The thing you have to realize is that it doesn't just happen to clueless grandmas. If someone has decided to devote 100% of their attention to tricking you, they will get you in the end. In practice, most move on to easier marks.