The question itself stems from confusion about what's being seen. It's not uncommon, especially with children, for people to become slightly confused while using a mirror. For instance, on seeing some smudge on their left cheek trying to wipe it from their right cheek. Once they start moving of course the error becomes obvious. Their initial understanding of the image is really what the image would be if you had a camera facing you and then used a video display as your mirror.
I have this problem all the time! I even shave by touch and then check in the mirror whether I've missed a spot rather than shave while looking in the mirror because I can't avoid being confused about this. I can, while looking at the mirror, reason about which way I need to move my hand to get at a certain spot on my face's reflection, but it's much faster to stop looking at the mirror and let (correct) instincts kick in.
I had my tongue pierced a few years back and even now (not that I do it very often), when looking in a mirror to put the ball back on the 'stick' (I don't know what it's called!) I drive myself nuts.
In one hand I'm trying to hold/'point' the 'stick' and in the other hand, not only am I trying to get the ball to the end of the stick, but at the same time trying to rotate it so that the hole in the ball lines up with the end of the stick. Everything points, moves and rolls totally against what I think it should be doing.
I, like you, find it easier to just do away with the mirror and try and work it out myself.
Having said that, this whole conversation has made me really think about the mirror and so I'm hoping that next time, it won't be so hard!