Agreed. I saw I got minimally downvoted for what I thought was a comment that really added to the conversation. It's disheartening to be downvoted simply because somebody doesn't agree with you.
That happens all the time. Don't fret about it, just ignore it and apply your votes the way you think things should work.
Over time those few downvotes are almost always balanced by the rest of HN.
Except for 'controversial' subjects, there anything can happen.
If you want to avoid being downvoted completely the easiest way is to avoid such subjects, specifically anything mentioning Ayn Rand, the glory of Capitalism, religion and emacs ;)