Maybe, but I don't think that most people who say this are trying to belittle their idol, rather complement them on the fact that they have a greater skill than others. That no matter how much time someone else puts into it, they will never be as good.
And there is some truth to this. You are born a good jocky, most people couldn't train hard to be that small.
I'm not saying it's always conscious, or that this applies to everything. But I think it has a lot to do with how you see the world, and whether you have what Carol Dweck would call a growth mindset or a fixed mindset.