I think it might be better to say that in the days when the meaning of the word "hacker" was still entirely up in the air, getting at the insides of things (especially infrastructural things) was often associated with being "one of the bad guys."
And the "thing" (for lack of a better word) about computing (and also to some extension all sorts of other fields) that interested the relevant loosely-defined subculture consisted, while certainly not entirely, at least substantially of the insides of things. So you ended up dealing with security even if that wasn't your primary interest.
That's about as clearly as I can put it, anyway. Please do note that I'm not saying it's a good thing.
And the "thing" (for lack of a better word) about computing (and also to some extension all sorts of other fields) that interested the relevant loosely-defined subculture consisted, while certainly not entirely, at least substantially of the insides of things. So you ended up dealing with security even if that wasn't your primary interest.
That's about as clearly as I can put it, anyway. Please do note that I'm not saying it's a good thing.