You're probably right, there's definitely a spectrum of libertarianism.
And it's not as if libertarians are entirely crazy. It's healthy to have some baseline skepticism towards authority. But it's also healthy to have some baseline skepticism towards market solutions. As usual, the best answer(s) are somewhere in compromise and in the middle.
I guess that ideas like the "non-aggression principle" are so alluring to some because they have a sort of superficial "intellectual purity" which that kind of compromising answer lacks.
I agree that it's a baffling world view though.