Fiberglass is at least pretty easy one to figure out. Don't have to get that stuff in your skin more than once to realize it's probably really, really bad to breathe.
My understanding is that fiberglass and mineral wool [1] are classified as skin (yes, it itches), eye, and lung irritants (coughing), but they're not carcinogenic. Considering how extremely widely they have been used for a long time, if they were we'd most likely have seen it by now.
Sure it's not good for you, so using PPE is certainly warranted, but it's on a whole different level of badness compared to asbestos.
[1] Per wikipedia, there are some varieties of mineral wool used for high temperatures (think insulating industrial furnaces and such) which are carcinogens.
