Our house has an asbestos flue in the bathroom. I'm very careful never to go near that thing, and never ever to cut/drill/attempt to remove it myself. But I wonder how many people would never know it was asbestos.

If you leave asbestos alone it's safe, and if you need to drill into it, use shaving cream so that the fibers don't get into the air. It's not a big deal if you are careful. But don't be reckless - you need to know what you are drilling into and do it slowly.

Source? Usually it is just tradesmen who worked with it directly, or their wives.

This is an interesting claim.

How many is plenty and what are the sources to back this?

This is not factual.

Asbestos is not kryptonite. One time exposure is not going to have short term or long term impact to your health.

There is a lot of FUD around asbestos, check out all of the panicked posts on reddit.