They don't have to always work well. If Alice's mask stops half the particles and Bob's mask stops half the particles, the chance of Alice infecting Bob is now 4x lower. 50% is not as good as 95%, but surely better than 0%.
All we need to do is bring the reproductive number below 1, not necessarily all the way to 0.
Healthcare workers wear masks. I'm pretty sure they have a pretty good idea whether masks work.