Taleb calls this Minority Rule in his book Skin in the Game. The most intolerant wins because of asymmetry:

"The minority rule will show us how it all it takes is a small number of intolerant virtuous people with skin in the game, in the form of courage, for society to function properly. "