No ILS is not an ATC rule, it's simply a way to get more flights to land (increasing airline/airport profits) and putting responsibility on the pilot (not ATC). Lufthansa has the policy exactly so that pilots do not get bullied into taking a visual approach, by ATC.
Most pilots would prefer a visual approach given the choice. I doubt this pilot actually believed it would be unsafe to fly the visual at SFO that evening: given the dozens of other jets actively doing it, that would be absurd. He was just following his company policy. If he had a real emergency that required him to land immediately, I'm almost certain he'd have flown that emergency landing visually.
I don't know for sure... but on a modern jet, I suspect he was able to contact his employer during that long hold to ask, and they told him "yes, we really want you to divert if you can't get the ILS". Either way, if there's any fault here, IMHO it's on his employer not doing their homework and putting their pilot in a no-win situation.
