Also have a single 4K 28"

Can confirm that is has to be DisplayPort

"Oh cool, 4K@60Hz KVMs aren't expensive anymore," I thought.

And then about 120 dollars in cables and adapters later, I can even get the 4K@60Hz signal to the monitor!

Never had that issue with DP before. Never had it negotiate to a spec it couldn't transmit. Never had it drop out 30s in.

HDMI is a mess.

Indeed. What HDMI can technically on paper or what's in the specification poorly reflects the real world, sometimes.