It's not a map/territory confusion. It's a warning about map/territory confusion. The problem is that we have only maps, so we cannot fully understand the territory.
This is a major concern in mathematics, whose reason for existence is to understand things precisely, not just rough and ready.
I'm not convinced there is a coherent "territory" when it comes to mathematics. Sometimes (remarkably!) mathematical concepts neatly line up with some part of the universe, and we gain some insight into how it works. But often that part of the universe lies entirely within somebody's mind - mathematics as an hobby or aesthetic pursuit, for instance, occupied with questions of pure logic.
Different ways of thinking about the same thing can lead to conflicting truths even without set theory getting in the way. The standard model of the real numbers has no infinitesimally small elements, for instance, but the hyperreal numbers do, despite satisfying all of the same first-order properties.
