I haven't tried XQuartz at all because I didn't think that kind of software has plenty of resource and is well maintained. Retina support is just one problem caused by lack of maintenance.
There are only two options: hack Virgil 3D or hack XQuartz. You cannot "just" run XQuartz and solve problems. And I choose hacking Virgil 3D because it should have less communication overhead and work with Wayland.
The main issue with XQuartz nowadays is that it doesn't support Retina displays in windowed mode. Everything runs pixel-doubled by default.