I agree, and to be honest I'd rather the TLS was optional because this would open Gemini to a lot of retro computers. Anyway, there is Mercury which is a simplified Gemini without encryption IIRC which solves that.
The important part is that you can treat the TLS as a black box and still hold the Gemini spec in your head. It is easy to implement by simply leveraging a TLS and a network lib. To implement a modern Web browser you need way more than that these days, which is why almost all browsers are chromium :-/ no company can justify the effort to develop new engines and clients anymore (my opinion).
If you want Gemini without TLS there is Gopher. LaGrange supports both Gemini and Gopher - and I believe there's at least one server out there that does serve both as well though I can't remember the name of it.
The important part is that you can treat the TLS as a black box and still hold the Gemini spec in your head. It is easy to implement by simply leveraging a TLS and a network lib. To implement a modern Web browser you need way more than that these days, which is why almost all browsers are chromium :-/ no company can justify the effort to develop new engines and clients anymore (my opinion).