Do any lisps have a "language server"? They kinda are their own language server right?

If you mean something like SLIME/SLY, than yeah, but I guess I never considered that a language server in the same way as all the other LSP backends.

Why not? There is an external IDE (SLIME in GNU Emacs) which interacts with a language server to get information about syntax highlighting, completion, documentation, source locations, ...

The idea of an 'inferior Lisp' connected to a GNU Emacs has a long tradition. Usually one did communicate with a Lisp as a process on the same machine. That was long before SLIME. Popular were ILISP (1990) or ELI (for Allegro CL).

The purpose is a more general, providing evaluation/compilation, backtraces, inspectors, browsers, etc.

Guiser fills the role of a language server for Guile and a handful of other Schemes.

Yes, there are a handful of Geiser implementations, but they vary differ in how feature-complete they are. Guile's implementation is the best supported and it's pretty great, but with Chez or MIT you'll have a trickier time.