> I would never want to use something like ollama in a production setting

If you can't get access to "real" datacenter GPUs for any reason and essentially do desktop, clientside deploys, it's your best bet.

It's not a common scenario, but a desktop with a 4090 or two is all you can get in some organizations.