I've seen the flip side of the coin in an app I inherited at work - it turns out server-side rendering can be significantly slower than client side rendering if you're rendering a lot of content, to the point where we hit an automatic timeout. If this app was a fat JS app, we probably wouldn't be trying to decommission it & it probably would load in seconds at worst, but the investment to modernize it/improve performance & scalability so it would meet our users' needs is too much.
That’s true in a lot of cases mainly because of the approach to rendering. So many languages focus on rendering the entire HTML structure before sending the first byte back, specifically so that the page can be wrapped in a cache. This forces a lot of memory allocation to build up strings and parts of strings that takes time on its own, is often repetitive over the same HTML tags and then all of that has to be garbage collected at the end of the request.
The approach taken by Elixir’s Phoenix framework is so fast because it avoids all of those pitfalls. Most people have to do a double take when they see the speed comparisons.
Your older app may be serializing a lot of work. For instance, it could be issuing 10 SQL queries, one after another, which take 200ms each to answer - that's 2s total just waiting for the database. Compare to a JS client which issues 10 backend API calls, each being a 200ms request from the server's perspective (and the JS may parallelize many of those requests)
If this is a factor, rearchitecting the backend wouldn't necessarily be easier than an SPA rewrite. But it's not a fundamental limitation of server-side rendering.