I think this is a false equivalence. The content of SVGs is non-semantic; to get from SVG code to Meaning one needs to do some sort of rendering and re-interpreting of the resulting image. Even if for some simple images that could be done in-brain, that is not usually the case. Whereas something like Mermaid is intended to be meaningful both as code and as rendered output, just like Markdown itself. Having that additional meaning inline can be very helpful to faster understanding of the content, which is not usually going to be the case for inline'd SVG.
I don't know if it's a false equivalence, because I've read (and certainly written) a fair number of mermaid diagrams which are extremely hard to understand without seeing the rendered content.
It's subjective, in my opinion. I might be fine with inline SVGs if they were "smaller", ie didn't have the doctype and root element and general XML verbosity on all nodes. PlantUML / Mermaid have a much more to-the-point syntax.
Also, I personally hate markdown tables since editing even a single cell means that I likely have to then resize either the entire row or the entire column, or both. So when I do need tables I use HTML tables anyway.
At any rate, a middle-ground might be to have the diagram source as references at the end of the document, and reference those from the use site, like `![Chart][#chart]` or something. So you don't have to see the diagram source inline if you don't like it, but it's still there in the same file.