This is a quite good example of the kind of problems you run into when you follow the philosophy of representing all data in informally-specified ad-hoc text formats. Everyone thinks they can just roll their own parser/serialiser, which they then neglect to test thoroughly enough, creating subtle bugs when the serialisation side forgets to escape data somewhere, or the parsing side doesn't even provide any way to escape grammar-significant characters.
No, the problem is the lack of a standard encoder function to go with the standard decoder function. It's not "everyone thinks they can", it's "everyone has to".
The problem is also YAGNI and validation-thru-testing, instead of up-front design. The "ad-hoc" and "text" parts aren't what's important, it's the whole approach of not doing any more than the bare minimum of up-front work. Which seems to historically give overall better results, even if it does come with interesting bugs that need fixing later.
Well, the real problem here is that there's not really a 'standard' you can rely on in the first place.
And while it's true that text isn't a necessary part of the general problem, in my experience text-based formats seem especially prone to it. How many times have you seen people attempt to use regular expressions to parse HTML/validate e-mail addresses/whatever?
I think this indicates a huge difference between Microsoft/UNIX mindsets. Microsoft allowed
rename *.txt *.bak
To do this, the "rename" command had to understand how to parse the asterisk character while being familiar with the contents of the directory.
However, creating a new replacement "rename" command is difficult, as well as creating new commands that can parse wildcards.
In the Unix environment, the shell expands the asterisk to all files that matches that pattern, and then passes these files to the "rename" command, who never sees the asterisk.
Therefore it's trivial to create a new "rename' utility because it doesn't need to parse wildcards. However, renaming all .txt to .bak is awkward in a UNIX system.
It may be awkward but it's no use to ruin a perfect system for just one usecase. For what it's worth, zsh provides a capable renamer tool called zmv. An example:
zmv -W '*.txt' '*.markdown'
And of course there are tools like rename(1) that work regardless of the shell used.
That's not at all user-friendly, but these are supposed to be programmer tools... If you want user-friendly file operations on UNIX command line, use midnight commander. It can do mass rename, etc.
I don't see how the problems you're listing have anything to do with text formats; you have these kind of issues with any "informally-specified" format.
