I do the same with REST APIs and CLIs, and for the latter often build --help as pretty much the first thing (whether for real or just as a text document).
I recently was adding a couple features to an existing utility, where I made some new options (--opt3 --opt4) following the style of what was there. As I was trying to write the help I realized there were several combinations that were totally invalid, and describing the situations where each could be used was really hard. It really boiled down to only about 3 unique situations, so I replaced all those options with a single --mode switch, making it way simpler to code, document and use. Without docs first that wouldn't have been obvious until I either was partway through coding (or maybe writing validation code), or worse, got bug reports about someone trying an invalid combination and it breaking in some strange way.
I'm (temporarily stalled) in the middle of writing a CLI manpage for roughly this kind of purpose (finding a public interface that makes me happy before finally cutting the first release).
It has been clarifying in some big ways, but I also feel like I wouldn't have been able to do this without an initial implementation that I and one or two others have been putting through the paces.
It's taken some time to use it in enough real-world cases that I feel like we've actually discovered a significant fraction of the features it needs.
It takes reflecting on each feature for long enough to understand if it is global or situational, to decide if it's just a change to the program, a new default you can disable, or a new optional feature you must enable. Or is it really a fellow traveller with a number of related features that signal the need for a new mode?
Finding an interface that doesn't annoy me has taken collecting the entire current feature list, the next several planned features, and some "I suspect someone will ask for this some day" sorts of things--and then stepping far enough back from them to distill a smaller number of verbs they can be grouped under.
I write a /lot/ of cli apps for personal and work use and I’ve always wrote READMEs first in the same way and had wrote it off as me procrastinating. Now I can think I’m actually being more productive ;)
Basically create a readme with example of how to use the lib with explanation.
Then I design the library to match my design.
I see lots of libraries (especially in .net) that are missing that extra something as if they weren’t designed first.