It's not hard, but it's a manual process which is always prone to errors or oversights. Having to support lots of versions just smells a little fishy to me, maybe justifiable if you have some type of firmware/hardware tie-in.
Why can't you, if you need to hotfix, git checkout the tag from master then git branch -b to something designating that it's a hotfix branch, and then when the hotfix is verified git tag the head of that branch with a specific version?
You can then merge that hotfix branch down into the head of master, or into any other later versions by doing a similar branch from the newer version tag first. Essentially it makes version branches lazy in that they're only created when they're actually needed, but you can still support lots of older versions to your hearts content with bugfixes, hotpatches and minor updates.
The big downside to this strategy, though, is that since you don't have a version branch until you decide you need it, new features for testing can't be merged into master until the current version is release ready (ie all outstanding bugs are fixed).
This isn't really a huge issue for smaller teams/companies or ones that can keep a fast continuous development pace, especially if your release schedule is short, because at most the merge freeze is only a couple days. But this will result in developers needing to handle merge conflicts if their feature branch exists for more than a single deployment cycle. IMO you want to keep the merge window to no more than the number of weeks between deployments in days (eg 1 week between deployments requires a merge freeze no longer than 1 day, and 3 weeks requires a merge freeze no longer than 3 days).
However, it's also really easy to grow from this into what this article describes as the team grows or the testing/deployment/merge freeze window lengthens, because you can switch from waiting to need a hotpatch for making a version branch to making a version branch every time you'd otherwise begin a merge freeze. You do have to remember to merge the commit tagged with that release back into master so that master gets all the bug fixes, but that's no more onerous than requiring hotpatches get merged into master as well.
Are there any reasons at all for using git flow over trunk based development with release branches? The latter seems far simpler but just as flexible.