Taxes don't go one way though. Governments provide plenty of useful services. If your government provides no useful services then that would be stealing, and you should probably move asap.
On a more local level this can apply in terms of city and state as well, so not just country. If you get literally no useful or beneficial services from a city government that's not a bad reason to move away unless you have no need for anything governmental.