Immigration and tax evasion generally don't directly hurt someone who'd be pissed off that the government let the crime happen, unlike e.g. theft, robbery, rape, or even just loitering around drunk in public.
I'm not talking about philosophy, I'm talking about what gets people angry at the government.
Imagine that you're running a pub. There's a regular that dresses funny, runs a small show, drinks like a hole, and at the end of the day, always runs out without paying his bill. He's stealing from you. However, there are many people who go to the pub just to watch his antics. The food and drinks he steals are nothing compared to the extra money you make from that.
You have every right to kick him out, and you might do it out of spite because you will not tolerate thieves... but the pragmatic thing to do is to look the other way, so it's not surprising when that's what happens.
Your hypothetical is hollow sophistry while taxation is both law and social contract. Miss me with that "I'm not talking about philosophy, now imagine you're a cowboy riding a dinosaur" bollocks.
