And if your neighbor is bigger and stronger than you? If 100 of your neighbors get together to take your stuff? If 100 of your neighbors independently want your stuff?

What if the government wants your stuff or 20-50% of it?

At the absolute barest minimum, even disregarding any context, it's better to have to one known entity demanding a share than an unlimited number of unknown ones doing the same.

If it makes no difference who is robbing or extorting you, I'd go with the ones I have a better chance of defending myself from: my neighbors.