Your scenario seems to imply that for your empty building (and for every other empty building), there is an identical plot of land that is vacant, that will pay the same land value tax.

Your tax would go up in the following situations. Maybe there is very little, or no vacant land. Maybe every other land plot has a 30-story mixed use office tower/apartment building that is currently paying many multiples of your property tax.

The discussion about land value taxes often involve dense areas where these situations are more likely.