For the remaining few who have nothing to hide I am willing to make myself available to rifle through your possessions to verify that you do in fact have nothing to hide.
But then you'd have unilateral coercive power over me. It needs to be a two-way street for either of us to be sure the other isn't a bad actor. I'll happily broadcast my entire (boring) existence so long as everyone else does the same. End the privacy arms race. It's exhausting and frankly immature.
The phrase doesn't mean, "I have nothing to hide from anyone at any time for any reason."
It means, "I have nothing to hide from authorities who are looking at my data in an official capacity, so I'm willing to cede my privacy to those particular actors in that specific context."
It might still be a bad argument, but your retort doesn't address it.
This is it. The people I talked with explained this to me exactly like this - they have nothing to hide from the authorities. Nor do they care that "someone in China" has access to things like the floor plan of their home. Baffling to me, but this is how they think, and if we plan to express our point, OP's aggressive line of "ok then let me rifle through your private stuff" won't work.