Supposedly it's about ability to punish. If someone can spy on you but can't do something horrendous to you and get away with it, then it's harm is far less. Loopholes such as you describe would be the same, since it still will be indirect surveillance.

Of course in practice it's not.