This is why I have a completely separate private key that I generated for my phone, using a different password that's unrelated to the passwords I use for all my other machines' private keys, and I use password protection on the phone itself to thwart casual intrusion. If I lose my phone, it's trivial to remove the phone's public key from my servers' authorized_keys files, at which point the phone can no longer be used to log in.

Encrypt your private key!

This is very easy to do, and quite important for devices that can be stolen - including your desktop computer.

It's easy enough to setup ssh-agent on your desktop so you don't have to keep entering the password.

If you have a copy of the private key and the public key, you can bruteforce the passphrase. So use a big passphrase instead of a password.