HN maintains that this is unethical because the users in your contact list have not consented to having their phone numbers shared with a social networking service.
You could use a key derivation function (scrypt or pbkdf2) to expand it out and make it quite difficult to even rainbow table it. You could also compound the effort by only storing a hash of the pairs (one hash or derived key of two phone numbers) with no associated metadata directly linked to it, only as an access gate. Collisions may occur but at least the positive results from a total search space would be significantly reduced.
I Am Not A Cryptographer, so I admit there is likely faulty reasoning in there somewhere, though.
The point is that the hash is effectively reversible at that scale. 10 numeric digits is 10 billion possibilities. It's actually far less than that because area codes don't span the entire 3-digit number range, and most area codes are not heavily populated. But even so, creating a lookup table by computing 10 billion hashes is today a comparatively easy task, even for expensive hash algorithms. It's a matter of only a few seconds to minutes work with relatively common hardware (high performance GPUs).