There actually has been work done on it, partially by me.
Unfortunately it's simply an unsolved problem to properly federate the 3pid mappings. The long and short of it is that you need to find some way to prove to other identity servers that you properly verified a 3pid. That's theoretically possible with e-mail and DKIM, except there are too many e-mail servers that don't implement it correctly.
For phone numbers it's even worse.
[edit]
I forgot: There is a working implementation of federated identity servers for e-mail, but that requires you to run an identity server on the domain of the e-mail address that you're trying to add as 3pid. That means it works fine if you host your own e-mail but it doesn't work for the gmails and hotmails of this world.
Unfortunately it's simply an unsolved problem to properly federate the 3pid mappings. The long and short of it is that you need to find some way to prove to other identity servers that you properly verified a 3pid. That's theoretically possible with e-mail and DKIM, except there are too many e-mail servers that don't implement it correctly.
For phone numbers it's even worse.
[edit] I forgot: There is a working implementation of federated identity servers for e-mail, but that requires you to run an identity server on the domain of the e-mail address that you're trying to add as 3pid. That means it works fine if you host your own e-mail but it doesn't work for the gmails and hotmails of this world.