Maybe each site should be able to designate who indexes it and robots can get that index from that indexer. Let the indexers compete. Let each site decide how frequently it can index. Allow the indexer that gets the business use the index immediately, with others getting access just once a day. Perhaps a standardized raw sharable index format could be created, with each search company processing it further for their own needs after pulling it.

And let the site notify the indexer when things change, so all the bandwidth isn't used looking for what's changed. Actual changes could make it in to the index more quickly if the site could draw attention to it immediately rather than an army of robots having to invade as frequently as inhumanly possible. The selected indexer could still visit once a day or week to make sure nothing gets missed.