Using GitHub for a CDN would be hotlinking to a .js file from your production website, not whats happening here. Other package managers do the same, for example "bower" will clone from github, it just does normal clones.
I think you're missing my distinction here. Yes, bower will clone from GitHub, but of course only the packages you actually use; in the same sense that Go can import source from GitHub repos. These repos are storing code and using GitHub as intended.
But CocoaPods/Specs is the equivalent of someone collecting all possible packages you could ever use in Bower in one big GitHub repo.