That's what I used to think but now that I see that there are that many chip makers contributing I'm starting to wonder. But it's probably as varikin says, they're working on different kinds of drivers.
Much of the hardware for the higher end Android devices is composed of stuff from a handful of OEMs, so that also helps. Also many of the devices that come out around the same time are using at least some identical components.
SoCs are all mostly Qualcomm or Samsung, radio chipsets are mostly Qualcomm, LCDs are Sony/Samsung/Sharp/LG and a few others, AMOLED are all Samsung as far as I know, GPUs are mostly Qualcomm/PowerVR/Nvidia.
Much of the time, the Android modding community ports newer versions of Android to other devices simply by hoping some OEM releases a newer version of Android for another device that happens to have some identical hardware component so they can either drop that particular driver into the other device or tweak it a bit. Porting the OS is never really the problem. The problems generally lie in outdated proprietary drivers no longer working with the newer version of Android (Camera/Bluetooth/Radio/sometimes GPU are the most likely candidates to break).
