Long hours are intrinsic to endurance sports, though. An avid cyclist or runner will easily spend ten hours or more training each week. By contrast, I doubt many people put that much into beer league softball or pick up basketball. (It's also not clear to me that rich people have less time than middle class or poor people.)
Rich people tend to travel a lot and have schedules which vary from week to week, which does make it hard to reliably show up for team practice. But they can do endurance sports on their own schedule.
Sorry, I realized my post was unclear/poorly worded. The primary issue is not just total time needed, but amount of time needed to synchronize schedules with other people.
More affluent people tend to have much more of their daily time scheduled, while less affluent people tend to be more flexible with their time - they can afford to just "hang out".
