This is an interesting argument! Wouldn't it also work for positions, though? That is, either the universe is finite, or, since there can't be a uniform distribution over an infinite space of positions, there must be some preferred "center" of the universe?
You'd think, but of course we know that not to be the case. It's hard to pinpoint the exact reason though. Sure we know time and space are rather special, but its hard to say exactly why.
In the end though I reckon the most obvious reason is that speed is a property that directly corresponds to energy, therefore for each region of space to have a well defined energy (which is required for e.g. general relativity) every region of space needs to have a well defined distribution of speeds.
I suppose this does leave open a small loophole, as you can easily correlate speed with position in order to get a distribution that is uniform in both (but correlated). But this goes against our assumption that the universe is uniform everywhere (which might turn out to be false, but so far it's holding up well).
