As an aside, it would be fun to see a programming challenge website focused on performance and optimizations, rather than code golf (shortest program) or edge case correctness (interview type sites). I took a course like this in uni where we learned low-level optimization and got to compete with other classmates to see who had the fastest program - a fun experience that I don't really see much of online.
The course had a machine solely for benchmarking. It would only compile and benchmark one program at a time and the students had to queue up.
It worked well for the course and the results were very consistent between runs, but that kind of setup doesn't scale well.
There may be a (slow) option though, and that would be benchmarking the code using an simulated processor. The runtime would be measured relative to the simulated processor as opposed to the computer hosting the simulator.
