cpoll-cppsp made huge performance gains from round 5 to 6. For example multiple queries went from 1,872 rps in round 5 to 7,252 in round 6. What caused that?
Also what happened to cpoll-cppsp in the plaintext test? Its performance there took a deep dive.
I know that the contributor of those tests made several changes. We weren't able to identify why it fails dramatically on the plaintext test, but I suspect it's a combination of the two unique characteristics of that test: the use of HTTP pipelining and higher client-side concurrency. My guess is that the pipelining in particular is causing problems.
I really hope you guys start doing more in-depth high-concurrency testing. This is a huge risk / problem with many frameworks and kits. It seems to be a blackhole in your tests for some reason, which is a shame because the are so well done in other regards.
Having a framework which falls down horribly at high concurrency is something people need to know about -- in how they structure and design applications and how they structure deploys. It can also help them pick frameworks that fit them better (some frameworks will do great at high concurrency, some won't).
Can't wait till you start adding high concurrency (which should NOT require any code changes from the implementation providers, you add a zeroes to the concurrency.. 100, 1000, 10000 tests).
Also what happened to cpoll-cppsp in the plaintext test? Its performance there took a deep dive.