I just finished implementing a simple backpressure algorithm. Once the queue is more than a certain percentage full, it rejects new items. These rejections get turned into HTTP 503s.
What is the purpose of cutting off the queue at a n% full? Isn't that effectively saying that the queue size is actually n% of the stated size, or am I missing detail?
Perhaps "rejects new items" is the key - a web server that hits a certain percentage utilization only allow existing clients/connections/logged in users/whatever to connect; refuse new connections.
Haven't you just moved the pressure back a step? Now you have a bunch of items retrying to get their stuff on the queue and a bunch of places to try and locate the problem, instead of an overly large queue. Is the idea that the messages aren't important enough to be placed on the queue in the first place if demand is high?
The idea is that it's not a sudden hard failure. Instead only a few things fail which helps in two ways: reducing the load by discarding some work and alerting that the system is beginning to be overloaded.
I think 429 is the preferred status code for this. Of course you can use whatever you want, but 503 can presumably come up in other scenarios and it's nice to have something unambiguous.
