Is any datacenter's water use significant compared to other industrial installations? According to that article, all datacenters in North Holland use 550 Ml/yr. North Holland has 2.95M residents [0], who use 129 l/person-day [1], 47 Kl/person-year, 139,000 Ml/year for the whole region. So the data centers use an estimated 0.4% of the region's water. Data centers use about 3% of the Netherlands' electricity.
Why do you think this is a lot of water? What are the alternatives to pulling from the local water utility and are those alternatives preferable?
> // Start a background task that takes the lock and holds it for a few seconds.
Holding a lock while waiting for IO can destroy a system's performance. With async Rust, we can prevent this by making the MutexGuard !Send, so it cannot be held across an await. Specifically, because it is !Send, it cannot be stored in the Future [2], so it must be dropped immediately, freeing the lock. This also prevents Futurelock deadlock.
This is how I wrote safina::sync::Mutex [0]. I did try to make it Send, like Tokio's MutexGuard, but stopped when I realized that it would become very complicated or require unsafe.
> You could imagine an unfair Mutex that always woke up all waiters and let them race to grab the lock again. That would not suffer from risk of futurelock, but it would have the thundering herd problem plus all the liveness issues associated with unfair synchronization primitives.
Thundering herd is when clients overload servers. This simple Mutex has O(n^2) runtime: every task must acquire and release the mutex, which adds all waiting tasks to the scheduler queue. In practice, scheduling a task is very fast (~600ns). As long as polling the lock-mutex-future is fast and you have <500 waiting tasks, then the O(n^2) runtime is fine.
Performance is hard to predict. I wrote Safina using the simplest possible implementations and assumed they would be slow. Then I wrote some micro-benchmarks and found that some parts (like the async Mutex) actually outperform Tokio's complicated versions [1]. I spent days coding optimizations that did not improve performance (work stealing) or even reduced performance (thread affinity). Now I'm hesitant to believe assumptions and predictions about performance, even if they are based on profiling data.
Considering this issue did also make me think - maybe the real footgun here is the async mutex. I think a better "rule" to avoid this issue might be something like, don't just use the tokio async mutex by default just because it's there and you're in an async function; instead default to a sync mutex that errors when held across awaits and think very hard about what you're really doing before you switch to the async one.
Actually I think I might be a little misguided here - confusing a mutex with an awaitable lock method versus blocking, and a mutex whose LockGuard is Send and can be held across other await points.
To clarify, I do still think it's probably wise to prefer using a mutex whose LockGuard is not Send. If you're in an async context though, it seems clearly preferable to use a mutex that lets you await on lock instead of possibly blocking. Looks like that's what that Safina gives you.
It does bring to mind the point though - does it really make sense to call all of these things Mutexes? Most Mutexes, including the one in std, seem relatively simplistic, with no provision for exactly what happens if multiple threads/tasks are waiting to acquire the lock. As if they're designed for the case of, it's probably rare to never for multiple threads to actually need this thing at once, but we have to guard against it just to be certain. The case of this resource is in high demand by a bunch of threads, we expect there to be a lot of time spent by a lot of threads waiting to get the lock, so it's actually important which lock requesters actually get the lock in what order, seems different enough that it maybe ought to have a different name and more flexibility and selection as to what algorithm is being used to control the lock order.
I would guess this is just to make the explanation of the bug easier.
In real world, the futurelock could occur even with very short locks, it just wouldn't be so deterministic. Having a minimal reproducer that you have to run a thousand times and it will maybe futurelock doesn't really make for a good example :)
>In real world, the futurelock could occur even with very short locks, it just wouldn't be so deterministic.
You have to explain the problem properly then. The problem here has nothing to do with duration whatsoever so don't bring that up. The problem here is that if you acquire a lock, you're inside a critical section. Critical sections have a programming paradigm that is equivalent to writing unsafe Rust. You're not allowed to panic inside unsafe Rust or inside critical sections. It's simply not allowed.
You're also not allowed to interrupt the critical section by something that does not have a hard guarantee that it will finish. This rules out await inside the critical section. You're not allowed to do await. It's simply not allowed. The only thing you're allowed to do is execute an instruction that guarantees that N-1 instructions are left to be executed, where N is a finite number. Alternatively you do the logical equivalent. You have a process that has a known finite bound on how long it will take to execute and you are waiting for that external process.
After that process has finished, you release the lock. Then you return to the scheduler and execute the next future. The next future cannot be blocked because the lock has already been released. It's simply impossible.
You now have to explain how the impossible happened. After all, by using the lock you've declared that you took all possible precautions to avoid interrupting the critical section. If you did not, then you deserve any bugs coming your way.
That's just how locks are.
I think you misunderstand the problem. The only purpose of the sleep in this example is to control interleaving of execution to ensure the problem happens. Here's a version where the background task (the initial lock holder) only runs a bounded number of instructions with the lock held, just as you suggest:
> After that process has finished, you release the lock. Then you return to the scheduler and execute the next future. The next future cannot be blocked because the lock has already been released. It's simply impossible.
This is true with threads and with tasks that only ever poll futures sequentially. It is not true in the various cases mentioned in this RFD (notably `tokio::select!`, but also others). Intuitively: when you have one task polling on multiple futures concurrently, you're essentially adding another layer to the scheduler (kernel thread scheduler, tokio task scheduler, now some task is acting as its own future scheduler). The problem is it's surprisingly easy to (1) not realize that and (2) accidentally have that "scheduler" not poll the next runnable future and then get stuck, just like if the kernel scheduler didn't wake up a runnable thread.
Work stealing is more a technique to function better when architecture is pessimal (think mixing slow and fast tasks in one queue) than something that make things go faster in general. It also tend to shuffle around complexity a bit, in ways that are sometimes nice.
Same thing with task preemption, though that one has less organisatorial impact.
In general, getting something to perform well enough on specific tasks is a lot easier than performing well enough on tasks in general. At the same time, most tasks have kinda specific needs when you start looking at them..
I spent some years working for a large NGO (Opportunity International) and living with people who work for NGOs.
NGOs must constantly raise money to fund their operations. The money that an NGO spends on fund-raising & administration is called "overhead". The percentage of annual revenue spent on overhead is the overhead percentage. Most NGOs publish this metric.
When a big donor stops contributing, the NGO must cut pay or lay off people and cut projects. I've never heard of an NGO "succumbing to excessive staff costs" like a startup running out of money. Financial mismanagement does occasionally happen and boards do replace CEOs. Board members are mostly donors, so they tend to donate more to help the NGO recover from mismanagement, instead of walking away.
NGOs pay less than other organizations, so they mostly attract workers who care about the NGO's mission. These are people with intrinsic motivation to make the NGO succeed in its mission. Financial incentives are a small part of their motivations. For example, my supervisor at Opportunity International refused several raises.
> So they go around addressing individual problems, taking sad pictures, and avoid addressing systemic problems.
Work on individual problems is valuable. For example, the Carter Center has prevented many millions of people from going blind from onchocerciasis and trachoma [0].
The Carter Center is not directly addressing the systemic problems of poverty and ineffective government health programs. That would take different expertise and different kinds of donors.
The world is extremely complicated and interconnected. The Carter Center's work preventing blindness directly supports worker productivity in many poor countries. Productivity helps economic growth and reduces poverty. And with more resources, government health programs run better.
Being effective in charity work requires humility and diligence to understand what can be done now, with the available resources. And then it requires tenacity to work in dangerous and backward places. It's an extremely hard job. People burn out. And we are all better off because of the work they do.
When we ignore the value of work on individual problems, because it doesn't address systemic problems, we practice binary thinking [1]. It's good to avoid binary thinking.
When I used AWS startup credits in 2019, the AWS console made it very difficult to estimate the bill after the credits ran out. I lost a lot of trust in AWS. Also, there were buried mines in the APIs, like the risk of bad logging running up a $70,000/day bill with CloudWatch Logs.
If I could go back and do it again, I would rent a single machine and deploy with ssh (git pull & docker-compose up) and backup to my laptop.
I think that async in Rust has a significant devex/velocity cost. Unfortunately, nearly all of the effort in Rust libraries has gone into async code, so the async libraries have outpaced the threaded libraries.
I wrote https://lib.rs/crates/servlin threaded web server. It uses async internally. It has 221K lines of code. Its hello-world example reaches 102Krps on my machine.
https://lib.rs/crates/ehttpd is another one but it has no tests and it seems abandoned. It does an impressive 113Krps without async, using only 8K lines of code.
For comparison, the popular Axum async web server has 4.3M lines of code and its hello-world example reaches 190Krps on my machine.
The popular threaded Postgres client uses Tokio internally and has 1M lines of code: http://lib.rs/postgres .
I took an "Architecting on AWS" class and half of the content was how to replicate complicated physical networking architectures on AWS's software-defined network: layers of VPCs, VPC peering, gateways, NATs, and impossible-to-debug firewall rules. AWS knows their customers tho. Without this, a lot of network engineers would block migrations from on-prem to AWS.
Ages ago I deployed a sophos virtual appliance in AWS, so I could centrally enforce some basic firewall rules, in a way that my management could understand. There was only 1 server behind it, the same thing could have been achieved simply using the standard built in security rules. I think about it often.
I do find Azures implementation of this stuff pretty baffling. Just in, networking concepts being digested by software engineers, and then regurgitated into a hierarchy that makes sense to them. Not impermeable, just weird.
I had a very interesting conversation with an AWS guy about how hard they tried to make sure things like Wireshark worked the same inside AWS, because they had some much pushback from network engineers that expected their jobs to be exactly the same inside as on-prem.
Main source of issues leading to overcomplex networking that I ever seen was "every VPC gets a 10./8" like approach replicated, so suddenly you have complex time trying to interconnect the networks later.
I think this part is somewhat legitimate. Every network engineer knows "it's always DNS," to the point that there are jokes about it. DNS is a brittle and inflexible protocol that works well when it's working, but unfortunately network engineers are the ones who get called when it's not.
A superior alternative to DNS would help a lot, but getting adoption for something at that level of the stack would be very hard.
I find that a lot of "it's always DNS" falls down to "I don't know routing beyond default gateway" and "I never learnt how to run DNS". Might be a tad elitist of me, I guess, but solid DHCP, routing, and DNS setup makes for way more reliable network than anything else.
DNS just tends to be part that is visible to random desktop user when things fail
>Might be a tad elitist of me, I guess, but solid DHCP, routing, and DNS setup makes for way more reliable network than anything else.
Depends on the network. If you are talking about a branch office, for sure.
>I find that a lot of "it's always DNS" falls down to "I don't know routing beyond default gateway"
I see it mostly with assumptions. Like DNS Server B MUST SURELY be configured the same as DNS Server A, thus my change will have no unexpected consequences.
Solid management of the services is important, yes. Also being prepared for when requirements change. I remember to this day when a bunch of small (rack-scale) deployments suddenly needed heavy-grade DNS because one of the deployed projects would generate a ton of DNS traffic. My predecessor set up dnsmasq, I didn't have a reason to change it before that, afterwards we had to setup total of 6 DNS servers per rack (1 primary authoritative, 2 secondary updating themselves from authoritative, 3 recursive).
I would say situation also changes a lot if you know/can deploy anycast routes for core network services - for example fc00::10-12 will always be recursive nameservers, and you configure routing so that it picks up the closest one, etc.
To handle errors in HTMX, I like to use config from [0] to swap responses into error dialogs and `hx-on-htmx-send-error` [1] and `hx-on-htmx-response-error` [2] to show the dialogs. For some components, I also use an `on-htmx-error` attribute handler:
VPN providers do not have reputations for making secure or reliable software.
Here's a good privacy proxy (VPN) setup: Set up a second wifi router, enable the "Internet kill switch", and connect it with Wireguard to a reputable VPN service. I recommend GL.iNet routers and Mullvad.
With this setup, one can move individual devices between the privacy wifi and identity-broadcasting wifi.
Why do you think this is a lot of water? What are the alternatives to pulling from the local water utility and are those alternatives preferable?
[0] https://en.wikipedia.org/wiki/North_Holland
[1] https://en.wikipedia.org/wiki/Water_supply_and_sanitation_in...
[2] https://www.dutchdatacenters.nl/en/statistics-2/