More performant than using HDF5?

I doubt it- I didn’t do a systematic comparison, but stuck with this because it was fast enough that it wasn’t the bottleneck

I am working on something super similar. If you remember the name of the startup I would appreciate it deeply

looked at my phd files

https://stormforge.io/

enjoy please reach out to me if you want

Depending on the size of your application you can calculate flops by hand

https://docs.nersc.gov/tools/performance/roofline/

In the desert? It’s the driest place in the world

this good enough for you?

https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd...

It is not a reasonable assumption to compare your local cluster to the largest clusters within DOE or their equivalents in Europe/Japan. These machines regularly run at >90% utilization and you will not be given an allocation if you can’t prove that you’ll actually use the machine.

I do see the phenomenon you describe on smaller university clusters, but these are not power users who know how to leverage HPC to the highest capacity. People in DOE spend their careers working to use as much as these machines as efficiently as possible.

DOE already mandates this

osti.gov

Econ has a big working paper culture

PFAS is able to be removed via reverse osmosis which direct potable reuse facilities utilize, so not really a problem