One thing I don't understand with Windows Server is that it seems no matter how fast the nvme drives I use, or I pair/pool, I can't get a normal file copy to go faster than around 1.5GB/s (that's local, no network). The underlying disks show multi GB/s performance under crystal disk mark. But I suspect something in the OS must get in the way.
Your system ~3+ years old or so? Your story screams DMI 3.0 or similar PCH/PCIe switch saturation issue. DMI 3.0 caps at ~3.5GB/s but about 1.5-1.7 when bidirectional, such as drive to drive. If 100% reads hit about 3.5 then you’ve all but confirmed it. ~1.5GB/s bidirectional is a super common issue for a super common hardware combination.
It’ll happen if your U.2 ports route through DMI 3.0 PCH/Chipset/PCIe switch rather than directly to the CPU PCIe lanes. Easiest to just check motherboard manual, but you can use hwinfo to inspect the PCI tree and see if your U.2 ports are under a “chipset” labeled node. You might have different ports on the mobo that are direct, or possibly bios changes to explore. Sometimes lots of options, sometimes none. Worst case a direct PCIe adapter will resolve it.
Actually more than 3y old. My servers are EPYC Gen 2 and Gen 3 mostly.
But I don't think it is a hardware thing, as I see in CrystalDiskMark (which I believe bypasses the Windows write cache) performances that are close to the SSD specs. But it is when I do windows copy, the performance goes down the drain.
And it's not just parallelism. If I copy one file from a fast nvme to fast nvme, it caps at about 1.5GB/s. If I copy two files in parallel, they seem to split that speed, even if file1 is copied from diskA to diskB while file2 is copied from diskC to diskD.
Ahh. EPYC doesn't use DMI, so there’s an easy elimination, and they have enough PCIe lanes so that switches only come up with boards supporting oodles of drives. It’s still worth checking your mobo manual to make sure there isn’t a specific mention of port selection related to PCIe switches, or a silly bios option.
There might be some confusion that’s holding you back on diagnostics though. If a PCIe switch was in the path and causing the issue then there’s no meaningful difference between “parallel” and “bidirectional.” The nature of the switch is that all the traffic goes through it and it has a top speed and that’s split among the operations. Read or write to a single drive gets full bandwidth, but copying from one to another is read and write, so each part gets 50%. Even on the same drive, write/read and write/read also get 50% each. Include other drives on the same switch and divide it again. “And” = divide.
Your platform makes that specific issue less likely, but hardware can be a bit more quirky than you might be giving it credit.
Of course you’ve turned off windows defender real-time scanning, but otherwise I can’t think of an on-by-default reason for Windows Server to cause that issue, but it isn’t inherent in the OS. I’ve used multiple versions in dozen GB/s arrangements - 1.5GB/s was beatable 20 years ago. There’s something going on with your settings or hardware. Good news is it might be fixable, bad news is you’ll have to go trekking around to find the issue. :/
No PCIe switch involved, every SSD is directly connected to the PCIe ports, the MB supports bifurcation. Observed on H11SSL-i and H12SSL-i motherboards with various models of SSDs (mostly Intel, Micron and Samsung). Windows defender switched off.
Hyper-V is on though. And I understand that when you switch on Hyper-V, even the host OS is really a VM on top of the hypervisor. So I wonder if that's not contributing (but disabling Hyper-V is not an option).
When you say 1.5GB/s was beatable, how do you measure it? Windows copy (or xcopy) or some benchmarking software?
In addition to my other comments about parallel IO and unbuffered IO, be aware that WS2022 has (had?) a rather slow NVMe driver. It has been improved in WS2025.
If it's over SMB/Windows file sharing then you might be looking at some kind of latency-induced limit. AFAIK SMB doesn't stream uploads, they occur as a sequence of individual write operations, which I'm going to guess also produce an acknowledgement from the other end. It's possible something like this (say, client waiting for an ACK before issuing a new pending IO) is responsible
What does iperf say about your client/server combination? If it's capping out at the same level then networking, else something somewhere else in the stack.
I noticed recently that OS X file IO performance is absolute garbage because of all the extra protection functionality they've been piling into newer versions. No idea how any of it works, all I know is some background process burns CPU just from simple operations like recursively listing directories
The problem I describe is local (U.2 to U.2 SSD on the same machine, drives that could easily performs at 4GB/s read/write, and even when I pool them in RAID0 in arrays that can do 10GB/s).
Windows has weird behaviors for copying. Like if I pool some SAS or NVMe SSD in storage space parity (~RAID5) the performance in CrystalDiskMark is abyssal (~250MB/s) but a windows copy will be stable at about 1GB/s over terabytes of data.
So it seems that whatever they do hurts in certain cases and severely limits the upside as well.
