The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on).
I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.
Pcap captures are not multithreaded so you are pinning to a single core.
This entire thread is interesting because it is highlighting a similar problem with my virtualized router. When I pinned the router vm to specific cpus the problem goes away. I switched to openstack which doesn’t give me the best control over cpu capabilities and the problem has manifested in a worse form.
My uninformed opinion is that there are underlying concurrency problems with multithreaded user land-kernel interaction and some nic drivers (consumer intel and Broadcom hardware)
Ah ! Thanks for this.
Being single threaded does not prevent you from having your process being migrated from one core to another though, no? Or do you mean that pcap captures are pinned?
The original symptom was very low throughput, which is what prompted the investigation . Without tcpdump, low throughput and high reordering, with tcpdump, high throughput ( which is why I couldn't figure out what was going on).
I'd be very interested if someone with kernel experience could tell me what's specific about tcpdump.