Re: Receive side performance issue with multi-10-GigE and NUMA

[Andrew Gallatin <gallatin@myri.com>]

Bill Fink wrote:

> <         Latency: 0, Cache Line Size: 64 bytes

<...>

>>       Latency: 0, Cache Line Size: 256 bytes


A cache line size of 256 clearly seems wrong for a Xeon.  I assume all
devices on the SuperMicro show the same value?

> Interestingly, the "WC Enabled" is only indicated on the first two

The WC is probably a red herring.

What does ethtool -S show for the DMA write bandwidth of the
NICs on the SuperMicro?

These values are obtained serially, as the driver resets
the NIC (reset happens at load time, and ifconfig up),
so they could easily sum to more than the memory bandwidth
of the system.  But it would be good to check for any anomalies.

I can send you a pointer to a tool we use internally, which loads
some custom firmware on the NIC, and can exercise the DMA engines
on all the NICs in parallel.  This would give an idea of the
aggregate DMA bandwidth available on the system.  Let me know
if you're interested.

Drew