Re: Major network performance regression in 3.7

Willy Tarreau
2013-01-06T00:50:53Z

On Sat, Jan 05, 2013 at 04:02:03PM -0800, Eric Dumazet wrote:
> On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote:
>
> > > 2) Another possibility would be that Myri card/driver doesnt like very
> > > well high order pages.
> >
> > It looks like it has not changed much since 3.6 :-/ I really suspect
> > something is wrong with memory allocation. I have tried reverting many
> > patches affecting the mm/ directory just in case but I did not come to
> > anything useful yet.
> >
>
> Hmm, I was referring to TCP stack now using order-3 pages instead of
> order-0 ones
>
> See commit 5640f7685831e088fe6c2e1f863a6805962f8e81
> (net: use a per task frag allocator)

OK, so you think there are two distinct problems ?

I have tried to revert this one but it did not change the performance, I'm
still saturating at ~6.9 Gbps.

> Could you please post :
>
> ethtool -S eth0

Yes, I've removed all zero counters in this short view for easier
reading (complete version appended at the end of this email). This
was after around 140 GB were transferred :

# ethtool -S eth1|grep -vw 0
NIC statistics:
rx_packets: 8001500
tx_packets: 10015409
rx_bytes: 480115998
tx_bytes: 148825674976
tx_boundary: 2048
WC: 1
irq: 45
MSI: 1
read_dma_bw_MBs: 1200
write_dma_bw_MBs: 1614
read_write_dma_bw_MBs: 2101
serial_number: 320061
link_changes: 2
link_up: 1
tx_pkt_start: 10015409
tx_pkt_done: 10015409
tx_req: 93407411
tx_done: 93407411
rx_small_cnt: 8001500
wake_queue: 187727
stop_queue: 187727
LRO aggregated: 146
LRO flushed: 146
LRO avg aggr: 1
LRO no_desc: 80

Quite honnestly, this is typically the pattern what I'm used to
observe here. I'm now trying to bisect, hopefully we'll get
something exploitable.

Cheers,
Willy

----- full ethtool -S ----

NIC statistics:
rx_packets: 8001500
tx_packets: 10015409
rx_bytes: 480115998
tx_bytes: 148825674976
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_boundary: 2048
WC: 1
irq: 45
MSI: 1
MSIX: 0
read_dma_bw_MBs: 1200
write_dma_bw_MBs: 1614
read_write_dma_bw_MBs: 2101
serial_number: 320061
watchdog_resets: 0
link_changes: 2
link_up: 1
dropped_link_overflow: 0
dropped_link_error_or_filtered: 0
dropped_pause: 0
dropped_bad_phy: 0
dropped_bad_crc32: 0
dropped_unicast_filtered: 0
dropped_multicast_filtered: 0
dropped_runt: 0
dropped_overrun: 0
dropped_no_small_buffer: 0
dropped_no_big_buffer: 0
----------- slice ---------: 0
tx_pkt_start: 10015409
tx_pkt_done: 10015409
tx_req: 93407411
tx_done: 93407411
rx_small_cnt: 8001500
rx_big_cnt: 0
wake_queue: 187727
stop_queue: 187727
tx_linearized: 0
LRO aggregated: 146
LRO flushed: 146
LRO avg aggr: 1
LRO no_desc: 80