From: "Paweł Staszewski" <pstaszewski@itcare.pl>
To: Linux Network Development list <netdev@vger.kernel.org>
Subject: Problem with ixgbe and TX locked on one cpu
Date: Fri, 14 Oct 2011 21:18:54 +0200 [thread overview]
Message-ID: <4E988B1E.5000606@itcare.pl> (raw)
[-- Attachment #1: Type: text/plain, Size: 4338 bytes --]
Hello
I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
Statistics for my ethernet - ixgbe driver:
ethtool -S eth4
NIC statistics:
rx_packets: 5815535848808
tx_packets: 5811202378421
rx_bytes: 4791001750842200
tx_bytes: 4781190419358301
rx_pkts_nic: 5815535848827
tx_pkts_nic: 5811202378510
rx_bytes_nic: 4837563124411799
tx_bytes_nic: 4829987507084013
lsc_int: 8
tx_busy: 0
non_eop_descs: 0
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 92494273
broadcast: 268718206
rx_no_buffer_count: 28829
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
rx_fifo_errors: 0
rx_missed_errors: 307051074
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_timeout_count: 0
tx_restart_queue: 15926219
rx_long_length_errors: 298
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 54173917
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
tx_queue_0_packets: 68694825
tx_queue_0_bytes: 9443750332
tx_queue_1_packets: 8410961
tx_queue_1_bytes: 2527763233
tx_queue_2_packets: 14411252
tx_queue_2_bytes: 1317132394
tx_queue_3_packets: 15013508147
tx_queue_3_bytes: 17364767277348
tx_queue_4_packets: 62779891
tx_queue_4_bytes: 63476596221
tx_queue_5_packets: 11176001
tx_queue_5_bytes: 2763600253
tx_queue_6_packets: 4416357
tx_queue_6_bytes: 611874984
tx_queue_7_packets: 8933405
tx_queue_7_bytes: 1837198524
tx_queue_8_packets: 13292669
tx_queue_8_bytes: 3241333510
tx_queue_9_packets: 10747236
tx_queue_9_bytes: 1805109931
tx_queue_10_packets: 5795935258380
tx_queue_10_bytes: 4763725304722245
tx_queue_11_packets: 12073934
tx_queue_11_bytes: 2982743045
tx_queue_12_packets: 10523764
tx_queue_12_bytes: 2637451199
tx_queue_13_packets: 12480552
tx_queue_13_bytes: 2434827407
tx_queue_14_packets: 7401777
tx_queue_14_bytes: 2413618099
tx_queue_15_packets: 8269270
tx_queue_15_bytes: 2854359576
rx_queue_0_packets: 361373769507
rx_queue_0_bytes: 298565751248279
rx_queue_1_packets: 369901571908
rx_queue_1_bytes: 303414679798160
rx_queue_2_packets: 362508961738
rx_queue_2_bytes: 299852439447157
rx_queue_3_packets: 363449272013
rx_queue_3_bytes: 299738390792515
rx_queue_4_packets: 361876234461
rx_queue_4_bytes: 297483366939732
rx_queue_5_packets: 361402926316
rx_queue_5_bytes: 297633876486533
rx_queue_6_packets: 362261522767
rx_queue_6_bytes: 298026696344647
rx_queue_7_packets: 361248593301
rx_queue_7_bytes: 296756459279986
rx_queue_8_packets: 361654143416
rx_queue_8_bytes: 298272433659520
rx_queue_9_packets: 362781764710
rx_queue_9_bytes: 298804803191595
rx_queue_10_packets: 361386593064
rx_queue_10_bytes: 297434987797644
rx_queue_11_packets: 369886597895
rx_queue_11_bytes: 302353350171712
rx_queue_12_packets: 361582732276
rx_queue_12_bytes: 298670408005971
rx_queue_13_packets: 365248093536
rx_queue_13_bytes: 302573023878287
rx_queue_14_packets: 366571142073
rx_queue_14_bytes: 302396739276514
rx_queue_15_packets: 362401929830
rx_queue_15_bytes: 299024344526029
The problem is with queue 10
tx_queue_10_packets: 5795935258380
tx_queue_10_bytes: 4763725304722245
as you can see most of the queue processing is used in queue 10
Average difference is 1,854271229903958e-6 - compared to other queues
and the problem is that almost all TX packet processing is on one CPU
cat /proc/interrupts - in attached file
Is this driver or kernel problem ?
Kernel is: 2.6.38.2
ixgbe driver is:
ethtool -i eth4
driver: ixgbe
version: 3.2.9-k2
firmware-version: 1.12-2
bus-info: 0000:04:00.0
Thanks
Pawel
--
[-- Attachment #2: interrupts.txt --]
[-- Type: text/plain, Size: 3675 bytes --]
cat /proc/interrupts | grep eth4
135: 3109261876 4289060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-0
136: 2738300312 2654348120 4055848 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-1
137: 43 2636245312 3776381478 4281702 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-2
138: 340 2156086460 3340495 3269054231 4487452 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-3
139: 38 0 2738519426 0 1088719123 4176363 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-4
140: 39 0 2632858749 3512903 0 2307156010 4310322 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-5
141: 41 0 0 2655130571 0 0 2492897896 4249569 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-6
142: 173 0 0 2625727263 0 0 0 2509835335 8038276 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-7
143: 44 0 0 0 2115559 0 0 0 3275187626 5066092 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-8
144: 51 0 0 0 0 0 0 0 2668196538 1238317991 4373599 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-9
145: 528852 0 0 0 386077 0 0 0 438158 294605430 1867115075 4806187 0 0 0 0 PCI-MSI-edge eth4-TxRx-10
146: 65 0 0 0 0 0 0 0 0 2378013639 3357280 1179087288 4668439 0 0 0 PCI-MSI-edge eth4-TxRx-11
147: 83 0 0 0 0 0 0 0 0 0 2447343915 0 1621496283 4718715 0 0 PCI-MSI-edge eth4-TxRx-12
148: 64 0 0 0 0 0 0 0 0 0 2719008413 3788138 0 2492359875 4697458 0 PCI-MSI-edge eth4-TxRx-13
149: 41 0 0 0 0 0 0 0 0 0 0 2569764726 0 0 3216415633 4546711 PCI-MSI-edge eth4-TxRx-14
150: 45 0 0 0 0 0 0 0 0 0 0 2553654902 0 0 0 2587543392 PCI-MSI-edge eth4-TxRx-15
151: 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 PCI-MSI-edge eth4:lsc
[-- Attachment #3: pstaszewski.vcf --]
[-- Type: text/x-vcard, Size: 336 bytes --]
begin:vcard
fn;quoted-printable:Pawe=C5=82 Staszewski
n;quoted-printable:Staszewski;Pawe=C5=82
org:ITCare
adr;quoted-printable;quoted-printable;dom:;;Sikorskiego 22;Libi=C4=85=C5=BC;Ma=C5=82opolskie;32-590
title:IT Manager
tel;work:+48 32 7203681
tel;fax:+48 32 7203682
tel;cell:+48 0 609911040
url:www.itcare.pl
version:2.1
end:vcard
next reply other threads:[~2011-10-14 19:25 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-14 19:18 Paweł Staszewski [this message]
2011-10-18 18:57 ` Problem with ixgbe and TX locked on one cpu Jesse Brandeburg
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
2011-10-18 21:23 ` Eric Dumazet
2011-10-18 21:34 ` Jim Sansing
2011-10-19 4:10 ` Eric Dumazet
2011-10-20 10:30 ` Pablo Neira Ayuso
2011-10-19 9:21 ` Problem with ixgbe and TX locked on one cpu Paweł Staszewski
2011-10-19 9:21 ` Paweł Staszewski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E988B1E.5000606@itcare.pl \
--to=pstaszewski@itcare.pl \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.