* Problem with ixgbe and TX locked on one cpu
@ 2011-10-14 19:18 Paweł Staszewski
2011-10-18 18:57 ` Jesse Brandeburg
0 siblings, 1 reply; 9+ messages in thread
From: Paweł Staszewski @ 2011-10-14 19:18 UTC (permalink / raw)
To: Linux Network Development list
[-- Attachment #1: Type: text/plain, Size: 4338 bytes --]
Hello
I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
Statistics for my ethernet - ixgbe driver:
ethtool -S eth4
NIC statistics:
rx_packets: 5815535848808
tx_packets: 5811202378421
rx_bytes: 4791001750842200
tx_bytes: 4781190419358301
rx_pkts_nic: 5815535848827
tx_pkts_nic: 5811202378510
rx_bytes_nic: 4837563124411799
tx_bytes_nic: 4829987507084013
lsc_int: 8
tx_busy: 0
non_eop_descs: 0
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 92494273
broadcast: 268718206
rx_no_buffer_count: 28829
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
rx_fifo_errors: 0
rx_missed_errors: 307051074
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_timeout_count: 0
tx_restart_queue: 15926219
rx_long_length_errors: 298
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 54173917
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
tx_queue_0_packets: 68694825
tx_queue_0_bytes: 9443750332
tx_queue_1_packets: 8410961
tx_queue_1_bytes: 2527763233
tx_queue_2_packets: 14411252
tx_queue_2_bytes: 1317132394
tx_queue_3_packets: 15013508147
tx_queue_3_bytes: 17364767277348
tx_queue_4_packets: 62779891
tx_queue_4_bytes: 63476596221
tx_queue_5_packets: 11176001
tx_queue_5_bytes: 2763600253
tx_queue_6_packets: 4416357
tx_queue_6_bytes: 611874984
tx_queue_7_packets: 8933405
tx_queue_7_bytes: 1837198524
tx_queue_8_packets: 13292669
tx_queue_8_bytes: 3241333510
tx_queue_9_packets: 10747236
tx_queue_9_bytes: 1805109931
tx_queue_10_packets: 5795935258380
tx_queue_10_bytes: 4763725304722245
tx_queue_11_packets: 12073934
tx_queue_11_bytes: 2982743045
tx_queue_12_packets: 10523764
tx_queue_12_bytes: 2637451199
tx_queue_13_packets: 12480552
tx_queue_13_bytes: 2434827407
tx_queue_14_packets: 7401777
tx_queue_14_bytes: 2413618099
tx_queue_15_packets: 8269270
tx_queue_15_bytes: 2854359576
rx_queue_0_packets: 361373769507
rx_queue_0_bytes: 298565751248279
rx_queue_1_packets: 369901571908
rx_queue_1_bytes: 303414679798160
rx_queue_2_packets: 362508961738
rx_queue_2_bytes: 299852439447157
rx_queue_3_packets: 363449272013
rx_queue_3_bytes: 299738390792515
rx_queue_4_packets: 361876234461
rx_queue_4_bytes: 297483366939732
rx_queue_5_packets: 361402926316
rx_queue_5_bytes: 297633876486533
rx_queue_6_packets: 362261522767
rx_queue_6_bytes: 298026696344647
rx_queue_7_packets: 361248593301
rx_queue_7_bytes: 296756459279986
rx_queue_8_packets: 361654143416
rx_queue_8_bytes: 298272433659520
rx_queue_9_packets: 362781764710
rx_queue_9_bytes: 298804803191595
rx_queue_10_packets: 361386593064
rx_queue_10_bytes: 297434987797644
rx_queue_11_packets: 369886597895
rx_queue_11_bytes: 302353350171712
rx_queue_12_packets: 361582732276
rx_queue_12_bytes: 298670408005971
rx_queue_13_packets: 365248093536
rx_queue_13_bytes: 302573023878287
rx_queue_14_packets: 366571142073
rx_queue_14_bytes: 302396739276514
rx_queue_15_packets: 362401929830
rx_queue_15_bytes: 299024344526029
The problem is with queue 10
tx_queue_10_packets: 5795935258380
tx_queue_10_bytes: 4763725304722245
as you can see most of the queue processing is used in queue 10
Average difference is 1,854271229903958e-6 - compared to other queues
and the problem is that almost all TX packet processing is on one CPU
cat /proc/interrupts - in attached file
Is this driver or kernel problem ?
Kernel is: 2.6.38.2
ixgbe driver is:
ethtool -i eth4
driver: ixgbe
version: 3.2.9-k2
firmware-version: 1.12-2
bus-info: 0000:04:00.0
Thanks
Pawel
--
[-- Attachment #2: interrupts.txt --]
[-- Type: text/plain, Size: 3675 bytes --]
cat /proc/interrupts | grep eth4
135: 3109261876 4289060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-0
136: 2738300312 2654348120 4055848 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-1
137: 43 2636245312 3776381478 4281702 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-2
138: 340 2156086460 3340495 3269054231 4487452 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-3
139: 38 0 2738519426 0 1088719123 4176363 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-4
140: 39 0 2632858749 3512903 0 2307156010 4310322 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-5
141: 41 0 0 2655130571 0 0 2492897896 4249569 0 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-6
142: 173 0 0 2625727263 0 0 0 2509835335 8038276 0 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-7
143: 44 0 0 0 2115559 0 0 0 3275187626 5066092 0 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-8
144: 51 0 0 0 0 0 0 0 2668196538 1238317991 4373599 0 0 0 0 0 PCI-MSI-edge eth4-TxRx-9
145: 528852 0 0 0 386077 0 0 0 438158 294605430 1867115075 4806187 0 0 0 0 PCI-MSI-edge eth4-TxRx-10
146: 65 0 0 0 0 0 0 0 0 2378013639 3357280 1179087288 4668439 0 0 0 PCI-MSI-edge eth4-TxRx-11
147: 83 0 0 0 0 0 0 0 0 0 2447343915 0 1621496283 4718715 0 0 PCI-MSI-edge eth4-TxRx-12
148: 64 0 0 0 0 0 0 0 0 0 2719008413 3788138 0 2492359875 4697458 0 PCI-MSI-edge eth4-TxRx-13
149: 41 0 0 0 0 0 0 0 0 0 0 2569764726 0 0 3216415633 4546711 PCI-MSI-edge eth4-TxRx-14
150: 45 0 0 0 0 0 0 0 0 0 0 2553654902 0 0 0 2587543392 PCI-MSI-edge eth4-TxRx-15
151: 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 PCI-MSI-edge eth4:lsc
[-- Attachment #3: pstaszewski.vcf --]
[-- Type: text/x-vcard, Size: 336 bytes --]
begin:vcard
fn;quoted-printable:Pawe=C5=82 Staszewski
n;quoted-printable:Staszewski;Pawe=C5=82
org:ITCare
adr;quoted-printable;quoted-printable;dom:;;Sikorskiego 22;Libi=C4=85=C5=BC;Ma=C5=82opolskie;32-590
title:IT Manager
tel;work:+48 32 7203681
tel;fax:+48 32 7203682
tel;cell:+48 0 609911040
url:www.itcare.pl
version:2.1
end:vcard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem with ixgbe and TX locked on one cpu
2011-10-14 19:18 Problem with ixgbe and TX locked on one cpu Paweł Staszewski
@ 2011-10-18 18:57 ` Jesse Brandeburg
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Jesse Brandeburg @ 2011-10-18 18:57 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Linux Network Development list, e1000-devel
CC: e1000-devel
2011/10/14 Paweł Staszewski <pstaszewski@itcare.pl>:
> Hello
>
> I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
what application are you running, how are you using ixgbe? looks like
a router. is something changing the skb->rx_queue entry (like
netfilter) or is there a layered device above ixgbe (bonding or ...) ?
why do your interrupts move after a long period? did you do it by
hand? we recommend disabling irqbalance and hand tuning interrupts
possibly with the set_irq_affinity.sh script.
> Statistics for my ethernet - ixgbe driver:
> ethtool -S eth4
> NIC statistics:
> rx_packets: 5815535848808
> tx_packets: 5811202378421
> rx_bytes: 4791001750842200
> tx_bytes: 4781190419358301
> rx_pkts_nic: 5815535848827
> tx_pkts_nic: 5811202378510
> rx_bytes_nic: 4837563124411799
> tx_bytes_nic: 4829987507084013
> lsc_int: 8
> tx_busy: 0
> non_eop_descs: 0
> rx_errors: 0
> tx_errors: 0
> rx_dropped: 0
> tx_dropped: 0
> multicast: 92494273
> broadcast: 268718206
> rx_no_buffer_count: 28829
> collisions: 0
> rx_over_errors: 0
> rx_crc_errors: 0
> rx_frame_errors: 0
> hw_rsc_aggregated: 0
> hw_rsc_flushed: 0
> fdir_match: 0
> fdir_miss: 0
> rx_fifo_errors: 0
> rx_missed_errors: 307051074
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> tx_timeout_count: 0
> tx_restart_queue: 15926219
> rx_long_length_errors: 298
> rx_short_length_errors: 0
> tx_flow_control_xon: 0
> rx_flow_control_xon: 0
> tx_flow_control_xoff: 0
> rx_flow_control_xoff: 0
> rx_csum_offload_errors: 54173917
> alloc_rx_page_failed: 0
> alloc_rx_buff_failed: 0
> rx_no_dma_resources: 0
> tx_queue_0_packets: 68694825
> tx_queue_0_bytes: 9443750332
> tx_queue_1_packets: 8410961
> tx_queue_1_bytes: 2527763233
> tx_queue_2_packets: 14411252
> tx_queue_2_bytes: 1317132394
> tx_queue_3_packets: 15013508147
> tx_queue_3_bytes: 17364767277348
> tx_queue_4_packets: 62779891
> tx_queue_4_bytes: 63476596221
> tx_queue_5_packets: 11176001
> tx_queue_5_bytes: 2763600253
> tx_queue_6_packets: 4416357
> tx_queue_6_bytes: 611874984
> tx_queue_7_packets: 8933405
> tx_queue_7_bytes: 1837198524
> tx_queue_8_packets: 13292669
> tx_queue_8_bytes: 3241333510
> tx_queue_9_packets: 10747236
> tx_queue_9_bytes: 1805109931
> tx_queue_10_packets: 5795935258380
> tx_queue_10_bytes: 4763725304722245
> tx_queue_11_packets: 12073934
> tx_queue_11_bytes: 2982743045
> tx_queue_12_packets: 10523764
> tx_queue_12_bytes: 2637451199
> tx_queue_13_packets: 12480552
> tx_queue_13_bytes: 2434827407
> tx_queue_14_packets: 7401777
> tx_queue_14_bytes: 2413618099
> tx_queue_15_packets: 8269270
> tx_queue_15_bytes: 2854359576
> rx_queue_0_packets: 361373769507
> rx_queue_0_bytes: 298565751248279
> rx_queue_1_packets: 369901571908
> rx_queue_1_bytes: 303414679798160
> rx_queue_2_packets: 362508961738
> rx_queue_2_bytes: 299852439447157
> rx_queue_3_packets: 363449272013
> rx_queue_3_bytes: 299738390792515
> rx_queue_4_packets: 361876234461
> rx_queue_4_bytes: 297483366939732
> rx_queue_5_packets: 361402926316
> rx_queue_5_bytes: 297633876486533
> rx_queue_6_packets: 362261522767
> rx_queue_6_bytes: 298026696344647
> rx_queue_7_packets: 361248593301
> rx_queue_7_bytes: 296756459279986
> rx_queue_8_packets: 361654143416
> rx_queue_8_bytes: 298272433659520
> rx_queue_9_packets: 362781764710
> rx_queue_9_bytes: 298804803191595
> rx_queue_10_packets: 361386593064
> rx_queue_10_bytes: 297434987797644
> rx_queue_11_packets: 369886597895
> rx_queue_11_bytes: 302353350171712
> rx_queue_12_packets: 361582732276
> rx_queue_12_bytes: 298670408005971
> rx_queue_13_packets: 365248093536
> rx_queue_13_bytes: 302573023878287
> rx_queue_14_packets: 366571142073
> rx_queue_14_bytes: 302396739276514
> rx_queue_15_packets: 362401929830
> rx_queue_15_bytes: 299024344526029
>
> The problem is with queue 10
> tx_queue_10_packets: 5795935258380
> tx_queue_10_bytes: 4763725304722245
>
> as you can see most of the queue processing is used in queue 10
> Average difference is 1,854271229903958e-6 - compared to other queues
>
> and the problem is that almost all TX packet processing is on one CPU
> cat /proc/interrupts - in attached file
>
> Is this driver or kernel problem ?
>
> Kernel is: 2.6.38.2
>
> ixgbe driver is:
> ethtool -i eth4
> driver: ixgbe
> version: 3.2.9-k2
> firmware-version: 1.12-2
> bus-info: 0000:04:00.0
^ permalink raw reply [flat|nested] 9+ messages in thread
* Comment on nf_queue NF_STOLEN patch
2011-10-18 18:57 ` Jesse Brandeburg
@ 2011-10-18 19:08 ` Jim Sansing
2011-10-18 21:23 ` Eric Dumazet
2011-10-19 9:21 ` Problem with ixgbe and TX locked on one cpu Paweł Staszewski
2011-10-19 9:21 ` Paweł Staszewski
2 siblings, 1 reply; 9+ messages in thread
From: Jim Sansing @ 2011-10-18 19:08 UTC (permalink / raw)
To: Linux Network Development list
I have been working on a kernel module that registers with netfilter,
and I noticed that a patch was added to nf_queue that changed the
handling of return code NF_FILTER from 'do nothing' to 'free the skb'.
I'm not sure which kernel version this went in, but the date of the
patch is Feb, 19, 2010.
Everything I have read about netfilter states that it is up to the
netfilter hook to free the skb if NF_STOLEN is returned. The
implications of this patch from a hook programming perspective are:
1) If the skb is used after the return from the hook, it must be cloned.
2) The original skb must not be freed.
I suggest that a comment be added to include/linux/netfilter.h that says
explicitly the skb will be freed if NF_STOLEN is returned.
Later . . . Jim
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Comment on nf_queue NF_STOLEN patch
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
@ 2011-10-18 21:23 ` Eric Dumazet
2011-10-18 21:34 ` Jim Sansing
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2011-10-18 21:23 UTC (permalink / raw)
To: Jim Sansing; +Cc: Linux Network Development list
Le mardi 18 octobre 2011 à 15:08 -0400, Jim Sansing a écrit :
> I have been working on a kernel module that registers with netfilter,
> and I noticed that a patch was added to nf_queue that changed the
> handling of return code NF_FILTER from 'do nothing' to 'free the skb'.
> I'm not sure which kernel version this went in, but the date of the
> patch is Feb, 19, 2010.
>
> Everything I have read about netfilter states that it is up to the
> netfilter hook to free the skb if NF_STOLEN is returned. The
> implications of this patch from a hook programming perspective are:
>
> 1) If the skb is used after the return from the hook, it must be cloned.
> 2) The original skb must not be freed.
>
> I suggest that a comment be added to include/linux/netfilter.h that says
> explicitly the skb will be freed if NF_STOLEN is returned.
But its not true. Just read the code.
If you are working on this stuff I recommend you take a look at
commits :
c6675233f9015d3c0460c8aab53ed9b99d915c64
(netfilter: nf_queue: reject NF_STOLEN verdicts from userspace)
fad54440438a7c231a6ae347738423cbabc936d9
(netfilter: avoid double free in nf_reinject)
64507fdbc29c3a622180378210ecea8659b14e40
(netfilter: nf_queue: fix NF_STOLEN skb leak)
3bc38712e3a6e0596ccb6f8299043a826f983701
([NETFILTER]: nf_queue: handle NF_STOP and unknown verdicts in
nf_reinject)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Comment on nf_queue NF_STOLEN patch
2011-10-18 21:23 ` Eric Dumazet
@ 2011-10-18 21:34 ` Jim Sansing
2011-10-19 4:10 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Jim Sansing @ 2011-10-18 21:34 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Network Development list
Eric Dumazet wrote:
> Le mardi 18 octobre 2011 à 15:08 -0400, Jim Sansing a écrit :
>
>> I have been working on a kernel module that registers with netfilter,
>> and I noticed that a patch was added to nf_queue that changed the
>> handling of return code NF_FILTER from 'do nothing' to 'free the skb'.
>> I'm not sure which kernel version this went in, but the date of the
>> patch is Feb, 19, 2010.
>>
>> Everything I have read about netfilter states that it is up to the
>> netfilter hook to free the skb if NF_STOLEN is returned. The
>> implications of this patch from a hook programming perspective are:
>>
>> 1) If the skb is used after the return from the hook, it must be cloned.
>> 2) The original skb must not be freed.
>>
>> I suggest that a comment be added to include/linux/netfilter.h that says
>> explicitly the skb will be freed if NF_STOLEN is returned.
>>
>
> But its not true. Just read the code.
>
> If you are working on this stuff I recommend you take a look at
> commits :
>
> c6675233f9015d3c0460c8aab53ed9b99d915c64
> (netfilter: nf_queue: reject NF_STOLEN verdicts from userspace)
>
> fad54440438a7c231a6ae347738423cbabc936d9
> (netfilter: avoid double free in nf_reinject)
>
> 64507fdbc29c3a622180378210ecea8659b14e40
> (netfilter: nf_queue: fix NF_STOLEN skb leak)
>
> 3bc38712e3a6e0596ccb6f8299043a826f983701
> ([NETFILTER]: nf_queue: handle NF_STOP and unknown verdicts in
> nf_reinject)
>
>
I see that fad54440438a7c231a6ae347738423cbabc936d9 (netfilter: avoid
double free in nf_reinject) returns the switch case for NF_STOLEN back
to the original state, but I just downloaded 3.0.4, and the skb is still
freed. So for some versions of the kernel, the situation exists.
Hopefully anyone who runs into it will find this thread.
Later . . . Jim
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Comment on nf_queue NF_STOLEN patch
2011-10-18 21:34 ` Jim Sansing
@ 2011-10-19 4:10 ` Eric Dumazet
2011-10-20 10:30 ` Pablo Neira Ayuso
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2011-10-19 4:10 UTC (permalink / raw)
To: Jim Sansing
Cc: Linux Network Development list, Netfilter Development Mailinglist,
Florian Westphal
Le mardi 18 octobre 2011 à 17:34 -0400, Jim Sansing a écrit :
> Eric Dumazet wrote:
> > Le mardi 18 octobre 2011 à 15:08 -0400, Jim Sansing a écrit :
> >
> >> I have been working on a kernel module that registers with netfilter,
> >> and I noticed that a patch was added to nf_queue that changed the
> >> handling of return code NF_FILTER from 'do nothing' to 'free the skb'.
> >> I'm not sure which kernel version this went in, but the date of the
> >> patch is Feb, 19, 2010.
> >>
> >> Everything I have read about netfilter states that it is up to the
> >> netfilter hook to free the skb if NF_STOLEN is returned. The
> >> implications of this patch from a hook programming perspective are:
> >>
> >> 1) If the skb is used after the return from the hook, it must be cloned.
> >> 2) The original skb must not be freed.
> >>
> >> I suggest that a comment be added to include/linux/netfilter.h that says
> >> explicitly the skb will be freed if NF_STOLEN is returned.
> >>
> >
> > But its not true. Just read the code.
> >
> > If you are working on this stuff I recommend you take a look at
> > commits :
> >
> > c6675233f9015d3c0460c8aab53ed9b99d915c64
> > (netfilter: nf_queue: reject NF_STOLEN verdicts from userspace)
> >
> > fad54440438a7c231a6ae347738423cbabc936d9
> > (netfilter: avoid double free in nf_reinject)
> >
> > 64507fdbc29c3a622180378210ecea8659b14e40
> > (netfilter: nf_queue: fix NF_STOLEN skb leak)
> >
> > 3bc38712e3a6e0596ccb6f8299043a826f983701
> > ([NETFILTER]: nf_queue: handle NF_STOP and unknown verdicts in
> > nf_reinject)
> >
> >
>
> I see that fad54440438a7c231a6ae347738423cbabc936d9 (netfilter: avoid
> double free in nf_reinject) returns the switch case for NF_STOLEN back
> to the original state, but I just downloaded 3.0.4, and the skb is still
> freed. So for some versions of the kernel, the situation exists.
> Hopefully anyone who runs into it will find this thread.
>
Hopefully netfilter guys (CCed) will sort out the problem and ask stable
submissions, if not already done. 3.0.4 is quite old :)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem with ixgbe and TX locked on one cpu
2011-10-18 18:57 ` Jesse Brandeburg
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
@ 2011-10-19 9:21 ` Paweł Staszewski
2011-10-19 9:21 ` Paweł Staszewski
2 siblings, 0 replies; 9+ messages in thread
From: Paweł Staszewski @ 2011-10-19 9:21 UTC (permalink / raw)
To: Jesse Brandeburg; +Cc: e1000-devel, Linux Network Development list
[-- Attachment #1.1: Type: text/plain, Size: 5738 bytes --]
W dniu 2011-10-18 20:57, Jesse Brandeburg pisze:
> CC: e1000-devel
>
> 2011/10/14 Paweł Staszewski<pstaszewski@itcare.pl>:
>> Hello
>>
>> I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
> what application are you running, how are you using ixgbe? looks like
> a router. is something changing the skb->rx_queue entry (like
> netfilter) or is there a layered device above ixgbe (bonding or ...) ?
Yes it is a router. - Quagga - BGP
There is few (10rules) iptables rules in INPUT - and nothing more
There are also namespaces running - and traffic is processed in namespaces.
no bonding
> why do your interrupts move after a long period? did you do it by
> hand? we recommend disabling irqbalance and hand tuning interrupts
> possibly with the set_irq_affinity.sh script.
Yes it was changed by hand to check why TX is processed on one interrupt.
Also now it is impossible to move this TX traffic to other interrupt.
All configuration for affinity is made by hand - no irqbalance running.
>> Statistics for my ethernet - ixgbe driver:
>> ethtool -S eth4
>> NIC statistics:
>> rx_packets: 5815535848808
>> tx_packets: 5811202378421
>> rx_bytes: 4791001750842200
>> tx_bytes: 4781190419358301
>> rx_pkts_nic: 5815535848827
>> tx_pkts_nic: 5811202378510
>> rx_bytes_nic: 4837563124411799
>> tx_bytes_nic: 4829987507084013
>> lsc_int: 8
>> tx_busy: 0
>> non_eop_descs: 0
>> rx_errors: 0
>> tx_errors: 0
>> rx_dropped: 0
>> tx_dropped: 0
>> multicast: 92494273
>> broadcast: 268718206
>> rx_no_buffer_count: 28829
>> collisions: 0
>> rx_over_errors: 0
>> rx_crc_errors: 0
>> rx_frame_errors: 0
>> hw_rsc_aggregated: 0
>> hw_rsc_flushed: 0
>> fdir_match: 0
>> fdir_miss: 0
>> rx_fifo_errors: 0
>> rx_missed_errors: 307051074
>> tx_aborted_errors: 0
>> tx_carrier_errors: 0
>> tx_fifo_errors: 0
>> tx_heartbeat_errors: 0
>> tx_timeout_count: 0
>> tx_restart_queue: 15926219
>> rx_long_length_errors: 298
>> rx_short_length_errors: 0
>> tx_flow_control_xon: 0
>> rx_flow_control_xon: 0
>> tx_flow_control_xoff: 0
>> rx_flow_control_xoff: 0
>> rx_csum_offload_errors: 54173917
>> alloc_rx_page_failed: 0
>> alloc_rx_buff_failed: 0
>> rx_no_dma_resources: 0
>> tx_queue_0_packets: 68694825
>> tx_queue_0_bytes: 9443750332
>> tx_queue_1_packets: 8410961
>> tx_queue_1_bytes: 2527763233
>> tx_queue_2_packets: 14411252
>> tx_queue_2_bytes: 1317132394
>> tx_queue_3_packets: 15013508147
>> tx_queue_3_bytes: 17364767277348
>> tx_queue_4_packets: 62779891
>> tx_queue_4_bytes: 63476596221
>> tx_queue_5_packets: 11176001
>> tx_queue_5_bytes: 2763600253
>> tx_queue_6_packets: 4416357
>> tx_queue_6_bytes: 611874984
>> tx_queue_7_packets: 8933405
>> tx_queue_7_bytes: 1837198524
>> tx_queue_8_packets: 13292669
>> tx_queue_8_bytes: 3241333510
>> tx_queue_9_packets: 10747236
>> tx_queue_9_bytes: 1805109931
>> tx_queue_10_packets: 5795935258380
>> tx_queue_10_bytes: 4763725304722245
>> tx_queue_11_packets: 12073934
>> tx_queue_11_bytes: 2982743045
>> tx_queue_12_packets: 10523764
>> tx_queue_12_bytes: 2637451199
>> tx_queue_13_packets: 12480552
>> tx_queue_13_bytes: 2434827407
>> tx_queue_14_packets: 7401777
>> tx_queue_14_bytes: 2413618099
>> tx_queue_15_packets: 8269270
>> tx_queue_15_bytes: 2854359576
>> rx_queue_0_packets: 361373769507
>> rx_queue_0_bytes: 298565751248279
>> rx_queue_1_packets: 369901571908
>> rx_queue_1_bytes: 303414679798160
>> rx_queue_2_packets: 362508961738
>> rx_queue_2_bytes: 299852439447157
>> rx_queue_3_packets: 363449272013
>> rx_queue_3_bytes: 299738390792515
>> rx_queue_4_packets: 361876234461
>> rx_queue_4_bytes: 297483366939732
>> rx_queue_5_packets: 361402926316
>> rx_queue_5_bytes: 297633876486533
>> rx_queue_6_packets: 362261522767
>> rx_queue_6_bytes: 298026696344647
>> rx_queue_7_packets: 361248593301
>> rx_queue_7_bytes: 296756459279986
>> rx_queue_8_packets: 361654143416
>> rx_queue_8_bytes: 298272433659520
>> rx_queue_9_packets: 362781764710
>> rx_queue_9_bytes: 298804803191595
>> rx_queue_10_packets: 361386593064
>> rx_queue_10_bytes: 297434987797644
>> rx_queue_11_packets: 369886597895
>> rx_queue_11_bytes: 302353350171712
>> rx_queue_12_packets: 361582732276
>> rx_queue_12_bytes: 298670408005971
>> rx_queue_13_packets: 365248093536
>> rx_queue_13_bytes: 302573023878287
>> rx_queue_14_packets: 366571142073
>> rx_queue_14_bytes: 302396739276514
>> rx_queue_15_packets: 362401929830
>> rx_queue_15_bytes: 299024344526029
>>
>> The problem is with queue 10
>> tx_queue_10_packets: 5795935258380
>> tx_queue_10_bytes: 4763725304722245
>>
>> as you can see most of the queue processing is used in queue 10
>> Average difference is 1,854271229903958e-6 - compared to other queues
>>
>> and the problem is that almost all TX packet processing is on one CPU
>> cat /proc/interrupts - in attached file
>>
>> Is this driver or kernel problem ?
>>
>> Kernel is: 2.6.38.2
>>
>> ixgbe driver is:
>> ethtool -i eth4
>> driver: ixgbe
>> version: 3.2.9-k2
>> firmware-version: 1.12-2
>> bus-info: 0000:04:00.0
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
[-- Attachment #2: Type: text/plain, Size: 377 bytes --]
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
[-- Attachment #3: Type: text/plain, Size: 257 bytes --]
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problem with ixgbe and TX locked on one cpu
2011-10-18 18:57 ` Jesse Brandeburg
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
2011-10-19 9:21 ` Problem with ixgbe and TX locked on one cpu Paweł Staszewski
@ 2011-10-19 9:21 ` Paweł Staszewski
2 siblings, 0 replies; 9+ messages in thread
From: Paweł Staszewski @ 2011-10-19 9:21 UTC (permalink / raw)
To: Jesse Brandeburg; +Cc: Linux Network Development list, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 5739 bytes --]
W dniu 2011-10-18 20:57, Jesse Brandeburg pisze:
> CC: e1000-devel
>
> 2011/10/14 Paweł Staszewski<pstaszewski@itcare.pl>:
>> Hello
>>
>> I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
> what application are you running, how are you using ixgbe? looks like
> a router. is something changing the skb->rx_queue entry (like
> netfilter) or is there a layered device above ixgbe (bonding or ...) ?
Yes it is a router. - Quagga - BGP
There is few (10rules) iptables rules in INPUT - and nothing more
There are also namespaces running - and traffic is processed in namespaces.
no bonding
> why do your interrupts move after a long period? did you do it by
> hand? we recommend disabling irqbalance and hand tuning interrupts
> possibly with the set_irq_affinity.sh script.
Yes it was changed by hand to check why TX is processed on one interrupt.
Also now it is impossible to move this TX traffic to other interrupt.
All configuration for affinity is made by hand - no irqbalance running.
>> Statistics for my ethernet - ixgbe driver:
>> ethtool -S eth4
>> NIC statistics:
>> rx_packets: 5815535848808
>> tx_packets: 5811202378421
>> rx_bytes: 4791001750842200
>> tx_bytes: 4781190419358301
>> rx_pkts_nic: 5815535848827
>> tx_pkts_nic: 5811202378510
>> rx_bytes_nic: 4837563124411799
>> tx_bytes_nic: 4829987507084013
>> lsc_int: 8
>> tx_busy: 0
>> non_eop_descs: 0
>> rx_errors: 0
>> tx_errors: 0
>> rx_dropped: 0
>> tx_dropped: 0
>> multicast: 92494273
>> broadcast: 268718206
>> rx_no_buffer_count: 28829
>> collisions: 0
>> rx_over_errors: 0
>> rx_crc_errors: 0
>> rx_frame_errors: 0
>> hw_rsc_aggregated: 0
>> hw_rsc_flushed: 0
>> fdir_match: 0
>> fdir_miss: 0
>> rx_fifo_errors: 0
>> rx_missed_errors: 307051074
>> tx_aborted_errors: 0
>> tx_carrier_errors: 0
>> tx_fifo_errors: 0
>> tx_heartbeat_errors: 0
>> tx_timeout_count: 0
>> tx_restart_queue: 15926219
>> rx_long_length_errors: 298
>> rx_short_length_errors: 0
>> tx_flow_control_xon: 0
>> rx_flow_control_xon: 0
>> tx_flow_control_xoff: 0
>> rx_flow_control_xoff: 0
>> rx_csum_offload_errors: 54173917
>> alloc_rx_page_failed: 0
>> alloc_rx_buff_failed: 0
>> rx_no_dma_resources: 0
>> tx_queue_0_packets: 68694825
>> tx_queue_0_bytes: 9443750332
>> tx_queue_1_packets: 8410961
>> tx_queue_1_bytes: 2527763233
>> tx_queue_2_packets: 14411252
>> tx_queue_2_bytes: 1317132394
>> tx_queue_3_packets: 15013508147
>> tx_queue_3_bytes: 17364767277348
>> tx_queue_4_packets: 62779891
>> tx_queue_4_bytes: 63476596221
>> tx_queue_5_packets: 11176001
>> tx_queue_5_bytes: 2763600253
>> tx_queue_6_packets: 4416357
>> tx_queue_6_bytes: 611874984
>> tx_queue_7_packets: 8933405
>> tx_queue_7_bytes: 1837198524
>> tx_queue_8_packets: 13292669
>> tx_queue_8_bytes: 3241333510
>> tx_queue_9_packets: 10747236
>> tx_queue_9_bytes: 1805109931
>> tx_queue_10_packets: 5795935258380
>> tx_queue_10_bytes: 4763725304722245
>> tx_queue_11_packets: 12073934
>> tx_queue_11_bytes: 2982743045
>> tx_queue_12_packets: 10523764
>> tx_queue_12_bytes: 2637451199
>> tx_queue_13_packets: 12480552
>> tx_queue_13_bytes: 2434827407
>> tx_queue_14_packets: 7401777
>> tx_queue_14_bytes: 2413618099
>> tx_queue_15_packets: 8269270
>> tx_queue_15_bytes: 2854359576
>> rx_queue_0_packets: 361373769507
>> rx_queue_0_bytes: 298565751248279
>> rx_queue_1_packets: 369901571908
>> rx_queue_1_bytes: 303414679798160
>> rx_queue_2_packets: 362508961738
>> rx_queue_2_bytes: 299852439447157
>> rx_queue_3_packets: 363449272013
>> rx_queue_3_bytes: 299738390792515
>> rx_queue_4_packets: 361876234461
>> rx_queue_4_bytes: 297483366939732
>> rx_queue_5_packets: 361402926316
>> rx_queue_5_bytes: 297633876486533
>> rx_queue_6_packets: 362261522767
>> rx_queue_6_bytes: 298026696344647
>> rx_queue_7_packets: 361248593301
>> rx_queue_7_bytes: 296756459279986
>> rx_queue_8_packets: 361654143416
>> rx_queue_8_bytes: 298272433659520
>> rx_queue_9_packets: 362781764710
>> rx_queue_9_bytes: 298804803191595
>> rx_queue_10_packets: 361386593064
>> rx_queue_10_bytes: 297434987797644
>> rx_queue_11_packets: 369886597895
>> rx_queue_11_bytes: 302353350171712
>> rx_queue_12_packets: 361582732276
>> rx_queue_12_bytes: 298670408005971
>> rx_queue_13_packets: 365248093536
>> rx_queue_13_bytes: 302573023878287
>> rx_queue_14_packets: 366571142073
>> rx_queue_14_bytes: 302396739276514
>> rx_queue_15_packets: 362401929830
>> rx_queue_15_bytes: 299024344526029
>>
>> The problem is with queue 10
>> tx_queue_10_packets: 5795935258380
>> tx_queue_10_bytes: 4763725304722245
>>
>> as you can see most of the queue processing is used in queue 10
>> Average difference is 1,854271229903958e-6 - compared to other queues
>>
>> and the problem is that almost all TX packet processing is on one CPU
>> cat /proc/interrupts - in attached file
>>
>> Is this driver or kernel problem ?
>>
>> Kernel is: 2.6.38.2
>>
>> ixgbe driver is:
>> ethtool -i eth4
>> driver: ixgbe
>> version: 3.2.9-k2
>> firmware-version: 1.12-2
>> bus-info: 0000:04:00.0
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
[-- Attachment #2: pstaszewski.vcf --]
[-- Type: text/x-vcard, Size: 336 bytes --]
begin:vcard
fn;quoted-printable:Pawe=C5=82 Staszewski
n;quoted-printable:Staszewski;Pawe=C5=82
org:ITCare
adr;quoted-printable;quoted-printable;dom:;;Sikorskiego 22;Libi=C4=85=C5=BC;Ma=C5=82opolskie;32-590
title:IT Manager
tel;work:+48 32 7203681
tel;fax:+48 32 7203682
tel;cell:+48 0 609911040
url:www.itcare.pl
version:2.1
end:vcard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Comment on nf_queue NF_STOLEN patch
2011-10-19 4:10 ` Eric Dumazet
@ 2011-10-20 10:30 ` Pablo Neira Ayuso
0 siblings, 0 replies; 9+ messages in thread
From: Pablo Neira Ayuso @ 2011-10-20 10:30 UTC (permalink / raw)
To: Eric Dumazet
Cc: Jim Sansing, Linux Network Development list,
Netfilter Development Mailinglist, Florian Westphal
On Wed, Oct 19, 2011 at 06:10:35AM +0200, Eric Dumazet wrote:
> Le mardi 18 octobre 2011 à 17:34 -0400, Jim Sansing a écrit :
> > Eric Dumazet wrote:
> > > Le mardi 18 octobre 2011 à 15:08 -0400, Jim Sansing a écrit :
> > >
> > >> I have been working on a kernel module that registers with netfilter,
> > >> and I noticed that a patch was added to nf_queue that changed the
> > >> handling of return code NF_FILTER from 'do nothing' to 'free the skb'.
> > >> I'm not sure which kernel version this went in, but the date of the
> > >> patch is Feb, 19, 2010.
> > >>
> > >> Everything I have read about netfilter states that it is up to the
> > >> netfilter hook to free the skb if NF_STOLEN is returned. The
> > >> implications of this patch from a hook programming perspective are:
> > >>
> > >> 1) If the skb is used after the return from the hook, it must be cloned.
> > >> 2) The original skb must not be freed.
> > >>
> > >> I suggest that a comment be added to include/linux/netfilter.h that says
> > >> explicitly the skb will be freed if NF_STOLEN is returned.
> > >>
> > >
> > > But its not true. Just read the code.
> > >
> > > If you are working on this stuff I recommend you take a look at
> > > commits :
> > >
> > > c6675233f9015d3c0460c8aab53ed9b99d915c64
> > > (netfilter: nf_queue: reject NF_STOLEN verdicts from userspace)
> > >
> > > fad54440438a7c231a6ae347738423cbabc936d9
> > > (netfilter: avoid double free in nf_reinject)
> > >
> > > 64507fdbc29c3a622180378210ecea8659b14e40
> > > (netfilter: nf_queue: fix NF_STOLEN skb leak)
> > >
> > > 3bc38712e3a6e0596ccb6f8299043a826f983701
> > > ([NETFILTER]: nf_queue: handle NF_STOP and unknown verdicts in
> > > nf_reinject)
> > >
> > >
> >
> > I see that fad54440438a7c231a6ae347738423cbabc936d9 (netfilter: avoid
> > double free in nf_reinject) returns the switch case for NF_STOLEN back
> > to the original state, but I just downloaded 3.0.4, and the skb is still
> > freed. So for some versions of the kernel, the situation exists.
> > Hopefully anyone who runs into it will find this thread.
> >
>
> Hopefully netfilter guys (CCed) will sort out the problem and ask stable
> submissions, if not already done. 3.0.4 is quite old :)
Not done yet, sorry. I'll do it asap.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-10-20 10:30 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-10-14 19:18 Problem with ixgbe and TX locked on one cpu Paweł Staszewski
2011-10-18 18:57 ` Jesse Brandeburg
2011-10-18 19:08 ` Comment on nf_queue NF_STOLEN patch Jim Sansing
2011-10-18 21:23 ` Eric Dumazet
2011-10-18 21:34 ` Jim Sansing
2011-10-19 4:10 ` Eric Dumazet
2011-10-20 10:30 ` Pablo Neira Ayuso
2011-10-19 9:21 ` Problem with ixgbe and TX locked on one cpu Paweł Staszewski
2011-10-19 9:21 ` Paweł Staszewski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).