* bnx2/5709: Strange interrupts spread
@ 2010-07-02 20:33 Christophe Ngo Van Duc
2010-07-02 20:58 ` Michael Chan
2010-07-19 15:55 ` Christophe Ngo Van Duc
0 siblings, 2 replies; 10+ messages in thread
From: Christophe Ngo Van Duc @ 2010-07-02 20:33 UTC (permalink / raw)
To: netdev
Dear list,
I hope I am posting to the correct place...
I am facing a strange issue on a HP DL 360.
I have 2 internal ethernet cards (the one that came by default with
the server) and 2 additional ethernet cards for a total for 4 ethernet
cards.
The 2 internal cards are running fine as of interrupts (for example eth1):
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
71: 604 11933 40 1537 0
0 0 6043 PCI-MSI-edge eth1-0
72: 24805 9795 3606 0 128
0 3365 0 PCI-MSI-edge eth1-1
73: 0 279 0 429 38
16540 0 30843 PCI-MSI-edge eth1-2
74: 0 0 25365 267 0
0 89 15541 PCI-MSI-edge eth1-3
75: 7244 24108 0 0 16488
0 240 0 PCI-MSI-edge eth1-4
76: 21378 3628 7726 0 49
247 2871 0 PCI-MSI-edge eth1-5
77: 0 0 47199 459 13
46 63064 18 PCI-MSI-edge eth1-6
78: 0 6230 67 283 259
82 7846 27130 PCI-MSI-edge eth1-7
On eth2 (external card) all interrupts goes to CPU0
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
80: 46973077 0 0 0 0
0 0 PCI-MSI-edge eth2-0
81: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-1
82: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-2
83: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-3
84: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-4
85: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-5
86: 0 0 2445 0 37
0 8463 13 PCI-MSI-edge eth2-6
87: 0 0 0 0 0
0 0 0 PCI-MSI-edge eth2-7
If I understand correctly the RSS hash is used to dispatch the packets
into the different queues running on the different CPU.
Why then my internal cards are running fine but the additional cards
(eth2 and eth3) are presenting this behavior where all interrupts goes
to one CPU?
Thanks for your help in understanding this. (see below for config details)
Christophe.
All are detected correctly at boot:
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.8e (April 13, 2010)
bnx2 0000:02:00.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
bnx2 0000:02:00.0: setting latency timer to 64
eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
at mem f4000000, IRQ 31, node addr f4:ce:46:86:a1:00
bnx2 0000:02:00.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
bnx2 0000:02:00.1: setting latency timer to 64
eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
at mem f2000000, IRQ 39, node addr f4:ce:46:86:a1:02
bnx2 0000:07:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
bnx2 0000:07:00.0: setting latency timer to 64
eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
at mem fa000000, IRQ 24, node addr 00:26:55:87:17:98
bnx2 0000:07:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34
bnx2 0000:07:00.1: setting latency timer to 64
eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
at mem f8000000, IRQ 34, node addr 00:26:55:87:17:9a
Kernel is 2.6.31-13
Broadcom driver bnx2 v2.0.8e
eth0 is a normal interface with an Ip address
eth1 is a normal interface with an Ip address
eth2 belongs to a bridge interface without an ip address, running tc (htb)
eth3 belongs to the same bridge interface without an ip address
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-02 20:33 bnx2/5709: Strange interrupts spread Christophe Ngo Van Duc
@ 2010-07-02 20:58 ` Michael Chan
2010-07-02 22:12 ` Christophe Ngo Van Duc
2010-07-19 15:55 ` Christophe Ngo Van Duc
1 sibling, 1 reply; 10+ messages in thread
From: Michael Chan @ 2010-07-02 20:58 UTC (permalink / raw)
To: Christophe Ngo Van Duc; +Cc: netdev@vger.kernel.org
On Fri, 2010-07-02 at 13:33 -0700, Christophe Ngo Van Duc wrote:
> On eth2 (external card) all interrupts goes to CPU0
>
>
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
> 80: 46973077 0 0 0 0 0 0 0 PCI-MSI-edge eth2-0
> 81: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-1
> 82: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-2
> 83: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-3
> 84: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-4
> 85: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-5
> 86: 0 0 2445 0 37 0 8463 13 PCI-MSI-edge eth2-6
> 87: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth2-7
Reformatted your output
> If I understand correctly the RSS hash is used to dispatch the packets
> into the different queues running on the different CPU.
It looks like most interrupts go to eth2-0, a few go to eth2-6. The rx
ring for eth2-0 is for non-IP packets. The RSS hash will hash IP
packets and place them on eth2-1 to eth2-7. eth2-0 also handles tx
interrupts for TX ring 0. TX traffic is hashed by the stack.
What kind of traffic is passing through eth2?
Thanks.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-02 20:58 ` Michael Chan
@ 2010-07-02 22:12 ` Christophe Ngo Van Duc
2010-07-04 20:36 ` Christophe Ngo Van Duc
0 siblings, 1 reply; 10+ messages in thread
From: Christophe Ngo Van Duc @ 2010-07-02 22:12 UTC (permalink / raw)
To: Michael Chan, netdev@vger.kernel.org
Hi
Well that's the strange thing: it is IP traffic. The only difference
with eth0 and eth1 is that eth2 and eth3 belongs to a bridge (br0).
Best Regards,
Christophe.
On 7/2/10, Michael Chan <mchan@broadcom.com> wrote:
>
> On Fri, 2010-07-02 at 13:33 -0700, Christophe Ngo Van Duc wrote:
>> On eth2 (external card) all interrupts goes to CPU0
>>
>>
>> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
>> CPU6 CPU7
>> 80: 46973077 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-0
>> 81: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-1
>> 82: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-2
>> 83: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-3
>> 84: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-4
>> 85: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-5
>> 86: 0 0 2445 0 37 0
>> 8463 13 PCI-MSI-edge eth2-6
>> 87: 0 0 0 0 0 0
>> 0 0 PCI-MSI-edge eth2-7
>
> Reformatted your output
>
>> If I understand correctly the RSS hash is used to dispatch the packets
>> into the different queues running on the different CPU.
>
> It looks like most interrupts go to eth2-0, a few go to eth2-6. The rx
> ring for eth2-0 is for non-IP packets. The RSS hash will hash IP
> packets and place them on eth2-1 to eth2-7. eth2-0 also handles tx
> interrupts for TX ring 0. TX traffic is hashed by the stack.
>
> What kind of traffic is passing through eth2?
>
> Thanks.
>
>
>
--
Sent from my mobile device
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-02 22:12 ` Christophe Ngo Van Duc
@ 2010-07-04 20:36 ` Christophe Ngo Van Duc
2010-07-05 22:56 ` Rick Jones
0 siblings, 1 reply; 10+ messages in thread
From: Christophe Ngo Van Duc @ 2010-07-04 20:36 UTC (permalink / raw)
To: netdev@vger.kernel.org
Is it a requirement that the interface has an IP address for the TX
and RX hash to work?
Christophe.
On Fri, Jul 2, 2010 at 7:12 PM, Christophe Ngo Van Duc
<cngovanduc@gmail.com> wrote:
> Hi
>
> Well that's the strange thing: it is IP traffic. The only difference
> with eth0 and eth1 is that eth2 and eth3 belongs to a bridge (br0).
>
> Best Regards,
> Christophe.
>
> On 7/2/10, Michael Chan <mchan@broadcom.com> wrote:
>>
>> On Fri, 2010-07-02 at 13:33 -0700, Christophe Ngo Van Duc wrote:
>>> On eth2 (external card) all interrupts goes to CPU0
>>>
>>>
>>> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
>>> CPU6 CPU7
>>> 80: 46973077 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-0
>>> 81: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-1
>>> 82: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-2
>>> 83: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-3
>>> 84: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-4
>>> 85: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-5
>>> 86: 0 0 2445 0 37 0
>>> 8463 13 PCI-MSI-edge eth2-6
>>> 87: 0 0 0 0 0 0
>>> 0 0 PCI-MSI-edge eth2-7
>>
>> Reformatted your output
>>
>>> If I understand correctly the RSS hash is used to dispatch the packets
>>> into the different queues running on the different CPU.
>>
>> It looks like most interrupts go to eth2-0, a few go to eth2-6. The rx
>> ring for eth2-0 is for non-IP packets. The RSS hash will hash IP
>> packets and place them on eth2-1 to eth2-7. eth2-0 also handles tx
>> interrupts for TX ring 0. TX traffic is hashed by the stack.
>>
>> What kind of traffic is passing through eth2?
>>
>> Thanks.
>>
>>
>>
>
> --
> Sent from my mobile device
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-04 20:36 ` Christophe Ngo Van Duc
@ 2010-07-05 22:56 ` Rick Jones
2010-07-08 20:06 ` Christophe Ngo Van Duc
0 siblings, 1 reply; 10+ messages in thread
From: Rick Jones @ 2010-07-05 22:56 UTC (permalink / raw)
To: Christophe Ngo Van Duc; +Cc: netdev@vger.kernel.org
Christophe Ngo Van Duc wrote:
> Is it a requirement that the interface has an IP address for the TX
> and RX hash to work?
At the risk of typing into Michael's keyboard, chances are, what the NIC does is
follow Microsoft's specs for RSS, which as far as I can tell, discusses hashing
either just on the IP addresses (v4 or v6) or the IP addresses and TCP port
numbers, so unless Broadcom has an enhancement/extension, if the traffic
arriving is not TCP/IP and not multiple connections, it probably does not get
spread-out.
rick jones
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-05 22:56 ` Rick Jones
@ 2010-07-08 20:06 ` Christophe Ngo Van Duc
0 siblings, 0 replies; 10+ messages in thread
From: Christophe Ngo Van Duc @ 2010-07-08 20:06 UTC (permalink / raw)
To: Rick Jones; +Cc: netdev@vger.kernel.org
Traffic is IP no doubt about that. Lots of TCP and UDP traffic. I can
provide a very short pcap if you want (lots of traffic so pcap can be
big really quickly).
- The 2 interfaces that have this problems are running in promiscuous
mode (without having an assiged IP address to the interface) inside a
linux bridge interface (br0), but the traffic that is bridged is IP.
- The other 2 that works correctly: are not in promiscuous and have an
IP assigned to the network card.
So I am thinking that for some reason the RSS is activating only when
a specific configuration is set on the interface but not just on if it
is IP traffic or not.
Best Regards,
Christophe.
On Mon, Jul 5, 2010 at 7:56 PM, Rick Jones <rick.jones2@hp.com> wrote:
> Christophe Ngo Van Duc wrote:
>>
>> Is it a requirement that the interface has an IP address for the TX
>> and RX hash to work?
>
> At the risk of typing into Michael's keyboard, chances are, what the NIC
> does is follow Microsoft's specs for RSS, which as far as I can tell,
> discusses hashing either just on the IP addresses (v4 or v6) or the IP
> addresses and TCP port numbers, so unless Broadcom has an
> enhancement/extension, if the traffic arriving is not TCP/IP and not
> multiple connections, it probably does not get spread-out.
>
> rick jones
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-02 20:33 bnx2/5709: Strange interrupts spread Christophe Ngo Van Duc
2010-07-02 20:58 ` Michael Chan
@ 2010-07-19 15:55 ` Christophe Ngo Van Duc
2010-07-19 18:47 ` Michael Chan
1 sibling, 1 reply; 10+ messages in thread
From: Christophe Ngo Van Duc @ 2010-07-19 15:55 UTC (permalink / raw)
To: netdev
Dear list,
So i've been able to do some test today:
If I put the 2 interface in a bridge with no IP adress, the interrupts
are on 1 CPU
If I put the 2 interface in a bridge with IP adress, the interrupts
are still on 1 CPU
If I put the 2 interface outside the bridge with IP address,
everything works fine the interrupts get spread on the CPU
So the conclusion seems to be that when the bnx2 is put into
promiscuous mode by the bridge, the RSS hash stop to work even if
traffic is IP in nature.
Best regards,
Christophe.
On Fri, Jul 2, 2010 at 5:33 PM, Christophe Ngo Van Duc
<cngovanduc@gmail.com> wrote:
> Dear list,
>
> I hope I am posting to the correct place...
>
> I am facing a strange issue on a HP DL 360.
>
> I have 2 internal ethernet cards (the one that came by default with
> the server) and 2 additional ethernet cards for a total for 4 ethernet
> cards.
>
> The 2 internal cards are running fine as of interrupts (for example eth1):
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
> CPU6 CPU7
>
> 71: 604 11933 40 1537 0
> 0 0 6043 PCI-MSI-edge eth1-0
> 72: 24805 9795 3606 0 128
> 0 3365 0 PCI-MSI-edge eth1-1
> 73: 0 279 0 429 38
> 16540 0 30843 PCI-MSI-edge eth1-2
> 74: 0 0 25365 267 0
> 0 89 15541 PCI-MSI-edge eth1-3
> 75: 7244 24108 0 0 16488
> 0 240 0 PCI-MSI-edge eth1-4
> 76: 21378 3628 7726 0 49
> 247 2871 0 PCI-MSI-edge eth1-5
> 77: 0 0 47199 459 13
> 46 63064 18 PCI-MSI-edge eth1-6
> 78: 0 6230 67 283 259
> 82 7846 27130 PCI-MSI-edge eth1-7
>
> On eth2 (external card) all interrupts goes to CPU0
> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
> CPU6 CPU7
> 80: 46973077 0 0 0 0
> 0 0 PCI-MSI-edge eth2-0
> 81: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-1
> 82: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-2
> 83: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-3
> 84: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-4
> 85: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-5
> 86: 0 0 2445 0 37
> 0 8463 13 PCI-MSI-edge eth2-6
> 87: 0 0 0 0 0
> 0 0 0 PCI-MSI-edge eth2-7
>
> If I understand correctly the RSS hash is used to dispatch the packets
> into the different queues running on the different CPU.
>
> Why then my internal cards are running fine but the additional cards
> (eth2 and eth3) are presenting this behavior where all interrupts goes
> to one CPU?
>
> Thanks for your help in understanding this. (see below for config details)
>
> Christophe.
>
> All are detected correctly at boot:
> Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.8e (April 13, 2010)
> bnx2 0000:02:00.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
> bnx2 0000:02:00.0: setting latency timer to 64
> eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
> at mem f4000000, IRQ 31, node addr f4:ce:46:86:a1:00
> bnx2 0000:02:00.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
> bnx2 0000:02:00.1: setting latency timer to 64
> eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
> at mem f2000000, IRQ 39, node addr f4:ce:46:86:a1:02
> bnx2 0000:07:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
> bnx2 0000:07:00.0: setting latency timer to 64
> eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
> at mem fa000000, IRQ 24, node addr 00:26:55:87:17:98
> bnx2 0000:07:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34
> bnx2 0000:07:00.1: setting latency timer to 64
> eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found
> at mem f8000000, IRQ 34, node addr 00:26:55:87:17:9a
>
> Kernel is 2.6.31-13
> Broadcom driver bnx2 v2.0.8e
>
> eth0 is a normal interface with an Ip address
> eth1 is a normal interface with an Ip address
> eth2 belongs to a bridge interface without an ip address, running tc (htb)
> eth3 belongs to the same bridge interface without an ip address
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-19 15:55 ` Christophe Ngo Van Duc
@ 2010-07-19 18:47 ` Michael Chan
2010-07-19 19:47 ` Eric Dumazet
0 siblings, 1 reply; 10+ messages in thread
From: Michael Chan @ 2010-07-19 18:47 UTC (permalink / raw)
To: Christophe Ngo Van Duc; +Cc: netdev@vger.kernel.org
On Mon, 2010-07-19 at 08:55 -0700, Christophe Ngo Van Duc wrote:
> So i've been able to do some test today:
> If I put the 2 interface in a bridge with no IP adress, the interrupts
> are on 1 CPU
> If I put the 2 interface in a bridge with IP adress, the interrupts
> are still on 1 CPU
> If I put the 2 interface outside the bridge with IP address,
> everything works fine the interrupts get spread on the CPU
>
> So the conclusion seems to be that when the bnx2 is put into
> promiscuous mode by the bridge, the RSS hash stop to work even if
> traffic is IP in nature.
I did a quick test with bridging and saw no problem with RSS. I did see
this though:
br0 received packet on queue 4, but number of RX queues is 1
Looks like it is a warning message from RPS.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-19 18:47 ` Michael Chan
@ 2010-07-19 19:47 ` Eric Dumazet
2010-07-19 20:29 ` Michael Chan
0 siblings, 1 reply; 10+ messages in thread
From: Eric Dumazet @ 2010-07-19 19:47 UTC (permalink / raw)
To: Michael Chan; +Cc: Christophe Ngo Van Duc, netdev@vger.kernel.org
Le lundi 19 juillet 2010 à 11:47 -0700, Michael Chan a écrit :
> On Mon, 2010-07-19 at 08:55 -0700, Christophe Ngo Van Duc wrote:
> > So i've been able to do some test today:
> > If I put the 2 interface in a bridge with no IP adress, the interrupts
> > are on 1 CPU
> > If I put the 2 interface in a bridge with IP adress, the interrupts
> > are still on 1 CPU
> > If I put the 2 interface outside the bridge with IP address,
> > everything works fine the interrupts get spread on the CPU
> >
> > So the conclusion seems to be that when the bnx2 is put into
> > promiscuous mode by the bridge, the RSS hash stop to work even if
> > traffic is IP in nature.
>
> I did a quick test with bridging and saw no problem with RSS. I did see
> this though:
>
> br0 received packet on queue 4, but number of RX queues is 1
>
> Looks like it is a warning message from RPS.
>
Christophe uses an old kernel, not RPS enabled ;)
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bnx2/5709: Strange interrupts spread
2010-07-19 19:47 ` Eric Dumazet
@ 2010-07-19 20:29 ` Michael Chan
0 siblings, 0 replies; 10+ messages in thread
From: Michael Chan @ 2010-07-19 20:29 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Christophe Ngo Van Duc, netdev@vger.kernel.org
On Mon, 2010-07-19 at 12:47 -0700, Eric Dumazet wrote:
> Le lundi 19 juillet 2010 à 11:47 -0700, Michael Chan a écrit :
> > On Mon, 2010-07-19 at 08:55 -0700, Christophe Ngo Van Duc wrote:
> > > So i've been able to do some test today:
> > > If I put the 2 interface in a bridge with no IP adress, the interrupts
> > > are on 1 CPU
> > > If I put the 2 interface in a bridge with IP adress, the interrupts
> > > are still on 1 CPU
> > > If I put the 2 interface outside the bridge with IP address,
> > > everything works fine the interrupts get spread on the CPU
> > >
> > > So the conclusion seems to be that when the bnx2 is put into
> > > promiscuous mode by the bridge, the RSS hash stop to work even if
> > > traffic is IP in nature.
> >
> > I did a quick test with bridging and saw no problem with RSS. I did see
> > this though:
> >
> > br0 received packet on queue 4, but number of RX queues is 1
> >
> > Looks like it is a warning message from RPS.
> >
>
> Christophe uses an old kernel, not RPS enabled ;)
>
>
Right, I'm reporting a related problem on a newer kernel with RPS
enabled. And the fact that it is receiving packets on queue 4 shows
that RSS is working together with bridging.
I'll try to use an older kernel to see what happens later today.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-07-19 20:30 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-07-02 20:33 bnx2/5709: Strange interrupts spread Christophe Ngo Van Duc
2010-07-02 20:58 ` Michael Chan
2010-07-02 22:12 ` Christophe Ngo Van Duc
2010-07-04 20:36 ` Christophe Ngo Van Duc
2010-07-05 22:56 ` Rick Jones
2010-07-08 20:06 ` Christophe Ngo Van Duc
2010-07-19 15:55 ` Christophe Ngo Van Duc
2010-07-19 18:47 ` Michael Chan
2010-07-19 19:47 ` Eric Dumazet
2010-07-19 20:29 ` Michael Chan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).