Re: packet re-ordering on SMP machines.

Re: packet re-ordering on SMP machines.

[David S. Miller]

Use IRQ affinity settings if you want strict packet receive ordering.

packet re-ordering on SMP machines.

[Ben Greear]

It appears from my initial tests on 2.4.20-pre4, that packets sent on
one port, and received on another port on the same machine (via a cross-over
cable), can be re-ordered.  I see about 2000 reordered packets per 5,000,000 packets
sent (sending about 70,000 packets-per-second on a dual-port e1000 NIC.)

By re-ordered, I mean that a method called from process_backlog in dev.c
is being handed packets in a different order than they are being poked into
the driver with hard_start_xmit on the other interface.  If each CPU can be running the
process_backlog, then I can see how this could be happening.


1)  Is this expected behaviour?

2)  Is there any standard (ie configurable) way to enforce strict ordering on an
     SMP system?

3)  If answer to 2 is no, would you all be interested in a patch that
     did allow strict ordering (if indeed I can figure out how to write one)?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: packet re-ordering on SMP machines.

[Manfred Spraul]

 > 2)  Is there any standard (ie configurable) way to enforce strict
 > ordering on an SMP system?

2 cpus, 2 network cards? What happens if you bind the interrupts to cpus?

echo 1 > /proc/irq/<irq_of_card_1>/smp_affinity
echo 2 > /proc/irq/<irq_of_card_2>/smp_affinity

--
	Manfred

Re: packet re-ordering on SMP machines.

[Alan Cox]

On Sun, 2002-08-25 at 08:18, Ben Greear wrote:
> By re-ordered, I mean that a method called from process_backlog in dev.c
> is being handed packets in a different order than they are being poked into
> the driver with hard_start_xmit on the other interface.  If each CPU can be running the
> process_backlog, then I can see how this could be happening.
> 
> 
> 1)  Is this expected behaviour?
Yes

> 2)  Is there any standard (ie configurable) way to enforce strict ordering on an
>      SMP system?
No

> 3)  If answer to 2 is no, would you all be interested in a patch that
>      did allow strict ordering (if indeed I can figure out how to write one)?

You should never need it. Ethernet, hubs, switches, routers, internet
backbones etc will all cause packet re-ordering. You should also expect
the percentage of re-ordered frames on the net to rise and rise. 

Re: packet re-ordering on SMP machines.

[jamal]

NAPI fixes packet reordering problems.
Could people please post network related questions to netdev please?
I think it even says so in the FAQ Richard Gooch maintains.
Believe it or not, quiet a few people are not subscribed to lk

cheers,
jamal

Re: packet re-ordering on SMP machines.

[Ben Greear]

Alan Cox wrote:
> On Sun, 2002-08-25 at 08:18, Ben Greear wrote:
> 
>>By re-ordered, I mean that a method called from process_backlog in dev.c
>>is being handed packets in a different order than they are being poked into
>>the driver with hard_start_xmit on the other interface.  If each CPU can be running the
>>process_backlog, then I can see how this could be happening.
>>
>>
>>1)  Is this expected behaviour?
> 
> Yes
> 
> 
>>2)  Is there any standard (ie configurable) way to enforce strict ordering on an
>>     SMP system?
> 
> No
> 
> 
>>3)  If answer to 2 is no, would you all be interested in a patch that
>>     did allow strict ordering (if indeed I can figure out how to write one)?
> 
> 
> You should never need it. Ethernet, hubs, switches, routers, internet
> backbones etc will all cause packet re-ordering. You should also expect
> the percentage of re-ordered frames on the net to rise and rise. 

I would like to detect the number of pkts that such backbone hardware does
re-order, so if my end machine is also re-ordering, I cannot get valid
numbers.

Thanks,
Ben

> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: packet re-ordering on SMP machines.

[Alan Cox]

On Sun, 2002-08-25 at 18:00, Ben Greear wrote:
> I would like to detect the number of pkts that such backbone hardware does
> re-order, so if my end machine is also re-ordering, I cannot get valid
> numbers.

Good point 

Re: packet re-ordering on SMP machines.

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: 25 Aug 2002 15:41:13 +0100

   > 3)  If answer to 2 is no, would you all be interested in a patch that
   >      did allow strict ordering (if indeed I can figure out how to write one)?
   
   You should never need it. Ethernet, hubs, switches, routers, internet
   backbones etc will all cause packet re-ordering. You should also expect
   the percentage of re-ordered frames on the net to rise and rise. 
   
But to be honest, NAPI was meant to dramatically reduce the "SMP
specific" causes of packet reordering.  When a driver does use NAPI,
the reordering due to SMP goes way down.