Linux SMP on 2.4.18-3

Linux SMP on 2.4.18-3

[Cheng Jin]

Hi,

Please excuse me for asking questions on a rather old kernel.  We decided
to do kernel modificatios against 2.4.18-3 so we can't back it out now.

On the SMP test machine we have at the lab (Dual 2.4 Ghz Xeons with one
SysKonnect Gigabit Ethernet card, SuperMicro P4DP6 MB), I observed TCP
functions being called simultaneously by both processors.  What I did was
to simply increment a counter (init to zero) and check whether it is one
in the functions under suspicion.  Sure enough, I see a lot of messages
printed out saying it is two.  Admittedly, my counter var is not protected
either, but seeing it becoming 2 is proof enough that the functions are
entered simultaneously (yes I decrement the counter before functions
return).

I looked at the code fairly extensively, and I didn't see any lock for
these functions, tcp_send_skb, tcp_push_one, update_send_head, where
packets_out gets incremented.  The problem I was having was that
tp->packets_out got out of sync with the number of unacked packets on the
sk->write_queue.

I would like to confirm with people that are involved with kernel
developement that what I observed was indeed correct.

Thanks,

Cheng

Lab # 626 395 8820

Re: Linux SMP on 2.4.18-3

[jamal]

The IP network stack in linux is totaly reentrant. You could have a
packet on _each_ processor in SMP concurently executing the same code. If
you add anything, you need to take this into account.
In non-NAPI based NICs such as yours, you could have reordering within
the system (this is described in the NAPI paper). Either get it NAPIfied
or get yourself a NAPI capable NIC such as tg3 based, e1000, Dlink gige
etc.

cheers,
jamal

On Sun, 27 Oct 2002, Cheng Jin wrote:

> Hi,
>
> Please excuse me for asking questions on a rather old kernel.  We decided
> to do kernel modificatios against 2.4.18-3 so we can't back it out now.
>
> On the SMP test machine we have at the lab (Dual 2.4 Ghz Xeons with one
> SysKonnect Gigabit Ethernet card, SuperMicro P4DP6 MB), I observed TCP
> functions being called simultaneously by both processors.  What I did was
> to simply increment a counter (init to zero) and check whether it is one
> in the functions under suspicion.  Sure enough, I see a lot of messages
> printed out saying it is two.  Admittedly, my counter var is not protected
> either, but seeing it becoming 2 is proof enough that the functions are
> entered simultaneously (yes I decrement the counter before functions
> return).
>
> I looked at the code fairly extensively, and I didn't see any lock for
> these functions, tcp_send_skb, tcp_push_one, update_send_head, where
> packets_out gets incremented.  The problem I was having was that
> tp->packets_out got out of sync with the number of unacked packets on the
> sk->write_queue.
>
> I would like to confirm with people that are involved with kernel
> developement that what I observed was indeed correct.
>
> Thanks,
>
> Cheng
>
> Lab # 626 395 8820
>
>

Re: Linux SMP on 2.4.18-3

[Cheng Jin]

Jamal,

Thanks for writing me back.

> The IP network stack in linux is totaly reentrant. You could have a
> packet on _each_ processor in SMP concurently executing the same code. If
> you add anything, you need to take this into account.

We did add code to the TCP layer, but I don't exactly see anything in the
original code where locking is used.  I assume the locks are in the IP
layer and lower?  The weirdest thing is that we noticed in
update_send_head, tp->packets_out sometimes increases by more than one
even though the code ++ it only once.  I guess I am not sure how we could
have screwed it up if the modifications are on the TCP layer.

> In non-NAPI based NICs such as yours, you could have reordering within
> the system (this is described in the NAPI paper). Either get it NAPIfied
> or get yourself a NAPI capable NIC such as tg3 based, e1000, Dlink gige
> etc.

Out of curiosity, what is the maximum send rate (pps) under Linux
2.4.18-3?  I read in the paper that sending across two Intel Gbe with one
CPU gets up to 36o Kpps.


Thanks,

Cheng

Re: Linux SMP on 2.4.18-3

[Boris Protopopov]

Jamal, could you give some pointers as to how "NAPIfy" a 2.4.18-3/e1000-4.3.15 
setup ? Boris.

jamal wrote:
> 
> The IP network stack in linux is totaly reentrant. You could have a
> packet on _each_ processor in SMP concurently executing the same code. If
> you add anything, you need to take this into account.
> In non-NAPI based NICs such as yours, you could have reordering within
> the system (this is described in the NAPI paper). Either get it NAPIfied
> or get yourself a NAPI capable NIC such as tg3 based, e1000, Dlink gige
> etc.
> 
> cheers,
> jamal
> 
> On Sun, 27 Oct 2002, Cheng Jin wrote:
> 
> 
>>Hi,
>>
>>Please excuse me for asking questions on a rather old kernel.  We decided
>>to do kernel modificatios against 2.4.18-3 so we can't back it out now.
>>
>>On the SMP test machine we have at the lab (Dual 2.4 Ghz Xeons with one
>>SysKonnect Gigabit Ethernet card, SuperMicro P4DP6 MB), I observed TCP
>>functions being called simultaneously by both processors.  What I did was
>>to simply increment a counter (init to zero) and check whether it is one
>>in the functions under suspicion.  Sure enough, I see a lot of messages
>>printed out saying it is two.  Admittedly, my counter var is not protected
>>either, but seeing it becoming 2 is proof enough that the functions are
>>entered simultaneously (yes I decrement the counter before functions
>>return).
>>
>>I looked at the code fairly extensively, and I didn't see any lock for
>>these functions, tcp_send_skb, tcp_push_one, update_send_head, where
>>packets_out gets incremented.  The problem I was having was that
>>tp->packets_out got out of sync with the number of unacked packets on the
>>sk->write_queue.
>>
>>I would like to confirm with people that are involved with kernel
>>developement that what I observed was indeed correct.
>>
>>Thanks,
>>
>>Cheng
>>
>>Lab # 626 395 8820
>>
>>
> 
> 
> 
> 
> 

Re: Linux SMP on 2.4.18-3

[Jeff Garzik]

Boris Protopopov wrote:

> Jamal, could you give some pointers as to how "NAPIfy" a 
> 2.4.18-3/e1000-4.3.15 setup ? Boris.



In general, read Documentation/networking/NAPI_HOWTO.txt.

For e1000 specifically, copy drivers/net/e1000/*.[ch] to 2.4.x.

    Jeff

Re: Linux SMP on 2.4.18-3

[Robert Olsson]

Cheng Jin writes:

 > Out of curiosity, what is the maximum send rate (pps) under Linux
 > 2.4.18-3?  I read in the paper that sending across two Intel Gbe with one
 > CPU gets up to 36o Kpps.

 Yes forwarding performance... 

 A good chip/driver can itself do a lot more. I have seen very decent numbers
 with e1000, tg3 and dl2k. For FE tulip has been a top performer for long 
 time but  I also now see 144 kpps from my laptop w. 3c59x driver.

 Cheers.
						--ro

Network Device-Driver/Layer Implementation: Help required

[Harish Kulkarni]

Hello all,
I have a T1/E1 device driver: ( completed till PCI initialization and other
device-internals initialization,including interrupts handling procedures),
now i am not understanding how to provide the interface to upper layers?. I
have walked through the lapb-module.txt and some code of IPX, but still the
things are not clear. Can any one help/guide me?
-Thanks
Hari


> -----Original Message-----
> From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com]On
> Behalf Of Robert Olsson
> Sent: Thursday, October 31, 2002 12:33 AM
> To: Cheng Jin
> Cc: jamal; netdev@oss.sgi.com
> Subject: Re: Linux SMP on 2.4.18-3
>
>
>
> Cheng Jin writes:
>
>  > Out of curiosity, what is the maximum send rate (pps) under Linux
>  > 2.4.18-3?  I read in the paper that sending across two Intel
> Gbe with one
>  > CPU gets up to 36o Kpps.
>
>  Yes forwarding performance...
>
>  A good chip/driver can itself do a lot more. I have seen very
> decent numbers
>  with e1000, tg3 and dl2k. For FE tulip has been a top performer for long
>  time but  I also now see 144 kpps from my laptop w. 3c59x driver.
>
>  Cheers.
> 						--ro
>
>
>

Re: Network Device-Driver/Layer Implementation: Help required

[James R. Leu]

What L2 protocols will it support?  This will determin how you present it to
the network stack.  At a minimum it will have to create a net_device which
has all sorts of function pointers in it that you will have to implement.

On Thu, Oct 31, 2002 at 01:25:18PM +0530, Harish Kulkarni wrote:
> Hello all,
> I have a T1/E1 device driver: ( completed till PCI initialization and other
> device-internals initialization,including interrupts handling procedures),
> now i am not understanding how to provide the interface to upper layers?. I
> have walked through the lapb-module.txt and some code of IPX, but still the
> things are not clear. Can any one help/guide me?
> -Thanks
> Hari
> 
> 
> > -----Original Message-----
> > From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com]On
> > Behalf Of Robert Olsson
> > Sent: Thursday, October 31, 2002 12:33 AM
> > To: Cheng Jin
> > Cc: jamal; netdev@oss.sgi.com
> > Subject: Re: Linux SMP on 2.4.18-3
> >
> >
> >
> > Cheng Jin writes:
> >
> >  > Out of curiosity, what is the maximum send rate (pps) under Linux
> >  > 2.4.18-3?  I read in the paper that sending across two Intel
> > Gbe with one
> >  > CPU gets up to 36o Kpps.
> >
> >  Yes forwarding performance...
> >
> >  A good chip/driver can itself do a lot more. I have seen very
> > decent numbers
> >  with e1000, tg3 and dl2k. For FE tulip has been a top performer for long
> >  time but  I also now see 144 kpps from my laptop w. 3c59x driver.
> >
> >  Cheers.
> > 						--ro
> >
> >
> >
> 

-- 
James R. Leu

Re: Linux SMP on 2.4.18-3

[jamal]

On Mon, 28 Oct 2002, Cheng Jin wrote:

> > The IP network stack in linux is totaly reentrant. You could have a
> > packet on _each_ processor in SMP concurently executing the same code. If
> > you add anything, you need to take this into account.
>
> We did add code to the TCP layer, but I don't exactly see anything in the
> original code where locking is used.  I assume the locks are in the IP
> layer and lower?

There are a few, but they are irrelevant in your case since you dont
touch that code (eg the dst cache lock).
As far as TCP is concerned, serialization between SMP processors happens
with the socket lock. This lock is also used to sequence packets
[Remember TCP has to ensure that packets are sequenced for processing
by user process. Also remember that TCP runs in both user and process
context].

> The weirdest thing is that we noticed in
> update_send_head, tp->packets_out sometimes increases by more than one
> even though the code ++ it only once.  I guess I am not sure how we could
> have screwed it up if the modifications are on the TCP layer.
>

Anything that you want to serialize access to you should put in the sock
struct; then use the sock lock to serialize.

cheers,
jamal