Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq network code on SMP

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq network code on SMP

[David Miller]

The whole reason the queues are per-cpu is so that we do not
have to touch remote processor state nor use locks of any
kind whatsoever.

With multi-queue networking cards becoming more and more
available, which will split up the packet workload in
hardware across all available cpus, there is less and less
reason to make a patch like this one.

We've known about this issue for ages, and if we felt it
was appropriate to make this change, we would have done
so years ago.

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq network code on SMP

[John Ye]

David,

Thanks for your reply. I understand it's not worth to do.

I have made it a loadable module to fulfill the function. it mainly for busy
NAT gateway server with SMP to speed up.

John Ye



----- Original Message -----
From: "David Miller" <davem@davemloft.net>
To: <johny@asimco.com.cn>
Cc: <netdev@vger.kernel.org>; <kuznet@ms2.inr.ac.ru>; <pekkas@netcore.fi>;
<jmorris@namei.org>; <kaber@coreworks.de>
Sent: Friday, September 21, 2007 1:46 AM
Subject: Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq
network code on SMP


>
> The whole reason the queues are per-cpu is so that we do not
> have to touch remote processor state nor use locks of any
> kind whatsoever.
>
> With multi-queue networking cards becoming more and more
> available, which will split up the packet workload in
> hardware across all available cpus, there is less and less
> reason to make a patch like this one.
>
> We've known about this issue for ages, and if we felt it
> was appropriate to make this change, we would have done
> so years ago.
>

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq network code on SMP

[jamal]

On Fri, 2007-21-09 at 17:25 +0800, John Ye wrote:
> David,
> 
> Thanks for your reply. I understand it's not worth to do.
> 
> I have made it a loadable module to fulfill the function. it mainly for busy
> NAT gateway server with SMP to speed up.
> 

John,

It was a little hard to read your code; however, it does seems to me
like will cause a massive amount of packet reordering to the end hosts
using you as the gateway especially when it is receiving a lot of
packets/second. 
You have a queue per CPU that connects your bottom and top half and
several CPUs that may service a single NIC in your bottom half.
one cpu in either bottom/top half has to be slightly loaded and you
loose the ordering where incoming doesnt match outgoing packet order. 

cheers,
jamal

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirq network code on SMP

[jamal]

On Fri, 2007-21-09 at 07:43 -0400, jamal wrote:

> one cpu in either bottom/top half has to be slightly loaded and you
> loose the ordering where incoming doesnt match outgoing packet order. 

Actually in your case it gets worse because (if i read that code
correctly) you randomly select the CPUs.

cheers,
jamal

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirqnetwork code on SMP

[john ye]

Dear Jamal,

Sorry, I sent to you all a not-good-formatted mail.
Thanks for instructions and corrections from you all.

I have thought that packet re-ordering for upper TCP protocol will become 
more intensive and this will make the network even slower.

I do randomly select a CPU to dispatch the skb to. Previously, I dispatch 
skb evenly to all CPUs( round robin, one by one). but I didn't find a quick 
coding. for_each_online_cpu is not quick enough.

According to my test result, it did make packet INPUT speed doubled because 
another CPU is used concurrently.
It seems the packets still keep "roughly ordering" after turning on BS 
patch.

The test is simple: use an 2400 lines of iptables -t filter -A INPUT -p 
tcp -s x.x.x.x --dport yy -j XXXX.
these rules make the current softirq be very busy on one CPU and make the 
incoming net very slow. after turning on BS, the speed doubled.

For NAT test, I didn't get a good result like INPUT because real environment 
limitation.
The test is very basic and is far from "full".

It seems to me that the cross-cpu spinlock_xxxx for the queue doesn't have 
big cost and is allowable in terms of CPU time consumption, compared with 
the gains by making other CPUs joint in the work.

I have made BS patch into a loadable module. 
http://linux.chinaunix.net/bbs/thread-909725-2-1.html and let others help 
with testing.

John Ye


----- Original Message ----- 
From: "jamal" <hadi@cyberus.ca>
To: "John Ye" <johny@asimco.com.cn>
Cc: "David Miller" <davem@davemloft.net>; <netdev@vger.kernel.org>; 
<kuznet@ms2.inr.ac.ru>; <pekkas@netcore.fi>; <jmorris@namei.org>; 
<kaber@coreworks.de>
Sent: Friday, September 21, 2007 7:43 PM
Subject: Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run 
softirqnetwork code on SMP


> On Fri, 2007-21-09 at 17:25 +0800, John Ye wrote:
>> David,
>>
>> Thanks for your reply. I understand it's not worth to do.
>>
>> I have made it a loadable module to fulfill the function. it mainly for 
>> busy
>> NAT gateway server with SMP to speed up.
>>
>
> John,
>
> It was a little hard to read your code; however, it does seems to me
> like will cause a massive amount of packet reordering to the end hosts
> using you as the gateway especially when it is receiving a lot of
> packets/second.
> You have a queue per CPU that connects your bottom and top half and
> several CPUs that may service a single NIC in your bottom half.
> one cpu in either bottom/top half has to be slightly loaded and you
> loose the ordering where incoming doesnt match outgoing packet order.
>
> cheers,
> jamal
>
> 

Re: [PATCH: 2.6.13-15-SMP 3/3] network: concurrently run softirqnetwork code on SMP

[jamal]

On Sun, 2007-23-09 at 12:45 +0800, john ye wrote:

>  I do randomly select a CPU to dispatch the skb to. Previously, I
> dispatch
>  skb evenly to all CPUs( round robin, one by one). but I didn't find a
> quick
>  coding. for_each_online_cpu is not quick enough.

for_each_online_cpu doenst look that expensive - but even round robin
wont fix the reordering problem. What you need to do is make sure that a
flow always goes to the same cpu over some period of time.

>  According to my test result, it did make packet INPUT speed doubled
> because
>  another CPU is used concurrently.

How did you measure "speed" - was it throughput? Did you measure how
much cpu was being utilized?

>  It seems the packets still keep "roughly ordering" after turning on
> BS patch.

Linux TCP is very resilient to reordering compared to other OSes, but
even then if you hit it with enough packets it is going to start
sweating it.

>  The test is simple: use an 2400 lines of iptables -t filter -A INPUT
> -p
>  tcp -s x.x.x.x --dport yy -j XXXX.
>  these rules make the current softirq be very busy on one CPU and make
> the
>  incoming net very slow. after turning on BS, the speed doubled.
> 
Ok, but how do you observe "doubled"?
Do you have conntrack on? It maybe that what you have just found is
netfilter needs to have its work defered from packet rcv.
You need some real fast traffic generator; possibly one that can do
thousands of tcp sessions.

>  For NAT test, I didn't get a good result like INPUT because real 
> environment limitation.
>  The test is very basic and is far from "full".

What happens when you totally compile out netfilter and you just use
this machine as a server?

>  It seems to me that the cross-cpu spinlock_xxxx for the queue doesn't
> have
>  big cost and is allowable in terms of CPU time consumption, compared
> with
>  the gains by making other CPUs joint in the work.
> 
>  I have made BS patch into a loadable module.
>  http://linux.chinaunix.net/bbs/thread-909725-2-1.html and let others
> help with testing.

It is still very hard to read; and i am not sure how you are going to
get the performance you claim eventually - you are registering as a tap
for ip packets, which means you will process two of each incoming
packets.

cheers,
jamal