Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote: 
> The regression is:
> 1)stoakley with 2 qual-core processors: 11%;
> 2)Tulsa with 4 dual-core(+hyperThread) processors:13%;
I have new update on this issue and also cc to netdev maillist.
Thank David Miller for pointing me the netdev maillist.

> 
> The test command is:
> #sudo taskset -c 7 ./netserver
> #sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
> 
> As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> regression is between 16%~11%.
> 
> I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> but the bisected kernel wasn't stable and went crazy.
> 
> I tried both CONFIG_SLUB=y and CONFIG_SLAB=y to make sure SLUB isn't the
> culprit.
> 
> The oprofile data of CONFIG_SLAB=y. Top cpu utilizations are:
> 1) 2.6.22 
> 2067379   9.4888  vmlinux                  schedule
> 1873604   8.5994  vmlinux                  mwait_idle
> 1568131   7.1974  vmlinux                  resched_task
> 1066976   4.8972  vmlinux                  tcp_v4_rcv
> 986641    4.5285  vmlinux                  tcp_rcv_established
> 979518    4.4958  vmlinux                  find_busiest_group
> 767069    3.5207  vmlinux                  sock_def_readable
> 736808    3.3818  vmlinux                  tcp_sendmsg
> 595889    2.7350  vmlinux                  task_rq_lock
> 557193    2.5574  vmlinux                  tcp_ack
> 470570    2.1598  vmlinux                  __mod_timer
> 392220    1.8002  vmlinux                  __alloc_skb
> 358106    1.6436  vmlinux                  skb_release_data
> 313372    1.4383  vmlinux                  skb_clone
> 
> 2) 2.6.24-rc7
> 2668426  12.4497  vmlinux                  vmlinux                  schedule
> 955698    4.4589  vmlinux                  vmlinux                  skb_release_data
> 836311    3.9018  vmlinux                  vmlinux                  tcp_v4_rcv
> 762398    3.5570  vmlinux                  vmlinux                  skb_release_all
> 728907    3.4007  vmlinux                  vmlinux                  task_rq_lock
> 705037    3.2894  vmlinux                  vmlinux                  __wake_up
> 694206    3.2388  vmlinux                  vmlinux                  __mod_timer
> 617616    2.8815  vmlinux                  vmlinux                  mwait_idle
> 
> It looks like tcp in 2.6.22 sends more packets, but frees far less skb than 2.6.24-rc6.
> tcp_rcv_established in 2.6.22 is highlighted on cpu utilization.
I instrumented kernel to capure the function call numbers.
1) 2.6.22
skb_release_data:50148649
tcp_ack:	 25062858	
tcp_transmit_skb:25063150	
tcp_v4_rcv:	 25063279	

2) 2.6.24-rc6
skb_release_data:21429692	
tcp_ack:	 10707710	
tcp_transmit_skb:10707866
tcp_v4_rcv:	 10707959		

The data doesn't show that 2.6.22 sends more packets while freeing far less skb than
2.6.24-rc6.

The data showed skb_release_data of kernel 2.6.22 is more than double of the one of
2.6.24-rc6. But netperf result just showed about 10% regression.

As the packet only has 1 byte, so I suspect 2.6.24-rc6 tries to merge packets after waiting for
a latency. 2.6.22 might haven't the wait latency or the latency is very small, so 2.6.22 almost
sends the packets immediately. I will check the source codes later.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Rick Jones]

>>The test command is:
>>#sudo taskset -c 7 ./netserver
>>#sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1

A couple of comments/questions on the command lines:

*) netperf/netserver support CPU affinity within themselves with the 
global -T option to netperf.  Is the result with taskset much different? 
   The equivalent to the above would be to run netperf with:

./netperf -T 0,7 ...

The one possibly salient difference between the two is that when done 
within netperf, the initial process creation will take place wherever 
the scheduler wants it.

*) The -i option to set the confidence iteration count will silently cap 
the max at 30.

happy benchmarking,

rick jones

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Fri, 2008-01-11 at 09:56 -0800, Rick Jones wrote:
> >>The test command is:
> >>#sudo taskset -c 7 ./netserver
> >>#sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
> 
> A couple of comments/questions on the command lines:
Thanks for your kind comments.

> 
> *) netperf/netserver support CPU affinity within themselves with the 
> global -T option to netperf.  Is the result with taskset much different? 
>    The equivalent to the above would be to run netperf with:
> 
> ./netperf -T 0,7 ..
I checked the source codes and didn't find this option.
I use netperf V2.3 (I found the number in the makefile).

> .
> 
> The one possibly salient difference between the two is that when done 
> within netperf, the initial process creation will take place wherever 
> the scheduler wants it.
> 
> *) The -i option to set the confidence iteration count will silently cap 
> the max at 30.
Indeed, you are right.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Rick Jones]

>>*) netperf/netserver support CPU affinity within themselves with the 
>>global -T option to netperf.  Is the result with taskset much different? 
>>   The equivalent to the above would be to run netperf with:
>>
>>./netperf -T 0,7 ..
> 
> I checked the source codes and didn't find this option.
> I use netperf V2.3 (I found the number in the makefile).

Indeed, that version pre-dates the -T option.  If you weren't already 
chasing a regression I'd suggest an upgrade to 2.4.mumble.  Once you are 
at a point where changing another variable won't muddle things you may 
want to consider upgrading.

happy benchmarking,

rick jones

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Mon, 2008-01-14 at 09:46 -0800, Rick Jones wrote:
> >>*) netperf/netserver support CPU affinity within themselves with the 
> >>global -T option to netperf.  Is the result with taskset much different? 
> >>   The equivalent to the above would be to run netperf with:
> >>
> >>./netperf -T 0,7 ..
> > 
> > I checked the source codes and didn't find this option.
> > I use netperf V2.3 (I found the number in the makefile).
> 
> Indeed, that version pre-dates the -T option.  If you weren't already 
> chasing a regression I'd suggest an upgrade to 2.4.mumble.  Once you are 
> at a point where changing another variable won't muddle things you may 
> want to consider upgrading.
> 
> happy benchmarking,
Rick,

I found my UDP_RR testing is just loop in netperf instead of ping-pang between
netserver and netperf. Is it correct? TCP_RR is ok.

#./netserver
#./netperf -t UDP_RR -l 60 -H 127.0.0.1 -i 30,3 -I 99,5 -- -P 12384 -r 1,1

Thanks,
-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Tue, 2008-01-22 at 13:24 +0800, Zhang, Yanmin wrote:
> On Mon, 2008-01-14 at 09:46 -0800, Rick Jones wrote:
> > >>*) netperf/netserver support CPU affinity within themselves with the 
> > >>global -T option to netperf.  Is the result with taskset much different? 
> > >>   The equivalent to the above would be to run netperf with:
> > >>
> > >>./netperf -T 0,7 ..
> > > 
> > > I checked the source codes and didn't find this option.
> > > I use netperf V2.3 (I found the number in the makefile).
> > 
> > Indeed, that version pre-dates the -T option.  If you weren't already 
> > chasing a regression I'd suggest an upgrade to 2.4.mumble.  Once you are 
> > at a point where changing another variable won't muddle things you may 
> > want to consider upgrading.
> > 
> > happy benchmarking,
> Rick,
> 
> I found my UDP_RR testing is just loop in netperf instead of ping-pang between
> netserver and netperf. Is it correct? TCP_RR is ok.
> 
> #./netserver
> #./netperf -t UDP_RR -l 60 -H 127.0.0.1 -i 30,3 -I 99,5 -- -P 12384 -r 1,1
I digged into netperf and netserver.

netperf binds ip 0 and port 12384 to its own socket. netserver binds ip
127.0.0.1 and port 12384 to its own socket. Then, netperf calls connect to setup server
127.0.0.1 and port 12384. Then, netperf starts sends UDP packets, but all packets netperf
sends are just received by netperf itself. netserver doesn't receive any packet.

I think netperf binding should fail, or netperf shouldn't get the packet it sends out, because
netserver already binds port 12384.

I am wondering if UDP stack in kernel has a bug.

TCP_RR testing hasn't such issue.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[David Miller]

From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
Date: Tue, 22 Jan 2008 14:07:19 +0800

> I am wondering if UDP stack in kernel has a bug.

If one server binds to INADDR_ANY with port N, then any other socket
can be bound to a specific IP address with port N.  When packets
come in destined for port N, the delivery will be prioritized
to whichever socket has the more specific and matching binding.

So the kernel is fine.

Netperf just needs to be more careful in order to handle this kind of
case more cleanly.

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Mon, 2008-01-21 at 22:22 -0800, David Miller wrote:
> From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
> Date: Tue, 22 Jan 2008 14:07:19 +0800
> 
> > I am wondering if UDP stack in kernel has a bug.
> 
> If one server binds to INADDR_ANY with port N, then any other socket
> can be bound to a specific IP address with port N.  When packets
> come in destined for port N, the delivery will be prioritized
> to whichever socket has the more specific and matching binding.
What does 'more specific' mean here? I assume 127.0.0.1 should be
prioritized before 0.0.0.0 which means packets should be queued to
127.0.0.1 firstly.

> 
> So the kernel is fine.
But kernel now queues packets to 0.0.0.0.

> 
> Netperf just needs to be more careful in order to handle this kind of
> case more cleanly.
It's better if kernel works more reasonable.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Eric Dumazet]

Zhang, Yanmin a écrit :
> On Mon, 2008-01-21 at 22:22 -0800, David Miller wrote:
>> From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
>> Date: Tue, 22 Jan 2008 14:07:19 +0800
>>
>>> I am wondering if UDP stack in kernel has a bug.
>> If one server binds to INADDR_ANY with port N, then any other socket
>> can be bound to a specific IP address with port N.  When packets
>> come in destined for port N, the delivery will be prioritized
>> to whichever socket has the more specific and matching binding.
> What does 'more specific' mean here? I assume 127.0.0.1 should be
> prioritized before 0.0.0.0 which means packets should be queued to
> 127.0.0.1 firstly.

vi +278 net/ipv4/udp.c

                         int score = (sk->sk_family == PF_INET ? 1 : 0);
                         if (inet->rcv_saddr) {
                                 if (inet->rcv_saddr != daddr)
                                         continue;
                                 score+=2;
                         }
                         if (inet->daddr) {
                                 if (inet->daddr != saddr)
                                         continue;
                                 score+=2;
                         }
                         if (inet->dport) {
                                 if (inet->dport != sport)
                                         continue;
                                 score+=2;
                         }
                         if (sk->sk_bound_dev_if) {
                                 if (sk->sk_bound_dev_if != dif)
                                         continue;
                                 score+=2;
                         }

So in your case, socket bound to 127.0.0.1 should have a better score (+2) 
than other one, unless the other one got an >= score because of another match 
(rcv_saddr set or bounded to an interface)

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Eric Dumazet]

Zhang, Yanmin a écrit :
> On Tue, 2008-01-22 at 13:24 +0800, Zhang, Yanmin wrote:
>> On Mon, 2008-01-14 at 09:46 -0800, Rick Jones wrote:
>>>>> *) netperf/netserver support CPU affinity within themselves with the 
>>>>> global -T option to netperf.  Is the result with taskset much different? 
>>>>>   The equivalent to the above would be to run netperf with:
>>>>>
>>>>> ./netperf -T 0,7 ..
>>>> I checked the source codes and didn't find this option.
>>>> I use netperf V2.3 (I found the number in the makefile).
>>> Indeed, that version pre-dates the -T option.  If you weren't already 
>>> chasing a regression I'd suggest an upgrade to 2.4.mumble.  Once you are 
>>> at a point where changing another variable won't muddle things you may 
>>> want to consider upgrading.
>>>
>>> happy benchmarking,
>> Rick,
>>
>> I found my UDP_RR testing is just loop in netperf instead of ping-pang between
>> netserver and netperf. Is it correct? TCP_RR is ok.
>>
>> #./netserver
>> #./netperf -t UDP_RR -l 60 -H 127.0.0.1 -i 30,3 -I 99,5 -- -P 12384 -r 1,1
> I digged into netperf and netserver.
> 
> netperf binds ip 0 and port 12384 to its own socket. netserver binds ip
> 127.0.0.1 and port 12384 to its own socket. Then, netperf calls connect to setup server
> 127.0.0.1 and port 12384. Then, netperf starts sends UDP packets, but all packets netperf
> sends are just received by netperf itself. netserver doesn't receive any packet.
> 
> I think netperf binding should fail, or netperf shouldn't get the packet it sends out, because
> netserver already binds port 12384.
> 
> I am wondering if UDP stack in kernel has a bug.

If :
- socket A is bound to 0.0.0.0:12384 and
- socket B is bound to 127.0.0.1:12384

Then packets sent to 127.0.0.1:12384 should be queued for socket B

If they are queued to socket A as you believe it is currently done, then yes 
there is a bug in kernel.

> 
> TCP_RR testing hasn't such issue.
> 
> -yanmin
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 2221 bytes --]

On Tue, 2008-01-22 at 07:27 +0100, Eric Dumazet wrote:
> Zhang, Yanmin a écrit :
> > On Tue, 2008-01-22 at 13:24 +0800, Zhang, Yanmin wrote:
> >> On Mon, 2008-01-14 at 09:46 -0800, Rick Jones wrote:
> >>>>> *) netperf/netserver support CPU affinity within themselves with the 
> >>>>> global -T option to netperf.  Is the result with taskset much different? 
> >>>>>   The equivalent to the above would be to run netperf with:
> >>>>>
> >>>>> ./netperf -T 0,7 ..
> >>>> I checked the source codes and didn't find this option.
> >>>> I use netperf V2.3 (I found the number in the makefile).
> >>> Indeed, that version pre-dates the -T option.  If you weren't already 
> >>> chasing a regression I'd suggest an upgrade to 2.4.mumble.  Once you are 
> >>> at a point where changing another variable won't muddle things you may 
> >>> want to consider upgrading.
> >>>
> >>> happy benchmarking,
> >> Rick,
> >>
> >> I found my UDP_RR testing is just loop in netperf instead of ping-pang between
> >> netserver and netperf. Is it correct? TCP_RR is ok.
> >>
> >> #./netserver
> >> #./netperf -t UDP_RR -l 60 -H 127.0.0.1 -i 30,3 -I 99,5 -- -P 12384 -r 1,1
> > I digged into netperf and netserver.
> > 
> > netperf binds ip 0 and port 12384 to its own socket. netserver binds ip
> > 127.0.0.1 and port 12384 to its own socket. Then, netperf calls connect to setup server
> > 127.0.0.1 and port 12384. Then, netperf starts sends UDP packets, but all packets netperf
> > sends are just received by netperf itself. netserver doesn't receive any packet.
> > 
> > I think netperf binding should fail, or netperf shouldn't get the packet it sends out, because
> > netserver already binds port 12384.
> > 
> > I am wondering if UDP stack in kernel has a bug.
> 
> If :
> - socket A is bound to 0.0.0.0:12384 and
> - socket B is bound to 127.0.0.1:12384
> 
> Then packets sent to 127.0.0.1:12384 should be queued for socket B
> 
> If they are queued to socket A as you believe it is currently done, then yes 
> there is a bug in kernel.
I double-checked it and they are queued to socket A. If I define a different local port
for netperf, packets will be queued to socket B.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[David Miller]

From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
Date: Tue, 22 Jan 2008 14:52:32 +0800

> I double-checked it and they are queued to socket A. If I define a
> different local port for netperf, packets will be queued to socket
> B.

This does not prove the kernel is buggy.

If netperf is binding to devices, that could make the kernel consider
the 0.0.0.0 bound socket equally preferable to the 127.0.0.1 bound
one.  When preference is equal, the first socket in the list is
choosen.

The algorithm is in net/ipv4/udp.c:__udp4_lib_lookup(), you
can look for yourself.  It uses a scoring system to decide
which socket to match.  Binding to a specific device gives
the score two points, so does binding to a specific local
address.

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Rick Jones]

When parsing the -P option in scan_socket_args() of src/nettest_bsd.c, 
netperf is using "break_args()" from src/netsh.c which indeed if the 
command line says "-P 12345" will set both the local and remote port 
numbers to 12345.  If instead you were to say "-P 12345,"  it will use 
12345 only for the netperf side.  If you say "-P ,12345" it will use 
12345 only for the netserver side.  To set both sides at once to 
different values it would be "-P 12345,54321"

In theory, send_udp_rr() in src/nettest_bsd.c (or I suppose 
scan_socket_args() could have more code added to it to check for a UDP 
test over loopback, but probably needs to be a check for any local IP, 
and unless this becomes something bigger than "Doctor! Doctor! It hurts 
when I do this!" :) I'm inclined to leave it as caveat benchmarker and 
perhaps some additional text in the manual.

rick jones

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Tue, 2008-01-22 at 10:36 -0800, Rick Jones wrote:
> When parsing the -P option in scan_socket_args() of src/nettest_bsd.c, 
> netperf is using "break_args()" from src/netsh.c which indeed if the 
> command line says "-P 12345" will set both the local and remote port 
> numbers to 12345.  If instead you were to say "-P 12345,"  it will use 
> 12345 only for the netperf side.  If you say "-P ,12345" it will use 
> 12345 only for the netserver side.  To set both sides at once to 
> different values it would be "-P 12345,54321"
> 
> In theory, send_udp_rr() in src/nettest_bsd.c (or I suppose 
> scan_socket_args() could have more code added to it to check for a UDP 
> test over loopback, but probably needs to be a check for any local IP, 
> and unless this becomes something bigger than "Doctor! Doctor! It hurts 
> when I do this!" :) I'm inclined to leave it as caveat benchmarker and 
> perhaps some additional text in the manual.
I will instrument kernel to see if kernel does work like it is expected.

When an issue is found, we shouldn't escape by saying it's nothing to do
with me.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Wed, 2008-01-23 at 08:42 +0800, Zhang, Yanmin wrote:
> On Tue, 2008-01-22 at 10:36 -0800, Rick Jones wrote:
> > When parsing the -P option in scan_socket_args() of src/nettest_bsd.c, 
> > netperf is using "break_args()" from src/netsh.c which indeed if the 
> > command line says "-P 12345" will set both the local and remote port 
> > numbers to 12345.  If instead you were to say "-P 12345,"  it will use 
> > 12345 only for the netperf side.  If you say "-P ,12345" it will use 
> > 12345 only for the netserver side.  To set both sides at once to 
> > different values it would be "-P 12345,54321"
> > 
> > In theory, send_udp_rr() in src/nettest_bsd.c (or I suppose 
> > scan_socket_args() could have more code added to it to check for a UDP 
> > test over loopback, but probably needs to be a check for any local IP, 
> > and unless this becomes something bigger than "Doctor! Doctor! It hurts 
> > when I do this!" :) I'm inclined to leave it as caveat benchmarker and 
> > perhaps some additional text in the manual.
> I will instrument kernel to see if kernel does work like it is expected.
> 
> When an issue is found, we shouldn't escape by saying it's nothing to do
> with me.
> 
I went through netperf source again and did a step debug with gdb.

Both sides bind 0.0.0.0:12384 to its own sockets. netperf binds firstly.
When netperf calls connect to configure server 127.0.0.1:12384, kernel chooses
socket A's queue. kernel is correct.

Anther question is no matter who binds 0.0.0.0:12384 firstly, netperf
always sends packets to its own socket. I suspect API connect called by netperf to
configure server ip/port has the side-effect, as server doesn't call connect.

It's good to add additional text in netperf manual.

Sorry and thanks for your guys kind response.

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Ilpo Järvinen]

On Fri, 11 Jan 2008, Zhang, Yanmin wrote:

> On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote: 
> > The regression is:
> > 1)stoakley with 2 qual-core processors: 11%;
> > 2)Tulsa with 4 dual-core(+hyperThread) processors:13%;
> I have new update on this issue and also cc to netdev maillist.
> Thank David Miller for pointing me the netdev maillist.
> 
> > 
> > The test command is:
> > #sudo taskset -c 7 ./netserver
> > #sudo taskset -c 0 ./netperf -t TCP_RR -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -r 1,1
> > 
> > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > regression is between 16%~11%.
> > 
> > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > but the bisected kernel wasn't stable and went crazy.

TCP work between that is very much non-existing.

Using git-reset's to select a nearby merge point instead of default 
commit where bisection lands might be help in case the bisected kernel 
breaks.

Also, limiting bisection under a subsystem might reduce probability of 
brokeness (might at least be able to narrow it down quite a lot), e.g.

git bisect start net/


-- 
 i.

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 628 bytes --]

On Mon, 14 Jan 2008, Ilpo Järvinen wrote:

> On Fri, 11 Jan 2008, Zhang, Yanmin wrote:
> 
> > On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote: 
> > > 
> > > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > > regression is between 16%~11%.
> > > 
> > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > but the bisected kernel wasn't stable and went crazy.
> 
> TCP work between that is very much non-existing.

I _really_ meant 2.6.22 - 2.6.23-rc1, not 2.6.24-rc1 in case you had a 
typo there which is not that uncommon while typing kernel versions... :-)

-- 
 i.

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1096 bytes --]

On Mon, 2008-01-14 at 11:21 +0200, Ilpo Järvinen wrote:
> On Mon, 14 Jan 2008, Ilpo Järvinen wrote:
> 
> > On Fri, 11 Jan 2008, Zhang, Yanmin wrote:
> > 
> > > On Wed, 2008-01-09 at 17:35 +0800, Zhang, Yanmin wrote: 
> > > > 
> > > > As a matter of fact, 2.6.23 has about 6% regression and 2.6.24-rc's
> > > > regression is between 16%~11%.
> > > > 
> > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > but the bisected kernel wasn't stable and went crazy.
> > 
> > TCP work between that is very much non-existing.
> 
> I _really_ meant 2.6.22 - 2.6.23-rc1, not 2.6.24-rc1 in case you had a 
> typo
I did bisect 2.6.22 - 2.6.23-rc1. I also tested it on the latest 2.6.24-rc.

>  there which is not that uncommon while typing kernel versions... :-)
Thanks. I will retry bisect and bind the server/client to the same logical processor, where
I hope the result is stable this time when bisecting.

Manual testing showed there is still same or more regression if I bind the
processes on the same cpu.


Thanks a lot!

-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Herbert Xu]

On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo Järvinen wrote:
>
> > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > but the bisected kernel wasn't stable and went crazy.
> 
> TCP work between that is very much non-existing.

Make sure you haven't switched between SLAB/SLUB while testing this.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 554 bytes --]

On Mon, 2008-01-14 at 21:53 +1100, Herbert Xu wrote:
> On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo Järvinen wrote:
> >
> > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > but the bisected kernel wasn't stable and went crazy.
> > 
> > TCP work between that is very much non-existing.
> 
> Make sure you haven't switched between SLAB/SLUB while testing this.
I can make sure. In addition, I tried both SLAB and SLUB and make sure the 
regression is still there if CONFIG_SLAB=y.

Thanks,
-yanmin

Re: Netperf TCP_RR(loopback) 10% regression in 2.6.24-rc6, comparing with 2.6.22

[Zhang, Yanmin]

On Wed, 2008-01-16 at 08:34 +0800, Zhang, Yanmin wrote:
> On Mon, 2008-01-14 at 21:53 +1100, Herbert Xu wrote:
> > On Mon, Jan 14, 2008 at 08:44:40AM +0000, Ilpo Jrvinen wrote:
> > >
> > > > > I tried to use bisect to locate the bad patch between 2.6.22 and 2.6.23-rc1,
> > > > > but the bisected kernel wasn't stable and went crazy.
> > > 
> > > TCP work between that is very much non-existing.
> > 
> > Make sure you haven't switched between SLAB/SLUB while testing this.
> I can make sure. In addition, I tried both SLAB and SLUB and make sure the 
> regression is still there if CONFIG_SLAB=y.
I retried bisect between 2.6.22 and 2.6.23-rc1. This time, I enabled CONFIG_SLAB=y,
and deleted the warmup procedure in the testing scripts. In addition, bind the 2
processes on the same logical processor. The regression is about 20% which is larger
than the one when binding 2 processes to different core.

The new bisect reported cfs core patch causes it. The results of every step look
stable.

dd41f596cda0d7d6e4a8b139ffdfabcefdd46528 is first bad commit
commit dd41f596cda0d7d6e4a8b139ffdfabcefdd46528
Author: Ingo Molnar <mingo@elte.hu>
Date:   Mon Jul 9 18:51:59 2007 +0200

    sched: cfs core code
    
    apply the CFS core code.
    
    this change switches over the scheduler core to CFS's modular
    design and makes use of kernel/sched_fair/rt/idletask.c to implement
    Linux's scheduling policies.
    
    thanks to Andrew Morton and Thomas Gleixner for lots of detailed review
    feedback and for fixlets.
    
    Signed-off-by: Ingo Molnar <mingo@elte.hu>
    Signed-off-by: Mike Galbraith <efault@gmx.de>
    Signed-off-by: Dmitry Adamushko <dmitry.adamushko@gmail.com>
    Signed-off-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>


-yanmin