Re: [Lse-tech] Re: (RFC): SKB Initialization

Re: [Lse-tech] Re: (RFC): SKB Initialization

[jamal]

Mala,

Could you please at least cc netdev on networking related issues?
It says so in the kernel FAQ.
I swore back around 95 to join lk only when Linux gets a IDE maintainer
who is not insane. Hasnt happened yet.

Can you repeat your tests with the hotlist turned off (i.e set to 0)?
Also if you would be doing tests on NAPI please either copy us or netdev;
it is not nice to read weeks after you post.

Also Robert and i did a few tests and we did find skb recycling (based on
a patch from Robert a few years back) was infact giving perfomance
improvements of upto 15% over regular slab.
Did you test with that patch for the e1000 he pointed you at?
I repeated the tests (around June/July) with the tulip with input rates of
a few 100K packets/sec and noticed a improvement over regular NAPI by
about 10%. Theres one bug on the tulip which we are chasing that
might be related to tulips alignment requirements;

The idea of only freeing on the same CPU a skb allocated is free with
the e1000 NAPI driver style but not in the tulip NAPI  where a txmit
interupt might happen on a different CPU. The skb recycler patch only
recylces if allocation and freeing are happening on the same CPU;
otherwise we let the slab take the hit. On the tulip this happens about
50% of the time.

cheers,
jamal

Re: [Lse-tech] Re: (RFC): SKB Initialization

[David S. Miller]

   From: jamal <hadi@cyberus.ca>
   Date: Sun, 25 Aug 2002 12:17:42 -0400 (EDT)

   I swore back around 95 to join lk only when Linux gets a IDE maintainer
   who is not insane. Hasnt happened yet.
   
Alan is now one of the 2.5.x IDE maintainers, would you like me to add
you to linux-kernel?  :-)

Franks a lot,
David S. Miller
davem@redhat.com

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

>On Tue, 27 Aug 2002, Mala Anand wrote:

>> SPECweb99 profile shows that __kfree_skb is in the top 5 hot routines.
We
>> will test the skb recycle patch on SPECweb99 and add skbinit patch
>> to that and see how it helps.  What I understand is that the skb recycle
>> patch does not attempt to recycle if the skbs are allocated on CPU
>> and freed on another CPU. Is that right? If so, skbinit patch will help
>> those cases.

>yes it will. Not significant is my current thinking. i.e i wouldn't write
>my mother to tell her about it.

I have not looked at Robert's recycle skb patch yet. I couldn't
find it in the link he sent me so I don't know how it works. However
I thought about it a little more and realized that even when you
recycle the skbs, they need to be initialized (cleaned up). I don't
understand how
can the recycle skb patch avoid calling constructors and destructors for
the
skb.  The skbs are given back to the driver instead of freeing to the skb
hot list
or to the slab. That does not eliminate the part of the code of kfree_skb
which
releases dst, initializes part of skb and executes destructor.
Tell me if I am wrong but wouldn't it break the code. So I do think that
recycle skb patch will not mitigate the benefits of the skb init
patch.


Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

Jamal wrote ..
>On Mon, 26 Aug 2002, Mala Anand wrote:

>> Troy Wilson (who works with me) posted SPECweb99 results using my
>> skbinit patch to lkml on Friday:
>>  http://www.uwsg.iu.edu/hypermail/linux/kernel/0208.2/1470.html
>> I know you don't subscribe to lkml. Have you seen these results?
>> On Numa machine it showed around 3% improvement using SPECweb99.

>The posting you pointed to says 1% - not that it matters. It becomes more
>insignificant when skb recycling comes in play mostly because the alloc
>and freeing of skbs doesnt really show up as hotlist item within
>the profile.
>I am not saying it is totaly useless -- anything that will save a few
>cycles is good;

SPECweb99 profile shows that __kfree_skb is in the top 5 hot routines. We
will test the skb recycle patch on SPECweb99 and add skbinit patch
to that and see how it helps.  What I understand is that the skb recycle
patch does not attempt to recycle if the skbs are allocated on CPU
and freed on another CPU. Is that right? If so, skbinit patch will help
those
cases. I think this patch is pretty safe and I anticipate greater gains
on NUMA systems.

BTW the 3% gain that I reported earlier on NUMA is done at another site
of IBM and it turned out to be it is not a NUMA machine. It is also
an 8-way SMP machine, however those are non-complaint SPECWeb99 runs so
I won't be able to use those results.

The alloc and free routines are not hot in netperf3 profiles. However
I am seeing some gains there also, not significant. I will post netperf3
results with skbinit patch later.



Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[jamal]

On Tue, 27 Aug 2002, Mala Anand wrote:

> SPECweb99 profile shows that __kfree_skb is in the top 5 hot routines. We
> will test the skb recycle patch on SPECweb99 and add skbinit patch
> to that and see how it helps.  What I understand is that the skb recycle
> patch does not attempt to recycle if the skbs are allocated on CPU
> and freed on another CPU. Is that right? If so, skbinit patch will help
> those cases.

yes it will. Not significant is my current thinking. i.e i wouldnt write
my mother to tell her about it.

cheers,
jamal

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

Robert Olsson wrote..
 >In slab terms you moved part of the destructor to the constructor
 >but the main problem is still there. The skb entered the "wrong" CPU
 >so to be "reused from the slab again" the work has to done regardless
 >if it's in the constructor or destructor.
That is true if it is a uni processor but in smp the initialization,
if happened in two different CPUs, affects performance due to cache
effects.

The problem of object (skb) allocation, usage and deallocation occurring
in multiple CPUs need to be addressed separately. This patch is not
attempting to address that.

 >Eventually if we accept some cache misses a skb could possibly be
re-routed
 >to the proper slab/CPU for this we would need some skb coloring.
You still can do this. I don't see skbinit patch hindering this.

Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

>>I think the skbinit patch and recycling skbs are mutually exclusive.

>I would say they are more orthogonal than mutually exclusive.
>Although ou still need to prove that relocating the code actually helps in
>real life. On paper it looks good.
Troy Wilson (who works with me) posted SPECweb99 results using my
skbinit patch to lkml on Friday:
 http://www.uwsg.iu.edu/hypermail/linux/kernel/0208.2/1470.html
I know you don't subscribe to lkml. Have you seen these results?
On Numa machine it showed around 3% improvement using SPECweb99.


Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Robert Olsson]

Mala Anand writes:
 > I know you don't subscribe to lkml. Have you seen these results?
 > On Numa machine it showed around 3% improvement using SPECweb99.

 In slab terms you moved part of the destructor to the constructor
 but the main problem is still there. The skb entered the "wrong" CPU
 so to be "reused from the slab again" the work has to done regardless
 if it's in the constructor or destructor.

 Eventually if we accept some cache misses a skb could possibly be re-routed
 to the proper slab/CPU for this we would need some skb coloring.

 
 Also I noticed your TCP results w. e1000. 
 Here is what I see:
 
Linux 2.4.20-pre4/UP PIII @ 933 MHz w. Intel's e100 2 port GIGE adapter.
e1000 4.3.2-k1 (current kernel version) and current NAPI patch. For NAPI 
e1000 driver used RxIntDelay=1. RxIntDelay=0 caused problem now. Non-NAPI
driver used RxIntDelay=64. (default)

Three tests: TCP, UDP, packet forwarding.


Netperf. TCP socket size 131070, Single TCP stream. Test length 30 s.

M-size   e1000    NAPI-e1000
============================
      4   20.74    20.69  Mbit/s data received.
    128  458.14   465.26 
    512  836.40   846.71 
   1024  936.11   937.93 
   2048  940.65   939.92 
   4096  940.86   937.59
   8192  940.87   939.95 
  16384  940.88   937.61
  32768  940.89   939.92
  65536  940.90   939.48
 131070  940.84   939.74


Netperf. UDP_STREAM. 1440 pkts. Single UDP stream. Test length 30 s.
         e1000    NAPI-e1000
====================================
         955.7    955.7   Mbit/s data received.


Forwarding test. 1 Mpkts at 970 kpps injected.
          e1000   NAPI-e1000
=============================================
Tput       305   298580   Pkts routed.



Cheers.

						--ro

Re: [Lse-tech] Re: (RFC): SKB Initialization

[jamal]

On Mon, 26 Aug 2002, Mala Anand wrote:

> Troy Wilson (who works with me) posted SPECweb99 results using my
> skbinit patch to lkml on Friday:
>  http://www.uwsg.iu.edu/hypermail/linux/kernel/0208.2/1470.html
> I know you don't subscribe to lkml. Have you seen these results?
> On Numa machine it showed around 3% improvement using SPECweb99.
>

The posting you pointed to says 1% - not that it matters. It becomes more
insignificant when skb recycling comes in play mostly because the alloc
and freeing of skbs doesnt really show up as hotlist item within
the profile.
I am not saying it is totaly useless -- anything that will save a few
cycles is good;

cheers,
jamal

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

Jamal wrote ..

>Could you please at least cc netdev on networking related issues?
>It says so in the kernel FAQ.
>I swore back around 95 to join lk only when Linux gets a IDE maintainer
>who is not insane. Hasnt happened yet.

 Yes I will, it is a mistake from my part,

>Can you repeat your tests with the hotlist turned off (i.e set to 0)?

 Even if I turned the hot list, the slab allocator has a per cpu array of
objects. In this case it keeps by default 60 objects and hot list keeps
126 objects. So there may be a difference. I will try this.

 This skb init work is the result of my probing in to the slab cache work.
Read
my posting on slab cache:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102773718023056&w=2
This work triggered the skb init patch. To quantify the effect of bouncing
the objects between cpus, I choose skb to measure. And it turns out that
the
limited cpu array is not the culprit in this case, it is how the objects
are
allocated in one cpu and freed in another cpu is what causing the bouncing
of objects between cpus.

>Also if you would be doing tests on NAPI please either copy us or netdev;
>it is not nice to read weeks after you post.

 Yes I will.

>Also Robert and i did a few tests and we did find skb recycling (based on
>a patch from Robert a few years back) was infact giving perfomance
>improvements of upto 15% over regular slab.
>Did you test with that patch for the e1000 he pointed you at?
>I repeated the tests (around June/July) with the tulip with input rates of
>a few 100K packets/sec and noticed a improvement over regular NAPI by
>about 10%. Theres one bug on the tulip which we are chasing that
>might be related to tulips alignment requirements;

Yes I got the patch from Robert and I am planning on testing the patch. My
understanding is that skbs are recylced in other operating systems as well
to
improve performance. And it particularly helps in architectures where pci
mapping is expensive and when skbs are recycled, remapping is eliminated. I
think the skbinit patch and recycling skbs are mutually exclusive.
Recycling
skbs will reduce the number of times we hit alloc_skb and __kfree_skb.

>The idea of only freeing on the same CPU a skb allocated is free with
>the e1000 NAPI driver style but not in the tulip NAPI  where a txmit
>interupt might happen on a different CPU. The skb recycler patch only
>recylces if allocation and freeing are happening on the same CPU;
>otherwise we let the slab take the hit. On the tulip this happens about
>50% of the time.
 So skbinit patch will help the other case.

Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[jamal]

On Sun, 25 Aug 2002, Mala Anand wrote:

>
> Jamal wrote ..
>
> >Can you repeat your tests with the hotlist turned off (i.e set to 0)?
>
>  Even if I turned the hot list, the slab allocator has a per cpu array of
> objects. In this case it keeps by default 60 objects and hot list keeps
> 126 objects. So there may be a difference. I will try this.
>

Well, the hotlist is supposed to be one way to reduce the effect of
initialization.

>  This skb init work is the result of my probing in to the slab cache work.
> Read
> my posting on slab cache:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=102773718023056&w=2
> This work triggered the skb init patch. To quantify the effect of bouncing
> the objects between cpus, I choose skb to measure. And it turns out that
> the
> limited cpu array is not the culprit in this case, it is how the objects
> are
> allocated in one cpu and freed in another cpu is what causing the bouncing
> of objects between cpus.
>

Same conclusion as us then

> >Also Robert and i did a few tests and we did find skb recycling (based on
> >a patch from Robert a few years back) was infact giving perfomance
> >improvements of upto 15% over regular slab.
> >Did you test with that patch for the e1000 he pointed you at?
> >I repeated the tests (around June/July) with the tulip with input rates of
> >a few 100K packets/sec and noticed a improvement over regular NAPI by
> >about 10%. Theres one bug on the tulip which we are chasing that
> >might be related to tulips alignment requirements;
>
> Yes I got the patch from Robert and I am planning on testing the patch. My
> understanding is that skbs are recylced in other operating systems as well
> to
> improve performance. And it particularly helps in architectures where pci
> mapping is expensive and when skbs are recycled, remapping is eliminated.

standard practise for eons on networking in rtoses at least.
Slab on Linux was supposed to get rid of this. Thats why Robert
hid his original patch.

>I think the skbinit patch and recycling skbs are mutually exclusive.

I would say they are more orthogonal than mutually exclusive.
Although ou still need to prove that relocating the code actually helps in
real life. On paper it looks good.

> Recycling
> skbs will reduce the number of times we hit alloc_skb and __kfree_skb.
>

Thats what we see.

> >The idea of only freeing on the same CPU a skb allocated is free with
> >the e1000 NAPI driver style but not in the tulip NAPI  where a txmit
> >interupt might happen on a different CPU. The skb recycler patch only
> >recylces if allocation and freeing are happening on the same CPU;
> >otherwise we let the slab take the hit. On the tulip this happens about
> >50% of the time.
>  So skbinit patch will help the other case.
>

It will. Note, however that on the e1000 style coding, the hit ration is
much higher -- almost 100% in theory at least.
Perhaps its time to convert the tulip ...

cheers,
jamal

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

From: Dave Hansen <haveblue@us.ibm.com>
   Date: Fri, 23 Aug 2002 09:39:13 -0700

   Where are interrupts disabled?   I just went through a set of kernprof
   data and traced up the call graph.  In the most common __kfree_skb
   case, I do not believe that it has interupts disabled.  I could be
   wrong, but I didn't see it.

>That's completely right.  interrupts should never be disabled when
>__kfree_skb is executed.  It used to be possible when we allowed
>it to be invoked from interrupt handlers, but that is illegal and
>we have kfree_skb_irq which just reschedules the actual __kfree_skb
>to a software interrupt.

>So I agree with you, Mala's claims seem totally bogus and not well
>founded at all.
To name a few, interrupts are disabled when skbs are put back to the
hot_list
and when the cache list is accessed in the slab allocator. Am I missing
something? Please help me to understand.


Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[David S. Miller]

   From: "Mala Anand" <manand@us.ibm.com>
   Date: Fri, 23 Aug 2002 18:38:59 -0500

   To name a few, interrupts are disabled when skbs are put back to the
   hot_list

A few cycles, at best, it should not be enough to skew the profiling

   and when the cache list is accessed in the slab allocator. Am I missing
   something? Please help me to understand.

Which means if this were enough to skew profiling, it would make the
program counter of the interrupt enable instruction in the SLAB code
show up clearly in the profiles.

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

Rick wrote..
>Note that Mala said "I measured the cycles for only the
>initialization code in alloc_skb and __kfree_skb" which could mean that
>even other parts of alloc_skb() or __kfree_skb() may have gotten worse
>and you would not have known.
 Please look at my reply to Ben LeHaise which has the cycles for
 alloc_skb() and __kfree_skb(). You don't have to guess that.

> Later she admits, "As the scope of the
>code measured widens the percentage improvement comes down" and finally
>observes "We measured it in a web serving workload and found that we
>get 0.7% improvement"  which is practically in the noise.
That was initial results which had more than the posted patch. We are
still working on getting numbers.

>Dave's
>observation was that it was slightly worse (0.35%).
Are you basing this 0.35% degradation on your profile. According to
Dave's SPECweb99 results there is a 2.97% improvement in simultaneous
connections with my patch. Is that right Dave?




Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

Dave Hansen wrote..
>Mala Anand wrote:
>> The third scope would be measuring this patch in a workload environment.
>> We measured it in a web serving workload and found that we get 0.7%
>> improvement.

>First of all, the patch doesn't apply at all against the current
>bitkeeper tree.  I can post the exact one I used if you like.

>I tried this under our Specweb99 setup.  Here's a snippet of
>readprofile with, then without the patch:

>alloc:free ratio: 1.226
>(__kfree_skb+alloc_skb)/total = 3.14%


>alloc:free ratio: 0.348
>(__kfree_skb+alloc_skb)/total = 2.79%

>You can see the entire readprofile here:
>http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/

>http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/

>No, I don't know why I have so much idle time.

Readprofile ticks are not as accurate as the cycles I measured.
Moreover readprofile can give misleading information as it profiles
on timer interrupts. The alloc_skb and __kfree_skb call memory
management routines and interrupts are disabled in many parts of that code.
So I don't trust the readprofile data.



Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Dave Hansen]

Mala Anand wrote:
> Readprofile ticks are not as accurate as the cycles I measured.
> Moreover readprofile can give misleading information as it profiles
> on timer interrupts. The alloc_skb and __kfree_skb call memory
> management routines and interrupts are disabled in many parts of that code.
> So I don't trust the readprofile data.

I don't believe your results to be accurate.  They may be _precise_ 
for a small case, but you couldn't have been measuring them for very 
long.  A claim of accuracy requires a large number of samples, which 
you apparently did not do.

I can't use oprofile or other NMI-based profilers on my hardware, so 
we'll just have to guess.  Is there any chance that you have access to 
a large Specweb setup on hardware that is close to mine and can run 
oprofile?

Where are interrupts disabled?   I just went through a set of kernprof 
data and traced up the call graph.  In the most common __kfree_skb 
case, I do not believe that it has interupts disabled.  I could be 
wrong, but I didn't see it.

http://www.sr71.net/~specweb99/old/run-specweb-2300-nodynpost-2.5.31-bk+profilers-08-14-2002-02.19.22/callgraph

The end result, as I can see it, is that your patches hurt Specweb 
performance.  They moved the profile around, but there was an overall 
decline in performance.  They partly address the symptom, but not the 
real problem.  We don't need to _tune_ it, we need to fix it.

The e1000's need to allocate/free fewer skbs.  NAPI's polling mode 
_should_ help this, or at least make it possible to batch them up.
-- 
Dave Hansen
haveblue@us.ibm.com

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Bill Hartner]

Dave Hansen wrote:
> 
> Mala Anand wrote:
> > Readprofile ticks are not as accurate as the cycles I measured.
> > Moreover readprofile can give misleading information as it profiles
> > on timer interrupts. The alloc_skb and __kfree_skb call memory
> > management routines and interrupts are disabled in many parts of that code.
> > So I don't trust the readprofile data.
> 
> I don't believe your results to be accurate.  They may be _precise_
> for a small case, but you couldn't have been measuring them for very
> long.  A claim of accuracy requires a large number of samples, which
> you apparently did not do.

Dave,

What is your definition of a "very long time" ?

Read the 1st email.  There were 2.4 million samples.

How many do you think is sufficient ?

> 
> I can't use oprofile or other NMI-based profilers on my hardware, so
> we'll just have to guess.  Is there any chance that you have access to
> a large Specweb setup on hardware that is close to mine and can run
> oprofile?

Why do you think oprofile is a better way to measure this ?
BTW, Mala works with Troy Wilson who is running SPECweb99 on
an 8-way system using Apache.  Troy has run with Mala's patch
and that data will be posted.

> 
> Where are interrupts disabled?   I just went through a set of kernprof
> data and traced up the call graph.  In the most common __kfree_skb
> case, I do not believe that it has interupts disabled.  I could be
> wrong, but I didn't see it.

What is the revelance of the above ?

> 
> http://www.sr71.net/~specweb99/old/run-specweb-2300-nodynpost-2.5.31-bk+profilers-08-14-2002-02.19.22/callgraph
> 
> The end result, as I can see it, is that your patches hurt Specweb
> performance.

Based on what ?  A callgraph ? A profile ? 

Bill

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Dave Hansen]

Bill Hartner wrote:
> 
> Dave Hansen wrote:
> 
>>Mala Anand wrote:
>>
>>>Readprofile ticks are not as accurate as the cycles I measured.
>>>Moreover readprofile can give misleading information as it profiles
>>>on timer interrupts. The alloc_skb and __kfree_skb call memory
>>>management routines and interrupts are disabled in many parts of that code.
>>>So I don't trust the readprofile data.
>>
>>I don't believe your results to be accurate.  They may be _precise_
>>for a small case, but you couldn't have been measuring them for very
>>long.  A claim of accuracy requires a large number of samples, which
>>you apparently did not do.
> 
> What is your definition of a "very long time" ?
> 
> Read the 1st email.  There were 2.4 million samples.
> 
> How many do you think is sufficient ?

I must have misunderstood the data from the first email.  I was under 
the impression that it was much smaller than that number.

>>I can't use oprofile or other NMI-based profilers on my hardware, so
>>we'll just have to guess.  Is there any chance that you have access to
>>a large Specweb setup on hardware that is close to mine and can run
>>oprofile?
> 
> Why do you think oprofile is a better way to measure this ?

Mala's main complaint about readprofile is that it cannot profile 
while interrupts are disabled.  oprofile's timer interrupts cannot be 
disabled, they _always_ occur.

> BTW, Mala works with Troy Wilson who is running SPECweb99 on
> an 8-way system using Apache.  Troy has run with Mala's patch
> and that data will be posted.

I look forward to seeing it.

>>Where are interrupts disabled?   I just went through a set of kernprof
>>data and traced up the call graph.  In the most common __kfree_skb
>>case, I do not believe that it has interupts disabled.  I could be
>>wrong, but I didn't see it.
> 
> What is the revelance of the above ?

Mala's main complaint about readprofile is that it cannot profile 
while interrupts are disabled.  I didn't see the case where it was 
being called with interrupts disabled.  I was hoping that you could 
point it out to me.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Troy Wilson]

 I found the number of simultaneous connections under SPECWeb99 *
to be improved by ~1% when using Mala's SKB patch.

   2.5.25 baseline = 2656 simultaneous connections
2.5.25 + SKB patch = 2688 simultaneous connections


*  SPEC(tm) and the benchmark name SPECweb(tm) are registered
trademarks of the Standard Performance Evaluation Corporation.
This benchmarking was performed for research purposes only,
and is non-compliant, with the following deviations from the
rules -

  1 - It was run on hardware that does not meed the SPEC
      availability-to-the-public criteria.  The machine is
      an engineering sample.

  2 - access_log wasn't kept for full accounting.  It was
      being written, but deleted every 200 seconds.

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Rick Lindsley]

    Read the 1st email.  There were 2.4 million samples.

    How many do you think is sufficient ?

I looked at my hand 2.4 million times and it was not wet each time.
Therefore, it is not raining.

Of course, if I am inside a roofed structure, the sampling is faulty.
And (correct me if I'm wrong here, Dave) I think that's what we're
asking about.  Are the samples you're getting pertinent and
significant?  If, as you suggested in another email, you disable
interrupts in the functions to take these measurements, you may be
significantly altering the very environment you hope to measure.

    Why do you think oprofile is a better way to measure this ?  BTW,
    Mala works with Troy Wilson who is running SPECweb99 on an 8-way
    system using Apache.  Troy has run with Mala's patch and that data
    will be posted.

That will be helpful.  Microbenchmarks which measure cycles are far
less interesting to the community than the end results of actual
workloads.  Note that Mala said "I measured the cycles for only the
initialization code in alloc_skb and __kfree_skb" which could mean that
even other parts of alloc_skb() or __kfree_skb() may have gotten worse
and you would not have known. Later she admits, "As the scope of the
code measured widens the percentage improvement comes down" and finally
observes "We measured it in a web serving workload and found that we
get 0.7% improvement"  which is practically in the noise.  Dave's
observation was that it was slightly worse (0.35%).  Either could be
statistical noise.  If the patch only creates statistical noise, the
community won't be interested.

Also, it is well known and widely recognized that more cpus add
increasing complexity with cache and code interactions.  Have you
tested this on an 8-way machine, rather than a 2-way, and do the
results still hold?  Things which look very good on 2-proc can start to
lose their lustre on 8-proc or bigger.

I'm unfamiliar with netperf -- does it yield "results" which can be
compared?  If so, since it was used to generate the load, how did the
results of the two runs compare?

Rick

Re: [Lse-tech] Re: (RFC): SKB Initialization

[David S. Miller]

   From: Dave Hansen <haveblue@us.ibm.com>
   Date: Fri, 23 Aug 2002 09:39:13 -0700

   Where are interrupts disabled?   I just went through a set of kernprof 
   data and traced up the call graph.  In the most common __kfree_skb 
   case, I do not believe that it has interupts disabled.  I could be 
   wrong, but I didn't see it.

That's completely right.  interrupts should never be disabled when
__kfree_skb is executed.  It used to be possible when we allowed
it to be invoked from interrupt handlers, but that is illegal and
we have kfree_skb_irq which just reschedules the actual __kfree_skb
to a software interrupt.

So I agree with you, Mala's claims seem totally bogus and not well
founded at all.

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

>On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>>
>> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
>> >> The patch reduces the number of cylces by 25%
>>
>> >The data you are reporting is flawed: where are the average cycle
>> >times spent in __kfree_skb with the patch?
>>
>> I measured the cycles for only the initialization code in alloc_skb
>> and __kfree_skb. Since the init code is removed from __kfree_skb,
>> no cycles are spent there.

>Then the testing technique is flawed.  You should include all of the
>operations included in an alloc_skb/kfree_skb pair in order to see
>the overall effect of the change, otherwise your change could have a
>net negative effect which would not be noticed.

Cycles for the whole routines alloc_skb and __kfree_skb are as follows:

Baseline 2.5.25
----------------
       alloc/free average cycles
       -------------------------
Runs:      1st              2nd          3rd

CPU0:    337/1163       336/1132      304/1100
CPU1:    318/1164       309/1153      311/1127


2.5.25+skbinit patch
--------------------

       alloc/free average cycles
       -------------------------
Runs:      1st          2nd            3rd

CPU0:   447/1015       580/846        402/905
CPU1:   419/1003       383/915        547/856

The above figures indicate that the cycles spent in alloc_skb and
__kfree_skb have gained 5% in the patch case.  However if you
take the absolute cycles and average them for the three runs it
comes around 145 cycles saving that is close to what I posted earlier
by measuring just the changed code. As the scope of the code measured
widens the percentage improvement comes down.

So the first two scopes, 1. measuring the cycles spent in changed code
2. measuring the cycles spent in alloc_skb and __kfree_skb, results
are consistent.

The third scope would be measuring this patch in a workload environment.
We measured it in a web serving workload and found that we get 0.7%
improvement.

I would like to stress again that this patch helps only when the
allocations
and frees occur on two different CPUs.  I measured it in a UNI system and
did not see any impact.

Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088




                                                                                                                                               
                      Benjamin LaHaise                                                                                                         
                      <bcrl@redhat.com>                To:       Mala Anand/Austin/IBM@IBMUS                                                   
                      Sent by:                         cc:       alan@lxorguk.ukuu.org.uk, Bill Hartner/Austin/IBM@IBMUS, davem@redhat.com,    
                      lse-tech-admin@lists.sour         linux-kernel@vger.kernel.org, lse-tech@lists.sourceforge.net                           
                      ceforge.net                      Subject:  [Lse-tech] Re: (RFC): SKB Initialization                                      
                                                                                                                                               
                                                                                                                                               
                      08/21/02 01:16 PM                                                                                                        
                                                                                                                                               
                                                                                                                                               



On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>
> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
> >> The patch reduces the numer of cylces by 25%
>
> >The data you are reporting is flawed: where are the average cycle
> >times spent in __kfree_skb with the patch?
>
> I measured the cycles for only the initialization code in alloc_skb
> and __kfree_skb. Since the init code is removed from __kfree_skb,
> no cycles are spent there.

Then the testing technique is flawed.  You should include all of the
operations included in an alloc_skb/kfree_skb pair in order to see
the overall effect of the change, otherwise your change could have a
net negative effect which would not be noticed.

                         -ben
--
"You will be reincarnated as a toad; and you will be much happier."


-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone?  Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
Lse-tech mailing list
Lse-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lse-tech

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Benjamin LaHaise]

On Thu, Aug 22, 2002 at 12:22:34PM -0500, Mala Anand wrote:
> I would like to stress again that this patch helps only when the
> allocations
> and frees occur on two different CPUs.  I measured it in a UNI system and
> did not see any impact.

Thanks, that looks a lot more complete.  We discussed this on irc a bit, and 
Andi Kleen pointed out that several years of hacking on skbs has probably 
changed the layout significantly from the original intention of keeping all 
the initializations to a cacheline or two.  I also pointed out that it might 
be worth looking at cache misses and perhaps adding a prefetch instruction 
or two, especially during allocation when an skb will be used immediately.  
Another point is to check the order of writes that gcc is generating to the 
skb: if the writes are sequential, the cpu can combine them and make use of 
the internal 64 bit bus to the cache.  In combination with write buffers in 
the cpu, that makes the writes in __kfree_skb almost free, but if the cache 
lines are spread out or cold, that would explain the degredation you're 
seeing.  Cheers,

		-ben

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Dave Hansen]

Mala Anand wrote:
> The third scope would be measuring this patch in a workload environment.
> We measured it in a web serving workload and found that we get 0.7%
> improvement.

First of all, the patch doesn't apply at all against the current 
bitkeeper tree.  I can post the exact one I used if you like.

I tried this under our Specweb99 setup.  Here's a snippet of 
readprofile with, then without the patch:

   8788 __kfree_skb
   8970 mod_timer
   9095 file_read_actor
  10778 alloc_skb
  10905 skb_clone
  11368 e1000_clean_tx_irq
  13595 e1000_intr
  18367 csum_partial_copy_generic
  27848 e1000_xmit_frame
225838 poll_idle
623160 total                                      0.4107

alloc:free ratio: 1.226
(__kfree_skb+alloc_skb)/total = 3.14%

   4535 alloc_skb
   4559 do_tcp_sendpages
   4596 e1000_clean_rx_irq
   4847 dev_queue_xmit
   5020 tcp_clean_rtx_queue
   5155 batch_entropy_store
   5165 kmalloc
   5309 tcp_transmit_skb
   6060 do_schedule
   6138 qdisc_restart
   6235 tcp_v4_rcv
   6393 kfree
   6787 do_gettimeofday
   7089 __d_lookup
   7810 ip_queue_xmit
   8303 skb_clone
   8858 file_read_actor
   8885 mod_timer
   9375 .text.lock.namei
  10267 .text.lock.dec_and_lock
  10936 e1000_clean_tx_irq
  13001 __kfree_skb
  13322 skb_release_data
  13562 e1000_intr
  18099 csum_partial_copy_generic
  27447 e1000_xmit_frame
225023 poll_idle
628695 total                                      0.4143

alloc:free ratio: 0.348
(__kfree_skb+alloc_skb)/total = 2.79%

You can see the entire readprofile here:
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
No, I don't know why I have so much idle time.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [Lse-tech] Re: (RFC): SKB Initialization

[William Lee Irwin III]

On Thu, Aug 22, 2002 at 12:02:27PM -0700, Dave Hansen wrote:
> You can see the entire readprofile here:
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
> No, I don't know why I have so much idle time.

Hmm, I found that tiobench was spending a lot of time idle due to
__wait_on_inode() and get_request_wait(). I bumped up the size of
the inode wait table to 1024 and the request queue size to 16384
and saw that most of them then spent their time stuck on ->i_sem
during the initial open of the file they were going to pound on
for the duration of the run.

I determined this by just ^C'ing with kgdb and backtracing various
"stuck" processes. I think various profiling patches might be able to
give you an idea of what people are going to sleep on too.


Cheers,
Bill

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Bill Hartner]

Mala Anand wrote:
> 
> Baseline 2.5.25
> ----------------
>        alloc/free average cycles
>        -------------------------
> Runs:      1st              2nd          3rd
> 
> CPU0:    337/1163       336/1132      304/1100
> CPU1:    318/1164       309/1153      311/1127
> 
> 2.5.25+skbinit patch
> --------------------
> 
>        alloc/free average cycles
>        -------------------------
> Runs:      1st          2nd            3rd
> 
> CPU0:   447/1015       580/846        402/905
> CPU1:   419/1003       383/915        547/856
> 
> The above figures indicate that the cycles spent in alloc_skb and
> __kfree_skb have gained 5% in the patch case.  However if you
> take the absolute cycles and average them for the three runs it
> comes around 145 cycles saving that is close to what I posted earlier
> by measuring just the changed code. As the scope of the code measured
> widens the percentage improvement comes down.

Measuring just the initialization code yielded a reduction of 156 cycles.
Measuring alloc_skb and __kfree_skb yielded a reduction of 145 cycles.
This was on a 2 CPU system.

The worst case scenario would be allocating the skb header on one
CPU then freeing it on another CPU.  The best case would be doing 
all of the allocs and frees on one CPU.

You can use process/irq affinity to create both of these cases.
Can you measure these ?

Bill