e1000: Question about polling

e1000: Question about polling

[Badalian Vyacheslav]

Hello all.

Interesting think:

Have PC that do NAT. Bandwidth about 600 mbs.

Have  4 CPU (2xCoRe 2 DUO "HT OFF" 3.2 HZ).

irqbalance in kernel is off.

nat2 ~ # cat /proc/irq/217/smp_affinity
00000001
nat2 ~ # cat /proc/irq/218/smp_affinity
00000003

Load SI on CPU0 and CPU1 is about 90%

Good... try do
echo ffffffff > /proc/irq/217/smp_affinity
echo ffffffff > /proc/irq/218/smp_affinity

Get 100% SI at CPU0

Question Why?

I listen that if use IRQ from 1 netdevice to 1 CPU i can get 30% 
perfomance... but i have 4 CPU... i must get more perfomance if i cat 
"ffffffff"  to smp_affinity.

picture looks liks this:
0-3 CPU get over 50% SI.... bandwith up.... 55% SI... bandwith up... 
100% SI on CPU0....

I remember patch to fix problem like it... patched function 
e1000_clean...  kernel on pc have this patch (2.6.24-rc7-git2)... e1000 
driver work much better (i up to 1.5-2x bandwidth before i get 100% SI), 
but i think that it not get 100% that it can =)

Thanks for answers and sorry for my English

Re: e1000: Question about polling

[Jarek Poplawski]

On 18-02-2008 10:18, Badalian Vyacheslav wrote:
> Hello all.

Hi,

> Interesting think:
> 
> Have PC that do NAT. Bandwidth about 600 mbs.
> 
> Have  4 CPU (2xCoRe 2 DUO "HT OFF" 3.2 HZ).
> 
> irqbalance in kernel is off.
> 
> nat2 ~ # cat /proc/irq/217/smp_affinity
> 00000001
> nat2 ~ # cat /proc/irq/218/smp_affinity
> 00000003
> 
> Load SI on CPU0 and CPU1 is about 90%
> 
> Good... try do
> echo ffffffff > /proc/irq/217/smp_affinity
> echo ffffffff > /proc/irq/218/smp_affinity
> 
> Get 100% SI at CPU0
> 
> Question Why?

I think you should show here /proc/interrupts in all these cases.

> 
> I listen that if use IRQ from 1 netdevice to 1 CPU i can get 30% 
> perfomance... but i have 4 CPU... i must get more perfomance if i cat 
> "ffffffff"  to smp_affinity.
> 
> picture looks liks this:
> 0-3 CPU get over 50% SI.... bandwith up.... 55% SI... bandwith up... 
> 100% SI on CPU0....
> 
> I remember patch to fix problem like it... patched function 
> e1000_clean...  kernel on pc have this patch (2.6.24-rc7-git2)... e1000 
> driver work much better (i up to 1.5-2x bandwidth before i get 100% SI), 
> but i think that it not get 100% that it can =)

If some patch works for you, and you can show here its advantages,
you should probably add here some link and request for merging.

BTW, I wonder if you tried to check if changing CONFIG_HZ makes any
difference here?

Regards,
Jarek P.

Re: e1000: Question about polling

[Badalian Vyacheslav]

Sorry for little information and mistakes in letter. Jesse Brandeburg 
ask for all my questions. In future i will try to be more accurate then 
write letters and post more info.
Please not think that it disrespect for you. Its simple language barrier =(
> On 18-02-2008 10:18, Badalian Vyacheslav wrote:
>   
>> Hello all.
>>     
>
> Hi,
>
>   
>> Interesting think:
>>
>> Have PC that do NAT. Bandwidth about 600 mbs.
>>
>> Have  4 CPU (2xCoRe 2 DUO "HT OFF" 3.2 HZ).
>>
>> irqbalance in kernel is off.
>>
>> nat2 ~ # cat /proc/irq/217/smp_affinity
>> 00000001
>> nat2 ~ # cat /proc/irq/218/smp_affinity
>> 00000003
>>
>> Load SI on CPU0 and CPU1 is about 90%
>>
>> Good... try do
>> echo ffffffff > /proc/irq/217/smp_affinity
>> echo ffffffff > /proc/irq/218/smp_affinity
>>
>> Get 100% SI at CPU0
>>
>> Question Why?
>>     
>
> I think you should show here /proc/interrupts in all these cases.
>
>   
>> I listen that if use IRQ from 1 netdevice to 1 CPU i can get 30% 
>> perfomance... but i have 4 CPU... i must get more perfomance if i cat 
>> "ffffffff"  to smp_affinity.
>>
>> picture looks liks this:
>> 0-3 CPU get over 50% SI.... bandwith up.... 55% SI... bandwith up... 
>> 100% SI on CPU0....
>>
>> I remember patch to fix problem like it... patched function 
>> e1000_clean...  kernel on pc have this patch (2.6.24-rc7-git2)... e1000 
>> driver work much better (i up to 1.5-2x bandwidth before i get 100% SI), 
>> but i think that it not get 100% that it can =)
>>     
>
> If some patch works for you, and you can show here its advantages,
> you should probably add here some link and request for merging.
>
> BTW, I wonder if you tried to check if changing CONFIG_HZ makes any
> difference here?
>
> Regards,
> Jarek P.
>
>   

Re: e1000: Question about polling

[Jarek Poplawski]

On Wed, Feb 20, 2008 at 12:25:32PM +0300, Badalian Vyacheslav wrote:
...
> Please not think that it disrespect for you. Its simple language barrier =(

OK! Don't disrespect for me -  I'll try fix my English next time!)

Jarek P.

Re: e1000: Question about polling

[Badalian Vyacheslav]

Khrm.... i try to say that i have language barrier and some time may 
wrong compose clauses. Example below =)
"I'll try fix my English next time!"

Vyacheslav
> On Wed, Feb 20, 2008 at 12:25:32PM +0300, Badalian Vyacheslav wrote:
> ...
>   
>> Please not think that it disrespect for you. Its simple language barrier =(
>>     
>
> OK! Don't disrespect for me -  I'll try fix my English next time!)
>
> Jarek P.
>
>   

Re: e1000: Question about polling

[Jarek Poplawski]

On Wed, Feb 20, 2008 at 02:54:27PM +0300, Badalian Vyacheslav wrote:
> Khrm.... i try to say that i have language barrier and some time may  
> wrong compose clauses. Example below =)

No, only a bit joking...

> "I'll try fix my English next time!"

Don't worry Vyacheslav: I think your message was understandable enough
if you got good answer from Jesse. (And I've learned something BTW too;
Thanks Jesse!) And after all it's not a language group: we care here
for serious problems!

Jarek P.

RE: e1000: Question about polling

[Brandeburg, Jesse]

Badalian Vyacheslav wrote:
> Hello all.
> 
> Interesting think:
> 
> Have PC that do NAT. Bandwidth about 600 mbs.
> 
> Have  4 CPU (2xCoRe 2 DUO "HT OFF" 3.2 HZ).
> 
> irqbalance in kernel is off.
> 
> nat2 ~ # cat /proc/irq/217/smp_affinity
> 00000001
this binds all 217 irq interrupts to cpu 0

> nat2 ~ # cat /proc/irq/218/smp_affinity
> 00000003

do you mean to be balancing interrupts between core 1 and 2 here?
1 = cpu 0
2 = cpu 1
4 = cpu 2
8 = cpu 3

so 1+2 = 3 for irq 218, ie balancing between the two.

sometimes the cpus will have a paired cache, depending on your bios it
will be organized like cpu 0/2 = shared cache, and cput 1/3 = shared
cache.
you can find this out by looking at physical ID and CORE ID in
/proc/cpuinfo

> Load SI on CPU0 and CPU1 is about 90%
> 
> Good... try do
> echo ffffffff > /proc/irq/217/smp_affinity
> echo ffffffff > /proc/irq/218/smp_affinity
> 
> Get 100% SI at CPU0
> 
> Question Why?

because as each adapter generating interrupts gets rotated through cpu0,
it gets "stuck" on cpu0 because the napi scheduling can only run one at
a time, and so each is always waiting in line behind the other to run
its napi poll, always fills its quota (work_done is always != 0) and
keeps interrupts disabled "forever"

> I listen that if use IRQ from 1 netdevice to 1 CPU i can get 30%
> perfomance... but i have 4 CPU... i must get more perfomance if i cat
> "ffffffff"  to smp_affinity.

only if your performance is not cache limited but cpu horsepower
limited.  you're sacrificing cache coherency for cpu power, but if that
works for you then great.
 
> picture looks liks this:
> 0-3 CPU get over 50% SI.... bandwith up.... 55% SI... bandwith up...
> 100% SI on CPU0....
> 
> I remember patch to fix problem like it... patched function
> e1000_clean...  kernel on pc have this patch (2.6.24-rc7-git2)...
> e1000 driver work much better (i up to 1.5-2x bandwidth before i get
> 100% SI), but i think that it not get 100% that it can =)

the patch helps a little because it decreases the amount of time the
driver spends in napi mode, basically shortening the exit condition
(which reenables interrupts, and therefore balancing) to work_done <
budget, not work_done == 0.

> Thanks for answers and sorry for my English

you basically can't get much more than one cpu can do for each nic.  its
possible to get a little more, but my guess is you won't get much.  The
best thing you can do is make sure as much traffic as possible stays in
the same cache, on two different cores.

you can try turning off NAPI mode either in the .config, or build the
sourceforge driver with CFLAGS_EXTRA=-DE1000_NO_NAPI,  which seems
counterintuitive, but with the non-napi e1000 pushing packets to the
backlog queue on each cpu, you may actually get better performance due
to the balancing.

some day soon (maybe) we'll have some coherent way to have one tx and rx
interrupt per core, and enough queues for each port to be able to handle
1 queue per core.

good luck,
  Jesse  

Re: e1000: Question about polling

[Badalian Vyacheslav]

Very big thanks for this answer. You ask for all my questions and for 
all future questions too. Thanks Again!
> Badalian Vyacheslav wrote:
>   
>> Hello all.
>>
>> Interesting think:
>>
>> Have PC that do NAT. Bandwidth about 600 mbs.
>>
>> Have  4 CPU (2xCoRe 2 DUO "HT OFF" 3.2 HZ).
>>
>> irqbalance in kernel is off.
>>
>> nat2 ~ # cat /proc/irq/217/smp_affinity
>> 00000001
>>     
> this binds all 217 irq interrupts to cpu 0
>
>   
>> nat2 ~ # cat /proc/irq/218/smp_affinity
>> 00000003
>>     
>
> do you mean to be balancing interrupts between core 1 and 2 here?
> 1 = cpu 0
> 2 = cpu 1
> 4 = cpu 2
> 8 = cpu 3
>
> so 1+2 = 3 for irq 218, ie balancing between the two.
>
> sometimes the cpus will have a paired cache, depending on your bios it
> will be organized like cpu 0/2 = shared cache, and cput 1/3 = shared
> cache.
> you can find this out by looking at physical ID and CORE ID in
> /proc/cpuinfo
>
>   
>> Load SI on CPU0 and CPU1 is about 90%
>>
>> Good... try do
>> echo ffffffff > /proc/irq/217/smp_affinity
>> echo ffffffff > /proc/irq/218/smp_affinity
>>
>> Get 100% SI at CPU0
>>
>> Question Why?
>>     
>
> because as each adapter generating interrupts gets rotated through cpu0,
> it gets "stuck" on cpu0 because the napi scheduling can only run one at
> a time, and so each is always waiting in line behind the other to run
> its napi poll, always fills its quota (work_done is always != 0) and
> keeps interrupts disabled "forever"
>
>   
>> I listen that if use IRQ from 1 netdevice to 1 CPU i can get 30%
>> perfomance... but i have 4 CPU... i must get more perfomance if i cat
>> "ffffffff"  to smp_affinity.
>>     
>
> only if your performance is not cache limited but cpu horsepower
> limited.  you're sacrificing cache coherency for cpu power, but if that
> works for you then great.
>  
>   
>> picture looks liks this:
>> 0-3 CPU get over 50% SI.... bandwith up.... 55% SI... bandwith up...
>> 100% SI on CPU0....
>>
>> I remember patch to fix problem like it... patched function
>> e1000_clean...  kernel on pc have this patch (2.6.24-rc7-git2)...
>> e1000 driver work much better (i up to 1.5-2x bandwidth before i get
>> 100% SI), but i think that it not get 100% that it can =)
>>     
>
> the patch helps a little because it decreases the amount of time the
> driver spends in napi mode, basically shortening the exit condition
> (which reenables interrupts, and therefore balancing) to work_done <
> budget, not work_done == 0.
>
>   
>> Thanks for answers and sorry for my English
>>     
>
> you basically can't get much more than one cpu can do for each nic.  its
> possible to get a little more, but my guess is you won't get much.  The
> best thing you can do is make sure as much traffic as possible stays in
> the same cache, on two different cores.
>
> you can try turning off NAPI mode either in the .config, or build the
> sourceforge driver with CFLAGS_EXTRA=-DE1000_NO_NAPI,  which seems
> counterintuitive, but with the non-napi e1000 pushing packets to the
> backlog queue on each cpu, you may actually get better performance due
> to the balancing.
>
> some day soon (maybe) we'll have some coherent way to have one tx and rx
> interrupt per core, and enough queues for each port to be able to handle
> 1 queue per core.
>
> good luck,
>   Jesse  
>
>