Re: Multicast packet loss

[Wes Chow <wchow@athenacr.com>]

Eric Dumazet wrote:
> Wes Chow a écrit :
>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>
>> First I'd like to note that there were a lot of bnx2 NAPI changes between 
>> 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
>> whereas loss in 2.6.22 is significant.
>>
>> Second, some CPU affinity info: if I do like Eric and pin all of the
>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>> ksoftirqd show up on top at all!
>>
>> If I pin half the processes on one CPU and the other half on another CPU, one 
>> ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
>> in that case is significant (25%).
>>
>> Now, the strange case: if I pin 3 processes to one CPU and 1 process to 
>> another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
>> of the apps takes significantly less CPU than the others, and all apps lose the
>> *exact same number of packets*. In all other situations where we see packet
>> loss, the actual number lost per application instance appears random.
> 
> You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see 
no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 
3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any 
other combination appears to produce loss (though I have not tried all 
28 combinations, this seems to be the case).

At first I thought maybe it had to do with processes pinned to the same 
CPU, but different cores. The machine is a dual quad core, which means 
that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 
produce packet loss.

I've also noticed that it does not matter which of the working pairs I 
pin to. For example, pinning 5 processes in any combination on either 
0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no 
packet loss.

The failures are also sudden. In all of the working cases mentioned 
above, I don't see ksoftirqd on top at all. But when I run 6 processes 
on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of 
packets.

> 
> Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?


Wes