Multicast packet loss

Multicast packet loss

[Kenny Chang]

Hi all,

We've been having some issues with multicast packet loss, we were wondering
if anyone knows anything about the behavior we're seeing.

Background: we use multicast messaging with lots of messages per sec for our
work. We recently transitioned many of our systems from an Ubuntu Dapper Drake
ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've
noticed much more multicast packet loss, and we think it's related to the
transition. Our particular theory is that it's specifically a 32 vs 64-bit
issue.

We narrowed the problem down to the attached program (mcasttest.cc).  Run
"mcasttest server" on one machine -- it'll send 500,000 messages small message
to a multicast group, 50,000 messages per second.  If we run "mcasttest client"
on another machine, it'll receive all those messages and print a count at the
end of how many messages it sees. It almost never loses any messages. However,
if we run 4 copies of the client on the same machine, receiving the same data,
then the programs usually sees fewer than 500,000 messages. We're running with:

for i in $(seq 1 4); do (./mcasttest client &); done

We know this because the program prints a count, but dropped packets also
show up in ifconfig's "RX packets" section.

Things we're curious about: do other people see similar problems?  The tests
we've done: we've tried this program on a bunch of different machines, all of
which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
machines have no problems but on certain machines, Hardy shows significant loss. 
We did some experiments on a troubled machine, varying the OS install, 
including mixed installations where the kernel was 64-bit and the userspace was
32-bit. This is what we found:

On machines that exhibit this problem, the ksoftirqd process seems to be 
pegged to 100% CPU when receiving packets.

Note: while we're on Ubuntu, we've tried this with other distros and have seen
similar results, we just haven't tabulated them.

> ----------------------------------------------------------------------------
> userland | userland arch | kernel           | kernel arch | mode           
> ----------------------------------------------------------------------------
> Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
> Dapper   |            32 | 2.6.22-generic   |          32 | no packet loss 
> Dapper   |            32 | 2.6.22-server    |          32 | no packet loss 
> Hardy    |            32 | 2.6.24-rt        |          32 | no packet loss
> Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
> Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss

> Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
> Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
> Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
> Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss

> Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
> Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
> Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
> Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss
> ----------------------------------------------------------------------------

It's not exactly clear what exactly the problem is but dapper shows no issues 
regardless of what we try. For hardy, userspace seem to matter: 
2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as the userspace 
is 32-bit.

Kernel comments:
2.6.15-28-server: This is Ubuntu Dapper's stock kernel build.
2.6.24-*: This is Ubuntu Hardy's stock kernel.
2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32.
2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.

We don't think it's related to our custom kernels, because the same phenomena
show up with the Ubuntu stock kernels.

Hardware:

The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz
dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.

We've also tried AMD machines, as well as machines with Tigon3
partno(BCM95704A6) tg3 network cards, they all show consistent behavior.

Our hardy x86_64 server machines all appear to have this problem, new and old.

On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE
seem to work fine.

All of the dapper ia32 machines have no trouble, even our older hardware.


Thanks,
Kenny Chang
Athena Capital Research

Re: Multicast packet loss

[Eric Dumazet]

Kenny Chang a écrit :
> Hi all,
> 
> We've been having some issues with multicast packet loss, we were wondering
> if anyone knows anything about the behavior we're seeing.
> 
> Background: we use multicast messaging with lots of messages per sec for
> our
> work. We recently transitioned many of our systems from an Ubuntu Dapper
> Drake
> ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've
> noticed much more multicast packet loss, and we think it's related to the
> transition. Our particular theory is that it's specifically a 32 vs 64-bit
> issue.
> 
> We narrowed the problem down to the attached program (mcasttest.cc).  Run
> "mcasttest server" on one machine -- it'll send 500,000 messages small
> message
> to a multicast group, 50,000 messages per second.  If we run "mcasttest
> client"
> on another machine, it'll receive all those messages and print a count
> at the
> end of how many messages it sees. It almost never loses any messages.
> However,
> if we run 4 copies of the client on the same machine, receiving the same
> data,
> then the programs usually sees fewer than 500,000 messages. We're
> running with:
> 
> for i in $(seq 1 4); do (./mcasttest client &); done
> 
> We know this because the program prints a count, but dropped packets also
> show up in ifconfig's "RX packets" section.
> 
> Things we're curious about: do other people see similar problems?  The
> tests
> we've done: we've tried this program on a bunch of different machines,
> all of
> which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
> machines have no problems but on certain machines, Hardy shows
> significant loss. We did some experiments on a troubled machine, varying
> the OS install, including mixed installations where the kernel was
> 64-bit and the userspace was
> 32-bit. This is what we found:
> 
> On machines that exhibit this problem, the ksoftirqd process seems to be
> pegged to 100% CPU when receiving packets.
> 
> Note: while we're on Ubuntu, we've tried this with other distros and
> have seen
> similar results, we just haven't tabulated them.
> 
>> ----------------------------------------------------------------------------
>>
>> userland | userland arch | kernel           | kernel arch |
>> mode          
>> ----------------------------------------------------------------------------
>>
>> Dapper   |            32 | 2.6.15-28-server |          32 | no packet
>> loss
>> Dapper   |            32 | 2.6.22-generic   |          32 | no packet
>> loss Dapper   |            32 | 2.6.22-server    |          32 | no
>> packet loss Hardy    |            32 | 2.6.24-rt        |          32
>> | no packet loss
>> Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet
>> loss
>> Hardy    |            32 | 2.6.24-server    |          32 | ~10%
>> packet loss
> 
>> Hardy    |            32 | 2.6.22-server    |          64 | no packet
>> loss
>> Hardy    |            32 | 2.6.24-rt        |          64 | no packet
>> loss
>> Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet
>> loss
>> Hardy    |            32 | 2.6.24-server    |          64 | 12% packet
>> loss
> 
>> Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
>> Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet
>> loss
>> Hardy    |            64 | 2.6.24-server    |          64 | ~30%
>> packet loss
>> Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet
>> loss
>> ----------------------------------------------------------------------------
>>
> 
> It's not exactly clear what exactly the problem is but dapper shows no
> issues regardless of what we try. For hardy, userspace seem to matter:
> 2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as
> the userspace is 32-bit.
> 
> Kernel comments:
> 2.6.15-28-server: This is Ubuntu Dapper's stock kernel build.
> 2.6.24-*: This is Ubuntu Hardy's stock kernel.
> 2.6.22-{generic,server}: This is a custom, in-house kernel build, built
> for ia32.
> 2.6.22-vanilla: This is our custom, in-house kernel build, built for
> x86_64.
> 
> We don't think it's related to our custom kernels, because the same
> phenomena
> show up with the Ubuntu stock kernels.
> 
> Hardware:
> 
> The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz
> dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.
> 
> We've also tried AMD machines, as well as machines with Tigon3
> partno(BCM95704A6) tg3 network cards, they all show consistent behavior.
> 
> Our hardy x86_64 server machines all appear to have this problem, new
> and old.
> 
> On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel
> 82566DC GigE
> seem to work fine.
> 
> All of the dapper ia32 machines have no trouble, even our older hardware.
> 
>

Hi Kenny

Interesting... You forgot the mcasttest.cc program

Any chance you try a recent kernel (2.6.29-rcX) ?

Could you post "cat /proc/interrupts" results (one for working
 setup, another for non working/droping setup)

Re: Multicast packet loss

[Denys Fedoryschenko]

On Friday 30 January 2009 19:49:48 Kenny Chang wrote:
> Hi all,
>
> We've been having some issues with multicast packet loss, we were wondering
> if anyone knows anything about the behavior we're seeing.

I didn't work much on multicast, but i have heavy unicast udp streaming (PEP 
for satellite).

First thing to check
net.core.wmem_max = 131071
net.core.wmem_default = 124928
net.ipv4.udp_mem = 379008       505344  758016

Usually they are too small by default.

next
netstat -s

Important part
Udp:
    1263992126 packets received
    260196 packets to unknown port received.
    627001 packet receive errors
    74235906 packets sent
    RcvbufErrors: 56683
    SndbufErrors: 4295851


in your case     SndbufErrors matter.

Re: Multicast packet loss

[Neil Horman]

On Fri, Jan 30, 2009 at 12:49:48PM -0500, Kenny Chang wrote:
> Hi all,
>
> We've been having some issues with multicast packet loss, we were wondering
> if anyone knows anything about the behavior we're seeing.
>
> Background: we use multicast messaging with lots of messages per sec for our
> work. We recently transitioned many of our systems from an Ubuntu Dapper Drake
> ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've
> noticed much more multicast packet loss, and we think it's related to the
> transition. Our particular theory is that it's specifically a 32 vs 64-bit
> issue.
>
> We narrowed the problem down to the attached program (mcasttest.cc).  Run
> "mcasttest server" on one machine -- it'll send 500,000 messages small message
> to a multicast group, 50,000 messages per second.  If we run "mcasttest client"
> on another machine, it'll receive all those messages and print a count at the
> end of how many messages it sees. It almost never loses any messages. However,
> if we run 4 copies of the client on the same machine, receiving the same data,
> then the programs usually sees fewer than 500,000 messages. We're running with:
>
> for i in $(seq 1 4); do (./mcasttest client &); done
>
> We know this because the program prints a count, but dropped packets also
> show up in ifconfig's "RX packets" section.
>
> Things we're curious about: do other people see similar problems?  The tests
> we've done: we've tried this program on a bunch of different machines, all of
> which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
> machines have no problems but on certain machines, Hardy shows 
> significant loss. We did some experiments on a troubled machine, varying 
> the OS install, including mixed installations where the kernel was 64-bit 
> and the userspace was
> 32-bit. This is what we found:
>
> On machines that exhibit this problem, the ksoftirqd process seems to be  
> pegged to 100% CPU when receiving packets.
>
> Note: while we're on Ubuntu, we've tried this with other distros and have seen
> similar results, we just haven't tabulated them.
>
>> ----------------------------------------------------------------------------
>> userland | userland arch | kernel           | kernel arch | mode        
>>     
>> ----------------------------------------------------------------------------
>> Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
>> Dapper   |            32 | 2.6.22-generic   |          32 | no packet 
>> loss Dapper   |            32 | 2.6.22-server    |          32 | no 
>> packet loss Hardy    |            32 | 2.6.24-rt        |          32 | 
>> no packet loss
>> Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
>> Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss
>
>> Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
>> Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
>> Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
>> Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss
>
>> Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
>> Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
>> Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
>> Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss
>> ----------------------------------------------------------------------------
>
> It's not exactly clear what exactly the problem is but dapper shows no 
> issues regardless of what we try. For hardy, userspace seem to matter:  
> 2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as 
> the userspace is 32-bit.
>
> Kernel comments:
> 2.6.15-28-server: This is Ubuntu Dapper's stock kernel build.
> 2.6.24-*: This is Ubuntu Hardy's stock kernel.
> 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32.
> 2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.
>
> We don't think it's related to our custom kernels, because the same phenomena
> show up with the Ubuntu stock kernels.
>
> Hardware:
>
> The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz
> dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.
>
> We've also tried AMD machines, as well as machines with Tigon3
> partno(BCM95704A6) tg3 network cards, they all show consistent behavior.
>
> Our hardy x86_64 server machines all appear to have this problem, new and old.
>
> On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE
> seem to work fine.
>
> All of the dapper ia32 machines have no trouble, even our older hardware.
>

Like Eric mentioned, I'd start with a latest kernel if at all possible.  If it
doesn't happen there, you're work is half over, you just need to figure out what
changed, and tell Canonical to backport it.

>From there, you can solve this like most packet loss issues are solved:

1) Determine if its a rx or tx packet loss.  From your comments above it sounds
like this is an rx side issue

2) Look at statistics from the hardware to the application.  Use ethtool &
/proc/net/dev to get hardware packet loss stats, /proc/net/snmp netstat -s to
get core network loss stats

3) Use those stats to identify where and why packets are getting dropped.
Posting some summary of that data here is something we can help with if need be

4) Determine how to reduce the loss (i.e. code change vs. tuning)

5) Lather, rinse repeat (given that eliminating a drop cause in one location
will likely increase througput, potentially putting strain on another location
in the code path, possibly leading to more drops elsewhere. 


You had mentioned that ifconfig was showing rx drops, which indicates that your
hardware rx buffer is likely overflowing.  Usually the best way to fix that is
to:

1) modify any available interrupt coalescing parameters on the driver such that
interrupts have less latency between packet arrival and assertion

2) increase (if possible) the napi weight (I think thats still the right term)
so that each napi poll interation receives more frames on the interface,
draining that queue more quickly.

Neil

Re: Multicast packet loss

[Kenny Chang]

[-- Attachment #1: Type: text/plain, Size: 7188 bytes --]

Ah, sorry, here's the test program attached.

We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the 
2.6.29.-rcX.

Right now, we are trying to step through the kernel versions until we 
see where the performance drops significantly.  We'll try 2.6.29-rc soon 
and post the result.

Neil Norman wrote:

1) Determine if its a rx or tx packet loss.  From your comments above it sounds
like this is an rx side issue

   We're pretty sure it's an rx issue.  Other machines receiving at the 
same time will
   get all the packets.

I'll gather the information mentioned and summarize in a subsequent email.

Thanks!
Kenny

Neil Horman wrote:
> On Fri, Jan 30, 2009 at 12:49:48PM -0500, Kenny Chang wrote:
>   
>> Hi all,
>>
>> We've been having some issues with multicast packet loss, we were wondering
>> if anyone knows anything about the behavior we're seeing.
>>
>> Background: we use multicast messaging with lots of messages per sec for our
>> work. We recently transitioned many of our systems from an Ubuntu Dapper Drake
>> ia32 distribution to Ubuntu Hardy Heron x86_64. Since the transition, we've
>> noticed much more multicast packet loss, and we think it's related to the
>> transition. Our particular theory is that it's specifically a 32 vs 64-bit
>> issue.
>>
>> We narrowed the problem down to the attached program (mcasttest.cc).  Run
>> "mcasttest server" on one machine -- it'll send 500,000 messages small message
>> to a multicast group, 50,000 messages per second.  If we run "mcasttest client"
>> on another machine, it'll receive all those messages and print a count at the
>> end of how many messages it sees. It almost never loses any messages. However,
>> if we run 4 copies of the client on the same machine, receiving the same data,
>> then the programs usually sees fewer than 500,000 messages. We're running with:
>>
>> for i in $(seq 1 4); do (./mcasttest client &); done
>>
>> We know this because the program prints a count, but dropped packets also
>> show up in ifconfig's "RX packets" section.
>>
>> Things we're curious about: do other people see similar problems?  The tests
>> we've done: we've tried this program on a bunch of different machines, all of
>> which are running either dapper ia32 or hardy x86_64. Uniformly, the dapper
>> machines have no problems but on certain machines, Hardy shows 
>> significant loss. We did some experiments on a troubled machine, varying 
>> the OS install, including mixed installations where the kernel was 64-bit 
>> and the userspace was
>> 32-bit. This is what we found:
>>
>> On machines that exhibit this problem, the ksoftirqd process seems to be  
>> pegged to 100% CPU when receiving packets.
>>
>> Note: while we're on Ubuntu, we've tried this with other distros and have seen
>> similar results, we just haven't tabulated them.
>>
>>     
>>> ----------------------------------------------------------------------------
>>> userland | userland arch | kernel           | kernel arch | mode        
>>>     
>>> ----------------------------------------------------------------------------
>>> Dapper   |            32 | 2.6.15-28-server |          32 | no packet loss
>>> Dapper   |            32 | 2.6.22-generic   |          32 | no packet 
>>> loss Dapper   |            32 | 2.6.22-server    |          32 | no 
>>> packet loss Hardy    |            32 | 2.6.24-rt        |          32 | 
>>> no packet loss
>>> Hardy    |            32 | 2.6.24-generic   |          32 | ~5% packet loss
>>> Hardy    |            32 | 2.6.24-server    |          32 | ~10% packet loss
>>>       
>>> Hardy    |            32 | 2.6.22-server    |          64 | no packet loss
>>> Hardy    |            32 | 2.6.24-rt        |          64 | no packet loss
>>> Hardy    |            32 | 2.6.24-generic   |          64 | 14% packet loss
>>> Hardy    |            32 | 2.6.24-server    |          64 | 12% packet loss
>>>       
>>> Hardy    |            64 | 2.6.22-vanilla   |          64 | packet loss
>>> Hardy    |            64 | 2.6.24-rt        |          64 | ~5% packet loss
>>> Hardy    |            64 | 2.6.24-server    |          64 | ~30% packet loss
>>> Hardy    |            64 | 2.6.24-generic   |          64 | ~5% packet loss
>>> ----------------------------------------------------------------------------
>>>       
>> It's not exactly clear what exactly the problem is but dapper shows no 
>> issues regardless of what we try. For hardy, userspace seem to matter:  
>> 2.6.24-rt kernel shows no packet loss for 32&64bit kernels, as long as 
>> the userspace is 32-bit.
>>
>> Kernel comments:
>> 2.6.15-28-server: This is Ubuntu Dapper's stock kernel build.
>> 2.6.24-*: This is Ubuntu Hardy's stock kernel.
>> 2.6.22-{generic,server}: This is a custom, in-house kernel build, built for ia32.
>> 2.6.22-vanilla: This is our custom, in-house kernel build, built for x86_64.
>>
>> We don't think it's related to our custom kernels, because the same phenomena
>> show up with the Ubuntu stock kernels.
>>
>> Hardware:
>>
>> The benchmark machine We've been using is an Intel Xeon E5440 @2.83GHz
>> dual-cpu quad-core with Broadcom NetXtreme II BCM5708 bnx2 networking.
>>
>> We've also tried AMD machines, as well as machines with Tigon3
>> partno(BCM95704A6) tg3 network cards, they all show consistent behavior.
>>
>> Our hardy x86_64 server machines all appear to have this problem, new and old.
>>
>> On the other hand, a desktop with Intel Q6600 quad core 2.4GHz and Intel 82566DC GigE
>> seem to work fine.
>>
>> All of the dapper ia32 machines have no trouble, even our older hardware.
>>
>>     
>
> Like Eric mentioned, I'd start with a latest kernel if at all possible.  If it
> doesn't happen there, you're work is half over, you just need to figure out what
> changed, and tell Canonical to backport it.
>
> From there, you can solve this like most packet loss issues are solved:
>
> 1) Determine if its a rx or tx packet loss.  From your comments above it sounds
> like this is an rx side issue
>
> 2) Look at statistics from the hardware to the application.  Use ethtool &
> /proc/net/dev to get hardware packet loss stats, /proc/net/snmp netstat -s to
> get core network loss stats
>
> 3) Use those stats to identify where and why packets are getting dropped.
> Posting some summary of that data here is something we can help with if need be
>
> 4) Determine how to reduce the loss (i.e. code change vs. tuning)
>
> 5) Lather, rinse repeat (given that eliminating a drop cause in one location
> will likely increase througput, potentially putting strain on another location
> in the code path, possibly leading to more drops elsewhere. 
>
>
> You had mentioned that ifconfig was showing rx drops, which indicates that your
> hardware rx buffer is likely overflowing.  Usually the best way to fix that is
> to:
>
> 1) modify any available interrupt coalescing parameters on the driver such that
> interrupts have less latency between packet arrival and assertion
>
> 2) increase (if possible) the napi weight (I think thats still the right term)
> so that each napi poll interation receives more frames on the interface,
> draining that queue more quickly.
>
> Neil
>
>   


[-- Attachment #2: mcasttest.c --]
[-- Type: text/x-csrc, Size: 3166 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/epoll.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <unistd.h>


void error(const char *s)
{
    fprintf(stderr, "%s\n", s);
    exit(1);
}

void check(int v)
{
    int myerr = errno;
    char *myerrstr = strerror(myerr);
    if(!v)
        error("bad return code");
}

const char *g_mcastaddr = "239.100.0.99";
int g_port = 10100;

int main(int argc, char **argv)
{
    if(argc != 2)
        error("usage: mcasttest (server|client)");
    if(strcmp(argv[1], "client") == 0)
    {
        // Client program: subscribes to a multicast group, receives messages
        // and prints a count of messages received once it's done.

        int s = socket(AF_INET, SOCK_DGRAM, 0);
        check(s > 0);
        int val = 1;
        check(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) == 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        addr.sin_addr.s_addr = htonl(INADDR_ANY);
        check(bind(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        struct ip_mreqn mreq;
        memset(&mreq, 0, sizeof(mreq));
        check(inet_pton(AF_INET, g_mcastaddr, &mreq.imr_multiaddr));
        mreq.imr_address.s_addr = htonl(INADDR_ANY);
        mreq.imr_ifindex = 0;
        check(setsockopt(s, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) == 0);

        int bufSz;
        socklen_t len = sizeof(bufSz);
        getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char*)(&bufSz), &len);
        printf("bufsz: %d\n", bufSz);

        int npackets = 0;
        char buf[1000];
        memset(buf, 0, sizeof(buf));
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            check(recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen) == 100);
            ++npackets;
            if(buf[0] == 1) // exit message
                break;
        }
        printf("received %d packets\n", npackets);
    }
    else if(strcmp(argv[1], "server") == 0)
    {
        // Server program: sends 50,000 packets per second to a multicast address,
        // for 10 seconds.
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        int val = 1;
        int i = 1;
        check(s > 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        check(inet_pton(AF_INET, g_mcastaddr, &addr.sin_addr.s_addr));
        check(connect(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        int npackets = 500000;
        char buf[100];
        memset(buf, 0, sizeof(buf));
        for(i = 1; i < npackets; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            usleep(20); // 50,000 messages per second
        }

        buf[0] = 1;
        for(i = 1; i < 5; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            sleep(1);
        }
    }
    else
        error("unknown mode");
    return 0;
}

Re: Multicast packet loss

[Eric Dumazet]

Kenny Chang a écrit :
> Ah, sorry, here's the test program attached.
> 
> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
> 2.6.29.-rcX.
> 
> Right now, we are trying to step through the kernel versions until we
> see where the performance drops significantly.  We'll try 2.6.29-rc soon
> and post the result.

2.6.29-rc contains UDP receive improvements (lockless)

Problem is multicast handling was not yet updated, but could be :)


I was asking you "cat /proc/interrupts" because I believe you might
have a problem NIC interrupts being handled by one CPU only (when having problems)

Re: Multicast packet loss

[Neil Horman]

On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
> Kenny Chang a écrit :
> > Ah, sorry, here's the test program attached.
> > 
> > We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
> > 2.6.29.-rcX.
> > 
> > Right now, we are trying to step through the kernel versions until we
> > see where the performance drops significantly.  We'll try 2.6.29-rc soon
> > and post the result.
> 
> 2.6.29-rc contains UDP receive improvements (lockless)
> 
> Problem is multicast handling was not yet updated, but could be :)
> 
> 
> I was asking you "cat /proc/interrupts" because I believe you might
> have a problem NIC interrupts being handled by one CPU only (when having problems)
> 
That would be expected (if irqbalance is running), and desireable, since
spreading high volume interrupts like NICS accross multiple cores (or more
specifically multiple L2 caches), is going increase your cache line miss rate
significantly and decrease rx throughput.

Although you do have a point here, if the system isn't running irqbalance, and
the NICS irq affinity is spread accross multiple L2 caches, that would be a
point of improvement performance-wise.  

Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
and your stats that I asked about earlier, that would be a big help.

Regards
Neil

Re: Multicast packet loss

[Kenny Chang]

Neil Horman wrote:
> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>   
>> Kenny Chang a écrit :
>>     
>>> Ah, sorry, here's the test program attached.
>>>
>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>> 2.6.29.-rcX.
>>>
>>> Right now, we are trying to step through the kernel versions until we
>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>> and post the result.
>>>       
>> 2.6.29-rc contains UDP receive improvements (lockless)
>>
>> Problem is multicast handling was not yet updated, but could be :)
>>
>>
>> I was asking you "cat /proc/interrupts" because I believe you might
>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>
>>     
> That would be expected (if irqbalance is running), and desireable, since
> spreading high volume interrupts like NICS accross multiple cores (or more
> specifically multiple L2 caches), is going increase your cache line miss rate
> significantly and decrease rx throughput.
>
> Although you do have a point here, if the system isn't running irqbalance, and
> the NICS irq affinity is spread accross multiple L2 caches, that would be a
> point of improvement performance-wise.  
>
> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
> and your stats that I asked about earlier, that would be a big help.
>
> Regards
> Neil
>
>   
Hi Neil,

Here's the information you requested.

Kenny

kchang@beast8:~$ uname -a
Linux beast8 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 
GNU/Linux
kchang@beast8:~$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5322.91
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 1
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.03
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 2
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.06
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 3
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.06
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 4
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 0
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.07
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 5
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 1
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.07
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 6
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 2
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.08
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 7
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 3
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.08
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

kchang@beast8:~$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       
CPU5       CPU6       CPU7      
  0:         67          0          1          0          0          
0          0          0   IO-APIC-edge      timer
  1:          0          0          0          0          0          
0          0          0   IO-APIC-edge      i8042
  8:          0          0          0          1          0          
0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0          0          
0          0          0   IO-APIC-fasteoi   acpi
 14:         12         13         13         13         12         
10         13         13   IO-APIC-edge      libata
 15:          0          0          0          0          0          
0          0          0   IO-APIC-edge      libata
 17:        294        295        293        294        294        
296        293        288   IO-APIC-fasteoi   aacraid
 22:          6          5          5          5          6          
6          6          6   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          7          8          8          7          7          
8          7          8   IO-APIC-fasteoi   ehci_hcd:usb1, 
uhci_hcd:usb2, uhci_hcd:usb4
2294:         48         46         48         48         49         
47         48         51   PCI-MSI-edge      eth0
NMI:          0          0          0          0          0          
0          0          0   Non-maskable interrupts
LOC:       5088       3394       3129       2835       2561       
2938       2576       2798   Local timer interrupts
RES:         59        119         58         36         34         
71         50         17   Rescheduling interrupts
CAL:        132        128        149        141        138        
140        152        140   function call interrupts
TLB:        285        178        278        183        297        
191        295        157   TLB shootdowns
TRM:          0          0          0          0          0          
0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          
0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0          0          
0          0          0   Spurious interrupts
ERR:          0

Re: Multicast packet loss

[Kenny Chang]

Neil Horman wrote:
> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>   
>> Kenny Chang a écrit :
>>     
>>> Ah, sorry, here's the test program attached.
>>>
>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>> 2.6.29.-rcX.
>>>
>>> Right now, we are trying to step through the kernel versions until we
>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>> and post the result.
>>>       
>> 2.6.29-rc contains UDP receive improvements (lockless)
>>
>> Problem is multicast handling was not yet updated, but could be :)
>>
>>
>> I was asking you "cat /proc/interrupts" because I believe you might
>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>
>>     
> That would be expected (if irqbalance is running), and desireable, since
> spreading high volume interrupts like NICS accross multiple cores (or more
> specifically multiple L2 caches), is going increase your cache line miss rate
> significantly and decrease rx throughput.
>
> Although you do have a point here, if the system isn't running irqbalance, and
> the NICS irq affinity is spread accross multiple L2 caches, that would be a
> point of improvement performance-wise.  
>
> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
> and your stats that I asked about earlier, that would be a big help.
>
> Regards
> Neil
>
>   
This is for a working setup.

-Kenny

kchang@fiji:~$ uname -a
Linux fiji 2.6.24-19-generic #1 SMP Wed Aug 20 17:53:40 UTC 2008 x86_64 
GNU/Linux
kchang@fiji:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping        : 11
cpu MHz         : 1600.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr lahf_lm
bogomips    : 4791.31
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 15
model name    : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping    : 11
cpu MHz        : 1600.000
cache size    : 4096 KB
physical id    : 0
siblings    : 4
core id        : 1
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr lahf_lm
bogomips    : 4788.05
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 15
model name    : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping    : 11
cpu MHz        : 1600.000
cache size    : 4096 KB
physical id    : 0
siblings    : 4
core id        : 2
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr lahf_lm
bogomips    : 4788.08
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 15
model name    : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping    : 11
cpu MHz        : 1600.000
cache size    : 4096 KB
physical id    : 0
siblings    : 4
core id        : 3
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr lahf_lm
bogomips    : 4788.07
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

kchang@fiji:~$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3      
  0:        165          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0   IO-APIC-edge      i8042
  8:          1          0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0   IO-APIC-edge      i8042
 16:   25614400          0          0          0   IO-APIC-fasteoi   
uhci_hcd:usb3, HDA Intel, eth1, nvidia
 17:     571932          0          0          0   IO-APIC-fasteoi   
uhci_hcd:usb4, uhci_hcd:usb6
 18:     102824          0          0          0   IO-APIC-fasteoi   
uhci_hcd:usb7
 22:          0          0          0          0   IO-APIC-fasteoi   
ehci_hcd:usb1
 23:    1636819          0          0          0   IO-APIC-fasteoi   
ehci_hcd:usb2, uhci_hcd:usb5
507:   12542966          0          0          0   PCI-MSI-edge      eth0
508:    1201118          0          0          0   PCI-MSI-edge      ahci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:   29214662   20141857   21777347   14279251   Local timer interrupts
RES:     205758     173268     238058     123958   Rescheduling interrupts
CAL:       2623       3732       3814       2747   function call interrupts
TLB:      29961      56621      31440      55783   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0

Re: Multicast packet loss

[Neil Horman]

On Mon, Feb 02, 2009 at 11:48:25AM -0500, Kenny Chang wrote:
> Neil Horman wrote:
>> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>>   
>>> Kenny Chang a écrit :
>>>     
>>>> Ah, sorry, here's the test program attached.
>>>>
>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>> 2.6.29.-rcX.
>>>>
>>>> Right now, we are trying to step through the kernel versions until we
>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>> and post the result.
>>>>       
>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>
>>> Problem is multicast handling was not yet updated, but could be :)
>>>
>>>
>>> I was asking you "cat /proc/interrupts" because I believe you might
>>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>>
>>>     
>> That would be expected (if irqbalance is running), and desireable, since
>> spreading high volume interrupts like NICS accross multiple cores (or more
>> specifically multiple L2 caches), is going increase your cache line miss rate
>> significantly and decrease rx throughput.
>>
>> Although you do have a point here, if the system isn't running irqbalance, and
>> the NICS irq affinity is spread accross multiple L2 caches, that would be a
>> point of improvement performance-wise.  
>>
>> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
>> and your stats that I asked about earlier, that would be a big help.
>>
>> Regards
>> Neil
>>
>>   
> This is for a working setup.
>

Are these quad core systems?  Or dual core w/ hyperthreading?  I ask because in
your working setup you have 1/2 the number of cpus' and was not sure if you
removed an entire package of if you just disabled hyperthreading.


Neil

Re: Multicast packet loss

[Kenny Chang]

Neil Horman wrote:
> On Mon, Feb 02, 2009 at 11:48:25AM -0500, Kenny Chang wrote:
>   
>> Neil Horman wrote:
>>     
>>> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>>>   
>>>       
>>>> Kenny Chang a écrit :
>>>>     
>>>>         
>>>>> Ah, sorry, here's the test program attached.
>>>>>
>>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>>> 2.6.29.-rcX.
>>>>>
>>>>> Right now, we are trying to step through the kernel versions until we
>>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>>> and post the result.
>>>>>       
>>>>>           
>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>>
>>>> Problem is multicast handling was not yet updated, but could be :)
>>>>
>>>>
>>>> I was asking you "cat /proc/interrupts" because I believe you might
>>>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>>>
>>>>     
>>>>         
>>> That would be expected (if irqbalance is running), and desireable, since
>>> spreading high volume interrupts like NICS accross multiple cores (or more
>>> specifically multiple L2 caches), is going increase your cache line miss rate
>>> significantly and decrease rx throughput.
>>>
>>> Although you do have a point here, if the system isn't running irqbalance, and
>>> the NICS irq affinity is spread accross multiple L2 caches, that would be a
>>> point of improvement performance-wise.  
>>>
>>> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
>>> and your stats that I asked about earlier, that would be a big help.
>>>
>>> Regards
>>> Neil
>>>
>>>   
>>>       
>> This is for a working setup.
>>
>>     
>
> Are these quad core systems?  Or dual core w/ hyperthreading?  I ask because in
> your working setup you have 1/2 the number of cpus' and was not sure if you
> removed an entire package of if you just disabled hyperthreading.
>
>
> Neil
>
>   
Yeah, these are quad core systems.  The 8 cpu system is a dual-processor 
quad-core.  The other is my desktop, single cpu quad core.

Kenny

Re: Multicast packet loss

[Neil Horman]

On Tue, Feb 03, 2009 at 10:20:13AM -0500, Kenny Chang wrote:
> Neil Horman wrote:
>> On Mon, Feb 02, 2009 at 11:48:25AM -0500, Kenny Chang wrote:
>>   
>>> Neil Horman wrote:
>>>     
>>>> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>>>>         
>>>>> Kenny Chang a écrit :
>>>>>             
>>>>>> Ah, sorry, here's the test program attached.
>>>>>>
>>>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>>>> 2.6.29.-rcX.
>>>>>>
>>>>>> Right now, we are trying to step through the kernel versions until we
>>>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>>>> and post the result.
>>>>>>                 
>>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>>>
>>>>> Problem is multicast handling was not yet updated, but could be :)
>>>>>
>>>>>
>>>>> I was asking you "cat /proc/interrupts" because I believe you might
>>>>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>>>>
>>>>>             
>>>> That would be expected (if irqbalance is running), and desireable, since
>>>> spreading high volume interrupts like NICS accross multiple cores (or more
>>>> specifically multiple L2 caches), is going increase your cache line miss rate
>>>> significantly and decrease rx throughput.
>>>>
>>>> Although you do have a point here, if the system isn't running irqbalance, and
>>>> the NICS irq affinity is spread accross multiple L2 caches, that would be a
>>>> point of improvement performance-wise.  
>>>>
>>>> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
>>>> and your stats that I asked about earlier, that would be a big help.
>>>>
>>>> Regards
>>>> Neil
>>>>
>>>>         
>>> This is for a working setup.
>>>
>>>     
>>
>> Are these quad core systems?  Or dual core w/ hyperthreading?  I ask because in
>> your working setup you have 1/2 the number of cpus' and was not sure if you
>> removed an entire package of if you just disabled hyperthreading.
>>
>>
>> Neil
>>
>>   
> Yeah, these are quad core systems.  The 8 cpu system is a dual-processor  
> quad-core.  The other is my desktop, single cpu quad core.
>
Ok, so their separate systms then.  Did you actually experience drops on the
8-core system since the last reboot?  I ask because even when its distributed
across all 8 cores, you only have about 500 total interrupts from the NIC, and
if you did get drops, something more than just affinity is wrong.

Regardless, spreading interrupts across cores is definately a problem.  As eric
says, quad core chips are actually 2x2 cores, so you'll want to either just run
irqbalance to assign an apropriate affinity to the NIC, or manually look at each
cores physical id and sibling id, to assign affininty to a core or cores that
share an L2 cache.  If you need to, as you've found, you may need to disable msi
interrupt mode on your bnx2 driver.  That kinda stinks, but bnx2 IIRC isn't
multiqueue, so its not like msi provides you any real performance gain.

Neil

> Kenny
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Multicast packet loss

[Kenny Chang]

Neil Horman wrote:
> On Tue, Feb 03, 2009 at 10:20:13AM -0500, Kenny Chang wrote:
>   
>> Neil Horman wrote:
>>     
>>> On Mon, Feb 02, 2009 at 11:48:25AM -0500, Kenny Chang wrote:
>>>   
>>>       
>>>> Neil Horman wrote:
>>>>     
>>>>         
>>>>> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>>>>>         
>>>>>           
>>>>>> Kenny Chang a écrit :
>>>>>>             
>>>>>>             
>>>>>>> Ah, sorry, here's the test program attached.
>>>>>>>
>>>>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>>>>> 2.6.29.-rcX.
>>>>>>>
>>>>>>> Right now, we are trying to step through the kernel versions until we
>>>>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>>>>> and post the result.
>>>>>>>                 
>>>>>>>               
>>>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>>>>
>>>>>> Problem is multicast handling was not yet updated, but could be :)
>>>>>>
>>>>>>
>>>>>> I was asking you "cat /proc/interrupts" because I believe you might
>>>>>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>>>>>
>>>>>>             
>>>>>>             
>>>>> That would be expected (if irqbalance is running), and desireable, since
>>>>> spreading high volume interrupts like NICS accross multiple cores (or more
>>>>> specifically multiple L2 caches), is going increase your cache line miss rate
>>>>> significantly and decrease rx throughput.
>>>>>
>>>>> Although you do have a point here, if the system isn't running irqbalance, and
>>>>> the NICS irq affinity is spread accross multiple L2 caches, that would be a
>>>>> point of improvement performance-wise.  
>>>>>
>>>>> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
>>>>> and your stats that I asked about earlier, that would be a big help.
>>>>>
>>>>> Regards
>>>>> Neil
>>>>>
>>>>>         
>>>>>           
>>>> This is for a working setup.
>>>>
>>>>     
>>>>         
>>> Are these quad core systems?  Or dual core w/ hyperthreading?  I ask because in
>>> your working setup you have 1/2 the number of cpus' and was not sure if you
>>> removed an entire package of if you just disabled hyperthreading.
>>>
>>>
>>> Neil
>>>
>>>   
>>>       
>> Yeah, these are quad core systems.  The 8 cpu system is a dual-processor  
>> quad-core.  The other is my desktop, single cpu quad core.
>>
>>     
> Ok, so their separate systms then.  Did you actually experience drops on the
> 8-core system since the last reboot?  I ask because even when its distributed
> across all 8 cores, you only have about 500 total interrupts from the NIC, and
> if you did get drops, something more than just affinity is wrong.
>
> Regardless, spreading interrupts across cores is definately a problem.  As eric
> says, quad core chips are actually 2x2 cores, so you'll want to either just run
> irqbalance to assign an apropriate affinity to the NIC, or manually look at each
> cores physical id and sibling id, to assign affininty to a core or cores that
> share an L2 cache.  If you need to, as you've found, you may need to disable msi
> interrupt mode on your bnx2 driver.  That kinda stinks, but bnx2 IIRC isn't
> multiqueue, so its not like msi provides you any real performance gain.
>
> Neil
>
>   
Hi Neil,

Yeah, we've been rebooting this system left and right switch kernels.  
The results are fairly consistent.  We were able to set the irq 
affinities, and as Wes had mentioned, what we see is that if we pin the 
softirq to 1 core, and pin the app to its sibling, we see really good 
performance, but as we load up other cores, the machine reaches a 
breaking point where all hell breaks loose and we drop a bunch.  (we 
hadn't turned off msi btw..)

While we were able to tune and adjust performance like that, in the end, 
it doesn't really explain the difference between earlier and recent 
kernels, also it doesn't quite explain the difference between machines.

You mentioned it would be good to see the interrupts for each kernel, in 
light of the above information, would it still be useful for me to 
provide that?

Kenny

Re: Multicast packet loss

[Wesley Chow]

>>>>>
>>>>>
>>>> Are these quad core systems?  Or dual core w/ hyperthreading?  I  
>>>> ask because in
>>>> your working setup you have 1/2 the number of cpus' and was not  
>>>> sure if you
>>>> removed an entire package of if you just disabled hyperthreading.
>>>>
>>>>
>>>> Neil
>>>>
>>>>
>>> Yeah, these are quad core systems.  The 8 cpu system is a dual- 
>>> processor  quad-core.  The other is my desktop, single cpu quad  
>>> core.
>>>
>>>


Just to be clear: on the 2 x quad core system, we can run with a  
2.6.15 kernel and see no packet drops. In fact, we can run with  
2.6.19, 2.6.20, and 2.6.21 just fine. 2.6.22 is the first kernel that  
shows problems.

Kenny posted results from a working setup on a different machine.

What I would really like to know is if whatever changed between 2.6.21  
and 2.6.22 that broke things is confined just to bnx2. To make this a  
rigorous test, we would need to use the same machine with a different  
nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is  
in the mail as I type this.

I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22  
(unsuccessfully), and building the most recent driver from the  
Broadcom site to Ubuntu Hardy's 2.6.24. The most recent driver with  
hardy 2.6.24 showed similar packet dropping problems. Hm, perhaps I'll  
try to build the most recent broadcom driver against 2.6.21.


Wes

Re: Multicast packet loss

[Eric Dumazet]

Wesley Chow a écrit :
>>>>>>
>>>>>>
>>>>> Are these quad core systems?  Or dual core w/ hyperthreading?  I
>>>>> ask because in
>>>>> your working setup you have 1/2 the number of cpus' and was not
>>>>> sure if you
>>>>> removed an entire package of if you just disabled hyperthreading.
>>>>>
>>>>>
>>>>> Neil
>>>>>
>>>>>
>>>> Yeah, these are quad core systems.  The 8 cpu system is a
>>>> dual-processor  quad-core.  The other is my desktop, single cpu quad
>>>> core.
>>>>
>>>>
> 
> 
> Just to be clear: on the 2 x quad core system, we can run with a 2.6.15
> kernel and see no packet drops. In fact, we can run with 2.6.19, 2.6.20,
> and 2.6.21 just fine. 2.6.22 is the first kernel that shows problems.
> 
> Kenny posted results from a working setup on a different machine.
> 
> What I would really like to know is if whatever changed between 2.6.21
> and 2.6.22 that broke things is confined just to bnx2. To make this a
> rigorous test, we would need to use the same machine with a different
> nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is
> in the mail as I type this.
> 
> I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22
> (unsuccessfully), and building the most recent driver from the Broadcom
> site to Ubuntu Hardy's 2.6.24. The most recent driver with hardy 2.6.24
> showed similar packet dropping problems. Hm, perhaps I'll try to build
> the most recent broadcom driver against 2.6.21.
> 

Try oprofile session, you shall see a scheduler effect (dont want to call
this a regression, no need for another flame war).

also give us "vmstat 1" results  (number of context switches per second)

On recent kernels, scheduler might be faster than before: You get more wakeups per
second and more work to do by softirq handler (it does more calls to scheduler,
thus less cpu cycles available for draining NIC RX queue in time)

opcontrol --vmlinux=/path/vmlinux --start
<run benchmark>
opreport -l /path/vmlinux | head -n 50

Recent schedulers tend to be optimum for lower latencies (and thus, on
a high level of wakeups, you get less bandwidth because of sofirq using
a whole CPU)

For example, if you have one tread receiving data on 4 or 8 sockets, you'll
probably notice better throughput (because it will sleep less often)

Multicast receiving on N sockets, with one thread waiting on each socket
is basically a way to trigger a scheduler storm. (N wakeups per packet).
So its more a benchmark to stress scheduler than stressing network stack...


Maybe its time to change user side, and not try to find an appropriate kernel :)

If you know you have to receive N frames per 20us units, then its better to :
Use non blocking sockets, and doing such loop :

{
usleep(20); // or try to compensate if this thread is slowed too much by following code
for (i = 0 ; i < N ; i++) {
	while (revfrom(socket[N], ....) != -1)
		receive_frame(...);
	}
}

That way, you are pretty sure network softirq handler wont have to spend time trying
to wakeup 400.000 time per second one thread. All cpu cycles can be spent in NIC driver
and network stack.

Your thread will do 50.000 calls to nanosleep() per second, that is not really expensive,
then N recvfrom() per iteration. It should work on all past , current and future kernels.

Re: Multicast packet loss

[Neil Horman]

On Wed, Feb 04, 2009 at 07:11:36PM +0100, Eric Dumazet wrote:
> Wesley Chow a écrit :
> >>>>>>
> >>>>>>
> >>>>> Are these quad core systems?  Or dual core w/ hyperthreading?  I
> >>>>> ask because in
> >>>>> your working setup you have 1/2 the number of cpus' and was not
> >>>>> sure if you
> >>>>> removed an entire package of if you just disabled hyperthreading.
> >>>>>
> >>>>>
> >>>>> Neil
> >>>>>
> >>>>>
> >>>> Yeah, these are quad core systems.  The 8 cpu system is a
> >>>> dual-processor  quad-core.  The other is my desktop, single cpu quad
> >>>> core.
> >>>>
> >>>>
> > 
> > 
> > Just to be clear: on the 2 x quad core system, we can run with a 2.6.15
> > kernel and see no packet drops. In fact, we can run with 2.6.19, 2.6.20,
> > and 2.6.21 just fine. 2.6.22 is the first kernel that shows problems.
> > 
> > Kenny posted results from a working setup on a different machine.
> > 
> > What I would really like to know is if whatever changed between 2.6.21
> > and 2.6.22 that broke things is confined just to bnx2. To make this a
> > rigorous test, we would need to use the same machine with a different
> > nic, which we don't have quite yet. An Intel Pro 1000 ethernet card is
> > in the mail as I type this.
> > 
> > I also tried forward porting the bnx2 driver in 2.6.21 to 2.6.22
> > (unsuccessfully), and building the most recent driver from the Broadcom
> > site to Ubuntu Hardy's 2.6.24. The most recent driver with hardy 2.6.24
> > showed similar packet dropping problems. Hm, perhaps I'll try to build
> > the most recent broadcom driver against 2.6.21.
> > 
> 
> Try oprofile session, you shall see a scheduler effect (dont want to call
> this a regression, no need for another flame war).
> 
> also give us "vmstat 1" results  (number of context switches per second)
> 
> On recent kernels, scheduler might be faster than before: You get more wakeups per
> second and more work to do by softirq handler (it does more calls to scheduler,
> thus less cpu cycles available for draining NIC RX queue in time)
> 
> opcontrol --vmlinux=/path/vmlinux --start
> <run benchmark>
> opreport -l /path/vmlinux | head -n 50
> 
> Recent schedulers tend to be optimum for lower latencies (and thus, on
> a high level of wakeups, you get less bandwidth because of sofirq using
> a whole CPU)
> 
> For example, if you have one tread receiving data on 4 or 8 sockets, you'll
> probably notice better throughput (because it will sleep less often)
> 
> Multicast receiving on N sockets, with one thread waiting on each socket
> is basically a way to trigger a scheduler storm. (N wakeups per packet).
> So its more a benchmark to stress scheduler than stressing network stack...
> 
> 
> Maybe its time to change user side, and not try to find an appropriate kernel :)
> 
> If you know you have to receive N frames per 20us units, then its better to :
> Use non blocking sockets, and doing such loop :
> 
> {
> usleep(20); // or try to compensate if this thread is slowed too much by following code
> for (i = 0 ; i < N ; i++) {
> 	while (revfrom(socket[N], ....) != -1)
> 		receive_frame(...);
> 	}
> }
> 
> That way, you are pretty sure network softirq handler wont have to spend time trying
> to wakeup 400.000 time per second one thread. All cpu cycles can be spent in NIC driver
> and network stack.
> 
> Your thread will do 50.000 calls to nanosleep() per second, that is not really expensive,
> then N recvfrom() per iteration. It should work on all past , current and future kernels.
> 
+1 to this idea.  Since the last oprofile traces showed significant variance in
the time spent in schedule(), it might be worthwhile to investigate the affects
of the application behavior on this.  I might also be worth adding a systemtap
probe to sys_recvmsg, to count how many times we receive frames on a working and
non-working system.  If the app is behaving differently on different kernels,
and its affecting the number of times you go to get a frame out of the stack,
that would affect your drop rates, and it would show up in sys_recvmsg

Neil

Re: Multicast packet loss

[Wesley Chow]

>>
>>
>> Maybe its time to change user side, and not try to find an  
>> appropriate kernel :)
>>
>> If you know you have to receive N frames per 20us units, then its  
>> better to :
>> Use non blocking sockets, and doing such loop :
>>
>> {
>> usleep(20); // or try to compensate if this thread is slowed too  
>> much by following code
>> for (i = 0 ; i < N ; i++) {
>> 	while (revfrom(socket[N], ....) != -1)
>> 		receive_frame(...);
>> 	}
>> }
>>
>> That way, you are pretty sure network softirq handler wont have to  
>> spend time trying
>> to wakeup 400.000 time per second one thread. All cpu cycles can be  
>> spent in NIC driver
>> and network stack.
>>
>> Your thread will do 50.000 calls to nanosleep() per second, that is  
>> not really expensive,
>> then N recvfrom() per iteration. It should work on all past ,  
>> current and future kernels.
>>
> +1 to this idea.  Since the last oprofile traces showed significant  
> variance in
> the time spent in schedule(), it might be worthwhile to investigate  
> the affects
> of the application behavior on this.  I might also be worth adding a  
> systemtap
> probe to sys_recvmsg, to count how many times we receive frames on a  
> working and
> non-working system.  If the app is behaving differently on different  
> kernels,
> and its affecting the number of times you go to get a frame out of  
> the stack,
> that would affect your drop rates, and it would show up in sys_recvmsg
>


I did some work to our test program to spin on a non-blocking socket  
and it indeed seems to fix the problem, at least for 2.6.28.1, which  
was a kernel we had problems with. The number of context switches  
drastically drops -- from 200,000+ to less than 50!

I haven't done totally comprehensive tests yet, so I don't want to  
officially state any results. I'm also out today, but Kenny may get a  
chance to play with it. Spinning on the socket is looking like an  
interesting solution, but we're a bit nervous about seeing our  
processes constantly running at 100% CPU. Does C++ have a  
MachineOnFire exception?


Wes

Re: Multicast packet loss

[Neil Horman]

On Wed, Feb 04, 2009 at 11:07:13AM -0500, Kenny Chang wrote:
> Neil Horman wrote:
>> On Tue, Feb 03, 2009 at 10:20:13AM -0500, Kenny Chang wrote:
>>   
>>> Neil Horman wrote:
>>>     
>>>> On Mon, Feb 02, 2009 at 11:48:25AM -0500, Kenny Chang wrote:
>>>>         
>>>>> Neil Horman wrote:
>>>>>             
>>>>>> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>>>>>>                   
>>>>>>> Kenny Chang a écrit :
>>>>>>>                         
>>>>>>>> Ah, sorry, here's the test program attached.
>>>>>>>>
>>>>>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>>>>>> 2.6.29.-rcX.
>>>>>>>>
>>>>>>>> Right now, we are trying to step through the kernel versions until we
>>>>>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>>>>>> and post the result.
>>>>>>>>                               
>>>>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>>>>>
>>>>>>> Problem is multicast handling was not yet updated, but could be :)
>>>>>>>
>>>>>>>
>>>>>>> I was asking you "cat /proc/interrupts" because I believe you might
>>>>>>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>>>>>>
>>>>>>>                         
>>>>>> That would be expected (if irqbalance is running), and desireable, since
>>>>>> spreading high volume interrupts like NICS accross multiple cores (or more
>>>>>> specifically multiple L2 caches), is going increase your cache line miss rate
>>>>>> significantly and decrease rx throughput.
>>>>>>
>>>>>> Although you do have a point here, if the system isn't running irqbalance, and
>>>>>> the NICS irq affinity is spread accross multiple L2 caches, that would be a
>>>>>> point of improvement performance-wise.  
>>>>>>
>>>>>> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
>>>>>> and your stats that I asked about earlier, that would be a big help.
>>>>>>
>>>>>> Regards
>>>>>> Neil
>>>>>>
>>>>>>                   
>>>>> This is for a working setup.
>>>>>
>>>>>             
>>>> Are these quad core systems?  Or dual core w/ hyperthreading?  I ask because in
>>>> your working setup you have 1/2 the number of cpus' and was not sure if you
>>>> removed an entire package of if you just disabled hyperthreading.
>>>>
>>>>
>>>> Neil
>>>>
>>>>         
>>> Yeah, these are quad core systems.  The 8 cpu system is a 
>>> dual-processor  quad-core.  The other is my desktop, single cpu quad 
>>> core.
>>>
>>>     
>> Ok, so their separate systms then.  Did you actually experience drops on the
>> 8-core system since the last reboot?  I ask because even when its distributed
>> across all 8 cores, you only have about 500 total interrupts from the NIC, and
>> if you did get drops, something more than just affinity is wrong.
>>
>> Regardless, spreading interrupts across cores is definately a problem.  As eric
>> says, quad core chips are actually 2x2 cores, so you'll want to either just run
>> irqbalance to assign an apropriate affinity to the NIC, or manually look at each
>> cores physical id and sibling id, to assign affininty to a core or cores that
>> share an L2 cache.  If you need to, as you've found, you may need to disable msi
>> interrupt mode on your bnx2 driver.  That kinda stinks, but bnx2 IIRC isn't
>> multiqueue, so its not like msi provides you any real performance gain.
>>
>> Neil
>>
>>   
> Hi Neil,
>
> Yeah, we've been rebooting this system left and right switch kernels.   
> The results are fairly consistent.  We were able to set the irq  
> affinities, and as Wes had mentioned, what we see is that if we pin the  
> softirq to 1 core, and pin the app to its sibling, we see really good  
> performance, but as we load up other cores, the machine reaches a  
> breaking point where all hell breaks loose and we drop a bunch.  (we  
> hadn't turned off msi btw..)
>
> While we were able to tune and adjust performance like that, in the end,  
> it doesn't really explain the difference between earlier and recent  
> kernels, also it doesn't quite explain the difference between machines.
>
> You mentioned it would be good to see the interrupts for each kernel, in  
> light of the above information, would it still be useful for me to  
> provide that?
>
In light of what you said, I probably don't need to see it no, although if you
go through testing on all these kernels again, I would suggest you take a look
at the /proc/interrupt numbers yourself.  Like you said, its odd that this
behavior changed, since the fast receive path is fairly consistent.  It may be
that te nic driver your using (bnx2 I think?), had a change that either broke
the ability to set affinity for msi interrupts, forcing an irq spread and
killing performance, or perhaps some large inefficiency was introduced either in
the interrupt handler itself, or in the napi poll method of the driver.  Another
good analysis technique would be to grab the latest kernel (which is 'broken' I
think your chart indicated), and the nic driver from the last working kernel.
Merge the driver into the latest kernel and see if the problem persists.  If
not, thats a pretty good indicator that a change in the driver has at least some
culpability.

Best
Neil

> Kenny
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Multicast packet loss

[Eric Dumazet]

Eric Dumazet a écrit :
> Kenny Chang a écrit :
>> Ah, sorry, here's the test program attached.
>>
>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>> 2.6.29.-rcX.
>>
>> Right now, we are trying to step through the kernel versions until we
>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>> and post the result.
> 

I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)

With 8 clients, about 10% packet loss, 

Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000
wakeups per second... But at least UDP receive path seems OK.

Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with
readers...

I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)

I tried playing with ethtool -C|c G|g params...
And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)

I can have 0% packet loss if booting with msi_disable and

echo 1 >/proc/irq/16/smp_affinities

(16 being interrupt of eth0 NIC)

then, a second run gave me errors, about 2%, oh well...


oprofile numbers without playing IRQ affinities:

CPU: Core 2, speed 2999.89 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        symbol name
327928   10.1427  schedule
259625    8.0301  mwait_idle
187337    5.7943  __skb_recv_datagram
109854    3.3977  lock_sock_nested
104713    3.2387  tick_nohz_stop_sched_tick
98831     3.0568  select_nohz_load_balancer
88163     2.7268  skb_release_data
78552     2.4296  update_curr
75241     2.3272  getnstimeofday
71400     2.2084  set_next_entity
67629     2.0917  get_next_timer_interrupt
67375     2.0839  sched_clock_tick
58112     1.7974  enqueue_entity
56462     1.7463  udp_recvmsg
55049     1.7026  copy_to_user
54277     1.6788  sched_clock_cpu
54031     1.6712  __copy_skb_header
51859     1.6040  __slab_free
51786     1.6017  prepare_to_wait_exclusive
51776     1.6014  sock_def_readable
50062     1.5484  try_to_wake_up
42182     1.3047  __switch_to
41631     1.2876  read_tsc
38337     1.1857  tick_nohz_restart_sched_tick
34358     1.0627  cpu_idle
34194     1.0576  native_sched_clock
33812     1.0458  pick_next_task_fair
33685     1.0419  resched_task
33340     1.0312  sys_recvfrom
33287     1.0296  dst_release
32439     1.0033  kmem_cache_free
32131     0.9938  hrtimer_start_range_ns
29807     0.9219  udp_queue_rcv_skb
27815     0.8603  task_rq_lock
26875     0.8312  __update_sched_clock
23912     0.7396  sock_queue_rcv_skb
21583     0.6676  __wake_up_sync
21001     0.6496  effective_load
20531     0.6350  hrtick_start_fair




With IRQ affinities and msi_disable (no packet drops)

CPU: Core 2, speed 3000.13 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        symbol name
79788    10.3815  schedule
69422     9.0328  mwait_idle
44877     5.8391  __skb_recv_datagram
28629     3.7250  tick_nohz_stop_sched_tick
27252     3.5459  select_nohz_load_balancer
24320     3.1644  lock_sock_nested
20833     2.7107  getnstimeofday
20666     2.6889  skb_release_data
18612     2.4217  set_next_entity
17785     2.3141  get_next_timer_interrupt
17691     2.3018  udp_recvmsg
17271     2.2472  sched_clock_tick
16032     2.0860  copy_to_user
14785     1.9237  update_curr
12512     1.6280  prepare_to_wait_exclusive
12498     1.6262  __slab_free
11380     1.4807  read_tsc
11145     1.4501  sched_clock_cpu
10598     1.3789  __switch_to
9588      1.2475  pick_next_task_fair
9480      1.2335  cpu_idle
9218      1.1994  sys_recvfrom
9008      1.1721  tick_nohz_restart_sched_tick
8977      1.1680  dst_release
8930      1.1619  native_sched_clock
8392      1.0919  kmem_cache_free
8124      1.0570  hrtimer_start_range_ns
7274      0.9464  bnx2_interrupt
7175      0.9336  __copy_skb_header
7006      0.9116  try_to_wake_up
6949      0.9042  sock_def_readable
6787      0.8831  enqueue_entity
6772      0.8811  __update_sched_clock
6349      0.8261  finish_task_switch
6164      0.8020  copy_from_user
5096      0.6631  resched_task
5007      0.6515  sysenter_past_esp


I will try to investigate a litle bit more in following days if time permits.

Re: Multicast packet loss

[Neil Horman]

On Sun, Feb 01, 2009 at 01:40:39PM +0100, Eric Dumazet wrote:
> Eric Dumazet a écrit :
> > Kenny Chang a écrit :
> >> Ah, sorry, here's the test program attached.
> >>
> >> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
> >> 2.6.29.-rcX.
> >>
> >> Right now, we are trying to step through the kernel versions until we
> >> see where the performance drops significantly.  We'll try 2.6.29-rc soon
> >> and post the result.
> > 
> 
> I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)
> 
> With 8 clients, about 10% packet loss, 
> 
> Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000
> wakeups per second... But at least UDP receive path seems OK.
> 
> Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with
> readers...
> 
> I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)
> 
> I tried playing with ethtool -C|c G|g params...
> And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)
> 
> I can have 0% packet loss if booting with msi_disable and
> 
> echo 1 >/proc/irq/16/smp_affinities
> 
> (16 being interrupt of eth0 NIC)
> 
> then, a second run gave me errors, about 2%, oh well...
> 
> 
> oprofile numbers without playing IRQ affinities:
> 
> CPU: Core 2, speed 2999.89 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
> samples  %        symbol name
> 327928   10.1427  schedule
> 259625    8.0301  mwait_idle
> 187337    5.7943  __skb_recv_datagram
> 109854    3.3977  lock_sock_nested
> 104713    3.2387  tick_nohz_stop_sched_tick
> 98831     3.0568  select_nohz_load_balancer
> 88163     2.7268  skb_release_data
> 78552     2.4296  update_curr
> 75241     2.3272  getnstimeofday
> 71400     2.2084  set_next_entity
> 67629     2.0917  get_next_timer_interrupt
> 67375     2.0839  sched_clock_tick
> 58112     1.7974  enqueue_entity
> 56462     1.7463  udp_recvmsg
> 55049     1.7026  copy_to_user
> 54277     1.6788  sched_clock_cpu
> 54031     1.6712  __copy_skb_header
> 51859     1.6040  __slab_free
> 51786     1.6017  prepare_to_wait_exclusive
> 51776     1.6014  sock_def_readable
> 50062     1.5484  try_to_wake_up
> 42182     1.3047  __switch_to
> 41631     1.2876  read_tsc
> 38337     1.1857  tick_nohz_restart_sched_tick
> 34358     1.0627  cpu_idle
> 34194     1.0576  native_sched_clock
> 33812     1.0458  pick_next_task_fair
> 33685     1.0419  resched_task
> 33340     1.0312  sys_recvfrom
> 33287     1.0296  dst_release
> 32439     1.0033  kmem_cache_free
> 32131     0.9938  hrtimer_start_range_ns
> 29807     0.9219  udp_queue_rcv_skb
> 27815     0.8603  task_rq_lock
> 26875     0.8312  __update_sched_clock
> 23912     0.7396  sock_queue_rcv_skb
> 21583     0.6676  __wake_up_sync
> 21001     0.6496  effective_load
> 20531     0.6350  hrtick_start_fair
> 
> 
> 
> 
> With IRQ affinities and msi_disable (no packet drops)
> 
> CPU: Core 2, speed 3000.13 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
> samples  %        symbol name
> 79788    10.3815  schedule
> 69422     9.0328  mwait_idle
> 44877     5.8391  __skb_recv_datagram
> 28629     3.7250  tick_nohz_stop_sched_tick
> 27252     3.5459  select_nohz_load_balancer
> 24320     3.1644  lock_sock_nested
> 20833     2.7107  getnstimeofday
> 20666     2.6889  skb_release_data
> 18612     2.4217  set_next_entity
> 17785     2.3141  get_next_timer_interrupt
> 17691     2.3018  udp_recvmsg
> 17271     2.2472  sched_clock_tick
> 16032     2.0860  copy_to_user
> 14785     1.9237  update_curr
> 12512     1.6280  prepare_to_wait_exclusive
> 12498     1.6262  __slab_free
> 11380     1.4807  read_tsc
> 11145     1.4501  sched_clock_cpu
> 10598     1.3789  __switch_to
> 9588      1.2475  pick_next_task_fair
> 9480      1.2335  cpu_idle
> 9218      1.1994  sys_recvfrom
> 9008      1.1721  tick_nohz_restart_sched_tick
> 8977      1.1680  dst_release
> 8930      1.1619  native_sched_clock
> 8392      1.0919  kmem_cache_free
> 8124      1.0570  hrtimer_start_range_ns
> 7274      0.9464  bnx2_interrupt
> 7175      0.9336  __copy_skb_header
> 7006      0.9116  try_to_wake_up
> 6949      0.9042  sock_def_readable
> 6787      0.8831  enqueue_entity
> 6772      0.8811  __update_sched_clock
> 6349      0.8261  finish_task_switch
> 6164      0.8020  copy_from_user
> 5096      0.6631  resched_task
> 5007      0.6515  sysenter_past_esp
> 
> 
> I will try to investigate a litle bit more in following days if time permits.
> 
I'm not 100% versed on this, but IIRC, some hardware simply can't set irq
affinity when operating in msi interrupt mode.  If this is the case with this
particular bnx2 card, then I would expect some packet loss, simply due to the
constant cache misses.  It would be interesting to re-run your oprofile cases,
counting L2 cache hits/misses (if your cpu supports that class of counter) for
both bnx2 running in msi enabled mode and msi disabled mode.  It would also be
interesting to use a different card, that can set irq affinity, and compare loss
with irqbalance on, and irqbalance off with irq afninty set to all cpus.

Neil

> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Multicast packet loss

[Eric Dumazet]

Neil Horman a écrit :
> On Sun, Feb 01, 2009 at 01:40:39PM +0100, Eric Dumazet wrote:
>> Eric Dumazet a écrit :
>>> Kenny Chang a écrit :
>>>> Ah, sorry, here's the test program attached.
>>>>
>>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>>> 2.6.29.-rcX.
>>>>
>>>> Right now, we are trying to step through the kernel versions until we
>>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>>> and post the result.
>> I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)
>>
>> With 8 clients, about 10% packet loss, 
>>
>> Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000
>> wakeups per second... But at least UDP receive path seems OK.
>>
>> Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with
>> readers...
>>
>> I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)
>>
>> I tried playing with ethtool -C|c G|g params...
>> And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)
>>
>> I can have 0% packet loss if booting with msi_disable and
>>
>> echo 1 >/proc/irq/16/smp_affinities
>>
>> (16 being interrupt of eth0 NIC)
>>
>> then, a second run gave me errors, about 2%, oh well...
>>
>>
>> oprofile numbers without playing IRQ affinities:
>>
>> CPU: Core 2, speed 2999.89 MHz (estimated)
>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
>> samples  %        symbol name
>> 327928   10.1427  schedule
>> 259625    8.0301  mwait_idle
>> 187337    5.7943  __skb_recv_datagram
>> 109854    3.3977  lock_sock_nested
>> 104713    3.2387  tick_nohz_stop_sched_tick
>> 98831     3.0568  select_nohz_load_balancer
>> 88163     2.7268  skb_release_data
>> 78552     2.4296  update_curr
>> 75241     2.3272  getnstimeofday
>> 71400     2.2084  set_next_entity
>> 67629     2.0917  get_next_timer_interrupt
>> 67375     2.0839  sched_clock_tick
>> 58112     1.7974  enqueue_entity
>> 56462     1.7463  udp_recvmsg
>> 55049     1.7026  copy_to_user
>> 54277     1.6788  sched_clock_cpu
>> 54031     1.6712  __copy_skb_header
>> 51859     1.6040  __slab_free
>> 51786     1.6017  prepare_to_wait_exclusive
>> 51776     1.6014  sock_def_readable
>> 50062     1.5484  try_to_wake_up
>> 42182     1.3047  __switch_to
>> 41631     1.2876  read_tsc
>> 38337     1.1857  tick_nohz_restart_sched_tick
>> 34358     1.0627  cpu_idle
>> 34194     1.0576  native_sched_clock
>> 33812     1.0458  pick_next_task_fair
>> 33685     1.0419  resched_task
>> 33340     1.0312  sys_recvfrom
>> 33287     1.0296  dst_release
>> 32439     1.0033  kmem_cache_free
>> 32131     0.9938  hrtimer_start_range_ns
>> 29807     0.9219  udp_queue_rcv_skb
>> 27815     0.8603  task_rq_lock
>> 26875     0.8312  __update_sched_clock
>> 23912     0.7396  sock_queue_rcv_skb
>> 21583     0.6676  __wake_up_sync
>> 21001     0.6496  effective_load
>> 20531     0.6350  hrtick_start_fair
>>
>>
>>
>>
>> With IRQ affinities and msi_disable (no packet drops)
>>
>> CPU: Core 2, speed 3000.13 MHz (estimated)
>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
>> samples  %        symbol name
>> 79788    10.3815  schedule
>> 69422     9.0328  mwait_idle
>> 44877     5.8391  __skb_recv_datagram
>> 28629     3.7250  tick_nohz_stop_sched_tick
>> 27252     3.5459  select_nohz_load_balancer
>> 24320     3.1644  lock_sock_nested
>> 20833     2.7107  getnstimeofday
>> 20666     2.6889  skb_release_data
>> 18612     2.4217  set_next_entity
>> 17785     2.3141  get_next_timer_interrupt
>> 17691     2.3018  udp_recvmsg
>> 17271     2.2472  sched_clock_tick
>> 16032     2.0860  copy_to_user
>> 14785     1.9237  update_curr
>> 12512     1.6280  prepare_to_wait_exclusive
>> 12498     1.6262  __slab_free
>> 11380     1.4807  read_tsc
>> 11145     1.4501  sched_clock_cpu
>> 10598     1.3789  __switch_to
>> 9588      1.2475  pick_next_task_fair
>> 9480      1.2335  cpu_idle
>> 9218      1.1994  sys_recvfrom
>> 9008      1.1721  tick_nohz_restart_sched_tick
>> 8977      1.1680  dst_release
>> 8930      1.1619  native_sched_clock
>> 8392      1.0919  kmem_cache_free
>> 8124      1.0570  hrtimer_start_range_ns
>> 7274      0.9464  bnx2_interrupt
>> 7175      0.9336  __copy_skb_header
>> 7006      0.9116  try_to_wake_up
>> 6949      0.9042  sock_def_readable
>> 6787      0.8831  enqueue_entity
>> 6772      0.8811  __update_sched_clock
>> 6349      0.8261  finish_task_switch
>> 6164      0.8020  copy_from_user
>> 5096      0.6631  resched_task
>> 5007      0.6515  sysenter_past_esp
>>
>>
>> I will try to investigate a litle bit more in following days if time permits.
>>
> I'm not 100% versed on this, but IIRC, some hardware simply can't set irq
> affinity when operating in msi interrupt mode.  If this is the case with this
> particular bnx2 card, then I would expect some packet loss, simply due to the
> constant cache misses.  It would be interesting to re-run your oprofile cases,
> counting L2 cache hits/misses (if your cpu supports that class of counter) for
> both bnx2 running in msi enabled mode and msi disabled mode.  It would also be
> interesting to use a different card, that can set irq affinity, and compare loss
> with irqbalance on, and irqbalance off with irq afninty set to all cpus.

booted with msi_disable=1, IRQ of eth0 handled by CPU0 only, so that
oprofile results sorted on CPU0 numbers.

We can see scheduler has hard time to cope with this workload with more of two CPUS

OK up to 30.000 (* 8 sockets) packets per second. 

CPU0 is 100% handling softirq (ksoftirqd/0)


CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
6152     12.5595  3         0.0098  3         0.0090  5         0.0156  1         0.0582  0              0  2         0.0065  3         0.0169  enqueue_entity
4453      9.0909  2         0.0065  3         0.0090  4         0.0125  5         0.2910  0              0  1         0.0033  2         0.0113  try_to_wake_up
3837      7.8333  3         0.0098  8         0.0241  0              0  0              0  0              0  0              0  0              0  sock_def_readable
3694      7.5414  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __copy_skb_header
2320      4.7363  1         0.0033  2         0.0060  2         0.0062  1         0.0582  1         0.0028  2         0.0065  0              0  resched_task
1818      3.7115  6         0.0196  32        0.0962  0              0  0              0  0              0  0              0  0              0  sock_queue_rcv_skb
1776      3.6257  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_queue_rcv_skb
1677      3.4236  0              0  1         0.0030  0              0  1         0.0582  1         0.0028  0              0  0              0  __slab_alloc
1658      3.3848  260       0.8496  303       0.9109  289       0.9021  24        1.3970  418       1.1730  326       1.0626  173       0.9733  sched_clock_cpu
1614      3.2950  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_sync
1600      3.2664  0              0  1         0.0030  0              0  1         0.0582  1         0.0028  0              0  0              0  select_task_rq_fair
1569      3.2032  1299      4.2447  1530      4.5996  1271      3.9675  6         0.3492  1677      4.7062  1275      4.1559  759       4.2703  update_curr
1532      3.1276  4         0.0131  4         0.0120  0              0  2         0.1164  1         0.0028  1         0.0033  1         0.0056  task_rq_lock
1325      2.7050  1         0.0033  7         0.0210  0              0  0              0  0              0  0              0  0              0  skb_queue_tail
1273      2.5989  1         0.0033  1         0.0030  1         0.0031  0              0  0              0  1         0.0033  0              0  enqueue_task_fair
1227      2.5050  0              0  0              0  0              0  0              0  0              0  0              0  0              0  effective_load
1071      2.1865  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __udp4_lib_rcv
1009      2.0599  0              0  0              0  0              0  2         0.1164  1         0.0028  0              0  0              0  activate_task
940       1.9190  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_common
930       1.8986  0              0  2         0.0060  0              0  0              0  0              0  0              0  1         0.0056  account_scheduler_latency
859       1.7537  0              0  0              0  0              0  0              0  0              0  0              0  1         0.0056  __skb_clone
609       1.2433  0              0  0              0  0              0  1         0.0582  1         0.0028  1         0.0033  0              0  enqueue_task
588       1.2004  3         0.0098  2         0.0060  5         0.0156  8         0.4657  2         0.0056  3         0.0098  2         0.0113  kmem_cache_alloc
477       0.9738  307       1.0032  322       0.9680  358       1.1175  27        1.5716  338       0.9485  315       1.0268  203       1.1421  native_sched_clock
441       0.9003  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_clone
408       0.8329  0              0  0              0  0              0  0              0  0              0  0              0  0              0  ip_route_input
375       0.7656  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_poll_work
366       0.7472  248       0.8104  269       0.8087  293       0.9146  22        1.2806  289       0.8110  332       1.0822  157       0.8833  __update_sched_clock
327       0.6676  1         0.0033  0              0  0              0  0              0  0              0  2         0.0065  1         0.0056  place_entity
265       0.5410  54        0.1765  62        0.1864  39        0.1217  3         0.1746  84        0.2357  61        0.1988  12        0.0675  rb_insert_color
194       0.3961  2662      8.6985  3291      9.8936  3231     10.0858  372      21.6531  2994      8.4021  3299     10.7533  1719      9.6714  mwait_idle



This problem completely disapears if I launch all clients bounded to CPU1
(and NIC irq still on CPU0)

taskset -p2 ./mcasttest.sh

(No packet loss, while CPU1 has 0% idle time...)
We have less context switches. Once awakened, a task can read several packets in its socket.

CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
25316    13.6664  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __copy_skb_header
14584     7.8729  13        0.0083  2         0.2670  0              0  0              0  1         0.0218  1         0.1577  1         0.2604  task_rq_lock
11624     6.2750  10657     6.7650  3         0.4005  2         0.5013  5         0.7278  36        0.7862  2         0.3155  1         0.2604  update_curr
10038     5.4188  318       0.2019  0              0  0              0  0              0  0              0  0              0  0              0  sock_def_readable
10021     5.4097  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_interrupt
7777      4.1983  11        0.0070  1         0.1335  2         0.5013  2         0.2911  8         0.1747  3         0.4732  1         0.2604  try_to_wake_up
6559      3.5408  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_queue_rcv_skb
6389      3.4490  257       0.1631  0              0  0              0  0              0  0              0  0              0  0              0  sock_queue_rcv_skb
6305      3.4036  6         0.0038  0              0  0              0  0              0  0              0  0              0  1         0.2604  __slab_alloc
5661      3.0560  44        0.0279  2         0.2670  1         0.2506  5         0.7278  0              0  1         0.1577  0              0  kmem_cache_alloc
5529      2.9847  5         0.0032  1         0.1335  4         1.0025  0              0  14        0.3057  3         0.4732  1         0.2604  enqueue_entity
4706      2.5404  64        0.0406  0              0  0              0  0              0  0              0  0              0  0              0  skb_queue_tail
4390      2.3699  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __udp4_lib_rcv
4043      2.1825  0              0  0              0  0              0  0              0  0              0  0              0  0              0  uhci_irq
3897      2.1037  0              0  0              0  0              0  0              0  0              0  0              0  0              0  bnx2_poll_work
3556      1.9196  0              0  296      39.5194  261      65.4135  258      37.5546  650      14.1952  263      41.4826  257      66.9271  mwait_idle
3449      1.8619  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __skb_clone
3348      1.8074  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_clone
3243      1.7507  1653      1.0493  7         0.9346  3         0.7519  2         0.2911  63        1.3758  1         0.1577  1         0.2604  sched_clock_cpu
3068      1.6562  1        6.3e-04  0              0  0              0  0              0  0              0  0              0  0              0  __wake_up_sync
2923      1.5779  1        6.3e-04  0              0  0              0  0              0  0              0  0              0  0              0  check_preempt_wakeup
2588      1.3971  1        6.3e-04  4         0.5340  2         0.5013  1         0.1456  1         0.0218  2         0.3155  0              0  enqueue_task_fair
2399      1.2951  0              0  0              0  0              0  0              0  0              0  0              0  0              0  ip_route_input
1986      1.0721  5         0.0032  0              0  0              0  0              0  1         0.0218  0              0  0              0  __wake_up_common
1777      0.9593  132       0.0838  0              0  0              0  3         0.4367  8         0.1747  1         0.1577  1         0.2604  rb_insert_color
1754      0.9469  34        0.0216  0              0  0              0  0              0  0              0  0              0  0              0  __sk_mem_schedule
1550      0.8367  0              0  0              0  0              0  0              0  0              0  0              0  0              0  irq_entries_start
1527      0.8243  0              0  13        1.7356  2         0.5013  0              0  104       2.2712  6         0.9464  0              0  get_next_timer_interrupt
1398      0.7547  6         0.0038  0              0  0              0  1         0.1456  1         0.0218  0              0  0              0  select_task_rq_fair
1159      0.6257  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __alloc_skb


Exchanging CPU0/CPU1 to get oprofile number sorted on the CPU used by user application :

CPU: Core 2, speed 3000.31 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        samples  %        symbol name
6040     10.1815  8         0.0134  4         0.3208  6         1.3699  1         0.1580  3         1.0909  5         0.6039  0              0  schedule
6014     10.1377  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __skb_recv_datagram
4730      7.9733  36        0.0604  0              0  0              0  0              0  0              0  0              0  0              0  skb_release_data
4014      6.7663  2         0.0034  33        2.6464  1         0.2283  6         0.9479  0              0  1         0.1208  0              0  copy_to_user
3732      6.2910  3708      6.2167  4         0.3208  1         0.2283  3         0.4739  3         1.0909  2         0.2415  0              0  update_curr
3446      5.8089  0              0  0              0  0              0  0              0  0              0  0              0  0              0  lock_sock_nested
2430      4.0962  0              0  0              0  0              0  0              0  0              0  0              0  0              0  udp_recvmsg
2028      3.4186  73        0.1224  0              0  0              0  2         0.3160  0              0  1         0.1208  1         0.3861  __slab_free
1898      3.1994  43        0.0721  0              0  0              0  0              0  0              0  0              0  0              0  dst_release
1645      2.7730  0              0  0              0  0              0  0              0  0              0  0              0  0              0  memcpy_toiovec
1635      2.7561  0              0  0              0  0              0  1         0.1580  0              0  0              0  0              0  copy_from_user
1407      2.3718  2         0.0034  0              0  2         0.4566  6         0.9479  0              0  5         0.6039  1         0.3861  sysenter_past_esp
1389      2.3414  58        0.0972  3         0.2406  3         0.6849  4         0.6319  2         0.7273  3         0.3623  0              0  kmem_cache_free
1135      1.9133  1         0.0017  0              0  0              0  0              0  0              0  0              0  0              0  release_sock
1069      1.8020  0              0  0              0  0              0  0              0  0              0  0              0  0              0  prepare_to_wait_exclusive
1031      1.7379  3         0.0050  0              0  0              0  0              0  0              0  1         0.1208  0              0  put_prev_task_fair
1007      1.6975  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sock_rfree
926       1.5609  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sys_recvfrom
838       1.4126  0              0  0              0  0              0  0              0  0              0  0              0  0              0  skb_copy_datagram_iovec
697       1.1749  27        0.0453  1         0.0802  0              0  0              0  1         0.3636  0              0  0              0  add_partial
697       1.1749  0              0  0              0  0              0  0              0  0              0  1         0.1208  0              0  dequeue_task
604       1.0182  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sock_recvmsg
582       0.9811  933       1.5642  17        1.3633  1         0.2283  2         0.3160  1         0.3636  6         0.7246  0              0  sched_clock_cpu
525       0.8850  0              0  0              0  0              0  0              0  0              0  0              0  0              0  __sk_mem_reclaim
512       0.8631  0              0  0              0  1         0.2283  0              0  0              0  1         0.1208  1         0.3861  __switch_to
489       0.8243  0              0  0              0  0              0  0              0  0              0  0              0  0              0  fget_light
450       0.7586  1         0.0017  2         0.1604  1         0.2283  0              0  0              0  4         0.4831  0              0  rb_erase
409       0.6894  95        0.1593  0              0  0              0  0              0  0              0  0              0  0              0  wakeup_preempt_entity
360       0.6068  0              0  0              0  0              0  0              0  0              0  0              0  0              0  move_addr_to_user
348       0.5866  0              0  0              0  0              0  0              0  0              0  0              0  0              0  sys_socketcall
347       0.5849  1         0.0017  0              0  0              0  0              0  0              0  0              0  1         0.3861  set_next_entity

Re: Multicast packet loss

[Neil Horman]

On Mon, Feb 02, 2009 at 05:57:24PM +0100, Eric Dumazet wrote:
> Neil Horman a écrit :
> > On Sun, Feb 01, 2009 at 01:40:39PM +0100, Eric Dumazet wrote:
> >> Eric Dumazet a écrit :
> >>> Kenny Chang a écrit :
> >>>> Ah, sorry, here's the test program attached.
> >>>>
> >>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
> >>>> 2.6.29.-rcX.
> >>>>
> >>>> Right now, we are trying to step through the kernel versions until we
> >>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
> >>>> and post the result.
> >> I tried your program on my dev machines and 2.6.29 (each machine : two quad core cpus, 32bits kernel)
> >>
> >> With 8 clients, about 10% packet loss, 
> >>
> >> Might be a scheduling problem, not sure... 50.000 packets per second, x 8 cpus = 400.000
> >> wakeups per second... But at least UDP receive path seems OK.
> >>
> >> Thing is the receiver (softirq that queues the packet) seems to fight on socket lock with
> >> readers...
> >>
> >> I tried to setup IRQ affinities, but it doesnt work any more on bnx2 (unless using msi_disable=1)
> >>
> >> I tried playing with ethtool -C|c G|g params...
> >> And /proc/net/core/rmem_max (and setsockopt(RCVBUF) to set bigger receive buffers in your program)
> >>
> >> I can have 0% packet loss if booting with msi_disable and
> >>
> >> echo 1 >/proc/irq/16/smp_affinities
> >>
> >> (16 being interrupt of eth0 NIC)
> >>
> >> then, a second run gave me errors, about 2%, oh well...
> >>
> >>
> >> oprofile numbers without playing IRQ affinities:
> >>
> >> CPU: Core 2, speed 2999.89 MHz (estimated)
> >> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
> >> samples  %        symbol name
> >> 327928   10.1427  schedule
> >> 259625    8.0301  mwait_idle
> >> 187337    5.7943  __skb_recv_datagram
> >> 109854    3.3977  lock_sock_nested
> >> 104713    3.2387  tick_nohz_stop_sched_tick
> >> 98831     3.0568  select_nohz_load_balancer
> >> 88163     2.7268  skb_release_data
> >> 78552     2.4296  update_curr
> >> 75241     2.3272  getnstimeofday
> >> 71400     2.2084  set_next_entity
> >> 67629     2.0917  get_next_timer_interrupt
> >> 67375     2.0839  sched_clock_tick
> >> 58112     1.7974  enqueue_entity
> >> 56462     1.7463  udp_recvmsg
> >> 55049     1.7026  copy_to_user
> >> 54277     1.6788  sched_clock_cpu
> >> 54031     1.6712  __copy_skb_header
> >> 51859     1.6040  __slab_free
> >> 51786     1.6017  prepare_to_wait_exclusive
> >> 51776     1.6014  sock_def_readable
> >> 50062     1.5484  try_to_wake_up
> >> 42182     1.3047  __switch_to
> >> 41631     1.2876  read_tsc
> >> 38337     1.1857  tick_nohz_restart_sched_tick
> >> 34358     1.0627  cpu_idle
> >> 34194     1.0576  native_sched_clock
> >> 33812     1.0458  pick_next_task_fair
> >> 33685     1.0419  resched_task
> >> 33340     1.0312  sys_recvfrom
> >> 33287     1.0296  dst_release
> >> 32439     1.0033  kmem_cache_free
> >> 32131     0.9938  hrtimer_start_range_ns
> >> 29807     0.9219  udp_queue_rcv_skb
> >> 27815     0.8603  task_rq_lock
> >> 26875     0.8312  __update_sched_clock
> >> 23912     0.7396  sock_queue_rcv_skb
> >> 21583     0.6676  __wake_up_sync
> >> 21001     0.6496  effective_load
> >> 20531     0.6350  hrtick_start_fair
> >>
> >>
> >>
> >>
> >> With IRQ affinities and msi_disable (no packet drops)
> >>
> >> CPU: Core 2, speed 3000.13 MHz (estimated)
> >> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
> >> samples  %        symbol name
> >> 79788    10.3815  schedule
> >> 69422     9.0328  mwait_idle
> >> 44877     5.8391  __skb_recv_datagram
> >> 28629     3.7250  tick_nohz_stop_sched_tick
> >> 27252     3.5459  select_nohz_load_balancer
> >> 24320     3.1644  lock_sock_nested
> >> 20833     2.7107  getnstimeofday
> >> 20666     2.6889  skb_release_data
> >> 18612     2.4217  set_next_entity
> >> 17785     2.3141  get_next_timer_interrupt
> >> 17691     2.3018  udp_recvmsg
> >> 17271     2.2472  sched_clock_tick
> >> 16032     2.0860  copy_to_user
> >> 14785     1.9237  update_curr
> >> 12512     1.6280  prepare_to_wait_exclusive
> >> 12498     1.6262  __slab_free
> >> 11380     1.4807  read_tsc
> >> 11145     1.4501  sched_clock_cpu
> >> 10598     1.3789  __switch_to
> >> 9588      1.2475  pick_next_task_fair
> >> 9480      1.2335  cpu_idle
> >> 9218      1.1994  sys_recvfrom
> >> 9008      1.1721  tick_nohz_restart_sched_tick
> >> 8977      1.1680  dst_release
> >> 8930      1.1619  native_sched_clock
> >> 8392      1.0919  kmem_cache_free
> >> 8124      1.0570  hrtimer_start_range_ns
> >> 7274      0.9464  bnx2_interrupt
> >> 7175      0.9336  __copy_skb_header
> >> 7006      0.9116  try_to_wake_up
> >> 6949      0.9042  sock_def_readable
> >> 6787      0.8831  enqueue_entity
> >> 6772      0.8811  __update_sched_clock
> >> 6349      0.8261  finish_task_switch
> >> 6164      0.8020  copy_from_user
> >> 5096      0.6631  resched_task
> >> 5007      0.6515  sysenter_past_esp
> >>
> >>
> >> I will try to investigate a litle bit more in following days if time permits.
> >>
> > I'm not 100% versed on this, but IIRC, some hardware simply can't set irq
> > affinity when operating in msi interrupt mode.  If this is the case with this
> > particular bnx2 card, then I would expect some packet loss, simply due to the
> > constant cache misses.  It would be interesting to re-run your oprofile cases,
> > counting L2 cache hits/misses (if your cpu supports that class of counter) for
> > both bnx2 running in msi enabled mode and msi disabled mode.  It would also be
> > interesting to use a different card, that can set irq affinity, and compare loss
> > with irqbalance on, and irqbalance off with irq afninty set to all cpus.
> 
> booted with msi_disable=1, IRQ of eth0 handled by CPU0 only, so that
> oprofile results sorted on CPU0 numbers.
> 
> We can see scheduler has hard time to cope with this workload with more of two CPUS
> 
> OK up to 30.000 (* 8 sockets) packets per second. 
> 
> CPU0 is 100% handling softirq (ksoftirqd/0)
> 

This explains alot.  if the application is scheduled to run on the same cpu that
has the irq for the NIC bound to it, you get a perf boost by not having to warm
up two caches (1 for the app cpu and one for the irq & softirq work), but you
loose it and then some fighting for cpu time.  If both the app and the irq are
on the same cpu, and we spend so much time in softirq context, we will
eventually overflow higher up the network stack, as the application doesn't have
enough time to dequeue frames.

It may also speak to the need to make the bnx2 napi routine more efficient :)

Neil

Re: Multicast packet loss

[Wes Chow]

(I'm Kenny's colleague, and I've been doing the kernel builds)

First I'd like to note that there were a lot of bnx2 NAPI changes between 
2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
whereas loss in 2.6.22 is significant.

Second, some CPU affinity info: if I do like Eric and pin all of the
apps onto a single CPU, I see no packet loss. Also, I do *not* see
ksoftirqd show up on top at all!

If I pin half the processes on one CPU and the other half on another CPU, one 
ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
in that case is significant (25%).

Now, the strange case: if I pin 3 processes to one CPU and 1 process to 
another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
of the apps takes significantly less CPU than the others, and all apps lose the
*exact same number of packets*. In all other situations where we see packet
loss, the actual number lost per application instance appears random.

We're about to plug in an Intel ethernet card into this machine to collect more 
rigorous testing data. Please note, though, that we have seen packet loss with
a tg3 chipset as well. For now, though, I'm assuming that this is purely a bnx2
problem.

If I understand correctly, when the nic signals a hardware interrupt, the 
kernel grabs it and defers the meaty work to the softirq handler -- how does it
decide which ksoftirqd gets the interrupts? Is this something determined by how
the driver implements the NAPI?


Wes

Re: Multicast packet loss

[Eric Dumazet]

Wes Chow a écrit :
> 
> (I'm Kenny's colleague, and I've been doing the kernel builds)
> 
> First I'd like to note that there were a lot of bnx2 NAPI changes between 
> 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
> whereas loss in 2.6.22 is significant.
> 
> Second, some CPU affinity info: if I do like Eric and pin all of the
> apps onto a single CPU, I see no packet loss. Also, I do *not* see
> ksoftirqd show up on top at all!
> 
> If I pin half the processes on one CPU and the other half on another CPU, one 
> ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
> in that case is significant (25%).
> 
> Now, the strange case: if I pin 3 processes to one CPU and 1 process to 
> another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
> of the apps takes significantly less CPU than the others, and all apps lose the
> *exact same number of packets*. In all other situations where we see packet
> loss, the actual number lost per application instance appears random.

You see same number of packet lost because they are lost at NIC level

(check ifconfig eth0 for droped packets)

if softirq is too busy to process packets, we are not able to get them
from hardware in time.

> 
> We're about to plug in an Intel ethernet card into this machine to collect more 
> rigorous testing data. Please note, though, that we have seen packet loss with
> a tg3 chipset as well. For now, though, I'm assuming that this is purely a bnx2
> problem.
> 
> If I understand correctly, when the nic signals a hardware interrupt, the 
> kernel grabs it and defers the meaty work to the softirq handler -- how does it
> decide which ksoftirqd gets the interrupts? Is this something determined by how
> the driver implements the NAPI?


Normaly, softirq runs on same cpu (the one handling hard irq)

Re: Multicast packet loss

[Wes Chow]

Eric Dumazet wrote:
> Wes Chow a écrit :
>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>
>> First I'd like to note that there were a lot of bnx2 NAPI changes between 
>> 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts of packet loss,
>> whereas loss in 2.6.22 is significant.
>>
>> Second, some CPU affinity info: if I do like Eric and pin all of the
>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>> ksoftirqd show up on top at all!
>>
>> If I pin half the processes on one CPU and the other half on another CPU, one 
>> ksoftirqd processes shows up in top and completely pegs one CPU. My packet loss
>> in that case is significant (25%).
>>
>> Now, the strange case: if I pin 3 processes to one CPU and 1 process to 
>> another, I get about 25% packet loss and ksoftirqd pins one CPU. However, one
>> of the apps takes significantly less CPU than the others, and all apps lose the
>> *exact same number of packets*. In all other situations where we see packet
>> loss, the actual number lost per application instance appears random.
> 
> You see same number of packet lost because they are lost at NIC level

Understood.

I have a new observation: if I pin processes to just CPUs 0 and 1, I see 
no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and 
3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any 
other combination appears to produce loss (though I have not tried all 
28 combinations, this seems to be the case).

At first I thought maybe it had to do with processes pinned to the same 
CPU, but different cores. The machine is a dual quad core, which means 
that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3 
produce packet loss.

I've also noticed that it does not matter which of the working pairs I 
pin to. For example, pinning 5 processes in any combination on either 
0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no 
packet loss.

The failures are also sudden. In all of the working cases mentioned 
above, I don't see ksoftirqd on top at all. But when I run 6 processes 
on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of 
packets.

> 
> Normaly, softirq runs on same cpu (the one handling hard irq)

What determines which CPU the hard irq occurs on?


Wes

Re: Multicast packet loss

[Eric Dumazet]

Wes Chow a écrit :
> 
> 
> Eric Dumazet wrote:
>> Wes Chow a écrit :
>>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>>
>>> First I'd like to note that there were a lot of bnx2 NAPI changes
>>> between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts
>>> of packet loss,
>>> whereas loss in 2.6.22 is significant.
>>>
>>> Second, some CPU affinity info: if I do like Eric and pin all of the
>>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>>> ksoftirqd show up on top at all!
>>>
>>> If I pin half the processes on one CPU and the other half on another
>>> CPU, one ksoftirqd processes shows up in top and completely pegs one
>>> CPU. My packet loss
>>> in that case is significant (25%).
>>>
>>> Now, the strange case: if I pin 3 processes to one CPU and 1 process
>>> to another, I get about 25% packet loss and ksoftirqd pins one CPU.
>>> However, one
>>> of the apps takes significantly less CPU than the others, and all
>>> apps lose the
>>> *exact same number of packets*. In all other situations where we see
>>> packet
>>> loss, the actual number lost per application instance appears random.
>>
>> You see same number of packet lost because they are lost at NIC level
> 
> Understood.
> 
> I have a new observation: if I pin processes to just CPUs 0 and 1, I see
> no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and
> 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any
> other combination appears to produce loss (though I have not tried all
> 28 combinations, this seems to be the case).
> 
> At first I thought maybe it had to do with processes pinned to the same
> CPU, but different cores. The machine is a dual quad core, which means
> that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3
> produce packet loss.

a quad core is really a 2 x 2 core

L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3 

You are at the limit of the machine with such workload, so as soon as your
CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.


> 
> I've also noticed that it does not matter which of the working pairs I
> pin to. For example, pinning 5 processes in any combination on either
> 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no
> packet loss.
> 
> The failures are also sudden. In all of the working cases mentioned
> above, I don't see ksoftirqd on top at all. But when I run 6 processes
> on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of
> packets.
> 
>>
>> Normaly, softirq runs on same cpu (the one handling hard irq)
> 
> What determines which CPU the hard irq occurs on?
> 

Check /proc/irq/{irqnumber}/smp_affinity

If you want IRQ16 only served by CPU0 :

echo 1 >/proc/irq/16/smp_affinity

Re: Multicast packet loss

[Kenny Chang]

[-- Attachment #1: Type: text/plain, Size: 4951 bytes --]

Eric Dumazet wrote:
> Wes Chow a écrit :
>   
>> Eric Dumazet wrote:
>>     
>>> Wes Chow a écrit :
>>>       
>>>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>>>
>>>> First I'd like to note that there were a lot of bnx2 NAPI changes
>>>> between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts
>>>> of packet loss,
>>>> whereas loss in 2.6.22 is significant.
>>>>
>>>> Second, some CPU affinity info: if I do like Eric and pin all of the
>>>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>>>> ksoftirqd show up on top at all!
>>>>
>>>> If I pin half the processes on one CPU and the other half on another
>>>> CPU, one ksoftirqd processes shows up in top and completely pegs one
>>>> CPU. My packet loss
>>>> in that case is significant (25%).
>>>>
>>>> Now, the strange case: if I pin 3 processes to one CPU and 1 process
>>>> to another, I get about 25% packet loss and ksoftirqd pins one CPU.
>>>> However, one
>>>> of the apps takes significantly less CPU than the others, and all
>>>> apps lose the
>>>> *exact same number of packets*. In all other situations where we see
>>>> packet
>>>> loss, the actual number lost per application instance appears random.
>>>>         
>>> You see same number of packet lost because they are lost at NIC level
>>>       
>> Understood.
>>
>> I have a new observation: if I pin processes to just CPUs 0 and 1, I see
>> no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and
>> 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any
>> other combination appears to produce loss (though I have not tried all
>> 28 combinations, this seems to be the case).
>>
>> At first I thought maybe it had to do with processes pinned to the same
>> CPU, but different cores. The machine is a dual quad core, which means
>> that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3
>> produce packet loss.
>>     
>
> a quad core is really a 2 x 2 core
>
> L2 cache is splited on two blocks, one block used by CPU0/1, other by CPU2/3 
>
> You are at the limit of the machine with such workload, so as soon as your
> CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.
>
>
>   
>> I've also noticed that it does not matter which of the working pairs I
>> pin to. For example, pinning 5 processes in any combination on either
>> 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no
>> packet loss.
>>
>> The failures are also sudden. In all of the working cases mentioned
>> above, I don't see ksoftirqd on top at all. But when I run 6 processes
>> on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of
>> packets.
>>
>>     
>>> Normaly, softirq runs on same cpu (the one handling hard irq)
>>>       
>> What determines which CPU the hard irq occurs on?
>>
>>     
>
> Check /proc/irq/{irqnumber}/smp_affinity
>
> If you want IRQ16 only served by CPU0 :
>
> echo 1 >/proc/irq/16/smp_affinity
>
>   
Hi everyone,

First, thanks for all the effort so far, I think we've learned so much 
more about the problem in the last couple of days than we had previously 
in a month.

Just to summarize where we are:

* pinning processes to specific cores/CPUs alleviate the problem
* issues exist from 2.6.22 up to 2.6.29-rc3
* issue does not appear to be isolated to 64-bit, 32-bits have problems 
too.
* I'm attaching an updated test program with the PR_SET_TIMERSTACK call 
added.
* on troubled machines, we are seeing high number of context switches 
and interrupts.
* we've ordered an Intel card to try in our machine to see if we can 
circumvent the issue with a different driver.

Kernel Version         Has Problem?     Notes
----------             ----------       ----------
2.6.15.x                N    
2.6.16.x                -
2.6.17.x                -               Doesn't build on Hardy
2.6.18.x                -               Doesn't boot (kernel panic)
2.6.19.7                N               ksoftirqd is up there, but not 
pegging a CPU.
                                        Takes roughly same amount of CPU 
as the other
                                        processes, all of which are from 
20-40%
2.6.20.21               N
2.6.21.7                N               sort of lopsided load, but no 
load from
                                        ksoftirqd -- strange
2.6.22.19               Y               First broken kernel
2.6.23.x                -
2.6.24-19               Y               (from Hardy)
2.6.25.x                -
2.6.26.x                -
2.6.27.x                Y               (from Intrepid)
2.6.28.1                Y
2.6.29-rc               Y


Correct me if I'm wrong, from what we've seen, it looks like its 
pointing to some inefficiency in the softirq handling.  The question is 
whether it's something in the driver or the kernel.  If we can isolate 
that, maybe we can take some action to have it fixed.

Kenny

[-- Attachment #2: mcasttest.c --]
[-- Type: text/x-csrc, Size: 3392 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/epoll.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>
#include <unistd.h>

#ifndef PR_SET_TIMERSLACK
#define PR_SET_TIMERSLACK 29
#endif

const char *g_mcastaddr = "239.100.0.99";
int g_port = 10100;

void error(const char *s)
{
    fprintf(stderr, "%s\n", s);
    exit(1);
}

void check(int v)
{
    int myerr = errno;
    char *myerrstr = strerror(myerr);
    if(!v)
        error("bad return code");
}

int main(int argc, char **argv)
{
    if(argc != 2)
        error("usage: mcasttest (server|client)");
    if(strcmp(argv[1], "client") == 0)
    {
        /*
         * Client program: subscribes to a multicast group, receives messages
         * and prints a count of messages received once it's done.
         */
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        check(s > 0);
        int val = 1;
        check(setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) == 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        addr.sin_addr.s_addr = htonl(INADDR_ANY);
        check(bind(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        struct ip_mreqn mreq;
        memset(&mreq, 0, sizeof(mreq));
        check(inet_pton(AF_INET, g_mcastaddr, &mreq.imr_multiaddr));
        mreq.imr_address.s_addr = htonl(INADDR_ANY);
        mreq.imr_ifindex = 0;
        check(setsockopt(s, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) == 0);

        int bufSz;
        socklen_t len = sizeof(bufSz);
        getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char*)(&bufSz), &len);
        printf("bufsz: %d\n", bufSz);

        int npackets = 0;
        char buf[1000];
        memset(buf, 0, sizeof(buf));
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            check(recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen) == 100);
            ++npackets;
            if(buf[0] == 1) // exit message
                break;
        }
        printf("received %d packets\n", npackets);
    }
    else if(strcmp(argv[1], "server") == 0)
    {
        /*
         * Setup a timer resolution of 1000 ns : 1 us
         */
        prctl(PR_SET_TIMERSLACK, 1000); 

        /*
         * Server program: sends 50,000 packets per second to a multicast address,
         * for 10 seconds.
         */
        int s = socket(AF_INET, SOCK_DGRAM, 0);
        int val = 1;
        int i = 1;
        check(s > 0);

        struct sockaddr_in addr;
        memset(&addr, 0, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons(g_port);
        check(inet_pton(AF_INET, g_mcastaddr, &addr.sin_addr.s_addr));
        check(connect(s, (struct sockaddr *) &addr, sizeof(addr)) == 0);

        int npackets = 500000;
        char buf[100];
        memset(buf, 0, sizeof(buf));
        for(i = 1; i < npackets; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            usleep(20); // 50,000 messages per second
        }

        buf[0] = 1;
        for(i = 1; i < 5; ++i)
        {
            check(send(s, buf, sizeof(buf), 0) > 0);
            sleep(1);
        }
    }
    else
        error("unknown mode");
    return 0;
}

Re: Multicast packet loss

[Neil Horman]

On Tue, Feb 03, 2009 at 12:34:54PM -0500, Kenny Chang wrote:
> Eric Dumazet wrote:
>> Wes Chow a écrit :
>>   
>>> Eric Dumazet wrote:
>>>     
>>>> Wes Chow a écrit :
>>>>       
>>>>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>>>>
>>>>> First I'd like to note that there were a lot of bnx2 NAPI changes
>>>>> between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts
>>>>> of packet loss,
>>>>> whereas loss in 2.6.22 is significant.
>>>>>
>>>>> Second, some CPU affinity info: if I do like Eric and pin all of the
>>>>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>>>>> ksoftirqd show up on top at all!
>>>>>
>>>>> If I pin half the processes on one CPU and the other half on another
>>>>> CPU, one ksoftirqd processes shows up in top and completely pegs one
>>>>> CPU. My packet loss
>>>>> in that case is significant (25%).
>>>>>
>>>>> Now, the strange case: if I pin 3 processes to one CPU and 1 process
>>>>> to another, I get about 25% packet loss and ksoftirqd pins one CPU.
>>>>> However, one
>>>>> of the apps takes significantly less CPU than the others, and all
>>>>> apps lose the
>>>>> *exact same number of packets*. In all other situations where we see
>>>>> packet
>>>>> loss, the actual number lost per application instance appears random.
>>>>>         
>>>> You see same number of packet lost because they are lost at NIC level
>>>>       
>>> Understood.
>>>
>>> I have a new observation: if I pin processes to just CPUs 0 and 1, I see
>>> no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and
>>> 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any
>>> other combination appears to produce loss (though I have not tried all
>>> 28 combinations, this seems to be the case).
>>>
>>> At first I thought maybe it had to do with processes pinned to the same
>>> CPU, but different cores. The machine is a dual quad core, which means
>>> that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3
>>> produce packet loss.
>>>     
>>
>> a quad core is really a 2 x 2 core
>>
>> L2 cache is splited on two blocks, one block used by CPU0/1, other by 
>> CPU2/3 
>>
>> You are at the limit of the machine with such workload, so as soon as your
>> CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.
>>
>>
>>   
>>> I've also noticed that it does not matter which of the working pairs I
>>> pin to. For example, pinning 5 processes in any combination on either
>>> 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no
>>> packet loss.
>>>
>>> The failures are also sudden. In all of the working cases mentioned
>>> above, I don't see ksoftirqd on top at all. But when I run 6 processes
>>> on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of
>>> packets.
>>>
>>>     
>>>> Normaly, softirq runs on same cpu (the one handling hard irq)
>>>>       
>>> What determines which CPU the hard irq occurs on?
>>>
>>>     
>>
>> Check /proc/irq/{irqnumber}/smp_affinity
>>
>> If you want IRQ16 only served by CPU0 :
>>
>> echo 1 >/proc/irq/16/smp_affinity
>>
>>   
> Hi everyone,
>
> First, thanks for all the effort so far, I think we've learned so much  
> more about the problem in the last couple of days than we had previously  
> in a month.
>
> Just to summarize where we are:
>
> * pinning processes to specific cores/CPUs alleviate the problem
> * issues exist from 2.6.22 up to 2.6.29-rc3
> * issue does not appear to be isolated to 64-bit, 32-bits have problems  
> too.
> * I'm attaching an updated test program with the PR_SET_TIMERSTACK call  
> added.
> * on troubled machines, we are seeing high number of context switches  
> and interrupts.
> * we've ordered an Intel card to try in our machine to see if we can  
> circumvent the issue with a different driver.
>
> Kernel Version         Has Problem?     Notes
> ----------             ----------       ----------
> 2.6.15.x                N    2.6.16.x                -
> 2.6.17.x                -               Doesn't build on Hardy
> 2.6.18.x                -               Doesn't boot (kernel panic)
> 2.6.19.7                N               ksoftirqd is up there, but not  
> pegging a CPU.
>                                        Takes roughly same amount of CPU  
> as the other
>                                        processes, all of which are from  
> 20-40%
> 2.6.20.21               N
> 2.6.21.7                N               sort of lopsided load, but no  
> load from
>                                        ksoftirqd -- strange
> 2.6.22.19               Y               First broken kernel
> 2.6.23.x                -
> 2.6.24-19               Y               (from Hardy)
> 2.6.25.x                -
> 2.6.26.x                -
> 2.6.27.x                Y               (from Intrepid)
> 2.6.28.1                Y
> 2.6.29-rc               Y
>
>
> Correct me if I'm wrong, from what we've seen, it looks like its  
> pointing to some inefficiency in the softirq handling.  The question is  
> whether it's something in the driver or the kernel.  If we can isolate  
> that, maybe we can take some action to have it fixed.
>
I don't think its sofirq ineffeciencies (oprofile would have shown that).  I
know I keep harping on this, but I still think irq affininty is your problem.
I'd be interested in knowning what your /proc/interrupts file looked like on
each of the above kenrels.  Perhaps its not that the bnx2 card you have can't
handle the setting of MSI interrupt affinities, but rather that something
changeed to break irq affinity on this card.

Neil

>

Re: Multicast packet loss

[Kenny Chang]

Neil Horman wrote:
> On Tue, Feb 03, 2009 at 12:34:54PM -0500, Kenny Chang wrote:
>> Eric Dumazet wrote:
>>> Wes Chow a écrit :
>>>   
>>>> Eric Dumazet wrote:
>>>>     
>>>>> Wes Chow a écrit :
>>>>>       
>>>>>> (I'm Kenny's colleague, and I've been doing the kernel builds)
>>>>>>
>>>>>> First I'd like to note that there were a lot of bnx2 NAPI changes
>>>>>> between 2.6.21 and 2.6.22. As a reminder, 2.6.21 shows tiny amounts
>>>>>> of packet loss,
>>>>>> whereas loss in 2.6.22 is significant.
>>>>>>
>>>>>> Second, some CPU affinity info: if I do like Eric and pin all of the
>>>>>> apps onto a single CPU, I see no packet loss. Also, I do *not* see
>>>>>> ksoftirqd show up on top at all!
>>>>>>
>>>>>> If I pin half the processes on one CPU and the other half on another
>>>>>> CPU, one ksoftirqd processes shows up in top and completely pegs one
>>>>>> CPU. My packet loss
>>>>>> in that case is significant (25%).
>>>>>>
>>>>>> Now, the strange case: if I pin 3 processes to one CPU and 1 process
>>>>>> to another, I get about 25% packet loss and ksoftirqd pins one CPU.
>>>>>> However, one
>>>>>> of the apps takes significantly less CPU than the others, and all
>>>>>> apps lose the
>>>>>> *exact same number of packets*. In all other situations where we see
>>>>>> packet
>>>>>> loss, the actual number lost per application instance appears random.
>>>>>>         
>>>>> You see same number of packet lost because they are lost at NIC level
>>>>>       
>>>> Understood.
>>>>
>>>> I have a new observation: if I pin processes to just CPUs 0 and 1, I see
>>>> no packet loss. Pinning to 0 and 2, I do see packet loss. Pinning 2 and
>>>> 3, no packet loss. 4 & 5 - no packet loss, 6 & 7 - no packet loss. Any
>>>> other combination appears to produce loss (though I have not tried all
>>>> 28 combinations, this seems to be the case).
>>>>
>>>> At first I thought maybe it had to do with processes pinned to the same
>>>> CPU, but different cores. The machine is a dual quad core, which means
>>>> that CPUs 0-3 should be a physical CPU, correct? Pinning to 0/2 and 0/3
>>>> produce packet loss.
>>>>     
>>> a quad core is really a 2 x 2 core
>>>
>>> L2 cache is splited on two blocks, one block used by CPU0/1, other by 
>>> CPU2/3 
>>>
>>> You are at the limit of the machine with such workload, so as soon as your
>>> CPUs have to transfert 64 bytes lines between those two L2 blocks, you loose.
>>>
>>>
>>>   
>>>> I've also noticed that it does not matter which of the working pairs I
>>>> pin to. For example, pinning 5 processes in any combination on either
>>>> 0/1 produce no packet loss, pinning all 5 to just CPU 0 also produces no
>>>> packet loss.
>>>>
>>>> The failures are also sudden. In all of the working cases mentioned
>>>> above, I don't see ksoftirqd on top at all. But when I run 6 processes
>>>> on a single CPU, ksoftirqd shoots up to 100% and I lose a huge number of
>>>> packets.
>>>>
>>>>     
>>>>> Normaly, softirq runs on same cpu (the one handling hard irq)
>>>>>       
>>>> What determines which CPU the hard irq occurs on?
>>>>
>>>>     
>>> Check /proc/irq/{irqnumber}/smp_affinity
>>>
>>> If you want IRQ16 only served by CPU0 :
>>>
>>> echo 1 >/proc/irq/16/smp_affinity
>>>
>>>   
>> Hi everyone,
>>
>> -snip-
>> Correct me if I'm wrong, from what we've seen, it looks like its  
>> pointing to some inefficiency in the softirq handling.  The question is  
>> whether it's something in the driver or the kernel.  If we can isolate  
>> that, maybe we can take some action to have it fixed.
>>
> I don't think its sofirq ineffeciencies (oprofile would have shown that).  I
> know I keep harping on this, but I still think irq affininty is your problem.
> I'd be interested in knowning what your /proc/interrupts file looked like on
> each of the above kenrels.  Perhaps its not that the bnx2 card you have can't
> handle the setting of MSI interrupt affinities, but rather that something
> changeed to break irq affinity on this card.
>
> Neil
>
>
It's been a while since I updated this thread.  We've been running 
through the different suggestions and tabulating their effects, as well 
as trying out an Intel card.  The short story is that setting affinity 
and MSI works to some extent, and the Intel card doesn't seem to change 
things significantly.  The results don't seem consistent enough for us 
to be able to point to a smoking gun.

It does look like the 2.6.29-rc4 kernel performs okay with the Intel 
card, but this is not a real-time build and it's not likely to be in a 
supported Ubuntu distribution real soon.  We've reached the point where 
we'd like to look for an expert dedicated to work on this problem for a 
period of time.  The final result being some sort of solution to produce 
a realtime configuration with a reasonably "aged" kernel (.24~.28) that 
has multicast performance greater than or equal to that of 2.6.15.

If anybody is interested in devoting some compensated time to this 
issue, we're offering up a bounty: 
http://www.athenacr.com/bounties/multicast-performance/

For completeness, here's the table of our experiment results:

====================== ================== ========= ========== 
=============== ============== ============== =================
Kernel                 flavor             IRQ       affinity   *4x 
mcasttest*  *5x mcasttest* *6x mcasttest*  *Mtools2* [4]_ 
====================== ================== ========= ========== 
=============== ============== ============== =================
Intel 
e1000e                                                                                                                  

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
2.6.24.19              rt                |          any       | 
OK              Maybe          X                             
2.6.24.19              rt                |          CPU0      | 
OK              OK             X                             
2.6.24.19              generic           |          any       | 
X                                                            
2.6.24.19              generic           |          CPU0      | 
OK                                                           
2.6.29-rc3             vanilla-server    |          any       | 
X                                                            
2.6.29-rc3             vanilla-server    |          CPU0      | 
OK                                                           
2.6.29-rc4             vanilla-generic   |          any       | 
X                                             OK             
2.6.29-rc4             vanilla-generic   |          CPU0      | OK   
           OK             OK [5]_        OK             
-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
Broadcom 
BNX2                                                                                                                 

-----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
2.6.24-19              rt                | MSI      any       | 
OK              OK             X                             
2.6.24-19              rt                | MSI      CPU0      | 
OK              Maybe          X                             
2.6.24-19              rt                | APIC     any       | 
OK              OK             X                             
2.6.24-19              rt                | APIC     CPU0      | 
OK              Maybe          X                             
2.6.24-19-bnx-latest   rt                | APIC     CPU0      | 
OK              X                                            
2.6.24-19              server            | MSI      any       | 
X                                                            
2.6.24-19              server            | MSI      CPU0      | 
OK                                                           
2.6.24-19              generic           | APIC     any       | 
X                                                            
2.6.24-19              generic           | APIC     CPU0      | 
OK                                                           
2.6.27-11              generic           | APIC     any       | 
X                                                            
2.6.27-11              generic           | APIC     CPU0      | 
OK              10% drop                                      
2.6.28-8               generic           | APIC     any       | 
OK              X                                             
2.6.28-8               generic           | APIC     CPU0      | 
OK              OK             0.5% drop                      
2.6.29-rc3             vanilla-server    | MSI      any       | 
X                                                            
2.6.29-rc3             vanilla-server    | MSI      CPU0      | 
X                                                            
2.6.29-rc3             vanilla-server    | APIC     any       | 
OK              X                                            
2.6.29-rc3             vanilla-server    | APIC     CPU0      | 
OK              OK                                           
2.6.29-rc4             vanilla-generic   | APIC     any       | 
X                                                            
2.6.29-rc4             vanilla-generic   | APIC     CPU0      | 
OK              3% drop        10% drop       X              
====================== 
==================+=========+==========+===============+==============+==============+=================
* [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
* [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped 
nothing.

Kenny

Re: Multicast packet loss

[Eric Dumazet]

Kenny Chang a écrit :
> It's been a while since I updated this thread.  We've been running
> through the different suggestions and tabulating their effects, as well
> as trying out an Intel card.  The short story is that setting affinity
> and MSI works to some extent, and the Intel card doesn't seem to change
> things significantly.  The results don't seem consistent enough for us
> to be able to point to a smoking gun.
> 
> It does look like the 2.6.29-rc4 kernel performs okay with the Intel
> card, but this is not a real-time build and it's not likely to be in a
> supported Ubuntu distribution real soon.  We've reached the point where
> we'd like to look for an expert dedicated to work on this problem for a
> period of time.  The final result being some sort of solution to produce
> a realtime configuration with a reasonably "aged" kernel (.24~.28) that
> has multicast performance greater than or equal to that of 2.6.15.
> 
> If anybody is interested in devoting some compensated time to this
> issue, we're offering up a bounty:
> http://www.athenacr.com/bounties/multicast-performance/
> 
> For completeness, here's the table of our experiment results:
> 
> ====================== ================== ========= ==========
> =============== ============== ============== =================
> Kernel                 flavor             IRQ       affinity   *4x
> mcasttest*  *5x mcasttest* *6x mcasttest*  *Mtools2* [4]_
> ====================== ================== ========= ==========
> =============== ============== ============== =================
> Intel
> e1000e                                                                                                                 
> 
> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
> 
> 2.6.24.19              rt                |          any       |
> OK              Maybe          X                            
> 2.6.24.19              rt                |          CPU0      |
> OK              OK             X                            
> 2.6.24.19              generic           |          any       |
> X                                                           
> 2.6.24.19              generic           |          CPU0      |
> OK                                                          
> 2.6.29-rc3             vanilla-server    |          any       |
> X                                                           
> 2.6.29-rc3             vanilla-server    |          CPU0      |
> OK                                                          
> 2.6.29-rc4             vanilla-generic   |          any       |
> X                                             OK            
> 2.6.29-rc4             vanilla-generic   |          CPU0      | OK  
>           OK             OK [5]_        OK            
> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
> 
> Broadcom
> BNX2                                                                                                                
> 
> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
> 
> 2.6.24-19              rt                | MSI      any       |
> OK              OK             X                            
> 2.6.24-19              rt                | MSI      CPU0      |
> OK              Maybe          X                            
> 2.6.24-19              rt                | APIC     any       |
> OK              OK             X                            
> 2.6.24-19              rt                | APIC     CPU0      |
> OK              Maybe          X                            
> 2.6.24-19-bnx-latest   rt                | APIC     CPU0      |
> OK              X                                           
> 2.6.24-19              server            | MSI      any       |
> X                                                           
> 2.6.24-19              server            | MSI      CPU0      |
> OK                                                          
> 2.6.24-19              generic           | APIC     any       |
> X                                                           
> 2.6.24-19              generic           | APIC     CPU0      |
> OK                                                          
> 2.6.27-11              generic           | APIC     any       |
> X                                                           
> 2.6.27-11              generic           | APIC     CPU0      |
> OK              10% drop                                     
> 2.6.28-8               generic           | APIC     any       |
> OK              X                                            
> 2.6.28-8               generic           | APIC     CPU0      |
> OK              OK             0.5% drop                     
> 2.6.29-rc3             vanilla-server    | MSI      any       |
> X                                                           
> 2.6.29-rc3             vanilla-server    | MSI      CPU0      |
> X                                                           
> 2.6.29-rc3             vanilla-server    | APIC     any       |
> OK              X                                           
> 2.6.29-rc3             vanilla-server    | APIC     CPU0      |
> OK              OK                                          
> 2.6.29-rc4             vanilla-generic   | APIC     any       |
> X                                                           
> 2.6.29-rc4             vanilla-generic   | APIC     CPU0      |
> OK              3% drop        10% drop       X             
> ======================
> ==================+=========+==========+===============+==============+==============+=================
> 
> * [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
> * [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped
> nothing.
> 
> Kenny
> 

Hi Kenny

I am investigating how to reduce contention (and schedule() calls) on this workload.

Following patch already gave me less packet drops (but not yet *perfect*)
(10% packet loss instead of 30%, if 8 receivers on my 8 cpus machine)


David, this is a preliminary work, not meant for inclusion as is,
comments are welcome.

Thank you

[PATCH] net: sk_forward_alloc becomes an atomic_t

Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting) introduced a regression for high rate UDP flows,
because of extra lock_sock() in udp_recvmsg()

In order to reduce need for lock_sock() in UDP receive path, we might need
to declare sk_forward_alloc as an atomic_t.

udp_recvmsg() can avoid a lock_sock()/release_sock() pair.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
---
 include/net/sock.h   |   14 +++++++-------
 net/core/sock.c      |   31 +++++++++++++++++++------------
 net/core/stream.c    |    2 +-
 net/ipv4/af_inet.c   |    2 +-
 net/ipv4/inet_diag.c |    2 +-
 net/ipv4/tcp_input.c |    2 +-
 net/ipv4/udp.c       |    2 --
 net/ipv6/udp.c       |    2 --
 net/sched/em_meta.c  |    2 +-
 9 files changed, 31 insertions(+), 28 deletions(-)

diff --git a/include/net/sock.h b/include/net/sock.h
index 4bb1ff9..c4befb9 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -250,7 +250,7 @@ struct sock {
 	struct sk_buff_head	sk_async_wait_queue;
 #endif
 	int			sk_wmem_queued;
-	int			sk_forward_alloc;
+	atomic_t		sk_forward_alloc;
 	gfp_t			sk_allocation;
 	int			sk_route_caps;
 	int			sk_gso_type;
@@ -823,7 +823,7 @@ static inline int sk_wmem_schedule(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return 1;
-	return size <= sk->sk_forward_alloc ||
+	return size <= atomic_read(&sk->sk_forward_alloc) ||
 		__sk_mem_schedule(sk, size, SK_MEM_SEND);
 }
 
@@ -831,7 +831,7 @@ static inline int sk_rmem_schedule(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return 1;
-	return size <= sk->sk_forward_alloc ||
+	return size <= atomic_read(&sk->sk_forward_alloc) ||
 		__sk_mem_schedule(sk, size, SK_MEM_RECV);
 }
 
@@ -839,7 +839,7 @@ static inline void sk_mem_reclaim(struct sock *sk)
 {
 	if (!sk_has_account(sk))
 		return;
-	if (sk->sk_forward_alloc >= SK_MEM_QUANTUM)
+	if (atomic_read(&sk->sk_forward_alloc) >= SK_MEM_QUANTUM)
 		__sk_mem_reclaim(sk);
 }
 
@@ -847,7 +847,7 @@ static inline void sk_mem_reclaim_partial(struct sock *sk)
 {
 	if (!sk_has_account(sk))
 		return;
-	if (sk->sk_forward_alloc > SK_MEM_QUANTUM)
+	if (atomic_read(&sk->sk_forward_alloc) > SK_MEM_QUANTUM)
 		__sk_mem_reclaim(sk);
 }
 
@@ -855,14 +855,14 @@ static inline void sk_mem_charge(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return;
-	sk->sk_forward_alloc -= size;
+	atomic_sub(size, &sk->sk_forward_alloc);
 }
 
 static inline void sk_mem_uncharge(struct sock *sk, int size)
 {
 	if (!sk_has_account(sk))
 		return;
-	sk->sk_forward_alloc += size;
+	atomic_add(size, &sk->sk_forward_alloc);
 }
 
 static inline void sk_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
diff --git a/net/core/sock.c b/net/core/sock.c
index 0620046..8489105 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1081,7 +1081,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 
 		newsk->sk_dst_cache	= NULL;
 		newsk->sk_wmem_queued	= 0;
-		newsk->sk_forward_alloc = 0;
+		atomic_set(&newsk->sk_forward_alloc, 0);
 		newsk->sk_send_head	= NULL;
 		newsk->sk_userlocks	= sk->sk_userlocks & ~SOCK_BINDPORT_LOCK;
 
@@ -1479,7 +1479,7 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind)
 	int amt = sk_mem_pages(size);
 	int allocated;
 
-	sk->sk_forward_alloc += amt * SK_MEM_QUANTUM;
+	atomic_add(amt * SK_MEM_QUANTUM, &sk->sk_forward_alloc);
 	allocated = atomic_add_return(amt, prot->memory_allocated);
 
 	/* Under limit. */
@@ -1520,7 +1520,7 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind)
 		if (prot->sysctl_mem[2] > alloc *
 		    sk_mem_pages(sk->sk_wmem_queued +
 				 atomic_read(&sk->sk_rmem_alloc) +
-				 sk->sk_forward_alloc))
+				 atomic_read(&sk->sk_forward_alloc)))
 			return 1;
 	}
 
@@ -1537,7 +1537,7 @@ suppress_allocation:
 	}
 
 	/* Alas. Undo changes. */
-	sk->sk_forward_alloc -= amt * SK_MEM_QUANTUM;
+	atomic_sub(amt * SK_MEM_QUANTUM, &sk->sk_forward_alloc);
 	atomic_sub(amt, prot->memory_allocated);
 	return 0;
 }
@@ -1551,14 +1551,21 @@ EXPORT_SYMBOL(__sk_mem_schedule);
 void __sk_mem_reclaim(struct sock *sk)
 {
 	struct proto *prot = sk->sk_prot;
-
-	atomic_sub(sk->sk_forward_alloc >> SK_MEM_QUANTUM_SHIFT,
-		   prot->memory_allocated);
-	sk->sk_forward_alloc &= SK_MEM_QUANTUM - 1;
-
-	if (prot->memory_pressure && *prot->memory_pressure &&
-	    (atomic_read(prot->memory_allocated) < prot->sysctl_mem[0]))
-		*prot->memory_pressure = 0;
+	int val = atomic_read(&sk->sk_forward_alloc);
+
+begin:
+	val = atomic_read(&sk->sk_forward_alloc);
+	if (val >= SK_MEM_QUANTUM) {
+		if (atomic_cmpxchg(&sk->sk_forward_alloc, val,
+				   val & (SK_MEM_QUANTUM - 1)) != val)
+			goto begin;
+		atomic_sub(val >> SK_MEM_QUANTUM_SHIFT,
+			   prot->memory_allocated);
+
+		if (prot->memory_pressure && *prot->memory_pressure &&
+		    (atomic_read(prot->memory_allocated) < prot->sysctl_mem[0]))
+			*prot->memory_pressure = 0;
+	}
 }
 
 EXPORT_SYMBOL(__sk_mem_reclaim);
diff --git a/net/core/stream.c b/net/core/stream.c
index 8727cea..4d04d28 100644
--- a/net/core/stream.c
+++ b/net/core/stream.c
@@ -198,7 +198,7 @@ void sk_stream_kill_queues(struct sock *sk)
 	sk_mem_reclaim(sk);
 
 	WARN_ON(sk->sk_wmem_queued);
-	WARN_ON(sk->sk_forward_alloc);
+	WARN_ON(atomic_read(&sk->sk_forward_alloc));
 
 	/* It is _impossible_ for the backlog to contain anything
 	 * when we get here.  All user references to this socket
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 627be4d..7a1475c 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -152,7 +152,7 @@ void inet_sock_destruct(struct sock *sk)
 	WARN_ON(atomic_read(&sk->sk_rmem_alloc));
 	WARN_ON(atomic_read(&sk->sk_wmem_alloc));
 	WARN_ON(sk->sk_wmem_queued);
-	WARN_ON(sk->sk_forward_alloc);
+	WARN_ON(atomic_read(&sk->sk_forward_alloc));
 
 	kfree(inet->opt);
 	dst_release(sk->sk_dst_cache);
diff --git a/net/ipv4/inet_diag.c b/net/ipv4/inet_diag.c
index 588a779..903ad66 100644
--- a/net/ipv4/inet_diag.c
+++ b/net/ipv4/inet_diag.c
@@ -158,7 +158,7 @@ static int inet_csk_diag_fill(struct sock *sk,
 	if (minfo) {
 		minfo->idiag_rmem = atomic_read(&sk->sk_rmem_alloc);
 		minfo->idiag_wmem = sk->sk_wmem_queued;
-		minfo->idiag_fmem = sk->sk_forward_alloc;
+		minfo->idiag_fmem = atomic_read(&sk->sk_forward_alloc);
 		minfo->idiag_tmem = atomic_read(&sk->sk_wmem_alloc);
 	}
 
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index a6961d7..5e08f37 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -5258,7 +5258,7 @@ int tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 				tcp_rcv_rtt_measure_ts(sk, skb);
 
-				if ((int)skb->truesize > sk->sk_forward_alloc)
+				if ((int)skb->truesize > atomic_read(&sk->sk_forward_alloc))
 					goto step5;
 
 				NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPHPHITS);
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 4bd178a..dcc246a 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -955,9 +955,7 @@ try_again:
 		err = ulen;
 
 out_free:
-	lock_sock(sk);
 	skb_free_datagram(sk, skb);
-	release_sock(sk);
 out:
 	return err;
 
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 84b1a29..582b80a 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -257,9 +257,7 @@ try_again:
 		err = ulen;
 
 out_free:
-	lock_sock(sk);
 	skb_free_datagram(sk, skb);
-	release_sock(sk);
 out:
 	return err;
 
diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c
index 72cf86e..94d90b6 100644
--- a/net/sched/em_meta.c
+++ b/net/sched/em_meta.c
@@ -383,7 +383,7 @@ META_COLLECTOR(int_sk_wmem_queued)
 META_COLLECTOR(int_sk_fwd_alloc)
 {
 	SKIP_NONLOCAL(skb);
-	dst->value = skb->sk->sk_forward_alloc;
+	dst->value = atomic_read(&skb->sk->sk_forward_alloc);
 }
 
 META_COLLECTOR(int_sk_sndbuf)

Re: Multicast packet loss

[Eric Dumazet]

Eric Dumazet a écrit :
> Kenny Chang a écrit :
>> It's been a while since I updated this thread.  We've been running
>> through the different suggestions and tabulating their effects, as well
>> as trying out an Intel card.  The short story is that setting affinity
>> and MSI works to some extent, and the Intel card doesn't seem to change
>> things significantly.  The results don't seem consistent enough for us
>> to be able to point to a smoking gun.
>>
>> It does look like the 2.6.29-rc4 kernel performs okay with the Intel
>> card, but this is not a real-time build and it's not likely to be in a
>> supported Ubuntu distribution real soon.  We've reached the point where
>> we'd like to look for an expert dedicated to work on this problem for a
>> period of time.  The final result being some sort of solution to produce
>> a realtime configuration with a reasonably "aged" kernel (.24~.28) that
>> has multicast performance greater than or equal to that of 2.6.15.
>>
>> If anybody is interested in devoting some compensated time to this
>> issue, we're offering up a bounty:
>> http://www.athenacr.com/bounties/multicast-performance/
>>
>> For completeness, here's the table of our experiment results:
>>
>> ====================== ================== ========= ==========
>> =============== ============== ============== =================
>> Kernel                 flavor             IRQ       affinity   *4x
>> mcasttest*  *5x mcasttest* *6x mcasttest*  *Mtools2* [4]_
>> ====================== ================== ========= ==========
>> =============== ============== ============== =================
>> Intel
>> e1000e                                                                                                                 
>>
>> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
>>
>> 2.6.24.19              rt                |          any       |
>> OK              Maybe          X                            
>> 2.6.24.19              rt                |          CPU0      |
>> OK              OK             X                            
>> 2.6.24.19              generic           |          any       |
>> X                                                           
>> 2.6.24.19              generic           |          CPU0      |
>> OK                                                          
>> 2.6.29-rc3             vanilla-server    |          any       |
>> X                                                           
>> 2.6.29-rc3             vanilla-server    |          CPU0      |
>> OK                                                          
>> 2.6.29-rc4             vanilla-generic   |          any       |
>> X                                             OK            
>> 2.6.29-rc4             vanilla-generic   |          CPU0      | OK  
>>           OK             OK [5]_        OK            
>> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
>>
>> Broadcom
>> BNX2                                                                                                                
>>
>> -----------------------------------------+---------+----------+---------------+--------------+--------------+-----------------
>>
>> 2.6.24-19              rt                | MSI      any       |
>> OK              OK             X                            
>> 2.6.24-19              rt                | MSI      CPU0      |
>> OK              Maybe          X                            
>> 2.6.24-19              rt                | APIC     any       |
>> OK              OK             X                            
>> 2.6.24-19              rt                | APIC     CPU0      |
>> OK              Maybe          X                            
>> 2.6.24-19-bnx-latest   rt                | APIC     CPU0      |
>> OK              X                                           
>> 2.6.24-19              server            | MSI      any       |
>> X                                                           
>> 2.6.24-19              server            | MSI      CPU0      |
>> OK                                                          
>> 2.6.24-19              generic           | APIC     any       |
>> X                                                           
>> 2.6.24-19              generic           | APIC     CPU0      |
>> OK                                                          
>> 2.6.27-11              generic           | APIC     any       |
>> X                                                           
>> 2.6.27-11              generic           | APIC     CPU0      |
>> OK              10% drop                                     
>> 2.6.28-8               generic           | APIC     any       |
>> OK              X                                            
>> 2.6.28-8               generic           | APIC     CPU0      |
>> OK              OK             0.5% drop                     
>> 2.6.29-rc3             vanilla-server    | MSI      any       |
>> X                                                           
>> 2.6.29-rc3             vanilla-server    | MSI      CPU0      |
>> X                                                           
>> 2.6.29-rc3             vanilla-server    | APIC     any       |
>> OK              X                                           
>> 2.6.29-rc3             vanilla-server    | APIC     CPU0      |
>> OK              OK                                          
>> 2.6.29-rc4             vanilla-generic   | APIC     any       |
>> X                                                           
>> 2.6.29-rc4             vanilla-generic   | APIC     CPU0      |
>> OK              3% drop        10% drop       X             
>> ======================
>> ==================+=========+==========+===============+==============+==============+=================
>>
>> * [4] MTools2 is a test from 29West: http://www.29west.com/docs/TestNet/
>> * [5] In 5 trials, 1 of the trials dropped 2%, 4 of the trials dropped
>> nothing.
>>
>> Kenny
>>
> 
> Hi Kenny
> 
> I am investigating how to reduce contention (and schedule() calls) on this workload.
> 

I bound NIC (gigabit BNX2) irq to cpu 0, so that oprofile results on this cpu can show us
where ksoftirqd is spending its time.

We can see scheduler at work :)

Also, one thing to note is __copy_skb_header() : 9.49 % of cpu0 time.
The problem comes from dst_clone() (6.05 % total, so 2/3 of __copy_skb_header()),
touching a highly contended cache line. (other cpus are doing the decrement of
dst refcounter)

CPU: Core 2, speed 3000.05 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) 
with a unit mask of 0x00 (Unhalted core cycles) count 100000
Samples on CPU 0
(samples for other cpus 1..7 omitted)
samples  cum. samples  %        cum. %     symbol name
23750    23750          9.8159   9.8159    try_to_wake_up
22972    46722          9.4944  19.3103    __copy_skb_header
20217    66939          8.3557  27.6660    enqueue_task_fair
14565    81504          6.0197  33.6857    sock_def_readable
13454    94958          5.5606  39.2463    task_rq_lock
13381    108339         5.5304  44.7767    resched_task
13090    121429         5.4101  50.1868    udp_queue_rcv_skb
11441    132870         4.7286  54.9154    skb_queue_tail
10109    142979         4.1781  59.0935    sock_queue_rcv_skb
10024    153003         4.1429  63.2364    __wake_up_sync
9952     162955         4.1132  67.3496    update_curr
8761     171716         3.6209  70.9705    sched_clock_cpu
7414     179130         3.0642  74.0347    rb_insert_color
7381     186511         3.0506  77.0853    select_task_rq_fair
6749     193260         2.7894  79.8747    __slab_alloc
5881     199141         2.4306  82.3053    __wake_up_common
5432     204573         2.2451  84.5504    __skb_clone
4306     208879         1.7797  86.3300    kmem_cache_alloc
3524     212403         1.4565  87.7865    place_entity
2783     215186         1.1502  88.9367    skb_clone
2576     217762         1.0647  90.0014    __udp4_lib_rcv
2430     220192         1.0043  91.0057    bnx2_poll_work
2184     222376         0.9027  91.9084    ipt_do_table
2090     224466         0.8638  92.7722    ip_route_input
1877     226343         0.7758  93.5479    __alloc_skb
1495     227838         0.6179  94.1658    native_sched_clock
1166     229004         0.4819  94.6477    __update_sched_clock
1083     230087         0.4476  95.0953    netif_receive_skb
1062     231149         0.4389  95.5343    activate_task
644      231793         0.2662  95.8004    __kmalloc_track_caller
638      232431         0.2637  96.0641    nf_iterate
549      232980         0.2269  96.2910    skb_put

Re: Multicast packet loss

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Sat, 28 Feb 2009 09:51:11 +0100

> David, this is a preliminary work, not meant for inclusion as is,
> comments are welcome.
> 
> [PATCH] net: sk_forward_alloc becomes an atomic_t
> 
> Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
> (UDP: Add memory accounting) introduced a regression for high rate UDP flows,
> because of extra lock_sock() in udp_recvmsg()
> 
> In order to reduce need for lock_sock() in UDP receive path, we might need
> to declare sk_forward_alloc as an atomic_t.
> 
> udp_recvmsg() can avoid a lock_sock()/release_sock() pair.
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

This adds new overhead for TCP which has to hold the socket
lock for other reasons in these paths.

I don't get how an atomic_t operation is cheaper than a
lock_sock/release_sock.  Is it the case that in many
executions of these paths only atomic_read()'s are necessary?

I actually think this scheme is racy.  There is a reason we
have to hold the socket lock when doing memory scheduling.
Two threads can get in there and say "hey I have enough space
already" even though only enough space is allocated for one
of their requests.

What did I miss? :)

Re: Multicast packet loss

[Eric Dumazet]

David Miller a écrit :
> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Sat, 28 Feb 2009 09:51:11 +0100
> 
>> David, this is a preliminary work, not meant for inclusion as is,
>> comments are welcome.
>>
>> [PATCH] net: sk_forward_alloc becomes an atomic_t
>>
>> Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
>> (UDP: Add memory accounting) introduced a regression for high rate UDP flows,
>> because of extra lock_sock() in udp_recvmsg()
>>
>> In order to reduce need for lock_sock() in UDP receive path, we might need
>> to declare sk_forward_alloc as an atomic_t.
>>
>> udp_recvmsg() can avoid a lock_sock()/release_sock() pair.
>>
>> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
> 
> This adds new overhead for TCP which has to hold the socket
> lock for other reasons in these paths.
> 
> I don't get how an atomic_t operation is cheaper than a
> lock_sock/release_sock.  Is it the case that in many
> executions of these paths only atomic_read()'s are necessary?
> 
> I actually think this scheme is racy.  There is a reason we
> have to hold the socket lock when doing memory scheduling.
> Two threads can get in there and say "hey I have enough space
> already" even though only enough space is allocated for one
> of their requests.
> 
> What did I miss? :)
> 

I believe you are right, and in fact was about to post a "dont look at this patch"
since it doesnt help the multicast reception at all, I redone tests more carefuly 
and got nothing but noise.

We have a cache line ping pong mess here, and need more thinking.

I rewrote Kenny prog to use non blocking sockets.

Receivers are doing :

        int delay = 50;
	fcntl(s, F_SETFL, O_NDELAY);
        while(1)
        {
            struct sockaddr_in from;
            socklen_t fromlen = sizeof(from);
            res = recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen);
            if (res == -1) {
                      delay++;
                      usleep(delay);
                      continue;
            }
            if (delay > 40)
                delay--;
            ++npackets;

With this litle user space change and 8 receivers on my dual quad core, softirqd
only takes 8% of one cpu and no drops at all (instead of 100% cpu and 30% drops)

So this is definitly a problem mixing scheduler cache line ping pongs with network
stack cache line ping pongs.

We could reorder fields so that fewer cache lines are touched by the softirq processing,
I tried this but still got packet drops.

Re: Multicast packet loss

[Eric Dumazet]

Eric Dumazet a écrit :
> David Miller a écrit :
>> From: Eric Dumazet <dada1@cosmosbay.com>
>> Date: Sat, 28 Feb 2009 09:51:11 +0100
>>
>>> David, this is a preliminary work, not meant for inclusion as is,
>>> comments are welcome.
>>>
>>> [PATCH] net: sk_forward_alloc becomes an atomic_t
>>>
>>> Commit 95766fff6b9a78d11fc2d3812dd035381690b55d
>>> (UDP: Add memory accounting) introduced a regression for high rate UDP flows,
>>> because of extra lock_sock() in udp_recvmsg()
>>>
>>> In order to reduce need for lock_sock() in UDP receive path, we might need
>>> to declare sk_forward_alloc as an atomic_t.
>>>
>>> udp_recvmsg() can avoid a lock_sock()/release_sock() pair.
>>>
>>> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
>> This adds new overhead for TCP which has to hold the socket
>> lock for other reasons in these paths.
>>
>> I don't get how an atomic_t operation is cheaper than a
>> lock_sock/release_sock.  Is it the case that in many
>> executions of these paths only atomic_read()'s are necessary?
>>
>> I actually think this scheme is racy.  There is a reason we
>> have to hold the socket lock when doing memory scheduling.
>> Two threads can get in there and say "hey I have enough space
>> already" even though only enough space is allocated for one
>> of their requests.
>>
>> What did I miss? :)
>>
> 
> I believe you are right, and in fact was about to post a "dont look at this patch"
> since it doesnt help the multicast reception at all, I redone tests more carefuly 
> and got nothing but noise.
> 
> We have a cache line ping pong mess here, and need more thinking.
> 
> I rewrote Kenny prog to use non blocking sockets.
> 
> Receivers are doing :
> 
>         int delay = 50;
> 	fcntl(s, F_SETFL, O_NDELAY);
>         while(1)
>         {
>             struct sockaddr_in from;
>             socklen_t fromlen = sizeof(from);
>             res = recvfrom(s, buf, 1000, 0, (struct sockaddr*)&from, &fromlen);
>             if (res == -1) {
>                       delay++;
>                       usleep(delay);
>                       continue;
>             }
>             if (delay > 40)
>                 delay--;
>             ++npackets;
> 
> With this litle user space change and 8 receivers on my dual quad core, softirqd
> only takes 8% of one cpu and no drops at all (instead of 100% cpu and 30% drops)
> 
> So this is definitly a problem mixing scheduler cache line ping pongs with network
> stack cache line ping pongs.
> 
> We could reorder fields so that fewer cache lines are touched by the softirq processing,
> I tried this but still got packet drops.
> 
> 
> 

I have more questions :

What is the maximum latency you can afford on the delivery of the packet(s) ?

Are user apps using real time scheduling ?

I had an idea, that keep cpu handling NIC interrupts only delivering packets to
socket queues, and not messing with scheduler : fast queueing, and wakeing up
a workqueue (on another cpu) to perform the scheduler work. But that means
some extra latency (in the order of 2 or 3 us I guess)

We could enter in this mode automatically, if the NIC rx handler *see* more than
N packets are waiting in NIC queue : In case of moderate or light trafic, no
extra latency would be necessary. This would mean some changes in NIC driver.

Hum, then, if NIC rx handler is run beside the ksoftirqd, we already know
we are in a stress situation, so maybe no driver changes are necessary :
Just test if we run ksoftirqd...

Re: Multicast packet loss

[Eric Dumazet]

Eric Dumazet a écrit :
> 
> I have more questions :
> 
> What is the maximum latency you can afford on the delivery of the packet(s) ?
> 
> Are user apps using real time scheduling ?
> 
> I had an idea, that keep cpu handling NIC interrupts only delivering packets to
> socket queues, and not messing with scheduler : fast queueing, and wakeing up
> a workqueue (on another cpu) to perform the scheduler work. But that means
> some extra latency (in the order of 2 or 3 us I guess)
> 
> We could enter in this mode automatically, if the NIC rx handler *see* more than
> N packets are waiting in NIC queue : In case of moderate or light trafic, no
> extra latency would be necessary. This would mean some changes in NIC driver.
> 
> Hum, then, if NIC rx handler is run beside the ksoftirqd, we already know
> we are in a stress situation, so maybe no driver changes are necessary :
> Just test if we run ksoftirqd...
> 

Here is a patch that helps. It's still an RFC of course, since its somewhat ugly :)

I am now able to have 8 receivers on my 8 cpus machine, with one multicast packet every 10 us,
without any loss. (standard setup, no affinity games)

oprofile results see that scheduler overhead vanished, we get back to pure network profile :)

(First offender being __copy_skb_header because of the atomic_inc on dst refcount)

CPU: Core 2, speed 3000.43 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  cum. samples  %        cum. %     symbol name
126329   126329        20.4296  20.4296    __copy_skb_header
31395    157724         5.0771  25.5067    udp_queue_rcv_skb
29191    186915         4.7207  30.2274    sock_def_readable
26284    213199         4.2506  34.4780    sock_queue_rcv_skb
26010    239209         4.2063  38.6842    kmem_cache_alloc
20040    259249         3.2408  41.9251    __udp4_lib_rcv
19570    278819         3.1648  45.0899    skb_queue_tail
17799    296618         2.8784  47.9683    bnx2_poll_work
17267    313885         2.7924  50.7606    skb_release_data
14663    328548         2.3713  53.1319    __skb_recv_datagram
14443    342991         2.3357  55.4676    __slab_alloc
13248    356239         2.1424  57.6100    copy_to_user
13138    369377         2.1246  59.7347    __sk_mem_schedule
12004    381381         1.9413  61.6759    __skb_clone
11924    393305         1.9283  63.6042    skb_clone
11077    404382         1.7913  65.3956    lock_sock_nested
10320    414702         1.6689  67.0645    ip_route_input
9622     424324         1.5560  68.6205    dst_release
8344     432668         1.3494  69.9699    __slab_free
8124     440792         1.3138  71.2837    mwait_idle
7066     447858         1.1427  72.4264    udp_recvmsg
6652     454510         1.0757  73.5021    netif_receive_skb
6386     460896         1.0327  74.5349    ipt_do_table
6010     466906         0.9719  75.5068    release_sock
6003     472909         0.9708  76.4776    memcpy_toiovec
5697     478606         0.9213  77.3989    __alloc_skb
5671     484277         0.9171  78.3160    copy_from_user
5031     489308         0.8136  79.1296    sysenter_past_esp
4753     494061         0.7686  79.8982    bnx2_interrupt
4429     498490         0.7162  80.6145    sock_rfree


[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example,
each received multicast frame can wakeup many threads. ksoftirqd is then
not able to drain NIC RX queues and we get frame losses and high latencies.

This patch adds an infrastructure to delay part of work done in
sock_def_readable() at end of do_softirq()


Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
---
 include/linux/interrupt.h |    9 +++++++++
 include/net/sock.h        |    1 +
 kernel/softirq.c          |   29 ++++++++++++++++++++++++++++-
 net/core/sock.c           |   21 +++++++++++++++++++--
 4 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 9127f6b..62caaae 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -295,6 +295,15 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_del {
+	struct softirq_del	*next;
+	void 			(*func)(struct softirq_del *);
+};
+int softirq_del(struct softirq_del *sdel, void (*func)(struct softirq_del *));
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
diff --git a/include/net/sock.h b/include/net/sock.h
index eefeeaf..95841de 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_del	sk_del;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
diff --git a/kernel/softirq.c b/kernel/softirq.c
index bdbe9de..40fe527 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -158,6 +158,33 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+static DEFINE_PER_CPU(struct softirq_del *, softirq_del_head);
+int softirq_del(struct softirq_del *sdel, void (*func)(struct softirq_del *))
+{
+	if (cmpxchg(&sdel->func, NULL, func) == NULL) {
+		sdel->next = __get_cpu_var(softirq_del_head);
+		__get_cpu_var(softirq_del_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+static void softirqdel_exec(void)
+{
+	struct softirq_del *sdel;
+	void (*func)(struct softirq_del *);
+
+	while ((sdel = __get_cpu_var(softirq_del_head)) != NULL) {
+		__get_cpu_var(softirq_del_head) = sdel->next;
+		func = sdel->func;
+		sdel->func = NULL;
+		(*func)(sdel);
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -219,7 +246,7 @@ restart:
 
 	if (pending)
 		wakeup_softirqd();
-
+	softirqdel_exec();
 	trace_softirq_exit();
 
 	account_system_vtime(current);
diff --git a/net/core/sock.c b/net/core/sock.c
index 5f97caa..f9ee8dd 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1026,6 +1026,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		newsk->sk_del.func = NULL;
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1634,12 +1635,27 @@ static void sock_def_error_report(struct sock *sk)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+static void sock_readable_defer(struct softirq_del *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_del);
+
+	wake_up_interruptible_sync(sk->sk_sleep);
+	read_unlock(&sk->sk_callback_lock);
+}
+
 static void sock_def_readable(struct sock *sk, int len)
 {
 	read_lock(&sk->sk_callback_lock);
-	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep))
-		wake_up_interruptible_sync(sk->sk_sleep);
 	sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (in_softirq()) {
+			if (!softirq_del(&sk->sk_del, sock_readable_defer))
+				goto unlock;
+			return;
+		}
+		wake_up_interruptible_sync(sk->sk_sleep);
+	}
+unlock:
 	read_unlock(&sk->sk_callback_lock);
 }
 
@@ -1720,6 +1736,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	sk->sk_del.func		=	NULL;
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,

Re: Multicast packet loss

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Sun, 08 Mar 2009 17:46:13 +0100

> +	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
> +		if (in_softirq()) {
> +			if (!softirq_del(&sk->sk_del, sock_readable_defer))
> +				goto unlock;
> +			return;
> +		}

This is interesting.

I think you should make softirq_del() more flexible.  Make it the
socket's job to make sure it doesn't try to defer different
functions, and put the onus on locking there as well.

The cmpxchg() and all of this checking is just wasted work.

I'd really like to get rid of that callback lock too, then we'd
really be in business. :-)

Re: Multicast packet loss

[Eric Dumazet]

David Miller a écrit :
> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Sun, 08 Mar 2009 17:46:13 +0100
> 
>> +	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
>> +		if (in_softirq()) {
>> +			if (!softirq_del(&sk->sk_del, sock_readable_defer))
>> +				goto unlock;
>> +			return;
>> +		}
> 
> This is interesting.
> 
> I think you should make softirq_del() more flexible.  Make it the
> socket's job to make sure it doesn't try to defer different
> functions, and put the onus on locking there as well.
> 
> The cmpxchg() and all of this checking is just wasted work.
> 
> I'd really like to get rid of that callback lock too, then we'd
> really be in business. :-)

First thanks for your review David.

I chose cmpxchg() because I needed some form of exclusion here.
I first added a spinlock inside "struct softirq_del" then I realize
I could use cmpxchg() instead and keep the structure small. As the
synchronization is only needed at queueing time, we could pass
the address of a spinlock XXX to sofirq_del() call.

Also, when an event was queued for later invocation, I also needed to keep
a reference on "struct socket" to make sure it doesnt disappear before
the invocation. Not all sockets are RCU guarded (we added RCU only for 
some protocols (TCP, UDP ...). So I found keeping a read_lock
on callback was the easyest thing to do. I now realize we might
overflow preempt_count, so special care is needed.

About your first point, maybe we should make sofirq_del() (poor name I confess)
only have one argument (pointer to struct softirq_del), and initialize
the function pointer at socket init time. That would insure "struct softirq_del"
is associated to one callback only. cmpxchg() test would have to be
done on "next" field then (or use the spinlock XXX)

I am not sure output path needs such tricks, since threads are rarely
blocking on output : We dont trigger 400.000 wakeups per second ?

Another point : I did a tbench test and got 2517 MB/s with the patch,
instead of 2538 MB/s (using Linus 2.6 git tree), thats ~ 0.8 % regression
for this workload.

Re: Multicast packet loss

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Mon, 09 Mar 2009 07:36:57 +0100

> I chose cmpxchg() because I needed some form of exclusion here.
> I first added a spinlock inside "struct softirq_del" then I realize
> I could use cmpxchg() instead and keep the structure small. As the
> synchronization is only needed at queueing time, we could pass
> the address of a spinlock XXX to sofirq_del() call.

I don't understand why you need the mutual exclusion in the
first place.  The function pointer always has the same value.
And this locking isn't protecting the list insertion either,
as that isn't even necessary.

It just looks like plain overhead to me.

> Also, when an event was queued for later invocation, I also needed to keep
> a reference on "struct socket" to make sure it doesnt disappear before
> the invocation. Not all sockets are RCU guarded (we added RCU only for 
> some protocols (TCP, UDP ...). So I found keeping a read_lock
> on callback was the easyest thing to do. I now realize we might
> overflow preempt_count, so special care is needed.

You're using this in UDP so... make the rule that you can't use
this with a non-RCU-quiescent protocol.

> About your first point, maybe we should make sofirq_del() (poor name
> I confess) only have one argument (pointer to struct softirq_del),
> and initialize the function pointer at socket init time. That would
> insure "struct softirq_del" is associated to one callback
> only. cmpxchg() test would have to be done on "next" field then (or
> use the spinlock XXX)

Why?  You run this from softirq safe context, nothing can run other
softirqs on this cpu and corrupt the list, therefore.

Re: Multicast packet loss

[Eric Dumazet]

David Miller a écrit :
> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Mon, 09 Mar 2009 07:36:57 +0100
> 
>> I chose cmpxchg() because I needed some form of exclusion here.
>> I first added a spinlock inside "struct softirq_del" then I realize
>> I could use cmpxchg() instead and keep the structure small. As the
>> synchronization is only needed at queueing time, we could pass
>> the address of a spinlock XXX to sofirq_del() call.
> 
> I don't understand why you need the mutual exclusion in the
> first place.  The function pointer always has the same value.
> And this locking isn't protecting the list insertion either,
> as that isn't even necessary.
> 
> It just looks like plain overhead to me.

I was lazy to check all callers (all protocols) had a lock on sock,
and prefered safety.

I was fooled by the read_lock(), and though several cpus could call
this function in //


> 
>> Also, when an event was queued for later invocation, I also needed to keep
>> a reference on "struct socket" to make sure it doesnt disappear before
>> the invocation. Not all sockets are RCU guarded (we added RCU only for 
>> some protocols (TCP, UDP ...). So I found keeping a read_lock
>> on callback was the easyest thing to do. I now realize we might
>> overflow preempt_count, so special care is needed.
> 
> You're using this in UDP so... make the rule that you can't use
> this with a non-RCU-quiescent protocol.

UDP/TCP only ? I though many other protocols (not all using RCU) were
using sock_def_readable() too...

Re: Multicast packet loss

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Fri, 13 Mar 2009 23:30:31 +0100

> David Miller a écrit :
> >> Also, when an event was queued for later invocation, I also needed to keep
> >> a reference on "struct socket" to make sure it doesnt disappear before
> >> the invocation. Not all sockets are RCU guarded (we added RCU only for 
> >> some protocols (TCP, UDP ...). So I found keeping a read_lock
> >> on callback was the easyest thing to do. I now realize we might
> >> overflow preempt_count, so special care is needed.
> > 
> > You're using this in UDP so... make the rule that you can't use
> > this with a non-RCU-quiescent protocol.
> 
> UDP/TCP only ? I though many other protocols (not all using RCU) were
> using sock_def_readable() too...

Maybe create a inet_def_readable() just for this purpose :-)

Re: Multicast packet loss

[Eric Dumazet]

David Miller a écrit :
> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Fri, 13 Mar 2009 23:30:31 +0100
> 
>> David Miller a écrit :
>>>> Also, when an event was queued for later invocation, I also needed to keep
>>>> a reference on "struct socket" to make sure it doesnt disappear before
>>>> the invocation. Not all sockets are RCU guarded (we added RCU only for 
>>>> some protocols (TCP, UDP ...). So I found keeping a read_lock
>>>> on callback was the easyest thing to do. I now realize we might
>>>> overflow preempt_count, so special care is needed.
>>> You're using this in UDP so... make the rule that you can't use
>>> this with a non-RCU-quiescent protocol.
>> UDP/TCP only ? I though many other protocols (not all using RCU) were
>> using sock_def_readable() too...
> 
> Maybe create a inet_def_readable() just for this purpose :-)

I must be tired, I should had this idea before you :)

I post a new patch after some rest, I definitly should not be still awake !

[PATCH] net: reorder fields of struct socket

[Eric Dumazet]

On x86_64, its rather unfortunate that "wait_queue_head_t wait"
field of "struct socket" spans two cache lines (assuming a 64
bytes cache line in current cpus)

offsetof(struct socket, wait)=0x30
sizeof(wait_queue_head_t)=0x18

This might explain why Kenny Chang noticed that his multicast workload
was performing bad with 64 bit kernels, since more cache lines ping pongs
were involved.

This litle patch moves "wait" field next "fasync_list" so that both
fields share a single cache line, to speedup sock_def_readable()

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

diff --git a/include/linux/net.h b/include/linux/net.h
index 4515efa..4fc2ffd 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -129,11 +129,15 @@ struct socket {
 	socket_state		state;
 	short			type;
 	unsigned long		flags;
-	const struct proto_ops	*ops;
+	/*
+	 * Please keep fasync_list & wait fields in the same cache line
+	 */
 	struct fasync_struct	*fasync_list;
+	wait_queue_head_t	wait;
+
 	struct file		*file;
 	struct sock		*sk;
-	wait_queue_head_t	wait;
+	const struct proto_ops	*ops;
 };
 
 struct vm_area_struct;

Re: [PATCH] net: reorder fields of struct socket

[David Miller]

From: Eric Dumazet <dada1@cosmosbay.com>
Date: Sat, 14 Mar 2009 10:03:54 +0100

> On x86_64, its rather unfortunate that "wait_queue_head_t wait"
> field of "struct socket" spans two cache lines (assuming a 64
> bytes cache line in current cpus)
> 
> offsetof(struct socket, wait)=0x30
> sizeof(wait_queue_head_t)=0x18
> 
> This might explain why Kenny Chang noticed that his multicast workload
> was performing bad with 64 bit kernels, since more cache lines ping pongs
> were involved.
> 
> This litle patch moves "wait" field next "fasync_list" so that both
> fields share a single cache line, to speedup sock_def_readable()
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>

Applied, thanks a lot Eric.

Re: Multicast packet loss

[Eric Dumazet]

David Miller a écrit :
> From: Eric Dumazet <dada1@cosmosbay.com>
> Date: Fri, 13 Mar 2009 23:30:31 +0100
> 
>> David Miller a écrit :
>>>> Also, when an event was queued for later invocation, I also needed to keep
>>>> a reference on "struct socket" to make sure it doesnt disappear before
>>>> the invocation. Not all sockets are RCU guarded (we added RCU only for 
>>>> some protocols (TCP, UDP ...). So I found keeping a read_lock
>>>> on callback was the easyest thing to do. I now realize we might
>>>> overflow preempt_count, so special care is needed.
>>> You're using this in UDP so... make the rule that you can't use
>>> this with a non-RCU-quiescent protocol.
>> UDP/TCP only ? I though many other protocols (not all using RCU) were
>> using sock_def_readable() too...
> 
> Maybe create a inet_def_readable() just for this purpose :-)


Here is the last incantation of the patch, that of course should be
split in two parts and better Changelog for further discussion on lkml.

We need to take a reference on sock when queued on a softirq delay
list. RCU wont help here because of SLAB_DESTROY_BY_RCU thing :
Another cpu could free/reuse the socket before we have a chance to
call softirq_delay_exec()

UDP & UDPLite use this delayed wakeup feature.

Thank you

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example,
each received multicast frame can wakeup many threads. ksoftirqd is then
not able to drain NIC RX queues in time and we get frame losses and high
latencies.

This patch adds an infrastructure to delay work done in
sock_def_readable() at end of do_softirq(). This needs to
make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS


Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
---
 include/linux/interrupt.h |   18 +++++++++++++++
 include/linux/irqflags.h  |   11 ++++-----
 include/linux/sched.h     |    2 -
 include/net/sock.h        |    2 +
 include/net/udplite.h     |    1
 kernel/lockdep.c          |    2 -
 kernel/softirq.c          |   42 ++++++++++++++++++++++++++++++++++--
 lib/locking-selftest.c    |    4 +--
 net/core/sock.c           |   41 +++++++++++++++++++++++++++++++++++
 net/ipv4/udp.c            |    7 ++++++
 net/ipv6/udp.c            |    7 ++++++
 11 files changed, 125 insertions(+), 12 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 9127f6b..a773d0c 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h
index 74bde13..30c1e01 100644
--- a/include/linux/irqflags.h
+++ b/include/linux/irqflags.h
@@ -13,19 +13,21 @@
 
 #include <linux/typecheck.h>
 
+#define softirq_enter()	do { current->softirq_context++; } while (0)
+#define softirq_exit()	do { current->softirq_context--; } while (0)
+#define softirq_context(p)	((p)->softirq_context)
+#define running_from_softirq()  (softirq_context(current) > 0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
   extern void trace_hardirqs_on(void);
   extern void trace_hardirqs_off(void);
 # define trace_hardirq_context(p)	((p)->hardirq_context)
-# define trace_softirq_context(p)	((p)->softirq_context)
 # define trace_hardirqs_enabled(p)	((p)->hardirqs_enabled)
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -33,13 +35,10 @@
 # define trace_softirqs_on(ip)		do { } while (0)
 # define trace_softirqs_off(ip)		do { } while (0)
 # define trace_hardirq_context(p)	0
-# define trace_softirq_context(p)	0
 # define trace_hardirqs_enabled(p)	0
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif
 
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8c216e0..5dd8487 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1320,8 +1320,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 48UL
 	u64 curr_chain_key;
diff --git a/include/net/sock.h b/include/net/sock.h
index 4bb1ff9..0160a83 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
@@ -960,6 +961,7 @@ extern void *sock_kmalloc(struct sock *sk, int size,
 			  gfp_t priority);
 extern void sock_kfree_s(struct sock *sk, void *mem, int size);
 extern void sk_send_sigurg(struct sock *sk);
+extern void inet_def_readable(struct sock *sk, int len);
 
 /*
  * Functions to fill in entries in struct proto_ops when a protocol
diff --git a/include/net/udplite.h b/include/net/udplite.h
index afdffe6..7ce0ee0 100644
--- a/include/net/udplite.h
+++ b/include/net/udplite.h
@@ -25,6 +25,7 @@ static __inline__ int udplite_getfrag(void *from, char *to, int  offset,
 /* Designate sk as UDP-Lite socket */
 static inline int udplite_sk_init(struct sock *sk)
 {
+	sk->sk_data_ready = inet_def_readable;
 	udp_sk(sk)->pcflag = UDPLITE_BIT;
 	return 0;
 }
diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index 06b0c35..9873b40 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -1807,7 +1807,7 @@ print_usage_bug(struct task_struct *curr, struct held_lock *this,
 	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
 		curr->comm, task_pid_nr(curr),
 		trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
-		trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
+		softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
 		trace_hardirqs_enabled(curr),
 		trace_softirqs_enabled(curr));
 	print_lock(this);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index bdbe9de..91a1714 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (!sdel->next) {
+		sdel->next = __get_cpu_var(softirq_delay_head);
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Because locking is provided by subsystem, please note
+ * that sdel->func(sdel) is responsible for setting sdel->next to NULL
+ */
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->func(sdel);	/*	sdel->next = NULL;*/
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -180,7 +216,7 @@ asmlinkage void __do_softirq(void)
 	account_system_vtime(current);
 
 	__local_bh_disable((unsigned long)__builtin_return_address(0));
-	trace_softirq_enter();
+	softirq_enter();
 
 	cpu = smp_processor_id();
 restart:
@@ -211,6 +247,8 @@ restart:
 		pending >>= 1;
 	} while (pending);
 
+	softirq_delay_exec();
+
 	local_irq_disable();
 
 	pending = local_softirq_pending();
@@ -220,7 +258,7 @@ restart:
 	if (pending)
 		wakeup_softirqd();
 
-	trace_softirq_exit();
+	softirq_exit();
 
 	account_system_vtime(current);
 	_local_bh_enable();
diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c
index 280332c..1aa7351 100644
--- a/lib/locking-selftest.c
+++ b/lib/locking-selftest.c
@@ -157,11 +157,11 @@ static void init_shared_classes(void)
 #define SOFTIRQ_ENTER()				\
 		local_bh_disable();		\
 		local_irq_disable();		\
-		trace_softirq_enter();		\
+		softirq_enter();		\
 		WARN_ON(!in_softirq());
 
 #define SOFTIRQ_EXIT()				\
-		trace_softirq_exit();		\
+		softirq_exit();		\
 		local_irq_enable();		\
 		local_bh_enable();
 
diff --git a/net/core/sock.c b/net/core/sock.c
index 0620046..c8745d1 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -213,6 +213,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
 /* Maximal space eaten by iovec or ancilliary data plus some space */
 int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
 
+static void sock_readable_defer(struct softirq_delay *sdel);
+
 static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
 {
 	struct timeval tv;
@@ -1074,6 +1076,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+/*
+ * helper function called by softirq_delay_exec(),
+ * if inet_def_readable() queued us.
+ */
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();
+	sock_def_readable(sk, 0);
+	sock_put(sk);
+}
+
+/*
+ * Custom version of sock_def_readable()
+ * We want to defer scheduler processing at the end of do_softirq()
+ * Called with socket locked.
+ */
+void inet_def_readable(struct sock *sk, int len)
+{
+	if (running_from_softirq()) {
+		if (softirq_delay_queue(&sk->sk_delay))
+			/*
+			 * If we queued this socket, take a reference on it
+			 * Caller owns socket lock, so write to sk_delay.next
+			 * will be committed before unlock.
+			 */
+			sock_hold(sk);
+	} else
+		sock_def_readable(sk, len);
+}
+
 static void sock_def_write_space(struct sock *sk)
 {
 	read_lock(&sk->sk_callback_lock);
@@ -1768,6 +1808,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 05b7abb..1cc0907 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1342,6 +1342,12 @@ void udp_destroy_sock(struct sock *sk)
 	release_sock(sk);
 }
 
+static int udp_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1559,6 +1565,7 @@ struct proto udp_prot = {
 	.connect	   = ip4_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init		   = udp_init_sock,
 	.destroy	   = udp_destroy_sock,
 	.setsockopt	   = udp_setsockopt,
 	.getsockopt	   = udp_getsockopt,
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 84b1a29..1a9f8d4 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -960,6 +960,12 @@ void udpv6_destroy_sock(struct sock *sk)
 	inet6_destroy_sock(sk);
 }
 
+static int udpv6_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1084,6 +1090,7 @@ struct proto udpv6_prot = {
 	.connect	   = ip6_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init 		   = udpv6_init_sock,
 	.destroy	   = udpv6_destroy_sock,
 	.setsockopt	   = udpv6_setsockopt,
 	.getsockopt	   = udpv6_getsockopt,

Re: Multicast packet loss

[Peter Zijlstra]

On Mon, 2009-03-16 at 23:22 +0100, Eric Dumazet wrote:

> Here is the last incantation of the patch, that of course should be
> split in two parts and better Changelog for further discussion on lkml.

I read the entire thread up to now, and I still don't really understand
the Changelog, sorry :(

> [PATCH] softirq: Introduce mechanism to defer wakeups
> 
> Some network workloads need to call scheduler too many times. For example,
> each received multicast frame can wakeup many threads. ksoftirqd is then
> not able to drain NIC RX queues in time and we get frame losses and high
> latencies.
> 
> This patch adds an infrastructure to delay work done in
> sock_def_readable() at end of do_softirq(). This needs to
> make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS

How does that solve the wakeup issue?

> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
> ---

> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
>  }
>  EXPORT_SYMBOL(local_bh_enable_ip);
>  
> +
> +#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
> +static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
> +	SOFTIRQ_DELAY_END
> +};

Why the magic termination value? Can't we NULL terminate the list

> +
> +/*
> + * Caller must disable preemption, and take care of appropriate
> + * locking and refcounting
> + */

Shouldn't we call it __softirq_delay_queue() if the caller needs to
disabled preemption?

Futhermore, don't we always require the caller to take care of lifetime
issues when we queue something?

> +int softirq_delay_queue(struct softirq_delay *sdel)
> +{
> +	if (!sdel->next) {
> +		sdel->next = __get_cpu_var(softirq_delay_head);
> +		__get_cpu_var(softirq_delay_head) = sdel;
> +		return 1;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Because locking is provided by subsystem, please note
> + * that sdel->func(sdel) is responsible for setting sdel->next to NULL
> + */
> +static void softirq_delay_exec(void)
> +{
> +	struct softirq_delay *sdel;
> +
> +	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
> +		__get_cpu_var(softirq_delay_head) = sdel->next;
> +		sdel->func(sdel);	/*	sdel->next = NULL;*/
> +		}
> +}

Why can't we write:

  struct softirq_delay *sdel, *next;

  sdel = __get_cpu_var(softirq_delay_head);
  __get_cpu_var(softirq_delay_head) = NULL;

  while (sdel) {
    next = sdel->next;
    sdel->func(sdel);
    sdel = next;
  }

Why does it matter what happens to sdel->next? we've done the callback.

Aah, the crux is in the re-use policy.. that most certainly does deserve
a comment.

How about we make sdel->next point to itself in the init case?

Then we can write:

  while (sdel) {
    next = sdel->next;
    sdel->next = sdel;
    sdel->func(sdel);
    sdel = next;
  }

and have the enqueue bit look like:

int __softirq_delay_queue(struct softirq_delay *sdel)
{
  struct softirq_delay **head;

  if (sdel->next != sdel)
    return 0;

  head = &__get_cpu_var(softirq_delay_head);
  sdel->next = *head;
  *head = sdel;
  return 1;
}
     
> @@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
>  	read_unlock(&sk->sk_callback_lock);
>  }
>  
> +/*
> + * helper function called by softirq_delay_exec(),
> + * if inet_def_readable() queued us.
> + */
> +static void sock_readable_defer(struct softirq_delay *sdel)
> +{
> +	struct sock *sk = container_of(sdel, struct sock, sk_delay);
> +
> +	sdel->next = NULL;
> +	/*
> +	 * At this point, we dont own a lock on socket, only a reference.
> +	 * We must commit above write, or another cpu could miss a wakeup
> +	 */
> +	smp_wmb();

Where's the matching barrier?

> +	sock_def_readable(sk, 0);
> +	sock_put(sk);
> +}
> +
> +/*
> + * Custom version of sock_def_readable()
> + * We want to defer scheduler processing at the end of do_softirq()
> + * Called with socket locked.
> + */
> +void inet_def_readable(struct sock *sk, int len)
> +{
> +	if (running_from_softirq()) {
> +		if (softirq_delay_queue(&sk->sk_delay))
> +			/*
> +			 * If we queued this socket, take a reference on it
> +			 * Caller owns socket lock, so write to sk_delay.next
> +			 * will be committed before unlock.
> +			 */
> +			sock_hold(sk);
> +	} else
> +		sock_def_readable(sk, len);
> +}

OK, so the idea is to handle a bunch of packets and instead of waking N
threads for each packet, only wake them once at the end of the batch?

Sounds like a sensible idea.. 

Re: Multicast packet loss

[Eric Dumazet]

Peter Zijlstra a écrit :
> On Mon, 2009-03-16 at 23:22 +0100, Eric Dumazet wrote:
> 
>> Here is the last incantation of the patch, that of course should be
>> split in two parts and better Changelog for further discussion on lkml.
> 
> I read the entire thread up to now, and I still don't really understand
> the Changelog, sorry :(

Sure, I should have taken more time, will repost this in a couple of hours,
with nice CHangelogs and split patches.

> 
>> [PATCH] softirq: Introduce mechanism to defer wakeups
>>
>> Some network workloads need to call scheduler too many times. For example,
>> each received multicast frame can wakeup many threads. ksoftirqd is then
>> not able to drain NIC RX queues in time and we get frame losses and high
>> latencies.
>>
>> This patch adds an infrastructure to delay work done in
>> sock_def_readable() at end of do_softirq(). This needs to
>> make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS
> 
> How does that solve the wakeup issue?

Apparently, on SMP machines this actually helps a lot, in case of multicast
trafic handled by many subscribers. skb_cloning involves atomic ops on
route cache entries, and if we wakeup threads as we currently do, they
start to consume skb while the feeder is still doing skb clones for
other sockets. Many cache line ping pongs are slowing down the softirq.

I will post the test program to reproduce the problem.

> 
>> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
>> ---
> 
>> --- a/kernel/softirq.c
>> +++ b/kernel/softirq.c
>> @@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
>>  }
>>  EXPORT_SYMBOL(local_bh_enable_ip);
>>  
>> +
>> +#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
>> +static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
>> +	SOFTIRQ_DELAY_END
>> +};
> 
> Why the magic termination value? Can't we NULL terminate the list

Yes we can, you are right.

> 
>> +
>> +/*
>> + * Caller must disable preemption, and take care of appropriate
>> + * locking and refcounting
>> + */
> 
> Shouldn't we call it __softirq_delay_queue() if the caller needs to
> disabled preemption?

I was wondering if some BUG_ON() can be added to crash if preemption is enabled
at this point. Could not find an existing check,
doing again the 'if (running_from_softirq())'" test might be overkill,
should I document caller should do :

skeleton :

    lock_my_data(data); /* barrier here */
    sdel = &data->sdel;
    if (running_from_softirq()) {
	if (softirq_delay_queue(sdel)) {
		hold a refcount on data;
	} else {
		/* already queued, nothing to do */
	}
    } else {
	/* cannot queue the work , must do it right now */
	do_work(data);
    }
    release_my_data(data);
}

> 
> Futhermore, don't we always require the caller to take care of lifetime
> issues when we queue something?

You mean comment is too verbose... or 

> 
>> +int softirq_delay_queue(struct softirq_delay *sdel)
>> +{
>> +	if (!sdel->next) {
>> +		sdel->next = __get_cpu_var(softirq_delay_head);
>> +		__get_cpu_var(softirq_delay_head) = sdel;
>> +		return 1;
>> +	}
>> +	return 0;
>> +}
>> +
>> +/*
>> + * Because locking is provided by subsystem, please note
>> + * that sdel->func(sdel) is responsible for setting sdel->next to NULL
>> + */
>> +static void softirq_delay_exec(void)
>> +{
>> +	struct softirq_delay *sdel;
>> +
>> +	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
>> +		__get_cpu_var(softirq_delay_head) = sdel->next;
>> +		sdel->func(sdel);	/*	sdel->next = NULL;*/
>> +		}
>> +}
> 
> Why can't we write:
> 
>   struct softirq_delay *sdel, *next;
> 
>   sdel = __get_cpu_var(softirq_delay_head);
>   __get_cpu_var(softirq_delay_head) = NULL;
> 
>   while (sdel) {
>     next = sdel->next;
>     sdel->func(sdel);
>     sdel = next;
>   }
> 
> Why does it matter what happens to sdel->next? we've done the callback.
> 
> Aah, the crux is in the re-use policy.. that most certainly does deserve
> a comment.

Hum, so my comment was not verbose enough :)

> 
> How about we make sdel->next point to itself in the init case?
> 
> Then we can write:
> 
>   while (sdel) {
>     next = sdel->next;
>     sdel->next = sdel;
>     sdel->func(sdel);
>     sdel = next;
>   }
> 
> and have the enqueue bit look like:
> 
> int __softirq_delay_queue(struct softirq_delay *sdel)
> {
>   struct softirq_delay **head;
> 
>   if (sdel->next != sdel)
>     return 0;

Yes we could do that

> 
>   head = &__get_cpu_var(softirq_delay_head);
>   sdel->next = *head;
>   *head = sdel;
>   return 1;
> }
>      
>> @@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
>>  	read_unlock(&sk->sk_callback_lock);
>>  }
>>  
>> +/*
>> + * helper function called by softirq_delay_exec(),
>> + * if inet_def_readable() queued us.
>> + */
>> +static void sock_readable_defer(struct softirq_delay *sdel)
>> +{
>> +	struct sock *sk = container_of(sdel, struct sock, sk_delay);
>> +
>> +	sdel->next = NULL;
>> +	/*
>> +	 * At this point, we dont own a lock on socket, only a reference.
>> +	 * We must commit above write, or another cpu could miss a wakeup
>> +	 */
>> +	smp_wmb();
> 
> Where's the matching barrier?

Check softirq_delay_exec(void) comment, where I stated synchronization had
to be done by the subsystem.

In this socket case, caller of softirq_delay_exec() has a lock on socket.

Problem is I dont want to get this lock again in sock_readable_defer() callback

if sdel->next is not committed, another cpu could call _softirq_delay_queue() and
find sdel->next being not null (or != sdel with your suggestion). Then next->func()
wont be called as it should (or called litle bit too soon)

So matching barrier is on "lock_my_data(data)" in previous skeleton ?

> 
>> +	sock_def_readable(sk, 0);
>> +	sock_put(sk);
>> +}
>> +
>> +/*
>> + * Custom version of sock_def_readable()
>> + * We want to defer scheduler processing at the end of do_softirq()
>> + * Called with socket locked.
>> + */
>> +void inet_def_readable(struct sock *sk, int len)
>> +{
>> +	if (running_from_softirq()) {
>> +		if (softirq_delay_queue(&sk->sk_delay))
>> +			/*
>> +			 * If we queued this socket, take a reference on it
>> +			 * Caller owns socket lock, so write to sk_delay.next
>> +			 * will be committed before unlock.
>> +			 */
>> +			sock_hold(sk);
>> +	} else
>> +		sock_def_readable(sk, len);
>> +}
> 
> OK, so the idea is to handle a bunch of packets and instead of waking N
> threads for each packet, only wake them once at the end of the batch?
> 
> Sounds like a sensible idea.. 

Idea is to batch wakeups() yes, and if we receive several packets for
the same socket(s), we reduce number of wakeups to one. In the multicast stress
situation of Athena CR, it really helps, no packets dropped instead of
30%

Thanks Peter

Re: Multicast packet loss

[Peter Zijlstra]

On Tue, 2009-03-17 at 12:08 +0100, Eric Dumazet wrote:

> >> +
> >> +/*
> >> + * Caller must disable preemption, and take care of appropriate
> >> + * locking and refcounting
> >> + */
> > 
> > Shouldn't we call it __softirq_delay_queue() if the caller needs to
> > disabled preemption?
> 
> I was wondering if some BUG_ON() can be added to crash if preemption is enabled
> at this point.

__get_cpu_var() has a preemption check and will generate BUGs when
CONFIG_DEBUG_PREEMPT similar to smp_processor_id().

>  Could not find an existing check,
> doing again the 'if (running_from_softirq())'" test might be overkill,
> should I document caller should do :
> 
> skeleton :
> 
>     lock_my_data(data); /* barrier here */
>     sdel = &data->sdel;
>     if (running_from_softirq()) {

Small nit: I don't particularly like the running_from_softirq() name,
but in_softirq() is already taken, and sadly means something slightly
different.

> 	if (softirq_delay_queue(sdel)) {
> 		hold a refcount on data;
> 	} else {
> 		/* already queued, nothing to do */
> 	}
>     } else {
> 	/* cannot queue the work , must do it right now */
> 	do_work(data);
>     }
>     release_my_data(data);
> }
> 
> > 
> > Futhermore, don't we always require the caller to take care of lifetime
> > issues when we queue something?
> 
> You mean comment is too verbose... or 

Yeah.

> > Aah, the crux is in the re-use policy.. that most certainly does deserve
> > a comment.
> 
> Hum, so my comment was not verbose enough :)

That too :-) 

> >> +static void sock_readable_defer(struct softirq_delay *sdel)
> >> +{
> >> +	struct sock *sk = container_of(sdel, struct sock, sk_delay);
> >> +
> >> +	sdel->next = NULL;
> >> +	/*
> >> +	 * At this point, we dont own a lock on socket, only a reference.
> >> +	 * We must commit above write, or another cpu could miss a wakeup
> >> +	 */
> >> +	smp_wmb();
> > 
> > Where's the matching barrier?
> 
> Check softirq_delay_exec(void) comment, where I stated synchronization had
> to be done by the subsystem.

afaiu the memory barrier semantics you cannot pair a wmb with a lock
barrier, it must either be a read, read_barrier_depends or full barrier.

> In this socket case, caller of softirq_delay_exec() has a lock on socket.
> 
> Problem is I dont want to get this lock again in sock_readable_defer() callback
> 
> if sdel->next is not committed, another cpu could call _softirq_delay_queue() and
> find sdel->next being not null (or != sdel with your suggestion). Then next->func()
> wont be called as it should (or called litle bit too soon)

Right, what we can do is put the wmb in the callback and the rmb right
before the __queue op, or simply integrate it into the framework.

> > OK, so the idea is to handle a bunch of packets and instead of waking N
> > threads for each packet, only wake them once at the end of the batch?
> > 
> > Sounds like a sensible idea.. 
> 
> Idea is to batch wakeups() yes, and if we receive several packets for
> the same socket(s), we reduce number of wakeups to one. In the multicast stress
> situation of Athena CR, it really helps, no packets dropped instead of
> 30%

Yes I can see that helping tremendously.

Re: Multicast packet loss

[Brian Bloniarz]

Eric Dumazet wrote:
> Sure, I should have taken more time, will repost this in a couple of hours,
> with nice CHangelogs and split patches.

One small thing: with CONFIG_IPV6=m, inet_def_readable needs to be exported,
right?

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Eric Dumazet wrote:
>> Sure, I should have taken more time, will repost this in a couple of
>> hours,
>> with nice CHangelogs and split patches.
> 
> One small thing: with CONFIG_IPV6=m, inet_def_readable needs to be
> exported,
> right?
> 

Absolutely, thank you !

Re: Multicast packet loss

[David Stevens]

I did some testing with this and see at least a 20% improvement
without drop.

I agree with Peter's recommended changes (esp. sentinel vs null),
and also the trivial brace indentation  in softirq_delay_exec(),
but otherwise looks  good to me. Nice work.

                                        +-DLS

Re: Multicast packet loss

[Eric Dumazet]

David Stevens a écrit :
> I did some testing with this and see at least a 20% improvement
> without drop.
> 
> I agree with Peter's recommended changes (esp. sentinel vs null),
> and also the trivial brace indentation  in softirq_delay_exec(),
> but otherwise looks  good to me. Nice work.
> 
>                                         +-DLS
> 
> 

Still I dont like very much all softirq.c changes. I feel very
uncomfortable to justify one extra call in do_softirq(), and
not very clean interface (stuff about locking, barriers...)

Easy way could be to add a SOFTIRQ but its not very wise.

I was wondering if we could use the infrastructure added in commit
54514a70adefe356afe854e2d3912d46668068e6
(softirq: Add support for triggering softirq work on softirqs.)
But I dont understand how it can works...
(softirq_work_list is feeded, but never processed)

Alternatively, we could use a framework dedicated to
network use, with well defined semantic :

Calling softirq_delay_exec() from net_rx_action(),
from this function, we know if time_squeeze was incremented,
or all netdev_budget consumed, and in this stress case 
try to give the wakeups job to another cpu.

Re: Multicast packet loss

[Brian Bloniarz]

Hi Eric,

We've been experimenting with this softirq-delay patch in production, and
have seen some hard-to-reproduce crashes. We finally managed to capture a
kexec crashdump this morning.

This is the dmesg:

[53417.592868] Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
[53417.598377]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.606300] PGD 32abb8067 PUD 32faf5067 PMD 0
[53417.610829] Oops: 0000 [1] SMP
[53417.614032] CPU 2
[53417.616083] Modules linked in: nfs lockd nfs_acl sunrpc openafs(P) autofs4 ipv6 ac sbs sbshc video output dock battery container iptable_filter ip_tables x_tables parport_pc lp parport loop joydev iTCO_wdt iTCO_vendor_support evdev button i5000_edac psmouse serio_raw pcspkr shpchp pci_hotplug edac_core ext3 jbd mbcache sr_mod cdrom ata_generic usbhid hid ata_piix sg sd_mod ehci_hcd pata_acpi uhci_hcd libata bnx2 aacraid usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
[53417.662067] Pid: 13039, comm: gball Tainted: P        2.6.24-19acr2-generic #1
[53417.669219] RIP: 0010:[<ffffffff80243643>]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.677368] RSP: 0018:ffff8103314f3f20  EFLAGS: 00010297
[53417.682697] RAX: ffff810084a1b000 RBX: ffffffff805ba530 RCX: 0000000000000000
[53417.689843] RDX: ffff8103305811e0 RSI: 0000000000000282 RDI: ffff810332ada580
[53417.696993] RBP: 0000000000000000 R08: ffff81032fad9f08 R09: ffff810332382000
[53417.704144] R10: 0000000000000000 R11: ffffffff80316ec0 R12: ffffffff8062b3d8
[53417.711294] R13: ffffffff8062b480 R14: 0000000000000002 R15: 000000000000000a
[53417.718447] FS:  00007fab0d7b8750(0000) GS:ffff810334401b80(0000) knlGS:0000000000000000
[53417.726568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[53417.732332] CR2: 0000000000000000 CR3: 0000000329e2d000 CR4: 00000000000006e0
[53417.739476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[53417.746637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[53417.753787] Process gball (pid: 13039, threadinfo ffff81032adde000, task ffff810329ff77d0)
[53417.761991] Stack:  ffffffff8062b3d8 0000000000000046 ffff8103314f3f68 0000000000000000
[53417.770146]  00000000000000a0 ffff81032addfee8 0000000000000000 ffffffff8020d50c
[53417.777660]  ffff8103314f3f68 00000000000000c1 ffffffff8020ed25 ffffffff8062c870
[53417.784961] Call Trace:
[53417.787635]  <IRQ>  [<ffffffff8020d50c>] call_softirq+0x1c/0x30
[53417.793597]  [<ffffffff8020ed25>] do_softirq+0x35/0x90
[53417.798747]  [<ffffffff80243578>] irq_exit+0x88/0x90
[53417.803727]  [<ffffffff8020ef70>] do_IRQ+0x80/0x100
[53417.808624]  [<ffffffff8020c891>] ret_from_intr+0x0/0xa
[53417.813862]  <EOI>  [<ffffffff803e53c8>] skb_release_all+0x18/0x150
[53417.820164]  [<ffffffff803e4ad9>] __kfree_skb+0x9/0x90
[53417.825327]  [<ffffffff80437612>] udp_recvmsg+0x222/0x260
[53417.830744]  [<ffffffff80231264>] source_load+0x34/0x70
[53417.835984]  [<ffffffff80232a9a>] find_busiest_group+0x1fa/0x850
[53417.842019]  [<ffffffff803e0100>] sock_common_recvmsg+0x30/0x50
[53417.847958]  [<ffffffff803de1ca>] sock_recvmsg+0x14a/0x160
[53417.853462]  [<ffffffff80231c21>] update_curr+0x71/0x100
[53419.858789]  [<ffffffff802320fd>] __dequeue_entity+0x3d/0x50
[53417.864469]  [<ffffffff80253ab0>] autoremove_wake_function+0x0/0x30
[53417.870758]  [<ffffffff8046662f>] thread_return+0x3a/0x57b
[53417.876262]  [<ffffffff803df73e>] sys_recvfrom+0xfe/0x190
[53417.881680]  [<ffffffff802e2a95>] sys_epoll_wait+0x245/0x4e0
[53417.887358]  [<ffffffff80233e20>] default_wake_function+0x0/0x10
[53417.893384]  [<ffffffff8020c37e>] system_call+0x7e/0x83
[53417.898628]
[53417.900134]
[53417.900134] Code: 48 8b 11 48 89 cf 65 48 8b 04 25 08 00 00 00 4a 89 14 20 ff
[53417.909430] RIP  [<ffffffff80243643>] __do_softirq+0xc3/0x150
[53417.915210]  RSP <ffff8103314f3f20>

The disassembly where it crashed:
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024361b:       d1 ed                   shr    %ebp
rcu_bh_qsctr_inc():
/local/home/bmb/doc/kernels/linux-hardy-eric/include/linux/rcupdate.h:130
ffffffff8024361d:       48 8b 40 08             mov    0x8(%rax),%rax
ffffffff80243621:       41 c7 44 05 08 01 00    movl   $0x1,0x8(%r13,%rax,1)
ffffffff80243628:       00 00
__do_softirq():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
ffffffff8024362a:       75 d8                   jne    ffffffff80243604 <__do_softirq+0x84>
softirq_delay_exec():
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
ffffffff8024362c:       48 8b 14 24             mov    (%rsp),%rdx
ffffffff80243630:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243637:       00 00
ffffffff80243639:       48 8b 0c 10             mov    (%rax,%rdx,1),%rcx
ffffffff8024363d:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff80243641:       74 29                   je     ffffffff8024366c <__do_softirq+0xec>
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
ffffffff80243643:       48 8b 11                mov    (%rcx),%rdx
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243646:       48 89 cf                mov    %rcx,%rdi
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
ffffffff80243649:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243650:       00 00
ffffffff80243652:       4a 89 14 20             mov    %rdx,(%rax,%r12,1)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
ffffffff80243656:       ff 51 08                callq  *0x8(%rcx)
/local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
ffffffff80243659:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
ffffffff80243660:       00 00
ffffffff80243662:       4a 8b 0c 20             mov    (%rax,%r12,1),%rcx
ffffffff80243666:       48 83 f9 01             cmp    $0x1,%rcx
ffffffff8024366a:       75 d7                   jne    ffffffff80243643 <__do_softirq+0xc3>
raw_local_irq_disable():
/local/home/bmb/doc/kernels/linux-hardy-eric/debian/build/build-generic/include2/asm/irqflags_64.h:76
ffffffff8024366c:       fa                      cli

And softirq.c line numbers:
    218   * Because locking is provided by subsystem, please note
    219   * that sdel->func(sdel) is responsible for setting sdel->next to NULL
    220   */
    221  static void softirq_delay_exec(void)
    222  {
    223          struct softirq_delay *sdel;
    224
    225          while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
    226                  __get_cpu_var(softirq_delay_head) = sdel->next;
    227                  sdel->func(sdel);       /*      sdel->next = NULL;*/
    228                  }
    229  }

So it's crashing because __get_cpu_var(softirq_delay_head)) is NULL somehow.

We aren't running a recent kernel -- we're running Ubuntu Hardy's 2.6.24-19,
with a backported version of this patch. One more atypical thing is that
we run openafs, 1.4.6.dfsg1-2.

Like I said, I have a full vmcore (3, actually) and would be happy to post any
more information you'd like to know.

Thanks,
Brian Bloniarz

Eric Dumazet wrote:
> David Miller a écrit :
>> From: Eric Dumazet <dada1@cosmosbay.com>
>> Date: Fri, 13 Mar 2009 23:30:31 +0100
>>
>>> David Miller a écrit :
>>>>> Also, when an event was queued for later invocation, I also needed to keep
>>>>> a reference on "struct socket" to make sure it doesnt disappear before
>>>>> the invocation. Not all sockets are RCU guarded (we added RCU only for 
>>>>> some protocols (TCP, UDP ...). So I found keeping a read_lock
>>>>> on callback was the easyest thing to do. I now realize we might
>>>>> overflow preempt_count, so special care is needed.
>>>> You're using this in UDP so... make the rule that you can't use
>>>> this with a non-RCU-quiescent protocol.
>>> UDP/TCP only ? I though many other protocols (not all using RCU) were
>>> using sock_def_readable() too...
>> Maybe create a inet_def_readable() just for this purpose :-)
> 
> 
> Here is the last incantation of the patch, that of course should be
> split in two parts and better Changelog for further discussion on lkml.
> 
> We need to take a reference on sock when queued on a softirq delay
> list. RCU wont help here because of SLAB_DESTROY_BY_RCU thing :
> Another cpu could free/reuse the socket before we have a chance to
> call softirq_delay_exec()
> 
> UDP & UDPLite use this delayed wakeup feature.
> 
> Thank you
> 
> [PATCH] softirq: Introduce mechanism to defer wakeups
> 
> Some network workloads need to call scheduler too many times. For example,
> each received multicast frame can wakeup many threads. ksoftirqd is then
> not able to drain NIC RX queues in time and we get frame losses and high
> latencies.
> 
> This patch adds an infrastructure to delay work done in
> sock_def_readable() at end of do_softirq(). This needs to
> make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS
> 
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
> ---
>  include/linux/interrupt.h |   18 +++++++++++++++
>  include/linux/irqflags.h  |   11 ++++-----
>  include/linux/sched.h     |    2 -
>  include/net/sock.h        |    2 +
>  include/net/udplite.h     |    1
>  kernel/lockdep.c          |    2 -
>  kernel/softirq.c          |   42 ++++++++++++++++++++++++++++++++++--
>  lib/locking-selftest.c    |    4 +--
>  net/core/sock.c           |   41 +++++++++++++++++++++++++++++++++++
>  net/ipv4/udp.c            |    7 ++++++
>  net/ipv6/udp.c            |    7 ++++++
>  11 files changed, 125 insertions(+), 12 deletions(-)
> 
> diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
> index 9127f6b..a773d0c 100644
> --- a/include/linux/interrupt.h
> +++ b/include/linux/interrupt.h
> @@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
>  extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
>  				  int this_cpu, int softirq);
>  
> +/*
> + * softirq delayed works : should be delayed at do_softirq() end
> + */
> +struct softirq_delay {
> +	struct softirq_delay	*next;
> +	void 			(*func)(struct softirq_delay *);
> +};
> +
> +int softirq_delay_queue(struct softirq_delay *sdel);
> +
> +static inline void softirq_delay_init(struct softirq_delay *sdel,
> +				      void (*func)(struct softirq_delay *))
> +{
> +	sdel->next = NULL;
> +	sdel->func = func;
> +}
> +
> +
>  /* Tasklets --- multithreaded analogue of BHs.
>  
>     Main feature differing them of generic softirqs: tasklet
> diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h
> index 74bde13..30c1e01 100644
> --- a/include/linux/irqflags.h
> +++ b/include/linux/irqflags.h
> @@ -13,19 +13,21 @@
>  
>  #include <linux/typecheck.h>
>  
> +#define softirq_enter()	do { current->softirq_context++; } while (0)
> +#define softirq_exit()	do { current->softirq_context--; } while (0)
> +#define softirq_context(p)	((p)->softirq_context)
> +#define running_from_softirq()  (softirq_context(current) > 0)
> +
>  #ifdef CONFIG_TRACE_IRQFLAGS
>    extern void trace_softirqs_on(unsigned long ip);
>    extern void trace_softirqs_off(unsigned long ip);
>    extern void trace_hardirqs_on(void);
>    extern void trace_hardirqs_off(void);
>  # define trace_hardirq_context(p)	((p)->hardirq_context)
> -# define trace_softirq_context(p)	((p)->softirq_context)
>  # define trace_hardirqs_enabled(p)	((p)->hardirqs_enabled)
>  # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
>  # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
>  # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
> -# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
> -# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
>  # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
>  #else
>  # define trace_hardirqs_on()		do { } while (0)
> @@ -33,13 +35,10 @@
>  # define trace_softirqs_on(ip)		do { } while (0)
>  # define trace_softirqs_off(ip)		do { } while (0)
>  # define trace_hardirq_context(p)	0
> -# define trace_softirq_context(p)	0
>  # define trace_hardirqs_enabled(p)	0
>  # define trace_softirqs_enabled(p)	0
>  # define trace_hardirq_enter()		do { } while (0)
>  # define trace_hardirq_exit()		do { } while (0)
> -# define trace_softirq_enter()		do { } while (0)
> -# define trace_softirq_exit()		do { } while (0)
>  # define INIT_TRACE_IRQFLAGS
>  #endif
>  
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 8c216e0..5dd8487 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -1320,8 +1320,8 @@ struct task_struct {
>  	unsigned long softirq_enable_ip;
>  	unsigned int softirq_enable_event;
>  	int hardirq_context;
> -	int softirq_context;
>  #endif
> +	int softirq_context;
>  #ifdef CONFIG_LOCKDEP
>  # define MAX_LOCK_DEPTH 48UL
>  	u64 curr_chain_key;
> diff --git a/include/net/sock.h b/include/net/sock.h
> index 4bb1ff9..0160a83 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -260,6 +260,7 @@ struct sock {
>  	unsigned long	        sk_lingertime;
>  	struct sk_buff_head	sk_error_queue;
>  	struct proto		*sk_prot_creator;
> +	struct softirq_delay	sk_delay;
>  	rwlock_t		sk_callback_lock;
>  	int			sk_err,
>  				sk_err_soft;
> @@ -960,6 +961,7 @@ extern void *sock_kmalloc(struct sock *sk, int size,
>  			  gfp_t priority);
>  extern void sock_kfree_s(struct sock *sk, void *mem, int size);
>  extern void sk_send_sigurg(struct sock *sk);
> +extern void inet_def_readable(struct sock *sk, int len);
>  
>  /*
>   * Functions to fill in entries in struct proto_ops when a protocol
> diff --git a/include/net/udplite.h b/include/net/udplite.h
> index afdffe6..7ce0ee0 100644
> --- a/include/net/udplite.h
> +++ b/include/net/udplite.h
> @@ -25,6 +25,7 @@ static __inline__ int udplite_getfrag(void *from, char *to, int  offset,
>  /* Designate sk as UDP-Lite socket */
>  static inline int udplite_sk_init(struct sock *sk)
>  {
> +	sk->sk_data_ready = inet_def_readable;
>  	udp_sk(sk)->pcflag = UDPLITE_BIT;
>  	return 0;
>  }
> diff --git a/kernel/lockdep.c b/kernel/lockdep.c
> index 06b0c35..9873b40 100644
> --- a/kernel/lockdep.c
> +++ b/kernel/lockdep.c
> @@ -1807,7 +1807,7 @@ print_usage_bug(struct task_struct *curr, struct held_lock *this,
>  	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
>  		curr->comm, task_pid_nr(curr),
>  		trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
> -		trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
> +		softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
>  		trace_hardirqs_enabled(curr),
>  		trace_softirqs_enabled(curr));
>  	print_lock(this);
> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index bdbe9de..91a1714 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -158,6 +158,42 @@ void local_bh_enable_ip(unsigned long ip)
>  }
>  EXPORT_SYMBOL(local_bh_enable_ip);
>  
> +
> +#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
> +static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
> +	SOFTIRQ_DELAY_END
> +};
> +
> +/*
> + * Caller must disable preemption, and take care of appropriate
> + * locking and refcounting
> + */
> +int softirq_delay_queue(struct softirq_delay *sdel)
> +{
> +	if (!sdel->next) {
> +		sdel->next = __get_cpu_var(softirq_delay_head);
> +		__get_cpu_var(softirq_delay_head) = sdel;
> +		return 1;
> +	}
> +	return 0;
> +}
> +
> +/*
> + * Because locking is provided by subsystem, please note
> + * that sdel->func(sdel) is responsible for setting sdel->next to NULL
> + */
> +static void softirq_delay_exec(void)
> +{
> +	struct softirq_delay *sdel;
> +
> +	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
> +		__get_cpu_var(softirq_delay_head) = sdel->next;
> +		sdel->func(sdel);	/*	sdel->next = NULL;*/
> +		}
> +}
> +
> +
> +
>  /*
>   * We restart softirq processing MAX_SOFTIRQ_RESTART times,
>   * and we fall back to softirqd after that.
> @@ -180,7 +216,7 @@ asmlinkage void __do_softirq(void)
>  	account_system_vtime(current);
>  
>  	__local_bh_disable((unsigned long)__builtin_return_address(0));
> -	trace_softirq_enter();
> +	softirq_enter();
>  
>  	cpu = smp_processor_id();
>  restart:
> @@ -211,6 +247,8 @@ restart:
>  		pending >>= 1;
>  	} while (pending);
>  
> +	softirq_delay_exec();
> +
>  	local_irq_disable();
>  
>  	pending = local_softirq_pending();
> @@ -220,7 +258,7 @@ restart:
>  	if (pending)
>  		wakeup_softirqd();
>  
> -	trace_softirq_exit();
> +	softirq_exit();
>  
>  	account_system_vtime(current);
>  	_local_bh_enable();
> diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c
> index 280332c..1aa7351 100644
> --- a/lib/locking-selftest.c
> +++ b/lib/locking-selftest.c
> @@ -157,11 +157,11 @@ static void init_shared_classes(void)
>  #define SOFTIRQ_ENTER()				\
>  		local_bh_disable();		\
>  		local_irq_disable();		\
> -		trace_softirq_enter();		\
> +		softirq_enter();		\
>  		WARN_ON(!in_softirq());
>  
>  #define SOFTIRQ_EXIT()				\
> -		trace_softirq_exit();		\
> +		softirq_exit();		\
>  		local_irq_enable();		\
>  		local_bh_enable();
>  
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 0620046..c8745d1 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -213,6 +213,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
>  /* Maximal space eaten by iovec or ancilliary data plus some space */
>  int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
>  
> +static void sock_readable_defer(struct softirq_delay *sdel);
> +
>  static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
>  {
>  	struct timeval tv;
> @@ -1074,6 +1076,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
>  #endif
>  
>  		rwlock_init(&newsk->sk_dst_lock);
> +		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
>  		rwlock_init(&newsk->sk_callback_lock);
>  		lockdep_set_class_and_name(&newsk->sk_callback_lock,
>  				af_callback_keys + newsk->sk_family,
> @@ -1691,6 +1694,43 @@ static void sock_def_readable(struct sock *sk, int len)
>  	read_unlock(&sk->sk_callback_lock);
>  }
>  
> +/*
> + * helper function called by softirq_delay_exec(),
> + * if inet_def_readable() queued us.
> + */
> +static void sock_readable_defer(struct softirq_delay *sdel)
> +{
> +	struct sock *sk = container_of(sdel, struct sock, sk_delay);
> +
> +	sdel->next = NULL;
> +	/*
> +	 * At this point, we dont own a lock on socket, only a reference.
> +	 * We must commit above write, or another cpu could miss a wakeup
> +	 */
> +	smp_wmb();
> +	sock_def_readable(sk, 0);
> +	sock_put(sk);
> +}
> +
> +/*
> + * Custom version of sock_def_readable()
> + * We want to defer scheduler processing at the end of do_softirq()
> + * Called with socket locked.
> + */
> +void inet_def_readable(struct sock *sk, int len)
> +{
> +	if (running_from_softirq()) {
> +		if (softirq_delay_queue(&sk->sk_delay))
> +			/*
> +			 * If we queued this socket, take a reference on it
> +			 * Caller owns socket lock, so write to sk_delay.next
> +			 * will be committed before unlock.
> +			 */
> +			sock_hold(sk);
> +	} else
> +		sock_def_readable(sk, len);
> +}
> +
>  static void sock_def_write_space(struct sock *sk)
>  {
>  	read_lock(&sk->sk_callback_lock);
> @@ -1768,6 +1808,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
>  		sk->sk_sleep	=	NULL;
>  
>  	rwlock_init(&sk->sk_dst_lock);
> +	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
>  	rwlock_init(&sk->sk_callback_lock);
>  	lockdep_set_class_and_name(&sk->sk_callback_lock,
>  			af_callback_keys + sk->sk_family,
> diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
> index 05b7abb..1cc0907 100644
> --- a/net/ipv4/udp.c
> +++ b/net/ipv4/udp.c
> @@ -1342,6 +1342,12 @@ void udp_destroy_sock(struct sock *sk)
>  	release_sock(sk);
>  }
>  
> +static int udp_init_sock(struct sock *sk)
> +{
> +	sk->sk_data_ready = inet_def_readable;
> +	return 0;
> +}
> +
>  /*
>   *	Socket option code for UDP
>   */
> @@ -1559,6 +1565,7 @@ struct proto udp_prot = {
>  	.connect	   = ip4_datagram_connect,
>  	.disconnect	   = udp_disconnect,
>  	.ioctl		   = udp_ioctl,
> +	.init		   = udp_init_sock,
>  	.destroy	   = udp_destroy_sock,
>  	.setsockopt	   = udp_setsockopt,
>  	.getsockopt	   = udp_getsockopt,
> diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
> index 84b1a29..1a9f8d4 100644
> --- a/net/ipv6/udp.c
> +++ b/net/ipv6/udp.c
> @@ -960,6 +960,12 @@ void udpv6_destroy_sock(struct sock *sk)
>  	inet6_destroy_sock(sk);
>  }
>  
> +static int udpv6_init_sock(struct sock *sk)
> +{
> +	sk->sk_data_ready = inet_def_readable;
> +	return 0;
> +}
> +
>  /*
>   *	Socket option code for UDP
>   */
> @@ -1084,6 +1090,7 @@ struct proto udpv6_prot = {
>  	.connect	   = ip6_datagram_connect,
>  	.disconnect	   = udp_disconnect,
>  	.ioctl		   = udp_ioctl,
> +	.init 		   = udpv6_init_sock,
>  	.destroy	   = udpv6_destroy_sock,
>  	.setsockopt	   = udpv6_setsockopt,
>  	.getsockopt	   = udpv6_getsockopt,
> 

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Hi Eric,
> 
> We've been experimenting with this softirq-delay patch in production, and
> have seen some hard-to-reproduce crashes. We finally managed to capture a
> kexec crashdump this morning.
> 
> This is the dmesg:
> 
> [53417.592868] Unable to handle kernel NULL pointer dereference at
> 0000000000000000 RIP:
> [53417.598377]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
> [53417.606300] PGD 32abb8067 PUD 32faf5067 PMD 0
> [53417.610829] Oops: 0000 [1] SMP
> [53417.614032] CPU 2
> [53417.616083] Modules linked in: nfs lockd nfs_acl sunrpc openafs(P)
> autofs4 ipv6 ac sbs sbshc video output dock battery container
> iptable_filter ip_tables x_tables parport_pc lp parport loop joydev
> iTCO_wdt iTCO_vendor_support evdev button i5000_edac psmouse serio_raw
> pcspkr shpchp pci_hotplug edac_core ext3 jbd mbcache sr_mod cdrom
> ata_generic usbhid hid ata_piix sg sd_mod ehci_hcd pata_acpi uhci_hcd
> libata bnx2 aacraid usbcore scsi_mod thermal processor fan fbcon
> tileblit font bitblit softcursor fuse
> [53417.662067] Pid: 13039, comm: gball Tainted: P       
> 2.6.24-19acr2-generic #1
> [53417.669219] RIP: 0010:[<ffffffff80243643>]  [<ffffffff80243643>]
> __do_softirq+0xc3/0x150
> [53417.677368] RSP: 0018:ffff8103314f3f20  EFLAGS: 00010297
> [53417.682697] RAX: ffff810084a1b000 RBX: ffffffff805ba530 RCX:
> 0000000000000000
> [53417.689843] RDX: ffff8103305811e0 RSI: 0000000000000282 RDI:
> ffff810332ada580
> [53417.696993] RBP: 0000000000000000 R08: ffff81032fad9f08 R09:
> ffff810332382000
> [53417.704144] R10: 0000000000000000 R11: ffffffff80316ec0 R12:
> ffffffff8062b3d8
> [53417.711294] R13: ffffffff8062b480 R14: 0000000000000002 R15:
> 000000000000000a
> [53417.718447] FS:  00007fab0d7b8750(0000) GS:ffff810334401b80(0000)
> knlGS:0000000000000000
> [53417.726568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [53417.732332] CR2: 0000000000000000 CR3: 0000000329e2d000 CR4:
> 00000000000006e0
> [53417.739476] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [53417.746637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [53417.753787] Process gball (pid: 13039, threadinfo ffff81032adde000,
> task ffff810329ff77d0)
> [53417.761991] Stack:  ffffffff8062b3d8 0000000000000046
> ffff8103314f3f68 0000000000000000
> [53417.770146]  00000000000000a0 ffff81032addfee8 0000000000000000
> ffffffff8020d50c
> [53417.777660]  ffff8103314f3f68 00000000000000c1 ffffffff8020ed25
> ffffffff8062c870
> [53417.784961] Call Trace:
> [53417.787635]  <IRQ>  [<ffffffff8020d50c>] call_softirq+0x1c/0x30
> [53417.793597]  [<ffffffff8020ed25>] do_softirq+0x35/0x90
> [53417.798747]  [<ffffffff80243578>] irq_exit+0x88/0x90
> [53417.803727]  [<ffffffff8020ef70>] do_IRQ+0x80/0x100
> [53417.808624]  [<ffffffff8020c891>] ret_from_intr+0x0/0xa
> [53417.813862]  <EOI>  [<ffffffff803e53c8>] skb_release_all+0x18/0x150
> [53417.820164]  [<ffffffff803e4ad9>] __kfree_skb+0x9/0x90
> [53417.825327]  [<ffffffff80437612>] udp_recvmsg+0x222/0x260
> [53417.830744]  [<ffffffff80231264>] source_load+0x34/0x70
> [53417.835984]  [<ffffffff80232a9a>] find_busiest_group+0x1fa/0x850
> [53417.842019]  [<ffffffff803e0100>] sock_common_recvmsg+0x30/0x50
> [53417.847958]  [<ffffffff803de1ca>] sock_recvmsg+0x14a/0x160
> [53417.853462]  [<ffffffff80231c21>] update_curr+0x71/0x100
> [53419.858789]  [<ffffffff802320fd>] __dequeue_entity+0x3d/0x50
> [53417.864469]  [<ffffffff80253ab0>] autoremove_wake_function+0x0/0x30
> [53417.870758]  [<ffffffff8046662f>] thread_return+0x3a/0x57b
> [53417.876262]  [<ffffffff803df73e>] sys_recvfrom+0xfe/0x190
> [53417.881680]  [<ffffffff802e2a95>] sys_epoll_wait+0x245/0x4e0
> [53417.887358]  [<ffffffff80233e20>] default_wake_function+0x0/0x10
> [53417.893384]  [<ffffffff8020c37e>] system_call+0x7e/0x83
> [53417.898628]
> [53417.900134]
> [53417.900134] Code: 48 8b 11 48 89 cf 65 48 8b 04 25 08 00 00 00 4a 89
> 14 20 ff
> [53417.909430] RIP  [<ffffffff80243643>] __do_softirq+0xc3/0x150
> [53417.915210]  RSP <ffff8103314f3f20>
> 
> The disassembly where it crashed:
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
> ffffffff8024361b:       d1 ed                   shr    %ebp
> rcu_bh_qsctr_inc():
> /local/home/bmb/doc/kernels/linux-hardy-eric/include/linux/rcupdate.h:130
> ffffffff8024361d:       48 8b 40 08             mov    0x8(%rax),%rax
> ffffffff80243621:       41 c7 44 05 08 01 00    movl  
> $0x1,0x8(%r13,%rax,1)
> ffffffff80243628:       00 00
> __do_softirq():
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
> ffffffff8024362a:       75 d8                   jne    ffffffff80243604
> <__do_softirq+0x84>
> softirq_delay_exec():
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
> ffffffff8024362c:       48 8b 14 24             mov    (%rsp),%rdx
> ffffffff80243630:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
> ffffffff80243637:       00 00
> ffffffff80243639:       48 8b 0c 10             mov    (%rax,%rdx,1),%rcx
> ffffffff8024363d:       48 83 f9 01             cmp    $0x1,%rcx
> ffffffff80243641:       74 29                   je     ffffffff8024366c
> <__do_softirq+0xec>
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
> ffffffff80243643:       48 8b 11                mov    (%rcx),%rdx
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
> ffffffff80243646:       48 89 cf                mov    %rcx,%rdi
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
> ffffffff80243649:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
> ffffffff80243650:       00 00
> ffffffff80243652:       4a 89 14 20             mov    %rdx,(%rax,%r12,1)
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
> ffffffff80243656:       ff 51 08                callq  *0x8(%rcx)
> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
> ffffffff80243659:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
> ffffffff80243660:       00 00
> ffffffff80243662:       4a 8b 0c 20             mov    (%rax,%r12,1),%rcx
> ffffffff80243666:       48 83 f9 01             cmp    $0x1,%rcx
> ffffffff8024366a:       75 d7                   jne    ffffffff80243643
> <__do_softirq+0xc3>
> raw_local_irq_disable():
> /local/home/bmb/doc/kernels/linux-hardy-eric/debian/build/build-generic/include2/asm/irqflags_64.h:76
> 
> ffffffff8024366c:       fa                      cli
> 
> And softirq.c line numbers:
>    218   * Because locking is provided by subsystem, please note
>    219   * that sdel->func(sdel) is responsible for setting sdel->next
> to NULL
>    220   */
>    221  static void softirq_delay_exec(void)
>    222  {
>    223          struct softirq_delay *sdel;
>    224
>    225          while ((sdel = __get_cpu_var(softirq_delay_head)) !=
> SOFTIRQ_DELAY_END) {
>    226                  __get_cpu_var(softirq_delay_head) = sdel->next;
>    227                  sdel->func(sdel);       /*      sdel->next =
> NULL;*/
>    228                  }
>    229  }
> 
> So it's crashing because __get_cpu_var(softirq_delay_head)) is NULL
> somehow.
> 
> We aren't running a recent kernel -- we're running Ubuntu Hardy's
> 2.6.24-19,
> with a backported version of this patch. One more atypical thing is that
> we run openafs, 1.4.6.dfsg1-2.
> 
> Like I said, I have a full vmcore (3, actually) and would be happy to
> post any
> more information you'd like to know.
> 
> Thanks,
> Brian Bloniarz

Hi Brian

2.6.24-19 kernel... hmm...

Could you please send me the diff of your backport against this kernel ?

I take you use Ubuntu Hardys 8.04 LTS server edition ?

Pointer being null might tell us that we managed to call inet_def_readable()
without socket lock hold...

Re: Multicast packet loss

[Brian Bloniarz]

Eric Dumazet wrote:
> Pointer being null might tell us that we managed to call inet_def_readable()
> without socket lock hold...

Trying to track this down: I added:
	BUG_ON(!spin_is_locked(&sk->sk_lock.slock));
to the top of inet_def_readable. This gives me the following panic:

[ 2528.745311] kernel BUG at net/core/sock.c:1674!
[ 2528.745311] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2528.745311] last sysfs file: /sys/devices/system/cpu/cpu7/crash_notes
[ 2528.745311] CPU 6
[ 2528.745311] Modules linked in: iptable_filter ip_tables x_tables parport_pc lp parport loop iTCO_wdt iTCO_vendor_support serio_raw psmouse pcspkr i5k_amb shpchp i5000_edac pci_hotplug button edac_core ipv6 ibmpex joydev ipmi_msghandler evdev ext3 jbd mbcache usbhid hid sr_mod cdrom pata_acpi ata_generic sg sd_mod ata_piix ehci_hcd uhci_hcd libata aacraid usbcore scsi_mod bnx2 thermal processor fan thermal_sys fuse
[ 2528.745311] Pid: 14507, comm: signalgen Not tainted 2.6.29.1-eric2-lowlat-lockdep #3 IBM System x3550 -[7978AC1]-
[ 2528.745311] RIP: 0010:[<ffffffff80444ec2>]  [<ffffffff80444ec2>] inet_def_readable+0x52/0x60
[ 2528.745311] RSP: 0018:ffff88043b985b58  EFLAGS: 00010246
[ 2528.745311] RAX: 0000000000000019 RBX: ffff88043b90c280 RCX: 0000000000000000
[ 2528.745311] RDX: 0000000000001919 RSI: 0000000000000068 RDI: ffff88043b90c280
[ 2528.745311] RBP: ffff88043b985b68 R08: 0000000000000000 R09: 0000000000000000
[ 2528.745311] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88043b811400
[ 2528.745311] R13: 0000000000000000 R14: 0000000000000068 R15: 0000000000000000
[ 2528.745311] FS:  00007f82f0742750(0000) GS:ffff88043dbc8280(0000) knlGS:0000000000000000
[ 2528.745311] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2528.745311] CR2: 000000000057f1a0 CR3: 000000043915e000 CR4: 00000000000406e0
[ 2528.745311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2528.745311] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2528.745311] Process signalgen (pid: 14507, threadinfo ffff88043b984000, task ffff8804309a9ef0)
[ 2528.745311] Stack:
[ 2528.745311]  ffff88043b811400 ffff88043b90c280 ffff88043b985b98 ffffffff80444ff6
[ 2528.745311]  ffff88043b90c280 ffff88043b811400 0000000000000000 ffff88043b90c2c0
[ 2528.745311]  ffff88043b985bc8 ffffffff8049ee67 ffff88043b985bc8 ffff88043b811400
[ 2528.745311] Call Trace:
[ 2528.745311]  [<ffffffff80444ff6>] sock_queue_rcv_skb+0xd6/0x120
[ 2528.745311]  [<ffffffff8049ee67>] __udp_queue_rcv_skb+0x27/0xe0
[ 2528.745311]  [<ffffffff8044406a>] release_sock+0x7a/0xe0
[ 2528.745311]  [<ffffffff804a1d0d>] udp_recvmsg+0x1ed/0x330
[ 2528.745311]  [<ffffffff804437e2>] sock_common_recvmsg+0x32/0x50
[ 2528.745311]  [<ffffffff80441449>] sock_recvmsg+0x139/0x150
[ 2528.745311]  [<ffffffff8025a590>] ? autoremove_wake_function+0x0/0x40
[ 2528.745311]  [<ffffffff8026c4d9>] ? validate_chain+0x469/0x1270
[ 2528.745311]  [<ffffffff8026d60e>] ? __lock_acquire+0x32e/0xa40
[ 2528.745311]  [<ffffffff804429df>] sys_recvfrom+0xaf/0x110
[ 2528.745311]  [<ffffffff804e6109>] ? mutex_unlock+0x9/0x10
[ 2528.745311]  [<ffffffff80310041>] ? sys_epoll_wait+0x4a1/0x510
[ 2528.745311]  [<ffffffff8020c55b>] system_call_fastpath+0x16/0x1b
[ 2528.745311] Code: 85 c0 7e 1b 48 8d bf 98 02 00 00 e8 29 34 e0 ff 85 c0 74 04 f0 ff 43 28 48 83 c4 08 5b c9 c3 e8 15 f3 ff ff 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec
[ 2528.745311] RIP  [<ffffffff80444ec2>] inet_def_readable+0x52/0x60
[ 2528.745311]  RSP <ffff88043b985b58>

Looks to me like __release_sock will call sk_backlog_rcv() with
the socket unlocked -- does that help at all?

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Brian Bloniarz]

Brian Bloniarz wrote:
>     BUG_ON(!spin_is_locked(&sk->sk_lock.slock));

Oh, sorry, I think I'm just misunderstanding how the socket
lock works. This doesn't actually check that the socket is locked,
right?

-Brian

Re: Multicast packet loss

[Brian Bloniarz]

Eric Dumazet wrote:
 > Brian Bloniarz a écrit :
 >> We've been experimenting with this softirq-delay patch in production, and
 >> have seen some hard-to-reproduce crashes. We finally managed to capture a
 >> kexec crashdump this morning.
 >
 > Pointer being null might tell us that we managed to call inet_def_readable()
 > without socket lock hold...

False alarm -- I think I did the backport to 2.6.24 incorrectly. 2.6.24 was
before the UDP receive path started taking the socket lock, so
inet_def_readable's assumption doesn't hold.

Sorry to waste everyone's time.

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Eric Dumazet wrote:
>> Brian Bloniarz a écrit :
>>> We've been experimenting with this softirq-delay patch in production,
> and
>>> have seen some hard-to-reproduce crashes. We finally managed to
> capture a
>>> kexec crashdump this morning.
>>
>> Pointer being null might tell us that we managed to call
> inet_def_readable()
>> without socket lock hold...
> 
> False alarm -- I think I did the backport to 2.6.24 incorrectly. 2.6.24 was
> before the UDP receive path started taking the socket lock, so
> inet_def_readable's assumption doesn't hold.
> 
> Sorry to waste everyone's time.
> 

Thanks for doing this discovery work and analysis. 

I am currently off-computers and could not do this until next week.

So, if you want to use 2.6.24, we need to back port other patches as well ?

Re: Multicast packet loss

[Brian Bloniarz]

Eric Dumazet wrote:
> Here is a patch that helps. It's still an RFC of course, since its somewhat ugly :)

Hi Eric,

I did some experimenting with this patch today -- we're users, not kernel hackers,
but the performance looks great. We see no loss with mcasttest, and no loss with
our internal test programs (which do much more user-space work). We're very
encouraged :)

One thing I'm curious about: previously, setting /proc/irq/<eth0>/smp_affinity
to one CPU made things perform better, but with this patch, performance is better
with smp_affinity == ff than with smp_affinity == 1. Do you know why that
is? Our tests are all with bnx2 msi_disable=1. I can investigate with oprofile
tomorrow.

Thank you for your continued help, we all deeply appreciate having someone
looking at this workload.

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Eric Dumazet wrote:
>> Here is a patch that helps. It's still an RFC of course, since its
>> somewhat ugly :)
> 
> Hi Eric,
> 
> I did some experimenting with this patch today -- we're users, not
> kernel hackers,
> but the performance looks great. We see no loss with mcasttest, and no
> loss with
> our internal test programs (which do much more user-space work). We're very
> encouraged :)
> 
> One thing I'm curious about: previously, setting
> /proc/irq/<eth0>/smp_affinity
> to one CPU made things perform better, but with this patch, performance
> is better
> with smp_affinity == ff than with smp_affinity == 1. Do you know why that
> is? Our tests are all with bnx2 msi_disable=1. I can investigate with
> oprofile
> tomorrow.
> 

Well, smp_affinity could help in my opininon if you dedicate
one cpu for the NIC, and others for user apps, if the average
work done per packet is large. If load is light, its better
to use the same cpu to perform all the work, since no expensive
bus trafic is needed between cpu to exchange memory lines.

If you only change /proc/irq/<eth0>/smp_affinity, and let scheduler
chose any cpu for your user-space work that can have long latencies,
I would not expect better performances.

Try to cpu affine your taks to 0xFE to get better determinism.

Re: Multicast packet loss

[Brian Bloniarz]

Hi Eric,

FYI: with your patch applied and lockdep enabled, I see:
[   39.114628] ================================================
[   39.121964] [ BUG: lock held when returning to user space! ]
[   39.127704] ------------------------------------------------
[   39.133461] msgtest/5242 is leaving the kernel with locks still held!
[   39.140132] 1 lock held by msgtest/5242:
[   39.144287]  #0:  (clock-AF_INET){-.-?}, at: [<ffffffff8041f5b9>] sock_def_readable+0x19/0xb0

I can't reproduced this with the mcasttest program yet, it
was with an internal test program which does some userspace
processing on the messages. I'll let you know if I find a way
to reproduce it with a simple program I can share.

 > Well, smp_affinity could help in my opininon if you dedicate
 > one cpu for the NIC, and others for user apps, if the average
 > work done per packet is large. If load is light, its better
 > to use the same cpu to perform all the work, since no expensive
 > bus trafic is needed between cpu to exchange memory lines.

I tried this setup as well: an 8-core box with 4 userspace
processes, each affined to an individual CPU1-4. The IRQ was on
CPU0. On most kernels, this setup loses fewer packets than the default
affinity (though they both lose some). With your patch enabled, the
default affinity loses 0 packets, and this setup loses some.

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Hi Eric,
> 
> FYI: with your patch applied and lockdep enabled, I see:
> [   39.114628] ================================================
> [   39.121964] [ BUG: lock held when returning to user space! ]
> [   39.127704] ------------------------------------------------
> [   39.133461] msgtest/5242 is leaving the kernel with locks still held!
> [   39.140132] 1 lock held by msgtest/5242:
> [   39.144287]  #0:  (clock-AF_INET){-.-?}, at: [<ffffffff8041f5b9>]
> sock_def_readable+0x19/0xb0

And you told me you were not a kernel hacker ;)

> 
> I can't reproduced this with the mcasttest program yet, it
> was with an internal test program which does some userspace
> processing on the messages. I'll let you know if I find a way
> to reproduce it with a simple program I can share.

I reproduced it as well here quite easily with a tcpdump of a tcp session,
thanks for the report.

It seems  "if (in_softirq())" doesnt do what I thought.

I wanted to test if we were called from __do_softirq() handler,
since only this function is calling softirq_delay_exec()
to dequeue events.

It appears I have to make current->softirq_context available
even if !CONFIG_TRACE_IRQFLAGS

Here is an updated version of the patch.

I also made the call to softirq_delay_exec() is performed
with interrupts enabled, and that preempt count wont
overflow if many events are queued.

[PATCH] softirq: Introduce mechanism to defer wakeups

Some network workloads need to call scheduler too many times. For example,
each received multicast frame can wakeup many threads. ksoftirqd is then
not able to drain NIC RX queues and we get frame losses and high latencies.

This patch adds an infrastructure to delay part of work done in
sock_def_readable() at end of do_softirq(). This need to
make available current->softirq_context even if !CONFIG_TRACE_IRQFLAGS


Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
---
 include/linux/interrupt.h |   18 +++++++++++++++++
 include/linux/irqflags.h  |    7 ++----
 include/linux/sched.h     |    2 -
 include/net/sock.h        |    1
 kernel/softirq.c          |   34 +++++++++++++++++++++++++++++++++
 net/core/sock.c           |   37 ++++++++++++++++++++++++++++++++++--
 6 files changed, 92 insertions(+), 7 deletions(-)


diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 9127f6b..a773d0c 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -295,6 +295,24 @@ extern void send_remote_softirq(struct call_single_data *cp, int cpu, int softir
 extern void __send_remote_softirq(struct call_single_data *cp, int cpu,
 				  int this_cpu, int softirq);
 
+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.
 
    Main feature differing them of generic softirqs: tasklet
diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h
index 74bde13..fe55ec4 100644
--- a/include/linux/irqflags.h
+++ b/include/linux/irqflags.h
@@ -13,6 +13,9 @@
 
 #include <linux/typecheck.h>
 
+#define trace_softirq_enter()	do { current->softirq_context++; } while (0)
+#define trace_softirq_exit()	do { current->softirq_context--; } while (0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
@@ -24,8 +27,6 @@
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while (0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while (0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -38,8 +39,6 @@
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif
 
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8c216e0..5dd8487 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1320,8 +1320,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 48UL
 	u64 curr_chain_key;
diff --git a/include/net/sock.h b/include/net/sock.h
index eefeeaf..1bfd9b8 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -260,6 +260,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9041ea7..c601730 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -158,6 +158,38 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);
 
+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};
+
+/*
+ * Preemption is disabled by caller
+ */
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (cmpxchg(&sdel->next, NULL, __get_cpu_var(softirq_delay_head)) == NULL) {
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->next = NULL;
+		sdel->func(sdel);
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -211,6 +243,8 @@ restart:
 		pending >>= 1;
 	} while (pending);
 
+	softirq_delay_exec();
+
 	local_irq_disable();
 
 	pending = local_softirq_pending();
diff --git a/net/core/sock.c b/net/core/sock.c
index 5f97caa..d51d57d 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -212,6 +212,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
 /* Maximal space eaten by iovec or ancilliary data plus some space */
 int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
 
+static void sock_readable_defer(struct softirq_delay *sdel);
+
 static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
 {
 	struct timeval tv;
@@ -1026,6 +1028,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority)
 #endif
 
 		rwlock_init(&newsk->sk_dst_lock);
+		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1634,12 +1637,41 @@ static void sock_def_error_report(struct sock *sk)
 	read_unlock(&sk->sk_callback_lock);
 }
 
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	wake_up_interruptible_sync(sk->sk_sleep);
+	/*
+	 * Before unlocking, we increase preempt_count,
+	 * as it was decreased in sock_def_readable()
+	 */
+	preempt_disable();
+	read_unlock(&sk->sk_callback_lock);
+}
+
 static void sock_def_readable(struct sock *sk, int len)
 {
 	read_lock(&sk->sk_callback_lock);
-	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep))
-		wake_up_interruptible_sync(sk->sk_sleep);
 	sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
+	if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) {
+		if (current->softirq_context) {
+			/*
+			 * If called from __do_softirq(), we want to delay
+			 * calls to wake_up_interruptible_sync()
+			 */
+			if (!softirq_delay_queue(&sk->sk_delay))
+				goto unlock;
+			/*
+			 * We keep sk->sk_callback_lock read locked,
+			 * but decrease preempt_count to avoid an overflow
+			 */
+			preempt_enable_no_resched();
+			return;
+		}
+		wake_up_interruptible_sync(sk->sk_sleep);
+	}
+unlock:
 	read_unlock(&sk->sk_callback_lock);
 }
 
@@ -1720,6 +1752,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 		sk->sk_sleep	=	NULL;
 
 	rwlock_init(&sk->sk_dst_lock);
+	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,

Re: Multicast packet loss

[Brian Bloniarz]

Eric Dumazet wrote:
> Here is an updated version of the patch.

This works great in all my tests so far.

Thanks,
Brian Bloniarz

Re: Multicast packet loss

[Eric Dumazet]

Brian Bloniarz a écrit :
> Eric Dumazet wrote:
>> Here is an updated version of the patch.
> 
> This works great in all my tests so far.
> 
> Thanks,
> Brian Bloniarz

Cool

I am wondering if we should extend the mechanism and change
softirq_delay_exec() to wakeup a workqueue instead of
doing the loop from softirq handler, in case a given
level of stress / load is hit.

This could help machines with several cpus, and one NIC (without
multi RX queues) flooded by messages (not necessarly multicast trafic).
Imagine a media/chat server receiving XXX.000 frames / second

One cpu could be dedicated to pure softirq/network handling,
and other cpus could participate and handle the scheduler part if any.

Condition could be : 

- We run __do_softirq() from ksoftirqd and 
- We queued more than N 'struct softirq_delay' in softirq_delay_head
- We have more than one cpu online

Re: Multicast packet loss

[Christoph Lameter]

On Fri, 30 Jan 2009, Eric Dumazet wrote:

> 2.6.29-rc contains UDP receive improvements (lockless)
>
> Problem is multicast handling was not yet updated, but could be :)

When will that happen?

Re: Multicast packet loss

[Eric Dumazet]

Christoph Lameter a écrit :
> On Fri, 30 Jan 2009, Eric Dumazet wrote:
> 
>> 2.6.29-rc contains UDP receive improvements (lockless)
>>
>> Problem is multicast handling was not yet updated, but could be :)
> 
> When will that happen?
> 

When proven necessary :)

Kenny problem is about scheduling storm. The extra spin_lock() in UDP 
multicast receive is not a problem.

Re: Multicast packet loss

[Christoph Lameter]

On Fri, 27 Feb 2009, Eric Dumazet wrote:

> Christoph Lameter a ?crit :
> > On Fri, 30 Jan 2009, Eric Dumazet wrote:
> >> 2.6.29-rc contains UDP receive improvements (lockless)
> >> Problem is multicast handling was not yet updated, but could be :)
> > When will that happen?
> When proven necessary :)
>
> Kenny problem is about scheduling storm. The extra spin_lock() in UDP
> multicast receive is not a problem.

My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via
UDP. This would fix a regression.....

Re: Multicast packet loss

[Eric Dumazet]

Christoph Lameter a écrit :
> On Fri, 27 Feb 2009, Eric Dumazet wrote:
> 
>> Christoph Lameter a ?crit :
>>> On Fri, 30 Jan 2009, Eric Dumazet wrote:
>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>> Problem is multicast handling was not yet updated, but could be :)
>>> When will that happen?
>> When proven necessary :)
>>
>> Kenny problem is about scheduling storm. The extra spin_lock() in UDP
>> multicast receive is not a problem.
> 
> My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via
> UDP. This would fix a regression.....
> 

Could you elaborate ?

I just retried Kenny test here. As one cpu is looping in ksoftirqd, only this cpu
touches the spin_lock, so spin_lock()/spin_unlock() is free.

oprofile shows that udp stack is lightweight in this case. Problem is about wakeing up
so many threads...

CPU: Core 2, speed 3000.16 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  cum. samples  %        cum. %     symbol name
356857   356857        15.1789  15.1789    schedule
274028   630885        11.6557  26.8346    mwait_idle
189218   820103         8.0484  34.8829    __skb_recv_datagram
116903   937006         4.9725  39.8554    skb_release_data
103152   1040158        4.3876  44.2430    lock_sock_nested
89600    1129758        3.8111  48.0541    udp_recvmsg
74171    1203929        3.1549  51.2089    copy_to_user
72299    1276228        3.0752  54.2842    set_next_entity
60392    1336620        2.5688  56.8529    sched_clock_cpu
54026    1390646        2.2980  59.1509    __slab_free
50212    1440858        2.1358  61.2867    prepare_to_wait_exclusive
38689    1479547        1.6456  62.9323    cpu_idle
38142    1517689        1.6224  64.5547    __switch_to
36701    1554390        1.5611  66.1157    hrtick_start_fair
36673    1591063        1.5599  67.6756    dst_release
36268    1627331        1.5427  69.2183    sys_recvfrom
35052    1662383        1.4909  70.7092    kmem_cache_free
32680    1695063        1.3900  72.0992    pick_next_task_fair
31209    1726272        1.3275  73.4267    try_to_wake_up
30382    1756654        1.2923  74.7190    dequeue_task_fair
29048    1785702        1.2356  75.9545    __copy_skb_header
28801    1814503        1.2250  77.1796    sock_def_readable
28655    1843158        1.2188  78.3984    enqueue_task_fair
27232    1870390        1.1583  79.5567    update_curr
21688    1892078        0.9225  80.4792    copy_from_user
18832    1910910        0.8010  81.2802    sysenter_past_esp
17732    1928642        0.7542  82.0345    finish_task_switch
17583    1946225        0.7479  82.7824    resched_task
17367    1963592        0.7387  83.5211    native_sched_clock
15691    1979283        0.6674  84.1885    task_rq_lock
15352    1994635        0.6530  84.8415    sock_queue_rcv_skb
15022    2009657        0.6390  85.4804    udp_queue_rcv_skb
13999    2023656        0.5954  86.0759    __update_sched_clock
12284    2035940        0.5225  86.5984    skb_copy_datagram_iovec
11869    2047809        0.5048  87.1032    release_sock
10986    2058795        0.4673  87.5705    __wake_up_sync
10488    2069283        0.4461  88.0166    sock_recvmsg
9686     2078969        0.4120  88.4286    skb_queue_tail
9425     2088394        0.4009  88.8295    sys_socketcall

Re: Multicast packet loss

[Eric Dumazet]

Eric Dumazet a écrit :
> Christoph Lameter a écrit :
>> On Fri, 27 Feb 2009, Eric Dumazet wrote:
>>
>>> Christoph Lameter a ?crit :
>>>> On Fri, 30 Jan 2009, Eric Dumazet wrote:
>>>>> 2.6.29-rc contains UDP receive improvements (lockless)
>>>>> Problem is multicast handling was not yet updated, but could be :)
>>>> When will that happen?
>>> When proven necessary :)
>>>
>>> Kenny problem is about scheduling storm. The extra spin_lock() in UDP
>>> multicast receive is not a problem.
>> My tests here show that 2.6.29-rc5 still looses ~5usec vs. 2.6.22 via
>> UDP. This would fix a regression.....
>>
> 
> Could you elaborate ?
> 
> I just retried Kenny test here. As one cpu is looping in ksoftirqd, only this cpu
> touches the spin_lock, so spin_lock()/spin_unlock() is free.
> 
> oprofile shows that udp stack is lightweight in this case. Problem is about wakeing up
> so many threads...
> 
> CPU: Core 2, speed 3000.16 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
> samples  cum. samples  %        cum. %     symbol name
> 356857   356857        15.1789  15.1789    schedule
> 274028   630885        11.6557  26.8346    mwait_idle
> 189218   820103         8.0484  34.8829    __skb_recv_datagram
> 116903   937006         4.9725  39.8554    skb_release_data
> 103152   1040158        4.3876  44.2430    lock_sock_nested
> 89600    1129758        3.8111  48.0541    udp_recvmsg
> 74171    1203929        3.1549  51.2089    copy_to_user
> 72299    1276228        3.0752  54.2842    set_next_entity
> 60392    1336620        2.5688  56.8529    sched_clock_cpu
> 54026    1390646        2.2980  59.1509    __slab_free
> 50212    1440858        2.1358  61.2867    prepare_to_wait_exclusive
> 38689    1479547        1.6456  62.9323    cpu_idle
> 38142    1517689        1.6224  64.5547    __switch_to
> 36701    1554390        1.5611  66.1157    hrtick_start_fair
> 36673    1591063        1.5599  67.6756    dst_release
> 36268    1627331        1.5427  69.2183    sys_recvfrom
> 35052    1662383        1.4909  70.7092    kmem_cache_free
> 32680    1695063        1.3900  72.0992    pick_next_task_fair
> 31209    1726272        1.3275  73.4267    try_to_wake_up
> 30382    1756654        1.2923  74.7190    dequeue_task_fair
> 29048    1785702        1.2356  75.9545    __copy_skb_header
> 28801    1814503        1.2250  77.1796    sock_def_readable
> 28655    1843158        1.2188  78.3984    enqueue_task_fair
> 27232    1870390        1.1583  79.5567    update_curr
> 21688    1892078        0.9225  80.4792    copy_from_user
> 18832    1910910        0.8010  81.2802    sysenter_past_esp
> 17732    1928642        0.7542  82.0345    finish_task_switch
> 17583    1946225        0.7479  82.7824    resched_task
> 17367    1963592        0.7387  83.5211    native_sched_clock
> 15691    1979283        0.6674  84.1885    task_rq_lock
> 15352    1994635        0.6530  84.8415    sock_queue_rcv_skb
> 15022    2009657        0.6390  85.4804    udp_queue_rcv_skb
> 13999    2023656        0.5954  86.0759    __update_sched_clock
> 12284    2035940        0.5225  86.5984    skb_copy_datagram_iovec
> 11869    2047809        0.5048  87.1032    release_sock
> 10986    2058795        0.4673  87.5705    __wake_up_sync
> 10488    2069283        0.4461  88.0166    sock_recvmsg
> 9686     2078969        0.4120  88.4286    skb_queue_tail
> 9425     2088394        0.4009  88.8295    sys_socketcall
> 
> 

My guess is commit 95766fff6b9a78d11fc2d3812dd035381690b55d
(UDP: Add memory accounting)
Hideo Aoki [Mon, 31 Dec 2007 08:29:24 +0000 (00:29 -0800)]

and 3ab224be6d69de912ee21302745ea45a99274dbc
[NET] CORE: Introducing new memory accounting interface.
Date:   Mon Dec 31 00:11:19 2007 -0800

are responsible for slowdown, because they add some
lock_sock()/release_sock() pairs.

function udp_recvmsg()

out_free:
+       lock_sock(sk);
        skb_free_datagram(sk, skb);
+       release_sock(sk);
 out:

I wonder why we can call __sk_mem_reclaim() when dequeing *one* UDP
frame in queue, while many others can still be in sk_receive_queue.
This defeats memory accounting, no ?

We should avoid lock_sock() if possible, or risk delaying
softirq RX in udp_queue_rcv_skb()

Re: Multicast packet loss

[Eric Dumazet]

Kenny Chang a écrit :
> Ah, sorry, here's the test program attached.
> 
> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
> 2.6.29.-rcX.
> 
> Right now, we are trying to step through the kernel versions until we
> see where the performance drops significantly.  We'll try 2.6.29-rc soon
> and post the result.
> 
> Neil Norman wrote:

On latest kernels, we have a "timer_slack_ns" default of 50.000 ns, aka 50us

So usleep(20) sleeps much more than expected.

You might add in your program a call to prcrl() to setup a smaller timer_slack :

#ifndef PR_SET_TIMERSLACK
#define PR_SET_TIMERSLACK 29
#endif
/*
 * Setup a timer resolution of 1000 ns : 1 us
 */
prctl(PR_SET_TIMERSLACK, 1000); 

Re: Multicast packet loss

[bmb]

> Could you please send me the diff of your backport against this kernel ?

Sure, the patch is at the bottom. It's against the tag: Ubuntu-2.6.24-19.41
from git://kernel.ubuntu.com/ubuntu/ubuntu-hardy.git

> I take you use Ubuntu Hardys 8.04 LTS server edition ?

Yes. We can only reproduce this in production right now, so trying out
a newer kernel would take some effort.

Thanks,
Brian Bloniarz

On Sun, 05 Apr 2009 15:49:14 +0200, Eric Dumazet <dada1@cosmosbay.com>
wrote:
> Brian Bloniarz a écrit :
>> Hi Eric,
>>
>> We've been experimenting with this softirq-delay patch in production,
> and
>> have seen some hard-to-reproduce crashes. We finally managed to capture
> a
>> kexec crashdump this morning.
>>
>> This is the dmesg:
>>
>> [53417.592868] Unable to handle kernel NULL pointer dereference at
>> 0000000000000000 RIP:
>> [53417.598377]  [<ffffffff80243643>] __do_softirq+0xc3/0x150
>> [53417.606300] PGD 32abb8067 PUD 32faf5067 PMD 0
>> [53417.610829] Oops: 0000 [1] SMP
>> [53417.614032] CPU 2
>> [53417.616083] Modules linked in: nfs lockd nfs_acl sunrpc openafs(P)
>> autofs4 ipv6 ac sbs sbshc video output dock battery container
>> iptable_filter ip_tables x_tables parport_pc lp parport loop joydev
>> iTCO_wdt iTCO_vendor_support evdev button i5000_edac psmouse serio_raw
>> pcspkr shpchp pci_hotplug edac_core ext3 jbd mbcache sr_mod cdrom
>> ata_generic usbhid hid ata_piix sg sd_mod ehci_hcd pata_acpi uhci_hcd
>> libata bnx2 aacraid usbcore scsi_mod thermal processor fan fbcon
>> tileblit font bitblit softcursor fuse
>> [53417.662067] Pid: 13039, comm: gball Tainted: P
>> 2.6.24-19acr2-generic #1
>> [53417.669219] RIP: 0010:[<ffffffff80243643>]  [<ffffffff80243643>]
>> __do_softirq+0xc3/0x150
>> [53417.677368] RSP: 0018:ffff8103314f3f20  EFLAGS: 00010297
>> [53417.682697] RAX: ffff810084a1b000 RBX: ffffffff805ba530 RCX:
>> 0000000000000000
>> [53417.689843] RDX: ffff8103305811e0 RSI: 0000000000000282 RDI:
>> ffff810332ada580
>> [53417.696993] RBP: 0000000000000000 R08: ffff81032fad9f08 R09:
>> ffff810332382000
>> [53417.704144] R10: 0000000000000000 R11: ffffffff80316ec0 R12:
>> ffffffff8062b3d8
>> [53417.711294] R13: ffffffff8062b480 R14: 0000000000000002 R15:
>> 000000000000000a
>> [53417.718447] FS:  00007fab0d7b8750(0000) GS:ffff810334401b80(0000)
>> knlGS:0000000000000000
>> [53417.726568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [53417.732332] CR2: 0000000000000000 CR3: 0000000329e2d000 CR4:
>> 00000000000006e0
>> [53417.739476] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [53417.746637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [53417.753787] Process gball (pid: 13039, threadinfo ffff81032adde000,
>> task ffff810329ff77d0)
>> [53417.761991] Stack:  ffffffff8062b3d8 0000000000000046
>> ffff8103314f3f68 0000000000000000
>> [53417.770146]  00000000000000a0 ffff81032addfee8 0000000000000000
>> ffffffff8020d50c
>> [53417.777660]  ffff8103314f3f68 00000000000000c1 ffffffff8020ed25
>> ffffffff8062c870
>> [53417.784961] Call Trace:
>> [53417.787635]  <IRQ>  [<ffffffff8020d50c>] call_softirq+0x1c/0x30
>> [53417.793597]  [<ffffffff8020ed25>] do_softirq+0x35/0x90
>> [53417.798747]  [<ffffffff80243578>] irq_exit+0x88/0x90
>> [53417.803727]  [<ffffffff8020ef70>] do_IRQ+0x80/0x100
>> [53417.808624]  [<ffffffff8020c891>] ret_from_intr+0x0/0xa
>> [53417.813862]  <EOI>  [<ffffffff803e53c8>] skb_release_all+0x18/0x150
>> [53417.820164]  [<ffffffff803e4ad9>] __kfree_skb+0x9/0x90
>> [53417.825327]  [<ffffffff80437612>] udp_recvmsg+0x222/0x260
>> [53417.830744]  [<ffffffff80231264>] source_load+0x34/0x70
>> [53417.835984]  [<ffffffff80232a9a>] find_busiest_group+0x1fa/0x850
>> [53417.842019]  [<ffffffff803e0100>] sock_common_recvmsg+0x30/0x50
>> [53417.847958]  [<ffffffff803de1ca>] sock_recvmsg+0x14a/0x160
>> [53417.853462]  [<ffffffff80231c21>] update_curr+0x71/0x100
>> [53419.858789]  [<ffffffff802320fd>] __dequeue_entity+0x3d/0x50
>> [53417.864469]  [<ffffffff80253ab0>] autoremove_wake_function+0x0/0x30
>> [53417.870758]  [<ffffffff8046662f>] thread_return+0x3a/0x57b
>> [53417.876262]  [<ffffffff803df73e>] sys_recvfrom+0xfe/0x190
>> [53417.881680]  [<ffffffff802e2a95>] sys_epoll_wait+0x245/0x4e0
>> [53417.887358]  [<ffffffff80233e20>] default_wake_function+0x0/0x10
>> [53417.893384]  [<ffffffff8020c37e>] system_call+0x7e/0x83
>> [53417.898628]
>> [53417.900134]
>> [53417.900134] Code: 48 8b 11 48 89 cf 65 48 8b 04 25 08 00 00 00 4a 89
>> 14 20 ff
>> [53417.909430] RIP  [<ffffffff80243643>] __do_softirq+0xc3/0x150
>> [53417.915210]  RSP <ffff8103314f3f20>
>>
>> The disassembly where it crashed:
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
>> ffffffff8024361b:       d1 ed                   shr    %ebp
>> rcu_bh_qsctr_inc():
>>
> /local/home/bmb/doc/kernels/linux-hardy-eric/include/linux/rcupdate.h:130
>> ffffffff8024361d:       48 8b 40 08             mov    0x8(%rax),%rax
>> ffffffff80243621:       41 c7 44 05 08 01 00    movl
>> $0x1,0x8(%r13,%rax,1)
>> ffffffff80243628:       00 00
>> __do_softirq():
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:273
>> ffffffff8024362a:       75 d8                   jne    ffffffff80243604
>> <__do_softirq+0x84>
>> softirq_delay_exec():
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
>> ffffffff8024362c:       48 8b 14 24             mov    (%rsp),%rdx
>> ffffffff80243630:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
>> ffffffff80243637:       00 00
>> ffffffff80243639:       48 8b 0c 10             mov
> (%rax,%rdx,1),%rcx
>> ffffffff8024363d:       48 83 f9 01             cmp    $0x1,%rcx
>> ffffffff80243641:       74 29                   je     ffffffff8024366c
>> <__do_softirq+0xec>
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
>> ffffffff80243643:       48 8b 11                mov    (%rcx),%rdx
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
>> ffffffff80243646:       48 89 cf                mov    %rcx,%rdi
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:226
>> ffffffff80243649:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
>> ffffffff80243650:       00 00
>> ffffffff80243652:       4a 89 14 20             mov
> %rdx,(%rax,%r12,1)
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:227
>> ffffffff80243656:       ff 51 08                callq  *0x8(%rcx)
>> /local/home/bmb/doc/kernels/linux-hardy-eric/kernel/softirq.c:225
>> ffffffff80243659:       65 48 8b 04 25 08 00    mov    %gs:0x8,%rax
>> ffffffff80243660:       00 00
>> ffffffff80243662:       4a 8b 0c 20             mov
> (%rax,%r12,1),%rcx
>> ffffffff80243666:       48 83 f9 01             cmp    $0x1,%rcx
>> ffffffff8024366a:       75 d7                   jne    ffffffff80243643
>> <__do_softirq+0xc3>
>> raw_local_irq_disable():
>>
>
/local/home/bmb/doc/kernels/linux-hardy-eric/debian/build/build-generic/include2/asm/irqflags_64.h:76
>>
>> ffffffff8024366c:       fa                      cli
>>
>> And softirq.c line numbers:
>>    218   * Because locking is provided by subsystem, please note
>>    219   * that sdel->func(sdel) is responsible for setting sdel->next
>> to NULL
>>    220   */
>>    221  static void softirq_delay_exec(void)
>>    222  {
>>    223          struct softirq_delay *sdel;
>>    224
>>    225          while ((sdel = __get_cpu_var(softirq_delay_head)) !=
>> SOFTIRQ_DELAY_END) {
>>    226                  __get_cpu_var(softirq_delay_head) = sdel->next;
>>    227                  sdel->func(sdel);       /*      sdel->next =
>> NULL;*/
>>    228                  }
>>    229  }
>>
>> So it's crashing because __get_cpu_var(softirq_delay_head)) is NULL
>> somehow.
>>
>> We aren't running a recent kernel -- we're running Ubuntu Hardy's
>> 2.6.24-19,
>> with a backported version of this patch. One more atypical thing is that
>> we run openafs, 1.4.6.dfsg1-2.
>>
>> Like I said, I have a full vmcore (3, actually) and would be happy to
>> post any
>> more information you'd like to know.
>>
>> Thanks,
>> Brian Bloniarz
>
> Hi Brian
>
> 2.6.24-19 kernel... hmm...
>
> Could you please send me the diff of your backport against this kernel ?
>
> I take you use Ubuntu Hardys 8.04 LTS server edition ?
>
> Pointer being null might tell us that we managed to call
> inet_def_readable()
> without socket lock hold...

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 2306920..b79a207 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -276,6 +276,24 @@ extern void FASTCALL(raise_softirq_irqoff(unsigned int
nr));
 extern void FASTCALL(raise_softirq(unsigned int nr));


+/*
+ * softirq delayed works : should be delayed at do_softirq() end
+ */
+struct softirq_delay {
+	struct softirq_delay	*next;
+	void 			(*func)(struct softirq_delay *);
+};
+
+int softirq_delay_queue(struct softirq_delay *sdel);
+
+static inline void softirq_delay_init(struct softirq_delay *sdel,
+				      void (*func)(struct softirq_delay *))
+{
+	sdel->next = NULL;
+	sdel->func = func;
+}
+
+
 /* Tasklets --- multithreaded analogue of BHs.

    Main feature differing them of generic softirqs: tasklet
diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h
index 412e025..f7b48a1 100644
--- a/include/linux/irqflags.h
+++ b/include/linux/irqflags.h
@@ -11,19 +11,21 @@
 #ifndef _LINUX_TRACE_IRQFLAGS_H
 #define _LINUX_TRACE_IRQFLAGS_H

+#define softirq_enter()	do { current->softirq_context++; } while (0)
+#define softirq_exit()	do { current->softirq_context--; } while (0)
+#define softirq_context(p)	((p)->softirq_context)
+#define running_from_softirq()  (softirq_context(current) > 0)
+
 #ifdef CONFIG_TRACE_IRQFLAGS
   extern void trace_hardirqs_on(void);
   extern void trace_hardirqs_off(void);
   extern void trace_softirqs_on(unsigned long ip);
   extern void trace_softirqs_off(unsigned long ip);
 # define trace_hardirq_context(p)	((p)->hardirq_context)
-# define trace_softirq_context(p)	((p)->softirq_context)
 # define trace_hardirqs_enabled(p)	((p)->hardirqs_enabled)
 # define trace_softirqs_enabled(p)	((p)->softirqs_enabled)
 # define trace_hardirq_enter()	do { current->hardirq_context++; } while
(0)
 # define trace_hardirq_exit()	do { current->hardirq_context--; } while (0)
-# define trace_softirq_enter()	do { current->softirq_context++; } while
(0)
-# define trace_softirq_exit()	do { current->softirq_context--; } while (0)
 # define INIT_TRACE_IRQFLAGS	.softirqs_enabled = 1,
 #else
 # define trace_hardirqs_on()		do { } while (0)
@@ -31,13 +33,10 @@
 # define trace_softirqs_on(ip)		do { } while (0)
 # define trace_softirqs_off(ip)		do { } while (0)
 # define trace_hardirq_context(p)	0
-# define trace_softirq_context(p)	0
 # define trace_hardirqs_enabled(p)	0
 # define trace_softirqs_enabled(p)	0
 # define trace_hardirq_enter()		do { } while (0)
 # define trace_hardirq_exit()		do { } while (0)
-# define trace_softirq_enter()		do { } while (0)
-# define trace_softirq_exit()		do { } while (0)
 # define INIT_TRACE_IRQFLAGS
 #endif

diff --git a/include/linux/net.h b/include/linux/net.h
index 596131e..9e762e9 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -115,12 +115,16 @@ enum sock_shutdown_cmd {
 struct socket {
 	socket_state		state;
 	unsigned long		flags;
-	const struct proto_ops	*ops;
+	/*
+	 * Please keep fasync_list & wait fields in the same cache line
+	 */
 	struct fasync_struct	*fasync_list;
+	wait_queue_head_t	wait;
+
 	struct file		*file;
 	struct sock		*sk;
-	wait_queue_head_t	wait;
 	short			type;
+	const struct proto_ops	*ops;
 };

 struct vm_area_struct;
diff --git a/include/linux/sched.h b/include/linux/sched.h
index dc2f4fa..bf0ff49 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1111,8 +1111,8 @@ struct task_struct {
 	unsigned long softirq_enable_ip;
 	unsigned int softirq_enable_event;
 	int hardirq_context;
-	int softirq_context;
 #endif
+	int softirq_context;
 #ifdef CONFIG_LOCKDEP
 # define MAX_LOCK_DEPTH 30UL
 	u64 curr_chain_key;
diff --git a/include/net/sock.h b/include/net/sock.h
index 6e1542d..fb0f719 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -236,6 +236,7 @@ struct sock {
 	unsigned long	        sk_lingertime;
 	struct sk_buff_head	sk_error_queue;
 	struct proto		*sk_prot_creator;
+	struct softirq_delay	sk_delay;
 	rwlock_t		sk_callback_lock;
 	int			sk_err,
 				sk_err_soft;
@@ -859,6 +860,7 @@ extern void *sock_kmalloc(struct sock *sk, int size,
 			  gfp_t priority);
 extern void sock_kfree_s(struct sock *sk, void *mem, int size);
 extern void sk_send_sigurg(struct sock *sk);
+extern void inet_def_readable(struct sock *sk, int len);

 /*
  * Functions to fill in entries in struct proto_ops when a protocol
diff --git a/include/net/udplite.h b/include/net/udplite.h
index 635b0ea..1589817 100644
--- a/include/net/udplite.h
+++ b/include/net/udplite.h
@@ -28,6 +28,7 @@ static __inline__ int udplite_getfrag(void *from, char
*to, int  offset,
 /* Designate sk as UDP-Lite socket */
 static inline int udplite_sk_init(struct sock *sk)
 {
+	sk->sk_data_ready = inet_def_readable;
 	udp_sk(sk)->pcflag = UDPLITE_BIT;
 	return 0;
 }
diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index e2c07ec..decb1f7 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -1643,7 +1643,7 @@ print_usage_bug(struct task_struct *curr, struct
held_lock *this,
 	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] takes:\n",
 		curr->comm, task_pid_nr(curr),
 		trace_hardirq_context(curr), hardirq_count() >> HARDIRQ_SHIFT,
-		trace_softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
+		softirq_context(curr), softirq_count() >> SOFTIRQ_SHIFT,
 		trace_hardirqs_enabled(curr),
 		trace_softirqs_enabled(curr));
 	print_lock(this);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index bd89bc4..fb116ac 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -194,6 +194,42 @@ void local_bh_enable_ip(unsigned long ip)
 }
 EXPORT_SYMBOL(local_bh_enable_ip);

+
+#define SOFTIRQ_DELAY_END (struct softirq_delay *)1L
+static DEFINE_PER_CPU(struct softirq_delay *, softirq_delay_head) = {
+	SOFTIRQ_DELAY_END
+};
+
+/*
+ * Caller must disable preemption, and take care of appropriate
+ * locking and refcounting
+ */
+int softirq_delay_queue(struct softirq_delay *sdel)
+{
+	if (!sdel->next) {
+		sdel->next = __get_cpu_var(softirq_delay_head);
+		__get_cpu_var(softirq_delay_head) = sdel;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Because locking is provided by subsystem, please note
+ * that sdel->func(sdel) is responsible for setting sdel->next to NULL
+ */
+static void softirq_delay_exec(void)
+{
+	struct softirq_delay *sdel;
+
+	while ((sdel = __get_cpu_var(softirq_delay_head)) != SOFTIRQ_DELAY_END) {
+		__get_cpu_var(softirq_delay_head) = sdel->next;
+		sdel->func(sdel);	/*	sdel->next = NULL;*/
+		}
+}
+
+
+
 /*
  * We restart softirq processing MAX_SOFTIRQ_RESTART times,
  * and we fall back to softirqd after that.
@@ -216,7 +252,7 @@ asmlinkage void __do_softirq(void)
 	account_system_vtime(current);

 	__local_bh_disable((unsigned long)__builtin_return_address(0));
-	trace_softirq_enter();
+	softirq_enter();

 	cpu = smp_processor_id();
 restart:
@@ -236,6 +272,8 @@ restart:
 		pending >>= 1;
 	} while (pending);

+	softirq_delay_exec();
+
 	local_irq_disable();

 	pending = local_softirq_pending();
@@ -245,7 +283,7 @@ restart:
 	if (pending)
 		wakeup_softirqd();

-	trace_softirq_exit();
+	softirq_exit();

 	account_system_vtime(current);
 	_local_bh_enable();
diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c
index 280332c..1aa7351 100644
--- a/lib/locking-selftest.c
+++ b/lib/locking-selftest.c
@@ -157,11 +157,11 @@ static void init_shared_classes(void)
 #define SOFTIRQ_ENTER()				\
 		local_bh_disable();		\
 		local_irq_disable();		\
-		trace_softirq_enter();		\
+		softirq_enter();		\
 		WARN_ON(!in_softirq());

 #define SOFTIRQ_EXIT()				\
-		trace_softirq_exit();		\
+		softirq_exit();		\
 		local_irq_enable();		\
 		local_bh_enable();

diff --git a/net/core/sock.c b/net/core/sock.c
index c519b43..cb70343 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -213,6 +213,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
 /* Maximal space eaten by iovec or ancilliary data plus some space */
 int sysctl_optmem_max __read_mostly = sizeof(unsigned
long)*(2*UIO_MAXIOV+512);

+static void sock_readable_defer(struct softirq_delay *sdel);
+
 static int sock_set_timeout(long *timeo_p, char __user *optval, int
optlen)
 {
 	struct timeval tv;
@@ -996,6 +998,7 @@ struct sock *sk_clone(const struct sock *sk, const
gfp_t priority)
 #endif

 		rwlock_init(&newsk->sk_dst_lock);
+		softirq_delay_init(&newsk->sk_delay, sock_readable_defer);
 		rwlock_init(&newsk->sk_callback_lock);
 		lockdep_set_class_and_name(&newsk->sk_callback_lock,
 				af_callback_keys + newsk->sk_family,
@@ -1509,6 +1512,45 @@ static void sock_def_readable(struct sock *sk, int
len)
 	read_unlock(&sk->sk_callback_lock);
 }

+/*
+ * helper function called by softirq_delay_exec(),
+ * if inet_def_readable() queued us.
+ */
+static void sock_readable_defer(struct softirq_delay *sdel)
+{
+	struct sock *sk = container_of(sdel, struct sock, sk_delay);
+
+	sdel->next = NULL;
+	/*
+	 * At this point, we dont own a lock on socket, only a reference.
+	 * We must commit above write, or another cpu could miss a wakeup
+	 */
+	smp_wmb();
+	sock_def_readable(sk, 0);
+	sock_put(sk);
+}
+
+/*
+ * Custom version of sock_def_readable()
+ * We want to defer scheduler processing at the end of do_softirq()
+ * Called with socket locked.
+ */
+void inet_def_readable(struct sock *sk, int len)
+{
+	if (running_from_softirq()) {
+		if (softirq_delay_queue(&sk->sk_delay))
+			/*
+			 * If we queued this socket, take a reference on it
+			 * Caller owns socket lock, so write to sk_delay.next
+			 * will be committed before unlock.
+			 */
+			sock_hold(sk);
+	} else
+		sock_def_readable(sk, len);
+}
+
+EXPORT_SYMBOL(inet_def_readable);
+
 static void sock_def_write_space(struct sock *sk)
 {
 	read_lock(&sk->sk_callback_lock);
@@ -1586,6 +1628,7 @@ void sock_init_data(struct socket *sock, struct sock
*sk)
 		sk->sk_sleep	=	NULL;

 	rwlock_init(&sk->sk_dst_lock);
+	softirq_delay_init(&sk->sk_delay, sock_readable_defer);
 	rwlock_init(&sk->sk_callback_lock);
 	lockdep_set_class_and_name(&sk->sk_callback_lock,
 			af_callback_keys + sk->sk_family,
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 03c400c..cfeb051 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1226,6 +1226,12 @@ int udp_destroy_sock(struct sock *sk)
 	return 0;
 }

+static int udp_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -1439,6 +1445,7 @@ struct proto udp_prot = {
 	.connect	   = ip4_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init		   = udp_init_sock,
 	.destroy	   = udp_destroy_sock,
 	.setsockopt	   = udp_setsockopt,
 	.getsockopt	   = udp_getsockopt,
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index ee1cc3f..fa9ce73 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -856,6 +856,12 @@ int udpv6_destroy_sock(struct sock *sk)
 	return 0;
 }

+static int udpv6_init_sock(struct sock *sk)
+{
+	sk->sk_data_ready = inet_def_readable;
+	return 0;
+}
+
 /*
  *	Socket option code for UDP
  */
@@ -979,6 +985,7 @@ struct proto udpv6_prot = {
 	.connect	   = ip6_datagram_connect,
 	.disconnect	   = udp_disconnect,
 	.ioctl		   = udp_ioctl,
+	.init 		   = udpv6_init_sock,
 	.destroy	   = udpv6_destroy_sock,
 	.setsockopt	   = udpv6_setsockopt,
 	.getsockopt	   = udpv6_getsockopt,