Re: ICMP echo reply fails

[Andy Fleming <afleming@gmail.com>]

On Fri, Mar 26, 2010 at 5:46 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le vendredi 26 mars 2010 à 23:06 +0100, Eric Dumazet a écrit :
>> Le vendredi 26 mars 2010 à 16:48 -0500, Andy Fleming a écrit :
>> > For various reasons, we have been running a stress test on one of our
>> > boards.  The test consists of initiating 2-3 flood pings from a
>> > Windows box running Cygwin, plus one additional ping we use as a
>> > "heartbeat".  The ping flood is overwhelming our board (we're dropping
>> > packets at a prodigious rate), but the board continues to respond for
>> > a while.  In addition, we are running a script on the  board which
>> > alternates bringing up and bringing down the interface every ten
>> > seconds.  After a highly variable amount of time, the board stops
>> > replying to the pings.  We suspected a driver issue, however, on
>> > closer inspection, we are still able to send and receive packets (I
>> > can ping *from* the board to the PC, and I can *telnet* from the PC to
>> > the board).  We tried pinging the board from another PC, and it also
>> > failed.  Essentially, ICMP echo requests are being ignored (A glance
>> > at memory indicates that packets are arriving, but no packets are
>> > being enqueued to the ethernet controller).  We still have a lot more
>> > debugging to do, but I was wondering if anyone had ever seen something
>> > like this, or might be quicker to realize the obvious mistake we're
>> > making.
>> >
>> > Thanks,
>> > Andy Fleming
>>
>>
>> kernel version ?
>>
>> NIC driver ?
>>
>> Are ICMP echo request received ? (grep Icmp /proc/net/snmp)
>>
>
> vi +1166 net/ipv4/icmp.c
>
>        /* Enough space for 2 64K ICMP packets, including
>         * sk_buff struct overhead.
>         */
>        sk->sk_sndbuf =
>                (2 * ((64 * 1024) + sizeof(struct sk_buff)));
>
>
> If many ICMP replies are lost/leaked by your driver when doing up/down
> things, ICMP socket can consume all its sndbuf reserve and no more icmp
> replies can be sent (a reboot is needed)
>
> You could try changing sk->sk_sndbuf to 0x7FFFFFFF  to see if the icmp
> replies survive longer to your tests. If this is the case, then find the
> leaks in your driver (tx path, maybe you forgot to free skbs in some
> reset cases ?)
>

Ah, that makes a bunch of sense.  I had a feeling the socket was
involved.  Thank you so much for your help.  I will test this as soon
as I have access to the board again!