From: Andrew Savchenko <bircoph@gmail.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: netdev@vger.kernel.org
Subject: Re: [BUG] Kernel recieves DNS reply, but doesn't deliver it to a waiting application
Date: Wed, 16 Jan 2013 20:36:44 +0400 [thread overview]
Message-ID: <20130116203644.8eb2a16f.bircoph@gmail.com> (raw)
In-Reply-To: <1356718263.21409.430.camel@edumazet-glaptop>
[-- Attachment #1.1: Type: text/plain, Size: 2135 bytes --]
Hello,
On Fri, 28 Dec 2012 10:11:03 -0800 Eric Dumazet wrote:
> On Sun, 2012-12-23 at 15:06 +0400, Andrew Savchenko wrote:
[...]
> > I hit this bug again on uptime 11 days on 3.7.0 vanilla kernel.
> > See kernel config, /prot/net/upd, netstat -s and dropwatch logs
> > attached to this mail. This bug happens on UDP DNS requests only,
> > TCP requests work fine, see dig.log attached.
> >
> > Increasing of net.ipv4.udp_mem from
> > 24150 32201 48300
> > to
> > 100000 150000 200000
> > helps, but I'm afraid only temporary again.
> >
> > Dropwatch data was collected in the following way:
> > - dropwatch.bug.* files contain data obtained after bug occurred;
> > - dropwatch.*.background files contain background data when no
> > host or dig test was running; this system has active firewall
> > and complicated routing, ipv6 disabled via sysctl, etc, so some
> > drops are normal;
> > - dropwatch.*.host.request shows dropped packets recorded during
> > host ya.ru request; of course, during this time some background
> > packets were recorded as well (dropwatch doesn't support filtering
> > at this moment);
> > - dropwatch.nobug.* data was collected after the bug was
> > workarounded via net.ipv4.upd_mem as described above.
> >
> > As can be seen from dropwatch logs, drop in __udp_queue_rcv_skb+61
> > happens only on host request on bug conditions, thus something is
> > wrong there.
> >
> > Best regards,
> > Andrew Savchenko
>
> Thanks a lot !
>
> I see strange drops in dev_hard_start_xmit()
>
> l2tp needs some care.
>
> Please try the following patch, as skb_cow_head() API
> doesnt really ease skb->truesize exact tracking anyway, better not mess
> with it.
Sorry for the delay, but I was able to reboot kernel only today.
Your patch is applied on top of the 3.7.2 vanilla kernel.
l2tp works fine and /proc/net/udp tx_queue values are normal now, see
attached /prot/net/udp output. This is a good hint that problem is
probably solved, but we need to wait at least several weeks to be
sure.
Best regards,
Andrew Savchenko
[-- Attachment #1.2: proc.net.udp --]
[-- Type: application/octet-stream, Size: 4224 bytes --]
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
0: 00000000:06A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5013 2 ffff88003d605c00 0
89: 00000000:90FE 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5074 2 ffff88003bcba700 0
90: 7E18340A:88FF 1400320A:06A5 01 00000000:00000000 00:00000000 00000000 0 0 5034 4 ffff88003bcba000 0
136: 00000000:892D 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5049 2 ffff88003bcba380 0
160: 0100007F:2745 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4495 2 ffff88003d604380 0
183: 0100007F:035C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4770 2 ffff88003d604a80 0
217: 00000000:857E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5146 2 ffff88003bd34700 0
310: 00000000:03DB 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4520 2 ffff88003d604700 0
318: 00000000:A9E3 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4776 2 ffff88003d604e00 0
348: 00000000:0801 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5137 2 ffff88003bd34380 0
400: 010013AC:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4876 2 ffff88003d605880 0
400: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4875 2 ffff88003d605500 0
414: 00000000:0043 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5176 2 ffff88003bd35500 0
458: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4486 2 ffff88003d604000 0
466: 00000000:0277 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4818 2 ffff88003d605180 0
470: 076A070A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 123 0 5526 2 ffff88003bd35c00 0
470: 010213AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5121 2 ffff88003bd34000 0
470: 010013AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5120 2 ffff88003bcbbc00 0
470: 00FCA8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5119 2 ffff88003bcbb880 0
470: 7E18340A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5118 2 ffff88003bcbb500 0
470: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5117 2 ffff88003bcbb180 0
470: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5111 2 ffff88003bcbaa80 0
484: FF0013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5298 2 ffff880039fbd500 0
484: 010013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5297 2 ffff880039fbd180 0
484: FF0213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5294 2 ffff880039fbc700 0
484: 010213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5293 2 ffff880039fbc380 0
484: 00000000:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5290 2 ffff88003bcbae00 0
485: FF0013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5300 2 ffff880039fbdc00 0
485: 010013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5299 2 ffff880039fbd880 0
485: FF0213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5296 2 ffff880039fbce00 0
485: 010213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5295 2 ffff880039fbca80 0
485: 00000000:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5291 2 ffff880039fbc000 0
[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]
next prev parent reply other threads:[~2013-01-16 16:43 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-03 19:25 Kernel recieves DNS reply, but doesn't deliver it to a waiting application Andrew Savchenko
2012-10-13 12:36 ` [BUG] " Andrew Savchenko
2012-10-13 13:44 ` Eric Dumazet
2012-10-13 23:11 ` Andrew Savchenko
2012-10-20 23:25 ` Andrew Savchenko
2012-10-21 12:52 ` Eric Dumazet
2012-10-22 3:36 ` Andrew Savchenko
2012-10-22 6:48 ` Eric Dumazet
2012-10-22 21:27 ` Andrew Savchenko
2012-12-12 8:27 ` Andrew Savchenko
2012-12-23 11:06 ` Andrew Savchenko
2012-12-28 18:11 ` Eric Dumazet
2013-01-16 16:36 ` Andrew Savchenko [this message]
2013-02-04 13:39 ` Andrew Savchenko
2013-02-04 15:21 ` Eric Dumazet
2012-11-23 7:45 ` Andrew Savchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130116203644.8eb2a16f.bircoph@gmail.com \
--to=bircoph@gmail.com \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.