From: Andrew Savchenko <bircoph@gmail.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: netdev@vger.kernel.org
Subject: Re: [BUG] Kernel recieves DNS reply, but doesn't deliver it to a waiting application
Date: Wed, 16 Jan 2013 20:36:44 +0400 [thread overview]
Message-ID: <20130116203644.8eb2a16f.bircoph@gmail.com> (raw)
In-Reply-To: <1356718263.21409.430.camel@edumazet-glaptop>
[-- Attachment #1.1: Type: text/plain, Size: 2135 bytes --]
Hello,
On Fri, 28 Dec 2012 10:11:03 -0800 Eric Dumazet wrote:
> On Sun, 2012-12-23 at 15:06 +0400, Andrew Savchenko wrote:
[...]
> > I hit this bug again on uptime 11 days on 3.7.0 vanilla kernel.
> > See kernel config, /prot/net/upd, netstat -s and dropwatch logs
> > attached to this mail. This bug happens on UDP DNS requests only,
> > TCP requests work fine, see dig.log attached.
> >
> > Increasing of net.ipv4.udp_mem from
> > 24150 32201 48300
> > to
> > 100000 150000 200000
> > helps, but I'm afraid only temporary again.
> >
> > Dropwatch data was collected in the following way:
> > - dropwatch.bug.* files contain data obtained after bug occurred;
> > - dropwatch.*.background files contain background data when no
> > host or dig test was running; this system has active firewall
> > and complicated routing, ipv6 disabled via sysctl, etc, so some
> > drops are normal;
> > - dropwatch.*.host.request shows dropped packets recorded during
> > host ya.ru request; of course, during this time some background
> > packets were recorded as well (dropwatch doesn't support filtering
> > at this moment);
> > - dropwatch.nobug.* data was collected after the bug was
> > workarounded via net.ipv4.upd_mem as described above.
> >
> > As can be seen from dropwatch logs, drop in __udp_queue_rcv_skb+61
> > happens only on host request on bug conditions, thus something is
> > wrong there.
> >
> > Best regards,
> > Andrew Savchenko
>
> Thanks a lot !
>
> I see strange drops in dev_hard_start_xmit()
>
> l2tp needs some care.
>
> Please try the following patch, as skb_cow_head() API
> doesnt really ease skb->truesize exact tracking anyway, better not mess
> with it.
Sorry for the delay, but I was able to reboot kernel only today.
Your patch is applied on top of the 3.7.2 vanilla kernel.
l2tp works fine and /proc/net/udp tx_queue values are normal now, see
attached /prot/net/udp output. This is a good hint that problem is
probably solved, but we need to wait at least several weeks to be
sure.
Best regards,
Andrew Savchenko
[-- Attachment #1.2: proc.net.udp --]
[-- Type: application/octet-stream, Size: 4224 bytes --]
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
0: 00000000:06A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5013 2 ffff88003d605c00 0
89: 00000000:90FE 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5074 2 ffff88003bcba700 0
90: 7E18340A:88FF 1400320A:06A5 01 00000000:00000000 00:00000000 00000000 0 0 5034 4 ffff88003bcba000 0
136: 00000000:892D 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5049 2 ffff88003bcba380 0
160: 0100007F:2745 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4495 2 ffff88003d604380 0
183: 0100007F:035C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4770 2 ffff88003d604a80 0
217: 00000000:857E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5146 2 ffff88003bd34700 0
310: 00000000:03DB 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4520 2 ffff88003d604700 0
318: 00000000:A9E3 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4776 2 ffff88003d604e00 0
348: 00000000:0801 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5137 2 ffff88003bd34380 0
400: 010013AC:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4876 2 ffff88003d605880 0
400: 0100007F:0035 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4875 2 ffff88003d605500 0
414: 00000000:0043 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5176 2 ffff88003bd35500 0
458: 00000000:006F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4486 2 ffff88003d604000 0
466: 00000000:0277 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 4818 2 ffff88003d605180 0
470: 076A070A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 123 0 5526 2 ffff88003bd35c00 0
470: 010213AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5121 2 ffff88003bd34000 0
470: 010013AC:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5120 2 ffff88003bcbbc00 0
470: 00FCA8C0:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5119 2 ffff88003bcbb880 0
470: 7E18340A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5118 2 ffff88003bcbb500 0
470: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5117 2 ffff88003bcbb180 0
470: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5111 2 ffff88003bcbaa80 0
484: FF0013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5298 2 ffff880039fbd500 0
484: 010013AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5297 2 ffff880039fbd180 0
484: FF0213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5294 2 ffff880039fbc700 0
484: 010213AC:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5293 2 ffff880039fbc380 0
484: 00000000:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5290 2 ffff88003bcbae00 0
485: FF0013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5300 2 ffff880039fbdc00 0
485: 010013AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5299 2 ffff880039fbd880 0
485: FF0213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5296 2 ffff880039fbce00 0
485: 010213AC:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5295 2 ffff880039fbca80 0
485: 00000000:008A 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 5291 2 ffff880039fbc000 0
[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]
next prev parent reply other threads:[~2013-01-16 16:43 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-03 19:25 Kernel recieves DNS reply, but doesn't deliver it to a waiting application Andrew Savchenko
2012-10-13 12:36 ` [BUG] " Andrew Savchenko
2012-10-13 13:44 ` Eric Dumazet
2012-10-13 23:11 ` Andrew Savchenko
2012-10-20 23:25 ` Andrew Savchenko
2012-10-21 12:52 ` Eric Dumazet
2012-10-22 3:36 ` Andrew Savchenko
2012-10-22 6:48 ` Eric Dumazet
2012-10-22 21:27 ` Andrew Savchenko
2012-12-12 8:27 ` Andrew Savchenko
2012-12-23 11:06 ` Andrew Savchenko
2012-12-28 18:11 ` Eric Dumazet
2013-01-16 16:36 ` Andrew Savchenko [this message]
2013-02-04 13:39 ` Andrew Savchenko
2013-02-04 15:21 ` Eric Dumazet
2012-11-23 7:45 ` Andrew Savchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130116203644.8eb2a16f.bircoph@gmail.com \
--to=bircoph@gmail.com \
--cc=eric.dumazet@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).