From: Tomasz Mloduchowski <q@qdot.me>
To: netdev@vger.kernel.org
Subject: TCP out of memory - possible bug [3.18.0-rc3] / sched?
Date: Mon, 03 Nov 2014 19:59:33 +0100 [thread overview]
Message-ID: <5457D095.9070502@qdot.me> (raw)
Hi List,
I hope this is the right place to report a networking issue with
3.18.0-rc2 and 3.18.0-rc3 - under heavy P2P load (tested both
rtorrent/libtorrent and bitcoind, so not protocol-specific), the system
quickly exhausts tcp_mem limits in a very strange sequence of events.
It might be scheduler or networking subsystem related.
It's 100% reproducible on my system, first observed under 3.18.0-rc2.
Neither terminating the offending application, nor removing the network
card seems to bring the 'mem' item in /proc/net/sockstat down from it's
extreme values.
http://static.qdot.me/tcp_mem_issue.png contains the plot of the 'mem'
and 'sockets' fields from sockstat - violet is the 'mem', quickly
exhausting the default 512k pages limit after a short period of reliable
operation.
Best Regards,
Tomasz
-- snip --
sched: RT throttling activated
kworker/dying (792) used greatest stack depth: 11984 bytes left
TCP: out of memory -- consider tuning tcp_mem
TCP: out of memory -- consider tuning tcp_mem
------------[ cut here ]------------
WARNING: CPU: 0 PID: 5802 at net/core/stream.c:201
sk_stream_kill_queues+0x12c/0x130()
Modules linked in:
CPU: 0 PID: 5802 Comm: main Not tainted 3.18.0-rc3 #2
Hardware name: LENOVO 37023L0/37023L0, BIOS GFET48WW (1.27 ) 07/01/2014
0000000000000009 ffff8800cae2fd88 ffffffff81b6c0ff 0000000000000000
0000000000000000 ffff8800cae2fdc8 ffffffff810c235c ffff8800cae2fda8
ffff8800bd996580 ffff8800bd996700 0000000000000004 ffff8800bd996610
Call Trace:
[<ffffffff81b6c0ff>] dump_stack+0x46/0x58
[<ffffffff810c235c>] warn_slowpath_common+0x7c/0xa0
[<ffffffff810c2425>] warn_slowpath_null+0x15/0x20
[<ffffffff81954fbc>] sk_stream_kill_queues+0x12c/0x130
[<ffffffff819b8b45>] inet_csk_destroy_sock+0x55/0x140
[<ffffffff819bd9ee>] tcp_close+0x22e/0x430
[<ffffffff819e3b82>] inet_release+0x72/0x80
[<ffffffff819455fa>] sock_release+0x1a/0x90
[<ffffffff8194567d>] sock_close+0xd/0x20
[<ffffffff811dfdf6>] __fput+0xc6/0x1d0
[<ffffffff811dff49>] ____fput+0x9/0x10
[<ffffffff810db94f>] task_work_run+0x8f/0xd0
[<ffffffff81046b12>] do_notify_resume+0x82/0xa0
[<ffffffff81b7613f>] int_signal+0x12/0x17
---[ end trace 080b1124407d2571 ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 5802 at net/ipv4/af_inet.c:153
inet_sock_destruct+0x1d9/0x1e0()
Modules linked in:
CPU: 0 PID: 5802 Comm: main Tainted: G W 3.18.0-rc3 #2
Hardware name: LENOVO 37023L0/37023L0, BIOS GFET48WW (1.27 ) 07/01/2014
0000000000000009 ffff8800cae2fd68 ffffffff81b6c0ff 0000000000000000
0000000000000000 ffff8800cae2fda8 ffffffff810c235c ffff8800cae2fd88
ffff8800bd996580 ffff8800bd996700 0000000000000004 ffff8800bd996610
Call Trace:
[<ffffffff81b6c0ff>] dump_stack+0x46/0x58
[<ffffffff810c235c>] warn_slowpath_common+0x7c/0xa0
[<ffffffff810c2425>] warn_slowpath_null+0x15/0x20
[<ffffffff819e5179>] inet_sock_destruct+0x1d9/0x1e0
[<ffffffff81949e7e>] __sk_free+0x1e/0x100
[<ffffffff81949f79>] sk_free+0x19/0x20
[<ffffffff819bd918>] tcp_close+0x158/0x430
[<ffffffff819e3b82>] inet_release+0x72/0x80
[<ffffffff819455fa>] sock_release+0x1a/0x90
[<ffffffff8194567d>] sock_close+0xd/0x20
[<ffffffff811dfdf6>] __fput+0xc6/0x1d0
[<ffffffff811dff49>] ____fput+0x9/0x10
[<ffffffff810db94f>] task_work_run+0x8f/0xd0
[<ffffffff81046b12>] do_notify_resume+0x82/0xa0
[<ffffffff81b7613f>] int_signal+0x12/0x17
---[ end trace 080b1124407d2572 ]---
kworker/dying (679) used greatest stack depth: 11784 bytes left
TCP: out of memory -- consider tuning tcp_mem
-- snip --
next reply other threads:[~2014-11-03 19:00 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-11-03 18:59 Tomasz Mloduchowski [this message]
2014-11-03 19:41 ` TCP out of memory - possible bug [3.18.0-rc3] / sched? Eric Dumazet
2015-01-04 8:42 ` Markus Trippelsdorf
2015-01-04 21:01 ` Eric Dumazet
2015-01-04 21:39 ` Markus Trippelsdorf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5457D095.9070502@qdot.me \
--to=q@qdot.me \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).