From: Philipp Hahn <pmhahn@pmhahn.de>
To: Sasha Levin <sasha.levin@oracle.com>,
Hannes Frederic Sowa <hannes@stressinduktion.org>,
Rainer Weikusat <rweikusat@mobileactivedefense.com>,
Andrey Vagin <avagin@openvz.org>,
Aaron Conole <aconole@bytheb.org>,
"David S. Miller" <davem@davemloft.net>,
linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Bug 4.1.16: self-detected stall in net/unix/?
Date: Tue, 2 Feb 2016 17:25:46 +0100 [thread overview]
Message-ID: <56B0D88A.1020609@pmhahn.de> (raw)
Hi,
we recently updated our kernel to 4.1.16 + patch for "unix: properly
account for FDs passed over unix sockets" and have since then
self-detected stalls triggered by the Samba daemon:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] INFO: rcu_sched self-detected stall on CPU { 3} (t=162780 jiffies g=47565 c=47564 q=1055670)
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] Task dump for CPU 3:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] smbd R running task 0 5938 1 0x0000000c
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] 0000000000000004 ffffffff81851340 ffffffff810d3c84 000000000000b9cd
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] ffff8801bfd97100 ffffffff81851340 ffffffff81851340 ffffffff818f6c60
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] ffffffff810d7659 0000000000000000 0000000000000000 00001e847fc2f700
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] Call Trace:
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] <IRQ> [<ffffffff810d3c84>] ? rcu_dump_cpu_stacks+0x84/0xc0
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810d7659>] ? rcu_check_callbacks+0x449/0x740
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec7c0>] ? tick_sched_do_timer+0x40/0x40
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810dcc54>] ? update_process_times+0x34/0x70
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec45c>] ? tick_sched_handle.isra.12+0x2c/0x70
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ec809>] ? tick_sched_timer+0x49/0x80
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810dd57d>] ? __run_hrtimer+0x6d/0x1b0
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff810ddd4d>] ? hrtimer_interrupt+0xed/0x210
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff815a0ed9>] ? smp_apic_timer_interrupt+0x39/0x50
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8159ef7e>] ? apic_timer_interrupt+0x6e/0x80
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] <EOI> [<ffffffff8159de85>] ? _raw_spin_lock+0x35/0x50
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8153b343>] ? unix_dgram_connect+0x93/0x200
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8147f248>] ? SYSC_connect+0xe8/0x100
> Feb 1 09:03:14 dcs1 kernel: [ 1152.840007] [<ffffffff8159e0f2>] ? system_call_fast_compare_end+0xc/0x6b
> Feb 1 11:48:13 ucs22f kernel: [307999.162254] INFO: rcu_sched self-detected stall on CPU { 0} (t=5250 jiffies g=6586733 c=6586732 q=6757)
> Feb 1 11:48:13 ucs22f kernel: [307999.162264] Task dump for CPU 0:
> Feb 1 11:48:13 ucs22f kernel: [307999.162267] smbd R running 0 4615 4609 0x00000008
> Feb 1 11:48:13 ucs22f kernel: [307999.162272] 00200082 f5863b90 c10b3fe9 c1682cc0 c1682cc0 c1682cc0 f79d2b00 f 5863bdc
> Feb 1 11:48:13 ucs22f kernel: [307999.162276] c10b722d c15bd400 00001482 0064816d 0064816c 00001a65 f79cd840 c 108b166
> Feb 1 11:48:13 ucs22f kernel: [307999.162280] 00000001 f5863bb8 f5863bb8 00000000 c1682cc0 f6808cf0 00001a65 f 6808cf0
> Feb 1 11:48:13 ucs22f kernel: [307999.162285] Call Trace:
> Feb 1 11:48:13 ucs22f kernel: [307999.162296] [<c10b3fe9>] ? rcu_dump_cpu_stacks+0x79/0xc0
> Feb 1 11:48:13 ucs22f kernel: [307999.162300] [<c10b722d>] ? rcu_check_callbacks+0x3cd/0x630
> Feb 1 11:48:13 ucs22f kernel: [307999.162304] [<c108b166>] ? account_process_tick+0x66/0x160
> Feb 1 11:48:13 ucs22f kernel: [307999.162307] [<c10bbe4f>] ? update_process_times+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162310] [<c10cbf9d>] ? tick_sched_handle.isra.12+0x2d/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162328] [<c10cc210>] ? tick_sched_timer+0x40/0x80
> Feb 1 11:48:13 ucs22f kernel: [307999.162331] [<c10bc6b0>] ? __remove_hrtimer+0x40/0xa0
> Feb 1 11:48:13 ucs22f kernel: [307999.162334] [<c10bc97f>] ? __run_hrtimer+0x6f/0x190
> Feb 1 11:48:13 ucs22f kernel: [307999.162337] [<c10cc1d0>] ? tick_sched_do_timer+0x30/0x30
> Feb 1 11:48:13 ucs22f kernel: [307999.162339] [<c10bd15f>] ? hrtimer_interrupt+0xef/0x260
> Feb 1 11:48:13 ucs22f kernel: [307999.162343] [<c119ae3d>] ? getname_kernel+0x2d/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162348] [<c1046f7f>] ? local_apic_timer_interrupt+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162353] [<c14e4543>] ? smp_apic_timer_interrupt+0x33/0x50
> Feb 1 11:48:13 ucs22f kernel: [307999.162355] [<c14e3c7c>] ? apic_timer_interrupt+0x34/0x3c
> Feb 1 11:48:13 ucs22f kernel: [307999.162358] [<c14e2dc1>] ? _raw_spin_lock+0x51/0x70
> Feb 1 11:48:13 ucs22f kernel: [307999.162362] [<c148c075>] ? unix_state_double_lock+0x25/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162365] [<c148de10>] ? unix_dgram_connect+0x90/0x1f0
> Feb 1 11:48:13 ucs22f kernel: [307999.162369] [<c13e4267>] ? SYSC_connect+0xc7/0xe0
> Feb 1 11:48:13 ucs22f kernel: [307999.162371] [<c13e2931>] ? sock_map_fd+0x41/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162374] [<c13e5014>] ? SYSC_socketcall+0x1b4/0xa20
> Feb 1 11:48:13 ucs22f kernel: [307999.162376] [<c10c2940>] ? ktime_get+0x50/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162379] [<c10466db>] ? lapic_next_event+0x1b/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162381] [<c10ca0ed>] ? clockevents_program_event+0x9d/0x140
> Feb 1 11:48:13 ucs22f kernel: [307999.162385] [<c129e068>] ? list_del+0x8/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162388] [<c1097ef7>] ? remove_wait_queue+0x27/0x40
> Feb 1 11:48:13 ucs22f kernel: [307999.162392] [<c11c8795>] ? inotify_read+0x295/0x340
> Feb 1 11:48:13 ucs22f kernel: [307999.162396] [<c10acc76>] ? handle_irq_event_percpu+0xa6/0x1a0
> Feb 1 11:48:13 ucs22f kernel: [307999.162399] [<c11a786f>] ? set_close_on_exec+0x2f/0x60
> Feb 1 11:48:13 ucs22f kernel: [307999.162402] [<c119d084>] ? do_fcntl+0x2f4/0x4e0
> Feb 1 11:48:13 ucs22f kernel: [307999.162405] [<c107d6df>] ? commit_creds+0xff/0x1f0
> Feb 1 11:48:13 ucs22f kernel: [307999.162407] [<c119d380>] ? SyS_fcntl64+0x60/0x100
> Feb 1 11:48:13 ucs22f kernel: [307999.162409] [<c13e5953>] ? SyS_socketcall+0x13/0x20
> Feb 1 11:48:13 ucs22f kernel: [307999.162412] [<c14e30db>] ? sysenter_do_call+0x12/0x12
We have not yet been able to reproduce the hang, but going back to our
previous kernel 4.1.12 makes the problem go away.
Is this a known issue or do you have an idea where to look?
What information should I collect next time it happens?
(Can unix_diag.ko with `ss` help?)
What other kernel configs should I enable do debug this dead-lock?
Thanks in advance
Philipp
next reply other threads:[~2016-02-02 16:31 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-02 16:25 Philipp Hahn [this message]
2016-02-03 1:43 ` Bug 4.1.16: self-detected stall in net/unix/? Hannes Frederic Sowa
2016-02-05 15:28 ` Philipp Hahn
2016-02-11 13:47 ` Philipp Hahn
2016-02-11 15:55 ` Rainer Weikusat
2016-02-11 17:03 ` Ben Hutchings
2016-02-11 17:40 ` Rainer Weikusat
2016-02-11 17:54 ` Rainer Weikusat
2016-02-11 18:31 ` Rainer Weikusat
2016-02-11 19:37 ` [PATCH net] af_unix: Guard against other == sk in unix_dgram_sendmsg Rainer Weikusat
2016-02-12 9:19 ` Philipp Hahn
2016-02-12 13:25 ` Rainer Weikusat
2016-02-12 19:54 ` Ben Hutchings
2016-02-12 20:17 ` Rainer Weikusat
2016-02-12 20:47 ` Ben Hutchings
2016-02-12 20:59 ` Rainer Weikusat
2016-02-16 17:54 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56B0D88A.1020609@pmhahn.de \
--to=pmhahn@pmhahn.de \
--cc=aconole@bytheb.org \
--cc=avagin@openvz.org \
--cc=davem@davemloft.net \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@stressinduktion.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rweikusat@mobileactivedefense.com \
--cc=sasha.levin@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.