From: Bart Van Assche <Bart.VanAssche@sandisk.com>
To: "tglx@linutronix.de" <tglx@linutronix.de>
Cc: "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
"mingo@kernel.org" <mingo@kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"hpa@zytor.com" <hpa@zytor.com>,
"akpm@linux-foundation.org" <akpm@linux-foundation.org>
Subject: Re: [GIT pull] CPU hotplug updates for 4.9
Date: Fri, 10 Mar 2017 21:32:52 +0000 [thread overview]
Message-ID: <1489181559.2548.9.camel@sandisk.com> (raw)
In-Reply-To: <alpine.DEB.2.20.1703091829530.3521@nanos>
On Thu, 2017-03-09 at 18:43 +0100, Thomas Gleixner wrote:
> On Thu, 9 Mar 2017, Bart Van Assche wrote:
> > INFO: task systemd-udevd:748 blocked for more than 480 seconds.
> > Tainted: G IO 4.11.0-rc1-dbg+ #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > systemd-udevd D 0 748 518 0x00000104
> > Call Trace:
> > __schedule+0x302/0xc30
> > schedule+0x38/0x90
> > schedule_timeout+0x255/0x490
> > wait_for_completion+0x103/0x170
> > cpuhp_issue_call+0xb9/0xe0
> > __cpuhp_setup_state+0xf6/0x180
> > coretemp_init+0x8d/0x1000 [coretemp]
> > do_one_initcall+0x3e/0x170
> > do_init_module+0x5a/0x1ed
> > load_module+0x2339/0x2a40
> > SYSC_finit_module+0xbc/0xf0
> > SyS_finit_module+0x9/0x10
> > do_syscall_64+0x57/0x140
> > entry_SYSCALL64_slow_path+0x25/0x25
> > Showing all locks held in the system:
> > 2 locks held by khungtaskd/91:
> > #0: (rcu_read_lock){......}, at: [<ffffffff8111a6f0>] watchdog+0xa0/0x5d0
> > #1: (tasklist_lock){.+.?..}, at: [<ffffffff810bf36d>] debug_show_all_locks+0x3d/0x1a0
> > 1 lock held by systemd-udevd/748:
> > #0: (cpu_hotplug.dep_map){++++++}, at: [<ffffffff8106712d>] get_online_cpus+0x2d/0x80
>
> Ok, so it's random. Now it would be interesting what the rest of the system
> does when this happens. I still have no idea why that IOAT setting has any
> influence.
Hello Thomas,
The following lines started to appear in the system log during boot after I
started testing kernels after v4.9 and appear before every cpuhp_issue_call()
complaint so this is probably related what causes the hang:
systemd-udevd[553]: seq 2998 '/devices/system/cpu/cpu0' killed
systemd-udevd[553]: seq 3009 '/devices/system/cpu/cpu9' killed
systemd-udevd[553]: seq 3008 '/devices/system/cpu/cpu8' killed
systemd-udevd[553]: seq 3007 '/devices/system/cpu/cpu7' killed
systemd-udevd[553]: seq 2999 '/devices/system/cpu/cpu1' killed
systemd-udevd[553]: seq 3006 '/devices/system/cpu/cpu6' killed
systemd-udevd[553]: seq 3001 '/devices/system/cpu/cpu11' killed
systemd-udevd[553]: seq 3004 '/devices/system/cpu/cpu4' killed
systemd-udevd[553]: seq 3003 '/devices/system/cpu/cpu3' killed
systemd-udevd[553]: seq 3002 '/devices/system/cpu/cpu2' killed
systemd-udevd[553]: seq 3005 '/devices/system/cpu/cpu5' killed
systemd-udevd[553]: seq 3000 '/devices/system/cpu/cpu10' killed
systemd-udevd[553]: worker [748] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [748] failed while handling '/devices/system/cpu/cpu4'
systemd-udevd[553]: worker [710] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [710] failed while handling '/devices/system/cpu/cpu1'
systemd-udevd[553]: worker [750] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [750] failed while handling '/devices/system/cpu/cpu5'
systemd-udevd[553]: worker [690] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [690] failed while handling '/devices/system/cpu/cpu0'
systemd-udevd[553]: worker [747] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [747] failed while handling '/devices/system/cpu/cpu8'
systemd-udevd[553]: worker [770] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [770] failed while handling '/devices/system/cpu/cpu2'
systemd-udevd[553]: worker [772] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [772] failed while handling '/devices/system/cpu/cpu10'
systemd-udevd[553]: worker [774] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [774] failed while handling '/devices/system/cpu/cpu6'
systemd-udevd[553]: worker [782] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [782] failed while handling '/devices/system/cpu/cpu9'
systemd-udevd[553]: worker [807] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [807] failed while handling '/devices/system/cpu/cpu11'
systemd-udevd[553]: worker [816] terminated by signal 9 (Killed)
systemd-udevd[553]: worker [816] failed while handling '/devices/system/cpu/cpu7'
Since this seems to reproduce easier than the hang I will try to use this to
guide a new bisect attempt.
Bart.
next prev parent reply other threads:[~2017-03-10 21:33 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-03 17:37 [GIT pull] CPU hotplug updates for 4.9 Thomas Gleixner
2017-03-07 1:52 ` Bart Van Assche
2017-03-07 9:30 ` Thomas Gleixner
2017-03-07 22:32 ` Bart Van Assche
2017-03-08 8:25 ` Thomas Gleixner
2017-03-08 19:32 ` Bart Van Assche
2017-03-09 10:22 ` Thomas Gleixner
2017-03-09 17:20 ` Bart Van Assche
2017-03-09 17:43 ` Thomas Gleixner
2017-03-09 22:48 ` Bart Van Assche
2017-03-10 7:23 ` Ingo Molnar
2017-03-10 17:37 ` Bart Van Assche
2017-03-10 21:32 ` Bart Van Assche [this message]
2017-03-14 15:06 ` [PATCH] cpu/hotplug: Serialize callback invocations proper Sebastian Andrzej Siewior
2017-03-14 17:38 ` Bart Van Assche
2017-03-14 17:43 ` Thomas Gleixner
2017-03-14 18:25 ` [tip:smp/urgent] " tip-bot for Sebastian Andrzej Siewior
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1489181559.2548.9.camel@sandisk.com \
--to=bart.vanassche@sandisk.com \
--cc=akpm@linux-foundation.org \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox