From: Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org>
To: Shawn Bohrer <shawn.bohrer-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: Li Zefan <lizefan-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>,
cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
Hugh Dickins <hughd-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>,
Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org>,
Markus Blank-Burian
<burian-iYtK5bfT9M8b1SvskN2V4Q@public.gmane.org>
Subject: Re: 3.10.16 cgroup_mutex deadlock
Date: Tue, 12 Nov 2013 15:31:47 +0100 [thread overview]
Message-ID: <20131112143147.GB6049@dhcp22.suse.cz> (raw)
In-Reply-To: <52820030.6000806-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
On Tue 12-11-13 18:17:20, Li Zefan wrote:
> Cc more people
>
> On 2013/11/12 6:06, Shawn Bohrer wrote:
> > Hello,
> >
> > This morning I had a machine running 3.10.16 go unresponsive but
> > before we killed it we were able to get the information below. I'm
> > not an expert here but it looks like most of the tasks below are
> > blocking waiting on the cgroup_mutex. You can see that the
> > resource_alloca:16502 task is holding the cgroup_mutex and that task
> > appears to be waiting on a lru_add_drain_all() to complete.
Do you have sysrq+l output as well by any chance? That would tell
us what the current CPUs are doing. Dumping all kworker stacks
might be helpful as well. We know that lru_add_drain_all waits for
schedule_on_each_cpu to return so it is waiting for workers to finish.
I would be really curious why some of lru_add_drain_cpu cannot finish
properly. The only reason would be that some work item(s) do not get CPU
or somebody is holding lru_lock.
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2013-11-12 14:31 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-11-11 22:06 3.10.16 cgroup_mutex deadlock Shawn Bohrer
[not found] ` <20131111220626.GA7509-/vebjAlq/uFE7V8Yqttd03bhEEblAqRIDbRjUBewulXQT0dZR+AlfA@public.gmane.org>
2013-11-12 10:17 ` Li Zefan
[not found] ` <52820030.6000806-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
2013-11-12 14:31 ` Michal Hocko [this message]
2013-11-12 15:55 ` Shawn Bohrer
[not found] ` <20131112155530.GA2860-/vebjAlq/uFE7V8Yqttd03bhEEblAqRIDbRjUBewulXQT0dZR+AlfA@public.gmane.org>
2013-11-12 16:55 ` Michal Hocko
[not found] ` <20131112165504.GF6049-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
2013-11-14 22:56 ` Shawn Bohrer
[not found] ` <20131114225649.GA16725-/vebjAlq/uFE7V8Yqttd03bhEEblAqRIDbRjUBewulXQT0dZR+AlfA@public.gmane.org>
2013-11-15 6:24 ` Tejun Heo
[not found] ` <20131115062458.GA9755-9pTldWuhBndy/B6EtB590w@public.gmane.org>
2013-11-15 7:54 ` Tejun Heo
[not found] ` <20131115075401.GB9755-9pTldWuhBndy/B6EtB590w@public.gmane.org>
2013-11-18 2:17 ` Hugh Dickins
[not found] ` <alpine.LNX.2.00.1311171746160.15789-fupSdm12i1nKWymIFiNcPA@public.gmane.org>
2013-11-18 20:10 ` Shawn Bohrer
[not found] ` <20131118201025.GA2747-/vebjAlq/uFE7V8Yqttd03bhEEblAqRIDbRjUBewulXQT0dZR+AlfA@public.gmane.org>
2013-11-19 2:55 ` Li Zefan
[not found] ` <528AD316.10001-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
2013-11-20 22:47 ` Shawn Bohrer
2013-11-22 20:59 ` William Dauchy
[not found] ` <CAJ75kXZYjKwV_XiEB493jNyGRqS395JZyY-S9xQBQJLyaCSOEQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-11-22 22:18 ` Tejun Heo
[not found] ` <20131122221839.GD8981-9pTldWuhBndy/B6EtB590w@public.gmane.org>
2013-11-22 22:54 ` William Dauchy
[not found] ` <CAJ75kXabrnxqdtb5SXqm_pYTrSih9yvP38DApF+_P+YZCepTMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-11-25 1:20 ` Li Zefan
[not found] ` <5292A5EA.1030501-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
2013-12-02 10:31 ` William Dauchy
2013-12-03 1:37 ` Li Zefan
2013-11-22 22:17 ` [PATCH cgroup/for-3.13-fixes] cgroup: use a dedicated workqueue for cgroup destruction Tejun Heo
2013-11-24 18:23 ` Hugh Dickins
[not found] ` <20131122221752.GC8981-9pTldWuhBndy/B6EtB590w@public.gmane.org>
2013-11-25 1:16 ` Li Zefan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131112143147.GB6049@dhcp22.suse.cz \
--to=mhocko-alswssmvlrq@public.gmane.org \
--cc=burian-iYtK5bfT9M8b1SvskN2V4Q@public.gmane.org \
--cc=cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org \
--cc=hughd-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org \
--cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=lizefan-hv44wF8Li93QT0dZR+AlfA@public.gmane.org \
--cc=shawn.bohrer-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox