From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Nikolay Borisov <kernel@kyup.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Christoph Lameter <cl@linux.com>, Linux MM <linux-mm@kvack.org>,
brouer@redhat.com
Subject: Re: Soft lockup in __slab_free (SLUB)
Date: Thu, 29 Sep 2016 04:10:23 -0700 [thread overview]
Message-ID: <20160929111023.GO14933@linux.vnet.ibm.com> (raw)
In-Reply-To: <57ECF1FA.6010908@kyup.com>
On Thu, Sep 29, 2016 at 01:50:34PM +0300, Nikolay Borisov wrote:
>
>
> On 09/29/2016 01:27 PM, Paul E. McKenney wrote:
> > On Thu, Sep 29, 2016 at 10:11:09AM +0300, Nikolay Borisov wrote:
> [SNIP]
>
> >> What in particular should I be looking for in ftrace? tracing the stacks
> >> on the stuck cpu?
> >
> > To start with, how about the sequence of functions that the stuck
> > CPU is executing?
>
> Unfortunately I do not know how to reproduce the issue, but it is being
> reproduced byt our production load - which is creating backups in this
> case. They are created by rsyncing files to a loop-back attached files
> wihch are then unmounted and unmapped.From this crash it is evident that
> the hang occurs while a volume is being unmounted.
>
> But the callstack is in my hang report, no? I have the crashdump with me
> so if you are interested in anything in particular I can go look for it.
> I believe an inode eviction was requested, since destroy_inode, which
> utilizes ext4_i_callback is called in the eviction + some errors paths.
> And this eviction is executed on this particular CPU. What in particular
> are you looking for?
>
> Unfortunately it's impossible for me to run:
>
> trace-cmd record -p function_graph -F <command that causes the issue>
Given that the hang appears to involve a few common functions, is it
possible to turn ftrace on for those functions (and related ones) at boot
time? Or using sysfs just after boot time?
Given that you cannot reproduce at will, your earlier suggestion of enabling
tracing of stacks might also make a lot of sense.
Thanx, Paul
> [SNIP]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-09-29 11:10 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-09-26 7:46 Soft lockup in __slab_free (SLUB) Nikolay Borisov
2016-09-28 5:31 ` Joonsoo Kim
2016-09-28 7:15 ` Nikolay Borisov
2016-09-28 11:12 ` Paul E. McKenney
2016-09-29 1:40 ` Joonsoo Kim
2016-09-29 2:11 ` Paul E. McKenney
2016-09-29 2:30 ` Nicholas Piggin
2016-09-29 2:47 ` Paul E. McKenney
2016-09-29 3:13 ` Nicholas Piggin
2016-09-29 10:30 ` Paul E. McKenney
2016-09-29 2:55 ` Joonsoo Kim
2016-09-29 7:11 ` Nikolay Borisov
2016-09-29 10:27 ` Paul E. McKenney
2016-09-29 10:50 ` Nikolay Borisov
2016-09-29 11:10 ` Paul E. McKenney [this message]
2016-10-04 14:58 ` Nikolay Borisov
2016-10-04 15:36 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160929111023.GO14933@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=brouer@redhat.com \
--cc=cl@linux.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=kernel@kyup.com \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).