From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Sasha Levin <sasha.levin@oracle.com>
Cc: Dave Jones <davej@redhat.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
htejun@gmail.com
Subject: Re: rcu_preempt detected stalls.
Date: Fri, 24 Oct 2014 09:13:37 -0700 [thread overview]
Message-ID: <20141024161337.GQ4977@linux.vnet.ibm.com> (raw)
In-Reply-To: <544A45F8.2030207@oracle.com>
On Fri, Oct 24, 2014 at 08:28:40AM -0400, Sasha Levin wrote:
> On 10/23/2014 03:58 PM, Paul E. McKenney wrote:
> > On Thu, Oct 23, 2014 at 02:55:43PM -0400, Sasha Levin wrote:
> >> > On 10/23/2014 02:39 PM, Paul E. McKenney wrote:
> >>> > > On Tue, Oct 14, 2014 at 10:35:10PM -0400, Sasha Levin wrote:
> >>>> > >> On 10/13/2014 01:35 PM, Dave Jones wrote:
> >>>>> > >>> oday in "rcu stall while fuzzing" news:
> >>>>> > >>>
> >>>>> > >>> INFO: rcu_preempt detected stalls on CPUs/tasks:
> >>>>> > >>> Tasks blocked on level-0 rcu_node (CPUs 0-3): P766 P646
> >>>>> > >>> Tasks blocked on level-0 rcu_node (CPUs 0-3): P766 P646
> >>>>> > >>> (detected by 0, t=6502 jiffies, g=75434, c=75433, q=0)
> >>>> > >>
> >>>> > >> I've complained about RCU stalls couple days ago (in a different context)
> >>>> > >> on -next. I guess whatever causing them made it into Linus's tree?
> >>>> > >>
> >>>> > >> https://lkml.org/lkml/2014/10/11/64
> >>> > >
> >>> > > And on that one, I must confess that I don't see where the RCU read-side
> >>> > > critical section might be.
> >>> > >
> >>> > > Hmmm... Maybe someone forgot to put an rcu_read_unlock() somewhere.
> >>> > > Can you reproduce this with CONFIG_PROVE_RCU=y?
> >> >
> >> > Paul, if that was directed to me - Yes, I see stalls with CONFIG_PROVE_RCU
> >> > set and nothing else is showing up before/after that.
> > Indeed it was directed to you. ;-)
> >
> > Does the following crude diagnostic patch turn up anything?
>
> Nope, seeing stalls but not seeing that pr_err() you added.
OK, color me confused. Could you please send me the full dmesg or a
pointer to it?
Thanx, Paul
> [ 5107.395916] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [ 5107.395916] 0: (776 ticks this GP) idle=a8d/140000000000002/0 softirq=16356/16356 last_accelerate: f5b7/55e5, nonlazy_posted: 24252, ..
> [ 5107.395916] (detected by 1, t=20502 jiffies, g=13949, c=13948, q=0)
> [ 5107.395916] Task dump for CPU 0:
> [ 5107.395916] trinity-c0 R running task 12848 20357 9041 0x0008000e
> [ 5107.395916] 0000000000000000 ffff88006bfd76c0 ffff88065722b988 ffffffffa10af964
> [ 5107.395916] ffff88065722b998 ffffffffa106ad23 ffff88065722b9c8 ffffffffa119dce5
> [ 5107.395916] 00000000001d76c0 ffff88006bfd76c0 00000000001d76c0 ffff8806473cbd10
> [ 5107.395916] Call Trace:
> [ 5107.395916] [<ffffffffa10af964>] ? kvm_clock_read+0x24/0x40
> [ 5107.395916] [<ffffffffa106ad23>] ? sched_clock+0x13/0x30
> [ 5107.395916] [<ffffffffa119dce5>] ? sched_clock_local+0x25/0x90
> [ 5107.395916] [<ffffffffa1303dfb>] ? __slab_free+0xbb/0x3a0
> [ 5107.395916] [<ffffffffa1b71167>] ? debug_smp_processor_id+0x17/0x20
> [ 5107.395916] [<ffffffffa451cb64>] ? _raw_spin_unlock_irqrestore+0x64/0xa0
> [ 5107.395916] [<ffffffffa1303dfb>] ? __slab_free+0xbb/0x3a0
> [ 5107.395916] [<ffffffffa1b71bce>] ? __debug_check_no_obj_freed+0x10e/0x210
> [ 5107.395916] [<ffffffffa1305871>] ? kmem_cache_free+0xb1/0x4f0
> [ 5107.395916] [<ffffffffa1305883>] ? kmem_cache_free+0xc3/0x4f0
> [ 5107.395916] [<ffffffffa1305bb2>] ? kmem_cache_free+0x3f2/0x4f0
> [ 5107.395916] [<ffffffffa12e0cbe>] ? unlink_anon_vmas+0x10e/0x180
> [ 5107.395916] [<ffffffffa12e0cbe>] ? unlink_anon_vmas+0x10e/0x180
> [ 5107.395916] [<ffffffffa12cfbdf>] ? free_pgtables+0x3f/0x130
> [ 5107.395916] [<ffffffffa12dc1a4>] ? exit_mmap+0xc4/0x180
> [ 5107.395916] [<ffffffffa13143fe>] ? __khugepaged_exit+0xbe/0x120
> [ 5107.395916] [<ffffffffa115bbb3>] ? mmput+0x73/0x110
> [ 5107.395916] [<ffffffffa1162eb7>] ? do_exit+0x2c7/0xd30
> [ 5107.395916] [<ffffffffa1173fb9>] ? get_signal+0x3c9/0xaf0
> [ 5107.395916] [<ffffffffa1b71167>] ? debug_smp_processor_id+0x17/0x20
> [ 5107.395916] [<ffffffffa11bccbe>] ? put_lock_stats.isra.13+0xe/0x30
> [ 5107.395916] [<ffffffffa451c810>] ? _raw_spin_unlock_irq+0x30/0x70
> [ 5107.395916] [<ffffffffa11639c2>] ? do_group_exit+0x52/0xe0
> [ 5107.395916] [<ffffffffa1173ef6>] ? get_signal+0x306/0xaf0
> [ 5107.395916] [<ffffffffa119dce5>] ? sched_clock_local+0x25/0x90
> [ 5107.395916] [<ffffffffa105f2f0>] ? do_signal+0x20/0x130
> [ 5107.395916] [<ffffffffa1298558>] ? context_tracking_user_exit+0x78/0x2d0
> [ 5107.395916] [<ffffffffa1b71183>] ? __this_cpu_preempt_check+0x13/0x20
> [ 5107.395916] [<ffffffffa11c04cb>] ? trace_hardirqs_on_caller+0xfb/0x280
> [ 5107.395916] [<ffffffffa11c065d>] ? trace_hardirqs_on+0xd/0x10
> [ 5107.395916] [<ffffffffa105f469>] ? do_notify_resume+0x69/0xb0
> [ 5107.395916] [<ffffffffa451d74f>] ? int_signal+0x12/0x17
>
>
> Thanks,
> Sasha
>
next prev parent reply other threads:[~2014-10-24 16:17 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-13 17:35 rcu_preempt detected stalls Dave Jones
2014-10-15 2:35 ` Sasha Levin
2014-10-23 18:39 ` Paul E. McKenney
2014-10-23 18:55 ` Sasha Levin
2014-10-23 19:58 ` Paul E. McKenney
2014-10-24 12:28 ` Sasha Levin
2014-10-24 16:13 ` Paul E. McKenney [this message]
2014-10-24 16:39 ` Sasha Levin
2014-10-27 21:13 ` Paul E. McKenney
2014-10-27 23:44 ` Paul E. McKenney
2014-11-13 23:07 ` Paul E. McKenney
2014-11-13 23:10 ` Sasha Levin
2014-10-30 23:41 ` Sasha Levin
2014-10-23 18:32 ` Paul E. McKenney
2014-10-23 18:40 ` Dave Jones
2014-10-23 19:28 ` Paul E. McKenney
2014-10-23 19:37 ` Dave Jones
2014-10-23 19:52 ` Paul E. McKenney
2014-10-23 20:28 ` Dave Jones
2014-10-23 20:44 ` Paul E. McKenney
2014-10-23 19:13 ` Oleg Nesterov
2014-10-23 19:38 ` Paul E. McKenney
2014-10-23 19:53 ` Oleg Nesterov
2014-10-23 20:24 ` Paul E. McKenney
2014-10-23 21:13 ` Oleg Nesterov
2014-10-23 21:38 ` Paul E. McKenney
2014-10-25 3:16 ` Dâniel Fraga
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141024161337.GQ4977@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=davej@redhat.com \
--cc=htejun@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=sasha.levin@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).