From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Arun KS <arunks.linux@gmail.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
josh@joshtriplett.org, rostedt@goodmis.org,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
laijs@cn.fujitsu.com, Arun KS <getarunks@gmail.com>
Subject: Re: [RCU] kernel hangs in wait_rcu_gp during suspend path
Date: Thu, 18 Dec 2014 12:05:06 -0800 [thread overview]
Message-ID: <20141218200506.GL11609@linux.vnet.ibm.com> (raw)
In-Reply-To: <CAKZGPAPNpci1UriB=vtT3OHcVQnMNtwi7qbT+d_Uqvp+A_xENw@mail.gmail.com>
On Thu, Dec 18, 2014 at 09:52:28PM +0530, Arun KS wrote:
> Hi Paul,
>
> On Thu, Dec 18, 2014 at 12:54 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Tue, Dec 16, 2014 at 11:00:20PM +0530, Arun KS wrote:
> >> Hello,
> >>
> >> Adding some more info.
> >>
> >> Below is the rcu_data data structure corresponding to cpu4.
> >
> > This shows that RCU is idle. What was the state of the system at the
> > time you collected this data?
>
> System initiated a suspend sequence and currently at disable_nonboot_cpus().
> It has hotplugged 0, 1 and 2 successfully. And even successful in hot
> plugging cpu3.
> But while calling the CPU_POST_DEAD notifier for cpu3, another driver tried to
> unregister an atomic notifier. Which eventually calls syncronize_rcu()
> which hangs the suspend task.
>
> bt as follows,
> PID: 202 TASK: edcd2a00 CPU: 4 COMMAND: "kworker/u16:4"
> #0 [<c0a1f8c0>] (__schedule) from [<c0a1d054>]
> #1 [<c0a1d054>] (schedule_timeout) from [<c0a1f018>]
> #2 [<c0a1f018>] (wait_for_common) from [<c013c570>]
> #3 [<c013c570>] (wait_rcu_gp) from [<c014407c>]
> #4 [<c014407c>] (atomic_notifier_chain_unregister) from [<c06be62c>]
> #5 [<c06be62c>] (cpufreq_interactive_disable_sched_input) from [<c06bee1c>]
> #6 [<c06bee1c>] (cpufreq_governor_interactive) from [<c06b7724>]
> #7 [<c06b7724>] (__cpufreq_governor) from [<c06b9f74>]
> #8 [<c06b9f74>] (__cpufreq_remove_dev_finish) from [<c06ba3a4>]
> #9 [<c06ba3a4>] (cpufreq_cpu_callback) from [<c0a22674>]
> #10 [<c0a22674>] (notifier_call_chain) from [<c012284c>]
> #11 [<c012284c>] (__cpu_notify) from [<c01229dc>]
> #12 [<c01229dc>] (cpu_notify_nofail) from [<c0a0dd1c>]
> #13 [<c0a0dd1c>] (_cpu_down) from [<c0122b48>]
> #14 [<c0122b48>] (disable_nonboot_cpus) from [<c0168cd8>]
> #15 [<c0168cd8>] (suspend_devices_and_enter) from [<c0169018>]
> #16 [<c0169018>] (pm_suspend) from [<c01691e0>]
> #17 [<c01691e0>] (try_to_suspend) from [<c01396f0>]
> #18 [<c01396f0>] (process_one_work) from [<c0139db0>]
> #19 [<c0139db0>] (worker_thread) from [<c013efa4>]
> #20 [<c013efa4>] (kthread) from [<c01061f8>]
>
> But the other cores 4-7 are active. I can see them going to idle tasks
> coming out from idle because of interrupts, scheduling kworkers etc.
> So when I took the data, all the online cores(4-7) were in idle as
> shown below from runq data structures.
>
> ------start--------------
> crash> runq
> CPU 0 [OFFLINE]
>
> CPU 1 [OFFLINE]
>
> CPU 2 [OFFLINE]
>
> CPU 3 [OFFLINE]
>
> CPU 4 RUNQUEUE: c5439040
> CURRENT: PID: 0 TASK: f0c9d400 COMMAND: "swapper/4"
> RT PRIO_ARRAY: c5439130
> [no tasks queued]
> CFS RB_ROOT: c54390c0
> [no tasks queued]
>
> CPU 5 RUNQUEUE: c5447040
> CURRENT: PID: 0 TASK: f0c9aa00 COMMAND: "swapper/5"
> RT PRIO_ARRAY: c5447130
> [no tasks queued]
> CFS RB_ROOT: c54470c0
> [no tasks queued]
>
> CPU 6 RUNQUEUE: c5455040
> CURRENT: PID: 0 TASK: f0c9ce00 COMMAND: "swapper/6"
> RT PRIO_ARRAY: c5455130
> [no tasks queued]
> CFS RB_ROOT: c54550c0
> [no tasks queued]
>
> CPU 7 RUNQUEUE: c5463040
> CURRENT: PID: 0 TASK: f0c9b000 COMMAND: "swapper/7"
> RT PRIO_ARRAY: c5463130
> [no tasks queued]
> CFS RB_ROOT: c54630c0
> [no tasks queued]
> ------end--------------
>
> but one strange thing i can see is that rcu_read_lock_nesting for idle
> tasks running on cpu 5 and cpu 6 are set to 1.
>
> PID: 0 TASK: f0c9d400 CPU: 4 COMMAND: "swapper/4"
> rcu_read_lock_nesting = 0,
>
> PID: 0 TASK: f0c9aa00 CPU: 5 COMMAND: "swapper/5"
> rcu_read_lock_nesting = 1,
>
> PID: 0 TASK: f0c9ce00 CPU: 6 COMMAND: "swapper/6"
> rcu_read_lock_nesting = 1,
>
> PID: 0 TASK: f0c9b000 CPU: 7 COMMAND: "swapper/7"
> rcu_read_lock_nesting = 0,
>
> Does this means that the current grace period(suspend thread is
> waiting on) is getting extended infinitely?
Indeed it does, good catch! Looks like someone entered an RCU read-side
critical section, then forgot to exit it, which would prevent grace
periods from ever completing. CONFIG_PROVE_RCU=y might be
helpful in tracking this down.
> Also attaching the per_cpu rcu_data for online and offline cores.
But these still look like there is no grace period in progress.
Still, it would be good to try CONFIG_PROVE_RCU=y and see what it
shows you.
Also, I am not seeing similar complaints about 3.10, so it is quite
possible that a recent change in the ARM-specific idle-loop code is
doing this to you. It might be well worth looking through recent
changes in this area, particularly if some older version works well
for you.
Thanx, Paul
> Thanks,
> Arun
>
> >
> > Thanx, Paul
> >
> >> struct rcu_data {
> >> completed = 5877,
> >> gpnum = 5877,
> >> passed_quiesce = true,
> >> qs_pending = false,
> >> beenonline = true,
> >> preemptible = true,
> >> mynode = 0xc117f340 <rcu_preempt_state>,
> >> grpmask = 16,
> >> nxtlist = 0xedaaec00,
> >> nxttail = {0xc54366c4, 0xe84d350c, 0xe84d350c, 0xe84d350c},
> >> nxtcompleted = {4294967035, 5878, 5878, 5878},
> >> qlen_lazy = 105,
> >> qlen = 415,
> >> qlen_last_fqs_check = 0,
> >> n_cbs_invoked = 86323,
> >> n_nocbs_invoked = 0,
> >> n_cbs_orphaned = 0,
> >> n_cbs_adopted = 139,
> >> n_force_qs_snap = 0,
> >> blimit = 10,
> >> dynticks = 0xc5436758,
> >> dynticks_snap = 7582140,
> >> dynticks_fqs = 41,
> >> offline_fqs = 0,
> >> n_rcu_pending = 59404,
> >> n_rp_qs_pending = 5,
> >> n_rp_report_qs = 4633,
> >> n_rp_cb_ready = 32,
> >> n_rp_cpu_needs_gp = 41088,
> >> n_rp_gp_completed = 2844,
> >> n_rp_gp_started = 1150,
> >> n_rp_need_nothing = 9657,
> >> barrier_head = {
> >> next = 0x0,
> >> func = 0x0
> >> },
> >> oom_head = {
> >> next = 0x0,
> >> func = 0x0
> >> },
> >> cpu = 4,
> >> rsp = 0xc117f340 <rcu_preempt_state>
> >> }
> >>
> >>
> >>
> >> Also pasting complete rcu_preempt_state.
> >>
> >>
> >>
> >> rcu_preempt_state = $9 = {
> >> node = {{
> >> lock = {
> >> raw_lock = {
> >> {
> >> slock = 3129850509,
> >> tickets = {
> >> owner = 47757,
> >> next = 47757
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> },
> >> gpnum = 5877,
> >> completed = 5877,
> >> qsmask = 0,
> >> expmask = 0,
> >> qsmaskinit = 240,
> >> grpmask = 0,
> >> grplo = 0,
> >> grphi = 7,
> >> grpnum = 0 '\000',
> >> level = 0 '\000',
> >> parent = 0x0,
> >> blkd_tasks = {
> >> next = 0xc117f378 <rcu_preempt_state+56>,
> >> prev = 0xc117f378 <rcu_preempt_state+56>
> >> },
> >> gp_tasks = 0x0,
> >> exp_tasks = 0x0,
> >> need_future_gp = {1, 0},
> >> fqslock = {
> >> raw_lock = {
> >> {
> >> slock = 0,
> >> tickets = {
> >> owner = 0,
> >> next = 0
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> }
> >> }},
> >> level = {0xc117f340 <rcu_preempt_state>},
> >> levelcnt = {1, 0, 0, 0, 0},
> >> levelspread = "\b",
> >> rda = 0xc115e6b0 <rcu_preempt_data>,
> >> call = 0xc01975ac <call_rcu>,
> >> fqs_state = 0 '\000',
> >> boost = 0 '\000',
> >> gpnum = 5877,
> >> completed = 5877,
> >> gp_kthread = 0xf0c9e600,
> >> gp_wq = {
> >> lock = {
> >> {
> >> rlock = {
> >> raw_lock = {
> >> {
> >> slock = 2160230594,
> >> tickets = {
> >> owner = 32962,
> >> next = 32962
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> }
> >> }
> >> },
> >> task_list = {
> >> next = 0xf0cd1f20,
> >> prev = 0xf0cd1f20
> >> }
> >> },
> >> gp_flags = 1,
> >> orphan_lock = {
> >> raw_lock = {
> >> {
> >> slock = 327685,
> >> tickets = {
> >> owner = 5,
> >> next = 5
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> },
> >> orphan_nxtlist = 0x0,
> >> orphan_nxttail = 0xc117f490 <rcu_preempt_state+336>,
> >> orphan_donelist = 0x0,
> >> orphan_donetail = 0xc117f498 <rcu_preempt_state+344>,
> >> qlen_lazy = 0,
> >> qlen = 0,
> >> onoff_mutex = {
> >> count = {
> >> counter = 1
> >> },
> >> wait_lock = {
> >> {
> >> rlock = {
> >> raw_lock = {
> >> {
> >> slock = 811479134,
> >> tickets = {
> >> owner = 12382,
> >> next = 12382
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> }
> >> }
> >> },
> >> wait_list = {
> >> next = 0xc117f4bc <rcu_preempt_state+380>,
> >> prev = 0xc117f4bc <rcu_preempt_state+380>
> >> },
> >> owner = 0x0,
> >> name = 0x0,
> >> magic = 0xc117f4a8 <rcu_preempt_state+360>
> >> },
> >> barrier_mutex = {
> >> count = {
> >> counter = 1
> >> },
> >> wait_lock = {
> >> {
> >> rlock = {
> >> raw_lock = {
> >> {
> >> slock = 0,
> >> tickets = {
> >> owner = 0,
> >> next = 0
> >> }
> >> }
> >> },
> >> magic = 3735899821,
> >> owner_cpu = 4294967295,
> >> owner = 0xffffffff
> >> }
> >> }
> >> },
> >> wait_list = {
> >> next = 0xc117f4e4 <rcu_preempt_state+420>,
> >> prev = 0xc117f4e4 <rcu_preempt_state+420>
> >> },
> >> owner = 0x0,
> >> name = 0x0,
> >> magic = 0xc117f4d0 <rcu_preempt_state+400>
> >> },
> >> barrier_cpu_count = {
> >> counter = 0
> >> },
> >> barrier_completion = {
> >> done = 0,
> >> wait = {
> >> lock = {
> >> {
> >> rlock = {
> >> raw_lock = {
> >> {
> >> slock = 0,
> >> tickets = {
> >> owner = 0,
> >> next = 0
> >> }
> >> }
> >> },
> >> magic = 0,
> >> owner_cpu = 0,
> >> owner = 0x0
> >> }
> >> }
> >> },
> >> task_list = {
> >> next = 0x0,
> >> prev = 0x0
> >> }
> >> }
> >> },
> >> n_barrier_done = 0,
> >> expedited_start = {
> >> counter = 0
> >> },
> >> expedited_done = {
> >> counter = 0
> >> },
> >> expedited_wrap = {
> >> counter = 0
> >> },
> >> expedited_tryfail = {
> >> counter = 0
> >> },
> >> expedited_workdone1 = {
> >> counter = 0
> >> },
> >> expedited_workdone2 = {
> >> counter = 0
> >> },
> >> expedited_normal = {
> >> counter = 0
> >> },
> >> expedited_stoppedcpus = {
> >> counter = 0
> >> },
> >> expedited_done_tries = {
> >> counter = 0
> >> },
> >> expedited_done_lost = {
> >> counter = 0
> >> },
> >> expedited_done_exit = {
> >> counter = 0
> >> },
> >> jiffies_force_qs = 4294963917,
> >> n_force_qs = 4028,
> >> n_force_qs_lh = 0,
> >> n_force_qs_ngp = 0,
> >> gp_start = 4294963911,
> >> jiffies_stall = 4294966011,
> >> gp_max = 17,
> >> name = 0xc0d833ab "rcu_preempt",
> >> abbr = 112 'p',
> >> flavors = {
> >> next = 0xc117f2ec <rcu_bh_state+556>,
> >> prev = 0xc117f300 <rcu_struct_flavors>
> >> },
> >> wakeup_work = {
> >> flags = 3,
> >> llnode = {
> >> next = 0x0
> >> },
> >> func = 0xc0195aa8 <rsp_wakeup>
> >> }
> >> }
> >>
> >> Hope this helps.
> >>
> >> Thanks,
> >> Arun
> >>
> >>
> >> On Tue, Dec 16, 2014 at 11:59 AM, Arun KS <arunks.linux@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > I dig little deeper to understand the situation.
> >> > All other cpus are in idle thread already.
> >> > As per my understanding, for the grace period to end, at-least one of
> >> > the following should happen on all online cpus,
> >> >
> >> > 1. a context switch.
> >> > 2. user space switch.
> >> > 3. switch to idle thread.
> >> >
> >> > In this situation, since all the other cores are already in idle, non
> >> > of the above are meet on all online cores.
> >> > So grace period is getting extended and never finishes. Below is the
> >> > state of runqueue when the hang happens.
> >> > --------------start------------------------------------
> >> > crash> runq
> >> > CPU 0 [OFFLINE]
> >> >
> >> > CPU 1 [OFFLINE]
> >> >
> >> > CPU 2 [OFFLINE]
> >> >
> >> > CPU 3 [OFFLINE]
> >> >
> >> > CPU 4 RUNQUEUE: c3192e40
> >> > CURRENT: PID: 0 TASK: f0874440 COMMAND: "swapper/4"
> >> > RT PRIO_ARRAY: c3192f20
> >> > [no tasks queued]
> >> > CFS RB_ROOT: c3192eb0
> >> > [no tasks queued]
> >> >
> >> > CPU 5 RUNQUEUE: c31a0e40
> >> > CURRENT: PID: 0 TASK: f0874980 COMMAND: "swapper/5"
> >> > RT PRIO_ARRAY: c31a0f20
> >> > [no tasks queued]
> >> > CFS RB_ROOT: c31a0eb0
> >> > [no tasks queued]
> >> >
> >> > CPU 6 RUNQUEUE: c31aee40
> >> > CURRENT: PID: 0 TASK: f0874ec0 COMMAND: "swapper/6"
> >> > RT PRIO_ARRAY: c31aef20
> >> > [no tasks queued]
> >> > CFS RB_ROOT: c31aeeb0
> >> > [no tasks queued]
> >> >
> >> > CPU 7 RUNQUEUE: c31bce40
> >> > CURRENT: PID: 0 TASK: f0875400 COMMAND: "swapper/7"
> >> > RT PRIO_ARRAY: c31bcf20
> >> > [no tasks queued]
> >> > CFS RB_ROOT: c31bceb0
> >> > [no tasks queued]
> >> > --------------end------------------------------------
> >> >
> >> > If my understanding is correct the below patch should help, because it
> >> > will expedite grace periods during suspend,
> >> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d1d74d14e98a6be740a6f12456c7d9ad47be9c9c
> >> >
> >> > But I wonder why it was not taken to stable trees. Can we take it?
> >> > Appreciate your help.
> >> >
> >> > Thanks,
> >> > Arun
> >> >
> >> > On Mon, Dec 15, 2014 at 10:34 PM, Arun KS <arunks.linux@gmail.com> wrote:
> >> >> Hi,
> >> >>
> >> >> Here is the backtrace of the process hanging in wait_rcu_gp,
> >> >>
> >> >> PID: 247 TASK: e16e7380 CPU: 4 COMMAND: "kworker/u16:5"
> >> >> #0 [<c09fead0>] (__schedule) from [<c09fcab0>]
> >> >> #1 [<c09fcab0>] (schedule_timeout) from [<c09fe050>]
> >> >> #2 [<c09fe050>] (wait_for_common) from [<c013b2b4>]
> >> >> #3 [<c013b2b4>] (wait_rcu_gp) from [<c0142f50>]
> >> >> #4 [<c0142f50>] (atomic_notifier_chain_unregister) from [<c06b2ab8>]
> >> >> #5 [<c06b2ab8>] (cpufreq_interactive_disable_sched_input) from [<c06b32a8>]
> >> >> #6 [<c06b32a8>] (cpufreq_governor_interactive) from [<c06abbf8>]
> >> >> #7 [<c06abbf8>] (__cpufreq_governor) from [<c06ae474>]
> >> >> #8 [<c06ae474>] (__cpufreq_remove_dev_finish) from [<c06ae8c0>]
> >> >> #9 [<c06ae8c0>] (cpufreq_cpu_callback) from [<c0a0185c>]
> >> >> #10 [<c0a0185c>] (notifier_call_chain) from [<c0121888>]
> >> >> #11 [<c0121888>] (__cpu_notify) from [<c0121a04>]
> >> >> #12 [<c0121a04>] (cpu_notify_nofail) from [<c09ee7f0>]
> >> >> #13 [<c09ee7f0>] (_cpu_down) from [<c0121b70>]
> >> >> #14 [<c0121b70>] (disable_nonboot_cpus) from [<c016788c>]
> >> >> #15 [<c016788c>] (suspend_devices_and_enter) from [<c0167bcc>]
> >> >> #16 [<c0167bcc>] (pm_suspend) from [<c0167d94>]
> >> >> #17 [<c0167d94>] (try_to_suspend) from [<c0138460>]
> >> >> #18 [<c0138460>] (process_one_work) from [<c0138b18>]
> >> >> #19 [<c0138b18>] (worker_thread) from [<c013dc58>]
> >> >> #20 [<c013dc58>] (kthread) from [<c01061b8>]
> >> >>
> >> >> Will this patch helps here,
> >> >> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d1d74d14e98a6be740a6f12456c7d9ad47be9c9c
> >> >>
> >> >> I couldn't really understand why it got struck in synchronize_rcu().
> >> >> Please give some pointers to debug this further.
> >> >>
> >> >> Below are the configs enable related to RCU.
> >> >>
> >> >> CONFIG_TREE_PREEMPT_RCU=y
> >> >> CONFIG_PREEMPT_RCU=y
> >> >> CONFIG_RCU_STALL_COMMON=y
> >> >> CONFIG_RCU_FANOUT=32
> >> >> CONFIG_RCU_FANOUT_LEAF=16
> >> >> CONFIG_RCU_FAST_NO_HZ=y
> >> >> CONFIG_RCU_CPU_STALL_TIMEOUT=21
> >> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
> >> >>
> >> >> Kernel version is 3.10.28
> >> >> Architecture is ARM
> >> >>
> >> >> Thanks,
> >> >> Arun
> >>
> >
> crash> struct rcu_data C54286B0
> struct rcu_data {
> completed = 2833,
> gpnum = 2833,
> passed_quiesce = false,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 8,
> nxtlist = 0x0,
> nxttail = {0xc54286c4, 0xc54286c4, 0xc54286c4, 0x0},
> nxtcompleted = {0, 4294967136, 4294967137, 4294967137},
> qlen_lazy = 0,
> qlen = 0,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 609,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 13,
> n_cbs_adopted = 0,
> n_force_qs_snap = 1428,
> blimit = 10,
> dynticks = 0xc5428758,
> dynticks_snap = 13206053,
> dynticks_fqs = 16,
> offline_fqs = 0,
> n_rcu_pending = 181,
> n_rp_qs_pending = 1,
> n_rp_report_qs = 21,
> n_rp_cb_ready = 0,
> n_rp_cpu_needs_gp = 0,
> n_rp_gp_completed = 22,
> n_rp_gp_started = 8,
> n_rp_need_nothing = 130,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 3,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data C541A6B0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 4,
> nxtlist = 0x0,
> nxttail = {0xc541a6c4, 0xc541a6c4, 0xc541a6c4, 0x0},
> nxtcompleted = {0, 5877, 5878, 5878},
> qlen_lazy = 0,
> qlen = 0,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 61565,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 139,
> n_cbs_adopted = 100,
> n_force_qs_snap = 0,
> blimit = 10,
> dynticks = 0xc541a758,
> dynticks_snap = 13901017,
> dynticks_fqs = 75,
> offline_fqs = 0,
> n_rcu_pending = 16546,
> n_rp_qs_pending = 3,
> n_rp_report_qs = 4539,
> n_rp_cb_ready = 69,
> n_rp_cpu_needs_gp = 782,
> n_rp_gp_completed = 4196,
> n_rp_gp_started = 1739,
> n_rp_need_nothing = 5221,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 2,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data C540C6B0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 2,
> nxtlist = 0x0,
> nxttail = {0xc540c6c4, 0xc540c6c4, 0xc540c6c4, 0x0},
> nxtcompleted = {4294967030, 5878, 5878, 5878},
> qlen_lazy = 0,
> qlen = 0,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 74292,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 100,
> n_cbs_adopted = 65,
> n_force_qs_snap = 0,
> blimit = 10,
> dynticks = 0xc540c758,
> dynticks_snap = 10753433,
> dynticks_fqs = 69,
> offline_fqs = 0,
> n_rcu_pending = 18350,
> n_rp_qs_pending = 6,
> n_rp_report_qs = 5009,
> n_rp_cb_ready = 50,
> n_rp_cpu_needs_gp = 915,
> n_rp_gp_completed = 4423,
> n_rp_gp_started = 1826,
> n_rp_need_nothing = 6127,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 1,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
> crash> struct rcu_data C53FE6B0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 1,
> nxtlist = 0x0,
> nxttail = {0xc53fe6c4, 0xc53fe6c4, 0xc53fe6c4, 0x0},
> nxtcompleted = {4294966997, 5875, 5876, 5876},
> qlen_lazy = 0,
> qlen = 0,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 123175,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 52,
> n_cbs_adopted = 0,
> n_force_qs_snap = 0,
> blimit = 10,
> dynticks = 0xc53fe758,
> dynticks_snap = 6330446,
> dynticks_fqs = 46,
> offline_fqs = 0,
> n_rcu_pending = 22529,
> n_rp_qs_pending = 3,
> n_rp_report_qs = 5290,
> n_rp_cb_ready = 279,
> n_rp_cpu_needs_gp = 740,
> n_rp_gp_completed = 2707,
> n_rp_gp_started = 1208,
> n_rp_need_nothing = 12305,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 0,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data c54366b0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 16,
> nxtlist = 0xedaaec00,
> nxttail = {0xc54366c4, 0xe84d350c, 0xe84d350c, 0xe84d350c},
> nxtcompleted = {4294967035, 5878, 5878, 5878},
> qlen_lazy = 105,
> qlen = 415,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 86323,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 0,
> n_cbs_adopted = 139,
> n_force_qs_snap = 0,
> blimit = 10,
> dynticks = 0xc5436758,
> dynticks_snap = 7582140,
> dynticks_fqs = 41,
> offline_fqs = 0,
> n_rcu_pending = 59404,
> n_rp_qs_pending = 5,
> n_rp_report_qs = 4633,
> n_rp_cb_ready = 32,
> n_rp_cpu_needs_gp = 41088,
> n_rp_gp_completed = 2844,
> n_rp_gp_started = 1150,
> n_rp_need_nothing = 9657,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 4,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data c54446b0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 32,
> nxtlist = 0xcf9e856c,
> nxttail = {0xc54446c4, 0xcfb3050c, 0xcfb3050c, 0xcfb3050c},
> nxtcompleted = {0, 5878, 5878, 5878},
> qlen_lazy = 0,
> qlen = 117,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 36951,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 0,
> n_cbs_adopted = 0,
> n_force_qs_snap = 1428,
> blimit = 10,
> dynticks = 0xc5444758,
> dynticks_snap = 86034,
> dynticks_fqs = 46,
> offline_fqs = 0,
> n_rcu_pending = 49104,
> n_rp_qs_pending = 3,
> n_rp_report_qs = 2360,
> n_rp_cb_ready = 18,
> n_rp_cpu_needs_gp = 40106,
> n_rp_gp_completed = 1334,
> n_rp_gp_started = 791,
> n_rp_need_nothing = 4495,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 5,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data c54526b0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 64,
> nxtlist = 0xe613d200,
> nxttail = {0xc54526c4, 0xe6fc9d0c, 0xe6fc9d0c, 0xe6fc9d0c},
> nxtcompleted = {0, 5878, 5878, 5878},
> qlen_lazy = 2,
> qlen = 35,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 34459,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 0,
> n_cbs_adopted = 0,
> n_force_qs_snap = 1428,
> blimit = 10,
> dynticks = 0xc5452758,
> dynticks_snap = 116840,
> dynticks_fqs = 47,
> offline_fqs = 0,
> n_rcu_pending = 48486,
> n_rp_qs_pending = 3,
> n_rp_report_qs = 2223,
> n_rp_cb_ready = 24,
> n_rp_cpu_needs_gp = 40101,
> n_rp_gp_completed = 1226,
> n_rp_gp_started = 789,
> n_rp_need_nothing = 4123,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 6,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> crash> struct rcu_data c54606b0
> struct rcu_data {
> completed = 5877,
> gpnum = 5877,
> passed_quiesce = true,
> qs_pending = false,
> beenonline = true,
> preemptible = true,
> mynode = 0xc117f340 <rcu_preempt_state>,
> grpmask = 128,
> nxtlist = 0xdec32a6c,
> nxttail = {0xc54606c4, 0xe6fcf10c, 0xe6fcf10c, 0xe6fcf10c},
> nxtcompleted = {0, 5878, 5878, 5878},
> qlen_lazy = 1,
> qlen = 30,
> qlen_last_fqs_check = 0,
> n_cbs_invoked = 31998,
> n_nocbs_invoked = 0,
> n_cbs_orphaned = 0,
> n_cbs_adopted = 0,
> n_force_qs_snap = 1428,
> blimit = 10,
> dynticks = 0xc5460758,
> dynticks_snap = 57846,
> dynticks_fqs = 54,
> offline_fqs = 0,
> n_rcu_pending = 47502,
> n_rp_qs_pending = 2,
> n_rp_report_qs = 2142,
> n_rp_cb_ready = 37,
> n_rp_cpu_needs_gp = 40049,
> n_rp_gp_completed = 1223,
> n_rp_gp_started = 661,
> n_rp_need_nothing = 3390,
> barrier_head = {
> next = 0x0,
> func = 0x0
> },
> oom_head = {
> next = 0x0,
> func = 0x0
> },
> cpu = 7,
> rsp = 0xc117f340 <rcu_preempt_state>
> }
>
> rcu_preempt_state = $9 = {
> node = {{
> lock = {
> raw_lock = {
> {
> slock = 3129850509,
> tickets = {
> owner = 47757,
> next = 47757
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> },
> gpnum = 5877,
> completed = 5877,
> qsmask = 0,
> expmask = 0,
> qsmaskinit = 240,
> grpmask = 0,
> grplo = 0,
> grphi = 7,
> grpnum = 0 '\000',
> level = 0 '\000',
> parent = 0x0,
> blkd_tasks = {
> next = 0xc117f378 <rcu_preempt_state+56>,
> prev = 0xc117f378 <rcu_preempt_state+56>
> },
> gp_tasks = 0x0,
> exp_tasks = 0x0,
> need_future_gp = {1, 0},
> fqslock = {
> raw_lock = {
> {
> slock = 0,
> tickets = {
> owner = 0,
> next = 0
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> }
> }},
> level = {0xc117f340 <rcu_preempt_state>},
> levelcnt = {1, 0, 0, 0, 0},
> levelspread = "\b",
> rda = 0xc115e6b0 <rcu_preempt_data>,
> call = 0xc01975ac <call_rcu>,
> fqs_state = 0 '\000',
> boost = 0 '\000',
> gpnum = 5877,
> completed = 5877,
> gp_kthread = 0xf0c9e600,
> gp_wq = {
> lock = {
> {
> rlock = {
> raw_lock = {
> {
> slock = 2160230594,
> tickets = {
> owner = 32962,
> next = 32962
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> }
> }
> },
> task_list = {
> next = 0xf0cd1f20,
> prev = 0xf0cd1f20
> }
> },
> gp_flags = 1,
> orphan_lock = {
> raw_lock = {
> {
> slock = 327685,
> tickets = {
> owner = 5,
> next = 5
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> },
> orphan_nxtlist = 0x0,
> orphan_nxttail = 0xc117f490 <rcu_preempt_state+336>,
> orphan_donelist = 0x0,
> orphan_donetail = 0xc117f498 <rcu_preempt_state+344>,
> qlen_lazy = 0,
> qlen = 0,
> onoff_mutex = {
> count = {
> counter = 1
> },
> wait_lock = {
> {
> rlock = {
> raw_lock = {
> {
> slock = 811479134,
> tickets = {
> owner = 12382,
> next = 12382
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> }
> }
> },
> wait_list = {
> next = 0xc117f4bc <rcu_preempt_state+380>,
> prev = 0xc117f4bc <rcu_preempt_state+380>
> },
> owner = 0x0,
> name = 0x0,
> magic = 0xc117f4a8 <rcu_preempt_state+360>
> },
> barrier_mutex = {
> count = {
> counter = 1
> },
> wait_lock = {
> {
> rlock = {
> raw_lock = {
> {
> slock = 0,
> tickets = {
> owner = 0,
> next = 0
> }
> }
> },
> magic = 3735899821,
> owner_cpu = 4294967295,
> owner = 0xffffffff
> }
> }
> },
> wait_list = {
> next = 0xc117f4e4 <rcu_preempt_state+420>,
> prev = 0xc117f4e4 <rcu_preempt_state+420>
> },
> owner = 0x0,
> name = 0x0,
> magic = 0xc117f4d0 <rcu_preempt_state+400>
> },
> barrier_cpu_count = {
> counter = 0
> },
> barrier_completion = {
> done = 0,
> wait = {
> lock = {
> {
> rlock = {
> raw_lock = {
> {
> slock = 0,
> tickets = {
> owner = 0,
> next = 0
> }
> }
> },
> magic = 0,
> owner_cpu = 0,
> owner = 0x0
> }
> }
> },
> task_list = {
> next = 0x0,
> prev = 0x0
> }
> }
> },
> n_barrier_done = 0,
> expedited_start = {
> counter = 0
> },
> expedited_done = {
> counter = 0
> },
> expedited_wrap = {
> counter = 0
> },
> expedited_tryfail = {
> counter = 0
> },
> expedited_workdone1 = {
> counter = 0
> },
> expedited_workdone2 = {
> counter = 0
> },
> expedited_normal = {
> counter = 0
> },
> expedited_stoppedcpus = {
> counter = 0
> },
> expedited_done_tries = {
> counter = 0
> },
> expedited_done_lost = {
> counter = 0
> },
> expedited_done_exit = {
> counter = 0
> },
> jiffies_force_qs = 4294963917,
> n_force_qs = 4028,
> n_force_qs_lh = 0,
> n_force_qs_ngp = 0,
> gp_start = 4294963911,
> jiffies_stall = 4294966011,
> gp_max = 17,
> name = 0xc0d833ab "rcu_preempt",
> abbr = 112 'p',
> flavors = {
> next = 0xc117f2ec <rcu_bh_state+556>,
> prev = 0xc117f300 <rcu_struct_flavors>
> },
> wakeup_work = {
> flags = 3,
> llnode = {
> next = 0x0
> },
> func = 0xc0195aa8 <rsp_wakeup>
> }
>
>
next prev parent reply other threads:[~2014-12-18 20:05 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-15 17:04 [RCU] kernel hangs in wait_rcu_gp during suspend path Arun KS
2014-12-16 6:29 ` Arun KS
2014-12-16 17:30 ` Arun KS
2014-12-17 19:24 ` Paul E. McKenney
2014-12-18 16:22 ` Arun KS
2014-12-18 20:05 ` Paul E. McKenney [this message]
2014-12-19 18:55 ` Arun KS
2014-12-20 0:25 ` Paul E. McKenney
2014-12-17 19:27 ` Paul E. McKenney
2014-12-16 20:19 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141218200506.GL11609@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=arunks.linux@gmail.com \
--cc=getarunks@gmail.com \
--cc=josh@joshtriplett.org \
--cc=laijs@cn.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.