public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Frederic Weisbecker <frederic@kernel.org>
To: Zhen Lei <thunder.leizhen@huawei.com>
Cc: "Paul E . McKenney" <paulmck@kernel.org>,
	Neeraj Upadhyay <quic_neeraju@quicinc.com>,
	Josh Triplett <josh@joshtriplett.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Lai Jiangshan <jiangshanlai@gmail.com>,
	Joel Fernandes <joel@joelfernandes.org>,
	rcu@vger.kernel.org, linux-kernel@vger.kernel.org,
	Robert Elliott <elliott@hpe.com>
Subject: Re: [PATCH v7 5/6] doc: Document CONFIG_RCU_CPU_STALL_CPUTIME=y stall information
Date: Wed, 16 Nov 2022 23:55:07 +0100	[thread overview]
Message-ID: <20221116225507.GA839220@lothringen> (raw)
In-Reply-To: <20221111130709.247-6-thunder.leizhen@huawei.com>

On Fri, Nov 11, 2022 at 09:07:08PM +0800, Zhen Lei wrote:
> +1. A CPU looping with interrupts disabled.::
> +
> +   rcu:          hardirqs   softirqs   csw/system
> +   rcu:  number:        0          0            0
> +65;6003;1c   rcu: cputime:        0          0            0   ==> 2500(ms)
> +
> +   Because interrupts have been disabled throughout the measurement
> +   interval, there are no interrupts and no context switches.
> +   Furthermore, because CPU time consumption was measured using interrupt
> +   handlers, the system CPU consumption is misleadingly measured as zero.
> +   This scenario will normally also have "(0 ticks this GP)" printed on
> +   this CPU's summary line.
> +
> +2. A CPU looping with bottom halves disabled.
> +
> +   This is similar to the previous example, but with non-zero number of
> +   and CPU time consumed by hard interrupts, along with non-zero CPU
> +   time consumed by in-kernel execution.::
> +
> +   rcu:          hardirqs   softirqs   csw/system
> +   rcu:  number:      624          0            0
> +   rcu: cputime:       49          0         2446   ==> 2500(ms)
> +
> +   The fact that there are zero softirqs gives a hint that these were
> +   disabled, perhaps via local_bh_disable().  It is of course possible
> +   that there were no softirqs, perhaps because all events that would
> +   result in softirq execution are confined to other CPUs.  In this case,
> +   the diagnosis should continue as shown in the next example.
> +
> +3. A CPU looping with preemption disabled.
> +
> +   Here, only the number of context switches is zero.::
> +
> +   rcu:          hardirqs   softirqs   csw/system
> +   rcu:  number:      624         45            0
> +   rcu: cputime:       69          1         2425   ==> 2500(ms)
> +
> +   This situation hints that the stalled CPU was looping with preemption
> +   disabled.
> +
> +4. No looping, but massive hard and soft interrupts.::
> +
> +   rcu:          hardirqs   softirqs   csw/system
> +   rcu:  number:       xx         xx            0
> +   rcu: cputime:       xx         xx            0   ==> 2500(ms)
> +
> +   Here, the number and CPU time of hard interrupts are all non-zero,
> +   but the number of context switches and the in-kernel CPU time consumed
> +   are zero. The number and cputime of soft interrupts will usually be
> +   non-zero, but could be zero, for example, if the CPU was spinning
> +   within a single hard interrupt handler.
> +
> +   If this type of RCU CPU stall warning can be reproduced, you can
> +   narrow it down by looking at /proc/interrupts or by writing code to
> +   trace each interrupt, for example, by referring to show_interrupts().

One last question I have. Usually all these informations can be deduced by
just looking at the stacktrace that comes along an RCU stall report. So on
which kind of situation the stacktrace is not enough?

Thanks.

  reply	other threads:[~2022-11-16 22:55 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-11 13:07 [PATCH v7 0/6] rcu: Add RCU stall diagnosis information Zhen Lei
2022-11-11 13:07 ` [PATCH v7 1/6] genirq: Fix the return type of kstat_cpu_irqs_sum() Zhen Lei
2022-11-11 13:07 ` [PATCH v7 2/6] sched: Add helper kstat_cpu_softirqs_sum() Zhen Lei
2022-11-14 11:42   ` Frederic Weisbecker
2022-11-14 12:45     ` Leizhen (ThunderTown)
2022-11-14 12:50       ` Frederic Weisbecker
2022-11-14 14:26         ` Leizhen (ThunderTown)
2022-11-11 13:07 ` [PATCH v7 3/6] sched: Add helper nr_context_switches_cpu() Zhen Lei
2022-11-11 13:07 ` [PATCH v7 4/6] rcu: Add RCU stall diagnosis information Zhen Lei
2022-11-14 11:24   ` Frederic Weisbecker
2022-11-14 12:32     ` Leizhen (ThunderTown)
2022-11-14 12:46       ` Frederic Weisbecker
2022-11-16 22:39   ` Frederic Weisbecker
2022-11-17  1:57     ` Leizhen (ThunderTown)
2022-11-17 12:22       ` Frederic Weisbecker
2022-11-17 13:25         ` Leizhen (ThunderTown)
2022-11-17 14:26           ` Frederic Weisbecker
2022-11-18  2:03             ` Leizhen (ThunderTown)
2022-11-11 13:07 ` [PATCH v7 5/6] doc: Document CONFIG_RCU_CPU_STALL_CPUTIME=y stall information Zhen Lei
2022-11-16 22:55   ` Frederic Weisbecker [this message]
2022-11-17  2:03     ` Leizhen (ThunderTown)
2022-11-17 12:23       ` Frederic Weisbecker
2022-11-11 13:07 ` [PATCH v7 6/6] rcu: Align the output of RCU stall Zhen Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20221116225507.GA839220@lothringen \
    --to=frederic@kernel.org \
    --cc=elliott@hpe.com \
    --cc=jiangshanlai@gmail.com \
    --cc=joel@joelfernandes.org \
    --cc=josh@joshtriplett.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=paulmck@kernel.org \
    --cc=quic_neeraju@quicinc.com \
    --cc=rcu@vger.kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=thunder.leizhen@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox