From: "Paul E. McKenney" <paulmck@kernel.org>
To: "Luck, Tony" <tony.luck@intel.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"x86@kernel.org" <x86@kernel.org>,
"linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
"bp@alien8.de" <bp@alien8.de>,
"tglx@linutronix.de" <tglx@linutronix.de>,
"mingo@redhat.com" <mingo@redhat.com>,
"hpa@zytor.com" <hpa@zytor.com>,
"kernel-team@fb.com" <kernel-team@fb.com>
Subject: Re: [PATCH RFC x86/mce] Make mce_timed_out() identify holdout CPUs
Date: Wed, 6 Jan 2021 11:17:08 -0800 [thread overview]
Message-ID: <20210106191708.GB2743@paulmck-ThinkPad-P72> (raw)
In-Reply-To: <3513b04e2bb543d2871ca8c152dcf5ae@intel.com>
On Wed, Jan 06, 2021 at 06:39:30PM +0000, Luck, Tony wrote:
> > The "Timeout: Not all CPUs entered broadcast exception handler" message
> > will appear from time to time given enough systems, but this message does
> > not identify which CPUs failed to enter the broadcast exception handler.
> > This information would be valuable if available, for example, in order to
> > correlated with other hardware-oriented error messages. This commit
> > therefore maintains a cpumask_t of CPUs that have entered this handler,
> > and prints out which ones failed to enter in the event of a timeout.
>
> I tried doing this a while back, but found that in my test case where I forced
> an error that would cause both threads from one core to be "missing", the
> output was highly unpredictable. Some random number of extra CPUs were
> reported as missing. After I added some extra breadcrumbs it became clear
> that pretty much all the CPUs (except the missing pair) entered do_machine_check(),
> but some got hung up at various points beyond the entry point. My only theory
> was that they were trying to snoop caches from the dead core (or access some
> other resource held by the dead core) and so they hung too.
>
> Your code is much neater than mine ... and perhaps works in other cases, but
> maybe the message needs to allow for the fact that some of the cores that
> are reported missing may just be collateral damage from the initial problem.
Understood. The system is probably not in the best shape if this code
is ever executed, after all. ;-)
So how about like this?
pr_info("%s: MCE holdout CPUs (may include false positives): %*pbl\n",
Easy enough if so!
> If I get time in the next day or two, I'll run my old test against your code to
> see what happens.
Thank you very much in advance!
For my own testing, is this still the right thing to use?
https://github.com/andikleen/mce-inject
Thanx, Paul
next prev parent reply other threads:[~2021-01-06 19:17 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-06 17:41 [PATCH RFC x86/mce] Make mce_timed_out() identify holdout CPUs Paul E. McKenney
2021-01-06 18:32 ` Borislav Petkov
2021-01-06 19:13 ` Paul E. McKenney
2021-01-07 7:07 ` Borislav Petkov
2021-01-07 17:08 ` Paul E. McKenney
2021-01-08 12:31 ` Borislav Petkov
2021-01-08 14:55 ` Paul E. McKenney
2021-01-08 16:57 ` Borislav Petkov
2021-01-06 18:39 ` Luck, Tony
2021-01-06 19:17 ` Paul E. McKenney [this message]
2021-01-06 22:49 ` Luck, Tony
2021-01-06 23:23 ` Paul E. McKenney
2021-01-07 0:26 ` Luck, Tony
2021-01-07 0:41 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210106191708.GB2743@paulmck-ThinkPad-P72 \
--to=paulmck@kernel.org \
--cc=bp@alien8.de \
--cc=hpa@zytor.com \
--cc=kernel-team@fb.com \
--cc=linux-edac@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox