From: Alex Elder <aelder@sgi.com>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 01/12] xfs: prevent NMI timeouts in cmn_err
Date: Tue, 11 Jan 2011 22:13:23 -0600 [thread overview]
Message-ID: <1294805603.3115.127.camel@doink> (raw)
In-Reply-To: <1294792553-8378-2-git-send-email-david@fromorbit.com>
On Wed, 2011-01-12 at 11:35 +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
>
> We currently have a global error message buffer in cmn_err that is
> protected by a spin lock that disables interrupts. Recently there
> have been reports of NMI timeouts occurring when the console is
> being flooded by SCSI error reports due to cmn_err() getting stuck
> trying to print to the console while holding this lock (i.e. with
> interrupts disabled). The NMI watchdog is seeing this CPU as
> non-responding and so is triggering a panic. While the trigger for
> the reported case is SCSI errors, pretty much anything that spams
> the kernel log could cause this to occur.
>
> Realistically the only reason that we have the intemediate message
> buffer is to prepend the correct kernel log level prefix to the log
> message. The only reason we have the lock is to protect the global
> message buffer and the only reason the message buffer is global is
> to keep it off the stack. Hence if we can avoid needing a global
> message buffer we avoid needing the lock, and we can do this with a
> small amount of cleanup and some preprocessor tricks:
>
> 1. clean up xfs_cmn_err() panic mask functionality to avoid
> needing debug code in xfs_cmn_err()
> 2. remove the couple of "!" message prefixes that still exist that
> the existing cmn_err() code steps over.
> 3. redefine CE_* levels directly to KERN_*
> 4. redefine cmn_err() and friends to use printk() directly
> via variable argument length macros.
>
> By doing this, we can completely remove the cmn_err() code and the
> lock that is causing the problems, and rely solely on printk()
> serialisation to ensure that we don't get garbled messages.
>
> A series of followup patches is really needed to clean up all the
> cmn_err() calls and related messages properly, but that results in a
> series that is not easily back portable to enterprise kernels. Hence
> this initial fix is only to address the direct problem in the lowest
> impact way possible.
I had two trivial remarks but, well, what you have is just fine...
Reviewed-by: Alex Elder <aelder@sgi.com>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
> fs/xfs/linux-2.6/xfs_sysctl.c | 23 ++++++++-
> fs/xfs/support/debug.c | 109 +++++++++++++++++++----------------------
> fs/xfs/support/debug.h | 25 ++++++---
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2011-01-12 4:13 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-12 0:35 [RFC, PATCH 0/12] xfs: rework error logging infrastructure Dave Chinner
2011-01-12 0:35 ` [PATCH 01/12] xfs: prevent NMI timeouts in cmn_err Dave Chinner
2011-01-12 4:13 ` Alex Elder [this message]
2011-01-12 12:10 ` Christoph Hellwig
2011-01-12 12:19 ` Dave Chinner
2011-01-12 15:56 ` XFS master branch update Alex Elder
2011-01-12 0:35 ` [PATCH 02/12] xfs: introduce new logging API Dave Chinner
2011-01-12 0:35 ` [PATCH 03/12] xfs: Convert linux-2.6/ files to new logging interface Dave Chinner
2011-01-12 0:35 ` [PATCH 04/12] xfs: Convert xlog_warn " Dave Chinner
2011-01-12 4:13 ` Alex Elder
2011-01-12 0:35 ` [PATCH 05/12] xfs: convert xfs_cmn_err to xfs_alert_tag Dave Chinner
2011-01-12 0:35 ` [PATCH 06/12] xfs: kill xfs_fs_repair_cmn_err() macro Dave Chinner
2011-01-12 0:35 ` [PATCH 07/12] xfs: kill xfs_fs_mount_cmn_err() macro Dave Chinner
2011-01-18 13:00 ` Christoph Hellwig
2011-01-18 21:46 ` Dave Chinner
2011-01-19 11:39 ` Christoph Hellwig
2011-01-12 0:35 ` [PATCH 08/12] xfs: convert xfs_fs_cmn_err to new error logging API Dave Chinner
2011-01-12 0:35 ` [PATCH 09/12] xfs: rename xfs_cmn_err_fsblock_zero() Dave Chinner
2011-01-12 0:35 ` [PATCH 10/12] xfs: convert the quota debug prints to new API Dave Chinner
2011-01-12 0:35 ` [PATCH 11/12] xfs: Convert remaining cmn_err() callers " Dave Chinner
2011-01-12 0:35 ` [PATCH 12/12] xfs: kill support/debug.[ch] Dave Chinner
2011-01-12 4:48 ` Alex Elder
2011-01-18 13:02 ` Christoph Hellwig
2011-01-12 4:16 ` [RFC, PATCH 0/12] xfs: rework error logging infrastructure Alex Elder
2011-01-12 7:43 ` Dave Chinner
2011-01-18 13:03 ` Christoph Hellwig
2011-01-18 21:48 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1294805603.3115.127.camel@doink \
--to=aelder@sgi.com \
--cc=david@fromorbit.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox