[RFC] crash_stop: crash_stop_common

[Keith Owens <kaos@sgi.com>]

The common crash_stop code, including the design level documentation.

This patch is missing the code to handle a notify_chain (see scenario 4
in kernel/crash_stop.c).  I want to make sure that the API design is
right before adding that code.

---
 kernel/Makefile     |    1 
 kernel/crash_stop.c |  563 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 564 insertions(+)

Index: linux/kernel/Makefile
===================================================================
--- linux.orig/kernel/Makefile
+++ linux/kernel/Makefile
@@ -51,6 +51,7 @@ obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
+obj-$(CONFIG_CRASH_STOP_SUPPORTED) += crash_stop.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <alan@linuxcare.com.au>, the -fno-omit-frame-pointer is
Index: linux/kernel/crash_stop.c
===================================================================
--- /dev/null
+++ linux/kernel/crash_stop.c
@@ -0,0 +1,563 @@
+/*
+ * linux/kernel/crash_stop.c
+ *
+ * Copyright (C) 2006 Keith Owens <kaos@sgi.com>
+ *
+ * Bring the system to a crash stop for debugging by stopping all the online
+ * cpus apart from the current cpu.  To interrupt the other cpus, first send a
+ * normal IPI, if any cpus have not responded after a few seconds then send a
+ * non-maskable interrupt.  Only used if CONFIG_SMP=y.
+ *
+ * These routines can be used by any debug style code that needs to stop the
+ * other cpus in the system, including those cpus that are not responding to
+ * normal interrupts.  Debug style code includes debuggers such as kdb, kgdb,
+ * nlkd as well as dump tools such as netdump, lkcd, kdump.  All these tools
+ * have the same basic synchronization requirements, the need to stop all the
+ * cpus, save the complete state of the tasks that were running then do some
+ * work on the current cpu.
+ *
+ * For each invocation of crash_stop, one cpu is the monarch, the other cpus
+ * are slaves.  There is no external guarantee of ordering between monarch and
+ * slave events.  The most common case is when the monarch is invoked via
+ * crash_stop(), it then drives the debugger's callback on the slave cpus,
+ * followed by the callback on the monarch cpu.
+ *
+ * Some architectures (IA64 in particular) define their own global machine
+ * synchronization events where a global event can drive the slave cpus ether
+ * before or after the monarch.  See INIT in Documentation/ia64/mca.txt.
+ *
+ * To hide the external monarch/slave races from the users of crash_stop, this
+ * code enforces a standard order on the events.  The debugger's callback
+ * routine is invoked on all the slaves "at the same time", followed 10 ms
+ * later by the callback on the monarch cpu.  Typically the callback will spin
+ * on the slave cpus until the monarch callback has done its work and released
+ * the slave cpus.
+ *
+ * There is no guarantee that all online cpus will be in crash_stop state when
+ * the monarch is entered.  If a cpu or chipset is so badly hung that it will
+ * not even respond to NMI then there will be no state for that cpu in
+ * crash_stop_running_process.
+ *
+ * A live locked system can result in a slave cpu processing the crash_stop IPI
+ * _after_ the monarch cpu has done its processing and left crash_stop status.
+ * The slave will not service the normal IPI fast enough (it is live locked
+ * with interrupts disabled) so it will be interrupted by NMI.  The monarch
+ * does its work and leaves crash_stop.  Later the slave gets out of the live
+ * lock and services the crash_stop IPI, but now there is no monarch to do
+ * anything.  To catch this delayed event, a crash_stop IPI is ignored if there
+ * is no current monarch.
+ *
+ * For some events, we cannot tell straight away if we want to debug the event
+ * or not.  For example, an IA64 MCA is architecturally defined to stop all the
+ * slaves before entering the monarch.  Only when the monarch is entered do we
+ * get any data on the event, it is only on the monarch that we can tell if the
+ * MCA is recoverable or not.  In this case, the monarch must call
+ * crash_stop_recovered() instead of crash_stop().  crash_stop_recovered()
+ * releases all the slaves.  Neither the slaves nor the monarch will use the
+ * callback routine.
+ *
+ * All routines are entered with interrupts disabled.  If necessary, the caller
+ * must disable interrupts before calling crash_stop.
+ */
+
+
+/* There are several possible scenarios for using crash_stop:
+ *
+ * (1) An explicit call to crash_stop from debugging code.  For example, a
+ *     direct entry into a debugger or an explicit request to dump via sysrq.
+ *     The debugging code calls crash_stop() which stops the slaves.
+ *
+ * (2) A nested call to crash_stop on the same cpu.  For example, a user is
+ *     debugging and they decide to take a kernel dump from inside the
+ *     debugger.  The debugger has already brought the system to crash_stop
+ *     state so the dump callback will be called on the current cpu (the
+ *     monarch) but not on the slaves.  The dump code uses the data that is
+ *     already in crash_stop_running_process[].
+ *
+ * (3) Concurrent calls to crash_stop on separate cpus.  One cpu will become
+ *     the monarch for one of the events and interrupt all the others,
+ *     including any cpus that are also trying to enter crash_stop.  When the
+ *     current monarch finishes, the other cpus will race for the crash_stop
+ *     lock and one will become the new monarch (assuming the system is still
+ *     usable).
+ *
+ * (4) A system error occurs and drives the notify_die callback chain, this one
+ *     can be tricky.  It is not known which entries on the notify_die chain
+ *     will do any work, but all of them need to see the same system state.  An
+ *     arch dependent crash_stop callback is called at the start and end of the
+ *     notify_die chain.  At the start it brings the system into crash_stop
+ *     state, using its own callbacks on the slave cpus.  Then it holds the
+ *     slave cpus and releases the monarch cpu.  This allows the rest of the
+ *     entries on the notify_die chain to run, each of them can call crash_stop
+ *     and run their callback on the current cpu and the slaves.  At the end of
+ *     the notify_die chain, the main crash_stop code releases the slave cpus.
+ *     This gives a consistent view of the system to all the entries on the
+ *     notify_die chain.
+ *
+ * The various states are a little complicated, because the code has to cope
+ * with normal calls, nested calls, concurrent calls on separate cpus plus
+ * keeping a consistent view for the life of a notify chain.  A few rules :-
+ *
+ *   Variables cs_lock_cpu, cs_monarch and cs_notify_chain hold a cpu number,
+ *   -1 is 'not set'.  These variables are only updated on the monarch cpu and
+ *   are all protected by cs_lock.
+ *
+ *   Entering a nested call only affects the monarch cpu.  The slave cpus will
+ *   continue to spin in the callback for the first crash_stop() event.
+ *
+ *   Returning from a nested call does not clear cs_monarch nor release the
+ *   slaves.
+ *
+ *   If a monarch gets the lock and cs_notify_chain is not the current cpu then
+ *   another cpu is already running a notify chain.  This monarch must back off
+ *   and wait for the other cpu to finish running its notify chain.
+ *
+ *   Returning from a notify_chain call clears cs_monarch but does not release
+ *   the slaves.  Instead the slaves loop inside this code, in the expectation
+ *   that another notify chain driven routine will call crash_stop and will
+ *   need the slaves.  Unlike a nested call, the slaves will use the supplied
+ *   callback for each entry on the notify chain that calls crash_stop().
+ *
+ * Why the difference between nested calls and a notify chain?  Mainly because
+ * the entries on a notify chain are defined to be separate, also crash_stop
+ * can easily detect the start and end of running the chain.  With a nested
+ * call, there is no way to tell if the first callback will use crash_stop() a
+ * second time.  Nested calls can result from explicit calls to other debug
+ * style code or from an oops in the current callback.  On a nested call, the
+ * monarch callback owns and controls the slaves, they are out of crash_stop()
+ * control.  Only the monarch callback can release the slaves by leaving
+ * crash_stop() state, at which point the sconed call to crash_stop is not a
+ * nested call.
+ */
+
+#include <linux/cpumask.h>
+#include <linux/delay.h>
+#include <linux/crash_stop.h>
+#include <linux/kernel.h>
+#include <linux/ptrace.h>
+#include <linux/nmi.h>
+#include <linux/spinlock.h>
+#include <linux/threads.h>
+
+DEFINE_SPINLOCK(cs_lock);
+int cs_lock_cpu = -1;
+int cs_monarch = -1;
+int cs_notify_chain = -1;
+
+static int cs_recovered;
+
+static cpumask_t cs_cpu_mask, cs_sent_ipi, cs_sent_nmi;
+
+static struct crash_stop_running_process crash_stop_running_process[NR_CPUS];
+
+struct cs_global cs_global;
+
+/* Use a local version of mdelay because RedHat patch the kernel to give a
+ * warning when mdelay is used with interrupts disabled.  Why do RedHat do
+ * these silly things, have they never heard of debugging?
+ */
+static void
+cs_mdelay(int ms)
+{
+	while (ms > 0) {
+		touch_nmi_watchdog();
+		udelay(1000);
+		--ms;
+	}
+}
+
+static void
+cpu_relax_watchdog(void)
+{
+	touch_nmi_watchdog();
+	cpu_relax();
+}
+
+/* If we cannot safely use an external print routine then save any messages in
+ * a local buffer.  This code is not performance sensitive so we take the time
+ * to left justify the entire buffer instead of using ring pointers, this
+ * removes the need for users to cope with wrapped cs_msg text when analysing a
+ * crash_stopped kernel.
+ */
+
+static char cs_msg[4096];
+
+static asmlinkage int
+cs_printk(const char * fmt, ...)
+{
+	int l, ret, shift;
+	va_list ap;
+	static DEFINE_SPINLOCK(cs_msg_lock);
+	spin_lock(&cs_msg_lock);
+	l = strlen(cs_msg);
+	while (1) {
+		va_start(ap, fmt);
+		ret = vsnprintf(cs_msg+l, sizeof(cs_msg)-l, fmt, ap);
+		va_end(ap);
+		if (l == 0 || ret < sizeof(cs_msg)-l)
+			break;
+		shift = sizeof(cs_msg) / 10 + 1;
+		shift = max(shift, ret);
+		shift = min(shift, l);
+		l -= shift;
+		memcpy(cs_msg, cs_msg+shift, l);
+		memset(cs_msg+l, 0, sizeof(cs_msg)-l);
+	}
+	spin_unlock(&cs_msg_lock);
+	return 0;
+}
+
+static void
+cs_online_cpu_status(const char *text)
+{
+	int slaves = num_online_cpus() - 1, count = 0, cpu, unknown;
+	if (!slaves)
+		return;
+	for_each_online_cpu(cpu) {
+		if (cpu_isset(cpu, cs_cpu_mask) &&
+		    cpu != smp_processor_id())
+			++count;
+	}
+	unknown = slaves - count;
+	if (unknown == 0)
+		cs_global.print(
+			"All cpus are in crash_stop for %s\n",
+			text);
+	else
+		cs_global.print(
+			"%d cpu%s not in crash_stop for %s, %s state is unknown\n",
+			unknown,
+			unknown == 1 ? " is" : "s are",
+			text,
+			unknown == 1 ? "its" : "their");
+}
+
+/* Should only be called by the arch interrupt handlers, when they receive the
+ * crash_stop specific IPI.
+ */
+void
+cs_common_ipi(struct pt_regs *regs)
+{
+	if (!cs_global.callback) {
+		printk(KERN_DEBUG "Ignoring late cs_ipi on cpu %d\n",
+		       smp_processor_id());
+		return;
+	}
+	crash_stop_cpu(0, regs);
+}
+
+/* Should only be called by the arch specific NMI handlers, to see if this NMI
+ * is for crash_stop or for something else.  On most architectures, an NMI
+ * signal carries no state so we have to deduce why it was sent.
+ */
+int
+crash_stop_sent_nmi(void)
+{
+	return cpu_isset(smp_processor_id(), cs_sent_nmi);
+}
+
+/* Should only be called by the arch specific crash_stop code, after they have
+ * saved any arch specific state.  The call chain is :-
+ *
+ * crash_stop() [monarch] or cs_ipi() [slave] ->
+ *   crash_stop_cpu() [common front end code] ->
+ *     cs_arch_cpu() [arch dependent code] ->
+ *       cs_common_cpu() [common back end code] ->
+ *         callback
+ *
+ * When cs_common_cpu() is entered for a slave cpu, it must spin while
+ * cs_monarch < 0.  That enforces the order of slave callbacks first, then
+ * monarch callback.
+ *
+ * When handling a notify chain, park the slave cpus in this holding routine
+ * while the monarch cpu runs down the notify chain.  If any entry on the
+ * notify chain calls crash_stop_cpu() than release the slaves to the
+ * corresponding crash_stop callback.  On return from the callback, put them
+ * back in a holding loop.  The state of the slave cpus is not significantly
+ * changed by this process and each caller of crash_stop_cpu() gets the same
+ * data in crash_stop_running_process.  IOW, all entries on the notify chain
+ * see the state that was saved by the first crash_stop entry on the chain, not
+ * some state that changes as we run the notify chain.
+ */
+void
+cs_common_cpu(int monarch)
+{
+	do {
+		/* wait until the monarch enters */
+		while (cs_monarch < 0)
+			cpu_relax_watchdog();
+		if (!cs_recovered)
+			cs_global.callback(monarch, cs_global.data);
+		if (monarch)
+			return;
+		/* wait until the monarch leaves */
+		while (cs_monarch >= 0)
+			cpu_relax_watchdog();
+	} while (cs_notify_chain >= 0);
+}
+
+/* Wait for at least 3 seconds, but allow an extra 100 ms per online cpu to
+ * cope with live lock on systems with large cpu counts.  These are arbitrary
+ * numbers, it might be worth exposing them as /sys values so sites can tune
+ * their debugging.  Review this after we have more experience with this
+ * code - KAO.
+ */
+static void
+cs_wait_for_cpus(void)
+{
+	int count, prev_count = 0, sent_nmi = 0, t, wait_secs, slaves, cpu;
+	slaves = num_online_cpus() - 1;
+	wait_secs = min(3, (slaves * 100) / 1000);
+	cs_mdelay(100);
+	for (t = 0; t < wait_secs; ++t) {
+		count = 0;
+		slaves = num_online_cpus() - 1;
+		for_each_online_cpu(cpu) {
+			if (cpu_isset(cpu, cs_cpu_mask))
+				++count;
+		}
+		if (count == slaves)
+			break;
+		if (prev_count != count) {
+			cs_global.print(
+				"  %d out of %d cpus in crash_stop, "
+				"waiting for the rest, timeout in %d "
+				"second(s)\n",
+				count, slaves, wait_secs-t);
+			prev_count = count;
+		}
+		cs_mdelay(1000);
+		if (!sent_nmi && t == min(wait_secs / 2, 5)) {
+			for_each_online_cpu(cpu) {
+				if (cpu_isset(cpu, cs_cpu_mask) ||
+				    cpu_isset(cpu, cs_sent_nmi) ||
+				    cpu == smp_processor_id())
+					continue;
+				if (!sent_nmi) {
+					cs_global.print(" sending NMI ");
+					sent_nmi = 1;
+				}
+				cpu_set(cpu, cs_sent_nmi);
+				wmb();
+				cs_arch_send_nmi(cpu);
+			}
+		}
+		if (t % 4 == 0)
+			cs_global.print(".");
+	}
+}
+
+static void
+cs_send_ipi(void)
+{
+	int sent_ipi = 0, cpu;
+	for_each_online_cpu(cpu) {
+		if (cpu_isset(cpu, cs_cpu_mask) ||
+		    cpu_isset(cpu, cs_sent_ipi) ||
+		    cpu == smp_processor_id())
+			continue;
+		cpu_set(cpu, cs_sent_ipi);
+		cs_arch_send_ipi(cpu);
+		sent_ipi = 1;
+	}
+	if (sent_ipi)
+		cs_wait_for_cpus();
+}
+
+/**
+ * crash_stop_cpu: - Put the current cpu into crash_stop state.
+ * @monarch: 0 for a slave cpu, 1 for the monarch cpu.
+ * @regs: pt_regs for the current interrupting event.
+ *
+ * Invoked on every cpu that is being stopped, with no externally defined order
+ * between monarch and slaves.  The arch independent running state is saved
+ * here, then cs_arch_cpu() saves any arch specific state, followed by
+ * invocation of cs_common_cpu() which drives the callback routine.
+ */
+void
+crash_stop_cpu(int monarch, struct pt_regs *regs)
+{
+	struct crash_stop_running_process *r, prev;
+	int cpu = smp_processor_id();
+	cpu_set(cpu, cs_cpu_mask);
+	r = crash_stop_running_process + cpu;
+	prev = *r;
+	r->p = current;
+	r->regs = regs;
+	r->prev = &prev;
+	if (monarch && cs_monarch < 0) {
+		cs_monarch = cpu;
+		wmb();
+		cs_mdelay(10);	/* give the slaves a chance to get going */
+	}
+	cs_arch_cpu(monarch, r);
+	*r = prev;
+	if (r->p == NULL) {
+		cpu_clear(cpu, cs_sent_ipi);
+		cpu_clear(cpu, cs_sent_nmi);
+		smp_mb__before_clear_bit();
+		cpu_clear(cpu, cs_cpu_mask);
+		if (monarch)
+			cs_monarch = -1;
+	}
+}
+
+/**
+ * crash_stop: - Bring the system to a crash stop for debugging.
+ * @callback: After each cpu has been interrupted, the callback is invoked on
+ * that cpu, with the monarch flag set to 0.  After all cpus have responded or
+ * the timeout has been reached then the callback is invoked on the current cpu
+ * with the monarch flag set to 1.
+ * @data: Callback specific data, crash_stop does not use this data.
+ * @print: Optionally, the name of a debugger specific print routine.  If this
+ * is NULL then crash_stop will default to using cs_printk(), messages will be
+ * left justified in cs_msg[].
+ *
+ * Unlike stop_machine(), crash_stop() does not ask if the other cpus are
+ * ready to be stopped and will use non-maskable interrupts to stop cpus that
+ * do not respond after a few seconds.
+ *
+ * crash_stop() must be entered with interrupts disabled, it can even be
+ * entered from an NMI event.  It is the caller's responsibility to ensure that
+ * their print routine (if any) is safe in the current context.
+ *
+ * If the system has already entered a globally stopped state then sending IPI
+ * or NMI is pointless and may even be unsafe.  This particularly applies to
+ * MCA or global INIT on IA64, these events are already defined to stop the
+ * entire machine and they also prevent crash_stop() from sending any IPI or
+ * NMI events.  Only send IPI/NMI to cpus that are not yet in crash_stop state.
+ *
+ * The global structure crash_stop_running_process is updated with information
+ * about the tasks that are running on each cpu.  The debugger can use this
+ * information to start the analysis of the running tasks.
+ *
+ * Returns: 0 normal
+ *
+ *          -ENOSYS crash_stop is not supported on this architecture.
+ */
+
+int
+crash_stop(void (*callback)(int monarch, void *data), void *data,
+	   struct pt_regs *regs, printk_t print,
+	   const char *text)
+{
+	int cpu;
+	struct cs_global csg_save, csg = {
+		.callback = callback,
+		.data = data,
+		.print = print ? print : cs_printk,
+	};
+
+	WARN_ON(!irqs_disabled());
+retry:
+	if (!spin_trylock(&cs_lock)) {
+		if (cs_lock_cpu == smp_processor_id()) {
+			/* nested call on the same cpu */
+			csg_save = cs_global;
+			cs_global = csg;
+			wmb();
+			cs_online_cpu_status(text);
+			crash_stop_cpu(1, regs);
+			cs_global = csg_save;
+			wmb();
+			return 0;
+		}
+		/* concurrent call on another cpu */
+		while (cs_lock_cpu != -1)
+			cpu_relax_watchdog();
+		goto retry;
+	}
+
+	if (cs_notify_chain >= 0 &&
+	    cs_notify_chain != smp_processor_id()) {
+		/* another cpu is running a notify chain, back off */
+		spin_unlock(&cs_lock);
+		cs_mdelay(1);
+		goto retry;
+	}
+
+	cs_lock_cpu = smp_processor_id();
+	csg_save = cs_global;
+	cs_global = csg;
+	wmb();
+	cs_send_ipi();
+	cs_online_cpu_status(text);
+	crash_stop_cpu(1, regs);
+	if (cs_monarch < 0 && cs_notify_chain < 0) {
+		/* leaving a normal call, wait for the slaves to exit */
+		for_each_online_cpu(cpu) {
+			while (cpu_isset(cpu, cs_cpu_mask))
+				cpu_relax_watchdog();
+		}
+	}
+	cs_global = csg_save;
+	cs_lock_cpu = -1;
+	spin_unlock(&cs_lock);
+	return 0;
+}
+
+/**
+ * crash_stop_recovered: - Release any slaves in crash_stop state.
+ *
+ * On architectures that define their own global synchronization methods, the
+ * slave cpus may enter crash_stop state before the monarch.  If the monarch
+ * decides that the event is recoverable then the slaves need to be released
+ * from crash_stop, without invoking any callbacks.
+ *
+ * For recovered events, we do not always force the other cpus into slave
+ * state.  The assumption is that crash_stop_recovered() is only required on
+ * architectures that define their own global synchronization methods (e.g.
+ * IA64 MCA), in which case the architecture has already take care of the
+ * slaves.  If no slave cpu is in crash_stop() state then do nothing, otherwise
+ * wait until all the slaves are in crash_stop().
+ *
+ * Note: this routine does not check for a nested call to crash_stop, nor does
+ * it handle notify chains.  It makes no sense to recover an error except at
+ * the top level.
+ */
+int
+crash_stop_recovered(void)
+{
+	int cpu, any_slaves = 0;
+
+	WARN_ON(!irqs_disabled());
+retry:
+	spin_lock(&cs_lock);
+	if (cs_notify_chain >= 0 &&
+	    cs_notify_chain != smp_processor_id()) {
+		/* another cpu is running a notify chain, back off */
+		spin_unlock(&cs_lock);
+		cs_mdelay(1);
+		goto retry;
+	}
+	BUG_ON(cs_notify_chain >= 0);
+	for_each_online_cpu(cpu) {
+		if (cpu_isset(cpu, cs_cpu_mask) &&
+		    cpu != smp_processor_id()) {
+			any_slaves = 1;
+			break;
+		}
+	}
+	if (any_slaves) {
+		/* give cs_send_ipi/cs_wait_for_cpus a safe print routine */
+		struct cs_global csg_save, csg = {
+			.print = cs_printk,
+		};
+		csg_save = cs_global;
+		cs_global = csg;
+		wmb();
+		cs_send_ipi();
+		cs_global = csg_save;
+	}
+	cs_recovered = 1;
+	wmb();
+	cs_monarch = smp_processor_id();
+	for_each_online_cpu(cpu) {
+		while (cpu_isset(cpu, cs_cpu_mask))
+			cpu_relax_watchdog();
+	}
+	cs_recovered = 0;
+	cs_monarch = -1;
+	spin_unlock(&cs_lock);
+	return 0;
+}