[RFC/PATCH] Make printk work for really early debugging

[RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

Currently printk is no use for early debugging because it refuses to actually
print anything to the console unless cpu_online(smp_processor_id()) is true.

The stated explanation is that console drivers may require per-cpu resources,
or otherwise barf, because the system is not yet setup correctly. Fair enough.

However some console drivers might be quite happy running early during boot,
in fact we have one, and so it'd be nice if printk understood that.

So I add a flag (which I would have called CON_BOOT, but that's taken) called
CON_ANYTIME, which indicates that a console is happy to be called anytime,
even if the cpu is not yet online.

Tested on a Power 5 machine, with both a CON_ANYTIME driver and a bogus
console driver that BUG()s if called while offline. No problems AFAICT.
Built for i386 UP & SMP.

Signed-off-by: Michael Ellerman <michael@ellerman.id.au>
---

 include/linux/console.h |    1 
 kernel/printk.c         |   50 +++++++++++++++++++++++++++++++-----------------
 2 files changed, 34 insertions(+), 17 deletions(-)

Index: to-merge/include/linux/console.h
===================================================================
--- to-merge.orig/include/linux/console.h
+++ to-merge/include/linux/console.h
@@ -87,6 +87,7 @@ void give_up_console(const struct consw 
 #define CON_CONSDEV	(2) /* Last on the command line */
 #define CON_ENABLED	(4)
 #define CON_BOOT	(8)
+#define CON_ANYTIME	(16) /* Safe to call when cpu is offline */
 
 struct console
 {
Index: to-merge/kernel/printk.c
===================================================================
--- to-merge.orig/kernel/printk.c
+++ to-merge/kernel/printk.c
@@ -326,7 +326,9 @@ static void __call_console_drivers(unsig
 	struct console *con;
 
 	for (con = console_drivers; con; con = con->next) {
-		if ((con->flags & CON_ENABLED) && con->write)
+		if ((con->flags & CON_ENABLED) && con->write &&
+				(cpu_online(smp_processor_id()) ||
+				(con->flags & CON_ANYTIME)))
 			con->write(con, &LOG_BUF(start), end - start);
 	}
 }
@@ -452,6 +454,18 @@ __attribute__((weak)) unsigned long long
 	return sched_clock();
 }
 
+/* Check if we have any console registered that can be called early in boot. */
+static int have_callable_console(void)
+{
+	struct console *con;
+
+	for (con = console_drivers; con; con = con->next)
+		if (con->flags & CON_ANYTIME)
+			return 1;
+
+	return 0;
+}
+
 /**
  * printk - print a kernel message
  * @fmt: format string
@@ -565,27 +579,29 @@ asmlinkage int vprintk(const char *fmt, 
 			log_level_unknown = 1;
 	}
 
-	if (!cpu_online(smp_processor_id())) {
+	if (!down_trylock(&console_sem)) {
 		/*
-		 * Some console drivers may assume that per-cpu resources have
-		 * been allocated.  So don't allow them to be called by this
-		 * CPU until it is officially up.  We shouldn't be calling into
-		 * random console drivers on a CPU which doesn't exist yet..
+		 * We own the drivers.  We can drop the spinlock and
+		 * let release_console_sem() print the text, maybe ...
 		 */
+		console_locked = 1;
 		printk_cpu = UINT_MAX;
 		spin_unlock_irqrestore(&logbuf_lock, flags);
-		goto out;
-	}
-	if (!down_trylock(&console_sem)) {
-		console_locked = 1;
+
 		/*
-		 * We own the drivers.  We can drop the spinlock and let
-		 * release_console_sem() print the text
+		 * Console drivers may assume that per-cpu resources have
+		 * been allocated. So unless they're explicitly marked as
+		 * being able to cope (CON_ANYTIME) don't call them until
+		 * this CPU is officially up.
 		 */
-		printk_cpu = UINT_MAX;
-		spin_unlock_irqrestore(&logbuf_lock, flags);
-		console_may_schedule = 0;
-		release_console_sem();
+		if (cpu_online(smp_processor_id()) || have_callable_console()) {
+			console_may_schedule = 0;
+			release_console_sem();
+		} else {
+			/* Release by hand to avoid flushing the buffer. */
+			console_locked = 0;
+			up(&console_sem);
+		}
 	} else {
 		/*
 		 * Someone else owns the drivers.  We drop the spinlock, which
@@ -595,7 +611,7 @@ asmlinkage int vprintk(const char *fmt, 
 		printk_cpu = UINT_MAX;
 		spin_unlock_irqrestore(&logbuf_lock, flags);
 	}
-out:
+
 	preempt_enable();
 	return printed_len;
 }

Re: [RFC/PATCH] Make printk work for really early debugging

[Andrew Morton]

Michael Ellerman <michael@ellerman.id.au> wrote:
>
> Currently printk is no use for early debugging because it refuses to actually
>  print anything to the console unless cpu_online(smp_processor_id()) is true.
> 
>  The stated explanation is that console drivers may require per-cpu resources,
>  or otherwise barf, because the system is not yet setup correctly. Fair enough.
> 
>  However some console drivers might be quite happy running early during boot,
>  in fact we have one, and so it'd be nice if printk understood that.
> 
>  So I add a flag (which I would have called CON_BOOT, but that's taken) called
>  CON_ANYTIME, which indicates that a console is happy to be called anytime,
>  even if the cpu is not yet online.
> 
>  Tested on a Power 5 machine, with both a CON_ANYTIME driver and a bogus
>  console driver that BUG()s if called while offline. No problems AFAICT.
>  Built for i386 UP & SMP.

hm, OK.  But iirc is was just one silly ia64 console driver which had this
problem.  It might be better to make the new behaviour be the default and mark
the ia64 driver CON_NEEDS_CPU_ONLINE or something.

No?

Or go through and audit the drivers and sprinkle CON_ANYTIME in all the
safe ones, maybe.

Re: [RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

On Thu, 2006-05-18 at 02:34 -0700, Andrew Morton wrote:
> Michael Ellerman <michael@ellerman.id.au> wrote:
> >
> > Currently printk is no use for early debugging because it refuses to actually
> >  print anything to the console unless cpu_online(smp_processor_id()) is true.
> > 
> >  The stated explanation is that console drivers may require per-cpu resources,
> >  or otherwise barf, because the system is not yet setup correctly. Fair enough.
> > 
> >  However some console drivers might be quite happy running early during boot,
> >  in fact we have one, and so it'd be nice if printk understood that.
> > 
> >  So I add a flag (which I would have called CON_BOOT, but that's taken) called
> >  CON_ANYTIME, which indicates that a console is happy to be called anytime,
> >  even if the cpu is not yet online.
> > 
> >  Tested on a Power 5 machine, with both a CON_ANYTIME driver and a bogus
> >  console driver that BUG()s if called while offline. No problems AFAICT.
> >  Built for i386 UP & SMP.
> 
> hm, OK.  But iirc is was just one silly ia64 console driver which had this
> problem.  It might be better to make the new behaviour be the default and mark
> the ia64 driver CON_NEEDS_CPU_ONLINE or something.
> 
> No?
> 
> Or go through and audit the drivers and sprinkle CON_ANYTIME in all the
> safe ones, maybe.

Quite possibly, I started from the assumption that we liked the current
behaviour. Inverting the logic, ie. CON_NEEDS_CPU_ONLINE, would be ok
with me, but it would be a much more intrusive change. All of a sudden
we'll be calling into all sorts of drivers that we didn't previously.

I'll trawl through the console drivers tomorrow and see if I can guess
what percentage look like they will/won't work, then we can decide which
way to flip it.

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: [RFC/PATCH] Make printk work for really early debugging

[Andrew Morton]

Michael Ellerman <michael@ellerman.id.au> wrote:
>
>  I'll trawl through the console drivers tomorrow and see if I can guess
>  what percentage look like they will/won't work, then we can decide which
>  way to flip it.

Yes, that's probably safer and saner, thanks.  Don't bust a gut over
it though - it's not your problem and if someone's hurting from it
then they can write and test the patch.

I guess many people are using earlyprintk or console=uart..

Re: [RFC/PATCH] Make printk work for really early debugging

[Andi Kleen]

Michael Ellerman <michael@ellerman.id.au> writes:

> Currently printk is no use for early debugging because it refuses to actually
> print anything to the console unless cpu_online(smp_processor_id()) is true.

On x86-64 this is simply solved by setting the boot processor online very early.

-Andi

Re: [RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

On Thu, 2006-05-18 at 15:55 +0200, Andi Kleen wrote:
> Michael Ellerman <michael@ellerman.id.au> writes:
> 
> > Currently printk is no use for early debugging because it refuses to actually
> > print anything to the console unless cpu_online(smp_processor_id()) is true.
> 
> On x86-64 this is simply solved by setting the boot processor online very early.

Yeah, someone suggested that. Unfortunately it doesn't work, we actually
want to call printk before we even know which cpu we're on :D

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: [RFC/PATCH] Make printk work for really early debugging

[Andi Kleen]

On Thursday 18 May 2006 17:41, Michael Ellerman wrote:
> On Thu, 2006-05-18 at 15:55 +0200, Andi Kleen wrote:
> > Michael Ellerman <michael@ellerman.id.au> writes:
> > 
> > > Currently printk is no use for early debugging because it refuses to actually
> > > print anything to the console unless cpu_online(smp_processor_id()) is true.
> > 
> > On x86-64 this is simply solved by setting the boot processor online very early.
> 
> Yeah, someone suggested that. Unfortunately it doesn't work, we actually
> want to call printk before we even know which cpu we're on :D

You mean smp_processor_id() returns a random value?

Then your patch is broken too because iirc it tested smp_processor_id()
before that other flag

-Andi

Re: [RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On Thu, 2006-05-18 at 18:00 +0200, Andi Kleen wrote:
> On Thursday 18 May 2006 17:41, Michael Ellerman wrote:
> > On Thu, 2006-05-18 at 15:55 +0200, Andi Kleen wrote:
> > > Michael Ellerman <michael@ellerman.id.au> writes:
> > > 
> > > > Currently printk is no use for early debugging because it refuses to actually
> > > > print anything to the console unless cpu_online(smp_processor_id()) is true.
> > > 
> > > On x86-64 this is simply solved by setting the boot processor online very early.
> > 
> > Yeah, someone suggested that. Unfortunately it doesn't work, we actually
> > want to call printk before we even know which cpu we're on :D
> 
> You mean smp_processor_id() returns a random value?
> 
> Then your patch is broken too because iirc it tested smp_processor_id()
> before that other flag

No it returns 0, but that seems to be the result of good luck rather
than good management. I think I owe you a beer :)

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: [RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

On Thu, 2006-05-18 at 03:07 -0700, Andrew Morton wrote:
> Michael Ellerman <michael@ellerman.id.au> wrote:
> >
> >  I'll trawl through the console drivers tomorrow and see if I can guess
> >  what percentage look like they will/won't work, then we can decide which
> >  way to flip it.
> 
> Yes, that's probably safer and saner, thanks.  Don't bust a gut over
> it though - it's not your problem and if someone's hurting from it
> then they can write and test the patch.

I had a quick gander at some of the consoles registered under arch. A
lot of them look like they'd be ok running with CON_ANYTIME, ie. they
just write to a fixed address or uart or something simple.

Having said that I still think we should leave this patch as is, and
leave it up to arch people to decide whether they want to add
CON_ANYTIME to their particular console driver.

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: [RFC/PATCH] Make printk work for really early debugging

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

On Thu, 2006-05-18 at 15:55 +0200, Andi Kleen wrote:
> Michael Ellerman <michael@ellerman.id.au> writes:
> 
> > Currently printk is no use for early debugging because it refuses to actually
> > print anything to the console unless cpu_online(smp_processor_id()) is true.
> 
> On x86-64 this is simply solved by setting the boot processor online very early.

I had a closer look at this, and we could actually fudge it so that
cpu_online() is true in our early boot code, even though we don't know
what cpu we're on.

But that kind of begs the question, what does "cpu_online(x)" actually
mean? If it doesn't mean percpu data is allocated, what does it mean?

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]