[PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Andreas Mohr]

Hi guys,

boy am I angry about this...

When trying to get my ASUS WL-500gP v2 to boot (via the very nice
OpenWrt infrastructure) from external media for a Debian-MIPSEL
installation on it
(see e.g. http://wpkg.org/Running_Debian_on_ASUS_WL-500G_deluxe ),
I was hitting the dreaded "No init found." boot message.

5 years (FIVE G*DDARN YEARS!! Leaving me wondering what Linux development
has been doing all the time...) earlier I was in a diploma thesis project
where I had the exact same issue, with the _exact same_ error message,
which managed to waste the _exact same_ 2 or 3 hours that I managed to waste
this time again (obviously my memory is not stellar enough to be able to
recall that it was a console issue).
I needed to search through a couple dozen forum threads (with many
thread participants being helpless) to finally nail the issue I was having
this time.

Needless to say this error message is totally and deadly precisely
UNHELPFUL, proof:
Google "No init found." linux: 160000 results.
Very conservative estimate: 5000 developers times 2 hours = 10000 hours
of wasted precious developer time. An estimated lowly $50 per developer hour,
thus a grand total of $500000 (FIVE HUNDRED THOUSAND AMERICAN DOLLARS, however
broken the dollar might be at any moment in time ;) wasted
on this unhelpfully implemented yet very critical (failure will leave you
hanging dry in a hard place, with no immediate tools for remedy, with
further unsuccessful image rebuilds taking > 30 minutes each)
transition step.


IMHO as catastrophic a failure in usability as it can ever get.


Note that this patch hasn't even been compiled yet (the
"See Linux " __FILE__ " for guidance." part might be the only thing to croak).
I just wanted to get this out the door ASAP despite a tight time frame.
You'll figure it out anyway ;)

Useful ChangeLog phrase: provide crucial explanations for the dreaded
"No init found." boot failure.

Thanks,

Signed-off-by: Andreas Mohr <andi@lisas.de>


--- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
+++ linux-2.6.32-rc6/init/main.c	2009-11-08 12:40:11.000000000 +0100
@@ -846,7 +846,47 @@ static noinline int init_post(void)
 	run_init_process("/bin/init");
 	run_init_process("/bin/sh");
 
-	panic("No init found.  Try passing init= option to kernel.");
+	panic("No init found.  Try passing init= option to kernel. "
+              "See Linux " __FILE__ " for guidance.");
+/* ok, so you've got this pretty unintuitive message and are wondering
+ * what the H*** went wrong.
+ * Some high-level reasons for failure (listed roughly in order of execution)
+ * to load the init binary are:
+ * A) Unable to mount root FS
+ * B) init binary doesn't exist on rootfs
+ * C) other requirements not met
+ * D) binary exists but dependencies not available
+ * E) binary cannot be loaded
+ *
+ * Detailed explanations:
+ * A) Please make sure you have the correct root FS type
+ * (and root= in bootloader or CONFIG_CMDLINE points to the correct partition),
+ * required drivers such as storage hardware (such as SCSI or USB!)
+ * and filesystem (ext3, jffs2 etc.) are builtin (alternatively as modules by
+ * using initrd)
+ * C) Possibly a conflict in console= setup --> initial console unavailable.
+ * E.g. some serial consoles are unreliable due to serial IRQ issues.
+ * Try using a different console= device or e.g. netconsole=.
+ * D) e.g. crucial library dependencies of the init binary such as
+ * /lib/ld-linux.so.2 missing or broken. Use readelf -d <INIT>|grep NEEDED
+ * to find out which libraries are required.
+ * E) make sure the binary's architecture matches your hardware.
+ * E.g. i386 vs. x86_64 mismatch, or trying to load x86 on ARM hardware.
+ * Or did you try loading a non-binary file here!?! (shell script?)
+ * To find out more, add patch here to display kernel_execve()s return values.
+ *
+ * Please extend this explanation whenever you find new failure causes
+ * (after all loading the init binary is a CRITICAL and hard transition step
+ * which needs to be made as painless as possible), then submit patch to LKML.
+ * Further TODOs:
+ * - Implement the various run_init_process() invocations via a struct array
+ *   which can then store the kernel_execve() result value and on failure
+ *   log it all by iterating over _all_ results (very important usability fix).
+ * - try to make the implementation itself more helpful in general,
+ *   e.g. by providing additional error messages at affected places.
+ *
+ * Andreas Mohr <andi at lisas period de>
+ */
 }
 
 static int __init kernel_init(void * unused)

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Américo Wang]

On Sun, Nov 08, 2009 at 01:00:08PM +0100, Andreas Mohr wrote:
>Hi guys,
>
>boy am I angry about this...
>
>When trying to get my ASUS WL-500gP v2 to boot (via the very nice
>OpenWrt infrastructure) from external media for a Debian-MIPSEL
>installation on it
>(see e.g. http://wpkg.org/Running_Debian_on_ASUS_WL-500G_deluxe ),
>I was hitting the dreaded "No init found." boot message.
>
>5 years (FIVE G*DDARN YEARS!! Leaving me wondering what Linux development
>has been doing all the time...) earlier I was in a diploma thesis project
>where I had the exact same issue, with the _exact same_ error message,
>which managed to waste the _exact same_ 2 or 3 hours that I managed to waste
>this time again (obviously my memory is not stellar enough to be able to
>recall that it was a console issue).
>I needed to search through a couple dozen forum threads (with many
>thread participants being helpless) to finally nail the issue I was having
>this time.
>
>Needless to say this error message is totally and deadly precisely
>UNHELPFUL, proof:
>Google "No init found." linux: 160000 results.
>Very conservative estimate: 5000 developers times 2 hours = 10000 hours
>of wasted precious developer time. An estimated lowly $50 per developer hour,
>thus a grand total of $500000 (FIVE HUNDRED THOUSAND AMERICAN DOLLARS, however
>broken the dollar might be at any moment in time ;) wasted
>on this unhelpfully implemented yet very critical (failure will leave you
>hanging dry in a hard place, with no immediate tools for remedy, with
>further unsuccessful image rebuilds taking > 30 minutes each)
>transition step.
>
>
>IMHO as catastrophic a failure in usability as it can ever get.
>
>
>Note that this patch hasn't even been compiled yet (the
>"See Linux " __FILE__ " for guidance." part might be the only thing to croak).
>I just wanted to get this out the door ASAP despite a tight time frame.
>You'll figure it out anyway ;)
>
>Useful ChangeLog phrase: provide crucial explanations for the dreaded
>"No init found." boot failure.
>
>Thanks,

Hi,

>
>Signed-off-by: Andreas Mohr <andi@lisas.de>
>
>
>--- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
>+++ linux-2.6.32-rc6/init/main.c	2009-11-08 12:40:11.000000000 +0100
>@@ -846,7 +846,47 @@ static noinline int init_post(void)
> 	run_init_process("/bin/init");
> 	run_init_process("/bin/sh");
> 
>-	panic("No init found.  Try passing init= option to kernel.");
>+	panic("No init found.  Try passing init= option to kernel. "
>+    y          "See Linux " __FILE__ " for guidance.");


I would like to put those guidelines in a doc file, instead
of being a FAT comment in the source code.

And, you're explaining the _user space_ reasons that causes
this problem, not related with kernel at all.



>+/* ok, so you've got this pretty unintuitive message and are wondering
>+ * what the H*** went wrong.
>+ * Some high-level reasons for failure (listed roughly in order of execution)
>+ * to load the init binary are:
>+ * A) Unable to mount root FS
>+ * B) init binary doesn't exist on rootfs
>+ * C) other requirements not met
>+ * D) binary exists but dependencies not available
>+ * E) binary cannot be loaded
>+ *
>+ * Detailed explanations:
>+ * A) Please make sure you have the correct root FS type
>+ * (and root= in bootloader or CONFIG_CMDLINE points to the correct partition),
>+ * required drivers such as storage hardware (such as SCSI or USB!)
>+ * and filesystem (ext3, jffs2 etc.) are builtin (alternatively as modules by
>+ * using initrd)
>+ * C) Possibly a conflict in console= setup --> initial console unavailable.
>+ * E.g. some serial consoles are unreliable due to serial IRQ issues.
>+ * Try using a different console= device or e.g. netconsole=.
>+ * D) e.g. crucial library dependencies of the init binary such as
>+ * /lib/ld-linux.so.2 missing or broken. Use readelf -d <INIT>|grep NEEDED
>+ * to find out which libraries are required.
>+ * E) make sure the binary's architecture matches your hardware.
>+ * E.g. i386 vs. x86_64 mismatch, or trying to load x86 on ARM hardware.
>+ * Or did you try loading a non-binary file here!?! (shell script?)
>+ * To find out more, add patch here to display kernel_execve()s return values.
>+ *
>+ * Please extend this explanation whenever you find new failure causes
>+ * (after all loading the init binary is a CRITICAL and hard transition step
>+ * which needs to be made as painless as possible), then submit patch to LKML.
>+ * Further TODOs:
>+ * - Implement the various run_init_process() invocations via a struct array
>+ *   which can then store the kernel_execve() result value and on failure
>+ *   log it all by iterating over _all_ results (very important usability fix).
>+ * - try to make the implementation itself more helpful in general,
>+ *   e.g. by providing additional error messages at affected places.
>+ *
>+ * Andreas Mohr <andi at lisas period de>
>+ */
> }
> 
> static int __init kernel_init(void * unused)
>--
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Live like a child, think like the god.
 

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Andreas Mohr]

Hi,

On Sun, Nov 08, 2009 at 10:25:06PM +0800, Américo Wang wrote:
> On Sun, Nov 08, 2009 at 01:00:08PM +0100, Andreas Mohr wrote:
> >--- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
> >+++ linux-2.6.32-rc6/init/main.c	2009-11-08 12:40:11.000000000 +0100
> >@@ -846,7 +846,47 @@ static noinline int init_post(void)
> > 	run_init_process("/bin/init");
> > 	run_init_process("/bin/sh");
> > 
> >-	panic("No init found.  Try passing init= option to kernel.");
> >+	panic("No init found.  Try passing init= option to kernel. "
> >+    y          "See Linux " __FILE__ " for guidance.");
> 
> 
> I would like to put those guidelines in a doc file, instead
> of being a FAT comment in the source code.

Heh, somehow I fully expected this objection to come up ;)
(and you're probably right after all)

So, where to put it?
Documentation/initrd.txt is related, however I'd create something
like Documentation/initialization/...
Or perhaps Documentation/bootup/init.txt ?

> And, you're explaining the _user space_ reasons that causes
> this problem, not related with kernel at all.

That's usually within the responsibility of the one passing control,
not the one gaining control.
Just like with children, where it's firmly the parents' job to take care
of getting their children safely to kindergarden, not the kindergarden's ;)

Plus in my case it _was_ a kernel issue (problematic/non-working serial
console device). And except for the linker issues of missing libraries,
many issues (USB functionality etc.) were kernel-related, too.

Thanks,

Andreas Mohr

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Américo Wang]

On Sun, Nov 08, 2009 at 03:45:50PM +0100, Andreas Mohr wrote:
>Hi,
>
>On Sun, Nov 08, 2009 at 10:25:06PM +0800, Américo Wang wrote:
>> On Sun, Nov 08, 2009 at 01:00:08PM +0100, Andreas Mohr wrote:
>> >--- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
>> >+++ linux-2.6.32-rc6/init/main.c	2009-11-08 12:40:11.000000000 +0100
>> >@@ -846,7 +846,47 @@ static noinline int init_post(void)
>> > 	run_init_process("/bin/init");
>> > 	run_init_process("/bin/sh");
>> > 
>> >-	panic("No init found.  Try passing init= option to kernel.");
>> >+	panic("No init found.  Try passing init= option to kernel. "
>> >+    y          "See Linux " __FILE__ " for guidance.");
>> 
>> 
>> I would like to put those guidelines in a doc file, instead
>> of being a FAT comment in the source code.
>
>Heh, somehow I fully expected this objection to come up ;)
>(and you're probably right after all)
>
>So, where to put it?
>Documentation/initrd.txt is related, however I'd create something
>like Documentation/initialization/...
>Or perhaps Documentation/bootup/init.txt ?

Personally, I prefer Documentation/init.txt, but adding it to
initrd.txt also makes sense for me. :)

>
>> And, you're explaining the _user space_ reasons that causes
>> this problem, not related with kernel at all.
>
>That's usually within the responsibility of the one passing control,
>not the one gaining control.
>Just like with children, where it's firmly the parents' job to take care
>of getting their children safely to kindergarden, not the kindergarden's ;)
>
>Plus in my case it _was_ a kernel issue (problematic/non-working serial
>console device). And except for the linker issues of missing libraries,
>many issues (USB functionality etc.) were kernel-related, too.

Yeah, I mentioned it to mean it is not suitable to explain
user-space problems in a kernel source file. Putting it into a kernel
doc should be fine.

Thanks.


-- 
Live like a child, think like the god.
 

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Arnd Bergmann]

On Sunday 08 November 2009, Américo Wang wrote:
> >Plus in my case it was a kernel issue (problematic/non-working serial
> >console device). And except for the linker issues of missing libraries,
> >many issues (USB functionality etc.) were kernel-related, too.
> 
> Yeah, I mentioned it to mean it is not suitable to explain
> user-space problems in a kernel source file. Putting it into a kernel
> doc should be fine.

I think it would be best to put it on a wiki page, e.g. on kernelnewbies.org
and point to that. That means you will be able to add information to it
independent of the kernel version.

	Arnd <><

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Andreas Mohr]

Hi,

On Sun, Nov 08, 2009 at 04:30:32PM +0100, Arnd Bergmann wrote:
> On Sunday 08 November 2009, Américo Wang wrote:
> > Yeah, I mentioned it to mean it is not suitable to explain
> > user-space problems in a kernel source file. Putting it into a kernel
> > doc should be fine.
> 
> I think it would be best to put it on a wiki page, e.g. on kernelnewbies.org
> and point to that. That means you will be able to add information to it
> independent of the kernel version.

I don't know, personally I think that's way too disconnected.
A Wiki entry might be down, might get renamed, might get converted to an
entirely different Wiki etc. Or might sit on a certain (admittedly large ;)
network that I don't have access to under certain circumstances.
In general these issues should be treated right where they're being created,
and that's the kernel.
And I don't see how "independent of the kernel version" is relevant
here. If I have such an issue, then I look into that file on a tree I
have here (which usually is a current version), and if that doesn't
help, then one could look into the exact version that (tries to) boots.


And IMHO what we should perhaps think of is inventing a reference number
scheme for people to easily look up kernel issues.
This could then be something like "LKE#1234: No init found." (L_inux
K_ernel E_rror).
OTOH this would bloat kernel messages, thus maybe there's a way to
handle it more intelligently.

Thanks,

Andreas Mohr

[PATCH] provide crucial explanations for the dreaded "No init found." boot failure.

[Andreas Mohr]

On Sun, Nov 08, 2009 at 11:11:53PM +0800, Américo Wang wrote:
> On Sun, Nov 08, 2009 at 03:45:50PM +0100, Andreas Mohr wrote:
> >Hi,
> >
> >On Sun, Nov 08, 2009 at 10:25:06PM +0800, Américo Wang wrote:
> >> On Sun, Nov 08, 2009 at 01:00:08PM +0100, Andreas Mohr wrote:
> >So, where to put it?
> >Documentation/initrd.txt is related, however I'd create something
> >like Documentation/initialization/...
> >Or perhaps Documentation/bootup/init.txt ?
> 
> Personally, I prefer Documentation/init.txt, but adding it to
> initrd.txt also makes sense for me. :)

Yup, chose Documentation/init.txt.

Patch against 2.6.32-rc6 attached, UNTESTED (checkpatch.pl is fine).

Please get something like this committed, half a million $$ in prior damage
ought to be enough for everybody.

Thanks,

Signed-off-by: Andreas Mohr <andi@lisas.de>


--- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
+++ linux-2.6.32-rc6/init/main.c	2009-11-09 21:34:24.000000000 +0100
@@ -846,7 +846,8 @@
 	run_init_process("/bin/init");
 	run_init_process("/bin/sh");
 
-	panic("No init found.  Try passing init= option to kernel.");
+	panic("No init found.  Try passing init= option to kernel. "
+	      "See Linux Documentation/init.txt for guidance.");
 }
 
 static int __init kernel_init(void * unused)
--- /dev/null	2009-10-09 14:49:35.924999847 +0200
+++ linux-2.6.32-rc6/Documentation/init.txt	2009-11-09 21:27:39.000000000 +0100
@@ -0,0 +1,42 @@
+Explaining the dreaded "No init found." boot hang message
+=========================================================
+
+OK, so you've got this pretty unintuitive message (currently located
+in init/main.c) and are wondering what the H*** went wrong.
+Some high-level reasons for failure (listed roughly in order of execution)
+to load the init binary are:
+A) Unable to mount root FS
+B) init binary doesn't exist on rootfs
+C) other requirements not met
+D) binary exists but dependencies not available
+E) binary cannot be loaded
+
+Detailed explanations:
+A) Please make sure you have the correct root FS type
+(and root= in bootloader or CONFIG_CMDLINE points to the correct partition),
+required drivers such as storage hardware (such as SCSI or USB!)
+and filesystem (ext3, jffs2 etc.) are builtin (alternatively as modules by
+using initrd)
+C) Possibly a conflict in console= setup --> initial console unavailable.
+E.g. some serial consoles are unreliable due to serial IRQ issues (e.g. missing
+interrupt-based configuration).
+Try using a different console= device or e.g. netconsole=.
+D) e.g. crucial library dependencies of the init binary such as
+/lib/ld-linux.so.2 missing or broken. Use readelf -d <INIT>|grep NEEDED
+to find out which libraries are required.
+E) make sure the binary's architecture matches your hardware.
+E.g. i386 vs. x86_64 mismatch, or trying to load x86 on ARM hardware.
+Or did you try loading a non-binary file here!?! (shell script?)
+To find out more, add code patch to display kernel_execve()s return values.
+
+Please extend this explanation whenever you find new failure causes
+(after all loading the init binary is a CRITICAL and hard transition step
+which needs to be made as painless as possible), then submit patch to LKML.
+Further TODOs:
+- Implement the various run_init_process() invocations via a struct array
+  which can then store the kernel_execve() result value and on failure
+  log it all by iterating over _all_ results (very important usability fix).
+- try to make the implementation itself more helpful in general,
+  e.g. by providing additional error messages at affected places.
+
+Andreas Mohr <andi at lisas period de>

Re: [PATCH] Please get this committed, _NOW_ (preferrably 5 years ago), thanks.

[Andi Kleen]

Andreas Mohr <andi@lisas.de> writes:

> boy am I angry about this...

Angry about yourself?

> --- linux-2.6.32-rc6/init/main.c.orig	2009-11-08 11:09:51.000000000 +0100
> +++ linux-2.6.32-rc6/init/main.c	2009-11-08 12:40:11.000000000 +0100
> @@ -846,7 +846,47 @@ static noinline int init_post(void)
>  	run_init_process("/bin/init");
>  	run_init_process("/bin/sh");
>  
> -	panic("No init found.  Try passing init= option to kernel.");
> +	panic("No init found.  Try passing init= option to kernel. "
> +              "See Linux " __FILE__ " for guidance.");

That seems like the wrong approach. The kernel can actually distingush
many of these cases, so it could just tell the user about them directly.

> +/* ok, so you've got this pretty unintuitive message and are wondering
> + * what the H*** went wrong.
> + * Some high-level reasons for failure (listed roughly in order of execution)
> + * to load the init binary are:
> + * A) Unable to mount root FS

This the kernel knows so it could just say. In fact it's already
printed usually, but it could be printed again.

> + * B) init binary doesn't exist on rootfs

This is knows also.

> + * C) other requirements not met

No idea what that should be.

> + * D) binary exists but dependencies not available

I think you're refering to shared libraries here. This is actually not correct,
because if dynamic linking for init fails this will not result in this error message.

> + * E) binary cannot be loaded

This it also knows.

So please just fix the output.

-Andi