CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Andy Isaacson]

% uname -m
x86_64
% git log -1 | head
commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
% grep XZ .config
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_XZ=y
CONFIG_RD_XZ=y
CONFIG_SQUASHFS_XZ=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_XZ_DEC_TEST=y
CONFIG_DECOMPRESS_XZ=y
% make -j4
...
  CC      arch/x86/boot/tty.o
  CC      arch/x86/oprofile/oprofile.mod.o
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
exec: 23: xz: not found
  CC      arch/x86/boot/video.o
  CC      crypto/crc32c.mod.o
  CC      crypto/pcrypt.mod.o
  CC      arch/x86/boot/video-mode.o
  CC      drivers/acpi/acpi_memhotplug.mod.o
  MKPIGGY arch/x86/boot/compressed/piggy.S
arch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
  CC      arch/x86/boot/version.o
  LD      arch/x86/boot/compressed/vmlinux
... [hundreds of lines of output snipped]
  LD [M]  net/sched/act_csum.ko
  IHEX    firmware/bnx2/bnx2-mips-09-6.2.1.fw
  IHEX    firmware/bnx2/bnx2-mips-06-6.2.1.fw
% echo $?
2
% ls -l arch/x86/boot/bzImage
-rw-r--r-- 1 adi adi 1991504 Feb  1 18:41 arch/x86/boot/bzImage

The failure is obscured by trailing make output, and since a broken
bzImage is created, "make install" succeeds, and the first visible
failure is after reboot when I get the lovely

    Decompressing Linux...

    Input is not in the XZ format (wrong magic bytes)

     -- System halted

Sometimes I run "make -j4 && make install" which would have caught this,
but the error should be more prominent and we shouldn't create bzImage
if the dependencies are not present.

-andy

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Pavel Vasilyev]

09.02.2011 22:52, Andy Isaacson пишет:
> % uname -m
> x86_64
> % git log -1 | head
> commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
> % grep XZ .config
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_KERNEL_XZ=y
> CONFIG_RD_XZ=y
> CONFIG_SQUASHFS_XZ=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_XZ_DEC_TEST=y
> CONFIG_DECOMPRESS_XZ=y
> % make -j4
> ...
>   CC      arch/x86/boot/tty.o
>   CC      arch/x86/oprofile/oprofile.mod.o
>   XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
> exec: 23: xz: not found
>   CC      arch/x86/boot/video.o
>   CC      crypto/crc32c.mod.o
>   CC      crypto/pcrypt.mod.o
>   CC      arch/x86/boot/video-mode.o
>   CC      drivers/acpi/acpi_memhotplug.mod.o
>   MKPIGGY arch/x86/boot/compressed/piggy.S
> arch/x86/boot/compressed/vmlinux.bin.xz: No such file or directory
>   CC      arch/x86/boot/version.o
>   LD      arch/x86/boot/compressed/vmlinux
> ... [hundreds of lines of output snipped]
>   LD [M]  net/sched/act_csum.ko
>   IHEX    firmware/bnx2/bnx2-mips-09-6.2.1.fw
>   IHEX    firmware/bnx2/bnx2-mips-06-6.2.1.fw
> % echo $?
> 2
> % ls -l arch/x86/boot/bzImage
> -rw-r--r-- 1 adi adi 1991504 Feb  1 18:41 arch/x86/boot/bzImage
> 
> The failure is obscured by trailing make output, and since a broken
> bzImage is created, "make install" succeeds, and the first visible
> failure is after reboot when I get the lovely
> 
>     Decompressing Linux...
> 
>     Input is not in the XZ format (wrong magic bytes)
> 
>      -- System halted
> 
> Sometimes I run "make -j4 && make install" which would have caught this,
> but the error should be more prominent and we shouldn't create bzImage
> if the dependencies are not present.

 :)

Signed-off-by: Pavel Vasilyev <pavel@pavlinux.ru>
---
diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh
index 17a5798..5ec9d36 100644
--- a/scripts/xz_wrap.sh
+++ b/scripts/xz_wrap.sh
@@ -20,4 +20,4 @@ case $ARCH in
        sparc)          BCJ=--sparc ;;
 esac

-exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
+exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1

-- 

                                                         Pavel.

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Andy Isaacson]

On Wed, Feb 09, 2011 at 11:07:32PM +0300, Pavel Vasilyev wrote:
> > The failure is obscured by trailing make output, and since a broken
> > bzImage is created, "make install" succeeds, and the first visible
[snip]
>  :)
> 
> Signed-off-by: Pavel Vasilyev <pavel@pavlinux.ru>
> ---
> diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh
> index 17a5798..5ec9d36 100644
> --- a/scripts/xz_wrap.sh
> +++ b/scripts/xz_wrap.sh
> @@ -20,4 +20,4 @@ case $ARCH in
>         sparc)          BCJ=--sparc ;;
>  esac
> 
> -exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
> +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1
> 

I get the same behavior with and without your patch, tested with both
bash and dash.  (FWIW, /bin/sh on my systems is a symlink to /bin/dash.)

% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% chmod +x foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory
./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or directory
126
% printf '#!/bin/bash\nexec /no/such/file\n' > foo.sh
% ./foo.sh; echo $?
./foo.sh: line 2: /no/such/file: No such file or directory
./foo.sh: line 2: exec: /no/such/file: cannot execute: No such file or
directory
126

% printf '#!/bin/dash\nexec /no/such/file\n' > foo.sh
% ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2
% printf '#!/bin/dash\nexec /no/such/file || exit 1\n' > foo.sh
% ./foo.sh; echo $?
exec: 2: /no/such/file: not found
2

-andy

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Pavel Vasilyev]

09.02.2011 23:38, Andy Isaacson пишет:
> On Wed, Feb 09, 2011 at 11:07:32PM +0300, Pavel Vasilyev wrote:
>>> The failure is obscured by trailing make output, and since a broken
>>> bzImage is created, "make install" succeeds, and the first visible
> [snip]
>>  :)
>>
>> Signed-off-by: Pavel Vasilyev <pavel@pavlinux.ru>
>> ---
>> diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh
>> index 17a5798..5ec9d36 100644
>> --- a/scripts/xz_wrap.sh
>> +++ b/scripts/xz_wrap.sh
>> @@ -20,4 +20,4 @@ case $ARCH in
>>         sparc)          BCJ=--sparc ;;
>>  esac
>>
>> -exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB
>> +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB || exit 1
>>
> 
> I get the same behavior with and without your patch, tested with both
> bash and dash.  (FWIW, /bin/sh on my systems is a symlink to /bin/dash.)
> 

# mv /usr/bin/xz /
# rm arch/x86/boot/compressed/vmlinux.bin*
# rm arch/x86/boot/bzImage
# make
# make
  CHK     include/linux/version.h
  CHK     include/generated/utsrelease.h
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     include/linux/version.h
make[2]: `scripts/unifdef' is up to date.
  XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
/media/kernel/linux-2.6/scripts/xz_wrap.sh: line 23: exec: xz: not found
make[2]: *** [arch/x86/boot/compressed/vmlinux.bin.xz] Error 1
make[1]: *** [arch/x86/boot/compressed/vmlinux] Error 2
make: *** [bzImage] Error 2

# make install
sh /media/kernel/linux-2.6/arch/x86/boot/install.sh 2.6.38-rc4+
arch/x86/boot/bzImage \
        System.map "/boot"

 *** Missing file: arch/x86/boot/bzImage
 *** You need to run "make" before "make install".

make[1]: *** [install] Error 1
make: *** [install] Error 2



All work, without and with patch.  :-/


# sh --version
GNU bash, version 4.1.9(1)-release (x86_64-suse-linux-gnu)


-- 

                                                         Pavel.

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[H. Peter Anvin]

On 02/09/2011 11:52 AM, Andy Isaacson wrote:
> 
> The failure is obscured by trailing make output, and since a broken
> bzImage is created, "make install" succeeds, and the first visible
> failure is after reboot when I get the lovely
> 
>     Decompressing Linux...
> 
>     Input is not in the XZ format (wrong magic bytes)
> 
>      -- System halted
> 
> Sometimes I run "make -j4 && make install" which would have caught this,
> but the error should be more prominent and we shouldn't create bzImage
> if the dependencies are not present.
> 

Agreed that the bzImage should not have been created.  That should make
the error prominent enough (your complaint that it gets obscured with
make -j is "just how things work, sorry.")

	-hpa

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Lasse Collin]

On 2011-02-09 Andy Isaacson wrote:
> % make -j4
> ...
>   CC      arch/x86/boot/tty.o
>   CC      arch/x86/oprofile/oprofile.mod.o
>   XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
> exec: 23: xz: not found

This may have something to do with the shell that "make" uses to run the 
commands. The compression command isn't seen as failed with dash for 
some reason.

It doesn't matter what shell is used to run xz_wrap.sh. The same problem 
exists with other compressors if they are missing.

> commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
[...]
> % ls -l arch/x86/boot/bzImage
> -rw-r--r-- 1 adi adi 1991504 Feb  1 18:41 arch/x86/boot/bzImage

"Feb 1" was about a week ago. The file is a few days older than the 
commit ID, assuming that your clock is set correctly.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Andy Isaacson]

On Thu, Feb 10, 2011 at 12:01:42AM +0200, Lasse Collin wrote:
> On 2011-02-09 Andy Isaacson wrote:
> > % make -j4
> > ...
> >   CC      arch/x86/boot/tty.o
> >   CC      arch/x86/oprofile/oprofile.mod.o
> >   XZKERN  arch/x86/boot/compressed/vmlinux.bin.xz
> > exec: 23: xz: not found
> 
> This may have something to do with the shell that "make" uses to run the 
> commands. The compression command isn't seen as failed with dash for 
> some reason.
> 
> It doesn't matter what shell is used to run xz_wrap.sh. The same problem 
> exists with other compressors if they are missing.
> 
> > commit 100b33c8bd8a3235fd0b7948338d6cbb3db3c63d
> [...]
> > % ls -l arch/x86/boot/bzImage
> > -rw-r--r-- 1 adi adi 1991504 Feb  1 18:41 arch/x86/boot/bzImage
> 
> "Feb 1" was about a week ago. The file is a few days older than the 
> commit ID, assuming that your clock is set correctly.

Ah, yes, this is my regular build tree; I routinely pull and build
there.

So the build failed, but the output file remains because it failed on an
intermediate step.  (This testing was from me reproducing it on a
different system, with 100b33).

Rewinding to the original system which failed to boot...

How did "make install" create a broken
/boot/vmlinuz-2.6.38-rc3-0312-gcb5520f that caused the boot-time error
message?

If I re-run "make -j4" in the tree that caused the problem, make fails
with exit code 2.

The following seems to reproduce the silent error:

rm .config
make defconfig
make -j4
edit .config and set CONFIG_KERNEL_XZ=y, CONFIG_KERNEL_GZIP=n

% make -j4
...
Kernel: arch/x86/boot/bzImage is ready  (#64)
% echo $?
0

The resulting bzImage fails to boot with the aforementioned "Input is
not in the XZ format" error.

FWIW, the machine that's showing the problem is an Ubuntu Karmic x86_64
desktop install with all updates through mid-January installed.

-andy

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Lasse Collin]

On 2011-02-10 Andy Isaacson wrote:
> On Thu, Feb 10, 2011 at 12:01:42AM +0200, Lasse Collin wrote:
> > On 2011-02-09 Andy Isaacson wrote:
> > > % ls -l arch/x86/boot/bzImage
> > > -rw-r--r-- 1 adi adi 1991504 Feb  1 18:41 arch/x86/boot/bzImage
> > 
> > "Feb 1" was about a week ago. The file is a few days older than the
> > commit ID, assuming that your clock is set correctly.
> 
> Ah, yes, this is my regular build tree; I routinely pull and build
> there.
> 
> So the build failed, but the output file remains because it failed on
> an intermediate step.  (This testing was from me reproducing it on a
> different system, with 100b33).
> 
> Rewinding to the original system which failed to boot...
> 
> How did "make install" create a broken
> /boot/vmlinuz-2.6.38-rc3-0312-gcb5520f that caused the boot-time
> error message?
> 
> If I re-run "make -j4" in the tree that caused the problem, make
> fails with exit code 2.
> 
> The following seems to reproduce the silent error:
> 
> rm .config
> make defconfig
> make -j4
> edit .config and set CONFIG_KERNEL_XZ=y, CONFIG_KERNEL_GZIP=n
> 
> % make -j4
> ...
> Kernel: arch/x86/boot/bzImage is ready  (#64)
> % echo $?
> 0

It looks a like a bug in dash. The following with dash prints foo bar 
and returns with exit status 0, but other shells print only foo and 
return with non-zero exit status:

    sh -c 'set -e ; echo foo ; ( false ) ; echo bar'

That is, "set -e" has no effect on a subshell returning a non-zero exit 
status. Without a subshell it works correctly:

    sh -c 'set -e ; echo foo ; { false ; } ; echo bar'

Kernel compression and mkpiggy use command lines that hit this bug. Here 
is a quick hack to avoid the unneeded subshell for these command lines, 
which also works around the dash bug:

    sed -i 's|( *\( *rm -f $@ ; false\) *)|{ \1 ; }|' \
        scripts/Makefile.lib arch/x86/boot/compressed/Makefile

There are similar "set -e" + subshell constructs in other makefiles and 
scripts. I don't know if it makes sense to accommodate a buggy but 
common /bin/sh by modifying makefiles and scripts to avoid the 
problematic construct. Maybe it would be better to have a test to catch 
the buggy shell and tell the user to use e.g. "make SHELL=/bin/bash".

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Re: CONFIG_KERNEL_XZ obscure build failure if xz(1) not installed

[Lasse Collin]

On 2011-02-10 Lasse Collin wrote:
> It looks a like a bug in dash. The following with dash prints foo bar
> and returns with exit status 0, but other shells print only foo and
> return with non-zero exit status:
> 
>     sh -c 'set -e ; echo foo ; ( false ) ; echo bar'

This dash bug has been fixed on 2010-07-06.

http://git.kernel.org/?p=utils/dash/dash.git;a=commitdiff;h=a92255d6fa8b8efb0f9fb093b77ac601fe3e9ede

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode