[bisected] broken make-kpkg kernel build in 3.15-rc1

[bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

The following commit breaks building the kernel using the debian
make-kpkg command

commit f4d4ffc03efc864645b990e1d579bbe1b8e358a4
Author: Jason Cooper <jason@lakedaemon.net>
Date:   Sun Dec 1 23:56:28 2013 +0000

    kbuild: dtbs_install: new make target


If I run this:
	make-kpkg --initrd --rootcmd fakeroot kernel_image

After this commit it fails with:

make[2]: Entering directory `/home/vince/research/linux-kernel/linux-git'
sh /home/vince/research/linux-kernel/linux-git/arch/x86/boot/install.sh 
3.14.0-rc6+ arch/x86/boot/bzImage \
		System.map 
"/home/vince/research/linux-kernel/linux-git/debian/linux-image-3.14.0-rc6+//boot"
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.14.0-rc6+ 
/home/vince/research/linux-kernel/linux-git/debian/linux-image-3.14.0-rc6+//boot/vmlinuz-3.14.0-rc6+
/etc/kernel/postinst.d/apt-auto-removal: 84: 
/etc/kernel/postinst.d/apt-auto-removal: cannot create 
/etc/apt/apt.conf.d//01autoremove-kernels.dpkg-new: Permission denied
run-parts: /etc/kernel/postinst.d/apt-auto-removal exited with return code 
2
make[3]: *** [install] Error 1
make[2]: *** [install] Error 2
make[2]: Leaving directory `/home/vince/research/linux-kernel/linux-git'
make[1]: *** [debian/stamp/install/linux-image-3.14.0-rc6+] Error 2
make[1]: Leaving directory `/home/vince/research/linux-kernel/linux-git'
make: *** [kernel_image] Error 2

Wasted a whole day tracking this down, thought it was a problem with a new 
version of apt.

Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Michal Marek]

On 2014-04-15 15:43, Vince Weaver wrote:
> 
> The following commit breaks building the kernel using the debian
> make-kpkg command
> 
> commit f4d4ffc03efc864645b990e1d579bbe1b8e358a4
> Author: Jason Cooper <jason@lakedaemon.net>
> Date:   Sun Dec 1 23:56:28 2013 +0000
> 
>     kbuild: dtbs_install: new make target
> 
> 
> If I run this:
> 	make-kpkg --initrd --rootcmd fakeroot kernel_image
> 
> After this commit it fails with:
> 
> make[2]: Entering directory `/home/vince/research/linux-kernel/linux-git'
> sh /home/vince/research/linux-kernel/linux-git/arch/x86/boot/install.sh 
> 3.14.0-rc6+ arch/x86/boot/bzImage \
> 		System.map 
> "/home/vince/research/linux-kernel/linux-git/debian/linux-image-3.14.0-rc6+//boot"
> run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.14.0-rc6+ 
> /home/vince/research/linux-kernel/linux-git/debian/linux-image-3.14.0-rc6+//boot/vmlinuz-3.14.0-rc6+
> /etc/kernel/postinst.d/apt-auto-removal: 84: 
> /etc/kernel/postinst.d/apt-auto-removal: cannot create 
> /etc/apt/apt.conf.d//01autoremove-kernels.dpkg-new: Permission denied
> run-parts: /etc/kernel/postinst.d/apt-auto-removal exited with return code 
> 2

I fail to see how the above commit is responsible for this. make install
starts /sbin/installkernel and the rest is the job of this script. To
me, it looks more like a fakeroot failure than a bug in make install.

Michal

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Christoph Lameter]

On Tue, 15 Apr 2014, Michal Marek wrote:

> I fail to see how the above commit is responsible for this. make install
> starts /sbin/installkernel and the rest is the job of this script. To
> me, it looks more like a fakeroot failure than a bug in make install.

The problem is that the kernel build process should not write to anything
in the /etc/apt directory. fakeroot only fakes root. Writing files may
occur to the users homedirectory (where the kernel build tree is) but not
to directories where the user running fakeroot has no write access.

I have the same issue here. Lets see if reverting that commit fixes it.

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Michal Marek]

On 2014-04-15 18:49, Christoph Lameter wrote:
> On Tue, 15 Apr 2014, Michal Marek wrote:
> 
>> I fail to see how the above commit is responsible for this. make install
>> starts /sbin/installkernel and the rest is the job of this script. To
>> me, it looks more like a fakeroot failure than a bug in make install.
> 
> The problem is that the kernel build process should not write to anything
> in the /etc/apt directory.

Again: make install starts /sbin/installkernel and the rest is the job
of this script. Which means you need to debug Debian's
/sbin/installkernel why it suddenly wants to modify files in /etc/apt.

Michal

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Christoph Lameter]

On Tue, 15 Apr 2014, Michal Marek wrote:

> > The problem is that the kernel build process should not write to anything
> > in the /etc/apt directory.
>
> Again: make install starts /sbin/installkernel and the rest is the job
> of this script. Which means you need to debug Debian's
> /sbin/installkernel why it suddenly wants to modify files in /etc/apt.

Why then does reverting the patch fix the issue?

Something in the patch causes install process to not divert correctly to
the install directory. It tries to go to the / directory instead.

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Russell King - ARM Linux]

On Tue, Apr 15, 2014 at 12:22:51PM -0500, Christoph Lameter wrote:
> On Tue, 15 Apr 2014, Michal Marek wrote:
> 
> > > The problem is that the kernel build process should not write to anything
> > > in the /etc/apt directory.
> >
> > Again: make install starts /sbin/installkernel and the rest is the job
> > of this script. Which means you need to debug Debian's
> > /sbin/installkernel why it suddenly wants to modify files in /etc/apt.
> 
> Why then does reverting the patch fix the issue?
> 
> Something in the patch causes install process to not divert correctly to
> the install directory. It tries to go to the / directory instead.

Right, so the thing to do is to run a build with the commit applied,
without any -j arguments and V=1, logging the output (so that predictable
output can be generated).

Then run the same thing with the commit reverted, again logging it.  Then
diff the two and see what's different.

I, too, can't see anything in that commit which would cause this regression.

Looking at arch/x86/boot/install.sh, it contains these lines:

if [ -x ~/bin/${INSTALLKERNEL} ]; then exec ~/bin/${INSTALLKERNEL} "$@"; fi
if [ -x /sbin/${INSTALLKERNEL} ]; then exec /sbin/${INSTALLKERNEL} "$@"; fi

which will have the effect of running /sbin/installkernel.

The Ubuntu /sbin/installkernel (which I assume is closely related to
Debian's version) installs the kernel in the specified directory
(indicated by $dir in the script), and then does this:

run-parts --verbose --exit-on-error --arg="$ver" --arg="$dir/$img_dest-$ver" \
  /etc/kernel/postinst.d

which has the effect of running all hooks in /etc/kernel/postinst.d.
One such hook is /etc/kernel/postinst.d/apt-auto-removal, which does
this:

eval $(apt-config shell APT_CONF_D Dir::Etc::parts/d)
test -n "${APT_CONF_D}" || APT_CONF_D="/etc/apt/apt.conf.d"
config_file=${APT_CONF_D}/01autoremove-kernels
...
cat > "$config_file".dpkg-new <<EOF
// File autogenerated by $0, do not edit
APT
{
...
};
EOF
mv "$config_file".dpkg-new "$config_file"

Now, the only additional thing that could affect any of debian's scripts
is the addition of INSTALL_DTBS_PATH in the enfironment.  So, I've tested
running "apt-config shell APT_CONF_D Dir::Etc::parts/d" with that set, and
it has no effect here.

So... like I say above, I think we need to see a diff between two build
attempts, one with the commit in place, and one without.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:

> Right, so the thing to do is to run a build with the commit applied,
> without any -j arguments and V=1, logging the output (so that predictable
> output can be generated).
> 
> Then run the same thing with the commit reverted, again logging it.  Then
> diff the two and see what's different.

OK, I captured the output.  No diffs, as of course reverting things
means one kernel was called 3.15-rc1 and one 3.15-rc1+ which meant more or 
less every line changed.

It seems that on stock 3.15-rc1 (which fails to build) after running 
depmod it kicks into

	restore_upstream_debianization

which goes down a path which causes the file permission failures

Wheras 3.15-rc1 with the problem commit reverted this doesn't happen.

This is a really weird bug, I've tried reverting individual hunks of the 
problem commit manually and it is not clear at all to me what is 
triggering this.

Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

On Tue, 15 Apr 2014, Vince Weaver wrote:

> On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> 
> > Right, so the thing to do is to run a build with the commit applied,
> > without any -j arguments and V=1, logging the output (so that predictable
> > output can be generated).
> > 
> > Then run the same thing with the commit reverted, again logging it.  Then
> > diff the two and see what's different.
> 
> OK, I captured the output.  No diffs, as of course reverting things
> means one kernel was called 3.15-rc1 and one 3.15-rc1+ which meant more or 
> less every line changed.
> 
> It seems that on stock 3.15-rc1 (which fails to build) after running 
> depmod it kicks into
> 
> 	restore_upstream_debianization
> 
> which goes down a path which causes the file permission failures
> 
> Wheras 3.15-rc1 with the problem commit reverted this doesn't happen.
> 
> This is a really weird bug, I've tried reverting individual hunks of the 
> problem commit manually and it is not clear at all to me what is 
> triggering this.

OK, in debian/image.mk we have this:

  ifeq ($(strip $(HAVE_INST_PATH)),)
        test ! -f System.map ||  cp System.map                         \
                        $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
        test ! -f System.map ||  chmod 644                             \
                        $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
        cp $(kimagesrc) $(kimagedest)
  else
        $(restore_upstream_debianization)
        $(MAKE) $(EXTRAV_ARG) INSTALL_MOD_PATH=$(INSTALL_MOD_PATH)           \
                INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)  \
                INSTALL_PATH=$(INT_IMAGE_DESTDIR) $(CROSS_ARG) $(KPKG_KBUILD_INSTALL_TARGET)
  endif


So in the good case it goes down the if, in the bad case down the else.


I'm assuming that the ($(strip $(HAVE_INST_PATH)),) is being confused by
the INSTALL_DTBS_PATH introduced by the changeset.

So I guess that makes this a debian bug?

Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

On Tue, 15 Apr 2014, Vince Weaver wrote:

> On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> 
> > Right, so the thing to do is to run a build with the commit applied,
> > without any -j arguments and V=1, logging the output (so that predictable
> > output can be generated).
> > 
> > Then run the same thing with the commit reverted, again logging it.  Then
> > diff the two and see what's different.
> 
> OK, I captured the output.  No diffs, as of course reverting things
> means one kernel was called 3.15-rc1 and one 3.15-rc1+ which meant more or 
> less every line changed.
> 
> It seems that on stock 3.15-rc1 (which fails to build) after running 
> depmod it kicks into
> 
> 	restore_upstream_debianization
> 
> which goes down a path which causes the file permission failures
> 
> Wheras 3.15-rc1 with the problem commit reverted this doesn't happen.
> 
> This is a really weird bug, I've tried reverting individual hunks of the 
> problem commit manually and it is not clear at all to me what is 
> triggering this.

OK, in debian/image.mk we have this:

  ifeq ($(strip $(HAVE_INST_PATH)),)
        test ! -f System.map ||  cp System.map                         \
                        $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
        test ! -f System.map ||  chmod 644                             \
                        $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
        cp $(kimagesrc) $(kimagedest)
  else
        $(restore_upstream_debianization)
        $(MAKE) $(EXTRAV_ARG) INSTALL_MOD_PATH=$(INSTALL_MOD_PATH)           \
                INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)  \
                INSTALL_PATH=$(INT_IMAGE_DESTDIR) $(CROSS_ARG) $(KPKG_KBUILD_INSTALL_TARGET)
  endif


So in the good case it goes down the if, in the bad case down the else.


I'm assuming that the ($(strip $(HAVE_INST_PATH)),) is being confused by
the INSTALL_DTBS_PATH introduced by the changeset.

So I guess that makes this a debian bug?

Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Russell King - ARM Linux]

On Tue, Apr 15, 2014 at 03:30:15PM -0400, Vince Weaver wrote:
>   ifeq ($(strip $(HAVE_INST_PATH)),)
>         test ! -f System.map ||  cp System.map                         \
>                         $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
>         test ! -f System.map ||  chmod 644                             \
>                         $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
>         cp $(kimagesrc) $(kimagedest)
>   else
>         $(restore_upstream_debianization)
>         $(MAKE) $(EXTRAV_ARG) INSTALL_MOD_PATH=$(INSTALL_MOD_PATH)           \
>                 INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)  \
>                 INSTALL_PATH=$(INT_IMAGE_DESTDIR) $(CROSS_ARG) $(KPKG_KBUILD_INSTALL_TARGET)
>   endif
> 
> 
> So in the good case it goes down the if, in the bad case down the else.
> 
> 
> I'm assuming that the ($(strip $(HAVE_INST_PATH)),) is being confused by
> the INSTALL_DTBS_PATH introduced by the changeset.
> 
> So I guess that makes this a debian bug?

Hmm, any ideas what HAVE_INST_PATH is set to?  It would be good to have
the complete picture on this bug. :)

Thanks.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> 
> Hmm, any ideas what HAVE_INST_PATH is set to?  It would be good to have
> the complete picture on this bug. :)

In debian/ruleset/misc/version_vars.mk

HAVE_INST_PATH  =$(call doit,grep -E '\(INSTALL_PATH\)' Makefile 2>/dev/null )


Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Russell King - ARM Linux]

On Tue, Apr 15, 2014 at 03:39:47PM -0400, Vince Weaver wrote:
> On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> > 
> > Hmm, any ideas what HAVE_INST_PATH is set to?  It would be good to have
> > the complete picture on this bug. :)
> 
> In debian/ruleset/misc/version_vars.mk
> 
> HAVE_INST_PATH  =$(call doit,grep -E '\(INSTALL_PATH\)' Makefile 2>/dev/null )

Ah, so that would match on:

export INSTALL_DTBS_PATH ?= $(INSTALL_PATH)/dtbs/$(KERNELRELEASE)
                             ^^^^^^^^^^^^^^

So it's matching on use of INSTALL_PATH not whether INSTALL_PATH has been
defined.  Interesting concept. :p

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Christoph Lameter]

Manoj: what is the right fix here?



On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:

> On Tue, Apr 15, 2014 at 03:39:47PM -0400, Vince Weaver wrote:
> > On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> > >
> > > Hmm, any ideas what HAVE_INST_PATH is set to?  It would be good to have
> > > the complete picture on this bug. :)
> >
> > In debian/ruleset/misc/version_vars.mk
> >
> > HAVE_INST_PATH  =$(call doit,grep -E '\(INSTALL_PATH\)' Makefile 2>/dev/null )
>
> Ah, so that would match on:
>
> export INSTALL_DTBS_PATH ?= $(INSTALL_PATH)/dtbs/$(KERNELRELEASE)
>                              ^^^^^^^^^^^^^^
>
> So it's matching on use of INSTALL_PATH not whether INSTALL_PATH has been
> defined.  Interesting concept. :p
>
>

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Vince Weaver]

On Tue, 15 Apr 2014, Vince Weaver wrote:

> On Tue, 15 Apr 2014, Vince Weaver wrote:
> 
> > On Tue, 15 Apr 2014, Russell King - ARM Linux wrote:
> > 
> > > Right, so the thing to do is to run a build with the commit applied,
> > > without any -j arguments and V=1, logging the output (so that predictable
> > > output can be generated).
> > > 
> > > Then run the same thing with the commit reverted, again logging it.  Then
> > > diff the two and see what's different.
> > 
> > OK, I captured the output.  No diffs, as of course reverting things
> > means one kernel was called 3.15-rc1 and one 3.15-rc1+ which meant more or 
> > less every line changed.
> > 
> > It seems that on stock 3.15-rc1 (which fails to build) after running 
> > depmod it kicks into
> > 
> > 	restore_upstream_debianization
> > 
> > which goes down a path which causes the file permission failures
> > 
> > Wheras 3.15-rc1 with the problem commit reverted this doesn't happen.
> > 
> > This is a really weird bug, I've tried reverting individual hunks of the 
> > problem commit manually and it is not clear at all to me what is 
> > triggering this.
> 
> OK, in debian/image.mk we have this:
> 
>   ifeq ($(strip $(HAVE_INST_PATH)),)
>         test ! -f System.map ||  cp System.map                         \
>                         $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
>         test ! -f System.map ||  chmod 644                             \
>                         $(TMPTOP)/$(IMAGEDIR)/System.map-$(KERNELRELEASE);
>         cp $(kimagesrc) $(kimagedest)
>   else
>         $(restore_upstream_debianization)
>         $(MAKE) $(EXTRAV_ARG) INSTALL_MOD_PATH=$(INSTALL_MOD_PATH)           \
>                 INSTALL_FW_PATH=$(INSTALL_MOD_PATH)/lib/firmware/$(KERNELRELEASE)  \
>                 INSTALL_PATH=$(INT_IMAGE_DESTDIR) $(CROSS_ARG) $(KPKG_KBUILD_INSTALL_TARGET)
>   endif
> 
> 
> So in the good case it goes down the if, in the bad case down the else.
> 
> 
> I'm assuming that the ($(strip $(HAVE_INST_PATH)),) is being confused by
> the INSTALL_DTBS_PATH introduced by the changeset.
> 
> So I guess that makes this a debian bug?

And it looks like this was reported to the list earlier and I somehow 
missed it :(

https://lkml.org/lkml/2014/4/3/333

Vince

Re: [bisected] broken make-kpkg kernel build in 3.15-rc1

[Christoph Lameter]

On Tue, 15 Apr 2014, Vince Weaver wrote:

> > Then run the same thing with the commit reverted, again logging it.  Then
> > diff the two and see what's different.
>
> OK, I captured the output.  No diffs, as of course reverting things
> means one kernel was called 3.15-rc1 and one 3.15-rc1+ which meant more or
> less every line changed.

Well just add another fake patch and they both will have the same name...
The diff would be very helpful.