pvgrub2 legacy_configfile issues: /sbin/init exec error -8

[Andy Smith <andy@strugglers.net>]

Hi,

I'm experimenting with pvgrub2 for the first time, and having some
problems getting a grub-legacy menu.lst config to work.

Using a Debian jessie Xen dom0 and guest. In dom0, grub is built
from git:

----------------------------------------------------------------------
$ sudo apt-get build-dep grub
$ git clone git://git.savannah.gnu.org/grub.git
$ cd grub/
$ ./autogen.sh
$ ./configure --prefix=/opt/grub --target=amd64 --with-platform=xen
$ make
$ sudo make install
$ sudo mkdir -vp /opt/grub/share/memdisk
$ sudo tar -C /opt/grub/share/memdisk -cvf /opt/grub/share/memdisk.tar grub.cfg
$ sudo /opt/grub/bin/grub-mkimage -O x86_64-xen -c /opt/grub/etc/grub-bootstrap.cfg -m /opt/grub/share/memdisk.tar -o /opt/grub/lib/grub-x86_64-xen.bin -p /boot/grub /opt/grub/lib/grub/x86_64-xen/*.mod
----------------------------------------------------------------------

Contents of /opt/grub/etc/grub-bootstrap.cfg:

----------------------------------------------------------------------
normal (memdisk)/grub.cfg
----------------------------------------------------------------------

Contents of /opt/grub/share/memdisk/grub.cfg:

----------------------------------------------------------------------
if search -s -f /boot/xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/boot/xen/pvboot-x86_64.elf"
        read
        multiboot "/boot/xen/pvboot-x86_64.elf"
        boot
fi

if search -s -f /xen/pvboot-x86_64.elf ; then
        echo "Chainloading (${root})/xen/pvboot-x86_64.elf"
        read
        multiboot "/xen/pvboot-x86_64.elf"
        boot
fi

if search -s -f /boot/grub/grub.cfg ; then
        echo "Reading (${root})/boot/grub/grub.cfg"
        read
        configfile /boot/grub/grub.cfg
fi

if search -s -f /grub/grub.cfg ; then
        echo "Reading (${root})/grub/grub.cfg"
        read
        configfile /grub/grub.cfg
fi

if search -s -f /boot/grub/menu.lst ; then
        echo "Reading (${root})/boot/grub/menu.lst"
        read
        legacy_configfile /boot/grub/menu.lst
fi

if search -s -f /grub/menu.lst ; then
        set root=(xen/xvda,msdos1)
        echo "Reading (${root})/grub/menu.lst"
        read
        legacy_configfile /grub/menu.lst
fi
----------------------------------------------------------------------

Contents of /boot/grub/menu.lst in guest:

----------------------------------------------------------------------
default         0
timeout         5
color cyan/blue white/blue
title           Debian GNU/Linux, kernel 3.16.0-4-amd64
root            (hd0,0)
kernel          /boot/vmlinuz-3.16.0-4-amd64 root=UUID=38420e46-6123-477d-ba23-baeba8ac0d59 ro 
initrd          /boot/initrd.img-3.16.0-4-amd64
----------------------------------------------------------------------

So, purpose of the grub.cfg in the dom0 memdisk is to try to, in
order:

1. Chainload pvgrub from guest, in boot/xen/ or in /xen/.

2. Parse guest's /boot/grub/grub.cfg or /grub/grub.cfg in pvgrub2.

3. Parse guest's legacy /boot/grub/menu.lst or /grub/menu.lst in
   pvgrub2.

(The "read" commands were used just to make it pause until I hit
return so I could read the "echo" before it.)

I've tested that outcomes (1) and (2) work. That is, chainload to
pvgrub2 in guest works and so does pvgrub parsing guest's grub.cfg.

After purging grub-xen and grub-pc packages from the guest and
installing grub-legacy, ensuring there is no /boot/xen/*, no
/boot/grub/grub.cfg, and that there is a /boot/grub/menu.lst as
above, an attempt to boot the guest results in a kernel panic at the
point where the guest kernel tries to execute /sbin/init:

----------------------------------------------------------------------
.
.
.
[    1.470403] Freeing unused kernel memory: 216K (ffff8800017ca000 - ffff880001800000)
[    1.471934] Failed to execute /init (error -8)
[    1.472673] Starting init: /sbin/init exists but couldn't execute it (error -8)
[    1.473359] Starting init: /bin/sh exists but couldn't execute it (error -8)
[    1.473365] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[    1.473369] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.39-1
[    1.473372]  0000000000000000 ffffffff81514c11 ffffffff81705310 ffff88003e23bf40
[    1.473377]  ffffffff8151195e ffffffff00000008 ffff88003e23bf50 ffff88003e23bef0
[    1.473380]  ffff88003e23bef8 0000000000000000 0000000000000495 0000000000000495
[    1.473384] Call Trace:
----------------------------------------------------------------------

Error -8 is an exec format error, almost as if it got booted as a
32-bit guest and tried to execute a 64-bit binary.

Switching the guest config back to pygrub allows it to boot without
any alteration of the menu.lst inside the guest.

So, I feel I am missing something very simple here, but much
searching has not produced any answer. Has anyone seen this before?

I appreciate I may have sent this query to the wrong place. It
doesn't feel like a Xen issue so I haven't yet tried xen-devel list.
Yet also probably not a bug in grub, just my misunderstanding, so I
didn't put anything in your bugzilla. If you think there is a more
appropriate place to send the query please do let me know.

Cheers,
Andy