pygrub: further improve grub2 support

pygrub: further improve grub2 support

[Ian Campbell]

Round 2 (3?) in the arms race against the Debian Squeeze grub packages.

      * Improve syntax error messages to say what actually went wrong
        instead of giving an arbitrary and basically useless integer. 
      * Improve handling of quoted values used with the "set" command,
        previously only the default variable was special cased to handle
        quoting. 
      * Allow for extra options to the menuentry command, syntax now
        appears to be
                        menuentry "TITLE" --option1 --option2 {...}

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Please include in 3.4 as well.

diff -r 4152a3ce90a7 tools/pygrub/src/GrubConf.py
--- a/tools/pygrub/src/GrubConf.py	Thu Mar 11 17:40:35 2010 +0000
+++ b/tools/pygrub/src/GrubConf.py	Mon Mar 15 08:51:07 2010 +0000
@@ -220,7 +220,6 @@
     def _get_default(self):
         return self._default
     def _set_default(self, val):
-        val = val.strip("\"")
         if val == "saved":
             self._default = 0
         else:
@@ -300,7 +299,15 @@
 
         if self.hasPassword():
             self.setPasswordAccess(False)
-    
+
+def grub2_handle_set(arg):
+    (com,arg) = grub_split(arg,2)
+    com="set:" + com
+    m = re.match("([\"\'])(.*)\\1", arg)
+    if m is not None:
+        arg=m.group(2) 
+    return (com,arg)
+
 class Grub2Image(_GrubImage):
     def __init__(self, title, lines):
         _GrubImage.__init__(self, title, lines)
@@ -309,9 +316,8 @@
         (com, arg) = grub_exact_split(line, 2)
 
         if com == "set":
-            (com,arg) = grub_split(arg,2)
-            com="set:" + com
-                
+            (com,arg) = grub2_handle_set(arg)
+            
         if self.commands.has_key(com):
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
@@ -373,17 +379,17 @@
                 continue
 
             # new image
-            title_match = re.match('^menuentry "(.*)" {', l)
+            title_match = re.match('^menuentry "(.*)" (.*){', l)
             if title_match:
                 if img is not None:
-                    raise RuntimeError, "syntax error 1 %d %s" % (len(img),img)
+                    raise RuntimeError, "syntax error: cannot nest menuentry (%d %s)" % (len(img),img)
                 img = []
                 title = title_match.group(1)
                 continue
             
             if l.startswith("}"):
                 if img is None:
-                    raise RuntimeError, "syntax error 2 %d %s" % (len(img),img)
+                    raise RuntimeError, "syntax error: closing brace without menuentry"
 
                 self.add_image(Grub2Image(title, img))
                 img = None
@@ -396,8 +402,7 @@
             (com, arg) = grub_exact_split(l, 2)
         
             if com == "set":
-                (com,arg) = grub_split(arg,2)
-                com="set:" + com
+                (com,arg) = grub2_handle_set(arg)
                 
             if self.commands.has_key(com):
                 if self.commands[com] is not None:
@@ -410,7 +415,7 @@
                 logging.warning("Unknown directive %s" %(com,))
             
         if img is not None:
-            raise RuntimeError, "syntax error 3 %d %s" % (len(img),img)
+            raise RuntimeError, "syntax error: end of file with open menuentry(%d %s)" % (len(img),img)
 
         if self.hasPassword():
             self.setPasswordAccess(False)

Re: pygrub: further improve grub2 support

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 4297 bytes --]

In Ubuntu 10.04 beta grub entry root looks like :

  set root='(/dev/sda,1)'  vs  set  root=(hd0,1) in 9.10

Standard trick with loading PV DomU via HVM image fails.
Looks like "pygrub" cannot parse new root's notation

Boris.
P.S.
Attempt just to replace with old one
doesn't help in my case.

--- On Mon, 3/15/10, Ian Campbell <Ian.Campbell@citrix.com> wrote:

From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: [Xen-devel] pygrub: further improve grub2 support
To: xen-devel@lists.xensource.com
Date: Monday, March 15, 2010, 4:56 AM


Round 2 (3?) in the arms race against the Debian Squeeze grub packages.

      * Improve syntax error messages to say what actually went wrong
        instead of giving an arbitrary and basically useless integer. 
      * Improve handling of quoted values used with the "set" command,
        previously only the default variable was special cased to handle
        quoting. 
      * Allow for extra options to the menuentry command, syntax now
        appears to be
                        menuentry "TITLE" --option1 --option2 {...}

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Please include in 3.4 as well.

diff -r 4152a3ce90a7 tools/pygrub/src/GrubConf.py
--- a/tools/pygrub/src/GrubConf.py    Thu Mar 11 17:40:35 2010 +0000
+++ b/tools/pygrub/src/GrubConf.py    Mon Mar 15 08:51:07 2010 +0000
@@ -220,7 +220,6 @@
     def _get_default(self):
         return self._default
     def _set_default(self, val):
-        val = val.strip("\"")
         if val == "saved":
             self._default = 0
         else:
@@ -300,7 +299,15 @@
 
         if self.hasPassword():
             self.setPasswordAccess(False)
-    
+
+def grub2_handle_set(arg):
+    (com,arg) = grub_split(arg,2)
+    com="set:" + com
+    m = re.match("([\"\'])(.*)\\1", arg)
+    if m is not None:
+        arg=m.group(2) 
+    return (com,arg)
+
 class Grub2Image(_GrubImage):
     def __init__(self, title, lines):
         _GrubImage.__init__(self, title, lines)
@@ -309,9 +316,8 @@
         (com, arg) = grub_exact_split(line, 2)
 
         if com == "set":
-            (com,arg) = grub_split(arg,2)
-            com="set:" + com
-                
+            (com,arg) = grub2_handle_set(arg)
+            
         if self.commands.has_key(com):
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
@@ -373,17 +379,17 @@
                 continue
 
             # new image
-            title_match = re.match('^menuentry "(.*)" {', l)
+            title_match = re.match('^menuentry "(.*)" (.*){', l)
             if title_match:
                 if img is not None:
-                    raise RuntimeError, "syntax error 1 %d %s" % (len(img),img)
+                    raise RuntimeError, "syntax error: cannot nest menuentry (%d %s)" % (len(img),img)
                 img = []
                 title = title_match.group(1)
                 continue
             
             if l.startswith("}"):
                 if img is None:
-                    raise RuntimeError, "syntax error 2 %d %s" % (len(img),img)
+                    raise RuntimeError, "syntax error: closing brace without menuentry"
 
                 self.add_image(Grub2Image(title, img))
                 img = None
@@ -396,8 +402,7 @@
             (com, arg) = grub_exact_split(l, 2)
         
             if com == "set":
-                (com,arg) = grub_split(arg,2)
-                com="set:" + com
+                (com,arg) = grub2_handle_set(arg)
                 
             if self.commands.has_key(com):
                 if self.commands[com] is not None:
@@ -410,7 +415,7 @@
                 logging.warning("Unknown directive %s" %(com,))
             
         if img is not None:
-            raise RuntimeError, "syntax error 3 %d %s" % (len(img),img)
+            raise RuntimeError, "syntax error: end of file with open menuentry(%d %s)" % (len(img),img)
 
         if self.hasPassword():
             self.setPasswordAccess(False)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 7514 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pygrub: further improve grub2 support

[Ian Campbell]

On Fri, 2010-03-26 at 10:41 +0000, Boris Derzhavets wrote:
> In Ubuntu 10.04 beta grub entry root looks like :
> 
>   set root='(/dev/sda,1)'  vs  set  root=(hd0,1) in 9.10

So grub in Ubuntu 10.04 uses _Linux_ device naming in the bootloader
configuration? That seems like an awfully big break with tradition.

Do you know how grub2 tries to interpret this, perhaps using a
device.map type file in /boot/grub? Or perhaps it is simply a bug in
10.04 beta?

sda isn't really a valid name for a modern PV domU kernel anyway. It
should be xvda. Looks like you need to do some additional tailoring to
convert your HVM Ubuntu domain into a PV one.

Ian.

Re: pygrub: further improve grub2 support

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1544 bytes --]

I've tried to PV domU with ready HVM domU via profile:-

memory = 2048
name = "UbuntuLynx"
bootloader="/usr/local/bin/pygrub"
vcpus = 2
vif = [ 'bridge=eth0' ]
disk = [ 'phy:/dev/sda8,xvda,w']
vfb= ['type=vnc,vncunused=1']

> Do you know how grub2 tries to interpret this, perhaps using a
> device.map type file in /boot/grub? Or perhaps it is simply a bug in
> 10.04 beta?

In meantime i don't know. I will take a look at device.map a bit latter.
But , it doesn't look to myself  as "beta"  issue

Boris.

--- On Fri, 3/26/10, Ian Campbell <Ian.Campbell@citrix.com> wrote:

From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: [Xen-devel] pygrub: further improve grub2 support
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 6:49 AM

On Fri, 2010-03-26 at 10:41 +0000, Boris Derzhavets wrote:
> In Ubuntu 10.04 beta grub entry root looks like :
> 
>   set root='(/dev/sda,1)'  vs  set  root=(hd0,1) in 9.10

So grub in Ubuntu 10.04 uses _Linux_ device naming in the bootloader
configuration? That seems like an awfully big break with tradition.

Do you know how grub2 tries to interpret this, perhaps using a
device.map type file in /boot/grub? Or perhaps it is simply a bug in
10.04 beta?

sda isn't really a valid name for a modern PV domU kernel anyway. It
should be xvda. Looks like you need to do some additional tailoring to
convert your HVM Ubuntu domain into a PV one.

Ian.




      

[-- Attachment #1.2: Type: text/html, Size: 1989 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pygrub: further improve grub2 support

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 2741 bytes --]

That's all i was able to get for now :-

root@Ubuntu10:/boot/grub# ls -l d*

-rw-r--r-- 1 root root 1916 2010-03-26 00:18 datehook.mod
-rw-r--r-- 1 root root 2404 2010-03-26 00:18 date.mod
-rw-r--r-- 1 root root 1333 2010-03-26 00:18 datetime.mod
-rw-r--r-- 1 root root  512 2010-03-26 00:18 diskboot.img
-rw-r--r-- 1 root root 1952 2010-03-26 00:18 dm_nv.mod
-rw-r--r-- 1 root root 5616 2010-03-26 00:18 drivemap.mod

root@Ubuntu10:/boot/grub# file drivemap.mod
drivemap.mod: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped

Boris.
P.S. I was installing Ubuntu Desktop on 10.04 HVM to get common exchange 
buffer.

--- On Fri, 3/26/10, Boris Derzhavets <bderzhavets@yahoo.com> wrote:

From: Boris Derzhavets <bderzhavets@yahoo.com>
Subject: Re: [Xen-devel] pygrub: further improve grub2 support
To: "Ian Campbell" <Ian.Campbell@citrix.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 6:57 AM

I've tried to PV domU with ready HVM domU via profile:-

memory = 2048
name = "UbuntuLynx"
bootloader="/usr/local/bin/pygrub"
vcpus = 2
vif = [ 'bridge=eth0' ]
disk = [ 'phy:/dev/sda8,xvda,w']
vfb= ['type=vnc,vncunused=1']

> Do you know how grub2 tries to interpret this, perhaps using a
> device.map type file in /boot/grub? Or perhaps it is simply a bug in
> 10.04 beta?

In meantime i don't know. I will take a look at device.map a bit latter.
But , it doesn't look to myself  as "beta"  issue

Boris.

--- On Fri, 3/26/10, Ian Campbell <Ian.Campbell@citrix.com> wrote:

From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: [Xen-devel] pygrub: further
 improve grub2 support
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 6:49 AM

On Fri, 2010-03-26 at 10:41 +0000, Boris Derzhavets wrote:
> In Ubuntu 10.04 beta grub entry root looks like :
> 
>   set root='(/dev/sda,1)'  vs  set  root=(hd0,1) in 9.10

So grub in Ubuntu 10.04 uses _Linux_ device naming in the bootloader
configuration? That seems like an awfully big break with tradition.

Do you know how grub2 tries to interpret this, perhaps using a
device.map type file in /boot/grub? Or perhaps it is simply a bug in
10.04 beta?

sda isn't really a valid name for a modern PV domU kernel anyway. It
should be xvda. Looks like you need to do some additional tailoring to
convert your HVM Ubuntu domain into a PV
 one.

Ian.




      
-----Inline Attachment Follows-----

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 3766 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pygrub: further improve grub2 support

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 3904 bytes --]

I would guess HVM image is bkoken on for Ubuntu 10.04 Server Beta
Attempt to load PV via profile :-

memory = 2048
name = "Ubuntu10.04"
kernel="/home/boris/lynx/vmlinuz-2.6.32-16-server"
ramdisk="/home/boris/lynx/initrd.img-2.6.32-16-server"
vcpus = 2
vif = [ 'bridge=eth0' ]
disk = [ 'phy:/dev/sda8,xvda,w']
extra="root=/dev/sda1 ro console=hvc0"

xm create -c Lynx.cfg

[    0.149124] XENBUS: Device with no driver: device/vbd/51712
[    0.149129] XENBUS: Device with no driver: device/vif/0
[    0.149135] XENBUS: Device with no driver: device/console/0
[    0.149149]   Magic number: 1:252:3141
[    0.149160] /build/buildd/linux-2.6.32/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.149167] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    0.149172] EDD information not available.
[    0.149254] Freeing unused kernel memory: 800k freed
[    0.149491] Write protecting the kernel read-only data: 7936k
Loading, please wait...
[    0.175675] udev: starting version 151
Begin: Loading essential drivers... ...
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
[    0.241186] blkfront: xvda: barriers enabled
[    0.241453]  xvda: xvda1 xvda2 < xvda5 >
Gave up waiting for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
   - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/sda1 does not exist. Dropping to a shell!


BusyBox v1.13.3 (Ubuntu 1:1.13.3-1ubuntu9) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) 


/dev/sda1 does show up as root FS for HVM has been built.

Boris.

--- On Fri, 3/26/10, Boris Derzhavets <bderzhavets@yahoo.com> wrote:

From: Boris Derzhavets <bderzhavets@yahoo.com>
Subject: Re: [Xen-devel] pygrub: further improve grub2 support
To: "Ian Campbell" <Ian.Campbell@citrix.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 6:57 AM

I've tried to PV domU with ready HVM domU via profile:-

memory = 2048
name = "UbuntuLynx"
bootloader="/usr/local/bin/pygrub"
vcpus = 2
vif = [ 'bridge=eth0' ]
disk = [ 'phy:/dev/sda8,xvda,w']
vfb= ['type=vnc,vncunused=1']

> Do you know how grub2 tries to interpret this, perhaps using a
> device.map type file in /boot/grub? Or perhaps it is simply a bug in
> 10.04 beta?

In meantime i don't know. I will take a look at device.map a bit latter.
But , it doesn't look to myself  as "beta"  issue

Boris.

--- On Fri, 3/26/10, Ian Campbell <Ian.Campbell@citrix.com> wrote:

From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: [Xen-devel] pygrub: further
 improve grub2 support
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 6:49 AM

On Fri, 2010-03-26 at 10:41 +0000, Boris Derzhavets wrote:
> In Ubuntu 10.04 beta grub entry root looks like :
> 
>   set root='(/dev/sda,1)'  vs  set  root=(hd0,1) in 9.10

So grub in Ubuntu 10.04 uses _Linux_ device naming in the bootloader
configuration? That seems like an awfully big break with tradition.

Do you know how grub2 tries to interpret this, perhaps using a
device.map type file in /boot/grub? Or perhaps it is simply a bug in
10.04 beta?

sda isn't really a valid name for a modern PV domU kernel anyway. It
should be xvda. Looks like you need to do some additional tailoring to
convert your HVM Ubuntu domain into a PV
 one.

Ian.




      
-----Inline Attachment Follows-----

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 5424 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pygrub: further improve grub2 support

[Ian Campbell]

On Fri, 2010-03-26 at 16:52 +0000, Boris Derzhavets wrote:
> I would guess HVM image is bkoken on for Ubuntu 10.04 Server Beta
> Attempt to load PV via profile :-
> 
> memory = 2048
> name = "Ubuntu10.04"
> kernel="/home/boris/lynx/vmlinuz-2.6.32-16-server"
> ramdisk="/home/boris/lynx/initrd.img-2.6.32-16-server"
> vcpus = 2
> vif = [ 'bridge=eth0' ]
> disk = [ 'phy:/dev/sda8,xvda,w']
> extra="root=/dev/sda1 ro console=hvc0"
> 

At a minimum this last line should be
extra="root=/dev/xvda1 ro console=hvc0"

sda is not a valid device name for a PV guest with a pvops kernel. There
may be other tailoring required within the guest as well.

Ian.

Re: pygrub: further improve grub2 support

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 3028 bytes --]

Thank you, i've got PV DomU loaded right away ( fixed error ):-

[    0.188837] udev: starting version 151
Begin: Loading essential drivers... ...
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
[    0.263402] blkfront: xvda: barriers enabled
[    0.263632]  xvda: xvda1 xvda2 < xvda5 >
Begin: Running /scripts/local-premount ...
Done.
[    0.537976] EXT4-fs (xvda1): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
fsck from util-linux-ng 2.17.2
/dev/xvda1: clean, 154798/960992 files, 786049/3837952 blocks

Ubuntu lucid (development branch) Ubuntu10 hvc0

Ubuntu10 login: root
Password: 
Last login: Fri Mar 26 13:32:29 MSK 2010 on tty1
Linux Ubuntu10 2.6.32-16-server #25-Ubuntu SMP Tue Mar 9 17:40:50 UTC 2010 x86_64

For documentation and useful resources, please visit:
 * http://ubuntu.com/server/doc

  System information as of Fri Mar 26 00:55:37 MSK 2010

  System load: 0.05              Memory usage: 2%   Processes:       79
  Usage of /:  5.8% of 14.41GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at https://landscape.canonical.com/

75 packages can be updated.
0 updates are security updates.

Checking for a new ubuntu release
No new release found

For documentation and useful resources, please visit:
 * http://ubuntu.com/server/doc

  System information as of Fri Mar 26 20:01:05 MSK 2010

  System load: 0.06               Memory usage: 1%   Processes:       101
  Usage of /:  19.2% of 14.41GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at https://landscape.canonical.com/

Checking for a new ubuntu release
No new release found
root@Ubuntu10:~# uname -a
Linux Ubuntu10 2.6.32-16-server #25-Ubuntu SMP Tue Mar 9 17:40:50 UTC 2010 x86_64 GNU/Linux


Boris.

--- On Fri, 3/26/10, Ian Campbell <Ian.Campbell@citrix.com> wrote:

From: Ian Campbell <Ian.Campbell@citrix.com>
Subject: Re: [Xen-devel] pygrub: further improve grub2 support
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Date: Friday, March 26, 2010, 12:55 PM

On Fri, 2010-03-26 at 16:52 +0000, Boris Derzhavets wrote:
> I would guess HVM image is bkoken on for Ubuntu 10.04 Server Beta
> Attempt to load PV via profile :-
> 
> memory = 2048
> name = "Ubuntu10.04"
> kernel="/home/boris/lynx/vmlinuz-2.6.32-16-server"
> ramdisk="/home/boris/lynx/initrd.img-2.6.32-16-server"
> vcpus = 2
> vif = [ 'bridge=eth0' ]
> disk = [ 'phy:/dev/sda8,xvda,w']
> extra="root=/dev/sda1 ro console=hvc0"
> 

At a minimum this last line should be
extra="root=/dev/xvda1 ro console=hvc0"

sda is not a valid device name for a PV guest with a pvops kernel. There
may be other tailoring required within the guest as well.

Ian.




      

[-- Attachment #1.2: Type: text/html, Size: 3959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel