moving ata initialisation to a command

moving ata initialisation to a command

[Robert Millan]

I'd like to move ata.mod initialisation away from its _init routine and into
a separate command.  This way it isn't a nuissance when it gets included in
monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
completely.

Does that sound fine?

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: moving ata initialisation to a command

[Christian Franke]

Robert Millan wrote:
> I'd like to move ata.mod initialisation away from its _init routine and into
> a separate command.  This way it isn't a nuissance when it gets included in
> monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
> completely.
>
> Does that sound fine?
>
>   

Yes.

Christian

Re: moving ata initialisation to a command

[Vesa Jääskeläinen]

Robert Millan wrote:
> I'd like to move ata.mod initialisation away from its _init routine and into
> a separate command.  This way it isn't a nuissance when it gets included in
> monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
> completely.
> 
> Does that sound fine?
> 

While you are there why not add optional IO-base argument so one could
use more than one controller lurking in other IO-bases (second/third PCI
?). Of course there needs to be some kind of auto detect for easier
usage for normal users.

Re: moving ata initialisation to a command

[Robert Millan]

On Sun, Dec 16, 2007 at 06:06:42PM +0200, Vesa Jääskeläinen wrote:
> Robert Millan wrote:
> > I'd like to move ata.mod initialisation away from its _init routine and into
> > a separate command.  This way it isn't a nuissance when it gets included in
> > monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
> > completely.
> > 
> > Does that sound fine?
> > 
> 
> While you are there why not add optional IO-base argument so one could
> use more than one controller lurking in other IO-bases (second/third PCI
> ?). Of course there needs to be some kind of auto detect for easier
> usage for normal users.

I don't like that.  Sounds like a workaround for not having proper PCI
support.. :-/

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: moving ata initialisation to a command

[Marco Gerards]

Robert Millan <rmh@aybabtu.com> writes:

[...]

>> While you are there why not add optional IO-base argument so one could
>> use more than one controller lurking in other IO-bases (second/third PCI
>> ?). Of course there needs to be some kind of auto detect for easier
>> usage for normal users.
>
> I don't like that.  Sounds like a workaround for not having proper PCI
> support.. :-/

Right, I plan to add PCI support soon.

--
Marco

Re: moving ata initialisation to a command

[Marco Gerards]

Robert Millan <rmh@aybabtu.com> writes:

> I'd like to move ata.mod initialisation away from its _init routine and into
> a separate command.  This way it isn't a nuissance when it gets included in
> monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
> completely.
>
> Does that sound fine?

No.  What if you put ata.mod in core.img?  You would expect your disks
will be detected and that grub.cfg will be loaded immediately.  The
user doesn't want to run a command to enable ata.mod and afterwards
load the grub.cfg manually.

A better solution, IMO, would be changing grub-mkrescue so it doesn't
load all modules.

--
Marco

Re: moving ata initialisation to a command

[Pavel Roskin]

On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
> A better solution, IMO, would be changing grub-mkrescue so it doesn't
> load all modules.

Maybe grub-mkrescue should create a filesystem?  Even FAT should be
fine.  This way, it will be possible to load problematic modules from
the filesystem.  The only problem would be dependency on filesystem
making tools.  Fortunately, mtools is quite common.

-- 
Regards,
Pavel Roskin

Re: moving ata initialisation to a command

[Marco Gerards]

Pavel Roskin <proski@gnu.org> writes:

> On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
>> A better solution, IMO, would be changing grub-mkrescue so it doesn't
>> load all modules.
>
> Maybe grub-mkrescue should create a filesystem?  Even FAT should be
> fine.  This way, it will be possible to load problematic modules from
> the filesystem.  The only problem would be dependency on filesystem
> making tools.  Fortunately, mtools is quite common.

This sounds like a good plan.  For the CDROM image, mkisofs can be
used.

--
Marco

Re: moving ata initialisation to a command

[Robert Millan]

On Mon, Dec 17, 2007 at 08:01:06PM +0100, Marco Gerards wrote:
> Pavel Roskin <proski@gnu.org> writes:
> 
> > On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
> >> A better solution, IMO, would be changing grub-mkrescue so it doesn't
> >> load all modules.
> >
> > Maybe grub-mkrescue should create a filesystem?  Even FAT should be
> > fine.  This way, it will be possible to load problematic modules from
> > the filesystem.  The only problem would be dependency on filesystem
> > making tools.  Fortunately, mtools is quite common.
> 
> This sounds like a good plan.  For the CDROM image, mkisofs can be
> used.

For non-CD images we could use genext2fs (same interface as genisoimage
/ mkisofs).

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: moving ata initialisation to a command

[Robert Millan]

On Mon, Dec 17, 2007 at 01:26:31PM -0500, Pavel Roskin wrote:
> On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
> > A better solution, IMO, would be changing grub-mkrescue so it doesn't
> > load all modules.
> 
> Maybe grub-mkrescue should create a filesystem?  Even FAT should be
> fine.  This way, it will be possible to load problematic modules from
> the filesystem.  The only problem would be dependency on filesystem
> making tools.  Fortunately, mtools is quite common.

I'd prefer to support embedding filesystems in core.img instead.  This
solves the problem for every situation instead of just PC boot media
(e.g. LinuxBIOS ELF payload image).

Anyway, as long as root privileges aren't made necessary, I'm fine with
grub-mkrescue creating a filesystem (one saner than FAT, preferably ;-).

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: moving ata initialisation to a command

[Marco Gerards]

Robert Millan <rmh@aybabtu.com> writes:

> On Mon, Dec 17, 2007 at 01:26:31PM -0500, Pavel Roskin wrote:
>> On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
>> > A better solution, IMO, would be changing grub-mkrescue so it doesn't
>> > load all modules.
>> 
>> Maybe grub-mkrescue should create a filesystem?  Even FAT should be
>> fine.  This way, it will be possible to load problematic modules from
>> the filesystem.  The only problem would be dependency on filesystem
>> making tools.  Fortunately, mtools is quite common.
>
> I'd prefer to support embedding filesystems in core.img instead.  This
> solves the problem for every situation instead of just PC boot media
> (e.g. LinuxBIOS ELF payload image).

Agreed, I would like to see such feature.  Also for stuff like
diskless boot.

What I had in mind was the following.  We could perhaps make it
possible to add all files using grub-mkimage, not only modules.  When
you add modules, those will be loaded automatically.  All other files
will be ignored.

Another filesystem module has to be written.  It scans over these
files and will make them available.  This can be simple and elegant.
I do not like the idea of adding a diskimage or so.  Are you
interested in working on this?

> Anyway, as long as root privileges aren't made necessary, I'm fine with
> grub-mkrescue creating a filesystem (one saner than FAT, preferably ;-).

Great!  Will you do that? :-)

--
Marco

Re: moving ata initialisation to a command

[Robert Millan]

On Fri, Dec 21, 2007 at 05:47:36PM +0100, Marco Gerards wrote:
> Robert Millan <rmh@aybabtu.com> writes:
> 
> > On Mon, Dec 17, 2007 at 01:26:31PM -0500, Pavel Roskin wrote:
> >> On Mon, 2007-12-17 at 17:01 +0100, Marco Gerards wrote:
> >> > A better solution, IMO, would be changing grub-mkrescue so it doesn't
> >> > load all modules.
> >> 
> >> Maybe grub-mkrescue should create a filesystem?  Even FAT should be
> >> fine.  This way, it will be possible to load problematic modules from
> >> the filesystem.  The only problem would be dependency on filesystem
> >> making tools.  Fortunately, mtools is quite common.
> >
> > I'd prefer to support embedding filesystems in core.img instead.  This
> > solves the problem for every situation instead of just PC boot media
> > (e.g. LinuxBIOS ELF payload image).
> 
> Agreed, I would like to see such feature.  Also for stuff like
> diskless boot.
> 
> What I had in mind was the following.  We could perhaps make it
> possible to add all files using grub-mkimage, not only modules.  When
> you add modules, those will be loaded automatically.  All other files
> will be ignored.
> 
> Another filesystem module has to be written.  It scans over these
> files and will make them available.  This can be simple and elegant.
> I do not like the idea of adding a diskimage or so.  Are you
> interested in working on this?

Well, I already have an almost-working implementation.  I guess I'll have to
defend it then! :-P

Really, it's clean and elegant.  Very simple and flexible.  Once I get the
address detection right, that is..

> > Anyway, as long as root privileges aren't made necessary, I'm fine with
> > grub-mkrescue creating a filesystem (one saner than FAT, preferably ;-).
> 
> Great!  Will you do that? :-)

No, I tried once and it was a mess of things.  Besides, when we have embedded
filesystem support this won't add any real advantage.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On Fri, Dec 21, 2007 at 07:04:05PM +0100, Robert Millan wrote:
> > >
> > > I'd prefer to support embedding filesystems in core.img instead.  This
> > > solves the problem for every situation instead of just PC boot media
> > > (e.g. LinuxBIOS ELF payload image).
> > 
> > Agreed, I would like to see such feature.  Also for stuff like
> > diskless boot.
> > 
> > What I had in mind was the following.  We could perhaps make it
> > possible to add all files using grub-mkimage, not only modules.  When
> > you add modules, those will be loaded automatically.  All other files
> > will be ignored.
> > 
> > Another filesystem module has to be written.  It scans over these
> > files and will make them available.  This can be simple and elegant.
> > I do not like the idea of adding a diskimage or so.  Are you
> > interested in working on this?
> 
> Well, I already have an almost-working implementation.  I guess I'll have to
> defend it then! :-P
> 
> Really, it's clean and elegant.  Very simple and flexible.  Once I get the
> address detection right, that is..

Please have a look at the attached patch.  This allows adding an arbitrary
number of files.  No extra filesystem module is needed, since any of the
supported filesystems can be used with this feature:

  genisoimage -o filesystem_image -r stuff
  grub-mkimage -o core.img -m filesystem_image

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

[-- Attachment #2: memdisk.diff --]
[-- Type: text/x-diff, Size: 10896 bytes --]

diff -x '*.mk' -Nurp grub2/conf/i386-pc.rmk grub2.memdisk/conf/i386-pc.rmk
--- grub2/conf/i386-pc.rmk	2007-12-21 12:41:23.000000000 +0100
+++ grub2.memdisk/conf/i386-pc.rmk	2007-12-21 21:47:57.000000000 +0100
@@ -39,7 +39,7 @@ kernel_img_HEADERS = arg.h boot.h cache.
 	env.h err.h file.h fs.h kernel.h loader.h misc.h mm.h net.h parser.h \
 	partition.h pc_partition.h rescue.h symbol.h term.h time.h types.h \
 	machine/biosdisk.h machine/boot.h machine/console.h machine/init.h \
-	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h
+	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h machine/kernel.h
 kernel_img_CFLAGS = $(COMMON_CFLAGS)
 kernel_img_ASFLAGS = $(COMMON_ASFLAGS)
 kernel_img_LDFLAGS = $(COMMON_LDFLAGS) -Wl,-N,-Ttext,8200 $(COMMON_CFLAGS)
@@ -136,7 +136,7 @@ pkgdata_MODULES = biosdisk.mod _chain.mo
 	_multiboot.mod chain.mod multiboot.mod reboot.mod halt.mod	\
 	vbe.mod vbetest.mod vbeinfo.mod video.mod gfxterm.mod \
 	videotest.mod play.mod bitmap.mod tga.mod cpuid.mod serial.mod ata.mod \
-	vga.mod
+	vga.mod memdisk.mod
 
 # For biosdisk.mod.
 biosdisk_mod_SOURCES = disk/i386/pc/biosdisk.c
@@ -263,4 +263,9 @@ vga_mod_SOURCES = term/i386/pc/vga.c
 vga_mod_CFLAGS = $(COMMON_CFLAGS)
 vga_mod_LDFLAGS = $(COMMON_LDFLAGS)
 
+# For memdisk.mod.
+memdisk_mod_SOURCES = disk/memdisk.c
+memdisk_mod_CFLAGS = $(COMMON_CFLAGS)
+memdisk_mod_LDFLAGS = $(COMMON_LDFLAGS)
+
 include $(srcdir)/conf/common.mk
diff -x '*.mk' -Nurp grub2/disk/memdisk.c grub2.memdisk/disk/memdisk.c
--- grub2/disk/memdisk.c	1970-01-01 01:00:00.000000000 +0100
+++ grub2.memdisk/disk/memdisk.c	2007-12-21 21:43:25.000000000 +0100
@@ -0,0 +1,92 @@
+/* memdisk.c - Access embedded memory disk.  */
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2007  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <grub/disk.h>
+#include <grub/dl.h>
+#include <grub/machine/kernel.h>
+#include <grub/misc.h>
+#include <grub/mm.h>
+#include <grub/types.h>
+
+static grub_addr_t memdisk_addr;
+
+static int
+grub_memdisk_iterate (int (*hook) (const char *name))
+{
+  return hook ("memdisk");
+}
+
+static grub_err_t
+grub_memdisk_open (const char *name, grub_disk_t disk)
+{
+  if (grub_strcmp (name, "memdisk"))
+      return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "not a memdisk");
+
+  disk->total_sectors = grub_memdisk_image_size / GRUB_DISK_SECTOR_SIZE;
+  disk->id = (int) 'mdsk';
+
+  disk->has_partitions = 0;
+  disk->data = NULL;
+
+  return GRUB_ERR_NONE;
+}
+
+static void
+grub_memdisk_close (grub_disk_t disk __attribute((unused)))
+{
+}
+
+static grub_err_t
+grub_memdisk_read (grub_disk_t disk, grub_disk_addr_t sector,
+		    grub_size_t size, char *buf)
+{
+  grub_memcpy (buf, memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), size << GRUB_DISK_SECTOR_BITS);
+  return 0;
+}
+
+static grub_err_t
+grub_memdisk_write (grub_disk_t disk, grub_disk_addr_t sector,
+		     grub_size_t size, const char *buf)
+{
+  grub_memcpy (memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), buf, size << GRUB_DISK_SECTOR_BITS);
+  return 0;
+}
+
+static struct grub_disk_dev grub_memdisk_dev =
+  {
+    .name = "memdisk",
+    .id = GRUB_DISK_DEVICE_MEMDISK_ID,
+    .iterate = grub_memdisk_iterate,
+    .open = grub_memdisk_open,
+    .close = grub_memdisk_close,
+    .read = grub_memdisk_read,
+    .write = grub_memdisk_write,
+    .next = 0
+  };
+
+GRUB_MOD_INIT(memdisk)
+{
+  memdisk_addr = 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;
+  grub_disk_dev_register (&grub_memdisk_dev);
+}
+
+GRUB_MOD_FINI(memdisk)
+{
+  grub_disk_dev_unregister (&grub_memdisk_dev);
+}
diff -x '*.mk' -Nurp grub2/include/grub/disk.h grub2.memdisk/include/grub/disk.h
--- grub2/include/grub/disk.h	2007-11-05 17:15:26.000000000 +0100
+++ grub2.memdisk/include/grub/disk.h	2007-12-21 17:47:06.000000000 +0100
@@ -35,7 +35,8 @@ enum grub_disk_dev_id
     GRUB_DISK_DEVICE_RAID_ID,
     GRUB_DISK_DEVICE_LVM_ID,
     GRUB_DISK_DEVICE_HOST_ID,
-    GRUB_DISK_DEVICE_ATA_ID
+    GRUB_DISK_DEVICE_ATA_ID,
+    GRUB_DISK_DEVICE_MEMDISK_ID,
   };
 
 struct grub_disk;
diff -x '*.mk' -Nurp grub2/include/grub/i386/pc/kernel.h grub2.memdisk/include/grub/i386/pc/kernel.h
--- grub2/include/grub/i386/pc/kernel.h	2007-07-22 01:32:23.000000000 +0200
+++ grub2.memdisk/include/grub/i386/pc/kernel.h	2007-12-21 18:22:26.000000000 +0100
@@ -34,8 +34,11 @@
 /* The offset of GRUB_INSTALL_BSD_PART.  */
 #define GRUB_KERNEL_MACHINE_INSTALL_BSD_PART	0x18
 
+/* The offset of GRUB_MEMDISK_IMAGE_SIZE.  */
+#define GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE	0x1c
+
 /* The offset of GRUB_PREFIX.  */
-#define GRUB_KERNEL_MACHINE_PREFIX		0x1c
+#define GRUB_KERNEL_MACHINE_PREFIX		0x20
 
 /* End of the data section. */
 #define GRUB_KERNEL_MACHINE_DATA_END		0x50
@@ -47,6 +50,10 @@
 
 #include <grub/types.h>
 
+extern grub_int32_t EXPORT_VAR(grub_kernel_image_size);
+extern grub_int32_t EXPORT_VAR(grub_total_module_size);
+extern grub_int32_t EXPORT_VAR(grub_memdisk_image_size);
+
 /* The DOS partition number of the installed partition.  */
 extern grub_int32_t grub_install_dos_part;
 
diff -x '*.mk' -Nurp grub2/kern/i386/pc/startup.S grub2.memdisk/kern/i386/pc/startup.S
--- grub2/kern/i386/pc/startup.S	2007-10-22 22:14:45.000000000 +0200
+++ grub2.memdisk/kern/i386/pc/startup.S	2007-12-21 17:53:01.000000000 +0100
@@ -95,6 +95,8 @@ VARIABLE(grub_install_dos_part)
 	.long	0xFFFFFFFF
 VARIABLE(grub_install_bsd_part)
 	.long	0xFFFFFFFF
+VARIABLE(grub_memdisk_image_size)
+	.long	0
 VARIABLE(grub_prefix)
 	/* to be filled by grub-mkimage */
 
diff -x '*.mk' -Nurp grub2/util/i386/pc/grub-mkimage.c grub2.memdisk/util/i386/pc/grub-mkimage.c
--- grub2/util/i386/pc/grub-mkimage.c	2007-07-22 01:32:32.000000000 +0200
+++ grub2.memdisk/util/i386/pc/grub-mkimage.c	2007-12-21 21:46:35.000000000 +0100
@@ -25,6 +25,7 @@
 #include <grub/disk.h>
 #include <grub/util/misc.h>
 #include <grub/util/resolve.h>
+#include <grub/misc.h>
 
 #include <stdio.h>
 #include <unistd.h>
@@ -75,11 +76,11 @@ compress_kernel (char *kernel_img, size_
 }
 
 static void
-generate_image (const char *dir, char *prefix, FILE *out, char *mods[])
+generate_image (const char *dir, char *prefix, FILE *out, char *mods[], char *memdisk_path)
 {
   grub_addr_t module_addr = 0;
   char *kernel_img, *boot_img, *core_img;
-  size_t kernel_size, boot_size, total_module_size, core_size;
+  size_t kernel_size, boot_size, total_module_size, core_size, memdisk_size = 0;
   char *kernel_path, *boot_path;
   unsigned num;
   size_t offset;
@@ -98,7 +99,13 @@ generate_image (const char *dir, char *p
 
   grub_util_info ("the total module size is 0x%x", total_module_size);
 
-  kernel_img = xmalloc (kernel_size + total_module_size);
+  if (memdisk_path)
+    {
+      memdisk_size = ALIGN_UP(grub_util_get_image_size (memdisk_path), 512);
+      grub_util_info ("the size of memory disk is 0x%x", memdisk_size);
+    }
+
+  kernel_img = xmalloc (kernel_size + total_module_size + memdisk_size);
   grub_util_load_image (kernel_path, kernel_img);
 
   if (GRUB_KERNEL_MACHINE_PREFIX + strlen (prefix) + 1 > GRUB_KERNEL_MACHINE_DATA_END)
@@ -122,13 +129,19 @@ generate_image (const char *dir, char *p
       header = (struct grub_module_header *) (kernel_img + offset);
       header->offset = grub_cpu_to_le32 (sizeof (*header));
       header->size = grub_cpu_to_le32 (mod_size + sizeof (*header));
-      
-      grub_util_load_image (p->name, kernel_img + offset + sizeof (*header));
+      offset += sizeof (*header);
+
+      grub_util_load_image (p->name, kernel_img + offset);
+      offset += mod_size;
+    }
 
-      offset += sizeof (*header) + mod_size;
+  if (memdisk_path)
+    {
+      grub_util_load_image (memdisk_path, kernel_img + offset);
+      offset += memdisk_size;
     }
 
-  compress_kernel (kernel_img, kernel_size + total_module_size,
+  compress_kernel (kernel_img, kernel_size + total_module_size + memdisk_size,
 		   &core_img, &core_size);
 
   grub_util_info ("the core size is 0x%x", core_size);
@@ -163,6 +176,8 @@ generate_image (const char *dir, char *p
     = grub_cpu_to_le32 (total_module_size);
   *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_KERNEL_IMAGE_SIZE))
     = grub_cpu_to_le32 (kernel_size);
+  *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE))
+    = grub_cpu_to_le32 (memdisk_size);
   *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_COMPRESSED_SIZE))
     = grub_cpu_to_le32 (core_size - GRUB_KERNEL_MACHINE_RAW_SIZE);
   
@@ -186,6 +201,7 @@ static struct option options[] =
   {
     {"directory", required_argument, 0, 'd'},
     {"prefix", required_argument, 0, 'p'},
+    {"memdisk", required_argument, 0, 'm'},
     {"output", required_argument, 0, 'o'},
     {"help", no_argument, 0, 'h'},
     {"version", no_argument, 0, 'V'},
@@ -206,6 +222,7 @@ Make a bootable image of GRUB.\n\
 \n\
   -d, --directory=DIR     use images and modules under DIR [default=%s]\n\
   -p, --prefix=DIR        set grub_prefix directory [default=%s]\n\
+  -m, --memdisk=FILE      embed FILE as a memdisk image\n\
   -o, --output=FILE       output a generated image to FILE [default=stdout]\n\
   -h, --help              display this message and exit\n\
   -V, --version           print version information and exit\n\
@@ -223,13 +240,14 @@ main (int argc, char *argv[])
   char *output = NULL;
   char *dir = NULL;
   char *prefix = NULL;
+  char *memdisk = NULL;
   FILE *fp = stdout;
 
   progname = "grub-mkimage";
   
   while (1)
     {
-      int c = getopt_long (argc, argv, "d:p:o:hVv", options, 0);
+      int c = getopt_long (argc, argv, "d:p:m:o:hVv", options, 0);
 
       if (c == -1)
 	break;
@@ -250,6 +268,13 @@ main (int argc, char *argv[])
 	    dir = xstrdup (optarg);
 	    break;
 
+	  case 'm':
+	    if (memdisk)
+	      free (memdisk);
+
+	    memdisk = xstrdup (optarg);
+	    break;
+
 	  case 'h':
 	    usage (0);
 	    break;
@@ -282,7 +307,7 @@ main (int argc, char *argv[])
 	grub_util_error ("cannot open %s", output);
     }
 
-  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind);
+  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind, memdisk);
 
   fclose (fp);
 

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Bean]

On Dec 22, 2007 4:54 AM, Robert Millan <rmh@aybabtu.com> wrote:
> On Fri, Dec 21, 2007 at 07:04:05PM +0100, Robert Millan wrote:
> > > >
> > > > I'd prefer to support embedding filesystems in core.img instead.  This
> > > > solves the problem for every situation instead of just PC boot media
> > > > (e.g. LinuxBIOS ELF payload image).
> > >
> > > Agreed, I would like to see such feature.  Also for stuff like
> > > diskless boot.
> > >
> > > What I had in mind was the following.  We could perhaps make it
> > > possible to add all files using grub-mkimage, not only modules.  When
> > > you add modules, those will be loaded automatically.  All other files
> > > will be ignored.
> > >
> > > Another filesystem module has to be written.  It scans over these
> > > files and will make them available.  This can be simple and elegant.
> > > I do not like the idea of adding a diskimage or so.  Are you
> > > interested in working on this?
> >
> > Well, I already have an almost-working implementation.  I guess I'll have to
> > defend it then! :-P
> >
> > Really, it's clean and elegant.  Very simple and flexible.  Once I get the
> > address detection right, that is..
>
> Please have a look at the attached patch.  This allows adding an arbitrary
> number of files.  No extra filesystem module is needed, since any of the
> supported filesystems can be used with this feature:
>
>   genisoimage -o filesystem_image -r stuff
>   grub-mkimage -o core.img -m filesystem_image
>

Yes, i think this concept is great, and i just think of an improvement
for the module. we can add support for a raw file system which store
files compactly,  so that very little space is wasted on file system
structure, for example, this is a possible implementation:

OFFSET_TO_NEXT_ITEM
FILE1_NAME_LEN
FILE1_NAME
FILE1_DATA
OFFSET_TO_NEXT_ITEM
FILE2_NAME_LEN
FILE2_NAME
FILE2_DATA

This is useful in situation where the size of core.img is limited, like bootrom.

-- 
Bean

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

On Sat, Dec 22, 2007 at 12:28:26PM +0800, Bean wrote:
> 
> Yes, i think this concept is great, and i just think of an improvement
> for the module.

I don't like the idea of making memdisk filesystem-dependant.  But adding
a more size-oriented filesystem to GRUB would be nice of course.

> we can add support for a raw file system which store
> files compactly,  so that very little space is wasted on file system
> structure, for example, this is a possible implementation:
> 
> OFFSET_TO_NEXT_ITEM
> FILE1_NAME_LEN
> FILE1_NAME
> FILE1_DATA
> OFFSET_TO_NEXT_ITEM
> FILE2_NAME_LEN
> FILE2_NAME
> FILE2_DATA

How about tar?  If we use an existing fs, we get the user tools for free.

> This is useful in situation where the size of core.img is limited, like bootrom.

Btw, note that the filesystem is embedded inside the LZO compressed block,
not just concatenated.  This should get rid of most of that 0x00 padding
in iso9660, I think.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Yoshinori K. Okuji]

On Saturday 22 December 2007 12:24, Robert Millan wrote:
> On Sat, Dec 22, 2007 at 12:28:26PM +0800, Bean wrote:
> > Yes, i think this concept is great, and i just think of an improvement
> > for the module.
>
> I don't like the idea of making memdisk filesystem-dependant.  But adding
> a more size-oriented filesystem to GRUB would be nice of course.
>
> > we can add support for a raw file system which store
> > files compactly,  so that very little space is wasted on file system
> > structure, for example, this is a possible implementation:
> >
> > OFFSET_TO_NEXT_ITEM
> > FILE1_NAME_LEN
> > FILE1_NAME
> > FILE1_DATA
> > OFFSET_TO_NEXT_ITEM
> > FILE2_NAME_LEN
> > FILE2_NAME
> > FILE2_DATA
>
> How about tar?  If we use an existing fs, we get the user tools for free.

How about using cpio, just like initrd? IIRC, cpio is easier to modify later, 
so we have a plenty of room to play with.

Okuji

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

Here's a new patch, with some cleanup.  The main difference is that memdisk.c
doesn't include any arch-specific code.

I've spotted a memory management problem.  The memdisk image, at the location
that it's usually uncompressed, tends to collide with the payload loading
region (grub_os_area_{addr,size}).

I'm not sure what would be a good solution to this.  Perhaps we could copy it
to dynamic memory in grub_memdisk_init() ?  However, if user plays with payload
images without booting them, and then loads memdisk.mod manually, things could
break.

Although, embedding a memdisk image without embedding memdisk.mod is kind of
silly;  perhaps grub-mkimage shouldn't allow it.

What do you think?

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Yoshinori K. Okuji]

On Saturday 22 December 2007 13:18, Robert Millan wrote:
> Here's a new patch, with some cleanup.  The main difference is that
> memdisk.c doesn't include any arch-specific code.

You forgot to attach it. :)

Okuji

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

On Sat, Dec 22, 2007 at 01:24:53PM +0100, Yoshinori K. Okuji wrote:
> On Saturday 22 December 2007 13:18, Robert Millan wrote:
> > Here's a new patch, with some cleanup.  The main difference is that
> > memdisk.c doesn't include any arch-specific code.
> 
> You forgot to attach it. :)

Oops.  Here it is.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

[-- Attachment #2: memdisk.diff --]
[-- Type: text/x-diff, Size: 12837 bytes --]

diff -x '*.mk' -Nurp grub2/conf/i386-pc.rmk grub2.memdisk/conf/i386-pc.rmk
--- grub2/conf/i386-pc.rmk	2007-12-21 12:41:23.000000000 +0100
+++ grub2.memdisk/conf/i386-pc.rmk	2007-12-21 21:47:57.000000000 +0100
@@ -39,7 +39,7 @@ kernel_img_HEADERS = arg.h boot.h cache.
 	env.h err.h file.h fs.h kernel.h loader.h misc.h mm.h net.h parser.h \
 	partition.h pc_partition.h rescue.h symbol.h term.h time.h types.h \
 	machine/biosdisk.h machine/boot.h machine/console.h machine/init.h \
-	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h
+	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h machine/kernel.h
 kernel_img_CFLAGS = $(COMMON_CFLAGS)
 kernel_img_ASFLAGS = $(COMMON_ASFLAGS)
 kernel_img_LDFLAGS = $(COMMON_LDFLAGS) -Wl,-N,-Ttext,8200 $(COMMON_CFLAGS)
@@ -136,7 +136,7 @@ pkgdata_MODULES = biosdisk.mod _chain.mo
 	_multiboot.mod chain.mod multiboot.mod reboot.mod halt.mod	\
 	vbe.mod vbetest.mod vbeinfo.mod video.mod gfxterm.mod \
 	videotest.mod play.mod bitmap.mod tga.mod cpuid.mod serial.mod ata.mod \
-	vga.mod
+	vga.mod memdisk.mod
 
 # For biosdisk.mod.
 biosdisk_mod_SOURCES = disk/i386/pc/biosdisk.c
@@ -263,4 +263,9 @@ vga_mod_SOURCES = term/i386/pc/vga.c
 vga_mod_CFLAGS = $(COMMON_CFLAGS)
 vga_mod_LDFLAGS = $(COMMON_LDFLAGS)
 
+# For memdisk.mod.
+memdisk_mod_SOURCES = disk/memdisk.c
+memdisk_mod_CFLAGS = $(COMMON_CFLAGS)
+memdisk_mod_LDFLAGS = $(COMMON_LDFLAGS)
+
 include $(srcdir)/conf/common.mk
diff -x '*.mk' -Nurp grub2/disk/memdisk.c grub2.memdisk/disk/memdisk.c
--- grub2/disk/memdisk.c	1970-01-01 01:00:00.000000000 +0100
+++ grub2.memdisk/disk/memdisk.c	2007-12-22 13:00:19.000000000 +0100
@@ -0,0 +1,94 @@
+/* memdisk.c - Access embedded memory disk.  */
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2007  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <grub/disk.h>
+#include <grub/dl.h>
+#include <grub/kernel.h>
+#include <grub/misc.h>
+#include <grub/mm.h>
+#include <grub/types.h>
+
+static grub_addr_t memdisk_addr;
+
+static int
+grub_memdisk_iterate (int (*hook) (const char *name))
+{
+  return hook ("memdisk");
+}
+
+static grub_err_t
+grub_memdisk_open (const char *name, grub_disk_t disk)
+{
+  if (grub_strcmp (name, "memdisk"))
+      return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "not a memdisk");
+
+  disk->total_sectors = grub_arch_memdisk_size () / GRUB_DISK_SECTOR_SIZE;
+  disk->id = (int) 'mdsk';
+  disk->has_partitions = 0;
+
+  return GRUB_ERR_NONE;
+}
+
+static void
+grub_memdisk_close (grub_disk_t disk __attribute((unused)))
+{
+}
+
+static grub_err_t
+grub_memdisk_read (grub_disk_t disk __attribute((unused)), grub_disk_addr_t sector,
+		    grub_size_t size, char *buf)
+{
+  grub_memcpy (buf, memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), size << GRUB_DISK_SECTOR_BITS);
+  return 0;
+}
+
+static grub_err_t
+grub_memdisk_write (grub_disk_t disk __attribute((unused)), grub_disk_addr_t sector,
+		     grub_size_t size, const char *buf)
+{
+  grub_memcpy (memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), buf, size << GRUB_DISK_SECTOR_BITS);
+  return 0;
+}
+
+static struct grub_disk_dev grub_memdisk_dev =
+  {
+    .name = "memdisk",
+    .id = GRUB_DISK_DEVICE_MEMDISK_ID,
+    .iterate = grub_memdisk_iterate,
+    .open = grub_memdisk_open,
+    .close = grub_memdisk_close,
+    .read = grub_memdisk_read,
+    .write = grub_memdisk_write,
+    .next = 0
+  };
+
+GRUB_MOD_INIT(memdisk)
+{
+  if (! grub_arch_memdisk_size ())
+    return;
+  memdisk_addr = grub_arch_memdisk_addr ();
+  grub_disk_dev_register (&grub_memdisk_dev);
+}
+
+GRUB_MOD_FINI(memdisk)
+{
+  if (! grub_arch_memdisk_size ())
+    return;
+  grub_disk_dev_unregister (&grub_memdisk_dev);
+}
diff -x '*.mk' -Nurp grub2/include/grub/disk.h grub2.memdisk/include/grub/disk.h
--- grub2/include/grub/disk.h	2007-11-05 17:15:26.000000000 +0100
+++ grub2.memdisk/include/grub/disk.h	2007-12-21 17:47:06.000000000 +0100
@@ -35,7 +35,8 @@ enum grub_disk_dev_id
     GRUB_DISK_DEVICE_RAID_ID,
     GRUB_DISK_DEVICE_LVM_ID,
     GRUB_DISK_DEVICE_HOST_ID,
-    GRUB_DISK_DEVICE_ATA_ID
+    GRUB_DISK_DEVICE_ATA_ID,
+    GRUB_DISK_DEVICE_MEMDISK_ID,
   };
 
 struct grub_disk;
diff -x '*.mk' -Nurp grub2/include/grub/i386/pc/kernel.h grub2.memdisk/include/grub/i386/pc/kernel.h
--- grub2/include/grub/i386/pc/kernel.h	2007-07-22 01:32:23.000000000 +0200
+++ grub2.memdisk/include/grub/i386/pc/kernel.h	2007-12-22 13:10:17.000000000 +0100
@@ -34,8 +34,11 @@
 /* The offset of GRUB_INSTALL_BSD_PART.  */
 #define GRUB_KERNEL_MACHINE_INSTALL_BSD_PART	0x18
 
+/* The offset of GRUB_MEMDISK_IMAGE_SIZE.  */
+#define GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE	0x1c
+
 /* The offset of GRUB_PREFIX.  */
-#define GRUB_KERNEL_MACHINE_PREFIX		0x1c
+#define GRUB_KERNEL_MACHINE_PREFIX		0x20
 
 /* End of the data section. */
 #define GRUB_KERNEL_MACHINE_DATA_END		0x50
@@ -47,12 +50,21 @@
 
 #include <grub/types.h>
 
+/* The size of kernel image.  */
+extern grub_int32_t grub_kernel_image_size;
+
+/* The total size of module images following the kernel.  */
+extern grub_int32_t grub_total_module_size;
+
 /* The DOS partition number of the installed partition.  */
 extern grub_int32_t grub_install_dos_part;
 
 /* The BSD partition number of the installed partition.  */
 extern grub_int32_t grub_install_bsd_part;
 
+/* The size of memory disk image, if present.  */
+extern grub_int32_t grub_memdisk_image_size;
+
 /* The prefix which points to the directory where GRUB modules and its
    configuration file are located.  */
 extern char grub_prefix[];
diff -x '*.mk' -Nurp grub2/include/grub/kernel.h grub2.memdisk/include/grub/kernel.h
--- grub2/include/grub/kernel.h	2007-07-22 01:32:21.000000000 +0200
+++ grub2.memdisk/include/grub/kernel.h	2007-12-22 12:54:18.000000000 +0100
@@ -20,6 +20,7 @@
 #define GRUB_KERNEL_HEADER	1
 
 #include <grub/types.h>
+#include <grub/symbol.h>
 
 /* The module header.  */
 struct grub_module_header
@@ -44,6 +45,8 @@ struct grub_module_info
 };
 
 extern grub_addr_t grub_arch_modules_addr (void);
+extern grub_addr_t EXPORT_FUNC(grub_arch_memdisk_addr) (void);
+extern grub_off_t EXPORT_FUNC(grub_arch_memdisk_size) (void);
 
 /* The start point of the C code.  */
 void grub_main (void);
diff -x '*.mk' -Nurp grub2/kern/i386/pc/init.c grub2.memdisk/kern/i386/pc/init.c
--- grub2/kern/i386/pc/init.c	2007-10-22 22:05:27.000000000 +0200
+++ grub2.memdisk/kern/i386/pc/init.c	2007-12-22 13:07:00.000000000 +0100
@@ -229,7 +229,7 @@ grub_machine_init (void)
       }
     else
       grub_mm_init_region ((void *) mem_regions[i].addr, mem_regions[i].size);
-  
+
   if (! grub_os_area_addr)
     grub_fatal ("no upper memory");
 }
@@ -253,3 +253,17 @@ grub_arch_modules_addr (void)
 {
   return grub_end_addr;
 }
+
+/* Return the start of the memdisk image.  */
+grub_addr_t
+grub_arch_memdisk_addr (void)
+{
+  return 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;
+}
+
+/* Return the size of the memdisk image.  */
+grub_off_t
+grub_arch_memdisk_size (void)
+{
+  return grub_memdisk_image_size;
+}
diff -x '*.mk' -Nurp grub2/kern/i386/pc/startup.S grub2.memdisk/kern/i386/pc/startup.S
--- grub2/kern/i386/pc/startup.S	2007-10-22 22:14:45.000000000 +0200
+++ grub2.memdisk/kern/i386/pc/startup.S	2007-12-21 17:53:01.000000000 +0100
@@ -95,6 +95,8 @@ VARIABLE(grub_install_dos_part)
 	.long	0xFFFFFFFF
 VARIABLE(grub_install_bsd_part)
 	.long	0xFFFFFFFF
+VARIABLE(grub_memdisk_image_size)
+	.long	0
 VARIABLE(grub_prefix)
 	/* to be filled by grub-mkimage */
 
diff -x '*.mk' -Nurp grub2/util/i386/pc/grub-mkimage.c grub2.memdisk/util/i386/pc/grub-mkimage.c
--- grub2/util/i386/pc/grub-mkimage.c	2007-07-22 01:32:32.000000000 +0200
+++ grub2.memdisk/util/i386/pc/grub-mkimage.c	2007-12-21 21:46:35.000000000 +0100
@@ -25,6 +25,7 @@
 #include <grub/disk.h>
 #include <grub/util/misc.h>
 #include <grub/util/resolve.h>
+#include <grub/misc.h>
 
 #include <stdio.h>
 #include <unistd.h>
@@ -75,11 +76,11 @@ compress_kernel (char *kernel_img, size_
 }
 
 static void
-generate_image (const char *dir, char *prefix, FILE *out, char *mods[])
+generate_image (const char *dir, char *prefix, FILE *out, char *mods[], char *memdisk_path)
 {
   grub_addr_t module_addr = 0;
   char *kernel_img, *boot_img, *core_img;
-  size_t kernel_size, boot_size, total_module_size, core_size;
+  size_t kernel_size, boot_size, total_module_size, core_size, memdisk_size = 0;
   char *kernel_path, *boot_path;
   unsigned num;
   size_t offset;
@@ -98,7 +99,13 @@ generate_image (const char *dir, char *p
 
   grub_util_info ("the total module size is 0x%x", total_module_size);
 
-  kernel_img = xmalloc (kernel_size + total_module_size);
+  if (memdisk_path)
+    {
+      memdisk_size = ALIGN_UP(grub_util_get_image_size (memdisk_path), 512);
+      grub_util_info ("the size of memory disk is 0x%x", memdisk_size);
+    }
+
+  kernel_img = xmalloc (kernel_size + total_module_size + memdisk_size);
   grub_util_load_image (kernel_path, kernel_img);
 
   if (GRUB_KERNEL_MACHINE_PREFIX + strlen (prefix) + 1 > GRUB_KERNEL_MACHINE_DATA_END)
@@ -122,13 +129,19 @@ generate_image (const char *dir, char *p
       header = (struct grub_module_header *) (kernel_img + offset);
       header->offset = grub_cpu_to_le32 (sizeof (*header));
       header->size = grub_cpu_to_le32 (mod_size + sizeof (*header));
-      
-      grub_util_load_image (p->name, kernel_img + offset + sizeof (*header));
+      offset += sizeof (*header);
+
+      grub_util_load_image (p->name, kernel_img + offset);
+      offset += mod_size;
+    }
 
-      offset += sizeof (*header) + mod_size;
+  if (memdisk_path)
+    {
+      grub_util_load_image (memdisk_path, kernel_img + offset);
+      offset += memdisk_size;
     }
 
-  compress_kernel (kernel_img, kernel_size + total_module_size,
+  compress_kernel (kernel_img, kernel_size + total_module_size + memdisk_size,
 		   &core_img, &core_size);
 
   grub_util_info ("the core size is 0x%x", core_size);
@@ -163,6 +176,8 @@ generate_image (const char *dir, char *p
     = grub_cpu_to_le32 (total_module_size);
   *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_KERNEL_IMAGE_SIZE))
     = grub_cpu_to_le32 (kernel_size);
+  *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE))
+    = grub_cpu_to_le32 (memdisk_size);
   *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_COMPRESSED_SIZE))
     = grub_cpu_to_le32 (core_size - GRUB_KERNEL_MACHINE_RAW_SIZE);
   
@@ -186,6 +201,7 @@ static struct option options[] =
   {
     {"directory", required_argument, 0, 'd'},
     {"prefix", required_argument, 0, 'p'},
+    {"memdisk", required_argument, 0, 'm'},
     {"output", required_argument, 0, 'o'},
     {"help", no_argument, 0, 'h'},
     {"version", no_argument, 0, 'V'},
@@ -206,6 +222,7 @@ Make a bootable image of GRUB.\n\
 \n\
   -d, --directory=DIR     use images and modules under DIR [default=%s]\n\
   -p, --prefix=DIR        set grub_prefix directory [default=%s]\n\
+  -m, --memdisk=FILE      embed FILE as a memdisk image\n\
   -o, --output=FILE       output a generated image to FILE [default=stdout]\n\
   -h, --help              display this message and exit\n\
   -V, --version           print version information and exit\n\
@@ -223,13 +240,14 @@ main (int argc, char *argv[])
   char *output = NULL;
   char *dir = NULL;
   char *prefix = NULL;
+  char *memdisk = NULL;
   FILE *fp = stdout;
 
   progname = "grub-mkimage";
   
   while (1)
     {
-      int c = getopt_long (argc, argv, "d:p:o:hVv", options, 0);
+      int c = getopt_long (argc, argv, "d:p:m:o:hVv", options, 0);
 
       if (c == -1)
 	break;
@@ -250,6 +268,13 @@ main (int argc, char *argv[])
 	    dir = xstrdup (optarg);
 	    break;
 
+	  case 'm':
+	    if (memdisk)
+	      free (memdisk);
+
+	    memdisk = xstrdup (optarg);
+	    break;
+
 	  case 'h':
 	    usage (0);
 	    break;
@@ -282,7 +307,7 @@ main (int argc, char *argv[])
 	grub_util_error ("cannot open %s", output);
     }
 
-  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind);
+  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind, memdisk);
 
   fclose (fp);
 

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Marco Gerards]

Robert Millan <rmh@aybabtu.com> writes:

> On Sat, Dec 22, 2007 at 01:24:53PM +0100, Yoshinori K. Okuji wrote:
>> On Saturday 22 December 2007 13:18, Robert Millan wrote:
>> > Here's a new patch, with some cleanup.  The main difference is that
>> > memdisk.c doesn't include any arch-specific code.
>> 
>> You forgot to attach it. :)
>
> Oops.  Here it is.

Hopefully this is the newest version?  You forgot the changelog entry.

Some comments below, otherwise it is fine :-)

--
Marco


> diff -x '*.mk' -Nurp grub2/conf/i386-pc.rmk grub2.memdisk/conf/i386-pc.rmk
> --- grub2/conf/i386-pc.rmk	2007-12-21 12:41:23.000000000 +0100
> +++ grub2.memdisk/conf/i386-pc.rmk	2007-12-21 21:47:57.000000000 +0100
> @@ -39,7 +39,7 @@ kernel_img_HEADERS = arg.h boot.h cache.
>  	env.h err.h file.h fs.h kernel.h loader.h misc.h mm.h net.h parser.h \
>  	partition.h pc_partition.h rescue.h symbol.h term.h time.h types.h \
>  	machine/biosdisk.h machine/boot.h machine/console.h machine/init.h \
> -	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h
> +	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h machine/kernel.h
>  kernel_img_CFLAGS = $(COMMON_CFLAGS)
>  kernel_img_ASFLAGS = $(COMMON_ASFLAGS)
>  kernel_img_LDFLAGS = $(COMMON_LDFLAGS) -Wl,-N,-Ttext,8200 $(COMMON_CFLAGS)
> @@ -136,7 +136,7 @@ pkgdata_MODULES = biosdisk.mod _chain.mo
>  	_multiboot.mod chain.mod multiboot.mod reboot.mod halt.mod	\
>  	vbe.mod vbetest.mod vbeinfo.mod video.mod gfxterm.mod \
>  	videotest.mod play.mod bitmap.mod tga.mod cpuid.mod serial.mod ata.mod \
> -	vga.mod
> +	vga.mod memdisk.mod
>  
>  # For biosdisk.mod.
>  biosdisk_mod_SOURCES = disk/i386/pc/biosdisk.c
> @@ -263,4 +263,9 @@ vga_mod_SOURCES = term/i386/pc/vga.c
>  vga_mod_CFLAGS = $(COMMON_CFLAGS)
>  vga_mod_LDFLAGS = $(COMMON_LDFLAGS)
>  
> +# For memdisk.mod.
> +memdisk_mod_SOURCES = disk/memdisk.c
> +memdisk_mod_CFLAGS = $(COMMON_CFLAGS)
> +memdisk_mod_LDFLAGS = $(COMMON_LDFLAGS)
> +
>  include $(srcdir)/conf/common.mk
> diff -x '*.mk' -Nurp grub2/disk/memdisk.c grub2.memdisk/disk/memdisk.c
> --- grub2/disk/memdisk.c	1970-01-01 01:00:00.000000000 +0100
> +++ grub2.memdisk/disk/memdisk.c	2007-12-22 13:00:19.000000000 +0100
> @@ -0,0 +1,94 @@
> +/* memdisk.c - Access embedded memory disk.  */
> +/*
> + *  GRUB  --  GRand Unified Bootloader
> + *  Copyright (C) 2007  Free Software Foundation, Inc.
> + *
> + *  GRUB is free software: you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation, either version 3 of the License, or
> + *  (at your option) any later version.
> + *
> + *  GRUB is distributed in the hope that it will be useful,
> + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + *  GNU General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License
> + *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <grub/disk.h>
> +#include <grub/dl.h>
> +#include <grub/kernel.h>
> +#include <grub/misc.h>
> +#include <grub/mm.h>
> +#include <grub/types.h>
> +
> +static grub_addr_t memdisk_addr;
> +
> +static int
> +grub_memdisk_iterate (int (*hook) (const char *name))
> +{
> +  return hook ("memdisk");
> +}
> +
> +static grub_err_t
> +grub_memdisk_open (const char *name, grub_disk_t disk)
> +{
> +  if (grub_strcmp (name, "memdisk"))
> +      return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "not a memdisk");
> +
> +  disk->total_sectors = grub_arch_memdisk_size () / GRUB_DISK_SECTOR_SIZE;
> +  disk->id = (int) 'mdsk';
> +  disk->has_partitions = 0;
> +
> +  return GRUB_ERR_NONE;
> +}
> +
> +static void
> +grub_memdisk_close (grub_disk_t disk __attribute((unused)))
> +{
> +}
> +
> +static grub_err_t
> +grub_memdisk_read (grub_disk_t disk __attribute((unused)), grub_disk_addr_t sector,
> +		    grub_size_t size, char *buf)
> +{
> +  grub_memcpy (buf, memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), size << GRUB_DISK_SECTOR_BITS);
> +  return 0;
> +}
> +
> +static grub_err_t
> +grub_memdisk_write (grub_disk_t disk __attribute((unused)), grub_disk_addr_t sector,
> +		     grub_size_t size, const char *buf)
> +{
> +  grub_memcpy (memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), buf, size << GRUB_DISK_SECTOR_BITS);
> +  return 0;
> +}
> +
> +static struct grub_disk_dev grub_memdisk_dev =
> +  {
> +    .name = "memdisk",
> +    .id = GRUB_DISK_DEVICE_MEMDISK_ID,
> +    .iterate = grub_memdisk_iterate,
> +    .open = grub_memdisk_open,
> +    .close = grub_memdisk_close,
> +    .read = grub_memdisk_read,
> +    .write = grub_memdisk_write,
> +    .next = 0
> +  };
> +
> +GRUB_MOD_INIT(memdisk)
> +{
> +  if (! grub_arch_memdisk_size ())
> +    return;
> +  memdisk_addr = grub_arch_memdisk_addr ();
> +  grub_disk_dev_register (&grub_memdisk_dev);
> +}
> +
> +GRUB_MOD_FINI(memdisk)
> +{
> +  if (! grub_arch_memdisk_size ())
> +    return;
> +  grub_disk_dev_unregister (&grub_memdisk_dev);
> +}
> diff -x '*.mk' -Nurp grub2/include/grub/disk.h grub2.memdisk/include/grub/disk.h
> --- grub2/include/grub/disk.h	2007-11-05 17:15:26.000000000 +0100
> +++ grub2.memdisk/include/grub/disk.h	2007-12-21 17:47:06.000000000 +0100
> @@ -35,7 +35,8 @@ enum grub_disk_dev_id
>      GRUB_DISK_DEVICE_RAID_ID,
>      GRUB_DISK_DEVICE_LVM_ID,
>      GRUB_DISK_DEVICE_HOST_ID,
> -    GRUB_DISK_DEVICE_ATA_ID
> +    GRUB_DISK_DEVICE_ATA_ID,
> +    GRUB_DISK_DEVICE_MEMDISK_ID,
>    };
>  
>  struct grub_disk;
> diff -x '*.mk' -Nurp grub2/include/grub/i386/pc/kernel.h grub2.memdisk/include/grub/i386/pc/kernel.h
> --- grub2/include/grub/i386/pc/kernel.h	2007-07-22 01:32:23.000000000 +0200
> +++ grub2.memdisk/include/grub/i386/pc/kernel.h	2007-12-22 13:10:17.000000000 +0100
> @@ -34,8 +34,11 @@
>  /* The offset of GRUB_INSTALL_BSD_PART.  */
>  #define GRUB_KERNEL_MACHINE_INSTALL_BSD_PART	0x18
>  
> +/* The offset of GRUB_MEMDISK_IMAGE_SIZE.  */
> +#define GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE	0x1c
> +
>  /* The offset of GRUB_PREFIX.  */
> -#define GRUB_KERNEL_MACHINE_PREFIX		0x1c
> +#define GRUB_KERNEL_MACHINE_PREFIX		0x20
>  
>  /* End of the data section. */
>  #define GRUB_KERNEL_MACHINE_DATA_END		0x50
> @@ -47,12 +50,21 @@
>  
>  #include <grub/types.h>
>  
> +/* The size of kernel image.  */
> +extern grub_int32_t grub_kernel_image_size;
> +
> +/* The total size of module images following the kernel.  */
> +extern grub_int32_t grub_total_module_size;
> +
>  /* The DOS partition number of the installed partition.  */
>  extern grub_int32_t grub_install_dos_part;
>  
>  /* The BSD partition number of the installed partition.  */
>  extern grub_int32_t grub_install_bsd_part;
>  
> +/* The size of memory disk image, if present.  */
> +extern grub_int32_t grub_memdisk_image_size;
> +
>  /* The prefix which points to the directory where GRUB modules and its
>     configuration file are located.  */
>  extern char grub_prefix[];
> diff -x '*.mk' -Nurp grub2/include/grub/kernel.h grub2.memdisk/include/grub/kernel.h
> --- grub2/include/grub/kernel.h	2007-07-22 01:32:21.000000000 +0200
> +++ grub2.memdisk/include/grub/kernel.h	2007-12-22 12:54:18.000000000 +0100
> @@ -20,6 +20,7 @@
>  #define GRUB_KERNEL_HEADER	1
>  
>  #include <grub/types.h>
> +#include <grub/symbol.h>
>  
>  /* The module header.  */
>  struct grub_module_header
> @@ -44,6 +45,8 @@ struct grub_module_info
>  };
>  
>  extern grub_addr_t grub_arch_modules_addr (void);
> +extern grub_addr_t EXPORT_FUNC(grub_arch_memdisk_addr) (void);
> +extern grub_off_t EXPORT_FUNC(grub_arch_memdisk_size) (void);
>  
>  /* The start point of the C code.  */
>  void grub_main (void);
> diff -x '*.mk' -Nurp grub2/kern/i386/pc/init.c grub2.memdisk/kern/i386/pc/init.c
> --- grub2/kern/i386/pc/init.c	2007-10-22 22:05:27.000000000 +0200
> +++ grub2.memdisk/kern/i386/pc/init.c	2007-12-22 13:07:00.000000000 +0100
> @@ -229,7 +229,7 @@ grub_machine_init (void)
>        }
>      else
>        grub_mm_init_region ((void *) mem_regions[i].addr, mem_regions[i].size);
> -  
> +

Please be careful with whitespaces...


>    if (! grub_os_area_addr)
>      grub_fatal ("no upper memory");
>  }
> @@ -253,3 +253,17 @@ grub_arch_modules_addr (void)
>  {
>    return grub_end_addr;
>  }
> +
> +/* Return the start of the memdisk image.  */
> +grub_addr_t
> +grub_arch_memdisk_addr (void)
> +{
> +  return 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;

This line seems way too long.

Isn't there some macro for 0x100000?

> +}
> +
> +/* Return the size of the memdisk image.  */
> +grub_off_t
> +grub_arch_memdisk_size (void)
> +{
> +  return grub_memdisk_image_size;
> +}
> diff -x '*.mk' -Nurp grub2/kern/i386/pc/startup.S grub2.memdisk/kern/i386/pc/startup.S
> --- grub2/kern/i386/pc/startup.S	2007-10-22 22:14:45.000000000 +0200
> +++ grub2.memdisk/kern/i386/pc/startup.S	2007-12-21 17:53:01.000000000 +0100
> @@ -95,6 +95,8 @@ VARIABLE(grub_install_dos_part)
>  	.long	0xFFFFFFFF
>  VARIABLE(grub_install_bsd_part)
>  	.long	0xFFFFFFFF
> +VARIABLE(grub_memdisk_image_size)
> +	.long	0
>  VARIABLE(grub_prefix)
>  	/* to be filled by grub-mkimage */
>  
> diff -x '*.mk' -Nurp grub2/util/i386/pc/grub-mkimage.c grub2.memdisk/util/i386/pc/grub-mkimage.c
> --- grub2/util/i386/pc/grub-mkimage.c	2007-07-22 01:32:32.000000000 +0200
> +++ grub2.memdisk/util/i386/pc/grub-mkimage.c	2007-12-21 21:46:35.000000000 +0100
> @@ -25,6 +25,7 @@
>  #include <grub/disk.h>
>  #include <grub/util/misc.h>
>  #include <grub/util/resolve.h>
> +#include <grub/misc.h>
>  
>  #include <stdio.h>
>  #include <unistd.h>
> @@ -75,11 +76,11 @@ compress_kernel (char *kernel_img, size_
>  }
>  
>  static void
> -generate_image (const char *dir, char *prefix, FILE *out, char *mods[])
> +generate_image (const char *dir, char *prefix, FILE *out, char *mods[], char *memdisk_path)
>  {
>    grub_addr_t module_addr = 0;
>    char *kernel_img, *boot_img, *core_img;
> -  size_t kernel_size, boot_size, total_module_size, core_size;
> +  size_t kernel_size, boot_size, total_module_size, core_size, memdisk_size = 0;
>    char *kernel_path, *boot_path;
>    unsigned num;
>    size_t offset;
> @@ -98,7 +99,13 @@ generate_image (const char *dir, char *p
>  
>    grub_util_info ("the total module size is 0x%x", total_module_size);
>  
> -  kernel_img = xmalloc (kernel_size + total_module_size);
> +  if (memdisk_path)
> +    {
> +      memdisk_size = ALIGN_UP(grub_util_get_image_size (memdisk_path), 512);
> +      grub_util_info ("the size of memory disk is 0x%x", memdisk_size);
> +    }
> +
> +  kernel_img = xmalloc (kernel_size + total_module_size + memdisk_size);
>    grub_util_load_image (kernel_path, kernel_img);
>  
>    if (GRUB_KERNEL_MACHINE_PREFIX + strlen (prefix) + 1 > GRUB_KERNEL_MACHINE_DATA_END)
> @@ -122,13 +129,19 @@ generate_image (const char *dir, char *p
>        header = (struct grub_module_header *) (kernel_img + offset);
>        header->offset = grub_cpu_to_le32 (sizeof (*header));
>        header->size = grub_cpu_to_le32 (mod_size + sizeof (*header));
> -      
> -      grub_util_load_image (p->name, kernel_img + offset + sizeof (*header));
> +      offset += sizeof (*header);
> +
> +      grub_util_load_image (p->name, kernel_img + offset);
> +      offset += mod_size;
> +    }
>  
> -      offset += sizeof (*header) + mod_size;
> +  if (memdisk_path)
> +    {
> +      grub_util_load_image (memdisk_path, kernel_img + offset);
> +      offset += memdisk_size;
>      }
>  
> -  compress_kernel (kernel_img, kernel_size + total_module_size,
> +  compress_kernel (kernel_img, kernel_size + total_module_size + memdisk_size,
>  		   &core_img, &core_size);
>  
>    grub_util_info ("the core size is 0x%x", core_size);
> @@ -163,6 +176,8 @@ generate_image (const char *dir, char *p
>      = grub_cpu_to_le32 (total_module_size);
>    *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_KERNEL_IMAGE_SIZE))
>      = grub_cpu_to_le32 (kernel_size);
> +  *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE))
> +    = grub_cpu_to_le32 (memdisk_size);
>    *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_COMPRESSED_SIZE))
>      = grub_cpu_to_le32 (core_size - GRUB_KERNEL_MACHINE_RAW_SIZE);
>    
> @@ -186,6 +201,7 @@ static struct option options[] =
>    {
>      {"directory", required_argument, 0, 'd'},
>      {"prefix", required_argument, 0, 'p'},
> +    {"memdisk", required_argument, 0, 'm'},
>      {"output", required_argument, 0, 'o'},
>      {"help", no_argument, 0, 'h'},
>      {"version", no_argument, 0, 'V'},
> @@ -206,6 +222,7 @@ Make a bootable image of GRUB.\n\
>  \n\
>    -d, --directory=DIR     use images and modules under DIR [default=%s]\n\
>    -p, --prefix=DIR        set grub_prefix directory [default=%s]\n\
> +  -m, --memdisk=FILE      embed FILE as a memdisk image\n\
>    -o, --output=FILE       output a generated image to FILE [default=stdout]\n\
>    -h, --help              display this message and exit\n\
>    -V, --version           print version information and exit\n\
> @@ -223,13 +240,14 @@ main (int argc, char *argv[])
>    char *output = NULL;
>    char *dir = NULL;
>    char *prefix = NULL;
> +  char *memdisk = NULL;
>    FILE *fp = stdout;
>  
>    progname = "grub-mkimage";
>    
>    while (1)
>      {
> -      int c = getopt_long (argc, argv, "d:p:o:hVv", options, 0);
> +      int c = getopt_long (argc, argv, "d:p:m:o:hVv", options, 0);
>  
>        if (c == -1)
>  	break;
> @@ -250,6 +268,13 @@ main (int argc, char *argv[])
>  	    dir = xstrdup (optarg);
>  	    break;
>  
> +	  case 'm':
> +	    if (memdisk)
> +	      free (memdisk);
> +
> +	    memdisk = xstrdup (optarg);
> +	    break;
> +
>  	  case 'h':
>  	    usage (0);
>  	    break;
> @@ -282,7 +307,7 @@ main (int argc, char *argv[])
>  	grub_util_error ("cannot open %s", output);
>      }
>  
> -  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind);
> +  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind, memdisk);
>  
>    fclose (fp);
>  
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

On Wed, Jan 16, 2008 at 09:10:30PM +0100, Marco Gerards wrote:
> Robert Millan <rmh@aybabtu.com> writes:
> 
> > On Sat, Dec 22, 2007 at 01:24:53PM +0100, Yoshinori K. Okuji wrote:
> >> On Saturday 22 December 2007 13:18, Robert Millan wrote:
> >> > Here's a new patch, with some cleanup.  The main difference is that
> >> > memdisk.c doesn't include any arch-specific code.
> >> 
> >> You forgot to attach it. :)
> >
> > Oops.  Here it is.
> 
> Hopefully this is the newest version?  You forgot the changelog entry.
> 
> Some comments below, otherwise it is fine :-)

Ok, fixed and committed!

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Bean]

On Dec 22, 2007 8:18 PM, Robert Millan <rmh@aybabtu.com> wrote:
>
> Here's a new patch, with some cleanup.  The main difference is that memdisk.c
> doesn't include any arch-specific code.
>
> I've spotted a memory management problem.  The memdisk image, at the location
> that it's usually uncompressed, tends to collide with the payload loading
> region (grub_os_area_{addr,size}).
>
> I'm not sure what would be a good solution to this.  Perhaps we could copy it
> to dynamic memory in grub_memdisk_init() ?  However, if user plays with payload
> images without booting them, and then loads memdisk.mod manually, things could
> break.
>
> Although, embedding a memdisk image without embedding memdisk.mod is kind of
> silly;  perhaps grub-mkimage shouldn't allow it.
>
> What do you think?

 I think you can copy the memdisk to the end of memory, and in
grub_machine_init, make sure to exclude this area:

	grub_mm_init_region ((void *) (grub_os_area_addr + grub_os_area_size),
			     quarter);

->

	grub_mm_init_region ((void *) (grub_os_area_addr + grub_os_area_size),
			     quarter - memdisk_size);

I also recommend to add a variable grub_memdisk_image_base in
startup.S, normally, this value will be zero, but loader like
lnxboot.S can set it to the address of the external initrd image.

-- 
Bean

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

On Mon, Dec 24, 2007 at 04:29:19PM +0800, Bean wrote:
> On Dec 22, 2007 8:18 PM, Robert Millan <rmh@aybabtu.com> wrote:
> >
> > Here's a new patch, with some cleanup.  The main difference is that memdisk.c
> > doesn't include any arch-specific code.
> >
> > I've spotted a memory management problem.  The memdisk image, at the location
> > that it's usually uncompressed, tends to collide with the payload loading
> > region (grub_os_area_{addr,size}).
> >
> > I'm not sure what would be a good solution to this.  Perhaps we could copy it
> > to dynamic memory in grub_memdisk_init() ?  However, if user plays with payload
> > images without booting them, and then loads memdisk.mod manually, things could
> > break.
> >
> > Although, embedding a memdisk image without embedding memdisk.mod is kind of
> > silly;  perhaps grub-mkimage shouldn't allow it.
> >
> > What do you think?
> 
>  I think you can copy the memdisk to the end of memory, and in
> grub_machine_init, make sure to exclude this area:
> 
> 	grub_mm_init_region ((void *) (grub_os_area_addr + grub_os_area_size),
> 			     quarter);
> 
> ->
> 
> 	grub_mm_init_region ((void *) (grub_os_area_addr + grub_os_area_size),
> 			     quarter - memdisk_size);

I was thinking in moving it to malloc'ed memory, which has the advantage that
you no longer need to remember a particular region is hardcoded for memdisk
(also, it hinders portability on platforms where top of memory is reserved).

but this doesn't solve the problem about its original region being corrupted
(by payload loaders) before we have a chance to copy it.

> I also recommend to add a variable grub_memdisk_image_base in
> startup.S, normally, this value will be zero,

That's in my patch already.  Or did you mean something else?

> but loader like
> lnxboot.S can set it to the address of the external initrd image.

Sounds consistent with what lnxboot does, although I'm not sure how useful
it'd be.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Bean]

On Dec 24, 2007 9:09 PM, Robert Millan <rmh@aybabtu.com> wrote:
> I was thinking in moving it to malloc'ed memory, which has the advantage that
> you no longer need to remember a particular region is hardcoded for memdisk
> (also, it hinders portability on platforms where top of memory is reserved).
>
> but this doesn't solve the problem about its original region being corrupted
> (by payload loaders) before we have a chance to copy it.

In that case, you can copy the memdisk back to conventional memory,
alongside with modules.
In startup.S:

codestart:
...
	/* copy modules before cleaning out the bss */
	movl	EXT_C(grub_total_module_size), %ecx
	movl	EXT_C(grub_kernel_image_size), %esi
	addl	%ecx, %esi

->
	movl	EXT_C(grub_total_module_size), %ecx
+      addl    EXT_C(grub_memdisk_image_size), %ecx
	movl	EXT_C(grub_kernel_image_size), %esi
	addl	%ecx, %esi

>
> > I also recommend to add a variable grub_memdisk_image_base in
> > startup.S, normally, this value will be zero,
>
> That's in my patch already.  Or did you mean something else?

But i only see the grub_memdisk_image_size, not the base address.


-- 
Bean

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

On Mon, Dec 24, 2007 at 09:27:05PM +0800, Bean wrote:
> On Dec 24, 2007 9:09 PM, Robert Millan <rmh@aybabtu.com> wrote:
> > I was thinking in moving it to malloc'ed memory, which has the advantage that
> > you no longer need to remember a particular region is hardcoded for memdisk
> > (also, it hinders portability on platforms where top of memory is reserved).
> >
> > but this doesn't solve the problem about its original region being corrupted
> > (by payload loaders) before we have a chance to copy it.
> 
> In that case, you can copy the memdisk back to conventional memory,
> alongside with modules.
> In startup.S:
> 
> codestart:
> ...
> 	/* copy modules before cleaning out the bss */
> 	movl	EXT_C(grub_total_module_size), %ecx
> 	movl	EXT_C(grub_kernel_image_size), %esi
> 	addl	%ecx, %esi
> 
> ->
> 	movl	EXT_C(grub_total_module_size), %ecx
> +      addl    EXT_C(grub_memdisk_image_size), %ecx
> 	movl	EXT_C(grub_kernel_image_size), %esi
> 	addl	%ecx, %esi

Doesn't this add a restriction on the maximum usable size, when it collides
with the assorted amenities in the 640 kiB / 1024 kiB area ?

> > > I also recommend to add a variable grub_memdisk_image_base in
> > > startup.S, normally, this value will be zero,
> >
> > That's in my patch already.  Or did you mean something else?
> 
> But i only see the grub_memdisk_image_size, not the base address.

Ah right;  well it's still there, just not directly:

+  return 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;

those two variables are from startup.S as well, IIRC.  Although it might make
sense to move them as a precalculated address.  The code to compute it probably
takes more kernel space than the address itself..

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Bean]

On Dec 25, 2007 12:24 AM, Robert Millan <rmh@aybabtu.com> wrote:
> Doesn't this add a restriction on the maximum usable size, when it collides
> with the assorted amenities in the 640 kiB / 1024 kiB area ?

yes, that's true. but i think the memdisk is not corrupted before
module initialization, so you can malloc and copy it safely in
grub_main:

void
grub_main (void)
{
  /* First of all, initialize the machine.  */
  grub_machine_init ();

  // malloc and copy memdisk
...

  /* Load pre-loaded modules and free the space.  */
  grub_register_exported_symbols ();
  grub_load_modules ();

 ..
}

>
> > > > I also recommend to add a variable grub_memdisk_image_base in
> > > > startup.S, normally, this value will be zero,
> > >
> > > That's in my patch already.  Or did you mean something else?
> >
> > But i only see the grub_memdisk_image_size, not the base address.
>
> Ah right;  well it's still there, just not directly:
>
> +  return 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;
>
> those two variables are from startup.S as well, IIRC.  Although it might make
> sense to move them as a precalculated address.  The code to compute it probably
> takes more kernel space than the address itself..

What i mean is that you can add an variable grub_total_module_addr, if
grub_total_module_addr=0, calculate the addr using the above fomula,
and if it's not, use it directly as the base address of memdisk. The
reason for this extra variable is that you can set it in pre loader,
for example,

kernel grub2.bin
initrd memdisk

lnxboot can extract the address of initrd image, and set
grub_total_module_addr before jumping to core.img.

-- 
Bean

Re: embedding filesystems in core image (Re: moving ata initialisation to a command)

[Robert Millan]

Hi,

Marco, do you have any comments on this patch?  You said you didn't like this
approach, but it looks good to me, and to others as well.  After having a look,
do you still have any objection ?

On Fri, Dec 21, 2007 at 09:54:31PM +0100, Robert Millan wrote:

> diff -x '*.mk' -Nurp grub2/conf/i386-pc.rmk grub2.memdisk/conf/i386-pc.rmk
> --- grub2/conf/i386-pc.rmk	2007-12-21 12:41:23.000000000 +0100
> +++ grub2.memdisk/conf/i386-pc.rmk	2007-12-21 21:47:57.000000000 +0100
> @@ -39,7 +39,7 @@ kernel_img_HEADERS = arg.h boot.h cache.
>  	env.h err.h file.h fs.h kernel.h loader.h misc.h mm.h net.h parser.h \
>  	partition.h pc_partition.h rescue.h symbol.h term.h time.h types.h \
>  	machine/biosdisk.h machine/boot.h machine/console.h machine/init.h \
> -	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h
> +	machine/memory.h machine/loader.h machine/vga.h machine/vbe.h machine/kernel.h
>  kernel_img_CFLAGS = $(COMMON_CFLAGS)
>  kernel_img_ASFLAGS = $(COMMON_ASFLAGS)
>  kernel_img_LDFLAGS = $(COMMON_LDFLAGS) -Wl,-N,-Ttext,8200 $(COMMON_CFLAGS)
> @@ -136,7 +136,7 @@ pkgdata_MODULES = biosdisk.mod _chain.mo
>  	_multiboot.mod chain.mod multiboot.mod reboot.mod halt.mod	\
>  	vbe.mod vbetest.mod vbeinfo.mod video.mod gfxterm.mod \
>  	videotest.mod play.mod bitmap.mod tga.mod cpuid.mod serial.mod ata.mod \
> -	vga.mod
> +	vga.mod memdisk.mod
>  
>  # For biosdisk.mod.
>  biosdisk_mod_SOURCES = disk/i386/pc/biosdisk.c
> @@ -263,4 +263,9 @@ vga_mod_SOURCES = term/i386/pc/vga.c
>  vga_mod_CFLAGS = $(COMMON_CFLAGS)
>  vga_mod_LDFLAGS = $(COMMON_LDFLAGS)
>  
> +# For memdisk.mod.
> +memdisk_mod_SOURCES = disk/memdisk.c
> +memdisk_mod_CFLAGS = $(COMMON_CFLAGS)
> +memdisk_mod_LDFLAGS = $(COMMON_LDFLAGS)
> +
>  include $(srcdir)/conf/common.mk
> diff -x '*.mk' -Nurp grub2/disk/memdisk.c grub2.memdisk/disk/memdisk.c
> --- grub2/disk/memdisk.c	1970-01-01 01:00:00.000000000 +0100
> +++ grub2.memdisk/disk/memdisk.c	2007-12-21 21:43:25.000000000 +0100
> @@ -0,0 +1,92 @@
> +/* memdisk.c - Access embedded memory disk.  */
> +/*
> + *  GRUB  --  GRand Unified Bootloader
> + *  Copyright (C) 2007  Free Software Foundation, Inc.
> + *
> + *  GRUB is free software: you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation, either version 3 of the License, or
> + *  (at your option) any later version.
> + *
> + *  GRUB is distributed in the hope that it will be useful,
> + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + *  GNU General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License
> + *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <grub/disk.h>
> +#include <grub/dl.h>
> +#include <grub/machine/kernel.h>
> +#include <grub/misc.h>
> +#include <grub/mm.h>
> +#include <grub/types.h>
> +
> +static grub_addr_t memdisk_addr;
> +
> +static int
> +grub_memdisk_iterate (int (*hook) (const char *name))
> +{
> +  return hook ("memdisk");
> +}
> +
> +static grub_err_t
> +grub_memdisk_open (const char *name, grub_disk_t disk)
> +{
> +  if (grub_strcmp (name, "memdisk"))
> +      return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "not a memdisk");
> +
> +  disk->total_sectors = grub_memdisk_image_size / GRUB_DISK_SECTOR_SIZE;
> +  disk->id = (int) 'mdsk';
> +
> +  disk->has_partitions = 0;
> +  disk->data = NULL;
> +
> +  return GRUB_ERR_NONE;
> +}
> +
> +static void
> +grub_memdisk_close (grub_disk_t disk __attribute((unused)))
> +{
> +}
> +
> +static grub_err_t
> +grub_memdisk_read (grub_disk_t disk, grub_disk_addr_t sector,
> +		    grub_size_t size, char *buf)
> +{
> +  grub_memcpy (buf, memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), size << GRUB_DISK_SECTOR_BITS);
> +  return 0;
> +}
> +
> +static grub_err_t
> +grub_memdisk_write (grub_disk_t disk, grub_disk_addr_t sector,
> +		     grub_size_t size, const char *buf)
> +{
> +  grub_memcpy (memdisk_addr + (sector << GRUB_DISK_SECTOR_BITS), buf, size << GRUB_DISK_SECTOR_BITS);
> +  return 0;
> +}
> +
> +static struct grub_disk_dev grub_memdisk_dev =
> +  {
> +    .name = "memdisk",
> +    .id = GRUB_DISK_DEVICE_MEMDISK_ID,
> +    .iterate = grub_memdisk_iterate,
> +    .open = grub_memdisk_open,
> +    .close = grub_memdisk_close,
> +    .read = grub_memdisk_read,
> +    .write = grub_memdisk_write,
> +    .next = 0
> +  };
> +
> +GRUB_MOD_INIT(memdisk)
> +{
> +  memdisk_addr = 0x100000 + (grub_kernel_image_size - GRUB_KERNEL_MACHINE_RAW_SIZE) + grub_total_module_size;
> +  grub_disk_dev_register (&grub_memdisk_dev);
> +}
> +
> +GRUB_MOD_FINI(memdisk)
> +{
> +  grub_disk_dev_unregister (&grub_memdisk_dev);
> +}
> diff -x '*.mk' -Nurp grub2/include/grub/disk.h grub2.memdisk/include/grub/disk.h
> --- grub2/include/grub/disk.h	2007-11-05 17:15:26.000000000 +0100
> +++ grub2.memdisk/include/grub/disk.h	2007-12-21 17:47:06.000000000 +0100
> @@ -35,7 +35,8 @@ enum grub_disk_dev_id
>      GRUB_DISK_DEVICE_RAID_ID,
>      GRUB_DISK_DEVICE_LVM_ID,
>      GRUB_DISK_DEVICE_HOST_ID,
> -    GRUB_DISK_DEVICE_ATA_ID
> +    GRUB_DISK_DEVICE_ATA_ID,
> +    GRUB_DISK_DEVICE_MEMDISK_ID,
>    };
>  
>  struct grub_disk;
> diff -x '*.mk' -Nurp grub2/include/grub/i386/pc/kernel.h grub2.memdisk/include/grub/i386/pc/kernel.h
> --- grub2/include/grub/i386/pc/kernel.h	2007-07-22 01:32:23.000000000 +0200
> +++ grub2.memdisk/include/grub/i386/pc/kernel.h	2007-12-21 18:22:26.000000000 +0100
> @@ -34,8 +34,11 @@
>  /* The offset of GRUB_INSTALL_BSD_PART.  */
>  #define GRUB_KERNEL_MACHINE_INSTALL_BSD_PART	0x18
>  
> +/* The offset of GRUB_MEMDISK_IMAGE_SIZE.  */
> +#define GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE	0x1c
> +
>  /* The offset of GRUB_PREFIX.  */
> -#define GRUB_KERNEL_MACHINE_PREFIX		0x1c
> +#define GRUB_KERNEL_MACHINE_PREFIX		0x20
>  
>  /* End of the data section. */
>  #define GRUB_KERNEL_MACHINE_DATA_END		0x50
> @@ -47,6 +50,10 @@
>  
>  #include <grub/types.h>
>  
> +extern grub_int32_t EXPORT_VAR(grub_kernel_image_size);
> +extern grub_int32_t EXPORT_VAR(grub_total_module_size);
> +extern grub_int32_t EXPORT_VAR(grub_memdisk_image_size);
> +
>  /* The DOS partition number of the installed partition.  */
>  extern grub_int32_t grub_install_dos_part;
>  
> diff -x '*.mk' -Nurp grub2/kern/i386/pc/startup.S grub2.memdisk/kern/i386/pc/startup.S
> --- grub2/kern/i386/pc/startup.S	2007-10-22 22:14:45.000000000 +0200
> +++ grub2.memdisk/kern/i386/pc/startup.S	2007-12-21 17:53:01.000000000 +0100
> @@ -95,6 +95,8 @@ VARIABLE(grub_install_dos_part)
>  	.long	0xFFFFFFFF
>  VARIABLE(grub_install_bsd_part)
>  	.long	0xFFFFFFFF
> +VARIABLE(grub_memdisk_image_size)
> +	.long	0
>  VARIABLE(grub_prefix)
>  	/* to be filled by grub-mkimage */
>  
> diff -x '*.mk' -Nurp grub2/util/i386/pc/grub-mkimage.c grub2.memdisk/util/i386/pc/grub-mkimage.c
> --- grub2/util/i386/pc/grub-mkimage.c	2007-07-22 01:32:32.000000000 +0200
> +++ grub2.memdisk/util/i386/pc/grub-mkimage.c	2007-12-21 21:46:35.000000000 +0100
> @@ -25,6 +25,7 @@
>  #include <grub/disk.h>
>  #include <grub/util/misc.h>
>  #include <grub/util/resolve.h>
> +#include <grub/misc.h>
>  
>  #include <stdio.h>
>  #include <unistd.h>
> @@ -75,11 +76,11 @@ compress_kernel (char *kernel_img, size_
>  }
>  
>  static void
> -generate_image (const char *dir, char *prefix, FILE *out, char *mods[])
> +generate_image (const char *dir, char *prefix, FILE *out, char *mods[], char *memdisk_path)
>  {
>    grub_addr_t module_addr = 0;
>    char *kernel_img, *boot_img, *core_img;
> -  size_t kernel_size, boot_size, total_module_size, core_size;
> +  size_t kernel_size, boot_size, total_module_size, core_size, memdisk_size = 0;
>    char *kernel_path, *boot_path;
>    unsigned num;
>    size_t offset;
> @@ -98,7 +99,13 @@ generate_image (const char *dir, char *p
>  
>    grub_util_info ("the total module size is 0x%x", total_module_size);
>  
> -  kernel_img = xmalloc (kernel_size + total_module_size);
> +  if (memdisk_path)
> +    {
> +      memdisk_size = ALIGN_UP(grub_util_get_image_size (memdisk_path), 512);
> +      grub_util_info ("the size of memory disk is 0x%x", memdisk_size);
> +    }
> +
> +  kernel_img = xmalloc (kernel_size + total_module_size + memdisk_size);
>    grub_util_load_image (kernel_path, kernel_img);
>  
>    if (GRUB_KERNEL_MACHINE_PREFIX + strlen (prefix) + 1 > GRUB_KERNEL_MACHINE_DATA_END)
> @@ -122,13 +129,19 @@ generate_image (const char *dir, char *p
>        header = (struct grub_module_header *) (kernel_img + offset);
>        header->offset = grub_cpu_to_le32 (sizeof (*header));
>        header->size = grub_cpu_to_le32 (mod_size + sizeof (*header));
> -      
> -      grub_util_load_image (p->name, kernel_img + offset + sizeof (*header));
> +      offset += sizeof (*header);
> +
> +      grub_util_load_image (p->name, kernel_img + offset);
> +      offset += mod_size;
> +    }
>  
> -      offset += sizeof (*header) + mod_size;
> +  if (memdisk_path)
> +    {
> +      grub_util_load_image (memdisk_path, kernel_img + offset);
> +      offset += memdisk_size;
>      }
>  
> -  compress_kernel (kernel_img, kernel_size + total_module_size,
> +  compress_kernel (kernel_img, kernel_size + total_module_size + memdisk_size,
>  		   &core_img, &core_size);
>  
>    grub_util_info ("the core size is 0x%x", core_size);
> @@ -163,6 +176,8 @@ generate_image (const char *dir, char *p
>      = grub_cpu_to_le32 (total_module_size);
>    *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_KERNEL_IMAGE_SIZE))
>      = grub_cpu_to_le32 (kernel_size);
> +  *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_MEMDISK_IMAGE_SIZE))
> +    = grub_cpu_to_le32 (memdisk_size);
>    *((grub_uint32_t *) (core_img + GRUB_KERNEL_MACHINE_COMPRESSED_SIZE))
>      = grub_cpu_to_le32 (core_size - GRUB_KERNEL_MACHINE_RAW_SIZE);
>    
> @@ -186,6 +201,7 @@ static struct option options[] =
>    {
>      {"directory", required_argument, 0, 'd'},
>      {"prefix", required_argument, 0, 'p'},
> +    {"memdisk", required_argument, 0, 'm'},
>      {"output", required_argument, 0, 'o'},
>      {"help", no_argument, 0, 'h'},
>      {"version", no_argument, 0, 'V'},
> @@ -206,6 +222,7 @@ Make a bootable image of GRUB.\n\
>  \n\
>    -d, --directory=DIR     use images and modules under DIR [default=%s]\n\
>    -p, --prefix=DIR        set grub_prefix directory [default=%s]\n\
> +  -m, --memdisk=FILE      embed FILE as a memdisk image\n\
>    -o, --output=FILE       output a generated image to FILE [default=stdout]\n\
>    -h, --help              display this message and exit\n\
>    -V, --version           print version information and exit\n\
> @@ -223,13 +240,14 @@ main (int argc, char *argv[])
>    char *output = NULL;
>    char *dir = NULL;
>    char *prefix = NULL;
> +  char *memdisk = NULL;
>    FILE *fp = stdout;
>  
>    progname = "grub-mkimage";
>    
>    while (1)
>      {
> -      int c = getopt_long (argc, argv, "d:p:o:hVv", options, 0);
> +      int c = getopt_long (argc, argv, "d:p:m:o:hVv", options, 0);
>  
>        if (c == -1)
>  	break;
> @@ -250,6 +268,13 @@ main (int argc, char *argv[])
>  	    dir = xstrdup (optarg);
>  	    break;
>  
> +	  case 'm':
> +	    if (memdisk)
> +	      free (memdisk);
> +
> +	    memdisk = xstrdup (optarg);
> +	    break;
> +
>  	  case 'h':
>  	    usage (0);
>  	    break;
> @@ -282,7 +307,7 @@ main (int argc, char *argv[])
>  	grub_util_error ("cannot open %s", output);
>      }
>  
> -  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind);
> +  generate_image (dir ? : GRUB_LIBDIR, prefix ? : DEFAULT_DIRECTORY, fp, argv + optind, memdisk);
>  
>    fclose (fp);
>  

> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel


-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

embedding data in ELF (Re: moving ata initialisation to a command)

[Robert Millan]

On Thu, Dec 20, 2007 at 06:36:07PM +0100, Robert Millan wrote:
> 
> I'd prefer to support embedding filesystems in core.img instead.  This
> solves the problem for every situation instead of just PC boot media
> (e.g. LinuxBIOS ELF payload image).

Btw, any ELF wizard around?  I wonder if ELF has any provisions for embedding
raw data to it.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)

Re: embedding data in ELF (Re: moving ata initialisation to a command)

[Pavel Roskin]

On Wed, 2007-12-26 at 09:00 +0100, Robert Millan wrote:
> On Thu, Dec 20, 2007 at 06:36:07PM +0100, Robert Millan wrote:
> > 
> > I'd prefer to support embedding filesystems in core.img instead.  This
> > solves the problem for every situation instead of just PC boot media
> > (e.g. LinuxBIOS ELF payload image).
> 
> Btw, any ELF wizard around?  I wonder if ELF has any provisions for embedding
> raw data to it.

You can add arbitrary sections, just make sure to avoid reserved names:
http://www.linux-foundation.org/spec/book/ELF-generic/ELF-generic/specialsections.html

objcopy supports --add-section, which can add sections from files.

Good starting point for information about ELF:
http://en.wikipedia.org/wiki/Executable_and_Linking_Format

-- 
Regards,
Pavel Roskin

avoid firmware in grub-mkrescue (Re: moving ata initialisation to a command)

[Robert Millan]

On Mon, Dec 17, 2007 at 05:01:42PM +0100, Marco Gerards wrote:
> Robert Millan <rmh@aybabtu.com> writes:
> 
> > I'd like to move ata.mod initialisation away from its _init routine and into
> > a separate command.  This way it isn't a nuissance when it gets included in
> > monolithic builds (such as the ones made by grub-mkrescue) and disables biosdisk
> > completely.
> >
> > Does that sound fine?
> 
> No.  What if you put ata.mod in core.img?  You would expect your disks
> will be detected and that grub.cfg will be loaded immediately.  The
> user doesn't want to run a command to enable ata.mod and afterwards
> load the grub.cfg manually.
> 
> A better solution, IMO, would be changing grub-mkrescue so it doesn't
> load all modules.

I'd suggest standarising in something that can work later for keyboards and
such.  How about:

  grub-mkrescue --avoid-firmware=disk  ?

So that we can later add "disk,keyboard,screen" etc, and at the same time it'd
also work if grub-mkrescue was implemented for other platforms.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)