grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

[Robert Millan]

2010/8/4, Seth Goldberg <seth.goldberg@oracle.com>:
>> I don't think this is the appropiate place for this kind of sanity check.
>> Keep in mind "grub-probe -t device" has many purposes, and for some of
>> them
>> (e.g. determining the partmap) the ZFS status is irrelevant.
>
>    Hmm.  I do see your point here, but then I'd ask how you plan to handle
> ensuring that all disks in a mirror are properly updated with boot/core.
> I think the proper thing would be either to have grub-probe output a set of
> devices, onto which grub-setup would iterate and install boot/core, or
> implement an abstraction that allows that to happen behind the scenes.  How
> does this work for lvm or other raids?

There are two approaches, one of them is simply to run grub-install for each
of the intended devices.  This is what Debian scripts do, and is independant
on the content in the partitions of those devices (lvm, raid, zfs, whatever).

The other is giving grub-setup special knowledge of the abstraction. We do
this for RAID1, see the code around:

util/i386/pc/grub-setup.c:      devicelist = grub_util_raid_getmembers
(dest_dev);

which allows user to e.g. "grub-install /dev/md0". Then the scripts
embed raid.mod
into core.img, and (in principle) after core.img has been loaded GRUB has RAID
mirroring protection to access /boot/grub/.

I suppose it's possible to do something similar for ZFS mirror layouts.

>    The question is: should GRUB2's installation be implicit or explicit for
> all members of a mirrored root pool when grub-setup is invoked.

Implicit approach only works for mirrored setup.  If you want to install to all
members in a non-mirrored pool, you still need explicit approach, so I'd only
implement implicit one if there's a clear benefit, such as the protection
offered by mirroring.  I would check if zfs.mod from grub-extras provides this
protection before trying to answer this question.

Re: grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

[Robert Millan]

I'd also point out that the device where grub-setup will embed
core.img is a minor issue here, since it's trivial to run grub-setup
multiple times.  The real problem is putting /boot/grub/ in a
mirrored filesystem so that either copy of core.img will be able
to access this directory from any of the disks.

Or if you're serious into this, from a combination of disks in a
non-mirror multi-device pool.

I think both goals require some work in zfs.mod.

Re: grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

[Seth Goldberg]

Quoting Robert Millan, who wrote the following on Wed, 4 Aug 2010:

> 2010/8/4, Seth Goldberg <seth.goldberg@oracle.com>:
>>> I don't think this is the appropiate place for this kind of sanity check.
>>> Keep in mind "grub-probe -t device" has many purposes, and for some of
>>> them
>>> (e.g. determining the partmap) the ZFS status is irrelevant.
>>
>>    Hmm.  I do see your point here, but then I'd ask how you plan to handle
>> ensuring that all disks in a mirror are properly updated with boot/core.
>> I think the proper thing would be either to have grub-probe output a set of
>> devices, onto which grub-setup would iterate and install boot/core, or
>> implement an abstraction that allows that to happen behind the scenes.  How
>> does this work for lvm or other raids?
>
> There are two approaches, one of them is simply to run grub-install for each
> of the intended devices.  This is what Debian scripts do, and is independant
> on the content in the partitions of those devices (lvm, raid, zfs, whatever).
>
> The other is giving grub-setup special knowledge of the abstraction. We do
> this for RAID1, see the code around:
>
> util/i386/pc/grub-setup.c:      devicelist = grub_util_raid_getmembers
> (dest_dev);
>
> which allows user to e.g. "grub-install /dev/md0". Then the scripts
> embed raid.mod
> into core.img, and (in principle) after core.img has been loaded GRUB has RAID
> mirroring protection to access /boot/grub/.
>
> I suppose it's possible to do something similar for ZFS mirror layouts.

   I think that's a great idea.  Doing it outside grub-setup seems a bit more 
error-prone to me (but that's actually what we do today in Solaris).

>
>>    The question is: should GRUB2's installation be implicit or explicit for
>> all members of a mirrored root pool when grub-setup is invoked.
>
> Implicit approach only works for mirrored setup.  If you want to install to all
> members in a non-mirrored pool, you still need explicit approach, so I'd only
> implement implicit one if there's a clear benefit, such as the protection
> offered by mirroring.  I would check if zfs.mod from grub-extras provides this
> protection before trying to answer this question.

   That's true -- right now, we only support root on an n-way mirrored pool. 
If support is expanded in the future, why wouldn't a similar 
grub_util_raid_getmembers() not work in an analagous way to the mirrored case?

  --S

Re: grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

[Robert Millan]

2010/8/16, Seth Goldberg <seth.goldberg@oracle.com>:
>> Implicit approach only works for mirrored setup.  If you want to install
>> to all
>> members in a non-mirrored pool, you still need explicit approach, so I'd
>> only
>> implement implicit one if there's a clear benefit, such as the protection
>> offered by mirroring.  I would check if zfs.mod from grub-extras provides
>> this
>> protection before trying to answer this question.
>
>    That's true -- right now, we only support root on an n-way mirrored pool.
> If support is expanded in the future, why wouldn't a similar
> grub_util_raid_getmembers() not work in an analagous way to the mirrored
> case?

Sorry, it would work.  I just didn't think of this possibility because I find it
confusing.

In a mirrored setup, grub-setup does what user expects (copiing core.img
to each disk).  In a non-mirrored array, user could think grub-setup will do
the analogous (interleaving core.img), but this wouldn't work, so what
you propose is to do something different which would work but is probably
not what user expects.

Not that this matters a lot (in the end user can boot), but I can't see any
advantage in providing two ways to do the same thing, specially when it
isn't obvious they are equivalent.

Also, in ZFS there isn't a single device name that identifies the whole pool
(as with mdraid), which would make the semantics even more confusing.

-- 
Robert Millan

Re: grub-setup on ZFS (Re: [PATCH] Enable `grub-probe -t device' resolution on ZFS)

[Seth Goldberg]

> Sorry, it would work.  I just didn't think of this possibility because I find it
> confusing.

  Why is it confusing?  If one has a set of disks that compose a root pool, all 
of those disks should be bootable.

>
> In a mirrored setup, grub-setup does what user expects (copiing core.img
> to each disk).  In a non-mirrored array, user could think grub-setup will do
> the analogous (interleaving core.img), but this wouldn't work, so what
> you propose is to do something different which would work but is probably
> not what user expects.

   I wouldn't think that interleaving core.img is the right thing to do -- 
even if there's a firmware implementation of zfs that can read from the 
device, for redundancy, core.img should be written to each disk.

>
> Not that this matters a lot (in the end user can boot), but I can't see any
> advantage in providing two ways to do the same thing, specially when it
> isn't obvious they are equivalent.

   What is the alternative?  core.img would need to be installed on each disk 
in turn in both cases.

  --S

GRUB2 and libzfs on Solaris

[Seth Goldberg]

Hi,

  Just wanted to report that the conftest that looks for libzfs on 
Solaris is failing.  From config.log:

gcc -o conftest   -I/usr/sfw/include  -L/usr/sfw/lib conftest.c -lzfs >&5
ld: warning: file libdladm.so.1: required by /usr/lib/libdlpi.so.1, not 
found
Undefined                       first referenced
  symbol                             in file
dladm_close                         /usr/lib/libdlpi.so.1
dladm_dev2linkid                    /usr/lib/libdlpi.so.1
dladm_phys_info                     /usr/lib/libdlpi.so.1
dladm_walk                          /usr/lib/libdlpi.so.1
dladm_open                          /usr/lib/libdlpi.so.1
dladm_parselink                     /usr/lib/libdlpi.so.1


  On Solaris, libzfs also requires libdlpi (from /usr/lib) and libdladm 
(from /lib), so adding some stuff to configure to detect Solaris and add 
the missing libraries is needed.

  --S

Re: GRUB2 and libzfs on Solaris

[Colin Watson]

On Tue, Aug 31, 2010 at 01:34:23PM -0700, Seth Goldberg wrote:
>  On Solaris, libzfs also requires libdlpi (from /usr/lib) and libdladm  
> (from /lib), so adding some stuff to configure to detect Solaris and add  
> the missing libraries is needed.

I don't suppose we're lucky enough that there's a pkg-config file,
rather than having to manually track all of libzfs' dependencies (which
are presumably subject to change)?

-- 
Colin Watson                                       [cjwatson@ubuntu.com]

Re: GRUB2 and libzfs on Solaris

[Seth Goldberg]

Quoting Colin Watson, who wrote the following on Tue, 31 Aug 2010:

> On Tue, Aug 31, 2010 at 01:34:23PM -0700, Seth Goldberg wrote:
>>  On Solaris, libzfs also requires libdlpi (from /usr/lib) and libdladm
>> (from /lib), so adding some stuff to configure to detect Solaris and add
>> the missing libraries is needed.
>
> I don't suppose we're lucky enough that there's a pkg-config file,
> rather than having to manually track all of libzfs' dependencies (which
> are presumably subject to change)?

   Unfortunately not, though you could always use ldd to check the set of 
libraries that it depends on (admittedly, that includes the transitive 
closure of its dependencies as well, but it's better than nothing ;)).

  --S

Re: GRUB2 and libzfs on Solaris

[Seth Goldberg]

Hi,

   This fixes the emu build when libzfs is in use.  Note that changes to 
$top_srcdir/Makefile.util.def are also required -- effectively just inserting 
"$(LIBZFS) $(LIBNVPAIR)" to the ldadd lines of each util program.


=== modified file 'grub-core/Makefile.core.def'
--- grub-core/Makefile.core.def	2010-08-31 23:29:58 +0000
+++ grub-core/Makefile.core.def	2010-09-01 02:00:30 +0000
@@ -157,6 +157,7 @@

    emu = disk/host.c;
    emu = gnulib/progname.c;
+  emu = gnulib/error.c;
    emu = kern/emu/console.c;
    emu = kern/emu/getroot.c;
    emu = kern/emu/hostdisk.c;
@@ -180,7 +181,7 @@

    ldadd = 'kernel.img$(EXEEXT)';
    ldadd = '$(MODULE_FILES)';
-  ldadd = '$(LIBUTIL) $(LIBCURSES) $(LIBSDL) $(LIBUSB) $(LIBPCIACCESS) $(LIBDEVMAPPER)';
+  ldadd = '$(LIBUTIL) $(LIBCURSES) $(LIBSDL) $(LIBUSB) $(LIBPCIACCESS) $(LIBDEVMAPPER) $(LIBZFS) $(LIBNVPAIR)';

    enable = emu;
  };
@@ -193,7 +194,7 @@
    emu_nodist = symlist.c;

    ldadd = 'kernel.img$(EXEEXT)';
-  ldadd = '$(LIBUTIL) $(LIBCURSES) $(LIBSDL) $(LIBUSB) $(LIBPCIACCESS) $(LIBDEVMAPPER)';
+  ldadd = '$(LIBUTIL) $(LIBCURSES) $(LIBSDL) $(LIBUSB) $(LIBPCIACCESS) $(LIBDEVMAPPER) $(LIBZFS) $(LIBNVPAIR)';

    enable = emu;
  };