Broken common.rmk change

Broken common.rmk change

[David Miller]

Robert, the set of objects used to build grub-mkdevicemap on
sparc64-ieee1275 is not the same as those used on other architectures.
So this change was not correct:

2009-11-26  Robert Millan  <rmh.grub@aybabtu.com>

	* conf/common.rmk (sbin_UTILITIES): Add `grub-mkdevicemap'.
 ...
	* conf/i386-coreboot.rmk (sbin_UTILITIES): Remove `grub-mkdevicemap'.
	(grub_mkdevicemap_SOURCES): Remove.
 ...
	* conf/sparc64-ieee1275.rmk: Likewise.

In particular, we use a special implementation devicemap.c on
sparc64-ieee1275 so that openfirmware device nodes are emitted instead
of "hd0" et al.

So when you moved the build rule into common.rmk you broke this.

This is probably the primary reason that the current tree works for
nobody on sparc64 :-)

Before I found this problem, I tested with an existing devicemap and
config file on a Niagara LDOM Linux guest and current trunk worked as
well as it did when I was last active several months ago and I was
able to boot Linux kernels with it.

Re: Broken common.rmk change

[Robert Millan]

On Sun, Dec 06, 2009 at 01:21:46AM -0800, David Miller wrote:
> 
> Robert, the set of objects used to build grub-mkdevicemap on
> sparc64-ieee1275 is not the same as those used on other architectures.
> So this change was not correct:
> 
> 2009-11-26  Robert Millan  <rmh.grub@aybabtu.com>
> 
> 	* conf/common.rmk (sbin_UTILITIES): Add `grub-mkdevicemap'.
>  ...
> 	* conf/i386-coreboot.rmk (sbin_UTILITIES): Remove `grub-mkdevicemap'.
> 	(grub_mkdevicemap_SOURCES): Remove.
>  ...
> 	* conf/sparc64-ieee1275.rmk: Likewise.
> 
> In particular, we use a special implementation devicemap.c on
> sparc64-ieee1275 so that openfirmware device nodes are emitted instead
> of "hd0" et al.
> 
> So when you moved the build rule into common.rmk you broke this.
> 
> This is probably the primary reason that the current tree works for
> nobody on sparc64 :-)
> 
> Before I found this problem, I tested with an existing devicemap and
> config file on a Niagara LDOM Linux guest and current trunk worked as
> well as it did when I was last active several months ago and I was
> able to boot Linux kernels with it.

We're actually in the process of getting rid of device.map;  in fact the
"hd0" names you see have no real effect on i386-pc, they're just ignored.

We now have more robust code that doesn't hardcode drive names.  grub-setup
now accepts plain system devices as arguments, even if they're not listed in
device.map.  grub-mkconfig generates a grub.cfg that relies on UUIDs instead
of hardcoding, etc.

But I might have missed some detail.  Is there a peculiarity of
sparc64-ieee1275 that makes this approach unpractical?

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Broken common.rmk change

[David Miller]

From: Robert Millan <rmh@aybabtu.com>
Date: Sun, 6 Dec 2009 14:10:26 +0100

> We're actually in the process of getting rid of device.map;

Meanwhile, please fix the regression you added to the tree. :-)

And I'll look into what you've mentioned to see if that kind of scheme
will work properly on Sparc.

Re: Broken common.rmk change

[Robert Millan]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Sun, Dec 06, 2009 at 04:21:50PM -0800, David Miller wrote:
> From: Robert Millan <rmh@aybabtu.com>
> Date: Sun, 6 Dec 2009 14:10:26 +0100
> 
> > We're actually in the process of getting rid of device.map;
> 
> Meanwhile, please fix the regression you added to the tree. :-)

I think this should do it.  Please can you test?

> And I'll look into what you've mentioned to see if that kind of scheme
> will work properly on Sparc.

Thanks!

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

[-- Attachment #2: sparc64_devicemap.diff --]
[-- Type: text/x-diff, Size: 572 bytes --]

=== modified file 'conf/common.rmk'
--- conf/common.rmk	2009-11-25 23:10:02 +0000
+++ conf/common.rmk	2009-12-07 14:03:09 +0000
@@ -3,7 +3,13 @@
 sbin_UTILITIES += grub-mkdevicemap
 grub_mkdevicemap_SOURCES = gnulib/progname.c util/grub-mkdevicemap.c \
 	util/deviceiter.c \
-	util/devicemap.c util/misc.c
+	util/misc.c
+
+ifeq ($(target_cpu)-$(platform), sparc64-ieee1275)
+grub_mkdevicemap_SOURCES += util/ieee1275/ofpath.c util/ieee1275/devicemap.c
+else
+grub_mkdevicemap_SOURCES += util/devicemap.c
+endif
 
 # For grub-mkelfimage.
 bin_UTILITIES += grub-mkelfimage

Re: Broken common.rmk change

[David Miller]

From: Robert Millan <rmh@aybabtu.com>
Date: Mon, 7 Dec 2009 15:04:11 +0100

> On Sun, Dec 06, 2009 at 04:21:50PM -0800, David Miller wrote:
>> From: Robert Millan <rmh@aybabtu.com>
>> Date: Sun, 6 Dec 2009 14:10:26 +0100
>> 
>> > We're actually in the process of getting rid of device.map;
>> 
>> Meanwhile, please fix the regression you added to the tree. :-)
> 
> I think this should do it.  Please can you test?

This patch works fine for me, thanks!

Re: Broken common.rmk change

[David Miller]

From: Robert Millan <rmh@aybabtu.com>
Date: Sun, 6 Dec 2009 14:10:26 +0100

> grub-mkconfig generates a grub.cfg that relies on UUIDs instead
> of hardcoding, etc.

Well, for one thing I don't use initrd's for my Linux kernels,
therefore specifying root devices via UUIDs isn't going to work.

Re: Broken common.rmk change

[Colin Watson]

On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
> From: Robert Millan <rmh@aybabtu.com>
> Date: Sun, 6 Dec 2009 14:10:26 +0100
> 
> > grub-mkconfig generates a grub.cfg that relies on UUIDs instead
> > of hardcoding, etc.
> 
> Well, for one thing I don't use initrd's for my Linux kernels,
> therefore specifying root devices via UUIDs isn't going to work.

Then don't use that feature of grub-mkconfig (i.e. set
GRUB_DISABLE_LINUX_UUID=false in /etc/default/grub). That's independent
of device.map handling, though.

-- 
Colin Watson                                       [cjwatson@ubuntu.com]

Re: Broken common.rmk change

[David Miller]

From: Colin Watson <cjwatson@ubuntu.com>
Date: Mon, 7 Dec 2009 10:20:16 +0000

> On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
>> From: Robert Millan <rmh@aybabtu.com>
>> Date: Sun, 6 Dec 2009 14:10:26 +0100
>> 
>> > grub-mkconfig generates a grub.cfg that relies on UUIDs instead
>> > of hardcoding, etc.
>> 
>> Well, for one thing I don't use initrd's for my Linux kernels,
>> therefore specifying root devices via UUIDs isn't going to work.
> 
> Then don't use that feature of grub-mkconfig (i.e. set
> GRUB_DISABLE_LINUX_UUID=false in /etc/default/grub). That's independent
> of device.map handling, though.

grub-mkconfig actually seems to do this automatically for me.

Re: Broken common.rmk change

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1282 bytes --]

David Miller wrote:
> From: Robert Millan <rmh@aybabtu.com>
> Date: Sun, 6 Dec 2009 14:10:26 +0100
>
>   
>> grub-mkconfig generates a grub.cfg that relies on UUIDs instead
>> of hardcoding, etc.
>>     
>
> Well, for one thing I don't use initrd's for my Linux kernels,
> therefore specifying root devices via UUIDs isn't going to work.
>
>   
The problem actually is the following:
When we're in OS we know the OS device name but not the firmware device
name. So we're able to generate correct root= for kernel command line
but can only guess which names you would need for grub's set root=. The
solution we're going for is to have UUID of fs stored in grub.cfg and go
through all partitions on boot and find the right one nd set it to root
variable. As for passing root= kernel parameter the decision of using
device name or UUID= is based on heuristics (sigh) but you can disable
them altogether in /etc/default/grub:

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to
Linux
#GRUB_DISABLE_LINUX_UUID=true

> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 293 bytes --]

Re: Broken common.rmk change

[David Miller]

From: Vladimir '^[$B&U^[(B-coder/phcoder' Serbinenko <phcoder@gmail.com>
Date: Mon, 07 Dec 2009 11:48:07 +0100

> The problem actually is the following:
> When we're in OS we know the OS device name but not the firmware device
> name. So we're able to generate correct root= for kernel command line
> but can only guess which names you would need for grub's set root=. The
> solution we're going for is to have UUID of fs stored in grub.cfg and go
> through all partitions on boot and find the right one nd set it to root
> variable. As for passing root= kernel parameter the decision of using
> device name or UUID= is based on heuristics (sigh) but you can disable
> them altogether in /etc/default/grub:
> 
> # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to
> Linux
> #GRUB_DISABLE_LINUX_UUID=true

UUIDs are convenient but can be very time consuming.

Some sparc64 systems come with 10 or 20 block OpenFirmware device
aliases, and we're going probe all of them, one by one, to look for
the UUID.

The firmware device name is so trivially obtainable, especially on
ieee1275.  That's what the grub-ofpathname tool I wrote does on
sparc64.  And it's what the 'ofpathname' script used by powerpc on
grub does as well.

Re: Broken common.rmk change

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

David Miller wrote:
> From: Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>
> Date: Mon, 07 Dec 2009 11:48:07 +0100
>
>   
>> The problem actually is the following:
>> When we're in OS we know the OS device name but not the firmware device
>> name. So we're able to generate correct root= for kernel command line
>> but can only guess which names you would need for grub's set root=. The
>> solution we're going for is to have UUID of fs stored in grub.cfg and go
>> through all partitions on boot and find the right one nd set it to root
>> variable. As for passing root= kernel parameter the decision of using
>> device name or UUID= is based on heuristics (sigh) but you can disable
>> them altogether in /etc/default/grub:
>>
>> # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to
>> Linux
>> #GRUB_DISABLE_LINUX_UUID=true
>>     
>
> UUIDs are convenient but can be very time consuming.
>
> Some sparc64 systems come with 10 or 20 block OpenFirmware device
> aliases, and we're going probe all of them, one by one, to look for
> the UUID.
>
> The firmware device name is so trivially obtainable, especially on
> ieee1275.  That's what the grub-ofpathname tool I wrote does on
> sparc64.  And it's what the 'ofpathname' script used by powerpc on
> grub does as well.
>
>   
I think we can go for the approach Seth Goldberg proposed: UUID with
hint. First we check if hinted device has correct UUID and if so we
return it, if not we do the search


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 293 bytes --]

Re: Broken common.rmk change

[Robert Millan]

On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
> From: Robert Millan <rmh@aybabtu.com>
> Date: Sun, 6 Dec 2009 14:10:26 +0100
> 
> > grub-mkconfig generates a grub.cfg that relies on UUIDs instead
> > of hardcoding, etc.
> 
> Well, for one thing I don't use initrd's for my Linux kernels,
> therefore specifying root devices via UUIDs isn't going to work.

I was referring to use of UUIDs internally by GRUB (e.g. the search command
instances grub-mkconfig generates).

We also pass UUIDs to Linux, but this can be disabled in /etc/default/grub.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Broken common.rmk change

[David Miller]

From: Robert Millan <rmh@aybabtu.com>
Date: Mon, 7 Dec 2009 15:05:16 +0100

> On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
>> From: Robert Millan <rmh@aybabtu.com>
>> Date: Sun, 6 Dec 2009 14:10:26 +0100
>> 
>> > grub-mkconfig generates a grub.cfg that relies on UUIDs instead
>> > of hardcoding, etc.
>> 
>> Well, for one thing I don't use initrd's for my Linux kernels,
>> therefore specifying root devices via UUIDs isn't going to work.
> 
> I was referring to use of UUIDs internally by GRUB (e.g. the search command
> instances grub-mkconfig generates).

Ok, but as noted in another thread this can make grub
very slow on ieee1275 systems as we are going to look
for the UUID on every ieee1275 device alias present and
there can be tons of those.

Re: Broken common.rmk change

[Robert Millan]

On Mon, Dec 07, 2009 at 11:50:33AM -0800, David Miller wrote:
> From: Robert Millan <rmh@aybabtu.com>
> Date: Mon, 7 Dec 2009 15:05:16 +0100
> 
> > On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
> >> From: Robert Millan <rmh@aybabtu.com>
> >> Date: Sun, 6 Dec 2009 14:10:26 +0100
> >> 
> >> > grub-mkconfig generates a grub.cfg that relies on UUIDs instead
> >> > of hardcoding, etc.
> >> 
> >> Well, for one thing I don't use initrd's for my Linux kernels,
> >> therefore specifying root devices via UUIDs isn't going to work.
> > 
> > I was referring to use of UUIDs internally by GRUB (e.g. the search command
> > instances grub-mkconfig generates).
> 
> Ok, but as noted in another thread this can make grub
> very slow on ieee1275 systems as we are going to look
> for the UUID on every ieee1275 device alias present and
> there can be tons of those.

I'm afraid I missed that thread.  In any case, we basically do/can do three
things to paliate that problem:

  - Cache the prepare_grub_to_access_device() calls at the shell/mkconfig
    layer (see util/grub.d/10_linux.in for an example).

  - Disk cache (kern/disk.c).

  - Adjust disk iteration order so that disks more likely to be the ones
    we're looking for show up first (e.g. on i386-pc we list hard disks
    before floppies or CD).  Note that UUID search aborts as soon as a
    match is found; it doesn't iterate the whole list.

More ideas welcome, of course :-)

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Broken common.rmk change

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Robert Millan wrote:
> On Mon, Dec 07, 2009 at 11:50:33AM -0800, David Miller wrote:
>   
>> From: Robert Millan <rmh@aybabtu.com>
>> Date: Mon, 7 Dec 2009 15:05:16 +0100
>>
>>     
>>> On Sun, Dec 06, 2009 at 05:22:52PM -0800, David Miller wrote:
>>>       
>>>> From: Robert Millan <rmh@aybabtu.com>
>>>> Date: Sun, 6 Dec 2009 14:10:26 +0100
>>>>
>>>>         
>>>>> grub-mkconfig generates a grub.cfg that relies on UUIDs instead
>>>>> of hardcoding, etc.
>>>>>           
>>>> Well, for one thing I don't use initrd's for my Linux kernels,
>>>> therefore specifying root devices via UUIDs isn't going to work.
>>>>         
>>> I was referring to use of UUIDs internally by GRUB (e.g. the search command
>>> instances grub-mkconfig generates).
>>>       
>> Ok, but as noted in another thread this can make grub
>> very slow on ieee1275 systems as we are going to look
>> for the UUID on every ieee1275 device alias present and
>> there can be tons of those.
>>     
>
> I'm afraid I missed that thread.  In any case, we basically do/can do three
> things to paliate that problem:
>
>   - Cache the prepare_grub_to_access_device() calls at the shell/mkconfig
>     layer (see util/grub.d/10_linux.in for an example).
>
>   - Disk cache (kern/disk.c).
>
>   - Adjust disk iteration order so that disks more likely to be the ones
>     we're looking for show up first (e.g. on i386-pc we list hard disks
>     before floppies or CD).  Note that UUID search aborts as soon as a
>     match is found; it doesn't iterate the whole list.
>
> More ideas welcome, of course :-)
>
>   
Seth proposed a hinting technique:
search --fs_uuid ... -s root --hint hd0,1
then hd0,1 will be checked first and only on failure it will revert to
scanning


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 293 bytes --]

Re: Broken common.rmk change

[Robert Millan]

On Mon, Dec 07, 2009 at 10:23:51PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
> Seth proposed a hinting technique:
> search --fs_uuid ... -s root --hint hd0,1
> then hd0,1 will be checked first and only on failure it will revert to
> scanning

Sounds good.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Broken common.rmk change

[BVK Chaitanya]

On Thu, Dec 10, 2009 at 3:15 AM, Robert Millan <rmh@aybabtu.com> wrote:
> On Mon, Dec 07, 2009 at 10:23:51PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
>> Seth proposed a hinting technique:
>> search --fs_uuid ... -s root --hint hd0,1
>> then hd0,1 will be checked first and only on failure it will revert to
>> scanning


I assume, this --hint argument can be given multiple times, and while
searching their order is followed ???



regards,
-- 
bvk.chaitanya

Re: Broken common.rmk change

[Seth Goldberg]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 782 bytes --]


  That was my intent, yes.

  --S

Quoting BVK Chaitanya, who wrote the following on Thu, 10 Dec 2009:

> On Thu, Dec 10, 2009 at 3:15 AM, Robert Millan <rmh@aybabtu.com> wrote:
>> On Mon, Dec 07, 2009 at 10:23:51PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
>>> Seth proposed a hinting technique:
>>> search --fs_uuid ... -s root --hint hd0,1
>>> then hd0,1 will be checked first and only on failure it will revert to
>>> scanning
>
>
> I assume, this --hint argument can be given multiple times, and while
> searching their order is followed ???
>
>
>
> regards,
> -- 
> bvk.chaitanya
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: Broken common.rmk change

[Robert Millan]

On Wed, Dec 09, 2009 at 10:45:16PM +0100, Robert Millan wrote:
> On Mon, Dec 07, 2009 at 10:23:51PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
> > Seth proposed a hinting technique:
> > search --fs_uuid ... -s root --hint hd0,1
> > then hd0,1 will be checked first and only on failure it will revert to
> > scanning
> 
> Sounds good.

Thinking a bit more, in the vast majority of times, $root variable is initially
defined to the drive that contains the filesystem we want.  Partition might be
wrong, but then again partition numbers are persistent.

Maybe our problem can be solved by making grub-mkconfig a bit smarter, so that
UUIDs are only used when /boot partition is in a different drive that the one
we embed core.img to.

-- 
Robert Millan

  "Be the change you want to see in the world" -- Gandhi