[linux-lvm] vgchange doesn't enable found VGs

[linux-lvm] vgchange doesn't enable found VGs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

Strange:

+ /usr/sbin/vgscan --ignorelockingfailure
  Reading all physical volumes.  This may take a while...
  Found volume group "foobar" using metadata type lvm2
+ /usr/sbin/vgchange -a y --ignorelockingfailure
  No volume groups found

It seems I have to tell vgchange to enable the foobar VG explicitly.

Why is this?

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] vgchange doesn't enable found VGs

[Bryn M. Reeves]

Brian J. Murrell wrote:
> Strange:
> 
> + /usr/sbin/vgscan --ignorelockingfailure Reading all physical
> volumes.  This may take a while... Found volume group "foobar"
> using metadata type lvm2 + /usr/sbin/vgchange -a y
> --ignorelockingfailure No volume groups found
> 
> It seems I have to tell vgchange to enable the foobar VG
> explicitly.
> 
> Why is this?

It's difficult to say from the output above. Try running the tools
with "-vvv" and looking at the detailed output. They'll produce a lot
of output when run this way - you may want to use something like
pastebin & send a link to the list rather than posting the entire output.

Why do you need the --ignorelockingfailure?

Bryn.

Re: [linux-lvm] vgchange doesn't enable found VGs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 2960 bytes --]

On Fri, 2009-02-27 at 16:29 +0000, Bryn M. Reeves wrote:
> 
> It's difficult to say from the output above. Try running the tools
> with "-vvv" and looking at the detailed output. They'll produce a lot
> of output when run this way

Not so much in fact.

> Why do you need the --ignorelockingfailure?

Because this is being done during system boot.

Here's "vgchange -v -v -v -a n" output:

        Processing: vgchange -v -v -v -a n
        O_DIRECT will be used
      Setting global/locking_type to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
    Finding all volume groups
        /dev/ramdisk: Not using O_DIRECT
        Opened /dev/ramdisk RO
        /dev/ramdisk: block size is 4096 bytes
      /dev/ramdisk: No label detected
        Closed /dev/ramdisk
        /dev/ram: Not using O_DIRECT
        Opened /dev/ram RO
        /dev/ram: block size is 4096 bytes
      /dev/ram: No label detected
        Closed /dev/ram
        /dev/ram2: Not using O_DIRECT
        Opened /dev/ram2 RO
        /dev/ram2: block size is 4096 bytes
      /dev/ram2: No label detected
        Closed /dev/ram2
...
        /dev/ram14: Not using O_DIRECT
        Opened /dev/ram14 RO
        /dev/ram14: block size is 4096 bytes
      /dev/ram14: No label detected
        Closed /dev/ram14
        /dev/ram15: Not using O_DIRECT
        Opened /dev/ram15 RO
        /dev/ram15: block size is 4096 bytes
      /dev/ram15: No label detected
        Closed /dev/ram15
  No volume groups found

The eclipses represents ram3, ram4, ram5, to ram 13.

When I specify the volume group, I see this:

         Processing: vgchange -v -v -v -a n lustre
        O_DIRECT will be used
      Setting global/locking_type to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
    Using volume group(s) on command line
      Locking /var/lock/lvm/V_lustre RB
    Finding volume group "lustre"
        /dev/ramdisk: Not using O_DIRECT
        Opened /dev/ramdisk RO
        /dev/ramdisk: block size is 4096 bytes
      /dev/ramdisk: No label detected
        Closed /dev/ramdisk
        /dev/ram: Not using O_DIRECT
        Opened /dev/ram RO
        /dev/ram: block size is 4096 bytes
      /dev/ram: No label detected
        Closed /dev/ram
        /dev/ram2: Not using O_DIRECT
        Opened /dev/ram2 RO
        /dev/ram2: block size is 4096 bytes
      /dev/ram2: No label detected
        Closed /dev/ram2
...
    Wiping cache of LVM-capable devices
        /dev/console: Not a block device
        /dev/core: Not a block device
        /dev/evtchn: Not a block device
        /dev/fd: Symbolic link to directory
...

and it considers a whole boatload more devices, including /dev/sda,
where my volume groups is.  Now the one thing that just occurred to me
as relevant is that my root filesystem (/) is read-only.

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] vgchange doesn't enable found VGs

[Bryn M. Reeves]

Brian J. Murrell wrote:
> /dev/ram14: Not using O_DIRECT Opened /dev/ram14 RO /dev/ram14:
> block size is 4096 bytes /dev/ram14: No label detected Closed
> /dev/ram14 /dev/ram15: Not using O_DIRECT Opened /dev/ram15 RO 
> /dev/ram15: block size is 4096 bytes /dev/ram15: No label detected 
> Closed /dev/ram15 No volume groups found

So it's not scanning /dev/sda at all (only ramdisks?). Do you have any
filter settings in lvm.conf and are you sure that the SD device nodes
will have been created at this point in the boot process?

You don't mention the distribution you're using but most 2.6 based
setups will use udev to manage driver loading and device node creation
during boot. It's possible for these to get out of sync in some cases.

Is there any way you can grab a listing of /dev at the moment the
vgchange runs during boot?

> 
> When I specify the volume group, I see this:
> 
> Processing: vgchange -v -v -v -a n lustre O_DIRECT will be used 
> Setting global/locking_type to 1 File-based locking selected. 
> Setting global/locking_dir to /var/lock/lvm Using volume group(s)
> on command line Locking /var/lock/lvm/V_lustre RB Finding volume
> group "lustre" ...

Did it scan /dev/sda this time?

> Wiping cache of LVM-capable devices /dev/console: Not a block
> device /dev/core: Not a block device /dev/evtchn: Not a block
> device /dev/fd: Symbolic link to directory ...
> 
> and it considers a whole boatload more devices, including /dev/sda,
>  where my volume groups is.  Now the one thing that just occurred
> to me as relevant is that my root filesystem (/) is read-only.

Shouldn't matter.

Just to be clear, are you only seeing this problem during boot or are
you able to reproduce once the system has completed booting (i.e.
vgchange -ay fails but vgchange -ay <vg name> succeeds)?

Your earlier messages didn't make that clear.

Regards,
Bryn.

Re: [linux-lvm] vgchange doesn't enable found VGs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1539 bytes --]

On Fri, 2009-02-27 at 17:02 +0000, Bryn M. Reeves wrote:
> 
> So it's not scanning /dev/sda at all (only ramdisks?).

So it seems, yes.

> Do you have any
> filter settings in lvm.conf

devices {
    dir = "/dev"
    scan = [ "/dev" ]
    preferred_names = [ ]
    filter = [ "a/.*/" ]
    cache_dir = "/etc/lvm/cache"
    cache_file_prefix = ""
    write_cache_state = 1
    sysfs_scan = 1      
    md_component_detection = 1
    ignore_suspended_devices = 0
}

>  and are you sure that the SD device nodes
> will have been created at this point in the boot process?

Well, the data from the previous message was done on a fully booted
machine, complete with /dev/sda being present for many, many minutes if
not hours.

> You don't mention the distribution you're using

RHEL5.2.

> but most 2.6 based
> setups will use udev to manage driver loading and device node creation
> during boot. It's possible for these to get out of sync in some cases.

Yes, but again, the data I am giving you is being taken from a
completely up and running system.

> Is there any way you can grab a listing of /dev at the moment the
> vgchange runs during boot?

Not relevant per above.

> Did it scan /dev/sda this time?

Yes.

> Just to be clear, are you only seeing this problem during boot or are
> you able to reproduce once the system has completed booting (i.e.
> vgchange -ay fails but vgchange -ay <vg name> succeeds)?

As per the above, I can reproduce on a completed booted system.

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] vgchange doesn't enable found VGs

[Alasdair G Kergon]

On Fri, Feb 27, 2009 at 12:30:30PM -0500, Brian J. Murrell wrote:
>     cache_dir = "/etc/lvm/cache"

The cache_dir must be *writeable*.
If you're attempting to use read-only root, then move it somewhere else
(is /var writeable? or make a ramfs) or delete that directory *and*
change

>     write_cache_state = 1
to 0.

(This will slow commands down.)

Or if all devices on your system are the same on every boot, 
place the correct version of that file into your read only image.
This won't work if devices can move about unless you include
all the possible locations in there.

> > Is there any way you can grab a listing of /dev at the moment the
> > vgchange runs during boot?
> Not relevant per above.

-vvvv output provides this information already.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] vgchange doesn't enable found VGs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Fri, 2009-02-27 at 20:00 +0000, Alasdair G Kergon wrote:
> On Fri, Feb 27, 2009 at 12:30:30PM -0500, Brian J. Murrell wrote:
> >     cache_dir = "/etc/lvm/cache"
> 
> The cache_dir must be *writeable*.

And persistent or just writable?

Interestingly enough, RHEL5.2's /etc/rwtab (the files and directories
that need to be made "magically" writable -- through bind mounting to
either persistent storage or tmpfs) has:

files   /etc/lvm/.cache

That's starting to look like a type methinks.

> If you're attempting to use read-only root, then move it somewhere else
> (is /var writeable? or make a ramfs) or delete that directory *and*
> change

Ahhh.  "a ramfs".  So persistence is not a requirement it seems.

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] vgchange doesn't enable found VGs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Fri, 2009-02-27 at 15:19 -0500, Brian J. Murrell wrote:
> On Fri, 2009-02-27 at 20:00 +0000, Alasdair G Kergon wrote:
> > On Fri, Feb 27, 2009 at 12:30:30PM -0500, Brian J. Murrell wrote:
> > >     cache_dir = "/etc/lvm/cache"
> > 
> > The cache_dir must be *writeable*.
> 
> And persistent or just writable?
> 
> Interestingly enough, RHEL5.2's /etc/rwtab (the files and directories
> that need to be made "magically" writable -- through bind mounting to
> either persistent storage or tmpfs) has:
> 
> files   /etc/lvm/.cache
> 
> That's starting to look like a type methinks.

But this is moot as the /etc/rwtab bind mounts are not configured until
after lvm in the rc.sysinit and crappity crap crap: there is nothing
mounted at that point in the boot that is writable.

But changing (only) "write_cache_state = 0" in the lvm.conf made no
difference in vgchange's inability to find the volume group.

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] vgchange doesn't enable found VGs

[Alasdair G Kergon]

You wrote:
  my root filesystem (/) is read-only.  

That's what I was reacting to.
With a read-only root, if /etc/lvm/cache/ is being used and is
wrong it will be unable to correct itself.
You must either make the cache file writeable or delete it.

vgchange -ay
and
vgchange -ay XXX

behave differently.
The first activates all VGs it sees, respecting the cache file
if it exists.
But with the second, if after doing that (and unless trustcache is set) it
can't find XXX, it considers the cache file might be wrong and tries again
without it, updating it afterwards iff write_cache_file is set.

Whether or not to preserve that file across boot depends on your system,
but typically vgscan will be run each time to regenerate it.

Alasdair
-- 
agk@redhat.com