Re: [linux-lvm] Meaning of LV -missing_<x>_<x> devices

Re: [linux-lvm] Meaning of LV -missing_<x>_<x> devices

[Enrico Ferro]

> > after the extention of a logical volume (mysqld1) a new "ghost"
> > logical volume appeared with a <missing> suffix:
> >
> > brw-rw---- 1 root disk 253, 38 Feb 15 20:53 vgmysql-mysqld1
> > brw-rw---- 1 root disk 253, 44 Jun 1 10:21
> > vgmysql-mysqld1-missing_4_0
>
> That means the lv was extended onto a PV that is no longer online.
> The missing device is the extents allocated on the missing PV.

Thank you Stuart for this information. We are now trying to
reproduce this strange behavior using virtual machines. We have got
the same error messages during the FS extension, but the -missing_
device did not appeared. We will try again using the SAN on a test
cluster (no virtual machines).

 > > Before performing this LV we partitioned and added a new LUN to the
> > existing vgmysql volume group (again, no errors reported).
> > We did not discovered this strange device immediately (the lvextend
> > command did not reported errors), only after resizing the file
> > system.
> > The ext2fs operation aborted and so we started with investigation.
>
> > So online resize of the FS failed, we unmounted the LV, extended
> > with
> > (apparent) success the filesystems and lived happy for a few days.
> > Only
> > while a deep corruption of the FS appeared. :-(
>
> You needed to do an fsck after the aborted filesystem resize. I'm
> suprised the fs resize didn't make you do so. If you did do the fsck,
> then there are serious problems with your new PV.

You are right: we did also the fsck before extending the unmounted LV, the check 
completed apparently without problems. Only after the catastrophe we discovered 
in the syslog lines like "buffer i/o errors"
during the umount-fscheck-extension-mount activity. It seems very strange that 
similar errors are not reported as failure by command lines tools like e2fsck, 
mount, etc... or causing a remount in read-only mode of the filesystem.

> > A reboot cleaned up the strange "missing" LV, it is no more present.
> > The point is: what is the meaning of -missing_* device? When are
> > they
> > generated (and why there are no warnings reported?)
>
> Apparently, the new PV went offline between lvextend and resize.

Thank you *very much* Stuart for your help, we are still investigating
but now we have a clue related to the storage...

Best regards,
-- 
Enrico

[linux-lvm] Meaning of LV -missing_<x>_<x> devices

[Enrico Ferro]

Hi all,

after the extention of a logical volume (mysqld1) a new "ghost" logical volume
appeared with a <missing> suffix:

brw-rw---- 1 root disk 253, 38 Feb 15 20:53 vgmysql-mysqld1
brw-rw---- 1 root disk 253, 44 Jun 1 10:21 vgmysql-mysqld1-missing_4_0

Before performing this LV we partitioned and added a new LUN to the existing
vgmysql volume group (again, no errors reported).

We did not discovered this strange device immediately (the lvextend
command did not reported errors), only after resizing the file system.
The ext2fs operation aborted and so we started with investigation.

Commands like vgdisplay reported errors like:

# vgdisplay -s
/dev/mapper/vgmysql-mysqld1-missing_4_0: read failed after 0 of 4096 at 0: 
Input/output error
"vg00" 132.59 GB [28.78 GB used / 103.81 GB free]
"vgmysql" 811.95 GB [793.00 GB used / 18.95 GB free]

So online resize of the FS failed, we unmounted the LV, extended with
(apparent) success the filesystems and lived happy for a few days. Only
while a deep corruption of the FS appeared. :-(

A reboot cleaned up the strange "missing" LV, it is no more present.

The point is: what is the meaning of -missing_* device? When are they
generated (and why there are no warnings reported?)

thank your for your suggestions,
best regards,
-- 
Enrico

Re: [linux-lvm] Meaning of LV -missing_<x>_<x> devices

[Stuart D. Gathman]

On Wed, 15 Jun 2011, Enrico Ferro wrote:

> after the extention of a logical volume (mysqld1) a new "ghost" logical 
> volume
> appeared with a <missing> suffix:
>
> brw-rw---- 1 root disk 253, 38 Feb 15 20:53 vgmysql-mysqld1
> brw-rw---- 1 root disk 253, 44 Jun 1 10:21 vgmysql-mysqld1-missing_4_0

That means the lv was extended onto a PV that is no longer online.
The missing device is the extents allocated on the missing PV.

> Before performing this LV we partitioned and added a new LUN to the existing
> vgmysql volume group (again, no errors reported).
>
> We did not discovered this strange device immediately (the lvextend
> command did not reported errors), only after resizing the file system.
> The ext2fs operation aborted and so we started with investigation.

> So online resize of the FS failed, we unmounted the LV, extended with
> (apparent) success the filesystems and lived happy for a few days. Only
> while a deep corruption of the FS appeared. :-(

You needed to do an fsck after the aborted filesystem resize.  I'm
suprised the fs resize didn't make you do so.  If you did do the fsck,
then there are serious problems with your new PV.

> A reboot cleaned up the strange "missing" LV, it is no more present.

> The point is: what is the meaning of -missing_* device? When are they
> generated (and why there are no warnings reported?)

Apparently, the new PV went offline between lvextend and resize.

--
 	      Stuart D. Gathman <stuart@bmsi.com>
     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.