[linux-lvm]corrupt /dev/lvm - bizzare properties

[linux-lvm]corrupt /dev/lvm - bizzare properties

[Chris Doherty]

I've been using LVM happily for several months without any complaints or 
problems.  This morning that all changed.  The boot process stopped cold when 
lvm-mod (w|c)ouldn't load.  My first impulse was to fsck, which I did as a 
matter of course.  Eventually, I stumbled across what appears to be the source 
of my problem:

root@connect4:~# ls -la /dev/lvm
?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm

The box is a fairly vanilla RedHat 9 install.  Here's a bit more info which 
might be relevant:

root@connect4:~# uname -a
Linux connect4 2.4.21 #1 Fri Aug 1 00:32:29 CST 2003 i586 i586 i386 GNU/Linux
root@connect4:~# dmesg |grep LVM
LVM version 1.0.5+(22/07/2002) module loaded
root@connect4:~# lsmod |grep lvm
lvm-mod                60000   0
root@connect4:~# /sbin/pvdisplay /dev/hdc1
pvdisplay -- LVM driver/module not loaded?

normally, the output would like this:

pvdisplay -- "/dev/hdc1" is a new physical volume of 23.29 GB

Googling hasn't been much help.  The closest thing I could find to my problem 
was a post to the LVM mailing list (Date: Thu Jun 13 2002 - 08:59:07 CEST) which

really just confirmed that, yes, what's happened is indeed a bad thing.

> Check that /dev/lvm has the right major/minor numbers: "ls -l /dev/lvm" 
> should give you something like this:
> 
> # ls -l /dev/lvm
> crw-r-----    1 root     root     109,   0 Mar 13  2001 /dev/lvm

I've thought about using lvmchange, but I can't discern from the manpage whether

or not this is the right thing to do.  I'm at a loss as to what to do next, 
since one false move could very easily hose my data.

[linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Måns Rullgård]

Chris Doherty <chris.doherty@adelaide.edu.au> writes:

> I've been using LVM happily for several months without any
> complaints or problems.  This morning that all changed.  The boot
> process stopped cold when lvm-mod (w|c)ouldn't load.  My first
> impulse was to fsck, which I did as a matter of course.  Eventually,
> I stumbled across what appears to be the source of my problem:
>
> root@connect4:~# ls -la /dev/lvm
> ?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm

Your filesystem seems to have taken some heavy blows.  You should fsck
it properly.  You might need the -f flag to fsck to force a complete
check if filesystem is marked clean.

-- 
M�ns Rullg�rd
mru@kth.se

[linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Chris Doherty]

Quoting M�ns Rullg�rd <mru@kth.se>:

> Chris Doherty <chris.doherty@adelaide.edu.au> writes:
> 
> > I've been using LVM happily for several months without any
> > complaints or problems.  This morning that all changed.  The boot
> > process stopped cold when lvm-mod (w|c)ouldn't load.  My first
> > impulse was to fsck, which I did as a matter of course.  Eventually,
> > I stumbled across what appears to be the source of my problem:
> >
> > root@connect4:~# ls -la /dev/lvm
> > ?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm
> 
> Your filesystem seems to have taken some heavy blows.  You should fsck
> it properly.  You might need the -f flag to fsck to force a complete
> check if filesystem is marked clean.

the filesystem can't be mounted (which is the really worrying part) so i can't 
fsck it.

root@connect4:~# mount -t ext3 /dev/vg1/lv1 /data
mount: /dev/vg1/lv1 is not a valid block device
root@connect4:~# ls /dev/vg1/
total 124
dr-xr-xr-x    2 root     root         4096 Feb  6 18:46 ./
drwxr-xr-x   22 root     root       118784 Feb  8 14:19 ../
crw-r-----    1 root     disk     109,   0 Feb  6 18:46 group
brw-rw----    1 root     disk      58,   0 Feb  6 18:46 lv1

the fsck that i mentioned previously was run on the other (available) 
filesystems, thinking that perhaps the LVM binaries (or some other related tool)

had been corrupted.

> -- 
> M�ns Rullg�rd
> mru@kth.se

Re: [linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Rainer krienke]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1313 bytes --]

Am Sonntag, 8. Februar 2004 13:00 schrieb Chris Doherty:
> Quoting Måns Rullgård <mru@kth.se>:
> > Chris Doherty <chris.doherty@adelaide.edu.au> writes:
> > > I've been using LVM happily for several months without any
> > > complaints or problems.  This morning that all changed.  The boot
> > > process stopped cold when lvm-mod (w|c)ouldn't load.  My first
> > > impulse was to fsck, which I did as a matter of course.  Eventually,
> > > I stumbled across what appears to be the source of my problem:
> > >
> > > root@connect4:~# ls -la /dev/lvm
> > > ?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm
> >
> > Your filesystem seems to have taken some heavy blows.  You should fsck
> > it properly.  You might need the -f flag to fsck to force a complete
> > check if filesystem is marked clean.
>
> the filesystem can't be mounted (which is the really worrying part) so i
> can't fsck it.
>

A filesystem that is to be fscked can and may *never* be mounted when 
performing an fsck. Usually all fsck tools won't do anything but stop if the 
filesystem (fsck was called for)  is currently mounted! If your root 
filesystem is corrupt boot from a rescue CD or similar and then check the 
filesystem. 

Have a nice day
Rainer
-- 
Rainer Krienke, rainer@krienke.org

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Chris Doherty]

Quoting Rainer krienke <rainer@krienke.org>:

> Am Sonntag, 8. Februar 2004 13:00 schrieb Chris Doherty:
> > Quoting M�ns Rullg�rd <mru@kth.se>:
> > > Chris Doherty <chris.doherty@adelaide.edu.au> writes:
> > > > I've been using LVM happily for several months without any
> > > > complaints or problems.  This morning that all changed.  The boot
> > > > process stopped cold when lvm-mod (w|c)ouldn't load.  My first
> > > > impulse was to fsck, which I did as a matter of course.  Eventually,
> > > > I stumbled across what appears to be the source of my problem:
> > > >
> > > > root@connect4:~# ls -la /dev/lvm
> > > > ?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm
> > >
> > > Your filesystem seems to have taken some heavy blows.  You should fsck
> > > it properly.  You might need the -f flag to fsck to force a complete
> > > check if filesystem is marked clean.
> >
> > the filesystem can't be mounted (which is the really worrying part) so i
> > can't fsck it.
> >
> 
> A filesystem that is to be fscked can and may *never* be mounted when 
> performing an fsck. Usually all fsck tools won't do anything but stop if the

oops, sorry my last statement was ambiguous.  what i meant was that the 
filesystem can't be mounted *and* fsck refuses to acknowledge that it is (or 
was?) a filesystem.

root@connect4:~# fsck -cvf -t ext3 /dev/vg1/lv1
fsck 1.32 (09-Nov-2002)
e2fsck 1.32 (09-Nov-2002)
fsck.ext3: No such device or address while trying to open /dev/vg1/lv1
Possibly non-existent or swap device?

it looks like what i need is a non-destructive MAKEDEV which can repair /dev/lvm

(in a dream world) or, more realistically to create a new LVM device and 
reassign all of the logical volumes to it.  can any LVM gurus confirm whether 
this would work?

i've been trying to find a way to make a backup so i can start experimenting, so

far without success.

root@connect4:~# tar cvf lvm.tar /dev/lvm
tar: /dev/lvm: Unknown file type; file ignored
tar: Error exit delayed from previous errors

root@connect4:~# dd if=/dev/lvm of=lvm.dd
dd: reading `/dev/lvm': Invalid argument
0+0 records in
0+0 records out

root@connect4:~# man dump
mdoc warning: Empty input line #609
DUMP(8)                   BSD System Manager�??s Manual                  
DUMP(8)

NAME
     dump - ext2 filesystem backup
<snip>

two problems with dump:

1) the filesystem on the LV is (was?) ext3
2) /dev/lvm isn't a filesystem

> filesystem (fsck was called for)  is currently mounted! If your root 
> filesystem is corrupt boot from a rescue CD or similar and then check the 
> filesystem. 

[linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Måns Rullgård]

Chris Doherty <chris.doherty@adelaide.edu.au> writes:

>> > > > root@connect4:~# ls -la /dev/lvm
>> > > > ?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm
>> > >
>> > > Your filesystem seems to have taken some heavy blows.  You should fsck
>> > > it properly.  You might need the -f flag to fsck to force a complete
>> > > check if filesystem is marked clean.
>> >
>> > the filesystem can't be mounted (which is the really worrying part) so i
>> > can't fsck it.
>> >
>> 
>> A filesystem that is to be fscked can and may *never* be mounted when 
>> performing an fsck. Usually all fsck tools won't do anything but stop if the
>
> oops, sorry my last statement was ambiguous.  what i meant was that the 
> filesystem can't be mounted *and* fsck refuses to acknowledge that it is (or 
> was?) a filesystem.

The filesystem containing /dev has been damaged somehow.  You should
fsck it.  Most likely fsck will remove /dev/lvm so you'll have to
recreate it with proper values.  Don't just rm it.  Something bad has
obviously happened and changing things without a proper fsck can make
things worse.

-- 
M�ns Rullg�rd
mru@kth.se

Re: [linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Chris Doherty]

M�ns Rullg�rd wrote:

> Chris Doherty <chris.doherty@adelaide.edu.au> writes:
> 
> 
>>>>>>root@connect4:~# ls -la /dev/lvm
>>>>>>?---rws-w-  8306 840966198 976250230 875573298 Sep 24  2004 /dev/lvm
>>>>>
>>>>>Your filesystem seems to have taken some heavy blows.  You should fsck
>>>>>it properly.  You might need the -f flag to fsck to force a complete
>>>>>check if filesystem is marked clean.
>>>>
>>>>the filesystem can't be mounted (which is the really worrying part) so i
>>>>can't fsck it.
>>>>
>>>
>>>A filesystem that is to be fscked can and may *never* be mounted when 
>>>performing an fsck. Usually all fsck tools won't do anything but stop if the
>>
>>oops, sorry my last statement was ambiguous.  what i meant was that the 
>>filesystem can't be mounted *and* fsck refuses to acknowledge that it is (or 
>>was?) a filesystem.
> 
> 
> The filesystem containing /dev has been damaged somehow.  You should
> fsck it.  Most likely fsck will remove /dev/lvm so you'll have to
> recreate it with proper values.  Don't just rm it.  Something bad has
> obviously happened and changing things without a proper fsck can make
> things worse.

thanks.  i'll try this tonight and report back. :)  my limited understanding 
of LVM leads me to believe that the volume group and logical volume within it 
are actually still safe and sound in /dev/vg1

 >>root@connect4:~# ls /dev/vg1/
 >>total 124
 >>dr-xr-xr-x    2 root     root         4096 Feb  6 18:46 ./
 >>drwxr-xr-x   22 root     root       118784 Feb  8 14:19 ../
 >>crw-r-----    1 root     disk     109,   0 Feb  6 18:46 group
 >>brw-rw----    1 root     disk      58,   0 Feb  6 18:46 lv1

the partition tables on the disks (hdc and hdd) still look ok (sorry i'm at 
work now and don't have the output of fdisk to demonstrate it) so i expect the 
contents of those disks is also ok.  as i've already shown, i can't check the 
the LVM physical volumes with pvdisplay because the kernel module won't / 
can't be loaded.

is /dev/lvm just a character device which is used to transmit data from the 
volume group (vg1) to the device driver?  you mention that fsck will probably 
remove /dev/lvm and that i should recreate it.  is there any risk to vg1/lv1 
if i do that?  (as i still haven't successfully backed up anything in it's 
current state)

[linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Måns Rullgård]

Chris Doherty <chris.doherty@adelaide.edu.au> writes:

>> The filesystem containing /dev has been damaged somehow.  You should
>> fsck it.  Most likely fsck will remove /dev/lvm so you'll have to
>> recreate it with proper values.  Don't just rm it.  Something bad has
>> obviously happened and changing things without a proper fsck can make
>> things worse.
>
> thanks.  i'll try this tonight and report back. :)  my limited
> understanding of LVM leads me to believe that the volume group and
> logical volume within it are actually still safe and sound in /dev/vg1

Chances are it's still there.

> is /dev/lvm just a character device which is used to transmit data
> from the volume group (vg1) to the device driver?

Yes.

> you mention that fsck will probably remove /dev/lvm and that i
> should recreate it.  is there any risk to vg1/lv1 if i do that?  (as
> i still haven't successfully backed up anything in it's current
> state)

You should be fine.  If you are paranoid and vg1 is on different disks
than the root fs you could unplug them while doing the fsck.

-- 
M�ns Rullg�rd
mru@kth.se

[linux-lvm] Re: [linux-lvm]Re: corrupt /dev/lvm - bizzare properties

[Chris Doherty]

>> The filesystem containing /dev has been damaged somehow.  You should
>> fsck it.  Most likely fsck will remove /dev/lvm so you'll have to
>> recreate it with proper values.  Don't just rm it.  Something bad has
>> obviously happened and changing things without a proper fsck can make
>> things worse.

i'm happy to report a fsck of the root filesystem fixed the problem.  now why 
didn't i do that in the first place? :)

it didn't remove /dev/lvm though, it just repaired it.  thanks to all for 
their suggestions.

[linux-lvm] LVM Statistics

[John]

hello everyone:

  Is there anyone know about LVM IO statistics, kind of want to trace IO
statistics but from the output lvdisplay -v only shows the frequency of read
and write activities to extend, rather than the actual number of bytes that
been written and read. Any clue that I can get those figure? thanks for
suggestion before i dig into the LVM source code. thanks