Crash recovery after a shutdown, lvm,reiser3.6,2.4.26, lost files

Crash recovery after a shutdown, lvm,reiser3.6,2.4.26, lost files

[Michele Nicosia]

Hi all,
    yesterday, after more tha three months of hard and good work, i 
shutted off my server, to test the start script in case o remote reboot.
After the shutdown, that apparently didn't show nothing abnormal, one of 
my larger lvm, with the home's users reported an error.
At a fist seen one disk seems to be not mounted becuase the vg/pv did 
not start.
After a restore of my lvm configuration, that vg/lv started correctly.
But many errors comes out :

Sep  7 20:14:26 giove kernel: for (lvm(58,1))
Sep  7 20:14:26 giove kernel: lvm(58,1):Using r5 hash to sort names
Sep  7 20:14:26 giove kernel: is_tree_node: node level 0 does not match 
to the expected one 2
Sep  7 20:14:26 giove kernel: lvm(58,1):vs-5150: search_by_key: invalid 
format found in block 12022576. Fsck?
Sep  7 20:14:26 giove kernel: lvm(58,1):vs-2140: finish_unfinished: 
search_by_key returned -2

And so on.
The reiserfsck without option, did not solve anything.
The disk was mountable, but a lot of damaged link insted of normal files 
were in many directory.
THe last thing to test was --rebuild-tree, that solved the problems of 
the damaged files, but not like i thinked, it was created a lost+found 
dir with a lot of  file, more than 30.000 and subdir.
I do not understand what happens, with an old server, 2.2.X and reiserfs 
3.5 without lvm, i switched of a lot of time with some partition of more 
than 100 g and NEVER a file was losted.
I forgot to say that into the lost+found i have only  file of  the 
courier maildir, but some well know  file, some avi's of 400mega, was 
completely lost!!
Why this happens?
Is the couple lvm-reiser less solid than reiser directly on a partition?

Thanks for any info, and sorry for the bad english.

Michele

Re: Crash recovery after a shutdown, lvm,reiser3.6,2.4.26, lost files

[Christian Mayrhuber]

On Wednesday 08 September 2004 14:43, Michele Nicosia wrote:
> Hi all,
>     yesterday, after more tha three months of hard and good work, i 
> shutted off my server, to test the start script in case o remote reboot.
> After the shutdown, that apparently didn't show nothing abnormal, one of 
> my larger lvm, with the home's users reported an error.
> At a fist seen one disk seems to be not mounted becuase the vg/pv did 
> not start.
> After a restore of my lvm configuration, that vg/lv started correctly.
> But many errors comes out :
> 
> Sep  7 20:14:26 giove kernel: for (lvm(58,1))
> Sep  7 20:14:26 giove kernel: lvm(58,1):Using r5 hash to sort names
> Sep  7 20:14:26 giove kernel: is_tree_node: node level 0 does not match 
> to the expected one 2
> Sep  7 20:14:26 giove kernel: lvm(58,1):vs-5150: search_by_key: invalid 
> format found in block 12022576. Fsck?
> Sep  7 20:14:26 giove kernel: lvm(58,1):vs-2140: finish_unfinished: 
> search_by_key returned -2
> 
> And so on.
> The reiserfsck without option, did not solve anything.
> The disk was mountable, but a lot of damaged link insted of normal files 
> were in many directory.
> THe last thing to test was --rebuild-tree, that solved the problems of 
> the damaged files, but not like i thinked, it was created a lost+found 
> dir with a lot of  file, more than 30.000 and subdir.
> I do not understand what happens, with an old server, 2.2.X and reiserfs 
> 3.5 without lvm, i switched of a lot of time with some partition of more 
> than 100 g and NEVER a file was losted.
> I forgot to say that into the lost+found i have only  file of  the 
> courier maildir, but some well know  file, some avi's of 400mega, was 
> completely lost!!
> Why this happens?
> Is the couple lvm-reiser less solid than reiser directly on a partition?
> 
> Thanks for any info, and sorry for the bad english.
> 
> Michele

1)
I once had such an insult on a machine I was playing around and some old
2.4.x kernel + ext2. (rebooting after partitioning did away with these 
problems)
I may be wrong, but there is the possibility that your lvm configuration was 
not quite right because you repartitioned your harddisk and created the lvm 
configuration without rebooting after repartitioning. This can happen because 
the kernel may not have read the correct partition table info. Thats why 
fdisk suggests rebooting. (At least there was a time it was necessary)
Your reboot after 3 months made the kernel read the correct
partition table and resulted for lvm to fail, because it's disk information 
did not match the kernel one.

2)
One of your harddisks could be defect. Try to run the badblocks command.

3)
Your RAM could be defect. Try to run memtest86 for about 2 days.

Note: ReiserFS has never failed for me on LVM and 2.4.x kernels (x >= 18,
I didn't use the combination lvm+reiserfs on earlier kernels) 

-- 
lg, Chris

Re: Crash recovery after a shutdown, lvm,reiser3.6,2.4.26, lost files

[Michele Nicosia]

Christian Mayrhuber wrote:

> 1)
>
>I once had such an insult on a machine I was playing around and some old
>2.4.x kernel + ext2. (rebooting after partitioning did away with these 
>problems)
>  
>
Hi,
    the primary idea starting play with reiserfs is that on crash the 
journal will save the job...
It is not the same of ext2 fs.

>I may be wrong, but there is the possibility that your lvm configuration was 
>not quite right because you repartitioned your harddisk and created the lvm 
>configuration without rebooting after repartitioning. This can happen because 
>the kernel may not have read the correct partition table info. Thats why 
>fdisk suggests rebooting. (At least there was a time it was necessary)
>Your reboot after 3 months made the kernel read the correct
>partition table and resulted for lvm to fail, because it's disk information 
>did not match the kernel one.
>
>  
>
After the partitionig i restarted the machine (i also founded the limti 
of 15 partition), but i do not restarted after i add another pv to the 
volume group e resized the volume, like i suppose is the thing normal to do.
Except this i can suppose tha another thing about th elvm could be 
worng, but it is not the issue, because after recover the pv, and 
mounted the fs all the files, many of them, were corrupted!!


>2)
>One of your harddisks could be defect. Try to run the badblocks command.
>
>  
>
smart array 641 with six ultra3 scsi drive, i think it is not a disk 
failure.

>3)
>Your RAM could be defect. Try to run memtest86 for about 2 days.
>
>  
>
e.c.c. 1,5 g on proliant G3 dual xeon, also this, i suppose the ram is 
good, not sure , i'm not tested it, but the probabilyt is low..

>Note: ReiserFS has never failed for me on LVM and 2.4.x kernels (x >= 18,
>I didn't use the combination lvm+reiserfs on earlier kernels) 
>  
>

2.4.26  > 2.4.18 and the lvm with reiserfs is the most improtnat and 
useful thing to use with linux, IMHO....

Thanks

Re: Crash recovery after a shutdown, lvm,reiser3.6,2.4.26, lost files

[Christian Mayrhuber]

On Wednesday 08 September 2004 16:52, Michele Nicosia wrote:
> Hi,
>     the primary idea starting play with reiserfs is that on crash the 
> journal will save the job...
> It is not the same of ext2 fs.
ReiserFS will not save your data, just the file metadata during a crash.
If you want it to save your data you need to use the 
data logging patches from SuSE and use the data=ordered or data=journal
mount options.
ftp://ftp.suse.com/pub/people/mason/patches/data-logging/2.4.25/
The patches for 2.4.25 work on 2.4.26 and 2.4.27, too.
Reiserfs cannot help in cases when the blocks are mixed somehow by
lvm.

> After the partitionig i restarted the machine (i also founded the limti 
> of 15 partition), but i do not restarted after i add another pv to the 
> volume group e resized the volume, like i suppose is the thing normal to do.
Yes, it is.

> Except this i can suppose tha another thing about th elvm could be 
> worng, but it is not the issue, because after recover the pv, and 
> mounted the fs all the files, many of them, were corrupted!!
You did rebuild the vg the right way for sure? I mean same order of pv's,
same extent size, etc.
reiserfsck --rebuild-tree is rather good at finding files even if lvm 
got rebuilt wrongly. 
reiserfsck --scan-whole-partition --rebuild-tree may even find more files,
maybe the 400MB one you are missing.
The only answer to such a big corruption you have experienced I have is that 
either the vg's are not rebuilt the right way or that you have a hardware 
defect, presumably some harddisk or the controller.
ReiserFS didn't have a report of such a desaster for a very long time. I guess
it's more than a year, now.

> >2)
> >One of your harddisks could be defect. Try to run the badblocks command.
> >
> >  
> >
> smart array 641 with six ultra3 scsi drive, i think it is not a disk 
> failure.
Thats no guarantee.
I once had a defect AMI Megaraid controller not detecting faulty disks, 
not even the ones with write errors. The result of reiserfschk --rebuild-tree
with two failed disks in a Raid-5 array was similiar to your experience.
Lots of files in lost+found with around 20% of the files containing corrupt 
data.

> 
> 2.4.26  > 2.4.18 and the lvm with reiserfs is the most improtnat and 
> useful thing to use with linux, IMHO....
Yes, I like the online expansion feature, too.

-- 
lg, Chris