[linux-lvm] File Corruption on LVM LV's in same VG

[linux-lvm] File Corruption on LVM LV's in same VG

[Erik Zweers]

I've been using LVM on a test system for a couple of months now, and
sofar have had no problems.  We have two VG's, one made up of multiple
SCSI drives and one consisting of a single large IDE drive.  The IDE VG
is used for more storage, while the SCSI VG is used for more active
applications (cyrus imap, home directores etc).

A few weeks ago the IDE VG was destroyed by attempting to install LILO
on the drive (there was confusion over how the system was setup).  The
system was also configured with the IDE as slave with no master.

The system was repaired and believed to be running properly, until
yesterday where we discovered that some of our ISO's were wrong,
consistently wrong in inconsistent locations.  I've pasted in a record
of the symptoms.  If anyone has any advice on what else to diagnose (if
your intrested in what is wrong) or how to repair It would be greatly
appreciated.

I have not rebooted the machine yet, wanting to be sure that any
diagnoses that can be done is done.

-----------------------------------------------------------------

in /var/export/iso
LV /dev/ide/iso
VG ide

[root@ike iso]# md5sum enigma-i386-disc1.iso enigma-i386-disc1.iso
enigma-i386-disc1.iso
83edcc827cdc3db2b4b0237bab59b1dd  enigma-i386-disc1.iso
13b425cda141e3263be34a37f237b410  enigma-i386-disc1.iso
5b4ade7fcc64bf587c266935ff9e4a2d  enigma-i386-disc1.iso
[root@ike iso]# md5sum seawolf-i386-disc1.iso seawolf-i386-disc1.iso
10bd7359091e3ce93b6b8da8eaed7afe  seawolf-i386-disc1.iso
e5ef0aa5922e45d39efffe04e904e739  seawolf-i386-disc1.iso

in /var/export/mp3/zweers
LV /dev/ide/mp3
VG ide

[root@ike zweers]# md5sum enigma-i386-disc1.iso enigma-i386-disc1.iso
18fd4da5a7b33e1598e88ff717aabbe6  enigma-i386-disc1.iso
b0a11d0a5efa523e136a0b5e2157334d  enigma-i386-disc1.iso

[root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
2549832  77  67
13431928 177 167
14489440 376 366
27451404 177 173
33149724 376 336
86298228 177 167
116907524 372 332
119130964  77  67
121124171 376 356


[root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
2549832  77  67
11455071 177 167
13431928 177 167
18043044 277 273
24780964 277 273
86298228 177 167
94133044 370 330
106966431 177 167


[root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
| wc -l     62
[root@ike iso]# ls -l enigma-i386-disc1.iso
-rw-r--r--    1 root     root     677961728 Feb  4 20:31
enigma-i386-disc1.iso

[root@ike iso]# fsck -n /dev/ide/iso
Parallelizing fsck version 1.23 (15-Aug-2001)
e2fsck 1.23, 15-Aug-2001 for EXT2 FS 0.5b, 95/08/09
Warning!  /dev/ide/iso is mounted.
/dev/ide/iso: clean, 24/1281696 files, 1351397/2560000 blocks

Re: [linux-lvm] File Corruption on LVM LV's in same VG

[Andreas Dilger]

On Feb 05, 2002  13:11 -0500, Erik Zweers wrote:
> The system was repaired and believed to be running properly, until
> yesterday where we discovered that some of our ISO's were wrong,
> consistently wrong in inconsistent locations.  I've pasted in a record
> of the symptoms.  If anyone has any advice on what else to diagnose (if
> your intrested in what is wrong) or how to repair It would be greatly
> appreciated.

I would suspect RAM or IDE problems.  If LVM is broken, it will stay
broken, and if it is working it will stay working in the absence of
other problems.  The failure symptoms would not look like this.

> I have not rebooted the machine yet, wanting to be sure that any
> diagnoses that can be done is done.

Any output in dmesg?

> [root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
> 2549832  77  67
> 13431928 177 167
> 14489440 376 366
> 27451404 177 173
> 33149724 376 336
> 86298228 177 167
> 116907524 372 332
> 119130964  77  67
> 121124171 376 356
> 
> 
> [root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
> 2549832  77  67
> 11455071 177 167
> 13431928 177 167
> 18043044 277 273
> 24780964 277 273
> 86298228 177 167
> 94133044 370 330
> 106966431 177 167

Looks exclusively like single-bit errors.  Either RAM or IDE cable/driver.
If it was RAM you would also have oopses or errors elsewhere.  Try turning
off UDMA on the drive, update the IDE driver, new cable, etc.

> | wc -l     62
> [root@ike iso]# ls -l enigma-i386-disc1.iso
> -rw-r--r--    1 root     root     677961728 Feb  4 20:31
> enigma-i386-disc1.iso
> 
> [root@ike iso]# fsck -n /dev/ide/iso
> Parallelizing fsck version 1.23 (15-Aug-2001)
> e2fsck 1.23, 15-Aug-2001 for EXT2 FS 0.5b, 95/08/09
> Warning!  /dev/ide/iso is mounted.
> /dev/ide/iso: clean, 24/1281696 files, 1351397/2560000 blocks

Given that you only have single-bit error rates in the order of 1/10^7,
the chance of such an error happening in the ext2 metadata is unlikely.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] File Corruption on LVM LV's in same VG

[Erik Zweers]

There are no error messages on the console.  This box does do a fair
amount of work and has no other issues.  The box has a second VG
consisting of SCSI drives.  If there were memory problems I would expect
this VG to contain errors too.

I did a MD5sum on the physical device twice and it returned the same sum
(while doing this on a file in a filesystem under LVM returns different
values each time).

Are there any good indepth tools that could analyze the structure of the
volume?  Everything I've seen sofar seems to just offer a summary.  The
box still has not been rebooted as I want to understand the problem
first.  I'm looking at using LVM for a production server and need to
find out whats wrong to go ahead with that project.



On Tue, 2002-02-05 at 13:41, Andreas Dilger wrote:
> On Feb 05, 2002  13:11 -0500, Erik Zweers wrote:
> > The system was repaired and believed to be running properly, until
> > yesterday where we discovered that some of our ISO's were wrong,
> > consistently wrong in inconsistent locations.  I've pasted in a record
> > of the symptoms.  If anyone has any advice on what else to diagnose (if
> > your intrested in what is wrong) or how to repair It would be greatly
> > appreciated.
> 
> I would suspect RAM or IDE problems.  If LVM is broken, it will stay
> broken, and if it is working it will stay working in the absence of
> other problems.  The failure symptoms would not look like this.
> 
> > I have not rebooted the machine yet, wanting to be sure that any
> > diagnoses that can be done is done.
> 
> Any output in dmesg?
> 
> > [root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
> > 2549832  77  67
> > 13431928 177 167
> > 14489440 376 366
> > 27451404 177 173
> > 33149724 376 336
> > 86298228 177 167
> > 116907524 372 332
> > 119130964  77  67
> > 121124171 376 356
> > 
> > 
> > [root@ike iso]# cmp -l enigma-i386-disc1.iso ~carl/enigma-i386-disc1.iso
> > 2549832  77  67
> > 11455071 177 167
> > 13431928 177 167
> > 18043044 277 273
> > 24780964 277 273
> > 86298228 177 167
> > 94133044 370 330
> > 106966431 177 167
> 
> Looks exclusively like single-bit errors.  Either RAM or IDE cable/driver.
> If it was RAM you would also have oopses or errors elsewhere.  Try turning
> off UDMA on the drive, update the IDE driver, new cable, etc.
> 
> > | wc -l     62
> > [root@ike iso]# ls -l enigma-i386-disc1.iso
> > -rw-r--r--    1 root     root     677961728 Feb  4 20:31
> > enigma-i386-disc1.iso
> > 
> > [root@ike iso]# fsck -n /dev/ide/iso
> > Parallelizing fsck version 1.23 (15-Aug-2001)
> > e2fsck 1.23, 15-Aug-2001 for EXT2 FS 0.5b, 95/08/09
> > Warning!  /dev/ide/iso is mounted.
> > /dev/ide/iso: clean, 24/1281696 files, 1351397/2560000 blocks
> 
> Given that you only have single-bit error rates in the order of 1/10^7,
> the chance of such an error happening in the ext2 metadata is unlikely.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> 
>