[linux-lvm] Strangenesses with snapshots, 0.9.1-beta6.

[linux-lvm] Strangenesses with snapshots, 0.9.1-beta6.

[Vegard Engen]

Hello,

The two last days I started testing 0.9.1-beta6 for my home-machine, hoping to get rid of the everlasting partitioning pain. Trying out the snapshots, I get
worrying errors when making a full backup of my ~4 GB /home partition. This is
a pretty long mail demonstrating the error....

I'm using kernel 2.4.3, with ext2 filesystems on the logical volumes. The
kernel is patched with LVM 0.9.1-beta6 patches, and these patches succeeded.
I followed the instructions and made a lvm-0.9.1_beta6-2.4.3.patch - which
applied flawlessly to the kernel.

I have made the following backup-script as a test:
----- snip ------
#!/bin/sh
if [ -e /dev/backup/home ] ; then
  umount /dev/backup/home;
  lvremove -f /dev/backup/home;
fi
if [ -e /dev/vg/backup ] ; then
  umount /dev/vg/backup
  lvremove -f /dev/vg/backup
fi

lvcreate -L 1000M -s -n backup /dev/vg/nhome
mount /dev/vg/backup /mnt/backup
lvcreate -L 5000M -n home backup
mount /dev/backup/home /mnt/bhome
cp -a /mnt/backup/* /mnt/bhome
----- snip ------

I tried it out. There was very little activity at all on the system when
running this, so filling up the snapshot-volume is not an issue here.

Well, the following is output from the script. This is a little long - I'm commenting inside the output, so please read....

middagspynten:/usr/local/sbin# ./backup_home.sh 
lvcreate
lvcreate -- WARNING: the snapshot must be disabled if it gets full
lvcreate -- INFO: using default snapshot chunk size of 64 KB for "/dev/vg/backup"
lvcreate -- doing automatic backup of "vg"
lvcreate -- logical volume "/dev/vg/backup" successfully created

mount: block device /dev/vg/backup is write-protected, mounting read-only
lvcreate
lvcreate -- doing automatic backup of "backup"
lvcreate -- logical volume "/dev/backup/home" successfully created

[*** these are normal files and are indeed readable in the normal /home ***]

cp: `/mnt/backup/vegard/Mail/irc/ircd-users/356' has unknown file type
cp: `/mnt/backup/vegard/Mail/cups/343' has unknown file type
cp: `/mnt/backup/vegard/Mail/cups/344' has unknown file type
cp: `/mnt/backup/vegard/Mail/cups/346' has unknown file type

[ *** some debug *** ]
This works:
middagspynten:/home/vegard/Mail/irc/ircd-users# cat 356

This is quite strange:
middagspynten:/mnt/backup/vegard/Mail/irc/ircd-users# ls -l 356
?--sr-srw-  15370 1734492719 1868774980 4482544195540565054 Feb 17  2003 356
middagspynten:/mnt/backup/vegard/Mail/irc/ircd-users# cat 356
cat: 356: Invalid argument

[ *** End debug, continue log *** ]

cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/314': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/315': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/316': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/317': Input/output error

[ *** This is another strangeness *** ]
middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# ls -l
.....
-rw-------    1 vegard   vegard       3454 Dec 11  1999 309
-rw-------    1 vegard   vegard       4530 Sep  9  1999 31
-rw-------    1 vegard   vegard       4147 Dec 11  1999 310
-rw-------    1 vegard   vegard       3294 Dec 11  1999 311
-rw-------    1 vegard   vegard       3452 Dec 11  1999 312
-rw-------    1 vegard   vegard       5775 Dec 13  1999 313
-rw-------    1 vegard   vegard       3292 Dec 22  1999 318
-rw-------    1 vegard   vegard       4318 Dec 22  1999 319
-rw-------    1 vegard   vegard       4384 Sep 11  1999 32
.....
middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# cat 314
314: Input/output error

This works:
middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# cat 318

[ *** These are the same as the first one *** ]
cp: `/mnt/backup/vegard/Mail/inbox/234' has unknown file type
cp: `/mnt/backup/vegard/Mail/inbox/235' has unknown file type
cp: `/mnt/backup/vegard/Mail/inbox/236' has unknown file type
cp: `/mnt/backup/vegard/Mail/inbox/237' has unknown file type

[ *** This *is* actually an error in my .procmailrc and the name of the
directory ;-) - the \255 is actually an \xad, though, so it's not an \255.
But, apart from that, it's exactly like the second case *** ]

cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/233': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/234': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/235': Input/output error
cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/236': Input/output error

[ *** This is a third strangeness - and more worrying for me, because this
time, it *works* in the snapshot volume, but gets corrupted on the copy *** ]

cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/357': No such device or address
cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/358': No such device or address
cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/359': No such device or address
cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/360': No such device or address

[ **** Debugging on the above problem **** ]
middagspynten:/mnt/backup/vegard/Mail/linux/linux-net# ls -l 359
-rw-------    1 vegard   vegard       5351 Dec  6  1999 359

This file is also cat'able.

But:
middagspynten:/mnt/bhome/vegard/Mail/linux/linux-net# ls -l 359
s-wx------    1 3233857728 3233857728        0 May 17  1936 359
middagspynten:/mnt/bhome/vegard/Mail/linux/linux-net# cat 359
cat: 359: No such device or address

[ *** Here, I don't know what happened. Thie is a workable file on /mnt/backup,
but in /mnt/bhome it turns into a directory, which it later tries to
overwrite *** ]
cp: cannot overwrite directory `/mnt/bhome/vegard/Mail/linux/linux-usb/111' with non-directory

[ *** The rest is cases described before *** ]


This is looks like pretty serious bugs to me. This seemed only to happen down
in my MH-directories. For those who doesn't know, MH is a mailfolder system
which stores each mail in one file, which it names according to the message
number in the folder. However, the MH-directory compromises quite a lot of
my home-directory, too, I think I counted it to ~ 80.000 files, last I counted.
Statistically, it's still strange that it shouldn't happen in the rest of
my /home partition, as the rest isn't exactly insignificant either. So I
*think* it must be somehow related to MH or how MH folders look like. Perhaps
something with the size of directories, or the number of levels in the
directory hierarchy. But this is only guesswork.

It does, however, seem to work on the real /home, I've not yet discovered any
errors here.

- Vegard

Re: [linux-lvm] Strangenesses with snapshots, 0.9.1-beta6.

[AJ Lewis]

[-- Attachment #1: Type: text/plain, Size: 8739 bytes --]

Are you sure those files were in a consistant state when you took the
snapshot?  You really need to get your file system into a consistant state,
otherwise the metadata could be partially written out when the snapshot is
made.  There are hooks put into the 2.4x series kernels for ReiserFS to
allow it to prepare for snapshots...but i don't think ext2 uses any
mechanisms like this.  I don't know a whole lot about it, but that's what it
looks like to me.

The other thing you could do to further aid in debugging this is upgrade to
0.9.1 Beta7 (including repatching the kernel) and testing it again to see if
you get the same kind of problem.

Hopefully others will have insight into this.

Regards,
aj

On Tue, Apr 17, 2001 at 10:08:04PM +0200, Vegard Engen wrote:
> Hello,
> 
> The two last days I started testing 0.9.1-beta6 for my home-machine, hoping to get rid of the everlasting partitioning pain. Trying out the snapshots, I get
> worrying errors when making a full backup of my ~4 GB /home partition. This is
> a pretty long mail demonstrating the error....
> 
> I'm using kernel 2.4.3, with ext2 filesystems on the logical volumes. The
> kernel is patched with LVM 0.9.1-beta6 patches, and these patches succeeded.
> I followed the instructions and made a lvm-0.9.1_beta6-2.4.3.patch - which
> applied flawlessly to the kernel.
> 
> I have made the following backup-script as a test:
> ----- snip ------
> #!/bin/sh
> if [ -e /dev/backup/home ] ; then
>   umount /dev/backup/home;
>   lvremove -f /dev/backup/home;
> fi
> if [ -e /dev/vg/backup ] ; then
>   umount /dev/vg/backup
>   lvremove -f /dev/vg/backup
> fi
> 
> lvcreate -L 1000M -s -n backup /dev/vg/nhome
> mount /dev/vg/backup /mnt/backup
> lvcreate -L 5000M -n home backup
> mount /dev/backup/home /mnt/bhome
> cp -a /mnt/backup/* /mnt/bhome
> ----- snip ------
> 
> I tried it out. There was very little activity at all on the system when
> running this, so filling up the snapshot-volume is not an issue here.
> 
> Well, the following is output from the script. This is a little long - I'm commenting inside the output, so please read....
> 
> middagspynten:/usr/local/sbin# ./backup_home.sh 
> lvcreate
> lvcreate -- WARNING: the snapshot must be disabled if it gets full
> lvcreate -- INFO: using default snapshot chunk size of 64 KB for "/dev/vg/backup"
> lvcreate -- doing automatic backup of "vg"
> lvcreate -- logical volume "/dev/vg/backup" successfully created
> 
> mount: block device /dev/vg/backup is write-protected, mounting read-only
> lvcreate
> lvcreate -- doing automatic backup of "backup"
> lvcreate -- logical volume "/dev/backup/home" successfully created
> 
> [*** these are normal files and are indeed readable in the normal /home ***]
> 
> cp: `/mnt/backup/vegard/Mail/irc/ircd-users/356' has unknown file type
> cp: `/mnt/backup/vegard/Mail/cups/343' has unknown file type
> cp: `/mnt/backup/vegard/Mail/cups/344' has unknown file type
> cp: `/mnt/backup/vegard/Mail/cups/346' has unknown file type
> 
> [ *** some debug *** ]
> This works:
> middagspynten:/home/vegard/Mail/irc/ircd-users# cat 356
> 
> This is quite strange:
> middagspynten:/mnt/backup/vegard/Mail/irc/ircd-users# ls -l 356
> ?--sr-srw-  15370 1734492719 1868774980 4482544195540565054 Feb 17  2003 356
> middagspynten:/mnt/backup/vegard/Mail/irc/ircd-users# cat 356
> cat: 356: Invalid argument
> 
> [ *** End debug, continue log *** ]
> 
> cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/314': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/315': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/316': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/317': Input/output error
> 
> [ *** This is another strangeness *** ]
> middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# ls -l
> .....
> -rw-------    1 vegard   vegard       3454 Dec 11  1999 309
> -rw-------    1 vegard   vegard       4530 Sep  9  1999 31
> -rw-------    1 vegard   vegard       4147 Dec 11  1999 310
> -rw-------    1 vegard   vegard       3294 Dec 11  1999 311
> -rw-------    1 vegard   vegard       3452 Dec 11  1999 312
> -rw-------    1 vegard   vegard       5775 Dec 13  1999 313
> -rw-------    1 vegard   vegard       3292 Dec 22  1999 318
> -rw-------    1 vegard   vegard       4318 Dec 22  1999 319
> -rw-------    1 vegard   vegard       4384 Sep 11  1999 32
> .....
> middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# cat 314
> 314: Input/output error
> 
> This works:
> middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# cat 318
> 
> [ *** These are the same as the first one *** ]
> cp: `/mnt/backup/vegard/Mail/inbox/234' has unknown file type
> cp: `/mnt/backup/vegard/Mail/inbox/235' has unknown file type
> cp: `/mnt/backup/vegard/Mail/inbox/236' has unknown file type
> cp: `/mnt/backup/vegard/Mail/inbox/237' has unknown file type
> 
> [ *** This *is* actually an error in my .procmailrc and the name of the
> directory ;-) - the \255 is actually an \xad, though, so it's not an \255.
> But, apart from that, it's exactly like the second case *** ]
> 
> cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/233': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/234': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/235': Input/output error
> cp: cannot stat `/mnt/backup/vegard/Mail/linux/tecra\255linux/236': Input/output error
> 
> [ *** This is a third strangeness - and more worrying for me, because this
> time, it *works* in the snapshot volume, but gets corrupted on the copy *** ]
> 
> cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/357': No such device or address
> cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/358': No such device or address
> cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/359': No such device or address
> cp: cannot create regular file `/mnt/bhome/vegard/Mail/linux/linux-net/360': No such device or address
> 
> [ **** Debugging on the above problem **** ]
> middagspynten:/mnt/backup/vegard/Mail/linux/linux-net# ls -l 359
> -rw-------    1 vegard   vegard       5351 Dec  6  1999 359
> 
> This file is also cat'able.
> 
> But:
> middagspynten:/mnt/bhome/vegard/Mail/linux/linux-net# ls -l 359
> s-wx------    1 3233857728 3233857728        0 May 17  1936 359
> middagspynten:/mnt/bhome/vegard/Mail/linux/linux-net# cat 359
> cat: 359: No such device or address
> 
> [ *** Here, I don't know what happened. Thie is a workable file on /mnt/backup,
> but in /mnt/bhome it turns into a directory, which it later tries to
> overwrite *** ]
> cp: cannot overwrite directory `/mnt/bhome/vegard/Mail/linux/linux-usb/111' with non-directory
> 
> [ *** The rest is cases described before *** ]
> 
> 
> This is looks like pretty serious bugs to me. This seemed only to happen down
> in my MH-directories. For those who doesn't know, MH is a mailfolder system
> which stores each mail in one file, which it names according to the message
> number in the folder. However, the MH-directory compromises quite a lot of
> my home-directory, too, I think I counted it to ~ 80.000 files, last I counted.
> Statistically, it's still strange that it shouldn't happen in the rest of
> my /home partition, as the rest isn't exactly insignificant either. So I
> *think* it must be somehow related to MH or how MH folders look like. Perhaps
> something with the size of directories, or the number of levels in the
> directory hierarchy. But this is only guesswork.
> 
> It does, however, seem to work on the real /home, I've not yet discovered any
> errors here.
> 
> - Vegard
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 
AJ Lewis
Sistina Software Inc.                  Voice:  612-379-3951
1313 5th St SE, Suite 111              Fax:    612-379-3952
Minneapolis, MN 55414                  E-Mail: lewis@sistina.com
http://www.sistina.com

Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B  52A0 941E 1261 0029 2648
Get my key at: http://www.sistina.com/~lewis/gpgkey
 (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys)

-----Begin Obligatory Humorous Quote----------------------------------------
Error: Keyboard not attached. Press F1 to continue.
-----End Obligatory Humorous Quote------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Strangenesses with snapshots, 0.9.1-beta6.

[Andreas Dilger]

AJ writes:
> Are you sure those files were in a consistant state when you took the
> snapshot?  You really need to get your file system into a consistant state,
> otherwise the metadata could be partially written out when the snapshot is
> made.  There are hooks put into the 2.4x series kernels for ReiserFS to
> allow it to prepare for snapshots...but i don't think ext2 uses any
> mechanisms like this.  I don't know a whole lot about it, but that's what it
> looks like to me.

Well, before the LVM snapshot is created, it does a full sync of the device,
so the amount of metadata that could be out-of-date would be very small (if
any).

Granted, there is a possibility that the original data is corrupt, but 
Vegard reports everything OK on the original filesystem.  I assume that
e2fsck -f on the unmounted /dev/vg/nhome works OK?

> > cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/314': Input/output error
> > cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/315': Input/output error
> > cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/316': Input/output error
> > cp: cannot stat `/mnt/backup/vegard/Mail/unix/openldap-general/317': Input/output error
> > 
> > [ *** This is another strangeness *** ]
> > middagspynten:/mnt/backup/vegard/Mail/unix/openldap-general# ls -l
> > .....
> > -rw-------    1 vegard   vegard       3454 Dec 11  1999 309
> > -rw-------    1 vegard   vegard       4530 Sep  9  1999 31
> > -rw-------    1 vegard   vegard       4147 Dec 11  1999 310
> > -rw-------    1 vegard   vegard       3294 Dec 11  1999 311
> > -rw-------    1 vegard   vegard       3452 Dec 11  1999 312
> > -rw-------    1 vegard   vegard       5775 Dec 13  1999 313
> > -rw-------    1 vegard   vegard       3292 Dec 22  1999 318
> > -rw-------    1 vegard   vegard       4318 Dec 22  1999 319
> > -rw-------    1 vegard   vegard       4384 Sep 11  1999 32

The same files with IO errors are those missing from the directory listing.
Based on the file timestamps, I would assume that these files have not
been changed in a long time, so it is not a metadata consistency issue.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Strangenesses with snapshots, 0.9.1-beta6.

[AJ Lewis]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

On Wed, Apr 18, 2001 at 12:15:09PM -0600, Andreas Dilger wrote:
> The same files with IO errors are those missing from the directory listing.
> Based on the file timestamps, I would assume that these files have not
> been changed in a long time, so it is not a metadata consistency issue.

Yep, looks like you're right.  Well, so much for the easy solution.  :(

-- 
AJ Lewis
Sistina Software Inc.                  Voice:  612-379-3951
1313 5th St SE, Suite 111              Fax:    612-379-3952
Minneapolis, MN 55414                  E-Mail: lewis@sistina.com
http://www.sistina.com

Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B  52A0 941E 1261 0029 2648
Get my key at: http://www.sistina.com/~lewis/gpgkey
 (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys)

-----Begin Obligatory Humorous Quote----------------------------------------
No-one suspects the butterfly!
-----End Obligatory Humorous Quote------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]