* Fw: BUG: Files corrupt after moving LVM volume to USB disk
@ 2007-03-22 8:03 Andrew Morton
2007-03-22 15:33 ` Alasdair G Kergon
0 siblings, 1 reply; 6+ messages in thread
From: Andrew Morton @ 2007-03-22 8:03 UTC (permalink / raw)
To: dm-devel
ho hum I'm getting rather bored of redirecting emails to the correct lists.
Does anyone there read linux-kernel?
Begin forwarded message:
Date: Tue, 20 Mar 2007 12:34:08 +0200
From: "Marti Raudsepp" <marti@juffo.org>
To: "Kernel hackers" <linux-kernel@vger.kernel.org>, fsdevel <linux-fsdevel@vger.kernel.org>
Subject: BUG: Files corrupt after moving LVM volume to USB disk
Hello LKML,
Second repost of "BUG: Killing and reviving files with USB disks", this time
also destined to linux-fsdevel.
This is a reproducible demonstration of the problem initially reported in my
first e-mail, titled "PROBLEM: 'bio too big device' after moving to a USB
disk" (http://lkml.org/lkml/2007/3/7/657)
Summary:
01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount
02. Populate the disk with files; sync; flush caches
03. Confirm that the files are still readable and non-corrupt (sha1sum)
04. Migrate logical volume to USB disk; sync; flush caches
05. Confirm that ALL PREVIOUSLY VERIFIED FILES ARE CORRUPT
06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
07. Run reiserfsck to check for corruptions -- none.
08. Migrate logical volume back to SATA disk; sync; flush caches
09. Confirm that files are readable and non-corrupt again
10. Clean up
Environment/configuration:
* Linux non 2.6.19-gentoo-r6 #1 Wed Feb 28 20:55:24 EET 2007 x86_64 AMD
Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
* USB disk 120GB Western Digital, model 00UE-00KVT0 (according to udev),
serial DEF10CD7F64C
* Older SATA disk 200GB Seagate 7200.7, model ST3200822AS
* Motherboard Asus A8N5X, nForce4 chipset
* Offending file system: reiserfs v3.6, mounted with noatime,barrier=flush
* dm-crypt using aes-256 with cbc-essiv:sha256; using assembly-optimized AES
on x86_64 (CONFIG_CRYPTO_AES_X86_64)
* Test partition is 68 extents times 16MiB = 1088 MiB large (that's the
largest I could allocate while remaining in one segment -- otherwise lvmove
wouldn't move the partition back to /dev/sda5 after "defragmenting" it.)
* LVM utilities version: 2.02.17 (2006-12-14)
* LVM library version: 1.02.12 (2006-10-13)
* LVM driver version: 4.10.0
* cryptsetup-luks 1.0.4 (user space interface to dm-crypt)
Involved block devices:
* /dev/sda: My old SATA disk.
* /dev/sda5: The LVM partition on the old disk.
* /dev/sdb: The new offending USB disk; whole disk is used as an LVM physical
volume.
* /dev/primary: LVM volume group 'primary', consisting of /dev/sdb and
/dev/sda5
* /dev/primary/punchbag: LVM volume 'punchbag' for demonstration purposes.
* /dev/mapper/crypt-punchbag: The dm-crypt "decrypted" loopback device.
----
00. PV information
[non]# pvdisplay /dev/sda5
--- Physical volume ---
PV Name /dev/sda5
VG Name primary
PV Size 145.83 GB / not usable 2.73 MB
Allocatable yes
PE Size (KByte) 16384
Total PE 9333
Free PE 117
Allocated PE 9216
PV UUID YdrDuL-jw5z-J0SA-EEKU-NOC4-6gGR-T90YCA
[non]# pvdisplay /dev/sdb
--- Physical volume ---
PV Name /dev/sdb
VG Name primary
PV Size 111.79 GB / not usable 9.46 MB
Allocatable yes
PE Size (KByte) 16384
Total PE 7154
Free PE 7129
Allocated PE 25
PV UUID nE8C5L-lfI1-VNgs-Q7ei-1Zz3-GeGT-UNhH4p
----
01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount
[non]# lvcreate -n punchbag --extents 68 primary /dev/sda5
Logical volume "punchbag" created
[non]# lvs --segments -o +devices
LV VG Attr #Str Type SSize Devices
[...]
punchbag primary -wi-a- 1 linear 1.06G /dev/sda5(0)
[...]
[non]# cryptsetup luksFormat /dev/primary/punchbag -c
aes-cbc-essiv:sha256 -h sha1
[...]
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.
[non]# cryptsetup luksOpen /dev/primary/punchbag crypt-punchbag
Enter LUKS passphrase:
key slot 0 unlocked.
Command successful.
[non]# mkfs.reiserfs /dev/mapper/crypt-punchbag
[...]
Guessing about desired format.. Kernel 2.6.19-gentoo-r6 is running.
Format 3.6 with standard journal
Count of blocks on the device: 343920
Number of blocks consumed by mkreiserfs formatting process: 8222
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: c1857208-5090-49dd-9163-9fb002d96a88
ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
ALL DATA WILL BE LOST ON '/dev/mapper/crypt-punchbag'!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
[...]
ReiserFS is successfully created on /dev/mapper/crypt-punchbag.
[non]# mkdir /mnt/punchbag
[non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o noatime,barrier=flush
----
02. Populate the disk with files; sync; flush caches
[non]# cp -r ~marti/mp3 /mnt/punchbag
[... yes, these are legal mp3s ;)]
cp: writing `/mnt/punchbag/mp3/...': No space left on device
^C
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches
[non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways
----
03. Confirm that the files are still readable and non-corrupt (sha1sum)
[non]# sha1sum -c *.sha1
mr Epic - Sideways - 01. Down Low.flac: OK
mr Epic - Sideways - 02. Blue Days.flac: OK
mr Epic - Sideways - 03. Sift.flac: OK
mr Epic - Sideways - 04. Slipping.flac: OK
mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
mr Epic - Sideways - 06. In.flac: OK
mr Epic - Sideways - 07. Out.flac: OK
----
04. Migrate logical volume to USB disk; sync; flush caches
[non]# pvmove -n punchbag /dev/sda5 /dev/sdb
[...]
/dev/sda5: Moved: 100.0%
[non]# lvs --segments -o +devices
LV VG Attr #Str Type SSize Devices
[...]
punchbag primary -wi-ao 1 linear 1.06G /dev/sdb(25)
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches
----
05. Confirm that all previously verified files are corrupt
[non]# sha1sum -c *.sha1
sha1sum: mr Epic - Sideways - 01. Down Low.flac: Input/output error
mr Epic - Sideways - 01. Down Low.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 02. Blue Days.flac: Input/output error
mr Epic - Sideways - 02. Blue Days.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 03. Sift.flac: Input/output error
mr Epic - Sideways - 03. Sift.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 04. Slipping.flac: Input/output error
mr Epic - Sideways - 04. Slipping.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 05. Ruff and Tumble.flac: Input/output error
mr Epic - Sideways - 05. Ruff and Tumble.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 06. In.flac: Input/output error
mr Epic - Sideways - 06. In.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 07. Out.flac: Input/output error
mr Epic - Sideways - 07. Out.flac: FAILED open or read
sha1sum: WARNING: 7 of 7 listed files could not be read
----
06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
[non]# dmesg |tail -n7
[101403.106825] bio too big device dm-0 (256 > 240)
[101403.115228] bio too big device dm-0 (256 > 240)
[101403.115912] bio too big device dm-0 (256 > 240)
[101403.130529] bio too big device dm-0 (256 > 240)
[101403.131220] bio too big device dm-0 (256 > 240)
[101403.141577] bio too big device dm-0 (256 > 240)
[101403.142255] bio too big device dm-0 (256 > 240)
----
07. Run reiserfsck to check for corruptions -- none.
[non]# cd
[non]# umount /mnt/punchbag
[non]# reiserfsck --check /dev/mapper/crypt-punchbag
[...]
Will read-only check consistency of the filesystem on
/dev/mapper/crypt-punchbag
Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Sat Mar 10 19:00:08 2007
###########
Replaying journal..
Reiserfs journal '/dev/mapper/crypt-punchbag' in blocks [18..8211]: 0
transactions replayed
Checking Internal tree...
0%....20%....40%....60%....80%....100%
Checking Semantic tree...
There are on the filesystem:
Leaves 282
Internal nodes 3
Directories 18
Other files 196
Data block pointers 268374 (0 of them are zero)
Safe links 0
No corruptions found
###########
reiserfsck finished at Sat Mar 10 19:00:10 2007
###########
----
08. Migrate logical volume back to SATA disk; sync; flush caches
[non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o noatime,barrier=flush
[non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways
[non]# pvmove -n punchbag /dev/sdb /dev/sda5
[...]
/dev/sdb: Moved: 100.0%
[non]# lvs --segments -o +devices
LV VG Attr #Str Type SSize Devices
[...]
punchbag primary -wi-a- 1 linear 1.06G /dev/sda5(0)
[...]
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches
----
09. Confirm that files are readable and non-corrupt again
[non]# sha1sum -c *.sha1
mr Epic - Sideways - 01. Down Low.flac: OK
mr Epic - Sideways - 02. Blue Days.flac: OK
mr Epic - Sideways - 03. Sift.flac: OK
mr Epic - Sideways - 04. Slipping.flac: OK
mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
mr Epic - Sideways - 06. In.flac: OK
mr Epic - Sideways - 07. Out.flac: OK
----
10. Clean up
[non]# cd
[non]# umount /mnt/punchbag
[non]# cryptsetup remove crypt-punchbag
[non]# lvremove primary/punchbag
Do you really want to remove active logical volume "punchbag"? [y/n]: y
Logical volume "punchbag" successfully removed
I repeated this five times and each time was able to reproduce the problem.
More details on the symptoms of the problem in my previous problem report:
http://lkml.org/lkml/2007/3/7/657
I'm not subscribed to the list, so please keep me in Cc:.
Regards,
Marti Raudsepp
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Fw: BUG: Files corrupt after moving LVM volume to USB disk
2007-03-22 8:03 Fw: BUG: Files corrupt after moving LVM volume to USB disk Andrew Morton
@ 2007-03-22 15:33 ` Alasdair G Kergon
2007-03-23 0:42 ` [dm-devel] " Marti Raudsepp
0 siblings, 1 reply; 6+ messages in thread
From: Alasdair G Kergon @ 2007-03-22 15:33 UTC (permalink / raw)
To: Marti Raudsepp; +Cc: dm-devel, Andrew Morton
On Thu, Mar 22, 2007 at 12:03:48AM -0800, Andrew Morton wrote:
> ho hum I'm getting rather bored of redirecting emails to the correct lists.
> Does anyone there read linux-kernel?
I for one generally only spot things with 'lvm' or device-mapper/dm in
the subject line. The original subject line included no clue this one
had anything to do with device-mapper:-(
A couple of patches to try:
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/2.6.19/dm-io-fix-bi_max_vecs.patch
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-merge-max_hw_sector.patch
and perhaps these three:
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-crypt-fix-call-to-clone_init.patch
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-crypt-fix-avoid-cloned-bio-ref-after-free.patch
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-crypt-fix-remove-first_clone.patch
Alasdair
--
agk@redhat.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dm-devel] Fw: BUG: Files corrupt after moving LVM volume to USB disk
2007-03-22 15:33 ` Alasdair G Kergon
@ 2007-03-23 0:42 ` Marti Raudsepp
2007-03-23 14:45 ` Milan Broz
0 siblings, 1 reply; 6+ messages in thread
From: Marti Raudsepp @ 2007-03-23 0:42 UTC (permalink / raw)
To: Alasdair G Kergon, Lennart Sorensen
Cc: device-mapper development, Kernel hackers, Milan Broz,
Andrew Morton
Summary of what I've managed to rule out so far:
1. this problem does not occur without dm-crypt
2. this problem does not occur when the file system hasn't been moved from
the SATA disk to USB
3. Both reiserfs and ext3 are affected by this problem, but ext3
merely slows down.
4. Unmounting and re-mounting has no effect.
My initial report contains more details.
On 3/22/07, Alasdair G Kergon <agk@redhat.com> wrote:
> A couple of patches to try:
> dm-io-fix-bi_max_vecs.patch
> dm-merge-max_hw_sector.patch
>
> and perhaps these three:
> dm-crypt-fix-call-to-clone_init.patch
> dm-crypt-fix-avoid-cloned-bio-ref-after-free.patch
> dm-crypt-fix-remove-first_clone.patch
No luck. :(
And thanks for reminding me how annoying reboots are...
On 3/21/07, Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:
> Does this happen only with this combination, or can you elliminate
> something as the cause?
Doesn't occur if dm-crypt is not involved (e.g., when placing reiserfs or
ext3 straight on top of /dev/primary/punchbag). I don't know if LVM is
necessary, as I don't have any unallocated space for "oldschool" partitions
left.
Nor do these problems occur when creating either file system directly on the
USB disk -- so, only when they have been lvmoved from the SATA disk, or when
the volume is physically dd'ed to the USB disk.
On 3/21/07, Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:
> Does it happen with ext3 or only reiserfs?
Tried with ext3 (LVM+dm-crypt+ext3), and the files didn't appear corrupt,
however, I could still see those messages in my dmesg, and disk access was
*extremely* slow, causing around 10k context switches per second (!?) reading
below 5 MB/s. Normally, I can get read speeds around 12-22 MB/s with this
particular USB disk. However, it's not CPU-bound, as over >80% of CPU is still
spent in iowait.
When the ext3 file system is created directly on a volume that is located on
the USB disk, however, these problems do not occur, just like with reiserfs.
However, when I copied the same files to a reiserfs volume when the LV had
already been pvmoved to /dev/sdb, the files were readable (as pointed out in
my initial post). Interestingly, it did not introduce the same slowdown as
with ext3, reading around 13-14 MB/s with ~3k context switches per second.
To clarify (I double-checked all these scenarios):
(1) files written *before* pvmove are corrupt with reiserfs
(2) files written *after* pvmove read fast with reiserfs
(3) files written *before* pvmove read slowly with ext3
(4) files written *after* pvmove read slowly with ext3
(5) files written to an ext3 volume that was formatted on the USB disk and not
pvmoved are fast and don't report these dmesg warnings.
This is purely my speculation, and I can't pretend to know much about the
inner workings of device-mapper/file systems, but it appears that ext3
statically derives some attributes during the creation of a file system,
while reiserfs gets them runtime. ext3 contains a slow workaround if these
attributes don't match (and still posts a warning), but reiserfs
returns outright I/O errors.
(P.S. does the LKML archive randomly omit some double-newlines in posts, or
is this a problem on my end?)
Marti Raudsepp
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] disable barriers in dm-crypt [was Re: BUG: Files corrupt after moving LVM volume to USB disk]
2007-03-23 0:42 ` [dm-devel] " Marti Raudsepp
@ 2007-03-23 14:45 ` Milan Broz
0 siblings, 0 replies; 6+ messages in thread
From: Milan Broz @ 2007-03-23 14:45 UTC (permalink / raw)
To: Marti Raudsepp
Cc: Christophe Saout, Kernel hackers, device-mapper development,
Lennart Sorensen, Andrew Morton, Alasdair G Kergon
> Summary of what I've managed to rule out so far:
>1. this problem does not occur without dm-crypt
Please could you test attached patch ?
Sorry for another annoying reboot :-)
Milan
--
mbroz@redhat.com
From: Milan Broz <mbroz@redhat.com>
Disable barriers in dm-crypt because of current workqueue processing
can reorder requests.
This must be addresed later but for now disabling barriers is needed
to prevent data corruption.
Signed-off-by: Milan Broz <mbroz@redhat.com>
Index: linux-2.6.20.1/drivers/md/dm-crypt.c
===================================================================
--- linux-2.6.20.1.orig/drivers/md/dm-crypt.c 2007-03-23 15:07:11.000000000 +0100
+++ linux-2.6.20.1/drivers/md/dm-crypt.c 2007-03-23 15:09:59.000000000 +0100
@@ -925,6 +925,9 @@ static int crypt_map(struct dm_target *t
struct crypt_config *cc = ti->private;
struct crypt_io *io;
+ if (bio_barrier(bio))
+ return -EOPNOTSUPP;
+
io = mempool_alloc(cc->io_pool, GFP_NOIO);
io->target = ti;
io->base_bio = bio;
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] disable barriers in dm-crypt [was Re: BUG: Files corrupt after moving LVM volume to USB disk]
@ 2007-03-23 14:45 ` Milan Broz
0 siblings, 0 replies; 6+ messages in thread
From: Milan Broz @ 2007-03-23 14:45 UTC (permalink / raw)
To: Marti Raudsepp
Cc: Alasdair G Kergon, Lennart Sorensen, device-mapper development,
Kernel hackers, Andrew Morton, Christophe Saout
> Summary of what I've managed to rule out so far:
>1. this problem does not occur without dm-crypt
Please could you test attached patch ?
Sorry for another annoying reboot :-)
Milan
--
mbroz@redhat.com
From: Milan Broz <mbroz@redhat.com>
Disable barriers in dm-crypt because of current workqueue processing
can reorder requests.
This must be addresed later but for now disabling barriers is needed
to prevent data corruption.
Signed-off-by: Milan Broz <mbroz@redhat.com>
Index: linux-2.6.20.1/drivers/md/dm-crypt.c
===================================================================
--- linux-2.6.20.1.orig/drivers/md/dm-crypt.c 2007-03-23 15:07:11.000000000 +0100
+++ linux-2.6.20.1/drivers/md/dm-crypt.c 2007-03-23 15:09:59.000000000 +0100
@@ -925,6 +925,9 @@ static int crypt_map(struct dm_target *t
struct crypt_config *cc = ti->private;
struct crypt_io *io;
+ if (bio_barrier(bio))
+ return -EOPNOTSUPP;
+
io = mempool_alloc(cc->io_pool, GFP_NOIO);
io->target = ti;
io->base_bio = bio;
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] disable barriers in dm-crypt [was Re: BUG: Files corrupt after moving LVM volume to USB disk]
2007-03-23 14:45 ` Milan Broz
(?)
@ 2007-03-26 0:05 ` Marti Raudsepp
-1 siblings, 0 replies; 6+ messages in thread
From: Marti Raudsepp @ 2007-03-26 0:05 UTC (permalink / raw)
To: Milan Broz
Cc: Alasdair G Kergon, Lennart Sorensen, device-mapper development,
Kernel hackers, Andrew Morton, Christophe Saout
On 3/23/07, Milan Broz <mbroz@redhat.com> wrote:
> Disable barriers in dm-crypt because of current workqueue processing
> can reorder requests.
>
> --- linux-2.6.20.1.orig/drivers/md/dm-crypt.c 2007-03-23 15:07:11.000000000 +0100
> +++ linux-2.6.20.1/drivers/md/dm-crypt.c 2007-03-23 15:09:59.000000000 +0100
> + if (bio_barrier(bio))
> + return -EOPNOTSUPP;
> +
Sorry, this doesn't help either. ext3 is still slow, reiserfs still fails.
(And sorry for the double e-mail, I forgot to reply to all)
Marti Raudsepp
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2007-03-26 0:05 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-03-22 8:03 Fw: BUG: Files corrupt after moving LVM volume to USB disk Andrew Morton
2007-03-22 15:33 ` Alasdair G Kergon
2007-03-23 0:42 ` [dm-devel] " Marti Raudsepp
2007-03-23 14:45 ` [PATCH] disable barriers in dm-crypt [was Re: BUG: Files corrupt after moving LVM volume to USB disk] Milan Broz
2007-03-23 14:45 ` Milan Broz
2007-03-26 0:05 ` Marti Raudsepp
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.