Re: Possible RBD inconsistencies with kvm+Windows 7

[Josh Pieper <jjp@pobox.com>]

Josh Durgin wrote:
> On 02/03/2012 10:19 AM, Josh Pieper wrote:
> >I have a Windows 7 guest running under kvm/libvirt with RBD as a
> >backend to a cluster of 3 OSDs.  With this setup, I am seeing behavior
> >that looks suspiciously like disk corruption in the guest VM executing
> >some of our workloads.
> >
> >For instance, in one occurance, there is a python function that
> >recursively deletes a large directory tree while the disk is otherwise
> >loaded.  For us, this occasionally fails because the OS reported that
> >all the files in the directory were deleted, but then reports the
> >directory is not empty when going to remove it.  In another, a simple
> >test application writes new files to a directory every 50ms, then
> >after 6s verifies that at least 3 files were written, also while the
> >disk is under heavy load.
> >
> >We have never ever seen these failures on bare metal, or on kvm
> >instances backed by a LVM volume in years of operation, but they
> >happen every couple of hours with RBD.  Unfortunately, I have been
> >unsuccessful when attempting to create synthetic test cases to
> >demonstrate the inconsistent RBD behavior.
> >
> >Has anyone else seen similar inconsistent RBD behavior, or have ideas
> >how to diagnose further?
> 
> What fs are your osds using? A while ago there was a bug in ext4's
> fiemap that sometimes caused incorrect reads - if you set
> filestore_fiemap_threshold larger than your object size, you can test
> whether fiemap is the problem.

The OSDs are using xfs.  In my testing with 0.40, btrfs had incredible
performance problems after a day or so of operation.  The last I
heard, ext4 could potentially have data loss due to its limited xattr
support.

> Are you using the rbd_writeback_window option? If so, does the
> corruption occur without it?

Yes I was.  In prior tests, performance was abysmal without it.  I
will test without it, but our runs will load the system very
differently when they are going so slowly.

> In any case, a log of this occurring with debug_ms=1 and
> debug_rbd=20 from qemu will tell us if there are out-of-order
> operations happening.

Great, I will attempt to record some.

Regards,
Josh