Re: Random data corruption in VM, possibly caused by rbd

[Oliver Francke <Oliver.Francke@filoo.de>]

Hi Guido,

yeah, there is something weird going on. I just started to establish 
some test-VM's. Freshly imported from running *.qcow2 images.
Kernel panic with INIT, seg-faults and other "funny" stuff.

Just added the rbd_cache=true in my config, voila. All is 
fast-n-up-n-running...
All my testing was done with cache enabled... Since our errors all came 
from rbd_writeback from former ceph-versions...

Josh? Sage? Help?!

Oliver.

On 06/08/2012 02:55 PM, Guido Winkelmann wrote:
> Am Donnerstag, 7. Juni 2012, 12:48:05 schrieben Sie:
>> On 06/07/2012 11:04 AM, Guido Winkelmann wrote:
>>> Hi,
>>>
>>> I'm using Ceph with RBD to provide network-transparent disk images for
>>> KVM-
>>> based virtual servers. The last two days, I've been hunting some weird
>>> elusive bug where data in the virtual machines would be corrupted in
>>> weird ways. It usually manifests in files having some random data -
>>> usually zeroes - at the start before the actual contents that should be
>>> in there start.
>> I definitely want to figure out what's going on with this.
>> A few questions:
>>
>> Are you using rbd caching? If so, what settings?
>>
>> In either case, does the corruption still occur if you
>> switch caching on/off? There are different I/O paths here,
>> and this might tell us if the problem is on the client side.
> Okay, I've tried enabling rbd caching now, and so far, the problem appears to
> be gone.
>
> I am using libvirt for starting and managing the virtual machines, and what I
> did was change the<source>  element for the virtual disk from
>
> <source protocol='rbd' name='rbd/name_of_image'>
>
> to
>
> <source protocol='rbd' name='rbd/name_of_image:rbd_cache=true'>
>
> and then restart the VM.
> (I found that in one of your mails on this list; there does not appear to be
> any proper documentation on this...)
>
> The iotester does not find any corruptions with these settings.
>
> The VM ist still horribly broken, but that's probably lingering filesystem
> damage from yesterday. I'll try with a fresh image next.
>
> I did not change anything else in the setup. In particular, the OSDs still use
> btrfs. One of the OSD has been restarted, though. I will run another test with
> a VM without rbd caching, to make sure it wasn't by random chance restarting
> that one osd that made the real difference.
>
> Enabling btrfs did not appear to make any difference wrt performance, but
> that's probably because my tests mostly create sustained sequential IO, for
> which caches are generally not very helpful.
>
> Enabling rbd caching is not a solution I particularly like, for two reasons:
>
> 1. In my setup, migrating VMs from one host to another is a normal part of
> operation, and I still don't know ho to prevent data corruption (in the form
> of silently lost writes) when combining rbd caching and migration.
>
> 2. I'm not really looking into speeding up single VM, I'm really more
> interested in just how many VMs I can run before performance starts degrading
> for everyone, and I don't think rbd caching will help with that.
>
> Regards,
> 	Guido
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 

Oliver Francke

filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh

Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz

Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html