Silent data corruption (3.11-rc4) in writethrough mode

Silent data corruption (3.11-rc4) in writethrough mode

[Frédéric VANNIÈRE]

Hello,

I'm using bcache for Xen hosting. The configuration is :
   - Servers HP Proliant DL360 G5/G7/Gen8
   - Backend : HP Smart Array with 2GB write cache (protected), it's a partition like /dev/sda2
   - Cache device : OWC Mercury Accelsior PCIe (2 x Sandforce in RAID0 on Marvell 88SE9230), the whole disk like /dev/sdb
   - Cache mode : writethrough
   - Kernel : 3.10.5 with bcache driver from 3.11-rc4
   - Setup process : make-bcache -B /dev/sda2 ; make-bcache -C /dev/sdb ; then attached cache device to backend.
   - LVM setup : pvcreate /dev/bcache0  , vgcreate xen_data /dev/bcache0 . Each VM has a LV volume on "xen_data"

On some hosts I've noticed very strange behavior with random mysql crashes, mysql innodb data corruption, filesystem corruption.

I'm running md5 check on system files each night and compares it to debian's database, on some VMs there where modified files (bad MD5).
When I detached the cache device, dropped virtualmemory cache on guest and synced FS, the MD5 of the files where OK again without rebooting.

It lookes like bcache had corrupted data in cache, they are no kernel logs on the host, no errors.

Frédéric.

Re: Silent data corruption (3.11-rc4) in writethrough mode

[Kent Overstreet]

On Mon, Aug 12, 2013 at 10:22:47AM +0200, Frédéric VANNIÈRE wrote:
> Hello,
> 
> I'm using bcache for Xen hosting. The configuration is :
>   - Servers HP Proliant DL360 G5/G7/Gen8
>   - Backend : HP Smart Array with 2GB write cache (protected), it's a partition like /dev/sda2
>   - Cache device : OWC Mercury Accelsior PCIe (2 x Sandforce in RAID0 on Marvell 88SE9230), the whole disk like /dev/sdb
>   - Cache mode : writethrough
>   - Kernel : 3.10.5 with bcache driver from 3.11-rc4
>   - Setup process : make-bcache -B /dev/sda2 ; make-bcache -C /dev/sdb ; then attached cache device to backend.
>   - LVM setup : pvcreate /dev/bcache0  , vgcreate xen_data /dev/bcache0 . Each VM has a LV volume on "xen_data"
> 
> On some hosts I've noticed very strange behavior with random mysql crashes, mysql innodb data corruption, filesystem corruption.
> 
> I'm running md5 check on system files each night and compares it to debian's database, on some VMs there where modified files (bad MD5).
> When I detached the cache device, dropped virtualmemory cache on guest and synced FS, the MD5 of the files where OK again without rebooting.
> 
> It lookes like bcache had corrupted data in cache, they are no kernel logs on the host, no errors.

Ouch... This wasn't occuring on 3.10?

Re: Silent data corruption (3.11-rc4) in writethrough mode

[Frédéric VANNIÈRE]

Le 12/08/2013 19:57, Kent Overstreet a écrit :
> Ouch... This wasn't occuring on 3.10? 

I don't know, I didn't use 3.10 long enough in production to see this appen. I'll revert to vanilla 3.10.6 and see if the issue is still present after my vacation.