public inbox for linux-bcache@vger.kernel.org
 help / color / mirror / Atom feed
From: "Frédéric VANNIÈRE" <f.vanniere-RVWVJRqVLmoxnA8XTFIEbw@public.gmane.org>
To: linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Silent data corruption (3.11-rc4) in writethrough mode
Date: Mon, 12 Aug 2013 10:22:47 +0200	[thread overview]
Message-ID: <52089B57.8000104@planet-work.com> (raw)

Hello,

I'm using bcache for Xen hosting. The configuration is :
   - Servers HP Proliant DL360 G5/G7/Gen8
   - Backend : HP Smart Array with 2GB write cache (protected), it's a partition like /dev/sda2
   - Cache device : OWC Mercury Accelsior PCIe (2 x Sandforce in RAID0 on Marvell 88SE9230), the whole disk like /dev/sdb
   - Cache mode : writethrough
   - Kernel : 3.10.5 with bcache driver from 3.11-rc4
   - Setup process : make-bcache -B /dev/sda2 ; make-bcache -C /dev/sdb ; then attached cache device to backend.
   - LVM setup : pvcreate /dev/bcache0  , vgcreate xen_data /dev/bcache0 . Each VM has a LV volume on "xen_data"

On some hosts I've noticed very strange behavior with random mysql crashes, mysql innodb data corruption, filesystem corruption.

I'm running md5 check on system files each night and compares it to debian's database, on some VMs there where modified files (bad MD5).
When I detached the cache device, dropped virtualmemory cache on guest and synced FS, the MD5 of the files where OK again without rebooting.

It lookes like bcache had corrupted data in cache, they are no kernel logs on the host, no errors.

Frédéric.

             reply	other threads:[~2013-08-12  8:22 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-08-12  8:22 Frédéric VANNIÈRE [this message]
     [not found] ` <52089B57.8000104-RVWVJRqVLmoxnA8XTFIEbw@public.gmane.org>
2013-08-12 17:57   ` Silent data corruption (3.11-rc4) in writethrough mode Kent Overstreet
     [not found]     ` <20130812175725.GA7662-jC9Py7bek1znysI04z7BkA@public.gmane.org>
2013-08-13  8:22       ` Frédéric VANNIÈRE

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=52089B57.8000104@planet-work.com \
    --to=f.vanniere-rvwvjrqvlmoxna8xtfiebw@public.gmane.org \
    --cc=linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox