Data corruption with bcache in 3.11

Data corruption with bcache in 3.11

[Kent Overstreet]

If you were one of the users seeing data corruption in 3.11 but not any
of the 3.10 stable kernel, I think I just fixed it. It's now clearer why
it was only hitting some users, too...

I want to get confirmation from someone who was seeing it that it fixes
the issue, so please test the fix and let me know what happens. It's in
my bcache-data-corruption-fix branch, and the patch is below:

commit 97ad43a31a7ea3c23a4eb64b8e26aad3cf21ae8d
Author: Kent Overstreet <kmo-PEzghdH756F8UrSeD/g0lQ@public.gmane.org>
Date:   Thu Sep 19 19:10:24 2013 -0700

    block: Fix bio_copy_data()
    
    The memcpy() in bio_copy_data() was using the wrong offset vars, leading
    to data corruption in weird unusual setups.
    
    Signed-off-by: Kent Overstreet <kmo-PEzghdH756F8UrSeD/g0lQ@public.gmane.org>
    Cc: linux-stable <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # >= v3.9

diff --git a/fs/bio.c b/fs/bio.c
index c5eae72..5e7507d 100644
--- a/fs/bio.c
+++ b/fs/bio.c
@@ -917,8 +917,8 @@ void bio_copy_data(struct bio *dst, struct bio *src)
 		src_p = kmap_atomic(src_bv->bv_page);
 		dst_p = kmap_atomic(dst_bv->bv_page);
 
-		memcpy(dst_p + dst_bv->bv_offset,
-		       src_p + src_bv->bv_offset,
+		memcpy(dst_p + dst_offset,
+		       src_p + src_offset,
 		       bytes);
 
 		kunmap_atomic(dst_p);

Re: Data corruption with bcache in 3.11

[Rolf Fokkens]

I'm using bcache on my system for about a week now, and all seems rock 
solid to me. I'm even running my root FS on bcache, which means my 
system might no longer boot when something goes wrong. Based on reported 
issues I also did some stress tests to see if anything breaks, e.g. by 
starting cp's and rsync's and see what happens. I can only tell they 
don't involve much caching, but there's no data corruption.

If I can help by trying to reproduce issues, let me know. I tried, but 
wasn't successful in reproducing issues. So provide any instructions on 
what to reproduce and I'll try to do that. Of course the easiest way to 
reproduce for me would be to do it in a VM, but I'm also OK with using 
my physical "test-configuration".

Rolf

Re: Data corruption with bcache in 3.11

[Kent Overstreet]

If you weren't seeing it before, don't worry about it. Normal setups
with the usual 4k block size won't ever see this.

On Fri, Sep 20, 2013 at 12:06 AM, Rolf Fokkens <rolf-6w2rdlBuEQTpMFipWq+H6g@public.gmane.org> wrote:
> I'm using bcache on my system for about a week now, and all seems rock solid
> to me. I'm even running my root FS on bcache, which means my system might no
> longer boot when something goes wrong. Based on reported issues I also did
> some stress tests to see if anything breaks, e.g. by starting cp's and
> rsync's and see what happens. I can only tell they don't involve much
> caching, but there's no data corruption.
>
> If I can help by trying to reproduce issues, let me know. I tried, but
> wasn't successful in reproducing issues. So provide any instructions on what
> to reproduce and I'll try to do that. Of course the easiest way to reproduce
> for me would be to do it in a VM, but I'm also OK with using my physical
> "test-configuration".
>
> Rolf
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Data corruption with bcache in 3.11

[Rolf Fokkens]

On 09/20/2013 09:44 AM, Kent Overstreet wrote:
> If you weren't seeing it before, don't worry about it. Normal setups
> with the usual 4k block size won't ever see this.
I wouldn't mind helping to reproduce other issues as well, like the 
iozone related issues and the "bcache hangs" issue. I actually tried, 
but bcache worked just fine.

Re: Data corruption with bcache in 3.11

[Zachary Palmer]

On 09/20/2013 09:44 AM, Kent Overstreet wrote:
> If you weren't seeing it before, don't worry about it. Normal setups
> with the usual 4k block size won't ever see this.
Fascinating.  That might have been my problem; my block size is 
currently 512.  (Is that ill-advised?)  I'm currently running a kernel 
built from the 3.11 corruption fix branch and I have no problems so 
far.  I'll report back in a few days; that's how long it usually took to 
start corrupting things and segfaulting the world. :)

Thanks,

Zach

Re: Data corruption with bcache in 3.11

[Joseph Yasi]

On 09/20/2013 02:35 AM, Kent Overstreet wrote:
> If you were one of the users seeing data corruption in 3.11 but not any
> of the 3.10 stable kernel, I think I just fixed it. It's now clearer why
> it was only hitting some users, too...
>
> I want to get confirmation from someone who was seeing it that it fixes
> the issue, so please test the fix and let me know what happens. It's in
> my bcache-data-corruption-fix branch, and the patch is below:
>
> commit 97ad43a31a7ea3c23a4eb64b8e26aad3cf21ae8d

This patch seems to fix the corruption problem for me caching an md
RAID10 array.