Strange read data corruption on ext4/LVM/md

[Pierre Ossman <pierre-list@ossman.eu>]

[-- Attachment #1: Type: text/plain, Size: 4329 bytes --]

I recently upgrade the storage stack here, and now I'm getting these
weird, intermittent read errors. Nothing complains anywhere, I only
notice that data is corrupt (man am I glad some formats have CRC
checks).

The errors come and go, and re-reading the same file eventually gets
the correct data (have to get it out of the page cache first though).

Unfortunately I changed a bit too many pieces at the same time, so I'm
not sure where things are breaking. I went from ext3 on md, to ext4 on
LVM on md on new harddrives.

Still, I managed to catch a hexdump of one of these corruptions.
Perhaps someone can spot something familiar in it that reveals the
source of the issue:

--- good	2010-05-19 22:49:00.992620437 +0200
+++ bad	2010-05-19 22:48:16.928619828 +0200
@@ -1429750,13 +1429750,13 @@
 015d0f50  75 ed d6 51 51 08 06 a3  8e 1f 70 aa 34 b9 74 7a  |u..QQ.....p.4.tz|
 015d0f60  ff 63 a2 5d 12 79 f6 ec  8a 13 32 51 1b e9 d0 9e  |.c.].y....2Q....|
 015d0f70  2f 8f f1 0c 3f 95 a4 ce  09 c4 dd 5e 9e 10 d9 98  |/...?......^....|
-015d0f80  86 db a6 48 e1 11 64 f3  dd c6 68 95 f5 41 28 e6  |...H..d...h..A(.|
-015d0f90  1a 59 4e ad 53 bc d8 55  56 26 09 bd 66 52 27 67  |.YN.S..UV&..fR'g|
-015d0fa0  e8 59 24 35 fe 74 c6 59  cb 23 4b 4c a1 ce 9b ab  |.Y$5.t.Y.#KL....|
-015d0fb0  d8 0b 22 31 9e c5 e5 89  b7 c0 87 d0 6d b9 6f 28  |.."1........m.o(|
-015d0fc0  22 a5 3a 2c 96 88 a5 5e  12 4c e4 ab 3d ab e5 27  |".:,...^.L..=..'|
-015d0fd0  58 17 c6 ff f6 66 02 e9  b2 6c f9 de d3 44 e5 6d  |X....f...l...D.m|
-015d0fe0  62 21 75 d2 04 94 82 d8  42 02 07 b3 a0 f4 eb 2b  |b!u.....B......+|
+015d0f80  2b 53 f6 39 fa 9a 71 df  d0 0e 7d 09 44 20 42 0c  |+S.9..q...}.D B.|
+015d0f90  fa 0e 0b 71 36 19 8d d0  ed 7f c6 7f 47 8d 88 78  |...q6.......G..x|
+015d0fa0  2e 23 ab b2 8a c5 e0 b0  45 36 0c a7 f4 c4 19 7b  |.#......E6.....{|
+015d0fb0  d2 fe 6f 99 b0 b9 08 c4  6b 42 62 50 9b ab 75 cc  |..o.....kBbP..u.|
+015d0fc0  3a 91 3e 4f b4 d0 68 5a  03 76 fc e4 ad 47 ed b3  |:.>O..hZ.v...G..|
+015d0fd0  59 04 f5 93 eb 3d fa ca  5a dc 2c 6c dd 5e da d8  |Y....=..Z.,l.^..|
+015d0fe0  71 30 75 89 12 99 a7 aa  42 02 07 b3 a0 f4 eb 2b  |q0u.....B......+|
 015d0ff0  55 7f 14 78 fe de b6 65  1d 9a 24 da 32 8b 69 17  |U..x...e..$.2.i.|
 015d1000  9c 27 64 a1 70 c5 48 bd  3e 6b ce 35 8d aa a9 5f  |.'d.p.H.>k.5..._|
 015d1010  7b 14 3e b0 58 1a 7f f3  77 02 bd bc dc fe 23 46  |{.>.X...w.....#F|
@@ -2232310,13 +2232310,13 @@
 0220ff50  ca a7 a7 ae b6 15 a6 6d  e5 37 20 0e 60 85 14 d6  |.......m.7 .`...|
 0220ff60  af f2 9d 02 79 a5 ad 94  3a fa 15 66 d0 58 52 0c  |....y...:..f.XR.|
 0220ff70  b1 e3 6e 5e f2 ea b3 f1  a7 10 3d eb 2b c7 df c7  |..n^......=.+...|
-0220ff80  f5 77 7e 7c 6e 09 be 03  d7 a1 af b1 31 d4 4f 77  |.w~|n.......1.Ow|
-0220ff90  da bc 16 95 45 4f 64 97  b5 95 c9 c5 ee 76 31 2b  |....EOd......v1+|
-0220ffa0  dc 58 88 90 52 8e 93 cd  9f 33 74 61 34 a6 8e 0a  |.X..R....3ta4...|
-0220ffb0  3e 62 8f ad d5 7b 50 ea  3b 96 72 8e 35 6a 5a 21  |>b...{P.;.r.5jZ!|
-0220ffc0  16 f4 60 ed ae 2f e8 c5  df 5f aa 15 cd 49 c1 f4  |..`../..._...I..|
-0220ffd0  b4 0b 2c 0a b4 15 82 0c  e6 74 84 e6 5e d8 89 8b  |..,......t..^...|
-0220ffe0  96 08 f2 e3 3e 88 79 89  9b ba b5 72 1e 93 5c 51  |....>.y....r..\Q|
+0220ff80  e7 3b 47 3d ae 02 53 d4  1a d0 24 9a 69 c6 81 8a  |.;G=..S...$.i...|
+0220ff90  82 1f 4e 6b 76 de 84 da  07 5d f2 89 50 9c 01 89  |..Nkv....]..P...|
+0220ffa0  d6 03 bb 7d 15 5a 39 05  c7 ee 4e 51 4b a9 84 1c  |...}.Z9...NQK...|
+0220ffb0  19 da 5e d8 95 2a 36 a5  cc 59 02 69 d6 11 e4 43  |..^..*6..Y.i...C|
+0220ffc0  5d 36 50 a0 ed 95 42 24  eb 9b e8 e7 ad 9d 23 9a  |]6P...B$......#.|
+0220ffd0  d5 cb 32 2b e0 37 16 36  96 b8 9a 01 d5 00 eb 86  |..2+.7.6........|
+0220ffe0  09 db 97 5b 76 ed 0b 4f  9b ba b5 72 1e 93 5c 51  |...[v..O...r..\Q|
 0220fff0  40 1b 68 c7 fc a8 8a 1b  16 7c 89 15 85 dc 26 df  |@.h......|....&.|
 02210000  66 51 51 a5 45 4a 6c 9d  9a 98 a8 ba 72 39 77 47  |fQQ.EJl.....r9wG|
 02210010  54 d0 30 94 ea 3b c0 24  12 fc cd db 64 c4 f7 71  |T.0..;.$....d..q|

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]