Strange read data corruption on ext4/LVM/md

Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 4329 bytes --]

I recently upgrade the storage stack here, and now I'm getting these
weird, intermittent read errors. Nothing complains anywhere, I only
notice that data is corrupt (man am I glad some formats have CRC
checks).

The errors come and go, and re-reading the same file eventually gets
the correct data (have to get it out of the page cache first though).

Unfortunately I changed a bit too many pieces at the same time, so I'm
not sure where things are breaking. I went from ext3 on md, to ext4 on
LVM on md on new harddrives.

Still, I managed to catch a hexdump of one of these corruptions.
Perhaps someone can spot something familiar in it that reveals the
source of the issue:

--- good	2010-05-19 22:49:00.992620437 +0200
+++ bad	2010-05-19 22:48:16.928619828 +0200
@@ -1429750,13 +1429750,13 @@
 015d0f50  75 ed d6 51 51 08 06 a3  8e 1f 70 aa 34 b9 74 7a  |u..QQ.....p.4.tz|
 015d0f60  ff 63 a2 5d 12 79 f6 ec  8a 13 32 51 1b e9 d0 9e  |.c.].y....2Q....|
 015d0f70  2f 8f f1 0c 3f 95 a4 ce  09 c4 dd 5e 9e 10 d9 98  |/...?......^....|
-015d0f80  86 db a6 48 e1 11 64 f3  dd c6 68 95 f5 41 28 e6  |...H..d...h..A(.|
-015d0f90  1a 59 4e ad 53 bc d8 55  56 26 09 bd 66 52 27 67  |.YN.S..UV&..fR'g|
-015d0fa0  e8 59 24 35 fe 74 c6 59  cb 23 4b 4c a1 ce 9b ab  |.Y$5.t.Y.#KL....|
-015d0fb0  d8 0b 22 31 9e c5 e5 89  b7 c0 87 d0 6d b9 6f 28  |.."1........m.o(|
-015d0fc0  22 a5 3a 2c 96 88 a5 5e  12 4c e4 ab 3d ab e5 27  |".:,...^.L..=..'|
-015d0fd0  58 17 c6 ff f6 66 02 e9  b2 6c f9 de d3 44 e5 6d  |X....f...l...D.m|
-015d0fe0  62 21 75 d2 04 94 82 d8  42 02 07 b3 a0 f4 eb 2b  |b!u.....B......+|
+015d0f80  2b 53 f6 39 fa 9a 71 df  d0 0e 7d 09 44 20 42 0c  |+S.9..q...}.D B.|
+015d0f90  fa 0e 0b 71 36 19 8d d0  ed 7f c6 7f 47 8d 88 78  |...q6.......G..x|
+015d0fa0  2e 23 ab b2 8a c5 e0 b0  45 36 0c a7 f4 c4 19 7b  |.#......E6.....{|
+015d0fb0  d2 fe 6f 99 b0 b9 08 c4  6b 42 62 50 9b ab 75 cc  |..o.....kBbP..u.|
+015d0fc0  3a 91 3e 4f b4 d0 68 5a  03 76 fc e4 ad 47 ed b3  |:.>O..hZ.v...G..|
+015d0fd0  59 04 f5 93 eb 3d fa ca  5a dc 2c 6c dd 5e da d8  |Y....=..Z.,l.^..|
+015d0fe0  71 30 75 89 12 99 a7 aa  42 02 07 b3 a0 f4 eb 2b  |q0u.....B......+|
 015d0ff0  55 7f 14 78 fe de b6 65  1d 9a 24 da 32 8b 69 17  |U..x...e..$.2.i.|
 015d1000  9c 27 64 a1 70 c5 48 bd  3e 6b ce 35 8d aa a9 5f  |.'d.p.H.>k.5..._|
 015d1010  7b 14 3e b0 58 1a 7f f3  77 02 bd bc dc fe 23 46  |{.>.X...w.....#F|
@@ -2232310,13 +2232310,13 @@
 0220ff50  ca a7 a7 ae b6 15 a6 6d  e5 37 20 0e 60 85 14 d6  |.......m.7 .`...|
 0220ff60  af f2 9d 02 79 a5 ad 94  3a fa 15 66 d0 58 52 0c  |....y...:..f.XR.|
 0220ff70  b1 e3 6e 5e f2 ea b3 f1  a7 10 3d eb 2b c7 df c7  |..n^......=.+...|
-0220ff80  f5 77 7e 7c 6e 09 be 03  d7 a1 af b1 31 d4 4f 77  |.w~|n.......1.Ow|
-0220ff90  da bc 16 95 45 4f 64 97  b5 95 c9 c5 ee 76 31 2b  |....EOd......v1+|
-0220ffa0  dc 58 88 90 52 8e 93 cd  9f 33 74 61 34 a6 8e 0a  |.X..R....3ta4...|
-0220ffb0  3e 62 8f ad d5 7b 50 ea  3b 96 72 8e 35 6a 5a 21  |>b...{P.;.r.5jZ!|
-0220ffc0  16 f4 60 ed ae 2f e8 c5  df 5f aa 15 cd 49 c1 f4  |..`../..._...I..|
-0220ffd0  b4 0b 2c 0a b4 15 82 0c  e6 74 84 e6 5e d8 89 8b  |..,......t..^...|
-0220ffe0  96 08 f2 e3 3e 88 79 89  9b ba b5 72 1e 93 5c 51  |....>.y....r..\Q|
+0220ff80  e7 3b 47 3d ae 02 53 d4  1a d0 24 9a 69 c6 81 8a  |.;G=..S...$.i...|
+0220ff90  82 1f 4e 6b 76 de 84 da  07 5d f2 89 50 9c 01 89  |..Nkv....]..P...|
+0220ffa0  d6 03 bb 7d 15 5a 39 05  c7 ee 4e 51 4b a9 84 1c  |...}.Z9...NQK...|
+0220ffb0  19 da 5e d8 95 2a 36 a5  cc 59 02 69 d6 11 e4 43  |..^..*6..Y.i...C|
+0220ffc0  5d 36 50 a0 ed 95 42 24  eb 9b e8 e7 ad 9d 23 9a  |]6P...B$......#.|
+0220ffd0  d5 cb 32 2b e0 37 16 36  96 b8 9a 01 d5 00 eb 86  |..2+.7.6........|
+0220ffe0  09 db 97 5b 76 ed 0b 4f  9b ba b5 72 1e 93 5c 51  |...[v..O...r..\Q|
 0220fff0  40 1b 68 c7 fc a8 8a 1b  16 7c 89 15 85 dc 26 df  |@.h......|....&.|
 02210000  66 51 51 a5 45 4a 6c 9d  9a 98 a8 ba 72 39 77 47  |fQQ.EJl.....r9wG|
 02210010  54 d0 30 94 ea 3b c0 24  12 fc cd db 64 c4 f7 71  |T.0..;.$....d..q|

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 2101 bytes --]

On Wed, 19 May 2010 22:56:53 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> Still, I managed to catch a hexdump of one of these corruptions.
> Perhaps someone can spot something familiar in it that reveals the
> source of the issue:
> 

Sent this off a bit too fast. Forgot the analysis I did so far:

The corruption is 104 bytes. Somewhat odd number. I would have expected
something more fundamental like a sector or a page.

The data in question seems to come from another part of the file. The
two bad data chunks can be found in the original file:

015d1380  2b 53 f6 39 fa 9a 71 df  d0 0e 7d 09 44 20 42 0c  |+S.9..q...}.D B.|
015d1390  fa 0e 0b 71 36 19 8d d0  ed 7f c6 7f 47 8d 88 78  |...q6.......G..x|
015d13a0  2e 23 ab b2 8a c5 e0 b0  45 36 0c a7 f4 c4 19 7b  |.#......E6.....{|
015d13b0  d2 fe 6f 99 b0 b9 08 c4  6b 42 62 50 9b ab 75 cc  |..o.....kBbP..u.|
015d13c0  3a 91 3e 4f b4 d0 68 5a  03 76 fc e4 ad 47 ed b3  |:.>O..hZ.v...G..|
015d13d0  59 04 f5 93 eb 3d fa ca  5a dc 2c 6c dd 5e da d8  |Y....=..Z.,l.^..|
015d13e0  71 30 75 89 12 99 a7 aa  d0 d1 4c a4 fd 2a f5 fa  |q0u.......L..*..|

02210380  e7 3b 47 3d ae 02 53 d4  1a d0 24 9a 69 c6 81 8a  |.;G=..S...$.i...|
02210390  82 1f 4e 6b 76 de 84 da  07 5d f2 89 50 9c 01 89  |..Nkv....]..P...|
022103a0  d6 03 bb 7d 15 5a 39 05  c7 ee 4e 51 4b a9 84 1c  |...}.Z9...NQK...|
022103b0  19 da 5e d8 95 2a 36 a5  cc 59 02 69 d6 11 e4 43  |..^..*6..Y.i...C|
022103c0  5d 36 50 a0 ed 95 42 24  eb 9b e8 e7 ad 9d 23 9a  |]6P...B$......#.|
022103d0  d5 cb 32 2b e0 37 16 36  96 b8 9a 01 d5 00 eb 86  |..2+.7.6........|
022103e0  09 db 97 5b 76 ed 0b 4f  6d 90 3b af 2a 1c 2a cc  |...[v..Om.;.*.*.|

The shifts are 015d1380 => 015d0f80 (-1024 bytes) and 02210380 =>
0220ff80 (also -1024 bytes). At least the offset is a nice, sane power
of two number.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 3979 bytes --]

Managed to provoke the bug twice more on this file:

--- good	2010-05-19 22:49:00.992620437 +0200
+++ bad2	2010-05-19 23:06:36.738619944 +0200
@@ -1223670,13 +1223670,13 @@
 012abf50  fa c2 39 be 53 35 a7 d4  03 1c 72 71 64 c8 02 2c  |..9.S5....rqd..,|
 012abf60  51 7f cc 03 96 9e a4 62  00 18 9a dc e4 f9 69 c4  |Q......b......i.|
 012abf70  30 b0 47 4b d4 5f a2 ef  d9 cc c3 59 c7 c5 c5 a4  |0.GK._.....Y....|
-012abf80  de af 90 f2 a6 46 36 93  4c 8c 38 95 cb 65 5a e7  |.....F6.L.8..eZ.|
-012abf90  42 ec 2f 7a 46 42 2d 01  e3 26 ce b8 14 90 d1 6b  |B./zFB-..&.....k|
-012abfa0  eb 17 2e f9 42 e9 a1 e2  f9 47 b1 ac 23 7e 52 18  |....B....G..#~R.|
-012abfb0  a1 40 99 28 a4 d4 13 5e  85 54 b5 73 58 59 53 63  |.@.(...^.T.sXYSc|
-012abfc0  3e 9c 4d 0e 31 4c 70 fc  ee c6 17 3d 4b 22 78 60  |>.M.1Lp....=K"x`|
-012abfd0  01 e1 2c 26 88 a5 a3 63  b9 dc a8 7f e9 90 da 53  |..,&...c.......S|
-012abfe0  f4 84 f9 b6 a0 85 ed 87  0f 6d 9f 14 04 1d 92 fa  |.........m......|
+012abf80  55 dd ed 51 6d bf 0c b6  f5 79 d9 5c a5 7f de f4  |U..Qm....y.\....|
+012abf90  ff 47 61 a7 a4 fb d5 72  3e cc 74 b1 61 e4 4e fd  |.Ga....r>.t.a.N.|
+012abfa0  7a 37 44 0f 32 e7 47 75  73 3a 46 16 3b d5 b0 47  |z7D.2.Gus:F.;..G|
+012abfb0  ed 24 37 22 36 98 1e 64  11 5a bf 80 63 4e b0 7b  |.$7"6..d.Z..cN.{|
+012abfc0  2b 00 bc 02 10 c2 69 cf  d4 82 f9 d7 ad 1e 70 37  |+.....i.......p7|
+012abfd0  37 a1 44 6a 6a cd f3 ea  31 7f 46 c5 0c 11 07 7c  |7.Djj...1.F....||
+012abfe0  33 ca 6d 53 ef f1 6c 5a  0f 6d 9f 14 04 1d 92 fa  |3.mS..lZ.m......|
 012abff0  13 af cf 24 7b cc 09 f9  55 a5 72 4a 92 da ca bf  |...${...U.rJ....|
 012ac000  fd 4f 96 09 ce 07 08 e2  d3 15 a6 f4 d9 30 08 84  |.O...........0..|
 012ac010  88 6b 89 8a 42 b3 56 98  9a 5f 6a 08 74 2a f5 c7  |.k..B.V.._j.t*..|

--- good	2010-05-19 22:49:00.992620437 +0200
+++ bad3	2010-05-19 23:21:17.600620102 +0200
@@ -693238,13 +693238,13 @@
 00a93f50  de 0f 0d 8c f3 62 a8 fa  a1 8c ec cf ca 0a 46 93  |.....b........F.|
 00a93f60  4e ee 6b 03 8c 44 29 00  52 0b e6 18 0f 97 96 91  |N.k..D).R.......|
 00a93f70  c0 b7 c5 8c 8b 0a 45 30  71 8a 8c 37 79 2b 0c 93  |......E0q..7y+..|
-00a93f80  87 5f e0 1f 0f 64 77 85  8d 42 db c1 01 1e d7 d8  |._...dw..B......|
-00a93f90  b0 00 2c 59 c9 c2 5f 45  50 ee c7 a5 cf 89 15 82  |..,Y.._EP.......|
-00a93fa0  9a 1e 00 f9 be 84 01 99  05 7c 70 c6 fa 1f ae 44  |.........|p....D|
-00a93fb0  e3 fa f7 a2 dc 9e 69 18  1c a2 06 5f a9 b8 65 d5  |......i...._..e.|
-00a93fc0  86 5e 97 96 08 0e a6 19  6e 65 2f 33 7f 6c 1e 3f  |.^......ne/3.l.?|
-00a93fd0  39 39 e9 94 c5 e1 06 52  2b f9 27 07 90 d9 67 d1  |99.....R+.'...g.|
-00a93fe0  32 d9 4c b7 99 8a 7b e0  9b 96 9c 1d cb 5b d9 7a  |2.L...{......[.z|
+00a93f80  a8 68 75 de 11 79 03 39  cc eb 1f d7 53 08 dc 0d  |.hu..y.9....S...|
+00a93f90  54 e9 55 fc d1 31 5b 18  f7 75 7a 16 3c ab 53 b5  |T.U..1[..uz.<.S.|
+00a93fa0  e5 f5 ac 46 0d 4d 0e 01  e7 82 60 df 36 44 cd a9  |...F.M....`.6D..|
+00a93fb0  01 42 a8 c6 5b ee b6 c6  84 0f 8c f0 50 9d 37 98  |.B..[.......P.7.|
+00a93fc0  44 52 2c 61 08 38 b6 03  2f 67 cf a9 e6 ab 75 8f  |DR,a.8../g....u.|
+00a93fd0  c8 de bc 0b 58 8b 61 68  09 34 35 11 81 23 ae cc  |....X.ah.45..#..|
+00a93fe0  ef 1f 85 8f fb ea ee 17  9b 96 9c 1d cb 5b d9 7a  |.............[.z|
 00a93ff0  f4 46 62 60 a3 a2 10 23  1f ba a9 6c 1b 40 e2 61  |.Fb`...#...l.@.a|
 00a94000  1c de ef d1 a4 67 de 38  3f d2 70 68 9f d2 44 43  |.....g.8?.ph..DC|
 00a94010  c9 32 c4 8a 48 b2 0f 9b  d0 d7 12 e7 4b 67 25 f1  |.2..H.......Kg%.|

Again these are "read aheads" of 104 bytes, at 1024 bytes later into
the file.

Noteworthy is also that the last three nibbles of the corruption are
always the same (xxxxx380 => xxxxxf80).

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Wed, 19 May 2010 23:29:06 +0200
Pierre Ossman <pierre@ossman.eu> wrote:

> 
> Noteworthy is also that the last three nibbles of the corruption are
> always the same (xxxxx380 => xxxxxf80).
> 

I'm mostly talking to myself at this point, but one thing that occurs
to me here is 4096 sectors line up decently with the numbers above.
0x380 is just at the end of a 512 byte sector, and 0xf80 is just at the
end of a 4096 byte one. Not sure it's relevant, but then again I've
stayed blissfully unaware of how this sector size transformation is
going to happen. :)

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

(adding linux-ide)

On Wed, 19 May 2010 23:34:08 +0200
Pierre Ossman <pierre@ossman.eu> wrote:

> 
> I'm mostly talking to myself at this point, but one thing that occurs
> to me here is 4096 sectors line up decently with the numbers above.
> 0x380 is just at the end of a 512 byte sector, and 0xf80 is just at the
> end of a 4096 byte one. Not sure it's relevant, but then again I've
> stayed blissfully unaware of how this sector size transformation is
> going to happen. :)
> 

Ignore the above. Math is hard.

I did some more testing though, and this might be a low level issue. I
did the following multiple times:

# dd if=/dev/sde skip=4k bs=4M count=500 | md5sum

And the results were:

13aa29adcd16f8d0faf3cb5c39f43826
d1e3df33c0b0d03c61f880a8f2bb6cfb
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
7a746328b60a63b76847c3e1319a8534
13aa29adcd16f8d0faf3cb5c39f43826

Note that this is a live system, so there is some chance that something
wrote to than area, then restored it to the previous state. I'm not
sure how likely that is.

If not, then it would seem that this is a problem in either the disks,
the controller or the controller driver. The components are WD
WD1002FAEX, sil3132 and sata_sil24 respectively.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Tejun Heo]

On 05/20/2010 09:14 AM, Pierre Ossman wrote:
> Note that this is a live system, so there is some chance that something
> wrote to than area, then restored it to the previous state. I'm not
> sure how likely that is.
> 
> If not, then it would seem that this is a problem in either the disks,
> the controller or the controller driver. The components are WD
> WD1002FAEX, sil3132 and sata_sil24 respectively.

There is a report that sil3124/32 recognizes FIS corruption but keeps
using the payload anyway thus leading to data corruption when the bus
condition on pci-e side isn't ideal.  Does moving the controller to
different slot make difference?

Thanks.

-- 
tejun

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Thu, 20 May 2010 10:57:29 +0200
Tejun Heo <tj@kernel.org> wrote:

> On 05/20/2010 09:14 AM, Pierre Ossman wrote:
> > Note that this is a live system, so there is some chance that something
> > wrote to than area, then restored it to the previous state. I'm not
> > sure how likely that is.
> > 
> > If not, then it would seem that this is a problem in either the disks,
> > the controller or the controller driver. The components are WD
> > WD1002FAEX, sil3132 and sata_sil24 respectively.
> 
> There is a report that sil3124/32 recognizes FIS corruption but keeps
> using the payload anyway thus leading to data corruption when the bus
> condition on pci-e side isn't ideal.  Does moving the controller to
> different slot make difference?
> 

The machine is rather crammed right now, with one controller in each of
the three available pci-e slots (5 disks). I am running continuous tests
on the disks right now though to see if the problems is on all disks or
just some. If just one slot is causing problems then we should see some
results there.

When you say FIS corruption, do you mean corruption in the sense of
randomly flipped bits? I don't know if you saw the first couple of
mails (before linux-ide was added), but the problem is data being moved
around, not just randomly changed.

Another note is that the problem seems to worsen under load. I'm
running the dd thing in the background, which seems to make read errors
more common on my test files on the filesystem level.

I also tried disabling NCQ without any noticeable change.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Tejun Heo]

Hello,

On 05/20/2010 11:29 AM, Pierre Ossman wrote:
> The machine is rather crammed right now, with one controller in each of
> the three available pci-e slots (5 disks). I am running continuous tests
> on the disks right now though to see if the problems is on all disks or
> just some. If just one slot is causing problems then we should see some
> results there.

I see.

> When you say FIS corruption, do you mean corruption in the sense of

Oh, not FIS, FIS is the name for SATA packets.  I meant the PCI-e
packets.  How were they called... yeap TLPs.

> randomly flipped bits? I don't know if you saw the first couple of
> mails (before linux-ide was added), but the problem is data being moved
> around, not just randomly changed.

I ony saw your previous posting.  TLP corruption can happen during
command setup phase and bit flipping in the command address part is
definitely possible, so reads and writes can be headed at wrong places
in both memory and disk.  I don't know whether this would fit your
symptom tho.

> Another note is that the problem seems to worsen under load. I'm
> running the dd thing in the background, which seems to make read errors
> more common on my test files on the filesystem level.

It would be great if you can try a different controller in similar
setup.  But please keep trying to narrow down the problem and if
possible please remove filesystem from the stack and test against the
block device directly.

Thanks.

-- 
tejun

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 3416 bytes --]

On Thu, 20 May 2010 11:42:29 +0200
Tejun Heo <tj@kernel.org> wrote:

> > randomly flipped bits? I don't know if you saw the first couple of
> > mails (before linux-ide was added), but the problem is data being moved
> > around, not just randomly changed.
> 
> I ony saw your previous posting.  TLP corruption can happen during
> command setup phase and bit flipping in the command address part is
> definitely possible, so reads and writes can be headed at wrong places
> in both memory and disk.  I don't know whether this would fit your
> symptom tho.
> 

Ah. Here's the problem description from a previous mail:

The corruption is 104 bytes. Somewhat odd number. I would have expected
something more fundamental like a sector or a page.

The data in question seems to come from another part of the file.

The shifts are 015d1380 => 015d0f80 (-1024 bytes) and 02210380 =>
0220ff80 (also -1024 bytes). At least the offset is a nice, sane power
of two number.

Noteworthy is also that the last three nibbles of the corruption are
always the same (xxxxx380 => xxxxxf80).

</recap>

Note that the above analysis is from files, so it involves the entire
stack. I've since focused on raw disks. See below.

> > Another note is that the problem seems to worsen under load. I'm
> > running the dd thing in the background, which seems to make read errors
> > more common on my test files on the filesystem level.
> 
> It would be great if you can try a different controller in similar
> setup.

I only stock sil3132 cards as those are the only decent add-on cards
I've found. AHCI stuff all seems to be onboard.

> But please keep trying to narrow down the problem and if
> possible please remove filesystem from the stack and test against the
> block device directly.

That's what I've been doing the last couple of runs. From a previous
mail:

I did some more testing though, and this might be a low level issue. I
did the following multiple times:

# dd if=/dev/sde skip=4k bs=4M count=500 | md5sum

And the results were:

13aa29adcd16f8d0faf3cb5c39f43826
d1e3df33c0b0d03c61f880a8f2bb6cfb
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
7a746328b60a63b76847c3e1319a8534
13aa29adcd16f8d0faf3cb5c39f43826

</recap2>

Since the amount of data is much larger here and the incidents more
rare, I haven't been able to confirm that the corruption is identical
to what I've seen in the files. I'm working on the assumption that it
is...

I've since constructed a script that keeps re-running the above over
all relevant disks and keeps track of how many unique md5 values we
get. It's been running for about 1.5 hours right now, and here are the
results so far:

sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 

sdd and sde are both on the same controller, so the problem you
mentioned could be relevant.

I'll let the test run for a few more hours and try moving things off
that controller later tonight.


Thanks for looking at this. Unstable data storage is one of those
things that can keep you up at night. :/

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Thu, 20 May 2010 12:22:27 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> I've since constructed a script that keeps re-running the above over
> all relevant disks and keeps track of how many unique md5 values we
> get. It's been running for about 1.5 hours right now, and here are the
> results so far:
> 
> sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 
> 
> sdd and sde are both on the same controller, so the problem you
> mentioned could be relevant.
> 
> I'll let the test run for a few more hours and try moving things off
> that controller later tonight.
> 

The trend continues and the numbers are now 9 and 8 for sdd and sde and
1 for the others.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Thu, 20 May 2010 16:00:49 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> On Thu, 20 May 2010 12:22:27 +0200
> Pierre Ossman <pierre-list@ossman.eu> wrote:
> 
> > 
> > I've since constructed a script that keeps re-running the above over
> > all relevant disks and keeps track of how many unique md5 values we
> > get. It's been running for about 1.5 hours right now, and here are the
> > results so far:
> > 
> > sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 
> > 
> > sdd and sde are both on the same controller, so the problem you
> > mentioned could be relevant.
> > 
> > I'll let the test run for a few more hours and try moving things off
> > that controller later tonight.
> > 
> 
> The trend continues and the numbers are now 9 and 8 for sdd and sde and
> 1 for the others.
> 

Controller now taken out of the picture and I no longer see any errors
on either filesystem nor direct disk level.

I have a sil3132 of a different OEM brand here. I'll try that in that
pci-e slot next.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On Thu, 20 May 2010 18:28:52 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> Controller now taken out of the picture and I no longer see any errors
> on either filesystem nor direct disk level.
> 
> I have a sil3132 of a different OEM brand here. I'll try that in that
> pci-e slot next.
> 

Status update:

1. Putting the other sil3132 controller did not make any problems
appear.

2. Next I swapped that controller with the one next to it to test if
this specific board design in combination with this specific slot was
causing the issues. Still no corruption.

3. Today I put the "bad" controller back in, but in a different slot.
Still not seeing any issues.

I could move it back to the inital slot to see if the problem
reappears, but I'm just happy the system is error free again so I'll
write this off as a temporary glitch or a bad combination of card and
slot.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Thu, 15 Jul 2010 21:38:01 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> Status update:
> 
> 1. Putting the other sil3132 controller did not make any problems
> appear.
> 
> 2. Next I swapped that controller with the one next to it to test if
> this specific board design in combination with this specific slot was
> causing the issues. Still no corruption.
> 
> 3. Today I put the "bad" controller back in, but in a different slot.
> Still not seeing any issues.
> 
> I could move it back to the inital slot to see if the problem
> reappears, but I'm just happy the system is error free again so I'll
> write this off as a temporary glitch or a bad combination of card and
> slot.
> 

Time for an update to this somewhat old thread. The problem has
remained, but being so very rare that I cannot reproduce it often
enough for systematic testing. Both motherboard and the "bad"
controller have been swapped out with different (but identical) cards.

These days there are some controller cards based on Marvell's 88SE9128,
which seems to be a decent chip. So the next step is chucking all the
sil3132 stuff in favour of those...

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 230 bytes --]