Re: Strange read data corruption on ext4/LVM/md

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

(adding linux-ide)

On Wed, 19 May 2010 23:34:08 +0200
Pierre Ossman <pierre@ossman.eu> wrote:

> 
> I'm mostly talking to myself at this point, but one thing that occurs
> to me here is 4096 sectors line up decently with the numbers above.
> 0x380 is just at the end of a 512 byte sector, and 0xf80 is just at the
> end of a 4096 byte one. Not sure it's relevant, but then again I've
> stayed blissfully unaware of how this sector size transformation is
> going to happen. :)
> 

Ignore the above. Math is hard.

I did some more testing though, and this might be a low level issue. I
did the following multiple times:

# dd if=/dev/sde skip=4k bs=4M count=500 | md5sum

And the results were:

13aa29adcd16f8d0faf3cb5c39f43826
d1e3df33c0b0d03c61f880a8f2bb6cfb
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
7a746328b60a63b76847c3e1319a8534
13aa29adcd16f8d0faf3cb5c39f43826

Note that this is a live system, so there is some chance that something
wrote to than area, then restored it to the previous state. I'm not
sure how likely that is.

If not, then it would seem that this is a problem in either the disks,
the controller or the controller driver. The components are WD
WD1002FAEX, sil3132 and sata_sil24 respectively.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Tejun Heo]

On 05/20/2010 09:14 AM, Pierre Ossman wrote:
> Note that this is a live system, so there is some chance that something
> wrote to than area, then restored it to the previous state. I'm not
> sure how likely that is.
> 
> If not, then it would seem that this is a problem in either the disks,
> the controller or the controller driver. The components are WD
> WD1002FAEX, sil3132 and sata_sil24 respectively.

There is a report that sil3124/32 recognizes FIS corruption but keeps
using the payload anyway thus leading to data corruption when the bus
condition on pci-e side isn't ideal.  Does moving the controller to
different slot make difference?

Thanks.

-- 
tejun

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Thu, 20 May 2010 10:57:29 +0200
Tejun Heo <tj@kernel.org> wrote:

> On 05/20/2010 09:14 AM, Pierre Ossman wrote:
> > Note that this is a live system, so there is some chance that something
> > wrote to than area, then restored it to the previous state. I'm not
> > sure how likely that is.
> > 
> > If not, then it would seem that this is a problem in either the disks,
> > the controller or the controller driver. The components are WD
> > WD1002FAEX, sil3132 and sata_sil24 respectively.
> 
> There is a report that sil3124/32 recognizes FIS corruption but keeps
> using the payload anyway thus leading to data corruption when the bus
> condition on pci-e side isn't ideal.  Does moving the controller to
> different slot make difference?
> 

The machine is rather crammed right now, with one controller in each of
the three available pci-e slots (5 disks). I am running continuous tests
on the disks right now though to see if the problems is on all disks or
just some. If just one slot is causing problems then we should see some
results there.

When you say FIS corruption, do you mean corruption in the sense of
randomly flipped bits? I don't know if you saw the first couple of
mails (before linux-ide was added), but the problem is data being moved
around, not just randomly changed.

Another note is that the problem seems to worsen under load. I'm
running the dd thing in the background, which seems to make read errors
more common on my test files on the filesystem level.

I also tried disabling NCQ without any noticeable change.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Tejun Heo]

Hello,

On 05/20/2010 11:29 AM, Pierre Ossman wrote:
> The machine is rather crammed right now, with one controller in each of
> the three available pci-e slots (5 disks). I am running continuous tests
> on the disks right now though to see if the problems is on all disks or
> just some. If just one slot is causing problems then we should see some
> results there.

I see.

> When you say FIS corruption, do you mean corruption in the sense of

Oh, not FIS, FIS is the name for SATA packets.  I meant the PCI-e
packets.  How were they called... yeap TLPs.

> randomly flipped bits? I don't know if you saw the first couple of
> mails (before linux-ide was added), but the problem is data being moved
> around, not just randomly changed.

I ony saw your previous posting.  TLP corruption can happen during
command setup phase and bit flipping in the command address part is
definitely possible, so reads and writes can be headed at wrong places
in both memory and disk.  I don't know whether this would fit your
symptom tho.

> Another note is that the problem seems to worsen under load. I'm
> running the dd thing in the background, which seems to make read errors
> more common on my test files on the filesystem level.

It would be great if you can try a different controller in similar
setup.  But please keep trying to narrow down the problem and if
possible please remove filesystem from the stack and test against the
block device directly.

Thanks.

-- 
tejun

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 3416 bytes --]

On Thu, 20 May 2010 11:42:29 +0200
Tejun Heo <tj@kernel.org> wrote:

> > randomly flipped bits? I don't know if you saw the first couple of
> > mails (before linux-ide was added), but the problem is data being moved
> > around, not just randomly changed.
> 
> I ony saw your previous posting.  TLP corruption can happen during
> command setup phase and bit flipping in the command address part is
> definitely possible, so reads and writes can be headed at wrong places
> in both memory and disk.  I don't know whether this would fit your
> symptom tho.
> 

Ah. Here's the problem description from a previous mail:

The corruption is 104 bytes. Somewhat odd number. I would have expected
something more fundamental like a sector or a page.

The data in question seems to come from another part of the file.

The shifts are 015d1380 => 015d0f80 (-1024 bytes) and 02210380 =>
0220ff80 (also -1024 bytes). At least the offset is a nice, sane power
of two number.

Noteworthy is also that the last three nibbles of the corruption are
always the same (xxxxx380 => xxxxxf80).

</recap>

Note that the above analysis is from files, so it involves the entire
stack. I've since focused on raw disks. See below.

> > Another note is that the problem seems to worsen under load. I'm
> > running the dd thing in the background, which seems to make read errors
> > more common on my test files on the filesystem level.
> 
> It would be great if you can try a different controller in similar
> setup.

I only stock sil3132 cards as those are the only decent add-on cards
I've found. AHCI stuff all seems to be onboard.

> But please keep trying to narrow down the problem and if
> possible please remove filesystem from the stack and test against the
> block device directly.

That's what I've been doing the last couple of runs. From a previous
mail:

I did some more testing though, and this might be a low level issue. I
did the following multiple times:

# dd if=/dev/sde skip=4k bs=4M count=500 | md5sum

And the results were:

13aa29adcd16f8d0faf3cb5c39f43826
d1e3df33c0b0d03c61f880a8f2bb6cfb
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
13aa29adcd16f8d0faf3cb5c39f43826
7a746328b60a63b76847c3e1319a8534
13aa29adcd16f8d0faf3cb5c39f43826

</recap2>

Since the amount of data is much larger here and the incidents more
rare, I haven't been able to confirm that the corruption is identical
to what I've seen in the files. I'm working on the assumption that it
is...

I've since constructed a script that keeps re-running the above over
all relevant disks and keeps track of how many unique md5 values we
get. It's been running for about 1.5 hours right now, and here are the
results so far:

sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 

sdd and sde are both on the same controller, so the problem you
mentioned could be relevant.

I'll let the test run for a few more hours and try moving things off
that controller later tonight.


Thanks for looking at this. Unstable data storage is one of those
things that can keep you up at night. :/

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Thu, 20 May 2010 12:22:27 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> I've since constructed a script that keeps re-running the above over
> all relevant disks and keeps track of how many unique md5 values we
> get. It's been running for about 1.5 hours right now, and here are the
> results so far:
> 
> sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 
> 
> sdd and sde are both on the same controller, so the problem you
> mentioned could be relevant.
> 
> I'll let the test run for a few more hours and try moving things off
> that controller later tonight.
> 

The trend continues and the numbers are now 9 and 8 for sdd and sde and
1 for the others.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Thu, 20 May 2010 16:00:49 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> On Thu, 20 May 2010 12:22:27 +0200
> Pierre Ossman <pierre-list@ossman.eu> wrote:
> 
> > 
> > I've since constructed a script that keeps re-running the above over
> > all relevant disks and keeps track of how many unique md5 values we
> > get. It's been running for about 1.5 hours right now, and here are the
> > results so far:
> > 
> > sdd - 3, sde - 4, sdf - 1, sdb - 1, sdc - 1, 
> > 
> > sdd and sde are both on the same controller, so the problem you
> > mentioned could be relevant.
> > 
> > I'll let the test run for a few more hours and try moving things off
> > that controller later tonight.
> > 
> 
> The trend continues and the numbers are now 9 and 8 for sdd and sde and
> 1 for the others.
> 

Controller now taken out of the picture and I no longer see any errors
on either filesystem nor direct disk level.

I have a sil3132 of a different OEM brand here. I'll try that in that
pci-e slot next.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On Thu, 20 May 2010 18:28:52 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> Controller now taken out of the picture and I no longer see any errors
> on either filesystem nor direct disk level.
> 
> I have a sil3132 of a different OEM brand here. I'll try that in that
> pci-e slot next.
> 

Status update:

1. Putting the other sil3132 controller did not make any problems
appear.

2. Next I swapped that controller with the one next to it to test if
this specific board design in combination with this specific slot was
causing the issues. Still no corruption.

3. Today I put the "bad" controller back in, but in a different slot.
Still not seeing any issues.

I could move it back to the inital slot to see if the problem
reappears, but I'm just happy the system is error free again so I'll
write this off as a temporary glitch or a bad combination of card and
slot.

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Strange read data corruption on ext4/LVM/md

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Thu, 15 Jul 2010 21:38:01 +0200
Pierre Ossman <pierre-list@ossman.eu> wrote:

> 
> Status update:
> 
> 1. Putting the other sil3132 controller did not make any problems
> appear.
> 
> 2. Next I swapped that controller with the one next to it to test if
> this specific board design in combination with this specific slot was
> causing the issues. Still no corruption.
> 
> 3. Today I put the "bad" controller back in, but in a different slot.
> Still not seeing any issues.
> 
> I could move it back to the inital slot to see if the problem
> reappears, but I'm just happy the system is error free again so I'll
> write this off as a temporary glitch or a bad combination of card and
> slot.
> 

Time for an update to this somewhat old thread. The problem has
remained, but being so very rare that I cannot reproduce it often
enough for systematic testing. Both motherboard and the "bad"
controller have been swapped out with different (but identical) cards.

These days there are some controller cards based on Marvell's 88SE9128,
which seems to be a decent chip. So the next step is chucking all the
sil3132 stuff in favour of those...

Rgds
-- 
     -- Pierre Ossman

  WARNING: This correspondence is being monitored by FRA, a
  Swedish intelligence agency. Make sure your server uses
  encryption for SMTP traffic and consider using PGP for
  end-to-end encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 230 bytes --]