OT: silent data corruption reading from hard drives

OT: silent data corruption reading from hard drives

[matt]

Quick intro: Last year I was having problems with an md array
continuously having a mismatch_cnt in the tens of thousands,
inexplicably.  After a week or two of hardware swapping and such, I
narrowed it down to bad reads of the hard drive block devices.  I used
scripts that would repetitively do something like this on all my drives:
      dd if=/dev/sdk1 bs=1024 count=50000000 |md5sum -b
Some devices would intermittently get different results.  I ended up
resolving (?) it by replacing the cheapo (Syba) SATA controller cards
with other cheapo (Rosewill) ones.  I've been fine for about a year
since then.

But now it's just started happening again.  Although this isn't an md
question per se, I'm hoping some of you raid/kernel/storage gurus can
give me some tips on how to trace this in a better way than my haphazard
method last year.  Is there any way to detect these bad reads when they
happen?  (Apparently not?)   What about finding out if the cause is the
motherboard, the controller card, the device driver, or the kernel?
(Besides swapping hardware?)  Can the md layer help out in this regard?
  Are there known bugs or hardware nuances that relate to this?  Is
silent data corruption like this simply to be expected when using cheap
commodity hardware?

Thanks for reading...

matt

Re: OT: silent data corruption reading from hard drives

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On Wed, 01 Aug 2012 08:02:38 -0400
matt <listy@fastmail.fm> wrote:

> Quick intro: Last year I was having problems with an md array
> continuously having a mismatch_cnt in the tens of thousands,
> inexplicably.  After a week or two of hardware swapping and such, I
> narrowed it down to bad reads of the hard drive block devices.  I used
> scripts that would repetitively do something like this on all my drives:
>       dd if=/dev/sdk1 bs=1024 count=50000000 |md5sum -b
> Some devices would intermittently get different results.  I ended up
> resolving (?) it by replacing the cheapo (Syba) SATA controller cards
> with other cheapo (Rosewill) ones.  I've been fine for about a year
> since then.

Syba was mentioned in a bad context last time we discussed this here, somewhere
down the thread: http://comments.gmane.org/gmane.linux.raid/33346
I don't remember if you have participated in that discussion, if not, have an
entertaining read. Is your new Rosewill card also Silicon Image based?

> Is silent data corruption like this simply to be expected when using cheap
> commodity hardware?

I'd say no. There are known good and known bad hardware, where both good and
bad is most easily measured by the amount (or lack of) problem reports you hear
or can find. One method to do that is asking a searching engine about "<chip
name> data corruption".

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: OT: silent data corruption reading from hard drives

[Stan Hoeppner]

On 8/1/2012 7:02 AM, matt wrote:
> Is silent data corruption like this simply to be expected when using cheap
> commodity hardware?

When you pay less than 1/4th the price of one drive for the controller
card directing all your drives, is it really necessary to ask this question?

Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card.
The pizza is consumed in one night, maybe some for breakfast.  Would you
trust a day worth of pizza with your RAID?  With your data?

One tank of gas for the average car today costs $50, the same as two of
these SATA cards.  The gas is gone in a week or two.  You want the cards
to run for 2-4 years.

Your drives cost anywhere from $400-$1000, yet your controller maybe
$50.  Do I really need to say any more?  Spend $150-250 on a decent
SAS/SATA controller such as an LSI or Adaptec and you won't have to
worry about this kind of thing.

-- 
Stan

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

well this will work on systems that have pci-express or pci slots
for systems that don´t have slots like arm systems, i don´t know if
this could work... maybe in some future we will have arm devices
running NAS solutions, and a more secure system is nice...

2012/8/1 Stan Hoeppner <stan@hardwarefreak.com>:
> On 8/1/2012 7:02 AM, matt wrote:
>> Is silent data corruption like this simply to be expected when using cheap
>> commodity hardware?
>
> When you pay less than 1/4th the price of one drive for the controller
> card directing all your drives, is it really necessary to ask this question?
>
> Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card.
> The pizza is consumed in one night, maybe some for breakfast.  Would you
> trust a day worth of pizza with your RAID?  With your data?
>
> One tank of gas for the average car today costs $50, the same as two of
> these SATA cards.  The gas is gone in a week or two.  You want the cards
> to run for 2-4 years.
>
> Your drives cost anywhere from $400-$1000, yet your controller maybe
> $50.  Do I really need to say any more?  Spend $150-250 on a decent
> SAS/SATA controller such as an LSI or Adaptec and you won't have to
> worry about this kind of thing.
>
> --
> Stan
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

just an example...
this arm device have USB3.0 and could plug external drivers with a fast speed

http://gizmodo.com/5646025/marvells-15ghz-triple-core-arm-processor-can-play-140-hours-of-music-on-one-charge

it´s a nice tool to implement since we don´t know what´s the quality
of USB external drives, and we could use it as a home NAS server


2012/8/1 Roberto Spadim <roberto@spadim.com.br>:
> well this will work on systems that have pci-express or pci slots
> for systems that don´t have slots like arm systems, i don´t know if
> this could work... maybe in some future we will have arm devices
> running NAS solutions, and a more secure system is nice...
>
> 2012/8/1 Stan Hoeppner <stan@hardwarefreak.com>:
>> On 8/1/2012 7:02 AM, matt wrote:
>>> Is silent data corruption like this simply to be expected when using cheap
>>> commodity hardware?
>>
>> When you pay less than 1/4th the price of one drive for the controller
>> card directing all your drives, is it really necessary to ask this question?
>>
>> Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card.
>> The pizza is consumed in one night, maybe some for breakfast.  Would you
>> trust a day worth of pizza with your RAID?  With your data?
>>
>> One tank of gas for the average car today costs $50, the same as two of
>> these SATA cards.  The gas is gone in a week or two.  You want the cards
>> to run for 2-4 years.
>>
>> Your drives cost anywhere from $400-$1000, yet your controller maybe
>> $50.  Do I really need to say any more?  Spend $150-250 on a decent
>> SAS/SATA controller such as an LSI or Adaptec and you won't have to
>> worry about this kind of thing.
>>
>> --
>> Stan
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Roberto Spadim
> Spadim Technology / SPAEmpresarial



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Adam Goryachev]

On 02/08/12 10:56, Stan Hoeppner wrote:
> Your drives cost anywhere from $400-$1000, yet your controller maybe $50. Do I really need to say any
more? Spend $150-250 on a decent SAS/SATA controller such as an LSI or
Adaptec and you won't have to worry about this kind of thing.
On a related note, is there a specific model card you would suggest that
fits the following:
a) 4 or 8 ports
b) PCI (for older systems)
c) affordable (note, not cheap, but not expensive)
d) does not need to support RAID since we will use software raid anyway

I am always unsure, and see very cheap SATA cards (around $50) and worry
as you suggest, and very expensive (around $1000), and feel that is a
waste of money for what should be a fairly simple card.

Also, would you use on board controller, or prefer to always use an
external controller? Or is it worthwhile using a combination (in case of
controller failure)?

Thanks for your opinions.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

check this link at oracle
https://oss.oracle.com/projects/data-integrity/

i don´t know if this is a good solution, but i think it´s a start point...


2012/8/1 Adam Goryachev <mailinglists@websitemanagers.com.au>:
> On 02/08/12 10:56, Stan Hoeppner wrote:
>> Your drives cost anywhere from $400-$1000, yet your controller maybe $50. Do I really need to say any
> more? Spend $150-250 on a decent SAS/SATA controller such as an LSI or
> Adaptec and you won't have to worry about this kind of thing.
> On a related note, is there a specific model card you would suggest that
> fits the following:
> a) 4 or 8 ports
> b) PCI (for older systems)
> c) affordable (note, not cheap, but not expensive)
> d) does not need to support RAID since we will use software raid anyway
>
> I am always unsure, and see very cheap SATA cards (around $50) and worry
> as you suggest, and very expensive (around $1000), and feel that is a
> waste of money for what should be a fairly simple card.
>
> Also, would you use on board controller, or prefer to always use an
> external controller? Or is it worthwhile using a combination (in case of
> controller failure)?
>
> Thanks for your opinions.
>
> Regards,
> Adam
>
> --
> Adam Goryachev
> Website Managers
> www.websitemanagers.com.au
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1931 bytes --]

On Wed, 01 Aug 2012 19:56:50 -0500
Stan Hoeppner <stan@hardwarefreak.com> wrote:

> On 8/1/2012 7:02 AM, matt wrote:
> > Is silent data corruption like this simply to be expected when using cheap
> > commodity hardware?
> 
> When you pay less than 1/4th the price of one drive for the controller
> card directing all your drives, is it really necessary to ask this question?
> 
> Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card.
> The pizza is consumed in one night, maybe some for breakfast.  Would you
> trust a day worth of pizza with your RAID?  With your data?
> 
> One tank of gas for the average car today costs $50, the same as two of
> these SATA cards.  The gas is gone in a week or two.  You want the cards
> to run for 2-4 years.
> 
> Your drives cost anywhere from $400-$1000, yet your controller maybe
> $50.  Do I really need to say any more?  Spend $150-250 on a decent
> SAS/SATA controller such as an LSI or Adaptec and you won't have to
> worry about this kind of thing.

While you're at it also don't forget to get yourself some quality
Ethernet cabling, you don't want any data corruption on your LAN, do you
http://www.amazon.com/Denon-AKDL1-Dedicated-Link-Cable/dp/tech-data/B000I1X6PM/ref=de_a_smtd

It's not a matter of "just pay more and all your problems will be solved".

There are some really crappy controllers even in the "enterprise" range
(e.g. those using the mv_sas driver). And there's a lot of $15 models giving
zero problems whatsoever (e.g. the ahci-compliant Marvell 88SE912x, JMicron
JMB36x). 

Also, if you only use the mdadm software RAID, getting an enterprise hardware
RAID controller is truly a waste of money, unless you really need the extra
port density.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: OT: silent data corruption reading from hard drives

[Stan Hoeppner]

On 8/1/2012 8:27 PM, Adam Goryachev wrote:

> On a related note, is there a specific model card you would suggest that
> fits the following:
> a) 4 or 8 ports
> b) PCI (for older systems)
> c) affordable (note, not cheap, but not expensive)
> d) does not need to support RAID since we will use software raid anyway

If you're stuck with PCI I'd stick with a more recognized vendor such as
Promise Tech.  They still sell a 4 port PCI card.  $70 USD at Newegg.

> I am always unsure, and see very cheap SATA cards (around $50) and worry
> as you suggest, and very expensive (around $1000), and feel that is a
> waste of money for what should be a fairly simple card.

A "very cheap" SATA card is a 2-port SiI 3512 based card--Syba,
Rosewill, Koutech, etc-- for $15 on sale, $20 regular price, all prices
USD.  A $50 card with the same port count *should* hopefully be much
higher quality.

> Also, would you use on board controller, or prefer to always use an
> external controller? Or is it worthwhile using a combination (in case of
> controller failure)?

Motherboard-down SATA controllers are usually fine for md/RAID as
they're part of an Intel or AMD chipset, and of high quality.  But this
also depends on the quality of the board as well--there have been plenty
of boards with good chipsets but also a buggy BIOS.  Overall they're
usually much better than a $20 card.

In 20+ years of computing I've never had an HBA or RAID card fail on me.
 I've had plenty of mobos go toes up and plenty of NICs and VGA cards
burn out.  I have a 128MB 3-channel AMI MegaRAID 428 UW-SCSI built in
1998 that still works and an Adaptec 1542 ISA SCSI card from 1993 that
still works.  I've tossed piles of dead mobos over the same period.

-- 
Stan

Re: OT: silent data corruption reading from hard drives

[Stan Hoeppner]

On 8/1/2012 10:19 PM, Roman Mamedov wrote:
> On Wed, 01 Aug 2012 19:56:50 -0500
> Stan Hoeppner <stan@hardwarefreak.com> wrote:
> 
>> On 8/1/2012 7:02 AM, matt wrote:
>>> Is silent data corruption like this simply to be expected when using cheap
>>> commodity hardware?
>>
>> When you pay less than 1/4th the price of one drive for the controller
>> card directing all your drives, is it really necessary to ask this question?
>>
>> Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card.
>> The pizza is consumed in one night, maybe some for breakfast.  Would you
>> trust a day worth of pizza with your RAID?  With your data?
>>
>> One tank of gas for the average car today costs $50, the same as two of
>> these SATA cards.  The gas is gone in a week or two.  You want the cards
>> to run for 2-4 years.
>>
>> Your drives cost anywhere from $400-$1000, yet your controller maybe
>> $50.  Do I really need to say any more?  Spend $150-250 on a decent
>> SAS/SATA controller such as an LSI or Adaptec and you won't have to
>> worry about this kind of thing.
> 
> While you're at it also don't forget to get yourself some quality
> Ethernet cabling, you don't want any data corruption on your LAN, do you
> http://www.amazon.com/Denon-AKDL1-Dedicated-Link-Cable/dp/tech-data/B000I1X6PM/ref=de_a_smtd

The hyperbole isn't necessary Roman, and detracts from your argument.

> It's not a matter of "just pay more and all your problems will be solved".

I didn't say, nor imply such a thing.  You missed my point entirely
about dollar amount.  It had two aspects:

1.  Some people are willing to spend hundreds or thousands on drives,
then hook them to $15 controllers without giving that ratio a thought,
no reflection upon the sanity of it.

2.  $150-250 is the price range of about a dozen high quality HBAs on
the market.  I don't have the time to go out and identify and then list
them all here.  Mentioning the price range simply makes sure people will
be looking at the right cards.

> There are some really crappy controllers even in the "enterprise" range
> (e.g. those using the mv_sas driver). 

First, there are not "enterprise" controllers that use the Marvell SAS
chip, period.  It's a low/mid cost, entry level 8 port SAS ASIC.  The
cards based on it a not crappy, but actually pretty decent.  They run
very well with Windows, *BSD, SCO, NetWare, and other OSes.  It just
happens that the Linux driver, yes mv_sas, totally sucks, despite
continuous development.  In fact, hop on the linux-scsi mailing list and
you can chat with who I believe is the primary developer of that driver,
Xiangliang Yu.  He works for Marvell.

> And there's a lot of $15 models giving
> zero problems whatsoever (e.g. the ahci-compliant Marvell 88SE912x, JMicron
> JMB36x). 

You're confusing SATA controller ASICs, which may be of high quality and
have quality drivers, with HBAs, which are the finished products that
incorporate these ASICs.  The HBAs also incorporate many other ICs and
analog parts, which are often of questionable quality in the $15-30
range of HBAs.  You're buying an HBA, not just the SATA ASIC.  QC/QA in
the manufacture of HBAs that retail for these prices is very low
compared to more expensive models.  And you can't get support from the
manufacturer.  Call Syba, Rosewill, Koutech, etc, and see how helpful
they are--if you can get a hold of anyone that is.  Then call LSI.

> Also, if you only use the mdadm software RAID, getting an enterprise hardware
> RAID controller is truly a waste of money, unless you really need the extra
> port density.

That logic is flawed.  If you have an enterprise hardware RAID
controller you're not going to use md/RAID, unless you're stitching LUNs
together with linear or RAID0, as I often do.  In fact that's my most
frequent use of md/RAID--concatenation of hardware RAID or SAN LUNs.

If you're talking about enterprise class HBAs, your logic is still
flawed.  Not only are they higher quality and performance, but you can
daisy chain multiple expanders to them.  The $175 LSI 9211-4i is a
4-port (single 8087) enterprise HBA that can attach up to 256 SAS/SATA
drives using fanned or daisy chained expanders.  You simply can't do
this with SATA port multipliers, which still tend to have compat issues
quite often.

The most you can get is 5 downstream SATA links per PMP.  And the PMPs,
the SiI based models, are almost $100 a pop.  So a 4 port SiI HBA with 4
SiI PMPs gets you 20 drives connected for ~$450 at PCIe 1.0 x1, for
250MB/s one way.  Or you can buy the LSI 9211-4i and the Intel 24-port
SAS expander for ~$400 at PCIe 2.0 x4, for 2GB/s one way.

In this case not only is the "enterprise" HBA solution cheaper, it's
also without compatibility issues or bugs, has excellent support, and
has an 8x faster bus, and a much faster ASIC able to handle over 20x the
IOPS.  Sure, there are Marvell based PCIe x4 cheapo HBAs, Rosewill makes
a 4 port model, but the Marvell SATA chips don't work with the SiI PMPs
very well, if at all, and AFAIK nobody offers a standalone Marvell PMP,
nor any other PMP.  SiI pretty much has the PMP market to itself.

-- 
Stan

Re: OT: silent data corruption reading from hard drives

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 2524 bytes --]

On Thu, 02 Aug 2012 02:51:08 -0500
Stan Hoeppner <stan@hardwarefreak.com> wrote:

> I didn't say, nor imply such a thing.  You missed my point entirely
> about dollar amount.  It had two aspects:
> 
> 1.  Some people are willing to spend hundreds or thousands on drives,
> then hook them to $15 controllers without giving that ratio a thought,
> no reflection upon the sanity of it.

Why should there be any reflection? There is no no automatic "Hmm, this thing
costs $XXX, so the other thing should also cost $XXX" rule.

$15 is simply the right price for a perfectly working 2 port SATA controller
(based on some of those chip models I named) on the market.

The rule is not "buy expensive", it's "avoid buying (known-)broken". Maybe
it's more difficult to come across a broken card in the expensive segment.
Maybe it's not. In any case my suggestion is to just do some research before
you buy, identify broken chips and cards, avoid those, and save yourself those
$185 or whatever.

> First, there are not "enterprise" controllers that use the Marvell SAS
> chip, period.  It's a low/mid cost, entry level 8 port SAS ASIC.  The
> cards based on it a not crappy, but actually pretty decent.  They run
> very well with Windows, *BSD, SCO, NetWare, and other OSes.  It just
> happens that the Linux driver, yes mv_sas, totally sucks

The end result for the user is the same, they accidentally buy that card, they
use GNU/Linux, they have to use that sucky driver. The best choice is to avoid
buying the card in the first place, but how do you know you should, if you
didn't do the research (see above) and just looked at the price tag.

> > Also, if you only use the mdadm software RAID, getting an enterprise hardware
> > RAID controller is truly a waste of money, unless you really need the extra
> > port density.
> 
> That logic is flawed.  If you have an enterprise hardware RAID
> controller you're not going to use md/RAID, unless you're stitching LUNs
> together with linear or RAID0, as I often do.  In fact that's my most
> frequent use of md/RAID--concatenation of hardware RAID or SAN LUNs.

I do not recommend using hardware RAID. It locks you into one card/vendor,
usually is much less flexible than mdadm, and often even provides lower
performance. See http://linux.yyz.us/why-software-raid.html

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: OT: silent data corruption reading from hard drives

[Stan Hoeppner]

On 8/2/2012 3:06 AM, Roman Mamedov wrote:
> On Thu, 02 Aug 2012 02:51:08 -0500
> Stan Hoeppner <stan@hardwarefreak.com> wrote:
> 
>> I didn't say, nor imply such a thing.  You missed my point entirely
>> about dollar amount.  It had two aspects:
>>
>> 1.  Some people are willing to spend hundreds or thousands on drives,
>> then hook them to $15 controllers without giving that ratio a thought,
>> no reflection upon the sanity of it.
> 
> Why should there be any reflection? There is no no automatic "Hmm, this thing
> costs $XXX, so the other thing should also cost $XXX" rule.

You're Russian.  In the U.S. there has been an axiom for generations
that says "You get what you pay for".  This isn't always true, but more
often than not.

> $15 is simply the right price for a perfectly working 2 port SATA controller
> (based on some of those chip models I named) on the market.

If it works out of the box, or after a month, yes.  But many of these
$15 cards are DOA or fail shortly after entering service, due to poor
QC.  This is one of the many problems one avoids with higher quality
boards, which are usually more expensive.  A twist on the above axiom,
"pay more, get more".

> The rule is not "buy expensive", 

Of course not.

> it's "avoid buying (known-)broken". Maybe

This is not knowable but for compatibility.  All the $15 cards have a
higher DOA/failure rate than the high quality cards.  By that logic one
should always avoid the cheap cards.  Coincidentally, this is one of the
reasons I recommend against them.  One is gambling whether his new Syba
is going to be DOA.  You can be pretty damn sure your new LSI won't be
DOA or fail after one week.

> it's more difficult to come across a broken card in the expensive segment.
> Maybe it's not. In any case my suggestion is to just do some research before
> you buy, identify broken chips and cards, avoid those, and save yourself those
> $185 or whatever.

With hundreds of SAS/SATA HBAs on the market, as with just about any
product, most people do not have the time to research every small
purchase.  Again, the axiom comes into play: "You get what you pay for".

>> First, there are not "enterprise" controllers that use the Marvell SAS
>> chip, period.  It's a low/mid cost, entry level 8 port SAS ASIC.  The
>> cards based on it a not crappy, but actually pretty decent.  They run
>> very well with Windows, *BSD, SCO, NetWare, and other OSes.  It just
>> happens that the Linux driver, yes mv_sas, totally sucks
> 
> The end result for the user is the same, they accidentally buy that card, they
> use GNU/Linux, they have to use that sucky driver. The best choice is to avoid
> buying the card in the first place, but how do you know you should, if you
> didn't do the research (see above) and just looked at the price tag.

You ask around, which smart people do, on various lists.  Or they suffer.

You stated the ASIC itself was junk.  I was simply correcting that
point.  Note what I recently stated on the linux-ide list:

On 7/29/2012 4:24 AM, Stan Hoeppner wrote:
> Speaking of which, don't even look at the $110 8 port Supermicro
> SAS/SATA controller.  It uses the Marvell SAS chip.  Although the chip
> itself is fine and works with Windows, the Linux driver *will* eat
> your data

I've made the same statement dozens of times on the various LKML sub lists.

>>> Also, if you only use the mdadm software RAID, getting an enterprise hardware
>>> RAID controller is truly a waste of money, unless you really need the extra
>>> port density.
>>
>> That logic is flawed.  If you have an enterprise hardware RAID
>> controller you're not going to use md/RAID, unless you're stitching LUNs
>> together with linear or RAID0, as I often do.  In fact that's my most
>> frequent use of md/RAID--concatenation of hardware RAID or SAN LUNs.
> 
> I do not recommend using hardware RAID. It locks you into one card/vendor,
> usually is much less flexible than mdadm, and often even provides lower
> performance. See http://linux.yyz.us/why-software-raid.html

Of course not.  You don't work in or support an environment that
benefits from it.  Tell me how well md/RAID works for you the next time
you need to setup 4 IBM Bladecenters each w/14 dual socket blades
booting ESX from FC SAN, and hosting hundreds of Linux guests.

I can have a Nexsan E60+E60x w/120 SAS drives configured into 6x 20
drive RAID10 volumes, carve up all my LUNs and export them, booting the
nodes within a few hours, including racking the two chassis and
inserting all the drives.

How many days (or weeks) would it take you part spec and order, build,
test, verify, and put into production a roughly equivalent 120 SAS drive
self built FC LUN serving SAN array, using md/RAID with DAS
controllers/drives on the back end?  A long damn time, if you could even
do it.  There are no guides out there for this scenario, though there
are a few for iSCSI.

This is precisely why proprietary hardware RAID sells, and precisely why
md/RAID isn't the solution for everything, but quite the opposite.  In
fact, md/RAID sees almost zero use in the business/corporate world in
the U.S., probably the same in other countries.  md/RAID is great for
what it does, but there is so much it simply cannot do, from the storage
network standpoint, or simply can't do without a boat load of custom
development.

-- 
Stan

Re: OT: silent data corruption reading from hard drives

[Iustin Pop]

On Thu, Aug 02, 2012 at 02:06:34PM +0600, Roman Mamedov wrote:
> I do not recommend using hardware RAID. It locks you into one card/vendor,
> usually is much less flexible than mdadm, and often even provides lower
> performance. See http://linux.yyz.us/why-software-raid.html

While that *can* be the case, I haven't ever seen md beating a HW RAID
adapter with battery-backed write-cache in certain metadata-intensive
workloads, so as usual it depends on what kind of HW raid you have, and
what performance you measure. Broad descriptions are not very helpful.

regards,
iustin

Re: OT: silent data corruption reading from hard drives

[Drew]

> On a related note, is there a specific model card you would suggest that
> fits the following:
> a) 4 or 8 ports
> b) PCI (for older systems)
> c) affordable (note, not cheap, but not expensive)
> d) does not need to support RAID since we will use software raid anyway

A good solid card I've found is the Promise SATA300TX4. 4port SATA2
controller that fits in a standard PCI slot.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: OT: silent data corruption reading from hard drives

[listy]

Thanks for the replies, everyone.  I didn't expect the debate on 
controller price ranges, though.  :)

My main concern was not really the price of the hardware ("OMG why 
doesn't my $15 card work?!?"), but that the entire setup, hardware + 
software, allows silent data corruption to occur. It's not hard to 
imagine that something similar might happen with an $800 controller 
card, for whatever reason.  And how would the user know, unless they 
noticed bad data, or happened to look at the mismatch_cnt?

(And in my case, I think that, once the weekly raid-check runs and 
updates mismatch_cnt, it's already too late, as the check process has 
potentially rewritten bad data to the drives based on the bogus reads.
Is that correct?   My array is RAID-6, btw.)

Anyway, I'm going give up on SI-based controllers for now and try a
StarTech PEXSAT34, as the Marvell 88SE9128 is supported natively by
the kernel's AHCI driver.

Thanks,
matt

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

well i think the integrity is know, but it´s not fully needed since
the security isn´t a problem we can buy secure sata/sas
controlers/disks
the main problem will be in some days when we are using SoC systems
and we only have USB to connect a harddrive... maybe when this become
more popular we will see a development of a module to have data
integrity (silient corruption detection and maybe repair)

2012/8/2  <listy@fastmail.fm>:
> Thanks for the replies, everyone.  I didn't expect the debate on
> controller price ranges, though.  :)
>
> My main concern was not really the price of the hardware ("OMG why
> doesn't my $15 card work?!?"), but that the entire setup, hardware +
> software, allows silent data corruption to occur. It's not hard to
> imagine that something similar might happen with an $800 controller
> card, for whatever reason.  And how would the user know, unless they
> noticed bad data, or happened to look at the mismatch_cnt?
>
> (And in my case, I think that, once the weekly raid-check runs and
> updates mismatch_cnt, it's already too late, as the check process has
> potentially rewritten bad data to the drives based on the bogus reads.
> Is that correct?   My array is RAID-6, btw.)
>
> Anyway, I'm going give up on SI-based controllers for now and try a
> StarTech PEXSAT34, as the Marvell 88SE9128 is supported natively by
> the kernel's AHCI driver.
>
> Thanks,
> matt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Jeff Johnson]

The only ways I know of to currently detect/repair silent data 
corruption are via the use of T10-DIF on SAS drives with 520-byte 
sectors and embedded per block CRCs (bytes 513-520) or via a patented 
algorithm used in a commercial Linux software RAID product 
(www.streamscale.com).

Neither approach is cost effective for small or personal use RAID 
applications.


On 8/2/12 10:04 AM, Roberto Spadim wrote:
> well i think the integrity is know, but it´s not fully needed since
> the security isn´t a problem we can buy secure sata/sas
> controlers/disks
> the main problem will be in some days when we are using SoC systems
> and we only have USB to connect a harddrive... maybe when this become
> more popular we will see a development of a module to have data
> integrity (silient corruption detection and maybe repair)
>

-- 
------------------------------
Jeff Johnson
Manager
Aeon Computing

jeff.johnson@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x101   f: 858-412-3845
m: 619-204-9061

/* New Address */
4170 Morena Boulevard, Suite D - San Diego, CA 92117

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

On Thu, 02 Aug 2012 10:13:55 -0700
Jeff Johnson <jeff.johnson@aeoncomputing.com> wrote:

> The only ways I know of to currently detect/repair silent data 
> corruption are via the use of T10-DIF on SAS drives with 520-byte 
> sectors and embedded per block CRCs (bytes 513-520) or via a patented 
> algorithm used in a commercial Linux software RAID product 
> (www.streamscale.com).

Well, you can simply use BTRFS RAID1/RAID10 (RAID5/6 to come some time soon).
It has per block checksums and auto-healing from other drive(s) if data on
some drive turns out not to match the checksums anymore.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

yes, that´s what i know too
the point is  develop a algorithm to make this in block level like a md device
it write to disk and check if it was write ok
it read from disk and check parity, if it´s wrong read again, after X
times reading wrong report it as a badblock (now a silent corruption
becomes a well know badblock disk corruption)

i think that´s all... just need someone to implement it, some ideas:

// create:
mdadm --create /dev/md0 --level=integrity /dev/sda1

// check:
echo "check" > /sys/block/md0/md/sync_action

// repair: (like harddisks badblock recovery, mark the block as
badblock, and it will never be used)
echo "repair" > /sys/block/md0/md/sync_action

when using it with raid1 raid10 or any other mirror mdadm device, the
device with silent corruption can be reported (checksum don´t match
data) as badblock (mdadm have news features to deal with badblocks...)
the disk with badblock will be ignored and read will be done in next
good disk...

we could use a md5 checksum (32bits?! - 4 bytes) and data with less
information, example today block is 512bytes.. change it to 508bytes
of data + 4 bytes of checksum

i think that´s all...


2012/8/2 Jeff Johnson <jeff.johnson@aeoncomputing.com>:
> The only ways I know of to currently detect/repair silent data corruption
> are via the use of T10-DIF on SAS drives with 520-byte sectors and embedded
> per block CRCs (bytes 513-520) or via a patented algorithm used in a
> commercial Linux software RAID product (www.streamscale.com).
>
> Neither approach is cost effective for small or personal use RAID
> applications.
>
>
> On 8/2/12 10:04 AM, Roberto Spadim wrote:
>>
>> well i think the integrity is know, but it愀 not fully needed since
>> the security isn愒 a problem we can buy secure sata/sas
>>
>> controlers/disks
>> the main problem will be in some days when we are using SoC systems
>> and we only have USB to connect a harddrive... maybe when this become
>> more popular we will see a development of a module to have data
>> integrity (silient corruption detection and maybe repair)
>>
>
> --
> ------------------------------
> Jeff Johnson
> Manager
> Aeon Computing
>
> jeff.johnson@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x101   f: 858-412-3845
> m: 619-204-9061
>
> /* New Address */
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

btrfs and jfs have silent data corruption protection, but it could be
nice to have this at block layer, since some databases (innodb at
mysql) can use block device to allocate information (as a performace
boost), and since i know, it don´t have silent data corruption
protection...
a solution at block device is better than filesystem? yes and no, but
at block device we can use with any filesystem, even with fat16 or
other old filesystem
we could add more features in future (like jornaling) but i think just
data integrity layer is nice... there´s others features like
reallocation (when write to a silent write detected block, realloc it
to another block like good SSD do)



2012/8/2 Roman Mamedov <rm@romanrm.ru>:
> On Thu, 02 Aug 2012 10:13:55 -0700
> Jeff Johnson <jeff.johnson@aeoncomputing.com> wrote:
>
>> The only ways I know of to currently detect/repair silent data
>> corruption are via the use of T10-DIF on SAS drives with 520-byte
>> sectors and embedded per block CRCs (bytes 513-520) or via a patented
>> algorithm used in a commercial Linux software RAID product
>> (www.streamscale.com).
>
> Well, you can simply use BTRFS RAID1/RAID10 (RAID5/6 to come some time soon).
> It has per block checksums and auto-healing from other drive(s) if data on
> some drive turns out not to match the checksums anymore.
>
> --
> With respect,
> Roman
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> "Stallman had a printer,
> with code he could not see.
> So he began to tinker,
> and set the software free."



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[listy]

On Thu, Aug 2, 2012, at 13:33, Phil Turmel wrote:
> You really do need to have a process check mismatch_cnt after your
> weekly check completes.


With Fedora, I get an email Monday morning, after the raid-check, which 
warns of a non-zero mismatch_cnt.


> Depends.  If you use "repair", bad data will be propagated.  If you use
> "check", it'll just be reported.


Ah, okay, good.  I thought I'd read here a while back that "check" & 
"repair" do the same thing.


> I've seen a great deal of good advice here, but nothing about the system
> component least likely to be protected in an "economy" system:
> RAM.  Does your Mobo have ECC ram?  

Good point.  It does not.  Might be time for me to upgrade to a mobo with 
ECC support.


> does your kernel support logging, and are you monitoring the
> machine check log?

klogd is not running, but I think the latest rsyslog handles the kernel 
messages.  There was nothing in the logs related to my corruption issues, 
however.

> Hard drives write extensive ECC payloads to catch corruptions there;
> SATA and SAS protocols have CRC checks on every frame transferred; the
> PCIe bus uses CRC checks on each lane, with low-level encoding very
> similar to SATA.  Even modern processors are using PCIe-style encoded

Thanks, this is good info, and kind of gets at my thinking when I posted my 
initial question.  In a typical consumer hardware setup, with a current 
linux kernel, do I have to take any steps to enable these kinds of checks?
Can the kernel log any failed checks at the levels you mention?  I guess my 
confusion with my silent data corruption issues stems from my naive 
assumption that all the various data transfers happening would have some 
way of detecting or flagging the bad reads as they happened.

But maybe as you suggest, my issue is related to memory, and ECC might help 
in the future?

Thanks,
matt

Re: OT: silent data corruption reading from hard drives

[Phil Turmel]

Hi Matt,

I now see that I hit the wrong "reply" button--my apologies to the list.
 You've quoted the important stuff, though, so I won't resend.

On 08/02/2012 02:32 PM, listy@fastmail.fm wrote:
> On Thu, Aug 2, 2012, at 13:33, Phil Turmel wrote:
>> You really do need to have a process check mismatch_cnt after your
>> weekly check completes.
> 
> 
> With Fedora, I get an email Monday morning, after the raid-check, which 
> warns of a non-zero mismatch_cnt.

Good to know.  I'm on gentoo, and I use my own script in logwatch, so
I'm not familiar with the various distros practice on this.

>> Depends.  If you use "repair", bad data will be propagated.  If you use
>> "check", it'll just be reported.
> 
> 
> Ah, okay, good.  I thought I'd read here a while back that "check" & 
> "repair" do the same thing.
> 
> 
>> I've seen a great deal of good advice here, but nothing about the system
>> component least likely to be protected in an "economy" system:
>> RAM.  Does your Mobo have ECC ram?  
> 
> Good point.  It does not.  Might be time for me to upgrade to a mobo with 
> ECC support.

In my opinion, any corruption noticed in a non-ECC system is most likely
due to the RAM.  You really need to run memtest86 on your system,
preferably for 24 hours or more.

>> does your kernel support logging, and are you monitoring the
>> machine check log?
> 
> klogd is not running, but I think the latest rsyslog handles the kernel 
> messages.  There was nothing in the logs related to my corruption issues, 
> however.

I meant logging of ECC RAM correction events (warnings) and
uncorrectable errors.  Your kernel has to support that.  I would be
shocked if Fedora didn't support it.  You also need the user space
"mcelog" package.  "mce" ==> "Machine Check Exception"

>> Hard drives write extensive ECC payloads to catch corruptions there;
>> SATA and SAS protocols have CRC checks on every frame transferred; the
>> PCIe bus uses CRC checks on each lane, with low-level encoding very
>> similar to SATA.  Even modern processors are using PCIe-style encoded
> 
> Thanks, this is good info, and kind of gets at my thinking when I posted my 
> initial question.  In a typical consumer hardware setup, with a current 
> linux kernel, do I have to take any steps to enable these kinds of checks?
> Can the kernel log any failed checks at the levels you mention?  I guess my 
> confusion with my silent data corruption issues stems from my naive 
> assumption that all the various data transfers happening would have some 
> way of detecting or flagging the bad reads as they happened.

You won't get ram corruption error reports if you don't have ECC ram.
Data transfer errors between CPU and chipset might generate machine
check exceptions, but if not recoverable, the machine just dies.  Errors
on PCIe lanes and SATA/SAS connections cause retransmissions until
success or the driver times out.  That would show up in dmesg.

> But maybe as you suggest, my issue is related to memory, and ECC might help 
> in the future?

You don't have to guess.  Boot into memtest86 and see.  And yes, any
machine handling data you really care about should have ECC ram.

Phil

Re: OT: silent data corruption reading from hard drives

[Peter Grandi]

[ ... ]

> In my opinion, any corruption noticed in a non-ECC system is
> most likely due to the RAM.

That's pretty common, but many disk drive models also have bugs,
and most hw RAID host adapters have many (terrible) bugs.

> You really need to run memtest86 on your system, preferably
> for 24 hours or more.

Even that is not conclusive. Some "memory" errors are due to
activity/noise spikes on the PCI/PCIe bus due to hw bugs or
poorly electrically designed cards.

>>> Hard drives write extensive ECC payloads to catch
>>> corruptions there; SATA and SAS protocols have CRC checks on
>>> every frame transferred;

A warning to the masses: USB mass storage is weak as to this and
in particular as to error recovery, and most USB chipsets
(especially USB-drive ones, but also motherboard ones) are
massively buggy.

>>> the PCIe bus uses CRC checks on each lane, with low-level
>>> encoding very similar to SATA.  Even modern processors are
>>> using PCIe-style encoded [ ... ]

> [ ... ] machine handling data you really care about

... should have end-to-end verification, that is the data itself
should be checksummed at least to detect corruption. For example
by putting it into checksummed containers (even just ZIP without
compression).

> should have ECC ram.

Oh yes, and any machine should have ECC RAM as the cost is
really modest. Unfortunately the usual evil marketers like to
segment artificially the market into cheap stuff without ECC and
premium stuff with ECC, and will not put ECC into cheap stuff to
avoid tempting business customers to buy it instead of the
premium stuff.

Re: OT: silent data corruption reading from hard drives

[Oliver Schinagl]

On 15-08-12 23:55, Peter Grandi wrote:
> [ ... ]
>
>> In my opinion, any corruption noticed in a non-ECC system is
>> most likely due to the RAM.
> That's pretty common, but many disk drive models also have bugs,
> and most hw RAID host adapters have many (terrible) bugs.
>
>> You really need to run memtest86 on your system, preferably
>> for 24 hours or more.
> Even that is not conclusive. Some "memory" errors are due to
> activity/noise spikes on the PCI/PCIe bus due to hw bugs or
> poorly electrically designed cards.
>
>>>> Hard drives write extensive ECC payloads to catch
>>>> corruptions there; SATA and SAS protocols have CRC checks on
>>>> every frame transferred;
> A warning to the masses: USB mass storage is weak as to this and
> in particular as to error recovery, and most USB chipsets
> (especially USB-drive ones, but also motherboard ones) are
> massively buggy.
>
>>>> the PCIe bus uses CRC checks on each lane, with low-level
>>>> encoding very similar to SATA.  Even modern processors are
>>>> using PCIe-style encoded [ ... ]
>> [ ... ] machine handling data you really care about
> ... should have end-to-end verification, that is the data itself
> should be checksummed at least to detect corruption. For example
> by putting it into checksummed containers (even just ZIP without
> compression).
>
>> should have ECC ram.
> Oh yes, and any machine should have ECC RAM as the cost is
> really modest. Unfortunately the usual evil marketers like to
> segment artificially the market into cheap stuff without ECC and
> premium stuff with ECC, and will not put ECC into cheap stuff to
> avoid tempting business customers to buy it instead of the
> premium stuff.
While I agree that all machine's should have ECC Ram (there are still 
some people think its not worth it), last time I checked on newegg, I 
found ECC prices not that much higher. My servers both run happily with 
ECC ram.

As for data corruption, I've also been there and know it simply just 
happens. Yes I had shitty IDE drives on a shitty 'rocketraid 404' 
controller, but that's no excuse to simply assume all data will always 
be right. Maybe in a few years from now, we'll have some 'open cores' 
for properly designed almost bug free hardware :)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OT: silent data corruption reading from hard drives

[Roberto Spadim]

well this discussion is very nice...
but let´s get back to our problem?

we can have data integrity with high cost (for some guys it´s high for
others it doesn´t matter), using 'especial' hardware...
what we can do to get integrity higher with a low cost hardware? any
software solution could gime more data integrity with more CRC or
other method of error check? could we develop it at block device level
(md device)?
i think that´s the point
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html