3 way mirror

3 way mirror

[Gandalf Corvotempesta]

let's assume a 3 way mirror (raid1 with 3 disks)
One disk got a bad sector detrcted by smartd
what happens trying to read or write to that sector?

is md smart enough to read from the other 2 disks and serve consistant data?

in other words, can i delay the disk replacement for a couple of days
(i've ordered the disk today, will came tuesday) ?

Re: 3 way mirror

[Geoff Back]

On 11/09/2022 10:08, Gandalf Corvotempesta wrote:
> let's assume a 3 way mirror (raid1 with 3 disks)
> One disk got a bad sector detrcted by smartd
> what happens trying to read or write to that sector?
>
> is md smart enough to read from the other 2 disks and serve consistant data?
>
> in other words, can i delay the disk replacement for a couple of days
> (i've ordered the disk today, will came tuesday) ?

If this is a recorded grown defect or block reallocation, the drive will
continue to operate normally anyway; the disk will have remapped the
sector internally to one of a few thousand spares kept for the purpose.

However for failed sectors in general, and especially ones that have not
been remapped, yes if MD tries to read from the block and the disk
reports a read error it will automatically redirect to another drive.  I
believe it will also attempt to rewrite the bad sector with the data
from the other drive in order to repair the mirror.  Write operations
will be sent to all three drives and the drive with the bad sector will
normally automatically remap to a spare sector if the internal write
fails.  Once available spare sectors are exhausted the drive will fail
the write, at which point md will fail the drive out of the array.

Of course, this is dependent on your disks returning errors in a timely
fashion rather than locking up on extensive retries.  If you are using
NAS-level or enterprise drives, you should be good, but if you are using
desktop-grade drives they are likely by default to retry for so long
that the kernel times out and the drive drops off the bus instead of
reporting an error.  At that point, of course, md sees the drive as failed.

Cheers,

Geoff.

Geoff Back
What if we're all just characters in someone's nightmares?

Re: 3 way mirror

[Wols Lists]

On 11/09/2022 10:08, Gandalf Corvotempesta wrote:
> let's assume a 3 way mirror (raid1 with 3 disks)
> One disk got a bad sector detrcted by smartd
> what happens trying to read or write to that sector?

I'm guessing when linux goes to read the data, the read will fail.
> 
> is md smart enough to read from the other 2 disks and serve consistant data?

Very much so, PROVIDED linux returns a read error for the disk.
> 
> in other words, can i delay the disk replacement for a couple of days
> (i've ordered the disk today, will came tuesday) ?

What sort of error? A new disk *may* be overkill ...


Firstly, this is a disk level problem. The whole point of raid is to 
protect your data from disk level problems :-)

Secondly, this is a disk level problem. The dodgy sector might not even 
be in use, so there's no data there to lose.

Thirdly, this is a disk level problem. It may be a simple case of the 
disk needs to rewrite the data and relocate the sector and everything 
will be hunky-dory again. BUT IT CAN'T DO THAT UNTIL LINUX GIVES IT NEW 
FRESH DATA.

So you need to quiesce the disk (basically, shut down as many processes 
as you can, maybe do this overnight), and run a scrub. That will tell 
you if linux/mdraid thinks there's a problem.

Then re-run smartd and see if it's fixed the problem.

Then look at the smartd output and ask yourself "do I really need a new 
disk?". I wouldn't send the new one back. Depending on how well you are 
off for disk space and SATA ports, now you've got the new disk, if the 
old one is still good I'd go for a 3-disk raid-5 plus spare. That's my 
current setup.

Cheers,
Wol

Re: 3 way mirror

[Gandalf Corvotempesta]

Il giorno dom 11 set 2022 alle ore 12:08 Wols Lists
<antlists@youngman.org.uk> ha scritto:
> What sort of error? A new disk *may* be overkill ...

It's a smart error, saying the sector at LBA can't be read, during the
scheduled long smart test.

> Then look at the smartd output and ask yourself "do I really need a new
> disk?". I wouldn't send the new one back. Depending on how well you are
> off for disk space and SATA ports, now you've got the new disk, if the
> old one is still good I'd go for a 3-disk raid-5 plus spare. That's my
> current setup.

A new (better) disk, an SSD, would cost 150 euros. Better safe than
sorry, so usually I change disks
as soon as I get a read error. (also, this disk is around 50k hours.....)

Re: 3 way mirror

[Reindl Harald]

Am 11.09.22 um 11:08 schrieb Gandalf Corvotempesta:
> let's assume a 3 way mirror (raid1 with 3 disks)
> One disk got a bad sector detrcted by smartd
> what happens trying to read or write to that sector?
> 
> is md smart enough to read from the other 2 disks and serve consistant data?

that's the whole point of RAID

> in other words, can i delay the disk replacement for a couple of days

just throw it out of the array - on a RAID 1 with two disk you have 
still redundancy and i won't trust a disk which recognized that it has 
errors on it's own

> (i've ordered the disk today, will came tuesday)?

what's the point of "can i delay" when you have no other options? :-)

Re: 3 way mirror

[Gandalf Corvotempesta]

Il giorno dom 11 set 2022 alle ore 12:52 Reindl Harald
<h.reindl@thelounge.net> ha scritto:
> just throw it out of the array - on a RAID 1 with two disk you have
> still redundancy and i won't trust a disk which recognized that it has
> errors on it's own

Why ? if it's just a single sector, the rest of the disk could be used
properly until the replacement.

> what's the point of "can i delay" when you have no other options? :-)

The option is: i've ordered the new disk, it should arrive this thursday.
If i'm still on the safe side, i'll wait, if not, i'll replace with
the first disk I have here, even if slower

Re: 3 way mirror

[Reindl Harald]

Am 11.09.22 um 13:35 schrieb Gandalf Corvotempesta:
> Il giorno dom 11 set 2022 alle ore 12:52 Reindl Harald
> <h.reindl@thelounge.net> ha scritto:
>> just throw it out of the array - on a RAID 1 with two disk you have
>> still redundancy and i won't trust a disk which recognized that it has
>> errors on it's own
> 
> Why ? if it's just a single sector, the rest of the disk could be used
> properly until the replacement.

i simply don't trust drives which have thrown errors
you don't lose anything when you push it out of the array

>> what's the point of "can i delay" when you have no other options? :-)
> 
> The option is: i've ordered the new disk, it should arrive this thursday.
> If i'm still on the safe side, i'll wait, if not, i'll replace with
> the first disk I have here, even if slower

you are clearly on the safe side with 3 mirrors and every rebuild wears 
out the remaining drives, 1 out of the 3 would be enough since they hold 
identical data

----------------------

the only thing one should think of: are all 3 drives identical types 
with identical age - then it's more likely that they fail all within a 
short time

that's why i try to mix different disks in a mirror to avoid failing 
both sides of a mirror due firmware-error or a charge of bad disks

----------------------

as i replaced my HDD to SSD i removed two drives out of my RAID10, as it 
didn't boot exchange two of them

so at that point 1 mirror of each stripe was a HDD and one a new SSD, 
after one year i replaced the remaining two disks

2 x 850 EVO 2TB
2 x 860 EVO 2TB

/dev/sda: GB Written: 72.020
/dev/sdb: GB Written: 69.709
/dev/sdc: GB Written: 91.367
/dev/sdd: GB Written: 94.134

pretty impossible that both sides of the same mirror will fail within a 
few hours

Re: 3 way mirror

[Wols Lists]

On 11/09/2022 13:20, Reindl Harald wrote:
> 
> 
> Am 11.09.22 um 13:35 schrieb Gandalf Corvotempesta:
>> Il giorno dom 11 set 2022 alle ore 12:52 Reindl Harald
>> <h.reindl@thelounge.net> ha scritto:
>>> just throw it out of the array - on a RAID 1 with two disk you have
>>> still redundancy and i won't trust a disk which recognized that it has
>>> errors on it's own
>>
>> Why ? if it's just a single sector, the rest of the disk could be used
>> properly until the replacement.
> 
> i simply don't trust drives which have thrown errors
> you don't lose anything when you push it out of the array

That's called paranoia - ALL drives have hiccups when there's absolutely 
nothing wrong with them. What are the manufacturer's error figures? 
Expect at least one error every two or three complete passes of pretty 
much every large big disk nowadays?

What's the point of having thousands of spare sectors, if you chuck the 
drive in the trash at the first hint of trouble?
> 
>>> what's the point of "can i delay" when you have no other options? :-)
>>
>> The option is: i've ordered the new disk, it should arrive this thursday.
>> If i'm still on the safe side, i'll wait, if not, i'll replace with
>> the first disk I have here, even if slower
> 
> you are clearly on the safe side with 3 mirrors and every rebuild wears 
> out the remaining drives, 1 out of the 3 would be enough since they hold 
> identical data
> 
> /dev/sdd: GB Written: 94.134
> 
> pretty impossible that both sides of the same mirror will fail within a 
> few hours
> 
You clearly have no clue why London Buses arrive in threes ... :-)

Cheers,
Wol

Re: 3 way mirror

[Gandalf Corvotempesta]

Il giorno dom 11 set 2022 alle ore 20:08 Wols Lists
<antlists@youngman.org.uk> ha scritto:
> That's called paranoia - ALL drives have hiccups when there's absolutely
> nothing wrong with them. What are the manufacturer's error figures?
> Expect at least one error every two or three complete passes of pretty
> much every large big disk nowadays?
>
> What's the point of having thousands of spare sectors, if you chuck the
> drive in the trash at the first hint of trouble?

even here we use to replace disk as soon as we get a read error.
The spare sectors are useful to give some time before the new disk arrives.

Re: 3 way mirror

[Phil Turmel]

Hi Wol,

On 9/11/22 06:08, Wols Lists wrote:

[trim /]

> old one is still good I'd go for a 3-disk raid-5 plus spare. That's my 
> current setup.

Eww.  Don't do that.  Always convert raid5+spare to raid6.

> Cheers,
> Wol

Re: 3 way mirror

[Wols Lists]

On 12/09/2022 01:58, Phil Turmel wrote:
> Hi Wol,
> 
> On 9/11/22 06:08, Wols Lists wrote:
> 
> [trim /]
> 
>> old one is still good I'd go for a 3-disk raid-5 plus spare. That's my 
>> current setup.
> 
> Eww.  Don't do that.  Always convert raid5+spare to raid6.
> 
At the moment the fourth drive is, in my mind, a transient backup. I 
don't want to make it an actual part of the array in case I want to 
remove it.

I'm thinking of getting a second matching drive, but seeing as I've 
bought 16TB of drive space in the recentish past, another 8TB seems 
excessive :-)

Cheers,
Wol

Re: 3 way mirror

[Reindl Harald]

Am 11.09.22 um 20:08 schrieb Wols Lists:
> On 11/09/2022 13:20, Reindl Harald wrote:
>>
>>
>> Am 11.09.22 um 13:35 schrieb Gandalf Corvotempesta:
>>> Il giorno dom 11 set 2022 alle ore 12:52 Reindl Harald
>>> <h.reindl@thelounge.net> ha scritto:
>>>> just throw it out of the array - on a RAID 1 with two disk you have
>>>> still redundancy and i won't trust a disk which recognized that it has
>>>> errors on it's own
>>>
>>> Why ? if it's just a single sector, the rest of the disk could be used
>>> properly until the replacement.
>>
>> i simply don't trust drives which have thrown errors
>> you don't lose anything when you push it out of the array
> 
> That's called paranoia

that's called common sense

> ALL drives have hiccups when there's absolutely 
> nothing wrong with them. What are the manufacturer's error figures? 
> Expect at least one error every two or three complete passes of pretty 
> much every large big disk nowadays?
i don't expect any SMART error at all and if one hits after years of 
uptime fine that the drive don't fail completly - but would i trust it? no!

Re: 3 way mirror

[Gandalf Corvotempesta]

Il giorno lun 12 set 2022 alle ore 10:16 Reindl Harald
<h.reindl@thelounge.net> ha scritto:
> i don't expect any SMART error at all and if one hits after years of
> uptime fine that the drive don't fail completly - but would i trust it? no!

Same here.

Re: 3 way mirror

[Roman Mamedov]

On Mon, 12 Sep 2022 10:16:55 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:

> > That's called paranoia
> 
> that's called common sense
> 
> > ALL drives have hiccups when there's absolutely 
> > nothing wrong with them. What are the manufacturer's error figures? 
> > Expect at least one error every two or three complete passes of pretty 
> > much every large big disk nowadays?
> i don't expect any SMART error at all and if one hits after years of 
> uptime fine that the drive don't fail completly - but would i trust it? no!

I have multiple drives (Hitachi) which have developed around 3 to 10
reallocated sectors, and then just continue to work for years, not increasing
those further. Throwing them away would be uneconomical. Not always a
reallocated sector is a sign of impeding failure, what you need to look for is
if there is more than one, and the count is increasing quickly.

Generally though, if you are way too concerned about individual drive failures,
it could be a sign that you do not have a robust enough backup scheme in place.

-- 
With respect,
Roman

Re: 3 way mirror

[Reindl Harald]

Am 12.09.22 um 11:26 schrieb Roman Mamedov:
> On Mon, 12 Sep 2022 10:16:55 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
> 
>>> That's called paranoia
>>
>> that's called common sense
>>
>>> ALL drives have hiccups when there's absolutely
>>> nothing wrong with them. What are the manufacturer's error figures?
>>> Expect at least one error every two or three complete passes of pretty
>>> much every large big disk nowadays?
>> i don't expect any SMART error at all and if one hits after years of
>> uptime fine that the drive don't fail completly - but would i trust it? no!
> 
> I have multiple drives (Hitachi) which have developed around 3 to 10
> reallocated sectors, and then just continue to work for years, not increasing
> those further. Throwing them away would be uneconomical. 

better safe than sorry

in the past 15 years i replaced 6 drives on 8 machines (each 4 disks) 
while some are running since 2011

on every location with machines i am resposible for is at least one 
replacment disk, switch, order a new reserve disk, case closed

> Generally though, if you are way too concerned about individual drive failures,
> it could be a sign that you do not have a robust enough backup scheme in place

my backup scheme is perfect but my personal lifetime is much more worth 
than a disk costs these days - the backups are regulary testet and in 
the past 20 years i didn't need them which is a good thing

no backup on this planet contains the last 30 minutes of work besides 
that outtakes typically happen in the moment where you have no time to 
deal with such events

come on a restore a backup in a production environment, your customers 
will be not much thankful when they lose the last 30 minutes of all 
mails and work....

Re: 3 way mirror

[Phil Turmel]

If you suffer a drive failure, your odds of a successful rebuild onto 
that spare are decidedly mixed.

https://marc.info/?l=linux-raid&m=139050322510249&w=2

On 9/12/22 04:06, Wols Lists wrote:
> On 12/09/2022 01:58, Phil Turmel wrote:
>> Hi Wol,
>>
>> On 9/11/22 06:08, Wols Lists wrote:
>>
>> [trim /]
>>
>>> old one is still good I'd go for a 3-disk raid-5 plus spare. That's 
>>> my current setup.
>>
>> Eww.  Don't do that.  Always convert raid5+spare to raid6.
>>
> At the moment the fourth drive is, in my mind, a transient backup. I 
> don't want to make it an actual part of the array in case I want to 
> remove it.
> 
> I'm thinking of getting a second matching drive, but seeing as I've 
> bought 16TB of drive space in the recentish past, another 8TB seems 
> excessive :-)
> 
> Cheers,
> Wol

Re: 3 way mirror

[Phil Turmel]

On 9/12/22 05:26, Roman Mamedov wrote:
> On Mon, 12 Sep 2022 10:16:55 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
> 
>>> That's called paranoia
>>
>> that's called common sense
>>
>>> ALL drives have hiccups when there's absolutely
>>> nothing wrong with them. What are the manufacturer's error figures?
>>> Expect at least one error every two or three complete passes of pretty
>>> much every large big disk nowadays?
>> i don't expect any SMART error at all and if one hits after years of
>> uptime fine that the drive don't fail completly - but would i trust it? no!
> 
> I have multiple drives (Hitachi) which have developed around 3 to 10
> reallocated sectors, and then just continue to work for years, not increasing
> those further. Throwing them away would be uneconomical. Not always a
> reallocated sector is a sign of impeding failure, what you need to look for is
> if there is more than one, and the count is increasing quickly.

Concur. I do not replace drives for reallocated sectors until they reach 
double digits.  Modern drives are expected to do this.

> Generally though, if you are way too concerned about individual drive failures,
> it could be a sign that you do not have a robust enough backup scheme in place.

Heh.

Phil