Nothing wrong, but is my website advice wonky?

Nothing wrong, but is my website advice wonky?

[Wols Lists]

Finally upgrading my system to raid-5 - two Seagate Ironwolves and a 
Barracuda ... :-(

As per my own advice, I added the third drive as a spare, then grew the 
array to raid-5 in two separate commands.

Trying to track what's going on, "cat /proc/mdstat" just shows two 
drives as sync'ing. "mdadm --detail" shows two active drives and a spare.

The drives are quite clearly working away - as I would expect.

So. What I *think* is happening is that my mirror is upgrading to a 
2-disk raid-5, and when that's finished it will add the spare and 
upgrade to a full 3-disk raid-5. Does that sound right?

What I *hoped* would happen (and thought *should* happen) was that it 
would spot the third drive, add it, and just resync straight away to 
full raid-5.

So at 7 or 8hrs per pass (3TB per drive) I'm now looking at my upgrade 
taking about 15 hours. Whoops.

So basically, does my scenario sound right? Would have explicitly 
changing raid-devices to 3 at the same time as converting to raid-5 
improved matters?

Okay, I'm going to build a new raid-testing computer in the near future 
so testing this sort of thing will be easy, but that is predicated on me 
finding enough time without upsetting the wife ...

Cheers,
Wol

Re: Nothing wrong, but is my website advice wonky?

[Wol]

Problem solved! My usual problem of missing the detail...

I'd described starting with a 2-drive mirror and converting it to a 
3-drive raid-5. I was actually starting with a 2-active-1-spare mirror ...

Cheers,
Wol

On 28/11/2021 09:25, Wols Lists wrote:
> Finally upgrading my system to raid-5 - two Seagate Ironwolves and a 
> Barracuda ... :-(
> 
> As per my own advice, I added the third drive as a spare, then grew the 
> array to raid-5 in two separate commands.
> 
> Trying to track what's going on, "cat /proc/mdstat" just shows two 
> drives as sync'ing. "mdadm --detail" shows two active drives and a spare.
> 
> The drives are quite clearly working away - as I would expect.
> 
> So. What I *think* is happening is that my mirror is upgrading to a 
> 2-disk raid-5, and when that's finished it will add the spare and 
> upgrade to a full 3-disk raid-5. Does that sound right?
> 
> What I *hoped* would happen (and thought *should* happen) was that it 
> would spot the third drive, add it, and just resync straight away to 
> full raid-5.
> 
> So at 7 or 8hrs per pass (3TB per drive) I'm now looking at my upgrade 
> taking about 15 hours. Whoops.
> 
> So basically, does my scenario sound right? Would have explicitly 
> changing raid-devices to 3 at the same time as converting to raid-5 
> improved matters?
> 
> Okay, I'm going to build a new raid-testing computer in the near future 
> so testing this sort of thing will be easy, but that is predicated on me 
> finding enough time without upsetting the wife ...
> 
> Cheers,
> Wol

Re: Nothing wrong, but is my website advice wonky?

[John Stoffel]

>>>>> "Wol" == Wol  <antlists@youngman.org.uk> writes:

Wol> Problem solved! My usual problem of missing the detail...

How was the problem solved?  Can you provide details on the before and
after setup?  It should be trivial to test with loop back devices.

Wol> I'd described starting with a 2-drive mirror and converting it to
Wol> a 3-drive raid-5. I was actually starting with a 2-active-1-spare
Wol> mirror ...

So if you had removed the spare before hand, would it have jumped
straight to a three drive RAID 5 setup?  And curious why you didn't
goto RAID6?  RAID5 just makes me nervous these days, I don't trust it
since drives tend to fail in waves.

But in my home system, I also mostly just mirror (or even triple
mirror!) my important data for both speed and ease of recovery.  

Wol> On 28/11/2021 09:25, Wols Lists wrote:
>> Finally upgrading my system to raid-5 - two Seagate Ironwolves and a 
>> Barracuda ... :-(
>> 
>> As per my own advice, I added the third drive as a spare, then grew the 
>> array to raid-5 in two separate commands.
>> 
>> Trying to track what's going on, "cat /proc/mdstat" just shows two 
>> drives as sync'ing. "mdadm --detail" shows two active drives and a spare.
>> 
>> The drives are quite clearly working away - as I would expect.
>> 
>> So. What I *think* is happening is that my mirror is upgrading to a 
>> 2-disk raid-5, and when that's finished it will add the spare and 
>> upgrade to a full 3-disk raid-5. Does that sound right?
>> 
>> What I *hoped* would happen (and thought *should* happen) was that it 
>> would spot the third drive, add it, and just resync straight away to 
>> full raid-5.
>> 
>> So at 7 or 8hrs per pass (3TB per drive) I'm now looking at my upgrade 
>> taking about 15 hours. Whoops.
>> 
>> So basically, does my scenario sound right? Would have explicitly 
>> changing raid-devices to 3 at the same time as converting to raid-5 
>> improved matters?
>> 
>> Okay, I'm going to build a new raid-testing computer in the near future 
>> so testing this sort of thing will be easy, but that is predicated on me 
>> finding enough time without upsetting the wife ...
>> 
>> Cheers,
>> Wol

Re: Nothing wrong, but is my website advice wonky?

[Wol]

On 28/11/2021 20:15, John Stoffel wrote:
>>>>>> "Wol" == Wol  <antlists@youngman.org.uk> writes:
> 
> Wol> Problem solved! My usual problem of missing the detail...
> 
> How was the problem solved?  Can you provide details on the before and
> after setup?  It should be trivial to test with loop back devices.
> 
> Wol> I'd described starting with a 2-drive mirror and converting it to
> Wol> a 3-drive raid-5. I was actually starting with a 2-active-1-spare
> Wol> mirror ...

So I converted it to raid-5, and because there were only two active 
drives, it carried on running as a degraded two-drive raid-5.

Once I brought the spare into service with --raid-devices=3, it started 
reshaping.

The website said to add the new drive and set raid-devices to three, I 
misread my own stuff and thought simply converting to raid-5 would bring 
in the spare.
> 
> So if you had removed the spare before hand, would it have jumped
> straight to a three drive RAID 5 setup? 

No - because I didn't tell it to use all three drives.

> And curious why you didn't
> goto RAID6?  RAID5 just makes me nervous these days, I don't trust it
> since drives tend to fail in waves.

Well, with two DVD drives and a floppy in an old chassis, I've only got 
room for three drives. Well, there should be room for four, but my 
drives are a bit too fat ... :-)

Plus (1) I've had no trouble so far with my drives (famous last words), 
(2) the drives are two new Ironwolves and an old Barracuda so I'm only 
likely to lose one so long as I keep an eye on things, and (3) all the 
drives have dm-integrity on them, so a corrupt drive should sort itself out.
> 
> But in my home system, I also mostly just mirror (or even triple
> mirror!) my important data for both speed and ease of recovery.
> 
The trouble with a plain mirror is if your data is corrupted ...

And if you've got raid you need to keep an eye on it. I'm planning to 
get an 8TB drive for backups ... probably configure it to snapshot once 
a week (I'm running lvm over raid over dm-integrity) and then use rsync 
to back up the volume over the network. Mind you I'm going to have to 
learn a lot of rsync - configure client/server and then get it to back 
up just the one file ...

Cheers,
Wol

> Wol> On 28/11/2021 09:25, Wols Lists wrote:
>>> Finally upgrading my system to raid-5 - two Seagate Ironwolves and a
>>> Barracuda ... :-(
>>>
>>> As per my own advice, I added the third drive as a spare, then grew the
>>> array to raid-5 in two separate commands.
>>>
>>> Trying to track what's going on, "cat /proc/mdstat" just shows two
>>> drives as sync'ing. "mdadm --detail" shows two active drives and a spare.
>>>
>>> The drives are quite clearly working away - as I would expect.
>>>
>>> So. What I *think* is happening is that my mirror is upgrading to a
>>> 2-disk raid-5, and when that's finished it will add the spare and
>>> upgrade to a full 3-disk raid-5. Does that sound right?
>>>
>>> What I *hoped* would happen (and thought *should* happen) was that it
>>> would spot the third drive, add it, and just resync straight away to
>>> full raid-5.
>>>
>>> So at 7 or 8hrs per pass (3TB per drive) I'm now looking at my upgrade
>>> taking about 15 hours. Whoops.
>>>
>>> So basically, does my scenario sound right? Would have explicitly
>>> changing raid-devices to 3 at the same time as converting to raid-5
>>> improved matters?
>>>
>>> Okay, I'm going to build a new raid-testing computer in the near future
>>> so testing this sort of thing will be easy, but that is predicated on me
>>> finding enough time without upsetting the wife ...
>>>
>>> Cheers,
>>> Wol

Re: Nothing wrong, but is my website advice wonky?

[John Stoffel]

>>>>> "Wol" == Wol  <antlists@youngman.org.uk> writes:

Wol> On 28/11/2021 20:15, John Stoffel wrote:
>>>>>>> "Wol" == Wol  <antlists@youngman.org.uk> writes:
>> 
Wol> Problem solved! My usual problem of missing the detail...
>> 
>> How was the problem solved?  Can you provide details on the before and
>> after setup?  It should be trivial to test with loop back devices.
>> 
Wol> I'd described starting with a 2-drive mirror and converting it to
Wol> a 3-drive raid-5. I was actually starting with a 2-active-1-spare
Wol> mirror ...

Wol> So I converted it to raid-5, and because there were only two active 
Wol> drives, it carried on running as a degraded two-drive raid-5.

Wol> Once I brought the spare into service with --raid-devices=3, it started 
Wol> reshaping.

Ah... that explains it.  

Wol> The website said to add the new drive and set raid-devices to three, I 
Wol> misread my own stuff and thought simply converting to raid-5 would bring 
Wol> in the spare.

>> So if you had removed the spare before hand, would it have jumped
>> straight to a three drive RAID 5 setup? 

Wol> No - because I didn't tell it to use all three drives.

>> And curious why you didn't
>> goto RAID6?  RAID5 just makes me nervous these days, I don't trust it
>> since drives tend to fail in waves.

Wol> Well, with two DVD drives and a floppy in an old chassis, I've
Wol> only got room for three drives. Well, there should be room for
Wol> four, but my drives are a bit too fat ... :-)

I understand.  We all have to work within our limits.  

Wol> Plus (1) I've had no trouble so far with my drives (famous last words), 
Wol> (2) the drives are two new Ironwolves and an old Barracuda so I'm only 
Wol> likely to lose one so long as I keep an eye on things, and (3) all the 
Wol> drives have dm-integrity on them, so a corrupt drive should sort itself out.

>> But in my home system, I also mostly just mirror (or even triple
>> mirror!) my important data for both speed and ease of recovery.
>> 
Wol> The trouble with a plain mirror is if your data is corrupted ...

Wol> And if you've got raid you need to keep an eye on it. I'm planning to 
Wol> get an 8TB drive for backups ... probably configure it to snapshot once 
Wol> a week (I'm running lvm over raid over dm-integrity) and then use rsync 
Wol> to back up the volume over the network. Mind you I'm going to have to 
Wol> learn a lot of rsync - configure client/server and then get it to back 
Wol> up just the one file ...

I've got an external USB-3 4tb drive I'm running rclone to, and
planning on taking it to the bank one of these days when I remember to
do so.