very degraded RAID5, or increasing capacity by adding discs

very degraded RAID5, or increasing capacity by adding discs

[Janek Kozicki]

Hello,

Recently I started to use mdadm and I'm very impressed by its
capabilities. 

I have raid0 (250+250 GB) on my workstation. And I want to have
raid5 (4*500 = 1500 GB) on my backup machine.

The backup machine currently doesn't have raid, just a single 500 GB
drive. I plan to buy more HDDs to have a bigger space for my
backups but since I cannot afford all HDDs at once I face a problem
of "expanding" an array. I'm able to add one 500 GB drive every few
months until I have all 4 drives.

But I cannot make a backup of a backup... so reformatting/copying all
data each time when I add new disc to the array is not possible for me.

Is it possible anyhow to create a "very degraded" raid array - a one
that consists of 4 drives, but has only TWO ?

This would involve some very tricky *hole* management on the block
device... A one that places holes in stripes on the block device,
until more discs are added to fill the holes. When the holes are
filled, the block device grows bigger, and with lvm I just increase
the filesystem size. This is perhaps coupled with some "unstripping"
that moves/reorganizes blocks around to fill/defragment the holes.

is it just a pipe dream?

best regards


PS: yes it's simple to make a degraded array of 3 drives, but I
cannot afford two discs at once...

-- 
Janek Kozicki                                                         |

Re: very degraded RAID5, or increasing capacity by adding discs

[Justin Piszcz]

On Mon, 8 Oct 2007, Janek Kozicki wrote:

> Hello,
>
> Recently I started to use mdadm and I'm very impressed by its
> capabilities.
>
> I have raid0 (250+250 GB) on my workstation. And I want to have
> raid5 (4*500 = 1500 GB) on my backup machine.
>
> The backup machine currently doesn't have raid, just a single 500 GB
> drive. I plan to buy more HDDs to have a bigger space for my
> backups but since I cannot afford all HDDs at once I face a problem
> of "expanding" an array. I'm able to add one 500 GB drive every few
> months until I have all 4 drives.
>
> But I cannot make a backup of a backup... so reformatting/copying all
> data each time when I add new disc to the array is not possible for me.
>
> Is it possible anyhow to create a "very degraded" raid array - a one
> that consists of 4 drives, but has only TWO ?
>
> This would involve some very tricky *hole* management on the block
> device... A one that places holes in stripes on the block device,
> until more discs are added to fill the holes. When the holes are
> filled, the block device grows bigger, and with lvm I just increase
> the filesystem size. This is perhaps coupled with some "unstripping"
> that moves/reorganizes blocks around to fill/defragment the holes.
>
> is it just a pipe dream?
>
> best regards
>
>
> PS: yes it's simple to make a degraded array of 3 drives, but I
> cannot afford two discs at once...
>
> -- 
> Janek Kozicki                                                         |
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

With raid1 you can create a degraded array with 1 disk- I have done this, 
I have always wondered if mdadm will let you make a degraded raid 5 array 
with 2 disks (you'd specify 3 and only give 2) - you can always expand 
later.

Justin.

Re: very degraded RAID5, or increasing capacity by adding discs

[Richard Scobie]

Janek Kozicki wrote:

> Is it possible anyhow to create a "very degraded" raid array - a one
> that consists of 4 drives, but has only TWO ?

No, but you can make a degraded 3 drive array, containing 2 drives and 
then add the next drive to complete it.

The array can then be grown (man mdadm, GROW section), to add the fourth.

Regards,

Richard

RE: very degraded RAID5, or increasing capacity by adding discs

[Guy Watkins]

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Richard Scobie
} Sent: Monday, October 08, 2007 3:27 PM
} To: linux-raid@vger.kernel.org
} Subject: Re: very degraded RAID5, or increasing capacity by adding discs
} 
} Janek Kozicki wrote:
} 
} > Is it possible anyhow to create a "very degraded" raid array - a one
} > that consists of 4 drives, but has only TWO ?
} 
} No, but you can make a degraded 3 drive array, containing 2 drives and
} then add the next drive to complete it.
} 
} The array can then be grown (man mdadm, GROW section), to add the fourth.
} 
} Regards,
} 
} Richard

I think someone once said you could create a 2 disk degraded RAID5 array
with just 1 disk.  Then add one later.  Then expand as needed.  Someone
should test this.

Re: very degraded RAID5, or increasing capacity by adding discs

[Janek Kozicki]

Richard Scobie said:     (by the date of Tue, 09 Oct 2007 08:26:35 +1300)

> No, but you can make a degraded 3 drive array, containing 2 drives and 
> then add the next drive to complete it.
> 
> The array can then be grown (man mdadm, GROW section), to add the fourth.

Oh, good. Thanks, I must've been blind that I missed this.
This completely solves my problem.

-- 
Janek Kozicki                                                         |

Re: very degraded RAID5, or increasing capacity by adding discs

[Janek Kozicki]

Janek Kozicki said:     (by the date of Tue, 9 Oct 2007 00:25:50 +0200)

> Richard Scobie said:     (by the date of Tue, 09 Oct 2007 08:26:35 +1300)
> 
> > No, but you can make a degraded 3 drive array, containing 2 drives and 
> > then add the next drive to complete it.
> > 
> > The array can then be grown (man mdadm, GROW section), to add the fourth.
> 
> Oh, good. Thanks, I must've been blind that I missed this.
> This completely solves my problem.

Uh, actually not :)

My 1st 500 GB drive is full now. When I buy a 2nd one I want to
create a 3-disc degraded array using just 2 discs, one of which
contains unbackupable data.

steps:
1. create degraded two-disc RAID5 on 1 new disc
2. copy data from old disc to new one
3. rebuild the array with old and new discs (now I have 500 GB on 2 discs)
4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2 discs)
...
5. when I buy 3rd drive I either grow the array, or just rebuild and
wait with growing until I buy a 4th drive.

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
grow an array to a degraded array (non existant disc). Is it possible?


PS: the fact, that degraded array will be unsafe for the data is an
intented motivating factor for buying next drive ;)

-- 
Janek Kozicki                                                         |

Re: very degraded RAID5, or increasing capacity by adding discs

[Michael Tokarev]

Janek Kozicki wrote:
> Hello,
> 
> Recently I started to use mdadm and I'm very impressed by its
> capabilities. 
> 
> I have raid0 (250+250 GB) on my workstation. And I want to have
> raid5 (4*500 = 1500 GB) on my backup machine.

Hmm.  Are you sure you need that much space on the backup, to
start with?  Maybe better backup strategy will help to avoid
hardware costs?  Such as using rsync for backups as discussed
on this mailinglist about a month back (rsync is able to keep
many ready to use copies of your filesystems but only store
files that actually changed since the last backup, thus
requiring much less space than many full backups).

> The backup machine currently doesn't have raid, just a single 500 GB
> drive. I plan to buy more HDDs to have a bigger space for my
> backups but since I cannot afford all HDDs at once I face a problem
> of "expanding" an array. I'm able to add one 500 GB drive every few
> months until I have all 4 drives.
> 
> But I cannot make a backup of a backup... so reformatting/copying all
> data each time when I add new disc to the array is not possible for me.
> 
> Is it possible anyhow to create a "very degraded" raid array - a one
> that consists of 4 drives, but has only TWO ?
> 
> This would involve some very tricky *hole* management on the block
> device... A one that places holes in stripes on the block device,
> until more discs are added to fill the holes. When the holes are
> filled, the block device grows bigger, and with lvm I just increase
> the filesystem size. This is perhaps coupled with some "unstripping"
> that moves/reorganizes blocks around to fill/defragment the holes.

It's definitely not possible with raid5.  Only option is to create a
raid5 array consisting of less drives than it should contain at the
end, and reshape it when you get more drives, as others noted in this
thread.  But do note the following points:

o degraded raid5 isn't really Raid - i.e, it's not any better than
  a raid0 array, that is, any disk fails => the whole array fails.
  So instead of creating a degraded raid5 array initially, create
  smaller one instead, but not degraded, and reshape it when
  necessary.

o reshaping takes time, and for this volume, reshape will take
  many hours, maybe days, to complete.

o During this reshape time, errors may be fatal to the whole array -
  while mdadm do have a sense of "critical section", but the
  whole procedure isn't as much tested as the rest of raid code,
  I for one will not rely on it, at least for now.  For example,
  a power failure at an "unexpected" moment, or some plain-stupid
  error in reshape code so that the whole array goes "boom" etc...

o A filesystem on the array has to be resized separately after
  re{siz,shap}ing the array.  And filesystems are different at
  this point, too - there are various limitations.  For example,
  it's problematic to grow ext[23]fs by large amounts, because
  when it gets initially created, mke2fs calculates sizes of
  certain internal data structures based on the device size,
  and those structures can't be grown significantly, only
  recreating the filesystem will do the trick.

> is it just a pipe dream?

I'd say it is... ;)

/mjt

RE: very degraded RAID5, or increasing capacity by adding discs

[Guy Watkins]

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Janek Kozicki
} Sent: Monday, October 08, 2007 6:47 PM
} To: linux-raid@vger.kernel.org
} Subject: Re: very degraded RAID5, or increasing capacity by adding discs
} 
} Janek Kozicki said:     (by the date of Tue, 9 Oct 2007 00:25:50 +0200)
} 
} > Richard Scobie said:     (by the date of Tue, 09 Oct 2007 08:26:35
} +1300)
} >
} > > No, but you can make a degraded 3 drive array, containing 2 drives and
} > > then add the next drive to complete it.
} > >
} > > The array can then be grown (man mdadm, GROW section), to add the
} fourth.
} >
} > Oh, good. Thanks, I must've been blind that I missed this.
} > This completely solves my problem.
} 
} Uh, actually not :)
} 
} My 1st 500 GB drive is full now. When I buy a 2nd one I want to
} create a 3-disc degraded array using just 2 discs, one of which
} contains unbackupable data.
} 
} steps:
} 1. create degraded two-disc RAID5 on 1 new disc
} 2. copy data from old disc to new one
} 3. rebuild the array with old and new discs (now I have 500 GB on 2 discs)
3. Add old disk to new array.  Once done RAID5 is redundant.

} 4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2
} discs)
4. Buy 3rd disk.
5. Add new 3rd disk to array and grow to 3 disk RAID5 array.  Once done,
array is redundant.

Repeat 4 and 5 each time you buy a new disk.

I don't think you can grow to a degraded array.  I think you must add a new
disk first.  But I am not sure.

} ...
} 5. when I buy 3rd drive I either grow the array, or just rebuild and
} wait with growing until I buy a 4th drive.
} 
} Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
} grow an array to a degraded array (non existant disc). Is it possible?
} 
} 
} PS: the fact, that degraded array will be unsafe for the data is an
} intented motivating factor for buying next drive ;)
} 
} --
} Janek Kozicki                                                         |
} -
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in
} the body of a message to majordomo@vger.kernel.org
} More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: very degraded RAID5, or increasing capacity by adding discs

[Neil Brown]

On Tuesday October 9, mjt@tls.msk.ru wrote:
> 
> o degraded raid5 isn't really Raid - i.e, it's not any better than
>   a raid0 array, that is, any disk fails => the whole array fails.
>   So instead of creating a degraded raid5 array initially, create
>   smaller one instead, but not degraded, and reshape it when
>   necessary.

Fully agree.

> 
> o reshaping takes time, and for this volume, reshape will take
>   many hours, maybe days, to complete.
> 
> o During this reshape time, errors may be fatal to the whole array -
>   while mdadm do have a sense of "critical section", but the
>   whole procedure isn't as much tested as the rest of raid code,
>   I for one will not rely on it, at least for now.  For example,
>   a power failure at an "unexpected" moment, or some plain-stupid
>   error in reshape code so that the whole array goes "boom" etc...

While it is true that the resize code is less tested than other code,
it is designed to handle a single failure at any time (so a power
failure is OK as long as the array is not running degraded), and I
have said that if anyone does suffer problems while performing a
reshape, I will do my absolute best to get the array functioning and
the data safe again.

> 
> o A filesystem on the array has to be resized separately after
>   re{siz,shap}ing the array.  And filesystems are different at
>   this point, too - there are various limitations.  For example,
>   it's problematic to grow ext[23]fs by large amounts, because
>   when it gets initially created, mke2fs calculates sizes of
>   certain internal data structures based on the device size,
>   and those structures can't be grown significantly, only
>   recreating the filesystem will do the trick.

This isn't entirely true.
For online resizing (while the filesystem is mounted) there are some
limitations as you suggest.  For offline resizing (while filesystem is
not mounted) there are no such limitations.


NeilBrown

Re: very degraded RAID5, or increasing capacity by adding discs

[Neil Brown]

On Tuesday October 9, janek_listy@wp.pl wrote:
> 
> Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
> grow an array to a degraded array (non existant disc). Is it possible?

Why not experiment with loop devices on files and find out?

But yes:  you can grow to a degraded array providing you specify a
--backup-file.

However I don't recommend it.  I would never recommend having a
degraded array by design.  It should only ever happen due to a
failure, and should last only until you can get a replacement
rebuilt. 

Remember that a degraded raid5 has a greater risk of data loss than a
single drive.

> 
> 
> PS: the fact, that degraded array will be unsafe for the data is an
> intented motivating factor for buying next drive ;)

:-)

NeilBrown

Re: very degraded RAID5, or increasing capacity by adding discs

[Michael Tokarev]

Neil Brown wrote:
> On Tuesday October 9, mjt@tls.msk.ru wrote:
[]
>> o During this reshape time, errors may be fatal to the whole array -
>>   while mdadm do have a sense of "critical section", but the
>>   whole procedure isn't as much tested as the rest of raid code,
>>   I for one will not rely on it, at least for now.  For example,
>>   a power failure at an "unexpected" moment, or some plain-stupid
>>   error in reshape code so that the whole array goes "boom" etc...
> 
> While it is true that the resize code is less tested than other code,
> it is designed to handle a single failure at any time (so a power
> failure is OK as long as the array is not running degraded), and I
> have said that if anyone does suffer problems while performing a
> reshape, I will do my absolute best to get the array functioning and
> the data safe again.

Well... Neil, it's your code, so you trust it - that's ok, I also
(tries to) trust my code until someone finds a bug in it.. ;)
And I'm a sysadmin (among other things), who's professional
property must be a bit of paranoia..  You got the idea ;)

>> o A filesystem on the array has to be resized separately after
>>   re{siz,shap}ing the array.  And filesystems are different at
>>   this point, too - there are various limitations.  For example,
>>   it's problematic to grow ext[23]fs by large amounts, because
>>   when it gets initially created, mke2fs calculates sizes of
>>   certain internal data structures based on the device size,
>>   and those structures can't be grown significantly, only
>>   recreating the filesystem will do the trick.
> 
> This isn't entirely true.
> For online resizing (while the filesystem is mounted) there are some
> limitations as you suggest.  For offline resizing (while filesystem is
> not mounted) there are no such limitations.

There still is - at least for ext[23].  Even offline resizers
can't do resizes from any to any size, extfs developers recommend
to recreate filesystem anyway if size changes significantly.
I'm too lazy to find a reference now, it has been mentioned here
on linux-raid at least this year.  It's sorta like fat (yea, that
ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb,
everything is ok, but if you want to go from 501Mb to 1Gb+1, you
have to recreate almost all data structures because sizes of
all internal fields changes - and here it's much safer to just
re-create it from scratch than trying to modify it in place.
Sure it's much better for extfs, but the point is still the same.

/mjt

Re: very degraded RAID5, or increasing capacity by adding discs

[Janek Kozicki]

Michael Tokarev said:     (by the date of Tue, 09 Oct 2007 02:52:06 +0400)

> Janek Kozicki wrote:
> > Hello,
> > 
> > Recently I started to use mdadm and I'm very impressed by its
> > capabilities. 
> > 
> > I have raid0 (250+250 GB) on my workstation. And I want to have
> > raid5 (4*500 = 1500 GB) on my backup machine.
> 
> Hmm.  Are you sure you need that much space on the backup, to
> start with?  Maybe better backup strategy will help to avoid
> hardware costs?  Such as using rsync for backups as discussed
> on this mailinglist about a month back (rsync is able to keep
> many ready to use copies of your filesystems but only store
> files that actually changed since the last backup, thus
> requiring much less space than many full backups).

yes, exactly. I am using rsnapshot, which is based on rsync and
hardlinks. It works exceptionally well - to my knowledge it's the
best backup solution I have ever seen. With plugin scripts I am even
mounting an lvm-snapshot of the drive being backupped.

from command 'rsnapshot du' I can see how many space is used (but
each directory tree is a full backup (made with hardlinks)):

278G    /backup/.sync
454M    /backup/hourly.0/
515M    /backup/hourly.1/
527M    /backup/daily.0/
30G     /backup/daily.1/
21G     /backup/daily.2/
561M    /backup/daily.3/
1.6G    /backup/daily.4/
3.0G    /backup/daily.5/
594M    /backup/daily.6/
1.4G    /backup/weekly.0/
11G     /backup/weekly.1/
9.3G    /backup/weekly.2/
23G     /backup/weekly.3/
33G     /backup/monthly.0/
3.7G    /backup/monthly.1/
415G    total


> It's definitely not possible with raid5.  Only option is to create a
> raid5 array consisting of less drives than it should contain at the
> end, and reshape it when you get more drives, as others noted in this
> thread.  But do note the following points:

<..snip..>

yes, I am aware of all those problems you listed. The data I'm
talking about is already a backup. While the real data is on my
workstation (a different linux box - albeit only the newest version
of my data). Only losing both of them simultaneoulsy will be
catastrophic for me. 

So I am inclined to do some experiments with the backup drives
configuration, while still doing my best at not losing it. An
exercise, you know :)

> > is it just a pipe dream?
> 
> I'd say it is... ;)

oh well. But I learnt a lot from your answers, thanks a lot!


PS: I'm receiving some mailing list posts twice, anybody knows why?
I'm used to mailman but looks like majordomo is being configured in a
different way - I cannot find a configure page. (I just subscribed).

-- 
Janek Kozicki                                                         |

Re: very degraded RAID5, or increasing capacity by adding discs

[Janek Kozicki]

Neil Brown said:     (by the date of Tue, 9 Oct 2007 13:32:09 +1000)

> On Tuesday October 9, janek_listy@wp.pl wrote:
> > 
> > Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
> > grow an array to a degraded array (non existant disc). Is it possible?
> 
> Why not experiment with loop devices on files and find out?
> 
> But yes:  you can grow to a degraded array providing you specify a
> --backup-file.

Thanks! I'll test this on loopback devices :)


-- 
Janek Kozicki                                                         |

Re: very degraded RAID5, or increasing capacity by adding discs

[Mr. James W. Laferriere]

 	Hello Neil ,

On Tue, 9 Oct 2007, Neil Brown wrote:
> On Tuesday October 9, janek_listy@wp.pl wrote:
>>
>> Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
>> grow an array to a degraded array (non existant disc). Is it possible?
>
> Why not experiment with loop devices on files and find out?
>
> But yes:  you can grow to a degraded array providing you specify a
> --backup-file.
 	Is there an estimate of how large this file can get ?
 	It's probably a calculation based on disk & array parameters .
 	But I was unable to find a reference on it from the manpage .

 		Tia ,  JimL

> However I don't recommend it.  I would never recommend having a
> degraded array by design.  It should only ever happen due to a
> failure, and should last only until you can get a replacement
> rebuilt.
>
> Remember that a degraded raid5 has a greater risk of data loss than a
> single drive.
>
>>
>>
>> PS: the fact, that degraded array will be unsafe for the data is an
>> intented motivating factor for buying next drive ;)
>
> :-)
>
> NeilBrown
-- 
+-----------------------------------------------------------------+
| James   W.   Laferriere | System   Techniques | Give me VMS     |
| Network        Engineer | 663  Beaumont  Blvd |  Give me Linux  |
| babydr@baby-dragons.com | Pacifica, CA. 94044 |   only  on  AXP |
+-----------------------------------------------------------------+

Re: very degraded RAID5, or increasing capacity by adding discs

[Neil Brown]

On Tuesday October 9, babydr@baby-dragons.com wrote:
>  	Hello Neil ,
> 
> On Tue, 9 Oct 2007, Neil Brown wrote:
> > On Tuesday October 9, janek_listy@wp.pl wrote:
> >>
> >> Problems at step 4.: 'man mdadm' doesn't tell if it's possible to
> >> grow an array to a degraded array (non existant disc). Is it possible?
> >
> > Why not experiment with loop devices on files and find out?
> >
> > But yes:  you can grow to a degraded array providing you specify a
> > --backup-file.
>  	Is there an estimate of how large this file can get ?
>  	It's probably a calculation based on disk & array parameters .
>  	But I was unable to find a reference on it from the manpage .
> 
>  		Tia ,  JimL

I think:

   chunk-size * old-ndisks * new-ndisks * 2 + 1K

It might be smaller than that, but it shouldn't be larger.

NeilBrown

Re: very degraded RAID5, or increasing capacity by adding discs

[Louis-David Mitterrand]

On Tue, Oct 09, 2007 at 01:48:50PM +0400, Michael Tokarev wrote:
> 
> There still is - at least for ext[23].  Even offline resizers
> can't do resizes from any to any size, extfs developers recommend
> to recreate filesystem anyway if size changes significantly.
> I'm too lazy to find a reference now, it has been mentioned here
> on linux-raid at least this year.  It's sorta like fat (yea, that
> ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb,
> everything is ok, but if you want to go from 501Mb to 1Gb+1, you
> have to recreate almost all data structures because sizes of
> all internal fields changes - and here it's much safer to just
> re-create it from scratch than trying to modify it in place.
> Sure it's much better for extfs, but the point is still the same.

I'll just mention that I once resized a multi-Tera ext3 filesystem and 
it took 8hours +, a comparable XFS online resize lasted all of 10 
seconds! 

Re: very degraded RAID5, or increasing capacity by adding discs

[Bill Davidsen]

Louis-David Mitterrand wrote:
> On Tue, Oct 09, 2007 at 01:48:50PM +0400, Michael Tokarev wrote:
>   
>> There still is - at least for ext[23].  Even offline resizers
>> can't do resizes from any to any size, extfs developers recommend
>> to recreate filesystem anyway if size changes significantly.
>> I'm too lazy to find a reference now, it has been mentioned here
>> on linux-raid at least this year.  It's sorta like fat (yea, that
>> ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb,
>> everything is ok, but if you want to go from 501Mb to 1Gb+1, you
>> have to recreate almost all data structures because sizes of
>> all internal fields changes - and here it's much safer to just
>> re-create it from scratch than trying to modify it in place.
>> Sure it's much better for extfs, but the point is still the same.
>>     
>
> I'll just mention that I once resized a multi-Tera ext3 filesystem and 
> it took 8hours +, a comparable XFS online resize lasted all of 10 
> seconds! 

Because of the different way these file systems do things, there is no 
comparable resize, at least in terms of work to be done. For many 
systems R/W operations are more common than resize, so the F/S type is 
selected to optimize that. ;-)

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979