Is shrinking raid5 possible?

Is shrinking raid5 possible?

[Paul Davidson]

Hi,

I'd like to shrink the size of a RAID5 array - is this
possible? My first attempt shrinking 1.4Tb to 600Gb,

mdadm --grow /dev/md5 --size=629145600

gives

mdadm: Cannot set device size/shape for /dev/md5: No space left on device

which is true but not particularly relevant :). If mdadm
doesn't support this for online arrays, can I do it offline
somehow?

I'd like to retain the ext3 filesystem on this device,
which I have already shrunk to 400Gb with resize2fs.

Thanks for any help,
Paul

Re: Is shrinking raid5 possible?

[Neil Brown]

On Monday June 19, Paul.Davidson@anu.edu.au wrote:
> Hi,
> 
> I'd like to shrink the size of a RAID5 array - is this
> possible? My first attempt shrinking 1.4Tb to 600Gb,
> 
> mdadm --grow /dev/md5 --size=629145600
> 
> gives
> 
> mdadm: Cannot set device size/shape for /dev/md5: No space left on device

Yep.
The '--size' option refers to:
              Amount  (in  Kibibytes)  of  space  to  use  from  each drive in
              RAID1/4/5/6.  This must be a multiple of  the  chunk  size,  and
              must  leave about 128Kb of space at the end of the drive for the
              RAID superblock.  
(from the man page).

So you were telling md to use the first 600GB of each device in the
array, and it told you there wasn't that much room.
If your array has N drives, you need to divide the target array size
by N-1 to find the target device size.
So if you have a 5 drive array, then you want
  --size=157286400

NeilBrown

Re: Is shrinking raid5 possible?

[Paul Davidson]

Neil Brown wrote:
> Yep.
> The '--size' option refers to:
>               Amount  (in  Kibibytes)  of  space  to  use  from  each drive in
>               RAID1/4/5/6.  This must be a multiple of  the  chunk  size,  and
>               must  leave about 128Kb of space at the end of the drive for the
>               RAID superblock.  
> (from the man page).
> 
> So you were telling md to use the first 600GB of each device in the
> array, and it told you there wasn't that much room.
> If your array has N drives, you need to divide the target array size
> by N-1 to find the target device size.
> So if you have a 5 drive array, then you want
>   --size=157286400
> 
> NeilBrown

Thanks, and sorry for not being able to read properly -- I read this at
least three times and didn't notice it was the drive size and not the
array size.

Cheers, Paul.

Re: Is shrinking raid5 possible?

[Bill Davidsen]

Neil Brown wrote:

>On Monday June 19, Paul.Davidson@anu.edu.au wrote:
>  
>
>>Hi,
>>
>>I'd like to shrink the size of a RAID5 array - is this
>>possible? My first attempt shrinking 1.4Tb to 600Gb,
>>
>>mdadm --grow /dev/md5 --size=629145600
>>
>>gives
>>
>>mdadm: Cannot set device size/shape for /dev/md5: No space left on device
>>    
>>
>
>Yep.
>The '--size' option refers to:
>              Amount  (in  Kibibytes)  of  space  to  use  from  each drive in
>              RAID1/4/5/6.  This must be a multiple of  the  chunk  size,  and
>              must  leave about 128Kb of space at the end of the drive for the
>              RAID superblock.  
>(from the man page).
>
>So you were telling md to use the first 600GB of each device in the
>array, and it told you there wasn't that much room.
>If your array has N drives, you need to divide the target array size
>by N-1 to find the target device size.
>So if you have a 5 drive array, then you want
>  --size=157286400
>

May I say in all honesty that making people do that math instead of the 
computer is a really bad user interface? Good, consider it said. An 
means to just set the target size of the resulting raid device would be 
a LOT less likely to cause bad user input, and while I'm complaining it 
should inderstand suffices 'k', 'm', and 'g'.

Far easier to use for the case where you need, for instance, 10G of 
storage for a database, tell mdadm what devices to use and what you need 
(and the level of course) and let the computer figure out the details, 
rounding up, leaving 128k, and phase of the moon if you decide to use it.

Sorry, I think the current approach is baaad human interface.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Is shrinking raid5 possible?

[Neil Brown]

On Thursday June 22, davidsen@tmr.com wrote:
> Neil Brown wrote:
> 
> >On Monday June 19, Paul.Davidson@anu.edu.au wrote:
> >  
> >
> >>Hi,
> >>
> >>I'd like to shrink the size of a RAID5 array - is this
> >>possible? My first attempt shrinking 1.4Tb to 600Gb,
> >>
> >>mdadm --grow /dev/md5 --size=629145600
> >>
> >>gives
> >>
> >>mdadm: Cannot set device size/shape for /dev/md5: No space left on device
> >>    
> >>
> >
> >Yep.
> >The '--size' option refers to:
> >              Amount  (in  Kibibytes)  of  space  to  use  from  each drive in
> >              RAID1/4/5/6.  This must be a multiple of  the  chunk  size,  and
> >              must  leave about 128Kb of space at the end of the drive for the
> >              RAID superblock.  
> >(from the man page).
> >
> >So you were telling md to use the first 600GB of each device in the
> >array, and it told you there wasn't that much room.
> >If your array has N drives, you need to divide the target array size
> >by N-1 to find the target device size.
> >So if you have a 5 drive array, then you want
> >  --size=157286400
> >
> 
> May I say in all honesty that making people do that math instead of the 
> computer is a really bad user interface? Good, consider it said. An 
> means to just set the target size of the resulting raid device would be 
> a LOT less likely to cause bad user input, and while I'm complaining it 
> should inderstand suffices 'k', 'm', and 'g'.

Let me put another perspective on this.

Why would you ever want to reduce the size of a raid5 in this way?
The only reason that I can think of is that you want to repartition
each device to use a smaller partition for the raid5, and free up some
space for something else.  If that is what you are doing, you will
have already done the math and you will know what size you want your
final partitions to be, so setting the device size is just as easy as
setting the array size.

If you really are interested in array size and have no interest in
recouping the wasted space on the drives, then there would be no point
in shrinking the array (that I can think of).
Just 'mkfs' a filesystem to the desired size and ignore the rest of
the array.


In short, reducing a raid5 to a particular size isn't something that
really makes sense to me.  Reducing the amount of each device that is
used does - though I would much more expect people to want to increase
that size.

If Paul really has a reason to reduce the array to a particular size
then fine.  I'm mildly curious, but it's his business and I'm happy
for mdadm to support it, though indirectly.  But I strongly suspect
that most people who want to resize their array will be thinking in
terms of the amount of each device that is used, so that is how mdadm
works.

> 
> Far easier to use for the case where you need, for instance, 10G of 
> storage for a database, tell mdadm what devices to use and what you need 
> (and the level of course) and let the computer figure out the details, 
> rounding up, leaving 128k, and phase of the moon if you decide to use it.
> 

mdadm is not intended to be a tool that manages your storage for you.  If
you want that, then I suspect EVMS is what you want (though I am only
guessing - I've never used it).  mdadm it a tool that enables YOU to
manage your storage.

NeilBrown



> Sorry, I think the current approach is baaad human interface.
> 
> -- 
> bill davidsen <davidsen@tmr.com>
>   CTO TMR Associates, Inc
>   Doing interesting things with small computers since 1979

Re: Is shrinking raid5 possible?

[Paul Davidson]

Neil Brown wrote:
> In short, reducing a raid5 to a particular size isn't something that
> really makes sense to me.  Reducing the amount of each device that is
> used does - though I would much more expect people to want to increase
> that size.
> 
> If Paul really has a reason to reduce the array to a particular size
> then fine.  I'm mildly curious, but it's his business and I'm happy
> for mdadm to support it, though indirectly.  But I strongly suspect
> that most people who want to resize their array will be thinking in
> terms of the amount of each device that is used, so that is how mdadm
> works.

By way of explanation, I was doing exactly what you thought - I had a
single ext3 filesystem on a raid5 device, and wanted to split it into
two filesystems. I'm not using LVM since it appears to affect read
performance  quite severely. I guess there may be other ways of doing
this but this seemed the most straightforward.

Incidentally, it wasn't clear to me what to do after shrinking the
raid5 device. My initial try at taking it offline and repartitioning
all the disks at once didn't work - I think because the superblocks
became 'lost'. I eventually realized I should go through a
fail-remove-repartition-add-recover cycle for each disk in turn with
the array online. It took a long time but worked in the end.

Would repartitioning them all at once have worked if I had chosen to
have the superblocks at the beginning of the partitions (v1.1 or 1.2
superblocks)?

As for Bill's comment about the mdadm interface, what probably would
have helped me the most is if the man page had had "from each drive"
in bold, flashing and preferably two-foot tall letters :-).

A more practical suggestion is if the error message had not been
    "No space left on device"
but something like
    "Maximum space available on each device is xxxxxxxx"
then I would have quickly realized my mistake.  As Neil points out,
you have to 'do the math' anyway when partitioning.

Cheers,
Paul

Re: Is shrinking raid5 possible?

[Henrik Holst]

Neil Brown wrote:
> In short, reducing a raid5 to a particular size isn't something that
> really makes sense to me.  Reducing the amount of each device that is
> used does - though I would much more expect people to want to increase
> that size.

Think about the poor people! :-) Those who can't afford to buy a new
disk after a failure but can give up some free space. I actually don't
think that that scenario is /highly unlikely/ to occur?

And also for the sake of symmetry: If growing is allowed- why should not
shrinking be just as valid?

Neil Brown wrote:
> If Paul really has a reason to reduce the array to a particular size
> then fine.  I'm mildly curious, but it's his business and I'm happy
> for mdadm to support it, though indirectly.  But I strongly suspect
> that most people who want to resize their array will be thinking in
> terms of the amount of each device that is used, so that is how mdadm
> works.

I agree with you here- keep the parameters "low level". In that way the
administrator (users use a GUI) have more control over the operation at
hand. (kmdadm anyone? :-))

--
Henrik Holst

Re: Is shrinking raid5 possible?

[Christian Pernegger]

> Why would you ever want to reduce the size of a raid5 in this way?

A feature that would have been useful to me a few times is the ability
to shrink an array by whole disks.

Example:

8x 300 GB disks -> 2100 GB raw capacity

shrink file system, remove 2 disks =

6x 300 GB disks --> 1500 GB raw capacity

Why?

If you're not backed up by a company budget, moving data to an new
array (extra / larger disks) is extremely difficult. A lot of cases
will hold 8 disks but not 16, never mind the extra RAID controller.
Building another temporary server and moving the data via Gigabit is
slow and expensive as well.

Shrinking the array step-by-step and unloading data onto a regular
filesystem on the freed disks would be a cheap (if time consuming) way
to migrate, because the data could be copied back to the new array a
disk at a time.

Thanks,

C.

Re: Is shrinking raid5 possible?

[Neil Brown]

On Friday June 23, pernegger@gmail.com wrote:
> > Why would you ever want to reduce the size of a raid5 in this way?
> 
> A feature that would have been useful to me a few times is the ability
> to shrink an array by whole disks.
> 
> Example:
> 
> 8x 300 GB disks -> 2100 GB raw capacity
> 
> shrink file system, remove 2 disks =
> 
> 6x 300 GB disks --> 1500 GB raw capacity
> 

This is shrinking an array by removing drives.  We were talking about
shrinking an array by reducing the size of drives - a very different
think.

Yes, it might be sometimes useful to reduce the number of drives in a
raid5.  This would be similar to adding a drive to a raid5 (now
possible), but the data copy would have to go in a different
direction, so there would need to be substantial changes to the code.
I'm not sure it is really worth the effort I'm afraid, but it might
get done, one day, especially if someone volunteers some code ... ;-)

NeilBrown


> Why?
> 
> If you're not backed up by a company budget, moving data to an new
> array (extra / larger disks) is extremely difficult. A lot of cases
> will hold 8 disks but not 16, never mind the extra RAID controller.
> Building another temporary server and moving the data via Gigabit is
> slow and expensive as well.
> 
> Shrinking the array step-by-step and unloading data onto a regular
> filesystem on the freed disks would be a cheap (if time consuming) way
> to migrate, because the data could be copied back to the new array a
> disk at a time.
> 
> Thanks,
> 
> C.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is shrinking raid5 possible?

[Christian Pernegger]

> This is shrinking an array by removing drives.  We were talking about
> shrinking an array by reducing the size of drives - a very different
> think.

Yes I know - I just wanted to get this in as an alternative shrinking semantic.

As for reducing the RAID (partition) size on the individual drives I
can only see two reasons:

1) One wants to replace / add a disk but the new disk is slightly
smaller than the existing ones. Actual capacity varies a lot for the
same nominal capacity, especially across brands.

2) One wants to do something else with some of the space, although for
a RAID5 I don't quite see the point - you'd end up with n small
partitions. Shrinking a 2-way mirror or stripe this way sounds far
more useful.

Having RAID tightly integrated with volume management and partitions
would be nice, but that's a pipe dream. (EVMS just integrates the UI
somewhat, you still have to mess about with layers.)

> I'm not sure it is really worth the effort I'm afraid.

Neither am I ... it just would have come in handy a few times, that's all.

Thanks,

C.

Re: Is shrinking raid5 possible?

[Henrik Holst]

Christian Pernegger wrote:
>> This is shrinking an array by removing drives.  We were talking about
>> shrinking an array by reducing the size of drives - a very different
>> think.
> 
> Yes I know - I just wanted to get this in as an alternative shrinking
> semantic.
> 
> As for reducing the RAID (partition) size on the individual drives I
> can only see two reasons:
> 
> 1) One wants to replace / add a disk but the new disk is slightly
> smaller than the existing ones. Actual capacity varies a lot for the
> same nominal capacity, especially across brands.
> 
> 2) One wants to do something else with some of the space, although for
> a RAID5 I don't quite see the point - you'd end up with n small
> partitions. Shrinking a 2-way mirror or stripe this way sounds far
> more useful.
> 
> Having RAID tightly integrated with volume management and partitions
> would be nice, but that's a pipe dream. (EVMS just integrates the UI
> somewhat, you still have to mess about with layers.)
> 
>> I'm not sure it is really worth the effort I'm afraid.
> 
> Neither am I ... it just would have come in handy a few times, that's all.
> 
> Thanks,
> 
> C.

I have an 8x120 GB RAID-5 volume. Or rather. But since of yesterday I
only have 7 120 GB harddrives. Since 120 GB disks are out of the
"priceworthy segment" for storage I want to reduce this to a
non-degenerate 7x120 GB RAID and not buy a relative expensive 120 GB
disk. And recreating the RAID as a 7x120 volume is out of the question
since I don't have the extra storage to move ~700 GB of data.

I think there IS an application for shrinking RAID:s, as well as
growing- and that is not only one of "symmetric beauty".

Is such a feature within the scope of software raid development? (I
guess I'm asking Neil B. or Steinar H. G.)

/Henrik Holst

Re: Is shrinking raid5 possible?

[Thomas Arthur Oehser]

> > 2) One wants to do something else with some of the space, although for
> > a RAID5 I don't quite see the point - you'd end up with n small
> > partitions. Shrinking a 2-way mirror or stripe this way sounds far
> > more useful.

Oooh, what would I want to do with "n small partitions"?  I know!  Raid!

Scenario:

existing:
sda1 sdb1 sdc1 254M + 254M + 254M = 254M raid-1 /boot
sda2 sdb2 sbc2 2G + 2G + 2G = 6G raid-0 swap
sda3 sdb3 sdc3 1.5T + 1.5T + 15.T = 3T raid-5 /

desired:
sda1 sdb1 sdc1 254M + 254M + 254M = 254M raid-1 /dev/md1 /boot
sda2 sdb2 sbc2 2G + 2G + 2G = 6G raid-0 swap /dev/md2
sda3 sdb3 sdc3 1.4T + 1.4T + 1.4T = 2.8T raid-5 /dev/md3 /
sda4 sdb4 sdc4 100G + 100G + 100G = 200G raid-5 /dev/md4 / <= alt for Koala

Since /dev/md3 is still only 70% full, I thought I could do something like:

 - resize2fs /dev/md3 <smaller>
 - mdadm -G -z <smaller> /dev/md3
 - fdisk /dev/sd[a-c]3 <smaller>

It _almost_ works fine... but... the mdadm -G keeps the (0.90) superblock
at the _end_ of the _device_ ... which hasn't been resized yet ...

How do I put the superblock back after fdisk, or make -G move or recreate?

-Tom

Re: Is shrinking raid5 possible?

[Asdo]

Thomas Arthur Oehser wrote:
> Oooh, what would I want to do with "n small partitions"?  I know!  Raid!
>
> Scenario:
>
> existing:
> sda1 sdb1 sdc1 254M + 254M + 254M = 254M raid-1 /boot
> sda2 sdb2 sbc2 2G + 2G + 2G = 6G raid-0 swap
> ...
>   
Putting swap in a raid-0 will hang your computer as soon as sda, sdb or 
sdc fails.
Are you OK with that?

Re: Is shrinking raid5 possible?

[Thomas Arthur Oehser]

> Putting swap in a raid-0 will hang your computer as soon as sda, sdb or  
> sdc fails.
> Are you OK with that?

I'll move the swap to the partitions then, though it is a moot point as
the machine has 4G physical RAM and runs with a load such that usually
about 2G is cache... I assume that if it never swaps, it never matters.

But what I care about is the raid-5 shrinking issue... does anyone know
if there is a reasonably safe way to shrink a raid-5 device (size, not
number of drives) that is superblock format 0.90?  I assume the --grow
just forces the superblock based on the _device_, but if I shrank the
_device_ before I shrank the raid, it wouldn't find the superblock in
the first place...

-Tom

Re: Is shrinking raid5 possible?

[Goswin von Brederlow]

Thomas Arthur Oehser <tom@toms.net> writes:

>> Putting swap in a raid-0 will hang your computer as soon as sda, sdb or  
>> sdc fails.
>> Are you OK with that?
>
> I'll move the swap to the partitions then, though it is a moot point as
> the machine has 4G physical RAM and runs with a load such that usually
> about 2G is cache... I assume that if it never swaps, it never matters.

Make it 2GB raid1 for swap then.

> But what I care about is the raid-5 shrinking issue... does anyone know
> if there is a reasonably safe way to shrink a raid-5 device (size, not
> number of drives) that is superblock format 0.90?  I assume the --grow
> just forces the superblock based on the _device_, but if I shrank the
> _device_ before I shrank the raid, it wouldn't find the superblock in
> the first place...
>
> -Tom

- shrink the raid
- fail+remove sda3
- make sda3 smaller
- add sda3 and resync
- repeat for sdb and sdc

You should have used LVM on your raid5. Makes suck things so much simpler.

MfG
        Goswin

Re: Is shrinking raid5 possible?

[John Robinson]

On 06/11/2009 13:17, Thomas Arthur Oehser wrote:
[...]
>  - resize2fs /dev/md3 <smaller>
>  - mdadm -G -z <smaller> /dev/md3
>  - fdisk /dev/sd[a-c]3 <smaller>
> 
> It _almost_ works fine... but... the mdadm -G keeps the (0.90) superblock
> at the _end_ of the _device_ ... which hasn't been resized yet ...
> 
> How do I put the superblock back after fdisk, or make -G move or recreate?

Are you sure it hasn't already put a copy of the superblock in the right 
place, within the <smaller> size, as well as leaving a copy at the end 
of the original larger partition? That's what I would do if I was the 
author, so the partition can still be recognised as a md one before the 
partition is resized; you wouldn't want the parition to become 
unrecognisable or unusable.

Cheers,

John.

Re: Is shrinking raid5 possible?

[Thomas Arthur Oehser]

On Fri, Nov 06, 2009 at 06:32:15PM +0000, John Robinson wrote:
> On 06/11/2009 13:17, Thomas Arthur Oehser wrote:
> [...]
>>  - resize2fs /dev/md3 <smaller>
>>  - mdadm -G -z <smaller> /dev/md3
>>  - fdisk /dev/sd[a-c]3 <smaller>
>>
>> It _almost_ works fine... but... the mdadm -G keeps the (0.90) superblock
>> at the _end_ of the _device_ ... which hasn't been resized yet ...
>>
>> How do I put the superblock back after fdisk, or make -G move or recreate?
>
> Are you sure it hasn't already put a copy of the superblock in the right  
> place, within the <smaller> size, as well as leaving a copy at the end  
> of the original larger partition? That's what I would do if I was the  
> author, so the partition can still be recognised as a md one before the  
> partition is resized; you wouldn't want the parition to become  
> unrecognisable or unusable.

Far as I can tell, no, because when I then shrink the partition, it can't
find the superblock.

Re: Is shrinking raid5 possible?

[Goswin von Brederlow]

Thomas Arthur Oehser <tom@toms.net> writes:

> On Fri, Nov 06, 2009 at 06:32:15PM +0000, John Robinson wrote:
>> On 06/11/2009 13:17, Thomas Arthur Oehser wrote:
>> [...]
>>>  - resize2fs /dev/md3 <smaller>
>>>  - mdadm -G -z <smaller> /dev/md3
>>>  - fdisk /dev/sd[a-c]3 <smaller>
>>>
>>> It _almost_ works fine... but... the mdadm -G keeps the (0.90) superblock
>>> at the _end_ of the _device_ ... which hasn't been resized yet ...
>>>
>>> How do I put the superblock back after fdisk, or make -G move or recreate?
>>
>> Are you sure it hasn't already put a copy of the superblock in the right  
>> place, within the <smaller> size, as well as leaving a copy at the end  
>> of the original larger partition? That's what I would do if I was the  
>> author, so the partition can still be recognised as a md one before the  
>> partition is resized; you wouldn't want the parition to become  
>> unrecognisable or unusable.
>
> Far as I can tell, no, because when I then shrink the partition, it can't
> find the superblock.

And you would have to shrink it to exactly the right size to the block
so that the shrunk superblock ends up at the end of the resized
device. Something most people won't be able to do right.

As alternative for fail/resync I mentioned in my last mail you could
also repartition and create a new raid with --assume-clean if you like
experimenting. Either way is bad.

MfG
        Goswin

Re: Is shrinking raid5 possible?

[John Robinson]

On 06/11/2009 19:30, Goswin von Brederlow wrote:
> Thomas Arthur Oehser <tom@toms.net> writes:
>> On Fri, Nov 06, 2009 at 06:32:15PM +0000, John Robinson wrote:
[...]
>>> Are you sure it hasn't already put a copy of the superblock in the right  
>>> place, within the <smaller> size, as well as leaving a copy at the end  
>>> of the original larger partition?
[...]
>> Far as I can tell, no, because when I then shrink the partition, it can't
>> find the superblock.
> 
> And you would have to shrink it to exactly the right size to the block
> so that the shrunk superblock ends up at the end of the resized
> device. Something most people won't be able to do right.

The thing is, I'm sure I've done it, and while I don't remember the 
details, I don't think I had to either recreate the array nor drop and 
re-add members. You're right though, getting the numbers right is the 
tricky bit, because the filesystem's smaller than the md, which in turn 
may be smaller than the partition, even when everything's "tight".

Cheers,

John.