Growing after replacing with larger discs

Growing after replacing with larger discs

[John Robinson]

Before I have to go to a customer's premises and try it, I wanted to ask 
if anyone had already done anything like this: I have a CentOS 5 system 
with 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I 
want to grow it by replacing the drives. If I dd the small partitions 
onto larger ones on the new discs, then reboot, will the system 
recognise the RAID-5 - which now has its metadata somewhere in the 
middle of the paritions not at the end - so that I can then --grow?

Cheers,

John.

Re: Growing after replacing with larger discs

[Majed B.]

I'm not sure if it is going to work or not, what you suggested.

But if you can run all disks at the same time, then why not create a
new array and copy the files over?

On Fri, Mar 12, 2010 at 8:39 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> Before I have to go to a customer's premises and try it, I wanted to ask if
> anyone had already done anything like this: I have a CentOS 5 system with
> 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I want to
> grow it by replacing the drives. If I dd the small partitions onto larger
> ones on the new discs, then reboot, will the system recognise the RAID-5 -
> which now has its metadata somewhere in the middle of the paritions not at
> the end - so that I can then --grow?
>
> Cheers,
>
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing after replacing with larger discs

[John Robinson]

On 12/03/2010 18:00, Majed B. wrote:
> I'm not sure if it is going to work or not, what you suggested.
> 
> But if you can run all disks at the same time, then why not create a
> new array and copy the files over?

I can't, I only have the 3 drive caddies, so I was going to shut down, 
start up from a CD, copy each drive individually, then put the 3 larger 
drives in all at once.

Cheers,

John.

Re: Growing after replacing with larger discs

[Luca Berra]

On Fri, Mar 12, 2010 at 05:39:47PM +0000, John Robinson wrote:
> Before I have to go to a customer's premises and try it, I wanted to ask if 
> anyone had already done anything like this: I have a CentOS 5 system with 
> 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I want to 
> grow it by replacing the drives. If I dd the small partitions onto larger 
> ones on the new discs, then reboot, will the system recognise the RAID-5 - 
> which now has its metadata somewhere in the middle of the paritions not at 
> the end - so that I can then --grow?
I believe it won't work
mdadm has no way to know where the metadata is in the above case.

you have to follow the painful procedure
fail one drive (mdadm /dev/mdX --fail /dev/sdY)
remove it (mdadm /dev/mdX --remove /dev/sdY)
replace drive with larger one
create larger partition
add drive again (mdadm /dev/mdX --add /dev/sdY)
wait for it to resync...
...and repeat until all drives have been replaced

L.
-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: Growing after replacing with larger discs

[Majed B.]

If you do this kind of thing a lot, or often enough, then it's better
to get a portable NAS, copy filesystem contents to it, then dump the
stuff back.

You can save yourself time with the new arrays by zeroing all the new
disks, and then when creating the new array, use assume-clean. That'll
prevent the array from resyncing, and you'll get full speed when
copying the filesystem contents back to the new array.

dd allows parallel writing to output devices.

On Fri, Mar 12, 2010 at 9:04 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 12/03/2010 18:00, Majed B. wrote:
>>
>> I'm not sure if it is going to work or not, what you suggested.
>>
>> But if you can run all disks at the same time, then why not create a
>> new array and copy the files over?
>
> I can't, I only have the 3 drive caddies, so I was going to shut down, start
> up from a CD, copy each drive individually, then put the 3 larger drives in
> all at once.
>
> Cheers,
>
> John.
>
>



-- 
       Majed B.

Re: Growing after replacing with larger discs

[John Robinson]

On 12/03/2010 18:09, Luca Berra wrote:
> On Fri, Mar 12, 2010 at 05:39:47PM +0000, John Robinson wrote:
>> Before I have to go to a customer's premises and try it, I wanted to 
>> ask if anyone had already done anything like this: I have a CentOS 5 
>> system with 0.90 metadata and type fd partitions with RAID-5 over 3 
>> discs. I want to grow it by replacing the drives. If I dd the small 
>> partitions onto larger ones on the new discs, then reboot, will the 
>> system recognise the RAID-5 - which now has its metadata somewhere in 
>> the middle of the paritions not at the end - so that I can then --grow?
> I believe it won't work
> mdadm has no way to know where the metadata is in the above case.
> 
> you have to follow the painful procedure
> fail one drive (mdadm /dev/mdX --fail /dev/sdY)
> remove it (mdadm /dev/mdX --remove /dev/sdY)
> replace drive with larger one
> create larger partition
> add drive again (mdadm /dev/mdX --add /dev/sdY)
> wait for it to resync...
> ...and repeat until all drives have been replaced

Yeuch. That'll take forever, because I'd need to rebuild 3 times. I see 
two other options:
(1) dd the drives as I said originally, and when the system can't find 
the array on the bigger discs, recreate the array using the same options 
the original array had
(2) degrade the existing array to two discs, replace the soft-failed 
disc with one of the new discs, temporarily make a filesystem on it, 
copy the data; replace the other two discs, make a new array on them 
with a missing drive, copy the data back; scrub the temporarily-made 
filesystem and add the drive to the array.

Actually I think something along the lines of Majed B's suggestion - in 
this case hang a spare drive out the side of the chassis, and just copy 
twice - would be quicker than (2). But (1) should work, shouldn't it?

Cheers,

John.

Re: Growing after replacing with larger discs

[Majed B.]

You should never degrade the array to copy its contents. Its very
risky. There might be bad sectors on one or more disks and if you
don't have all disks at hand, you may not be able to rebuild the array
on the new disk.

As for using dd, as others have pointed, the metadata won't be in
place and if you create a new array after using dd, it'll still
require a resync and the data will be destroyed.

On Sat, Mar 13, 2010 at 6:18 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 12/03/2010 18:09, Luca Berra wrote:
>>
>> On Fri, Mar 12, 2010 at 05:39:47PM +0000, John Robinson wrote:
>>>
>>> Before I have to go to a customer's premises and try it, I wanted to ask
>>> if anyone had already done anything like this: I have a CentOS 5 system with
>>> 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I want to
>>> grow it by replacing the drives. If I dd the small partitions onto larger
>>> ones on the new discs, then reboot, will the system recognise the RAID-5 -
>>> which now has its metadata somewhere in the middle of the paritions not at
>>> the end - so that I can then --grow?
>>
>> I believe it won't work
>> mdadm has no way to know where the metadata is in the above case.
>>
>> you have to follow the painful procedure
>> fail one drive (mdadm /dev/mdX --fail /dev/sdY)
>> remove it (mdadm /dev/mdX --remove /dev/sdY)
>> replace drive with larger one
>> create larger partition
>> add drive again (mdadm /dev/mdX --add /dev/sdY)
>> wait for it to resync...
>> ...and repeat until all drives have been replaced
>
> Yeuch. That'll take forever, because I'd need to rebuild 3 times. I see two
> other options:
> (1) dd the drives as I said originally, and when the system can't find the
> array on the bigger discs, recreate the array using the same options the
> original array had
> (2) degrade the existing array to two discs, replace the soft-failed disc
> with one of the new discs, temporarily make a filesystem on it, copy the
> data; replace the other two discs, make a new array on them with a missing
> drive, copy the data back; scrub the temporarily-made filesystem and add the
> drive to the array.
>
> Actually I think something along the lines of Majed B's suggestion - in this
> case hang a spare drive out the side of the chassis, and just copy twice -
> would be quicker than (2). But (1) should work, shouldn't it?
>
> Cheers,
>
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing after replacing with larger discs

[John Robinson]

On 13/03/2010 15:21, Majed B. wrote:
> You should never degrade the array to copy its contents. Its very
> risky. There might be bad sectors on one or more disks and if you
> don't have all disks at hand, you may not be able to rebuild the array
> on the new disk.

OK that's fair, you're right I don't want to risk losing data because I 
was using a degraded array.

> As for using dd, as others have pointed, the metadata won't be in
> place and if you create a new array after using dd, it'll still
> require a resync and the data will be destroyed.

I don't see whay either it'd require a resync - although I didn't say 
it, I was planning to zero the rest of the larger drives and recreate 
with --assume-clean - nor why data would be destroyed if I didn't create 
with --assume-clean, after all the data will all be in the right place.

But I think on balance I'm going to save the contents of the original 
array to an extra drive, create a new array on the big discs, and copy 
the data back. It shouldn't take too much more time than copying the 
discs individually with dd, even though I'm copying the data twice 
rather than once, and it's both safer and gives me a free defrag.

Cheers,

John.

Re: Growing after replacing with larger discs

[Majed B.]

What I meant was: If you use dd to clone an old disk to a new one of a
larger size, metadata isn't where it should be and the new disk won't
be recognized as part of an array.

So if you do that to all disks, then attempt to create a new array on
the new disks, it will cause a resync. And even if it didn't resync,
data mapping may be incorrect.

For argument's sake, let's assume that cloning the disks would work.
It means cloning 3 disks.

The other option that you presented, which I hope you avoid, is
degrading the array and presenting the new disks. This would also
require 3 resyncs.

The option where you copy the data off the old array then back to the
new one consist of 2 copy operations only. May not be as fast as a dd
operation, but still should consume less time than 3 resyncs.

Also, while copying the original data, if the filesystem has problems,
you'll see them right away and probably identify which files are
affected. I hope everything goes smooth for you, but in case such
problems occur, do NOT run fsck!
Clone the filesystem first for best assurance of data safety.

Good luck!

On Sat, Mar 13, 2010 at 9:13 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 13/03/2010 15:21, Majed B. wrote:
>>
>> You should never degrade the array to copy its contents. Its very
>> risky. There might be bad sectors on one or more disks and if you
>> don't have all disks at hand, you may not be able to rebuild the array
>> on the new disk.
>
> OK that's fair, you're right I don't want to risk losing data because I was
> using a degraded array.
>
>> As for using dd, as others have pointed, the metadata won't be in
>> place and if you create a new array after using dd, it'll still
>> require a resync and the data will be destroyed.
>
> I don't see whay either it'd require a resync - although I didn't say it, I
> was planning to zero the rest of the larger drives and recreate with
> --assume-clean - nor why data would be destroyed if I didn't create with
> --assume-clean, after all the data will all be in the right place.
>
> But I think on balance I'm going to save the contents of the original array
> to an extra drive, create a new array on the big discs, and copy the data
> back. It shouldn't take too much more time than copying the discs
> individually with dd, even though I'm copying the data twice rather than
> once, and it's both safer and gives me a free defrag.
>
> Cheers,
>
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing after replacing with larger discs

[Beolach]

Forgot to send to the list...

On Sat, Mar 13, 2010 at 13:42, Beolach <beolach@gmail.com> wrote:
> On Fri, Mar 12, 2010 at 10:39, John Robinson
> <john.robinson@anonymous.org.uk> wrote:
>> Before I have to go to a customer's premises and try it, I wanted to ask if
>> anyone had already done anything like this: I have a CentOS 5 system with
>> 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I want to
>> grow it by replacing the drives. If I dd the small partitions onto larger
>> ones on the new discs, then reboot, will the system recognise the RAID-5 -
>> which now has its metadata somewhere in the middle of the paritions not at
>> the end - so that I can then --grow?
>>
>
> I have no idea if this would actually work, but maybe try partitioning
> the new, larger drives so the partitions are identical in size to the
> old discs, copying the old discs to the new partitions, starting the
> array with the new partitions, resizing the new partitions, and then
> running --grow.  If mdadm can re-check the size of the member devices
> of a running array at --grow time, this might work, but if it only
> checks member device sizes at creation or assemble time, then it
> probably won't.
>
>
> Good luck,
> Conway S. Smith
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing after replacing with larger discs

[Majed B.]

Beolach,

That involves double the work!

On Sat, Mar 13, 2010 at 11:45 PM, Beolach <beolach@gmail.com> wrote:
> Forgot to send to the list...
>
> On Sat, Mar 13, 2010 at 13:42, Beolach <beolach@gmail.com> wrote:
>> On Fri, Mar 12, 2010 at 10:39, John Robinson
>> <john.robinson@anonymous.org.uk> wrote:
>>> Before I have to go to a customer's premises and try it, I wanted to ask if
>>> anyone had already done anything like this: I have a CentOS 5 system with
>>> 0.90 metadata and type fd partitions with RAID-5 over 3 discs. I want to
>>> grow it by replacing the drives. If I dd the small partitions onto larger
>>> ones on the new discs, then reboot, will the system recognise the RAID-5 -
>>> which now has its metadata somewhere in the middle of the paritions not at
>>> the end - so that I can then --grow?
>>>
>>
>> I have no idea if this would actually work, but maybe try partitioning
>> the new, larger drives so the partitions are identical in size to the
>> old discs, copying the old discs to the new partitions, starting the
>> array with the new partitions, resizing the new partitions, and then
>> running --grow.  If mdadm can re-check the size of the member devices
>> of a running array at --grow time, this might work, but if it only
>> checks member device sizes at creation or assemble time, then it
>> probably won't.
>>
>>
>> Good luck,
>> Conway S. Smith
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing after replacing with larger discs

[Stefan /*St0fF*/ Hübner]

Hi all,

Am 13.03.2010 19:22, schrieb Majed B.:
> What I meant was: If you use dd to clone an old disk to a new one of a
> larger size, metadata isn't where it should be and the new disk won't
> be recognized as part of an array.

I mean to have read that he uses 0.9 metadata.  mdadm finds the device
size (i.e. partition size) and locates the metadata according to that.
So if he has partitioned the drives containing the raid, the dd-way
should work on one hand (if he clones the complete drive, not only the
partitions).  On the other hand he'd have a backup at hand if something
goes terribly wrong.  Or might I be mistaken now?

> So if you do that to all disks, then attempt to create a new array on
> the new disks, it will cause a resync. And even if it didn't resync,
> data mapping may be incorrect.

I second that - recreating is only good on same size devices - but
again: if the partition is same size, it should work.
> 
> For argument's sake, let's assume that cloning the disks would work.
> It means cloning 3 disks.

Yes, using sgp_dd this works very well and quickly.  And you don't have
problems arising from broken read-sectors being skipped. (yes, I know
there is conv=noerror,sync; but I had my problems with that...)

> The other option that you presented, which I hope you avoid, is
> degrading the array and presenting the new disks. This would also
> require 3 resyncs.

Yes, that is a bad idea.

> The option where you copy the data off the old array then back to the
> new one consist of 2 copy operations only. May not be as fast as a dd
> operation, but still should consume less time than 3 resyncs.

right on one hand.  the other hand is: on my laboratory computer (which
is used for raid-recoveries and such) 4 parallel sgp_dd operations work
at about 90MB/s (if the drives are quick enough, starting with 115MB/s,
and sinking speed as the r/w-heads move outside).  But one copy only
reaches 50MB/s.  Multiple parallel copies do not give any speedup.
> 
> Also, while copying the original data, if the filesystem has problems,
> you'll see them right away and probably identify which files are
> affected. I hope everything goes smooth for you, but in case such
> problems occur, do NOT run fsck!

second that - if fsck is needed, create a dd-backup to run it on.

> Clone the filesystem first for best assurance of data safety.
> 
> Good luck!
> 
[...]

Also my best wishes for the operation...

Stefan