messed up changing chunk size

messed up changing chunk size

[Konstantin Svist]

  I've been playing around with RAID config and may have finally messed 
it up.

Initially, I created the array with 3 300G drives:
# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 
/dev/sdb1 /dev/sdc1
# mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0

It's been working nicely so far, and I decided to add a 4th 300G drive:
# mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0

That finished overnight, while I looked around and found that chunk size 
of 512 should work better. I unmounted the FS and ran
# mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
mdadm: component size 293033536K is not a multiple of chunksize 512K

so I sized it down a bit:
# mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0

and then back to resizing chunks:
mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0

It's running right now:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
       293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]
       [====>................]  reshape = 22.7% (66540032/293033472) 
finish=947.3min speed=3984K/sec


But just now I tried to mount the filesystem and it's failing:
EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of 
device (73258368 blocks)

Here's the question, then: am I royally screwed or is my data still 
there? How do I recover?



Yes, in retrospect I should've probably checked around whether this 
process has been successful for others.. but that's why they say 
hindsight is 20/20

RE: messed up changing chunk size

[Steven Haigh]

It seems that when you shrank the array a little bit that you chopped off the end of the ext4 filesystem...

The raid array is now smaller than the filesystem says it is - hence your error.

--
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299


> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Konstantin Svist
> Sent: Monday, 19 July 2010 10:32 AM
> To: linux-raid@vger.kernel.org
> Subject: messed up changing chunk size
> 
>   I've been playing around with RAID config and may have finally messed it
> up.
> 
> Initially, I created the array with 3 300G drives:
> # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4
> /dev/sdb1 /dev/sdc1
> # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
> 
> It's been working nicely so far, and I decided to add a 4th 300G drive:
> # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak
> /dev/md0
> 
> That finished overnight, while I looked around and found that chunk size of
> 512 should work better. I unmounted the FS and ran # mdadm --grow -c 512 -
> -backup-file=/root/grow_md0_rechunk.bak /dev/md0
> mdadm: component size 293033536K is not a multiple of chunksize 512K
> 
> so I sized it down a bit:
> # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak
> /dev/md0
> 
> and then back to resizing chunks:
> mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak
> /dev/md0
> 
> It's running right now:
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
>        293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>        [====>................]  reshape = 22.7% (66540032/293033472)
> finish=947.3min speed=3984K/sec
> 
> 
> But just now I tried to mount the filesystem and it's failing:
> EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of device
> (73258368 blocks)
> 
> Here's the question, then: am I royally screwed or is my data still there? How
> do I recover?
> 
> 
> 
> Yes, in retrospect I should've probably checked around whether this
> process has been successful for others.. but that's why they say
> hindsight is 20/20
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: messed up changing chunk size

[Konstantin Svist]

  On 07/18/2010 05:41 PM, Steven Haigh wrote:
> It seems that when you shrank the array a little bit that you chopped off the end of the ext4 filesystem...
>
> The raid array is now smaller than the filesystem says it is - hence your error.

That's what I thought, too.
But, again, the question is whether or not I can recover the data...

RE: messed up changing chunk size

[Guy Watkins]

I think this should work:
Put the chunk size back, resize the array to the original size, then run
fsck and hope it did not need the data on the end that got trashed.  Then
shrink your file system first (by a really safe margin), then resize the
array, then change chunk size.  Then grow the file system to use the max
allowed.

Maybe the data on the end did not get trashed, but I don't know.


} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Konstantin Svist
} Sent: Sunday, July 18, 2010 8:32 PM
} To: linux-raid@vger.kernel.org
} Subject: messed up changing chunk size
} 
}   I've been playing around with RAID config and may have finally messed
} it up.
} 
} Initially, I created the array with 3 300G drives:
} # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4
} /dev/sdb1 /dev/sdc1
} # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
} 
} It's been working nicely so far, and I decided to add a 4th 300G drive:
} # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
} 
} That finished overnight, while I looked around and found that chunk size
} of 512 should work better. I unmounted the FS and ran
} # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
} mdadm: component size 293033536K is not a multiple of chunksize 512K
} 
} so I sized it down a bit:
} # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
} 
} and then back to resizing chunks:
} mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
} 
} It's running right now:
} # cat /proc/mdstat
} Personalities : [raid6] [raid5] [raid4]
} md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
}        293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4]
} [UUUU]
}        [====>................]  reshape = 22.7% (66540032/293033472)
} finish=947.3min speed=3984K/sec
} 
} 
} But just now I tried to mount the filesystem and it's failing:
} EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of
} device (73258368 blocks)
} 
} Here's the question, then: am I royally screwed or is my data still
} there? How do I recover?
} 
} 
} 
} Yes, in retrospect I should've probably checked around whether this
} process has been successful for others.. but that's why they say
} hindsight is 20/20
} 
} --
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in
} the body of a message to majordomo@vger.kernel.org
} More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: messed up changing chunk size

[Konstantin Svist]

  Right now the reshape is at "32.1% (94138368/293033472) 
finish=861.4min speed=3847K/sec"
Is it safe to reshape back right now, will it pick up the command?

On 07/18/2010 06:29 PM, Guy Watkins wrote:
> I think this should work:
> Put the chunk size back, resize the array to the original size, then run
> fsck and hope it did not need the data on the end that got trashed.  Then
> shrink your file system first (by a really safe margin), then resize the
> array, then change chunk size.  Then grow the file system to use the max
> allowed.
>
> Maybe the data on the end did not get trashed, but I don't know.
>
>
> } -----Original Message-----
> } From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> } owner@vger.kernel.org] On Behalf Of Konstantin Svist
> } Sent: Sunday, July 18, 2010 8:32 PM
> } To: linux-raid@vger.kernel.org
> } Subject: messed up changing chunk size
> }
> }   I've been playing around with RAID config and may have finally messed
> } it up.
> }
> } Initially, I created the array with 3 300G drives:
> } # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4
> } /dev/sdb1 /dev/sdc1
> } # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
> }
> } It's been working nicely so far, and I decided to add a 4th 300G drive:
> } # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
> }
> } That finished overnight, while I looked around and found that chunk size
> } of 512 should work better. I unmounted the FS and ran
> } # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
> } mdadm: component size 293033536K is not a multiple of chunksize 512K
> }
> } so I sized it down a bit:
> } # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
> }
> } and then back to resizing chunks:
> } mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
> }
> } It's running right now:
> } # cat /proc/mdstat
> } Personalities : [raid6] [raid5] [raid4]
> } md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
> }        293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4]
> } [UUUU]
> }        [====>................]  reshape = 22.7% (66540032/293033472)
> } finish=947.3min speed=3984K/sec
> }
> }
> } But just now I tried to mount the filesystem and it's failing:
> } EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of
> } device (73258368 blocks)
> }
> } Here's the question, then: am I royally screwed or is my data still
> } there? How do I recover?
> }
> }
> }
> } Yes, in retrospect I should've probably checked around whether this
> } process has been successful for others.. but that's why they say
> } hindsight is 20/20
> }
> } --
> } To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> } the body of a message to majordomo@vger.kernel.org
> } More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

RE: messed up changing chunk size

[Guy Watkins]

No idea, but I would not risk it.  Let it finish, unless someone else has a
useful tip.

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Konstantin Svist
} Sent: Sunday, July 18, 2010 9:46 PM
} To: linux-raid@vger.kernel.org
} Subject: Re: messed up changing chunk size
} 
}   Right now the reshape is at "32.1% (94138368/293033472)
} finish=861.4min speed=3847K/sec"
} Is it safe to reshape back right now, will it pick up the command?
} 
} On 07/18/2010 06:29 PM, Guy Watkins wrote:
} > I think this should work:
} > Put the chunk size back, resize the array to the original size, then run
} > fsck and hope it did not need the data on the end that got trashed.
} Then
} > shrink your file system first (by a really safe margin), then resize the
} > array, then change chunk size.  Then grow the file system to use the max
} > allowed.
} >
} > Maybe the data on the end did not get trashed, but I don't know.
} >
} >
} > } -----Original Message-----
} > } From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} > } owner@vger.kernel.org] On Behalf Of Konstantin Svist
} > } Sent: Sunday, July 18, 2010 8:32 PM
} > } To: linux-raid@vger.kernel.org
} > } Subject: messed up changing chunk size
} > }
} > }   I've been playing around with RAID config and may have finally
} messed
} > } it up.
} > }
} > } Initially, I created the array with 3 300G drives:
} > } # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3
} /dev/sda4
} > } /dev/sdb1 /dev/sdc1
} > } # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
} > }
} > } It's been working nicely so far, and I decided to add a 4th 300G
} drive:
} > } # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak
} /dev/md0
} > }
} > } That finished overnight, while I looked around and found that chunk
} size
} > } of 512 should work better. I unmounted the FS and ran
} > } # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak
} /dev/md0
} > } mdadm: component size 293033536K is not a multiple of chunksize 512K
} > }
} > } so I sized it down a bit:
} > } # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak
} /dev/md0
} > }
} > } and then back to resizing chunks:
} > } mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
} > }
} > } It's running right now:
} > } # cat /proc/mdstat
} > } Personalities : [raid6] [raid5] [raid4]
} > } md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
} > }        293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2
} [4/4]
} > } [UUUU]
} > }        [====>................]  reshape = 22.7% (66540032/293033472)
} > } finish=947.3min speed=3984K/sec
} > }
} > }
} > } But just now I tried to mount the filesystem and it's failing:
} > } EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of
} > } device (73258368 blocks)
} > }
} > } Here's the question, then: am I royally screwed or is my data still
} > } there? How do I recover?
} > }
} > }
} > }
} > } Yes, in retrospect I should've probably checked around whether this
} > } process has been successful for others.. but that's why they say
} > } hindsight is 20/20
} > }
} > } --
} > } To unsubscribe from this list: send the line "unsubscribe linux-raid"
} in
} > } the body of a message to majordomo@vger.kernel.org
} > } More majordomo info at  http://vger.kernel.org/majordomo-info.html
} >
} >
} 
} --
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in
} the body of a message to majordomo@vger.kernel.org
} More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: messed up changing chunk size

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

On Sun, 18 Jul 2010 17:32:25 -0700
Konstantin Svist <fry.kun@gmail.com> wrote:

> I looked around and found that chunk size 
> of 512 should work better.

Not true, at least with RAID5/6 a chunk size of 64K performs faster, see
http://louwrentius.blogspot.com/2010/05/raid-level-and-chunk-size-benchmarks.html
http://alephnull.com/benchmarks/sata2009/chunksize.html

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: messed up changing chunk size

[Jools Wills]

On 19/07/10 04:53, Roman Mamedov wrote:
> On Sun, 18 Jul 2010 17:32:25 -0700
> Konstantin Svist<fry.kun@gmail.com>  wrote:
>
>> I looked around and found that chunk size
>> of 512 should work better.
>
> Not true, at least with RAID5/6 a chunk size of 64K performs faster, see
> http://louwrentius.blogspot.com/2010/05/raid-level-and-chunk-size-benchmarks.html
> http://alephnull.com/benchmarks/sata2009/chunksize.html

There is no mention in this benchmark tests if he modified the stripe 
cache. With an increased stripe size a larger stripe cache would be 
needed for good performance (at least from my experience)

Best Regards

Jools

Re: messed up changing chunk size

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

On Mon, 19 Jul 2010 14:24:12 +0100
Jools Wills <jools@oxfordinspire.co.uk> wrote:

> >> I looked around and found that chunk size
> >> of 512 should work better.
> >
> > Not true, at least with RAID5/6 a chunk size of 64K performs faster, see
> > http://louwrentius.blogspot.com/2010/05/raid-level-and-chunk-size-benchmarks.html
> > http://alephnull.com/benchmarks/sata2009/chunksize.html
> 
> There is no mention in this benchmark tests if he modified the stripe 
> cache. With an increased stripe size a larger stripe cache would be 
> needed for good performance (at least from my experience)

That's true, but increasing the stripe cache helps across all chunk sizes, even
with relatively low 128K, see [1]. And I don't think that a large-chunk(512K)
configuration will significantly, if at all, outperform a small-chunks(64K)
one at any given stripe cache size (set to the same value in both cases).

There's also an opinion I heard from a couple of sources, that the stripe size
better be chosen so that either one stripe, or a whole stride fits inside the
CPU's L2 or L3 cache. Sounds logical, though I am not convinced that this is
what causes the performance difference between various stripe sizes.


[1]
http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: messed up changing chunk size

[Keld Simonsen]

On Mon, Jul 19, 2010 at 09:53:51AM +0600, Roman Mamedov wrote:
> On Sun, 18 Jul 2010 17:32:25 -0700
> Konstantin Svist <fry.kun@gmail.com> wrote:
> 
> > I looked around and found that chunk size 
> > of 512 should work better.
> 
> Not true, at least with RAID5/6 a chunk size of 64K performs faster, see
> http://louwrentius.blogspot.com/2010/05/raid-level-and-chunk-size-benchmarks.html
> http://alephnull.com/benchmarks/sata2009/chunksize.html

Thanks for these benchmarks, I added them to our benchmarks page
on the wiki. 

It seems odd to me to use sata controllers on a 32-bit PCI bus,
this should limit performance to about 130 MB/s - which is also seen tin the graphs.
I think people would be better off using the on-board SATA controller, this would
give normally something like 4 SATA ports connected to the southbridge, which
normally is much faster than a 32-bit PCI bus, and also faster than many
PCI-E busses.  So I think using all the on-board SATA ports first, and the
using the SATA ports on a bus-connceted controller would give better
performance.

In the louwrentius page, I miss info on which layout of RAID10 that is used.
The 3 layouts (near, far, offset) has very different characteristics.

best regards
keld

Re: messed up changing chunk size

[Konstantin Svist]

  On 07/18/2010 05:32 PM, Konstantin Svist wrote:
>  I've been playing around with RAID config and may have finally messed 
> it up.
>
> Initially, I created the array with 3 300G drives:
> # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 
> /dev/sda4 /dev/sdb1 /dev/sdc1
> # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
>
> It's been working nicely so far, and I decided to add a 4th 300G drive:
> # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
>
> That finished overnight, while I looked around and found that chunk 
> size of 512 should work better. I unmounted the FS and ran
> # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
> mdadm: component size 293033536K is not a multiple of chunksize 512K
>
> so I sized it down a bit:
> # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak 
> /dev/md0
>
> and then back to resizing chunks:
> mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
>
> It's running right now:
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
>       293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 
> [4/4] [UUUU]
>       [====>................]  reshape = 22.7% (66540032/293033472) 
> finish=947.3min speed=3984K/sec
>
>
> But just now I tried to mount the filesystem and it's failing:
> EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of 
> device (73258368 blocks)
>
> Here's the question, then: am I royally screwed or is my data still 
> there? How do I recover?
>
>
>
> Yes, in retrospect I should've probably checked around whether this 
> process has been successful for others.. but that's why they say 
> hindsight is 20/20
>


Just noticed that I got this in /var/log/messages when I sized it down:
Jul 18 12:00:44 slon kernel: md0: detected capacity change from 
900199022592 to 300066275328
Jul 18 12:00:44 slon kernel: md0: unknown partition table

And scraped from command line session:

[root@slon ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
       879100608 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[root@slon ~]# mdadm --grow -z 293033472 
--backup-file=/root/grow_md0_size.bak /dev/md0
mdadm: component size of /dev/md0 has been set to 293033472K
[root@slon ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
       293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

Looks like instead of setting the number of blocks per-drive (as per the 
man page), the -z option set the number of total blocks for the array, 
instead. Which means I'm very thoroughly screwed.

Right now I'm trying to reshape back to the old chunk size and will then 
try to resize the array back to the previous size. If the chunk size 
change and array resize are both fully reversible -- and I don't think 
I'm THAT lucky -- then I might be able to get some stuff back.
Will see, I guess.

Re: messed up changing chunk size

[Konstantin Svist]

On 07/19/2010 11:04 AM, Konstantin Svist wrote:
> Just noticed that I got this in /var/log/messages when I sized it down:
> Jul 18 12:00:44 slon kernel: md0: detected capacity change from 
> 900199022592 to 300066275328
> Jul 18 12:00:44 slon kernel: md0: unknown partition table
>
> And scraped from command line session:
>
> [root@slon ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
>       879100608 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
> [root@slon ~]# mdadm --grow -z 293033472 
> --backup-file=/root/grow_md0_size.bak /dev/md0
> mdadm: component size of /dev/md0 has been set to 293033472K
> [root@slon ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
>       293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>
> Looks like instead of setting the number of blocks per-drive (as per 
> the man page), the -z option set the number of total blocks for the 
> array, instead. Which means I'm very thoroughly screwed.
>
> Right now I'm trying to reshape back to the old chunk size and will 
> then try to resize the array back to the previous size. If the chunk 
> size change and array resize are both fully reversible -- and I don't 
> think I'm THAT lucky -- then I might be able to get some stuff back.
> Will see, I guess.

Can someone confirm that this command should work to revert the size?
mdadm --grow -z 879100608 --backup-file=/root/grow_md0_size_revert.bak 
/dev/md0

Re: messed up changing chunk size

[Konstantin Svist]

  On 07/19/2010 09:28 PM, Konstantin Svist wrote:
> Can someone confirm that this command should work to revert the size?
> mdadm --grow -z 879100608 --backup-file=/root/grow_md0_size_revert.bak 
> /dev/md0

Finally sat down and thought about it - created a temporary raid using 
/dev/ram## and copied some files to it, did the same operations on it.
Amazingly, all the files came back. Will see if the real thing works too

[SOLVED] Re: messed up changing chunk size

[Konstantin Svist]

  On 07/19/2010 11:29 PM, Konstantin Svist wrote:
>  On 07/19/2010 09:28 PM, Konstantin Svist wrote:
>> Can someone confirm that this command should work to revert the size?
>> mdadm --grow -z 879100608 
>> --backup-file=/root/grow_md0_size_revert.bak /dev/md0
>
> Finally sat down and thought about it - created a temporary raid using 
> /dev/ram## and copied some files to it, did the same operations on it.
> Amazingly, all the files came back. Will see if the real thing works too

For some reason, mdadm refused to resize back to max:

# mdadm --grow -z max --backup-file=/root/grow_md0_size_back.bak /dev/md0
mdadm: component size of /dev/md0 has been set to 293033536K
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
       293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]


After stopping and re-assembling the array, though, it comes up with 
full size but with only 3 of 4 drives:

# mdadm --assemble --scan
mdadm: /dev/md0 has been started with 3 drives (out of 4).
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda4[0] sdc1[2] sdb1[1]
       879100608 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

The array is degraded but data is all there, so I'm backing it up right 
now. No idea why it doesn't like the 4th drive.

Re: [SOLVED] Re: messed up changing chunk size

[Konstantin Svist]

On 07/20/2010 09:30 AM, Konstantin Svist wrote:
> For some reason, mdadm refused to resize back to max:
>
> # mdadm --grow -z max --backup-file=/root/grow_md0_size_back.bak /dev/md0
> mdadm: component size of /dev/md0 has been set to 293033536K
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
>       293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>
>
> After stopping and re-assembling the array, though, it comes up with 
> full size but with only 3 of 4 drives:
>
> # mdadm --assemble --scan
> mdadm: /dev/md0 has been started with 3 drives (out of 4).
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sda4[0] sdc1[2] sdb1[1]
>       879100608 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
>
> The array is degraded but data is all there, so I'm backing it up 
> right now. No idea why it doesn't like the 4th drive.
>

3 out of 4 was due to bad /etc/mdadm.conf -- I forgot the / in front of 
/dev/sdd1
It's now rebuilding and everything is fine so far