RAID5 member disks shrunk

RAID5 member disks shrunk

[Alex Leach]

Dear list,

I hope you all had a good festive period. Happy New Year!

Sorry to burden you with this, but I'm in real need of some help! I  
recently suffered from a failed hard disk in my RAID5 array, and have made  
some rookie errors since.. I'll try and detail what I did as accurately  
and concisely as possible. Some help in recovering data from my ext4  
partition would be greatly appreciated!

Set up:-
   1. RAID0 was pre-configured on new machine, with 2x300GB WD  
Velociraptors. (2010)
   2. Got another, identical drive and upgrade to RAID5 using Windows Intel  
Matrix Manager. (2011)

RAID Partitions (roughly):
   1. 100MB windows boot loader.
   2. 342GB NTFS Windows installation partition
   3. 256GB ext4 partition with Kubuntu 12.10, and dmraid.

Non-RAID Partitions (for completeness):
   Backup Drive: 2TB.
      1. 200GB Ubuntu 12.10 ext4 (recovabuntu) with dmraid.
      2. 1800GB NTFS backup and media partition.
   SAS drive: 600GB
      1. 600GB ext4 partition, for SQL and other databases.


Problem, started about two weeks ago:
   1. Suffer an IO error on member disk, whilst in Kubuntu. Array DEGRADED.
   2. Goes unnoticed for at most a day. Shut down immediately, replace HD  
with new, identical disk.
   3. Boot into Windows, use Matrix Storage manager to rebuild array on to  
new disk. Array now fine.
   4. At some point after the array was rebuilt, back in Kubuntu, the new  
disk also raised an IO error. DEGRADED again.

Stop gap:
   5. Panic, and boot into recovabuntu on non-raid disk. The 2TB drive is  
not a drive I want to (over-)use when not strictly necessary, as this has  
all my media on it and a couple of (old) backups.
   6. WD have been stalling the RMA on the failed drive(s) over Christmas,  
and I didn't want to stress out my 2TB drive too much.
   7. Decide to get an SSD and use that as primary boot device.
   8. Whilst at it, I also bought and installed a 5 bay Icy Box HD  
backplane, upgrading from a 3 bay version. This was trickier than I  
thought; I had to completely disassemble the PC and mod the 5.25" bays in  
the case, with drill, dremel, rivet gun, and some spray paint for good  
measure :)

Human error:
   9. Putting it back together, I accidentally connected one of the good  
RAID member drives to the SAS controller, and the SAS drive into one of  
the SATA ports, which has an Intel ICH10R controller.
   10. Took a couple boots into BIOS and recovabuntu to realise what I'd  
done wrong. Array now wants to rebuild on to good disk, using data from  
the drive that IO Error'd on me. Don't like the sound of that, so I leave  
it degraded.

(Botched) Recovery:
   11. Install Arch Linux and Windows onto separate partitions on the SSD  
drive.
   12. Read that Intel now support mdadm over dmraid, so install that on  
Arch.
   13. Backup some information:

     $ sudo mdadm -D /dev/md/RAID5

/dev/md/RAID5:
       Container : /dev/md/imsm0, member 0
      Raid Level : raid5
      Array Size : 586066944 (558.92 GiB 600.13 GB)
   Used Dev Size : 293033600 (279.46 GiB 300.07 GB)
    Raid Devices : 3
   Total Devices : 2

           State : active, degraded
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-asymmetric
      Chunk Size : 64K


            UUID : 789c5fd2:da9dd3d2:b57d7def:89f68d3c
     Number   Major   Minor   RaidDevice State
        1       8        0        0      active sync   /dev/sda
        1       0        0        1      removed
        0       8       16        2      active sync   /dev/sdb


     $ sudo mdadm -D /dev/md/imsm0

/dev/md/imsm0:
         Version : imsm
      Raid Level : container
   Total Devices : 3

Working Devices : 3


            UUID : 4528e473:abbe0e9f:25a8bb6b:bb9e9999
   Member Arrays : /dev/md/RAID5

     Number   Major   Minor   RaidDevice

        0       8        0        -        /dev/sda
        1       8       96        -        /dev/sdg
        2       8       16        -        /dev/sdb


     $ sfdisk -d /dev/sda > sda.out (and same for sdg and sdb)
     $ ls -l /dev/sd[agb].out
-rw-r--r-- 1 albl500 users   259 Dec 18 10:45 sda.out
-rw-r--r-- 1 albl500 users     0 Dec 18 10:47 sdb.out
-rw-r--r-- 1 albl500 users     0 Dec 18 10:47 sdg.out

     $ cat sda.out

# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size=   204800, Id= 7, bootable
/dev/sda2 : start=   206848, size=667994112, Id= 7
/dev/sda3 : start=668200960, size=503928832, Id= 7
/dev/sda4 : start=        0, size=        0, Id= 0

   14. Figure I should zero the superblocks and re-create the array. Really  
should have backed up as much as possible before this...
     $ sudo mdadm --misc --zero-superblock /dev/sd[agb]

     $ sudo mdadm --create /dev/md/imsm0 --raid-devices=2  
--uuid='4528e473:abbe0e9f:25a8bb6b:bb9e9999' --metadata=imsm /dev/sda  
/dev/sdg /dev/sdb
mdadm: container /dev/md/imsm0 prepared.

     $ sudo mdadm --create --verbose /dev/md/RAID5 --raid-devices=3  
--level=5 --chunk=64 --layout=la --size=293033600  
--uuid='789c5fd2b57d7def:89f68d3c' -e imsm /dev/sda /dev/sdg /dev/sdb
mdadm: /dev/sdb not enough space (586066062 < 586067200)
mdadm: /dev/sdb is smaller than given size. 0K < 293033600K + metadata
mdadm: /dev/sdc not enough space (586066062 < 586067200)
mdadm: /dev/sdc is not suitable for this array.
mdadm: create aborted


   15. If I leave off the size option, then the array gets built, and  
verifies all 3 drives without a single IO error. So the drives seem to be  
okay now. But when they are assembled, the Array Size and Used Dev Size  
come out smaller than with the previous array, whose details are above.  
Now, after re-creating and verifying (with no size option), I get these  
results:

     $ sudo mdadm -D /dev/md/RAID5

/dev/md/RAID5:
       Container : /dev/md/imsm0, member 0
      Raid Level : raid5
      Array Size : 586065920 (558.92 GiB 600.13 GB)
   Used Dev Size : 293033024 (279.46 GiB 300.07 GB)
    Raid Devices : 3
   Total Devices : 3

           State : clean
  Active Devices : 3
Working Devices : 3
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-asymmetric
      Chunk Size : 64K


            UUID : 2cdc3f5c:7d91eb7d:51b57c72:bdbfc1fb
     Number   Major   Minor   RaidDevice State
        2       8       16        0      active sync   /dev/sda
        1       8      112        1      active sync   /dev/sdg
        0       8       32        2      active sync   /dev/sdb

     $ sudo sfdisk -d /dev/md126  # this is: /dev/md/RAID5

# partition table of /dev/md126
unit: sectors

/dev/md126p1 : start=     2048, size=   204800, Id= 7, bootable
/dev/md126p2 : start=   206848, size=667994112, Id= 7
/dev/md126p3 : start=668200960, size=503928832, Id= 7
/dev/md126p4 : start=        0, size=        0, Id= 0

   16. When I saw these partitions in /dev, I was very happy! I thought I  
had everything back, but I couldn't mount a single one of the file  
systems. I used testdisk to try and recover the partitions, and this  
changed the partition table to point to different start and end positions  
for p1 and p2, deleting p3. I could mount and recover data from my NTFS  
windows partition, but the ext4 partition could not be recovered at all; I  
suspect because of the new Minor numbers and the apparent reduced device  
and array sizes.

   17. It seems the Used Dev size has reduced by 576kB ( = 512kB + 64kB),  
which I thought could be partially explained by an MBR being present on  
the disks, but I have been unable to think where the extra 64kB could be  
taken up.

   18. So I backup the first 512kB of each member disk, and write zeros to  
those areas, with dd.

     $ sudo dd if=/dev/sdX of=sdX.out bs=512 count=1
     $ sudo dd if=/dev/zero of=/dev/sdX bs=512 count=1

   19. Still unable to create an array with the right size (after zeroing  
superblocks). Updating the super-minor doesn't seem to apply to imsm  
containers, and I can't think of any other way to get this to re-assemble  
correctly.

So, I've got myself into a real mess here. I don't have the space to  
backup every disk image, which I've seen recommended pretty much  
everywhere.
Any help getting me out of this mess would be really, really appreciated!

Kind regards,
Alex

-- 
Using Opera's mail client: http://www.opera.com/mail/

Re: RAID5 member disks shrunk

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Thu, 03 Jan 2013 15:33:55 -0000
"Alex Leach" <beamesleach@gmail.com> wrote:

>    9. Putting it back together, I accidentally connected one of the good  
> RAID member drives to the SAS controller, and the SAS drive into one of  
> the SATA ports, which has an Intel ICH10R controller.

Try checking if your controller/BIOS have shrunk the disk by establishing an
HPA on it, the easiest way to check if you have any is perhaps with 

  dmesg | grep HPA

If that is indeed the case, you can control it with "hdparm -N" (see its man page).

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: RAID5 member disks shrunk

[Alex Leach]

(Replying again, on list. Sorry Roman)

On Thu, 03 Jan 2013 15:42:36 -0000, Roman Mamedov <rm@romanrm.ru> wrote:

> Try checking if your controller/BIOS have shrunk the disk by  
> establishing an
> HPA on it, the easiest way to check if you have any is perhaps with
>
>   dmesg | grep HPA
>
> If that is indeed the case, you can control it with "hdparm -N" (see its  
> man page).
>

Thanks for the reply.
The dmesg command produces no output. Anywhere else I should look?

Cheers,
Alex

-- 
Using Opera's mail client: http://www.opera.com/mail/

Re: RAID5 member disks shrunk

[Alex Leach]

On Thu, 03 Jan 2013 15:42:36 -0000, Roman Mamedov <rm@romanrm.ru> wrote:

> If that is indeed the case, you can control it with "hdparm -N" (see its  
> man page).
>

Just checked out the man page, and ran "hdparm -N" on all RAID member
devices. They all report that HPA is disabled.
e.g.
      $ sudo hdparm -N /dev/sdc

/dev/sdc:
    max sectors   = 586072368/586072368, HPA is disabled

This number of sectors does seem like it would fit all the partitions from
the previous array. However, the Size reported by mdadm seems to be
smaller than what hdparm reports, judging from the error messages below.

On Thu, 03 Jan 2013 15:33:55 -0000, Alex Leach <beamesleach@gmail.com>
wrote:

>      $ sudo mdadm --create --verbose /dev/md/RAID5 --raid-devices=3  
> --level=5 --chunk=64 --layout=la --size=293033600  
> --uuid='789c5fd2b57d7def:89f68d3c' -e imsm /dev/sda /dev/sdg /dev/sdb
> mdadm: /dev/sdb not enough space (586066062 < 586067200)
> mdadm: /dev/sdb is smaller than given size. 0K < 293033600K + metadata
> mdadm: /dev/sdc not enough space (586066062 < 586067200)
> mdadm: /dev/sdc is not suitable for this array.
> mdadm: create aborted
>


Cheers,
Alex

Re: RAID5 member disks shrunk

[Benjamin ESTRABAUD]

Hi,

Answers inline.

On 03/01/13 15:33, Alex Leach wrote:
> Dear list,
>
> I hope you all had a good festive period. Happy New Year!
>
> Sorry to burden you with this, but I'm in real need of some help! I 
> recently suffered from a failed hard disk in my RAID5 array, and have 
> made some rookie errors since.. I'll try and detail what I did as 
> accurately and concisely as possible. Some help in recovering data 
> from my ext4 partition would be greatly appreciated!
>
> Set up:-
>   1. RAID0 was pre-configured on new machine, with 2x300GB WD 
> Velociraptors. (2010)
>   2. Got another, identical drive and upgrade to RAID5 using Windows 
> Intel Matrix Manager. (2011)
>
> RAID Partitions (roughly):
>   1. 100MB windows boot loader.
>   2. 342GB NTFS Windows installation partition
>   3. 256GB ext4 partition with Kubuntu 12.10, and dmraid.
>
> Non-RAID Partitions (for completeness):
>   Backup Drive: 2TB.
>      1. 200GB Ubuntu 12.10 ext4 (recovabuntu) with dmraid.
>      2. 1800GB NTFS backup and media partition.
>   SAS drive: 600GB
>      1. 600GB ext4 partition, for SQL and other databases.
>
>
> Problem, started about two weeks ago:
>   1. Suffer an IO error on member disk, whilst in Kubuntu. Array 
> DEGRADED.
>   2. Goes unnoticed for at most a day. Shut down immediately, replace 
> HD with new, identical disk.
>   3. Boot into Windows, use Matrix Storage manager to rebuild array on 
> to new disk. Array now fine.
>   4. At some point after the array was rebuilt, back in Kubuntu, the 
> new disk also raised an IO error. DEGRADED again.
>
> Stop gap:
>   5. Panic, and boot into recovabuntu on non-raid disk. The 2TB drive 
> is not a drive I want to (over-)use when not strictly necessary, as 
> this has all my media on it and a couple of (old) backups.
>   6. WD have been stalling the RMA on the failed drive(s) over 
> Christmas, and I didn't want to stress out my 2TB drive too much.
>   7. Decide to get an SSD and use that as primary boot device.
>   8. Whilst at it, I also bought and installed a 5 bay Icy Box HD 
> backplane, upgrading from a 3 bay version. This was trickier than I 
> thought; I had to completely disassemble the PC and mod the 5.25" bays 
> in the case, with drill, dremel, rivet gun, and some spray paint for 
> good measure :)
>
> Human error:
>   9. Putting it back together, I accidentally connected one of the 
> good RAID member drives to the SAS controller, and the SAS drive into 
> one of the SATA ports, which has an Intel ICH10R controller.
>   10. Took a couple boots into BIOS and recovabuntu to realise what 
> I'd done wrong. Array now wants to rebuild on to good disk, using data 
> from the drive that IO Error'd on me. Don't like the sound of that, so 
> I leave it degraded.
>
> (Botched) Recovery:
>   11. Install Arch Linux and Windows onto separate partitions on the 
> SSD drive.
>   12. Read that Intel now support mdadm over dmraid, so install that 
> on Arch.
>   13. Backup some information:
>
>     $ sudo mdadm -D /dev/md/RAID5
>
> /dev/md/RAID5:
>       Container : /dev/md/imsm0, member 0
>      Raid Level : raid5
>      Array Size : 586066944 (558.92 GiB 600.13 GB)
>   Used Dev Size : 293033600 (279.46 GiB 300.07 GB)
>    Raid Devices : 3
>   Total Devices : 2
>
>           State : active, degraded
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-asymmetric
>      Chunk Size : 64K
>
>
>            UUID : 789c5fd2:da9dd3d2:b57d7def:89f68d3c
>     Number   Major   Minor   RaidDevice State
>        1       8        0        0      active sync   /dev/sda
>        1       0        0        1      removed
>        0       8       16        2      active sync   /dev/sdb
>
>
>     $ sudo mdadm -D /dev/md/imsm0
>
> /dev/md/imsm0:
>         Version : imsm
>      Raid Level : container
>   Total Devices : 3
>
> Working Devices : 3
>
>
>            UUID : 4528e473:abbe0e9f:25a8bb6b:bb9e9999
>   Member Arrays : /dev/md/RAID5
>
>     Number   Major   Minor   RaidDevice
>
>        0       8        0        -        /dev/sda
>        1       8       96        -        /dev/sdg
>        2       8       16        -        /dev/sdb
>
>
>     $ sfdisk -d /dev/sda > sda.out (and same for sdg and sdb)
>     $ ls -l /dev/sd[agb].out
> -rw-r--r-- 1 albl500 users   259 Dec 18 10:45 sda.out
> -rw-r--r-- 1 albl500 users     0 Dec 18 10:47 sdb.out
> -rw-r--r-- 1 albl500 users     0 Dec 18 10:47 sdg.out
>
>     $ cat sda.out
>
> # partition table of /dev/sda
> unit: sectors
>
> /dev/sda1 : start=     2048, size=   204800, Id= 7, bootable
> /dev/sda2 : start=   206848, size=667994112, Id= 7
> /dev/sda3 : start=668200960, size=503928832, Id= 7
> /dev/sda4 : start=        0, size=        0, Id= 0
>
>   14. Figure I should zero the superblocks and re-create the array. 
> Really should have backed up as much as possible before this...
>     $ sudo mdadm --misc --zero-superblock /dev/sd[agb]
>
>     $ sudo mdadm --create /dev/md/imsm0 --raid-devices=2 
> --uuid='4528e473:abbe0e9f:25a8bb6b:bb9e9999' --metadata=imsm /dev/sda 
> /dev/sdg /dev/sdb
> mdadm: container /dev/md/imsm0 prepared.
>
>     $ sudo mdadm --create --verbose /dev/md/RAID5 --raid-devices=3 
> --level=5 --chunk=64 --layout=la --size=293033600 
> --uuid='789c5fd2b57d7def:89f68d3c' -e imsm /dev/sda /dev/sdg /dev/sdb
> mdadm: /dev/sdb not enough space (586066062 < 586067200)
> mdadm: /dev/sdb is smaller than given size. 0K < 293033600K + metadata
> mdadm: /dev/sdc not enough space (586066062 < 586067200)
> mdadm: /dev/sdc is not suitable for this array.
> mdadm: create aborted

Not sure if this will help but I had something similar recently. After 
updating from mdadm-2.6.9 on kernel 2.6.35 to mdadm-3.2.6 on kernel 
3.6.6 I noticed that reusing the same "create" command with the same 
"size" argument did not work anymore, complaining about devices being 
too small.

The command having worked before, I looked deeper into it and realized 
that between the time the array was initially created on mdadm-2.6.9 and 
when it was re-created on mdadm-3.2.6 some changes to mdadm were made 
that reserved more space for superblocks etc.

While I used to simply allocate an extra 256KB of space for the 
superblock when passing in the "size" argument, I changed this to a lot 
higher for any subsequent array.

Considering that your array was built on one system and that you tried 
to run the "create" command on another, this is what could have happened.

In a more positive light, the fact that it did not do anything probably 
saved you from it overwriting the old array by creating it at a so 
slightly different array.

Not sure what the guideline is here in this case, but I would recommend 
only "recreating" arrays to repair them in last resort, and if necessary 
to do, only by using the same version of mdadm/kernel, just to be sure 
there aren't any changes in the code that will cause your array to be 
recreated not in the same exact way.

>
>
>   15. If I leave off the size option, then the array gets built, and 
> verifies all 3 drives without a single IO error. So the drives seem to 
> be okay now. But when they are assembled, the Array Size and Used Dev 
> Size come out smaller than with the previous array, whose details are 
> above. Now, after re-creating and verifying (with no size option), I 
> get these results:
>
>     $ sudo mdadm -D /dev/md/RAID5
>
> /dev/md/RAID5:
>       Container : /dev/md/imsm0, member 0
>      Raid Level : raid5
>      Array Size : 586065920 (558.92 GiB 600.13 GB)
>   Used Dev Size : 293033024 (279.46 GiB 300.07 GB)
>    Raid Devices : 3
>   Total Devices : 3
>
>           State : clean
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-asymmetric
>      Chunk Size : 64K
>
>
>            UUID : 2cdc3f5c:7d91eb7d:51b57c72:bdbfc1fb
>     Number   Major   Minor   RaidDevice State
>        2       8       16        0      active sync   /dev/sda
>        1       8      112        1      active sync   /dev/sdg
>        0       8       32        2      active sync   /dev/sdb
>
>     $ sudo sfdisk -d /dev/md126  # this is: /dev/md/RAID5
>
> # partition table of /dev/md126
> unit: sectors
>
> /dev/md126p1 : start=     2048, size=   204800, Id= 7, bootable
> /dev/md126p2 : start=   206848, size=667994112, Id= 7
> /dev/md126p3 : start=668200960, size=503928832, Id= 7
> /dev/md126p4 : start=        0, size=        0, Id= 0
>
>   16. When I saw these partitions in /dev, I was very happy! I thought 
> I had everything back, but I couldn't mount a single one of the file 
> systems. I used testdisk to try and recover the partitions, and this 
> changed the partition table to point to different start and end 
> positions for p1 and p2, deleting p3. I could mount and recover data 
> from my NTFS windows partition, but the ext4 partition could not be 
> recovered at all; I suspect because of the new Minor numbers and the 
> apparent reduced device and array sizes.
Minors numbers are irrelevant to ext4, a device can be allocated 
different minor numbers and still have its filesystem read (devices do 
not always mount on the same sdX device after all)

My guess is that there is corruption (to an unknown extent) but that 
NTFS is more forgiving about it. Have you made sure that the data you 
recovered on your NTFS partition is still good?

The fact that your ext4 partition "shrunk" (assuming it is the last one) 
should not (to be tested, simply a hunch out of memory) prevent it from 
mounting, at least when trying to force it.

>
>   17. It seems the Used Dev size has reduced by 576kB ( = 512kB + 
> 64kB), which I thought could be partially explained by an MBR being 
> present on the disks, but I have been unable to think where the extra 
> 64kB could be taken up.
>
>   18. So I backup the first 512kB of each member disk, and write zeros 
> to those areas, with dd.
>
>     $ sudo dd if=/dev/sdX of=sdX.out bs=512 count=1
>     $ sudo dd if=/dev/zero of=/dev/sdX bs=512 count=1
>
>   19. Still unable to create an array with the right size (after 
> zeroing superblocks). Updating the super-minor doesn't seem to apply 
> to imsm containers, and I can't think of any other way to get this to 
> re-assemble correctly.
>
> So, I've got myself into a real mess here. I don't have the space to 
> backup every disk image, which I've seen recommended pretty much 
> everywhere.
> Any help getting me out of this mess would be really, really appreciated!
>
> Kind regards,
> Alex
>
Mhhhh, had not seen the part where your array got created in the end. If 
your NTFS partition on the RAID is here, chances are the other should be 
too. Try to look for ext3 file headers by examining the contents of your 
RAID with hexdump at the offset at which your partition is supposed to 
start.

The fact that "create" was ran and that there are no backups of the 
images does not help your case.

HTH.

Regards,
Ben.

Fwd: Re: RAID5 member disks shrunk

[Alex Leach]

(Sorry, keep replying only to sender!).

Hi,

Thanks for the reply. Really helpful :)


On Fri, 04 Jan 2013 16:13:02 -0000, Benjamin ESTRABAUD <be@mpstor.com>
wrote:

>
> Not sure if this will help but I had something similar recently. After  
> updating from mdadm-2.6.9 on kernel 2.6.35 to mdadm-3.2.6 on kernel  
> 3.6.6 I noticed that reusing the same "create" command with the same  
> "size" argument did not work anymore, complaining about devices being  
> too small.

I'm also currently running mdadm-3.2.6, but on kernel 3.7.1.
The Kubuntu ext4 partition (used to be my main OS) was on about kernel
3.5.5, but was running dmraid (don't know off-hand what version, but I
kept everything up to date with the main Ubuntu repositories).

The RAID arrays weren't originally created in Linux. I imagine the initial
RAID0 was created in the BIOS (by the PC assemblers), but they might have
done it in Windows.. I can phone up and ask tomorrow. Don't know off-hand
what version of the Windows utility was installed, but could figure it out
if necessary.

I've tried (and failed) to upgrade the Intel BIOS Option ROM. This is what
is and has always been installed:

       $ mdadm --detail-platform
          Platform : Intel(R) Matrix Storage Manager
           Version : 8.0.0.1038
       RAID Levels : raid0 raid1 raid10 raid5
       Chunk Sizes : 4k 8k 16k 32k 64k 128k
       2TB volumes : supported
         2TB disks : not supported
         Max Disks : 6
       Max Volumes : 2 per array, 4 per controller
    I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)

I've seen it recommended to always stick to whatever tools you used to
create the array in the first place, and don't change between dmraid and
mdadm. I've always used dmraid (until I installed Arch, a couple weeks
ago), but never for creating or manipulating the array. mdadm seems a bit
safer and has more, specific recovery options in this sort of scenario,
and did just fine when accessing the initially degraded array.


I saw this excellent answer / set of tests on serverfault.com, which
furthered my confidence in mdadm :
     http://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using/347786#347786

Having read that and seen all the extra functionality mdadm provides over
dmraid, I'd quite like to stick with mdadm over dmraid.


>
> The command having worked before, I looked deeper into it and realized  
> that between the time the array was initially created on mdadm-2.6.9 and  
> when it was re-created on mdadm-3.2.6 some changes to mdadm were made  
> that reserved more space for superblocks etc.
>
> While I used to simply allocate an extra 256KB of space for the  
> superblock when passing in the "size" argument, I changed this to a lot  
> higher for any subsequent array.

Thanks for the tip. Bit late now, but I'll be sure to keep some extra tail
space in the future!

>
> Considering that your array was built on one system and that you tried  
> to run the "create" command on another, this is what could have happened.

Intel's manual says all data will be lost when you create an array, which
is why I've avoided using their Windows tool for this recovery, even
though they say the same for mdadm. AFAICT, the windows utility doesn't
seem to allow specifying device order, essential if I'm to recover
anything. Perhaps I should have a go in the BIOS, after zero'ing the
superblocks? Any thoughts for, or (more importantly) against that?


>
> In a more positive light, the fact that it did not do anything probably  
> saved you from it overwriting the old array by creating it at a so  
> slightly different array.

Yes, cheers. I'm hoping there wasn't anything essential at the end of the
ext4 partition, but I haven't read (or understood) anything specifying an
essential bit of ext4 data at the end of a partition. I have a suspicion
that there's some sort of marker at the end of an ext4 partition, which
may have been overwritten by an enlarged RAID superblock. What I'm
confused about is how the partition starts seem to have changed location,
higher up the disk, according to testdisk results...

>
> Not sure what the guideline is here in this case, but I would recommend  
> only "recreating" arrays to repair them in last resort, and if necessary  
> to do, only by using the same version of mdadm/kernel, just to be sure  
> there aren't any changes in the code that will cause your array to be  
> recreated not in the same exact way.
>

The same RAID utility would be Intel's windows one.. Would mdadm have been
compatible with whatever Windows version was available at the time? I
could figure out what mdadm version was available at the same time, and
could potentially use that from an older recovabuntu kernel.



> Minors numbers are irrelevant to ext4, a device can be allocated  
> different minor numbers and still have its filesystem read (devices do  
> not always mount on the same sdX device after all)

Cool, thanks for the clarification. Yea, I noticed that; after writing a
partition table with testdisk, the sdX codes had changed. Fortunately, I
managed to match the new letters with the old (and have been posting
consistent drive letters here for simplicity).


>
> My guess is that there is corruption (to an unknown extent) but that  
> NTFS is more forgiving about it. Have you made sure that the data you  
> recovered on your NTFS partition is still good?

Haven't checked it all.. I've played through a couple albums I copied off
the NTFS partition, without a hitch, and the directory structures and file
names of my Documents seem to be fine. I've rescued a 6GB Ubuntu VM from
the NTFS partition; haven't tried running it since, but I think that'd be
a good test for integrity...

>
> The fact that your ext4 partition "shrunk" (assuming it is the last one)  
> should not (to be tested, simply a hunch out of memory) prevent it from  
> mounting, at least when trying to force it.

That would be cool. Do you know, if I was to use `mke2fs` to update the
partition table to fit on the drive and not to overlap the previous, NTFS
partition, is there a possibility the old file system and journal could be
picked up? Would it work even if the partition table said it extended
beyond the available size of the array? With the old partition table, I
was unable to mount any of the partitions... I guess a valid partition
table, even if inconsistent with the old one, would allow me to then use
extundelete, which I've been unable to do without a partition / device
file.


>>
> Mhhhh, had not seen the part where your array got created in the end. If  
> your NTFS partition on the RAID is here, chances are the other should be  
> too. Try to look for ext3 file headers by examining the contents of your  
> RAID with hexdump at the offset at which your partition is supposed to  
> start.

I could give that a go, but don't have a clue what to expect ext3 file
headers to look like. Haven't used hexdump before, but I'll see if I can
figure out what to do from the man page.

>
> The fact that "create" was ran and that there are no backups of the  
> images does not help your case.

Yep, I know :( Tragic circumstances...



Feel like that was a bit all over the place. So gonna detail my plan of
action.. Any comments extremely welcome.

    Current working status
    ======================
     NB I did another re-create yesterday, and minor numbers changed back,
but size still small :-/

       $ sudo mdadm -D /dev/md/RAID5

/dev/md/RAID5:
         Container : /dev/md/imsm0, member 0
        Raid Level : raid5
        Array Size : 586065920 (558.92 GiB 600.13 GB)
     Used Dev Size : 293033024 (279.46 GiB 300.07 GB)
      Raid Devices : 3
     Total Devices : 3

             State : clean
    Active Devices : 3
Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-asymmetric
        Chunk Size : 64K


              UUID : 1a11fb7c:d558fd23:c0481cee:268cb5e0
       Number   Major   Minor   RaidDevice State
          0       8        0        0      active sync   /dev/sda
          3       8       96        1      active sync   /dev/sdg
          1       8       16        2      active sync   /dev/sdb


       $ sudo sfdisk -d /dev/md/RAID5
       $ sudo testdisk /dev/md/RAID5
# Intel Partition -> Analyse -> Quick Search
Disk /dev/md/RAID5 - 600 GB / 558 GiB - CHS 146516480 2 4
Analysing cylinder ...... (57%)
Warning: Incorrect number of heads/cylinder 255 (NTFS) != 2 (HD)
Warning: Incorrect number of sectors per track 63 (NTFS) != 4 (HD)
     HPFS - NTFS            266   0  1 25865   1  4     204800
Warning: Incorrect number of heads/cylinder 255 (NTFS) != 2 (HD)
Warning: Incorrect number of sectors per track 63 (NTFS) != 4 (HD)
     HPFS - NTFS          26816   0  1 83526079   1  4  667994112
     Linux                83526080   0  1 146517183   1  4  503928832

# Stop (at ~57%)
The following partition can't be recovered:
        Partition               Start        End    Size in sectors
>  Linux                83526080   0  1 146517183   1  4  503928832

# Continue
Disk /dev/md/RAID5 - 600 GB / 558 GiB - CHS 146516480 2 4
        Partition               Start        End    Size in sectors
> * HPFS - NTFS            266   0  1 25865   1  4     204800
    P HPFS - NTFS          26816   0  1 83526079   1  4  667994112

Structure: Ok.  Use Up/Down Arrow keys to select partition.
Use Left/Right Arrow keys to CHANGE partition characteristics:
*=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted

# Continue -> Quit.


How is the start sector now 266? Is it something to do with the CHS
values? testdisk reports partition boundaries that don't seem to be
aligned with the cylinder boundaries, but if I change the Sectors per head
to 63, and heads per cylinder to 255, in testdisk options, it then doesn't
find these partitions, which are almost definitely my old partitions that
I want to recover...


     2. Make a copy of old partition table file; the one I originally dumped
   from sfdisk, here for convenience:

>>     $ sudo sfdisk -d /dev/md126  # this is: /dev/md/RAID5
>>
>> # partition table of /dev/md126
>> unit: sectors
>>
>> /dev/md126p1 : start=     2048, size=   204800, Id= 7, bootable
>> /dev/md126p2 : start=   206848, size=667994112, Id= 7
>> /dev/md126p3 : start=668200960, size=503928832, Id= 7
>> /dev/md126p4 : start=        0, size=        0, Id= 0
>>

     3. Copy new start and end numbers reported by testdisk into above copy
of sfdisk output. Figure out start and end sectors for ext4 partition.
These will be for the first sector after NTFS partition, and last sector
before end of array. Copy that into sfdisk output. I'm quite confused by
the numbers testdisk reports, as the start and end numbers reported
wouldn't fit the 'size in sectors'..


     4. $ sudo sfdisk /dev/md/RAID5 < md126.out

     5. Hope udev picks up the partition table change, reboot if nothing
happens, and hope for the best.

     6. See what extundelete can find...

     7. Backup! Backup! Backup!

Thanks again for the help and advice.

Cheers,
Alex


-- 
Using Opera's mail client: http://www.opera.com/mail/

Re: RAID5 member disks shrunk

[Alex Leach]

On Fri, 04 Jan 2013 16:13:02 -0000, Benjamin ESTRABAUD wrote:

> The fact that your ext4 partition "shrunk" (assuming it is the last one)  
> should not (to be tested, simply a hunch out of memory) prevent it from  
> mounting, at least when trying to force it.

How do you force a mount? Just looked through `man mount` and didn't see
anything fitting the description.



I figured some stuff out. Thought I'd detail and plan it here, before I go
ahead and do anything rash (like I haven't already)..

On Fri, 04 Jan 2013 19:24:09 -0000, Alex Leach wrote:

>
>   Current working status
>   ======================

>      $ sudo testdisk /dev/md/RAID5
> # Intel Partition -> Analyse -> Quick Search
> Disk /dev/md/RAID5 - 600 GB / 558 GiB - CHS 146516480 2 4
> Analysing cylinder ...... (57%)
> Warning: Incorrect number of heads/cylinder 255 (NTFS) != 2 (HD)
> Warning: Incorrect number of sectors per track 63 (NTFS) != 4 (HD)
>    HPFS - NTFS            266   0  1 25865   1  4     204800
> Warning: Incorrect number of heads/cylinder 255 (NTFS) != 2 (HD)
> Warning: Incorrect number of sectors per track 63 (NTFS) != 4 (HD)
>    HPFS - NTFS          26816   0  1 83526079   1  4  667994112
>    Linux                83526080   0  1 146517183   1  4  503928832
>
> # Stop (at ~57%)
> The following partition can't be recovered:
>       Partition               Start        End    Size in sectors
>>  Linux                83526080   0  1 146517183   1  4  503928832
>
> # Continue
> Disk /dev/md/RAID5 - 600 GB / 558 GiB - CHS 146516480 2 4
>       Partition               Start        End    Size in sectors
>> * HPFS - NTFS            266   0  1 25865   1  4     204800
>   P HPFS - NTFS          26816   0  1 83526079   1  4  667994112
>

> How is the start sector now 266? Is it something to do with the CHS  
> values? testdisk reports partition boundaries that don't seem to be  
> aligned with the cylinder boundaries, but if I change the Sectors per  
> head to 63, and heads per cylinder to 255, in testdisk options, it then  
> doesn't find these partitions, which are almost definitely my old  
> partitions that I want to recover...


So the testdisk units must be cylinder numbers. Multiplying them by 8 ( =
2 x 4 <-- from CHS values) gives the start sector number. This is the
partition table that testdisk creates:


# partition table of /dev/md/RAID5
unit: sectors

/dev/md/RAID5p1 : start=     2128, size=   204800, Id= 7, bootable
/dev/md/RAID5p2 : start=   214528, size=667994112, Id= 7
/dev/md/RAID5p3 : start=        0, size=        0, Id= 0
/dev/md/RAID5p4 : start=        0, size=        0, Id= 0

NB. There is now a noticeable gap between p1 and p2, of 7,600 sectors. p2
appears to be fine, but p1 doesn't mount:

     $ sudo mount -t ntfs-3g /dev/md/RAID5p1 /mnt
ntfs_mst_post_read_fixup_warn: magic: 0x44524352  size: 1024   usa_ofs:
40  usa_count: 8: Invalid argument
Record 0 has no FILE magic (0x44524352)
Failed to load $MFT: Input/output error
Failed to mount '/dev/md126p1': Input/output error
...


>
>    3. Copy new start and end numbers reported by testdisk into above  
> copy of sfdisk output. Figure out start and end sectors for ext4  
> partition. These will be for the first sector after NTFS partition, and  
> last sector before end of array. Copy that into sfdisk output. I'm quite  
> confused by the numbers testdisk reports, as the start and end numbers  
> reported wouldn't fit the 'size in sectors'..


I'm wary of reducing the size of the partition, as presumably the size of
the inode table would differ and the start location of the data block
would be out.

https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Layout

So I loaded in the following partition table, wherein the final partition
extends beyond the end of the array, but it is of the correct size.

     $ sudo sfdisk -f /dev/md/RAID5  -O md126.testdisk.saved <
md126.new
...
Warning: given size (503928824) exceeds max allowable size (503923200)
New situation:
Units: sectors of 512 bytes, counting from 0

                    Device Boot    Start       End   #sectors  Id System
/dev/md/RAID5p1   *      9728    214527     204800   7  HPFS/NTFS/exFAT
/dev/md/RAID5p2        214528 668208639  667994112   7  HPFS/NTFS/exFAT
/dev/md/RAID5p3     668208640 1172137463  503928824  83  Linux
/dev/md/RAID5p4             0         -          0   0  Empty
Warning: partition 3 extends past end of disk
Successfully wrote the new partition table
...
     $ partprobe /dev/md/RAID5
Error: Can't have a partition outside the disk!
     $ ls /dev/md/RAID5*
/dev/md/RAID5 /dev/md/RAID5p1 /dev/md/RAID5p2


Is there a way to force udev to create the p3 /dev entry?

I think my only option is to zero the superblock with mdadm and try to  
recreate the array in Windows, with
whatever version of Intel Matrix Storage Manager was initially installed
on the machine, hoping to God that the array contents don't get
overwritten. Then, hopefully the original array size would be available
again and the ext4 partition would fit within it. Sounds dangerous...

Any alternative solutions?

Kind regards,
Alex

Re: RAID5 member disks shrunk

[Alex Leach]

Dear all,

Sorry about the break; I've been away and doing other things...


On Sun, 06 Jan 2013 12:59:07 -0000, Alex Leach <beamesleach@gmail.com>  
wrote:

> I think my only option is to zero the superblock with mdadm and try to  
> recreate the array in Windows, with
> whatever version of Intel Matrix Storage Manager was initially installed
> on the machine, hoping to God that the array contents don't get
> overwritten. Then, hopefully the original array size would be available
> again and the ext4 partition would fit within it. Sounds dangerous...


So, that is what I did. Specifically:-

0. Used serial numbers from /dev/disk/by-id to figure out the original  
order that the member disks were plugged into the motherboard. Swapped  
/dev/sdg and /dev/sdb, so that they were plugged into the motherboard  
ports in the same order as when I first created the array.

1. mdadm --zero-superblock /dev/sd[abg]

2. Re-create RAID5 array in Intel Matrix Storage Manager 8.9.

    This just initialised the container and member array i.e. wrote the  
imsm container superblock. The re-sync was pending.

3. Reboot into Arch linux.

    mdadm started re-sync'ing the array. I let it finish...

4. mdadm -D /dev/md/RAID5 | grep Size

      Array Size : 586066944 (558.92 GiB 600.13 GB)
   Used Dev Size : 293033600 (279.46 GiB 300.07 GB)
      Chunk Size : 64K

Assuming the above units are in Kibibytes, I figure multiplying the Array  
Size by 2 should give the usable number of sectors: 1,172,133,888.

The array size and used dev size is now as it was before everything went  
tits up. That's good, but testdisk still finds that the partitions have  
moved up the disk. The discovered ext4 partition now extends 3,576 sectors  
beyond the end of the array.


In sectors, these are the partitions testdisk finds:

Partition     Start            End  (Diff.)          Size
1             2,128        206,920     +73        204,800
2           214,528    668,208,632  +7,673    667,994,112
3       668,208,640  1,172,137,464  +7,680    503,928,832

where Diff is the number of sectors the partition seems to have moved, cf.  
the original partition table.

---------------------


Partition 2 is still recoverable. I can browse the array contents using  
testdisk and can see that the Windows directory structure is still there.  
Partition 1 seems corrupted. I can't browse the contents in testdisk and  
was unable to mount the partition, even before this last array re-creation.

Partition 3 is still a problem. testdisk refuses to recover it and I can't  
browse its contents.

I seem to have a new problem, too. I am now unable to write a partition  
table to the disk! I've tried using sfdisk, parted and testdisk. Each of  
these programs hangs indefinitely and the process is invincible to kill  
commands. The machine needs to be hard-rebooted in order to kill the  
hanging process.

Two minutes after trying to write a partition table, dmesg reports the  
following traceback each and every 2 minutes:

[  479.779519] INFO: task sfdisk:1020 blocked for more than 120 seconds.
[  479.779522] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables  
this message.
[  479.779525] sfdisk          D 0000000000000001     0  1020   1019  
0x00000000
[  479.779529]  ffff8805e9cb3978 0000000000000086 ffff8805e9cb3801  
ffff8805e9cb3fd8
[  479.779534]  ffff8805e9cb3948 ffff8805e9cb3fd8 ffff8805e9cb3fd8  
ffff8805e9cb3fd8
[  479.779538]  ffff88061e0c5040 ffff8805e9e76450 ffff8805e9cb3908  
ffff8805edd86200
[  479.779542] Call Trace:
[  479.779551]  [<ffffffff81164024>] ?  
__mem_cgroup_commit_charge+0xd4/0x350
[  479.779558]  [<ffffffff8107b79b>] ? prepare_to_wait+0x5b/0x90
[  479.779569]  [<ffffffffa03efc55>] md_write_start+0xb5/0x1a0 [md_mod]
[  479.779573]  [<ffffffff8107b9a0>] ? abort_exclusive_wait+0xb0/0xb0
[  479.779577]  [<ffffffffa02bad0b>] make_request+0x3b/0x6c0 [raid456]
[  479.779582]  [<ffffffff81067e68>] ? lock_timer_base.isra.38+0x38/0x70
[  479.779585]  [<ffffffff81067400>] ? internal_add_timer+0x20/0x50
[  479.779591]  [<ffffffffa03eac6c>] md_make_request+0xfc/0x240 [md_mod]
[  479.779595]  [<ffffffff81113075>] ? mempool_alloc_slab+0x15/0x20
[  479.779601]  [<ffffffff8122a9c2>] generic_make_request+0xc2/0x110
[  479.779604]  [<ffffffff8122aa89>] submit_bio+0x79/0x160
[  479.779609]  [<ffffffff811a2c15>] ? bio_alloc_bioset+0x65/0x120
[  479.779612]  [<ffffffff8119d1e5>] submit_bh+0x125/0x210
[  479.779616]  [<ffffffff811a0210>] __block_write_full_page+0x1f0/0x360
[  479.779620]  [<ffffffff8119e0b0>] ? end_buffer_async_read+0x200/0x200
[  479.779623]  [<ffffffff811a3df0>] ? I_BDEV+0x10/0x10
[  479.779627]  [<ffffffff811a3df0>] ? I_BDEV+0x10/0x10
[  479.779630]  [<ffffffff8119e0b0>] ? end_buffer_async_read+0x200/0x200
[  479.779633]  [<ffffffff811a0466>] block_write_full_page_endio+0xe6/0x130
[  479.779637]  [<ffffffff811a04c5>] block_write_full_page+0x15/0x20
[  479.779641]  [<ffffffff811a4478>] blkdev_writepage+0x18/0x20
[  479.779644]  [<ffffffff81119ac7>] __writepage+0x17/0x50
[  479.779647]  [<ffffffff81119f92>] write_cache_pages+0x1f2/0x4e0
[  479.779650]  [<ffffffff81119ab0>] ? global_dirtyable_memory+0x40/0x40
[  479.779655]  [<ffffffff8116c047>] ? do_sync_write+0xa7/0xe0
[  479.779658]  [<ffffffff8111a2ca>] generic_writepages+0x4a/0x70
[  479.779662]  [<ffffffff8111bace>] do_writepages+0x1e/0x40
[  479.779666]  [<ffffffff81111a49>] __filemap_fdatawrite_range+0x59/0x60
[  479.779669]  [<ffffffff81111b50>] filemap_write_and_wait_range+0x50/0x70
[  479.779673]  [<ffffffff811a4744>] blkdev_fsync+0x24/0x50
[  479.779676]  [<ffffffff8119b6dd>] do_fsync+0x5d/0x90
[  479.779680]  [<ffffffff8116ca62>] ? sys_write+0x52/0xa0
[  479.779683]  [<ffffffff8119ba90>] sys_fsync+0x10/0x20
[  479.779688]  [<ffffffff81495edd>] system_call_fastpath+0x1a/0x1f

Apologies for not having all the debug symbols installed. I've been unable  
to locate the necessary package/s on Arch that contain them.

This is with mdadm-3.2.6. I think I'll try again with an old version of  
dmraid on recovabuntu, one released around the time I first created the  
ext4 partition, which was in Ubuntu.

As to why I get this traceback, no idea.

---------------------------

I'd really appreciate some suggestions on how to recover the ext4  
partition. My current idea is this:

1. Use ddrescue to copy the hard disk from sector 668,208,640 (location  
testdisk found) up to the end.

     $ ddrescue obs=512 seek=668208640 bs=64 if=/dev/md/RAID5  
of=/media/bigdisk/kubuntu.ext4

2. Probably get a new HDD, create an MBR in Windows 7, and make a new  
partition exactly the same size as before.

     $ [cs]?fdisk ...

Not exactly sure what information I'd need to specify here, other than  
size, primary and partition Id (83). Use whichever program allows all the  
necessary options.
Assume, just created partition /dev/sdXx.

3. Copy the backed up image on to the raw device / partition file.

     $ dd bs=64 if=/media/bigdisk/kubuntu.ext4 of=/dev/sdXx

4. Hope fsck works...

     $ fsck.ext4 -f -p /dev/sdXx

5. Hopefully mount the partition and browse data.

6. Re-create and re-format partitions on the RAID array and copy stuff  
back.

7. Get an incremental back-up system running.

Does anyone know if there is even a remote possibility this could work?  
I've read through some of the kernel wiki on ext4 partitions  
(https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout), but doubt my  
ability to use any of that information effectively.

[Question - any ext4 gurus here who would know?]

Could I increase the size of the output file of step 1, so that it appears  
to be the same size as the original partition? i.e. If I were to pad it  
with zeros, could I attempt to mount the file as an ext4 partition?  
Something like:

    $ dd if=/dev/zeros bs=512 count=3576 >> /media/bigdisk/kubuntu.ext4
    $ e2fsck /media/bigdisk/kubuntu.ext4
    $ mount -t ext4 /media/bigdisk/kubuntu.ext4 /mnt



Any comments, suggestions or insults completely welcome.

Kind regards,
Alex

P.S. Sorry that I've been unable to report any hexdump results. Basically,  
I don't (yet) have a clue how I'd locate the ext3 file headers, or what to  
expect them to look like. Over the course of this week, I'll try and make  
my way through this ext3 recovery tutorial:  
http://carlo17.home.xs4all.nl/howto/undelete_ext3.html

-- 
Using Opera's mail client: http://www.opera.com/mail/