* Re: [PATCH] IMSM: do not use comma expression for simple assignments
From: Artur Paszkiewicz @ 2014-12-11 9:06 UTC (permalink / raw)
To: Jürg Billeter, NeilBrown; +Cc: linux-raid, Baldysiak, Pawel
In-Reply-To: <1407313179-24598-1-git-send-email-j@bitron.ch>
On 08/06/2014 10:19 AM, Jürg Billeter wrote:
> Fixes build with gcc 4.9.
>
> error: right-hand operand of comma expression has no effect
> snprintf((char *) spare->sig, MAX_SIGNATURE_LENGTH,
> ^
>
> Signed-off-by: Jürg Billeter <j@bitron.ch>
> ---
> super-intel.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/super-intel.c b/super-intel.c
> index e28ac7d..f24d101 100644
> --- a/super-intel.c
> +++ b/super-intel.c
> @@ -5069,13 +5069,13 @@ static int write_super_imsm_spares(struct intel_super *super, int doclose)
> __u32 sum;
> struct dl *d;
>
> - spare->mpb_size = __cpu_to_le32(sizeof(struct imsm_super)),
> - spare->generation_num = __cpu_to_le32(1UL),
> + spare->mpb_size = __cpu_to_le32(sizeof(struct imsm_super));
> + spare->generation_num = __cpu_to_le32(1UL);
> spare->attributes = MPB_ATTRIB_CHECKSUM_VERIFY;
> - spare->num_disks = 1,
> - spare->num_raid_devs = 0,
> - spare->cache_size = mpb->cache_size,
> - spare->pwr_cycle_count = __cpu_to_le32(1),
> + spare->num_disks = 1;
> + spare->num_raid_devs = 0;
> + spare->cache_size = mpb->cache_size;
> + spare->pwr_cycle_count = __cpu_to_le32(1);
>
> snprintf((char *) spare->sig, MAX_SIGNATURE_LENGTH,
> MPB_SIGNATURE MPB_VERSION_RAID0);
>
Hi Neil,
We verified that this patch indeed is needed to build mdadm with gcc
4.9. Can you apply it?
Thanks,
Artur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-09 12:00 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141209090114.GA30324@cthulhu.home.robinhill.me.uk>
You're right! I just changed it to sdd3 sdb3 sdc3 missing and fsck -n
/dev/md0 detected everything said it was clean.
Thanks a lot. I will backup my important files and write back a quick
summary of what we did to fix this situation.
On Tue, Dec 9, 2014 at 4:01 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Tue Dec 09, 2014 at 12:35:14AM -0500, Emery Guevremont wrote:
>> >> >> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> >> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
>> >> >> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>> >> >> >> >> >
>> >> >> >> >> >> The long story and what I've done.
>> >> >> >> >> >>
>> >> >> >> >> >> /dev/md0 is assembled with 4 drives
>> >> >> >> >> >> /dev/sda3
>> >> >> >> >> >> /dev/sdb3
>> >> >> >> >> >> /dev/sdc3
>> >> >> >> >> >> /dev/sdd3
>> >> >> >> >> >>
>> >> >> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> >> >> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> >> >> >> >> >> the server and until I received a replacement drive.
>> >> >> >> >> >>
>> >> >> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
>> >> >> >> >> >> single user mode and did this:
>> >> >> >> >> >>
>> >> >> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> >> >> >> >> >> confirmed the resyncing process. The last time I checked it was up to
>> >> >> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> >> >> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
>> >> >> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> >> >> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> >> >> >> >> >> everything as is and to go to bed.
>> >> >> >> >> >>
>> >> >> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
>> >> >> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
>> >> >> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> >> >> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> >> >> >> >> >> looks of this.
>> >> >> >> >> >>
>> >> >> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> >> >> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> >> >> >> >> >> was eventually able to read the bad sector on a retry. I followed up
>> >> >> >> >> >> by also cloning with ddrescue, sdb and sdc.
>> >> >> >> >> >>
>> >> >> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
>> >> >> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
>> >> >> >> >> >> all drives are added as spares. Running mdadm --examine on each
>> >> >> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
>> >> >> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
>> >> >> >> >> >> makes sense. I believe I could re-assemble my array if I could define
>> >> >> >> >> >> the raid level and raid devices.
>> >> >> >> >> >>
>> >> >> >> >> >> I wanted to know if there are a way to restore my superblocks from the
>> >> >> >> >> >> examine command I ran at the beginning? If not, what mdadm create
>> >> >> >> >> >> command should I run? Also please let me know if drive ordering is
>> >> >> >> >> >> important, and how I can determine this with the examine output I'll
>> >> >> >> >> >> got?
>> >> >> >> >> >>
>> >> >> >> >> >> Thank you.
>> >> >> >> >> >>
>> >> >> >> >> You'll see from the examine output, raid level and devices aren't
>> >> >> >> >> defined and notice the role of each drives. The examine output (I
>> >> >> >> >> attached 4 files) that I took right after the read error during the
>> >> >> >> >> synching process seems to show a more accurate superblock. Here's also
>> >> >> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
>> >> >> >> >> error:
>> >> >> >> >>
>> >> >> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> name=runts:0
>> >> >> >> >> spares=1
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Here's the output of how things currently are:
>> >> >> >> >>
>> >> >> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >> >> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
>> >> >> >> >> start the array.
>> >> >> >> >>
>> >> >> >> >> dmesg
>> >> >> >> >> [27903.423895] md: md127 stopped.
>> >> >> >> >> [27903.434327] md: bind<sdc3>
>> >> >> >> >> [27903.434767] md: bind<sdd3>
>> >> >> >> >> [27903.434963] md: bind<sdb3>
>> >> >> >> >>
>> >> >> >> >> cat /proc/mdstat
>> >> >> >> >> root@ubuntu:~# cat /proc/mdstat
>> >> >> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
>> >> >> >> >> [raid1] [raid10]
>> >> >> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
>> >> >> >> >> 5858387208 blocks super 1.2
>> >> >> >> >>
>> >> >> >> >> mdadm --examine /dev/sd[bcd]3
>> >> >> >> >> /dev/sdb3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0
>> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> >> Raid Level : -unknown-
>> >> >> >> >> Raid Devices : 0
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : active
>> >> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >> >>
>> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> >> Checksum : 5e8cfc9a - correct
>> >> >> >> >> Events : 1
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Device Role : spare
>> >> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >> >> /dev/sdc3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0
>> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> >> Raid Level : -unknown-
>> >> >> >> >> Raid Devices : 0
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : active
>> >> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> >> >>
>> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> >> Checksum : f69518c - correct
>> >> >> >> >> Events : 1
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Device Role : spare
>> >> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >> >> /dev/sdd3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0
>> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> >> Raid Level : -unknown-
>> >> >> >> >> Raid Devices : 0
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : active
>> >> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> >> >>
>> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> >> Checksum : 571ad2bd - correct
>> >> >> >> >> Events : 1
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Device Role : spare
>> >> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >> >>
>> >> >> >> >> and finally kernel and mdadm versions:
>> >> >> >> >>
>> >> >> >> >> uname -a
>> >> >> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
>> >> >> >> >> 2012 i686 i686 i386 GNU/Linux
>> >> >> >> >>
>> >> >> >> >> mdadm -V
>> >> >> >> >> mdadm - v3.2.3 - 23rd December 2011
>> >> >> >> >
>> >> >> >> >> /dev/sda3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> >> Raid Level : raid5
>> >> >> >> >> Raid Devices : 4
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : clean
>> >> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >> >>
>> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> >> Checksum : 5ed5b898 - correct
>> >> >> >> >> Events : 3925676
>> >> >> >> >>
>> >> >> >> >> Layout : left-symmetric
>> >> >> >> >> Chunk Size : 512K
>> >> >> >> >>
>> >> >> >> >> Device Role : spare
>> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >> >
>> >> >> >> >> /dev/sdb3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> >> Raid Level : raid5
>> >> >> >> >> Raid Devices : 4
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : clean
>> >> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> >> >>
>> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> >> Checksum : 57638ebb - correct
>> >> >> >> >> Events : 3925676
>> >> >> >> >>
>> >> >> >> >> Layout : left-symmetric
>> >> >> >> >> Chunk Size : 512K
>> >> >> >> >>
>> >> >> >> >> Device Role : Active device 0
>> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >> >
>> >> >> >> >> /dev/sdc3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> >> Raid Level : raid5
>> >> >> >> >> Raid Devices : 4
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : clean
>> >> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> >> >>
>> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> >> Checksum : fb20d8a - correct
>> >> >> >> >> Events : 3925676
>> >> >> >> >>
>> >> >> >> >> Layout : left-symmetric
>> >> >> >> >> Chunk Size : 512K
>> >> >> >> >>
>> >> >> >> >> Device Role : Active device 2
>> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >> >
>> >> >> >> >> /dev/sdd3:
>> >> >> >> >> Magic : a92b4efc
>> >> >> >> >> Version : 1.2
>> >> >> >> >> Feature Map : 0x0
>> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> >> Raid Level : raid5
>> >> >> >> >> Raid Devices : 4
>> >> >> >> >>
>> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> >> Data Offset : 2048 sectors
>> >> >> >> >> Super Offset : 8 sectors
>> >> >> >> >> State : clean
>> >> >> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>> >> >> >> >>
>> >> >> >> >> Update Time : Tue Dec 2 23:14:03 2014
>> >> >> >> >> Checksum : a126853f - correct
>> >> >> >> >> Events : 3925672
>> >> >> >> >>
>> >> >> >> >> Layout : left-symmetric
>> >> >> >> >> Chunk Size : 512K
>> >> >> >> >>
>> >> >> >> >> Device Role : Active device 1
>> >> >> >> >> Array State : AAAA ('A' == active, '.' == missing)
>> >> >> >> >
>> >> >> >> > At least you have the previous data anyway, which should allow
>> >> >> >> > reconstruction of the array. The device names have changed between your
>> >> >> >> > two reports though, so I'd advise double-checking which is which before
>> >> >> >> > proceeding.
>> >> >> >> >
>> >> >> >> > The reports indicate that the original array order (based on the device
>> >> >> >> > role field) for the four devices was (using device UUIDs as they're
>> >> >> >> > consistent):
>> >> >> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> >> > 4156ab46:bd42c10d:8565d5af:74856641
>> >> >> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> >> > b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >> >
>> >> >> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
>> >> >> >> > have the current data for sda3, but that's the only missing UUID).
>> >> >> >> >
>> I had forgotten that I took a pic of the read error message, which
>> also contained an output of /proc/mdstat, so I was able to determine
>> the ordering and I ran this command:
>>
> What did that indicate, and how did you map it to the device order below?
>
>> root@ubuntu:~# mdadm -v --create --assume-clean --level=5 --chunk=512
>> --size=1952795136 --raid-devices=4 /dev/md0 /dev/sdd3 /dev/sdb3
>> missing /dev/sdc3
>> mdadm: layout defaults to left-symmetric
>> mdadm: layout defaults to left-symmetric
>> mdadm: /dev/sdd3 appears to be part of a raid array:
>> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
>> mdadm: layout defaults to left-symmetric
>> mdadm: /dev/sdb3 appears to be part of a raid array:
>> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
>> mdadm: layout defaults to left-symmetric
>> mdadm: /dev/sdc3 appears to be part of a raid array:
>> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
>> Continue creating array? y
>> mdadm: Defaulting to version 1.2 metadata
>> mdadm: array /dev/md0 started.
>>
>> I did mdadm -E and everything seemed to be consistent with the
>> original output of the examine command. So I ran fsck -n
>>
>> root@ubuntu:~# fsck -n /dev/md0
>> fsck from util-linux 2.20.1
>> e2fsck 1.42 (29-Nov-2011)
>> fsck.ext4: Group descriptors look bad... trying backup blocks...
>> Error writing block 1 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 2 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 3 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 4 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 5 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 6 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>> ...
>> ...
>> Error writing block 343 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> Error writing block 344 (Attempt to write block to filesystem resulted
>> in short write). Ignore error? no
>>
>> fsck.ext4: Device or resource busy while trying to open /dev/md0
>> Filesystem mounted or opened exclusively by another program?
>>
>>
>> I believe I made some progress. But before I continue, I wanted to
>> know if I was on the right track?
>>
>> I tried to mount /dev/md0 but got this:
>>
>> root@ubuntu:~# mount -t ext4 /dev/md0 /mnt/
>> mount: wrong fs type, bad option, bad superblock on /dev/md0,
>> missing codepage or helper program, or other error
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so
>>
>> Am I at a point to run fsck to repair the ext4 superblock?
>>
> No, that output would definitely suggest you have the wrong order.
> That looks to be far too many errors for a normal unclean shutdown
> situation.
>
>> I also tried a different ordering to see what fsck -n would give and I got:
>>
>> root@ubuntu:~# fsck -n /dev/md0
>> fsck from util-linux 2.20.1
>> e2fsck 1.42 (29-Nov-2011)
>> fsck.ext4: Filesystem revision too high while trying to open /dev/md0
>> The filesystem revision is apparently too high for this version of e2fsck.
>> (Or the filesystem superblock is corrupt)
>>
>>
>> The superblock could not be read or does not describe a correct ext2
>> filesystem. If the device is valid and it really contains an ext2
>> filesystem (and not swap or ufs or something else), then the superblock
>> is corrupt, and you might try running e2fsck with an alternate superblock:
>> e2fsck -b 8193 <device>
>>
>> Which seems to confirm my first attempt at the ordering was good.
>>
> No, it confirms that the first device was correct - the filesystem
> superblock will be entirely within the first chunk, so only the first
> disk needs to be correct for that to be readable.
>
> Have you tried running it in the order I advised (sdd3, sda3, sdc3,
> missing) or in the order of the UUIDs (if the device order has changed)?
> 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> 4156ab46:bd42c10d:8565d5af:74856641
> 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> b2bf0462:e0722254:0e233a72:aa5df4da
>
> If not, please do so first and see whether the fsck output is any
> better.
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-09 9:01 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sY5eFjGOFBD+Kyrc-uoGUbNQdMz6rG1F4rnLY2TEwD8VQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 18574 bytes --]
On Tue Dec 09, 2014 at 12:35:14AM -0500, Emery Guevremont wrote:
> >> >> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> >> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
> >> >> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> >> >> >> >> >
> >> >> >> >> >> The long story and what I've done.
> >> >> >> >> >>
> >> >> >> >> >> /dev/md0 is assembled with 4 drives
> >> >> >> >> >> /dev/sda3
> >> >> >> >> >> /dev/sdb3
> >> >> >> >> >> /dev/sdc3
> >> >> >> >> >> /dev/sdd3
> >> >> >> >> >>
> >> >> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> >> >> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> >> >> >> >> >> the server and until I received a replacement drive.
> >> >> >> >> >>
> >> >> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
> >> >> >> >> >> single user mode and did this:
> >> >> >> >> >>
> >> >> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> >> >> >> >> >> confirmed the resyncing process. The last time I checked it was up to
> >> >> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
> >> >> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
> >> >> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
> >> >> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> >> >> >> >> >> everything as is and to go to bed.
> >> >> >> >> >>
> >> >> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
> >> >> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
> >> >> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> >> >> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> >> >> >> >> >> looks of this.
> >> >> >> >> >>
> >> >> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> >> >> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> >> >> >> >> >> was eventually able to read the bad sector on a retry. I followed up
> >> >> >> >> >> by also cloning with ddrescue, sdb and sdc.
> >> >> >> >> >>
> >> >> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
> >> >> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
> >> >> >> >> >> all drives are added as spares. Running mdadm --examine on each
> >> >> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
> >> >> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
> >> >> >> >> >> makes sense. I believe I could re-assemble my array if I could define
> >> >> >> >> >> the raid level and raid devices.
> >> >> >> >> >>
> >> >> >> >> >> I wanted to know if there are a way to restore my superblocks from the
> >> >> >> >> >> examine command I ran at the beginning? If not, what mdadm create
> >> >> >> >> >> command should I run? Also please let me know if drive ordering is
> >> >> >> >> >> important, and how I can determine this with the examine output I'll
> >> >> >> >> >> got?
> >> >> >> >> >>
> >> >> >> >> >> Thank you.
> >> >> >> >> >>
> >> >> >> >> You'll see from the examine output, raid level and devices aren't
> >> >> >> >> defined and notice the role of each drives. The examine output (I
> >> >> >> >> attached 4 files) that I took right after the read error during the
> >> >> >> >> synching process seems to show a more accurate superblock. Here's also
> >> >> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
> >> >> >> >> error:
> >> >> >> >>
> >> >> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> name=runts:0
> >> >> >> >> spares=1
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Here's the output of how things currently are:
> >> >> >> >>
> >> >> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
> >> >> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
> >> >> >> >> start the array.
> >> >> >> >>
> >> >> >> >> dmesg
> >> >> >> >> [27903.423895] md: md127 stopped.
> >> >> >> >> [27903.434327] md: bind<sdc3>
> >> >> >> >> [27903.434767] md: bind<sdd3>
> >> >> >> >> [27903.434963] md: bind<sdb3>
> >> >> >> >>
> >> >> >> >> cat /proc/mdstat
> >> >> >> >> root@ubuntu:~# cat /proc/mdstat
> >> >> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
> >> >> >> >> [raid1] [raid10]
> >> >> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
> >> >> >> >> 5858387208 blocks super 1.2
> >> >> >> >>
> >> >> >> >> mdadm --examine /dev/sd[bcd]3
> >> >> >> >> /dev/sdb3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0
> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> >> Raid Level : -unknown-
> >> >> >> >> Raid Devices : 0
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : active
> >> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >> >>
> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> >> Checksum : 5e8cfc9a - correct
> >> >> >> >> Events : 1
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Device Role : spare
> >> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >> >> /dev/sdc3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0
> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> >> Raid Level : -unknown-
> >> >> >> >> Raid Devices : 0
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : active
> >> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> >> >>
> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> >> Checksum : f69518c - correct
> >> >> >> >> Events : 1
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Device Role : spare
> >> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >> >> /dev/sdd3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0
> >> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> >> Raid Level : -unknown-
> >> >> >> >> Raid Devices : 0
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : active
> >> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> >> >>
> >> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> >> Checksum : 571ad2bd - correct
> >> >> >> >> Events : 1
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Device Role : spare
> >> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >> >>
> >> >> >> >> and finally kernel and mdadm versions:
> >> >> >> >>
> >> >> >> >> uname -a
> >> >> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
> >> >> >> >> 2012 i686 i686 i386 GNU/Linux
> >> >> >> >>
> >> >> >> >> mdadm -V
> >> >> >> >> mdadm - v3.2.3 - 23rd December 2011
> >> >> >> >
> >> >> >> >> /dev/sda3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> >> Raid Level : raid5
> >> >> >> >> Raid Devices : 4
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : clean
> >> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >> >>
> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> >> Checksum : 5ed5b898 - correct
> >> >> >> >> Events : 3925676
> >> >> >> >>
> >> >> >> >> Layout : left-symmetric
> >> >> >> >> Chunk Size : 512K
> >> >> >> >>
> >> >> >> >> Device Role : spare
> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >> >
> >> >> >> >> /dev/sdb3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> >> Raid Level : raid5
> >> >> >> >> Raid Devices : 4
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : clean
> >> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> >> >>
> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> >> Checksum : 57638ebb - correct
> >> >> >> >> Events : 3925676
> >> >> >> >>
> >> >> >> >> Layout : left-symmetric
> >> >> >> >> Chunk Size : 512K
> >> >> >> >>
> >> >> >> >> Device Role : Active device 0
> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >> >
> >> >> >> >> /dev/sdc3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> >> Raid Level : raid5
> >> >> >> >> Raid Devices : 4
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : clean
> >> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> >> >>
> >> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> >> Checksum : fb20d8a - correct
> >> >> >> >> Events : 3925676
> >> >> >> >>
> >> >> >> >> Layout : left-symmetric
> >> >> >> >> Chunk Size : 512K
> >> >> >> >>
> >> >> >> >> Device Role : Active device 2
> >> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >> >
> >> >> >> >> /dev/sdd3:
> >> >> >> >> Magic : a92b4efc
> >> >> >> >> Version : 1.2
> >> >> >> >> Feature Map : 0x0
> >> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> >> Raid Level : raid5
> >> >> >> >> Raid Devices : 4
> >> >> >> >>
> >> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> >> Data Offset : 2048 sectors
> >> >> >> >> Super Offset : 8 sectors
> >> >> >> >> State : clean
> >> >> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
> >> >> >> >>
> >> >> >> >> Update Time : Tue Dec 2 23:14:03 2014
> >> >> >> >> Checksum : a126853f - correct
> >> >> >> >> Events : 3925672
> >> >> >> >>
> >> >> >> >> Layout : left-symmetric
> >> >> >> >> Chunk Size : 512K
> >> >> >> >>
> >> >> >> >> Device Role : Active device 1
> >> >> >> >> Array State : AAAA ('A' == active, '.' == missing)
> >> >> >> >
> >> >> >> > At least you have the previous data anyway, which should allow
> >> >> >> > reconstruction of the array. The device names have changed between your
> >> >> >> > two reports though, so I'd advise double-checking which is which before
> >> >> >> > proceeding.
> >> >> >> >
> >> >> >> > The reports indicate that the original array order (based on the device
> >> >> >> > role field) for the four devices was (using device UUIDs as they're
> >> >> >> > consistent):
> >> >> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> >> > 4156ab46:bd42c10d:8565d5af:74856641
> >> >> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> >> > b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >> >
> >> >> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
> >> >> >> > have the current data for sda3, but that's the only missing UUID).
> >> >> >> >
> I had forgotten that I took a pic of the read error message, which
> also contained an output of /proc/mdstat, so I was able to determine
> the ordering and I ran this command:
>
What did that indicate, and how did you map it to the device order below?
> root@ubuntu:~# mdadm -v --create --assume-clean --level=5 --chunk=512
> --size=1952795136 --raid-devices=4 /dev/md0 /dev/sdd3 /dev/sdb3
> missing /dev/sdc3
> mdadm: layout defaults to left-symmetric
> mdadm: layout defaults to left-symmetric
> mdadm: /dev/sdd3 appears to be part of a raid array:
> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
> mdadm: layout defaults to left-symmetric
> mdadm: /dev/sdb3 appears to be part of a raid array:
> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
> mdadm: layout defaults to left-symmetric
> mdadm: /dev/sdc3 appears to be part of a raid array:
> level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
>
> I did mdadm -E and everything seemed to be consistent with the
> original output of the examine command. So I ran fsck -n
>
> root@ubuntu:~# fsck -n /dev/md0
> fsck from util-linux 2.20.1
> e2fsck 1.42 (29-Nov-2011)
> fsck.ext4: Group descriptors look bad... trying backup blocks...
> Error writing block 1 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 2 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 3 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 4 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 5 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 6 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
> ...
> ...
> Error writing block 343 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> Error writing block 344 (Attempt to write block to filesystem resulted
> in short write). Ignore error? no
>
> fsck.ext4: Device or resource busy while trying to open /dev/md0
> Filesystem mounted or opened exclusively by another program?
>
>
> I believe I made some progress. But before I continue, I wanted to
> know if I was on the right track?
>
> I tried to mount /dev/md0 but got this:
>
> root@ubuntu:~# mount -t ext4 /dev/md0 /mnt/
> mount: wrong fs type, bad option, bad superblock on /dev/md0,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> Am I at a point to run fsck to repair the ext4 superblock?
>
No, that output would definitely suggest you have the wrong order.
That looks to be far too many errors for a normal unclean shutdown
situation.
> I also tried a different ordering to see what fsck -n would give and I got:
>
> root@ubuntu:~# fsck -n /dev/md0
> fsck from util-linux 2.20.1
> e2fsck 1.42 (29-Nov-2011)
> fsck.ext4: Filesystem revision too high while trying to open /dev/md0
> The filesystem revision is apparently too high for this version of e2fsck.
> (Or the filesystem superblock is corrupt)
>
>
> The superblock could not be read or does not describe a correct ext2
> filesystem. If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
> e2fsck -b 8193 <device>
>
> Which seems to confirm my first attempt at the ordering was good.
>
No, it confirms that the first device was correct - the filesystem
superblock will be entirely within the first chunk, so only the first
disk needs to be correct for that to be readable.
Have you tried running it in the order I advised (sdd3, sda3, sdc3,
missing) or in the order of the UUIDs (if the device order has changed)?
92589cc2:9d5ed86c:1467efc2:2e6b7f09
4156ab46:bd42c10d:8565d5af:74856641
390bd4a2:07a28c01:528ed41e:a9d0fcf0
b2bf0462:e0722254:0e233a72:aa5df4da
If not, please do so first and see whether the fsck output is any
better.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: md raid performance with 3-18-rc3
From: Manish Awasthi @ 2014-12-09 8:26 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <5486B1C2.1000508@caviumnetworks.com>
[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]
this time with attachment:
manish
On 12/09/2014 01:54 PM, Manish Awasthi wrote:
> resending:
>
> dirty_ratio same for both the kernels.
>>
>> vm.dirty_background_bytes = 0
>> vm.dirty_background_ratio = 10
>> vm.dirty_bytes = 0
>> vm.dirty_expire_centisecs = 3000
>> vm.dirty_ratio = 20
>> vm.dirty_writeback_centisecs = 500
>>
>>
>> I re-ran the tests with the same set of kernel without enabling
>> multithread support on 3.18 and measured a few things with perf.
>>
>> perf-stat-<kernel>.txt: test ran for some time and measured various
>> parameters.
>>
>> Meanwhile I'm also running complete test under perf record. I'll
>> share the results soon.
>>
>> Manish
>>
>> On 12/03/2014 11:51 AM, NeilBrown wrote:
>>> On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
>>> <manish.awasthi@caviumnetworks.com> wrote:
>>>
>>>> Whatever data I have on comparison is attached, I have consolidated
>>>> this
>>>> from log files to excel. See if this helps.
>>> raid_3_18_performance.xls shows read throughput to be consistently
>>> 20% down
>>> on 3.18 compared to 3.6.11.
>>>
>>> Writes are a few percent better for 4G/8G files, 20% better for
>>> 16G/32G files.
>>> unchanged above that.
>>> Given that you have 8G of RAM, that seems like it could be some
>>> change in
>>> caching behaviour, and not necessarily a change in RAID behaviour.
>>>
>>> The CPU utilization roughly follows the throughput: 40% higher when
>>> write
>>> throughput is 20% better.
>>> Could you check if the value of /proc/sys/vm/dirty_ratio is the same
>>> for both
>>> tests. That number has changed occasionally and could affect these
>>> tests.
>>>
>>>
>>> The second file, 3SSDs-perf-2-Cores-3.18-rc1 has the "change" numbers
>>> negative where I expected positive.. i.e. negative mean an increase.
>>>
>>> Writes consistently have higher CPU utilisation.
>>> Reads consistently have much lower CPU utilization.
>>>
>>> I don't know what that means ... it might not mean anything.
>>>
>>> Could you please run the tests between the two kernels *with* RAID.
>>> i.e.
>>> directly on an SSD. That will give us a baseline for what changes
>>> are caused
>>> by other parts of the kernel (filesystem, block layer, MM, etc).
>>> Then we can
>>> see how much change RAID5 is contributing.
>>>
>>> The third file, 3SSDs-perf-4Core.xls seems to show significantly
>>> reduced
>>> throughput across the board.
>>> CPU utilization is less (better) for writes, but worse for reads.
>>> That is
>>> the reverse of what the second file shows.
>>>
>>> I might try running some tests across a set of kernel versions and
>>> see what I
>>> can come up with.
>>>
>>> NeilBrown
>>
>
[-- Attachment #2: perf-stat-3.6.11.txt --]
[-- Type: text/plain, Size: 3546 bytes --]
perf stat on md125_raid5 -- kernel 3.6.11
# perf stat -p 2613 -e cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles,stalled-cycles-frontend,ref-cycles,cpu-clock,task-clock,faults,context-switches,cpu-migrations,minor-faults,major-faults,alignment-faults,emulation-faults,L1-dcache-load-misses,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,LLC-loads,LLC-stores,LLC-prefetches,dTLB-load-misses,dTLB-store-misses,iTLB-loads,iTLB-load-misses,branch-loads,branch-load-misses
^C
Performance counter stats for process id '2613':
103,200,677,721 cycles # 2.848 GHz [22.72%]
69,669,813,983 instructions # 0.68 insns per cycle
# 1.07 stalled cycles per insn [27.26%]
2,668,465,769 cache-references # 73.648 M/sec [27.35%]
1,408,493,680 cache-misses # 52.783 % of all cache refs [27.17%]
13,609,211,321 branches # 375.607 M/sec [27.19%]
121,593,598 branch-misses # 0.89% of all branches [27.32%]
3,420,725,359 bus-cycles # 94.410 M/sec [18.07%]
74,362,368,252 stalled-cycles-frontend # 72.06% frontend cycles idle [18.16%]
112,553,945,650 ref-cycles # 3106.427 M/sec [22.76%]
36233.766411 cpu-clock (msec)
36232.605499 task-clock (msec) # 0.181 CPUs utilized
0 faults # 0.000 K/sec
442,885 context-switches # 0.012 M/sec
9,646 cpu-migrations # 0.266 K/sec
0 minor-faults # 0.000 K/sec
0 major-faults # 0.000 K/sec
0 alignment-faults # 0.000 K/sec
0 emulation-faults # 0.000 K/sec
3,188,865,936 L1-dcache-load-misses # 88.011 M/sec [22.96%]
1,658,831,957 L1-dcache-store-misses # 45.783 M/sec [22.89%]
338,744,029 L1-dcache-prefetch-misses # 9.349 M/sec [23.04%]
445,066,995 L1-icache-load-misses # 12.284 M/sec [22.99%]
1,578,067,225 LLC-loads # 43.554 M/sec [18.19%]
1,317,822,999 LLC-stores # 36.371 M/sec [18.23%]
798,004,610 LLC-prefetches # 22.024 M/sec [ 9.09%]
0 dTLB-load-misses # 0.000 K/sec [13.52%]
7,633,236 dTLB-store-misses # 0.211 M/sec [18.03%]
10,024,464 iTLB-loads # 0.277 M/sec [17.92%]
3,157,141 iTLB-load-misses # 31.49% of all iTLB cache hits [18.12%]
13,616,857,645 branch-loads # 375.818 M/sec [18.16%]
119,250,450 branch-load-misses # 3.291 M/sec [18.14%]
200.190181623 seconds time elapsed
[-- Attachment #3: perf-stat-3.18.txt --]
[-- Type: text/plain, Size: 3543 bytes --]
perf stat on md125_raid5 -- kernel 3.18
# perf stat -p 2778 -e cycles,instructions,cache-references,cache-misses,branches,branch-misses,bus-cycles,stalled-cycles-frontend,ref-cycles,cpu-clock,task-clock,faults,context-switches,cpu-migrations,minor-faults,major-faults,alignment-faults,emulation-faults,L1-dcache-load-misses,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,LLC-loads,LLC-stores,LLC-prefetches,dTLB-load-misses,dTLB-store-misses,iTLB-loads,iTLB-load-misses,branch-loads,branch-load-misses
^C
Performance counter stats for process id '2778':
191,212,778,981 cycles # 2.942 GHz [22.99%]
160,318,628,367 instructions # 0.84 insns per cycle
# 0.77 stalled cycles per insn [27.49%]
3,800,688,695 cache-references # 58.485 M/sec [27.40%]
1,418,431,693 cache-misses # 37.320 % of all cache refs [27.27%]
33,635,552,951 branches # 517.586 M/sec [27.12%]
352,264,516 branch-misses # 1.05% of all branches [27.19%]
6,035,806,867 bus-cycles # 92.879 M/sec [18.21%]
122,980,401,285 stalled-cycles-frontend # 64.32% frontend cycles idle [18.16%]
197,829,618,312 ref-cycles # 3044.216 M/sec [22.72%]
65039.738267 cpu-clock (msec)
64985.415568 task-clock (msec) # 0.186 CPUs utilized
0 faults # 0.000 K/sec
3,437,945 context-switches # 0.053 M/sec
237 cpu-migrations # 0.004 K/sec
0 minor-faults # 0.000 K/sec
0 major-faults # 0.000 K/sec
0 alignment-faults # 0.000 K/sec
0 emulation-faults # 0.000 K/sec
5,329,711,939 L1-dcache-load-misses # 82.014 M/sec [22.83%]
2,138,400,107 L1-dcache-store-misses # 32.906 M/sec [22.52%]
667,646,968 L1-dcache-prefetch-misses # 10.274 M/sec [22.48%]
2,259,425,830 L1-icache-load-misses # 34.768 M/sec [22.45%]
2,090,596,777 LLC-loads # 32.170 M/sec [17.93%]
1,679,287,271 LLC-stores # 25.841 M/sec [18.04%]
1,120,086,147 LLC-prefetches # 17.236 M/sec [ 9.09%]
465,142,622 dTLB-load-misses # 7.158 M/sec [13.69%]
26,672,298 dTLB-store-misses # 0.410 M/sec [18.26%]
66,723,475 iTLB-loads # 1.027 M/sec [18.37%]
9,736,729 iTLB-load-misses # 14.59% of all iTLB cache hits [18.43%]
33,238,082,664 branch-loads # 511.470 M/sec [18.44%]
346,025,993 branch-load-misses # 5.325 M/sec [18.46%]
348.946853958 seconds time elapsed
^ permalink raw reply
* Re: md raid performance with 3-18-rc3
From: Manish Awasthi @ 2014-12-09 8:24 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <5486B15C.8060109@caviumnetworks.com>
resending:
dirty_ratio same for both the kernels.
>
> vm.dirty_background_bytes = 0
> vm.dirty_background_ratio = 10
> vm.dirty_bytes = 0
> vm.dirty_expire_centisecs = 3000
> vm.dirty_ratio = 20
> vm.dirty_writeback_centisecs = 500
>
>
> I re-ran the tests with the same set of kernel without enabling
> multithread support on 3.18 and measured a few things with perf.
>
> perf-stat-<kernel>.txt: test ran for some time and measured various
> parameters.
>
> Meanwhile I'm also running complete test under perf record. I'll share
> the results soon.
>
> Manish
>
> On 12/03/2014 11:51 AM, NeilBrown wrote:
>> On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
>> <manish.awasthi@caviumnetworks.com> wrote:
>>
>>> Whatever data I have on comparison is attached, I have consolidated this
>>> from log files to excel. See if this helps.
>> raid_3_18_performance.xls shows read throughput to be consistently 20% down
>> on 3.18 compared to 3.6.11.
>>
>> Writes are a few percent better for 4G/8G files, 20% better for 16G/32G files.
>> unchanged above that.
>> Given that you have 8G of RAM, that seems like it could be some change in
>> caching behaviour, and not necessarily a change in RAID behaviour.
>>
>> The CPU utilization roughly follows the throughput: 40% higher when write
>> throughput is 20% better.
>> Could you check if the value of /proc/sys/vm/dirty_ratio is the same for both
>> tests. That number has changed occasionally and could affect these tests.
>>
>>
>> The second file, 3SSDs-perf-2-Cores-3.18-rc1 has the "change" numbers
>> negative where I expected positive.. i.e. negative mean an increase.
>>
>> Writes consistently have higher CPU utilisation.
>> Reads consistently have much lower CPU utilization.
>>
>> I don't know what that means ... it might not mean anything.
>>
>> Could you please run the tests between the two kernels *with* RAID. i.e.
>> directly on an SSD. That will give us a baseline for what changes are caused
>> by other parts of the kernel (filesystem, block layer, MM, etc). Then we can
>> see how much change RAID5 is contributing.
>>
>> The third file, 3SSDs-perf-4Core.xls seems to show significantly reduced
>> throughput across the board.
>> CPU utilization is less (better) for writes, but worse for reads. That is
>> the reverse of what the second file shows.
>>
>> I might try running some tests across a set of kernel versions and see what I
>> can come up with.
>>
>> NeilBrown
>
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-09 5:35 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141208181619.GA8104@cthulhu.home.robinhill.me.uk>
I had forgotten that I took a pic of the read error message, which
also contained an output of /proc/mdstat, so I was able to determine
the ordering and I ran this command:
root@ubuntu:~# mdadm -v --create --assume-clean --level=5 --chunk=512
--size=1952795136 --raid-devices=4 /dev/md0 /dev/sdd3 /dev/sdb3
missing /dev/sdc3
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdd3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdb3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdc3 appears to be part of a raid array:
level=raid5 devices=4 ctime=Tue Dec 9 05:17:53 2014
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
I did mdadm -E and everything seemed to be consistent with the
original output of the examine command. So I ran fsck -n
root@ubuntu:~# fsck -n /dev/md0
fsck from util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
fsck.ext4: Group descriptors look bad... trying backup blocks...
Error writing block 1 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 2 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 3 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 4 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 5 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 6 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
...
...
Error writing block 343 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
Error writing block 344 (Attempt to write block to filesystem resulted
in short write). Ignore error? no
fsck.ext4: Device or resource busy while trying to open /dev/md0
Filesystem mounted or opened exclusively by another program?
I believe I made some progress. But before I continue, I wanted to
know if I was on the right track?
I tried to mount /dev/md0 but got this:
root@ubuntu:~# mount -t ext4 /dev/md0 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
Am I at a point to run fsck to repair the ext4 superblock?
I also ran dump2fs and got this:
root@ubuntu:~# dumpe2fs /dev/md0
dumpe2fs 1.42 (29-Nov-2011)
Filesystem volume name: <none>
Last mounted on: /
Filesystem UUID: 8e314cca-1a2b-4554-a1d5-cd5111240783
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file
huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 366149632
Block count: 1464596352
Reserved block count: 29280500
Free blocks: 177343551
Free inodes: 363706506
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 674
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stride: 128
RAID stripe width: 128
Flex block group size: 16
Filesystem created: Tue Jul 26 03:28:40 2011
Last mount time: Wed Dec 3 03:11:10 2014
Last write time: Tue Nov 25 02:19:04 2014
Mount count: 15
Maximum mount count: 22
Last checked: Wed Aug 13 20:01:25 2014
Check interval: 15552000 (6 months)
Next check after: Mon Feb 9 20:01:25 2015
Lifetime writes: 3824 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: afdd257c-8a09-4154-8fc7-a723af0c675b
Journal backup: inode blocks
Journal features: journal_incompat_revoke
Journal size: 128M
Journal length: 32768
Journal sequence: 0x018fc7c8
Journal start: 55
After which it just hung there, had to ctrl+c.
I also tried a different ordering to see what fsck -n would give and I got:
root@ubuntu:~# fsck -n /dev/md0
fsck from util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
fsck.ext4: Filesystem revision too high while trying to open /dev/md0
The filesystem revision is apparently too high for this version of e2fsck.
(Or the filesystem superblock is corrupt)
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
Which seems to confirm my first attempt at the ordering was good.
On Mon, Dec 8, 2014 at 1:16 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Mon Dec 08, 2014 at 12:22:40PM -0500, Emery Guevremont wrote:
>
>> On Mon, Dec 8, 2014 at 11:55 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> > On Mon Dec 08, 2014 at 11:31:09AM -0500, Emery Guevremont wrote:
>> >> On Mon, Dec 8, 2014 at 10:14 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> > On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
>> >> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
>> >> >> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>> >> >> >> >
>> >> >> >> >> The long story and what I've done.
>> >> >> >> >>
>> >> >> >> >> /dev/md0 is assembled with 4 drives
>> >> >> >> >> /dev/sda3
>> >> >> >> >> /dev/sdb3
>> >> >> >> >> /dev/sdc3
>> >> >> >> >> /dev/sdd3
>> >> >> >> >>
>> >> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> >> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> >> >> >> >> the server and until I received a replacement drive.
>> >> >> >> >>
>> >> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
>> >> >> >> >> single user mode and did this:
>> >> >> >> >>
>> >> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> >> >> >> >> confirmed the resyncing process. The last time I checked it was up to
>> >> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> >> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
>> >> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> >> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> >> >> >> >> everything as is and to go to bed.
>> >> >> >> >>
>> >> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
>> >> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
>> >> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> >> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> >> >> >> >> looks of this.
>> >> >> >> >>
>> >> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> >> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> >> >> >> >> was eventually able to read the bad sector on a retry. I followed up
>> >> >> >> >> by also cloning with ddrescue, sdb and sdc.
>> >> >> >> >>
>> >> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
>> >> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
>> >> >> >> >> all drives are added as spares. Running mdadm --examine on each
>> >> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
>> >> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
>> >> >> >> >> makes sense. I believe I could re-assemble my array if I could define
>> >> >> >> >> the raid level and raid devices.
>> >> >> >> >>
>> >> >> >> >> I wanted to know if there are a way to restore my superblocks from the
>> >> >> >> >> examine command I ran at the beginning? If not, what mdadm create
>> >> >> >> >> command should I run? Also please let me know if drive ordering is
>> >> >> >> >> important, and how I can determine this with the examine output I'll
>> >> >> >> >> got?
>> >> >> >> >>
>> >> >> >> >> Thank you.
>> >> >> >> >>
>> >> >> >> > Have you tried --assemble --force? You'll need to make sure the array's
>> >> >> >> > stopped first, but that's the usual way to get the array back up and
>> >> >> >> > running in that sort of situation.
>> >> >> >> >
>> >> >> >> > If that doesn't work, stop the array again and post:
>> >> >> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
>> >> >> >> > - any dmesg output corresponding with the above
>> >> >> >> > - --examine output for all disks
>> >> >> >> > - kernel and mdadm versions
>> >> >> >> >
>> >> >> >> > Good luck,
>> >> >> >> > Robin
>> >> >> >
>> >> >> >> You'll see from the examine output, raid level and devices aren't
>> >> >> >> defined and notice the role of each drives. The examine output (I
>> >> >> >> attached 4 files) that I took right after the read error during the
>> >> >> >> synching process seems to show a more accurate superblock. Here's also
>> >> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
>> >> >> >> error:
>> >> >> >>
>> >> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> name=runts:0
>> >> >> >> spares=1
>> >> >> >>
>> >> >> >>
>> >> >> >> Here's the output of how things currently are:
>> >> >> >>
>> >> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
>> >> >> >> start the array.
>> >> >> >>
>> >> >> >> dmesg
>> >> >> >> [27903.423895] md: md127 stopped.
>> >> >> >> [27903.434327] md: bind<sdc3>
>> >> >> >> [27903.434767] md: bind<sdd3>
>> >> >> >> [27903.434963] md: bind<sdb3>
>> >> >> >>
>> >> >> >> cat /proc/mdstat
>> >> >> >> root@ubuntu:~# cat /proc/mdstat
>> >> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
>> >> >> >> [raid1] [raid10]
>> >> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
>> >> >> >> 5858387208 blocks super 1.2
>> >> >> >>
>> >> >> >> mdadm --examine /dev/sd[bcd]3
>> >> >> >> /dev/sdb3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0
>> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> Raid Level : -unknown-
>> >> >> >> Raid Devices : 0
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : active
>> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >>
>> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> Checksum : 5e8cfc9a - correct
>> >> >> >> Events : 1
>> >> >> >>
>> >> >> >>
>> >> >> >> Device Role : spare
>> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >> /dev/sdc3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0
>> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> Raid Level : -unknown-
>> >> >> >> Raid Devices : 0
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : active
>> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> >>
>> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> Checksum : f69518c - correct
>> >> >> >> Events : 1
>> >> >> >>
>> >> >> >>
>> >> >> >> Device Role : spare
>> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >> /dev/sdd3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0
>> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> >> Raid Level : -unknown-
>> >> >> >> Raid Devices : 0
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : active
>> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> >>
>> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> >> Checksum : 571ad2bd - correct
>> >> >> >> Events : 1
>> >> >> >>
>> >> >> >>
>> >> >> >> Device Role : spare
>> >> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> >>
>> >> >> >> and finally kernel and mdadm versions:
>> >> >> >>
>> >> >> >> uname -a
>> >> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
>> >> >> >> 2012 i686 i686 i386 GNU/Linux
>> >> >> >>
>> >> >> >> mdadm -V
>> >> >> >> mdadm - v3.2.3 - 23rd December 2011
>> >> >> >
>> >> >> > The missing data looks similar to a bug fixed a couple of years ago
>> >> >> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
>> >> >> > don't match and the missing data is somewhat different - it may be that
>> >> >> > the relevant patches were backported to the vendor kernel you're using.
>> >> >> >
>> >> >> > With that data missing there's no way to assemble though, so a re-create
>> >> >> > is required in this case (it's a last resort, but I don't see any other
>> >> >> > option).
>> >> >> >
>> >> >> >> /dev/sda3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> Raid Level : raid5
>> >> >> >> Raid Devices : 4
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : clean
>> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >>
>> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> Checksum : 5ed5b898 - correct
>> >> >> >> Events : 3925676
>> >> >> >>
>> >> >> >> Layout : left-symmetric
>> >> >> >> Chunk Size : 512K
>> >> >> >>
>> >> >> >> Device Role : spare
>> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >
>> >> >> >> /dev/sdb3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> Raid Level : raid5
>> >> >> >> Raid Devices : 4
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : clean
>> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> >>
>> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> Checksum : 57638ebb - correct
>> >> >> >> Events : 3925676
>> >> >> >>
>> >> >> >> Layout : left-symmetric
>> >> >> >> Chunk Size : 512K
>> >> >> >>
>> >> >> >> Device Role : Active device 0
>> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >
>> >> >> >> /dev/sdc3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> Raid Level : raid5
>> >> >> >> Raid Devices : 4
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : clean
>> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> >>
>> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> >> Checksum : fb20d8a - correct
>> >> >> >> Events : 3925676
>> >> >> >>
>> >> >> >> Layout : left-symmetric
>> >> >> >> Chunk Size : 512K
>> >> >> >>
>> >> >> >> Device Role : Active device 2
>> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >> >
>> >> >> >> /dev/sdd3:
>> >> >> >> Magic : a92b4efc
>> >> >> >> Version : 1.2
>> >> >> >> Feature Map : 0x0
>> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> >> Name : runts:0 (local to host runts)
>> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> >> Raid Level : raid5
>> >> >> >> Raid Devices : 4
>> >> >> >>
>> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> >> Data Offset : 2048 sectors
>> >> >> >> Super Offset : 8 sectors
>> >> >> >> State : clean
>> >> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>> >> >> >>
>> >> >> >> Update Time : Tue Dec 2 23:14:03 2014
>> >> >> >> Checksum : a126853f - correct
>> >> >> >> Events : 3925672
>> >> >> >>
>> >> >> >> Layout : left-symmetric
>> >> >> >> Chunk Size : 512K
>> >> >> >>
>> >> >> >> Device Role : Active device 1
>> >> >> >> Array State : AAAA ('A' == active, '.' == missing)
>> >> >> >
>> >> >> > At least you have the previous data anyway, which should allow
>> >> >> > reconstruction of the array. The device names have changed between your
>> >> >> > two reports though, so I'd advise double-checking which is which before
>> >> >> > proceeding.
>> >> >> >
>> >> >> > The reports indicate that the original array order (based on the device
>> >> >> > role field) for the four devices was (using device UUIDs as they're
>> >> >> > consistent):
>> >> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >> > 4156ab46:bd42c10d:8565d5af:74856641
>> >> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >> > b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >> >
>> >> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
>> >> >> > have the current data for sda3, but that's the only missing UUID).
>> >> >> >
>> >> >> > The create command would therefore be:
>> >> >> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
>> >> >> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
>> >> >> >
>> >> >> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
>> >> >> > array, but you may want to double-check that with a test array on a
>> >> >> > couple of loopback devices first. If not, you'll need to grab the
>> >> >> > latest release and add the --data-offset=2048 parameter to the above
>> >> >> > create command.
>> >> >> >
>> >> >> > You should also follow the instructions for using overlay files at
>> >> >> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>> >> >> > in order to safely test out the above without risking damage to the
>> >> >> > array data.
>> >> >> >
>> >> >> > Once you've run the create, run a "fsck -n" on the filesystem to check
>> >> >> > that the data looks okay. If not, the order or parameters may be
>> >> >> > incorrect - check the --examine output for any differences from the
>> >> >> > original results.
>> >> >> >
>> >> >> Just to double check, would this be the right command to run?
>> >> >>
>> >> >> mdadm --create --assume-clean --level=5 --size=5858385408
>> >> >> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >> >>
>> >> >> Are there any other options I would need to add? Should I specify
>> >> >> --chunk and --size (and if I entered the right size)?
>> >> >>
>> >> > You don't need --assume-clean as there's a missing device, so no scope
>> >> > for rebuilding one of the disks (which is all the flag prevents). It
>> >> > won't do any harm leaving it in though.
>> >> >
>> >> > The size should be the per-device size in kiB (which is half the Used
>> >> > Dev Size value listed in the --examine output, as that's given in
>> >> > 512-byte blocks) and I gave you the correct value above. I'd recommend
>> >> > including this as it will ensure that mdadm isn't calculating the size
>> >> > any different from the version originally used to create the array.
>> >> >
>> >> > The device order you've given is incorrect for either the original
>> >> > device numbering or the numbering you posted as being the most recent.
>> >> > The order I gave above is based on the order as in the latest --examine
>> >> > results you gave. If you've rebooted since then, you'll need to verify
>> >> > the order based on the UUIDs of the devices though (again, the original
>> >> > order should be the one I gave above, based on the device role order in
>> >> > your original --examine output). If you're using different disks, you'll
>> >> > need to be sure which one was mirrored from which original. If you use
>> >> > the incorrect order, you'll get a lot of errors in the "fsck -n" output
>> >> > but, as long as you don't actually write to the array, it shouldn't
>> >> > cause any data corruption as only the metadata will be overwritten.
>> >> >
>> >> > There shouldn't be any need to specify the chunk size, as 512k should be
>> >> > the default value, but I'd probably still stick it in anyway, just to be
>> >> > on the safe side.
>> >> >
>> >> > Similarly with the metadata version - 1.2 is the default (currently
>> >> > anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
>> >> > I'd add it in to be on the safe side.
>> >> >
>> >> >> By the way thanks for the help.
>> >> >>
>> >> >
>> >> > No problem.
>> >> >
>> >> > Cheers,
>> >> > Robin
>> >>
>> >> Here's the adjusted command.
>> >>
>> >> mdadm --create --assume-clean --level=5 --metadata=1.2 --chunk=512
>> >> --size=1952795136 --raid-devices=4 /dev/md0 missing \
>> >> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 \
>> >> 390bd4a2:07a28c01:528ed41e:a9d0fcf0 \
>> >> 4156ab46:bd42c10d:8565d5af:74856641
>> >>
>> > No, the missing should come last - the original --examine info you gave
>> > had info for device roles 0, 1, & 2, so the original failed disk must
>> > have been role 3.
>>
>> As for the ordering this is what I can confirm to you.
>>
>> After sda3 failed, a cat /proc/mdstat displayed _UUU.
>> At that point I haven't done any mdadm -E commands.
>>
>> I rebooted with a new sda hard drive installed.
>>
>> After sdd3 got the read error during the re-sync process, cat
>> /proc/mdstat gave _U_U.
>> But the mdadm -E |grep "Array State " output gave A.A. Is it normal
>> that /proc/mdstat displays the output in reverse? Which one should I
>> rely on to guestimate the ordering?
>>
>> One thing to note about my array, is that it originally was a RAID5
>> with 3 devices. A few years back, one drive failed (possibly sdc if
>> memory serves) and I replaced it and right after that, I added a 4th
>> drive to the aray and made it grow.
>>
> That is rather confusing. I'd expect the /proc/mdstat order to match
> the device role order. Also, the --examine output has /dev/sdd3 showing
> all array devices present, whereas /dev/sdb3 and /dev/sdc3 both show 2
> missing (and the update times are only a couple of minutes apart). That
> would suggest the array was fully built, then sdd3 failed first,
> immediately followed by another device (sda3?). However, sda3 is showing
> as a spare, which would suggest it didn't complete the rebuild - unless
> the rebuild completed, then sdd3 failed after updating its metadata,
> then sda3 failed before updating its metadata (or that for sdb3 or
> sdc3), though the event count was updated on sda3, sdb3 and sdc3, so
> some of the metadata must have been updated (I've no idea how the kernel
> would be expected to handle that sort of situation though).
>
> Anyway, I'd definitely go with the --examine output in preference to the
> /proc/mdstat output. I'd also advise using the overlay devices rather
> than working directly on the physical drives to start with, as that
> pretty much rules out any chance of further data corruption.
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: Disks never stop spinning
From: Phillip Susi @ 2014-12-08 20:38 UTC (permalink / raw)
To: P. Gautschi; +Cc: Peter Grandi, Linux RAID
In-Reply-To: <54831DC7.4070304@gautschi.net>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12/6/2014 10:16 AM, P. Gautschi wrote:
>> Are you running udisks? If so then that is the culprit. It
>> polls the drive's SMART stats every 10 minutes unless the drive
>> is already asleep, so if the timeout is>= 10 minutes ( which it
>> seems WD drives refuse to go any lower anyhow ), then the drive
>> will never go to standby. I posted some patches to fix this a
>> while back on the devkit-devel mailing list but I think the
>> udisks maintainer has still not gotten around to applying them.
>
> This is indeed the cause in my case. There are the two problems
> that overlap: if a shorter time then 10 minutes is set as a standby
> time, the WD disk rounds this up to 10 minutes. And Kubuntu is
> using udisksd. It is started during a KDE login and never stopped
> when logging out.
>
> Is it possible to configure udisks to not do any polling or at a
> much longer interval? Or is it possible to safely disable udisksd
> somehow? I'm still looking for a practical solution.
You can build and install my patched version. See:
https://launchpad.net/bugs/1281588
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUhgxfAAoJENRVrw2cjl5RGxYH/3x1szkPGfky9XKg+rImoVS7
1W8pWCuIww1qLXGTrbapu8+9u5DWteTaLTkvrJvG4V8Gy4NrK1AwbrvF9N9Ox1Nx
aRzI3HuNaFCdqPBDkOAtcguflO2sbEr41NzWtqoW1zytVzcFOlPzOhIitTAh0BR6
ng3QuWf6DrDfLIdQoDkkYOFnaZydGKYMMhIFEMqPWrlJ760Yjrb9GUEVR3g0C2yP
pVmFoL7g4lRyfw4+eXu4+XQFHtjeB+SFvn7nRBIo7ZVWYVimBjX3HmobrIwsJJJ0
nY/AVswj5gMfJAzdmDDiIEJZtzwIbnO9Ogh8Oghdl7Fxg55V/i4Krp9Kvp0GSB0=
=dlgf
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-08 18:16 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sYdbgzZ=fAzoTELnutOG0ge+TcYifJwKjD=W0n2uKVhNw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 20673 bytes --]
On Mon Dec 08, 2014 at 12:22:40PM -0500, Emery Guevremont wrote:
> On Mon, Dec 8, 2014 at 11:55 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> > On Mon Dec 08, 2014 at 11:31:09AM -0500, Emery Guevremont wrote:
> >> On Mon, Dec 8, 2014 at 10:14 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> > On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
> >> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
> >> >> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> >> >> >> >
> >> >> >> >> The long story and what I've done.
> >> >> >> >>
> >> >> >> >> /dev/md0 is assembled with 4 drives
> >> >> >> >> /dev/sda3
> >> >> >> >> /dev/sdb3
> >> >> >> >> /dev/sdc3
> >> >> >> >> /dev/sdd3
> >> >> >> >>
> >> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> >> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> >> >> >> >> the server and until I received a replacement drive.
> >> >> >> >>
> >> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
> >> >> >> >> single user mode and did this:
> >> >> >> >>
> >> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> >> >> >> >> confirmed the resyncing process. The last time I checked it was up to
> >> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
> >> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
> >> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
> >> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> >> >> >> >> everything as is and to go to bed.
> >> >> >> >>
> >> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
> >> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
> >> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> >> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> >> >> >> >> looks of this.
> >> >> >> >>
> >> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> >> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> >> >> >> >> was eventually able to read the bad sector on a retry. I followed up
> >> >> >> >> by also cloning with ddrescue, sdb and sdc.
> >> >> >> >>
> >> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
> >> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
> >> >> >> >> all drives are added as spares. Running mdadm --examine on each
> >> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
> >> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
> >> >> >> >> makes sense. I believe I could re-assemble my array if I could define
> >> >> >> >> the raid level and raid devices.
> >> >> >> >>
> >> >> >> >> I wanted to know if there are a way to restore my superblocks from the
> >> >> >> >> examine command I ran at the beginning? If not, what mdadm create
> >> >> >> >> command should I run? Also please let me know if drive ordering is
> >> >> >> >> important, and how I can determine this with the examine output I'll
> >> >> >> >> got?
> >> >> >> >>
> >> >> >> >> Thank you.
> >> >> >> >>
> >> >> >> > Have you tried --assemble --force? You'll need to make sure the array's
> >> >> >> > stopped first, but that's the usual way to get the array back up and
> >> >> >> > running in that sort of situation.
> >> >> >> >
> >> >> >> > If that doesn't work, stop the array again and post:
> >> >> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
> >> >> >> > - any dmesg output corresponding with the above
> >> >> >> > - --examine output for all disks
> >> >> >> > - kernel and mdadm versions
> >> >> >> >
> >> >> >> > Good luck,
> >> >> >> > Robin
> >> >> >
> >> >> >> You'll see from the examine output, raid level and devices aren't
> >> >> >> defined and notice the role of each drives. The examine output (I
> >> >> >> attached 4 files) that I took right after the read error during the
> >> >> >> synching process seems to show a more accurate superblock. Here's also
> >> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
> >> >> >> error:
> >> >> >>
> >> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> name=runts:0
> >> >> >> spares=1
> >> >> >>
> >> >> >>
> >> >> >> Here's the output of how things currently are:
> >> >> >>
> >> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
> >> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
> >> >> >> start the array.
> >> >> >>
> >> >> >> dmesg
> >> >> >> [27903.423895] md: md127 stopped.
> >> >> >> [27903.434327] md: bind<sdc3>
> >> >> >> [27903.434767] md: bind<sdd3>
> >> >> >> [27903.434963] md: bind<sdb3>
> >> >> >>
> >> >> >> cat /proc/mdstat
> >> >> >> root@ubuntu:~# cat /proc/mdstat
> >> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
> >> >> >> [raid1] [raid10]
> >> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
> >> >> >> 5858387208 blocks super 1.2
> >> >> >>
> >> >> >> mdadm --examine /dev/sd[bcd]3
> >> >> >> /dev/sdb3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0
> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> Raid Level : -unknown-
> >> >> >> Raid Devices : 0
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : active
> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >>
> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> Checksum : 5e8cfc9a - correct
> >> >> >> Events : 1
> >> >> >>
> >> >> >>
> >> >> >> Device Role : spare
> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >> /dev/sdc3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0
> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> Raid Level : -unknown-
> >> >> >> Raid Devices : 0
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : active
> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> >>
> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> Checksum : f69518c - correct
> >> >> >> Events : 1
> >> >> >>
> >> >> >>
> >> >> >> Device Role : spare
> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >> /dev/sdd3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0
> >> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> >> Raid Level : -unknown-
> >> >> >> Raid Devices : 0
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : active
> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> >>
> >> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> >> Checksum : 571ad2bd - correct
> >> >> >> Events : 1
> >> >> >>
> >> >> >>
> >> >> >> Device Role : spare
> >> >> >> Array State : ('A' == active, '.' == missing)
> >> >> >>
> >> >> >> and finally kernel and mdadm versions:
> >> >> >>
> >> >> >> uname -a
> >> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
> >> >> >> 2012 i686 i686 i386 GNU/Linux
> >> >> >>
> >> >> >> mdadm -V
> >> >> >> mdadm - v3.2.3 - 23rd December 2011
> >> >> >
> >> >> > The missing data looks similar to a bug fixed a couple of years ago
> >> >> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
> >> >> > don't match and the missing data is somewhat different - it may be that
> >> >> > the relevant patches were backported to the vendor kernel you're using.
> >> >> >
> >> >> > With that data missing there's no way to assemble though, so a re-create
> >> >> > is required in this case (it's a last resort, but I don't see any other
> >> >> > option).
> >> >> >
> >> >> >> /dev/sda3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> Raid Level : raid5
> >> >> >> Raid Devices : 4
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : clean
> >> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >>
> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> Checksum : 5ed5b898 - correct
> >> >> >> Events : 3925676
> >> >> >>
> >> >> >> Layout : left-symmetric
> >> >> >> Chunk Size : 512K
> >> >> >>
> >> >> >> Device Role : spare
> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >
> >> >> >> /dev/sdb3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> Raid Level : raid5
> >> >> >> Raid Devices : 4
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : clean
> >> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> >>
> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> Checksum : 57638ebb - correct
> >> >> >> Events : 3925676
> >> >> >>
> >> >> >> Layout : left-symmetric
> >> >> >> Chunk Size : 512K
> >> >> >>
> >> >> >> Device Role : Active device 0
> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >
> >> >> >> /dev/sdc3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> Raid Level : raid5
> >> >> >> Raid Devices : 4
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : clean
> >> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> >>
> >> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> >> Checksum : fb20d8a - correct
> >> >> >> Events : 3925676
> >> >> >>
> >> >> >> Layout : left-symmetric
> >> >> >> Chunk Size : 512K
> >> >> >>
> >> >> >> Device Role : Active device 2
> >> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >> >
> >> >> >> /dev/sdd3:
> >> >> >> Magic : a92b4efc
> >> >> >> Version : 1.2
> >> >> >> Feature Map : 0x0
> >> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> >> Name : runts:0 (local to host runts)
> >> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> >> Raid Level : raid5
> >> >> >> Raid Devices : 4
> >> >> >>
> >> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> >> Data Offset : 2048 sectors
> >> >> >> Super Offset : 8 sectors
> >> >> >> State : clean
> >> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
> >> >> >>
> >> >> >> Update Time : Tue Dec 2 23:14:03 2014
> >> >> >> Checksum : a126853f - correct
> >> >> >> Events : 3925672
> >> >> >>
> >> >> >> Layout : left-symmetric
> >> >> >> Chunk Size : 512K
> >> >> >>
> >> >> >> Device Role : Active device 1
> >> >> >> Array State : AAAA ('A' == active, '.' == missing)
> >> >> >
> >> >> > At least you have the previous data anyway, which should allow
> >> >> > reconstruction of the array. The device names have changed between your
> >> >> > two reports though, so I'd advise double-checking which is which before
> >> >> > proceeding.
> >> >> >
> >> >> > The reports indicate that the original array order (based on the device
> >> >> > role field) for the four devices was (using device UUIDs as they're
> >> >> > consistent):
> >> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >> > 4156ab46:bd42c10d:8565d5af:74856641
> >> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >> > b2bf0462:e0722254:0e233a72:aa5df4da
> >> >> >
> >> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
> >> >> > have the current data for sda3, but that's the only missing UUID).
> >> >> >
> >> >> > The create command would therefore be:
> >> >> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
> >> >> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
> >> >> >
> >> >> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
> >> >> > array, but you may want to double-check that with a test array on a
> >> >> > couple of loopback devices first. If not, you'll need to grab the
> >> >> > latest release and add the --data-offset=2048 parameter to the above
> >> >> > create command.
> >> >> >
> >> >> > You should also follow the instructions for using overlay files at
> >> >> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> >> >> > in order to safely test out the above without risking damage to the
> >> >> > array data.
> >> >> >
> >> >> > Once you've run the create, run a "fsck -n" on the filesystem to check
> >> >> > that the data looks okay. If not, the order or parameters may be
> >> >> > incorrect - check the --examine output for any differences from the
> >> >> > original results.
> >> >> >
> >> >> Just to double check, would this be the right command to run?
> >> >>
> >> >> mdadm --create --assume-clean --level=5 --size=5858385408
> >> >> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
> >> >>
> >> >> Are there any other options I would need to add? Should I specify
> >> >> --chunk and --size (and if I entered the right size)?
> >> >>
> >> > You don't need --assume-clean as there's a missing device, so no scope
> >> > for rebuilding one of the disks (which is all the flag prevents). It
> >> > won't do any harm leaving it in though.
> >> >
> >> > The size should be the per-device size in kiB (which is half the Used
> >> > Dev Size value listed in the --examine output, as that's given in
> >> > 512-byte blocks) and I gave you the correct value above. I'd recommend
> >> > including this as it will ensure that mdadm isn't calculating the size
> >> > any different from the version originally used to create the array.
> >> >
> >> > The device order you've given is incorrect for either the original
> >> > device numbering or the numbering you posted as being the most recent.
> >> > The order I gave above is based on the order as in the latest --examine
> >> > results you gave. If you've rebooted since then, you'll need to verify
> >> > the order based on the UUIDs of the devices though (again, the original
> >> > order should be the one I gave above, based on the device role order in
> >> > your original --examine output). If you're using different disks, you'll
> >> > need to be sure which one was mirrored from which original. If you use
> >> > the incorrect order, you'll get a lot of errors in the "fsck -n" output
> >> > but, as long as you don't actually write to the array, it shouldn't
> >> > cause any data corruption as only the metadata will be overwritten.
> >> >
> >> > There shouldn't be any need to specify the chunk size, as 512k should be
> >> > the default value, but I'd probably still stick it in anyway, just to be
> >> > on the safe side.
> >> >
> >> > Similarly with the metadata version - 1.2 is the default (currently
> >> > anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
> >> > I'd add it in to be on the safe side.
> >> >
> >> >> By the way thanks for the help.
> >> >>
> >> >
> >> > No problem.
> >> >
> >> > Cheers,
> >> > Robin
> >>
> >> Here's the adjusted command.
> >>
> >> mdadm --create --assume-clean --level=5 --metadata=1.2 --chunk=512
> >> --size=1952795136 --raid-devices=4 /dev/md0 missing \
> >> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 \
> >> 390bd4a2:07a28c01:528ed41e:a9d0fcf0 \
> >> 4156ab46:bd42c10d:8565d5af:74856641
> >>
> > No, the missing should come last - the original --examine info you gave
> > had info for device roles 0, 1, & 2, so the original failed disk must
> > have been role 3.
>
> As for the ordering this is what I can confirm to you.
>
> After sda3 failed, a cat /proc/mdstat displayed _UUU.
> At that point I haven't done any mdadm -E commands.
>
> I rebooted with a new sda hard drive installed.
>
> After sdd3 got the read error during the re-sync process, cat
> /proc/mdstat gave _U_U.
> But the mdadm -E |grep "Array State " output gave A.A. Is it normal
> that /proc/mdstat displays the output in reverse? Which one should I
> rely on to guestimate the ordering?
>
> One thing to note about my array, is that it originally was a RAID5
> with 3 devices. A few years back, one drive failed (possibly sdc if
> memory serves) and I replaced it and right after that, I added a 4th
> drive to the aray and made it grow.
>
That is rather confusing. I'd expect the /proc/mdstat order to match
the device role order. Also, the --examine output has /dev/sdd3 showing
all array devices present, whereas /dev/sdb3 and /dev/sdc3 both show 2
missing (and the update times are only a couple of minutes apart). That
would suggest the array was fully built, then sdd3 failed first,
immediately followed by another device (sda3?). However, sda3 is showing
as a spare, which would suggest it didn't complete the rebuild - unless
the rebuild completed, then sdd3 failed after updating its metadata,
then sda3 failed before updating its metadata (or that for sdb3 or
sdc3), though the event count was updated on sda3, sdb3 and sdc3, so
some of the metadata must have been updated (I've no idea how the kernel
would be expected to handle that sort of situation though).
Anyway, I'd definitely go with the --examine output in preference to the
/proc/mdstat output. I'd also advise using the overlay devices rather
than working directly on the physical drives to start with, as that
pretty much rules out any chance of further data corruption.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-08 17:22 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141208165535.GC8530@cthulhu.home.robinhill.me.uk>
On Mon, Dec 8, 2014 at 11:55 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Mon Dec 08, 2014 at 11:31:09AM -0500, Emery Guevremont wrote:
>> On Mon, Dec 8, 2014 at 10:14 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> > On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
>> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
>> >> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>> >> >> >
>> >> >> >> The long story and what I've done.
>> >> >> >>
>> >> >> >> /dev/md0 is assembled with 4 drives
>> >> >> >> /dev/sda3
>> >> >> >> /dev/sdb3
>> >> >> >> /dev/sdc3
>> >> >> >> /dev/sdd3
>> >> >> >>
>> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> >> >> >> the server and until I received a replacement drive.
>> >> >> >>
>> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
>> >> >> >> single user mode and did this:
>> >> >> >>
>> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> >> >> >> confirmed the resyncing process. The last time I checked it was up to
>> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
>> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> >> >> >> everything as is and to go to bed.
>> >> >> >>
>> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
>> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
>> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> >> >> >> looks of this.
>> >> >> >>
>> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> >> >> >> was eventually able to read the bad sector on a retry. I followed up
>> >> >> >> by also cloning with ddrescue, sdb and sdc.
>> >> >> >>
>> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
>> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
>> >> >> >> all drives are added as spares. Running mdadm --examine on each
>> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
>> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
>> >> >> >> makes sense. I believe I could re-assemble my array if I could define
>> >> >> >> the raid level and raid devices.
>> >> >> >>
>> >> >> >> I wanted to know if there are a way to restore my superblocks from the
>> >> >> >> examine command I ran at the beginning? If not, what mdadm create
>> >> >> >> command should I run? Also please let me know if drive ordering is
>> >> >> >> important, and how I can determine this with the examine output I'll
>> >> >> >> got?
>> >> >> >>
>> >> >> >> Thank you.
>> >> >> >>
>> >> >> > Have you tried --assemble --force? You'll need to make sure the array's
>> >> >> > stopped first, but that's the usual way to get the array back up and
>> >> >> > running in that sort of situation.
>> >> >> >
>> >> >> > If that doesn't work, stop the array again and post:
>> >> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
>> >> >> > - any dmesg output corresponding with the above
>> >> >> > - --examine output for all disks
>> >> >> > - kernel and mdadm versions
>> >> >> >
>> >> >> > Good luck,
>> >> >> > Robin
>> >> >
>> >> >> You'll see from the examine output, raid level and devices aren't
>> >> >> defined and notice the role of each drives. The examine output (I
>> >> >> attached 4 files) that I took right after the read error during the
>> >> >> synching process seems to show a more accurate superblock. Here's also
>> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
>> >> >> error:
>> >> >>
>> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> name=runts:0
>> >> >> spares=1
>> >> >>
>> >> >>
>> >> >> Here's the output of how things currently are:
>> >> >>
>> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
>> >> >> start the array.
>> >> >>
>> >> >> dmesg
>> >> >> [27903.423895] md: md127 stopped.
>> >> >> [27903.434327] md: bind<sdc3>
>> >> >> [27903.434767] md: bind<sdd3>
>> >> >> [27903.434963] md: bind<sdb3>
>> >> >>
>> >> >> cat /proc/mdstat
>> >> >> root@ubuntu:~# cat /proc/mdstat
>> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
>> >> >> [raid1] [raid10]
>> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
>> >> >> 5858387208 blocks super 1.2
>> >> >>
>> >> >> mdadm --examine /dev/sd[bcd]3
>> >> >> /dev/sdb3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0
>> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> Raid Level : -unknown-
>> >> >> Raid Devices : 0
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : active
>> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >>
>> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> Checksum : 5e8cfc9a - correct
>> >> >> Events : 1
>> >> >>
>> >> >>
>> >> >> Device Role : spare
>> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> /dev/sdc3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0
>> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> Raid Level : -unknown-
>> >> >> Raid Devices : 0
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : active
>> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >>
>> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> Checksum : f69518c - correct
>> >> >> Events : 1
>> >> >>
>> >> >>
>> >> >> Device Role : spare
>> >> >> Array State : ('A' == active, '.' == missing)
>> >> >> /dev/sdd3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0
>> >> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> >> Raid Level : -unknown-
>> >> >> Raid Devices : 0
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : active
>> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >>
>> >> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> >> Checksum : 571ad2bd - correct
>> >> >> Events : 1
>> >> >>
>> >> >>
>> >> >> Device Role : spare
>> >> >> Array State : ('A' == active, '.' == missing)
>> >> >>
>> >> >> and finally kernel and mdadm versions:
>> >> >>
>> >> >> uname -a
>> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
>> >> >> 2012 i686 i686 i386 GNU/Linux
>> >> >>
>> >> >> mdadm -V
>> >> >> mdadm - v3.2.3 - 23rd December 2011
>> >> >
>> >> > The missing data looks similar to a bug fixed a couple of years ago
>> >> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
>> >> > don't match and the missing data is somewhat different - it may be that
>> >> > the relevant patches were backported to the vendor kernel you're using.
>> >> >
>> >> > With that data missing there's no way to assemble though, so a re-create
>> >> > is required in this case (it's a last resort, but I don't see any other
>> >> > option).
>> >> >
>> >> >> /dev/sda3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0 (local to host runts)
>> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> Raid Level : raid5
>> >> >> Raid Devices : 4
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : clean
>> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >>
>> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> Checksum : 5ed5b898 - correct
>> >> >> Events : 3925676
>> >> >>
>> >> >> Layout : left-symmetric
>> >> >> Chunk Size : 512K
>> >> >>
>> >> >> Device Role : spare
>> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >
>> >> >> /dev/sdb3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0 (local to host runts)
>> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> Raid Level : raid5
>> >> >> Raid Devices : 4
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : clean
>> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> >>
>> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> Checksum : 57638ebb - correct
>> >> >> Events : 3925676
>> >> >>
>> >> >> Layout : left-symmetric
>> >> >> Chunk Size : 512K
>> >> >>
>> >> >> Device Role : Active device 0
>> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >
>> >> >> /dev/sdc3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0 (local to host runts)
>> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> Raid Level : raid5
>> >> >> Raid Devices : 4
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : clean
>> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> >>
>> >> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> >> Checksum : fb20d8a - correct
>> >> >> Events : 3925676
>> >> >>
>> >> >> Layout : left-symmetric
>> >> >> Chunk Size : 512K
>> >> >>
>> >> >> Device Role : Active device 2
>> >> >> Array State : A.A. ('A' == active, '.' == missing)
>> >> >
>> >> >> /dev/sdd3:
>> >> >> Magic : a92b4efc
>> >> >> Version : 1.2
>> >> >> Feature Map : 0x0
>> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> >> Name : runts:0 (local to host runts)
>> >> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> >> Raid Level : raid5
>> >> >> Raid Devices : 4
>> >> >>
>> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> >> Data Offset : 2048 sectors
>> >> >> Super Offset : 8 sectors
>> >> >> State : clean
>> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>> >> >>
>> >> >> Update Time : Tue Dec 2 23:14:03 2014
>> >> >> Checksum : a126853f - correct
>> >> >> Events : 3925672
>> >> >>
>> >> >> Layout : left-symmetric
>> >> >> Chunk Size : 512K
>> >> >>
>> >> >> Device Role : Active device 1
>> >> >> Array State : AAAA ('A' == active, '.' == missing)
>> >> >
>> >> > At least you have the previous data anyway, which should allow
>> >> > reconstruction of the array. The device names have changed between your
>> >> > two reports though, so I'd advise double-checking which is which before
>> >> > proceeding.
>> >> >
>> >> > The reports indicate that the original array order (based on the device
>> >> > role field) for the four devices was (using device UUIDs as they're
>> >> > consistent):
>> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >> > 4156ab46:bd42c10d:8565d5af:74856641
>> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >> > b2bf0462:e0722254:0e233a72:aa5df4da
>> >> >
>> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
>> >> > have the current data for sda3, but that's the only missing UUID).
>> >> >
>> >> > The create command would therefore be:
>> >> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
>> >> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
>> >> >
>> >> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
>> >> > array, but you may want to double-check that with a test array on a
>> >> > couple of loopback devices first. If not, you'll need to grab the
>> >> > latest release and add the --data-offset=2048 parameter to the above
>> >> > create command.
>> >> >
>> >> > You should also follow the instructions for using overlay files at
>> >> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>> >> > in order to safely test out the above without risking damage to the
>> >> > array data.
>> >> >
>> >> > Once you've run the create, run a "fsck -n" on the filesystem to check
>> >> > that the data looks okay. If not, the order or parameters may be
>> >> > incorrect - check the --examine output for any differences from the
>> >> > original results.
>> >> >
>> >> Just to double check, would this be the right command to run?
>> >>
>> >> mdadm --create --assume-clean --level=5 --size=5858385408
>> >> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >>
>> >> Are there any other options I would need to add? Should I specify
>> >> --chunk and --size (and if I entered the right size)?
>> >>
>> > You don't need --assume-clean as there's a missing device, so no scope
>> > for rebuilding one of the disks (which is all the flag prevents). It
>> > won't do any harm leaving it in though.
>> >
>> > The size should be the per-device size in kiB (which is half the Used
>> > Dev Size value listed in the --examine output, as that's given in
>> > 512-byte blocks) and I gave you the correct value above. I'd recommend
>> > including this as it will ensure that mdadm isn't calculating the size
>> > any different from the version originally used to create the array.
>> >
>> > The device order you've given is incorrect for either the original
>> > device numbering or the numbering you posted as being the most recent.
>> > The order I gave above is based on the order as in the latest --examine
>> > results you gave. If you've rebooted since then, you'll need to verify
>> > the order based on the UUIDs of the devices though (again, the original
>> > order should be the one I gave above, based on the device role order in
>> > your original --examine output). If you're using different disks, you'll
>> > need to be sure which one was mirrored from which original. If you use
>> > the incorrect order, you'll get a lot of errors in the "fsck -n" output
>> > but, as long as you don't actually write to the array, it shouldn't
>> > cause any data corruption as only the metadata will be overwritten.
>> >
>> > There shouldn't be any need to specify the chunk size, as 512k should be
>> > the default value, but I'd probably still stick it in anyway, just to be
>> > on the safe side.
>> >
>> > Similarly with the metadata version - 1.2 is the default (currently
>> > anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
>> > I'd add it in to be on the safe side.
>> >
>> >> By the way thanks for the help.
>> >>
>> >
>> > No problem.
>> >
>> > Cheers,
>> > Robin
>>
>> Here's the adjusted command.
>>
>> mdadm --create --assume-clean --level=5 --metadata=1.2 --chunk=512
>> --size=1952795136 --raid-devices=4 /dev/md0 missing \
>> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 \
>> 390bd4a2:07a28c01:528ed41e:a9d0fcf0 \
>> 4156ab46:bd42c10d:8565d5af:74856641
>>
> No, the missing should come last - the original --examine info you gave
> had info for device roles 0, 1, & 2, so the original failed disk must
> have been role 3.
As for the ordering this is what I can confirm to you.
After sda3 failed, a cat /proc/mdstat displayed _UUU.
At that point I haven't done any mdadm -E commands.
I rebooted with a new sda hard drive installed.
After sdd3 got the read error during the re-sync process, cat
/proc/mdstat gave _U_U.
But the mdadm -E |grep "Array State " output gave A.A. Is it normal
that /proc/mdstat displays the output in reverse? Which one should I
rely on to guestimate the ordering?
One thing to note about my array, is that it originally was a RAID5
with 3 devices. A few years back, one drive failed (possibly sdc if
memory serves) and I replaced it and right after that, I added a 4th
drive to the aray and made it grow.
>
>> For the --size option, I'm not quite sure I understood what you tried
>> to explain to me. I re-read the manpage and I came up with this 2
>> equations:
>>
>> (My understanding of your explanation) Used Dev size (3905590272)
>> divided by 2 = size (1952795136)
>> (My understanding from the manpages) Used Dev size (3905590272)
>> divided by chunk size (512) = size (7628106)
>>
> No, the mdadm manual page says that it has to be a multiple of the chunk
> size, not that it's given in multiples of the chunk size. It also says
> (in the 3,3,1 release anyway) that it's the "Amount (in Kibibytes)".
> It's not spelt out that the Used Dev size is in 512-byte blocks, but
> that's obvious from the corresponding Gib size given. You can check by
> creating some loopback devices and testing creating an array if you
> like.
>
>> As for the device, I should order them with the device UUID (as shown
>> above) and I replace those UUID with the /dev/sdX3 that returns the
>> same device uuid from a mdadm -E command I will currently get? i.e.
>> mdadm -E /dev/sdd3 returns a device uuid of
>> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 , my first device with be
>> /dev/sdd3...?
>>
> That's correct, yes.
>
>> One last question, after running mdadm --create command, can I run
>> mdadm -E and verify the values I get (chunk size, used dev size...)
>> match the ones I got from my first mdadm -E command, and if it
>> doesn't, to rerun the mdadm --create command to eventually get
>> matching values?
>>
> Yes, the --create command will only overwrite the array metadata, so as
> long as your array offset is correct then the actual array data will be
> untouched (as the 1.2 superblock is near the start of the device, even a
> size error won't damage the data). You'll want to ensure that the chunk
> size & dev size match the originals, and that the device role is correct
> for the corresponding device UUID.
>
> Once that all matches, you can do the "fsck -f -n" and check that there
> are no errors (or only a handful - there may be somea errors after the
> array failure anyway).
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-08 16:55 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sZNxNDO_f0X_uXwLuM8kcRgT=p+f8+-wtYDhEQPy_dX5Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 19507 bytes --]
On Mon Dec 08, 2014 at 11:31:09AM -0500, Emery Guevremont wrote:
> On Mon, Dec 8, 2014 at 10:14 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> > On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
> >> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
> >> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> >> >> >
> >> >> >> The long story and what I've done.
> >> >> >>
> >> >> >> /dev/md0 is assembled with 4 drives
> >> >> >> /dev/sda3
> >> >> >> /dev/sdb3
> >> >> >> /dev/sdc3
> >> >> >> /dev/sdd3
> >> >> >>
> >> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> >> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> >> >> >> the server and until I received a replacement drive.
> >> >> >>
> >> >> >> This week, I replaced the dying drive with my new drive. Booted into
> >> >> >> single user mode and did this:
> >> >> >>
> >> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> >> >> >> confirmed the resyncing process. The last time I checked it was up to
> >> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
> >> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
> >> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
> >> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> >> >> >> everything as is and to go to bed.
> >> >> >>
> >> >> >> The next day, I shutdown the server and reboot with a live usb distro
> >> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
> >> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> >> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> >> >> >> looks of this.
> >> >> >>
> >> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> >> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> >> >> >> was eventually able to read the bad sector on a retry. I followed up
> >> >> >> by also cloning with ddrescue, sdb and sdc.
> >> >> >>
> >> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
> >> >> >> Currently running mdadm --assemble --scan, will activate my array, but
> >> >> >> all drives are added as spares. Running mdadm --examine on each
> >> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
> >> >> >> and raid level is -unknown- for some reason. The rest seems fine and
> >> >> >> makes sense. I believe I could re-assemble my array if I could define
> >> >> >> the raid level and raid devices.
> >> >> >>
> >> >> >> I wanted to know if there are a way to restore my superblocks from the
> >> >> >> examine command I ran at the beginning? If not, what mdadm create
> >> >> >> command should I run? Also please let me know if drive ordering is
> >> >> >> important, and how I can determine this with the examine output I'll
> >> >> >> got?
> >> >> >>
> >> >> >> Thank you.
> >> >> >>
> >> >> > Have you tried --assemble --force? You'll need to make sure the array's
> >> >> > stopped first, but that's the usual way to get the array back up and
> >> >> > running in that sort of situation.
> >> >> >
> >> >> > If that doesn't work, stop the array again and post:
> >> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
> >> >> > - any dmesg output corresponding with the above
> >> >> > - --examine output for all disks
> >> >> > - kernel and mdadm versions
> >> >> >
> >> >> > Good luck,
> >> >> > Robin
> >> >
> >> >> You'll see from the examine output, raid level and devices aren't
> >> >> defined and notice the role of each drives. The examine output (I
> >> >> attached 4 files) that I took right after the read error during the
> >> >> synching process seems to show a more accurate superblock. Here's also
> >> >> the output of mdadm --detail /dev/md0 that I took when I got the first
> >> >> error:
> >> >>
> >> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
> >> >> name=runts:0
> >> >> spares=1
> >> >>
> >> >>
> >> >> Here's the output of how things currently are:
> >> >>
> >> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
> >> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
> >> >> start the array.
> >> >>
> >> >> dmesg
> >> >> [27903.423895] md: md127 stopped.
> >> >> [27903.434327] md: bind<sdc3>
> >> >> [27903.434767] md: bind<sdd3>
> >> >> [27903.434963] md: bind<sdb3>
> >> >>
> >> >> cat /proc/mdstat
> >> >> root@ubuntu:~# cat /proc/mdstat
> >> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
> >> >> [raid1] [raid10]
> >> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
> >> >> 5858387208 blocks super 1.2
> >> >>
> >> >> mdadm --examine /dev/sd[bcd]3
> >> >> /dev/sdb3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0
> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> Raid Level : -unknown-
> >> >> Raid Devices : 0
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : active
> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >>
> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> Checksum : 5e8cfc9a - correct
> >> >> Events : 1
> >> >>
> >> >>
> >> >> Device Role : spare
> >> >> Array State : ('A' == active, '.' == missing)
> >> >> /dev/sdc3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0
> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> Raid Level : -unknown-
> >> >> Raid Devices : 0
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : active
> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >>
> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> Checksum : f69518c - correct
> >> >> Events : 1
> >> >>
> >> >>
> >> >> Device Role : spare
> >> >> Array State : ('A' == active, '.' == missing)
> >> >> /dev/sdd3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0
> >> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> >> Raid Level : -unknown-
> >> >> Raid Devices : 0
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : active
> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >>
> >> >> Update Time : Sat Dec 6 12:46:40 2014
> >> >> Checksum : 571ad2bd - correct
> >> >> Events : 1
> >> >>
> >> >>
> >> >> Device Role : spare
> >> >> Array State : ('A' == active, '.' == missing)
> >> >>
> >> >> and finally kernel and mdadm versions:
> >> >>
> >> >> uname -a
> >> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
> >> >> 2012 i686 i686 i386 GNU/Linux
> >> >>
> >> >> mdadm -V
> >> >> mdadm - v3.2.3 - 23rd December 2011
> >> >
> >> > The missing data looks similar to a bug fixed a couple of years ago
> >> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
> >> > don't match and the missing data is somewhat different - it may be that
> >> > the relevant patches were backported to the vendor kernel you're using.
> >> >
> >> > With that data missing there's no way to assemble though, so a re-create
> >> > is required in this case (it's a last resort, but I don't see any other
> >> > option).
> >> >
> >> >> /dev/sda3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0 (local to host runts)
> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> Raid Level : raid5
> >> >> Raid Devices : 4
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : clean
> >> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >> >>
> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> Checksum : 5ed5b898 - correct
> >> >> Events : 3925676
> >> >>
> >> >> Layout : left-symmetric
> >> >> Chunk Size : 512K
> >> >>
> >> >> Device Role : spare
> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >
> >> >> /dev/sdb3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0 (local to host runts)
> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> Raid Level : raid5
> >> >> Raid Devices : 4
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : clean
> >> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> >>
> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> Checksum : 57638ebb - correct
> >> >> Events : 3925676
> >> >>
> >> >> Layout : left-symmetric
> >> >> Chunk Size : 512K
> >> >>
> >> >> Device Role : Active device 0
> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >
> >> >> /dev/sdc3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0 (local to host runts)
> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> Raid Level : raid5
> >> >> Raid Devices : 4
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : clean
> >> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> >>
> >> >> Update Time : Tue Dec 2 23:15:37 2014
> >> >> Checksum : fb20d8a - correct
> >> >> Events : 3925676
> >> >>
> >> >> Layout : left-symmetric
> >> >> Chunk Size : 512K
> >> >>
> >> >> Device Role : Active device 2
> >> >> Array State : A.A. ('A' == active, '.' == missing)
> >> >
> >> >> /dev/sdd3:
> >> >> Magic : a92b4efc
> >> >> Version : 1.2
> >> >> Feature Map : 0x0
> >> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> >> Name : runts:0 (local to host runts)
> >> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> >> Raid Level : raid5
> >> >> Raid Devices : 4
> >> >>
> >> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> >> Data Offset : 2048 sectors
> >> >> Super Offset : 8 sectors
> >> >> State : clean
> >> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
> >> >>
> >> >> Update Time : Tue Dec 2 23:14:03 2014
> >> >> Checksum : a126853f - correct
> >> >> Events : 3925672
> >> >>
> >> >> Layout : left-symmetric
> >> >> Chunk Size : 512K
> >> >>
> >> >> Device Role : Active device 1
> >> >> Array State : AAAA ('A' == active, '.' == missing)
> >> >
> >> > At least you have the previous data anyway, which should allow
> >> > reconstruction of the array. The device names have changed between your
> >> > two reports though, so I'd advise double-checking which is which before
> >> > proceeding.
> >> >
> >> > The reports indicate that the original array order (based on the device
> >> > role field) for the four devices was (using device UUIDs as they're
> >> > consistent):
> >> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >> > 4156ab46:bd42c10d:8565d5af:74856641
> >> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >> > b2bf0462:e0722254:0e233a72:aa5df4da
> >> >
> >> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
> >> > have the current data for sda3, but that's the only missing UUID).
> >> >
> >> > The create command would therefore be:
> >> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
> >> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
> >> >
> >> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
> >> > array, but you may want to double-check that with a test array on a
> >> > couple of loopback devices first. If not, you'll need to grab the
> >> > latest release and add the --data-offset=2048 parameter to the above
> >> > create command.
> >> >
> >> > You should also follow the instructions for using overlay files at
> >> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> >> > in order to safely test out the above without risking damage to the
> >> > array data.
> >> >
> >> > Once you've run the create, run a "fsck -n" on the filesystem to check
> >> > that the data looks okay. If not, the order or parameters may be
> >> > incorrect - check the --examine output for any differences from the
> >> > original results.
> >> >
> >> Just to double check, would this be the right command to run?
> >>
> >> mdadm --create --assume-clean --level=5 --size=5858385408
> >> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
> >>
> >> Are there any other options I would need to add? Should I specify
> >> --chunk and --size (and if I entered the right size)?
> >>
> > You don't need --assume-clean as there's a missing device, so no scope
> > for rebuilding one of the disks (which is all the flag prevents). It
> > won't do any harm leaving it in though.
> >
> > The size should be the per-device size in kiB (which is half the Used
> > Dev Size value listed in the --examine output, as that's given in
> > 512-byte blocks) and I gave you the correct value above. I'd recommend
> > including this as it will ensure that mdadm isn't calculating the size
> > any different from the version originally used to create the array.
> >
> > The device order you've given is incorrect for either the original
> > device numbering or the numbering you posted as being the most recent.
> > The order I gave above is based on the order as in the latest --examine
> > results you gave. If you've rebooted since then, you'll need to verify
> > the order based on the UUIDs of the devices though (again, the original
> > order should be the one I gave above, based on the device role order in
> > your original --examine output). If you're using different disks, you'll
> > need to be sure which one was mirrored from which original. If you use
> > the incorrect order, you'll get a lot of errors in the "fsck -n" output
> > but, as long as you don't actually write to the array, it shouldn't
> > cause any data corruption as only the metadata will be overwritten.
> >
> > There shouldn't be any need to specify the chunk size, as 512k should be
> > the default value, but I'd probably still stick it in anyway, just to be
> > on the safe side.
> >
> > Similarly with the metadata version - 1.2 is the default (currently
> > anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
> > I'd add it in to be on the safe side.
> >
> >> By the way thanks for the help.
> >>
> >
> > No problem.
> >
> > Cheers,
> > Robin
>
> Here's the adjusted command.
>
> mdadm --create --assume-clean --level=5 --metadata=1.2 --chunk=512
> --size=1952795136 --raid-devices=4 /dev/md0 missing \
> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 \
> 390bd4a2:07a28c01:528ed41e:a9d0fcf0 \
> 4156ab46:bd42c10d:8565d5af:74856641
>
No, the missing should come last - the original --examine info you gave
had info for device roles 0, 1, & 2, so the original failed disk must
have been role 3.
> For the --size option, I'm not quite sure I understood what you tried
> to explain to me. I re-read the manpage and I came up with this 2
> equations:
>
> (My understanding of your explanation) Used Dev size (3905590272)
> divided by 2 = size (1952795136)
> (My understanding from the manpages) Used Dev size (3905590272)
> divided by chunk size (512) = size (7628106)
>
No, the mdadm manual page says that it has to be a multiple of the chunk
size, not that it's given in multiples of the chunk size. It also says
(in the 3,3,1 release anyway) that it's the "Amount (in Kibibytes)".
It's not spelt out that the Used Dev size is in 512-byte blocks, but
that's obvious from the corresponding Gib size given. You can check by
creating some loopback devices and testing creating an array if you
like.
> As for the device, I should order them with the device UUID (as shown
> above) and I replace those UUID with the /dev/sdX3 that returns the
> same device uuid from a mdadm -E command I will currently get? i.e.
> mdadm -E /dev/sdd3 returns a device uuid of
> 92589cc2:9d5ed86c:1467efc2:2e6b7f09 , my first device with be
> /dev/sdd3...?
>
That's correct, yes.
> One last question, after running mdadm --create command, can I run
> mdadm -E and verify the values I get (chunk size, used dev size...)
> match the ones I got from my first mdadm -E command, and if it
> doesn't, to rerun the mdadm --create command to eventually get
> matching values?
>
Yes, the --create command will only overwrite the array metadata, so as
long as your array offset is correct then the actual array data will be
untouched (as the 1.2 superblock is near the start of the device, even a
size error won't damage the data). You'll want to ensure that the chunk
size & dev size match the originals, and that the device role is correct
for the corresponding device UUID.
Once that all matches, you can do the "fsck -f -n" and check that there
are no errors (or only a handful - there may be somea errors after the
array failure anyway).
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-08 16:31 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141208151419.GB8530@cthulhu.home.robinhill.me.uk>
Here's the adjusted command.
mdadm --create --assume-clean --level=5 --metadata=1.2 --chunk=512
--size=1952795136 --raid-devices=4 /dev/md0 missing \
92589cc2:9d5ed86c:1467efc2:2e6b7f09 \
390bd4a2:07a28c01:528ed41e:a9d0fcf0 \
4156ab46:bd42c10d:8565d5af:74856641
For the --size option, I'm not quite sure I understood what you tried
to explain to me. I re-read the manpage and I came up with this 2
equations:
(My understanding of your explanation) Used Dev size (3905590272)
divided by 2 = size (1952795136)
(My understanding from the manpages) Used Dev size (3905590272)
divided by chunk size (512) = size (7628106)
As for the device, I should order them with the device UUID (as shown
above) and I replace those UUID with the /dev/sdX3 that returns the
same device uuid from a mdadm -E command I will currently get? i.e.
mdadm -E /dev/sdd3 returns a device uuid of
92589cc2:9d5ed86c:1467efc2:2e6b7f09 , my first device with be
/dev/sdd3...?
One last question, after running mdadm --create command, can I run
mdadm -E and verify the values I get (chunk size, used dev size...)
match the ones I got from my first mdadm -E command, and if it
doesn't, to rerun the mdadm --create command to eventually get
matching values?
On Mon, Dec 8, 2014 at 10:14 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
>> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
>> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
>> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>> >> >
>> >> >> The long story and what I've done.
>> >> >>
>> >> >> /dev/md0 is assembled with 4 drives
>> >> >> /dev/sda3
>> >> >> /dev/sdb3
>> >> >> /dev/sdc3
>> >> >> /dev/sdd3
>> >> >>
>> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> >> >> the server and until I received a replacement drive.
>> >> >>
>> >> >> This week, I replaced the dying drive with my new drive. Booted into
>> >> >> single user mode and did this:
>> >> >>
>> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> >> >> confirmed the resyncing process. The last time I checked it was up to
>> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
>> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> >> >> everything as is and to go to bed.
>> >> >>
>> >> >> The next day, I shutdown the server and reboot with a live usb distro
>> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
>> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> >> >> looks of this.
>> >> >>
>> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> >> >> was eventually able to read the bad sector on a retry. I followed up
>> >> >> by also cloning with ddrescue, sdb and sdc.
>> >> >>
>> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
>> >> >> Currently running mdadm --assemble --scan, will activate my array, but
>> >> >> all drives are added as spares. Running mdadm --examine on each
>> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
>> >> >> and raid level is -unknown- for some reason. The rest seems fine and
>> >> >> makes sense. I believe I could re-assemble my array if I could define
>> >> >> the raid level and raid devices.
>> >> >>
>> >> >> I wanted to know if there are a way to restore my superblocks from the
>> >> >> examine command I ran at the beginning? If not, what mdadm create
>> >> >> command should I run? Also please let me know if drive ordering is
>> >> >> important, and how I can determine this with the examine output I'll
>> >> >> got?
>> >> >>
>> >> >> Thank you.
>> >> >>
>> >> > Have you tried --assemble --force? You'll need to make sure the array's
>> >> > stopped first, but that's the usual way to get the array back up and
>> >> > running in that sort of situation.
>> >> >
>> >> > If that doesn't work, stop the array again and post:
>> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
>> >> > - any dmesg output corresponding with the above
>> >> > - --examine output for all disks
>> >> > - kernel and mdadm versions
>> >> >
>> >> > Good luck,
>> >> > Robin
>> >
>> >> You'll see from the examine output, raid level and devices aren't
>> >> defined and notice the role of each drives. The examine output (I
>> >> attached 4 files) that I took right after the read error during the
>> >> synching process seems to show a more accurate superblock. Here's also
>> >> the output of mdadm --detail /dev/md0 that I took when I got the first
>> >> error:
>> >>
>> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
>> >> name=runts:0
>> >> spares=1
>> >>
>> >>
>> >> Here's the output of how things currently are:
>> >>
>> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
>> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
>> >> start the array.
>> >>
>> >> dmesg
>> >> [27903.423895] md: md127 stopped.
>> >> [27903.434327] md: bind<sdc3>
>> >> [27903.434767] md: bind<sdd3>
>> >> [27903.434963] md: bind<sdb3>
>> >>
>> >> cat /proc/mdstat
>> >> root@ubuntu:~# cat /proc/mdstat
>> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
>> >> [raid1] [raid10]
>> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
>> >> 5858387208 blocks super 1.2
>> >>
>> >> mdadm --examine /dev/sd[bcd]3
>> >> /dev/sdb3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0
>> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> Raid Level : -unknown-
>> >> Raid Devices : 0
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : active
>> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >>
>> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> Checksum : 5e8cfc9a - correct
>> >> Events : 1
>> >>
>> >>
>> >> Device Role : spare
>> >> Array State : ('A' == active, '.' == missing)
>> >> /dev/sdc3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0
>> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> Raid Level : -unknown-
>> >> Raid Devices : 0
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : active
>> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >>
>> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> Checksum : f69518c - correct
>> >> Events : 1
>> >>
>> >>
>> >> Device Role : spare
>> >> Array State : ('A' == active, '.' == missing)
>> >> /dev/sdd3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0
>> >> Creation Time : Tue Jul 26 03:27:39 2011
>> >> Raid Level : -unknown-
>> >> Raid Devices : 0
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : active
>> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >>
>> >> Update Time : Sat Dec 6 12:46:40 2014
>> >> Checksum : 571ad2bd - correct
>> >> Events : 1
>> >>
>> >>
>> >> Device Role : spare
>> >> Array State : ('A' == active, '.' == missing)
>> >>
>> >> and finally kernel and mdadm versions:
>> >>
>> >> uname -a
>> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
>> >> 2012 i686 i686 i386 GNU/Linux
>> >>
>> >> mdadm -V
>> >> mdadm - v3.2.3 - 23rd December 2011
>> >
>> > The missing data looks similar to a bug fixed a couple of years ago
>> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
>> > don't match and the missing data is somewhat different - it may be that
>> > the relevant patches were backported to the vendor kernel you're using.
>> >
>> > With that data missing there's no way to assemble though, so a re-create
>> > is required in this case (it's a last resort, but I don't see any other
>> > option).
>> >
>> >> /dev/sda3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0 (local to host runts)
>> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> Raid Level : raid5
>> >> Raid Devices : 4
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : clean
>> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>> >>
>> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> Checksum : 5ed5b898 - correct
>> >> Events : 3925676
>> >>
>> >> Layout : left-symmetric
>> >> Chunk Size : 512K
>> >>
>> >> Device Role : spare
>> >> Array State : A.A. ('A' == active, '.' == missing)
>> >
>> >> /dev/sdb3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0 (local to host runts)
>> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> Raid Level : raid5
>> >> Raid Devices : 4
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : clean
>> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> >>
>> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> Checksum : 57638ebb - correct
>> >> Events : 3925676
>> >>
>> >> Layout : left-symmetric
>> >> Chunk Size : 512K
>> >>
>> >> Device Role : Active device 0
>> >> Array State : A.A. ('A' == active, '.' == missing)
>> >
>> >> /dev/sdc3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0 (local to host runts)
>> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> Raid Level : raid5
>> >> Raid Devices : 4
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : clean
>> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> >>
>> >> Update Time : Tue Dec 2 23:15:37 2014
>> >> Checksum : fb20d8a - correct
>> >> Events : 3925676
>> >>
>> >> Layout : left-symmetric
>> >> Chunk Size : 512K
>> >>
>> >> Device Role : Active device 2
>> >> Array State : A.A. ('A' == active, '.' == missing)
>> >
>> >> /dev/sdd3:
>> >> Magic : a92b4efc
>> >> Version : 1.2
>> >> Feature Map : 0x0
>> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> >> Name : runts:0 (local to host runts)
>> >> Creation Time : Mon Jul 25 23:27:39 2011
>> >> Raid Level : raid5
>> >> Raid Devices : 4
>> >>
>> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> >> Data Offset : 2048 sectors
>> >> Super Offset : 8 sectors
>> >> State : clean
>> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>> >>
>> >> Update Time : Tue Dec 2 23:14:03 2014
>> >> Checksum : a126853f - correct
>> >> Events : 3925672
>> >>
>> >> Layout : left-symmetric
>> >> Chunk Size : 512K
>> >>
>> >> Device Role : Active device 1
>> >> Array State : AAAA ('A' == active, '.' == missing)
>> >
>> > At least you have the previous data anyway, which should allow
>> > reconstruction of the array. The device names have changed between your
>> > two reports though, so I'd advise double-checking which is which before
>> > proceeding.
>> >
>> > The reports indicate that the original array order (based on the device
>> > role field) for the four devices was (using device UUIDs as they're
>> > consistent):
>> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>> > 4156ab46:bd42c10d:8565d5af:74856641
>> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>> > b2bf0462:e0722254:0e233a72:aa5df4da
>> >
>> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
>> > have the current data for sda3, but that's the only missing UUID).
>> >
>> > The create command would therefore be:
>> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
>> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
>> >
>> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
>> > array, but you may want to double-check that with a test array on a
>> > couple of loopback devices first. If not, you'll need to grab the
>> > latest release and add the --data-offset=2048 parameter to the above
>> > create command.
>> >
>> > You should also follow the instructions for using overlay files at
>> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
>> > in order to safely test out the above without risking damage to the
>> > array data.
>> >
>> > Once you've run the create, run a "fsck -n" on the filesystem to check
>> > that the data looks okay. If not, the order or parameters may be
>> > incorrect - check the --examine output for any differences from the
>> > original results.
>> >
>> Just to double check, would this be the right command to run?
>>
>> mdadm --create --assume-clean --level=5 --size=5858385408
>> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
>>
>> Are there any other options I would need to add? Should I specify
>> --chunk and --size (and if I entered the right size)?
>>
> You don't need --assume-clean as there's a missing device, so no scope
> for rebuilding one of the disks (which is all the flag prevents). It
> won't do any harm leaving it in though.
>
> The size should be the per-device size in kiB (which is half the Used
> Dev Size value listed in the --examine output, as that's given in
> 512-byte blocks) and I gave you the correct value above. I'd recommend
> including this as it will ensure that mdadm isn't calculating the size
> any different from the version originally used to create the array.
>
> The device order you've given is incorrect for either the original
> device numbering or the numbering you posted as being the most recent.
> The order I gave above is based on the order as in the latest --examine
> results you gave. If you've rebooted since then, you'll need to verify
> the order based on the UUIDs of the devices though (again, the original
> order should be the one I gave above, based on the device role order in
> your original --examine output). If you're using different disks, you'll
> need to be sure which one was mirrored from which original. If you use
> the incorrect order, you'll get a lot of errors in the "fsck -n" output
> but, as long as you don't actually write to the array, it shouldn't
> cause any data corruption as only the metadata will be overwritten.
>
> There shouldn't be any need to specify the chunk size, as 512k should be
> the default value, but I'd probably still stick it in anyway, just to be
> on the safe side.
>
> Similarly with the metadata version - 1.2 is the default (currently
> anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
> I'd add it in to be on the safe side.
>
>> By the way thanks for the help.
>>
>
> No problem.
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-08 15:14 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sYfFNaeUNiCFhidGVuRtMOP1XjJ4e=edsXp1bqe-FYBww@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 15651 bytes --]
On Mon Dec 08, 2014 at 09:13:13AM -0500, Emery Guevremont wrote:
> On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> > On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
> >> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> >> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> >> >
> >> >> The long story and what I've done.
> >> >>
> >> >> /dev/md0 is assembled with 4 drives
> >> >> /dev/sda3
> >> >> /dev/sdb3
> >> >> /dev/sdc3
> >> >> /dev/sdd3
> >> >>
> >> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> >> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> >> >> the server and until I received a replacement drive.
> >> >>
> >> >> This week, I replaced the dying drive with my new drive. Booted into
> >> >> single user mode and did this:
> >> >>
> >> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> >> >> confirmed the resyncing process. The last time I checked it was up to
> >> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
> >> >> read error message on /dev/sdd3 (have a pic of it if interested)
> >> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
> >> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> >> >> everything as is and to go to bed.
> >> >>
> >> >> The next day, I shutdown the server and reboot with a live usb distro
> >> >> (Ubuntu rescue remix). After booting into the live distro, a cat
> >> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> >> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> >> >> looks of this.
> >> >>
> >> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> >> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> >> >> was eventually able to read the bad sector on a retry. I followed up
> >> >> by also cloning with ddrescue, sdb and sdc.
> >> >>
> >> >> So now I have cloned copies of sdb, sdc and sdd to work with.
> >> >> Currently running mdadm --assemble --scan, will activate my array, but
> >> >> all drives are added as spares. Running mdadm --examine on each
> >> >> drives, shows the same Array UUID number, but the Raid Devices is 0
> >> >> and raid level is -unknown- for some reason. The rest seems fine and
> >> >> makes sense. I believe I could re-assemble my array if I could define
> >> >> the raid level and raid devices.
> >> >>
> >> >> I wanted to know if there are a way to restore my superblocks from the
> >> >> examine command I ran at the beginning? If not, what mdadm create
> >> >> command should I run? Also please let me know if drive ordering is
> >> >> important, and how I can determine this with the examine output I'll
> >> >> got?
> >> >>
> >> >> Thank you.
> >> >>
> >> > Have you tried --assemble --force? You'll need to make sure the array's
> >> > stopped first, but that's the usual way to get the array back up and
> >> > running in that sort of situation.
> >> >
> >> > If that doesn't work, stop the array again and post:
> >> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
> >> > - any dmesg output corresponding with the above
> >> > - --examine output for all disks
> >> > - kernel and mdadm versions
> >> >
> >> > Good luck,
> >> > Robin
> >
> >> You'll see from the examine output, raid level and devices aren't
> >> defined and notice the role of each drives. The examine output (I
> >> attached 4 files) that I took right after the read error during the
> >> synching process seems to show a more accurate superblock. Here's also
> >> the output of mdadm --detail /dev/md0 that I took when I got the first
> >> error:
> >>
> >> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
> >> name=runts:0
> >> spares=1
> >>
> >>
> >> Here's the output of how things currently are:
> >>
> >> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
> >> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
> >> start the array.
> >>
> >> dmesg
> >> [27903.423895] md: md127 stopped.
> >> [27903.434327] md: bind<sdc3>
> >> [27903.434767] md: bind<sdd3>
> >> [27903.434963] md: bind<sdb3>
> >>
> >> cat /proc/mdstat
> >> root@ubuntu:~# cat /proc/mdstat
> >> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
> >> [raid1] [raid10]
> >> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
> >> 5858387208 blocks super 1.2
> >>
> >> mdadm --examine /dev/sd[bcd]3
> >> /dev/sdb3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0
> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> Raid Level : -unknown-
> >> Raid Devices : 0
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : active
> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >>
> >> Update Time : Sat Dec 6 12:46:40 2014
> >> Checksum : 5e8cfc9a - correct
> >> Events : 1
> >>
> >>
> >> Device Role : spare
> >> Array State : ('A' == active, '.' == missing)
> >> /dev/sdc3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0
> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> Raid Level : -unknown-
> >> Raid Devices : 0
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : active
> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >>
> >> Update Time : Sat Dec 6 12:46:40 2014
> >> Checksum : f69518c - correct
> >> Events : 1
> >>
> >>
> >> Device Role : spare
> >> Array State : ('A' == active, '.' == missing)
> >> /dev/sdd3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0
> >> Creation Time : Tue Jul 26 03:27:39 2011
> >> Raid Level : -unknown-
> >> Raid Devices : 0
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : active
> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >>
> >> Update Time : Sat Dec 6 12:46:40 2014
> >> Checksum : 571ad2bd - correct
> >> Events : 1
> >>
> >>
> >> Device Role : spare
> >> Array State : ('A' == active, '.' == missing)
> >>
> >> and finally kernel and mdadm versions:
> >>
> >> uname -a
> >> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
> >> 2012 i686 i686 i386 GNU/Linux
> >>
> >> mdadm -V
> >> mdadm - v3.2.3 - 23rd December 2011
> >
> > The missing data looks similar to a bug fixed a couple of years ago
> > (http://neil.brown.name/blog/20120615073245), though the kernel versions
> > don't match and the missing data is somewhat different - it may be that
> > the relevant patches were backported to the vendor kernel you're using.
> >
> > With that data missing there's no way to assemble though, so a re-create
> > is required in this case (it's a last resort, but I don't see any other
> > option).
> >
> >> /dev/sda3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0 (local to host runts)
> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : clean
> >> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
> >>
> >> Update Time : Tue Dec 2 23:15:37 2014
> >> Checksum : 5ed5b898 - correct
> >> Events : 3925676
> >>
> >> Layout : left-symmetric
> >> Chunk Size : 512K
> >>
> >> Device Role : spare
> >> Array State : A.A. ('A' == active, '.' == missing)
> >
> >> /dev/sdb3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0 (local to host runts)
> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : clean
> >> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> >>
> >> Update Time : Tue Dec 2 23:15:37 2014
> >> Checksum : 57638ebb - correct
> >> Events : 3925676
> >>
> >> Layout : left-symmetric
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 0
> >> Array State : A.A. ('A' == active, '.' == missing)
> >
> >> /dev/sdc3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0 (local to host runts)
> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : clean
> >> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> >>
> >> Update Time : Tue Dec 2 23:15:37 2014
> >> Checksum : fb20d8a - correct
> >> Events : 3925676
> >>
> >> Layout : left-symmetric
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 2
> >> Array State : A.A. ('A' == active, '.' == missing)
> >
> >> /dev/sdd3:
> >> Magic : a92b4efc
> >> Version : 1.2
> >> Feature Map : 0x0
> >> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> >> Name : runts:0 (local to host runts)
> >> Creation Time : Mon Jul 25 23:27:39 2011
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> >> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> >> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> >> Data Offset : 2048 sectors
> >> Super Offset : 8 sectors
> >> State : clean
> >> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
> >>
> >> Update Time : Tue Dec 2 23:14:03 2014
> >> Checksum : a126853f - correct
> >> Events : 3925672
> >>
> >> Layout : left-symmetric
> >> Chunk Size : 512K
> >>
> >> Device Role : Active device 1
> >> Array State : AAAA ('A' == active, '.' == missing)
> >
> > At least you have the previous data anyway, which should allow
> > reconstruction of the array. The device names have changed between your
> > two reports though, so I'd advise double-checking which is which before
> > proceeding.
> >
> > The reports indicate that the original array order (based on the device
> > role field) for the four devices was (using device UUIDs as they're
> > consistent):
> > 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> > 4156ab46:bd42c10d:8565d5af:74856641
> > 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> > b2bf0462:e0722254:0e233a72:aa5df4da
> >
> > That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
> > have the current data for sda3, but that's the only missing UUID).
> >
> > The create command would therefore be:
> > mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
> > /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
> >
> > mdadm 3.2.3 should use a data offset of 2048, the same as your old
> > array, but you may want to double-check that with a test array on a
> > couple of loopback devices first. If not, you'll need to grab the
> > latest release and add the --data-offset=2048 parameter to the above
> > create command.
> >
> > You should also follow the instructions for using overlay files at
> > https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> > in order to safely test out the above without risking damage to the
> > array data.
> >
> > Once you've run the create, run a "fsck -n" on the filesystem to check
> > that the data looks okay. If not, the order or parameters may be
> > incorrect - check the --examine output for any differences from the
> > original results.
> >
> Just to double check, would this be the right command to run?
>
> mdadm --create --assume-clean --level=5 --size=5858385408
> --raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
>
> Are there any other options I would need to add? Should I specify
> --chunk and --size (and if I entered the right size)?
>
You don't need --assume-clean as there's a missing device, so no scope
for rebuilding one of the disks (which is all the flag prevents). It
won't do any harm leaving it in though.
The size should be the per-device size in kiB (which is half the Used
Dev Size value listed in the --examine output, as that's given in
512-byte blocks) and I gave you the correct value above. I'd recommend
including this as it will ensure that mdadm isn't calculating the size
any different from the version originally used to create the array.
The device order you've given is incorrect for either the original
device numbering or the numbering you posted as being the most recent.
The order I gave above is based on the order as in the latest --examine
results you gave. If you've rebooted since then, you'll need to verify
the order based on the UUIDs of the devices though (again, the original
order should be the one I gave above, based on the device role order in
your original --examine output). If you're using different disks, you'll
need to be sure which one was mirrored from which original. If you use
the incorrect order, you'll get a lot of errors in the "fsck -n" output
but, as long as you don't actually write to the array, it shouldn't
cause any data corruption as only the metadata will be overwritten.
There shouldn't be any need to specify the chunk size, as 512k should be
the default value, but I'd probably still stick it in anyway, just to be
on the safe side.
Similarly with the metadata version - 1.2 is the default (currently
anyway, I'm not certain with 3.2.3), so shouldn't be necessary. Again,
I'd add it in to be on the safe side.
> By the way thanks for the help.
>
No problem.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-08 14:13 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141208094841.GA8530@cthulhu.home.robinhill.me.uk>
Just to double check, would this be the right command to run?
mdadm --create --assume-clean --level=5 --size=5858385408
--raid-devices=4 /dev/md0 missing /dev/sdb3 /dev/sdc3 /dev/sdd3
Are there any other options I would need to add? Should I specify
--chunk and --size (and if I entered the right size)?
By the way thanks for the help.
On Mon, Dec 8, 2014 at 4:48 AM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
>> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>> >
>> >> The long story and what I've done.
>> >>
>> >> /dev/md0 is assembled with 4 drives
>> >> /dev/sda3
>> >> /dev/sdb3
>> >> /dev/sdc3
>> >> /dev/sdd3
>> >>
>> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> >> the server and until I received a replacement drive.
>> >>
>> >> This week, I replaced the dying drive with my new drive. Booted into
>> >> single user mode and did this:
>> >>
>> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> >> confirmed the resyncing process. The last time I checked it was up to
>> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> >> read error message on /dev/sdd3 (have a pic of it if interested)
>> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> >> everything as is and to go to bed.
>> >>
>> >> The next day, I shutdown the server and reboot with a live usb distro
>> >> (Ubuntu rescue remix). After booting into the live distro, a cat
>> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> >> looks of this.
>> >>
>> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> >> was eventually able to read the bad sector on a retry. I followed up
>> >> by also cloning with ddrescue, sdb and sdc.
>> >>
>> >> So now I have cloned copies of sdb, sdc and sdd to work with.
>> >> Currently running mdadm --assemble --scan, will activate my array, but
>> >> all drives are added as spares. Running mdadm --examine on each
>> >> drives, shows the same Array UUID number, but the Raid Devices is 0
>> >> and raid level is -unknown- for some reason. The rest seems fine and
>> >> makes sense. I believe I could re-assemble my array if I could define
>> >> the raid level and raid devices.
>> >>
>> >> I wanted to know if there are a way to restore my superblocks from the
>> >> examine command I ran at the beginning? If not, what mdadm create
>> >> command should I run? Also please let me know if drive ordering is
>> >> important, and how I can determine this with the examine output I'll
>> >> got?
>> >>
>> >> Thank you.
>> >>
>> > Have you tried --assemble --force? You'll need to make sure the array's
>> > stopped first, but that's the usual way to get the array back up and
>> > running in that sort of situation.
>> >
>> > If that doesn't work, stop the array again and post:
>> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
>> > - any dmesg output corresponding with the above
>> > - --examine output for all disks
>> > - kernel and mdadm versions
>> >
>> > Good luck,
>> > Robin
>
>> You'll see from the examine output, raid level and devices aren't
>> defined and notice the role of each drives. The examine output (I
>> attached 4 files) that I took right after the read error during the
>> synching process seems to show a more accurate superblock. Here's also
>> the output of mdadm --detail /dev/md0 that I took when I got the first
>> error:
>>
>> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
>> name=runts:0
>> spares=1
>>
>>
>> Here's the output of how things currently are:
>>
>> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
>> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
>> start the array.
>>
>> dmesg
>> [27903.423895] md: md127 stopped.
>> [27903.434327] md: bind<sdc3>
>> [27903.434767] md: bind<sdd3>
>> [27903.434963] md: bind<sdb3>
>>
>> cat /proc/mdstat
>> root@ubuntu:~# cat /proc/mdstat
>> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
>> [raid1] [raid10]
>> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
>> 5858387208 blocks super 1.2
>>
>> mdadm --examine /dev/sd[bcd]3
>> /dev/sdb3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0
>> Creation Time : Tue Jul 26 03:27:39 2011
>> Raid Level : -unknown-
>> Raid Devices : 0
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : active
>> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>>
>> Update Time : Sat Dec 6 12:46:40 2014
>> Checksum : 5e8cfc9a - correct
>> Events : 1
>>
>>
>> Device Role : spare
>> Array State : ('A' == active, '.' == missing)
>> /dev/sdc3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0
>> Creation Time : Tue Jul 26 03:27:39 2011
>> Raid Level : -unknown-
>> Raid Devices : 0
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : active
>> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>>
>> Update Time : Sat Dec 6 12:46:40 2014
>> Checksum : f69518c - correct
>> Events : 1
>>
>>
>> Device Role : spare
>> Array State : ('A' == active, '.' == missing)
>> /dev/sdd3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0
>> Creation Time : Tue Jul 26 03:27:39 2011
>> Raid Level : -unknown-
>> Raid Devices : 0
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : active
>> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>>
>> Update Time : Sat Dec 6 12:46:40 2014
>> Checksum : 571ad2bd - correct
>> Events : 1
>>
>>
>> Device Role : spare
>> Array State : ('A' == active, '.' == missing)
>>
>> and finally kernel and mdadm versions:
>>
>> uname -a
>> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
>> 2012 i686 i686 i386 GNU/Linux
>>
>> mdadm -V
>> mdadm - v3.2.3 - 23rd December 2011
>
> The missing data looks similar to a bug fixed a couple of years ago
> (http://neil.brown.name/blog/20120615073245), though the kernel versions
> don't match and the missing data is somewhat different - it may be that
> the relevant patches were backported to the vendor kernel you're using.
>
> With that data missing there's no way to assemble though, so a re-create
> is required in this case (it's a last resort, but I don't see any other
> option).
>
>> /dev/sda3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0 (local to host runts)
>> Creation Time : Mon Jul 25 23:27:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>>
>> Update Time : Tue Dec 2 23:15:37 2014
>> Checksum : 5ed5b898 - correct
>> Events : 3925676
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : spare
>> Array State : A.A. ('A' == active, '.' == missing)
>
>> /dev/sdb3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0 (local to host runts)
>> Creation Time : Mon Jul 25 23:27:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>>
>> Update Time : Tue Dec 2 23:15:37 2014
>> Checksum : 57638ebb - correct
>> Events : 3925676
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : A.A. ('A' == active, '.' == missing)
>
>> /dev/sdc3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0 (local to host runts)
>> Creation Time : Mon Jul 25 23:27:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>>
>> Update Time : Tue Dec 2 23:15:37 2014
>> Checksum : fb20d8a - correct
>> Events : 3925676
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : A.A. ('A' == active, '.' == missing)
>
>> /dev/sdd3:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
>> Name : runts:0 (local to host runts)
>> Creation Time : Mon Jul 25 23:27:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
>> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
>> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>>
>> Update Time : Tue Dec 2 23:14:03 2014
>> Checksum : a126853f - correct
>> Events : 3925672
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AAAA ('A' == active, '.' == missing)
>
> At least you have the previous data anyway, which should allow
> reconstruction of the array. The device names have changed between your
> two reports though, so I'd advise double-checking which is which before
> proceeding.
>
> The reports indicate that the original array order (based on the device
> role field) for the four devices was (using device UUIDs as they're
> consistent):
> 92589cc2:9d5ed86c:1467efc2:2e6b7f09
> 4156ab46:bd42c10d:8565d5af:74856641
> 390bd4a2:07a28c01:528ed41e:a9d0fcf0
> b2bf0462:e0722254:0e233a72:aa5df4da
>
> That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
> have the current data for sda3, but that's the only missing UUID).
>
> The create command would therefore be:
> mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
> /dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
>
> mdadm 3.2.3 should use a data offset of 2048, the same as your old
> array, but you may want to double-check that with a test array on a
> couple of loopback devices first. If not, you'll need to grab the
> latest release and add the --data-offset=2048 parameter to the above
> create command.
>
> You should also follow the instructions for using overlay files at
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
> in order to safely test out the above without risking damage to the
> array data.
>
> Once you've run the create, run a "fsck -n" on the filesystem to check
> that the data looks okay. If not, the order or parameters may be
> incorrect - check the --examine output for any differences from the
> original results.
>
> Cheers,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-08 9:48 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sYkWFO0hC-rUHednzy=BWZoBaVS_DhoBedg-viPnTdU-g@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 12439 bytes --]
On Sat Dec 06, 2014 at 03:49:10PM -0500, Emery Guevremont wrote:
> On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> > On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> >
> >> The long story and what I've done.
> >>
> >> /dev/md0 is assembled with 4 drives
> >> /dev/sda3
> >> /dev/sdb3
> >> /dev/sdc3
> >> /dev/sdd3
> >>
> >> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> >> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> >> the server and until I received a replacement drive.
> >>
> >> This week, I replaced the dying drive with my new drive. Booted into
> >> single user mode and did this:
> >>
> >> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> >> confirmed the resyncing process. The last time I checked it was up to
> >> 11%. After a few minutes later, I noticed that the syncing stopped. A
> >> read error message on /dev/sdd3 (have a pic of it if interested)
> >> appear on the console. It appears that /dev/sdd3 might be going bad. A
> >> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> >> everything as is and to go to bed.
> >>
> >> The next day, I shutdown the server and reboot with a live usb distro
> >> (Ubuntu rescue remix). After booting into the live distro, a cat
> >> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> >> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> >> looks of this.
> >>
> >> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> >> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> >> was eventually able to read the bad sector on a retry. I followed up
> >> by also cloning with ddrescue, sdb and sdc.
> >>
> >> So now I have cloned copies of sdb, sdc and sdd to work with.
> >> Currently running mdadm --assemble --scan, will activate my array, but
> >> all drives are added as spares. Running mdadm --examine on each
> >> drives, shows the same Array UUID number, but the Raid Devices is 0
> >> and raid level is -unknown- for some reason. The rest seems fine and
> >> makes sense. I believe I could re-assemble my array if I could define
> >> the raid level and raid devices.
> >>
> >> I wanted to know if there are a way to restore my superblocks from the
> >> examine command I ran at the beginning? If not, what mdadm create
> >> command should I run? Also please let me know if drive ordering is
> >> important, and how I can determine this with the examine output I'll
> >> got?
> >>
> >> Thank you.
> >>
> > Have you tried --assemble --force? You'll need to make sure the array's
> > stopped first, but that's the usual way to get the array back up and
> > running in that sort of situation.
> >
> > If that doesn't work, stop the array again and post:
> > - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
> > - any dmesg output corresponding with the above
> > - --examine output for all disks
> > - kernel and mdadm versions
> >
> > Good luck,
> > Robin
> You'll see from the examine output, raid level and devices aren't
> defined and notice the role of each drives. The examine output (I
> attached 4 files) that I took right after the read error during the
> synching process seems to show a more accurate superblock. Here's also
> the output of mdadm --detail /dev/md0 that I took when I got the first
> error:
>
> ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
> name=runts:0
> spares=1
>
>
> Here's the output of how things currently are:
>
> mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
> mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
> start the array.
>
> dmesg
> [27903.423895] md: md127 stopped.
> [27903.434327] md: bind<sdc3>
> [27903.434767] md: bind<sdd3>
> [27903.434963] md: bind<sdb3>
>
> cat /proc/mdstat
> root@ubuntu:~# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
> [raid1] [raid10]
> md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
> 5858387208 blocks super 1.2
>
> mdadm --examine /dev/sd[bcd]3
> /dev/sdb3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0
> Creation Time : Tue Jul 26 03:27:39 2011
> Raid Level : -unknown-
> Raid Devices : 0
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>
> Update Time : Sat Dec 6 12:46:40 2014
> Checksum : 5e8cfc9a - correct
> Events : 1
>
>
> Device Role : spare
> Array State : ('A' == active, '.' == missing)
> /dev/sdc3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0
> Creation Time : Tue Jul 26 03:27:39 2011
> Raid Level : -unknown-
> Raid Devices : 0
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>
> Update Time : Sat Dec 6 12:46:40 2014
> Checksum : f69518c - correct
> Events : 1
>
>
> Device Role : spare
> Array State : ('A' == active, '.' == missing)
> /dev/sdd3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0
> Creation Time : Tue Jul 26 03:27:39 2011
> Raid Level : -unknown-
> Raid Devices : 0
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>
> Update Time : Sat Dec 6 12:46:40 2014
> Checksum : 571ad2bd - correct
> Events : 1
>
>
> Device Role : spare
> Array State : ('A' == active, '.' == missing)
>
> and finally kernel and mdadm versions:
>
> uname -a
> Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
> 2012 i686 i686 i386 GNU/Linux
>
> mdadm -V
> mdadm - v3.2.3 - 23rd December 2011
The missing data looks similar to a bug fixed a couple of years ago
(http://neil.brown.name/blog/20120615073245), though the kernel versions
don't match and the missing data is somewhat different - it may be that
the relevant patches were backported to the vendor kernel you're using.
With that data missing there's no way to assemble though, so a re-create
is required in this case (it's a last resort, but I don't see any other
option).
> /dev/sda3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0 (local to host runts)
> Creation Time : Mon Jul 25 23:27:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
>
> Update Time : Tue Dec 2 23:15:37 2014
> Checksum : 5ed5b898 - correct
> Events : 3925676
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : spare
> Array State : A.A. ('A' == active, '.' == missing)
> /dev/sdb3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0 (local to host runts)
> Creation Time : Mon Jul 25 23:27:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
>
> Update Time : Tue Dec 2 23:15:37 2014
> Checksum : 57638ebb - correct
> Events : 3925676
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : A.A. ('A' == active, '.' == missing)
> /dev/sdc3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0 (local to host runts)
> Creation Time : Mon Jul 25 23:27:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
>
> Update Time : Tue Dec 2 23:15:37 2014
> Checksum : fb20d8a - correct
> Events : 3925676
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : A.A. ('A' == active, '.' == missing)
> /dev/sdd3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
> Name : runts:0 (local to host runts)
> Creation Time : Mon Jul 25 23:27:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
> Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
> Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
>
> Update Time : Tue Dec 2 23:14:03 2014
> Checksum : a126853f - correct
> Events : 3925672
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing)
At least you have the previous data anyway, which should allow
reconstruction of the array. The device names have changed between your
two reports though, so I'd advise double-checking which is which before
proceeding.
The reports indicate that the original array order (based on the device
role field) for the four devices was (using device UUIDs as they're
consistent):
92589cc2:9d5ed86c:1467efc2:2e6b7f09
4156ab46:bd42c10d:8565d5af:74856641
390bd4a2:07a28c01:528ed41e:a9d0fcf0
b2bf0462:e0722254:0e233a72:aa5df4da
That would give a current device order of sdd3,sda3,sdc3,sdb3 (I don't
have the current data for sda3, but that's the only missing UUID).
The create command would therefore be:
mdadm -C -l 5 -n 4 -c 512 -e 1.2 -z 1952795136 \
/dev/md0 /dev/sdd3 /dev/sda3 /dev/sdc3 missing
mdadm 3.2.3 should use a data offset of 2048, the same as your old
array, but you may want to double-check that with a test array on a
couple of loopback devices first. If not, you'll need to grab the
latest release and add the --data-offset=2048 parameter to the above
create command.
You should also follow the instructions for using overlay files at
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
in order to safely test out the above without risking damage to the
array data.
Once you've run the create, run a "fsck -n" on the filesystem to check
that the data looks okay. If not, the order or parameters may be
incorrect - check the --examine output for any differences from the
original results.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-06 20:49 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
In-Reply-To: <20141206185600.GA2343@cthulhu.home.robinhill.me.uk>
[-- Attachment #1: Type: text/plain, Size: 6166 bytes --]
You'll see from the examine output, raid level and devices aren't
defined and notice the role of each drives. The examine output (I
attached 4 files) that I took right after the read error during the
synching process seems to show a more accurate superblock. Here's also
the output of mdadm --detail /dev/md0 that I took when I got the first
error:
ARRAY /dev/md/0 metadata=1.2 UUID=cf9db8fa:0c2bb553:46865912:704cceae
name=runts:0
spares=1
Here's the output of how things currently are:
mdadm --assemble --force /dev/md127 /dev/sdb3 /dev/sdc3 /dev/sdd3
mdadm: /dev/md127 assembled from 0 drives and 3 spares - not enough to
start the array.
dmesg
[27903.423895] md: md127 stopped.
[27903.434327] md: bind<sdc3>
[27903.434767] md: bind<sdd3>
[27903.434963] md: bind<sdb3>
cat /proc/mdstat
root@ubuntu:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md127 : inactive sdb3[4](S) sdd3[0](S) sdc3[5](S)
5858387208 blocks super 1.2
mdadm --examine /dev/sd[bcd]3
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0
Creation Time : Tue Jul 26 03:27:39 2011
Raid Level : -unknown-
Raid Devices : 0
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
Update Time : Sat Dec 6 12:46:40 2014
Checksum : 5e8cfc9a - correct
Events : 1
Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0
Creation Time : Tue Jul 26 03:27:39 2011
Raid Level : -unknown-
Raid Devices : 0
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
Update Time : Sat Dec 6 12:46:40 2014
Checksum : f69518c - correct
Events : 1
Device Role : spare
Array State : ('A' == active, '.' == missing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0
Creation Time : Tue Jul 26 03:27:39 2011
Raid Level : -unknown-
Raid Devices : 0
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
Update Time : Sat Dec 6 12:46:40 2014
Checksum : 571ad2bd - correct
Events : 1
Device Role : spare
Array State : ('A' == active, '.' == missing)
and finally kernel and mdadm versions:
uname -a
Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:41:14 UTC
2012 i686 i686 i386 GNU/Linux
mdadm -V
mdadm - v3.2.3 - 23rd December 2011
On Sat, Dec 6, 2014 at 1:56 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
>
>> The long story and what I've done.
>>
>> /dev/md0 is assembled with 4 drives
>> /dev/sda3
>> /dev/sdb3
>> /dev/sdc3
>> /dev/sdd3
>>
>> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
>> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
>> the server and until I received a replacement drive.
>>
>> This week, I replaced the dying drive with my new drive. Booted into
>> single user mode and did this:
>>
>> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
>> confirmed the resyncing process. The last time I checked it was up to
>> 11%. After a few minutes later, I noticed that the syncing stopped. A
>> read error message on /dev/sdd3 (have a pic of it if interested)
>> appear on the console. It appears that /dev/sdd3 might be going bad. A
>> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
>> everything as is and to go to bed.
>>
>> The next day, I shutdown the server and reboot with a live usb distro
>> (Ubuntu rescue remix). After booting into the live distro, a cat
>> /proc/mdstat showed that my /dev/md0 was detected but all drives had
>> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
>> looks of this.
>>
>> I ran ddrescue to copy /dev/sdd onto my new replacement disk
>> (/dev/sda). Everything, worked, ddrescue got only one read error, but
>> was eventually able to read the bad sector on a retry. I followed up
>> by also cloning with ddrescue, sdb and sdc.
>>
>> So now I have cloned copies of sdb, sdc and sdd to work with.
>> Currently running mdadm --assemble --scan, will activate my array, but
>> all drives are added as spares. Running mdadm --examine on each
>> drives, shows the same Array UUID number, but the Raid Devices is 0
>> and raid level is -unknown- for some reason. The rest seems fine and
>> makes sense. I believe I could re-assemble my array if I could define
>> the raid level and raid devices.
>>
>> I wanted to know if there are a way to restore my superblocks from the
>> examine command I ran at the beginning? If not, what mdadm create
>> command should I run? Also please let me know if drive ordering is
>> important, and how I can determine this with the examine output I'll
>> got?
>>
>> Thank you.
>>
> Have you tried --assemble --force? You'll need to make sure the array's
> stopped first, but that's the usual way to get the array back up and
> running in that sort of situation.
>
> If that doesn't work, stop the array again and post:
> - the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
> - any dmesg output corresponding with the above
> - --examine output for all disks
> - kernel and mdadm versions
>
> Good luck,
> Robin
> --
> ___
> ( ' } | Robin Hill <robin@robinhill.me.uk> |
> / / ) | Little Jim says .... |
> // !! | "He fallen in de water !!" |
[-- Attachment #2: sda3.examine --]
[-- Type: application/octet-stream, Size: 814 bytes --]
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0 (local to host runts)
Creation Time : Mon Jul 25 23:27:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b2bf0462:e0722254:0e233a72:aa5df4da
Update Time : Tue Dec 2 23:15:37 2014
Checksum : 5ed5b898 - correct
Events : 3925676
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : A.A. ('A' == active, '.' == missing)
[-- Attachment #3: sdb3.examine --]
[-- Type: application/octet-stream, Size: 824 bytes --]
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0 (local to host runts)
Creation Time : Mon Jul 25 23:27:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 92589cc2:9d5ed86c:1467efc2:2e6b7f09
Update Time : Tue Dec 2 23:15:37 2014
Checksum : 57638ebb - correct
Events : 3925676
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : A.A. ('A' == active, '.' == missing)
[-- Attachment #4: sdc3.examine --]
[-- Type: application/octet-stream, Size: 823 bytes --]
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0 (local to host runts)
Creation Time : Mon Jul 25 23:27:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 390bd4a2:07a28c01:528ed41e:a9d0fcf0
Update Time : Tue Dec 2 23:15:37 2014
Checksum : fb20d8a - correct
Events : 3925676
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.A. ('A' == active, '.' == missing)
[-- Attachment #5: sdd3.examine --]
[-- Type: application/octet-stream, Size: 824 bytes --]
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : cf9db8fa:0c2bb553:46865912:704cceae
Name : runts:0 (local to host runts)
Creation Time : Mon Jul 25 23:27:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3905591472 (1862.33 GiB 1999.66 GB)
Array Size : 5858385408 (5586.99 GiB 5998.99 GB)
Used Dev Size : 3905590272 (1862.33 GiB 1999.66 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4156ab46:bd42c10d:8565d5af:74856641
Update Time : Tue Dec 2 23:14:03 2014
Checksum : a126853f - correct
Events : 3925672
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
^ permalink raw reply
* Re: On RAID5 read error during syncing - array .A.A
From: Robin Hill @ 2014-12-06 18:56 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sZoinCMbp7wrjYaQ=XgCOiBWZBLHkMsV_g+nUyM9pc7CQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3002 bytes --]
On Sat Dec 06, 2014 at 01:35:50pm -0500, Emery Guevremont wrote:
> The long story and what I've done.
>
> /dev/md0 is assembled with 4 drives
> /dev/sda3
> /dev/sdb3
> /dev/sdc3
> /dev/sdd3
>
> 2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
> _UUU. smarctl also confirmed that the drive was dying. So I shutdown
> the server and until I received a replacement drive.
>
> This week, I replaced the dying drive with my new drive. Booted into
> single user mode and did this:
>
> mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
> confirmed the resyncing process. The last time I checked it was up to
> 11%. After a few minutes later, I noticed that the syncing stopped. A
> read error message on /dev/sdd3 (have a pic of it if interested)
> appear on the console. It appears that /dev/sdd3 might be going bad. A
> cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
> everything as is and to go to bed.
>
> The next day, I shutdown the server and reboot with a live usb distro
> (Ubuntu rescue remix). After booting into the live distro, a cat
> /proc/mdstat showed that my /dev/md0 was detected but all drives had
> an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
> looks of this.
>
> I ran ddrescue to copy /dev/sdd onto my new replacement disk
> (/dev/sda). Everything, worked, ddrescue got only one read error, but
> was eventually able to read the bad sector on a retry. I followed up
> by also cloning with ddrescue, sdb and sdc.
>
> So now I have cloned copies of sdb, sdc and sdd to work with.
> Currently running mdadm --assemble --scan, will activate my array, but
> all drives are added as spares. Running mdadm --examine on each
> drives, shows the same Array UUID number, but the Raid Devices is 0
> and raid level is -unknown- for some reason. The rest seems fine and
> makes sense. I believe I could re-assemble my array if I could define
> the raid level and raid devices.
>
> I wanted to know if there are a way to restore my superblocks from the
> examine command I ran at the beginning? If not, what mdadm create
> command should I run? Also please let me know if drive ordering is
> important, and how I can determine this with the examine output I'll
> got?
>
> Thank you.
>
Have you tried --assemble --force? You'll need to make sure the array's
stopped first, but that's the usual way to get the array back up and
running in that sort of situation.
If that doesn't work, stop the array again and post:
- the output from mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcd]3
- any dmesg output corresponding with the above
- --examine output for all disks
- kernel and mdadm versions
Good luck,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-06 18:35 UTC (permalink / raw)
To: linux-raid
The long story and what I've done.
/dev/md0 is assembled with 4 drives
/dev/sda3
/dev/sdb3
/dev/sdc3
/dev/sdd3
2 weeks ago, mdadm marked /dev/sda3 as failed. cat /proc/mdstat showed
_UUU. smarctl also confirmed that the drive was dying. So I shutdown
the server and until I received a replacement drive.
This week, I replaced the dying drive with my new drive. Booted into
single user mode and did this:
mdadm --manage /dev/md0 --add /dev/sda3 a cat of /proc/mdstat
confirmed the resyncing process. The last time I checked it was up to
11%. After a few minutes later, I noticed that the syncing stopped. A
read error message on /dev/sdd3 (have a pic of it if interested)
appear on the console. It appears that /dev/sdd3 might be going bad. A
cat /proc/mdstat showed _U_U. Now I panic, and decide to leave
everything as is and to go to bed.
The next day, I shutdown the server and reboot with a live usb distro
(Ubuntu rescue remix). After booting into the live distro, a cat
/proc/mdstat showed that my /dev/md0 was detected but all drives had
an (S) next to it. i.e. /dev/sda3 (S)... Naturally I don't like the
looks of this.
I ran ddrescue to copy /dev/sdd onto my new replacement disk
(/dev/sda). Everything, worked, ddrescue got only one read error, but
was eventually able to read the bad sector on a retry. I followed up
by also cloning with ddrescue, sdb and sdc.
So now I have cloned copies of sdb, sdc and sdd to work with.
Currently running mdadm --assemble --scan, will activate my array, but
all drives are added as spares. Running mdadm --examine on each
drives, shows the same Array UUID number, but the Raid Devices is 0
and raid level is -unknown- for some reason. The rest seems fine and
makes sense. I believe I could re-assemble my array if I could define
the raid level and raid devices.
I wanted to know if there are a way to restore my superblocks from the
examine command I ran at the beginning? If not, what mdadm create
command should I run? Also please let me know if drive ordering is
important, and how I can determine this with the examine output I'll
got?
Thank you.
^ permalink raw reply
* Re: Disks never stop spinning
From: P. Gautschi @ 2014-12-06 15:16 UTC (permalink / raw)
To: Phillip Susi; +Cc: Peter Grandi, Linux RAID
In-Reply-To: <5480B726.8070103@ubuntu.com>
> Are you running udisks? If so then that is the culprit. It polls the
> drive's SMART stats every 10 minutes unless the drive is already
> asleep, so if the timeout is>= 10 minutes ( which it seems WD drives
> refuse to go any lower anyhow ), then the drive will never go to
> standby. I posted some patches to fix this a while back on the
> devkit-devel mailing list but I think the udisks maintainer has still
> not gotten around to applying them.
This is indeed the cause in my case. There are the two problems that
overlap: if a shorter time then 10 minutes is set as a standby time,
the WD disk rounds this up to 10 minutes.
And Kubuntu is using udisksd. It is started during a KDE login and
never stopped when logging out.
Is it possible to configure udisks to not do any polling or at a much
longer interval? Or is it possible to safely disable udisksd somehow?
I'm still looking for a practical solution.
regards
Patrick
^ permalink raw reply
* Re: Multiple arrays out of devices belonging to single array
From: NeilBrown @ 2014-12-05 11:17 UTC (permalink / raw)
To: Michael Tokarev; +Cc: linux-raid
In-Reply-To: <54817A83.6020000@msgid.tls.msk.ru>
[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]
On Fri, 05 Dec 2014 12:27:31 +0300 Michael Tokarev <mjt@tls.msk.ru> wrote:
> Hello.
>
> We in Debian have 2 rather similar bugreports which are somewhat old:
> http://bugs.debian.org/599352
> http://bugs.debian.org/694513
> Last bugreport is against mdadm-3.2.5, so not THAT old.
>
> Basically, it smells like there's a possibility for mdadm to assemble
> more than one (degraded) array out of components of the same array.
>
> So I wonder if mdadm or kernel have some protection of this situation
> from happening. Something like looking at already active/assembled
> devices to see if the same UUID is already used before trying to
> assemble another array.
>
> It might be better done in kernel because this way the process might
> be freee from races when two mdadm instances tries to do the work in
> parallel.
>
> Are these bugs for real?
Certainly are.
Fix in 3.3 I think.
commit 0431869cec4c673309d9aa30a2df4b778bc0bd24
Author: NeilBrown <neilb@suse.de>
Date: Wed Oct 10 18:27:32 2012 +1100
Fix up interactions between --assemble and --incremental
is one part of the fix. I cannot quickly find another patch which obviously
relates to the symptoms described, but my vague memories suggest there was
more than one patch.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Bug#763917: mdadm: rounding errors in human_size()
From: Michael Tokarev @ 2014-12-05 10:28 UTC (permalink / raw)
To: 763917; +Cc: linux-raid
In-Reply-To: <20141003182454.GA9120@goneko.de>
Neil, will you take the patch in this bugreport for the next version?
http://bugs.debian.org/763917.
Thanks,
/mjt
On Fri, 3 Oct 2014 20:24:54 +0200 Jan Echternach <jan@goneko.de> wrote:
> Package: mdadm
> Version: 3.3.2-1
> Severity: minor
> Tags: patch
>
>
> While setting up a new system, I noticed an incorrect value in the output
> of mdadm --examine:
>
> Avail Dev Size : 2095080 (1023.16 MiB 1072.68 MB)
>
> The 1023.16 MiB were quite irritating because the underlying partition has
> a size of exactly 1023 MiB.
>
> The number of sectors seems plausible: 2095080 sectors * 512 bytes/sector
> are 1022.988 MiB or 1072.681 MB. I looked into the code and found an
> inaccuracy in human_size() and human_size_brief(). The formula used for
> the MiB value is essentially
>
> long cMiB = (bytes / ( (1LL<<20) / 200LL ) +1) /2;
>
> but (1LL<<20) / 200LL is not an integer. It's rounded down and cMiB becomes
> too large. The quick fix would have been multiplying by 200 before dividing
> by 1<<20, but that might cause integer overflows in the GiB case.
>
> The following patch uses a more complicated formula that computes the
> fractional portion separately from the integer portion. It also changes some
> longs to long longs to eliminate a different cause of integer overflows.
>
>
> --- mdadm-3.3.2/util.c 2014-10-03 19:06:51.000000000 +0200
> +++ mdadm-3.3.2/util.c 2014-10-03 19:08:06.000000000 +0200
> @@ -671,15 +671,17 @@
> if (bytes < 5000*1024)
> buf[0] = 0;
> else if (bytes < 2*1024LL*1024LL*1024LL) {
> - long cMiB = (bytes / ( (1LL<<20) / 200LL ) +1) /2;
> + long cMiB = bytes / (1LL<<20) * 100
> + + ((bytes % (1LL<<20)) * 200 / (1LL<<20) +1) /2;
> long cMB = (bytes / ( 1000000LL / 200LL ) +1) /2;
> snprintf(buf, sizeof(buf), " (%ld.%02ld MiB %ld.%02ld MB)",
> cMiB/100 , cMiB % 100,
> cMB/100, cMB % 100);
> } else {
> - long cGiB = (bytes / ( (1LL<<30) / 200LL ) +1) /2;
> - long cGB = (bytes / (1000000000LL/200LL ) +1) /2;
> - snprintf(buf, sizeof(buf), " (%ld.%02ld GiB %ld.%02ld GB)",
> + long long cGiB = bytes / (1LL<<30) * 100
> + + ((bytes % (1LL<<30)) * 200 / (1LL<<30) +1) /2;
> + long long cGB = (bytes / (1000000000LL/200LL ) +1) /2;
> + snprintf(buf, sizeof(buf), " (%lld.%02lld GiB %lld.%02lld GB)",
> cGiB/100 , cGiB % 100,
> cGB/100, cGB % 100);
> }
> @@ -706,12 +708,14 @@
> buf[0] = 0;
> else if (prefix == IEC) {
> if (bytes < 2*1024LL*1024LL*1024LL) {
> - long cMiB = (bytes / ( (1LL<<20) / 200LL ) +1) /2;
> + long cMiB = bytes / (1LL<<20) * 100
^ permalink raw reply
* Multiple arrays out of devices belonging to single array
From: Michael Tokarev @ 2014-12-05 9:27 UTC (permalink / raw)
To: linux-raid
Hello.
We in Debian have 2 rather similar bugreports which are somewhat old:
http://bugs.debian.org/599352
http://bugs.debian.org/694513
Last bugreport is against mdadm-3.2.5, so not THAT old.
Basically, it smells like there's a possibility for mdadm to assemble
more than one (degraded) array out of components of the same array.
So I wonder if mdadm or kernel have some protection of this situation
from happening. Something like looking at already active/assembled
devices to see if the same UUID is already used before trying to
assemble another array.
It might be better done in kernel because this way the process might
be freee from races when two mdadm instances tries to do the work in
parallel.
Are these bugs for real?
Thanks,
/mjt
^ permalink raw reply
* Re: mdadm creates corrupt superblock
From: Hans Kraus @ 2014-12-04 20:59 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux-RAID
In-Reply-To: <20141125111233.22cd9ca1@notabene.brown>
Hi Neil,
finally I found the culprit: the SATA interface (an addidtional card)
was it. I rewired the whole system and connected the drives to different
interfaces. The error wandered with the interface.
If someone asks: "why all the hassle?" That is a system for cold data,
consisting of an older motherboard, not otherwise used SATA interface
cards and small (at least by today's standards) HDs. For that system
I'm looking forward for mdadm with three or more parity drives.
Many thanks for all the support,
Hans
Am 25.11.2014 01:12, schrieb NeilBrown:
> On Thu, 20 Nov 2014 16:58:58 +0100 Hans Kraus <hans@hanswkraus.com> wrote:
>
>> Hi Neil,
>>
>> the partition table (output of 'sfdisk -d /dev/sdh'):
>>
>> # partition table of /dev/sdh
>> unit: sectors
>>
>> /dev/sdh1 : start= 2048, size=1953523120, Id=83
>> /dev/sdh2 : start= 0, size= 0, Id= 0
>> /dev/sdh3 : start= 0, size= 0, Id= 0
>> /dev/sdh4 : start= 0, size= 0, Id= 0
>>
>> Makes that any sense to you?
>
> This tells me that sdh1 starts at sector 2048 in sdh.
> So sector 64 of sdh is not in any partition. So if sector 64 of sdh is
> faulty, that will not affect an array in sdh1....
>
> But you mentioned sdb before ... did the names change (I know that does
> happen).
>
> How did you run "badblocks" and what exactly was the output?
>
> There must be something we are missing here.
>
> NeilBrown
>
>
>>
>> Regards, Hans
>>
>> On 11/19/2014 10:47 PM, NeilBrown wrote:
>>> On Wed, 19 Nov 2014 10:55:05 +0100 Hans Kraus <hans@hanswkraus.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I think I found the problem. 'badblocks' reports sector 64 of the HD as
>>>> errornous. But the HD neither reports a read error nor remaps that
>>>> sector.
>>>>
>>>> Is it possible to parametrize mdadm that it doesn't use that sector?
>>>
>>> I'm surprised that md is using sector 64.
>>> The superblock is at sectors 8,9
>>> The bitmap at 16..24, or something like that
>>> The data is at 262144 onwards.
>>>
>>> But that are offsets in sdb1, maybe the '64' is a sector number in 'sdb'.
>>>
>>> Where does 'sdb1' start in 'sdb'??
>>>
>>> NeilBrown
>>>
>>>
>>>>
>>>> Regards, Hans
>>>>
>>>> Am 29.10.2014 22:11, schrieb Hans Kraus:
>>>>> Hi Neil,
>>>>>
>>>>> many thanks for your response. I rebooted the machine and got new drive
>>>>> assignments. The drive in question is now sdc1. mdadm gave an error with
>>>>> the dump, results below (nothing to report by dmesg) [by the way, I
>>>>> wasn'nt even able to create an array on that drive with
>>>>> 'mdadm --create --level=1 -n 2 /dev/sdc1 missing', same error(s)]:
>>>>> -------------------------------------------------------------------------
>>>>> root@nashorn:~# mdadm -E /dev/sdb1
>>>>> /dev/sdb1:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x1
>>>>> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
>>>>> Name : nashorn:126 (local to host nashorn)
>>>>> Creation Time : Mon Oct 27 15:58:38 2014
>>>>> Raid Level : raid1
>>>>> Raid Devices : 2
>>>>>
>>>>> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
>>>>> Array Size : 976630488 (931.39 GiB 1000.07 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262056 sectors, after=0 sectors
>>>>> State : clean
>>>>> Device UUID : f7a91c46:e4cda0c0:1b770fd7:876634f8
>>>>>
>>>>> Internal Bitmap : 8 sectors from superblock
>>>>> Update Time : Wed Oct 29 18:42:16 2014
>>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>>> Checksum : 939b22d1 - correct
>>>>> Events : 3325
>>>>>
>>>>>
>>>>> Device Role : Active device 0
>>>>> Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
>>>>> root@nashorn:~#
>>>>> -------------------------------------------------------------------------
>>>>> root@nashorn:~# mdadm -E /dev/sdh1
>>>>> /dev/sdh1:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x1
>>>>> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
>>>>> Name : nashorn:126 (local to host nashorn)
>>>>> Creation Time : Mon Oct 27 15:58:38 2014
>>>>> Raid Level : raid1
>>>>> Raid Devices : 2
>>>>>
>>>>> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
>>>>> Array Size : 976630488 (931.39 GiB 1000.07 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262046 sectors, after=0 sectors
>>>>> State : clean
>>>>> Device UUID : 089d1d0e:4d5ca222:dd716acb:0813815c
>>>>>
>>>>> Internal Bitmap : 8 sectors from superblock
>>>>> Update Time : Wed Oct 29 18:42:16 2014
>>>>> Bad Block Log : 512 entries available at offset 82 sectors
>>>>> Checksum : 16454b96 - correct
>>>>> Events : 3325
>>>>>
>>>>>
>>>>> Device Role : Active device 1
>>>>> Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
>>>>> root@nashorn:~#
>>>>> -------------------------------------------------------------------------
>>>>> root@nashorn:~# mdadm -E /dev/sdc1
>>>>> /dev/sdc1:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x1
>>>>> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
>>>>> Name : nashorn:126 (local to host nashorn)
>>>>> Creation Time : Mon Oct 27 15:58:38 2014
>>>>> Raid Level : raid1
>>>>> Raid Devices : 2
>>>>>
>>>>> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
>>>>> Array Size : 0
>>>>> Used Dev Size : 0
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262056 sectors, after=1953260976 sectors
>>>>> State : clean
>>>>> Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
>>>>>
>>>>> Internal Bitmap : 8 sectors from superblock
>>>>> Update Time : Tue Oct 28 11:53:49 2014
>>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>>> Checksum : 6f02d040 - expected 891eedff
>>>>> Events : 0
>>>>>
>>>>>
>>>>> Device Role : spare
>>>>> Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
>>>>> -------------------------------------------------------------------------
>>>>> root@nashorn:~# mkdir -p /tmp/dump
>>>>> root@nashorn:~# mdadm --dump /tmp/dump /dev/sdc1
>>>>> mdadm: Failed to copy metadata from /dev/sdc1 to /tmp/dump/sdc1
>>>>> -------------------------------------------------------------------------
>>>>> Kind regards, Hans
>>>>>
>>>>> Am 28.10.2014 22:00, schrieb NeilBrown:
>>>>>> On Tue, 28 Oct 2014 19:06:20 +0100 Hans Kraus <hans@hanswkraus.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I debugged my problem a bit more: it seems that mdadm creates a defect
>>>>>>> superblock. That's repeatedly happening, but only for one drive. I
>>>>>>> copied zeros to that drive (via ddrescue /dev/zero ...) and the drive
>>>>>>> looks OK. The info I got:
>>>>>>>
>>>>>>
>>>>>> and you send me the metadata of all your devices please?
>>>>>>
>>>>>> mkdir /tmp/dump
>>>>>> mdadm --dump /tmp/dump /dev/sde1
>>>>>> tar czvf /tmp/dump.tgz /tmp/dump
>>>>>>
>>>>>> and then send /tmp/dump.tgz
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> NeilBrown
>>>>>>
>>>>>> =================================================================================================================================================================================================
>>>>>>
>>>>>>> root@nashorn:/home/kraush/work/smartctrl# mdadm --zero-superblock
>>>>>>> /dev/sde1
>>>>>>> mdadm: Unrecognised md component device - /dev/sde1
>>>>>>> root@nashorn:/home/kraush/work/smartctrl# mdadm --add /dev/md126
>>>>>>> /dev/sde1
>>>>>>> mdadm: add new device failed for /dev/sde1 as 4: Invalid argument
>>>>>>> root@nashorn:/home/kraush/work/smartctrl# mdadm -E /dev/sde1
>>>>>>> /dev/sde1:
>>>>>>> Magic : a92b4efc
>>>>>>> Version : 1.2
>>>>>>> Feature Map : 0x1
>>>>>>> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
>>>>>>> Name : nashorn:126 (local to host nashorn)
>>>>>>> Creation Time : Mon Oct 27 15:58:38 2014
>>>>>>> Raid Level : raid1
>>>>>>> Raid Devices : 2
>>>>>>>
>>>>>>> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
>>>>>>> Array Size : 0
>>>>>>> Used Dev Size : 0
>>>>>>> Data Offset : 262144 sectors
>>>>>>> Super Offset : 8 sectors
>>>>>>> Unused Space : before=262056 sectors, after=1953260976 sectors
>>>>>>> State : clean
>>>>>>> Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
>>>>>>>
>>>>>>> Internal Bitmap : 8 sectors from superblock
>>>>>>> Update Time : Tue Oct 28 11:53:49 2014
>>>>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>>>>> Checksum : 6f02d040 - expected 891eedff
>>>>>>> Events : 0
>>>>>>>
>>>>>>>
>>>>>>> Device Role : spare
>>>>>>> Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>> root@nashorn:/home/kraush/work/smartctrl#
>>>>>>> -----------------------------------------------------------------------------------------------------------
>>>>>>>
>>>>>>> root@nashorn:/home/kraush/work/smartctrl# cat /proc/mdstat
>>>>>>> Personalities : [raid1] [raid6] [raid5] [raid4]
>>>>>>> md127 : active (auto-read-only) raid6 sda1[0] sdk1[6](S) sdl2[7](S)
>>>>>>> sdh1[5] sdg1[4] sdf1[3] sdc1[2] sdb1[1]
>>>>>>> 1953017856 blocks super 1.2 level 6, 512k chunk, algorithm 2
>>>>>>> [6/6] [UUUUUU]
>>>>>>> bitmap: 0/4 pages [0KB], 65536KB chunk
>>>>>>>
>>>>>>> md10 : active (auto-read-only) raid1 sdi2[0] sdl3[1]
>>>>>>> 87833408 blocks super 1.2 [2/2] [UU]
>>>>>>> bitmap: 0/1 pages [0KB], 65536KB chunk
>>>>>>>
>>>>>>> md126 : active raid1 sdd1[3] sdj1[2]
>>>>>>> 488254464 blocks super 1.2 [2/2] [UU]
>>>>>>> bitmap: 0/4 pages [0KB], 65536KB chunk
>>>>>>>
>>>>>>> md0 : active raid1 sdl1[4] sdd2[3] sdi1[5]
>>>>>>> 156157824 blocks super 1.2 [3/1] [U__]
>>>>>>> [===================>.] recovery = 95.6% (149290048/156157824)
>>>>>>> finish=8.4min speed=13496K/sec
>>>>>>> bitmap: 1/2 pages [4KB], 65536KB chunk
>>>>>>>
>>>>>>> unused devices: <none>
>>>>>>> root@nashorn:/home/kraush/work/smartctrl#
>>>>>>> -----------------------------------------------------------------------------------------------------------
>>>>>>>
>>>>>>> dmesg:
>>>>>>> [33299.387382] md: invalid superblock checksum on sde1
>>>>>>> [33299.387385] md: sde1 does not have a valid v1.2 superblock, not
>>>>>>> importing!
>>>>>>> [33299.387408] md: md_import_device returned -22
>>>>>>> root@nashorn:/home/kraush/work/smartctrl#
>>>>>>> -----------------------------------------------------------------------------------------------------------
>>>>>>>
>>>>>>> root@nashorn:/home/kraush/work/mdadm# mdadm --version
>>>>>>> mdadm - v3.3-161-gfed12d4 - 21st August 2014
>>>>>>> =================================================================================================================================================================================================
>>>>>>>
>>>>>>>
>>>>>>> Kind regards, Hans
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phil Turmel @ 2014-12-04 20:02 UTC (permalink / raw)
To: Phillip Susi, P. Gautschi; +Cc: Vince, linux-raid
In-Reply-To: <5480B5FF.5050407@ubuntu.com>
Hi Phillip,
On 12/04/2014 02:29 PM, Phillip Susi wrote:
> On 11/7/2014 10:36 PM, Phil Turmel wrote:
>> However, if the device with the bad sector is trying to recover
>> longer than the linux low level driver's timeout, bad things^TM
>> happen. Specifically, the driver resets the SATA (or SCSI)
>> connection and attempts to reconnect. During this brief time, it
>> will not accept further I/O, so the write back of the reconstructed
>> data fails. Then the device has experienced a *write* error, so MD
>> fails the drive. This is the out-of-the-box behavior of
>> consumer-grade drives in raid arrays.
>
> What? During the recovery action ( reset and retry ), a write being
> issued to the drive should just sit in the request queue until after
> the drive finishes being reset; it should not just be failed outright.
It's been a few years since I've directly tested this myself, but that's
what would happen. The window to reject the write might be small, but
it's there (unless the fix is recent).
I'm not an expert on the driver stack, though. YMMV.
Phil
^ permalink raw reply
* Re: Disks never stop spinning
From: Phillip Susi @ 2014-12-04 19:33 UTC (permalink / raw)
To: Patrick Gautschi, Peter Grandi; +Cc: Linux RAID
In-Reply-To: <54657187.6020306@gautschi.net>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 11/13/2014 10:05 PM, Patrick Gautschi wrote:
>> That means something else is waking up those disks. To verify
>> this use something like:
>
> FS are not mounted and arrays stopped. There is nothing that wakes
> them up. If a switch them off manually with hdparm -Y they stay
> off.
Are you running udisks? If so then that is the culprit. It polls the
drive's SMART stats every 10 minutes unless the drive is already
asleep, so if the timeout is >= 10 minutes ( which it seems WD drives
refuse to go any lower anyhow ), then the drive will never go to
standby. I posted some patches to fix this a while back on the
devkit-devel mailing list but I think the udisks maintainer has still
not gotten around to applying them.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUgLcmAAoJENRVrw2cjl5RNkgIAKY2UyBO/I5Tkm9gA6ALn9wx
mcQWsS5gacWitu8ALa/dO7ON75vs+KV0NHjeEc0J+x9jdo02M7gg5PXG3YH1cxzi
mzW78NsSJXwImaZsaCDL4bwYCx6o91wUCuW29Kybvzh+J8u41jW/+5YnX10Mg7jp
o/CMY6O2vgG+MAOd2YIi7LMjgJRWg3my3PPrc4ahaBT4+9L7vHWExy9eRhemQvDN
1e2Q9sDk+uGvG1KCAVG24+2JcNyTEjN53x4FaPkw9EQ0bmUNLrYPksQUl6D/CbGB
mg+Jrcq65cTHp5vpbI3UAd4uKAl2TnoyThIs1cO98spiF6s1MhhRFp+3Vtu8tLg=
=CcWo
-----END PGP SIGNATURE-----
^ permalink raw reply
* Re: Raid5 drive fail during grow and no backup
From: Phillip Susi @ 2014-12-04 19:29 UTC (permalink / raw)
To: Phil Turmel, P. Gautschi; +Cc: Vince, linux-raid
In-Reply-To: <545D8FBA.9090701@turmel.org>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 11/7/2014 10:36 PM, Phil Turmel wrote:
> However, if the device with the bad sector is trying to recover
> longer than the linux low level driver's timeout, bad things^TM
> happen. Specifically, the driver resets the SATA (or SCSI)
> connection and attempts to reconnect. During this brief time, it
> will not accept further I/O, so the write back of the reconstructed
> data fails. Then the device has experienced a *write* error, so MD
> fails the drive. This is the out-of-the-box behavior of
> consumer-grade drives in raid arrays.
What? During the recovery action ( reset and retry ), a write being
issued to the drive should just sit in the request queue until after
the drive finishes being reset; it should not just be failed outright.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJUgLX+AAoJENRVrw2cjl5RHb4H+wWzuTFekQMwIoX7Vov5QjLh
XyEmgbwqgtdcnsbQqtnNiQK0k8KVxQDW3xzWkB30PkOjWMfldES3dRjFXuNbZ0r1
FnJeIYbChFBnfJLp/BqHHOnL5YHD81HvENJ4M/OW6t9SpSiFuOieFe7WTEwHoh5t
t9v/J0+x84CQu1q/AF7FRMkLE1fYhZieAMLTyKhbo5TmMm5XSP8eXumMCz+PXmvV
tVN6rYejSozl1wfwa0l4N9jwkyYWLgbzFRIR7PuQNacywFyLhg0WtIPnqjNV6YuL
rAl5VBFbHEn6BwklgxDWkzSIuOIt2ce6KIE0JZtqTGlDajhSUM+ojTPALLFcmLE=
=4mKS
-----END PGP SIGNATURE-----
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox