* Re: Hot replacing device in RAID 6
From: Keith Keller @ 2014-10-05 18:30 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAAOsTSmn3MDn-a6usUTpmBObwYngvFNNRUXXF6zhmDKyUTHU_w@mail.gmail.com>
On 2014-10-05, Patrik Horník <patrik@dsl.sk> wrote:
>
> - We discussed couple of years ago hot replacing the drive. I see it
> is now supported in mdadm. Is it rock stable and can be used on
> production data?
I have used replace a handful of times with great success and no
problems. I have not had to shut down the array during the rebuild
operation, however. I suspect (but have no direct evidence) that a
clean shutdown should be fine. I used kernel 3.9 from ELRepo for
the replace.
--keith
--
kkeller@wombat.san-francisco.ca.us
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: UEFI and mdadm questions.
From: Phil Turmel @ 2014-10-05 18:18 UTC (permalink / raw)
To: Chris Murphy, Linux-RAID
In-Reply-To: <E82F134B-634B-46B9-A835-846CA89728FB@colorremedies.com>
On 10/03/2014 01:04 AM, Chris Murphy wrote:
>
> On Oct 1, 2014, at 12:33 PM, "Wilson, Jonathan"
> <piercing_male@hotmail.com> wrote:
>
>> From what I can tell with UEFI I need to set up a UEFI partition
>> with a FAT format.
>
> It's a particular kind of FAT, that's defined as EFI FAT, the idea
> being that if the originating FAT ever changes, EFI FAT won't.
>
>>
>> On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M
>> and / Raid 1 40G.
>>
>> Obviously Grub installs to the mbr, and then installs a bit into
>> Biosboot which can read raids, hence it can read and boot from
>> /boot.
>
> BIOSboot applies to GPT disks on BIOS computers, not MBR. On MBR
> disks, the GRUB stage1 code jumps to stage2 code in the MBR gap which
> is the region between the MBR and the first partition's starting
> LBA.
>
>>
>> Further, from what I can tell, into the UEFI partition can go
>> either a kernel & initramfs with UEFI support, or a "loader" that
>> then loads the kernel.
>
> No. An OSLoader is required, it's an EFI application. Its job is to
> load a kernel and initramfs. The kernel and initramfs could be on the
> ESP (EFI System partition) but this is fraught with limitations. The
> expectation is that the kernel and initramfs are on some other
> partition of the same disk. Of course if you're using GRUB it doesn't
> care and will find a kernel/initramfs off another disk also, or even
> off md raid.
An option to consider is to compile a kernel using the EFI stub option,
a pre-set command line, and an embedded initramfs. Then the kernel can
boot directly from the EFI FAT partition with *no bootloader*. The
embedded initramfs can support any raid/lvm/partitioning scheme
what-so-ever.
http://kroah.com/log/blog/2013/09/02/booting-a-self-signed-linux-kernel/
The cryptography is optional if you just want to boot really fast.
If your BIOS can be configured to try multiple boot images, it should be
possible to have true raid fallback without using motherboard or
hardware raid. (Set up md raid1 with metadata v1.0 of multiple copies
of the EFI FAT partition.) I've been meaning to try this....
Phil
^ permalink raw reply
* Re: Hot replacing device in RAID 6
From: Mikael Abrahamsson @ 2014-10-05 16:46 UTC (permalink / raw)
To: Patrik Horník; +Cc: Neil Brown, linux-raid
In-Reply-To: <CAAOsTSmn3MDn-a6usUTpmBObwYngvFNNRUXXF6zhmDKyUTHU_w@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1030 bytes --]
On Sun, 5 Oct 2014, Patrik Horník wrote:
> I want to ask for best solution for replacing device in RAID 6 array.
>
> - We discussed couple of years ago hot replacing the drive. I see it
> is now supported in mdadm. Is it rock stable and can be used on
> production data? I have kernel and mdadm from debian testing, 3.14-2
> and 3.3-2. Were there some important fixes from these versions? I have
> this server on UPS but not on aggregator so there is real danger of
> power outage and need to shutdown the array during operation.
I have successfully used the replace function without ill effects, and I
believe this was with a 3.8.x kernel. The function itself has been around
since 3.3 or so.
I don't want to give you any recommendations, but next time I need to
replace a failing drive that is still operational enough to stay in the
raid, personally I will use the replace function. The code is a few years
old by now and I imagine it has been used quite a lot by now.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Hot replacing device in RAID 6
From: Patrik Horník @ 2014-10-05 16:36 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
Hello Neil,
I want to ask for best solution for replacing device in RAID 6 array.
I want to replace device (actually RAID 0 array of two drives), which
is working OK but there is known problem with firmware of one drive of
that RAID 0 and no solution for it. (It can cause lock up of the
drive, which happened to me.)
I want to minimize risk of problems on any device / devices during
adding new drive, so I am considering other solutions than remove old
and add new.
- We discussed couple of years ago hot replacing the drive. I see it
is now supported in mdadm. Is it rock stable and can be used on
production data? I have kernel and mdadm from debian testing, 3.14-2
and 3.3-2. Were there some important fixes from these versions? I have
this server on UPS but not on aggregator so there is real danger of
power outage and need to shutdown the array during operation.
- I can take the array offline. Is it good solution to simply copy the
old device to new device?
- Or do you suggest as safest solution to remove old device and add new one?
Thanks.
Best regards,
Patrik
^ permalink raw reply
* Re: Raid5 hang in 3.14.19
From: BillStuff @ 2014-10-05 16:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141001085409.41399b13@notabene.brown>
On 09/30/2014 05:54 PM, NeilBrown wrote:
> [removed alot of stuff about raid5 check hanging]
> Thanks for the testing! You have included enough information.
> I didn't really like that 'sync_starting' variable when I wrote the patch,
> but it seemed do the right thing. It doesn't.
>
> If md_check_recovery() runs again immediately after scheduling the sync
> thread to run, it will not have set sync_starting but will find ->sync_thread
> is NULL and so will clear MD_RECOVERY_RUNNING. The next time it runs, that
> flag is still clear and ->sync_thread is not NULL so it will try to stop the
> thread, which deadlocks.
>
> This patch on top of what you have should fix it... but I might end up
> redoing the logic a bit.
>
Neil,
With the second patch, my test has been running well for close to 5 days
now,
but something odd happened yesterday.
It's on a raid1 array (md1) with both ext3 and xfs on LVM, but I suspect
it happened silently on my
raid5 test also.
md1 : active raid1 sdh3[0] sdg3[1]
76959296 blocks [2/2] [UU]
[=>...................] check = 9.7% (7532800/76959296)
finish=108.8min speed=10628K/sec
bitmap: 0/1 pages [0KB], 65536KB chunk
Again this is running kernel builds, read / write loops, and remove/
(re)add / check loops.
At some point while removing a disk from the array, something bad
happened, the errors below
appeared in the log, and the ext3 filesystem remounted readonly. xfs
plowed right on through.
There's no evidence of any read or write errors to the member disks,
either in logs or on the
disks themselves, and the raid1 checks came back with zero mismatches.
fsck for the ext3 fs
complained a lot but the filesystem came back into a useable state and
I've restarted my tests.
Below, dm-3 is xfs, dm-1 is ext3. The interesting stuff happens around
12:27:04. This was
immediately after (or during?) removing sdh3 from md1.
Oct 4 12:26:01 xplane kernel: md: unbind<sdg3>
Oct 4 12:26:01 xplane kernel: md: export_rdev(sdg3)
Oct 4 12:26:11 xplane kernel: md: bind<sdg3>
Oct 4 12:26:11 xplane kernel: RAID1 conf printout:
Oct 4 12:26:11 xplane kernel: --- wd:1 rd:2
Oct 4 12:26:11 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
Oct 4 12:26:11 xplane kernel: disk 1, wo:1, o:1, dev:sdg3
Oct 4 12:26:12 xplane kernel: md: recovery of RAID array md1
Oct 4 12:26:12 xplane kernel: md: minimum _guaranteed_ speed: 10000
KB/sec/disk.
Oct 4 12:26:12 xplane kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for recovery.
Oct 4 12:26:12 xplane kernel: md: using 128k window, over a total of
76959296k.
Oct 4 12:27:03 xplane kernel: md: md1: recovery done.
Oct 4 12:27:03 xplane kernel: RAID1 conf printout:
Oct 4 12:27:03 xplane kernel: --- wd:2 rd:2
Oct 4 12:27:03 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
Oct 4 12:27:03 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 4 12:27:04 xplane kernel: md/raid1:md1: Disk failure on sdh3,
disabling device.
Oct 4 12:27:04 xplane kernel: md/raid1:md1: Operation continuing on 1
devices.
Oct 4 12:27:05 xplane kernel: quiet_error: 912 callbacks suppressed
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331468
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331469
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331470
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331471
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331472
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331473
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331474
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331475
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331476
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Buffer I/O error on device dm-3, logical
block 6331477
Oct 4 12:27:05 xplane kernel: lost page write due to I/O error on dm-3
Oct 4 12:27:05 xplane kernel: Aborting journal on device dm-1.
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_blocks:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
ext3_journal_start_sb: Detected aborted journal
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error: remounting
filesystem read-only
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error:
ext3_journal_start_sb: Detected aborted journal
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
ext3_writeback_write_end: IO failure
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_orphan_add:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in ext3_new_inode:
Journal has aborted
Oct 4 12:27:05 xplane kernel: EXT3-fs (dm-1): error in
ext3_dirty_inode: IO failure
Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
Oct 4 12:27:05 xplane kernel: disk 0, wo:1, o:0, dev:sdh3
Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 4 12:27:05 xplane kernel: RAID1 conf printout:
Oct 4 12:27:05 xplane kernel: --- wd:1 rd:2
Oct 4 12:27:05 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 4 12:27:14 xplane kernel: md: unbind<sdh3>
Oct 4 12:27:14 xplane kernel: md: export_rdev(sdh3)
Oct 4 12:27:24 xplane kernel: md: bind<sdh3>
Oct 4 12:27:24 xplane kernel: RAID1 conf printout:
Oct 4 12:27:24 xplane kernel: --- wd:1 rd:2
Oct 4 12:27:24 xplane kernel: disk 0, wo:1, o:1, dev:sdh3
Oct 4 12:27:24 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 4 12:27:24 xplane kernel: md: recovery of RAID array md1
Oct 4 12:27:24 xplane kernel: md: minimum _guaranteed_ speed: 10000
KB/sec/disk.
Oct 4 12:27:24 xplane kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for recovery.
Oct 4 12:27:24 xplane kernel: md: using 128k window, over a total of
76959296k.
Oct 4 12:27:33 xplane kernel: md: md1: recovery done.
Oct 4 12:27:33 xplane kernel: RAID1 conf printout:
Oct 4 12:27:33 xplane kernel: --- wd:2 rd:2
Oct 4 12:27:33 xplane kernel: disk 0, wo:0, o:1, dev:sdh3
Oct 4 12:27:33 xplane kernel: disk 1, wo:0, o:1, dev:sdg3
Oct 4 12:27:35 xplane kernel: md: data-check of RAID array md1
Oct 4 12:27:35 xplane kernel: md: minimum _guaranteed_ speed: 10000
KB/sec/disk.
Oct 4 12:27:35 xplane kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for data-check.
Oct 4 12:27:35 xplane kernel: md: using 128k window, over a total of
76959296k.
Oct 4 12:29:59 xplane kernel: __journal_remove_journal_head: freeing
b_committed_data
It seems like something got out of sync as the disk was being removed,
but before the remove completed.
Again, this is 3.14.19 with these 8 patches:
md/raid1: intialise start_next_window for READ case to avoid hang
md/raid1: be more cautious where we read-balance during resync.
md/raid1: clean up request counts properly in close_sync()
md/raid1: make sure resync waits for conflicting writes to complete.
md/raid1: Don't use next_resync to determine how far resync has
progressed
md/raid1: update next_resync under resync_lock.
md/raid1: count resync requests in nr_pending.
md/raid1: fix_read_error should act on all non-faulty devices.
and the two patches for the check start hang.
Any ideas on what happened here?
Thanks,
Bill
^ permalink raw reply
* Re: Please advise, strange "not enough to start the array while not clean"
From: Patrik Horník @ 2014-10-05 15:40 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20140922201701.77f5cc66@notabene.brown>
Hello,
so it worked as we expected and it was most probably caused by that bug.
Thanks for the assistance.
Patrik
2014-09-22 12:17 GMT+02:00 NeilBrown <neilb@suse.de>:
> On Mon, 22 Sep 2014 09:20:35 +0200 Patrik Horník <patrik@dsl.sk> wrote:
>
>> Well I browsed through the sources of latest mdadm version at night
>> instead of sleeping :) and was searching how it got clean flag set to
>> 0. And was not sure exactly about that line there and from which
>> device it gets clean state. So the bug was that it can get it from not
>> current device? It makes sense because first device identified by
>> mdadm is old md101 device.
>
> Correct.
>
>>
>> So will it work then if I use 3.3 and somehow dont give it md101
>> device? By stopping it before -A call or by manually specifying other
>> drives? Or you really recommend to build latest version of mdadm?
>
> mdadm -S /dev/mdXX list-of-devices-that-are-working
>
> should start the array for you using 3.3.
> Then
> mdadm /dev/mdXX --re-add /dev/md101
>
> will re-add the device and hopefully do a quick bitmap-based rebuild.
>
>>
>> What is expected behaviour with 3.3.1+? Can it be started with all
>> devices and it should automatically start to recover md101? If so what
>> is best way, to start it without md101 and then use -re-add to add it,
>> start it without md101 and use -add or start it also with md101?
>
> --add should have the same effect as --re-add.
>
> I think mdadm 3.3.1 will just assemble the array without md101 and you then
> have to --re-add that yourself. To get it re-added automatically you needed
> to have "policy action=re-add" or similar in mdadm.conf, and then use
> mdadm -I /dev/devicename
> on each device. That should put the whole array together and re-add anything
> that needs it.
>
> I think.
>
> NeilBrown
>
>
>>
>> Thank you very much.
>>
>> 2014-09-22 8:56 GMT+02:00 NeilBrown <neilb@suse.de>:
>> > On Mon, 22 Sep 2014 08:34:21 +0200 Patrik Horník <patrik@dsl.sk> wrote:
>> >
>> >> - Well what is exact meaning of --no-degraded then? Because I am using
>> >> it also on RAID6 arrays that are missing one drive and mdadm starts
>> >> them. I thought until today that it is against assembling for example
>> >> RAID6 array with missing more than two drives or to be more precise
>> >> array with number of drives it used last time. (I did not look at the
>> >> code what does it exactly. It is mdadm 3.3 on Debian.)
>> >
>> > Sorry, I confused myself.
>> > "--no-degraded" means "Only start the array if all expected devices are
>> > present".
>> > So if the array "knows" that one device is missing, it will start if all
>> > other devices are present. But if it "thinks" that all devices are working,
>> > then it will only start if all the devices ar there.
>> >
>> >>
>> >> - Well array was shutdown cleanly manually by mdadm -S. Cant the not
>> >> clean classification be result of md101 device between find devices or
>> >> result of first two assemble tries?
>> >
>> > If the state still says "Clean" (which it does, thanks), the mdadm should
>> > treat it as 'clean'.
>> >
>> > I think you are probably hitting the bug fixed by
>> >
>> > http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=56bbc588f7f0f3bdd3ec23f02109b427c1d3b8f1
>> >
>> > which is in 3.3.1.
>> >
>> > So a new version of mdadm should fix it.
>> >
>> > NeilBrown
>> >
>> >
>> >
>> >>
>> >> - Anyway as I mentioned superblock on all five devices has clean state. Example:
>> >> /dev/sdk1:
>> >> Magic : XXXXXXX
>> >> Version : 1.2
>> >> Feature Map : 0x1
>> >> Array UUID : XXXXXXXXXXXXXXXXXXXXX
>> >> Name :
>> >> Creation Time : Thu Aug XXXXXXXX
>> >> Raid Level : raid6
>> >> Raid Devices : 6
>> >>
>> >> Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
>> >> Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
>> >> Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
>> >> Data Offset : 262144 sectors
>> >> Super Offset : 8 sectors
>> >> Unused Space : before=262056 sectors, after=911 sectors
>> >> State : clean
>> >> Device UUID : YYYYYYYYYYYYYYYYYYYY
>> >>
>> >> Internal Bitmap : 8 sectors from superblock
>> >> Update Time : Mon Sep 22 02:23:45 2014
>> >> Bad Block Log : 512 entries available at offset 72 sectors
>> >> Checksum : ZZZZZZZZ - correct
>> >> Events : EEEEEE
>> >>
>> >> Layout : left-symmetric
>> >> Chunk Size : 512K
>> >>
>> >> Device Role : Active device 4
>> >> Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>> >>
>> >> - md101 has Events count lower by 16 than others devices.
>> >>
>> >> - Please I need little more assurance what is exact state of array and
>> >> explain why it is behaving as it is behaving, so I can be sure what
>> >> steps are needed and what happens. The data on array is important.
>> >> Patrik Horník
>> >> šéfredaktor www.DSL.sk
>> >> Tel.: +421 905 385 666
>> >> Email: patrik@dsl.sk
>> >>
>> >>
>> >> 2014-09-22 5:19 GMT+02:00 NeilBrown <neilb@suse.de>:
>> >> > On Mon, 22 Sep 2014 04:11:20 +0200 Patrik Horník <patrik@dsl.sk> wrote:
>> >> >
>> >> >> Hello Neil,
>> >> >>
>> >> >> I've got this situation unfamiliar to me on RAID6 array md1 with important data.
>> >> >>
>> >> >> - It is RAID6 with 6 devices, 5 are partitions and 1 is another RAID0
>> >> >> array md101 from two smaller drives. One of the smaller drives froze,
>> >> >> so md101 got kicked out from md1 and marked as faulty in md1. After
>> >> >> while I've stopped md1 without removing md101 from it first. Then I
>> >> >> rebooted and assembled md101.
>> >> >>
>> >> >> - First I tried mdadm -A --no-degraded -u UUID /dev/md1 but got
>> >> >> "mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started."
>> >> >> so I stopped the md1.
>> >> >>
>> >> >> - Second time I started it with -v and got:
>> >> >>
>> >> >> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
>> >> >> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
>> >> >> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
>> >> >> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
>> >> >> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
>> >> >> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
>> >> >> mdadm: added /dev/sdi1 to /dev/md1 as 1
>> >> >> mdadm: added /dev/sdh1 to /dev/md1 as 2
>> >> >> mdadm: added /dev/sde1 to /dev/md1 as 3
>> >> >> mdadm: added /dev/sdk1 to /dev/md1 as 4
>> >> >> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
>> >> >> mdadm: added /dev/sdg1 to /dev/md1 as 0
>> >> >> mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started.
>> >> >>
>> >> >> - On third time I tried without --nodegraded with mdadm -A -v -u UUID
>> >> >> /dev/md1. This is what I've got:
>> >> >>
>> >> >> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
>> >> >> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
>> >> >> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
>> >> >> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
>> >> >> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
>> >> >> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
>> >> >> mdadm: added /dev/sdi1 to /dev/md1 as 1
>> >> >> mdadm: added /dev/sdh1 to /dev/md1 as 2
>> >> >> mdadm: added /dev/sde1 to /dev/md1 as 3
>> >> >> mdadm: added /dev/sdk1 to /dev/md1 as 4
>> >> >> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
>> >> >> mdadm: added /dev/sdg1 to /dev/md1 as 0
>> >> >> mdadm: /dev/md1 assembled from 5 drives - not enough to start the
>> >> >> array while not clean - consider --force.
>> >> >>
>> >> >> Array md1 has bitmap. All drive devices have all same Events, their
>> >> >> state is clean and Device Role is Active device. md101 has active
>> >> >> state and lower Events.
>> >> >>
>> >> >> Is this expected behavior? My theory is that it is caused by md101 and
>> >> >> I should start array md1 without it (by for example stopping md101)
>> >> >> and then re-add it. Is that a case or is it something else?
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> Best regards,
>> >> >>
>> >> >> Patrik
>> >> >
>> >> >
>> >> > The array is clearly degraded as one of the devices failed and hasn't been
>> >> > recovered yet, so using --nodegraded is counter productive, as you
>> >> > discovered.
>> >> >
>> >> > It appears that the array is also marked as 'dirty'. That suggests that it
>> >> > wasn't shut down cleanly.
>> >> > What does "mdadm --examine" of some device show?
>> >> >
>> >> > You probably need to re-assemble the array with --force like it suggests,
>> >> > then add the failed device and let it recover.
>> >> >
>> >> > NeilBrown
>> >> >
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: 2 disk raid 5 failure
From: Jean-Paul Sergent @ 2014-10-05 9:55 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141005205242.31d630da@notabene.brown>
Great, so its mostly media, movies and stuff. So no real tragedy if I
loose it. but I'll go ahead and do both then copy the few essential
files I think I have off of it.
Do you know why version 3.3 had those problems and 3.3.2 works?
Anyways, thanks again.
-JP
On Sun, Oct 5, 2014 at 2:52 AM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 5 Oct 2014 02:38:53 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
> wrote:
>
>> haha, you're awesome, did an apt-get update && apt-get install mdadm.
>> Got version:
>>
>> mdadm - v3.3.2 - 21st August 2014
>>
>> and all is good, re-added drives automatically, threw out the one with
>> the oldest event, which had bad sectors.
>
> Excellent :-)
>
>>
>> Thanks so much.
>>
>> One last question though, the filesystem was XFS. Should I repair the
>> degraded raid first with a spare disk? or should I do an xfs scrub
>> first?
>
> It hardly matters. If another device is going to fail, either action could
> cause it by putting stress on the system. If not, doing both in parallel is
> perfectly safe.
>
> If you have some really really important files, it might make sense to copy
> them off before doing anything else.
> I would probably start the array recovering, then start running the xfs scrub
> tool.
>
> NeilBrown
>
>
>>
>> -JP
>>
>> On Sun, Oct 5, 2014 at 2:21 AM, Jean-Paul Sergent <jpsergent@gmail.com> wrote:
>> > I'm recovering with a liveUSB of debian, the system this raid is on
>> > normally runs fedora 20, I'm not sure what version of mdadm was
>> > running on that system. I can find out if I need to.
>> >
>> >
>> > root@debian:~# mdadm --version
>> > mdadm - v3.3 - 3rd September 2013
>> >
>> >
>> > root@debian:~# mdadm -A /dev/md0 --force -vv /dev/sdb /dev/sdc
>> > /dev/sde /dev/sdf /dev/sdg
>> > mdadm: looking for devices for /dev/md0
>> > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 2.
>> > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
>> > mdadm: /dev/sde is identified as a member of /dev/md0, slot 0.
>> > mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
>> > mdadm: /dev/sdg is identified as a member of /dev/md0, slot 4.
>> > mdadm: added /dev/sdc to /dev/md0 as 1
>> > mdadm: added /dev/sdb to /dev/md0 as 2
>> > mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
>> > mdadm: added /dev/sdg to /dev/md0 as 4 (possibly out of date)
>> > mdadm: added /dev/sde to /dev/md0 as 0
>> > mdadm: /dev/md0 assembled from 3 drives - not enough to start the array.
>> >
>> > Thanks,
>> > -JP
>> >
>> > On Sun, Oct 5, 2014 at 1:41 AM, NeilBrown <neilb@suse.de> wrote:
>> >> On Sat, 4 Oct 2014 22:43:01 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
>> >> wrote:
>> >>
>> >>> Greetings,
>> >>>
>> >>> Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
>> >>> cable. It was a Y splitter and it shorted... it was cheap. I was wondering
>> >>> if there was any chance in getting my data back.
>> >>>
>> >>> Of the 2 disks that blew out, one actually had bad/unreadable sectors on
>> >>> the disk and the other disk seems fine. I have cloned both disks with DD to
>> >>> 2 new disks and forced the one disk to clone even with errors. The
>> >>> remaining 3 disks are in tact. The events number on the array for all 5
>> >>> disks are very close to each other:
>> >>>
>> >>> Events : 201636
>> >>> Events : 201636
>> >>> Events : 201636
>> >>> Events : 201630
>> >>> Events : 201633
>> >>>
>> >>> Which from my reading gives me some hope, but I'm not sure. I have not done
>> >>> "recovering a failed software raid" on the wiki yet, the part about
>> >>> using a loop device to protect your array. I thought I would send a
>> >>> message out to this list first before going down
>> >>> that route.
>> >>>
>> >>> I did try to do a mdadm --force --assemble on the array but is says that it
>> >>> only has 3 disks which isn't enough to start the array. I don't want to do
>> >>> anything else before consulting the mailing list first.
>> >>
>> >> --force --assemble really is what you want. It should work.
>> >> What does
>> >> mdadm -A /dev/md1 --force -vv /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg
>> >>
>> >> report??
>> >> What version of mdadm (mdadm -V) do you have?
>> >>
>> >> NeilBrown
>> >>
>> >>
>> >>>
>> >>> below I have pasted the mdadm --examine from each member drive. Any help would
>> >>> be greatly appreciated.
>> >>>
>> >>> Thanks,
>> >>> -JP
>> >>>
>> >>> /dev/sdb:
>> >>> Magic : a92b4efc
>> >>> Version : 1.2
>> >>> Feature Map : 0x0
>> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> >>> Name : b1ackb0x:1
>> >>> Creation Time : Sun Jan 13 00:01:44 2013
>> >>> Raid Level : raid5
>> >>> Raid Devices : 5
>> >>>
>> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> >>> Data Offset : 2048 sectors
>> >>> Super Offset : 8 sectors
>> >>> Unused Space : before=1968 sectors, after=816 sectors
>> >>> State : clean
>> >>> Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
>> >>>
>> >>> Update Time : Tue Apr 29 02:49:21 2014
>> >>> Checksum : cd29f83c - correct
>> >>> Events : 201636
>> >>>
>> >>> Layout : left-symmetric
>> >>> Chunk Size : 512K
>> >>>
>> >>> Device Role : Active device 2
>> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> >>> /dev/sdc:
>> >>> Magic : a92b4efc
>> >>> Version : 1.2
>> >>> Feature Map : 0x0
>> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> >>> Name : b1ackb0x:1
>> >>> Creation Time : Sun Jan 13 00:01:44 2013
>> >>> Raid Level : raid5
>> >>> Raid Devices : 5
>> >>>
>> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> >>> Data Offset : 2048 sectors
>> >>> Super Offset : 8 sectors
>> >>> Unused Space : before=1968 sectors, after=816 sectors
>> >>> State : clean
>> >>> Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
>> >>>
>> >>> Update Time : Tue Apr 29 02:49:21 2014
>> >>> Checksum : 1e5353af - correct
>> >>> Events : 201636
>> >>>
>> >>> Layout : left-symmetric
>> >>> Chunk Size : 512K
>> >>>
>> >>> Device Role : Active device 1
>> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> >>> /dev/sde:
>> >>> Magic : a92b4efc
>> >>> Version : 1.2
>> >>> Feature Map : 0x0
>> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> >>> Name : b1ackb0x:1
>> >>> Creation Time : Sun Jan 13 00:01:44 2013
>> >>> Raid Level : raid5
>> >>> Raid Devices : 5
>> >>>
>> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> >>> Data Offset : 2048 sectors
>> >>> Super Offset : 8 sectors
>> >>> Unused Space : before=1968 sectors, after=816 sectors
>> >>> State : clean
>> >>> Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
>> >>>
>> >>> Update Time : Tue Apr 29 02:49:21 2014
>> >>> Checksum : 24a3f586 - correct
>> >>> Events : 201636
>> >>>
>> >>> Layout : left-symmetric
>> >>> Chunk Size : 512K
>> >>>
>> >>> Device Role : Active device 0
>> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> >>> /dev/sdf:
>> >>> Magic : a92b4efc
>> >>> Version : 1.2
>> >>> Feature Map : 0x0
>> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> >>> Name : b1ackb0x:1
>> >>> Creation Time : Sun Jan 13 00:01:44 2013
>> >>> Raid Level : raid5
>> >>> Raid Devices : 5
>> >>>
>> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> >>> Data Offset : 2048 sectors
>> >>> Super Offset : 8 sectors
>> >>> Unused Space : before=1968 sectors, after=976752816 sectors
>> >>> State : clean
>> >>> Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
>> >>>
>> >>> Update Time : Tue Apr 29 02:37:38 2014
>> >>> Checksum : 2621f9d5 - correct
>> >>> Events : 201630
>> >>>
>> >>> Layout : left-symmetric
>> >>> Chunk Size : 512K
>> >>>
>> >>> Device Role : Active device 3
>> >>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>> >>> /dev/sdg:
>> >>> Magic : a92b4efc
>> >>> Version : 1.2
>> >>> Feature Map : 0x0
>> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> >>> Name : b1ackb0x:1
>> >>> Creation Time : Sun Jan 13 00:01:44 2013
>> >>> Raid Level : raid5
>> >>> Raid Devices : 5
>> >>>
>> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> >>> Data Offset : 2048 sectors
>> >>> Super Offset : 8 sectors
>> >>> Unused Space : before=1968 sectors, after=976752816 sectors
>> >>> State : clean
>> >>> Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
>> >>>
>> >>> Update Time : Tue Apr 29 02:48:01 2014
>> >>> Checksum : db9e6008 - correct
>> >>> Events : 201633
>> >>>
>> >>> Layout : left-symmetric
>> >>> Chunk Size : 512K
>> >>>
>> >>> Device Role : Active device 4
>> >>> Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
>> >>> --
>> >>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >>> the body of a message to majordomo@vger.kernel.org
>> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: 2 disk raid 5 failure
From: NeilBrown @ 2014-10-05 9:52 UTC (permalink / raw)
To: Jean-Paul Sergent; +Cc: linux-raid
In-Reply-To: <CANuKptnEodSbv5ywzMtBq4TZXeOmdd7g0Awo1SXB=xm9kD=EQg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 9923 bytes --]
On Sun, 5 Oct 2014 02:38:53 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
wrote:
> haha, you're awesome, did an apt-get update && apt-get install mdadm.
> Got version:
>
> mdadm - v3.3.2 - 21st August 2014
>
> and all is good, re-added drives automatically, threw out the one with
> the oldest event, which had bad sectors.
Excellent :-)
>
> Thanks so much.
>
> One last question though, the filesystem was XFS. Should I repair the
> degraded raid first with a spare disk? or should I do an xfs scrub
> first?
It hardly matters. If another device is going to fail, either action could
cause it by putting stress on the system. If not, doing both in parallel is
perfectly safe.
If you have some really really important files, it might make sense to copy
them off before doing anything else.
I would probably start the array recovering, then start running the xfs scrub
tool.
NeilBrown
>
> -JP
>
> On Sun, Oct 5, 2014 at 2:21 AM, Jean-Paul Sergent <jpsergent@gmail.com> wrote:
> > I'm recovering with a liveUSB of debian, the system this raid is on
> > normally runs fedora 20, I'm not sure what version of mdadm was
> > running on that system. I can find out if I need to.
> >
> >
> > root@debian:~# mdadm --version
> > mdadm - v3.3 - 3rd September 2013
> >
> >
> > root@debian:~# mdadm -A /dev/md0 --force -vv /dev/sdb /dev/sdc
> > /dev/sde /dev/sdf /dev/sdg
> > mdadm: looking for devices for /dev/md0
> > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 2.
> > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
> > mdadm: /dev/sde is identified as a member of /dev/md0, slot 0.
> > mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
> > mdadm: /dev/sdg is identified as a member of /dev/md0, slot 4.
> > mdadm: added /dev/sdc to /dev/md0 as 1
> > mdadm: added /dev/sdb to /dev/md0 as 2
> > mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
> > mdadm: added /dev/sdg to /dev/md0 as 4 (possibly out of date)
> > mdadm: added /dev/sde to /dev/md0 as 0
> > mdadm: /dev/md0 assembled from 3 drives - not enough to start the array.
> >
> > Thanks,
> > -JP
> >
> > On Sun, Oct 5, 2014 at 1:41 AM, NeilBrown <neilb@suse.de> wrote:
> >> On Sat, 4 Oct 2014 22:43:01 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
> >> wrote:
> >>
> >>> Greetings,
> >>>
> >>> Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
> >>> cable. It was a Y splitter and it shorted... it was cheap. I was wondering
> >>> if there was any chance in getting my data back.
> >>>
> >>> Of the 2 disks that blew out, one actually had bad/unreadable sectors on
> >>> the disk and the other disk seems fine. I have cloned both disks with DD to
> >>> 2 new disks and forced the one disk to clone even with errors. The
> >>> remaining 3 disks are in tact. The events number on the array for all 5
> >>> disks are very close to each other:
> >>>
> >>> Events : 201636
> >>> Events : 201636
> >>> Events : 201636
> >>> Events : 201630
> >>> Events : 201633
> >>>
> >>> Which from my reading gives me some hope, but I'm not sure. I have not done
> >>> "recovering a failed software raid" on the wiki yet, the part about
> >>> using a loop device to protect your array. I thought I would send a
> >>> message out to this list first before going down
> >>> that route.
> >>>
> >>> I did try to do a mdadm --force --assemble on the array but is says that it
> >>> only has 3 disks which isn't enough to start the array. I don't want to do
> >>> anything else before consulting the mailing list first.
> >>
> >> --force --assemble really is what you want. It should work.
> >> What does
> >> mdadm -A /dev/md1 --force -vv /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg
> >>
> >> report??
> >> What version of mdadm (mdadm -V) do you have?
> >>
> >> NeilBrown
> >>
> >>
> >>>
> >>> below I have pasted the mdadm --examine from each member drive. Any help would
> >>> be greatly appreciated.
> >>>
> >>> Thanks,
> >>> -JP
> >>>
> >>> /dev/sdb:
> >>> Magic : a92b4efc
> >>> Version : 1.2
> >>> Feature Map : 0x0
> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> >>> Name : b1ackb0x:1
> >>> Creation Time : Sun Jan 13 00:01:44 2013
> >>> Raid Level : raid5
> >>> Raid Devices : 5
> >>>
> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> >>> Data Offset : 2048 sectors
> >>> Super Offset : 8 sectors
> >>> Unused Space : before=1968 sectors, after=816 sectors
> >>> State : clean
> >>> Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
> >>>
> >>> Update Time : Tue Apr 29 02:49:21 2014
> >>> Checksum : cd29f83c - correct
> >>> Events : 201636
> >>>
> >>> Layout : left-symmetric
> >>> Chunk Size : 512K
> >>>
> >>> Device Role : Active device 2
> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>> /dev/sdc:
> >>> Magic : a92b4efc
> >>> Version : 1.2
> >>> Feature Map : 0x0
> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> >>> Name : b1ackb0x:1
> >>> Creation Time : Sun Jan 13 00:01:44 2013
> >>> Raid Level : raid5
> >>> Raid Devices : 5
> >>>
> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> >>> Data Offset : 2048 sectors
> >>> Super Offset : 8 sectors
> >>> Unused Space : before=1968 sectors, after=816 sectors
> >>> State : clean
> >>> Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
> >>>
> >>> Update Time : Tue Apr 29 02:49:21 2014
> >>> Checksum : 1e5353af - correct
> >>> Events : 201636
> >>>
> >>> Layout : left-symmetric
> >>> Chunk Size : 512K
> >>>
> >>> Device Role : Active device 1
> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>> /dev/sde:
> >>> Magic : a92b4efc
> >>> Version : 1.2
> >>> Feature Map : 0x0
> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> >>> Name : b1ackb0x:1
> >>> Creation Time : Sun Jan 13 00:01:44 2013
> >>> Raid Level : raid5
> >>> Raid Devices : 5
> >>>
> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> >>> Data Offset : 2048 sectors
> >>> Super Offset : 8 sectors
> >>> Unused Space : before=1968 sectors, after=816 sectors
> >>> State : clean
> >>> Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
> >>>
> >>> Update Time : Tue Apr 29 02:49:21 2014
> >>> Checksum : 24a3f586 - correct
> >>> Events : 201636
> >>>
> >>> Layout : left-symmetric
> >>> Chunk Size : 512K
> >>>
> >>> Device Role : Active device 0
> >>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> >>> /dev/sdf:
> >>> Magic : a92b4efc
> >>> Version : 1.2
> >>> Feature Map : 0x0
> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> >>> Name : b1ackb0x:1
> >>> Creation Time : Sun Jan 13 00:01:44 2013
> >>> Raid Level : raid5
> >>> Raid Devices : 5
> >>>
> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> >>> Data Offset : 2048 sectors
> >>> Super Offset : 8 sectors
> >>> Unused Space : before=1968 sectors, after=976752816 sectors
> >>> State : clean
> >>> Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
> >>>
> >>> Update Time : Tue Apr 29 02:37:38 2014
> >>> Checksum : 2621f9d5 - correct
> >>> Events : 201630
> >>>
> >>> Layout : left-symmetric
> >>> Chunk Size : 512K
> >>>
> >>> Device Role : Active device 3
> >>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
> >>> /dev/sdg:
> >>> Magic : a92b4efc
> >>> Version : 1.2
> >>> Feature Map : 0x0
> >>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> >>> Name : b1ackb0x:1
> >>> Creation Time : Sun Jan 13 00:01:44 2013
> >>> Raid Level : raid5
> >>> Raid Devices : 5
> >>>
> >>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> >>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> >>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> >>> Data Offset : 2048 sectors
> >>> Super Offset : 8 sectors
> >>> Unused Space : before=1968 sectors, after=976752816 sectors
> >>> State : clean
> >>> Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
> >>>
> >>> Update Time : Tue Apr 29 02:48:01 2014
> >>> Checksum : db9e6008 - correct
> >>> Events : 201633
> >>>
> >>> Layout : left-symmetric
> >>> Chunk Size : 512K
> >>>
> >>> Device Role : Active device 4
> >>> Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: 2 disk raid 5 failure
From: Jean-Paul Sergent @ 2014-10-05 9:38 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <CANuKptm=Y3L-xUfGfB5GO3yV63buHeWsfxMYKjUVPP8paGhs0A@mail.gmail.com>
haha, you're awesome, did an apt-get update && apt-get install mdadm.
Got version:
mdadm - v3.3.2 - 21st August 2014
and all is good, re-added drives automatically, threw out the one with
the oldest event, which had bad sectors.
Thanks so much.
One last question though, the filesystem was XFS. Should I repair the
degraded raid first with a spare disk? or should I do an xfs scrub
first?
-JP
On Sun, Oct 5, 2014 at 2:21 AM, Jean-Paul Sergent <jpsergent@gmail.com> wrote:
> I'm recovering with a liveUSB of debian, the system this raid is on
> normally runs fedora 20, I'm not sure what version of mdadm was
> running on that system. I can find out if I need to.
>
>
> root@debian:~# mdadm --version
> mdadm - v3.3 - 3rd September 2013
>
>
> root@debian:~# mdadm -A /dev/md0 --force -vv /dev/sdb /dev/sdc
> /dev/sde /dev/sdf /dev/sdg
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sde is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdg is identified as a member of /dev/md0, slot 4.
> mdadm: added /dev/sdc to /dev/md0 as 1
> mdadm: added /dev/sdb to /dev/md0 as 2
> mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
> mdadm: added /dev/sdg to /dev/md0 as 4 (possibly out of date)
> mdadm: added /dev/sde to /dev/md0 as 0
> mdadm: /dev/md0 assembled from 3 drives - not enough to start the array.
>
> Thanks,
> -JP
>
> On Sun, Oct 5, 2014 at 1:41 AM, NeilBrown <neilb@suse.de> wrote:
>> On Sat, 4 Oct 2014 22:43:01 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
>> wrote:
>>
>>> Greetings,
>>>
>>> Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
>>> cable. It was a Y splitter and it shorted... it was cheap. I was wondering
>>> if there was any chance in getting my data back.
>>>
>>> Of the 2 disks that blew out, one actually had bad/unreadable sectors on
>>> the disk and the other disk seems fine. I have cloned both disks with DD to
>>> 2 new disks and forced the one disk to clone even with errors. The
>>> remaining 3 disks are in tact. The events number on the array for all 5
>>> disks are very close to each other:
>>>
>>> Events : 201636
>>> Events : 201636
>>> Events : 201636
>>> Events : 201630
>>> Events : 201633
>>>
>>> Which from my reading gives me some hope, but I'm not sure. I have not done
>>> "recovering a failed software raid" on the wiki yet, the part about
>>> using a loop device to protect your array. I thought I would send a
>>> message out to this list first before going down
>>> that route.
>>>
>>> I did try to do a mdadm --force --assemble on the array but is says that it
>>> only has 3 disks which isn't enough to start the array. I don't want to do
>>> anything else before consulting the mailing list first.
>>
>> --force --assemble really is what you want. It should work.
>> What does
>> mdadm -A /dev/md1 --force -vv /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg
>>
>> report??
>> What version of mdadm (mdadm -V) do you have?
>>
>> NeilBrown
>>
>>
>>>
>>> below I have pasted the mdadm --examine from each member drive. Any help would
>>> be greatly appreciated.
>>>
>>> Thanks,
>>> -JP
>>>
>>> /dev/sdb:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>>> Name : b1ackb0x:1
>>> Creation Time : Sun Jan 13 00:01:44 2013
>>> Raid Level : raid5
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1968 sectors, after=816 sectors
>>> State : clean
>>> Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
>>>
>>> Update Time : Tue Apr 29 02:49:21 2014
>>> Checksum : cd29f83c - correct
>>> Events : 201636
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 2
>>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>>> /dev/sdc:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>>> Name : b1ackb0x:1
>>> Creation Time : Sun Jan 13 00:01:44 2013
>>> Raid Level : raid5
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1968 sectors, after=816 sectors
>>> State : clean
>>> Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
>>>
>>> Update Time : Tue Apr 29 02:49:21 2014
>>> Checksum : 1e5353af - correct
>>> Events : 201636
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 1
>>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>>> /dev/sde:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>>> Name : b1ackb0x:1
>>> Creation Time : Sun Jan 13 00:01:44 2013
>>> Raid Level : raid5
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1968 sectors, after=816 sectors
>>> State : clean
>>> Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
>>>
>>> Update Time : Tue Apr 29 02:49:21 2014
>>> Checksum : 24a3f586 - correct
>>> Events : 201636
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 0
>>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>>> /dev/sdf:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>>> Name : b1ackb0x:1
>>> Creation Time : Sun Jan 13 00:01:44 2013
>>> Raid Level : raid5
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1968 sectors, after=976752816 sectors
>>> State : clean
>>> Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
>>>
>>> Update Time : Tue Apr 29 02:37:38 2014
>>> Checksum : 2621f9d5 - correct
>>> Events : 201630
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 3
>>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>> /dev/sdg:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>>> Name : b1ackb0x:1
>>> Creation Time : Sun Jan 13 00:01:44 2013
>>> Raid Level : raid5
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=1968 sectors, after=976752816 sectors
>>> State : clean
>>> Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
>>>
>>> Update Time : Tue Apr 29 02:48:01 2014
>>> Checksum : db9e6008 - correct
>>> Events : 201633
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 4
>>> Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
^ permalink raw reply
* MD Raid1 hangs system on read error (3.10)
From: Matthijs Kooijman @ 2014-10-05 9:27 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 15059 bytes --]
Hey folks,
a few times now I've found my system being locked up after a read error
from a hard disk. It looks like the MD code that handles the read error
messes up and causes a GPF.
After this happens, the system becomes completely unresponsive - it
responds to ping and opens TCP connections, but no data comes out. The
serial console also gives no response.
After rebooting, the disk in question showed a pending sector. In the
most recent occurence, I found that the array was also resyncing. I'm
not sure if this also happened in the earlier occurences (but since the
pending sector didn't disappear in the next day, I'd expect no resync
happened before). The read errors always happened during a routine check
of the array.
Over the last months, this happend three or four times now. In between,
I've also seen some read errors that did get handled properly without a
crash.
I've included the kernel output from the most recent case below. If it
helps, I could dig in my logs to find the earlier occurences too. I've
also included some details about the raid arrays. The read error
happened in sdd / md127.
Gr.
Matthijs
Oct 5 01:08:02 tika kernel: [2383614.474387] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Oct 5 01:08:02 tika kernel: [2383614.481134] ata4.00: irq_stat 0x40000008
Oct 5 01:08:02 tika kernel: [2383614.485360] ata4.00: failed command: READ FPDMA QUEUED
Oct 5 01:08:02 tika kernel: [2383614.490775] ata4.00: cmd 60/00:00:00:60:3f/04:00:04:00:00/40 tag 0 ncq 524288 in
Oct 5 01:08:02 tika kernel: [2383614.490775] res 41/40:00:ae:60:3f/00:00:04:00:00/40 Emask 0x409 (media error) <F>
Oct 5 01:08:02 tika kernel: [2383614.506945] ata4.00: status: { DRDY ERR }
Oct 5 01:08:02 tika kernel: [2383614.511213] ata4.00: error: { UNC }
Oct 5 01:08:02 tika kernel: [2383614.534796] ata4.00: configured for UDMA/133
Oct 5 01:08:02 tika kernel: [2383614.539397] sd 3:0:0:0: [sdd] Unhandled sense code
Oct 5 01:08:02 tika kernel: [2383614.544489] sd 3:0:0:0: [sdd]
Oct 5 01:08:02 tika kernel: [2383614.547885] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 5 01:08:02 tika kernel: [2383614.553790] sd 3:0:0:0: [sdd]
Oct 5 01:08:02 tika kernel: [2383614.557417] Sense Key : Medium Error [current] [descriptor]
Oct 5 01:08:02 tika kernel: [2383614.563428] Descriptor sense data with sense descriptors (in hex):
Oct 5 01:08:02 tika kernel: [2383614.569937] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Oct 5 01:08:02 tika kernel: [2383614.577640] 04 3f 60 ae
Oct 5 01:08:02 tika kernel: [2383614.581453] sd 3:0:0:0: [sdd]
Oct 5 01:08:02 tika kernel: [2383614.584856] Add. Sense: Unrecovered read error - auto reallocate failed
Oct 5 01:08:02 tika kernel: [2383614.591847] sd 3:0:0:0: [sdd] CDB:
Oct 5 01:08:02 tika kernel: [2383614.595727] Read(10): 28 00 04 3f 60 00 00 04 00 00
Oct 5 01:08:02 tika kernel: [2383614.657093] general protection fault: 0000 [#1] SMP
Oct 5 01:08:02 tika kernel: [2383614.660531] Modules linked in: joydev fuse btrfs raid6_pq zlib_deflate xor ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs crc32c libcrc32c reiserfs ext2 efivars xt_CLASSIFY xt_helper xt_mac xt_mark nf_conntrack_netlink tcp_diag inet_diag xt_multiport ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables ipt_REJECT xt_nat xt_tcpudp xt_connmark xt_conntrack xt_NFLOG nfnetlink_log nfnetlink xt_limit ipt_rpfilter iptable_raw veth nf_nat_ftp nf_conntrack_ftp xt_state iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_filter ip_tables x_tables bridge stp llc ext4 crc16 jbd2 ipmi_si ipmi_devintf ipmi_msghandler loop hid_generic usbhid hid iTCO_wdt iTCO_vendor_support mperf coretemp snd_pcsp e1000e snd_pcm snd_page_alloc snd_timer lpc_ich i2c_i801 snd psmouse evdev ptp microcode i2c_core soundcore serio_raw mfd_core pps_core ehci_pci uhci_hcd ehci_hcd usbcore processor button usb_co
Oct 5 01:08:02 tika kernel: mmon thermal_sys ext3 mbcache jbd dm_mod raid1 md_mod sd_mod crc_t10dif ahci libahci libata scsi_mod
Oct 5 01:08:02 tika kernel: [2383614.660531] CPU: 3 PID: 232 Comm: md127_raid1 Not tainted 3.10-1-amd64 #1 Debian 3.10.3-1
Oct 5 01:08:02 tika kernel: [2383614.660531] Hardware name: Supermicro X7SPA-HF/X7SPA-HF, BIOS 1.2a 02/21/12
Oct 5 01:08:02 tika kernel: [2383614.660531] task: ffff880139b2e080 ti: ffff8801396a0000 task.ti: ffff8801396a0000
Oct 5 01:08:02 tika kernel: [2383614.660531] RIP: 0010:[<ffffffff8112fe9b>] [<ffffffff8112fe9b>] bio_copy_data+0x138/0x158
Oct 5 01:08:02 tika kernel: [2383614.660531] RSP: 0018:ffff8801396a1d20 EFLAGS: 00010286
Oct 5 01:08:02 tika kernel: [2383614.660531] RAX: ffff88005b451980 RBX: 0000000000000006 RCX: 0000000000001000
Oct 5 01:08:02 tika kernel: [2383614.660531] RDX: ffff880112510a80 RSI: db73880000000006 RDI: ffff88010319c000
Oct 5 01:08:02 tika kernel: [2383614.660531] RBP: 0000000000001000 R08: ffff880112510a00 R09: ffff88005b451800
Oct 5 01:08:02 tika kernel: [2383614.660531] R10: ffff8801396a0000 R11: 0000000000000000 R12: 0000160000000000
Oct 5 01:08:02 tika kernel: [2383614.660531] R13: ffff8801396a1fd8 R14: 6db6db6db6db6db7 R15: ffff880000000000
Oct 5 01:08:02 tika kernel: [2383614.660531] FS: 0000000000000000(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
Oct 5 01:08:02 tika kernel: [2383614.660531] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 5 01:08:02 tika kernel: [2383614.660531] CR2: 00007fffc589aa80 CR3: 0000000125a83000 CR4: 00000000000007e0
Oct 5 01:08:02 tika kernel: [2383614.660531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 5 01:08:02 tika kernel: [2383614.868070] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 5 01:08:02 tika kernel: [2383614.868070] Stack:
Oct 5 01:08:02 tika kernel: [2383614.868070] ffff88010319c000 ffff88011dc91240 ffff88011dc91278 ffff88013952e780
Oct 5 01:08:02 tika kernel: [2383614.868070] ffff880112510a00 ffff8801394cb800 ffff88013952e780 ffffffffa0063785
Oct 5 01:08:02 tika kernel: [2383614.868070] ffffffff8105eaf0 ffff88013aecd040 6db6db6db6db6db7 ffff880000000000
Oct 5 01:08:02 tika kernel: [2383614.868070] Call Trace:
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffffa0063785>] ? raid1d+0x6cd/0xacc [raid1]
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff8105eaf0>] ? mmdrop+0xd/0x1c
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffffa00af0d0>] ? md_thread+0x114/0x132 [md_mod]
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff81057c48>] ? abort_exclusive_wait+0x79/0x79
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffffa00aefbc>] ? signal_pending+0x10/0x10 [md_mod]
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff810572f0>] ? kthread+0x7d/0x85
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff810408ef>] ? do_exit+0x901/0x918
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff81057273>] ? __kthread_parkme+0x59/0x59
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff8138d2bc>] ? ret_from_fork+0x7c/0xb0
Oct 5 01:08:02 tika kernel: [2383614.868070] [<ffffffff81057273>] ? __kthread_parkme+0x59/0x59
Oct 5 01:08:02 tika kernel: [2383614.868070] Code: fe 03 49 0f af ce 49 0f af f6 48 c1 e1 0c 4c 01 f9 48 c1 e6 0c 48 01 cf 4c 01 fe 89 e9 48 89 3c 24 8b 78 0c 48 01 fe 48 8b 3c 24 <f3> a4 41 ff 4a 1c 41 ff 4a 1c 01 eb 41 01 eb e9 0b ff ff ff 58
Oct 5 01:08:02 tika kernel: [2383614.868070] RIP [<ffffffff8112fe9b>] bio_copy_data+0x138/0x158
Oct 5 01:08:02 tika kernel: [2383614.868070] RSP <ffff8801396a1d20>
Oct 5 01:08:02 tika kernel: [2383615.003013] ---[ end trace 83bab722baa22e03 ]---
Oct 5 01:08:02 tika kernel: [2383615.007968] BUG: scheduling while atomic: md127_raid1/232/0x10000002
Oct 5 01:08:02 tika kernel: [2383615.014640] Modules linked in: joydev fuse btrfs raid6_pq zlib_deflate xor ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs crc32c libcrc32c reiserfs ext2 efivars xt_CLASSIFY xt_helper xt_mac xt_mark nf_conntrack_netlink tcp_diag inet_diag xt_multiport ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables ipt_REJECT xt_nat xt_tcpudp xt_connmark xt_conntrack xt_NFLOG nfnetlink_log nfnetlink xt_limit ipt_rpfilter iptable_raw veth nf_nat_ftp nf_conntrack_ftp xt_state iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_filter ip_tables x_tables bridge stp llc ext4 crc16 jbd2 ipmi_si ipmi_devintf ipmi_msghandler loop hid_generic usbhid hid iTCO_wdt iTCO_vendor_support mperf coretemp snd_pcsp e1000e snd_pcm snd_page_alloc snd_timer lpc_ich i2c_i801 snd psmouse evdev ptp microcode i2c_core soundcore serio_raw mfd_core pps_core ehci_pci uhci_hcd ehci_hcd usbcore processor button usb_co
Oct 5 01:08:02 tika kernel: mmon thermal_sys ext3 mbcache jbd dm_mod raid1 md_mod sd_mod crc_t10dif ahci libahci libata scsi_mod
Oct 5 01:08:02 tika kernel: [2383615.119035] CPU: 3 PID: 232 Comm: md127_raid1 Tainted: G D 3.10-1-amd64 #1 Debian 3.10.3-1
Oct 5 01:08:02 tika kernel: [2383615.128462] Hardware name: Supermicro X7SPA-HF/X7SPA-HF, BIOS 1.2a 02/21/12
Oct 5 01:08:02 tika kernel: [2383615.136470] 0000000000000000 ffffffff81382c1d ffffffff81386f6e 0000000000000046
Oct 5 01:08:02 tika kernel: [2383615.144420] 0000000000014040 ffff8801396a1fd8 ffff8801396a1fd8 ffff880139b2e080
Oct 5 01:08:02 tika kernel: [2383615.152521] ffff8801396a0000 000000000000000b 0000000000000246 ffff8801396a1fd8
Oct 5 01:08:02 tika kernel: [2383615.160479] Call Trace:
Oct 5 01:08:02 tika kernel: [2383615.163228] [<ffffffff81382c1d>] ? __schedule_bug+0x42/0x4f
Oct 5 01:08:02 tika kernel: [2383615.169195] [<ffffffff81386f6e>] ? __schedule+0x85/0x532
Oct 5 01:08:02 tika kernel: [2383615.174893] [<ffffffff810606fc>] ? __cond_resched+0x1d/0x26
Oct 5 01:08:02 tika kernel: [2383615.180883] [<ffffffff81387464>] ? _cond_resched+0x10/0x18
Oct 5 01:08:02 tika kernel: [2383615.186732] [<ffffffff81386bd2>] ? down_read+0x9/0x19
Oct 5 01:08:02 tika kernel: [2383615.192207] [<ffffffff8104b6f4>] ? exit_signals+0x1a/0x110
Oct 5 01:08:02 tika kernel: [2383615.198080] [<ffffffff810400fb>] ? do_exit+0x10d/0x918
Oct 5 01:08:02 tika kernel: [2383615.203594] [<ffffffff81382641>] ? printk+0x4f/0x51
Oct 5 01:08:02 tika kernel: [2383615.208859] [<ffffffff81388f91>] ? oops_end+0xa9/0xae
Oct 5 01:08:02 tika kernel: [2383615.214327] [<ffffffff81388568>] ? general_protection+0x28/0x30
Oct 5 01:08:02 tika kernel: [2383615.220824] [<ffffffff8112fe9b>] ? bio_copy_data+0x138/0x158
Oct 5 01:08:02 tika kernel: [2383615.226937] [<ffffffffa0063785>] ? raid1d+0x6cd/0xacc [raid1]
Oct 5 01:08:02 tika kernel: [2383615.233105] [<ffffffff8105eaf0>] ? mmdrop+0xd/0x1c
Oct 5 01:08:02 tika kernel: [2383615.238324] [<ffffffffa00af0d0>] ? md_thread+0x114/0x132 [md_mod]
Oct 5 01:08:02 tika kernel: [2383615.244801] [<ffffffff81057c48>] ? abort_exclusive_wait+0x79/0x79
Oct 5 01:08:02 tika kernel: [2383615.251334] [<ffffffffa00aefbc>] ? signal_pending+0x10/0x10 [md_mod]
Oct 5 01:08:02 tika kernel: [2383615.258046] [<ffffffff810572f0>] ? kthread+0x7d/0x85
Oct 5 01:08:02 tika kernel: [2383615.263448] [<ffffffff810408ef>] ? do_exit+0x901/0x918
Oct 5 01:08:02 tika kernel: [2383615.269032] [<ffffffff81057273>] ? __kthread_parkme+0x59/0x59
Oct 5 01:08:02 tika kernel: [2383615.275217] [<ffffffff8138d2bc>] ? ret_from_fork+0x7c/0xb0
Oct 5 01:08:02 tika kernel: [2383615.281102] [<ffffffff81057273>] ? __kthread_parkme+0x59/0x59
Oct 5 01:08:02 tika kernel: [2383615.287286] note: md127_raid1[232] exited with preempt_count 2
Oct 5 01:08:02 tika kernel: [2383615.293427] BUG: scheduling while atomic: md127_raid1/232/0x10000002
Oct 5 01:08:03 tika kernel: [2383615.300072] Modules linked in: joydev fuse btrfs raid6_pq zlib_deflate xor ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs crc32c libcrc32c reiserfs ext2 efivars xt_CLASSIFY xt_helper xt_mac xt_mark nf_conntrack_netlink tcp_diag inet_diag xt_multiport ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ip6table_mangle ip6table_filter ip6_tables ipt_REJECT xt_nat xt_tcpudp xt_connmark xt_conntrack xt_NFLOG nfnetlink_log nfnetlink xt_limit ipt_rpfilter iptable_raw veth nf_nat_ftp nf_conntrack_ftp xt_state iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_filter ip_tables x_tables bridge stp llc ext4 crc16 jbd2 ipmi_si ipmi_devintf ipmi_msghandler loop hid_generic usbhid hid iTCO_wdt iTCO_vendor_support mperf coretemp snd_pcsp e1000e snd_pcm snd_page_alloc snd_timer lpc_ich i2c_i801 snd psmouse evdev ptp microcode i2c_core soundcore serio_raw mfd_core pps_core ehci_pci uhci_hcd ehci_hcd usbcore processor button usb_co
Oct 5 01:08:03 tika kernel: mmon thermal_sys ext3 mbcache jbd dm_mod raid1 md_mod sd_mod crc_t10dif ahci libahci libata scsi_mod
Oct 5 01:08:03 tika kernel: [2383615.404198] CPU: 1 PID: 232 Comm: md127_raid1 Tainted: G D W 3.10-1-amd64 #1 Debian 3.10.3-1
Oct 5 01:08:03 tika kernel: [2383615.413670] Hardware name: Supermicro X7SPA-HF/X7SPA-HF, BIOS 1.2a 02/21/12
(This is where the log ends, probably things went FUBAR at this point)
matthijs@tika:~$ sudo mdadm --misc -D /dev/md126
/dev/md126:
Version : 1.2
Creation Time : Thu Apr 10 19:04:30 2014
Raid Level : raid1
Array Size : 104791936 (99.94 GiB 107.31 GB)
Used Dev Size : 104791936 (99.94 GiB 107.31 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun Oct 5 11:17:35 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : tika:ssd (local to host tika)
UUID : 16e28bb4:db7d5c69:81cb1b64:ae4f760d
Events : 125
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
matthijs@tika:~$ sudo mdadm --misc -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Tue Sep 21 12:30:00 2010
Raid Level : raid1
Array Size : 488384536 (465.76 GiB 500.11 GB)
Used Dev Size : 488384536 (465.76 GiB 500.11 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun Oct 5 11:12:55 2014
State : active, resyncing
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Resync Status : 16% complete
Name : tika:hdd (local to host tika)
UUID : f201f80b:8c1e6f0b:ccc25f2e:98fdccc6
Events : 922
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply
* Re: 2 disk raid 5 failure
From: Jean-Paul Sergent @ 2014-10-05 9:21 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141005194105.4377c295@notabene.brown>
I'm recovering with a liveUSB of debian, the system this raid is on
normally runs fedora 20, I'm not sure what version of mdadm was
running on that system. I can find out if I need to.
root@debian:~# mdadm --version
mdadm - v3.3 - 3rd September 2013
root@debian:~# mdadm -A /dev/md0 --force -vv /dev/sdb /dev/sdc
/dev/sde /dev/sdf /dev/sdg
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 4.
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdb to /dev/md0 as 2
mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
mdadm: added /dev/sdg to /dev/md0 as 4 (possibly out of date)
mdadm: added /dev/sde to /dev/md0 as 0
mdadm: /dev/md0 assembled from 3 drives - not enough to start the array.
Thanks,
-JP
On Sun, Oct 5, 2014 at 1:41 AM, NeilBrown <neilb@suse.de> wrote:
> On Sat, 4 Oct 2014 22:43:01 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
> wrote:
>
>> Greetings,
>>
>> Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
>> cable. It was a Y splitter and it shorted... it was cheap. I was wondering
>> if there was any chance in getting my data back.
>>
>> Of the 2 disks that blew out, one actually had bad/unreadable sectors on
>> the disk and the other disk seems fine. I have cloned both disks with DD to
>> 2 new disks and forced the one disk to clone even with errors. The
>> remaining 3 disks are in tact. The events number on the array for all 5
>> disks are very close to each other:
>>
>> Events : 201636
>> Events : 201636
>> Events : 201636
>> Events : 201630
>> Events : 201633
>>
>> Which from my reading gives me some hope, but I'm not sure. I have not done
>> "recovering a failed software raid" on the wiki yet, the part about
>> using a loop device to protect your array. I thought I would send a
>> message out to this list first before going down
>> that route.
>>
>> I did try to do a mdadm --force --assemble on the array but is says that it
>> only has 3 disks which isn't enough to start the array. I don't want to do
>> anything else before consulting the mailing list first.
>
> --force --assemble really is what you want. It should work.
> What does
> mdadm -A /dev/md1 --force -vv /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg
>
> report??
> What version of mdadm (mdadm -V) do you have?
>
> NeilBrown
>
>
>>
>> below I have pasted the mdadm --examine from each member drive. Any help would
>> be greatly appreciated.
>>
>> Thanks,
>> -JP
>>
>> /dev/sdb:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> Name : b1ackb0x:1
>> Creation Time : Sun Jan 13 00:01:44 2013
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=816 sectors
>> State : clean
>> Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
>>
>> Update Time : Tue Apr 29 02:49:21 2014
>> Checksum : cd29f83c - correct
>> Events : 201636
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> /dev/sdc:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> Name : b1ackb0x:1
>> Creation Time : Sun Jan 13 00:01:44 2013
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=816 sectors
>> State : clean
>> Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
>>
>> Update Time : Tue Apr 29 02:49:21 2014
>> Checksum : 1e5353af - correct
>> Events : 201636
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> /dev/sde:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> Name : b1ackb0x:1
>> Creation Time : Sun Jan 13 00:01:44 2013
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=816 sectors
>> State : clean
>> Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
>>
>> Update Time : Tue Apr 29 02:49:21 2014
>> Checksum : 24a3f586 - correct
>> Events : 201636
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
>> /dev/sdf:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> Name : b1ackb0x:1
>> Creation Time : Sun Jan 13 00:01:44 2013
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
>>
>> Update Time : Tue Apr 29 02:37:38 2014
>> Checksum : 2621f9d5 - correct
>> Events : 201630
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>> /dev/sdg:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
>> Name : b1ackb0x:1
>> Creation Time : Sun Jan 13 00:01:44 2013
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
>> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
>> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=976752816 sectors
>> State : clean
>> Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
>>
>> Update Time : Tue Apr 29 02:48:01 2014
>> Checksum : db9e6008 - correct
>> Events : 201633
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 4
>> Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: 2 disk raid 5 failure
From: NeilBrown @ 2014-10-05 8:41 UTC (permalink / raw)
To: Jean-Paul Sergent; +Cc: linux-raid
In-Reply-To: <CANuKpt=QJsham0TQWR1Cu1EC8GLyc0aSdU4p3DOznBRrEea11A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6711 bytes --]
On Sat, 4 Oct 2014 22:43:01 -0700 Jean-Paul Sergent <jpsergent@gmail.com>
wrote:
> Greetings,
>
> Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
> cable. It was a Y splitter and it shorted... it was cheap. I was wondering
> if there was any chance in getting my data back.
>
> Of the 2 disks that blew out, one actually had bad/unreadable sectors on
> the disk and the other disk seems fine. I have cloned both disks with DD to
> 2 new disks and forced the one disk to clone even with errors. The
> remaining 3 disks are in tact. The events number on the array for all 5
> disks are very close to each other:
>
> Events : 201636
> Events : 201636
> Events : 201636
> Events : 201630
> Events : 201633
>
> Which from my reading gives me some hope, but I'm not sure. I have not done
> "recovering a failed software raid" on the wiki yet, the part about
> using a loop device to protect your array. I thought I would send a
> message out to this list first before going down
> that route.
>
> I did try to do a mdadm --force --assemble on the array but is says that it
> only has 3 disks which isn't enough to start the array. I don't want to do
> anything else before consulting the mailing list first.
--force --assemble really is what you want. It should work.
What does
mdadm -A /dev/md1 --force -vv /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg
report??
What version of mdadm (mdadm -V) do you have?
NeilBrown
>
> below I have pasted the mdadm --examine from each member drive. Any help would
> be greatly appreciated.
>
> Thanks,
> -JP
>
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> Name : b1ackb0x:1
> Creation Time : Sun Jan 13 00:01:44 2013
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=816 sectors
> State : clean
> Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
>
> Update Time : Tue Apr 29 02:49:21 2014
> Checksum : cd29f83c - correct
> Events : 201636
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdc:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> Name : b1ackb0x:1
> Creation Time : Sun Jan 13 00:01:44 2013
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=816 sectors
> State : clean
> Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
>
> Update Time : Tue Apr 29 02:49:21 2014
> Checksum : 1e5353af - correct
> Events : 201636
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sde:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> Name : b1ackb0x:1
> Creation Time : Sun Jan 13 00:01:44 2013
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=816 sectors
> State : clean
> Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
>
> Update Time : Tue Apr 29 02:49:21 2014
> Checksum : 24a3f586 - correct
> Events : 201636
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdf:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> Name : b1ackb0x:1
> Creation Time : Sun Jan 13 00:01:44 2013
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=976752816 sectors
> State : clean
> Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
>
> Update Time : Tue Apr 29 02:37:38 2014
> Checksum : 2621f9d5 - correct
> Events : 201630
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdg:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 7de100f5:4f30f751:62456293:fe98f735
> Name : b1ackb0x:1
> Creation Time : Sun Jan 13 00:01:44 2013
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=976752816 sectors
> State : clean
> Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
>
> Update Time : Tue Apr 29 02:48:01 2014
> Checksum : db9e6008 - correct
> Events : 201633
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 4
> Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* 2 disk raid 5 failure
From: Jean-Paul Sergent @ 2014-10-05 5:43 UTC (permalink / raw)
To: linux-raid
Greetings,
Recently I lost 2 disks, out of 5, in my raid 5 array from a bad SATA power
cable. It was a Y splitter and it shorted... it was cheap. I was wondering
if there was any chance in getting my data back.
Of the 2 disks that blew out, one actually had bad/unreadable sectors on
the disk and the other disk seems fine. I have cloned both disks with DD to
2 new disks and forced the one disk to clone even with errors. The
remaining 3 disks are in tact. The events number on the array for all 5
disks are very close to each other:
Events : 201636
Events : 201636
Events : 201636
Events : 201630
Events : 201633
Which from my reading gives me some hope, but I'm not sure. I have not done
"recovering a failed software raid" on the wiki yet, the part about
using a loop device to protect your array. I thought I would send a
message out to this list first before going down
that route.
I did try to do a mdadm --force --assemble on the array but is says that it
only has 3 disks which isn't enough to start the array. I don't want to do
anything else before consulting the mailing list first.
below I have pasted the mdadm --examine from each member drive. Any help would
be greatly appreciated.
Thanks,
-JP
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7de100f5:4f30f751:62456293:fe98f735
Name : b1ackb0x:1
Creation Time : Sun Jan 13 00:01:44 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=816 sectors
State : clean
Device UUID : 71d7c3d7:7b232399:51571715:711da6f6
Update Time : Tue Apr 29 02:49:21 2014
Checksum : cd29f83c - correct
Events : 201636
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7de100f5:4f30f751:62456293:fe98f735
Name : b1ackb0x:1
Creation Time : Sun Jan 13 00:01:44 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=816 sectors
State : clean
Device UUID : b47c32b5:b2f9e81a:37150c33:8e3fa6ca
Update Time : Tue Apr 29 02:49:21 2014
Checksum : 1e5353af - correct
Events : 201636
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7de100f5:4f30f751:62456293:fe98f735
Name : b1ackb0x:1
Creation Time : Sun Jan 13 00:01:44 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=816 sectors
State : clean
Device UUID : 0398da5b:0bcddd81:8f7e77e9:6689ee0c
Update Time : Tue Apr 29 02:49:21 2014
Checksum : 24a3f586 - correct
Events : 201636
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7de100f5:4f30f751:62456293:fe98f735
Name : b1ackb0x:1
Creation Time : Sun Jan 13 00:01:44 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=976752816 sectors
State : clean
Device UUID : 356c6d85:627a994f:753dec0d:db4fa4f2
Update Time : Tue Apr 29 02:37:38 2014
Checksum : 2621f9d5 - correct
Events : 201630
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7de100f5:4f30f751:62456293:fe98f735
Name : b1ackb0x:1
Creation Time : Sun Jan 13 00:01:44 2013
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=976752816 sectors
State : clean
Device UUID : 3dc152d8:832dd43a:a6d638e3:6e12b394
Update Time : Tue Apr 29 02:48:01 2014
Checksum : db9e6008 - correct
Events : 201633
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAA.A ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Re: raid6 appears not-degraded just after creation
From: Anssi Hannula @ 2014-10-05 0:24 UTC (permalink / raw)
To: Ethan Wilson, linux-raid
In-Reply-To: <54305509.30001@shiftmail.org>
04.10.2014, 23:14, Ethan Wilson kirjoitti:
> MD Raid6 appears not-degraded during initial resync just after creation:
>
> md106 : active raid6 sdbv1[7] sdbr1[6] sdbq1[5] sdbp1[4] sdbo1[3]
> sdbn1[2] sdbm1[1] sdbl1[0]
> 3144938496 blocks super 1.2 level 6, 512k chunk, algorithm 2
> [8/8] [UUUUUUUU]
> [======>..............] resync = 34.3% (179834012/524156416)
> finish=54.3min speed=105637K/sec
>
> and also from sys filesystem:
>
> cat /sys/block/md106/md/degraded
> 0
>
> (kernel 3.14.19)
>
> Is this normal?
I'd say so.
(Re)syncing does not mean the array is degraded - everything you write
to the new array still has full redundancy immediately.
--
Anssi Hannula
^ permalink raw reply
* Re: build-sys: do not depend on check_rundir for executables
From: NeilBrown @ 2014-10-04 20:58 UTC (permalink / raw)
To: Michael Tokarev; +Cc: linux-raid
In-Reply-To: <1412443628.428300.8315.nullmailer@msgid.tls.msk.ru>
[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]
On Sat, 04 Oct 2014 21:27:08 +0400 Michael Tokarev <mjt@tls.msk.ru> wrote:
> The problem is that the link lines will always be executed
> even if the executables are up to date.
>
> If anything, this check_rundir should be a dependency of
> install target, or some other "phony" target like this,
> since check_rundir is phony by itself.
>
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
>
> --- a/Makefile
> +++ b/Makefile
> @@ -172,7 +172,7 @@ everything-test: all mdadm.static swap_super test_stripe \
> # mdadm.uclibc and mdassemble.uclibc don't work on x86-64
> # mdadm.tcc doesn't work..
>
> -mdadm : check_rundir $(OBJS)
> +mdadm : $(OBJS)
> $(CC) $(CFLAGS) $(LDFLAGS) -o mdadm $(OBJS) $(LDLIBS)
>
> mdadm.static : $(OBJS) $(STATICOBJS)
> @@ -195,7 +195,7 @@ mdmon.O2 : $(MON_SRCS) $(INCL) mdmon.h
> $(CC) -o mdmon.O2 $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -DHAVE_STDINT_H -O2 -D_FORTIFY_SOURCE=2 $(MON_SRCS)
>
> # use '-z now' to guarantee no dynamic linker interactions with the monitor thread
> -mdmon : check_rundir $(MON_OBJS)
> +mdmon : $(MON_OBJS)
> $(CC) $(CFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -Wl,-z,now -o mdmon $(MON_OBJS) $(LDLIBS)
> msg.o: msg.c msg.h
>
>
Thanks for the report... I might fix it like this.
NeilBrown
diff --git a/Makefile b/Makefile
index f058a22ee2dc..a7d8c5c1b7fa 100644
--- a/Makefile
+++ b/Makefile
@@ -153,7 +153,7 @@ ASSEMBLE_SRCS += $(ASSEMBLE_AUTO_SRCS)
ASSEMBLE_FLAGS += -DMDASSEMBLE_AUTO
endif
-all : check_rundir mdadm mdmon
+all : mdadm mdmon
man : mdadm.man md.man mdadm.conf.man mdmon.man raid6check.man
check_rundir:
@@ -172,7 +172,7 @@ everything-test: all mdadm.static swap_super test_stripe \
# mdadm.uclibc and mdassemble.uclibc don't work on x86-64
# mdadm.tcc doesn't work..
-mdadm : check_rundir $(OBJS)
+mdadm : $(OBJS) | check_rundir
$(CC) $(CFLAGS) $(LDFLAGS) -o mdadm $(OBJS) $(LDLIBS)
mdadm.static : $(OBJS) $(STATICOBJS)
@@ -195,7 +195,7 @@ mdmon.O2 : $(MON_SRCS) $(INCL) mdmon.h
$(CC) -o mdmon.O2 $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -DHAVE_STDINT_H -O2 -D_FORTIFY_SOURCE=2 $(MON_SRCS)
# use '-z now' to guarantee no dynamic linker interactions with the monitor thread
-mdmon : check_rundir $(MON_OBJS)
+mdmon : $(MON_OBJS) | check_rundir
$(CC) $(CFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -Wl,-z,now -o mdmon $(MON_OBJS) $(LDLIBS)
msg.o: msg.c msg.h
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* raid6 appears not-degraded just after creation
From: Ethan Wilson @ 2014-10-04 20:14 UTC (permalink / raw)
To: linux-raid
MD Raid6 appears not-degraded during initial resync just after creation:
md106 : active raid6 sdbv1[7] sdbr1[6] sdbq1[5] sdbp1[4] sdbo1[3]
sdbn1[2] sdbm1[1] sdbl1[0]
3144938496 blocks super 1.2 level 6, 512k chunk, algorithm 2
[8/8] [UUUUUUUU]
[======>..............] resync = 34.3% (179834012/524156416)
finish=54.3min speed=105637K/sec
and also from sys filesystem:
cat /sys/block/md106/md/degraded
0
(kernel 3.14.19)
Is this normal?
EW
^ permalink raw reply
* build-sys: do not depend on check_rundir for executables
From: Michael Tokarev @ 2014-10-04 17:27 UTC (permalink / raw)
To: linux-raid
The problem is that the link lines will always be executed
even if the executables are up to date.
If anything, this check_rundir should be a dependency of
install target, or some other "phony" target like this,
since check_rundir is phony by itself.
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
--- a/Makefile
+++ b/Makefile
@@ -172,7 +172,7 @@ everything-test: all mdadm.static swap_super test_stripe \
# mdadm.uclibc and mdassemble.uclibc don't work on x86-64
# mdadm.tcc doesn't work..
-mdadm : check_rundir $(OBJS)
+mdadm : $(OBJS)
$(CC) $(CFLAGS) $(LDFLAGS) -o mdadm $(OBJS) $(LDLIBS)
mdadm.static : $(OBJS) $(STATICOBJS)
@@ -195,7 +195,7 @@ mdmon.O2 : $(MON_SRCS) $(INCL) mdmon.h
$(CC) -o mdmon.O2 $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -DHAVE_STDINT_H -O2 -D_FORTIFY_SOURCE=2 $(MON_SRCS)
# use '-z now' to guarantee no dynamic linker interactions with the monitor thread
-mdmon : check_rundir $(MON_OBJS)
+mdmon : $(MON_OBJS)
$(CC) $(CFLAGS) $(LDFLAGS) $(MON_LDFLAGS) -Wl,-z,now -o mdmon $(MON_OBJS) $(LDLIBS)
msg.o: msg.c msg.h
^ permalink raw reply
* Extremely High mismatch_cnt on RAID1 system
From: Dennis Grant @ 2014-10-04 13:46 UTC (permalink / raw)
To: linux-raid
Hello all.
I recently updated an Ubuntu 12.04 LTS system to a 14.04 LTS system.
This upgrade did not go particularly smoothly, and I've been dealing
with various types of weirdness ever since.
This system has 4 hard drives in it, comprising 3 RAID 1 arrays. One
pair of drives contains two arrays - one for / (md0), and one for
/home (md2). The other array is a pair of older drives (the original
array from when the machine was first built) which I have mounted as
/home/backups (md3)
Some investigation has me thinking that perhaps the arrays had gotten
out of sync, or perhaps there was even a failing drive. So I've been
checking and repairing arrays to see what happens.
All drives passed their long SMART tests with no issues.
Even after multiple checks, repairs, and rebuilds, the arrays on the
bigger drives (/ and /home) are showing insanely high mismatch_cnt
values. This has me concerned.
Here are the details:
$cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md3 : active raid1 sdb4[3] sda4[2]
660213568 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sda2[2] sdb2[3]
307068736 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdc1[0] sdd1[1]
2930133824 blocks super 1.2 [2/2] [UU]
unused devices: <none>
$ sudo mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Jun 9 19:19:32 2013
Raid Level : raid1
Array Size : 307068736 (292.84 GiB 314.44 GB)
Used Dev Size : 307068736 (292.84 GiB 314.44 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Oct 4 10:18:58 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : karzai:0 (local to host karzai)
UUID : ff878502:6d567c41:4dbd32f7:7a1122be
Events : 256772
$ sudo mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Sat Jun 8 04:09:23 2013
Raid Level : raid1
Array Size : 2930133824 (2794.39 GiB 3000.46 GB)
Used Dev Size : 2930133824 (2794.39 GiB 3000.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Oct 4 10:19:33 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : karzai:2 (local to host karzai)
UUID : dd5af8bf:02a9500e:0b73f986:72a2ba43
Events : 459
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
Number Major Minor RaidDevice State
3 8 18 0 active sync /dev/sdb2
2 8 2 1 active sync /dev/sda2
$ sudo mdadm -D /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Sun Jun 9 20:05:28 2013
Raid Level : raid1
Array Size : 660213568 (629.63 GiB 676.06 GB)
Used Dev Size : 660213568 (629.63 GiB 676.06 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Oct 4 09:52:47 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : karzai:3 (local to host karzai)
UUID : e60332c8:e5568487:df2f3b61:94fd2b2c
Events : 14971
Number Major Minor RaidDevice State
3 8 20 0 active sync /dev/sdb4
2 8 4 1 active sync /dev/sda4
$ grep . /sys/block/md?/md/mismatch_cnt
/sys/block/md0/md/mismatch_cnt:3148032
/sys/block/md2/md/mismatch_cnt:20217856
/sys/block/md3/md/mismatch_cnt:0
fsck of md2 and md3 were clean
Should I be concerned? The fact that the high counts on on arrays that
share the same physical drives has me very nervous, notwithstanding
the passed SMART tests.
Thank you.
^ permalink raw reply
* [GIT PULL REQUEST] one last-minute update for md/raid5
From: NeilBrown @ 2014-10-03 5:38 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux RAID, lkml, Martin K. Petersen, Shaohua Li, Mike Snitzer,
Heinz Mauelshagen
[-- Attachment #1: Type: text/plain, Size: 4720 bytes --]
Hi Linus,
I hope this is in time for 3.17...
It turns out that 'discard_zeroes_data' is not a guarantee, and RAID4/5/6
really depends on it for DISCARD to be handled safely.
So it is safest to default to disabling DISCARD on these arrays until some
sort of whitelist arrangement can be found.
Thanks,
NeilBrown
The following changes since commit fe82dcec644244676d55a1384c958d5f67979adb:
Linux 3.17-rc7 (2014-09-28 14:29:07 -0700)
are available in the git repository at:
git://neil.brown.name/md tags/md/3.17-final-fix
for you to fetch changes up to 8e0e99ba64c7ba46133a7c8a3e3f7de01f23bd93:
md/raid5: disable 'DISCARD' by default due to safety concerns. (2014-10-02 13:45:00 +1000)
----------------------------------------------------------------
One fix for raid5 discard issue.
----------------------------------------------------------------
NeilBrown (1):
md/raid5: disable 'DISCARD' by default due to safety concerns.
drivers/md/raid5.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
commit 8e0e99ba64c7ba46133a7c8a3e3f7de01f23bd93
Author: NeilBrown <neilb@suse.de>
Date: Thu Oct 2 13:45:00 2014 +1000
md/raid5: disable 'DISCARD' by default due to safety concerns.
It has come to my attention (thanks Martin) that 'discard_zeroes_data'
is only a hint. Some devices in some cases don't do what it
says on the label.
The use of DISCARD in RAID5 depends on reads from discarded regions
being predictably zero. If a write to a previously discarded region
performs a read-modify-write cycle it assumes that the parity block
was consistent with the data blocks. If all were zero, this would
be the case. If some are and some aren't this would not be the case.
This could lead to data corruption after a device failure when
data needs to be reconstructed from the parity.
As we cannot trust 'discard_zeroes_data', ignore it by default
and so disallow DISCARD on all raid4/5/6 arrays.
As many devices are trustworthy, and as there are benefits to using
DISCARD, add a module parameter to over-ride this caution and cause
DISCARD to work if discard_zeroes_data is set.
If a site want to enable DISCARD on some arrays but not on others they
should select DISCARD support at the filesystem level, and set the
raid456 module parameter.
raid456.devices_handle_discard_safely=Y
As this is a data-safety issue, I believe this patch is suitable for
-stable.
DISCARD support for RAID456 was added in 3.7
Cc: Shaohua Li <shli@kernel.org>
Cc: "Martin K. Petersen" <martin.petersen@oracle.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Heinz Mauelshagen <heinzm@redhat.com>
Cc: stable@vger.kernel.org (3.7+)
Acked-by: Martin K. Petersen <martin.petersen@oracle.com>
Acked-by: Mike Snitzer <snitzer@redhat.com>
Fixes: 620125f2bf8ff0c4969b79653b54d7bcc9d40637
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 183588b11fc1..9f0fbecd1eb5 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -64,6 +64,10 @@
#define cpu_to_group(cpu) cpu_to_node(cpu)
#define ANY_GROUP NUMA_NO_NODE
+static bool devices_handle_discard_safely = false;
+module_param(devices_handle_discard_safely, bool, 0644);
+MODULE_PARM_DESC(devices_handle_discard_safely,
+ "Set to Y if all devices in each array reliably return zeroes on reads from discarded regions");
static struct workqueue_struct *raid5_wq;
/*
* Stripe cache
@@ -6208,7 +6212,7 @@ static int run(struct mddev *mddev)
mddev->queue->limits.discard_granularity = stripe;
/*
* unaligned part of discard request will be ignored, so can't
- * guarantee discard_zerors_data
+ * guarantee discard_zeroes_data
*/
mddev->queue->limits.discard_zeroes_data = 0;
@@ -6233,6 +6237,18 @@ static int run(struct mddev *mddev)
!bdev_get_queue(rdev->bdev)->
limits.discard_zeroes_data)
discard_supported = false;
+ /* Unfortunately, discard_zeroes_data is not currently
+ * a guarantee - just a hint. So we only allow DISCARD
+ * if the sysadmin has confirmed that only safe devices
+ * are in use by setting a module parameter.
+ */
+ if (!devices_handle_discard_safely) {
+ if (discard_supported) {
+ pr_info("md/raid456: discard support disabled due to uncertainty.\n");
+ pr_info("Set raid456.devices_handle_discard_safely=Y to override.\n");
+ }
+ discard_supported = false;
+ }
}
if (discard_supported &&
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: UEFI and mdadm questions.
From: Chris Murphy @ 2014-10-03 5:04 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <BLU436-SMTP10330CBEBC0CA1E27853BE498B80@phx.gbl>
On Oct 1, 2014, at 12:33 PM, "Wilson, Jonathan" <piercing_male@hotmail.com> wrote:
> From what I can tell with UEFI I need to set up a UEFI partition with a
> FAT format.
It's a particular kind of FAT, that's defined as EFI FAT, the idea being that if the originating FAT ever changes, EFI FAT won't.
>
> On my current BIOS system I have a Biosboot 1M, /boot Raid1 200M and /
> Raid 1 40G.
>
> Obviously Grub installs to the mbr, and then installs a bit into
> Biosboot which can read raids, hence it can read and boot from /boot.
BIOSboot applies to GPT disks on BIOS computers, not MBR. On MBR disks, the GRUB stage1 code jumps to stage2 code in the MBR gap which is the region between the MBR and the first partition's starting LBA.
>
> Further, from what I can tell, into the UEFI partition can go either a
> kernel & initramfs with UEFI support, or a "loader" that then loads the
> kernel.
No. An OSLoader is required, it's an EFI application. Its job is to load a kernel and initramfs. The kernel and initramfs could be on the ESP (EFI System partition) but this is fraught with limitations. The expectation is that the kernel and initramfs are on some other partition of the same disk. Of course if you're using GRUB it doesn't care and will find a kernel/initramfs off another disk also, or even off md raid.
In the case of GRUB, it directly understands pretty much every filesystem used on linux, so it can read kernel+initramfs from ext4, or ext4 on mdraid, or on LVM, or on Btrfs (including multiple devices).
>
> What I am unsure about are...
>
> 1) Can the loader/kernel understand md raid? so the / can be in a bog
> standard raid1 v1.2?
If the EFI OSLoader is GRUB2, then yes.
>
> 2) I'm guessing I would no longer need the /boot as that would be
> replaced by what ever was in the UEFI partition?
The ESP isn't a replacement for /boot. But you can have a unified /boot and / on a single file system, and GRUB2 can boot from it, including if that volume is on an md device. GRUB2 even boots off md raid6 (it's sorta crazy and badass at the same time, but in my testing it does work with the BIOS being the limiting factor as to how many drives are recognized at this stage).
>
> 3) as my "/boot" is currently in a raid 1 my life is simple, should any
> changes occur they are replicated to the drives I have set up as /boot
> raid 1, and I installed the mbr portion of boot loader manually on each
> disk and tested pulling one, and then booting from another.. it
> worked :-)
> So I would like to keep things, if not simple, at least less likely to
> have problems because I forgot to install duplicates of the UEFI on all
> the disks… so can I use a .90v raid1 on the UEFI partition,
That solution isn't that simple because it requires too many things that aren't supported by any distribution. And also it increases the chances of corruption because of course the EFI firmware only sees these partition/volumes as single devices, not as md devices that need to be assembled first into a logical device. Some EFI firmwares do reportedly write to the ESP, so if they write to one and not the other, while also not changing the metadata on either partition, there's no way to know which one is valid now. So that's just a mess. And even if your firmware doesn't write to the ESP ever, what about dual boot? That too would be proscribed because the ESP is shared among multiple OS's. So the idea is just way off the rails of anything standardized or widely supported. Will it work? Yes. But do you want to support something this non-standard?
An alternative is a single static grub.cfg on the EFI System partition that points to the real grub.cfg on an mdraid1 / or /boot. Since the ESP's are identical, and never need updating again, this is much safer. You can also dispense with this ridiculously bad idea most all linux distros have right now of persistently mounting the ESP at /boot/efi (and persistently mount it rw as well! really bad!).
The way to do this is with the grub configfile command. The most straightforward way to do it is let grub-mkconfig create a grub.cfg for you, and then copy past the parts that tell GRUB how to find /. For example it needs to know the mduuid so it knows there's an mdraid volume to assembled, and also the volume uuid so it knows what filesystem volume it should find the file one, and then "configfile /boot/grub/grub.cfg" and it'll load that grub.cfg. And it's that grub.cfg that your system should update - of course you have to figure out how your system knows to update the grub.cfg so that it does it correctly without stepping on your ESP grub.cfg (for one comment out the /boot/efi line in fstab and then unmount the ESP). This is distribution specific.
Either way it's a bit complicated to figure out initially.
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [dm-devel] dm-raid: add RAID discard support
From: Martin K. Petersen @ 2014-10-03 1:14 UTC (permalink / raw)
To: NeilBrown
Cc: Mike Snitzer, Heinz Mauelshagen, device-mapper development,
Shaohua Li, Martin K. Petersen, linux RAID, lkml
In-Reply-To: <20141002140410.1b379f74@notabene.brown>
>>>>> "Neil" == NeilBrown <neilb@suse.de> writes:
Neil> I plan to submit this to Linus tomorrow, hopefully for 3.7, unless
Neil> there are complaints. It is in my for-next branch now.
Acked-by: Martin K. Petersen <martin.petersen@oracle.com>
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: dm-raid: add RAID discard support
From: Mike Snitzer @ 2014-10-02 18:00 UTC (permalink / raw)
To: NeilBrown
Cc: Heinz Mauelshagen, device-mapper development, Shaohua Li,
Martin K. Petersen, linux RAID, lkml
In-Reply-To: <20141002135232.GA25298@redhat.com>
On Thu, Oct 02 2014 at 9:52am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:
> Thanks Neil, I'll get something comparable staged for dm-raid for 3.18
> (less urgency given that dm-raid doesn't yet support discards).
FYI, I've staged the following dm-raid change in linux-next for 3.18:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=bfc3f335f9886b68d281240aab751b96259d8860
^ permalink raw reply
* Re: [PATCH] dm-log-userspace: fix memory leak on failure path in dm_ulog_tfr_init()
From: Brassow Jonathan @ 2014-10-02 16:57 UTC (permalink / raw)
To: Alexey Khoroshilov
Cc: Alasdair Kergon, Mike Snitzer, dm-devel, Neil Brown, linux-raid,
linux-kernel, ldv-project
In-Reply-To: <1412197115-32593-1-git-send-email-khoroshilov@ispras.ru>
On Oct 1, 2014, at 3:58 PM, Alexey Khoroshilov <khoroshilov@ispras.ru> wrote:
> If cn_add_callback() fails in dm_ulog_tfr_init(), it does not
> deallocate prealloced memory but calls cn_del_callback().
> It looks like a misprint.
>
> Found by Linux Driver Verification project (linuxtesting.org).
>
> Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
^ permalink raw reply
* Re: dm-raid: add RAID discard support
From: Mike Snitzer @ 2014-10-02 13:52 UTC (permalink / raw)
To: NeilBrown
Cc: Heinz Mauelshagen, device-mapper development, Shaohua Li,
Martin K. Petersen, linux RAID, lkml
In-Reply-To: <20141002140410.1b379f74@notabene.brown>
On Thu, Oct 02 2014 at 12:04am -0400,
NeilBrown <neilb@suse.de> wrote:
>
> I plan to submit this to Linus tomorrow, hopefully for 3.7, unless there are
> complaints. It is in my for-next branch now.
>
> Thanks,
> NeilBrown
>
>
> From aec6f821ed92fac5ae4f1db50279a3999de5872a Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Thu, 2 Oct 2014 13:45:00 +1000
> Subject: [PATCH] md/raid5: disable 'DISCARD' by default due to safety
> concerns.
>
> It has come to my attention (thanks Martin) that 'discard_zeroes_data'
> is only a hint. Some devices in some cases don't do what it
> says on the label.
>
> The use of DISCARD in RAID5 depends on reads from discarded regions
> being predictably zero. If a write to a previously discarded region
> performs a read-modify-write cycle it assumes that the parity block
> was consistent with the data blocks. If all were zero, this would
> be the case. If some are and some aren't this would not be the case.
> This could lead to data corruption after a device failure when
> data needs to be reconstructed from the parity.
>
> As we cannot trust 'discard_zeroes_data', ignore it by default
> and so disallow DISCARD on all raid4/5/6 arrays.
>
> As many devices are trustworthy, and as there are benefits to using
> DISCARD, add a module parameter to over-ride this caution and cause
> DISCARD to work if discard_zeroes_data is set.
>
> If a site want to enable DISCARD on some arrays but not on others they
> should select DISCARD support at the filesystem level, and set the
> raid456 module parameter.
> raid456.devices_handle_discard_safely=Y
>
> As this is a data-safety issue, I believe this patch is suitable for
> -stable.
> DISCARD support for RAID456 was added in 3.7
>
> Cc: Shaohua Li <shli@kernel.org>
> Cc: "Martin K. Petersen" <martin.petersen@oracle.com>
> Cc: Mike Snitzer <snitzer@redhat.com>
> Cc: Heinz Mauelshagen <heinzm@redhat.com>
> Cc: stable@vger.kernel.org (3.7+)
> Fixes: 620125f2bf8ff0c4969b79653b54d7bcc9d40637
> Signed-off-by: NeilBrown <neilb@suse.de>
Acked-by: Mike Snitzer <snitzer@redhat.com>
Thanks Neil, I'll get something comparable staged for dm-raid for 3.18
(less urgency given that dm-raid doesn't yet support discards).
^ permalink raw reply
* Why MD often doesn't correct read errors?
From: Ethan Wilson @ 2014-10-02 9:11 UTC (permalink / raw)
To: linux-raid
Hello all,
I am testing a system with a failing disk.
This is an MD raid5 with bitmap, disks are over LSI SAS. A pretty normal
setup.
I can show very long dmesgs in which most read errors are apparently not
corrected. However upper layers such as the filesystem do not complain
either, e.g. the filesystem does not go readonly, and no "read error"
received from userspace. So everything actually works, but I can't
understand why!?
Here is one piece of dmesg in which only at [2204.845894] some errors
get corrected by MD, so I see at least 3 errors before that (at time
729, 1207, 2071) which are apparently ignored by everything:
[ 289.360928] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Opts: (null)
[ 729.141449] sd 6:0:33:0: [sdah] Unhandled sense code
[ 729.141460] sd 6:0:33:0: [sdah]
[ 729.141463] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 729.141466] sd 6:0:33:0: [sdah]
[ 729.141467] Sense Key : Medium Error [current]
[ 729.141471] Info fld=0xcba7e3c
[ 729.141473] sd 6:0:33:0: [sdah]
[ 729.141476] Add. Sense: Unrecovered read error
[ 729.141478] sd 6:0:33:0: [sdah] CDB:
[ 729.141480] Read(10): 28 00 0c ba 7e 00 00 00 a0 00
[ 729.141488] end_request: critical medium error, dev sdah, sector
213548604
[ 781.088413] perf samples too long (2510 > 2500), lowering
kernel.perf_event_max_sample_rate to 50000
[ 1207.475752] sd 6:0:33:0: [sdah] Unhandled sense code
[ 1207.475761] sd 6:0:33:0: [sdah]
[ 1207.475762] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1207.475764] sd 6:0:33:0: [sdah]
[ 1207.475765] Sense Key : Medium Error [current]
[ 1207.475767] Info fld=0xd2d89d2
[ 1207.475769] sd 6:0:33:0: [sdah]
[ 1207.475770] Add. Sense: Unrecovered read error
[ 1207.475772] sd 6:0:33:0: [sdah] CDB:
[ 1207.475773] Read(10): 28 00 0d 2d 88 c0 00 01 98 00
[ 1207.475778] end_request: critical medium error, dev sdah, sector
221088210
[ 2071.445584] sd 6:0:33:0: [sdah] Unhandled sense code
[ 2071.445596] sd 6:0:33:0: [sdah]
[ 2071.445599] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 2071.445601] sd 6:0:33:0: [sdah]
[ 2071.445603] Sense Key : Medium Error [current]
[ 2071.445607] Info fld=0xc8fd800
[ 2071.445612] sd 6:0:33:0: [sdah]
[ 2071.445614] Add. Sense: Unrecovered read error
[ 2071.445615] sd 6:0:33:0: [sdah] CDB:
[ 2071.445617] Read(10): 28 00 0c 8f d8 00 00 01 c8 00
[ 2071.445622] end_request: critical medium error, dev sdah, sector
210753536
[ 2201.018508] sd 6:0:33:0: [sdah] Unhandled sense code
[ 2201.018522] sd 6:0:33:0: [sdah]
[ 2201.018525] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 2201.018528] sd 6:0:33:0: [sdah]
[ 2201.018530] Sense Key : Medium Error [current]
[ 2201.018534] Info fld=0xc8fb450
[ 2201.018537] sd 6:0:33:0: [sdah]
[ 2201.018546] Add. Sense: Unrecovered read error
[ 2201.018551] sd 6:0:33:0: [sdah] CDB:
[ 2201.018552] Read(10): 28 00 0c 8f b4 48 00 00 38 00
[ 2201.018561] end_request: critical medium error, dev sdah, sector
210744400
[ 2203.651727] sd 6:0:33:0: [sdah] Unhandled sense code
[ 2203.651740] sd 6:0:33:0: [sdah]
[ 2203.651743] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 2203.651745] sd 6:0:33:0: [sdah]
[ 2203.651747] Sense Key : Medium Error [current]
[ 2203.651752] Info fld=0xc8fb450
[ 2203.651754] sd 6:0:33:0: [sdah]
[ 2203.651756] Add. Sense: Unrecovered read error
[ 2203.651759] sd 6:0:33:0: [sdah] CDB:
[ 2203.651761] Read(10): 28 00 0c 8f b4 50 00 00 30 00
[ 2203.651769] end_request: critical medium error, dev sdah, sector
210744400
[ 2204.845894] md/raid:md201: read error corrected (8 sectors at 996432
on sdah2)
[ 2204.845912] md/raid:md201: read error corrected (8 sectors at 996440
on sdah2)
[ 2204.845915] md/raid:md201: read error corrected (8 sectors at 996448
on sdah2)
[ 2204.845918] md/raid:md201: read error corrected (8 sectors at 996456
on sdah2)
[ 2204.845920] md/raid:md201: read error corrected (8 sectors at 996464
on sdah2)
[ 2204.845923] md/raid:md201: read error corrected (8 sectors at 996472
on sdah2)
Here is a time in which they get corrected a bit more often, but as you
can see most are still skipped:
[97939.727497] sd 6:0:33:0: [sdah] Unhandled sense code
[97939.727512] sd 6:0:33:0: [sdah]
[97939.727515] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97939.727518] sd 6:0:33:0: [sdah]
[97939.727520] Sense Key : Medium Error [current]
[97939.727524] Info fld=0xd439400
[97939.727526] sd 6:0:33:0: [sdah]
[97939.727529] Add. Sense: Unrecovered read error
[97939.727531] sd 6:0:33:0: [sdah] CDB:
[97939.727533] Read(10): 28 00 0d 43 94 00 00 00 28 00
[97939.727541] end_request: critical medium error, dev sdah, sector
222532608
[97942.216365] sd 6:0:33:0: [sdah] Unhandled sense code
[97942.216378] sd 6:0:33:0: [sdah]
[97942.216381] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97942.216382] sd 6:0:33:0: [sdah]
[97942.216384] Sense Key : Medium Error [current]
[97942.216387] Info fld=0xd439400
[97942.216388] sd 6:0:33:0: [sdah]
[97942.216390] Add. Sense: Unrecovered read error
[97942.216391] sd 6:0:33:0: [sdah] CDB:
[97942.216393] Read(10): 28 00 0d 43 94 00 00 00 28 00
[97942.216398] end_request: critical medium error, dev sdah, sector
222532608
[97942.625805] md/raid:md201: read error corrected (8 sectors at
12784640 on sdah2)
[97942.625884] md/raid:md201: read error corrected (8 sectors at
12784648 on sdah2)
[97942.625887] md/raid:md201: read error corrected (8 sectors at
12784656 on sdah2)
[97942.625888] md/raid:md201: read error corrected (8 sectors at
12784664 on sdah2)
[97942.625890] md/raid:md201: read error corrected (8 sectors at
12784672 on sdah2)
[98112.230660] sd 6:0:33:0: [sdah] Unhandled sense code
[98112.230687] sd 6:0:33:0: [sdah]
[98112.230690] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[98112.230692] sd 6:0:33:0: [sdah]
[98112.230694] Sense Key : Medium Error [current]
[98112.230698] Info fld=0xcbaca40
[98112.230700] sd 6:0:33:0: [sdah]
[98112.230703] Add. Sense: Unrecovered read error
[98112.230705] sd 6:0:33:0: [sdah] CDB:
[98112.230707] Read(10): 28 00 0c ba ca 40 00 00 08 00
[98112.230715] end_request: critical medium error, dev sdah, sector
213568064
[99107.714394] sd 6:0:33:0: [sdah] Unhandled sense code
[99107.714443] sd 6:0:33:0: [sdah]
[99107.714444] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[99107.714446] sd 6:0:33:0: [sdah]
[99107.714447] Sense Key : Medium Error [current]
[99107.714450] Info fld=0xcba46c8
[99107.714451] sd 6:0:33:0: [sdah]
[99107.714453] Add. Sense: Unrecovered read error
[99107.714455] sd 6:0:33:0: [sdah] CDB:
[99107.714456] Read(10): 28 00 0c ba 46 c0 00 00 20 00
[99107.714461] end_request: critical medium error, dev sdah, sector
213534408
[99110.123110] sd 6:0:33:0: [sdah] Unhandled sense code
[99110.123167] sd 6:0:33:0: [sdah]
[99110.123170] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[99110.123173] sd 6:0:33:0: [sdah]
[99110.123175] Sense Key : Medium Error [current]
[99110.123179] Info fld=0xcba46c8
[99110.123181] sd 6:0:33:0: [sdah]
[99110.123184] Add. Sense: Unrecovered read error
[99110.123187] sd 6:0:33:0: [sdah] CDB:
[99110.123189] Read(10): 28 00 0c ba 46 c0 00 00 20 00
[99110.123197] end_request: critical medium error, dev sdah, sector
213534408
[99111.169398] md/raid:md201: read error corrected (8 sectors at 3786440
on sdah2)
[99111.169404] md/raid:md201: read error corrected (8 sectors at 3786448
on sdah2)
[99111.169406] md/raid:md201: read error corrected (8 sectors at 3786456
on sdah2)
[101221.285568] mpt2sas0: _scsih_sas_broadcast_primitive_event: enter:
phy number(1), width(16)
[101221.288095] mpt2sas0: _scsih_sas_broadcast_primitive_event: enter:
phy number(1), width(16)
[101221.290937] mpt2sas0: _scsih_sas_broadcast_primitive_event: enter:
phy number(1), width(16)
[101221.293768] mpt2sas0: _scsih_sas_broadcast_primitive_event: enter:
phy number(1), width(16)
[101491.327771] sd 6:0:33:0: [sdah] Unhandled sense code
[101491.327813] sd 6:0:33:0: [sdah]
[101491.327815] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[101491.327817] sd 6:0:33:0: [sdah]
[101491.327819] Sense Key : Medium Error [current]
[101491.327822] Info fld=0xd2d7c1c
[101491.327824] sd 6:0:33:0: [sdah]
[101491.327826] Add. Sense: Unrecovered read error
[101491.327828] sd 6:0:33:0: [sdah] CDB:
[101491.327830] Read(10): 28 00 0d 2d 7c 18 00 00 08 00
[101491.327836] end_request: critical medium error, dev sdah, sector
221084700
[112965.864443] sd 6:0:33:0: [sdah] Unhandled sense code
[112965.864469] sd 6:0:33:0: [sdah]
[112965.864471] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[112965.864474] sd 6:0:33:0: [sdah]
[112965.864476] Sense Key : Medium Error [current]
[112965.864480] Info fld=0xc8e1cb1
[112968.322232] sd 6:0:33:0: [sdah]
[112968.322233] Add. Sense: Unrecovered read error
[112968.322235] sd 6:0:33:0: [sdah] CDB:
[112968.322236] Read(10): 28 00 0c 8e 1c 00 00 00 d8 00
[112968.322241] end_request: critical medium error, dev sdah, sector
210640049
[112969.127941] md/raid:md201: read error corrected (8 sectors at 892080
on sdah2)
[112969.127952] md/raid:md201: read error corrected (8 sectors at 892088
on sdah2)
[112969.127954] md/raid:md201: read error corrected (8 sectors at 892096
on sdah2)
[112969.127955] md/raid:md201: read error corrected (8 sectors at 892104
on sdah2)
[112969.127957] md/raid:md201: read error corrected (8 sectors at 892112
on sdah2)
[113352.100011] sd 6:0:33:0: [sdah] Unhandled sense code
[113352.100068] sd 6:0:33:0: [sdah]
[113352.100071] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[113352.100074] sd 6:0:33:0: [sdah]
[113352.100076] Sense Key : Medium Error [current]
[113352.100080] Info fld=0xc8e8448
[113352.100083] sd 6:0:33:0: [sdah]
[113352.100086] Add. Sense: Unrecovered read error
[113352.100088] sd 6:0:33:0: [sdah] CDB:
[113352.100090] Read(10): 28 00 0c 8e 84 30 00 00 38 00
[113352.100099] end_request: critical medium error, dev sdah, sector
210666568
[113354.850395] sd 6:0:33:0: [sdah] Unhandled sense code
[113354.850404] sd 6:0:33:0: [sdah]
[113354.850406] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[113354.850408] sd 6:0:33:0: [sdah]
[113354.850409] Sense Key : Medium Error [current]
[113354.850412] Info fld=0xc8e8448
[113354.850414] sd 6:0:33:0: [sdah]
[113354.850416] Add. Sense: Unrecovered read error
[113354.850417] sd 6:0:33:0: [sdah] CDB:
[113354.850419] Read(10): 28 00 0c 8e 84 30 00 00 38 00
[113354.850424] end_request: critical medium error, dev sdah, sector
210666568
[113355.387298] md/raid:md201: read error corrected (8 sectors at 918600
on sdah2)
[113355.387303] md/raid:md201: read error corrected (8 sectors at 918608
on sdah2)
[113355.387305] md/raid:md201: read error corrected (8 sectors at 918616
on sdah2)
[113355.387307] md/raid:md201: read error corrected (8 sectors at 918624
on sdah2)
As I wrote above, no error is noticed by userspace, so it actually
works, but I don't know why!?
Thanks for info
EW
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox