From: "John Stoffel" <john@stoffel.org>
To: Peter Bates <peter.thebates@gmail.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: mdadm - stuck reshape operation
Date: Wed, 27 Apr 2016 22:33:40 -0400 [thread overview]
Message-ID: <22305.30340.565924.810276@quad.stoffel.home> (raw)
In-Reply-To: <CAASHjeD2HqH=8ON_VSy1jbmR9NAK+NXQAmHjFw-PsW-gwNWB9g@mail.gmail.com>
>>>>> "Peter" == Peter Bates <peter.thebates@gmail.com> writes:
Peter> I have a 3 disk RAID 5 array that I tried to add a 4th disk to.
>> mdadm --add /dev/md6 /dev/sdb1
>> mdadm --grow --raid-devices=4 /dev/md6
Peter> This operation started successfully and proceeded until it hit 51.1%
>> cat /proc/mdstat
Peter> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
Peter> [raid4] [multipath] [faulty]
Peter> md6 : active raid5 sda1[0] sdb1[5] sdf1[3] sde1[4]
Peter> 3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
Peter> [==========>..........] reshape = 51.1% (998533632/1953382400)
Peter> finish=9046506.1min speed=1K/sec
Peter> bitmap: 0/15 pages [0KB], 65536KB chunk
Peter> It has been sitting on the same 998533632 position for
Peter> days. I've tried a few reboots, but it never progresses.
Peter> Stopping the array, or trying to start the logical volume in it
Peter> hangs. Altering the min / max speed parameters has no effect.
Peter> When I reboot and resemble the array the speed indicated
Peter> steadily drops to almost 0.
>> mdadm --assemble /dev/md6 --verbose --uuid 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2
I looked back in my email archives, and I wonder if maybe you have
SElinux enabled? If so, please turn it off and see if that helps.
What happens when you use dd on each of the drives and dump the output
to /dev/null?
Are there any messages in the logs, or dmesg output after the stuff
you showed? Can you maybe 'strace' the mdadm process, or even go grab
the latest version using git from:
git clone git://neil.brown.name/mdadm
And see if compiling it yourself from the master might do the trick.
Peter> I haven't tried anything more drastic than a reboot yet,
Peter> Below is as much information as I can think to provide at this stage.
Peter> Please let me know what else I can do.
Peter> I'm happy to change kernels, kernel config or anything else require to
Peter> get better info.
Peter> Kernel: 4.4.3
Peter> mdadm 3.4
>> ps aux | grep md6
Peter> root 5041 99.9 0.0 0 0 ? R 07:10 761:58 [md6_raid5]
Peter> root 5042 0.0 0.0 0 0 ? D 07:10 0:00 [md6_reshape]
Peter> This is consistent. 100% cpu on the raid component, but not the reshape
>> mdadm --detail --verbose /dev/md6
Peter> /dev/md6:
Peter> Version : 1.2
Peter> Creation Time : Fri Aug 29 21:13:52 2014
Peter> Raid Level : raid5
Peter> Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Peter> Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Peter> Raid Devices : 4
Peter> Total Devices : 4
Peter> Persistence : Superblock is persistent
Peter> Intent Bitmap : Internal
Peter> Update Time : Wed Apr 27 07:10:07 2016
Peter> State : clean, reshaping
Peter> Active Devices : 4
Peter> Working Devices : 4
Peter> Failed Devices : 0
Peter> Spare Devices : 0
Peter> Layout : left-symmetric
Peter> Chunk Size : 512K
Peter> Reshape Status : 51% complete
Peter> Delta Devices : 1, (3->4)
Peter> Name : Alpheus:6 (local to host Alpheus)
Peter> UUID : 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2
Peter> Events : 47975
Peter> Number Major Minor RaidDevice State
Peter> 0 8 1 0 active sync /dev/sda1
Peter> 4 8 65 1 active sync /dev/sde1
Peter> 3 8 81 2 active sync /dev/sdf1
Peter> 5 8 17 3 active sync /dev/sdb1
>> iostat
Peter> Linux 4.4.3-gentoo (Alpheus) 04/27/2016 _x86_64_ (4 CPU)
Peter> avg-cpu: %user %nice %system %iowait %steal %idle
Peter> 1.84 0.00 24.50 0.09 0.00 73.57
Peter> Looking at the individual disks I can see minor activity on the MD6
Peter> members. This activity tends to match up with the overall rate
Peter> reported by /proc/mdstat
Peter> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
Peter> sda 0.02 2.72 1.69 128570 79957
Peter> sdb 0.01 0.03 1.69 1447 79889
Peter> sdd 3.85 2.27 56.08 106928 2646042
Peter> sde 0.02 2.73 1.69 128610 79961
Peter> sdf 0.02 2.72 1.69 128128 79961
Peter> sdc 4.08 5.44 56.08 256899 2646042
Peter> md0 2.91 7.62 55.08 359714 2598725
Peter> dm-0 0.00 0.03 0.00 1212 0
Peter> dm-1 0.00 0.05 0.00 2151 9
Peter> dm-2 2.65 6.52 3.42 307646 161296
Peter> dm-3 0.19 1.03 51.66 48377 2437420
Peter> md6 0.00 0.02 0.00 1036 0
>> dmesg
Peter> [ 1199.426995] md: bind<sde1>
Peter> [ 1199.427779] md: bind<sdf1>
Peter> [ 1199.428379] md: bind<sdb1>
Peter> [ 1199.428592] md: bind<sda1>
Peter> [ 1199.429260] md/raid:md6: reshape will continue
Peter> [ 1199.429274] md/raid:md6: device sda1 operational as raid disk 0
Peter> [ 1199.429275] md/raid:md6: device sdb1 operational as raid disk 3
Peter> [ 1199.429276] md/raid:md6: device sdf1 operational as raid disk 2
Peter> [ 1199.429277] md/raid:md6: device sde1 operational as raid disk 1
Peter> [ 1199.429498] md/raid:md6: allocated 4338kB
Peter> [ 1199.429807] md/raid:md6: raid level 5 active with 4 out of 4
Peter> devices, algorithm 2
Peter> [ 1199.429810] RAID conf printout:
Peter> [ 1199.429811] --- level:5 rd:4 wd:4
Peter> [ 1199.429812] disk 0, o:1, dev:sda1
Peter> [ 1199.429814] disk 1, o:1, dev:sde1
Peter> [ 1199.429816] disk 2, o:1, dev:sdf1
Peter> [ 1199.429817] disk 3, o:1, dev:sdb1
Peter> [ 1199.429993] created bitmap (15 pages) for device md6
Peter> [ 1199.430297] md6: bitmap initialized from disk: read 1 pages, set 0
Peter> of 29807 bits
Peter> [ 1199.474604] md6: detected capacity change from 0 to 4000527155200
Peter> [ 1199.474611] md: reshape of RAID array md6
Peter> [ 1199.474613] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Peter> [ 1199.474614] md: using maximum available idle IO bandwidth (but not
Peter> more than 200000 KB/sec) for reshape.
Peter> [ 1199.474617] md: using 128k window, over a total of 1953382400k.
>> lsblk
Peter> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
Peter> sda 8:0 0 1.8T 0 disk
Peter> └─sda1 8:1 0 1.8T 0 part
Peter> └─md6 9:6 0 3.7T 0 raid5
Peter> sdb 8:16 0 1.8T 0 disk
Peter> └─sdb1 8:17 0 1.8T 0 part
Peter> └─md6 9:6 0 3.7T 0 raid5
Peter> sdc 8:32 0 2.7T 0 disk
Peter> ├─sdc1 8:33 0 16M 0 part
Peter> └─sdc2 8:34 0 2.7T 0 part
Peter> └─md0 9:0 0 2.7T 0 raid1
Peter> ├─vg--mirror-swap 253:0 0 4G 0 lvm [SWAP]
Peter> ├─vg--mirror-boot 253:1 0 256M 0 lvm /boot
Peter> ├─vg--mirror-root 253:2 0 256G 0 lvm /
Peter> └─vg--mirror-data--mirror 253:3 0 2.5T 0 lvm /data/mirror
Peter> sdd 8:48 0 2.7T 0 disk
Peter> ├─sdd1 8:49 0 16M 0 part
Peter> └─sdd2 8:50 0 2.7T 0 part
Peter> └─md0 9:0 0 2.7T 0 raid1
Peter> ├─vg--mirror-swap 253:0 0 4G 0 lvm [SWAP]
Peter> ├─vg--mirror-boot 253:1 0 256M 0 lvm /boot
Peter> ├─vg--mirror-root 253:2 0 256G 0 lvm /
Peter> └─vg--mirror-data--mirror 253:3 0 2.5T 0 lvm /data/mirror
Peter> sde 8:64 0 1.8T 0 disk
Peter> └─sde1 8:65 0 1.8T 0 part
Peter> └─md6 9:6 0 3.7T 0 raid5
Peter> sdf 8:80 0 1.8T 0 disk
Peter> └─sdf1 8:81 0 1.8T 0 part
Peter> └─md6 9:6 0 3.7T 0 raid5
Peter> Thanks for any pointers
Peter> Peter Bates
Peter> peter.thebates@gmail.com
Peter> --
Peter> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Peter> the body of a message to majordomo@vger.kernel.org
Peter> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2016-04-28 2:33 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-27 12:21 mdadm - stuck reshape operation Peter Bates
2016-04-28 2:33 ` John Stoffel [this message]
2016-04-28 21:31 ` Peter Bates
2016-05-17 20:01 ` Erkki Seppala
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=22305.30340.565924.810276@quad.stoffel.home \
--to=john@stoffel.org \
--cc=linux-raid@vger.kernel.org \
--cc=peter.thebates@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox