* Re: [PATCH] IMSM: do not use comma expression for simple assignments
From: Artur Paszkiewicz @ 2014-12-11 9:06 UTC (permalink / raw)
To: Jürg Billeter, NeilBrown; +Cc: linux-raid, Baldysiak, Pawel
In-Reply-To: <1407313179-24598-1-git-send-email-j@bitron.ch>
On 08/06/2014 10:19 AM, Jürg Billeter wrote:
> Fixes build with gcc 4.9.
>
> error: right-hand operand of comma expression has no effect
> snprintf((char *) spare->sig, MAX_SIGNATURE_LENGTH,
> ^
>
> Signed-off-by: Jürg Billeter <j@bitron.ch>
> ---
> super-intel.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/super-intel.c b/super-intel.c
> index e28ac7d..f24d101 100644
> --- a/super-intel.c
> +++ b/super-intel.c
> @@ -5069,13 +5069,13 @@ static int write_super_imsm_spares(struct intel_super *super, int doclose)
> __u32 sum;
> struct dl *d;
>
> - spare->mpb_size = __cpu_to_le32(sizeof(struct imsm_super)),
> - spare->generation_num = __cpu_to_le32(1UL),
> + spare->mpb_size = __cpu_to_le32(sizeof(struct imsm_super));
> + spare->generation_num = __cpu_to_le32(1UL);
> spare->attributes = MPB_ATTRIB_CHECKSUM_VERIFY;
> - spare->num_disks = 1,
> - spare->num_raid_devs = 0,
> - spare->cache_size = mpb->cache_size,
> - spare->pwr_cycle_count = __cpu_to_le32(1),
> + spare->num_disks = 1;
> + spare->num_raid_devs = 0;
> + spare->cache_size = mpb->cache_size;
> + spare->pwr_cycle_count = __cpu_to_le32(1);
>
> snprintf((char *) spare->sig, MAX_SIGNATURE_LENGTH,
> MPB_SIGNATURE MPB_VERSION_RAID0);
>
Hi Neil,
We verified that this patch indeed is needed to build mdadm with gcc
4.9. Can you apply it?
Thanks,
Artur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Growing RAID-10 (near-2) array - which kernel needed?
From: cvb @ 2014-12-11 18:31 UTC (permalink / raw)
To: linux-raid
Apologies for raising this topic again, but I can't seem to find
anything googling, and man mdadm doesn't also give me a clue. Which
kernel version is needed to grow a near-2 RAID 10 array?
I'm currently running kernel 3.13.0 with mdadm 3.2.5. But I could also
use a live CD with a newer kernel, if needed.
I have replaced the 4 disks here with larger ones (2TB drives -> 4 TB
drives), and this is what the array currently looks like:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Aug 20 08:15:07 2011
Raid Level : raid10
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Thu Dec 11 08:31:15 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Name : alpha:0 (local to host alpha)
UUID : 8199e640:58fc3a04:99b52fe6:80a1574d
Events : 918971
Number Major Minor RaidDevice State
6 8 1 0 active sync /dev/sda1
5 8 33 1 active sync /dev/sdc1
7 8 17 2 active sync /dev/sdb1
4 8 49 3 active sync /dev/sdd1
^ permalink raw reply
* Re: Growing RAID-10 (near-2) array - which kernel needed?
From: Robin Hill @ 2014-12-11 19:21 UTC (permalink / raw)
To: cvb; +Cc: linux-raid
In-Reply-To: <b906dbe55e09c7cf24abcacfbf97efed@kruemel.org>
[-- Attachment #1: Type: text/plain, Size: 2416 bytes --]
On Thu Dec 11, 2014 at 07:31:31PM +0100, cvb@kruemel.org wrote:
> Apologies for raising this topic again, but I can't seem to find
> anything googling, and man mdadm doesn't also give me a clue. Which
> kernel version is needed to grow a near-2 RAID 10 array?
>
Full reshape support (changing number of devices, layout or chunk size)
was introduced in 3.5 (http://kernelnewbies.org/Linux_3.5).
> I'm currently running kernel 3.13.0 with mdadm 3.2.5. But I could also
> use a live CD with a newer kernel, if needed.
>
It also requires mdadm 3.3 or above.
> I have replaced the 4 disks here with larger ones (2TB drives -> 4 TB
> drives), and this is what the array currently looks like:
>
> # mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Sat Aug 20 08:15:07 2011
> Raid Level : raid10
> Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
> Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
> Raid Devices : 4
> Total Devices : 4
> Persistence : Superblock is persistent
>
> Update Time : Thu Dec 11 08:31:15 2014
> State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : near=2
> Chunk Size : 512K
>
> Name : alpha:0 (local to host alpha)
> UUID : 8199e640:58fc3a04:99b52fe6:80a1574d
> Events : 918971
>
> Number Major Minor RaidDevice State
> 6 8 1 0 active sync /dev/sda1
> 5 8 33 1 active sync /dev/sdc1
> 7 8 17 2 active sync /dev/sdb1
> 4 8 49 3 active sync /dev/sdd1
>
I can't find anything explicitly covering growing a RAID 10 by just
increasing the device sizes. I would have thought that would be an
easier task than doing so by adding disks, so should be supported in an
earlier version. mdadm 3.2.1 mentions supporting converting between
RAID-10 and RAID-1, which can definitely be grown.
Have you tried doing a grow with the versions you have?
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Growing RAID-10 (near-2) array - which kernel needed?
From: NeilBrown @ 2014-12-11 20:28 UTC (permalink / raw)
To: Robin Hill; +Cc: cvb, linux-raid
In-Reply-To: <20141211192139.GA4233@cthulhu.home.robinhill.me.uk>
[-- Attachment #1: Type: text/plain, Size: 2789 bytes --]
On Thu, 11 Dec 2014 19:21:39 +0000 Robin Hill <robin@robinhill.me.uk> wrote:
> On Thu Dec 11, 2014 at 07:31:31PM +0100, cvb@kruemel.org wrote:
>
> > Apologies for raising this topic again, but I can't seem to find
> > anything googling, and man mdadm doesn't also give me a clue. Which
> > kernel version is needed to grow a near-2 RAID 10 array?
> >
> Full reshape support (changing number of devices, layout or chunk size)
> was introduced in 3.5 (http://kernelnewbies.org/Linux_3.5).
>
> > I'm currently running kernel 3.13.0 with mdadm 3.2.5. But I could also
> > use a live CD with a newer kernel, if needed.
> >
> It also requires mdadm 3.3 or above.
>
> > I have replaced the 4 disks here with larger ones (2TB drives -> 4 TB
> > drives), and this is what the array currently looks like:
> >
> > # mdadm --detail /dev/md0
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Sat Aug 20 08:15:07 2011
> > Raid Level : raid10
> > Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
> > Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
> > Raid Devices : 4
> > Total Devices : 4
> > Persistence : Superblock is persistent
> >
> > Update Time : Thu Dec 11 08:31:15 2014
> > State : clean
> > Active Devices : 4
> > Working Devices : 4
> > Failed Devices : 0
> > Spare Devices : 0
> >
> > Layout : near=2
> > Chunk Size : 512K
> >
> > Name : alpha:0 (local to host alpha)
> > UUID : 8199e640:58fc3a04:99b52fe6:80a1574d
> > Events : 918971
> >
> > Number Major Minor RaidDevice State
> > 6 8 1 0 active sync /dev/sda1
> > 5 8 33 1 active sync /dev/sdc1
> > 7 8 17 2 active sync /dev/sdb1
> > 4 8 49 3 active sync /dev/sdd1
> >
> I can't find anything explicitly covering growing a RAID 10 by just
> increasing the device sizes.
commit 006a09a0ae0a494473a8cd82c8d1d653e37e6663
Author: NeilBrown <neilb@suse.de>
Date: Mon Mar 19 12:46:40 2012 +1100
md/raid10 - support resizing some RAID10 arrays.
$ git describe --contains 006a09a0ae0a494473a8cd82c8d1d653e37e6663
v3.4-rc1~147^2~8
So Linux-3.4 is needed.
I don't think you need a particularly new mdadm.
NeilBrown
> I would have thought that would be an
> easier task than doing so by adding disks, so should be supported in an
> earlier version. mdadm 3.2.1 mentions supporting converting between
> RAID-10 and RAID-1, which can definitely be grown.
>
> Have you tried doing a grow with the versions you have?
>
> Cheers,
> Robin
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Growing RAID-10 (near-2) array - which kernel needed?
From: cvb @ 2014-12-12 11:04 UTC (permalink / raw)
To: linux-raid, robin
In-Reply-To: <20141211192139.GA4233@cthulhu.home.robinhill.me.uk>
> Have you tried doing a grow with the versions you have?
Not yet, I thought it'd be wise to ask first, as this appears to be
nowhere documented. And I like restricting the cases where I need to
revert to my backup to the absolute minimum. :-)
Thanks, Chris
^ permalink raw reply
* Re: Growing RAID-10 (near-2) array - which kernel needed?
From: cvb @ 2014-12-12 12:03 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141212072814.65b1e134@notabene.brown>
> So Linux-3.4 is needed.
> I don't think you need a particularly new mdadm.
Ok, thanks. Will try it with what I have, and report back.
Chris
^ permalink raw reply
* raid10: 6 out of 8 disks marked as stale on every restart
From: Peter Kieser @ 2014-12-12 17:04 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 5408 bytes --]
Hello,
I have a 8 disk RAID10 array, 6 of the disks are on an LSISAS2008
controller and 2 are on a 82801JI (ICH10 Family) SATA AHCI controller.
I upgraded the kernel from 3.17.1 to 3.17.6 when the issue I am having
started to occur, but reverting to an older kernel does not resolve the
issue.
Restarting the machine causes the array not to start (or be visible in
/proc/mdstat or any mention in kernel messages.) If I try to assemble
the drives, mdraid complains that 6 out of the 8 disks (coincidentally
all on the LSISAS2008 controller) are non-fresh:
root@kvm:~# mdadm --assemble /dev/md3 /dev/sde /dev/sdf /dev/sdg
/dev/sdh /dev/sdi /dev/sdj /dev/sda /dev/sdb
Dec 11 21:08:25 kvm kernel: [ 528.503736] md: kicking non-fresh sdi
from array!
Dec 11 21:08:25 kvm kernel: [ 528.503747] md: unbind<sdi>
Dec 11 21:08:25 kvm kernel: [ 528.523775] md: export_rdev(sdi)
Dec 11 21:08:25 kvm kernel: [ 528.523802] md: kicking non-fresh sdg
from array!
Dec 11 21:08:25 kvm kernel: [ 528.523809] md: unbind<sdg>
Dec 11 21:08:25 kvm kernel: [ 528.531753] md: export_rdev(sdg)
Dec 11 21:08:25 kvm kernel: [ 528.531780] md: kicking non-fresh sdf
from array!
Dec 11 21:08:25 kvm kernel: [ 528.531788] md: unbind<sdf>
Dec 11 21:08:25 kvm kernel: [ 528.539749] md: export_rdev(sdf)
Dec 11 21:08:25 kvm kernel: [ 528.539776] md: kicking non-fresh sdh
from array!
Dec 11 21:08:25 kvm kernel: [ 528.539785] md: unbind<sdh>
Dec 11 21:08:25 kvm kernel: [ 528.547744] md: export_rdev(sdh)
Dec 11 21:08:25 kvm kernel: [ 528.547771] md: kicking non-fresh sdj
from array!
Dec 11 21:08:25 kvm kernel: [ 528.547779] md: unbind<sdj>
Dec 11 21:08:25 kvm kernel: [ 528.555755] md: export_rdev(sdj)
Dec 11 21:08:25 kvm kernel: [ 528.555782] md: kicking non-fresh sde
from array!
Dec 11 21:08:25 kvm kernel: [ 528.555790] md: unbind<sde>
Dec 11 21:08:25 kvm kernel: [ 528.563758] md: export_rdev(sde)
Dec 11 21:08:25 kvm kernel: [ 528.565831] md/raid10:md3: not enough
operational mirrors.
Dec 11 21:08:25 kvm kernel: [ 528.567230] md: pers->run() failed ...
/dev/sda and /dev/sdb are the only drives not on the LSI controller. If
I force the assembly with 6 out of the 8 drives the RAID array comes up:
root@kvm:~# mdadm --assemble /dev/md3 /dev/sde /dev/sdf /dev/sdg
/dev/sdh /dev/sdi /dev/sdj --run
Then I add the extra drives:
root@kvm:~# mdadm --manage /dev/md3 --add /dev/sda
root@kvm:~# mdadm --manage /dev/md3 --add /dev/sdb
root@kvm:~# mdadm --detail /dev/md3
/dev/md3:
Version : 1.0
Creation Time : Thu Sep 12 18:43:56 2013
Raid Level : raid10
Array Size : 7814055936 (7452.06 GiB 8001.59 GB)
Used Dev Size : 1953513984 (1863.02 GiB 2000.40 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Fri Dec 12 08:58:19 2014
State : active, degraded, recovering
Active Devices : 6
Working Devices : 8
Failed Devices : 0
Spare Devices : 2
Layout : near=2
Chunk Size : 512K
Rebuild Status : 76% complete
Name : kvm.taylor.kieser.ca:3
UUID : f0bc8469:9879a709:e4cc94a7:521bd273
Events : 82901
Number Major Minor RaidDevice State
0 8 128 0 active sync /dev/sdi
8 8 96 1 active sync /dev/sdg
11 8 0 2 spare rebuilding /dev/sda
3 8 112 3 active sync /dev/sdh
4 0 0 4 removed
10 8 80 5 active sync /dev/sdf
6 8 64 6 active sync /dev/sde
9 8 144 7 active sync /dev/sdj
12 8 16 - spare /dev/sdb
This occurs every time I restart the machine. Thoughts? I tried
rebuilding the initramfs but this didn't resolve the issue. I'm also
running bcache on this machine, but on top of the mdraid.
/etc/mdadm.conf:
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.0 UUID=3b174514:49f3e22e:550cf9a7:8ed93920
name=linux:0
ARRAY /dev/md/1 metadata=1.0 UUID=8e23f81d:73f9b393:addd1f7f:5ee1833a
name=linux:1
ARRAY /dev/md/2 metadata=1.0 UUID=cc5a0495:b5262855:fb3cd40a:8b237162
name=kvm.taylor.kieser.ca:2
ARRAY /dev/md/3 metadata=1.0 UUID=f0bc8469:9879a709:e4cc94a7:521bd273
name=kvm.taylor.kieser.ca:3
root@kvm:~# uname -a
Linux kvm 3.17.6 #3 SMP Sun Dec 7 12:16:45 PST 2014 x86_64 x86_64 x86_64
GNU/Linux
root@kvm:~# mdadm -V
mdadm - v3.2.5 - 18th May 2012
root@kvm:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md127 : inactive sdk[2](S)
1465138448 blocks super 1.0
md3 : active raid10 sdb[12](S) sda[11] sdi[0] sdj[9] sde[6] sdf[10]
sdh[3] sdg[8]
7814055936 blocks super 1.0 512K chunks 2 near-copies [8/6]
[UU_U_UUU]
[===============>.....] recovery = 76.6% (1498279040/1953513984)
finish=4710.1min speed=1610K/sec
md1 : active raid1 sdd5[3] sdc5[2]
25164672 blocks super 1.0 [2/2] [UU]
md0 : active raid1 sdd1[3] sdc1[2]
16779136 blocks super 1.0 [2/2] [UU]
md2 : active raid1 sdd6[3] sdc6[2]
192472960 blocks super 1.0 [2/2] [UU]
unused devices: <none>
-Peter
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4291 bytes --]
^ permalink raw reply
* Lost raid 5 volume
From: Neil . @ 2014-12-13 0:48 UTC (permalink / raw)
To: linux-raid
I am looking for some help in trying to recover a raid 5 volume. Is
this the right place? What data should I provide to get the ball
rolling?
The back story is its 4 x 6tb drives in raid 5. Was created on a WD
EX4 NAS box. The NAS reported 2 simultaneous drive failures on wake up
from standby mode. On reboot both drives appear fine. No SMART errors
etc but the raid 5 volume is gone.
It's the second time the NAS box has lost data like this. Previous
time it was a mix of 2 and 3 tb drives with a 6 tb raid 5 and a 1 tb
mirror. mdadm on the nas is from 2006 (don't have the exact version to
hand).
I was thinking to put the drives into a PC, boot to a recent distro on
usb stick and see if I can recover the data there. Then setup it up
more or less permanently (and ditch the NAS). Not sure of the best way
to proceed.
I'd be really grateful for any assistance or guidance.
Cheers
^ permalink raw reply
* Re: Lost raid 5 volume
From: Emery Guevremont @ 2014-12-14 0:06 UTC (permalink / raw)
To: Neil .; +Cc: linux-raid
In-Reply-To: <CADhQzuraHNZ1MaMw4OP0-3+vxRcB_TAN-_R2_eaTOQcEfUv8+w@mail.gmail.com>
Stop your raid with mdadm -S /dev/md0 or whatever your raid device name is.
Probably the safest thing to do is clone your drives with ddrescue.
You might also want to view your S.M.A.R.T. log with smarctl -a
/dev/sda or which ever is your device name.
After, what you'd need to start doing is taking a backup of mdadm
--examine /dev/sda1 or which ever partition is your partition used for
raid. This will give us info on you md superblock. Post this info and
from there we'll be able to see how everything is setup and have a
better idea of your current situation.
On Fri, Dec 12, 2014 at 7:48 PM, Neil . <neil.perrie@gmail.com> wrote:
> I am looking for some help in trying to recover a raid 5 volume. Is
> this the right place? What data should I provide to get the ball
> rolling?
>
> The back story is its 4 x 6tb drives in raid 5. Was created on a WD
> EX4 NAS box. The NAS reported 2 simultaneous drive failures on wake up
> from standby mode. On reboot both drives appear fine. No SMART errors
> etc but the raid 5 volume is gone.
>
> It's the second time the NAS box has lost data like this. Previous
> time it was a mix of 2 and 3 tb drives with a 6 tb raid 5 and a 1 tb
> mirror. mdadm on the nas is from 2006 (don't have the exact version to
> hand).
>
> I was thinking to put the drives into a PC, boot to a recent distro on
> usb stick and see if I can recover the data there. Then setup it up
> more or less permanently (and ditch the NAS). Not sure of the best way
> to proceed.
>
> I'd be really grateful for any assistance or guidance.
>
> Cheers
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [GIT PULL REQUEST] md updates for 3.19
From: NeilBrown @ 2014-12-14 1:37 UTC (permalink / raw)
To: Linus Torvalds; +Cc: lkml, linux RAID, Henry Cai
[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]
hi Linus,
please pull these three patches.
I did have a largish set of locking changes queued, but late testing showed
they weren't quite as stable as I thought and while I fixed what I found, I
decided it safer to delay them a release ... particularly as I'll be AFK for
a few weeks. So expect a larger batch next time :-)
Thanks,
NeilBrow
The following changes since commit 3a18ca061311f2f1ee9c44012f89c7436d392117:
Merge tag 'ext4_for_linus_urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4 (2014-12-01 20:11:49 -0800)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/3.19
for you to fetch changes up to f851b60db0fd83a10034c5cc9d9e58c758457b1c:
md: Check MD_RECOVERY_RUNNING as well as ->sync_thread. (2014-12-11 10:02:10 +1100)
----------------------------------------------------------------
Three fixes for md
----------------------------------------------------------------
NeilBrown (2):
md/raid5: fetch_block must fetch all the blocks handle_stripe_dirtying wants.
md: Check MD_RECOVERY_RUNNING as well as ->sync_thread.
kbuild test robot (1):
md: fix semicolon.cocci warnings
drivers/md/md.c | 38 +++++++++++++++++++++++++++-----------
drivers/md/raid5.c | 7 +++++--
2 files changed, 32 insertions(+), 13 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Lost raid 5 volume
From: Neil . @ 2014-12-14 4:30 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8sZUdZkLRPQXS+ANam=k4Tvjynkh8UzgL5WXE3qBwYUK6Q@mail.gmail.com>
smartctl says Overall health...: passed for all drives.
This is interesting(?)
mdadm --examine for sda2 and b2 list:
Array State as : AAAA
But sdc2 and d2 list it as ..AA
root@sysresccd /mnt % mdadm --examine /dev/sda2
/dev/sda2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : cf0bf1b9:f57b96be:8c749fcb:cea10311
Name : 'localhost.localdomain':1
Creation Time : Sat Oct 25 16:11:56 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11712653168 (5585.03 GiB 5996.88 GB)
Array Size : 17568979392 (16755.08 GiB 17990.63 GB)
Used Dev Size : 11712652928 (5585.03 GiB 5996.88 GB)
Super Offset : 11712653296 sectors
Unused Space : before=0 sectors, after=368 sectors
State : clean
Device UUID : 0953e84c:a25760b0:28a20bab:bd1dc41b
Internal Bitmap : 2 sectors from superblock
Update Time : Sat Dec 6 14:00:02 2014
Checksum : dfe27505 - correct
Events : 3
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@sysresccd /mnt % mdadm --examine /dev/sdb2
/dev/sdb2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : cf0bf1b9:f57b96be:8c749fcb:cea10311
Name : 'localhost.localdomain':1
Creation Time : Sat Oct 25 16:11:56 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11712653168 (5585.03 GiB 5996.88 GB)
Array Size : 17568979392 (16755.08 GiB 17990.63 GB)
Used Dev Size : 11712652928 (5585.03 GiB 5996.88 GB)
Super Offset : 11712653296 sectors
Unused Space : before=0 sectors, after=368 sectors
State : clean
Device UUID : d96a56d4:c5ac346a:24765692:501f6f22
Internal Bitmap : 2 sectors from superblock
Update Time : Sat Dec 6 14:00:02 2014
Checksum : f1ab789 - correct
Events : 3
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@sysresccd /mnt % mdadm --examine /dev/sdc2
/dev/sdc2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : cf0bf1b9:f57b96be:8c749fcb:cea10311
Name : 'localhost.localdomain':1
Creation Time : Sat Oct 25 16:11:56 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11712653168 (5585.03 GiB 5996.88 GB)
Array Size : 17568979392 (16755.08 GiB 17990.63 GB)
Used Dev Size : 11712652928 (5585.03 GiB 5996.88 GB)
Super Offset : 11712653296 sectors
Unused Space : before=0 sectors, after=368 sectors
State : clean
Device UUID : 17d61cf2:2c0c4765:cb4c478c:4828aefc
Internal Bitmap : 2 sectors from superblock
Update Time : Sun Dec 7 11:18:06 2014
Checksum : fc228d5e - correct
Events : 8
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing, 'R' == replacing)
root@sysresccd /mnt % mdadm --examine /dev/sdd2
/dev/sdd2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : cf0bf1b9:f57b96be:8c749fcb:cea10311
Name : 'localhost.localdomain':1
Creation Time : Sat Oct 25 16:11:56 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11712653168 (5585.03 GiB 5996.88 GB)
Array Size : 17568979392 (16755.08 GiB 17990.63 GB)
Used Dev Size : 11712652928 (5585.03 GiB 5996.88 GB)
Super Offset : 11712653296 sectors
Unused Space : before=0 sectors, after=368 sectors
State : clean
Device UUID : 306bde8a:3beebe54:52e6acd2:f7681367
Internal Bitmap : 2 sectors from superblock
Update Time : Sun Dec 7 11:18:06 2014
Checksum : 3526df7d - correct
Events : 8
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing, 'R' == replacing)
On 14 December 2014 at 00:06, Emery Guevremont
<emery.guevremont@gmail.com> wrote:
> Stop your raid with mdadm -S /dev/md0 or whatever your raid device name is.
>
> Probably the safest thing to do is clone your drives with ddrescue.
> You might also want to view your S.M.A.R.T. log with smarctl -a
> /dev/sda or which ever is your device name.
>
> After, what you'd need to start doing is taking a backup of mdadm
> --examine /dev/sda1 or which ever partition is your partition used for
> raid. This will give us info on you md superblock. Post this info and
> from there we'll be able to see how everything is setup and have a
> better idea of your current situation.
>
> On Fri, Dec 12, 2014 at 7:48 PM, Neil . <neil.perrie@gmail.com> wrote:
>> I am looking for some help in trying to recover a raid 5 volume. Is
>> this the right place? What data should I provide to get the ball
>> rolling?
>>
-snip-
^ permalink raw reply
* Re: "bitmap file is out of date, doing full recovery"
From: Alexander Lyakas @ 2014-12-14 12:11 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <CAGRgLy4YjfUtziFfFyZmC5SHGm+PLUzmLACJfanBBN8SS7JjCg@mail.gmail.com>
Hi Neil,
This issue keeps happening to us. Do you see any problem in always
incrementing the event count?
Thanks,
Alex.
On Tue, Nov 4, 2014 at 11:17 AM, Alexander Lyakas
<alex.bolshoy@gmail.com> wrote:
> Hi Neil,
> thank you for your comments.
>
> On Wed, Oct 29, 2014 at 1:19 AM, NeilBrown <neilb@suse.de> wrote:
>> On Thu, 23 Oct 2014 19:04:48 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
>> wrote:
>>
>>> Hi Neil,
>>> I found at least one way of this happening. The problem is that in
>>> md_update_sb() we allow to decrease the event count:
>>>
>>> /* If this is just a dirty<->clean transition, and the array is clean
>>> * and 'events' is odd, we can roll back to the previous clean state */
>>> if (nospares
>>> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
>>> && mddev->can_decrease_events
>>> && mddev->events != 1) {
>>> mddev->events--;
>>> mddev->can_decrease_events = 0;
>>>
>>> Then we call bitmap_update_sb(). If we crash after we update (the
>>> first or all of) bitmap superblocks, then after reboot, we will see
>>> that bitmap event count is less than MD superblock event count. Then
>>> we decide to do full resync.
>>>
>>> This can be easily reproduced by hacking bitmap_update_sb() to call
>>> BUG(), after it calls write_page() in case event count was decreased.
>>>
>>> Why we are decreasing the event count??? Can we always increase it?
>>> u64 is a lot to increase...
>>
>> The reason for decreasing the event count is so that we don't need to update
>> the event count on spares - they can be left spun down.
>> We for simple clean/dirty transitions with increment for clean->dirty and
>> decrement for dirty->clean. But we should only use this optimisation when
>> everything is simple.
>> We really shouldn't do this when the array is degraded.
>> Do this fix your problem?
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 2c73fcb82593..98fd97b10e13 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -2244,6 +2244,7 @@ repeat:
>> * and 'events' is odd, we can roll back to the previous clean state */
>> if (nospares
>> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
>> + && mddev->degraded == 0
>> && mddev->can_decrease_events
>> && mddev->events != 1) {
>> mddev->events--;
>>
>>
> No, unfortunately, this doesn't fix the problem. In my case, the array
> is never degraded. Both drives are present and operational, then the
> box crashes, and after reboot the bitmap event counter is lower than
> we expect. Again, this is easily reproduced by hacking
> bitmap_update_sb() as I mentioned earlier.
>
> In my case array does not have spares. (There is some other system on
> top, which monitors the array, and, if needed, adds a spare from a
> "global" spare pool). Is this ok in this case to always increment the
> event count?
>
> Thanks,
> Alex.
>
>
>>>
>>> Some other doubt that I have is that bitmap_unplug() and
>>> bitmap_daemon_work() call write_page() on page index=0. This page
>>> contains both the superblock and also some dirty bits (could not we
>>> waste 4KB on bitmap superblock???). I am not sure, but I wonder
>>> whether this call can race with md_update_sb (which explicitly calls
>>> bitmap_update_sb), and somehow write the outdated superblock, after
>>> bitmap_update_sb has completed writing it.
>>>
>>
>> storage.sb_page is exactly the same as storage.filemap[0]
>> So once an update has happened, the "outdated superblock" doesn't exist
>> anywhere to be written out from.
>>
>>> Yet another suspect is when loading the bitmap we basically load it
>>> from the first up-to-date drive. Maybe we should have scanned all the
>>> bitmap superblocks, and selected one that has the higher event count
>>> (although as we saw "higher" does not necessarily mean "more
>>> up-to-date").
>>>
>>> Anyways, back to decrementing the event count. Do you see any issue
>>> with not doing this and always incrementing?
>>>
>>> Thanks,
>>> Alex.
>>>
>>
>> Thanks,
>> NeilBrown
^ permalink raw reply
* Re: Lost raid 5 volume
From: Phil Turmel @ 2014-12-14 16:23 UTC (permalink / raw)
To: Neil ., Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CADhQzuo57BoMT_fLVHd=ufGR6ztSFaRqsRLZtFSwvii+YzAcLQ@mail.gmail.com>
Good morning Neil,
[Etiquette on kernel.org is to trim replies and either bottom-post or
interleave.]
On 12/13/2014 11:30 PM, Neil . wrote:
> smartctl says Overall health...: passed for all drives.
This is good, but not relevant. Lots of desktop drives will report this
even though they are part of the problem. Please show the complete
output of "smartctl -x" for the two troublesome drives. Maybe also for
the other two.
> Update Time : Sat Dec 6 14:00:02 2014
> Events : 3
> Update Time : Sat Dec 6 14:00:02 2014
> Events : 3
> Update Time : Sun Dec 7 11:18:06 2014
> Events : 8
> Update Time : Sun Dec 7 11:18:06 2014
> Events : 8
This looks strange. It suggests that the two drives failed well before
your reboot. If you have dmesg from Saturday afternoon, that might be
enlightening.
However, as a raid 5, you are stuck unless you include at least one of
the two stale drives. The correct tool for this is forced assembly:
mdadm --assemble --force --verbose /dev/mdX /dev/sd[abcd]2
If it fails, show its output.
Whether it fails or not, you need to investigate why the drives were
dropped. Simultaneous drops suggest a hardware problem.
Phil
^ permalink raw reply
* Re: Lost raid 5 volume
From: Neil . @ 2014-12-15 13:37 UTC (permalink / raw)
To: Emery Guevremont; +Cc: linux-raid
In-Reply-To: <CAB_L8saPqvR8HjqBFUvK+10H4Nou_yOrby5iPFkeiSiNj=RSww@mail.gmail.com>
On 15 December 2014 at 11:30, Emery Guevremont
<emery.guevremont@gmail.com> wrote:
> it's a good sign.
>
> mont -o ro,noload /dev/md0 /mnt
>
> and backup.
Emery! Phil! Thank you so much!
It mounted and I can get to the data. Really cool.
^ permalink raw reply
* Re: RAID1 + power fail + unwanted activation
From: Csordás Csaba @ 2014-12-15 14:19 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CABpXQ7bb4Sz7DZshAsry_+XO1x0p3jaSR3diYNj3t3CqCYV5pA@mail.gmail.com>
Although there still isn't any reactions I am still patient in order
to fully recover things. I'm open to anything, including telling me if
my questions are stupid or not understandable. Since my _previous_
notes about md and mdadm were on the RAID, I am stuck. I just won't
touch it until it will be clear for me. Also, I just can mount both
filesystems and run a diff -r against them hoping it results a 100%
match, and then recreate the RAID from scratch. It is soooo miserable,
isn't it?
The hard part is that presumably I don't understand something (:D) and
I don't know anybody else not amongst you who might know md better.
Since unfortunately I am not "used to" read C source code, any pointer
from remarkable people would be helpful.
Csordás Csaba
On Wed, Dec 3, 2014 at 5:20 PM, Csordás Csaba <cscsordas@gmail.com> wrote:
> Hi,
>
> after a power fail I was afraid so I was thinking about making a full
> gzipped image with dd from both members of my level1 array to be able
> to go back to the "original" (before the crash) state in case if
> *something* goes wrong during rebuild/lvm activation/fsck.
>
> The other machine I put the disk in unfortunately also had mdadm
> installed which automatically activated the degraded array (only one
> member of the array was connected) at boot time.
>
> Now, if i just put the disks back in the crashed machine, will the
> right thing happen while the kernel is resyncing the blocks between
> the devices? The internal bitmap was turned on, but the event counter
> on the mistakenly activated member of the array might have changed. I
> don't know how this -or other things I am not aware of- relates to the
> actual reassembly of my RAID1 array. I'm looking for an authoritative
> answer to understand the situation.
>
>
> Csordás Csaba
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* udisks error creating watch file in md127 container directory
From: John L. Center @ 2014-12-15 16:38 UTC (permalink / raw)
To: linux-raid
Hi,
I'm running Ubuntu 14.04 with the 3.17.5 kernel. I have 3 IMSM RAID1
arrays set up & I noticed the following syslog messages:
Dec 15 09:49:06 mariposa udisksd[3255]: udisks daemon version 2.1.3 starting
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md123/md/sync_action: No such file or
directory (g-file-error-quark, 4)
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md123/md/degraded: No such file or directory
(g-file-error-quark, 4)
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md125/md/sync_action: No such file or
directory (g-file-error-quark, 4)
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md125/md/degraded: No such file or directory
(g-file-error-quark, 4)
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md127/md/sync_action: No such file or
directory (g-file-error-quark, 4)
Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
/sys/devices/virtual/block/md127/md/degraded: No such file or directory
(g-file-error-quark, 4)
Dec 15 09:49:06 mariposa dbus[1230]: [system] Successfully activated
service 'org.freedesktop.UDisks2'
The files (degraded,sync_action) don't exist in the container (md127,
etc.) directories, but do exist in the array (md126, etc.) directories
themselves. I never noticed this before, is this something I should be
concerned about? I recently ran software updates, but udisks was not
one of the packages patched.
Thanks.
-John
^ permalink raw reply
* RE: udisks error creating watch file in md127 container directory
From: Baldysiak, Pawel @ 2014-12-15 17:30 UTC (permalink / raw)
To: John L. Center, linux-raid@vger.kernel.org
In-Reply-To: <548F0E75.1050502@comcast.net>
> On Monday, December 15, 2014 5:38 PM John L. Center wrote:
> To: linux-raid@vger.kernel.org
> Subject: udisks error creating watch file in md127 container directory
>
> Hi,
>
> I'm running Ubuntu 14.04 with the 3.17.5 kernel. I have 3 IMSM RAID1 arrays
> set up & I noticed the following syslog messages:
>
> Dec 15 09:49:06 mariposa udisksd[3255]: udisks daemon version 2.1.3 starting
> Dec 15 09:49:06 mariposa udisksd[3255]: Error creating watch for file
> /sys/devices/virtual/block/md123/md/sync_action: No such file or directory
> (g-file-error-quark, 4) Dec 15 09:49:06 mariposa udisksd[3255]: Error creating
> watch for file
> /sys/devices/virtual/block/md123/md/degraded: No such file or directory (g-
> file-error-quark, 4) Dec 15 09:49:06 mariposa udisksd[3255]: Error creating
> watch for file
> /sys/devices/virtual/block/md125/md/sync_action: No such file or directory
> (g-file-error-quark, 4) Dec 15 09:49:06 mariposa udisksd[3255]: Error creating
> watch for file
> /sys/devices/virtual/block/md125/md/degraded: No such file or directory (g-
> file-error-quark, 4) Dec 15 09:49:06 mariposa udisksd[3255]: Error creating
> watch for file
> /sys/devices/virtual/block/md127/md/sync_action: No such file or directory
> (g-file-error-quark, 4) Dec 15 09:49:06 mariposa udisksd[3255]: Error creating
> watch for file
> /sys/devices/virtual/block/md127/md/degraded: No such file or directory (g-
> file-error-quark, 4) Dec 15 09:49:06 mariposa dbus[1230]: [system]
> Successfully activated service 'org.freedesktop.UDisks2'
>
> The files (degraded,sync_action) don't exist in the container (md127,
> etc.) directories, but do exist in the array (md126, etc.) directories
> themselves. I never noticed this before, is this something I should be
> concerned about? I recently ran software updates, but udisks was not one
> of the packages patched.
>
> Thanks.
>
> -John
Hi John,
Everything seems to be alright - those files are not created for container devices (only array can be degraded, or proceed with sync).
I do not know why udisks is trying to create watch in this case.
Thanks
Pawel Baldysiak
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] Update CONFIG_DM_CRYPT help-text link
From: Loic Pefferkorn @ 2014-12-15 20:23 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel
Hello,
This patch fixes the obsolete help-text link of CONFIG_DM_CRYPT.
Applies against next-20141215.
Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=75111
Signed-off-by: Loic Pefferkorn <loic@loicp.eu>
---
drivers/md/Kconfig | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index 5bdedf6..09c89a4b 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -230,9 +230,8 @@ config DM_CRYPT
transparently encrypts the data on it. You'll need to activate
the ciphers you're going to use in the cryptoapi configuration.
- Information on how to use dm-crypt can be found on
-
- <http://www.saout.de/misc/dm-crypt/>
+ For further information on dm-crypt and userspace tools see:
+ <http://code.google.com/p/cryptsetup/wiki/DMCrypt>
To compile this code as a module, choose M here: the module will
be called dm-crypt.
--
2.1.3
^ permalink raw reply related
* Re: [PATCH] Update CONFIG_DM_CRYPT help-text link
From: NeilBrown @ 2014-12-15 20:30 UTC (permalink / raw)
To: Loic Pefferkorn; +Cc: linux-raid, linux-kernel
In-Reply-To: <81c1a7329d2146d37b22d7efa96b3a449cb66d14.1418674665.git.loic@loicp.eu>
[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]
On Mon, 15 Dec 2014 21:23:35 +0100 Loic Pefferkorn <loic@loicp.eu> wrote:
> Hello,
>
> This patch fixes the obsolete help-text link of CONFIG_DM_CRYPT.
> Applies against next-20141215.
Please direct this according to:
DEVICE-MAPPER (LVM)
M: Alasdair Kergon <agk@redhat.com>
M: Mike Snitzer <snitzer@redhat.com>
M: dm-devel@redhat.com
L: dm-devel@redhat.com
W: http://sources.redhat.com/dm
Q: http://patchwork.kernel.org/project/dm-devel/list/
T: git git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
T: quilt http://people.redhat.com/agk/patches/linux/editing/
S: Maintained
F: Documentation/device-mapper/
F: drivers/md/dm*
F: drivers/md/persistent-data/
F: include/linux/device-mapper.h
F: include/linux/dm-*.h
F: include/uapi/linux/dm-*.h
I guess get-maintainer.pl doesn't cope when when on file has separate parts
which are separately maintained.
Thanks,
NeilBrown
>
> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=75111
>
> Signed-off-by: Loic Pefferkorn <loic@loicp.eu>
> ---
> drivers/md/Kconfig | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
> index 5bdedf6..09c89a4b 100644
> --- a/drivers/md/Kconfig
> +++ b/drivers/md/Kconfig
> @@ -230,9 +230,8 @@ config DM_CRYPT
> transparently encrypts the data on it. You'll need to activate
> the ciphers you're going to use in the cryptoapi configuration.
>
> - Information on how to use dm-crypt can be found on
> -
> - <http://www.saout.de/misc/dm-crypt/>
> + For further information on dm-crypt and userspace tools see:
> + <http://code.google.com/p/cryptsetup/wiki/DMCrypt>
>
> To compile this code as a module, choose M here: the module will
> be called dm-crypt.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH] Update CONFIG_DM_CRYPT help-text link
From: Loic Pefferkorn @ 2014-12-15 21:11 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, linux-kernel
In-Reply-To: <20141216073037.75ec447b@notabene.brown>
On Tue, Dec 16, 2014 at 07:30:37AM +1100, NeilBrown wrote:
>
> Please direct this according to:
>
> DEVICE-MAPPER (LVM)
> M: Alasdair Kergon <agk@redhat.com>
> M: Mike Snitzer <snitzer@redhat.com>
> M: dm-devel@redhat.com
> L: dm-devel@redhat.com
> W: http://sources.redhat.com/dm
> Q: http://patchwork.kernel.org/project/dm-devel/list/
> T: git git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
> T: quilt http://people.redhat.com/agk/patches/linux/editing/
> S: Maintained
> F: Documentation/device-mapper/
> F: drivers/md/dm*
> F: drivers/md/persistent-data/
> F: include/linux/device-mapper.h
> F: include/linux/dm-*.h
> F: include/uapi/linux/dm-*.h
>
> I guess get-maintainer.pl doesn't cope when when on file has separate parts
> which are separately maintained.
>
> Thanks,
> NeilBrown
I will do that, thanks for the info!
--
Cheers,
Loïc
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-12-16 16:25 UTC (permalink / raw)
To: NeilBrown; +Cc: Mdadm
In-Reply-To: <CAK-d5dajGYxda_UFK_fBET14+KDr68km0uEKfrQVqs5oShWGfg@mail.gmail.com>
On 2 December 2014 at 17:26, Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> It works! (Atleast on a sample 5 MB device with 5 x 1MB partitions :-)
> will find more space on my drives and do a larger test but don't see
> why it shouldn't work)
> Here are the following caveats (and questions):
> - Neil, like you pointed out, the power of 2 chunk size will probably
> need a code change (in the kernel or only in the userspace tool?)
> - Any performance or other reasons why a terabyte size chunk may
> not be feasible?
> - Implications of safe_mode_delay
> - Would the metadata be updated on the block device be written to
> and the parity device as well?
> - If the drive fails which is the same as the drive being written
> to, would that lack of metadata updates to the other devices affect
> reconstruction?
> - Adding new devices (is it possible to move the parity to the disk
> being added? How does device addition work for RAID4 ...is it added as
> a zero-ed out device with parity disk remaining the same)
>
>
Neil, sorry to try to bump this thread. Could you please look over the
questions and address the points on the remaining items that can make
it a working solution? Thanks
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: NeilBrown @ 2014-12-16 21:49 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Mdadm
In-Reply-To: <CAK-d5db_ubRobL=uiia_9yOFA0PMovptAG8O+Y17ORT_--FsQg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]
On Tue, 16 Dec 2014 21:55:15 +0530 Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> On 2 December 2014 at 17:26, Anshuman Aggarwal
> <anshuman.aggarwal@gmail.com> wrote:
> > It works! (Atleast on a sample 5 MB device with 5 x 1MB partitions :-)
> > will find more space on my drives and do a larger test but don't see
> > why it shouldn't work)
> > Here are the following caveats (and questions):
> > - Neil, like you pointed out, the power of 2 chunk size will probably
> > need a code change (in the kernel or only in the userspace tool?)
In the kernel too.
> > - Any performance or other reasons why a terabyte size chunk may
> > not be feasible?
Not that I can think of.
> > - Implications of safe_mode_delay
> > - Would the metadata be updated on the block device be written to
> > and the parity device as well?
Probably. Hard to give a specific answer to vague question.
> > - If the drive fails which is the same as the drive being written
> > to, would that lack of metadata updates to the other devices affect
> > reconstruction?
Again, to give a precise answer, a detailed question is needed. Obviously
any change would have to made in such a way to ensure that things which
needed to work, did work.
> > - Adding new devices (is it possible to move the parity to the disk
> > being added? How does device addition work for RAID4 ...is it added as
> > a zero-ed out device with parity disk remaining the same)
RAID5 or RAID6 with ALGORITHM_PARITY_0 puts the parity on the early devices.
Currently if you add a device to such an array ...... I'm not sure what it
will do. It should be possible to make it just write zeros out.
NeilBrown
> >
> >
>
> Neil, sorry to try to bump this thread. Could you please look over the
> questions and address the points on the remaining items that can make
> it a working solution? Thanks
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: "bitmap file is out of date, doing full recovery"
From: NeilBrown @ 2014-12-16 22:26 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: linux-raid
In-Reply-To: <CAGRgLy7EW=3Tcy0swpcHNnyMz+A=9LiOoWXkx4-6R5YL1nVE_w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4693 bytes --]
On Sun, 14 Dec 2014 14:11:05 +0200 Alexander Lyakas <alex.bolshoy@gmail.com>
wrote:
> Hi Neil,
> This issue keeps happening to us. Do you see any problem in always
> incrementing the event count?
The reason we don't always increment the event count is that it wakes up
spare device unnecessarily.
Maybe the event counts on spare devices should be ignored....
NeilBrown
>
> Thanks,
> Alex.
>
> On Tue, Nov 4, 2014 at 11:17 AM, Alexander Lyakas
> <alex.bolshoy@gmail.com> wrote:
> > Hi Neil,
> > thank you for your comments.
> >
> > On Wed, Oct 29, 2014 at 1:19 AM, NeilBrown <neilb@suse.de> wrote:
> >> On Thu, 23 Oct 2014 19:04:48 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
> >> wrote:
> >>
> >>> Hi Neil,
> >>> I found at least one way of this happening. The problem is that in
> >>> md_update_sb() we allow to decrease the event count:
> >>>
> >>> /* If this is just a dirty<->clean transition, and the array is clean
> >>> * and 'events' is odd, we can roll back to the previous clean state */
> >>> if (nospares
> >>> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
> >>> && mddev->can_decrease_events
> >>> && mddev->events != 1) {
> >>> mddev->events--;
> >>> mddev->can_decrease_events = 0;
> >>>
> >>> Then we call bitmap_update_sb(). If we crash after we update (the
> >>> first or all of) bitmap superblocks, then after reboot, we will see
> >>> that bitmap event count is less than MD superblock event count. Then
> >>> we decide to do full resync.
> >>>
> >>> This can be easily reproduced by hacking bitmap_update_sb() to call
> >>> BUG(), after it calls write_page() in case event count was decreased.
> >>>
> >>> Why we are decreasing the event count??? Can we always increase it?
> >>> u64 is a lot to increase...
> >>
> >> The reason for decreasing the event count is so that we don't need to update
> >> the event count on spares - they can be left spun down.
> >> We for simple clean/dirty transitions with increment for clean->dirty and
> >> decrement for dirty->clean. But we should only use this optimisation when
> >> everything is simple.
> >> We really shouldn't do this when the array is degraded.
> >> Do this fix your problem?
> >>
> >> diff --git a/drivers/md/md.c b/drivers/md/md.c
> >> index 2c73fcb82593..98fd97b10e13 100644
> >> --- a/drivers/md/md.c
> >> +++ b/drivers/md/md.c
> >> @@ -2244,6 +2244,7 @@ repeat:
> >> * and 'events' is odd, we can roll back to the previous clean state */
> >> if (nospares
> >> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
> >> + && mddev->degraded == 0
> >> && mddev->can_decrease_events
> >> && mddev->events != 1) {
> >> mddev->events--;
> >>
> >>
> > No, unfortunately, this doesn't fix the problem. In my case, the array
> > is never degraded. Both drives are present and operational, then the
> > box crashes, and after reboot the bitmap event counter is lower than
> > we expect. Again, this is easily reproduced by hacking
> > bitmap_update_sb() as I mentioned earlier.
> >
> > In my case array does not have spares. (There is some other system on
> > top, which monitors the array, and, if needed, adds a spare from a
> > "global" spare pool). Is this ok in this case to always increment the
> > event count?
> >
> > Thanks,
> > Alex.
> >
> >
> >>>
> >>> Some other doubt that I have is that bitmap_unplug() and
> >>> bitmap_daemon_work() call write_page() on page index=0. This page
> >>> contains both the superblock and also some dirty bits (could not we
> >>> waste 4KB on bitmap superblock???). I am not sure, but I wonder
> >>> whether this call can race with md_update_sb (which explicitly calls
> >>> bitmap_update_sb), and somehow write the outdated superblock, after
> >>> bitmap_update_sb has completed writing it.
> >>>
> >>
> >> storage.sb_page is exactly the same as storage.filemap[0]
> >> So once an update has happened, the "outdated superblock" doesn't exist
> >> anywhere to be written out from.
> >>
> >>> Yet another suspect is when loading the bitmap we basically load it
> >>> from the first up-to-date drive. Maybe we should have scanned all the
> >>> bitmap superblocks, and selected one that has the higher event count
> >>> (although as we saw "higher" does not necessarily mean "more
> >>> up-to-date").
> >>>
> >>> Anyways, back to decrementing the event count. Do you see any issue
> >>> with not doing this and always incrementing?
> >>>
> >>> Thanks,
> >>> Alex.
> >>>
> >>
> >> Thanks,
> >> NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Question about my patch
From: NeilBrown @ 2014-12-17 5:26 UTC (permalink / raw)
To: nick; +Cc: linux-raid, linux-kernel
In-Reply-To: <5491043D.8020703@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2646 bytes --]
On Tue, 16 Dec 2014 23:19:09 -0500 nick <xerofoify@gmail.com> wrote:
> Greetings Neil,
> As you our the maintainer for this patch I created:
>
> >From ad324f9c2c8117b2f74ad73cb9c6e8185edf5395 Mon Sep 17 00:00:00 2001
> From: Nicholas Krause <xerofoify@gmail.com>
> Date: Tue, 16 Dec 2014 22:54:10 -0500
> Subject: [PATCH] drivers:md: Remove unneeded argument to the function,
> sync_request in raid5.c
>
> Removes unneeded argument of go_faster for the function sync_request as this parameter is never used in the function. Further more do to this we can remove this parameter for the function sync_request
> in the file, raid5.c safely and not break any callers of this function successfully.
>
> Signed-off-by: Nicholas Krause <xerofoify@gmail.com>
> ---
> drivers/md/raid5.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index c1b0d52..e7da686 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -4986,8 +4986,7 @@ ret:
> return reshape_sectors;
> }
>
> -/* FIXME go_faster isn't used */
> -static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipped, int go_faster)
> +static inline sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipped)
> {
> struct r5conf *conf = mddev->private;
> struct stripe_head *sh;
>
> --
> 2.1.0
>
> I am wondering as I am getting a few build errors if I should sent the patch anyway or create one that has no warnings. Below our the warnings I am getting.
>
Adding new warning is definitely not acceptable.
The only way you could get ride of the warning is to not change the arguments to the function, or
to make the same change to the arguments of every sync_request function.
The former is easier.
i.e. there is no bug here, and nothing to fix.
Thanks,
NeilBrown
> Warnings from Build:
> drivers/md/raid5.c:7047:2: warning: initialization from incompatible pointer type
> .sync_request = sync_request,
> ^
> drivers/md/raid5.c:7047:2: warning: (near initialization for ‘raid6_personality.sync_request’)
> drivers/md/raid5.c:7069:2: warning: initialization from incompatible pointer type
> .sync_request = sync_request,
> ^
> drivers/md/raid5.c:7069:2: warning: (near initialization for ‘raid5_personality.sync_request’)
> drivers/md/raid5.c:7092:2: warning: initialization from incompatible pointer type
> .sync_request = sync_request,
> ^
> Thanks Nick
> P.S. Sorry about the bad layout if it's hard to read tried to do my best to make it readable :).
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Manibalan P @ 2014-12-17 6:31 UTC (permalink / raw)
To: linux-raid; +Cc: NeilBrown
[-- Attachment #1: Type: text/plain, Size: 23516 bytes --]
Dear Neil,
We are facing IO struck issue with raid5 in the following scenario. (please see the attachment for the complete information)
In RAID5 array, if a drive is removed while initialization and the same time if IO is happening to that md. Then IO is getting struck, and md_raid5 thread is using 100 % of CPU. Also the md state showing as resync=PENDING
Kernel : Issue found in the following kernels
RHEL 6.5 (2.6.32-431.el6.x86_64)
CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
Steps to Reproduce the issue:
1. Created a raid 5 md with 4 drives using the below mdadm command.
mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6
2. Make the md writable
mdadm –readwrite /dev/md0
3. Now md will start initialization
4. Run FIO Tool, the the below said configuration
/usr/bin/fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
4. During MD initialzing, remove a drive(either using MDADM set faulty/remove or remove manually)
5. Now the IO will struck, and cat /proc/mdstat shows states with resync=PENDING
---------------------------------------------------------------------------------------------
top - output show, md_raid5 using 100% cpu
top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
---------------------------------------------------------------------------------------------
dmesg - show the stack trace
INFO: task fio:2715 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000a 0 2715 2654 0x00000080
ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8140fa39>] ? md_wakeup_thread+0x39/0x70
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffffa0308f66>] ? make_request+0x306/0xc6c [raid456]
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81122283>] ? mempool_alloc+0x63/0x140
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0
[<ffffffff811c8196>] __blockdev_direct_IO_newtrunc+0x346/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2717 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000004 0 2717 2654 0x00000080
ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2718 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000005 0 2718 2654 0x00000080
ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2719 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000001 0 2719 2654 0x00000080
ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2720 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000008 0 2720 2654 0x00000080
ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2721 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2721 2654 0x00000080
ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2722 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2722 2654 0x00000080
ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2723 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000006 0 2723 2654 0x00000080
ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2724 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000b 0 2724 2654 0x00000080
ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2725 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000003 0 2725 2654 0x00000080
ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[root@root ~]# cat /proc/2690/stack
[<ffffffff810686da>] __cond_resched+0x2a/0x40
[<ffffffffa030361c>] ops_run_io+0x2c/0x920 [raid456]
[<ffffffffa03052cc>] handle_stripe+0x9cc/0x2980 [raid456]
[<ffffffffa03078a4>] raid5d+0x624/0x850 [raid456]
[<ffffffff81416f05>] md_thread+0x115/0x150
[<ffffffff8109aef6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffffffffffff>] 0xffffffffffffffff
[root@root ~]# cat /proc/2690/stat
2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17 2 0 0 6855 0 0
[root@root ~]# cat /proc/2690/statm
0 0 0 0 0 0 0
[root@root ~]# cat /proc/2690/stat
stat statm status
[root@root ~]# cat /proc/2690/status
Name: md0_raid5
State: R (running)
Tgid: 2690
Pid: 2690
PPid: 2
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
Utrace: 0
FDSize: 64
Groups:
Threads: 1
SigQ: 2/128402
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: fffffffffffffeff
SigCgt: 0000000000000100
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: fffffffffffffeff
CapBnd: ffffffffffffffff
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 5411612
nonvoluntary_ctxt_switches: 257032
Thanks,
Manibalan.
[-- Attachment #2: md_raid5-hang-resync-PENDING.txt --]
[-- Type: text/plain, Size: 29686 bytes --]
Issue:
md_raid5 using 100% CPU and hang with resync=PENDING status, if a drive is removed during initialization
Discription:
In RAID5 array, if a drive is removed during initialization and the same time if IO is happening to that md.
Then IO is getting struck, and md_raid5 thread is using 100 % of CPU. Also the md state showing as resync=PENDING
Kernel : Issue found in the following kernels
>RHEL 6.5 (2.6.32-431.el6.x86_64)
>CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
Steps to Reproduce the issue:
1. Created a raid 5 md with 4 drives using the below mdadm command.
mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6
2. Make the md writable
mdadm –readwrite /dev/md0
3. Now md will start initialization
4. Run FIO Tool, the the below said configuration
/usr/bin/fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
4. During MD initialzing, remove a drive(either using MDADM set faulty/remove or remove manually)
5. Now the IO will struck, and cat /proc/mdstat shows states with resync=PENDING
Step done one by one to reproduce the issue, and the Observation during each step:
1. System Information:
[root@root ~]# uname -a
Linux root 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@root ~]# mdadm -V
mdadm - v3.2.6 - 25th October 2012
[root@root ~]# fio --version
fio-2.1.10
[root@root ~]# lsscsi
[0:0:0:0] disk SEAGATE ST31000640SS 0003 /dev/sda
[0:0:1:0] disk SEAGATE ST2000NM0001 0002 /dev/sdb
[0:0:2:0] enclosu LSI CORP SAS2X36 0424 -
[0:0:3:0] disk SEAGATE ST200FM0002 0003 /dev/sdc
[0:0:4:0] disk SEAGATE ST200FM0002 0003 /dev/sdd
[0:0:5:0] disk SEAGATE ST200FM0002 0003 /dev/sde
[0:0:6:0] disk SEAGATE ST200FM0002 0003 /dev/sdf
2. Creating raid5 md
[root@root ~]# mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sd[cdef]6
mdadm: /dev/sdc6 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Dec 15 18:23:17 2014
mdadm: /dev/sdd6 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Dec 15 18:23:17 2014
mdadm: /dev/sde6 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Dec 15 18:23:17 2014
mdadm: /dev/sdf6 appears to be part of a raid array:
level=raid5 devices=4 ctime=Mon Dec 15 18:23:17 2014
Continue creating array? y
mdadm: array /dev/md0 started.
dmesg
md: unbind<sdf6>
md: export_rdev(sdf6)
md: unbind<sde6>
md: export_rdev(sde6)
md: unbind<sdd6>
md: export_rdev(sdd6)
md: unbind<sdc6>
md: export_rdev(sdc6)
md: bind<sdc6>
md: bind<sdd6>
md: bind<sde6>
md: bind<sdf6>
async_tx: api initialized (async)
xor: automatically using best checksumming function: generic_sse
generic_sse: 9976.000 MB/sec
xor: using function: generic_sse (9976.000 MB/sec)
raid6: sse2x1 6386 MB/s
raid6: sse2x2 7464 MB/s
raid6: sse2x4 8199 MB/s
raid6: using algorithm sse2x4 (8199 MB/s)
raid6: using ssse3x2 recovery algorithm
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
bio: create slab <bio-1> at 1
md/raid:md0: not clean -- starting background reconstruction
md/raid:md0: device sdf6 operational as raid disk 3
md/raid:md0: device sde6 operational as raid disk 2
md/raid:md0: device sdd6 operational as raid disk 1
md/raid:md0: device sdc6 operational as raid disk 0
md/raid:md0: allocated 4314kB
md/raid:md0: raid level 5 active with 4 out of 4 devices, algorithm 2
RAID conf printout:
--- level:5 rd:4 wd:4
disk 0, o:1, dev:sdc6
disk 1, o:1, dev:sdd6
disk 2, o:1, dev:sde6
disk 3, o:1, dev:sdf6
md0: detected capacity change from 0 to 576636125184
md: resync of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
md: using 128k window, over a total of 187707072k.
md0: unknown partition table
[root@root ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf6[3] sde6[2] sdd6[1] sdc6[0]
563121216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[=>...................] resync = 8.2% (15404908/187707072) finish=16.6min speed=172462K/sec
unused devices: <none>
[root@root ~]# echo 10000 > /sys/block/md0/md/sync_speed_min
[root@root ~]# echo 30000 > /sys/block/md0/md/sync_speed_max
[root@root ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf6[3] sde6[2] sdd6[1] sdc6[0]
563121216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[===>.................] resync = 16.1% (30226432/187707072) finish=47.3min speed=55459K/sec
unused devices: <none>
3. Start FIO
[root@root ~]# /usr/bin/fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
md0: (g=0): rw=randwrite, bs=8704-8704/8704-8704/8704-8704, ioengine=libaio, iodepth=4000
...
fio-2.1.10
Starting 10 threads
Jobs: 10 (f=10): [wwwwwwwwww] [12.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 43m:37s]
4. Remove a drive from md arry using mdadm command
[root@root ~]# mdadm /dev/md0 --set-faulty /dev/sdc6
dmesg
md/raid:md0: Disk failure on sdc6, disabling device.
md/raid:md0: Operation continuing on 3 devices.
md: md0: resync done.
md: checkpointing resync of md0.
5. System state after the drive is removed
[root@root ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf6[3] sde6[2] sdd6[1] sdc6[0](F)
563121216 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
resync=PENDING
unused devices: <none>
top
top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
Tasks: 313 total, 2 running, 311 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 6.3%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16455916k total, 780184k used, 15675732k free, 29628k buffers
Swap: 6127608k total, 0k used, 6127608k free, 116212k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
235 root 39 19 0 0 0 S 0.3 0.0 0:12.95 kipmi0
2650 root 20 0 98.1m 4456 3348 S 0.3 0.0 0:00.21 sshd
1 root 20 0 19364 1536 1232 S 0.0 0.0 0:01.42 init
Dmesg
INFO: task fio:2715 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000a 0 2715 2654 0x00000080
ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8140fa39>] ? md_wakeup_thread+0x39/0x70
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffffa0308f66>] ? make_request+0x306/0xc6c [raid456]
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81122283>] ? mempool_alloc+0x63/0x140
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0
[<ffffffff811c8196>] __blockdev_direct_IO_newtrunc+0x346/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2717 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000004 0 2717 2654 0x00000080
ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2718 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000005 0 2718 2654 0x00000080
ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2719 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000001 0 2719 2654 0x00000080
ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2720 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000008 0 2720 2654 0x00000080
ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2721 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2721 2654 0x00000080
ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2722 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2722 2654 0x00000080
ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2723 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000006 0 2723 2654 0x00000080
ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2724 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000b 0 2724 2654 0x00000080
ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2725 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000003 0 2725 2654 0x00000080
ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8
Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70
[<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456]
[<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81415b41>] md_make_request+0xe1/0x230
[<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110
[<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230
[<ffffffff81266c50>] generic_make_request+0x240/0x5a0
[<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81267020>] submit_bio+0x70/0x120
[<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20
[<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170
[<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0
[<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0
[<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200
[<ffffffff811d6924>] aio_run_iocb+0x64/0x170
[<ffffffff811d7d51>] do_io_submit+0x291/0x920
[<ffffffff811d83f0>] sys_io_submit+0x10/0x20
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[root@root ~]# cat /proc/2690/stack
[<ffffffff810686da>] __cond_resched+0x2a/0x40
[<ffffffffa030361c>] ops_run_io+0x2c/0x920 [raid456]
[<ffffffffa03052cc>] handle_stripe+0x9cc/0x2980 [raid456]
[<ffffffffa03078a4>] raid5d+0x624/0x850 [raid456]
[<ffffffff81416f05>] md_thread+0x115/0x150
[<ffffffff8109aef6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffffffffffff>] 0xffffffffffffffff
[root@root ~]# cat /proc/2690/stat
2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17 2 0 0 6855 0 0
[root@root ~]# cat /proc/2690/statm
0 0 0 0 0 0 0
[root@root ~]# cat /proc/2690/stat
stat statm status
[root@root ~]# cat /proc/2690/status
Name: md0_raid5
State: R (running)
Tgid: 2690
Pid: 2690
PPid: 2
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
Utrace: 0
FDSize: 64
Groups:
Threads: 1
SigQ: 2/128402
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: fffffffffffffeff
SigCgt: 0000000000000100
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: fffffffffffffeff
CapBnd: ffffffffffffffff
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 5411612
nonvoluntary_ctxt_switches: 257032
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox