* Re: RAID1 removing failed disk returns EBUSY
From: Xiao Ni @ 2015-01-30 4:27 UTC (permalink / raw)
To: Joe Lawrence; +Cc: NeilBrown, linux-raid, Bill Kuzeja
In-Reply-To: <1282724195.2440807.1422584341926.JavaMail.zimbra@redhat.com>
----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: "Joe Lawrence" <joe.lawrence@stratus.com>
> Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> Sent: Friday, January 30, 2015 10:19:01 AM
> Subject: Re: RAID1 removing failed disk returns EBUSY
>
>
>
> ----- Original Message -----
> > From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill Kuzeja"
> > <william.kuzeja@stratus.com>
> > Sent: Friday, January 23, 2015 11:11:29 PM
> > Subject: Re: RAID1 removing failed disk returns EBUSY
> >
> > On Tue, 20 Jan 2015 02:16:46 -0500
> > Xiao Ni <xni@redhat.com> wrote:
> > > Joe
> > >
> > > Thanks for the explanation. So echo "idle" to sync_action is a
> > > workaround
> > > without the patch.
> > >
> > > It looks like the patch is not enough to fix the problem.
> > > Do you have a try with the new patch? Is the problem still exist in
> > > your environment?
> > >
> > > If your environment have no problem, can you give me the version
> > > number?
> > > I'll
> > > have a try with the same version too.
> >
> > Hi Xiao,
> >
> > Bill and I did some more testing yesterday and I think we've figured
> > out the confusion. Running a 3.18+ kernel and an upstream mdadm, it
> > was the udev invocation of "mdadm -If <dev>" that was automatically
> > removing the device for us.
> >
> > If we ran with an older mdadm and got the MD wedged in the faulty
> > condition, then nothing we echoed into the sysfs state file ('idle'
> > 'fail' or 'remove') would change anything. I think this agrees with
> > your testing report.
> >
> > So two things:
> >
> > 1 - Did you make / make install the latest mdadm and see it try to run
> > mdadm -If on the removed disk? (You could also try manually running
> > it.)
>
> I make sure I have install the latest mdadm
> [root@dhcp-12-133 ~]# mdadm --version
> mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014
>
> It can prove this, right?
>
> It's strange when I ran mdadm -If
>
> [root@dhcp-12-133 ~]# mdadm -If sdc
> mdadm: sdc does not appear to be a component of any array
> [root@dhcp-12-133 ~]# cat /proc/mdstat
> Personalities : [raid1]
> md0 : active (auto-read-only) raid1 sdd1[1] sdc1[0](F)
> 5238784 blocks super 1.2 [2/1] [_U]
>
> unused devices: <none>
>
> I unplug the device manually from the machine. The machine is on my desk.
Hi Joe
Sorry for this. I input the command wrongly.
[root@dhcp-12-133 ~]# mdadm -If sdc1
mdadm: set sdc1 faulty in md0
mdadm: hot remove failed for sdc1: Device or resource busy
>
>
> >
> > 2 - I think the sysfs interface to the removed disks is still broken in
> > cases where (1) doesn't occur.
> >
> > Thanks,
> >
> > -- Joe
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [PATCH] md:Remove unneeded fixme comment for the function, pool_ctr
From: Nicholas Krause @ 2015-01-31 5:36 UTC (permalink / raw)
To: agk; +Cc: linux-raid, dm-devel, linux-kernel, snitzer
Removes the no longer needed fix me comment for removing the validation check
for the number of agruments passed by the caller to outside the mutex_lock in
the function,pool_ctr. This comment is no longer needed as we will clearly jump
to the out_lock block of this function if the validation fails and therefore this
is no real concern for holding the mutex lock longer then required if the validation
check for number of agruments from the caller of fails.
Signed-off-by: Nicholas Krause <xerofoify@gmail.com>
---
drivers/md/dm-thin.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index 07705ee..7d22334 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -2849,9 +2849,6 @@ static int pool_ctr(struct dm_target *ti, unsigned argc, char **argv)
struct dm_dev *metadata_dev;
fmode_t metadata_mode;
- /*
- * FIXME Remove validation from scope of lock.
- */
mutex_lock(&dm_thin_pool_table.mutex);
if (argc < 4) {
--
2.1.0
^ permalink raw reply related
* Apply!
From: Easy Finance @ 2015-01-31 7:15 UTC (permalink / raw)
To: Recipients
Good & Bad Credit Loan With No Credit Check. Email us amount needed & duration.
===================================================================
^ permalink raw reply
* unsynchronized raid10 with different events count
From: LuVar @ 2015-01-31 13:52 UTC (permalink / raw)
To: linux-raid
Hi,
I have raid with 9 devices, layout n3. It is not possible to autoassemble it with all devices, because 4 devices has different number of events... How can I assemble it fully and sync with 5 devices which have slightly more events? My current state after mdadm --assemble /dev/...... is this: http://pastebin.com/vEpd8WWW
I know that I can delete those 4 disks and add them again, but I do not want to experiment if removed and newly added disk will be on correct place in raid and also I would like to not synchronize whole disks. They miss only a few events. Is there any possibility?
PS: here is complete examine output: http://cwillu.com:8080/188.121.181.8/8 and cat /proc/mdstat is here: http://cwillu.com:8080/188.121.181.8/9
^ permalink raw reply
* Re: unsynchronized raid10 with different events count
From: LuVar @ 2015-01-31 14:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1468090491.36611422712337250.JavaMail.root@shiva>
Hi,
I have found out something myself. According https://raid.wiki.kernel.org/index.php/RAID_Recovery#Trying_to_assemble_using_--force wiki page, assembling with force can solve my problem. By reading manpage ob mdadm, I have reat this:
<cite>
-f, --force
Assemble the array even if the metadata on some devices appears to be out-of-date. If mdadm cannot find enough working devices to start the array, but can find some devices that are recorded as having failed, then it will mark those devices as working so that the array can be started. An array which requires --force to be started may contain data corruption. Use it carefully.
</cite>
My new answer is, would it use in raid10 (layout n3) disks with same, highest, event count? If yes, I assume, that result will be 100% consistent. Currently, I have these events count:
Events : 55060
Events : 55041
Events : 55060
Events : 55060
Events : 55041
Events : 55041
Events : 55060
Events : 55060
Events : 55041
So for each part of raid0, I have three disks in raid1. In each raid1, there is at least one disk with events = 55060. So if force will use them to restore (sync) content on other raid1 disks, it should be 100% consistent. WHY does not do this mdraid automatically when I request assemble? There is no risk of dataloss from my point ow view, if the biggest event count is contained at least on one part of raid.
Thanks for fast reply,
LuVar
----- "LuVar" <luvar@plaintext.sk> wrote:
> Hi,
> I have raid with 9 devices, layout n3. It is not possible to
> autoassemble it with all devices, because 4 devices has different
> number of events... How can I assemble it fully and sync with 5
> devices which have slightly more events? My current state after mdadm
> --assemble /dev/...... is this: http://pastebin.com/vEpd8WWW
>
> I know that I can delete those 4 disks and add them again, but I do
> not want to experiment if removed and newly added disk will be on
> correct place in raid and also I would like to not synchronize whole
> disks. They miss only a few events. Is there any possibility?
>
> PS: here is complete examine output:
> http://cwillu.com:8080/188.121.181.8/8 and cat /proc/mdstat is here:
> http://cwillu.com:8080/188.121.181.8/9
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: unsynchronized raid10 with different events count
From: LuVar @ 2015-01-31 16:14 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <2049102750.36641422714825909.JavaMail.root@shiva>
Hi,
finally,
I have run --re-add thing. It seems that it is what I wanted. My array is now OK.
So only one question remains. Is there any possibility to make improvements and make array autoassemble if there are clear that in raid10, there is consistent highest events count for at least one drive for raid1 subparts? I have layout=n3 for situations when one drive fails, to be able to correctly recover from others two same copies and to know that given data are probably ok if they are same.
----- "LuVar" <luvar@plaintext.sk> wrote:
> Hi,
> I have found out something myself. According
> https://raid.wiki.kernel.org/index.php/RAID_Recovery#Trying_to_assemble_using_--force
> wiki page, assembling with force can solve my problem. By reading
> manpage ob mdadm, I have reat this:
>
> <cite>
> -f, --force
> Assemble the array even if the metadata on some devices
> appears to be out-of-date. If mdadm cannot find enough working
> devices to start the array, but can find some devices that are
> recorded as having failed, then it will mark those devices as working
> so that the array can be started. An array which requires --force to
> be started may contain data corruption. Use it carefully.
> </cite>
>
> My new answer is, would it use in raid10 (layout n3) disks with same,
> highest, event count? If yes, I assume, that result will be 100%
> consistent. Currently, I have these events count:
>
> Events : 55060
> Events : 55041
> Events : 55060
>
> Events : 55060
> Events : 55041
> Events : 55041
>
> Events : 55060
> Events : 55060
> Events : 55041
>
> So for each part of raid0, I have three disks in raid1. In each raid1,
> there is at least one disk with events = 55060. So if force will use
> them to restore (sync) content on other raid1 disks, it should be 100%
> consistent. WHY does not do this mdraid automatically when I request
> assemble? There is no risk of dataloss from my point ow view, if the
> biggest event count is contained at least on one part of raid.
>
> Thanks for fast reply,
> LuVar
>
> ----- "LuVar" <luvar@plaintext.sk> wrote:
>
> > Hi,
> > I have raid with 9 devices, layout n3. It is not possible to
> > autoassemble it with all devices, because 4 devices has different
> > number of events... How can I assemble it fully and sync with 5
> > devices which have slightly more events? My current state after
> mdadm
> > --assemble /dev/...... is this: http://pastebin.com/vEpd8WWW
> >
> > I know that I can delete those 4 disks and add them again, but I do
> > not want to experiment if removed and newly added disk will be on
> > correct place in raid and also I would like to not synchronize whole
> > disks. They miss only a few events. Is there any possibility?
> >
> > PS: here is complete examine output:
> > http://cwillu.com:8080/188.121.181.8/8 and cat /proc/mdstat is here:
> > http://cwillu.com:8080/188.121.181.8/9
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> linux-raid"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] md: do_release_stripe(): No need to call md_wakeup_thread() twice
From: NeilBrown @ 2015-02-01 23:25 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid
In-Reply-To: <1422553109-29247-1-git-send-email-Jes.Sorensen@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]
On Thu, 29 Jan 2015 12:38:29 -0500 Jes.Sorensen@redhat.com wrote:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> 67f455486d2ea20b2d94d6adf5b9b783d079e321 introduced a call to
> md_wakeup_thread() when adding to the delayed_list. However the md
> thread is woken up unconditionally just below.
>
> Remove the unnecessary wakeup call.
>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> drivers/md/raid5.c | 7 ++-----
> 1 file changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index c1b0d52..41494d9 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -296,12 +296,9 @@ static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh,
> BUG_ON(atomic_read(&conf->active_stripes)==0);
> if (test_bit(STRIPE_HANDLE, &sh->state)) {
> if (test_bit(STRIPE_DELAYED, &sh->state) &&
> - !test_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) {
> + !test_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
> list_add_tail(&sh->lru, &conf->delayed_list);
> - if (atomic_read(&conf->preread_active_stripes)
> - < IO_THRESHOLD)
> - md_wakeup_thread(conf->mddev->thread);
> - } else if (test_bit(STRIPE_BIT_DELAY, &sh->state) &&
> + else if (test_bit(STRIPE_BIT_DELAY, &sh->state) &&
> sh->bm_seq - conf->seq_write > 0)
> list_add_tail(&sh->lru, &conf->bitmap_list);
> else {
Applied, thanks.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [dm-devel] [PATCH] md: fix raid5 livelock
From: NeilBrown @ 2015-02-02 0:06 UTC (permalink / raw)
To: Heinz Mauelshagen
Cc: device-mapper development, linux-raid, Brassow Jonathan,
Alasdair G Kergon, Mikulas Patocka, Marian Csontos
In-Reply-To: <54CA1850.1000909@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2173 bytes --]
On Thu, 29 Jan 2015 12:24:00 +0100 Heinz Mauelshagen <heinzm@redhat.com>
wrote:
>
> Neil,
>
> the patch worked fine in overnight test runs without the previous livelock.
> No regressions have been triggered.
>
> Yes, tidying up that optimization logic (e.g. in fetch_block()) is very
> much appreciated :-)
>
Thanks!
The following is what should appear in -next soonish. If there are any *-by:
tags to be added or changed, please let me know.
NeilBrown
From: NeilBrown <neilb@suse.de>
Date: Mon, 2 Feb 2015 10:44:29 +1100
Subject: [PATCH] md/raid5: fix another livelock caused by non-aligned writes.
If a non-page-aligned write is destined for a device which
is missing/faulty, we can deadlock.
As the target device is missing, a read-modify-write cycle
is not possible.
As the write is not for a full-page, a recontruct-write cycle
is not possible.
This should be handled by logic in fetch_block() which notices
there is a non-R5_OVERWRITE write to a missing device, and so
loads all blocks.
However since commit 67f455486d2ea2, that code requires
STRIPE_PREREAD_ACTIVE before it will active, and those circumstances
never set STRIPE_PREREAD_ACTIVE.
So: in handle_stripe_dirtying, if neither rmw or rcw was possible,
set STRIPE_DELAYED, which will cause STRIPE_PREREAD_ACTIVE be set
after a suitable delay.
Fixes: 67f455486d2ea20b2d94d6adf5b9b783d079e321
Cc: stable@vger.kernel.org (v3.16+)
Reported-by: Mikulas Patocka <mpatocka@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 41494d904859..274db1834d43 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3192,6 +3192,11 @@ static void handle_stripe_dirtying(struct r5conf *conf,
(unsigned long long)sh->sector,
rcw, qread, test_bit(STRIPE_DELAYED, &sh->state));
}
+
+ if (rcw > disks && rmw > disks &&
+ !test_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
+ set_bit(STRIPE_DELAYED, &sh->state);
+
/* now if nothing is locked, and if we have enough data,
* we can start a write request
*/
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: RAID1 removing failed disk returns EBUSY
From: NeilBrown @ 2015-02-02 6:36 UTC (permalink / raw)
To: Xiao Ni; +Cc: Joe Lawrence, linux-raid, Bill Kuzeja
In-Reply-To: <371504811.2053160.1422533656432.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 6808 bytes --]
On Thu, 29 Jan 2015 07:14:16 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
>
>
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: "Joe Lawrence" <joe.lawrence@stratus.com>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> > Sent: Thursday, January 29, 2015 11:52:17 AM
> > Subject: Re: RAID1 removing failed disk returns EBUSY
> >
> > On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
> >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill
> > > > Kuzeja" <william.kuzeja@stratus.com>
> > > > Sent: Friday, January 16, 2015 11:10:31 PM
> > > > Subject: Re: RAID1 removing failed disk returns EBUSY
> > > >
> > > > On Fri, 16 Jan 2015 00:20:12 -0500
> > > > Xiao Ni <xni@redhat.com> wrote:
> > > > >
> > > > > Hi Joe
> > > > >
> > > > > Thanks for reminding me. I didn't do that. Now it can remove
> > > > > successfully after writing
> > > > > "idle" to sync_action.
> > > > >
> > > > > I thought wrongly that the patch referenced in this mail is fixed
> > > > > for
> > > > > the problem.
> > > >
> > > > So it sounds like even with 3.18 and a new mdadm, this bug still
> > > > persists?
> > > >
> > > > -- Joe
> > > >
> > > > --
> > >
> > > Hi Joe
> > >
> > > I'm a little confused now. Does the patch
> > > 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> > > resolve the problem?
> > >
> > > My environment is:
> > >
> > > [root@dhcp-12-133 mdadm]# mdadm --version
> > > mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014 (this is the newest
> > > upstream)
> > > [root@dhcp-12-133 mdadm]# uname -r
> > > 3.18.2
> > >
> > >
> > > My steps are:
> > >
> > > [root@dhcp-12-133 mdadm]# lsblk
> > > sdb 8:16 0 931.5G 0 disk
> > > └─sdb1 8:17 0 5G 0 part
> > > sdc 8:32 0 186.3G 0 disk
> > > sdd 8:48 0 931.5G 0 disk
> > > └─sdd1 8:49 0 5G 0 part
> > > [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1 /dev/sdd1
> > > --assume-clean
> > > mdadm: Note: this array has metadata at the start and
> > > may not be suitable as a boot device. If you plan to
> > > store '/boot' on this device please ensure that
> > > your boot-loader understands md/v1.x metadata, or use
> > > --metadata=0.90
> > > mdadm: Defaulting to version 1.2 metadata
> > > mdadm: array /dev/md0 started.
> > >
> > > Then I unplug the disk.
> > >
> > > [root@dhcp-12-133 mdadm]# lsblk
> > > sdc 8:32 0 186.3G 0 disk
> > > sdd 8:48 0 931.5G 0 disk
> > > └─sdd1 8:49 0 5G 0 part
> > > └─md0 9:0 0 5G 0 raid1
> > > [root@dhcp-12-133 mdadm]# echo faulty > /sys/block/md0/md/dev-sdb1/state
> > > [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state
> > > -bash: echo: write error: Device or resource busy
> > > [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action
> > > [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state
> > >
> >
> > I cannot reproduce this - using linux 3.18.2. I'd be surprised if mdadm
> > version affects things.
>
> Hi Neil
>
> I'm very curious, because it can reproduce in my machine 100%.
>
> >
> > This error (Device or resoource busy) implies that rdev->raid_disk is >= 0
> > (tested in state_store()).
> >
> > ->raid_disk is set to -1 by remove_and_add_spares() providing:
> > 1/ it isn't Blocked (which is very unlikely)
> > 2/ hot_remove_disk succeeds, which it will if nr_pending is zero, and
> > 3/ nr_pending is zero.
>
> I remember I have tired to check those reasons. But it's really is the reason 1
> which is very unlikely.
>
> I add some code in the function array_state_show
>
> array_state_show(struct mddev *mddev, char *page) {
> enum array_state st = inactive;
> struct md_rdev *rdev;
>
> rdev_for_each_rcu(rdev, mddev) {
> printk(KERN_ALERT "search for %s\n", rdev->bdev->bd_disk->disk_name);
> if (test_bit(Blocked, &rdev->flags))
> printk(KERN_ALERT "rdev is Blocked\n");
> else
> printk(KERN_ALERT "rdev is not Blocked\n");
> }
>
> When I echo 1 > /sys/block/sdc/device/delete, then I ran command:
>
> [root@dhcp-12-133 md]# cat /sys/block/md0/md/array_state
> read-auto
^^^^^^^^^
I think that is half the explanation.
You must have the md_mod.start_ro parameter set to '1'.
> [root@dhcp-12-133 md]# dmesg
> [ 2679.559185] search for sdc
> [ 2679.559189] rdev is Blocked
> [ 2679.559190] search for sdb
> [ 2679.559190] rdev is not Blocked
>
> So sdc is Blocked
and that is the other half - thanks.
(yes, I was wrong. Sometimes it is easier than being right, but still
yields results).
When a device fails, it is Blocked until the metadata is updated to record
the failure. This ensures that no writes succeed without writing to that
device, until we a certain that no read will try reading from that device,
even after a crash/restart.
Blocked is cleared after the metadata is written, but read-auto (and
read-only) devices never write out their metadata. So blocked doesn't get
cleared.
When you "echo idle > .../sync_action" one of the side effects is to with
from 'read-auto' to fully active. This allows the metadata to be written,
Blocked to be cleared, and the device to be removed.
If you
echo none > /sys/block/md0/md/dev-sdc/slot
first, then the remove will work.
We could possibly fix it with something like the following, but I'm not sure
I like it. There is no guarantee that I can see which would ensure the
superblock got updated before the first write if the array switch to
read/write.
NeilBrown
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 9233c71138f1..b3d1e8e5e067 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7528,7 +7528,7 @@ static int remove_and_add_spares(struct mddev *mddev,
rdev_for_each(rdev, mddev)
if ((this == NULL || rdev == this) &&
rdev->raid_disk >= 0 &&
- !test_bit(Blocked, &rdev->flags) &&
+ (!test_bit(Blocked, &rdev->flags) || mddev->ro) &&
(test_bit(Faulty, &rdev->flags) ||
! test_bit(In_sync, &rdev->flags)) &&
atomic_read(&rdev->nr_pending)==0) {
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Manibalan P @ 2015-02-02 7:10 UTC (permalink / raw)
To: Pasi Kärkkäinen, neilb@suse.de, linux-raid
In-Reply-To: <20141231164800.GL19091@reaktio.net>
Dear All,
Any updates on this issue.
Thanks,
Manibalan.
-----Original Message-----
From: Manibalan P
Sent: Wednesday, January 14, 2015 3:55 PM
To: 'Pasi Kärkkäinen'
Cc: 'neilb@suse.de'; 'linux-raid'
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear Pasi,
Could you able to find something on this issue.
Thanks,
Manibalan.
-----Original Message-----
From: Manibalan P
Sent: Friday, January 2, 2015 12:08 PM
To: 'Pasi Kärkkäinen'
Cc: neilb@suse.de; linux-raid
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear Pasi,
I have add the bug in https://bugzilla.redhat.com/show_bug.cgi?id=1178080
Thanks,
Manibalan.
-----Original Message-----
From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
Sent: Wednesday, December 31, 2014 10:18 PM
To: Manibalan P
Cc: neilb@suse.de; linux-raid
Subject: Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
On Tue, Dec 30, 2014 at 11:06:47AM +0000, Manibalan P wrote:
> Dear Neil,
>
Hello,
> Few this for you kind attention,
> 1. I tried the same test with FC11 (2.6.32 kernel before MD code
> change). And the issue is not there 2. But with Centos 6.4 (2.6.32 kernel after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Also, a bug has been raise with RHEL regarding this issue. Please find the bug link "https://access.redhat.com/support/cases/#/case/01320319"
>
That support case URL can only be accessed by you and Redhat. Do you happen to have a public bugzilla link?
Thanks,
-- Pasi
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 24, 2014 12:15 PM
> To: neilb@suse.de; 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
>
> Dear Neil,
>
> Few this for you kind attention,
> 1. I tried the same tesst with FC11 (2.6 kernel before MD code change). And the issue is not there 2. But with Centos 6.4 (2.6 after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Thursday, December 18, 2014 11:38 AM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear neil,
>
> I also compiled the latest 3.18 kernel on CentOS 6.4 with GIT MD pull patches form 3.19, that also ran in to the same issue after removing a drive during resync.
>
> Dec 17 19:07:32 ITX002590129362 kernel: Linux version 3.18.0 (root@mycentos6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Dec 17 15:59:09 EST 2014 Dec 17 19:07:32 ITX002590129362 kernel: Command line: ro root=/dev/md255 rd_NO_LVM rd_NO_DM rhgb quiet md_mod.start_ro=1 nmi_watchdog=1 md_mod.start_dirty_degraded=1 ??? Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sda6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdb6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdc6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdh6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdi6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdj6> Dec 17 19:10:15 ITX002590129362 kernel: async_tx: api initialized (async) Dec 17 19:10:15 ITX002590129362 kernel: xor: measuring software checksum speed
> Dec 17 19:10:15 ITX002590129362 kernel: prefetch64-sse: 10048.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: generic_sse: 8824.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: xor: using function: prefetch64-sse (10048.000 MB/sec)
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x1 5921 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x2 6933 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x4 7476 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: using algorithm sse2x4
> (7476 MB/s) Dec 17 19:10:15 ITX002590129362 kernel: raid6: using
> ssse3x2 recovery algorithm Dec 17 19:10:15 ITX002590129362 kernel: md:
> raid6 personality registered for level 6 Dec 17 19:10:15
> ITX002590129362 kernel: md: raid5 personality registered for level 5
> Dec 17 19:10:15 ITX002590129362 kernel: md: raid4 personality
> registered for level 4 Dec 17 19:10:15 ITX002590129362 kernel:
> md/raid:md0: not clean -- starting background reconstruction Dec 17
> 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdj6 operational
> as raid disk 5 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> device sdi6 operational as raid disk 4 Dec 17 19:10:15 ITX002590129362
> kernel: md/raid:md0: device sdh6 operational as raid disk 3 Dec 17
> 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdc6 operational
> as raid disk 2 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> device sdb6 operational as raid disk 1 Dec 17 19:10:15 ITX002590129362
> kernel: md/ra
> id:md0: device sda6 operational as raid disk 0 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: allocated 0kB Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: raid level 5 active with 6 out of 6 devices, algorithm 2 Dec 17 19:10:15 ITX002590129362 kernel: md0: detected capacity change from 0 to 2361059573760 Dec 17 19:10:15 ITX002590129362 kernel: md0: unknown partition table Dec 17 19:10:35 ITX002590129362 kernel: md: md0 switched to read-write mode.
> Dec 17 19:10:35 ITX002590129362 kernel: md: resync of RAID array md0 Dec 17 19:10:35 ITX002590129362 kernel: md: minimum _guaranteed_ speed: 10000 KB/sec/disk.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using maximum available idle IO bandwidth (but not more than 30000 KB/sec) for resync.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using 128k window, over a total of 461144448k.
> ???
> Started IOs using fio tool.
>
> ./fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> --blockalign=512
>
> ???
> Removed a drive form the system..
>
> Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) ..
> Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 02 69 03 70 00 00 10 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 40436592 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 51 b3 d0 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 206681040 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 3a f3 40 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 205189952 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK ??? Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb 00 00 00 08 00 Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:25 ITX002590129362 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb f0 00 00 10 00 Dec 17 19:13:25 ITX002590129362 aghswap: devpath [0:0:7:0] action [remove] devtype [scsi_disk] Dec 17 19:13:25 ITX002590129362 aghswap: MHSA: Sent event 0 0 7 0 remove scsi_disk Dec 17 19:13:25 ITX002590129362 kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x500605ba0101e305) Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Disk failure on sdh6, disabling device.
> Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Operation continuing on 5 devices.
> Dec 17 19:13:25 ITX002590129362 kernel: md: md0: resync interrupted.
> Dec 17 19:13:25 ITX002590129362 kernel: md: checkpointing resync of md0.
> ..
> Log messages after enabling debufgs on raid5.c, it is getting repeated continuously.
>
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0 handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328 for sector 273480328, rmw=2 rcw=1 handling stripe 65238568, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081a956b00 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=65238568 for sector 65238568, rmw=2 rcw=1 handling stripe 713868672, state=0x2041 cnt=1, pd_idx=4, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x10 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081f020100 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=713868672 for sector 713868672, rmw=2 rcw=1 handling stripe 729622496, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622496 for sector 729622496, rmw=2 rcw=1 handling stripe 729622504, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622504 for sector 729622504, rmw=2 rcw=1 handling stripe 245773680, state=0x2041 cnt=1, pd_idx=0, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081cab7a00 written (null)
> check 0: state 0x10 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=245773680 for sector 245773680, rmw=2 rcw=1 handling stripe 867965560, state=0x2041 cnt=1, pd_idx=1, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff880802b2bf00 written (null)
> check 1: state 0x10 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=867965560 for sector 867965560, rmw=2 rcw=1 handling stripe 550162280, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x18 read (null) write ffff880802b08800 written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=550162280 for
> sector 550162280, rmw=2 rcw=1
>
>
> Thanks,
> Manibalan
>
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:11 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> The same Issue is reproducible in the latest upstream kernel also.
>
> Tested in "3.17.6" latest stable upstream kernel and find the same issue.
>
> [root@root ~]# modinfo raid456
> filename: /lib/modules/3.17.6/kernel/drivers/md/raid456.ko
> alias: raid6
> alias: raid5
> alias: md-level-6
> alias: md-raid6
> alias: md-personality-8
> alias: md-level-4
> alias: md-level-5
> alias: md-raid4
> alias: md-raid5
> alias: md-personality-4
> description: RAID4/5/6 (striping with parity) personality for MD
> license: GPL
> srcversion: 0EEF680023FDC7410F7989A
> depends: async_raid6_recov,async_pq,async_tx,async_memcpy,async_xor
> intree: Y
> vermagic: 3.17.6 SMP mod_unload modversions
> parm: devices_handle_discard_safely:Set to Y if all devices in each array reliably return zeroes on reads from discarded regions (bool)
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:01 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> We are facing IO struck issue with raid5 in the following scenario.
> (please see the attachment for the complete information) In RAID5
> array, if a drive is removed while initialization and the same time if
> IO is happening to that md. Then IO is getting struck, and md_raid5
> thread is using 100 % of CPU. Also the md state showing as
> resync=PENDING
>
> Kernel : Issue found in the following kernels RHEL 6.5
> (2.6.32-431.el6.x86_64) CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
>
> Steps to Reproduce the issue:
>
> 1. Created a raid 5 md with 4 drives using the below mdadm command.
> mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6
> /dev/sdd6 /dev/sde6
>
> 2. Make the md writable
> mdadm ???readwrite /dev/md0
>
> 3. Now md will start initialization
>
> 4. Run FIO Tool, the the below said configuration /usr/bin/fio
> --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> --blockalign=512
>
> 4. During MD initialzing, remove a drive(either using MDADM set
> faulty/remove or remove manually)
>
> 5. Now the IO will struck, and cat /proc/mdstat shows states with
> resync=PENDING
> ----------------------------------------------------------------------
> ----------------------- top - output show, md_raid5 using 100% cpu
>
> top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
> ----------------------------------------------------------------------
> -----------------------
> dmesg - show the stack trace
>
> INFO: task fio:2715 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000a 0 2715 2654 0x00000080
> ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
> ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff8140fa39>] ?
> md_wakeup_thread+0x39/0x70 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffffa0308f66>] ?
> make_request+0x306/0xc6c [raid456] [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81122283>] ?
> mempool_alloc+0x63/0x140 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0 [<ffffffff811c8196>]
> __blockdev_direct_IO_newtrunc+0x346/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2717 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000004 0 2717 2654 0x00000080
> ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
> ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2718 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000005 0 2718 2654 0x00000080
> ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
> ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2719 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000001 0 2719 2654 0x00000080
> ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
> ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2720 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000008 0 2720 2654 0x00000080
> ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
> ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2721 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2721 2654 0x00000080
> ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
> ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2722 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2722 2654 0x00000080
> ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
> ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2723 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000006 0 2723 2654 0x00000080
> ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
> ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2724 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000b 0 2724 2654 0x00000080
> ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
> ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2725 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000003 0 2725 2654 0x00000080
> ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
> ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
>
> [root@root ~]# cat /proc/2690/stack
> [<ffffffff810686da>] __cond_resched+0x2a/0x40 [<ffffffffa030361c>]
> ops_run_io+0x2c/0x920 [raid456] [<ffffffffa03052cc>]
> handle_stripe+0x9cc/0x2980 [raid456] [<ffffffffa03078a4>]
> raid5d+0x624/0x850 [raid456] [<ffffffff81416f05>]
> md_thread+0x115/0x150 [<ffffffff8109aef6>] kthread+0x96/0xa0
> [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffffffffffff>]
> 0xffffffffffffffff
>
> [root@root ~]# cat /proc/2690/stat
> 2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0
> 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17
> 2 0 0 6855 0 0 [root@root ~]# cat /proc/2690/statm
> 0 0 0 0 0 0 0
> [root@root ~]# cat /proc/2690/stat
> stat statm status
> [root@root ~]# cat /proc/2690/status
> Name: md0_raid5
> State: R (running)
> Tgid: 2690
> Pid: 2690
> PPid: 2
> TracerPid: 0
> Uid: 0 0 0 0
> Gid: 0 0 0 0
> Utrace: 0
> FDSize: 64
> Groups:
> Threads: 1
> SigQ: 2/128402
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: fffffffffffffeff
> SigCgt: 0000000000000100
> CapInh: 0000000000000000
> CapPrm: ffffffffffffffff
> CapEff: fffffffffffffeff
> CapBnd: ffffffffffffffff
> Cpus_allowed: ffffff
> Cpus_allowed_list: 0-23
> Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
> Mems_allowed_list: 0-1
> voluntary_ctxt_switches: 5411612
> nonvoluntary_ctxt_switches: 257032
>
>
> Thanks,
> Manibalan.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Array died during grow; now resync stopped
From: Jörg Habenicht @ 2015-02-02 9:41 UTC (permalink / raw)
To: linux-raid
Hi all,
I had a server crash during an array grow.
Commandline was "mdadm --grow /dev/md0 --raid-devices=6 --chunk=1M"
Now the sync is stuck at 27% and wont continue.
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6]
[UUUUUU]
[=====>...............] reshape = 27.9% (410229760/1465137152)
finish=8670020128.0min speed=0K/sec
unused devices: <none>
$ mdadm -D /dev/md0
/dev/md0:
Version : 1.0
Creation Time : Thu Oct 7 09:28:04 2010
Raid Level : raid5
Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
Used Dev Size : 1465137152 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Sun Feb 1 13:30:05 2015
State : clean, reshaping
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Reshape Status : 27% complete
Delta Devices : 1, (5->6)
New Chunksize : 1024K
Name : stelli:3 (local to host stelli)
UUID : 52857d77:3806e446:477d4865:d711451e
Events : 2254869
Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sde1
5 8 81 1 active sync /dev/sdf1
8 8 49 2 active sync /dev/sdd1
7 8 17 3 active sync /dev/sdb1
6 8 33 4 active sync /dev/sdc1
9 8 97 5 active sync /dev/sdg1
smartcrl reports the disks are OK. No remapped sectors, no pending writes, etc.
The system load keeps at 2.0:
$ cat /proc/loadavg
2.00 2.00 1.95 1/140 2937
which may be caused by udevd and md0_reshape
$ ps fax
PID TTY STAT TIME COMMAND
2 ? S 0:00 [kthreadd]
...
1671 ? D 0:00 \_ [md0_reshape]
...
1289 ? Ss 0:01 /sbin/udevd --daemon
1672 ? D 0:00 \_ /sbin/udevd --daemon
Could this be caused by a software lock?
The system got 2G RAM and 2G swap. Is this sufficient to complete?
$ free
total used free shared buffers cached
Mem: 1799124 351808 1447316 540 14620 286216
-/+ buffers/cache: 50972 1748152
Swap: 2104508 0 2104508
And in dmesg I found this:
$ dmesg | less
[ 5.456941] md: bind<sdg1>
[ 11.015014] xor: measuring software checksum speed
[ 11.051384] prefetch64-sse: 3291.000 MB/sec
[ 11.091375] generic_sse: 3129.000 MB/sec
[ 11.091378] xor: using function: prefetch64-sse (3291.000 MB/sec)
[ 11.159365] raid6: sse2x1 1246 MB/s
[ 11.227343] raid6: sse2x2 2044 MB/s
[ 11.295327] raid6: sse2x4 2487 MB/s
[ 11.295331] raid6: using algorithm sse2x4 (2487 MB/s)
[ 11.295334] raid6: using intx1 recovery algorithm
[ 11.328771] md: raid6 personality registered for level 6
[ 11.328776] md: raid5 personality registered for level 5
[ 11.328779] md: raid4 personality registered for level 4
[ 19.840890] bio: create slab <bio-1> at 1
[ 159.701406] md: md0 stopped.
[ 159.701413] md: unbind<sdg1>
[ 159.709902] md: export_rdev(sdg1)
[ 159.709980] md: unbind<sdd1>
[ 159.721856] md: export_rdev(sdd1)
[ 159.721955] md: unbind<sdb1>
[ 159.733883] md: export_rdev(sdb1)
[ 159.733991] md: unbind<sdc1>
[ 159.749856] md: export_rdev(sdc1)
[ 159.749954] md: unbind<sdf1>
[ 159.769885] md: export_rdev(sdf1)
[ 159.769985] md: unbind<sde1>
[ 159.781873] md: export_rdev(sde1)
[ 160.471460] md: md0 stopped.
[ 160.490329] md: bind<sdf1>
[ 160.490478] md: bind<sdd1>
[ 160.490689] md: bind<sdb1>
[ 160.490911] md: bind<sdc1>
[ 160.491164] md: bind<sdg1>
[ 160.491408] md: bind<sde1>
[ 160.492616] md/raid:md0: reshape will continue
[ 160.492638] md/raid:md0: device sde1 operational as raid disk 0
[ 160.492640] md/raid:md0: device sdg1 operational as raid disk 5
[ 160.492641] md/raid:md0: device sdc1 operational as raid disk 4
[ 160.492642] md/raid:md0: device sdb1 operational as raid disk 3
[ 160.492644] md/raid:md0: device sdd1 operational as raid disk 2
[ 160.492645] md/raid:md0: device sdf1 operational as raid disk 1
[ 160.493187] md/raid:md0: allocated 0kB
[ 160.493253] md/raid:md0: raid level 5 active with 6 out of 6 devices,
algorithm 2
[ 160.493256] RAID conf printout:
[ 160.493257] --- level:5 rd:6 wd:6
[ 160.493259] disk 0, o:1, dev:sde1
[ 160.493261] disk 1, o:1, dev:sdf1
[ 160.493262] disk 2, o:1, dev:sdd1
[ 160.493263] disk 3, o:1, dev:sdb1
[ 160.493264] disk 4, o:1, dev:sdc1
[ 160.493266] disk 5, o:1, dev:sdg1
[ 160.493336] md0: detected capacity change from 0 to 6001201774592
[ 160.493340] md: reshape of RAID array md0
[ 160.493342] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 160.493343] md: using maximum available idle IO bandwidth (but not more
than 200000 KB/sec) for reshape.
[ 160.493351] md: using 128k window, over a total of 1465137152k.
[ 160.951404] md0: unknown partition table
[ 190.984871] udevd[1289]: worker [1672] /devices/virtual/block/md0
timeout; kill it
[ 190.984901] udevd[1289]: seq 2259 '/devices/virtual/block/md0' killed
$ mdadm --version
mdadm - v3.3.1 - 5th June 2014
uname -a
Linux XXXXX 3.14.14-gentoo #3 SMP Sat Jan 31 18:45:04 CET 2015 x86_64 AMD
Athlon(tm) II X2 240e Processor AuthenticAMD GNU/Linux
Currently I can't access the array to read the remaining data, nor can I
continue the array grow.
Can you help me get it running?
best regards
Jörg
^ permalink raw reply
* update mdadm version to 3.3.2 in Centos 6.4
From: alpha lin @ 2015-02-02 14:30 UTC (permalink / raw)
To: linux-raid
Hi All,
I would like to know the possibility of update the mdadm package
from 3.2.5 to 3.3.2 in Redhat Centos 6.4 x64 system.
The problem I have is I found that I can create a raid volume
without any problem. But when reboot the system. The md volume become
inactive.
Below is my step:
1. Check out mdadm package from the github.
2. unzip the mdadm package and execute "make" and "make install"
3. reboot.
4. after system reboot, run mdadm to create a imsm RAID Volume.
[root@localhost rules.d]# mdadm --version
mdadm - v3.3.2 - 21st August 2014
[root@localhost rules.d]# mdadm --detail-platform
Platform : Intel(R) Rapid Storage Technology enterprise
Version : 3.7.0.1049
RAID Levels : raid0 raid1 raid10 raid5
Chunk Sizes : 4k 8k 16k 32k 64k 128k
2TB volumes : supported
2TB disks : supported
Max Disks : 6
Max Volumes : 2 per array, 4 per controller
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
Port2 : /dev/sda (3JV9BZRP)
Port3 : /dev/sdb (1346095A3033)
Port4 : /dev/sdc (1346095A2FEC)
Port5 : /dev/sdd (1346095A3042)
Port0 : - no device attached -
Port1 : - no device attached -
[root@localhost rules.d]# mdadm -C /dev/md0 /dev/sd[b-d] -n 3 -e imsm
mdadm: /dev/sdb appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
mdadm: /dev/sdc appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
mdadm: /dev/sdd appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
Continue creating array? y
mdadm: container /dev/md0 prepared.
[root@localhost rules.d]# mdadm -C /dev/md/Volume0 /dev/md0 -n 3 -l 5
mdadm: array /dev/md/Volume0 started.
5. check raid stat
[root@localhost rules.d]# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sdb[2] sdd[1] sdc[0]
234434560 blocks super external:/md0/0 level 5, 128k chunk,
algorithm 0 [3/3] [UUU]
[>....................] resync = 0.8% (1027072/117217664)
finish=18.8min speed=102707K/sec
md0 : inactive sdd[2](S) sdc[1](S) sdb[0](S)
3315 blocks super external:imsm
unused devices: <none>
[root@localhost rules.d]#
[root@localhost dev]# mdmon md0
mdmon: md0 already managed
6. reboot the system again. check the md again.
[root@localhost Desktop]# cat /proc/mdstat
Personalities :
md125 : inactive sdi[1](S) sdh[0](S)
6306 blocks super external:imsm
md126 : inactive sdg[0](S)
3153 blocks super external:imsm
md127 : inactive sdg[0]
117217664 blocks super external:/md126/0
unused devices: <none>
[root@localhost Desktop]#
I found the device label change from sd[a-d] to sd[f-i]
[root@localhost rules.d]# mdadm --detail-platform
Platform : Intel(R) Rapid Storage Technology enterprise
Version : 3.7.0.1049
RAID Levels : raid0 raid1 raid10 raid5
Chunk Sizes : 4k 8k 16k 32k 64k 128k
2TB volumes : supported
2TB disks : supported
Max Disks : 6
Max Volumes : 2 per array, 4 per controller
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
Port2 : /dev/sdf (3JV9BZRP)
Port3 : /dev/sdg (1346095A3033)
Port4 : /dev/sdh (1346095A2FEC)
Port5 : /dev/sdi (1346095A3042)
Port0 : - no device attached -
Port1 : - no device attached -
Any idea? Does anything missing in my setup ?
^ permalink raw reply
* [PATCH 0/3] md: Deletion of a few unnecessary checks
From: SF Markus Elfring @ 2015-02-02 15:11 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, linux-raid
Cc: LKML, kernel-janitors, Julia Lawall
In-Reply-To: <5317A59D.4@users.sourceforge.net>
From: Markus Elfring <elfring@users.sourceforge.net>
Date: Mon, 2 Feb 2015 15:51:43 +0100
Further update suggestions were taken into account together with results
from static source code analysis.
Markus Elfring (3):
dm snapshot: Deletion of unnecessary checks before the function call "vfree"
md/bitmap: Delete an unnecessary check before the function call "kfree"
dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
drivers/md/bitmap.c | 10 +++++-----
drivers/md/dm-ioctl.c | 3 +--
drivers/md/dm-snap-persistent.c | 14 ++++----------
3 files changed, 10 insertions(+), 17 deletions(-)
--
2.2.2
^ permalink raw reply
* [PATCH 1/3] dm snapshot: Deletion of unnecessary checks before the function call "vfree"
From: SF Markus Elfring @ 2015-02-02 15:17 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, dm-devel, linux-raid
Cc: LKML, kernel-janitors, Julia Lawall
In-Reply-To: <54CF93BB.40206@users.sourceforge.net>
From: Markus Elfring <elfring@users.sourceforge.net>
Date: Mon, 2 Feb 2015 14:38:29 +0100
The vfree() function performs also input parameter validation.
Thus the test around the call is not needed.
This issue was detected by using the Coccinelle software.
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
---
drivers/md/dm-snap-persistent.c | 14 ++++----------
1 file changed, 4 insertions(+), 10 deletions(-)
diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c
index d6e8817..808b841 100644
--- a/drivers/md/dm-snap-persistent.c
+++ b/drivers/md/dm-snap-persistent.c
@@ -200,16 +200,11 @@ err_area:
static void free_area(struct pstore *ps)
{
- if (ps->area)
- vfree(ps->area);
+ vfree(ps->area);
ps->area = NULL;
-
- if (ps->zero_area)
- vfree(ps->zero_area);
+ vfree(ps->zero_area);
ps->zero_area = NULL;
-
- if (ps->header_area)
- vfree(ps->header_area);
+ vfree(ps->header_area);
ps->header_area = NULL;
}
@@ -605,8 +600,7 @@ static void persistent_dtr(struct dm_exception_store *store)
free_area(ps);
/* Allocated in persistent_read_metadata */
- if (ps->callbacks)
- vfree(ps->callbacks);
+ vfree(ps->callbacks);
kfree(ps);
}
--
2.2.2
^ permalink raw reply related
* [PATCH 2/3] md/bitmap: Delete an unnecessary check before the function call "kfree"
From: SF Markus Elfring @ 2015-02-02 15:20 UTC (permalink / raw)
To: Neil Brown, linux-raid; +Cc: LKML, kernel-janitors, Julia Lawall
In-Reply-To: <54CF93BB.40206@users.sourceforge.net>
From: Markus Elfring <elfring@users.sourceforge.net>
Date: Mon, 2 Feb 2015 15:10:57 +0100
The kfree() function tests whether its argument is NULL and then
returns immediately. Thus the test around the call is not needed.
* This issue was detected by using the Coccinelle software.
* Let us also move an assignment for the variable "pages" to the place
directly before it is really needed for a loop.
* Let us also move another kfree() call into a block which should belong
to a previous check for the variable "bp".
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
---
drivers/md/bitmap.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index da3604e..47d72df 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1586,15 +1586,15 @@ static void bitmap_free(struct bitmap *bitmap)
bitmap_file_unmap(&bitmap->storage);
bp = bitmap->counts.bp;
- pages = bitmap->counts.pages;
/* free all allocated memory */
-
- if (bp) /* deallocate the page memory */
+ if (bp) { /* deallocate the page memory */
+ pages = bitmap->counts.pages;
for (k = 0; k < pages; k++)
- if (bp[k].map && !bp[k].hijacked)
+ if (!bp[k].hijacked)
kfree(bp[k].map);
- kfree(bp);
+ kfree(bp);
+ }
kfree(bitmap);
}
--
2.2.2
^ permalink raw reply related
* [PATCH 3/3] dm ioctl: Delete an unnecessary check before the function call "dm_table_destroy"
From: SF Markus Elfring @ 2015-02-02 15:23 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, dm-devel, linux-raid
Cc: LKML, kernel-janitors, Julia Lawall
In-Reply-To: <54CF93BB.40206@users.sourceforge.net>
From: Markus Elfring <elfring@users.sourceforge.net>
Date: Mon, 2 Feb 2015 15:30:37 +0100
The dm_table_destroy() function tests whether its argument is NULL and then
returns immediately. Thus the test around the call is not needed.
This issue was detected by using the Coccinelle software.
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
---
drivers/md/dm-ioctl.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index 73f791b..4fac6cf 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1053,8 +1053,7 @@ static int do_resume(struct dm_ioctl *param)
* Since dm_swap_table synchronizes RCU, nobody should be in
* read-side critical section already.
*/
- if (old_map)
- dm_table_destroy(old_map);
+ dm_table_destroy(old_map);
if (!r)
__dev_status(md, param);
--
2.2.2
^ permalink raw reply related
* Re: Array died during grow; now resync stopped
From: NeilBrown @ 2015-02-02 19:40 UTC (permalink / raw)
To: Jörg Habenicht; +Cc: linux-raid
In-Reply-To: <loom.20150202T100454-579@post.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 6762 bytes --]
On Mon, 2 Feb 2015 09:41:02 +0000 (UTC) Jörg Habenicht <j.habenicht@gmx.de>
wrote:
> Hi all,
>
> I had a server crash during an array grow.
> Commandline was "mdadm --grow /dev/md0 --raid-devices=6 --chunk=1M"
>
> Now the sync is stuck at 27% and wont continue.
> $ cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sde1[0] sdg1[9] sdc1[6] sdb1[7] sdd1[8] sdf1[5]
> 5860548608 blocks super 1.0 level 5, 256k chunk, algorithm 2 [6/6]
> [UUUUUU]
> [=====>...............] reshape = 27.9% (410229760/1465137152)
> finish=8670020128.0min speed=0K/sec
>
> unused devices: <none>
>
>
> $ mdadm -D /dev/md0
> /dev/md0:
> Version : 1.0
> Creation Time : Thu Oct 7 09:28:04 2010
> Raid Level : raid5
> Array Size : 5860548608 (5589.05 GiB 6001.20 GB)
> Used Dev Size : 1465137152 (1397.26 GiB 1500.30 GB)
> Raid Devices : 6
> Total Devices : 6
> Persistence : Superblock is persistent
>
> Update Time : Sun Feb 1 13:30:05 2015
> State : clean, reshaping
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Reshape Status : 27% complete
> Delta Devices : 1, (5->6)
> New Chunksize : 1024K
>
> Name : stelli:3 (local to host stelli)
> UUID : 52857d77:3806e446:477d4865:d711451e
> Events : 2254869
>
> Number Major Minor RaidDevice State
> 0 8 65 0 active sync /dev/sde1
> 5 8 81 1 active sync /dev/sdf1
> 8 8 49 2 active sync /dev/sdd1
> 7 8 17 3 active sync /dev/sdb1
> 6 8 33 4 active sync /dev/sdc1
> 9 8 97 5 active sync /dev/sdg1
>
>
> smartcrl reports the disks are OK. No remapped sectors, no pending writes, etc.
>
> The system load keeps at 2.0:
> $ cat /proc/loadavg
> 2.00 2.00 1.95 1/140 2937
> which may be caused by udevd and md0_reshape
> $ ps fax
> PID TTY STAT TIME COMMAND
> 2 ? S 0:00 [kthreadd]
> ...
> 1671 ? D 0:00 \_ [md0_reshape]
> ...
> 1289 ? Ss 0:01 /sbin/udevd --daemon
> 1672 ? D 0:00 \_ /sbin/udevd --daemon
>
>
> Could this be caused by a software lock?
Some sort of software problem I suspect.
What does
cat /proc/1671/stack
cat /proc/1672/stack
show?
Alternatively,
echo w > /proc/sysrq-trigger
and see what appears in 'dmesg'.
>
> The system got 2G RAM and 2G swap. Is this sufficient to complete?
Memory shouldn't be a problem.
However it wouldn't hurt to see what value is in
/sys/block/md0/md/stripe_cache_size
and double it.
If all else fails a reboot should be safe and will probably start the reshape
properly. md is very careful about surviving reboots.
NeilBrown
> $ free
> total used free shared buffers cached
> Mem: 1799124 351808 1447316 540 14620 286216
> -/+ buffers/cache: 50972 1748152
> Swap: 2104508 0 2104508
>
>
> And in dmesg I found this:
> $ dmesg | less
> [ 5.456941] md: bind<sdg1>
> [ 11.015014] xor: measuring software checksum speed
> [ 11.051384] prefetch64-sse: 3291.000 MB/sec
> [ 11.091375] generic_sse: 3129.000 MB/sec
> [ 11.091378] xor: using function: prefetch64-sse (3291.000 MB/sec)
> [ 11.159365] raid6: sse2x1 1246 MB/s
> [ 11.227343] raid6: sse2x2 2044 MB/s
> [ 11.295327] raid6: sse2x4 2487 MB/s
> [ 11.295331] raid6: using algorithm sse2x4 (2487 MB/s)
> [ 11.295334] raid6: using intx1 recovery algorithm
> [ 11.328771] md: raid6 personality registered for level 6
> [ 11.328776] md: raid5 personality registered for level 5
> [ 11.328779] md: raid4 personality registered for level 4
> [ 19.840890] bio: create slab <bio-1> at 1
> [ 159.701406] md: md0 stopped.
> [ 159.701413] md: unbind<sdg1>
> [ 159.709902] md: export_rdev(sdg1)
> [ 159.709980] md: unbind<sdd1>
> [ 159.721856] md: export_rdev(sdd1)
> [ 159.721955] md: unbind<sdb1>
> [ 159.733883] md: export_rdev(sdb1)
> [ 159.733991] md: unbind<sdc1>
> [ 159.749856] md: export_rdev(sdc1)
> [ 159.749954] md: unbind<sdf1>
> [ 159.769885] md: export_rdev(sdf1)
> [ 159.769985] md: unbind<sde1>
> [ 159.781873] md: export_rdev(sde1)
> [ 160.471460] md: md0 stopped.
> [ 160.490329] md: bind<sdf1>
> [ 160.490478] md: bind<sdd1>
> [ 160.490689] md: bind<sdb1>
> [ 160.490911] md: bind<sdc1>
> [ 160.491164] md: bind<sdg1>
> [ 160.491408] md: bind<sde1>
> [ 160.492616] md/raid:md0: reshape will continue
> [ 160.492638] md/raid:md0: device sde1 operational as raid disk 0
> [ 160.492640] md/raid:md0: device sdg1 operational as raid disk 5
> [ 160.492641] md/raid:md0: device sdc1 operational as raid disk 4
> [ 160.492642] md/raid:md0: device sdb1 operational as raid disk 3
> [ 160.492644] md/raid:md0: device sdd1 operational as raid disk 2
> [ 160.492645] md/raid:md0: device sdf1 operational as raid disk 1
> [ 160.493187] md/raid:md0: allocated 0kB
> [ 160.493253] md/raid:md0: raid level 5 active with 6 out of 6 devices,
> algorithm 2
> [ 160.493256] RAID conf printout:
> [ 160.493257] --- level:5 rd:6 wd:6
> [ 160.493259] disk 0, o:1, dev:sde1
> [ 160.493261] disk 1, o:1, dev:sdf1
> [ 160.493262] disk 2, o:1, dev:sdd1
> [ 160.493263] disk 3, o:1, dev:sdb1
> [ 160.493264] disk 4, o:1, dev:sdc1
> [ 160.493266] disk 5, o:1, dev:sdg1
> [ 160.493336] md0: detected capacity change from 0 to 6001201774592
> [ 160.493340] md: reshape of RAID array md0
> [ 160.493342] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 160.493343] md: using maximum available idle IO bandwidth (but not more
> than 200000 KB/sec) for reshape.
> [ 160.493351] md: using 128k window, over a total of 1465137152k.
> [ 160.951404] md0: unknown partition table
> [ 190.984871] udevd[1289]: worker [1672] /devices/virtual/block/md0
> timeout; kill it
> [ 190.984901] udevd[1289]: seq 2259 '/devices/virtual/block/md0' killed
>
>
>
> $ mdadm --version
> mdadm - v3.3.1 - 5th June 2014
>
> uname -a
> Linux XXXXX 3.14.14-gentoo #3 SMP Sat Jan 31 18:45:04 CET 2015 x86_64 AMD
> Athlon(tm) II X2 240e Processor AuthenticAMD GNU/Linux
>
>
> Currently I can't access the array to read the remaining data, nor can I
> continue the array grow.
> Can you help me get it running?
>
>
> best regards
> Jörg
>
>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 2/3] md/bitmap: Delete an unnecessary check before the function call "kfree"
From: NeilBrown @ 2015-02-02 19:46 UTC (permalink / raw)
To: SF Markus Elfring; +Cc: linux-raid, LKML, kernel-janitors, Julia Lawall
In-Reply-To: <54CF95CA.90102@users.sourceforge.net>
[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]
On Mon, 02 Feb 2015 16:20:42 +0100 SF Markus Elfring
<elfring@users.sourceforge.net> wrote:
> From: Markus Elfring <elfring@users.sourceforge.net>
> Date: Mon, 2 Feb 2015 15:10:57 +0100
>
> The kfree() function tests whether its argument is NULL and then
> returns immediately. Thus the test around the call is not needed.
>
> * This issue was detected by using the Coccinelle software.
>
> * Let us also move an assignment for the variable "pages" to the place
> directly before it is really needed for a loop.
>
> * Let us also move another kfree() call into a block which should belong
> to a previous check for the variable "bp".
>
> Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
> ---
> drivers/md/bitmap.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index da3604e..47d72df 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1586,15 +1586,15 @@ static void bitmap_free(struct bitmap *bitmap)
> bitmap_file_unmap(&bitmap->storage);
>
> bp = bitmap->counts.bp;
> - pages = bitmap->counts.pages;
>
> /* free all allocated memory */
> -
> - if (bp) /* deallocate the page memory */
> + if (bp) { /* deallocate the page memory */
> + pages = bitmap->counts.pages;
> for (k = 0; k < pages; k++)
> - if (bp[k].map && !bp[k].hijacked)
> + if (!bp[k].hijacked)
> kfree(bp[k].map);
> - kfree(bp);
> + kfree(bp);
> + }
> kfree(bitmap);
> }
>
Hi,
I'm somewhat amused that you removed a test for one kfree, but imposed a
test on another. I realised the second test was already there, but why not
just:
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index da3604e73e8a..ad13b2e1bf1f 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1592,7 +1592,7 @@ static void bitmap_free(struct bitmap *bitmap)
if (bp) /* deallocate the page memory */
for (k = 0; k < pages; k++)
- if (bp[k].map && !bp[k].hijacked)
+ if (!bp[k].hijacked)
kfree(bp[k].map);
kfree(bp);
kfree(bitmap);
It makes the intention of the patch much clearer.
I'd probably prefer to leave the code as it is. I don't think either patch
is really an improvement in readability, and readability trumps performance
in places like this.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Fwd: update mdadm version to 3.3.2 in Centos 6.4
From: alpha lin @ 2015-02-02 22:26 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAKtQoftv3uf-Y1va7QV=qFiJZO0CdSJ7QtfHj7P1Mg5oyj0fRw@mail.gmail.com>
Forgot to mention that my Centos 6.4 x64 system has upgrade kernel to 3.16.0.
So the failure is base on kernel 3.16.0
---------- Forwarded message ----------
From: alpha lin <lin.alpha@gmail.com>
Date: Mon, Feb 2, 2015 at 10:30 PM
Subject: update mdadm version to 3.3.2 in Centos 6.4
To: linux-raid@vger.kernel.org
Hi All,
I would like to know the possibility of update the mdadm package
from 3.2.5 to 3.3.2 in Redhat Centos 6.4 x64 system.
The problem I have is I found that I can create a raid volume
without any problem. But when reboot the system. The md volume become
inactive.
Below is my step:
1. Check out mdadm package from the github.
2. unzip the mdadm package and execute "make" and "make install"
3. reboot.
4. after system reboot, run mdadm to create a imsm RAID Volume.
[root@localhost rules.d]# mdadm --version
mdadm - v3.3.2 - 21st August 2014
[root@localhost rules.d]# mdadm --detail-platform
Platform : Intel(R) Rapid Storage Technology enterprise
Version : 3.7.0.1049
RAID Levels : raid0 raid1 raid10 raid5
Chunk Sizes : 4k 8k 16k 32k 64k 128k
2TB volumes : supported
2TB disks : supported
Max Disks : 6
Max Volumes : 2 per array, 4 per controller
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
Port2 : /dev/sda (3JV9BZRP)
Port3 : /dev/sdb (1346095A3033)
Port4 : /dev/sdc (1346095A2FEC)
Port5 : /dev/sdd (1346095A3042)
Port0 : - no device attached -
Port1 : - no device attached -
[root@localhost rules.d]# mdadm -C /dev/md0 /dev/sd[b-d] -n 3 -e imsm
mdadm: /dev/sdb appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
mdadm: /dev/sdc appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
mdadm: /dev/sdd appears to be part of a raid array:
level=container devices=0 ctime=Thu Jan 1 08:00:00 1970
Continue creating array? y
mdadm: container /dev/md0 prepared.
[root@localhost rules.d]# mdadm -C /dev/md/Volume0 /dev/md0 -n 3 -l 5
mdadm: array /dev/md/Volume0 started.
5. check raid stat
[root@localhost rules.d]# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sdb[2] sdd[1] sdc[0]
234434560 blocks super external:/md0/0 level 5, 128k chunk,
algorithm 0 [3/3] [UUU]
[>....................] resync = 0.8% (1027072/117217664)
finish=18.8min speed=102707K/sec
md0 : inactive sdd[2](S) sdc[1](S) sdb[0](S)
3315 blocks super external:imsm
unused devices: <none>
[root@localhost rules.d]#
[root@localhost dev]# mdmon md0
mdmon: md0 already managed
6. reboot the system again. check the md again.
[root@localhost Desktop]# cat /proc/mdstat
Personalities :
md125 : inactive sdi[1](S) sdh[0](S)
6306 blocks super external:imsm
md126 : inactive sdg[0](S)
3153 blocks super external:imsm
md127 : inactive sdg[0]
117217664 blocks super external:/md126/0
unused devices: <none>
[root@localhost Desktop]#
I found the device label change from sd[a-d] to sd[f-i]
[root@localhost rules.d]# mdadm --detail-platform
Platform : Intel(R) Rapid Storage Technology enterprise
Version : 3.7.0.1049
RAID Levels : raid0 raid1 raid10 raid5
Chunk Sizes : 4k 8k 16k 32k 64k 128k
2TB volumes : supported
2TB disks : supported
Max Disks : 6
Max Volumes : 2 per array, 4 per controller
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
Port2 : /dev/sdf (3JV9BZRP)
Port3 : /dev/sdg (1346095A3033)
Port4 : /dev/sdh (1346095A2FEC)
Port5 : /dev/sdi (1346095A3042)
Port0 : - no device attached -
Port1 : - no device attached -
Any idea? Does anything missing in my setup ?
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: NeilBrown @ 2015-02-02 22:30 UTC (permalink / raw)
To: Manibalan P; +Cc: Pasi Kärkkäinen, linux-raid
In-Reply-To: <CD8664C5675EDF49A5E76D7DB099D7B326130D6E@VENUS1.in.megatrends.com>
[-- Attachment #1: Type: text/plain, Size: 44625 bytes --]
On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
wrote:
> Dear All,
> Any updates on this issue.
Probably the same as:
http://marc.info/?l=linux-raid&m=142283560704091&w=2
which follows on from
http://marc.info/?t=142221642300001&r=1&w=2
and
http://marc.info/?t=142172432500001&r=1&w=2
NeilBrown
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, January 14, 2015 3:55 PM
> To: 'Pasi Kärkkäinen'
> Cc: 'neilb@suse.de'; 'linux-raid'
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> Dear Pasi,
> Could you able to find something on this issue.
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Friday, January 2, 2015 12:08 PM
> To: 'Pasi Kärkkäinen'
> Cc: neilb@suse.de; linux-raid
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> Dear Pasi,
>
> I have add the bug in https://bugzilla.redhat.com/show_bug.cgi?id=1178080
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Sent: Wednesday, December 31, 2014 10:18 PM
> To: Manibalan P
> Cc: neilb@suse.de; linux-raid
> Subject: Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> On Tue, Dec 30, 2014 at 11:06:47AM +0000, Manibalan P wrote:
> > Dear Neil,
> >
>
> Hello,
>
> > Few this for you kind attention,
> > 1. I tried the same test with FC11 (2.6.32 kernel before MD code
> > change). And the issue is not there 2. But with Centos 6.4 (2.6.32 kernel after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
> >
> > Also, a bug has been raise with RHEL regarding this issue. Please find the bug link "https://access.redhat.com/support/cases/#/case/01320319"
> >
>
> That support case URL can only be accessed by you and Redhat. Do you happen to have a public bugzilla link?
>
>
> Thanks,
>
> -- Pasi
>
> > Thanks,
> > Manibalan.
> >
> > -----Original Message-----
> > From: Manibalan P
> > Sent: Wednesday, December 24, 2014 12:15 PM
> > To: neilb@suse.de; 'linux-raid'
> > Cc: 'NeilBrown'
> > Subject: RE: md_raid5 using 100% CPU and hang with status
> > resync=PENDING, if a drive is removed during initialization
> >
> >
> > Dear Neil,
> >
> > Few this for you kind attention,
> > 1. I tried the same tesst with FC11 (2.6 kernel before MD code change). And the issue is not there 2. But with Centos 6.4 (2.6 after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
> >
> > Thanks,
> > Manibalan.
> >
> > -----Original Message-----
> > From: Manibalan P
> > Sent: Thursday, December 18, 2014 11:38 AM
> > To: 'linux-raid'
> > Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> > Subject: RE: md_raid5 using 100% CPU and hang with status
> > resync=PENDING, if a drive is removed during initialization
> >
> > Dear neil,
> >
> > I also compiled the latest 3.18 kernel on CentOS 6.4 with GIT MD pull patches form 3.19, that also ran in to the same issue after removing a drive during resync.
> >
> > Dec 17 19:07:32 ITX002590129362 kernel: Linux version 3.18.0 (root@mycentos6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Dec 17 15:59:09 EST 2014 Dec 17 19:07:32 ITX002590129362 kernel: Command line: ro root=/dev/md255 rd_NO_LVM rd_NO_DM rhgb quiet md_mod.start_ro=1 nmi_watchdog=1 md_mod.start_dirty_degraded=1 ??? Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sda6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdb6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdc6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdh6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdi6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdj6> Dec 17 19:10:15 ITX002590129362 kernel: async_tx: api initialized (async) Dec 17 19:10:15 ITX002590129362 kernel: xor: measuring software checksum speed
> > Dec 17 19:10:15 ITX002590129362 kernel: prefetch64-sse: 10048.000 MB/sec
> > Dec 17 19:10:15 ITX002590129362 kernel: generic_sse: 8824.000 MB/sec
> > Dec 17 19:10:15 ITX002590129362 kernel: xor: using function: prefetch64-sse (10048.000 MB/sec)
> > Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x1 5921 MB/s
> > Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x2 6933 MB/s
> > Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x4 7476 MB/s
> > Dec 17 19:10:15 ITX002590129362 kernel: raid6: using algorithm sse2x4
> > (7476 MB/s) Dec 17 19:10:15 ITX002590129362 kernel: raid6: using
> > ssse3x2 recovery algorithm Dec 17 19:10:15 ITX002590129362 kernel: md:
> > raid6 personality registered for level 6 Dec 17 19:10:15
> > ITX002590129362 kernel: md: raid5 personality registered for level 5
> > Dec 17 19:10:15 ITX002590129362 kernel: md: raid4 personality
> > registered for level 4 Dec 17 19:10:15 ITX002590129362 kernel:
> > md/raid:md0: not clean -- starting background reconstruction Dec 17
> > 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdj6 operational
> > as raid disk 5 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> > device sdi6 operational as raid disk 4 Dec 17 19:10:15 ITX002590129362
> > kernel: md/raid:md0: device sdh6 operational as raid disk 3 Dec 17
> > 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdc6 operational
> > as raid disk 2 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> > device sdb6 operational as raid disk 1 Dec 17 19:10:15 ITX002590129362
> > kernel: md/ra
> > id:md0: device sda6 operational as raid disk 0 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: allocated 0kB Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: raid level 5 active with 6 out of 6 devices, algorithm 2 Dec 17 19:10:15 ITX002590129362 kernel: md0: detected capacity change from 0 to 2361059573760 Dec 17 19:10:15 ITX002590129362 kernel: md0: unknown partition table Dec 17 19:10:35 ITX002590129362 kernel: md: md0 switched to read-write mode.
> > Dec 17 19:10:35 ITX002590129362 kernel: md: resync of RAID array md0 Dec 17 19:10:35 ITX002590129362 kernel: md: minimum _guaranteed_ speed: 10000 KB/sec/disk.
> > Dec 17 19:10:35 ITX002590129362 kernel: md: using maximum available idle IO bandwidth (but not more than 30000 KB/sec) for resync.
> > Dec 17 19:10:35 ITX002590129362 kernel: md: using 128k window, over a total of 461144448k.
> > ???
> > Started IOs using fio tool.
> >
> > ./fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> > --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> > --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> > --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> > --blockalign=512
> >
> > ???
> > Removed a drive form the system..
> >
> > Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) ..
> > Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> > Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 02 69 03 70 00 00 10 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 40436592 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> > Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 51 b3 d0 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 206681040 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> > Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 3a f3 40 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 205189952 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK ??? Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> > Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb 00 00 00 08 00 Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:25 ITX002590129362 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> > Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb f0 00 00 10 00 Dec 17 19:13:25 ITX002590129362 aghswap: devpath [0:0:7:0] action [remove] devtype [scsi_disk] Dec 17 19:13:25 ITX002590129362 aghswap: MHSA: Sent event 0 0 7 0 remove scsi_disk Dec 17 19:13:25 ITX002590129362 kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x500605ba0101e305) Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Disk failure on sdh6, disabling device.
> > Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Operation continuing on 5 devices.
> > Dec 17 19:13:25 ITX002590129362 kernel: md: md0: resync interrupted.
> > Dec 17 19:13:25 ITX002590129362 kernel: md: checkpointing resync of md0.
> > ..
> > Log messages after enabling debufgs on raid5.c, it is getting repeated continuously.
> >
> > __get_priority_stripe: handle: busy hold: empty full_writes: 0
> > bypass_count: 0
> > __get_priority_stripe: handle: busy hold: empty full_writes: 0
> > bypass_count: 0
> > __get_priority_stripe: handle: busy hold: empty full_writes: 0
> > bypass_count: 0
> > __get_priority_stripe: handle: busy hold: empty full_writes: 0
> > bypass_count: 0
> > __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0 handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x10 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x11 read (null) write (null) written (null)
> > check 1: state 0x11 read (null) write (null) written (null)
> > check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328 for sector 273480328, rmw=2 rcw=1 handling stripe 65238568, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x10 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x18 read (null) write ffff88081a956b00 written (null)
> > check 1: state 0x11 read (null) write (null) written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=65238568 for sector 65238568, rmw=2 rcw=1 handling stripe 713868672, state=0x2041 cnt=1, pd_idx=4, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x10 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x18 read (null) write ffff88081f020100 written (null)
> > check 1: state 0x11 read (null) write (null) written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=713868672 for sector 713868672, rmw=2 rcw=1 handling stripe 729622496, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x10 read (null) write (null) written (null)
> > check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622496 for sector 729622496, rmw=2 rcw=1 handling stripe 729622504, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x10 read (null) write (null) written (null)
> > check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622504 for sector 729622504, rmw=2 rcw=1 handling stripe 245773680, state=0x2041 cnt=1, pd_idx=0, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x11 read (null) write (null) written (null)
> > check 1: state 0x18 read (null) write ffff88081cab7a00 written (null)
> > check 0: state 0x10 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=245773680 for sector 245773680, rmw=2 rcw=1 handling stripe 867965560, state=0x2041 cnt=1, pd_idx=1, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x11 read (null) write (null) written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x18 read (null) write ffff880802b2bf00 written (null)
> > check 1: state 0x10 read (null) write (null) written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=867965560 for sector 867965560, rmw=2 rcw=1 handling stripe 550162280, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> > check 5: state 0x11 read (null) write (null) written (null)
> > check 4: state 0x18 read (null) write ffff880802b08800 written (null)
> > check 3: state 0x0 read (null) write (null) written (null)
> > check 2: state 0x10 read (null) write (null) written (null)
> > check 1: state 0x11 read (null) write (null) written (null)
> > check 0: state 0x11 read (null) write (null) written (null)
> > locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> > force RCW max_degraded=1, recovery_cp=7036944 sh->sector=550162280 for
> > sector 550162280, rmw=2 rcw=1
> >
> >
> > Thanks,
> > Manibalan
> >
> >
> > -----Original Message-----
> > From: Manibalan P
> > Sent: Wednesday, December 17, 2014 12:11 PM
> > To: 'linux-raid'
> > Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> > Subject: RE: md_raid5 using 100% CPU and hang with status
> > resync=PENDING, if a drive is removed during initialization
> >
> > Dear Neil,
> >
> > The same Issue is reproducible in the latest upstream kernel also.
> >
> > Tested in "3.17.6" latest stable upstream kernel and find the same issue.
> >
> > [root@root ~]# modinfo raid456
> > filename: /lib/modules/3.17.6/kernel/drivers/md/raid456.ko
> > alias: raid6
> > alias: raid5
> > alias: md-level-6
> > alias: md-raid6
> > alias: md-personality-8
> > alias: md-level-4
> > alias: md-level-5
> > alias: md-raid4
> > alias: md-raid5
> > alias: md-personality-4
> > description: RAID4/5/6 (striping with parity) personality for MD
> > license: GPL
> > srcversion: 0EEF680023FDC7410F7989A
> > depends: async_raid6_recov,async_pq,async_tx,async_memcpy,async_xor
> > intree: Y
> > vermagic: 3.17.6 SMP mod_unload modversions
> > parm: devices_handle_discard_safely:Set to Y if all devices in each array reliably return zeroes on reads from discarded regions (bool)
> >
> > Thanks,
> > Manibalan.
> >
> > -----Original Message-----
> > From: Manibalan P
> > Sent: Wednesday, December 17, 2014 12:01 PM
> > To: 'linux-raid'
> > Cc: 'NeilBrown'
> > Subject: RE: md_raid5 using 100% CPU and hang with status
> > resync=PENDING, if a drive is removed during initialization
> >
> > Dear Neil,
> >
> > We are facing IO struck issue with raid5 in the following scenario.
> > (please see the attachment for the complete information) In RAID5
> > array, if a drive is removed while initialization and the same time if
> > IO is happening to that md. Then IO is getting struck, and md_raid5
> > thread is using 100 % of CPU. Also the md state showing as
> > resync=PENDING
> >
> > Kernel : Issue found in the following kernels RHEL 6.5
> > (2.6.32-431.el6.x86_64) CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
> >
> > Steps to Reproduce the issue:
> >
> > 1. Created a raid 5 md with 4 drives using the below mdadm command.
> > mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6
> > /dev/sdd6 /dev/sde6
> >
> > 2. Make the md writable
> > mdadm ???readwrite /dev/md0
> >
> > 3. Now md will start initialization
> >
> > 4. Run FIO Tool, the the below said configuration /usr/bin/fio
> > --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> > --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> > --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> > --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> > --blockalign=512
> >
> > 4. During MD initialzing, remove a drive(either using MDADM set
> > faulty/remove or remove manually)
> >
> > 5. Now the IO will struck, and cat /proc/mdstat shows states with
> > resync=PENDING
> > ----------------------------------------------------------------------
> > ----------------------- top - output show, md_raid5 using 100% cpu
> >
> > top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
> > ----------------------------------------------------------------------
> > -----------------------
> > dmesg - show the stack trace
> >
> > INFO: task fio:2715 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 000000000000000a 0 2715 2654 0x00000080
> > ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
> > ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff8140fa39>] ?
> > md_wakeup_thread+0x39/0x70 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffffa0308f66>] ?
> > make_request+0x306/0xc6c [raid456] [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81122283>] ?
> > mempool_alloc+0x63/0x140 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0 [<ffffffff811c8196>]
> > __blockdev_direct_IO_newtrunc+0x346/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2717 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000004 0 2717 2654 0x00000080
> > ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
> > ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2718 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000005 0 2718 2654 0x00000080
> > ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
> > ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2719 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000001 0 2719 2654 0x00000080
> > ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
> > ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2720 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000008 0 2720 2654 0x00000080
> > ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
> > ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2721 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000000 0 2721 2654 0x00000080
> > ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
> > ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2722 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000000 0 2722 2654 0x00000080
> > ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
> > ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2723 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000006 0 2723 2654 0x00000080
> > ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
> > ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2724 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 000000000000000b 0 2724 2654 0x00000080
> > ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
> > ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> > INFO: task fio:2725 blocked for more than 120 seconds.
> > Not tainted 2.6.32-431.el6.x86_64 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > fio D 0000000000000003 0 2725 2654 0x00000080
> > ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
> > ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> > ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8 Call Trace:
> > [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> > md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> > get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> > default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> > prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> > make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> > autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> > mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> > md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> > bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> > __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> > generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> > do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> > [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> > [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> > __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> > blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> > blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> > generic_file_direct_write+0xc2/0x190
> > [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> > [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> > blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> > blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> > aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> > aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> > [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> > system_call_fastpath+0x16/0x1b
> >
> > [root@root ~]# cat /proc/2690/stack
> > [<ffffffff810686da>] __cond_resched+0x2a/0x40 [<ffffffffa030361c>]
> > ops_run_io+0x2c/0x920 [raid456] [<ffffffffa03052cc>]
> > handle_stripe+0x9cc/0x2980 [raid456] [<ffffffffa03078a4>]
> > raid5d+0x624/0x850 [raid456] [<ffffffff81416f05>]
> > md_thread+0x115/0x150 [<ffffffff8109aef6>] kthread+0x96/0xa0
> > [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffffffffffff>]
> > 0xffffffffffffffff
> >
> > [root@root ~]# cat /proc/2690/stat
> > 2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0
> > 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17
> > 2 0 0 6855 0 0 [root@root ~]# cat /proc/2690/statm
> > 0 0 0 0 0 0 0
> > [root@root ~]# cat /proc/2690/stat
> > stat statm status
> > [root@root ~]# cat /proc/2690/status
> > Name: md0_raid5
> > State: R (running)
> > Tgid: 2690
> > Pid: 2690
> > PPid: 2
> > TracerPid: 0
> > Uid: 0 0 0 0
> > Gid: 0 0 0 0
> > Utrace: 0
> > FDSize: 64
> > Groups:
> > Threads: 1
> > SigQ: 2/128402
> > SigPnd: 0000000000000000
> > ShdPnd: 0000000000000000
> > SigBlk: 0000000000000000
> > SigIgn: fffffffffffffeff
> > SigCgt: 0000000000000100
> > CapInh: 0000000000000000
> > CapPrm: ffffffffffffffff
> > CapEff: fffffffffffffeff
> > CapBnd: ffffffffffffffff
> > Cpus_allowed: ffffff
> > Cpus_allowed_list: 0-23
> > Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
> > Mems_allowed_list: 0-1
> > voluntary_ctxt_switches: 5411612
> > nonvoluntary_ctxt_switches: 257032
> >
> >
> > Thanks,
> > Manibalan.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Xiao Ni @ 2015-02-03 8:10 UTC (permalink / raw)
To: NeilBrown; +Cc: Joe Lawrence, linux-raid, Bill Kuzeja
In-Reply-To: <20150202173601.1ab02927@notabene.brown>
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: "Joe Lawrence" <joe.lawrence@stratus.com>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> Sent: Monday, February 2, 2015 2:36:01 PM
> Subject: Re: RAID1 removing failed disk returns EBUSY
>
> On Thu, 29 Jan 2015 07:14:16 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: "Joe Lawrence" <joe.lawrence@stratus.com>,
> > > linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> > > Sent: Thursday, January 29, 2015 11:52:17 AM
> > > Subject: Re: RAID1 removing failed disk returns EBUSY
> > >
> > > On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:
> > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > > > > To: "Xiao Ni" <xni@redhat.com>
> > > > > Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill
> > > > > Kuzeja" <william.kuzeja@stratus.com>
> > > > > Sent: Friday, January 16, 2015 11:10:31 PM
> > > > > Subject: Re: RAID1 removing failed disk returns EBUSY
> > > > >
> > > > > On Fri, 16 Jan 2015 00:20:12 -0500
> > > > > Xiao Ni <xni@redhat.com> wrote:
> > > > > >
> > > > > > Hi Joe
> > > > > >
> > > > > > Thanks for reminding me. I didn't do that. Now it can remove
> > > > > > successfully after writing
> > > > > > "idle" to sync_action.
> > > > > >
> > > > > > I thought wrongly that the patch referenced in this mail is
> > > > > > fixed
> > > > > > for
> > > > > > the problem.
> > > > >
> > > > > So it sounds like even with 3.18 and a new mdadm, this bug still
> > > > > persists?
> > > > >
> > > > > -- Joe
> > > > >
> > > > > --
> > > >
> > > > Hi Joe
> > > >
> > > > I'm a little confused now. Does the patch
> > > > 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> > > > resolve the problem?
> > > >
> > > > My environment is:
> > > >
> > > > [root@dhcp-12-133 mdadm]# mdadm --version
> > > > mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014 (this is the newest
> > > > upstream)
> > > > [root@dhcp-12-133 mdadm]# uname -r
> > > > 3.18.2
> > > >
> > > >
> > > > My steps are:
> > > >
> > > > [root@dhcp-12-133 mdadm]# lsblk
> > > > sdb 8:16 0 931.5G 0 disk
> > > > └─sdb1 8:17 0 5G 0 part
> > > > sdc 8:32 0 186.3G 0 disk
> > > > sdd 8:48 0 931.5G 0 disk
> > > > └─sdd1 8:49 0 5G 0 part
> > > > [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1
> > > > /dev/sdd1
> > > > --assume-clean
> > > > mdadm: Note: this array has metadata at the start and
> > > > may not be suitable as a boot device. If you plan to
> > > > store '/boot' on this device please ensure that
> > > > your boot-loader understands md/v1.x metadata, or use
> > > > --metadata=0.90
> > > > mdadm: Defaulting to version 1.2 metadata
> > > > mdadm: array /dev/md0 started.
> > > >
> > > > Then I unplug the disk.
> > > >
> > > > [root@dhcp-12-133 mdadm]# lsblk
> > > > sdc 8:32 0 186.3G 0 disk
> > > > sdd 8:48 0 931.5G 0 disk
> > > > └─sdd1 8:49 0 5G 0 part
> > > > └─md0 9:0 0 5G 0 raid1
> > > > [root@dhcp-12-133 mdadm]# echo faulty >
> > > > /sys/block/md0/md/dev-sdb1/state
> > > > [root@dhcp-12-133 mdadm]# echo remove >
> > > > /sys/block/md0/md/dev-sdb1/state
> > > > -bash: echo: write error: Device or resource busy
> > > > [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action
> > > > [root@dhcp-12-133 mdadm]# echo remove >
> > > > /sys/block/md0/md/dev-sdb1/state
> > > >
> > >
> > > I cannot reproduce this - using linux 3.18.2. I'd be surprised if mdadm
> > > version affects things.
> >
> > Hi Neil
> >
> > I'm very curious, because it can reproduce in my machine 100%.
> >
> > >
> > > This error (Device or resoource busy) implies that rdev->raid_disk is >=
> > > 0
> > > (tested in state_store()).
> > >
> > > ->raid_disk is set to -1 by remove_and_add_spares() providing:
> > > 1/ it isn't Blocked (which is very unlikely)
> > > 2/ hot_remove_disk succeeds, which it will if nr_pending is zero, and
> > > 3/ nr_pending is zero.
> >
> > I remember I have tired to check those reasons. But it's really is the
> > reason 1
> > which is very unlikely.
> >
> > I add some code in the function array_state_show
> >
> > array_state_show(struct mddev *mddev, char *page) {
> > enum array_state st = inactive;
> > struct md_rdev *rdev;
> >
> > rdev_for_each_rcu(rdev, mddev) {
> > printk(KERN_ALERT "search for %s\n",
> > rdev->bdev->bd_disk->disk_name);
> > if (test_bit(Blocked, &rdev->flags))
> > printk(KERN_ALERT "rdev is Blocked\n");
> > else
> > printk(KERN_ALERT "rdev is not Blocked\n");
> > }
> >
> > When I echo 1 > /sys/block/sdc/device/delete, then I ran command:
> >
> > [root@dhcp-12-133 md]# cat /sys/block/md0/md/array_state
> > read-auto
> ^^^^^^^^^
>
> I think that is half the explanation.
> You must have the md_mod.start_ro parameter set to '1'.
>
>
> > [root@dhcp-12-133 md]# dmesg
> > [ 2679.559185] search for sdc
> > [ 2679.559189] rdev is Blocked
> > [ 2679.559190] search for sdb
> > [ 2679.559190] rdev is not Blocked
> >
> > So sdc is Blocked
>
> and that is the other half - thanks.
> (yes, I was wrong. Sometimes it is easier than being right, but still
> yields results).
>
> When a device fails, it is Blocked until the metadata is updated to record
> the failure. This ensures that no writes succeed without writing to that
> device, until we a certain that no read will try reading from that device,
> even after a crash/restart.
>
> Blocked is cleared after the metadata is written, but read-auto (and
> read-only) devices never write out their metadata. So blocked doesn't get
> cleared.
>
> When you "echo idle > .../sync_action" one of the side effects is to with
> from 'read-auto' to fully active. This allows the metadata to be written,
> Blocked to be cleared, and the device to be removed.
>
> If you
> echo none > /sys/block/md0/md/dev-sdc/slot
>
> first, then the remove will work.
>
> We could possibly fix it with something like the following, but I'm not sure
> I like it. There is no guarantee that I can see which would ensure the
> superblock got updated before the first write if the array switch to
> read/write.
>
> NeilBrown
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 9233c71138f1..b3d1e8e5e067 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -7528,7 +7528,7 @@ static int remove_and_add_spares(struct mddev *mddev,
> rdev_for_each(rdev, mddev)
> if ((this == NULL || rdev == this) &&
> rdev->raid_disk >= 0 &&
> - !test_bit(Blocked, &rdev->flags) &&
> + (!test_bit(Blocked, &rdev->flags) || mddev->ro) &&
> (test_bit(Faulty, &rdev->flags) ||
> ! test_bit(In_sync, &rdev->flags)) &&
> atomic_read(&rdev->nr_pending)==0) {
>
>
>
Hi Neil
I have tried the patch and the problem can be fixed by it. But I'm sorry that I can't
give more advices for better idea about this. I'm not familiar with the metadata part about
the md. I'll try to get more time to read the code about md.
Best Regards
Xiao
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Can't reshape raid0 to raid10
From: Xiao Ni @ 2015-02-03 8:13 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <484995116.1735572.1419909221275.JavaMail.zimbra@redhat.com>
----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: linux-raid@vger.kernel.org
> Sent: Tuesday, December 30, 2014 11:13:41 AM
> Subject: Can't reshape raid0 to raid10
>
> Hi Neil
>
> When I try to reshape a raid0 to raid10, it'll fail like this:
>
> [root@dhcp-12-133 mdadm-3.3.2]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 111.8G 0 disk
> ├─sda1 8:1 0 1000M 0 part /boot
> ├─sda2 8:2 0 29.3G 0 part /
> ├─sda3 8:3 0 512M 0 part [SWAP]
> ├─sda4 8:4 0 1K 0 part
> ├─sda5 8:5 0 102M 0 part
> └─sda6 8:6 0 10.1G 0 part
> └─VolGroup00-LogVol00 254:0 0 9.9G 0 lvm
> sdb 8:16 0 111.8G 0 disk
> ├─sdb1 8:17 0 2G 0 part
> └─sdb2 8:18 0 10G 0 part
> sdc 8:32 0 186.3G 0 disk
> ├─sdc1 8:33 0 2G 0 part
> └─sdc2 8:34 0 10G 0 part
> sdd 8:48 0 111.8G 0 disk
> ├─sdd1 8:49 0 2G 0 part
> └─sdd2 8:50 0 10G 0 part
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1
> /dev/sdc1 /dev/sdd1
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2
> /dev/sdc2 /dev/sdd2
> mdadm: level of /dev/md0 changed to raid10
> mdadm: add new device failed for /dev/sdb2 as 6: No space left on device
>
>
> But if I reshape the raid0 to raid5, reshape raid5 to raid0, then reshape
> raid0 to raid10 use
> the same command it'll succeed.
>
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1
> /dev/sdc1 /dev/sdd1
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l5
> [root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
> md0 : active raid5 sdd1[2] sdc1[1] sdb1[0]
> 6285312 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
>
> unused devices: <none>
>
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l0
> [root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
> md0 : active raid0 sdd1[2] sdc1[1] sdb1[0]
> 6285312 blocks super 1.2 512k chunks
>
> unused devices: <none>
> [root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2
> /dev/sdc2 /dev/sdd2
> mdadm: level of /dev/md0 changed to raid10
> mdadm: added /dev/sdb2
> mdadm: added /dev/sdc2
> mdadm: added /dev/sdd2
>
> So I guess it's the problem add the disk to raid10 after the reshaping.
> In the function
> super_1_validate, it'll set the mddev->dev_sectors using the superblock read
> from disks.
> If it's raid0, the le64_to_cpu(sb-size) is 0. So when add disk to raid10
> bind_rdev_to_array
> return -ENOSPC.
>
> When create raid0, it doesn't write give the value to s->size. So the
> sb-size is 0.
> I modify the code about Create.c. I'm not sure whether it's right to do so.
> But it can resolve
> the problem.
>
> diff --git a/Create.c b/Create.c
> index 330c5b4..f3135c5 100644
> --- a/Create.c
> +++ b/Create.c
> @@ -489,7 +489,7 @@ int Create(struct supertype *st, char *mddev,
> pr_err("no size and no drives given - aborting
> create.\n");
> return 1;
> }
> - if (s->level > 0 || s->level == LEVEL_MULTIPATH
> + if (s->level >= 0 || s->level == LEVEL_MULTIPATH
> || s->level == LEVEL_FAULTY
> || st->ss->external ) {
> /* size is meaningful */
>
Hi Neil
Any update for this?
Best Regards
Xiao
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Aw: Re: Array died during grow; now resync stopped
From: "Jörg Habenicht" @ 2015-02-03 10:35 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150203064056.5dbba8c5@notabene.brown>
Hello Neil,
thank you for caring.
(And sorry for the malformed structure, I have to use webmail.. )
On Mon, 2 Feb 2015 09:41:02 +0000 (UTC) Jörg Habenicht <j.habenicht@gmx.de>
wrote:
> Hi all,
>
> I had a server crash during an array grow.
> Commandline was "mdadm --grow /dev/md0 --raid-devices=6 --chunk=1M"
>
>
>
> Could this be caused by a software lock?
>Some sort of software problem I suspect.
>What does
>cat /proc/1671/stack
>cat /proc/1672/stack
>show?
$ cat /proc/1671/stack
cat: /proc/1671/stack: No such file or directory
Huch?
$ ls /proc/1671
ls: cannot read symbolic link /proc/1671/exe: No such file or directory
attr comm fdinfo mounts oom_score stat
autogroup coredump_filter io mountstats oom_score_adj statm
auxv cwd limits net pagemap status
cgroup environ maps ns personality syscall
clear_refs exe mem numa_maps root task
cmdline fd mountinfo oom_adj smaps wchan
$ id
uid=0(root) gid=0(root) groups=0(root), ...
$ cat /proc/1672/stack
cat: /proc/1672/stack: No such file or directory
>Alternatively,
>echo w > /proc/sysrq-trigger
>and see what appears in 'dmesg'.
No good:
[99166.625796] SysRq : Show Blocked State
[99166.625829] task PC stack pid father
[99166.625845] md0_reshape D ffff88006cb81e08 0 1671 2 0x00000000
[99166.625854] ffff88006a17fb30 0000000000000046 000000000000a000 ffff88006cc9b7e0
[99166.625861] ffff88006a17ffd8 ffff88006cc9b7e0 ffff88006fc11830 ffff88006fc11830
[99166.625866] 0000000000000001 ffffffff81068670 ffff88006ca56848 ffff88006fc11830
[99166.625871] Call Trace:
[99166.625884] [<ffffffff81068670>] ? __dequeue_entity+0x40/0x50
[99166.625891] [<ffffffff8106b966>] ? pick_next_task_fair+0x56/0x1b0
[99166.625898] [<ffffffff813f4a50>] ? __schedule+0x2a0/0x820
[99166.625905] [<ffffffff8106273d>] ? ttwu_do_wakeup+0xd/0x80
[99166.625914] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
[99166.625922] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
[99166.625929] [<ffffffffa027e83a>] ? reshape_request+0x21a/0x860 [raid456]
[99166.625935] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
[99166.625942] [<ffffffffa02744f6>] ? sync_request+0x236/0x380 [raid456]
[99166.625955] [<ffffffffa01557ad>] ? md_do_sync+0x82d/0xd00 [md_mod]
[99166.625961] [<ffffffff810684b4>] ? update_curr+0x64/0xe0
[99166.625971] [<ffffffffa0152197>] ? md_thread+0xf7/0x110 [md_mod]
[99166.625977] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
[99166.625985] [<ffffffffa01520a0>] ? md_register_thread+0xf0/0xf0 [md_mod]
[99166.625991] [<ffffffff81059de8>] ? kthread+0xb8/0xd0
[99166.625997] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
[99166.626003] [<ffffffff813f837c>] ? ret_from_fork+0x7c/0xb0
[99166.626008] [<ffffffff81059d30>] ? kthread_create_on_node+0x180/0x180
[99166.626012] udevd D ffff88006cb81e08 0 1672 1289 0x00000004
[99166.626017] ffff88006a1819e8 0000000000000086 000000000000a000 ffff88006c4967a0
[99166.626022] ffff88006a181fd8 ffff88006c4967a0 0000000000000000 0000000000000000
[99166.626027] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[99166.626032] Call Trace:
[99166.626039] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
[99166.626044] [<ffffffff810c24ed>] ? zone_statistics+0x9d/0xa0
[99166.626050] [<ffffffff810b13e7>] ? get_page_from_freelist+0x507/0x850
[99166.626057] [<ffffffffa027b4c5>] ? get_active_stripe+0x185/0x5c0 [raid456]
[99166.626063] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
[99166.626069] [<ffffffffa027f627>] ? make_request+0x7a7/0xa00 [raid456]
[99166.626075] [<ffffffff81080afd>] ? ktime_get_ts+0x3d/0xd0
[99166.626080] [<ffffffff81072110>] ? __wake_up_sync+0x10/0x10
[99166.626089] [<ffffffffa014ea12>] ? md_make_request+0xd2/0x210 [md_mod]
[99166.626096] [<ffffffff811e649d>] ? generic_make_request_checks+0x23d/0x270
[99166.626100] [<ffffffff810acc68>] ? mempool_alloc+0x58/0x140
[99166.626106] [<ffffffff811e7238>] ? generic_make_request+0xa8/0xf0
[99166.626111] [<ffffffff811e72e7>] ? submit_bio+0x67/0x130
[99166.626117] [<ffffffff8112a638>] ? bio_alloc_bioset+0x1b8/0x2a0
[99166.626123] [<ffffffff81126a57>] ? _submit_bh+0x127/0x200
[99166.626129] [<ffffffff8112815d>] ? block_read_full_page+0x1fd/0x290
[99166.626133] [<ffffffff8112b680>] ? I_BDEV+0x10/0x10
[99166.626140] [<ffffffff810aad2b>] ? add_to_page_cache_locked+0x6b/0xc0
[99166.626146] [<ffffffff810b5520>] ? __do_page_cache_readahead+0x1b0/0x220
[99166.626152] [<ffffffff810b5812>] ? force_page_cache_readahead+0x62/0xa0
[99166.626159] [<ffffffff810ac936>] ? generic_file_aio_read+0x4b6/0x6c0
[99166.626166] [<ffffffff810f9f87>] ? do_sync_read+0x57/0x90
[99166.626172] [<ffffffff810fa571>] ? vfs_read+0xa1/0x180
[99166.626178] [<ffffffff810fb0ab>] ? SyS_read+0x4b/0xc0
[99166.626183] [<ffffffff813f7f72>] ? page_fault+0x22/0x30
[99166.626190] [<ffffffff813f8422>] ? system_call_fastpath+0x16/0x1b
>
> The system got 2G RAM and 2G swap. Is this sufficient to complete?
>Memory shouldn't be a problem.
>However it wouldn't hurt to see what value is in
>/sys/block/md0/md/stripe_cache_size
>and double it.
$ cat /sys/block/md0/md/stripe_cache_size
256
I did not change it due to the crash in md_reshape
>If all else fails a reboot should be safe and will probably start the reshape
>properly. md is very careful about surviving reboots.
I already did reboot twice before I wrote to the list. Same result.
>NeilBrown
cu,
Joerg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: update mdadm version to 3.3.2 in Centos 6.4
From: Jes Sorensen @ 2015-02-03 13:19 UTC (permalink / raw)
To: alpha lin; +Cc: linux-raid
In-Reply-To: <CAKtQoftv3uf-Y1va7QV=qFiJZO0CdSJ7QtfHj7P1Mg5oyj0fRw@mail.gmail.com>
alpha lin <lin.alpha@gmail.com> writes:
> Hi All,
>
> I would like to know the possibility of update the mdadm package
> from 3.2.5 to 3.3.2 in Redhat Centos 6.4 x64 system.
> The problem I have is I found that I can create a raid volume
> without any problem. But when reboot the system. The md volume become
> inactive.
> Below is my step:
> 1. Check out mdadm package from the github.
> 2. unzip the mdadm package and execute "make" and "make install"
> 3. reboot.
> 4. after system reboot, run mdadm to create a imsm RAID Volume.
> [root@localhost rules.d]# mdadm --version
> mdadm - v3.3.2 - 21st August 2014
Any reason why you haven't upgraded to centos 6.6? 6.4 is ancient, and
6.6 will at least come with mdadm-3.3.
Jes
^ permalink raw reply
* Off-Topic Write cache disabling?
From: Weedy @ 2015-02-03 17:45 UTC (permalink / raw)
To: linux-raid
Is there a kernel option or sysfs toggle that disables write caching?
Or forces the kernel to commit everything constantly.
------
I don't really want to join another ML, especially a higher traffic
one just to ask this when it only bugs me sometimes. But I'll shut up
if this is unwanted.
------
I use a similar kernel config with respect to selected options on all
my systems but this only effect my laptop.
On any of my personal system or system I have remote access too
nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
the system isn't working hard).
On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
go up to 10ish if I untar something but then almost immediately go
back to 0. If I didn't know better I would swear dirty_*_centisecs or
something was set to a near instant commit interval but I haven't
found evidence of that. The hard drive light blinks almost constantly
once a second, even if I'm at a X login screen.
As I said this doesn't bug me most of the time but if I let my FF
session get too large or start multiple VMs anything that might make
me swap a little, the machine pretty much dies from IOWAIT. Which I'm
guessing is because it's trying to flush (syncfs?) imediately and
constantly.
You guys spend all day in the IO subsystem, any idea where I can keep
looking? It has persisted across reboots and kernel updates.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox