Re: RAID1 removing failed disk returns EBUSY

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 5752 bytes --]

On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni <xni@redhat.com> wrote:

> 
> 
> ----- Original Message -----
> > From: "Joe Lawrence" <joe.lawrence@stratus.com>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: "NeilBrown" <neilb@suse.de>, linux-raid@vger.kernel.org, "Bill Kuzeja" <william.kuzeja@stratus.com>
> > Sent: Friday, January 16, 2015 11:10:31 PM
> > Subject: Re: RAID1 removing failed disk returns EBUSY
> > 
> > On Fri, 16 Jan 2015 00:20:12 -0500
> > Xiao Ni <xni@redhat.com> wrote:
> > > 
> > > Hi Joe
> > > 
> > >    Thanks for reminding me. I didn't do that. Now it can remove
> > >    successfully after writing
> > > "idle" to sync_action.
> > > 
> > >    I thought wrongly that the patch referenced in this mail is fixed for
> > >    the problem.
> > 
> > So it sounds like even with 3.18 and a new mdadm, this bug still
> > persists?
> > 
> > -- Joe
> > 
> > --
> 
> Hi Joe
> 
>    I'm a little confused now. Does the patch 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
> resolve the problem?
> 
>    My environment is:
> 
> [root@dhcp-12-133 mdadm]# mdadm --version
> mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014  (this is the newest upstream)
> [root@dhcp-12-133 mdadm]# uname -r
> 3.18.2
> 
> 
>    My steps are:
> 
> [root@dhcp-12-133 mdadm]# lsblk 
> sdb                       8:16   0 931.5G  0 disk 
> └─sdb1                    8:17   0     5G  0 part 
> sdc                       8:32   0 186.3G  0 disk 
> sdd                       8:48   0 931.5G  0 disk 
> └─sdd1                    8:49   0     5G  0 part 
> [root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1 /dev/sdd1 --assume-clean
> mdadm: Note: this array has metadata at the start and
>     may not be suitable as a boot device.  If you plan to
>     store '/boot' on this device please ensure that
>     your boot-loader understands md/v1.x metadata, or use
>     --metadata=0.90
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> 
>    Then I unplug the disk.
> 
> [root@dhcp-12-133 mdadm]# lsblk 
> sdc                       8:32   0 186.3G  0 disk  
> sdd                       8:48   0 931.5G  0 disk  
> └─sdd1                    8:49   0     5G  0 part  
>   └─md0                   9:0    0     5G  0 raid1 
> [root@dhcp-12-133 mdadm]# echo faulty > /sys/block/md0/md/dev-sdb1/state 
> [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state 
> -bash: echo: write error: Device or resource busy
> [root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action 
> [root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state 
> 

I cannot reproduce this - using linux 3.18.2.  I'd be surprised if mdadm
version affects things.

This error (Device or resoource busy) implies that rdev->raid_disk is >= 0
(tested in state_store()).

->raid_disk is set to -1 by remove_and_add_spares() providing:
  1/ it isn't Blocked (which is very unlikely)
  2/ hot_remove_disk succeeds, which it will if nr_pending is zero, and
  3/ nr_pending is zero.

So it seems most likely that either:
 1/ nr_pending is non-zero, or
 2/ remove_and_add_spares() didn't run.

nr_pending can only get set if IO is generated, and your sequence of steps
don't show any IO.  It is possible that something else (e.g. started by udev)
triggered some IO.  How long that IO can stay pending might depend on exactly
how you unplug the device.
In my tests I used
   echo 1 > /sys/block/sdXX/../../delete
which may have a different effect to what you do.

However the fact that writing 'idle' to sync_action releases the device seems
to suggest the nr_pending has dropped to zero.  So either
  - remove_and_add_spares didn't run, or
  - remove_and_add_spares ran during a small window when nr_pending was
    elevated, and then didn't run again when nr_pending was reduced to zero.

Ahh.... that rings bells....

I have the following patch in the SLES kernel which I have applied to
mainline yet (and given how old it is, that is really slack of me).

Can you apply the following and see if the symptom goes away please?

Thanks,
NeilBrown

From: Hannes Reinecke <hare@suse.de>
Date: Thu, 26 Jul 2012 11:12:18 +0200
Subject: [PATCH] md: wakeup thread upon rdev_dec_pending()

After each call to rdev_dec_pending() we should wakeup the
md thread if the device is found to be faulty.
Otherwise we'll incur heavy delays on failing devices.

Signed-off-by: Neil Brown <nfbrown@suse.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>

diff --git a/drivers/md/md.h b/drivers/md/md.h
index 03cec5bdcaae..4cc2f59b2994 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -439,13 +439,6 @@ struct mddev {
 	void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev);
 };
 
-static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
-{
-	int faulty = test_bit(Faulty, &rdev->flags);
-	if (atomic_dec_and_test(&rdev->nr_pending) && faulty)
-		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
-}
-
 static inline void md_sync_acct(struct block_device *bdev, unsigned long nr_sectors)
 {
 	atomic_add(nr_sectors, &bdev->bd_contains->bd_disk->sync_io);
@@ -624,4 +617,14 @@ static inline int mddev_check_plugged(struct mddev *mddev)
 	return !!blk_check_plugged(md_unplug, mddev,
 				   sizeof(struct blk_plug_cb));
 }
+
+static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
+{
+	int faulty = test_bit(Faulty, &rdev->flags);
+	if (atomic_dec_and_test(&rdev->nr_pending) && faulty) {
+		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+		md_wakeup_thread(mddev->thread);
+	}
+}
+
 #endif /* _MD_MD_H */


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]