Spare fails to transfer between RAID groups

Spare fails to transfer between RAID groups

[Garth Snyder]

Hi raiders --

I'm having a problem getting a spare partition moved to the right RAID group
after a failure. I see some past discussion of issues related to spare
groups in the list archive, but no exact fit for the behavior I'm seeing.
Here's the outline:

1) Two RAID arrays (/dev/md_d0 and /dev/md_d3) are assigned to one
spare-group in /etc/mdadm/mdadm.conf. The spare partition is initially
assigned to /dev/md_d0.

2) I use mdadm --fail to fail a disk that belongs to the nondefault RAID
group, /dev/md_d3. (This is a testing scenario, not an actual failure.)

3) mdadm --monitor is running and working correctly, as evidenced by email
noting the Fail event on /dev/md_d3.

4) The spare partition is successfully removed from its original RAID group,
/dev/md_d0.

5) The spare partition is never added to /dev/md_d3, nor is it returned to
its original group, /dev/md_d0.

6) Two error messages are submitted via syslog to /var/log/kern.log in quick
succession, both with the same message: "HOT_ADD may only be used with
version-0 superblocks". The first message is tagged "md_d3" and the second
"md_d0".

7) There are no other syslog messages or developments. /dev/md_d3 remains
degraded.

8) The system is Ubuntu Jaunty Jackalope, kernel  2.6.28-11-generic, mdadm
--version = v2.6.7.1 - 15th October 2008.

From the error messages, I gather that the issue occurs when adding the
spare to the new array. After that proves impossible, mdadm tries to return
the spare to its original array, but that fails too. The weird things is
that "mdadm /dev/md_d3 --add /dev/sdc1" works just fine when run by hand, as
does the analogous command to return /dev/sdc1 to /dev/md_d0.

All superblocks involved are version 1.2. This is a new setup, and every
mdadm --create command has included --metadata=1.2. There should be no
legacy or old RAID superblocks around.

Any help or pointers would be appreciated. Is there some limitation I'm not
aware of that's preventing the hot add?

Thanks,
Garth

Some details:

/etc/mdadm/mdadm.conf:

DEVICE /dev/sd[abcehi][1234] /dev/sd[dgf][12356] /dev/hfspare/hfspare1
/dev/hfspare/hfspare2
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR garth@grsweb.us
ARRAY /dev/md_d0 level=raid10 metadata=1.2 num-devices=3
UUID=c766cb59:4e5fc5f6:509aac41:fa5c9c45 name=nutrient:evolution1
spare-group=evonent
ARRAY /dev/md_d3 level=raid5 metadata=1.2 num-devices=3
UUID=d4c020c9:4e5fc5f6:509aac41:fa5c9c45 name=nutrient:entgegen
spare-group=evonent
[other RAID arrays omitted]

/proc/mdstat, right after /dev/sde1 is failed:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
[raid10] 

md0 : active raid10 sdg2[1] sdf2[2] sdd2[0] sdc1[3](S)
      468756288 blocks super 1.2 64K chunks 2 far-copies [3/3] [UUU]
      
md4 : active raid5 sdg6[1] sdf6[2] sdd6[0] sda4[4]
      516216192 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4]
[UUUU]
      
md1 : active raid5 sdf3[2] sdg3[1] sdd3[0] sde2[5] sda2[3]
      312495872 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5]
[UUUUU]
      
md2 : active raid5 sdf5[2] sdg5[1] sdd5[0] sda3[4]
      1230465984 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4]
[UUUU]
      
md3 : active raid5 sde1[1](F) sda1[0] sdh1[3]
      624992384 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [U_U]

Re: Spare fails to transfer between RAID groups

[Neil Brown]

On Sunday May 10, garth@grsweb.us wrote:
> Hi raiders --
> 
> I'm having a problem getting a spare partition moved to the right RAID group
> after a failure. I see some past discussion of issues related to spare
> groups in the list archive, but no exact fit for the behavior I'm seeing.
> Here's the outline:
> 
> 1) Two RAID arrays (/dev/md_d0 and /dev/md_d3) are assigned to one
> spare-group in /etc/mdadm/mdadm.conf. The spare partition is initially
> assigned to /dev/md_d0.
> 
...
> 6) Two error messages are submitted via syslog to /var/log/kern.log in quick
> succession, both with the same message: "HOT_ADD may only be used with
> version-0 superblocks". The first message is tagged "md_d3" and the second
> "md_d0".
> 
....
> 
> All superblocks involved are version 1.2. This is a new setup, and every
> mdadm --create command has included --metadata=1.2. There should be no
> legacy or old RAID superblocks around.
> 

This would be because no one has tried the spare-group feature with
1.x metadata yet.  You are the first (or the first to report when it
failed at least).

The following patch should make it work for you.  This patch is against
2.6.9, though it should apply to any reasonably recent 2.x mdadm.
(it won't apply to 3.0).

Please let me know if it works for you.

Thanks for the report,
NeilBrown

diff --git a/Manage.c b/Manage.c
index 33a0bc7..fa4bb60 100644
--- a/Manage.c
+++ b/Manage.c
@@ -292,11 +292,15 @@ int Manage_subdevs(char *devname, int fd,
 		} else {
 			j = 0;
 
-			if (stat(dv->devname, &stb)) {
+			tfd = dev_open(dv->devname, O_RDONLY);
+			if (tfd < 0 || fstat(tfd, &stb) != 0) {
 				fprintf(stderr, Name ": cannot find %s: %s\n",
 					dv->devname, strerror(errno));
+				if (tfd >= 0)
+					close(tfd);
 				return 1;
 			}
+			close(tfd);
 			if ((stb.st_mode & S_IFMT) != S_IFBLK) {
 				fprintf(stderr, Name ": %s is not a "
 					"block device.\n",
@@ -313,7 +317,7 @@ int Manage_subdevs(char *devname, int fd,
 			/* add the device */
 
 			/* Make sure it isn't in use (in 2.6 or later) */
-			tfd = open(dv->devname, O_RDONLY|O_EXCL);
+			tfd = dev_open(dv->devname, O_RDONLY|O_EXCL);
 			if (tfd < 0) {
 				fprintf(stderr, Name ": Cannot open %s: %s\n",
 					dv->devname, strerror(errno));
diff --git a/Monitor.c b/Monitor.c
index 3825600..e0a9d2a 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -470,16 +470,25 @@ int Monitor(mddev_dev_t devlist,
 							}
 						}
 						if (dev > 0) {
-							if (ioctl(fd2, HOT_REMOVE_DISK,
-								  (unsigned long)dev) == 0) {
-								if (ioctl(fd1, HOT_ADD_DISK,
-									  (unsigned long)dev) == 0) {
+							struct mddev_dev_s devlist;
+							char devname[20];
+							devlist.next = NULL;
+							devlist.used = 0;
+							devlist.re_add = 0;
+							devlist.writemostly = 0;
+							devlist.devname = devname;
+							sprintf(devname, "%d:%d", major(dev), minor(dev));
+
+							devlist.disposition = 'r';
+							if (Manage_subdevs(st2->devname, fd2, &devlist, -1) == 0) {
+								devlist.disposition = 'a';
+								if (Manage_subdevs(st->devname, fd1, &devlist, -1) == 0) {
 									alert("MoveSpare", st->devname, st2->devname, mailaddr, mailfrom, alert_cmd, dosyslog);
 									close(fd1);
 									close(fd2);
 									break;
 								}
-								else ioctl(fd2, HOT_ADD_DISK, (unsigned long) dev);
+								else Manage_subdevs(st2->devname, fd2, &devlist, -1);
 							}
 						}
 						close(fd1);
diff --git a/super0.c b/super0.c
index 90fdf23..ebba534 100644
--- a/super0.c
+++ b/super0.c
@@ -672,7 +672,7 @@ static int write_init_super0(struct supertype *st,
 			     mdu_disk_info_t *dinfo, char *devname)
 {
 	mdp_super_t *sb = st->sb;
-	int fd = open(devname, O_RDWR|O_EXCL);
+	int fd = dev_open(devname, O_RDWR|O_EXCL);
 	int rv;
 
 	if (fd < 0) {
diff --git a/super1.c b/super1.c
index 037c5eb..b86a28b 100644
--- a/super1.c
+++ b/super1.c
@@ -887,7 +887,7 @@ static int write_init_super1(struct supertype *st,
 {
 	struct mdp_superblock_1 *sb = st->sb;
 	struct supertype refst;
-	int fd = open(devname, O_RDWR | O_EXCL);
+	int fd = dev_open(devname, O_RDWR | O_EXCL);
 	int rfd;
 	int rv;
 	int bm_space;

RE: Spare fails to transfer between RAID groups

[Garth Snyder]

On Sunday May 10, I (garth@grsweb.us) wrote:

> I'm having a problem getting a spare partition moved to the right RAID
group
> after a failure...All superblocks involved are version 1.2.

Neil Brown replied:

> This would be because no one has tried the spare-group feature with
> 1.x metadata yet...The following patch should make it work...This patch
> is against 2.6.9, though it should apply to any reasonably recent 2.x
mdadm.

Thanks for the response, Neil. The patch works great; now I get Fail,
MoveSpare, and Rebuild events in sequence with no errors. /proc/mdstat shows
that the appropriate spare was moved into the failed array and that the
array is being rebuilt onto it.

I patched the 2.6.7.1 tree, which was Jaunty's default for "apt-get source
mdadm" (and the same version included in the binary package). Manage.c and
Monitor.c already matched the "after" state of your patch, so the only code
that actually changed was the one-liners in super0.c and super1.c.

Thanks for the lightning-fast patch, and please let me know if there's
anything more I can do to help validate the fix.

Cheers,
Garth

RE: Spare fails to transfer between RAID groups

[Neil Brown]

On Tuesday May 12, garth@grsweb.us wrote:
> On Sunday May 10, I (garth@grsweb.us) wrote:
> 
> > I'm having a problem getting a spare partition moved to the right RAID
> group
> > after a failure...All superblocks involved are version 1.2.
> 
> Neil Brown replied:
> 
> > This would be because no one has tried the spare-group feature with
> > 1.x metadata yet...The following patch should make it work...This patch
> > is against 2.6.9, though it should apply to any reasonably recent 2.x
> mdadm.
> 
> Thanks for the response, Neil. The patch works great; now I get Fail,
> MoveSpare, and Rebuild events in sequence with no errors. /proc/mdstat shows
> that the appropriate spare was moved into the failed array and that the
> array is being rebuilt onto it.

Excellent.  Thanks for testing and reporting.

> 
> I patched the 2.6.7.1 tree, which was Jaunty's default for "apt-get source
> mdadm" (and the same version included in the binary package). Manage.c and
> Monitor.c already matched the "after" state of your patch, so the only code
> that actually changed was the one-liners in super0.c and super1.c.

That is impossible.  I had only just written that patch, so there is
no way the 'after' bit could possibly be in Jaunty.  Maybe something
got confused when you ran patch.... doesn't really matter - it works,
and that is the important thing.

NeilBrown

RE: Spare fails to transfer between RAID groups

[Garth Snyder]

I wrote: 

> I patched the 2.6.7.1 tree, which was Jaunty's default for "apt-get source
> mdadm" (and the same version included in the binary package). Manage.c and
> Monitor.c already matched the "after" state of your patch, so the only
code
> that actually changed was the one-liners in super0.c and super1.c.

Neil Brown replied:

> That is impossible. I had only just written that patch, so there is
> no way the 'after' bit could possibly be in Jaunty.

Sorry, my mistake - I received error messages when applying the Manage.c and
Monitor.c portions, so I thought that the patches hadn't been applied when I
inspected the files. I am in fact running the complete set of patches to all
four files and that's what's been working for me.

Garth