md multipath restart problem

md multipath restart problem

[John Madden]

I'm connecting to an IBM 6800 SAN via two QLogic 2340's to a single LUN and I've
been wrestling with multipathing to them for about a week now.  "Both" LUNs
are visible as two scsi devices, /dev/sda and /dev/sdb.

So I slap a Linux (0x83) partition on the device (and it is visible through
the "second" lun) and I create the md device:

# ./mdadm --create /dev/md0 --force --level=multipath --raid-disks=2 \
/.dev/scsi/host0/bus0/target0/lun0/part1 \
/.dev/scsi/host1/bus0/target0/lun0/part1
mdadm: array /dev/md0 started.

That can be formatted, mounted, written to, etc., all reliably.  Info:

# dmesg
multipath: array md0 active with 2 out of 2 IO paths

# ./mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Fri Jun 10 16:07:16 2005
     Raid Level : multipath
     Array Size : 314568640 (299.100 GiB 322.12 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Jun 10 16:07:16 2005
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : a917893d:4570204d:bcf77929:a2564bef
         Events : 0.2

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sda1
       1       8       65        1      active sync   /dev/sdb1

The config file, switched to devfs names since sda and sdb could move
around:

DEVICE /.dev/scsi/host0/bus0/target0/lun0/part1
/.dev/scsi/host0/bus0/target0/lun0/part1
ARRAY /dev/md0 level=multipath num-devices=2
UUID=a917893d:4570204d:bcf77929:a2564bef
devices=/.dev/scsi/host0/bus0/target0/lun0/part1,/.dev/scsi/host1/bus0/target0/l

And then we reboot.

When the device is brought up the second time (and I don't think a reboot is
even needed), only one of the paths to the storage will be added, the other
failing with "Device or resource busy."

dmesg after boot:
md: md0 stopped.
md: bind<sdb1>
md: export_rdev(sda1)
multipath: array md0 active with 1 out of 2 IO paths

When I stop it manually, then re-activate it:

# ./mdadm -A /dev/md0
mdadm: device 1 in /dev/md0 has wrong state in superblock, but
/.dev/scsi/host1/bus0/target0/lun0/part1 seems ok
mdadm: failed to add /.dev/scsi/host0/bus0/target0/lun0/part1 to /dev/md0:
Device or resource busy
mdadm: /dev/md0 has been started with 1 drive (out of 2).

So I'd guess that something's going wrong there but I have little idea what.
The bad part is that while troubleshooting this, I managed to make any
number of things go wrong, so I'm not even sure anymore if this is the root
of my problem, but it's where I am right now.  Any suggestions?

Thanks,
  John



-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: md multipath restart problem

[John Madden]

> When I stop it manually, then re-activate it:
>
> # ./mdadm -A /dev/md0
> mdadm: device 1 in /dev/md0 has wrong state in superblock, but
> /.dev/scsi/host1/bus0/target0/lun0/part1 seems ok
> mdadm: failed to add /.dev/scsi/host0/bus0/target0/lun0/part1 to /dev/md0:
> Device or resource busy
> mdadm: /dev/md0 has been started with 1 drive (out of 2).

FWIW, I found after more playing with it that rather than using -A, if I use
--build, I'm able to bring the raidset back up with both IO paths active.  Is
there a bug in -A with it not unlocking the devices a it traverses them to bring
them into the array?

Thanks,
  John



-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

RE: md multipath restart problem

[Nagpure, Dinesh]

John,

Have you found a solution to the problem? If not then which kernel version are you using. The problem you are talking about seems like a bug in md that got fixed recently. 

-dinesh 

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org]On Behalf Of John Madden
Sent: Sunday, June 12, 2005 12:31 PM
To: linux-raid@vger.kernel.org
Subject: Re: md multipath restart problem


> When I stop it manually, then re-activate it:
>
> # ./mdadm -A /dev/md0
> mdadm: device 1 in /dev/md0 has wrong state in superblock, but
> /.dev/scsi/host1/bus0/target0/lun0/part1 seems ok
> mdadm: failed to add /.dev/scsi/host0/bus0/target0/lun0/part1 to /dev/md0:
> Device or resource busy
> mdadm: /dev/md0 has been started with 1 drive (out of 2).

FWIW, I found after more playing with it that rather than using -A, if I use
--build, I'm able to bring the raidset back up with both IO paths active.  Is
there a bug in -A with it not unlocking the devices a it traverses them to bring
them into the array?

Thanks,
  John



-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: md multipath restart problem

[John Madden]

> Have you found a solution to the problem? If not then which kernel version are you
> using. The problem you are talking about seems like a bug in md that got fixed
> recently.

And to think I was just about to give up and unsubscribe... :)

I did actually manage to fix the issue by just not doing things correctly -- if I
--build the array instead of starting it with -A, it works.  (See my previous
posts.) I'm running kernel 2.6.11.8 so far, what version has a/the fix?

Thanks,
  John




-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

RE: md multipath restart problem

[Nagpure, Dinesh]

John,

I encountered the bug using RHEL 4, fixed it and then found it already fixed in  linux-2.6.12-rc2.
The bug is in the function super_90_load() in md.c where it checks sb->level against MULTIPATH which should be LEVEL_MULTIPATH as below:

        rdev->preferred_minor = sb->md_minor;
        rdev->data_offset = 0;

        if (sb->level == LEVEL_MULTIPATH) /* THIS IS THE FIX - dinesh*/
                rdev->desc_nr = -1;
        else
                rdev->desc_nr = sb->this_disk.number;

        if (refdev == 0)
                ret = 1;

BTW what version of mdadm are you using? just curious, can you fail and then re-insert a path to a multipath array and then dump the superblock information using mdadm to see both the paths active again? I think paths get activated but the superblock is not updated so the information mdadm dumps is sort of incorrect.

-dinesh



 

-----Original Message-----
From: John Madden [mailto:jmadden@ivytech.edu]
Sent: Monday, June 20, 2005 10:42 AM
To: Nagpure, Dinesh
Cc: linux-raid@vger.kernel.org; Nagpure, Dinesh
Subject: RE: md multipath restart problem


> Have you found a solution to the problem? If not then which kernel version are you
> using. The problem you are talking about seems like a bug in md that got fixed
> recently.

And to think I was just about to give up and unsubscribe... :)

I did actually manage to fix the issue by just not doing things correctly -- if I
--build the array instead of starting it with -A, it works.  (See my previous
posts.) I'm running kernel 2.6.11.8 so far, what version has a/the fix?

Thanks,
  John




-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

RE: md multipath restart problem

[John Madden]

> I encountered the bug using RHEL 4, fixed it and then found it already fixed in
> linux-2.6.12-rc2.

Interesting - I'll try upgrading.

> BTW what version of mdadm are you using? just curious, can you fail and then
> re-insert a path to a multipath array and then dump the superblock information
> using mdadm to see both the paths active again? I think paths get activated but
> the superblock is not updated so the information mdadm dumps is sort of incorrect.

I had tried the stock Ubuntu version (1.9.0) and then switched to 1.11.  Failover
works, failback doesn't - the device has to be reinserted manually.  Given the
email notification that mdadm sends out though, I can live with that.

John





-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu