2 HBA's and Multipath.

2 HBA's and Multipath.

[SoulBlazer]

Question,

I have 2 Qlogic Fiberchannel HBA's (2100F's) connected directly to both FCAL 
ports on my Sun A5200 22 Disk disk array.  If I enable multipath, do I get 
twice the throughput between my linux box and the array ? Or is multipath 
purely for failover.

Additionally do I get twice the bandwidth anyhow since both HBA's see the 
same disk ?  How can I tell if either one or the other or both hba's are 
being utilized...

These are private loops being run in full duplex.

Re: 2 HBA's and Multipath.

[Luca Berra]

On Mon, Apr 08, 2002 at 02:02:56PM -0500, SoulBlazer wrote:
> Question,
> 
> I have 2 Qlogic Fiberchannel HBA's (2100F's) connected directly to both FCAL 
> ports on my Sun A5200 22 Disk disk array.  If I enable multipath, do I get 
> twice the throughput between my linux box and the array ? Or is multipath 
> purely for failover.
> 
> Additionally do I get twice the bandwidth anyhow since both HBA's see the 
> same disk ?  How can I tell if either one or the other or both hba's are 
> being utilized...

multipath does not do any read-balancing (see code below),
while i don't think it would be difficult to implement (cut&paste from raid1.c)
i did not have any time to try

only way i found to do read-balancing at the moment is if you have multiple luns on your
array you can create md configs with alternating channel order.

regards,
Luca

------- from 2.4.18-rc1/drivers/md/multipath.c
static int multipath_read_balance (multipath_conf_t *conf)
{
        int disk;

        for (disk = 0; disk < conf->raid_disks; disk++)
                if (conf->multipaths[disk].operational)
                        return disk;
        BUG();
        return 0;
}

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: 2 HBA's and Multipath.

[SoulBlazer]

What about write balancing ?

I would also be interested in discussing a possible patch to the raid code to 
do write/read balancing over N hba's if it is that trivial.

On April 8, 2002 03:16 pm, Luca Berra wrote:
> On Mon, Apr 08, 2002 at 02:02:56PM -0500, SoulBlazer wrote:
> > Question,
> >
> > I have 2 Qlogic Fiberchannel HBA's (2100F's) connected directly to both
> > FCAL ports on my Sun A5200 22 Disk disk array.  If I enable multipath, do
> > I get twice the throughput between my linux box and the array ? Or is
> > multipath purely for failover.
> >
> > Additionally do I get twice the bandwidth anyhow since both HBA's see the
> > same disk ?  How can I tell if either one or the other or both hba's are
> > being utilized...
>
> multipath does not do any read-balancing (see code below),
> while i don't think it would be difficult to implement (cut&paste from
> raid1.c) i did not have any time to try
>
> only way i found to do read-balancing at the moment is if you have multiple
> luns on your array you can create md configs with alternating channel
> order.
>
> regards,
> Luca
>
> ------- from 2.4.18-rc1/drivers/md/multipath.c
> static int multipath_read_balance (multipath_conf_t *conf)
> {
>         int disk;
>
>         for (disk = 0; disk < conf->raid_disks; disk++)
>                 if (conf->multipaths[disk].operational)
>                         return disk;
>         BUG();
>         return 0;
> }

Re: 2 HBA's and Multipath.

[Luca Berra]

On Mon, Apr 08, 2002 at 03:03:00PM -0500, SoulBlazer wrote:
> What about write balancing ?
> 
> I would also be interested in discussing a possible patch to the raid code to 
> do write/read balancing over N hba's if it is that trivial.

the multipath_read_balance routine is actually used both for reading and writing
besides with multipath we don't need to check the head position of the disk
since they are supposed to be the same (if the concept of head position has
any sense with your storage, which i strongly doubt).

what we would need is adding an
int last_used;
to struct multipath_private_data
in include/linux/raid/multipath.h

then rewrite multipath_read_balance in drivers/md/multipath.c
to look something like this.

static int multipath_read_balance (multipath_conf_t *conf)
{
    int disk;

    for (disk = conf->last_used + 1; disk != conf->last_used; disk++) {
	if (disk > conf->raid_disks)
	    disk = 0;
        if (conf->multipaths[disk].operational)
	    return disk;
	}
    BUG();
    return 0;
}

that's all folks, unless i did some very big fuck-up in the code


-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: 2 HBA's and Multipath.

[SoulBlazer]

Well I applied those modifications however I am noticing the following :

--BEGIN SNIPPIT--

# : mdadm -D /dev/md/1
/dev/md/1:
        Version : 00.90.00
  Creation Time : Fri Apr 19 12:14:13 2002
     Raid Level : multipath
     Array Size : 17688448 (16.86 GiB 18.11 GB)
   Raid Devices : 1
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Fri Apr 19 12:14:13 2002
          State : dirty, no-errors
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

    Number   Major   Minor   RaidDevice State
       0      65      113        0      active sync   
/dev/scsi/host3/bus0/target1/lun0/part1
       1       8       17        1        
/dev/scsi/host2/bus0/target1/lun0/part1

--END SNIPPIT--

Additionally during a write operation to the volume I noticed the following 
during a dd(1) on the multipath device:

--BEGIN SNIPPIT--
# cd /proc/scsi/isp
# until [ 1 -gt 2 ] ; do grep Request 2 3; sleep 1; done
2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 222   Req Out 226   Result 82 Nactv 255 HiWater 295 QAVAIL 3 WtQHi 0

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 744   Req Out 869   Result 82 Nactv 53 HiWater 295 QAVAIL 124 WtQHi

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 1001 Req Out 1016 Result 82 Nactv 295 HiWater 295 QAVAIL 14 WtQHi 0

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 592   Req Out 595   Result 82 Nactv 256 HiWater 295 QAVAIL 2 WtQHi 0
--END SNIPPIT--

This confirms what mdadm reported, in that only 1 of my FCAL Hba's (host3) 
was doing the work (active).

Is it possible to activate the spare device, I realize that multipath is for 
failover.. however could you not multiplex or trunk the FCAL hba's ?  The 
intended goal is to get 2gbps out of this sun a5200 22 disk fcal array in 
linux (the a5200 has 2 FCAL SES Gbic ports .. I am connecting 1 Hba per a5200 
port from my Linux box, each hba registers its own private loop to the array).

-- Off topic --
I am using the Qlogic FCAL driver written by Feral Softwares Matthew Jacob 
which is LGPL v2 for linux.  This driver is infinitely better then the 
bundled driver in the stock linux kernels and so far better then the 4.27B 
drivers from Qlogic themselves which seem riddled with bugs.  Why this has 
not been merged as of yet is beyond me as the other drivers seem unusable.

ftp://ftp.feral.com/pub/isp/isp_dist.tgz <-- the source.

Kudos again to Matthew Jacob for such an excellent driver for without which I 
would be forced to use Solaris and disksuite (ugh).


On April 11, 2002 05:49 am, Luca Berra wrote:
> On Mon, Apr 08, 2002 at 03:03:00PM -0500, SoulBlazer wrote:
> > What about write balancing ?
> >
> > I would also be interested in discussing a possible patch to the raid
> > code to do write/read balancing over N hba's if it is that trivial.
>
> the multipath_read_balance routine is actually used both for reading and
> writing besides with multipath we don't need to check the head position of
> the disk since they are supposed to be the same (if the concept of head
> position has any sense with your storage, which i strongly doubt).
>
> what we would need is adding an
> int last_used;
> to struct multipath_private_data
> in include/linux/raid/multipath.h
>
> then rewrite multipath_read_balance in drivers/md/multipath.c
> to look something like this.
>
> static int multipath_read_balance (multipath_conf_t *conf)
> {
>     int disk;
>
>     for (disk = conf->last_used + 1; disk != conf->last_used; disk++) {
> 	if (disk > conf->raid_disks)
> 	    disk = 0;
>         if (conf->multipaths[disk].operational)
> 	    return disk;
> 	}
>     BUG();
>     return 0;
> }
>
> that's all folks, unless i did some very big fuck-up in the code

Re: 2 HBA's and Multipath.

[SoulBlazer]

Additionally this is an email I wrote to Matthew Jacob describing what im 
doing in more detail:


Subject: FCAL/Linux : Perhaps something different
Date: Fri, 19 Apr 2002 14:25:46 -0400
X-Mailer: KMail [version 1.3.2]
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Status: RO
X-Status: S
 
Hey Matthew,

I would like to start off by thanking you for such a superb Linux qlogic 
isp2x00 driver.  You cannot imagine the headaches I have had trying to get 
the bundled kernel qlogicfc, or the Qlogic 4.27B drivers to work properly.

Now if you would be so kind as to indulge a problem I am wrestling with at 
the moment.

=)


--Intended Goal--
Use Linux/x86 with two Qlogic ISP2100F FCAL HBA's to access an StoreEdge 
a5200 Storage array at 2Gbps* (1Gbps*/HBA).  Accomplish this by trunking / 
bonding / loadbalancing / ? both hba's to both GBIC ports on the array via 
independent private loops.

* The speed being subjective, I'm told that I can expect on average 85%+ 
'real' raw throughput out of these cards.

--Testbed Setup--
I have a Linux Server :

Dual AMD 1.2GHz, Tyan ThunderK7 PCI-64
2x Qlogic ISP2100F HBA's 
Kernel 2.4.18

Connected to: 

Sun StoreEdge A5200, 22 Disk (18.2GB/drive)
2x SES FCAL GBIC Ports

Where : 

HBA 1 (Linux) is wired directly via multimode fibre to A5200 GBIC port 1

and 

HBA 2 (Linux) is wired directly via multimode fibre to A5200 GBIC port 2

--Kernel Snippage--

# insmod isp_mod.o isp_fcduplex=1

ISP SCSI and Fibre Channel Host Adapter Driver
      Linux Platform Version 2.1
      Common Core Code Version 2.5
      Built on Apr 17 2002, 17:41:10
isp0: Board Type 2100, Chip Revision 0x4, loaded F/W Revision 1.19.20
isp0: Installed in 64-Bit PCI slot
isp0: Last F/W revision was 0.789.8224
isp0: NVRAM Port WWN 0x200000e08b02ba29
isp0: Loop ID 11, AL_PA 0xd4, Port ID 0xd4, State 0x2, Topology 'Private Loop'
isp1: Board Type 2100, Chip Revision 0x3, loaded F/W Revision 1.19.20
isp1: Installed in 64-Bit PCI slot
isp1: NVRAM Port WWN 0x200000e08b006999
scsi2 : Driver for a Qlogic ISP 2100 Host Adapter
scsi3 : Driver for a Qlogic ISP 2100 Host Adapter
isp1: Loop ID 125, AL_PA 0x1, Port ID 0x1, State 0x2, Topology 'Private Loop'
scsi::resize_dma_pool: WARNING, dma_sectors=64, wanted=28576, scaling
scsi::resize_dma_pool: WARNING, dma_sectors=64, wanted=21440, scaling
<Description of all the detected Drives>
...

--Notes and Questions--

First off, is what I'm trying to do here even possible ?

Assuming it is, I have tried creating a multipath meta-device via the 
software raid facility in the 2.4 kernel.

--Snippit from /etc/raidtab--
raiddev /dev/md/0
        raid-level      multipath
        nr-raid-disks   2

        device          /dev/scsi/host2/bus0/target0/lun0/part1
        raid-disk       0
        device          /dev/scsi/host3/bus0/target0/lun0/part1
        raid-disk       1

--Snippit from /proc/mdstat--
# : cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath]
read_ahead 1024 sectors

md0 : active multipath scsi/host3/bus0/target0/lun0/part1[0]                  
                                      scsi/host2/bus0/target0/lun0/part1[1]
                                      17688448 blocks [1/1] [U]

Where host2 is the qlogic hba registered as isp0, host3 is the qlogic hba 
registered as isp1, and target0/lun0/part1 is the first detected disk in the 
a5200.

Now I realise that multipath in linux is really meant for failover/HA however 
after discussing this on linux-raid, Luca Berra proposed the following 
modifications to the existing multipath code:

Add "int last_used;"  to struct multipath_private_data in 
"include/linux/raid/multipath.h".

Replace the original multipath_read_balance function in multipath.c
with the following:

static int multipath_read_balance (multipath_conf_t *conf)
{
    int disk;

    for (disk = conf->last_used + 1; disk != conf->last_used; disk++) {
        if (disk > conf->raid_disks)
            disk = 0;
        if (conf->multipaths[disk].operational)
            return disk;
        }
    BUG();
    return 0;
}

I made these changes and recompiled, after which I noticed that while doing a 
dd(1) write to a raid-0 set consisting of multipath volumes the following was 
happening: 

--BEGIN SNIPPIT--
# cd /proc/scsi/isp
# until [ 1 -gt 2 ] ; do grep Request 2 3; sleep 1; done
2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 222   Req Out 226   Result 82 Nactv 255 HiWater 295 QAVAIL 3 WtQHi 0

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 744   Req Out 869   Result 82 Nactv 53 HiWater 295 QAVAIL 124 WtQHi

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 1001 Req Out 1016 Result 82 Nactv 295 HiWater 295 QAVAIL 14 WtQHi 0

2: Req In 159   Req Out 158   Result 82 Nactv 0 HiWater 1 QAVAIL 1022 WtQHi 0
3: Req In 592   Req Out 595   Result 82 Nactv 256 HiWater 295 QAVAIL 2 WtQHi 0
--END SNIPPIT--

This confirms that only 1 of my FCAL Hba's (host3/isp1) was doing the work 
(active) while the other (host2/isp0) remained dormant.

Anyhow any feedback you can offer would be appreciated, perhaps I'm going 
about this all wrong or have overlooked something.


Cheers,
-- 
Ex Ignis, Palam Tempestas, Electus Evasto

Re: 2 HBA's and Multipath.

[Luca Berra]

On Fri, Apr 19, 2002 at 12:18:11PM -0400, SoulBlazer wrote:
> Well I applied those modifications however I am noticing the following :
> 
> --BEGIN SNIPPIT--
> 
> # : mdadm -D /dev/md/1
> /dev/md/1:
>         Version : 00.90.00
>   Creation Time : Fri Apr 19 12:14:13 2002
>      Raid Level : multipath
>      Array Size : 17688448 (16.86 GiB 18.11 GB)
>    Raid Devices : 1
>   Total Devices : 2
> Preferred Minor : 1
>     Persistence : Superblock is persistent
> 
>     Update Time : Fri Apr 19 12:14:13 2002
>           State : dirty, no-errors
>  Active Devices : 1
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 1
> 
>     Number   Major   Minor   RaidDevice State
>        0      65      113        0      active sync   
> /dev/scsi/host3/bus0/target1/lun0/part1
>        1       8       17        1        
> /dev/scsi/host2/bus0/target1/lun0/part1
> 

Oops,
seems i forgot about spares :(

multipath.c sets all disks as spares to start with.
a dirty hack could be:

look for:
        /*
         * Mark all disks as spare to start with, then pick our
         * active disk.  If we have a disk that is marked active
         * in the sb, then use it, else use the first rdev.
         */

below you will find:
       if (disk_active(desc)) {
            if(!conf->working_disks) { //remove this line
                printk(OPERATIONAL, partition_name(rdev->dev),
                    desc->raid_disk);
                disk->operational = 1;
                disk->spare = 0;
                conf->working_disks++;
                def_rdev = rdev;
            } else { //remove this line
                mark_disk_spare(desc); //remove this line
            } //remove this line
        } else
            mark_disk_spare(desc);

remove the lines i market with //remove this line
and it should activate the disks

i wont have a system at hand to test this, so please let me know
if it works

we then need to clean up the logic for initialization
since is nonsense to set a disk as spare then reset it back
again as operational, but this could work for a start

good luck

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: 2 HBA's and Multipath.

[SoulBlazer]

Luca, 

Here are the results the modifications you suggested :

# : mdadm -D /dev/md/0
/dev/md/0:
        Version : 00.90.00
  Creation Time : Tue Apr 23 15:55:33 2002
     Raid Level : multipath
     Array Size : 17688448 (16.86 GiB 18.11 GB)
   Raid Devices : 1
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Apr 23 16:00:31 2002
          State : dirty, no-errors
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1


    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   
/dev/scsi/host2/bus0/target0/lun0/part1
       1      65       97        1      active   
/dev/scsi/host3/bus0/target0/lun0/part1
           UUID : b0191dc2:e89b155f:66c3e4e4:fa925d37

# : cat /proc/mdstat
md0 : active multipath scsi/host3/bus0/target0/lun0/part1[1] 
scsi/host2/bus0/target0/lun0/part1[0]
      17688448 blocks [1/2] [U]

Again only one HBA is utilized, which is verified via the previous testing I 
had run.  I believe that even after we hack the code to set all disks to 
'active', the raid code will still only use the first drive it finds that is 
not a spare.

--Snippit-- (multipath.c)
  * Mark all disks as spare to start with, then pick our
  * active disk.  If we have a disk that is marked active
  * in the sb, then use it, else use the first rdev.  
--Snippit--

Additionally I have noticed the following in my syslog on the mkraid of the 
multipath device.

--Snippit--
[events: 00000002]
[events: 00000002]
md: autorun ...
md: considering scsi/host3/bus0/target0/lun0/part1 ...
md:  adding scsi/host3/bus0/target0/lun0/part1 ...
md:  adding scsi/host2/bus0/target0/lun0/part1 ...
md: created md0
md: bind<scsi/host2/bus0/target0/lun0/part1,1>
md: bind<scsi/host3/bus0/target0/lun0/part1,2>
md: running: 
<scsi/host3/bus0/target0/lun0/part1><scsi/host2/bus0/target0/lun0/part1>
md: scsi/host3/bus0/target0/lun0/part1's event counter: 00000002
md: scsi/host2/bus0/target0/lun0/part1's event counter: 00000002
md0: max total readahead window set to 124k
md0: 1 data-disks, max readahead per data-disk: 124k
multipath: making IO path scsi/host3/bus0/target0/lun0/part1 a spare path 
(not in sync)
multipath: device scsi/host3/bus0/target0/lun0/part1 operational as IO path 1
multipath: device scsi/host2/bus0/target0/lun0/part1 operational as IO path 0
(checking disk 0)
(checking disk 1)
multipath: array md0 active with 1 out of 1 IO paths (1 spare IO paths)
md: updating md0 RAID superblock on device
md: (skipping alias scsi/host3/bus0/target0/lun0/part1 )
md: scsi/host2/bus0/target0/lun0/part1 [events: 00000003]<6>(write) 
scsi/host2/bus0/target0/lun0/part1's sb offset: 17688448

--Snippit--

Please note the comment about host3 being marked a spare path (not in sync).



Any comments as always, are welcome.


On April 19, 2002 05:58 pm, Luca Berra wrote:

> Oops,
> seems i forgot about spares :(
>
> multipath.c sets all disks as spares to start with.
> a dirty hack could be:
>
> look for:
>         /*
>          * Mark all disks as spare to start with, then pick our
>          * active disk.  If we have a disk that is marked active
>          * in the sb, then use it, else use the first rdev.
>          */
>
> below you will find:
>        if (disk_active(desc)) {
>             if(!conf->working_disks) { //remove this line
>                 printk(OPERATIONAL, partition_name(rdev->dev),
>                     desc->raid_disk);
>                 disk->operational = 1;
>                 disk->spare = 0;
>                 conf->working_disks++;
>                 def_rdev = rdev;
>             } else { //remove this line
>                 mark_disk_spare(desc); //remove this line
>             } //remove this line
>         } else
>             mark_disk_spare(desc);
>
> remove the lines i market with //remove this line
> and it should activate the disks
>
> i wont have a system at hand to test this, so please let me know
> if it works
>
> we then need to clean up the logic for initialization
> since is nonsense to set a disk as spare then reset it back
> again as operational, but this could work for a start
>
> good luck
>
> L.