[PATCH] Make sure the state of a path is set properly when controller is swapped from passive to active

[PATCH] Make sure the state of a path is set properly when controller is swapped from passive to active

[Chandra Seetharaman]

When the controller ownership is changed (from passive to active),
check_ownership() doesn't set the state of the device to ACTIVE.

This patch fixes the problem.

Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Reported and tested by: "Moger, Babu" <Babu.Moger@lsi.com>
---
Index: linux-2.6.28-rc3/drivers/scsi/device_handler/scsi_dh_rdac.c
===================================================================
--- linux-2.6.28-rc3.orig/drivers/scsi/device_handler/scsi_dh_rdac.c
+++ linux-2.6.28-rc3/drivers/scsi/device_handler/scsi_dh_rdac.c
@@ -387,6 +387,7 @@ static int check_ownership(struct scsi_d
 	struct c9_inquiry *inqp;
 
 	h->lun_state = RDAC_LUN_UNOWNED;
+	h->state = RDAC_STATE_ACTIVE;
 	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
 	if (err == SCSI_DH_OK) {
 		inqp = &h->inq.c9;

[PATCH] scsi_dh: Adding LSI vendor and product ids in rdac device list

[Moger, Babu]

From: Babu Moger <babu.moger@lsi.com>

This patch adds LSI vendor and product ids in rdac device handler.

Signed-off-by: Babu Moger <Babu.Moger@lsi.com>

---

--- linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c.orig    2008-11-10 16:56:01.000000000 -0600
+++ linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c 2008-11-10 16:56:20.000000000 -0600
@@ -594,6 +594,8 @@ static const struct scsi_dh_devlist rdac
        {"SUN", "LCSM100_F"},
        {"DELL", "MD3000"},
        {"DELL", "MD3000i"},
+       {"LSI", "INF-01-00"},
+       {"ENGENIO", "INF-01-00"},
        {NULL, NULL},
 };

DM does not activate the paths if there are more than one path in path group during failover

[Moger, Babu]

Hi All,
  I have noticed one more problem with device mapper today.

Summary:  If there are more than one path in the path group, device mapper does not activate the all the paths during the failover. DM only activates the first path in the path group. It does not call the function activate_path for the second path (or third so on). This seems like a major problem for me. Current code does not keep track of whether the path is already activated or not.  This leads DM using only first path and all other paths become unusable. This happens during the failover.

Test case.
1. Start the IO with all the paths.
2. Fail the active path.
3. Failover (mode select) will happen and passive path will be activated
4. Only first path in path group is activated and other paths will be failed. I would expect the DM to activate all the paths in the path group. However this does not happen.

Output of multipath -ll before the test.

mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
[size=2.0G][features=0][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=4][enabled]
 \_ 3:0:0:0 sde 8:64  [active][undef] running
 \_ 3:0:2:0 sdg 8:96  [active][undef] running
\_ round-robin 0 [prio=2][enabled]
 \_ 3:0:1:0 sdf 8:80  [active][undef] running
 \_ 3:0:3:0 sdh 8:112 [active][undef] running


Output of multipath -ll after the test. Notice that second path has failed.

mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
[size=2.0G][features=0][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 3:0:1:0 sdf 8:80  [active][undef] running
 \_ 3:0:3:0 sdh 8:112 [failed][undef] running


Here is the patch, I have used to work-around the problem. I am sure this is not the place we want to add the fix. However, this patch should give you understanding of the problem. This patch will set the flags
m->pg_init_required and m->queue_io whenever D-M select new path in the path group whenever the repeat_count is exhausted.

--- linux-2.6.28-rc4/drivers/md/dm-mpath.c.orig 2008-11-10 17:50:24.000000000 -0600
+++ linux-2.6.28-rc4/drivers/md/dm-mpath.c      2008-11-10 17:51:36.000000000 -0600
@@ -245,6 +245,10 @@ static int __choose_path_in_pg(struct mu
        if (!path)
                return -ENXIO;

+       /* Set the pg_init_required flag to activate this path */
+       m->pg_init_required = 1;
+       m->queue_io = 1;
+
        m->current_pgpath = path_to_pgpath(path);

        if (m->current_pg != pg)

Re: DM does not activate the paths if there are more than one path in path group during failover

[Alasdair G Kergon]

pg_init is a function intended to be called *once* when there is a switch to
a new group of paths.  It is not per-path initialisation.

Alasdair
-- 
agk@redhat.com

Re: DM does not activate the paths if there are more than one path in path group during failover

[Chandra Seetharaman]

On Mon, 2008-11-10 at 17:45 -0700, Moger, Babu wrote:
> Hi All,
>   I have noticed one more problem with device mapper today.
> 
> Summary:  If there are more than one path in the path group, device mapper does not activate the all the paths during the failover. DM only activates the first path in the path group. It does not call the function activate_path for the second path (or third so on). This seems like a major problem for me. Current code does not keep track of whether the path is already activated or not.  This leads DM using only first path and all other paths become unusable. This happens during the failover.

We do not need to send activate to each "path" as all paths in a path
group lead to the same controller and that controller is activated when
the activate is sent on the first path itself.

Is your multipathd running ?

BTW, why is the physical path state is showing "undef", it should be
"ghost" or "ready"
> 
> Test case.
> 1. Start the IO with all the paths.
> 2. Fail the active path.
> 3. Failover (mode select) will happen and passive path will be activated
> 4. Only first path in path group is activated and other paths will be failed. I would expect the DM to activate all the paths in the path group. However this does not happen.
> 
> Output of multipath -ll before the test.
> 
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=4][enabled]
>  \_ 3:0:0:0 sde 8:64  [active][undef] running
>  \_ 3:0:2:0 sdg 8:96  [active][undef] running
> \_ round-robin 0 [prio=2][enabled]
>  \_ 3:0:1:0 sdf 8:80  [active][undef] running
>  \_ 3:0:3:0 sdh 8:112 [active][undef] running
> 
> 
> Output of multipath -ll after the test. Notice that second path has failed.
> 
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=2][enabled]
>  \_ 3:0:1:0 sdf 8:80  [active][undef] running
>  \_ 3:0:3:0 sdh 8:112 [failed][undef] running
> 
> 
> Here is the patch, I have used to work-around the problem. I am sure this is not the place we want to add the fix. However, this patch should give you understanding of the problem. This patch will set the flags
> m->pg_init_required and m->queue_io whenever D-M select new path in the path group whenever the repeat_count is exhausted.
> 
> --- linux-2.6.28-rc4/drivers/md/dm-mpath.c.orig 2008-11-10 17:50:24.000000000 -0600
> +++ linux-2.6.28-rc4/drivers/md/dm-mpath.c      2008-11-10 17:51:36.000000000 -0600
> @@ -245,6 +245,10 @@ static int __choose_path_in_pg(struct mu
>         if (!path)
>                 return -ENXIO;
> 
> +       /* Set the pg_init_required flag to activate this path */
> +       m->pg_init_required = 1;
> +       m->queue_io = 1;
> +
>         m->current_pgpath = path_to_pgpath(path);
> 
>         if (m->current_pg != pg)
> 
> 
> 

RE: DM does not activate the paths if there are more than one path in path group during failover

[Moger, Babu]

Is your multipathd running?
  Yes, multipathd deamon is running.

BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"

  This is because i am running the mainstream multipath-tools. That is the reason you are seeing "undef".  I have reverted back to native tools and i am still seeing the problem. Here the output of multipath -ll with native tools.

mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
[size=2.0G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:1:0 sdf 8:80  [active][ready]
 \_ 3:0:3:0 sdh 8:112 [failed][ready]



-----Original Message-----
From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
Sent: Monday, November 10, 2008 8:06 PM
To: Moger, Babu
Cc: linux-scsi@vger.kernel.org; device-mapper development; andmike@linux.vnet.ibm.com
Subject: Re: DM does not activate the paths if there are more than one path in path group during failover


On Mon, 2008-11-10 at 17:45 -0700, Moger, Babu wrote:
> Hi All,
>   I have noticed one more problem with device mapper today.
>
> Summary:  If there are more than one path in the path group, device mapper does not activate the all the paths during the failover. DM only activates the first path in the path group. It does not call the function activate_path for the second path (or third so on). This seems like a major problem for me. Current code does not keep track of whether the path is already activated or not.  This leads DM using only first path and all other paths become unusable. This happens during the failover.

We do not need to send activate to each "path" as all paths in a path group lead to the same controller and that controller is activated when the activate is sent on the first path itself.

Is your multipathd running ?

BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"
>
> Test case.
> 1. Start the IO with all the paths.
> 2. Fail the active path.
> 3. Failover (mode select) will happen and passive path will be
> activated 4. Only first path in path group is activated and other paths will be failed. I would expect the DM to activate all the paths in the path group. However this does not happen.
>
> Output of multipath -ll before the test.
>
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> [prio=4][enabled]  \_ 3:0:0:0 sde 8:64  [active][undef] running  \_
> 3:0:2:0 sdg 8:96  [active][undef] running \_ round-robin 0
> [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> 3:0:3:0 sdh 8:112 [active][undef] running
>
>
> Output of multipath -ll after the test. Notice that second path has failed.
>
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> 3:0:3:0 sdh 8:112 [failed][undef] running
>
>
> Here is the patch, I have used to work-around the problem. I am sure
> this is not the place we want to add the fix. However, this patch
> should give you understanding of the problem. This patch will set the
> flags
> m->pg_init_required and m->queue_io whenever D-M select new path in the path group whenever the repeat_count is exhausted.
>
> --- linux-2.6.28-rc4/drivers/md/dm-mpath.c.orig 2008-11-10
> 17:50:24.000000000 -0600
> +++ linux-2.6.28-rc4/drivers/md/dm-mpath.c      2008-11-10 17:51:36.000000000 -0600
> @@ -245,6 +245,10 @@ static int __choose_path_in_pg(struct mu
>         if (!path)
>                 return -ENXIO;
>
> +       /* Set the pg_init_required flag to activate this path */
> +       m->pg_init_required = 1;
> +       m->queue_io = 1;
> +
>         m->current_pgpath = path_to_pgpath(path);
>
>         if (m->current_pg != pg)
>
>
>

RE: DM does not activate the paths if there are more than one path in path group during failover

[Chandra Seetharaman]

On Tue, 2008-11-11 at 10:07 -0700, Moger, Babu wrote:
> Is your multipathd running?
>   Yes, multipathd deamon is running.
> 
> BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"
> 
>   This is because i am running the mainstream multipath-tools. That is the reason you are seeing "undef".  I have reverted back to native tools and i am still seeing the problem. Here the output of multipath -ll with native tools.
> 
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac]
> \_ round-robin 0 [prio=0][active]
>  \_ 3:0:1:0 sdf 8:80  [active][ready]
>  \_ 3:0:3:0 sdh 8:112 [failed][ready]
> 

can you send me the full o/p multipath -ll mpathf;

prio=0 means that this path group is connected to the passive
controller, in which case "failed" is correct.

> 
> 
> -----Original Message-----
> From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
> Sent: Monday, November 10, 2008 8:06 PM
> To: Moger, Babu
> Cc: linux-scsi@vger.kernel.org; device-mapper development; andmike@linux.vnet.ibm.com
> Subject: Re: DM does not activate the paths if there are more than one path in path group during failover
> 
> 
> On Mon, 2008-11-10 at 17:45 -0700, Moger, Babu wrote:
> > Hi All,
> >   I have noticed one more problem with device mapper today.
> >
> > Summary:  If there are more than one path in the path group, device mapper does not activate the all the paths during the failover. DM only activates the first path in the path group. It does not call the function activate_path for the second path (or third so on). This seems like a major problem for me. Current code does not keep track of whether the path is already activated or not.  This leads DM using only first path and all other paths become unusable. This happens during the failover.
> 
> We do not need to send activate to each "path" as all paths in a path group lead to the same controller and that controller is activated when the activate is sent on the first path itself.
> 
> Is your multipathd running ?
> 
> BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"
> >
> > Test case.
> > 1. Start the IO with all the paths.
> > 2. Fail the active path.
> > 3. Failover (mode select) will happen and passive path will be
> > activated 4. Only first path in path group is activated and other paths will be failed. I would expect the DM to activate all the paths in the path group. However this does not happen.
> >
> > Output of multipath -ll before the test.
> >
> > mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> > [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> > [prio=4][enabled]  \_ 3:0:0:0 sde 8:64  [active][undef] running  \_
> > 3:0:2:0 sdg 8:96  [active][undef] running \_ round-robin 0
> > [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> > 3:0:3:0 sdh 8:112 [active][undef] running
> >
> >
> > Output of multipath -ll after the test. Notice that second path has failed.
> >
> > mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> > [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> > [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> > 3:0:3:0 sdh 8:112 [failed][undef] running
> >
> >
> > Here is the patch, I have used to work-around the problem. I am sure
> > this is not the place we want to add the fix. However, this patch
> > should give you understanding of the problem. This patch will set the
> > flags
> > m->pg_init_required and m->queue_io whenever D-M select new path in the path group whenever the repeat_count is exhausted.
> >
> > --- linux-2.6.28-rc4/drivers/md/dm-mpath.c.orig 2008-11-10
> > 17:50:24.000000000 -0600
> > +++ linux-2.6.28-rc4/drivers/md/dm-mpath.c      2008-11-10 17:51:36.000000000 -0600
> > @@ -245,6 +245,10 @@ static int __choose_path_in_pg(struct mu
> >         if (!path)
> >                 return -ENXIO;
> >
> > +       /* Set the pg_init_required flag to activate this path */
> > +       m->pg_init_required = 1;
> > +       m->queue_io = 1;
> > +
> >         m->current_pgpath = path_to_pgpath(path);
> >
> >         if (m->current_pg != pg)
> >
> >
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: DM does not activate the paths if there are more than one path in path group during failover

[Moger, Babu]

Thanks for the response.

If pg_init is intended to be called only once then problem is elsewhere.

During the pg_init, the activate_path is called. The activate_path calls scsi_dh_activate(then rdac_activate) which will set path state to active(h->state = RDAC_STATE_ACTIVE;). My understanding is, h->state is specific to each path. How does d-m sets h->state to RDAC_STATE_ACTIVE for all the paths available in the path group.

This is what happening in my case. On path group failure, d-m switches the path group initiates path group. It then selects the first path (in the path group) starts I/O. After exhausting the repeat_count it then selects the next path in the path group.  Then rdac_prep_fn is called. This function finds h->state is still passive and fails the path which becomes unusable.

This function returns BLKPREP_KILL which is leading to fail failure.

static int rdac_prep_fn(struct scsi_device *sdev, struct request *req)
{
        struct rdac_dh_data *h = get_rdac_data(sdev);
        int ret = BLKPREP_OK;

        if (h->state != RDAC_STATE_ACTIVE) {
                ret = BLKPREP_KILL;
                req->cmd_flags |= REQ_QUIET;
        }
        return ret;
}

This is my understanding so far. Please correct me if there is anything wrong.


-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Alasdair G Kergon
Sent: Monday, November 10, 2008 6:59 PM
To: device-mapper development
Cc: andmike@linux.vnet.ibm.com; linux-scsi@vger.kernel.org
Subject: Re: [dm-devel] DM does not activate the paths if there are more than one path in path group during failover

pg_init is a function intended to be called *once* when there is a switch to a new group of paths.  It is not per-path initialisation.

Alasdair
--
agk@redhat.com

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: DM does not activate the paths if there are more than one path in path group during failover

[Moger, Babu]

Here is full output of multipath -ll mpathf before the test.

[root@mymachine ~]# multipath -ll mpathf
error calling out /lib/udev/scsi_id -g -u -d /dev/sda
error calling out /lib/udev/scsi_id -g -u -d /dev/sdb
error calling out /lib/udev/scsi_id -g -u -d /dev/sdc
error calling out /lib/udev/scsi_id -g -u -d /dev/sdd
error calling out /lib/udev/scsi_id -g -u -d /dev/sdk
error calling out /lib/udev/scsi_id -g -u -d /dev/sdn
mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
[size=2.0G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=200][active]
 \_ 3:0:0:0 sde 8:64  [active][ready]
 \_ 3:0:2:0 sdg 8:96  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 3:0:1:0 sdf 8:80  [active][ghost]
 \_ 3:0:3:0 sdh 8:112 [active][ghost]

Thanks
Babu Moger

-----Original Message-----
From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
Sent: Tuesday, November 11, 2008 11:10 AM
To: Moger, Babu
Cc: linux-scsi@vger.kernel.org; device-mapper development; andmike@linux.vnet.ibm.com
Subject: RE: DM does not activate the paths if there are more than one path in path group during failover


On Tue, 2008-11-11 at 10:07 -0700, Moger, Babu wrote:
> Is your multipathd running?
>   Yes, multipathd deamon is running.
>
> BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"
>
>   This is because i am running the mainstream multipath-tools. That is the reason you are seeing "undef".  I have reverted back to native tools and i am still seeing the problem. Here the output of multipath -ll with native tools.
>
> mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> [size=2.0G][features=0][hwhandler=1 rdac]
> \_ round-robin 0 [prio=0][active]
>  \_ 3:0:1:0 sdf 8:80  [active][ready]
>  \_ 3:0:3:0 sdh 8:112 [failed][ready]
>

can you send me the full o/p multipath -ll mpathf;

prio=0 means that this path group is connected to the passive
controller, in which case "failed" is correct.

>
>
> -----Original Message-----
> From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
> Sent: Monday, November 10, 2008 8:06 PM
> To: Moger, Babu
> Cc: linux-scsi@vger.kernel.org; device-mapper development; andmike@linux.vnet.ibm.com
> Subject: Re: DM does not activate the paths if there are more than one path in path group during failover
>
>
> On Mon, 2008-11-10 at 17:45 -0700, Moger, Babu wrote:
> > Hi All,
> >   I have noticed one more problem with device mapper today.
> >
> > Summary:  If there are more than one path in the path group, device mapper does not activate the all the paths during the failover. DM only activates the first path in the path group. It does not call the function activate_path for the second path (or third so on). This seems like a major problem for me. Current code does not keep track of whether the path is already activated or not.  This leads DM using only first path and all other paths become unusable. This happens during the failover.
>
> We do not need to send activate to each "path" as all paths in a path group lead to the same controller and that controller is activated when the activate is sent on the first path itself.
>
> Is your multipathd running ?
>
> BTW, why is the physical path state is showing "undef", it should be "ghost" or "ready"
> >
> > Test case.
> > 1. Start the IO with all the paths.
> > 2. Fail the active path.
> > 3. Failover (mode select) will happen and passive path will be
> > activated 4. Only first path in path group is activated and other paths will be failed. I would expect the DM to activate all the paths in the path group. However this does not happen.
> >
> > Output of multipath -ll before the test.
> >
> > mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> > [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> > [prio=4][enabled]  \_ 3:0:0:0 sde 8:64  [active][undef] running  \_
> > 3:0:2:0 sdg 8:96  [active][undef] running \_ round-robin 0
> > [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> > 3:0:3:0 sdh 8:112 [active][undef] running
> >
> >
> > Output of multipath -ll after the test. Notice that second path has failed.
> >
> > mpathf (3600a0b80000f519c0000cc8a48fc7d0b) dm-2 LSI,INF-01-00
> > [size=2.0G][features=0][hwhandler=1 rdac][rw] \_ round-robin 0
> > [prio=2][enabled]  \_ 3:0:1:0 sdf 8:80  [active][undef] running  \_
> > 3:0:3:0 sdh 8:112 [failed][undef] running
> >
> >
> > Here is the patch, I have used to work-around the problem. I am sure
> > this is not the place we want to add the fix. However, this patch
> > should give you understanding of the problem. This patch will set the
> > flags
> > m->pg_init_required and m->queue_io whenever D-M select new path in the path group whenever the repeat_count is exhausted.
> >
> > --- linux-2.6.28-rc4/drivers/md/dm-mpath.c.orig 2008-11-10
> > 17:50:24.000000000 -0600
> > +++ linux-2.6.28-rc4/drivers/md/dm-mpath.c      2008-11-10 17:51:36.000000000 -0600
> > @@ -245,6 +245,10 @@ static int __choose_path_in_pg(struct mu
> >         if (!path)
> >                 return -ENXIO;
> >
> > +       /* Set the pg_init_required flag to activate this path */
> > +       m->pg_init_required = 1;
> > +       m->queue_io = 1;
> > +
> >         m->current_pgpath = path_to_pgpath(path);
> >
> >         if (m->current_pg != pg)
> >
> >
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: DM does not activate the paths if there are more than one path in path group during failover

[Chandra Seetharaman]

Will look into this problem and get back to you.

chandra
On Tue, 2008-11-11 at 10:13 -0700, Moger, Babu wrote:
> Thanks for the response.
> 
> If pg_init is intended to be called only once then problem is elsewhere.
> 
> During the pg_init, the activate_path is called. The activate_path calls scsi_dh_activate(then rdac_activate) which will set path state to active(h->state = RDAC_STATE_ACTIVE;). My understanding is, h->state is specific to each path. How does d-m sets h->state to RDAC_STATE_ACTIVE for all the paths available in the path group.
> 
> This is what happening in my case. On path group failure, d-m switches the path group initiates path group. It then selects the first path (in the path group) starts I/O. After exhausting the repeat_count it then selects the next path in the path group.  Then rdac_prep_fn is called. This function finds h->state is still passive and fails the path which becomes unusable.
> 
> This function returns BLKPREP_KILL which is leading to fail failure.
> 
> static int rdac_prep_fn(struct scsi_device *sdev, struct request *req)
> {
>         struct rdac_dh_data *h = get_rdac_data(sdev);
>         int ret = BLKPREP_OK;
> 
>         if (h->state != RDAC_STATE_ACTIVE) {
>                 ret = BLKPREP_KILL;
>                 req->cmd_flags |= REQ_QUIET;
>         }
>         return ret;
> }
> 
> This is my understanding so far. Please correct me if there is anything wrong.
> 
> 
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Alasdair G Kergon
> Sent: Monday, November 10, 2008 6:59 PM
> To: device-mapper development
> Cc: andmike@linux.vnet.ibm.com; linux-scsi@vger.kernel.org
> Subject: Re: [dm-devel] DM does not activate the paths if there are more than one path in path group during failover
> 
> pg_init is a function intended to be called *once* when there is a switch to a new group of paths.  It is not per-path initialisation.
> 
> Alasdair
> --
> agk@redhat.com
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

[RFC][PATCH] Handle multipath paths in a path group properly during pg_init (was "RE: [dm-devel] DM does not activate the paths if there are more than one path in path group during failover")

[Chandra Seetharaman]

The problem reported by Moger Babu was caused due to the architectural
change made when we moved from dm hardware handler to SCSI hardware
handler.

Thanks Babu for finding and reporting the bug.

All of the hardware handlers, do have a state now, and they are set to
active and (some form of) inactive. All of them have prep_fn, which use
this "state" to fail the I/O without it ever being sent to the device.

As Babu has noted in his email, the pg_init/activate is sent on only one
path and the "state" of that path is changed appropriately to "active"
while other paths in the same path group are never changed as they never
got an "activate".

Attached is a patch (compiled, tested, but not clean yet), which makes
changes in the dm-multipath layer to send an "activate" on each paths in
the path groups.

Mike (Anderson) and I had a discussion about whether to implement this
in the dm-mulitpath layer or in the SCSI hardware handler layer and we
came to a conclusion that it is best suited to be in the dm-mulitpath
layer as it is the one that knows the relationship between different
paths.

If it were to be done at the Hardware handler layer, then the hardware
handler may end up having a different notion of the path relationship
and hence may not work as expected by the dm-multipath layer.

Please provide your feedback on this approach and any additional
comments.
----------

Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>

---
Index: linux-2.6.28-rc3/drivers/md/dm-mpath.c
===================================================================
--- linux-2.6.28-rc3.orig/drivers/md/dm-mpath.c
+++ linux-2.6.28-rc3/drivers/md/dm-mpath.c
@@ -36,6 +36,7 @@ struct pgpath {
 
 	struct dm_path path;
 	struct work_struct deactivate_path;
+	struct work_struct activate_path;
 };
 
 #define path_to_pgpath(__pgp) container_of((__pgp), struct pgpath, path)
@@ -65,8 +66,6 @@ struct multipath {
 	spinlock_t lock;
 
 	const char *hw_handler_name;
-	struct work_struct activate_path;
-	struct pgpath *pgpath_to_activate;
 	unsigned nr_priority_groups;
 	struct list_head priority_groups;
 	unsigned pg_init_required;	/* pg_init needs calling? */
@@ -129,6 +128,7 @@ static struct pgpath *alloc_pgpath(void)
 	if (pgpath) {
 		pgpath->is_active = 1;
 		INIT_WORK(&pgpath->deactivate_path, deactivate_path);
+		INIT_WORK(&pgpath->activate_path, activate_path);
 	}
 
 	return pgpath;
@@ -170,10 +170,6 @@ static void free_pgpaths(struct list_hea
 		if (m->hw_handler_name)
 			scsi_dh_detach(bdev_get_queue(pgpath->path.dev->bdev));
 		dm_put_device(ti, pgpath->path.dev);
-		spin_lock_irqsave(&m->lock, flags);
-		if (m->pgpath_to_activate == pgpath)
-			m->pgpath_to_activate = NULL;
-		spin_unlock_irqrestore(&m->lock, flags);
 		free_pgpath(pgpath);
 	}
 }
@@ -203,7 +199,6 @@ static struct multipath *alloc_multipath
 		m->queue_io = 1;
 		INIT_WORK(&m->process_queued_ios, process_queued_ios);
 		INIT_WORK(&m->trigger_event, trigger_event);
-		INIT_WORK(&m->activate_path, activate_path);
 		m->mpio_pool = mempool_create_slab_pool(MIN_IOS, _mpio_cache);
 		if (!m->mpio_pool) {
 			kfree(m);
@@ -428,8 +423,8 @@ static void process_queued_ios(struct wo
 {
 	struct multipath *m =
 		container_of(work, struct multipath, process_queued_ios);
-	struct pgpath *pgpath = NULL;
-	unsigned init_required = 0, must_queue = 1;
+	struct pgpath *pgpath = NULL, *tmp;
+	unsigned must_queue = 1;
 	unsigned long flags;
 
 	spin_lock_irqsave(&m->lock, flags);
@@ -447,19 +442,15 @@ static void process_queued_ios(struct wo
 		must_queue = 0;
 
 	if (m->pg_init_required && !m->pg_init_in_progress && pgpath) {
-		m->pgpath_to_activate = pgpath;
 		m->pg_init_count++;
 		m->pg_init_required = 0;
-		m->pg_init_in_progress = 1;
-		init_required = 1;
+		list_for_each_entry(tmp, &pgpath->pg->pgpaths, list) {
+			queue_work(kmpath_handlerd, &tmp->activate_path);
+			m->pg_init_in_progress++;
+		}
 	}
-
 out:
 	spin_unlock_irqrestore(&m->lock, flags);
-
-	if (init_required)
-		queue_work(kmpath_handlerd, &m->activate_path);
-
 	if (!must_queue)
 		dispatch_queued_ios(m);
 }
@@ -1111,27 +1102,20 @@ static void pg_init_done(struct dm_path 
 		pg->bypassed = 0;
 	}
 
-	m->pg_init_in_progress = 0;
-	queue_work(kmultipathd, &m->process_queued_ios);
+	m->pg_init_in_progress--;
+	if (!m->pg_init_in_progress)
+		queue_work(kmultipathd, &m->process_queued_ios);
 	spin_unlock_irqrestore(&m->lock, flags);
 }
 
 static void activate_path(struct work_struct *work)
 {
 	int ret;
-	struct multipath *m =
-		container_of(work, struct multipath, activate_path);
-	struct dm_path *path;
-	unsigned long flags;
+	struct pgpath *pgpath =
+		container_of(work, struct pgpath, activate_path);
 
-	spin_lock_irqsave(&m->lock, flags);
-	path = &m->pgpath_to_activate->path;
-	m->pgpath_to_activate = NULL;
-	spin_unlock_irqrestore(&m->lock, flags);
-	if (!path)
-		return;
-	ret = scsi_dh_activate(bdev_get_queue(path->dev->bdev));
-	pg_init_done(path, ret);
+	ret = scsi_dh_activate(bdev_get_queue(pgpath->path.dev->bdev));
+	pg_init_done(&pgpath->path, ret);
 }
 
 /*

Re: [RFC][PATCH] Handle multipath paths in a path group properly during pg_init (was "RE: DM does not activate the paths if there are more than one path in path group during failover")

[Alasdair G Kergon]

On Thu, Nov 13, 2008 at 03:48:58PM -0800, Chandra Seetharaman wrote:
> The problem reported by Moger Babu was caused due to the architectural
> change made when we moved from dm hardware handler to SCSI hardware
> handler.
 
pg_init is single initialisation for the group.
It only needs to happen on one path.

It sounds to me that what you're adding here is a *per-path* initialisation,
something *separate* from (and additional to) the existing *per-group* pg_init.

Has this patch been tested on EMC?

Alasdair
-- 
agk@redhat.com

Re: [RFC][PATCH] Handle multipath paths in a path group properly during pg_init (was "RE: [dm-devel] DM does not activate the paths if there are more than one path in path group during failover")

[Chandra Seetharaman]

On Fri, 2008-11-14 at 00:12 +0000, Alasdair G Kergon wrote:
> On Thu, Nov 13, 2008 at 03:48:58PM -0800, Chandra Seetharaman wrote:
> > The problem reported by Moger Babu was caused due to the architectural
> > change made when we moved from dm hardware handler to SCSI hardware
> > handler.
> 
> pg_init is single initialisation for the group.
> It only needs to happen on one path.

Yes, that was my initial response too.

dm hardware handlers did not maintain a "state" for each path in which
case activating/initializing the controller from one path was sufficient
for all the paths connected to the controller to work as expected.

SCSI Hardware Handlers maintain a "state" for each path, so that they
can fail an I/O at prep_fn, if the path is not an "active" path, which
is what lead to the bug Babu saw.

> 
> It sounds to me that what you're adding here is a *per-path* initialisation,
> something *separate* from (and additional to) the existing *per-group* pg_init.

No, I "replaced" the "per-group" pg_init with "per-path" init.

> 
> Has this patch been tested on EMC?

No, we have tested it with LSI RDAC. I will send a separate email to EMC
folks (on Cc here) describing the problem and the fix and ask their help
in testing.

> 
> Alasdair

Re: [RFC][PATCH] Handle multipath paths in a path group properly during pg_init (was "RE: DM does not activate the paths if there are more than one path in path group during failover")

[Alasdair G Kergon]

On Fri, Nov 14, 2008 at 11:08:48AM -0800, Chandra Seetharaman wrote:
> No, I "replaced" the "per-group" pg_init with "per-path" init.
 
Indeed, but I'm pointing out that the per-group initialisation was
originally added for EMC and arguing that per-group initialisation
and per-path initialisation are independent conceptually and the
code may need to support both: adding one to deal with the problem
reported here need not necessitate removing the other.

Alasdair
-- 
agk@redhat.com

Re: [RFC][PATCH] Handle multipath paths in a path group properly during pg_init (was "RE: [dm-devel] DM does not activate the paths if there are more than one path in path group during failover")

[Chandra Seetharaman]

On Fri, 2008-11-14 at 19:34 +0000, Alasdair G Kergon wrote:
> On Fri, Nov 14, 2008 at 11:08:48AM -0800, Chandra Seetharaman wrote:
> > No, I "replaced" the "per-group" pg_init with "per-path" init.
> 
> Indeed, but I'm pointing out that the per-group initialisation was
> originally added for EMC and arguing that per-group initialisation
> and per-path initialisation are independent conceptually and the
> code may need to support both: adding one to deal with the problem
> reported here need not necessitate removing the other.

It was the case earlier. But Ed changed it (to send a mode select only
when needed) before I moved the emc code to be under SCSI Hardware
Handler. Later Hannes made it still better.

IOW, The EMC code in SCSI hardware handler is smart to find out if the
trespass/mode_select command need to be sent down the wire (they first
do a inquiry and if the state is "OWNED", they just return).

Hannes, MikeC, correct me if I am wrong.

> 
> Alasdair

Re: [PATCH] scsi_dh: Adding LSI vendor and product ids in rdac device list

[James Bottomley]

On Mon, 2008-11-10 at 16:17 -0700, Moger, Babu wrote:
> From: Babu Moger <babu.moger@lsi.com>
> 
> This patch adds LSI vendor and product ids in rdac device handler.
> 
> Signed-off-by: Babu Moger <Babu.Moger@lsi.com>
> 
> ---
> 
> --- linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c.orig    2008-11-10 16:56:01.000000000 -0600
> +++ linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c 2008-11-10 16:56:20.000000000 -0600
> @@ -594,6 +594,8 @@ static const struct scsi_dh_devlist rdac
>         {"SUN", "LCSM100_F"},
>         {"DELL", "MD3000"},
>         {"DELL", "MD3000i"},
> +       {"LSI", "INF-01-00"},
> +       {"ENGENIO", "INF-01-00"},
>         {NULL, NULL},
>  };

This patch is whitespace damaged.  Your tabs have been converted to
spaces by your email tool.  I've fixed it up this time, since it was
only two lines, but please see

Documentation/email-clients.txt

For hints on how to get your MUA to work with patches.

James

RE: [PATCH] scsi_dh: Adding LSI vendor and product ids in rdac device list

[Moger, Babu]

Thank You..  Will take care next time.

Thanks
Babu Moger
-----Original Message-----
From: James Bottomley [mailto:James.Bottomley@HansenPartnership.com]
Sent: Sunday, November 23, 2008 6:05 PM
To: Moger, Babu
Cc: linux-scsi@vger.kernel.org; sekharan@us.ibm.com; andmike@linux.vnet.ibm.com; device-mapper development
Subject: Re: [PATCH] scsi_dh: Adding LSI vendor and product ids in rdac device list

On Mon, 2008-11-10 at 16:17 -0700, Moger, Babu wrote:
> From: Babu Moger <babu.moger@lsi.com>
>
> This patch adds LSI vendor and product ids in rdac device handler.
>
> Signed-off-by: Babu Moger <Babu.Moger@lsi.com>
>
> ---
>
> --- linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c.orig    2008-11-10 16:56:01.000000000 -0600
> +++ linux-2.6.28-rc4/drivers/scsi/device_handler/scsi_dh_rdac.c 2008-11-10 16:56:20.000000000 -0600
> @@ -594,6 +594,8 @@ static const struct scsi_dh_devlist rdac
>         {"SUN", "LCSM100_F"},
>         {"DELL", "MD3000"},
>         {"DELL", "MD3000i"},
> +       {"LSI", "INF-01-00"},
> +       {"ENGENIO", "INF-01-00"},
>         {NULL, NULL},
>  };

This patch is whitespace damaged.  Your tabs have been converted to
spaces by your email tool.  I've fixed it up this time, since it was
only two lines, but please see

Documentation/email-clients.txt

For hints on how to get your MUA to work with patches.

James

qla2xxx 0000:03:0c.0: Firmware image unavailable

[Moger, Babu]

Hi All,
Have you guys seen this following error? 

Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware image unavailable. 
Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.

After this error, qlogic driver fails to detect the LUNs attached. 

I am running 2.6.29-rc2 kernel on x86_64 machine.

I tried couple of things without any success. I downloaded all the qlogic firmwares available and kept it in /lib/firmware. Tried rebuilding the kernel. Did not help. Thanks in advance. 


Thanks
Babu Moger 

Re: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Seokmann Ju]

On Jan 21, 2009, at 4:00 PM, Moger, Babu wrote:

> Hi All,
> Have you guys seen this following error?
>
> Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware image  
> unavailable.
> Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware images  
> can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
>
> After this error, qlogic driver fails to detect the LUNs attached.
>
> I am running 2.6.29-rc2 kernel on x86_64 machine.
>
> I tried couple of things without any success. I downloaded all the  
> qlogic firmwares available and kept it in /lib/firmware. Tried  
> rebuilding the kernel. Did not help. Thanks in advance.
Could you double check that those firmware binaries under /lib/ 
firmware have correct naming convention as defined in qla_os.c:  
FW_FILE_ISP2XXX?

>
>
>
> Thanks
> Babu Moger
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Moger, Babu]

Yes, the naming conventions look good. Here is the output of ls

-rw-r--r-- 1 root root  76802 Jan 21 16:41 ql2100_fw.bin
-rw-r--r-- 1 root root  84566 Jan 21 16:41 ql2200_fw.bin
-rw-r--r-- 1 root root 125088 Jan 21 16:41 ql2300_fw.bin
-rw-r--r-- 1 root root 135896 Jan 21 16:41 ql2322_fw.bin
-rw-r--r-- 1 root root 228056 Jan 21 16:41 ql2400_fw.bin
-rw-r--r-- 1 root root 239052 Jan 21 16:41 ql2400_fw.bin_mid
-rw-r--r-- 1 root root 200064 Jan 21 16:41 ql2500_fw.bin
-rw-r--r-- 1 root root 219924 Jan 21 16:41 ql2500_fw.bin_mid

Thanks
Babu Moger 

-----Original Message-----
From: Seokmann Ju [mailto:seokmann.ju@qlogic.com] 
Sent: Wednesday, January 21, 2009 6:26 PM
To: Moger, Babu
Cc: linux-scsi@vger.kernel.org; device-mapper development
Subject: Re: qla2xxx 0000:03:0c.0: Firmware image unavailable


On Jan 21, 2009, at 4:00 PM, Moger, Babu wrote:

> Hi All,
> Have you guys seen this following error?
>
> Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware image  
> unavailable.
> Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: Firmware images  
> can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
>
> After this error, qlogic driver fails to detect the LUNs attached.
>
> I am running 2.6.29-rc2 kernel on x86_64 machine.
>
> I tried couple of things without any success. I downloaded all the  
> qlogic firmwares available and kept it in /lib/firmware. Tried  
> rebuilding the kernel. Did not help. Thanks in advance.
Could you double check that those firmware binaries under /lib/ 
firmware have correct naming convention as defined in qla_os.c:  
FW_FILE_ISP2XXX?

>
>
>
> Thanks
> Babu Moger
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Sudhakar Pannerselvam]

Firmware is loaded through udev hotplug interface i.e using
request_firmware(). Can you verify if udev hotplug is functional in your
setup.

To reproduce this in my local setup, I have kill'ed udevd daemon and
loaded qlogic module which failed with firmware unavailable message. 

Regards,
Sudhakar

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Moger, Babu
> Sent: Thursday, January 22, 2009 7:44 PM
> To: Seokmann Ju
> Cc: linux-scsi@vger.kernel.org; device-mapper development
> Subject: RE: qla2xxx 0000:03:0c.0: Firmware image unavailable
> 
> Yes, the naming conventions look good. Here is the output of ls
> 
> -rw-r--r-- 1 root root  76802 Jan 21 16:41 ql2100_fw.bin
> -rw-r--r-- 1 root root  84566 Jan 21 16:41 ql2200_fw.bin
> -rw-r--r-- 1 root root 125088 Jan 21 16:41 ql2300_fw.bin
> -rw-r--r-- 1 root root 135896 Jan 21 16:41 ql2322_fw.bin
> -rw-r--r-- 1 root root 228056 Jan 21 16:41 ql2400_fw.bin
> -rw-r--r-- 1 root root 239052 Jan 21 16:41 ql2400_fw.bin_mid
> -rw-r--r-- 1 root root 200064 Jan 21 16:41 ql2500_fw.bin
> -rw-r--r-- 1 root root 219924 Jan 21 16:41 ql2500_fw.bin_mid
> 
> Thanks
> Babu Moger 
> 
> -----Original Message-----
> From: Seokmann Ju [mailto:seokmann.ju@qlogic.com]
> Sent: Wednesday, January 21, 2009 6:26 PM
> To: Moger, Babu
> Cc: linux-scsi@vger.kernel.org; device-mapper development
> Subject: Re: qla2xxx 0000:03:0c.0: Firmware image unavailable
> 
> 
> On Jan 21, 2009, at 4:00 PM, Moger, Babu wrote:
> 
> > Hi All,
> > Have you guys seen this following error?
> >
> > Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: 
> Firmware image  
> > unavailable.
> > Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: 
> Firmware images  
> > can be retrieved from: 
> ftp://ftp.qlogic.com/outgoing/linux/firmware/.
> >
> > After this error, qlogic driver fails to detect the LUNs attached.
> >
> > I am running 2.6.29-rc2 kernel on x86_64 machine.
> >
> > I tried couple of things without any success. I downloaded all the  
> > qlogic firmwares available and kept it in /lib/firmware. Tried  
> > rebuilding the kernel. Did not help. Thanks in advance.
> Could you double check that those firmware binaries under /lib/ 
> firmware have correct naming convention as defined in qla_os.c:  
> FW_FILE_ISP2XXX?
> 
> >
> >
> >
> > Thanks
> > Babu Moger
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux- 
> > scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[James Bottomley]

On Thu, 2009-01-22 at 07:14 -0700, Moger, Babu wrote:
> Yes, the naming conventions look good. Here is the output of ls
> 
> -rw-r--r-- 1 root root  76802 Jan 21 16:41 ql2100_fw.bin
> -rw-r--r-- 1 root root  84566 Jan 21 16:41 ql2200_fw.bin
> -rw-r--r-- 1 root root 125088 Jan 21 16:41 ql2300_fw.bin
> -rw-r--r-- 1 root root 135896 Jan 21 16:41 ql2322_fw.bin
> -rw-r--r-- 1 root root 228056 Jan 21 16:41 ql2400_fw.bin
> -rw-r--r-- 1 root root 239052 Jan 21 16:41 ql2400_fw.bin_mid
> -rw-r--r-- 1 root root 200064 Jan 21 16:41 ql2500_fw.bin
> -rw-r--r-- 1 root root 219924 Jan 21 16:41 ql2500_fw.bin_mid

OK, so logical choice number 2 is that udev, which feeds the firmware
into the kernel, isn't working for you.  This is really a distribution
problem.

James

RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Moger, Babu]

You are right. Looks like qlogic driver loads even before udev hotplug is functional.  Driver loads successfully when I re-load the driver after the system in up.  Is there any solution or work-around for this problem?

Thanks
Babu Moger 

-----Original Message-----
From: Sudhakar Pannerselvam [mailto:sudhakarp@inmage.net] 
Sent: Thursday, January 22, 2009 8:40 AM
To: Moger, Babu; Seokmann Ju
Cc: linux-scsi@vger.kernel.org; device-mapper development
Subject: RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

Firmware is loaded through udev hotplug interface i.e using
request_firmware(). Can you verify if udev hotplug is functional in your
setup.

To reproduce this in my local setup, I have kill'ed udevd daemon and
loaded qlogic module which failed with firmware unavailable message. 

Regards,
Sudhakar

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Moger, Babu
> Sent: Thursday, January 22, 2009 7:44 PM
> To: Seokmann Ju
> Cc: linux-scsi@vger.kernel.org; device-mapper development
> Subject: RE: qla2xxx 0000:03:0c.0: Firmware image unavailable
> 
> Yes, the naming conventions look good. Here is the output of ls
> 
> -rw-r--r-- 1 root root  76802 Jan 21 16:41 ql2100_fw.bin
> -rw-r--r-- 1 root root  84566 Jan 21 16:41 ql2200_fw.bin
> -rw-r--r-- 1 root root 125088 Jan 21 16:41 ql2300_fw.bin
> -rw-r--r-- 1 root root 135896 Jan 21 16:41 ql2322_fw.bin
> -rw-r--r-- 1 root root 228056 Jan 21 16:41 ql2400_fw.bin
> -rw-r--r-- 1 root root 239052 Jan 21 16:41 ql2400_fw.bin_mid
> -rw-r--r-- 1 root root 200064 Jan 21 16:41 ql2500_fw.bin
> -rw-r--r-- 1 root root 219924 Jan 21 16:41 ql2500_fw.bin_mid
> 
> Thanks
> Babu Moger 
> 
> -----Original Message-----
> From: Seokmann Ju [mailto:seokmann.ju@qlogic.com]
> Sent: Wednesday, January 21, 2009 6:26 PM
> To: Moger, Babu
> Cc: linux-scsi@vger.kernel.org; device-mapper development
> Subject: Re: qla2xxx 0000:03:0c.0: Firmware image unavailable
> 
> 
> On Jan 21, 2009, at 4:00 PM, Moger, Babu wrote:
> 
> > Hi All,
> > Have you guys seen this following error?
> >
> > Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: 
> Firmware image  
> > unavailable.
> > Jan 21 16:30:44 txmpp2 kernel: qla2xxx 0000:03:0c.0: 
> Firmware images  
> > can be retrieved from: 
> ftp://ftp.qlogic.com/outgoing/linux/firmware/.
> >
> > After this error, qlogic driver fails to detect the LUNs attached.
> >
> > I am running 2.6.29-rc2 kernel on x86_64 machine.
> >
> > I tried couple of things without any success. I downloaded all the  
> > qlogic firmwares available and kept it in /lib/firmware. Tried  
> > rebuilding the kernel. Did not help. Thanks in advance.
> Could you double check that those firmware binaries under /lib/ 
> firmware have correct naming convention as defined in qla_os.c:  
> FW_FILE_ISP2XXX?
> 
> >
> >
> >
> > Thanks
> > Babu Moger
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux- 
> > scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Dave Holland]

On Thu, Jan 22, 2009 at 08:20:48AM -0700, Moger, Babu wrote:
> You are right. Looks like qlogic driver loads even before udev hotplug
> is functional.  Driver loads successfully when I re-load the driver
> after the system in up.  Is there any solution or work-around for this
> problem?

So the problem only happens at boot time? Are you using an initrd, and
if so, have you got the firmware images in the initrd image?
(mkinitramfs, update-initramfs, etc)

Dave
-- 
** Dave Holland ** Systems Support -- Infrastructure Management **
** 01223 496923 ** The Sanger Institute, Hinxton, Cambridge, UK **
"Your bumper is my crumple zone."


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

RE: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Moger, Babu]

Yes, firmware images are in the initrd. I have verified that. 

Thanks
Babu Moger 

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Dave Holland
Sent: Thursday, January 22, 2009 10:46 AM
To: device-mapper development
Cc: linux-scsi@vger.kernel.org
Subject: Re: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

On Thu, Jan 22, 2009 at 08:20:48AM -0700, Moger, Babu wrote:
> You are right. Looks like qlogic driver loads even before udev hotplug
> is functional.  Driver loads successfully when I re-load the driver
> after the system in up.  Is there any solution or work-around for this
> problem?

So the problem only happens at boot time? Are you using an initrd, and
if so, have you got the firmware images in the initrd image?
(mkinitramfs, update-initramfs, etc)

Dave
-- 
** Dave Holland ** Systems Support -- Infrastructure Management **
** 01223 496923 ** The Sanger Institute, Hinxton, Cambridge, UK **
"Your bumper is my crumple zone."


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Sudhakar Pannerselvam]

As James mentioned earlier, this is a distribution issue. I had seen
this issue with redhat. If you observe redhat distro qlogic drivers,
they compile firmware files along with the driver and hence they
wouldn't observe this issue when they load through initrd as there is no
firmware loading procedure through hotplug interface, whereas if you
compile qlogic driver by hand without inbuilt firmware files and using
distro's like redhat, you would encounter this issue. This has to be
addressed in nash hotplug code. 

One workaround that I could think of, as I mentioned above this is an
issue with nash which is active only when you load qlogic through initrd
mechanism. Check if you really need to compile your qlogic driver as
part of initrd image and recreate initrd image without qlogic module.
Later you could load the qlogic module(again using hotplug mechanism)
when kernel comes alive.

Regards
Sudhakar

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Moger, Babu
> Sent: Friday, January 23, 2009 5:46 AM
> To: device-mapper development
> Cc: linux-scsi@vger.kernel.org
> Subject: RE: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware 
> image unavailable
> 
> 
> Yes, firmware images are in the initrd. I have verified that. 
> 
> Thanks
> Babu Moger 
> 
> -----Original Message-----
> From: dm-devel-bounces@redhat.com 
> [mailto:dm-devel-bounces@redhat.com] On Behalf Of Dave Holland
> Sent: Thursday, January 22, 2009 10:46 AM
> To: device-mapper development
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware 
> image unavailable
> 
> On Thu, Jan 22, 2009 at 08:20:48AM -0700, Moger, Babu wrote:
> > You are right. Looks like qlogic driver loads even before 
> udev hotplug 
> > is functional.  Driver loads successfully when I re-load the driver 
> > after the system in up.  Is there any solution or 
> work-around for this 
> > problem?
> 
> So the problem only happens at boot time? Are you using an 
> initrd, and if so, have you got the firmware images in the 
> initrd image?
> (mkinitramfs, update-initramfs, etc)
> 
> Dave
> --
> ** Dave Holland ** Systems Support -- Infrastructure Management **
> ** 01223 496923 ** The Sanger Institute, Hinxton, Cambridge, 
> UK ** "Your bumper is my crumple zone."
> 
> 
> -- 
>  The Wellcome Trust Sanger Institute is operated by Genome Research 
>  Limited, a charity registered in England with number 1021457 and a 
>  company registered in England with number 2742969, whose registered 
>  office is 215 Euston Road, London, NW1 2BE. 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: [dm-devel] RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[James Bottomley]

On Thu, 2009-01-22 at 22:22 -0800, Sudhakar Pannerselvam wrote:
> As James mentioned earlier, this is a distribution issue. I had seen
> this issue with redhat. If you observe redhat distro qlogic drivers,
> they compile firmware files along with the driver and hence they
> wouldn't observe this issue when they load through initrd as there is no
> firmware loading procedure through hotplug interface, whereas if you
> compile qlogic driver by hand without inbuilt firmware files and using
> distro's like redhat, you would encounter this issue. This has to be
> addressed in nash hotplug code. 

OK, there is one little wrinkle with Red Hat:  nash is a sort of all in
one process and it keeps going wrong.  I've opened a few bugs with red
hat about nash over the years with hotplug issues, which this is, but
they always ignore them and then close them when the distro goes end of
life.

The way I fixed my RH nash problems was to boot the RH distro with a
debian created initrd ... which is a bit of a time consuming hack, but
it worked.

These problems should be going away when Dave Jones finally kills off
nash in his initrd/boot system rewrite.

James

Re: RE: qla2xxx 0000:03:0c.0: Firmware image unavailable

[Mike Snitzer]

On Fri, Jan 23, 2009 at 10:04 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Thu, 2009-01-22 at 22:22 -0800, Sudhakar Pannerselvam wrote:
>> As James mentioned earlier, this is a distribution issue. I had seen
>> this issue with redhat. If you observe redhat distro qlogic drivers,
>> they compile firmware files along with the driver and hence they
>> wouldn't observe this issue when they load through initrd as there is no
>> firmware loading procedure through hotplug interface, whereas if you
>> compile qlogic driver by hand without inbuilt firmware files and using
>> distro's like redhat, you would encounter this issue. This has to be
>> addressed in nash hotplug code.
>
> OK, there is one little wrinkle with Red Hat:  nash is a sort of all in
> one process and it keeps going wrong.  I've opened a few bugs with red
> hat about nash over the years with hotplug issues, which this is, but
> they always ignore them and then close them when the distro goes end of
> life.

FYI, RHEL5 had this specific nash issue; but RedHat fixed it, see:
https://bugzilla.redhat.com/show_bug.cgi?id=235277

> These problems should be going away when Dave Jones finally kills off
> nash in his initrd/boot system rewrite.

Cool, sounds like it will be a nice improvement.

Mike