RE: [PATCH 07/12] qla2xxx: Convert to host_lock less w/ interrupts disabled externally

Madhu Iyengar
2010-12-21T00:38:05Z

[-- Attachment #1: Type: text/plain, Size: 5830 bytes --]

All,

We had (internally @ QLogic) come up with a couple of patches (attached).

Patch 1: Similar to the patch from NAB and from Mathew Wilcox
Patch 2: A bit more changes with the host_lost within the qla2xxx driver.

The above 2 patches have gone through some testing as well, so far so good. Please take a look and let us know.

Cheers,
Madhu

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Nicholas A. Bellinger
Sent: Monday, December 20, 2010 1:24 AM
To: Matthew Wilcox
Cc: linux-scsi; linux-kernel; James Bottomley; Jeff Garzik; Christoph Hellwig; FUJITA Tomonori; Hannes Reinecke; Mike Christie; Mike Anderson; Tejun Heo; Vasu Dev; Tim Chen; Andi Kleen; Ravi Anand; Andrew Vasquez; Joe Eykholt; James Smart; Douglas Gilbert; adam radford; Kashyap Desai; MPTFusionLinux
Subject: Re: [PATCH 07/12] qla2xxx: Convert to host_lock less w/ interrupts disabled externally

On Sun, 2010-12-19 at 16:11 -0700, Matthew Wilcox wrote:
> On Sun, Dec 19, 2010 at 01:22:02PM -0800, Nicholas A. Bellinger wrote:
> > This patch converts qla2xxx to run in host_lock less mode with the new
> > IRQ_DISABLE_SCSI_QCMD() that disables interrupts while calling ->queuecommand()
> > dispatch. It also drops the legacy host_lock unlock optimization.
>
> I'm not sure this is the right direction to go. Now that Jeff's done
> the pushdown and put in the compatibility macros, I don't think it makes
> sense to do another partial transition on each driver. Much better to
> take our time, analyse each driver thoroughly, and kill the DEF_SCSI_QCMD
> in each driver without introducing IRQ_DISABLE_SCSI_QCMD.
>
> In particular for this driver, it explicitly re-enables interrupts,
> so it's pretty easy to do a full conversion. Compile-tested only.
>
> Convert qla2xxx driver to run without the shost lock
>
> Signed-off-by: Matthew Wilcox <willy@linux.intel.com>

Ok, this patch is functioning with LTP disktest I/O using .37-rc6
initiator side lock_less operation.

This has been added as an incremental patch for lock_less-LLDs-for-38-v3
here:

commit 355798dc3d97a0f07b14d1f3891ecca1802c9094
Author: Nicholas Bellinger <nab@linux-iscsi.org>
Date: Mon Dec 20 01:24:12 2010 -0800

qla2xxx: Convert qla2xxx driver to run without the shost lock

This patch converts qla2xxx LLD code to run in fully lock-less mode
and removes IRQ_DISABLE_SCSI_QCMD usage. This also includes a handful
of cleanups for cmd->scsi_done assignment removal from qla2x00_get_new_sp()
and renames one goto of the exception patch with the '_lock' suffix in
qla2xxx_queuecommand().

So far this patch has been lightly tested with basic I/O functionality on
bare-metal x86_64 .37-rc6 using ISP-2532 8 Gb/sec HW.

Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
Signed-off-by: Nicholas A. Bellinger <nab@linux-iscsi.org>

Thanks!

--nab

>
> diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
> index 2c0876c..b44d986 100644
> --- a/drivers/scsi/qla2xxx/qla_os.c
> +++ b/drivers/scsi/qla2xxx/qla_os.c
> @@ -513,7 +513,7 @@ qla24xx_fw_version_str(struct scsi_qla_host *vha, char *str)
>
> static inline srb_t *
> qla2x00_get_new_sp(scsi_qla_host_t *vha, fc_port_t *fcport,
> - struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
> + struct scsi_cmnd *cmd)
> {
> srb_t *sp;
> struct qla_hw_data *ha = vha->hw;
> @@ -527,16 +527,15 @@ qla2x00_get_new_sp(scsi_qla_host_t *vha, fc_port_t *fcport,
> sp->cmd = cmd;
> sp->flags = 0;
> CMD_SP(cmd) = (void *)sp;
> - cmd->scsi_done = done;
> sp->ctx = NULL;
>
> return sp;
> }
>
> static int
> -qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
> +qla2xxx_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *cmd)
> {
> - scsi_qla_host_t *vha = shost_priv(cmd->device->host);
> + scsi_qla_host_t *vha = shost_priv(shost);
> fc_port_t *fcport = (struct fc_port *) cmd->device->hostdata;
> struct fc_rport *rport = starget_to_rport(scsi_target(cmd->device));
> struct qla_hw_data *ha = vha->hw;
> @@ -544,7 +543,6 @@ qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *)
> srb_t *sp;
> int rval;
>
> - spin_unlock_irq(vha->host->host_lock);
> if (ha->flags.eeh_busy) {
> if (ha->flags.pci_channel_io_perm_failure)
> cmd->result = DID_NO_CONNECT << 16;
> @@ -577,39 +575,32 @@ qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *)
> goto qc24_target_busy;
> }
>
> - sp = qla2x00_get_new_sp(base_vha, fcport, cmd, done);
> + sp = qla2x00_get_new_sp(base_vha, fcport, cmd);
> if (!sp)
> - goto qc24_host_busy_lock;
> + goto qc24_host_busy;
>
> rval = ha->isp_ops->start_scsi(sp);
> if (rval != QLA_SUCCESS)
> goto qc24_host_busy_free_sp;
>
> - spin_lock_irq(vha->host->host_lock);
> -
> return 0;
>
> qc24_host_busy_free_sp:
> qla2x00_sp_free_dma(sp);
> mempool_free(sp, ha->srb_mempool);
>
> -qc24_host_busy_lock:
> - spin_lock_irq(vha->host->host_lock);
> +qc24_host_busy:
> return SCSI_MLQUEUE_HOST_BUSY;
>
> qc24_target_busy:
> - spin_lock_irq(vha->host->host_lock);
> return SCSI_MLQUEUE_TARGET_BUSY;
>
> qc24_fail_command:
> - spin_lock_irq(vha->host->host_lock);
> - done(cmd);
> + cmd->scsi_done(cmd);
>
> return 0;
> }
>
> -static DEF_SCSI_QCMD(qla2xxx_queuecommand)
> -
>
> /*
> * qla2x00_eh_wait_on_command
>

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: qla2xxx-Remove-host_lock-in-queuecommand-function.patch --]
[-- Type: application/octet-stream, Size: 2854 bytes --]

From: Madhuranath Iyengar <Madhu.Iyengar@qlogic.com>
Date: Mon, 20 Dec 2010 22:42:40 +0000 (-0800)
Subject: qla2xxx: Remove host_lock in queuecommand function
X-Git-Url: http://gitlnx/git?p=miyengar%2Fqla2xxx-upstream.git;a=commitdiff_plain;h=559013849273c0200c472b13c1f7e798027e83b5

qla2xxx: Remove host_lock in queuecommand function

Also in qla_os.c, rename the function, remove DEF_SCSI_QCMD, etc.

Signed-off-by: Madhuranath Iyengar <Madhu.Iyengar@qlogic.com>
---

diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index 2c0876c..e6bd4ae 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -513,7 +513,7 @@ qla24xx_fw_version_str(struct scsi_qla_host *vha, char *str)

static inline srb_t *
qla2x00_get_new_sp(scsi_qla_host_t *vha, fc_port_t *fcport,
- struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+ struct scsi_cmnd *cmd)
{
srb_t *sp;
struct qla_hw_data *ha = vha->hw;
@@ -527,14 +527,13 @@ qla2x00_get_new_sp(scsi_qla_host_t *vha, fc_port_t *fcport,
sp->cmd = cmd;
sp->flags = 0;
CMD_SP(cmd) = (void *)sp;
- cmd->scsi_done = done;
sp->ctx = NULL;

return sp;
}

static int
-qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+qla2xxx_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
{
scsi_qla_host_t *vha = shost_priv(cmd->device->host);
fc_port_t *fcport = (struct fc_port *) cmd->device->hostdata;
@@ -544,7 +543,6 @@ qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *)
srb_t *sp;
int rval;

- spin_unlock_irq(vha->host->host_lock);
if (ha->flags.eeh_busy) {
if (ha->flags.pci_channel_io_perm_failure)
cmd->result = DID_NO_CONNECT << 16;
@@ -577,40 +575,32 @@ qla2xxx_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *)
goto qc24_target_busy;
}

- sp = qla2x00_get_new_sp(base_vha, fcport, cmd, done);
+ sp = qla2x00_get_new_sp(base_vha, fcport, cmd);
if (!sp)
- goto qc24_host_busy_lock;
+ goto qc24_host_busy;

rval = ha->isp_ops->start_scsi(sp);
if (rval != QLA_SUCCESS)
goto qc24_host_busy_free_sp;

- spin_lock_irq(vha->host->host_lock);
-
return 0;

qc24_host_busy_free_sp:
qla2x00_sp_free_dma(sp);
mempool_free(sp, ha->srb_mempool);

-qc24_host_busy_lock:
- spin_lock_irq(vha->host->host_lock);
+qc24_host_busy:
return SCSI_MLQUEUE_HOST_BUSY;

qc24_target_busy:
- spin_lock_irq(vha->host->host_lock);
return SCSI_MLQUEUE_TARGET_BUSY;

qc24_fail_command:
- spin_lock_irq(vha->host->host_lock);
- done(cmd);
+ cmd->scsi_done(cmd);

return 0;
}

-static DEF_SCSI_QCMD(qla2xxx_queuecommand)
-
-
/*
* qla2x00_eh_wait_on_command
* Waits for the command to be returned by the Firmware for some

[-- Attachment #3: qla2xxx-Change-from-irq-to-irqsave-with-host_lock.txt --]
[-- Type: text/plain, Size: 3741 bytes --]

From: Madhuranath Iyengar <Madhu.Iyengar@qlogic.com>
Date: Tue, 7 Dec 2010 23:34:59 +0000 (-0800)
Subject: qla2xxx: Change from irq to irqsave with host_lock
X-Git-Url: http://gitlnx/git?p=miyengar%2Fqla2xxx-upstream.git;a=commitdiff_plain;h=cfdfbb52ef299abcb702fc7395f321556fd1cab0

qla2xxx: Change from irq to irqsave with host_lock

Make the driver safer by using irqsave/irqrestore with host_lock.

Signed-off-by: Madhuranath Iyengar <Madhu.Iyengar@qlogic.com>
---

diff --git a/drivers/scsi/qla2xxx/qla_attr.c b/drivers/scsi/qla2xxx/qla_attr.c
index bc8194f..30f9231 100644
--- a/drivers/scsi/qla2xxx/qla_attr.c
+++ b/drivers/scsi/qla2xxx/qla_attr.c
@@ -1534,6 +1534,7 @@ qla2x00_dev_loss_tmo_callbk(struct fc_rport *rport)
{
struct Scsi_Host *host = rport_to_shost(rport);
fc_port_t *fcport = *(fc_port_t **)rport->dd_data;
+ unsigned long flags;

if (!fcport)
return;
@@ -1546,10 +1547,10 @@ qla2x00_dev_loss_tmo_callbk(struct fc_rport *rport)
* Transport has effectively 'deleted' the rport, clear
* all local references.
*/
- spin_lock_irq(host->host_lock);
+ spin_lock_irqsave(host->host_lock, flags);
fcport->rport = fcport->drport = NULL;
*((fc_port_t **)rport->dd_data) = NULL;
- spin_unlock_irq(host->host_lock);
+ spin_unlock_irqrestore(host->host_lock, flags);

if (test_bit(ABORT_ISP_ACTIVE, &fcport->vha->dpc_flags))
return;
diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c
index 259f511..dfed504 100644
--- a/drivers/scsi/qla2xxx/qla_init.c
+++ b/drivers/scsi/qla2xxx/qla_init.c
@@ -2503,11 +2503,12 @@ qla2x00_rport_del(void *data)
{
fc_port_t *fcport = data;
struct fc_rport *rport;
+ unsigned long flags;

- spin_lock_irq(fcport->vha->host->host_lock);
+ spin_lock_irqsave(fcport->vha->host->host_lock, flags);
rport = fcport->drport ? fcport->drport: fcport->rport;
fcport->drport = NULL;
- spin_unlock_irq(fcport->vha->host->host_lock);
+ spin_unlock_irqrestore(fcport->vha->host->host_lock, flags);
if (rport)
fc_remote_port_delete(rport);
}
@@ -2877,6 +2878,7 @@ qla2x00_reg_remote_port(scsi_qla_host_t *vha, fc_port_t *fcport)
struct fc_rport_identifiers rport_ids;
struct fc_rport *rport;
struct qla_hw_data *ha = vha->hw;
+ unsigned long flags;

qla2x00_rport_del(fcport);

@@ -2891,9 +2893,9 @@ qla2x00_reg_remote_port(scsi_qla_host_t *vha, fc_port_t *fcport)
"Unable to allocate fc remote port!\n");
return;
}
- spin_lock_irq(fcport->vha->host->host_lock);
+ spin_lock_irqsave(fcport->vha->host->host_lock, flags);
*((fc_port_t **)rport->dd_data) = fcport;
- spin_unlock_irq(fcport->vha->host->host_lock);
+ spin_unlock_irqrestore(fcport->vha->host->host_lock, flags);

rport->supported_classes = fcport->supported_classes;

diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index e6bd4ae..3b441c8 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -2510,6 +2510,7 @@ qla2x00_schedule_rport_del(struct scsi_qla_host *vha, fc_port_t *fcport,
{
struct fc_rport *rport;
scsi_qla_host_t *base_vha;
+ unsigned long flags;

if (!fcport->rport)
return;
@@ -2517,9 +2518,9 @@ qla2x00_schedule_rport_del(struct scsi_qla_host *vha, fc_port_t *fcport,
rport = fcport->rport;
if (defer) {
base_vha = pci_get_drvdata(vha->hw->pdev);
- spin_lock_irq(vha->host->host_lock);
+ spin_lock_irqsave(vha->host->host_lock, flags);
fcport->drport = rport;
- spin_unlock_irq(vha->host->host_lock);
+ spin_unlock_irqrestore(vha->host->host_lock, flags);
set_bit(FCPORT_UPDATE_NEEDED, &base_vha->dpc_flags);
qla2xxx_wake_dpc(base_vha);
} else