RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Long Li]

> -----Original Message-----
> From: Bart Van Assche [mailto:Bart.VanAssche@sandisk.com]
> Sent: Monday, May 15, 2017 4:03 PM
> To: jejb@linux.vnet.ibm.com; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org; Long Li <longli@microsoft.com>;
> martin.petersen@oracle.com
> Cc: Long Li <longli@microsoft.com>
> Subject: Re: [Possible Phish Fraud][PATCH] scsi: zero per-cmd driver data for
> each MQ I/O
> 
> On Wed, 2017-05-10 at 14:07 -0700, Long Li wrote:
> > From: Long Li <longli@microsoft.com>
> >
> > Lower layer driver may not initialize private data before use. Zero
> > them out to prevent use of stale data.
> >
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> >  drivers/scsi/scsi_lib.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index
> > 19125d7..a821593 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -1850,7 +1850,7 @@ static int scsi_mq_prep_fn(struct request *req)
> >
> >  	/* zero out the cmd, except for the embedded scsi_request */
> >  	memset((char *)cmd + sizeof(cmd->req), 0,
> > -		sizeof(*cmd) - sizeof(cmd->req));
> > +		sizeof(*cmd) - sizeof(cmd->req) + shost->hostt->cmd_size);
> >
> >  	req->special = cmd;
> 
> Hello Long,
> 
> Sorry but this patch looks wrong to me. Since scsi_mq_prep_fn() is called
> after scsi_req_init(), erasing struct scsi_request from scsi_mq_prep_fn() will
> erase the values that were set by scsi_req_init(). That includes information
> like the pointer to the SCSI CDB and the CDB itself. See e.g.
> scsi_execute().
> 
> Did you come up with this patch after source reading or did you come up with
> this patch while chasing a bug?

Thanks for looking! Yes this is for chasing a bug. 

Actually scsi_mq_prep_fn() doesn't touch cmd->req (which is the struct scsi_request mentioned in your email).

With the patch, we also zero the private data used by lower layer driver, in addition to the private data in scsi_cmnd.

> 
> Thanks,
> 
> Bart.

Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Bart Van Assche]

On Mon, 2017-05-15 at 23:32 +0000, Long Li wrote:
> Thanks for looking! Yes this is for chasing a bug. 
> 
> With the patch, we also zero the private data used by lower layer driver, in
> addition to the private data in scsi_cmnd.

Hello Long,

What bug did you encounter, with which combination of ULP (sd?) and LLD SCSI
driver(s) and for which request type (REQ_OP_*)? You will have to mention that
information in the patch description anyway if you want your patch to get
accepted.

If the bug that you encountered only occurs with a single LLD, would it be
possible to implement a fix by modifying the LLD instead of the SCSI core?

Thanks,

Bart.

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Long Li]

> -----Original Message-----
> From: Bart Van Assche [mailto:Bart.VanAssche@sandisk.com]
> Sent: Monday, May 15, 2017 5:01 PM
> To: jejb@linux.vnet.ibm.com; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org; Long Li <longli@microsoft.com>;
> martin.petersen@oracle.com
> Cc: Stephen Hemminger <sthemmin@microsoft.com>; KY Srinivasan
> <kys@microsoft.com>
> Subject: Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> 
> On Mon, 2017-05-15 at 23:32 +0000, Long Li wrote:
> > Thanks for looking! Yes this is for chasing a bug.
> >
> > With the patch, we also zero the private data used by lower layer
> > driver, in addition to the private data in scsi_cmnd.
> 
> Hello Long,
> 
> What bug did you encounter, with which combination of ULP (sd?) and LLD SCSI
> driver(s) and for which request type (REQ_OP_*)? You will have to mention
> that information in the patch description anyway if you want your patch to get
> accepted.
> 
> If the bug that you encountered only occurs with a single LLD, would it be
> possible to implement a fix by modifying the LLD instead of the SCSI core?

The bug I encounter is that in hv_storvsc (a LLD), sometime we are getting stale data in the private driver data memory allocated by SCSI. As a LLD, we expect the memory allocated by SCSI to be zeroed. If not we may send unexpected commands to the device.

A little background on private data: In LLD's scsi_host_template, the driver may optionally ask SCSI to allocate its private driver memory for each command, by specifying cmd_size. This memory is allocated at the end of scsi_cmnd by SCSI. Later when SCSI queues a command, the LLD can use scsi_cmd_priv to get its private data.

hv_storvsc doesn't clear its private data before use. I'm not sure about other LLD drivers. Although it's possible to fix it in LLD not SCSI core, I think that is not the ideal place to do it. Whoever is allocating the SCSI command should also zero it.

There is a similar patch that fixed a similar issue for non-MQ case:
commit ee5242360424b9b967454e9183767323d10cf985

I'm sorry I should have put more details in the patch. 

> 
> Thanks,
> 
> Bart.

Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Bart Van Assche]

On Tue, 2017-05-16 at 17:31 +0000, Long Li wrote:
> > -----Original Message-----
> > From: Bart Van Assche [mailto:Bart.VanAssche@sandisk.com]
> > Sent: Monday, May 15, 2017 5:01 PM
> > To: jejb@linux.vnet.ibm.com; linux-scsi@vger.kernel.org; linux-
> > kernel@vger.kernel.org; Long Li <longli@microsoft.com>;
> > martin.petersen@oracle.com
> > Cc: Stephen Hemminger <sthemmin@microsoft.com>; KY Srinivasan
> > <kys@microsoft.com>
> > Subject: Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> > 
> > On Mon, 2017-05-15 at 23:32 +0000, Long Li wrote:
> > > Thanks for looking! Yes this is for chasing a bug.
> > > 
> > > With the patch, we also zero the private data used by lower layer
> > > driver, in addition to the private data in scsi_cmnd.
> > 
> > Hello Long,
> > 
> > What bug did you encounter, with which combination of ULP (sd?) and LLD SCSI
> > driver(s) and for which request type (REQ_OP_*)? You will have to mention
> > that information in the patch description anyway if you want your patch to get
> > accepted.
> > 
> > If the bug that you encountered only occurs with a single LLD, would it be
> > possible to implement a fix by modifying the LLD instead of the SCSI core?
> 
> The bug I encounter is that in hv_storvsc (a LLD), sometime we are getting stale data in the private driver data memory allocated by SCSI. As a LLD, we expect the memory allocated by SCSI to be zeroed. If not we may send unexpected commands to the device.
> 
> A little background on private data: In LLD's scsi_host_template, the driver may optionally ask SCSI to allocate its private driver memory for each command, by specifying cmd_size. This memory is allocated at the end of scsi_cmnd by SCSI. Later when SCSI queues a command, the LLD can use scsi_cmd_priv to get its private data.
> 
> hv_storvsc doesn't clear its private data before use. I'm not sure about other LLD drivers. Although it's possible to fix it in LLD not SCSI core, I think that is not the ideal place to do it. Whoever is allocating the SCSI command should also zero it.
> 
> There is a similar patch that fixed a similar issue for non-MQ case:
> commit ee5242360424b9b967454e9183767323d10cf985
> 
> I'm sorry I should have put more details in the patch. 

Hello Long,

Thank you for the feedback. I'm working on a patch series that merges the scsi-sq
and scsi-mq code paths for command initialization and that should fix the bug you
encountered.

Bart.

Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Christoph Hellwig]

On Wed, May 17, 2017 at 11:05:18PM +0000, Bart Van Assche wrote:
> Thank you for the feedback. I'm working on a patch series that merges the scsi-sq
> and scsi-mq code paths for command initialization and that should fix the bug you
> encountered.

While that sounds great (I tried it a while ago but gave up due to
priorities) I think we should merge this patch as-is and backport it
to stable for now and rebase your series on top of it.

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[KY Srinivasan]

> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@infradead.org]
> Sent: Wednesday, May 17, 2017 11:55 PM
> To: Bart Van Assche <Bart.VanAssche@sandisk.com>
> Cc: jejb@linux.vnet.ibm.com; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org; Long Li <longli@microsoft.com>;
> martin.petersen@oracle.com; Stephen Hemminger
> <sthemmin@microsoft.com>; KY Srinivasan <kys@microsoft.com>
> Subject: Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> 
> On Wed, May 17, 2017 at 11:05:18PM +0000, Bart Van Assche wrote:
> > Thank you for the feedback. I'm working on a patch series that merges the
> scsi-sq
> > and scsi-mq code paths for command initialization and that should fix the
> bug you
> > encountered.
> 
> While that sounds great (I tried it a while ago but gave up due to
> priorities) I think we should merge this patch as-is and backport it
> to stable for now and rebase your series on top of it.
Thanks Christoph.

K. Y

Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Bart Van Assche]

On Wed, 2017-05-17 at 23:54 -0700, Christoph Hellwig wrote:
> On Wed, May 17, 2017 at 11:05:18PM +0000, Bart Van Assche wrote:
> > Thank you for the feedback. I'm working on a patch series that merges the scsi-sq
> > and scsi-mq code paths for command initialization and that should fix the bug you
> > encountered.
> 
> While that sounds great (I tried it a while ago but gave up due to
> priorities) I think we should merge this patch as-is and backport it
> to stable for now and rebase your series on top of it.

Hello Christoph,

I will rebase my patch series on top of Long Li's patch. Long, please
repost your patch with a more detailed description and with the Reviewed-by
and Cc: stable tags added.

Thanks,

Bart.

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Long Li]

> -----Original Message-----
> From: Bart Van Assche [mailto:Bart.VanAssche@sandisk.com]
> Sent: Thursday, May 18, 2017 8:52 AM
> To: hch@infradead.org
> Cc: jejb@linux.vnet.ibm.com; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org; Stephen Hemminger <sthemmin@microsoft.com>; KY
> Srinivasan <kys@microsoft.com>; Long Li <longli@microsoft.com>;
> martin.petersen@oracle.com
> Subject: Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> 
> On Wed, 2017-05-17 at 23:54 -0700, Christoph Hellwig wrote:
> > On Wed, May 17, 2017 at 11:05:18PM +0000, Bart Van Assche wrote:
> > > Thank you for the feedback. I'm working on a patch series that
> > > merges the scsi-sq and scsi-mq code paths for command initialization
> > > and that should fix the bug you encountered.
> >
> > While that sounds great (I tried it a while ago but gave up due to
> > priorities) I think we should merge this patch as-is and backport it
> > to stable for now and rebase your series on top of it.
> 
> Hello Christoph,
> 
> I will rebase my patch series on top of Long Li's patch. Long, please repost your
> patch with a more detailed description and with the Reviewed-by and Cc:
> stable tags added.

Will do that.

> 
> Thanks,
> 
> Bart.

[PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Long Li]

From: Long Li <longli@microsoft.com>

Lower layer driver may not initialize private data before use. Zero them
out to prevent use of stale data.

Signed-off-by: Long Li <longli@microsoft.com>
---
 drivers/scsi/scsi_lib.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 19125d7..a821593 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1850,7 +1850,7 @@ static int scsi_mq_prep_fn(struct request *req)
 
 	/* zero out the cmd, except for the embedded scsi_request */
 	memset((char *)cmd + sizeof(cmd->req), 0,
-		sizeof(*cmd) - sizeof(cmd->req));
+		sizeof(*cmd) - sizeof(cmd->req) + shost->hostt->cmd_size);
 
 	req->special = cmd;
 
-- 
2.7.4

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Long Li]

> -----Original Message-----
> From: Long Li [mailto:longli@exchange.microsoft.com]
> Sent: Wednesday, May 10, 2017 2:07 PM
> To: James E.J. Bottomley <jejb@linux.vnet.ibm.com>; Martin K. Petersen
> <martin.petersen@oracle.com>; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Cc: Long Li <longli@microsoft.com>
> Subject: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> 
> From: Long Li <longli@microsoft.com>
> 
> Lower layer driver may not initialize private data before use. Zero them out
> to prevent use of stale data.
> 
> Signed-off-by: Long Li <longli@microsoft.com>

Can someone help review this patch?

> ---
>  drivers/scsi/scsi_lib.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index
> 19125d7..a821593 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -1850,7 +1850,7 @@ static int scsi_mq_prep_fn(struct request *req)
> 
>         /* zero out the cmd, except for the embedded scsi_request */
>         memset((char *)cmd + sizeof(cmd->req), 0,
> -               sizeof(*cmd) - sizeof(cmd->req));
> +               sizeof(*cmd) - sizeof(cmd->req) +
> + shost->hostt->cmd_size);
> 
>         req->special = cmd;
> 
> --
> 2.7.4

RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[KY Srinivasan]

> -----Original Message-----
> From: Long Li
> Sent: Monday, May 15, 2017 3:48 PM
> To: Long Li <longli@microsoft.com>; James E.J. Bottomley
> <jejb@linux.vnet.ibm.com>; Martin K. Petersen
> <martin.petersen@oracle.com>; linux-scsi@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Cc: KY Srinivasan <kys@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>
> Subject: RE: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> 
> > -----Original Message-----
> > From: Long Li [mailto:longli@exchange.microsoft.com]
> > Sent: Wednesday, May 10, 2017 2:07 PM
> > To: James E.J. Bottomley <jejb@linux.vnet.ibm.com>; Martin K. Petersen
> > <martin.petersen@oracle.com>; linux-scsi@vger.kernel.org; linux-
> > kernel@vger.kernel.org
> > Cc: Long Li <longli@microsoft.com>
> > Subject: [PATCH] scsi: zero per-cmd driver data for each MQ I/O
> >
> > From: Long Li <longli@microsoft.com>
> >
> > Lower layer driver may not initialize private data before use. Zero them out
> > to prevent use of stale data.
> >
> > Signed-off-by: Long Li <longli@microsoft.com>
> 
> Can someone help review this patch?
Reviewed-by:  K. Y. Srinivasan <kys@microsoft.com>

> 
> > ---
> >  drivers/scsi/scsi_lib.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index
> > 19125d7..a821593 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -1850,7 +1850,7 @@ static int scsi_mq_prep_fn(struct request *req)
> >
> >         /* zero out the cmd, except for the embedded scsi_request */
> >         memset((char *)cmd + sizeof(cmd->req), 0,
> > -               sizeof(*cmd) - sizeof(cmd->req));
> > +               sizeof(*cmd) - sizeof(cmd->req) +
> > + shost->hostt->cmd_size);
> >
> >         req->special = cmd;
> >
> > --
> > 2.7.4

Re: [PATCH] scsi: zero per-cmd driver data for each MQ I/O

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>