* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-07 23:31 [PATCH] ibmvscsi: add write memory barrier to CRQ processing Tyrel Datwyler
@ 2016-12-08 9:06 ` Johannes Thumshirn
2016-12-09 2:56 ` Tyrel Datwyler
2016-12-08 14:21 ` Brian King
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Johannes Thumshirn @ 2016-12-08 9:06 UTC (permalink / raw)
To: Tyrel Datwyler
Cc: james.bottomley, martin.petersen, linux-scsi, linuxppc-dev,
brking, nfont
On Wed, Dec 07, 2016 at 05:31:26PM -0600, Tyrel Datwyler wrote:
> The first byte of each CRQ entry is used to indicate whether an entry is
> a valid response or free for the VIOS to use. After processing a
> response the driver sets the valid byte to zero to indicate the entry is
> now free to be reused. Add a memory barrier after this write to ensure
> no other stores are reordered when updating the valid byte.
>
> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
> ---
> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
> index d9534ee..2f5b07e 100644
> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
> while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
> ibmvscsi_handle_crq(crq, hostdata);
> crq->valid = VIOSRP_CRQ_FREE;
> + wmb();
> }
>
> vio_enable_interrupts(vdev);
> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
> vio_disable_interrupts(vdev);
> ibmvscsi_handle_crq(crq, hostdata);
> crq->valid = VIOSRP_CRQ_FREE;
> + wmb();
> } else {
> done = 1;
> }
Is this something you have seen in the wild or just a "better save than sorry"
barrier?
Thanks,
Johannes
--
Johannes Thumshirn Storage
jthumshirn@suse.de +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-08 9:06 ` Johannes Thumshirn
@ 2016-12-09 2:56 ` Tyrel Datwyler
0 siblings, 0 replies; 8+ messages in thread
From: Tyrel Datwyler @ 2016-12-09 2:56 UTC (permalink / raw)
To: Johannes Thumshirn, Tyrel Datwyler
Cc: james.bottomley, martin.petersen, linux-scsi, linuxppc-dev,
brking, nfont
On 12/08/2016 01:06 AM, Johannes Thumshirn wrote:
> On Wed, Dec 07, 2016 at 05:31:26PM -0600, Tyrel Datwyler wrote:
>> The first byte of each CRQ entry is used to indicate whether an entry is
>> a valid response or free for the VIOS to use. After processing a
>> response the driver sets the valid byte to zero to indicate the entry is
>> now free to be reused. Add a memory barrier after this write to ensure
>> no other stores are reordered when updating the valid byte.
>>
>> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
>> ---
>> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> index d9534ee..2f5b07e 100644
>> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
>> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
>> while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
>> ibmvscsi_handle_crq(crq, hostdata);
>> crq->valid = VIOSRP_CRQ_FREE;
>> + wmb();
>> }
>>
>> vio_enable_interrupts(vdev);
>> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
>> vio_disable_interrupts(vdev);
>> ibmvscsi_handle_crq(crq, hostdata);
>> crq->valid = VIOSRP_CRQ_FREE;
>> + wmb();
>> } else {
>> done = 1;
>> }
>
> Is this something you have seen in the wild or just a "better save than sorry"
> barrier?
I myself have not observed or heard of anybody hitting an issue here.
However, based on conversation with the VIOS developers, who have
indicated it is required, this is a "better safe than sorry" scenario.
Further, it matches what we already do in the ibmvfc driver for the CRQ
processing logic.
-Tyrel
>
> Thanks,
> Johannes
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-07 23:31 [PATCH] ibmvscsi: add write memory barrier to CRQ processing Tyrel Datwyler
2016-12-08 9:06 ` Johannes Thumshirn
@ 2016-12-08 14:21 ` Brian King
2016-12-08 23:29 ` Paolo Bonzini
2016-12-09 21:20 ` Benjamin Herrenschmidt
3 siblings, 0 replies; 8+ messages in thread
From: Brian King @ 2016-12-08 14:21 UTC (permalink / raw)
To: Tyrel Datwyler, james.bottomley
Cc: martin.petersen, linux-scsi, linuxppc-dev, nfont
Reviewed-by: Brian King <brking@linux.vnet.ibm.com>
--
Brian King
Power Linux I/O
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-07 23:31 [PATCH] ibmvscsi: add write memory barrier to CRQ processing Tyrel Datwyler
2016-12-08 9:06 ` Johannes Thumshirn
2016-12-08 14:21 ` Brian King
@ 2016-12-08 23:29 ` Paolo Bonzini
2016-12-09 2:52 ` Tyrel Datwyler
2016-12-09 21:20 ` Benjamin Herrenschmidt
3 siblings, 1 reply; 8+ messages in thread
From: Paolo Bonzini @ 2016-12-08 23:29 UTC (permalink / raw)
To: Tyrel Datwyler, james.bottomley
Cc: martin.petersen, linux-scsi, linuxppc-dev, brking, nfont
On 08/12/2016 00:31, Tyrel Datwyler wrote:
> The first byte of each CRQ entry is used to indicate whether an entry is
> a valid response or free for the VIOS to use. After processing a
> response the driver sets the valid byte to zero to indicate the entry is
> now free to be reused. Add a memory barrier after this write to ensure
> no other stores are reordered when updating the valid byte.
>
> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
> ---
> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
> index d9534ee..2f5b07e 100644
> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
> while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
> ibmvscsi_handle_crq(crq, hostdata);
> crq->valid = VIOSRP_CRQ_FREE;
> + wmb();
> }
>
> vio_enable_interrupts(vdev);
> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
> vio_disable_interrupts(vdev);
> ibmvscsi_handle_crq(crq, hostdata);
> crq->valid = VIOSRP_CRQ_FREE;
> + wmb();
Should this driver use virt_wmb instead?
Paolo
> } else {
> done = 1;
> }
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-08 23:29 ` Paolo Bonzini
@ 2016-12-09 2:52 ` Tyrel Datwyler
0 siblings, 0 replies; 8+ messages in thread
From: Tyrel Datwyler @ 2016-12-09 2:52 UTC (permalink / raw)
To: Paolo Bonzini, Tyrel Datwyler, james.bottomley
Cc: brking, nfont, linuxppc-dev, linux-scsi, martin.petersen
On 12/08/2016 03:29 PM, Paolo Bonzini wrote:
>
>
> On 08/12/2016 00:31, Tyrel Datwyler wrote:
>> The first byte of each CRQ entry is used to indicate whether an entry is
>> a valid response or free for the VIOS to use. After processing a
>> response the driver sets the valid byte to zero to indicate the entry is
>> now free to be reused. Add a memory barrier after this write to ensure
>> no other stores are reordered when updating the valid byte.
>>
>> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
>> ---
>> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> index d9534ee..2f5b07e 100644
>> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
>> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
>> while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
>> ibmvscsi_handle_crq(crq, hostdata);
>> crq->valid = VIOSRP_CRQ_FREE;
>> + wmb();
>> }
>>
>> vio_enable_interrupts(vdev);
>> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
>> vio_disable_interrupts(vdev);
>> ibmvscsi_handle_crq(crq, hostdata);
>> crq->valid = VIOSRP_CRQ_FREE;
>> + wmb();
>
> Should this driver use virt_wmb instead?
Both virt_wmb and wmb reduce to a lwsync instruction under PowerPC.
-Tyrel
>
> Paolo
>
>> } else {
>> done = 1;
>> }
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-07 23:31 [PATCH] ibmvscsi: add write memory barrier to CRQ processing Tyrel Datwyler
` (2 preceding siblings ...)
2016-12-08 23:29 ` Paolo Bonzini
@ 2016-12-09 21:20 ` Benjamin Herrenschmidt
2016-12-21 17:35 ` Tyrel Datwyler
3 siblings, 1 reply; 8+ messages in thread
From: Benjamin Herrenschmidt @ 2016-12-09 21:20 UTC (permalink / raw)
To: Tyrel Datwyler, james.bottomley
Cc: martin.petersen, linux-scsi, brking, nfont, linuxppc-dev
On Wed, 2016-12-07 at 17:31 -0600, Tyrel Datwyler wrote:
> The first byte of each CRQ entry is used to indicate whether an entry is
> a valid response or free for the VIOS to use. After processing a
> response the driver sets the valid byte to zero to indicate the entry is
> now free to be reused. Add a memory barrier after this write to ensure
> no other stores are reordered when updating the valid byte.
Which "other stores" specifically ? This smells fishy without that
precision. It's important to always understand what exactly barriers
order with.
Cheers,
Ben.
> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
> ---
> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
> index d9534ee..2f5b07e 100644
> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
> > while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
> > ibmvscsi_handle_crq(crq, hostdata);
> > crq->valid = VIOSRP_CRQ_FREE;
> > + wmb();
> > }
>
> > vio_enable_interrupts(vdev);
> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
> > vio_disable_interrupts(vdev);
> > ibmvscsi_handle_crq(crq, hostdata);
> > crq->valid = VIOSRP_CRQ_FREE;
> > + wmb();
> > } else {
> > done = 1;
> > }
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] ibmvscsi: add write memory barrier to CRQ processing
2016-12-09 21:20 ` Benjamin Herrenschmidt
@ 2016-12-21 17:35 ` Tyrel Datwyler
0 siblings, 0 replies; 8+ messages in thread
From: Tyrel Datwyler @ 2016-12-21 17:35 UTC (permalink / raw)
To: benh, Tyrel Datwyler, james.bottomley
Cc: martin.petersen, linux-scsi, brking, nfont, linuxppc-dev
On 12/09/2016 01:20 PM, Benjamin Herrenschmidt wrote:
> On Wed, 2016-12-07 at 17:31 -0600, Tyrel Datwyler wrote:
>> The first byte of each CRQ entry is used to indicate whether an entry is
>> a valid response or free for the VIOS to use. After processing a
>> response the driver sets the valid byte to zero to indicate the entry is
>> now free to be reused. Add a memory barrier after this write to ensure
>> no other stores are reordered when updating the valid byte.
>
> Which "other stores" specifically ? This smells fishy without that
> precision. It's important to always understand what exactly barriers
> order with.
So, this patch initially came about while chasing a data integrity issue
based on the observation that we were already doing this same write
barrier in the virtual fibre channel driver.
However, the more I stare at it I agree it does seem fishy and I can't
see any other stores that we need to order with here. In terms of the
16byte CRQ entries we only ever write to the first byte as a sort of
doorbell to tell the VIOS that we have processed the data and the entry
is free to be re-used. The remainder of the CRQ is only ever read from.
The wmb() in the ibmvfc driver was part of a commit from Brian that also
involved a necessary rmb() that prevented stale data from load
re-ordering by ensuring that a read from the first byte completed and
contained a valid value prior to doing any other reads of the CRQ entry.
I'd have to defer to Brian as to whether he remembers a legitimate
reason for the wmb().
-Tyrel
>
> Cheers,
> Ben.
>
>> Signed-off-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
>> ---
>> drivers/scsi/ibmvscsi/ibmvscsi.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> index d9534ee..2f5b07e 100644
>> --- a/drivers/scsi/ibmvscsi/ibmvscsi.c
>> +++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
>> @@ -232,6 +232,7 @@ static void ibmvscsi_task(void *data)
>>> while ((crq = crq_queue_next_crq(&hostdata->queue)) != NULL) {
>>> ibmvscsi_handle_crq(crq, hostdata);
>>> crq->valid = VIOSRP_CRQ_FREE;
>>> + wmb();
>>> }
>>
>>> vio_enable_interrupts(vdev);
>> @@ -240,6 +241,7 @@ static void ibmvscsi_task(void *data)
>>> vio_disable_interrupts(vdev);
>>> ibmvscsi_handle_crq(crq, hostdata);
>>> crq->valid = VIOSRP_CRQ_FREE;
>>> + wmb();
>>> } else {
>>> done = 1;
>>> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 8+ messages in thread