[PATCH RFC v2 0/2] libibverbs memory barrier fixes

[PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Steve Wise]

This is v2 of the ARM64 mb* support, plus an additional patch to fail
compiles if there is no platform implementation of the memory barriers.
I've included the 2nd patch because I think it is important to not
assume any default barrier implementation.  Getting it wrong can result
in data corruption.

Changes since V1:

Put the RFC tag back on because I want to retest this series.

Implemented the ARM64 memory barrier macros from scratch using the AMD
reference docs.

Added 2nd patch to fail compiles if no mb* macros exist for the platform.

Steve Wise (2):
  libibverbs: add ARM64 memory barrier macros
  Fail compiles if no platform specific memory barriers exist

 include/infiniband/arch.h |   15 +++++++++------
 1 files changed, 9 insertions(+), 6 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On 05/20/2016 03:43 PM, Steve Wise wrote:
> This is v2 of the ARM64 mb* support, plus an additional patch to fail
> compiles if there is no platform implementation of the memory barriers.
> I've included the 2nd patch because I think it is important to not
> assume any default barrier implementation.  Getting it wrong can result
> in data corruption.
> 
> Changes since V1:
> 
> Put the RFC tag back on because I want to retest this series.
> 
> Implemented the ARM64 memory barrier macros from scratch using the AMD
> reference docs.
> 
> Added 2nd patch to fail compiles if no mb* macros exist for the platform.
> 
> Steve Wise (2):
>   libibverbs: add ARM64 memory barrier macros
>   Fail compiles if no platform specific memory barriers exist
> 
>  include/infiniband/arch.h |   15 +++++++++------
>  1 files changed, 9 insertions(+), 6 deletions(-)
> 

Please remember to use --thread=shallow when sending patch emails using
git send-email (which is what I assume you did).

Let me know how the testing comes out.  The patches look fine to me
(thanks for adding the second one, it's the right thing to do).

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

RE: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Steve Wise]

> > This is v2 of the ARM64 mb* support, plus an additional patch to fail
> > compiles if there is no platform implementation of the memory barriers.
> > I've included the 2nd patch because I think it is important to not
> > assume any default barrier implementation.  Getting it wrong can result
> > in data corruption.
> >
> > Changes since V1:
> >
> > Put the RFC tag back on because I want to retest this series.
> >
> > Implemented the ARM64 memory barrier macros from scratch using the AMD
> > reference docs.
> >
> > Added 2nd patch to fail compiles if no mb* macros exist for the platform.
> >
> > Steve Wise (2):
> >   libibverbs: add ARM64 memory barrier macros
> >   Fail compiles if no platform specific memory barriers exist
> >
> >  include/infiniband/arch.h |   15 +++++++++------
> >  1 files changed, 9 insertions(+), 6 deletions(-)
> >
> 
> Please remember to use --thread=shallow when sending patch emails using
> git send-email (which is what I assume you did).
>

Will do.  I use git format-patch.  I added -t, but that apparently does nothing.
:)  I'll use --thread=shallow next time.

 
> Let me know how the testing comes out.  The patches look fine to me
> (thanks for adding the second one, it's the right thing to do).
>

It will be next week, and I'll send out a final version of the series then.  

Thanks!

Steve



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Noa Osherovich]

Hi,

We'll be testing those patches as well.

-
Thanks,
Noa Osherovich  |  SW Engineer  |  Tel: 074-7236018


-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Steve Wise
Sent: Friday, May 20, 2016 11:28 PM
To: 'Doug Ledford' <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: RE: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

> > This is v2 of the ARM64 mb* support, plus an additional patch to 
> > fail compiles if there is no platform implementation of the memory barriers.
> > I've included the 2nd patch because I think it is important to not 
> > assume any default barrier implementation.  Getting it wrong can 
> > result in data corruption.
> >
> > Changes since V1:
> >
> > Put the RFC tag back on because I want to retest this series.
> >
> > Implemented the ARM64 memory barrier macros from scratch using the 
> > AMD reference docs.
> >
> > Added 2nd patch to fail compiles if no mb* macros exist for the platform.
> >
> > Steve Wise (2):
> >   libibverbs: add ARM64 memory barrier macros
> >   Fail compiles if no platform specific memory barriers exist
> >
> >  include/infiniband/arch.h |   15 +++++++++------
> >  1 files changed, 9 insertions(+), 6 deletions(-)
> >
> 
> Please remember to use --thread=shallow when sending patch emails 
> using git send-email (which is what I assume you did).
>

Will do.  I use git format-patch.  I added -t, but that apparently does nothing.
:)  I'll use --thread=shallow next time.

 
> Let me know how the testing comes out.  The patches look fine to me 
> (thanks for adding the second one, it's the right thing to do).
>

It will be next week, and I'll send out a final version of the series then.  

Thanks!

Steve



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Steve Wise]

> On 05/20/2016 03:43 PM, Steve Wise wrote:
> > This is v2 of the ARM64 mb* support, plus an additional patch to fail
> > compiles if there is no platform implementation of the memory barriers.
> > I've included the 2nd patch because I think it is important to not
> > assume any default barrier implementation.  Getting it wrong can result
> > in data corruption.
> >
> > Changes since V1:
> >
> > Put the RFC tag back on because I want to retest this series.
> >
> > Implemented the ARM64 memory barrier macros from scratch using the AMD
> > reference docs.
> >
> > Added 2nd patch to fail compiles if no mb* macros exist for the platform.
> >
> > Steve Wise (2):
> >   libibverbs: add ARM64 memory barrier macros
> >   Fail compiles if no platform specific memory barriers exist
> >
> >  include/infiniband/arch.h |   15 +++++++++------
> >  1 files changed, 9 insertions(+), 6 deletions(-)
> >
> 
> Please remember to use --thread=shallow when sending patch emails using
> git send-email (which is what I assume you did).
> 
> Let me know how the testing comes out.  The patches look fine to me
> (thanks for adding the second one, it's the right thing to do).
> 

Hey Doug, the series tests good with cxgb4 on ARM64.  Do I need to repost a
final series, or can you merge v2? 

Hey Noa, have you all had a chance to test this series?

Thanks,

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 1491 bytes --]

On 5/25/2016 10:51 AM, Steve Wise wrote:
>> On 05/20/2016 03:43 PM, Steve Wise wrote:
>>> This is v2 of the ARM64 mb* support, plus an additional patch to fail
>>> compiles if there is no platform implementation of the memory barriers.
>>> I've included the 2nd patch because I think it is important to not
>>> assume any default barrier implementation.  Getting it wrong can result
>>> in data corruption.
>>>
>>> Changes since V1:
>>>
>>> Put the RFC tag back on because I want to retest this series.
>>>
>>> Implemented the ARM64 memory barrier macros from scratch using the AMD
>>> reference docs.
>>>
>>> Added 2nd patch to fail compiles if no mb* macros exist for the platform.
>>>
>>> Steve Wise (2):
>>>   libibverbs: add ARM64 memory barrier macros
>>>   Fail compiles if no platform specific memory barriers exist
>>>
>>>  include/infiniband/arch.h |   15 +++++++++------
>>>  1 files changed, 9 insertions(+), 6 deletions(-)
>>>
>>
>> Please remember to use --thread=shallow when sending patch emails using
>> git send-email (which is what I assume you did).
>>
>> Let me know how the testing comes out.  The patches look fine to me
>> (thanks for adding the second one, it's the right thing to do).
>>
> 
> Hey Doug, the series tests good with cxgb4 on ARM64.  Do I need to repost a
> final series, or can you merge v2? 
> 
> Hey Noa, have you all had a chance to test this series?
> 
> Thanks,
> 
> Steve.
> 

Repost a final please.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Noa Osherovich]

On 5/25/2016 5:51 PM, Steve Wise wrote:
>> On 05/20/2016 03:43 PM, Steve Wise wrote:
>>> This is v2 of the ARM64 mb* support, plus an additional patch to fail
>>> compiles if there is no platform implementation of the memory barriers.
>>> I've included the 2nd patch because I think it is important to not
>>> assume any default barrier implementation.  Getting it wrong can result
>>> in data corruption.
>>>
>>> Changes since V1:
>>>
>>> Put the RFC tag back on because I want to retest this series.
>>>
>>> Implemented the ARM64 memory barrier macros from scratch using the AMD
>>> reference docs.
>>>
>>> Added 2nd patch to fail compiles if no mb* macros exist for the platform.
>>>
>>> Steve Wise (2):
>>>   libibverbs: add ARM64 memory barrier macros
>>>   Fail compiles if no platform specific memory barriers exist
>>>
>>>  include/infiniband/arch.h |   15 +++++++++------
>>>  1 files changed, 9 insertions(+), 6 deletions(-)
>>>
>> Please remember to use --thread=shallow when sending patch emails using
>> git send-email (which is what I assume you did).
>>
>> Let me know how the testing comes out.  The patches look fine to me
>> (thanks for adding the second one, it's the right thing to do).
>>
> Hey Doug, the series tests good with cxgb4 on ARM64.  Do I need to repost a
> final series, or can you merge v2? 
>
> Hey Noa, have you all had a chance to test this series?

We didn't encounter any issue so far that seems to be related to this patch.
However, some extra functional / performance testing are needed to evaluate.
>From internal code review it seems that wmb() is quite strict in your patch and
may have a performance penalty.
Any reason not to use dsb instead of dmb for wmb()?
#define wmb()asm volatile("dsb st" ::: "memory")
#define wmb()asm volatile("dmb st" ::: "memory")

> Thanks,
>
> Steve.
>
Thanks,
Noa

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH RFC v2 0/2] libibverbs memory barrier fixes

[Steve Wise]

> On 5/25/2016 5:51 PM, Steve Wise wrote:
> >> On 05/20/2016 03:43 PM, Steve Wise wrote:
> >>> This is v2 of the ARM64 mb* support, plus an additional patch to fail
> >>> compiles if there is no platform implementation of the memory barriers.
> >>> I've included the 2nd patch because I think it is important to not
> >>> assume any default barrier implementation.  Getting it wrong can result
> >>> in data corruption.
> >>>
> >>> Changes since V1:
> >>>
> >>> Put the RFC tag back on because I want to retest this series.
> >>>
> >>> Implemented the ARM64 memory barrier macros from scratch using the AMD
> >>> reference docs.
> >>>
> >>> Added 2nd patch to fail compiles if no mb* macros exist for the platform.
> >>>
> >>> Steve Wise (2):
> >>>   libibverbs: add ARM64 memory barrier macros
> >>>   Fail compiles if no platform specific memory barriers exist
> >>>
> >>>  include/infiniband/arch.h |   15 +++++++++------
> >>>  1 files changed, 9 insertions(+), 6 deletions(-)
> >>>
> >> Please remember to use --thread=shallow when sending patch emails using
> >> git send-email (which is what I assume you did).
> >>
> >> Let me know how the testing comes out.  The patches look fine to me
> >> (thanks for adding the second one, it's the right thing to do).
> >>
> > Hey Doug, the series tests good with cxgb4 on ARM64.  Do I need to repost a
> > final series, or can you merge v2?
> >
> > Hey Noa, have you all had a chance to test this series?
> 
> We didn't encounter any issue so far that seems to be related to this patch.
> However, some extra functional / performance testing are needed to evaluate.
> >From internal code review it seems that wmb() is quite strict in your patch and
> may have a performance penalty.
> Any reason not to use dsb instead of dmb for wmb()?
> #define wmb()asm volatile("dsb st" ::: "memory")
> #define wmb()asm volatile("dmb st" ::: "memory")

I was being conservative.   Unless you measure some drop in performance, I'd rather leave this as dsb for now.   Then perhaps we can optimize them later more performance work.

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html