[PATCH 0/2] Add support for enhanced atomic operations

[PATCH 0/2] Add support for enhanced atomic operations

[Vladimir Sokolovsky]

Hi Roland,

This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and add

These operations enable using a smaller amount of memory when using multiple locks
by using portions of a 64 bit value in an atomic operation.

Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations
mlx4/IB: Add support for enhanced atomic operations

  drivers/infiniband/hw/mlx4/cq.c   |    8 ++++++++
  drivers/infiniband/hw/mlx4/main.c |    2 ++
  drivers/infiniband/hw/mlx4/qp.c   |   27 +++++++++++++++++++++++++++
  include/linux/mlx4/device.h       |    4 ++--
  include/linux/mlx4/qp.h           |    7 +++++++
  include/rdma/ib_verbs.h           |    9 ++++++++-
  6 files changed, 54 insertions(+), 3 deletions(-)

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/2] Add support for enhanced atomic operations

[Sean Hefty]

>This patchset adds support for the following enhanced atomic operations:
>- Masked atomic compare and swap
>- Masked atomic fetch and add

Can you explain these in more detail?  How exactly is a mask used?  Are there
any restrictions on the format of the mask?  How is the mask carried over the
wire?  etc.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Vladimir Sokolovsky]

Sean Hefty wrote:
>> This patchset adds support for the following enhanced atomic operations:
>> - Masked atomic compare and swap
>> - Masked atomic fetch and add
> 
> Can you explain these in more detail?  How exactly is a mask used?  Are there
> any restrictions on the format of the mask?  How is the mask carried over the
> wire?  etc.
> 
> --

Masked Compare and Swap (MskCmpSwap)
The MskCmpSwap atomic operation is an extension to the CmpSwap operation defined in the IB
spec. MskCmpSwap allows the user to select a portion of the 64 bit target data for the “compare”
check as well as to restrict the swap to a (possibly different) portion. The pseudo code below
describes the operation:

-----
atomic_response = *va
if (((cmp XOR *va) AND cmp_mask) is ZERO) then
	*va = (*va AND NOT(swap_mask)) OR (swap AND swap_mask)

return atomic_response
-----

The additional operands are carried in the Extended Transport Header. Atomic response generation
and packet format for MskCmpSwap is as for standard IB Atomic operations.


Masked Fetch and Add (MFetchAdd)
The MFetchAdd Atomic operation extends the functionality of the standard IB FetchAdd by
allowing the user to split the target into multiple fields of selectable length. The atomic add is done
independently on each one of this fields. A bit set in the field_boundary parameter specifies the
field boundaries. The pseudo code below describes the operation:

------------
bit_adder(ci, b1, b2, *co)
{
         value = ci + b1 + b2;

         if (value & 2)
                 *co = 1
         else
                 *co = 0

         return value & 1
}

mask = 1
carry = 0
atomic_response = *va
#define MASK_IS_SET(mask, attr)      (((mask)&(attr))!=0)

for i = 0 to 63
{
         if ( i != 0 )
                 mask =  mask << 1;

         bit_add_res = bit_adder(carry, MASK_IS_SET(atomic_response, mask), MASK_IS_SET(add_value, mask), &new_carry)
         if (bit_add_res)
                 atomic_response |= mask

         carry = ((new_carry) && (!MASK_IS_SET(cmp_mask, mask)))
}

return atomic_response
------------

The additional operands are carried in the Extended Transport Header. Atomic response generation
and packet format for MFetchAdd is as for standard IB Atomic operations.

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Hal Rosenstock]

On Tue, Feb 2, 2010 at 5:44 AM, Vladimir Sokolovsky
<vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Sean Hefty wrote:
>>>
>>> This patchset adds support for the following enhanced atomic operations:
>>> - Masked atomic compare and swap
>>> - Masked atomic fetch and add
>>
>> Can you explain these in more detail?  How exactly is a mask used?  Are
>> there
>> any restrictions on the format of the mask?  How is the mask carried over
>> the
>> wire?  etc.
>>
>> --
>
> Masked Compare and Swap (MskCmpSwap)
> The MskCmpSwap atomic operation is an extension to the CmpSwap operation
> defined in the IB
> spec. MskCmpSwap allows the user to select a portion of the 64 bit target
> data for the “compare”
> check as well as to restrict the swap to a (possibly different) portion. The
> pseudo code below
> describes the operation:
>
> -----
> atomic_response = *va
> if (((cmp XOR *va) AND cmp_mask) is ZERO) then
>        *va = (*va AND NOT(swap_mask)) OR (swap AND swap_mask)
>
> return atomic_response
> -----
>
> The additional operands are carried in the Extended Transport Header. Atomic
> response generation
> and packet format for MskCmpSwap is as for standard IB Atomic operations.
>
>
> Masked Fetch and Add (MFetchAdd)
> The MFetchAdd Atomic operation extends the functionality of the standard IB
> FetchAdd by
> allowing the user to split the target into multiple fields of selectable
> length. The atomic add is done
> independently on each one of this fields. A bit set in the field_boundary
> parameter specifies the
> field boundaries. The pseudo code below describes the operation:
>
> ------------
> bit_adder(ci, b1, b2, *co)
> {
>        value = ci + b1 + b2;
>
>        if (value & 2)
>                *co = 1
>        else
>                *co = 0
>
>        return value & 1
> }
>
> mask = 1
> carry = 0
> atomic_response = *va
> #define MASK_IS_SET(mask, attr)      (((mask)&(attr))!=0)
>
> for i = 0 to 63
> {
>        if ( i != 0 )
>                mask =  mask << 1;
>
>        bit_add_res = bit_adder(carry, MASK_IS_SET(atomic_response, mask),
> MASK_IS_SET(add_value, mask), &new_carry)
>        if (bit_add_res)
>                atomic_response |= mask
>
>        carry = ((new_carry) && (!MASK_IS_SET(cmp_mask, mask)))
> }
>
> return atomic_response
> ------------
>
> The additional operands are carried in the Extended Transport Header. Atomic
> response generation
> and packet format for MFetchAdd is as for standard IB Atomic operations.

Will Mellanox be adding this option to the IB spec rather than keep it
as vendor proprietary ?

-- Hal

>
> Regards,
> Vladimir
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Håkon Bugge]

On Feb 2, 2010, at 16:54 , Hal Rosenstock wrote:

> On Tue, Feb 2, 2010 at 5:44 AM, Vladimir Sokolovsky
>> [snip]
>> Masked Fetch and Add (MFetchAdd)
>> The MFetchAdd Atomic operation extends the functionality of the standard IB
>> FetchAdd by
>> allowing the user to split the target into multiple fields of selectable
>> length. The atomic add is done
>> independently on each one of this fields. A bit set in the field_boundary
>> parameter specifies the
>> field boundaries. The pseudo code below describes the operation:


As discussed by private email, my take is that it is more important to support adjacent fields than a single-bit fetch-and-add. Hence, encoding the mask slightly different where a one-to-zero bit transition indicates break-of-carry, but mask-wise treated as a one, allows adjacent bit-fields to be added.

Just my two cents.


Håkon

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/2] Add support for enhanced atomic operations

[Sean Hefty]

>Will Mellanox be adding this option to the IB spec rather than keep it
>as vendor proprietary ?

Along with that question, what is the use case for this feature?  The only
benefit mentioned was saving a few bytes of memory, at the cost of carrying
extra network headers.  Atomics are only 64-bits to begin with...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Vladimir Sokolovsky]

Sean Hefty wrote:
>> Will Mellanox be adding this option to the IB spec rather than keep it
>> as vendor proprietary ?
> 
> Along with that question, what is the use case for this feature?  The only
> benefit mentioned was saving a few bytes of memory, at the cost of carrying
> extra network headers.  Atomics are only 64-bits to begin with...
> 


For some applications the memory savings are very significant. One
example is fine grain lock implementations for huge data sets. In other
cases, the benefit is the ability to update multiple fields with a
single io operation.

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Hal Rosenstock]

On Wed, Feb 3, 2010 at 3:01 AM, Vladimir Sokolovsky
<vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Sean Hefty wrote:
>>>
>>> Will Mellanox be adding this option to the IB spec rather than keep it
>>> as vendor proprietary ?
>>
>> Along with that question, what is the use case for this feature?  The only
>> benefit mentioned was saving a few bytes of memory, at the cost of
>> carrying
>> extra network headers.  Atomics are only 64-bits to begin with...
>>
>
>
> For some applications the memory savings are very significant. One
> example is fine grain lock implementations for huge data sets. In other
> cases, the benefit is the ability to update multiple fields with a
> single io operation.

What happens when the other end doesn't support this feature (new
opcodes) ? How can that be determined remotely ? Is some CM related
change also needed ?

-- Hal

>
> Regards,
> Vladimir
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Vladimir Sokolovsky]

Hal Rosenstock wrote:
> On Wed, Feb 3, 2010 at 3:01 AM, Vladimir Sokolovsky
> <vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>> Sean Hefty wrote:
>>>> Will Mellanox be adding this option to the IB spec rather than keep it
>>>> as vendor proprietary ?
>>> Along with that question, what is the use case for this feature?  The only
>>> benefit mentioned was saving a few bytes of memory, at the cost of
>>> carrying
>>> extra network headers.  Atomics are only 64-bits to begin with...
>>>
>>
>> For some applications the memory savings are very significant. One
>> example is fine grain lock implementations for huge data sets. In other
>> cases, the benefit is the ability to update multiple fields with a
>> single io operation.
> 
> What happens when the other end doesn't support this feature (new
> opcodes) ? How can that be determined remotely ? Is some CM related
> change also needed ?
> 
> -- Hal
> 

In this case the responder (the other end) will return a NAK-Invalid Request
to the requester.
There is no need to extend the CM protocol. This can be done as part of
application negotiation. Atomic is an optional feature and the CM is not
involved in negotiation of this. Application to application should decide
if they want and they can use atomic. The same should be done for extended
atomics.

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Add support for enhanced atomic operations

[Tziporet Koren]

On 2/2/2010 5:54 PM, Hal Rosenstock wrote:
> Will Mellanox be adding this option to the IB spec rather than keep it
> as vendor proprietary ?
>    
Yes, it's in our plans

Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html