[PATCH V2 1/2] IB/core: Add support for enhanced atomic operations

[PATCH V2 1/2] IB/core: Add support for enhanced atomic operations

[Vladimir Sokolovsky]

- Add a new IB_WR_ATOMIC_MASKED_CMP_AND_SWP and IB_WR_ATOMIC_MASKED_FETCH_AND_ADD send
opcodes that can be used to mark a "masked atomic compare and swap" and
"masked atomic fetch and add" work request correspondingly.
- Add IB_DEVICE_MASKED_ATOMIC capability bit.
- Add mask fields to atomic struct of ib_send_wr
- Add new opcodes to ib_wc_opcode

Signed-off-by: Vladimir Sokolovsky <vlad-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 include/rdma/ib_verbs.h |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 09509ed..338cff1 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -105,6 +105,7 @@ enum ib_device_cap_flags {
 	IB_DEVICE_UD_TSO		= (1<<19),
 	IB_DEVICE_MEM_MGT_EXTENSIONS	= (1<<21),
 	IB_DEVICE_BLOCK_MULTICAST_LOOPBACK = (1<<22),
+	IB_DEVICE_MASKED_ATOMIC		= (1<<23),
 };
 
 enum ib_atomic_cap {
@@ -467,6 +468,8 @@ enum ib_wc_opcode {
 	IB_WC_LSO,
 	IB_WC_LOCAL_INV,
 	IB_WC_FAST_REG_MR,
+	IB_WC_MASKED_COMP_SWAP,
+	IB_WC_MASKED_FETCH_ADD,
 /*
  * Set value of IB_WC_RECV so consumers can test if a completion is a
  * receive by testing (opcode & IB_WC_RECV).
@@ -689,6 +692,8 @@ enum ib_wr_opcode {
 	IB_WR_RDMA_READ_WITH_INV,
 	IB_WR_LOCAL_INV,
 	IB_WR_FAST_REG_MR,
+	IB_WR_ATOMIC_MASKED_CMP_AND_SWP,
+	IB_WR_ATOMIC_MASKED_FETCH_AND_ADD,
 };
 
 enum ib_send_flags {
@@ -732,6 +737,8 @@ struct ib_send_wr {
 			u64	compare_add;
 			u64	swap;
 			u32	rkey;
+			u64	compare_add_mask;
+			u64	swap_mask;
 		} atomic;
 		struct {
 			struct ib_ah *ah;
-- 
1.6.6.GIT

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V2 1/2] IB/core: Add support for enhanced atomic operations

[Roland Dreier]

I think I'd be OK adding this for 2.6.34, if there were some case made
that this would be useful for someone working on code that used this.
So what case can be made for adding this to 2.6.34 without an in-tree user?

 > - Add a new IB_WR_ATOMIC_MASKED_CMP_AND_SWP and IB_WR_ATOMIC_MASKED_FETCH_AND_ADD send
 > opcodes that can be used to mark a "masked atomic compare and swap" and
 > "masked atomic fetch and add" work request correspondingly.
 > - Add IB_DEVICE_MASKED_ATOMIC capability bit.
 > - Add mask fields to atomic struct of ib_send_wr
 > - Add new opcodes to ib_wc_opcode

The description of these operations from the mlx4 patch really belongs
here.

 > +	IB_WR_ATOMIC_MASKED_CMP_AND_SWP,
 > +	IB_WR_ATOMIC_MASKED_FETCH_AND_ADD,

Even in your description you talk about "masked atomic..." which makes
the ATOMIC_MASKED read a little strangely.  These enum values would make
more sense as:

	IB_WR_MASKED_ATOMIC_CMP_AND_SWP,
	IB_WR_MASKED_ATOMIC_FETCH_AND_ADD,

 >  			u64	swap;
 >  			u32	rkey;
 > +			u64	compare_add_mask;
 > +			u64	swap_mask;

It doesn't make any practical difference here, but it would be cleaner
to add the new fields before the rkey field, so that we don't leave a
hole due to padding.

 - R.
-- 
Roland Dreier  <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V2 1/2] IB/core: Add support for enhanced atomic operations

[Andy Grover]

Roland Dreier wrote:
> I think I'd be OK adding this for 2.6.34, if there were some case made
> that this would be useful for someone working on code that used this.
> So what case can be made for adding this to 2.6.34 without an in-tree user?

Support needs to be there before anyone can use it :)

Here is a git tree of our atomics work so far:
http://www.openfabrics.org/git/?p=~agrover/ofed_1_5/linux-2.6.git;a=shortlog;h=future/20100203

Still in-development, but it should be ready very soon. We will be the
first in-kernel user of atomics, as well as masked atomics. This tree
does not support masked atomics yet because it is based on ofed 1.5.1.
When 1.5.2 is officially released, I'll rebase, add mask support, and
plan on pushing to mainline and ofed 1.6 as soon as it opens.

Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V2 1/2] IB/core: Add support for enhanced atomic operations

[Håkon Bugge]

> Still in-development, but it should be ready very soon. We will be the
> first in-kernel user of atomics, as well as masked atomics. This tree
> does not support masked atomics yet because it is based on ofed 1.5.1.
> When 1.5.2 is officially released, I'll rebase, add mask support, and
> plan on pushing to mainline and ofed 1.6 as soon as it opens.



May be I missed something, but how do you guys intend to reflect this new functionality vs. the capabilities? A new atomic_enhanced_cap ?

The reason for asking is that the IB ordinary atomic repertoire fits nicely with those of PCIe Gen 3, so one could possible assume new HCAs supporting PCIe Gen3 to possess the ATOMIC_GLOB capability for atomic_cap. But I do not see that happening to the proposed Enhanced IB Atomics.


Thanks, Håkon

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html