[PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

A bad cast causes the iova_start, which in this case is a DMA bus address,
to be truncated on 32b systems.  No cast is needed.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
---

 net/sunrpc/xprtrdma/verbs.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 3b21e0c..3a374f5 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,
 	memset(&frmr_wr, 0, sizeof frmr_wr);
 	frmr_wr.opcode = IB_WR_FAST_REG_MR;
 	frmr_wr.send_flags = 0;			/* unsignaled */
-	frmr_wr.wr.fast_reg.iova_start = (unsigned long)seg1->mr_dma;
+	frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma;
 	frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;
 	frmr_wr.wr.fast_reg.page_list_len = i;
 	frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Trond,

There is a similar bug in the server side too.  I'll repost a single 
patch that fixes both.

Steve.

Steve Wise wrote:
> A bad cast causes the iova_start, which in this case is a DMA bus address,
> to be truncated on 32b systems.  No cast is needed.
>
> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> ---
>
>  net/sunrpc/xprtrdma/verbs.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 3b21e0c..3a374f5 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,
>  	memset(&frmr_wr, 0, sizeof frmr_wr);
>  	frmr_wr.opcode = IB_WR_FAST_REG_MR;
>  	frmr_wr.send_flags = 0;			/* unsignaled */
> -	frmr_wr.wr.fast_reg.iova_start = (unsigned long)seg1->mr_dma;
> +	frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma;
>  	frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;
>  	frmr_wr.wr.fast_reg.page_list_len = i;
>  	frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Hey Trond,

Turns out the server side is fine.  So this patch is good to go.

Thanks,

Steve.

Steve Wise wrote:
> Trond,
>
> There is a similar bug in the server side too.  I'll repost a single 
> patch that fixes both.
>
> Steve.
>
> Steve Wise wrote:
>> A bad cast causes the iova_start, which in this case is a DMA bus 
>> address,
>> to be truncated on 32b systems.  No cast is needed.
>>
>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
>> ---
>>
>>  net/sunrpc/xprtrdma/verbs.c |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>> index 3b21e0c..3a374f5 100644
>> --- a/net/sunrpc/xprtrdma/verbs.c
>> +++ b/net/sunrpc/xprtrdma/verbs.c
>> @@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct 
>> rpcrdma_mr_seg *seg,
>>      memset(&frmr_wr, 0, sizeof frmr_wr);
>>      frmr_wr.opcode = IB_WR_FAST_REG_MR;
>>      frmr_wr.send_flags = 0;            /* unsignaled */
>> -    frmr_wr.wr.fast_reg.iova_start = (unsigned long)seg1->mr_dma;
>> +    frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma;
>>      frmr_wr.wr.fast_reg.page_list = 
>> seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;
>>      frmr_wr.wr.fast_reg.page_list_len = i;
>>      frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>   
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Tom Talpey]

At 02:57 PM 4/26/2009, Steve Wise wrote:
>Hey Trond,
>
>Turns out the server side is fine.  So this patch is good to go.

I'll ACK the patch, but I do wonder if the code will compile cleanly on all
platforms. The iova_start is a u64, whereas the mr_dma is a dma_addr_t,
which is variable sized, depending on platform. Would a (u64) cast be a
safer patch, warning-wise?

Tom.

>
>Thanks,
>
>Steve.
>
>Steve Wise wrote:
>> Trond,
>>
>> There is a similar bug in the server side too.  I'll repost a single 
>> patch that fixes both.
>>
>> Steve.
>>
>> Steve Wise wrote:
>>> A bad cast causes the iova_start, which in this case is a DMA bus 
>>> address,
>>> to be truncated on 32b systems.  No cast is needed.
>>>
>>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
>>> ---
>>>
>>>  net/sunrpc/xprtrdma/verbs.c |    2 +-
>>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>>> index 3b21e0c..3a374f5 100644
>>> --- a/net/sunrpc/xprtrdma/verbs.c
>>> +++ b/net/sunrpc/xprtrdma/verbs.c
>>> @@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct 
>>> rpcrdma_mr_seg *seg,
>>>      memset(&frmr_wr, 0, sizeof frmr_wr);
>>>      frmr_wr.opcode = IB_WR_FAST_REG_MR;
>>>      frmr_wr.send_flags = 0;            /* unsignaled */
>>> -    frmr_wr.wr.fast_reg.iova_start = (unsigned long)seg1->mr_dma;
>>> +    frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma;
>>>      frmr_wr.wr.fast_reg.page_list = 
>>> seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;
>>>      frmr_wr.wr.fast_reg.page_list_len = i;
>>>      frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>   
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Tom Talpey wrote:
> At 02:57 PM 4/26/2009, Steve Wise wrote:
>   
>> Hey Trond,
>>
>> Turns out the server side is fine.  So this patch is good to go.
>>     
>
> I'll ACK the patch, but I do wonder if the code will compile cleanly on all
> platforms. The iova_start is a u64, whereas the mr_dma is a dma_addr_t,
> which is variable sized, depending on platform. Would a (u64) cast be a
> safer patch, warning-wise?
>
>   

Well, if dma_addr_t is smaller, I don't think you'll get a warning, will 
you?  And if its larger than 64b, then you _want_ the warning, because 
nothing will work. :)

Its up to you and/or Trond.

Steve.


> Tom.
>
>   
>> Thanks,
>>
>> Steve.
>>
>> Steve Wise wrote:
>>     
>>> Trond,
>>>
>>> There is a similar bug in the server side too.  I'll repost a single 
>>> patch that fixes both.
>>>
>>> Steve.
>>>
>>> Steve Wise wrote:
>>>       
>>>> A bad cast causes the iova_start, which in this case is a DMA bus 
>>>> address,
>>>> to be truncated on 32b systems.  No cast is needed.
>>>>
>>>> Signed-off-by: Steve Wise <swise@opengridcomputing.com>
>>>> ---
>>>>
>>>>  net/sunrpc/xprtrdma/verbs.c |    2 +-
>>>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>>>> index 3b21e0c..3a374f5 100644
>>>> --- a/net/sunrpc/xprtrdma/verbs.c
>>>> +++ b/net/sunrpc/xprtrdma/verbs.c
>>>> @@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct 
>>>> rpcrdma_mr_seg *seg,
>>>>      memset(&frmr_wr, 0, sizeof frmr_wr);
>>>>      frmr_wr.opcode = IB_WR_FAST_REG_MR;
>>>>      frmr_wr.send_flags = 0;            /* unsignaled */
>>>> -    frmr_wr.wr.fast_reg.iova_start = (unsigned long)seg1->mr_dma;
>>>> +    frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma;
>>>>      frmr_wr.wr.fast_reg.page_list = 
>>>> seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;
>>>>      frmr_wr.wr.fast_reg.page_list_len = i;
>>>>      frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>   
>>>>         
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>       
>>     

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-04-27 at 12:37 -0500, Steve Wise wrote:
> Tom Talpey wrote:
> > At 02:57 PM 4/26/2009, Steve Wise wrote:
> >   
> >> Hey Trond,
> >>
> >> Turns out the server side is fine.  So this patch is good to go.
> >>     
> >
> > I'll ACK the patch, but I do wonder if the code will compile cleanly on all
> > platforms. The iova_start is a u64, whereas the mr_dma is a dma_addr_t,
> > which is variable sized, depending on platform. Would a (u64) cast be a
> > safer patch, warning-wise?
> >
> >   
> 
> Well, if dma_addr_t is smaller, I don't think you'll get a warning, will 
> you?  And if its larger than 64b, then you _want_ the warning, because 
> nothing will work. :)
> 
> Its up to you and/or Trond.

It looks looks as though the bug is really that the IB code is using a
u64 to store dma handles. As an external user of the IB api, we really
shouldn't have to perform this sort of transformation. If it is
absolutely necessary, then it should be done by means of specialised
accessor functions to initialise/read iova_start value when given a
dma_addr_t.

I'd therefore prefer the no-cast version (with eventual compiler
warnings), in the hope that eventually the IB folks will fix their
interface.

Cheers
   Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
> It looks looks as though the bug is really that the IB code is using a
> u64 to store dma handles. As an external user of the IB api, we really
> shouldn't have to perform this sort of transformation. If it is
> absolutely necessary, then it should be done by means of specialised
> accessor functions to initialise/read iova_start value when given a
> dma_addr_t.
> 
> I'd therefore prefer the no-cast version (with eventual compiler
> warnings), in the hope that eventually the IB folks will fix their
> interface.

Translation: It looks to me as if the interface that we're using is a
bit too corrupted with IB low level implementation grime. In the future,
I'd like to see someone come up with a more high level interface for use
by external code such as the sunrpc module.

Cheers
  Trond

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Trond Myklebust wrote:
> On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
>   
>> It looks looks as though the bug is really that the IB code is using a
>> u64 to store dma handles. As an external user of the IB api, we really
>> shouldn't have to perform this sort of transformation. If it is
>> absolutely necessary, then it should be done by means of specialised
>> accessor functions to initialise/read iova_start value when given a
>> dma_addr_t.
>>
>> I'd therefore prefer the no-cast version (with eventual compiler
>> warnings), in the hope that eventually the IB folks will fix their
>> interface.
>>     
>
> Translation: It looks to me as if the interface that we're using is a
> bit too corrupted with IB low level implementation grime. In the future,
> I'd like to see someone come up with a more high level interface for use
> by external code such as the sunrpc module.
>
>   

Clarification:  The iova_start isn't used to store dma handles.  The 
iova_start is the "address" base value that is advertised to a peer to 
describe the base address of a memory region.  The contents of that can 
be more than just a dma handle...its up to the application.  For 
instance, you could advertise a iova_start of zero or a kernel VA as the 
rdma server does.  Also, the type is u64 because that is the size used 
on the wire as part of the rdma (IB and iWARP) protocols. 


Steve.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Tom Talpey]

At 03:32 PM 4/27/2009, Steve Wise wrote:
>Trond Myklebust wrote:
>> On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
>>   
>>> It looks looks as though the bug is really that the IB code is using a
>>> u64 to store dma handles. As an external user of the IB api, we really
>>> shouldn't have to perform this sort of transformation. If it is
>>> absolutely necessary, then it should be done by means of specialised
>>> accessor functions to initialise/read iova_start value when given a
>>> dma_addr_t.
>>>
>>> I'd therefore prefer the no-cast version (with eventual compiler
>>> warnings), in the hope that eventually the IB folks will fix their
>>> interface.
>>>     
>>
>> Translation: It looks to me as if the interface that we're using is a
>> bit too corrupted with IB low level implementation grime. In the future,
>> I'd like to see someone come up with a more high level interface for use
>> by external code such as the sunrpc module.
>>
>>   
>
>Clarification:  The iova_start isn't used to store dma handles.  The 

Agreed, it's more of a hardware register, that ends up on the wire as well.

I think the net of this is that the mr_dma should have a more sensible
up-cast that yields the right bits in the iova_start. Maybe a nice
machine-dependent macro, defined in the RDMA layer, would be a good
approach. Surely the other upper layers need it too.

While I have the floor, why doesn't the server have this issue? Looking
at the code, it has the same (unsigned long) cast as the client when
initializing its iova_start.

Tom.

>iova_start is the "address" base value that is advertised to a peer to 
>describe the base address of a memory region.  The contents of that can 
>be more than just a dma handle...its up to the application.  For 
>instance, you could advertise a iova_start of zero or a kernel VA as the 
>rdma server does.  Also, the type is u64 because that is the size used 
>on the wire as part of the rdma (IB and iWARP) protocols. 
>
>
>Steve.
>

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Tom Talpey wrote:
> At 03:32 PM 4/27/2009, Steve Wise wrote:
>   
>> Trond Myklebust wrote:
>>     
>>> On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
>>>   
>>>       
>>>> It looks looks as though the bug is really that the IB code is using a
>>>> u64 to store dma handles. As an external user of the IB api, we really
>>>> shouldn't have to perform this sort of transformation. If it is
>>>> absolutely necessary, then it should be done by means of specialised
>>>> accessor functions to initialise/read iova_start value when given a
>>>> dma_addr_t.
>>>>
>>>> I'd therefore prefer the no-cast version (with eventual compiler
>>>> warnings), in the hope that eventually the IB folks will fix their
>>>> interface.
>>>>     
>>>>         
>>> Translation: It looks to me as if the interface that we're using is a
>>> bit too corrupted with IB low level implementation grime. In the future,
>>> I'd like to see someone come up with a more high level interface for use
>>> by external code such as the sunrpc module.
>>>
>>>   
>>>       
>> Clarification:  The iova_start isn't used to store dma handles.  The 
>>     
>
> Agreed, it's more of a hardware register, that ends up on the wire as well.
>
> I think the net of this is that the mr_dma should have a more sensible
> up-cast that yields the right bits in the iova_start. Maybe a nice
> machine-dependent macro, defined in the RDMA layer, would be a good
> approach. Surely the other upper layers need it too.
>
> While I have the floor, why doesn't the server have this issue? Looking
> at the code, it has the same (unsigned long) cast as the client when
> initializing its iova_start.
>
>   

The server isn't using the dma address as the iova_start, but rather a 
kernel virtual address pointer, which is 32b on a i386 system.  If you 
take the cast off, then the the signed bit gets extended into the u64.  
Apparently pointers are signed? 

For instance, the server had a kva of 0xf5b75000.  If you remove the 
(unsigned long) cast and stuff that into a u64, it ends up as 
0xfffffffff5b75000.

here was a trace I took of the server doing the first rdma write using 
an frmr:

Apr 26 13:14:07 rac2 kernel: build_fastreg iova_start 0xfffffffff5b75000 
rkey 0x500 len 4096
Apr 26 13:14:07 rac2 kernel: build_fastreg pbl[0] 0x35b75000
Apr 26 13:14:07 rac2 kernel: build_rdma_write sge[0] lkey 0x500 addr 
0xf5b75000 len 24
Apr 26 13:14:07 rac2 kernel: post_qp_event - AE qpid 0x23 opcode 0 
status 0x6 type 1 wrid.hi 0x1 wrid.lo 0x0


So the frmr registration ends up with 0xfffffffff5b75000 as the 
iova_start, yet the rdma write work request has 0xf5b75000 as the sge 
address entry.   And the rnic fails this WR with a base/bounds violation 
(status 0x6 in the chelsio Async Event).

Steve.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Tom Talpey]

At 03:50 PM 4/27/2009, Steve Wise wrote:
>Tom Talpey wrote:
>> At 03:32 PM 4/27/2009, Steve Wise wrote:
>>   
>>> Trond Myklebust wrote:
>>>     
>>>> On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
>>>>   
>>>>       
>>>>> It looks looks as though the bug is really that the IB code is using a
>>>>> u64 to store dma handles. As an external user of the IB api, we really
>>>>> shouldn't have to perform this sort of transformation. If it is
>>>>> absolutely necessary, then it should be done by means of specialised
>>>>> accessor functions to initialise/read iova_start value when given a
>>>>> dma_addr_t.
>>>>>
>>>>> I'd therefore prefer the no-cast version (with eventual compiler
>>>>> warnings), in the hope that eventually the IB folks will fix their
>>>>> interface.
>>>>>     
>>>>>         
>>>> Translation: It looks to me as if the interface that we're using is a
>>>> bit too corrupted with IB low level implementation grime. In the future,
>>>> I'd like to see someone come up with a more high level interface for use
>>>> by external code such as the sunrpc module.
>>>>
>>>>   
>>>>       
>>> Clarification:  The iova_start isn't used to store dma handles.  The 
>>>     
>>
>> Agreed, it's more of a hardware register, that ends up on the wire as well.
>>
>> I think the net of this is that the mr_dma should have a more sensible
>> up-cast that yields the right bits in the iova_start. Maybe a nice
>> machine-dependent macro, defined in the RDMA layer, would be a good
>> approach. Surely the other upper layers need it too.
>>
>> While I have the floor, why doesn't the server have this issue? Looking
>> at the code, it has the same (unsigned long) cast as the client when
>> initializing its iova_start.
>>
>>   
>
>The server isn't using the dma address as the iova_start, but rather a 
>kernel virtual address pointer, which is 32b on a i386 system.  If you 
>take the cast off, then the the signed bit gets extended into the u64.  
>Apparently pointers are signed? 

Why is the server using a u64 to store a naked pointer? That has to be
a bug. Casting to (unsigned long) is just hiding it.

Does this address get handed to the RNIC to perform some sort of local
DMA? If so, how does it work if there's an IOMMU in the system? The
kva isn't necessarily the same as the dma_addr, right?

BTW, pointers are unsigned, but the assignment to u64 causes the
compiler to convert the pointer into a ptrdiff_t, in effect evaluating
((pointer) - NULL). Then, since the ptrdiff_t is a signed 32 bits, the
promotion results in the sign extension. I think! IOW, bug.

Tom.

>
>For instance, the server had a kva of 0xf5b75000.  If you remove the 
>(unsigned long) cast and stuff that into a u64, it ends up as 
>0xfffffffff5b75000.
>
>here was a trace I took of the server doing the first rdma write using 
>an frmr:
>
>Apr 26 13:14:07 rac2 kernel: build_fastreg iova_start 0xfffffffff5b75000 
>rkey 0x500 len 4096
>Apr 26 13:14:07 rac2 kernel: build_fastreg pbl[0] 0x35b75000
>Apr 26 13:14:07 rac2 kernel: build_rdma_write sge[0] lkey 0x500 addr 
>0xf5b75000 len 24
>Apr 26 13:14:07 rac2 kernel: post_qp_event - AE qpid 0x23 opcode 0 
>status 0x6 type 1 wrid.hi 0x1 wrid.lo 0x0
>
>
>So the frmr registration ends up with 0xfffffffff5b75000 as the 
>iova_start, yet the rdma write work request has 0xf5b75000 as the sge 
>address entry.   And the rnic fails this WR with a base/bounds violation 
>(status 0x6 in the chelsio Async Event).
>
>Steve.
>
>

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Tom Talpey wrote:
> At 03:50 PM 4/27/2009, Steve Wise wrote:
>   
>> Tom Talpey wrote:
>>     
>>> At 03:32 PM 4/27/2009, Steve Wise wrote:
>>>   
>>>       
>>>> Trond Myklebust wrote:
>>>>     
>>>>         
>>>>> On Mon, 2009-04-27 at 14:05 -0400, Trond Myklebust wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> It looks looks as though the bug is really that the IB code is using a
>>>>>> u64 to store dma handles. As an external user of the IB api, we really
>>>>>> shouldn't have to perform this sort of transformation. If it is
>>>>>> absolutely necessary, then it should be done by means of specialised
>>>>>> accessor functions to initialise/read iova_start value when given a
>>>>>> dma_addr_t.
>>>>>>
>>>>>> I'd therefore prefer the no-cast version (with eventual compiler
>>>>>> warnings), in the hope that eventually the IB folks will fix their
>>>>>> interface.
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> Translation: It looks to me as if the interface that we're using is a
>>>>> bit too corrupted with IB low level implementation grime. In the future,
>>>>> I'd like to see someone come up with a more high level interface for use
>>>>> by external code such as the sunrpc module.
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>> Clarification:  The iova_start isn't used to store dma handles.  The 
>>>>     
>>>>         
>>> Agreed, it's more of a hardware register, that ends up on the wire as well.
>>>
>>> I think the net of this is that the mr_dma should have a more sensible
>>> up-cast that yields the right bits in the iova_start. Maybe a nice
>>> machine-dependent macro, defined in the RDMA layer, would be a good
>>> approach. Surely the other upper layers need it too.
>>>
>>> While I have the floor, why doesn't the server have this issue? Looking
>>> at the code, it has the same (unsigned long) cast as the client when
>>> initializing its iova_start.
>>>
>>>   
>>>       
>> The server isn't using the dma address as the iova_start, but rather a 
>> kernel virtual address pointer, which is 32b on a i386 system.  If you 
>> take the cast off, then the the signed bit gets extended into the u64.  
>> Apparently pointers are signed? 
>>     
>
> Why is the server using a u64 to store a naked pointer? That has to be
> a bug. Casting to (unsigned long) is just hiding it.
>   


That is what it wants to use as the registration for its frmr, which in 
this case is used as the source of an RDMA Write.


> Does this address get handed to the RNIC to perform some sort of local
> DMA? 

No.
> If so, how does it work if there's an IOMMU in the system? The
> kva isn't necessarily the same as the dma_addr, right?
>
>   

Correct. 

This kva is used as the iova_start for the fast-registered memory 
region.   All it is used for is to mark the base value for "addresses" 
passed in via sge entries in the work requests, and also for incoming 
"addresses" in rdma packets.   So you can use the kva when you fastreg 
the mr, and then also use the kva + any offset in the sge entries of 
your work requests that utilize it.  Additionally, you can advertise the 
fastreg rkey, iova_start, and length to the peer for doing rdma into 
that region.  The HW will validate any SGE entry in and any incoming 
rdma packets to ensure that the rkey/addr/len in the sge/packet is 
within the bounds of the fastregmr.  Namely that the sge/packet address 
and length fall within the iova_start and iova_start+fastreg_len.

 
> BTW, pointers are unsigned, but the assignment to u64 causes the
> compiler to convert the pointer into a ptrdiff_t, in effect evaluating
> ((pointer) - NULL). Then, since the ptrdiff_t is a signed 32 bits, the
> promotion results in the sign extension. I think! IOW, bug.
>
>   

I see.  And that's why the cast is needed for the server side.


Steve.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-04-27 at 14:32 -0500, Steve Wise wrote:
> Clarification:  The iova_start isn't used to store dma handles.  The 
> iova_start is the "address" base value that is advertised to a peer to 
> describe the base address of a memory region.  The contents of that can 
> be more than just a dma handle...its up to the application.  For 
> instance, you could advertise a iova_start of zero or a kernel VA as the 
> rdma server does.  Also, the type is u64 because that is the size used 
> on the wire as part of the rdma (IB and iWARP) protocols. 

OK, but my point is we shouldn't be having this discussion at all. I
shouldn't be required to know that the wire protocol uses a 64-bit
unsigned little-endian/big endian integer in order to use the rdma api.

All I should need to know is that I can advertise either dma handles or
kernel VAs, and know that I can choose between two functions, say,
ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to
initialise the ib_send_wr structure correctly.

Trond

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Trond Myklebust wrote:
> On Mon, 2009-04-27 at 14:32 -0500, Steve Wise wrote:
>   
>> Clarification:  The iova_start isn't used to store dma handles.  The 
>> iova_start is the "address" base value that is advertised to a peer to 
>> describe the base address of a memory region.  The contents of that can 
>> be more than just a dma handle...its up to the application.  For 
>> instance, you could advertise a iova_start of zero or a kernel VA as the 
>> rdma server does.  Also, the type is u64 because that is the size used 
>> on the wire as part of the rdma (IB and iWARP) protocols. 
>>     
>
> OK, but my point is we shouldn't be having this discussion at all. I
> shouldn't be required to know that the wire protocol uses a 64-bit
> unsigned little-endian/big endian integer in order to use the rdma api.
>
> All I should need to know is that I can advertise either dma handles or
> kernel VAs, and know that I can choose between two functions, say,
> ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to
> initialise the ib_send_wr structure correctly.
>   

I hear ya.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Hey Trond,

Will this bug fix make 2.6.30?

Thanks,

Steve.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
> Hey Trond,
> 
> Will this bug fix make 2.6.30?
> 
> Thanks,
> 
> Steve.

Not in the form it is in now. As I've said earlier, I'm not happy about
the sunrpc layer having to circumvent ordinary type checking on
non-sunrpc structures.

Cheers
    Trond

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Trond Myklebust wrote:
> On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
>   
>> Hey Trond,
>>
>> Will this bug fix make 2.6.30?
>>
>> Thanks,
>>
>> Steve.
>>     
>
> Not in the form it is in now. As I've said earlier, I'm not happy about
> the sunrpc layer having to circumvent ordinary type checking on
> non-sunrpc structures.
>
> Cheers
>     Trond
How is it circumventing?  It's currently incorrectly casting a pointer 
into a u64.  That seems just broken to me.  Also, its really the sunrpc 
rdma transport layer.  It deals specifically with rdma.  It _should_ 
know about rdma interfaces and types.

But  I'll whip up get/put type accessor methods for this field if that's 
what you require.  In the meantime, customers will just crash I guess.


Steve.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Tom Talpey]

At 08:13 PM 5/11/2009, Steve Wise wrote:
>Trond Myklebust wrote:
>> On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
>>   
>>> Hey Trond,
>>>
>>> Will this bug fix make 2.6.30?
>>>
>>> Thanks,
>>>
>>> Steve.
>>>     
>>
>> Not in the form it is in now. As I've said earlier, I'm not happy about
>> the sunrpc layer having to circumvent ordinary type checking on
>> non-sunrpc structures.
>>
>> Cheers
>>     Trond
>How is it circumventing?  It's currently incorrectly casting a pointer 
>into a u64.  That seems just broken to me.

The cast is definitely broken, and since your patch removes it I agree
with the change, for the record. Especially since with the change, the
code doesn't work on 32-bit CPU / 64-bit IOMMU platforms.

>  Also, its really the sunrpc 
>rdma transport layer.  It deals specifically with rdma.  It _should_ 
>know about rdma interfaces and types.

IOW, this is an issue for the OFA API to address? I agree with Trond that
there are some hardware-specific warts in there. It might be a good idea
for us to go through both the client and server code overall - you up for
doing that? I'm in, if so.

Tom.


>
>But  I'll whip up get/put type accessor methods for this field if that's 
>what you require.  In the meantime, customers will just crash I guess.
>
>
>Steve.
>
>
>
>

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-05-11 at 19:13 -0500, Steve Wise wrote:
> Trond Myklebust wrote:
> > On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
> >   
> >> Hey Trond,
> >>
> >> Will this bug fix make 2.6.30?
> >>
> >> Thanks,
> >>
> >> Steve.
> >>     
> >
> > Not in the form it is in now. As I've said earlier, I'm not happy about
> > the sunrpc layer having to circumvent ordinary type checking on
> > non-sunrpc structures.
> >
> > Cheers
> >     Trond
> How is it circumventing?  It's currently incorrectly casting a pointer 
> into a u64.  That seems just broken to me.  Also, its really the sunrpc 
> rdma transport layer.  It deals specifically with rdma.  It _should_ 
> know about rdma interfaces and types.

The fact is that I'm simply not interested enough in rdma to tolerate
hacks. If it isn't done cleanly, in a manner that I can maintain, then
the whole transport layer comes out...

Trond

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Tom Talpey wrote:
> At 08:13 PM 5/11/2009, Steve Wise wrote:
>   
>> Trond Myklebust wrote:
>>     
>>> On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
>>>   
>>>       
>>>> Hey Trond,
>>>>
>>>> Will this bug fix make 2.6.30?
>>>>
>>>> Thanks,
>>>>
>>>> Steve.
>>>>     
>>>>         
>>> Not in the form it is in now. As I've said earlier, I'm not happy about
>>> the sunrpc layer having to circumvent ordinary type checking on
>>> non-sunrpc structures.
>>>
>>> Cheers
>>>     Trond
>>>       
>> How is it circumventing?  It's currently incorrectly casting a pointer 
>> into a u64.  That seems just broken to me.
>>     
>
> The cast is definitely broken, and since your patch removes it I agree
> with the change, for the record. Especially since with the change, the
> code doesn't work on 32-bit CPU / 64-bit IOMMU platforms.
>
>   

That was my short-term concern.  Its just broke.


>>  Also, its really the sunrpc 
>> rdma transport layer.  It deals specifically with rdma.  It _should_ 
>> know about rdma interfaces and types.
>>     
>
> IOW, this is an issue for the OFA API to address? I agree with Trond that
> there are some hardware-specific warts in there. It might be a good idea
> for us to go through both the client and server code overall - you up for
> doing that? I'm in, if so.
>
>   

Ok.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Tom Talpey]

At 08:44 PM 5/11/2009, Trond Myklebust wrote:
>On Mon, 2009-05-11 at 19:13 -0500, Steve Wise wrote:
>> Trond Myklebust wrote:
>> > On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
>> >   
>> >> Hey Trond,
>> >>
>> >> Will this bug fix make 2.6.30?
>> >>
>> >> Thanks,
>> >>
>> >> Steve.
>> >>     
>> >
>> > Not in the form it is in now. As I've said earlier, I'm not happy about
>> > the sunrpc layer having to circumvent ordinary type checking on
>> > non-sunrpc structures.
>> >
>> > Cheers
>> >     Trond
>> How is it circumventing?  It's currently incorrectly casting a pointer 
>> into a u64.  That seems just broken to me.  Also, its really the sunrpc 
>> rdma transport layer.  It deals specifically with rdma.  It _should_ 
>> know about rdma interfaces and types.
>
>The fact is that I'm simply not interested enough in rdma to tolerate
>hacks. If it isn't done cleanly, in a manner that I can maintain, then
>the whole transport layer comes out...

I know exactly what you want - it's not what the code does now and
it's not an accessor function to set the hardware's u64 field. What's
needed is a new function to manage the entire RDMA triplet, and the
memory registration behind it, in the OFA code side. Put the hardware
goop below the line, IOW. I'll dust up Steve on this.

Tom.

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Trond Myklebust]

On Mon, 2009-05-11 at 21:14 -0400, Tom Talpey wrote:
> At 08:44 PM 5/11/2009, Trond Myklebust wrote:
> >On Mon, 2009-05-11 at 19:13 -0500, Steve Wise wrote:
> >> Trond Myklebust wrote:
> >> > On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
> >> >   
> >> >> Hey Trond,
> >> >>
> >> >> Will this bug fix make 2.6.30?
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Steve.
> >> >>     
> >> >
> >> > Not in the form it is in now. As I've said earlier, I'm not happy about
> >> > the sunrpc layer having to circumvent ordinary type checking on
> >> > non-sunrpc structures.
> >> >
> >> > Cheers
> >> >     Trond
> >> How is it circumventing?  It's currently incorrectly casting a pointer 
> >> into a u64.  That seems just broken to me.  Also, its really the sunrpc 
> >> rdma transport layer.  It deals specifically with rdma.  It _should_ 
> >> know about rdma interfaces and types.
> >
> >The fact is that I'm simply not interested enough in rdma to tolerate
> >hacks. If it isn't done cleanly, in a manner that I can maintain, then
> >the whole transport layer comes out...
> 
> I know exactly what you want - it's not what the code does now and
> it's not an accessor function to set the hardware's u64 field. What's
> needed is a new function to manage the entire RDMA triplet, and the
> memory registration behind it, in the OFA code side. Put the hardware
> goop below the line, IOW. I'll dust up Steve on this.

This does indeed sound like what I'd looking for.

There is a huge difference between having code that depends on well
defined rdma interfaces, and code that depends on rdma hacks. A piece of
code that requires casts from a non-local opaque type into another
protocol-dependent non-local type will definitely fall in the latter
category. I really don't care what the current code does, but a fix for
that code is something that does it _correctly_; it is not yet another
hack, whether or not it fixes a bug in the short term.

   Trond

Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Trond Myklebust wrote:
> On Mon, 2009-05-11 at 21:14 -0400, Tom Talpey wrote:
>   
>> At 08:44 PM 5/11/2009, Trond Myklebust wrote:
>>     
>>> On Mon, 2009-05-11 at 19:13 -0500, Steve Wise wrote:
>>>       
>>>> Trond Myklebust wrote:
>>>>         
>>>>> On Mon, 2009-05-11 at 17:25 -0500, Steve Wise wrote:
>>>>>   
>>>>>           
>>>>>> Hey Trond,
>>>>>>
>>>>>> Will this bug fix make 2.6.30?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Steve.
>>>>>>     
>>>>>>             
>>>>> Not in the form it is in now. As I've said earlier, I'm not happy about
>>>>> the sunrpc layer having to circumvent ordinary type checking on
>>>>> non-sunrpc structures.
>>>>>
>>>>> Cheers
>>>>>     Trond
>>>>>           
>>>> How is it circumventing?  It's currently incorrectly casting a pointer 
>>>> into a u64.  That seems just broken to me.  Also, its really the sunrpc 
>>>> rdma transport layer.  It deals specifically with rdma.  It _should_ 
>>>> know about rdma interfaces and types.
>>>>         
>>> The fact is that I'm simply not interested enough in rdma to tolerate
>>> hacks. If it isn't done cleanly, in a manner that I can maintain, then
>>> the whole transport layer comes out...
>>>       
>> I know exactly what you want - it's not what the code does now and
>> it's not an accessor function to set the hardware's u64 field. What's
>> needed is a new function to manage the entire RDMA triplet, and the
>> memory registration behind it, in the OFA code side. Put the hardware
>> goop below the line, IOW. I'll dust up Steve on this.
>>     
>
> This does indeed sound like what I'd looking for.
>
> There is a huge difference between having code that depends on well
> defined rdma interfaces, and code that depends on rdma hacks. A piece of
> code that requires casts from a non-local opaque type into another
> protocol-dependent non-local type will definitely fall in the latter
> category. I really don't care what the current code does, but a fix for
> that code is something that does it _correctly_; it is not yet another
> hack, whether or not it fixes a bug in the short term.
>
>    Trond
>
>   

Trond, I get your point, and we can certainly work on improving this 
with the rdma developer community.  But removing the one-line-broken 
cast will resolve a current crash situation for 2.6.30.  Can't we get 
this fix in 2.6.30 and work on the API improvements for 2.6.31?  I've 
CCed Roland and the ofa general list to get everyone involved in this 
thread so we can get this API design change going.

I agree we can clean this up moving forward, but lets fix the broken 
2.6.30 code.  

Will this work?

Steve.

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Steve Wise wrote:

 >Trond Myklebust wrote (earlier in this thread):
 >
 > All I should need to know is that I can advertise either dma handles or
 > kernel VAs, and know that I can choose between two functions, say,
 > ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to
 > initialise the ib_send_wr structure correctly.


To align more with the rest of the fast_reg API in ib_verbs.h, I propose:

static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr 
*send_wr, dma_addr_t dma);
static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr 
*send_wr, void *kva);

Thoughts?

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Steve Wise wrote:
> Steve Wise wrote:
>
> >Trond Myklebust wrote (earlier in this thread):
> >
> > All I should need to know is that I can advertise either dma handles or
> > kernel VAs, and know that I can choose between two functions, say,
> > ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to
> > initialise the ib_send_wr structure correctly.
>
>
> To align more with the rest of the fast_reg API in ib_verbs.h, I propose:
>
> static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr 
> *send_wr, dma_addr_t dma);
> static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr 
> *send_wr, void *kva);
>
> Thoughts?
>
>
uncompiled patch:

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index c179318..fb56930 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1940,6 +1940,30 @@ static inline void ib_update_fast_reg_key(struct 
ib_mr *mr, u8 newkey)
 }
 
 /**
+ * ib_init_fast_reg_iova_start_dma - initializes the iova_start field
+ *   based on a dma address supplied by the user.
+ * @wr - struct ib_send_wr pointer to be initialized
+ * @addr - dma_addr_t value to be used as the iova_start
+ */
+static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr *wr,
+                                                  dma_addr_t addr)
+{
+       wr->wr.fast_reg.iova_start = addr;
+}
+
+/**
+ * ib_init_fast_reg_iova_start_kva - initializes the iova_start field
+ *   based on a kernel virtual address supplied by the user.
+ * @wr - struct ib_send_wr pointer to be initialized
+ * @addr - void * address to be used as the iova_start
+ */
+static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr,
+                                                  void *addr)
+{
+       wr->wr.fast_reg.iova_start = (unsigned long)addr;
+}
+
+/**
  * ib_alloc_mw - Allocates a memory window.
  * @pd: The protection domain associated with the memory window.
  */

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Roland Dreier]

 > Trond Myklebust wrote (earlier in this thread):
 > >
 > > All I should need to know is that I can advertise either dma handles or
 > > kernel VAs, and know that I can choose between two functions, say,
 > > ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to
 > > initialise the ib_send_wr structure correctly.

I skimmed the earlier thread, and I have to say that I don't quite see
what the problem with assigning things to a u64 directly is.  You can
use any address you want, and I don't quite understand why using the
correct cast to avoid sign extension or truncation problems is such a
big maintenance burden?

The code below really just looks like obfuscation to me -- are we going
to want to add something like

/**
 * ib_init_fast_reg_iova_start_u64 - initializes the iova_start field
 *   based on a 64-bit address supplied by the user.
 * @wr - struct ib_send_wr pointer to be initialized
 * @addr - void * address to be used as the iova_start
 */
static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr,
						   u64 addr)
{
	wr->wr.fast_reg.iova_start = addr;
}

next, to make sure we don't get confused about assigning a u64 to a u64?
It all looks a bit overcomplicated to me.

 - R.

 > /**
 > + * ib_init_fast_reg_iova_start_dma - initializes the iova_start field
 > + *   based on a dma address supplied by the user.
 > + * @wr - struct ib_send_wr pointer to be initialized
 > + * @addr - dma_addr_t value to be used as the iova_start
 > + */
 > +static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr *wr,
 > +                                                  dma_addr_t addr)
 > +{
 > +       wr->wr.fast_reg.iova_start = addr;
 > +}
 > +
 > +/**
 > + * ib_init_fast_reg_iova_start_kva - initializes the iova_start field
 > + *   based on a kernel virtual address supplied by the user.
 > + * @wr - struct ib_send_wr pointer to be initialized
 > + * @addr - void * address to be used as the iova_start
 > + */
 > +static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr,
 > +                                                  void *addr)
 > +{
 > +       wr->wr.fast_reg.iova_start = (unsigned long)addr;
 > +}
 > +
 > +/**
 >  * ib_alloc_mw - Allocates a memory window.
 >  * @pd: The protection domain associated with the memory window.
 >  */

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Or Gerlitz]

> Trond Myklebust wrote 
>> All I should need to know is that I can advertise either dma handles or kernel VAs

Maybe its obvious to some people here, but may I ask why there's a need 
to post either dma address or kernel virtual address? is it application 
need? hardware (e.g IB vs iWARP vs  vendor implementation) specific? or 
something else?

Or.

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Steve Wise]

Or Gerlitz wrote:
>> Trond Myklebust wrote
>>> All I should need to know is that I can advertise either dma handles 
>>> or kernel VAs
>
> Maybe its obvious to some people here, but may I ask why there's a 
> need to post either dma address or kernel virtual address? is it 
> application need? hardware (e.g IB vs iWARP vs  vendor implementation) 
> specific? or something else?
>
> Or.
>
>

The NFSRDMA transport uses Fast Register Memory Regions.  In this 
particular section of code, the NFSRDMA client is building a fastreg 
work request to bind a page list to a fastreg mr.  You can read about 
this in the IBTA spec on memory management extensions, or in the RDMA 
Verbs draft.


Steve.

Re: [ofa-general] Re: [PATCH 2.6.30] xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.

[Or Gerlitz]

Steve Wise wrote:
> The NFSRDMA transport uses Fast Register Memory Regions.  In this
> particular section of code, the NFSRDMA client is building a fastreg
> work request to bind a page list to a fastreg mr.  You can read about
> this in the IBTA spec on memory management extensions, or in the RDMA Verbs draft.

Hi Steve,

I was aware for the context being fastreg work request. I was thinking
that the spec mandates either dma addr or kva on the iova but from your reply
I assume to be wrong, thanks.

Or.