* Is it possible to transfer a large file between two computers using RDMA UD?
@ 2020-01-02 14:47 Terry Toole
2020-01-02 18:29 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Terry Toole @ 2020-01-02 14:47 UTC (permalink / raw)
To: linux-rdma
Hi,
Is it possible to transfer a large file, say 25GB, between two computers using
RDMA UD, and have an exact copy of the original file on the receiving side? My
understanding is that the order of the messages is not guaranteed with UD.
But I thought that if I only use one QP I could ensure that the ordering of the
data will be predictable.
This is a proof of concept at this time. My thought was to load the file into
memory on the client side and advance the address pointer after each
transfer. The same scheme would be used on the receiving side. That is,
allocate a large region of memory and increment the address pointer
after each message is received.
I have been using the program ib_send_bw from perftest as a starting point.
After increasing the buffer size on the client side and advancing sg_list->addr
on the client side, I seem to be sending what I intend to send. At the present
time I am not yet advancing sg_list->addr on the server side. When I print out
the first 100B of data after each transfer, I often see the data I
expect, but not
always.
The commands I've been using to launch my modified version of ib_send_bw are
client:
./ib_send_bw -R -d mlx5_0 -c UD -i 1 -F -q 1 --mmap=myInputfile.dat
--report_gbits -n 1000 10.10.10.3
server:
./ib_send_bw -R -r 50 -d mlx5_0 -c UD -i 1 -q 1 -F -n 1000 --report_gbits
The "-n 1000" is just for debugging. It will be increased later.
Am I going about this in the correct way? Should I be able to copy a file using
RDMA UD? Any thoughts or suggestions would be most appreciated.
Thanks,
Terry
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Is it possible to transfer a large file between two computers using RDMA UD?
2020-01-02 14:47 Is it possible to transfer a large file between two computers using RDMA UD? Terry Toole
@ 2020-01-02 18:29 ` Jason Gunthorpe
2020-01-02 19:11 ` Terry Toole
0 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2020-01-02 18:29 UTC (permalink / raw)
To: Terry Toole; +Cc: linux-rdma
On Thu, Jan 02, 2020 at 09:47:06AM -0500, Terry Toole wrote:
> Hi,
> Is it possible to transfer a large file, say 25GB, between two computers using
> RDMA UD, and have an exact copy of the original file on the receiving side? My
> understanding is that the order of the messages is not guaranteed with UD.
> But I thought that if I only use one QP I could ensure that the ordering of the
> data will be predictable.
It is not guaranteed to be in order.
Why would you do this anyhow? The overhead to use RC is pretty small
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Is it possible to transfer a large file between two computers using RDMA UD?
2020-01-02 18:29 ` Jason Gunthorpe
@ 2020-01-02 19:11 ` Terry Toole
2020-01-02 19:25 ` Hefty, Sean
0 siblings, 1 reply; 6+ messages in thread
From: Terry Toole @ 2020-01-02 19:11 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: linux-rdma
On Thu, Jan 2, 2020 at 1:29 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Thu, Jan 02, 2020 at 09:47:06AM -0500, Terry Toole wrote:
> > Hi,
> > Is it possible to transfer a large file, say 25GB, between two computers using
> > RDMA UD, and have an exact copy of the original file on the receiving side? My
> > understanding is that the order of the messages is not guaranteed with UD.
> > But I thought that if I only use one QP I could ensure that the ordering of the
> > data will be predictable.
>
> It is not guaranteed to be in order.
>
> Why would you do this anyhow? The overhead to use RC is pretty small
>
So even if I am using only 1 QP, it is still not guaranteed to be in
order. Ok. Thanks.
The reason I am interested in doing this is that I am working on a
project where we
want to transfer data between an FPGA and a computer. Memory needs to be added
on the FPGA side to buffer data for retries when RC is used. Adding
memory makes
the electronics on the sending side more complex. If we could transfer
the data in UD mode, it may simplify other aspects of the project.
Thanks again for your response,
Terry
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Is it possible to transfer a large file between two computers using RDMA UD?
2020-01-02 19:11 ` Terry Toole
@ 2020-01-02 19:25 ` Hefty, Sean
2020-01-02 19:31 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Hefty, Sean @ 2020-01-02 19:25 UTC (permalink / raw)
To: Terry Toole, Jason Gunthorpe; +Cc: linux-rdma@vger.kernel.org
> > > Is it possible to transfer a large file, say 25GB, between two computers using
> > > RDMA UD, and have an exact copy of the original file on the receiving side? My
> > > understanding is that the order of the messages is not guaranteed with UD.
> > > But I thought that if I only use one QP I could ensure that the ordering of the
> > > data will be predictable.
> >
> > It is not guaranteed to be in order.
> >
> > Why would you do this anyhow? The overhead to use RC is pretty small
> >
> So even if I am using only 1 QP, it is still not guaranteed to be in
> order. Ok. Thanks.
The spec doesn't guarantee ordering, though I doubt messages would ever be unordered in practice. You could transfer a sequence number as immediate data to allow the receiver to verify (and correct) packet ordering, and detect lost packets.
- Sean
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Is it possible to transfer a large file between two computers using RDMA UD?
2020-01-02 19:25 ` Hefty, Sean
@ 2020-01-02 19:31 ` Jason Gunthorpe
2020-01-02 19:45 ` Terry Toole
0 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2020-01-02 19:31 UTC (permalink / raw)
To: Hefty, Sean; +Cc: Terry Toole, linux-rdma@vger.kernel.org
On Thu, Jan 02, 2020 at 07:25:34PM +0000, Hefty, Sean wrote:
> > > > Is it possible to transfer a large file, say 25GB, between two computers using
> > > > RDMA UD, and have an exact copy of the original file on the receiving side? My
> > > > understanding is that the order of the messages is not guaranteed with UD.
> > > > But I thought that if I only use one QP I could ensure that the ordering of the
> > > > data will be predictable.
> > >
> > > It is not guaranteed to be in order.
> > >
> > > Why would you do this anyhow? The overhead to use RC is pretty small
> > >
> > So even if I am using only 1 QP, it is still not guaranteed to be in
> > order. Ok. Thanks.
>
> The spec doesn't guarantee ordering, though I doubt messages would
> ever be unordered in practice. You could transfer a sequence number
> as immediate data to allow the receiver to verify (and correct)
> packet ordering, and detect lost packets.
They could always become lost though, due to a link BER or
something. And that is only on IB, not Ethernet.
I don't think you can avoid needing retransmit on the sender...
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Is it possible to transfer a large file between two computers using RDMA UD?
2020-01-02 19:31 ` Jason Gunthorpe
@ 2020-01-02 19:45 ` Terry Toole
0 siblings, 0 replies; 6+ messages in thread
From: Terry Toole @ 2020-01-02 19:45 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: Hefty, Sean, linux-rdma@vger.kernel.org
On Thu, Jan 2, 2020 at 2:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Thu, Jan 02, 2020 at 07:25:34PM +0000, Hefty, Sean wrote:
> > > > > Is it possible to transfer a large file, say 25GB, between two computers using
> > > > > RDMA UD, and have an exact copy of the original file on the receiving side? My
> > > > > understanding is that the order of the messages is not guaranteed with UD.
> > > > > But I thought that if I only use one QP I could ensure that the ordering of the
> > > > > data will be predictable.
> > > >
> > > > It is not guaranteed to be in order.
> > > >
> > > > Why would you do this anyhow? The overhead to use RC is pretty small
> > > >
> > > So even if I am using only 1 QP, it is still not guaranteed to be in
> > > order. Ok. Thanks.
> >
> > The spec doesn't guarantee ordering, though I doubt messages would
> > ever be unordered in practice. You could transfer a sequence number
> > as immediate data to allow the receiver to verify (and correct)
> > packet ordering, and detect lost packets.
>
> They could always become lost though, due to a link BER or
> something. And that is only on IB, not Ethernet.
>
> I don't think you can avoid needing retransmit on the sender...
>
My plan was to try to send large files between two computers.
From many transfers, I was intending to get an estimate of
what types of losses we could reasonably expect. From my reading,
I got the impression that with a point to point connection, the expected
message loss with RDMA UD was quite low.
Terry
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-02 19:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-01-02 14:47 Is it possible to transfer a large file between two computers using RDMA UD? Terry Toole
2020-01-02 18:29 ` Jason Gunthorpe
2020-01-02 19:11 ` Terry Toole
2020-01-02 19:25 ` Hefty, Sean
2020-01-02 19:31 ` Jason Gunthorpe
2020-01-02 19:45 ` Terry Toole
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox