Re: Integration of SCST in the mainstream Linux kernel

[Erez Zilber <erezz@Voltaire.COM>]

Bart Van Assche wrote:
> On Feb 18, 2008 10:43 AM, Erez Zilber <erezz@voltaire.com> wrote:
>   
>> If you use a high value for FirstBurstLength, all (or most) of your data
>> will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA
>> engine, so you miss the advantage of IB.
>>     
>
> Hello Erez,
>
> Did you notice the e-mail Roland Dreier wrote on Februari 6, 2008 ?
> This is what Roland wrote:
>   
>> I think the confusion here is caused by a slight misuse of the term
>> "RDMA".  It is true that all data is always transported over an
>> InfiniBand connection when iSER is used, but not all such transfers
>> are one-sided RDMA operations; some data can be transferred using
>> send/receive operations.
>>     
>
>   
Yes, I saw that. I tried to give an explanation with more details.

> Or: data sent during the first burst is not transferred via one-sided
> remote memory reads or writes but via two-sided send/receive
> operations. At least on my setup, these operations are as fast as
> one-sided remote memory reads or writes. As an example, I obtained the
> following numbers on my setup (SDR 4x network);
> ib_write_bw: 933 MB/s.
> ib_read_bw: 905 MB/s.
> ib_send_bw: 931 MB/s.
>
>   
According to these numbers one can think that you don't need RDMA at
all, just send iSCSI PDUs over IB. The benchmarks that you use are
synthetic IB benchmarks that are not equivalent to iSCSI over iSER. They
just send IB packets. I'm not surprised that you got more or less the
same performance because, AFAIK, ib_send_bw doesn't copy data (unlike
iSCSI that has to copy data that is sent/received without RDMA).

When you use RDMA with iSCSI (i.e. iSER), you don't need to create iSCSI
PDUs and process them. The CPU is not busy as it is with iSCSI over TCP
because no data copies are required. Another advantage is that you don't
need header/data digest because the IB HW does that.

Erez