Re: Integration of SCST in the mainstream Linux kernel

[Vladislav Bolkhovitin <vst@vlnb.net>]

FUJITA Tomonori wrote:
> On Tue, 29 Jan 2008 13:31:52 -0800
> Roland Dreier <rdreier@cisco.com> wrote:
> 
> 
>> > .                           .   STGT read     SCST read    .    STGT read      SCST read    .
>> > .                           .  performance   performance   . performance    performance   .
>> > .                           .  (0.5K, MB/s)  (0.5K, MB/s)  .   (1 MB, MB/s)   (1 MB, MB/s)  .
>> > . iSER     (8 Gb/s network) .     250            N/A       .       360           N/A       .
>> > . SRP      (8 Gb/s network) .     N/A            421       .       N/A           683       .
>>
>> > On the comparable figures, which only seem to be IPoIB they're showing a
>> > 13-18% variance, aren't they?  Which isn't an incredible difference.
>>
>>Maybe I'm all wet, but I think iSER vs. SRP should be roughly
>>comparable.  The exact formatting of various messages etc. is
>>different but the data path using RDMA is pretty much identical.  So
>>the big difference between STGT iSER and SCST SRP hints at some big
>>difference in the efficiency of the two implementations.
> 
> 
> iSER has parameters to limit the maximum size of RDMA (it needs to
> repeat RDMA with a poor configuration)?
> 
> 
> Anyway, here's the results from Robin Humble:
> 
> iSER to 7G ramfs, x86_64, centos4.6, 2.6.22 kernels, git tgtd,
> initiator end booted with mem=512M, target with 8G ram
> 
>  direct i/o dd
>   write/read  800/751 MB/s
>     dd if=/dev/zero of=/dev/sdc bs=1M count=5000 oflag=direct
>     dd of=/dev/null if=/dev/sdc bs=1M count=5000 iflag=direct
> 
> http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg13502.html
> 
> I think that STGT is pretty fast with the fast backing storage. 

How fast SCST will be on the same hardware?

> I don't think that there is the notable perfornace difference between
> kernel-space and user-space SRP (or ISER) implementations about moving
> data between hosts. IB is expected to enable user-space applications
> to move data between hosts quickly (if not, what can IB provide us?).
> 
> I think that the question is how fast user-space applications can do
> I/Os ccompared with I/Os in kernel space. STGT is eager for the advent
> of good asynchronous I/O and event notification interfances.
> 
> One more possible optimization for STGT is zero-copy data
> transfer. STGT uses pre-registered buffers and move data between page
> cache and thsse buffers, and then does RDMA transfer. If we implement
> own caching mechanism to use pre-registered buffers directly with (AIO
> and O_DIRECT), then STGT can move data without data copies.

Great! So, you are going to duplicate Linux page cache in the user 
space. You will continue keeping the in-kernel code as small as possible 
and its mainteinership effort as low as possible by the cost that the 
user space part's code size and complexity (and, hence, its 
mainteinership effort) will rocket to the sky. Apparently, this doesn't 
look like a good design decision.

> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>