Re: [SPDK] SCST Usermode iSCSI Storage Server now handles Intel SPDK backing storage

[Vladislav Bolkhovitin <vst at vlnb.net>]

[-- Attachment #1: Type: text/plain, Size: 2872 bytes --]

Wonderful!

The only note would be that, as I have already mentioned before, tcmu does data copy
between user mode module and kernel, so usage with SCST zero-copy scst_user instead
would be more performance efficient.

Thanks,
Vlad

David Butterfield wrote on 09/05/2017 12:25 AM:
> The SCST Usermode iSCSI Storage Server can now utilize backing storage through
> the Intel Storage Performance Development Kit (SPDK) API.
> 
> The SCST Usermode Server is a port of about 80 KLOC of the SCST Linux kernel
> software to run entirely in usermode on an unmodified kernel, with virtually
> no change to the existing SCST source code.
> 
> The diagram on the left side of this PDF page compares the usual kernel-based
> SCST configuration [blue box] with the configuration adapted for usermode
> [purple box]
> https://github.com/DavidButterfield/SCST-Usermode-Adaptation/blob/usermode/usermode/scstu_tcmur.pdf
> 
> The diagram on the right side of that page illustrates the datapath from
> Initiator to backing storage API -- showing paths through LIO (in-kernel), and
> through Usermode SCST [purple box].  The Usermode SCST server can access
> backing storage through any of these interfaces: preadv(2) and pwritev(2),
> aio(7), or the tcmu-runner backstorage API [red arrow].
> 
> The tcmu-runner backstorage API is a usermode interface point between the
> kernel-based LIO facility and usermode backstore-specific handlers.  The
> tcmu-runner project implements backstore handlers for Ceph/rbd, Gluster/glfs,
> and QEMU/qcow [green box].  I have re-used that same API for Usermode SCST so
> that it can make use of those same backstore handlers [red arrow].
> 
> I have also implemented two additional backstore handlers: a "ram" driver that
> uses mmap(2) either anonymously or with a persistent backing file; and most
> recently, an interface module to the Intel Storage Performance Development Kit
> (SPDK) [red circle -- note that the new SPDK module is a prototype, presently
> functional with Usermode SCST, but not yet through the LIO datapath].
> 
> Project is at https://github.com/DavidButterfield/SCST-Usermode-Adaptation --
> the README there has a few diagrams and a link to a technical paper.  The new
> SPDK backstore handler is in usermode/spdk.c
> 
> The paper starts by describing the port of SCST from the Linux kernel to
> usermode, including diagrams showing how this was done without changing the
> SCST source code.  Next I specify the configuration used for performance
> measurements, followed by plots and analysis interpreting the results.  I
> introduce an experimental "Adaptive Nagle" algorithm to improve performance of
> small Read operations.  An appendix develops a performance model that attempts
> to maintain some intuition in a fairly complicated analysis.
> 
> David Butterfield