[SPDK] Re: SPDK socket abstraction layer

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6187 bytes --]

On Wed, 2019-10-30 at 17:54 +0000, Harris, James R wrote:
> Hi Sasha,
> 
> Tomek is only talking about the VPP implementation.  There are no plans to
> remove the socket abstraction layer.  If anything, the project needs to look
> at extending it in ways as you suggested.

To expand on this, there's a lot of activity right now in the SPDK sock
abstraction layer to begin to implement asynchronous operations, zero copy
operations, etc. For example, see:

Asynchronous writev
https://review.gerrithub.io/c/spdk/spdk/+/470523

MSG_ZEROCOPY use in the posix implementation
https://review.gerrithub.io/c/spdk/spdk/+/471752

A new sock implementation based on io_uring/libaio:
https://review.gerrithub.io/c/spdk/spdk/+/471314

And a new sock implementation based on Seastar:
https://review.gerrithub.io/c/spdk/spdk/+/466629

So not only is the sock abstraction layer sticking around, but it's getting a
lot of focus going forward. There is a lot of innovation happening in the Linux
kernel around networking at all layers that we need to keep up with.

One thing I would like community feedback on is what to do about the current VPP
implementation. As we make improvements and additions to the sock abstraction,
it will necessarily require updates to the VPP implementation. We can of course
continue to make those, but does the community see value in maintaining support
here? I'd really love to see someone take up the mantle on VPP if they believe
there is value that we just haven't been able to unlock yet, but absent that
it's just a maintenance burden.

Personally speaking, it would be easier for me, as someone trying to evolve the
sock abstraction layer, to drop VPP. That's one less implementation that I then
have to go update and test each time. But I'm very open to opinions and feedback
here if anyone has something to say. SPDK obviously can't just drop support
without strong consensus and a considerable amount of forewarning.

Thanks,
Ben

> 
> -Jim
> 
> 
> On 10/30/19, 10:50 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il>
> wrote:
> 
>     Hi Tomek,
>     
>     Are you looking for community feedback regarding VPP implementation of TCP
>     stack, or about having socket abstraction layer in SPDK?
>     I think, socket abstraction layer is critical for future integration
> between
>     SPDK and user-space stacks. In Mellanox, we're evaluating integration
>     between VMA (https://github.com/Mellanox/libvma) and SPDK. Although, VMA
> can
>     be used as replacement for Kernel implementation of Posix socket
> interface,
>     we see great potential in "deep" integration, which definitely needs keep
>     existing abstraction layer. For example, one of potential improvements can
>     be zero-copy in RX (receive) flow. I don't see how that can be implemented
>     on top of Linux Kernel stack. 
>     
>     Best regards
>     Sasha
>     
>     -----Original Message-----
>     From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
>     Sent: Monday, October 21, 2019 3:01 PM
>     To: Storage Performance Development Kit <spdk(a)lists.01.org>
>     Subject: [SPDK] SPDK socket abstraction layer
>     
>     Hello everyone,
>     
>     Summary:
>     
>     With this message I wanted to update SPDK community on state of VPP socket
>     abstraction as of SPDK 19.07 release.
>     At this time there does not seem to be a clear efficiency improvements
> with
>     VPP. There is no further work planned on SPDK and VPP integration.
>     
>     Details:
>     
>     As some of you may remember, SPDK 18.04 release introduced support for
>     alternative socket types. Along with that release, Vector Packet
> Processing
>     (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by
>     expanding socket abstraction to use VPP Communications Library (VCL).
> TCP/IP
>     stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early stages
> back
>     then and has seen improvements throughout the last year.
>     
>     To better use VPP capabilities, following fruitful collaboration with VPP
>     team, in SPDK 19.07, this implementation was changed from VCL to VPP
> Session
>     API from VPP 19.04.2.
>     
>     VPP socket abstraction has met some challenges due to inherent design of
>     both projects, in particular related to running separate processes and
>     memory copies.
>     Seeing improvements from original implementation was encouraging, yet
>     measuring against posix socket abstraction (taking into consideration
> entire
>     system, i.e. both processes), results are comparable. In other words,  at
>     this time there does not seem to be a clear benefit of either socket
>     abstraction from standpoint of CPU efficiency or IOPS.
>     
>     With this message I just wanted to update SPDK community on state of
> socket
>     abstraction layers as of SPDK 19.07 release. Each SPDK release always
> brings
>     improvements to the abstraction and its implementations, with exciting
> work
>     on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK
>     20.01.
>     
>     However there is no active involvement at this point around VPP
>     implementation of socket abstraction in SPDK. Contributions in this area
> are
>     always welcome. In case you're interested in implementing further
>     enhancements of VPP and SPDK integration feel free to reply, or to use one
>     of the many SPDK community communications
>     channels<https://spdk.io/community/>;.
>     
>     Thanks,
>     Tomek
>     
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org