[SPDK] Re: SPDK socket abstraction layer

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 8451 bytes --]

On Wed, 2019-10-30 at 21:55 +0200, Sasha Kotchubievsky wrote:
> Hi Ben, 
> 
> Great list of patches,
> This work will , definitely, take NVME-OF TCP at the next level.
> 
> We, also, work on zero-copy (TX) in initiator side based on MSG_ZEROCOPY.
> Preliminary results are great. In the target side, we still investigate the
> solution.

I'm very interested to hear about your work here. I've been doing it primarily
on the target side and the preliminary results we're seeing are that it's slower
for 4K I/O. That's not really what I expected at all, and I feel like I must be
missing something with getting the page pinning to hit the fast path
consistently. It's early days with all of these things, so it's a safe
assumption that I either coded something incorrectly or the system isn't
configured right.

> 
> We run a lot of tests and see great potential for zero-copy. It look like a
> real bottleneck. On the send side, it can be removed with POSIX interface
> (MSG_ZEROCOPY), in receive side, it needs deep integration between TCP stack
> and SPDK. Next week we will have internal brain-storming, and, I hope, on dev
> meetup, I'll be ready for discussions.
> 
> We will invest in both directions: user-space path and in Linux Kernel
> path.  But, in user-space area, at this stage, VPP is out of our interest. 
> 
> Best regards
> Sasha  
> 
> -----Original Message-----
> From: Walker, Benjamin <benjamin.walker(a)intel.com> 
> Sent: Wednesday, October 30, 2019 8:47 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Re: SPDK socket abstraction layer
> 
> On Wed, 2019-10-30 at 17:54 +0000, Harris, James R wrote:
> > Hi Sasha,
> > 
> > Tomek is only talking about the VPP implementation.  There are no 
> > plans to remove the socket abstraction layer.  If anything, the 
> > project needs to look at extending it in ways as you suggested.
> 
> To expand on this, there's a lot of activity right now in the SPDK sock
> abstraction layer to begin to implement asynchronous operations, zero copy
> operations, etc. For example, see:
> 
> Asynchronous writev
> https://review.gerrithub.io/c/spdk/spdk/+/470523
> 
> MSG_ZEROCOPY use in the posix implementation
> https://review.gerrithub.io/c/spdk/spdk/+/471752
> 
> A new sock implementation based on io_uring/libaio:
> https://review.gerrithub.io/c/spdk/spdk/+/471314
> 
> And a new sock implementation based on Seastar:
> https://review.gerrithub.io/c/spdk/spdk/+/466629
> 
> So not only is the sock abstraction layer sticking around, but it's getting a
> lot of focus going forward. There is a lot of innovation happening in the
> Linux kernel around networking at all layers that we need to keep up with.
> 
> One thing I would like community feedback on is what to do about the current
> VPP implementation. As we make improvements and additions to the sock
> abstraction, it will necessarily require updates to the VPP implementation. We
> can of course continue to make those, but does the community see value in
> maintaining support here? I'd really love to see someone take up the mantle on
> VPP if they believe there is value that we just haven't been able to unlock
> yet, but absent that it's just a maintenance burden.
> 
> Personally speaking, it would be easier for me, as someone trying to evolve
> the sock abstraction layer, to drop VPP. That's one less implementation that I
> then have to go update and test each time. But I'm very open to opinions and
> feedback here if anyone has something to say. SPDK obviously can't just drop
> support without strong consensus and a considerable amount of forewarning.
> 
> Thanks,
> Ben
> 
> > -Jim
> > 
> > 
> > On 10/30/19, 10:50 AM, "Sasha Kotchubievsky" 
> > <sashakot(a)dev.mellanox.co.il>
> > wrote:
> > 
> >     Hi Tomek,
> >     
> >     Are you looking for community feedback regarding VPP implementation of
> > TCP
> >     stack, or about having socket abstraction layer in SPDK?
> >     I think, socket abstraction layer is critical for future 
> > integration between
> >     SPDK and user-space stacks. In Mellanox, we're evaluating integration
> >     between VMA (https://github.com/Mellanox/libvma) and SPDK. 
> > Although, VMA can
> >     be used as replacement for Kernel implementation of Posix socket 
> > interface,
> >     we see great potential in "deep" integration, which definitely needs
> > keep
> >     existing abstraction layer. For example, one of potential improvements
> > can
> >     be zero-copy in RX (receive) flow. I don't see how that can be
> > implemented
> >     on top of Linux Kernel stack. 
> >     
> >     Best regards
> >     Sasha
> >     
> >     -----Original Message-----
> >     From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
> >     Sent: Monday, October 21, 2019 3:01 PM
> >     To: Storage Performance Development Kit <spdk(a)lists.01.org>
> >     Subject: [SPDK] SPDK socket abstraction layer
> >     
> >     Hello everyone,
> >     
> >     Summary:
> >     
> >     With this message I wanted to update SPDK community on state of VPP
> > socket
> >     abstraction as of SPDK 19.07 release.
> >     At this time there does not seem to be a clear efficiency 
> > improvements with
> >     VPP. There is no further work planned on SPDK and VPP integration.
> >     
> >     Details:
> >     
> >     As some of you may remember, SPDK 18.04 release introduced support for
> >     alternative socket types. Along with that release, Vector Packet 
> > Processing
> >     (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by
> >     expanding socket abstraction to use VPP Communications Library (VCL).
> > TCP/IP
> >     stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early 
> > stages back
> >     then and has seen improvements throughout the last year.
> >     
> >     To better use VPP capabilities, following fruitful collaboration with
> > VPP
> >     team, in SPDK 19.07, this implementation was changed from VCL to 
> > VPP Session
> >     API from VPP 19.04.2.
> >     
> >     VPP socket abstraction has met some challenges due to inherent design of
> >     both projects, in particular related to running separate processes and
> >     memory copies.
> >     Seeing improvements from original implementation was encouraging, yet
> >     measuring against posix socket abstraction (taking into 
> > consideration entire
> >     system, i.e. both processes), results are comparable. In other
> > words,  at
> >     this time there does not seem to be a clear benefit of either socket
> >     abstraction from standpoint of CPU efficiency or IOPS.
> >     
> >     With this message I just wanted to update SPDK community on state 
> > of socket
> >     abstraction layers as of SPDK 19.07 release. Each SPDK release 
> > always brings
> >     improvements to the abstraction and its implementations, with 
> > exciting work
> >     on more efficient use of kernel TCP stack - changes in SPDK 19.10 and
> > SPDK
> >     20.01.
> >     
> >     However there is no active involvement at this point around VPP
> >     implementation of socket abstraction in SPDK. Contributions in 
> > this area are
> >     always welcome. In case you're interested in implementing further
> >     enhancements of VPP and SPDK integration feel free to reply, or to use
> > one
> >     of the many SPDK community communications
> >     channels<https://spdk.io/community/>;;.
> >     
> >     Thanks,
> >     Tomek
> >     
> >     _______________________________________________
> >     SPDK mailing list -- spdk(a)lists.01.org
> >     To unsubscribe send an email to spdk-leave(a)lists.01.org
> >     _______________________________________________
> >     SPDK mailing list -- spdk(a)lists.01.org
> >     To unsubscribe send an email to spdk-leave(a)lists.01.org
> >     
> > 
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> > spdk-leave(a)lists.01.org
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org