From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5622154130923995713==" MIME-Version: 1.0 From: Sasha Kotchubievsky Subject: [SPDK] Re: SPDK socket abstraction layer Date: Wed, 30 Oct 2019 21:55:08 +0200 Message-ID: <010401d58f5b$efae6f80$cf0b4e80$@dev.mellanox.co.il> In-Reply-To: b372372e456ce2f236845058d42ae283af8ca662.camel@intel.com List-ID: To: spdk@lists.01.org --===============5622154130923995713== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Ben, = Great list of patches, This work will , definitely, take NVME-OF TCP at the next level. We, also, work on zero-copy (TX) in initiator side based on MSG_ZEROCOPY. P= reliminary results are great. In the target side, we still investigate the = solution. We run a lot of tests and see great potential for zero-copy. It look like a= real bottleneck. On the send side, it can be removed with POSIX interface = (MSG_ZEROCOPY), in receive side, it needs deep integration between TCP stac= k and SPDK. Next week we will have internal brain-storming, and, I hope, on= dev meetup, I'll be ready for discussions. We will invest in both directions: user-space path and in Linux Kernel path= . But, in user-space area, at this stage, VPP is out of our interest. = Best regards Sasha = -----Original Message----- From: Walker, Benjamin = Sent: Wednesday, October 30, 2019 8:47 PM To: spdk(a)lists.01.org Subject: [SPDK] Re: SPDK socket abstraction layer On Wed, 2019-10-30 at 17:54 +0000, Harris, James R wrote: > Hi Sasha, > = > Tomek is only talking about the VPP implementation. There are no = > plans to remove the socket abstraction layer. If anything, the = > project needs to look at extending it in ways as you suggested. To expand on this, there's a lot of activity right now in the SPDK sock abs= traction layer to begin to implement asynchronous operations, zero copy ope= rations, etc. For example, see: Asynchronous writev https://review.gerrithub.io/c/spdk/spdk/+/470523 MSG_ZEROCOPY use in the posix implementation https://review.gerrithub.io/c/spdk/spdk/+/471752 A new sock implementation based on io_uring/libaio: https://review.gerrithub.io/c/spdk/spdk/+/471314 And a new sock implementation based on Seastar: https://review.gerrithub.io/c/spdk/spdk/+/466629 So not only is the sock abstraction layer sticking around, but it's getting= a lot of focus going forward. There is a lot of innovation happening in th= e Linux kernel around networking at all layers that we need to keep up with. One thing I would like community feedback on is what to do about the curren= t VPP implementation. As we make improvements and additions to the sock abs= traction, it will necessarily require updates to the VPP implementation. We= can of course continue to make those, but does the community see value in = maintaining support here? I'd really love to see someone take up the mantle= on VPP if they believe there is value that we just haven't been able to un= lock yet, but absent that it's just a maintenance burden. Personally speaking, it would be easier for me, as someone trying to evolve= the sock abstraction layer, to drop VPP. That's one less implementation th= at I then have to go update and test each time. But I'm very open to opinio= ns and feedback here if anyone has something to say. SPDK obviously can't j= ust drop support without strong consensus and a considerable amount of fore= warning. Thanks, Ben > = > -Jim > = > = > =EF=BB=BFOn 10/30/19, 10:50 AM, "Sasha Kotchubievsky" = > > wrote: > = > Hi Tomek, > = > Are you looking for community feedback regarding VPP implementation o= f TCP > stack, or about having socket abstraction layer in SPDK? > I think, socket abstraction layer is critical for future = > integration between > SPDK and user-space stacks. In Mellanox, we're evaluating integration > between VMA (https://github.com/Mellanox/libvma) and SPDK. = > Although, VMA can > be used as replacement for Kernel implementation of Posix socket = > interface, > we see great potential in "deep" integration, which definitely needs = keep > existing abstraction layer. For example, one of potential improvement= s can > be zero-copy in RX (receive) flow. I don't see how that can be implem= ented > on top of Linux Kernel stack. = > = > Best regards > Sasha > = > -----Original Message----- > From: Zawadzki, Tomasz = > Sent: Monday, October 21, 2019 3:01 PM > To: Storage Performance Development Kit > Subject: [SPDK] SPDK socket abstraction layer > = > Hello everyone, > = > Summary: > = > With this message I wanted to update SPDK community on state of VPP s= ocket > abstraction as of SPDK 19.07 release. > At this time there does not seem to be a clear efficiency = > improvements with > VPP. There is no further work planned on SPDK and VPP integration. > = > Details: > = > As some of you may remember, SPDK 18.04 release introduced support for > alternative socket types. Along with that release, Vector Packet = > Processing > (VPP) 18.01 was integrated with SPDK, by > expanding socket abstraction to use VPP Communications Library (VCL). > TCP/IP > stack in VPP was in early = > stages back > then and has seen improvements throughout the last year. > = > To better use VPP capabilities, following fruitful collaboration with= VPP > team, in SPDK 19.07, this implementation was changed from VCL to = > VPP Session > API from VPP 19.04.2. > = > VPP socket abstraction has met some challenges due to inherent design= of > both projects, in particular related to running separate processes and > memory copies. > Seeing improvements from original implementation was encouraging, yet > measuring against posix socket abstraction (taking into = > consideration entire > system, i.e. both processes), results are comparable. In other words,= at > this time there does not seem to be a clear benefit of either socket > abstraction from standpoint of CPU efficiency or IOPS. > = > With this message I just wanted to update SPDK community on state = > of socket > abstraction layers as of SPDK 19.07 release. Each SPDK release = > always brings > improvements to the abstraction and its implementations, with = > exciting work > on more efficient use of kernel TCP stack - changes in SPDK 19.10 and= SPDK > 20.01. > = > However there is no active involvement at this point around VPP > implementation of socket abstraction in SPDK. Contributions in = > this area are > always welcome. In case you're interested in implementing further > enhancements of VPP and SPDK integration feel free to reply, or to us= e one > of the many SPDK community communications > channels;. > = > Thanks, > Tomek > = > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > = > = > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to = > spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org --===============5622154130923995713==--