From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0886839989730869651==" MIME-Version: 1.0 From: Walker, Benjamin Subject: [SPDK] Re: SPDK socket abstraction layer Date: Wed, 30 Oct 2019 20:28:43 +0000 Message-ID: In-Reply-To: 010401d58f5b$efae6f80$cf0b4e80$@dev.mellanox.co.il List-ID: To: spdk@lists.01.org --===============0886839989730869651== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, 2019-10-30 at 21:55 +0200, Sasha Kotchubievsky wrote: > Hi Ben, = > = > Great list of patches, > This work will , definitely, take NVME-OF TCP at the next level. > = > We, also, work on zero-copy (TX) in initiator side based on MSG_ZEROCOPY. > Preliminary results are great. In the target side, we still investigate t= he > solution. I'm very interested to hear about your work here. I've been doing it primar= ily on the target side and the preliminary results we're seeing are that it's s= lower for 4K I/O. That's not really what I expected at all, and I feel like I mus= t be missing something with getting the page pinning to hit the fast path consistently. It's early days with all of these things, so it's a safe assumption that I either coded something incorrectly or the system isn't configured right. > = > We run a lot of tests and see great potential for zero-copy. It look like= a > real bottleneck. On the send side, it can be removed with POSIX interface > (MSG_ZEROCOPY), in receive side, it needs deep integration between TCP st= ack > and SPDK. Next week we will have internal brain-storming, and, I hope, on= dev > meetup, I'll be ready for discussions. > = > We will invest in both directions: user-space path and in Linux Kernel > path. But, in user-space area, at this stage, VPP is out of our interest= . = > = > Best regards > Sasha = > = > -----Original Message----- > From: Walker, Benjamin = > Sent: Wednesday, October 30, 2019 8:47 PM > To: spdk(a)lists.01.org > Subject: [SPDK] Re: SPDK socket abstraction layer > = > On Wed, 2019-10-30 at 17:54 +0000, Harris, James R wrote: > > Hi Sasha, > > = > > Tomek is only talking about the VPP implementation. There are no = > > plans to remove the socket abstraction layer. If anything, the = > > project needs to look at extending it in ways as you suggested. > = > To expand on this, there's a lot of activity right now in the SPDK sock > abstraction layer to begin to implement asynchronous operations, zero copy > operations, etc. For example, see: > = > Asynchronous writev > https://review.gerrithub.io/c/spdk/spdk/+/470523 > = > MSG_ZEROCOPY use in the posix implementation > https://review.gerrithub.io/c/spdk/spdk/+/471752 > = > A new sock implementation based on io_uring/libaio: > https://review.gerrithub.io/c/spdk/spdk/+/471314 > = > And a new sock implementation based on Seastar: > https://review.gerrithub.io/c/spdk/spdk/+/466629 > = > So not only is the sock abstraction layer sticking around, but it's getti= ng a > lot of focus going forward. There is a lot of innovation happening in the > Linux kernel around networking at all layers that we need to keep up with. > = > One thing I would like community feedback on is what to do about the curr= ent > VPP implementation. As we make improvements and additions to the sock > abstraction, it will necessarily require updates to the VPP implementatio= n. We > can of course continue to make those, but does the community see value in > maintaining support here? I'd really love to see someone take up the mant= le on > VPP if they believe there is value that we just haven't been able to unlo= ck > yet, but absent that it's just a maintenance burden. > = > Personally speaking, it would be easier for me, as someone trying to evol= ve > the sock abstraction layer, to drop VPP. That's one less implementation t= hat I > then have to go update and test each time. But I'm very open to opinions = and > feedback here if anyone has something to say. SPDK obviously can't just d= rop > support without strong consensus and a considerable amount of forewarning. > = > Thanks, > Ben > = > > -Jim > > = > > = > > =EF=BB=BFOn 10/30/19, 10:50 AM, "Sasha Kotchubievsky" = > > > > wrote: > > = > > Hi Tomek, > > = > > Are you looking for community feedback regarding VPP implementation= of > > TCP > > stack, or about having socket abstraction layer in SPDK? > > I think, socket abstraction layer is critical for future = > > integration between > > SPDK and user-space stacks. In Mellanox, we're evaluating integrati= on > > between VMA (https://github.com/Mellanox/libvma) and SPDK. = > > Although, VMA can > > be used as replacement for Kernel implementation of Posix socket = > > interface, > > we see great potential in "deep" integration, which definitely needs > > keep > > existing abstraction layer. For example, one of potential improveme= nts > > can > > be zero-copy in RX (receive) flow. I don't see how that can be > > implemented > > on top of Linux Kernel stack. = > > = > > Best regards > > Sasha > > = > > -----Original Message----- > > From: Zawadzki, Tomasz = > > Sent: Monday, October 21, 2019 3:01 PM > > To: Storage Performance Development Kit > > Subject: [SPDK] SPDK socket abstraction layer > > = > > Hello everyone, > > = > > Summary: > > = > > With this message I wanted to update SPDK community on state of VPP > > socket > > abstraction as of SPDK 19.07 release. > > At this time there does not seem to be a clear efficiency = > > improvements with > > VPP. There is no further work planned on SPDK and VPP integration. > > = > > Details: > > = > > As some of you may remember, SPDK 18.04 release introduced support = for > > alternative socket types. Along with that release, Vector Packet = > > Processing > > (VPP) 18.01 was integrated with SPDK, = by > > expanding socket abstraction to use VPP Communications Library (VCL= ). > > TCP/IP > > stack in VPP was in early = > > stages back > > then and has seen improvements throughout the last year. > > = > > To better use VPP capabilities, following fruitful collaboration wi= th > > VPP > > team, in SPDK 19.07, this implementation was changed from VCL to = > > VPP Session > > API from VPP 19.04.2. > > = > > VPP socket abstraction has met some challenges due to inherent desi= gn of > > both projects, in particular related to running separate processes = and > > memory copies. > > Seeing improvements from original implementation was encouraging, y= et > > measuring against posix socket abstraction (taking into = > > consideration entire > > system, i.e. both processes), results are comparable. In other > > words, at > > this time there does not seem to be a clear benefit of either socket > > abstraction from standpoint of CPU efficiency or IOPS. > > = > > With this message I just wanted to update SPDK community on state = > > of socket > > abstraction layers as of SPDK 19.07 release. Each SPDK release = > > always brings > > improvements to the abstraction and its implementations, with = > > exciting work > > on more efficient use of kernel TCP stack - changes in SPDK 19.10 a= nd > > SPDK > > 20.01. > > = > > However there is no active involvement at this point around VPP > > implementation of socket abstraction in SPDK. Contributions in = > > this area are > > always welcome. In case you're interested in implementing further > > enhancements of VPP and SPDK integration feel free to reply, or to = use > > one > > of the many SPDK community communications > > channels;;. > > = > > Thanks, > > Tomek > > = > > _______________________________________________ > > SPDK mailing list -- spdk(a)lists.01.org > > To unsubscribe send an email to spdk-leave(a)lists.01.org > > _______________________________________________ > > SPDK mailing list -- spdk(a)lists.01.org > > To unsubscribe send an email to spdk-leave(a)lists.01.org > > = > > = > > _______________________________________________ > > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email t= o = > > spdk-leave(a)lists.01.org > = > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org --===============0886839989730869651==--