From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5552591872785517267==" MIME-Version: 1.0 From: Walker, Benjamin Subject: [SPDK] Re: SPDK socket abstraction layer Date: Thu, 07 Nov 2019 18:45:57 +0000 Message-ID: In-Reply-To: CAJ3xEMjP+8GTzisXHPe4EU8aPBnSrg770Jok-BxEt+Fs-T-E8Q@mail.gmail.com List-ID: To: spdk@lists.01.org --===============5552591872785517267== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, 2019-11-07 at 18:26 +0200, Or Gerlitz wrote: > On Tue, Nov 5, 2019 at 8:08 PM Walker, Benjamin > wrote: > > On my system currently the network stack is electing to do a deferred c= opy > > so the performance is not good. > = > Ben, can't prove that instantly, but I tend to think that if you use > dma buffers for the wire pdu headers, > things would be better for you in that respect (deferred copy), spdk's > dma buffers originate (dpdk memory > allocators) from huge pages which afaik on linux are always pinned. > Hence the kernel just needs to ref/unref > when you use them with MSG_ZEROCOPY and nothing beyond that. This is > the initiator patch [1] but should > be straight forward to apply it for the target. I agree and I've had similar thoughts - putting all of the data into pre-pi= nned memory is much more likely to hit a fast path in the pinning logic. But I w= ould have expected it to work even if it wasn't optimal, and it does look like y= our colleagues at Mellanox got my patch working to some extent. We're just goin= g to try it again using the exact set up they used and confirm that it does func= tion. In parallel I'm going to change the allocations for the PDUs on the target = side to DMA memory like you've done on the initiator. > = > https://review.gerrithub.io/c/spdk/spdk/+/473278/4 > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org --===============5552591872785517267==--