[SPDK] Re: SPDK socket abstraction layer

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

On Thu, 2019-11-07 at 18:26 +0200, Or Gerlitz wrote:
> On Tue, Nov 5, 2019 at 8:08 PM Walker, Benjamin
> <benjamin.walker(a)intel.com> wrote:
> > On my system currently the network stack is electing to do a deferred copy
> > so the performance is not good.
> 
> Ben, can't prove that instantly, but I tend to think that if you use
> dma buffers for the wire pdu headers,
> things would be better for you in that respect (deferred copy), spdk's
> dma buffers originate (dpdk memory
> allocators) from huge pages which afaik on linux are always pinned.
> Hence the kernel just needs to ref/unref
> when you use them with MSG_ZEROCOPY and nothing beyond that. This is
> the initiator patch [1] but should
> be straight forward to apply it for the target.

I agree and I've had similar thoughts - putting all of the data into pre-pinned
memory is much more likely to hit a fast path in the pinning logic. But I would
have expected it to work even if it wasn't optimal, and it does look like your
colleagues at Mellanox got my patch working to some extent. We're just going to
try it again using the exact set up they used and confirm that it does function.

In parallel I'm going to change the allocations for the PDUs on the target side
to DMA memory like you've done on the initiator.

> 
> https://review.gerrithub.io/c/spdk/spdk/+/473278/4
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org