From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============2445346956541306066=="
MIME-Version: 1.0
From: Mittal, Rishabh <rimittal at ebay.com>
Subject: Re: [SPDK] NBD with SPDK
Date: Thu, 15 Aug 2019 23:34:30 +0000
Message-ID: <47AD3F09-682C-4DCC-AEF6-4EE1D5B24751@ebay.com>
In-Reply-To: 0A4299FB-9034-4530-A633-D4DB7EC1269A@intel.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============2445346956541306066==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hi Jim

What tool you use to take profiling. =


Thanks
Rishabh Mittal

=EF=BB=BFOn 8/14/19, 9:54 AM, "Harris, James R" <james.r.harris(a)intel.com=
> wrote:

    =

    =

    On 8/14/19, 9:18 AM, "Walker, Benjamin" <benjamin.walker(a)intel.com> w=
rote:
    =

    <trim>
        =

        When an I/O is performed in the process initiating the I/O to a fil=
e, the data
        goes into the OS page cache buffers at a layer far above the bio st=
ack
        (somewhere up in VFS). If SPDK were to reserve some memory and hand=
 it off to
        your kernel driver, your kernel driver would still need to copy it =
to that
        location out of the page cache buffers. We can't safely share the p=
age cache
        buffers with a user space process.
       =

    I think Rishabh was suggesting the SPDK reserve the virtual address spa=
ce only.
    Then the kernel could map the page cache buffers into that virtual addr=
ess space.
    That would not require a data copy, but would require the mapping opera=
tions.
    =

    I think the profiling data would be really helpful - to quantify how mu=
ch of the 50us
    Is due to copying the 4KB of data.  That can help drive next steps on h=
ow to optimize
    the SPDK NBD module.
    =

    Thanks,
    =

    -Jim
    =

    =

        As Paul said, I'm skeptical that the memcpy is significant in the o=
verall
        performance you're measuring. I encourage you to go look at some pr=
ofiling data
        and confirm that the memcpy is really showing up. I suspect the ove=
rhead is
        instead primarily in these spots:
        =

        1) Dynamic buffer allocation in the SPDK NBD backend.
        =

        As Paul indicated, the NBD target is dynamically allocating memory =
for each I/O.
        The NBD backend wasn't designed to be fast - it was designed to be =
simple.
        Pooling would be a lot faster and is something fairly easy to imple=
ment.
        =

        2) The way SPDK does the syscalls when it implements the NBD backen=
d.
        =

        Again, the code was designed to be simple, not high performance. It=
 simply calls
        read() and write() on the socket for each command. There are much h=
igher
        performance ways of doing this, they're just more complex to implem=
ent.
        =

        3) The lack of multi-queue support in NBD
        =

        Every I/O is funneled through a single sockpair up to user space. T=
hat means
        there is locking going on. I believe this is just a limitation of N=
BD today - it
        doesn't plug into the block-mq stuff in the kernel and expose multi=
ple
        sockpairs. But someone more knowledgeable on the kernel stack would=
 need to take
        a look.
        =

        Thanks,
        Ben
        =

        > =

        > Couple of things that I am not really sure in this flow is :- 1. =
How memory
        > registration is going to work with RDMA driver.
        > 2. What changes are required in spdk memory management
        > =

        > Thanks
        > Rishabh Mittal
        =

    =

    =



--===============2445346956541306066==--