Re: [SPDK] spdk/vpp performance

Re: [SPDK] spdk/vpp performance

[Jonathan Richardson]

[-- Attachment #1: Type: text/plain, Size: 2777 bytes --]

Hi Sasha,

Thank you for your feedback. Looks like making vpp/nvmeotcp
zero copy is not straight forward at this point but at least we know
where we stand.

Jon

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha
Kotchubievsky
Sent: Thursday, September 12, 2019 9:25 AM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] spdk/vpp performance

Hi,

I'd recommend to use VMA user-space stack for TCP

https://github.com/Mellanox/libvma

With VMA we see better performance than with VPP.

I think, zero-copy is not feasible in case of VPP.

We have plans to implement zero-copy in NVME-OF TCP above VMA.

Best regards

Sasha

On 10-Sep-19 10:42 PM, Jonathan Richardson via SPDK wrote:
> Hi,
>
> I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about
the
> architecture of the session API with nvmeotcp. I attached my perf
results
> for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues
and
> the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
> cpu usage on the spdk threads when it writes the completion into shared
> memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
> You can flip those results around for a random write. L3 cache miss is
bad
> at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
> is more of a bottleneck than cpu on my ARM system, so the spdk design
> worked out well being zero copy. It would be better if the copying in
and
> out of the shmem were avoided though I'm not sure it's feasible and need
> to study the code further.
>
> Are there any plans to change the session API or the way it uses memory?
>
> Spdk and vpp both use dpdk independently with different memory regions
> (--file-prefix). But the shared memory in the session API uses a libc
> malloc'd buffer. Would using a dpdk mempool be better for cache
efficiency
> and synchronization with other dpdk memory usage?
>
> Any thoughts on shared dpdk mempools between vpp and spdk instead of
> copying into the session API's shared memory? It looks like there are 2
> copies for 1 read, the nvme completion into shmem, and the vpp copy out
> (haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
> getting rid of one of the memcpy's could help.
>
> We are looking into possible optimizations so any guidance is
appreciated
> since we want any changes merged into the mainline.
>
> Thanks,
> Jon
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] spdk/vpp performance

[Jonathan Richardson]

[-- Attachment #1: Type: text/plain, Size: 4364 bytes --]

Hi Ben,

Thanks very much for the feedback. That's all I really needed for now. Just
wanted to get a high level idea of what it might take. I didn't think it
would be a trivial undertaking so it's good to hear from someone other than
myself.

We might investigate "Open Fast Path" as well. Possibly a multi-threaded
library would be easier to bring into spdk with a new sock interface and
just
have the stack running in another thread within spdk process. It's all high
level discussion at this point and would require proper investigation.

Thanks,
Jon

-----Original Message-----
From: Walker, Benjamin [mailto:benjamin.walker(a)intel.com]
Sent: Monday, September 16, 2019 12:54 PM
To: spdk(a)lists.01.org
Cc: jonathan.richardson(a)broadcom.com
Subject: Re: [SPDK] spdk/vpp performance

On Tue, 2019-09-10 at 12:42 -0700, Jonathan Richardson via SPDK wrote:
> Hi,
>
> I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the
> architecture of the session API with nvmeotcp. I attached my perf results
> for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and
> the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
> cpu usage on the spdk threads when it writes the completion into shared
> memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
> You can flip those results around for a random write. L3 cache miss is bad
> at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
> is more of a bottleneck than cpu on my ARM system, so the spdk design
> worked out well being zero copy. It would be better if the copying in and
> out of the shmem were avoided though I'm not sure it's feasible and need
> to study the code further.
>
> Are there any plans to change the session API or the way it uses memory?

I've been investigating a bit here and I'm struggling to figure out
specifically
what the optimal way to use the session API would be here - it's
undocumented as
far as I can tell. I wish I had an answer, but I don't. Certainly if we
could
get some experts on VPP to weight in it would be awesome.

>
> Spdk and vpp both use dpdk independently with different memory regions
> (--file-prefix). But the shared memory in the session API uses a libc
> malloc'd buffer. Would using a dpdk mempool be better for cache efficiency
> and synchronization with other dpdk memory usage?

If the DMA operations could occur directly to this memory, yes. If due to
some
other architecture reason it still copies into and out of it, then it won't
help
to make it a memory pool.

>
> Any thoughts on shared dpdk mempools between vpp and spdk instead of
> copying into the session API's shared memory? It looks like there are 2
> copies for 1 read, the nvme completion into shmem, and the vpp copy out
> (haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
> getting rid of one of the memcpy's could help.
>
> We are looking into possible optimizations so any guidance is appreciated
> since we want any changes merged into the mainline.

The guidance we've received from VPP on this is that the copies are
unavoidable
via the session or vcl APIs. To avoid the copies, we'd have to make SPDK a
VPP
plugin and run within their framework. I haven't entirely comprehended what
that
would mean for us. SPDK is designed to integrate reasonably well into other
cooperative multi-tasking frameworks. For example, see here where Jim
integrated
SPDK into Seastar: https://review.gerrithub.io/c/spdk/spdk/+/466629

In order to integrate into another framework, we just need to call
spdk_thread_lib_init and pass in a function pointer that can schedule our
lightweight threads onto whatever event loop the framework has. I think that
means creating "input" nodes for each lightweight thread so that they get
polled
on each pass through the loop, and then re-implementing our sock abstraction
so
that it just passes any data to the "next node" which would be the TCP
processing node. But there's obviously a huge amount of detail to be filled
in
beyond what I've just said, and I have no idea currently how to do it.

>
> Thanks,
> Jon
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] spdk/vpp performance

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

On Tue, 2019-09-10 at 12:42 -0700, Jonathan Richardson via SPDK wrote:
> Hi,
> 
> I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the
> architecture of the session API with nvmeotcp. I attached my perf results
> for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and
> the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
> cpu usage on the spdk threads when it writes the completion into shared
> memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
> You can flip those results around for a random write. L3 cache miss is bad
> at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
> is more of a bottleneck than cpu on my ARM system, so the spdk design
> worked out well being zero copy. It would be better if the copying in and
> out of the shmem were avoided though I'm not sure it's feasible and need
> to study the code further.
> 
> Are there any plans to change the session API or the way it uses memory?

I've been investigating a bit here and I'm struggling to figure out specifically
what the optimal way to use the session API would be here - it's undocumented as
far as I can tell. I wish I had an answer, but I don't. Certainly if we could
get some experts on VPP to weight in it would be awesome.

> 
> Spdk and vpp both use dpdk independently with different memory regions
> (--file-prefix). But the shared memory in the session API uses a libc
> malloc'd buffer. Would using a dpdk mempool be better for cache efficiency
> and synchronization with other dpdk memory usage?

If the DMA operations could occur directly to this memory, yes. If due to some
other architecture reason it still copies into and out of it, then it won't help
to make it a memory pool.

> 
> Any thoughts on shared dpdk mempools between vpp and spdk instead of
> copying into the session API's shared memory? It looks like there are 2
> copies for 1 read, the nvme completion into shmem, and the vpp copy out
> (haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
> getting rid of one of the memcpy's could help.
> 
> We are looking into possible optimizations so any guidance is appreciated
> since we want any changes merged into the mainline.

The guidance we've received from VPP on this is that the copies are unavoidable
via the session or vcl APIs. To avoid the copies, we'd have to make SPDK a VPP
plugin and run within their framework. I haven't entirely comprehended what that
would mean for us. SPDK is designed to integrate reasonably well into other
cooperative multi-tasking frameworks. For example, see here where Jim integrated
SPDK into Seastar: https://review.gerrithub.io/c/spdk/spdk/+/466629

In order to integrate into another framework, we just need to call
spdk_thread_lib_init and pass in a function pointer that can schedule our
lightweight threads onto whatever event loop the framework has. I think that
means creating "input" nodes for each lightweight thread so that they get polled
on each pass through the loop, and then re-implementing our sock abstraction so
that it just passes any data to the "next node" which would be the TCP
processing node. But there's obviously a huge amount of detail to be filled in
beyond what I've just said, and I have no idea currently how to do it.

> 
> Thanks,
> Jon
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] spdk/vpp performance

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

Hi,

I'd recommend to use VMA user-space stack for TCP

https://github.com/Mellanox/libvma

With VMA we see better performance than with VPP.

I think, zero-copy is not feasible in case of VPP.

We have plans to implement zero-copy in NVME-OF TCP above VMA.

Best regards

Sasha

On 10-Sep-19 10:42 PM, Jonathan Richardson via SPDK wrote:
> Hi,
>
> I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the
> architecture of the session API with nvmeotcp. I attached my perf results
> for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and
> the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
> cpu usage on the spdk threads when it writes the completion into shared
> memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
> You can flip those results around for a random write. L3 cache miss is bad
> at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
> is more of a bottleneck than cpu on my ARM system, so the spdk design
> worked out well being zero copy. It would be better if the copying in and
> out of the shmem were avoided though I'm not sure it's feasible and need
> to study the code further.
>
> Are there any plans to change the session API or the way it uses memory?
>
> Spdk and vpp both use dpdk independently with different memory regions
> (--file-prefix). But the shared memory in the session API uses a libc
> malloc'd buffer. Would using a dpdk mempool be better for cache efficiency
> and synchronization with other dpdk memory usage?
>
> Any thoughts on shared dpdk mempools between vpp and spdk instead of
> copying into the session API's shared memory? It looks like there are 2
> copies for 1 read, the nvme completion into shmem, and the vpp copy out
> (haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
> getting rid of one of the memcpy's could help.
>
> We are looking into possible optimizations so any guidance is appreciated
> since we want any changes merged into the mainline.
>
> Thanks,
> Jon
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] spdk/vpp performance

[Jonathan Richardson]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

Hi,

I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the
architecture of the session API with nvmeotcp. I attached my perf results
for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and
the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
cpu usage on the spdk threads when it writes the completion into shared
memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
You can flip those results around for a random write. L3 cache miss is bad
at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
is more of a bottleneck than cpu on my ARM system, so the spdk design
worked out well being zero copy. It would be better if the copying in and
out of the shmem were avoided though I'm not sure it's feasible and need
to study the code further.

Are there any plans to change the session API or the way it uses memory?

Spdk and vpp both use dpdk independently with different memory regions
(--file-prefix). But the shared memory in the session API uses a libc
malloc'd buffer. Would using a dpdk mempool be better for cache efficiency
and synchronization with other dpdk memory usage?

Any thoughts on shared dpdk mempools between vpp and spdk instead of
copying into the session API's shared memory? It looks like there are 2
copies for 1 read, the nvme completion into shmem, and the vpp copy out
(haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
getting rid of one of the memcpy's could help.

We are looking into possible optimizations so any guidance is appreciated
since we want any changes merged into the mainline.

Thanks,
Jon

[-- Attachment #2: vpp-performance.txt --]
[-- Type: text/plain, Size: 7970 bytes --]

------------------------------------------------
SPDK CORE
------------------------------------------------

+   98.23%     0.00%  reactor_6        libc-2.27.so        [.] thread_start
+   98.23%     0.00%  reactor_6        libpthread-2.27.so  [.] start_thread
+   98.23%     0.00%  reactor_6        nvmf_tgt            [.] eal_thread_loop
+   97.52%     0.38%  reactor_6        nvmf_tgt            [.] _spdk_reactor_run
+   96.49%     1.36%  reactor_6        nvmf_tgt            [.] spdk_thread_poll
+   69.56%     0.61%  reactor_6        nvmf_tgt            [.] bdev_nvme_poll
+   68.85%     3.49%  reactor_6        nvmf_tgt            [.] spdk_nvme_qpair_process_completions
+   65.79%     4.32%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_process
+   65.43%     4.21%  reactor_6        nvmf_tgt            [.] nvme_pcie_qpair_process_completions
+   61.18%     0.34%  reactor_6        nvmf_tgt            [.] nvme_pcie_qpair_complete_tracker
+   58.07%     1.27%  reactor_6        nvmf_tgt            [.] nvmf_bdev_ctrlr_complete_cmd
+   56.51%     0.83%  reactor_6        nvmf_tgt            [.] spdk_nvmf_request_complete
+   54.66%     0.03%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_complete
+   53.66%     0.84%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_send_c2h_data
+   51.92%     0.06%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_qpair_flush_pdus
+   51.69%     0.71%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_qpair_flush_pdus_internal
+   49.28%     3.08%  reactor_6        nvmf_tgt            [.] spdk_vpp_sock_writev
-   30.94%    30.90%  reactor_6        libc-2.27.so        [.] __memcpy_generic
     30.90% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
      - spdk_thread_poll
         - 29.62% bdev_nvme_poll
              spdk_nvme_qpair_process_completions
              nvme_pcie_qpair_process_completions
              nvme_pcie_qpair_complete_tracker
              nvmf_bdev_ctrlr_complete_cmd
              spdk_nvmf_request_complete
              spdk_nvmf_tcp_req_complete
              spdk_nvmf_tcp_req_process
              spdk_nvmf_tcp_send_c2h_data
              spdk_nvmf_tcp_qpair_flush_pdus
              spdk_nvmf_tcp_qpair_flush_pdus_internal
            - spdk_vpp_sock_writev
                 29.61% __memcpy_generic
         + 1.28% spdk_nvmf_poll_group_poll
+   25.31%     0.14%  reactor_6        nvmf_tgt            [.] spdk_nvmf_poll_group_poll
+   25.18%     0.27%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_poll_group_poll
+   17.62%     0.79%  reactor_6        nvmf_tgt            [.] spdk_sock_group_poll_count
-   15.13%    15.11%  reactor_6        libsvm.so.19.04.2   [.] svm_fifo_enqueue_nowait_ma
     15.11% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
        spdk_thread_poll
        bdev_nvme_poll
        spdk_nvme_qpair_process_completions
        nvme_pcie_qpair_process_completions
        nvme_pcie_qpair_complete_tracker
        nvmf_bdev_ctrlr_complete_cmd
        spdk_nvmf_request_complete
        spdk_nvmf_tcp_req_complete
        spdk_nvmf_tcp_req_process
        spdk_nvmf_tcp_send_c2h_data
        spdk_nvmf_tcp_qpair_flush_pdus
      - spdk_nvmf_tcp_qpair_flush_pdus_internal
         - 15.11% spdk_vpp_sock_writev
              svm_fifo_enqueue_nowait_ma
+   12.02%     0.43%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_sock_cb
+    7.15%     0.48%  reactor_6        nvmf_tgt            [.] spdk_nvmf_request_exec
-    7.02%     7.01%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_set_state
     7.01% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
      - spdk_thread_poll
         - 5.31% spdk_nvmf_poll_group_poll
            - spdk_nvmf_tcp_poll_group_poll
               - 5.10% spdk_sock_group_poll_count
                    4.75% spdk_nvmf_tcp_req_set_state
         + 1.70% bdev_nvme_poll
+    6.28%     0.12%  reactor_6        nvmf_tgt            [.] spdk_nvmf_bdev_ctrlr_read_cmd
+    6.17%     0.02%  reactor_6        nvmf_tgt            [.] spdk_bdev_readv_blocks
+    5.49%     0.26%  reactor_6        nvmf_tgt            [.] _spdk_bdev_readv_blocks_with_md
+    4.98%     0.03%  reactor_6        nvmf_tgt            [.] nvme_tcp_read_data
+    4.87%     0.23%  reactor_6        nvmf_tgt            [.] spdk_vpp_sock_recv
+    4.61%     0.33%  reactor_6        nvmf_tgt            [.] spdk_bdev_io_submit
+    4.05%     0.15%  reactor_6        nvmf_tgt            [.] bdev_nvme_get_buf_cb
+    3.89%     1.76%  reactor_6        nvmf_tgt            [.] spdk_nvme_ns_cmd_readv_with_md
+    3.22%     3.22%  reactor_6        libsvm.so.19.04.2   [.] svm_fifo_dequeue_nowait_ma
+    2.74%     2.74%  reactor_6        nvmf_tgt            [.] spdk_bdev_io_complete

------------------------------------------------
VPP CORE
------------------------------------------------

+  100.00%     0.00%  vpp_wk_1  libvppinfra.so.19.04.2  [.] clib_calljmp
+   99.98%     0.39%  vpp_wk_1  libvlib.so.19.04.2      [.] vlib_worker_loop
+   62.76%     0.71%  vpp_wk_1  libvnet.so.19.04.2      [.] session_queue_node_fn
+   59.80%     3.82%  vpp_wk_1  libvnet.so.19.04.2      [.] session_tx_fifo_peek_and_snd
-   54.59%    54.54%  vpp_wk_1  libc-2.27.so            [.] __memcpy_generic
     54.54% clib_calljmp
      - vlib_worker_loop
         - 52.14% session_queue_node_fn
            - 52.05% session_tx_fifo_peek_and_snd
                 __memcpy_generic
         - 2.39% dispatch_pending_node
            - 2.35% tcp4_established_node_fn
                 tcp_segment_rcv
                 session_enqueue_stream_connection
                 __memcpy_generic
+   34.04%     0.75%  vpp_wk_1  libvlib.so.19.04.2      [.] dispatch_pending_node
+   14.14%     3.99%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_device_class_tx_fn
+   10.14%     2.58%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_xmit_pkts
-    7.28%     6.49%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_handle_tx_cp
   - 6.49% clib_calljmp
        vlib_worker_loop
        dispatch_pending_node
      - dpdk_device_class_tx_fn
         - 6.48% bnxt_xmit_pkts
              bnxt_handle_tx_cp
   + 0.79% bnxt_handle_tx_cp
+    6.93%     0.66%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_established_node_fn
+    3.84%     0.05%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_segment_rcv
+    3.68%     0.11%  vpp_wk_1  libvnet.so.19.04.2      [.] session_enqueue_stream_connection
+    3.57%     3.30%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_output_node_fn
+    2.85%     2.80%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_rewrite_node_fn
+    2.31%     0.63%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_input_node_fn
+    2.16%     1.96%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_lookup_node_fn
+    1.64%     1.64%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_peek_ma
+    1.49%     1.49%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_recv_pkts
+    1.31%     1.30%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_session_push_header
+    1.20%     1.20%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_enqueue_nowait_ma
+    1.06%     0.08%  vpp_wk_1  libvnet.so.19.04.2      [.] transport_update_time
+    0.84%     0.84%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_dequeue_drop
+    0.84%     0.78%  vpp_wk_1  libvnet.so.19.04.2      [.] ethernet_input_node_fn
+    0.79%     0.26%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_input_node_fn
+    0.76%     0.69%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_input_no_checksum_node_fn
+    0.69%     0.69%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_ops_vpp_enqueue
+    0.67%     0.01%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_update_time
+    0.66%     0.66%  vpp_wk_1  libvlib.so.19.04.2      [.] vlib_get_next_frame_internal