[SPDK] spdk/vpp performance

[Jonathan Richardson <jonathan.richardson at broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

Hi,

I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the
architecture of the session API with nvmeotcp. I attached my perf results
for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and
the other 5 cores are for spdk. The memcpy is the bottleneck taking 31%
cpu usage on the spdk threads when it writes the completion into shared
memory. The vpp thread memcpy is 54% when it pulls the data out for tx.
You can flip those results around for a random write. L3 cache miss is bad
at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory
is more of a bottleneck than cpu on my ARM system, so the spdk design
worked out well being zero copy. It would be better if the copying in and
out of the shmem were avoided though I'm not sure it's feasible and need
to study the code further.

Are there any plans to change the session API or the way it uses memory?

Spdk and vpp both use dpdk independently with different memory regions
(--file-prefix). But the shared memory in the session API uses a libc
malloc'd buffer. Would using a dpdk mempool be better for cache efficiency
and synchronization with other dpdk memory usage?

Any thoughts on shared dpdk mempools between vpp and spdk instead of
copying into the session API's shared memory? It looks like there are 2
copies for 1 read, the nvme completion into shmem, and the vpp copy out
(haven't looked where session_tx_fifo_peek_and_snd is copying to). Even
getting rid of one of the memcpy's could help.

We are looking into possible optimizations so any guidance is appreciated
since we want any changes merged into the mainline.

Thanks,
Jon

[-- Attachment #2: vpp-performance.txt --]
[-- Type: text/plain, Size: 7970 bytes --]

------------------------------------------------
SPDK CORE
------------------------------------------------

+   98.23%     0.00%  reactor_6        libc-2.27.so        [.] thread_start
+   98.23%     0.00%  reactor_6        libpthread-2.27.so  [.] start_thread
+   98.23%     0.00%  reactor_6        nvmf_tgt            [.] eal_thread_loop
+   97.52%     0.38%  reactor_6        nvmf_tgt            [.] _spdk_reactor_run
+   96.49%     1.36%  reactor_6        nvmf_tgt            [.] spdk_thread_poll
+   69.56%     0.61%  reactor_6        nvmf_tgt            [.] bdev_nvme_poll
+   68.85%     3.49%  reactor_6        nvmf_tgt            [.] spdk_nvme_qpair_process_completions
+   65.79%     4.32%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_process
+   65.43%     4.21%  reactor_6        nvmf_tgt            [.] nvme_pcie_qpair_process_completions
+   61.18%     0.34%  reactor_6        nvmf_tgt            [.] nvme_pcie_qpair_complete_tracker
+   58.07%     1.27%  reactor_6        nvmf_tgt            [.] nvmf_bdev_ctrlr_complete_cmd
+   56.51%     0.83%  reactor_6        nvmf_tgt            [.] spdk_nvmf_request_complete
+   54.66%     0.03%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_complete
+   53.66%     0.84%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_send_c2h_data
+   51.92%     0.06%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_qpair_flush_pdus
+   51.69%     0.71%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_qpair_flush_pdus_internal
+   49.28%     3.08%  reactor_6        nvmf_tgt            [.] spdk_vpp_sock_writev
-   30.94%    30.90%  reactor_6        libc-2.27.so        [.] __memcpy_generic
     30.90% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
      - spdk_thread_poll
         - 29.62% bdev_nvme_poll
              spdk_nvme_qpair_process_completions
              nvme_pcie_qpair_process_completions
              nvme_pcie_qpair_complete_tracker
              nvmf_bdev_ctrlr_complete_cmd
              spdk_nvmf_request_complete
              spdk_nvmf_tcp_req_complete
              spdk_nvmf_tcp_req_process
              spdk_nvmf_tcp_send_c2h_data
              spdk_nvmf_tcp_qpair_flush_pdus
              spdk_nvmf_tcp_qpair_flush_pdus_internal
            - spdk_vpp_sock_writev
                 29.61% __memcpy_generic
         + 1.28% spdk_nvmf_poll_group_poll
+   25.31%     0.14%  reactor_6        nvmf_tgt            [.] spdk_nvmf_poll_group_poll
+   25.18%     0.27%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_poll_group_poll
+   17.62%     0.79%  reactor_6        nvmf_tgt            [.] spdk_sock_group_poll_count
-   15.13%    15.11%  reactor_6        libsvm.so.19.04.2   [.] svm_fifo_enqueue_nowait_ma
     15.11% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
        spdk_thread_poll
        bdev_nvme_poll
        spdk_nvme_qpair_process_completions
        nvme_pcie_qpair_process_completions
        nvme_pcie_qpair_complete_tracker
        nvmf_bdev_ctrlr_complete_cmd
        spdk_nvmf_request_complete
        spdk_nvmf_tcp_req_complete
        spdk_nvmf_tcp_req_process
        spdk_nvmf_tcp_send_c2h_data
        spdk_nvmf_tcp_qpair_flush_pdus
      - spdk_nvmf_tcp_qpair_flush_pdus_internal
         - 15.11% spdk_vpp_sock_writev
              svm_fifo_enqueue_nowait_ma
+   12.02%     0.43%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_sock_cb
+    7.15%     0.48%  reactor_6        nvmf_tgt            [.] spdk_nvmf_request_exec
-    7.02%     7.01%  reactor_6        nvmf_tgt            [.] spdk_nvmf_tcp_req_set_state
     7.01% thread_start
        start_thread
        eal_thread_loop
        _spdk_reactor_run
      - spdk_thread_poll
         - 5.31% spdk_nvmf_poll_group_poll
            - spdk_nvmf_tcp_poll_group_poll
               - 5.10% spdk_sock_group_poll_count
                    4.75% spdk_nvmf_tcp_req_set_state
         + 1.70% bdev_nvme_poll
+    6.28%     0.12%  reactor_6        nvmf_tgt            [.] spdk_nvmf_bdev_ctrlr_read_cmd
+    6.17%     0.02%  reactor_6        nvmf_tgt            [.] spdk_bdev_readv_blocks
+    5.49%     0.26%  reactor_6        nvmf_tgt            [.] _spdk_bdev_readv_blocks_with_md
+    4.98%     0.03%  reactor_6        nvmf_tgt            [.] nvme_tcp_read_data
+    4.87%     0.23%  reactor_6        nvmf_tgt            [.] spdk_vpp_sock_recv
+    4.61%     0.33%  reactor_6        nvmf_tgt            [.] spdk_bdev_io_submit
+    4.05%     0.15%  reactor_6        nvmf_tgt            [.] bdev_nvme_get_buf_cb
+    3.89%     1.76%  reactor_6        nvmf_tgt            [.] spdk_nvme_ns_cmd_readv_with_md
+    3.22%     3.22%  reactor_6        libsvm.so.19.04.2   [.] svm_fifo_dequeue_nowait_ma
+    2.74%     2.74%  reactor_6        nvmf_tgt            [.] spdk_bdev_io_complete

------------------------------------------------
VPP CORE
------------------------------------------------

+  100.00%     0.00%  vpp_wk_1  libvppinfra.so.19.04.2  [.] clib_calljmp
+   99.98%     0.39%  vpp_wk_1  libvlib.so.19.04.2      [.] vlib_worker_loop
+   62.76%     0.71%  vpp_wk_1  libvnet.so.19.04.2      [.] session_queue_node_fn
+   59.80%     3.82%  vpp_wk_1  libvnet.so.19.04.2      [.] session_tx_fifo_peek_and_snd
-   54.59%    54.54%  vpp_wk_1  libc-2.27.so            [.] __memcpy_generic
     54.54% clib_calljmp
      - vlib_worker_loop
         - 52.14% session_queue_node_fn
            - 52.05% session_tx_fifo_peek_and_snd
                 __memcpy_generic
         - 2.39% dispatch_pending_node
            - 2.35% tcp4_established_node_fn
                 tcp_segment_rcv
                 session_enqueue_stream_connection
                 __memcpy_generic
+   34.04%     0.75%  vpp_wk_1  libvlib.so.19.04.2      [.] dispatch_pending_node
+   14.14%     3.99%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_device_class_tx_fn
+   10.14%     2.58%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_xmit_pkts
-    7.28%     6.49%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_handle_tx_cp
   - 6.49% clib_calljmp
        vlib_worker_loop
        dispatch_pending_node
      - dpdk_device_class_tx_fn
         - 6.48% bnxt_xmit_pkts
              bnxt_handle_tx_cp
   + 0.79% bnxt_handle_tx_cp
+    6.93%     0.66%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_established_node_fn
+    3.84%     0.05%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_segment_rcv
+    3.68%     0.11%  vpp_wk_1  libvnet.so.19.04.2      [.] session_enqueue_stream_connection
+    3.57%     3.30%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_output_node_fn
+    2.85%     2.80%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_rewrite_node_fn
+    2.31%     0.63%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_input_node_fn
+    2.16%     1.96%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_lookup_node_fn
+    1.64%     1.64%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_peek_ma
+    1.49%     1.49%  vpp_wk_1  dpdk_plugin.so          [.] bnxt_recv_pkts
+    1.31%     1.30%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_session_push_header
+    1.20%     1.20%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_enqueue_nowait_ma
+    1.06%     0.08%  vpp_wk_1  libvnet.so.19.04.2      [.] transport_update_time
+    0.84%     0.84%  vpp_wk_1  libsvm.so.19.04.2       [.] svm_fifo_dequeue_drop
+    0.84%     0.78%  vpp_wk_1  libvnet.so.19.04.2      [.] ethernet_input_node_fn
+    0.79%     0.26%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp4_input_node_fn
+    0.76%     0.69%  vpp_wk_1  libvnet.so.19.04.2      [.] ip4_input_no_checksum_node_fn
+    0.69%     0.69%  vpp_wk_1  dpdk_plugin.so          [.] dpdk_ops_vpp_enqueue
+    0.67%     0.01%  vpp_wk_1  libvnet.so.19.04.2      [.] tcp_update_time
+    0.66%     0.66%  vpp_wk_1  libvlib.so.19.04.2      [.] vlib_get_next_frame_internal