Re: [SPDK] Number of NVMe devices per core

[Verma, Vishal4 <vishal4.verma at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 7548 bytes --]

Hi Zed,

It is good that you are able to scale performance with # of cores… Our testing has showed that we can get close to 3M IOPs/Core using perf benchmark. Few questions I have for you regarding the configuration.

What is queue depth you are using while running perf benchmark? Can you share your exact perf command line? What is the model of your NVMe drives? Do you know which CPU socket are your NVMes connected? Should try and run perf core mask to the same socket where NVMes are connected. This would help avoid any cross-socket traffic.

Also, In order to achieve max performance/core, we generally enable Turbo to allow CPU to run at max frequency.

Thanks,
Vishal

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 8:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core

cat CONFIG.local returns CONFIG_DPDK_DIR?=/spdk_test/spdk/dpdk/build
AFAIR to get debug build I have to add some command line argument to configure, I just ran ./configure and make


On Thu, Feb 15, 2018 at 5:35 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Can you confirm you aren’t using a debug build?  ‘cat CONFIG.local’ will confirm.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 8:26 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

Dual socket Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, HT off
8 x Intel® SSD DC P3700 Series
16 x 16 DIMM DDR4  2133 MHz
Ubuntu 17.10
4.13.0-32-generic
SPDK version - cloned today

The hardware quite close to the one in the presentation. now results I get with perf for 1, 2 and 4 cores

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid3180 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  206803.20     807.83    4951.17    1088.65   28467.64
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 0:  206803.20     807.83    4951.71    4392.74   25127.60
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  206803.20     807.83    4952.46    4379.57   21966.62
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 0:  206803.20     807.83    4953.35    4379.25   23165.75
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  206803.20     807.83    4954.31    4357.84   30302.48
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 0:  206803.20     807.83    4955.33    4366.51   37664.17
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  206803.20     807.83    4956.35    4357.68   45025.49
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 0:  206803.20     807.83    4957.39    3205.82   52456.83
========================================================
Total                                                  : 1654425.60    6462.60    4954.01    1088.65   52456.83

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 3 --file-prefix=spdk_pid3192 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  438592.00    1713.25    2334.48    1363.58   16128.43
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 1:  438592.00    1713.25    2335.47    2073.78   14921.21
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  438592.00    1713.25    2336.68    2038.52   29808.68
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 1:  438592.00    1713.25    2337.99    1827.81   44872.18
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  437977.60    1710.85    2337.78    1409.41   15688.80
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  437977.60    1710.85    2338.77    2034.40   14879.24
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  437977.60    1710.85    2339.98    2037.44   29795.93
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  437977.60    1710.85    2341.29    2037.46   44764.63
========================================================
Total                                                  : 3506278.40   13696.40    2337.80    1363.58   44872.18

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c F --file-prefix=spdk_pid3205 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 3:  614814.65    2401.62    1665.48     827.93   13076.89
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 3:  481525.05    1880.96    2128.57    1085.12   37295.10
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 2:  542172.10    2117.86    1888.83     920.16   19435.55
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 2:  488315.25    1907.48    2099.24    1070.72   42946.29
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  585409.45    2286.76    1749.10     897.17   10901.34
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  488748.90    1909.18    2097.28    1126.79   43355.29
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  742426.50    2900.10    1379.04     706.27   10713.54
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  488740.80    1909.14    2096.44    1136.78   31606.15
========================================================
Total                                                  : 4432152.70   17313.10    1849.11     706.27   43355.29

Any idea what possibly could go wrong here?


On Thu, Feb 15, 2018 at 4:50 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Ernest,

The answer depends on many different factors – the IOPs limit per device, CPU core frequency, Turbo, etc.  As a frame of reference, the team here at Intel has measured over 3M IO/s on a single Intel Xeon core[1].

-Jim

[1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160809_FA12_P3_Prepalli.pdf - slide 63


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 6:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Number of NVMe devices per core

Hi All,
I ran 'perf' with different settings to see how to get most of NVMe disks. I see that performance degrades once cpu mask puts more than 2 devises per core. Is it OK? what is the theoretical (or empirical) limit of devices one core can handle?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 26696 bytes --]