Re: [SPDK] Number of NVMe devices per core

[Verma, Vishal4 <vishal4.verma at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 10830 bytes --]

Thanks Ernest!

Can you try –q 128 (or 256) instead and –t to something like 120 (2 minutes)? I think perf applies –q parameter value to each nvme drive. So you don’t have to multiply 128 by 8.
There might be different ways to check CPU socket connection of a particular drive. Quickest would be to check using “lspci | grep –i Non” and look for BDF for each of your NVMe drive. Anything greater than 80 in the BDF eg: “80:00.0” would mean it is connected to socket 1.

Thanks,
Vishal
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 9:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core

>> What is queue depth you are using while running perf benchmark?
>> Can you share your exact perf command line?
echo 8dev x 1CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20
echo ===============================================================================
echo 8dev x 2CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 3
echo ===============================================================================
echo 8dev x 3CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 7
echo ===============================================================================
echo 8dev x 4CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c F
>> What is the model of your NVMe drives?
INTEL SSDPE2MD800G4
>> Do you know which CPU socket are your NVMes connected?
Nope, not sure how to check

Regarding Turbo setting, will check in BIOS, however I'm out of office for the weekend. Will get back as soon as I get back.

In addition, I'm attaching the full run of my test script, it may provide additional info.

Sincerely,
Ernest


On Thu, Feb 15, 2018 at 6:17 PM, Verma, Vishal4 <vishal4.verma(a)intel.com<mailto:vishal4.verma(a)intel.com>> wrote:
Hi Zed,

It is good that you are able to scale performance with # of cores… Our testing has showed that we can get close to 3M IOPs/Core using perf benchmark. Few questions I have for you regarding the configuration.

What is queue depth you are using while running perf benchmark? Can you share your exact perf command line? What is the model of your NVMe drives? Do you know which CPU socket are your NVMes connected? Should try and run perf core mask to the same socket where NVMes are connected. This would help avoid any cross-socket traffic.

Also, In order to achieve max performance/core, we generally enable Turbo to allow CPU to run at max frequency.

Thanks,
Vishal

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 8:52 AM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

cat CONFIG.local returns CONFIG_DPDK_DIR?=/spdk_test/spdk/dpdk/build
AFAIR to get debug build I have to add some command line argument to configure, I just ran ./configure and make


On Thu, Feb 15, 2018 at 5:35 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Can you confirm you aren’t using a debug build?  ‘cat CONFIG.local’ will confirm.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 8:26 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

Dual socket Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, HT off
8 x Intel® SSD DC P3700 Series
16 x 16 DIMM DDR4  2133 MHz
Ubuntu 17.10
4.13.0-32-generic
SPDK version - cloned today

The hardware quite close to the one in the presentation. now results I get with perf for 1, 2 and 4 cores

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid3180 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  206803.20     807.83    4951.17    1088.65   28467.64
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 0:  206803.20     807.83    4951.71    4392.74   25127.60
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  206803.20     807.83    4952.46    4379.57   21966.62
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 0:  206803.20     807.83    4953.35    4379.25   23165.75
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  206803.20     807.83    4954.31    4357.84   30302.48
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 0:  206803.20     807.83    4955.33    4366.51   37664.17
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  206803.20     807.83    4956.35    4357.68   45025.49
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 0:  206803.20     807.83    4957.39    3205.82   52456.83
========================================================
Total                                                  : 1654425.60    6462.60    4954.01    1088.65   52456.83

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 3 --file-prefix=spdk_pid3192 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  438592.00    1713.25    2334.48    1363.58   16128.43
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 1:  438592.00    1713.25    2335.47    2073.78   14921.21
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  438592.00    1713.25    2336.68    2038.52   29808.68
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 1:  438592.00    1713.25    2337.99    1827.81   44872.18
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  437977.60    1710.85    2337.78    1409.41   15688.80
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  437977.60    1710.85    2338.77    2034.40   14879.24
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  437977.60    1710.85    2339.98    2037.44   29795.93
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  437977.60    1710.85    2341.29    2037.46   44764.63
========================================================
Total                                                  : 3506278.40   13696.40    2337.80    1363.58   44872.18

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c F --file-prefix=spdk_pid3205 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 3:  614814.65    2401.62    1665.48     827.93   13076.89
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 3:  481525.05    1880.96    2128.57    1085.12   37295.10
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 2:  542172.10    2117.86    1888.83     920.16   19435.55
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 2:  488315.25    1907.48    2099.24    1070.72   42946.29
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  585409.45    2286.76    1749.10     897.17   10901.34
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  488748.90    1909.18    2097.28    1126.79   43355.29
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  742426.50    2900.10    1379.04     706.27   10713.54
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  488740.80    1909.14    2096.44    1136.78   31606.15
========================================================
Total                                                  : 4432152.70   17313.10    1849.11     706.27   43355.29

Any idea what possibly could go wrong here?


On Thu, Feb 15, 2018 at 4:50 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Ernest,

The answer depends on many different factors – the IOPs limit per device, CPU core frequency, Turbo, etc.  As a frame of reference, the team here at Intel has measured over 3M IO/s on a single Intel Xeon core[1].

-Jim

[1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160809_FA12_P3_Prepalli.pdf - slide 63


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 6:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Number of NVMe devices per core

Hi All,
I ran 'perf' with different settings to see how to get most of NVMe disks. I see that performance degrades once cpu mask puts more than 2 devises per core. Is it OK? what is the theoretical (or empirical) limit of devices one core can handle?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 37236 bytes --]