Re: [SPDK] Number of NVMe devices per core

[Verma, Vishal4 <vishal4.verma at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 12808 bytes --]

After looking at your perf command line more closely, It looks like your system is hooked up with 8 nvme drives (4 on each socket). It really depends on your platform, if it allows you to connect 8 drives on the same socket or not. Basically does it have enough PCie slots for you to allow hook up 8 drives on same socket.

As far as I remember I think our 3-3.5M IOPS # was achieved on the platform which has all the drives connected to same socket. Our recommendation would be to see if you can enable Turbo to clock the CPU to higher frequency and run with –q 128.

E5- 2697v4 can go as high as 3.6GHz @ Turbo I believe. I think that should definitely help you get more IOPs/core.

Thanks,
Vishal

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 10:42 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core

Vishal, I wast multiplying 128 by 8, just was checking how queue depth would affect the performance, I dont remember all values I've tried, but I started from something low, like 32, however I didnt see any significant perf. impact, could be considered as measurement error, so I left it 1024. Will try longer runs and check the socket assignment as soon as I get back to office.
When I see the BDF 80+, means I have to assign mask (for 18 core CPU) 0 and 18? BTW, how do I assign all NVMe devices to the same NUMA node? otherwise, how come the aforementioned intel presentation mention 8 NVMe on single core? They all have be assigned to the same socket to get max. perf. right?

On Thu, Feb 15, 2018 at 6:54 PM, Verma, Vishal4 <vishal4.verma(a)intel.com<mailto:vishal4.verma(a)intel.com>> wrote:
Thanks Ernest!

Can you try –q 128 (or 256) instead and –t to something like 120 (2 minutes)? I think perf applies –q parameter value to each nvme drive. So you don’t have to multiply 128 by 8.
There might be different ways to check CPU socket connection of a particular drive. Quickest would be to check using “lspci | grep –i Non” and look for BDF for each of your NVMe drive. Anything greater than 80 in the BDF eg: “80:00.0” would mean it is connected to socket 1.

Thanks,
Vishal
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 9:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

>> What is queue depth you are using while running perf benchmark?
>> Can you share your exact perf command line?
echo 8dev x 1CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20
echo ===============================================================================
echo 8dev x 2CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 3
echo ===============================================================================
echo 8dev x 3CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 7
echo ===============================================================================
echo 8dev x 4CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c F
>> What is the model of your NVMe drives?
INTEL SSDPE2MD800G4
>> Do you know which CPU socket are your NVMes connected?
Nope, not sure how to check

Regarding Turbo setting, will check in BIOS, however I'm out of office for the weekend. Will get back as soon as I get back.

In addition, I'm attaching the full run of my test script, it may provide additional info.

Sincerely,
Ernest


On Thu, Feb 15, 2018 at 6:17 PM, Verma, Vishal4 <vishal4.verma(a)intel.com<mailto:vishal4.verma(a)intel.com>> wrote:
Hi Zed,

It is good that you are able to scale performance with # of cores… Our testing has showed that we can get close to 3M IOPs/Core using perf benchmark. Few questions I have for you regarding the configuration.

What is queue depth you are using while running perf benchmark? Can you share your exact perf command line? What is the model of your NVMe drives? Do you know which CPU socket are your NVMes connected? Should try and run perf core mask to the same socket where NVMes are connected. This would help avoid any cross-socket traffic.

Also, In order to achieve max performance/core, we generally enable Turbo to allow CPU to run at max frequency.

Thanks,
Vishal

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 8:52 AM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

cat CONFIG.local returns CONFIG_DPDK_DIR?=/spdk_test/spdk/dpdk/build
AFAIR to get debug build I have to add some command line argument to configure, I just ran ./configure and make


On Thu, Feb 15, 2018 at 5:35 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Can you confirm you aren’t using a debug build?  ‘cat CONFIG.local’ will confirm.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 8:26 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Number of NVMe devices per core

Dual socket Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, HT off
8 x Intel® SSD DC P3700 Series
16 x 16 DIMM DDR4  2133 MHz
Ubuntu 17.10
4.13.0-32-generic
SPDK version - cloned today

The hardware quite close to the one in the presentation. now results I get with perf for 1, 2 and 4 cores

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid3180 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  206803.20     807.83    4951.17    1088.65   28467.64
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 0:  206803.20     807.83    4951.71    4392.74   25127.60
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  206803.20     807.83    4952.46    4379.57   21966.62
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 0:  206803.20     807.83    4953.35    4379.25   23165.75
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  206803.20     807.83    4954.31    4357.84   30302.48
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 0:  206803.20     807.83    4955.33    4366.51   37664.17
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  206803.20     807.83    4956.35    4357.68   45025.49
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 0:  206803.20     807.83    4957.39    3205.82   52456.83
========================================================
Total                                                  : 1654425.60    6462.60    4954.01    1088.65   52456.83

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 3 --file-prefix=spdk_pid3192 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  438592.00    1713.25    2334.48    1363.58   16128.43
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 1:  438592.00    1713.25    2335.47    2073.78   14921.21
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  438592.00    1713.25    2336.68    2038.52   29808.68
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 1:  438592.00    1713.25    2337.99    1827.81   44872.18
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  437977.60    1710.85    2337.78    1409.41   15688.80
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  437977.60    1710.85    2338.77    2034.40   14879.24
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  437977.60    1710.85    2339.98    2037.44   29795.93
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  437977.60    1710.85    2341.29    2037.46   44764.63
========================================================
Total                                                  : 3506278.40   13696.40    2337.80    1363.58   44872.18

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c F --file-prefix=spdk_pid3205 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 3:  614814.65    2401.62    1665.48     827.93   13076.89
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 3:  481525.05    1880.96    2128.57    1085.12   37295.10
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 2:  542172.10    2117.86    1888.83     920.16   19435.55
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 2:  488315.25    1907.48    2099.24    1070.72   42946.29
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  585409.45    2286.76    1749.10     897.17   10901.34
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  488748.90    1909.18    2097.28    1126.79   43355.29
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  742426.50    2900.10    1379.04     706.27   10713.54
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  488740.80    1909.14    2096.44    1136.78   31606.15
========================================================
Total                                                  : 4432152.70   17313.10    1849.11     706.27   43355.29

Any idea what possibly could go wrong here?


On Thu, Feb 15, 2018 at 4:50 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Ernest,

The answer depends on many different factors – the IOPs limit per device, CPU core frequency, Turbo, etc.  As a frame of reference, the team here at Intel has measured over 3M IO/s on a single Intel Xeon core[1].

-Jim

[1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160809_FA12_P3_Prepalli.pdf - slide 63


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 6:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Number of NVMe devices per core

Hi All,
I ran 'perf' with different settings to see how to get most of NVMe disks. I see that performance degrades once cpu mask puts more than 2 devises per core. Is it OK? what is the theoretical (or empirical) limit of devices one core can handle?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 43698 bytes --]