After looking at your perf command line more closely, It looks like your system is hooked up with 8 nvme drives (4 on each socket). It really depends on your
platform, if it allows you to connect 8 drives on the same socket or not. Basically does it have enough PCie slots for you to allow hook up 8 drives on same socket.
As far as I remember I think our 3-3.5M IOPS # was achieved on the platform which has all the drives connected to same socket. Our recommendation would be to
see if you can enable Turbo to clock the CPU to higher frequency and run with –q 128.
E5- 2697v4 can go as high as 3.6GHz @ Turbo I believe. I think that should definitely help you get more IOPs/core.
Thanks,
Vishal
From: SPDK [mailto:spdk-bounces@lists.01.org]
On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 10:42 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core
Vishal, I wast multiplying 128 by 8, just was checking how queue depth would affect the performance, I dont remember all values I've tried, but I started from something low, like 32, however I didnt see any significant perf. impact, could
be considered as measurement error, so I left it 1024. Will try longer runs and check the socket assignment as soon as I get back to office.
When I see the BDF 80+, means I have to assign mask (for 18 core CPU) 0 and 18? BTW, how do I assign all NVMe devices to the same NUMA node? otherwise, how come the aforementioned intel presentation mention 8 NVMe on single core? They all
have be assigned to the same socket to get max. perf. right?
On Thu, Feb 15, 2018 at 6:54 PM, Verma, Vishal4 <vishal4.verma@intel.com> wrote:
Thanks Ernest!
Can you try –q 128 (or 256) instead and –t to something like 120 (2 minutes)? I think perf applies –q parameter value to each nvme drive. So you don’t have to multiply 128 by 8.
There might be different ways to check CPU socket connection of a particular drive. Quickest would be to check using “lspci | grep –i Non” and look for BDF for each of your NVMe drive. Anything greater than 80 in the BDF eg: “80:00.0” would mean it is connected to socket 1.
Vishal
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 9:46 AM
To: Storage Performance Development Kit <spdk@lists.01.org>Subject: Re: [SPDK] Number of NVMe devices per core
>> What is queue depth you are using while running perf benchmark?
>> Can you share your exact perf command line?
echo 8dev x 1CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20
echo ===============================================================================
echo 8dev x 2CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 3
echo ===============================================================================
echo 8dev x 3CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c 7
echo ===============================================================================
echo 8dev x 4CPUs
/spdk_test/spdk/examples/nvme/perf/perf -w randread -s 4096 -r 'trtype:PCIe traddr:03:00.0' -r 'trtype:PCIe traddr:04:00.0' -r 'trtype:PCIe traddr:05:00.0' -r 'trtype:PCIe traddr:06:00.0' -r 'trtype:PCIe traddr:83:00.0' -r 'trtype:PCIe traddr:84:00.0' -r 'trtype:PCIe traddr:85:00.0' -r 'trtype:PCIe traddr:86:00.0' -q 1024 -t 20 -c F
>> What is the model of your NVMe drives?
INTEL SSDPE2MD800G4
>> Do you know which CPU socket are your NVMes connected?
Nope, not sure how to check
Regarding Turbo setting, will check in BIOS, however I'm out of office for the weekend. Will get back as soon as I get back.
In addition, I'm attaching the full run of my test script, it may provide additional info.
Sincerely,
Ernest
On Thu, Feb 15, 2018 at 6:17 PM, Verma, Vishal4 <vishal4.verma@intel.com> wrote:
Hi Zed,
It is good that you are able to scale performance with # of cores… Our testing has showed that we can get close to 3M IOPs/Core using perf benchmark. Few questions I have for you regarding the configuration.
What is queue depth you are using while running perf benchmark? Can you share your exact perf command line? What is the model of your NVMe drives? Do you know which CPU socket are your NVMes connected? Should try and run perf core mask to the same socket where NVMes are connected. This would help avoid any cross-socket traffic.
Also, In order to achieve max performance/core, we generally enable Turbo to allow CPU to run at max frequency.
Thanks,
Vishal
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Ernest Zed
Sent: Thursday, February 15, 2018 8:52 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core
cat CONFIG.local returns CONFIG_DPDK_DIR?=/spdk_test/spdk/dpdk/build
AFAIR to get debug build I have to add some command line argument to configure, I just ran ./configure and make
On Thu, Feb 15, 2018 at 5:35 PM, Harris, James R <james.r.harris@intel.com> wrote:
Can you confirm you aren’t using a debug build? ‘cat CONFIG.local’ will confirm.
-Jim
From: SPDK <spdk-bounces@lists.01.org> on behalf of Ernest Zed <kreuzerkrieg@gmail.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Thursday, February 15, 2018 at 8:26 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core
Dual socket Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, HT off
8 x Intel® SSD DC P3700 Series
16 x 16 DIMM DDR4 2133 MHz
Ubuntu 17.10
4.13.0-32-generic
SPDK version - cloned today
The hardware quite close to the one in the presentation. now results I get with perf for 1, 2 and 4 cores
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid3180 ]
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD800G4 (CVFT5471002L800HGN ) from core 0: 206803.20 807.83 4951.17 1088.65 28467.64
INTEL SSDPE2MD800G4 (CVFT5471002S800HGN ) from core 0: 206803.20 807.83 4951.71 4392.74 25127.60
INTEL SSDPE2MD800G4 (CVFT5471001U800HGN ) from core 0: 206803.20 807.83 4952.46 4379.57 21966.62
INTEL SSDPE2MD800G4 (CVFT5471002K800HGN ) from core 0: 206803.20 807.83 4953.35 4379.25 23165.75
INTEL SSDPE2MD800G4 (CVFT54710002800HGN ) from core 0: 206803.20 807.83 4954.31 4357.84 30302.48
INTEL SSDPE2MD800G4 (CVFT5471001L800HGN ) from core 0: 206803.20 807.83 4955.33 4366.51 37664.17
INTEL SSDPE2MD800G4 (CVFT5471002H800HGN ) from core 0: 206803.20 807.83 4956.35 4357.68 45025.49
INTEL SSDPE2MD800G4 (CVFT54710015800HGN ) from core 0: 206803.20 807.83 4957.39 3205.82 52456.83
========================================================
Total : 1654425.60 6462.60 4954.01 1088.65 52456.83
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 3 --file-prefix=spdk_pid3192 ]
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD800G4 (CVFT5471002S800HGN ) from core 1: 438592.00 1713.25 2334.48 1363.58 16128.43
INTEL SSDPE2MD800G4 (CVFT5471002K800HGN ) from core 1: 438592.00 1713.25 2335.47 2073.78 14921.21
INTEL SSDPE2MD800G4 (CVFT5471001L800HGN ) from core 1: 438592.00 1713.25 2336.68 2038.52 29808.68
INTEL SSDPE2MD800G4 (CVFT54710015800HGN ) from core 1: 438592.00 1713.25 2337.99 1827.81 44872.18
INTEL SSDPE2MD800G4 (CVFT5471002L800HGN ) from core 0: 437977.60 1710.85 2337.78 1409.41 15688.80
INTEL SSDPE2MD800G4 (CVFT5471001U800HGN ) from core 0: 437977.60 1710.85 2338.77 2034.40 14879.24
INTEL SSDPE2MD800G4 (CVFT54710002800HGN ) from core 0: 437977.60 1710.85 2339.98 2037.44 29795.93
INTEL SSDPE2MD800G4 (CVFT5471002H800HGN ) from core 0: 437977.60 1710.85 2341.29 2037.46 44764.63
========================================================
Total : 3506278.40 13696.40 2337.80 1363.58 44872.18
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c F --file-prefix=spdk_pid3205 ]
Device Information : IOPS MB/s Average min max
INTEL SSDPE2MD800G4 (CVFT5471002K800HGN ) from core 3: 614814.65 2401.62 1665.48 827.93 13076.89
INTEL SSDPE2MD800G4 (CVFT54710015800HGN ) from core 3: 481525.05 1880.96 2128.57 1085.12 37295.10
INTEL SSDPE2MD800G4 (CVFT5471001U800HGN ) from core 2: 542172.10 2117.86 1888.83 920.16 19435.55
INTEL SSDPE2MD800G4 (CVFT5471002H800HGN ) from core 2: 488315.25 1907.48 2099.24 1070.72 42946.29
INTEL SSDPE2MD800G4 (CVFT5471002S800HGN ) from core 1: 585409.45 2286.76 1749.10 897.17 10901.34
INTEL SSDPE2MD800G4 (CVFT5471001L800HGN ) from core 1: 488748.90 1909.18 2097.28 1126.79 43355.29
INTEL SSDPE2MD800G4 (CVFT5471002L800HGN ) from core 0: 742426.50 2900.10 1379.04 706.27 10713.54
INTEL SSDPE2MD800G4 (CVFT54710002800HGN ) from core 0: 488740.80 1909.14 2096.44 1136.78 31606.15
========================================================
Total : 4432152.70 17313.10 1849.11 706.27 43355.29
Any idea what possibly could go wrong here?
On Thu, Feb 15, 2018 at 4:50 PM, Harris, James R <james.r.harris@intel.com> wrote:
Hi Ernest,
The answer depends on many different factors – the IOPs limit per device, CPU core frequency, Turbo, etc. As a frame of reference, the team here at Intel has measured over 3M IO/s on a single Intel Xeon core[1].
-Jim
[1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160809_FA12_P3_Prepalli.pdf - slide 63
From: SPDK <spdk-bounces@lists.01.org> on behalf of Ernest Zed <kreuzerkrieg@gmail.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Thursday, February 15, 2018 at 6:46 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [SPDK] Number of NVMe devices per core
Hi All,
I ran 'perf' with different settings to see how to get most of NVMe disks. I see that performance degrades once cpu mask puts more than 2 devises per core. Is it OK? what is the theoretical (or empirical) limit of devices one core can handle?
Sincerely,
Ernest
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk