Re: [SPDK] Number of NVMe devices per core

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6025 bytes --]

Can you confirm you aren’t using a debug build?  ‘cat CONFIG.local’ will confirm.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, February 15, 2018 at 8:26 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Number of NVMe devices per core

Dual socket Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, HT off
8 x Intel® SSD DC P3700 Series
16 x 16 DIMM DDR4  2133 MHz
Ubuntu 17.10
4.13.0-32-generic
SPDK version - cloned today

The hardware quite close to the one in the presentation. now results I get with perf for 1, 2 and 4 cores

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid3180 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  206803.20     807.83    4951.17    1088.65   28467.64
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 0:  206803.20     807.83    4951.71    4392.74   25127.60
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  206803.20     807.83    4952.46    4379.57   21966.62
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 0:  206803.20     807.83    4953.35    4379.25   23165.75
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  206803.20     807.83    4954.31    4357.84   30302.48
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 0:  206803.20     807.83    4955.33    4366.51   37664.17
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  206803.20     807.83    4956.35    4357.68   45025.49
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 0:  206803.20     807.83    4957.39    3205.82   52456.83
========================================================
Total                                                  : 1654425.60    6462.60    4954.01    1088.65   52456.83

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 3 --file-prefix=spdk_pid3192 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  438592.00    1713.25    2334.48    1363.58   16128.43
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 1:  438592.00    1713.25    2335.47    2073.78   14921.21
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  438592.00    1713.25    2336.68    2038.52   29808.68
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 1:  438592.00    1713.25    2337.99    1827.81   44872.18
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  437977.60    1710.85    2337.78    1409.41   15688.80
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 0:  437977.60    1710.85    2338.77    2034.40   14879.24
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  437977.60    1710.85    2339.98    2037.44   29795.93
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 0:  437977.60    1710.85    2341.29    2037.46   44764.63
========================================================
Total                                                  : 3506278.40   13696.40    2337.80    1363.58   44872.18

Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c F --file-prefix=spdk_pid3205 ]
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPE2MD800G4  (CVFT5471002K800HGN  ) from core 3:  614814.65    2401.62    1665.48     827.93   13076.89
INTEL SSDPE2MD800G4  (CVFT54710015800HGN  ) from core 3:  481525.05    1880.96    2128.57    1085.12   37295.10
INTEL SSDPE2MD800G4  (CVFT5471001U800HGN  ) from core 2:  542172.10    2117.86    1888.83     920.16   19435.55
INTEL SSDPE2MD800G4  (CVFT5471002H800HGN  ) from core 2:  488315.25    1907.48    2099.24    1070.72   42946.29
INTEL SSDPE2MD800G4  (CVFT5471002S800HGN  ) from core 1:  585409.45    2286.76    1749.10     897.17   10901.34
INTEL SSDPE2MD800G4  (CVFT5471001L800HGN  ) from core 1:  488748.90    1909.18    2097.28    1126.79   43355.29
INTEL SSDPE2MD800G4  (CVFT5471002L800HGN  ) from core 0:  742426.50    2900.10    1379.04     706.27   10713.54
INTEL SSDPE2MD800G4  (CVFT54710002800HGN  ) from core 0:  488740.80    1909.14    2096.44    1136.78   31606.15
========================================================
Total                                                  : 4432152.70   17313.10    1849.11     706.27   43355.29

Any idea what possibly could go wrong here?


On Thu, Feb 15, 2018 at 4:50 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Ernest,

The answer depends on many different factors – the IOPs limit per device, CPU core frequency, Turbo, etc.  As a frame of reference, the team here at Intel has measured over 3M IO/s on a single Intel Xeon core[1].

-Jim

[1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160809_FA12_P3_Prepalli.pdf - slide 63


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of Ernest Zed <kreuzerkrieg(a)gmail.com<mailto:kreuzerkrieg(a)gmail.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, February 15, 2018 at 6:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Number of NVMe devices per core

Hi All,
I ran 'perf' with different settings to see how to get most of NVMe disks. I see that performance degrades once cpu mask puts more than 2 devises per core. Is it OK? what is the theoretical (or empirical) limit of devices one core can handle?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 18671 bytes --]