Re: [SPDK] Helloworld program fails during device initialization.

[Sreeni (Sreenivasa) Busam (Stellus) <s.busam at stellus.com>]

[-- Attachment #1: Type: text/plain, Size: 6051 bytes --]

Thanks Daniel for your comments. I think the SSD devices are of same model looking at lspci output.
Let me confirm it with my lab administrator.
I will add the debug code and give you the output.

Sreeni

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Tuesday, November 14, 2017 9:43 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Helloworld program fails during device initialization.

It looks to me like the device at 0000:ad:00.0 worked in both tests, and the device at 0000:ae:00.0 failed during identify and wasn't used in perf due to the -r argument specifying only 0000:ad:00.0.

Are these two devices the same model/firmware of NVMe controller, or is there some difference between them?

It would also be useful to see the debug output from the NVMe library for the failing controller; you can enable this in hello_world.c by adding these two lines at the beginning of main():

  spdk_log_set_print_level(SPDK_LOG_DEBUG);
  spdk_log_set_trace_flag("all");

Thanks,
-- Daniel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, November 14, 2017 6:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Helloworld program fails during device initialization.

Well, maybe not.  Could very well be an issue in the driver that's only exposed via this hardware especially given that hello_world failed and perf did not.

Sreeni, do you have any other data points that you can share? Like, for example, have you tried multiple SSDs of that same model or any other model? Does hello consistently fail and perf consistently pass? I haven't done a ton of debug with either but with a little more info, if someone else doesn't jump in with the answer, I can pretty easily help compare the init sequence in a few of these and maybe start to narrow down what's different.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wu, WenzhongX
Sent: Tuesday, November 14, 2017 2:13 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Helloworld program fails during device initialization.

Hi Sreeni,

Verify with newest spdk source code with intel P3700 ,run helloword everything is ok:
root(a)waikiki5-target:~/workspace/spdk1102/examples/nvme/hello_world# ./hello_world
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: hello_world -c 0x1 --file-prefix=spdk0 --base-virtaddr=0x1000000000 --proc-type=auto ]
EAL: Detected 36 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:81:00.0
EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:84:00.0
EAL: PCI device 0000:85:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:85:00.0
Attached to 0000:85:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT6401011B800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Attached to 0000:81:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT5373002D800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Attached to 0000:84:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT53640003800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Initialization complete.
Hello world!
Hello world!
Hello world!

Seem  it's your hardware cause this issues.

Thanks
Wenzhong

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Tuesday, November 14, 2017 2:36 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Helloworld program fails during device initialization.

I was testing the helloworld program on ubuntu with NVMe SSD and the probe() is failing. Please let me know if this is a known problem.
Let me know if there is workaround.
The perf program did not have any problem.
I have attached the output for both tests.
lspci| grep Non
ad:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
ae:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
uname -a
Linux sjca5radevct13 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
1. helloworld output:
---------------------------
Attached to 0000:ad:00.0
Using controller SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) with 1 namespaces.
  Namespace ID: 1 size: 480GB
nvme_ctrlr.c:1475:nvme_ctrlr_process_init: *ERROR*: Initialization timed out in state 1
nvme_ctrlr.c: 523:nvme_ctrlr_shutdown: *ERROR*: did not shutdown within 5 seconds
spdk_nvme_probe() failed

2. perf output:
sudo ./perf -q 1 -s 4096 -t 10 -w read -r "trtype:PCIe traddr:0000:ad:00.0"
---------------
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid7647 ]
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
Attached to NVMe Controller at 0000:ad:00.0 [144d:a802]
Associating SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
                                                                                            Latency(us)
Device Information                                     :       IOPS       MB/s    Average        min        max
SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) from core 0:   13773.50      53.80      72.59      60.20     243.82
========================================================
Total                                                  :   13773.50      53.80      72.59      60.20     243.82

Thanks for your help,
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16983 bytes --]