Re: [SPDK] Helloworld program fails during device initialization.

[Sreeni (Sreenivasa) Busam (Stellus) <s.busam at stellus.com>]

[-- Attachment #1: Type: text/plain, Size: 5769 bytes --]

Hi Paul,

Thanks for your comments. Please see my response inline.


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, November 14, 2017 5:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Helloworld program fails during device initialization.

Well, maybe not.  Could very well be an issue in the driver that's only exposed via this hardware especially given that hello_world failed and perf did not.

Sreeni, do you have any other data points that you can share? Like, for example, have you tried multiple SSDs of that same model or any other model? Does hello consistently fail and perf consistently pass?
>> As you have seen from the log, we know it fails in initialization of SSD device.
nvme_ctrlr.c:1475:nvme_ctrlr_process_init: *ERROR*: Initialization timed out in state 1
I have 2 SSD's and one of them fails in initialization, in case of other device, looks it has hardware problem and the SPDK driver does not see it. I am going to get another SSD and try the same test and will post you the results after testing it. It may take a while as I have to contact my Lab personnel. I have not tested on other models.
Yes, the test on perf always passes and the hello_world failed every time. I tried at least 5 times.
I haven't done a ton of debug with either but with a little more info, if someone else doesn't jump in with the answer, I can pretty easily help compare the init sequence in a few of these and maybe start to narrow down what's different.
>> Do you think it would be good to try to increase the timeout and check if initialization succeeds? Any help would be appreciated.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wu, WenzhongX
Sent: Tuesday, November 14, 2017 2:13 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Helloworld program fails during device initialization.

Hi Sreeni,

Verify with newest spdk source code with intel P3700 ,run helloword everything is ok:
root(a)waikiki5-target:~/workspace/spdk1102/examples/nvme/hello_world# ./hello_world
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: hello_world -c 0x1 --file-prefix=spdk0 --base-virtaddr=0x1000000000 --proc-type=auto ]
EAL: Detected 36 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:81:00.0
EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:84:00.0
EAL: PCI device 0000:85:00.0 on NUMA socket 1
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:85:00.0
Attached to 0000:85:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT6401011B800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Attached to 0000:81:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT5373002D800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Attached to 0000:84:00.0
Using controller INTEL SSDPEDMD800G4  (PHFT53640003800CGN  ) with 1 namespaces.
  Namespace ID: 1 size: 800GB
Initialization complete.
Hello world!
Hello world!
Hello world!

Seem  it's your hardware cause this issues.

Thanks
Wenzhong

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Tuesday, November 14, 2017 2:36 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Helloworld program fails during device initialization.

I was testing the helloworld program on ubuntu with NVMe SSD and the probe() is failing. Please let me know if this is a known problem.
Let me know if there is workaround.
The perf program did not have any problem.
I have attached the output for both tests.
lspci| grep Non
ad:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
ae:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
uname -a
Linux sjca5radevct13 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
1. helloworld output:
---------------------------
Attached to 0000:ad:00.0
Using controller SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) with 1 namespaces.
  Namespace ID: 1 size: 480GB
nvme_ctrlr.c:1475:nvme_ctrlr_process_init: *ERROR*: Initialization timed out in state 1
nvme_ctrlr.c: 523:nvme_ctrlr_shutdown: *ERROR*: did not shutdown within 5 seconds
spdk_nvme_probe() failed

2. perf output:
sudo ./perf -q 1 -s 4096 -t 10 -w read -r "trtype:PCIe traddr:0000:ad:00.0"
---------------
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid7647 ]
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
Attached to NVMe Controller at 0000:ad:00.0 [144d:a802]
Associating SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
                                                                                            Latency(us)
Device Information                                     :       IOPS       MB/s    Average        min        max
SAMSUNG MZQWV480HCGM (S1TRNYAFA00148      ) from core 0:   13773.50      53.80      72.59      60.20     243.82
========================================================
Total                                                  :   13773.50      53.80      72.59      60.20     243.82

Thanks for your help,
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 15563 bytes --]