Re: [SPDK] NVMf Target configuration issue

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 9131 bytes --]

Hi Gyan,

libnuma/numactl is a recently added package dependency.  You can run scripts/pkgdep.sh or just install it explicitly – numactl-devel on RHEL/CentOS or  libnuma-dev on Ubuntu/Debian.

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Gyan Prakash <gyapra2016(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, October 17, 2017 at 5:36 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] NVMf Target configuration issue

Hi all,
I got the latest master of SPDK and getting following error:

/root/test-spdk/spdk/dpdk/lib/librte_eal/linuxapp/eal/eal_memory.c:56:18: fatal error: numa.h: No such file or directory
 #include <numa.h>
                  ^
compilation terminated.
  CC eal_timer.o
  CC eal_interrupts.o
  CC eal_alarm.o
make[8]: *** [eal_memory.o] Error 1
make[8]: *** Waiting for unfinished jobs....
make[7]: *** [eal] Error 2
make[6]: *** [linuxapp] Error 2
make[5]: *** [librte_eal] Error 2
make[4]: *** [lib] Error 2
make[3]: *** [all] Error 2
make[3]: Leaving directory `/root/test-spdk/spdk/dpdk'
make[2]: *** [all] Error 2
make[1]: *** [all] Error 2
make: *** [dpdkbuild] Error 2

I am using the in-built DPDK and I do not see the file "numa.h" anywhere. Am I supposed to get something installed on my system to get the numa.h?
I used "make CONFIG_RDMA=y CONFIG_DEBUG=y" as my build command.

Thanks & regards,
Gyan


On Tue, Oct 17, 2017 at 1:03 PM, Gyan Prakash <gyapra2016(a)gmail.com<mailto:gyapra2016(a)gmail.com>> wrote:
Hi Daniel,

Thanks for the reply. You are correct, I am using SPDK v17.03. I will test with SPDK v17.07.

I will also check the referenced documents for the configuration.

Thanks,
Gyan

On Tue, Oct 17, 2017 at 12:54 PM, Verkamp, Daniel <daniel.verkamp(a)intel.com<mailto:daniel.verkamp(a)intel.com>> wrote:
Hi Gyan,

It looks like you are using an older version of SPDK before the NVMe-oF target was changed to always use the bdev abstraction layer.

Can you retest with the latest master, or at least SPDK v17.07?  To do this, you will need to update your nvmf.conf to specify the NVMe devices you want to use as bdevs – see the NVMe bdev configuration docs [1] and NVMe-oF target Getting Started guide [2] for more details.

Thanks,
-- Daniel

[1]: http://www.spdk.io/doc/bdev.html#bdev_config_nvme
[2]: http://www.spdk.io/doc/nvmf.html#nvmf_getting_started

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Gyan Prakash
Sent: Tuesday, October 17, 2017 12:05 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] NVMf Target configuration issue

Hi all,
I see issue with NVMf Target configuration. I also see the thread  [SPDK] NVMf target configuration issue Thu Sep 7 01:29:33 PDT 2017 which is for the similar issue what I am seeing. I tried the suggestion from that thread, but it was not helpful for my problem.

 Error Message:
 # ./nvmf_tgt -t all -c nvmf.conf
Starting DPDK 17.02.0 initialization...
[ DPDK EAL parameters: nvmf -c fff --file-prefix=spdk_pid3911 ]
EAL: Detected 12 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Occupied cpu core mask is 0xfff
Occupied cpu socket mask is 0x1
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL:   probe driver: 8086:2f20 spdk_ioat
 Found matching device at 0000:00:04.0 vendor:0x8086 device:0x2f20
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL:   probe driver: 8086:2f21 spdk_ioat
 Found matching device at 0000:00:04.1 vendor:0x8086 device:0x2f21
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL:   probe driver: 8086:2f22 spdk_ioat
 Found matching device at 0000:00:04.2 vendor:0x8086 device:0x2f22
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL:   probe driver: 8086:2f23 spdk_ioat
 Found matching device at 0000:00:04.3 vendor:0x8086 device:0x2f23
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL:   probe driver: 8086:2f24 spdk_ioat
 Found matching device at 0000:00:04.4 vendor:0x8086 device:0x2f24
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL:   probe driver: 8086:2f25 spdk_ioat
 Found matching device at 0000:00:04.5 vendor:0x8086 device:0x2f25
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL:   probe driver: 8086:2f26 spdk_ioat
 Found matching device at 0000:00:04.6 vendor:0x8086 device:0x2f26
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL:   probe driver: 8086:2f27 spdk_ioat
 Found matching device at 0000:00:04.7 vendor:0x8086 device:0x2f27
Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
Probing device 0000:01:00.0
[nvme] nvme_ctrlr.c: 502:nvme_ctrlr_set_state: setting state to init (no timeout)
[nvme] nvme_ctrlr.c:1135:nvme_ctrlr_process_init: CC.EN = 1
[nvme] nvme_ctrlr.c:1149:nvme_ctrlr_process_init: Setting CC.EN = 0
[nvme] nvme_ctrlr.c: 506:nvme_ctrlr_set_state: setting state to disable and wait for CSTS.RDY = 0 (timeout 20000 ms)
[nvme] nvme_ctrlr.c:1206:nvme_ctrlr_process_init: CC.EN = 0 && CSTS.RDY = 0 - enabling controller
[nvme] nvme_ctrlr.c:1208:nvme_ctrlr_process_init: Setting CC.EN = 1
[nvme] nvme_ctrlr.c: 506:nvme_ctrlr_set_state: setting state to enable and wait for CSTS.RDY = 1 (timeout 20000 ms)
[nvme] nvme_ctrlr.c:1217:nvme_ctrlr_process_init: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
[nvme] nvme_ctrlr.c: 594:nvme_ctrlr_identify: transport max_xfer_size 2072576
[nvme] nvme_ctrlr.c: 598:nvme_ctrlr_identify: MDTS max_xfer_size 131072
[nvme] nvme_ctrlr.c: 673:nvme_ctrlr_set_keep_alive_timeout: Controller KAS is 0 - not enabling Keep Alive
[nvme] nvme_ctrlr.c: 502:nvme_ctrlr_set_state: setting state to ready (no timeout)
[debug] bdev.c: 932:spdk_bdev_register: Inserting bdev Nvme0n1 into list
Total cores available: 12
Reactor started on core 1 on socket 0
Reactor started on core 2 on socket 0
Reactor started on core 3 on socket 0
Reactor started on core 4 on socket 0
Reactor started on core 5 on socket 0
Reactor started on core 6 on socket 0
Reactor started on core 7 on socket 0
Reactor started on core 8 on socket 0
Reactor started on core 9 on socket 0
Reactor started on core 10 on socket 0
Reactor started on core 0 on socket 0
[nvmf] nvmf.c:  68:spdk_nvmf_tgt_init: Max Queues Per Session: 4
[nvmf] nvmf.c:  69:spdk_nvmf_tgt_init: Max Queue Depth: 128
[nvmf] nvmf.c:  70:spdk_nvmf_tgt_init: Max In Capsule Data: 4096 bytes
[nvmf] nvmf.c:  71:spdk_nvmf_tgt_init: Max I/O Size: 131072 bytes
*** RDMA Transport Init ***
Reactor started on core 11 on socket 0
allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 5 on socket 0
[rdma] rdma.c:1177:spdk_nvmf_rdma_listen: For listen id 0x1720840 with context 0x1720db0, created completion channel 0x1720ad0
conf.c: 555:spdk_nvmf_construct_subsystem: ***ERROR*** Subsystem nqn.2016-06.io.spdk:cnode1: missing NVMe directive
[nvmf] subsystem.c: 231:spdk_nvmf_delete_subsystem: subsystem is 0x1720b20
nvmf_tgt.c: 313:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf() failed
[nvmf] subsystem.c: 231:spdk_nvmf_delete_subsystem: subsystem is 0x17205b0
[nvme] nvme_ctrlr.c: 406:nvme_ctrlr_shutdown: shutdown complete





# ./setup.sh
0000:01:00.0 (8086 0953): nvme -> uio_pci_generic
0000:00:04.0 (8086 2f20): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2f21): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2f22): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2f23): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2f24): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2f25): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2f26): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2f27): ioatdma -> uio_pci_generic

# lspci | grep -i vola*
01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)



# lspci -v -s 0000:01:00.0
01:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Intel Corporation DC P3700 SSD
        Physical Slot: 1
        Flags: fast devsel, IRQ 24, NUMA node 0
        Memory at fb410000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fb400000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI-X: Enable- Count=32 Masked-
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Virtual Channel
        Capabilities: [180] Power Budgeting <?>
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [270] Device Serial Number 55-cd-2e-41-4d-34-1e-1b
        Capabilities: [2a0] #19
        Kernel driver in use: uio_pci_generic
        Kernel modules: nvme



_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 39779 bytes --]