Hi Ranjan, Nice to meet you. Iโ€™m adding SPDK mailing list for wider audience (someone may find it helpful someday ๐Ÿ˜Š ). Maybe it has to do something with additional comments on each line in configuration file โ€“ try to remove them. Also in โ€œspdk_rpc_port =4420โ€ you are missing space after โ€œ=โ€ โ€“ I doubt that this may has something to do with that missing space, but it is always worth trying removing it. Everything else looks good. Let me know if that wonโ€™t work โ€“ Iโ€™ll try to reproduce environment on my side. Regards, Maciej From: Kumar Ranjan Sent: Monday, March 23, 2020 6:57 PM To: Szwed, Maciej Subject: Fail to start cinder service Hi Maciej, I am Ranjan from WDC. Request your help in understanding as why I am not able to start the cinder service. I saw some of your patches and looks like you are more into this field. Hence pinging you. For experiment : devstack as well my client is running in the same server. My host IP for Horizon (private network) : 192.168.1.14 My target IP where rpc is listening : 192.168.1.111 My target has 2 data ports , one is : 192.168.1.111 and another : 172.168.1.111 Spdk entried in my Cinder.conf [spdk] spdk_rpc_ip = 192.168.1.111 #address to machine with SPDK spdk_rpc_port =4420 #port to use to send commands to SPDK once running with -r #spdk_rpc_username = admin#username set in remote_rpc.py #spdk_rpc_password = admin#password set in remote_rpc.py target_ip_address = 172.168.1.111 #NVMe-oF interface target_port = 4420 target_protocol = nvmet_rdma target_helper = spdk-nvmeof target_prefix = nqn.1992-05.com.wdc volume_driver = cinder.volume.drivers.spdk.SPDKDriver volume_backend_name = SPDK --------------- root(a)ubuntu:~# systemctl status devstack(a)c-vol.service โ— devstack(a)c-vol.service - Devstack devstack(a)c-vol.service Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2020-03-23 23:16:59 IST; 5s ago Process: 15636 ExecStart=/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf (code=exited, status=1/FAILURE) Main PID: 15636 (code=exited, status=1/FAILURE) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume return self._conf._get(name, self._group) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2639, in _get Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume value, loc = self._do_get(name, group, namespace) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2713, in _do_get Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume raise ConfigFileValueError(message) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume oslo_config.cfg.ConfigFileValueError: Value for option spdk_rpc_port from LocationInfo(location=: Main process exited, code=exited, status=1/FAILURE Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service: Failed with result 'exit-code'. Target : root(a)KAMSHED:/usr/share/spdk/scripts# ps aux | grep nvmf root 14676 691 5.0 4271148 409508 pts/0 SLl 01:28 37:21 /usr/sbin/nvmf_tgt -o -j -c /usr/share/spdk/nvmf.conf -r 192.168.1.111:4420 root 15844 0.0 0.0 2316 464 pts/0 S+ 01:33 0:00 grep nvmf Regards, Ranjan