[SPDK] Re: Fail to start cinder service

[SPDK] Re: Fail to start cinder service

[Szwed, Maciej]

[-- Attachment #1: Type: text/plain, Size: 4129 bytes --]

Hi Ranjan,
Nice to meet you. I’m adding SPDK mailing list for wider audience (someone may find it helpful someday 😊 ). Maybe it has to do something with additional comments on each line in configuration file – try to remove them. Also in “spdk_rpc_port =4420” you are missing space after “=” – I doubt that this may has something to do with that missing space, but it is always worth trying removing it. Everything else looks good. Let me know if that won’t work – I’ll try to reproduce environment on my side.

Regards,
Maciej


From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com>
Sent: Monday, March 23, 2020 6:57 PM
To: Szwed, Maciej <maciej.szwed(a)intel.com>
Subject: Fail to start cinder service

Hi Maciej,
I am Ranjan from WDC. Request your  help in understanding as why I am not able to start the cinder service.
I saw some of your patches and looks like you are more into this field. Hence pinging you.

For experiment : devstack as well my client is running in the same server.

My host IP for Horizon (private network) : 192.168.1.14
My target IP where rpc is listening : 192.168.1.111
My target has 2 data ports , one is : 192.168.1.111 and another : 172.168.1.111

Spdk entried in my  Cinder.conf

[spdk]
spdk_rpc_ip = 192.168.1.111 #address to machine with SPDK
spdk_rpc_port =4420 #port to use to send commands to SPDK once running with -r
#spdk_rpc_username = admin#username set in remote_rpc.py
#spdk_rpc_password = admin#password set in remote_rpc.py
target_ip_address = 172.168.1.111 #NVMe-oF interface
target_port = 4420
target_protocol = nvmet_rdma
target_helper = spdk-nvmeof
target_prefix = nqn.1992-05.com.wdc
volume_driver = cinder.volume.drivers.spdk.SPDKDriver
volume_backend_name = SPDK

---------------


root(a)ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
● devstack(a)c-vol.service<mailto:devstack(a)c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
   Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack(a)c-vol.service>; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-03-23 23:16:59 IST; 5s ago
  Process: 15636 ExecStart=/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf (code=exited, status=1/FAILURE)
Main PID: 15636 (code=exited, status=1/FAILURE)

Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     return self._conf._get(name, self._group)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume   File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2639, in _get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     value, loc = self._do_get(name, group, namespace)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume   File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2713, in _do_get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     raise ConfigFileValueError(message)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume oslo_config.cfg.ConfigFileValueError: Value for option spdk_rpc_port from LocationInfo(location=<Locations.user: (4, Tru
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume [None req-5dddc9ec-a2a6-4c94-913c-caea68014ac5 None None] No volume service(s) started successfully, terminating.
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>: Main process exited, code=exited, status=1/FAILURE
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>: Failed with result 'exit-code'.


Target :

root(a)KAMSHED:/usr/share/spdk/scripts# ps aux | grep nvmf
root     14676  691  5.0 4271148 409508 pts/0  SLl  01:28  37:21 /usr/sbin/nvmf_tgt -o -j -c /usr/share/spdk/nvmf.conf -r 192.168.1.111:4420
root     15844  0.0  0.0   2316   464 pts/0    S+   01:33   0:00 grep nvmf



Regards,
Ranjan

[SPDK] Re: Fail to start cinder service

[Szwed, Maciej]

[-- Attachment #1: Type: text/plain, Size: 8043 bytes --]

Hi Ranjan,
I happy that this helped to proceed further. The log you pasted below is cut (lines are cut after some number of words) – could you please try to get the rest of the lines as it seems that most important information was there?

Thanks,
Maciek

From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com>
Sent: Tuesday, March 31, 2020 7:42 AM
To: Szwed, Maciej <maciej.szwed(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: Fail to start cinder service

Thanks a ton for such instant response Szwed. My bad, I just missed this email.
I removed the comments, and at least the service is running. I am yet to check if I can access the volume from horizon. Will update you soon on this.

root(a)ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
● devstack(a)c-vol.service<mailto:devstack(a)c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
   Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack(a)c-vol.service>; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-31 11:09:03 IST; 23s ago
Main PID: 18375 (cinder-volume)
    Tasks: 2 (limit: 6143)
   CGroup: /system.slice/system-devstack.slice/devstack(a)c-vol.service<mailto:/system.slice/system-devstack.slice/devstack(a)c-vol.service>
           ├─18375 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf
           └─18421 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf

Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager     raise exception.VolumeBackendAPIException(data=err_msg)
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Could no
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Creating RPC server for service cinder-volume {{(pid=18421) start
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG oslo_db.sqlalchemy.engines [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_A
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Pinning object versions for RPC server serializer to 1.38 {{(pid=
Mar 31 11:09:13 ubuntu cinder-volume[18375]: INFO cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Initializing RPC dependent components of volume driver SPDK
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.utils [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Volume driver SPDKDriver not initialized
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Cannot complete RPC initialization because driver isn't in
Mar 31 11:09:23 ubuntu cinder-volume[18375]: ERROR cinder.service [-] Manager for service cinder-volume ubuntu(a)spdk is reporting problems, not sending heartbeat. Service will appear "down".


Regards,
Ranjan

From: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed(a)intel.com>>
Sent: Tuesday, March 24, 2020 7:38 PM
To: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan(a)wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: RE: Fail to start cinder service

CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.

Hi Ranjan,
Nice to meet you. I’m adding SPDK mailing list for wider audience (someone may find it helpful someday 😊 ). Maybe it has to do something with additional comments on each line in configuration file – try to remove them. Also in “spdk_rpc_port =4420” you are missing space after “=” – I doubt that this may has something to do with that missing space, but it is always worth trying removing it. Everything else looks good. Let me know if that won’t work – I’ll try to reproduce environment on my side.

Regards,
Maciej


From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan(a)wdc.com>>
Sent: Monday, March 23, 2020 6:57 PM
To: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed(a)intel.com>>
Subject: Fail to start cinder service

Hi Maciej,
I am Ranjan from WDC. Request your  help in understanding as why I am not able to start the cinder service.
I saw some of your patches and looks like you are more into this field. Hence pinging you.

For experiment : devstack as well my client is running in the same server.

My host IP for Horizon (private network) : 192.168.1.14
My target IP where rpc is listening : 192.168.1.111
My target has 2 data ports , one is : 192.168.1.111 and another : 172.168.1.111

Spdk entried in my  Cinder.conf

[spdk]
spdk_rpc_ip = 192.168.1.111 #address to machine with SPDK
spdk_rpc_port =4420 #port to use to send commands to SPDK once running with -r
#spdk_rpc_username = admin#username set in remote_rpc.py
#spdk_rpc_password = admin#password set in remote_rpc.py
target_ip_address = 172.168.1.111 #NVMe-oF interface
target_port = 4420
target_protocol = nvmet_rdma
target_helper = spdk-nvmeof
target_prefix = nqn.1992-05.com.wdc
volume_driver = cinder.volume.drivers.spdk.SPDKDriver
volume_backend_name = SPDK

---------------


root(a)ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
● devstack(a)c-vol.service<mailto:devstack(a)c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>
   Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack(a)c-vol.service>; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-03-23 23:16:59 IST; 5s ago
  Process: 15636 ExecStart=/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf (code=exited, status=1/FAILURE)
Main PID: 15636 (code=exited, status=1/FAILURE)

Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     return self._conf._get(name, self._group)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume   File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2639, in _get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     value, loc = self._do_get(name, group, namespace)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume   File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2713, in _do_get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume     raise ConfigFileValueError(message)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume oslo_config.cfg.ConfigFileValueError: Value for option spdk_rpc_port from LocationInfo(location=<Locations.user: (4, Tru
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume [None req-5dddc9ec-a2a6-4c94-913c-caea68014ac5 None None] No volume service(s) started successfully, terminating.
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>: Main process exited, code=exited, status=1/FAILURE
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack(a)c-vol.service>: Failed with result 'exit-code'.


Target :

root(a)KAMSHED:/usr/share/spdk/scripts# ps aux | grep nvmf
root     14676  691  5.0 4271148 409508 pts/0  SLl  01:28  37:21 /usr/sbin/nvmf_tgt -o -j -c /usr/share/spdk/nvmf.conf -r 192.168.1.111:4420
root     15844  0.0  0.0   2316   464 pts/0    S+   01:33   0:00 grep nvmf



Regards,
Ranjan