Comment 4 for bug 1987663

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Copy/paste of commands to check before/after:

... Reproducer (15-second startup delay in tgt.service)

 FILE=/etc/systemd/system/tgt.service.d/start-delay.conf
 mkdir -p $(dirname $FILE)
 cat <<EOF > $FILE
 [Service]
 ExecStartPre=$(which sleep) 15
 EOF
 systemctl daemon-reload

Before:
---

... Restart both services at the same time:

 # date; systemctl restart cinder-volume.service tgt.service; date
 Mon Oct 16 21:57:12 UTC 2023
 Mon Oct 16 21:57:27 UTC 2023

... Notice that cinder-volume.service is Started _BEFORE_ tgt.service

 # journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3
 Oct 16 21:57:12 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.
 Oct 16 21:57:12 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
 Oct 16 21:57:27 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.

.. Log error:

 # grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log
 ...
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager [None req-ce03264a-6765-41de-8016-a6f27d2685e4 - - - - - -] Failed to re-export volume, setting to ERROR.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
 ...
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Exit code: 107
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stdout: ''
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n'
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager

... Volume error:

 # cinder list
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | error | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

... Undo error state:

 # systemctl restart cinder-volume.service
 # cinder reset-state --state in-use test-volume
 # cinder list

 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

After:
---

... Patched with Wants=

 # systemctl show cinder-volume.service | grep Wants=
 Wants=network-online.target tgt.service

 # cinder list
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

... Restart both services at the same time:

 # date; systemctl restart cinder-volume.service tgt.service; date
 Wed Oct 18 15:29:01 UTC 2023
 Wed Oct 18 15:29:17 UTC 2023

... Notice that cinder-volume.service is Started _AFTER_ tgt.service

 # journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3
 Oct 18 15:29:01 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
 Oct 18 15:29:17 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.
 Oct 18 15:29:17 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.

... Volume not in error:

 # cinder list
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

... Without tgt.service installed:

 # apt remove --purge --yes tgt

 # systemctl status tgt.service
 Unit tgt.service could not be found.

 # systemctl show cinder-volume.service | grep Wants=
 Wants=tgt.service network-online.target

 # systemctl restart cinder-volume.service
 # echo $?
 0

 # systemctl status cinder-volume.service | grep Active:
      Active: active (running) since Wed 2023-10-18 16:47:14 UTC; 11s ago

 # journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -1
 Oct 18 16:47:14 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.