Verification done on {mantic,jammy,focal}-proposed. The detailed verification steps as per comment #3 are attached. This is the summary with key steps. With the packages from -proposed, when (re)starting {cinder-volume,tgt}.service, the cinder-volume service is started _after_ the tgt service, and the volumes remain with 'in-use' status (no longer 'error' status). Reproducer (45-second startup delay in tgt.service) --- FILE=/etc/systemd/system/tgt.service.d/start-delay.conf mkdir -p $(dirname $FILE) cat < $FILE [Service] ExecStartPre=$(which sleep) 45 EOF systemctl daemon-reload # systemctl show tgt.service | grep ^ExecStartPre= ExecStartPre={ path=/usr/bin/sleep ; argv[]=/usr/bin/sleep 45 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } # date; systemctl restart cinder-volume.service tgt.service; date Before / packages from -updates: --- ... Notice that cinder-volume.service is Started _BEFORE_ tgt.service ... And the 'tgtadm' error _IS_ seen in the logs. ... And volumes are listed with 'error' Status. mantic: root@cinder-mantic:~# dpkg -l | grep cinder-volume ii cinder-volume 2:23.0.0-0ubuntu1.1 all Cinder storage service - Volume server root@cinder-mantic:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 15:54:43 UTC 2024 Tue Apr 30 15:55:29 UTC 2024 root@cinder-mantic:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 15:54:43 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume. Apr 30 15:54:44 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon... Apr 30 15:55:29 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon. root@cinder-mantic:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-mantic:~# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 45b39ae0-2552-4703-a664-ae8de77a23a1 | error | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ jammy: root@cinder-jammy:~# dpkg -l | grep cinder-volume ii cinder-volume 2:20.3.1-0ubuntu1.1 all Cinder storage service - Volume server root@cinder-jammy:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 15:54:45 UTC 2024 Tue Apr 30 15:55:30 UTC 2024 root@cinder-jammy:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 15:54:45 cinder-jammy systemd[1]: Started OpenStack Cinder Volume. Apr 30 15:54:45 cinder-jammy systemd[1]: Starting (i)SCSI target daemon... Apr 30 15:55:30 cinder-jammy systemd[1]: Started (i)SCSI target daemon. root@cinder-jammy:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-jammy:~# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 88024eca-203c-4f8f-9ff0-4de704041a2f | error | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ focal: root@cinder-focal:~# dpkg -l | grep cinder-volume ii cinder-volume 2:16.4.2-0ubuntu2.4 all Cinder storage service - Volume server root@cinder-focal:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 15:54:46 UTC 2024 Tue Apr 30 15:55:31 UTC 2024 root@cinder-focal:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 15:54:46 cinder-focal systemd[1]: Started OpenStack Cinder Volume. Apr 30 15:54:46 cinder-focal systemd[1]: Starting (i)SCSI target daemon... Apr 30 15:55:31 cinder-focal systemd[1]: Started (i)SCSI target daemon. root@cinder-focal:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-focal:~# cinder list +--------------------------------------+--------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+-------------+----------+-------------+ | 5fa41f17-afaf-4a56-8494-8c9246531523 | error | test-volume | 1 | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+-------------+----------+-------------+ After / packages from -proposed ... Notice that cinder-volume.service is Started *AFTER* tgt.service now ... And now the 'tgtadm' error is *NOT* seen in the logs. ... And the volumes are listed with Status 'in-use' (no longer 'error') mantic: root@cinder-mantic:~# apt-cache policy cinder-volume cinder-volume: Installed: 2:23.0.0-0ubuntu1.2 Candidate: 2:23.0.0-0ubuntu1.2 Version table: *** 2:23.0.0-0ubuntu1.2 500 400 http://archive.ubuntu.com/ubuntu mantic-proposed/main amd64 Packages ... root@cinder-mantic:~# dpkg -l | grep cinder-volume ii cinder-volume 2:23.0.0-0ubuntu1.2 all Cinder storage service - Volume server root@cinder-mantic:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 16:34:03 UTC 2024 Tue Apr 30 16:34:49 UTC 2024 root@cinder-mantic:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 16:34:03 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon... Apr 30 16:34:49 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon. Apr 30 16:34:49 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume. (error timestamps are from the previous/non-proposed run) root@cinder-mantic:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-mantic:~# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 45b39ae0-2552-4703-a664-ae8de77a23a1 | in-use | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ jammy: root@cinder-jammy:~# apt-cache policy cinder-volume cinder-volume: Installed: 2:20.3.1-0ubuntu1.2 Candidate: 2:20.3.1-0ubuntu1.2 Version table: *** 2:20.3.1-0ubuntu1.2 400 400 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages ... root@cinder-jammy:~# dpkg -l | grep cinder-volume ii cinder-volume 2:20.3.1-0ubuntu1.2 all Cinder storage service - Volume server root@cinder-jammy:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 16:34:04 UTC 2024 Tue Apr 30 16:34:50 UTC 2024 root@cinder-jammy:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 16:34:04 cinder-jammy systemd[1]: Starting (i)SCSI target daemon... Apr 30 16:34:50 cinder-jammy systemd[1]: Started (i)SCSI target daemon. Apr 30 16:34:50 cinder-jammy systemd[1]: Started OpenStack Cinder Volume. (error timestamps are from the previous/non-proposed run) root@cinder-jammy:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:55.918 14576 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-jammy:~# cinder list +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ | 88024eca-203c-4f8f-9ff0-4de704041a2f | in-use | test-volume | 1 | True | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+ focal: root@cinder-focal:~# apt-cache policy cinder-volume cinder-volume: Installed: 2:16.4.2-0ubuntu2.5 Candidate: 2:16.4.2-0ubuntu2.5 Version table: *** 2:16.4.2-0ubuntu2.5 500 400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages ... root@cinder-focal:~# dpkg -l | grep cinder-volume ii cinder-volume 2:16.4.2-0ubuntu2.5 all Cinder storage service - Volume server root@cinder-focal:~# date; systemctl restart cinder-volume.service tgt.service; date Tue Apr 30 16:34:05 UTC 2024 Tue Apr 30 16:34:51 UTC 2024 root@cinder-focal:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3 Apr 30 16:34:05 cinder-focal systemd[1]: Starting (i)SCSI target daemon... Apr 30 16:34:51 cinder-focal systemd[1]: Started (i)SCSI target daemon. Apr 30 16:34:51 cinder-focal systemd[1]: Started OpenStack Cinder Volume. (error timestamps are from the previous/non-proposed run) root@cinder-focal:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show() 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target 2024-04-30 15:54:55.830 15834 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n' root@cinder-focal:~# cinder list +--------------------------------------+--------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+-------------+------+-------------+----------+-------------+ | 5fa41f17-afaf-4a56-8494-8c9246531523 | in-use | test-volume | 1 | __DEFAULT__ | false | | +--------------------------------------+--------+-------------+------+-------------+----------+-------------+