cinder-volume: "Failed to re-export volume, setting to ERROR" with "tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected" on service startup

Bug #1987663 reported by Mauricio Faria de Oliveira
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Status tracked in Caracal
Antelope
Triaged
Undecided
Unassigned
Bobcat
In Progress
Undecided
Unassigned
Caracal
Fix Released
Undecided
Unassigned
Ussuri
Triaged
Undecided
Unassigned
Victoria
Won't Fix
Undecided
Unassigned
Wallaby
Won't Fix
Undecided
Unassigned
Xena
Won't Fix
Undecided
Unassigned
Yoga
In Progress
Undecided
Unassigned
Zed
Won't Fix
Undecided
Unassigned
cinder (Ubuntu)
Fix Released
Medium
Mauricio Faria de Oliveira
Bionic
Won't Fix
Medium
Unassigned
Focal
Fix Committed
Medium
Mauricio Faria de Oliveira
Jammy
Fix Committed
Medium
Mauricio Faria de Oliveira
Lunar
Won't Fix
Undecided
Unassigned
Mantic
Fix Released
Medium
Mauricio Faria de Oliveira
Noble
Fix Released
Medium
Mauricio Faria de Oliveira

Bug Description

[Impact]

 * The cinder-volume service might fail to re-export volumes
   in-use on startup if tgt.service isn't fully started yet.

 * This affects the 'lvm' driver with 'tgtadm' target helper
   (which runs 'tgtadm' commands that need the service ready).

 * Snippets from /var/log/cinder/cinder-volume.log:

   Failed to re-export volume, setting to ERROR.
   ...
   Command: tgtadm --lld iscsi --op show --mode target
   ...
   Stderr: 'tgtadm: failed to send request hdr to tgt daemon,
   Transport endpoint is not connected\n'

 * This issue is more common in openstack compute nodes
   with networking (ovs/ovn) that takes long to startup,
   which might delay the startup of tgt.service _after_
   cinder-volume.service.

[Test Steps]

 * Steps to reproduce are detailed in comment #3.
   Summary:

 * Install mysql, rabbitmq-server, keystone, and cinder
   (controller and storage nodes; backup node unneeded).

 * Configure cinder-volume (storage node) for LVM backend
   and tgtadm iSCSI helper (tgt.service).

 * Create a cinder volume, and configure it as 'in-use'.

 * Simulate a start delay on tgt.service with a drop-in.

 * Restart services: cinder-volume.service tgt.service

 * Check sequence of service startup.

 * Check status of the cinder volume:
   'in-use' (expected) or 'error' (bug).

 * Check log file /var/log/cinder/cinder-volume.log for
   'tgtadm: failed to send request hdr to tgt daemon'.

[Regression Potential]

 * The fix introduces systemd unit 'After=' and 'Wants='
   properties for tgt.service in cinder-volume.service,
   thus might delay the boot process (multi-user.target).

     $ systemctl show cinder-volume.service | grep WantedBy=
     WantedBy=multi-user.target

 * However, the boot process already waits on tgt.service
   anyway, thus the difference (if any) should not be big,
   and would provide more correct behavior.

     $ systemctl show tgt.service | grep WantedBy=
     WantedBy=multi-user.target

 * If tgt.service is not present (tgt package not installed)
   _no errors_ occur, as both 'After=' and 'Wants=' are weak
   ordering/dependency properties (man 5 systemd.unit).

[Other Info]

 * The fix uses a systemd service drop-in snippet because
   the service unit is generated by openstack-pkg-tools
   (pkgos-gen-systemd-unit) based on the 'init' service,
   and it only emits 'Wants=' for network-online.target.

 * Changing that in openstack-pkg-tools changes behavior
   in stable releases, and only manifest at build time,
   for many openstack packages that have no issues now.

 * We'll continue to pursue the general improvement in
   Debian, so it comes into Ubuntu development release,
   but for the Ubuntu stable releases, this should do.

[Original Bug Description]

Real-world example in comment #2.

Related branches

Changed in cinder (Ubuntu):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
description: updated
Changed in cinder (Debian):
status: Unknown → Fix Released
summary: - "Failed to re-export volume, setting to ERROR" with "tgtadm: failed to
- send request hdr to tgt daemon, Transport endpoint is not connected" on
- cinder-volume service start
+ cinder-volume: "Failed to re-export volume, setting to ERROR" with
+ "tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is
+ not connected" on service startup
description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (5.2 KiB)

Real-world example:

1) Unit `ovs-vswitchd.service` took 2 minutes to start up.
2) That delayed `network.target` (`ovs-vswitchd.service` has `Before=network.target`).
3) That delayed `tgt.service` too (it has `After=network.target`).

4) BUT that did _not_ delay `cinder-volume.service` (it does _not_ have `After=tgt.service`)
5) So it failed to talk to tgtd with tgtadm, and volume re-export failed.

 $ cat sos_commands/logs/journalctl_--no-pager_--catalog_--boot \
   | grep -E -B1 'Subject: Unit (cinder-volume.service|tgt.service|ovs-vswitchd.service|network.target)'

 Aug 10 06:23:15 <HOST> systemd[1]: Started OpenStack Cinder Volume.
 -- Subject: Unit cinder-volume.service has finished start-up
 --
 <<< ERROR in cinder-volume.log >>>

 Aug 10 06:23:16 <HOST> systemd[1]: Starting Open vSwitch Forwarding Unit...
 -- Subject: Unit ovs-vswitchd.service has begun start-up
 --
 <<< DELAY of 2 minutes >>>

 Aug 10 06:25:17 <HOST> systemd[1]: Started Open vSwitch Forwarding Unit.
 -- Subject: Unit ovs-vswitchd.service has finished start-up
 --
 Aug 10 06:25:17 <HOST> systemd[1]: Reached target Network.
 -- Subject: Unit network.target has finished start-up
 --
 Aug 10 06:25:17 <HOST> systemd[1]: Starting (i)SCSI target daemon...
 -- Subject: Unit tgt.service has begun start-up
 --
 Aug 10 06:25:19 <HOST> systemd[1]: Started (i)SCSI target daemon.
 -- Subject: Unit tgt.service has finished start-up

 <<< NOW tgtd is running >>>

@ var/log/cinder/cinder-volume.log

  9901 2022-08-10 06:23:21.515 3939 INFO cinder.service [-] Starting cinder-volume node (version 12.0.9)
 ...
  9932 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager [req-e67a852a-6cce-4073-a2de-d3487c85d586 - - - - -] Failed to re-export volume, setting to ERROR.: ProcessExecutionError: Unexpected error while running command.
  9933 Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target
  9934 Exit code: 107
  9935 Stdout: u''
  9936 Stderr: u'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n'
  9937 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager Traceback (most recent call last):
  9938 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 486, in init_host
  9939 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager self.driver.ensure_export(ctxt, volume)
  9940 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 826, in ensure_export
  9941 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager self.target_driver.ensure_export(context, volume, volume_path)
  9942 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/targets/iscsi.py", line 261, in ensure_export
  9943 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager old_name=None, **portals_config)
  9944 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 818, in _wrapper
  9945 2022-08-10 06:23:23.398 3932 ERROR cinder.volume.manager return r.call(f, ...

Read more...

description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (13.1 KiB)

Steps to Reproduce (Mantic)

# LXD VM

$ lxc launch --vm --config limits.cpu=2 --config limits.memory=3GiB ubuntu:mantic cinder-mantic
$ lxc shell cinder-mantic

# Apt

apt remove -y unattended-upgrades
apt update

# Network

echo '127.0.0.1 controller' >>/etc/hosts

# MySQL

apt install -y mysql-server python3-pymysql

# RabbitMQ

apt install -y rabbitmq-server

rabbitmqctl add_user openstack RABBIT_PASS
rabbitmqctl set_permissions openstack ".*" ".*" ".*"

# Keystone

cat <<EOF | mysql
CREATE DATABASE keystone;
CREATE USER 'keystone'@'localhost' IDENTIFIED BY 'KEYSTONE_DBPASS';
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost';
EOF

apt install -y keystone

@ /etc/keystone/keystone.conf

[database]
connection = mysql+pymysql://keystone:KEYSTONE_DBPASS@controller/keystone

[token]
provider = fernet

...

   # mantic has some exceptions ignored in these, but worked fine:

su -s /bin/sh -c "keystone-manage db_sync" keystone # exit code 0
keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
keystone-manage credential_setup --keystone-user keystone --keystone-group keystone
keystone-manage bootstrap --bootstrap-password ADMIN_PASS \
  --bootstrap-admin-url http://controller:5000/v3/ \
  --bootstrap-internal-url http://controller:5000/v3/ \
  --bootstrap-public-url http://controller:5000/v3/ \
  --bootstrap-region-id RegionOne

echo 'ServerName controller' >>/etc/apache2/apache2.conf
systemctl restart apache2.service

# Openstack Client

cat <<EOF >openstack.rc
export OS_USERNAME=admin
export OS_PASSWORD=ADMIN_PASS
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_AUTH_URL=http://controller:5000/v3
export OS_IDENTITY_API_VERSION=3
EOF

source openstack.rc

apt install -y python3-openstackclient

openstack project create --domain default \
  --description "Service Project" service

+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Service Project |
| domain_id | default |
| enabled | True |
| id | 229ef5671d0b4136b32f1a60584ab725 |
| is_domain | False |
| name | service |
| options | {} |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+

# openstack project list
+----------------------------------+---------+
| ID | Name |
+----------------------------------+---------+
| 0e204837047a4323b8f57ed23bffa4f8 | admin |
| 229ef5671d0b4136b32f1a60584ab725 | service |
+----------------------------------+---------+

# Cinder

cat <<EOF | mysql
CREATE DATABASE cinder;
CREATE USER 'cinder'@'localhost' IDENTIFIED BY 'CINDER_DBPASS';
GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost';
EOF

openstack user create --domain default --password CINDER_USERPASS cinder
openstack role add --project service --user cinder admin

open...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (5.9 KiB)

Copy/paste of commands to check before/after:

... Reproducer (15-second startup delay in tgt.service)

 FILE=/etc/systemd/system/tgt.service.d/start-delay.conf
 mkdir -p $(dirname $FILE)
 cat <<EOF > $FILE
 [Service]
 ExecStartPre=$(which sleep) 15
 EOF
 systemctl daemon-reload

Before:
---

... Restart both services at the same time:

 # date; systemctl restart cinder-volume.service tgt.service; date
 Mon Oct 16 21:57:12 UTC 2023
 Mon Oct 16 21:57:27 UTC 2023

... Notice that cinder-volume.service is Started _BEFORE_ tgt.service

 # journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3
 Oct 16 21:57:12 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.
 Oct 16 21:57:12 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
 Oct 16 21:57:27 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.

.. Log error:

 # grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log
 ...
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager [None req-ce03264a-6765-41de-8016-a6f27d2685e4 - - - - - -] Failed to re-export volume, setting to ERROR.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
 ...
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Exit code: 107
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stdout: ''
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n'
 2023-10-16 21:57:18.301 1658 ERROR cinder.volume.manager

... Volume error:

 # cinder list
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | error | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

... Undo error state:

 # systemctl restart cinder-volume.service
 # cinder reset-state --state in-use test-volume
 # cinder list

 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 17c4f736-058b-4cfc-9864-a64ab0995957 | in-use | test-volume | 1 | True | __DEFAULT__ | false | |
 +---------...

Read more...

Changed in cinder (Ubuntu Bionic):
status: New → Won't Fix
Changed in cinder (Ubuntu Focal):
status: New → Confirmed
Changed in cinder (Ubuntu Jammy):
status: New → Confirmed
Changed in cinder (Ubuntu Lunar):
status: New → Confirmed
no longer affects: cinder (Debian)
Changed in cinder (Ubuntu Bionic):
importance: Undecided → Medium
Changed in cinder (Ubuntu Focal):
importance: Undecided → Medium
Changed in cinder (Ubuntu Jammy):
importance: Undecided → Medium
Changed in cinder (Ubuntu Lunar):
importance: Undecided → Medium
Changed in cinder (Ubuntu Bionic):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in cinder (Ubuntu Focal):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in cinder (Ubuntu Jammy):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in cinder (Ubuntu Lunar):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Marking noble as Fix Committed because the fix is in cinder/2:23.0.0-0ubuntu2 [1],
amended by 2:23.0.0-0ubuntu3 [2] (fix FTBFS with Python 3.12) in noble-proposed.

[1] https://launchpad.net/ubuntu/+source/cinder/2:23.0.0-0ubuntu2/
[2] https://launchpad.net/ubuntu/+source/cinder/2:23.0.0-0ubuntu3/

Changed in cinder (Ubuntu Noble):
status: In Progress → Fix Committed
Changed in cinder (Ubuntu Mantic):
status: In Progress → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:23.0.0-0ubuntu3

---------------
cinder (2:23.0.0-0ubuntu3) noble; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * d/p/py312-tests-mock-assert.patch: Add prefix assert_
    to mock object assertions missing it (stricter in 3.12).

  [ Corey Bryant ]
  * d/control: Update min version of python3-taskflow to
    ensure it supports Python 3.12.

 -- Corey Bryant <email address hidden> Tue, 12 Dec 2023 17:04:19 -0500

Changed in cinder (Ubuntu Noble):
status: Fix Committed → Fix Released
Changed in cinder (Ubuntu Lunar):
status: Confirmed → Won't Fix
assignee: Mauricio Faria de Oliveira (mfo) → nobody
Changed in cinder (Ubuntu Bionic):
assignee: Mauricio Faria de Oliveira (mfo) → nobody
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

jammy debdiff

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

focal debdiff

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

mantic debdiff

Changed in cinder (Ubuntu Focal):
status: Confirmed → In Progress
Changed in cinder (Ubuntu Jammy):
status: Confirmed → In Progress
Changed in cinder (Ubuntu Mantic):
status: Confirmed → In Progress
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

Hi @mfo,

I've finished uploading the rest of the debdiffs and updated the status to in-progress, please let me know if there are any problems.

Thanks,
Heather Lemon

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hey Heather,

Thanks! The changes look aligned with the MR for stable releases which Corey approved [1].

I recall doing testing for the MR (e.g., see comment #3 section 'Patched:'),
but it's been several months ago, and not for all of the stable releases.

Can you confirm you have (re)tested for expected behavior with the patch?

Thanks!
Mauricio

[1] https://code.launchpad.net/~mfo/ubuntu/+source/cinder/+git/cinder/+merge/454428/comments/1218602

Changed in cinder (Ubuntu Mantic):
status: In Progress → Incomplete
Changed in cinder (Ubuntu Jammy):
status: In Progress → Incomplete
Changed in cinder (Ubuntu Focal):
status: In Progress → Incomplete
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

The only one I can confirm fixes the issue is Mantic.
I tried testing Noble series, but ran into issues executing Openstack commands with the steps above (500).

Thanks,
Heather Lemon

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi Heather,

Ok, thanks!

Let's hold on Jammy and Focal for your testing, before we upload those, per SRU policy
(template: "It is assumed that any SRU candidate patch is well-tested before upload") [1].

If you have any issues or need help, please just let me know!

[1] https://wiki.ubuntu.com/StableReleaseUpdates

Revision history for this message
James Page (james-page) wrote :

I've merged and pushed the updates for mantic and jammy into the packaging repo alongside the fix for bug 1988942

James Page (james-page)
Changed in cinder (Ubuntu Mantic):
status: Incomplete → New
Changed in cinder (Ubuntu Jammy):
status: Incomplete → New
Changed in cinder (Ubuntu Focal):
status: Incomplete → New
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Re: comment #13, I had re-tested Mantic, Jammy, and Focal with/without the patch (they all work/fail as expected), but forgot to update the bug.

Changed in cinder (Ubuntu Lunar):
importance: Medium → Undecided
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The packages built successfully in Test PPA ppa:mfo/cinder-lp1987663-lp1988942-lp1994521

All build-time unit tests passed in the Test PPA (as in the Ubuntu Archive).

Mantic:
---

mantic-updates:

 BUILDLOG='https://launchpad.net/ubuntu/+source/cinder/2:23.0.0-0ubuntu1.1/+build/27741243/+files/buildlog_ubuntu-mantic-amd64.cinder_2%3A23.0.0-0ubuntu1.1_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | grep -e '^Ran .* tests' -e '^OK (skipped='

 Ran 17293 tests in 343.725s
 OK (skipped=44)

mantic test ppa:

 BUILDLOG='https://launchpadlibrarian.net/725471761/buildlog_ubuntu-mantic-amd64.cinder_2%3A23.0.0-0ubuntu1.2_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | grep -e '^Ran .* tests' -e '^OK (skipped='

 Ran 17293 tests in 333.393s
 OK (skipped=44)

Jammy:
---

jammy-updates:

 BUILDLOG='https://launchpad.net/ubuntu/+source/cinder/2:20.3.1-0ubuntu1.1/+build/27741270/+files/buildlog_ubuntu-jammy-amd64.cinder_2%3A20.3.1-0ubuntu1.1_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | grep -e '^Ran .* tests' -e '^OK (skipped='

 Ran 16048 tests in 367.044s
 OK (skipped=48)

jammy test ppa:

 BUILDLOG='https://launchpad.net/~mfo/+archive/ubuntu/cinder-lp1987663-lp1988942-lp1994521/+build/28125465/+files/buildlog_ubuntu-jammy-amd64.cinder_2%3A20.3.1-0ubuntu1.2_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | grep -e '^Ran .* tests' -e '^OK (skipped='

 Ran 16054 tests in 343.184s
 OK (skipped=48)

Focal:
---

focal-updates:

 BUILDLOG='https://launchpadlibrarian.net/666807464/buildlog_ubuntu-focal-amd64.cinder_2%3A16.4.2-0ubuntu2.4_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | sed -n -e '/^Ran:/,/^Sum of/p'

 Ran: 12772 tests in 205.1574 sec.
  - Passed: 12760
  - Skipped: 12
  - Expected Fail: 0
  - Unexpected Success: 0
  - Failed: 0
 Sum of execute time for each test: 802.0120 sec.

focal test ppa:

 BUILDLOG='https://launchpad.net/~mfo/+archive/ubuntu/cinder-lp1987663-lp1988942-lp1994521/+build/28125762/+files/buildlog_ubuntu-focal-amd64.cinder_2%3A16.4.2-0ubuntu2.5_BUILDING.txt.gz'
 curl -sL "$BUILDLOG" -o - | gzip -dc | sed -n -e '/^Ran:/,/^Sum of/p'

 Ran: 12772 tests in 228.6830 sec.
  - Passed: 12760
  - Skipped: 12
  - Expected Fail: 0
  - Unexpected Success: 0
  - Failed: 0
 Sum of execute time for each test: 866.5233 sec.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote (last edit ):

Uploaded to Mantic, Jammy, Focal.

This upload has been coordinated with the Openstack Engineering team (~james-page) for bugs 1987663, 1988942, and 1994521.
Test PPA and build-time unit test summary in https://bugs.launchpad.net/ubuntu/+source/cinder/+bug/1987663/comments/16

Changed in cinder (Ubuntu Mantic):
status: New → In Progress
Changed in cinder (Ubuntu Jammy):
status: New → In Progress
Changed in cinder (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Cloud Archive:

Caracal is addressed with Noble
Bobcat is addressed with Mantic
* Antelope is _not_ addressed with Lunar (no longer supported), needs patching in UCA [Triaged]
Zed is no longer supported (2024/04)
Yoga is addressed with Jammy
Wallaby is no longer supported (2024/04)
Victoria is no longer supported (2022/04)
Ussuri is _not_ adddressed with Focal (Bionic is in ESM/Pro), needs patching in Pro [Triaged]

Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Mauricio, or anyone else affected,

Accepted cinder into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:23.0.0-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cinder (Ubuntu Mantic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-mantic
Changed in cinder (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Mauricio, or anyone else affected,

Accepted cinder into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:20.3.1-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cinder (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Mauricio, or anyone else affected,

Accepted cinder into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:16.4.2-0ubuntu2.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote (last edit ):
Download full text (12.9 KiB)

Verification done on {mantic,jammy,focal}-proposed.

The detailed verification steps as per comment #3
are attached. This is the summary with key steps.

With the packages from -proposed, when (re)starting {cinder-volume,tgt}.service,
the cinder-volume service is started _after_ the tgt service, and
the volumes remain with 'in-use' status (no longer 'error' status).

Reproducer (45-second startup delay in tgt.service)
---

 FILE=/etc/systemd/system/tgt.service.d/start-delay.conf
 mkdir -p $(dirname $FILE)
 cat <<EOF > $FILE
 [Service]
 ExecStartPre=$(which sleep) 45
 EOF
 systemctl daemon-reload

 # systemctl show tgt.service | grep ^ExecStartPre=
 ExecStartPre={ path=/usr/bin/sleep ; argv[]=/usr/bin/sleep 45 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }

 # date; systemctl restart cinder-volume.service tgt.service; date

Before / packages from -updates:
---

... Notice that cinder-volume.service is Started _BEFORE_ tgt.service
... And the 'tgtadm' error _IS_ seen in the logs.
... And volumes are listed with 'error' Status.

mantic:

 root@cinder-mantic:~# dpkg -l | grep cinder-volume
 ii cinder-volume 2:23.0.0-0ubuntu1.1 all Cinder storage service - Volume server

 root@cinder-mantic:~# date; systemctl restart cinder-volume.service tgt.service; date
 Tue Apr 30 15:54:43 UTC 2024
 Tue Apr 30 15:55:29 UTC 2024

 root@cinder-mantic:~# journalctl -b -u cinder-volume.service -u tgt.service | grep Start | tail -3
 Apr 30 15:54:43 cinder-mantic systemd[1]: Started cinder-volume.service - OpenStack Cinder Volume.
 Apr 30 15:54:44 cinder-mantic systemd[1]: Starting tgt.service - (i)SCSI target daemon...
 Apr 30 15:55:29 cinder-mantic systemd[1]: Started tgt.service - (i)SCSI target daemon.

 root@cinder-mantic:~# grep 'ERROR cinder.volume.manager' /var/log/cinder/cinder-volume.log | grep tgtadm
 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager (out, err) = cinder.privsep.targets.tgt.tgtadm_show()
 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Command: tgtadm --lld iscsi --op show --mode target
 2024-04-30 15:54:54.603 13692 ERROR cinder.volume.manager Stderr: 'tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected\n'

 root@cinder-mantic:~# cinder list
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | ID | Status | Name | Size | Consumes Quota | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+
 | 45b39ae0-2552-4703-a664-ae8de77a23a1 | error | test-volume | 1 | True | __DEFAULT__ | false | |
 +--------------------------------------+--------+-------------+------+----------------+-------------+----------+-------------+

jammy:

 root@cinder-jammy:~# dpkg -l | grep cinder-volume
 ii cinder-volume 2:20.3.1-0ubuntu1.1 all Cinder storage service - Volume server

 root@cinder-jammy:~# date; syste...

tags: added: verification-done verification-done-focal verification-done-jammy verification-done-mantic
removed: verification-needed verification-needed-focal verification-needed-jammy verification-needed-mantic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:23.0.0-0ubuntu1.2

---------------
cinder (2:23.0.0-0ubuntu1.2) mantic; urgency=medium

  [ Jorge Merlino ]
  * Increase size of volume image metadata values to 65535 bytes
    (LP: #1988942)

  [ Heather Lemon ]
  * Start cinder-volume.service after tgt.service started (LP: #1987663)
    - d/cinder-volume.service.conf: drop-in with 'After=' and 'Wants='
      ('Wants=' is not generated by pkgos-gen-systemd-unit currently).
    - d/cinder-volume.install: ship the systemd service drop-in file.

 -- Mauricio Faria de Oliveira <email address hidden> Fri, 19 Apr 2024 10:09:31 -0300

Changed in cinder (Ubuntu Mantic):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for cinder has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.