cloud-init-23.4 cannot read "- Azure" datasource_list format

Bug #2051147 reported by Alberto Contreras
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Undecided
Brett Holman
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Mantic
Fix Released
Undecided
Unassigned

Bug Description

[ Impact ]

When an invalid configuration in /etc/cloud/cloud.cfg.d/ includes an invalid (empty) datasource_list, ds-identify will identify the wrong datasource.

While the datasource_list is invalid, it does represent a regression because previous ds-identify behavior was to silently ignore the invalid configuration and now it needlessly prevents cloud-init from running rather than identifying the correct datasource like it used to.

[ Test Plan ]

1. Create a VM on Azure, change the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg to:

datasource:
  Azure:
    apply_network_config: false
datasource_list:
 \- Azure
2. Clean cloud-init and reboot VM

 $sudo cloud-init clean
 $sudo reboot

Expected results
cloud-init service can start successfully.

Bug results
cloud-init service fails to start.

[ Where problems could occur ]

This is a revert of a commit added this cycle, so it's possible we could bring back any other undiscovered bugs we didn't know about in the old code.

[ Other Info ]

Upstream bug: https://github.com/canonical/cloud-init/issues/4794

This issue was introduced as a fix for https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2030729 in https://github.com/canonical/cloud-init/pull/4327

PR reverting and fixing the issue: https://github.com/canonical/cloud-init/pull/4797

[ Original Description ]

In Image Builder images, the datasource_list in /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg is:

datasource_list:
 \- Azure

which cannot be parsed by cloud-init.

2024-01-18 10:27:12,433 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.rhel.Distro'>
2024-01-18 10:27:12,441 - _{_}init{_}_.py[DEBUG]: Looking for data source in: [\\{'datasource_list': None}, 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM', 'NETWORK']
2024-01-18 10:27:12,441 - util.py[WARNING]: failed stage init
2024-01-18 10:27:12,441 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 781, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 394, in main_init
    init.fetch(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 493, in fetch
    return self._get_data_source(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 367, in _get_data_source
    self.reporter,
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1001, in find_source
    ds_list = list_sources(cfg_list, ds_deps, pkg_list)
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1047, in list_sources
    ds_name = importer.match_case_insensitive_module_name(ds)
  File "/usr/lib/python3.6/site-packages/cloudinit/importer.py", line 40, in match_case_insensitive_module_name
    if "nocloud-net" == mod_name.lower():
AttributeError: 'dict' object has no attribute 'lower'

## Steps to reproduce the problem
1. Create a VM on Azure, change the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg to:

datasource:
  Azure:
    apply_network_config: false
datasource_list:
 \- Azure
2. Clean cloud-init and reboot VM

 $sudo cloud-init clean
 $sudo reboot

Expected results
cloud-init service can start successfully.

Actual results
cloud-init service fails to start(logs are in the description)
If change the

datasource_list:
 \- Azure
to:

datasource_list: [ Azure ]
Then cloud-init clean && reboot, the cloud-init service and start successfully.

## Environment details
- Cloud-init version: cloud-init-23.4
- Operating System Distribution: RHEL
- Cloud provider, platform or installer type: Azure

## cloud-init logs
1. $ sudo cat /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg
 datasource_list:
 \- Azure

2. cloud-init.log

2024-01-18 10:27:12,433 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.rhel.Distro'>
2024-01-18 10:27:12,441 - _{_}init{_}_.py[DEBUG]: Looking for data source in: [\\{'datasource_list': None}, 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM', 'NETWORK']
2024-01-18 10:27:12,441 - util.py[WARNING]: failed stage init
2024-01-18 10:27:12,441 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 781, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 394, in main_init
    init.fetch(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 493, in fetch
    return self._get_data_source(existing=existing)
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 367, in _get_data_source
    self.reporter,
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1001, in find_source
    ds_list = list_sources(cfg_list, ds_deps, pkg_list)
  File "/usr/lib/python3.6/site-packages/cloudinit/sources/_{_}init{_}_.py", line 1047, in list_sources
    ds_name = importer.match_case_insensitive_module_name(ds)
  File "/usr/lib/python3.6/site-packages/cloudinit/importer.py", line 40, in match_case_insensitive_module_name
    if "nocloud-net" == mod_name.lower():
AttributeError: 'dict' object has no attribute 'lower'

## Additional info:
1. No such issue in cloud-init-23.1.1, just met this issue after rebase to 23.4, so could you please help to address which commit did this change between 23.1.1 and 23.4?
2. RHEL was using ds-identify before 23.1.1
3. I am afraid users who were using the format "- Azure" in config file will meet this issue after upgrade, so could you please help to check if there is a way to be compatible with this format? Thanks

---

Upstream bug: https://github.com/canonical/cloud-init/issues/4794

This issue was introduced as a fix for https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2030729 in https://github.com/canonical/cloud-init/pull/4327

PR reverting and fixing the issue: https://github.com/canonical/cloud-init/pull/4797

Changed in cloud-init (Ubuntu):
status: New → In Progress
Changed in cloud-init (Ubuntu):
assignee: nobody → Brett Holman (holmanb)
James Falcon (falcojr)
description: updated
description: updated
Revision history for this message
Chad Smith (chad.smith) wrote :
Changed in cloud-init (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote :

published to Ubuntu Noble: cloud-init 24.1~4gd9677655-0ubuntu1 (Accepted)

Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
James Falcon (falcojr)
tags: added: regression-proposed
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Please test proposed package

Hello Alberto, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Mantic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-mantic
Changed in cloud-init (Ubuntu Jammy):
status: New → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Alberto, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Alberto, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.2-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Alberto, or anyone else affected,

Accepted cloud-init into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Alberto, or anyone else affected,

Accepted cloud-init into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hello Alberto, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/23.4.3-0ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Alberto Contreras (aciba) wrote :

I have verified that the bug is not reproducible with 23.4.3-0ubuntu0~23.10.1, 23.4.3-0ubuntu0~22.04.1 and 23.4.3-0ubuntu0~20.04.1. Attached test script and logs.

James Falcon (falcojr)
tags: added: verification-done verification-done-focal verification-done-jammy verification-done-mantic
removed: verification-needed verification-needed-focal verification-needed-jammy verification-needed-mantic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~23.10.1

---------------
cloud-init (23.4.3-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~23.10.1) mantic; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~23.10.2) mantic; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~23.10.1) mantic; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * d/p/do-not-block-user-login.patch:
    - revert redacted patch content introduced in 23.4-0
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches and dropped cherry-picks entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~23.10.1) mantic; urgency=medium

  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * refresh patches:
    - d/p/do-not-block-user-login.patch
  * drop the following cherry-picks now included:
    - cpick-0d9f149a-Pytestify-apt-config-test-modules-4424
    - cpick-5023e9f9-Refactor-test_apt_source_v1.py-to-use-pytest-4427
    - cpick-e9cdd7e3-Install-gnupg-if-gpg-not-found-4431
    - cpick-015543d3-apt-install-software-properties-common-when-absent-but
    - cpick-2ab1f340-fix-cc_apt_configure-avoid-unneeded-call-to-apt-install
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 16:00:04 -0600

Changed in cloud-init (Ubuntu Mantic):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~22.04.1

---------------
cloud-init (23.4.3-0ubuntu0~22.04.1) jammy; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~22.04.1) jammy; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~22.04.2) jammy; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/p/series: bring back retain-old-groups.patch.
    This patch was inadvertently dropped in 5d4a3cf.
  * refresh patches:
    - d/p/do-not-block-user-login.patch
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 15:59:14 -0600

Changed in cloud-init (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 23.4.3-0ubuntu0~20.04.1

---------------
cloud-init (23.4.3-0ubuntu0~20.04.1) focal; urgency=medium

  * Upstream snapshot based on 23.4.3. (LP: #2046483).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.3/ChangeLog

cloud-init (23.4.2-0ubuntu0~20.04.1) focal; urgency=medium

  * Upstream snapshot based on 23.4.2. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2051147)

cloud-init (23.4.1-0ubuntu1~20.04.2) focal; urgency=medium

  * d/p/status-retain-recoverable-error-exit-code.patch:
    Retain exit code in cloud-init status for recoverable errors.
    (LP: #2048522).

cloud-init (23.4.1-0ubuntu1~20.04.1) focal; urgency=medium

  * d/p/retain-apt-pre-deb822.patch:
    - Disable apt source list generation with DEB822 style
  * refresh patches:
    - d/p/status-do-not-remove-duplicated-data.patch
  * d/changelog: amend 23.4-0 refresh patches entry
  * Upstream snapshot based on 23.4.1. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4.1/ChangeLog

cloud-init (23.4-0ubuntu1~20.04.1) focal; urgency=medium

  * d/p/status-do-not-remove-duplicated-data.patch:
    - Revert behavior downstream, leave duplicate data
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * refresh patches:
    - d/p/do-not-block-user-login.patch
    - d/p/netplan99-cannot-use-default.patch
  * Upstream snapshot based on 23.4. (LP: #2045582).
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/23.4/ChangeLog

 -- James Falcon <email address hidden> Fri, 02 Feb 2024 16:00:45 -0600

Changed in cloud-init (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.