VMs hardening with the noexec option in /tmp and /var/tmp which is causing issues to get an IP with cloud-init , reason why the VM takes like 25 min to start

Bug #1962343 reported by Mauricio Saenz
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
Alberto Contreras

Bug Description

Hardening Azure VM - Ubuntu 18.04 with the noexec option in /tmp and /var/tmp is causing issues with the dhclient to get an IP with cloud-init , reason why the VM takes like 25 min to start

Hardening:

root@ubu1804repro:~# cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=5b1ab5d4-8b76-46c5-928f-8db42fbe3af6 / ext4 defaults,discard 0 1
UUID=91B6-4BB7 /boot/efi vfat umask=0077 0 1
UUID="fadc7d49-1a88-4eed-8964-94b78ee7dfa6" /tmp ext4 rw,nodev,nosuid,noexec,discard 0 0
/tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

Error:

[ OK ] Reached target System Time Synchronized.
[ OK ] Started AppArmor initialization.
         Starting Load AppArmor profiles managed internally by snapd...
         Starting Initial cloud-init job (pre-networking)...
[ 8.062136] sh[795]: + [ -e /var/lib/cloud/instance/obj.pkl ]
[ OK ] [ 8.097225] sh[795]: + echo cleaning persistent cloud-init object
Started Load AppArmor profiles managed internally by snapd.
[ 8.100207] sh[795]: cleaning persistent cloud-init object
[ 8.106214] sh[795]: + rm /var/lib/cloud/instance/obj.pkl
[ 8.112706] sh[795]: + exit 0
[ 14.435302] cloud-init[813]: Cloud-init v. 21.4-0ubuntu1~18.04.1 running 'init-local' at Fri, 25 Feb 2022 17:18:50 +0000. Up 8.71 seconds.
[ 14.445225] cloud-init[813]: 2022-02-25 17:18:56,105 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid
[ 14.453129] cloud-init[813]: 2022-02-25 17:18:56,107 - azure.py[WARNING]: exception while getting metadata:
[ 14.460876] cloud-init[813]: 2022-02-25 17:18:56,109 - azure.py[ERROR]: Could not crawl Azure metadata:
[ 19.626878] cloud-init[813]: 2022-02-25 17:19:01,297 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid
[ 19.664700] cloud-init[813]: 2022-02-25 17:19:01,333 - azure.py[ERROR]: Failed to read /var/lib/dhcp/dhclient.eth0.leases: [Errno 2] No such file or directory: '/var/lib/dhcp/dhclient.eth0.leases'
[ 19.674221] cloud-init[813]: 2022-02-25 17:19:01,333 - azure.py[WARNING]: No lease found; using default endpoint: a8:3f:81:10

Cloud-Init Version :

root@ubu1804repro:~# cloud-init --version
/usr/bin/cloud-init 21.4-0ubuntu1~18.04.1
root@ubu1804repro:~#

OS version:

root@ubu1804repro:~# uname -a
Linux ubu1804repro 5.4.0-1069-azure #72~18.04.1-Ubuntu SMP Mon Feb 7 11:12:24 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@ubu1804repro:~#

root@ubu1804repro:~# cat /etc/*rele*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@ubu1804repro:~#

Workaround : Remove the noexec option from /tmp and /tmp/var entries in /etc/fstab.

Revision history for this message
Mauricio Saenz (msaenzbo2022) wrote :
Revision history for this message
Chad Smith (chad.smith) wrote :

Thank you for filing this bug and improving Ubuntu and cloud-init.
I confirm this looks to be a problem from your attached cloud-init.log

Looks like cloud-init should be a bit more resilient in determining the appropriate temporary directory from which to execute a preliminary dhclient call prior to network being setup on the system.

the module function where we'd likely need to address this is cloudinit.temp_files._tempfile_dir_arg.

A preflight check of util.mounts() can inform cloud-init if it's temporary directory choice would lead to noexec type errors.

Changed in cloud-init:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Chad Smith (chad.smith) wrote :

This type of issue with /var/tmp partitions setup as noexec will also likely affect the following additional config modules:
- cc_puppet.py
- cc_growpart.py
- cc_chef.py

Revision history for this message
Chad Smith (chad.smith) wrote :

Marking this is high because it affects a number of components.

Changed in cloud-init:
importance: Medium → High
Revision history for this message
Chad Smith (chad.smith) wrote :

Looking into this more, I'm thinking cloudinit.temp_utils should probably check util.get_mount_info("/var/tmp/cloud-init") to ensure noexec is not set on the mounted filesystem when needs_exe=True. It may want to fallback to use os.join.path(Distro.usr_lib_exec, "cloud-init/clouddir) in that case as /usr/lib/cloud-init or /usr/libexec/cloud-init will be mounted with exec perms.

Revision history for this message
Chris Patterson (cjp256) wrote :

Another customer reported this issue. Is it on the agenda to get fixed? I can help, if needed.

Changed in cloud-init:
assignee: nobody → Alberto Contreras (aciba)
Revision history for this message
Alberto Contreras (aciba) wrote :
Changed in cloud-init:
status: Triaged → Fix Committed
Revision history for this message
Mauricio Saenz (msaenzbo2022) wrote :

@Alberto Contreras

Thanks for the commit.
On which version the cloud-init will contains this fix?
This fix will be released as well on other Linux Distributions?

Revision history for this message
Alberto Contreras (aciba) wrote :

My pleasure. This patch will be included in cloud-init 22.4, which is expected to be released the 1st of November: https://discourse.ubuntu.com/t/cloud-init-2022-release-schedule/25413 .

Other distros will release it as well, but I am not sure about their schedule.

Revision history for this message
James Falcon (falcojr) wrote : Fixed in cloud-init version 22.4.

This bug is believed to be fixed in cloud-init in version 22.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.