cloud-init is calling localedef (and getting killed)

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2038945

Would be really good to understand when this started happening, since this cycle we for instance switched to use ubuntu-image 3.0 for building the image - so I'm of course thinking if maybe the way we prepare the cloud-init configuration could be wrong or something. Or is it cloud-init itself changing behavior?

(by the image I mean the pi images)

Two pieces to this. The raspi images are generated with ubuntu-image rather than the legacy livecd-rootfs code; but the legacy livecd-rootfs code includes this fixup in live-build/auto/build:

        case $PROJECT:${SUBPROJECT:-} in
                ubuntu-server:*|ubuntu-cpc:*|ubuntu:desktop-preinstalled|ubuntu-
wsl:*)
                        # Set locale to C.UTF-8 by default. We should
                        # probably do this for all images early in the
                        # 18.10 cycle but for now just do it for
                        # server and cpc products.
                        echo "LANG=C.UTF-8" > chroot/etc/default/locale
                        ;;
        esac

Instead, the Pi image has:

$ cat /mnt/etc/default/locale
# File generated by update-locale
$

So there is no default locale set.

This is wrong, because when there is no locale set in the environment, glibc defaults to C instead of C.UTF-8 and we do not want to ever anywhere be using ASCII as our default character set instead of UTF-8. (I *think* this is still the meaning of 'POSIX' as a locale in glibc. If 'POSIX' does now map to C.UTF-8, then it's not actually wrong but as we don't know that's the case and haven't de-configured the other images, we should for the time being assume glibc is wrong and make sure our images enforce the intended default.)

It is also wrong that, when no locale is configured in the target, cloud-init decides the default should be en_US.UTF-8. cloud-init should ALSO default to C.UTF-8, which is statically available in our glibc package, instead of generating an en_US.UTF-8 locale.

jiggling severities - ubuntu-image can be fixed for release, cloud-init is not something we want to change Tuesday of release week.

A few additional notes on this:

In answer to Łukasz's question of when this started, presumably at the point we started building the pi images with ubuntu-image instead of livecd-rootfs. However, this specific failure (OOM killed) didn't occur in the beta so I suspect we were "sailing close to the wind" with memory during localedef and just about got away with it. That brings me onto...

I've implemented a workaround for this; in the OOM output you can see the next biggest process is multipathd which is something we simply don't care about on Pi. I've added "multipath=off" to the classic pi-gadget to disable running multipathd on boot (without actually removing the package or changing our seeds) and this frees up enough that the initial boot succeeds on both Pi Zero 2W and 3A+. So, we're still sailing close to the wind with localedef and ideally wouldn't be running it at all, but if the images are re-spun with that change they will at least survive first boot ... until someone runs localedef with other things loaded :)

The cloud-init part is being addressed in the following upstream PR: https://github.com/canonical/cloud-init/pull/4503

This bug was fixed in the package cloud-init - 23.4~3g0cb0b80f-0ubuntu1

---------------
cloud-init (23.4~3g0cb0b80f-0ubuntu1) noble; urgency=medium

  * d/p/do-not-block-user-login.patch:
    - Revert behavior, allow user login after cloud-init stage (LP: #2039505)
  * d/control: add python3-apt as Recommends to read APT config from apt_pkg
  * drop the following cherry-picks now included:
    + d/p/cpick-0d9f149a-Pytestify-apt-config-test-modules-4424
    + d/p/cpick-5023e9f9-Refactor-test_apt_source_v1.py-to-use-pytest-4427
    + d/p/cpick-e9cdd7e3-Install-gnupg-if-gpg-not-found-4431
    + d/p/cpick-015543d3-apt-install-software-properties-common-when-absent-but
  * Upstream snapshot based on upstream/main at 0cb0b80f.
    - Bugs fixed in this snapshot: (LP: #2034273, #2030729)
      (LP: #2038945, #2039453)

 -- Chad Smith <email address hidden> Wed, 08 Nov 2023 22:24:09 -0700