machine get stuck at boot if specified 'console=ttyS* ' doesn't exist.

I've managed to reproduce in 4.4, 4.15, 5.4 (all from Ubuntu releases) and 5.7-rc4-next; also, I think bare-metal usually doesn't reproduce.

Seems like a live-lock situation, I'll attach some calltraces I gathered using sysrqs.
Cheers,

Guilherme

from dmesg (took from virsh dump)

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.15.0-101-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=ttyS4
......
[ 1.053683] Warning: unable to open an initial console.

I don't know yet if related, but I also observed the following:

syslog:May 21 17:25:27 ubuntu kernel: [ 0.000000] random: get_random_bytes called from start_kernel+0x99/0x4fd with crng_init=0
syslog:May 21 17:25:27 ubuntu kernel: [ 1.902484] random: lvm: uninitialized urandom read (4 bytes read)

Could the missing ttyS* have an effect on entropy ?

I observed the random kernel spew on my reproducer but I also want to add that it was also noticed on my VMware impacted contact who brought this up to my attention at the first place.

Related public bugs found here and there:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960355
https://github.com/systemd/systemd/issues/15783
https://github.com/systemd/systemd/issues/15656

This was greatly debugged by LP user WGH in https://bugs.launchpad.net/cloud-images/+bug/1573095/comments/46 - it's really a flaw on initramfs, I managed to workaround the issue with "quiet" parameter (system boots normally ,even with the wrong serial console).

Investigation continues... I guess this LP may end-up being a duplicate of 1573095.
Thanks,

Guilherme

I tested with debian buster and I wasn't able to reproduce (w/o quiet parameter)

I'm not yet too sure if it's a pure initramfs-tools or something introduced by a package hooks.

I first thought that plymouth could have beeb a potential candidate, but I disabled it and even purge it and problem persisted.

- Eric

Eric, I'll close this LP as dup of #1879987 - seems it's the same issue. I'll try also the Debian release top see what's different there...we can comment in the other bug about Debian status and what are the differences.

Thanks,

Guilherme

Sorry, dup of LP #1573095.

Although this is closed as dup, I thought would be good to clarify why Debian Buster doesn't reproduce (at first). What happens is that Debian includes the fsck on initrd by default, and in Ubuntu that doesn't happen - the reason behind it is that rootfs is marked as always-verify in the 6th field of fstab (fs_passno) in the Debian cloud image I got, whereas Ubuntu cloud image use the default value, 0.

Once I changed Debian to not include fsck on initrd, it reproduced the issue. And once I set Ubuntu rootfs to have value 1 on fstab's 6th field, it stopped to reproduce. When fsck is on initrd, we don't log_warn "fsck not present, so skipping $NAME file system", so we don't fail in _checkfs_once(), hence there's no infinite loop on checkfs().

Cheers,

Guilherme

Let me clarify why I'm un-marking this as duplicate of #1573095. This LP bug is indeed a duplicate of #1573095, but we have a lot of noise there, and potentially multiple different issues reported. The main one is the wrong ttySX causing the infinite loop in initramfs-tools due to error on write.

Hence, I'll use this LP to do the SRU for Ubuntu initramfs-tools. Also, worth notice that we have Debian bugs reported [0] and a Merge Request in Debian Salsa [1], both ignored by the Debian maintainer for 6 weeks. So, we'll go with Ubuntu fix for now, and hopefully Debian will merge that in the future.

Cheers,

Guilherme

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962545 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960355

[1] https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/30

Worth to notice that the debdiffs (except Xenial's) include a fix for LP #1879980 - we are doing a single SRU for 2 bugs.

Attaching a new Bionic debdiff, which now includes a fix for a third bug (LP #1820929).

After about 10 weeks, finally my patch was merged by the Debian maintainer: https://salsa.debian.org/kernel-team/initramfs-tools/-/commit/c3cbf355

Cheers,

Guilherme

An issue on initramfs-tools autopkgtest was found in Groovy and Focal (see LP #1893675) - it's non-related with the fixes proposed here, but we need to make autopkgtest happy or we cannot get the package released, so here goes the V2 of the initramfs-tools debdiffs.
Notice the SRU is mainly driven in this LP and/or LP #1879980.

This bug was fixed in the package initramfs-tools - 0.137ubuntu12

---------------
initramfs-tools (0.137ubuntu12) groovy; urgency=medium

  * d/tests: Add explicit call to partprobe on net test, specially in
    prep-image and run-image. (LP: #1893675)

initramfs-tools (0.137ubuntu11) groovy; urgency=medium

  * scripts/functions: Prevent printf error carry over if the wrong
    console is set. (LP: #1879987)
      The function _log_msg() is "void" typed, returning whatever its
      last command returns. This function is the basic building block
      for all error/warning messages in initramfs-tools. If a bad console
      is provided to kernel on command-line, printf returns error, and so
      this error is carried over in _log_msg(). Happens that checkfs()
      function has a loop that runs forever in this scenario (*if* fsck
      is not present in initramfs and "quiet" is not passed in the
      command-line). If that happens, boot is stuck and cannot progress.
      The simple fix hereby merged is to return zero on _log_msg().

  * scripts/local: Re-execute cryptroot local-block script. (LP: #1879980)
      Currently, if an encrypted rootfs is configured on top of a MD RAID1
      array and such array gets degraded (like a member is removed/failed),
      initramfs-tools cannot mount the rootfs and the boot fails. We fix
      that issue here by allowing cryptroot script to re-run on local-block
      stage, given that mdadm is able to activate a degraded array in that
      point. There is a cryptsetup counter-part for this fix, but alone the
      initramfs-tools portion is innocuous.

 -- <email address hidden> (Guilherme G. Piccoli) Mon, 31 Aug 2020 18:04:00 -0300

[sts-sponsor]

Sponsored in Focal/Bionic.

Thanks for your contribution.

Hello Eric, or anyone else affected,

Accepted initramfs-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.136ubuntu6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted initramfs-tools (0.136ubuntu6.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.136ubuntu6.3 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I've tested the Focal-proposed initramfs-tools (version 0.136ubuntu6.3 ) according to the "Test Case" section in the description, and it's working as expected. Also, the armhf autopkgtest failure is no longer an issue (test was re-executed with success), so marking as verified!
Cheers,

Guilherme

This bug was fixed in the package initramfs-tools - 0.136ubuntu6.3

---------------
initramfs-tools (0.136ubuntu6.3) focal; urgency=medium

  * scripts/functions: Prevent printf error carry over if the wrong
    console is set. (LP: #1879987)
      The function _log_msg() is "void" typed, returning whatever its
      last command returns. This function is the basic building block
      for all error/warning messages in initramfs-tools. If a bad console
      is provided to kernel on command-line, printf returns error, and so
      this error is carried over in _log_msg(). Happens that checkfs()
      function has a loop that runs forever in this scenario (*if* fsck
      is not present in initramfs and "quiet" is not passed in the
      command-line). If that happens, boot is stuck and cannot progress.
      The simple fix hereby merged is to return zero on _log_msg().

  * scripts/local: Re-execute cryptroot local-block script. (LP: #1879980)
      Currently, if an encrypted rootfs is configured on top of a MD RAID1
      array and such array gets degraded (like a member is removed/failed),
      initramfs-tools cannot mount the rootfs and the boot fails. We fix
      that issue here by allowing cryptroot script to re-run on local-block
      stage, given that mdadm is able to activate a degraded array in that
      point. There is a cryptsetup counter-part for this fix, but alone the
      initramfs-tools portion is innocuous.

  * d/tests: Add explicit call to partprobe on net test, specially in
    prep-image and run-image. (LP: #1893675)

 -- <email address hidden> (Guilherme G. Piccoli) Mon, 31 Aug 2020 13:43:29 -0300

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Uploaded initramfs-tools to Bionic and Xenial.
Attaching the updated debdiff for reference.
(Rebased on top of the more recent -updates.)

Hello Eric, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Eric, or anyone else affected,

Accepted initramfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.122ubuntu8.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verification done for bionic:

bionic-updates:

- boots fine w/ console=ttyS0, can ssh in 10 seconds.
- boot fails w/ console=ttyS1, cannot SSH, qemu on 100% CPU (initramfs-tools looping)

bionic-proposed:

- boots fine w/ console=ttyS0, can ssh in 10 seconds. (no regression)
- boots fine w/ console=ttyS1, can ssh in 10 seconds, qemu on low CPU% (issue fixed)

details:
-------

$ lsb_release -cs
bionic

bionic-updates:
---

$ dpkg -s initramfs-tools | grep -i version:
Version: 0.130ubuntu3.10

console=ttyS0:

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.15.0-99-generic root=UUID=f513680e-7260-45e3-b4aa-89e4a210be77 ro console=ttyS0

console=ttyS1:

 (cannot SSH into system. CPU% always high for QEMU.)

 $ top -b -d1 | grep -e CPU -e qemu
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142522 libvirt+ 20 0 3745600 485332 23116 S 100.0 3.0 1:30.08 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142522 libvirt+ 20 0 3745600 485332 23116 S 99.0 3.0 1:31.08 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142522 libvirt+ 20 0 3745600 485332 23116 S 98.0 3.0 1:32.07 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142522 libvirt+ 20 0 3745600 485332 23116 S 99.0 3.0 1:33.07 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142522 libvirt+ 20 0 3745600 485332 23116 S 99.0 3.0 1:34.06 qemu-system-x86
 ^C

bionic-proposed:
---

$ dpkg -s initramfs-tools | grep -i version:
Version: 0.130ubuntu3.11

console=ttyS0:

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.15.0-99-generic root=UUID=f513680e-7260-45e3-b4aa-89e4a210be77 ro console=ttyS0

console=ttyS1:

 (can SSH into system. CPU% low for QEMU)

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.15.0-99-generic root=UUID=f513680e-7260-45e3-b4aa-89e4a210be77 ro console=ttyS1

     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142731 libvirt+ 20 0 3656512 602160 22816 S 83.2 3.7 0:38.73 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142731 libvirt+ 20 0 3656512 602160 22816 S 29.7 3.7 0:39.03 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142731 libvirt+ 20 0 3656512 602160 22816 S 1.0 3.7 0:39.04 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142731 libvirt+ 20 0 3656512 602160 22816 S 2.0 3.7 0:39.06 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  142731 libvirt+ 20 0 3656512 602160 22816 S 2.9 3.7 0:39.09 qemu-system-x86

Verification done for xenial:

xenial-updates:

- boots fine w/ console=ttyS0, can ssh in 10 seconds.
- boot fails w/ console=ttyS1, cannot SSH, qemu on 100% CPU (initramfs-tools looping)

xenial-proposed:

- boots fine w/ console=ttyS0, can ssh in 10 seconds. (no regression)
- boots fine w/ console=ttyS1, can ssh in 10 seconds, qemu on low CPU% (issue fixed)

details:
-------

$ lsb_release -cs
xenial

xenial-updates:
---

$ dpkg -s initramfs-tools | grep -i version:
Version: 0.122ubuntu8.16

console=ttyS0:

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.4.0-184-generic root=UUID=c763ea2f-6085-4419-8895-3cefa32d1477 ro console=ttyS0

console=ttyS1:

 (cannot SSH into system. CPU% always high for QEMU.)

 $ top -b -d1 | grep -e CPU -e qemu
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143433 libvirt+ 20 0 3886900 366840 22900 S 106.7 2.3 0:32.52 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143433 libvirt+ 20 0 3886900 366840 22900 S 101.0 2.3 0:33.54 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143433 libvirt+ 20 0 3886900 366840 22900 S 100.0 2.3 0:34.55 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143433 libvirt+ 20 0 3886900 366840 22900 S 100.0 2.3 0:35.56 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143433 libvirt+ 20 0 3886900 366840 22900 S 101.0 2.3 0:36.57 qemu-system-x86
 ^C

xenial-proposed:
---

$ dpkg -s initramfs-tools | grep -i version:
Version: 0.122ubuntu8.17

console=ttyS0:

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.4.0-184-generic root=UUID=c763ea2f-6085-4419-8895-3cefa32d1477 ro console=ttyS0

console=ttyS1:

 (can SSH into system. CPU% low for QEMU)

 $ cat /proc/cmdline
 BOOT_IMAGE=/boot/vmlinuz-4.4.0-184-generic root=UUID=c763ea2f-6085-4419-8895-3cefa32d1477 ro console=ttyS1

  143646 libvirt+ 20 0 4596800 692248 22972 R 158.4 4.3 0:53.95 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143646 libvirt+ 20 0 4596800 692248 22972 S 104.0 4.3 0:55.00 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143646 libvirt+ 20 0 4596800 692248 22972 S 2.0 4.3 0:55.02 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143646 libvirt+ 20 0 4596800 692248 22972 S 2.0 4.3 0:55.04 qemu-system-x86
     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  143646 libvirt+ 20 0 4596800 692248 22972 S 37.3 4.3 0:55.42 qemu-system-x86

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.11

---------------
initramfs-tools (0.130ubuntu3.11) bionic; urgency=medium

  [ Guilherme G. Piccoli ]
  * scripts/functions: Prevent printf error carry over if the wrong
    console is set. (LP: #1879987)
      The function _log_msg() is "void" typed, returning whatever its
      last command returns. This function is the basic building block
      for all error/warning messages in initramfs-tools. If a bad console
      is provided to kernel on command-line, printf returns error, and so
      this error is carried over in _log_msg(). Happens that checkfs()
      function has a loop that runs forever in this scenario (*if* fsck
      is not present in initramfs and "quiet" is not passed in the
      command-line). If that happens, boot is stuck and cannot progress.
      The simple fix hereby merged is to return zero on _log_msg().

  * scripts/local: Re-execute cryptroot local-block script. (LP: #1879980)
      Currently, if an encrypted rootfs is configured on top of a MD RAID1
      array and such array gets degraded (like a member is removed/failed),
      initramfs-tools cannot mount the rootfs and the boot fails. We fix
      that issue here by allowing cryptroot script to re-run on local-block
      stage, given that mdadm is able to activate a degraded array in that
      point. There is a cryptsetup counter-part for this fix, but alone the
      initramfs-tools portion is innocuous.

  [ Jay Vosburgh ]
  * scripts/functions: Change netplan render for net_failover master
    devices. (LP: #1820929)
      Modify the _render_netplan function to check for network interfaces
      that are net_failover master devices. When found, such devices are
      matched only by name, not by MAC address, as the MAC is not a unique
      identifier for the net_failover case. In the net_failover architecture,
      the MAC address is used to manage the membership of the net_failover
      interface set, thus multiple interfaces will be assigned the same MAC
      address.

 -- <email address hidden> (Guilherme G. Piccoli) Wed, 12 Aug 2020 17:12:11 -0300

This bug was fixed in the package initramfs-tools - 0.122ubuntu8.17

---------------
initramfs-tools (0.122ubuntu8.17) xenial; urgency=medium

  * scripts/functions: Prevent printf error carry over if the wrong
    console is set. (LP: #1879987)
      The function _log_msg() is "void" typed, returning whatever its
      last command returns. This function is the basic building block
      for all error/warning messages in initramfs-tools. If a bad console
      is provided to kernel on command-line, printf returns error, and so
      this error is carried over in _log_msg(). Happens that checkfs()
      function has a loop that runs forever in this scenario (*if* fsck
      is not present in initramfs and "quiet" is not passed in the
      command-line). If that happens, boot is stuck and cannot progress.
      The simple fix hereby merged is to return zero on _log_msg().

 -- <email address hidden> (Guilherme G. Piccoli) Thu, 23 Jul 2020 13:20:14 -0300