[Raspberry Pi/lunar] systemd-oomd fails with "ConditionControlGroupController=memory was not met"

Bug #2017209 reported by fprietog
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-raspi (Ubuntu)
Fix Released
Undecided
Unassigned
Lunar
Fix Released
Undecided
Unassigned
systemd (Ubuntu)
Invalid
Undecided
Unassigned
Lunar
Invalid
Undecided
Unassigned

Bug Description

PROBLEM SUMMARY:
----------------
Just upgraded to Ubuntu 23.04 (lunar) from Ubuntu 22.10 (kinetic). systemd-oomd fails with "ConditionControlGroupController=memory was not met" during boot.
If you manually started it after boot it starts but after a few minutes it fails with the same error.

These are the journalctl entries regarding this problem:

abr 20 23:12:48 fpgrpi systemd[1]: Listening on systemd-oomd.socket - Userspace Out-Of-Memory (OOM) Killer Socket.
abr 20 23:12:48 fpgrpi systemd[1]: systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer was skipped because of an unmet condition check (ConditionControlGroupController=memory).
--- Line repeat 20 times ---
abr 20 23:12:49 fpgrpi systemd[1]: systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer was skipped because of an unmet condition check (ConditionControlGroupController=memory).
abr 20 23:12:49 fpgrpi systemd[1]: systemd-oomd.socket: Trigger limit hit, refusing further activation.
abr 20 23:12:49 fpgrpi systemd[1]: systemd-oomd.socket: Failed with result 'trigger-limit-hit'.

This is the systemctl status for systemd-oomd.service and systemd-oomd.socket:

# systemctl status systemd-oomd.service
○ systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
     Loaded: loaded (/lib/systemd/system/systemd-oomd.service; enabled; preset: enabled)
     Active: inactive (dead)
TriggeredBy: × systemd-oomd.socket
  Condition: start condition failed at Thu 2023-04-20 23:21:07 CEST; 9h ago
             └─ ConditionControlGroupController=memory was not met
       Docs: man:systemd-oomd.service(8)

# systemctl status systemd-oomd.socket
× systemd-oomd.socket - Userspace Out-Of-Memory (OOM) Killer Socket
     Loaded: loaded (/lib/systemd/system/systemd-oomd.socket; enabled; preset: enabled)
     Active: failed (Result: trigger-limit-hit) since Thu 2023-04-20 23:21:07 CEST; 9h ago
   Duration: 2min 2.252s
   Triggers: ● systemd-oomd.service
       Docs: man:systemd-oomd.service(8)
     Listen: /run/systemd/oom/io.system.ManagedOOM (Stream)

ADITIONAL INFO:
---------------
Machine is an arm64 "Raspberry Pi 4 Model B Rev 1.4 8GB"

# uname -a
Linux fpgrpi 6.2.0-1004-raspi #5-Ubuntu SMP PREEMPT Mon Apr 3 11:15:14 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

# lsb_release -rd
No LSB modules are available.
Description: Ubuntu 23.04
Release: 23.04

# apt-cache policy systemd-oomd
systemd-oomd:
  Instalados: 252.5-2ubuntu3
  Candidato: 252.5-2ubuntu3
  Tabla de versión:
 *** 252.5-2ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
fprietog (fprietog) wrote :

I see that memory cgroup is not enabled:

# cat /proc/cgroups | grep y
#subsys_name hierarchy num_cgroups enabled
memory 0 121 0

So a workaround is to enable it adding "cgroup_enable=memory" to "/boot/firmware/cmdline.txt" kernel parameters file for Raspberry Pi boards.

As systemd-oomd worked in Ubuntu 22.10 (Kinetic) without the need of set this kernel parameter the only conclusion I've reached is that this cgroup has been disabled in a point between kernel 5.19 (kinetic) and kernel 6.2 (lunar) releases.

So I've searched the Raspberry Pi kernel commits related and found this one:

https://github.com/raspberrypi/linux/commit/28aec65bb1743c9bfa53b036999f9835c889704e

cgroup: Disable cgroup "memory" by default
Some Raspberry Pis have limited RAM and most users won't use the
cgroup memory support so it is disabled by default. Enable with:
    cgroup_enable=memory

At this point I don't know what team should fix this bug. I've updated it for linux-raspi package.

affects: systemd (Ubuntu) → linux-raspi (Ubuntu)
fprietog (fprietog)
tags: added: linux-raspi raspberry-pi
fprietog (fprietog)
summary: - systemd-oomd fails with "ConditionControlGroupController=memory was not
- met"
+ [Raspberry Pi/lunar] systemd-oomd fails with
+ "ConditionControlGroupController=memory was not met"
Juerg Haefliger (juergh)
tags: added: kern-6572
Revision history for this message
Juerg Haefliger (juergh) wrote :

Sigh. I missed to forward-port the revert of that commit from kinetic to lunar.

Juerg Haefliger (juergh)
Changed in systemd (Ubuntu):
status: New → Invalid
Changed in linux-raspi (Ubuntu Lunar):
status: New → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-raspi/6.2.0-1005.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-lunar-linux-raspi verification-needed-lunar
Revision history for this message
fprietog (fprietog) wrote (last edit ):

There may be a problem with the proposed repo.

The .deb files version in lunar-proposed repo is 6.2.0-1005.7 but inside the file http://ports.ubuntu.com/ubuntu-ports/dists/lunar-proposed/main/binary-arm64/Packages.gz there is also a non-existent version 6.2.0.1005.8 that causes conflict. Notice that even it's name is not correct because it got a third dot instead of a minus sign.

In fact, I don't know if its correct or no but the proposed repo doesn't work for me. Maybe my packages database is broken or so.

I'll install the packagaes .deb manually.

Revision history for this message
fprietog (fprietog) wrote :

Just installed and tested the proposed kernel: memory cgroup is enabled by default and problem is solved.

tags: added: verification-done-lunar
removed: verification-needed-lunar
Revision history for this message
Juerg Haefliger (juergh) wrote :

Thanks for the verification.

> there is also a non-existent version 6.2.0.1005.8 that causes conflict.

That's a valid meta package version and the meta packages should be available from lunar-proposed.

http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux-meta-raspi/

Revision history for this message
Juerg Haefliger (juergh) wrote :

ubuntu@rpi-4b-rev1d2-164b:~$ cat /etc/apt/preferences.d/proposed-updates
Package: *
Pin: release a=lunar-proposed
Pin-Priority: 500

ubuntu@rpi-4b-rev1d2-164b:~$ apt-cache policy linux-raspi
linux-raspi:
  Installed: 6.2.0.1004.6
  Candidate: 6.2.0.1005.8
  Version table:
     6.2.0.1005.8 500
        500 http://ports.ubuntu.com/ubuntu-ports lunar-proposed/main arm64 Packages
 *** 6.2.0.1004.6 500
        500 http://ports.ubuntu.com/ubuntu-ports lunar/main arm64 Packages
        100 /var/lib/dpkg/status

ubuntu@rpi-4b-rev1d2-164b:~$ apt list --upgradable
Listing... Done
linux-firmware/lunar-proposed 20230323.gitbcdcfbcf-0ubuntu1.1 all [upgradable from: 20230323.gitbcdcfbcf-0ubuntu1]
linux-headers-raspi/lunar-proposed 6.2.0.1005.8 arm64 [upgradable from: 6.2.0.1004.6]
linux-image-raspi/lunar-proposed 6.2.0.1005.8 arm64 [upgradable from: 6.2.0.1004.6]
linux-raspi/lunar-proposed 6.2.0.1005.8 arm64 [upgradable from: 6.2.0.1004.6]
mdadm/lunar-proposed 4.2-5ubuntu2 arm64 [upgradable from: 4.2-5ubuntu1]

ubuntu@rpi-4b-rev1d2-164b:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-headers-6.2.0-1005-raspi linux-image-6.2.0-1005-raspi linux-modules-6.2.0-1005-raspi linux-raspi-headers-6.2.0-1005
The following packages will be upgraded:
  linux-firmware linux-headers-raspi linux-image-raspi linux-raspi mdadm
5 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 338 MB of archives.
After this operation, 287 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Revision history for this message
fprietog (fprietog) wrote (last edit ):

The file /etc/apt/preferences.d/proposed-updates didn't exists in my system. I've created it with your content and proposed updates now are working.

But, AFAIK, I've never needed to pin proposed updates repo to make it work until now (?)

Edit: the only difference between kinetic and lunar proposed repos definition I've found is that lunar have these two additional options:

NotAutomatic: yes
ButAutomaticUpgrades: yes

It may explain why proposed updates in lunar are not offered by default, but my apt knowledge is very limited.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (74.5 KiB)

This bug was fixed in the package linux-raspi - 6.2.0-1006.8

---------------
linux-raspi (6.2.0-1006.8) lunar; urgency=medium

  * lunar/linux-raspi: 6.2.0-1006.8 -proposed tracker (LP: #2019843)

  * introduce do_lib_rust=true|false to enable/disable linux-lib-rust package
    (LP: #2021605)
    - [Packaging] introduce do_lib_rust and enable it only on generic amd64

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] resync getabis
    - debian/dkms-versions -- update from kernel-versions (main/2023.05.15)

  [ Ubuntu: 6.2.0-23.23 ]

  * lunar/linux: 6.2.0-23.23 -proposed tracker (LP: #2019845)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/2023.05.15)
  * Fix flicker display problem on some panels which support PSR2 (LP: #2002968)
    - drm/i915/psr: Add continuous full frame bit together with single
  * Kernel 6.1 bumped the disk consumption on default images by 15%
    (LP: #2015867)
    - [Packaging] introduce a separate linux-lib-rust package
  * Update I915 PSR calculation on Linux 6.2 (LP: #2018655)
    - drm/i915: Fix fast wake AUX sync len
    - drm/i915: Explain the magic numbers for AUX SYNC/precharge length
  * Computer with Intel Atom CPU will not boot with Kernel 6.2.0-20
    (LP: #2017444)
    - [Config]: Disable CONFIG_INTEL_ATOMISP
  * udev fails to make prctl() syscall with apparmor=0 (as used by maas by
    default) (LP: #2016908)
    - SAUCE: (no-up) Stacking v38: Fix prctl() syscall with apparmor=0
  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase
  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()
  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg
  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()
  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1
  * LSM stacking and AppArmor for 6.2: additional fixes (LP: #2017903)
    - SAUCE: (no-up) apparmor: fix policy_compat perms remap for file dfa
    - SAUCE: (no-up) apparmor: fix profile verification and enable it
    - SAUCE: (no-up) apparmor: fix: add missing failure check in
      compute_xmatch_perms
    - SAUCE: (no-up) apparmor: fix: kzalloc perms tables for shared dfas
  * Lunar update: v6.2.12 upstream stable release (LP: #2017219)
    - Revert "pinctrl: amd: Disable and mask interrupts on resume"
    - drm/amd/display: Pass the right info to drm_dp_remove_payload
    - drm/i915: Workaround ICL CSC_MODE sticky arming
    - ALSA: emu10k1: fix capture interrupt handler unlinking
    - ALSA: hda/sigmatel: add pin overrides for Intel DP45SG motherboard
    - ALSA: i2c/cs8427: fix iec958 mixer control deactivation
    - ALSA: hda: patch_realtek: add quirk for Asus N7601ZM
    - ALSA: hda/realtek: Add quirks for Lenovo Z13/Z16 Gen2
    - ALSA: firewire-tascam: add missing unwind goto in
      snd_tscm_stream_start_duplex()
    - ALSA: emu10k1: don't create old pass-through playback device on Aud...

Changed in linux-raspi (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (9.8 KiB)

This bug was fixed in the package linux-raspi - 6.5.0-1002.2

---------------
linux-raspi (6.5.0-1002.2) mantic; urgency=medium

  * mantic/linux-raspi: 6.5.0-1002.2 -proposed tracker (LP: #2033637)

  * Packaging resync (LP: #1786013)
    - [Packaging] update update.conf

  * Miscellaneous Ubuntu changes
    - [Config] raspi: Add notes to configs that differ from master
    - [Config] raspi: Set CONFIG_ARM_ERRATA_<foo>=n
    - [Config] raspi: Set PREEMPT_DYNAMIC=y
    - [Config] raspi: Set STAGING_MEDIA_DEPRECATED=n
    - [Config] raspi: Set EFI=n
    - [Config] raspi: Set POWER_RESET_GPIO_RESTART=n
    - [Config] raspi: Set HID_BPF=n
    - [Config] raspi: Set COMMON_CLK_FIXED_MMIO=n
    - [Config] raspi: Set PWM_STMPE=n
    - [Config] raspi: Set LEDS_SYSCON=n
    - [Packaging] raspi: Include bcm2835_smi module in linux-modules
    - SAUCE: (no-up) Disable FIQ split-transaction FSM in dwc_otg

  * Miscellaneous upstream changes
    - drivers: media: imx296: Updated imx296 driver for external trigger
    - char: broadcom: vc_mem: Fix preprocessor conditional
    - drivers: dwc_otg: Fix fallthrough warnings
    - vc04_services/vc-sm-cma: Switch one-bit bitfields to bool
    - media: i2c: ov2311: Fix uninitialized variable usage
    - drm/panel: Fix default values for Waveshare 7.9 inch DSI touchscreen (#5565)
    - dtoverlays: Add i2c bus overrides to edt-ft5406 overlay
    - dtoverlays: Fix README text for i2c-fan
    - drivers: irqchip: irq-bcm2835: Concurrency fix
    - defconfigs: Add TINYDRM_ILI9486 to defconfigs
    - dtoverlays: Add drm option to piscreen overlay
    - drm/ili9486: Resolve clash in spi_device_id names
    - input: ads7846: Add missing spi_device_id strings
    - staging: bcm2835-codec: Downgrade the level for a debug message
    - configs: Raise 8250 UART limit to 5 on BCM2711

  [ Ubuntu: 6.5.0-5.5 ]

  * mantic/linux: 6.5.0-5.5 -proposed tracker (LP: #2034546)
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/d2023.08.23)

  [ Ubuntu: 6.5.0-4.4 ]

  * mantic/linux: 6.5.0-4.4 -proposed tracker (LP: #2034042)
  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/d2023.08.23)

  [ Ubuntu: 6.5.0-3.3 ]

  * mantic/linux: 6.5.0-3.3 -proposed tracker (LP: #2033904)
  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/d2023.08.23)
  * [23.10] Please test secure-boot and lockdown on the early 6.5 kernel (s390x)
    (LP: #2026833)
    - [Packaging] re-enable signing for s390x
  * Miscellaneous upstream changes
    - module/decompress: use vmalloc() for zstd decompression workspace

  [ Ubuntu: 6.5.0-2.2 ]

  * mantic/linux: 6.5.0-2.2 -proposed tracker (LP: #2033240)
  * Soundwire support for Dell SKU0C87 devices (LP: #2029281)
    - SAUCE: ASoC: Intel: soc-acpi: add support for Dell SKU0C87 devices
  * Fix numerous AER related issues (LP: #2033025)
    - SAUCE: PCI/AER: Disable AER service during suspend, again
    - SAUCE: PCI/DPC: Disable DPC service during suspend, again
  * Support Realtek RTL8852CE WiFi 6E/BT Combo (LP: #2025672)
    - wifi: r...

Changed in linux-raspi (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.