libsmartcols: Revert back to previous behaviour of non-shell parsable column output (lsblk -P)

Bug #1961542 reported by Matthew Ruffell
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
util-linux (Ubuntu)
Fix Released
High
Matthew Ruffell
Jammy
Fix Released
High
Matthew Ruffell

Bug Description

[Impact]

util-linux 2.37 in Jammy has introduced some new behaviour for lsblk and similar tools which depend on libsmartcols. This switched the -P / --pairs parameter from printing column names as normal, to changing the names to shell compatible names instead.

e.g. lsblk -P now outputs LOG_SEC instead of LOG-SEC.

The change broke some core tooling which rely on on the output of lsblk -P, most notably, curtin and MAAS, but I am sure there will be more applications affected.

Affected MAAS users will see the following traceback when attempting to deploy 22.04:

Traceback (most recent call last):
File "/curtin/curtin/block/__init__.py", line 785, in get_blockdev_sector_size
    logical = info[parent]['LOG-SEC']
KeyError: 'LOG-SEC'
'LOG-SEC'
curtin: Installation failed with exception: Unexpected error while running

This is documented in MAAS bug 1956613.

MAAS decided to fix it by changing from -P to -J, in the following commit:
https://git.launchpad.net/maas/commit/?id=e2c01963430e6837198a54bc1eadf3efc9fdd9a2

Curtin now checks for MAJ_MIN, and changes it back to MAJ:MIN in:
https://github.com/canonical/curtin/commit/ce811db127fe1ce46498b83615f8faed8c7dfeb6

The issue is that these commits are not tagged to any MAAS release, and users would be forced to upgrade MAAS to the latest stable release when available if they want to deploy 22.04.

There are many users out there that don't want to upgrade MAAS, so returning to the previous column output is the most desirable solution.

[Testcase]

On a Jammy install, simply run lsblk with either -P or --pairs:

$ sudo lsblk -P
...
NAME="sda" MAJ_MIN="8:0" RM="0" SIZE="465.8G" RO="0" TYPE="disk" MOUNTPOINTS=""
...

Affected installs will see MAJ_MIN.

There is a test package available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf326660-test

If you install it, you will see MAJ:MIN, just like it is on Impish and previous.

[Where problems could occur]

We are changing the column output for -P and --pairs for the following applications:

* lsblk
* findmnt
* lsipc

If any application has been modified to depend on the new column output, it will break. I don't have any examples of something that will break, because MAAS and curtin were modified such that they would be compatible with both column name formats.

It should be noted that the manpage documents that lsblk output can change at any time:

> The default output, as well as the default output from options like --fs and --topology, is subject to change.
> So whenever possible, you should avoid using default outputs in your scripts.
> Always explicitly define expected columns by using --output columns-list and --list in environments where a stable output is required.

If a regression should occur, we will need to fix up these affected packages also.

[Other info]

The change came about when a user asked upstream to make -P / --pairs shell parsable, in Issue 1201 upstream:

https://github.com/util-linux/util-linux/issues/1201

Karel Zak obliged, and it was implemented in the following commit:

commit 58b510e5805d8350c31bfb81a47bcd38ea9fdd7e
From: Karel Zak <email address hidden>
Date: Thu, 3 Dec 2020 12:14:10 +0100
Subject: libsmartcols: sanitize variable names on export output
Link: https://github.com/util-linux/util-linux/commit/58b510e5805d8350c31bfb81a47bcd38ea9fdd7e

I wrote to Karel Zak with the regressions introduced by changing the format, and asked to revert back, and instead implement the shell parsable logic as a new parameter.

This happened in upstream issue 1594:

https://github.com/util-linux/util-linux/issues/1594

Karel Zak was happy to oblige again, and we now have the following commits:

338ad4a93 findmnt: commit missing flag
0f843ab64 lsblk: update --help output for -y
eba05f308 lsipc: add -y,--shell
152c17aa4 findmnt: add -y,--shell
9c7e81ff1 lslogins: add -y,--shell
25fb0638a lsblk: add -y/--shell
39679ea0c lsfd: use new libsmartcols functions
6fd0e3590 column: use new libsmartcols functions
0b3c2e80d include/carefulputc: remove unused function
3b5db50f7 libsmartcols: change "export" behavior, add "shellvar" flag

While we got the intended behaviour, these commits won't land until util-linux 2.38, which will be after Jammy releases, and the other issue is that this changes a significant amount of code, like nearly 1k lines, and is spread over 10+ commits.

I wrote to ubuntu-devel asking for advice, on either 1) not changing anything 2) backporting the 10+ new commits, or 3) simply reverting the commit which changed the behaviour.

https://lists.ubuntu.com/archives/ubuntu-devel/2022-February/041870.html

ubuntu-devel had strong support for option (3).

Hence, we will revert the below commit to ensure Jammy can be deployed on all existing MAAS releases.
58b510e580 libsmartcols: sanitize variable names on export output

Tags: jammy patch sts
Changed in util-linux (Ubuntu Jammy):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Matthew Ruffell (mruffell)
tags: added: jammy sts
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Attached is a debdiff for Jammy util-linux.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff for util-linux on Jammy" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.37.2-4ubuntu3

---------------
util-linux (2.37.2-4ubuntu3) jammy; urgency=medium

  * Revert the change to libsmartcols that outputs shell parsable column
    names when -P / --pairs is used in lsblk. The change breaks older
    MAAS and curtin versions, such that they cannot deploy Jammy.
    (LP: #1961542)
    - d/p/ubuntu/lp-1961542-Revert-libsmartcols-sanitize-variable-names-on-ex.patch

 -- Matthew Ruffell <email address hidden> Mon, 21 Feb 2022 14:49:57 +1300

Changed in util-linux (Ubuntu Jammy):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.