Basic vmtest run fails on s390x

Bug #1866663 reported by Paride Legovini
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
High
Ryan Harper

Bug Description

The BionicTestBasic vmtests fail to run on s390x. The Xenial and Focal based tests fail in a similar way. I also tried running BionicTestScsiBasic but it's not working either.

Test run stdout:

ubuntu@s1lp04:~/paride/curtin$ CURTIN_VMTEST_TOPDIR=`pwd`/output CURTIN_VMTEST_BOOT_TIMEOUT=3000 ./tools/jenkins-runner tests/vmtests/test_basic.py:BionicTestBasic
CURTIN_VMTEST_BOOT_TIMEOUT=3000
CURTIN_VMTEST_CURTIN_EXE_VERSION=19.3-30-gc650f6c2
CURTIN_VMTEST_CURTIN_VERSION=19.3-30-gc650f6c2
CURTIN_VMTEST_IMAGE_SYNC=0
CURTIN_VMTEST_ISCSI_PORTAL=10.245.237.4:1234
CURTIN_VMTEST_KEEP_DATA_FAIL=logs,collect
CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
CURTIN_VMTEST_LOG=/home/ubuntu/paride/curtin/output/debug.log
CURTIN_VMTEST_PARALLEL=0
CURTIN_VMTEST_REUSE_TOPDIR=0
CURTIN_VMTEST_TAR_DISKS=0
CURTIN_VMTEST_TOPDIR=/home/ubuntu/paride/curtin/output
TGT_IPC_SOCKET=/home/ubuntu/paride/curtin/output/tgt.d/socket
TGT_LOG_D=/home/ubuntu/paride/curtin/output/tgt.d
TGT_PID=226081
TGT_PORTAL=10.245.237.4:1234
http_proxy=
https_proxy=
no_proxy=
Quering synced ephemeral images/kernels in /srv/images
======================================================================================
 Release Codename ImageDate Arch /SubArch Path
--------------------------------------------------------------------------------------
   16.04 xenial 20200229 s390x/ga-16.04 xenial/s390x/20200229/squashfs
   16.04 xenial 20200229 s390x/hwe-16.04 xenial/s390x/20200229/squashfs
   16.04 xenial 20200229 s390x/hwe-16.04-edge xenial/s390x/20200229/squashfs
   18.04 bionic 20200304 s390x/ga-18.04 bionic/s390x/20200304/squashfs
   18.04 bionic 20200304 s390x/hwe-18.04 bionic/s390x/20200304/squashfs
   18.04 bionic 20200304 s390x/hwe-18.04-edge bionic/s390x/20200304/squashfs
   19.10 eoan 20200307 s390x/ga-19.10 eoan/s390x/20200307/squashfs
   20.04 focal 20200305 s390x/ga-20.04 focal/s390x/20200305/squashfs
--------------------------------------------------------------------------------------
======================================================================================

Mon, 09 Mar 2020 16:18:36 +0000: vmtest start: nosetests3 -vv --nologcapture tests/vmtests/test_basic.py:BionicTestBasic
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
2020-03-09 16:18:37,260 - vmtests - INFO - Logfile: /home/ubuntu/paride/curtin/output/debug.log . Working dir: /home/ubuntu/paride/curtin/output
2020-03-09 16:18:37,265 - BionicTestBasic - INFO - Logfile: /home/ubuntu/paride/curtin/output/debug.log . Working dir: /home/ubuntu/paride/curtin/output
2020-03-09 16:18:37,267 - BionicTestBasic - INFO - Starting setup for testclass: BionicTestBasic (ubuntu/bionic -> ubuntu/bionic)
2020-03-09 16:18:37,334 - BionicTestBasic - INFO - Using tempdir: /home/ubuntu/paride/curtin/output/BionicTestBasic
2020-03-09 16:18:37,334 - BionicTestBasic - INFO - Loading testcase config file: examples/tests/basic.yaml
2020-03-09 16:18:37,352 - BionicTestBasic - INFO - Publishing ephemeral image as --publish=/srv/images/bionic/s390x/20200304/squashfs:root/squashfs
2020-03-09 16:18:37,352 - BionicTestBasic - INFO - Curtin install source URI: cp:///media/root-ro
2020-03-09 16:18:37,367 - BionicTestBasic - INFO - Loading testcase config file: examples/tests/basic.yaml
2020-03-09 16:18:37,381 - BionicTestBasic - INFO - disk_serials: ['serial=disk-a', 'serial=disk-b', 'serial=disk-c', 'serial=disk-d']
2020-03-09 16:18:37,381 - BionicTestBasic - INFO - nvme_serials: []
2020-03-09 16:18:37,381 - BionicTestBasic - INFO - nvme disks: []
2020-03-09 16:18:37,397 - BionicTestBasic - INFO - Using root_disk: []
2020-03-09 16:18:37,398 - BionicTestBasic - INFO - Running curtin installer: /home/ubuntu/paride/curtin/output/BionicTestBasic/logs/install-serial.log
2020-03-09 16:20:50,191 - BionicTestBasic - INFO - BionicTestBasic[install]: boot took 132.79 seconds. returned True
2020-03-09 16:20:50,421 - BionicTestBasic - INFO - Install OK
2020-03-09 16:20:50,421 - BionicTestBasic - INFO - Booting target image: /home/ubuntu/paride/curtin/output/BionicTestBasic/logs/boot-serial.log
2020-03-09 16:20:50,657 - BionicTestBasic - INFO - BionicTestBasic[first_boot]: boot took 0.24 seconds. returned True
2020-03-09 16:20:50,700 - BionicTestBasic - INFO - BionicTestBasic: setUpClass finished. took 133.43 seconds. Running testcases.
get_test_files (vmtests.test_basic.BionicTestBasic) ... ok
test_clear_holders_ran (vmtests.test_basic.BionicTestBasic) ... ERROR
test_curtin_install_version (vmtests.test_basic.BionicTestBasic) ... ok
test_dname (vmtests.test_basic.BionicTestBasic) ... FAIL
test_dname_rules (vmtests.test_basic.BionicTestBasic) ... FAIL
test_fstab (vmtests.test_basic.BionicTestBasic) ... ERROR
test_installed_correct_kernel_package (vmtests.test_basic.BionicTestBasic) ... ERROR
test_interfacesd_eth0_removed (vmtests.test_basic.BionicTestBasic) ... ERROR
test_kernel_img_conf (vmtests.test_basic.BionicTestBasic) ... FAIL
test_output_files_exist (vmtests.test_basic.BionicTestBasic) ... FAIL
test_partition_is_prep (vmtests.test_basic.BionicTestBasic) ... ERROR
test_partition_is_zero (vmtests.test_basic.BionicTestBasic) ... ERROR
test_partition_numbers (vmtests.test_basic.BionicTestBasic) ... ERROR
test_proxy_set (vmtests.test_basic.BionicTestBasic) ... FAIL
test_ptable (vmtests.test_basic.BionicTestBasic) ... FAIL
test_reporting_data (vmtests.test_basic.BionicTestBasic) ... ok
test_swaps_used (vmtests.test_basic.BionicTestBasic) ... ERROR
test_whole_disk_uuid (vmtests.test_basic.BionicTestBasic) ... ERROR

======================================================================
ERROR: test_clear_holders_ran (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1815, in test_clear_holders_ran
    if not self.has_storage_config():
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1911, in has_storage_config
    return len(self.get_storage_config()) > 0
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1906, in get_storage_config
    cfg = load_config(self.collect_path("root/curtin-install-cfg.yaml"))
  File "/home/ubuntu/paride/curtin/curtin/config.py", line 99, in load_config
    with open(cfg_file, "r") as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/root/curtin-install-cfg.yaml'

======================================================================
ERROR: test_fstab (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1617, in test_fstab
    expected = self.get_fstab_expected()
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 139, in get_fstab_expected
    rootdev = self._serial_to_kname('disk-a')
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1712, in _serial_to_kname
    ls_byid = self.load_collect_file("ls_al_byid")
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/ls_al_byid'

======================================================================
ERROR: test_installed_correct_kernel_package (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1810, in test_installed_correct_kernel_package
    self.assertIn(kpackage, self.debian_packages)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1880, in debian_packages
    data = self.load_collect_file("debian-packages.txt")
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/debian-packages.txt'

======================================================================
ERROR: test_interfacesd_eth0_removed (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1798, in test_interfacesd_eth0_removed
    interfacesd = self.load_collect_file("find_interfacesd")
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/find_interfacesd'

======================================================================
ERROR: test_partition_is_prep (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 173, in test_partition_is_prep
    self._test_partition_is_prep("udev_info.out")
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 104, in _test_partition_is_prep
    udev_info = self.load_collect_file(info_file).rstrip()
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/udev_info.out'

======================================================================
ERROR: test_partition_is_zero (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 394, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 178, in test_partition_is_zero
    self._test_partition_is_zero("cmp_prep.out")
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 118, in _test_partition_is_zero
    self.assertEqual(0, int(self.load_collect_file(cmp_file).rstrip()))
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/cmp_prep.out'

======================================================================
ERROR: test_partition_numbers (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 134, in test_partition_numbers
    disk = self._serial_to_kname('disk-d')
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1712, in _serial_to_kname
    ls_byid = self.load_collect_file("ls_al_byid")
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/ls_al_byid'

======================================================================
ERROR: test_swaps_used (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1915, in test_swaps_used
    if not self.has_storage_config():
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1911, in has_storage_config
    return len(self.get_storage_config()) > 0
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1906, in get_storage_config
    cfg = load_config(self.collect_path("root/curtin-install-cfg.yaml"))
  File "/home/ubuntu/paride/curtin/curtin/config.py", line 99, in load_config
    with open(cfg_file, "r") as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/root/curtin-install-cfg.yaml'

======================================================================
ERROR: test_whole_disk_uuid (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 149, in test_whole_disk_uuid
    self._serial_to_kname('disk-c'),
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1712, in _serial_to_kname
    ls_byid = self.load_collect_file("ls_al_byid")
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1548, in load_collect_file
    with open(self.collect_path(filename), mode) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/paride/curtin/output/BionicTestBasic/collect/ls_al_byid'

======================================================================
FAIL: test_dname (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1652, in test_dname
    self.output_files_exist(["ls_dname"])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

======================================================================
FAIL: test_dname_rules (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1672, in test_dname_rules
    self.output_files_exist(["udev_rules.d"])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

======================================================================
FAIL: test_kernel_img_conf (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1844, in test_kernel_img_conf
    self.output_files_exist([kconf])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

======================================================================
FAIL: test_output_files_exist (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 124, in test_output_files_exist
    "root/curtin-install.log", "root/curtin-install-cfg.yaml"])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

======================================================================
FAIL: test_proxy_set (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 155, in test_proxy_set
    self.output_files_exist(['apt-proxy'])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

======================================================================
FAIL: test_ptable (vmtests.test_basic.BionicTestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 130, in test_ptable
    self._test_ptable("blkid_output_diska", expected_ptable)
  File "/home/ubuntu/paride/curtin/tests/vmtests/test_basic.py", line 72, in _test_ptable
    self.output_files_exist([blkid_output])
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1537, in output_files_exist
    msg="expected collected files do not exist.")
AssertionError: False is not true : expected collected files do not exist.

----------------------------------------------------------------------
Ran 18 tests in 134.236s

FAILED (errors=9, failures=6)
Mon, 09 Mar 2020 16:20:51 +0000: vmtest end [1] in 136s

Related branches

Ryan Harper (raharper)
Changed in curtin:
assignee: nobody → Ryan Harper (raharper)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Paride Legovini (paride) wrote :

I tested the fix/vmtest-basic/s390x branch, it corrently skips the TestBasic tests but still fails while running BionicTestScsiBasic and FocalTestScsiBasic. The output for the Bionic failure is (Focal is identical):

======================================================================
ERROR: test_clear_holders_ran (vmtests.test_basic.FocalTestScsiBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 379, in test_wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1840, in test_clear_holders_ran
    install_log = self.load_collect_file(install_logfile)
  File "/home/ubuntu/paride/curtin/tests/vmtests/__init__.py", line 1556, in load_collect_file
    return fp.read()
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 82285: ordinal not in range(128)

Revision history for this message
Ryan Harper (raharper) wrote :

As we debugged, this is not related to this fix; just unfortunate output that triggered this scenario.

I was going to include a fix in this branch but refactoring it touches 3 methods of vmtest which are heavily utilized so I'll fix the load issue in another branch:

https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/380571

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit b10aeef0 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=b10aeef0

Changed in curtin:
status: In Progress → Fix Committed
Revision history for this message
Ryan Harper (raharper) wrote : Fixed in curtin version 20.1.

This bug is believed to be fixed in curtin in version 20.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.