Some time back (5.6) the packaging was changed so failing tests during the build did not cause the build to fail. It's not clear why, as the test suite should at least be expected to pass on platforms that are supported by upstream.
[Impact]
Failing tests might not be noticed (there are occasionally some differences between build test runs and dep8 test runs). Failing builds can take longer because they're only spotted during dep8 runs.
Passing builds pointlessly spend time on the test runs, but ignore the result.
There are multiple elements to the change:
* Test failures will cause build failures on i386 and amd64 (for now. We will expand this list as we verify green test runs on other platforms.)
* We run the mtr suite with --force, meaning it doesn't abort on the first failure. This will help us get a complete list of which tests fail on which platforms
* We add a simple system for skipping spurious tests until they can be fixed or removed upstream (dep8 already has this)
* The test datadir_permissions has been consistently failing at build-time since 5.7.13, so we add a fix for it.
[Test case]
* Build package (the test datadir_permission will fail, but the failure is ignored)
[Regression potential]
* Spurious tests can cause build failures and not just dep8 failures (though this is technically a good thing, since they should be reported upstream and fixed.), so we might see more random build failures.
* --force makes builds failing due to tests take longer
This is present in 5.7.17-0ubuntu1 (present since 5.7.16-0ubuntu2).