[MASTER] package policykit-1 0.96-1 failed to install/upgrade: start-stop-daemon: nothing in /proc - not mounted? when system is run from external card

* also memtest86 failed to load or upgrade..........

Update from Karmic to lucid going wrong on an eeepc 701 4G, on an external 8G SSD

After hitting the "policykit-1" error, all packages upgrades then seem to fail

The lucid upgrade then aborted "in the middle of the road", leaving a unusable system in place (will need to download the iso and reinstall it from the usb stick).

Note that a few failures occurred during package download, but this did not seem to influence the installation process, which was able to retrieve packages from where it stopped.

Attached is an archive of the dist-upgrade directory on the hanged system.

attachment to previous post

Setting up policykit-1 (0.96-1) ...
start-stop-daemon: nothing in /proc - not mounted?
dpkg: error processing policykit-1 (--configure):
 subprocess installed post-installation script returned error exit status 2

Hi,

This is a local problem it seems, if /proc isn't mounted then not many things will
work well.

It's possible that something in the upgrade unmounted /proc, but I have no idea
what would cause that.

I'm marking this as "Incomplete", as it's not a problem with this package, and there
isn't enough information to know where the problem lies.

Thanks,

James

I've not yet reinstalled Lucid (although my usb stick is ready, I've tried Lucid live with it)

Is there anything else from the failed SSD I could provide or look at, which would help investigate ?

Does this need to be opened as a separate issue ?

I thought this was somehow dangerous, especially for unskilled users, that's why I chose to report this behavior.

Nicolas

Whether /proc is mounted depends on the running system, so I'm not
sure whether we could get anything useful from the disk.

Thanks,

James

I'm setting the status to 'confirmed' due to duplicates.

From those duplicates, the conditions of the failure seems to be during an upgrade from 9.10 to 10.04 when the system is run from an external card.

I experienced what I believe to be this same problem on my desktop machine. My desktop machine is not running on an external card, but it is running on a RAID controller which Ubuntu seems to think is 'removable.'

In any event, I have been able to replicate this problem by installing 9.10 on a USB flash drive and then upgrading it to 10.04. After a significant amount of troubleshooting, I have found that things appear to go wrong during the postinstall of the 'udisks' package. Running this line ('udevadm trigger --subsystem-match=block --action=change') in udisks.postinst results in empty /proc and /sys filesystems. Thus, the failure of policykit-1 to install is just a symptom of /proc being empty.

I'm still not sure whether this is a bug in udisks or possibly udev, but I guess I'll reassign it to udisks for now.

Repeated my test on an upgrade from Hardy -> Lucid on a USB Flash Drive. This worked. So this appears to be a Karmic -> Lucid only issue. Continuing to investigate why /proc and /sys appear to be getting unmounted somehow.

Scott,

seems that wasn't a scenario where the udevadm trigger would cause /proc to be unmounted -- in your "mount output after trigger", /proc is still mounted.

So this bug could potentially live in either udisks itself (jjust in case it does something really strange with /proc, but usually it doesn't touch those virtual FSes at all), or in mountall (which I believe mounts /proc and /sys in the first place).

Can someone who is able to reproduce this please do

  udisks --monitor-detail 2>&1 | tee /tmp/udisks.log

then reproduce the situation with the "udevadm trigger --subsystem-match=block --action=change" command or upgrading packages, ensuring that /proc is not mounted any more afterwards, and then attach /tmp/udisks.log here? Thanks!

On Sat, 26 Jun 2010, Martin Pitt wrote:

> seems that wasn't a scenario where the udevadm trigger would cause /proc
> to be unmounted -- in your "mount output after trigger", /proc is still
> mounted.

You're right - it was technically still mounted (per 'mount') but /proc
was empty. Ie, 'ls -l /proc' returns an empty directory.

> You're right - it was technically still mounted (per 'mount') but /proc was empty. Ie, 'ls -l /proc' returns an empty directory.

That sounds seriously wrong. What is the exact output of "mount" and "cat /proc/mounts" when this happens?

On Mon, 28 Jun 2010, Martin Pitt wrote:

>> You're right - it was technically still mounted (per 'mount') but
> /proc was empty. Ie, 'ls -l /proc' returns an empty directory.
>
> That sounds seriously wrong. What is the exact output of "mount" and
> "cat /proc/mounts" when this happens?

The output of "mount" is as I attached in 'mount.after'. If I try to "cat
/proc/mounts" it fails because mounts doesn't exist. I can try to get you
the exact output tonight.

I'm still trying to get you output from udisks --monitor-detail.
Unfortunately, this it seems to be hard to reproduce this problem outside
of running under the update manager.

Further troubleshooting on this...the empty /proc & /sys occurs during the karmic->lucid upgrade when "udevadm trigger --subsystem-match=block --action=change" is called AFTER the devicekit-disks package has been removed. I can reproduce this outside the upgrade installer by manually removing devicekit-disks and then running the udevadm trigger command.

I tried that on lucid (install devicekit-disks, devkit-disks --dump, purge, and trigger), and /proc and /sys are still fine. How exactly did you reproduce this? On karmic, before the upgrade? Any other steps than I did?

> I tried that on lucid (install devicekit-disks, devkit-disks --dump,
> purge, and trigger), and /proc and /sys are still fine. How exactly did
> you reproduce this? On karmic, before the upgrade? Any other steps than
> I did?

On Karmic I did:
dpkg -r devicekit-disks --ignore-depends=devicekit-disks
udevadm trigger --subsystem-match=block --action=change
ls -l /proc ==> empty

Note that the other key factor seems to be that Ubuntu needs to be
installed on disk that is seen as 'removable', e.g. a flash drive.

I haven't tried the above steps on a Lucid install.

Further narrowing it down, removal of the file /lib/udev/rules.d/95-devkit-disks.rules followed by the udevadm trigger causes the empty /proc.

One possibly interesting thing that I've noticed about this is when I look at the output of 'udevadm monitor' while running the 'udevadm trigger --subsystem-match=block --action=change' command which causes /proc to go away, it shows this line (which it doesn't show if I run the 'udevadm trigger --subsystem-match=block --action=change' command any other time):

UDEV [1277516625.070541] remove /devices/virtual/bdi/0:21 (bdi)

First, shouldn't udev only be issuing 'change' events and not 'remove' events? Second what exactly is 'bdi'? I can't seem to find a lot of information on it.

Scott

Scott Talbert [2010-07-19 4:00 -0000]:
> UDEV [1277516625.070541] remove /devices/virtual/bdi/0:21 (bdi)

Hm, I'm not sure about this. I don't see anything in udev or udisks
which would issue events on bdi devices, and usually they originate
from the kernel. Does this also appear in udevadm monitor --kernel?

> First, shouldn't udev only be issuing 'change' events and not 'remove'
> events?

In general it should mirror whatever the kernel sends, including
removal ones. However, in this "trigger" case if should only
synthesize change events, of course. I don't believe that this bdi
remove event is originating from udev itself.

> Second what exactly is 'bdi'? I can't seem to find a lot of
> information on it.

Me neither,
http://www.mjmwired.net/kernel/Documentation/ABI/testing/sysfs-class-bdi
was the closest thing I found.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

Yes, the bdi event does appear as a kernel event as well.

One thing I noticed tonight is that, when things go wrong, /dev/sda also no longer exists. (/dev/sda1, /dev/sda2, and /dev/sda5 do still exist, though.) I'll attach a log of udevd in debug mode when this occurs. I couldn't see anything obviously out of the ordinary myself.

OK, I think I finally have a reasonably good idea of what's going on with this now. I had been focused on udev and udisks, but it turns out that the bug appears to actually be in devicekit-disks. Here's a rough timeline of what happens:

1. During the Karmic->Lucid upgrade, devicekit-disks package is removed, which includes its udev rules (/lib/udev/rules.d/95-devkit-disks.rules). The daemon (devkit-disks-daemon) is still hanging around, however.
2. Later on, during the upgrade, the command 'udevadm trigger --subsystem-match=block --action=change' is issued (it happens to be in udisks). When this happens, devicekit-disks' data items (e.g. DKD_*) are removed from the udev database.
3. The devkit-disks-daemon gets notified that 'sda' has changed. During the change processing, it discovers that 'sda' is a 'removable' device and because the DKD_* udev entries are no longer present, it determines that 'sda' has no media present and then proceeds to perform a force unmount on /dev/sda1. This unsurprisingly results in a broken system.

Since devicekit-disks is deprecated, I am thinking that perhaps the simplest fix would be to add a devicekit-disks.prerm that simply stops the daemon before removing the package. I've run the upgrade with the attached devicekit-disks.prerm file and the upgrade completed successfully.

Note that udisks does not appear to exhibit this same issue because the media_available logic has been changed. See this change: http://cgit.freedesktop.org/udisks/commit/?id=c8d02e0cc26a7559063b98de05f8a4ee07f1de2b. Perhaps a safer fix would be to backport this change to devicekit-disks (and not rely on the daemon not being running).

So I went ahead and backported the media detection change to devicekit-disks. It is attached. My test system successfully upgraded from Karmic->Lucid with the patched devicekit-disks in place.

Scott, thanks a lot for your investigations, this makes a lot of sense.

To be honest I would be rather cautious with backporting the media detection patch. A lot of other layers (gvfs, blkid, udev, etc.) have changed since then, and it is not immediately obvious that this patch would not cause regressions with some media types or situations.

My gut feeling is that it is much safer to kill the daemon in the prerm, as you already pointed out. The postinst already does this for upgrades, so we can just copy the script and replace "configure" with "remove". Do you want to prepare an SRU/debdiff yourself, or want me to do it?

Thanks, Martin

On Mon, 26 Jul 2010, Martin Pitt wrote:

> so we can just copy the script and replace "configure" with "remove". Do
> you want to prepare an SRU/debdiff yourself, or want me to do it?

I haven't done one in a while, but I can take a crack at it.

Scott

Debdiff attached.

Thanks Scott!

Accepted devicekit-disks into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Installed the karmic-proposed package onto my test Karmic image. Upgraded successfully from Karmic->Lucid.

This bug was fixed in the package devicekit-disks - 007-2ubuntu7

---------------
devicekit-disks (007-2ubuntu7) karmic-proposed; urgency=low

  * Add devicekit-disks.prerm script which stops the devicekit-disks
    daemon when devicekit-disks is removed. This fixes a problem with
    the Karmic to Lucid upgrade on systems with 'removable' disks.
    (LP: #554718)
 -- Scott Talbert <email address hidden> Mon, 26 Jul 2010 19:14:54 -0400