mountall needs to flush plymouth message queue before emitting upstart events

I'm seeing a variation on this, with significant similarities - I'll try the method above.

In my case, Plymouth comes up, and tells me that it will check all the disks, but then halts at 90% or 8% (depending on whether it checks one or both of the disks). It provides the option of pressing C to cancel the check, but does not respond if C is pressed (shift + c or just c).

If I press escape, I get several screens full of "udevd[xxx]: SYSFS{}= will be removed in a future udev version, please use ATTR{}= to match the event device, or ATTRS= to match a parent device, in /etc/udev/rules.d/65-libmatp.rules:89" or variations upon that theme.

Often I also get a "ureadahead-other main process terminated with status 4" message or two

At end, it also drops several instances of the aforementioned "mountall: Plymouth command failed".

At which point, the computer appears to hang, and no further progress is made towards booting.

I tested other kernel versions, and I get roughly the same series of events.

A time-out issue may be causing this. My disk has a mixture of ext3 and ext4 partitions, with one of the ext3 partitions at 91% capacity. This causes fsck to take a long time. Below is the relevant portion of '$ df -h'. I'm also attaching a copy of my '/etc/fstab', in case this helps in resolving the issue:

Filesystem Size Used Avail Use% Mounted on
/dev/sda6 12G 6.5G 4.8G 58% / (Ext4)
/dev/sda1 122M 66M 50M 57% /boot (Ext3)
/dev/sda5 15G 14G 1.4G 91% /root/snapshot (Ext3)
/dev/sda2 63G 44G 19G 71% /home (Ext4)

In response to fejes:

> In my case, Plymouth comes up, and tells me that it will check all the
> disks, but then halts at 90% or 8%

I normally boot without the Plymouth splash screen (i.e. I've appended 'nosplash' to the boot options in grub). However, I've just tried booting with the splash screen. The progress indicator changes intermittently over the course of the disk check, rather than advance at a constant rate. For example, it will quickly advance to 30%, then pause for a minute before advancing again. For the record, my boot times from grub to the kdm logon screen are:
Normal, no fsck: 35 seconds
File /forcefsck exists: 8 minutes

The presence or absence of a splash screen doesn't significantly alter these times.

> If I press escape, I get several screens full of "udevd[xxx]: SYSFS{}=
> will be removed in a future udev version, please use ATTR{}= to match
> the event device, or ATTRS= to match a parent device, in
> /etc/udev/rules.d/65-libmatp.rules:89" or variations upon that theme.

I don't see these messages, and grep fails to find the 'SYSFS' string in either /etc/udev/rules.d/* or /lib/udev/rules.d/*

> Often I also get a "ureadahead-other main process terminated with status
> 4" message or two

This is the normal exit status when ureadahead doesn't need to read any data on a given partition. See comment at http://ubuntuforums.org/showpost.php?p=8998483&postcount=1

Hi Matthew, Thanks for the helpful comments - those are great explanations for what I'm seeing.

Just to reply to a couple points:

> Normal, no fsck: 35 seconds
> File /forcefsck exists: 8 minutes

This morning, I left it for over an hour once it hit 90%, and no change was visible.

> The presence or absence of a splash screen doesn't significantly alter these times.

I often boot with nosplash, and you're correct, I don't see any difference in behaviour with or without it.

> I don't see these messages, and grep fails to find the 'SYSFS' string in either /etc/udev/rules.d/* or /lib/udev/rules.d/*

It's possible that this is something has failed to upgrade over several different iterations of ubuntu. This particular computer has been upgraded repeatedly since 2007, and may be due for a reinstall anyhow. I alo found a bug report that seemed similar to this, so it may be entirely unrelated, however I included it for completeness.

Just a quick update - There was corruption on one of the hard drives, which was fixed by running a fsck from an old 8.10 server boot disk. It turned out that it wasn't getting stuck at 90%, but rather that it was restarting repeatedly at 90% after a long pause. At any rate, that part is no longer a problem.

Most of my other problems seem to have been the result of my disk order being changed. My SATA disk went from being /dev/hda to /dev/hdc, with predictable consequences. Changing the /etc/fstab file designation of all of my drives fixed the issue.

possible duplicate of 554079 ?

Perhaps this doesn't need to be here, but for completeness, I figured I should include one final update, the /dev/sd* designations of my hard drives began to be assigned inconsistently from one boot to the next, causing similar but not identical mount errors. This was solved by converting my /etc/fstab to use UUIDs instead of indicating the /dev/sd* notation.

> possible duplicate of 554079 ?

This bug is similar to (at least) 554079, 554737 and 557161. However, there are differences in each case:

In bug #554079, it appears that plymouth is stalling during fsck, which results in GDM not loading. 554079 also makes no mention of the "mountall: Plymouth command failed" error message. In my case KDM loads properly and fsck appears to operate correctly.

In bug #554737, the mountall process appears to be blocked by a plymouthd process, and vice versa. In my case, there are no plymouth processes running after boot.

In bug #557161, the mountall process terminates shortly after KDM loads. In my case, the mountall process continues running for several minutes.

Some additional details:

If fsck only checks my root partition at boot time (setting a high mount count with tune2fs), mountall terminates promptly. The "mountall: Plymouth command failed" errors only appear 30 or so times before ceasing, as in bug #557161.

I can replicate the symptom of mountall failing to terminate by running fsck on only /dev/sda5 at boot. This is a 15 GiB, ext3 partition, 91% full. A fsck on this partition normally takes less than 4 minutes.
However, at boot time, mountall continues running for 17 minutes. Once the system had booted (approx. 4 minutes), no fsck processes were running, nor was there any HDD activity.
  $ ps -F -C mountall
    UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
    root 356 1 82 10598 40852 0 13:42 ? 00:17:16 mountall --daemon
The mountall process terminated less than 2 seconds later.

I am also attaching a backtrace of the mountall process, as requested at https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/554737/comments/22. However, unlike that bug, there were no plymouth processes running at the time ('ps -ef | grep ply' failed to find anything).

I've uploaded a new version of mountall (2.13~ppa1) to https://launchpad.net/~scott/+archive/ppa

Please test and see whether it solves this problem

Thanks

Hello Scott,

The updated mountall package (2.13~ppa1) appears to create more problems than it solves.

Although the mountall process no longer spams the console with "mountall: Plymouth command failed", there is now a more serious problem. The system no longer boots, even if fsck isn't doing disk checks. The last boot status messages show the disk status. After this, the system still responds to the Alt-Ctrl-F? commands to switch consoles, but no further advancement is seen in the boot process (KDM doesn't start, no logon prompts appear on the consoles). SysRq-R, SysRq-E, SysRq-I causes the system to launch the mountall-shell recovery shell, with the root partition mounted read only. The following output is observed on the console:

This output is transcribed. I've contracted some parts between the '<>' tags.
--- Output Begins ---

fsck from util-linux-ng 2.17.2
Root: clean, <xxx>/<yyy> files, <ppp>/<qqq> blocks
<Similar messages for all other partitions fsck usually checks>

SysRq : Keyboard mode set to system default
SysRq : Kill All Tasks

init: plymouth main process (348) killed by KILL signal
init: plymouth-stop pre-start process (854) terminated with status 1
init: udev main process (387) killed by KILL signal
init: udev main process ended, respawing
[ <uptime> ] udev: starting version 151
General error mounting filesystem

<Recovery shell boilerplate>
root@<host>:~#

--- Output Ends ---

I'm also attaching the list of running processes 'ps -eF', obtained once the root filesystem was remounted read/write.

Package versions:
mountall: 2.13~ppa1
plymouth: 0.8.2-2
kernel: 2.6.32-21 i386

Hello,

Looks like version 2.13~ppa1 works correctly when fsck actually checks disks, but crashes when checks are skipped (i.e. when filesystems are clean).

If filesystems are clean, splash screen animation loops endlessly and nothing happens. But it is possible to press key 'C' and get to recovery shell.

Then I noticed the following line in dmesg:
[ 28.948962] mountall[322]: segfault at 50 ip 00007fc03da6b0a7 sp 00007ffff41fd930 error 4 in mountall[7fc03da60000+15000]

Then I run mountall and then system starts and everything works fine.

Hope this information helps.

Dear Scott,
Per your directions in #9 above, mountall~ppa1 is working fine for Lucid 10.04 beta 2 x86_64; however I'm still wondering why the checkdisk utility now desires to run each time boot-up occurs. I will run a memory test to be sure and report back on the results. It would be very strange to see this harddrive (<year old) have any corruption at this stage.

OK, it looks like the hdd is fine after running some integrity tests. However, now I'm having a problem where sometimes if I press 'C' to cancel the check disk at boot, I get launched into tty1 and the command 'startx' says something like 'failed to find device.'

As commented in bug #553290, this change introduces a regression because of a bug in ply_boot_client_flush(). So regrettably, I'm once again backing out a mountall fix and uploading 2.13 with the other critical fix that we already have pending.

Alright, I'm not having anymore problems now that the new mountall is in the updates. Thanks!

I take back that last post, now this is really weird:

1) After update to mountall version 2.14 (or the latest one) the system booted correctly 3 times last night.
2) This morning, X11 failed to start, and I was kicked again to tty1. I tried reconfiguring xserver-xorg etc, nothing.
3) The only way to get into the GUI interface is to do 'sudo startx', at which point I'm logged in as root. If I try to log out so I can go to my normal desktop, I get booted to tty1 again.

Since the update to mountall 2.14, there were no modifications to the system. I can't understand why it would work one day and not the next.

Please help!!!

It happens to me at random times. A dialog box comes up which includes the option "restart x" or something similar (always the last option.). Selecting it always causes it to start correctly immediately. (I tried the other options once, and it didn't help - just restarting it apparently is the only correct solution.)

Hello.

I had this same problem where mountall consumed a lot of CPU cycles (and appeared to be busy-looping) in case fsck was performed. The program was not busy-looping, and did eventually exit by itself.

This behaviour was caused by an assertion at libply module, file ply-list.c, function ply_list_unlink_node(). The last line of the function asserts that the node was unliked properly by scanning through the list (lineat time operation).

When for (some reason) connection from mountall to plymouth was broken, the event loop noticed this, and application started to cancel pending requests (I had somehing like 300000 of those). The assertion, if present, basically leads to time quadratic behaviour for this cancellation operation.

This should not happen on non-debug (-DNDEBUG) builts.

More proper fix would be to figure out why the socket towards plymouth is closed, and keep that open as long as mountall/fsck is running.

Br.
--
tmo

OK guys, I have a fix to hold us over until the gdm/plymouth/mountall debacle is over.

1) If you get kicked to terminal, type 'sudo startx' to go log into the x-server as root.
2) Goto synaptic package manager, and find gdm, gdm-guest-session and plymouth-x11. Mark both for uninstall.
3) Find xdm and mark for install.
4) Click on apply changes. A little config screen will pop up asking you to select the default, choose xdm (second choice).
5) Reboot, you should be able to login as normal even tho the screen looks different. For some reason, the colors in X11 were kinda messed up for me, but who cares until a better fix comes out.

Yes I know this solution isn't elegant, but if you want your desktop back do as I say!
-Jabba

scratch that, now I'm having the same problem again. ugh

Alright, I have narrowed the problem down to gdm not being initialized during the boot process, for when I am kicked to tty1, I simply 'sudo gdm' and I arrive at the gnome login screen and business as usual. None of the subsequent modifications to mountall nor initramfs that are now in the repository have changed this yet.

I am running the x86_64 lucid and have an nvidia geforce driver, which doesn't seem to be the problem at all.

I have no idea what's going on, but something is really odd with mountall; the machine I just upgraded to lucid refused to boot until I rewrote /etc/fstab to only contain /. Otherwise I'd hang on some nondescript plymouth/mountall error (each in a different console).

sudo apt-get remove ubuntu

jk

Have the same problem in RC!!! It will kill my SSD in EEEPC. Can you fix it?

I've committed what I believe to be the correct fix to the plymouth bzr branch. With this change, mountall reliably emits all signals for me, instead of hanging like before.

The only remaining issue I see here is that trying to cancel fsck when there are more than 2 filesystem checks queued results in the additional ones marked failed and showing the 'i'gnore, 's'kip, 'm'aintenance prompt. Pressing 'i' succeeds in progressing for the first filesystem, but for the second filesystem there appears to be no reaction - the message stays on the screen and the boot stops. But as this isn't a regression (prior to this upload of plymouth, pressing 'C' to cancel fsck doesn't work *at all*), and the remaining problem doesn't appear to be a bug in plymouth (it appears to be a bug in mountall), I don't think it's a reason to block the plymouth half of this fix, at least. I've uploaded plymouth to the lucid unapproved queue.

This bug was fixed in the package plymouth - 0.8.2-2ubuntu1

---------------
plymouth (0.8.2-2ubuntu1) lucid; urgency=low

  * src/main.c: if the splash screen isn't up yet, queue message requests
    instead of discarding them. LP: #507881.
  * src/client/ply-boot-client.c: some replies may be sent out of order
    because they depend on user input, so pay attention to the message type
    when picking the handler instead of handing the response to the first
    handler in the list; without this, cancelling fsck in mountall will
    never work. LP: #562811.
  * src/client/ply-boot-client.c: instead of trying to read from the server
    pipe if there are any outstanding requests, call
    ply_event_loop_process_pending_events() which already knows whether we
    can read from the pipe. LP: #559761.
  * add the pixel display bpp symbols to libplymouth2.symbols with a correct
    version, so that packages using them don't wind up with overly-strict
    dependencies on libplymouth2.
 -- Steve Langasek <email address hidden> Sun, 25 Apr 2010 16:15:37 +0100

This bug was fixed in the package mountall - 2.14

---------------
mountall (2.14) lucid; urgency=low

  [ Scott James Remnant ]
  * Flush updates to Plymouth before emitting Upstart events, in case
    the event kills Plymouth. LP: #559761.
  * Don't mark a filesystem "nodev" just because it's got "none" in the
    device column; this will block the "virtual-filesystems" event which
    is the one that can't use Plymouth to prompt. LP: #507881.
  * When cancelling in-progress fsck, don't deference the NULL mount
    record. LP: #562811.
  * mountall is missing a very important line of code that increases the
    udev buffer size; without this it's possible we may miss events during
    busy periods. LP: #561390.

  [ Steve Langasek ]
  * If we're not marking all nodev filesystems as virtual, we need to
    at least mark our placeholder filesystem entries (type=none && dev=none)
    this way.
 -- Steve Langasek <email address hidden> Sun, 25 Apr 2010 23:36:01 +0100

Folks,

This issue is still present with the following versions:

plymouth: 0.8.2-2ubuntu1
mountall: 2.15~ppa1

Fsck progress displayed at boot screen stops advancing at 70%.
20 seconds after that the login screen appears.

Meanwhile mountall keeps logging: "mountall: Plymouth command failed" on console.
Mountall is still running after login and is busy removing outgoing events (see comment #18) on the following stack trace after it has noticed that the socket towards plymouthd has been closed.

#0 0x00007f2b3c5d06e3 in ply_list_find_node () from /lib/libply.so.2
#1 0x00007f2b3c5d0933 in ply_list_remove_node () from /lib/libply.so.2
#2 0x00007f2b3c3c3c64 in ?? () from /lib/libply-boot-client.so.2
#3 0x00007f2b3c3c3d9e in ?? () from /lib/libply-boot-client.so.2
#4 0x00007f2b3c5cd24b in ply_event_loop_process_pending_events ()
   from /lib/libply.so.2
#5 0x00007f2b3d262482 in nih_io_handle_fds () from /lib/libnih.so.1
#6 0x00007f2b3d263f0f in nih_main_loop () from /lib/libnih.so.1
#7 0x000000000040e2d4 in main (argc=3, argv=0x7fff1f547d78) at mountall.c:3313

To repeat: touch /forcefsck && reboot

This x86_64 host was upgraded from 9.10.

Workaround: Remove the assertion causing time-quadratic behaviour from ply_list_remove_node(), and make mountall less verbose when it fails to issue plymouth commands.
--
tmo

No, you're seeing bug #553745, a different bug.

I have a Point of View Ion 330 with Ubuntu server running as a home server.
apt-get dist-upgrade shows no pending packages.
I also get the "mountall Plymouth command failed" error at every boot.
The system runs in command line mode only and shows the login prompt shortly after the mountall error.
Nonetheless I hope the mountall error gets fixed soon.

2.6.32-33-server #72-Ubuntu SMP Fri Jul 29 21:21:55 UTC 2011 x86_64 GNU/Linux.
Description: Ubuntu 10.04.3 LTS
Release: 10.04
Codename: lucid

H&#283;ll&#248;!

&#205; &#229;m &#229; h&#229;cker wh&#248; h&#229;s &#229;ccess t&#248; y&#248;&#252;r &#248;p&#283;r&#229;t&#237;ng syst&#283;m.
&#205; &#229;ls&#248; h&#229;v&#283; full &#229;cc&#283;ss t&#248; y&#248;&#252;r &#229;cc&#248;&#252;&#328;t.

&#205;'v&#283; b&#283;&#283;n w&#229;tch&#237;ng y&#248;&#252; f&#248;r &#229; f&#283;w m&#248;nths n&#248;w.
Th&#283; f&#229;ct &#237;s th&#229;t y&#248;&#252; w&#283;r&#283; &#237;nf&#283;ct&#283;d w&#237;th m&#229;lw&#229;r&#283; thr&#248;&#252;gh &#229;n &#229;d&#252;lt s&#237;t&#283; th&#229;t y&#248;&#252; v&#237;s&#237;t&#283;d.

&#205;f y&#248;&#252; &#229;r&#283; n&#248;t f&#229;m&#237;l&#237;&#229;r w&#237;th th&#237;s, &#205; w&#237;ll &#283;xpl&#229;&#237;n.
Tr&#248;j&#229;n V&#237;r&#252;s g&#237;v&#283;s m&#283; f&#252;ll &#229;cc&#283;ss &#229;nd c&#248;ntr&#248;l &#248;v&#283;r &#229; c&#248;mp&#252;t&#283;r &#248;r &#248;th&#283;r d&#283;v&#237;c&#283;.
Th&#237;s m&#283;&#229;ns th&#229;t &#205; c&#229;n s&#283;&#283; &#283;v&#283;ryth&#237;ng &#248;n y&#248;&#252;r scr&#283;&#283;n, t&#252;rn &#248;n th&#283; c&#229;m&#283;r&#229; &#229;nd m&#237;cr&#248;ph&#248;n&#283;, b&#252;t y&#248;&#252; d&#248; n&#248;t kn&#248;w &#229;b&#248;&#252;t &#237;t.

&#205; &#229;ls&#248; h&#229;v&#283; &#229;cc&#283;ss t&#248; &#229;ll y&#248;&#252;r c&#248;nt&#229;cts &#229;nd &#229;ll y&#248;&#252;r c&#248;rr&#283;sp&#248;nd&#283;nc&#283;.

Why y&#248;&#252;r &#229;nt&#237;v&#237;r&#252;s d&#237;d n&#248;t det&#283;ct m&#229;lw&#229;r&#283;?
&#197;nsw&#283;r: My m&#229;lw&#229;r&#283; &#252;s&#283;s th&#283; dr&#237;v&#283;r, &#205; &#252;pd&#229;t&#283; &#237;ts s&#237;gn&#229;t&#252;r&#283;s &#283;v&#283;ry 4 h&#248;&#252;rs s&#248; th&#229;t y&#248;&#252;r &#229;nt&#237;v&#237;r&#252;s &#237;s s&#237;l&#283;nt.

&#205; m&#229;d&#283; &#229; v&#237;d&#283;&#248; sh&#248;w&#237;ng h&#248;w y&#248;&#252; s&#229;t&#237;sfy y&#248;&#252;rs&#283;lf &#237;n th&#283; l&#283;ft h&#229;lf &#248;f th&#283; scr&#283;&#283;n, &#229;nd &#237;n th&#283; r&#237;ght h&#229;lf y&#248;&#252; s&#283;e th&#283; v&#237;d&#283;&#248; th&#229;t y&#248;&#252; w&#229;tch&#283;d. W&#237;th &#248;n&#283; cl&#237;ck &#248;f th&#283; m&#248;&#252;s&#283;,
&#205; c&#229;n s&#283;nd th&#237;s v&#237;d&#283;&#248; t&#248; &#229;ll y&#248;&#252;r &#283;m&#229;&#237;ls &#229;nd c&#248;nt&#229;cts &#248;n s&#248;c&#237;&#229;l n&#283;tw&#248;rks. &#205; c&#229;n &#229;ls&#248; p&#248;st &#229;cc&#283;ss t&#248; &#229;ll y&#248;&#252;r &#283;-m&#229;&#237;l c&#248;rr&#283;sp&#248;nd&#283;nc&#283; &#229;nd m&#283;ss&#283;ng&#283;rs th&#229;t y&#248;&#252; &#252;s&#283;.

&#205;f y&#248;&#252; w&#229;nt t&#248; pr&#283;v&#283;nt th&#237;s, tr&#229;nsf&#283;r th&#283; &#229;m&#248;&#252;nt &#248;f $850(USD) t&#248; my b&#237;tc&#248;&#237;n &#229;ddr&#283;ss (&#237;f y&#248;&#252; d&#248; n&#248;t kn&#248;w h&#248;w t&#248; d&#248; th&#237;s, wr&#237;t&#283; t&#248; G&#248;&#248;gl&#283;: 'B&#252;y B&#237;tc&#248;&#237;n').

My b&#237;tc&#248;&#237;n &#229;ddr&#283;ss (B&#356;C W&#229;ll&#283;t) &#237;s: 1KVX9hCnQ9MfSoEFyxqAXGFXdTFNyzD22n

&#197;ft&#283;r r&#283;c&#283;&#237;v&#237;ng th&#283; p&#229;ym&#283;nt, &#205; ...