mountall blocks on timeout waiting for a partition, rather than supplying prompt and picking it up later

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in lvm2.

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately we can't fix it without more information. Please include the information requested at https://wiki.ubuntu.com/DebuggingLvm as separate attachments.

I have a similar problem with lvm. But I don't know if it's the same.
Sometimes (very often indeed) during boot process, my lvm volumes are mounted but empty ! Of course, I'm unable to access to my desktop.
After some boots, my volumes are good and all works fine.
dmesg not show anything.

OK, looking at this finally. I'm getting ready for my wedding, so
sorry for the slow response.

First, there is no /scripts/init-premount/udev

Second, the --verbose and --suppress-syslog options are not valid for
udevd inside of initramfs

Thanks,

Deven

On Thu, Feb 25, 2010 at 2:16 PM, Anzenketh <email address hidden> wrote:
> Thank you for taking the time to report this bug and helping to make
> Ubuntu better. Unfortunately we can't fix it without more information.
> Please include the information requested at
> https://wiki.ubuntu.com/DebuggingLvm as separate attachments.
>
> ** Changed in: lvm2 (Ubuntu)
>       Status: New => Incomplete
>
> --
> LVM Not mounting in Lucid
> https://bugs.launchpad.net/bugs/527666
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Additionally, the /sbin/udevtrigger command does not exist on Lucid.

I'm setting this from Incomplete to Confirmed. The reason is
1) what Deven said below (the instructions are somewhat not applyable to lucid)
2) We have at least two people hitting the very same problem (one of them is my person and the other one is amitk (check http://irclogs.ubuntu.com/2010/03/08/%23ubuntu-devel.html)

The way to reproduce:
1. create a VG
2. create a LV on the VG
3. Mount the LV via fstab and reboot your server (it happened on ubuntu lucid server flavour)
4. wait and see

This is a regression from karmic and should be fixed before release.

@bug triaging team: please set the correct "regression" tag (as I don't know the correct workflow)

I wonder if we should move this bug from lvm2 to initramfs, because lvm2 in general does work as a charm...it's only the boot up area.

I use the attached script to create partitions on servers as a step in my kickstart installation.

I can confirm a regression since the last LTS release. When I reboot Lucid with the new partition setup, the boot screen hangs on random partition, usually /tmp, /var or /usr.

Last tested of me on Lucid Alpha 3 in a VirtualBox setup on x86.

LVM works fine before reboot. The boot goes very fast, does it wait for proper LVM discovery?

Hmmm...
since the last dist-upgrade with new kernel, new initramfs-tools and new mountall package it works here for me...

I'm trying to reproduce it somehow, because I think there is something like a timing race condition..I'm not sure

It seemed to work fine for me last night. I created an LVM snapshot of my 9.10 root, rebooted using the snapshot as the root, then upgraded to lucid. Rebooted back into the original 9.10 root, then again into the lucid snapshot without issue.

Did a new test today.

Kickstart installation with netboot image. I used no.archive.ubuntu.com as mirror.

Installation and and the first reboot went fine. All six LVM-volumes was discovered:

/home
/opt
/tmp
/usr
/var
/var/log

All reboots after the first missed one or more LVM volumes and started to hang on random volume again. I am at a loss here. Why did it work on the first reboot, but not with subsequent reboots?

Suggestion how I should debug this would be appreciated. Or is there any sensible place to insert a delay somewhere to work around this problem?

I have to agree with Philip. On my system with just /home as an LVM
volume, the latest updates appear to have fixed my problem. Now, I
would remind you that this is Ubuntu Desktop, 64 bit.

Thanks,

Deven

On Thu, Mar 11, 2010 at 4:50 AM, Arnulf Heimsbakk
<email address hidden> wrote:
> Did a new test today.
>
> Kickstart installation with netboot image. I used no.archive.ubuntu.com
> as mirror.
>
> Installation and and the first reboot went fine. All six LVM-volumes was
> discovered:
>
> /home
> /opt
> /tmp
> /usr
> /var
> /var/log
>
> All reboots after the first missed one or more LVM volumes and started
> to hang on random volume again. I am at a loss here. Why did it work on
> the first reboot, but not with subsequent reboots?
>
> Suggestion how I should debug this would be appreciated. Or is there any
> sensible place to insert a delay somewhere to work around this problem?
>
> --
> LVM Not mounting in Lucid
> https://bugs.launchpad.net/bugs/527666
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Hi,

I can reproduce this problem every new install (ubuntu-minimal on x86).

I've been testing around. Using only two or three LVM partitions seem to work every time. No problem booting there.

Using four LVM partitions or in my case six partitions seems to be a problem. It varies on how many LVM partitions which is detected in the boot sequence. Usually four, but at some boots five and even three.

Arnulf

When you say LVM partition do you mean physical partition, LVM volume
group or LVM logical volume? I experience this problem intermittently
wtih 1 volume group and 3 logical volumes.

I have several PVs in 1 VG with several LVs, but only the /home volume is
automounted by fstab. I believe that others here are saying multiple LVs
mounted by fstab.

Deven

On Mar 11, 2010 9:10 PM, "doclist" <email address hidden> wrote:

When you say LVM partition do you mean physical partition, LVM volume
group or LVM logical volume? I experience this problem intermittently
wtih 1 volume group and 3 logical volumes.

--
LVM Not mounting in Lucid
https://bugs.launchpad.net/bugs/527666
You received this bug notifica...

I create one VG on one PV. In that VG I create six LVs.

I try to mount all six LVs by fstab:

/home
/opt
/tmp
/usr
/var
/var/log

Confirming that it 'hangs' after the first boot for multiple LVM mounts. Increasing importance and assigning to Scott.

I guess Scott will want debug output after adding --debug to the mountall command in /etc/init/mountall.conf

Picture of output with --debug is captured here:

http://share.ovi.com/media/amit.kucheria.Public/amit.kucheria.10006

I'm adding my fstab and mountall debug output. This time it hangs on /var.

The postfix .ah is my initials.

Yep, just happened to me again after a reboot to install some updates.
Here's my configuration details:

LVM2 -
root@dphillips-desktop:~# pvdisplay
  --- Physical volume ---
  PV Name /dev/sdb
  VG Name VirtualMachines
  PV Size 465.76 GiB / not usable 12.02 MiB
  Allocatable yes
  PE Size 16.00 MiB
  Total PE 29808
  Free PE 2928
  Allocated PE 26880
  PV UUID tr32vc-YYU7-97Wj-fJ2r-Q5pT-VmmY-jkdmvC

root@dphillips-desktop:~# lvdisplay
  --- Logical volume ---
  LV Name /dev/VirtualMachines/TERM15
  VG Name VirtualMachines
  LV UUID IILyg1-Jrrh-Rg8a-bA0d-TvEf-tvVK-5lPmVn
  LV Write Access read/write
  LV Status available
  # open 0
  LV Size 30.00 GiB
  Current LE 1920
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:0

  --- Logical volume ---
  LV Name /dev/VirtualMachines/WINXP
  VG Name VirtualMachines
  LV UUID Jvrlrc-F3DD-gkcA-9vKd-LY8f-QXi3-Lqj5Vf
  LV Write Access read/write
  LV Status available
  # open 0
  LV Size 40.00 GiB
  Current LE 2560
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:1

  --- Logical volume ---
  LV Name /dev/VirtualMachines/CentOS
  VG Name VirtualMachines
  LV UUID VXRec0-xRzp-ndR7-GsSW-nYPn-mzpu-B4eRqM
  LV Write Access read/write
  LV Status available
  # open 0
  LV Size 30.00 GiB
  Current LE 1920
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:2

  --- Logical volume ---
  LV Name /dev/VirtualMachines/BackUp
  VG Name VirtualMachines
  LV UUID eBW0Na-tTTt-88Nx-711s-4S7m-6ihi-2NeX3c
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 100.00 GiB
  Current LE 6400
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:3

  --- Logical volume ---
  LV Name /dev/VirtualMachines/Music
  VG Name VirtualMachines
  LV UUID rhk9FS-kJ8a-rtCm-9gqS-tMMA-uu1n-b2MRb3
  LV Write Access read/write
  LV Status available
  # open 0
  LV Size 200.00 GiB
  Current LE 12800
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 252:4

  --- Logical volume ---
  LV Name /dev/VirtualMachines/Temp
  VG Name VirtualMachines
  LV UUID u4CeIR-FuXN-NYuz...

Should this bug be filed under package mountall? It doesn't seem to be a lvm-bug.

I think also this bug is related to mountall.
if I press S in the boot sequence, I have my gdm login. Switching to VT1 an logging as root, I can mount fine my LV.

This bug does not seem to be directly related to lvm, but rather to the mountall command.

Should the LVs be owned by root:root or root:disk?

I have 3 LVs on the new disk - Home, Private and Shared.
I found that all the LVs in /dev/mapper are owned by root:root except for Private and Shared that are owned by root:disk. And these are the two that are not mounted and cause the wait messages. Will go through the udev logs as I find time.

I have same bug. Install fresh Lucid beta 2 and add /data lvm/reiserfs mountpoint to fstab. System is often not start. Sometimes i press reset, if "Waiting for /data [SM]" is shown, and sytem on next boot start disc check and booted to gdm.

If I press enter on message "Waiting...", often start maintenance shell. After running mount -a, all mountpoint successfully mounted.

Logs from two boots by
/etc/init/mountall.conf
    exec mountall --debug --daemon $force_fsck $fsck_fix >/dev/mountall-stdout.log 2>/dev/mountall-stderr.log

I have created 3 logical volumes on a removable USB drive - one volume group of one physical partition. If I have fstab entries for these LV's the system will start to boot then give me a "Waiting ..." message on the first LV if the drive has not been connected; enter gives a maintenance shell. If the drive is connected at startup/restart, bootup is not a problem. This problem does not occur under 9.04.

Please amend last line to read 9.10 not 9.04.

I've tried but have been unable to reproduce this. I'm not entirely sure that my environment is equivalent though, so let me explain what I did and if you have suggestions for other things to try, I can give it a shot.

I created a brand new kvm vm x86_64 w/ a 40G disk, 512MB. I grabbed the lucid-beta1 64bit server iso and did a fresh install. When it came time to partition the disk, I created one VG on the PV. I created 6 LVs on the VG:

root -> /
home -> /home
opt -> /opt
tmp -> /tmp
var -> /var
varlog -> /var/log

with various sizes ranging from about 5G to 10G apiece. Everything installed and booted perfectly fine. No hang, all filesystems mounted correctly. In fact, boot was so blazingly fast I blinked and it was done.

I updated all packages and rebooted about 10 times. I never had a hang or failure to mount any partitions. Boot never took longer than a second or two. I added --debug to mountall as in orgoj's comment #26 and mountall-stderr.log was never anything but empty. mountall-stdout.log didn't have any indications of problems (on the contrary, it looked quite reasonable).

Is this a reasonable test of the reported issue? Is there anything else I can try to get a better reproduction of the bug?

I can't say, but I would suggest trying it without using VMs.

Deven

On Fri, Mar 26, 2010 at 4:26 PM, Barry Warsaw <email address hidden> wrote:
> I've tried but have been unable to reproduce this.  I'm not entirely
> sure that my environment is equivalent though, so let me explain what I
> did and if you have suggestions for other things to try, I can give it a
> shot.
>
> I created a brand new kvm vm x86_64 w/ a 40G disk, 512MB.  I grabbed the
> lucid-beta1 64bit server iso and did a fresh install.  When it came time
> to partition the disk, I created one VG on the PV.  I created 6 LVs on
> the VG:
>
> root -> /
> home -> /home
> opt -> /opt
> tmp -> /tmp
> var -> /var
> varlog -> /var/log
>
> with various sizes ranging from about 5G to 10G apiece.  Everything
> installed and booted perfectly fine.  No hang, all filesystems mounted
> correctly.  In fact, boot was so blazingly fast I blinked and it was
> done.
>
> I updated all packages and rebooted about 10 times.  I never had a hang
> or failure to mount any partitions.  Boot never took longer than a
> second or two.  I added --debug to mountall as in orgoj's comment #26
> and mountall-stderr.log was never anything but empty.  mountall-
> stdout.log didn't have any indications of problems (on the contrary, it
> looked quite reasonable).
>
> Is this a reasonable test of the reported issue?  Is there anything else
> I can try to get a better reproduction of the bug?
>
> --
> multiple LVM volumes not mounted in Lucid
> https://bugs.launchpad.net/bugs/527666
> You received this bug notification because you are a direct subscriber
> of the bug.
>

@Deven: yeah, unfortunately i haven't got any free hardware laying about ;). I'll have to see if I can cobble something together.

I can confirm this. I have one lv which is owned by root:disk, when this lv is present in /etc/fstab, the system refuses to boot, it keeps waiting for this lv to be mounted. If I remove this lv from /etc/fstab, it boots. Unfortunately, I cannot for the life of me find where these permissions are stored! Why does just this one lv have a different group?

Okay, I'm going to dig up some physical hardware to see if I can reproduce this. I've had no luck reproducing it in VMs, even with a layout suggested by someone in IRC.

I make fresh install Ubuntu 10.4 i386 beta1, manual partition whole disk to:
sda1 /boot ext2 256MB
sda2 swap 2GB
sda3 / ext3 12GB
sda5 lvm main 'rest of disk'
/data/main/home /home reiserfs 40GB

After restart and reboot system hang on "Wait for /home [SM]".

@orgoj: interesting. does the same thing happen if you use ext4 instead of reiserfs?

It happens on my machine, and I'm using ext4.

On Fri, Mar 26, 2010 at 9:45 PM, Barry Warsaw <email address hidden> wrote:
> @orgoj: interesting.  does the same thing happen if you use ext4 instead
> of reiserfs?
>
> --
> multiple LVM volumes not mounted in Lucid
> https://bugs.launchpad.net/bugs/527666
> You received this bug notification because you are a direct subscriber
> of the bug.
>

I noticed the message on boot in both cases if system boot or hang:

udevd-work[70]: inotify_aff_watch(6, /dev/sdb1, 10) failed: No such file or directory

/dev/sdb1 is second part of my lvm storage on main PC.

Logs from mountall if system successfully booted.

I try the same as reiserfs but with ext4 for / and /home(lvm).

System hang on boot, but no "Wait for.." is shown. After pressing M, console is shown. Mount don't show mounted /home. Mount -a mount /home without errors. After Ctrl-D system successfully mounted.

I can confirm that the mount issue still exists. I'm adding debug from mountall and /var/log/udev. Note. I used [s] to skip waiting for /var and /var/log. My test setup is as follows.

lvdisplay:

  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  homelv rootvg -wi-ao 1.00g
  loglv rootvg -wi-a- 2.00g
  optlv rootvg -wi-ao 1.00g
  tmplv rootvg -wi-ao 2.00g
  varlv rootvg -wi-a- 2.00g

fstab:

proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=d5e5232c-0357-454e-8cb4-2fca244f4f42 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=1b4eae89-00cd-42d3-a9a3-0e08c885a1f1 none swap sw 0 0
UUID=414e0c6d-29b0-446f-bde2-6540af2c73ae /home ext4 defaults 01
UUID=816b3834-02a7-4cb1-b591-cab186c2502d /opt ext4 defaults 01
UUID=35f9fad7-e5a0-48d7-8b29-50c9fd096054 /tmp ext4 defaults 01
UUID=40767f37-c576-42f6-82ab-80aa31f85617 /var ext4 defaults 01
UUID=e1ae56e7-0cd0-42bf-9079-690c5711461d /var/log ext4 defaults0 1

I have also discovered, that if I move /usr to an lvm partition then I get the message

error: file not found.

on my console when/right after the kernel boots. I have no idea where that comes from or if it is related to this problem. Tips for debugging is appreciated.

Arnulf

My /var/log/udev

Can the status of this bug be changed from "Fix Released" to "Confirmed" since it is still an issue?

Arnulf

@arnulf: done

I see 2 issues :
- mountall stop working while user is prompted to press S/M (corrected in mountall 2.10, cf #58)
- the latest devices created in /dev/mapper is owned by root.disk instead of root.root

On my machine, "mountall --version" returns "2.8" while "apt-cache show mountall" returns "2.10" (and apt-get tells my I already have the latest version). Maybe this 2.10 package contains a mistake ?

Waiting for the dm device to be mounted is pointless, only changing permissions on /dev/mapper/logicalvolume can make Lucid to boot correctly. This is easy to do, but not really clean.

thamieu

On Tue, 2010-04-06 at 11:47 +0000, Arnulf Heimsbakk wrote:

> Can the status of this bug be changed from "Fix Released" to "Confirmed"
> since it is still an issue?
>
No.

If you are still having issues, you must have had a different bug to the
original reporter all along.

Please open a new bug.

Scott
--
Scott James Remnant
<email address hidden>

Hi,

I'm having the same problem as orgoj and some of the others on this bug. Did someone create a new bug for this, possibly different, bug which looks like this one?

Thanks,
Tim

similar to tamieu. But my /dev/mapper contains only one file - control :((( Should I open new bug as well?

An 'grep swap' extract from /var/log/boot.log with mountall --debug:

local 6/6 remote 0/0 virtual 11/11 swap 0/1
try_mount: /dev/mapper/lucid-swap_1 waiting for device
try_udev_device: block /dev/mapper/lucid-swap_1 e26b43b0-7782-44da-9a8f-78c7955e3c74 (null)
try_udev_device: /dev/mapper/lucid-swap_1 by name
run_fsck: /dev/mapper/lucid-swap_1: no check required
activating /dev/mapper/lucid-swap_1
spawn: swapon /dev/mapper/lucid-swap_1
spawn: swapon /dev/mapper/lucid-swap_1 [1021]
swapon: /dev/mapper/lucid-swap_1: swapon failed: Device or resource busy
mountall: swapon /dev/mapper/lucid-swap_1 [1021] terminated with status 255
mountall: Problem activating swap: /dev/mapper/lucid-swap_1
mounted: /dev/mapper/lucid-swap_1
swap finished
local 6/6 remote 0/0 virtual 11/11 swap 1/1

Just a guess here... If each of the mountall discovered FSs are mounted in the background by a spawned process (assumed from the logging) then as /home is generally the largest mount on a default install and is going to take the longest, could it be possible that it just happens that as the swap mount has failed all but the /home has mounted ok but mountall has given up waiting due to the failure and killed off the spawned mounts?

On Wed, 2010-04-07 at 16:53 +0000, Tim Jones wrote:

> I'm having the same problem as orgoj and some of the others on this bug.
> Did someone create a new bug for this, possibly different, bug which
> looks like this one?
>
If you could each create a new one using "ubuntu-bug mountall", I would
really appreciate that.

It's quite possible that you each have a different problem at this
point.

Scott
--
Scott James Remnant
<email address hidden>

I would also ask that everyone post back here with the new bug numbers so that I and others will be able to track the trail to the other bugs should we land here.

Thanks

I opened a new bug about the ownership issue : #557909.

I have my bug related to "lost" lvs/vg: #554478

Having this same issue, mountall version 2.11

I try same fresh install from 10.4b2 alternative i386 with home reiserfs on LVM as before and 3 next reboots are ok.

On main system from version 2.11 mostly boot and from version 2.12 I not see hang on boot.

After update and install nvidia-96 driver, system hang on boot with same symptom. On maintenance console I see home not mounted, mount -a forking fine and after Ctrl-D system boot. Gdm after login hang and restart.

Having this this same issue, mountall 2.12. Trying to debug it. Seems like an 'add/change' event is not getting to mountall from udev, as symlinks in /dev/vg are being created...

Further to above:

There is still a race condition in mountall, proabably due to teh integration with plymouth boot screen.

Add/change events from udev are being dropped.

When I get the error, I press 'M', and sulogin. The links are there in /dev/<volume_group>, so I believe that udev is getting its job done, and letting mountall know about it.

two things should be done:

1) add code to try 2 mount attempts before giving up on a file system in /etc/fsta on boot.
2) Find the race and fix it.

1) is the belt and braces - not mounting file systems on boot is a SERIOUS problem.

Condition can be debugged when a machines running by creating a volume group with about 10 logical volumes, deactivating it with 'vgchange -a n /dev/<volume_group>, and then activating it. mountall would have to be running in a separate window, not issuing events to upstart init.

Still having the same problem as well, pulled down the latest packages for install this morning. Attaching screenshot, fstab, mount after boot, and LVM displays.

lsb_release -rd
Description: Ubuntu lucid (development branch)
Release: 10.04

apt-cache policy mountall
mountall:
  Installed: 2.13
  Candidate: 2.13
  Version table:
 *** 2.13 0
        500 http://apt-dev/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

I am experiencing exactly the same problem since I upgraded to Lucid yesterday. I wonder if this is a udev problem, because I've also discovered an issue with the permissions of /dev/shm.

Since, the upgrade to Lucid, boots hangup indefinitely. I have to execute the following procedure to get to the GDM screen:
1. Enter M (for Manual Recovery)
2. Execute "mount -a" which mounts all filesystems on LVM without problems.
3. CTRL-D to close the shell and continue with the reboot.

After login to Gnome Session, launching Google Chrome fails, because /dev/shm has permissions rw-r--r-t. Google Chrome starts working normally after setting /dev/shm permissions manually to rw-rw-rwt, but the permissions do not survive a reboot.

Works for me now !

- plymouth 0.8.2-2
- udev 151-12
- mountall 2.13

Something is clearly resetting the permissions of /dev/shm - I don't think it'll be udev, udev would have removed the "t" as well

Yesterday, I had posted a comment with details of the issue I am experiencing since I upgraded to Lucid.

Lucid hangs up indefinitely with the "Waiting for 'some partition'" error. The partitions that cause the problem are on LVs. Amit Kucheria mentioned that at this point some of the LVs have root:root ownership whereas others have root:disk ownership, and apparently the LVs that hang are the ones with root:disk ownership.

Simply changing the owner ship of the device node in /dev/mapper is a no fix, because the permissions are not persistent through reboots. So, I went ahead and added the following line in mountall.conf before the line that launches the daemon with exec:

chown root:root /dev/mapper/*

Adding this line fixed the problem completely. This test seems to confirm that the problem is with the ownership of the LVM device nodes, but I have no idea why some nodes end up having the root:disk ownership, while others have root:root in the first place.

Same here, but sometimes everything works (about half the time):

I have /home under LVM:
/dev/cassini/cassini-home /home ext4 errors=remount-ro 0 1

The boot process (sometimes) hangs with the following message:
The disk drive for /home is not ready yet or not present.
Continue to wait; or Press S to skip mounting or M for manual recovery

I press M
# mount /home
# exit

and then the boot proceeds. See my mountall logs at comment 3.

Sorry, I meant at comment https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/561390/comments/3

Bug 561390 tracks this specific issue, it could be marked a duplicate if that bug was reopened instead, depending on where Scott prefers to track the issue.

With following line in mountall.conf to fix permissions for LVs, everything seems to work fine:
chown root:root /dev/mapper/*

But, I've discovered that on battery power things get even worse. The boot seems to hangup about the same place, but I cannot obtain a recovery shell and I have found no way to get a running system. I am not sure if this is another bug somewhere else or the bug is related.

With the change to fix de ownership issue of DM device nodes, things seemed to be working, but then I started having problems again today even with mains power. I second Thierry Carrez that the boot fails about half the time. In fact the things have become worse, because sometimes pressing M for manual recovery does not work and the only way to get the system to boot is to reboot over and over again to get a working session.

As far as I can gather Lucid boot process is failing completely for many people that have multiple filesystems on LVM. I've been using Ubuntu since with LVM since 7.04, and all the upgrades up to 9.10 worked without problems. Judging by the comments of other the problem is not limited to upgrades either, This bug really seems to be a show-stopper, because a system that was working perfectly, does not even get to a login prompt with Lucid.

I will be glad to help to identify a solution, but I do not know how.

Seems like the problem does not occur consistently with every possible setup, because I have a another laptop with 8 LVs that I upgraded to Lucid yesterday and it has been booting without problems; just the usual error messages for statd and ureadahead startup for having /var on a separate partition. I've uninstalled ureadahead for fixing the error messages with ureadahead which apparently does not work with /var n a separate partition, but the statd error messages are still there.

On my machine (two disks, no RAID, LVM), the boot process still hangs in about 1 out of 5 boots. Strg+Alt+Del to reboot almost always works. (Lucid with all updates as of yesterday, 64 bit.)

We still experience this bug on lucid today. Bootchart shows that the boot stalls on mountall.

The system boots in some cases, so it is possible to compare OK and KO cases. Attached are:

* The two bootcharts (OK and KO) ;
* The two mountall --debug logs (OK and KO);
* Our /etc/fstab

The bootcharts clearly show that mountall is the process blocking the boot with LVM (until we press 'S' or 'M'). In the KO case, the mountall debug logs read:

  Received SIGUSR1 (network device up)
  try_mount: /WOO waiting for device

which seems to be blocking all the depending mounts.

Sorry I could only add one attachment, so here is a tarball with all the attachments listed in the previous comment.

Upon debugging further, it seems mountall is waiting for /dev/HEBEX/WOO_BASE, and rightly so. After skipping with 'S' during boot, here is what we see:

root@malorent:~# lvscan
  ACTIVE '/dev/HEBEX/WOO_BASE' [1.00 GiB] inherit
  ACTIVE '/dev/HEBEX/WOO_LOG' [1.00 GiB] inherit
  ACTIVE '/dev/HEBEX/WOO' [5.00 GiB] inherit
  ACTIVE '/dev/HEBEX/WOO_PROG' [2.00 GiB] inherit
  ACTIVE '/dev/HEBEX/VAR_LOG' [3.00 GiB] inherit

root@malorent:~# ls -l /dev/mapper/
total 0
brw-rw---- 1 root disk 251, 4 Apr 23 14:57 HEBEX-VAR_LOG
brw-rw---- 1 root disk 251, 2 Apr 23 14:57 HEBEX-WOO
brw-rw---- 1 root disk 251, 1 Apr 23 14:57 HEBEX-WOO_LOG
brw-rw---- 1 root disk 251, 3 Apr 23 14:57 HEBEX-WOO_PROG
crw-rw---- 1 root root 10, 59 Apr 23 14:57 control

root@malorent:~# ls -l /dev/HEBEX/
total 0
lrwxrwxrwx 1 root root 23 Apr 23 14:57 VAR_LOG -> ../mapper/HEBEX-VAR_LOG
lrwxrwxrwx 1 root root 19 Apr 23 14:57 WOO -> ../mapper/HEBEX-WOO
lrwxrwxrwx 1 root root 23 Apr 23 14:57 WOO_LOG -> ../mapper/HEBEX-WOO_LOG
lrwxrwxrwx 1 root root 24 Apr 23 14:57 WOO_PROG -> ../mapper/HEBEX-WOO_PROG

root@malorent:~# lvdisplay /dev/HEBEX/WOO_BASE
  /dev/mapper/HEBEX-WOO_BASE: open failed: No such file or directory
  /dev/mapper/HEBEX-WOO_BASE: open failed: No such file or directory
  --- Logical volume ---
  LV Name /dev/HEBEX/WOO_BASE
  VG Name HEBEX
  LV UUID 1an8Zg-aYse-AkF3-NGs2-0BGF-4mfO-1m3wUA
  LV Write Access read/write
  LV Status NOT available
  LV Size 1.00 GiB
  Current LE 256
  Segments 1
  Allocation inherit
  Read ahead sectors auto

So LVM finds the missing device internally, but the device is not created by udev.

Update: the /dev nodes seem to only be missing when two LVM partitions fail.

On Fri, 2010-04-23 at 09:49 +0000, Mathieu Alorent wrote:

> We still experience this bug on lucid today.
>
No, this bug has been fixed. You are experiencing a different bug, I'd
appreciate it if you could open a new bug with "ubuntu-bug mountall"
which will some of the information we need from you.

Scott
--
Scott James Remnant
<email address hidden>

Mathieu: actually, after reviewing the data you did attach, it's a high
probability you're experiencing bug #561390

Scott
--
Scott James Remnant
<email address hidden>

Hi Scott,

The bugs 561390 and 527666 seem to be pointing to the very same issue to me. Infact, I was quite tempted to mark them as duplicates:
* In both cases the same error message is displayed and the only way to continue with the boot process is to enter the Recovery Shell and mount the missing partitions manually.
* The filesystems that do not get mounted are on LVM.
* There is usually something wrong with the permissions of /dev/mapper devices and /dev/shm, when the problem occurs.
* Both bugs seem to point to a critical regression in Lucid; the partition setup that was working perfectly with karmic, causes problems after Lucid upgrade.

Why do you think the two bugs refer to separate issues? In what way do the two bugs differ? How can I identify exactly which issue I am experiencing?

On Sat, 2010-04-24 at 17:42 +0000, Ali Onur Uyar wrote:

> The bugs 561390 and 527666 seem to be pointing to the very same issue to me.
>
They are not.

527666 (this bug) describes an issue where mountall simply doesn't wait
long enough for block devices to appear that *do* appear.

561390 describes an issue where mountall never receives notification of
LVM devices from the kernel.

> Infact, I was quite tempted to mark them as duplicates:
>
Do not.

> Why do you think the two bugs refer to separate issues? In what way do
> the two bugs differ? How can I identify exactly which issue I am
> experiencing?
>
Since this bug (mountall doesn't wait long enough) has been fixed, if
you are experiencing issues you are either experiencing bug 561390
(which has not been marked Fix Released) or a different bug entirely.

It's always best to just file a new bug describing your own problems,
and allow the developers to triage that bug and determine themselves
whether it's a duplicate of a known problem or a new problem not
previously known.

Scott
--
Scott James Remnant
<email address hidden>

Thanks Scott, for the detailed explanation. Even though I've been using Ubuntu for the last few years, I am quite new to launchpad.

On Mon, 2010-04-26 at 00:01 +0000, Ali Onur Uyar wrote:

> Thanks Scott, for the detailed explanation. Even though I've been using
> Ubuntu for the last few years, I am quite new to launchpad.
>
It's not really a Launchpad thing.

The confusion becomes because there's a tendancy for users to classify
bugs by their symptoms ("black screen", "big loud noise", etc.) -
whereas developers classify bugs by the cause.

While these two bugs have the same apparent symptom, the cause is
actually quite different.

In fact, I'm now convinced there are *three* bugs; two of which have
been fixed. You have the third.

Scott
--
Scott James Remnant
<email address hidden>

For those still experiencing problems, and not yet subscribed to bug #561390, in my PPA you'll find a new dmsetup package, could you try it out and see whether it makes things better or worse?

  sudo add-apt-repository ppa:scott/ppa
  sudo apt-get update
  sudo apt-get upgrade

Check you have dmsetup 2.02.54-1ubuntu4~ppa1

  dpkg-query -W dmsetup

Then reboot.