[Karmic, security] Encrypted /tmp no longer mounting after upgrade to karmic

Hmmm... Scott assigned this bug to the cryptsetup package, but I still believe it's an upstart / initscripts issue.
Cryptsetup does its job good, but I believe there is an order or synchronization issue with the way the initscripts or Upstart call cryptsetup / fsck / mount of encrypted filesystems (i.e. try to mount them before activating the encrypted volumes).

De-duplicated this bug, affected to "mountall", updated description.

The interaction issue between mountall and cryptsetup has been described in detail in bug #475936 and some cryptsetup fixes have been attempted, that don't fix the race condition with mountall.

mountall logs are available in comments 19 and 20 of bug #475936

Bug purposely affected to mountall, please do not reaffect this bug to cryptsetup - There's already a cryptsetup bug discussing this issue (#475936), but this issue (most probably) needs to be fixed mainly in mountall, not cryptsetup (unless the contrary can be demonstrated otherwise than silently reaffecting this bug back to cryptsetup...)

There is no issue with mountall; the problem is that cryptsetup is not decrypting the disks - as soon as the block devices listed in fstab (decrypted cryptsetup disks) appear, mountall will mount them just fine

Negative, Scott. I have tested this thouroughly and an encrypted /tmp IS PROPERLY configured and formated by cryptsetup, BUT NOT MOUNTED by mountall.

If I just type "mount -a" once the system is finished booting, the encrypted /tmp will actually mount. Everytime.

So it's not an issue with "cryptsetup not decrypting", it's an issue with mountall not waiting for cryptsetup having made the device ready before giving up in attempting to mount it.

mountall never "gives up" attempting to mount devices, as soon as udev notifies mountall that the device is ready, it is mounted

This bug was reassigned to mountall because of the following referenced comment on bug #475936:

> Just confirmed that using the test case in lucid, the /tmp directory fails to be mounted. However, this is because mountall
> treats the /tmp directory specially. Varying the /etc/fstab line in either of two ways is enough to fix this:

> - set the 'pass' option to non-zero
> - mount it somewhere other than /tmp
>
> After making either change, mountall successfully mounts the device for me.

> I've updated the test case using the first of these options.

> (If you believe the mountall behavior is wrong, please file a separate bug report on the mountall package for this.)

That's not a bug in cryptsetup.

If you replace the encrypted /tmp mountpoint with an unencrypted mountpoint, does it work?

Perhaps a silly question, but at the point that /dev/mapper/c_tmp is announced to the system - does it have a filesystem on it?

If cryptsetup doesn't call mke2fs until after creating the dm device with that name, then udev is told about the block device before the filesystem is created, so blkid returns no filesystem, so mountall will ignore it (if it didn't ignore it - it would fail to mount anyway).

(The two suggested workarounds would introduce enough of a delay that it might get away with it.)

This still leaves it as a cryptsetup bug, for announcing a block device without a filesystem and not following up with a "change" event when the block device is ready.

Answer to question in comment 8 : Yes

Answer to comment 9 : As far as I understood, the latest cryptsetup fix (currently in "karmic-proposed), which I have installed on my system (and that didn't help), is for fixing the issue you discuss. It creates the encrypted temp under a temporary device name, mke2fs it, then renames it (using dm_rename or something like that I believe) to its final name.So when the device "appears in the system" under its final name, it supposedly has the filesystem on it.

You say "mountall nevers gives up", so well... I have hard times understanding why the filesystem will mount if I type "mount -a" which will mount all fstab defined filesystems, altough mountall didn't mount it... Unless of course it didn't receive the notification that the device was present... Then wouldn't be "creating it under a temporary name, mke2fs it, then rename it" a good solution ?

Whether or not cryptsetup is to blame, I believe we have something wrong with mountall / upstart anyway, for in the current situation the system will startup with a *mandatory* filesystem unmounted, then daemons start and spit their temp files and sockets into the root filesystem, which is definitely no good.

If for some reason a mandatory FS that mountall *must* mount cannot be mounted, this should be a showstopper for system startup...

Also one thing that made me file this bug under mountall is that cryptsetup-encrypted /tmp has worked for me for many previous Ubuntu version and suddently fails after upgrading to Karmic.

I don't believe that cryptsetup was much changed between Jaunty and Karmic, while the system startup sequence and the way filesystems are mounted has been deeply modified.

Observing that this (from a sysadmin PoV) results in a regression - something that used to work doesn't anymore - it looks quite obvious to me that the cause of the regression relates to what has been modified in a major way, not to what stayed the same but the overall result is that it doesn't work anymore...

And also, I could hardly consider that starting up a system with some fstab-defined filesystems unmounted is not a "bug"... of a serious kind, again from a sysadmin PoV.

Attached is a mountall --debug log corresponding to the following setup in karmic, using cryptsetup 2:1.0.6+20090405.svn49-1ubuntu7.1:

$ grep crypttmp /etc/fstab
/dev/mapper/crypttmp /tmp ext2 defaults 0 0
$ grep crypttmp /etc/crypttab
crypttmp /dev/sda8 /dev/urandom tmp
$

Key lines are:
try_mount: /tmp waiting for device /dev/mapper/crypttmp
update_mount: /tmp: /dev/mapper/crypttmp_unformatted ext2 defaults
mounted: /tmp

But at the end, mountall has exited and /tmp is *not* mounted.

So there are two issues here.
1) mountall thinks /tmp is mounted when it isn't. That looks like a mountall bug to me, and I think should be tracked in this bug report.
2) Now instead of seeing the device before it has a filesystem on it, mountall sees the device before the device node is under its final name. This indicates the fix for bug #475936 is incomplete, so I'll mark that bug as verification-failed for SRU. Unfortunately, I'm not sure how to fix this; if 'dmsetup rename' isn't a reliable way to expose the device to mountall under the new name, I don't know what will be.

Scott, if you can provide any insight into a correct means of notifying mountall that the filesystem is created and ready for mounting, I would appreciate it.

I think that Scott James Remnant has hit the nail on the head in comment #9. dm-crypt creates encrypted block devices, which are transparent to the system. So what happens is something like this:

1. dm-crypt creates an encrypted block device, and mounts it to /dev/mapper/ctmp.
2. dm-crypt then runs mkfs.ext2 on /dev/mapper/ctmp.
3. mountall wakes up and sees the device it has been waiting for exits, and tries to mount it. Either step 2 hasn't started yet or hasn't finished yet, and so the mount fails.

I get the following output when my system boots, which makes me believe the above is the case:

* ctmp (starting)
mount: /dev/mapper/ctmp already mounted or /tmp busy
mountall: mount /tmp [1004] terminated with status 32
mountall: Filesystem could not be mounted: /tmp
init: mountall main process (488) terminated with status 4
Mount of root filesystem failed.
A maintenance shell will now be started.
CONTROL-D will terminate this shell and reboot the system.
* ctmp (started)... [ OK ]

@Creature : I feel you may have posted your comment without having read the comments past #9 ...

Actually this log is saying exactly what is happening - cryptsetup is *mounting* /dev/mapper/crypttmp_unformatted on /tmp - mountall get notification of that via /proc/self/mountinfo and that's why you see:

mountinfo_watcher: mountinfo changed, reparsing
parse_mountinfo_file: updating mounts
  :
update_mount: /tmp: /dev/mapper/crypttmp_unformatted ext2 defaults
mounted: /tmp