hald crashed with SIGSEGV in hotplug_event_begin_add_blockdev when assembling mdraid devices

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please try to obtain a backtrace following the instructions at http://wiki.ubuntu.com/DebuggingProgramCrash and upload the backtrace (as an attachment) to the bug report. This will greatly help us in tracking down your problem.

Thank you for your work.

I am seeing the same segfault when hald probes my md raid1 devices.

Thanks. It would be useful also for you to be able to run "sudo hald --verbose=yes --daemon=no 2>&1 | tee ~/hald.log", then recreate the steps to trigger the crash and attach the log here.

Thanks

Would you also mind trying the build of HAL from my PPA [1]?

[1] - https://launchpad.net/~chrisccoulson/+archive/ppa

Please find attached the requested log

Your build seems to work.

Thanks. I don't know if my patch is the right way to fix it. With my build, could you also please run "lshal > lshal.log" after assembling the raid volume, and attaching "lshal.log" to the bug report. Once that is done, I will send all this upstream.

Thanks

Your patch is working for me.

I'll leave it to the original poster to post his lshal.log unless you'd like mine as well.

This bug only manifested on 2.6.28-11 - booting 2.6.27-11 on Jaunty beta was fine.

If you can provide the output, then it would be appreciated (from both kernels).

Please find the log attached.

Thanks Noumayos. That was before you assembled your raid array though wasn't it?

The result of lshal is the same before and after the raid array.

Would you mind running "lshal -m", assembling your array and then posting any output?

Thanks

I have no output when assembling my array.

Here's my lshal.log. As I said, your updated package is working for me (RAID1)

I hope this helps.

Works for me too. Good work, many thanks! :)

Hi,

I upgraded today from Intrepid to Jaunty, and had the same problem with no mouse/keyboard in X, and a message in system log about a segfault with HAL. I am using software raid.

I downloaded and installed the package supplied by Chris and it works. I hope this patch will be applied to the official package as well.

Thanks for the help!

Also used the hald supplied by Chris and the keyboard and mouse work again.

One more: I also used the hald supplied by Chris and it fixed it. Same symptom - md's causing hal to crash.

I can confirm this when using a RAID 10 array. Chris's HAL builds also fixed the problem for me.

Chris's HAL builds helped for me too. I think that the bug status should be 'In progress' instead of 'Incomplete'.

How is it possible that this bug is only listed as "Medium" importance?!?!?

I discovered what I think is this bug when I upgraded my system using system update. Upon a later reboot I found that I had no access to the system whatsoever. After trying multiple things to get back in from the root recovery console, I re-formatted, losing a lot of configuration.

Fortunately, I am somewhat tech savvy and I didn't lose everything, but this cost me 2 days and will probably cost more. And for the average user, this could equate to complete data loss. I would think this would be a high importance bug since it disables a working system!

Does anyone agree with me, or am I missing something here?

--mobrien118

FYI, it looks like Chris's PPA packages fixed it, though.

Thank goodness! If I had been remote from this machine (which I will be for the next few months) it would have been a nightmare!

Mobrien118, I agree that there should be some consideration of its importance. Maybe it isn't that big of a deal since few people have raid sets on their computers. It cost me several hours of rebooting and googling before I thought to boot into recovery console, install openssh and then log in with another machine to look at the dmsg output. I considered myself fortunate to have stumbled upon the solution here since the failure case is very confusing. If hald is so critical maybe there should be a better way of reporting these errors to the user.

I've not had much time to look any further at this, and I haven't proposed my patch as a fix yet because I don't know if it is the right way to fix it. What I need to do really is send this bug report upstream, and also have a play around with a mdraid setup myself, but I don't have a clue how to set one of those up.

Perhaps someone here could help me set one up ;)

I think I can get you started.

I mean, the first step to creating a RAID volume is to think it out. Where do you need redundancy? Where do you need speed? Then map out your partitions (especially if you have different sized disks).

Remember that RAID will cause a slight (in the case of RAID0 or RAID1) to slightly greater (RAID5 or RAID6) processor and I/O load. That is the trade-off for getting better overall disk performance.

Although it is supposedly not needed to RAID0 your swap partitions across disks (supposedly the swap daemon manages multiple disks very well) it doesn't hurt to do so and is an easy and safe way to get started with RAID. You might consider making this your test case.

This page lays out mdadm and Linux RAID pretty well: http://ubuntuforums.org/showthread.php?t=408461

Basically:
1. format disks you want to use as "RAID" partitions
2. create a RAID array using "mdadm --create /dev/md0 --level=[level] --raid-devices=[number of devices] [device1] [device2]...[deviceN]
3. assemble the array
4. format the array in the filesystem of your choice (like any other partition)
5. mount it like you would any other disk partition

Pretty simple, eh?

Thanks for your help mobrien118 and Russell (who also contacted me privately with some help). I've managed to recreate the crash now.

Now I understand it properly, I've re-written the patch, tested it and sent upstream.

To the main Ubuntu repos or to your PPA?

The patch is in my bzr branch, waiting to be merged in to the ubuntu-core-dev branch. I've also sent the patch to the upstream freedesktop bug tracker: https://bugs.freedesktop.org/show_bug.cgi?id=21603

Also confirmed, problem appeared after creating mdadm raidset and reboot.

Chris' build fixed my problem as well
Thanks!

Adding my voice to the chorus - same problem.

Haven't tried the patch yet (I'm away from the box right now), but will ASAP. I agree that there should be consideration of increasing importance if that's still appropriate.

I committed the fix upstream, thanks Chris!

This bug was fixed in the package hal - 0.5.12+git20090512-0ubuntu2

---------------
hal (0.5.12+git20090512-0ubuntu2) karmic; urgency=low

  * debian/patches/50_no_crash_on_md_blockdev.patch:
    - When adding a block device, don't assume that the parent
      has storage capability. This fixes a crash where the device
      is re-parented to the root computer device object (such as
      with mdraid devices). LP: #361689.

 -- Chris Coulson <email address hidden> Fri, 15 May 2009 18:34:58 +0200

So... may I suggest the new hald build be uploaded to the jaunty repository?

I've spent the last 3 hours debugging what seemed to be a DBus problem, but turned out to be this. Then I needed to pull the fix from karmic because the fix is not available in the jaunty repo.

Update seems to be in repository already, waiting for the index to be updated.
These builds can be downloaded @
http://nl3.archive.ubuntu.com/ubuntu/pool/main/h/hal/hal_0.5.12+git20090512-0ubuntu2_i386.deb
http://nl3.archive.ubuntu.com/ubuntu/pool/main/h/hal/hal_0.5.12+git20090512-0ubuntu2_amd64.deb

excuse my ignorance, but do we have any idea when that fix will be available when I update my system?
shouldn't the severity be high as it breaks any installation that is using raid afaik? shouldn't that issue be listed in the release note (i would not have upgraded if I had known..).

I've updated on saturday from 8.10 and now nothing is working and I am not too keen on applying a temporary fix, knowing that it has few depencies (libblkid1). the alternative would be to re-install 8.10 unless I am told a fix would be available in the next few days.

many thanks for your work and for your answer.
eric

Same deal here, except that the fixed hal .deb depends on version 2.15 of libblkid1 which is not available, at least on amd64, in the repositories either.

My system is now working (after some painful googling with links) after applying the following:

http://launchpadlibrarian.net/26631965/libblkid1_2.15-1ubuntu2_amd64.deb

Then applying Chris's patch from above.

Gigthanks, Chris. And like everyone else, I think the raid-running world ought to be warned off of Jaunty until this is fixed in the distribution.

Regards,
ded

Spoke too soon. After installing both Chris's fixed hal and the libblkid1 update above, my AMD64 system failed to boot, pausing about 1/5 of the way through. If I hit Alt-Ctl-Del, I could get the boot to resume but with my / mounted read-only. I suspect something is wrong with the libblkid1 that is causing the problem.

Anyone else seeing this?

Yes, I have the same problem.
From my quick investigation, the problem was when the findfs binary was called and it was stuck on that binary execution.

Eric, thanks for confirming. Is your system AMD64 or i386?

Please don't do silly things like install libblkid from karmic - that's totally unsupported and is likely to break your machine.

Everyone is at UDS at the moment but I'll see if this could be considered for a SRU when everyone gets back.

my system is AMD64.

It's not something I would have thought of on my own, but it appears to be a dependency in the amd .deb hal package you posted above:

root@saturn:/home/ded/Downloads# dpkg -i hal_0.5.12+git20090512-0ubuntu2_amd64.deb
(Reading database ... 94344 files and directories currently installed.)
Preparing to replace hal 0.5.12~rc1+git20090403-0ubuntu1 (using hal_0.5.12+git20090512-0ubuntu2_amd64.deb) ...
 * Stopping Hardware abstraction layer hald [ OK ]
Unpacking replacement hal ...
dpkg: dependency problems prevent configuration of hal:
 hal depends on libblkid1 (>= 2.15~rc2-1ubuntu1); however:
  Version of libblkid1 on system is 1.41.4-1ubuntu1.
dpkg: error processing hal (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db ...
Errors were encountered while processing:
 hal

Is karmic poison? Why the warning?

Thanks.

Chris et. al.,

OK, I get it now. karmic is the next release---my bad, just an end user here.

Still, is there some repository that has version 2.15 of libblkid1 for jaunty? Would someone please post a link or the sources entry for such a repository? Looks like Chris's fix to hal needs it from somewhere at least on the 64-bit systems.

Thanks.

All,

Has everyone else been able to work around this issue, or is it just me? Since my jaunty update, I can't get mouse or keyboard with mdadm installed.

I was hoping someone would tell me what to do about the libblkid1 dependency in Chris's hal update---at least on amd64---but no traffic here for several days.

Chris, any help? Any one else? Does it work on an i386?

Regards,

Here's a debdiff for the Jaunty update

Accepted hal into jaunty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

The patch from jaunty-proposed WORKS for me! Thank you!

Patch also worked for me on AMD64. Thanks, Chris.

This bug was fixed in the package hal - 0.5.12~rc1+git20090403-0ubuntu2

---------------
hal (0.5.12~rc1+git20090403-0ubuntu2) jaunty-proposed; urgency=low

  * debian/patches/50_no_crash_on_md_blockdev.patch:
    - When adding a block device, don't assume that the parent
      has storage capability. This fixes a crash where the device
      is re-parented to the root computer device object (such as
      with mdraid devices). LP: #361689.

 -- Chris Coulson <email address hidden> Fri, 05 Jun 2009 12:25:50 +0200

After applying this update I am having trouble with my X server. There are the following errors in the log:

====
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
(EE) config/hal: NewInputDeviceRequest failed (8)
====

Do you think it could be a side effect?

Oli Wade [2009-06-17 9:41 -0000]:
> After applying this update I am having trouble with my X server. There
> are the following errors in the log:
>
> ====
> (EE) config/hal: NewInputDeviceRequest failed (8)
>
> Do you think it could be a side effect?

The hal update didn't change anything wrt. input devices. If you
downgrade to the previous hal again [1], does it work again?

Does that only happen right after the package upgrade, or also after a
restart of the machine?

[1] sudo apt-get install hal/jaunty-updates
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

It happened after a reboot - I blamed this update due to the "hal" in the error message.

I've downgraded ("sudo apt-get install hal/jaunty libhal1/jaunty libhal-storage1/jaunty") but the problem remained through several reboots until I did a shutdown and then poweron.

Therefore I suspect some part(s) of the hardware might have been in a weird state and there is nothing wrong with the update.

Oli Wade [2009-06-18 8:56 -0000]:
> Therefore I suspect some part(s) of the hardware might have been in a
> weird state and there is nothing wrong with the update.

OK, thanks for checking!

Ahhh! Is this bug back?

The server I was having a problem with is 1000 miles away from me now and I rebooted it and it didn't come back up. Thinking back a few hours, I remember that "update-manager" installed a HAL update.

Noooooooooooo! I need my server and this is going to force me into weeks of downtime! How is this not a "Critical" bug, and how did this update cause a regression?

This is absolutely horrible. The first time I experienced this bug, it cost me hours/possibly days of troubleshooting, now I have indefinite unscheduled downtime. Seriously CRITICAL!

Anyone have any suggestions?

Didn't mean to sound upset at anyone in my previous post. I know that Chris did an awesome job with the first patch. I'm just upset and looking for a support group :-)

The sooner we can get a permanent fix for this issue, the sooner I can get a good night's sleep.

Please, anyone who is capable, help out with this!

Also, can we change the status back to "confirmed" or "incomplete" so it will bubble back up and get noticed?

This bug hasn't regressed, as the recent HAL update was completely unrelated, and didn't even touch any code AFAICT. If you're experiencing any issues, it's defaintely not related to this bug, even if you are experiencing a HAL crash.

You should open a new bug report, preferably by submitting a crash report using Apport. You might need to enable apport in /etc/default/apport and restart though.