Kernel 2.6.24-23-rt locks up hard under heavy rt load

Bug #324275 reported by Matthias Lüscher
14
Affects Status Importance Assigned to Milestone
linux-rt (Ubuntu)
Invalid
Undecided
Unassigned
Hardy
Won't Fix
Medium
Alessio Igor Bogani
Jaunty
Invalid
Undecided
Unassigned

Bug Description

Description: Ubuntu 8.04.2
Release: 8.04
linux-rt:
  Installed: 2.6.24.23.25
  Candidate: 2.6.24.23.25
  Version table:
 *** 2.6.24.23.25 0
        500 http://ch.archive.ubuntu.com hardy-updates/multiverse Packages
        100 /var/lib/dpkg/status
     2.6.24.22.24 0
        500 http://security.ubuntu.com hardy-security/multiverse Packages
     2.6.24.16.18 0
        500 http://ch.archive.ubuntu.com hardy/multiverse Packages

The latest rt-kernel locks up hard under heavy load:
The problem can be reproduced easily by using the cyclictest program (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git) with the following command line options:
sudo ./cyclictest -p99 -t10 -n -i250

Additional observations:
- The problem got introduced with the upgrade to the rt21 patch - it did not happen with older kernel versions based on previous rt patches.
- The lockup is really hard: there is no output on a netconsole and even the magic sysrequest keys to reboot the system (Alt+SysRQ+b) do not work anymore.
- It also happens to a vanilla kernel with the rt21 patch (I did not try any newer patches yet)
- with a lower realtime priority the system seems to run at least more stable (e.g. sudo ./cyclictest -p95 -t10 -n -i250)
- the problem seems to appear only on multicore systems

Revision history for this message
Matthias Lüscher (m-luescher) wrote :

The above mentioned problem can also be reproduced with vanilla 2.6.24.7 with the latest rt patch (rt26).

Revision history for this message
Matthias Lüscher (m-luescher) wrote :

The lockup is currently being discussed on linux-rt-users (http://marc.info/?t=123360942700006&r=1&w=2). It looks like the issue is related to a patch that was introduced with 2.6.24.7-rt13.

Revision history for this message
Matthias Lüscher (m-luescher) wrote :

The fix for the crash can be found here:
http://marc.info/?l=linux-rt-users&m=123393014501532&w=2

The described patch should get included into the quilt series of the rt kernel:
[ubuntu/ubuntu-hardy.git]/debian/binary-custom.d/rt/patchset/

Revision history for this message
Matthias Lüscher (m-luescher) wrote :

The above mentioned problem might also crash ubuntu studio (jackd, http://marc.info/?l=linux-rt-users&m=123323476722881&w=2, 2.6.26.7-rt13 had the same problem).

The attached patch has made it into 2.6.24.7-rt27.

Changed in linux-rt:
assignee: nobody → abogani
status: New → In Progress
Changed in linux-rt:
importance: Undecided → Medium
status: In Progress → Fix Committed
Revision history for this message
Alessio Igor Bogani (abogani) wrote :

The fix will be available with first kernel after 2.6.24-24.50.

Revision history for this message
Martin Pitt (pitti) wrote :

Please clarify the situation in Jaunty and close the task if it is already fixed.

tags: added: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Accepted linux into hardy-proposed; please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Revision history for this message
Alessio Igor Bogani (abogani) wrote :

Jaunty is not affected.

Changed in linux-rt:
assignee: nobody → abogani
importance: Undecided → Medium
status: New → Fix Committed
status: Fix Committed → Invalid
assignee: abogani → nobody
importance: Medium → Undecided
Revision history for this message
stryd_one (stryd-one) wrote :

After installing these updates my onboard sound, virtual midi, lirc and USB TV tuner failed. Rolled back to .50 and all is well. Errors in the logs are as attached.

Revision history for this message
Alessio Igor Bogani (abogani) wrote :

@stryd_one,

Do you have tested 51-generic also?

Thanks!

Revision history for this message
stryd_one (stryd-one) wrote :

Sorry Alessio... it took me a few hours to get it back to working order (my first experience with rolling back kernel updates), so I haven't found such a window of opportunity since then... I'll try it ASAP.

Revision history for this message
stryd_one (stryd-one) wrote :

Short version: Still no good with .51-rt, but okay with .51-generic. I'm back to 2.6.24-23-rt for the time being.

Long version: Using 2.6.24-24.51-rt, I still have logs full of those symbol errors (same thing repeated for all the devices involved). I didn't attach logs because it's the same thing with new timestamps, but I'm happy to if you need them.
Using 2.6.24-24.51-generic, I didn't see any symbol errors, but I did have some troubles with my TV Tuner, which works OK with 2.6.24-24.50-rt and -generic.

I suspected this to be related to the DKMS build of v4l-dvb I'm using and not directly related to this patch. I did notice that libc was updated too though, so perhaps that has a part to play in this? I gather that the DKMS system builds the kernel module automatically when the kernel is updated. It would have used libc to do so, so perhaps this explains the malfunctional driver. I wanted to be complete, so I thought it worth a mention. Otherwise the .51-generic kernel works OK, sound and midi and lirc are behaving.

I'm too linux-newb to know what this indicates... Is there anything I can do to assist, or is this more likely to be a problem with my particular machine?

Revision history for this message
stryd_one (stryd-one) wrote :

Any word on this? There is an update I'm being alerted to today, to 2.6.24-23.52, but I'm concerned about installing that update as that is now my only working kernel version... Sorry to nag!

Revision history for this message
stryd_one (stryd-one) wrote :

Don't suppose you'd like to know what was wrong with the package?

Revision history for this message
Rolf Leggewie (r0lf) wrote :

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

Changed in linux-rt (Ubuntu Hardy):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.