WARNING: at /build/buildd/linux-2.6.31/fs/notify/inotify/inotify_fsnotify.c:129 idr_callback+0x4f/0x80()

This problem seems to be a race condition in the new notify code in the lastest (2.6.31) kernels. A test kernel package for both i386 and amd64 with a patch is located at:

http://people.canonical.com/~lieb /{i386,amd64}

Choose the appropriate package for your system, install and test it please.

NOTE: these are test packages. The only supported packages will be delivered via the update process once testing confirms the fix.

Thanks

Chatted to lieb on irc and confirmed that the fix is suspected to be this one:

  commit 9f0d793b52eb2266359661369ef6303838904855
  Author: Eric Paris <email address hidden>
  Date: Fri Sep 11 13:03:19 2009 -0400

    fsnotify: do not set group for a mark before it is on the i_list

This same patch is in testing in the 2.6.31.6 stable update.

To have a go on this, you would find the latest kernel at https://launchpad.net/~stefan-bader-canonical/+archive/karmic. Please let us know if that works for you. Thanks you.

I updated for the latest kernel and had no issue so far. But as I can't reproduce this bug, I can't affirm it worked.

Thanks. I have no idea what the problem was. Just that Ubuntu said something
serious went wrong in the kernel and wanted to report a bug so I followed
the steps. There were no symptoms so I can't help to know if it was fixed.

2009/11/13 Pierre Faivre <email address hidden>

> I updated for the latest kernel and had no issue so far. But as I can't
> reproduce this bug, I can't affirm it worked.
>
> --
> WARNING: at
> /build/buildd/linux-2.6.31/fs/notify/inotify/inotify_fsnotify.c:129
> idr_callback+0x4f/0x80()
> https://bugs.launchpad.net/bugs/442966
> You received this bug notification because you are a direct subscriber
> of a duplicate bug.
>
> Status in “linux” package in Ubuntu: In Progress
>
> Bug description:
> Unsure what has happened. I got this bug when booted up beta kubuntu 9.10.
>
> ProblemType: KernelOops
> Annotation: Your system might become unstable now and might need to be
> restarted.
> Architecture: i386
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC0: phoenixlink 8185 F.... pulseaudio
> phoenixlink 9583 F.... kmix
> /dev/snd/controlC1: phoenixlink 8185 F.... pulseaudio
> phoenixlink 9583 F.... kmix
> CRDA: Error: [Errno 2] No such file or directory
> Card0.Amixer.info:
> Card hw:0 'NVidia'/'HDA NVidia at 0xfae78000 irq 21'
> Mixer name : 'Realtek ALC662 rev1'
> Components : 'HDA:10ec0662,18493662,00100101'
> Controls : 33
> Simple ctrls : 17
> Card1.Amixer.info:
> Card hw:1 'Q9000'/'Logitech, Inc. QuickCam Pro 9000 at usb-0000:00:02.1-3,
> high speed'
> Mixer name : 'USB Mixer'
> Components : 'USB046d:0990'
> Controls : 2
> Simple ctrls : 1
> Card1.Amixer.values:
> Simple mixer control 'Mic',0
> Capabilities: cvolume cvolume-joined cswitch cswitch-joined
> Capture channels: Mono
> Limits: Capture 0 - 3072
> Mono: Capture 3072 [100%] [18.00dB] [off]
> Date: Mon Oct 5 05:29:35 2009
> DistroRelease: Ubuntu 9.10
> Failure: oops
> HibernationDevice: RESUME=UUID=49686331-26fb-4cad-b801-3d5af4363eab
> IwConfig:
> lo no wireless extensions.
>
> eth1 no wireless extensions.
>
> vboxnet0 no wireless extensions.
> MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
> NonfreeKernelModules: nvidia
> Package: linux-image-2.6.31-11-generic 2.6.31-11.38
> ProcCmdLine: root=UUID=260c2e1a-26d3-4a86-bec2-effb2c4858af ro quiet splash
> ProcVersionSignature: Ubuntu 2.6.31-11.38-generic
> RelatedPackageVersions:
> linux-backports-modules-2.6.31-11-generic N/A
> linux-firmware 1.20
> RfKill:
>
> SourcePackage: linux
> Tags: kernel-oops
> Title: [174353.492058] ------------[ cut here ]------------
> Uname: Linux 2.6.31-11-generic i686
> WpaSupplicantLog:
>
> dmi.bios.date: 04/24/2009
> dmi.bios.vendor: American Megatrends Inc.
> dmi.bios.version: P1.40
> dmi.board.name: K10N78M
> dmi.board.vendor: ASRock
> dmi.chassis.asset.tag: To Be Filled By O.E.M.
> dmi.chassis.type: 3
> dmi.chassis.vendor: To Be Filled By O.E.M.
> dmi.chassis.version: To Be Filled By O.E.M.
> dmi.modalias:
> dmi:bvnAmericanMegatrendsInc.:bvrP1.40:bd04/24/2009:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledB...

I've not seen this bug in at least 7 days now. I haven't installed any updates from any PPAs, only updates from Karmic and Karmic proposed.

I am still seeing this problem on Ubuntu 9.10. The real problem is that when this happens (as near as I can tell), my system becomes unresponsive to network traffic. However, when I go to the console (via KVM) and simply press enter, everything returns to normal.

This happens intermittently, often (it seems) around midnight. I am running dovecot imap, and that would seem to be involved in the information below. There was another bug reported with nearly indentical details as below and involving dovecot.

I am running a server in this case, no gdm but lots of other usual server things (apache, postfix, mysql, dovecot, etc). I need this system to be reliable, and despite apparent reports that this problem was fixed in 2.6.31-6, clearly it has not been.

So I offer this data point in the hopes that it is helpful, and in the hope that this issue get some increased priority and gets fixed once and for all! (For what it's worth, I also run Ubuntu 9.10 on a laptop and on a system at work, and neither suffer from this. But then, neither of those systems runs dovecot or any other server-like stuff.

If there is more information I can provide, I am happy to do so. Just contact me. mark at granoff dot net

$ uname -a
Linux opus 2.6.31-16-generic #52-Ubuntu SMP Thu Dec 3 22:00:22 UTC 2009 i686 GNU/Linux

From kern.log:

Dec 9 00:16:58 opus kernel: [147655.680560] ------------[ cut here ]------------
Dec 9 00:16:58 opus kernel: [147655.680583] WARNING: at /build/buildd/linux-2.6.31/fs/notify/inotify/inotify_fsnotify.c:129 idr_callback+0x4f/0x80()
Dec 9 00:16:58 opus kernel: [147655.680588] Hardware name: A7V400-MX
Dec 9 00:16:58 opus kernel: [147655.680592] inotify closing but id=0 for entry=ef9c6840 in group=f6346580 still in idr. Probably leaking memory
Dec 9 00:16:58 opus kernel: [147655.680596] Modules linked in: snd_via82xx snd_ac97_codec iptable_filter ip_tables i2c_viapro ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device x_tables lp snd via_ircc irda soundcore ns558 gameport via_agp ppdev parport_pc parport shpchp agpgart crc_ccitt raid10 raid456 raid6_pq async_xor async_memcpy async_tx xor raid1 raid0 multipath linear via_rhine mii
Dec 9 00:16:58 opus kernel: [147655.680647] Pid: 3969, comm: imap Not tainted 2.6.31-16-generic #52-Ubuntu
Dec 9 00:16:58 opus kernel: [147655.680651] Call Trace:
Dec 9 00:16:58 opus kernel: [147655.680666] [<c014518d>] warn_slowpath_common+0x6d/0xa0
Dec 9 00:16:58 opus kernel: [147655.680672] [<c021620f>] ? idr_callback+0x4f/0x80
Dec 9 00:16:58 opus kernel: [147655.680676] [<c021620f>] ? idr_callback+0x4f/0x80
Dec 9 00:16:58 opus kernel: [147655.680682] [<c0145206>] warn_slowpath_fmt+0x26/0x30
Dec 9 00:16:58 opus kernel: [147655.680686] [<c021620f>] idr_callback+0x4f/0x80
Dec 9 00:16:58 opus kernel: [147655.680694] [<c0310fc6>] idr_for_each+0x76/0xd0
Dec 9 00:16:58 opus kernel: [147655.680699] [<c02161c0>] ? idr_callback+0x0/0x80
Dec 9 00:16:58 opus kernel: [147655.680704] [<c0214357>] ? fsnotify_put_event+0x37/0x50
Dec 9 00:16:58 opus kernel: [147655.680709] [<c0214...

Just to confirm that this bug is still not fixed: I just got this report on my up-to-date server. The system was more or less idle with the usual server stuff running (apache, postfix, bind,...).

Dec 24 12:27:59 batleth kernel: [1217678.140504] ------------[ cut here ]------------
Dec 24 12:27:59 batleth kernel: [1217678.140515] WARNING: at /build/buildd/linux-2.6.31/fs/notify/inotify/inotify_fsnotify.c:129 idr_callback+0x49/0x70()
Dec 24 12:27:59 batleth kernel: [1217678.140518] Hardware name: X8STi
Dec 24 12:27:59 batleth kernel: [1217678.140521] inotify closing but id=0 for entry=ffff8801b586bee0 in group=ffff880155c8cc00 still in idr. Probably leaking memory
Dec 24 12:27:59 batleth kernel: [1217678.140523] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs ipt_REJECT xt_multiport ipt_LOG xt_limit xt_state xt_tcpudp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables psmouse lp serio_raw parport raid10 raid456 raid6_pq async_xor async_memcpy async_tx xor usbhid usb_storage raid1 raid0 multipath linear e1000e
Dec 24 12:27:59 batleth kernel: [1217678.140561] Pid: 14547, comm: gam_server Not tainted 2.6.31-16-server #53-Ubuntu
Dec 24 12:27:59 batleth kernel: [1217678.140564] Call Trace:
Dec 24 12:27:59 batleth kernel: [1217678.140571] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0
Dec 24 12:27:59 batleth kernel: [1217678.140575] [<ffffffff8105e6ac>] warn_slowpath_fmt+0x3c/0x40
Dec 24 12:27:59 batleth kernel: [1217678.140581] [<ffffffff81524bc4>] ? wait_for_common+0x124/0x170
Dec 24 12:27:59 batleth kernel: [1217678.140586] [<ffffffff81153039>] idr_callback+0x49/0x70
Dec 24 12:27:59 batleth kernel: [1217678.140590] [<ffffffff810538a0>] ? default_wake_function+0x0/0x10
Dec 24 12:27:59 batleth kernel: [1217678.140595] [<ffffffff81271769>] idr_for_each+0x89/0xe0
Dec 24 12:27:59 batleth kernel: [1217678.140599] [<ffffffff81152ff0>] ? idr_callback+0x0/0x70
Dec 24 12:27:59 batleth kernel: [1217678.140603] [<ffffffff81150bb4>] ? fsnotify_put_event+0x44/0x60
Dec 24 12:27:59 batleth kernel: [1217678.140607] [<ffffffff81152fd2>] inotify_free_group_priv+0x22/0x40
Dec 24 12:27:59 batleth kernel: [1217678.140611] [<ffffffff81150d13>] fsnotify_final_destroy_group+0x23/0x40
Dec 24 12:27:59 batleth kernel: [1217678.140615] [<ffffffff81150e18>] fsnotify_put_group+0x98/0xb0
Dec 24 12:27:59 batleth kernel: [1217678.140619] [<ffffffff8115325f>] inotify_release+0x2f/0x50
Dec 24 12:27:59 batleth kernel: [1217678.140622] [<ffffffff811201ba>] __fput+0xda/0x1f0
Dec 24 12:27:59 batleth kernel: [1217678.140625] [<ffffffff811202ed>] fput+0x1d/0x30
Dec 24 12:27:59 batleth kernel: [1217678.140630] [<ffffffff8111c3f8>] filp_close+0x58/0x90
Dec 24 12:27:59 batleth kernel: [1217678.140634] [<ffffffff81060487>] put_files_struct+0x77/0xe0
Dec 24 12:27:59 batleth kernel: [1217678.140637] [<ffffffff8106053f>] exit_files+0x4f/0x60
Dec 24 12:27:59 batleth kernel: [1217678.140641] [<ffffffff810628cb>] do_exit+0x14b/0x360
Dec 24 12:27:59 batleth kernel: [1217678.140644] [<ffffffff81062b29>] do_group_exit+0x49/0xc0
Dec 24 12:27:59 batleth ke...

And once again I was hit by this bug, this time with kernel 2.6.31-19-server #56-Ubuntu, which was just superseded by kernel 2.6.31-20-server #58-Ubuntu, which is running now on my server.

This bug should have been fixed in 2.6.31-17.54. Can anyone confirm that they get this on a kernel later than that? Please make sure that you actually get this oops, and not another similar one.

I just got one yesterday running kernel 2.6.31-20.58, but I'm unsure whether or not this is the same oops. If you think this is not the same oops, please tell me so that I can open another bug report.

[2643126.913740] WARNING: at /build/buildd/linux-2.6.31/fs/notify/inotify/inotify_fsnotify.c:129 idr_callback+0x49/0x70()
[... see full oops in the attachment]

Hmm, seems #485556 would be the right bug for my report - if so, I apologize. What a subtle difference...

It was a strange one. I only got it twice.

Mike

On Friday 23 April 2010 10:26:18 you wrote:
> Hmm, seems #485556 would be the right bug for my report - if so, I
> apologize. What a subtle difference...
>
--
Mike McGinn FACOCM
No electrons were harmed in sending this message.
My deeds are my dearest companions. I am the beneficiary of my deeds.
My deeds are the ground on which I stand. ** Registered Linux User 377849

Marking as Fixed Released. If anyone does get this again, feel free to reopen this bug report.