Kernel panics on iSER (iscsi over IB) logouts

This bug was discovered when fixing iSER enablement for Trusty in case:

https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1401575

Kernel 3.16 does not suffer from this issue. I'll cherry-pick fixes either by checking git commits or bisecting development tree (since I have a reproducer).

I started bisecting this issue:

Upstream "linux-stable" git repo v3.13 -> BISECT BAD (same issue as description)
Upstream "linux-stable" git repo v3.16 -> BISECT GOOD (no issues after iSER iscsi login and iscsid turned off)

It will take sometime but I'll come up with the commit that fixes this behavior:

- Kernel 3.13 panicing on iscsi logout after node has logged in and mapped target using iSER.

After taking long time bisecting kernel (3.13 <-> 3.16) it came out that the following commit (and its dependencies) are the ones that fixes the issue:

commit b73c3adabdb1e2cb2f2c69bc3cbb9306aa3f9700
Author: Ariel Nahum <email address hidden>
Date: Thu May 22 11:00:18 2014 +0300

    IB/iser: Simplify connection management

    iSER relies on refcounting to manage iser connections establishment
    and teardown.

    Following commit 39ff05dbbbdb ("IB/iser: Enhance disconnection logic
    for multi-pathing"), iser connection maintain 3 references:

     - iscsi_endpoint (at creation stage)
     - cma_id (at connection request stage)
     - iscsi_conn (at bind stage)

    We can avoid taking explicit refcounts by correctly serializing iser
    teardown flows (graceful and non-graceful).

    Our approach is to trigger a scheduled work to handle ordered teardown
    by gracefully waiting for 2 cleanup stages to complete:

     1. Cleanup of live pending tasks indicated by iscsi_conn_stop completion
     2. Flush errors processing

    Each completed stage will notify a waiting worker thread when it is
    done to allow teardwon continuation.

    Since iSCSI connection establishment may trigger endpoint disconnect
    without a successful endpoint connect, we rely on the iscsi <-> iser
    binding (.conn_bind) to learn about the teardown policy we should take
    wrt cleanup stages.

    Since all cleanup worker threads are scheduled (release_wq) in
    .ep_disconnect it is safe to assume that when module_exit is called,
    all cleanup workers are already scheduled. Thus proper module unload
    shall flush all scheduled works before allowing safe exit, to
    guarantee no resources got left behind.

    Signed-off-by: Ariel Nahum <email address hidden>
    Signed-off-by: Sagi Grimberg <email address hidden>
    Reviewed-by: Roi Dayan <email address hidden>
    Reviewed-by: Or Gerlitz <email address hidden>
    Signed-off-by: Roland Dreier <email address hidden>

I have backported this fix to 3.13 (attached as a .patch for review if anyone is willing to). Will do more tests, provide PPA for feedback for hopefully will ask kernel team for SRU.

No Kernel panics for iSER Logout at this time (with this patch):

Linux dixie 3.13.11-ckt18-test01 #3 SMP Fri Apr 24 13:46:29 BRT 2015 x86_64 x86_64 x86_64 GNU/Linux

root@dixie:~# iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 172.16.0.1 -I iser
172.16.0.1:3260,1 iqn.2001-04.com.hertz-lunit1

root@dixie:~# iscsiadm -m node -l
Logging in to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] (multiple)
Login to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.

root@dixie:~# dd if=/dev/zero of=/dev/sdc bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.996458 s, 1.1 GB/s

root@dixie:~# service open-iscsi stop
 * Unmounting iscsi-backed filesystems [ OK ]
 * Disconnecting iSCSI targets
Logging out of session [sid: 5, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260]
Logout of [sid: 5, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.
 * Stopping iSCSI initiator service [ OK ]

And then again:

root@dixie:~# iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 172.16.0.1 -I iser
172.16.0.1:3260,1 iqn.2001-04.com.hertz-lunit1

root@dixie:~# iscsiadm -m node -l
Logging in to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] (multiple)
Login to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.

root@dixie:~# dd if=/dev/zero of=/dev/sdc bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.844185 s, 1.3 GB/s

root@dixie:~# service open-iscsi stop
 * Unmounting iscsi-backed filesystems [ OK ]
 * Disconnecting iSCSI targets
Logging out of session [sid: 2, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260]
Logout of [sid: 2, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.
 * Stopping iSCSI initiator service

No kernel panic again. Will do some stress testing for iSER to make sure everything is good.

Anyone can use the following PPA for hotfixed 3.13 (Trusty) kernel for x86/x64:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1443648-x64

And the following PPA for hotfixed kernel for arm64 (moonshot):

https://launchpad.net/~inaddy/+archive/ubuntu/lp1443648-arm64

These PPAs should be used for any Trusty machine being an iSER (iSCSI over IB) client.

PLEASE ->

provide feedback for these kernels so I can ask kernel team to SRU my backport.

OBS:
I had to deal with some conflicts when backporting code from 3.16 to 3.13.
I already tested myself and it looks okay but I'll stress test the 3.13 iSER code a bit more.

Proposed this fix to kernel team mailing list. Waiting on SRU.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

+ Kamal

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Brad Figg
Sent: Wednesday, August 5, 2015 9:15 PM
To: Amir Vadai <email address hidden>
Subject: [Bug 1443648] Re: Kernel panics on iSER (iscsi over IB) logouts
Importance: High

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

** Tags added: verification-needed-trusty

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1443648

Title:
  Kernel panics on iSER (iscsi over IB) logouts

Status in linux package in Ubuntu:
  Invalid
Status in linux source package in Trusty:
  Fix Committed

Bug description:
  SRU Justification:

  Impact: After iSER was fixed by LP: #1409904 iscsi logouts cause panics.
  Fix: Upstream development and devel bisection between 3.13 and 3.16.
  Testcase: To use iSER and shut down the open-iscsi daemon (comment #8).

  ---------------------

  While shutting down the "open-iscsi" daemon (with an iSER mapped
  volume from other host) I'm getting the following kernel panic for
  kernel 3.13.0-49:

  """
  [272820.948946] scsi15 : iSCSI Initiator over iSER
  [272843.474551] sd 14:0:0:1: [sdc] Synchronizing SCSI cache
  [272843.998231] BUG: unable to handle kernel paging request at ffffffffa03840b7
  [272843.998258] IP: [<ffffffffa03840b7>] 0xffffffffa03840b7
  [272843.998278] PGD 1c11067 PUD 1c12063 PMD 80afb5067 PTE 0
  [272843.998306] Oops: 0010 [#1] SMP
  [272843.998324] Modules linked in: rdma_ucm ib_ucm ib_srp scsi_transport_srp scsi_tgt ib_srpt ib_mthca ib_ipa
  [272843.998742] CPU: 1 PID: 32399 Comm: kworker/1:0 Not tainted 3.13.0-49-generic #81-Ubuntu
  [272843.998758] Hardware name: HP ProLiant DL360e Gen8, BIOS P73 11/12/2013
  [272843.998774] Workqueue: ib_cm cm_work_handler [ib_cm]
  [272843.998789] task: ffff88080b72c800 ti: ffff880802522000 task.ti: ffff880802522000
  [272843.998827] RIP: 0010:[<ffffffffa03840b7>] [<ffffffffa03840b7>] 0xffffffffa03840b7
  [272843.998870] RSP: 0018:ffff880802523c28 EFLAGS: 00010296
  [272843.998893] RAX: ffff8800b851d801 RBX: ffff8807af8482b8 RCX: 0000000180200008
  [272843.998929] RDX: 0000000180200009 RSI: 0000000080200008 RDI: ffff88081f003900
  [272843.998965] RBP: ffff880802523c30 R08: 0000000000000000 R09: 0000000000000001
  [272843.999002] R10: ffffea0002e14740 R11: ffffffffa00e0b9a R12: ffff88080953e400
  [272843.999038] R13: ffff8807af8482b8 R14: ffff88080953e400 R15: 0000000000000000
  [272843.999075] FS: 0000000000000000(0000) GS:ffff88083f420000(0000) knlGS:0000000000000000
  [272843.999113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [272843.999136] CR2: ffffffffa03840b7 CR3: 00000...

Verifying this right now, will share results soon.

I was able to verify that the new kernel fixes the issue:

root@dixie:~# uname -a
Linux dixie 3.13.0-62-generic #101-Ubuntu SMP Thu Jul 30 09:01:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@dixie:~# iscsiadm -m discovery --op=show --type sendtargets --portal 172.16.0.1 -I iser
172.16.0.1:3260,1 iqn.2001-04.com.hertz-lunit1

root@dixie:~# iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 172.16.0.1 -I iser
172.16.0.1:3260,1 iqn.2001-04.com.hertz-lunit1

root@dixie:~# iscsiadm -m node -l
Logging in to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] (multiple)
Login to [iface: iser, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.

root@dixie:~# ls /dev/sdc
/dev/sdc

root@dixie:~# dd if=/dev/zero of=/dev/sdc bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.01833 s, 1.1 GB/s

root@dixie:~# service open-iscsi stop
 * Unmounting iscsi-backed filesystems [ OK ]
 * Disconnecting iSCSI targets Logging out of session [sid: 3, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260]

### NO KERNEL PANIC!!! (as it used to happen)

Logout of [sid: 3, target: iqn.2001-04.com.hertz-lunit1, portal: 172.16.0.1,3260] successful.
                                                                                  [ OK ]
 * Stopping iSCSI initiator service [ OK ]
root@dixie:~#
root@dixie:~#
root@dixie:~# service open-iscsi start
 * Starting iSCSI initiator service iscsid [ OK ]
 * Setting up iSCSI targets
iscsiadm: No records found
                                                                                  [ OK ]
 * Mounting network filesystems [ OK ]

2nd verification:

https://bugs.launchpad.net/ubuntu/+source/libibverbs/+bug/1409904/comments/88

inaddy@mylinux  ~/Work/Kernel/Ubuntu/ubuntu-trusty   master  git tag --contains d6cdcda08c712cae7f071866e7e2e331242d66c3
Ubuntu-3.13.0-60.99
Ubuntu-3.13.0-62.101
Ubuntu-3.13.0-62.102
Ubuntu-3.13.0-63.103
Ubuntu-3.13.0-64.104
Ubuntu-3.13.0-65.105

 linux-image-generic | 3.13.0.24.28 | trusty | amd64, arm64, armhf, i386, ppc64el
 linux-image-generic | 3.13.0.65.71 | trusty-security | amd64, arm64, armhf, i386, ppc64el
 linux-image-generic | 3.13.0.65.71 | trusty-updates | amd64, arm64, armhf, i386, ppc64el

This is already Fix Released.