From: keith.busch@intel.com (Busch, Keith)
Subject: [PATCH 8/9] nvme: remove dead controllers from a work item
Date: Thu, 22 Oct 2015 18:10:15 +0000 [thread overview]
Message-ID: <20151022181014.GE21840@localhost.localdomain> (raw)
In-Reply-To: <1445515421-4940-9-git-send-email-hch@lst.de>
On Thu, Oct 22, 2015@02:03:40PM +0200, Christoph Hellwig wrote:
> Compared to the kthread this gives us multiple call prevention for free.
My apologies in advance, I usually try to be more helpful debugging
new issues, but I'm running low on time today. I might be able to dig
into this more tomorrow, but if you could, please verify what happens
if an admin command times out during initialization. The reset_work
gets queued twice on two different work_queues and causes trouble. I
get one of several crashes or warnings after applying the whole series,
but this is the most frequent I'm seeing so far:
[ 348.845441] BUG: unable to handle kernel NULL pointer dereference at 0000000000000014
[ 348.846615] IP: [<ffffffffa02c1a48>] nvme_dev_shutdown+0x303/0x3e8 [nvme]
[ 348.847615] PGD b8749067 PUD b8748067 PMD 0
[ 348.848328] Oops: 0002 [#1] SMP
[ 348.848826] Modules linked in: nvme bnep rfcomm bluetooth rfkill nfsd auth_rpcgss oid_registry nfs_acl nfs lockdgrace fscache sunrpc dm_round_robin loop dm_multipath parport_pc parport evdev psmouse pcspkr serio_raw processor button ext4 crc16 jbd2 mbcache sg sr_mod sd_mod cdrom ata_generic floppy e1000 ata_piix libata scsi_mod
[ 348.849428] CPU: 0 PID: 3415 Comm: kworker/0:0 Not tainted 4.3.0-rc4+ #4
[ 348.849428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
[ 348.849428] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
[ 348.849428] task: ffff88013afe6ec0 ti: ffff88013afe8000 task.ti: ffff88013afe8000
[ 348.849428] RIP: 0010:[<ffffffffa02c1a48>] [<ffffffffa02c1a48>] nvme_dev_shutdown+0x303/0x3e8 [nvme]
[ 348.849428] RSP: 0018:ffff88013afebcd8 EFLAGS: 00010202
[ 348.849428] RAX: 0000000000464001 RBX: ffff8800369c2800 RCX: 0000005138ced189
[ 348.849428] RDX: 0000000000000000 RSI: ffffffff81409bf0 RDI: ffff88013ab53f10
[ 348.849428] RBP: ffff88013afe6ec0 R08: ffff88013afe8000 R09: 0000000000000000
[ 348.849428] R10: 0000000000000000 R11: ffff88013fc142c0 R12: 0000000000000000
[ 348.849428] R13: ffff880036f0c540 R14: ffff88013afebcf8 R15: 0000000000000000
[ 348.849428] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[ 348.849428] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 348.849428] CR2: 0000000000000014 CR3: 00000000b8cb2000 CR4: 00000000000006f0
[ 348.849428] Stack:
[ 348.849428] ffff8800b8e556b8 0000000000000000 ffff8800b8e55640 0000000000000082
[ 348.849428] ffff88013afe6ec0 ffff88013afebd10 ffff880100000000 0000000000000000
[ 348.849428] ffff88013afebd18 ffff88013afebd18 0000000000000000 0000000000000000
[ 348.849428] Call Trace:
[ 348.849428] [<ffffffffa02c1bd6>] ? nvme_remove+0x6f/0xc0 [nvme]
[ 348.849428] [<ffffffff8121b67a>] ? pci_device_remove+0x43/0x8e
[ 348.849428] [<ffffffff812bb882>] ? __device_release_driver+0x91/0x109
[ 348.849428] [<ffffffff812bb90b>] ? device_release_driver+0x11/0x1a
[ 348.849428] [<ffffffff812165e2>] ? pci_stop_bus_device+0x62/0x85
[ 348.849428] [<ffffffff81216798>] ? pci_stop_and_remove_bus_device+0x9/0x12
[ 348.849428] [<ffffffff812167b9>] ? pci_stop_and_remove_bus_device_locked+0x18/0x21
[ 348.849428] [<ffffffffa02bfd44>] ? nvme_remove_dead_ctrl_work+0x1e/0x28 [nvme]
[ 348.849428] [<ffffffff810560b6>] ? process_one_work+0x177/0x27b
[ 348.849428] [<ffffffff81056394>] ? worker_thread+0x1b4/0x28b
[ 348.849428] [<ffffffff810561e0>] ? process_scheduled_works+0x26/0x26
[ 348.849428] [<ffffffff8105a4b6>] ? kthread+0x99/0xa1
[ 348.849428] [<ffffffff8105a41d>] ? kthread_parkme+0x16/0x16
[ 348.849428] [<ffffffff813c9d6f>] ? ret_from_fork+0x3f/0x70
[ 348.849428] [<ffffffff8105a41d>] ? kthread_parkme+0x16/0x16
[ 348.849428] Code: 48 89 44 24 18 48 8b 44 24 18 4c 89 ef e8 ce 88 d9 e0 8b 83 2c 01 00 00 48 8b 93 38 01 00 00 80 e4 3f 80 cc 40 89 83 2c 01 00 00 <89> 42 14 0f b6 2d fe 46 00 00 48 8b 05 a7 85 34 e1 65 4c 8b 24
[ 348.849428] RIP [<ffffffffa02c1a48>] nvme_dev_shutdown+0x303/0x3e8 [nvme]
[ 348.849428] RSP <ffff88013afebcd8>
[ 348.849428] CR2: 0000000000000014
[ 348.849428] ---[ end trace f11db05b8d65bab0 ]---
next prev parent reply other threads:[~2015-10-22 18:10 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-22 12:03 nvme abort and reset fixes Christoph Hellwig
2015-10-22 12:03 ` [PATCH 1/9] nvme: only add a controller to dev_list after it's been fully initialized Christoph Hellwig
2015-10-22 12:03 ` [PATCH 2/9] nvme: don't take the I/O queue q_lock in nvme_timeout Christoph Hellwig
2015-10-22 12:03 ` [PATCH 3/9] nvme: merge nvme_abort_req and nvme_timeout Christoph Hellwig
2015-10-22 12:03 ` [PATCH 4/9] nvme: do not restart the request timeout if we're resetting the controller Christoph Hellwig
2015-10-22 16:27 ` Busch, Keith
2015-10-22 16:30 ` Christoph Hellwig
2015-10-22 17:15 ` Busch, Keith
2015-10-22 18:17 ` Christoph Hellwig
2015-10-22 12:03 ` [PATCH 5/9] nvme: simplify resets Christoph Hellwig
2015-10-22 12:03 ` [PATCH 6/9] nvme: abort requests on the reqeueue list when shutting down a controller Christoph Hellwig
2015-10-22 14:44 ` Busch, Keith
2015-10-22 14:58 ` Christoph Hellwig
2015-10-22 15:16 ` Busch, Keith
2015-10-22 16:27 ` Christoph Hellwig
2015-10-22 12:03 ` [PATCH 7/9] nvme: merge probe_work and reset_work Christoph Hellwig
2015-10-22 12:03 ` [PATCH 8/9] nvme: remove dead controllers from a work item Christoph Hellwig
2015-10-22 18:10 ` Busch, Keith [this message]
2015-10-22 18:12 ` Christoph Hellwig
2015-10-22 20:36 ` Busch, Keith
2015-10-23 5:57 ` Christoph Hellwig
2015-10-23 14:51 ` Busch, Keith
2015-10-23 19:31 ` Busch, Keith
2015-10-22 12:03 ` [PATCH 9/9] nvme: switch abort_limit to an atomic_t Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151022181014.GE21840@localhost.localdomain \
--to=keith.busch@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).