From: John Sperbeck <jsperbeck@google.com>
To: Bean Huo <beanhuo@micron.com>, Sagi Grimberg <sagi@grimberg.me>
Cc: khazhy@google.com, Jens Axboe <axboe@kernel.dk>, stable@vger.kernel.org
Subject: Crash in NVME tracing on 5.10LTS
Date: Tue, 9 Jan 2024 10:17:22 -0800 [thread overview]
Message-ID: <20240109181722.228783-1-jsperbeck@google.com> (raw)
With 5.10LTS (e.g., 5.10.206), on a machine using an NVME device, the
following tracing commands will trigger a crash due to a NULL pointer
dereference:
KDIR=/sys/kernel/debug/tracing
echo 1 > $KDIR/tracing_on
echo 1 > $KDIR/events/nvme/enable
echo "Waiting for trace events..."
cat $KDIR/trace_pipe
The backtrace looks something like this:
Call Trace:
<IRQ>
? __die_body+0x6b/0xb0
? __die+0x9e/0xb0
? no_context+0x3eb/0x460
? ttwu_do_activate+0xf0/0x120
? __bad_area_nosemaphore+0x157/0x200
? select_idle_sibling+0x2f/0x410
? bad_area_nosemaphore+0x13/0x20
? do_user_addr_fault+0x2ab/0x360
? exc_page_fault+0x69/0x180
? asm_exc_page_fault+0x1e/0x30
? trace_event_raw_event_nvme_complete_rq+0xba/0x170
? trace_event_raw_event_nvme_complete_rq+0xa3/0x170
nvme_complete_rq+0x168/0x170
nvme_pci_complete_rq+0x16c/0x1f0
nvme_handle_cqe+0xde/0x190
nvme_irq+0x78/0x100
__handle_irq_event_percpu+0x77/0x1e0
handle_irq_event+0x54/0xb0
handle_edge_irq+0xdf/0x230
asm_call_irq_on_stack+0xf/0x20
</IRQ>
common_interrupt+0x9e/0x150
asm_common_interrupt+0x1e/0x40
It looks to me like these two upstream commits were backported to 5.10:
679c54f2de67 ("nvme: use command_id instead of req->tag in trace_nvme_complete_rq()")
e7006de6c238 ("nvme: code command_id with a genctr for use-after-free validation")
But they depend on this upstream commit to initialize the 'cmd' field in
some cases:
f4b9e6c90c57 ("nvme: use driver pdu command for passthrough")
Does it sound like I'm on the right track? The 5.15LTS and later seems to be okay.
next reply other threads:[~2024-01-09 18:17 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-09 18:17 John Sperbeck [this message]
2024-01-11 9:46 ` Crash in NVME tracing on 5.10LTS Greg KH
2024-01-11 17:00 ` John Sperbeck
2024-01-11 19:38 ` Jens Axboe
2024-01-13 9:34 ` Greg KH
2024-01-13 16:11 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240109181722.228783-1-jsperbeck@google.com \
--to=jsperbeck@google.com \
--cc=axboe@kernel.dk \
--cc=beanhuo@micron.com \
--cc=khazhy@google.com \
--cc=sagi@grimberg.me \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox