* [PATCH blktests] check: add kmemleak support to blktests
@ 2026-01-13 9:51 Nilay Shroff
2026-01-14 11:31 ` Johannes Thumshirn
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Nilay Shroff @ 2026-01-13 9:51 UTC (permalink / raw)
To: linux-block; +Cc: shinichiro.kawasaki, yi.zhang, gjoyce, Nilay Shroff
Running blktests can also help uncover kernel memory leaks when the
kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the
blktests framework had no way to automatically detect or report such
leaks. Users typically had to manually setup kmemleak and trigger
scans after running tests[1][2].
This change integrates kmemleak support directly into the blktests
framework. Before running each test, the framework checks for the
presence of /sys/kernel/debug/kmemleak to determine whether kmemleak
is enabled for the running kernel. If available, before running a test,
any existing kmemleak reports are cleared to avoid false positives
from previous tests. After the test completes, the framework explicitly
triggers a kmemleak scan. If memory leaks are detected, they are written
to a per-test file at, "results/.../.../<test>.kmemleak" and the
corresponding test is marked as FAIL. Users can then inspect the
<test>.kmemleak file to analyze the reported leaks.
With this enhancement, blktests can automatically detect kernel memory
leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on
a per-test basis, removing the need for manual kmemleak setup and scans.
This should make it easier and faster to identify memory leaks
introduced by individual tests.
[1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
[2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
check | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/check b/check
index 6d77d8e..3a6e837 100755
--- a/check
+++ b/check
@@ -183,6 +183,36 @@ _check_dmesg() {
fi
}
+_setup_kmemleak() {
+ local f="/sys/kernel/debug/kmemleak"
+
+ if [[ ! -e $f || ! -r $f ]]; then
+ return 0
+ fi
+
+ echo clear > "$f"
+}
+
+_check_kmemleak() {
+ local kmemleak
+ local f="/sys/kernel/debug/kmemleak"
+
+ if [[ ! -e $f || ! -r $f ]]; then
+ return 0
+ fi
+
+ echo scan > "$f"
+ sleep 1
+ kmemleak=$(cat "$f")
+
+ if [[ -z $kmemleak ]]; then
+ return 0
+ fi
+
+ printf '%s\n' "$kmemleak" > "${seqres}.kmemleak"
+ return 1
+}
+
_read_last_test_run() {
local seqres="${RESULTS_DIR}/${TEST_NAME}"
@@ -377,6 +407,8 @@ _call_test() {
if [[ -v SKIP_REASONS ]]; then
TEST_RUN["status"]="not run"
else
+ _setup_kmemleak
+
if [[ -w /dev/kmsg ]]; then
local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}"
echo "$dmesg_marker" >> /dev/kmsg
@@ -414,6 +446,9 @@ _call_test() {
elif ! _check_dmesg "$dmesg_marker"; then
TEST_RUN["status"]=fail
TEST_RUN["reason"]=dmesg
+ elif ! _check_kmemleak; then
+ TEST_RUN["status"]=fail
+ TEST_RUN["reason"]=kmemleak
else
TEST_RUN["status"]=pass
fi
@@ -451,6 +486,18 @@ _call_test() {
print \" \" \$0
}" "${seqres}.dmesg"
;;
+ kmemleak)
+ echo " kmemleak detected:"
+ awk "
+ {
+ if (NR > 10) {
+ print \" ...\"
+ print \" (See '${seqres}.kmemleak' for the entire message)\"
+ exit
+ }
+ print \" \" \$0
+ }" "${seqres}.kmemleak"
+ ;;
esac
return 1
else
--
2.52.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH blktests] check: add kmemleak support to blktests
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
@ 2026-01-14 11:31 ` Johannes Thumshirn
2026-01-16 10:19 ` Shinichiro Kawasaki
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Johannes Thumshirn @ 2026-01-14 11:31 UTC (permalink / raw)
To: Nilay Shroff, linux-block@vger.kernel.org
Cc: Shinichiro Kawasaki, yi.zhang@redhat.com, gjoyce@ibm.com
Looks good to me,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH blktests] check: add kmemleak support to blktests
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
2026-01-14 11:31 ` Johannes Thumshirn
@ 2026-01-16 10:19 ` Shinichiro Kawasaki
2026-01-16 11:14 ` Yi Zhang
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Shinichiro Kawasaki @ 2026-01-16 10:19 UTC (permalink / raw)
To: Nilay Shroff
Cc: linux-block@vger.kernel.org, yi.zhang@redhat.com, gjoyce@ibm.com
On Jan 13, 2026 / 15:21, Nilay Shroff wrote:
> Running blktests can also help uncover kernel memory leaks when the
> kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the
> blktests framework had no way to automatically detect or report such
> leaks. Users typically had to manually setup kmemleak and trigger
> scans after running tests[1][2].
>
> This change integrates kmemleak support directly into the blktests
> framework. Before running each test, the framework checks for the
> presence of /sys/kernel/debug/kmemleak to determine whether kmemleak
> is enabled for the running kernel. If available, before running a test,
> any existing kmemleak reports are cleared to avoid false positives
> from previous tests. After the test completes, the framework explicitly
> triggers a kmemleak scan. If memory leaks are detected, they are written
> to a per-test file at, "results/.../.../<test>.kmemleak" and the
> corresponding test is marked as FAIL. Users can then inspect the
> <test>.kmemleak file to analyze the reported leaks.
>
> With this enhancement, blktests can automatically detect kernel memory
> leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on
> a per-test basis, removing the need for manual kmemleak setup and scans.
> This should make it easier and faster to identify memory leaks
> introduced by individual tests.
>
> [1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
> [2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
>
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Nilay, thank you very much. This idea is excellent, and the patch looks good
except one nit comment below. I also did trial runs, and confirmed memory leaks
can be detected and reported as test case failures. Great :)
> diff --git a/check b/check
> index 6d77d8e..3a6e837 100755
> --- a/check
> +++ b/check
[...]
> @@ -451,6 +486,18 @@ _call_test() {
> print \" \" \$0
> }" "${seqres}.dmesg"
> ;;
> + kmemleak)
> + echo " kmemleak detected:"
> + awk "
> + {
> + if (NR > 10) {
> + print \" ...\"
> + print \" (See '${seqres}.kmemleak' for the entire message)\"
> + exit
> + }
> + print \" \" \$0
> + }" "${seqres}.kmemleak"
> + ;;
Nit: the hunk above uses spaces for indent. Assuming there is no other review
comments on this patch, I will replace the spaces with tabs when I apply it. I
will wait a few more days to see someone makes comments or not. Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH blktests] check: add kmemleak support to blktests
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
2026-01-14 11:31 ` Johannes Thumshirn
2026-01-16 10:19 ` Shinichiro Kawasaki
@ 2026-01-16 11:14 ` Yi Zhang
2026-01-19 9:54 ` Daniel Wagner
2026-01-22 4:06 ` Shinichiro Kawasaki
4 siblings, 0 replies; 6+ messages in thread
From: Yi Zhang @ 2026-01-16 11:14 UTC (permalink / raw)
To: Nilay Shroff; +Cc: linux-block, shinichiro.kawasaki, gjoyce
On Tue, Jan 13, 2026 at 5:52 PM Nilay Shroff <nilay@linux.ibm.com> wrote:
>
> Running blktests can also help uncover kernel memory leaks when the
> kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the
> blktests framework had no way to automatically detect or report such
> leaks. Users typically had to manually setup kmemleak and trigger
> scans after running tests[1][2].
>
> This change integrates kmemleak support directly into the blktests
> framework. Before running each test, the framework checks for the
> presence of /sys/kernel/debug/kmemleak to determine whether kmemleak
> is enabled for the running kernel. If available, before running a test,
> any existing kmemleak reports are cleared to avoid false positives
> from previous tests. After the test completes, the framework explicitly
> triggers a kmemleak scan. If memory leaks are detected, they are written
> to a per-test file at, "results/.../.../<test>.kmemleak" and the
> corresponding test is marked as FAIL. Users can then inspect the
> <test>.kmemleak file to analyze the reported leaks.
>
> With this enhancement, blktests can automatically detect kernel memory
> leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on
> a per-test basis, removing the need for manual kmemleak setup and scans.
> This should make it easier and faster to identify memory leaks
> introduced by individual tests.
>
> [1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
> [2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
>
Thanks for the patch, that's really helpful to catch the kmemleak issue.
Reviewed-by: Yi Zhang <yi.zhang@redhat.com>
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
> ---
> check | 47 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 47 insertions(+)
>
> diff --git a/check b/check
> index 6d77d8e..3a6e837 100755
> --- a/check
> +++ b/check
> @@ -183,6 +183,36 @@ _check_dmesg() {
> fi
> }
>
> +_setup_kmemleak() {
> + local f="/sys/kernel/debug/kmemleak"
> +
> + if [[ ! -e $f || ! -r $f ]]; then
> + return 0
> + fi
> +
> + echo clear > "$f"
> +}
> +
> +_check_kmemleak() {
> + local kmemleak
> + local f="/sys/kernel/debug/kmemleak"
> +
> + if [[ ! -e $f || ! -r $f ]]; then
> + return 0
> + fi
> +
> + echo scan > "$f"
> + sleep 1
> + kmemleak=$(cat "$f")
> +
> + if [[ -z $kmemleak ]]; then
> + return 0
> + fi
> +
> + printf '%s\n' "$kmemleak" > "${seqres}.kmemleak"
> + return 1
> +}
> +
> _read_last_test_run() {
> local seqres="${RESULTS_DIR}/${TEST_NAME}"
>
> @@ -377,6 +407,8 @@ _call_test() {
> if [[ -v SKIP_REASONS ]]; then
> TEST_RUN["status"]="not run"
> else
> + _setup_kmemleak
> +
> if [[ -w /dev/kmsg ]]; then
> local dmesg_marker="run blktests $TEST_NAME at ${TEST_RUN["date"]}"
> echo "$dmesg_marker" >> /dev/kmsg
> @@ -414,6 +446,9 @@ _call_test() {
> elif ! _check_dmesg "$dmesg_marker"; then
> TEST_RUN["status"]=fail
> TEST_RUN["reason"]=dmesg
> + elif ! _check_kmemleak; then
> + TEST_RUN["status"]=fail
> + TEST_RUN["reason"]=kmemleak
> else
> TEST_RUN["status"]=pass
> fi
> @@ -451,6 +486,18 @@ _call_test() {
> print \" \" \$0
> }" "${seqres}.dmesg"
> ;;
> + kmemleak)
> + echo " kmemleak detected:"
> + awk "
> + {
> + if (NR > 10) {
> + print \" ...\"
> + print \" (See '${seqres}.kmemleak' for the entire message)\"
> + exit
> + }
> + print \" \" \$0
> + }" "${seqres}.kmemleak"
> + ;;
> esac
> return 1
> else
> --
> 2.52.0
>
--
Best Regards,
Yi Zhang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH blktests] check: add kmemleak support to blktests
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
` (2 preceding siblings ...)
2026-01-16 11:14 ` Yi Zhang
@ 2026-01-19 9:54 ` Daniel Wagner
2026-01-22 4:06 ` Shinichiro Kawasaki
4 siblings, 0 replies; 6+ messages in thread
From: Daniel Wagner @ 2026-01-19 9:54 UTC (permalink / raw)
To: Nilay Shroff; +Cc: linux-block, shinichiro.kawasaki, yi.zhang, gjoyce
On Tue, Jan 13, 2026 at 03:21:03PM +0530, Nilay Shroff wrote:
> Running blktests can also help uncover kernel memory leaks when the
> kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the
> blktests framework had no way to automatically detect or report such
> leaks. Users typically had to manually setup kmemleak and trigger
> scans after running tests[1][2].
>
> This change integrates kmemleak support directly into the blktests
> framework. Before running each test, the framework checks for the
> presence of /sys/kernel/debug/kmemleak to determine whether kmemleak
> is enabled for the running kernel. If available, before running a test,
> any existing kmemleak reports are cleared to avoid false positives
> from previous tests. After the test completes, the framework explicitly
> triggers a kmemleak scan. If memory leaks are detected, they are written
> to a per-test file at, "results/.../.../<test>.kmemleak" and the
> corresponding test is marked as FAIL. Users can then inspect the
> <test>.kmemleak file to analyze the reported leaks.
>
> With this enhancement, blktests can automatically detect kernel memory
> leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on
> a per-test basis, removing the need for manual kmemleak setup and scans.
> This should make it easier and faster to identify memory leaks
> introduced by individual tests.
>
> [1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
> [2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
Nice!
Reviewed-by: Daniel Wagner <dwagner@suse.de>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH blktests] check: add kmemleak support to blktests
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
` (3 preceding siblings ...)
2026-01-19 9:54 ` Daniel Wagner
@ 2026-01-22 4:06 ` Shinichiro Kawasaki
4 siblings, 0 replies; 6+ messages in thread
From: Shinichiro Kawasaki @ 2026-01-22 4:06 UTC (permalink / raw)
To: Nilay Shroff
Cc: linux-block@vger.kernel.org, yi.zhang@redhat.com, gjoyce@ibm.com
On Jan 13, 2026 / 15:21, Nilay Shroff wrote:
> Running blktests can also help uncover kernel memory leaks when the
> kernel is built with CONFIG_DEBUG_KMEMLEAK. However, until now the
> blktests framework had no way to automatically detect or report such
> leaks. Users typically had to manually setup kmemleak and trigger
> scans after running tests[1][2].
>
> This change integrates kmemleak support directly into the blktests
> framework. Before running each test, the framework checks for the
> presence of /sys/kernel/debug/kmemleak to determine whether kmemleak
> is enabled for the running kernel. If available, before running a test,
> any existing kmemleak reports are cleared to avoid false positives
> from previous tests. After the test completes, the framework explicitly
> triggers a kmemleak scan. If memory leaks are detected, they are written
> to a per-test file at, "results/.../.../<test>.kmemleak" and the
> corresponding test is marked as FAIL. Users can then inspect the
> <test>.kmemleak file to analyze the reported leaks.
>
> With this enhancement, blktests can automatically detect kernel memory
> leaks (if kerel is configured with CONFIG_DEBUG_KMEMLEAK support) on
> a per-test basis, removing the need for manual kmemleak setup and scans.
> This should make it easier and faster to identify memory leaks
> introduced by individual tests.
>
> [1] https://lore.kernel.org/all/CAHj4cs8oJFvz=daCvjHM5dYCNQH4UXwSySPPU4v-WHce_kZXZA@mail.gmail.com/
> [2] https://lore.kernel.org/all/CAHj4cs9wv3SdPo+N01Fw2SHBYDs9tj2M_e1-GdQOkRy=DsBB1w@mail.gmail.com/
>
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
I applied this patch replacing the spaces with tabs. Nilay, thank you for the
contribution :)
I ran whole blktests with this change and observed two kmemleak failures at
nvme/061 and rnbd/002. FYI, here share the failure symptoms. Debug support the
nvme/061 failure will be appreciated. I plan to post a fix patch for the
rnbd/002 failure.
1) nvme/061 (tr=rdma, tr=tcp)
When nvme/061 is ran for rdma transport and tcp transport in sequence,
kemeleak error below was observed at the run for tcp transport. The
backtrace indicates the leak happened for rdma transport.
unreferenced object 0xffff88812b7d8cc0 (size 32):
comm "kworker/2:0H", pid 66707, jiffies 4298507259
hex dump (first 32 bytes):
82 69 85 06 00 ea ff ff 00 00 00 00 00 10 00 00 .i..............
00 60 5a a1 81 88 ff ff 00 10 00 00 00 00 00 00 .`Z.............
backtrace (crc e0e63884):
__kmalloc_noprof+0x62f/0x8b0
sgl_alloc_order+0x74/0x330
nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
0xffffffffc1d60f0d
0xffffffffc1d67064
__ib_process_cq+0x14f/0x3e0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x868/0x1490
worker_thread+0x5ee/0xfd0
kthread+0x3af/0x770
ret_from_fork+0x55c/0x810
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff88812b7d8a00 (size 32):
comm "kworker/2:0H", pid 66707, jiffies 4298507259
hex dump (first 32 bytes):
c2 65 85 06 00 ea ff ff 00 00 00 00 00 10 00 00 .e..............
00 70 59 a1 81 88 ff ff 00 10 00 00 00 00 00 00 .pY.............
backtrace (crc e120dc78):
__kmalloc_noprof+0x62f/0x8b0
sgl_alloc_order+0x74/0x330
nvmet_req_alloc_sgls+0x283/0x4e0 [nvmet]
0xffffffffc1d60f0d
0xffffffffc1d67064
__ib_process_cq+0x14f/0x3e0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x868/0x1490
worker_thread+0x5ee/0xfd0
kthread+0x3af/0x770
ret_from_fork+0x55c/0x810
ret_from_fork_asm+0x1a/0x30
...
2) rnbd/002
The kememleak message was as follows.
unreferenced object 0xffff88826c5f8720 (size 8):
comm "check", pid 145048, jiffies 4301507712
hex dump (first 8 bytes):
72 6e 62 64 00 15 b3 a6 rnbd....
backtrace (crc f6c244b4):
__kmalloc_node_track_caller_noprof+0x69d/0x900
kstrdup+0x42/0xc0
kobject_set_name_vargs+0x44/0x110
kobject_init_and_add+0xcf/0x150
rnbd_clt_map_device_store.cold+0x164/0x792 [rnbd_client]
kernfs_fop_write_iter+0x3d6/0x5e0
vfs_write+0x52c/0xf80
ksys_write+0xfb/0x200
do_syscall_64+0x95/0x540
entry_SYSCALL_64_after_hwframe+0x76/0x7e
I tried out a fix patch below, and it avoided the kmemleak
message.
diff --git a/drivers/block/rnbd/rnbd-clt.c b/drivers/block/rnbd/rnbd-clt.c
index 8194a970f002..db6890121012 100644
--- a/drivers/block/rnbd/rnbd-clt.c
+++ b/drivers/block/rnbd/rnbd-clt.c
@@ -60,6 +60,7 @@ static void rnbd_clt_put_dev(struct rnbd_clt_dev *dev)
kfree(dev->pathname);
rnbd_clt_put_sess(dev->sess);
mutex_destroy(&dev->lock);
+ kobject_put(&dev->kobj);
kfree(dev);
}
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-01-22 4:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-13 9:51 [PATCH blktests] check: add kmemleak support to blktests Nilay Shroff
2026-01-14 11:31 ` Johannes Thumshirn
2026-01-16 10:19 ` Shinichiro Kawasaki
2026-01-16 11:14 ` Yi Zhang
2026-01-19 9:54 ` Daniel Wagner
2026-01-22 4:06 ` Shinichiro Kawasaki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox