* Re: [PATCH v3 2/2] lib/crypto: tests: Add KUnit tests for ML-DSA verification
From: David Howells @ 2025-12-03 9:01 UTC (permalink / raw)
To: Eric Biggers
Cc: dhowells, linux-crypto, Herbert Xu, Luis Chamberlain, Petr Pavlu,
Daniel Gomez, Sami Tolvanen, Jason A . Donenfeld, Ard Biesheuvel,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251203072844.484893-3-ebiggers@kernel.org>
I'm seeing:
ERROR: modpost: module mldsa_kunit uses symbol mldsa_use_hint from namespace EXPORTED_FOR_KUNIT_TESTING, but does not import it.
I have this in my .config:
CONFIG_CRYPTO_LIB_MLDSA=m
CONFIG_CRYPTO_LIB_MLDSA_KUNIT_TEST=m
The problem appears to be here:
#if IS_ENABLED(CONFIG_CRYPTO_LIB_MLDSA_KUNIT_TEST)
/* Allow the __always_inline function use_hint() to be unit-tested. */
s32 mldsa_use_hint(u8 h, s32 r, s32 gamma2)
{
return use_hint(h, r, gamma2);
}
EXPORT_SYMBOL_IF_KUNIT(mldsa_use_hint);
#endif
It works if EXPORT_SYMBOL_IF_KUNIT() is changed to EXPORT_SYMBOL_GPL().
David
^ permalink raw reply
* Re: [PATCH V2] mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction
From: Vlastimil Babka @ 2025-12-03 9:05 UTC (permalink / raw)
To: Harry Yoo
Cc: surenb, Liam.Howlett, cl, rientjes, roman.gushchin, urezki,
sidhartha.kumar, linux-mm, linux-kernel, rcu, maple-tree,
linux-modules, mcgrof, petr.pavlu, samitolvanen, atomlin,
lucas.demarchi, akpm, jonathanh, stable, Daniel Gomez
In-Reply-To: <20251202101626.783736-1-harry.yoo@oracle.com>
On 12/2/25 11:16, Harry Yoo wrote:
> Currently, kvfree_rcu_barrier() flushes RCU sheaves across all slab
> caches when a cache is destroyed. This is unnecessary; only the RCU
> sheaves belonging to the cache being destroyed need to be flushed.
>
> As suggested by Vlastimil Babka, introduce a weaker form of
> kvfree_rcu_barrier() that operates on a specific slab cache.
>
> Factor out flush_rcu_sheaves_on_cache() from flush_all_rcu_sheaves() and
> call it from flush_all_rcu_sheaves() and kvfree_rcu_barrier_on_cache().
>
> Call kvfree_rcu_barrier_on_cache() instead of kvfree_rcu_barrier() on
> cache destruction.
>
> The performance benefit is evaluated on a 12 core 24 threads AMD Ryzen
> 5900X machine (1 socket), by loading slub_kunit module.
>
> Before:
> Total calls: 19
> Average latency (us): 18127
> Total time (us): 344414
>
> After:
> Total calls: 19
> Average latency (us): 10066
> Total time (us): 191264
>
> Two performance regression have been reported:
> - stress module loader test's runtime increases by 50-60% (Daniel)
> - internal graphics test's runtime on Tegra23 increases by 35% (Jon)
>
> They are fixed by this change.
>
> Suggested-by: Vlastimil Babka <vbabka@suse.cz>
> Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
> Cc: <stable@vger.kernel.org>
> Link: https://lore.kernel.org/linux-mm/1bda09da-93be-4737-aef0-d47f8c5c9301@suse.cz
> Reported-and-tested-by: Daniel Gomez <da.gomez@samsung.com>
> Closes: https://lore.kernel.org/linux-mm/0406562e-2066-4cf8-9902-b2b0616dd742@kernel.org
> Reported-and-tested-by: Jon Hunter <jonathanh@nvidia.com>
> Closes: https://lore.kernel.org/linux-mm/e988eff6-1287-425e-a06c-805af5bbf262@nvidia.com
> Signed-off-by: Harry Yoo <harry.yoo@oracle.com>
Thanks a lot! Added to slab/for-next-fixes
^ permalink raw reply
* Re: [PATCH v3 0/2] lib/crypto: ML-DSA verification support
From: David Howells @ 2025-12-03 10:09 UTC (permalink / raw)
To: Eric Biggers
Cc: dhowells, linux-crypto, Herbert Xu, Luis Chamberlain, Petr Pavlu,
Daniel Gomez, Sami Tolvanen, Jason A . Donenfeld, Ard Biesheuvel,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, keyrings,
linux-modules, linux-kernel
In-Reply-To: <20251203072844.484893-1-ebiggers@kernel.org>
Give or take fixing the symbol export build failure:
Reviewed-by: David Howells <dhowells@redhat.com>
Tested-by: David Howells <dhowells@redhat.com>
^ permalink raw reply
* Re: [PATCH v3 0/2] lib/crypto: ML-DSA verification support
From: Harald Freudenberger @ 2025-12-03 14:19 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, David Howells, Herbert Xu, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, Jason A . Donenfeld,
Ard Biesheuvel, Stephan Mueller, Lukas Wunner, Ignat Korchagin,
keyrings, linux-modules, linux-kernel
In-Reply-To: <20251203072844.484893-1-ebiggers@kernel.org>
On 2025-12-03 08:28, Eric Biggers wrote:
> This series can also be retrieved from:
>
> git fetch
> https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git
> mldsa-v3
>
> This series adds support for verifying ML-DSA signatures to
> lib/crypto/.
> Patch 1 is the ML-DSA implementation itself. See that for full
> details.
> Patch 2 adds the KUnit test suite.
>
> The initial use case for this will be kernel module signature
> verification. For more details, see David Howells' patchset
> https://lore.kernel.org/linux-crypto/20251120104439.2620205-1-dhowells@redhat.com/
>
> Changed in v3:
> - Moved SHAKE128 block buffer off the stack
> - Reworked use_hint() again, simplifying it further
> - Added unit test for use_hint()
> - Moved some of the reduction logic into invntt_and_mul_2_32() and
> simplified it slightly
> - Updated Zq_mult() to not rely on signed integer overflow having
> defined behavior, though this is unnecessary in the kernel
> - Formatted the documented return values into a list
> - Other minor tweaks
>
> Changed in v2:
> - Reworked the KUnit test suite
> - Improved commit messages and comments
> - Added missing MODULE_DESCRIPTION() and MODULE_LICENSE()
> - Made the return values of mldsa_verify() differentiate between an
> input being malformed and the "real" signature check failing
> - Refactored w1 encoding into a helper function
> - Used kfree() instead of kfree_sensitive()
> - Avoided unusal C syntax by accessing the hint vector via 'u8 *'
> - Reworked use_hint() to be better optimized and documented
>
> Eric Biggers (2):
> lib/crypto: Add ML-DSA verification support
> lib/crypto: tests: Add KUnit tests for ML-DSA verification
>
> include/crypto/mldsa.h | 60 +
> lib/crypto/Kconfig | 7 +
> lib/crypto/Makefile | 5 +
> lib/crypto/mldsa.c | 652 ++++++++++
> lib/crypto/tests/Kconfig | 9 +
> lib/crypto/tests/Makefile | 1 +
> lib/crypto/tests/mldsa-testvecs.h | 1887 +++++++++++++++++++++++++++++
> lib/crypto/tests/mldsa_kunit.c | 437 +++++++
> 8 files changed, 3058 insertions(+)
> create mode 100644 include/crypto/mldsa.h
> create mode 100644 lib/crypto/mldsa.c
> create mode 100644 lib/crypto/tests/mldsa-testvecs.h
> create mode 100644 lib/crypto/tests/mldsa_kunit.c
>
>
> base-commit: 3f9f0252130e7dd60d41be0802bf58f6471c691d
Hi Eric, I pulled your branch but it does not build (on s390):
[root@b83lp71 ebiggers-linux]# make
CALL scripts/checksyscalls.sh
MODPOST Module.symvers
ERROR: modpost: module mldsa_kunit uses symbol mldsa_use_hint from
namespace EXPORTED_FOR_KUNIT_TESTING, but does not import it.
make[2]: *** [scripts/Makefile.modpost:147: Module.symvers] Error 1
make[1]: *** [/ext/ebiggers-linux/Makefile:1963: modpost] Error 2
make: *** [Makefile:248: __sub-make] Error 2
[root@b83lp71 ebiggers-linux]#
any ideas?
Harald Freudenberger
^ permalink raw reply
* Re: [PATCH v3 0/2] lib/crypto: ML-DSA verification support
From: David Howells @ 2025-12-03 14:40 UTC (permalink / raw)
To: freude
Cc: dhowells, Eric Biggers, linux-crypto, Herbert Xu,
Luis Chamberlain, Petr Pavlu, Daniel Gomez, Sami Tolvanen,
Jason A . Donenfeld, Ard Biesheuvel, Stephan Mueller,
Lukas Wunner, Ignat Korchagin, keyrings, linux-modules,
linux-kernel
In-Reply-To: <94cb715bd8782b93e10a285c6dc6ec58@linux.ibm.com>
Harald Freudenberger <freude@linux.ibm.com> wrote:
> ERROR: modpost: module mldsa_kunit uses symbol mldsa_use_hint from namespace
> EXPORTED_FOR_KUNIT_TESTING, but does not import it.
Change EXPORT_SYMBOL_FOR_KUNIT() in patch 1 to EXPORT_SYMBOL_GPL().
David
^ permalink raw reply
* [PATCH v2 0/4] Unload linux/kernel.h
From: Yury Norov (NVIDIA) @ 2025-12-03 16:23 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Christophe Leroy, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Yury Norov (NVIDIA)
kernel.h hosts declarations that can be placed better.
Testing (0-Day):
alpha defconfig gcc-15.1.0
arc defconfig gcc-15.1.0
arc randconfig-001-20251202 gcc-8.5.0
arc randconfig-002-20251202 gcc-12.5.0
arm defconfig clang-22
arm randconfig-001-20251202 clang-17
arm randconfig-002-20251202 clang-22
arm randconfig-003-20251202 clang-22
arm randconfig-004-20251202 gcc-14.3.0
arm spear13xx_defconfig gcc-15.1.0
arm64 defconfig gcc-15.1.0
arm64 randconfig-001-20251202 clang-22
arm64 randconfig-002-20251202 clang-22
arm64 randconfig-003-20251202 clang-16
arm64 randconfig-004-20251202 clang-22
csky defconfig gcc-15.1.0
csky randconfig-001-20251202 gcc-13.4.0
csky randconfig-002-20251202 gcc-15.1.0
hexagon defconfig clang-22
hexagon randconfig-001-20251202 clang-22
hexagon randconfig-002-20251202 clang-22
i386 allmodconfig gcc-14
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001-20251202 clang-20
i386 buildonly-randconfig-002-20251202 gcc-14
i386 buildonly-randconfig-003-20251202 clang-20
i386 buildonly-randconfig-004-20251202 clang-20
i386 buildonly-randconfig-005-20251202 clang-20
i386 buildonly-randconfig-006-20251202 gcc-14
i386 defconfig clang-20
i386 randconfig-001-20251202 clang-20
i386 randconfig-002-20251202 gcc-14
i386 randconfig-003-20251202 gcc-14
i386 randconfig-004-20251202 clang-20
i386 randconfig-005-20251202 gcc-14
i386 randconfig-006-20251202 clang-20
i386 randconfig-007-20251202 clang-20
i386 randconfig-011-20251202 clang-20
i386 randconfig-012-20251202 clang-20
i386 randconfig-013-20251202 gcc-14
i386 randconfig-014-20251202 gcc-14
i386 randconfig-015-20251202 gcc-14
i386 randconfig-016-20251202 clang-20
i386 randconfig-017-20251202 clang-20
loongarch defconfig clang-19
loongarch randconfig-001-20251202 gcc-15.1.0
loongarch randconfig-002-20251202 clang-22
m68k defconfig gcc-15.1.0
microblaze allyesconfig gcc-15.1.0
microblaze defconfig gcc-15.1.0
nios2 10m50_defconfig gcc-11.5.0
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig gcc-11.5.0
nios2 defconfig gcc-11.5.0
nios2 randconfig-001-20251202 gcc-8.5.0
nios2 randconfig-002-20251202 gcc-8.5.0
openrisc allmodconfig gcc-15.1.0
openrisc allnoconfig gcc-15.1.0
openrisc defconfig gcc-15.1.0
parisc allnoconfig gcc-15.1.0
parisc allyesconfig gcc-15.1.0
parisc defconfig gcc-15.1.0
parisc generic-32bit_defconfig gcc-15.1.0
parisc randconfig-001-20251202 gcc-9.5.0
parisc randconfig-002-20251202 gcc-15.1.0
parisc64 defconfig gcc-15.1.0
powerpc allmodconfig gcc-15.1.0
powerpc allnoconfig gcc-15.1.0
powerpc randconfig-001-20251202 clang-22
powerpc randconfig-002-20251202 gcc-10.5.0
powerpc64 randconfig-001-20251202 clang-22
powerpc64 randconfig-002-20251202 clang-22
riscv allnoconfig gcc-15.1.0
riscv defconfig clang-22
riscv nommu_k210_defconfig clang-22
riscv randconfig-001-20251202 gcc-9.5.0
riscv randconfig-002-20251202 clang-22
s390 allnoconfig clang-22
s390 defconfig clang-22
s390 randconfig-001-20251202 clang-17
s390 randconfig-002-20251202 gcc-13.4.0
sh allnoconfig gcc-15.1.0
sh defconfig gcc-15.1.0
sh randconfig-001-20251202 gcc-15.1.0
sh randconfig-002-20251202 gcc-12.5.0
sh rsk7269_defconfig gcc-15.1.0
sh sh7710voipgw_defconfig gcc-15.1.0
sparc allnoconfig gcc-15.1.0
sparc defconfig gcc-15.1.0
sparc randconfig-001-20251202 gcc-8.5.0
sparc randconfig-002-20251202 gcc-8.5.0
sparc64 allmodconfig clang-22
sparc64 defconfig clang-20
sparc64 randconfig-001-20251202 gcc-8.5.0
sparc64 randconfig-002-20251202 gcc-10.5.0
um allnoconfig clang-22
um defconfig clang-22
um i386_defconfig gcc-14
um randconfig-001-20251202 clang-17
um randconfig-002-20251202 gcc-14
um x86_64_defconfig clang-22
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20251202 gcc-14
x86_64 buildonly-randconfig-002-20251202 gcc-14
x86_64 buildonly-randconfig-003-20251202 clang-20
x86_64 buildonly-randconfig-004-20251202 gcc-13
x86_64 buildonly-randconfig-005-20251202 gcc-14
x86_64 buildonly-randconfig-006-20251202 gcc-13
x86_64 defconfig gcc-14
x86_64 randconfig-011-20251202 clang-20
x86_64 randconfig-012-20251202 gcc-13
x86_64 randconfig-013-20251202 gcc-14
x86_64 randconfig-014-20251202 clang-20
x86_64 randconfig-015-20251202 clang-20
x86_64 randconfig-016-20251202 clang-20
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig gcc-15.1.0
xtensa randconfig-001-20251202 gcc-11.5.0
xtensa randconfig-002-20251202 gcc-11.5.0
v1: https://lore.kernel.org/all/20251129195304.204082-1-yury.norov@gmail.com/
v2:
- more historical context for STACK_MAGIC removal (Christophe);
- place STACK_MAGIC in i915_selftest.h (Jan);
- new patch #2: moduleparam: include required headers explicitly (Petr);
- align Documentation/filesystems/sysfs.rst (Andy);
- more motivation for linux/tracing.h (Andy, Steven);
Yury Norov (NVIDIA) (4):
kernel.h: drop STACK_MAGIC macro
moduleparam: include required headers explicitly
kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
tracing: move tracing declarations from kernel.h to a dedicated header
Documentation/filesystems/sysfs.rst | 2 +-
MAINTAINERS | 1 +
.../drm/i915/gt/selftest_ring_submission.c | 1 +
drivers/gpu/drm/i915/i915_selftest.h | 2 +
include/linux/kernel.h | 209 +-----------------
include/linux/moduleparam.h | 7 +-
include/linux/sysfs.h | 13 ++
include/linux/tracing.h | 203 +++++++++++++++++
8 files changed, 228 insertions(+), 210 deletions(-)
create mode 100644 include/linux/tracing.h
--
2.43.0
^ permalink raw reply
* [PATCH v2 1/4] kernel.h: drop STACK_MAGIC macro
From: Yury Norov (NVIDIA) @ 2025-12-03 16:23 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Christophe Leroy, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Yury Norov (NVIDIA), Jani Nikula
In-Reply-To: <20251203162329.280182-1-yury.norov@gmail.com>
The macro was introduced in 1994, v1.0.4, for stacks protection. Since
that, people found better ways to protect stacks, and now the macro is
only used by i915 selftests. Move it to a local header and drop from
the kernel.h.
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Acked-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
---
drivers/gpu/drm/i915/gt/selftest_ring_submission.c | 1 +
drivers/gpu/drm/i915/i915_selftest.h | 2 ++
include/linux/kernel.h | 2 --
3 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/gt/selftest_ring_submission.c b/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
index 87ceb0f374b6..600333ae6c8c 100644
--- a/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
+++ b/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
@@ -3,6 +3,7 @@
* Copyright © 2020 Intel Corporation
*/
+#include "i915_selftest.h"
#include "intel_engine_pm.h"
#include "selftests/igt_flush_test.h"
diff --git a/drivers/gpu/drm/i915/i915_selftest.h b/drivers/gpu/drm/i915/i915_selftest.h
index bdf3e22c0a34..72922028f4ba 100644
--- a/drivers/gpu/drm/i915/i915_selftest.h
+++ b/drivers/gpu/drm/i915/i915_selftest.h
@@ -26,6 +26,8 @@
#include <linux/types.h>
+#define STACK_MAGIC 0xdeadbeef
+
struct pci_dev;
struct drm_i915_private;
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 5b46924fdff5..61d63c57bc2d 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -40,8 +40,6 @@
#include <uapi/linux/kernel.h>
-#define STACK_MAGIC 0xdeadbeef
-
struct completion;
struct user;
--
2.43.0
^ permalink raw reply related
* [PATCH v2 2/4] moduleparam: include required headers explicitly
From: Yury Norov (NVIDIA) @ 2025-12-03 16:23 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Christophe Leroy, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Yury Norov (NVIDIA)
In-Reply-To: <20251203162329.280182-1-yury.norov@gmail.com>
The following patch drops moduleparam.h dependency on kernel.h. In
preparation to it, list all the required headers explicitly.
Suggested-by: Petr Pavlu <petr.pavlu@suse.com>
CC: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
---
include/linux/moduleparam.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/include/linux/moduleparam.h b/include/linux/moduleparam.h
index 6907aedc4f74..ca7c8107c7c8 100644
--- a/include/linux/moduleparam.h
+++ b/include/linux/moduleparam.h
@@ -2,9 +2,14 @@
#ifndef _LINUX_MODULE_PARAMS_H
#define _LINUX_MODULE_PARAMS_H
/* (C) Copyright 2001, 2002 Rusty Russell IBM Corporation */
+
+#include <linux/array_size.h>
+#include <linux/build_bug.h>
+#include <linux/compiler.h>
#include <linux/init.h>
#include <linux/stringify.h>
#include <linux/kernel.h>
+#include <linux/types.h>
/*
* The maximum module name length, including the NUL byte.
--
2.43.0
^ permalink raw reply related
* [PATCH v2 3/4] kernel.h: move VERIFY_OCTAL_PERMISSIONS() to sysfs.h
From: Yury Norov (NVIDIA) @ 2025-12-03 16:23 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Christophe Leroy, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Yury Norov (NVIDIA)
In-Reply-To: <20251203162329.280182-1-yury.norov@gmail.com>
The macro is related to sysfs, but is defined in kernel.h. Move it to
the proper header, and unload the generic kernel.h.
Now that the macro is removed from kernel.h, linux/moduleparam.h is
decoupled, and kernel.h inclusion can be removed.
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
---
Documentation/filesystems/sysfs.rst | 2 +-
include/linux/kernel.h | 12 ------------
include/linux/moduleparam.h | 2 +-
include/linux/sysfs.h | 13 +++++++++++++
4 files changed, 15 insertions(+), 14 deletions(-)
diff --git a/Documentation/filesystems/sysfs.rst b/Documentation/filesystems/sysfs.rst
index 2703c04af7d0..ffcef4d6bc8d 100644
--- a/Documentation/filesystems/sysfs.rst
+++ b/Documentation/filesystems/sysfs.rst
@@ -120,7 +120,7 @@ is equivalent to doing::
.store = store_foo,
};
-Note as stated in include/linux/kernel.h "OTHER_WRITABLE? Generally
+Note as stated in include/linux/sysfs.h "OTHER_WRITABLE? Generally
considered a bad idea." so trying to set a sysfs file writable for
everyone will fail reverting to RO mode for "Others".
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 61d63c57bc2d..5b879bfea948 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -389,16 +389,4 @@ static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
# define REBUILD_DUE_TO_DYNAMIC_FTRACE
#endif
-/* Permissions on a sysfs file: you didn't miss the 0 prefix did you? */
-#define VERIFY_OCTAL_PERMISSIONS(perms) \
- (BUILD_BUG_ON_ZERO((perms) < 0) + \
- BUILD_BUG_ON_ZERO((perms) > 0777) + \
- /* USER_READABLE >= GROUP_READABLE >= OTHER_READABLE */ \
- BUILD_BUG_ON_ZERO((((perms) >> 6) & 4) < (((perms) >> 3) & 4)) + \
- BUILD_BUG_ON_ZERO((((perms) >> 3) & 4) < ((perms) & 4)) + \
- /* USER_WRITABLE >= GROUP_WRITABLE */ \
- BUILD_BUG_ON_ZERO((((perms) >> 6) & 2) < (((perms) >> 3) & 2)) + \
- /* OTHER_WRITABLE? Generally considered a bad idea. */ \
- BUILD_BUG_ON_ZERO((perms) & 2) + \
- (perms))
#endif
diff --git a/include/linux/moduleparam.h b/include/linux/moduleparam.h
index ca7c8107c7c8..dd2d990b2611 100644
--- a/include/linux/moduleparam.h
+++ b/include/linux/moduleparam.h
@@ -8,7 +8,7 @@
#include <linux/compiler.h>
#include <linux/init.h>
#include <linux/stringify.h>
-#include <linux/kernel.h>
+#include <linux/sysfs.h>
#include <linux/types.h>
/*
diff --git a/include/linux/sysfs.h b/include/linux/sysfs.h
index 9a25a2911652..15ee3ef33991 100644
--- a/include/linux/sysfs.h
+++ b/include/linux/sysfs.h
@@ -798,4 +798,17 @@ static inline void sysfs_put(struct kernfs_node *kn)
kernfs_put(kn);
}
+/* Permissions on a sysfs file: you didn't miss the 0 prefix did you? */
+#define VERIFY_OCTAL_PERMISSIONS(perms) \
+ (BUILD_BUG_ON_ZERO((perms) < 0) + \
+ BUILD_BUG_ON_ZERO((perms) > 0777) + \
+ /* USER_READABLE >= GROUP_READABLE >= OTHER_READABLE */ \
+ BUILD_BUG_ON_ZERO((((perms) >> 6) & 4) < (((perms) >> 3) & 4)) + \
+ BUILD_BUG_ON_ZERO((((perms) >> 3) & 4) < ((perms) & 4)) + \
+ /* USER_WRITABLE >= GROUP_WRITABLE */ \
+ BUILD_BUG_ON_ZERO((((perms) >> 6) & 2) < (((perms) >> 3) & 2)) + \
+ /* OTHER_WRITABLE? Generally considered a bad idea. */ \
+ BUILD_BUG_ON_ZERO((perms) & 2) + \
+ (perms))
+
#endif /* _SYSFS_H_ */
--
2.43.0
^ permalink raw reply related
* [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Yury Norov (NVIDIA) @ 2025-12-03 16:23 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Andy Shevchenko, Christophe Leroy, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Yury Norov (NVIDIA)
In-Reply-To: <20251203162329.280182-1-yury.norov@gmail.com>
Tracing is a half of the kernel.h in terms of LOCs, although it's
a self-consistent part. It is intended for quick debugging purposes
and isn't used by the normal tracing utilities.
Move it to a separate header. If someone needs to just throw a
trace_printk() in their driver, they will not have to pull all
the heavy tracing machinery.
This is a pure move, except for removing a few 'extern's.
Acked-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
---
MAINTAINERS | 1 +
include/linux/kernel.h | 195 +-------------------------------------
include/linux/tracing.h | 203 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 205 insertions(+), 194 deletions(-)
create mode 100644 include/linux/tracing.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 3468f6827b13..107cadb613da 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -26080,6 +26080,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
F: Documentation/trace/*
F: fs/tracefs/
F: include/linux/trace*.h
+F: include/linux/tracing.h
F: include/trace/
F: kernel/trace/
F: kernel/tracepoint.c
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 5b879bfea948..2e9d65177f45 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -32,6 +32,7 @@
#include <linux/build_bug.h>
#include <linux/sprintf.h>
#include <linux/static_call_types.h>
+#include <linux/tracing.h>
#include <linux/instruction_pointer.h>
#include <linux/util_macros.h>
#include <linux/wordpart.h>
@@ -190,200 +191,6 @@ enum system_states {
};
extern enum system_states system_state;
-/*
- * General tracing related utility functions - trace_printk(),
- * tracing_on/tracing_off and tracing_start()/tracing_stop
- *
- * Use tracing_on/tracing_off when you want to quickly turn on or off
- * tracing. It simply enables or disables the recording of the trace events.
- * This also corresponds to the user space /sys/kernel/tracing/tracing_on
- * file, which gives a means for the kernel and userspace to interact.
- * Place a tracing_off() in the kernel where you want tracing to end.
- * From user space, examine the trace, and then echo 1 > tracing_on
- * to continue tracing.
- *
- * tracing_stop/tracing_start has slightly more overhead. It is used
- * by things like suspend to ram where disabling the recording of the
- * trace is not enough, but tracing must actually stop because things
- * like calling smp_processor_id() may crash the system.
- *
- * Most likely, you want to use tracing_on/tracing_off.
- */
-
-enum ftrace_dump_mode {
- DUMP_NONE,
- DUMP_ALL,
- DUMP_ORIG,
- DUMP_PARAM,
-};
-
-#ifdef CONFIG_TRACING
-void tracing_on(void);
-void tracing_off(void);
-int tracing_is_on(void);
-void tracing_snapshot(void);
-void tracing_snapshot_alloc(void);
-
-extern void tracing_start(void);
-extern void tracing_stop(void);
-
-static inline __printf(1, 2)
-void ____trace_printk_check_format(const char *fmt, ...)
-{
-}
-#define __trace_printk_check_format(fmt, args...) \
-do { \
- if (0) \
- ____trace_printk_check_format(fmt, ##args); \
-} while (0)
-
-/**
- * trace_printk - printf formatting in the ftrace buffer
- * @fmt: the printf format for printing
- *
- * Note: __trace_printk is an internal function for trace_printk() and
- * the @ip is passed in via the trace_printk() macro.
- *
- * This function allows a kernel developer to debug fast path sections
- * that printk is not appropriate for. By scattering in various
- * printk like tracing in the code, a developer can quickly see
- * where problems are occurring.
- *
- * This is intended as a debugging tool for the developer only.
- * Please refrain from leaving trace_printks scattered around in
- * your code. (Extra memory is used for special buffers that are
- * allocated when trace_printk() is used.)
- *
- * A little optimization trick is done here. If there's only one
- * argument, there's no need to scan the string for printf formats.
- * The trace_puts() will suffice. But how can we take advantage of
- * using trace_puts() when trace_printk() has only one argument?
- * By stringifying the args and checking the size we can tell
- * whether or not there are args. __stringify((__VA_ARGS__)) will
- * turn into "()\0" with a size of 3 when there are no args, anything
- * else will be bigger. All we need to do is define a string to this,
- * and then take its size and compare to 3. If it's bigger, use
- * do_trace_printk() otherwise, optimize it to trace_puts(). Then just
- * let gcc optimize the rest.
- */
-
-#define trace_printk(fmt, ...) \
-do { \
- char _______STR[] = __stringify((__VA_ARGS__)); \
- if (sizeof(_______STR) > 3) \
- do_trace_printk(fmt, ##__VA_ARGS__); \
- else \
- trace_puts(fmt); \
-} while (0)
-
-#define do_trace_printk(fmt, args...) \
-do { \
- static const char *trace_printk_fmt __used \
- __section("__trace_printk_fmt") = \
- __builtin_constant_p(fmt) ? fmt : NULL; \
- \
- __trace_printk_check_format(fmt, ##args); \
- \
- if (__builtin_constant_p(fmt)) \
- __trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args); \
- else \
- __trace_printk(_THIS_IP_, fmt, ##args); \
-} while (0)
-
-extern __printf(2, 3)
-int __trace_bprintk(unsigned long ip, const char *fmt, ...);
-
-extern __printf(2, 3)
-int __trace_printk(unsigned long ip, const char *fmt, ...);
-
-/**
- * trace_puts - write a string into the ftrace buffer
- * @str: the string to record
- *
- * Note: __trace_bputs is an internal function for trace_puts and
- * the @ip is passed in via the trace_puts macro.
- *
- * This is similar to trace_printk() but is made for those really fast
- * paths that a developer wants the least amount of "Heisenbug" effects,
- * where the processing of the print format is still too much.
- *
- * This function allows a kernel developer to debug fast path sections
- * that printk is not appropriate for. By scattering in various
- * printk like tracing in the code, a developer can quickly see
- * where problems are occurring.
- *
- * This is intended as a debugging tool for the developer only.
- * Please refrain from leaving trace_puts scattered around in
- * your code. (Extra memory is used for special buffers that are
- * allocated when trace_puts() is used.)
- *
- * Returns: 0 if nothing was written, positive # if string was.
- * (1 when __trace_bputs is used, strlen(str) when __trace_puts is used)
- */
-
-#define trace_puts(str) ({ \
- static const char *trace_printk_fmt __used \
- __section("__trace_printk_fmt") = \
- __builtin_constant_p(str) ? str : NULL; \
- \
- if (__builtin_constant_p(str)) \
- __trace_bputs(_THIS_IP_, trace_printk_fmt); \
- else \
- __trace_puts(_THIS_IP_, str, strlen(str)); \
-})
-extern int __trace_bputs(unsigned long ip, const char *str);
-extern int __trace_puts(unsigned long ip, const char *str, int size);
-
-extern void trace_dump_stack(int skip);
-
-/*
- * The double __builtin_constant_p is because gcc will give us an error
- * if we try to allocate the static variable to fmt if it is not a
- * constant. Even with the outer if statement.
- */
-#define ftrace_vprintk(fmt, vargs) \
-do { \
- if (__builtin_constant_p(fmt)) { \
- static const char *trace_printk_fmt __used \
- __section("__trace_printk_fmt") = \
- __builtin_constant_p(fmt) ? fmt : NULL; \
- \
- __ftrace_vbprintk(_THIS_IP_, trace_printk_fmt, vargs); \
- } else \
- __ftrace_vprintk(_THIS_IP_, fmt, vargs); \
-} while (0)
-
-extern __printf(2, 0) int
-__ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap);
-
-extern __printf(2, 0) int
-__ftrace_vprintk(unsigned long ip, const char *fmt, va_list ap);
-
-extern void ftrace_dump(enum ftrace_dump_mode oops_dump_mode);
-#else
-static inline void tracing_start(void) { }
-static inline void tracing_stop(void) { }
-static inline void trace_dump_stack(int skip) { }
-
-static inline void tracing_on(void) { }
-static inline void tracing_off(void) { }
-static inline int tracing_is_on(void) { return 0; }
-static inline void tracing_snapshot(void) { }
-static inline void tracing_snapshot_alloc(void) { }
-
-static inline __printf(1, 2)
-int trace_printk(const char *fmt, ...)
-{
- return 0;
-}
-static __printf(1, 0) inline int
-ftrace_vprintk(const char *fmt, va_list ap)
-{
- return 0;
-}
-static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
-#endif /* CONFIG_TRACING */
-
/* Rebuild everything on CONFIG_DYNAMIC_FTRACE */
#ifdef CONFIG_DYNAMIC_FTRACE
# define REBUILD_DUE_TO_DYNAMIC_FTRACE
diff --git a/include/linux/tracing.h b/include/linux/tracing.h
new file mode 100644
index 000000000000..1989e6328c59
--- /dev/null
+++ b/include/linux/tracing.h
@@ -0,0 +1,203 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_TRACING_H
+#define _LINUX_TRACING_H
+
+#include <linux/compiler_attributes.h>
+#include <linux/instruction_pointer.h>
+#include <linux/stringify.h>
+
+/*
+ * General tracing related utility functions - trace_printk(),
+ * tracing_on/tracing_off and tracing_start()/tracing_stop
+ *
+ * Use tracing_on/tracing_off when you want to quickly turn on or off
+ * tracing. It simply enables or disables the recording of the trace events.
+ * This also corresponds to the user space /sys/kernel/tracing/tracing_on
+ * file, which gives a means for the kernel and userspace to interact.
+ * Place a tracing_off() in the kernel where you want tracing to end.
+ * From user space, examine the trace, and then echo 1 > tracing_on
+ * to continue tracing.
+ *
+ * tracing_stop/tracing_start has slightly more overhead. It is used
+ * by things like suspend to ram where disabling the recording of the
+ * trace is not enough, but tracing must actually stop because things
+ * like calling smp_processor_id() may crash the system.
+ *
+ * Most likely, you want to use tracing_on/tracing_off.
+ */
+
+enum ftrace_dump_mode {
+ DUMP_NONE,
+ DUMP_ALL,
+ DUMP_ORIG,
+ DUMP_PARAM,
+};
+
+#ifdef CONFIG_TRACING
+void tracing_on(void);
+void tracing_off(void);
+int tracing_is_on(void);
+void tracing_snapshot(void);
+void tracing_snapshot_alloc(void);
+
+void tracing_start(void);
+void tracing_stop(void);
+
+static inline __printf(1, 2)
+void ____trace_printk_check_format(const char *fmt, ...)
+{
+}
+#define __trace_printk_check_format(fmt, args...) \
+do { \
+ if (0) \
+ ____trace_printk_check_format(fmt, ##args); \
+} while (0)
+
+/**
+ * trace_printk - printf formatting in the ftrace buffer
+ * @fmt: the printf format for printing
+ *
+ * Note: __trace_printk is an internal function for trace_printk() and
+ * the @ip is passed in via the trace_printk() macro.
+ *
+ * This function allows a kernel developer to debug fast path sections
+ * that printk is not appropriate for. By scattering in various
+ * printk like tracing in the code, a developer can quickly see
+ * where problems are occurring.
+ *
+ * This is intended as a debugging tool for the developer only.
+ * Please refrain from leaving trace_printks scattered around in
+ * your code. (Extra memory is used for special buffers that are
+ * allocated when trace_printk() is used.)
+ *
+ * A little optimization trick is done here. If there's only one
+ * argument, there's no need to scan the string for printf formats.
+ * The trace_puts() will suffice. But how can we take advantage of
+ * using trace_puts() when trace_printk() has only one argument?
+ * By stringifying the args and checking the size we can tell
+ * whether or not there are args. __stringify((__VA_ARGS__)) will
+ * turn into "()\0" with a size of 3 when there are no args, anything
+ * else will be bigger. All we need to do is define a string to this,
+ * and then take its size and compare to 3. If it's bigger, use
+ * do_trace_printk() otherwise, optimize it to trace_puts(). Then just
+ * let gcc optimize the rest.
+ */
+
+#define trace_printk(fmt, ...) \
+do { \
+ char _______STR[] = __stringify((__VA_ARGS__)); \
+ if (sizeof(_______STR) > 3) \
+ do_trace_printk(fmt, ##__VA_ARGS__); \
+ else \
+ trace_puts(fmt); \
+} while (0)
+
+#define do_trace_printk(fmt, args...) \
+do { \
+ static const char *trace_printk_fmt __used \
+ __section("__trace_printk_fmt") = \
+ __builtin_constant_p(fmt) ? fmt : NULL; \
+ \
+ __trace_printk_check_format(fmt, ##args); \
+ \
+ if (__builtin_constant_p(fmt)) \
+ __trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args); \
+ else \
+ __trace_printk(_THIS_IP_, fmt, ##args); \
+} while (0)
+
+__printf(2, 3)
+int __trace_bprintk(unsigned long ip, const char *fmt, ...);
+
+__printf(2, 3)
+int __trace_printk(unsigned long ip, const char *fmt, ...);
+
+/**
+ * trace_puts - write a string into the ftrace buffer
+ * @str: the string to record
+ *
+ * Note: __trace_bputs is an internal function for trace_puts and
+ * the @ip is passed in via the trace_puts macro.
+ *
+ * This is similar to trace_printk() but is made for those really fast
+ * paths that a developer wants the least amount of "Heisenbug" effects,
+ * where the processing of the print format is still too much.
+ *
+ * This function allows a kernel developer to debug fast path sections
+ * that printk is not appropriate for. By scattering in various
+ * printk like tracing in the code, a developer can quickly see
+ * where problems are occurring.
+ *
+ * This is intended as a debugging tool for the developer only.
+ * Please refrain from leaving trace_puts scattered around in
+ * your code. (Extra memory is used for special buffers that are
+ * allocated when trace_puts() is used.)
+ *
+ * Returns: 0 if nothing was written, positive # if string was.
+ * (1 when __trace_bputs is used, strlen(str) when __trace_puts is used)
+ */
+
+#define trace_puts(str) ({ \
+ static const char *trace_printk_fmt __used \
+ __section("__trace_printk_fmt") = \
+ __builtin_constant_p(str) ? str : NULL; \
+ \
+ if (__builtin_constant_p(str)) \
+ __trace_bputs(_THIS_IP_, trace_printk_fmt); \
+ else \
+ __trace_puts(_THIS_IP_, str, strlen(str)); \
+})
+int __trace_bputs(unsigned long ip, const char *str);
+int __trace_puts(unsigned long ip, const char *str, int size);
+
+void trace_dump_stack(int skip);
+
+/*
+ * The double __builtin_constant_p is because gcc will give us an error
+ * if we try to allocate the static variable to fmt if it is not a
+ * constant. Even with the outer if statement.
+ */
+#define ftrace_vprintk(fmt, vargs) \
+do { \
+ if (__builtin_constant_p(fmt)) { \
+ static const char *trace_printk_fmt __used \
+ __section("__trace_printk_fmt") = \
+ __builtin_constant_p(fmt) ? fmt : NULL; \
+ \
+ __ftrace_vbprintk(_THIS_IP_, trace_printk_fmt, vargs); \
+ } else \
+ __ftrace_vprintk(_THIS_IP_, fmt, vargs); \
+} while (0)
+
+__printf(2, 0) int
+__ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap);
+
+__printf(2, 0) int
+__ftrace_vprintk(unsigned long ip, const char *fmt, va_list ap);
+
+void ftrace_dump(enum ftrace_dump_mode oops_dump_mode);
+#else
+static inline void tracing_start(void) { }
+static inline void tracing_stop(void) { }
+static inline void trace_dump_stack(int skip) { }
+
+static inline void tracing_on(void) { }
+static inline void tracing_off(void) { }
+static inline int tracing_is_on(void) { return 0; }
+static inline void tracing_snapshot(void) { }
+static inline void tracing_snapshot_alloc(void) { }
+
+static inline __printf(1, 2)
+int trace_printk(const char *fmt, ...)
+{
+ return 0;
+}
+static __printf(1, 0) inline int
+ftrace_vprintk(const char *fmt, va_list ap)
+{
+ return 0;
+}
+static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
+#endif /* CONFIG_TRACING */
+
+#endif
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v2 2/4] moduleparam: include required headers explicitly
From: Andy Shevchenko @ 2025-12-03 17:00 UTC (permalink / raw)
To: Yury Norov (NVIDIA)
Cc: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251203162329.280182-3-yury.norov@gmail.com>
On Wed, Dec 03, 2025 at 11:23:23AM -0500, Yury Norov (NVIDIA) wrote:
> The following patch drops moduleparam.h dependency on kernel.h. In
> preparation to it, list all the required headers explicitly.
> Suggested-by: Petr Pavlu <petr.pavlu@suse.com>
> CC: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
You can move this line after --- or use --cc when formatting patch,
no need to pollute the commit message with it.
In any case, the change LGTM and you can just replace the above with
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Steven Rostedt @ 2025-12-03 17:10 UTC (permalink / raw)
To: Yury Norov (NVIDIA)
Cc: Masami Hiramatsu, Mathieu Desnoyers, Andy Shevchenko,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251203162329.280182-5-yury.norov@gmail.com>
On Wed, 3 Dec 2025 11:23:25 -0500
"Yury Norov (NVIDIA)" <yury.norov@gmail.com> wrote:
> diff --git a/include/linux/tracing.h b/include/linux/tracing.h
> new file mode 100644
> index 000000000000..1989e6328c59
> --- /dev/null
> +++ b/include/linux/tracing.h
Sorry, and I hope this isn't too much bike shedding, but I'm thinking we
should call it trace_printk.h. It's mostly about using trace_printk() and
other similar debugging aspects. Calling it "tracing.h" makes it sound like
it is part of the generic tracing infrastructure.
Thanks,
-- Steve
^ permalink raw reply
* Re: [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Andy Shevchenko @ 2025-12-03 17:10 UTC (permalink / raw)
To: Yury Norov (NVIDIA)
Cc: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251203162329.280182-5-yury.norov@gmail.com>
On Wed, Dec 03, 2025 at 11:23:25AM -0500, Yury Norov (NVIDIA) wrote:
> Tracing is a half of the kernel.h in terms of LOCs, although it's
> a self-consistent part. It is intended for quick debugging purposes
> and isn't used by the normal tracing utilities.
>
> Move it to a separate header. If someone needs to just throw a
> trace_printk() in their driver, they will not have to pull all
> the heavy tracing machinery.
>
> This is a pure move, except for removing a few 'extern's.
...
> #include <linux/build_bug.h>
> #include <linux/sprintf.h>
> #include <linux/static_call_types.h>
> +#include <linux/tracing.h>
> #include <linux/instruction_pointer.h>
And kill this one? We really don't care about the mess in the kernel.h right
now, and your header will bring that anyway. Also consider killing other
inclusions that has been moved from kernel.h to tracing.h.
> #include <linux/util_macros.h>
> #include <linux/wordpart.h>
...
> +#ifndef _LINUX_TRACING_H
> +#define _LINUX_TRACING_H
> +
> +#include <linux/compiler_attributes.h>
> +#include <linux/instruction_pointer.h>
+ stddef.h // you use NULL
+ string.h // use of strlen()
> +#include <linux/stringify.h>
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Andy Shevchenko @ 2025-12-03 17:11 UTC (permalink / raw)
To: Steven Rostedt
Cc: Yury Norov (NVIDIA), Masami Hiramatsu, Mathieu Desnoyers,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251203121007.407eaf3d@gandalf.local.home>
On Wed, Dec 03, 2025 at 12:10:07PM -0500, Steven Rostedt wrote:
> On Wed, 3 Dec 2025 11:23:25 -0500
> "Yury Norov (NVIDIA)" <yury.norov@gmail.com> wrote:
>
> > diff --git a/include/linux/tracing.h b/include/linux/tracing.h
> > new file mode 100644
> > index 000000000000..1989e6328c59
> > --- /dev/null
> > +++ b/include/linux/tracing.h
>
> Sorry, and I hope this isn't too much bike shedding, but I'm thinking we
> should call it trace_printk.h. It's mostly about using trace_printk() and
> other similar debugging aspects. Calling it "tracing.h" makes it sound like
> it is part of the generic tracing infrastructure.
I am fine with the proposed name, and it makes sense to me.
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Yury Norov @ 2025-12-03 17:12 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, Mathieu Desnoyers, Andy Shevchenko,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <20251203121007.407eaf3d@gandalf.local.home>
On Wed, Dec 03, 2025 at 12:10:07PM -0500, Steven Rostedt wrote:
> On Wed, 3 Dec 2025 11:23:25 -0500
> "Yury Norov (NVIDIA)" <yury.norov@gmail.com> wrote:
>
> > diff --git a/include/linux/tracing.h b/include/linux/tracing.h
> > new file mode 100644
> > index 000000000000..1989e6328c59
> > --- /dev/null
> > +++ b/include/linux/tracing.h
>
> Sorry, and I hope this isn't too much bike shedding, but I'm thinking we
> should call it trace_printk.h. It's mostly about using trace_printk() and
> other similar debugging aspects. Calling it "tracing.h" makes it sound like
> it is part of the generic tracing infrastructure.
I agree. I didn't change the name because it seemingly was a consensus
about it during v1 discussion. Let's wait for more comments. If no
other issues, I can send a quick v3, or you can fix it inplace,
assuming you're moving it yourself.
Thanks,
Yury
^ permalink raw reply
* Re: [PATCH v2 4/4] tracing: move tracing declarations from kernel.h to a dedicated header
From: Andy Shevchenko @ 2025-12-03 17:12 UTC (permalink / raw)
To: Yury Norov (NVIDIA)
Cc: Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Christophe Leroy, Randy Dunlap, Ingo Molnar, Jani Nikula,
Joonas Lahtinen, David Laight, Petr Pavlu, Andi Shyti,
Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez, Greg Kroah-Hartman,
Rafael J. Wysocki, Danilo Krummrich, Andrew Morton, linux-kernel,
intel-gfx, dri-devel, linux-modules, linux-trace-kernel
In-Reply-To: <aTBvE_Arj16yB83I@smile.fi.intel.com>
On Wed, Dec 03, 2025 at 07:10:53PM +0200, Andy Shevchenko wrote:
> On Wed, Dec 03, 2025 at 11:23:25AM -0500, Yury Norov (NVIDIA) wrote:
...
> > +#ifndef _LINUX_TRACING_H
> > +#define _LINUX_TRACING_H
> > +
> > +#include <linux/compiler_attributes.h>
> > +#include <linux/instruction_pointer.h>
>
> + stddef.h // you use NULL
>
> + string.h // use of strlen()
Why it should not conflict, it might be worth to research more and perhaps even
use intrinsic instead of strlen(). Just a thought spoken loudly.
> > +#include <linux/stringify.h>
--
With Best Regards,
Andy Shevchenko
^ permalink raw reply
* Re: [PATCH v3 2/2] lib/crypto: tests: Add KUnit tests for ML-DSA verification
From: Eric Biggers @ 2025-12-03 17:43 UTC (permalink / raw)
To: David Howells
Cc: linux-crypto, Herbert Xu, Luis Chamberlain, Petr Pavlu,
Daniel Gomez, Sami Tolvanen, Jason A . Donenfeld, Ard Biesheuvel,
Stephan Mueller, Lukas Wunner, Ignat Korchagin, keyrings,
linux-modules, linux-kernel
In-Reply-To: <1682768.1764752475@warthog.procyon.org.uk>
On Wed, Dec 03, 2025 at 09:01:15AM +0000, David Howells wrote:
> I'm seeing:
>
> ERROR: modpost: module mldsa_kunit uses symbol mldsa_use_hint from namespace EXPORTED_FOR_KUNIT_TESTING, but does not import it.
>
Yep, sorry for the trouble. mldsa_kunit.c needs:
MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
Would be nice if the kunit_test_suite() macro did that so that test
authors don't have to remember to add the namespace import separately.
But maybe there's a reason why they need to be separate.
- Eric
^ permalink raw reply
* [GIT PULL] Modules changes for v6.19-rc1
From: Daniel Gomez @ 2025-12-03 23:48 UTC (permalink / raw)
To: Linus Torvalds
Cc: Luis Chamberlain, Petr Pavlu, Sami Tolvanen, Daniel Gomez,
Mauro Carvalho Chehab, Hans Verkuil, Andreas Hindborg,
Uwe Kleine-König, Aaron Tomlin, Kees Cook, linux-modules,
linux-kernel
The following changes since commit dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa:
Linux 6.18-rc3 (2025-10-26 15:59:49 -0700)
are available in the Git repository at:
ssh://git@gitolite.kernel.org/pub/scm/linux/kernel/git/modules/linux.git/ tags/modules-6.19-rc1
for you to fetch changes up to 1ddac5cd7f278345b2e8298c930e4bffe0911a45:
MAINTAINERS: Add myself as reviewer for module support (2025-11-10 16:20:35 +0100)
----------------------------------------------------------------
Modules changes for v6.19-rc1
Rust module parameter support:
- Add Rust module parameter support, enabling Rust kernel modules to declare
and use module parameters. The rust_minimal sample module demonstrates this,
and the rust null block driver will be the first to use it in the next cycle.
This also adds the Rust module files under the modules subsystem as agreed
between the Rust and modules maintainers.
Hardening:
- Add compile-time check for embedded NUL characters in MODULE_*() macros. This
module metadata was once used (and maybe still) to bypass license enforcement
(LWN article [1] from 2003). This change required a sparse fix [2] which you
reviewed.
MAINTAINERS:
- Add Aaron Tomlin as reviewer for the Modules subsystem.
The changes have been in linux-next for 4 weeks. Recent 0day reports for UM [3]
and arm64 [4] builds were not reproducible and traced to a buggy bindgen version
combined with unreleased clang-22 in 0day. The Rust team has reported this to
0day.
As discussed previously, we rotate module maintainership among co-maintainers
every 6 months. Sami Tolvanen is next in line and will send the next pull
request. As a reminder, Luis has already announced [5] he will gradually step
away as maintainer.
Link: https://lwn.net/Articles/82305/ [1]
Link: https://lore.kernel.org/linux-sparse/CACePvbVG2KrGQq4cNKV=wbO5h=jp3M0RO1SdfX8kV4OukjPG8A@mail.gmail.com/T/#mf838b3e2e3245d88c30a801ea7473d5a5c0eb121 [2]
Link: https://lore.kernel.org/oe-kbuild-all/202511210858.uwVivgvn-lkp@intel.com/ [3]
Link: https://lore.kernel.org/oe-kbuild-all/202512020454.Tf36WHw5-lkp@intel.com/ [4]
Link: https://lore.kernel.org/linux-modules/aGiAF8IQ4PRYn0th@bombadil.infradead.org/ [5]
Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
----------------------------------------------------------------
Conflicts:
There are conflicts with the pwm and rust trees. Both are resolved in
linux-next.
pwm tree:
739ad9be61e5 ("rust: macros: Add support for 'imports_ns' to module!")
conflicts with:
0b24f9740f26 ("rust: module: update the module macro with module parameter
support")
3809d7a89fe5 ("rust: module: use a reference in macros::module::module")
Resolution:
--- a/rust/macros/module.rs
+++ b/rust/macros/module.rs
@@@ -205,52 -98,9 +205,53 @@@ struct ModuleInfo
description: Option<String>,
alias: Option<Vec<String>>,
firmware: Option<Vec<String>>,
+ params: Option<Vec<Parameter>>,
+ imports_ns: Option<Vec<String>>,
}
+#[derive(Debug)]
+struct Parameter {
+ name: String,
+ ptype: String,
+ default: String,
+ description: String,
+}
+
+fn expect_params(it: &mut token_stream::IntoIter) -> Vec<Parameter> {
+ let params = expect_group(it);
+ assert_eq!(params.delimiter(), Delimiter::Brace);
+ let mut it = params.stream().into_iter();
+ let mut parsed = Vec::new();
+
+ loop {
+ let param_name = match it.next() {
+ Some(TokenTree::Ident(ident)) => ident.to_string(),
+ Some(_) => panic!("Expected Ident or end"),
+ None => break,
+ };
+
+ assert_eq!(expect_punct(&mut it), ':');
+ let param_type = expect_ident(&mut it);
+ let group = expect_group(&mut it);
+ assert_eq!(group.delimiter(), Delimiter::Brace);
+ assert_eq!(expect_punct(&mut it), ',');
+
+ let mut param_it = group.stream().into_iter();
+ let param_default = expect_param_default(&mut param_it);
+ let param_description = expect_string_field(&mut param_it, "description");
+ expect_end(&mut param_it);
+
+ parsed.push(Parameter {
+ name: param_name,
+ ptype: param_type,
+ default: param_default,
+ description: param_description,
+ })
+ }
+
+ parsed
+}
+
impl ModuleInfo {
fn parse(it: &mut token_stream::IntoIter) -> Self {
let mut info = ModuleInfo::default();
@@@ -263,7 -113,7 +264,8 @@@
"license",
"alias",
"firmware",
+ "params",
+ "imports_ns",
];
const REQUIRED_KEYS: &[&str] = &["type", "name", "license"];
let mut seen_keys = Vec::new();
@@@ -289,7 -139,7 +291,8 @@@
"license" => info.license = expect_string_ascii(it),
"alias" => info.alias = Some(expect_string_array(it)),
"firmware" => info.firmware = Some(expect_string_array(it)),
+ "params" => info.params = Some(expect_params(it)),
+ "imports_ns" => info.imports_ns = Some(expect_string_array(it)),
_ => panic!("Unknown key \"{key}\". Valid keys are: {EXPECTED_KEYS:?}."),
}
@@@ -329,25 -179,30 +332,30 @@@ pub(crate) fn module(ts: TokenStream) -
// Rust does not allow hyphens in identifiers, use underscore instead.
let ident = info.name.replace('-', "_");
let mut modinfo = ModInfoBuilder::new(ident.as_ref());
- if let Some(authors) = info.authors {
+ if let Some(authors) = &info.authors {
for author in authors {
- modinfo.emit("author", &author);
+ modinfo.emit("author", author);
}
}
- if let Some(description) = info.description {
- modinfo.emit("description", &description);
+ if let Some(description) = &info.description {
+ modinfo.emit("description", description);
}
modinfo.emit("license", &info.license);
- if let Some(aliases) = info.alias {
+ if let Some(aliases) = &info.alias {
for alias in aliases {
- modinfo.emit("alias", &alias);
+ modinfo.emit("alias", alias);
}
}
- if let Some(firmware) = info.firmware {
+ if let Some(firmware) = &info.firmware {
for fw in firmware {
- modinfo.emit("firmware", &fw);
+ modinfo.emit("firmware", fw);
}
}
- if let Some(imports) = info.imports_ns {
++ if let Some(imports) = &info.imports_ns {
+ for ns in imports {
- modinfo.emit("import_ns", &ns);
++ modinfo.emit("import_ns", ns);
+ }
+ }
rust tree commit:
3b83f5d5e78a ("rust: replace `CStr` with `core::ffi::CStr`")
conflicts with 2 modules' commits. The first is:
51d9ee90ea90 ("rust: str: add radix prefixed integer parsing functions")
Resolution:
diff --cc rust/kernel/str.rs
index a1a3581eb546,7593d758fbb7..000000000000
--- a/rust/kernel/str.rs
+++ b/rust/kernel/str.rs
@@@ -10,11 -10,11 +10,13 @@@ use crate::
};
use core::{
marker::PhantomData,
- ops::{self, Deref, DerefMut, Index},
+ ops::{Deref, DerefMut, Index},
};
+ pub use crate::prelude::CStr;
+
+pub mod parse_int;
+
/// Byte string without UTF-8 validity guarantee.
#[repr(transparent)]
pub struct BStr([u8]);
And the second is:
0b08fc292842 ("rust: introduce module_param module")
which causes build error [6]. Resolution:
diff --git a/rust/kernel/module_param.rs b/rust/kernel/module_param.rs
index e7d5c930a467d..6a8a7a875643a 100644
--- a/rust/kernel/module_param.rs
+++ b/rust/kernel/module_param.rs
@@ -70,6 +70,7 @@ pub trait ModuleParam: Sized + Copy {
// SAFETY: By function safety requirement, val is non-null, null-terminated
// and valid for reads for the duration of this function.
let arg = unsafe { CStr::from_char_ptr(val) };
+ let arg: &BStr = arg.as_ref();
crate::error::from_result(|| {
let new_value = T::try_from_param_arg(arg)?;
diff --git a/rust/macros/module.rs b/rust/macros/module.rs
index d62e9c1e2a898..decb0849f3d33 100644
--- a/rust/macros/module.rs
+++ b/rust/macros/module.rs
@@ -133,10 +133,10 @@ fn emit_params(&mut self, info: &ModuleInfo) {
::kernel::module_param::KernelParam::new(
::kernel::bindings::kernel_param {{
name: if ::core::cfg!(MODULE) {{
- ::kernel::c_str!(\"{param_name}\").as_bytes_with_nul()
+ ::kernel::c_str!(\"{param_name}\").to_bytes_with_nul()
}} else {{
::kernel::c_str!(\"{module_name}.{param_name}\")
- .as_bytes_with_nul()
+ .to_bytes_with_nul()
}}.as_ptr(),
// SAFETY: `__this_module` is constructed by the kernel at load
// time and will not be freed until the module is unloaded.
Link: https://lore.kernel.org/all/20251104154500.5acb5340@canb.auug.org.au/ [6]
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE73Ua4R8Pc+G5xjxTQJ6jxB8ZUfsFAmkwygIACgkQQJ6jxB8Z
UfvZ2Q//YAkK9V1Hk8imngjOxmiT1BGzo0feKSOgDHc0K3G3VkutmYMKTPymLS8Q
6EbvpbBRke990lOB7PloEL5ih27i9jmdL0QKpgU+uijRy5RssYEOoDMEz9JuKnqX
L8BzR61YzRoEIZBgZWij1Di+ITTu+qHn5VxnJUCqydDS4uqqcgO/9xibmN1JtToO
HpI63Y3R0VSMnJYfyVYJuKVCVWBhJzOzgIC8ZJCDUSceZlOAAjTsMyeUPS5m8j03
28o78aH3XTLRpL46vKBt4hpmeNrqE47tj6meMybVEew9SmEF78B9wbaQD3oR8Jod
BiFAhCNkwQao6aQAaKHAUZyWl+Udqsk8kJEgSeo/Sn5p1A6c2pGbddg++2W5jk75
gjYQEwdv+VZuym1YraM8E+mKIU/9+X1NXpwuusC5Vli7xz8DYf3w6llZNOgRQqTr
E1fXRQv2X5rUz3o6gzHgDF14XUEH0GM/3kYdUFSO9mqAQJCsMIRv0xIzRddsAcXH
ylqNX+o7cO+wuwcIvBIkhlYwS/MnAP/iDGFp8NTGGZsDrorCkNu5mFlO+xzGVLWd
gizeWnzgKrCNTzlR9oUzsGuPjTaQMBkNMTwlE+7InlLFH2CUc3vyKrkANzcL/vGn
jHBdg/pNsboAfbERgNG42d8YqrrCuLvYVrI6TRw9RhPPBFt8coE=
=ujQB
-----END PGP SIGNATURE-----
----------------------------------------------------------------
Aaron Tomlin (1):
MAINTAINERS: Add myself as reviewer for module support
Andreas Hindborg (7):
rust: sync: add `SetOnce`
rust: str: add radix prefixed integer parsing functions
rust: introduce module_param module
rust: module: use a reference in macros::module::module
rust: module: update the module macro with module parameter support
rust: samples: add a module parameter to the rust_minimal sample
modules: add rust modules files to MAINTAINERS
Kees Cook (3):
media: dvb-usb-v2: lmedm04: Fix firmware macro definitions
media: radio: si470x: Fix DRIVER_AUTHOR macro definition
module: Add compile-time check for embedded NUL characters
MAINTAINERS | 3 +
drivers/media/radio/si470x/radio-si470x-i2c.c | 2 +-
drivers/media/usb/dvb-usb-v2/lmedm04.c | 12 +-
include/linux/moduleparam.h | 3 +
rust/kernel/lib.rs | 1 +
rust/kernel/module_param.rs | 181 ++++++++++++++++++++++++
rust/kernel/str.rs | 2 +
rust/kernel/str/parse_int.rs | 148 ++++++++++++++++++++
rust/kernel/sync.rs | 2 +
rust/kernel/sync/set_once.rs | 125 +++++++++++++++++
rust/macros/helpers.rs | 25 ++++
rust/macros/lib.rs | 31 ++++
rust/macros/module.rs | 194 +++++++++++++++++++++++---
samples/rust/rust_minimal.rs | 10 ++
14 files changed, 714 insertions(+), 25 deletions(-)
create mode 100644 rust/kernel/module_param.rs
create mode 100644 rust/kernel/str/parse_int.rs
create mode 100644 rust/kernel/sync/set_once.rs
^ permalink raw reply related
* Re: [PATCH v2 1/4] kernel.h: drop STACK_MAGIC macro
From: Christophe Leroy (CS GROUP) @ 2025-12-04 7:43 UTC (permalink / raw)
To: Yury Norov (NVIDIA), Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Andy Shevchenko, Randy Dunlap, Ingo Molnar,
Jani Nikula, Joonas Lahtinen, David Laight, Petr Pavlu,
Andi Shyti, Rodrigo Vivi, Tvrtko Ursulin, Daniel Gomez,
Greg Kroah-Hartman, Rafael J. Wysocki, Danilo Krummrich,
Andrew Morton, linux-kernel, intel-gfx, dri-devel, linux-modules,
linux-trace-kernel
Cc: Jani Nikula
In-Reply-To: <20251203162329.280182-2-yury.norov@gmail.com>
Le 03/12/2025 à 17:23, Yury Norov (NVIDIA) a écrit :
> The macro was introduced in 1994, v1.0.4, for stacks protection. Since
> that, people found better ways to protect stacks, and now the macro is
> only used by i915 selftests. Move it to a local header and drop from
> the kernel.h.
>
> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Acked-by: Randy Dunlap <rdunlap@infradead.org>
> Acked-by: Jani Nikula <jani.nikula@intel.com>
> Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
> ---
> drivers/gpu/drm/i915/gt/selftest_ring_submission.c | 1 +
> drivers/gpu/drm/i915/i915_selftest.h | 2 ++
> include/linux/kernel.h | 2 --
> 3 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gt/selftest_ring_submission.c b/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
> index 87ceb0f374b6..600333ae6c8c 100644
> --- a/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
> +++ b/drivers/gpu/drm/i915/gt/selftest_ring_submission.c
> @@ -3,6 +3,7 @@
> * Copyright © 2020 Intel Corporation
> */
>
> +#include "i915_selftest.h"
> #include "intel_engine_pm.h"
> #include "selftests/igt_flush_test.h"
>
> diff --git a/drivers/gpu/drm/i915/i915_selftest.h b/drivers/gpu/drm/i915/i915_selftest.h
> index bdf3e22c0a34..72922028f4ba 100644
> --- a/drivers/gpu/drm/i915/i915_selftest.h
> +++ b/drivers/gpu/drm/i915/i915_selftest.h
> @@ -26,6 +26,8 @@
>
> #include <linux/types.h>
>
> +#define STACK_MAGIC 0xdeadbeef
> +
> struct pci_dev;
> struct drm_i915_private;
>
> diff --git a/include/linux/kernel.h b/include/linux/kernel.h
> index 5b46924fdff5..61d63c57bc2d 100644
> --- a/include/linux/kernel.h
> +++ b/include/linux/kernel.h
> @@ -40,8 +40,6 @@
>
> #include <uapi/linux/kernel.h>
>
> -#define STACK_MAGIC 0xdeadbeef
> -
> struct completion;
> struct user;
>
^ permalink raw reply
* Re: [GIT PULL] Modules changes for v6.19-rc1
From: Daniel Gomez @ 2025-12-04 11:52 UTC (permalink / raw)
To: Linus Torvalds
Cc: Luis Chamberlain, Petr Pavlu, Sami Tolvanen, Daniel Gomez,
Mauro Carvalho Chehab, Hans Verkuil, Andreas Hindborg,
Uwe Kleine-König, Aaron Tomlin, Kees Cook, linux-modules,
linux-kernel
In-Reply-To: <20251203234840.3720-1-da.gomez@kernel.org>
On 04/12/2025 00.48, Daniel Gomez wrote:
> The following changes since commit dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa:
>
> Linux 6.18-rc3 (2025-10-26 15:59:49 -0700)
>
> are available in the Git repository at:
>
> ssh://git@gitolite.kernel.org/pub/scm/linux/kernel/git/modules/linux.git/ tags/modules-6.19-rc1
I know the preference is to use git:// but my git config resolved to the above
link. Here what I intended to do:
git://git.kernel.org/pub/scm/linux/kernel/git/modules/linux.git tags/modules-6.19-rc1
> The changes have been in linux-next for 4 weeks. Recent 0day reports for UM [3]
> and arm64 [4] builds were not reproducible and traced to a buggy bindgen version
> combined with unreleased clang-22 in 0day. The Rust team has reported this to
> 0day.
FYI, this has been confirmed by 0day:
https://lore.kernel.org/all/aTEi3KUUkPVGS3ix@xsang-OptiPlex-9020/
> @@@ -329,25 -179,30 +332,30 @@@ pub(crate) fn module(ts: TokenStream) -
> // Rust does not allow hyphens in identifiers, use underscore instead.
> let ident = info.name.replace('-', "_");
> let mut modinfo = ModInfoBuilder::new(ident.as_ref());
> - if let Some(authors) = info.authors {
> + if let Some(authors) = &info.authors {
> for author in authors {
> - modinfo.emit("author", &author);
> + modinfo.emit("author", author);
> }
> }
> - if let Some(description) = info.description {
> - modinfo.emit("description", &description);
> + if let Some(description) = &info.description {
> + modinfo.emit("description", description);
> }
> modinfo.emit("license", &info.license);
> - if let Some(aliases) = info.alias {
> + if let Some(aliases) = &info.alias {
> for alias in aliases {
> - modinfo.emit("alias", &alias);
> + modinfo.emit("alias", alias);
> }
> }
> - if let Some(firmware) = info.firmware {
> + if let Some(firmware) = &info.firmware {
> for fw in firmware {
> - modinfo.emit("firmware", &fw);
> + modinfo.emit("firmware", fw);
> }
> }
> - if let Some(imports) = info.imports_ns {
> ++ if let Some(imports) = &info.imports_ns {
> + for ns in imports {
> - modinfo.emit("import_ns", &ns);
> ++ modinfo.emit("import_ns", ns);
> + }
> + }
This last part is the only non-trivial for this conflict.
^ permalink raw reply
* Re: [PATCH V2] mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction
From: Suren Baghdasaryan @ 2025-12-04 22:05 UTC (permalink / raw)
To: Harry Yoo
Cc: vbabka, Liam.Howlett, cl, rientjes, roman.gushchin, urezki,
sidhartha.kumar, linux-mm, linux-kernel, rcu, maple-tree,
linux-modules, mcgrof, petr.pavlu, samitolvanen, atomlin,
lucas.demarchi, akpm, jonathanh, stable, Daniel Gomez
In-Reply-To: <20251202101626.783736-1-harry.yoo@oracle.com>
On Tue, Dec 2, 2025 at 2:16 AM Harry Yoo <harry.yoo@oracle.com> wrote:
>
> Currently, kvfree_rcu_barrier() flushes RCU sheaves across all slab
> caches when a cache is destroyed. This is unnecessary; only the RCU
> sheaves belonging to the cache being destroyed need to be flushed.
>
> As suggested by Vlastimil Babka, introduce a weaker form of
> kvfree_rcu_barrier() that operates on a specific slab cache.
>
> Factor out flush_rcu_sheaves_on_cache() from flush_all_rcu_sheaves() and
> call it from flush_all_rcu_sheaves() and kvfree_rcu_barrier_on_cache().
>
> Call kvfree_rcu_barrier_on_cache() instead of kvfree_rcu_barrier() on
> cache destruction.
>
> The performance benefit is evaluated on a 12 core 24 threads AMD Ryzen
> 5900X machine (1 socket), by loading slub_kunit module.
>
> Before:
> Total calls: 19
> Average latency (us): 18127
> Total time (us): 344414
>
> After:
> Total calls: 19
> Average latency (us): 10066
> Total time (us): 191264
>
> Two performance regression have been reported:
> - stress module loader test's runtime increases by 50-60% (Daniel)
> - internal graphics test's runtime on Tegra23 increases by 35% (Jon)
>
> They are fixed by this change.
>
> Suggested-by: Vlastimil Babka <vbabka@suse.cz>
> Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
> Cc: <stable@vger.kernel.org>
> Link: https://lore.kernel.org/linux-mm/1bda09da-93be-4737-aef0-d47f8c5c9301@suse.cz
> Reported-and-tested-by: Daniel Gomez <da.gomez@samsung.com>
> Closes: https://lore.kernel.org/linux-mm/0406562e-2066-4cf8-9902-b2b0616dd742@kernel.org
> Reported-and-tested-by: Jon Hunter <jonathanh@nvidia.com>
> Closes: https://lore.kernel.org/linux-mm/e988eff6-1287-425e-a06c-805af5bbf262@nvidia.com
> Signed-off-by: Harry Yoo <harry.yoo@oracle.com>
> ---
>
> No code change, added proper tags and updated changelog.
>
> include/linux/slab.h | 5 ++++
> mm/slab.h | 1 +
> mm/slab_common.c | 52 +++++++++++++++++++++++++++++------------
> mm/slub.c | 55 ++++++++++++++++++++++++--------------------
> 4 files changed, 73 insertions(+), 40 deletions(-)
>
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index cf443f064a66..937c93d44e8c 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -1149,6 +1149,10 @@ static inline void kvfree_rcu_barrier(void)
> {
> rcu_barrier();
> }
> +static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> +{
> + rcu_barrier();
> +}
>
> static inline void kfree_rcu_scheduler_running(void) { }
> #else
> @@ -1156,6 +1160,7 @@ void kvfree_rcu_barrier(void);
>
> void kfree_rcu_scheduler_running(void);
> #endif
> +void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
Should the above line be before the #endif? I was expecting something like this:
#ifndef CONFIG_KVFREE_RCU_BATCHED
...
static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
{
rcu_barrier();
}
#else
...
void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
#endif
but when I apply this patch on mm-new I get this:
#ifndef CONFIG_KVFREE_RCU_BATCHED
...
static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
{
rcu_barrier();
}
#else
...
#endif
void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
Other than that LGTM
>
> /**
> * kmalloc_size_roundup - Report allocation bucket size for the given size
> diff --git a/mm/slab.h b/mm/slab.h
> index f730e012553c..e767aa7e91b0 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -422,6 +422,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
>
> bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> void flush_all_rcu_sheaves(void);
> +void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
>
> #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
> SLAB_CACHE_DMA32 | SLAB_PANIC | \
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 84dfff4f7b1f..dd8a49d6f9cc 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -492,7 +492,7 @@ void kmem_cache_destroy(struct kmem_cache *s)
> return;
>
> /* in-flight kfree_rcu()'s may include objects from our cache */
> - kvfree_rcu_barrier();
> + kvfree_rcu_barrier_on_cache(s);
>
> if (IS_ENABLED(CONFIG_SLUB_RCU_DEBUG) &&
> (s->flags & SLAB_TYPESAFE_BY_RCU)) {
> @@ -2038,25 +2038,13 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> }
> EXPORT_SYMBOL_GPL(kvfree_call_rcu);
>
> -/**
> - * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete.
> - *
> - * Note that a single argument of kvfree_rcu() call has a slow path that
> - * triggers synchronize_rcu() following by freeing a pointer. It is done
> - * before the return from the function. Therefore for any single-argument
> - * call that will result in a kfree() to a cache that is to be destroyed
> - * during module exit, it is developer's responsibility to ensure that all
> - * such calls have returned before the call to kmem_cache_destroy().
> - */
> -void kvfree_rcu_barrier(void)
> +static inline void __kvfree_rcu_barrier(void)
> {
> struct kfree_rcu_cpu_work *krwp;
> struct kfree_rcu_cpu *krcp;
> bool queued;
> int i, cpu;
>
> - flush_all_rcu_sheaves();
> -
> /*
> * Firstly we detach objects and queue them over an RCU-batch
> * for all CPUs. Finally queued works are flushed for each CPU.
> @@ -2118,8 +2106,43 @@ void kvfree_rcu_barrier(void)
> }
> }
> }
> +
> +/**
> + * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete.
> + *
> + * Note that a single argument of kvfree_rcu() call has a slow path that
> + * triggers synchronize_rcu() following by freeing a pointer. It is done
> + * before the return from the function. Therefore for any single-argument
> + * call that will result in a kfree() to a cache that is to be destroyed
> + * during module exit, it is developer's responsibility to ensure that all
> + * such calls have returned before the call to kmem_cache_destroy().
> + */
> +void kvfree_rcu_barrier(void)
> +{
> + flush_all_rcu_sheaves();
> + __kvfree_rcu_barrier();
> +}
> EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
>
> +/**
> + * kvfree_rcu_barrier_on_cache - Wait for in-flight kvfree_rcu() calls on a
> + * specific slab cache.
> + * @s: slab cache to wait for
> + *
> + * See the description of kvfree_rcu_barrier() for details.
> + */
> +void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> +{
> + if (s->cpu_sheaves)
> + flush_rcu_sheaves_on_cache(s);
> + /*
> + * TODO: Introduce a version of __kvfree_rcu_barrier() that works
> + * on a specific slab cache.
> + */
> + __kvfree_rcu_barrier();
> +}
> +EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
> +
> static unsigned long
> kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
> {
> @@ -2215,4 +2238,3 @@ void __init kvfree_rcu_init(void)
> }
>
> #endif /* CONFIG_KVFREE_RCU_BATCHED */
> -
> diff --git a/mm/slub.c b/mm/slub.c
> index 785e25a14999..7cec2220712b 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -4118,42 +4118,47 @@ static void flush_rcu_sheaf(struct work_struct *w)
>
>
> /* needed for kvfree_rcu_barrier() */
> -void flush_all_rcu_sheaves(void)
> +void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
> {
> struct slub_flush_work *sfw;
> - struct kmem_cache *s;
> unsigned int cpu;
>
> - cpus_read_lock();
> - mutex_lock(&slab_mutex);
> + mutex_lock(&flush_lock);
>
> - list_for_each_entry(s, &slab_caches, list) {
> - if (!s->cpu_sheaves)
> - continue;
> + for_each_online_cpu(cpu) {
> + sfw = &per_cpu(slub_flush, cpu);
>
> - mutex_lock(&flush_lock);
> + /*
> + * we don't check if rcu_free sheaf exists - racing
> + * __kfree_rcu_sheaf() might have just removed it.
> + * by executing flush_rcu_sheaf() on the cpu we make
> + * sure the __kfree_rcu_sheaf() finished its call_rcu()
> + */
>
> - for_each_online_cpu(cpu) {
> - sfw = &per_cpu(slub_flush, cpu);
> + INIT_WORK(&sfw->work, flush_rcu_sheaf);
> + sfw->s = s;
> + queue_work_on(cpu, flushwq, &sfw->work);
> + }
>
> - /*
> - * we don't check if rcu_free sheaf exists - racing
> - * __kfree_rcu_sheaf() might have just removed it.
> - * by executing flush_rcu_sheaf() on the cpu we make
> - * sure the __kfree_rcu_sheaf() finished its call_rcu()
> - */
> + for_each_online_cpu(cpu) {
> + sfw = &per_cpu(slub_flush, cpu);
> + flush_work(&sfw->work);
> + }
>
> - INIT_WORK(&sfw->work, flush_rcu_sheaf);
> - sfw->s = s;
> - queue_work_on(cpu, flushwq, &sfw->work);
> - }
> + mutex_unlock(&flush_lock);
> +}
>
> - for_each_online_cpu(cpu) {
> - sfw = &per_cpu(slub_flush, cpu);
> - flush_work(&sfw->work);
> - }
> +void flush_all_rcu_sheaves(void)
> +{
> + struct kmem_cache *s;
> +
> + cpus_read_lock();
> + mutex_lock(&slab_mutex);
>
> - mutex_unlock(&flush_lock);
> + list_for_each_entry(s, &slab_caches, list) {
> + if (!s->cpu_sheaves)
> + continue;
> + flush_rcu_sheaves_on_cache(s);
> }
>
> mutex_unlock(&slab_mutex);
> --
> 2.43.0
>
^ permalink raw reply
* Re: [PATCH V2] mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction
From: Harry Yoo @ 2025-12-05 1:13 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: vbabka, Liam.Howlett, cl, rientjes, roman.gushchin, urezki,
sidhartha.kumar, linux-mm, linux-kernel, rcu, maple-tree,
linux-modules, mcgrof, petr.pavlu, samitolvanen, atomlin,
lucas.demarchi, akpm, jonathanh, stable, Daniel Gomez
In-Reply-To: <CAJuCfpE+g65Dm8-r=psDJQf_O1rfBG62DOzx4mE1mb+ottUKmQ@mail.gmail.com>
On Thu, Dec 04, 2025 at 02:05:07PM -0800, Suren Baghdasaryan wrote:
> On Tue, Dec 2, 2025 at 2:16 AM Harry Yoo <harry.yoo@oracle.com> wrote:
> >
> > Currently, kvfree_rcu_barrier() flushes RCU sheaves across all slab
> > caches when a cache is destroyed. This is unnecessary; only the RCU
> > sheaves belonging to the cache being destroyed need to be flushed.
> >
> > As suggested by Vlastimil Babka, introduce a weaker form of
> > kvfree_rcu_barrier() that operates on a specific slab cache.
> >
> > Factor out flush_rcu_sheaves_on_cache() from flush_all_rcu_sheaves() and
> > call it from flush_all_rcu_sheaves() and kvfree_rcu_barrier_on_cache().
> >
> > Call kvfree_rcu_barrier_on_cache() instead of kvfree_rcu_barrier() on
> > cache destruction.
> >
> > The performance benefit is evaluated on a 12 core 24 threads AMD Ryzen
> > 5900X machine (1 socket), by loading slub_kunit module.
> >
> > Before:
> > Total calls: 19
> > Average latency (us): 18127
> > Total time (us): 344414
> >
> > After:
> > Total calls: 19
> > Average latency (us): 10066
> > Total time (us): 191264
> >
> > Two performance regression have been reported:
> > - stress module loader test's runtime increases by 50-60% (Daniel)
> > - internal graphics test's runtime on Tegra23 increases by 35% (Jon)
> >
> > They are fixed by this change.
> >
> > Suggested-by: Vlastimil Babka <vbabka@suse.cz>
> > Fixes: ec66e0d59952 ("slab: add sheaf support for batching kfree_rcu() operations")
> > Cc: <stable@vger.kernel.org>
> > Link: https://lore.kernel.org/linux-mm/1bda09da-93be-4737-aef0-d47f8c5c9301@suse.cz
> > Reported-and-tested-by: Daniel Gomez <da.gomez@samsung.com>
> > Closes: https://lore.kernel.org/linux-mm/0406562e-2066-4cf8-9902-b2b0616dd742@kernel.org
> > Reported-and-tested-by: Jon Hunter <jonathanh@nvidia.com>
> > Closes: https://lore.kernel.org/linux-mm/e988eff6-1287-425e-a06c-805af5bbf262@nvidia.com
> > Signed-off-by: Harry Yoo <harry.yoo@oracle.com>
> > ---
> >
> > No code change, added proper tags and updated changelog.
> >
> > include/linux/slab.h | 5 ++++
> > mm/slab.h | 1 +
> > mm/slab_common.c | 52 +++++++++++++++++++++++++++++------------
> > mm/slub.c | 55 ++++++++++++++++++++++++--------------------
> > 4 files changed, 73 insertions(+), 40 deletions(-)
> >
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index cf443f064a66..937c93d44e8c 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -1149,6 +1149,10 @@ static inline void kvfree_rcu_barrier(void)
> > {
> > rcu_barrier();
> > }
> > +static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> > +{
> > + rcu_barrier();
> > +}
> >
> > static inline void kfree_rcu_scheduler_running(void) { }
> > #else
> > @@ -1156,6 +1160,7 @@ void kvfree_rcu_barrier(void);
> >
> > void kfree_rcu_scheduler_running(void);
> > #endif
> > +void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
>
> Should the above line be before the #endif? I was expecting something like this:
>
> #ifndef CONFIG_KVFREE_RCU_BATCHED
> ...
> static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> {
> rcu_barrier();
> }
> #else
> ...
> void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
> #endif
>
> but when I apply this patch on mm-new I get this:
>
> #ifndef CONFIG_KVFREE_RCU_BATCHED
> ...
> static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> {
> rcu_barrier();
> }
> #else
> ...
> #endif
> void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
Oops, nice catch!
Interestingly this didn't break CONFIG_KVFREE_RCU_BATCHED=n builds...
I'll send V3 shortly.
> Other than that LGTM
Thanks!
> > /**
> > * kmalloc_size_roundup - Report allocation bucket size for the given size
> > diff --git a/mm/slab.h b/mm/slab.h
> > index f730e012553c..e767aa7e91b0 100644
> > --- a/mm/slab.h
> > +++ b/mm/slab.h
> > @@ -422,6 +422,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
> >
> > bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> > void flush_all_rcu_sheaves(void);
> > +void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
> >
> > #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
> > SLAB_CACHE_DMA32 | SLAB_PANIC | \
> > diff --git a/mm/slab_common.c b/mm/slab_common.c
> > index 84dfff4f7b1f..dd8a49d6f9cc 100644
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -492,7 +492,7 @@ void kmem_cache_destroy(struct kmem_cache *s)
> > return;
> >
> > /* in-flight kfree_rcu()'s may include objects from our cache */
> > - kvfree_rcu_barrier();
> > + kvfree_rcu_barrier_on_cache(s);
> >
> > if (IS_ENABLED(CONFIG_SLUB_RCU_DEBUG) &&
> > (s->flags & SLAB_TYPESAFE_BY_RCU)) {
> > @@ -2038,25 +2038,13 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> > }
> > EXPORT_SYMBOL_GPL(kvfree_call_rcu);
> >
> > -/**
> > - * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete.
> > - *
> > - * Note that a single argument of kvfree_rcu() call has a slow path that
> > - * triggers synchronize_rcu() following by freeing a pointer. It is done
> > - * before the return from the function. Therefore for any single-argument
> > - * call that will result in a kfree() to a cache that is to be destroyed
> > - * during module exit, it is developer's responsibility to ensure that all
> > - * such calls have returned before the call to kmem_cache_destroy().
> > - */
> > -void kvfree_rcu_barrier(void)
> > +static inline void __kvfree_rcu_barrier(void)
> > {
> > struct kfree_rcu_cpu_work *krwp;
> > struct kfree_rcu_cpu *krcp;
> > bool queued;
> > int i, cpu;
> >
> > - flush_all_rcu_sheaves();
> > -
> > /*
> > * Firstly we detach objects and queue them over an RCU-batch
> > * for all CPUs. Finally queued works are flushed for each CPU.
> > @@ -2118,8 +2106,43 @@ void kvfree_rcu_barrier(void)
> > }
> > }
> > }
> > +
> > +/**
> > + * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete.
> > + *
> > + * Note that a single argument of kvfree_rcu() call has a slow path that
> > + * triggers synchronize_rcu() following by freeing a pointer. It is done
> > + * before the return from the function. Therefore for any single-argument
> > + * call that will result in a kfree() to a cache that is to be destroyed
> > + * during module exit, it is developer's responsibility to ensure that all
> > + * such calls have returned before the call to kmem_cache_destroy().
> > + */
> > +void kvfree_rcu_barrier(void)
> > +{
> > + flush_all_rcu_sheaves();
> > + __kvfree_rcu_barrier();
> > +}
> > EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
> >
> > +/**
> > + * kvfree_rcu_barrier_on_cache - Wait for in-flight kvfree_rcu() calls on a
> > + * specific slab cache.
> > + * @s: slab cache to wait for
> > + *
> > + * See the description of kvfree_rcu_barrier() for details.
> > + */
> > +void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> > +{
> > + if (s->cpu_sheaves)
> > + flush_rcu_sheaves_on_cache(s);
> > + /*
> > + * TODO: Introduce a version of __kvfree_rcu_barrier() that works
> > + * on a specific slab cache.
> > + */
> > + __kvfree_rcu_barrier();
> > +}
> > +EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
> > +
> > static unsigned long
> > kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
> > {
> > @@ -2215,4 +2238,3 @@ void __init kvfree_rcu_init(void)
> > }
> >
> > #endif /* CONFIG_KVFREE_RCU_BATCHED */
> > -
> > diff --git a/mm/slub.c b/mm/slub.c
> > index 785e25a14999..7cec2220712b 100644
> > --- a/mm/slub.c
> > +++ b/mm/slub.c
> > @@ -4118,42 +4118,47 @@ static void flush_rcu_sheaf(struct work_struct *w)
> >
> >
> > /* needed for kvfree_rcu_barrier() */
> > -void flush_all_rcu_sheaves(void)
> > +void flush_rcu_sheaves_on_cache(struct kmem_cache *s)
> > {
> > struct slub_flush_work *sfw;
> > - struct kmem_cache *s;
> > unsigned int cpu;
> >
> > - cpus_read_lock();
> > - mutex_lock(&slab_mutex);
> > + mutex_lock(&flush_lock);
> >
> > - list_for_each_entry(s, &slab_caches, list) {
> > - if (!s->cpu_sheaves)
> > - continue;
> > + for_each_online_cpu(cpu) {
> > + sfw = &per_cpu(slub_flush, cpu);
> >
> > - mutex_lock(&flush_lock);
> > + /*
> > + * we don't check if rcu_free sheaf exists - racing
> > + * __kfree_rcu_sheaf() might have just removed it.
> > + * by executing flush_rcu_sheaf() on the cpu we make
> > + * sure the __kfree_rcu_sheaf() finished its call_rcu()
> > + */
> >
> > - for_each_online_cpu(cpu) {
> > - sfw = &per_cpu(slub_flush, cpu);
> > + INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > + sfw->s = s;
> > + queue_work_on(cpu, flushwq, &sfw->work);
> > + }
> >
> > - /*
> > - * we don't check if rcu_free sheaf exists - racing
> > - * __kfree_rcu_sheaf() might have just removed it.
> > - * by executing flush_rcu_sheaf() on the cpu we make
> > - * sure the __kfree_rcu_sheaf() finished its call_rcu()
> > - */
> > + for_each_online_cpu(cpu) {
> > + sfw = &per_cpu(slub_flush, cpu);
> > + flush_work(&sfw->work);
> > + }
> >
> > - INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > - sfw->s = s;
> > - queue_work_on(cpu, flushwq, &sfw->work);
> > - }
> > + mutex_unlock(&flush_lock);
> > +}
> >
> > - for_each_online_cpu(cpu) {
> > - sfw = &per_cpu(slub_flush, cpu);
> > - flush_work(&sfw->work);
> > - }
> > +void flush_all_rcu_sheaves(void)
> > +{
> > + struct kmem_cache *s;
> > +
> > + cpus_read_lock();
> > + mutex_lock(&slab_mutex);
> >
> > - mutex_unlock(&flush_lock);
> > + list_for_each_entry(s, &slab_caches, list) {
> > + if (!s->cpu_sheaves)
> > + continue;
> > + flush_rcu_sheaves_on_cache(s);
> > }
> >
> > mutex_unlock(&slab_mutex);
> > --
> > 2.43.0
> >
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH v18 00/42] DEPT(DEPendency Tracker)
From: Byungchul Park @ 2025-12-05 7:18 UTC (permalink / raw)
To: linux-kernel
Cc: kernel_team, torvalds, damien.lemoal, linux-ide, adilger.kernel,
linux-ext4, mingo, peterz, will, tglx, rostedt, joel, sashal,
daniel.vetter, duyuyang, johannes.berg, tj, tytso, willy, david,
amir73il, gregkh, kernel-team, linux-mm, akpm, mhocko, minchan,
hannes, vdavydov.dev, sj, jglisse, dennis, cl, penberg, rientjes,
vbabka, ngupta, linux-block, josef, linux-fsdevel, jack, jlayton,
dan.j.williams, hch, djwong, dri-devel, rodrigosiqueiramelo,
melissa.srw, hamohammed.sa, harry.yoo, chris.p.wilson,
gwan-gyeong.mun, max.byungchul.park, boqun.feng, longman,
yunseong.kim, ysk, yeoreum.yun, netdev, matthew.brost, her0gyugyu,
corbet, catalin.marinas, bp, x86, hpa, luto, sumit.semwal,
gustavo, christian.koenig, andi.shyti, arnd, lorenzo.stoakes,
Liam.Howlett, rppt, surenb, mcgrof, petr.pavlu, da.gomez,
samitolvanen, paulmck, frederic, neeraj.upadhyay, joelagnelf,
josh, urezki, mathieu.desnoyers, jiangshanlai, qiang.zhang,
juri.lelli, vincent.guittot, dietmar.eggemann, bsegall, mgorman,
vschneid, chuck.lever, neil, okorniev, Dai.Ngo, tom, trondmy,
anna, kees, bigeasy, clrkwllms, mark.rutland, ada.coupriediaz,
kristina.martsenko, wangkefeng.wang, broonie, kevin.brodsky, dwmw,
shakeel.butt, ast, ziy, yuzhao, baolin.wang, usamaarif642,
joel.granados, richard.weiyang, geert+renesas, tim.c.chen, linux,
alexander.shishkin, lillian, chenhuacai, francesco,
guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel,
2407018371, dakr, miguel.ojeda.sandonis, neilb, bagasdotme,
wsa+renesas, dave.hansen, geert, ojeda, alex.gaynor, gary,
bjorn3_gh, lossin, a.hindborg, aliceryhl, tmgross, rust-for-linux
I'm happy to see that DEPT reported real problems in practice:
https://lore.kernel.org/lkml/6383cde5-cf4b-facf-6e07-1378a485657d@I-love.SAKURA.ne.jp/
https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/
https://lore.kernel.org/all/b6e00e77-4a8c-4e05-ab79-266bf05fcc2d@igalia.com/
I’ve added documentation describing DEPT — this should help you
understand what DEPT is and how it works. You can use DEPT simply by
enabling CONFIG_DEPT and checking dmesg at runtime.
---
Hi Linus and folks,
I’ve been developing a tool to detect deadlock possibilities by tracking
waits/events — rather than lock acquisition order — to cover all the
synchronization mechanisms. To summarize the design rationale, starting
from the problem statement, through analysis, to the solution:
CURRENT STATUS
--------------
Lockdep tracks lock acquisition order to identify deadlock conditions.
Additionally, it tracks IRQ state changes — via {en,dis}able — to
detect cases where locks are acquired unintentionally during
interrupt handling.
PROBLEM
-------
Waits and their associated events that are never reachable can
eventually lead to deadlocks. However, since Lockdep focuses solely
on lock acquisition order, it has inherent limitations when handling
waits and events.
Moreover, by tracking only lock acquisition order, Lockdep cannot
properly handle read locks or cross-event scenarios — such as
wait_for_completion() and complete() — making it increasingly
inadequate as a general-purpose deadlock detection tool.
SOLUTION
--------
Once again, waits and their associated events that are never
reachable can eventually lead to deadlocks. The new solution, DEPT,
focuses directly on waits and events. DEPT monitors waits and events,
and reports them when any become unreachable.
DEPT provides:
* Correct handling of read locks.
* Support for general waits and events.
* Continuous operation, even after multiple reports.
* Simple, intuitive annotation APIs.
There are still false positives, and some are already being worked on
for suppression. Especially splitting the folio class into several
appropriate classes e.g. block device mapping class and regular file
mapping class, is currently under active development by me and Yeoreum
Yun.
Anyway, these efforts will need to continue for a while, as we’ve seen
with lockdep over two decades. DEPT is tagged as EXPERIMENTAL in
Kconfig — meaning it’s not yet suitable for use as an automation tool.
However, for those who are interested in using DEPT to analyze complex
synchronization patterns and extract dependency insights, DEPT would be
a great tool for the purpose.
Thanks for your support and contributions to:
Harry Yoo <harry.yoo@oracle.com>
Gwan-gyeong Mun <gwan-gyeong.mun@intel.com>
Yunseong Kim <ysk@kzalloc.com>
Yeoreum Yun <yeoreum.yun@arm.com>
FAQ
---
Q. Is this the first attempt to solve this problem?
A. No. The cross-release feature (commit b09be676e0ff2) attempted to
address it — as a Lockdep extension. It was merged, but quickly
reverted, because:
While it uncovered valuable hidden issues, it also introduced false
positives. Since these false positives mask further real problems
with Lockdep — and developers strongly dislike them — the feature was
rolled back.
Q. Why wasn’t DEPT built as a Lockdep extension?
A. Lockdep is the result of years of work by kernel developers — and is
now very stable. But I chose to build DEPT separately, because:
While reusing BFS(Breadth First Search) and Lockdep’s hashing is
beneficial, the rest of the system must be rebuilt from scratch to
align with DEPT’s wait-event model — since Lockdep was originally
designed for tracking lock acquisition orders, not wait-event
dependencies.
Q. Do you plan to replace Lockdep entirely?
A. Not at all — Lockdep still plays a vital role in validating correct
lock usage. While its dependency-checking logic should eventually be
superseded by DEPT, the rest of its functionality should stay.
Q. Should we replace the dependency check immediately?
A. Absolutely not. Lockdep’s stability is the result of years of hard
work by kernel developers. Lockdep and DEPT should run side by side
until DEPT matures.
Q. Stronger detection often leads to more false positives — which was a
major pain point when cross-release was added. Is DEPT designed to
handle this?
A. Yes. DEPT’s simple, generalized design enables flexible reporting —
so while false positives still need fixing, they’re far less
disruptive than they were under the Lockdep extension, cross-release.
Q. Why not fix all false positives out-of-tree before merging?
A. Since the affected subsystems span the entire kernel, like Lockdep,
which has relied on annotations to avoid false positives over the
last two decades, DEPT too will require the annotation efforts.
Performing annotation work within the mainline will help us add
annotations more appropriately and will also make DEPT a useful tool
for a wider range of users more quickly.
CONFIG_DEPT is marked EXPERIMENTAL, so it’s opt-in. Some users are
already interested in using DEPT to analyze complex synchronization
patterns and extract dependency insights.
Byungchul
---
Changes from v17:
1. Rebase on the mainline as of 2025 Dec 5.
2. Convert the documents' format from txt to rst. (feedbacked
by Jonathan Corbet and Bagas Sanjaya)
3. Move the documents from 'Documentation/dependency' to
'Documentation/dev-tools'. (feedbakced by Jonathan Corbet)
4. Improve the documentation. (feedbacked by NeilBrown)
5. Use a common function, enter_from_user_mode(), instead of
arch specific code, to notice context switch from user mode.
(feedbacked by Dave Hansen, Mark Rutland, and Mark Brown)
6. Resolve the header dependency issue by using dept's internal
header, instead of relocating 'struct llist_{head,node}' to
another header. (feedbacked by Greg KH)
7. Improve page(or folio) usage type APIs.
8. Add rust helper for wait_for_completion(). (feedbacked by
Guangbo Cui, Boqun Feng, and Danilo Krummrich)
9. Refine some commit messages.
Changes from v16:
1. Rebase on v6.17.
2. Fix a false positive from rcu (by Yunseong Kim)
3. Introduce APIs to set page's usage, dept_set_page_usage() and
dept_reset_page_usage() to avoid false positives.
4. Consider lock_page() as a potential wait unconditionally.
5. Consider folio_lock_killable() as a potential wait
unconditionally.
6. Add support for tracking PG_writeback waits and events.
7. Fix two build errors due to the additional debug information
added by dept. (by Yunseong Kim)
Changes from v15:
1. Fix typo and improve comments and commit messages (feedbacked
by ALOK TIWARI, Waiman Long, and kernel test robot).
2. Do not stop dept on detection of cicular dependency of
recover event, allowing to keep reporting.
3. Add SK hynix to copyright.
4. Consider folio_lock() as a potential wait unconditionally.
5. Fix Kconfig dependency bug (feedbacked by kernel test rebot).
6. Do not suppress reports that involve classes even that have
already involved in other reports, allowing to keep
reporting.
Changes from v14:
1. Rebase on the current latest, v6.15-rc6.
2. Refactor dept code.
3. With multi event sites for a single wait, even if an event
forms a circular dependency, the event can be recovered by
other event(or wake up) paths. Even though informing the
circular dependency is worthy but it should be suppressed
once informing it, if it doesn't lead an actual deadlock. So
introduce APIs to annotate the relationship between event
site and recover site, that are, event_site() and
dept_recover_event().
4. wait_for_completion() worked with dept map embedded in struct
completion. However, it generates a few false positves since
all the waits using the instance of struct completion, share
the map and key. To avoid the false positves, make it not to
share the map and key but each wait_for_completion() caller
have its own key by default. Of course, external maps also
can be used if needed.
5. Fix a bug about hardirq on/off tracing.
6. Implement basic unit test for dept.
7. Add more supports for dma fence synchronization.
8. Add emergency stop of dept e.g. on panic().
9. Fix false positives by mmu_notifier_invalidate_*().
10. Fix recursive call bug by DEPT_WARN_*() and DEPT_STOP().
11. Fix trivial bugs in DEPT_WARN_*() and DEPT_STOP().
12. Fix a bug that a spin lock, dept_pool_spin, is used in
both contexts of irq disabled and enabled without irq
disabled.
13. Suppress reports with classes, any of that already have
been reported, even though they have different chains but
being barely meaningful.
14. Print stacktrace of the wait that an event is now waking up,
not only stacktrace of the event.
15. Make dept aware of lockdep_cmp_fn() that is used to avoid
false positives in lockdep so that dept can also avoid them.
16. Do do_event() only if there are no ecxts have been
delimited.
17. Fix a bug that was not synchronized for stage_m in struct
dept_task, using a spin lock, dept_task()->stage_lock.
18. Fix a bug that dept didn't handle the case that multiple
ttwus for a single waiter can be called at the same time
e.i. a race issue.
19. Distinguish each kernel context from others, not only by
system call but also by user oriented fault so that dept can
work with more accuracy information about kernel context.
That helps to avoid a few false positives.
20. Limit dept's working to x86_64 and arm64.
Changes from v13:
1. Rebase on the current latest version, v6.9-rc7.
2. Add 'dept' documentation describing dept APIs.
Changes from v12:
1. Refine the whole document for dept.
2. Add 'Interpret dept report' section in the document, using a
deadlock report obtained in practice. Hope this version of
document helps guys understand dept better.
https://lore.kernel.org/lkml/6383cde5-cf4b-facf-6e07-1378a485657d@I-love.SAKURA.ne.jp/#t
https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/
Changes from v11:
1. Add 'dept' documentation describing the concept of dept.
2. Rewrite the commit messages of the following commits for
using weaker lockdep annotation, for better description.
fs/jbd2: Use a weaker annotation in journal handling
cpu/hotplug: Use a weaker annotation in AP thread
(feedbacked by Thomas Gleixner)
Changes from v10:
1. Fix noinstr warning when building kernel source.
2. dept has been reporting some false positives due to the folio
lock's unfairness. Reflect it and make dept work based on
dept annotaions instead of just wait and wake up primitives.
3. Remove the support for PG_writeback while working on 2. I
will add the support later if needed.
4. dept didn't print stacktrace for [S] if the participant of a
deadlock is not lock mechanism but general wait and event.
However, it made hard to interpret the report in that case.
So add support to print stacktrace of the requestor who asked
the event context to run - usually a waiter of the event does
it just before going to wait state.
5. Give up tracking raw_local_irq_{disable,enable}() since it
totally messed up dept's irq tracking. So make it work in the
same way as lockdep does. I will consider it once any false
positives by those are observed again.
6. Change the manual rwsem_acquire_read(->j_trans_commit_map)
annotation in fs/jbd2/transaction.c to the try version so
that it works as much as it exactly needs.
7. Remove unnecessary 'inline' keyword in dept.c and add
'__maybe_unused' to a needed place.
Changes from v9:
1. Fix a bug. SDT tracking didn't work well because of my big
mistake that I should've used waiter's map to indentify its
class but it had been working with waker's one. FYI,
PG_locked and PG_writeback weren't affected. They still
worked well. (reported by YoungJun)
Changes from v8:
1. Fix build error by adding EXPORT_SYMBOL(PG_locked_map) and
EXPORT_SYMBOL(PG_writeback_map) for kernel module build -
appologize for that. (reported by kernel test robot)
2. Fix build error by removing header file's circular dependency
that was caused by "atomic.h", "kernel.h" and "irqflags.h",
which I introduced - appolgize for that. (reported by kernel
test robot)
Changes from v7:
1. Fix a bug that cannot track rwlock dependency properly,
introduced in v7. (reported by Boqun and lockdep selftest)
2. Track wait/event of PG_{locked,writeback} more aggressively
assuming that when a bit of PG_{locked,writeback} is cleared
there might be waits on the bit. (reported by Linus, Hillf
and syzbot)
3. Fix and clean bad style code e.i. unnecessarily introduced
a randome pattern and so on. (pointed out by Linux)
4. Clean code for applying dept to wait_for_completion().
Changes from v6:
1. Tie to task scheduler code to track sleep and try_to_wake_up()
assuming sleeps cause waits, try_to_wake_up()s would be the
events that those are waiting for, of course with proper dept
annotations, sdt_might_sleep_weak(), sdt_might_sleep_strong()
and so on. For these cases, class is classified at sleep
entrance rather than the synchronization initialization code.
Which would extremely reduce false alarms.
2. Remove the dept associated instance in each page struct for
tracking dependencies by PG_locked and PG_writeback thanks to
the 1. work above.
3. Introduce CONFIG_dept_AGGRESIVE_TIMEOUT_WAIT to suppress
reports that waits with timeout set are involved, for those
who don't like verbose reporting.
4. Add a mechanism to refill the internal memory pools on
running out so that dept could keep working as long as free
memory is available in the system.
5. Re-enable tracking hashed-waitqueue wait. That's going to no
longer generate false positives because class is classified
at sleep entrance rather than the waitqueue initailization.
6. Refactor to make it easier to port onto each new version of
the kernel.
7. Apply dept to dma fence.
8. Do trivial optimizaitions.
Changes from v5:
1. Use just pr_warn_once() rather than WARN_ONCE() on the lack
of internal resources because WARN_*() printing stacktrace is
too much for informing the lack. (feedback from Ted, Hyeonggon)
2. Fix trivial bugs like missing initializing a struct before
using it.
3. Assign a different class per task when handling onstack
variables for waitqueue or the like. Which makes dept
distinguish between onstack variables of different tasks so
as to prevent false positives. (reported by Hyeonggon)
4. Make dept aware of even raw_local_irq_*() to prevent false
positives. (reported by Hyeonggon)
5. Don't consider dependencies between the events that might be
triggered within __schedule() and the waits that requires
__schedule(), real ones. (reported by Hyeonggon)
6. Unstage the staged wait that has prepare_to_wait_event()'ed
*and* yet to get to __schedule(), if we encounter __schedule()
in-between for another sleep, which is possible if e.g. a
mutex_lock() exists in 'condition' of ___wait_event().
7. Turn on CONFIG_PROVE_LOCKING when CONFIG_DEPT is on, to rely
on the hardirq and softirq entrance tracing to make dept more
portable for now.
Changes from v4:
1. Fix some bugs that produce false alarms.
2. Distinguish each syscall context from another *for arm64*.
3. Make it not warn it but just print it in case dept ring
buffer gets exhausted. (feedback from Hyeonggon)
4. Explicitely describe "EXPERIMENTAL" and "dept might produce
false positive reports" in Kconfig. (feedback from Ted)
Changes from v3:
1. dept shouldn't create dependencies between different depths
of a class that were indicated by *_lock_nested(). dept
normally doesn't but it does once another lock class comes
in. So fixed it. (feedback from Hyeonggon)
2. dept considered a wait as a real wait once getting to
__schedule() even if it has been set to TASK_RUNNING by wake
up sources in advance. Fixed it so that dept doesn't consider
the case as a real wait. (feedback from Jan Kara)
3. Stop tracking dependencies with a map once the event
associated with the map has been handled. dept will start to
work with the map again, on the next sleep.
Changes from v2:
1. Disable dept on bit_wait_table[] in sched/wait_bit.c
reporting a lot of false positives, which is my fault.
Wait/event for bit_wait_table[] should've been tagged in a
higher layer for better work, which is a future work.
(feedback from Jan Kara)
2. Disable dept on crypto_larval's completion to prevent a false
positive.
Changes from v1:
1. Fix coding style and typo. (feedback from Steven)
2. Distinguish each work context from another in workqueue.
3. Skip checking lock acquisition with nest_lock, which is about
correct lock usage that should be checked by lockdep.
Changes from RFC(v0):
1. Prevent adding a wait tag at prepare_to_wait() but __schedule().
(feedback from Linus and Matthew)
2. Use try version at lockdep_acquire_cpus_lock() annotation.
3. Distinguish each syscall context from another.
Byungchul Park (41):
dept: implement DEPT(DEPendency Tracker)
dept: add single event dependency tracker APIs
dept: add lock dependency tracker APIs
dept: tie to lockdep and IRQ tracing
dept: add proc knobs to show stats and dependency graph
dept: distinguish each kernel context from another
dept: distinguish each work from another
dept: add a mechanism to refill the internal memory pools on running
out
dept: record the latest one out of consecutive waits of the same class
dept: apply sdt_might_sleep_{start,end}() to
wait_for_completion()/complete()
dept: apply sdt_might_sleep_{start,end}() to swait
dept: apply sdt_might_sleep_{start,end}() to waitqueue wait
dept: apply sdt_might_sleep_{start,end}() to hashed-waitqueue wait
dept: apply sdt_might_sleep_{start,end}() to dma fence
dept: track timeout waits separately with a new Kconfig
dept: apply timeout consideration to wait_for_completion()/complete()
dept: apply timeout consideration to swait
dept: apply timeout consideration to waitqueue wait
dept: apply timeout consideration to hashed-waitqueue wait
dept: apply timeout consideration to dma fence wait
dept: make dept able to work with an external wgen
dept: track PG_locked with dept
dept: print staged wait's stacktrace on report
locking/lockdep: prevent various lockdep assertions when
lockdep_off()'ed
dept: add documents for dept
cpu/hotplug: use a weaker annotation in AP thread
dept: assign dept map to mmu notifier invalidation synchronization
dept: assign unique dept_key to each distinct dma fence caller
dept: make dept aware of lockdep_set_lock_cmp_fn() annotation
dept: make dept stop from working on debug_locks_off()
dept: assign unique dept_key to each distinct wait_for_completion()
caller
completion, dept: introduce init_completion_dmap() API
dept: introduce a new type of dependency tracking between multi event
sites
dept: add module support for struct dept_event_site and
dept_event_site_dep
dept: introduce event_site() to disable event tracking if it's
recoverable
dept: implement a basic unit test for dept
dept: call dept_hardirqs_off() in local_irq_*() regardless of irq
state
dept: introduce APIs to set page usage and use subclasses_evt for the
usage
dept: track PG_writeback with dept
SUNRPC: relocate struct rcu_head to the first field of struct rpc_xprt
mm: percpu: increase PERCPU_DYNAMIC_SIZE_SHIFT on DEPT and large
PAGE_SIZE
Yunseong Kim (1):
rcu/update: fix same dept key collision between various types of RCU
Documentation/dev-tools/dept.rst | 778 ++++++
Documentation/dev-tools/dept_api.rst | 125 +
drivers/dma-buf/dma-fence.c | 23 +-
include/asm-generic/vmlinux.lds.h | 13 +-
include/linux/completion.h | 124 +-
include/linux/dept.h | 402 +++
include/linux/dept_ldt.h | 78 +
include/linux/dept_sdt.h | 68 +
include/linux/dept_unit_test.h | 67 +
include/linux/dma-fence.h | 74 +-
include/linux/hardirq.h | 3 +
include/linux/irq-entry-common.h | 4 +
include/linux/irqflags.h | 21 +-
include/linux/local_lock_internal.h | 1 +
include/linux/lockdep.h | 105 +-
include/linux/lockdep_types.h | 3 +
include/linux/mm_types.h | 4 +
include/linux/mmu_notifier.h | 26 +
include/linux/module.h | 5 +
include/linux/mutex.h | 1 +
include/linux/page-flags.h | 217 +-
include/linux/pagemap.h | 37 +-
include/linux/percpu-rwsem.h | 2 +-
include/linux/percpu.h | 4 +
include/linux/rcupdate_wait.h | 13 +-
include/linux/rtmutex.h | 1 +
include/linux/rwlock_types.h | 1 +
include/linux/rwsem.h | 1 +
include/linux/sched.h | 118 +
include/linux/seqlock.h | 2 +-
include/linux/spinlock_types_raw.h | 3 +
include/linux/srcu.h | 2 +-
include/linux/sunrpc/xprt.h | 9 +-
include/linux/swait.h | 3 +
include/linux/wait.h | 3 +
include/linux/wait_bit.h | 3 +
init/init_task.c | 2 +
init/main.c | 2 +
kernel/Makefile | 1 +
kernel/cpu.c | 2 +-
kernel/dependency/Makefile | 5 +
kernel/dependency/dept.c | 3499 ++++++++++++++++++++++++++
kernel/dependency/dept_hash.h | 10 +
kernel/dependency/dept_internal.h | 314 +++
kernel/dependency/dept_object.h | 13 +
kernel/dependency/dept_proc.c | 94 +
kernel/dependency/dept_unit_test.c | 173 ++
kernel/exit.c | 1 +
kernel/fork.c | 2 +
kernel/locking/lockdep.c | 33 +
kernel/module/main.c | 19 +
kernel/rcu/rcu.h | 1 +
kernel/rcu/update.c | 5 +-
kernel/sched/completion.c | 62 +-
kernel/sched/core.c | 9 +
kernel/workqueue.c | 3 +
lib/Kconfig.debug | 48 +
lib/debug_locks.c | 2 +
lib/locking-selftest.c | 2 +
mm/filemap.c | 38 +
mm/mm_init.c | 3 +
mm/mmu_notifier.c | 31 +-
rust/helpers/completion.c | 5 +
63 files changed, 6602 insertions(+), 121 deletions(-)
create mode 100644 Documentation/dev-tools/dept.rst
create mode 100644 Documentation/dev-tools/dept_api.rst
create mode 100644 include/linux/dept.h
create mode 100644 include/linux/dept_ldt.h
create mode 100644 include/linux/dept_sdt.h
create mode 100644 include/linux/dept_unit_test.h
create mode 100644 kernel/dependency/Makefile
create mode 100644 kernel/dependency/dept.c
create mode 100644 kernel/dependency/dept_hash.h
create mode 100644 kernel/dependency/dept_internal.h
create mode 100644 kernel/dependency/dept_object.h
create mode 100644 kernel/dependency/dept_proc.c
create mode 100644 kernel/dependency/dept_unit_test.c
base-commit: 43dfc13ca972988e620a6edb72956981b75ab6b0
--
2.17.1
^ permalink raw reply
* [PATCH v18 02/42] dept: add single event dependency tracker APIs
From: Byungchul Park @ 2025-12-05 7:18 UTC (permalink / raw)
To: linux-kernel
Cc: kernel_team, torvalds, damien.lemoal, linux-ide, adilger.kernel,
linux-ext4, mingo, peterz, will, tglx, rostedt, joel, sashal,
daniel.vetter, duyuyang, johannes.berg, tj, tytso, willy, david,
amir73il, gregkh, kernel-team, linux-mm, akpm, mhocko, minchan,
hannes, vdavydov.dev, sj, jglisse, dennis, cl, penberg, rientjes,
vbabka, ngupta, linux-block, josef, linux-fsdevel, jack, jlayton,
dan.j.williams, hch, djwong, dri-devel, rodrigosiqueiramelo,
melissa.srw, hamohammed.sa, harry.yoo, chris.p.wilson,
gwan-gyeong.mun, max.byungchul.park, boqun.feng, longman,
yunseong.kim, ysk, yeoreum.yun, netdev, matthew.brost, her0gyugyu,
corbet, catalin.marinas, bp, x86, hpa, luto, sumit.semwal,
gustavo, christian.koenig, andi.shyti, arnd, lorenzo.stoakes,
Liam.Howlett, rppt, surenb, mcgrof, petr.pavlu, da.gomez,
samitolvanen, paulmck, frederic, neeraj.upadhyay, joelagnelf,
josh, urezki, mathieu.desnoyers, jiangshanlai, qiang.zhang,
juri.lelli, vincent.guittot, dietmar.eggemann, bsegall, mgorman,
vschneid, chuck.lever, neil, okorniev, Dai.Ngo, tom, trondmy,
anna, kees, bigeasy, clrkwllms, mark.rutland, ada.coupriediaz,
kristina.martsenko, wangkefeng.wang, broonie, kevin.brodsky, dwmw,
shakeel.butt, ast, ziy, yuzhao, baolin.wang, usamaarif642,
joel.granados, richard.weiyang, geert+renesas, tim.c.chen, linux,
alexander.shishkin, lillian, chenhuacai, francesco,
guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel,
2407018371, dakr, miguel.ojeda.sandonis, neilb, bagasdotme,
wsa+renesas, dave.hansen, geert, ojeda, alex.gaynor, gary,
bjorn3_gh, lossin, a.hindborg, aliceryhl, tmgross, rust-for-linux
In-Reply-To: <20251205071855.72743-1-byungchul@sk.com>
Wrapped the base APIs for easier annotation on wait and event. Start
with supporting waiters on each single event. More general support for
multiple events is a future work. Do more when the need arises.
How to annotate:
1. Initaialize a map for the interesting wait.
/*
* Place along with the wait instance.
*/
struct dept_map my_wait;
/*
* Place in the initialization code.
*/
sdt_map_init(&my_wait);
2. Place the following at the wait code.
sdt_wait(&my_wait);
3. Place the following at the event code.
sdt_event(&my_wait);
That's it!
Signed-off-by: Byungchul Park <byungchul@sk.com>
---
include/linux/dept_sdt.h | 65 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
create mode 100644 include/linux/dept_sdt.h
diff --git a/include/linux/dept_sdt.h b/include/linux/dept_sdt.h
new file mode 100644
index 000000000000..0535f763b21b
--- /dev/null
+++ b/include/linux/dept_sdt.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Single-event Dependency Tracker
+ *
+ * Started by Byungchul Park <max.byungchul.park@gmail.com>:
+ *
+ * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park
+ * Copyright (c) 2024 SK hynix, Inc., Byungchul Park
+ */
+
+#ifndef __LINUX_DEPT_SDT_H
+#define __LINUX_DEPT_SDT_H
+
+#include <linux/kernel.h>
+#include <linux/dept.h>
+
+#ifdef CONFIG_DEPT
+#define sdt_map_init(m) \
+ do { \
+ static struct dept_key __key; \
+ dept_map_init(m, &__key, 0, #m); \
+ } while (0)
+
+#define sdt_map_init_key(m, k) dept_map_init(m, k, 0, #m)
+
+#define sdt_wait(m) \
+ do { \
+ dept_request_event(m); \
+ dept_wait(m, 1UL, _THIS_IP_, __func__, 0); \
+ } while (0)
+
+/*
+ * sdt_might_sleep() and its family will be committed in __schedule()
+ * when it actually gets to __schedule(). Both dept_request_event() and
+ * dept_wait() will be performed on the commit.
+ */
+
+/*
+ * Use the code location as the class key if an explicit map is not used.
+ */
+#define sdt_might_sleep_start(m) \
+ do { \
+ struct dept_map *__m = m; \
+ static struct dept_key __key; \
+ dept_stage_wait(__m, __m ? NULL : &__key, _THIS_IP_, __func__);\
+ } while (0)
+
+#define sdt_might_sleep_end() dept_clean_stage()
+
+#define sdt_ecxt_enter(m) dept_ecxt_enter(m, 1UL, _THIS_IP_, "start", "event", 0)
+#define sdt_event(m) dept_event(m, 1UL, _THIS_IP_, __func__)
+#define sdt_ecxt_exit(m) dept_ecxt_exit(m, 1UL, _THIS_IP_)
+#define sdt_request_event(m) dept_request_event(m)
+#else /* !CONFIG_DEPT */
+#define sdt_map_init(m) do { } while (0)
+#define sdt_map_init_key(m, k) do { (void)(k); } while (0)
+#define sdt_wait(m) do { } while (0)
+#define sdt_might_sleep_start(m) do { } while (0)
+#define sdt_might_sleep_end() do { } while (0)
+#define sdt_ecxt_enter(m) do { } while (0)
+#define sdt_event(m) do { } while (0)
+#define sdt_ecxt_exit(m) do { } while (0)
+#define sdt_request_event(m) do { } while (0)
+#endif
+#endif /* __LINUX_DEPT_SDT_H */
--
2.17.1
^ permalink raw reply related
* [PATCH v18 01/42] dept: implement DEPT(DEPendency Tracker)
From: Byungchul Park @ 2025-12-05 7:18 UTC (permalink / raw)
To: linux-kernel
Cc: kernel_team, torvalds, damien.lemoal, linux-ide, adilger.kernel,
linux-ext4, mingo, peterz, will, tglx, rostedt, joel, sashal,
daniel.vetter, duyuyang, johannes.berg, tj, tytso, willy, david,
amir73il, gregkh, kernel-team, linux-mm, akpm, mhocko, minchan,
hannes, vdavydov.dev, sj, jglisse, dennis, cl, penberg, rientjes,
vbabka, ngupta, linux-block, josef, linux-fsdevel, jack, jlayton,
dan.j.williams, hch, djwong, dri-devel, rodrigosiqueiramelo,
melissa.srw, hamohammed.sa, harry.yoo, chris.p.wilson,
gwan-gyeong.mun, max.byungchul.park, boqun.feng, longman,
yunseong.kim, ysk, yeoreum.yun, netdev, matthew.brost, her0gyugyu,
corbet, catalin.marinas, bp, x86, hpa, luto, sumit.semwal,
gustavo, christian.koenig, andi.shyti, arnd, lorenzo.stoakes,
Liam.Howlett, rppt, surenb, mcgrof, petr.pavlu, da.gomez,
samitolvanen, paulmck, frederic, neeraj.upadhyay, joelagnelf,
josh, urezki, mathieu.desnoyers, jiangshanlai, qiang.zhang,
juri.lelli, vincent.guittot, dietmar.eggemann, bsegall, mgorman,
vschneid, chuck.lever, neil, okorniev, Dai.Ngo, tom, trondmy,
anna, kees, bigeasy, clrkwllms, mark.rutland, ada.coupriediaz,
kristina.martsenko, wangkefeng.wang, broonie, kevin.brodsky, dwmw,
shakeel.butt, ast, ziy, yuzhao, baolin.wang, usamaarif642,
joel.granados, richard.weiyang, geert+renesas, tim.c.chen, linux,
alexander.shishkin, lillian, chenhuacai, francesco,
guoweikang.kernel, link, jpoimboe, masahiroy, brauner,
thomas.weissschuh, oleg, mjguzik, andrii, wangfushuai, linux-doc,
linux-arm-kernel, linux-media, linaro-mm-sig, linux-i2c,
linux-arch, linux-modules, rcu, linux-nfs, linux-rt-devel,
2407018371, dakr, miguel.ojeda.sandonis, neilb, bagasdotme,
wsa+renesas, dave.hansen, geert, ojeda, alex.gaynor, gary,
bjorn3_gh, lossin, a.hindborg, aliceryhl, tmgross, rust-for-linux
In-Reply-To: <20251205071855.72743-1-byungchul@sk.com>
CURRENT STATUS
--------------
Lockdep tracks lock acquisition order to identify deadlock conditions.
Additionally, it tracks IRQ state changes — via {en,dis}able — to detect
cases where locks are acquired unintentionally during interrupt handling.
PROBLEM
-------
Waits and their associated events that are never reachable can
eventually lead to deadlocks. However, since Lockdep focuses solely on
lock acquisition order, it has inherent limitations when handling waits
and events.
Moreover, by tracking only lock acquisition order, Lockdep cannot
properly handle read locks or cross-event scenarios — such as
wait_for_completion() and complete() — making it increasingly inadequate
as a general-purpose deadlock detection tool.
SOLUTION
--------
Once again, waits and their associated events that are never reachable
can eventually lead to deadlocks. The new solution, DEPT, focuses
directly on waits and events. DEPT monitors waits and events, and
reports them when any become unreachable.
DEPT provides:
* Correct handling of read locks.
* Support for general waits and events.
* Continuous operation, even after multiple reports.
* Simple, intuitive annotation APIs.
Q & A
-----
Q. Is this the first attempt to solve this problem?
A. No. The cross-release feature (commit b09be676e0ff2) attempted to
address it — as a Lockdep extension. It was merged, but quickly
reverted, because:
While it uncovered valuable hidden issues, it also introduced false
positives. Since these false positives mask further real problems
with Lockdep — and developers strongly dislike them — the feature was
rolled back.
Q. Why wasn’t DEPT built as a Lockdep extension?
A. Lockdep is the result of years of work by kernel developers — and is
now very stable. But I chose to build DEPT separately, because:
While reusing BFS(Breadth First Search) and Lockdep’s hashing is
beneficial, the rest of the system must be rebuilt from scratch to
align with DEPT’s wait-event model — since Lockdep was originally
designed for tracking lock acquisition orders, not wait-event
dependencies.
Q. Do you plan to replace Lockdep entirely?
A. Not at all — Lockdep still plays a vital role in validating correct
lock usage. While its dependency-checking logic should eventually be
superseded by DEPT, the rest of its functionality should stay.
Q. Should we replace the dependency check immediately?
A. Absolutely not. Lockdep’s stability is the result of years of hard
work by kernel developers. Lockdep and DEPT should run side by side
until DEPT matures.
Q. Stronger detection often leads to more false positives — which was a
major pain point when cross-release was added. Is DEPT designed to
handle this?
A. Yes. DEPT’s simple, generalized design enables flexible reporting —
so while false positives still need fixing, they’re far less
disruptive than they were under the Lockdep extension, cross-release.
Q. Why not fix all false positives out-of-tree before merging?
A. Since the affected subsystems span the entire kernel, like Lockdep,
which has relied on annotations to avoid false positives over the
last two decades, dept too will require the annotation efforts.
Performing annotation work within the mainline will help us add
annotations more appropriately and will also make dept a useful tool
for a wider range of users more quickly.
CONFIG_DEPT is marked EXPERIMENTAL, so it’s opt-in. Some users are
already interested in using DEPT to analyze complex synchronization
patterns and extract dependency insights.
Signed-off-by: Byungchul Park <byungchul@sk.com>
Tested-by: Harry Yoo <harry.yoo@oracle.com>
Tested-by: Gwan-gyeong Mun <gwan-gyeong.mun@intel.com>
Tested-by: Yunseong Kim <ysk@kzalloc.com>
Tested-by: Yeoreum Yun <yeoreum.yun@arm.com>
---
include/linux/dept.h | 208 ++
include/linux/hardirq.h | 3 +
include/linux/sched.h | 108 ++
init/init_task.c | 2 +
init/main.c | 2 +
kernel/Makefile | 1 +
kernel/dependency/Makefile | 3 +
kernel/dependency/dept.c | 2977 +++++++++++++++++++++++++++++
kernel/dependency/dept_hash.h | 10 +
kernel/dependency/dept_internal.h | 283 +++
kernel/dependency/dept_object.h | 13 +
kernel/exit.c | 1 +
kernel/fork.c | 2 +
kernel/module/main.c | 4 +
kernel/sched/core.c | 9 +
lib/Kconfig.debug | 26 +
lib/locking-selftest.c | 2 +
17 files changed, 3654 insertions(+)
create mode 100644 include/linux/dept.h
create mode 100644 kernel/dependency/Makefile
create mode 100644 kernel/dependency/dept.c
create mode 100644 kernel/dependency/dept_hash.h
create mode 100644 kernel/dependency/dept_internal.h
create mode 100644 kernel/dependency/dept_object.h
diff --git a/include/linux/dept.h b/include/linux/dept.h
new file mode 100644
index 000000000000..a6d14b585a8d
--- /dev/null
+++ b/include/linux/dept.h
@@ -0,0 +1,208 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * DEPT(DEPendency Tracker) - runtime dependency tracker
+ *
+ * Started by Byungchul Park <max.byungchul.park@gmail.com>:
+ *
+ * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park
+ * Copyright (c) 2024 SK hynix, Inc., Byungchul Park
+ */
+
+#ifndef __LINUX_DEPT_H
+#define __LINUX_DEPT_H
+
+#ifdef CONFIG_DEPT
+
+struct task_struct;
+
+#define DEPT_MAX_STACK_ENTRY 16
+#define DEPT_MAX_WAIT_HIST 64
+#define DEPT_MAX_ECXT_HELD 48
+
+#define DEPT_MAX_SUBCLASSES 16
+#define DEPT_MAX_SUBCLASSES_EVT 2
+#define DEPT_MAX_SUBCLASSES_USR (DEPT_MAX_SUBCLASSES / DEPT_MAX_SUBCLASSES_EVT)
+#define DEPT_MAX_SUBCLASSES_CACHE 2
+
+#define DEPT_SIRQ 0
+#define DEPT_HIRQ 1
+#define DEPT_IRQS_NR 2
+#define DEPT_SIRQF (1UL << DEPT_SIRQ)
+#define DEPT_HIRQF (1UL << DEPT_HIRQ)
+
+struct dept_key {
+ union {
+ /*
+ * Each byte-wise address will be used as its key.
+ */
+ char base[DEPT_MAX_SUBCLASSES];
+
+ /*
+ * for caching the main class pointer
+ */
+ struct dept_class *classes[DEPT_MAX_SUBCLASSES_CACHE];
+ };
+};
+
+struct dept_map {
+ const char *name;
+ struct dept_key *keys;
+
+ /*
+ * subclass that can be set from user
+ */
+ int sub_u;
+
+ /*
+ * It's local copy for fast access to the associated classes.
+ * Also used for dept_key for static maps.
+ */
+ struct dept_key map_key;
+
+ /*
+ * wait timestamp associated to this map
+ */
+ unsigned int wgen;
+
+ /*
+ * whether this map should be going to be checked or not
+ */
+ bool nocheck;
+};
+
+#define DEPT_MAP_INITIALIZER(n, k) \
+{ \
+ .name = #n, \
+ .keys = (struct dept_key *)(k), \
+ .sub_u = 0, \
+ .map_key = { .classes = { NULL, } }, \
+ .wgen = 0U, \
+ .nocheck = false, \
+}
+
+struct dept_ecxt_held {
+ /*
+ * associated event context
+ */
+ struct dept_ecxt *ecxt;
+
+ /*
+ * unique key for this dept_ecxt_held
+ */
+ struct dept_map *map;
+
+ /*
+ * class of the ecxt of this dept_ecxt_held
+ */
+ struct dept_class *class;
+
+ /*
+ * the wgen when the event context started
+ */
+ unsigned int wgen;
+
+ /*
+ * subclass that only works in the local context
+ */
+ int sub_l;
+};
+
+struct dept_wait_hist {
+ /*
+ * associated wait
+ */
+ struct dept_wait *wait;
+
+ /*
+ * unique id of all waits system-wise until wrapped
+ */
+ unsigned int wgen;
+
+ /*
+ * local context id to identify IRQ context
+ */
+ unsigned int ctxt_id;
+};
+
+extern void dept_on(void);
+extern void dept_off(void);
+extern void dept_init(void);
+extern void dept_task_init(struct task_struct *t);
+extern void dept_task_exit(struct task_struct *t);
+extern void dept_free_range(void *start, unsigned int sz);
+
+extern void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u, const char *n);
+extern void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, const char *n);
+extern void dept_map_copy(struct dept_map *to, struct dept_map *from);
+extern void dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l);
+extern void dept_stage_wait(struct dept_map *m, struct dept_key *k, unsigned long ip, const char *w_fn);
+extern void dept_request_event_wait_commit(void);
+extern void dept_clean_stage(void);
+extern void dept_ttwu_stage_wait(struct task_struct *t, unsigned long ip);
+extern void dept_ecxt_enter(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *c_fn, const char *e_fn, int sub_l);
+extern bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f);
+extern void dept_request_event(struct dept_map *m);
+extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *e_fn);
+extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long ip);
+extern void dept_sched_enter(void);
+extern void dept_sched_exit(void);
+
+static inline void dept_ecxt_enter_nokeep(struct dept_map *m)
+{
+ dept_ecxt_enter(m, 0UL, 0UL, NULL, NULL, 0);
+}
+
+/*
+ * for users who want to manage external keys
+ */
+extern void dept_key_init(struct dept_key *k);
+extern void dept_key_destroy(struct dept_key *k);
+extern void dept_map_ecxt_modify(struct dept_map *m, unsigned long e_f, struct dept_key *new_k, unsigned long new_e_f, unsigned long new_ip, const char *new_c_fn, const char *new_e_fn, int new_sub_l);
+
+extern void dept_softirq_enter(void);
+extern void dept_hardirq_enter(void);
+extern void dept_softirqs_on_ip(unsigned long ip);
+extern void dept_hardirqs_on(void);
+extern void dept_softirqs_off(void);
+extern void dept_hardirqs_off(void);
+#else /* !CONFIG_DEPT */
+struct dept_key { };
+struct dept_map { };
+
+#define DEPT_MAP_INITIALIZER(n, k) { }
+
+#define dept_on() do { } while (0)
+#define dept_off() do { } while (0)
+#define dept_init() do { } while (0)
+#define dept_task_init(t) do { } while (0)
+#define dept_task_exit(t) do { } while (0)
+#define dept_free_range(s, sz) do { } while (0)
+
+#define dept_map_init(m, k, su, n) do { (void)(n); (void)(k); } while (0)
+#define dept_map_reinit(m, k, su, n) do { (void)(n); (void)(k); } while (0)
+#define dept_map_copy(t, f) do { } while (0)
+#define dept_wait(m, w_f, ip, w_fn, sl) do { (void)(w_fn); } while (0)
+#define dept_stage_wait(m, k, ip, w_fn) do { (void)(k); (void)(w_fn); } while (0)
+#define dept_request_event_wait_commit() do { } while (0)
+#define dept_clean_stage() do { } while (0)
+#define dept_ttwu_stage_wait(t, ip) do { } while (0)
+#define dept_ecxt_enter(m, e_f, ip, c_fn, e_fn, sl) do { (void)(c_fn); (void)(e_fn); } while (0)
+#define dept_ecxt_holding(m, e_f) false
+#define dept_request_event(m) do { } while (0)
+#define dept_event(m, e_f, ip, e_fn) do { (void)(e_fn); } while (0)
+#define dept_ecxt_exit(m, e_f, ip) do { } while (0)
+#define dept_sched_enter() do { } while (0)
+#define dept_sched_exit() do { } while (0)
+#define dept_ecxt_enter_nokeep(m) do { } while (0)
+#define dept_key_init(k) do { (void)(k); } while (0)
+#define dept_key_destroy(k) do { (void)(k); } while (0)
+#define dept_map_ecxt_modify(m, e_f, n_k, n_e_f, n_ip, n_c_fn, n_e_fn, n_sl) do { (void)(n_k); (void)(n_c_fn); (void)(n_e_fn); } while (0)
+
+#define dept_softirq_enter() do { } while (0)
+#define dept_hardirq_enter() do { } while (0)
+#define dept_softirqs_on_ip(ip) do { } while (0)
+#define dept_hardirqs_on() do { } while (0)
+#define dept_softirqs_off() do { } while (0)
+#define dept_hardirqs_off() do { } while (0)
+#endif
+#endif /* __LINUX_DEPT_H */
diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h
index d57cab4d4c06..bb279dbbe748 100644
--- a/include/linux/hardirq.h
+++ b/include/linux/hardirq.h
@@ -5,6 +5,7 @@
#include <linux/context_tracking_state.h>
#include <linux/preempt.h>
#include <linux/lockdep.h>
+#include <linux/dept.h>
#include <linux/ftrace_irq.h>
#include <linux/sched.h>
#include <linux/vtime.h>
@@ -106,6 +107,7 @@ void irq_exit_rcu(void);
*/
#define __nmi_enter() \
do { \
+ dept_off(); \
lockdep_off(); \
arch_nmi_enter(); \
BUG_ON(in_nmi() == NMI_MASK); \
@@ -128,6 +130,7 @@ void irq_exit_rcu(void);
__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \
arch_nmi_exit(); \
lockdep_on(); \
+ dept_on(); \
} while (0)
#define nmi_exit() \
diff --git a/include/linux/sched.h b/include/linux/sched.h
index d395f2810fac..33e713c3ba70 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -49,6 +49,8 @@
#include <linux/tracepoint-defs.h>
#include <linux/unwind_deferred_types.h>
#include <asm/kmap_size.h>
+#include <linux/spinlock.h>
+#include <linux/dept.h>
#ifndef COMPILE_OFFSETS
#include <generated/rq-offsets.h>
#endif
@@ -816,6 +818,110 @@ struct kmap_ctrl {
#endif
};
+#ifdef CONFIG_DEPT
+struct dept_task {
+ /*
+ * all event contexts that have entered and before exiting
+ */
+ struct dept_ecxt_held ecxt_held[DEPT_MAX_ECXT_HELD];
+ int ecxt_held_pos;
+
+ /*
+ * ring buffer holding all waits that have happened
+ */
+ struct dept_wait_hist wait_hist[DEPT_MAX_WAIT_HIST];
+ int wait_hist_pos;
+
+ /*
+ * sequential id to identify each IRQ context
+ */
+ unsigned int irq_id[DEPT_IRQS_NR];
+
+ /*
+ * for tracking IRQ-enabled points with cross-event
+ */
+ unsigned int wgen_enirq[DEPT_IRQS_NR];
+
+ /*
+ * for keeping up-to-date IRQ-enabled points
+ */
+ unsigned long enirq_ip[DEPT_IRQS_NR];
+
+ /*
+ * for reserving a current stack instance at each operation
+ */
+ struct dept_stack *stack;
+
+ /*
+ * for preventing recursive call into DEPT engine
+ */
+ int recursive;
+
+ /*
+ * for preventing reentrance to WARN*() while warning
+ */
+ int in_warning;
+
+ /*
+ * for staging data to commit a wait
+ */
+ struct dept_map stage_m;
+ struct dept_map *stage_real_m;
+ bool stage_sched_map;
+ const char *stage_w_fn;
+ unsigned long stage_ip;
+ arch_spinlock_t stage_lock;
+
+ /*
+ * the number of missing ecxts
+ */
+ int missing_ecxt;
+
+ /*
+ * for tracking IRQ-enable state
+ */
+ bool hardirqs_enabled;
+ bool softirqs_enabled;
+
+ /*
+ * whether the current is on do_exit()
+ */
+ bool task_exit;
+
+ /*
+ * whether the current is running __schedule()
+ */
+ bool in_sched;
+};
+
+#define DEPT_TASK_INITIALIZER(t) \
+{ \
+ .wait_hist = { { .wait = NULL, } }, \
+ .ecxt_held_pos = 0, \
+ .wait_hist_pos = 0, \
+ .irq_id = { 0U }, \
+ .wgen_enirq = { 0U }, \
+ .enirq_ip = { 0UL }, \
+ .stack = NULL, \
+ .recursive = 0, \
+ .in_warning = 0, \
+ .stage_m = DEPT_MAP_INITIALIZER((t)->stage_m, NULL), \
+ .stage_real_m = NULL, \
+ .stage_sched_map = false, \
+ .stage_w_fn = NULL, \
+ .stage_ip = 0UL, \
+ .stage_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED,\
+ .missing_ecxt = 0, \
+ .hardirqs_enabled = false, \
+ .softirqs_enabled = false, \
+ .task_exit = false, \
+ .in_sched = false, \
+}
+#else
+struct dept_task { };
+#define DEPT_TASK_INITIALIZER(t) { }
+#endif
+
struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK
/*
@@ -1274,6 +1380,8 @@ struct task_struct {
struct held_lock held_locks[MAX_LOCK_DEPTH];
#endif
+ struct dept_task dept_task;
+
#if defined(CONFIG_UBSAN) && !defined(CONFIG_UBSAN_TRAP)
unsigned int in_ubsan;
#endif
diff --git a/init/init_task.c b/init/init_task.c
index 49b13d7c3985..8ce30b835fc3 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -14,6 +14,7 @@
#include <linux/numa.h>
#include <linux/scs.h>
#include <linux/plist.h>
+#include <linux/dept.h>
#include <linux/uaccess.h>
@@ -234,6 +235,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
.curr_chain_key = INITIAL_CHAIN_KEY,
.lockdep_recursion = 0,
#endif
+ .dept_task = DEPT_TASK_INITIALIZER(init_task),
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
.ret_stack = NULL,
.tracing_graph_pause = ATOMIC_INIT(0),
diff --git a/init/main.c b/init/main.c
index 07a3116811c5..a3d7db9e3d7a 100644
--- a/init/main.c
+++ b/init/main.c
@@ -66,6 +66,7 @@
#include <linux/debug_locks.h>
#include <linux/debugobjects.h>
#include <linux/lockdep.h>
+#include <linux/dept.h>
#include <linux/kmemleak.h>
#include <linux/padata.h>
#include <linux/pid_namespace.h>
@@ -1051,6 +1052,7 @@ void start_kernel(void)
panic_param);
lockdep_init();
+ dept_init();
/*
* Need to run this when irqs are enabled, because it wants
diff --git a/kernel/Makefile b/kernel/Makefile
index 9fe722305c9b..859dab7af192 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -56,6 +56,7 @@ obj-y += dma/
obj-y += entry/
obj-y += unwind/
obj-$(CONFIG_MODULES) += module/
+obj-y += dependency/
obj-$(CONFIG_KCMP) += kcmp.o
obj-$(CONFIG_FREEZER) += freezer.o
diff --git a/kernel/dependency/Makefile b/kernel/dependency/Makefile
new file mode 100644
index 000000000000..b5cfb8a03c0c
--- /dev/null
+++ b/kernel/dependency/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_DEPT) += dept.o
diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c
new file mode 100644
index 000000000000..f2a9456f63f6
--- /dev/null
+++ b/kernel/dependency/dept.c
@@ -0,0 +1,2977 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DEPT(DEPendency Tracker) - Runtime dependency tracker
+ *
+ * Started by Byungchul Park <max.byungchul.park@gmail.com>:
+ *
+ * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park
+ * Copyright (c) 2024 SK hynix, Inc., Byungchul Park
+ *
+ * DEPT provides a general way to detect potential deadlocks at runtime
+ * and the interest is not limited to typical lock but to every
+ * synchronization primitives.
+ *
+ * The following ideas were borrowed from LOCKDEP:
+ *
+ * 1) Use a graph to track relationship between classes.
+ * 2) Prevent performance regression using hash.
+ *
+ * The following items were enhanced from LOCKDEP:
+ *
+ * 1) Cover more deadlock cases.
+ * 2) Allow multiple reports.
+ *
+ * TODO: Both LOCKDEP and DEPT should co-exist until DEPT is considered
+ * stable. Then the dependency check routine should be replaced with
+ * DEPT after. It should finally look like:
+ *
+ *
+ *
+ * As is:
+ *
+ * LOCKDEP
+ * +-----------------------------------------+
+ * | Lock usage correctness check | <-> locks
+ * | |
+ * | |
+ * | +-------------------------------------+ |
+ * | | Dependency check | |
+ * | | (by tracking lock acquisition order)| |
+ * | +-------------------------------------+ |
+ * | |
+ * +-----------------------------------------+
+ *
+ * DEPT
+ * +-----------------------------------------+
+ * | Dependency check | <-> waits/events
+ * | (by tracking wait and event context) |
+ * +-----------------------------------------+
+ *
+ *
+ *
+ * To be:
+ *
+ * LOCKDEP
+ * +-----------------------------------------+
+ * | Lock usage correctness check | <-> locks
+ * | |
+ * | |
+ * | (Request dependency check) |
+ * | T |
+ * +--------------------|--------------------+
+ * |
+ * DEPT V
+ * +-----------------------------------------+
+ * | Dependency check | <-> waits/events
+ * | (by tracking wait and event context) |
+ * +-----------------------------------------+
+ */
+
+#include <linux/sched.h>
+#include <linux/stacktrace.h>
+#include <linux/spinlock.h>
+#include <linux/kallsyms.h>
+#include <linux/hash.h>
+#include <linux/dept.h>
+#include <linux/utsname.h>
+#include <linux/kernel.h>
+#include "dept_internal.h"
+
+static int dept_stop;
+static int dept_per_cpu_ready;
+
+static inline struct dept_task *dept_task(void)
+{
+ return ¤t->dept_task;
+}
+
+#define DEPT_READY_WARN (!oops_in_progress && !dept_task()->in_warning)
+
+/*
+ * Make all operations using DEPT_WARN_ON() fail on oops_in_progress and
+ * prevent warning message.
+ */
+#define DEPT_WARN_ON_ONCE(c) \
+ ({ \
+ int __ret = !!(c); \
+ \
+ if (likely(DEPT_READY_WARN)) { \
+ ++dept_task()->in_warning; \
+ WARN_ONCE(c, "DEPT_WARN_ON_ONCE: " #c); \
+ --dept_task()->in_warning; \
+ } \
+ __ret; \
+ })
+
+#define DEPT_WARN_ONCE(s...) \
+ ({ \
+ if (likely(DEPT_READY_WARN)) { \
+ ++dept_task()->in_warning; \
+ WARN_ONCE(1, "DEPT_WARN_ONCE: " s); \
+ --dept_task()->in_warning; \
+ } \
+ })
+
+#define DEPT_WARN_ON(c) \
+ ({ \
+ int __ret = !!(c); \
+ \
+ if (likely(DEPT_READY_WARN)) { \
+ ++dept_task()->in_warning; \
+ WARN(c, "DEPT_WARN_ON: " #c); \
+ --dept_task()->in_warning; \
+ } \
+ __ret; \
+ })
+
+#define DEPT_WARN(s...) \
+ ({ \
+ if (likely(DEPT_READY_WARN)) { \
+ ++dept_task()->in_warning; \
+ WARN(1, "DEPT_WARN: " s); \
+ --dept_task()->in_warning; \
+ } \
+ })
+
+#define DEPT_STOP(s...) \
+ ({ \
+ WRITE_ONCE(dept_stop, 1); \
+ if (likely(DEPT_READY_WARN)) { \
+ ++dept_task()->in_warning; \
+ WARN(1, "DEPT_STOP: " s); \
+ --dept_task()->in_warning; \
+ } \
+ })
+
+#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s)
+
+static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
+
+/*
+ * DEPT internal engine should be cautious in using outside functions
+ * e.g. printk at reporting since that kind of usage might cause
+ * untrackable deadlock.
+ */
+static atomic_t dept_outworld = ATOMIC_INIT(0);
+
+static void dept_outworld_enter(void)
+{
+ atomic_inc(&dept_outworld);
+}
+
+static void dept_outworld_exit(void)
+{
+ atomic_dec(&dept_outworld);
+}
+
+static bool dept_outworld_entered(void)
+{
+ return atomic_read(&dept_outworld);
+}
+
+static bool dept_lock(void)
+{
+ while (!arch_spin_trylock(&dept_spin))
+ if (unlikely(dept_outworld_entered()))
+ return false;
+ return true;
+}
+
+static void dept_unlock(void)
+{
+ arch_spin_unlock(&dept_spin);
+}
+
+enum bfs_ret {
+ BFS_CONTINUE,
+ BFS_DONE,
+ BFS_SKIP,
+};
+
+static bool before(unsigned int a, unsigned int b)
+{
+ return (int)(a - b) < 0;
+}
+
+static bool valid_stack(struct dept_stack *s)
+{
+ return s && s->nr > 0;
+}
+
+static bool valid_class(struct dept_class *c)
+{
+ return c->key;
+}
+
+static void invalidate_class(struct dept_class *c)
+{
+ c->key = 0UL;
+}
+
+static struct dept_ecxt *dep_e(struct dept_dep *d)
+{
+ return d->ecxt;
+}
+
+static struct dept_wait *dep_w(struct dept_dep *d)
+{
+ return d->wait;
+}
+
+static struct dept_class *dep_fc(struct dept_dep *d)
+{
+ return dep_e(d)->class;
+}
+
+static struct dept_class *dep_tc(struct dept_dep *d)
+{
+ return dep_w(d)->class;
+}
+
+static const char *irq_str(int irq)
+{
+ if (irq == DEPT_SIRQ)
+ return "softirq";
+ if (irq == DEPT_HIRQ)
+ return "hardirq";
+ return "(unknown)";
+}
+
+/*
+ * Dept doesn't work either when it's stopped by DEPT_STOP() or in a nmi
+ * context.
+ */
+static bool dept_working(void)
+{
+ return !READ_ONCE(dept_stop) && !in_nmi();
+}
+
+/*
+ * Even k == NULL is considered as a valid key because it would use
+ * &->map_key as the key in that case.
+ */
+struct dept_key __dept_no_validate__;
+static bool valid_key(struct dept_key *k)
+{
+ return &__dept_no_validate__ != k;
+}
+
+/*
+ * Pool
+ * =====================================================================
+ * DEPT maintains pools to provide objects in a safe way.
+ *
+ * 1) Static pool is used at the beginning of booting time.
+ * 2) Local pool is tried first before the static pool. Objects that
+ * have been freed will be placed.
+ */
+
+enum object_t {
+#define OBJECT(id, nr) OBJECT_##id,
+ #include "dept_object.h"
+#undef OBJECT
+ OBJECT_NR,
+};
+
+#define OBJECT(id, nr) \
+static struct dept_##id spool_##id[nr]; \
+static DEFINE_PER_CPU(struct llist_head, lpool_##id);
+ #include "dept_object.h"
+#undef OBJECT
+
+static struct dept_pool dept_pool[OBJECT_NR] = {
+#define OBJECT(id, nr) { \
+ .name = #id, \
+ .obj_sz = sizeof(struct dept_##id), \
+ .obj_nr = ATOMIC_INIT(nr), \
+ .node_off = offsetof(struct dept_##id, pool_node), \
+ .spool = spool_##id, \
+ .lpool = &lpool_##id, },
+ #include "dept_object.h"
+#undef OBJECT
+};
+
+/*
+ * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is
+ * enabled or not because NMI and other contexts in the same CPU never
+ * run inside of DEPT concurrently by preventing reentrance.
+ */
+static void *from_pool(enum object_t t)
+{
+ struct dept_pool *p;
+ struct llist_head *h;
+ struct llist_node *n;
+
+ /*
+ * llist_del_first() doesn't allow concurrent access e.g.
+ * between process and IRQ context.
+ */
+ if (DEPT_WARN_ON(!irqs_disabled()))
+ return NULL;
+
+ p = &dept_pool[t];
+
+ /*
+ * Try local pool first.
+ */
+ if (likely(dept_per_cpu_ready))
+ h = this_cpu_ptr(p->lpool);
+ else
+ h = &p->boot_pool;
+
+ n = llist_del_first(h);
+ if (n)
+ return (void *)n - p->node_off;
+
+ /*
+ * Try static pool.
+ */
+ if (atomic_read(&p->obj_nr) > 0) {
+ int idx = atomic_dec_return(&p->obj_nr);
+
+ if (idx >= 0)
+ return p->spool + (idx * p->obj_sz);
+ }
+
+ DEPT_INFO_ONCE("---------------------------------------------\n"
+ " Some of Dept internal resources are run out.\n"
+ " Dept might still work if the resources get freed.\n"
+ " However, the chances are Dept will suffer from\n"
+ " the lack from now. Needs to extend the internal\n"
+ " resource pools. Ask max.byungchul.park@gmail.com\n");
+ return NULL;
+}
+
+static void to_pool(void *o, enum object_t t)
+{
+ struct dept_pool *p = &dept_pool[t];
+ struct llist_head *h;
+
+ preempt_disable();
+ if (likely(dept_per_cpu_ready))
+ h = this_cpu_ptr(p->lpool);
+ else
+ h = &p->boot_pool;
+
+ llist_add(o + p->node_off, h);
+ preempt_enable();
+}
+
+#define OBJECT(id, nr) \
+static void (*ctor_##id)(struct dept_##id *a); \
+static void (*dtor_##id)(struct dept_##id *a); \
+static struct dept_##id *new_##id(void) \
+{ \
+ struct dept_##id *a; \
+ \
+ a = (struct dept_##id *)from_pool(OBJECT_##id); \
+ if (unlikely(!a)) \
+ return NULL; \
+ \
+ atomic_set(&a->ref, 1); \
+ \
+ if (ctor_##id) \
+ ctor_##id(a); \
+ \
+ return a; \
+} \
+ \
+static struct dept_##id *get_##id(struct dept_##id *a) \
+{ \
+ atomic_inc(&a->ref); \
+ return a; \
+} \
+ \
+static void put_##id(struct dept_##id *a) \
+{ \
+ if (!atomic_dec_return(&a->ref)) { \
+ if (dtor_##id) \
+ dtor_##id(a); \
+ to_pool(a, OBJECT_##id); \
+ } \
+} \
+ \
+static void del_##id(struct dept_##id *a) \
+{ \
+ put_##id(a); \
+} \
+ \
+static bool __maybe_unused id##_consumed(struct dept_##id *a) \
+{ \
+ return a && atomic_read(&a->ref) > 1; \
+}
+#include "dept_object.h"
+#undef OBJECT
+
+#define SET_CONSTRUCTOR(id, f) \
+static void (*ctor_##id)(struct dept_##id *a) = f
+
+static void initialize_dep(struct dept_dep *d)
+{
+ INIT_LIST_HEAD(&d->dep_node);
+ INIT_LIST_HEAD(&d->dep_rev_node);
+}
+SET_CONSTRUCTOR(dep, initialize_dep);
+
+static void initialize_class(struct dept_class *c)
+{
+ int i;
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ struct dept_iecxt *ie = &c->iecxt[i];
+ struct dept_iwait *iw = &c->iwait[i];
+
+ ie->ecxt = NULL;
+ ie->enirq = i;
+ ie->staled = false;
+
+ iw->wait = NULL;
+ iw->irq = i;
+ iw->staled = false;
+ iw->touched = false;
+ }
+ c->bfs_gen = 0U;
+
+ INIT_LIST_HEAD(&c->all_node);
+ INIT_LIST_HEAD(&c->dep_head);
+ INIT_LIST_HEAD(&c->dep_rev_head);
+ INIT_LIST_HEAD(&c->bfs_node);
+}
+SET_CONSTRUCTOR(class, initialize_class);
+
+static void initialize_ecxt(struct dept_ecxt *e)
+{
+ int i;
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ e->enirq_stack[i] = NULL;
+ e->enirq_ip[i] = 0UL;
+ }
+ e->ecxt_ip = 0UL;
+ e->ecxt_stack = NULL;
+ e->enirqf = 0UL;
+ e->event_ip = 0UL;
+ e->event_stack = NULL;
+}
+SET_CONSTRUCTOR(ecxt, initialize_ecxt);
+
+static void initialize_wait(struct dept_wait *w)
+{
+ int i;
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ w->irq_stack[i] = NULL;
+ w->irq_ip[i] = 0UL;
+ }
+ w->wait_ip = 0UL;
+ w->wait_stack = NULL;
+ w->irqf = 0UL;
+}
+SET_CONSTRUCTOR(wait, initialize_wait);
+
+static void initialize_stack(struct dept_stack *s)
+{
+ s->nr = 0;
+}
+SET_CONSTRUCTOR(stack, initialize_stack);
+
+#define OBJECT(id, nr) \
+static void (*ctor_##id)(struct dept_##id *a);
+ #include "dept_object.h"
+#undef OBJECT
+
+#undef SET_CONSTRUCTOR
+
+#define SET_DESTRUCTOR(id, f) \
+static void (*dtor_##id)(struct dept_##id *a) = f
+
+static void destroy_dep(struct dept_dep *d)
+{
+ if (dep_e(d))
+ put_ecxt(dep_e(d));
+ if (dep_w(d))
+ put_wait(dep_w(d));
+}
+SET_DESTRUCTOR(dep, destroy_dep);
+
+static void destroy_ecxt(struct dept_ecxt *e)
+{
+ int i;
+
+ for (i = 0; i < DEPT_IRQS_NR; i++)
+ if (e->enirq_stack[i])
+ put_stack(e->enirq_stack[i]);
+ if (e->class)
+ put_class(e->class);
+ if (e->ecxt_stack)
+ put_stack(e->ecxt_stack);
+ if (e->event_stack)
+ put_stack(e->event_stack);
+}
+SET_DESTRUCTOR(ecxt, destroy_ecxt);
+
+static void destroy_wait(struct dept_wait *w)
+{
+ int i;
+
+ for (i = 0; i < DEPT_IRQS_NR; i++)
+ if (w->irq_stack[i])
+ put_stack(w->irq_stack[i]);
+ if (w->class)
+ put_class(w->class);
+ if (w->wait_stack)
+ put_stack(w->wait_stack);
+}
+SET_DESTRUCTOR(wait, destroy_wait);
+
+#define OBJECT(id, nr) \
+static void (*dtor_##id)(struct dept_##id *a);
+ #include "dept_object.h"
+#undef OBJECT
+
+#undef SET_DESTRUCTOR
+
+/*
+ * Caching and hashing
+ * =====================================================================
+ * DEPT makes use of caching and hashing to improve performance. Each
+ * object can be obtained in O(1) with its key.
+ *
+ * NOTE: Currently we assume all the objects in the hashs will never be
+ * removed. Implement it when needed.
+ */
+
+/*
+ * Some information might be lost but it's only for hashing key.
+ */
+static unsigned long mix(unsigned long a, unsigned long b)
+{
+ int halfbits = sizeof(unsigned long) * 8 / 2;
+ unsigned long halfmask = (1UL << halfbits) - 1UL;
+
+ return (a << halfbits) | (b & halfmask);
+}
+
+static bool cmp_dep(struct dept_dep *d1, struct dept_dep *d2)
+{
+ return dep_fc(d1)->key == dep_fc(d2)->key &&
+ dep_tc(d1)->key == dep_tc(d2)->key;
+}
+
+static unsigned long key_dep(struct dept_dep *d)
+{
+ return mix(dep_fc(d)->key, dep_tc(d)->key);
+}
+
+static bool cmp_class(struct dept_class *c1, struct dept_class *c2)
+{
+ return c1->key == c2->key;
+}
+
+static unsigned long key_class(struct dept_class *c)
+{
+ return c->key;
+}
+
+#define HASH(id, bits) \
+static struct hlist_head table_##id[1 << (bits)]; \
+ \
+static struct hlist_head *head_##id(struct dept_##id *a) \
+{ \
+ return table_##id + hash_long(key_##id(a), bits); \
+} \
+ \
+static struct dept_##id *hash_lookup_##id(struct dept_##id *a) \
+{ \
+ struct dept_##id *b; \
+ \
+ hlist_for_each_entry_rcu(b, head_##id(a), hash_node) \
+ if (cmp_##id(a, b)) \
+ return b; \
+ return NULL; \
+} \
+ \
+static void hash_add_##id(struct dept_##id *a) \
+{ \
+ get_##id(a); \
+ hlist_add_head_rcu(&a->hash_node, head_##id(a)); \
+} \
+ \
+static void hash_del_##id(struct dept_##id *a) \
+{ \
+ hlist_del_rcu(&a->hash_node); \
+ put_##id(a); \
+}
+#include "dept_hash.h"
+#undef HASH
+
+static struct dept_dep *lookup_dep(struct dept_class *fc,
+ struct dept_class *tc)
+{
+ struct dept_ecxt onetime_e = { .class = fc };
+ struct dept_wait onetime_w = { .class = tc };
+ struct dept_dep onetime_d = { .ecxt = &onetime_e,
+ .wait = &onetime_w };
+ return hash_lookup_dep(&onetime_d);
+}
+
+static struct dept_class *lookup_class(unsigned long key)
+{
+ struct dept_class onetime_c = { .key = key };
+
+ return hash_lookup_class(&onetime_c);
+}
+
+/*
+ * Report
+ * =====================================================================
+ * DEPT prints useful information to help debugging on detection of
+ * problematic dependency.
+ */
+
+static void print_ip_stack(unsigned long ip, struct dept_stack *s)
+{
+ if (ip)
+ print_ip_sym(KERN_WARNING, ip);
+
+#ifdef CONFIG_DEPT_DEBUG
+ if (!s)
+ pr_warn("stack is NULL.\n");
+ else if (!s->nr)
+ pr_warn("stack->nr is 0.\n");
+ if (s)
+ pr_warn("stack ref is %d.\n", atomic_read(&s->ref));
+#endif
+
+ if (valid_stack(s)) {
+ pr_warn("stacktrace:\n");
+ stack_trace_print(s->raw, s->nr, 5);
+ }
+
+ if (!ip && !valid_stack(s))
+ pr_warn("(N/A)\n");
+}
+
+#define print_spc(spc, fmt, ...) \
+ pr_warn("%*c" fmt, (spc) * 3, ' ', ##__VA_ARGS__)
+
+static void print_diagram(struct dept_dep *d)
+{
+ struct dept_ecxt *e = dep_e(d);
+ struct dept_wait *w = dep_w(d);
+ struct dept_class *fc = dep_fc(d);
+ struct dept_class *tc = dep_tc(d);
+ unsigned long irqf;
+ int irq;
+ bool firstline = true;
+ int spc = 1;
+ const char *w_fn = w->wait_fn ?: "(unknown)";
+ const char *e_fn = e->event_fn ?: "(unknown)";
+ const char *c_fn = e->ecxt_fn ?: "(unknown)";
+ const char *fc_n = fc->sched_map ? "<sched>" : (fc->name ?: "(unknown)");
+ const char *tc_n = tc->sched_map ? "<sched>" : (tc->name ?: "(unknown)");
+
+ irqf = e->enirqf & w->irqf;
+ for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) {
+ if (!firstline)
+ pr_warn("\nor\n\n");
+ firstline = false;
+
+ print_spc(spc, "[S] %s(%s:%d)\n", c_fn, fc_n, fc->sub_id);
+ print_spc(spc, " <%s interrupt>\n", irq_str(irq));
+ print_spc(spc + 1, "[W] %s(%s:%d)\n", w_fn, tc_n, tc->sub_id);
+ print_spc(spc, "[E] %s(%s:%d)\n", e_fn, fc_n, fc->sub_id);
+ }
+
+ if (!irqf) {
+ print_spc(spc, "[S] %s(%s:%d)\n", c_fn, fc_n, fc->sub_id);
+ print_spc(spc, "[W] %s(%s:%d)\n", w_fn, tc_n, tc->sub_id);
+ print_spc(spc, "[E] %s(%s:%d)\n", e_fn, fc_n, fc->sub_id);
+ }
+}
+
+static void print_dep(struct dept_dep *d)
+{
+ struct dept_ecxt *e = dep_e(d);
+ struct dept_wait *w = dep_w(d);
+ struct dept_class *fc = dep_fc(d);
+ struct dept_class *tc = dep_tc(d);
+ unsigned long irqf;
+ int irq;
+ const char *w_fn = w->wait_fn ?: "(unknown)";
+ const char *e_fn = e->event_fn ?: "(unknown)";
+ const char *c_fn = e->ecxt_fn ?: "(unknown)";
+ const char *fc_n = fc->sched_map ? "<sched>" : (fc->name ?: "(unknown)");
+ const char *tc_n = tc->sched_map ? "<sched>" : (tc->name ?: "(unknown)");
+
+ irqf = e->enirqf & w->irqf;
+ for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) {
+ pr_warn("%s has been enabled:\n", irq_str(irq));
+ print_ip_stack(e->enirq_ip[irq], e->enirq_stack[irq]);
+ pr_warn("\n");
+
+ pr_warn("[S] %s(%s:%d):\n", c_fn, fc_n, fc->sub_id);
+ print_ip_stack(e->ecxt_ip, e->ecxt_stack);
+ pr_warn("\n");
+
+ pr_warn("[W] %s(%s:%d) in %s context:\n",
+ w_fn, tc_n, tc->sub_id, irq_str(irq));
+ print_ip_stack(w->irq_ip[irq], w->irq_stack[irq]);
+ pr_warn("\n");
+
+ pr_warn("[E] %s(%s:%d):\n", e_fn, fc_n, fc->sub_id);
+ print_ip_stack(e->event_ip, e->event_stack);
+ }
+
+ if (!irqf) {
+ pr_warn("[S] %s(%s:%d):\n", c_fn, fc_n, fc->sub_id);
+ print_ip_stack(e->ecxt_ip, e->ecxt_stack);
+ pr_warn("\n");
+
+ pr_warn("[W] %s(%s:%d):\n", w_fn, tc_n, tc->sub_id);
+ print_ip_stack(w->wait_ip, w->wait_stack);
+ pr_warn("\n");
+
+ pr_warn("[E] %s(%s:%d):\n", e_fn, fc_n, fc->sub_id);
+ print_ip_stack(e->event_ip, e->event_stack);
+ }
+}
+
+static void save_current_stack(int skip);
+
+/*
+ * Print all classes in a circle.
+ */
+static void print_circle(struct dept_class *c)
+{
+ struct dept_class *fc = c->bfs_parent;
+ struct dept_class *tc = c;
+ int i;
+
+ dept_outworld_enter();
+ save_current_stack(6);
+
+ pr_warn("===================================================\n");
+ pr_warn("DEPT: Circular dependency has been detected.\n");
+ pr_warn("%s %.*s %s\n", init_utsname()->release,
+ (int)strcspn(init_utsname()->version, " "),
+ init_utsname()->version,
+ print_tainted());
+ pr_warn("---------------------------------------------------\n");
+ pr_warn("summary\n");
+ pr_warn("---------------------------------------------------\n");
+
+ if (fc == tc)
+ pr_warn("*** AA DEADLOCK ***\n\n");
+ else
+ pr_warn("*** DEADLOCK ***\n\n");
+
+ i = 0;
+ do {
+ struct dept_dep *d = lookup_dep(fc, tc);
+
+ pr_warn("context %c\n", 'A' + (i++));
+ print_diagram(d);
+ if (fc != c)
+ pr_warn("\n");
+
+ tc = fc;
+ fc = fc->bfs_parent;
+ } while (tc != c);
+
+ pr_warn("\n");
+ pr_warn("[S]: start of the event context\n");
+ pr_warn("[W]: the wait blocked\n");
+ pr_warn("[E]: the event not reachable\n");
+
+ i = 0;
+ do {
+ struct dept_dep *d = lookup_dep(fc, tc);
+
+ pr_warn("---------------------------------------------------\n");
+ pr_warn("context %c's detail\n", 'A' + i);
+ pr_warn("---------------------------------------------------\n");
+ pr_warn("context %c\n", 'A' + (i++));
+ print_diagram(d);
+ pr_warn("\n");
+ print_dep(d);
+
+ tc = fc;
+ fc = fc->bfs_parent;
+ } while (tc != c);
+
+ pr_warn("---------------------------------------------------\n");
+ pr_warn("information that might be helpful\n");
+ pr_warn("---------------------------------------------------\n");
+ dump_stack();
+
+ dept_outworld_exit();
+}
+
+/*
+ * BFS(Breadth First Search)
+ * =====================================================================
+ * Whenever a new dependency is added into the graph, search the graph
+ * for a new circular dependency.
+ */
+
+struct bfs_ops {
+ void (*bfs_init)(void *, void *, void **);
+ void (*extend)(struct list_head *, void *);
+ void *(*dequeue)(struct list_head *);
+ enum bfs_ret (*callback)(void *, void *, void **);
+};
+
+static unsigned int bfs_gen;
+
+/*
+ * NOTE: Must be called with dept_lock held.
+ */
+static void bfs(void *root, struct bfs_ops *ops, void *in, void **out)
+{
+ LIST_HEAD(q);
+ enum bfs_ret ret;
+
+ if (DEPT_WARN_ON(!ops || !ops->bfs_init || !ops->extend ||
+ !ops->dequeue || !ops->callback))
+ return;
+
+ /*
+ * Avoid zero bfs_gen.
+ */
+ bfs_gen = bfs_gen + 1 ?: 1;
+ ops->bfs_init(root, in, out);
+
+ ret = ops->callback(root, in, out);
+ if (ret != BFS_CONTINUE)
+ return;
+
+ ops->extend(&q, root);
+ while (!list_empty(&q)) {
+ void *node = ops->dequeue(&q);
+
+ if (ret == BFS_DONE)
+ continue;
+
+ ret = ops->callback(node, in, out);
+ if (ret == BFS_CONTINUE)
+ ops->extend(&q, node);
+ }
+}
+
+/*
+ * Main operations
+ * =====================================================================
+ * Add dependencies - Each new dependency is added into the graph and
+ * checked if it forms a circular dependency.
+ *
+ * Track waits - Waits are queued into the ring buffer for later use to
+ * generate appropriate dependencies with cross-event.
+ *
+ * Track event contexts(ecxt) - Event contexts are pushed into local
+ * stack for later use to generate appropriate dependencies with waits.
+ */
+
+static unsigned long cur_enirqf(void);
+static int cur_irq(void);
+static unsigned int cur_ctxt_id(void);
+
+static struct dept_iecxt *iecxt(struct dept_class *c, int irq)
+{
+ return &c->iecxt[irq];
+}
+
+static struct dept_iwait *iwait(struct dept_class *c, int irq)
+{
+ return &c->iwait[irq];
+}
+
+static void stale_iecxt(struct dept_iecxt *ie)
+{
+ if (ie->ecxt)
+ put_ecxt(ie->ecxt);
+
+ WRITE_ONCE(ie->ecxt, NULL);
+ WRITE_ONCE(ie->staled, true);
+}
+
+static void set_iecxt(struct dept_iecxt *ie, struct dept_ecxt *e)
+{
+ /*
+ * ->ecxt will never be updated once getting set until the class
+ * gets removed.
+ */
+ if (ie->ecxt)
+ DEPT_WARN_ON(1);
+ else
+ WRITE_ONCE(ie->ecxt, get_ecxt(e));
+}
+
+static void stale_iwait(struct dept_iwait *iw)
+{
+ if (iw->wait)
+ put_wait(iw->wait);
+
+ WRITE_ONCE(iw->wait, NULL);
+ WRITE_ONCE(iw->staled, true);
+}
+
+static void set_iwait(struct dept_iwait *iw, struct dept_wait *w)
+{
+ /*
+ * ->wait will never be updated once getting set until the class
+ * gets removed.
+ */
+ if (iw->wait)
+ DEPT_WARN_ON(1);
+ else
+ WRITE_ONCE(iw->wait, get_wait(w));
+
+ iw->touched = true;
+}
+
+static void touch_iwait(struct dept_iwait *iw)
+{
+ iw->touched = true;
+}
+
+static void untouch_iwait(struct dept_iwait *iw)
+{
+ iw->touched = false;
+}
+
+static struct dept_stack *get_current_stack(void)
+{
+ struct dept_stack *s = dept_task()->stack;
+
+ return s ? get_stack(s) : NULL;
+}
+
+static void prepare_current_stack(void)
+{
+ DEPT_WARN_ON(dept_task()->stack);
+
+ dept_task()->stack = new_stack();
+}
+
+static void save_current_stack(int skip)
+{
+ struct dept_stack *s = dept_task()->stack;
+
+ if (!s)
+ return;
+
+ if (valid_stack(s))
+ return;
+
+ s->nr = stack_trace_save(s->raw, DEPT_MAX_STACK_ENTRY, skip);
+}
+
+static void finish_current_stack(void)
+{
+ struct dept_stack *s = dept_task()->stack;
+
+ /*
+ * Fill the struct dept_stack with a valid stracktrace if it has
+ * been referred at least once.
+ */
+ if (stack_consumed(s))
+ save_current_stack(2);
+
+ dept_task()->stack = NULL;
+
+ /*
+ * Actual deletion will happen at put_stack() if the stack has
+ * been referred.
+ */
+ if (s)
+ del_stack(s);
+}
+
+/*
+ * FIXME: For now, disable LOCKDEP while DEPT is working.
+ *
+ * Both LOCKDEP and DEPT report it on a deadlock detection using
+ * printk taking the risk of another deadlock that might be caused by
+ * locks of console or printk between inside and outside of them.
+ *
+ * For DEPT, it's no problem since multiple reports are allowed. But it
+ * would be a bad idea for LOCKDEP since it will stop even on a singe
+ * report. So we need to prevent LOCKDEP from its reporting the risk
+ * DEPT would take when reporting something.
+ */
+#include <linux/lockdep.h>
+
+void noinstr dept_off(void)
+{
+ dept_task()->recursive++;
+ lockdep_off();
+}
+
+void noinstr dept_on(void)
+{
+ lockdep_on();
+ dept_task()->recursive--;
+}
+
+static unsigned long dept_enter(void)
+{
+ unsigned long flags;
+
+ flags = arch_local_irq_save();
+ dept_off();
+ prepare_current_stack();
+ return flags;
+}
+
+static void dept_exit(unsigned long flags)
+{
+ finish_current_stack();
+ dept_on();
+ arch_local_irq_restore(flags);
+}
+
+static unsigned long dept_enter_recursive(void)
+{
+ unsigned long flags;
+
+ flags = arch_local_irq_save();
+ return flags;
+}
+
+static void dept_exit_recursive(unsigned long flags)
+{
+ arch_local_irq_restore(flags);
+}
+
+/*
+ * NOTE: Must be called with dept_lock held.
+ */
+static struct dept_dep *__add_dep(struct dept_ecxt *e,
+ struct dept_wait *w)
+{
+ struct dept_dep *d;
+
+ if (DEPT_WARN_ON(!valid_class(e->class)))
+ return NULL;
+
+ if (DEPT_WARN_ON(!valid_class(w->class)))
+ return NULL;
+
+ if (lookup_dep(e->class, w->class))
+ return NULL;
+
+ d = new_dep();
+ if (unlikely(!d))
+ return NULL;
+
+ d->ecxt = get_ecxt(e);
+ d->wait = get_wait(w);
+
+ /*
+ * Add the dependency into hash and graph.
+ */
+ hash_add_dep(d);
+ list_add(&d->dep_node, &dep_fc(d)->dep_head);
+ list_add(&d->dep_rev_node, &dep_tc(d)->dep_rev_head);
+ return d;
+}
+
+static void bfs_init_check_dl(void *node, void *in, void **out)
+{
+ struct dept_class *root = (struct dept_class *)node;
+ struct dept_dep *new = (struct dept_dep *)in;
+
+ root->bfs_gen = bfs_gen;
+ dep_tc(new)->bfs_parent = dep_fc(new);
+}
+
+static void bfs_extend_dep(struct list_head *h, void *node)
+{
+ struct dept_class *cur = (struct dept_class *)node;
+ struct dept_dep *d;
+
+ list_for_each_entry(d, &cur->dep_head, dep_node) {
+ struct dept_class *next = dep_tc(d);
+
+ if (bfs_gen == next->bfs_gen)
+ continue;
+ next->bfs_parent = cur;
+ next->bfs_gen = bfs_gen;
+ list_add_tail(&next->bfs_node, h);
+ }
+}
+
+static void *bfs_dequeue_dep(struct list_head *h)
+{
+ struct dept_class *c;
+
+ DEPT_WARN_ON(list_empty(h));
+
+ c = list_first_entry(h, struct dept_class, bfs_node);
+ list_del(&c->bfs_node);
+ return c;
+}
+
+static enum bfs_ret cb_check_dl(void *node, void *in, void **out)
+{
+ struct dept_class *cur = (struct dept_class *)node;
+ struct dept_dep *new = (struct dept_dep *)in;
+
+ if (cur == dep_fc(new)) {
+ print_circle(dep_tc(new));
+ return BFS_DONE;
+ }
+
+ return BFS_CONTINUE;
+}
+
+/*
+ * This function is actually in charge of reporting.
+ */
+static void check_dl_bfs(struct dept_dep *d)
+{
+ struct bfs_ops ops = {
+ .bfs_init = bfs_init_check_dl,
+ .extend = bfs_extend_dep,
+ .dequeue = bfs_dequeue_dep,
+ .callback = cb_check_dl,
+ };
+
+ bfs((void *)dep_tc(d), &ops, (void *)d, NULL);
+}
+
+static void bfs_init_dep(void *node, void *in, void **out)
+{
+ struct dept_class *root = (struct dept_class *)node;
+
+ root->bfs_gen = bfs_gen;
+}
+
+static void bfs_extend_dep_rev(struct list_head *h, void *node)
+{
+ struct dept_class *cur = (struct dept_class *)node;
+ struct dept_dep *d;
+
+ list_for_each_entry(d, &cur->dep_rev_head, dep_rev_node) {
+ struct dept_class *next = dep_fc(d);
+
+ if (bfs_gen == next->bfs_gen)
+ continue;
+ next->bfs_parent = cur;
+ next->bfs_gen = bfs_gen;
+ list_add_tail(&next->bfs_node, h);
+ }
+}
+
+static enum bfs_ret cb_find_iw(void *node, void *in, void **out)
+{
+ struct dept_class *cur = (struct dept_class *)node;
+ int irq = *(int *)in;
+ struct dept_iwait *iw;
+
+ if (DEPT_WARN_ON(!out))
+ return BFS_DONE;
+
+ iw = iwait(cur, irq);
+
+ /*
+ * If any parent's ->wait was set, then the children would've
+ * been touched.
+ */
+ if (!iw->touched)
+ return BFS_SKIP;
+
+ if (!iw->wait)
+ return BFS_CONTINUE;
+
+ *out = iw;
+ return BFS_DONE;
+}
+
+static struct dept_iwait *find_iw_bfs(struct dept_class *c, int irq)
+{
+ struct dept_iwait *iw = iwait(c, irq);
+ struct dept_iwait *found = NULL;
+ struct bfs_ops ops = {
+ .bfs_init = bfs_init_dep,
+ .extend = bfs_extend_dep_rev,
+ .dequeue = bfs_dequeue_dep,
+ .callback = cb_find_iw,
+ };
+
+ bfs((void *)c, &ops, (void *)&irq, (void **)&found);
+
+ if (found)
+ return found;
+
+ untouch_iwait(iw);
+ return NULL;
+}
+
+static enum bfs_ret cb_touch_iw_find_ie(void *node, void *in, void **out)
+{
+ struct dept_class *cur = (struct dept_class *)node;
+ int irq = *(int *)in;
+ struct dept_iecxt *ie = iecxt(cur, irq);
+ struct dept_iwait *iw = iwait(cur, irq);
+
+ if (DEPT_WARN_ON(!out))
+ return BFS_DONE;
+
+ touch_iwait(iw);
+
+ if (!ie->ecxt)
+ return BFS_CONTINUE;
+ if (!*out)
+ *out = ie;
+
+ /*
+ * Do touch_iwait() all the way.
+ */
+ return BFS_CONTINUE;
+}
+
+static struct dept_iecxt *touch_iw_find_ie_bfs(struct dept_class *c,
+ int irq)
+{
+ struct dept_iecxt *found = NULL;
+ struct bfs_ops ops = {
+ .bfs_init = bfs_init_dep,
+ .extend = bfs_extend_dep,
+ .dequeue = bfs_dequeue_dep,
+ .callback = cb_touch_iw_find_ie,
+ };
+
+ bfs((void *)c, &ops, (void *)&irq, (void **)&found);
+ return found;
+}
+
+/*
+ * Should be called with dept_lock held.
+ */
+static void __add_idep(struct dept_iecxt *ie, struct dept_iwait *iw)
+{
+ struct dept_dep *new;
+
+ /*
+ * There's nothing to do.
+ */
+ if (!ie || !iw || !ie->ecxt || !iw->wait)
+ return;
+
+ new = __add_dep(ie->ecxt, iw->wait);
+
+ /*
+ * Deadlock detected. Let check_dl_bfs() report it.
+ */
+ if (new) {
+ check_dl_bfs(new);
+ stale_iecxt(ie);
+ stale_iwait(iw);
+ }
+
+ /*
+ * If !new, it would be the case of lack of object resource.
+ * Just let it go and get checked by other chances. Retrying is
+ * meaningless in that case.
+ */
+}
+
+static void set_check_iecxt(struct dept_class *c, int irq,
+ struct dept_ecxt *e)
+{
+ struct dept_iecxt *ie = iecxt(c, irq);
+
+ set_iecxt(ie, e);
+ __add_idep(ie, find_iw_bfs(c, irq));
+}
+
+static void set_check_iwait(struct dept_class *c, int irq,
+ struct dept_wait *w)
+{
+ struct dept_iwait *iw = iwait(c, irq);
+
+ set_iwait(iw, w);
+ __add_idep(touch_iw_find_ie_bfs(c, irq), iw);
+}
+
+static void add_iecxt(struct dept_class *c, int irq, struct dept_ecxt *e,
+ bool stack)
+{
+ /*
+ * This access is safe since we ensure e->class has set locally.
+ */
+ struct dept_task *dt = dept_task();
+ struct dept_iecxt *ie = iecxt(c, irq);
+
+ if (DEPT_WARN_ON(!valid_class(c)))
+ return;
+
+ if (unlikely(READ_ONCE(ie->staled)))
+ return;
+
+ /*
+ * Skip add_iecxt() if ie->ecxt has ever been set at least once.
+ * Which means it has a valid ->ecxt or been staled.
+ */
+ if (READ_ONCE(ie->ecxt))
+ return;
+
+ if (unlikely(!dept_lock()))
+ return;
+
+ if (unlikely(ie->staled))
+ goto unlock;
+ if (ie->ecxt)
+ goto unlock;
+
+ e->enirqf |= (1UL << irq);
+
+ /*
+ * Should be NULL since it's the first time that these
+ * enirq_{ip,stack}[irq] have ever set.
+ */
+ DEPT_WARN_ON(e->enirq_ip[irq]);
+ DEPT_WARN_ON(e->enirq_stack[irq]);
+
+ e->enirq_ip[irq] = dt->enirq_ip[irq];
+ e->enirq_stack[irq] = stack ? get_current_stack() : NULL;
+
+ set_check_iecxt(c, irq, e);
+unlock:
+ dept_unlock();
+}
+
+static void add_iwait(struct dept_class *c, int irq, struct dept_wait *w)
+{
+ struct dept_iwait *iw = iwait(c, irq);
+
+ if (DEPT_WARN_ON(!valid_class(c)))
+ return;
+
+ if (unlikely(READ_ONCE(iw->staled)))
+ return;
+
+ /*
+ * Skip add_iwait() if iw->wait has ever been set at least once.
+ * Which means it has a valid ->wait or been staled.
+ */
+ if (READ_ONCE(iw->wait))
+ return;
+
+ if (unlikely(!dept_lock()))
+ return;
+
+ if (unlikely(iw->staled))
+ goto unlock;
+ if (iw->wait)
+ goto unlock;
+
+ w->irqf |= (1UL << irq);
+
+ /*
+ * Should be NULL since it's the first time that these
+ * irq_{ip,stack}[irq] have ever set.
+ */
+ DEPT_WARN_ON(w->irq_ip[irq]);
+ DEPT_WARN_ON(w->irq_stack[irq]);
+
+ w->irq_ip[irq] = w->wait_ip;
+ w->irq_stack[irq] = get_current_stack();
+
+ set_check_iwait(c, irq, w);
+unlock:
+ dept_unlock();
+}
+
+static struct dept_wait_hist *hist(int pos)
+{
+ struct dept_task *dt = dept_task();
+
+ return dt->wait_hist + (pos % DEPT_MAX_WAIT_HIST);
+}
+
+static int hist_pos_next(void)
+{
+ struct dept_task *dt = dept_task();
+
+ return dt->wait_hist_pos % DEPT_MAX_WAIT_HIST;
+}
+
+static void hist_advance(void)
+{
+ struct dept_task *dt = dept_task();
+
+ dt->wait_hist_pos++;
+ dt->wait_hist_pos %= DEPT_MAX_WAIT_HIST;
+}
+
+static struct dept_wait_hist *new_hist(void)
+{
+ struct dept_wait_hist *wh = hist(hist_pos_next());
+
+ hist_advance();
+ return wh;
+}
+
+static void add_hist(struct dept_wait *w, unsigned int wg, unsigned int ctxt_id)
+{
+ struct dept_wait_hist *wh = new_hist();
+
+ if (likely(wh->wait))
+ put_wait(wh->wait);
+
+ wh->wait = get_wait(w);
+ wh->wgen = wg;
+ wh->ctxt_id = ctxt_id;
+}
+
+/*
+ * Should be called after setting up e's iecxt and w's iwait.
+ */
+static void add_dep(struct dept_ecxt *e, struct dept_wait *w)
+{
+ struct dept_class *fc = e->class;
+ struct dept_class *tc = w->class;
+ struct dept_dep *d;
+ int i;
+
+ if (lookup_dep(fc, tc))
+ return;
+
+ if (unlikely(!dept_lock()))
+ return;
+
+ /*
+ * __add_dep() will lookup_dep() again with lock held.
+ */
+ d = __add_dep(e, w);
+ if (d) {
+ check_dl_bfs(d);
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ struct dept_iwait *fiw = iwait(fc, i);
+ struct dept_iecxt *found_ie;
+ struct dept_iwait *found_iw;
+
+ /*
+ * '->touched == false' guarantees there's no
+ * parent that has been set ->wait.
+ */
+ if (!fiw->touched)
+ continue;
+
+ /*
+ * find_iw_bfs() will untouch the iwait if
+ * not found.
+ */
+ found_iw = find_iw_bfs(fc, i);
+
+ if (!found_iw)
+ continue;
+
+ found_ie = touch_iw_find_ie_bfs(tc, i);
+ __add_idep(found_ie, found_iw);
+ }
+ }
+ dept_unlock();
+}
+
+static atomic_t wgen = ATOMIC_INIT(1);
+
+static int next_wgen(void)
+{
+ /*
+ * Avoid zero wgen.
+ */
+ return atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen);
+}
+
+static void add_wait(struct dept_class *c, unsigned long ip,
+ const char *w_fn, int sub_l, bool sched_sleep)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_wait *w;
+ unsigned int wg;
+ int irq;
+ int i;
+
+ if (DEPT_WARN_ON(!valid_class(c)))
+ return;
+
+ w = new_wait();
+ if (unlikely(!w))
+ return;
+
+ WRITE_ONCE(w->class, get_class(c));
+ w->wait_ip = ip;
+ w->wait_fn = w_fn;
+ w->wait_stack = get_current_stack();
+ w->sched_sleep = sched_sleep;
+
+ irq = cur_irq();
+ if (irq < DEPT_IRQS_NR)
+ add_iwait(c, irq, w);
+
+ /*
+ * Avoid adding dependency between user aware nested ecxt and
+ * wait.
+ */
+ for (i = dt->ecxt_held_pos - 1; i >= 0; i--) {
+ struct dept_ecxt_held *eh;
+
+ eh = dt->ecxt_held + i;
+
+ /*
+ * the case of invalid key'ed one
+ */
+ if (!eh->ecxt)
+ continue;
+
+ if (eh->ecxt->class != c || eh->sub_l == sub_l)
+ add_dep(eh->ecxt, w);
+ }
+
+ wg = next_wgen();
+ add_hist(w, wg, cur_ctxt_id());
+
+ del_wait(w);
+}
+
+static struct dept_ecxt_held *add_ecxt(struct dept_map *m,
+ struct dept_class *c, unsigned long ip, const char *c_fn,
+ const char *e_fn, int sub_l)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_ecxt_held *eh;
+ struct dept_ecxt *e;
+ unsigned long irqf;
+ unsigned int wg;
+ int irq;
+
+ if (DEPT_WARN_ON(!valid_class(c)))
+ return NULL;
+
+ if (DEPT_WARN_ON_ONCE(dt->ecxt_held_pos >= DEPT_MAX_ECXT_HELD))
+ return NULL;
+
+ wg = next_wgen();
+ if (m->nocheck) {
+ eh = dt->ecxt_held + (dt->ecxt_held_pos++);
+ eh->ecxt = NULL;
+ eh->map = m;
+ eh->class = get_class(c);
+ eh->wgen = wg;
+ eh->sub_l = sub_l;
+
+ return eh;
+ }
+
+ e = new_ecxt();
+ if (unlikely(!e))
+ return NULL;
+
+ e->class = get_class(c);
+ e->ecxt_ip = ip;
+ e->ecxt_stack = ip ? get_current_stack() : NULL;
+ e->event_fn = e_fn;
+ e->ecxt_fn = c_fn;
+
+ eh = dt->ecxt_held + (dt->ecxt_held_pos++);
+ eh->ecxt = get_ecxt(e);
+ eh->map = m;
+ eh->class = get_class(c);
+ eh->wgen = wg;
+ eh->sub_l = sub_l;
+
+ irqf = cur_enirqf();
+ for_each_set_bit(irq, &irqf, DEPT_IRQS_NR)
+ add_iecxt(c, irq, e, false);
+
+ del_ecxt(e);
+ return eh;
+}
+
+static int find_ecxt_pos(struct dept_map *m, struct dept_class *c,
+ bool newfirst)
+{
+ struct dept_task *dt = dept_task();
+ int i;
+
+ if (newfirst) {
+ for (i = dt->ecxt_held_pos - 1; i >= 0; i--) {
+ struct dept_ecxt_held *eh;
+
+ eh = dt->ecxt_held + i;
+ if (eh->map == m && eh->class == c)
+ return i;
+ }
+ } else {
+ for (i = 0; i < dt->ecxt_held_pos; i++) {
+ struct dept_ecxt_held *eh;
+
+ eh = dt->ecxt_held + i;
+ if (eh->map == m && eh->class == c)
+ return i;
+ }
+ }
+ return -1;
+}
+
+static bool pop_ecxt(struct dept_map *m, struct dept_class *c)
+{
+ struct dept_task *dt = dept_task();
+ int pos;
+ int i;
+
+ pos = find_ecxt_pos(m, c, true);
+ if (pos == -1)
+ return false;
+
+ if (dt->ecxt_held[pos].class)
+ put_class(dt->ecxt_held[pos].class);
+
+ if (dt->ecxt_held[pos].ecxt)
+ put_ecxt(dt->ecxt_held[pos].ecxt);
+
+ dt->ecxt_held_pos--;
+
+ for (i = pos; i < dt->ecxt_held_pos; i++)
+ dt->ecxt_held[i] = dt->ecxt_held[i + 1];
+ return true;
+}
+
+static bool good_hist(struct dept_wait_hist *wh, unsigned int wg)
+{
+ return wh->wait != NULL && before(wg, wh->wgen);
+}
+
+/*
+ * Binary-search the ring buffer for the earliest valid wait.
+ */
+static int find_hist_pos(unsigned int wg)
+{
+ int oldest;
+ int l;
+ int r;
+ int pos;
+
+ oldest = hist_pos_next();
+ if (unlikely(good_hist(hist(oldest), wg))) {
+ DEPT_INFO_ONCE("Need to expand the ring buffer.\n");
+ return oldest;
+ }
+
+ l = oldest + 1;
+ r = oldest + DEPT_MAX_WAIT_HIST - 1;
+ for (pos = (l + r) / 2; l <= r; pos = (l + r) / 2) {
+ struct dept_wait_hist *p = hist(pos - 1);
+ struct dept_wait_hist *wh = hist(pos);
+
+ if (!good_hist(p, wg) && good_hist(wh, wg))
+ return pos % DEPT_MAX_WAIT_HIST;
+ if (good_hist(wh, wg))
+ r = pos - 1;
+ else
+ l = pos + 1;
+ }
+ return -1;
+}
+
+static void do_event(struct dept_map *m, struct dept_map *real_m,
+ struct dept_class *c, unsigned int wg, unsigned long ip,
+ const char *e_fn)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_wait_hist *wh;
+ struct dept_ecxt_held *eh;
+ unsigned int ctxt_id;
+ int end;
+ int pos;
+ int i;
+
+ if (DEPT_WARN_ON(!valid_class(c)))
+ return;
+
+ if (m->nocheck)
+ return;
+
+ /*
+ * The event was triggered before wait.
+ */
+ if (!wg)
+ return;
+
+ /*
+ * If an ecxt for this map exists, let the ecxt work for this
+ * event and do not proceed it in do_event().
+ */
+ if (find_ecxt_pos(real_m, c, false) != -1)
+ return;
+ eh = add_ecxt(m, c, 0UL, NULL, e_fn, 0);
+
+ if (!eh)
+ return;
+
+ if (DEPT_WARN_ON(!eh->ecxt))
+ goto out;
+
+ eh->ecxt->event_ip = ip;
+ eh->ecxt->event_stack = get_current_stack();
+
+ pos = find_hist_pos(wg);
+ if (pos == -1)
+ goto out;
+
+ ctxt_id = cur_ctxt_id();
+ end = hist_pos_next();
+ end = end > pos ? end : end + DEPT_MAX_WAIT_HIST;
+ for (wh = hist(pos); pos < end; wh = hist(++pos)) {
+ if (dt->in_sched && wh->wait->sched_sleep)
+ continue;
+
+ if (wh->ctxt_id == ctxt_id)
+ add_dep(eh->ecxt, wh->wait);
+ }
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ struct dept_ecxt *e;
+
+ if (before(dt->wgen_enirq[i], wg))
+ continue;
+
+ e = eh->ecxt;
+ add_iecxt(e->class, i, e, false);
+ }
+out:
+ /*
+ * Pop ecxt that temporarily has been added to handle this event.
+ */
+ pop_ecxt(m, c);
+}
+
+static void del_dep_rcu(struct rcu_head *rh)
+{
+ struct dept_dep *d = container_of(rh, struct dept_dep, rh);
+
+ preempt_disable();
+ del_dep(d);
+ preempt_enable();
+}
+
+/*
+ * NOTE: Must be called with dept_lock held.
+ */
+static void disconnect_class(struct dept_class *c)
+{
+ struct dept_dep *d, *n;
+ int i;
+
+ list_for_each_entry_safe(d, n, &c->dep_head, dep_node) {
+ list_del_rcu(&d->dep_node);
+ list_del_rcu(&d->dep_rev_node);
+ hash_del_dep(d);
+ call_rcu(&d->rh, del_dep_rcu);
+ }
+
+ list_for_each_entry_safe(d, n, &c->dep_rev_head, dep_rev_node) {
+ list_del_rcu(&d->dep_node);
+ list_del_rcu(&d->dep_rev_node);
+ hash_del_dep(d);
+ call_rcu(&d->rh, del_dep_rcu);
+ }
+
+ for (i = 0; i < DEPT_IRQS_NR; i++) {
+ stale_iecxt(iecxt(c, i));
+ stale_iwait(iwait(c, i));
+ }
+}
+
+/*
+ * Context control
+ * =====================================================================
+ * Whether a wait is in {hard,soft}-IRQ context or whether
+ * {hard,soft}-IRQ has been enabled on the way to an event is very
+ * important to check dependency. All those things should be tracked.
+ */
+
+static unsigned long cur_enirqf(void)
+{
+ struct dept_task *dt = dept_task();
+ int he = dt->hardirqs_enabled;
+ int se = dt->softirqs_enabled;
+
+ if (he)
+ return DEPT_HIRQF | (se ? DEPT_SIRQF : 0UL);
+ return 0UL;
+}
+
+static int cur_irq(void)
+{
+ if (lockdep_softirq_context(current))
+ return DEPT_SIRQ;
+ if (lockdep_hardirq_context())
+ return DEPT_HIRQ;
+ return DEPT_IRQS_NR;
+}
+
+static unsigned int cur_ctxt_id(void)
+{
+ struct dept_task *dt = dept_task();
+ int irq = cur_irq();
+
+ /*
+ * Normal process context
+ */
+ if (irq == DEPT_IRQS_NR)
+ return 0U;
+
+ return dt->irq_id[irq] | (1UL << irq);
+}
+
+static void enirq_transition(int irq)
+{
+ struct dept_task *dt = dept_task();
+ int i;
+
+ /*
+ * IRQ can cut in on the way to the event. Used for cross-event
+ * detection.
+ *
+ * wait context event context(ecxt)
+ * ------------ -------------------
+ * wait event
+ * UPDATE wgen
+ * observe IRQ enabled
+ * UPDATE wgen
+ * keep the wgen locally
+ *
+ * on the event
+ * check the wgen kept
+ */
+
+ dt->wgen_enirq[irq] = next_wgen();
+
+ for (i = dt->ecxt_held_pos - 1; i >= 0; i--) {
+ struct dept_ecxt_held *eh;
+ struct dept_ecxt *e;
+
+ eh = dt->ecxt_held + i;
+ e = eh->ecxt;
+ if (e)
+ add_iecxt(e->class, irq, e, true);
+ }
+}
+
+static void dept_enirq(unsigned long ip)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long irqf = cur_enirqf();
+ int irq;
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ /*
+ * IRQ ON/OFF transition might happen while Dept is working.
+ * We cannot handle recursive entrance. Just ignore it.
+ * Only transitions outside of Dept will be considered.
+ */
+ if (dt->recursive)
+ return;
+
+ flags = dept_enter();
+
+ for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) {
+ dt->enirq_ip[irq] = ip;
+ enirq_transition(irq);
+ }
+
+ dept_exit(flags);
+}
+
+void dept_softirqs_on_ip(unsigned long ip)
+{
+ /*
+ * Assumes that it's called with IRQ disabled so that accessing
+ * current's fields is not racy.
+ */
+ dept_task()->softirqs_enabled = true;
+ dept_enirq(ip);
+}
+
+void dept_hardirqs_on(void)
+{
+ /*
+ * Assumes that it's called with IRQ disabled so that accessing
+ * current's fields is not racy.
+ */
+ dept_task()->hardirqs_enabled = true;
+ dept_enirq(_RET_IP_);
+}
+
+void dept_softirqs_off(void)
+{
+ /*
+ * Assumes that it's called with IRQ disabled so that accessing
+ * current's fields is not racy.
+ */
+ dept_task()->softirqs_enabled = false;
+}
+
+void dept_hardirqs_off(void)
+{
+ /*
+ * Assumes that it's called with IRQ disabled so that accessing
+ * current's fields is not racy.
+ */
+ dept_task()->hardirqs_enabled = false;
+}
+
+/*
+ * Ensure it's the outmost softirq context.
+ */
+void dept_softirq_enter(void)
+{
+ struct dept_task *dt = dept_task();
+
+ dt->irq_id[DEPT_SIRQ] += 1UL << DEPT_IRQS_NR;
+}
+
+/*
+ * Ensure it's the outmost hardirq context.
+ */
+void dept_hardirq_enter(void)
+{
+ struct dept_task *dt = dept_task();
+
+ dt->irq_id[DEPT_HIRQ] += 1UL << DEPT_IRQS_NR;
+}
+
+void dept_sched_enter(void)
+{
+ dept_task()->in_sched = true;
+}
+
+void dept_sched_exit(void)
+{
+ dept_task()->in_sched = false;
+}
+
+/*
+ * Exposed APIs
+ * =====================================================================
+ */
+
+static void clean_classes_cache(struct dept_key *k)
+{
+ int i;
+
+ for (i = 0; i < DEPT_MAX_SUBCLASSES_CACHE; i++) {
+ if (!READ_ONCE(k->classes[i]))
+ continue;
+
+ WRITE_ONCE(k->classes[i], NULL);
+ }
+}
+
+/*
+ * Assume we don't have to consider race with the map when
+ * dept_map_init() is called.
+ */
+void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u,
+ const char *n)
+{
+ unsigned long flags;
+
+ if (unlikely(!dept_working())) {
+ m->nocheck = true;
+ return;
+ }
+
+ if (DEPT_WARN_ON(sub_u < 0)) {
+ m->nocheck = true;
+ return;
+ }
+
+ if (DEPT_WARN_ON(sub_u >= DEPT_MAX_SUBCLASSES_USR)) {
+ m->nocheck = true;
+ return;
+ }
+
+ /*
+ * Allow recursive entrance.
+ */
+ flags = dept_enter_recursive();
+
+ clean_classes_cache(&m->map_key);
+
+ m->keys = k;
+ m->sub_u = sub_u;
+ m->name = n;
+ m->wgen = 0U;
+ m->nocheck = !valid_key(k);
+
+ dept_exit_recursive(flags);
+}
+EXPORT_SYMBOL_GPL(dept_map_init);
+
+/*
+ * Assume we don't have to consider race with the map when
+ * dept_map_reinit() is called.
+ */
+void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u,
+ const char *n)
+{
+ unsigned long flags;
+
+ if (unlikely(!dept_working())) {
+ m->nocheck = true;
+ return;
+ }
+
+ /*
+ * Allow recursive entrance.
+ */
+ flags = dept_enter_recursive();
+
+ if (k) {
+ clean_classes_cache(&m->map_key);
+ m->keys = k;
+ m->nocheck = !valid_key(k);
+ }
+
+ if (sub_u >= 0 && sub_u < DEPT_MAX_SUBCLASSES_USR)
+ m->sub_u = sub_u;
+
+ if (n)
+ m->name = n;
+
+ m->wgen = 0U;
+
+ dept_exit_recursive(flags);
+}
+EXPORT_SYMBOL_GPL(dept_map_reinit);
+
+void dept_map_copy(struct dept_map *to, struct dept_map *from)
+{
+ if (unlikely(!dept_working())) {
+ to->nocheck = true;
+ return;
+ }
+
+ *to = *from;
+
+ /*
+ * XXX: 'to' might be in a stack or something. Using the address
+ * in a stack segment as a key is meaningless. Just ignore the
+ * case for now.
+ */
+ if (!to->keys) {
+ to->nocheck = true;
+ return;
+ }
+
+ /*
+ * Since the class cache can be modified concurrently we could
+ * observe half pointers (64bit arch using 32bit copy
+ * instructions). Therefore clear the caches and take the
+ * performance hit.
+ */
+ clean_classes_cache(&to->map_key);
+}
+
+static LIST_HEAD(dept_classes);
+
+static bool within(const void *addr, void *start, unsigned long size)
+{
+ return addr >= start && addr < start + size;
+}
+
+void dept_free_range(void *start, unsigned int sz)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_class *c, *n;
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive) {
+ DEPT_STOP("Failed to successfully free Dept objects.\n");
+ return;
+ }
+
+ flags = dept_enter();
+
+ /*
+ * dept_free_range() should not fail.
+ *
+ * FIXME: Should be fixed if dept_free_range() causes deadlock
+ * with dept_lock().
+ */
+ while (unlikely(!dept_lock()))
+ cpu_relax();
+
+ list_for_each_entry_safe(c, n, &dept_classes, all_node) {
+ if (!within((void *)c->key, start, sz) &&
+ !within(c->name, start, sz))
+ continue;
+
+ hash_del_class(c);
+ disconnect_class(c);
+ list_del(&c->all_node);
+ invalidate_class(c);
+
+ /*
+ * Actual deletion will happen on the rcu callback
+ * that has been added in disconnect_class().
+ */
+ del_class(c);
+ }
+ dept_unlock();
+ dept_exit(flags);
+
+ /*
+ * Wait until even lockless hash_lookup_class() for the class
+ * returns NULL.
+ */
+ might_sleep();
+ synchronize_rcu();
+}
+
+static int sub_id(struct dept_map *m, int e)
+{
+ return (m ? m->sub_u : 0) + e * DEPT_MAX_SUBCLASSES_USR;
+}
+
+static struct dept_class *check_new_class(struct dept_key *local,
+ struct dept_key *k, int sub_id,
+ const char *n, bool sched_map)
+{
+ struct dept_class *c = NULL;
+
+ if (DEPT_WARN_ON(sub_id >= DEPT_MAX_SUBCLASSES))
+ return NULL;
+
+ if (DEPT_WARN_ON(!k))
+ return NULL;
+
+ /*
+ * XXX: Assume that users prevent the map from using if any of
+ * the cached keys has been invalidated. If not, the cache,
+ * local->classes should not be used because it would be racy
+ * with class deletion.
+ */
+ if (local && sub_id < DEPT_MAX_SUBCLASSES_CACHE)
+ c = READ_ONCE(local->classes[sub_id]);
+
+ if (c)
+ return c;
+
+ c = lookup_class((unsigned long)k->base + sub_id);
+ if (c)
+ goto caching;
+
+ if (unlikely(!dept_lock()))
+ return NULL;
+
+ c = lookup_class((unsigned long)k->base + sub_id);
+ if (unlikely(c))
+ goto unlock;
+
+ c = new_class();
+ if (unlikely(!c))
+ goto unlock;
+
+ c->name = n;
+ c->sched_map = sched_map;
+ c->sub_id = sub_id;
+ c->key = (unsigned long)(k->base + sub_id);
+ hash_add_class(c);
+ list_add(&c->all_node, &dept_classes);
+unlock:
+ dept_unlock();
+caching:
+ if (local && sub_id < DEPT_MAX_SUBCLASSES_CACHE)
+ WRITE_ONCE(local->classes[sub_id], c);
+
+ return c;
+}
+
+/*
+ * Called between dept_enter() and dept_exit().
+ */
+static void __dept_wait(struct dept_map *m, unsigned long w_f,
+ unsigned long ip, const char *w_fn, int sub_l,
+ bool sched_sleep, bool sched_map)
+{
+ int e;
+
+ /*
+ * Be as conservative as possible. In case of multiple waits for
+ * a single dept_map, we are going to keep only the last wait's
+ * wgen for simplicity - keeping all wgens seems overengineering.
+ *
+ * Of course, it might cause missing some dependencies that
+ * would rarely, probably never, happen but it helps avoid
+ * false positive reports.
+ */
+ for_each_set_bit(e, &w_f, DEPT_MAX_SUBCLASSES_EVT) {
+ struct dept_class *c;
+ struct dept_key *k;
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k,
+ sub_id(m, e), m->name, sched_map);
+ if (!c)
+ continue;
+
+ add_wait(c, ip, w_fn, sub_l, sched_sleep);
+ }
+}
+
+/*
+ * Called between dept_enter() and dept_exit().
+ */
+static void __dept_event(struct dept_map *m, struct dept_map *real_m,
+ unsigned long e_f, unsigned long ip, const char *e_fn,
+ bool sched_map)
+{
+ struct dept_class *c;
+ struct dept_key *k;
+ int e;
+
+ e = find_first_bit(&e_f, DEPT_MAX_SUBCLASSES_EVT);
+
+ if (DEPT_WARN_ON(e >= DEPT_MAX_SUBCLASSES_EVT))
+ return;
+
+ /*
+ * An event is an event. If the caller passed more than single
+ * event, then warn it and handle the event corresponding to
+ * the first bit anyway.
+ */
+ DEPT_WARN_ON(1UL << e != e_f);
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k, sub_id(m, e), m->name, sched_map);
+
+ if (c)
+ do_event(m, real_m, c, READ_ONCE(m->wgen), ip, e_fn);
+}
+
+void dept_wait(struct dept_map *m, unsigned long w_f,
+ unsigned long ip, const char *w_fn, int sub_l)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive)
+ return;
+
+ if (m->nocheck)
+ return;
+
+ flags = dept_enter();
+
+ __dept_wait(m, w_f, ip, w_fn, sub_l, false, false);
+
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_wait);
+
+void dept_stage_wait(struct dept_map *m, struct dept_key *k,
+ unsigned long ip, const char *w_fn)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (m && m->nocheck)
+ return;
+
+ /*
+ * Either m or k should be passed. Which means Dept relies on
+ * either its own map or the caller's position in the code when
+ * determining its class.
+ */
+ if (DEPT_WARN_ON(!m && !k))
+ return;
+
+ /*
+ * Allow recursive entrance.
+ */
+ flags = dept_enter_recursive();
+
+ /*
+ * Ensure the outmost dept_stage_wait() works.
+ */
+ if (dt->stage_m.keys)
+ goto exit;
+
+ arch_spin_lock(&dt->stage_lock);
+ if (m) {
+ dt->stage_m = *m;
+ dt->stage_real_m = m;
+
+ /*
+ * Ensure dt->stage_m.keys != NULL and it works with the
+ * map's map_key, not stage_m's one when ->keys == NULL.
+ */
+ if (!m->keys)
+ dt->stage_m.keys = &m->map_key;
+ } else {
+ dt->stage_m.name = w_fn;
+ dt->stage_sched_map = true;
+ dt->stage_real_m = &dt->stage_m;
+ }
+
+ /*
+ * dept_map_reinit() includes WRITE_ONCE(->wgen, 0U) that
+ * effectively disables the map just in case real sleep won't
+ * happen. dept_request_event_wait_commit() will enable it.
+ */
+ dept_map_reinit(&dt->stage_m, k, -1, NULL);
+
+ dt->stage_w_fn = w_fn;
+ dt->stage_ip = ip;
+ arch_spin_unlock(&dt->stage_lock);
+exit:
+ dept_exit_recursive(flags);
+}
+EXPORT_SYMBOL_GPL(dept_stage_wait);
+
+static void __dept_clean_stage(struct dept_task *dt)
+{
+ memset(&dt->stage_m, 0x0, sizeof(struct dept_map));
+ dt->stage_real_m = NULL;
+ dt->stage_sched_map = false;
+ dt->stage_w_fn = NULL;
+ dt->stage_ip = 0UL;
+}
+
+void dept_clean_stage(void)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ /*
+ * Allow recursive entrance.
+ */
+ flags = dept_enter_recursive();
+ arch_spin_lock(&dt->stage_lock);
+ __dept_clean_stage(dt);
+ arch_spin_unlock(&dt->stage_lock);
+ dept_exit_recursive(flags);
+}
+EXPORT_SYMBOL_GPL(dept_clean_stage);
+
+/*
+ * Always called from __schedule().
+ */
+void dept_request_event_wait_commit(void)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ unsigned int wg;
+ unsigned long ip;
+ const char *w_fn;
+ bool sched_map;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ /*
+ * It's impossible that __schedule() is called while Dept is
+ * working that already disabled IRQ at the entrance.
+ */
+ if (DEPT_WARN_ON(dt->recursive))
+ return;
+
+ flags = dept_enter();
+
+ arch_spin_lock(&dt->stage_lock);
+
+ /*
+ * Checks if current has staged a wait.
+ */
+ if (!dt->stage_m.keys) {
+ arch_spin_unlock(&dt->stage_lock);
+ goto exit;
+ }
+
+ w_fn = dt->stage_w_fn;
+ ip = dt->stage_ip;
+ sched_map = dt->stage_sched_map;
+
+ wg = next_wgen();
+ WRITE_ONCE(dt->stage_m.wgen, wg);
+ arch_spin_unlock(&dt->stage_lock);
+
+ __dept_wait(&dt->stage_m, 1UL, ip, w_fn, 0, true, sched_map);
+exit:
+ dept_exit(flags);
+}
+
+/*
+ * Always called from try_to_wake_up().
+ */
+void dept_ttwu_stage_wait(struct task_struct *requestor, unsigned long ip)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_task *dt_req = &requestor->dept_task;
+ unsigned long flags;
+ struct dept_map m;
+ struct dept_map *real_m;
+ bool sched_map;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive)
+ return;
+
+ flags = dept_enter();
+
+ arch_spin_lock(&dt_req->stage_lock);
+
+ /*
+ * Serializing is unnecessary as long as it always comes from
+ * try_to_wake_up().
+ */
+ m = dt_req->stage_m;
+ sched_map = dt_req->stage_sched_map;
+ real_m = dt_req->stage_real_m;
+ __dept_clean_stage(dt_req);
+ arch_spin_unlock(&dt_req->stage_lock);
+
+ /*
+ * ->stage_m.keys should not be NULL if it's in use. Should
+ * make sure that it's not NULL when staging a valid map.
+ */
+ if (!m.keys)
+ goto exit;
+
+ __dept_event(&m, real_m, 1UL, ip, "try_to_wake_up", sched_map);
+exit:
+ dept_exit(flags);
+}
+
+/*
+ * Modifies the latest ecxt corresponding to m and e_f.
+ */
+void dept_map_ecxt_modify(struct dept_map *m, unsigned long e_f,
+ struct dept_key *new_k, unsigned long new_e_f,
+ unsigned long new_ip, const char *new_c_fn,
+ const char *new_e_fn, int new_sub_l)
+{
+ struct dept_task *dt = dept_task();
+ struct dept_ecxt_held *eh;
+ struct dept_class *c;
+ struct dept_key *k;
+ unsigned long flags;
+ int pos = -1;
+ int new_e;
+ int e;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ /*
+ * XXX: Couldn't handle re-enterance cases. Ignore it for now.
+ */
+ if (dt->recursive)
+ return;
+
+ /*
+ * Should go ahead no matter whether ->nocheck == true or not
+ * because ->nocheck value can be changed within the ecxt area
+ * delimitated by dept_ecxt_enter() and dept_ecxt_exit().
+ */
+
+ flags = dept_enter();
+
+ for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) {
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k,
+ sub_id(m, e), m->name, false);
+ if (!c)
+ continue;
+
+ /*
+ * When it found an ecxt for any event in e_f, done.
+ */
+ pos = find_ecxt_pos(m, c, true);
+ if (pos != -1)
+ break;
+ }
+
+ if (unlikely(pos == -1))
+ goto exit;
+
+ eh = dt->ecxt_held + pos;
+ new_sub_l = new_sub_l >= 0 ? new_sub_l : eh->sub_l;
+
+ new_e = find_first_bit(&new_e_f, DEPT_MAX_SUBCLASSES_EVT);
+
+ if (new_e < DEPT_MAX_SUBCLASSES_EVT)
+ /*
+ * Let it work with the first bit anyway.
+ */
+ DEPT_WARN_ON(1UL << new_e != new_e_f);
+ else
+ new_e = e;
+
+ pop_ecxt(m, c);
+
+ /*
+ * Apply the key to the map.
+ */
+ if (new_k)
+ dept_map_reinit(m, new_k, -1, NULL);
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k, sub_id(m, new_e), m->name, false);
+
+ if (c && add_ecxt(m, c, new_ip, new_c_fn, new_e_fn, new_sub_l))
+ goto exit;
+
+ /*
+ * Successfully pop_ecxt()ed but failed to add_ecxt().
+ */
+ dt->missing_ecxt++;
+exit:
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_map_ecxt_modify);
+
+void dept_ecxt_enter(struct dept_map *m, unsigned long e_f, unsigned long ip,
+ const char *c_fn, const char *e_fn, int sub_l)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ struct dept_class *c;
+ struct dept_key *k;
+ int e;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive) {
+ dt->missing_ecxt++;
+ return;
+ }
+
+ /*
+ * Should go ahead no matter whether ->nocheck == true or not
+ * because ->nocheck value can be changed within the ecxt area
+ * delimitated by dept_ecxt_enter() and dept_ecxt_exit().
+ */
+
+ flags = dept_enter();
+
+ e = find_first_bit(&e_f, DEPT_MAX_SUBCLASSES_EVT);
+
+ if (e >= DEPT_MAX_SUBCLASSES_EVT)
+ goto missing_ecxt;
+
+ /*
+ * An event is an event. If the caller passed more than single
+ * event, then warn it and handle the event corresponding to
+ * the first bit anyway.
+ */
+ DEPT_WARN_ON(1UL << e != e_f);
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k, sub_id(m, e), m->name, false);
+
+ if (c && add_ecxt(m, c, ip, c_fn, e_fn, sub_l))
+ goto exit;
+missing_ecxt:
+ dt->missing_ecxt++;
+exit:
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_ecxt_enter);
+
+bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ bool ret = false;
+ int e;
+
+ if (unlikely(!dept_working()))
+ return false;
+
+ if (dt->recursive)
+ return false;
+
+ flags = dept_enter();
+
+ for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) {
+ struct dept_class *c;
+ struct dept_key *k;
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k,
+ sub_id(m, e), m->name, false);
+ if (!c)
+ continue;
+
+ if (find_ecxt_pos(m, c, true) != -1) {
+ ret = true;
+ break;
+ }
+ }
+
+ dept_exit(flags);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(dept_ecxt_holding);
+
+void dept_request_event(struct dept_map *m)
+{
+ unsigned long flags;
+ unsigned int wg;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (m->nocheck)
+ return;
+
+ /*
+ * Allow recursive entrance.
+ */
+ flags = dept_enter_recursive();
+
+ wg = next_wgen();
+ WRITE_ONCE(m->wgen, wg);
+
+ dept_exit_recursive(flags);
+}
+EXPORT_SYMBOL_GPL(dept_request_event);
+
+void dept_event(struct dept_map *m, unsigned long e_f,
+ unsigned long ip, const char *e_fn)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (m->nocheck)
+ return;
+
+ if (dt->recursive) {
+ /*
+ * Dept won't work with this even though an event
+ * context has been asked. Don't make it confused at
+ * handling the event. Disable it until the next.
+ */
+ WRITE_ONCE(m->wgen, 0U);
+ return;
+ }
+
+ flags = dept_enter();
+
+ __dept_event(m, m, e_f, ip, e_fn, false);
+
+ /*
+ * Keep the map diabled until the next sleep.
+ */
+ WRITE_ONCE(m->wgen, 0U);
+
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_event);
+
+void dept_ecxt_exit(struct dept_map *m, unsigned long e_f,
+ unsigned long ip)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ int e;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive) {
+ dt->missing_ecxt--;
+ return;
+ }
+
+ /*
+ * Should go ahead no matter whether ->nocheck == true or not
+ * because ->nocheck value can be changed within the ecxt area
+ * delimitated by dept_ecxt_enter() and dept_ecxt_exit().
+ */
+
+ flags = dept_enter();
+
+ for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) {
+ struct dept_class *c;
+ struct dept_key *k;
+
+ k = m->keys ?: &m->map_key;
+ c = check_new_class(&m->map_key, k,
+ sub_id(m, e), m->name, false);
+ if (!c)
+ continue;
+
+ /*
+ * When it found an ecxt for any event in e_f, done.
+ */
+ if (pop_ecxt(m, c))
+ goto exit;
+ }
+
+ dt->missing_ecxt--;
+exit:
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_ecxt_exit);
+
+void dept_task_exit(struct task_struct *t)
+{
+ struct dept_task *dt = &t->dept_task;
+ int i;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ raw_local_irq_disable();
+
+ if (dt->stack) {
+ put_stack(dt->stack);
+ dt->stack = NULL;
+ }
+
+ for (i = 0; i < dt->ecxt_held_pos; i++) {
+ if (dt->ecxt_held[i].class) {
+ put_class(dt->ecxt_held[i].class);
+ dt->ecxt_held[i].class = NULL;
+ }
+ if (dt->ecxt_held[i].ecxt) {
+ put_ecxt(dt->ecxt_held[i].ecxt);
+ dt->ecxt_held[i].ecxt = NULL;
+ }
+ }
+
+ for (i = 0; i < DEPT_MAX_WAIT_HIST; i++) {
+ if (dt->wait_hist[i].wait) {
+ put_wait(dt->wait_hist[i].wait);
+ dt->wait_hist[i].wait = NULL;
+ }
+ }
+
+ dt->task_exit = true;
+ dept_off();
+
+ raw_local_irq_enable();
+}
+
+void dept_task_init(struct task_struct *t)
+{
+ memset(&t->dept_task, 0x0, sizeof(struct dept_task));
+ t->dept_task.stage_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
+}
+
+void dept_key_init(struct dept_key *k)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ int sub_id;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive) {
+ DEPT_STOP("Key initialization fails.\n");
+ return;
+ }
+
+ flags = dept_enter();
+
+ clean_classes_cache(k);
+
+ /*
+ * dept_key_init() should not fail.
+ *
+ * FIXME: Should be fixed if dept_key_init() causes deadlock
+ * with dept_lock().
+ */
+ while (unlikely(!dept_lock()))
+ cpu_relax();
+
+ for (sub_id = 0; sub_id < DEPT_MAX_SUBCLASSES; sub_id++) {
+ struct dept_class *c;
+
+ c = lookup_class((unsigned long)k->base + sub_id);
+ if (!c)
+ continue;
+
+ DEPT_STOP("The class(%s/%d) has not been removed.\n",
+ c->name, sub_id);
+ break;
+ }
+
+ dept_unlock();
+ dept_exit(flags);
+}
+EXPORT_SYMBOL_GPL(dept_key_init);
+
+void dept_key_destroy(struct dept_key *k)
+{
+ struct dept_task *dt = dept_task();
+ unsigned long flags;
+ int sub_id;
+
+ if (unlikely(!dept_working()))
+ return;
+
+ if (dt->recursive == 1 && dt->task_exit) {
+ /*
+ * Need to allow to go ahead in this case where
+ * ->recursive has been set to 1 by dept_off() in
+ * dept_task_exit() and ->task_exit has been set to
+ * true in dept_task_exit().
+ */
+ } else if (dt->recursive) {
+ DEPT_STOP("Key destroying fails.\n");
+ return;
+ }
+
+ flags = dept_enter();
+
+ /*
+ * dept_key_destroy() should not fail.
+ *
+ * FIXME: Should be fixed if dept_key_destroy() causes deadlock
+ * with dept_lock().
+ */
+ while (unlikely(!dept_lock()))
+ cpu_relax();
+
+ for (sub_id = 0; sub_id < DEPT_MAX_SUBCLASSES; sub_id++) {
+ struct dept_class *c;
+
+ c = lookup_class((unsigned long)k->base + sub_id);
+ if (!c)
+ continue;
+
+ hash_del_class(c);
+ disconnect_class(c);
+ list_del(&c->all_node);
+ invalidate_class(c);
+
+ /*
+ * Actual deletion will happen on the rcu callback
+ * that has been added in disconnect_class().
+ */
+ del_class(c);
+ }
+
+ dept_unlock();
+ dept_exit(flags);
+
+ /*
+ * Wait until even lockless hash_lookup_class() for the class
+ * returns NULL.
+ */
+ might_sleep();
+ synchronize_rcu();
+}
+EXPORT_SYMBOL_GPL(dept_key_destroy);
+
+static void move_llist(struct llist_head *to, struct llist_head *from)
+{
+ struct llist_node *first = llist_del_all(from);
+ struct llist_node *last = first;
+
+ if (!first)
+ return;
+
+ while (llist_next(last))
+ last = llist_next(last);
+ llist_add_batch(first, last, to);
+}
+
+static void migrate_per_cpu_pool(void)
+{
+ const int boot_cpu = 0;
+ int i;
+
+ /*
+ * The boot CPU has been using the temporal local pool so far.
+ * From now on that per_cpu areas have been ready, use the
+ * per_cpu local pool instead.
+ */
+ DEPT_WARN_ON(smp_processor_id() != boot_cpu);
+ for (i = 0; i < OBJECT_NR; i++) {
+ struct llist_head *from;
+ struct llist_head *to;
+
+ from = &dept_pool[i].boot_pool;
+ to = per_cpu_ptr(dept_pool[i].lpool, boot_cpu);
+ move_llist(to, from);
+ }
+}
+
+#define B2KB(B) ((B) / 1024)
+
+/*
+ * Should be called after setup_per_cpu_areas() and before no non-boot
+ * CPUs have been on.
+ */
+void __init dept_init(void)
+{
+ size_t mem_total = 0;
+
+ local_irq_disable();
+ dept_per_cpu_ready = 1;
+ migrate_per_cpu_pool();
+ local_irq_enable();
+
+#define HASH(id, bits) BUILD_BUG_ON(1 << (bits) <= 0);
+ #include "dept_hash.h"
+#undef HASH
+#define OBJECT(id, nr) mem_total += sizeof(struct dept_##id) * nr;
+ #include "dept_object.h"
+#undef OBJECT
+#define HASH(id, bits) mem_total += sizeof(struct hlist_head) * (1 << (bits));
+ #include "dept_hash.h"
+#undef HASH
+
+ pr_info("DEPendency Tracker: Copyright (c) 2020 LG Electronics, Inc., Byungchul Park\n");
+ pr_info("... DEPT_MAX_STACK_ENTRY: %d\n", DEPT_MAX_STACK_ENTRY);
+ pr_info("... DEPT_MAX_WAIT_HIST : %d\n", DEPT_MAX_WAIT_HIST);
+ pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD);
+ pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES);
+#define OBJECT(id, nr) \
+ pr_info("... memory used by %s: %zu KB\n", \
+ #id, B2KB(sizeof(struct dept_##id) * nr));
+ #include "dept_object.h"
+#undef OBJECT
+#define HASH(id, bits) \
+ pr_info("... hash list head used by %s: %zu KB\n", \
+ #id, B2KB(sizeof(struct hlist_head) * (1 << (bits))));
+ #include "dept_hash.h"
+#undef HASH
+ pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total));
+ pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task));
+}
diff --git a/kernel/dependency/dept_hash.h b/kernel/dependency/dept_hash.h
new file mode 100644
index 000000000000..fd85aab1fdfb
--- /dev/null
+++ b/kernel/dependency/dept_hash.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * HASH(id, bits)
+ *
+ * id : Id for the object of struct dept_##id.
+ * bits: 1UL << bits is the hash table size.
+ */
+
+HASH(dep, 12)
+HASH(class, 12)
diff --git a/kernel/dependency/dept_internal.h b/kernel/dependency/dept_internal.h
new file mode 100644
index 000000000000..9b28398fecfd
--- /dev/null
+++ b/kernel/dependency/dept_internal.h
@@ -0,0 +1,283 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Dept(DEPendency Tracker) - runtime dependency tracker internal header
+ *
+ * Started by Byungchul Park <max.byungchul.park@gmail.com>:
+ *
+ * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park
+ * Copyright (c) 2024 SK hynix, Inc., Byungchul Park
+ */
+
+#ifndef __DEPT_INTERNAL_H
+#define __DEPT_INTERNAL_H
+
+#ifdef CONFIG_DEPT
+#include <linux/dept.h>
+#include <linux/percpu.h>
+#include <linux/llist.h>
+#include <linux/types.h>
+
+struct dept_pool {
+ const char *name;
+
+ /*
+ * object size
+ */
+ size_t obj_sz;
+
+ /*
+ * the number of the static array
+ */
+ atomic_t obj_nr;
+
+ /*
+ * offset of ->pool_node
+ */
+ size_t node_off;
+
+ /*
+ * pointer to the pool
+ */
+ void *spool;
+ struct llist_head boot_pool;
+ struct llist_head __percpu *lpool;
+};
+
+struct dept_ecxt;
+struct dept_iecxt {
+ struct dept_ecxt *ecxt;
+ int enirq;
+ /*
+ * for preventing to add a new ecxt
+ */
+ bool staled;
+};
+
+struct dept_wait;
+struct dept_iwait {
+ struct dept_wait *wait;
+ int irq;
+ /*
+ * for preventing to add a new wait
+ */
+ bool staled;
+ bool touched;
+};
+
+struct dept_class {
+ union {
+ struct llist_node pool_node;
+ struct {
+ /*
+ * reference counter for object management
+ */
+ atomic_t ref;
+
+ /*
+ * unique information about the class
+ */
+ const char *name;
+ unsigned long key;
+ int sub_id;
+
+ /*
+ * for BFS
+ */
+ unsigned int bfs_gen;
+ struct dept_class *bfs_parent;
+ struct list_head bfs_node;
+
+ /*
+ * for hashing this object
+ */
+ struct hlist_node hash_node;
+
+ /*
+ * for linking all classes
+ */
+ struct list_head all_node;
+
+ /*
+ * for associating its dependencies
+ */
+ struct list_head dep_head;
+ struct list_head dep_rev_head;
+
+ /*
+ * for tracking IRQ dependencies
+ */
+ struct dept_iecxt iecxt[DEPT_IRQS_NR];
+ struct dept_iwait iwait[DEPT_IRQS_NR];
+
+ /*
+ * classified by a map embedded in task_struct,
+ * not an explicit map
+ */
+ bool sched_map;
+ };
+ };
+};
+
+struct dept_stack {
+ union {
+ struct llist_node pool_node;
+ struct {
+ /*
+ * reference counter for object management
+ */
+ atomic_t ref;
+
+ /*
+ * backtrace entries
+ */
+ unsigned long raw[DEPT_MAX_STACK_ENTRY];
+ int nr;
+ };
+ };
+};
+
+struct dept_ecxt {
+ union {
+ struct llist_node pool_node;
+ struct {
+ /*
+ * reference counter for object management
+ */
+ atomic_t ref;
+
+ /*
+ * function that entered to this ecxt
+ */
+ const char *ecxt_fn;
+
+ /*
+ * event function
+ */
+ const char *event_fn;
+
+ /*
+ * associated class
+ */
+ struct dept_class *class;
+
+ /*
+ * flag indicating which IRQ has been
+ * enabled within the event context
+ */
+ unsigned long enirqf;
+
+ /*
+ * where the IRQ-enabled happened
+ */
+ unsigned long enirq_ip[DEPT_IRQS_NR];
+ struct dept_stack *enirq_stack[DEPT_IRQS_NR];
+
+ /*
+ * where the event context started
+ */
+ unsigned long ecxt_ip;
+ struct dept_stack *ecxt_stack;
+
+ /*
+ * where the event triggered
+ */
+ unsigned long event_ip;
+ struct dept_stack *event_stack;
+ };
+ };
+};
+
+struct dept_wait {
+ union {
+ struct llist_node pool_node;
+ struct {
+ /*
+ * reference counter for object management
+ */
+ atomic_t ref;
+
+ /*
+ * function causing this wait
+ */
+ const char *wait_fn;
+
+ /*
+ * the associated class
+ */
+ struct dept_class *class;
+
+ /*
+ * which IRQ the wait was placed in
+ */
+ unsigned long irqf;
+
+ /*
+ * where the IRQ wait happened
+ */
+ unsigned long irq_ip[DEPT_IRQS_NR];
+ struct dept_stack *irq_stack[DEPT_IRQS_NR];
+
+ /*
+ * where the wait happened
+ */
+ unsigned long wait_ip;
+ struct dept_stack *wait_stack;
+
+ /*
+ * whether this wait is for commit in scheduler
+ */
+ bool sched_sleep;
+ };
+ };
+};
+
+struct dept_dep {
+ union {
+ struct llist_node pool_node;
+ struct {
+ /*
+ * reference counter for object management
+ */
+ atomic_t ref;
+
+ /*
+ * key data of dependency
+ */
+ struct dept_ecxt *ecxt;
+ struct dept_wait *wait;
+
+ /*
+ * This object can be referred without dept_lock
+ * held but with IRQ disabled, e.g. for hash
+ * lookup. So deferred deletion is needed.
+ */
+ struct rcu_head rh;
+
+ /*
+ * for hashing this object
+ */
+ struct hlist_node hash_node;
+
+ /*
+ * for linking to a class object
+ */
+ struct list_head dep_node;
+ struct list_head dep_rev_node;
+ };
+ };
+};
+
+struct dept_hash {
+ /*
+ * hash table
+ */
+ struct hlist_head *table;
+
+ /*
+ * size of the table e.i. 2^bits
+ */
+ int bits;
+};
+
+#endif
+#endif /* __DEPT_INTERNAL_H */
diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h
new file mode 100644
index 000000000000..0b7eb16fe9fb
--- /dev/null
+++ b/kernel/dependency/dept_object.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * OBJECT(id, nr)
+ *
+ * id: Id for the object of struct dept_##id.
+ * nr: # of the object that should be kept in the pool.
+ */
+
+OBJECT(dep, 1024 * 8)
+OBJECT(class, 1024 * 8)
+OBJECT(stack, 1024 * 32)
+OBJECT(ecxt, 1024 * 16)
+OBJECT(wait, 1024 * 32)
diff --git a/kernel/exit.c b/kernel/exit.c
index 4dc1918db67b..99c848b58177 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -1013,6 +1013,7 @@ void __noreturn do_exit(long code)
exit_tasks_rcu_finish();
lockdep_free_task(tsk);
+ dept_task_exit(tsk);
do_task_dead();
}
diff --git a/kernel/fork.c b/kernel/fork.c
index 25d243718048..07c87562d11d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -106,6 +106,7 @@
#include <linux/pidfs.h>
#include <linux/tick.h>
#include <linux/unwind_deferred.h>
+#include <linux/dept.h>
#include <asm/pgalloc.h>
#include <linux/uaccess.h>
@@ -2130,6 +2131,7 @@ __latent_entropy struct task_struct *copy_process(
p->pagefault_disabled = 0;
lockdep_init_task(p);
+ dept_task_init(p);
p->blocked_on = NULL; /* not blocked yet */
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 7b3ec2fa6e7c..03ed63f2adf0 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -1375,12 +1375,14 @@ static void free_mod_mem(struct module *mod)
/* Free lock-classes; relies on the preceding sync_rcu(). */
lockdep_free_key_range(mod_mem->base, mod_mem->size);
+ dept_free_range(mod_mem->base, mod_mem->size);
if (mod_mem->size)
module_memory_free(mod, type);
}
/* MOD_DATA hosts mod, so free it at last */
lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size);
+ dept_free_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size);
module_memory_free(mod, MOD_DATA);
}
@@ -3548,6 +3550,8 @@ static int load_module(struct load_info *info, const char __user *uargs,
for_class_mod_mem_type(type, core_data) {
lockdep_free_key_range(mod->mem[type].base,
mod->mem[type].size);
+ dept_free_range(mod->mem[type].base,
+ mod->mem[type].size);
}
module_memory_restore_rox(mod);
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b7801cd05d5a..d41112efaeae 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -69,6 +69,7 @@
#include <linux/wait_api.h>
#include <linux/workqueue_api.h>
#include <linux/livepatch_sched.h>
+#include <linux/dept.h>
#ifdef CONFIG_PREEMPT_DYNAMIC
# ifdef CONFIG_GENERIC_IRQ_ENTRY
@@ -4136,6 +4137,8 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
break;
+ dept_ttwu_stage_wait(p, _RET_IP_);
+
/*
* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be
* possible to, falsely, observe p->on_cpu == 0.
@@ -6737,6 +6740,11 @@ static void __sched notrace __schedule(int sched_mode)
rq = cpu_rq(cpu);
prev = rq->curr;
+ prev_state = READ_ONCE(prev->__state);
+ if (sched_mode != SM_PREEMPT && prev_state & TASK_NORMAL)
+ dept_request_event_wait_commit();
+
+ dept_sched_enter();
schedule_debug(prev, preempt);
if (sched_feat(HRTICK) || sched_feat(HRTICK_DL))
@@ -6871,6 +6879,7 @@ static void __sched notrace __schedule(int sched_mode)
raw_spin_rq_unlock_irq(rq);
}
trace_sched_exit_tp(is_switch);
+ dept_sched_exit();
}
void __noreturn do_task_dead(void)
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 742b23ef0d8b..712514e2c149 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1368,6 +1368,32 @@ config DEBUG_PREEMPT
menu "Lock Debugging (spinlocks, mutexes, etc...)"
+config DEPT
+ bool "Dependency tracking (EXPERIMENTAL)"
+ depends on DEBUG_KERNEL && LOCK_DEBUGGING_SUPPORT
+ select DEBUG_SPINLOCK
+ select DEBUG_MUTEXES if !PREEMPT_RT
+ select DEBUG_RT_MUTEXES if RT_MUTEXES
+ select DEBUG_RWSEMS if !PREEMPT_RT
+ select DEBUG_WW_MUTEX_SLOWPATH
+ select DEBUG_LOCK_ALLOC
+ select TRACE_IRQFLAGS
+ select STACKTRACE
+ select KALLSYMS
+ select KALLSYMS_ALL
+ select PROVE_LOCKING
+ default n
+ help
+ Check dependencies between wait and event and report it if
+ deadlock possibility has been detected. Multiple reports are
+ allowed if there are more than a single problem.
+
+ This feature is considered EXPERIMENTAL that might produce
+ false positive reports because new dependencies start to be
+ tracked, that have never been tracked before. It's worth
+ noting, to mitigate the impact by the false positives, multi
+ reporting has been supported.
+
config LOCK_DEBUGGING_SUPPORT
bool
depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c
index d939403331b5..a7f8e59d0092 100644
--- a/lib/locking-selftest.c
+++ b/lib/locking-selftest.c
@@ -1398,6 +1398,8 @@ static void reset_locks(void)
local_irq_disable();
lockdep_free_key_range(&ww_lockdep.acquire_key, 1);
lockdep_free_key_range(&ww_lockdep.mutex_key, 1);
+ dept_free_range(&ww_lockdep.acquire_key, 1);
+ dept_free_range(&ww_lockdep.mutex_key, 1);
I1(A); I1(B); I1(C); I1(D);
I1(X1); I1(X2); I1(Y1); I1(Y2); I1(Z1); I1(Z2);
--
2.17.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox