+ kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

+ kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[Andrew Morton]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 5022 bytes --]


The patch titled
     Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
has been added to the -mm mm-nonmm-unstable branch.  Its filename is
     kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch

This patch will shortly appear at
     https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch

This patch will later appear in the mm-nonmm-unstable branch at
    git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Before you just go and hit "reply", please:
   a) Consider who else should be cc'ed
   b) Prefer to cc a suitable mailing list as well
   c) Ideally: find the original patch on the mailing list and do a
      reply-to-all to that, adding suitable additional cc's

*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***

The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days

------------------------------------------------------
From: Jim Cromie <jim.cromie@gmail.com>
Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
Date: Thu, 08 Oct 2026 17:19:12 -0600

commit 4fafd1165b33 replaced hardcoded values which determine marker array
size, with new values that accelerate symbol-name lookups.  These symbols
are needed in user-space to generate the lookup tables, and in kernel to
read the generated table.

To avoid 2-defns problem, they were added to a kernel header (with ifdef
guards) and included by scripts/kallsyms.c.  But that creates packaging
problems elsewhere, so just add the 2nd defns, with an explicit comment
about the user-kernel coupling.

Link: https://lore.kernel.org/20261008-fix-kallsyms-v1-1-3339fdac2ab3@gmail.com
Fixes: 4fafd1165b33 ("kallsyms: increase marker density to 16:1 to accelerate lookups")
Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Tested-by: Nicolas Schier <nsc@kernel.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Denis Benato <denis.benato@linux.dev>
Cc: Kees Cook <kees@kernel.org>
Cc: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Cc: James Le Cuirot <chewi@gentoo.org>
Cc: Abel Vesa <abelvesa@kernel.org>
Cc: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Jiri Olsa <olsajiri@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Luis Chamberlain <mcgrof@kernel.org>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Zhen Lei <thunder.leizhen@huawei.com>
---

 kernel/kallsyms_internal.h |   12 +++++-------
 scripts/kallsyms.c         |    9 ++++++++-
 2 files changed, 13 insertions(+), 8 deletions(-)

--- a/kernel/kallsyms_internal.h~kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix
+++ a/kernel/kallsyms_internal.h
@@ -2,18 +2,17 @@
 #ifndef LINUX_KALLSYMS_INTERNAL_H_
 #define LINUX_KALLSYMS_INTERNAL_H_
 
+#include <linux/types.h>
+
 /*
- * Provide compile-constants for scripts/kallsyms.c
- * so it can build the corresponding kallsyms_marker[] table.
- * and wrap the rest in __KERNEL__
+ * These constants determine the size of the kallsyms_markers array
+ * written by scripts/kallsyms.c during the build.  The symbol values
+ * must be kept in sync.
  */
 #define KALLSYMS_MARKER_SHIFT 4
 #define KALLSYMS_MARKER_SIZE  (1U << KALLSYMS_MARKER_SHIFT)
 #define KALLSYMS_MARKER_MASK  (KALLSYMS_MARKER_SIZE - 1U)
 
-#ifdef __KERNEL__
-#include <linux/types.h>
-
 extern const int kallsyms_offsets[];
 extern const u8 kallsyms_names[];
 
@@ -24,6 +23,5 @@ extern const u16 kallsyms_token_index[];
 
 extern const unsigned int kallsyms_markers[];
 extern const u8 kallsyms_seqs_of_names[];
-#endif /* __KERNEL__ */
 
 #endif // LINUX_KALLSYMS_INTERNAL_H_
--- a/scripts/kallsyms.c~kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix
+++ a/scripts/kallsyms.c
@@ -29,7 +29,14 @@
 
 #include <xalloc.h>
 
-#include "../kernel/kallsyms_internal.h"
+/*
+ * These constants determine the size of the marker array filled here,
+ * and they're duplicated in kernel/kallsyms.c, where they're used in
+ * get_symbol_offset().  They must be kept in agreement.
+ */
+#define KALLSYMS_MARKER_SHIFT 4
+#define KALLSYMS_MARKER_SIZE  (1U << KALLSYMS_MARKER_SHIFT)
+#define KALLSYMS_MARKER_MASK  (KALLSYMS_MARKER_SIZE - 1U)
 
 #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof(arr[0]))
 
_

Patches currently in -mm which might be from jim.cromie@gmail.com are

kallsyms-match-compressed-tokens-on-the-fly-during-binary-search.patch
kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups.patch
kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch
kallsyms-unroll-24-bit-sequence-reconstruction-in-get_symbol_seq.patch

Re: + kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[David Laight]

On Sat, 10 Oct 2026 17:23:40 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> The patch titled
>      Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> has been added to the -mm mm-nonmm-unstable branch.  Its filename is
>      kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch

There is a problem with the way this change in implemented.
Basically the kernel Makefile (etc) only builds scripts/kallsyms if it doesn't exist.
This means that if the header file is changed you can easily end up with a kernel
that expects a different marker density from that in the data file.
That will lead to all sorts of strange/confusing issues.

It would be better to put the 'shift' value into the .S file.
It isn't used anywhere where that will make a performance difference.

David

> 
> This patch will shortly appear at
>      https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch
> 
> This patch will later appear in the mm-nonmm-unstable branch at
>     git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> 
> Before you just go and hit "reply", please:
>    a) Consider who else should be cc'ed
>    b) Prefer to cc a suitable mailing list as well
>    c) Ideally: find the original patch on the mailing list and do a
>       reply-to-all to that, adding suitable additional cc's
> 
> *** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
> 
> The -mm tree is included into linux-next via various
> branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> and is updated there most days
> 
> ------------------------------------------------------
> From: Jim Cromie <jim.cromie@gmail.com>
> Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> Date: Thu, 08 Oct 2026 17:19:12 -0600
> 
> commit 4fafd1165b33 replaced hardcoded values which determine marker array
> size, with new values that accelerate symbol-name lookups.  These symbols
> are needed in user-space to generate the lookup tables, and in kernel to
> read the generated table.
> 
> To avoid 2-defns problem, they were added to a kernel header (with ifdef
> guards) and included by scripts/kallsyms.c.  But that creates packaging
> problems elsewhere, so just add the 2nd defns, with an explicit comment
> about the user-kernel coupling.
> 
> Link: https://lore.kernel.org/20261008-fix-kallsyms-v1-1-3339fdac2ab3@gmail.com
> Fixes: 4fafd1165b33 ("kallsyms: increase marker density to 16:1 to accelerate lookups")
> Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> Tested-by: Nicolas Schier <nsc@kernel.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>
> Cc: Nathan Chancellor <nathan@kernel.org>
> Cc: Denis Benato <denis.benato@linux.dev>
> Cc: Kees Cook <kees@kernel.org>
> Cc: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Cc: James Le Cuirot <chewi@gentoo.org>
> Cc: Abel Vesa <abelvesa@kernel.org>
> Cc: Andrey Grodzovsky <andrey.grodzovsky@crowdstrike.com>
> Cc: David Laight <david.laight.linux@gmail.com>
> Cc: Geert Uytterhoeven <geert@linux-m68k.org>
> Cc: Jiri Olsa <olsajiri@gmail.com>
> Cc: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Luis Chamberlain <mcgrof@kernel.org>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Petr Mladek <pmladek@suse.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Zhen Lei <thunder.leizhen@huawei.com>
> ---
> 
>  kernel/kallsyms_internal.h |   12 +++++-------
>  scripts/kallsyms.c         |    9 ++++++++-
>  2 files changed, 13 insertions(+), 8 deletions(-)
> 
> --- a/kernel/kallsyms_internal.h~kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix
> +++ a/kernel/kallsyms_internal.h
> @@ -2,18 +2,17 @@
>  #ifndef LINUX_KALLSYMS_INTERNAL_H_
>  #define LINUX_KALLSYMS_INTERNAL_H_
>  
> +#include <linux/types.h>
> +
>  /*
> - * Provide compile-constants for scripts/kallsyms.c
> - * so it can build the corresponding kallsyms_marker[] table.
> - * and wrap the rest in __KERNEL__
> + * These constants determine the size of the kallsyms_markers array
> + * written by scripts/kallsyms.c during the build.  The symbol values
> + * must be kept in sync.
>   */
>  #define KALLSYMS_MARKER_SHIFT 4
>  #define KALLSYMS_MARKER_SIZE  (1U << KALLSYMS_MARKER_SHIFT)
>  #define KALLSYMS_MARKER_MASK  (KALLSYMS_MARKER_SIZE - 1U)
>  
> -#ifdef __KERNEL__
> -#include <linux/types.h>
> -
>  extern const int kallsyms_offsets[];
>  extern const u8 kallsyms_names[];
>  
> @@ -24,6 +23,5 @@ extern const u16 kallsyms_token_index[];
>  
>  extern const unsigned int kallsyms_markers[];
>  extern const u8 kallsyms_seqs_of_names[];
> -#endif /* __KERNEL__ */
>  
>  #endif // LINUX_KALLSYMS_INTERNAL_H_
> --- a/scripts/kallsyms.c~kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix
> +++ a/scripts/kallsyms.c
> @@ -29,7 +29,14 @@
>  
>  #include <xalloc.h>
>  
> -#include "../kernel/kallsyms_internal.h"
> +/*
> + * These constants determine the size of the marker array filled here,
> + * and they're duplicated in kernel/kallsyms.c, where they're used in
> + * get_symbol_offset().  They must be kept in agreement.
> + */
> +#define KALLSYMS_MARKER_SHIFT 4
> +#define KALLSYMS_MARKER_SIZE  (1U << KALLSYMS_MARKER_SHIFT)
> +#define KALLSYMS_MARKER_MASK  (KALLSYMS_MARKER_SIZE - 1U)
>  
>  #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof(arr[0]))
>  
> _
> 
> Patches currently in -mm which might be from jim.cromie@gmail.com are
> 
> kallsyms-match-compressed-tokens-on-the-fly-during-binary-search.patch
> kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups.patch
> kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch
> kallsyms-unroll-24-bit-sequence-reconstruction-in-get_symbol_seq.patch
> 

Re: + kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[Andrew Morton]

On Sun, 11 Oct 2026 15:44:01 +0100 David Laight <david.laight.linux@gmail.com> wrote:

> On Sat, 10 Oct 2026 17:23:40 -0700
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > The patch titled
> >      Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> > has been added to the -mm mm-nonmm-unstable branch.  Its filename is
> >      kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch
> 
> There is a problem with the way this change in implemented.
> Basically the kernel Makefile (etc) only builds scripts/kallsyms if it doesn't exist.
> This means that if the header file is changed you can easily end up with a kernel
> that expects a different marker density from that in the data file.
> That will lead to all sorts of strange/confusing issues.
> 
> It would be better to put the 'shift' value into the .S file.
> It isn't used anywhere where that will make a performance difference.

OK, thanks.  I'll remove the v7 series from the 7.3-rcX queue - let's
revisit for the next -rc cycle.

Re: + kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[David Laight]

On Sun, 11 Oct 2026 08:15:52 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Sun, 11 Oct 2026 15:44:01 +0100 David Laight <david.laight.linux@gmail.com> wrote:
> 
> > On Sat, 10 Oct 2026 17:23:40 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> >   
> > > The patch titled
> > >      Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> > > has been added to the -mm mm-nonmm-unstable branch.  Its filename is
> > >      kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch  
> > 
> > There is a problem with the way this change in implemented.
> > Basically the kernel Makefile (etc) only builds scripts/kallsyms if it doesn't exist.
> > This means that if the header file is changed you can easily end up with a kernel
> > that expects a different marker density from that in the data file.
> > That will lead to all sorts of strange/confusing issues.
> > 
> > It would be better to put the 'shift' value into the .S file.
> > It isn't used anywhere where that will make a performance difference.  
> 
> OK, thanks.  I'll remove the v7 series from the 7.3-rcX queue - let's
> revisit for the next -rc cycle.
> 

The patch that changes the name comparison can go in.
That saves 512 bytes of stack but doesn't make (much) difference to
lookups because other things dominate (probably unless you have some
very long symbols from rust code).

I am working on an additional change.
(I posted a v1, v2 adds some extra changes and is taking a while.)
Performance wise it improves name lookup by a factor of ~18 while
only increasing the 'density' to 64:1 (1 bit per two pixels).

I've make the density a kernel config option (default 6 => 64:1).

One of the reasons it is taking 'ages' is because I discovered that four
of the 'token' values aren't actually used in any symbol names (in the
dataset I'm using) - so they can be reused reducing the main table by
~8k (in 1.7M).

David

Re: + kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[jim.cromie]

hi David

On Mon, Oct 12, 2026 at 4:14 AM David Laight
<david.laight.linux@gmail.com> wrote:
>
> On Sun, 11 Oct 2026 08:15:52 -0700
> Andrew Morton <akpm@linux-foundation.org> wrote:
>
> > On Sun, 11 Oct 2026 15:44:01 +0100 David Laight <david.laight.linux@gmail.com> wrote:
> >
> > > On Sat, 10 Oct 2026 17:23:40 -0700
> > > Andrew Morton <akpm@linux-foundation.org> wrote:
> > >
> > > > The patch titled
> > > >      Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> > > > has been added to the -mm mm-nonmm-unstable branch.  Its filename is
> > > >      kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch
> > >
> > > There is a problem with the way this change in implemented.
> > > Basically the kernel Makefile (etc) only builds scripts/kallsyms if it doesn't exist.
> > > This means that if the header file is changed you can easily end up with a kernel
> > > that expects a different marker density from that in the data file.
> > > That will lead to all sorts of strange/confusing issues.
> > >


Is this problem fixed by making these all depend upon a CONFIG_  item ?

Nathan Chancellor suggested this earlier,
I demurred cuz it seemed like an unnecessary knob,
but solving a build problem is a higher purpose.

of course your v2 might make this moot, I'll try it when it lands here.


> > > It would be better to put the 'shift' value into the .S file.
> > > It isn't used anywhere where that will make a performance difference.
> >
> > OK, thanks.  I'll remove the v7 series from the 7.3-rcX queue - let's
> > revisit for the next -rc cycle.
> >
>
> The patch that changes the name comparison can go in.
> That saves 512 bytes of stack but doesn't make (much) difference to
> lookups because other things dominate (probably unless you have some
> very long symbols from rust code).
>
> I am working on an additional change.
> (I posted a v1, v2 adds some extra changes and is taking a while.)
> Performance wise it improves name lookup by a factor of ~18 while
> only increasing the 'density' to 64:1 (1 bit per two pixels).
>
> I've make the density a kernel config option (default 6 => 64:1).
>
> One of the reasons it is taking 'ages' is because I discovered that four
> of the 'token' values aren't actually used in any symbol names (in the
> dataset I'm using) - so they can be reused reducing the main table by
> ~8k (in 1.7M).
>
> David

Re: + kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch added to mm-nonmm-unstable branch

[David Laight]

On Mon, 12 Oct 2026 16:39:41 -0600
jim.cromie@gmail.com wrote:

> hi David
> 
> On Mon, Oct 12, 2026 at 4:14 AM David Laight
> <david.laight.linux@gmail.com> wrote:
> >
> > On Sun, 11 Oct 2026 08:15:52 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> >  
> > > On Sun, 11 Oct 2026 15:44:01 +0100 David Laight <david.laight.linux@gmail.com> wrote:
> > >  
> > > > On Sat, 10 Oct 2026 17:23:40 -0700
> > > > Andrew Morton <akpm@linux-foundation.org> wrote:
> > > >  
> > > > > The patch titled
> > > > >      Subject: kallsyms: copy constants from __KERNEL__ to user, drop the inclusion
> > > > > has been added to the -mm mm-nonmm-unstable branch.  Its filename is
> > > > >      kallsyms-increase-marker-density-to-16-1-to-accelerate-lookups-fix.patch  
> > > >
> > > > There is a problem with the way this change in implemented.
> > > > Basically the kernel Makefile (etc) only builds scripts/kallsyms if it doesn't exist.
> > > > This means that if the header file is changed you can easily end up with a kernel
> > > > that expects a different marker density from that in the data file.
> > > > That will lead to all sorts of strange/confusing issues.
> > > >  
> 
> 
> Is this problem fixed by making these all depend upon a CONFIG_  item ?
> 
> Nathan Chancellor suggested this earlier,
> I demurred cuz it seemed like an unnecessary knob,
> but solving a build problem is a higher purpose.
> 
> of course your v2 might make this moot, I'll try it when it lands here.

That is what I end up doing, it has to get passed to scripts/kallsyms
as a command line parameter and then written to the output file.
That way nothing can go wrong (honest).

My v2 should be along soon.
I'm busy going through my changes and reordering them into a reasonable
set of patches.
It's raining, progress might be made.

Oh, try looking up _sub_D_65535_0 in an allyesconfig kernel; it is repeated
some 24610 times. I'm not trying that one with the old code.
Even a normal kernel has __func__.0 repeated hundreds of times.

David

> 
> 
> > > > It would be better to put the 'shift' value into the .S file.
> > > > It isn't used anywhere where that will make a performance difference.  
> > >
> > > OK, thanks.  I'll remove the v7 series from the 7.3-rcX queue - let's
> > > revisit for the next -rc cycle.
> > >  
> >
> > The patch that changes the name comparison can go in.
> > That saves 512 bytes of stack but doesn't make (much) difference to
> > lookups because other things dominate (probably unless you have some
> > very long symbols from rust code).
> >
> > I am working on an additional change.
> > (I posted a v1, v2 adds some extra changes and is taking a while.)
> > Performance wise it improves name lookup by a factor of ~18 while
> > only increasing the 'density' to 64:1 (1 bit per two pixels).
> >
> > I've make the density a kernel config option (default 6 => 64:1).
> >
> > One of the reasons it is taking 'ages' is because I discovered that four
> > of the 'token' values aren't actually used in any symbol names (in the
> > dataset I'm using) - so they can be reused reducing the main table by
> > ~8k (in 1.7M).
> >
> > David