* [PATCH v5 2/7] module: add kflagstab section to vmlinux and modules
From: Siddharth Nayyar @ 2026-03-26 21:21 UTC (permalink / raw)
To: Luis Chamberlain, Petr Pavlu, Daniel Gomez, Sami Tolvanen,
Aaron Tomlin, Arnd Bergmann, Nathan Chancellor, Nicolas Schier,
Jonathan Corbet, Shuah Khan
Cc: linux-modules, linux-kernel, linux-arch, linux-kbuild, linux-doc,
Siddharth Nayyar, gprocida
In-Reply-To: <20260326-kflagstab-v5-0-455cd723dddf@google.com>
This section will contain read-only data for values of kernel symbol
flags in the form of an 8-bit bitsets for each kernel symbol. Each bit
in the bitset represents a flag value defined by ksym_flags enumeration.
The kflagstab section introduces a 1-byte overhead for each symbol
exported in the ksymtab. Given that typical kernel builds contain
roughly a few thousand exported symbols, the resulting memory increase
is negligible.
Signed-off-by: Siddharth Nayyar <sidnayyar@google.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
---
include/asm-generic/vmlinux.lds.h | 7 +++++++
scripts/module.lds.S | 1 +
2 files changed, 8 insertions(+)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 1e1580febe4b..d64a475c468a 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -536,6 +536,13 @@
__stop___kcrctab_gpl = .; \
} \
\
+ /* Kernel symbol flags table */ \
+ __kflagstab : AT(ADDR(__kflagstab) - LOAD_OFFSET) { \
+ __start___kflagstab = .; \
+ KEEP(*(SORT(___kflagstab+*))) \
+ __stop___kflagstab = .; \
+ } \
+ \
/* Kernel symbol table: strings */ \
__ksymtab_strings : AT(ADDR(__ksymtab_strings) - LOAD_OFFSET) { \
*(__ksymtab_strings) \
diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index 054ef99e8288..d7a8ba278dfc 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -23,6 +23,7 @@ SECTIONS {
__ksymtab_gpl 0 : ALIGN(8) { *(SORT(___ksymtab_gpl+*)) }
__kcrctab 0 : ALIGN(4) { *(SORT(___kcrctab+*)) }
__kcrctab_gpl 0 : ALIGN(4) { *(SORT(___kcrctab_gpl+*)) }
+ __kflagstab 0 : ALIGN(1) { *(SORT(___kflagstab+*)) }
.ctors 0 : ALIGN(8) { *(SORT(.ctors.*)) *(.ctors) }
.init_array 0 : ALIGN(8) { *(SORT(.init_array.*)) *(.init_array) }
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v5 1/7] module: define ksym_flags enumeration to represent kernel symbol flags
From: Siddharth Nayyar @ 2026-03-26 21:21 UTC (permalink / raw)
To: Luis Chamberlain, Petr Pavlu, Daniel Gomez, Sami Tolvanen,
Aaron Tomlin, Arnd Bergmann, Nathan Chancellor, Nicolas Schier,
Jonathan Corbet, Shuah Khan
Cc: linux-modules, linux-kernel, linux-arch, linux-kbuild, linux-doc,
Siddharth Nayyar, gprocida
In-Reply-To: <20260326-kflagstab-v5-0-455cd723dddf@google.com>
Symbol flags is an enumeration used to represent flags as a bitset, for
example a flag to tell if a symbol is GPL only.
The said bitset is introduced in subsequent patches and will contain
values of kernel symbol flags. These bitset will then be used to infer
flag values rather than fragmenting ksymtab for separating symbols with
different flag values, thereby eliminating the need to fragment the
ksymtab.
Signed-off-by: Siddharth Nayyar <sidnayyar@google.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
---
include/linux/module_symbol.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/include/linux/module_symbol.h b/include/linux/module_symbol.h
index 77c9895b9ddb..574609aced99 100644
--- a/include/linux/module_symbol.h
+++ b/include/linux/module_symbol.h
@@ -2,6 +2,11 @@
#ifndef _LINUX_MODULE_SYMBOL_H
#define _LINUX_MODULE_SYMBOL_H
+/* Kernel symbol flags bitset. */
+enum ksym_flags {
+ KSYM_FLAG_GPL_ONLY = 1 << 0,
+};
+
/* This ignores the intensely annoying "mapping symbols" found in ELF files. */
static inline bool is_mapping_symbol(const char *str)
{
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v5 0/7] scalable symbol flags with __kflagstab
From: Siddharth Nayyar @ 2026-03-26 21:21 UTC (permalink / raw)
To: Luis Chamberlain, Petr Pavlu, Daniel Gomez, Sami Tolvanen,
Aaron Tomlin, Arnd Bergmann, Nathan Chancellor, Nicolas Schier,
Jonathan Corbet, Shuah Khan
Cc: linux-modules, linux-kernel, linux-arch, linux-kbuild, linux-doc,
Siddharth Nayyar, gprocida
This patch series implements a mechanism for scalable exported symbol
flags using a separate section called __kflagstab. The series introduces
__kflagstab support, removes *_gpl sections in favor of a GPL flag,
simplifies symbol resolution during module loading.
The __kflagstab contains an 8-bit bitset which can represent up to 8
boolean flags per symbol exported in the __ksymtab. The patch series
also uses this bitset to store GPL-only flag values for kernel symbols,
thereby eliminating the need for *_gpl sections for representing GPL
only symbols.
Petr Pavlu ran a small test to get a better understanding of the
different section sizes resulting from this patch series. He used
v6.17-rc6 together with the openSUSE x86_64 config [1], which is fairly
large. The resulting vmlinux.bin (no debuginfo) had an on-disk size of
58 MiB, and included 5937 + 6589 (GPL-only) exported symbols.
The following table summarizes his measurements and calculations
regarding the sizes of all sections related to exported symbols:
| HAVE_ARCH_PREL32_RELOCATIONS | !HAVE_ARCH_PREL32_RELOCATIONS
Section | Base [B] | Ext. [B] | Sep. [B] | Base [B] | Ext. [B] | Sep. [B]
----------------------------------------------------------------------------------------
__ksymtab | 71244 | 200416 | 150312 | 142488 | 400832 | 300624
__ksymtab_gpl | 79068 | NA | NA | 158136 | NA | NA
__kcrctab | 23748 | 50104 | 50104 | 23748 | 50104 | 50104
__kcrctab_gpl | 26356 | NA | NA | 26356 | NA | NA
__ksymtab_strings | 253628 | 253628 | 253628 | 253628 | 253628 | 253628
__kflagstab | NA | NA | 12526 | NA | NA | 12526
----------------------------------------------------------------------------------------
Total | 454044 | 504148 | 466570 | 604356 | 704564 | 616882
Increase to base [%] | NA | 11.0 | 2.8 | NA | 16.6 | 2.1
The column "HAVE_ARCH_PREL32_RELOCATIONS -> Base" contains themeasured
numbers. The rest of the values are calculated. The "Ext." column
represents an alternative approach of extending __ksymtab to include a
bitset of symbol flags, and the "Sep." column represents the approach of
having a separate __kflagstab. With HAVE_ARCH_PREL32_RELOCATIONS, each
kernel_symbol is 12 B in size and is extended to 16 B. With
!HAVE_ARCH_PREL32_RELOCATIONS, it is 24 B, extended to 32 B. Note that
this does not include the metadata needed to relocate __ksymtab*, which
is freed after the initial processing.
The base export data in this case totals 0.43 MiB. About 50% is used for
storing the names of exported symbols.
Adding __kflagstab as a separate section has a negligible impact, as
expected. When extending __ksymtab (kernel_symbol) instead, the worst
case with !HAVE_ARCH_PREL32_RELOCATIONS increases the export data size
by 16.6%. Note that the larger increase in size for the latter approach
is due to 4-byte alignment of kernel_symbol data structure, instead of
1-byte alignment for the flags bitset in __kflagstab in the former
approach.
Based on the above, it was concluded that introducing __kflagstab makes
senses, as the added complexity is minimal over extending kernel_symbol,
and there is overall simplification of symbol finding logic in the
module loader.
Thank you Petr Pavlu for doing a section size analysis as well as Sami
Tolvanen, Petr Pavlu and Jonathan Corbet for their valuable feedback.
---
Changes from v4:
- squashed patches #4 and #5 to fix a bisecting issue
v4:
https://lore.kernel.org/r/20260305-kflagstab-v4-0-6a76bf8b83c7@google.com
Changes from v3:
- made commit messages more descriptive
v3:
https://lore.kernel.org/20251103161954.1351784-1-sidnayyar@google.com/
Changes from v2:
- dropped symbol import protection to spin off into its own series
v2:
https://lore.kernel.org/20251013153918.2206045-1-sidnayyar@google.com/
Changes from v1:
- added a check to ensure __kflagstab is present
- added warnings for the obsolete *_gpl sections
- moved protected symbol check before ref_module() call
- moved protected symbol check failure warning to issue detection point
v1:
https://lore.kernel.org/20250829105418.3053274-1-sidnayyar@google.com/
[1] https://github.com/openSUSE/kernel-source/blob/307f149d9100a0e229eb94cbb997ae61187995c3/config/x86_64/default
Signed-off-by: Siddharth Nayyar <sidnayyar@google.com>
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
---
Siddharth Nayyar (7):
module: define ksym_flags enumeration to represent kernel symbol flags
module: add kflagstab section to vmlinux and modules
module: populate kflagstab in modpost
module: use kflagstab instead of *_gpl sections
module: deprecate usage of *_gpl sections in module loader
module: remove *_gpl sections from vmlinux and modules
documentation: remove references to *_gpl sections
Documentation/kbuild/modules.rst | 11 +++--
include/asm-generic/vmlinux.lds.h | 21 +++-----
include/linux/export-internal.h | 28 +++++++----
include/linux/module.h | 4 +-
include/linux/module_symbol.h | 5 ++
kernel/module/internal.h | 4 +-
kernel/module/main.c | 101 ++++++++++++++++++--------------------
scripts/mod/modpost.c | 16 ++++--
scripts/module.lds.S | 3 +-
9 files changed, 98 insertions(+), 95 deletions(-)
---
base-commit: 0138af2472dfdef0d56fc4697416eaa0ff2589bd
change-id: 20260305-kflagstab-51a08efed244
Best regards,
--
Siddharth Nayyar <sidnayyar@google.com>
^ permalink raw reply
* Re: [PATCH v3 0/8] module: Move 'struct module_signature' to UAPI
From: Sami Tolvanen @ 2026-03-26 20:06 UTC (permalink / raw)
To: David Howells, David Woodhouse, Luis Chamberlain, Petr Pavlu,
Daniel Gomez, Aaron Tomlin, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
Mimi Zohar, Roberto Sassu, Dmitry Kasatkin, Eric Snowberg,
Paul Moore, James Morris, Serge E. Hallyn, Nathan Chancellor,
Nicolas Schier, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, John Fastabend, KP Singh, Stanislav Fomichev,
Hao Luo, Jiri Olsa, Shuah Khan, Thomas Weißschuh
Cc: keyrings, linux-kernel, linux-modules, linux-s390,
linux-integrity, linux-security-module, linux-kbuild, bpf,
linux-kselftest
In-Reply-To: <20260305-module-signature-uapi-v3-0-92f45ea6028c@linutronix.de>
On Thu, 05 Mar 2026 10:31:36 +0100, Thomas Weißschuh wrote:
> This structure definition is used outside the kernel proper.
> For example in kmod and the kernel build environment.
>
> To allow reuse, move it to a new UAPI header.
>
> While it is not a true UAPI, it is a common practice to have
> non-UAPI interface definitions in the kernel's UAPI headers.
>
> [...]
Applied to modules-next, thanks!
[1/8] extract-cert: drop unused definition of PKEY_ID_PKCS7
commit: 137676d4482d8b8d755890b4ed29fe8223661d20
[2/8] module: Drop unused signature types
commit: 8988913aacee82e5401bf3b96839731982dcbde7
[3/8] module: Give 'enum pkey_id_type' a more specific name
commit: acd87264af525dba6e9355310e8acdf066a5f6b5
[4/8] module: Give MODULE_SIG_STRING a more descriptive name
commit: 2ae4ea2d9aaf25cb74fbc23450b1b8f0a5b7aa89
[5/8] module: Move 'struct module_signature' to UAPI
commit: f9909cf0a2dcc9e99377f3fcc965ccd93e518e34
[6/8] tools uapi headers: add linux/module_signature.h
commit: d2d7561dc656748f592cc34d34bf5db8d5c67f7b
[7/8] sign-file: use 'struct module_signature' from the UAPI headers
commit: e340db306c3bb85877490f33a78eb80549ac43a7
[8/8] selftests/bpf: verify_pkcs7_sig: Use 'struct module_signature' from the UAPI headers
commit: 55722b3f80377103bac6ac748554129108c75651
Best regards,
Sami
^ permalink raw reply
* Re: [PATCH v4 0/8] scalable symbol flags with __kflagstab
From: Petr Pavlu @ 2026-03-25 12:49 UTC (permalink / raw)
To: Siddharth Nayyar
Cc: Luis Chamberlain, Daniel Gomez, Sami Tolvanen, Aaron Tomlin,
Arnd Bergmann, Nathan Chancellor, Nicolas Schier, Jonathan Corbet,
Shuah Khan, linux-modules, linux-kernel, linux-arch, linux-kbuild,
linux-doc, maennich, gprocida
In-Reply-To: <20260305-kflagstab-v4-0-6a76bf8b83c7@google.com>
On 3/5/26 5:55 PM, Siddharth Nayyar wrote:
> This patch series implements a mechanism for scalable exported symbol
> flags using a separate section called __kflagstab. The series introduces
> __kflagstab support, removes *_gpl sections in favor of a GPL flag,
> simplifies symbol resolution during module loading.
I noticed that the series has a bisecting issue. The module loader
doesn't see any GPL-only exports after patch #4. I think you'll need to
squash patches #4 and #5 to fix this. Alternatively, the patches could
be swapped, with the caveat that GPL-only symbols would lose their GPL
property for one commit.
Nit: Please use simply the "module" prefix in commit subjects:
#1: module: define ksym_flags enumeration to represent kernel symbol flags
#2: module: add kflagstab section to vmlinux and modules
#4: module: use kflagstab instead of *_gpl sections
#6: module: deprecate usage of *_gpl sections
#7: module: remove *_gpl sections from vmlinux and modules
The changes look otherwise ok to me. With the above fixed, feel free to
add:
Reviewed-by: Petr Pavlu <petr.pavlu@suse.com>
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH] module/kallsyms: sort function symbols and use binary search
From: Stanislaw Gruszka @ 2026-03-25 10:02 UTC (permalink / raw)
To: Petr Pavlu
Cc: linux-modules, Sami Tolvanen, Luis Chamberlain, linux-kernel,
linux-trace-kernel, live-patching, Daniel Gomez, Aaron Tomlin,
Steven Rostedt, Masami Hiramatsu, Jordan Rome, Viktor Malik
In-Reply-To: <20260325082648.GA18968@wp.pl>
On Wed, Mar 25, 2026 at 09:26:56AM +0100, Stanislaw Gruszka wrote:
> On Tue, Mar 24, 2026 at 05:00:19PM +0100, Petr Pavlu wrote:
> > On 3/24/26 1:53 PM, Stanislaw Gruszka wrote:
> > > Hi,
> > >
> > > On Mon, Mar 23, 2026 at 02:06:43PM +0100, Petr Pavlu wrote:
> > >> On 3/17/26 12:04 PM, Stanislaw Gruszka wrote:
> > >>> Module symbol lookup via find_kallsyms_symbol() performs a linear scan
> > >>> over the entire symtab when resolving an address. The number of symbols
> > >>> in module symtabs has grown over the years, largely due to additional
> > >>> metadata in non-standard sections, making this lookup very slow.
> > >>>
> > >>> Improve this by separating function symbols during module load, placing
> > >>> them at the beginning of the symtab, sorting them by address, and using
> > >>> binary search when resolving addresses in module text.
> > >>
> > >> Doesn't considering only function symbols break the expected behavior
> > >> with CONFIG_KALLSYMS_ALL=y. For instance, when using kdb, is it still
> > >> able to see all symbols in a module? The module loader should be remain
> > >> consistent with the main kallsyms code regarding which symbols can be
> > >> looked up.
> > >
> > > We already have a CONFIG_KALLSYMS_ALL=y inconsistency between kernel and
> > > module symbol lookup, independent of this patch. find_kallsyms_symbol()
> > > restricts the search to MOD_TEXT (or MOD_INIT_TEXT) address ranges, so
> > > it cannot resolve data or rodata symbols.
> >
> > My understanding is that find_kallsyms_symbol() can identify all symbols
> > in a module by their addresses. However, the issue I see with
> > MOD_TEXT/MOD_INIT_TEXT is that the function may incorrectly calculate
> > the size of symbols that are not within these ranges, which is a bug
> > that should be fixed.
>
> You are right, I misinterpreted the code:
>
> if (within_module_init(addr, mod))
> mod_mem = &mod->mem[MOD_INIT_TEXT];
> else
> mod_mem = &mod->mem[MOD_TEXT];
>
> nextval = (unsigned long)mod_mem->base + mod_mem->size;
>
> bestval = kallsyms_symbol_value(&kallsyms->symtab[best]);
>
> For best = 0, bestval is also 0 as it comes from the ELF null symbol.
>
> > A test using kdb confirms that non-text symbols can be found by their
> > addresses. The following shows the current behavior with 7.0-rc5 when
> > printing a module parameter in mlx4_en:
> >
> > [1]kdb> mds __param_arr_num_vfs
> > 0xffffffffc1209f20 0000000100000003 ........
> > 0xffffffffc1209f28 ffffffffc0fbf07c [mlx4_core]num_vfs_argc
> > 0xffffffffc1209f30 ffffffff8844bba0 param_ops_byte
> > 0xffffffffc1209f38 ffffffffc0fbf080 [mlx4_core]num_vfs
> > 0xffffffffc1209f40 000000785f69736d msi_x...
> > 0xffffffffc1209f48 656c5f6775626564 debug_le
> > 0xffffffffc1209f50 00000000006c6576 vel.....
> > 0xffffffffc1209f58 0000000000000000 ........
> >
> > .. and the behavior with the proposed patch:
> >
> > [1]kdb> mds __param_arr_num_vfs
> > 0xffffffffc1077f20 0000000100000003 ........
> > 0xffffffffc1077f28 ffffffffc104707c |p......
> > 0xffffffffc1077f30 ffffffffb4a4bba0 param_ops_byte
> > 0xffffffffc1077f38 ffffffffc1047080 .p......
> > 0xffffffffc1077f40 000000785f69736d msi_x...
> > 0xffffffffc1077f48 656c5f6775626564 debug_le
> > 0xffffffffc1077f50 00000000006c6576 vel.....
> > 0xffffffffc1077f58 0000000000000000 ........
>
> Thanks for testing and pointing this out. Patch indeed breaks
> the CONFIG_KALLSYMS_ALL case.
>
> I think, possible fix would be to track the relevant sections in
> __layout_sections() and use defined symbols from those sections,
> instead of just function symbols.
I considered sorting data symbols as well, but this is nontrivial,
it is difficult to reliably distinguish real data sections from metadata
sections containing symbols we do not want to include.
An alternative approach is to check the module memory type and fall back to
a linear search for ranges other than MOD_TEXT. This approach would also
fix the incorrect nextval/size problem.
Regards
Stanislaw
^ permalink raw reply
* Re: [PATCH] module/kallsyms: sort function symbols and use binary search
From: Stanislaw Gruszka @ 2026-03-25 8:26 UTC (permalink / raw)
To: Petr Pavlu
Cc: linux-modules, Sami Tolvanen, Luis Chamberlain, linux-kernel,
linux-trace-kernel, live-patching, Daniel Gomez, Aaron Tomlin,
Steven Rostedt, Masami Hiramatsu, Jordan Rome, Viktor Malik
In-Reply-To: <282574df-7689-4677-929b-b844e7201bd5@suse.com>
On Tue, Mar 24, 2026 at 05:00:19PM +0100, Petr Pavlu wrote:
> On 3/24/26 1:53 PM, Stanislaw Gruszka wrote:
> > Hi,
> >
> > On Mon, Mar 23, 2026 at 02:06:43PM +0100, Petr Pavlu wrote:
> >> On 3/17/26 12:04 PM, Stanislaw Gruszka wrote:
> >>> Module symbol lookup via find_kallsyms_symbol() performs a linear scan
> >>> over the entire symtab when resolving an address. The number of symbols
> >>> in module symtabs has grown over the years, largely due to additional
> >>> metadata in non-standard sections, making this lookup very slow.
> >>>
> >>> Improve this by separating function symbols during module load, placing
> >>> them at the beginning of the symtab, sorting them by address, and using
> >>> binary search when resolving addresses in module text.
> >>
> >> Doesn't considering only function symbols break the expected behavior
> >> with CONFIG_KALLSYMS_ALL=y. For instance, when using kdb, is it still
> >> able to see all symbols in a module? The module loader should be remain
> >> consistent with the main kallsyms code regarding which symbols can be
> >> looked up.
> >
> > We already have a CONFIG_KALLSYMS_ALL=y inconsistency between kernel and
> > module symbol lookup, independent of this patch. find_kallsyms_symbol()
> > restricts the search to MOD_TEXT (or MOD_INIT_TEXT) address ranges, so
> > it cannot resolve data or rodata symbols.
>
> My understanding is that find_kallsyms_symbol() can identify all symbols
> in a module by their addresses. However, the issue I see with
> MOD_TEXT/MOD_INIT_TEXT is that the function may incorrectly calculate
> the size of symbols that are not within these ranges, which is a bug
> that should be fixed.
You are right, I misinterpreted the code:
if (within_module_init(addr, mod))
mod_mem = &mod->mem[MOD_INIT_TEXT];
else
mod_mem = &mod->mem[MOD_TEXT];
nextval = (unsigned long)mod_mem->base + mod_mem->size;
bestval = kallsyms_symbol_value(&kallsyms->symtab[best]);
For best = 0, bestval is also 0 as it comes from the ELF null symbol.
> A test using kdb confirms that non-text symbols can be found by their
> addresses. The following shows the current behavior with 7.0-rc5 when
> printing a module parameter in mlx4_en:
>
> [1]kdb> mds __param_arr_num_vfs
> 0xffffffffc1209f20 0000000100000003 ........
> 0xffffffffc1209f28 ffffffffc0fbf07c [mlx4_core]num_vfs_argc
> 0xffffffffc1209f30 ffffffff8844bba0 param_ops_byte
> 0xffffffffc1209f38 ffffffffc0fbf080 [mlx4_core]num_vfs
> 0xffffffffc1209f40 000000785f69736d msi_x...
> 0xffffffffc1209f48 656c5f6775626564 debug_le
> 0xffffffffc1209f50 00000000006c6576 vel.....
> 0xffffffffc1209f58 0000000000000000 ........
>
> .. and the behavior with the proposed patch:
>
> [1]kdb> mds __param_arr_num_vfs
> 0xffffffffc1077f20 0000000100000003 ........
> 0xffffffffc1077f28 ffffffffc104707c |p......
> 0xffffffffc1077f30 ffffffffb4a4bba0 param_ops_byte
> 0xffffffffc1077f38 ffffffffc1047080 .p......
> 0xffffffffc1077f40 000000785f69736d msi_x...
> 0xffffffffc1077f48 656c5f6775626564 debug_le
> 0xffffffffc1077f50 00000000006c6576 vel.....
> 0xffffffffc1077f58 0000000000000000 ........
Thanks for testing and pointing this out. Patch indeed breaks
the CONFIG_KALLSYMS_ALL case.
I think, possible fix would be to track the relevant sections in
__layout_sections() and use defined symbols from those sections,
instead of just function symbols.
Regards
Stanislaw
^ permalink raw reply
* Re: [RFC] btf: split core BTF parsing out of BPF subsystem into kernel/btf/
From: bot+bpf-ci @ 2026-03-25 2:15 UTC (permalink / raw)
To: sashal, sashal
Cc: James.Bottomley, adobriyan, akpm, alexei.starovoitov, ast, bp,
bpf, corbet, dave.hansen, davidgow, deller, geert, gregkh, hpa,
jgross, jpoimboe, kees, laurent.pinchart, linux-doc, linux-kbuild,
linux-kernel, linux-modules, masahiroy, mcgrof, mingo, nathan,
nsc, peterz, petr.pavlu, pmladek, rdunlap, rostedt, tglx, vbabka,
x86, ast, andrii, daniel, martin.lau, eddyz87, yonghong.song, clm,
ihor.solodrai
In-Reply-To: <20260325011853.657295-1-sashal@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2607 bytes --]
> diff --git a/kernel/btf/bpf.c b/kernel/btf/bpf.c
> new file mode 100644
> index 0000000000000..ef973771bb2ab
> --- /dev/null
> +++ b/kernel/btf/bpf.c
[ ... ]
> +struct btf_struct_ops_tab {
> + u32 cnt;
> + u32 capacity;
> + struct bpf_struct_ops_desc ops[];
> +};
> + u32 id;
> + struct rcu_head rcu;
> + struct btf_kfunc_set_tab *kfunc_set_tab;
> + struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
> + struct btf_struct_metas *struct_meta_tab;
> + struct btf_struct_ops_tab *struct_ops_tab;
> +s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
These six declarations between the closing brace of struct
btf_struct_ops_tab and bpf_find_btf_id() appear to be orphaned
fields from the original struct btf definition that were left
behind during the split.
They are parsed as file-scope global variable definitions. With
-fno-common, the name "id" in particular is likely to collide at
link time. Were these meant to be removed here, given that they
already appear correctly inside struct btf in kernel/btf/btf.h
under the CONFIG_BPF_SYSCALL guard?
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 93f356d2b3d95..0009a7ac78462 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -395,14 +395,23 @@ config DEBUG_INFO_SPLIT
[ ... ]
> +config BTF
> + bool "BTF type information support"
> + depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
> + depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
> + depends on PAHOLE_VERSION >= 122
> + # pahole uses elfutils, which does not have support for Hexagon relocations
> + depends on !HEXAGON
[ ... ]
> +config DEBUG_INFO_BTF
> + bool "Generate BTF type information"
> + select BTF
The old DEBUG_INFO_BTF had its own depends on lines:
depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
depends on BPF_SYSCALL
depends on PAHOLE_VERSION >= 122
depends on !HEXAGON
Those constraints now live only on CONFIG_BTF, and DEBUG_INFO_BTF
uses "select BTF" with no depends of its own. Since select bypasses
the selected symbol's depends-on checks, DEBUG_INFO_BTF is now
visible and selectable even when BTF's dependencies are not met
(e.g. on Hexagon, or with PAHOLE_VERSION < 122).
Should DEBUG_INFO_BTF use "depends on BTF" instead of (or in
addition to) "select BTF", so that it inherits the same
constraints?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23520643445
^ permalink raw reply
* [RFC] btf: split core BTF parsing out of BPF subsystem into kernel/btf/
From: Sasha Levin @ 2026-03-25 1:18 UTC (permalink / raw)
To: sashal
Cc: James.Bottomley, adobriyan, akpm, alexei.starovoitov, ast, bp,
bpf, corbet, dave.hansen, davidgow, deller, geert, gregkh, hpa,
jgross, jpoimboe, kees, laurent.pinchart, linux-doc, linux-kbuild,
linux-kernel, linux-modules, masahiroy, mcgrof, mingo, nathan,
nsc, peterz, petr.pavlu, pmladek, rdunlap, rostedt, tglx, vbabka,
x86
In-Reply-To: <acLbnMAPrHCpoIpr@laps>
Move BTF type format parsing and inspection code out of the BPF
subsystem into its own kernel/btf/ directory, separating core BTF
functionality from BPF-specific extensions.
CONFIG_DEBUG_INFO_BTF currently depends on CONFIG_BPF_SYSCALL, which
prevents embedded, automotive, and safety-critical environments from
using BTF. These platforms often disable BPF for security and size
reasons but would benefit from BTF type information for crash
diagnostics and debugging.
Introduce CONFIG_BTF for the core BTF runtime support (type parsing,
lookup, and display) which can be enabled independently of
CONFIG_BPF_SYSCALL, and remove the BPF_SYSCALL dependency from
CONFIG_DEBUG_INFO_BTF. The code is split into:
kernel/btf/btf.c - Core BTF: parsing, verification, type lookup,
display infrastructure. No BPF dependencies.
kernel/btf/bpf.c - BPF extensions: kfunc registration, struct_ops,
verifier integration, FD interface, module BTF.
kernel/btf/btf.h - Internal shared header.
When CONFIG_BPF_SYSCALL=n, only btf.c is compiled, providing the
minimal BTF API. When CONFIG_BPF_SYSCALL=y, both files are compiled
and bpf.c overrides weak symbols in btf.c for BPF-specific cleanup
(IDR management, RCU-deferred freeing, kfunc/struct_ops tab cleanup).
No functional change when CONFIG_BPF_SYSCALL=y.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/btf.h | 80 +-
kernel/Makefile | 1 +
kernel/bpf/Makefile | 2 +-
kernel/btf/Makefile | 4 +
kernel/{bpf/btf.c => btf/bpf.c} | 6386 ++++---------------------------
kernel/btf/btf.c | 4803 +++++++++++++++++++++++
kernel/btf/btf.h | 135 +
lib/Kconfig.debug | 15 +-
8 files changed, 5796 insertions(+), 5630 deletions(-)
create mode 100644 kernel/btf/Makefile
rename kernel/{bpf/btf.c => btf/bpf.c} (50%)
create mode 100644 kernel/btf/btf.c
create mode 100644 kernel/btf/btf.h
diff --git a/include/linux/btf.h b/include/linux/btf.h
index 48108471c5b13..d4d64878324ff 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -139,17 +139,11 @@ struct btf_struct_metas {
struct btf_struct_meta types[];
};
-extern const struct file_operations btf_fops;
-
const char *btf_get_name(const struct btf *btf);
void btf_get(struct btf *btf);
void btf_put(struct btf *btf);
const struct btf_header *btf_header(const struct btf *btf);
-int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
-struct btf *btf_get_by_fd(int fd);
-int btf_get_info_by_fd(const struct btf *btf,
- const union bpf_attr *attr,
- union bpf_attr __user *uattr);
+
/* Figure out the size of a type_id. If type_id is a modifier
* (e.g. const), it will be resolved to find out the type with size.
*
@@ -213,12 +207,9 @@ int btf_type_seq_show_flags(const struct btf *btf, u32 type_id, void *obj,
int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
char *buf, int len, u64 flags);
-int btf_get_fd_by_id(u32 id);
-u32 btf_obj_id(const struct btf *btf);
bool btf_is_kernel(const struct btf *btf);
bool btf_is_module(const struct btf *btf);
bool btf_is_vmlinux(const struct btf *btf);
-struct module *btf_try_get_module(const struct btf *btf);
u32 btf_nr_types(const struct btf *btf);
u32 btf_named_start_id(const struct btf *btf, bool own);
struct btf *btf_base_btf(const struct btf *btf);
@@ -228,12 +219,8 @@ bool btf_type_is_primitive(const struct btf_type *t);
bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
const struct btf_member *m,
u32 expected_offset, u32 expected_size);
-struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, u32 value_size);
-int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
bool btf_type_is_void(const struct btf_type *t);
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
-s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
u32 id, u32 *res_id);
const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
@@ -245,6 +232,20 @@ btf_resolve_size(const struct btf *btf, const struct btf_type *type,
u32 *type_size);
const char *btf_type_str(const struct btf_type *t);
+/* BPF-specific declarations */
+extern const struct file_operations btf_fops;
+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
+struct btf *btf_get_by_fd(int fd);
+int btf_get_info_by_fd(const struct btf *btf,
+ const union bpf_attr *attr,
+ union bpf_attr __user *uattr);
+int btf_get_fd_by_id(u32 id);
+u32 btf_obj_id(const struct btf *btf);
+struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, u32 value_size);
+int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
+s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
+
#define for_each_member(i, struct_type, member) \
for (i = 0, member = btf_type_member(struct_type); \
i < btf_type_vlen(struct_type); \
@@ -524,9 +525,6 @@ static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
bool btf_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix);
-int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
- u32 arg_no);
-u32 btf_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto, int off);
struct bpf_verifier_log;
@@ -564,16 +562,39 @@ struct btf_field_iter {
int vlen;
};
-#ifdef CONFIG_BPF_SYSCALL
+#ifdef CONFIG_BTF
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
+const char *btf_name_by_offset(const struct btf *btf, u32 offset);
+const char *btf_str_by_offset(const struct btf *btf, u32 offset);
+#else
+static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
+ u32 type_id)
+{
+ return NULL;
+}
+
+static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
+{
+}
+
+static inline const char *btf_name_by_offset(const struct btf *btf,
+ u32 offset)
+{
+ return NULL;
+}
+#endif /* CONFIG_BTF */
+
+#ifdef CONFIG_BPF_SYSCALL
+struct module *btf_try_get_module(const struct btf *btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **map_ids);
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind);
__u32 *btf_field_iter_next(struct btf_field_iter *it);
-const char *btf_name_by_offset(const struct btf *btf, u32 offset);
-const char *btf_str_by_offset(const struct btf *btf, u32 offset);
+int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
+ u32 arg_no);
+u32 btf_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto, int off);
struct btf *btf_parse_vmlinux(void);
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
@@ -606,17 +627,7 @@ static inline bool btf_type_is_struct_ptr(struct btf *btf, const struct btf_type
return btf_type_is_struct(t);
}
#else
-static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
- u32 type_id)
-{
- return NULL;
-}
-
-static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
-{
-}
-
-static inline int btf_relocate(void *log, struct btf *btf, const struct btf *base_btf,
+static inline int btf_relocate(struct btf *btf, const struct btf *base_btf,
__u32 **map_ids)
{
return -EOPNOTSUPP;
@@ -633,11 +644,6 @@ static inline __u32 *btf_field_iter_next(struct btf_field_iter *it)
return NULL;
}
-static inline const char *btf_name_by_offset(const struct btf *btf,
- u32 offset)
-{
- return NULL;
-}
static inline u32 *btf_kfunc_id_set_contains(const struct btf *btf,
u32 kfunc_btf_id,
struct bpf_prog *prog)
@@ -683,5 +689,5 @@ static inline int btf_check_iter_arg(struct btf *btf, const struct btf_type *fun
{
return -EOPNOTSUPP;
}
-#endif
+#endif /* CONFIG_BPF_SYSCALL */
#endif
diff --git a/kernel/Makefile b/kernel/Makefile
index 6785982013dce..d09d7f102c335 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -121,6 +121,7 @@ obj-$(CONFIG_RETHOOK) += trace/
obj-$(CONFIG_IRQ_WORK) += irq_work.o
obj-$(CONFIG_CPU_PM) += cpu_pm.o
obj-$(CONFIG_BPF) += bpf/
+obj-$(CONFIG_BTF) += btf/
obj-$(CONFIG_KCSAN) += kcsan/
obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o
obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 79cf22860a99b..37d7ebf4808f3 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -14,7 +14,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
-obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o
+obj-$(CONFIG_BPF_SYSCALL) += memalloc.o rqspinlock.o stream.o
ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)
obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o
endif
diff --git a/kernel/btf/Makefile b/kernel/btf/Makefile
new file mode 100644
index 0000000000000..e0365d5c7e9bd
--- /dev/null
+++ b/kernel/btf/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_BTF) += btf.o
+obj-$(CONFIG_BPF_SYSCALL) += bpf.o
diff --git a/kernel/bpf/btf.c b/kernel/btf/bpf.c
similarity index 50%
rename from kernel/bpf/btf.c
rename to kernel/btf/bpf.c
index 71f9143fe90f3..986d29f4bc3ba 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/btf/bpf.c
@@ -1,33 +1,35 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2018 Facebook */
+/* BPF-specific BTF extensions */
#include <uapi/linux/btf.h>
#include <uapi/linux/bpf.h>
-#include <uapi/linux/bpf_perf_event.h>
#include <uapi/linux/types.h>
-#include <linux/seq_file.h>
#include <linux/compiler.h>
#include <linux/ctype.h>
#include <linux/errno.h>
#include <linux/slab.h>
-#include <linux/anon_inodes.h>
#include <linux/file.h>
-#include <linux/uaccess.h>
#include <linux/kernel.h>
-#include <linux/idr.h>
#include <linux/sort.h>
+#include <linux/seq_file.h>
#include <linux/bpf_verifier.h>
#include <linux/btf.h>
#include <linux/btf_ids.h>
+#include <linux/bsearch.h>
+#include <linux/string.h>
+#include <linux/overflow.h>
+#include <linux/kobject.h>
+
+#include <uapi/linux/bpf_perf_event.h>
+#include <linux/anon_inodes.h>
+#include <linux/uaccess.h>
+#include <linux/idr.h>
#include <linux/bpf.h>
#include <linux/bpf_lsm.h>
#include <linux/skmsg.h>
#include <linux/perf_event.h>
-#include <linux/bsearch.h>
-#include <linux/kobject.h>
-#include <linux/string.h>
#include <linux/sysfs.h>
-#include <linux/overflow.h>
#include <net/netfilter/nf_bpf_link.h>
@@ -35,173 +37,7 @@
#include <net/xdp.h>
#include "../tools/lib/bpf/relo_core.h"
-/* BTF (BPF Type Format) is the meta data format which describes
- * the data types of BPF program/map. Hence, it basically focus
- * on the C programming language which the modern BPF is primary
- * using.
- *
- * ELF Section:
- * ~~~~~~~~~~~
- * The BTF data is stored under the ".BTF" ELF section
- *
- * struct btf_type:
- * ~~~~~~~~~~~~~~~
- * Each 'struct btf_type' object describes a C data type.
- * Depending on the type it is describing, a 'struct btf_type'
- * object may be followed by more data. F.e.
- * To describe an array, 'struct btf_type' is followed by
- * 'struct btf_array'.
- *
- * 'struct btf_type' and any extra data following it are
- * 4 bytes aligned.
- *
- * Type section:
- * ~~~~~~~~~~~~~
- * The BTF type section contains a list of 'struct btf_type' objects.
- * Each one describes a C type. Recall from the above section
- * that a 'struct btf_type' object could be immediately followed by extra
- * data in order to describe some particular C types.
- *
- * type_id:
- * ~~~~~~~
- * Each btf_type object is identified by a type_id. The type_id
- * is implicitly implied by the location of the btf_type object in
- * the BTF type section. The first one has type_id 1. The second
- * one has type_id 2...etc. Hence, an earlier btf_type has
- * a smaller type_id.
- *
- * A btf_type object may refer to another btf_type object by using
- * type_id (i.e. the "type" in the "struct btf_type").
- *
- * NOTE that we cannot assume any reference-order.
- * A btf_type object can refer to an earlier btf_type object
- * but it can also refer to a later btf_type object.
- *
- * For example, to describe "const void *". A btf_type
- * object describing "const" may refer to another btf_type
- * object describing "void *". This type-reference is done
- * by specifying type_id:
- *
- * [1] CONST (anon) type_id=2
- * [2] PTR (anon) type_id=0
- *
- * The above is the btf_verifier debug log:
- * - Each line started with "[?]" is a btf_type object
- * - [?] is the type_id of the btf_type object.
- * - CONST/PTR is the BTF_KIND_XXX
- * - "(anon)" is the name of the type. It just
- * happens that CONST and PTR has no name.
- * - type_id=XXX is the 'u32 type' in btf_type
- *
- * NOTE: "void" has type_id 0
- *
- * String section:
- * ~~~~~~~~~~~~~~
- * The BTF string section contains the names used by the type section.
- * Each string is referred by an "offset" from the beginning of the
- * string section.
- *
- * Each string is '\0' terminated.
- *
- * The first character in the string section must be '\0'
- * which is used to mean 'anonymous'. Some btf_type may not
- * have a name.
- */
-
-/* BTF verification:
- *
- * To verify BTF data, two passes are needed.
- *
- * Pass #1
- * ~~~~~~~
- * The first pass is to collect all btf_type objects to
- * an array: "btf->types".
- *
- * Depending on the C type that a btf_type is describing,
- * a btf_type may be followed by extra data. We don't know
- * how many btf_type is there, and more importantly we don't
- * know where each btf_type is located in the type section.
- *
- * Without knowing the location of each type_id, most verifications
- * cannot be done. e.g. an earlier btf_type may refer to a later
- * btf_type (recall the "const void *" above), so we cannot
- * check this type-reference in the first pass.
- *
- * In the first pass, it still does some verifications (e.g.
- * checking the name is a valid offset to the string section).
- *
- * Pass #2
- * ~~~~~~~
- * The main focus is to resolve a btf_type that is referring
- * to another type.
- *
- * We have to ensure the referring type:
- * 1) does exist in the BTF (i.e. in btf->types[])
- * 2) does not cause a loop:
- * struct A {
- * struct B b;
- * };
- *
- * struct B {
- * struct A a;
- * };
- *
- * btf_type_needs_resolve() decides if a btf_type needs
- * to be resolved.
- *
- * The needs_resolve type implements the "resolve()" ops which
- * essentially does a DFS and detects backedge.
- *
- * During resolve (or DFS), different C types have different
- * "RESOLVED" conditions.
- *
- * When resolving a BTF_KIND_STRUCT, we need to resolve all its
- * members because a member is always referring to another
- * type. A struct's member can be treated as "RESOLVED" if
- * it is referring to a BTF_KIND_PTR. Otherwise, the
- * following valid C struct would be rejected:
- *
- * struct A {
- * int m;
- * struct A *a;
- * };
- *
- * When resolving a BTF_KIND_PTR, it needs to keep resolving if
- * it is referring to another BTF_KIND_PTR. Otherwise, we cannot
- * detect a pointer loop, e.g.:
- * BTF_KIND_CONST -> BTF_KIND_PTR -> BTF_KIND_CONST -> BTF_KIND_PTR +
- * ^ |
- * +-----------------------------------------+
- *
- */
-
-#define BITS_PER_U128 (sizeof(u64) * BITS_PER_BYTE * 2)
-#define BITS_PER_BYTE_MASK (BITS_PER_BYTE - 1)
-#define BITS_PER_BYTE_MASKED(bits) ((bits) & BITS_PER_BYTE_MASK)
-#define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
-#define BITS_ROUNDUP_BYTES(bits) \
- (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
-
-#define BTF_INFO_MASK 0x9f00ffff
-#define BTF_INT_MASK 0x0fffffff
-#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
-#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
-
-/* 16MB for 64k structs and each has 16 members and
- * a few MB spaces for the string section.
- * The hard limit is S32_MAX.
- */
-#define BTF_MAX_SIZE (16 * 1024 * 1024)
-
-#define for_each_member_from(i, from, struct_type, member) \
- for (i = from, member = btf_type_member(struct_type) + from; \
- i < btf_type_vlen(struct_type); \
- i++, member++)
-
-#define for_each_vsi_from(i, from, struct_type, member) \
- for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
- i < btf_type_vlen(struct_type); \
- i++, member++)
+#include "btf.h"
DEFINE_IDR(btf_idr);
DEFINE_SPINLOCK(btf_idr_lock);
@@ -250,431 +86,12 @@ struct btf_struct_ops_tab {
u32 capacity;
struct bpf_struct_ops_desc ops[];
};
-
-struct btf {
- void *data;
- struct btf_type **types;
- u32 *resolved_ids;
- u32 *resolved_sizes;
- const char *strings;
- void *nohdr_data;
- struct btf_header hdr;
- u32 nr_types; /* includes VOID for base BTF */
- u32 named_start_id;
- u32 types_size;
- u32 data_size;
- refcount_t refcnt;
u32 id;
struct rcu_head rcu;
struct btf_kfunc_set_tab *kfunc_set_tab;
struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
struct btf_struct_metas *struct_meta_tab;
struct btf_struct_ops_tab *struct_ops_tab;
-
- /* split BTF support */
- struct btf *base_btf;
- u32 start_id; /* first type ID in this BTF (0 for base BTF) */
- u32 start_str_off; /* first string offset (0 for base BTF) */
- char name[MODULE_NAME_LEN];
- bool kernel_btf;
- __u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
-};
-
-enum verifier_phase {
- CHECK_META,
- CHECK_TYPE,
-};
-
-struct resolve_vertex {
- const struct btf_type *t;
- u32 type_id;
- u16 next_member;
-};
-
-enum visit_state {
- NOT_VISITED,
- VISITED,
- RESOLVED,
-};
-
-enum resolve_mode {
- RESOLVE_TBD, /* To Be Determined */
- RESOLVE_PTR, /* Resolving for Pointer */
- RESOLVE_STRUCT_OR_ARRAY, /* Resolving for struct/union
- * or array
- */
-};
-
-#define MAX_RESOLVE_DEPTH 32
-
-struct btf_sec_info {
- u32 off;
- u32 len;
-};
-
-struct btf_verifier_env {
- struct btf *btf;
- u8 *visit_states;
- struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
- struct bpf_verifier_log log;
- u32 log_type_id;
- u32 top_stack;
- enum verifier_phase phase;
- enum resolve_mode resolve_mode;
-};
-
-static const char * const btf_kind_str[NR_BTF_KINDS] = {
- [BTF_KIND_UNKN] = "UNKNOWN",
- [BTF_KIND_INT] = "INT",
- [BTF_KIND_PTR] = "PTR",
- [BTF_KIND_ARRAY] = "ARRAY",
- [BTF_KIND_STRUCT] = "STRUCT",
- [BTF_KIND_UNION] = "UNION",
- [BTF_KIND_ENUM] = "ENUM",
- [BTF_KIND_FWD] = "FWD",
- [BTF_KIND_TYPEDEF] = "TYPEDEF",
- [BTF_KIND_VOLATILE] = "VOLATILE",
- [BTF_KIND_CONST] = "CONST",
- [BTF_KIND_RESTRICT] = "RESTRICT",
- [BTF_KIND_FUNC] = "FUNC",
- [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO",
- [BTF_KIND_VAR] = "VAR",
- [BTF_KIND_DATASEC] = "DATASEC",
- [BTF_KIND_FLOAT] = "FLOAT",
- [BTF_KIND_DECL_TAG] = "DECL_TAG",
- [BTF_KIND_TYPE_TAG] = "TYPE_TAG",
- [BTF_KIND_ENUM64] = "ENUM64",
-};
-
-const char *btf_type_str(const struct btf_type *t)
-{
- return btf_kind_str[BTF_INFO_KIND(t->info)];
-}
-
-/* Chunk size we use in safe copy of data to be shown. */
-#define BTF_SHOW_OBJ_SAFE_SIZE 32
-
-/*
- * This is the maximum size of a base type value (equivalent to a
- * 128-bit int); if we are at the end of our safe buffer and have
- * less than 16 bytes space we can't be assured of being able
- * to copy the next type safely, so in such cases we will initiate
- * a new copy.
- */
-#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
-
-/* Type name size */
-#define BTF_SHOW_NAME_SIZE 80
-
-/*
- * The suffix of a type that indicates it cannot alias another type when
- * comparing BTF IDs for kfunc invocations.
- */
-#define NOCAST_ALIAS_SUFFIX "___init"
-
-/*
- * Common data to all BTF show operations. Private show functions can add
- * their own data to a structure containing a struct btf_show and consult it
- * in the show callback. See btf_type_show() below.
- *
- * One challenge with showing nested data is we want to skip 0-valued
- * data, but in order to figure out whether a nested object is all zeros
- * we need to walk through it. As a result, we need to make two passes
- * when handling structs, unions and arrays; the first path simply looks
- * for nonzero data, while the second actually does the display. The first
- * pass is signalled by show->state.depth_check being set, and if we
- * encounter a non-zero value we set show->state.depth_to_show to
- * the depth at which we encountered it. When we have completed the
- * first pass, we will know if anything needs to be displayed if
- * depth_to_show > depth. See btf_[struct,array]_show() for the
- * implementation of this.
- *
- * Another problem is we want to ensure the data for display is safe to
- * access. To support this, the anonymous "struct {} obj" tracks the data
- * object and our safe copy of it. We copy portions of the data needed
- * to the object "copy" buffer, but because its size is limited to
- * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
- * traverse larger objects for display.
- *
- * The various data type show functions all start with a call to
- * btf_show_start_type() which returns a pointer to the safe copy
- * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
- * raw data itself). btf_show_obj_safe() is responsible for
- * using copy_from_kernel_nofault() to update the safe data if necessary
- * as we traverse the object's data. skbuff-like semantics are
- * used:
- *
- * - obj.head points to the start of the toplevel object for display
- * - obj.size is the size of the toplevel object
- * - obj.data points to the current point in the original data at
- * which our safe data starts. obj.data will advance as we copy
- * portions of the data.
- *
- * In most cases a single copy will suffice, but larger data structures
- * such as "struct task_struct" will require many copies. The logic in
- * btf_show_obj_safe() handles the logic that determines if a new
- * copy_from_kernel_nofault() is needed.
- */
-struct btf_show {
- u64 flags;
- void *target; /* target of show operation (seq file, buffer) */
- __printf(2, 0) void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
- const struct btf *btf;
- /* below are used during iteration */
- struct {
- u8 depth;
- u8 depth_to_show;
- u8 depth_check;
- u8 array_member:1,
- array_terminated:1;
- u16 array_encoding;
- u32 type_id;
- int status; /* non-zero for error */
- const struct btf_type *type;
- const struct btf_member *member;
- char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
- } state;
- struct {
- u32 size;
- void *head;
- void *data;
- u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
- } obj;
-};
-
-struct btf_kind_operations {
- s32 (*check_meta)(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left);
- int (*resolve)(struct btf_verifier_env *env,
- const struct resolve_vertex *v);
- int (*check_member)(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type);
- int (*check_kflag_member)(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type);
- void (*log_details)(struct btf_verifier_env *env,
- const struct btf_type *t);
- void (*show)(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offsets,
- struct btf_show *show);
-};
-
-static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
-static struct btf_type btf_void;
-
-static int btf_resolve(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id);
-
-static int btf_func_check(struct btf_verifier_env *env,
- const struct btf_type *t);
-
-static bool btf_type_is_modifier(const struct btf_type *t)
-{
- /* Some of them is not strictly a C modifier
- * but they are grouped into the same bucket
- * for BTF concern:
- * A type (t) that refers to another
- * type through t->type AND its size cannot
- * be determined without following the t->type.
- *
- * ptr does not fall into this bucket
- * because its size is always sizeof(void *).
- */
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_TYPEDEF:
- case BTF_KIND_VOLATILE:
- case BTF_KIND_CONST:
- case BTF_KIND_RESTRICT:
- case BTF_KIND_TYPE_TAG:
- return true;
- }
-
- return false;
-}
-
-static int btf_start_id(const struct btf *btf)
-{
- return btf->start_id + (btf->base_btf ? 0 : 1);
-}
-
-bool btf_type_is_void(const struct btf_type *t)
-{
- return t == &btf_void;
-}
-
-static bool btf_type_is_datasec(const struct btf_type *t)
-{
- return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
-}
-
-static bool btf_type_is_decl_tag(const struct btf_type *t)
-{
- return BTF_INFO_KIND(t->info) == BTF_KIND_DECL_TAG;
-}
-
-static bool btf_type_nosize(const struct btf_type *t)
-{
- return btf_type_is_void(t) || btf_type_is_fwd(t) ||
- btf_type_is_func(t) || btf_type_is_func_proto(t) ||
- btf_type_is_decl_tag(t);
-}
-
-static bool btf_type_nosize_or_null(const struct btf_type *t)
-{
- return !t || btf_type_nosize(t);
-}
-
-static bool btf_type_is_decl_tag_target(const struct btf_type *t)
-{
- return btf_type_is_func(t) || btf_type_is_struct(t) ||
- btf_type_is_var(t) || btf_type_is_typedef(t);
-}
-
-bool btf_is_vmlinux(const struct btf *btf)
-{
- return btf->kernel_btf && !btf->base_btf;
-}
-
-u32 btf_nr_types(const struct btf *btf)
-{
- u32 total = 0;
-
- while (btf) {
- total += btf->nr_types;
- btf = btf->base_btf;
- }
-
- return total;
-}
-
-/*
- * Note that vmlinux and kernel module BTFs are always sorted
- * during the building phase.
- */
-static void btf_check_sorted(struct btf *btf)
-{
- u32 i, n, named_start_id = 0;
-
- n = btf_nr_types(btf);
- if (btf_is_vmlinux(btf)) {
- for (i = btf_start_id(btf); i < n; i++) {
- const struct btf_type *t = btf_type_by_id(btf, i);
- const char *n = btf_name_by_offset(btf, t->name_off);
-
- if (n[0] != '\0') {
- btf->named_start_id = i;
- return;
- }
- }
- return;
- }
-
- for (i = btf_start_id(btf) + 1; i < n; i++) {
- const struct btf_type *ta = btf_type_by_id(btf, i - 1);
- const struct btf_type *tb = btf_type_by_id(btf, i);
- const char *na = btf_name_by_offset(btf, ta->name_off);
- const char *nb = btf_name_by_offset(btf, tb->name_off);
-
- if (strcmp(na, nb) > 0)
- return;
-
- if (named_start_id == 0 && na[0] != '\0')
- named_start_id = i - 1;
- if (named_start_id == 0 && nb[0] != '\0')
- named_start_id = i;
- }
-
- if (named_start_id)
- btf->named_start_id = named_start_id;
-}
-
-/*
- * btf_named_start_id - Get the named starting ID for the BTF
- * @btf: Pointer to the target BTF object
- * @own: Flag indicating whether to query only the current BTF (true = current BTF only,
- * false = recursively traverse the base BTF chain)
- *
- * Return value rules:
- * 1. For a sorted btf, return its named_start_id
- * 2. Else for a split BTF, return its start_id
- * 3. Else for a base BTF, return 1
- */
-u32 btf_named_start_id(const struct btf *btf, bool own)
-{
- const struct btf *base_btf = btf;
-
- while (!own && base_btf->base_btf)
- base_btf = base_btf->base_btf;
-
- return base_btf->named_start_id ?: (base_btf->start_id ?: 1);
-}
-
-static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)
-{
- const struct btf_type *t;
- const char *tname;
- s32 l, r, m;
-
- l = btf_named_start_id(btf, true);
- r = btf_nr_types(btf) - 1;
- while (l <= r) {
- m = l + (r - l) / 2;
- t = btf_type_by_id(btf, m);
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) >= 0) {
- if (l == r)
- return r;
- r = m;
- } else {
- l = m + 1;
- }
- }
-
- return btf_nr_types(btf);
-}
-
-s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
-{
- const struct btf *base_btf = btf_base_btf(btf);
- const struct btf_type *t;
- const char *tname;
- s32 id, total;
-
- if (base_btf) {
- id = btf_find_by_name_kind(base_btf, name, kind);
- if (id > 0)
- return id;
- }
-
- total = btf_nr_types(btf);
- if (btf->named_start_id > 0 && name[0]) {
- id = btf_find_by_name_kind_bsearch(btf, name);
- for (; id < total; id++) {
- t = btf_type_by_id(btf, id);
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) != 0)
- return -ENOENT;
- if (BTF_INFO_KIND(t->info) == kind)
- return id;
- }
- } else {
- for (id = btf_start_id(btf); id < total; id++) {
- t = btf_type_by_id(btf, id);
- if (BTF_INFO_KIND(t->info) != kind)
- continue;
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) == 0)
- return id;
- }
- }
-
- return -ENOENT;
-}
-
s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
{
struct btf *btf;
@@ -719,4964 +136,976 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
return ret;
}
EXPORT_SYMBOL_GPL(bpf_find_btf_id);
-
-const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
- u32 id, u32 *res_id)
+static int btf_alloc_id(struct btf *btf)
{
- const struct btf_type *t = btf_type_by_id(btf, id);
+ int id;
- while (btf_type_is_modifier(t)) {
- id = t->type;
- t = btf_type_by_id(btf, t->type);
- }
+ idr_preload(GFP_KERNEL);
+ spin_lock_bh(&btf_idr_lock);
+ id = idr_alloc_cyclic(&btf_idr, btf, 1, INT_MAX, GFP_ATOMIC);
+ if (id > 0)
+ btf->id = id;
+ spin_unlock_bh(&btf_idr_lock);
+ idr_preload_end();
- if (res_id)
- *res_id = id;
+ if (WARN_ON_ONCE(!id))
+ return -ENOSPC;
- return t;
+ return id > 0 ? 0 : id;
}
-const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
- u32 id, u32 *res_id)
+static void btf_free_id(struct btf *btf)
{
- const struct btf_type *t;
-
- t = btf_type_skip_modifiers(btf, id, NULL);
- if (!btf_type_is_ptr(t))
- return NULL;
+ unsigned long flags;
- return btf_type_skip_modifiers(btf, t->type, res_id);
+ /*
+ * In map-in-map, calling map_delete_elem() on outer
+ * map will call bpf_map_put on the inner map.
+ * It will then eventually call btf_free_id()
+ * on the inner map. Some of the map_delete_elem()
+ * implementation may have irq disabled, so
+ * we need to use the _irqsave() version instead
+ * of the _bh() version.
+ */
+ spin_lock_irqsave(&btf_idr_lock, flags);
+ if (btf->id) {
+ idr_remove(&btf_idr, btf->id);
+ /*
+ * Clear the id here to make this function idempotent, since it will get
+ * called a couple of times for module BTFs: on module unload, and then
+ * the final btf_put(). btf_alloc_id() starts IDs with 1, so we can use
+ * 0 as sentinel value.
+ */
+ WRITE_ONCE(btf->id, 0);
+ }
+ spin_unlock_irqrestore(&btf_idr_lock, flags);
}
-const struct btf_type *btf_type_resolve_func_ptr(const struct btf *btf,
- u32 id, u32 *res_id)
+static void btf_free_kfunc_set_tab(struct btf *btf)
{
- const struct btf_type *ptype;
-
- ptype = btf_type_resolve_ptr(btf, id, res_id);
- if (ptype && btf_type_is_func_proto(ptype))
- return ptype;
+ struct btf_kfunc_set_tab *tab = btf->kfunc_set_tab;
+ int hook;
- return NULL;
+ if (!tab)
+ return;
+ for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
+ kfree(tab->sets[hook]);
+ kfree(tab);
+ btf->kfunc_set_tab = NULL;
}
-/* Types that act only as a source, not sink or intermediate
- * type when resolving.
- */
-static bool btf_type_is_resolve_source_only(const struct btf_type *t)
+static void btf_free_dtor_kfunc_tab(struct btf *btf)
{
- return btf_type_is_var(t) ||
- btf_type_is_decl_tag(t) ||
- btf_type_is_datasec(t);
+ struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
+
+ if (!tab)
+ return;
+ kfree(tab);
+ btf->dtor_kfunc_tab = NULL;
}
-/* What types need to be resolved?
- *
- * btf_type_is_modifier() is an obvious one.
- *
- * btf_type_is_struct() because its member refers to
- * another type (through member->type).
- *
- * btf_type_is_var() because the variable refers to
- * another type. btf_type_is_datasec() holds multiple
- * btf_type_is_var() types that need resolving.
- *
- * btf_type_is_array() because its element (array->type)
- * refers to another type. Array can be thought of a
- * special case of struct while array just has the same
- * member-type repeated by array->nelems of times.
- */
-static bool btf_type_needs_resolve(const struct btf_type *t)
+static void btf_struct_metas_free(struct btf_struct_metas *tab)
{
- return btf_type_is_modifier(t) ||
- btf_type_is_ptr(t) ||
- btf_type_is_struct(t) ||
- btf_type_is_array(t) ||
- btf_type_is_var(t) ||
- btf_type_is_func(t) ||
- btf_type_is_decl_tag(t) ||
- btf_type_is_datasec(t);
+ int i;
+
+ if (!tab)
+ return;
+ for (i = 0; i < tab->cnt; i++)
+ btf_record_free(tab->types[i].record);
+ kfree(tab);
}
-/* t->size can be used */
-static bool btf_type_has_size(const struct btf_type *t)
+static void btf_free_struct_meta_tab(struct btf *btf)
{
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_INT:
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- case BTF_KIND_ENUM:
- case BTF_KIND_DATASEC:
- case BTF_KIND_FLOAT:
- case BTF_KIND_ENUM64:
- return true;
- }
+ struct btf_struct_metas *tab = btf->struct_meta_tab;
- return false;
+ btf_struct_metas_free(tab);
+ btf->struct_meta_tab = NULL;
}
-static const char *btf_int_encoding_str(u8 encoding)
+static void btf_free_struct_ops_tab(struct btf *btf)
{
- if (encoding == 0)
- return "(none)";
- else if (encoding == BTF_INT_SIGNED)
- return "SIGNED";
- else if (encoding == BTF_INT_CHAR)
- return "CHAR";
- else if (encoding == BTF_INT_BOOL)
- return "BOOL";
- else
- return "UNKN";
-}
+ struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
+ u32 i;
-static u32 btf_type_int(const struct btf_type *t)
-{
- return *(u32 *)(t + 1);
-}
+ if (!tab)
+ return;
-static const struct btf_array *btf_type_array(const struct btf_type *t)
-{
- return (const struct btf_array *)(t + 1);
-}
+ for (i = 0; i < tab->cnt; i++)
+ bpf_struct_ops_desc_release(&tab->ops[i]);
-static const struct btf_enum *btf_type_enum(const struct btf_type *t)
-{
- return (const struct btf_enum *)(t + 1);
+ kfree(tab);
+ btf->struct_ops_tab = NULL;
}
-static const struct btf_var *btf_type_var(const struct btf_type *t)
+void btf_free_bpf_data(struct btf *btf)
{
- return (const struct btf_var *)(t + 1);
+ btf_free_struct_meta_tab(btf);
+ btf_free_dtor_kfunc_tab(btf);
+ btf_free_kfunc_set_tab(btf);
+ btf_free_struct_ops_tab(btf);
}
-static const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t)
+static void btf_free_rcu(struct rcu_head *rcu)
{
- return (const struct btf_decl_tag *)(t + 1);
-}
+ struct btf *btf = container_of(rcu, struct btf, rcu);
-static const struct btf_enum64 *btf_type_enum64(const struct btf_type *t)
-{
- return (const struct btf_enum64 *)(t + 1);
+ btf_free(btf);
}
-static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
+void btf_put_bpf(struct btf *btf)
{
- return kind_ops[BTF_INFO_KIND(t->info)];
+ btf_free_id(btf);
+ call_rcu(&btf->rcu, btf_free_rcu);
}
+enum {
+ BTF_FIELD_IGNORE = 0,
+ BTF_FIELD_FOUND = 1,
+};
-static bool btf_name_offset_valid(const struct btf *btf, u32 offset)
-{
- if (!BTF_STR_OFFSET_VALID(offset))
- return false;
-
- while (offset < btf->start_str_off)
- btf = btf->base_btf;
-
- offset -= btf->start_str_off;
- return offset < btf->hdr.str_len;
-}
+struct btf_field_info {
+ enum btf_field_type type;
+ u32 off;
+ union {
+ struct {
+ u32 type_id;
+ } kptr;
+ struct {
+ const char *node_name;
+ u32 value_btf_id;
+ } graph_root;
+ };
+};
-static bool __btf_name_char_ok(char c, bool first)
+static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
+ u32 off, int sz, enum btf_field_type field_type,
+ struct btf_field_info *info)
{
- if ((first ? !isalpha(c) :
- !isalnum(c)) &&
- c != '_' &&
- c != '.')
- return false;
- return true;
+ if (!__btf_type_is_struct(t))
+ return BTF_FIELD_IGNORE;
+ if (t->size != sz)
+ return BTF_FIELD_IGNORE;
+ info->type = field_type;
+ info->off = off;
+ return BTF_FIELD_FOUND;
}
-const char *btf_str_by_offset(const struct btf *btf, u32 offset)
+static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
+ u32 off, int sz, struct btf_field_info *info, u32 field_mask)
{
- while (offset < btf->start_str_off)
- btf = btf->base_btf;
-
- offset -= btf->start_str_off;
- if (offset < btf->hdr.str_len)
- return &btf->strings[offset];
-
- return NULL;
-}
+ enum btf_field_type type;
+ const char *tag_value;
+ bool is_type_tag;
+ u32 res_id;
-static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
-{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
+ /* Permit modifiers on the pointer itself */
+ if (btf_type_is_volatile(t))
+ t = btf_type_by_id(btf, t->type);
+ /* For PTR, sz is always == 8 */
+ if (!btf_type_is_ptr(t))
+ return BTF_FIELD_IGNORE;
+ t = btf_type_by_id(btf, t->type);
+ is_type_tag = btf_type_is_type_tag(t) && !btf_type_kflag(t);
+ if (!is_type_tag)
+ return BTF_FIELD_IGNORE;
+ /* Reject extra tags */
+ if (btf_type_is_type_tag(btf_type_by_id(btf, t->type)))
+ return -EINVAL;
+ tag_value = __btf_name_by_offset(btf, t->name_off);
+ if (!strcmp("kptr_untrusted", tag_value))
+ type = BPF_KPTR_UNREF;
+ else if (!strcmp("kptr", tag_value))
+ type = BPF_KPTR_REF;
+ else if (!strcmp("percpu_kptr", tag_value))
+ type = BPF_KPTR_PERCPU;
+ else if (!strcmp("uptr", tag_value))
+ type = BPF_UPTR;
+ else
+ return -EINVAL;
- if (!__btf_name_char_ok(*src, true))
- return false;
+ if (!(type & field_mask))
+ return BTF_FIELD_IGNORE;
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- src++;
- while (*src && src < src_limit) {
- if (!__btf_name_char_ok(*src, false))
- return false;
- src++;
- }
+ /* Get the base type */
+ t = btf_type_skip_modifiers(btf, t->type, &res_id);
+ /* Only pointer to struct is allowed */
+ if (!__btf_type_is_struct(t))
+ return -EINVAL;
- return !*src;
+ info->type = type;
+ info->off = off;
+ info->kptr.type_id = res_id;
+ return BTF_FIELD_FOUND;
}
-/* Allow any printable character in DATASEC names */
-static bool btf_name_valid_section(const struct btf *btf, u32 offset)
+int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
+ int comp_idx, const char *tag_key, int last_id)
{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
+ int len = strlen(tag_key);
+ int i, n;
- if (!*src)
- return false;
+ for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
+ const struct btf_type *t = btf_type_by_id(btf, i);
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- while (*src && src < src_limit) {
- if (!isprint(*src))
- return false;
- src++;
+ if (!btf_type_is_decl_tag(t))
+ continue;
+ if (pt != btf_type_by_id(btf, t->type))
+ continue;
+ if (btf_type_decl_tag(t)->component_idx != comp_idx)
+ continue;
+ if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
+ continue;
+ return i;
}
-
- return !*src;
-}
-
-static const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
-{
- const char *name;
-
- if (!offset)
- return "(anon)";
-
- name = btf_str_by_offset(btf, offset);
- return name ?: "(invalid-name-offset)";
+ return -ENOENT;
}
-const char *btf_name_by_offset(const struct btf *btf, u32 offset)
+const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
+ int comp_idx, const char *tag_key)
{
- return btf_str_by_offset(btf, offset);
-}
+ const char *value = NULL;
+ const struct btf_type *t;
+ int len, id;
-const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
+ id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
+ btf_named_start_id(btf, false) - 1);
+ if (id < 0)
+ return ERR_PTR(id);
- type_id -= btf->start_id;
- if (type_id >= btf->nr_types)
- return NULL;
- return btf->types[type_id];
-}
-EXPORT_SYMBOL_GPL(btf_type_by_id);
-
-/*
- * Check that the type @t is a regular int. This means that @t is not
- * a bit field and it has the same size as either of u8/u16/u32/u64
- * or __int128. If @expected_size is not zero, then size of @t should
- * be the same. A caller should already have checked that the type @t
- * is an integer.
- */
-static bool __btf_type_int_is_regular(const struct btf_type *t, size_t expected_size)
-{
- u32 int_data = btf_type_int(t);
- u8 nr_bits = BTF_INT_BITS(int_data);
- u8 nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
-
- return BITS_PER_BYTE_MASKED(nr_bits) == 0 &&
- BTF_INT_OFFSET(int_data) == 0 &&
- (nr_bytes <= 16 && is_power_of_2(nr_bytes)) &&
- (expected_size == 0 || nr_bytes == expected_size);
-}
+ t = btf_type_by_id(btf, id);
+ len = strlen(tag_key);
+ value = __btf_name_by_offset(btf, t->name_off) + len;
-static bool btf_type_int_is_regular(const struct btf_type *t)
-{
- return __btf_type_int_is_regular(t, 0);
-}
+ /* Prevent duplicate entries for same type */
+ id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, id);
+ if (id >= 0)
+ return ERR_PTR(-EEXIST);
-bool btf_type_is_i32(const struct btf_type *t)
-{
- return btf_type_is_int(t) && __btf_type_int_is_regular(t, 4);
+ return value;
}
-bool btf_type_is_i64(const struct btf_type *t)
+static int
+btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
+ const struct btf_type *t, int comp_idx, u32 off,
+ int sz, struct btf_field_info *info,
+ enum btf_field_type head_type)
{
- return btf_type_is_int(t) && __btf_type_int_is_regular(t, 8);
-}
+ const char *node_field_name;
+ const char *value_type;
+ s32 id;
-bool btf_type_is_primitive(const struct btf_type *t)
-{
- return (btf_type_is_int(t) && btf_type_int_is_regular(t)) ||
- btf_is_any_enum(t);
+ if (!__btf_type_is_struct(t))
+ return BTF_FIELD_IGNORE;
+ if (t->size != sz)
+ return BTF_FIELD_IGNORE;
+ value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
+ if (IS_ERR(value_type))
+ return -EINVAL;
+ node_field_name = strstr(value_type, ":");
+ if (!node_field_name)
+ return -EINVAL;
+ value_type = kstrndup(value_type, node_field_name - value_type,
+ GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!value_type)
+ return -ENOMEM;
+ id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
+ kfree(value_type);
+ if (id < 0)
+ return id;
+ node_field_name++;
+ if (str_is_empty(node_field_name))
+ return -EINVAL;
+ info->type = head_type;
+ info->off = off;
+ info->graph_root.value_btf_id = id;
+ info->graph_root.node_name = node_field_name;
+ return BTF_FIELD_FOUND;
}
-/*
- * Check that given struct member is a regular int with expected
- * offset and size.
- */
-bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
- const struct btf_member *m,
- u32 expected_offset, u32 expected_size)
+static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_type,
+ u32 field_mask, u32 *seen_mask, int *align, int *sz)
{
- const struct btf_type *t;
- u32 id, int_data;
- u8 nr_bits;
+ const struct {
+ enum btf_field_type type;
+ const char *const name;
+ const bool is_unique;
+ } field_types[] = {
+ { BPF_SPIN_LOCK, "bpf_spin_lock", true },
+ { BPF_RES_SPIN_LOCK, "bpf_res_spin_lock", true },
+ { BPF_TIMER, "bpf_timer", true },
+ { BPF_WORKQUEUE, "bpf_wq", true },
+ { BPF_TASK_WORK, "bpf_task_work", true },
+ { BPF_LIST_HEAD, "bpf_list_head", false },
+ { BPF_LIST_NODE, "bpf_list_node", false },
+ { BPF_RB_ROOT, "bpf_rb_root", false },
+ { BPF_RB_NODE, "bpf_rb_node", false },
+ { BPF_REFCOUNT, "bpf_refcount", false },
+ };
+ int type = 0, i;
+ const char *name = __btf_name_by_offset(btf, var_type->name_off);
+ const char *field_type_name;
+ enum btf_field_type field_type;
+ bool is_unique;
- id = m->type;
- t = btf_type_id_size(btf, &id, NULL);
- if (!t || !btf_type_is_int(t))
- return false;
+ for (i = 0; i < ARRAY_SIZE(field_types); ++i) {
+ field_type = field_types[i].type;
+ field_type_name = field_types[i].name;
+ is_unique = field_types[i].is_unique;
+ if (!(field_mask & field_type) || strcmp(name, field_type_name))
+ continue;
+ if (is_unique) {
+ if (*seen_mask & field_type)
+ return -E2BIG;
+ *seen_mask |= field_type;
+ }
+ type = field_type;
+ goto end;
+ }
- int_data = btf_type_int(t);
- nr_bits = BTF_INT_BITS(int_data);
- if (btf_type_kflag(s)) {
- u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
- u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
-
- /* if kflag set, int should be a regular int and
- * bit offset should be at byte boundary.
- */
- return !bitfield_size &&
- BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
- BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
- }
-
- if (BTF_INT_OFFSET(int_data) ||
- BITS_PER_BYTE_MASKED(m->offset) ||
- BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
- BITS_PER_BYTE_MASKED(nr_bits) ||
- BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
- return false;
-
- return true;
-}
-
-/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
-static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
- u32 id)
-{
- const struct btf_type *t = btf_type_by_id(btf, id);
-
- while (btf_type_is_modifier(t) &&
- BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
- t = btf_type_by_id(btf, t->type);
+ /* Only return BPF_KPTR when all other types with matchable names fail */
+ if (field_mask & (BPF_KPTR | BPF_UPTR) && !__btf_type_is_struct(var_type)) {
+ type = BPF_KPTR_REF;
+ goto end;
}
-
- return t;
+ return 0;
+end:
+ *sz = btf_field_type_size(type);
+ *align = btf_field_type_align(type);
+ return type;
}
-#define BTF_SHOW_MAX_ITER 10
-
-#define BTF_KIND_BIT(kind) (1ULL << kind)
-
-/*
- * Populate show->state.name with type name information.
- * Format of type name is
+/* Repeat a number of fields for a specified number of times.
*
- * [.member_name = ] (type_name)
+ * Copy the fields starting from the first field and repeat them for
+ * repeat_cnt times. The fields are repeated by adding the offset of each
+ * field with
+ * (i + 1) * elem_size
+ * where i is the repeat index and elem_size is the size of an element.
*/
-static const char *btf_show_name(struct btf_show *show)
+static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
+ u32 field_cnt, u32 repeat_cnt, u32 elem_size)
{
- /* BTF_MAX_ITER array suffixes "[]" */
- const char *array_suffixes = "[][][][][][][][][][]";
- const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
- /* BTF_MAX_ITER pointer suffixes "*" */
- const char *ptr_suffixes = "**********";
- const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
- const char *name = NULL, *prefix = "", *parens = "";
- const struct btf_member *m = show->state.member;
- const struct btf_type *t;
- const struct btf_array *array;
- u32 id = show->state.type_id;
- const char *member = NULL;
- bool show_member = false;
- u64 kinds = 0;
- int i;
-
- show->state.name[0] = '\0';
-
- /*
- * Don't show type name if we're showing an array member;
- * in that case we show the array type so don't need to repeat
- * ourselves for each member.
- */
- if (show->state.array_member)
- return "";
-
- /* Retrieve member name, if any. */
- if (m) {
- member = btf_name_by_offset(show->btf, m->name_off);
- show_member = strlen(member) > 0;
- id = m->type;
- }
-
- /*
- * Start with type_id, as we have resolved the struct btf_type *
- * via btf_modifier_show() past the parent typedef to the child
- * struct, int etc it is defined as. In such cases, the type_id
- * still represents the starting type while the struct btf_type *
- * in our show->state points at the resolved type of the typedef.
- */
- t = btf_type_by_id(show->btf, id);
- if (!t)
- return "";
+ u32 i, j;
+ u32 cur;
- /*
- * The goal here is to build up the right number of pointer and
- * array suffixes while ensuring the type name for a typedef
- * is represented. Along the way we accumulate a list of
- * BTF kinds we have encountered, since these will inform later
- * display; for example, pointer types will not require an
- * opening "{" for struct, we will just display the pointer value.
- *
- * We also want to accumulate the right number of pointer or array
- * indices in the format string while iterating until we get to
- * the typedef/pointee/array member target type.
- *
- * We start by pointing at the end of pointer and array suffix
- * strings; as we accumulate pointers and arrays we move the pointer
- * or array string backwards so it will show the expected number of
- * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
- * and/or arrays and typedefs are supported as a precaution.
- *
- * We also want to get typedef name while proceeding to resolve
- * type it points to so that we can add parentheses if it is a
- * "typedef struct" etc.
- */
- for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
-
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_TYPEDEF:
- if (!name)
- name = btf_name_by_offset(show->btf,
- t->name_off);
- kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
- id = t->type;
- break;
- case BTF_KIND_ARRAY:
- kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
- parens = "[";
- if (!t)
- return "";
- array = btf_type_array(t);
- if (array_suffix > array_suffixes)
- array_suffix -= 2;
- id = array->type;
- break;
- case BTF_KIND_PTR:
- kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
- if (ptr_suffix > ptr_suffixes)
- ptr_suffix -= 1;
- id = t->type;
+ /* Ensure not repeating fields that should not be repeated. */
+ for (i = 0; i < field_cnt; i++) {
+ switch (info[i].type) {
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ case BPF_LIST_HEAD:
+ case BPF_RB_ROOT:
break;
default:
- id = 0;
- break;
+ return -EINVAL;
}
- if (!id)
- break;
- t = btf_type_skip_qualifiers(show->btf, id);
- }
- /* We may not be able to represent this type; bail to be safe */
- if (i == BTF_SHOW_MAX_ITER)
- return "";
-
- if (!name)
- name = btf_name_by_offset(show->btf, t->name_off);
-
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
- "struct" : "union";
- /* if it's an array of struct/union, parens is already set */
- if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
- parens = "{";
- break;
- case BTF_KIND_ENUM:
- case BTF_KIND_ENUM64:
- prefix = "enum";
- break;
- default:
- break;
}
- /* pointer does not require parens */
- if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
- parens = "";
- /* typedef does not require struct/union/enum prefix */
- if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
- prefix = "";
-
- if (!name)
- name = "";
-
- /* Even if we don't want type name info, we want parentheses etc */
- if (show->flags & BTF_SHOW_NONAME)
- snprintf(show->state.name, sizeof(show->state.name), "%s",
- parens);
- else
- snprintf(show->state.name, sizeof(show->state.name),
- "%s%s%s(%s%s%s%s%s%s)%s",
- /* first 3 strings comprise ".member = " */
- show_member ? "." : "",
- show_member ? member : "",
- show_member ? " = " : "",
- /* ...next is our prefix (struct, enum, etc) */
- prefix,
- strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
- /* ...this is the type name itself */
- name,
- /* ...suffixed by the appropriate '*', '[]' suffixes */
- strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
- array_suffix, parens);
-
- return show->state.name;
-}
+ /* The type of struct size or variable size is u32,
+ * so the multiplication will not overflow.
+ */
+ if (field_cnt * (repeat_cnt + 1) > info_cnt)
+ return -E2BIG;
-static const char *__btf_show_indent(struct btf_show *show)
-{
- const char *indents = " ";
- const char *indent = &indents[strlen(indents)];
+ cur = field_cnt;
+ for (i = 0; i < repeat_cnt; i++) {
+ memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
+ for (j = 0; j < field_cnt; j++)
+ info[cur++].off += (i + 1) * elem_size;
+ }
- if ((indent - show->state.depth) >= indents)
- return indent - show->state.depth;
- return indents;
+ return 0;
}
-static const char *btf_show_indent(struct btf_show *show)
-{
- return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
-}
+static int btf_find_struct_field(const struct btf *btf,
+ const struct btf_type *t, u32 field_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level);
-static const char *btf_show_newline(struct btf_show *show)
+/* Find special fields in the struct type of a field.
+ *
+ * This function is used to find fields of special types that is not a
+ * global variable or a direct field of a struct type. It also handles the
+ * repetition if it is the element type of an array.
+ */
+static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
+ u32 off, u32 nelems,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt, u32 level)
{
- return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
-}
+ int ret, err, i;
-static const char *btf_show_delim(struct btf_show *show)
-{
- if (show->state.depth == 0)
- return "";
+ level++;
+ if (level >= MAX_RESOLVE_DEPTH)
+ return -E2BIG;
- if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
- BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
- return "|";
+ ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
- return ",";
-}
+ if (ret <= 0)
+ return ret;
-__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
-{
- va_list args;
+ /* Shift the offsets of the nested struct fields to the offsets
+ * related to the container.
+ */
+ for (i = 0; i < ret; i++)
+ info[i].off += off;
- if (!show->state.depth_check) {
- va_start(args, fmt);
- show->showfn(show, fmt, args);
- va_end(args);
+ if (nelems > 1) {
+ err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
+ if (err == 0)
+ ret *= nelems;
+ else
+ ret = err;
}
-}
-
-/* Macros are used here as btf_show_type_value[s]() prepends and appends
- * format specifiers to the format specifier passed in; these do the work of
- * adding indentation, delimiters etc while the caller simply has to specify
- * the type value(s) in the format specifier + value(s).
- */
-#define btf_show_type_value(show, fmt, value) \
- do { \
- if ((value) != (__typeof__(value))0 || \
- (show->flags & BTF_SHOW_ZERO) || \
- show->state.depth == 0) { \
- btf_show(show, "%s%s" fmt "%s%s", \
- btf_show_indent(show), \
- btf_show_name(show), \
- value, btf_show_delim(show), \
- btf_show_newline(show)); \
- if (show->state.depth > show->state.depth_to_show) \
- show->state.depth_to_show = show->state.depth; \
- } \
- } while (0)
-
-#define btf_show_type_values(show, fmt, ...) \
- do { \
- btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
- btf_show_name(show), \
- __VA_ARGS__, btf_show_delim(show), \
- btf_show_newline(show)); \
- if (show->state.depth > show->state.depth_to_show) \
- show->state.depth_to_show = show->state.depth; \
- } while (0)
-
-/* How much is left to copy to safe buffer after @data? */
-static int btf_show_obj_size_left(struct btf_show *show, void *data)
-{
- return show->obj.head + show->obj.size - data;
-}
-
-/* Is object pointed to by @data of @size already copied to our safe buffer? */
-static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
-{
- return data >= show->obj.data &&
- (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
-}
-/*
- * If object pointed to by @data of @size falls within our safe buffer, return
- * the equivalent pointer to the same safe data. Assumes
- * copy_from_kernel_nofault() has already happened and our safe buffer is
- * populated.
- */
-static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
-{
- if (btf_show_obj_is_safe(show, data, size))
- return show->obj.safe + (data - show->obj.data);
- return NULL;
+ return ret;
}
-/*
- * Return a safe-to-access version of data pointed to by @data.
- * We do this by copying the relevant amount of information
- * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
- *
- * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
- * safe copy is needed.
- *
- * Otherwise we need to determine if we have the required amount
- * of data (determined by the @data pointer and the size of the
- * largest base type we can encounter (represented by
- * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
- * that we will be able to print some of the current object,
- * and if more is needed a copy will be triggered.
- * Some objects such as structs will not fit into the buffer;
- * in such cases additional copies when we iterate over their
- * members may be needed.
- *
- * btf_show_obj_safe() is used to return a safe buffer for
- * btf_show_start_type(); this ensures that as we recurse into
- * nested types we always have safe data for the given type.
- * This approach is somewhat wasteful; it's possible for example
- * that when iterating over a large union we'll end up copying the
- * same data repeatedly, but the goal is safety not performance.
- * We use stack data as opposed to per-CPU buffers because the
- * iteration over a type can take some time, and preemption handling
- * would greatly complicate use of the safe buffer.
- */
-static void *btf_show_obj_safe(struct btf_show *show,
- const struct btf_type *t,
- void *data)
+static int btf_find_field_one(const struct btf *btf,
+ const struct btf_type *var,
+ const struct btf_type *var_type,
+ int var_idx,
+ u32 off, u32 expected_size,
+ u32 field_mask, u32 *seen_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level)
{
- const struct btf_type *rt;
- int size_left, size;
- void *safe = NULL;
-
- if (show->flags & BTF_SHOW_UNSAFE)
- return data;
-
- rt = btf_resolve_size(show->btf, t, &size);
- if (IS_ERR(rt)) {
- show->state.status = PTR_ERR(rt);
- return NULL;
- }
+ int ret, align, sz, field_type;
+ struct btf_field_info tmp;
+ const struct btf_array *array;
+ u32 i, nelems = 1;
- /*
- * Is this toplevel object? If so, set total object size and
- * initialize pointers. Otherwise check if we still fall within
- * our safe object data.
+ /* Walk into array types to find the element type and the number of
+ * elements in the (flattened) array.
*/
- if (show->state.depth == 0) {
- show->obj.size = size;
- show->obj.head = data;
- } else {
- /*
- * If the size of the current object is > our remaining
- * safe buffer we _may_ need to do a new copy. However
- * consider the case of a nested struct; it's size pushes
- * us over the safe buffer limit, but showing any individual
- * struct members does not. In such cases, we don't need
- * to initiate a fresh copy yet; however we definitely need
- * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
- * in our buffer, regardless of the current object size.
- * The logic here is that as we resolve types we will
- * hit a base type at some point, and we need to be sure
- * the next chunk of data is safely available to display
- * that type info safely. We cannot rely on the size of
- * the current object here because it may be much larger
- * than our current buffer (e.g. task_struct is 8k).
- * All we want to do here is ensure that we can print the
- * next basic type, which we can if either
- * - the current type size is within the safe buffer; or
- * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
- * the safe buffer.
- */
- safe = __btf_show_obj_safe(show, data,
- min(size,
- BTF_SHOW_OBJ_BASE_TYPE_SIZE));
+ for (i = 0; i < MAX_RESOLVE_DEPTH && btf_type_is_array(var_type); i++) {
+ array = btf_array(var_type);
+ nelems *= array->nelems;
+ var_type = btf_type_by_id(btf, array->type);
}
+ if (i == MAX_RESOLVE_DEPTH)
+ return -E2BIG;
+ if (nelems == 0)
+ return 0;
- /*
- * We need a new copy to our safe object, either because we haven't
- * yet copied and are initializing safe data, or because the data
- * we want falls outside the boundaries of the safe object.
- */
- if (!safe) {
- size_left = btf_show_obj_size_left(show, data);
- if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
- size_left = BTF_SHOW_OBJ_SAFE_SIZE;
- show->state.status = copy_from_kernel_nofault(show->obj.safe,
- data, size_left);
- if (!show->state.status) {
- show->obj.data = data;
- safe = show->obj.safe;
- }
+ field_type = btf_get_field_type(btf, var_type,
+ field_mask, seen_mask, &align, &sz);
+ /* Look into variables of struct types */
+ if (!field_type && __btf_type_is_struct(var_type)) {
+ sz = var_type->size;
+ if (expected_size && expected_size != sz * nelems)
+ return 0;
+ ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
+ &info[0], info_cnt, level);
+ return ret;
}
- return safe;
-}
+ if (field_type == 0)
+ return 0;
+ if (field_type < 0)
+ return field_type;
-/*
- * Set the type we are starting to show and return a safe data pointer
- * to be used for showing the associated data.
- */
-static void *btf_show_start_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id, void *data)
-{
- show->state.type = t;
- show->state.type_id = type_id;
- show->state.name[0] = '\0';
+ if (expected_size && expected_size != sz * nelems)
+ return 0;
+ if (off % align)
+ return 0;
- return btf_show_obj_safe(show, t, data);
-}
+ switch (field_type) {
+ case BPF_SPIN_LOCK:
+ case BPF_RES_SPIN_LOCK:
+ case BPF_TIMER:
+ case BPF_WORKQUEUE:
+ case BPF_LIST_NODE:
+ case BPF_RB_NODE:
+ case BPF_REFCOUNT:
+ case BPF_TASK_WORK:
+ ret = btf_find_struct(btf, var_type, off, sz, field_type,
+ info_cnt ? &info[0] : &tmp);
+ if (ret < 0)
+ return ret;
+ break;
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ ret = btf_find_kptr(btf, var_type, off, sz,
+ info_cnt ? &info[0] : &tmp, field_mask);
+ if (ret < 0)
+ return ret;
+ break;
+ case BPF_LIST_HEAD:
+ case BPF_RB_ROOT:
+ ret = btf_find_graph_root(btf, var, var_type,
+ var_idx, off, sz,
+ info_cnt ? &info[0] : &tmp,
+ field_type);
+ if (ret < 0)
+ return ret;
+ break;
+ default:
+ return -EFAULT;
+ }
-static void btf_show_end_type(struct btf_show *show)
-{
- show->state.type = NULL;
- show->state.type_id = 0;
- show->state.name[0] = '\0';
+ if (ret == BTF_FIELD_IGNORE)
+ return 0;
+ if (!info_cnt)
+ return -E2BIG;
+ if (nelems > 1) {
+ ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
+ if (ret < 0)
+ return ret;
+ }
+ return nelems;
}
-static void *btf_show_start_aggr_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id, void *data)
+static int btf_find_struct_field(const struct btf *btf,
+ const struct btf_type *t, u32 field_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level)
{
- void *safe_data = btf_show_start_type(show, t, type_id, data);
-
- if (!safe_data)
- return safe_data;
+ int ret, idx = 0;
+ const struct btf_member *member;
+ u32 i, off, seen_mask = 0;
- btf_show(show, "%s%s%s", btf_show_indent(show),
- btf_show_name(show),
- btf_show_newline(show));
- show->state.depth++;
- return safe_data;
-}
+ for_each_member(i, t, member) {
+ const struct btf_type *member_type = btf_type_by_id(btf,
+ member->type);
-static void btf_show_end_aggr_type(struct btf_show *show,
- const char *suffix)
-{
- show->state.depth--;
- btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
- btf_show_delim(show), btf_show_newline(show));
- btf_show_end_type(show);
-}
-
-static void btf_show_start_member(struct btf_show *show,
- const struct btf_member *m)
-{
- show->state.member = m;
-}
-
-static void btf_show_start_array_member(struct btf_show *show)
-{
- show->state.array_member = 1;
- btf_show_start_member(show, NULL);
-}
-
-static void btf_show_end_member(struct btf_show *show)
-{
- show->state.member = NULL;
-}
-
-static void btf_show_end_array_member(struct btf_show *show)
-{
- show->state.array_member = 0;
- btf_show_end_member(show);
-}
+ off = __btf_member_bit_offset(t, member);
+ if (off % 8)
+ /* valid C code cannot generate such BTF */
+ return -EINVAL;
+ off /= 8;
-static void *btf_show_start_array_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id,
- u16 array_encoding,
- void *data)
-{
- show->state.array_encoding = array_encoding;
- show->state.array_terminated = 0;
- return btf_show_start_aggr_type(show, t, type_id, data);
+ ret = btf_find_field_one(btf, t, member_type, i,
+ off, 0,
+ field_mask, &seen_mask,
+ &info[idx], info_cnt - idx, level);
+ if (ret < 0)
+ return ret;
+ idx += ret;
+ }
+ return idx;
}
-static void btf_show_end_array_type(struct btf_show *show)
+static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt, u32 level)
{
- show->state.array_encoding = 0;
- show->state.array_terminated = 0;
- btf_show_end_aggr_type(show, "]");
-}
+ int ret, idx = 0;
+ const struct btf_var_secinfo *vsi;
+ u32 i, off, seen_mask = 0;
-static void *btf_show_start_struct_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id,
- void *data)
-{
- return btf_show_start_aggr_type(show, t, type_id, data);
-}
+ for_each_vsi(i, t, vsi) {
+ const struct btf_type *var = btf_type_by_id(btf, vsi->type);
+ const struct btf_type *var_type = btf_type_by_id(btf, var->type);
-static void btf_show_end_struct_type(struct btf_show *show)
-{
- btf_show_end_aggr_type(show, "}");
+ off = vsi->offset;
+ ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
+ field_mask, &seen_mask,
+ &info[idx], info_cnt - idx,
+ level);
+ if (ret < 0)
+ return ret;
+ idx += ret;
+ }
+ return idx;
}
-__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
- const char *fmt, ...)
+static int btf_find_field(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt)
{
- va_list args;
-
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ if (__btf_type_is_struct(t))
+ return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
+ else if (btf_type_is_datasec(t))
+ return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
+ return -EINVAL;
}
-__printf(2, 3) static void btf_verifier_log(struct btf_verifier_env *env,
- const char *fmt, ...)
+/* Callers have to ensure the life cycle of btf if it is program BTF */
+static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
+ struct module *mod = NULL;
+ const struct btf_type *t;
+ /* If a matching btf type is found in kernel or module BTFs, kptr_ref
+ * is that BTF, otherwise it's program BTF
+ */
+ struct btf *kptr_btf;
+ int ret;
+ s32 id;
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
-}
+ /* Find type in map BTF, and use it to look up the matching type
+ * in vmlinux or module BTFs, by name and kind.
+ */
+ t = btf_type_by_id(btf, info->kptr.type_id);
+ id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
+ &kptr_btf);
+ if (id == -ENOENT) {
+ /* btf_parse_kptr should only be called w/ btf = program BTF */
+ WARN_ON_ONCE(btf_is_kernel(btf));
-__printf(4, 5) static void __btf_verifier_log_type(struct btf_verifier_env *env,
- const struct btf_type *t,
- bool log_details,
- const char *fmt, ...)
-{
- struct bpf_verifier_log *log = &env->log;
- struct btf *btf = env->btf;
- va_list args;
+ /* Type exists only in program BTF. Assume that it's a MEM_ALLOC
+ * kptr allocated via bpf_obj_new
+ */
+ field->kptr.dtor = NULL;
+ id = info->kptr.type_id;
+ kptr_btf = (struct btf *)btf;
+ goto found_dtor;
+ }
+ if (id < 0)
+ return id;
- if (!bpf_verifier_log_needed(log))
- return;
+ /* Find and stash the function pointer for the destruction function that
+ * needs to be eventually invoked from the map free path.
+ */
+ if (info->type == BPF_KPTR_REF) {
+ const struct btf_type *dtor_func;
+ const char *dtor_func_name;
+ unsigned long addr;
+ s32 dtor_btf_id;
- if (log->level == BPF_LOG_KERNEL) {
- /* btf verifier prints all types it is processing via
- * btf_verifier_log_type(..., fmt = NULL).
- * Skip those prints for in-kernel BTF verification.
+ /* This call also serves as a whitelist of allowed objects that
+ * can be used as a referenced pointer and be stored in a map at
+ * the same time.
*/
- if (!fmt)
- return;
-
- /* Skip logging when loading module BTF with mismatches permitted */
- if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
- return;
- }
+ dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
+ if (dtor_btf_id < 0) {
+ ret = dtor_btf_id;
+ goto end_btf;
+ }
- __btf_verifier_log(log, "[%u] %s %s%s",
- env->log_type_id,
- btf_type_str(t),
- __btf_name_by_offset(btf, t->name_off),
- log_details ? " " : "");
+ dtor_func = btf_type_by_id(kptr_btf, dtor_btf_id);
+ if (!dtor_func) {
+ ret = -ENOENT;
+ goto end_btf;
+ }
- if (log_details)
- btf_type_ops(t)->log_details(env, t);
+ if (btf_is_module(kptr_btf)) {
+ mod = btf_try_get_module(kptr_btf);
+ if (!mod) {
+ ret = -ENXIO;
+ goto end_btf;
+ }
+ }
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ /* We already verified dtor_func to be btf_type_is_func
+ * in register_btf_id_dtor_kfuncs.
+ */
+ dtor_func_name = __btf_name_by_offset(kptr_btf, dtor_func->name_off);
+ addr = kallsyms_lookup_name(dtor_func_name);
+ if (!addr) {
+ ret = -EINVAL;
+ goto end_mod;
+ }
+ field->kptr.dtor = (void *)addr;
}
- __btf_verifier_log(log, "\n");
+found_dtor:
+ field->kptr.btf_id = id;
+ field->kptr.btf = kptr_btf;
+ field->kptr.module = mod;
+ return 0;
+end_mod:
+ module_put(mod);
+end_btf:
+ btf_put(kptr_btf);
+ return ret;
}
-#define btf_verifier_log_type(env, t, ...) \
- __btf_verifier_log_type((env), (t), true, __VA_ARGS__)
-#define btf_verifier_log_basic(env, t, ...) \
- __btf_verifier_log_type((env), (t), false, __VA_ARGS__)
-
-__printf(4, 5)
-static void btf_verifier_log_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const char *fmt, ...)
+static int btf_parse_graph_root(const struct btf *btf,
+ struct btf_field *field,
+ struct btf_field_info *info,
+ const char *node_type_name,
+ size_t node_type_align)
{
- struct bpf_verifier_log *log = &env->log;
- struct btf *btf = env->btf;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
-
- if (log->level == BPF_LOG_KERNEL) {
- if (!fmt)
- return;
-
- /* Skip logging when loading module BTF with mismatches permitted */
- if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
- return;
- }
+ const struct btf_type *t, *n = NULL;
+ const struct btf_member *member;
+ u32 offset;
+ int i;
- /* The CHECK_META phase already did a btf dump.
- *
- * If member is logged again, it must hit an error in
- * parsing this member. It is useful to print out which
- * struct this member belongs to.
+ t = btf_type_by_id(btf, info->graph_root.value_btf_id);
+ /* We've already checked that value_btf_id is a struct type. We
+ * just need to figure out the offset of the list_node, and
+ * verify its type.
*/
- if (env->phase != CHECK_META)
- btf_verifier_log_type(env, struct_type, NULL);
-
- if (btf_type_kflag(struct_type))
- __btf_verifier_log(log,
- "\t%s type_id=%u bitfield_size=%u bits_offset=%u",
- __btf_name_by_offset(btf, member->name_off),
- member->type,
- BTF_MEMBER_BITFIELD_SIZE(member->offset),
- BTF_MEMBER_BIT_OFFSET(member->offset));
- else
- __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
- __btf_name_by_offset(btf, member->name_off),
- member->type, member->offset);
+ for_each_member(i, t, member) {
+ if (strcmp(info->graph_root.node_name,
+ __btf_name_by_offset(btf, member->name_off)))
+ continue;
+ /* Invalid BTF, two members with same name */
+ if (n)
+ return -EINVAL;
+ n = btf_type_by_id(btf, member->type);
+ if (!__btf_type_is_struct(n))
+ return -EINVAL;
+ if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off)))
+ return -EINVAL;
+ offset = __btf_member_bit_offset(n, member);
+ if (offset % 8)
+ return -EINVAL;
+ offset /= 8;
+ if (offset % node_type_align)
+ return -EINVAL;
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ field->graph_root.btf = (struct btf *)btf;
+ field->graph_root.value_btf_id = info->graph_root.value_btf_id;
+ field->graph_root.node_offset = offset;
}
-
- __btf_verifier_log(log, "\n");
+ if (!n)
+ return -ENOENT;
+ return 0;
}
-__printf(4, 5)
-static void btf_verifier_log_vsi(struct btf_verifier_env *env,
- const struct btf_type *datasec_type,
- const struct btf_var_secinfo *vsi,
- const char *fmt, ...)
+static int btf_parse_list_head(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
- if (log->level == BPF_LOG_KERNEL && !fmt)
- return;
- if (env->phase != CHECK_META)
- btf_verifier_log_type(env, datasec_type, NULL);
-
- __btf_verifier_log(log, "\t type_id=%u offset=%u size=%u",
- vsi->type, vsi->offset, vsi->size);
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
- }
-
- __btf_verifier_log(log, "\n");
+ return btf_parse_graph_root(btf, field, info, "bpf_list_node",
+ __alignof__(struct bpf_list_node));
}
-static void btf_verifier_log_hdr(struct btf_verifier_env *env,
- u32 btf_data_size)
+static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- const struct btf *btf = env->btf;
- const struct btf_header *hdr;
-
- if (!bpf_verifier_log_needed(log))
- return;
-
- if (log->level == BPF_LOG_KERNEL)
- return;
- hdr = &btf->hdr;
- __btf_verifier_log(log, "magic: 0x%x\n", hdr->magic);
- __btf_verifier_log(log, "version: %u\n", hdr->version);
- __btf_verifier_log(log, "flags: 0x%x\n", hdr->flags);
- __btf_verifier_log(log, "hdr_len: %u\n", hdr->hdr_len);
- __btf_verifier_log(log, "type_off: %u\n", hdr->type_off);
- __btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
- __btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
- __btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
- __btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
+ return btf_parse_graph_root(btf, field, info, "bpf_rb_node",
+ __alignof__(struct bpf_rb_node));
}
-static int btf_add_type(struct btf_verifier_env *env, struct btf_type *t)
+static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
{
- struct btf *btf = env->btf;
-
- if (btf->types_size == btf->nr_types) {
- /* Expand 'types' array */
-
- struct btf_type **new_types;
- u32 expand_by, new_size;
-
- if (btf->start_id + btf->types_size == BTF_MAX_TYPE) {
- btf_verifier_log(env, "Exceeded max num of types");
- return -E2BIG;
- }
-
- expand_by = max_t(u32, btf->types_size >> 2, 16);
- new_size = min_t(u32, BTF_MAX_TYPE,
- btf->types_size + expand_by);
-
- new_types = kvzalloc_objs(*new_types, new_size,
- GFP_KERNEL | __GFP_NOWARN);
- if (!new_types)
- return -ENOMEM;
-
- if (btf->nr_types == 0) {
- if (!btf->base_btf) {
- /* lazily init VOID type */
- new_types[0] = &btf_void;
- btf->nr_types++;
- }
- } else {
- memcpy(new_types, btf->types,
- sizeof(*btf->types) * btf->nr_types);
- }
-
- kvfree(btf->types);
- btf->types = new_types;
- btf->types_size = new_size;
- }
-
- btf->types[btf->nr_types++] = t;
+ const struct btf_field *a = (const struct btf_field *)_a;
+ const struct btf_field *b = (const struct btf_field *)_b;
+ if (a->offset < b->offset)
+ return -1;
+ else if (a->offset > b->offset)
+ return 1;
return 0;
}
-static int btf_alloc_id(struct btf *btf)
+struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, u32 value_size)
{
- int id;
+ struct btf_field_info info_arr[BTF_FIELDS_MAX];
+ u32 next_off = 0, field_type_size;
+ struct btf_record *rec;
+ int ret, i, cnt;
- idr_preload(GFP_KERNEL);
- spin_lock_bh(&btf_idr_lock);
- id = idr_alloc_cyclic(&btf_idr, btf, 1, INT_MAX, GFP_ATOMIC);
- if (id > 0)
- btf->id = id;
- spin_unlock_bh(&btf_idr_lock);
- idr_preload_end();
+ ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
+ if (ret < 0)
+ return ERR_PTR(ret);
+ if (!ret)
+ return NULL;
- if (WARN_ON_ONCE(!id))
- return -ENOSPC;
-
- return id > 0 ? 0 : id;
-}
-
-static void btf_free_id(struct btf *btf)
-{
- unsigned long flags;
-
- /*
- * In map-in-map, calling map_delete_elem() on outer
- * map will call bpf_map_put on the inner map.
- * It will then eventually call btf_free_id()
- * on the inner map. Some of the map_delete_elem()
- * implementation may have irq disabled, so
- * we need to use the _irqsave() version instead
- * of the _bh() version.
- */
- spin_lock_irqsave(&btf_idr_lock, flags);
- if (btf->id) {
- idr_remove(&btf_idr, btf->id);
- /*
- * Clear the id here to make this function idempotent, since it will get
- * called a couple of times for module BTFs: on module unload, and then
- * the final btf_put(). btf_alloc_id() starts IDs with 1, so we can use
- * 0 as sentinel value.
- */
- WRITE_ONCE(btf->id, 0);
- }
- spin_unlock_irqrestore(&btf_idr_lock, flags);
-}
-
-static void btf_free_kfunc_set_tab(struct btf *btf)
-{
- struct btf_kfunc_set_tab *tab = btf->kfunc_set_tab;
- int hook;
-
- if (!tab)
- return;
- for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
- kfree(tab->sets[hook]);
- kfree(tab);
- btf->kfunc_set_tab = NULL;
-}
-
-static void btf_free_dtor_kfunc_tab(struct btf *btf)
-{
- struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
-
- if (!tab)
- return;
- kfree(tab);
- btf->dtor_kfunc_tab = NULL;
-}
-
-static void btf_struct_metas_free(struct btf_struct_metas *tab)
-{
- int i;
-
- if (!tab)
- return;
- for (i = 0; i < tab->cnt; i++)
- btf_record_free(tab->types[i].record);
- kfree(tab);
-}
-
-static void btf_free_struct_meta_tab(struct btf *btf)
-{
- struct btf_struct_metas *tab = btf->struct_meta_tab;
-
- btf_struct_metas_free(tab);
- btf->struct_meta_tab = NULL;
-}
-
-static void btf_free_struct_ops_tab(struct btf *btf)
-{
- struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
- u32 i;
-
- if (!tab)
- return;
-
- for (i = 0; i < tab->cnt; i++)
- bpf_struct_ops_desc_release(&tab->ops[i]);
-
- kfree(tab);
- btf->struct_ops_tab = NULL;
-}
-
-static void btf_free(struct btf *btf)
-{
- btf_free_struct_meta_tab(btf);
- btf_free_dtor_kfunc_tab(btf);
- btf_free_kfunc_set_tab(btf);
- btf_free_struct_ops_tab(btf);
- kvfree(btf->types);
- kvfree(btf->resolved_sizes);
- kvfree(btf->resolved_ids);
- /* vmlinux does not allocate btf->data, it simply points it at
- * __start_BTF.
+ cnt = ret;
+ /* This needs to be kzalloc to zero out padding and unused fields, see
+ * comment in btf_record_equal.
*/
- if (!btf_is_vmlinux(btf))
- kvfree(btf->data);
- kvfree(btf->base_id_map);
- kfree(btf);
-}
-
-static void btf_free_rcu(struct rcu_head *rcu)
-{
- struct btf *btf = container_of(rcu, struct btf, rcu);
-
- btf_free(btf);
-}
-
-const char *btf_get_name(const struct btf *btf)
-{
- return btf->name;
-}
-
-void btf_get(struct btf *btf)
-{
- refcount_inc(&btf->refcnt);
-}
-
-void btf_put(struct btf *btf)
-{
- if (btf && refcount_dec_and_test(&btf->refcnt)) {
- btf_free_id(btf);
- call_rcu(&btf->rcu, btf_free_rcu);
- }
-}
-
-struct btf *btf_base_btf(const struct btf *btf)
-{
- return btf->base_btf;
-}
-
-const struct btf_header *btf_header(const struct btf *btf)
-{
- return &btf->hdr;
-}
-
-void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
-{
- btf->base_btf = (struct btf *)base_btf;
- btf->start_id = btf_nr_types(base_btf);
- btf->start_str_off = base_btf->hdr.str_len;
-}
-
-static int env_resolve_init(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- u32 nr_types = btf->nr_types;
- u32 *resolved_sizes = NULL;
- u32 *resolved_ids = NULL;
- u8 *visit_states = NULL;
-
- resolved_sizes = kvcalloc(nr_types, sizeof(*resolved_sizes),
- GFP_KERNEL | __GFP_NOWARN);
- if (!resolved_sizes)
- goto nomem;
-
- resolved_ids = kvcalloc(nr_types, sizeof(*resolved_ids),
- GFP_KERNEL | __GFP_NOWARN);
- if (!resolved_ids)
- goto nomem;
-
- visit_states = kvcalloc(nr_types, sizeof(*visit_states),
- GFP_KERNEL | __GFP_NOWARN);
- if (!visit_states)
- goto nomem;
-
- btf->resolved_sizes = resolved_sizes;
- btf->resolved_ids = resolved_ids;
- env->visit_states = visit_states;
-
- return 0;
-
-nomem:
- kvfree(resolved_sizes);
- kvfree(resolved_ids);
- kvfree(visit_states);
- return -ENOMEM;
-}
-
-static void btf_verifier_env_free(struct btf_verifier_env *env)
-{
- kvfree(env->visit_states);
- kfree(env);
-}
-
-static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
- const struct btf_type *next_type)
-{
- switch (env->resolve_mode) {
- case RESOLVE_TBD:
- /* int, enum or void is a sink */
- return !btf_type_needs_resolve(next_type);
- case RESOLVE_PTR:
- /* int, enum, void, struct, array, func or func_proto is a sink
- * for ptr
- */
- return !btf_type_is_modifier(next_type) &&
- !btf_type_is_ptr(next_type);
- case RESOLVE_STRUCT_OR_ARRAY:
- /* int, enum, void, ptr, func or func_proto is a sink
- * for struct and array
- */
- return !btf_type_is_modifier(next_type) &&
- !btf_type_is_array(next_type) &&
- !btf_type_is_struct(next_type);
- default:
- BUG();
- }
-}
-
-static bool env_type_is_resolved(const struct btf_verifier_env *env,
- u32 type_id)
-{
- /* base BTF types should be resolved by now */
- if (type_id < env->btf->start_id)
- return true;
-
- return env->visit_states[type_id - env->btf->start_id] == RESOLVED;
-}
-
-static int env_stack_push(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id)
-{
- const struct btf *btf = env->btf;
- struct resolve_vertex *v;
-
- if (env->top_stack == MAX_RESOLVE_DEPTH)
- return -E2BIG;
-
- if (type_id < btf->start_id
- || env->visit_states[type_id - btf->start_id] != NOT_VISITED)
- return -EEXIST;
-
- env->visit_states[type_id - btf->start_id] = VISITED;
-
- v = &env->stack[env->top_stack++];
- v->t = t;
- v->type_id = type_id;
- v->next_member = 0;
-
- if (env->resolve_mode == RESOLVE_TBD) {
- if (btf_type_is_ptr(t))
- env->resolve_mode = RESOLVE_PTR;
- else if (btf_type_is_struct(t) || btf_type_is_array(t))
- env->resolve_mode = RESOLVE_STRUCT_OR_ARRAY;
- }
-
- return 0;
-}
-
-static void env_stack_set_next_member(struct btf_verifier_env *env,
- u16 next_member)
-{
- env->stack[env->top_stack - 1].next_member = next_member;
-}
-
-static void env_stack_pop_resolved(struct btf_verifier_env *env,
- u32 resolved_type_id,
- u32 resolved_size)
-{
- u32 type_id = env->stack[--(env->top_stack)].type_id;
- struct btf *btf = env->btf;
+ rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!rec)
+ return ERR_PTR(-ENOMEM);
- type_id -= btf->start_id; /* adjust to local type id */
- btf->resolved_sizes[type_id] = resolved_size;
- btf->resolved_ids[type_id] = resolved_type_id;
- env->visit_states[type_id] = RESOLVED;
-}
+ rec->spin_lock_off = -EINVAL;
+ rec->res_spin_lock_off = -EINVAL;
+ rec->timer_off = -EINVAL;
+ rec->wq_off = -EINVAL;
+ rec->refcount_off = -EINVAL;
+ rec->task_work_off = -EINVAL;
+ for (i = 0; i < cnt; i++) {
+ field_type_size = btf_field_type_size(info_arr[i].type);
+ if (info_arr[i].off + field_type_size > value_size) {
+ WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size);
+ ret = -EFAULT;
+ goto end;
+ }
+ if (info_arr[i].off < next_off) {
+ ret = -EEXIST;
+ goto end;
+ }
+ next_off = info_arr[i].off + field_type_size;
-static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
-{
- return env->top_stack ? &env->stack[env->top_stack - 1] : NULL;
-}
+ rec->field_mask |= info_arr[i].type;
+ rec->fields[i].offset = info_arr[i].off;
+ rec->fields[i].type = info_arr[i].type;
+ rec->fields[i].size = field_type_size;
-/* Resolve the size of a passed-in "type"
- *
- * type: is an array (e.g. u32 array[x][y])
- * return type: type "u32[x][y]", i.e. BTF_KIND_ARRAY,
- * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
- * corresponds to the return type.
- * *elem_type: u32
- * *elem_id: id of u32
- * *total_nelems: (x * y). Hence, individual elem size is
- * (*type_size / *total_nelems)
- * *type_id: id of type if it's changed within the function, 0 if not
- *
- * type: is not an array (e.g. const struct X)
- * return type: type "struct X"
- * *type_size: sizeof(struct X)
- * *elem_type: same as return type ("struct X")
- * *elem_id: 0
- * *total_nelems: 1
- * *type_id: id of type if it's changed within the function, 0 if not
- */
-static const struct btf_type *
-__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
- u32 *type_size, const struct btf_type **elem_type,
- u32 *elem_id, u32 *total_nelems, u32 *type_id)
-{
- const struct btf_type *array_type = NULL;
- const struct btf_array *array = NULL;
- u32 i, size, nelems = 1, id = 0;
-
- for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
- switch (BTF_INFO_KIND(type->info)) {
- /* type->size can be used */
- case BTF_KIND_INT:
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- case BTF_KIND_ENUM:
- case BTF_KIND_FLOAT:
- case BTF_KIND_ENUM64:
- size = type->size;
- goto resolved;
-
- case BTF_KIND_PTR:
- size = sizeof(void *);
- goto resolved;
-
- /* Modifiers */
- case BTF_KIND_TYPEDEF:
- case BTF_KIND_VOLATILE:
- case BTF_KIND_CONST:
- case BTF_KIND_RESTRICT:
- case BTF_KIND_TYPE_TAG:
- id = type->type;
- type = btf_type_by_id(btf, type->type);
+ switch (info_arr[i].type) {
+ case BPF_SPIN_LOCK:
+ WARN_ON_ONCE(rec->spin_lock_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->spin_lock_off = rec->fields[i].offset;
break;
-
- case BTF_KIND_ARRAY:
- if (!array_type)
- array_type = type;
- array = btf_type_array(type);
- if (nelems && array->nelems > U32_MAX / nelems)
- return ERR_PTR(-EINVAL);
- nelems *= array->nelems;
- type = btf_type_by_id(btf, array->type);
+ case BPF_RES_SPIN_LOCK:
+ WARN_ON_ONCE(rec->spin_lock_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->res_spin_lock_off = rec->fields[i].offset;
+ break;
+ case BPF_TIMER:
+ WARN_ON_ONCE(rec->timer_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->timer_off = rec->fields[i].offset;
+ break;
+ case BPF_WORKQUEUE:
+ WARN_ON_ONCE(rec->wq_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->wq_off = rec->fields[i].offset;
+ break;
+ case BPF_TASK_WORK:
+ WARN_ON_ONCE(rec->task_work_off >= 0);
+ rec->task_work_off = rec->fields[i].offset;
+ break;
+ case BPF_REFCOUNT:
+ WARN_ON_ONCE(rec->refcount_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->refcount_off = rec->fields[i].offset;
+ break;
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_LIST_HEAD:
+ ret = btf_parse_list_head(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_RB_ROOT:
+ ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_LIST_NODE:
+ case BPF_RB_NODE:
break;
-
- /* type without size */
default:
- return ERR_PTR(-EINVAL);
+ ret = -EFAULT;
+ goto end;
}
+ rec->cnt++;
}
- return ERR_PTR(-EINVAL);
-
-resolved:
- if (nelems && size > U32_MAX / nelems)
- return ERR_PTR(-EINVAL);
-
- *type_size = nelems * size;
- if (total_nelems)
- *total_nelems = nelems;
- if (elem_type)
- *elem_type = type;
- if (elem_id)
- *elem_id = array ? array->type : 0;
- if (type_id && id)
- *type_id = id;
-
- return array_type ? : type;
-}
-
-const struct btf_type *
-btf_resolve_size(const struct btf *btf, const struct btf_type *type,
- u32 *type_size)
-{
- return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
-}
-
-static u32 btf_resolved_type_id(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
-
- return btf->resolved_ids[type_id - btf->start_id];
-}
-
-/* The input param "type_id" must point to a needs_resolve type */
-static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
- u32 *type_id)
-{
- *type_id = btf_resolved_type_id(btf, *type_id);
- return btf_type_by_id(btf, *type_id);
-}
-
-static u32 btf_resolved_type_size(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
-
- return btf->resolved_sizes[type_id - btf->start_id];
-}
-
-const struct btf_type *btf_type_id_size(const struct btf *btf,
- u32 *type_id, u32 *ret_size)
-{
- const struct btf_type *size_type;
- u32 size_type_id = *type_id;
- u32 size = 0;
-
- size_type = btf_type_by_id(btf, size_type_id);
- if (btf_type_nosize_or_null(size_type))
- return NULL;
-
- if (btf_type_has_size(size_type)) {
- size = size_type->size;
- } else if (btf_type_is_array(size_type)) {
- size = btf_resolved_type_size(btf, size_type_id);
- } else if (btf_type_is_ptr(size_type)) {
- size = sizeof(void *);
- } else {
- if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) &&
- !btf_type_is_var(size_type)))
- return NULL;
-
- size_type_id = btf_resolved_type_id(btf, size_type_id);
- size_type = btf_type_by_id(btf, size_type_id);
- if (btf_type_nosize_or_null(size_type))
- return NULL;
- else if (btf_type_has_size(size_type))
- size = size_type->size;
- else if (btf_type_is_array(size_type))
- size = btf_resolved_type_size(btf, size_type_id);
- else if (btf_type_is_ptr(size_type))
- size = sizeof(void *);
- else
- return NULL;
- }
-
- *type_id = size_type_id;
- if (ret_size)
- *ret_size = size;
-
- return size_type;
-}
-
-static int btf_df_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- btf_verifier_log_basic(env, struct_type,
- "Unsupported check_member");
- return -EINVAL;
-}
-
-static int btf_df_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- btf_verifier_log_basic(env, struct_type,
- "Unsupported check_kflag_member");
- return -EINVAL;
-}
-
-/* Used for ptr, array struct/union and float type members.
- * int, enum and modifier types have their specific callback functions.
- */
-static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- /* bitfield size is 0, so member->offset represents bit offset only.
- * It is safe to call non kflag check_member variants.
- */
- return btf_type_ops(member_type)->check_member(env, struct_type,
- member,
- member_type);
-}
-
-static int btf_df_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- btf_verifier_log_basic(env, v->t, "Unsupported resolve");
- return -EINVAL;
-}
-
-static void btf_df_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offsets,
- struct btf_show *show)
-{
- btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
-}
-
-static int btf_int_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 int_data = btf_type_int(member_type);
- u32 struct_bits_off = member->offset;
- u32 struct_size = struct_type->size;
- u32 nr_copy_bits;
- u32 bytes_offset;
-
- if (U32_MAX - struct_bits_off < BTF_INT_OFFSET(int_data)) {
- btf_verifier_log_member(env, struct_type, member,
- "bits_offset exceeds U32_MAX");
- return -EINVAL;
- }
-
- struct_bits_off += BTF_INT_OFFSET(int_data);
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- nr_copy_bits = BTF_INT_BITS(int_data) +
- BITS_PER_BYTE_MASKED(struct_bits_off);
-
- if (nr_copy_bits > BITS_PER_U128) {
- btf_verifier_log_member(env, struct_type, member,
- "nr_copy_bits exceeds 128");
- return -EINVAL;
- }
-
- if (struct_size < bytes_offset ||
- struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_int_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
- u32 int_data = btf_type_int(member_type);
- u32 struct_size = struct_type->size;
- u32 nr_copy_bits;
-
- /* a regular int type is required for the kflag int member */
- if (!btf_type_int_is_regular(member_type)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member base type");
- return -EINVAL;
- }
-
- /* check sanity of bitfield size */
- nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
- struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
- nr_int_data_bits = BTF_INT_BITS(int_data);
- if (!nr_bits) {
- /* Not a bitfield member, member offset must be at byte
- * boundary.
- */
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member offset");
- return -EINVAL;
- }
-
- nr_bits = nr_int_data_bits;
- } else if (nr_bits > nr_int_data_bits) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
- if (nr_copy_bits > BITS_PER_U128) {
- btf_verifier_log_member(env, struct_type, member,
- "nr_copy_bits exceeds 128");
- return -EINVAL;
- }
-
- if (struct_size < bytes_offset ||
- struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_int_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 int_data, nr_bits, meta_needed = sizeof(int_data);
- u16 encoding;
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- int_data = btf_type_int(t);
- if (int_data & ~BTF_INT_MASK) {
- btf_verifier_log_basic(env, t, "Invalid int_data:%x",
- int_data);
- return -EINVAL;
- }
-
- nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
-
- if (nr_bits > BITS_PER_U128) {
- btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
- BITS_PER_U128);
- return -EINVAL;
- }
-
- if (BITS_ROUNDUP_BYTES(nr_bits) > t->size) {
- btf_verifier_log_type(env, t, "nr_bits exceeds type_size");
- return -EINVAL;
- }
-
- /*
- * Only one of the encoding bits is allowed and it
- * should be sufficient for the pretty print purpose (i.e. decoding).
- * Multiple bits can be allowed later if it is found
- * to be insufficient.
- */
- encoding = BTF_INT_ENCODING(int_data);
- if (encoding &&
- encoding != BTF_INT_SIGNED &&
- encoding != BTF_INT_CHAR &&
- encoding != BTF_INT_BOOL) {
- btf_verifier_log_type(env, t, "Unsupported encoding");
- return -ENOTSUPP;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_int_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- int int_data = btf_type_int(t);
-
- btf_verifier_log(env,
- "size=%u bits_offset=%u nr_bits=%u encoding=%s",
- t->size, BTF_INT_OFFSET(int_data),
- BTF_INT_BITS(int_data),
- btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
-}
-
-static void btf_int128_print(struct btf_show *show, void *data)
-{
- /* data points to a __int128 number.
- * Suppose
- * int128_num = *(__int128 *)data;
- * The below formulas shows what upper_num and lower_num represents:
- * upper_num = int128_num >> 64;
- * lower_num = int128_num & 0xffffffffFFFFFFFFULL;
- */
- u64 upper_num, lower_num;
-
-#ifdef __BIG_ENDIAN_BITFIELD
- upper_num = *(u64 *)data;
- lower_num = *(u64 *)(data + 8);
-#else
- upper_num = *(u64 *)(data + 8);
- lower_num = *(u64 *)data;
-#endif
- if (upper_num == 0)
- btf_show_type_value(show, "0x%llx", lower_num);
- else
- btf_show_type_values(show, "0x%llx%016llx", upper_num,
- lower_num);
-}
-
-static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
- u16 right_shift_bits)
-{
- u64 upper_num, lower_num;
-
-#ifdef __BIG_ENDIAN_BITFIELD
- upper_num = print_num[0];
- lower_num = print_num[1];
-#else
- upper_num = print_num[1];
- lower_num = print_num[0];
-#endif
-
- /* shake out un-needed bits by shift/or operations */
- if (left_shift_bits >= 64) {
- upper_num = lower_num << (left_shift_bits - 64);
- lower_num = 0;
- } else {
- upper_num = (upper_num << left_shift_bits) |
- (lower_num >> (64 - left_shift_bits));
- lower_num = lower_num << left_shift_bits;
- }
-
- if (right_shift_bits >= 64) {
- lower_num = upper_num >> (right_shift_bits - 64);
- upper_num = 0;
- } else {
- lower_num = (lower_num >> right_shift_bits) |
- (upper_num << (64 - right_shift_bits));
- upper_num = upper_num >> right_shift_bits;
- }
-
-#ifdef __BIG_ENDIAN_BITFIELD
- print_num[0] = upper_num;
- print_num[1] = lower_num;
-#else
- print_num[0] = lower_num;
- print_num[1] = upper_num;
-#endif
-}
-
-static void btf_bitfield_show(void *data, u8 bits_offset,
- u8 nr_bits, struct btf_show *show)
-{
- u16 left_shift_bits, right_shift_bits;
- u8 nr_copy_bytes;
- u8 nr_copy_bits;
- u64 print_num[2] = {};
-
- nr_copy_bits = nr_bits + bits_offset;
- nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
-
- memcpy(print_num, data, nr_copy_bytes);
-
-#ifdef __BIG_ENDIAN_BITFIELD
- left_shift_bits = bits_offset;
-#else
- left_shift_bits = BITS_PER_U128 - nr_copy_bits;
-#endif
- right_shift_bits = BITS_PER_U128 - nr_bits;
-
- btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
- btf_int128_print(show, print_num);
-}
-
-
-static void btf_int_bits_show(const struct btf *btf,
- const struct btf_type *t,
- void *data, u8 bits_offset,
- struct btf_show *show)
-{
- u32 int_data = btf_type_int(t);
- u8 nr_bits = BTF_INT_BITS(int_data);
- u8 total_bits_offset;
-
- /*
- * bits_offset is at most 7.
- * BTF_INT_OFFSET() cannot exceed 128 bits.
- */
- total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
- data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
- bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
- btf_bitfield_show(data, bits_offset, nr_bits, show);
-}
-
-static void btf_int_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- u32 int_data = btf_type_int(t);
- u8 encoding = BTF_INT_ENCODING(int_data);
- bool sign = encoding & BTF_INT_SIGNED;
- u8 nr_bits = BTF_INT_BITS(int_data);
- void *safe_data;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- if (bits_offset || BTF_INT_OFFSET(int_data) ||
- BITS_PER_BYTE_MASKED(nr_bits)) {
- btf_int_bits_show(btf, t, safe_data, bits_offset, show);
- goto out;
- }
-
- switch (nr_bits) {
- case 128:
- btf_int128_print(show, safe_data);
- break;
- case 64:
- if (sign)
- btf_show_type_value(show, "%lld", *(s64 *)safe_data);
- else
- btf_show_type_value(show, "%llu", *(u64 *)safe_data);
- break;
- case 32:
- if (sign)
- btf_show_type_value(show, "%d", *(s32 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u32 *)safe_data);
- break;
- case 16:
- if (sign)
- btf_show_type_value(show, "%d", *(s16 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u16 *)safe_data);
- break;
- case 8:
- if (show->state.array_encoding == BTF_INT_CHAR) {
- /* check for null terminator */
- if (show->state.array_terminated)
- break;
- if (*(char *)data == '\0') {
- show->state.array_terminated = 1;
- break;
- }
- if (isprint(*(char *)data)) {
- btf_show_type_value(show, "'%c'",
- *(char *)safe_data);
- break;
- }
- }
- if (sign)
- btf_show_type_value(show, "%d", *(s8 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u8 *)safe_data);
- break;
- default:
- btf_int_bits_show(btf, t, safe_data, bits_offset, show);
- break;
- }
-out:
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations int_ops = {
- .check_meta = btf_int_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_int_check_member,
- .check_kflag_member = btf_int_check_kflag_member,
- .log_details = btf_int_log,
- .show = btf_int_show,
-};
-
-static int btf_modifier_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- const struct btf_type *resolved_type;
- u32 resolved_type_id = member->type;
- struct btf_member resolved_member;
- struct btf *btf = env->btf;
-
- resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
- if (!resolved_type) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member");
- return -EINVAL;
- }
-
- resolved_member = *member;
- resolved_member.type = resolved_type_id;
-
- return btf_type_ops(resolved_type)->check_member(env, struct_type,
- &resolved_member,
- resolved_type);
-}
-
-static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- const struct btf_type *resolved_type;
- u32 resolved_type_id = member->type;
- struct btf_member resolved_member;
- struct btf *btf = env->btf;
-
- resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
- if (!resolved_type) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member");
- return -EINVAL;
- }
-
- resolved_member = *member;
- resolved_member.type = resolved_type_id;
-
- return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
- &resolved_member,
- resolved_type);
-}
-
-static int btf_ptr_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_size, struct_bits_off, bytes_offset;
-
- struct_size = struct_type->size;
- struct_bits_off = member->offset;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- if (struct_size - bytes_offset < sizeof(void *)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_ref_type_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const char *value;
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t) && !btf_type_is_type_tag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!BTF_TYPE_ID_VALID(t->type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- /* typedef/type_tag type must have a valid name, and other ref types,
- * volatile, const, restrict, should have a null name.
- */
- if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- } else if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPE_TAG) {
- value = btf_name_by_offset(env->btf, t->name_off);
- if (!value || !value[0]) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- } else {
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_modifier_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *t = v->t;
- const struct btf_type *next_type;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- /* Figure out the resolved next_type_id with size.
- * They will be stored in the current modifier's
- * resolved_ids and resolved_sizes such that it can
- * save us a few type-following when we use it later (e.g. in
- * pretty print).
- */
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- if (env_type_is_resolved(env, next_type_id))
- next_type = btf_type_id_resolve(btf, &next_type_id);
-
- /* "typedef void new_void", "const void"...etc */
- if (!btf_type_is_void(next_type) &&
- !btf_type_is_fwd(next_type) &&
- !btf_type_is_func_proto(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static int btf_var_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- if (btf_type_is_modifier(next_type)) {
- const struct btf_type *resolved_type;
- u32 resolved_type_id;
-
- resolved_type_id = next_type_id;
- resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
-
- if (btf_type_is_ptr(resolved_type) &&
- !env_type_is_resolve_sink(env, resolved_type) &&
- !env_type_is_resolved(env, resolved_type_id))
- return env_stack_push(env, resolved_type,
- resolved_type_id);
- }
-
- /* We must resolve to something concrete at this point, no
- * forward types or similar that would resolve to size of
- * zero is allowed.
- */
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static int btf_ptr_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- /* If the modifier was RESOLVED during RESOLVE_STRUCT_OR_ARRAY,
- * the modifier may have stopped resolving when it was resolved
- * to a ptr (last-resolved-ptr).
- *
- * We now need to continue from the last-resolved-ptr to
- * ensure the last-resolved-ptr will not referring back to
- * the current ptr (t).
- */
- if (btf_type_is_modifier(next_type)) {
- const struct btf_type *resolved_type;
- u32 resolved_type_id;
-
- resolved_type_id = next_type_id;
- resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
-
- if (btf_type_is_ptr(resolved_type) &&
- !env_type_is_resolve_sink(env, resolved_type) &&
- !env_type_is_resolved(env, resolved_type_id))
- return env_stack_push(env, resolved_type,
- resolved_type_id);
- }
-
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- if (env_type_is_resolved(env, next_type_id))
- next_type = btf_type_id_resolve(btf, &next_type_id);
-
- if (!btf_type_is_void(next_type) &&
- !btf_type_is_fwd(next_type) &&
- !btf_type_is_func_proto(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static void btf_modifier_show(const struct btf *btf,
- const struct btf_type *t,
- u32 type_id, void *data,
- u8 bits_offset, struct btf_show *show)
-{
- if (btf->resolved_ids)
- t = btf_type_id_resolve(btf, &type_id);
- else
- t = btf_type_skip_modifiers(btf, type_id, NULL);
-
- btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
-}
-
-static void btf_var_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- t = btf_type_id_resolve(btf, &type_id);
-
- btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
-}
-
-static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- void *safe_data;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
- if (show->flags & BTF_SHOW_PTR_RAW)
- btf_show_type_value(show, "0x%px", *(void **)safe_data);
- else
- btf_show_type_value(show, "0x%p", *(void **)safe_data);
- btf_show_end_type(show);
-}
-
-static void btf_ref_type_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "type_id=%u", t->type);
-}
-
-static const struct btf_kind_operations modifier_ops = {
- .check_meta = btf_ref_type_check_meta,
- .resolve = btf_modifier_resolve,
- .check_member = btf_modifier_check_member,
- .check_kflag_member = btf_modifier_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_modifier_show,
-};
-
-static const struct btf_kind_operations ptr_ops = {
- .check_meta = btf_ref_type_check_meta,
- .resolve = btf_ptr_resolve,
- .check_member = btf_ptr_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_ptr_show,
-};
-
-static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (t->type) {
- btf_verifier_log_type(env, t, "type != 0");
- return -EINVAL;
- }
-
- /* fwd type must have a valid name */
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static void btf_fwd_type_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "%s", btf_type_kflag(t) ? "union" : "struct");
-}
-
-static const struct btf_kind_operations fwd_ops = {
- .check_meta = btf_fwd_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_fwd_type_log,
- .show = btf_df_show,
-};
-
-static int btf_array_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
- u32 array_type_id, array_size;
- struct btf *btf = env->btf;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- array_type_id = member->type;
- btf_type_id_size(btf, &array_type_id, &array_size);
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < array_size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_array_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_array *array = btf_type_array(t);
- u32 meta_needed = sizeof(*array);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- /* array type should not have a name */
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (t->size) {
- btf_verifier_log_type(env, t, "size != 0");
- return -EINVAL;
- }
-
- /* Array elem type and index type cannot be in type void,
- * so !array->type and !array->index_type are not allowed.
- */
- if (!array->type || !BTF_TYPE_ID_VALID(array->type)) {
- btf_verifier_log_type(env, t, "Invalid elem");
- return -EINVAL;
- }
-
- if (!array->index_type || !BTF_TYPE_ID_VALID(array->index_type)) {
- btf_verifier_log_type(env, t, "Invalid index");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static int btf_array_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_array *array = btf_type_array(v->t);
- const struct btf_type *elem_type, *index_type;
- u32 elem_type_id, index_type_id;
- struct btf *btf = env->btf;
- u32 elem_size;
-
- /* Check array->index_type */
- index_type_id = array->index_type;
- index_type = btf_type_by_id(btf, index_type_id);
- if (btf_type_nosize_or_null(index_type) ||
- btf_type_is_resolve_source_only(index_type)) {
- btf_verifier_log_type(env, v->t, "Invalid index");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, index_type) &&
- !env_type_is_resolved(env, index_type_id))
- return env_stack_push(env, index_type, index_type_id);
-
- index_type = btf_type_id_size(btf, &index_type_id, NULL);
- if (!index_type || !btf_type_is_int(index_type) ||
- !btf_type_int_is_regular(index_type)) {
- btf_verifier_log_type(env, v->t, "Invalid index");
- return -EINVAL;
- }
-
- /* Check array->type */
- elem_type_id = array->type;
- elem_type = btf_type_by_id(btf, elem_type_id);
- if (btf_type_nosize_or_null(elem_type) ||
- btf_type_is_resolve_source_only(elem_type)) {
- btf_verifier_log_type(env, v->t,
- "Invalid elem");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, elem_type) &&
- !env_type_is_resolved(env, elem_type_id))
- return env_stack_push(env, elem_type, elem_type_id);
-
- elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
- if (!elem_type) {
- btf_verifier_log_type(env, v->t, "Invalid elem");
- return -EINVAL;
- }
-
- if (btf_type_is_int(elem_type) && !btf_type_int_is_regular(elem_type)) {
- btf_verifier_log_type(env, v->t, "Invalid array of int");
- return -EINVAL;
- }
-
- if (array->nelems && elem_size > U32_MAX / array->nelems) {
- btf_verifier_log_type(env, v->t,
- "Array size overflows U32_MAX");
- return -EINVAL;
- }
-
- env_stack_pop_resolved(env, elem_type_id, elem_size * array->nelems);
-
- return 0;
-}
-
-static void btf_array_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_array *array = btf_type_array(t);
-
- btf_verifier_log(env, "type_id=%u index_type_id=%u nr_elems=%u",
- array->type, array->index_type, array->nelems);
-}
-
-static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_array *array = btf_type_array(t);
- const struct btf_kind_operations *elem_ops;
- const struct btf_type *elem_type;
- u32 i, elem_size = 0, elem_type_id;
- u16 encoding = 0;
-
- elem_type_id = array->type;
- elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
- if (elem_type && btf_type_has_size(elem_type))
- elem_size = elem_type->size;
-
- if (elem_type && btf_type_is_int(elem_type)) {
- u32 int_type = btf_type_int(elem_type);
-
- encoding = BTF_INT_ENCODING(int_type);
-
- /*
- * BTF_INT_CHAR encoding never seems to be set for
- * char arrays, so if size is 1 and element is
- * printable as a char, we'll do that.
- */
- if (elem_size == 1)
- encoding = BTF_INT_CHAR;
- }
-
- if (!btf_show_start_array_type(show, t, type_id, encoding, data))
- return;
-
- if (!elem_type)
- goto out;
- elem_ops = btf_type_ops(elem_type);
-
- for (i = 0; i < array->nelems; i++) {
-
- btf_show_start_array_member(show);
-
- elem_ops->show(btf, elem_type, elem_type_id, data,
- bits_offset, show);
- data += elem_size;
-
- btf_show_end_array_member(show);
-
- if (show->state.array_terminated)
- break;
- }
-out:
- btf_show_end_array_type(show);
-}
-
-static void btf_array_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *m = show->state.member;
-
- /*
- * First check if any members would be shown (are non-zero).
- * See comments above "struct btf_show" definition for more
- * details on how this works at a high-level.
- */
- if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
- if (!show->state.depth_check) {
- show->state.depth_check = show->state.depth + 1;
- show->state.depth_to_show = 0;
- }
- __btf_array_show(btf, t, type_id, data, bits_offset, show);
- show->state.member = m;
-
- if (show->state.depth_check != show->state.depth + 1)
- return;
- show->state.depth_check = 0;
-
- if (show->state.depth_to_show <= show->state.depth)
- return;
- /*
- * Reaching here indicates we have recursed and found
- * non-zero array member(s).
- */
- }
- __btf_array_show(btf, t, type_id, data, bits_offset, show);
-}
-
-static const struct btf_kind_operations array_ops = {
- .check_meta = btf_array_check_meta,
- .resolve = btf_array_resolve,
- .check_member = btf_array_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_array_log,
- .show = btf_array_show,
-};
-
-static int btf_struct_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < member_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_struct_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- bool is_union = BTF_INFO_KIND(t->info) == BTF_KIND_UNION;
- const struct btf_member *member;
- u32 meta_needed, last_offset;
- struct btf *btf = env->btf;
- u32 struct_size = t->size;
- u32 offset;
- u16 i;
-
- meta_needed = btf_type_vlen(t) * sizeof(*member);
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- /* struct type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- last_offset = 0;
- for_each_member(i, t, member) {
- if (!btf_name_offset_valid(btf, member->name_off)) {
- btf_verifier_log_member(env, t, member,
- "Invalid member name_offset:%u",
- member->name_off);
- return -EINVAL;
- }
-
- /* struct member either no name or a valid one */
- if (member->name_off &&
- !btf_name_valid_identifier(btf, member->name_off)) {
- btf_verifier_log_member(env, t, member, "Invalid name");
- return -EINVAL;
- }
- /* A member cannot be in type void */
- if (!member->type || !BTF_TYPE_ID_VALID(member->type)) {
- btf_verifier_log_member(env, t, member,
- "Invalid type_id");
- return -EINVAL;
- }
-
- offset = __btf_member_bit_offset(t, member);
- if (is_union && offset) {
- btf_verifier_log_member(env, t, member,
- "Invalid member bits_offset");
- return -EINVAL;
- }
-
- /*
- * ">" instead of ">=" because the last member could be
- * "char a[0];"
- */
- if (last_offset > offset) {
- btf_verifier_log_member(env, t, member,
- "Invalid member bits_offset");
- return -EINVAL;
- }
-
- if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
- btf_verifier_log_member(env, t, member,
- "Member bits_offset exceeds its struct size");
- return -EINVAL;
- }
-
- btf_verifier_log_member(env, t, member, NULL);
- last_offset = offset;
- }
-
- return meta_needed;
-}
-
-static int btf_struct_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_member *member;
- int err;
- u16 i;
-
- /* Before continue resolving the next_member,
- * ensure the last member is indeed resolved to a
- * type with size info.
- */
- if (v->next_member) {
- const struct btf_type *last_member_type;
- const struct btf_member *last_member;
- u32 last_member_type_id;
-
- last_member = btf_type_member(v->t) + v->next_member - 1;
- last_member_type_id = last_member->type;
- if (WARN_ON_ONCE(!env_type_is_resolved(env,
- last_member_type_id)))
- return -EINVAL;
-
- last_member_type = btf_type_by_id(env->btf,
- last_member_type_id);
- if (btf_type_kflag(v->t))
- err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
- last_member,
- last_member_type);
- else
- err = btf_type_ops(last_member_type)->check_member(env, v->t,
- last_member,
- last_member_type);
- if (err)
- return err;
- }
-
- for_each_member_from(i, v->next_member, v->t, member) {
- u32 member_type_id = member->type;
- const struct btf_type *member_type = btf_type_by_id(env->btf,
- member_type_id);
-
- if (btf_type_nosize_or_null(member_type) ||
- btf_type_is_resolve_source_only(member_type)) {
- btf_verifier_log_member(env, v->t, member,
- "Invalid member");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, member_type) &&
- !env_type_is_resolved(env, member_type_id)) {
- env_stack_set_next_member(env, i + 1);
- return env_stack_push(env, member_type, member_type_id);
- }
-
- if (btf_type_kflag(v->t))
- err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
- member,
- member_type);
- else
- err = btf_type_ops(member_type)->check_member(env, v->t,
- member,
- member_type);
- if (err)
- return err;
- }
-
- env_stack_pop_resolved(env, 0, 0);
-
- return 0;
-}
-
-static void btf_struct_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-enum {
- BTF_FIELD_IGNORE = 0,
- BTF_FIELD_FOUND = 1,
-};
-
-struct btf_field_info {
- enum btf_field_type type;
- u32 off;
- union {
- struct {
- u32 type_id;
- } kptr;
- struct {
- const char *node_name;
- u32 value_btf_id;
- } graph_root;
- };
-};
-
-static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
- u32 off, int sz, enum btf_field_type field_type,
- struct btf_field_info *info)
-{
- if (!__btf_type_is_struct(t))
- return BTF_FIELD_IGNORE;
- if (t->size != sz)
- return BTF_FIELD_IGNORE;
- info->type = field_type;
- info->off = off;
- return BTF_FIELD_FOUND;
-}
-
-static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
- u32 off, int sz, struct btf_field_info *info, u32 field_mask)
-{
- enum btf_field_type type;
- const char *tag_value;
- bool is_type_tag;
- u32 res_id;
-
- /* Permit modifiers on the pointer itself */
- if (btf_type_is_volatile(t))
- t = btf_type_by_id(btf, t->type);
- /* For PTR, sz is always == 8 */
- if (!btf_type_is_ptr(t))
- return BTF_FIELD_IGNORE;
- t = btf_type_by_id(btf, t->type);
- is_type_tag = btf_type_is_type_tag(t) && !btf_type_kflag(t);
- if (!is_type_tag)
- return BTF_FIELD_IGNORE;
- /* Reject extra tags */
- if (btf_type_is_type_tag(btf_type_by_id(btf, t->type)))
- return -EINVAL;
- tag_value = __btf_name_by_offset(btf, t->name_off);
- if (!strcmp("kptr_untrusted", tag_value))
- type = BPF_KPTR_UNREF;
- else if (!strcmp("kptr", tag_value))
- type = BPF_KPTR_REF;
- else if (!strcmp("percpu_kptr", tag_value))
- type = BPF_KPTR_PERCPU;
- else if (!strcmp("uptr", tag_value))
- type = BPF_UPTR;
- else
- return -EINVAL;
-
- if (!(type & field_mask))
- return BTF_FIELD_IGNORE;
-
- /* Get the base type */
- t = btf_type_skip_modifiers(btf, t->type, &res_id);
- /* Only pointer to struct is allowed */
- if (!__btf_type_is_struct(t))
- return -EINVAL;
-
- info->type = type;
- info->off = off;
- info->kptr.type_id = res_id;
- return BTF_FIELD_FOUND;
-}
-
-int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
- int comp_idx, const char *tag_key, int last_id)
-{
- int len = strlen(tag_key);
- int i, n;
-
- for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
- const struct btf_type *t = btf_type_by_id(btf, i);
-
- if (!btf_type_is_decl_tag(t))
- continue;
- if (pt != btf_type_by_id(btf, t->type))
- continue;
- if (btf_type_decl_tag(t)->component_idx != comp_idx)
- continue;
- if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
- continue;
- return i;
- }
- return -ENOENT;
-}
-
-const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
- int comp_idx, const char *tag_key)
-{
- const char *value = NULL;
- const struct btf_type *t;
- int len, id;
-
- id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
- btf_named_start_id(btf, false) - 1);
- if (id < 0)
- return ERR_PTR(id);
-
- t = btf_type_by_id(btf, id);
- len = strlen(tag_key);
- value = __btf_name_by_offset(btf, t->name_off) + len;
-
- /* Prevent duplicate entries for same type */
- id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, id);
- if (id >= 0)
- return ERR_PTR(-EEXIST);
-
- return value;
-}
-
-static int
-btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
- const struct btf_type *t, int comp_idx, u32 off,
- int sz, struct btf_field_info *info,
- enum btf_field_type head_type)
-{
- const char *node_field_name;
- const char *value_type;
- s32 id;
-
- if (!__btf_type_is_struct(t))
- return BTF_FIELD_IGNORE;
- if (t->size != sz)
- return BTF_FIELD_IGNORE;
- value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
- if (IS_ERR(value_type))
- return -EINVAL;
- node_field_name = strstr(value_type, ":");
- if (!node_field_name)
- return -EINVAL;
- value_type = kstrndup(value_type, node_field_name - value_type,
- GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
- if (!value_type)
- return -ENOMEM;
- id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
- kfree(value_type);
- if (id < 0)
- return id;
- node_field_name++;
- if (str_is_empty(node_field_name))
- return -EINVAL;
- info->type = head_type;
- info->off = off;
- info->graph_root.value_btf_id = id;
- info->graph_root.node_name = node_field_name;
- return BTF_FIELD_FOUND;
-}
-
-static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_type,
- u32 field_mask, u32 *seen_mask, int *align, int *sz)
-{
- const struct {
- enum btf_field_type type;
- const char *const name;
- const bool is_unique;
- } field_types[] = {
- { BPF_SPIN_LOCK, "bpf_spin_lock", true },
- { BPF_RES_SPIN_LOCK, "bpf_res_spin_lock", true },
- { BPF_TIMER, "bpf_timer", true },
- { BPF_WORKQUEUE, "bpf_wq", true },
- { BPF_TASK_WORK, "bpf_task_work", true },
- { BPF_LIST_HEAD, "bpf_list_head", false },
- { BPF_LIST_NODE, "bpf_list_node", false },
- { BPF_RB_ROOT, "bpf_rb_root", false },
- { BPF_RB_NODE, "bpf_rb_node", false },
- { BPF_REFCOUNT, "bpf_refcount", false },
- };
- int type = 0, i;
- const char *name = __btf_name_by_offset(btf, var_type->name_off);
- const char *field_type_name;
- enum btf_field_type field_type;
- bool is_unique;
-
- for (i = 0; i < ARRAY_SIZE(field_types); ++i) {
- field_type = field_types[i].type;
- field_type_name = field_types[i].name;
- is_unique = field_types[i].is_unique;
- if (!(field_mask & field_type) || strcmp(name, field_type_name))
- continue;
- if (is_unique) {
- if (*seen_mask & field_type)
- return -E2BIG;
- *seen_mask |= field_type;
- }
- type = field_type;
- goto end;
- }
-
- /* Only return BPF_KPTR when all other types with matchable names fail */
- if (field_mask & (BPF_KPTR | BPF_UPTR) && !__btf_type_is_struct(var_type)) {
- type = BPF_KPTR_REF;
- goto end;
- }
- return 0;
-end:
- *sz = btf_field_type_size(type);
- *align = btf_field_type_align(type);
- return type;
-}
-
-/* Repeat a number of fields for a specified number of times.
- *
- * Copy the fields starting from the first field and repeat them for
- * repeat_cnt times. The fields are repeated by adding the offset of each
- * field with
- * (i + 1) * elem_size
- * where i is the repeat index and elem_size is the size of an element.
- */
-static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
- u32 field_cnt, u32 repeat_cnt, u32 elem_size)
-{
- u32 i, j;
- u32 cur;
-
- /* Ensure not repeating fields that should not be repeated. */
- for (i = 0; i < field_cnt; i++) {
- switch (info[i].type) {
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- case BPF_LIST_HEAD:
- case BPF_RB_ROOT:
- break;
- default:
- return -EINVAL;
- }
- }
-
- /* The type of struct size or variable size is u32,
- * so the multiplication will not overflow.
- */
- if (field_cnt * (repeat_cnt + 1) > info_cnt)
- return -E2BIG;
-
- cur = field_cnt;
- for (i = 0; i < repeat_cnt; i++) {
- memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
- for (j = 0; j < field_cnt; j++)
- info[cur++].off += (i + 1) * elem_size;
- }
-
- return 0;
-}
-
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level);
-
-/* Find special fields in the struct type of a field.
- *
- * This function is used to find fields of special types that is not a
- * global variable or a direct field of a struct type. It also handles the
- * repetition if it is the element type of an array.
- */
-static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
- u32 off, u32 nelems,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level)
-{
- int ret, err, i;
-
- level++;
- if (level >= MAX_RESOLVE_DEPTH)
- return -E2BIG;
-
- ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
-
- if (ret <= 0)
- return ret;
-
- /* Shift the offsets of the nested struct fields to the offsets
- * related to the container.
- */
- for (i = 0; i < ret; i++)
- info[i].off += off;
-
- if (nelems > 1) {
- err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
- if (err == 0)
- ret *= nelems;
- else
- ret = err;
- }
-
- return ret;
-}
-
-static int btf_find_field_one(const struct btf *btf,
- const struct btf_type *var,
- const struct btf_type *var_type,
- int var_idx,
- u32 off, u32 expected_size,
- u32 field_mask, u32 *seen_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level)
-{
- int ret, align, sz, field_type;
- struct btf_field_info tmp;
- const struct btf_array *array;
- u32 i, nelems = 1;
-
- /* Walk into array types to find the element type and the number of
- * elements in the (flattened) array.
- */
- for (i = 0; i < MAX_RESOLVE_DEPTH && btf_type_is_array(var_type); i++) {
- array = btf_array(var_type);
- nelems *= array->nelems;
- var_type = btf_type_by_id(btf, array->type);
- }
- if (i == MAX_RESOLVE_DEPTH)
- return -E2BIG;
- if (nelems == 0)
- return 0;
-
- field_type = btf_get_field_type(btf, var_type,
- field_mask, seen_mask, &align, &sz);
- /* Look into variables of struct types */
- if (!field_type && __btf_type_is_struct(var_type)) {
- sz = var_type->size;
- if (expected_size && expected_size != sz * nelems)
- return 0;
- ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
- &info[0], info_cnt, level);
- return ret;
- }
-
- if (field_type == 0)
- return 0;
- if (field_type < 0)
- return field_type;
-
- if (expected_size && expected_size != sz * nelems)
- return 0;
- if (off % align)
- return 0;
-
- switch (field_type) {
- case BPF_SPIN_LOCK:
- case BPF_RES_SPIN_LOCK:
- case BPF_TIMER:
- case BPF_WORKQUEUE:
- case BPF_LIST_NODE:
- case BPF_RB_NODE:
- case BPF_REFCOUNT:
- case BPF_TASK_WORK:
- ret = btf_find_struct(btf, var_type, off, sz, field_type,
- info_cnt ? &info[0] : &tmp);
- if (ret < 0)
- return ret;
- break;
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- ret = btf_find_kptr(btf, var_type, off, sz,
- info_cnt ? &info[0] : &tmp, field_mask);
- if (ret < 0)
- return ret;
- break;
- case BPF_LIST_HEAD:
- case BPF_RB_ROOT:
- ret = btf_find_graph_root(btf, var, var_type,
- var_idx, off, sz,
- info_cnt ? &info[0] : &tmp,
- field_type);
- if (ret < 0)
- return ret;
- break;
- default:
- return -EFAULT;
- }
-
- if (ret == BTF_FIELD_IGNORE)
- return 0;
- if (!info_cnt)
- return -E2BIG;
- if (nelems > 1) {
- ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
- if (ret < 0)
- return ret;
- }
- return nelems;
-}
-
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level)
-{
- int ret, idx = 0;
- const struct btf_member *member;
- u32 i, off, seen_mask = 0;
-
- for_each_member(i, t, member) {
- const struct btf_type *member_type = btf_type_by_id(btf,
- member->type);
-
- off = __btf_member_bit_offset(t, member);
- if (off % 8)
- /* valid C code cannot generate such BTF */
- return -EINVAL;
- off /= 8;
-
- ret = btf_find_field_one(btf, t, member_type, i,
- off, 0,
- field_mask, &seen_mask,
- &info[idx], info_cnt - idx, level);
- if (ret < 0)
- return ret;
- idx += ret;
- }
- return idx;
-}
-
-static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level)
-{
- int ret, idx = 0;
- const struct btf_var_secinfo *vsi;
- u32 i, off, seen_mask = 0;
-
- for_each_vsi(i, t, vsi) {
- const struct btf_type *var = btf_type_by_id(btf, vsi->type);
- const struct btf_type *var_type = btf_type_by_id(btf, var->type);
-
- off = vsi->offset;
- ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
- field_mask, &seen_mask,
- &info[idx], info_cnt - idx,
- level);
- if (ret < 0)
- return ret;
- idx += ret;
- }
- return idx;
-}
-
-static int btf_find_field(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt)
-{
- if (__btf_type_is_struct(t))
- return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
- else if (btf_type_is_datasec(t))
- return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
- return -EINVAL;
-}
-
-/* Callers have to ensure the life cycle of btf if it is program BTF */
-static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- struct module *mod = NULL;
- const struct btf_type *t;
- /* If a matching btf type is found in kernel or module BTFs, kptr_ref
- * is that BTF, otherwise it's program BTF
- */
- struct btf *kptr_btf;
- int ret;
- s32 id;
-
- /* Find type in map BTF, and use it to look up the matching type
- * in vmlinux or module BTFs, by name and kind.
- */
- t = btf_type_by_id(btf, info->kptr.type_id);
- id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
- &kptr_btf);
- if (id == -ENOENT) {
- /* btf_parse_kptr should only be called w/ btf = program BTF */
- WARN_ON_ONCE(btf_is_kernel(btf));
-
- /* Type exists only in program BTF. Assume that it's a MEM_ALLOC
- * kptr allocated via bpf_obj_new
- */
- field->kptr.dtor = NULL;
- id = info->kptr.type_id;
- kptr_btf = (struct btf *)btf;
- goto found_dtor;
- }
- if (id < 0)
- return id;
-
- /* Find and stash the function pointer for the destruction function that
- * needs to be eventually invoked from the map free path.
- */
- if (info->type == BPF_KPTR_REF) {
- const struct btf_type *dtor_func;
- const char *dtor_func_name;
- unsigned long addr;
- s32 dtor_btf_id;
-
- /* This call also serves as a whitelist of allowed objects that
- * can be used as a referenced pointer and be stored in a map at
- * the same time.
- */
- dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
- if (dtor_btf_id < 0) {
- ret = dtor_btf_id;
- goto end_btf;
- }
-
- dtor_func = btf_type_by_id(kptr_btf, dtor_btf_id);
- if (!dtor_func) {
- ret = -ENOENT;
- goto end_btf;
- }
-
- if (btf_is_module(kptr_btf)) {
- mod = btf_try_get_module(kptr_btf);
- if (!mod) {
- ret = -ENXIO;
- goto end_btf;
- }
- }
-
- /* We already verified dtor_func to be btf_type_is_func
- * in register_btf_id_dtor_kfuncs.
- */
- dtor_func_name = __btf_name_by_offset(kptr_btf, dtor_func->name_off);
- addr = kallsyms_lookup_name(dtor_func_name);
- if (!addr) {
- ret = -EINVAL;
- goto end_mod;
- }
- field->kptr.dtor = (void *)addr;
- }
-
-found_dtor:
- field->kptr.btf_id = id;
- field->kptr.btf = kptr_btf;
- field->kptr.module = mod;
- return 0;
-end_mod:
- module_put(mod);
-end_btf:
- btf_put(kptr_btf);
- return ret;
-}
-
-static int btf_parse_graph_root(const struct btf *btf,
- struct btf_field *field,
- struct btf_field_info *info,
- const char *node_type_name,
- size_t node_type_align)
-{
- const struct btf_type *t, *n = NULL;
- const struct btf_member *member;
- u32 offset;
- int i;
-
- t = btf_type_by_id(btf, info->graph_root.value_btf_id);
- /* We've already checked that value_btf_id is a struct type. We
- * just need to figure out the offset of the list_node, and
- * verify its type.
- */
- for_each_member(i, t, member) {
- if (strcmp(info->graph_root.node_name,
- __btf_name_by_offset(btf, member->name_off)))
- continue;
- /* Invalid BTF, two members with same name */
- if (n)
- return -EINVAL;
- n = btf_type_by_id(btf, member->type);
- if (!__btf_type_is_struct(n))
- return -EINVAL;
- if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off)))
- return -EINVAL;
- offset = __btf_member_bit_offset(n, member);
- if (offset % 8)
- return -EINVAL;
- offset /= 8;
- if (offset % node_type_align)
- return -EINVAL;
-
- field->graph_root.btf = (struct btf *)btf;
- field->graph_root.value_btf_id = info->graph_root.value_btf_id;
- field->graph_root.node_offset = offset;
- }
- if (!n)
- return -ENOENT;
- return 0;
-}
-
-static int btf_parse_list_head(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- return btf_parse_graph_root(btf, field, info, "bpf_list_node",
- __alignof__(struct bpf_list_node));
-}
-
-static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- return btf_parse_graph_root(btf, field, info, "bpf_rb_node",
- __alignof__(struct bpf_rb_node));
-}
-
-static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
-{
- const struct btf_field *a = (const struct btf_field *)_a;
- const struct btf_field *b = (const struct btf_field *)_b;
-
- if (a->offset < b->offset)
- return -1;
- else if (a->offset > b->offset)
- return 1;
- return 0;
-}
-
-struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, u32 value_size)
-{
- struct btf_field_info info_arr[BTF_FIELDS_MAX];
- u32 next_off = 0, field_type_size;
- struct btf_record *rec;
- int ret, i, cnt;
-
- ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
- if (ret < 0)
- return ERR_PTR(ret);
- if (!ret)
- return NULL;
-
- cnt = ret;
- /* This needs to be kzalloc to zero out padding and unused fields, see
- * comment in btf_record_equal.
- */
- rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
- if (!rec)
- return ERR_PTR(-ENOMEM);
-
- rec->spin_lock_off = -EINVAL;
- rec->res_spin_lock_off = -EINVAL;
- rec->timer_off = -EINVAL;
- rec->wq_off = -EINVAL;
- rec->refcount_off = -EINVAL;
- rec->task_work_off = -EINVAL;
- for (i = 0; i < cnt; i++) {
- field_type_size = btf_field_type_size(info_arr[i].type);
- if (info_arr[i].off + field_type_size > value_size) {
- WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size);
- ret = -EFAULT;
- goto end;
- }
- if (info_arr[i].off < next_off) {
- ret = -EEXIST;
- goto end;
- }
- next_off = info_arr[i].off + field_type_size;
-
- rec->field_mask |= info_arr[i].type;
- rec->fields[i].offset = info_arr[i].off;
- rec->fields[i].type = info_arr[i].type;
- rec->fields[i].size = field_type_size;
-
- switch (info_arr[i].type) {
- case BPF_SPIN_LOCK:
- WARN_ON_ONCE(rec->spin_lock_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->spin_lock_off = rec->fields[i].offset;
- break;
- case BPF_RES_SPIN_LOCK:
- WARN_ON_ONCE(rec->spin_lock_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->res_spin_lock_off = rec->fields[i].offset;
- break;
- case BPF_TIMER:
- WARN_ON_ONCE(rec->timer_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->timer_off = rec->fields[i].offset;
- break;
- case BPF_WORKQUEUE:
- WARN_ON_ONCE(rec->wq_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->wq_off = rec->fields[i].offset;
- break;
- case BPF_TASK_WORK:
- WARN_ON_ONCE(rec->task_work_off >= 0);
- rec->task_work_off = rec->fields[i].offset;
- break;
- case BPF_REFCOUNT:
- WARN_ON_ONCE(rec->refcount_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->refcount_off = rec->fields[i].offset;
- break;
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_LIST_HEAD:
- ret = btf_parse_list_head(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_RB_ROOT:
- ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_LIST_NODE:
- case BPF_RB_NODE:
- break;
- default:
- ret = -EFAULT;
- goto end;
- }
- rec->cnt++;
- }
-
- if (rec->spin_lock_off >= 0 && rec->res_spin_lock_off >= 0) {
- ret = -EINVAL;
- goto end;
- }
-
- /* bpf_{list_head, rb_node} require bpf_spin_lock */
- if ((btf_record_has_field(rec, BPF_LIST_HEAD) ||
- btf_record_has_field(rec, BPF_RB_ROOT)) &&
- (rec->spin_lock_off < 0 && rec->res_spin_lock_off < 0)) {
- ret = -EINVAL;
- goto end;
- }
-
- if (rec->refcount_off < 0 &&
- btf_record_has_field(rec, BPF_LIST_NODE) &&
- btf_record_has_field(rec, BPF_RB_NODE)) {
- ret = -EINVAL;
- goto end;
- }
-
- sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
- NULL, rec);
-
- return rec;
-end:
- btf_record_free(rec);
- return ERR_PTR(ret);
-}
-
-int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
-{
- int i;
-
- /* There are three types that signify ownership of some other type:
- * kptr_ref, bpf_list_head, bpf_rb_root.
- * kptr_ref only supports storing kernel types, which can't store
- * references to program allocated local types.
- *
- * Hence we only need to ensure that bpf_{list_head,rb_root} ownership
- * does not form cycles.
- */
- if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & (BPF_GRAPH_ROOT | BPF_UPTR)))
- return 0;
- for (i = 0; i < rec->cnt; i++) {
- struct btf_struct_meta *meta;
- const struct btf_type *t;
- u32 btf_id;
-
- if (rec->fields[i].type == BPF_UPTR) {
- /* The uptr only supports pinning one page and cannot
- * point to a kernel struct
- */
- if (btf_is_kernel(rec->fields[i].kptr.btf))
- return -EINVAL;
- t = btf_type_by_id(rec->fields[i].kptr.btf,
- rec->fields[i].kptr.btf_id);
- if (!t->size)
- return -EINVAL;
- if (t->size > PAGE_SIZE)
- return -E2BIG;
- continue;
- }
-
- if (!(rec->fields[i].type & BPF_GRAPH_ROOT))
- continue;
- btf_id = rec->fields[i].graph_root.value_btf_id;
- meta = btf_find_struct_meta(btf, btf_id);
- if (!meta)
- return -EFAULT;
- rec->fields[i].graph_root.value_rec = meta->record;
-
- /* We need to set value_rec for all root types, but no need
- * to check ownership cycle for a type unless it's also a
- * node type.
- */
- if (!(rec->field_mask & BPF_GRAPH_NODE))
- continue;
-
- /* We need to ensure ownership acyclicity among all types. The
- * proper way to do it would be to topologically sort all BTF
- * IDs based on the ownership edges, since there can be multiple
- * bpf_{list_head,rb_node} in a type. Instead, we use the
- * following resaoning:
- *
- * - A type can only be owned by another type in user BTF if it
- * has a bpf_{list,rb}_node. Let's call these node types.
- * - A type can only _own_ another type in user BTF if it has a
- * bpf_{list_head,rb_root}. Let's call these root types.
- *
- * We ensure that if a type is both a root and node, its
- * element types cannot be root types.
- *
- * To ensure acyclicity:
- *
- * When A is an root type but not a node, its ownership
- * chain can be:
- * A -> B -> C
- * Where:
- * - A is an root, e.g. has bpf_rb_root.
- * - B is both a root and node, e.g. has bpf_rb_node and
- * bpf_list_head.
- * - C is only an root, e.g. has bpf_list_node
- *
- * When A is both a root and node, some other type already
- * owns it in the BTF domain, hence it can not own
- * another root type through any of the ownership edges.
- * A -> B
- * Where:
- * - A is both an root and node.
- * - B is only an node.
- */
- if (meta->record->field_mask & BPF_GRAPH_ROOT)
- return -ELOOP;
- }
- return 0;
-}
-
-static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *member;
- void *safe_data;
- u32 i;
-
- safe_data = btf_show_start_struct_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- for_each_member(i, t, member) {
- const struct btf_type *member_type = btf_type_by_id(btf,
- member->type);
- const struct btf_kind_operations *ops;
- u32 member_offset, bitfield_size;
- u32 bytes_offset;
- u8 bits8_offset;
-
- btf_show_start_member(show, member);
-
- member_offset = __btf_member_bit_offset(t, member);
- bitfield_size = __btf_member_bitfield_size(t, member);
- bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
- bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
- if (bitfield_size) {
- safe_data = btf_show_start_type(show, member_type,
- member->type,
- data + bytes_offset);
- if (safe_data)
- btf_bitfield_show(safe_data,
- bits8_offset,
- bitfield_size, show);
- btf_show_end_type(show);
- } else {
- ops = btf_type_ops(member_type);
- ops->show(btf, member_type, member->type,
- data + bytes_offset, bits8_offset, show);
- }
-
- btf_show_end_member(show);
- }
-
- btf_show_end_struct_type(show);
-}
-
-static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *m = show->state.member;
-
- /*
- * First check if any members would be shown (are non-zero).
- * See comments above "struct btf_show" definition for more
- * details on how this works at a high-level.
- */
- if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
- if (!show->state.depth_check) {
- show->state.depth_check = show->state.depth + 1;
- show->state.depth_to_show = 0;
- }
- __btf_struct_show(btf, t, type_id, data, bits_offset, show);
- /* Restore saved member data here */
- show->state.member = m;
- if (show->state.depth_check != show->state.depth + 1)
- return;
- show->state.depth_check = 0;
-
- if (show->state.depth_to_show <= show->state.depth)
- return;
- /*
- * Reaching here indicates we have recursed and found
- * non-zero child values.
- */
- }
-
- __btf_struct_show(btf, t, type_id, data, bits_offset, show);
-}
-
-static const struct btf_kind_operations struct_ops = {
- .check_meta = btf_struct_check_meta,
- .resolve = btf_struct_resolve,
- .check_member = btf_struct_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_struct_log,
- .show = btf_struct_show,
-};
-
-static int btf_enum_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < member_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off, nr_bits, bytes_end, struct_size;
- u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
-
- struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
- nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
- if (!nr_bits) {
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- nr_bits = int_bitsize;
- } else if (nr_bits > int_bitsize) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
- if (struct_size < bytes_end) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_enum_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_enum *enums = btf_type_enum(t);
- struct btf *btf = env->btf;
- const char *fmt_str;
- u16 i, nr_enums;
- u32 meta_needed;
-
- nr_enums = btf_type_vlen(t);
- meta_needed = nr_enums * sizeof(*enums);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->size > 8 || !is_power_of_2(t->size)) {
- btf_verifier_log_type(env, t, "Unexpected size");
- return -EINVAL;
- }
-
- /* enum type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for (i = 0; i < nr_enums; i++) {
- if (!btf_name_offset_valid(btf, enums[i].name_off)) {
- btf_verifier_log(env, "\tInvalid name_offset:%u",
- enums[i].name_off);
- return -EINVAL;
- }
-
- /* enum member must have a valid name */
- if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (env->log.level == BPF_LOG_KERNEL)
- continue;
- fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
- btf_verifier_log(env, fmt_str,
- __btf_name_by_offset(btf, enums[i].name_off),
- enums[i].val);
- }
-
- return meta_needed;
-}
-
-static void btf_enum_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_enum *enums = btf_type_enum(t);
- u32 i, nr_enums = btf_type_vlen(t);
- void *safe_data;
- int v;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- v = *(int *)safe_data;
-
- for (i = 0; i < nr_enums; i++) {
- if (v != enums[i].val)
- continue;
-
- btf_show_type_value(show, "%s",
- __btf_name_by_offset(btf,
- enums[i].name_off));
-
- btf_show_end_type(show);
- return;
- }
-
- if (btf_type_kflag(t))
- btf_show_type_value(show, "%d", v);
- else
- btf_show_type_value(show, "%u", v);
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations enum_ops = {
- .check_meta = btf_enum_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_enum_check_member,
- .check_kflag_member = btf_enum_check_kflag_member,
- .log_details = btf_enum_log,
- .show = btf_enum_show,
-};
-
-static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_enum64 *enums = btf_type_enum64(t);
- struct btf *btf = env->btf;
- const char *fmt_str;
- u16 i, nr_enums;
- u32 meta_needed;
-
- nr_enums = btf_type_vlen(t);
- meta_needed = nr_enums * sizeof(*enums);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->size > 8 || !is_power_of_2(t->size)) {
- btf_verifier_log_type(env, t, "Unexpected size");
- return -EINVAL;
- }
-
- /* enum type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for (i = 0; i < nr_enums; i++) {
- if (!btf_name_offset_valid(btf, enums[i].name_off)) {
- btf_verifier_log(env, "\tInvalid name_offset:%u",
- enums[i].name_off);
- return -EINVAL;
- }
-
- /* enum member must have a valid name */
- if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (env->log.level == BPF_LOG_KERNEL)
- continue;
-
- fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
- btf_verifier_log(env, fmt_str,
- __btf_name_by_offset(btf, enums[i].name_off),
- btf_enum64_value(enums + i));
- }
-
- return meta_needed;
-}
-
-static void btf_enum64_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_enum64 *enums = btf_type_enum64(t);
- u32 i, nr_enums = btf_type_vlen(t);
- void *safe_data;
- s64 v;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- v = *(u64 *)safe_data;
-
- for (i = 0; i < nr_enums; i++) {
- if (v != btf_enum64_value(enums + i))
- continue;
-
- btf_show_type_value(show, "%s",
- __btf_name_by_offset(btf,
- enums[i].name_off));
-
- btf_show_end_type(show);
- return;
- }
-
- if (btf_type_kflag(t))
- btf_show_type_value(show, "%lld", v);
- else
- btf_show_type_value(show, "%llu", v);
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations enum64_ops = {
- .check_meta = btf_enum64_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_enum_check_member,
- .check_kflag_member = btf_enum_check_kflag_member,
- .log_details = btf_enum_log,
- .show = btf_enum64_show,
-};
-
-static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 meta_needed = btf_type_vlen(t) * sizeof(struct btf_param);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_func_proto_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_param *args = (const struct btf_param *)(t + 1);
- u16 nr_args = btf_type_vlen(t), i;
-
- btf_verifier_log(env, "return=%u args=(", t->type);
- if (!nr_args) {
- btf_verifier_log(env, "void");
- goto done;
- }
-
- if (nr_args == 1 && !args[0].type) {
- /* Only one vararg */
- btf_verifier_log(env, "vararg");
- goto done;
- }
-
- btf_verifier_log(env, "%u %s", args[0].type,
- __btf_name_by_offset(env->btf,
- args[0].name_off));
- for (i = 1; i < nr_args - 1; i++)
- btf_verifier_log(env, ", %u %s", args[i].type,
- __btf_name_by_offset(env->btf,
- args[i].name_off));
-
- if (nr_args > 1) {
- const struct btf_param *last_arg = &args[nr_args - 1];
-
- if (last_arg->type)
- btf_verifier_log(env, ", %u %s", last_arg->type,
- __btf_name_by_offset(env->btf,
- last_arg->name_off));
- else
- btf_verifier_log(env, ", vararg");
- }
-
-done:
- btf_verifier_log(env, ")");
-}
-
-static const struct btf_kind_operations func_proto_ops = {
- .check_meta = btf_func_proto_check_meta,
- .resolve = btf_df_resolve,
- /*
- * BTF_KIND_FUNC_PROTO cannot be directly referred by
- * a struct's member.
- *
- * It should be a function pointer instead.
- * (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
- *
- * Hence, there is no btf_func_check_member().
- */
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_func_proto_log,
- .show = btf_df_show,
-};
-
-static s32 btf_func_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t) > BTF_FUNC_GLOBAL) {
- btf_verifier_log_type(env, t, "Invalid func linkage");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_func_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- int err;
-
- err = btf_func_check(env, t);
- if (err)
- return err;
-
- env_stack_pop_resolved(env, next_type_id, 0);
- return 0;
-}
-
-static const struct btf_kind_operations func_ops = {
- .check_meta = btf_func_check_meta,
- .resolve = btf_func_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_df_show,
-};
-
-static s32 btf_var_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_var *var;
- u32 meta_needed = sizeof(*var);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- /* A var cannot be in type void */
- if (!t->type || !BTF_TYPE_ID_VALID(t->type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- var = btf_type_var(t);
- if (var->linkage != BTF_VAR_STATIC &&
- var->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
- btf_verifier_log_type(env, t, "Linkage not supported");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t)
-{
- const struct btf_var *var = btf_type_var(t);
-
- btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage);
-}
-
-static const struct btf_kind_operations var_ops = {
- .check_meta = btf_var_check_meta,
- .resolve = btf_var_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_var_log,
- .show = btf_var_show,
-};
-
-static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_var_secinfo *vsi;
- u64 last_vsi_end_off = 0, sum = 0;
- u32 i, meta_needed;
-
- meta_needed = btf_type_vlen(t) * sizeof(*vsi);
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (!t->size) {
- btf_verifier_log_type(env, t, "size == 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!t->name_off ||
- !btf_name_valid_section(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for_each_vsi(i, t, vsi) {
- /* A var cannot be in type void */
- if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid type_id");
- return -EINVAL;
- }
-
- if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid offset");
- return -EINVAL;
- }
-
- if (!vsi->size || vsi->size > t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid size");
- return -EINVAL;
- }
-
- last_vsi_end_off = vsi->offset + vsi->size;
- if (last_vsi_end_off > t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid offset+size");
- return -EINVAL;
- }
-
- btf_verifier_log_vsi(env, t, vsi, NULL);
- sum += vsi->size;
- }
-
- if (t->size < sum) {
- btf_verifier_log_type(env, t, "Invalid btf_info size");
- return -EINVAL;
- }
-
- return meta_needed;
-}
-
-static int btf_datasec_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_var_secinfo *vsi;
- struct btf *btf = env->btf;
- u16 i;
-
- env->resolve_mode = RESOLVE_TBD;
- for_each_vsi_from(i, v->next_member, v->t, vsi) {
- u32 var_type_id = vsi->type, type_id, type_size = 0;
- const struct btf_type *var_type = btf_type_by_id(env->btf,
- var_type_id);
- if (!var_type || !btf_type_is_var(var_type)) {
- btf_verifier_log_vsi(env, v->t, vsi,
- "Not a VAR kind member");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, var_type) &&
- !env_type_is_resolved(env, var_type_id)) {
- env_stack_set_next_member(env, i + 1);
- return env_stack_push(env, var_type, var_type_id);
- }
-
- type_id = var_type->type;
- if (!btf_type_id_size(btf, &type_id, &type_size)) {
- btf_verifier_log_vsi(env, v->t, vsi, "Invalid type");
- return -EINVAL;
- }
-
- if (vsi->size < type_size) {
- btf_verifier_log_vsi(env, v->t, vsi, "Invalid size");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, 0, 0);
- return 0;
-}
-
-static void btf_datasec_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-static void btf_datasec_show(const struct btf *btf,
- const struct btf_type *t, u32 type_id,
- void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_var_secinfo *vsi;
- const struct btf_type *var;
- u32 i;
-
- if (!btf_show_start_type(show, t, type_id, data))
- return;
-
- btf_show_type_value(show, "section (\"%s\") = {",
- __btf_name_by_offset(btf, t->name_off));
- for_each_vsi(i, t, vsi) {
- var = btf_type_by_id(btf, vsi->type);
- if (i)
- btf_show(show, ",");
- btf_type_ops(var)->show(btf, var, vsi->type,
- data + vsi->offset, bits_offset, show);
- }
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations datasec_ops = {
- .check_meta = btf_datasec_check_meta,
- .resolve = btf_datasec_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_datasec_log,
- .show = btf_datasec_show,
-};
-
-static s32 btf_float_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (t->size != 2 && t->size != 4 && t->size != 8 && t->size != 12 &&
- t->size != 16) {
- btf_verifier_log_type(env, t, "Invalid type_size");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_float_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u64 start_offset_bytes;
- u64 end_offset_bytes;
- u64 misalign_bits;
- u64 align_bytes;
- u64 align_bits;
-
- /* Different architectures have different alignment requirements, so
- * here we check only for the reasonable minimum. This way we ensure
- * that types after CO-RE can pass the kernel BTF verifier.
- */
- align_bytes = min_t(u64, sizeof(void *), member_type->size);
- align_bits = align_bytes * BITS_PER_BYTE;
- div64_u64_rem(member->offset, align_bits, &misalign_bits);
- if (misalign_bits) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not properly aligned");
- return -EINVAL;
- }
-
- start_offset_bytes = member->offset / BITS_PER_BYTE;
- end_offset_bytes = start_offset_bytes + member_type->size;
- if (end_offset_bytes > struct_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static void btf_float_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u", t->size);
-}
-
-static const struct btf_kind_operations float_ops = {
- .check_meta = btf_float_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_float_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_float_log,
- .show = btf_df_show,
-};
-
-static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_decl_tag *tag;
- u32 meta_needed = sizeof(*tag);
- s32 component_idx;
- const char *value;
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- value = btf_name_by_offset(env->btf, t->name_off);
- if (!value || !value[0]) {
- btf_verifier_log_type(env, t, "Invalid value");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- component_idx = btf_type_decl_tag(t)->component_idx;
- if (component_idx < -1) {
- btf_verifier_log_type(env, t, "Invalid component_idx");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static int btf_decl_tag_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
- s32 component_idx;
- u32 vlen;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || !btf_type_is_decl_tag_target(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- component_idx = btf_type_decl_tag(t)->component_idx;
- if (component_idx != -1) {
- if (btf_type_is_var(next_type) || btf_type_is_typedef(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid component_idx");
- return -EINVAL;
- }
-
- if (btf_type_is_struct(next_type)) {
- vlen = btf_type_vlen(next_type);
- } else {
- /* next_type should be a function */
- next_type = btf_type_by_id(btf, next_type->type);
- vlen = btf_type_vlen(next_type);
- }
-
- if ((u32)component_idx >= vlen) {
- btf_verifier_log_type(env, v->t, "Invalid component_idx");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static void btf_decl_tag_log(struct btf_verifier_env *env, const struct btf_type *t)
-{
- btf_verifier_log(env, "type=%u component_idx=%d", t->type,
- btf_type_decl_tag(t)->component_idx);
-}
-
-static const struct btf_kind_operations decl_tag_ops = {
- .check_meta = btf_decl_tag_check_meta,
- .resolve = btf_decl_tag_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_decl_tag_log,
- .show = btf_df_show,
-};
-
-static int btf_func_proto_check(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_type *ret_type;
- const struct btf_param *args;
- const struct btf *btf;
- u16 nr_args, i;
- int err;
-
- btf = env->btf;
- args = (const struct btf_param *)(t + 1);
- nr_args = btf_type_vlen(t);
-
- /* Check func return type which could be "void" (t->type == 0) */
- if (t->type) {
- u32 ret_type_id = t->type;
-
- ret_type = btf_type_by_id(btf, ret_type_id);
- if (!ret_type) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
-
- if (btf_type_is_resolve_source_only(ret_type)) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
-
- if (btf_type_needs_resolve(ret_type) &&
- !env_type_is_resolved(env, ret_type_id)) {
- err = btf_resolve(env, ret_type, ret_type_id);
- if (err)
- return err;
- }
-
- /* Ensure the return type is a type that has a size */
- if (!btf_type_id_size(btf, &ret_type_id, NULL)) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
- }
-
- if (!nr_args)
- return 0;
-
- /* Last func arg type_id could be 0 if it is a vararg */
- if (!args[nr_args - 1].type) {
- if (args[nr_args - 1].name_off) {
- btf_verifier_log_type(env, t, "Invalid arg#%u",
- nr_args);
- return -EINVAL;
- }
- nr_args--;
- }
-
- for (i = 0; i < nr_args; i++) {
- const struct btf_type *arg_type;
- u32 arg_type_id;
-
- arg_type_id = args[i].type;
- arg_type = btf_type_by_id(btf, arg_type_id);
- if (!arg_type) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (btf_type_is_resolve_source_only(arg_type)) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (args[i].name_off &&
- (!btf_name_offset_valid(btf, args[i].name_off) ||
- !btf_name_valid_identifier(btf, args[i].name_off))) {
- btf_verifier_log_type(env, t,
- "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (btf_type_needs_resolve(arg_type) &&
- !env_type_is_resolved(env, arg_type_id)) {
- err = btf_resolve(env, arg_type, arg_type_id);
- if (err)
- return err;
- }
-
- if (!btf_type_id_size(btf, &arg_type_id, NULL)) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
- }
-
- return 0;
-}
-
-static int btf_func_check(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_type *proto_type;
- const struct btf_param *args;
- const struct btf *btf;
- u16 nr_args, i;
-
- btf = env->btf;
- proto_type = btf_type_by_id(btf, t->type);
-
- if (!proto_type || !btf_type_is_func_proto(proto_type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- args = (const struct btf_param *)(proto_type + 1);
- nr_args = btf_type_vlen(proto_type);
- for (i = 0; i < nr_args; i++) {
- if (!args[i].name_off && args[i].type) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
- }
-
- return 0;
-}
-
-static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = {
- [BTF_KIND_INT] = &int_ops,
- [BTF_KIND_PTR] = &ptr_ops,
- [BTF_KIND_ARRAY] = &array_ops,
- [BTF_KIND_STRUCT] = &struct_ops,
- [BTF_KIND_UNION] = &struct_ops,
- [BTF_KIND_ENUM] = &enum_ops,
- [BTF_KIND_FWD] = &fwd_ops,
- [BTF_KIND_TYPEDEF] = &modifier_ops,
- [BTF_KIND_VOLATILE] = &modifier_ops,
- [BTF_KIND_CONST] = &modifier_ops,
- [BTF_KIND_RESTRICT] = &modifier_ops,
- [BTF_KIND_FUNC] = &func_ops,
- [BTF_KIND_FUNC_PROTO] = &func_proto_ops,
- [BTF_KIND_VAR] = &var_ops,
- [BTF_KIND_DATASEC] = &datasec_ops,
- [BTF_KIND_FLOAT] = &float_ops,
- [BTF_KIND_DECL_TAG] = &decl_tag_ops,
- [BTF_KIND_TYPE_TAG] = &modifier_ops,
- [BTF_KIND_ENUM64] = &enum64_ops,
-};
-
-static s32 btf_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 saved_meta_left = meta_left;
- s32 var_meta_size;
-
- if (meta_left < sizeof(*t)) {
- btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu",
- env->log_type_id, meta_left, sizeof(*t));
- return -EINVAL;
- }
- meta_left -= sizeof(*t);
-
- if (t->info & ~BTF_INFO_MASK) {
- btf_verifier_log(env, "[%u] Invalid btf_info:%x",
- env->log_type_id, t->info);
- return -EINVAL;
- }
-
- if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
- BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
- btf_verifier_log(env, "[%u] Invalid kind:%u",
- env->log_type_id, BTF_INFO_KIND(t->info));
- return -EINVAL;
- }
-
- if (!btf_name_offset_valid(env->btf, t->name_off)) {
- btf_verifier_log(env, "[%u] Invalid name_offset:%u",
- env->log_type_id, t->name_off);
- return -EINVAL;
- }
-
- var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left);
- if (var_meta_size < 0)
- return var_meta_size;
-
- meta_left -= var_meta_size;
-
- return saved_meta_left - meta_left;
-}
-
-static int btf_check_all_metas(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- struct btf_header *hdr;
- void *cur, *end;
-
- hdr = &btf->hdr;
- cur = btf->nohdr_data + hdr->type_off;
- end = cur + hdr->type_len;
-
- env->log_type_id = btf->base_btf ? btf->start_id : 1;
- while (cur < end) {
- struct btf_type *t = cur;
- s32 meta_size;
-
- meta_size = btf_check_meta(env, t, end - cur);
- if (meta_size < 0)
- return meta_size;
-
- btf_add_type(env, t);
- cur += meta_size;
- env->log_type_id++;
- }
-
- return 0;
-}
-
-static bool btf_resolve_valid(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 type_id)
-{
- struct btf *btf = env->btf;
-
- if (!env_type_is_resolved(env, type_id))
- return false;
-
- if (btf_type_is_struct(t) || btf_type_is_datasec(t))
- return !btf_resolved_type_id(btf, type_id) &&
- !btf_resolved_type_size(btf, type_id);
-
- if (btf_type_is_decl_tag(t) || btf_type_is_func(t))
- return btf_resolved_type_id(btf, type_id) &&
- !btf_resolved_type_size(btf, type_id);
-
- if (btf_type_is_modifier(t) || btf_type_is_ptr(t) ||
- btf_type_is_var(t)) {
- t = btf_type_id_resolve(btf, &type_id);
- return t &&
- !btf_type_is_modifier(t) &&
- !btf_type_is_var(t) &&
- !btf_type_is_datasec(t);
- }
-
- if (btf_type_is_array(t)) {
- const struct btf_array *array = btf_type_array(t);
- const struct btf_type *elem_type;
- u32 elem_type_id = array->type;
- u32 elem_size;
-
- elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
- return elem_type && !btf_type_is_modifier(elem_type) &&
- (array->nelems * elem_size ==
- btf_resolved_type_size(btf, type_id));
- }
-
- return false;
-}
-
-static int btf_resolve(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id)
-{
- u32 save_log_type_id = env->log_type_id;
- const struct resolve_vertex *v;
- int err = 0;
-
- env->resolve_mode = RESOLVE_TBD;
- env_stack_push(env, t, type_id);
- while (!err && (v = env_stack_peak(env))) {
- env->log_type_id = v->type_id;
- err = btf_type_ops(v->t)->resolve(env, v);
- }
-
- env->log_type_id = type_id;
- if (err == -E2BIG) {
- btf_verifier_log_type(env, t,
- "Exceeded max resolving depth:%u",
- MAX_RESOLVE_DEPTH);
- } else if (err == -EEXIST) {
- btf_verifier_log_type(env, t, "Loop detected");
- }
-
- /* Final sanity check */
- if (!err && !btf_resolve_valid(env, t, type_id)) {
- btf_verifier_log_type(env, t, "Invalid resolve state");
- err = -EINVAL;
- }
-
- env->log_type_id = save_log_type_id;
- return err;
-}
-
-static int btf_check_all_types(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- const struct btf_type *t;
- u32 type_id, i;
- int err;
-
- err = env_resolve_init(env);
- if (err)
- return err;
-
- env->phase++;
- for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
- type_id = btf->start_id + i;
- t = btf_type_by_id(btf, type_id);
-
- env->log_type_id = type_id;
- if (btf_type_needs_resolve(t) &&
- !env_type_is_resolved(env, type_id)) {
- err = btf_resolve(env, t, type_id);
- if (err)
- return err;
- }
-
- if (btf_type_is_func_proto(t)) {
- err = btf_func_proto_check(env, t);
- if (err)
- return err;
- }
- }
-
- return 0;
-}
-
-static int btf_parse_type_sec(struct btf_verifier_env *env)
-{
- const struct btf_header *hdr = &env->btf->hdr;
- int err;
-
- /* Type section must align to 4 bytes */
- if (hdr->type_off & (sizeof(u32) - 1)) {
- btf_verifier_log(env, "Unaligned type_off");
- return -EINVAL;
- }
-
- if (!env->btf->base_btf && !hdr->type_len) {
- btf_verifier_log(env, "No type found");
- return -EINVAL;
- }
-
- err = btf_check_all_metas(env);
- if (err)
- return err;
-
- return btf_check_all_types(env);
-}
-
-static int btf_parse_str_sec(struct btf_verifier_env *env)
-{
- const struct btf_header *hdr;
- struct btf *btf = env->btf;
- const char *start, *end;
-
- hdr = &btf->hdr;
- start = btf->nohdr_data + hdr->str_off;
- end = start + hdr->str_len;
-
- if (end != btf->data + btf->data_size) {
- btf_verifier_log(env, "String section is not at the end");
- return -EINVAL;
- }
-
- btf->strings = start;
-
- if (btf->base_btf && !hdr->str_len)
- return 0;
- if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_NAME_OFFSET || end[-1]) {
- btf_verifier_log(env, "Invalid string section");
- return -EINVAL;
- }
- if (!btf->base_btf && start[0]) {
- btf_verifier_log(env, "Invalid string section");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static const size_t btf_sec_info_offset[] = {
- offsetof(struct btf_header, type_off),
- offsetof(struct btf_header, str_off),
-};
-
-static int btf_sec_info_cmp(const void *a, const void *b)
-{
- const struct btf_sec_info *x = a;
- const struct btf_sec_info *y = b;
-
- return (int)(x->off - y->off) ? : (int)(x->len - y->len);
-}
+ if (rec->spin_lock_off >= 0 && rec->res_spin_lock_off >= 0) {
+ ret = -EINVAL;
+ goto end;
+ }
-static int btf_check_sec_info(struct btf_verifier_env *env,
- u32 btf_data_size)
-{
- struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
- u32 total, expected_total, i;
- const struct btf_header *hdr;
- const struct btf *btf;
-
- btf = env->btf;
- hdr = &btf->hdr;
-
- /* Populate the secs from hdr */
- for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++)
- secs[i] = *(struct btf_sec_info *)((void *)hdr +
- btf_sec_info_offset[i]);
-
- sort(secs, ARRAY_SIZE(btf_sec_info_offset),
- sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
-
- /* Check for gaps and overlap among sections */
- total = 0;
- expected_total = btf_data_size - hdr->hdr_len;
- for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++) {
- if (expected_total < secs[i].off) {
- btf_verifier_log(env, "Invalid section offset");
- return -EINVAL;
- }
- if (total < secs[i].off) {
- /* gap */
- btf_verifier_log(env, "Unsupported section found");
- return -EINVAL;
- }
- if (total > secs[i].off) {
- btf_verifier_log(env, "Section overlap found");
- return -EINVAL;
- }
- if (expected_total - total < secs[i].len) {
- btf_verifier_log(env,
- "Total section length too long");
- return -EINVAL;
- }
- total += secs[i].len;
+ /* bpf_{list_head, rb_node} require bpf_spin_lock */
+ if ((btf_record_has_field(rec, BPF_LIST_HEAD) ||
+ btf_record_has_field(rec, BPF_RB_ROOT)) &&
+ (rec->spin_lock_off < 0 && rec->res_spin_lock_off < 0)) {
+ ret = -EINVAL;
+ goto end;
}
- /* There is data other than hdr and known sections */
- if (expected_total != total) {
- btf_verifier_log(env, "Unsupported section found");
- return -EINVAL;
+ if (rec->refcount_off < 0 &&
+ btf_record_has_field(rec, BPF_LIST_NODE) &&
+ btf_record_has_field(rec, BPF_RB_NODE)) {
+ ret = -EINVAL;
+ goto end;
}
- return 0;
+ sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
+ NULL, rec);
+
+ return rec;
+end:
+ btf_record_free(rec);
+ return ERR_PTR(ret);
}
-static int btf_parse_hdr(struct btf_verifier_env *env)
+int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
{
- u32 hdr_len, hdr_copy, btf_data_size;
- const struct btf_header *hdr;
- struct btf *btf;
-
- btf = env->btf;
- btf_data_size = btf->data_size;
-
- if (btf_data_size < offsetofend(struct btf_header, hdr_len)) {
- btf_verifier_log(env, "hdr_len not found");
- return -EINVAL;
- }
-
- hdr = btf->data;
- hdr_len = hdr->hdr_len;
- if (btf_data_size < hdr_len) {
- btf_verifier_log(env, "btf_header not found");
- return -EINVAL;
- }
+ int i;
- /* Ensure the unsupported header fields are zero */
- if (hdr_len > sizeof(btf->hdr)) {
- u8 *expected_zero = btf->data + sizeof(btf->hdr);
- u8 *end = btf->data + hdr_len;
+ /* There are three types that signify ownership of some other type:
+ * kptr_ref, bpf_list_head, bpf_rb_root.
+ * kptr_ref only supports storing kernel types, which can't store
+ * references to program allocated local types.
+ *
+ * Hence we only need to ensure that bpf_{list_head,rb_root} ownership
+ * does not form cycles.
+ */
+ if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & (BPF_GRAPH_ROOT | BPF_UPTR)))
+ return 0;
+ for (i = 0; i < rec->cnt; i++) {
+ struct btf_struct_meta *meta;
+ const struct btf_type *t;
+ u32 btf_id;
- for (; expected_zero < end; expected_zero++) {
- if (*expected_zero) {
- btf_verifier_log(env, "Unsupported btf_header");
+ if (rec->fields[i].type == BPF_UPTR) {
+ /* The uptr only supports pinning one page and cannot
+ * point to a kernel struct
+ */
+ if (btf_is_kernel(rec->fields[i].kptr.btf))
+ return -EINVAL;
+ t = btf_type_by_id(rec->fields[i].kptr.btf,
+ rec->fields[i].kptr.btf_id);
+ if (!t->size)
+ return -EINVAL;
+ if (t->size > PAGE_SIZE)
return -E2BIG;
- }
+ continue;
}
- }
-
- hdr_copy = min_t(u32, hdr_len, sizeof(btf->hdr));
- memcpy(&btf->hdr, btf->data, hdr_copy);
- hdr = &btf->hdr;
-
- btf_verifier_log_hdr(env, btf_data_size);
-
- if (hdr->magic != BTF_MAGIC) {
- btf_verifier_log(env, "Invalid magic");
- return -EINVAL;
- }
-
- if (hdr->version != BTF_VERSION) {
- btf_verifier_log(env, "Unsupported version");
- return -ENOTSUPP;
- }
+ if (!(rec->fields[i].type & BPF_GRAPH_ROOT))
+ continue;
+ btf_id = rec->fields[i].graph_root.value_btf_id;
+ meta = btf_find_struct_meta(btf, btf_id);
+ if (!meta)
+ return -EFAULT;
+ rec->fields[i].graph_root.value_rec = meta->record;
- if (hdr->flags) {
- btf_verifier_log(env, "Unsupported flags");
- return -ENOTSUPP;
- }
+ /* We need to set value_rec for all root types, but no need
+ * to check ownership cycle for a type unless it's also a
+ * node type.
+ */
+ if (!(rec->field_mask & BPF_GRAPH_NODE))
+ continue;
- if (!btf->base_btf && btf_data_size == hdr->hdr_len) {
- btf_verifier_log(env, "No data");
- return -EINVAL;
+ /* We need to ensure ownership acyclicity among all types. The
+ * proper way to do it would be to topologically sort all BTF
+ * IDs based on the ownership edges, since there can be multiple
+ * bpf_{list_head,rb_node} in a type. Instead, we use the
+ * following resaoning:
+ *
+ * - A type can only be owned by another type in user BTF if it
+ * has a bpf_{list,rb}_node. Let's call these node types.
+ * - A type can only _own_ another type in user BTF if it has a
+ * bpf_{list_head,rb_root}. Let's call these root types.
+ *
+ * We ensure that if a type is both a root and node, its
+ * element types cannot be root types.
+ *
+ * To ensure acyclicity:
+ *
+ * When A is an root type but not a node, its ownership
+ * chain can be:
+ * A -> B -> C
+ * Where:
+ * - A is an root, e.g. has bpf_rb_root.
+ * - B is both a root and node, e.g. has bpf_rb_node and
+ * bpf_list_head.
+ * - C is only an root, e.g. has bpf_list_node
+ *
+ * When A is both a root and node, some other type already
+ * owns it in the BTF domain, hence it can not own
+ * another root type through any of the ownership edges.
+ * A -> B
+ * Where:
+ * - A is both an root and node.
+ * - B is only an node.
+ */
+ if (meta->record->field_mask & BPF_GRAPH_ROOT)
+ return -ELOOP;
}
-
- return btf_check_sec_info(env, btf_data_size);
+ return 0;
}
-
static const char *alloc_obj_fields[] = {
"bpf_spin_lock",
"bpf_list_head",
@@ -5818,54 +1247,6 @@ struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id)
return NULL;
return bsearch(&btf_id, tab->types, tab->cnt, sizeof(tab->types[0]), btf_id_cmp_func);
}
-
-static int btf_check_type_tags(struct btf_verifier_env *env,
- struct btf *btf, int start_id)
-{
- int i, n, good_id = start_id - 1;
- bool in_tags;
-
- n = btf_nr_types(btf);
- for (i = start_id; i < n; i++) {
- const struct btf_type *t;
- int chain_limit = 32;
- u32 cur_id = i;
-
- t = btf_type_by_id(btf, i);
- if (!t)
- return -EINVAL;
- if (!btf_type_is_modifier(t))
- continue;
-
- cond_resched();
-
- in_tags = btf_type_is_type_tag(t);
- while (btf_type_is_modifier(t)) {
- if (!chain_limit--) {
- btf_verifier_log(env, "Max chain length or cycle detected");
- return -ELOOP;
- }
- if (btf_type_is_type_tag(t)) {
- if (!in_tags) {
- btf_verifier_log(env, "Type tags don't precede modifiers");
- return -EINVAL;
- }
- } else if (in_tags) {
- in_tags = false;
- }
- if (cur_id <= good_id)
- break;
- /* Move to next type */
- cur_id = t->type;
- t = btf_type_by_id(btf, cur_id);
- if (!t)
- return -EINVAL;
- }
- good_id = i;
- }
- return 0;
-}
-
static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)
{
u32 log_true_size;
@@ -6313,60 +1694,6 @@ int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_ty
}
BTF_ID_LIST_SINGLE(bpf_ctx_convert_btf_id, struct, bpf_ctx_convert)
-
-static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
- void *data, unsigned int data_size)
-{
- struct btf *btf = NULL;
- int err;
-
- if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
- return ERR_PTR(-ENOENT);
-
- btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
- if (!btf) {
- err = -ENOMEM;
- goto errout;
- }
- env->btf = btf;
-
- btf->data = data;
- btf->data_size = data_size;
- btf->kernel_btf = true;
- btf->named_start_id = 0;
- strscpy(btf->name, name);
-
- err = btf_parse_hdr(env);
- if (err)
- goto errout;
-
- btf->nohdr_data = btf->data + btf->hdr.hdr_len;
-
- err = btf_parse_str_sec(env);
- if (err)
- goto errout;
-
- err = btf_check_all_metas(env);
- if (err)
- goto errout;
-
- err = btf_check_type_tags(env, btf, 1);
- if (err)
- goto errout;
-
- btf_check_sorted(btf);
- refcount_set(&btf->refcnt, 1);
-
- return btf;
-
-errout:
- if (btf) {
- kvfree(btf->types);
- kfree(btf);
- }
- return ERR_PTR(err);
-}
-
struct btf *btf_parse_vmlinux(void)
{
struct btf_verifier_env *env = NULL;
@@ -8029,96 +3356,6 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return 0;
}
-
-static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
- struct btf_show *show)
-{
- const struct btf_type *t = btf_type_by_id(btf, type_id);
-
- show->btf = btf;
- memset(&show->state, 0, sizeof(show->state));
- memset(&show->obj, 0, sizeof(show->obj));
-
- btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
-}
-
-__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,
- va_list args)
-{
- seq_vprintf((struct seq_file *)show->target, fmt, args);
-}
-
-int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
- void *obj, struct seq_file *m, u64 flags)
-{
- struct btf_show sseq;
-
- sseq.target = m;
- sseq.showfn = btf_seq_show;
- sseq.flags = flags;
-
- btf_type_show(btf, type_id, obj, &sseq);
-
- return sseq.state.status;
-}
-
-void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
- struct seq_file *m)
-{
- (void) btf_type_seq_show_flags(btf, type_id, obj, m,
- BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
- BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
-}
-
-struct btf_show_snprintf {
- struct btf_show show;
- int len_left; /* space left in string */
- int len; /* length we would have written */
-};
-
-__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
- va_list args)
-{
- struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
- int len;
-
- len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
-
- if (len < 0) {
- ssnprintf->len_left = 0;
- ssnprintf->len = len;
- } else if (len >= ssnprintf->len_left) {
- /* no space, drive on to get length we would have written */
- ssnprintf->len_left = 0;
- ssnprintf->len += len;
- } else {
- ssnprintf->len_left -= len;
- ssnprintf->len += len;
- show->target += len;
- }
-}
-
-int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
- char *buf, int len, u64 flags)
-{
- struct btf_show_snprintf ssnprintf;
-
- ssnprintf.show.target = buf;
- ssnprintf.show.flags = flags;
- ssnprintf.show.showfn = btf_snprintf_show;
- ssnprintf.len_left = len;
- ssnprintf.len = 0;
-
- btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
-
- /* If we encountered an error, return it. */
- if (ssnprintf.show.state.status)
- return ssnprintf.show.state.status;
-
- /* Otherwise return length we would have written */
- return ssnprintf.len;
-}
-
#ifdef CONFIG_PROC_FS
static void bpf_btf_show_fdinfo(struct seq_file *m, struct file *filp)
{
@@ -8271,17 +3508,6 @@ u32 btf_obj_id(const struct btf *btf)
{
return READ_ONCE(btf->id);
}
-
-bool btf_is_kernel(const struct btf *btf)
-{
- return btf->kernel_btf;
-}
-
-bool btf_is_module(const struct btf *btf)
-{
- return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
-}
-
enum {
BTF_MODULE_F_LIVE = (1 << 0),
};
@@ -9740,21 +4966,3 @@ int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
}
EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
#endif
-
-bool btf_param_match_suffix(const struct btf *btf,
- const struct btf_param *arg,
- const char *suffix)
-{
- int suffix_len = strlen(suffix), len;
- const char *param_name;
-
- /* In the future, this can be ported to use BTF tagging */
- param_name = btf_name_by_offset(btf, arg->name_off);
- if (str_is_empty(param_name))
- return false;
- len = strlen(param_name);
- if (len <= suffix_len)
- return false;
- param_name += len - suffix_len;
- return !strncmp(param_name, suffix, suffix_len);
-}
diff --git a/kernel/btf/btf.c b/kernel/btf/btf.c
new file mode 100644
index 0000000000000..0eb383926b7d3
--- /dev/null
+++ b/kernel/btf/btf.c
@@ -0,0 +1,4803 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2018 Facebook */
+
+#include <uapi/linux/btf.h>
+#include <uapi/linux/bpf.h>
+#include <uapi/linux/types.h>
+#include <linux/seq_file.h>
+#include <linux/compiler.h>
+#include <linux/ctype.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/file.h>
+#include <linux/kernel.h>
+#include <linux/sort.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/bsearch.h>
+#include <linux/kobject.h>
+#include <linux/string.h>
+#include <linux/overflow.h>
+#include "btf.h"
+
+#ifndef CONFIG_BPF_SYSCALL
+/*
+ * When CONFIG_BPF_SYSCALL is not set, log.c is not compiled, so
+ * bpf_verifier_vlog() is unavailable. Provide a simple implementation
+ * that handles the BPF_LOG_KERNEL case (pr_err), which is the only
+ * log level used by the core BTF verifier for kernel BTF parsing.
+ */
+void bpf_verifier_vlog(struct bpf_verifier_log *log, const char *fmt,
+ va_list args)
+{
+ char buf[256];
+
+ vscnprintf(buf, sizeof(buf), fmt, args);
+ if (log->level == BPF_LOG_KERNEL)
+ pr_err("BPF: %s", buf);
+}
+
+__printf(2, 3) void bpf_log(struct bpf_verifier_log *log,
+ const char *fmt, ...)
+{
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+#endif /* !CONFIG_BPF_SYSCALL */
+
+/* BTF (BPF Type Format) is the meta data format which describes
+ * the data types of BPF program/map. Hence, it basically focus
+ * on the C programming language which the modern BPF is primary
+ * using.
+ *
+ * ELF Section:
+ * ~~~~~~~~~~~
+ * The BTF data is stored under the ".BTF" ELF section
+ *
+ * struct btf_type:
+ * ~~~~~~~~~~~~~~~
+ * Each 'struct btf_type' object describes a C data type.
+ * Depending on the type it is describing, a 'struct btf_type'
+ * object may be followed by more data. F.e.
+ * To describe an array, 'struct btf_type' is followed by
+ * 'struct btf_array'.
+ *
+ * 'struct btf_type' and any extra data following it are
+ * 4 bytes aligned.
+ *
+ * Type section:
+ * ~~~~~~~~~~~~~
+ * The BTF type section contains a list of 'struct btf_type' objects.
+ * Each one describes a C type. Recall from the above section
+ * that a 'struct btf_type' object could be immediately followed by extra
+ * data in order to describe some particular C types.
+ *
+ * type_id:
+ * ~~~~~~~
+ * Each btf_type object is identified by a type_id. The type_id
+ * is implicitly implied by the location of the btf_type object in
+ * the BTF type section. The first one has type_id 1. The second
+ * one has type_id 2...etc. Hence, an earlier btf_type has
+ * a smaller type_id.
+ *
+ * A btf_type object may refer to another btf_type object by using
+ * type_id (i.e. the "type" in the "struct btf_type").
+ *
+ * NOTE that we cannot assume any reference-order.
+ * A btf_type object can refer to an earlier btf_type object
+ * but it can also refer to a later btf_type object.
+ *
+ * For example, to describe "const void *". A btf_type
+ * object describing "const" may refer to another btf_type
+ * object describing "void *". This type-reference is done
+ * by specifying type_id:
+ *
+ * [1] CONST (anon) type_id=2
+ * [2] PTR (anon) type_id=0
+ *
+ * The above is the btf_verifier debug log:
+ * - Each line started with "[?]" is a btf_type object
+ * - [?] is the type_id of the btf_type object.
+ * - CONST/PTR is the BTF_KIND_XXX
+ * - "(anon)" is the name of the type. It just
+ * happens that CONST and PTR has no name.
+ * - type_id=XXX is the 'u32 type' in btf_type
+ *
+ * NOTE: "void" has type_id 0
+ *
+ * String section:
+ * ~~~~~~~~~~~~~~
+ * The BTF string section contains the names used by the type section.
+ * Each string is referred by an "offset" from the beginning of the
+ * string section.
+ *
+ * Each string is '\0' terminated.
+ *
+ * The first character in the string section must be '\0'
+ * which is used to mean 'anonymous'. Some btf_type may not
+ * have a name.
+ */
+
+/* BTF verification:
+ *
+ * To verify BTF data, two passes are needed.
+ *
+ * Pass #1
+ * ~~~~~~~
+ * The first pass is to collect all btf_type objects to
+ * an array: "btf->types".
+ *
+ * Depending on the C type that a btf_type is describing,
+ * a btf_type may be followed by extra data. We don't know
+ * how many btf_type is there, and more importantly we don't
+ * know where each btf_type is located in the type section.
+ *
+ * Without knowing the location of each type_id, most verifications
+ * cannot be done. e.g. an earlier btf_type may refer to a later
+ * btf_type (recall the "const void *" above), so we cannot
+ * check this type-reference in the first pass.
+ *
+ * In the first pass, it still does some verifications (e.g.
+ * checking the name is a valid offset to the string section).
+ *
+ * Pass #2
+ * ~~~~~~~
+ * The main focus is to resolve a btf_type that is referring
+ * to another type.
+ *
+ * We have to ensure the referring type:
+ * 1) does exist in the BTF (i.e. in btf->types[])
+ * 2) does not cause a loop:
+ * struct A {
+ * struct B b;
+ * };
+ *
+ * struct B {
+ * struct A a;
+ * };
+ *
+ * btf_type_needs_resolve() decides if a btf_type needs
+ * to be resolved.
+ *
+ * The needs_resolve type implements the "resolve()" ops which
+ * essentially does a DFS and detects backedge.
+ *
+ * During resolve (or DFS), different C types have different
+ * "RESOLVED" conditions.
+ *
+ * When resolving a BTF_KIND_STRUCT, we need to resolve all its
+ * members because a member is always referring to another
+ * type. A struct's member can be treated as "RESOLVED" if
+ * it is referring to a BTF_KIND_PTR. Otherwise, the
+ * following valid C struct would be rejected:
+ *
+ * struct A {
+ * int m;
+ * struct A *a;
+ * };
+ *
+ * When resolving a BTF_KIND_PTR, it needs to keep resolving if
+ * it is referring to another BTF_KIND_PTR. Otherwise, we cannot
+ * detect a pointer loop, e.g.:
+ * BTF_KIND_CONST -> BTF_KIND_PTR -> BTF_KIND_CONST -> BTF_KIND_PTR +
+ * ^ |
+ * +-----------------------------------------+
+ *
+ */
+
+#define BITS_PER_U128 (sizeof(u64) * BITS_PER_BYTE * 2)
+
+#define BTF_INFO_MASK 0x9f00ffff
+#define BTF_INT_MASK 0x0fffffff
+#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
+#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
+
+
+#define for_each_member_from(i, from, struct_type, member) \
+ for (i = from, member = btf_type_member(struct_type) + from; \
+ i < btf_type_vlen(struct_type); \
+ i++, member++)
+
+#define for_each_vsi_from(i, from, struct_type, member) \
+ for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
+ i < btf_type_vlen(struct_type); \
+ i++, member++)
+
+
+
+
+enum visit_state {
+ NOT_VISITED,
+ VISITED,
+ RESOLVED,
+};
+
+struct btf_sec_info {
+ u32 off;
+ u32 len;
+};
+
+const char * const btf_kind_str[NR_BTF_KINDS] = {
+ [BTF_KIND_UNKN] = "UNKNOWN",
+ [BTF_KIND_INT] = "INT",
+ [BTF_KIND_PTR] = "PTR",
+ [BTF_KIND_ARRAY] = "ARRAY",
+ [BTF_KIND_STRUCT] = "STRUCT",
+ [BTF_KIND_UNION] = "UNION",
+ [BTF_KIND_ENUM] = "ENUM",
+ [BTF_KIND_FWD] = "FWD",
+ [BTF_KIND_TYPEDEF] = "TYPEDEF",
+ [BTF_KIND_VOLATILE] = "VOLATILE",
+ [BTF_KIND_CONST] = "CONST",
+ [BTF_KIND_RESTRICT] = "RESTRICT",
+ [BTF_KIND_FUNC] = "FUNC",
+ [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO",
+ [BTF_KIND_VAR] = "VAR",
+ [BTF_KIND_DATASEC] = "DATASEC",
+ [BTF_KIND_FLOAT] = "FLOAT",
+ [BTF_KIND_DECL_TAG] = "DECL_TAG",
+ [BTF_KIND_TYPE_TAG] = "TYPE_TAG",
+ [BTF_KIND_ENUM64] = "ENUM64",
+};
+
+const char *btf_type_str(const struct btf_type *t)
+{
+ return btf_kind_str[BTF_INFO_KIND(t->info)];
+}
+
+/* Chunk size we use in safe copy of data to be shown. */
+#define BTF_SHOW_OBJ_SAFE_SIZE 32
+
+/*
+ * This is the maximum size of a base type value (equivalent to a
+ * 128-bit int); if we are at the end of our safe buffer and have
+ * less than 16 bytes space we can't be assured of being able
+ * to copy the next type safely, so in such cases we will initiate
+ * a new copy.
+ */
+#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
+
+/* Type name size */
+#define BTF_SHOW_NAME_SIZE 80
+
+
+/*
+ * Common data to all BTF show operations. Private show functions can add
+ * their own data to a structure containing a struct btf_show and consult it
+ * in the show callback. See btf_type_show() below.
+ *
+ * One challenge with showing nested data is we want to skip 0-valued
+ * data, but in order to figure out whether a nested object is all zeros
+ * we need to walk through it. As a result, we need to make two passes
+ * when handling structs, unions and arrays; the first path simply looks
+ * for nonzero data, while the second actually does the display. The first
+ * pass is signalled by show->state.depth_check being set, and if we
+ * encounter a non-zero value we set show->state.depth_to_show to
+ * the depth at which we encountered it. When we have completed the
+ * first pass, we will know if anything needs to be displayed if
+ * depth_to_show > depth. See btf_[struct,array]_show() for the
+ * implementation of this.
+ *
+ * Another problem is we want to ensure the data for display is safe to
+ * access. To support this, the anonymous "struct {} obj" tracks the data
+ * object and our safe copy of it. We copy portions of the data needed
+ * to the object "copy" buffer, but because its size is limited to
+ * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
+ * traverse larger objects for display.
+ *
+ * The various data type show functions all start with a call to
+ * btf_show_start_type() which returns a pointer to the safe copy
+ * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
+ * raw data itself). btf_show_obj_safe() is responsible for
+ * using copy_from_kernel_nofault() to update the safe data if necessary
+ * as we traverse the object's data. skbuff-like semantics are
+ * used:
+ *
+ * - obj.head points to the start of the toplevel object for display
+ * - obj.size is the size of the toplevel object
+ * - obj.data points to the current point in the original data at
+ * which our safe data starts. obj.data will advance as we copy
+ * portions of the data.
+ *
+ * In most cases a single copy will suffice, but larger data structures
+ * such as "struct task_struct" will require many copies. The logic in
+ * btf_show_obj_safe() handles the logic that determines if a new
+ * copy_from_kernel_nofault() is needed.
+ */
+struct btf_show {
+ u64 flags;
+ void *target; /* target of show operation (seq file, buffer) */
+ __printf(2, 0) void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
+ const struct btf *btf;
+ /* below are used during iteration */
+ struct {
+ u8 depth;
+ u8 depth_to_show;
+ u8 depth_check;
+ u8 array_member:1,
+ array_terminated:1;
+ u16 array_encoding;
+ u32 type_id;
+ int status; /* non-zero for error */
+ const struct btf_type *type;
+ const struct btf_member *member;
+ char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
+ } state;
+ struct {
+ u32 size;
+ void *head;
+ void *data;
+ u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
+ } obj;
+};
+
+struct btf_kind_operations {
+ s32 (*check_meta)(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left);
+ int (*resolve)(struct btf_verifier_env *env,
+ const struct resolve_vertex *v);
+ int (*check_member)(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type);
+ int (*check_kflag_member)(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type);
+ void (*log_details)(struct btf_verifier_env *env,
+ const struct btf_type *t);
+ void (*show)(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offsets,
+ struct btf_show *show);
+};
+
+static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
+static struct btf_type btf_void;
+
+static int btf_resolve(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id);
+
+static int btf_func_check(struct btf_verifier_env *env,
+ const struct btf_type *t);
+
+bool btf_type_is_modifier(const struct btf_type *t)
+{
+ /* Some of them is not strictly a C modifier
+ * but they are grouped into the same bucket
+ * for BTF concern:
+ * A type (t) that refers to another
+ * type through t->type AND its size cannot
+ * be determined without following the t->type.
+ *
+ * ptr does not fall into this bucket
+ * because its size is always sizeof(void *).
+ */
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_TYPEDEF:
+ case BTF_KIND_VOLATILE:
+ case BTF_KIND_CONST:
+ case BTF_KIND_RESTRICT:
+ case BTF_KIND_TYPE_TAG:
+ return true;
+ }
+
+ return false;
+}
+
+static int btf_start_id(const struct btf *btf)
+{
+ return btf->start_id + (btf->base_btf ? 0 : 1);
+}
+
+bool btf_type_is_void(const struct btf_type *t)
+{
+ return t == &btf_void;
+}
+
+bool btf_type_is_datasec(const struct btf_type *t)
+{
+ return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
+}
+
+bool btf_type_is_decl_tag(const struct btf_type *t)
+{
+ return BTF_INFO_KIND(t->info) == BTF_KIND_DECL_TAG;
+}
+
+static bool btf_type_nosize(const struct btf_type *t)
+{
+ return btf_type_is_void(t) || btf_type_is_fwd(t) ||
+ btf_type_is_func(t) || btf_type_is_func_proto(t) ||
+ btf_type_is_decl_tag(t);
+}
+
+static bool btf_type_nosize_or_null(const struct btf_type *t)
+{
+ return !t || btf_type_nosize(t);
+}
+
+static bool btf_type_is_decl_tag_target(const struct btf_type *t)
+{
+ return btf_type_is_func(t) || btf_type_is_struct(t) ||
+ btf_type_is_var(t) || btf_type_is_typedef(t);
+}
+
+bool btf_is_vmlinux(const struct btf *btf)
+{
+ return btf->kernel_btf && !btf->base_btf;
+}
+
+u32 btf_nr_types(const struct btf *btf)
+{
+ u32 total = 0;
+
+ while (btf) {
+ total += btf->nr_types;
+ btf = btf->base_btf;
+ }
+
+ return total;
+}
+
+/*
+ * Note that vmlinux and kernel module BTFs are always sorted
+ * during the building phase.
+ */
+void btf_check_sorted(struct btf *btf)
+{
+ u32 i, n, named_start_id = 0;
+
+ n = btf_nr_types(btf);
+ if (btf_is_vmlinux(btf)) {
+ for (i = btf_start_id(btf); i < n; i++) {
+ const struct btf_type *t = btf_type_by_id(btf, i);
+ const char *n = btf_name_by_offset(btf, t->name_off);
+
+ if (n[0] != '\0') {
+ btf->named_start_id = i;
+ return;
+ }
+ }
+ return;
+ }
+
+ for (i = btf_start_id(btf) + 1; i < n; i++) {
+ const struct btf_type *ta = btf_type_by_id(btf, i - 1);
+ const struct btf_type *tb = btf_type_by_id(btf, i);
+ const char *na = btf_name_by_offset(btf, ta->name_off);
+ const char *nb = btf_name_by_offset(btf, tb->name_off);
+
+ if (strcmp(na, nb) > 0)
+ return;
+
+ if (named_start_id == 0 && na[0] != '\0')
+ named_start_id = i - 1;
+ if (named_start_id == 0 && nb[0] != '\0')
+ named_start_id = i;
+ }
+
+ if (named_start_id)
+ btf->named_start_id = named_start_id;
+}
+
+/*
+ * btf_named_start_id - Get the named starting ID for the BTF
+ * @btf: Pointer to the target BTF object
+ * @own: Flag indicating whether to query only the current BTF (true = current BTF only,
+ * false = recursively traverse the base BTF chain)
+ *
+ * Return value rules:
+ * 1. For a sorted btf, return its named_start_id
+ * 2. Else for a split BTF, return its start_id
+ * 3. Else for a base BTF, return 1
+ */
+u32 btf_named_start_id(const struct btf *btf, bool own)
+{
+ const struct btf *base_btf = btf;
+
+ while (!own && base_btf->base_btf)
+ base_btf = base_btf->base_btf;
+
+ return base_btf->named_start_id ?: (base_btf->start_id ?: 1);
+}
+
+static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)
+{
+ const struct btf_type *t;
+ const char *tname;
+ s32 l, r, m;
+
+ l = btf_named_start_id(btf, true);
+ r = btf_nr_types(btf) - 1;
+ while (l <= r) {
+ m = l + (r - l) / 2;
+ t = btf_type_by_id(btf, m);
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) >= 0) {
+ if (l == r)
+ return r;
+ r = m;
+ } else {
+ l = m + 1;
+ }
+ }
+
+ return btf_nr_types(btf);
+}
+
+s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
+{
+ const struct btf *base_btf = btf_base_btf(btf);
+ const struct btf_type *t;
+ const char *tname;
+ s32 id, total;
+
+ if (base_btf) {
+ id = btf_find_by_name_kind(base_btf, name, kind);
+ if (id > 0)
+ return id;
+ }
+
+ total = btf_nr_types(btf);
+ if (btf->named_start_id > 0 && name[0]) {
+ id = btf_find_by_name_kind_bsearch(btf, name);
+ for (; id < total; id++) {
+ t = btf_type_by_id(btf, id);
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) != 0)
+ return -ENOENT;
+ if (BTF_INFO_KIND(t->info) == kind)
+ return id;
+ }
+ } else {
+ for (id = btf_start_id(btf); id < total; id++) {
+ t = btf_type_by_id(btf, id);
+ if (BTF_INFO_KIND(t->info) != kind)
+ continue;
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) == 0)
+ return id;
+ }
+ }
+
+ return -ENOENT;
+}
+
+
+const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *t = btf_type_by_id(btf, id);
+
+ while (btf_type_is_modifier(t)) {
+ id = t->type;
+ t = btf_type_by_id(btf, t->type);
+ }
+
+ if (res_id)
+ *res_id = id;
+
+ return t;
+}
+
+const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *t;
+
+ t = btf_type_skip_modifiers(btf, id, NULL);
+ if (!btf_type_is_ptr(t))
+ return NULL;
+
+ return btf_type_skip_modifiers(btf, t->type, res_id);
+}
+
+const struct btf_type *btf_type_resolve_func_ptr(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *ptype;
+
+ ptype = btf_type_resolve_ptr(btf, id, res_id);
+ if (ptype && btf_type_is_func_proto(ptype))
+ return ptype;
+
+ return NULL;
+}
+
+/* Types that act only as a source, not sink or intermediate
+ * type when resolving.
+ */
+static bool btf_type_is_resolve_source_only(const struct btf_type *t)
+{
+ return btf_type_is_var(t) ||
+ btf_type_is_decl_tag(t) ||
+ btf_type_is_datasec(t);
+}
+
+/* What types need to be resolved?
+ *
+ * btf_type_is_modifier() is an obvious one.
+ *
+ * btf_type_is_struct() because its member refers to
+ * another type (through member->type).
+ *
+ * btf_type_is_var() because the variable refers to
+ * another type. btf_type_is_datasec() holds multiple
+ * btf_type_is_var() types that need resolving.
+ *
+ * btf_type_is_array() because its element (array->type)
+ * refers to another type. Array can be thought of a
+ * special case of struct while array just has the same
+ * member-type repeated by array->nelems of times.
+ */
+static bool btf_type_needs_resolve(const struct btf_type *t)
+{
+ return btf_type_is_modifier(t) ||
+ btf_type_is_ptr(t) ||
+ btf_type_is_struct(t) ||
+ btf_type_is_array(t) ||
+ btf_type_is_var(t) ||
+ btf_type_is_func(t) ||
+ btf_type_is_decl_tag(t) ||
+ btf_type_is_datasec(t);
+}
+
+/* t->size can be used */
+bool btf_type_has_size(const struct btf_type *t)
+{
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_INT:
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ case BTF_KIND_ENUM:
+ case BTF_KIND_DATASEC:
+ case BTF_KIND_FLOAT:
+ case BTF_KIND_ENUM64:
+ return true;
+ }
+
+ return false;
+}
+
+static const char *btf_int_encoding_str(u8 encoding)
+{
+ if (encoding == 0)
+ return "(none)";
+ else if (encoding == BTF_INT_SIGNED)
+ return "SIGNED";
+ else if (encoding == BTF_INT_CHAR)
+ return "CHAR";
+ else if (encoding == BTF_INT_BOOL)
+ return "BOOL";
+ else
+ return "UNKN";
+}
+
+static u32 btf_type_int(const struct btf_type *t)
+{
+ return *(u32 *)(t + 1);
+}
+
+static const struct btf_array *btf_type_array(const struct btf_type *t)
+{
+ return (const struct btf_array *)(t + 1);
+}
+
+static const struct btf_enum *btf_type_enum(const struct btf_type *t)
+{
+ return (const struct btf_enum *)(t + 1);
+}
+
+static const struct btf_var *btf_type_var(const struct btf_type *t)
+{
+ return (const struct btf_var *)(t + 1);
+}
+
+const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t)
+{
+ return (const struct btf_decl_tag *)(t + 1);
+}
+
+static const struct btf_enum64 *btf_type_enum64(const struct btf_type *t)
+{
+ return (const struct btf_enum64 *)(t + 1);
+}
+
+static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
+{
+ return kind_ops[BTF_INFO_KIND(t->info)];
+}
+
+static bool btf_name_offset_valid(const struct btf *btf, u32 offset)
+{
+ if (!BTF_STR_OFFSET_VALID(offset))
+ return false;
+
+ while (offset < btf->start_str_off)
+ btf = btf->base_btf;
+
+ offset -= btf->start_str_off;
+ return offset < btf->hdr.str_len;
+}
+
+static bool __btf_name_char_ok(char c, bool first)
+{
+ if ((first ? !isalpha(c) :
+ !isalnum(c)) &&
+ c != '_' &&
+ c != '.')
+ return false;
+ return true;
+}
+
+const char *btf_str_by_offset(const struct btf *btf, u32 offset)
+{
+ while (offset < btf->start_str_off)
+ btf = btf->base_btf;
+
+ offset -= btf->start_str_off;
+ if (offset < btf->hdr.str_len)
+ return &btf->strings[offset];
+
+ return NULL;
+}
+
+static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
+{
+ /* offset must be valid */
+ const char *src = btf_str_by_offset(btf, offset);
+ const char *src_limit;
+
+ if (!__btf_name_char_ok(*src, true))
+ return false;
+
+ /* set a limit on identifier length */
+ src_limit = src + KSYM_NAME_LEN;
+ src++;
+ while (*src && src < src_limit) {
+ if (!__btf_name_char_ok(*src, false))
+ return false;
+ src++;
+ }
+
+ return !*src;
+}
+
+/* Allow any printable character in DATASEC names */
+static bool btf_name_valid_section(const struct btf *btf, u32 offset)
+{
+ /* offset must be valid */
+ const char *src = btf_str_by_offset(btf, offset);
+ const char *src_limit;
+
+ if (!*src)
+ return false;
+
+ /* set a limit on identifier length */
+ src_limit = src + KSYM_NAME_LEN;
+ while (*src && src < src_limit) {
+ if (!isprint(*src))
+ return false;
+ src++;
+ }
+
+ return !*src;
+}
+
+const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
+{
+ const char *name;
+
+ if (!offset)
+ return "(anon)";
+
+ name = btf_str_by_offset(btf, offset);
+ return name ?: "(invalid-name-offset)";
+}
+
+const char *btf_name_by_offset(const struct btf *btf, u32 offset)
+{
+ return btf_str_by_offset(btf, offset);
+}
+
+const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ type_id -= btf->start_id;
+ if (type_id >= btf->nr_types)
+ return NULL;
+ return btf->types[type_id];
+}
+EXPORT_SYMBOL_GPL(btf_type_by_id);
+
+/*
+ * Check that the type @t is a regular int. This means that @t is not
+ * a bit field and it has the same size as either of u8/u16/u32/u64
+ * or __int128. If @expected_size is not zero, then size of @t should
+ * be the same. A caller should already have checked that the type @t
+ * is an integer.
+ */
+static bool __btf_type_int_is_regular(const struct btf_type *t, size_t expected_size)
+{
+ u32 int_data = btf_type_int(t);
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ u8 nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
+
+ return BITS_PER_BYTE_MASKED(nr_bits) == 0 &&
+ BTF_INT_OFFSET(int_data) == 0 &&
+ (nr_bytes <= 16 && is_power_of_2(nr_bytes)) &&
+ (expected_size == 0 || nr_bytes == expected_size);
+}
+
+static bool btf_type_int_is_regular(const struct btf_type *t)
+{
+ return __btf_type_int_is_regular(t, 0);
+}
+
+bool btf_type_is_i32(const struct btf_type *t)
+{
+ return btf_type_is_int(t) && __btf_type_int_is_regular(t, 4);
+}
+
+bool btf_type_is_i64(const struct btf_type *t)
+{
+ return btf_type_is_int(t) && __btf_type_int_is_regular(t, 8);
+}
+
+bool btf_type_is_primitive(const struct btf_type *t)
+{
+ return (btf_type_is_int(t) && btf_type_int_is_regular(t)) ||
+ btf_is_any_enum(t);
+}
+
+/*
+ * Check that given struct member is a regular int with expected
+ * offset and size.
+ */
+bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
+ const struct btf_member *m,
+ u32 expected_offset, u32 expected_size)
+{
+ const struct btf_type *t;
+ u32 id, int_data;
+ u8 nr_bits;
+
+ id = m->type;
+ t = btf_type_id_size(btf, &id, NULL);
+ if (!t || !btf_type_is_int(t))
+ return false;
+
+ int_data = btf_type_int(t);
+ nr_bits = BTF_INT_BITS(int_data);
+ if (btf_type_kflag(s)) {
+ u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
+ u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
+
+ /* if kflag set, int should be a regular int and
+ * bit offset should be at byte boundary.
+ */
+ return !bitfield_size &&
+ BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
+ BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
+ }
+
+ if (BTF_INT_OFFSET(int_data) ||
+ BITS_PER_BYTE_MASKED(m->offset) ||
+ BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
+ BITS_PER_BYTE_MASKED(nr_bits) ||
+ BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
+ return false;
+
+ return true;
+}
+
+/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
+static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
+ u32 id)
+{
+ const struct btf_type *t = btf_type_by_id(btf, id);
+
+ while (btf_type_is_modifier(t) &&
+ BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
+ t = btf_type_by_id(btf, t->type);
+ }
+
+ return t;
+}
+
+#define BTF_SHOW_MAX_ITER 10
+
+#define BTF_KIND_BIT(kind) (1ULL << kind)
+
+/*
+ * Populate show->state.name with type name information.
+ * Format of type name is
+ *
+ * [.member_name = ] (type_name)
+ */
+static const char *btf_show_name(struct btf_show *show)
+{
+ /* BTF_MAX_ITER array suffixes "[]" */
+ const char *array_suffixes = "[][][][][][][][][][]";
+ const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
+ /* BTF_MAX_ITER pointer suffixes "*" */
+ const char *ptr_suffixes = "**********";
+ const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
+ const char *name = NULL, *prefix = "", *parens = "";
+ const struct btf_member *m = show->state.member;
+ const struct btf_type *t;
+ const struct btf_array *array;
+ u32 id = show->state.type_id;
+ const char *member = NULL;
+ bool show_member = false;
+ u64 kinds = 0;
+ int i;
+
+ show->state.name[0] = '\0';
+
+ /*
+ * Don't show type name if we're showing an array member;
+ * in that case we show the array type so don't need to repeat
+ * ourselves for each member.
+ */
+ if (show->state.array_member)
+ return "";
+
+ /* Retrieve member name, if any. */
+ if (m) {
+ member = btf_name_by_offset(show->btf, m->name_off);
+ show_member = strlen(member) > 0;
+ id = m->type;
+ }
+
+ /*
+ * Start with type_id, as we have resolved the struct btf_type *
+ * via btf_modifier_show() past the parent typedef to the child
+ * struct, int etc it is defined as. In such cases, the type_id
+ * still represents the starting type while the struct btf_type *
+ * in our show->state points at the resolved type of the typedef.
+ */
+ t = btf_type_by_id(show->btf, id);
+ if (!t)
+ return "";
+
+ /*
+ * The goal here is to build up the right number of pointer and
+ * array suffixes while ensuring the type name for a typedef
+ * is represented. Along the way we accumulate a list of
+ * BTF kinds we have encountered, since these will inform later
+ * display; for example, pointer types will not require an
+ * opening "{" for struct, we will just display the pointer value.
+ *
+ * We also want to accumulate the right number of pointer or array
+ * indices in the format string while iterating until we get to
+ * the typedef/pointee/array member target type.
+ *
+ * We start by pointing at the end of pointer and array suffix
+ * strings; as we accumulate pointers and arrays we move the pointer
+ * or array string backwards so it will show the expected number of
+ * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
+ * and/or arrays and typedefs are supported as a precaution.
+ *
+ * We also want to get typedef name while proceeding to resolve
+ * type it points to so that we can add parentheses if it is a
+ * "typedef struct" etc.
+ */
+ for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
+
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_TYPEDEF:
+ if (!name)
+ name = btf_name_by_offset(show->btf,
+ t->name_off);
+ kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
+ id = t->type;
+ break;
+ case BTF_KIND_ARRAY:
+ kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
+ parens = "[";
+ if (!t)
+ return "";
+ array = btf_type_array(t);
+ if (array_suffix > array_suffixes)
+ array_suffix -= 2;
+ id = array->type;
+ break;
+ case BTF_KIND_PTR:
+ kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
+ if (ptr_suffix > ptr_suffixes)
+ ptr_suffix -= 1;
+ id = t->type;
+ break;
+ default:
+ id = 0;
+ break;
+ }
+ if (!id)
+ break;
+ t = btf_type_skip_qualifiers(show->btf, id);
+ }
+ /* We may not be able to represent this type; bail to be safe */
+ if (i == BTF_SHOW_MAX_ITER)
+ return "";
+
+ if (!name)
+ name = btf_name_by_offset(show->btf, t->name_off);
+
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
+ "struct" : "union";
+ /* if it's an array of struct/union, parens is already set */
+ if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
+ parens = "{";
+ break;
+ case BTF_KIND_ENUM:
+ case BTF_KIND_ENUM64:
+ prefix = "enum";
+ break;
+ default:
+ break;
+ }
+
+ /* pointer does not require parens */
+ if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
+ parens = "";
+ /* typedef does not require struct/union/enum prefix */
+ if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
+ prefix = "";
+
+ if (!name)
+ name = "";
+
+ /* Even if we don't want type name info, we want parentheses etc */
+ if (show->flags & BTF_SHOW_NONAME)
+ snprintf(show->state.name, sizeof(show->state.name), "%s",
+ parens);
+ else
+ snprintf(show->state.name, sizeof(show->state.name),
+ "%s%s%s(%s%s%s%s%s%s)%s",
+ /* first 3 strings comprise ".member = " */
+ show_member ? "." : "",
+ show_member ? member : "",
+ show_member ? " = " : "",
+ /* ...next is our prefix (struct, enum, etc) */
+ prefix,
+ strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
+ /* ...this is the type name itself */
+ name,
+ /* ...suffixed by the appropriate '*', '[]' suffixes */
+ strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
+ array_suffix, parens);
+
+ return show->state.name;
+}
+
+static const char *__btf_show_indent(struct btf_show *show)
+{
+ const char *indents = " ";
+ const char *indent = &indents[strlen(indents)];
+
+ if ((indent - show->state.depth) >= indents)
+ return indent - show->state.depth;
+ return indents;
+}
+
+static const char *btf_show_indent(struct btf_show *show)
+{
+ return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
+}
+
+static const char *btf_show_newline(struct btf_show *show)
+{
+ return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
+}
+
+static const char *btf_show_delim(struct btf_show *show)
+{
+ if (show->state.depth == 0)
+ return "";
+
+ if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
+ BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
+ return "|";
+
+ return ",";
+}
+
+__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
+{
+ va_list args;
+
+ if (!show->state.depth_check) {
+ va_start(args, fmt);
+ show->showfn(show, fmt, args);
+ va_end(args);
+ }
+}
+
+/* Macros are used here as btf_show_type_value[s]() prepends and appends
+ * format specifiers to the format specifier passed in; these do the work of
+ * adding indentation, delimiters etc while the caller simply has to specify
+ * the type value(s) in the format specifier + value(s).
+ */
+#define btf_show_type_value(show, fmt, value) \
+ do { \
+ if ((value) != (__typeof__(value))0 || \
+ (show->flags & BTF_SHOW_ZERO) || \
+ show->state.depth == 0) { \
+ btf_show(show, "%s%s" fmt "%s%s", \
+ btf_show_indent(show), \
+ btf_show_name(show), \
+ value, btf_show_delim(show), \
+ btf_show_newline(show)); \
+ if (show->state.depth > show->state.depth_to_show) \
+ show->state.depth_to_show = show->state.depth; \
+ } \
+ } while (0)
+
+#define btf_show_type_values(show, fmt, ...) \
+ do { \
+ btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
+ btf_show_name(show), \
+ __VA_ARGS__, btf_show_delim(show), \
+ btf_show_newline(show)); \
+ if (show->state.depth > show->state.depth_to_show) \
+ show->state.depth_to_show = show->state.depth; \
+ } while (0)
+
+/* How much is left to copy to safe buffer after @data? */
+static int btf_show_obj_size_left(struct btf_show *show, void *data)
+{
+ return show->obj.head + show->obj.size - data;
+}
+
+/* Is object pointed to by @data of @size already copied to our safe buffer? */
+static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
+{
+ return data >= show->obj.data &&
+ (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
+}
+
+/*
+ * If object pointed to by @data of @size falls within our safe buffer, return
+ * the equivalent pointer to the same safe data. Assumes
+ * copy_from_kernel_nofault() has already happened and our safe buffer is
+ * populated.
+ */
+static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
+{
+ if (btf_show_obj_is_safe(show, data, size))
+ return show->obj.safe + (data - show->obj.data);
+ return NULL;
+}
+
+/*
+ * Return a safe-to-access version of data pointed to by @data.
+ * We do this by copying the relevant amount of information
+ * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
+ *
+ * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
+ * safe copy is needed.
+ *
+ * Otherwise we need to determine if we have the required amount
+ * of data (determined by the @data pointer and the size of the
+ * largest base type we can encounter (represented by
+ * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
+ * that we will be able to print some of the current object,
+ * and if more is needed a copy will be triggered.
+ * Some objects such as structs will not fit into the buffer;
+ * in such cases additional copies when we iterate over their
+ * members may be needed.
+ *
+ * btf_show_obj_safe() is used to return a safe buffer for
+ * btf_show_start_type(); this ensures that as we recurse into
+ * nested types we always have safe data for the given type.
+ * This approach is somewhat wasteful; it's possible for example
+ * that when iterating over a large union we'll end up copying the
+ * same data repeatedly, but the goal is safety not performance.
+ * We use stack data as opposed to per-CPU buffers because the
+ * iteration over a type can take some time, and preemption handling
+ * would greatly complicate use of the safe buffer.
+ */
+static void *btf_show_obj_safe(struct btf_show *show,
+ const struct btf_type *t,
+ void *data)
+{
+ const struct btf_type *rt;
+ int size_left, size;
+ void *safe = NULL;
+
+ if (show->flags & BTF_SHOW_UNSAFE)
+ return data;
+
+ rt = btf_resolve_size(show->btf, t, &size);
+ if (IS_ERR(rt)) {
+ show->state.status = PTR_ERR(rt);
+ return NULL;
+ }
+
+ /*
+ * Is this toplevel object? If so, set total object size and
+ * initialize pointers. Otherwise check if we still fall within
+ * our safe object data.
+ */
+ if (show->state.depth == 0) {
+ show->obj.size = size;
+ show->obj.head = data;
+ } else {
+ /*
+ * If the size of the current object is > our remaining
+ * safe buffer we _may_ need to do a new copy. However
+ * consider the case of a nested struct; it's size pushes
+ * us over the safe buffer limit, but showing any individual
+ * struct members does not. In such cases, we don't need
+ * to initiate a fresh copy yet; however we definitely need
+ * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
+ * in our buffer, regardless of the current object size.
+ * The logic here is that as we resolve types we will
+ * hit a base type at some point, and we need to be sure
+ * the next chunk of data is safely available to display
+ * that type info safely. We cannot rely on the size of
+ * the current object here because it may be much larger
+ * than our current buffer (e.g. task_struct is 8k).
+ * All we want to do here is ensure that we can print the
+ * next basic type, which we can if either
+ * - the current type size is within the safe buffer; or
+ * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
+ * the safe buffer.
+ */
+ safe = __btf_show_obj_safe(show, data,
+ min(size,
+ BTF_SHOW_OBJ_BASE_TYPE_SIZE));
+ }
+
+ /*
+ * We need a new copy to our safe object, either because we haven't
+ * yet copied and are initializing safe data, or because the data
+ * we want falls outside the boundaries of the safe object.
+ */
+ if (!safe) {
+ size_left = btf_show_obj_size_left(show, data);
+ if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
+ size_left = BTF_SHOW_OBJ_SAFE_SIZE;
+ show->state.status = copy_from_kernel_nofault(show->obj.safe,
+ data, size_left);
+ if (!show->state.status) {
+ show->obj.data = data;
+ safe = show->obj.safe;
+ }
+ }
+
+ return safe;
+}
+
+/*
+ * Set the type we are starting to show and return a safe data pointer
+ * to be used for showing the associated data.
+ */
+static void *btf_show_start_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id, void *data)
+{
+ show->state.type = t;
+ show->state.type_id = type_id;
+ show->state.name[0] = '\0';
+
+ return btf_show_obj_safe(show, t, data);
+}
+
+static void btf_show_end_type(struct btf_show *show)
+{
+ show->state.type = NULL;
+ show->state.type_id = 0;
+ show->state.name[0] = '\0';
+}
+
+static void *btf_show_start_aggr_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id, void *data)
+{
+ void *safe_data = btf_show_start_type(show, t, type_id, data);
+
+ if (!safe_data)
+ return safe_data;
+
+ btf_show(show, "%s%s%s", btf_show_indent(show),
+ btf_show_name(show),
+ btf_show_newline(show));
+ show->state.depth++;
+ return safe_data;
+}
+
+static void btf_show_end_aggr_type(struct btf_show *show,
+ const char *suffix)
+{
+ show->state.depth--;
+ btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
+ btf_show_delim(show), btf_show_newline(show));
+ btf_show_end_type(show);
+}
+
+static void btf_show_start_member(struct btf_show *show,
+ const struct btf_member *m)
+{
+ show->state.member = m;
+}
+
+static void btf_show_start_array_member(struct btf_show *show)
+{
+ show->state.array_member = 1;
+ btf_show_start_member(show, NULL);
+}
+
+static void btf_show_end_member(struct btf_show *show)
+{
+ show->state.member = NULL;
+}
+
+static void btf_show_end_array_member(struct btf_show *show)
+{
+ show->state.array_member = 0;
+ btf_show_end_member(show);
+}
+
+static void *btf_show_start_array_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id,
+ u16 array_encoding,
+ void *data)
+{
+ show->state.array_encoding = array_encoding;
+ show->state.array_terminated = 0;
+ return btf_show_start_aggr_type(show, t, type_id, data);
+}
+
+static void btf_show_end_array_type(struct btf_show *show)
+{
+ show->state.array_encoding = 0;
+ show->state.array_terminated = 0;
+ btf_show_end_aggr_type(show, "]");
+}
+
+static void *btf_show_start_struct_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id,
+ void *data)
+{
+ return btf_show_start_aggr_type(show, t, type_id, data);
+}
+
+static void btf_show_end_struct_type(struct btf_show *show)
+{
+ btf_show_end_aggr_type(show, "}");
+}
+
+__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
+ const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+
+__printf(2, 3) static void btf_verifier_log(struct btf_verifier_env *env,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+
+__printf(4, 5) static void __btf_verifier_log_type(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ bool log_details,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ struct btf *btf = env->btf;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL) {
+ /* btf verifier prints all types it is processing via
+ * btf_verifier_log_type(..., fmt = NULL).
+ * Skip those prints for in-kernel BTF verification.
+ */
+ if (!fmt)
+ return;
+
+ /* Skip logging when loading module BTF with mismatches permitted */
+ if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
+ return;
+ }
+
+ __btf_verifier_log(log, "[%u] %s %s%s",
+ env->log_type_id,
+ btf_type_str(t),
+ __btf_name_by_offset(btf, t->name_off),
+ log_details ? " " : "");
+
+ if (log_details)
+ btf_type_ops(t)->log_details(env, t);
+
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+#define btf_verifier_log_type(env, t, ...) \
+ __btf_verifier_log_type((env), (t), true, __VA_ARGS__)
+#define btf_verifier_log_basic(env, t, ...) \
+ __btf_verifier_log_type((env), (t), false, __VA_ARGS__)
+
+__printf(4, 5)
+static void btf_verifier_log_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ struct btf *btf = env->btf;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL) {
+ if (!fmt)
+ return;
+
+ /* Skip logging when loading module BTF with mismatches permitted */
+ if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
+ return;
+ }
+
+ /* The CHECK_META phase already did a btf dump.
+ *
+ * If member is logged again, it must hit an error in
+ * parsing this member. It is useful to print out which
+ * struct this member belongs to.
+ */
+ if (env->phase != CHECK_META)
+ btf_verifier_log_type(env, struct_type, NULL);
+
+ if (btf_type_kflag(struct_type))
+ __btf_verifier_log(log,
+ "\t%s type_id=%u bitfield_size=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type,
+ BTF_MEMBER_BITFIELD_SIZE(member->offset),
+ BTF_MEMBER_BIT_OFFSET(member->offset));
+ else
+ __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type, member->offset);
+
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+__printf(4, 5)
+static void btf_verifier_log_vsi(struct btf_verifier_env *env,
+ const struct btf_type *datasec_type,
+ const struct btf_var_secinfo *vsi,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+ if (log->level == BPF_LOG_KERNEL && !fmt)
+ return;
+ if (env->phase != CHECK_META)
+ btf_verifier_log_type(env, datasec_type, NULL);
+
+ __btf_verifier_log(log, "\t type_id=%u offset=%u size=%u",
+ vsi->type, vsi->offset, vsi->size);
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+static void btf_verifier_log_hdr(struct btf_verifier_env *env,
+ u32 btf_data_size)
+{
+ struct bpf_verifier_log *log = &env->log;
+ const struct btf *btf = env->btf;
+ const struct btf_header *hdr;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL)
+ return;
+ hdr = &btf->hdr;
+ __btf_verifier_log(log, "magic: 0x%x\n", hdr->magic);
+ __btf_verifier_log(log, "version: %u\n", hdr->version);
+ __btf_verifier_log(log, "flags: 0x%x\n", hdr->flags);
+ __btf_verifier_log(log, "hdr_len: %u\n", hdr->hdr_len);
+ __btf_verifier_log(log, "type_off: %u\n", hdr->type_off);
+ __btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
+ __btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
+ __btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
+ __btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
+}
+
+static int btf_add_type(struct btf_verifier_env *env, struct btf_type *t)
+{
+ struct btf *btf = env->btf;
+
+ if (btf->types_size == btf->nr_types) {
+ /* Expand 'types' array */
+
+ struct btf_type **new_types;
+ u32 expand_by, new_size;
+
+ if (btf->start_id + btf->types_size == BTF_MAX_TYPE) {
+ btf_verifier_log(env, "Exceeded max num of types");
+ return -E2BIG;
+ }
+
+ expand_by = max_t(u32, btf->types_size >> 2, 16);
+ new_size = min_t(u32, BTF_MAX_TYPE,
+ btf->types_size + expand_by);
+
+ new_types = kvzalloc_objs(*new_types, new_size,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!new_types)
+ return -ENOMEM;
+
+ if (btf->nr_types == 0) {
+ if (!btf->base_btf) {
+ /* lazily init VOID type */
+ new_types[0] = &btf_void;
+ btf->nr_types++;
+ }
+ } else {
+ memcpy(new_types, btf->types,
+ sizeof(*btf->types) * btf->nr_types);
+ }
+
+ kvfree(btf->types);
+ btf->types = new_types;
+ btf->types_size = new_size;
+ }
+
+ btf->types[btf->nr_types++] = t;
+
+ return 0;
+}
+
+
+void __weak btf_free_bpf_data(struct btf *btf)
+{
+}
+
+void btf_free(struct btf *btf)
+{
+ btf_free_bpf_data(btf);
+ kvfree(btf->types);
+ kvfree(btf->resolved_sizes);
+ kvfree(btf->resolved_ids);
+ /* vmlinux does not allocate btf->data, it simply points it at
+ * __start_BTF.
+ */
+ if (!btf_is_vmlinux(btf))
+ kvfree(btf->data);
+ kvfree(btf->base_id_map);
+ kfree(btf);
+}
+
+
+const char *btf_get_name(const struct btf *btf)
+{
+ return btf->name;
+}
+
+void btf_get(struct btf *btf)
+{
+ refcount_inc(&btf->refcnt);
+}
+
+void __weak btf_put_bpf(struct btf *btf)
+{
+ btf_free(btf);
+}
+
+void btf_put(struct btf *btf)
+{
+ if (btf && refcount_dec_and_test(&btf->refcnt)) {
+ btf_put_bpf(btf);
+ }
+}
+
+struct btf *btf_base_btf(const struct btf *btf)
+{
+ return btf->base_btf;
+}
+
+const struct btf_header *btf_header(const struct btf *btf)
+{
+ return &btf->hdr;
+}
+
+void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
+{
+ btf->base_btf = (struct btf *)base_btf;
+ btf->start_id = btf_nr_types(base_btf);
+ btf->start_str_off = base_btf->hdr.str_len;
+}
+
+static int env_resolve_init(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ u32 nr_types = btf->nr_types;
+ u32 *resolved_sizes = NULL;
+ u32 *resolved_ids = NULL;
+ u8 *visit_states = NULL;
+
+ resolved_sizes = kvcalloc(nr_types, sizeof(*resolved_sizes),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!resolved_sizes)
+ goto nomem;
+
+ resolved_ids = kvcalloc(nr_types, sizeof(*resolved_ids),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!resolved_ids)
+ goto nomem;
+
+ visit_states = kvcalloc(nr_types, sizeof(*visit_states),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!visit_states)
+ goto nomem;
+
+ btf->resolved_sizes = resolved_sizes;
+ btf->resolved_ids = resolved_ids;
+ env->visit_states = visit_states;
+
+ return 0;
+
+nomem:
+ kvfree(resolved_sizes);
+ kvfree(resolved_ids);
+ kvfree(visit_states);
+ return -ENOMEM;
+}
+
+void btf_verifier_env_free(struct btf_verifier_env *env)
+{
+ kvfree(env->visit_states);
+ kfree(env);
+}
+
+static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
+ const struct btf_type *next_type)
+{
+ switch (env->resolve_mode) {
+ case RESOLVE_TBD:
+ /* int, enum or void is a sink */
+ return !btf_type_needs_resolve(next_type);
+ case RESOLVE_PTR:
+ /* int, enum, void, struct, array, func or func_proto is a sink
+ * for ptr
+ */
+ return !btf_type_is_modifier(next_type) &&
+ !btf_type_is_ptr(next_type);
+ case RESOLVE_STRUCT_OR_ARRAY:
+ /* int, enum, void, ptr, func or func_proto is a sink
+ * for struct and array
+ */
+ return !btf_type_is_modifier(next_type) &&
+ !btf_type_is_array(next_type) &&
+ !btf_type_is_struct(next_type);
+ default:
+ BUG();
+ }
+}
+
+static bool env_type_is_resolved(const struct btf_verifier_env *env,
+ u32 type_id)
+{
+ /* base BTF types should be resolved by now */
+ if (type_id < env->btf->start_id)
+ return true;
+
+ return env->visit_states[type_id - env->btf->start_id] == RESOLVED;
+}
+
+static int env_stack_push(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id)
+{
+ const struct btf *btf = env->btf;
+ struct resolve_vertex *v;
+
+ if (env->top_stack == MAX_RESOLVE_DEPTH)
+ return -E2BIG;
+
+ if (type_id < btf->start_id
+ || env->visit_states[type_id - btf->start_id] != NOT_VISITED)
+ return -EEXIST;
+
+ env->visit_states[type_id - btf->start_id] = VISITED;
+
+ v = &env->stack[env->top_stack++];
+ v->t = t;
+ v->type_id = type_id;
+ v->next_member = 0;
+
+ if (env->resolve_mode == RESOLVE_TBD) {
+ if (btf_type_is_ptr(t))
+ env->resolve_mode = RESOLVE_PTR;
+ else if (btf_type_is_struct(t) || btf_type_is_array(t))
+ env->resolve_mode = RESOLVE_STRUCT_OR_ARRAY;
+ }
+
+ return 0;
+}
+
+static void env_stack_set_next_member(struct btf_verifier_env *env,
+ u16 next_member)
+{
+ env->stack[env->top_stack - 1].next_member = next_member;
+}
+
+static void env_stack_pop_resolved(struct btf_verifier_env *env,
+ u32 resolved_type_id,
+ u32 resolved_size)
+{
+ u32 type_id = env->stack[--(env->top_stack)].type_id;
+ struct btf *btf = env->btf;
+
+ type_id -= btf->start_id; /* adjust to local type id */
+ btf->resolved_sizes[type_id] = resolved_size;
+ btf->resolved_ids[type_id] = resolved_type_id;
+ env->visit_states[type_id] = RESOLVED;
+}
+
+static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
+{
+ return env->top_stack ? &env->stack[env->top_stack - 1] : NULL;
+}
+
+/* Resolve the size of a passed-in "type"
+ *
+ * type: is an array (e.g. u32 array[x][y])
+ * return type: type "u32[x][y]", i.e. BTF_KIND_ARRAY,
+ * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
+ * corresponds to the return type.
+ * *elem_type: u32
+ * *elem_id: id of u32
+ * *total_nelems: (x * y). Hence, individual elem size is
+ * (*type_size / *total_nelems)
+ * *type_id: id of type if it's changed within the function, 0 if not
+ *
+ * type: is not an array (e.g. const struct X)
+ * return type: type "struct X"
+ * *type_size: sizeof(struct X)
+ * *elem_type: same as return type ("struct X")
+ * *elem_id: 0
+ * *total_nelems: 1
+ * *type_id: id of type if it's changed within the function, 0 if not
+ */
+const struct btf_type *
+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size, const struct btf_type **elem_type,
+ u32 *elem_id, u32 *total_nelems, u32 *type_id)
+{
+ const struct btf_type *array_type = NULL;
+ const struct btf_array *array = NULL;
+ u32 i, size, nelems = 1, id = 0;
+
+ for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
+ switch (BTF_INFO_KIND(type->info)) {
+ /* type->size can be used */
+ case BTF_KIND_INT:
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ case BTF_KIND_ENUM:
+ case BTF_KIND_FLOAT:
+ case BTF_KIND_ENUM64:
+ size = type->size;
+ goto resolved;
+
+ case BTF_KIND_PTR:
+ size = sizeof(void *);
+ goto resolved;
+
+ /* Modifiers */
+ case BTF_KIND_TYPEDEF:
+ case BTF_KIND_VOLATILE:
+ case BTF_KIND_CONST:
+ case BTF_KIND_RESTRICT:
+ case BTF_KIND_TYPE_TAG:
+ id = type->type;
+ type = btf_type_by_id(btf, type->type);
+ break;
+
+ case BTF_KIND_ARRAY:
+ if (!array_type)
+ array_type = type;
+ array = btf_type_array(type);
+ if (nelems && array->nelems > U32_MAX / nelems)
+ return ERR_PTR(-EINVAL);
+ nelems *= array->nelems;
+ type = btf_type_by_id(btf, array->type);
+ break;
+
+ /* type without size */
+ default:
+ return ERR_PTR(-EINVAL);
+ }
+ }
+
+ return ERR_PTR(-EINVAL);
+
+resolved:
+ if (nelems && size > U32_MAX / nelems)
+ return ERR_PTR(-EINVAL);
+
+ *type_size = nelems * size;
+ if (total_nelems)
+ *total_nelems = nelems;
+ if (elem_type)
+ *elem_type = type;
+ if (elem_id)
+ *elem_id = array ? array->type : 0;
+ if (type_id && id)
+ *type_id = id;
+
+ return array_type ? : type;
+}
+
+const struct btf_type *
+btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size)
+{
+ return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
+}
+
+static u32 btf_resolved_type_id(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ return btf->resolved_ids[type_id - btf->start_id];
+}
+
+/* The input param "type_id" must point to a needs_resolve type */
+static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
+ u32 *type_id)
+{
+ *type_id = btf_resolved_type_id(btf, *type_id);
+ return btf_type_by_id(btf, *type_id);
+}
+
+static u32 btf_resolved_type_size(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ return btf->resolved_sizes[type_id - btf->start_id];
+}
+
+const struct btf_type *btf_type_id_size(const struct btf *btf,
+ u32 *type_id, u32 *ret_size)
+{
+ const struct btf_type *size_type;
+ u32 size_type_id = *type_id;
+ u32 size = 0;
+
+ size_type = btf_type_by_id(btf, size_type_id);
+ if (btf_type_nosize_or_null(size_type))
+ return NULL;
+
+ if (btf_type_has_size(size_type)) {
+ size = size_type->size;
+ } else if (btf_type_is_array(size_type)) {
+ size = btf_resolved_type_size(btf, size_type_id);
+ } else if (btf_type_is_ptr(size_type)) {
+ size = sizeof(void *);
+ } else {
+ if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) &&
+ !btf_type_is_var(size_type)))
+ return NULL;
+
+ size_type_id = btf_resolved_type_id(btf, size_type_id);
+ size_type = btf_type_by_id(btf, size_type_id);
+ if (btf_type_nosize_or_null(size_type))
+ return NULL;
+ else if (btf_type_has_size(size_type))
+ size = size_type->size;
+ else if (btf_type_is_array(size_type))
+ size = btf_resolved_type_size(btf, size_type_id);
+ else if (btf_type_is_ptr(size_type))
+ size = sizeof(void *);
+ else
+ return NULL;
+ }
+
+ *type_id = size_type_id;
+ if (ret_size)
+ *ret_size = size;
+
+ return size_type;
+}
+
+static int btf_df_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ btf_verifier_log_basic(env, struct_type,
+ "Unsupported check_member");
+ return -EINVAL;
+}
+
+static int btf_df_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ btf_verifier_log_basic(env, struct_type,
+ "Unsupported check_kflag_member");
+ return -EINVAL;
+}
+
+/* Used for ptr, array struct/union and float type members.
+ * int, enum and modifier types have their specific callback functions.
+ */
+static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ /* bitfield size is 0, so member->offset represents bit offset only.
+ * It is safe to call non kflag check_member variants.
+ */
+ return btf_type_ops(member_type)->check_member(env, struct_type,
+ member,
+ member_type);
+}
+
+static int btf_df_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ btf_verifier_log_basic(env, v->t, "Unsupported resolve");
+ return -EINVAL;
+}
+
+static void btf_df_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offsets,
+ struct btf_show *show)
+{
+ btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
+}
+
+static int btf_int_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 int_data = btf_type_int(member_type);
+ u32 struct_bits_off = member->offset;
+ u32 struct_size = struct_type->size;
+ u32 nr_copy_bits;
+ u32 bytes_offset;
+
+ if (U32_MAX - struct_bits_off < BTF_INT_OFFSET(int_data)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "bits_offset exceeds U32_MAX");
+ return -EINVAL;
+ }
+
+ struct_bits_off += BTF_INT_OFFSET(int_data);
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ nr_copy_bits = BTF_INT_BITS(int_data) +
+ BITS_PER_BYTE_MASKED(struct_bits_off);
+
+ if (nr_copy_bits > BITS_PER_U128) {
+ btf_verifier_log_member(env, struct_type, member,
+ "nr_copy_bits exceeds 128");
+ return -EINVAL;
+ }
+
+ if (struct_size < bytes_offset ||
+ struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_int_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
+ u32 int_data = btf_type_int(member_type);
+ u32 struct_size = struct_type->size;
+ u32 nr_copy_bits;
+
+ /* a regular int type is required for the kflag int member */
+ if (!btf_type_int_is_regular(member_type)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member base type");
+ return -EINVAL;
+ }
+
+ /* check sanity of bitfield size */
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_int_data_bits = BTF_INT_BITS(int_data);
+ if (!nr_bits) {
+ /* Not a bitfield member, member offset must be at byte
+ * boundary.
+ */
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member offset");
+ return -EINVAL;
+ }
+
+ nr_bits = nr_int_data_bits;
+ } else if (nr_bits > nr_int_data_bits) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
+ if (nr_copy_bits > BITS_PER_U128) {
+ btf_verifier_log_member(env, struct_type, member,
+ "nr_copy_bits exceeds 128");
+ return -EINVAL;
+ }
+
+ if (struct_size < bytes_offset ||
+ struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_int_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 int_data, nr_bits, meta_needed = sizeof(int_data);
+ u16 encoding;
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ int_data = btf_type_int(t);
+ if (int_data & ~BTF_INT_MASK) {
+ btf_verifier_log_basic(env, t, "Invalid int_data:%x",
+ int_data);
+ return -EINVAL;
+ }
+
+ nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
+
+ if (nr_bits > BITS_PER_U128) {
+ btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
+ BITS_PER_U128);
+ return -EINVAL;
+ }
+
+ if (BITS_ROUNDUP_BYTES(nr_bits) > t->size) {
+ btf_verifier_log_type(env, t, "nr_bits exceeds type_size");
+ return -EINVAL;
+ }
+
+ /*
+ * Only one of the encoding bits is allowed and it
+ * should be sufficient for the pretty print purpose (i.e. decoding).
+ * Multiple bits can be allowed later if it is found
+ * to be insufficient.
+ */
+ encoding = BTF_INT_ENCODING(int_data);
+ if (encoding &&
+ encoding != BTF_INT_SIGNED &&
+ encoding != BTF_INT_CHAR &&
+ encoding != BTF_INT_BOOL) {
+ btf_verifier_log_type(env, t, "Unsupported encoding");
+ return -ENOTSUPP;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_int_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ int int_data = btf_type_int(t);
+
+ btf_verifier_log(env,
+ "size=%u bits_offset=%u nr_bits=%u encoding=%s",
+ t->size, BTF_INT_OFFSET(int_data),
+ BTF_INT_BITS(int_data),
+ btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
+}
+
+static void btf_int128_print(struct btf_show *show, void *data)
+{
+ /* data points to a __int128 number.
+ * Suppose
+ * int128_num = *(__int128 *)data;
+ * The below formulas shows what upper_num and lower_num represents:
+ * upper_num = int128_num >> 64;
+ * lower_num = int128_num & 0xffffffffFFFFFFFFULL;
+ */
+ u64 upper_num, lower_num;
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ upper_num = *(u64 *)data;
+ lower_num = *(u64 *)(data + 8);
+#else
+ upper_num = *(u64 *)(data + 8);
+ lower_num = *(u64 *)data;
+#endif
+ if (upper_num == 0)
+ btf_show_type_value(show, "0x%llx", lower_num);
+ else
+ btf_show_type_values(show, "0x%llx%016llx", upper_num,
+ lower_num);
+}
+
+static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
+ u16 right_shift_bits)
+{
+ u64 upper_num, lower_num;
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ upper_num = print_num[0];
+ lower_num = print_num[1];
+#else
+ upper_num = print_num[1];
+ lower_num = print_num[0];
+#endif
+
+ /* shake out un-needed bits by shift/or operations */
+ if (left_shift_bits >= 64) {
+ upper_num = lower_num << (left_shift_bits - 64);
+ lower_num = 0;
+ } else {
+ upper_num = (upper_num << left_shift_bits) |
+ (lower_num >> (64 - left_shift_bits));
+ lower_num = lower_num << left_shift_bits;
+ }
+
+ if (right_shift_bits >= 64) {
+ lower_num = upper_num >> (right_shift_bits - 64);
+ upper_num = 0;
+ } else {
+ lower_num = (lower_num >> right_shift_bits) |
+ (upper_num << (64 - right_shift_bits));
+ upper_num = upper_num >> right_shift_bits;
+ }
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ print_num[0] = upper_num;
+ print_num[1] = lower_num;
+#else
+ print_num[0] = lower_num;
+ print_num[1] = upper_num;
+#endif
+}
+
+static void btf_bitfield_show(void *data, u8 bits_offset,
+ u8 nr_bits, struct btf_show *show)
+{
+ u16 left_shift_bits, right_shift_bits;
+ u8 nr_copy_bytes;
+ u8 nr_copy_bits;
+ u64 print_num[2] = {};
+
+ nr_copy_bits = nr_bits + bits_offset;
+ nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
+
+ memcpy(print_num, data, nr_copy_bytes);
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ left_shift_bits = bits_offset;
+#else
+ left_shift_bits = BITS_PER_U128 - nr_copy_bits;
+#endif
+ right_shift_bits = BITS_PER_U128 - nr_bits;
+
+ btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
+ btf_int128_print(show, print_num);
+}
+
+
+static void btf_int_bits_show(const struct btf *btf,
+ const struct btf_type *t,
+ void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ u32 int_data = btf_type_int(t);
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ u8 total_bits_offset;
+
+ /*
+ * bits_offset is at most 7.
+ * BTF_INT_OFFSET() cannot exceed 128 bits.
+ */
+ total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
+ data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
+ bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
+ btf_bitfield_show(data, bits_offset, nr_bits, show);
+}
+
+static void btf_int_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ u32 int_data = btf_type_int(t);
+ u8 encoding = BTF_INT_ENCODING(int_data);
+ bool sign = encoding & BTF_INT_SIGNED;
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ void *safe_data;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ if (bits_offset || BTF_INT_OFFSET(int_data) ||
+ BITS_PER_BYTE_MASKED(nr_bits)) {
+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
+ goto out;
+ }
+
+ switch (nr_bits) {
+ case 128:
+ btf_int128_print(show, safe_data);
+ break;
+ case 64:
+ if (sign)
+ btf_show_type_value(show, "%lld", *(s64 *)safe_data);
+ else
+ btf_show_type_value(show, "%llu", *(u64 *)safe_data);
+ break;
+ case 32:
+ if (sign)
+ btf_show_type_value(show, "%d", *(s32 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u32 *)safe_data);
+ break;
+ case 16:
+ if (sign)
+ btf_show_type_value(show, "%d", *(s16 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u16 *)safe_data);
+ break;
+ case 8:
+ if (show->state.array_encoding == BTF_INT_CHAR) {
+ /* check for null terminator */
+ if (show->state.array_terminated)
+ break;
+ if (*(char *)data == '\0') {
+ show->state.array_terminated = 1;
+ break;
+ }
+ if (isprint(*(char *)data)) {
+ btf_show_type_value(show, "'%c'",
+ *(char *)safe_data);
+ break;
+ }
+ }
+ if (sign)
+ btf_show_type_value(show, "%d", *(s8 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u8 *)safe_data);
+ break;
+ default:
+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
+ break;
+ }
+out:
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations int_ops = {
+ .check_meta = btf_int_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_int_check_member,
+ .check_kflag_member = btf_int_check_kflag_member,
+ .log_details = btf_int_log,
+ .show = btf_int_show,
+};
+
+static int btf_modifier_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id = member->type;
+ struct btf_member resolved_member;
+ struct btf *btf = env->btf;
+
+ resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
+ if (!resolved_type) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ resolved_member = *member;
+ resolved_member.type = resolved_type_id;
+
+ return btf_type_ops(resolved_type)->check_member(env, struct_type,
+ &resolved_member,
+ resolved_type);
+}
+
+static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id = member->type;
+ struct btf_member resolved_member;
+ struct btf *btf = env->btf;
+
+ resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
+ if (!resolved_type) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ resolved_member = *member;
+ resolved_member.type = resolved_type_id;
+
+ return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
+ &resolved_member,
+ resolved_type);
+}
+
+static int btf_ptr_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_size, struct_bits_off, bytes_offset;
+
+ struct_size = struct_type->size;
+ struct_bits_off = member->offset;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ if (struct_size - bytes_offset < sizeof(void *)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_ref_type_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const char *value;
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t) && !btf_type_is_type_tag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!BTF_TYPE_ID_VALID(t->type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ /* typedef/type_tag type must have a valid name, and other ref types,
+ * volatile, const, restrict, should have a null name.
+ */
+ if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ } else if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPE_TAG) {
+ value = btf_name_by_offset(env->btf, t->name_off);
+ if (!value || !value[0]) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ } else {
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_modifier_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *t = v->t;
+ const struct btf_type *next_type;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ /* Figure out the resolved next_type_id with size.
+ * They will be stored in the current modifier's
+ * resolved_ids and resolved_sizes such that it can
+ * save us a few type-following when we use it later (e.g. in
+ * pretty print).
+ */
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ if (env_type_is_resolved(env, next_type_id))
+ next_type = btf_type_id_resolve(btf, &next_type_id);
+
+ /* "typedef void new_void", "const void"...etc */
+ if (!btf_type_is_void(next_type) &&
+ !btf_type_is_fwd(next_type) &&
+ !btf_type_is_func_proto(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static int btf_var_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ if (btf_type_is_modifier(next_type)) {
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id;
+
+ resolved_type_id = next_type_id;
+ resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
+
+ if (btf_type_is_ptr(resolved_type) &&
+ !env_type_is_resolve_sink(env, resolved_type) &&
+ !env_type_is_resolved(env, resolved_type_id))
+ return env_stack_push(env, resolved_type,
+ resolved_type_id);
+ }
+
+ /* We must resolve to something concrete at this point, no
+ * forward types or similar that would resolve to size of
+ * zero is allowed.
+ */
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static int btf_ptr_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ /* If the modifier was RESOLVED during RESOLVE_STRUCT_OR_ARRAY,
+ * the modifier may have stopped resolving when it was resolved
+ * to a ptr (last-resolved-ptr).
+ *
+ * We now need to continue from the last-resolved-ptr to
+ * ensure the last-resolved-ptr will not referring back to
+ * the current ptr (t).
+ */
+ if (btf_type_is_modifier(next_type)) {
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id;
+
+ resolved_type_id = next_type_id;
+ resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
+
+ if (btf_type_is_ptr(resolved_type) &&
+ !env_type_is_resolve_sink(env, resolved_type) &&
+ !env_type_is_resolved(env, resolved_type_id))
+ return env_stack_push(env, resolved_type,
+ resolved_type_id);
+ }
+
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ if (env_type_is_resolved(env, next_type_id))
+ next_type = btf_type_id_resolve(btf, &next_type_id);
+
+ if (!btf_type_is_void(next_type) &&
+ !btf_type_is_fwd(next_type) &&
+ !btf_type_is_func_proto(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static void btf_modifier_show(const struct btf *btf,
+ const struct btf_type *t,
+ u32 type_id, void *data,
+ u8 bits_offset, struct btf_show *show)
+{
+ if (btf->resolved_ids)
+ t = btf_type_id_resolve(btf, &type_id);
+ else
+ t = btf_type_skip_modifiers(btf, type_id, NULL);
+
+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
+}
+
+static void btf_var_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ t = btf_type_id_resolve(btf, &type_id);
+
+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
+}
+
+static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ void *safe_data;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
+ if (show->flags & BTF_SHOW_PTR_RAW)
+ btf_show_type_value(show, "0x%px", *(void **)safe_data);
+ else
+ btf_show_type_value(show, "0x%p", *(void **)safe_data);
+ btf_show_end_type(show);
+}
+
+static void btf_ref_type_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "type_id=%u", t->type);
+}
+
+static const struct btf_kind_operations modifier_ops = {
+ .check_meta = btf_ref_type_check_meta,
+ .resolve = btf_modifier_resolve,
+ .check_member = btf_modifier_check_member,
+ .check_kflag_member = btf_modifier_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_modifier_show,
+};
+
+static const struct btf_kind_operations ptr_ops = {
+ .check_meta = btf_ref_type_check_meta,
+ .resolve = btf_ptr_resolve,
+ .check_member = btf_ptr_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_ptr_show,
+};
+
+static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (t->type) {
+ btf_verifier_log_type(env, t, "type != 0");
+ return -EINVAL;
+ }
+
+ /* fwd type must have a valid name */
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static void btf_fwd_type_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "%s", btf_type_kflag(t) ? "union" : "struct");
+}
+
+static const struct btf_kind_operations fwd_ops = {
+ .check_meta = btf_fwd_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_fwd_type_log,
+ .show = btf_df_show,
+};
+
+static int btf_array_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+ u32 array_type_id, array_size;
+ struct btf *btf = env->btf;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ array_type_id = member->type;
+ btf_type_id_size(btf, &array_type_id, &array_size);
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < array_size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_array_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_array *array = btf_type_array(t);
+ u32 meta_needed = sizeof(*array);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ /* array type should not have a name */
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (t->size) {
+ btf_verifier_log_type(env, t, "size != 0");
+ return -EINVAL;
+ }
+
+ /* Array elem type and index type cannot be in type void,
+ * so !array->type and !array->index_type are not allowed.
+ */
+ if (!array->type || !BTF_TYPE_ID_VALID(array->type)) {
+ btf_verifier_log_type(env, t, "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (!array->index_type || !BTF_TYPE_ID_VALID(array->index_type)) {
+ btf_verifier_log_type(env, t, "Invalid index");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static int btf_array_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_array *array = btf_type_array(v->t);
+ const struct btf_type *elem_type, *index_type;
+ u32 elem_type_id, index_type_id;
+ struct btf *btf = env->btf;
+ u32 elem_size;
+
+ /* Check array->index_type */
+ index_type_id = array->index_type;
+ index_type = btf_type_by_id(btf, index_type_id);
+ if (btf_type_nosize_or_null(index_type) ||
+ btf_type_is_resolve_source_only(index_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid index");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, index_type) &&
+ !env_type_is_resolved(env, index_type_id))
+ return env_stack_push(env, index_type, index_type_id);
+
+ index_type = btf_type_id_size(btf, &index_type_id, NULL);
+ if (!index_type || !btf_type_is_int(index_type) ||
+ !btf_type_int_is_regular(index_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid index");
+ return -EINVAL;
+ }
+
+ /* Check array->type */
+ elem_type_id = array->type;
+ elem_type = btf_type_by_id(btf, elem_type_id);
+ if (btf_type_nosize_or_null(elem_type) ||
+ btf_type_is_resolve_source_only(elem_type)) {
+ btf_verifier_log_type(env, v->t,
+ "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, elem_type) &&
+ !env_type_is_resolved(env, elem_type_id))
+ return env_stack_push(env, elem_type, elem_type_id);
+
+ elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
+ if (!elem_type) {
+ btf_verifier_log_type(env, v->t, "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_int(elem_type) && !btf_type_int_is_regular(elem_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid array of int");
+ return -EINVAL;
+ }
+
+ if (array->nelems && elem_size > U32_MAX / array->nelems) {
+ btf_verifier_log_type(env, v->t,
+ "Array size overflows U32_MAX");
+ return -EINVAL;
+ }
+
+ env_stack_pop_resolved(env, elem_type_id, elem_size * array->nelems);
+
+ return 0;
+}
+
+static void btf_array_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_array *array = btf_type_array(t);
+
+ btf_verifier_log(env, "type_id=%u index_type_id=%u nr_elems=%u",
+ array->type, array->index_type, array->nelems);
+}
+
+static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_array *array = btf_type_array(t);
+ const struct btf_kind_operations *elem_ops;
+ const struct btf_type *elem_type;
+ u32 i, elem_size = 0, elem_type_id;
+ u16 encoding = 0;
+
+ elem_type_id = array->type;
+ elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
+ if (elem_type && btf_type_has_size(elem_type))
+ elem_size = elem_type->size;
+
+ if (elem_type && btf_type_is_int(elem_type)) {
+ u32 int_type = btf_type_int(elem_type);
+
+ encoding = BTF_INT_ENCODING(int_type);
+
+ /*
+ * BTF_INT_CHAR encoding never seems to be set for
+ * char arrays, so if size is 1 and element is
+ * printable as a char, we'll do that.
+ */
+ if (elem_size == 1)
+ encoding = BTF_INT_CHAR;
+ }
+
+ if (!btf_show_start_array_type(show, t, type_id, encoding, data))
+ return;
+
+ if (!elem_type)
+ goto out;
+ elem_ops = btf_type_ops(elem_type);
+
+ for (i = 0; i < array->nelems; i++) {
+
+ btf_show_start_array_member(show);
+
+ elem_ops->show(btf, elem_type, elem_type_id, data,
+ bits_offset, show);
+ data += elem_size;
+
+ btf_show_end_array_member(show);
+
+ if (show->state.array_terminated)
+ break;
+ }
+out:
+ btf_show_end_array_type(show);
+}
+
+static void btf_array_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *m = show->state.member;
+
+ /*
+ * First check if any members would be shown (are non-zero).
+ * See comments above "struct btf_show" definition for more
+ * details on how this works at a high-level.
+ */
+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
+ if (!show->state.depth_check) {
+ show->state.depth_check = show->state.depth + 1;
+ show->state.depth_to_show = 0;
+ }
+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
+ show->state.member = m;
+
+ if (show->state.depth_check != show->state.depth + 1)
+ return;
+ show->state.depth_check = 0;
+
+ if (show->state.depth_to_show <= show->state.depth)
+ return;
+ /*
+ * Reaching here indicates we have recursed and found
+ * non-zero array member(s).
+ */
+ }
+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
+}
+
+static const struct btf_kind_operations array_ops = {
+ .check_meta = btf_array_check_meta,
+ .resolve = btf_array_resolve,
+ .check_member = btf_array_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_array_log,
+ .show = btf_array_show,
+};
+
+static int btf_struct_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < member_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_struct_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ bool is_union = BTF_INFO_KIND(t->info) == BTF_KIND_UNION;
+ const struct btf_member *member;
+ u32 meta_needed, last_offset;
+ struct btf *btf = env->btf;
+ u32 struct_size = t->size;
+ u32 offset;
+ u16 i;
+
+ meta_needed = btf_type_vlen(t) * sizeof(*member);
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ /* struct type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ last_offset = 0;
+ for_each_member(i, t, member) {
+ if (!btf_name_offset_valid(btf, member->name_off)) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member name_offset:%u",
+ member->name_off);
+ return -EINVAL;
+ }
+
+ /* struct member either no name or a valid one */
+ if (member->name_off &&
+ !btf_name_valid_identifier(btf, member->name_off)) {
+ btf_verifier_log_member(env, t, member, "Invalid name");
+ return -EINVAL;
+ }
+ /* A member cannot be in type void */
+ if (!member->type || !BTF_TYPE_ID_VALID(member->type)) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid type_id");
+ return -EINVAL;
+ }
+
+ offset = __btf_member_bit_offset(t, member);
+ if (is_union && offset) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member bits_offset");
+ return -EINVAL;
+ }
+
+ /*
+ * ">" instead of ">=" because the last member could be
+ * "char a[0];"
+ */
+ if (last_offset > offset) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member bits_offset");
+ return -EINVAL;
+ }
+
+ if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
+ btf_verifier_log_member(env, t, member,
+ "Member bits_offset exceeds its struct size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_member(env, t, member, NULL);
+ last_offset = offset;
+ }
+
+ return meta_needed;
+}
+
+static int btf_struct_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_member *member;
+ int err;
+ u16 i;
+
+ /* Before continue resolving the next_member,
+ * ensure the last member is indeed resolved to a
+ * type with size info.
+ */
+ if (v->next_member) {
+ const struct btf_type *last_member_type;
+ const struct btf_member *last_member;
+ u32 last_member_type_id;
+
+ last_member = btf_type_member(v->t) + v->next_member - 1;
+ last_member_type_id = last_member->type;
+ if (WARN_ON_ONCE(!env_type_is_resolved(env,
+ last_member_type_id)))
+ return -EINVAL;
+
+ last_member_type = btf_type_by_id(env->btf,
+ last_member_type_id);
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
+ last_member,
+ last_member_type);
+ else
+ err = btf_type_ops(last_member_type)->check_member(env, v->t,
+ last_member,
+ last_member_type);
+ if (err)
+ return err;
+ }
+
+ for_each_member_from(i, v->next_member, v->t, member) {
+ u32 member_type_id = member->type;
+ const struct btf_type *member_type = btf_type_by_id(env->btf,
+ member_type_id);
+
+ if (btf_type_nosize_or_null(member_type) ||
+ btf_type_is_resolve_source_only(member_type)) {
+ btf_verifier_log_member(env, v->t, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, member_type) &&
+ !env_type_is_resolved(env, member_type_id)) {
+ env_stack_set_next_member(env, i + 1);
+ return env_stack_push(env, member_type, member_type_id);
+ }
+
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
+ member,
+ member_type);
+ else
+ err = btf_type_ops(member_type)->check_member(env, v->t,
+ member,
+ member_type);
+ if (err)
+ return err;
+ }
+
+ env_stack_pop_resolved(env, 0, 0);
+
+ return 0;
+}
+
+static void btf_struct_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+
+static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *member;
+ void *safe_data;
+ u32 i;
+
+ safe_data = btf_show_start_struct_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ for_each_member(i, t, member) {
+ const struct btf_type *member_type = btf_type_by_id(btf,
+ member->type);
+ const struct btf_kind_operations *ops;
+ u32 member_offset, bitfield_size;
+ u32 bytes_offset;
+ u8 bits8_offset;
+
+ btf_show_start_member(show, member);
+
+ member_offset = __btf_member_bit_offset(t, member);
+ bitfield_size = __btf_member_bitfield_size(t, member);
+ bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
+ bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
+ if (bitfield_size) {
+ safe_data = btf_show_start_type(show, member_type,
+ member->type,
+ data + bytes_offset);
+ if (safe_data)
+ btf_bitfield_show(safe_data,
+ bits8_offset,
+ bitfield_size, show);
+ btf_show_end_type(show);
+ } else {
+ ops = btf_type_ops(member_type);
+ ops->show(btf, member_type, member->type,
+ data + bytes_offset, bits8_offset, show);
+ }
+
+ btf_show_end_member(show);
+ }
+
+ btf_show_end_struct_type(show);
+}
+
+static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *m = show->state.member;
+
+ /*
+ * First check if any members would be shown (are non-zero).
+ * See comments above "struct btf_show" definition for more
+ * details on how this works at a high-level.
+ */
+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
+ if (!show->state.depth_check) {
+ show->state.depth_check = show->state.depth + 1;
+ show->state.depth_to_show = 0;
+ }
+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
+ /* Restore saved member data here */
+ show->state.member = m;
+ if (show->state.depth_check != show->state.depth + 1)
+ return;
+ show->state.depth_check = 0;
+
+ if (show->state.depth_to_show <= show->state.depth)
+ return;
+ /*
+ * Reaching here indicates we have recursed and found
+ * non-zero child values.
+ */
+ }
+
+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
+}
+
+static const struct btf_kind_operations struct_ops = {
+ .check_meta = btf_struct_check_meta,
+ .resolve = btf_struct_resolve,
+ .check_member = btf_struct_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_struct_log,
+ .show = btf_struct_show,
+};
+
+static int btf_enum_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < member_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, bytes_end, struct_size;
+ u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
+
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ if (!nr_bits) {
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ nr_bits = int_bitsize;
+ } else if (nr_bits > int_bitsize) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
+ if (struct_size < bytes_end) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_enum_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_enum *enums = btf_type_enum(t);
+ struct btf *btf = env->btf;
+ const char *fmt_str;
+ u16 i, nr_enums;
+ u32 meta_needed;
+
+ nr_enums = btf_type_vlen(t);
+ meta_needed = nr_enums * sizeof(*enums);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->size > 8 || !is_power_of_2(t->size)) {
+ btf_verifier_log_type(env, t, "Unexpected size");
+ return -EINVAL;
+ }
+
+ /* enum type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for (i = 0; i < nr_enums; i++) {
+ if (!btf_name_offset_valid(btf, enums[i].name_off)) {
+ btf_verifier_log(env, "\tInvalid name_offset:%u",
+ enums[i].name_off);
+ return -EINVAL;
+ }
+
+ /* enum member must have a valid name */
+ if (!enums[i].name_off ||
+ !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (env->log.level == BPF_LOG_KERNEL)
+ continue;
+ fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
+ btf_verifier_log(env, fmt_str,
+ __btf_name_by_offset(btf, enums[i].name_off),
+ enums[i].val);
+ }
+
+ return meta_needed;
+}
+
+static void btf_enum_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_enum *enums = btf_type_enum(t);
+ u32 i, nr_enums = btf_type_vlen(t);
+ void *safe_data;
+ int v;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ v = *(int *)safe_data;
+
+ for (i = 0; i < nr_enums; i++) {
+ if (v != enums[i].val)
+ continue;
+
+ btf_show_type_value(show, "%s",
+ __btf_name_by_offset(btf,
+ enums[i].name_off));
+
+ btf_show_end_type(show);
+ return;
+ }
+
+ if (btf_type_kflag(t))
+ btf_show_type_value(show, "%d", v);
+ else
+ btf_show_type_value(show, "%u", v);
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations enum_ops = {
+ .check_meta = btf_enum_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_enum_check_member,
+ .check_kflag_member = btf_enum_check_kflag_member,
+ .log_details = btf_enum_log,
+ .show = btf_enum_show,
+};
+
+static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_enum64 *enums = btf_type_enum64(t);
+ struct btf *btf = env->btf;
+ const char *fmt_str;
+ u16 i, nr_enums;
+ u32 meta_needed;
+
+ nr_enums = btf_type_vlen(t);
+ meta_needed = nr_enums * sizeof(*enums);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->size > 8 || !is_power_of_2(t->size)) {
+ btf_verifier_log_type(env, t, "Unexpected size");
+ return -EINVAL;
+ }
+
+ /* enum type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for (i = 0; i < nr_enums; i++) {
+ if (!btf_name_offset_valid(btf, enums[i].name_off)) {
+ btf_verifier_log(env, "\tInvalid name_offset:%u",
+ enums[i].name_off);
+ return -EINVAL;
+ }
+
+ /* enum member must have a valid name */
+ if (!enums[i].name_off ||
+ !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (env->log.level == BPF_LOG_KERNEL)
+ continue;
+
+ fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
+ btf_verifier_log(env, fmt_str,
+ __btf_name_by_offset(btf, enums[i].name_off),
+ btf_enum64_value(enums + i));
+ }
+
+ return meta_needed;
+}
+
+static void btf_enum64_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_enum64 *enums = btf_type_enum64(t);
+ u32 i, nr_enums = btf_type_vlen(t);
+ void *safe_data;
+ s64 v;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ v = *(u64 *)safe_data;
+
+ for (i = 0; i < nr_enums; i++) {
+ if (v != btf_enum64_value(enums + i))
+ continue;
+
+ btf_show_type_value(show, "%s",
+ __btf_name_by_offset(btf,
+ enums[i].name_off));
+
+ btf_show_end_type(show);
+ return;
+ }
+
+ if (btf_type_kflag(t))
+ btf_show_type_value(show, "%lld", v);
+ else
+ btf_show_type_value(show, "%llu", v);
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations enum64_ops = {
+ .check_meta = btf_enum64_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_enum_check_member,
+ .check_kflag_member = btf_enum_check_kflag_member,
+ .log_details = btf_enum_log,
+ .show = btf_enum64_show,
+};
+
+static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 meta_needed = btf_type_vlen(t) * sizeof(struct btf_param);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_func_proto_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_param *args = (const struct btf_param *)(t + 1);
+ u16 nr_args = btf_type_vlen(t), i;
+
+ btf_verifier_log(env, "return=%u args=(", t->type);
+ if (!nr_args) {
+ btf_verifier_log(env, "void");
+ goto done;
+ }
+
+ if (nr_args == 1 && !args[0].type) {
+ /* Only one vararg */
+ btf_verifier_log(env, "vararg");
+ goto done;
+ }
+
+ btf_verifier_log(env, "%u %s", args[0].type,
+ __btf_name_by_offset(env->btf,
+ args[0].name_off));
+ for (i = 1; i < nr_args - 1; i++)
+ btf_verifier_log(env, ", %u %s", args[i].type,
+ __btf_name_by_offset(env->btf,
+ args[i].name_off));
+
+ if (nr_args > 1) {
+ const struct btf_param *last_arg = &args[nr_args - 1];
+
+ if (last_arg->type)
+ btf_verifier_log(env, ", %u %s", last_arg->type,
+ __btf_name_by_offset(env->btf,
+ last_arg->name_off));
+ else
+ btf_verifier_log(env, ", vararg");
+ }
+
+done:
+ btf_verifier_log(env, ")");
+}
+
+static const struct btf_kind_operations func_proto_ops = {
+ .check_meta = btf_func_proto_check_meta,
+ .resolve = btf_df_resolve,
+ /*
+ * BTF_KIND_FUNC_PROTO cannot be directly referred by
+ * a struct's member.
+ *
+ * It should be a function pointer instead.
+ * (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
+ *
+ * Hence, there is no btf_func_check_member().
+ */
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_func_proto_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_func_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t) > BTF_FUNC_GLOBAL) {
+ btf_verifier_log_type(env, t, "Invalid func linkage");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_func_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ int err;
+
+ err = btf_func_check(env, t);
+ if (err)
+ return err;
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+ return 0;
+}
+
+static const struct btf_kind_operations func_ops = {
+ .check_meta = btf_func_check_meta,
+ .resolve = btf_func_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_var_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_var *var;
+ u32 meta_needed = sizeof(*var);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ /* A var cannot be in type void */
+ if (!t->type || !BTF_TYPE_ID_VALID(t->type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ var = btf_type_var(t);
+ if (var->linkage != BTF_VAR_STATIC &&
+ var->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
+ btf_verifier_log_type(env, t, "Linkage not supported");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t)
+{
+ const struct btf_var *var = btf_type_var(t);
+
+ btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage);
+}
+
+static const struct btf_kind_operations var_ops = {
+ .check_meta = btf_var_check_meta,
+ .resolve = btf_var_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_var_log,
+ .show = btf_var_show,
+};
+
+static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_var_secinfo *vsi;
+ u64 last_vsi_end_off = 0, sum = 0;
+ u32 i, meta_needed;
+
+ meta_needed = btf_type_vlen(t) * sizeof(*vsi);
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (!t->size) {
+ btf_verifier_log_type(env, t, "size == 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!t->name_off ||
+ !btf_name_valid_section(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for_each_vsi(i, t, vsi) {
+ /* A var cannot be in type void */
+ if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid offset");
+ return -EINVAL;
+ }
+
+ if (!vsi->size || vsi->size > t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid size");
+ return -EINVAL;
+ }
+
+ last_vsi_end_off = vsi->offset + vsi->size;
+ if (last_vsi_end_off > t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid offset+size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_vsi(env, t, vsi, NULL);
+ sum += vsi->size;
+ }
+
+ if (t->size < sum) {
+ btf_verifier_log_type(env, t, "Invalid btf_info size");
+ return -EINVAL;
+ }
+
+ return meta_needed;
+}
+
+static int btf_datasec_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_var_secinfo *vsi;
+ struct btf *btf = env->btf;
+ u16 i;
+
+ env->resolve_mode = RESOLVE_TBD;
+ for_each_vsi_from(i, v->next_member, v->t, vsi) {
+ u32 var_type_id = vsi->type, type_id, type_size = 0;
+ const struct btf_type *var_type = btf_type_by_id(env->btf,
+ var_type_id);
+ if (!var_type || !btf_type_is_var(var_type)) {
+ btf_verifier_log_vsi(env, v->t, vsi,
+ "Not a VAR kind member");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, var_type) &&
+ !env_type_is_resolved(env, var_type_id)) {
+ env_stack_set_next_member(env, i + 1);
+ return env_stack_push(env, var_type, var_type_id);
+ }
+
+ type_id = var_type->type;
+ if (!btf_type_id_size(btf, &type_id, &type_size)) {
+ btf_verifier_log_vsi(env, v->t, vsi, "Invalid type");
+ return -EINVAL;
+ }
+
+ if (vsi->size < type_size) {
+ btf_verifier_log_vsi(env, v->t, vsi, "Invalid size");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, 0, 0);
+ return 0;
+}
+
+static void btf_datasec_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+static void btf_datasec_show(const struct btf *btf,
+ const struct btf_type *t, u32 type_id,
+ void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_var_secinfo *vsi;
+ const struct btf_type *var;
+ u32 i;
+
+ if (!btf_show_start_type(show, t, type_id, data))
+ return;
+
+ btf_show_type_value(show, "section (\"%s\") = {",
+ __btf_name_by_offset(btf, t->name_off));
+ for_each_vsi(i, t, vsi) {
+ var = btf_type_by_id(btf, vsi->type);
+ if (i)
+ btf_show(show, ",");
+ btf_type_ops(var)->show(btf, var, vsi->type,
+ data + vsi->offset, bits_offset, show);
+ }
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations datasec_ops = {
+ .check_meta = btf_datasec_check_meta,
+ .resolve = btf_datasec_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_datasec_log,
+ .show = btf_datasec_show,
+};
+
+static s32 btf_float_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (t->size != 2 && t->size != 4 && t->size != 8 && t->size != 12 &&
+ t->size != 16) {
+ btf_verifier_log_type(env, t, "Invalid type_size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_float_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u64 start_offset_bytes;
+ u64 end_offset_bytes;
+ u64 misalign_bits;
+ u64 align_bytes;
+ u64 align_bits;
+
+ /* Different architectures have different alignment requirements, so
+ * here we check only for the reasonable minimum. This way we ensure
+ * that types after CO-RE can pass the kernel BTF verifier.
+ */
+ align_bytes = min_t(u64, sizeof(void *), member_type->size);
+ align_bits = align_bytes * BITS_PER_BYTE;
+ div64_u64_rem(member->offset, align_bits, &misalign_bits);
+ if (misalign_bits) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not properly aligned");
+ return -EINVAL;
+ }
+
+ start_offset_bytes = member->offset / BITS_PER_BYTE;
+ end_offset_bytes = start_offset_bytes + member_type->size;
+ if (end_offset_bytes > struct_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void btf_float_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u", t->size);
+}
+
+static const struct btf_kind_operations float_ops = {
+ .check_meta = btf_float_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_float_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_float_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_decl_tag *tag;
+ u32 meta_needed = sizeof(*tag);
+ s32 component_idx;
+ const char *value;
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ value = btf_name_by_offset(env->btf, t->name_off);
+ if (!value || !value[0]) {
+ btf_verifier_log_type(env, t, "Invalid value");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ component_idx = btf_type_decl_tag(t)->component_idx;
+ if (component_idx < -1) {
+ btf_verifier_log_type(env, t, "Invalid component_idx");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static int btf_decl_tag_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+ s32 component_idx;
+ u32 vlen;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || !btf_type_is_decl_tag_target(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ component_idx = btf_type_decl_tag(t)->component_idx;
+ if (component_idx != -1) {
+ if (btf_type_is_var(next_type) || btf_type_is_typedef(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid component_idx");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_struct(next_type)) {
+ vlen = btf_type_vlen(next_type);
+ } else {
+ /* next_type should be a function */
+ next_type = btf_type_by_id(btf, next_type->type);
+ vlen = btf_type_vlen(next_type);
+ }
+
+ if ((u32)component_idx >= vlen) {
+ btf_verifier_log_type(env, v->t, "Invalid component_idx");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static void btf_decl_tag_log(struct btf_verifier_env *env, const struct btf_type *t)
+{
+ btf_verifier_log(env, "type=%u component_idx=%d", t->type,
+ btf_type_decl_tag(t)->component_idx);
+}
+
+static const struct btf_kind_operations decl_tag_ops = {
+ .check_meta = btf_decl_tag_check_meta,
+ .resolve = btf_decl_tag_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_decl_tag_log,
+ .show = btf_df_show,
+};
+
+static int btf_func_proto_check(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_type *ret_type;
+ const struct btf_param *args;
+ const struct btf *btf;
+ u16 nr_args, i;
+ int err;
+
+ btf = env->btf;
+ args = (const struct btf_param *)(t + 1);
+ nr_args = btf_type_vlen(t);
+
+ /* Check func return type which could be "void" (t->type == 0) */
+ if (t->type) {
+ u32 ret_type_id = t->type;
+
+ ret_type = btf_type_by_id(btf, ret_type_id);
+ if (!ret_type) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_resolve_source_only(ret_type)) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+
+ if (btf_type_needs_resolve(ret_type) &&
+ !env_type_is_resolved(env, ret_type_id)) {
+ err = btf_resolve(env, ret_type, ret_type_id);
+ if (err)
+ return err;
+ }
+
+ /* Ensure the return type is a type that has a size */
+ if (!btf_type_id_size(btf, &ret_type_id, NULL)) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+ }
+
+ if (!nr_args)
+ return 0;
+
+ /* Last func arg type_id could be 0 if it is a vararg */
+ if (!args[nr_args - 1].type) {
+ if (args[nr_args - 1].name_off) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u",
+ nr_args);
+ return -EINVAL;
+ }
+ nr_args--;
+ }
+
+ for (i = 0; i < nr_args; i++) {
+ const struct btf_type *arg_type;
+ u32 arg_type_id;
+
+ arg_type_id = args[i].type;
+ arg_type = btf_type_by_id(btf, arg_type_id);
+ if (!arg_type) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (btf_type_is_resolve_source_only(arg_type)) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (args[i].name_off &&
+ (!btf_name_offset_valid(btf, args[i].name_off) ||
+ !btf_name_valid_identifier(btf, args[i].name_off))) {
+ btf_verifier_log_type(env, t,
+ "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (btf_type_needs_resolve(arg_type) &&
+ !env_type_is_resolved(env, arg_type_id)) {
+ err = btf_resolve(env, arg_type, arg_type_id);
+ if (err)
+ return err;
+ }
+
+ if (!btf_type_id_size(btf, &arg_type_id, NULL)) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int btf_func_check(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_type *proto_type;
+ const struct btf_param *args;
+ const struct btf *btf;
+ u16 nr_args, i;
+
+ btf = env->btf;
+ proto_type = btf_type_by_id(btf, t->type);
+
+ if (!proto_type || !btf_type_is_func_proto(proto_type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ args = (const struct btf_param *)(proto_type + 1);
+ nr_args = btf_type_vlen(proto_type);
+ for (i = 0; i < nr_args; i++) {
+ if (!args[i].name_off && args[i].type) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = {
+ [BTF_KIND_INT] = &int_ops,
+ [BTF_KIND_PTR] = &ptr_ops,
+ [BTF_KIND_ARRAY] = &array_ops,
+ [BTF_KIND_STRUCT] = &struct_ops,
+ [BTF_KIND_UNION] = &struct_ops,
+ [BTF_KIND_ENUM] = &enum_ops,
+ [BTF_KIND_FWD] = &fwd_ops,
+ [BTF_KIND_TYPEDEF] = &modifier_ops,
+ [BTF_KIND_VOLATILE] = &modifier_ops,
+ [BTF_KIND_CONST] = &modifier_ops,
+ [BTF_KIND_RESTRICT] = &modifier_ops,
+ [BTF_KIND_FUNC] = &func_ops,
+ [BTF_KIND_FUNC_PROTO] = &func_proto_ops,
+ [BTF_KIND_VAR] = &var_ops,
+ [BTF_KIND_DATASEC] = &datasec_ops,
+ [BTF_KIND_FLOAT] = &float_ops,
+ [BTF_KIND_DECL_TAG] = &decl_tag_ops,
+ [BTF_KIND_TYPE_TAG] = &modifier_ops,
+ [BTF_KIND_ENUM64] = &enum64_ops,
+};
+
+static s32 btf_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 saved_meta_left = meta_left;
+ s32 var_meta_size;
+
+ if (meta_left < sizeof(*t)) {
+ btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu",
+ env->log_type_id, meta_left, sizeof(*t));
+ return -EINVAL;
+ }
+ meta_left -= sizeof(*t);
+
+ if (t->info & ~BTF_INFO_MASK) {
+ btf_verifier_log(env, "[%u] Invalid btf_info:%x",
+ env->log_type_id, t->info);
+ return -EINVAL;
+ }
+
+ if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
+ BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
+ btf_verifier_log(env, "[%u] Invalid kind:%u",
+ env->log_type_id, BTF_INFO_KIND(t->info));
+ return -EINVAL;
+ }
+
+ if (!btf_name_offset_valid(env->btf, t->name_off)) {
+ btf_verifier_log(env, "[%u] Invalid name_offset:%u",
+ env->log_type_id, t->name_off);
+ return -EINVAL;
+ }
+
+ var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left);
+ if (var_meta_size < 0)
+ return var_meta_size;
+
+ meta_left -= var_meta_size;
+
+ return saved_meta_left - meta_left;
+}
+
+int btf_check_all_metas(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ struct btf_header *hdr;
+ void *cur, *end;
+
+ hdr = &btf->hdr;
+ cur = btf->nohdr_data + hdr->type_off;
+ end = cur + hdr->type_len;
+
+ env->log_type_id = btf->base_btf ? btf->start_id : 1;
+ while (cur < end) {
+ struct btf_type *t = cur;
+ s32 meta_size;
+
+ meta_size = btf_check_meta(env, t, end - cur);
+ if (meta_size < 0)
+ return meta_size;
+
+ btf_add_type(env, t);
+ cur += meta_size;
+ env->log_type_id++;
+ }
+
+ return 0;
+}
+
+static bool btf_resolve_valid(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 type_id)
+{
+ struct btf *btf = env->btf;
+
+ if (!env_type_is_resolved(env, type_id))
+ return false;
+
+ if (btf_type_is_struct(t) || btf_type_is_datasec(t))
+ return !btf_resolved_type_id(btf, type_id) &&
+ !btf_resolved_type_size(btf, type_id);
+
+ if (btf_type_is_decl_tag(t) || btf_type_is_func(t))
+ return btf_resolved_type_id(btf, type_id) &&
+ !btf_resolved_type_size(btf, type_id);
+
+ if (btf_type_is_modifier(t) || btf_type_is_ptr(t) ||
+ btf_type_is_var(t)) {
+ t = btf_type_id_resolve(btf, &type_id);
+ return t &&
+ !btf_type_is_modifier(t) &&
+ !btf_type_is_var(t) &&
+ !btf_type_is_datasec(t);
+ }
+
+ if (btf_type_is_array(t)) {
+ const struct btf_array *array = btf_type_array(t);
+ const struct btf_type *elem_type;
+ u32 elem_type_id = array->type;
+ u32 elem_size;
+
+ elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
+ return elem_type && !btf_type_is_modifier(elem_type) &&
+ (array->nelems * elem_size ==
+ btf_resolved_type_size(btf, type_id));
+ }
+
+ return false;
+}
+
+static int btf_resolve(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id)
+{
+ u32 save_log_type_id = env->log_type_id;
+ const struct resolve_vertex *v;
+ int err = 0;
+
+ env->resolve_mode = RESOLVE_TBD;
+ env_stack_push(env, t, type_id);
+ while (!err && (v = env_stack_peak(env))) {
+ env->log_type_id = v->type_id;
+ err = btf_type_ops(v->t)->resolve(env, v);
+ }
+
+ env->log_type_id = type_id;
+ if (err == -E2BIG) {
+ btf_verifier_log_type(env, t,
+ "Exceeded max resolving depth:%u",
+ MAX_RESOLVE_DEPTH);
+ } else if (err == -EEXIST) {
+ btf_verifier_log_type(env, t, "Loop detected");
+ }
+
+ /* Final sanity check */
+ if (!err && !btf_resolve_valid(env, t, type_id)) {
+ btf_verifier_log_type(env, t, "Invalid resolve state");
+ err = -EINVAL;
+ }
+
+ env->log_type_id = save_log_type_id;
+ return err;
+}
+
+static int btf_check_all_types(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ const struct btf_type *t;
+ u32 type_id, i;
+ int err;
+
+ err = env_resolve_init(env);
+ if (err)
+ return err;
+
+ env->phase++;
+ for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
+ type_id = btf->start_id + i;
+ t = btf_type_by_id(btf, type_id);
+
+ env->log_type_id = type_id;
+ if (btf_type_needs_resolve(t) &&
+ !env_type_is_resolved(env, type_id)) {
+ err = btf_resolve(env, t, type_id);
+ if (err)
+ return err;
+ }
+
+ if (btf_type_is_func_proto(t)) {
+ err = btf_func_proto_check(env, t);
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+int btf_parse_type_sec(struct btf_verifier_env *env)
+{
+ const struct btf_header *hdr = &env->btf->hdr;
+ int err;
+
+ /* Type section must align to 4 bytes */
+ if (hdr->type_off & (sizeof(u32) - 1)) {
+ btf_verifier_log(env, "Unaligned type_off");
+ return -EINVAL;
+ }
+
+ if (!env->btf->base_btf && !hdr->type_len) {
+ btf_verifier_log(env, "No type found");
+ return -EINVAL;
+ }
+
+ err = btf_check_all_metas(env);
+ if (err)
+ return err;
+
+ return btf_check_all_types(env);
+}
+
+int btf_parse_str_sec(struct btf_verifier_env *env)
+{
+ const struct btf_header *hdr;
+ struct btf *btf = env->btf;
+ const char *start, *end;
+
+ hdr = &btf->hdr;
+ start = btf->nohdr_data + hdr->str_off;
+ end = start + hdr->str_len;
+
+ if (end != btf->data + btf->data_size) {
+ btf_verifier_log(env, "String section is not at the end");
+ return -EINVAL;
+ }
+
+ btf->strings = start;
+
+ if (btf->base_btf && !hdr->str_len)
+ return 0;
+ if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_NAME_OFFSET || end[-1]) {
+ btf_verifier_log(env, "Invalid string section");
+ return -EINVAL;
+ }
+ if (!btf->base_btf && start[0]) {
+ btf_verifier_log(env, "Invalid string section");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const size_t btf_sec_info_offset[] = {
+ offsetof(struct btf_header, type_off),
+ offsetof(struct btf_header, str_off),
+};
+
+static int btf_sec_info_cmp(const void *a, const void *b)
+{
+ const struct btf_sec_info *x = a;
+ const struct btf_sec_info *y = b;
+
+ return (int)(x->off - y->off) ? : (int)(x->len - y->len);
+}
+
+static int btf_check_sec_info(struct btf_verifier_env *env,
+ u32 btf_data_size)
+{
+ struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
+ u32 total, expected_total, i;
+ const struct btf_header *hdr;
+ const struct btf *btf;
+
+ btf = env->btf;
+ hdr = &btf->hdr;
+
+ /* Populate the secs from hdr */
+ for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++)
+ secs[i] = *(struct btf_sec_info *)((void *)hdr +
+ btf_sec_info_offset[i]);
+
+ sort(secs, ARRAY_SIZE(btf_sec_info_offset),
+ sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
+
+ /* Check for gaps and overlap among sections */
+ total = 0;
+ expected_total = btf_data_size - hdr->hdr_len;
+ for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++) {
+ if (expected_total < secs[i].off) {
+ btf_verifier_log(env, "Invalid section offset");
+ return -EINVAL;
+ }
+ if (total < secs[i].off) {
+ /* gap */
+ btf_verifier_log(env, "Unsupported section found");
+ return -EINVAL;
+ }
+ if (total > secs[i].off) {
+ btf_verifier_log(env, "Section overlap found");
+ return -EINVAL;
+ }
+ if (expected_total - total < secs[i].len) {
+ btf_verifier_log(env,
+ "Total section length too long");
+ return -EINVAL;
+ }
+ total += secs[i].len;
+ }
+
+ /* There is data other than hdr and known sections */
+ if (expected_total != total) {
+ btf_verifier_log(env, "Unsupported section found");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int btf_parse_hdr(struct btf_verifier_env *env)
+{
+ u32 hdr_len, hdr_copy, btf_data_size;
+ const struct btf_header *hdr;
+ struct btf *btf;
+
+ btf = env->btf;
+ btf_data_size = btf->data_size;
+
+ if (btf_data_size < offsetofend(struct btf_header, hdr_len)) {
+ btf_verifier_log(env, "hdr_len not found");
+ return -EINVAL;
+ }
+
+ hdr = btf->data;
+ hdr_len = hdr->hdr_len;
+ if (btf_data_size < hdr_len) {
+ btf_verifier_log(env, "btf_header not found");
+ return -EINVAL;
+ }
+
+ /* Ensure the unsupported header fields are zero */
+ if (hdr_len > sizeof(btf->hdr)) {
+ u8 *expected_zero = btf->data + sizeof(btf->hdr);
+ u8 *end = btf->data + hdr_len;
+
+ for (; expected_zero < end; expected_zero++) {
+ if (*expected_zero) {
+ btf_verifier_log(env, "Unsupported btf_header");
+ return -E2BIG;
+ }
+ }
+ }
+
+ hdr_copy = min_t(u32, hdr_len, sizeof(btf->hdr));
+ memcpy(&btf->hdr, btf->data, hdr_copy);
+
+ hdr = &btf->hdr;
+
+ btf_verifier_log_hdr(env, btf_data_size);
+
+ if (hdr->magic != BTF_MAGIC) {
+ btf_verifier_log(env, "Invalid magic");
+ return -EINVAL;
+ }
+
+ if (hdr->version != BTF_VERSION) {
+ btf_verifier_log(env, "Unsupported version");
+ return -ENOTSUPP;
+ }
+
+ if (hdr->flags) {
+ btf_verifier_log(env, "Unsupported flags");
+ return -ENOTSUPP;
+ }
+
+ if (!btf->base_btf && btf_data_size == hdr->hdr_len) {
+ btf_verifier_log(env, "No data");
+ return -EINVAL;
+ }
+
+ return btf_check_sec_info(env, btf_data_size);
+}
+
+
+int btf_check_type_tags(struct btf_verifier_env *env,
+ struct btf *btf, int start_id)
+{
+ int i, n, good_id = start_id - 1;
+ bool in_tags;
+
+ n = btf_nr_types(btf);
+ for (i = start_id; i < n; i++) {
+ const struct btf_type *t;
+ int chain_limit = 32;
+ u32 cur_id = i;
+
+ t = btf_type_by_id(btf, i);
+ if (!t)
+ return -EINVAL;
+ if (!btf_type_is_modifier(t))
+ continue;
+
+ cond_resched();
+
+ in_tags = btf_type_is_type_tag(t);
+ while (btf_type_is_modifier(t)) {
+ if (!chain_limit--) {
+ btf_verifier_log(env, "Max chain length or cycle detected");
+ return -ELOOP;
+ }
+ if (btf_type_is_type_tag(t)) {
+ if (!in_tags) {
+ btf_verifier_log(env, "Type tags don't precede modifiers");
+ return -EINVAL;
+ }
+ } else if (in_tags) {
+ in_tags = false;
+ }
+ if (cur_id <= good_id)
+ break;
+ /* Move to next type */
+ cur_id = t->type;
+ t = btf_type_by_id(btf, cur_id);
+ if (!t)
+ return -EINVAL;
+ }
+ good_id = i;
+ }
+ return 0;
+}
+
+
+struct btf *
+btf_parse_base(struct btf_verifier_env *env, const char *name,
+ void *data, unsigned int data_size)
+{
+ struct btf *btf = NULL;
+ int err;
+
+ if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
+ return ERR_PTR(-ENOENT);
+
+ btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
+ if (!btf) {
+ err = -ENOMEM;
+ goto errout;
+ }
+ env->btf = btf;
+
+ btf->data = data;
+ btf->data_size = data_size;
+ btf->kernel_btf = true;
+ btf->named_start_id = 0;
+ strscpy(btf->name, name);
+
+ err = btf_parse_hdr(env);
+ if (err)
+ goto errout;
+
+ btf->nohdr_data = btf->data + btf->hdr.hdr_len;
+
+ err = btf_parse_str_sec(env);
+ if (err)
+ goto errout;
+
+ err = btf_check_all_metas(env);
+ if (err)
+ goto errout;
+
+ err = btf_check_type_tags(env, btf, 1);
+ if (err)
+ goto errout;
+
+ btf_check_sorted(btf);
+ refcount_set(&btf->refcnt, 1);
+
+ return btf;
+
+errout:
+ if (btf) {
+ kvfree(btf->types);
+ kfree(btf);
+ }
+ return ERR_PTR(err);
+}
+
+
+static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
+ struct btf_show *show)
+{
+ const struct btf_type *t = btf_type_by_id(btf, type_id);
+
+ show->btf = btf;
+ memset(&show->state, 0, sizeof(show->state));
+ memset(&show->obj, 0, sizeof(show->obj));
+
+ btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
+}
+
+__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,
+ va_list args)
+{
+ seq_vprintf((struct seq_file *)show->target, fmt, args);
+}
+
+int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
+ void *obj, struct seq_file *m, u64 flags)
+{
+ struct btf_show sseq;
+
+ sseq.target = m;
+ sseq.showfn = btf_seq_show;
+ sseq.flags = flags;
+
+ btf_type_show(btf, type_id, obj, &sseq);
+
+ return sseq.state.status;
+}
+
+void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
+ struct seq_file *m)
+{
+ (void) btf_type_seq_show_flags(btf, type_id, obj, m,
+ BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
+ BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
+}
+
+struct btf_show_snprintf {
+ struct btf_show show;
+ int len_left; /* space left in string */
+ int len; /* length we would have written */
+};
+
+__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
+ va_list args)
+{
+ struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
+ int len;
+
+ len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
+
+ if (len < 0) {
+ ssnprintf->len_left = 0;
+ ssnprintf->len = len;
+ } else if (len >= ssnprintf->len_left) {
+ /* no space, drive on to get length we would have written */
+ ssnprintf->len_left = 0;
+ ssnprintf->len += len;
+ } else {
+ ssnprintf->len_left -= len;
+ ssnprintf->len += len;
+ show->target += len;
+ }
+}
+
+int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
+ char *buf, int len, u64 flags)
+{
+ struct btf_show_snprintf ssnprintf;
+
+ ssnprintf.show.target = buf;
+ ssnprintf.show.flags = flags;
+ ssnprintf.show.showfn = btf_snprintf_show;
+ ssnprintf.len_left = len;
+ ssnprintf.len = 0;
+
+ btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
+
+ /* If we encountered an error, return it. */
+ if (ssnprintf.show.state.status)
+ return ssnprintf.show.state.status;
+
+ /* Otherwise return length we would have written */
+ return ssnprintf.len;
+}
+
+
+bool btf_is_kernel(const struct btf *btf)
+{
+ return btf->kernel_btf;
+}
+
+bool btf_is_module(const struct btf *btf)
+{
+ return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
+}
+
+
+bool btf_param_match_suffix(const struct btf *btf,
+ const struct btf_param *arg,
+ const char *suffix)
+{
+ int suffix_len = strlen(suffix), len;
+ const char *param_name;
+
+ /* In the future, this can be ported to use BTF tagging */
+ param_name = btf_name_by_offset(btf, arg->name_off);
+ if (str_is_empty(param_name))
+ return false;
+ len = strlen(param_name);
+ if (len <= suffix_len)
+ return false;
+ param_name += len - suffix_len;
+ return !strncmp(param_name, suffix, suffix_len);
+}
diff --git a/kernel/btf/btf.h b/kernel/btf/btf.h
new file mode 100644
index 0000000000000..f66d9300adb31
--- /dev/null
+++ b/kernel/btf/btf.h
@@ -0,0 +1,135 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2018 Facebook */
+/* Internal BTF header for kernel/btf/ */
+
+#ifndef _KERNEL_BTF_BTF_H
+#define _KERNEL_BTF_BTF_H
+
+#include <linux/types.h>
+#include <linux/btf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/module.h>
+#include <linux/rculist.h>
+
+/* Forward declarations */
+struct btf_show;
+struct btf_struct_metas;
+
+/* btf_verifier_env and its dependencies (needed by both btf.c and bpf.c) */
+enum verifier_phase {
+ CHECK_META,
+ CHECK_TYPE,
+};
+
+struct resolve_vertex {
+ const struct btf_type *t;
+ u32 type_id;
+ u16 next_member;
+};
+
+enum resolve_mode {
+ RESOLVE_TBD, /* To Be Determined */
+ RESOLVE_PTR, /* Resolving for Pointer */
+ RESOLVE_STRUCT_OR_ARRAY, /* Resolving for struct/union
+ * or array
+ */
+};
+
+#define MAX_RESOLVE_DEPTH 32
+
+struct btf_verifier_env {
+ struct btf *btf;
+ u8 *visit_states;
+ struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
+ struct bpf_verifier_log log;
+ u32 log_type_id;
+ u32 top_stack;
+ enum verifier_phase phase;
+ enum resolve_mode resolve_mode;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+struct btf_kfunc_set_tab;
+struct btf_id_dtor_kfunc_tab;
+struct btf_struct_ops_tab;
+#endif
+
+struct btf {
+ void *data;
+ struct btf_type **types;
+ u32 *resolved_ids;
+ u32 *resolved_sizes;
+ const char *strings;
+ void *nohdr_data;
+ struct btf_header hdr;
+ u32 nr_types; /* includes VOID for base BTF */
+ u32 named_start_id;
+ u32 types_size;
+ u32 data_size;
+ refcount_t refcnt;
+#ifdef CONFIG_BPF_SYSCALL
+ u32 id;
+ struct rcu_head rcu;
+ struct btf_kfunc_set_tab *kfunc_set_tab;
+ struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
+ struct btf_struct_metas *struct_meta_tab;
+ struct btf_struct_ops_tab *struct_ops_tab;
+#endif
+
+ /* split BTF support */
+ struct btf *base_btf;
+ u32 start_id; /* first type ID in this BTF (0 for base BTF) */
+ u32 start_str_off; /* first string offset (0 for base BTF) */
+ char name[MODULE_NAME_LEN];
+ bool kernel_btf;
+ __u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
+};
+
+#define BITS_PER_BYTE_MASK (BITS_PER_BYTE - 1)
+#define BITS_PER_BYTE_MASKED(bits) ((bits) & BITS_PER_BYTE_MASK)
+#define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
+#define BITS_ROUNDUP_BYTES(bits) \
+ (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
+
+/* 16MB for 64k structs and each has 16 members and
+ * a few MB spaces for the string section.
+ * The hard limit is S32_MAX.
+ */
+#define BTF_MAX_SIZE (16 * 1024 * 1024)
+
+/*
+ * The suffix of a type that indicates it cannot alias another type when
+ * comparing BTF IDs for kfunc invocations.
+ */
+#define NOCAST_ALIAS_SUFFIX "___init"
+
+/* Core BTF functions needed by bpf.c */
+const char *__btf_name_by_offset(const struct btf *btf, u32 offset);
+const struct btf_type *
+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size, const struct btf_type **elem_type,
+ u32 *elem_id, u32 *total_nelems, u32 *type_id);
+int btf_check_all_metas(struct btf_verifier_env *env);
+void btf_check_sorted(struct btf *btf);
+int btf_check_type_tags(struct btf_verifier_env *env,
+ struct btf *btf, int start_id);
+void btf_free(struct btf *btf);
+struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
+ void *data, unsigned int data_size);
+int btf_parse_hdr(struct btf_verifier_env *env);
+int btf_parse_str_sec(struct btf_verifier_env *env);
+int btf_parse_type_sec(struct btf_verifier_env *env);
+const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t);
+bool btf_type_has_size(const struct btf_type *t);
+bool btf_type_is_datasec(const struct btf_type *t);
+bool btf_type_is_decl_tag(const struct btf_type *t);
+bool btf_type_is_modifier(const struct btf_type *t);
+void btf_verifier_env_free(struct btf_verifier_env *env);
+
+extern const char * const btf_kind_str[NR_BTF_KINDS];
+
+/* Weak symbols - overridden by bpf.c when CONFIG_BPF_SYSCALL=y */
+void btf_free_bpf_data(struct btf *btf);
+void btf_put_bpf(struct btf *btf);
+
+#endif /* _KERNEL_BTF_BTF_H */
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 93f356d2b3d95..0009a7ac78462 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -395,14 +395,23 @@ config DEBUG_INFO_SPLIT
to know about the .dwo files and include them.
Incompatible with older versions of ccache.
-config DEBUG_INFO_BTF
- bool "Generate BTF type information"
+config BTF
+ bool "BTF type information support"
depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
- depends on BPF_SYSCALL
depends on PAHOLE_VERSION >= 122
# pahole uses elfutils, which does not have support for Hexagon relocations
depends on !HEXAGON
+ help
+ Enable the core BTF (BPF Type Format) infrastructure. This provides
+ BTF parsing, type lookup, and display functionality used by BPF and
+ other kernel subsystems.
+
+ This option is typically selected automatically by DEBUG_INFO_BTF.
+
+config DEBUG_INFO_BTF
+ bool "Generate BTF type information"
+ select BTF
help
Generate deduplicated BTF type information from DWARF debug info.
Turning this on requires pahole v1.22 or later, which will convert
--
2.51.0
^ permalink raw reply related
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Sasha Levin @ 2026-03-24 18:51 UTC (permalink / raw)
To: Alan Maguire
Cc: Alexei Starovoitov, Andrew Morton, Masahiro Yamada,
Nathan Chancellor, Nicolas Schier, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Peter Zijlstra,
Josh Poimboeuf, Petr Mladek, Alexei Starovoitov, Jonathan Corbet,
David Gow, Kees Cook, Greg KH, Luis Chamberlain, Steven Rostedt,
Helge Deller, Randy Dunlap, Geert Uytterhoeven, Juergen Gross,
James Bottomley, Alexey Dobriyan, Vlastimil Babka,
Laurent Pinchart, Petr Pavlu, X86 ML, LKML,
Linux Kbuild mailing list, open list:DOCUMENTATION, linux-modules,
bpf
In-Reply-To: <cbeb9f50-9398-4afb-9fb7-243d2841187e@oracle.com>
On Tue, Mar 24, 2026 at 05:34:06PM +0000, Alan Maguire wrote:
>On 24/03/2026 16:00, Sasha Levin wrote:
>> On Tue, Mar 24, 2026 at 08:03:30AM -0700, Alexei Starovoitov wrote:
>>> On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>>>>
>>>> Embed DWARF-derived function parameter name and type information in the
>>>> kernel image so that oops and WARN dumps display the crashing function's
>>>> register-passed arguments with their names, types, and values.
>>>>
>>>> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
>>>> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
>>>> parameter names and human-readable type strings. The resulting tables are
>>>> stored in .rodata using the same two-phase link approach as lineinfo.
>>>>
>>>> At runtime, kallsyms_show_paraminfo() performs a binary search on the
>>>> paraminfo tables, maps parameters to x86-64 calling convention registers
>>>> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
>>>> and value from pt_regs. If a parameter value matches the page fault
>>>> address (CR2), it is highlighted with "<-- fault address".
>>>>
>>>> Integration at show_regs() means this works for both oops and WARN()
>>>> automatically, since both paths provide full pt_regs at the exception
>>>> point.
>>>>
>>>> Example output:
>>>>
>>>> Function parameters (ext4_readdir):
>>>> file (struct file *) = 0xffff888123456000
>>>> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
>>>>
>>>> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
>>>> Adds approximately 1-2 MB to the kernel image for ~58K functions.
>>>>
>>>> Assisted-by: Claude:claude-opus-4-6
>>>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>>>
>>> Nack.
>>>
>>> You asked claude to reinvent pahole and BTF and it did it
>>> completely missing years of fine tuning that pahole does.
>>
>> Let's keep this on the technical side please.
>>
>>> dwarf cannot be trusted as-is. pahole converts it carefully
>>> by analyzing optimized out arguments and dropping signatures
>>
>> Fair point about pahole and optimized-out args. The problem is that BTF depends
>> on BPF_SYSCALL, and the environments I care about can't enable either.
>> Automotive, robotics, and safety configs all have DWARF and KALLSYMS but no
>> path to BTF.
>>
>
>Curious what the blockers are to BTF adoption? Hopefully we can tackle some
>of these or get them on a roadmap at least. I know some embedded folks want
>vmlinux BTF as a module instead of directly contained in the vmlinux binary
>to minimize vmlinux size; is this the problem you run into? Are there other
>issues? Any info you could provide would be great as the aim is to make BTF
>feasible in as many environments as possible. Thanks!
So the biggest reason I'm aware of is that those systems do not want to enable
BPF, and BTF is hidden behind BPF.
Other than that:
1. Toolchain qualifications for safety uses (we'd need to get pahole safety
certified).
2. On the ecosystem side, from what I saw, BTF isn't part of most BSPs that
vendors produce.
3. I saw concerns in the past about interactions with PREEMPT_RT, but I'm not
sure if it's still a thing.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Sasha Levin @ 2026-03-24 18:44 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <CAADnVQLr5Sx+vG6D4Jxm8r2vPxu7ygFz60LGwmqfkc=VB0-Miw@mail.gmail.com>
On Tue, Mar 24, 2026 at 09:04:03AM -0700, Alexei Starovoitov wrote:
>On Tue, Mar 24, 2026 at 9:00 AM Sasha Levin <sashal@kernel.org> wrote:
>>
>> On Tue, Mar 24, 2026 at 08:03:30AM -0700, Alexei Starovoitov wrote:
>> >On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>> >>
>> >> Embed DWARF-derived function parameter name and type information in the
>> >> kernel image so that oops and WARN dumps display the crashing function's
>> >> register-passed arguments with their names, types, and values.
>> >>
>> >> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
>> >> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
>> >> parameter names and human-readable type strings. The resulting tables are
>> >> stored in .rodata using the same two-phase link approach as lineinfo.
>> >>
>> >> At runtime, kallsyms_show_paraminfo() performs a binary search on the
>> >> paraminfo tables, maps parameters to x86-64 calling convention registers
>> >> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
>> >> and value from pt_regs. If a parameter value matches the page fault
>> >> address (CR2), it is highlighted with "<-- fault address".
>> >>
>> >> Integration at show_regs() means this works for both oops and WARN()
>> >> automatically, since both paths provide full pt_regs at the exception
>> >> point.
>> >>
>> >> Example output:
>> >>
>> >> Function parameters (ext4_readdir):
>> >> file (struct file *) = 0xffff888123456000
>> >> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
>> >>
>> >> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
>> >> Adds approximately 1-2 MB to the kernel image for ~58K functions.
>> >>
>> >> Assisted-by: Claude:claude-opus-4-6
>> >> Signed-off-by: Sasha Levin <sashal@kernel.org>
>> >
>> >Nack.
>> >
>> >You asked claude to reinvent pahole and BTF and it did it
>> >completely missing years of fine tuning that pahole does.
>>
>> Let's keep this on the technical side please.
>>
>> >dwarf cannot be trusted as-is. pahole converts it carefully
>> >by analyzing optimized out arguments and dropping signatures
>>
>> Fair point about pahole and optimized-out args. The problem is that BTF depends
>> on BPF_SYSCALL, and the environments I care about can't enable either.
>
>This is trivially fixable without reinventing pahole.
Hmm...
Looking at the code, I'd need to:
- Split BTF parsing from kernel/bpf/btf.c to somewhere outside of kernel/bpf/.
- New init path for btf_vmlinux outside BPF verifier.
- Refactor btf_parse_vmlinux() BPF-specific bits.
- Remove BPF_SYSCALL dependency from DEBUG_INFO_BTF.
- Somehow make BTF work with DEBUG_INFO_REDUCED.
I suppose that the first 4 are straightforward, but I don't have an idea about
DEBUG_INFO_REDUCED. Though we can probably tackle it later.
Does that make sense? Did I miss anything?
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Alan Maguire @ 2026-03-24 17:34 UTC (permalink / raw)
To: Sasha Levin, Alexei Starovoitov
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <acK1M_CvbYCtq7im@laps>
On 24/03/2026 16:00, Sasha Levin wrote:
> On Tue, Mar 24, 2026 at 08:03:30AM -0700, Alexei Starovoitov wrote:
>> On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>>>
>>> Embed DWARF-derived function parameter name and type information in the
>>> kernel image so that oops and WARN dumps display the crashing function's
>>> register-passed arguments with their names, types, and values.
>>>
>>> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
>>> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
>>> parameter names and human-readable type strings. The resulting tables are
>>> stored in .rodata using the same two-phase link approach as lineinfo.
>>>
>>> At runtime, kallsyms_show_paraminfo() performs a binary search on the
>>> paraminfo tables, maps parameters to x86-64 calling convention registers
>>> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
>>> and value from pt_regs. If a parameter value matches the page fault
>>> address (CR2), it is highlighted with "<-- fault address".
>>>
>>> Integration at show_regs() means this works for both oops and WARN()
>>> automatically, since both paths provide full pt_regs at the exception
>>> point.
>>>
>>> Example output:
>>>
>>> Function parameters (ext4_readdir):
>>> file (struct file *) = 0xffff888123456000
>>> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
>>>
>>> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
>>> Adds approximately 1-2 MB to the kernel image for ~58K functions.
>>>
>>> Assisted-by: Claude:claude-opus-4-6
>>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>>
>> Nack.
>>
>> You asked claude to reinvent pahole and BTF and it did it
>> completely missing years of fine tuning that pahole does.
>
> Let's keep this on the technical side please.
>
>> dwarf cannot be trusted as-is. pahole converts it carefully
>> by analyzing optimized out arguments and dropping signatures
>
> Fair point about pahole and optimized-out args. The problem is that BTF depends
> on BPF_SYSCALL, and the environments I care about can't enable either.
> Automotive, robotics, and safety configs all have DWARF and KALLSYMS but no
> path to BTF.
>
Curious what the blockers are to BTF adoption? Hopefully we can tackle some
of these or get them on a roadmap at least. I know some embedded folks want
vmlinux BTF as a module instead of directly contained in the vmlinux binary
to minimize vmlinux size; is this the problem you run into? Are there other
issues? Any info you could provide would be great as the aim is to make BTF
feasible in as many environments as possible. Thanks!
Alan
^ permalink raw reply
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Alexei Starovoitov @ 2026-03-24 16:04 UTC (permalink / raw)
To: Sasha Levin
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <acK1M_CvbYCtq7im@laps>
On Tue, Mar 24, 2026 at 9:00 AM Sasha Levin <sashal@kernel.org> wrote:
>
> On Tue, Mar 24, 2026 at 08:03:30AM -0700, Alexei Starovoitov wrote:
> >On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
> >>
> >> Embed DWARF-derived function parameter name and type information in the
> >> kernel image so that oops and WARN dumps display the crashing function's
> >> register-passed arguments with their names, types, and values.
> >>
> >> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
> >> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
> >> parameter names and human-readable type strings. The resulting tables are
> >> stored in .rodata using the same two-phase link approach as lineinfo.
> >>
> >> At runtime, kallsyms_show_paraminfo() performs a binary search on the
> >> paraminfo tables, maps parameters to x86-64 calling convention registers
> >> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
> >> and value from pt_regs. If a parameter value matches the page fault
> >> address (CR2), it is highlighted with "<-- fault address".
> >>
> >> Integration at show_regs() means this works for both oops and WARN()
> >> automatically, since both paths provide full pt_regs at the exception
> >> point.
> >>
> >> Example output:
> >>
> >> Function parameters (ext4_readdir):
> >> file (struct file *) = 0xffff888123456000
> >> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
> >>
> >> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
> >> Adds approximately 1-2 MB to the kernel image for ~58K functions.
> >>
> >> Assisted-by: Claude:claude-opus-4-6
> >> Signed-off-by: Sasha Levin <sashal@kernel.org>
> >
> >Nack.
> >
> >You asked claude to reinvent pahole and BTF and it did it
> >completely missing years of fine tuning that pahole does.
>
> Let's keep this on the technical side please.
>
> >dwarf cannot be trusted as-is. pahole converts it carefully
> >by analyzing optimized out arguments and dropping signatures
>
> Fair point about pahole and optimized-out args. The problem is that BTF depends
> on BPF_SYSCALL, and the environments I care about can't enable either.
This is trivially fixable without reinventing pahole.
^ permalink raw reply
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Sasha Levin @ 2026-03-24 16:00 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <CAADnVQJjJwRtUQNZAhLoXF7DYprhU97xJReZg9izV7n3f7=uJQ@mail.gmail.com>
On Tue, Mar 24, 2026 at 08:03:30AM -0700, Alexei Starovoitov wrote:
>On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>>
>> Embed DWARF-derived function parameter name and type information in the
>> kernel image so that oops and WARN dumps display the crashing function's
>> register-passed arguments with their names, types, and values.
>>
>> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
>> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
>> parameter names and human-readable type strings. The resulting tables are
>> stored in .rodata using the same two-phase link approach as lineinfo.
>>
>> At runtime, kallsyms_show_paraminfo() performs a binary search on the
>> paraminfo tables, maps parameters to x86-64 calling convention registers
>> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
>> and value from pt_regs. If a parameter value matches the page fault
>> address (CR2), it is highlighted with "<-- fault address".
>>
>> Integration at show_regs() means this works for both oops and WARN()
>> automatically, since both paths provide full pt_regs at the exception
>> point.
>>
>> Example output:
>>
>> Function parameters (ext4_readdir):
>> file (struct file *) = 0xffff888123456000
>> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
>>
>> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
>> Adds approximately 1-2 MB to the kernel image for ~58K functions.
>>
>> Assisted-by: Claude:claude-opus-4-6
>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>
>Nack.
>
>You asked claude to reinvent pahole and BTF and it did it
>completely missing years of fine tuning that pahole does.
Let's keep this on the technical side please.
>dwarf cannot be trusted as-is. pahole converts it carefully
>by analyzing optimized out arguments and dropping signatures
Fair point about pahole and optimized-out args. The problem is that BTF depends
on BPF_SYSCALL, and the environments I care about can't enable either.
Automotive, robotics, and safety configs all have DWARF and KALLSYMS but no
path to BTF.
>from BTF that are not accurate. This work is still ongoing.
>For example see this set:
>https://lore.kernel.org/bpf/20260320190917.1970524-1-yonghong.song@linux.dev/
Ack. I wasn't familiar with this, and looks like it makes the 2nd patch in this
series unnecessary.
>pahole isn't perfect, but what you attempted to do here
>is just broken.
I hear you that raw DWARF isn't perfect with optimized code, but I'd rather
show best-effort info than nothing. Happy to mark it as such in the output.
Open to suggestions on improving accuracy without the BTF dependency.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH] module/kallsyms: sort function symbols and use binary search
From: Petr Pavlu @ 2026-03-24 16:00 UTC (permalink / raw)
To: Stanislaw Gruszka
Cc: linux-modules, Sami Tolvanen, Luis Chamberlain, linux-kernel,
linux-trace-kernel, live-patching, Daniel Gomez, Aaron Tomlin,
Steven Rostedt, Masami Hiramatsu, Jordan Rome, Viktor Malik
In-Reply-To: <20260324125304.GA15972@wp.pl>
On 3/24/26 1:53 PM, Stanislaw Gruszka wrote:
> Hi,
>
> On Mon, Mar 23, 2026 at 02:06:43PM +0100, Petr Pavlu wrote:
>> On 3/17/26 12:04 PM, Stanislaw Gruszka wrote:
>>> Module symbol lookup via find_kallsyms_symbol() performs a linear scan
>>> over the entire symtab when resolving an address. The number of symbols
>>> in module symtabs has grown over the years, largely due to additional
>>> metadata in non-standard sections, making this lookup very slow.
>>>
>>> Improve this by separating function symbols during module load, placing
>>> them at the beginning of the symtab, sorting them by address, and using
>>> binary search when resolving addresses in module text.
>>
>> Doesn't considering only function symbols break the expected behavior
>> with CONFIG_KALLSYMS_ALL=y. For instance, when using kdb, is it still
>> able to see all symbols in a module? The module loader should be remain
>> consistent with the main kallsyms code regarding which symbols can be
>> looked up.
>
> We already have a CONFIG_KALLSYMS_ALL=y inconsistency between kernel and
> module symbol lookup, independent of this patch. find_kallsyms_symbol()
> restricts the search to MOD_TEXT (or MOD_INIT_TEXT) address ranges, so
> it cannot resolve data or rodata symbols.
My understanding is that find_kallsyms_symbol() can identify all symbols
in a module by their addresses. However, the issue I see with
MOD_TEXT/MOD_INIT_TEXT is that the function may incorrectly calculate
the size of symbols that are not within these ranges, which is a bug
that should be fixed.
A test using kdb confirms that non-text symbols can be found by their
addresses. The following shows the current behavior with 7.0-rc5 when
printing a module parameter in mlx4_en:
[1]kdb> mds __param_arr_num_vfs
0xffffffffc1209f20 0000000100000003 ........
0xffffffffc1209f28 ffffffffc0fbf07c [mlx4_core]num_vfs_argc
0xffffffffc1209f30 ffffffff8844bba0 param_ops_byte
0xffffffffc1209f38 ffffffffc0fbf080 [mlx4_core]num_vfs
0xffffffffc1209f40 000000785f69736d msi_x...
0xffffffffc1209f48 656c5f6775626564 debug_le
0xffffffffc1209f50 00000000006c6576 vel.....
0xffffffffc1209f58 0000000000000000 ........
.. and the behavior with the proposed patch:
[1]kdb> mds __param_arr_num_vfs
0xffffffffc1077f20 0000000100000003 ........
0xffffffffc1077f28 ffffffffc104707c |p......
0xffffffffc1077f30 ffffffffb4a4bba0 param_ops_byte
0xffffffffc1077f38 ffffffffc1047080 .p......
0xffffffffc1077f40 000000785f69736d msi_x...
0xffffffffc1077f48 656c5f6775626564 debug_le
0xffffffffc1077f50 00000000006c6576 vel.....
0xffffffffc1077f58 0000000000000000 ........
--
Thanks,
Petr
^ permalink raw reply
* Re: [PATCH 2/2] kallsyms: add BTF-based deep parameter rendering in oops dumps
From: Alexei Starovoitov @ 2026-03-24 15:07 UTC (permalink / raw)
To: Sasha Levin
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <20260323164858.1939248-3-sashal@kernel.org>
On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>
> +static const char *extract_struct_name(const char *type_str, bool *is_union,
> + char *name_buf, size_t bufsz)
> +{
> + const char *p, *end;
> +
> + *is_union = false;
> +
> + /* Must end with " *" to be a pointer */
> + end = type_str + strlen(type_str);
> + if (end - type_str < 3 || end[-1] != '*' || end[-2] != ' ')
> + return NULL;
> +
> + if (!strncmp(type_str, "struct ", 7)) {
> + p = type_str + 7;
> + } else if (!strncmp(type_str, "union ", 6)) {
> + p = type_str + 6;
> + *is_union = true;
> + } else {
> + return NULL;
> + }
> +
> + /* Copy name up to the " *" */
> + {
> + size_t len = (end - 2) - p;
> +
> + if (len == 0 || len >= bufsz)
> + return NULL;
> + memcpy(name_buf, p, len);
> + name_buf[len] = '\0';
> + }
> +
> + return name_buf;
Nack. This is just awful.
You didn't even bother to reformat what claude spat out.
Which means you didn't think it through.
It prints something that looks plausible
which is the opposite of what kernel crash dump should be.
crash output should be accurate and not a guess work.
^ permalink raw reply
* Re: [PATCH 1/2] kallsyms: show function parameter info in oops/WARN dumps
From: Alexei Starovoitov @ 2026-03-24 15:03 UTC (permalink / raw)
To: Sasha Levin
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
X86 ML, LKML, Linux Kbuild mailing list, open list:DOCUMENTATION,
linux-modules, bpf
In-Reply-To: <20260323164858.1939248-2-sashal@kernel.org>
On Mon, Mar 23, 2026 at 9:49 AM Sasha Levin <sashal@kernel.org> wrote:
>
> Embed DWARF-derived function parameter name and type information in the
> kernel image so that oops and WARN dumps display the crashing function's
> register-passed arguments with their names, types, and values.
>
> A new build-time tool (scripts/gen_paraminfo.c) parses DW_TAG_subprogram
> and DW_TAG_formal_parameter entries from DWARF .debug_info, extracting
> parameter names and human-readable type strings. The resulting tables are
> stored in .rodata using the same two-phase link approach as lineinfo.
>
> At runtime, kallsyms_show_paraminfo() performs a binary search on the
> paraminfo tables, maps parameters to x86-64 calling convention registers
> (RDI, RSI, RDX, RCX, R8, R9), and prints each parameter's name, type,
> and value from pt_regs. If a parameter value matches the page fault
> address (CR2), it is highlighted with "<-- fault address".
>
> Integration at show_regs() means this works for both oops and WARN()
> automatically, since both paths provide full pt_regs at the exception
> point.
>
> Example output:
>
> Function parameters (ext4_readdir):
> file (struct file *) = 0xffff888123456000
> ctx (struct dir_context *) = 0x0000000000001234 <-- fault address
>
> Gated behind CONFIG_KALLSYMS_PARAMINFO (depends on CONFIG_KALLSYMS_LINEINFO).
> Adds approximately 1-2 MB to the kernel image for ~58K functions.
>
> Assisted-by: Claude:claude-opus-4-6
> Signed-off-by: Sasha Levin <sashal@kernel.org>
Nack.
You asked claude to reinvent pahole and BTF and it did it
completely missing years of fine tuning that pahole does.
dwarf cannot be trusted as-is. pahole converts it carefully
by analyzing optimized out arguments and dropping signatures
from BTF that are not accurate. This work is still ongoing.
For example see this set:
https://lore.kernel.org/bpf/20260320190917.1970524-1-yonghong.song@linux.dev/
pahole isn't perfect, but what you attempted to do here
is just broken.
^ permalink raw reply
* Re: [PATCH] module/kallsyms: sort function symbols and use binary search
From: Stanislaw Gruszka @ 2026-03-24 12:53 UTC (permalink / raw)
To: Petr Pavlu
Cc: linux-modules, Sami Tolvanen, Luis Chamberlain, linux-kernel,
linux-trace-kernel, live-patching, Daniel Gomez, Aaron Tomlin,
Steven Rostedt, Masami Hiramatsu, Jordan Rome, Viktor Malik
In-Reply-To: <b6030f42-b4d2-4e52-acec-76e25c0f40db@suse.com>
Hi,
On Mon, Mar 23, 2026 at 02:06:43PM +0100, Petr Pavlu wrote:
> On 3/17/26 12:04 PM, Stanislaw Gruszka wrote:
> > Module symbol lookup via find_kallsyms_symbol() performs a linear scan
> > over the entire symtab when resolving an address. The number of symbols
> > in module symtabs has grown over the years, largely due to additional
> > metadata in non-standard sections, making this lookup very slow.
> >
> > Improve this by separating function symbols during module load, placing
> > them at the beginning of the symtab, sorting them by address, and using
> > binary search when resolving addresses in module text.
>
> Doesn't considering only function symbols break the expected behavior
> with CONFIG_KALLSYMS_ALL=y. For instance, when using kdb, is it still
> able to see all symbols in a module? The module loader should be remain
> consistent with the main kallsyms code regarding which symbols can be
> looked up.
We already have a CONFIG_KALLSYMS_ALL=y inconsistency between kernel and
module symbol lookup, independent of this patch. find_kallsyms_symbol()
restricts the search to MOD_TEXT (or MOD_INIT_TEXT) address ranges, so
it cannot resolve data or rodata symbols.
This appears to be acceptable in practice, most kallsyms_lookup() users are
interested in function symbols. Users relying on CONFIG_KALLSYMS_ALL=y
seems to use name-based lookups or iterate over the full symtab. Though kdb
looks like the exception: it can resolve data symbols by address in the kernel,
but not in modules. But, I think, resolving symbols by name is more common for
kdb.
To make the behavior consistent, we could either: extend find_kallsyms_symbol()
to cover data/rodata symbols (for CONFIG_KALLSYSM_ALL), or restrict
kallsyms_lookup() to text symbols and introduce a separate API for data symbols
lookup for users that really need that. I think second option is better, as
some (maybe most) users are not interested in all symbols, even if
CONFIG_KALLSYSM_ALL is set.
However, either would require substantial rework and is outside the scope
of this patch.
Regards
Stanislaw
> > This also should improve times for linear symbol name lookups, as valid
> > function symbols are now located at the beginning of the symtab.
> >
> > The cost of sorting is small relative to module load time. In repeated
> > module load tests [1], depending on .config options, this change
> > increases load time between 2% and 4%. With cold caches, the difference
> > is not measurable, as memory access latency dominates.
> >
> > The sorting theoretically could be done in compile time, but much more
> > complicated as we would have to simulate kernel addresses resolution
> > for symbols, and then correct relocation entries. That would be risky
> > if get out of sync.
> >
> > The improvement can be observed when listing ftrace filter functions:
> >
> > root@nano:~# time cat /sys/kernel/tracing/available_filter_functions | wc -l
> > 74908
> >
> > real 0m1.315s
> > user 0m0.000s
> > sys 0m1.312s
> >
> > After:
> >
> > root@nano:~# time cat /sys/kernel/tracing/available_filter_functions | wc -l
> > 74911
> >
> > real 0m0.167s
> > user 0m0.004s
> > sys 0m0.175s
> >
> > (there are three more symbols introduced by the patch)
>
> This looks as a reasonable improvement.
>
> >
> > For livepatch modules, the symtab layout is preserved and the existing
> > linear search is used. For this case, it should be possible to keep
> > the original ELF symtab instead of copying it 1:1, but that is outside
> > the scope of this patch.
>
> Livepatch modules are already handled specially by the kallsyms module
> code so excluding them from this optimization is probably ok.
>
> However, it might be worth revisiting this exception. I believe that
> livepatch support requires the original symbol table for relocations to
> remain usable. It might make sense to investigate whether updating the
> relocation data with the adjusted symbol indexes would be sensible.
>
> --
> Thanks,
> Petr
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Sasha Levin @ 2026-03-24 11:39 UTC (permalink / raw)
To: Jiri Olsa
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
x86, linux-kernel, linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <acJR51EAjn-7EOPm@krava>
On Tue, Mar 24, 2026 at 09:57:11AM +0100, Jiri Olsa wrote:
>On Mon, Mar 23, 2026 at 12:48:55PM -0400, Sasha Levin wrote:
>> Building on the lineinfo series, this adds typed function parameter
>
>hi,
>could you please specify the exact tree/commit and point to the
>series this patchset is based on?
Yup, sorry, I should have noted that. This series is based on Andrew Morton's
mm-nonmm-unstable branch (git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
mm-nonmm-unstable) since he took the base lineinfo patches and these build on
code adjacent to it.
There's no direct dependency between the two, but they touch the same parts of
the code.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Jiri Olsa @ 2026-03-24 8:57 UTC (permalink / raw)
To: Sasha Levin
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
x86, linux-kernel, linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <20260323164858.1939248-1-sashal@kernel.org>
On Mon, Mar 23, 2026 at 12:48:55PM -0400, Sasha Levin wrote:
> Building on the lineinfo series, this adds typed function parameter
hi,
could you please specify the exact tree/commit and point to the
series this patchset is based on?
thanks,
jirka
> display to oops and WARN dumps. A build-time tool extracts parameter
> names and types from DWARF, and the kernel maps pt_regs to the calling
> convention at crash time. When BTF is available, struct pointer
> parameters are dereferenced and their members displayed.
>
> Example output from a WARN in a function receiving struct new_utsname *
> (kernel version info) and struct file * parameters:
>
> ------------[ cut here ]------------
> WARNING: drivers/tty/sysrq.c:1209 at demo_crash+0xf/0x20 (drivers/tty/sysrq.c:1209)
> CPU: 2 UID: 0 PID: 323 Comm: bash
> RIP: 0010:demo_crash+0xf/0x20 (drivers/tty/sysrq.c:1209)
> ...
> RDI: ffffffffb8ca8d00
> RSI: ffffa0a3c250acc0
> ...
> Function parameters (paraminfo_demo_crash):
> uts (struct new_utsname *) = 0xffffffffb8ca8d00
> .sysname = "Linux" .nodename = "localhost"
> .release = "7.0.0-rc2-00006-g3190..." .version = "#45 SMP PRE"
> file (struct file * ) = 0xffffa0a3c250acc0
> .f_mode = (fmode_t)67993630 .f_op = (struct file_operations *)0xffffffffb7237620
> .f_flags = (unsigned int)32769 .f_cred = (struct cred *)0xffffa0a3c2e06a80
> .dentry = (struct dentry *)0xffffa0a3c0978cc0
> .prev_pos = (loff_t)-1
> Call Trace:
> <TASK>
> write_sysrq_trigger+0x96/0xb0 (drivers/tty/sysrq.c:1222)
> proc_reg_write+0x54/0xa0 (fs/proc/inode.c:330)
> vfs_write+0xc9/0x480 (fs/read_write.c:686)
> ksys_write+0x6e/0xe0 (fs/read_write.c:738)
> do_syscall_64+0xe2/0x570 (arch/x86/entry/syscall_64.c:62)
> entry_SYSCALL_64_after_hwframe+0x77/0x7f (arch/x86/entry/entry_64.S:121)
>
> Patch 1 adds the core paraminfo infrastructure (DWARF extraction,
> kernel-side lookup, register-to-parameter mapping, ~1-2 MB overhead).
> Patch 2 adds optional BTF-based struct rendering, gated behind
> CONFIG_KALLSYMS_PARAMINFO_BTF.
>
> Sasha Levin (2):
> kallsyms: show function parameter info in oops/WARN dumps
> kallsyms: add BTF-based deep parameter rendering in oops dumps
>
> .../admin-guide/kallsyms-lineinfo.rst | 31 +
> arch/x86/kernel/dumpstack.c | 6 +-
> include/linux/kallsyms.h | 9 +
> init/Kconfig | 40 ++
> kernel/Makefile | 1 +
> kernel/kallsyms.c | 182 ++++++
> kernel/kallsyms_internal.h | 6 +
> kernel/kallsyms_paraminfo_btf.c | 199 ++++++
> lib/Kconfig.debug | 11 +
> lib/tests/Makefile | 3 +
> lib/tests/paraminfo_kunit.c | 249 ++++++++
> scripts/Makefile | 3 +
> scripts/empty_paraminfo.S | 18 +
> scripts/gen_paraminfo.c | 597 ++++++++++++++++++
> scripts/link-vmlinux.sh | 44 +-
> 15 files changed, 1393 insertions(+), 6 deletions(-)
> create mode 100644 kernel/kallsyms_paraminfo_btf.c
> create mode 100644 lib/tests/paraminfo_kunit.c
> create mode 100644 scripts/empty_paraminfo.S
> create mode 100644 scripts/gen_paraminfo.c
>
> --
> 2.51.0
>
>
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Sasha Levin @ 2026-03-23 23:08 UTC (permalink / raw)
To: Andrew Morton
Cc: Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
x86, linux-kernel, linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <20260323155057.29b8e17d10421962d5ed798d@linux-foundation.org>
On Mon, Mar 23, 2026 at 03:50:57PM -0700, Andrew Morton wrote:
>On Mon, 23 Mar 2026 12:48:55 -0400 Sasha Levin <sashal@kernel.org> wrote:
>
>> Building on the lineinfo series, this adds typed function parameter
>> display to oops and WARN dumps. A build-time tool extracts parameter
>> names and types from DWARF, and the kernel maps pt_regs to the calling
>> convention at crash time. When BTF is available, struct pointer
>> parameters are dereferenced and their members displayed.
>
>mm.git is full and I'm seriously looking at loadshedded. Can we please
>leave this until next cycle, give your "kallsyms: embed source file:line
>info in kernel stack traces", v4 time to settle in?
Definitely.
If you want to ignore these, and related patches that might appear in the next
couple of weeks, I'll plan to send them to you in an organized series after
-rc1.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Sasha Levin @ 2026-03-23 22:58 UTC (permalink / raw)
To: Alexey Dobriyan
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Vlastimil Babka, Laurent Pinchart, Petr Pavlu, x86, linux-kernel,
linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <cdd61497-8d50-4fc5-aec8-47286e23d537@p183>
On Mon, Mar 23, 2026 at 09:43:59PM +0300, Alexey Dobriyan wrote:
>On Mon, Mar 23, 2026 at 12:48:55PM -0400, Sasha Levin wrote:
>> Function parameters (paraminfo_demo_crash):
>> uts (struct new_utsname *) = 0xffffffffb8ca8d00
>> .sysname = "Linux" .nodename = "localhost"
>> .release = "7.0.0-rc2-00006-g3190..." .version = "#45 SMP PRE"
>> file (struct file * ) = 0xffffa0a3c250acc0
>> .f_mode = (fmode_t)67993630 .f_op = (struct file_operations *)0xffffffffb7237620
>> .f_flags = (unsigned int)32769 .f_cred = (struct cred *)0xffffa0a3c2e06a80
>> .dentry = (struct dentry *)0xffffa0a3c0978cc0
>
>Should this be in crash's format?
>
> struct dentry ffffffffffff0000
The format currently used comes from the kernel's own BTF show infrastructure.
crash's struct dentry ffffffffffff0000 syntax is specific to crash. drgn, GDB,
bpftool, and the kernel's own BTF rendering all use the (struct type *)0xaddr
notation we're already using. Adopting crash's format would make this output
inconsistent with all other BTF-based output in the kernel, and would also lose
the member name context (.dentry).
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Andrew Morton @ 2026-03-23 22:50 UTC (permalink / raw)
To: Sasha Levin
Cc: Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Alexey Dobriyan, Vlastimil Babka, Laurent Pinchart, Petr Pavlu,
x86, linux-kernel, linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <20260323164858.1939248-1-sashal@kernel.org>
On Mon, 23 Mar 2026 12:48:55 -0400 Sasha Levin <sashal@kernel.org> wrote:
> Building on the lineinfo series, this adds typed function parameter
> display to oops and WARN dumps. A build-time tool extracts parameter
> names and types from DWARF, and the kernel maps pt_regs to the calling
> convention at crash time. When BTF is available, struct pointer
> parameters are dereferenced and their members displayed.
mm.git is full and I'm seriously looking at loadshedded. Can we please
leave this until next cycle, give your "kallsyms: embed source file:line
info in kernel stack traces", v4 time to settle in?
^ permalink raw reply
* Re: [PATCH v4 0/4] kallsyms: embed source file:line info in kernel stack traces
From: Andrew Morton @ 2026-03-23 19:25 UTC (permalink / raw)
To: Sasha Levin
Cc: Masahiro Yamada, Luis Chamberlain, Linus Torvalds,
Richard Weinberger, Juergen Gross, Geert Uytterhoeven,
James Bottomley, Jonathan Corbet, Nathan Chancellor,
Nicolas Schier, Petr Pavlu, Daniel Gomez, Greg KH, Petr Mladek,
Steven Rostedt, Kees Cook, Peter Zijlstra, Thorsten Leemhuis,
Vlastimil Babka, Helge Deller, Randy Dunlap, Laurent Pinchart,
Vivian Wang, linux-kernel, linux-kbuild, linux-modules, linux-doc
In-Reply-To: <20260322131543.971079-1-sashal@kernel.org>
On Sun, 22 Mar 2026 09:15:39 -0400 Sasha Levin <sashal@kernel.org> wrote:
> This series adds CONFIG_KALLSYMS_LINEINFO, which embeds source file:line
> information directly in the kernel image so that stack traces annotate
> every frame with the originating source location - no external tools, no
> debug symbols at runtime, and safe to use in NMI/panic context.
Thanks, I've updated mm.git's mm-nonmm-unstable branch to this version.
> Changes since v3
> =================
>
> - Remove redundant gen_lineinfo entry in scripts/Makefile for
> CONFIG_KALLSYMS_LINEINFO_MODULES (depends on CONFIG_KALLSYMS_LINEINFO
> which already builds it). (Reported by Petr Pavlu)
>
> - Use R_* constants from <elf.h> instead of hardcoded relocation type
> values in r_type_abs32(). (Reported by Petr Pavlu)
>
> - Simplify duplicated-path detection in make_relative(): replace loop
> over every '/' with a direct midpoint check, since true path
> duplication always splits at len/2. (Suggested by Petr Pavlu)
>
> - Fix comment in process_dwarf(): sections in ET_REL objects have
> sh_addr == 0 and therefore overlapping address ranges; this is
> expected behavior, not a "may" situation. (Reported by Petr Pavlu)
>
> - Use U32_MAX instead of UINT_MAX for the module raw_offset bounds
> check, matching the u32 type of the addrs array.
> (Reported by Petr Pavlu)
>
> - Document the assumption that .text is at the start of the MOD_TEXT
> segment in module_lookup_lineinfo(). A proper fix using ELF
> relocations is planned for a future series.
> (Reported by Petr Pavlu)
>
> - Wrap -fno-inline-functions-called-once in $(call cc-option,...) for
> clang compatibility. Clang does not support this GCC-specific flag;
> the noinline attribute is sufficient.
Here's how v3 altered mm.git:
kernel/module/kallsyms.c | 8 +++++--
lib/tests/Makefile | 2 -
scripts/Makefile | 1
scripts/gen_lineinfo.c | 40 +++++++++++++++++--------------------
4 files changed, 26 insertions(+), 25 deletions(-)
--- a/kernel/module/kallsyms.c~b
+++ a/kernel/module/kallsyms.c
@@ -547,13 +547,17 @@ bool module_lookup_lineinfo(struct modul
if (hdr->files_size < hdr->num_files * sizeof(u32))
return false;
- /* Compute offset from module .text base */
+ /*
+ * Compute offset from module .text base.
+ * NOTE: This assumes .text is at the start of the MOD_TEXT segment.
+ * A proper fix would use ELF relocations to reference .text directly.
+ */
text_base = (unsigned long)mod->mem[MOD_TEXT].base;
if (addr < text_base)
return false;
raw_offset = addr - text_base;
- if (raw_offset > UINT_MAX)
+ if (raw_offset > U32_MAX)
return false;
tbl.blk_addrs = base + hdr->blocks_offset;
--- a/lib/tests/Makefile~b
+++ a/lib/tests/Makefile
@@ -36,7 +36,7 @@ obj-$(CONFIG_LIVEUPDATE_TEST) += liveupd
CFLAGS_longest_symbol_kunit.o += $(call cc-disable-warning, missing-prototypes)
obj-$(CONFIG_LONGEST_SYM_KUNIT_TEST) += longest_symbol_kunit.o
-CFLAGS_lineinfo_kunit.o += -fno-inline-functions-called-once
+CFLAGS_lineinfo_kunit.o += $(call cc-option,-fno-inline-functions-called-once)
obj-$(CONFIG_LINEINFO_KUNIT_TEST) += lineinfo_kunit.o
obj-$(CONFIG_MEMCPY_KUNIT_TEST) += memcpy_kunit.o
--- a/scripts/gen_lineinfo.c~b
+++ a/scripts/gen_lineinfo.c
@@ -206,14 +206,11 @@ static const char *make_relative(const c
*/
{
size_t len = strlen(path);
+ size_t mid = len / 2;
- for (p = path; (p = strchr(p, '/')) != NULL; p++) {
- size_t prefix = p - path;
- size_t rest = len - prefix - 1;
-
- if (rest == prefix && !memcmp(path, p + 1, prefix))
- return p + 1;
- }
+ if (len > 1 && path[mid] == '/' &&
+ !memcmp(path, path + mid + 1, mid))
+ return path + mid + 1;
}
/*
@@ -340,17 +337,17 @@ static void find_text_section_range(Elf
static unsigned int r_type_abs32(unsigned int e_machine)
{
switch (e_machine) {
- case EM_X86_64: return 10; /* R_X86_64_32 */
- case EM_386: return 1; /* R_386_32 */
- case EM_AARCH64: return 258; /* R_AARCH64_ABS32 */
- case EM_ARM: return 2; /* R_ARM_ABS32 */
- case EM_RISCV: return 1; /* R_RISCV_32 */
- case EM_S390: return 4; /* R_390_32 */
- case EM_MIPS: return 2; /* R_MIPS_32 */
- case EM_PPC64: return 1; /* R_PPC64_ADDR32 */
- case EM_PPC: return 1; /* R_PPC_ADDR32 */
- case EM_LOONGARCH: return 1; /* R_LARCH_32 */
- case EM_PARISC: return 1; /* R_PARISC_DIR32 */
+ case EM_X86_64: return R_X86_64_32;
+ case EM_386: return R_386_32;
+ case EM_AARCH64: return R_AARCH64_ABS32;
+ case EM_ARM: return R_ARM_ABS32;
+ case EM_RISCV: return R_RISCV_32;
+ case EM_S390: return R_390_32;
+ case EM_MIPS: return R_MIPS_32;
+ case EM_PPC64: return R_PPC64_ADDR32;
+ case EM_PPC: return R_PPC_ADDR32;
+ case EM_LOONGARCH: return R_LARCH_32;
+ case EM_PARISC: return R_PARISC_DIR32;
default: return 0;
}
}
@@ -492,9 +489,10 @@ static void process_dwarf(Dwarf *dwarf,
/*
* In module mode, keep only .text addresses.
- * In ET_REL .ko files, .init.text/.exit.text may
- * overlap with .text address ranges, so we must
- * explicitly check against the .text bounds.
+ * In ET_REL .ko files, .text, .init.text and
+ * .exit.text all have sh_addr == 0 and therefore
+ * overlapping address ranges. Explicitly check
+ * against the .text bounds.
*/
if (module_mode && text_section_end > text_section_start &&
(addr < text_section_start || addr >= text_section_end))
--- a/scripts/Makefile~b
+++ a/scripts/Makefile
@@ -5,7 +5,6 @@
hostprogs-always-$(CONFIG_KALLSYMS) += kallsyms
hostprogs-always-$(CONFIG_KALLSYMS_LINEINFO) += gen_lineinfo
-hostprogs-always-$(CONFIG_KALLSYMS_LINEINFO_MODULES) += gen_lineinfo
hostprogs-always-$(BUILD_C_RECORDMCOUNT) += recordmcount
hostprogs-always-$(CONFIG_BUILDTIME_TABLE_SORT) += sorttable
hostprogs-always-$(CONFIG_ASN1) += asn1_compiler
_
^ permalink raw reply
* Re: [PATCH 0/2] kallsyms: show typed function parameters in oops/WARN dumps
From: Alexey Dobriyan @ 2026-03-23 18:43 UTC (permalink / raw)
To: Sasha Levin
Cc: Andrew Morton, Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra, Josh Poimboeuf, Petr Mladek,
Alexei Starovoitov, Jonathan Corbet, David Gow, Kees Cook,
Greg KH, Luis Chamberlain, Steven Rostedt, Helge Deller,
Randy Dunlap, Geert Uytterhoeven, Juergen Gross, James Bottomley,
Vlastimil Babka, Laurent Pinchart, Petr Pavlu, x86, linux-kernel,
linux-kbuild, linux-doc, linux-modules, bpf
In-Reply-To: <20260323164858.1939248-1-sashal@kernel.org>
On Mon, Mar 23, 2026 at 12:48:55PM -0400, Sasha Levin wrote:
> Function parameters (paraminfo_demo_crash):
> uts (struct new_utsname *) = 0xffffffffb8ca8d00
> .sysname = "Linux" .nodename = "localhost"
> .release = "7.0.0-rc2-00006-g3190..." .version = "#45 SMP PRE"
> file (struct file * ) = 0xffffa0a3c250acc0
> .f_mode = (fmode_t)67993630 .f_op = (struct file_operations *)0xffffffffb7237620
> .f_flags = (unsigned int)32769 .f_cred = (struct cred *)0xffffa0a3c2e06a80
> .dentry = (struct dentry *)0xffffa0a3c0978cc0
Should this be in crash's format?
struct dentry ffffffffffff0000
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox