* Re: [PATCH v8 02/10] x86/bhi: Make clear_bhb_loop() effective on newer CPUs
From: Pawan Gupta @ 2026-03-24 22:13 UTC (permalink / raw)
To: Borislav Petkov
Cc: x86, Jon Kohler, Nikolay Borisov, H. Peter Anvin, Josh Poimboeuf,
David Kaplan, Sean Christopherson, Dave Hansen, Peter Zijlstra,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, KP Singh,
Jiri Olsa, David S. Miller, David Laight, Andy Lutomirski,
Thomas Gleixner, Ingo Molnar, David Ahern, Martin KaFai Lau,
Eduard Zingerman, Song Liu, Yonghong Song, John Fastabend,
Stanislav Fomichev, Hao Luo, Paolo Bonzini, Jonathan Corbet,
linux-kernel, kvm, Asit Mallick, Tao Zhang, bpf, netdev,
linux-doc
In-Reply-To: <20260324205930.GQacL7Mp7vwGBKX1W7@fat_crate.local>
On Tue, Mar 24, 2026 at 09:59:30PM +0100, Borislav Petkov wrote:
> On Tue, Mar 24, 2026 at 11:16:51AM -0700, Pawan Gupta wrote:
> > As a mitigation for BHI, clear_bhb_loop() executes branches that overwrites
> > the Branch History Buffer (BHB). On Alder Lake and newer parts this
> > sequence is not sufficient because it doesn't clear enough entries. This
> > was not an issue because these CPUs have a hardware control (BHI_DIS_S)
> > that mitigates BHI in kernel.
> >
> > BHI variant of VMSCAPE requires isolating branch history between guests and
> > userspace. Note that there is no equivalent hardware control for userspace.
> > To effectively isolate branch history on newer CPUs, clear_bhb_loop()
> > should execute sufficient number of branches to clear a larger BHB.
> >
> > Dynamically set the loop count of clear_bhb_loop() such that it is
> > effective on newer CPUs too. Use the hardware control enumeration
> > X86_FEATURE_BHI_CTRL to select the appropriate loop count.
> >
> > Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
> > Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
> > Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> > ---
> > arch/x86/entry/entry_64.S | 21 ++++++++++++++++-----
> > arch/x86/net/bpf_jit_comp.c | 7 -------
> > 2 files changed, 16 insertions(+), 12 deletions(-)
>
> Ok, pls tell me why this below doesn't work?
>
> The additional indirection makes even the BHB loop code simpler.
>
> (I didn't pay too much attention to the labels, 2: is probably weird there).
>
> ---
>
> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
> index 3a180a36ca0e..95c7ed9afbbe 100644
> --- a/arch/x86/entry/entry_64.S
> +++ b/arch/x86/entry/entry_64.S
> @@ -1532,11 +1532,13 @@ SYM_CODE_END(rewind_stack_and_make_dead)
> * Note, callers should use a speculation barrier like LFENCE immediately after
> * a call to this function to ensure BHB is cleared before indirect branches.
> */
> -SYM_FUNC_START(clear_bhb_loop)
> +SYM_FUNC_START(__clear_bhb_loop)
> ANNOTATE_NOENDBR
> push %rbp
> + /* BPF caller may require %rax to be preserved */
> + push %rax
> mov %rsp, %rbp
> - movl $5, %ecx
> +
> ANNOTATE_INTRA_FUNCTION_CALL
> call 1f
> jmp 5f
> @@ -1557,17 +1559,17 @@ SYM_FUNC_START(clear_bhb_loop)
> * but some Clang versions (e.g. 18) don't like this.
> */
> .skip 32 - 18, 0xcc
> -2: movl $5, %eax
> +2:
> 3: jmp 4f
> nop
> -4: sub $1, %eax
> +4: sub $1, %rsi
%rsi needs to be loaded again with $inner_loop_count once per every
outer loop iteration. We probably need another register to hold that.
> jnz 3b
> - sub $1, %ecx
> + sub $1, %rdi
> jnz 1b
> .Lret2: RET
> 5:
> + pop %rax
> pop %rbp
> RET
> -SYM_FUNC_END(clear_bhb_loop)
> -EXPORT_SYMBOL_FOR_KVM(clear_bhb_loop)
> -STACK_FRAME_NON_STANDARD(clear_bhb_loop)
> +SYM_FUNC_END(__clear_bhb_loop)
> +STACK_FRAME_NON_STANDARD(__clear_bhb_loop)
> diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
> index 70b377fcbc1c..a9f406941e11 100644
> --- a/arch/x86/include/asm/nospec-branch.h
> +++ b/arch/x86/include/asm/nospec-branch.h
> @@ -390,6 +390,7 @@ extern void write_ibpb(void);
>
> #ifdef CONFIG_X86_64
> extern void clear_bhb_loop(void);
> +extern void __clear_bhb_loop(unsigned int a, unsigned int b);
> #endif
>
> extern void (*x86_return_thunk)(void);
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 83f51cab0b1e..c41b0548cf2a 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -3735,3 +3735,11 @@ void __warn_thunk(void)
> {
> WARN_ONCE(1, "Unpatched return thunk in use. This should not happen!\n");
> }
> +
> +void clear_bhb_loop(void)
> +{
> + if (cpu_feature_enabled(X86_FEATURE_BHI_CTRL))
> + __clear_bhb_loop(12, 7);
> + else
> + __clear_bhb_loop(5, 5);
> +}
This is cleaner. A few things to consider are, CLEAR_BRANCH_HISTORY that
calls clear_bhb_loop() would be calling into C code very early during the
kernel entry. The code generated here may vary based on the compiler. Any
indirect branch here would be security risk. This needs to be noinstr so
that it can't be hijacked by probes and ftraces.
At kernel entry, calling into C before mitigations are applied is risky.
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 63d6c9fa5e80..e2cceabb23e8 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -1614,11 +1614,6 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
> u8 *func;
>
> if (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) {
> - /* The clearing sequence clobbers eax and ecx. */
> - EMIT1(0x50); /* push rax */
> - EMIT1(0x51); /* push rcx */
> - ip += 2;
> -
> func = (u8 *)clear_bhb_loop;
Although call to clear_bhb_loop() will be inserted at the end of the BPF
program before it returns, I am not sure if it is safe to assume that
trashing registers in the path clear_bhb_loop() -> __clear_bhb_loop() is
okay? Especially, when we don't know what code compiler generated for
clear_bhb_loop(). BPF experts would know better?
^ permalink raw reply
* Re: [PATCH net-next v3 03/13] net: introduce ndo_set_rx_mode_async and dev_rx_mode_work
From: Stanislav Fomichev @ 2026-03-24 22:49 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Stanislav Fomichev, netdev, davem, edumazet, pabeni, horms,
corbet, skhan, andrew+netdev, michael.chan, pavan.chebbi,
anthony.l.nguyen, przemyslaw.kitszel, saeedm, tariqt, mbloch,
alexanderduyck, kernel-team, johannes, sd, jianbol, dtatulea,
mohsin.bashr, jacob.e.keller, willemb, skhawaja, bestswngs,
aleksandr.loktionov, kees, linux-doc, linux-kernel,
intel-wired-lan, linux-rdma, linux-wireless, linux-kselftest,
leon
In-Reply-To: <20260324142114.216fcb01@kernel.org>
On 03/24, Jakub Kicinski wrote:
> On Tue, 24 Mar 2026 11:13:04 -0700 Stanislav Fomichev wrote:
> > > > + netif_addr_lock_bh(dev);
> > > > +
> > > > + err = __hw_addr_list_snapshot(&uc_snap, &dev->uc,
> > > > + dev->addr_len);
> > > > + if (!err)
> > > > + err = __hw_addr_list_snapshot(&uc_ref, &dev->uc,
> > > > + dev->addr_len);
> > > > + if (!err)
> > > > + err = __hw_addr_list_snapshot(&mc_snap, &dev->mc,
> > > > + dev->addr_len);
> > > > + if (!err)
> > > > + err = __hw_addr_list_snapshot(&mc_ref, &dev->mc,
> > > > + dev->addr_len);
> > >
> > > This doesn't get slow with a few thousands of addresses?
> >
> > I can add kunit benchmark and attach the output? Although not sure where
> > to go from that. The alternative to this is allocating an array of entries.
> > I started with that initially but __hw_addr_sync_dev wants to kfree the
> > individual entries and I decided not to have a separate helpers to
> > manage the snapshots.
>
> Let's see what the benchmark says. Hopefully it's fast enough and
> we don't have to worry. Is keeping these lists around between the
> invocations of the work tricky?
Yeah, that sounds doable, don't think it's too tricky, just extra
list_head on net_device + change the alloc/free to use it.
And then we keep this cache around until unregister? I will try to add it as
a separate patch to cache these entries to keep it simple for review..
> > > Can we give the work a reference on the netdev (at init time) and
> > > cancel + release it here instead of flushing / waiting?
> >
> > Not sure why cancel+release, maybe you're thinking about the unregister
> > path? This is rtnl_unlock -> netdev_run_todo -> __rtnl_unlock + some
> > extras.
> >
> > And the flush is here to plumb the addresses to the real devices
> > before we return to the callers. Mostly because of the following
> > things we have in the tests:
> >
> > # TEST: team cleanup mode lacp [FAIL]
> > # macvlan unicast address not found on a slave
> >
> > Can you explain a bit more on the suggestion?
>
> Oh, I thought it's here for unregister! Feels like it'd be cleaner to
> add the flush in dev_*c_add() and friends? How hard would it be to
> identify the callers in atomic context?
Not sure we can do it in dev_xc_add because it runs under rtnl :-(
I currently do flush in netdev_run_todo because that's the place that
doesn't hold rtnl. Otherwise flush will get stuck because the work
handler grabs it...
^ permalink raw reply
* Re: [PATCH v2 00/16] fs,x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <cover.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
>
> This series adds support for Privilege-Level Zero Association (PLZA) to the
> resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID
> and/or RMID for execution in kernel mode (privilege level zero), so that
> kernel work is not subject to the same resource constrains as the current
> user-space task. This avoids kernel operations being aggressively throttled
> when a task's memory bandwidth is heavily limited.
>
> The feature documentation is not yet publicly available, but it is expected
> to be released in the next few weeks. In the meantime, a brief description
> of the features is provided below.
>
> Privilege Level Zero Association (PLZA)
>
> Privilege Level Zero Association (PLZA) allows the hardware to
> automatically associate execution in Privilege Level Zero (CPL=0) with a
> specific COS (Class of Service) and/or RMID (Resource Monitoring
> Identifier). The QoS feature set already has a mechanism to associate
> execution on each logical processor with an RMID or COS. PLZA allows the
> system to override this per-thread association for a thread that is
> executing with CPL=0.
> ------------------------------------------------------------------------
>
> The series introduces the feature in a way that supports the interface in
> a generic manner to accomodate MPAM or other vendor specific implimentation.
>
> Below is the detailed requirements provided by Reinette:
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
Our discussion considered how resctrl could support PLZA in a generic way while
also preparing to support MPAM's variants and how PLZA may evolve to have similar
capabilities when considering the capabilities of its registers.
This does not mean that your work needs to implement everything that was discussed.
Instead, this work is expected to just support what PLZA is capable of today but
do so in a way that the future enhancements could be added to.
This series is quite difficult to follow since it appears to implement a full
featured generic interface while PLZA cannot take advantage of it.
Could you please simplify this work to focus on just enabling PLZA and only
add interfaces needed to do so?
>
> Summary:
> 1. Kernel-mode/PLZA controls and status should be exposed under the resctrl
> info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.
>
> 2. Add two info files
>
> a. kernel_mode
> Purpose: Control how resource allocation and monitoring apply in kernel mode
> (e.g. inherit from task vs global assign).
>
> Read: List supported modes and show current one (e.g. with [brackets]).
> Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).
>
> b. kernel_mode_assignment
>
> Purpose: When a “global assign” kernel mode is active, specify which resctrl group
> (CLOSID/RMID) is used for kernel work.
>
> Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
> Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).
>
> The patches are based on top of commit (v7.0.0-rc3)
> 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
> ------------------------------------------------------------------------
>
> Examples: kernel_mode and kernel_mode_assignment
>
> All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means
> /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform
> must support the relevant modes (e.g. AMD with PLZA).
>
> 1) kernel_mode — show and set the current kernel mode
>
> Read supported modes and which one is active (current in brackets):
>
> $ cat info/kernel_mode
> [inherit_ctrl_and_mon]
> global_assign_ctrl_inherit_mon
> global_assign_ctrl_assign_mon
>
> Set the active mode (e.g. use one CLOSID+RMID for all kernel work):
>
> $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
> $ cat info/kernel_mode
> inherit_ctrl_and_mon
> global_assign_ctrl_inherit_mon
> [global_assign_ctrl_assign_mon]
>
> Mode meanings:
> - inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
> - global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
> - global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.
>
> 2) kernel_mode_assignment — show and set which group is used for kernel work
>
> Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the
To help with future usages please connect visibility of this file with the mode in
info/kernel_mode. This helps us to support future modes with other resctrl files, possible
within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon",
while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.
> currently assigned group (path format is "CTRL_MON/MON/"):
The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon"
then it should only contain a control group, alternatively, if the mode is
"global_assign_ctrl_assign_mon" then it contains control and mon group. This gives
resctrl future flexibility to change format for future modes.
We should also consider the scenario when it is a "monitoring only" system, which can
happen independent from what hardware actually supports, for example, if user boots
with "rdt=!l3cat,!l2cat,!mba,!smba". In this case I assume CLOS should just always be
zero and thus only "default control group" is accepted?
>
> $ cat info/kernel_mode_assignment
> //
>
> "//" means the default CTRL_MON group is assigned. Assign a specific
> group instead (e.g. a CTRL_MON group "ctrl1", or a MON group "mon1" under it):
>
> $ echo "ctrl1//" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1//
>
> $ echo "ctrl1/mon1/" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1/mon1/
>
> Clear the assignment (no dedicated group for kernel work):
>
> $ echo >> info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> Kmode is not configured
This does not look right. Would this not create a conflict between info/kernel_mode
and info/kernel_mode_assignment about what the current mode is? The way I see it
info/kernel_mode_assignment must always contain a valid group.
>
> Errors (e.g. invalid group name or unsupported mode) are reported in
> info/last_cmd_status.
>
> ---
>
> v2:
> This is similar to RFC with new proposal. Names of the some interfaces
> are not final. Lets fix that later as we move forward.
>
> Separated the two features: Global Bandwidth Enforcement (GLBE) and
> Privilege Level Zero Association (PLZA).
>
> This series only adds support for PLZA.
>
> Used the name of the feature as kmode instead of PLZA. That can be changed as well.
>
> Tony suggested using global variables to store the kernel mode
> CLOSID and RMID. However, the kernel mode CLOSID and RMID are
> coming from rdtgroup structure with the new interface. Accessing
> them requires holding the associated lock, which would make the
> context switch path unnecessarily expensive. So, dropped the idea.
> https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
> Let me know if there are other ways to optimize this.
I do not see why the context switch path needs to be touched at all with this
implementation. Since PLZA only supports global assignment does it not mean that resctrl
only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and
info/kernel_mode_assignment?
Consider some of the scenarios:
resctrl mount with default state:
# cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_inherit_mon
global_assign_ctrl_assign_mon
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
enable global_assign_ctrl_assign_mon mode:
# echo "global_assign_ctrl_assign_mon" > info/kernel_mode
Expectation here is that when user space sets this mode as above then resctrl would
in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
I do not see why it is necessary to maintain any per-CPU or per-task state or needing
to touch the context switch code. Since PLZA only supports global could it not
just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it?
Only caveat is that if a CPU is offline then this setting needs to be stashed
so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.
The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing
with RDT_RESOURCE_L3 and traverses the resource domain list and resource group
CPU mask seems unnecessary to me as well as error prone since the system may only
have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set
MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?
To continue the scenarios ...
After user's setting above related files read:
# cat info/kernel_mode
inherit_ctrl_and_mon
global_assign_ctrl_inherit_mon
[global_assign_ctrl_assign_mon]
# cat info/kernel_mode_assignment
//
Modify group used by global_assign_ctrl_assign_mon mode:
# echo 'ctrl1/mon1/' > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1>
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
Enable global_assign_ctrl_inherit_mon mode:
# echo "global_assign_ctrl_inherit_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode
inherit_ctrl_and_mon
[global_assign_ctrl_inherit_mon]
global_assign_ctrl_assign_mon
# cat info/kernel_mode_assignment <==== returns just a ctrl group
/
Modify group used by global_assign_ctrl_inherit_mon mode:
# echo ctrl1 > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode_assignment <==== returns just a ctrl group
ctrl/
Enable inherit_ctrl_and_mon mode:
# echo "inherit_ctrl_and_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=0
MSR_IA32_PQR_PLZA_ASSOC.plza_en=0
At this point info/kernel_mode_assignment is not visible anymore:
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
From what I understand above exposes and enables full capability of PLZA. All the other
per-task and per-cpu handling in this series is not something that PLZA can benefit from.
If this is not the case, what am I missing? Could this series be simplified to just support
PLZA today? When next hardware with more capability needs to be supported resctrl could be
enhanced to support it by using the more accurate information about what the hardware is
capable of.
We also do not really know what use cases users prefer. This may even be sufficient.
Reinette
^ permalink raw reply
* Re: [PATCH v2 01/16] fs/resctrl: Add kernel mode (kmode) data structures and arch hook
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <c3381fbbce357b00895252572907645a92cd9718.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Add resctrl_kmode, resctrl_kmode_cfg, kernel mode bit defines, and
> resctrl_arch_get_kmode_cfg() for resctrl kernel mode (e.g. PLZA) support.
We should not have to start every series from scratch.
Documentation/process/maintainer-tip.rst. Always.
> ---
> include/linux/resctrl.h | 10 ++++++++++
> include/linux/resctrl_types.h | 30 ++++++++++++++++++++++++++++++
> 2 files changed, 40 insertions(+)
>
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 006e57fd7ca5..2c36d1ac392f 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -699,6 +699,16 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
> */
> bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
>
> +/**
> + * resctrl_arch_get_kmode_cfg() - Get resctrl kernel mode configuration
> + * @kcfg: Filled with current kernel mode config (kmode, kmode_cur, k_rdtgrp).
> + *
> + * Used by the arch (e.g. x86) to report which kernel mode is active and,
> + * when a global assign mode is in use, which rdtgroup is assigned to
> + * kernel work.
> + */
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg);
This interface does not look right. Would it not be resctrl fs that determines
which resource group is assigned? This cannot be set by arch. Why does arch decide
which mode is active? Is this not also resctrl fs? Should arch not just tell
resctrl fs what it supports?
> +
> extern unsigned int resctrl_rmid_realloc_threshold;
> extern unsigned int resctrl_rmid_realloc_limit;
>
> diff --git a/include/linux/resctrl_types.h b/include/linux/resctrl_types.h
> index a5f56faa18d2..6b78b08eab29 100644
> --- a/include/linux/resctrl_types.h
> +++ b/include/linux/resctrl_types.h
> @@ -65,7 +65,37 @@ enum resctrl_event_id {
> QOS_NUM_EVENTS,
> };
>
> +/**
> + * struct resctrl_kmode - Resctrl kernel mode descriptor
> + * @name: Human-readable name of the kernel mode.
> + * @val: Bitmask value for the kernel mode (e.g. INHERIT_CTRL_AND_MON).
> + */
> +struct resctrl_kmode {
> + char name[32];
> + u32 val;
> +};
There is no reason why this needs to be in a central header exposed to archs. Could
this not be a static within the only function that uses it? Something like
rdt_mode_str[]?
> +
> +/**
> + * struct resctrl_kmode_cfg - Resctrl kernel mode configuration
> + * @kmode: Requested kernel mode.
> + * @kmode_cur: Currently active kernel mode.
> + * @k_rdtgrp: Resource control structure in use, or NULL otherwise.
> + */
> +struct resctrl_kmode_cfg {
> + u32 kmode;
> + u32 kmode_cur;
> + struct rdtgroup *k_rdtgrp;
> +};
> +
> #define QOS_NUM_L3_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
> #define MBM_STATE_IDX(evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
>
> +/* Resctrl kernel mode bits (e.g. for PLZA). */
> +#define INHERIT_CTRL_AND_MON BIT(0) /* Kernel uses same CLOSID/RMID as user. */
> +/* One CLOSID for all kernel work; RMID inherited from user. */
> +#define GLOBAL_ASSIGN_CTRL_INHERIT_MON BIT(1)
> +/* One resource group (CLOSID+RMID) for all kernel work. */
> +#define GLOBAL_ASSIGN_CTRL_ASSIGN_MON BIT(2)
> +#define RESCTRL_KERNEL_MODES_NUM 3
I think it will make the code much easier to understand if the different modes are described by an
enum. For example,
enum resctrl_kernel_modes {
INHERIT_CTRL_AND_MON,
GLOBAL_ASSIGN_CTRL_INHERIT_MON,
GLOBAL_ASSIGN_CTRL_ASSIGN_MON,
RESCTRL_KMODE_LAST = GLOBAL_ASSIGN_CTRL_ASSIGN_MON
};
#define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
The supported kernel modes can still be managed as a bitmap with intuitive API using the
enum that will make the code easier to read. For example, __set_bit(INHERIT_CTRL_AND_MON, ...)
or BIT(INHERIT_CTRL_AND_MON). The naming is awkward at the moment though, we should improve here.
Reinette
^ permalink raw reply
* Re: [PATCH v2 02/16] fs, x86/resctrl: Add architecture routines for kernel mode initialization
From: Reinette Chatre @ 2026-03-24 22:53 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <72315f83ddf35b0de53fb911f46623782d95f507.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Implement the resctrl kernel mode (kmode) arch initialization.
>
> - Add resctrl_arch_get_kmode_cfg() to fill the default kernel mode
> (INHERIT_CTRL_AND_MON). This can be extended later (e.g. for PLZA) to set
> additional modes.
I do not think this is something that the architecture should set, at least
at this time. Every mode has different requirements and this just lets the arch set
it without any support for what configurations it implies. For example, if
arch sets a different default mode than INHERIT_CTRL_AND_MON then PQR_PLZA_ASSOC
needs to be programmed as the CPUs come online and this does not seem to
accommodate this. This implementation appears to have significant assumptions on
what architecture will end up setting since it is only considering PLZA.
>
> - Add global resctrl_kcfg and resctrl_kmode_init() to initialize default
> values.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
> ---
> arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
> fs/resctrl/rdtgroup.c | 10 ++++++++++
> 2 files changed, 17 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 7667cf7c4e94..4c3ab2d93909 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -892,6 +892,13 @@ bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
> }
> }
>
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg)
> +{
> + kcfg->kmode = INHERIT_CTRL_AND_MON;
> + kcfg->kmode_cur = INHERIT_CTRL_AND_MON;
> + kcfg->k_rdtgrp = NULL;
> +}
I already commented on the arch vs filesystem settings.
When using an arch helper this forces all architectures to support this helper. Is a
helper required? Is it perhaps possible for arch to set a property instead? For example,
how enumeration is handled?
I think the assumption here is that INHERIT_CTRL_AND_MON is the default and expected to
be supported by all architectures. I do not see why arch should set this as default but
instead this should be from resctrl fs. At the same time it is expected that the
architecture supports this mode so there needs to be a failure if an architecture does
not support this mode?
I'm going to stop here. I think the comments so far may result in major changes already
making further detailed review of patches unnecessary.
Reinette
^ permalink raw reply
* Re: [PATCH V9 1/8] dax: move dax_pgoff_to_phys from [drivers/dax/] device.c to bus.c
From: Ira Weiny @ 2026-03-24 23:44 UTC (permalink / raw)
To: Jonathan Cameron, John Groves
Cc: John Groves, Miklos Szeredi, Dan Williams, Bernd Schubert,
Alison Schofield, John Groves, Jonathan Corbet, Shuah Khan,
Vishal Verma, Dave Jiang, Matthew Wilcox, Jan Kara,
Alexander Viro, David Hildenbrand, Christian Brauner,
Darrick J . Wong, Randy Dunlap, Jeff Layton, Amir Goldstein,
Stefan Hajnoczi, Joanne Koong, Josef Bacik, Bagas Sanjaya,
Chen Linxuan, James Morse, Fuad Tabba, Sean Christopherson,
Shivank Garg, Ackerley Tng, Gregory Price, Aravind Ramesh,
Ajay Joshi, venkataravis@micron.com, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, nvdimm@lists.linux.dev,
linux-cxl@vger.kernel.org, linux-fsdevel@vger.kernel.org,
Ira Weiny
In-Reply-To: <20260324141806.000003f7@huawei.com>
Jonathan Cameron wrote:
> On Tue, 24 Mar 2026 00:37:53 +0000
> John Groves <john@jagalactic.com> wrote:
>
> > From: John Groves <john@groves.net>
> >
> > This function will be used by both device.c and fsdev.c, but both are
> > loadable modules. Moving to bus.c puts it in core and makes it available
> > to both.
> >
> > No code changes - just relocated.
> >
> > Reviewed-by: Ira Weiny <ira.weiny@intel.com>
> > Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> > Signed-off-by: John Groves <john@groves.net>
> Obviously this is a straight forward code move... But I can't resist
> commenting on what is moving (feel free to ignore! or maybe a follow
> up patch if you agree.
>
> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Added this to the series. LMK if I missed something.
Ira
---
commit ccc1878ab00178e82108bdd1ece497388a24290b (HEAD -> nvdimm-famfs-dax)
Author: Ira Weiny <ira.weiny@intel.com>
Date: Tue Mar 24 12:36:19 2026 -0500
dax: Modernize dax_pgoff_to_phys()
The patch to move dax_pgoff_to_phys() to bus.c revealed that the
function could be improved with more modern style and the newer
in_range() utility function.
Update it while we are moving it around.
Link: https://lore.kernel.org/all/20260324141806.000003f7@huawei.com/
Suggested-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c
index e4bd5c9f006c..1b412264bb36 100644
--- a/drivers/dax/bus.c
+++ b/drivers/dax/bus.c
@@ -1421,16 +1421,12 @@ static const struct device_type dev_dax_type = {
__weak phys_addr_t dax_pgoff_to_phys(struct dev_dax *dev_dax, pgoff_t pgoff,
unsigned long size)
{
- int i;
-
- for (i = 0; i < dev_dax->nr_range; i++) {
+ for (int i = 0; i < dev_dax->nr_range; i++) {
struct dev_dax_range *dax_range = &dev_dax->ranges[i];
struct range *range = &dax_range->range;
- unsigned long long pgoff_end;
phys_addr_t phys;
- pgoff_end = dax_range->pgoff + PHYS_PFN(range_len(range)) - 1;
- if (pgoff < dax_range->pgoff || pgoff > pgoff_end)
+ if (!in_range(pgoff, dax_range->pgoff, PHYS_PFN(range_len(range))))
continue;
phys = PFN_PHYS(pgoff - dax_range->pgoff) + range->start;
if (phys + size - 1 <= range->end)
^ permalink raw reply related
* Re: [PATCH v4 2/4] workflow: process/changes.rst: expand and cleanup list
From: Manuel Ebner @ 2026-03-23 17:50 UTC (permalink / raw)
To: Jonathan Corbet, Collin Funk, Shuah Khan
Cc: workflows, linux-doc, linux-kernel
In-Reply-To: <87fr5ycxao.fsf@trenco.lwn.net>
On Tue, 2026-03-17 at 09:04 -0600, Jonathan Corbet wrote:
> Manuel Ebner <manuelebner@airmail.cc> writes:
> > remove footnote for sphinx
> > in the paragraph before the list it says:
> > "Also, not all tools are necessary on all systems;
> > obviously, if you don't have any PC Card hardware, for example,
> > you probably needn't concern yourself with pcmciautils."
i understand this paragraph like this:
most or all tools can be optional, depending on the specific use case.
e.g. quota-tools is optional for people compiling the kernel.
that's why i removed the optional markings.
and pcmciautils, the example in the the file, is not marked as optional in changes.rst:
====================== =============== ========================================
Program Minimal version Command to check the version
====================== =============== ========================================
[...]
pcmciautils 004 pccardctl -V
> > remove "(optional)" for the same reason as above
>
> Here you have lost some information - now the document doesn't say
> *which* tools are optional. That is, I think, not ideal.
that's true.
my question is: what are the tools optional for?
or what is the criteria for "optional"?
Manuel
^ permalink raw reply
* Re: [PATCH v3 1/2] docs: allow long links to wrap per character to prevent page overflow
From: Randy Dunlap @ 2026-03-25 0:50 UTC (permalink / raw)
To: Rito Rhymes, Jonathan Corbet, linux-doc; +Cc: Shuah Khan, linux-kernel
In-Reply-To: <20260323152428.30483-1-rito@ritovision.com>
On 3/23/26 8:24 AM, Rito Rhymes wrote:
> Some documentation pages contain long link text without natural
> break points, which can force page-wide horizontal scroll overflow
> on small screens.
>
> Use overflow-wrap: anywhere for anchor text in the docs stylesheet so
> links can wrap per character as a fallback when normal word boundaries
> are unavailable.
>
> Examples:
> https://docs.kernel.org/6.15/firmware-guide/acpi/non-d0-probe.html
> https://docs.kernel.org/6.15/arch/x86/earlyprintk.html
>
> Signed-off-by: Rito Rhymes <rito@ritovision.com>
> Assisted-by: Codex:GPT-5.4
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
thanks.
> ---
> v3: add latest public versioned URL examples to the patchlog
>
> Documentation/sphinx-static/custom.css | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/Documentation/sphinx-static/custom.css b/Documentation/sphinx-static/custom.css
> index db24f4344..4ec617d40 100644
> --- a/Documentation/sphinx-static/custom.css
> +++ b/Documentation/sphinx-static/custom.css
> @@ -149,6 +149,11 @@ div.language-selection ul li:hover {
> background: #dddddd;
> }
>
> +/* Let long link text wrap instead of forcing overflow. */
> +a {
> + overflow-wrap: anywhere;
> +}
> +
> /* Make xrefs more universally visible */
> a.reference, a.reference:hover {
> border-bottom: none;
--
~Randy
^ permalink raw reply
* Re: [PATCH v3 2/2] docs: allow long table reference links to wrap and prevent overflow
From: Randy Dunlap @ 2026-03-25 0:50 UTC (permalink / raw)
To: Rito Rhymes, Jonathan Corbet, linux-doc; +Cc: Shuah Khan, linux-kernel
In-Reply-To: <20260323152428.30483-2-rito@ritovision.com>
On 3/23/26 8:24 AM, Rito Rhymes wrote:
> Some documentation pages contain docutils tables with reference links
> that use long unbroken strings. Those strings can expand the table
> width beyond the content column and cause page-wide horizontal
> overflow.
>
> Allow reference links in docutils tables in the main document body to
> wrap when needed so the table stays within the content column and does
> not break page layout.
>
> Examples:
> https://docs.kernel.org/6.15/arch/openrisc/openrisc_port.html
> https://docs.kernel.org/6.15/filesystems/ext2.html
>
> Signed-off-by: Rito Rhymes <rito@ritovision.com>
> Assisted-by: Codex:GPT-5.4
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
thanks.
> ---
> v3: add latest public versioned URL examples to the patchlog
>
> Documentation/sphinx-static/custom.css | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/Documentation/sphinx-static/custom.css b/Documentation/sphinx-static/custom.css
> index 4ec617d40..b41c54c71 100644
> --- a/Documentation/sphinx-static/custom.css
> +++ b/Documentation/sphinx-static/custom.css
> @@ -154,6 +154,11 @@ a {
> overflow-wrap: anywhere;
> }
>
> +/* Let rendered reference links in tables wrap when needed. */
> +div.body table.docutils a.reference {
> + overflow-wrap: anywhere;
> +}
> +
> /* Make xrefs more universally visible */
> a.reference, a.reference:hover {
> border-bottom: none;
--
~Randy
^ permalink raw reply
* [RFC] btf: split core BTF parsing out of BPF subsystem into kernel/btf/
From: Sasha Levin @ 2026-03-25 1:18 UTC (permalink / raw)
To: sashal
Cc: James.Bottomley, adobriyan, akpm, alexei.starovoitov, ast, bp,
bpf, corbet, dave.hansen, davidgow, deller, geert, gregkh, hpa,
jgross, jpoimboe, kees, laurent.pinchart, linux-doc, linux-kbuild,
linux-kernel, linux-modules, masahiroy, mcgrof, mingo, nathan,
nsc, peterz, petr.pavlu, pmladek, rdunlap, rostedt, tglx, vbabka,
x86
In-Reply-To: <acLbnMAPrHCpoIpr@laps>
Move BTF type format parsing and inspection code out of the BPF
subsystem into its own kernel/btf/ directory, separating core BTF
functionality from BPF-specific extensions.
CONFIG_DEBUG_INFO_BTF currently depends on CONFIG_BPF_SYSCALL, which
prevents embedded, automotive, and safety-critical environments from
using BTF. These platforms often disable BPF for security and size
reasons but would benefit from BTF type information for crash
diagnostics and debugging.
Introduce CONFIG_BTF for the core BTF runtime support (type parsing,
lookup, and display) which can be enabled independently of
CONFIG_BPF_SYSCALL, and remove the BPF_SYSCALL dependency from
CONFIG_DEBUG_INFO_BTF. The code is split into:
kernel/btf/btf.c - Core BTF: parsing, verification, type lookup,
display infrastructure. No BPF dependencies.
kernel/btf/bpf.c - BPF extensions: kfunc registration, struct_ops,
verifier integration, FD interface, module BTF.
kernel/btf/btf.h - Internal shared header.
When CONFIG_BPF_SYSCALL=n, only btf.c is compiled, providing the
minimal BTF API. When CONFIG_BPF_SYSCALL=y, both files are compiled
and bpf.c overrides weak symbols in btf.c for BPF-specific cleanup
(IDR management, RCU-deferred freeing, kfunc/struct_ops tab cleanup).
No functional change when CONFIG_BPF_SYSCALL=y.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/btf.h | 80 +-
kernel/Makefile | 1 +
kernel/bpf/Makefile | 2 +-
kernel/btf/Makefile | 4 +
kernel/{bpf/btf.c => btf/bpf.c} | 6386 ++++---------------------------
kernel/btf/btf.c | 4803 +++++++++++++++++++++++
kernel/btf/btf.h | 135 +
lib/Kconfig.debug | 15 +-
8 files changed, 5796 insertions(+), 5630 deletions(-)
create mode 100644 kernel/btf/Makefile
rename kernel/{bpf/btf.c => btf/bpf.c} (50%)
create mode 100644 kernel/btf/btf.c
create mode 100644 kernel/btf/btf.h
diff --git a/include/linux/btf.h b/include/linux/btf.h
index 48108471c5b13..d4d64878324ff 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -139,17 +139,11 @@ struct btf_struct_metas {
struct btf_struct_meta types[];
};
-extern const struct file_operations btf_fops;
-
const char *btf_get_name(const struct btf *btf);
void btf_get(struct btf *btf);
void btf_put(struct btf *btf);
const struct btf_header *btf_header(const struct btf *btf);
-int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
-struct btf *btf_get_by_fd(int fd);
-int btf_get_info_by_fd(const struct btf *btf,
- const union bpf_attr *attr,
- union bpf_attr __user *uattr);
+
/* Figure out the size of a type_id. If type_id is a modifier
* (e.g. const), it will be resolved to find out the type with size.
*
@@ -213,12 +207,9 @@ int btf_type_seq_show_flags(const struct btf *btf, u32 type_id, void *obj,
int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
char *buf, int len, u64 flags);
-int btf_get_fd_by_id(u32 id);
-u32 btf_obj_id(const struct btf *btf);
bool btf_is_kernel(const struct btf *btf);
bool btf_is_module(const struct btf *btf);
bool btf_is_vmlinux(const struct btf *btf);
-struct module *btf_try_get_module(const struct btf *btf);
u32 btf_nr_types(const struct btf *btf);
u32 btf_named_start_id(const struct btf *btf, bool own);
struct btf *btf_base_btf(const struct btf *btf);
@@ -228,12 +219,8 @@ bool btf_type_is_primitive(const struct btf_type *t);
bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
const struct btf_member *m,
u32 expected_offset, u32 expected_size);
-struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, u32 value_size);
-int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
bool btf_type_is_void(const struct btf_type *t);
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
-s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
u32 id, u32 *res_id);
const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
@@ -245,6 +232,20 @@ btf_resolve_size(const struct btf *btf, const struct btf_type *type,
u32 *type_size);
const char *btf_type_str(const struct btf_type *t);
+/* BPF-specific declarations */
+extern const struct file_operations btf_fops;
+int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
+struct btf *btf_get_by_fd(int fd);
+int btf_get_info_by_fd(const struct btf *btf,
+ const union bpf_attr *attr,
+ union bpf_attr __user *uattr);
+int btf_get_fd_by_id(u32 id);
+u32 btf_obj_id(const struct btf *btf);
+struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, u32 value_size);
+int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
+s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
+
#define for_each_member(i, struct_type, member) \
for (i = 0, member = btf_type_member(struct_type); \
i < btf_type_vlen(struct_type); \
@@ -524,9 +525,6 @@ static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
bool btf_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix);
-int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
- u32 arg_no);
-u32 btf_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto, int off);
struct bpf_verifier_log;
@@ -564,16 +562,39 @@ struct btf_field_iter {
int vlen;
};
-#ifdef CONFIG_BPF_SYSCALL
+#ifdef CONFIG_BTF
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
+const char *btf_name_by_offset(const struct btf *btf, u32 offset);
+const char *btf_str_by_offset(const struct btf *btf, u32 offset);
+#else
+static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
+ u32 type_id)
+{
+ return NULL;
+}
+
+static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
+{
+}
+
+static inline const char *btf_name_by_offset(const struct btf *btf,
+ u32 offset)
+{
+ return NULL;
+}
+#endif /* CONFIG_BTF */
+
+#ifdef CONFIG_BPF_SYSCALL
+struct module *btf_try_get_module(const struct btf *btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **map_ids);
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind);
__u32 *btf_field_iter_next(struct btf_field_iter *it);
-const char *btf_name_by_offset(const struct btf *btf, u32 offset);
-const char *btf_str_by_offset(const struct btf *btf, u32 offset);
+int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
+ u32 arg_no);
+u32 btf_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto, int off);
struct btf *btf_parse_vmlinux(void);
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog);
@@ -606,17 +627,7 @@ static inline bool btf_type_is_struct_ptr(struct btf *btf, const struct btf_type
return btf_type_is_struct(t);
}
#else
-static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
- u32 type_id)
-{
- return NULL;
-}
-
-static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
-{
-}
-
-static inline int btf_relocate(void *log, struct btf *btf, const struct btf *base_btf,
+static inline int btf_relocate(struct btf *btf, const struct btf *base_btf,
__u32 **map_ids)
{
return -EOPNOTSUPP;
@@ -633,11 +644,6 @@ static inline __u32 *btf_field_iter_next(struct btf_field_iter *it)
return NULL;
}
-static inline const char *btf_name_by_offset(const struct btf *btf,
- u32 offset)
-{
- return NULL;
-}
static inline u32 *btf_kfunc_id_set_contains(const struct btf *btf,
u32 kfunc_btf_id,
struct bpf_prog *prog)
@@ -683,5 +689,5 @@ static inline int btf_check_iter_arg(struct btf *btf, const struct btf_type *fun
{
return -EOPNOTSUPP;
}
-#endif
+#endif /* CONFIG_BPF_SYSCALL */
#endif
diff --git a/kernel/Makefile b/kernel/Makefile
index 6785982013dce..d09d7f102c335 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -121,6 +121,7 @@ obj-$(CONFIG_RETHOOK) += trace/
obj-$(CONFIG_IRQ_WORK) += irq_work.o
obj-$(CONFIG_CPU_PM) += cpu_pm.o
obj-$(CONFIG_BPF) += bpf/
+obj-$(CONFIG_BTF) += btf/
obj-$(CONFIG_KCSAN) += kcsan/
obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o
obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 79cf22860a99b..37d7ebf4808f3 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -14,7 +14,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
-obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o
+obj-$(CONFIG_BPF_SYSCALL) += memalloc.o rqspinlock.o stream.o
ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)
obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o
endif
diff --git a/kernel/btf/Makefile b/kernel/btf/Makefile
new file mode 100644
index 0000000000000..e0365d5c7e9bd
--- /dev/null
+++ b/kernel/btf/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_BTF) += btf.o
+obj-$(CONFIG_BPF_SYSCALL) += bpf.o
diff --git a/kernel/bpf/btf.c b/kernel/btf/bpf.c
similarity index 50%
rename from kernel/bpf/btf.c
rename to kernel/btf/bpf.c
index 71f9143fe90f3..986d29f4bc3ba 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/btf/bpf.c
@@ -1,33 +1,35 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2018 Facebook */
+/* BPF-specific BTF extensions */
#include <uapi/linux/btf.h>
#include <uapi/linux/bpf.h>
-#include <uapi/linux/bpf_perf_event.h>
#include <uapi/linux/types.h>
-#include <linux/seq_file.h>
#include <linux/compiler.h>
#include <linux/ctype.h>
#include <linux/errno.h>
#include <linux/slab.h>
-#include <linux/anon_inodes.h>
#include <linux/file.h>
-#include <linux/uaccess.h>
#include <linux/kernel.h>
-#include <linux/idr.h>
#include <linux/sort.h>
+#include <linux/seq_file.h>
#include <linux/bpf_verifier.h>
#include <linux/btf.h>
#include <linux/btf_ids.h>
+#include <linux/bsearch.h>
+#include <linux/string.h>
+#include <linux/overflow.h>
+#include <linux/kobject.h>
+
+#include <uapi/linux/bpf_perf_event.h>
+#include <linux/anon_inodes.h>
+#include <linux/uaccess.h>
+#include <linux/idr.h>
#include <linux/bpf.h>
#include <linux/bpf_lsm.h>
#include <linux/skmsg.h>
#include <linux/perf_event.h>
-#include <linux/bsearch.h>
-#include <linux/kobject.h>
-#include <linux/string.h>
#include <linux/sysfs.h>
-#include <linux/overflow.h>
#include <net/netfilter/nf_bpf_link.h>
@@ -35,173 +37,7 @@
#include <net/xdp.h>
#include "../tools/lib/bpf/relo_core.h"
-/* BTF (BPF Type Format) is the meta data format which describes
- * the data types of BPF program/map. Hence, it basically focus
- * on the C programming language which the modern BPF is primary
- * using.
- *
- * ELF Section:
- * ~~~~~~~~~~~
- * The BTF data is stored under the ".BTF" ELF section
- *
- * struct btf_type:
- * ~~~~~~~~~~~~~~~
- * Each 'struct btf_type' object describes a C data type.
- * Depending on the type it is describing, a 'struct btf_type'
- * object may be followed by more data. F.e.
- * To describe an array, 'struct btf_type' is followed by
- * 'struct btf_array'.
- *
- * 'struct btf_type' and any extra data following it are
- * 4 bytes aligned.
- *
- * Type section:
- * ~~~~~~~~~~~~~
- * The BTF type section contains a list of 'struct btf_type' objects.
- * Each one describes a C type. Recall from the above section
- * that a 'struct btf_type' object could be immediately followed by extra
- * data in order to describe some particular C types.
- *
- * type_id:
- * ~~~~~~~
- * Each btf_type object is identified by a type_id. The type_id
- * is implicitly implied by the location of the btf_type object in
- * the BTF type section. The first one has type_id 1. The second
- * one has type_id 2...etc. Hence, an earlier btf_type has
- * a smaller type_id.
- *
- * A btf_type object may refer to another btf_type object by using
- * type_id (i.e. the "type" in the "struct btf_type").
- *
- * NOTE that we cannot assume any reference-order.
- * A btf_type object can refer to an earlier btf_type object
- * but it can also refer to a later btf_type object.
- *
- * For example, to describe "const void *". A btf_type
- * object describing "const" may refer to another btf_type
- * object describing "void *". This type-reference is done
- * by specifying type_id:
- *
- * [1] CONST (anon) type_id=2
- * [2] PTR (anon) type_id=0
- *
- * The above is the btf_verifier debug log:
- * - Each line started with "[?]" is a btf_type object
- * - [?] is the type_id of the btf_type object.
- * - CONST/PTR is the BTF_KIND_XXX
- * - "(anon)" is the name of the type. It just
- * happens that CONST and PTR has no name.
- * - type_id=XXX is the 'u32 type' in btf_type
- *
- * NOTE: "void" has type_id 0
- *
- * String section:
- * ~~~~~~~~~~~~~~
- * The BTF string section contains the names used by the type section.
- * Each string is referred by an "offset" from the beginning of the
- * string section.
- *
- * Each string is '\0' terminated.
- *
- * The first character in the string section must be '\0'
- * which is used to mean 'anonymous'. Some btf_type may not
- * have a name.
- */
-
-/* BTF verification:
- *
- * To verify BTF data, two passes are needed.
- *
- * Pass #1
- * ~~~~~~~
- * The first pass is to collect all btf_type objects to
- * an array: "btf->types".
- *
- * Depending on the C type that a btf_type is describing,
- * a btf_type may be followed by extra data. We don't know
- * how many btf_type is there, and more importantly we don't
- * know where each btf_type is located in the type section.
- *
- * Without knowing the location of each type_id, most verifications
- * cannot be done. e.g. an earlier btf_type may refer to a later
- * btf_type (recall the "const void *" above), so we cannot
- * check this type-reference in the first pass.
- *
- * In the first pass, it still does some verifications (e.g.
- * checking the name is a valid offset to the string section).
- *
- * Pass #2
- * ~~~~~~~
- * The main focus is to resolve a btf_type that is referring
- * to another type.
- *
- * We have to ensure the referring type:
- * 1) does exist in the BTF (i.e. in btf->types[])
- * 2) does not cause a loop:
- * struct A {
- * struct B b;
- * };
- *
- * struct B {
- * struct A a;
- * };
- *
- * btf_type_needs_resolve() decides if a btf_type needs
- * to be resolved.
- *
- * The needs_resolve type implements the "resolve()" ops which
- * essentially does a DFS and detects backedge.
- *
- * During resolve (or DFS), different C types have different
- * "RESOLVED" conditions.
- *
- * When resolving a BTF_KIND_STRUCT, we need to resolve all its
- * members because a member is always referring to another
- * type. A struct's member can be treated as "RESOLVED" if
- * it is referring to a BTF_KIND_PTR. Otherwise, the
- * following valid C struct would be rejected:
- *
- * struct A {
- * int m;
- * struct A *a;
- * };
- *
- * When resolving a BTF_KIND_PTR, it needs to keep resolving if
- * it is referring to another BTF_KIND_PTR. Otherwise, we cannot
- * detect a pointer loop, e.g.:
- * BTF_KIND_CONST -> BTF_KIND_PTR -> BTF_KIND_CONST -> BTF_KIND_PTR +
- * ^ |
- * +-----------------------------------------+
- *
- */
-
-#define BITS_PER_U128 (sizeof(u64) * BITS_PER_BYTE * 2)
-#define BITS_PER_BYTE_MASK (BITS_PER_BYTE - 1)
-#define BITS_PER_BYTE_MASKED(bits) ((bits) & BITS_PER_BYTE_MASK)
-#define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
-#define BITS_ROUNDUP_BYTES(bits) \
- (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
-
-#define BTF_INFO_MASK 0x9f00ffff
-#define BTF_INT_MASK 0x0fffffff
-#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
-#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
-
-/* 16MB for 64k structs and each has 16 members and
- * a few MB spaces for the string section.
- * The hard limit is S32_MAX.
- */
-#define BTF_MAX_SIZE (16 * 1024 * 1024)
-
-#define for_each_member_from(i, from, struct_type, member) \
- for (i = from, member = btf_type_member(struct_type) + from; \
- i < btf_type_vlen(struct_type); \
- i++, member++)
-
-#define for_each_vsi_from(i, from, struct_type, member) \
- for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
- i < btf_type_vlen(struct_type); \
- i++, member++)
+#include "btf.h"
DEFINE_IDR(btf_idr);
DEFINE_SPINLOCK(btf_idr_lock);
@@ -250,431 +86,12 @@ struct btf_struct_ops_tab {
u32 capacity;
struct bpf_struct_ops_desc ops[];
};
-
-struct btf {
- void *data;
- struct btf_type **types;
- u32 *resolved_ids;
- u32 *resolved_sizes;
- const char *strings;
- void *nohdr_data;
- struct btf_header hdr;
- u32 nr_types; /* includes VOID for base BTF */
- u32 named_start_id;
- u32 types_size;
- u32 data_size;
- refcount_t refcnt;
u32 id;
struct rcu_head rcu;
struct btf_kfunc_set_tab *kfunc_set_tab;
struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
struct btf_struct_metas *struct_meta_tab;
struct btf_struct_ops_tab *struct_ops_tab;
-
- /* split BTF support */
- struct btf *base_btf;
- u32 start_id; /* first type ID in this BTF (0 for base BTF) */
- u32 start_str_off; /* first string offset (0 for base BTF) */
- char name[MODULE_NAME_LEN];
- bool kernel_btf;
- __u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
-};
-
-enum verifier_phase {
- CHECK_META,
- CHECK_TYPE,
-};
-
-struct resolve_vertex {
- const struct btf_type *t;
- u32 type_id;
- u16 next_member;
-};
-
-enum visit_state {
- NOT_VISITED,
- VISITED,
- RESOLVED,
-};
-
-enum resolve_mode {
- RESOLVE_TBD, /* To Be Determined */
- RESOLVE_PTR, /* Resolving for Pointer */
- RESOLVE_STRUCT_OR_ARRAY, /* Resolving for struct/union
- * or array
- */
-};
-
-#define MAX_RESOLVE_DEPTH 32
-
-struct btf_sec_info {
- u32 off;
- u32 len;
-};
-
-struct btf_verifier_env {
- struct btf *btf;
- u8 *visit_states;
- struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
- struct bpf_verifier_log log;
- u32 log_type_id;
- u32 top_stack;
- enum verifier_phase phase;
- enum resolve_mode resolve_mode;
-};
-
-static const char * const btf_kind_str[NR_BTF_KINDS] = {
- [BTF_KIND_UNKN] = "UNKNOWN",
- [BTF_KIND_INT] = "INT",
- [BTF_KIND_PTR] = "PTR",
- [BTF_KIND_ARRAY] = "ARRAY",
- [BTF_KIND_STRUCT] = "STRUCT",
- [BTF_KIND_UNION] = "UNION",
- [BTF_KIND_ENUM] = "ENUM",
- [BTF_KIND_FWD] = "FWD",
- [BTF_KIND_TYPEDEF] = "TYPEDEF",
- [BTF_KIND_VOLATILE] = "VOLATILE",
- [BTF_KIND_CONST] = "CONST",
- [BTF_KIND_RESTRICT] = "RESTRICT",
- [BTF_KIND_FUNC] = "FUNC",
- [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO",
- [BTF_KIND_VAR] = "VAR",
- [BTF_KIND_DATASEC] = "DATASEC",
- [BTF_KIND_FLOAT] = "FLOAT",
- [BTF_KIND_DECL_TAG] = "DECL_TAG",
- [BTF_KIND_TYPE_TAG] = "TYPE_TAG",
- [BTF_KIND_ENUM64] = "ENUM64",
-};
-
-const char *btf_type_str(const struct btf_type *t)
-{
- return btf_kind_str[BTF_INFO_KIND(t->info)];
-}
-
-/* Chunk size we use in safe copy of data to be shown. */
-#define BTF_SHOW_OBJ_SAFE_SIZE 32
-
-/*
- * This is the maximum size of a base type value (equivalent to a
- * 128-bit int); if we are at the end of our safe buffer and have
- * less than 16 bytes space we can't be assured of being able
- * to copy the next type safely, so in such cases we will initiate
- * a new copy.
- */
-#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
-
-/* Type name size */
-#define BTF_SHOW_NAME_SIZE 80
-
-/*
- * The suffix of a type that indicates it cannot alias another type when
- * comparing BTF IDs for kfunc invocations.
- */
-#define NOCAST_ALIAS_SUFFIX "___init"
-
-/*
- * Common data to all BTF show operations. Private show functions can add
- * their own data to a structure containing a struct btf_show and consult it
- * in the show callback. See btf_type_show() below.
- *
- * One challenge with showing nested data is we want to skip 0-valued
- * data, but in order to figure out whether a nested object is all zeros
- * we need to walk through it. As a result, we need to make two passes
- * when handling structs, unions and arrays; the first path simply looks
- * for nonzero data, while the second actually does the display. The first
- * pass is signalled by show->state.depth_check being set, and if we
- * encounter a non-zero value we set show->state.depth_to_show to
- * the depth at which we encountered it. When we have completed the
- * first pass, we will know if anything needs to be displayed if
- * depth_to_show > depth. See btf_[struct,array]_show() for the
- * implementation of this.
- *
- * Another problem is we want to ensure the data for display is safe to
- * access. To support this, the anonymous "struct {} obj" tracks the data
- * object and our safe copy of it. We copy portions of the data needed
- * to the object "copy" buffer, but because its size is limited to
- * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
- * traverse larger objects for display.
- *
- * The various data type show functions all start with a call to
- * btf_show_start_type() which returns a pointer to the safe copy
- * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
- * raw data itself). btf_show_obj_safe() is responsible for
- * using copy_from_kernel_nofault() to update the safe data if necessary
- * as we traverse the object's data. skbuff-like semantics are
- * used:
- *
- * - obj.head points to the start of the toplevel object for display
- * - obj.size is the size of the toplevel object
- * - obj.data points to the current point in the original data at
- * which our safe data starts. obj.data will advance as we copy
- * portions of the data.
- *
- * In most cases a single copy will suffice, but larger data structures
- * such as "struct task_struct" will require many copies. The logic in
- * btf_show_obj_safe() handles the logic that determines if a new
- * copy_from_kernel_nofault() is needed.
- */
-struct btf_show {
- u64 flags;
- void *target; /* target of show operation (seq file, buffer) */
- __printf(2, 0) void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
- const struct btf *btf;
- /* below are used during iteration */
- struct {
- u8 depth;
- u8 depth_to_show;
- u8 depth_check;
- u8 array_member:1,
- array_terminated:1;
- u16 array_encoding;
- u32 type_id;
- int status; /* non-zero for error */
- const struct btf_type *type;
- const struct btf_member *member;
- char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
- } state;
- struct {
- u32 size;
- void *head;
- void *data;
- u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
- } obj;
-};
-
-struct btf_kind_operations {
- s32 (*check_meta)(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left);
- int (*resolve)(struct btf_verifier_env *env,
- const struct resolve_vertex *v);
- int (*check_member)(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type);
- int (*check_kflag_member)(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type);
- void (*log_details)(struct btf_verifier_env *env,
- const struct btf_type *t);
- void (*show)(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offsets,
- struct btf_show *show);
-};
-
-static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
-static struct btf_type btf_void;
-
-static int btf_resolve(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id);
-
-static int btf_func_check(struct btf_verifier_env *env,
- const struct btf_type *t);
-
-static bool btf_type_is_modifier(const struct btf_type *t)
-{
- /* Some of them is not strictly a C modifier
- * but they are grouped into the same bucket
- * for BTF concern:
- * A type (t) that refers to another
- * type through t->type AND its size cannot
- * be determined without following the t->type.
- *
- * ptr does not fall into this bucket
- * because its size is always sizeof(void *).
- */
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_TYPEDEF:
- case BTF_KIND_VOLATILE:
- case BTF_KIND_CONST:
- case BTF_KIND_RESTRICT:
- case BTF_KIND_TYPE_TAG:
- return true;
- }
-
- return false;
-}
-
-static int btf_start_id(const struct btf *btf)
-{
- return btf->start_id + (btf->base_btf ? 0 : 1);
-}
-
-bool btf_type_is_void(const struct btf_type *t)
-{
- return t == &btf_void;
-}
-
-static bool btf_type_is_datasec(const struct btf_type *t)
-{
- return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
-}
-
-static bool btf_type_is_decl_tag(const struct btf_type *t)
-{
- return BTF_INFO_KIND(t->info) == BTF_KIND_DECL_TAG;
-}
-
-static bool btf_type_nosize(const struct btf_type *t)
-{
- return btf_type_is_void(t) || btf_type_is_fwd(t) ||
- btf_type_is_func(t) || btf_type_is_func_proto(t) ||
- btf_type_is_decl_tag(t);
-}
-
-static bool btf_type_nosize_or_null(const struct btf_type *t)
-{
- return !t || btf_type_nosize(t);
-}
-
-static bool btf_type_is_decl_tag_target(const struct btf_type *t)
-{
- return btf_type_is_func(t) || btf_type_is_struct(t) ||
- btf_type_is_var(t) || btf_type_is_typedef(t);
-}
-
-bool btf_is_vmlinux(const struct btf *btf)
-{
- return btf->kernel_btf && !btf->base_btf;
-}
-
-u32 btf_nr_types(const struct btf *btf)
-{
- u32 total = 0;
-
- while (btf) {
- total += btf->nr_types;
- btf = btf->base_btf;
- }
-
- return total;
-}
-
-/*
- * Note that vmlinux and kernel module BTFs are always sorted
- * during the building phase.
- */
-static void btf_check_sorted(struct btf *btf)
-{
- u32 i, n, named_start_id = 0;
-
- n = btf_nr_types(btf);
- if (btf_is_vmlinux(btf)) {
- for (i = btf_start_id(btf); i < n; i++) {
- const struct btf_type *t = btf_type_by_id(btf, i);
- const char *n = btf_name_by_offset(btf, t->name_off);
-
- if (n[0] != '\0') {
- btf->named_start_id = i;
- return;
- }
- }
- return;
- }
-
- for (i = btf_start_id(btf) + 1; i < n; i++) {
- const struct btf_type *ta = btf_type_by_id(btf, i - 1);
- const struct btf_type *tb = btf_type_by_id(btf, i);
- const char *na = btf_name_by_offset(btf, ta->name_off);
- const char *nb = btf_name_by_offset(btf, tb->name_off);
-
- if (strcmp(na, nb) > 0)
- return;
-
- if (named_start_id == 0 && na[0] != '\0')
- named_start_id = i - 1;
- if (named_start_id == 0 && nb[0] != '\0')
- named_start_id = i;
- }
-
- if (named_start_id)
- btf->named_start_id = named_start_id;
-}
-
-/*
- * btf_named_start_id - Get the named starting ID for the BTF
- * @btf: Pointer to the target BTF object
- * @own: Flag indicating whether to query only the current BTF (true = current BTF only,
- * false = recursively traverse the base BTF chain)
- *
- * Return value rules:
- * 1. For a sorted btf, return its named_start_id
- * 2. Else for a split BTF, return its start_id
- * 3. Else for a base BTF, return 1
- */
-u32 btf_named_start_id(const struct btf *btf, bool own)
-{
- const struct btf *base_btf = btf;
-
- while (!own && base_btf->base_btf)
- base_btf = base_btf->base_btf;
-
- return base_btf->named_start_id ?: (base_btf->start_id ?: 1);
-}
-
-static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)
-{
- const struct btf_type *t;
- const char *tname;
- s32 l, r, m;
-
- l = btf_named_start_id(btf, true);
- r = btf_nr_types(btf) - 1;
- while (l <= r) {
- m = l + (r - l) / 2;
- t = btf_type_by_id(btf, m);
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) >= 0) {
- if (l == r)
- return r;
- r = m;
- } else {
- l = m + 1;
- }
- }
-
- return btf_nr_types(btf);
-}
-
-s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
-{
- const struct btf *base_btf = btf_base_btf(btf);
- const struct btf_type *t;
- const char *tname;
- s32 id, total;
-
- if (base_btf) {
- id = btf_find_by_name_kind(base_btf, name, kind);
- if (id > 0)
- return id;
- }
-
- total = btf_nr_types(btf);
- if (btf->named_start_id > 0 && name[0]) {
- id = btf_find_by_name_kind_bsearch(btf, name);
- for (; id < total; id++) {
- t = btf_type_by_id(btf, id);
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) != 0)
- return -ENOENT;
- if (BTF_INFO_KIND(t->info) == kind)
- return id;
- }
- } else {
- for (id = btf_start_id(btf); id < total; id++) {
- t = btf_type_by_id(btf, id);
- if (BTF_INFO_KIND(t->info) != kind)
- continue;
- tname = btf_name_by_offset(btf, t->name_off);
- if (strcmp(tname, name) == 0)
- return id;
- }
- }
-
- return -ENOENT;
-}
-
s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
{
struct btf *btf;
@@ -719,4964 +136,976 @@ s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
return ret;
}
EXPORT_SYMBOL_GPL(bpf_find_btf_id);
-
-const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
- u32 id, u32 *res_id)
+static int btf_alloc_id(struct btf *btf)
{
- const struct btf_type *t = btf_type_by_id(btf, id);
+ int id;
- while (btf_type_is_modifier(t)) {
- id = t->type;
- t = btf_type_by_id(btf, t->type);
- }
+ idr_preload(GFP_KERNEL);
+ spin_lock_bh(&btf_idr_lock);
+ id = idr_alloc_cyclic(&btf_idr, btf, 1, INT_MAX, GFP_ATOMIC);
+ if (id > 0)
+ btf->id = id;
+ spin_unlock_bh(&btf_idr_lock);
+ idr_preload_end();
- if (res_id)
- *res_id = id;
+ if (WARN_ON_ONCE(!id))
+ return -ENOSPC;
- return t;
+ return id > 0 ? 0 : id;
}
-const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
- u32 id, u32 *res_id)
+static void btf_free_id(struct btf *btf)
{
- const struct btf_type *t;
-
- t = btf_type_skip_modifiers(btf, id, NULL);
- if (!btf_type_is_ptr(t))
- return NULL;
+ unsigned long flags;
- return btf_type_skip_modifiers(btf, t->type, res_id);
+ /*
+ * In map-in-map, calling map_delete_elem() on outer
+ * map will call bpf_map_put on the inner map.
+ * It will then eventually call btf_free_id()
+ * on the inner map. Some of the map_delete_elem()
+ * implementation may have irq disabled, so
+ * we need to use the _irqsave() version instead
+ * of the _bh() version.
+ */
+ spin_lock_irqsave(&btf_idr_lock, flags);
+ if (btf->id) {
+ idr_remove(&btf_idr, btf->id);
+ /*
+ * Clear the id here to make this function idempotent, since it will get
+ * called a couple of times for module BTFs: on module unload, and then
+ * the final btf_put(). btf_alloc_id() starts IDs with 1, so we can use
+ * 0 as sentinel value.
+ */
+ WRITE_ONCE(btf->id, 0);
+ }
+ spin_unlock_irqrestore(&btf_idr_lock, flags);
}
-const struct btf_type *btf_type_resolve_func_ptr(const struct btf *btf,
- u32 id, u32 *res_id)
+static void btf_free_kfunc_set_tab(struct btf *btf)
{
- const struct btf_type *ptype;
-
- ptype = btf_type_resolve_ptr(btf, id, res_id);
- if (ptype && btf_type_is_func_proto(ptype))
- return ptype;
+ struct btf_kfunc_set_tab *tab = btf->kfunc_set_tab;
+ int hook;
- return NULL;
+ if (!tab)
+ return;
+ for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
+ kfree(tab->sets[hook]);
+ kfree(tab);
+ btf->kfunc_set_tab = NULL;
}
-/* Types that act only as a source, not sink or intermediate
- * type when resolving.
- */
-static bool btf_type_is_resolve_source_only(const struct btf_type *t)
+static void btf_free_dtor_kfunc_tab(struct btf *btf)
{
- return btf_type_is_var(t) ||
- btf_type_is_decl_tag(t) ||
- btf_type_is_datasec(t);
+ struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
+
+ if (!tab)
+ return;
+ kfree(tab);
+ btf->dtor_kfunc_tab = NULL;
}
-/* What types need to be resolved?
- *
- * btf_type_is_modifier() is an obvious one.
- *
- * btf_type_is_struct() because its member refers to
- * another type (through member->type).
- *
- * btf_type_is_var() because the variable refers to
- * another type. btf_type_is_datasec() holds multiple
- * btf_type_is_var() types that need resolving.
- *
- * btf_type_is_array() because its element (array->type)
- * refers to another type. Array can be thought of a
- * special case of struct while array just has the same
- * member-type repeated by array->nelems of times.
- */
-static bool btf_type_needs_resolve(const struct btf_type *t)
+static void btf_struct_metas_free(struct btf_struct_metas *tab)
{
- return btf_type_is_modifier(t) ||
- btf_type_is_ptr(t) ||
- btf_type_is_struct(t) ||
- btf_type_is_array(t) ||
- btf_type_is_var(t) ||
- btf_type_is_func(t) ||
- btf_type_is_decl_tag(t) ||
- btf_type_is_datasec(t);
+ int i;
+
+ if (!tab)
+ return;
+ for (i = 0; i < tab->cnt; i++)
+ btf_record_free(tab->types[i].record);
+ kfree(tab);
}
-/* t->size can be used */
-static bool btf_type_has_size(const struct btf_type *t)
+static void btf_free_struct_meta_tab(struct btf *btf)
{
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_INT:
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- case BTF_KIND_ENUM:
- case BTF_KIND_DATASEC:
- case BTF_KIND_FLOAT:
- case BTF_KIND_ENUM64:
- return true;
- }
+ struct btf_struct_metas *tab = btf->struct_meta_tab;
- return false;
+ btf_struct_metas_free(tab);
+ btf->struct_meta_tab = NULL;
}
-static const char *btf_int_encoding_str(u8 encoding)
+static void btf_free_struct_ops_tab(struct btf *btf)
{
- if (encoding == 0)
- return "(none)";
- else if (encoding == BTF_INT_SIGNED)
- return "SIGNED";
- else if (encoding == BTF_INT_CHAR)
- return "CHAR";
- else if (encoding == BTF_INT_BOOL)
- return "BOOL";
- else
- return "UNKN";
-}
+ struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
+ u32 i;
-static u32 btf_type_int(const struct btf_type *t)
-{
- return *(u32 *)(t + 1);
-}
+ if (!tab)
+ return;
-static const struct btf_array *btf_type_array(const struct btf_type *t)
-{
- return (const struct btf_array *)(t + 1);
-}
+ for (i = 0; i < tab->cnt; i++)
+ bpf_struct_ops_desc_release(&tab->ops[i]);
-static const struct btf_enum *btf_type_enum(const struct btf_type *t)
-{
- return (const struct btf_enum *)(t + 1);
+ kfree(tab);
+ btf->struct_ops_tab = NULL;
}
-static const struct btf_var *btf_type_var(const struct btf_type *t)
+void btf_free_bpf_data(struct btf *btf)
{
- return (const struct btf_var *)(t + 1);
+ btf_free_struct_meta_tab(btf);
+ btf_free_dtor_kfunc_tab(btf);
+ btf_free_kfunc_set_tab(btf);
+ btf_free_struct_ops_tab(btf);
}
-static const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t)
+static void btf_free_rcu(struct rcu_head *rcu)
{
- return (const struct btf_decl_tag *)(t + 1);
-}
+ struct btf *btf = container_of(rcu, struct btf, rcu);
-static const struct btf_enum64 *btf_type_enum64(const struct btf_type *t)
-{
- return (const struct btf_enum64 *)(t + 1);
+ btf_free(btf);
}
-static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
+void btf_put_bpf(struct btf *btf)
{
- return kind_ops[BTF_INFO_KIND(t->info)];
+ btf_free_id(btf);
+ call_rcu(&btf->rcu, btf_free_rcu);
}
+enum {
+ BTF_FIELD_IGNORE = 0,
+ BTF_FIELD_FOUND = 1,
+};
-static bool btf_name_offset_valid(const struct btf *btf, u32 offset)
-{
- if (!BTF_STR_OFFSET_VALID(offset))
- return false;
-
- while (offset < btf->start_str_off)
- btf = btf->base_btf;
-
- offset -= btf->start_str_off;
- return offset < btf->hdr.str_len;
-}
+struct btf_field_info {
+ enum btf_field_type type;
+ u32 off;
+ union {
+ struct {
+ u32 type_id;
+ } kptr;
+ struct {
+ const char *node_name;
+ u32 value_btf_id;
+ } graph_root;
+ };
+};
-static bool __btf_name_char_ok(char c, bool first)
+static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
+ u32 off, int sz, enum btf_field_type field_type,
+ struct btf_field_info *info)
{
- if ((first ? !isalpha(c) :
- !isalnum(c)) &&
- c != '_' &&
- c != '.')
- return false;
- return true;
+ if (!__btf_type_is_struct(t))
+ return BTF_FIELD_IGNORE;
+ if (t->size != sz)
+ return BTF_FIELD_IGNORE;
+ info->type = field_type;
+ info->off = off;
+ return BTF_FIELD_FOUND;
}
-const char *btf_str_by_offset(const struct btf *btf, u32 offset)
+static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
+ u32 off, int sz, struct btf_field_info *info, u32 field_mask)
{
- while (offset < btf->start_str_off)
- btf = btf->base_btf;
-
- offset -= btf->start_str_off;
- if (offset < btf->hdr.str_len)
- return &btf->strings[offset];
-
- return NULL;
-}
+ enum btf_field_type type;
+ const char *tag_value;
+ bool is_type_tag;
+ u32 res_id;
-static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
-{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
+ /* Permit modifiers on the pointer itself */
+ if (btf_type_is_volatile(t))
+ t = btf_type_by_id(btf, t->type);
+ /* For PTR, sz is always == 8 */
+ if (!btf_type_is_ptr(t))
+ return BTF_FIELD_IGNORE;
+ t = btf_type_by_id(btf, t->type);
+ is_type_tag = btf_type_is_type_tag(t) && !btf_type_kflag(t);
+ if (!is_type_tag)
+ return BTF_FIELD_IGNORE;
+ /* Reject extra tags */
+ if (btf_type_is_type_tag(btf_type_by_id(btf, t->type)))
+ return -EINVAL;
+ tag_value = __btf_name_by_offset(btf, t->name_off);
+ if (!strcmp("kptr_untrusted", tag_value))
+ type = BPF_KPTR_UNREF;
+ else if (!strcmp("kptr", tag_value))
+ type = BPF_KPTR_REF;
+ else if (!strcmp("percpu_kptr", tag_value))
+ type = BPF_KPTR_PERCPU;
+ else if (!strcmp("uptr", tag_value))
+ type = BPF_UPTR;
+ else
+ return -EINVAL;
- if (!__btf_name_char_ok(*src, true))
- return false;
+ if (!(type & field_mask))
+ return BTF_FIELD_IGNORE;
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- src++;
- while (*src && src < src_limit) {
- if (!__btf_name_char_ok(*src, false))
- return false;
- src++;
- }
+ /* Get the base type */
+ t = btf_type_skip_modifiers(btf, t->type, &res_id);
+ /* Only pointer to struct is allowed */
+ if (!__btf_type_is_struct(t))
+ return -EINVAL;
- return !*src;
+ info->type = type;
+ info->off = off;
+ info->kptr.type_id = res_id;
+ return BTF_FIELD_FOUND;
}
-/* Allow any printable character in DATASEC names */
-static bool btf_name_valid_section(const struct btf *btf, u32 offset)
+int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
+ int comp_idx, const char *tag_key, int last_id)
{
- /* offset must be valid */
- const char *src = btf_str_by_offset(btf, offset);
- const char *src_limit;
+ int len = strlen(tag_key);
+ int i, n;
- if (!*src)
- return false;
+ for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
+ const struct btf_type *t = btf_type_by_id(btf, i);
- /* set a limit on identifier length */
- src_limit = src + KSYM_NAME_LEN;
- while (*src && src < src_limit) {
- if (!isprint(*src))
- return false;
- src++;
+ if (!btf_type_is_decl_tag(t))
+ continue;
+ if (pt != btf_type_by_id(btf, t->type))
+ continue;
+ if (btf_type_decl_tag(t)->component_idx != comp_idx)
+ continue;
+ if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
+ continue;
+ return i;
}
-
- return !*src;
-}
-
-static const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
-{
- const char *name;
-
- if (!offset)
- return "(anon)";
-
- name = btf_str_by_offset(btf, offset);
- return name ?: "(invalid-name-offset)";
+ return -ENOENT;
}
-const char *btf_name_by_offset(const struct btf *btf, u32 offset)
+const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
+ int comp_idx, const char *tag_key)
{
- return btf_str_by_offset(btf, offset);
-}
+ const char *value = NULL;
+ const struct btf_type *t;
+ int len, id;
-const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
+ id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
+ btf_named_start_id(btf, false) - 1);
+ if (id < 0)
+ return ERR_PTR(id);
- type_id -= btf->start_id;
- if (type_id >= btf->nr_types)
- return NULL;
- return btf->types[type_id];
-}
-EXPORT_SYMBOL_GPL(btf_type_by_id);
-
-/*
- * Check that the type @t is a regular int. This means that @t is not
- * a bit field and it has the same size as either of u8/u16/u32/u64
- * or __int128. If @expected_size is not zero, then size of @t should
- * be the same. A caller should already have checked that the type @t
- * is an integer.
- */
-static bool __btf_type_int_is_regular(const struct btf_type *t, size_t expected_size)
-{
- u32 int_data = btf_type_int(t);
- u8 nr_bits = BTF_INT_BITS(int_data);
- u8 nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
-
- return BITS_PER_BYTE_MASKED(nr_bits) == 0 &&
- BTF_INT_OFFSET(int_data) == 0 &&
- (nr_bytes <= 16 && is_power_of_2(nr_bytes)) &&
- (expected_size == 0 || nr_bytes == expected_size);
-}
+ t = btf_type_by_id(btf, id);
+ len = strlen(tag_key);
+ value = __btf_name_by_offset(btf, t->name_off) + len;
-static bool btf_type_int_is_regular(const struct btf_type *t)
-{
- return __btf_type_int_is_regular(t, 0);
-}
+ /* Prevent duplicate entries for same type */
+ id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, id);
+ if (id >= 0)
+ return ERR_PTR(-EEXIST);
-bool btf_type_is_i32(const struct btf_type *t)
-{
- return btf_type_is_int(t) && __btf_type_int_is_regular(t, 4);
+ return value;
}
-bool btf_type_is_i64(const struct btf_type *t)
+static int
+btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
+ const struct btf_type *t, int comp_idx, u32 off,
+ int sz, struct btf_field_info *info,
+ enum btf_field_type head_type)
{
- return btf_type_is_int(t) && __btf_type_int_is_regular(t, 8);
-}
+ const char *node_field_name;
+ const char *value_type;
+ s32 id;
-bool btf_type_is_primitive(const struct btf_type *t)
-{
- return (btf_type_is_int(t) && btf_type_int_is_regular(t)) ||
- btf_is_any_enum(t);
+ if (!__btf_type_is_struct(t))
+ return BTF_FIELD_IGNORE;
+ if (t->size != sz)
+ return BTF_FIELD_IGNORE;
+ value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
+ if (IS_ERR(value_type))
+ return -EINVAL;
+ node_field_name = strstr(value_type, ":");
+ if (!node_field_name)
+ return -EINVAL;
+ value_type = kstrndup(value_type, node_field_name - value_type,
+ GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!value_type)
+ return -ENOMEM;
+ id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
+ kfree(value_type);
+ if (id < 0)
+ return id;
+ node_field_name++;
+ if (str_is_empty(node_field_name))
+ return -EINVAL;
+ info->type = head_type;
+ info->off = off;
+ info->graph_root.value_btf_id = id;
+ info->graph_root.node_name = node_field_name;
+ return BTF_FIELD_FOUND;
}
-/*
- * Check that given struct member is a regular int with expected
- * offset and size.
- */
-bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
- const struct btf_member *m,
- u32 expected_offset, u32 expected_size)
+static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_type,
+ u32 field_mask, u32 *seen_mask, int *align, int *sz)
{
- const struct btf_type *t;
- u32 id, int_data;
- u8 nr_bits;
+ const struct {
+ enum btf_field_type type;
+ const char *const name;
+ const bool is_unique;
+ } field_types[] = {
+ { BPF_SPIN_LOCK, "bpf_spin_lock", true },
+ { BPF_RES_SPIN_LOCK, "bpf_res_spin_lock", true },
+ { BPF_TIMER, "bpf_timer", true },
+ { BPF_WORKQUEUE, "bpf_wq", true },
+ { BPF_TASK_WORK, "bpf_task_work", true },
+ { BPF_LIST_HEAD, "bpf_list_head", false },
+ { BPF_LIST_NODE, "bpf_list_node", false },
+ { BPF_RB_ROOT, "bpf_rb_root", false },
+ { BPF_RB_NODE, "bpf_rb_node", false },
+ { BPF_REFCOUNT, "bpf_refcount", false },
+ };
+ int type = 0, i;
+ const char *name = __btf_name_by_offset(btf, var_type->name_off);
+ const char *field_type_name;
+ enum btf_field_type field_type;
+ bool is_unique;
- id = m->type;
- t = btf_type_id_size(btf, &id, NULL);
- if (!t || !btf_type_is_int(t))
- return false;
+ for (i = 0; i < ARRAY_SIZE(field_types); ++i) {
+ field_type = field_types[i].type;
+ field_type_name = field_types[i].name;
+ is_unique = field_types[i].is_unique;
+ if (!(field_mask & field_type) || strcmp(name, field_type_name))
+ continue;
+ if (is_unique) {
+ if (*seen_mask & field_type)
+ return -E2BIG;
+ *seen_mask |= field_type;
+ }
+ type = field_type;
+ goto end;
+ }
- int_data = btf_type_int(t);
- nr_bits = BTF_INT_BITS(int_data);
- if (btf_type_kflag(s)) {
- u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
- u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
-
- /* if kflag set, int should be a regular int and
- * bit offset should be at byte boundary.
- */
- return !bitfield_size &&
- BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
- BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
- }
-
- if (BTF_INT_OFFSET(int_data) ||
- BITS_PER_BYTE_MASKED(m->offset) ||
- BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
- BITS_PER_BYTE_MASKED(nr_bits) ||
- BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
- return false;
-
- return true;
-}
-
-/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
-static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
- u32 id)
-{
- const struct btf_type *t = btf_type_by_id(btf, id);
-
- while (btf_type_is_modifier(t) &&
- BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
- t = btf_type_by_id(btf, t->type);
+ /* Only return BPF_KPTR when all other types with matchable names fail */
+ if (field_mask & (BPF_KPTR | BPF_UPTR) && !__btf_type_is_struct(var_type)) {
+ type = BPF_KPTR_REF;
+ goto end;
}
-
- return t;
+ return 0;
+end:
+ *sz = btf_field_type_size(type);
+ *align = btf_field_type_align(type);
+ return type;
}
-#define BTF_SHOW_MAX_ITER 10
-
-#define BTF_KIND_BIT(kind) (1ULL << kind)
-
-/*
- * Populate show->state.name with type name information.
- * Format of type name is
+/* Repeat a number of fields for a specified number of times.
*
- * [.member_name = ] (type_name)
+ * Copy the fields starting from the first field and repeat them for
+ * repeat_cnt times. The fields are repeated by adding the offset of each
+ * field with
+ * (i + 1) * elem_size
+ * where i is the repeat index and elem_size is the size of an element.
*/
-static const char *btf_show_name(struct btf_show *show)
+static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
+ u32 field_cnt, u32 repeat_cnt, u32 elem_size)
{
- /* BTF_MAX_ITER array suffixes "[]" */
- const char *array_suffixes = "[][][][][][][][][][]";
- const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
- /* BTF_MAX_ITER pointer suffixes "*" */
- const char *ptr_suffixes = "**********";
- const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
- const char *name = NULL, *prefix = "", *parens = "";
- const struct btf_member *m = show->state.member;
- const struct btf_type *t;
- const struct btf_array *array;
- u32 id = show->state.type_id;
- const char *member = NULL;
- bool show_member = false;
- u64 kinds = 0;
- int i;
-
- show->state.name[0] = '\0';
-
- /*
- * Don't show type name if we're showing an array member;
- * in that case we show the array type so don't need to repeat
- * ourselves for each member.
- */
- if (show->state.array_member)
- return "";
-
- /* Retrieve member name, if any. */
- if (m) {
- member = btf_name_by_offset(show->btf, m->name_off);
- show_member = strlen(member) > 0;
- id = m->type;
- }
-
- /*
- * Start with type_id, as we have resolved the struct btf_type *
- * via btf_modifier_show() past the parent typedef to the child
- * struct, int etc it is defined as. In such cases, the type_id
- * still represents the starting type while the struct btf_type *
- * in our show->state points at the resolved type of the typedef.
- */
- t = btf_type_by_id(show->btf, id);
- if (!t)
- return "";
+ u32 i, j;
+ u32 cur;
- /*
- * The goal here is to build up the right number of pointer and
- * array suffixes while ensuring the type name for a typedef
- * is represented. Along the way we accumulate a list of
- * BTF kinds we have encountered, since these will inform later
- * display; for example, pointer types will not require an
- * opening "{" for struct, we will just display the pointer value.
- *
- * We also want to accumulate the right number of pointer or array
- * indices in the format string while iterating until we get to
- * the typedef/pointee/array member target type.
- *
- * We start by pointing at the end of pointer and array suffix
- * strings; as we accumulate pointers and arrays we move the pointer
- * or array string backwards so it will show the expected number of
- * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
- * and/or arrays and typedefs are supported as a precaution.
- *
- * We also want to get typedef name while proceeding to resolve
- * type it points to so that we can add parentheses if it is a
- * "typedef struct" etc.
- */
- for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
-
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_TYPEDEF:
- if (!name)
- name = btf_name_by_offset(show->btf,
- t->name_off);
- kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
- id = t->type;
- break;
- case BTF_KIND_ARRAY:
- kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
- parens = "[";
- if (!t)
- return "";
- array = btf_type_array(t);
- if (array_suffix > array_suffixes)
- array_suffix -= 2;
- id = array->type;
- break;
- case BTF_KIND_PTR:
- kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
- if (ptr_suffix > ptr_suffixes)
- ptr_suffix -= 1;
- id = t->type;
+ /* Ensure not repeating fields that should not be repeated. */
+ for (i = 0; i < field_cnt; i++) {
+ switch (info[i].type) {
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ case BPF_LIST_HEAD:
+ case BPF_RB_ROOT:
break;
default:
- id = 0;
- break;
+ return -EINVAL;
}
- if (!id)
- break;
- t = btf_type_skip_qualifiers(show->btf, id);
- }
- /* We may not be able to represent this type; bail to be safe */
- if (i == BTF_SHOW_MAX_ITER)
- return "";
-
- if (!name)
- name = btf_name_by_offset(show->btf, t->name_off);
-
- switch (BTF_INFO_KIND(t->info)) {
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
- "struct" : "union";
- /* if it's an array of struct/union, parens is already set */
- if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
- parens = "{";
- break;
- case BTF_KIND_ENUM:
- case BTF_KIND_ENUM64:
- prefix = "enum";
- break;
- default:
- break;
}
- /* pointer does not require parens */
- if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
- parens = "";
- /* typedef does not require struct/union/enum prefix */
- if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
- prefix = "";
-
- if (!name)
- name = "";
-
- /* Even if we don't want type name info, we want parentheses etc */
- if (show->flags & BTF_SHOW_NONAME)
- snprintf(show->state.name, sizeof(show->state.name), "%s",
- parens);
- else
- snprintf(show->state.name, sizeof(show->state.name),
- "%s%s%s(%s%s%s%s%s%s)%s",
- /* first 3 strings comprise ".member = " */
- show_member ? "." : "",
- show_member ? member : "",
- show_member ? " = " : "",
- /* ...next is our prefix (struct, enum, etc) */
- prefix,
- strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
- /* ...this is the type name itself */
- name,
- /* ...suffixed by the appropriate '*', '[]' suffixes */
- strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
- array_suffix, parens);
-
- return show->state.name;
-}
+ /* The type of struct size or variable size is u32,
+ * so the multiplication will not overflow.
+ */
+ if (field_cnt * (repeat_cnt + 1) > info_cnt)
+ return -E2BIG;
-static const char *__btf_show_indent(struct btf_show *show)
-{
- const char *indents = " ";
- const char *indent = &indents[strlen(indents)];
+ cur = field_cnt;
+ for (i = 0; i < repeat_cnt; i++) {
+ memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
+ for (j = 0; j < field_cnt; j++)
+ info[cur++].off += (i + 1) * elem_size;
+ }
- if ((indent - show->state.depth) >= indents)
- return indent - show->state.depth;
- return indents;
+ return 0;
}
-static const char *btf_show_indent(struct btf_show *show)
-{
- return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
-}
+static int btf_find_struct_field(const struct btf *btf,
+ const struct btf_type *t, u32 field_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level);
-static const char *btf_show_newline(struct btf_show *show)
+/* Find special fields in the struct type of a field.
+ *
+ * This function is used to find fields of special types that is not a
+ * global variable or a direct field of a struct type. It also handles the
+ * repetition if it is the element type of an array.
+ */
+static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
+ u32 off, u32 nelems,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt, u32 level)
{
- return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
-}
+ int ret, err, i;
-static const char *btf_show_delim(struct btf_show *show)
-{
- if (show->state.depth == 0)
- return "";
+ level++;
+ if (level >= MAX_RESOLVE_DEPTH)
+ return -E2BIG;
- if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
- BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
- return "|";
+ ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
- return ",";
-}
+ if (ret <= 0)
+ return ret;
-__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
-{
- va_list args;
+ /* Shift the offsets of the nested struct fields to the offsets
+ * related to the container.
+ */
+ for (i = 0; i < ret; i++)
+ info[i].off += off;
- if (!show->state.depth_check) {
- va_start(args, fmt);
- show->showfn(show, fmt, args);
- va_end(args);
+ if (nelems > 1) {
+ err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
+ if (err == 0)
+ ret *= nelems;
+ else
+ ret = err;
}
-}
-
-/* Macros are used here as btf_show_type_value[s]() prepends and appends
- * format specifiers to the format specifier passed in; these do the work of
- * adding indentation, delimiters etc while the caller simply has to specify
- * the type value(s) in the format specifier + value(s).
- */
-#define btf_show_type_value(show, fmt, value) \
- do { \
- if ((value) != (__typeof__(value))0 || \
- (show->flags & BTF_SHOW_ZERO) || \
- show->state.depth == 0) { \
- btf_show(show, "%s%s" fmt "%s%s", \
- btf_show_indent(show), \
- btf_show_name(show), \
- value, btf_show_delim(show), \
- btf_show_newline(show)); \
- if (show->state.depth > show->state.depth_to_show) \
- show->state.depth_to_show = show->state.depth; \
- } \
- } while (0)
-
-#define btf_show_type_values(show, fmt, ...) \
- do { \
- btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
- btf_show_name(show), \
- __VA_ARGS__, btf_show_delim(show), \
- btf_show_newline(show)); \
- if (show->state.depth > show->state.depth_to_show) \
- show->state.depth_to_show = show->state.depth; \
- } while (0)
-
-/* How much is left to copy to safe buffer after @data? */
-static int btf_show_obj_size_left(struct btf_show *show, void *data)
-{
- return show->obj.head + show->obj.size - data;
-}
-
-/* Is object pointed to by @data of @size already copied to our safe buffer? */
-static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
-{
- return data >= show->obj.data &&
- (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
-}
-/*
- * If object pointed to by @data of @size falls within our safe buffer, return
- * the equivalent pointer to the same safe data. Assumes
- * copy_from_kernel_nofault() has already happened and our safe buffer is
- * populated.
- */
-static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
-{
- if (btf_show_obj_is_safe(show, data, size))
- return show->obj.safe + (data - show->obj.data);
- return NULL;
+ return ret;
}
-/*
- * Return a safe-to-access version of data pointed to by @data.
- * We do this by copying the relevant amount of information
- * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
- *
- * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
- * safe copy is needed.
- *
- * Otherwise we need to determine if we have the required amount
- * of data (determined by the @data pointer and the size of the
- * largest base type we can encounter (represented by
- * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
- * that we will be able to print some of the current object,
- * and if more is needed a copy will be triggered.
- * Some objects such as structs will not fit into the buffer;
- * in such cases additional copies when we iterate over their
- * members may be needed.
- *
- * btf_show_obj_safe() is used to return a safe buffer for
- * btf_show_start_type(); this ensures that as we recurse into
- * nested types we always have safe data for the given type.
- * This approach is somewhat wasteful; it's possible for example
- * that when iterating over a large union we'll end up copying the
- * same data repeatedly, but the goal is safety not performance.
- * We use stack data as opposed to per-CPU buffers because the
- * iteration over a type can take some time, and preemption handling
- * would greatly complicate use of the safe buffer.
- */
-static void *btf_show_obj_safe(struct btf_show *show,
- const struct btf_type *t,
- void *data)
+static int btf_find_field_one(const struct btf *btf,
+ const struct btf_type *var,
+ const struct btf_type *var_type,
+ int var_idx,
+ u32 off, u32 expected_size,
+ u32 field_mask, u32 *seen_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level)
{
- const struct btf_type *rt;
- int size_left, size;
- void *safe = NULL;
-
- if (show->flags & BTF_SHOW_UNSAFE)
- return data;
-
- rt = btf_resolve_size(show->btf, t, &size);
- if (IS_ERR(rt)) {
- show->state.status = PTR_ERR(rt);
- return NULL;
- }
+ int ret, align, sz, field_type;
+ struct btf_field_info tmp;
+ const struct btf_array *array;
+ u32 i, nelems = 1;
- /*
- * Is this toplevel object? If so, set total object size and
- * initialize pointers. Otherwise check if we still fall within
- * our safe object data.
+ /* Walk into array types to find the element type and the number of
+ * elements in the (flattened) array.
*/
- if (show->state.depth == 0) {
- show->obj.size = size;
- show->obj.head = data;
- } else {
- /*
- * If the size of the current object is > our remaining
- * safe buffer we _may_ need to do a new copy. However
- * consider the case of a nested struct; it's size pushes
- * us over the safe buffer limit, but showing any individual
- * struct members does not. In such cases, we don't need
- * to initiate a fresh copy yet; however we definitely need
- * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
- * in our buffer, regardless of the current object size.
- * The logic here is that as we resolve types we will
- * hit a base type at some point, and we need to be sure
- * the next chunk of data is safely available to display
- * that type info safely. We cannot rely on the size of
- * the current object here because it may be much larger
- * than our current buffer (e.g. task_struct is 8k).
- * All we want to do here is ensure that we can print the
- * next basic type, which we can if either
- * - the current type size is within the safe buffer; or
- * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
- * the safe buffer.
- */
- safe = __btf_show_obj_safe(show, data,
- min(size,
- BTF_SHOW_OBJ_BASE_TYPE_SIZE));
+ for (i = 0; i < MAX_RESOLVE_DEPTH && btf_type_is_array(var_type); i++) {
+ array = btf_array(var_type);
+ nelems *= array->nelems;
+ var_type = btf_type_by_id(btf, array->type);
}
+ if (i == MAX_RESOLVE_DEPTH)
+ return -E2BIG;
+ if (nelems == 0)
+ return 0;
- /*
- * We need a new copy to our safe object, either because we haven't
- * yet copied and are initializing safe data, or because the data
- * we want falls outside the boundaries of the safe object.
- */
- if (!safe) {
- size_left = btf_show_obj_size_left(show, data);
- if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
- size_left = BTF_SHOW_OBJ_SAFE_SIZE;
- show->state.status = copy_from_kernel_nofault(show->obj.safe,
- data, size_left);
- if (!show->state.status) {
- show->obj.data = data;
- safe = show->obj.safe;
- }
+ field_type = btf_get_field_type(btf, var_type,
+ field_mask, seen_mask, &align, &sz);
+ /* Look into variables of struct types */
+ if (!field_type && __btf_type_is_struct(var_type)) {
+ sz = var_type->size;
+ if (expected_size && expected_size != sz * nelems)
+ return 0;
+ ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
+ &info[0], info_cnt, level);
+ return ret;
}
- return safe;
-}
+ if (field_type == 0)
+ return 0;
+ if (field_type < 0)
+ return field_type;
-/*
- * Set the type we are starting to show and return a safe data pointer
- * to be used for showing the associated data.
- */
-static void *btf_show_start_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id, void *data)
-{
- show->state.type = t;
- show->state.type_id = type_id;
- show->state.name[0] = '\0';
+ if (expected_size && expected_size != sz * nelems)
+ return 0;
+ if (off % align)
+ return 0;
- return btf_show_obj_safe(show, t, data);
-}
+ switch (field_type) {
+ case BPF_SPIN_LOCK:
+ case BPF_RES_SPIN_LOCK:
+ case BPF_TIMER:
+ case BPF_WORKQUEUE:
+ case BPF_LIST_NODE:
+ case BPF_RB_NODE:
+ case BPF_REFCOUNT:
+ case BPF_TASK_WORK:
+ ret = btf_find_struct(btf, var_type, off, sz, field_type,
+ info_cnt ? &info[0] : &tmp);
+ if (ret < 0)
+ return ret;
+ break;
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ ret = btf_find_kptr(btf, var_type, off, sz,
+ info_cnt ? &info[0] : &tmp, field_mask);
+ if (ret < 0)
+ return ret;
+ break;
+ case BPF_LIST_HEAD:
+ case BPF_RB_ROOT:
+ ret = btf_find_graph_root(btf, var, var_type,
+ var_idx, off, sz,
+ info_cnt ? &info[0] : &tmp,
+ field_type);
+ if (ret < 0)
+ return ret;
+ break;
+ default:
+ return -EFAULT;
+ }
-static void btf_show_end_type(struct btf_show *show)
-{
- show->state.type = NULL;
- show->state.type_id = 0;
- show->state.name[0] = '\0';
+ if (ret == BTF_FIELD_IGNORE)
+ return 0;
+ if (!info_cnt)
+ return -E2BIG;
+ if (nelems > 1) {
+ ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
+ if (ret < 0)
+ return ret;
+ }
+ return nelems;
}
-static void *btf_show_start_aggr_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id, void *data)
+static int btf_find_struct_field(const struct btf *btf,
+ const struct btf_type *t, u32 field_mask,
+ struct btf_field_info *info, int info_cnt,
+ u32 level)
{
- void *safe_data = btf_show_start_type(show, t, type_id, data);
-
- if (!safe_data)
- return safe_data;
+ int ret, idx = 0;
+ const struct btf_member *member;
+ u32 i, off, seen_mask = 0;
- btf_show(show, "%s%s%s", btf_show_indent(show),
- btf_show_name(show),
- btf_show_newline(show));
- show->state.depth++;
- return safe_data;
-}
+ for_each_member(i, t, member) {
+ const struct btf_type *member_type = btf_type_by_id(btf,
+ member->type);
-static void btf_show_end_aggr_type(struct btf_show *show,
- const char *suffix)
-{
- show->state.depth--;
- btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
- btf_show_delim(show), btf_show_newline(show));
- btf_show_end_type(show);
-}
-
-static void btf_show_start_member(struct btf_show *show,
- const struct btf_member *m)
-{
- show->state.member = m;
-}
-
-static void btf_show_start_array_member(struct btf_show *show)
-{
- show->state.array_member = 1;
- btf_show_start_member(show, NULL);
-}
-
-static void btf_show_end_member(struct btf_show *show)
-{
- show->state.member = NULL;
-}
-
-static void btf_show_end_array_member(struct btf_show *show)
-{
- show->state.array_member = 0;
- btf_show_end_member(show);
-}
+ off = __btf_member_bit_offset(t, member);
+ if (off % 8)
+ /* valid C code cannot generate such BTF */
+ return -EINVAL;
+ off /= 8;
-static void *btf_show_start_array_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id,
- u16 array_encoding,
- void *data)
-{
- show->state.array_encoding = array_encoding;
- show->state.array_terminated = 0;
- return btf_show_start_aggr_type(show, t, type_id, data);
+ ret = btf_find_field_one(btf, t, member_type, i,
+ off, 0,
+ field_mask, &seen_mask,
+ &info[idx], info_cnt - idx, level);
+ if (ret < 0)
+ return ret;
+ idx += ret;
+ }
+ return idx;
}
-static void btf_show_end_array_type(struct btf_show *show)
+static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt, u32 level)
{
- show->state.array_encoding = 0;
- show->state.array_terminated = 0;
- btf_show_end_aggr_type(show, "]");
-}
+ int ret, idx = 0;
+ const struct btf_var_secinfo *vsi;
+ u32 i, off, seen_mask = 0;
-static void *btf_show_start_struct_type(struct btf_show *show,
- const struct btf_type *t,
- u32 type_id,
- void *data)
-{
- return btf_show_start_aggr_type(show, t, type_id, data);
-}
+ for_each_vsi(i, t, vsi) {
+ const struct btf_type *var = btf_type_by_id(btf, vsi->type);
+ const struct btf_type *var_type = btf_type_by_id(btf, var->type);
-static void btf_show_end_struct_type(struct btf_show *show)
-{
- btf_show_end_aggr_type(show, "}");
+ off = vsi->offset;
+ ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
+ field_mask, &seen_mask,
+ &info[idx], info_cnt - idx,
+ level);
+ if (ret < 0)
+ return ret;
+ idx += ret;
+ }
+ return idx;
}
-__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
- const char *fmt, ...)
+static int btf_find_field(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, struct btf_field_info *info,
+ int info_cnt)
{
- va_list args;
-
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ if (__btf_type_is_struct(t))
+ return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
+ else if (btf_type_is_datasec(t))
+ return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
+ return -EINVAL;
}
-__printf(2, 3) static void btf_verifier_log(struct btf_verifier_env *env,
- const char *fmt, ...)
+/* Callers have to ensure the life cycle of btf if it is program BTF */
+static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
+ struct module *mod = NULL;
+ const struct btf_type *t;
+ /* If a matching btf type is found in kernel or module BTFs, kptr_ref
+ * is that BTF, otherwise it's program BTF
+ */
+ struct btf *kptr_btf;
+ int ret;
+ s32 id;
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
-}
+ /* Find type in map BTF, and use it to look up the matching type
+ * in vmlinux or module BTFs, by name and kind.
+ */
+ t = btf_type_by_id(btf, info->kptr.type_id);
+ id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
+ &kptr_btf);
+ if (id == -ENOENT) {
+ /* btf_parse_kptr should only be called w/ btf = program BTF */
+ WARN_ON_ONCE(btf_is_kernel(btf));
-__printf(4, 5) static void __btf_verifier_log_type(struct btf_verifier_env *env,
- const struct btf_type *t,
- bool log_details,
- const char *fmt, ...)
-{
- struct bpf_verifier_log *log = &env->log;
- struct btf *btf = env->btf;
- va_list args;
+ /* Type exists only in program BTF. Assume that it's a MEM_ALLOC
+ * kptr allocated via bpf_obj_new
+ */
+ field->kptr.dtor = NULL;
+ id = info->kptr.type_id;
+ kptr_btf = (struct btf *)btf;
+ goto found_dtor;
+ }
+ if (id < 0)
+ return id;
- if (!bpf_verifier_log_needed(log))
- return;
+ /* Find and stash the function pointer for the destruction function that
+ * needs to be eventually invoked from the map free path.
+ */
+ if (info->type == BPF_KPTR_REF) {
+ const struct btf_type *dtor_func;
+ const char *dtor_func_name;
+ unsigned long addr;
+ s32 dtor_btf_id;
- if (log->level == BPF_LOG_KERNEL) {
- /* btf verifier prints all types it is processing via
- * btf_verifier_log_type(..., fmt = NULL).
- * Skip those prints for in-kernel BTF verification.
+ /* This call also serves as a whitelist of allowed objects that
+ * can be used as a referenced pointer and be stored in a map at
+ * the same time.
*/
- if (!fmt)
- return;
-
- /* Skip logging when loading module BTF with mismatches permitted */
- if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
- return;
- }
+ dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
+ if (dtor_btf_id < 0) {
+ ret = dtor_btf_id;
+ goto end_btf;
+ }
- __btf_verifier_log(log, "[%u] %s %s%s",
- env->log_type_id,
- btf_type_str(t),
- __btf_name_by_offset(btf, t->name_off),
- log_details ? " " : "");
+ dtor_func = btf_type_by_id(kptr_btf, dtor_btf_id);
+ if (!dtor_func) {
+ ret = -ENOENT;
+ goto end_btf;
+ }
- if (log_details)
- btf_type_ops(t)->log_details(env, t);
+ if (btf_is_module(kptr_btf)) {
+ mod = btf_try_get_module(kptr_btf);
+ if (!mod) {
+ ret = -ENXIO;
+ goto end_btf;
+ }
+ }
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ /* We already verified dtor_func to be btf_type_is_func
+ * in register_btf_id_dtor_kfuncs.
+ */
+ dtor_func_name = __btf_name_by_offset(kptr_btf, dtor_func->name_off);
+ addr = kallsyms_lookup_name(dtor_func_name);
+ if (!addr) {
+ ret = -EINVAL;
+ goto end_mod;
+ }
+ field->kptr.dtor = (void *)addr;
}
- __btf_verifier_log(log, "\n");
+found_dtor:
+ field->kptr.btf_id = id;
+ field->kptr.btf = kptr_btf;
+ field->kptr.module = mod;
+ return 0;
+end_mod:
+ module_put(mod);
+end_btf:
+ btf_put(kptr_btf);
+ return ret;
}
-#define btf_verifier_log_type(env, t, ...) \
- __btf_verifier_log_type((env), (t), true, __VA_ARGS__)
-#define btf_verifier_log_basic(env, t, ...) \
- __btf_verifier_log_type((env), (t), false, __VA_ARGS__)
-
-__printf(4, 5)
-static void btf_verifier_log_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const char *fmt, ...)
+static int btf_parse_graph_root(const struct btf *btf,
+ struct btf_field *field,
+ struct btf_field_info *info,
+ const char *node_type_name,
+ size_t node_type_align)
{
- struct bpf_verifier_log *log = &env->log;
- struct btf *btf = env->btf;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
-
- if (log->level == BPF_LOG_KERNEL) {
- if (!fmt)
- return;
-
- /* Skip logging when loading module BTF with mismatches permitted */
- if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
- return;
- }
+ const struct btf_type *t, *n = NULL;
+ const struct btf_member *member;
+ u32 offset;
+ int i;
- /* The CHECK_META phase already did a btf dump.
- *
- * If member is logged again, it must hit an error in
- * parsing this member. It is useful to print out which
- * struct this member belongs to.
+ t = btf_type_by_id(btf, info->graph_root.value_btf_id);
+ /* We've already checked that value_btf_id is a struct type. We
+ * just need to figure out the offset of the list_node, and
+ * verify its type.
*/
- if (env->phase != CHECK_META)
- btf_verifier_log_type(env, struct_type, NULL);
-
- if (btf_type_kflag(struct_type))
- __btf_verifier_log(log,
- "\t%s type_id=%u bitfield_size=%u bits_offset=%u",
- __btf_name_by_offset(btf, member->name_off),
- member->type,
- BTF_MEMBER_BITFIELD_SIZE(member->offset),
- BTF_MEMBER_BIT_OFFSET(member->offset));
- else
- __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
- __btf_name_by_offset(btf, member->name_off),
- member->type, member->offset);
+ for_each_member(i, t, member) {
+ if (strcmp(info->graph_root.node_name,
+ __btf_name_by_offset(btf, member->name_off)))
+ continue;
+ /* Invalid BTF, two members with same name */
+ if (n)
+ return -EINVAL;
+ n = btf_type_by_id(btf, member->type);
+ if (!__btf_type_is_struct(n))
+ return -EINVAL;
+ if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off)))
+ return -EINVAL;
+ offset = __btf_member_bit_offset(n, member);
+ if (offset % 8)
+ return -EINVAL;
+ offset /= 8;
+ if (offset % node_type_align)
+ return -EINVAL;
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
+ field->graph_root.btf = (struct btf *)btf;
+ field->graph_root.value_btf_id = info->graph_root.value_btf_id;
+ field->graph_root.node_offset = offset;
}
-
- __btf_verifier_log(log, "\n");
+ if (!n)
+ return -ENOENT;
+ return 0;
}
-__printf(4, 5)
-static void btf_verifier_log_vsi(struct btf_verifier_env *env,
- const struct btf_type *datasec_type,
- const struct btf_var_secinfo *vsi,
- const char *fmt, ...)
+static int btf_parse_list_head(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- va_list args;
-
- if (!bpf_verifier_log_needed(log))
- return;
- if (log->level == BPF_LOG_KERNEL && !fmt)
- return;
- if (env->phase != CHECK_META)
- btf_verifier_log_type(env, datasec_type, NULL);
-
- __btf_verifier_log(log, "\t type_id=%u offset=%u size=%u",
- vsi->type, vsi->offset, vsi->size);
- if (fmt && *fmt) {
- __btf_verifier_log(log, " ");
- va_start(args, fmt);
- bpf_verifier_vlog(log, fmt, args);
- va_end(args);
- }
-
- __btf_verifier_log(log, "\n");
+ return btf_parse_graph_root(btf, field, info, "bpf_list_node",
+ __alignof__(struct bpf_list_node));
}
-static void btf_verifier_log_hdr(struct btf_verifier_env *env,
- u32 btf_data_size)
+static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
+ struct btf_field_info *info)
{
- struct bpf_verifier_log *log = &env->log;
- const struct btf *btf = env->btf;
- const struct btf_header *hdr;
-
- if (!bpf_verifier_log_needed(log))
- return;
-
- if (log->level == BPF_LOG_KERNEL)
- return;
- hdr = &btf->hdr;
- __btf_verifier_log(log, "magic: 0x%x\n", hdr->magic);
- __btf_verifier_log(log, "version: %u\n", hdr->version);
- __btf_verifier_log(log, "flags: 0x%x\n", hdr->flags);
- __btf_verifier_log(log, "hdr_len: %u\n", hdr->hdr_len);
- __btf_verifier_log(log, "type_off: %u\n", hdr->type_off);
- __btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
- __btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
- __btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
- __btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
+ return btf_parse_graph_root(btf, field, info, "bpf_rb_node",
+ __alignof__(struct bpf_rb_node));
}
-static int btf_add_type(struct btf_verifier_env *env, struct btf_type *t)
+static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
{
- struct btf *btf = env->btf;
-
- if (btf->types_size == btf->nr_types) {
- /* Expand 'types' array */
-
- struct btf_type **new_types;
- u32 expand_by, new_size;
-
- if (btf->start_id + btf->types_size == BTF_MAX_TYPE) {
- btf_verifier_log(env, "Exceeded max num of types");
- return -E2BIG;
- }
-
- expand_by = max_t(u32, btf->types_size >> 2, 16);
- new_size = min_t(u32, BTF_MAX_TYPE,
- btf->types_size + expand_by);
-
- new_types = kvzalloc_objs(*new_types, new_size,
- GFP_KERNEL | __GFP_NOWARN);
- if (!new_types)
- return -ENOMEM;
-
- if (btf->nr_types == 0) {
- if (!btf->base_btf) {
- /* lazily init VOID type */
- new_types[0] = &btf_void;
- btf->nr_types++;
- }
- } else {
- memcpy(new_types, btf->types,
- sizeof(*btf->types) * btf->nr_types);
- }
-
- kvfree(btf->types);
- btf->types = new_types;
- btf->types_size = new_size;
- }
-
- btf->types[btf->nr_types++] = t;
+ const struct btf_field *a = (const struct btf_field *)_a;
+ const struct btf_field *b = (const struct btf_field *)_b;
+ if (a->offset < b->offset)
+ return -1;
+ else if (a->offset > b->offset)
+ return 1;
return 0;
}
-static int btf_alloc_id(struct btf *btf)
+struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
+ u32 field_mask, u32 value_size)
{
- int id;
+ struct btf_field_info info_arr[BTF_FIELDS_MAX];
+ u32 next_off = 0, field_type_size;
+ struct btf_record *rec;
+ int ret, i, cnt;
- idr_preload(GFP_KERNEL);
- spin_lock_bh(&btf_idr_lock);
- id = idr_alloc_cyclic(&btf_idr, btf, 1, INT_MAX, GFP_ATOMIC);
- if (id > 0)
- btf->id = id;
- spin_unlock_bh(&btf_idr_lock);
- idr_preload_end();
+ ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
+ if (ret < 0)
+ return ERR_PTR(ret);
+ if (!ret)
+ return NULL;
- if (WARN_ON_ONCE(!id))
- return -ENOSPC;
-
- return id > 0 ? 0 : id;
-}
-
-static void btf_free_id(struct btf *btf)
-{
- unsigned long flags;
-
- /*
- * In map-in-map, calling map_delete_elem() on outer
- * map will call bpf_map_put on the inner map.
- * It will then eventually call btf_free_id()
- * on the inner map. Some of the map_delete_elem()
- * implementation may have irq disabled, so
- * we need to use the _irqsave() version instead
- * of the _bh() version.
- */
- spin_lock_irqsave(&btf_idr_lock, flags);
- if (btf->id) {
- idr_remove(&btf_idr, btf->id);
- /*
- * Clear the id here to make this function idempotent, since it will get
- * called a couple of times for module BTFs: on module unload, and then
- * the final btf_put(). btf_alloc_id() starts IDs with 1, so we can use
- * 0 as sentinel value.
- */
- WRITE_ONCE(btf->id, 0);
- }
- spin_unlock_irqrestore(&btf_idr_lock, flags);
-}
-
-static void btf_free_kfunc_set_tab(struct btf *btf)
-{
- struct btf_kfunc_set_tab *tab = btf->kfunc_set_tab;
- int hook;
-
- if (!tab)
- return;
- for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
- kfree(tab->sets[hook]);
- kfree(tab);
- btf->kfunc_set_tab = NULL;
-}
-
-static void btf_free_dtor_kfunc_tab(struct btf *btf)
-{
- struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
-
- if (!tab)
- return;
- kfree(tab);
- btf->dtor_kfunc_tab = NULL;
-}
-
-static void btf_struct_metas_free(struct btf_struct_metas *tab)
-{
- int i;
-
- if (!tab)
- return;
- for (i = 0; i < tab->cnt; i++)
- btf_record_free(tab->types[i].record);
- kfree(tab);
-}
-
-static void btf_free_struct_meta_tab(struct btf *btf)
-{
- struct btf_struct_metas *tab = btf->struct_meta_tab;
-
- btf_struct_metas_free(tab);
- btf->struct_meta_tab = NULL;
-}
-
-static void btf_free_struct_ops_tab(struct btf *btf)
-{
- struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
- u32 i;
-
- if (!tab)
- return;
-
- for (i = 0; i < tab->cnt; i++)
- bpf_struct_ops_desc_release(&tab->ops[i]);
-
- kfree(tab);
- btf->struct_ops_tab = NULL;
-}
-
-static void btf_free(struct btf *btf)
-{
- btf_free_struct_meta_tab(btf);
- btf_free_dtor_kfunc_tab(btf);
- btf_free_kfunc_set_tab(btf);
- btf_free_struct_ops_tab(btf);
- kvfree(btf->types);
- kvfree(btf->resolved_sizes);
- kvfree(btf->resolved_ids);
- /* vmlinux does not allocate btf->data, it simply points it at
- * __start_BTF.
+ cnt = ret;
+ /* This needs to be kzalloc to zero out padding and unused fields, see
+ * comment in btf_record_equal.
*/
- if (!btf_is_vmlinux(btf))
- kvfree(btf->data);
- kvfree(btf->base_id_map);
- kfree(btf);
-}
-
-static void btf_free_rcu(struct rcu_head *rcu)
-{
- struct btf *btf = container_of(rcu, struct btf, rcu);
-
- btf_free(btf);
-}
-
-const char *btf_get_name(const struct btf *btf)
-{
- return btf->name;
-}
-
-void btf_get(struct btf *btf)
-{
- refcount_inc(&btf->refcnt);
-}
-
-void btf_put(struct btf *btf)
-{
- if (btf && refcount_dec_and_test(&btf->refcnt)) {
- btf_free_id(btf);
- call_rcu(&btf->rcu, btf_free_rcu);
- }
-}
-
-struct btf *btf_base_btf(const struct btf *btf)
-{
- return btf->base_btf;
-}
-
-const struct btf_header *btf_header(const struct btf *btf)
-{
- return &btf->hdr;
-}
-
-void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
-{
- btf->base_btf = (struct btf *)base_btf;
- btf->start_id = btf_nr_types(base_btf);
- btf->start_str_off = base_btf->hdr.str_len;
-}
-
-static int env_resolve_init(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- u32 nr_types = btf->nr_types;
- u32 *resolved_sizes = NULL;
- u32 *resolved_ids = NULL;
- u8 *visit_states = NULL;
-
- resolved_sizes = kvcalloc(nr_types, sizeof(*resolved_sizes),
- GFP_KERNEL | __GFP_NOWARN);
- if (!resolved_sizes)
- goto nomem;
-
- resolved_ids = kvcalloc(nr_types, sizeof(*resolved_ids),
- GFP_KERNEL | __GFP_NOWARN);
- if (!resolved_ids)
- goto nomem;
-
- visit_states = kvcalloc(nr_types, sizeof(*visit_states),
- GFP_KERNEL | __GFP_NOWARN);
- if (!visit_states)
- goto nomem;
-
- btf->resolved_sizes = resolved_sizes;
- btf->resolved_ids = resolved_ids;
- env->visit_states = visit_states;
-
- return 0;
-
-nomem:
- kvfree(resolved_sizes);
- kvfree(resolved_ids);
- kvfree(visit_states);
- return -ENOMEM;
-}
-
-static void btf_verifier_env_free(struct btf_verifier_env *env)
-{
- kvfree(env->visit_states);
- kfree(env);
-}
-
-static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
- const struct btf_type *next_type)
-{
- switch (env->resolve_mode) {
- case RESOLVE_TBD:
- /* int, enum or void is a sink */
- return !btf_type_needs_resolve(next_type);
- case RESOLVE_PTR:
- /* int, enum, void, struct, array, func or func_proto is a sink
- * for ptr
- */
- return !btf_type_is_modifier(next_type) &&
- !btf_type_is_ptr(next_type);
- case RESOLVE_STRUCT_OR_ARRAY:
- /* int, enum, void, ptr, func or func_proto is a sink
- * for struct and array
- */
- return !btf_type_is_modifier(next_type) &&
- !btf_type_is_array(next_type) &&
- !btf_type_is_struct(next_type);
- default:
- BUG();
- }
-}
-
-static bool env_type_is_resolved(const struct btf_verifier_env *env,
- u32 type_id)
-{
- /* base BTF types should be resolved by now */
- if (type_id < env->btf->start_id)
- return true;
-
- return env->visit_states[type_id - env->btf->start_id] == RESOLVED;
-}
-
-static int env_stack_push(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id)
-{
- const struct btf *btf = env->btf;
- struct resolve_vertex *v;
-
- if (env->top_stack == MAX_RESOLVE_DEPTH)
- return -E2BIG;
-
- if (type_id < btf->start_id
- || env->visit_states[type_id - btf->start_id] != NOT_VISITED)
- return -EEXIST;
-
- env->visit_states[type_id - btf->start_id] = VISITED;
-
- v = &env->stack[env->top_stack++];
- v->t = t;
- v->type_id = type_id;
- v->next_member = 0;
-
- if (env->resolve_mode == RESOLVE_TBD) {
- if (btf_type_is_ptr(t))
- env->resolve_mode = RESOLVE_PTR;
- else if (btf_type_is_struct(t) || btf_type_is_array(t))
- env->resolve_mode = RESOLVE_STRUCT_OR_ARRAY;
- }
-
- return 0;
-}
-
-static void env_stack_set_next_member(struct btf_verifier_env *env,
- u16 next_member)
-{
- env->stack[env->top_stack - 1].next_member = next_member;
-}
-
-static void env_stack_pop_resolved(struct btf_verifier_env *env,
- u32 resolved_type_id,
- u32 resolved_size)
-{
- u32 type_id = env->stack[--(env->top_stack)].type_id;
- struct btf *btf = env->btf;
+ rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!rec)
+ return ERR_PTR(-ENOMEM);
- type_id -= btf->start_id; /* adjust to local type id */
- btf->resolved_sizes[type_id] = resolved_size;
- btf->resolved_ids[type_id] = resolved_type_id;
- env->visit_states[type_id] = RESOLVED;
-}
+ rec->spin_lock_off = -EINVAL;
+ rec->res_spin_lock_off = -EINVAL;
+ rec->timer_off = -EINVAL;
+ rec->wq_off = -EINVAL;
+ rec->refcount_off = -EINVAL;
+ rec->task_work_off = -EINVAL;
+ for (i = 0; i < cnt; i++) {
+ field_type_size = btf_field_type_size(info_arr[i].type);
+ if (info_arr[i].off + field_type_size > value_size) {
+ WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size);
+ ret = -EFAULT;
+ goto end;
+ }
+ if (info_arr[i].off < next_off) {
+ ret = -EEXIST;
+ goto end;
+ }
+ next_off = info_arr[i].off + field_type_size;
-static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
-{
- return env->top_stack ? &env->stack[env->top_stack - 1] : NULL;
-}
+ rec->field_mask |= info_arr[i].type;
+ rec->fields[i].offset = info_arr[i].off;
+ rec->fields[i].type = info_arr[i].type;
+ rec->fields[i].size = field_type_size;
-/* Resolve the size of a passed-in "type"
- *
- * type: is an array (e.g. u32 array[x][y])
- * return type: type "u32[x][y]", i.e. BTF_KIND_ARRAY,
- * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
- * corresponds to the return type.
- * *elem_type: u32
- * *elem_id: id of u32
- * *total_nelems: (x * y). Hence, individual elem size is
- * (*type_size / *total_nelems)
- * *type_id: id of type if it's changed within the function, 0 if not
- *
- * type: is not an array (e.g. const struct X)
- * return type: type "struct X"
- * *type_size: sizeof(struct X)
- * *elem_type: same as return type ("struct X")
- * *elem_id: 0
- * *total_nelems: 1
- * *type_id: id of type if it's changed within the function, 0 if not
- */
-static const struct btf_type *
-__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
- u32 *type_size, const struct btf_type **elem_type,
- u32 *elem_id, u32 *total_nelems, u32 *type_id)
-{
- const struct btf_type *array_type = NULL;
- const struct btf_array *array = NULL;
- u32 i, size, nelems = 1, id = 0;
-
- for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
- switch (BTF_INFO_KIND(type->info)) {
- /* type->size can be used */
- case BTF_KIND_INT:
- case BTF_KIND_STRUCT:
- case BTF_KIND_UNION:
- case BTF_KIND_ENUM:
- case BTF_KIND_FLOAT:
- case BTF_KIND_ENUM64:
- size = type->size;
- goto resolved;
-
- case BTF_KIND_PTR:
- size = sizeof(void *);
- goto resolved;
-
- /* Modifiers */
- case BTF_KIND_TYPEDEF:
- case BTF_KIND_VOLATILE:
- case BTF_KIND_CONST:
- case BTF_KIND_RESTRICT:
- case BTF_KIND_TYPE_TAG:
- id = type->type;
- type = btf_type_by_id(btf, type->type);
+ switch (info_arr[i].type) {
+ case BPF_SPIN_LOCK:
+ WARN_ON_ONCE(rec->spin_lock_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->spin_lock_off = rec->fields[i].offset;
break;
-
- case BTF_KIND_ARRAY:
- if (!array_type)
- array_type = type;
- array = btf_type_array(type);
- if (nelems && array->nelems > U32_MAX / nelems)
- return ERR_PTR(-EINVAL);
- nelems *= array->nelems;
- type = btf_type_by_id(btf, array->type);
+ case BPF_RES_SPIN_LOCK:
+ WARN_ON_ONCE(rec->spin_lock_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->res_spin_lock_off = rec->fields[i].offset;
+ break;
+ case BPF_TIMER:
+ WARN_ON_ONCE(rec->timer_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->timer_off = rec->fields[i].offset;
+ break;
+ case BPF_WORKQUEUE:
+ WARN_ON_ONCE(rec->wq_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->wq_off = rec->fields[i].offset;
+ break;
+ case BPF_TASK_WORK:
+ WARN_ON_ONCE(rec->task_work_off >= 0);
+ rec->task_work_off = rec->fields[i].offset;
+ break;
+ case BPF_REFCOUNT:
+ WARN_ON_ONCE(rec->refcount_off >= 0);
+ /* Cache offset for faster lookup at runtime */
+ rec->refcount_off = rec->fields[i].offset;
+ break;
+ case BPF_KPTR_UNREF:
+ case BPF_KPTR_REF:
+ case BPF_KPTR_PERCPU:
+ case BPF_UPTR:
+ ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_LIST_HEAD:
+ ret = btf_parse_list_head(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_RB_ROOT:
+ ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]);
+ if (ret < 0)
+ goto end;
+ break;
+ case BPF_LIST_NODE:
+ case BPF_RB_NODE:
break;
-
- /* type without size */
default:
- return ERR_PTR(-EINVAL);
+ ret = -EFAULT;
+ goto end;
}
+ rec->cnt++;
}
- return ERR_PTR(-EINVAL);
-
-resolved:
- if (nelems && size > U32_MAX / nelems)
- return ERR_PTR(-EINVAL);
-
- *type_size = nelems * size;
- if (total_nelems)
- *total_nelems = nelems;
- if (elem_type)
- *elem_type = type;
- if (elem_id)
- *elem_id = array ? array->type : 0;
- if (type_id && id)
- *type_id = id;
-
- return array_type ? : type;
-}
-
-const struct btf_type *
-btf_resolve_size(const struct btf *btf, const struct btf_type *type,
- u32 *type_size)
-{
- return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
-}
-
-static u32 btf_resolved_type_id(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
-
- return btf->resolved_ids[type_id - btf->start_id];
-}
-
-/* The input param "type_id" must point to a needs_resolve type */
-static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
- u32 *type_id)
-{
- *type_id = btf_resolved_type_id(btf, *type_id);
- return btf_type_by_id(btf, *type_id);
-}
-
-static u32 btf_resolved_type_size(const struct btf *btf, u32 type_id)
-{
- while (type_id < btf->start_id)
- btf = btf->base_btf;
-
- return btf->resolved_sizes[type_id - btf->start_id];
-}
-
-const struct btf_type *btf_type_id_size(const struct btf *btf,
- u32 *type_id, u32 *ret_size)
-{
- const struct btf_type *size_type;
- u32 size_type_id = *type_id;
- u32 size = 0;
-
- size_type = btf_type_by_id(btf, size_type_id);
- if (btf_type_nosize_or_null(size_type))
- return NULL;
-
- if (btf_type_has_size(size_type)) {
- size = size_type->size;
- } else if (btf_type_is_array(size_type)) {
- size = btf_resolved_type_size(btf, size_type_id);
- } else if (btf_type_is_ptr(size_type)) {
- size = sizeof(void *);
- } else {
- if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) &&
- !btf_type_is_var(size_type)))
- return NULL;
-
- size_type_id = btf_resolved_type_id(btf, size_type_id);
- size_type = btf_type_by_id(btf, size_type_id);
- if (btf_type_nosize_or_null(size_type))
- return NULL;
- else if (btf_type_has_size(size_type))
- size = size_type->size;
- else if (btf_type_is_array(size_type))
- size = btf_resolved_type_size(btf, size_type_id);
- else if (btf_type_is_ptr(size_type))
- size = sizeof(void *);
- else
- return NULL;
- }
-
- *type_id = size_type_id;
- if (ret_size)
- *ret_size = size;
-
- return size_type;
-}
-
-static int btf_df_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- btf_verifier_log_basic(env, struct_type,
- "Unsupported check_member");
- return -EINVAL;
-}
-
-static int btf_df_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- btf_verifier_log_basic(env, struct_type,
- "Unsupported check_kflag_member");
- return -EINVAL;
-}
-
-/* Used for ptr, array struct/union and float type members.
- * int, enum and modifier types have their specific callback functions.
- */
-static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- /* bitfield size is 0, so member->offset represents bit offset only.
- * It is safe to call non kflag check_member variants.
- */
- return btf_type_ops(member_type)->check_member(env, struct_type,
- member,
- member_type);
-}
-
-static int btf_df_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- btf_verifier_log_basic(env, v->t, "Unsupported resolve");
- return -EINVAL;
-}
-
-static void btf_df_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offsets,
- struct btf_show *show)
-{
- btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
-}
-
-static int btf_int_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 int_data = btf_type_int(member_type);
- u32 struct_bits_off = member->offset;
- u32 struct_size = struct_type->size;
- u32 nr_copy_bits;
- u32 bytes_offset;
-
- if (U32_MAX - struct_bits_off < BTF_INT_OFFSET(int_data)) {
- btf_verifier_log_member(env, struct_type, member,
- "bits_offset exceeds U32_MAX");
- return -EINVAL;
- }
-
- struct_bits_off += BTF_INT_OFFSET(int_data);
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- nr_copy_bits = BTF_INT_BITS(int_data) +
- BITS_PER_BYTE_MASKED(struct_bits_off);
-
- if (nr_copy_bits > BITS_PER_U128) {
- btf_verifier_log_member(env, struct_type, member,
- "nr_copy_bits exceeds 128");
- return -EINVAL;
- }
-
- if (struct_size < bytes_offset ||
- struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_int_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
- u32 int_data = btf_type_int(member_type);
- u32 struct_size = struct_type->size;
- u32 nr_copy_bits;
-
- /* a regular int type is required for the kflag int member */
- if (!btf_type_int_is_regular(member_type)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member base type");
- return -EINVAL;
- }
-
- /* check sanity of bitfield size */
- nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
- struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
- nr_int_data_bits = BTF_INT_BITS(int_data);
- if (!nr_bits) {
- /* Not a bitfield member, member offset must be at byte
- * boundary.
- */
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member offset");
- return -EINVAL;
- }
-
- nr_bits = nr_int_data_bits;
- } else if (nr_bits > nr_int_data_bits) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
- if (nr_copy_bits > BITS_PER_U128) {
- btf_verifier_log_member(env, struct_type, member,
- "nr_copy_bits exceeds 128");
- return -EINVAL;
- }
-
- if (struct_size < bytes_offset ||
- struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_int_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 int_data, nr_bits, meta_needed = sizeof(int_data);
- u16 encoding;
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- int_data = btf_type_int(t);
- if (int_data & ~BTF_INT_MASK) {
- btf_verifier_log_basic(env, t, "Invalid int_data:%x",
- int_data);
- return -EINVAL;
- }
-
- nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
-
- if (nr_bits > BITS_PER_U128) {
- btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
- BITS_PER_U128);
- return -EINVAL;
- }
-
- if (BITS_ROUNDUP_BYTES(nr_bits) > t->size) {
- btf_verifier_log_type(env, t, "nr_bits exceeds type_size");
- return -EINVAL;
- }
-
- /*
- * Only one of the encoding bits is allowed and it
- * should be sufficient for the pretty print purpose (i.e. decoding).
- * Multiple bits can be allowed later if it is found
- * to be insufficient.
- */
- encoding = BTF_INT_ENCODING(int_data);
- if (encoding &&
- encoding != BTF_INT_SIGNED &&
- encoding != BTF_INT_CHAR &&
- encoding != BTF_INT_BOOL) {
- btf_verifier_log_type(env, t, "Unsupported encoding");
- return -ENOTSUPP;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_int_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- int int_data = btf_type_int(t);
-
- btf_verifier_log(env,
- "size=%u bits_offset=%u nr_bits=%u encoding=%s",
- t->size, BTF_INT_OFFSET(int_data),
- BTF_INT_BITS(int_data),
- btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
-}
-
-static void btf_int128_print(struct btf_show *show, void *data)
-{
- /* data points to a __int128 number.
- * Suppose
- * int128_num = *(__int128 *)data;
- * The below formulas shows what upper_num and lower_num represents:
- * upper_num = int128_num >> 64;
- * lower_num = int128_num & 0xffffffffFFFFFFFFULL;
- */
- u64 upper_num, lower_num;
-
-#ifdef __BIG_ENDIAN_BITFIELD
- upper_num = *(u64 *)data;
- lower_num = *(u64 *)(data + 8);
-#else
- upper_num = *(u64 *)(data + 8);
- lower_num = *(u64 *)data;
-#endif
- if (upper_num == 0)
- btf_show_type_value(show, "0x%llx", lower_num);
- else
- btf_show_type_values(show, "0x%llx%016llx", upper_num,
- lower_num);
-}
-
-static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
- u16 right_shift_bits)
-{
- u64 upper_num, lower_num;
-
-#ifdef __BIG_ENDIAN_BITFIELD
- upper_num = print_num[0];
- lower_num = print_num[1];
-#else
- upper_num = print_num[1];
- lower_num = print_num[0];
-#endif
-
- /* shake out un-needed bits by shift/or operations */
- if (left_shift_bits >= 64) {
- upper_num = lower_num << (left_shift_bits - 64);
- lower_num = 0;
- } else {
- upper_num = (upper_num << left_shift_bits) |
- (lower_num >> (64 - left_shift_bits));
- lower_num = lower_num << left_shift_bits;
- }
-
- if (right_shift_bits >= 64) {
- lower_num = upper_num >> (right_shift_bits - 64);
- upper_num = 0;
- } else {
- lower_num = (lower_num >> right_shift_bits) |
- (upper_num << (64 - right_shift_bits));
- upper_num = upper_num >> right_shift_bits;
- }
-
-#ifdef __BIG_ENDIAN_BITFIELD
- print_num[0] = upper_num;
- print_num[1] = lower_num;
-#else
- print_num[0] = lower_num;
- print_num[1] = upper_num;
-#endif
-}
-
-static void btf_bitfield_show(void *data, u8 bits_offset,
- u8 nr_bits, struct btf_show *show)
-{
- u16 left_shift_bits, right_shift_bits;
- u8 nr_copy_bytes;
- u8 nr_copy_bits;
- u64 print_num[2] = {};
-
- nr_copy_bits = nr_bits + bits_offset;
- nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
-
- memcpy(print_num, data, nr_copy_bytes);
-
-#ifdef __BIG_ENDIAN_BITFIELD
- left_shift_bits = bits_offset;
-#else
- left_shift_bits = BITS_PER_U128 - nr_copy_bits;
-#endif
- right_shift_bits = BITS_PER_U128 - nr_bits;
-
- btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
- btf_int128_print(show, print_num);
-}
-
-
-static void btf_int_bits_show(const struct btf *btf,
- const struct btf_type *t,
- void *data, u8 bits_offset,
- struct btf_show *show)
-{
- u32 int_data = btf_type_int(t);
- u8 nr_bits = BTF_INT_BITS(int_data);
- u8 total_bits_offset;
-
- /*
- * bits_offset is at most 7.
- * BTF_INT_OFFSET() cannot exceed 128 bits.
- */
- total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
- data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
- bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
- btf_bitfield_show(data, bits_offset, nr_bits, show);
-}
-
-static void btf_int_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- u32 int_data = btf_type_int(t);
- u8 encoding = BTF_INT_ENCODING(int_data);
- bool sign = encoding & BTF_INT_SIGNED;
- u8 nr_bits = BTF_INT_BITS(int_data);
- void *safe_data;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- if (bits_offset || BTF_INT_OFFSET(int_data) ||
- BITS_PER_BYTE_MASKED(nr_bits)) {
- btf_int_bits_show(btf, t, safe_data, bits_offset, show);
- goto out;
- }
-
- switch (nr_bits) {
- case 128:
- btf_int128_print(show, safe_data);
- break;
- case 64:
- if (sign)
- btf_show_type_value(show, "%lld", *(s64 *)safe_data);
- else
- btf_show_type_value(show, "%llu", *(u64 *)safe_data);
- break;
- case 32:
- if (sign)
- btf_show_type_value(show, "%d", *(s32 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u32 *)safe_data);
- break;
- case 16:
- if (sign)
- btf_show_type_value(show, "%d", *(s16 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u16 *)safe_data);
- break;
- case 8:
- if (show->state.array_encoding == BTF_INT_CHAR) {
- /* check for null terminator */
- if (show->state.array_terminated)
- break;
- if (*(char *)data == '\0') {
- show->state.array_terminated = 1;
- break;
- }
- if (isprint(*(char *)data)) {
- btf_show_type_value(show, "'%c'",
- *(char *)safe_data);
- break;
- }
- }
- if (sign)
- btf_show_type_value(show, "%d", *(s8 *)safe_data);
- else
- btf_show_type_value(show, "%u", *(u8 *)safe_data);
- break;
- default:
- btf_int_bits_show(btf, t, safe_data, bits_offset, show);
- break;
- }
-out:
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations int_ops = {
- .check_meta = btf_int_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_int_check_member,
- .check_kflag_member = btf_int_check_kflag_member,
- .log_details = btf_int_log,
- .show = btf_int_show,
-};
-
-static int btf_modifier_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- const struct btf_type *resolved_type;
- u32 resolved_type_id = member->type;
- struct btf_member resolved_member;
- struct btf *btf = env->btf;
-
- resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
- if (!resolved_type) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member");
- return -EINVAL;
- }
-
- resolved_member = *member;
- resolved_member.type = resolved_type_id;
-
- return btf_type_ops(resolved_type)->check_member(env, struct_type,
- &resolved_member,
- resolved_type);
-}
-
-static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- const struct btf_type *resolved_type;
- u32 resolved_type_id = member->type;
- struct btf_member resolved_member;
- struct btf *btf = env->btf;
-
- resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
- if (!resolved_type) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member");
- return -EINVAL;
- }
-
- resolved_member = *member;
- resolved_member.type = resolved_type_id;
-
- return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
- &resolved_member,
- resolved_type);
-}
-
-static int btf_ptr_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_size, struct_bits_off, bytes_offset;
-
- struct_size = struct_type->size;
- struct_bits_off = member->offset;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- if (struct_size - bytes_offset < sizeof(void *)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_ref_type_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const char *value;
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t) && !btf_type_is_type_tag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!BTF_TYPE_ID_VALID(t->type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- /* typedef/type_tag type must have a valid name, and other ref types,
- * volatile, const, restrict, should have a null name.
- */
- if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- } else if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPE_TAG) {
- value = btf_name_by_offset(env->btf, t->name_off);
- if (!value || !value[0]) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- } else {
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_modifier_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *t = v->t;
- const struct btf_type *next_type;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- /* Figure out the resolved next_type_id with size.
- * They will be stored in the current modifier's
- * resolved_ids and resolved_sizes such that it can
- * save us a few type-following when we use it later (e.g. in
- * pretty print).
- */
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- if (env_type_is_resolved(env, next_type_id))
- next_type = btf_type_id_resolve(btf, &next_type_id);
-
- /* "typedef void new_void", "const void"...etc */
- if (!btf_type_is_void(next_type) &&
- !btf_type_is_fwd(next_type) &&
- !btf_type_is_func_proto(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static int btf_var_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- if (btf_type_is_modifier(next_type)) {
- const struct btf_type *resolved_type;
- u32 resolved_type_id;
-
- resolved_type_id = next_type_id;
- resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
-
- if (btf_type_is_ptr(resolved_type) &&
- !env_type_is_resolve_sink(env, resolved_type) &&
- !env_type_is_resolved(env, resolved_type_id))
- return env_stack_push(env, resolved_type,
- resolved_type_id);
- }
-
- /* We must resolve to something concrete at this point, no
- * forward types or similar that would resolve to size of
- * zero is allowed.
- */
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static int btf_ptr_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || btf_type_is_resolve_source_only(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- /* If the modifier was RESOLVED during RESOLVE_STRUCT_OR_ARRAY,
- * the modifier may have stopped resolving when it was resolved
- * to a ptr (last-resolved-ptr).
- *
- * We now need to continue from the last-resolved-ptr to
- * ensure the last-resolved-ptr will not referring back to
- * the current ptr (t).
- */
- if (btf_type_is_modifier(next_type)) {
- const struct btf_type *resolved_type;
- u32 resolved_type_id;
-
- resolved_type_id = next_type_id;
- resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
-
- if (btf_type_is_ptr(resolved_type) &&
- !env_type_is_resolve_sink(env, resolved_type) &&
- !env_type_is_resolved(env, resolved_type_id))
- return env_stack_push(env, resolved_type,
- resolved_type_id);
- }
-
- if (!btf_type_id_size(btf, &next_type_id, NULL)) {
- if (env_type_is_resolved(env, next_type_id))
- next_type = btf_type_id_resolve(btf, &next_type_id);
-
- if (!btf_type_is_void(next_type) &&
- !btf_type_is_fwd(next_type) &&
- !btf_type_is_func_proto(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static void btf_modifier_show(const struct btf *btf,
- const struct btf_type *t,
- u32 type_id, void *data,
- u8 bits_offset, struct btf_show *show)
-{
- if (btf->resolved_ids)
- t = btf_type_id_resolve(btf, &type_id);
- else
- t = btf_type_skip_modifiers(btf, type_id, NULL);
-
- btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
-}
-
-static void btf_var_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- t = btf_type_id_resolve(btf, &type_id);
-
- btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
-}
-
-static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- void *safe_data;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
- if (show->flags & BTF_SHOW_PTR_RAW)
- btf_show_type_value(show, "0x%px", *(void **)safe_data);
- else
- btf_show_type_value(show, "0x%p", *(void **)safe_data);
- btf_show_end_type(show);
-}
-
-static void btf_ref_type_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "type_id=%u", t->type);
-}
-
-static const struct btf_kind_operations modifier_ops = {
- .check_meta = btf_ref_type_check_meta,
- .resolve = btf_modifier_resolve,
- .check_member = btf_modifier_check_member,
- .check_kflag_member = btf_modifier_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_modifier_show,
-};
-
-static const struct btf_kind_operations ptr_ops = {
- .check_meta = btf_ref_type_check_meta,
- .resolve = btf_ptr_resolve,
- .check_member = btf_ptr_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_ptr_show,
-};
-
-static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (t->type) {
- btf_verifier_log_type(env, t, "type != 0");
- return -EINVAL;
- }
-
- /* fwd type must have a valid name */
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static void btf_fwd_type_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "%s", btf_type_kflag(t) ? "union" : "struct");
-}
-
-static const struct btf_kind_operations fwd_ops = {
- .check_meta = btf_fwd_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_fwd_type_log,
- .show = btf_df_show,
-};
-
-static int btf_array_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
- u32 array_type_id, array_size;
- struct btf *btf = env->btf;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- array_type_id = member->type;
- btf_type_id_size(btf, &array_type_id, &array_size);
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < array_size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_array_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_array *array = btf_type_array(t);
- u32 meta_needed = sizeof(*array);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- /* array type should not have a name */
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (t->size) {
- btf_verifier_log_type(env, t, "size != 0");
- return -EINVAL;
- }
-
- /* Array elem type and index type cannot be in type void,
- * so !array->type and !array->index_type are not allowed.
- */
- if (!array->type || !BTF_TYPE_ID_VALID(array->type)) {
- btf_verifier_log_type(env, t, "Invalid elem");
- return -EINVAL;
- }
-
- if (!array->index_type || !BTF_TYPE_ID_VALID(array->index_type)) {
- btf_verifier_log_type(env, t, "Invalid index");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static int btf_array_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_array *array = btf_type_array(v->t);
- const struct btf_type *elem_type, *index_type;
- u32 elem_type_id, index_type_id;
- struct btf *btf = env->btf;
- u32 elem_size;
-
- /* Check array->index_type */
- index_type_id = array->index_type;
- index_type = btf_type_by_id(btf, index_type_id);
- if (btf_type_nosize_or_null(index_type) ||
- btf_type_is_resolve_source_only(index_type)) {
- btf_verifier_log_type(env, v->t, "Invalid index");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, index_type) &&
- !env_type_is_resolved(env, index_type_id))
- return env_stack_push(env, index_type, index_type_id);
-
- index_type = btf_type_id_size(btf, &index_type_id, NULL);
- if (!index_type || !btf_type_is_int(index_type) ||
- !btf_type_int_is_regular(index_type)) {
- btf_verifier_log_type(env, v->t, "Invalid index");
- return -EINVAL;
- }
-
- /* Check array->type */
- elem_type_id = array->type;
- elem_type = btf_type_by_id(btf, elem_type_id);
- if (btf_type_nosize_or_null(elem_type) ||
- btf_type_is_resolve_source_only(elem_type)) {
- btf_verifier_log_type(env, v->t,
- "Invalid elem");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, elem_type) &&
- !env_type_is_resolved(env, elem_type_id))
- return env_stack_push(env, elem_type, elem_type_id);
-
- elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
- if (!elem_type) {
- btf_verifier_log_type(env, v->t, "Invalid elem");
- return -EINVAL;
- }
-
- if (btf_type_is_int(elem_type) && !btf_type_int_is_regular(elem_type)) {
- btf_verifier_log_type(env, v->t, "Invalid array of int");
- return -EINVAL;
- }
-
- if (array->nelems && elem_size > U32_MAX / array->nelems) {
- btf_verifier_log_type(env, v->t,
- "Array size overflows U32_MAX");
- return -EINVAL;
- }
-
- env_stack_pop_resolved(env, elem_type_id, elem_size * array->nelems);
-
- return 0;
-}
-
-static void btf_array_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_array *array = btf_type_array(t);
-
- btf_verifier_log(env, "type_id=%u index_type_id=%u nr_elems=%u",
- array->type, array->index_type, array->nelems);
-}
-
-static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_array *array = btf_type_array(t);
- const struct btf_kind_operations *elem_ops;
- const struct btf_type *elem_type;
- u32 i, elem_size = 0, elem_type_id;
- u16 encoding = 0;
-
- elem_type_id = array->type;
- elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
- if (elem_type && btf_type_has_size(elem_type))
- elem_size = elem_type->size;
-
- if (elem_type && btf_type_is_int(elem_type)) {
- u32 int_type = btf_type_int(elem_type);
-
- encoding = BTF_INT_ENCODING(int_type);
-
- /*
- * BTF_INT_CHAR encoding never seems to be set for
- * char arrays, so if size is 1 and element is
- * printable as a char, we'll do that.
- */
- if (elem_size == 1)
- encoding = BTF_INT_CHAR;
- }
-
- if (!btf_show_start_array_type(show, t, type_id, encoding, data))
- return;
-
- if (!elem_type)
- goto out;
- elem_ops = btf_type_ops(elem_type);
-
- for (i = 0; i < array->nelems; i++) {
-
- btf_show_start_array_member(show);
-
- elem_ops->show(btf, elem_type, elem_type_id, data,
- bits_offset, show);
- data += elem_size;
-
- btf_show_end_array_member(show);
-
- if (show->state.array_terminated)
- break;
- }
-out:
- btf_show_end_array_type(show);
-}
-
-static void btf_array_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *m = show->state.member;
-
- /*
- * First check if any members would be shown (are non-zero).
- * See comments above "struct btf_show" definition for more
- * details on how this works at a high-level.
- */
- if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
- if (!show->state.depth_check) {
- show->state.depth_check = show->state.depth + 1;
- show->state.depth_to_show = 0;
- }
- __btf_array_show(btf, t, type_id, data, bits_offset, show);
- show->state.member = m;
-
- if (show->state.depth_check != show->state.depth + 1)
- return;
- show->state.depth_check = 0;
-
- if (show->state.depth_to_show <= show->state.depth)
- return;
- /*
- * Reaching here indicates we have recursed and found
- * non-zero array member(s).
- */
- }
- __btf_array_show(btf, t, type_id, data, bits_offset, show);
-}
-
-static const struct btf_kind_operations array_ops = {
- .check_meta = btf_array_check_meta,
- .resolve = btf_array_resolve,
- .check_member = btf_array_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_array_log,
- .show = btf_array_show,
-};
-
-static int btf_struct_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < member_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_struct_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- bool is_union = BTF_INFO_KIND(t->info) == BTF_KIND_UNION;
- const struct btf_member *member;
- u32 meta_needed, last_offset;
- struct btf *btf = env->btf;
- u32 struct_size = t->size;
- u32 offset;
- u16 i;
-
- meta_needed = btf_type_vlen(t) * sizeof(*member);
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- /* struct type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- last_offset = 0;
- for_each_member(i, t, member) {
- if (!btf_name_offset_valid(btf, member->name_off)) {
- btf_verifier_log_member(env, t, member,
- "Invalid member name_offset:%u",
- member->name_off);
- return -EINVAL;
- }
-
- /* struct member either no name or a valid one */
- if (member->name_off &&
- !btf_name_valid_identifier(btf, member->name_off)) {
- btf_verifier_log_member(env, t, member, "Invalid name");
- return -EINVAL;
- }
- /* A member cannot be in type void */
- if (!member->type || !BTF_TYPE_ID_VALID(member->type)) {
- btf_verifier_log_member(env, t, member,
- "Invalid type_id");
- return -EINVAL;
- }
-
- offset = __btf_member_bit_offset(t, member);
- if (is_union && offset) {
- btf_verifier_log_member(env, t, member,
- "Invalid member bits_offset");
- return -EINVAL;
- }
-
- /*
- * ">" instead of ">=" because the last member could be
- * "char a[0];"
- */
- if (last_offset > offset) {
- btf_verifier_log_member(env, t, member,
- "Invalid member bits_offset");
- return -EINVAL;
- }
-
- if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
- btf_verifier_log_member(env, t, member,
- "Member bits_offset exceeds its struct size");
- return -EINVAL;
- }
-
- btf_verifier_log_member(env, t, member, NULL);
- last_offset = offset;
- }
-
- return meta_needed;
-}
-
-static int btf_struct_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_member *member;
- int err;
- u16 i;
-
- /* Before continue resolving the next_member,
- * ensure the last member is indeed resolved to a
- * type with size info.
- */
- if (v->next_member) {
- const struct btf_type *last_member_type;
- const struct btf_member *last_member;
- u32 last_member_type_id;
-
- last_member = btf_type_member(v->t) + v->next_member - 1;
- last_member_type_id = last_member->type;
- if (WARN_ON_ONCE(!env_type_is_resolved(env,
- last_member_type_id)))
- return -EINVAL;
-
- last_member_type = btf_type_by_id(env->btf,
- last_member_type_id);
- if (btf_type_kflag(v->t))
- err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
- last_member,
- last_member_type);
- else
- err = btf_type_ops(last_member_type)->check_member(env, v->t,
- last_member,
- last_member_type);
- if (err)
- return err;
- }
-
- for_each_member_from(i, v->next_member, v->t, member) {
- u32 member_type_id = member->type;
- const struct btf_type *member_type = btf_type_by_id(env->btf,
- member_type_id);
-
- if (btf_type_nosize_or_null(member_type) ||
- btf_type_is_resolve_source_only(member_type)) {
- btf_verifier_log_member(env, v->t, member,
- "Invalid member");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, member_type) &&
- !env_type_is_resolved(env, member_type_id)) {
- env_stack_set_next_member(env, i + 1);
- return env_stack_push(env, member_type, member_type_id);
- }
-
- if (btf_type_kflag(v->t))
- err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
- member,
- member_type);
- else
- err = btf_type_ops(member_type)->check_member(env, v->t,
- member,
- member_type);
- if (err)
- return err;
- }
-
- env_stack_pop_resolved(env, 0, 0);
-
- return 0;
-}
-
-static void btf_struct_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-enum {
- BTF_FIELD_IGNORE = 0,
- BTF_FIELD_FOUND = 1,
-};
-
-struct btf_field_info {
- enum btf_field_type type;
- u32 off;
- union {
- struct {
- u32 type_id;
- } kptr;
- struct {
- const char *node_name;
- u32 value_btf_id;
- } graph_root;
- };
-};
-
-static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
- u32 off, int sz, enum btf_field_type field_type,
- struct btf_field_info *info)
-{
- if (!__btf_type_is_struct(t))
- return BTF_FIELD_IGNORE;
- if (t->size != sz)
- return BTF_FIELD_IGNORE;
- info->type = field_type;
- info->off = off;
- return BTF_FIELD_FOUND;
-}
-
-static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
- u32 off, int sz, struct btf_field_info *info, u32 field_mask)
-{
- enum btf_field_type type;
- const char *tag_value;
- bool is_type_tag;
- u32 res_id;
-
- /* Permit modifiers on the pointer itself */
- if (btf_type_is_volatile(t))
- t = btf_type_by_id(btf, t->type);
- /* For PTR, sz is always == 8 */
- if (!btf_type_is_ptr(t))
- return BTF_FIELD_IGNORE;
- t = btf_type_by_id(btf, t->type);
- is_type_tag = btf_type_is_type_tag(t) && !btf_type_kflag(t);
- if (!is_type_tag)
- return BTF_FIELD_IGNORE;
- /* Reject extra tags */
- if (btf_type_is_type_tag(btf_type_by_id(btf, t->type)))
- return -EINVAL;
- tag_value = __btf_name_by_offset(btf, t->name_off);
- if (!strcmp("kptr_untrusted", tag_value))
- type = BPF_KPTR_UNREF;
- else if (!strcmp("kptr", tag_value))
- type = BPF_KPTR_REF;
- else if (!strcmp("percpu_kptr", tag_value))
- type = BPF_KPTR_PERCPU;
- else if (!strcmp("uptr", tag_value))
- type = BPF_UPTR;
- else
- return -EINVAL;
-
- if (!(type & field_mask))
- return BTF_FIELD_IGNORE;
-
- /* Get the base type */
- t = btf_type_skip_modifiers(btf, t->type, &res_id);
- /* Only pointer to struct is allowed */
- if (!__btf_type_is_struct(t))
- return -EINVAL;
-
- info->type = type;
- info->off = off;
- info->kptr.type_id = res_id;
- return BTF_FIELD_FOUND;
-}
-
-int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
- int comp_idx, const char *tag_key, int last_id)
-{
- int len = strlen(tag_key);
- int i, n;
-
- for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
- const struct btf_type *t = btf_type_by_id(btf, i);
-
- if (!btf_type_is_decl_tag(t))
- continue;
- if (pt != btf_type_by_id(btf, t->type))
- continue;
- if (btf_type_decl_tag(t)->component_idx != comp_idx)
- continue;
- if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
- continue;
- return i;
- }
- return -ENOENT;
-}
-
-const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
- int comp_idx, const char *tag_key)
-{
- const char *value = NULL;
- const struct btf_type *t;
- int len, id;
-
- id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
- btf_named_start_id(btf, false) - 1);
- if (id < 0)
- return ERR_PTR(id);
-
- t = btf_type_by_id(btf, id);
- len = strlen(tag_key);
- value = __btf_name_by_offset(btf, t->name_off) + len;
-
- /* Prevent duplicate entries for same type */
- id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, id);
- if (id >= 0)
- return ERR_PTR(-EEXIST);
-
- return value;
-}
-
-static int
-btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
- const struct btf_type *t, int comp_idx, u32 off,
- int sz, struct btf_field_info *info,
- enum btf_field_type head_type)
-{
- const char *node_field_name;
- const char *value_type;
- s32 id;
-
- if (!__btf_type_is_struct(t))
- return BTF_FIELD_IGNORE;
- if (t->size != sz)
- return BTF_FIELD_IGNORE;
- value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
- if (IS_ERR(value_type))
- return -EINVAL;
- node_field_name = strstr(value_type, ":");
- if (!node_field_name)
- return -EINVAL;
- value_type = kstrndup(value_type, node_field_name - value_type,
- GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
- if (!value_type)
- return -ENOMEM;
- id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
- kfree(value_type);
- if (id < 0)
- return id;
- node_field_name++;
- if (str_is_empty(node_field_name))
- return -EINVAL;
- info->type = head_type;
- info->off = off;
- info->graph_root.value_btf_id = id;
- info->graph_root.node_name = node_field_name;
- return BTF_FIELD_FOUND;
-}
-
-static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_type,
- u32 field_mask, u32 *seen_mask, int *align, int *sz)
-{
- const struct {
- enum btf_field_type type;
- const char *const name;
- const bool is_unique;
- } field_types[] = {
- { BPF_SPIN_LOCK, "bpf_spin_lock", true },
- { BPF_RES_SPIN_LOCK, "bpf_res_spin_lock", true },
- { BPF_TIMER, "bpf_timer", true },
- { BPF_WORKQUEUE, "bpf_wq", true },
- { BPF_TASK_WORK, "bpf_task_work", true },
- { BPF_LIST_HEAD, "bpf_list_head", false },
- { BPF_LIST_NODE, "bpf_list_node", false },
- { BPF_RB_ROOT, "bpf_rb_root", false },
- { BPF_RB_NODE, "bpf_rb_node", false },
- { BPF_REFCOUNT, "bpf_refcount", false },
- };
- int type = 0, i;
- const char *name = __btf_name_by_offset(btf, var_type->name_off);
- const char *field_type_name;
- enum btf_field_type field_type;
- bool is_unique;
-
- for (i = 0; i < ARRAY_SIZE(field_types); ++i) {
- field_type = field_types[i].type;
- field_type_name = field_types[i].name;
- is_unique = field_types[i].is_unique;
- if (!(field_mask & field_type) || strcmp(name, field_type_name))
- continue;
- if (is_unique) {
- if (*seen_mask & field_type)
- return -E2BIG;
- *seen_mask |= field_type;
- }
- type = field_type;
- goto end;
- }
-
- /* Only return BPF_KPTR when all other types with matchable names fail */
- if (field_mask & (BPF_KPTR | BPF_UPTR) && !__btf_type_is_struct(var_type)) {
- type = BPF_KPTR_REF;
- goto end;
- }
- return 0;
-end:
- *sz = btf_field_type_size(type);
- *align = btf_field_type_align(type);
- return type;
-}
-
-/* Repeat a number of fields for a specified number of times.
- *
- * Copy the fields starting from the first field and repeat them for
- * repeat_cnt times. The fields are repeated by adding the offset of each
- * field with
- * (i + 1) * elem_size
- * where i is the repeat index and elem_size is the size of an element.
- */
-static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
- u32 field_cnt, u32 repeat_cnt, u32 elem_size)
-{
- u32 i, j;
- u32 cur;
-
- /* Ensure not repeating fields that should not be repeated. */
- for (i = 0; i < field_cnt; i++) {
- switch (info[i].type) {
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- case BPF_LIST_HEAD:
- case BPF_RB_ROOT:
- break;
- default:
- return -EINVAL;
- }
- }
-
- /* The type of struct size or variable size is u32,
- * so the multiplication will not overflow.
- */
- if (field_cnt * (repeat_cnt + 1) > info_cnt)
- return -E2BIG;
-
- cur = field_cnt;
- for (i = 0; i < repeat_cnt; i++) {
- memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
- for (j = 0; j < field_cnt; j++)
- info[cur++].off += (i + 1) * elem_size;
- }
-
- return 0;
-}
-
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level);
-
-/* Find special fields in the struct type of a field.
- *
- * This function is used to find fields of special types that is not a
- * global variable or a direct field of a struct type. It also handles the
- * repetition if it is the element type of an array.
- */
-static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
- u32 off, u32 nelems,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level)
-{
- int ret, err, i;
-
- level++;
- if (level >= MAX_RESOLVE_DEPTH)
- return -E2BIG;
-
- ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
-
- if (ret <= 0)
- return ret;
-
- /* Shift the offsets of the nested struct fields to the offsets
- * related to the container.
- */
- for (i = 0; i < ret; i++)
- info[i].off += off;
-
- if (nelems > 1) {
- err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
- if (err == 0)
- ret *= nelems;
- else
- ret = err;
- }
-
- return ret;
-}
-
-static int btf_find_field_one(const struct btf *btf,
- const struct btf_type *var,
- const struct btf_type *var_type,
- int var_idx,
- u32 off, u32 expected_size,
- u32 field_mask, u32 *seen_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level)
-{
- int ret, align, sz, field_type;
- struct btf_field_info tmp;
- const struct btf_array *array;
- u32 i, nelems = 1;
-
- /* Walk into array types to find the element type and the number of
- * elements in the (flattened) array.
- */
- for (i = 0; i < MAX_RESOLVE_DEPTH && btf_type_is_array(var_type); i++) {
- array = btf_array(var_type);
- nelems *= array->nelems;
- var_type = btf_type_by_id(btf, array->type);
- }
- if (i == MAX_RESOLVE_DEPTH)
- return -E2BIG;
- if (nelems == 0)
- return 0;
-
- field_type = btf_get_field_type(btf, var_type,
- field_mask, seen_mask, &align, &sz);
- /* Look into variables of struct types */
- if (!field_type && __btf_type_is_struct(var_type)) {
- sz = var_type->size;
- if (expected_size && expected_size != sz * nelems)
- return 0;
- ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
- &info[0], info_cnt, level);
- return ret;
- }
-
- if (field_type == 0)
- return 0;
- if (field_type < 0)
- return field_type;
-
- if (expected_size && expected_size != sz * nelems)
- return 0;
- if (off % align)
- return 0;
-
- switch (field_type) {
- case BPF_SPIN_LOCK:
- case BPF_RES_SPIN_LOCK:
- case BPF_TIMER:
- case BPF_WORKQUEUE:
- case BPF_LIST_NODE:
- case BPF_RB_NODE:
- case BPF_REFCOUNT:
- case BPF_TASK_WORK:
- ret = btf_find_struct(btf, var_type, off, sz, field_type,
- info_cnt ? &info[0] : &tmp);
- if (ret < 0)
- return ret;
- break;
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- ret = btf_find_kptr(btf, var_type, off, sz,
- info_cnt ? &info[0] : &tmp, field_mask);
- if (ret < 0)
- return ret;
- break;
- case BPF_LIST_HEAD:
- case BPF_RB_ROOT:
- ret = btf_find_graph_root(btf, var, var_type,
- var_idx, off, sz,
- info_cnt ? &info[0] : &tmp,
- field_type);
- if (ret < 0)
- return ret;
- break;
- default:
- return -EFAULT;
- }
-
- if (ret == BTF_FIELD_IGNORE)
- return 0;
- if (!info_cnt)
- return -E2BIG;
- if (nelems > 1) {
- ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
- if (ret < 0)
- return ret;
- }
- return nelems;
-}
-
-static int btf_find_struct_field(const struct btf *btf,
- const struct btf_type *t, u32 field_mask,
- struct btf_field_info *info, int info_cnt,
- u32 level)
-{
- int ret, idx = 0;
- const struct btf_member *member;
- u32 i, off, seen_mask = 0;
-
- for_each_member(i, t, member) {
- const struct btf_type *member_type = btf_type_by_id(btf,
- member->type);
-
- off = __btf_member_bit_offset(t, member);
- if (off % 8)
- /* valid C code cannot generate such BTF */
- return -EINVAL;
- off /= 8;
-
- ret = btf_find_field_one(btf, t, member_type, i,
- off, 0,
- field_mask, &seen_mask,
- &info[idx], info_cnt - idx, level);
- if (ret < 0)
- return ret;
- idx += ret;
- }
- return idx;
-}
-
-static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt, u32 level)
-{
- int ret, idx = 0;
- const struct btf_var_secinfo *vsi;
- u32 i, off, seen_mask = 0;
-
- for_each_vsi(i, t, vsi) {
- const struct btf_type *var = btf_type_by_id(btf, vsi->type);
- const struct btf_type *var_type = btf_type_by_id(btf, var->type);
-
- off = vsi->offset;
- ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
- field_mask, &seen_mask,
- &info[idx], info_cnt - idx,
- level);
- if (ret < 0)
- return ret;
- idx += ret;
- }
- return idx;
-}
-
-static int btf_find_field(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, struct btf_field_info *info,
- int info_cnt)
-{
- if (__btf_type_is_struct(t))
- return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
- else if (btf_type_is_datasec(t))
- return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
- return -EINVAL;
-}
-
-/* Callers have to ensure the life cycle of btf if it is program BTF */
-static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- struct module *mod = NULL;
- const struct btf_type *t;
- /* If a matching btf type is found in kernel or module BTFs, kptr_ref
- * is that BTF, otherwise it's program BTF
- */
- struct btf *kptr_btf;
- int ret;
- s32 id;
-
- /* Find type in map BTF, and use it to look up the matching type
- * in vmlinux or module BTFs, by name and kind.
- */
- t = btf_type_by_id(btf, info->kptr.type_id);
- id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
- &kptr_btf);
- if (id == -ENOENT) {
- /* btf_parse_kptr should only be called w/ btf = program BTF */
- WARN_ON_ONCE(btf_is_kernel(btf));
-
- /* Type exists only in program BTF. Assume that it's a MEM_ALLOC
- * kptr allocated via bpf_obj_new
- */
- field->kptr.dtor = NULL;
- id = info->kptr.type_id;
- kptr_btf = (struct btf *)btf;
- goto found_dtor;
- }
- if (id < 0)
- return id;
-
- /* Find and stash the function pointer for the destruction function that
- * needs to be eventually invoked from the map free path.
- */
- if (info->type == BPF_KPTR_REF) {
- const struct btf_type *dtor_func;
- const char *dtor_func_name;
- unsigned long addr;
- s32 dtor_btf_id;
-
- /* This call also serves as a whitelist of allowed objects that
- * can be used as a referenced pointer and be stored in a map at
- * the same time.
- */
- dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
- if (dtor_btf_id < 0) {
- ret = dtor_btf_id;
- goto end_btf;
- }
-
- dtor_func = btf_type_by_id(kptr_btf, dtor_btf_id);
- if (!dtor_func) {
- ret = -ENOENT;
- goto end_btf;
- }
-
- if (btf_is_module(kptr_btf)) {
- mod = btf_try_get_module(kptr_btf);
- if (!mod) {
- ret = -ENXIO;
- goto end_btf;
- }
- }
-
- /* We already verified dtor_func to be btf_type_is_func
- * in register_btf_id_dtor_kfuncs.
- */
- dtor_func_name = __btf_name_by_offset(kptr_btf, dtor_func->name_off);
- addr = kallsyms_lookup_name(dtor_func_name);
- if (!addr) {
- ret = -EINVAL;
- goto end_mod;
- }
- field->kptr.dtor = (void *)addr;
- }
-
-found_dtor:
- field->kptr.btf_id = id;
- field->kptr.btf = kptr_btf;
- field->kptr.module = mod;
- return 0;
-end_mod:
- module_put(mod);
-end_btf:
- btf_put(kptr_btf);
- return ret;
-}
-
-static int btf_parse_graph_root(const struct btf *btf,
- struct btf_field *field,
- struct btf_field_info *info,
- const char *node_type_name,
- size_t node_type_align)
-{
- const struct btf_type *t, *n = NULL;
- const struct btf_member *member;
- u32 offset;
- int i;
-
- t = btf_type_by_id(btf, info->graph_root.value_btf_id);
- /* We've already checked that value_btf_id is a struct type. We
- * just need to figure out the offset of the list_node, and
- * verify its type.
- */
- for_each_member(i, t, member) {
- if (strcmp(info->graph_root.node_name,
- __btf_name_by_offset(btf, member->name_off)))
- continue;
- /* Invalid BTF, two members with same name */
- if (n)
- return -EINVAL;
- n = btf_type_by_id(btf, member->type);
- if (!__btf_type_is_struct(n))
- return -EINVAL;
- if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off)))
- return -EINVAL;
- offset = __btf_member_bit_offset(n, member);
- if (offset % 8)
- return -EINVAL;
- offset /= 8;
- if (offset % node_type_align)
- return -EINVAL;
-
- field->graph_root.btf = (struct btf *)btf;
- field->graph_root.value_btf_id = info->graph_root.value_btf_id;
- field->graph_root.node_offset = offset;
- }
- if (!n)
- return -ENOENT;
- return 0;
-}
-
-static int btf_parse_list_head(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- return btf_parse_graph_root(btf, field, info, "bpf_list_node",
- __alignof__(struct bpf_list_node));
-}
-
-static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
- struct btf_field_info *info)
-{
- return btf_parse_graph_root(btf, field, info, "bpf_rb_node",
- __alignof__(struct bpf_rb_node));
-}
-
-static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
-{
- const struct btf_field *a = (const struct btf_field *)_a;
- const struct btf_field *b = (const struct btf_field *)_b;
-
- if (a->offset < b->offset)
- return -1;
- else if (a->offset > b->offset)
- return 1;
- return 0;
-}
-
-struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
- u32 field_mask, u32 value_size)
-{
- struct btf_field_info info_arr[BTF_FIELDS_MAX];
- u32 next_off = 0, field_type_size;
- struct btf_record *rec;
- int ret, i, cnt;
-
- ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
- if (ret < 0)
- return ERR_PTR(ret);
- if (!ret)
- return NULL;
-
- cnt = ret;
- /* This needs to be kzalloc to zero out padding and unused fields, see
- * comment in btf_record_equal.
- */
- rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
- if (!rec)
- return ERR_PTR(-ENOMEM);
-
- rec->spin_lock_off = -EINVAL;
- rec->res_spin_lock_off = -EINVAL;
- rec->timer_off = -EINVAL;
- rec->wq_off = -EINVAL;
- rec->refcount_off = -EINVAL;
- rec->task_work_off = -EINVAL;
- for (i = 0; i < cnt; i++) {
- field_type_size = btf_field_type_size(info_arr[i].type);
- if (info_arr[i].off + field_type_size > value_size) {
- WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size);
- ret = -EFAULT;
- goto end;
- }
- if (info_arr[i].off < next_off) {
- ret = -EEXIST;
- goto end;
- }
- next_off = info_arr[i].off + field_type_size;
-
- rec->field_mask |= info_arr[i].type;
- rec->fields[i].offset = info_arr[i].off;
- rec->fields[i].type = info_arr[i].type;
- rec->fields[i].size = field_type_size;
-
- switch (info_arr[i].type) {
- case BPF_SPIN_LOCK:
- WARN_ON_ONCE(rec->spin_lock_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->spin_lock_off = rec->fields[i].offset;
- break;
- case BPF_RES_SPIN_LOCK:
- WARN_ON_ONCE(rec->spin_lock_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->res_spin_lock_off = rec->fields[i].offset;
- break;
- case BPF_TIMER:
- WARN_ON_ONCE(rec->timer_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->timer_off = rec->fields[i].offset;
- break;
- case BPF_WORKQUEUE:
- WARN_ON_ONCE(rec->wq_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->wq_off = rec->fields[i].offset;
- break;
- case BPF_TASK_WORK:
- WARN_ON_ONCE(rec->task_work_off >= 0);
- rec->task_work_off = rec->fields[i].offset;
- break;
- case BPF_REFCOUNT:
- WARN_ON_ONCE(rec->refcount_off >= 0);
- /* Cache offset for faster lookup at runtime */
- rec->refcount_off = rec->fields[i].offset;
- break;
- case BPF_KPTR_UNREF:
- case BPF_KPTR_REF:
- case BPF_KPTR_PERCPU:
- case BPF_UPTR:
- ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_LIST_HEAD:
- ret = btf_parse_list_head(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_RB_ROOT:
- ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]);
- if (ret < 0)
- goto end;
- break;
- case BPF_LIST_NODE:
- case BPF_RB_NODE:
- break;
- default:
- ret = -EFAULT;
- goto end;
- }
- rec->cnt++;
- }
-
- if (rec->spin_lock_off >= 0 && rec->res_spin_lock_off >= 0) {
- ret = -EINVAL;
- goto end;
- }
-
- /* bpf_{list_head, rb_node} require bpf_spin_lock */
- if ((btf_record_has_field(rec, BPF_LIST_HEAD) ||
- btf_record_has_field(rec, BPF_RB_ROOT)) &&
- (rec->spin_lock_off < 0 && rec->res_spin_lock_off < 0)) {
- ret = -EINVAL;
- goto end;
- }
-
- if (rec->refcount_off < 0 &&
- btf_record_has_field(rec, BPF_LIST_NODE) &&
- btf_record_has_field(rec, BPF_RB_NODE)) {
- ret = -EINVAL;
- goto end;
- }
-
- sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
- NULL, rec);
-
- return rec;
-end:
- btf_record_free(rec);
- return ERR_PTR(ret);
-}
-
-int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
-{
- int i;
-
- /* There are three types that signify ownership of some other type:
- * kptr_ref, bpf_list_head, bpf_rb_root.
- * kptr_ref only supports storing kernel types, which can't store
- * references to program allocated local types.
- *
- * Hence we only need to ensure that bpf_{list_head,rb_root} ownership
- * does not form cycles.
- */
- if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & (BPF_GRAPH_ROOT | BPF_UPTR)))
- return 0;
- for (i = 0; i < rec->cnt; i++) {
- struct btf_struct_meta *meta;
- const struct btf_type *t;
- u32 btf_id;
-
- if (rec->fields[i].type == BPF_UPTR) {
- /* The uptr only supports pinning one page and cannot
- * point to a kernel struct
- */
- if (btf_is_kernel(rec->fields[i].kptr.btf))
- return -EINVAL;
- t = btf_type_by_id(rec->fields[i].kptr.btf,
- rec->fields[i].kptr.btf_id);
- if (!t->size)
- return -EINVAL;
- if (t->size > PAGE_SIZE)
- return -E2BIG;
- continue;
- }
-
- if (!(rec->fields[i].type & BPF_GRAPH_ROOT))
- continue;
- btf_id = rec->fields[i].graph_root.value_btf_id;
- meta = btf_find_struct_meta(btf, btf_id);
- if (!meta)
- return -EFAULT;
- rec->fields[i].graph_root.value_rec = meta->record;
-
- /* We need to set value_rec for all root types, but no need
- * to check ownership cycle for a type unless it's also a
- * node type.
- */
- if (!(rec->field_mask & BPF_GRAPH_NODE))
- continue;
-
- /* We need to ensure ownership acyclicity among all types. The
- * proper way to do it would be to topologically sort all BTF
- * IDs based on the ownership edges, since there can be multiple
- * bpf_{list_head,rb_node} in a type. Instead, we use the
- * following resaoning:
- *
- * - A type can only be owned by another type in user BTF if it
- * has a bpf_{list,rb}_node. Let's call these node types.
- * - A type can only _own_ another type in user BTF if it has a
- * bpf_{list_head,rb_root}. Let's call these root types.
- *
- * We ensure that if a type is both a root and node, its
- * element types cannot be root types.
- *
- * To ensure acyclicity:
- *
- * When A is an root type but not a node, its ownership
- * chain can be:
- * A -> B -> C
- * Where:
- * - A is an root, e.g. has bpf_rb_root.
- * - B is both a root and node, e.g. has bpf_rb_node and
- * bpf_list_head.
- * - C is only an root, e.g. has bpf_list_node
- *
- * When A is both a root and node, some other type already
- * owns it in the BTF domain, hence it can not own
- * another root type through any of the ownership edges.
- * A -> B
- * Where:
- * - A is both an root and node.
- * - B is only an node.
- */
- if (meta->record->field_mask & BPF_GRAPH_ROOT)
- return -ELOOP;
- }
- return 0;
-}
-
-static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *member;
- void *safe_data;
- u32 i;
-
- safe_data = btf_show_start_struct_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- for_each_member(i, t, member) {
- const struct btf_type *member_type = btf_type_by_id(btf,
- member->type);
- const struct btf_kind_operations *ops;
- u32 member_offset, bitfield_size;
- u32 bytes_offset;
- u8 bits8_offset;
-
- btf_show_start_member(show, member);
-
- member_offset = __btf_member_bit_offset(t, member);
- bitfield_size = __btf_member_bitfield_size(t, member);
- bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
- bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
- if (bitfield_size) {
- safe_data = btf_show_start_type(show, member_type,
- member->type,
- data + bytes_offset);
- if (safe_data)
- btf_bitfield_show(safe_data,
- bits8_offset,
- bitfield_size, show);
- btf_show_end_type(show);
- } else {
- ops = btf_type_ops(member_type);
- ops->show(btf, member_type, member->type,
- data + bytes_offset, bits8_offset, show);
- }
-
- btf_show_end_member(show);
- }
-
- btf_show_end_struct_type(show);
-}
-
-static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_member *m = show->state.member;
-
- /*
- * First check if any members would be shown (are non-zero).
- * See comments above "struct btf_show" definition for more
- * details on how this works at a high-level.
- */
- if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
- if (!show->state.depth_check) {
- show->state.depth_check = show->state.depth + 1;
- show->state.depth_to_show = 0;
- }
- __btf_struct_show(btf, t, type_id, data, bits_offset, show);
- /* Restore saved member data here */
- show->state.member = m;
- if (show->state.depth_check != show->state.depth + 1)
- return;
- show->state.depth_check = 0;
-
- if (show->state.depth_to_show <= show->state.depth)
- return;
- /*
- * Reaching here indicates we have recursed and found
- * non-zero child values.
- */
- }
-
- __btf_struct_show(btf, t, type_id, data, bits_offset, show);
-}
-
-static const struct btf_kind_operations struct_ops = {
- .check_meta = btf_struct_check_meta,
- .resolve = btf_struct_resolve,
- .check_member = btf_struct_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_struct_log,
- .show = btf_struct_show,
-};
-
-static int btf_enum_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off = member->offset;
- u32 struct_size, bytes_offset;
-
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
- if (struct_size - bytes_offset < member_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u32 struct_bits_off, nr_bits, bytes_end, struct_size;
- u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
-
- struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
- nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
- if (!nr_bits) {
- if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not byte aligned");
- return -EINVAL;
- }
-
- nr_bits = int_bitsize;
- } else if (nr_bits > int_bitsize) {
- btf_verifier_log_member(env, struct_type, member,
- "Invalid member bitfield_size");
- return -EINVAL;
- }
-
- struct_size = struct_type->size;
- bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
- if (struct_size < bytes_end) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static s32 btf_enum_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_enum *enums = btf_type_enum(t);
- struct btf *btf = env->btf;
- const char *fmt_str;
- u16 i, nr_enums;
- u32 meta_needed;
-
- nr_enums = btf_type_vlen(t);
- meta_needed = nr_enums * sizeof(*enums);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->size > 8 || !is_power_of_2(t->size)) {
- btf_verifier_log_type(env, t, "Unexpected size");
- return -EINVAL;
- }
-
- /* enum type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for (i = 0; i < nr_enums; i++) {
- if (!btf_name_offset_valid(btf, enums[i].name_off)) {
- btf_verifier_log(env, "\tInvalid name_offset:%u",
- enums[i].name_off);
- return -EINVAL;
- }
-
- /* enum member must have a valid name */
- if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (env->log.level == BPF_LOG_KERNEL)
- continue;
- fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
- btf_verifier_log(env, fmt_str,
- __btf_name_by_offset(btf, enums[i].name_off),
- enums[i].val);
- }
-
- return meta_needed;
-}
-
-static void btf_enum_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_enum *enums = btf_type_enum(t);
- u32 i, nr_enums = btf_type_vlen(t);
- void *safe_data;
- int v;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- v = *(int *)safe_data;
-
- for (i = 0; i < nr_enums; i++) {
- if (v != enums[i].val)
- continue;
-
- btf_show_type_value(show, "%s",
- __btf_name_by_offset(btf,
- enums[i].name_off));
-
- btf_show_end_type(show);
- return;
- }
-
- if (btf_type_kflag(t))
- btf_show_type_value(show, "%d", v);
- else
- btf_show_type_value(show, "%u", v);
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations enum_ops = {
- .check_meta = btf_enum_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_enum_check_member,
- .check_kflag_member = btf_enum_check_kflag_member,
- .log_details = btf_enum_log,
- .show = btf_enum_show,
-};
-
-static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_enum64 *enums = btf_type_enum64(t);
- struct btf *btf = env->btf;
- const char *fmt_str;
- u16 i, nr_enums;
- u32 meta_needed;
-
- nr_enums = btf_type_vlen(t);
- meta_needed = nr_enums * sizeof(*enums);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->size > 8 || !is_power_of_2(t->size)) {
- btf_verifier_log_type(env, t, "Unexpected size");
- return -EINVAL;
- }
-
- /* enum type either no name or a valid one */
- if (t->name_off &&
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for (i = 0; i < nr_enums; i++) {
- if (!btf_name_offset_valid(btf, enums[i].name_off)) {
- btf_verifier_log(env, "\tInvalid name_offset:%u",
- enums[i].name_off);
- return -EINVAL;
- }
-
- /* enum member must have a valid name */
- if (!enums[i].name_off ||
- !btf_name_valid_identifier(btf, enums[i].name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (env->log.level == BPF_LOG_KERNEL)
- continue;
-
- fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
- btf_verifier_log(env, fmt_str,
- __btf_name_by_offset(btf, enums[i].name_off),
- btf_enum64_value(enums + i));
- }
-
- return meta_needed;
-}
-
-static void btf_enum64_show(const struct btf *btf, const struct btf_type *t,
- u32 type_id, void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_enum64 *enums = btf_type_enum64(t);
- u32 i, nr_enums = btf_type_vlen(t);
- void *safe_data;
- s64 v;
-
- safe_data = btf_show_start_type(show, t, type_id, data);
- if (!safe_data)
- return;
-
- v = *(u64 *)safe_data;
-
- for (i = 0; i < nr_enums; i++) {
- if (v != btf_enum64_value(enums + i))
- continue;
-
- btf_show_type_value(show, "%s",
- __btf_name_by_offset(btf,
- enums[i].name_off));
-
- btf_show_end_type(show);
- return;
- }
-
- if (btf_type_kflag(t))
- btf_show_type_value(show, "%lld", v);
- else
- btf_show_type_value(show, "%llu", v);
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations enum64_ops = {
- .check_meta = btf_enum64_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_enum_check_member,
- .check_kflag_member = btf_enum_check_kflag_member,
- .log_details = btf_enum_log,
- .show = btf_enum64_show,
-};
-
-static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 meta_needed = btf_type_vlen(t) * sizeof(struct btf_param);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (t->name_off) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_func_proto_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_param *args = (const struct btf_param *)(t + 1);
- u16 nr_args = btf_type_vlen(t), i;
-
- btf_verifier_log(env, "return=%u args=(", t->type);
- if (!nr_args) {
- btf_verifier_log(env, "void");
- goto done;
- }
-
- if (nr_args == 1 && !args[0].type) {
- /* Only one vararg */
- btf_verifier_log(env, "vararg");
- goto done;
- }
-
- btf_verifier_log(env, "%u %s", args[0].type,
- __btf_name_by_offset(env->btf,
- args[0].name_off));
- for (i = 1; i < nr_args - 1; i++)
- btf_verifier_log(env, ", %u %s", args[i].type,
- __btf_name_by_offset(env->btf,
- args[i].name_off));
-
- if (nr_args > 1) {
- const struct btf_param *last_arg = &args[nr_args - 1];
-
- if (last_arg->type)
- btf_verifier_log(env, ", %u %s", last_arg->type,
- __btf_name_by_offset(env->btf,
- last_arg->name_off));
- else
- btf_verifier_log(env, ", vararg");
- }
-
-done:
- btf_verifier_log(env, ")");
-}
-
-static const struct btf_kind_operations func_proto_ops = {
- .check_meta = btf_func_proto_check_meta,
- .resolve = btf_df_resolve,
- /*
- * BTF_KIND_FUNC_PROTO cannot be directly referred by
- * a struct's member.
- *
- * It should be a function pointer instead.
- * (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
- *
- * Hence, there is no btf_func_check_member().
- */
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_func_proto_log,
- .show = btf_df_show,
-};
-
-static s32 btf_func_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t) > BTF_FUNC_GLOBAL) {
- btf_verifier_log_type(env, t, "Invalid func linkage");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_func_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- int err;
-
- err = btf_func_check(env, t);
- if (err)
- return err;
-
- env_stack_pop_resolved(env, next_type_id, 0);
- return 0;
-}
-
-static const struct btf_kind_operations func_ops = {
- .check_meta = btf_func_check_meta,
- .resolve = btf_func_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_ref_type_log,
- .show = btf_df_show,
-};
-
-static s32 btf_var_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_var *var;
- u32 meta_needed = sizeof(*var);
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!t->name_off ||
- !btf_name_valid_identifier(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- /* A var cannot be in type void */
- if (!t->type || !BTF_TYPE_ID_VALID(t->type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- var = btf_type_var(t);
- if (var->linkage != BTF_VAR_STATIC &&
- var->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
- btf_verifier_log_type(env, t, "Linkage not supported");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t)
-{
- const struct btf_var *var = btf_type_var(t);
-
- btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage);
-}
-
-static const struct btf_kind_operations var_ops = {
- .check_meta = btf_var_check_meta,
- .resolve = btf_var_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_var_log,
- .show = btf_var_show,
-};
-
-static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_var_secinfo *vsi;
- u64 last_vsi_end_off = 0, sum = 0;
- u32 i, meta_needed;
-
- meta_needed = btf_type_vlen(t) * sizeof(*vsi);
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- if (!t->size) {
- btf_verifier_log_type(env, t, "size == 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (!t->name_off ||
- !btf_name_valid_section(env->btf, t->name_off)) {
- btf_verifier_log_type(env, t, "Invalid name");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- for_each_vsi(i, t, vsi) {
- /* A var cannot be in type void */
- if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid type_id");
- return -EINVAL;
- }
-
- if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid offset");
- return -EINVAL;
- }
-
- if (!vsi->size || vsi->size > t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid size");
- return -EINVAL;
- }
-
- last_vsi_end_off = vsi->offset + vsi->size;
- if (last_vsi_end_off > t->size) {
- btf_verifier_log_vsi(env, t, vsi,
- "Invalid offset+size");
- return -EINVAL;
- }
-
- btf_verifier_log_vsi(env, t, vsi, NULL);
- sum += vsi->size;
- }
-
- if (t->size < sum) {
- btf_verifier_log_type(env, t, "Invalid btf_info size");
- return -EINVAL;
- }
-
- return meta_needed;
-}
-
-static int btf_datasec_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_var_secinfo *vsi;
- struct btf *btf = env->btf;
- u16 i;
-
- env->resolve_mode = RESOLVE_TBD;
- for_each_vsi_from(i, v->next_member, v->t, vsi) {
- u32 var_type_id = vsi->type, type_id, type_size = 0;
- const struct btf_type *var_type = btf_type_by_id(env->btf,
- var_type_id);
- if (!var_type || !btf_type_is_var(var_type)) {
- btf_verifier_log_vsi(env, v->t, vsi,
- "Not a VAR kind member");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, var_type) &&
- !env_type_is_resolved(env, var_type_id)) {
- env_stack_set_next_member(env, i + 1);
- return env_stack_push(env, var_type, var_type_id);
- }
-
- type_id = var_type->type;
- if (!btf_type_id_size(btf, &type_id, &type_size)) {
- btf_verifier_log_vsi(env, v->t, vsi, "Invalid type");
- return -EINVAL;
- }
-
- if (vsi->size < type_size) {
- btf_verifier_log_vsi(env, v->t, vsi, "Invalid size");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, 0, 0);
- return 0;
-}
-
-static void btf_datasec_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
-}
-
-static void btf_datasec_show(const struct btf *btf,
- const struct btf_type *t, u32 type_id,
- void *data, u8 bits_offset,
- struct btf_show *show)
-{
- const struct btf_var_secinfo *vsi;
- const struct btf_type *var;
- u32 i;
-
- if (!btf_show_start_type(show, t, type_id, data))
- return;
-
- btf_show_type_value(show, "section (\"%s\") = {",
- __btf_name_by_offset(btf, t->name_off));
- for_each_vsi(i, t, vsi) {
- var = btf_type_by_id(btf, vsi->type);
- if (i)
- btf_show(show, ",");
- btf_type_ops(var)->show(btf, var, vsi->type,
- data + vsi->offset, bits_offset, show);
- }
- btf_show_end_type(show);
-}
-
-static const struct btf_kind_operations datasec_ops = {
- .check_meta = btf_datasec_check_meta,
- .resolve = btf_datasec_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_datasec_log,
- .show = btf_datasec_show,
-};
-
-static s32 btf_float_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- if (btf_type_kflag(t)) {
- btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
- return -EINVAL;
- }
-
- if (t->size != 2 && t->size != 4 && t->size != 8 && t->size != 12 &&
- t->size != 16) {
- btf_verifier_log_type(env, t, "Invalid type_size");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return 0;
-}
-
-static int btf_float_check_member(struct btf_verifier_env *env,
- const struct btf_type *struct_type,
- const struct btf_member *member,
- const struct btf_type *member_type)
-{
- u64 start_offset_bytes;
- u64 end_offset_bytes;
- u64 misalign_bits;
- u64 align_bytes;
- u64 align_bits;
-
- /* Different architectures have different alignment requirements, so
- * here we check only for the reasonable minimum. This way we ensure
- * that types after CO-RE can pass the kernel BTF verifier.
- */
- align_bytes = min_t(u64, sizeof(void *), member_type->size);
- align_bits = align_bytes * BITS_PER_BYTE;
- div64_u64_rem(member->offset, align_bits, &misalign_bits);
- if (misalign_bits) {
- btf_verifier_log_member(env, struct_type, member,
- "Member is not properly aligned");
- return -EINVAL;
- }
-
- start_offset_bytes = member->offset / BITS_PER_BYTE;
- end_offset_bytes = start_offset_bytes + member_type->size;
- if (end_offset_bytes > struct_type->size) {
- btf_verifier_log_member(env, struct_type, member,
- "Member exceeds struct_size");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static void btf_float_log(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- btf_verifier_log(env, "size=%u", t->size);
-}
-
-static const struct btf_kind_operations float_ops = {
- .check_meta = btf_float_check_meta,
- .resolve = btf_df_resolve,
- .check_member = btf_float_check_member,
- .check_kflag_member = btf_generic_check_kflag_member,
- .log_details = btf_float_log,
- .show = btf_df_show,
-};
-
-static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- const struct btf_decl_tag *tag;
- u32 meta_needed = sizeof(*tag);
- s32 component_idx;
- const char *value;
-
- if (meta_left < meta_needed) {
- btf_verifier_log_basic(env, t,
- "meta_left:%u meta_needed:%u",
- meta_left, meta_needed);
- return -EINVAL;
- }
-
- value = btf_name_by_offset(env->btf, t->name_off);
- if (!value || !value[0]) {
- btf_verifier_log_type(env, t, "Invalid value");
- return -EINVAL;
- }
-
- if (btf_type_vlen(t)) {
- btf_verifier_log_type(env, t, "vlen != 0");
- return -EINVAL;
- }
-
- component_idx = btf_type_decl_tag(t)->component_idx;
- if (component_idx < -1) {
- btf_verifier_log_type(env, t, "Invalid component_idx");
- return -EINVAL;
- }
-
- btf_verifier_log_type(env, t, NULL);
-
- return meta_needed;
-}
-
-static int btf_decl_tag_resolve(struct btf_verifier_env *env,
- const struct resolve_vertex *v)
-{
- const struct btf_type *next_type;
- const struct btf_type *t = v->t;
- u32 next_type_id = t->type;
- struct btf *btf = env->btf;
- s32 component_idx;
- u32 vlen;
-
- next_type = btf_type_by_id(btf, next_type_id);
- if (!next_type || !btf_type_is_decl_tag_target(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid type_id");
- return -EINVAL;
- }
-
- if (!env_type_is_resolve_sink(env, next_type) &&
- !env_type_is_resolved(env, next_type_id))
- return env_stack_push(env, next_type, next_type_id);
-
- component_idx = btf_type_decl_tag(t)->component_idx;
- if (component_idx != -1) {
- if (btf_type_is_var(next_type) || btf_type_is_typedef(next_type)) {
- btf_verifier_log_type(env, v->t, "Invalid component_idx");
- return -EINVAL;
- }
-
- if (btf_type_is_struct(next_type)) {
- vlen = btf_type_vlen(next_type);
- } else {
- /* next_type should be a function */
- next_type = btf_type_by_id(btf, next_type->type);
- vlen = btf_type_vlen(next_type);
- }
-
- if ((u32)component_idx >= vlen) {
- btf_verifier_log_type(env, v->t, "Invalid component_idx");
- return -EINVAL;
- }
- }
-
- env_stack_pop_resolved(env, next_type_id, 0);
-
- return 0;
-}
-
-static void btf_decl_tag_log(struct btf_verifier_env *env, const struct btf_type *t)
-{
- btf_verifier_log(env, "type=%u component_idx=%d", t->type,
- btf_type_decl_tag(t)->component_idx);
-}
-
-static const struct btf_kind_operations decl_tag_ops = {
- .check_meta = btf_decl_tag_check_meta,
- .resolve = btf_decl_tag_resolve,
- .check_member = btf_df_check_member,
- .check_kflag_member = btf_df_check_kflag_member,
- .log_details = btf_decl_tag_log,
- .show = btf_df_show,
-};
-
-static int btf_func_proto_check(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_type *ret_type;
- const struct btf_param *args;
- const struct btf *btf;
- u16 nr_args, i;
- int err;
-
- btf = env->btf;
- args = (const struct btf_param *)(t + 1);
- nr_args = btf_type_vlen(t);
-
- /* Check func return type which could be "void" (t->type == 0) */
- if (t->type) {
- u32 ret_type_id = t->type;
-
- ret_type = btf_type_by_id(btf, ret_type_id);
- if (!ret_type) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
-
- if (btf_type_is_resolve_source_only(ret_type)) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
-
- if (btf_type_needs_resolve(ret_type) &&
- !env_type_is_resolved(env, ret_type_id)) {
- err = btf_resolve(env, ret_type, ret_type_id);
- if (err)
- return err;
- }
-
- /* Ensure the return type is a type that has a size */
- if (!btf_type_id_size(btf, &ret_type_id, NULL)) {
- btf_verifier_log_type(env, t, "Invalid return type");
- return -EINVAL;
- }
- }
-
- if (!nr_args)
- return 0;
-
- /* Last func arg type_id could be 0 if it is a vararg */
- if (!args[nr_args - 1].type) {
- if (args[nr_args - 1].name_off) {
- btf_verifier_log_type(env, t, "Invalid arg#%u",
- nr_args);
- return -EINVAL;
- }
- nr_args--;
- }
-
- for (i = 0; i < nr_args; i++) {
- const struct btf_type *arg_type;
- u32 arg_type_id;
-
- arg_type_id = args[i].type;
- arg_type = btf_type_by_id(btf, arg_type_id);
- if (!arg_type) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (btf_type_is_resolve_source_only(arg_type)) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (args[i].name_off &&
- (!btf_name_offset_valid(btf, args[i].name_off) ||
- !btf_name_valid_identifier(btf, args[i].name_off))) {
- btf_verifier_log_type(env, t,
- "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
-
- if (btf_type_needs_resolve(arg_type) &&
- !env_type_is_resolved(env, arg_type_id)) {
- err = btf_resolve(env, arg_type, arg_type_id);
- if (err)
- return err;
- }
-
- if (!btf_type_id_size(btf, &arg_type_id, NULL)) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
- }
-
- return 0;
-}
-
-static int btf_func_check(struct btf_verifier_env *env,
- const struct btf_type *t)
-{
- const struct btf_type *proto_type;
- const struct btf_param *args;
- const struct btf *btf;
- u16 nr_args, i;
-
- btf = env->btf;
- proto_type = btf_type_by_id(btf, t->type);
-
- if (!proto_type || !btf_type_is_func_proto(proto_type)) {
- btf_verifier_log_type(env, t, "Invalid type_id");
- return -EINVAL;
- }
-
- args = (const struct btf_param *)(proto_type + 1);
- nr_args = btf_type_vlen(proto_type);
- for (i = 0; i < nr_args; i++) {
- if (!args[i].name_off && args[i].type) {
- btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
- return -EINVAL;
- }
- }
-
- return 0;
-}
-
-static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = {
- [BTF_KIND_INT] = &int_ops,
- [BTF_KIND_PTR] = &ptr_ops,
- [BTF_KIND_ARRAY] = &array_ops,
- [BTF_KIND_STRUCT] = &struct_ops,
- [BTF_KIND_UNION] = &struct_ops,
- [BTF_KIND_ENUM] = &enum_ops,
- [BTF_KIND_FWD] = &fwd_ops,
- [BTF_KIND_TYPEDEF] = &modifier_ops,
- [BTF_KIND_VOLATILE] = &modifier_ops,
- [BTF_KIND_CONST] = &modifier_ops,
- [BTF_KIND_RESTRICT] = &modifier_ops,
- [BTF_KIND_FUNC] = &func_ops,
- [BTF_KIND_FUNC_PROTO] = &func_proto_ops,
- [BTF_KIND_VAR] = &var_ops,
- [BTF_KIND_DATASEC] = &datasec_ops,
- [BTF_KIND_FLOAT] = &float_ops,
- [BTF_KIND_DECL_TAG] = &decl_tag_ops,
- [BTF_KIND_TYPE_TAG] = &modifier_ops,
- [BTF_KIND_ENUM64] = &enum64_ops,
-};
-
-static s32 btf_check_meta(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 meta_left)
-{
- u32 saved_meta_left = meta_left;
- s32 var_meta_size;
-
- if (meta_left < sizeof(*t)) {
- btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu",
- env->log_type_id, meta_left, sizeof(*t));
- return -EINVAL;
- }
- meta_left -= sizeof(*t);
-
- if (t->info & ~BTF_INFO_MASK) {
- btf_verifier_log(env, "[%u] Invalid btf_info:%x",
- env->log_type_id, t->info);
- return -EINVAL;
- }
-
- if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
- BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
- btf_verifier_log(env, "[%u] Invalid kind:%u",
- env->log_type_id, BTF_INFO_KIND(t->info));
- return -EINVAL;
- }
-
- if (!btf_name_offset_valid(env->btf, t->name_off)) {
- btf_verifier_log(env, "[%u] Invalid name_offset:%u",
- env->log_type_id, t->name_off);
- return -EINVAL;
- }
-
- var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left);
- if (var_meta_size < 0)
- return var_meta_size;
-
- meta_left -= var_meta_size;
-
- return saved_meta_left - meta_left;
-}
-
-static int btf_check_all_metas(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- struct btf_header *hdr;
- void *cur, *end;
-
- hdr = &btf->hdr;
- cur = btf->nohdr_data + hdr->type_off;
- end = cur + hdr->type_len;
-
- env->log_type_id = btf->base_btf ? btf->start_id : 1;
- while (cur < end) {
- struct btf_type *t = cur;
- s32 meta_size;
-
- meta_size = btf_check_meta(env, t, end - cur);
- if (meta_size < 0)
- return meta_size;
-
- btf_add_type(env, t);
- cur += meta_size;
- env->log_type_id++;
- }
-
- return 0;
-}
-
-static bool btf_resolve_valid(struct btf_verifier_env *env,
- const struct btf_type *t,
- u32 type_id)
-{
- struct btf *btf = env->btf;
-
- if (!env_type_is_resolved(env, type_id))
- return false;
-
- if (btf_type_is_struct(t) || btf_type_is_datasec(t))
- return !btf_resolved_type_id(btf, type_id) &&
- !btf_resolved_type_size(btf, type_id);
-
- if (btf_type_is_decl_tag(t) || btf_type_is_func(t))
- return btf_resolved_type_id(btf, type_id) &&
- !btf_resolved_type_size(btf, type_id);
-
- if (btf_type_is_modifier(t) || btf_type_is_ptr(t) ||
- btf_type_is_var(t)) {
- t = btf_type_id_resolve(btf, &type_id);
- return t &&
- !btf_type_is_modifier(t) &&
- !btf_type_is_var(t) &&
- !btf_type_is_datasec(t);
- }
-
- if (btf_type_is_array(t)) {
- const struct btf_array *array = btf_type_array(t);
- const struct btf_type *elem_type;
- u32 elem_type_id = array->type;
- u32 elem_size;
-
- elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
- return elem_type && !btf_type_is_modifier(elem_type) &&
- (array->nelems * elem_size ==
- btf_resolved_type_size(btf, type_id));
- }
-
- return false;
-}
-
-static int btf_resolve(struct btf_verifier_env *env,
- const struct btf_type *t, u32 type_id)
-{
- u32 save_log_type_id = env->log_type_id;
- const struct resolve_vertex *v;
- int err = 0;
-
- env->resolve_mode = RESOLVE_TBD;
- env_stack_push(env, t, type_id);
- while (!err && (v = env_stack_peak(env))) {
- env->log_type_id = v->type_id;
- err = btf_type_ops(v->t)->resolve(env, v);
- }
-
- env->log_type_id = type_id;
- if (err == -E2BIG) {
- btf_verifier_log_type(env, t,
- "Exceeded max resolving depth:%u",
- MAX_RESOLVE_DEPTH);
- } else if (err == -EEXIST) {
- btf_verifier_log_type(env, t, "Loop detected");
- }
-
- /* Final sanity check */
- if (!err && !btf_resolve_valid(env, t, type_id)) {
- btf_verifier_log_type(env, t, "Invalid resolve state");
- err = -EINVAL;
- }
-
- env->log_type_id = save_log_type_id;
- return err;
-}
-
-static int btf_check_all_types(struct btf_verifier_env *env)
-{
- struct btf *btf = env->btf;
- const struct btf_type *t;
- u32 type_id, i;
- int err;
-
- err = env_resolve_init(env);
- if (err)
- return err;
-
- env->phase++;
- for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
- type_id = btf->start_id + i;
- t = btf_type_by_id(btf, type_id);
-
- env->log_type_id = type_id;
- if (btf_type_needs_resolve(t) &&
- !env_type_is_resolved(env, type_id)) {
- err = btf_resolve(env, t, type_id);
- if (err)
- return err;
- }
-
- if (btf_type_is_func_proto(t)) {
- err = btf_func_proto_check(env, t);
- if (err)
- return err;
- }
- }
-
- return 0;
-}
-
-static int btf_parse_type_sec(struct btf_verifier_env *env)
-{
- const struct btf_header *hdr = &env->btf->hdr;
- int err;
-
- /* Type section must align to 4 bytes */
- if (hdr->type_off & (sizeof(u32) - 1)) {
- btf_verifier_log(env, "Unaligned type_off");
- return -EINVAL;
- }
-
- if (!env->btf->base_btf && !hdr->type_len) {
- btf_verifier_log(env, "No type found");
- return -EINVAL;
- }
-
- err = btf_check_all_metas(env);
- if (err)
- return err;
-
- return btf_check_all_types(env);
-}
-
-static int btf_parse_str_sec(struct btf_verifier_env *env)
-{
- const struct btf_header *hdr;
- struct btf *btf = env->btf;
- const char *start, *end;
-
- hdr = &btf->hdr;
- start = btf->nohdr_data + hdr->str_off;
- end = start + hdr->str_len;
-
- if (end != btf->data + btf->data_size) {
- btf_verifier_log(env, "String section is not at the end");
- return -EINVAL;
- }
-
- btf->strings = start;
-
- if (btf->base_btf && !hdr->str_len)
- return 0;
- if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_NAME_OFFSET || end[-1]) {
- btf_verifier_log(env, "Invalid string section");
- return -EINVAL;
- }
- if (!btf->base_btf && start[0]) {
- btf_verifier_log(env, "Invalid string section");
- return -EINVAL;
- }
-
- return 0;
-}
-
-static const size_t btf_sec_info_offset[] = {
- offsetof(struct btf_header, type_off),
- offsetof(struct btf_header, str_off),
-};
-
-static int btf_sec_info_cmp(const void *a, const void *b)
-{
- const struct btf_sec_info *x = a;
- const struct btf_sec_info *y = b;
-
- return (int)(x->off - y->off) ? : (int)(x->len - y->len);
-}
+ if (rec->spin_lock_off >= 0 && rec->res_spin_lock_off >= 0) {
+ ret = -EINVAL;
+ goto end;
+ }
-static int btf_check_sec_info(struct btf_verifier_env *env,
- u32 btf_data_size)
-{
- struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
- u32 total, expected_total, i;
- const struct btf_header *hdr;
- const struct btf *btf;
-
- btf = env->btf;
- hdr = &btf->hdr;
-
- /* Populate the secs from hdr */
- for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++)
- secs[i] = *(struct btf_sec_info *)((void *)hdr +
- btf_sec_info_offset[i]);
-
- sort(secs, ARRAY_SIZE(btf_sec_info_offset),
- sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
-
- /* Check for gaps and overlap among sections */
- total = 0;
- expected_total = btf_data_size - hdr->hdr_len;
- for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++) {
- if (expected_total < secs[i].off) {
- btf_verifier_log(env, "Invalid section offset");
- return -EINVAL;
- }
- if (total < secs[i].off) {
- /* gap */
- btf_verifier_log(env, "Unsupported section found");
- return -EINVAL;
- }
- if (total > secs[i].off) {
- btf_verifier_log(env, "Section overlap found");
- return -EINVAL;
- }
- if (expected_total - total < secs[i].len) {
- btf_verifier_log(env,
- "Total section length too long");
- return -EINVAL;
- }
- total += secs[i].len;
+ /* bpf_{list_head, rb_node} require bpf_spin_lock */
+ if ((btf_record_has_field(rec, BPF_LIST_HEAD) ||
+ btf_record_has_field(rec, BPF_RB_ROOT)) &&
+ (rec->spin_lock_off < 0 && rec->res_spin_lock_off < 0)) {
+ ret = -EINVAL;
+ goto end;
}
- /* There is data other than hdr and known sections */
- if (expected_total != total) {
- btf_verifier_log(env, "Unsupported section found");
- return -EINVAL;
+ if (rec->refcount_off < 0 &&
+ btf_record_has_field(rec, BPF_LIST_NODE) &&
+ btf_record_has_field(rec, BPF_RB_NODE)) {
+ ret = -EINVAL;
+ goto end;
}
- return 0;
+ sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
+ NULL, rec);
+
+ return rec;
+end:
+ btf_record_free(rec);
+ return ERR_PTR(ret);
}
-static int btf_parse_hdr(struct btf_verifier_env *env)
+int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
{
- u32 hdr_len, hdr_copy, btf_data_size;
- const struct btf_header *hdr;
- struct btf *btf;
-
- btf = env->btf;
- btf_data_size = btf->data_size;
-
- if (btf_data_size < offsetofend(struct btf_header, hdr_len)) {
- btf_verifier_log(env, "hdr_len not found");
- return -EINVAL;
- }
-
- hdr = btf->data;
- hdr_len = hdr->hdr_len;
- if (btf_data_size < hdr_len) {
- btf_verifier_log(env, "btf_header not found");
- return -EINVAL;
- }
+ int i;
- /* Ensure the unsupported header fields are zero */
- if (hdr_len > sizeof(btf->hdr)) {
- u8 *expected_zero = btf->data + sizeof(btf->hdr);
- u8 *end = btf->data + hdr_len;
+ /* There are three types that signify ownership of some other type:
+ * kptr_ref, bpf_list_head, bpf_rb_root.
+ * kptr_ref only supports storing kernel types, which can't store
+ * references to program allocated local types.
+ *
+ * Hence we only need to ensure that bpf_{list_head,rb_root} ownership
+ * does not form cycles.
+ */
+ if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & (BPF_GRAPH_ROOT | BPF_UPTR)))
+ return 0;
+ for (i = 0; i < rec->cnt; i++) {
+ struct btf_struct_meta *meta;
+ const struct btf_type *t;
+ u32 btf_id;
- for (; expected_zero < end; expected_zero++) {
- if (*expected_zero) {
- btf_verifier_log(env, "Unsupported btf_header");
+ if (rec->fields[i].type == BPF_UPTR) {
+ /* The uptr only supports pinning one page and cannot
+ * point to a kernel struct
+ */
+ if (btf_is_kernel(rec->fields[i].kptr.btf))
+ return -EINVAL;
+ t = btf_type_by_id(rec->fields[i].kptr.btf,
+ rec->fields[i].kptr.btf_id);
+ if (!t->size)
+ return -EINVAL;
+ if (t->size > PAGE_SIZE)
return -E2BIG;
- }
+ continue;
}
- }
-
- hdr_copy = min_t(u32, hdr_len, sizeof(btf->hdr));
- memcpy(&btf->hdr, btf->data, hdr_copy);
- hdr = &btf->hdr;
-
- btf_verifier_log_hdr(env, btf_data_size);
-
- if (hdr->magic != BTF_MAGIC) {
- btf_verifier_log(env, "Invalid magic");
- return -EINVAL;
- }
-
- if (hdr->version != BTF_VERSION) {
- btf_verifier_log(env, "Unsupported version");
- return -ENOTSUPP;
- }
+ if (!(rec->fields[i].type & BPF_GRAPH_ROOT))
+ continue;
+ btf_id = rec->fields[i].graph_root.value_btf_id;
+ meta = btf_find_struct_meta(btf, btf_id);
+ if (!meta)
+ return -EFAULT;
+ rec->fields[i].graph_root.value_rec = meta->record;
- if (hdr->flags) {
- btf_verifier_log(env, "Unsupported flags");
- return -ENOTSUPP;
- }
+ /* We need to set value_rec for all root types, but no need
+ * to check ownership cycle for a type unless it's also a
+ * node type.
+ */
+ if (!(rec->field_mask & BPF_GRAPH_NODE))
+ continue;
- if (!btf->base_btf && btf_data_size == hdr->hdr_len) {
- btf_verifier_log(env, "No data");
- return -EINVAL;
+ /* We need to ensure ownership acyclicity among all types. The
+ * proper way to do it would be to topologically sort all BTF
+ * IDs based on the ownership edges, since there can be multiple
+ * bpf_{list_head,rb_node} in a type. Instead, we use the
+ * following resaoning:
+ *
+ * - A type can only be owned by another type in user BTF if it
+ * has a bpf_{list,rb}_node. Let's call these node types.
+ * - A type can only _own_ another type in user BTF if it has a
+ * bpf_{list_head,rb_root}. Let's call these root types.
+ *
+ * We ensure that if a type is both a root and node, its
+ * element types cannot be root types.
+ *
+ * To ensure acyclicity:
+ *
+ * When A is an root type but not a node, its ownership
+ * chain can be:
+ * A -> B -> C
+ * Where:
+ * - A is an root, e.g. has bpf_rb_root.
+ * - B is both a root and node, e.g. has bpf_rb_node and
+ * bpf_list_head.
+ * - C is only an root, e.g. has bpf_list_node
+ *
+ * When A is both a root and node, some other type already
+ * owns it in the BTF domain, hence it can not own
+ * another root type through any of the ownership edges.
+ * A -> B
+ * Where:
+ * - A is both an root and node.
+ * - B is only an node.
+ */
+ if (meta->record->field_mask & BPF_GRAPH_ROOT)
+ return -ELOOP;
}
-
- return btf_check_sec_info(env, btf_data_size);
+ return 0;
}
-
static const char *alloc_obj_fields[] = {
"bpf_spin_lock",
"bpf_list_head",
@@ -5818,54 +1247,6 @@ struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id)
return NULL;
return bsearch(&btf_id, tab->types, tab->cnt, sizeof(tab->types[0]), btf_id_cmp_func);
}
-
-static int btf_check_type_tags(struct btf_verifier_env *env,
- struct btf *btf, int start_id)
-{
- int i, n, good_id = start_id - 1;
- bool in_tags;
-
- n = btf_nr_types(btf);
- for (i = start_id; i < n; i++) {
- const struct btf_type *t;
- int chain_limit = 32;
- u32 cur_id = i;
-
- t = btf_type_by_id(btf, i);
- if (!t)
- return -EINVAL;
- if (!btf_type_is_modifier(t))
- continue;
-
- cond_resched();
-
- in_tags = btf_type_is_type_tag(t);
- while (btf_type_is_modifier(t)) {
- if (!chain_limit--) {
- btf_verifier_log(env, "Max chain length or cycle detected");
- return -ELOOP;
- }
- if (btf_type_is_type_tag(t)) {
- if (!in_tags) {
- btf_verifier_log(env, "Type tags don't precede modifiers");
- return -EINVAL;
- }
- } else if (in_tags) {
- in_tags = false;
- }
- if (cur_id <= good_id)
- break;
- /* Move to next type */
- cur_id = t->type;
- t = btf_type_by_id(btf, cur_id);
- if (!t)
- return -EINVAL;
- }
- good_id = i;
- }
- return 0;
-}
-
static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)
{
u32 log_true_size;
@@ -6313,60 +1694,6 @@ int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_ty
}
BTF_ID_LIST_SINGLE(bpf_ctx_convert_btf_id, struct, bpf_ctx_convert)
-
-static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
- void *data, unsigned int data_size)
-{
- struct btf *btf = NULL;
- int err;
-
- if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
- return ERR_PTR(-ENOENT);
-
- btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
- if (!btf) {
- err = -ENOMEM;
- goto errout;
- }
- env->btf = btf;
-
- btf->data = data;
- btf->data_size = data_size;
- btf->kernel_btf = true;
- btf->named_start_id = 0;
- strscpy(btf->name, name);
-
- err = btf_parse_hdr(env);
- if (err)
- goto errout;
-
- btf->nohdr_data = btf->data + btf->hdr.hdr_len;
-
- err = btf_parse_str_sec(env);
- if (err)
- goto errout;
-
- err = btf_check_all_metas(env);
- if (err)
- goto errout;
-
- err = btf_check_type_tags(env, btf, 1);
- if (err)
- goto errout;
-
- btf_check_sorted(btf);
- refcount_set(&btf->refcnt, 1);
-
- return btf;
-
-errout:
- if (btf) {
- kvfree(btf->types);
- kfree(btf);
- }
- return ERR_PTR(err);
-}
-
struct btf *btf_parse_vmlinux(void)
{
struct btf_verifier_env *env = NULL;
@@ -8029,96 +3356,6 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return 0;
}
-
-static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
- struct btf_show *show)
-{
- const struct btf_type *t = btf_type_by_id(btf, type_id);
-
- show->btf = btf;
- memset(&show->state, 0, sizeof(show->state));
- memset(&show->obj, 0, sizeof(show->obj));
-
- btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
-}
-
-__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,
- va_list args)
-{
- seq_vprintf((struct seq_file *)show->target, fmt, args);
-}
-
-int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
- void *obj, struct seq_file *m, u64 flags)
-{
- struct btf_show sseq;
-
- sseq.target = m;
- sseq.showfn = btf_seq_show;
- sseq.flags = flags;
-
- btf_type_show(btf, type_id, obj, &sseq);
-
- return sseq.state.status;
-}
-
-void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
- struct seq_file *m)
-{
- (void) btf_type_seq_show_flags(btf, type_id, obj, m,
- BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
- BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
-}
-
-struct btf_show_snprintf {
- struct btf_show show;
- int len_left; /* space left in string */
- int len; /* length we would have written */
-};
-
-__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
- va_list args)
-{
- struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
- int len;
-
- len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
-
- if (len < 0) {
- ssnprintf->len_left = 0;
- ssnprintf->len = len;
- } else if (len >= ssnprintf->len_left) {
- /* no space, drive on to get length we would have written */
- ssnprintf->len_left = 0;
- ssnprintf->len += len;
- } else {
- ssnprintf->len_left -= len;
- ssnprintf->len += len;
- show->target += len;
- }
-}
-
-int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
- char *buf, int len, u64 flags)
-{
- struct btf_show_snprintf ssnprintf;
-
- ssnprintf.show.target = buf;
- ssnprintf.show.flags = flags;
- ssnprintf.show.showfn = btf_snprintf_show;
- ssnprintf.len_left = len;
- ssnprintf.len = 0;
-
- btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
-
- /* If we encountered an error, return it. */
- if (ssnprintf.show.state.status)
- return ssnprintf.show.state.status;
-
- /* Otherwise return length we would have written */
- return ssnprintf.len;
-}
-
#ifdef CONFIG_PROC_FS
static void bpf_btf_show_fdinfo(struct seq_file *m, struct file *filp)
{
@@ -8271,17 +3508,6 @@ u32 btf_obj_id(const struct btf *btf)
{
return READ_ONCE(btf->id);
}
-
-bool btf_is_kernel(const struct btf *btf)
-{
- return btf->kernel_btf;
-}
-
-bool btf_is_module(const struct btf *btf)
-{
- return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
-}
-
enum {
BTF_MODULE_F_LIVE = (1 << 0),
};
@@ -9740,21 +4966,3 @@ int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
}
EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
#endif
-
-bool btf_param_match_suffix(const struct btf *btf,
- const struct btf_param *arg,
- const char *suffix)
-{
- int suffix_len = strlen(suffix), len;
- const char *param_name;
-
- /* In the future, this can be ported to use BTF tagging */
- param_name = btf_name_by_offset(btf, arg->name_off);
- if (str_is_empty(param_name))
- return false;
- len = strlen(param_name);
- if (len <= suffix_len)
- return false;
- param_name += len - suffix_len;
- return !strncmp(param_name, suffix, suffix_len);
-}
diff --git a/kernel/btf/btf.c b/kernel/btf/btf.c
new file mode 100644
index 0000000000000..0eb383926b7d3
--- /dev/null
+++ b/kernel/btf/btf.c
@@ -0,0 +1,4803 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2018 Facebook */
+
+#include <uapi/linux/btf.h>
+#include <uapi/linux/bpf.h>
+#include <uapi/linux/types.h>
+#include <linux/seq_file.h>
+#include <linux/compiler.h>
+#include <linux/ctype.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/file.h>
+#include <linux/kernel.h>
+#include <linux/sort.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/bsearch.h>
+#include <linux/kobject.h>
+#include <linux/string.h>
+#include <linux/overflow.h>
+#include "btf.h"
+
+#ifndef CONFIG_BPF_SYSCALL
+/*
+ * When CONFIG_BPF_SYSCALL is not set, log.c is not compiled, so
+ * bpf_verifier_vlog() is unavailable. Provide a simple implementation
+ * that handles the BPF_LOG_KERNEL case (pr_err), which is the only
+ * log level used by the core BTF verifier for kernel BTF parsing.
+ */
+void bpf_verifier_vlog(struct bpf_verifier_log *log, const char *fmt,
+ va_list args)
+{
+ char buf[256];
+
+ vscnprintf(buf, sizeof(buf), fmt, args);
+ if (log->level == BPF_LOG_KERNEL)
+ pr_err("BPF: %s", buf);
+}
+
+__printf(2, 3) void bpf_log(struct bpf_verifier_log *log,
+ const char *fmt, ...)
+{
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+#endif /* !CONFIG_BPF_SYSCALL */
+
+/* BTF (BPF Type Format) is the meta data format which describes
+ * the data types of BPF program/map. Hence, it basically focus
+ * on the C programming language which the modern BPF is primary
+ * using.
+ *
+ * ELF Section:
+ * ~~~~~~~~~~~
+ * The BTF data is stored under the ".BTF" ELF section
+ *
+ * struct btf_type:
+ * ~~~~~~~~~~~~~~~
+ * Each 'struct btf_type' object describes a C data type.
+ * Depending on the type it is describing, a 'struct btf_type'
+ * object may be followed by more data. F.e.
+ * To describe an array, 'struct btf_type' is followed by
+ * 'struct btf_array'.
+ *
+ * 'struct btf_type' and any extra data following it are
+ * 4 bytes aligned.
+ *
+ * Type section:
+ * ~~~~~~~~~~~~~
+ * The BTF type section contains a list of 'struct btf_type' objects.
+ * Each one describes a C type. Recall from the above section
+ * that a 'struct btf_type' object could be immediately followed by extra
+ * data in order to describe some particular C types.
+ *
+ * type_id:
+ * ~~~~~~~
+ * Each btf_type object is identified by a type_id. The type_id
+ * is implicitly implied by the location of the btf_type object in
+ * the BTF type section. The first one has type_id 1. The second
+ * one has type_id 2...etc. Hence, an earlier btf_type has
+ * a smaller type_id.
+ *
+ * A btf_type object may refer to another btf_type object by using
+ * type_id (i.e. the "type" in the "struct btf_type").
+ *
+ * NOTE that we cannot assume any reference-order.
+ * A btf_type object can refer to an earlier btf_type object
+ * but it can also refer to a later btf_type object.
+ *
+ * For example, to describe "const void *". A btf_type
+ * object describing "const" may refer to another btf_type
+ * object describing "void *". This type-reference is done
+ * by specifying type_id:
+ *
+ * [1] CONST (anon) type_id=2
+ * [2] PTR (anon) type_id=0
+ *
+ * The above is the btf_verifier debug log:
+ * - Each line started with "[?]" is a btf_type object
+ * - [?] is the type_id of the btf_type object.
+ * - CONST/PTR is the BTF_KIND_XXX
+ * - "(anon)" is the name of the type. It just
+ * happens that CONST and PTR has no name.
+ * - type_id=XXX is the 'u32 type' in btf_type
+ *
+ * NOTE: "void" has type_id 0
+ *
+ * String section:
+ * ~~~~~~~~~~~~~~
+ * The BTF string section contains the names used by the type section.
+ * Each string is referred by an "offset" from the beginning of the
+ * string section.
+ *
+ * Each string is '\0' terminated.
+ *
+ * The first character in the string section must be '\0'
+ * which is used to mean 'anonymous'. Some btf_type may not
+ * have a name.
+ */
+
+/* BTF verification:
+ *
+ * To verify BTF data, two passes are needed.
+ *
+ * Pass #1
+ * ~~~~~~~
+ * The first pass is to collect all btf_type objects to
+ * an array: "btf->types".
+ *
+ * Depending on the C type that a btf_type is describing,
+ * a btf_type may be followed by extra data. We don't know
+ * how many btf_type is there, and more importantly we don't
+ * know where each btf_type is located in the type section.
+ *
+ * Without knowing the location of each type_id, most verifications
+ * cannot be done. e.g. an earlier btf_type may refer to a later
+ * btf_type (recall the "const void *" above), so we cannot
+ * check this type-reference in the first pass.
+ *
+ * In the first pass, it still does some verifications (e.g.
+ * checking the name is a valid offset to the string section).
+ *
+ * Pass #2
+ * ~~~~~~~
+ * The main focus is to resolve a btf_type that is referring
+ * to another type.
+ *
+ * We have to ensure the referring type:
+ * 1) does exist in the BTF (i.e. in btf->types[])
+ * 2) does not cause a loop:
+ * struct A {
+ * struct B b;
+ * };
+ *
+ * struct B {
+ * struct A a;
+ * };
+ *
+ * btf_type_needs_resolve() decides if a btf_type needs
+ * to be resolved.
+ *
+ * The needs_resolve type implements the "resolve()" ops which
+ * essentially does a DFS and detects backedge.
+ *
+ * During resolve (or DFS), different C types have different
+ * "RESOLVED" conditions.
+ *
+ * When resolving a BTF_KIND_STRUCT, we need to resolve all its
+ * members because a member is always referring to another
+ * type. A struct's member can be treated as "RESOLVED" if
+ * it is referring to a BTF_KIND_PTR. Otherwise, the
+ * following valid C struct would be rejected:
+ *
+ * struct A {
+ * int m;
+ * struct A *a;
+ * };
+ *
+ * When resolving a BTF_KIND_PTR, it needs to keep resolving if
+ * it is referring to another BTF_KIND_PTR. Otherwise, we cannot
+ * detect a pointer loop, e.g.:
+ * BTF_KIND_CONST -> BTF_KIND_PTR -> BTF_KIND_CONST -> BTF_KIND_PTR +
+ * ^ |
+ * +-----------------------------------------+
+ *
+ */
+
+#define BITS_PER_U128 (sizeof(u64) * BITS_PER_BYTE * 2)
+
+#define BTF_INFO_MASK 0x9f00ffff
+#define BTF_INT_MASK 0x0fffffff
+#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
+#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
+
+
+#define for_each_member_from(i, from, struct_type, member) \
+ for (i = from, member = btf_type_member(struct_type) + from; \
+ i < btf_type_vlen(struct_type); \
+ i++, member++)
+
+#define for_each_vsi_from(i, from, struct_type, member) \
+ for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
+ i < btf_type_vlen(struct_type); \
+ i++, member++)
+
+
+
+
+enum visit_state {
+ NOT_VISITED,
+ VISITED,
+ RESOLVED,
+};
+
+struct btf_sec_info {
+ u32 off;
+ u32 len;
+};
+
+const char * const btf_kind_str[NR_BTF_KINDS] = {
+ [BTF_KIND_UNKN] = "UNKNOWN",
+ [BTF_KIND_INT] = "INT",
+ [BTF_KIND_PTR] = "PTR",
+ [BTF_KIND_ARRAY] = "ARRAY",
+ [BTF_KIND_STRUCT] = "STRUCT",
+ [BTF_KIND_UNION] = "UNION",
+ [BTF_KIND_ENUM] = "ENUM",
+ [BTF_KIND_FWD] = "FWD",
+ [BTF_KIND_TYPEDEF] = "TYPEDEF",
+ [BTF_KIND_VOLATILE] = "VOLATILE",
+ [BTF_KIND_CONST] = "CONST",
+ [BTF_KIND_RESTRICT] = "RESTRICT",
+ [BTF_KIND_FUNC] = "FUNC",
+ [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO",
+ [BTF_KIND_VAR] = "VAR",
+ [BTF_KIND_DATASEC] = "DATASEC",
+ [BTF_KIND_FLOAT] = "FLOAT",
+ [BTF_KIND_DECL_TAG] = "DECL_TAG",
+ [BTF_KIND_TYPE_TAG] = "TYPE_TAG",
+ [BTF_KIND_ENUM64] = "ENUM64",
+};
+
+const char *btf_type_str(const struct btf_type *t)
+{
+ return btf_kind_str[BTF_INFO_KIND(t->info)];
+}
+
+/* Chunk size we use in safe copy of data to be shown. */
+#define BTF_SHOW_OBJ_SAFE_SIZE 32
+
+/*
+ * This is the maximum size of a base type value (equivalent to a
+ * 128-bit int); if we are at the end of our safe buffer and have
+ * less than 16 bytes space we can't be assured of being able
+ * to copy the next type safely, so in such cases we will initiate
+ * a new copy.
+ */
+#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
+
+/* Type name size */
+#define BTF_SHOW_NAME_SIZE 80
+
+
+/*
+ * Common data to all BTF show operations. Private show functions can add
+ * their own data to a structure containing a struct btf_show and consult it
+ * in the show callback. See btf_type_show() below.
+ *
+ * One challenge with showing nested data is we want to skip 0-valued
+ * data, but in order to figure out whether a nested object is all zeros
+ * we need to walk through it. As a result, we need to make two passes
+ * when handling structs, unions and arrays; the first path simply looks
+ * for nonzero data, while the second actually does the display. The first
+ * pass is signalled by show->state.depth_check being set, and if we
+ * encounter a non-zero value we set show->state.depth_to_show to
+ * the depth at which we encountered it. When we have completed the
+ * first pass, we will know if anything needs to be displayed if
+ * depth_to_show > depth. See btf_[struct,array]_show() for the
+ * implementation of this.
+ *
+ * Another problem is we want to ensure the data for display is safe to
+ * access. To support this, the anonymous "struct {} obj" tracks the data
+ * object and our safe copy of it. We copy portions of the data needed
+ * to the object "copy" buffer, but because its size is limited to
+ * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
+ * traverse larger objects for display.
+ *
+ * The various data type show functions all start with a call to
+ * btf_show_start_type() which returns a pointer to the safe copy
+ * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
+ * raw data itself). btf_show_obj_safe() is responsible for
+ * using copy_from_kernel_nofault() to update the safe data if necessary
+ * as we traverse the object's data. skbuff-like semantics are
+ * used:
+ *
+ * - obj.head points to the start of the toplevel object for display
+ * - obj.size is the size of the toplevel object
+ * - obj.data points to the current point in the original data at
+ * which our safe data starts. obj.data will advance as we copy
+ * portions of the data.
+ *
+ * In most cases a single copy will suffice, but larger data structures
+ * such as "struct task_struct" will require many copies. The logic in
+ * btf_show_obj_safe() handles the logic that determines if a new
+ * copy_from_kernel_nofault() is needed.
+ */
+struct btf_show {
+ u64 flags;
+ void *target; /* target of show operation (seq file, buffer) */
+ __printf(2, 0) void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
+ const struct btf *btf;
+ /* below are used during iteration */
+ struct {
+ u8 depth;
+ u8 depth_to_show;
+ u8 depth_check;
+ u8 array_member:1,
+ array_terminated:1;
+ u16 array_encoding;
+ u32 type_id;
+ int status; /* non-zero for error */
+ const struct btf_type *type;
+ const struct btf_member *member;
+ char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
+ } state;
+ struct {
+ u32 size;
+ void *head;
+ void *data;
+ u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
+ } obj;
+};
+
+struct btf_kind_operations {
+ s32 (*check_meta)(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left);
+ int (*resolve)(struct btf_verifier_env *env,
+ const struct resolve_vertex *v);
+ int (*check_member)(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type);
+ int (*check_kflag_member)(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type);
+ void (*log_details)(struct btf_verifier_env *env,
+ const struct btf_type *t);
+ void (*show)(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offsets,
+ struct btf_show *show);
+};
+
+static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
+static struct btf_type btf_void;
+
+static int btf_resolve(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id);
+
+static int btf_func_check(struct btf_verifier_env *env,
+ const struct btf_type *t);
+
+bool btf_type_is_modifier(const struct btf_type *t)
+{
+ /* Some of them is not strictly a C modifier
+ * but they are grouped into the same bucket
+ * for BTF concern:
+ * A type (t) that refers to another
+ * type through t->type AND its size cannot
+ * be determined without following the t->type.
+ *
+ * ptr does not fall into this bucket
+ * because its size is always sizeof(void *).
+ */
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_TYPEDEF:
+ case BTF_KIND_VOLATILE:
+ case BTF_KIND_CONST:
+ case BTF_KIND_RESTRICT:
+ case BTF_KIND_TYPE_TAG:
+ return true;
+ }
+
+ return false;
+}
+
+static int btf_start_id(const struct btf *btf)
+{
+ return btf->start_id + (btf->base_btf ? 0 : 1);
+}
+
+bool btf_type_is_void(const struct btf_type *t)
+{
+ return t == &btf_void;
+}
+
+bool btf_type_is_datasec(const struct btf_type *t)
+{
+ return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
+}
+
+bool btf_type_is_decl_tag(const struct btf_type *t)
+{
+ return BTF_INFO_KIND(t->info) == BTF_KIND_DECL_TAG;
+}
+
+static bool btf_type_nosize(const struct btf_type *t)
+{
+ return btf_type_is_void(t) || btf_type_is_fwd(t) ||
+ btf_type_is_func(t) || btf_type_is_func_proto(t) ||
+ btf_type_is_decl_tag(t);
+}
+
+static bool btf_type_nosize_or_null(const struct btf_type *t)
+{
+ return !t || btf_type_nosize(t);
+}
+
+static bool btf_type_is_decl_tag_target(const struct btf_type *t)
+{
+ return btf_type_is_func(t) || btf_type_is_struct(t) ||
+ btf_type_is_var(t) || btf_type_is_typedef(t);
+}
+
+bool btf_is_vmlinux(const struct btf *btf)
+{
+ return btf->kernel_btf && !btf->base_btf;
+}
+
+u32 btf_nr_types(const struct btf *btf)
+{
+ u32 total = 0;
+
+ while (btf) {
+ total += btf->nr_types;
+ btf = btf->base_btf;
+ }
+
+ return total;
+}
+
+/*
+ * Note that vmlinux and kernel module BTFs are always sorted
+ * during the building phase.
+ */
+void btf_check_sorted(struct btf *btf)
+{
+ u32 i, n, named_start_id = 0;
+
+ n = btf_nr_types(btf);
+ if (btf_is_vmlinux(btf)) {
+ for (i = btf_start_id(btf); i < n; i++) {
+ const struct btf_type *t = btf_type_by_id(btf, i);
+ const char *n = btf_name_by_offset(btf, t->name_off);
+
+ if (n[0] != '\0') {
+ btf->named_start_id = i;
+ return;
+ }
+ }
+ return;
+ }
+
+ for (i = btf_start_id(btf) + 1; i < n; i++) {
+ const struct btf_type *ta = btf_type_by_id(btf, i - 1);
+ const struct btf_type *tb = btf_type_by_id(btf, i);
+ const char *na = btf_name_by_offset(btf, ta->name_off);
+ const char *nb = btf_name_by_offset(btf, tb->name_off);
+
+ if (strcmp(na, nb) > 0)
+ return;
+
+ if (named_start_id == 0 && na[0] != '\0')
+ named_start_id = i - 1;
+ if (named_start_id == 0 && nb[0] != '\0')
+ named_start_id = i;
+ }
+
+ if (named_start_id)
+ btf->named_start_id = named_start_id;
+}
+
+/*
+ * btf_named_start_id - Get the named starting ID for the BTF
+ * @btf: Pointer to the target BTF object
+ * @own: Flag indicating whether to query only the current BTF (true = current BTF only,
+ * false = recursively traverse the base BTF chain)
+ *
+ * Return value rules:
+ * 1. For a sorted btf, return its named_start_id
+ * 2. Else for a split BTF, return its start_id
+ * 3. Else for a base BTF, return 1
+ */
+u32 btf_named_start_id(const struct btf *btf, bool own)
+{
+ const struct btf *base_btf = btf;
+
+ while (!own && base_btf->base_btf)
+ base_btf = base_btf->base_btf;
+
+ return base_btf->named_start_id ?: (base_btf->start_id ?: 1);
+}
+
+static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)
+{
+ const struct btf_type *t;
+ const char *tname;
+ s32 l, r, m;
+
+ l = btf_named_start_id(btf, true);
+ r = btf_nr_types(btf) - 1;
+ while (l <= r) {
+ m = l + (r - l) / 2;
+ t = btf_type_by_id(btf, m);
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) >= 0) {
+ if (l == r)
+ return r;
+ r = m;
+ } else {
+ l = m + 1;
+ }
+ }
+
+ return btf_nr_types(btf);
+}
+
+s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
+{
+ const struct btf *base_btf = btf_base_btf(btf);
+ const struct btf_type *t;
+ const char *tname;
+ s32 id, total;
+
+ if (base_btf) {
+ id = btf_find_by_name_kind(base_btf, name, kind);
+ if (id > 0)
+ return id;
+ }
+
+ total = btf_nr_types(btf);
+ if (btf->named_start_id > 0 && name[0]) {
+ id = btf_find_by_name_kind_bsearch(btf, name);
+ for (; id < total; id++) {
+ t = btf_type_by_id(btf, id);
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) != 0)
+ return -ENOENT;
+ if (BTF_INFO_KIND(t->info) == kind)
+ return id;
+ }
+ } else {
+ for (id = btf_start_id(btf); id < total; id++) {
+ t = btf_type_by_id(btf, id);
+ if (BTF_INFO_KIND(t->info) != kind)
+ continue;
+ tname = btf_name_by_offset(btf, t->name_off);
+ if (strcmp(tname, name) == 0)
+ return id;
+ }
+ }
+
+ return -ENOENT;
+}
+
+
+const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *t = btf_type_by_id(btf, id);
+
+ while (btf_type_is_modifier(t)) {
+ id = t->type;
+ t = btf_type_by_id(btf, t->type);
+ }
+
+ if (res_id)
+ *res_id = id;
+
+ return t;
+}
+
+const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *t;
+
+ t = btf_type_skip_modifiers(btf, id, NULL);
+ if (!btf_type_is_ptr(t))
+ return NULL;
+
+ return btf_type_skip_modifiers(btf, t->type, res_id);
+}
+
+const struct btf_type *btf_type_resolve_func_ptr(const struct btf *btf,
+ u32 id, u32 *res_id)
+{
+ const struct btf_type *ptype;
+
+ ptype = btf_type_resolve_ptr(btf, id, res_id);
+ if (ptype && btf_type_is_func_proto(ptype))
+ return ptype;
+
+ return NULL;
+}
+
+/* Types that act only as a source, not sink or intermediate
+ * type when resolving.
+ */
+static bool btf_type_is_resolve_source_only(const struct btf_type *t)
+{
+ return btf_type_is_var(t) ||
+ btf_type_is_decl_tag(t) ||
+ btf_type_is_datasec(t);
+}
+
+/* What types need to be resolved?
+ *
+ * btf_type_is_modifier() is an obvious one.
+ *
+ * btf_type_is_struct() because its member refers to
+ * another type (through member->type).
+ *
+ * btf_type_is_var() because the variable refers to
+ * another type. btf_type_is_datasec() holds multiple
+ * btf_type_is_var() types that need resolving.
+ *
+ * btf_type_is_array() because its element (array->type)
+ * refers to another type. Array can be thought of a
+ * special case of struct while array just has the same
+ * member-type repeated by array->nelems of times.
+ */
+static bool btf_type_needs_resolve(const struct btf_type *t)
+{
+ return btf_type_is_modifier(t) ||
+ btf_type_is_ptr(t) ||
+ btf_type_is_struct(t) ||
+ btf_type_is_array(t) ||
+ btf_type_is_var(t) ||
+ btf_type_is_func(t) ||
+ btf_type_is_decl_tag(t) ||
+ btf_type_is_datasec(t);
+}
+
+/* t->size can be used */
+bool btf_type_has_size(const struct btf_type *t)
+{
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_INT:
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ case BTF_KIND_ENUM:
+ case BTF_KIND_DATASEC:
+ case BTF_KIND_FLOAT:
+ case BTF_KIND_ENUM64:
+ return true;
+ }
+
+ return false;
+}
+
+static const char *btf_int_encoding_str(u8 encoding)
+{
+ if (encoding == 0)
+ return "(none)";
+ else if (encoding == BTF_INT_SIGNED)
+ return "SIGNED";
+ else if (encoding == BTF_INT_CHAR)
+ return "CHAR";
+ else if (encoding == BTF_INT_BOOL)
+ return "BOOL";
+ else
+ return "UNKN";
+}
+
+static u32 btf_type_int(const struct btf_type *t)
+{
+ return *(u32 *)(t + 1);
+}
+
+static const struct btf_array *btf_type_array(const struct btf_type *t)
+{
+ return (const struct btf_array *)(t + 1);
+}
+
+static const struct btf_enum *btf_type_enum(const struct btf_type *t)
+{
+ return (const struct btf_enum *)(t + 1);
+}
+
+static const struct btf_var *btf_type_var(const struct btf_type *t)
+{
+ return (const struct btf_var *)(t + 1);
+}
+
+const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t)
+{
+ return (const struct btf_decl_tag *)(t + 1);
+}
+
+static const struct btf_enum64 *btf_type_enum64(const struct btf_type *t)
+{
+ return (const struct btf_enum64 *)(t + 1);
+}
+
+static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
+{
+ return kind_ops[BTF_INFO_KIND(t->info)];
+}
+
+static bool btf_name_offset_valid(const struct btf *btf, u32 offset)
+{
+ if (!BTF_STR_OFFSET_VALID(offset))
+ return false;
+
+ while (offset < btf->start_str_off)
+ btf = btf->base_btf;
+
+ offset -= btf->start_str_off;
+ return offset < btf->hdr.str_len;
+}
+
+static bool __btf_name_char_ok(char c, bool first)
+{
+ if ((first ? !isalpha(c) :
+ !isalnum(c)) &&
+ c != '_' &&
+ c != '.')
+ return false;
+ return true;
+}
+
+const char *btf_str_by_offset(const struct btf *btf, u32 offset)
+{
+ while (offset < btf->start_str_off)
+ btf = btf->base_btf;
+
+ offset -= btf->start_str_off;
+ if (offset < btf->hdr.str_len)
+ return &btf->strings[offset];
+
+ return NULL;
+}
+
+static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
+{
+ /* offset must be valid */
+ const char *src = btf_str_by_offset(btf, offset);
+ const char *src_limit;
+
+ if (!__btf_name_char_ok(*src, true))
+ return false;
+
+ /* set a limit on identifier length */
+ src_limit = src + KSYM_NAME_LEN;
+ src++;
+ while (*src && src < src_limit) {
+ if (!__btf_name_char_ok(*src, false))
+ return false;
+ src++;
+ }
+
+ return !*src;
+}
+
+/* Allow any printable character in DATASEC names */
+static bool btf_name_valid_section(const struct btf *btf, u32 offset)
+{
+ /* offset must be valid */
+ const char *src = btf_str_by_offset(btf, offset);
+ const char *src_limit;
+
+ if (!*src)
+ return false;
+
+ /* set a limit on identifier length */
+ src_limit = src + KSYM_NAME_LEN;
+ while (*src && src < src_limit) {
+ if (!isprint(*src))
+ return false;
+ src++;
+ }
+
+ return !*src;
+}
+
+const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
+{
+ const char *name;
+
+ if (!offset)
+ return "(anon)";
+
+ name = btf_str_by_offset(btf, offset);
+ return name ?: "(invalid-name-offset)";
+}
+
+const char *btf_name_by_offset(const struct btf *btf, u32 offset)
+{
+ return btf_str_by_offset(btf, offset);
+}
+
+const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ type_id -= btf->start_id;
+ if (type_id >= btf->nr_types)
+ return NULL;
+ return btf->types[type_id];
+}
+EXPORT_SYMBOL_GPL(btf_type_by_id);
+
+/*
+ * Check that the type @t is a regular int. This means that @t is not
+ * a bit field and it has the same size as either of u8/u16/u32/u64
+ * or __int128. If @expected_size is not zero, then size of @t should
+ * be the same. A caller should already have checked that the type @t
+ * is an integer.
+ */
+static bool __btf_type_int_is_regular(const struct btf_type *t, size_t expected_size)
+{
+ u32 int_data = btf_type_int(t);
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ u8 nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
+
+ return BITS_PER_BYTE_MASKED(nr_bits) == 0 &&
+ BTF_INT_OFFSET(int_data) == 0 &&
+ (nr_bytes <= 16 && is_power_of_2(nr_bytes)) &&
+ (expected_size == 0 || nr_bytes == expected_size);
+}
+
+static bool btf_type_int_is_regular(const struct btf_type *t)
+{
+ return __btf_type_int_is_regular(t, 0);
+}
+
+bool btf_type_is_i32(const struct btf_type *t)
+{
+ return btf_type_is_int(t) && __btf_type_int_is_regular(t, 4);
+}
+
+bool btf_type_is_i64(const struct btf_type *t)
+{
+ return btf_type_is_int(t) && __btf_type_int_is_regular(t, 8);
+}
+
+bool btf_type_is_primitive(const struct btf_type *t)
+{
+ return (btf_type_is_int(t) && btf_type_int_is_regular(t)) ||
+ btf_is_any_enum(t);
+}
+
+/*
+ * Check that given struct member is a regular int with expected
+ * offset and size.
+ */
+bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
+ const struct btf_member *m,
+ u32 expected_offset, u32 expected_size)
+{
+ const struct btf_type *t;
+ u32 id, int_data;
+ u8 nr_bits;
+
+ id = m->type;
+ t = btf_type_id_size(btf, &id, NULL);
+ if (!t || !btf_type_is_int(t))
+ return false;
+
+ int_data = btf_type_int(t);
+ nr_bits = BTF_INT_BITS(int_data);
+ if (btf_type_kflag(s)) {
+ u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
+ u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
+
+ /* if kflag set, int should be a regular int and
+ * bit offset should be at byte boundary.
+ */
+ return !bitfield_size &&
+ BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
+ BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
+ }
+
+ if (BTF_INT_OFFSET(int_data) ||
+ BITS_PER_BYTE_MASKED(m->offset) ||
+ BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
+ BITS_PER_BYTE_MASKED(nr_bits) ||
+ BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
+ return false;
+
+ return true;
+}
+
+/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
+static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
+ u32 id)
+{
+ const struct btf_type *t = btf_type_by_id(btf, id);
+
+ while (btf_type_is_modifier(t) &&
+ BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
+ t = btf_type_by_id(btf, t->type);
+ }
+
+ return t;
+}
+
+#define BTF_SHOW_MAX_ITER 10
+
+#define BTF_KIND_BIT(kind) (1ULL << kind)
+
+/*
+ * Populate show->state.name with type name information.
+ * Format of type name is
+ *
+ * [.member_name = ] (type_name)
+ */
+static const char *btf_show_name(struct btf_show *show)
+{
+ /* BTF_MAX_ITER array suffixes "[]" */
+ const char *array_suffixes = "[][][][][][][][][][]";
+ const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
+ /* BTF_MAX_ITER pointer suffixes "*" */
+ const char *ptr_suffixes = "**********";
+ const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
+ const char *name = NULL, *prefix = "", *parens = "";
+ const struct btf_member *m = show->state.member;
+ const struct btf_type *t;
+ const struct btf_array *array;
+ u32 id = show->state.type_id;
+ const char *member = NULL;
+ bool show_member = false;
+ u64 kinds = 0;
+ int i;
+
+ show->state.name[0] = '\0';
+
+ /*
+ * Don't show type name if we're showing an array member;
+ * in that case we show the array type so don't need to repeat
+ * ourselves for each member.
+ */
+ if (show->state.array_member)
+ return "";
+
+ /* Retrieve member name, if any. */
+ if (m) {
+ member = btf_name_by_offset(show->btf, m->name_off);
+ show_member = strlen(member) > 0;
+ id = m->type;
+ }
+
+ /*
+ * Start with type_id, as we have resolved the struct btf_type *
+ * via btf_modifier_show() past the parent typedef to the child
+ * struct, int etc it is defined as. In such cases, the type_id
+ * still represents the starting type while the struct btf_type *
+ * in our show->state points at the resolved type of the typedef.
+ */
+ t = btf_type_by_id(show->btf, id);
+ if (!t)
+ return "";
+
+ /*
+ * The goal here is to build up the right number of pointer and
+ * array suffixes while ensuring the type name for a typedef
+ * is represented. Along the way we accumulate a list of
+ * BTF kinds we have encountered, since these will inform later
+ * display; for example, pointer types will not require an
+ * opening "{" for struct, we will just display the pointer value.
+ *
+ * We also want to accumulate the right number of pointer or array
+ * indices in the format string while iterating until we get to
+ * the typedef/pointee/array member target type.
+ *
+ * We start by pointing at the end of pointer and array suffix
+ * strings; as we accumulate pointers and arrays we move the pointer
+ * or array string backwards so it will show the expected number of
+ * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
+ * and/or arrays and typedefs are supported as a precaution.
+ *
+ * We also want to get typedef name while proceeding to resolve
+ * type it points to so that we can add parentheses if it is a
+ * "typedef struct" etc.
+ */
+ for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
+
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_TYPEDEF:
+ if (!name)
+ name = btf_name_by_offset(show->btf,
+ t->name_off);
+ kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
+ id = t->type;
+ break;
+ case BTF_KIND_ARRAY:
+ kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
+ parens = "[";
+ if (!t)
+ return "";
+ array = btf_type_array(t);
+ if (array_suffix > array_suffixes)
+ array_suffix -= 2;
+ id = array->type;
+ break;
+ case BTF_KIND_PTR:
+ kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
+ if (ptr_suffix > ptr_suffixes)
+ ptr_suffix -= 1;
+ id = t->type;
+ break;
+ default:
+ id = 0;
+ break;
+ }
+ if (!id)
+ break;
+ t = btf_type_skip_qualifiers(show->btf, id);
+ }
+ /* We may not be able to represent this type; bail to be safe */
+ if (i == BTF_SHOW_MAX_ITER)
+ return "";
+
+ if (!name)
+ name = btf_name_by_offset(show->btf, t->name_off);
+
+ switch (BTF_INFO_KIND(t->info)) {
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
+ "struct" : "union";
+ /* if it's an array of struct/union, parens is already set */
+ if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
+ parens = "{";
+ break;
+ case BTF_KIND_ENUM:
+ case BTF_KIND_ENUM64:
+ prefix = "enum";
+ break;
+ default:
+ break;
+ }
+
+ /* pointer does not require parens */
+ if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
+ parens = "";
+ /* typedef does not require struct/union/enum prefix */
+ if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
+ prefix = "";
+
+ if (!name)
+ name = "";
+
+ /* Even if we don't want type name info, we want parentheses etc */
+ if (show->flags & BTF_SHOW_NONAME)
+ snprintf(show->state.name, sizeof(show->state.name), "%s",
+ parens);
+ else
+ snprintf(show->state.name, sizeof(show->state.name),
+ "%s%s%s(%s%s%s%s%s%s)%s",
+ /* first 3 strings comprise ".member = " */
+ show_member ? "." : "",
+ show_member ? member : "",
+ show_member ? " = " : "",
+ /* ...next is our prefix (struct, enum, etc) */
+ prefix,
+ strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
+ /* ...this is the type name itself */
+ name,
+ /* ...suffixed by the appropriate '*', '[]' suffixes */
+ strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
+ array_suffix, parens);
+
+ return show->state.name;
+}
+
+static const char *__btf_show_indent(struct btf_show *show)
+{
+ const char *indents = " ";
+ const char *indent = &indents[strlen(indents)];
+
+ if ((indent - show->state.depth) >= indents)
+ return indent - show->state.depth;
+ return indents;
+}
+
+static const char *btf_show_indent(struct btf_show *show)
+{
+ return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
+}
+
+static const char *btf_show_newline(struct btf_show *show)
+{
+ return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
+}
+
+static const char *btf_show_delim(struct btf_show *show)
+{
+ if (show->state.depth == 0)
+ return "";
+
+ if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
+ BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
+ return "|";
+
+ return ",";
+}
+
+__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
+{
+ va_list args;
+
+ if (!show->state.depth_check) {
+ va_start(args, fmt);
+ show->showfn(show, fmt, args);
+ va_end(args);
+ }
+}
+
+/* Macros are used here as btf_show_type_value[s]() prepends and appends
+ * format specifiers to the format specifier passed in; these do the work of
+ * adding indentation, delimiters etc while the caller simply has to specify
+ * the type value(s) in the format specifier + value(s).
+ */
+#define btf_show_type_value(show, fmt, value) \
+ do { \
+ if ((value) != (__typeof__(value))0 || \
+ (show->flags & BTF_SHOW_ZERO) || \
+ show->state.depth == 0) { \
+ btf_show(show, "%s%s" fmt "%s%s", \
+ btf_show_indent(show), \
+ btf_show_name(show), \
+ value, btf_show_delim(show), \
+ btf_show_newline(show)); \
+ if (show->state.depth > show->state.depth_to_show) \
+ show->state.depth_to_show = show->state.depth; \
+ } \
+ } while (0)
+
+#define btf_show_type_values(show, fmt, ...) \
+ do { \
+ btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
+ btf_show_name(show), \
+ __VA_ARGS__, btf_show_delim(show), \
+ btf_show_newline(show)); \
+ if (show->state.depth > show->state.depth_to_show) \
+ show->state.depth_to_show = show->state.depth; \
+ } while (0)
+
+/* How much is left to copy to safe buffer after @data? */
+static int btf_show_obj_size_left(struct btf_show *show, void *data)
+{
+ return show->obj.head + show->obj.size - data;
+}
+
+/* Is object pointed to by @data of @size already copied to our safe buffer? */
+static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
+{
+ return data >= show->obj.data &&
+ (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
+}
+
+/*
+ * If object pointed to by @data of @size falls within our safe buffer, return
+ * the equivalent pointer to the same safe data. Assumes
+ * copy_from_kernel_nofault() has already happened and our safe buffer is
+ * populated.
+ */
+static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
+{
+ if (btf_show_obj_is_safe(show, data, size))
+ return show->obj.safe + (data - show->obj.data);
+ return NULL;
+}
+
+/*
+ * Return a safe-to-access version of data pointed to by @data.
+ * We do this by copying the relevant amount of information
+ * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
+ *
+ * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
+ * safe copy is needed.
+ *
+ * Otherwise we need to determine if we have the required amount
+ * of data (determined by the @data pointer and the size of the
+ * largest base type we can encounter (represented by
+ * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
+ * that we will be able to print some of the current object,
+ * and if more is needed a copy will be triggered.
+ * Some objects such as structs will not fit into the buffer;
+ * in such cases additional copies when we iterate over their
+ * members may be needed.
+ *
+ * btf_show_obj_safe() is used to return a safe buffer for
+ * btf_show_start_type(); this ensures that as we recurse into
+ * nested types we always have safe data for the given type.
+ * This approach is somewhat wasteful; it's possible for example
+ * that when iterating over a large union we'll end up copying the
+ * same data repeatedly, but the goal is safety not performance.
+ * We use stack data as opposed to per-CPU buffers because the
+ * iteration over a type can take some time, and preemption handling
+ * would greatly complicate use of the safe buffer.
+ */
+static void *btf_show_obj_safe(struct btf_show *show,
+ const struct btf_type *t,
+ void *data)
+{
+ const struct btf_type *rt;
+ int size_left, size;
+ void *safe = NULL;
+
+ if (show->flags & BTF_SHOW_UNSAFE)
+ return data;
+
+ rt = btf_resolve_size(show->btf, t, &size);
+ if (IS_ERR(rt)) {
+ show->state.status = PTR_ERR(rt);
+ return NULL;
+ }
+
+ /*
+ * Is this toplevel object? If so, set total object size and
+ * initialize pointers. Otherwise check if we still fall within
+ * our safe object data.
+ */
+ if (show->state.depth == 0) {
+ show->obj.size = size;
+ show->obj.head = data;
+ } else {
+ /*
+ * If the size of the current object is > our remaining
+ * safe buffer we _may_ need to do a new copy. However
+ * consider the case of a nested struct; it's size pushes
+ * us over the safe buffer limit, but showing any individual
+ * struct members does not. In such cases, we don't need
+ * to initiate a fresh copy yet; however we definitely need
+ * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
+ * in our buffer, regardless of the current object size.
+ * The logic here is that as we resolve types we will
+ * hit a base type at some point, and we need to be sure
+ * the next chunk of data is safely available to display
+ * that type info safely. We cannot rely on the size of
+ * the current object here because it may be much larger
+ * than our current buffer (e.g. task_struct is 8k).
+ * All we want to do here is ensure that we can print the
+ * next basic type, which we can if either
+ * - the current type size is within the safe buffer; or
+ * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
+ * the safe buffer.
+ */
+ safe = __btf_show_obj_safe(show, data,
+ min(size,
+ BTF_SHOW_OBJ_BASE_TYPE_SIZE));
+ }
+
+ /*
+ * We need a new copy to our safe object, either because we haven't
+ * yet copied and are initializing safe data, or because the data
+ * we want falls outside the boundaries of the safe object.
+ */
+ if (!safe) {
+ size_left = btf_show_obj_size_left(show, data);
+ if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
+ size_left = BTF_SHOW_OBJ_SAFE_SIZE;
+ show->state.status = copy_from_kernel_nofault(show->obj.safe,
+ data, size_left);
+ if (!show->state.status) {
+ show->obj.data = data;
+ safe = show->obj.safe;
+ }
+ }
+
+ return safe;
+}
+
+/*
+ * Set the type we are starting to show and return a safe data pointer
+ * to be used for showing the associated data.
+ */
+static void *btf_show_start_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id, void *data)
+{
+ show->state.type = t;
+ show->state.type_id = type_id;
+ show->state.name[0] = '\0';
+
+ return btf_show_obj_safe(show, t, data);
+}
+
+static void btf_show_end_type(struct btf_show *show)
+{
+ show->state.type = NULL;
+ show->state.type_id = 0;
+ show->state.name[0] = '\0';
+}
+
+static void *btf_show_start_aggr_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id, void *data)
+{
+ void *safe_data = btf_show_start_type(show, t, type_id, data);
+
+ if (!safe_data)
+ return safe_data;
+
+ btf_show(show, "%s%s%s", btf_show_indent(show),
+ btf_show_name(show),
+ btf_show_newline(show));
+ show->state.depth++;
+ return safe_data;
+}
+
+static void btf_show_end_aggr_type(struct btf_show *show,
+ const char *suffix)
+{
+ show->state.depth--;
+ btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
+ btf_show_delim(show), btf_show_newline(show));
+ btf_show_end_type(show);
+}
+
+static void btf_show_start_member(struct btf_show *show,
+ const struct btf_member *m)
+{
+ show->state.member = m;
+}
+
+static void btf_show_start_array_member(struct btf_show *show)
+{
+ show->state.array_member = 1;
+ btf_show_start_member(show, NULL);
+}
+
+static void btf_show_end_member(struct btf_show *show)
+{
+ show->state.member = NULL;
+}
+
+static void btf_show_end_array_member(struct btf_show *show)
+{
+ show->state.array_member = 0;
+ btf_show_end_member(show);
+}
+
+static void *btf_show_start_array_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id,
+ u16 array_encoding,
+ void *data)
+{
+ show->state.array_encoding = array_encoding;
+ show->state.array_terminated = 0;
+ return btf_show_start_aggr_type(show, t, type_id, data);
+}
+
+static void btf_show_end_array_type(struct btf_show *show)
+{
+ show->state.array_encoding = 0;
+ show->state.array_terminated = 0;
+ btf_show_end_aggr_type(show, "]");
+}
+
+static void *btf_show_start_struct_type(struct btf_show *show,
+ const struct btf_type *t,
+ u32 type_id,
+ void *data)
+{
+ return btf_show_start_aggr_type(show, t, type_id, data);
+}
+
+static void btf_show_end_struct_type(struct btf_show *show)
+{
+ btf_show_end_aggr_type(show, "}");
+}
+
+__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
+ const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+
+__printf(2, 3) static void btf_verifier_log(struct btf_verifier_env *env,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+}
+
+__printf(4, 5) static void __btf_verifier_log_type(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ bool log_details,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ struct btf *btf = env->btf;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL) {
+ /* btf verifier prints all types it is processing via
+ * btf_verifier_log_type(..., fmt = NULL).
+ * Skip those prints for in-kernel BTF verification.
+ */
+ if (!fmt)
+ return;
+
+ /* Skip logging when loading module BTF with mismatches permitted */
+ if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
+ return;
+ }
+
+ __btf_verifier_log(log, "[%u] %s %s%s",
+ env->log_type_id,
+ btf_type_str(t),
+ __btf_name_by_offset(btf, t->name_off),
+ log_details ? " " : "");
+
+ if (log_details)
+ btf_type_ops(t)->log_details(env, t);
+
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+#define btf_verifier_log_type(env, t, ...) \
+ __btf_verifier_log_type((env), (t), true, __VA_ARGS__)
+#define btf_verifier_log_basic(env, t, ...) \
+ __btf_verifier_log_type((env), (t), false, __VA_ARGS__)
+
+__printf(4, 5)
+static void btf_verifier_log_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ struct btf *btf = env->btf;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL) {
+ if (!fmt)
+ return;
+
+ /* Skip logging when loading module BTF with mismatches permitted */
+ if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
+ return;
+ }
+
+ /* The CHECK_META phase already did a btf dump.
+ *
+ * If member is logged again, it must hit an error in
+ * parsing this member. It is useful to print out which
+ * struct this member belongs to.
+ */
+ if (env->phase != CHECK_META)
+ btf_verifier_log_type(env, struct_type, NULL);
+
+ if (btf_type_kflag(struct_type))
+ __btf_verifier_log(log,
+ "\t%s type_id=%u bitfield_size=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type,
+ BTF_MEMBER_BITFIELD_SIZE(member->offset),
+ BTF_MEMBER_BIT_OFFSET(member->offset));
+ else
+ __btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
+ __btf_name_by_offset(btf, member->name_off),
+ member->type, member->offset);
+
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+__printf(4, 5)
+static void btf_verifier_log_vsi(struct btf_verifier_env *env,
+ const struct btf_type *datasec_type,
+ const struct btf_var_secinfo *vsi,
+ const char *fmt, ...)
+{
+ struct bpf_verifier_log *log = &env->log;
+ va_list args;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+ if (log->level == BPF_LOG_KERNEL && !fmt)
+ return;
+ if (env->phase != CHECK_META)
+ btf_verifier_log_type(env, datasec_type, NULL);
+
+ __btf_verifier_log(log, "\t type_id=%u offset=%u size=%u",
+ vsi->type, vsi->offset, vsi->size);
+ if (fmt && *fmt) {
+ __btf_verifier_log(log, " ");
+ va_start(args, fmt);
+ bpf_verifier_vlog(log, fmt, args);
+ va_end(args);
+ }
+
+ __btf_verifier_log(log, "\n");
+}
+
+static void btf_verifier_log_hdr(struct btf_verifier_env *env,
+ u32 btf_data_size)
+{
+ struct bpf_verifier_log *log = &env->log;
+ const struct btf *btf = env->btf;
+ const struct btf_header *hdr;
+
+ if (!bpf_verifier_log_needed(log))
+ return;
+
+ if (log->level == BPF_LOG_KERNEL)
+ return;
+ hdr = &btf->hdr;
+ __btf_verifier_log(log, "magic: 0x%x\n", hdr->magic);
+ __btf_verifier_log(log, "version: %u\n", hdr->version);
+ __btf_verifier_log(log, "flags: 0x%x\n", hdr->flags);
+ __btf_verifier_log(log, "hdr_len: %u\n", hdr->hdr_len);
+ __btf_verifier_log(log, "type_off: %u\n", hdr->type_off);
+ __btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
+ __btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
+ __btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
+ __btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
+}
+
+static int btf_add_type(struct btf_verifier_env *env, struct btf_type *t)
+{
+ struct btf *btf = env->btf;
+
+ if (btf->types_size == btf->nr_types) {
+ /* Expand 'types' array */
+
+ struct btf_type **new_types;
+ u32 expand_by, new_size;
+
+ if (btf->start_id + btf->types_size == BTF_MAX_TYPE) {
+ btf_verifier_log(env, "Exceeded max num of types");
+ return -E2BIG;
+ }
+
+ expand_by = max_t(u32, btf->types_size >> 2, 16);
+ new_size = min_t(u32, BTF_MAX_TYPE,
+ btf->types_size + expand_by);
+
+ new_types = kvzalloc_objs(*new_types, new_size,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!new_types)
+ return -ENOMEM;
+
+ if (btf->nr_types == 0) {
+ if (!btf->base_btf) {
+ /* lazily init VOID type */
+ new_types[0] = &btf_void;
+ btf->nr_types++;
+ }
+ } else {
+ memcpy(new_types, btf->types,
+ sizeof(*btf->types) * btf->nr_types);
+ }
+
+ kvfree(btf->types);
+ btf->types = new_types;
+ btf->types_size = new_size;
+ }
+
+ btf->types[btf->nr_types++] = t;
+
+ return 0;
+}
+
+
+void __weak btf_free_bpf_data(struct btf *btf)
+{
+}
+
+void btf_free(struct btf *btf)
+{
+ btf_free_bpf_data(btf);
+ kvfree(btf->types);
+ kvfree(btf->resolved_sizes);
+ kvfree(btf->resolved_ids);
+ /* vmlinux does not allocate btf->data, it simply points it at
+ * __start_BTF.
+ */
+ if (!btf_is_vmlinux(btf))
+ kvfree(btf->data);
+ kvfree(btf->base_id_map);
+ kfree(btf);
+}
+
+
+const char *btf_get_name(const struct btf *btf)
+{
+ return btf->name;
+}
+
+void btf_get(struct btf *btf)
+{
+ refcount_inc(&btf->refcnt);
+}
+
+void __weak btf_put_bpf(struct btf *btf)
+{
+ btf_free(btf);
+}
+
+void btf_put(struct btf *btf)
+{
+ if (btf && refcount_dec_and_test(&btf->refcnt)) {
+ btf_put_bpf(btf);
+ }
+}
+
+struct btf *btf_base_btf(const struct btf *btf)
+{
+ return btf->base_btf;
+}
+
+const struct btf_header *btf_header(const struct btf *btf)
+{
+ return &btf->hdr;
+}
+
+void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
+{
+ btf->base_btf = (struct btf *)base_btf;
+ btf->start_id = btf_nr_types(base_btf);
+ btf->start_str_off = base_btf->hdr.str_len;
+}
+
+static int env_resolve_init(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ u32 nr_types = btf->nr_types;
+ u32 *resolved_sizes = NULL;
+ u32 *resolved_ids = NULL;
+ u8 *visit_states = NULL;
+
+ resolved_sizes = kvcalloc(nr_types, sizeof(*resolved_sizes),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!resolved_sizes)
+ goto nomem;
+
+ resolved_ids = kvcalloc(nr_types, sizeof(*resolved_ids),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!resolved_ids)
+ goto nomem;
+
+ visit_states = kvcalloc(nr_types, sizeof(*visit_states),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!visit_states)
+ goto nomem;
+
+ btf->resolved_sizes = resolved_sizes;
+ btf->resolved_ids = resolved_ids;
+ env->visit_states = visit_states;
+
+ return 0;
+
+nomem:
+ kvfree(resolved_sizes);
+ kvfree(resolved_ids);
+ kvfree(visit_states);
+ return -ENOMEM;
+}
+
+void btf_verifier_env_free(struct btf_verifier_env *env)
+{
+ kvfree(env->visit_states);
+ kfree(env);
+}
+
+static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
+ const struct btf_type *next_type)
+{
+ switch (env->resolve_mode) {
+ case RESOLVE_TBD:
+ /* int, enum or void is a sink */
+ return !btf_type_needs_resolve(next_type);
+ case RESOLVE_PTR:
+ /* int, enum, void, struct, array, func or func_proto is a sink
+ * for ptr
+ */
+ return !btf_type_is_modifier(next_type) &&
+ !btf_type_is_ptr(next_type);
+ case RESOLVE_STRUCT_OR_ARRAY:
+ /* int, enum, void, ptr, func or func_proto is a sink
+ * for struct and array
+ */
+ return !btf_type_is_modifier(next_type) &&
+ !btf_type_is_array(next_type) &&
+ !btf_type_is_struct(next_type);
+ default:
+ BUG();
+ }
+}
+
+static bool env_type_is_resolved(const struct btf_verifier_env *env,
+ u32 type_id)
+{
+ /* base BTF types should be resolved by now */
+ if (type_id < env->btf->start_id)
+ return true;
+
+ return env->visit_states[type_id - env->btf->start_id] == RESOLVED;
+}
+
+static int env_stack_push(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id)
+{
+ const struct btf *btf = env->btf;
+ struct resolve_vertex *v;
+
+ if (env->top_stack == MAX_RESOLVE_DEPTH)
+ return -E2BIG;
+
+ if (type_id < btf->start_id
+ || env->visit_states[type_id - btf->start_id] != NOT_VISITED)
+ return -EEXIST;
+
+ env->visit_states[type_id - btf->start_id] = VISITED;
+
+ v = &env->stack[env->top_stack++];
+ v->t = t;
+ v->type_id = type_id;
+ v->next_member = 0;
+
+ if (env->resolve_mode == RESOLVE_TBD) {
+ if (btf_type_is_ptr(t))
+ env->resolve_mode = RESOLVE_PTR;
+ else if (btf_type_is_struct(t) || btf_type_is_array(t))
+ env->resolve_mode = RESOLVE_STRUCT_OR_ARRAY;
+ }
+
+ return 0;
+}
+
+static void env_stack_set_next_member(struct btf_verifier_env *env,
+ u16 next_member)
+{
+ env->stack[env->top_stack - 1].next_member = next_member;
+}
+
+static void env_stack_pop_resolved(struct btf_verifier_env *env,
+ u32 resolved_type_id,
+ u32 resolved_size)
+{
+ u32 type_id = env->stack[--(env->top_stack)].type_id;
+ struct btf *btf = env->btf;
+
+ type_id -= btf->start_id; /* adjust to local type id */
+ btf->resolved_sizes[type_id] = resolved_size;
+ btf->resolved_ids[type_id] = resolved_type_id;
+ env->visit_states[type_id] = RESOLVED;
+}
+
+static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
+{
+ return env->top_stack ? &env->stack[env->top_stack - 1] : NULL;
+}
+
+/* Resolve the size of a passed-in "type"
+ *
+ * type: is an array (e.g. u32 array[x][y])
+ * return type: type "u32[x][y]", i.e. BTF_KIND_ARRAY,
+ * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
+ * corresponds to the return type.
+ * *elem_type: u32
+ * *elem_id: id of u32
+ * *total_nelems: (x * y). Hence, individual elem size is
+ * (*type_size / *total_nelems)
+ * *type_id: id of type if it's changed within the function, 0 if not
+ *
+ * type: is not an array (e.g. const struct X)
+ * return type: type "struct X"
+ * *type_size: sizeof(struct X)
+ * *elem_type: same as return type ("struct X")
+ * *elem_id: 0
+ * *total_nelems: 1
+ * *type_id: id of type if it's changed within the function, 0 if not
+ */
+const struct btf_type *
+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size, const struct btf_type **elem_type,
+ u32 *elem_id, u32 *total_nelems, u32 *type_id)
+{
+ const struct btf_type *array_type = NULL;
+ const struct btf_array *array = NULL;
+ u32 i, size, nelems = 1, id = 0;
+
+ for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
+ switch (BTF_INFO_KIND(type->info)) {
+ /* type->size can be used */
+ case BTF_KIND_INT:
+ case BTF_KIND_STRUCT:
+ case BTF_KIND_UNION:
+ case BTF_KIND_ENUM:
+ case BTF_KIND_FLOAT:
+ case BTF_KIND_ENUM64:
+ size = type->size;
+ goto resolved;
+
+ case BTF_KIND_PTR:
+ size = sizeof(void *);
+ goto resolved;
+
+ /* Modifiers */
+ case BTF_KIND_TYPEDEF:
+ case BTF_KIND_VOLATILE:
+ case BTF_KIND_CONST:
+ case BTF_KIND_RESTRICT:
+ case BTF_KIND_TYPE_TAG:
+ id = type->type;
+ type = btf_type_by_id(btf, type->type);
+ break;
+
+ case BTF_KIND_ARRAY:
+ if (!array_type)
+ array_type = type;
+ array = btf_type_array(type);
+ if (nelems && array->nelems > U32_MAX / nelems)
+ return ERR_PTR(-EINVAL);
+ nelems *= array->nelems;
+ type = btf_type_by_id(btf, array->type);
+ break;
+
+ /* type without size */
+ default:
+ return ERR_PTR(-EINVAL);
+ }
+ }
+
+ return ERR_PTR(-EINVAL);
+
+resolved:
+ if (nelems && size > U32_MAX / nelems)
+ return ERR_PTR(-EINVAL);
+
+ *type_size = nelems * size;
+ if (total_nelems)
+ *total_nelems = nelems;
+ if (elem_type)
+ *elem_type = type;
+ if (elem_id)
+ *elem_id = array ? array->type : 0;
+ if (type_id && id)
+ *type_id = id;
+
+ return array_type ? : type;
+}
+
+const struct btf_type *
+btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size)
+{
+ return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
+}
+
+static u32 btf_resolved_type_id(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ return btf->resolved_ids[type_id - btf->start_id];
+}
+
+/* The input param "type_id" must point to a needs_resolve type */
+static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
+ u32 *type_id)
+{
+ *type_id = btf_resolved_type_id(btf, *type_id);
+ return btf_type_by_id(btf, *type_id);
+}
+
+static u32 btf_resolved_type_size(const struct btf *btf, u32 type_id)
+{
+ while (type_id < btf->start_id)
+ btf = btf->base_btf;
+
+ return btf->resolved_sizes[type_id - btf->start_id];
+}
+
+const struct btf_type *btf_type_id_size(const struct btf *btf,
+ u32 *type_id, u32 *ret_size)
+{
+ const struct btf_type *size_type;
+ u32 size_type_id = *type_id;
+ u32 size = 0;
+
+ size_type = btf_type_by_id(btf, size_type_id);
+ if (btf_type_nosize_or_null(size_type))
+ return NULL;
+
+ if (btf_type_has_size(size_type)) {
+ size = size_type->size;
+ } else if (btf_type_is_array(size_type)) {
+ size = btf_resolved_type_size(btf, size_type_id);
+ } else if (btf_type_is_ptr(size_type)) {
+ size = sizeof(void *);
+ } else {
+ if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) &&
+ !btf_type_is_var(size_type)))
+ return NULL;
+
+ size_type_id = btf_resolved_type_id(btf, size_type_id);
+ size_type = btf_type_by_id(btf, size_type_id);
+ if (btf_type_nosize_or_null(size_type))
+ return NULL;
+ else if (btf_type_has_size(size_type))
+ size = size_type->size;
+ else if (btf_type_is_array(size_type))
+ size = btf_resolved_type_size(btf, size_type_id);
+ else if (btf_type_is_ptr(size_type))
+ size = sizeof(void *);
+ else
+ return NULL;
+ }
+
+ *type_id = size_type_id;
+ if (ret_size)
+ *ret_size = size;
+
+ return size_type;
+}
+
+static int btf_df_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ btf_verifier_log_basic(env, struct_type,
+ "Unsupported check_member");
+ return -EINVAL;
+}
+
+static int btf_df_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ btf_verifier_log_basic(env, struct_type,
+ "Unsupported check_kflag_member");
+ return -EINVAL;
+}
+
+/* Used for ptr, array struct/union and float type members.
+ * int, enum and modifier types have their specific callback functions.
+ */
+static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ /* bitfield size is 0, so member->offset represents bit offset only.
+ * It is safe to call non kflag check_member variants.
+ */
+ return btf_type_ops(member_type)->check_member(env, struct_type,
+ member,
+ member_type);
+}
+
+static int btf_df_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ btf_verifier_log_basic(env, v->t, "Unsupported resolve");
+ return -EINVAL;
+}
+
+static void btf_df_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offsets,
+ struct btf_show *show)
+{
+ btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
+}
+
+static int btf_int_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 int_data = btf_type_int(member_type);
+ u32 struct_bits_off = member->offset;
+ u32 struct_size = struct_type->size;
+ u32 nr_copy_bits;
+ u32 bytes_offset;
+
+ if (U32_MAX - struct_bits_off < BTF_INT_OFFSET(int_data)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "bits_offset exceeds U32_MAX");
+ return -EINVAL;
+ }
+
+ struct_bits_off += BTF_INT_OFFSET(int_data);
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ nr_copy_bits = BTF_INT_BITS(int_data) +
+ BITS_PER_BYTE_MASKED(struct_bits_off);
+
+ if (nr_copy_bits > BITS_PER_U128) {
+ btf_verifier_log_member(env, struct_type, member,
+ "nr_copy_bits exceeds 128");
+ return -EINVAL;
+ }
+
+ if (struct_size < bytes_offset ||
+ struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_int_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
+ u32 int_data = btf_type_int(member_type);
+ u32 struct_size = struct_type->size;
+ u32 nr_copy_bits;
+
+ /* a regular int type is required for the kflag int member */
+ if (!btf_type_int_is_regular(member_type)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member base type");
+ return -EINVAL;
+ }
+
+ /* check sanity of bitfield size */
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_int_data_bits = BTF_INT_BITS(int_data);
+ if (!nr_bits) {
+ /* Not a bitfield member, member offset must be at byte
+ * boundary.
+ */
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member offset");
+ return -EINVAL;
+ }
+
+ nr_bits = nr_int_data_bits;
+ } else if (nr_bits > nr_int_data_bits) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
+ if (nr_copy_bits > BITS_PER_U128) {
+ btf_verifier_log_member(env, struct_type, member,
+ "nr_copy_bits exceeds 128");
+ return -EINVAL;
+ }
+
+ if (struct_size < bytes_offset ||
+ struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_int_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 int_data, nr_bits, meta_needed = sizeof(int_data);
+ u16 encoding;
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ int_data = btf_type_int(t);
+ if (int_data & ~BTF_INT_MASK) {
+ btf_verifier_log_basic(env, t, "Invalid int_data:%x",
+ int_data);
+ return -EINVAL;
+ }
+
+ nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
+
+ if (nr_bits > BITS_PER_U128) {
+ btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
+ BITS_PER_U128);
+ return -EINVAL;
+ }
+
+ if (BITS_ROUNDUP_BYTES(nr_bits) > t->size) {
+ btf_verifier_log_type(env, t, "nr_bits exceeds type_size");
+ return -EINVAL;
+ }
+
+ /*
+ * Only one of the encoding bits is allowed and it
+ * should be sufficient for the pretty print purpose (i.e. decoding).
+ * Multiple bits can be allowed later if it is found
+ * to be insufficient.
+ */
+ encoding = BTF_INT_ENCODING(int_data);
+ if (encoding &&
+ encoding != BTF_INT_SIGNED &&
+ encoding != BTF_INT_CHAR &&
+ encoding != BTF_INT_BOOL) {
+ btf_verifier_log_type(env, t, "Unsupported encoding");
+ return -ENOTSUPP;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_int_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ int int_data = btf_type_int(t);
+
+ btf_verifier_log(env,
+ "size=%u bits_offset=%u nr_bits=%u encoding=%s",
+ t->size, BTF_INT_OFFSET(int_data),
+ BTF_INT_BITS(int_data),
+ btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
+}
+
+static void btf_int128_print(struct btf_show *show, void *data)
+{
+ /* data points to a __int128 number.
+ * Suppose
+ * int128_num = *(__int128 *)data;
+ * The below formulas shows what upper_num and lower_num represents:
+ * upper_num = int128_num >> 64;
+ * lower_num = int128_num & 0xffffffffFFFFFFFFULL;
+ */
+ u64 upper_num, lower_num;
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ upper_num = *(u64 *)data;
+ lower_num = *(u64 *)(data + 8);
+#else
+ upper_num = *(u64 *)(data + 8);
+ lower_num = *(u64 *)data;
+#endif
+ if (upper_num == 0)
+ btf_show_type_value(show, "0x%llx", lower_num);
+ else
+ btf_show_type_values(show, "0x%llx%016llx", upper_num,
+ lower_num);
+}
+
+static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
+ u16 right_shift_bits)
+{
+ u64 upper_num, lower_num;
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ upper_num = print_num[0];
+ lower_num = print_num[1];
+#else
+ upper_num = print_num[1];
+ lower_num = print_num[0];
+#endif
+
+ /* shake out un-needed bits by shift/or operations */
+ if (left_shift_bits >= 64) {
+ upper_num = lower_num << (left_shift_bits - 64);
+ lower_num = 0;
+ } else {
+ upper_num = (upper_num << left_shift_bits) |
+ (lower_num >> (64 - left_shift_bits));
+ lower_num = lower_num << left_shift_bits;
+ }
+
+ if (right_shift_bits >= 64) {
+ lower_num = upper_num >> (right_shift_bits - 64);
+ upper_num = 0;
+ } else {
+ lower_num = (lower_num >> right_shift_bits) |
+ (upper_num << (64 - right_shift_bits));
+ upper_num = upper_num >> right_shift_bits;
+ }
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ print_num[0] = upper_num;
+ print_num[1] = lower_num;
+#else
+ print_num[0] = lower_num;
+ print_num[1] = upper_num;
+#endif
+}
+
+static void btf_bitfield_show(void *data, u8 bits_offset,
+ u8 nr_bits, struct btf_show *show)
+{
+ u16 left_shift_bits, right_shift_bits;
+ u8 nr_copy_bytes;
+ u8 nr_copy_bits;
+ u64 print_num[2] = {};
+
+ nr_copy_bits = nr_bits + bits_offset;
+ nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
+
+ memcpy(print_num, data, nr_copy_bytes);
+
+#ifdef __BIG_ENDIAN_BITFIELD
+ left_shift_bits = bits_offset;
+#else
+ left_shift_bits = BITS_PER_U128 - nr_copy_bits;
+#endif
+ right_shift_bits = BITS_PER_U128 - nr_bits;
+
+ btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
+ btf_int128_print(show, print_num);
+}
+
+
+static void btf_int_bits_show(const struct btf *btf,
+ const struct btf_type *t,
+ void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ u32 int_data = btf_type_int(t);
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ u8 total_bits_offset;
+
+ /*
+ * bits_offset is at most 7.
+ * BTF_INT_OFFSET() cannot exceed 128 bits.
+ */
+ total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
+ data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
+ bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
+ btf_bitfield_show(data, bits_offset, nr_bits, show);
+}
+
+static void btf_int_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ u32 int_data = btf_type_int(t);
+ u8 encoding = BTF_INT_ENCODING(int_data);
+ bool sign = encoding & BTF_INT_SIGNED;
+ u8 nr_bits = BTF_INT_BITS(int_data);
+ void *safe_data;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ if (bits_offset || BTF_INT_OFFSET(int_data) ||
+ BITS_PER_BYTE_MASKED(nr_bits)) {
+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
+ goto out;
+ }
+
+ switch (nr_bits) {
+ case 128:
+ btf_int128_print(show, safe_data);
+ break;
+ case 64:
+ if (sign)
+ btf_show_type_value(show, "%lld", *(s64 *)safe_data);
+ else
+ btf_show_type_value(show, "%llu", *(u64 *)safe_data);
+ break;
+ case 32:
+ if (sign)
+ btf_show_type_value(show, "%d", *(s32 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u32 *)safe_data);
+ break;
+ case 16:
+ if (sign)
+ btf_show_type_value(show, "%d", *(s16 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u16 *)safe_data);
+ break;
+ case 8:
+ if (show->state.array_encoding == BTF_INT_CHAR) {
+ /* check for null terminator */
+ if (show->state.array_terminated)
+ break;
+ if (*(char *)data == '\0') {
+ show->state.array_terminated = 1;
+ break;
+ }
+ if (isprint(*(char *)data)) {
+ btf_show_type_value(show, "'%c'",
+ *(char *)safe_data);
+ break;
+ }
+ }
+ if (sign)
+ btf_show_type_value(show, "%d", *(s8 *)safe_data);
+ else
+ btf_show_type_value(show, "%u", *(u8 *)safe_data);
+ break;
+ default:
+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
+ break;
+ }
+out:
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations int_ops = {
+ .check_meta = btf_int_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_int_check_member,
+ .check_kflag_member = btf_int_check_kflag_member,
+ .log_details = btf_int_log,
+ .show = btf_int_show,
+};
+
+static int btf_modifier_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id = member->type;
+ struct btf_member resolved_member;
+ struct btf *btf = env->btf;
+
+ resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
+ if (!resolved_type) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ resolved_member = *member;
+ resolved_member.type = resolved_type_id;
+
+ return btf_type_ops(resolved_type)->check_member(env, struct_type,
+ &resolved_member,
+ resolved_type);
+}
+
+static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id = member->type;
+ struct btf_member resolved_member;
+ struct btf *btf = env->btf;
+
+ resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
+ if (!resolved_type) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ resolved_member = *member;
+ resolved_member.type = resolved_type_id;
+
+ return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
+ &resolved_member,
+ resolved_type);
+}
+
+static int btf_ptr_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_size, struct_bits_off, bytes_offset;
+
+ struct_size = struct_type->size;
+ struct_bits_off = member->offset;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ if (struct_size - bytes_offset < sizeof(void *)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_ref_type_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const char *value;
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t) && !btf_type_is_type_tag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!BTF_TYPE_ID_VALID(t->type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ /* typedef/type_tag type must have a valid name, and other ref types,
+ * volatile, const, restrict, should have a null name.
+ */
+ if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ } else if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPE_TAG) {
+ value = btf_name_by_offset(env->btf, t->name_off);
+ if (!value || !value[0]) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ } else {
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_modifier_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *t = v->t;
+ const struct btf_type *next_type;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ /* Figure out the resolved next_type_id with size.
+ * They will be stored in the current modifier's
+ * resolved_ids and resolved_sizes such that it can
+ * save us a few type-following when we use it later (e.g. in
+ * pretty print).
+ */
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ if (env_type_is_resolved(env, next_type_id))
+ next_type = btf_type_id_resolve(btf, &next_type_id);
+
+ /* "typedef void new_void", "const void"...etc */
+ if (!btf_type_is_void(next_type) &&
+ !btf_type_is_fwd(next_type) &&
+ !btf_type_is_func_proto(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static int btf_var_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ if (btf_type_is_modifier(next_type)) {
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id;
+
+ resolved_type_id = next_type_id;
+ resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
+
+ if (btf_type_is_ptr(resolved_type) &&
+ !env_type_is_resolve_sink(env, resolved_type) &&
+ !env_type_is_resolved(env, resolved_type_id))
+ return env_stack_push(env, resolved_type,
+ resolved_type_id);
+ }
+
+ /* We must resolve to something concrete at this point, no
+ * forward types or similar that would resolve to size of
+ * zero is allowed.
+ */
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static int btf_ptr_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || btf_type_is_resolve_source_only(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ /* If the modifier was RESOLVED during RESOLVE_STRUCT_OR_ARRAY,
+ * the modifier may have stopped resolving when it was resolved
+ * to a ptr (last-resolved-ptr).
+ *
+ * We now need to continue from the last-resolved-ptr to
+ * ensure the last-resolved-ptr will not referring back to
+ * the current ptr (t).
+ */
+ if (btf_type_is_modifier(next_type)) {
+ const struct btf_type *resolved_type;
+ u32 resolved_type_id;
+
+ resolved_type_id = next_type_id;
+ resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
+
+ if (btf_type_is_ptr(resolved_type) &&
+ !env_type_is_resolve_sink(env, resolved_type) &&
+ !env_type_is_resolved(env, resolved_type_id))
+ return env_stack_push(env, resolved_type,
+ resolved_type_id);
+ }
+
+ if (!btf_type_id_size(btf, &next_type_id, NULL)) {
+ if (env_type_is_resolved(env, next_type_id))
+ next_type = btf_type_id_resolve(btf, &next_type_id);
+
+ if (!btf_type_is_void(next_type) &&
+ !btf_type_is_fwd(next_type) &&
+ !btf_type_is_func_proto(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static void btf_modifier_show(const struct btf *btf,
+ const struct btf_type *t,
+ u32 type_id, void *data,
+ u8 bits_offset, struct btf_show *show)
+{
+ if (btf->resolved_ids)
+ t = btf_type_id_resolve(btf, &type_id);
+ else
+ t = btf_type_skip_modifiers(btf, type_id, NULL);
+
+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
+}
+
+static void btf_var_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ t = btf_type_id_resolve(btf, &type_id);
+
+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
+}
+
+static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ void *safe_data;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
+ if (show->flags & BTF_SHOW_PTR_RAW)
+ btf_show_type_value(show, "0x%px", *(void **)safe_data);
+ else
+ btf_show_type_value(show, "0x%p", *(void **)safe_data);
+ btf_show_end_type(show);
+}
+
+static void btf_ref_type_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "type_id=%u", t->type);
+}
+
+static const struct btf_kind_operations modifier_ops = {
+ .check_meta = btf_ref_type_check_meta,
+ .resolve = btf_modifier_resolve,
+ .check_member = btf_modifier_check_member,
+ .check_kflag_member = btf_modifier_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_modifier_show,
+};
+
+static const struct btf_kind_operations ptr_ops = {
+ .check_meta = btf_ref_type_check_meta,
+ .resolve = btf_ptr_resolve,
+ .check_member = btf_ptr_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_ptr_show,
+};
+
+static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (t->type) {
+ btf_verifier_log_type(env, t, "type != 0");
+ return -EINVAL;
+ }
+
+ /* fwd type must have a valid name */
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static void btf_fwd_type_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "%s", btf_type_kflag(t) ? "union" : "struct");
+}
+
+static const struct btf_kind_operations fwd_ops = {
+ .check_meta = btf_fwd_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_fwd_type_log,
+ .show = btf_df_show,
+};
+
+static int btf_array_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+ u32 array_type_id, array_size;
+ struct btf *btf = env->btf;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ array_type_id = member->type;
+ btf_type_id_size(btf, &array_type_id, &array_size);
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < array_size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_array_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_array *array = btf_type_array(t);
+ u32 meta_needed = sizeof(*array);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ /* array type should not have a name */
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (t->size) {
+ btf_verifier_log_type(env, t, "size != 0");
+ return -EINVAL;
+ }
+
+ /* Array elem type and index type cannot be in type void,
+ * so !array->type and !array->index_type are not allowed.
+ */
+ if (!array->type || !BTF_TYPE_ID_VALID(array->type)) {
+ btf_verifier_log_type(env, t, "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (!array->index_type || !BTF_TYPE_ID_VALID(array->index_type)) {
+ btf_verifier_log_type(env, t, "Invalid index");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static int btf_array_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_array *array = btf_type_array(v->t);
+ const struct btf_type *elem_type, *index_type;
+ u32 elem_type_id, index_type_id;
+ struct btf *btf = env->btf;
+ u32 elem_size;
+
+ /* Check array->index_type */
+ index_type_id = array->index_type;
+ index_type = btf_type_by_id(btf, index_type_id);
+ if (btf_type_nosize_or_null(index_type) ||
+ btf_type_is_resolve_source_only(index_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid index");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, index_type) &&
+ !env_type_is_resolved(env, index_type_id))
+ return env_stack_push(env, index_type, index_type_id);
+
+ index_type = btf_type_id_size(btf, &index_type_id, NULL);
+ if (!index_type || !btf_type_is_int(index_type) ||
+ !btf_type_int_is_regular(index_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid index");
+ return -EINVAL;
+ }
+
+ /* Check array->type */
+ elem_type_id = array->type;
+ elem_type = btf_type_by_id(btf, elem_type_id);
+ if (btf_type_nosize_or_null(elem_type) ||
+ btf_type_is_resolve_source_only(elem_type)) {
+ btf_verifier_log_type(env, v->t,
+ "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, elem_type) &&
+ !env_type_is_resolved(env, elem_type_id))
+ return env_stack_push(env, elem_type, elem_type_id);
+
+ elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
+ if (!elem_type) {
+ btf_verifier_log_type(env, v->t, "Invalid elem");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_int(elem_type) && !btf_type_int_is_regular(elem_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid array of int");
+ return -EINVAL;
+ }
+
+ if (array->nelems && elem_size > U32_MAX / array->nelems) {
+ btf_verifier_log_type(env, v->t,
+ "Array size overflows U32_MAX");
+ return -EINVAL;
+ }
+
+ env_stack_pop_resolved(env, elem_type_id, elem_size * array->nelems);
+
+ return 0;
+}
+
+static void btf_array_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_array *array = btf_type_array(t);
+
+ btf_verifier_log(env, "type_id=%u index_type_id=%u nr_elems=%u",
+ array->type, array->index_type, array->nelems);
+}
+
+static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_array *array = btf_type_array(t);
+ const struct btf_kind_operations *elem_ops;
+ const struct btf_type *elem_type;
+ u32 i, elem_size = 0, elem_type_id;
+ u16 encoding = 0;
+
+ elem_type_id = array->type;
+ elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
+ if (elem_type && btf_type_has_size(elem_type))
+ elem_size = elem_type->size;
+
+ if (elem_type && btf_type_is_int(elem_type)) {
+ u32 int_type = btf_type_int(elem_type);
+
+ encoding = BTF_INT_ENCODING(int_type);
+
+ /*
+ * BTF_INT_CHAR encoding never seems to be set for
+ * char arrays, so if size is 1 and element is
+ * printable as a char, we'll do that.
+ */
+ if (elem_size == 1)
+ encoding = BTF_INT_CHAR;
+ }
+
+ if (!btf_show_start_array_type(show, t, type_id, encoding, data))
+ return;
+
+ if (!elem_type)
+ goto out;
+ elem_ops = btf_type_ops(elem_type);
+
+ for (i = 0; i < array->nelems; i++) {
+
+ btf_show_start_array_member(show);
+
+ elem_ops->show(btf, elem_type, elem_type_id, data,
+ bits_offset, show);
+ data += elem_size;
+
+ btf_show_end_array_member(show);
+
+ if (show->state.array_terminated)
+ break;
+ }
+out:
+ btf_show_end_array_type(show);
+}
+
+static void btf_array_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *m = show->state.member;
+
+ /*
+ * First check if any members would be shown (are non-zero).
+ * See comments above "struct btf_show" definition for more
+ * details on how this works at a high-level.
+ */
+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
+ if (!show->state.depth_check) {
+ show->state.depth_check = show->state.depth + 1;
+ show->state.depth_to_show = 0;
+ }
+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
+ show->state.member = m;
+
+ if (show->state.depth_check != show->state.depth + 1)
+ return;
+ show->state.depth_check = 0;
+
+ if (show->state.depth_to_show <= show->state.depth)
+ return;
+ /*
+ * Reaching here indicates we have recursed and found
+ * non-zero array member(s).
+ */
+ }
+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
+}
+
+static const struct btf_kind_operations array_ops = {
+ .check_meta = btf_array_check_meta,
+ .resolve = btf_array_resolve,
+ .check_member = btf_array_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_array_log,
+ .show = btf_array_show,
+};
+
+static int btf_struct_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < member_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_struct_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ bool is_union = BTF_INFO_KIND(t->info) == BTF_KIND_UNION;
+ const struct btf_member *member;
+ u32 meta_needed, last_offset;
+ struct btf *btf = env->btf;
+ u32 struct_size = t->size;
+ u32 offset;
+ u16 i;
+
+ meta_needed = btf_type_vlen(t) * sizeof(*member);
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ /* struct type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ last_offset = 0;
+ for_each_member(i, t, member) {
+ if (!btf_name_offset_valid(btf, member->name_off)) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member name_offset:%u",
+ member->name_off);
+ return -EINVAL;
+ }
+
+ /* struct member either no name or a valid one */
+ if (member->name_off &&
+ !btf_name_valid_identifier(btf, member->name_off)) {
+ btf_verifier_log_member(env, t, member, "Invalid name");
+ return -EINVAL;
+ }
+ /* A member cannot be in type void */
+ if (!member->type || !BTF_TYPE_ID_VALID(member->type)) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid type_id");
+ return -EINVAL;
+ }
+
+ offset = __btf_member_bit_offset(t, member);
+ if (is_union && offset) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member bits_offset");
+ return -EINVAL;
+ }
+
+ /*
+ * ">" instead of ">=" because the last member could be
+ * "char a[0];"
+ */
+ if (last_offset > offset) {
+ btf_verifier_log_member(env, t, member,
+ "Invalid member bits_offset");
+ return -EINVAL;
+ }
+
+ if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
+ btf_verifier_log_member(env, t, member,
+ "Member bits_offset exceeds its struct size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_member(env, t, member, NULL);
+ last_offset = offset;
+ }
+
+ return meta_needed;
+}
+
+static int btf_struct_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_member *member;
+ int err;
+ u16 i;
+
+ /* Before continue resolving the next_member,
+ * ensure the last member is indeed resolved to a
+ * type with size info.
+ */
+ if (v->next_member) {
+ const struct btf_type *last_member_type;
+ const struct btf_member *last_member;
+ u32 last_member_type_id;
+
+ last_member = btf_type_member(v->t) + v->next_member - 1;
+ last_member_type_id = last_member->type;
+ if (WARN_ON_ONCE(!env_type_is_resolved(env,
+ last_member_type_id)))
+ return -EINVAL;
+
+ last_member_type = btf_type_by_id(env->btf,
+ last_member_type_id);
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
+ last_member,
+ last_member_type);
+ else
+ err = btf_type_ops(last_member_type)->check_member(env, v->t,
+ last_member,
+ last_member_type);
+ if (err)
+ return err;
+ }
+
+ for_each_member_from(i, v->next_member, v->t, member) {
+ u32 member_type_id = member->type;
+ const struct btf_type *member_type = btf_type_by_id(env->btf,
+ member_type_id);
+
+ if (btf_type_nosize_or_null(member_type) ||
+ btf_type_is_resolve_source_only(member_type)) {
+ btf_verifier_log_member(env, v->t, member,
+ "Invalid member");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, member_type) &&
+ !env_type_is_resolved(env, member_type_id)) {
+ env_stack_set_next_member(env, i + 1);
+ return env_stack_push(env, member_type, member_type_id);
+ }
+
+ if (btf_type_kflag(v->t))
+ err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
+ member,
+ member_type);
+ else
+ err = btf_type_ops(member_type)->check_member(env, v->t,
+ member,
+ member_type);
+ if (err)
+ return err;
+ }
+
+ env_stack_pop_resolved(env, 0, 0);
+
+ return 0;
+}
+
+static void btf_struct_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+
+static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *member;
+ void *safe_data;
+ u32 i;
+
+ safe_data = btf_show_start_struct_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ for_each_member(i, t, member) {
+ const struct btf_type *member_type = btf_type_by_id(btf,
+ member->type);
+ const struct btf_kind_operations *ops;
+ u32 member_offset, bitfield_size;
+ u32 bytes_offset;
+ u8 bits8_offset;
+
+ btf_show_start_member(show, member);
+
+ member_offset = __btf_member_bit_offset(t, member);
+ bitfield_size = __btf_member_bitfield_size(t, member);
+ bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
+ bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
+ if (bitfield_size) {
+ safe_data = btf_show_start_type(show, member_type,
+ member->type,
+ data + bytes_offset);
+ if (safe_data)
+ btf_bitfield_show(safe_data,
+ bits8_offset,
+ bitfield_size, show);
+ btf_show_end_type(show);
+ } else {
+ ops = btf_type_ops(member_type);
+ ops->show(btf, member_type, member->type,
+ data + bytes_offset, bits8_offset, show);
+ }
+
+ btf_show_end_member(show);
+ }
+
+ btf_show_end_struct_type(show);
+}
+
+static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_member *m = show->state.member;
+
+ /*
+ * First check if any members would be shown (are non-zero).
+ * See comments above "struct btf_show" definition for more
+ * details on how this works at a high-level.
+ */
+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
+ if (!show->state.depth_check) {
+ show->state.depth_check = show->state.depth + 1;
+ show->state.depth_to_show = 0;
+ }
+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
+ /* Restore saved member data here */
+ show->state.member = m;
+ if (show->state.depth_check != show->state.depth + 1)
+ return;
+ show->state.depth_check = 0;
+
+ if (show->state.depth_to_show <= show->state.depth)
+ return;
+ /*
+ * Reaching here indicates we have recursed and found
+ * non-zero child values.
+ */
+ }
+
+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
+}
+
+static const struct btf_kind_operations struct_ops = {
+ .check_meta = btf_struct_check_meta,
+ .resolve = btf_struct_resolve,
+ .check_member = btf_struct_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_struct_log,
+ .show = btf_struct_show,
+};
+
+static int btf_enum_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off = member->offset;
+ u32 struct_size, bytes_offset;
+
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
+ if (struct_size - bytes_offset < member_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u32 struct_bits_off, nr_bits, bytes_end, struct_size;
+ u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
+
+ struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
+ nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
+ if (!nr_bits) {
+ if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not byte aligned");
+ return -EINVAL;
+ }
+
+ nr_bits = int_bitsize;
+ } else if (nr_bits > int_bitsize) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Invalid member bitfield_size");
+ return -EINVAL;
+ }
+
+ struct_size = struct_type->size;
+ bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
+ if (struct_size < bytes_end) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static s32 btf_enum_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_enum *enums = btf_type_enum(t);
+ struct btf *btf = env->btf;
+ const char *fmt_str;
+ u16 i, nr_enums;
+ u32 meta_needed;
+
+ nr_enums = btf_type_vlen(t);
+ meta_needed = nr_enums * sizeof(*enums);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->size > 8 || !is_power_of_2(t->size)) {
+ btf_verifier_log_type(env, t, "Unexpected size");
+ return -EINVAL;
+ }
+
+ /* enum type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for (i = 0; i < nr_enums; i++) {
+ if (!btf_name_offset_valid(btf, enums[i].name_off)) {
+ btf_verifier_log(env, "\tInvalid name_offset:%u",
+ enums[i].name_off);
+ return -EINVAL;
+ }
+
+ /* enum member must have a valid name */
+ if (!enums[i].name_off ||
+ !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (env->log.level == BPF_LOG_KERNEL)
+ continue;
+ fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
+ btf_verifier_log(env, fmt_str,
+ __btf_name_by_offset(btf, enums[i].name_off),
+ enums[i].val);
+ }
+
+ return meta_needed;
+}
+
+static void btf_enum_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_enum *enums = btf_type_enum(t);
+ u32 i, nr_enums = btf_type_vlen(t);
+ void *safe_data;
+ int v;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ v = *(int *)safe_data;
+
+ for (i = 0; i < nr_enums; i++) {
+ if (v != enums[i].val)
+ continue;
+
+ btf_show_type_value(show, "%s",
+ __btf_name_by_offset(btf,
+ enums[i].name_off));
+
+ btf_show_end_type(show);
+ return;
+ }
+
+ if (btf_type_kflag(t))
+ btf_show_type_value(show, "%d", v);
+ else
+ btf_show_type_value(show, "%u", v);
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations enum_ops = {
+ .check_meta = btf_enum_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_enum_check_member,
+ .check_kflag_member = btf_enum_check_kflag_member,
+ .log_details = btf_enum_log,
+ .show = btf_enum_show,
+};
+
+static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_enum64 *enums = btf_type_enum64(t);
+ struct btf *btf = env->btf;
+ const char *fmt_str;
+ u16 i, nr_enums;
+ u32 meta_needed;
+
+ nr_enums = btf_type_vlen(t);
+ meta_needed = nr_enums * sizeof(*enums);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->size > 8 || !is_power_of_2(t->size)) {
+ btf_verifier_log_type(env, t, "Unexpected size");
+ return -EINVAL;
+ }
+
+ /* enum type either no name or a valid one */
+ if (t->name_off &&
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for (i = 0; i < nr_enums; i++) {
+ if (!btf_name_offset_valid(btf, enums[i].name_off)) {
+ btf_verifier_log(env, "\tInvalid name_offset:%u",
+ enums[i].name_off);
+ return -EINVAL;
+ }
+
+ /* enum member must have a valid name */
+ if (!enums[i].name_off ||
+ !btf_name_valid_identifier(btf, enums[i].name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (env->log.level == BPF_LOG_KERNEL)
+ continue;
+
+ fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
+ btf_verifier_log(env, fmt_str,
+ __btf_name_by_offset(btf, enums[i].name_off),
+ btf_enum64_value(enums + i));
+ }
+
+ return meta_needed;
+}
+
+static void btf_enum64_show(const struct btf *btf, const struct btf_type *t,
+ u32 type_id, void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_enum64 *enums = btf_type_enum64(t);
+ u32 i, nr_enums = btf_type_vlen(t);
+ void *safe_data;
+ s64 v;
+
+ safe_data = btf_show_start_type(show, t, type_id, data);
+ if (!safe_data)
+ return;
+
+ v = *(u64 *)safe_data;
+
+ for (i = 0; i < nr_enums; i++) {
+ if (v != btf_enum64_value(enums + i))
+ continue;
+
+ btf_show_type_value(show, "%s",
+ __btf_name_by_offset(btf,
+ enums[i].name_off));
+
+ btf_show_end_type(show);
+ return;
+ }
+
+ if (btf_type_kflag(t))
+ btf_show_type_value(show, "%lld", v);
+ else
+ btf_show_type_value(show, "%llu", v);
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations enum64_ops = {
+ .check_meta = btf_enum64_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_enum_check_member,
+ .check_kflag_member = btf_enum_check_kflag_member,
+ .log_details = btf_enum_log,
+ .show = btf_enum64_show,
+};
+
+static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 meta_needed = btf_type_vlen(t) * sizeof(struct btf_param);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (t->name_off) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_func_proto_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_param *args = (const struct btf_param *)(t + 1);
+ u16 nr_args = btf_type_vlen(t), i;
+
+ btf_verifier_log(env, "return=%u args=(", t->type);
+ if (!nr_args) {
+ btf_verifier_log(env, "void");
+ goto done;
+ }
+
+ if (nr_args == 1 && !args[0].type) {
+ /* Only one vararg */
+ btf_verifier_log(env, "vararg");
+ goto done;
+ }
+
+ btf_verifier_log(env, "%u %s", args[0].type,
+ __btf_name_by_offset(env->btf,
+ args[0].name_off));
+ for (i = 1; i < nr_args - 1; i++)
+ btf_verifier_log(env, ", %u %s", args[i].type,
+ __btf_name_by_offset(env->btf,
+ args[i].name_off));
+
+ if (nr_args > 1) {
+ const struct btf_param *last_arg = &args[nr_args - 1];
+
+ if (last_arg->type)
+ btf_verifier_log(env, ", %u %s", last_arg->type,
+ __btf_name_by_offset(env->btf,
+ last_arg->name_off));
+ else
+ btf_verifier_log(env, ", vararg");
+ }
+
+done:
+ btf_verifier_log(env, ")");
+}
+
+static const struct btf_kind_operations func_proto_ops = {
+ .check_meta = btf_func_proto_check_meta,
+ .resolve = btf_df_resolve,
+ /*
+ * BTF_KIND_FUNC_PROTO cannot be directly referred by
+ * a struct's member.
+ *
+ * It should be a function pointer instead.
+ * (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
+ *
+ * Hence, there is no btf_func_check_member().
+ */
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_func_proto_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_func_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t) > BTF_FUNC_GLOBAL) {
+ btf_verifier_log_type(env, t, "Invalid func linkage");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_func_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ int err;
+
+ err = btf_func_check(env, t);
+ if (err)
+ return err;
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+ return 0;
+}
+
+static const struct btf_kind_operations func_ops = {
+ .check_meta = btf_func_check_meta,
+ .resolve = btf_func_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_ref_type_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_var_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_var *var;
+ u32 meta_needed = sizeof(*var);
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!t->name_off ||
+ !btf_name_valid_identifier(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ /* A var cannot be in type void */
+ if (!t->type || !BTF_TYPE_ID_VALID(t->type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ var = btf_type_var(t);
+ if (var->linkage != BTF_VAR_STATIC &&
+ var->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
+ btf_verifier_log_type(env, t, "Linkage not supported");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t)
+{
+ const struct btf_var *var = btf_type_var(t);
+
+ btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage);
+}
+
+static const struct btf_kind_operations var_ops = {
+ .check_meta = btf_var_check_meta,
+ .resolve = btf_var_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_var_log,
+ .show = btf_var_show,
+};
+
+static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_var_secinfo *vsi;
+ u64 last_vsi_end_off = 0, sum = 0;
+ u32 i, meta_needed;
+
+ meta_needed = btf_type_vlen(t) * sizeof(*vsi);
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ if (!t->size) {
+ btf_verifier_log_type(env, t, "size == 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (!t->name_off ||
+ !btf_name_valid_section(env->btf, t->name_off)) {
+ btf_verifier_log_type(env, t, "Invalid name");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ for_each_vsi(i, t, vsi) {
+ /* A var cannot be in type void */
+ if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid offset");
+ return -EINVAL;
+ }
+
+ if (!vsi->size || vsi->size > t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid size");
+ return -EINVAL;
+ }
+
+ last_vsi_end_off = vsi->offset + vsi->size;
+ if (last_vsi_end_off > t->size) {
+ btf_verifier_log_vsi(env, t, vsi,
+ "Invalid offset+size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_vsi(env, t, vsi, NULL);
+ sum += vsi->size;
+ }
+
+ if (t->size < sum) {
+ btf_verifier_log_type(env, t, "Invalid btf_info size");
+ return -EINVAL;
+ }
+
+ return meta_needed;
+}
+
+static int btf_datasec_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_var_secinfo *vsi;
+ struct btf *btf = env->btf;
+ u16 i;
+
+ env->resolve_mode = RESOLVE_TBD;
+ for_each_vsi_from(i, v->next_member, v->t, vsi) {
+ u32 var_type_id = vsi->type, type_id, type_size = 0;
+ const struct btf_type *var_type = btf_type_by_id(env->btf,
+ var_type_id);
+ if (!var_type || !btf_type_is_var(var_type)) {
+ btf_verifier_log_vsi(env, v->t, vsi,
+ "Not a VAR kind member");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, var_type) &&
+ !env_type_is_resolved(env, var_type_id)) {
+ env_stack_set_next_member(env, i + 1);
+ return env_stack_push(env, var_type, var_type_id);
+ }
+
+ type_id = var_type->type;
+ if (!btf_type_id_size(btf, &type_id, &type_size)) {
+ btf_verifier_log_vsi(env, v->t, vsi, "Invalid type");
+ return -EINVAL;
+ }
+
+ if (vsi->size < type_size) {
+ btf_verifier_log_vsi(env, v->t, vsi, "Invalid size");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, 0, 0);
+ return 0;
+}
+
+static void btf_datasec_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
+}
+
+static void btf_datasec_show(const struct btf *btf,
+ const struct btf_type *t, u32 type_id,
+ void *data, u8 bits_offset,
+ struct btf_show *show)
+{
+ const struct btf_var_secinfo *vsi;
+ const struct btf_type *var;
+ u32 i;
+
+ if (!btf_show_start_type(show, t, type_id, data))
+ return;
+
+ btf_show_type_value(show, "section (\"%s\") = {",
+ __btf_name_by_offset(btf, t->name_off));
+ for_each_vsi(i, t, vsi) {
+ var = btf_type_by_id(btf, vsi->type);
+ if (i)
+ btf_show(show, ",");
+ btf_type_ops(var)->show(btf, var, vsi->type,
+ data + vsi->offset, bits_offset, show);
+ }
+ btf_show_end_type(show);
+}
+
+static const struct btf_kind_operations datasec_ops = {
+ .check_meta = btf_datasec_check_meta,
+ .resolve = btf_datasec_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_datasec_log,
+ .show = btf_datasec_show,
+};
+
+static s32 btf_float_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ if (btf_type_kflag(t)) {
+ btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
+ return -EINVAL;
+ }
+
+ if (t->size != 2 && t->size != 4 && t->size != 8 && t->size != 12 &&
+ t->size != 16) {
+ btf_verifier_log_type(env, t, "Invalid type_size");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return 0;
+}
+
+static int btf_float_check_member(struct btf_verifier_env *env,
+ const struct btf_type *struct_type,
+ const struct btf_member *member,
+ const struct btf_type *member_type)
+{
+ u64 start_offset_bytes;
+ u64 end_offset_bytes;
+ u64 misalign_bits;
+ u64 align_bytes;
+ u64 align_bits;
+
+ /* Different architectures have different alignment requirements, so
+ * here we check only for the reasonable minimum. This way we ensure
+ * that types after CO-RE can pass the kernel BTF verifier.
+ */
+ align_bytes = min_t(u64, sizeof(void *), member_type->size);
+ align_bits = align_bytes * BITS_PER_BYTE;
+ div64_u64_rem(member->offset, align_bits, &misalign_bits);
+ if (misalign_bits) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member is not properly aligned");
+ return -EINVAL;
+ }
+
+ start_offset_bytes = member->offset / BITS_PER_BYTE;
+ end_offset_bytes = start_offset_bytes + member_type->size;
+ if (end_offset_bytes > struct_type->size) {
+ btf_verifier_log_member(env, struct_type, member,
+ "Member exceeds struct_size");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void btf_float_log(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ btf_verifier_log(env, "size=%u", t->size);
+}
+
+static const struct btf_kind_operations float_ops = {
+ .check_meta = btf_float_check_meta,
+ .resolve = btf_df_resolve,
+ .check_member = btf_float_check_member,
+ .check_kflag_member = btf_generic_check_kflag_member,
+ .log_details = btf_float_log,
+ .show = btf_df_show,
+};
+
+static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ const struct btf_decl_tag *tag;
+ u32 meta_needed = sizeof(*tag);
+ s32 component_idx;
+ const char *value;
+
+ if (meta_left < meta_needed) {
+ btf_verifier_log_basic(env, t,
+ "meta_left:%u meta_needed:%u",
+ meta_left, meta_needed);
+ return -EINVAL;
+ }
+
+ value = btf_name_by_offset(env->btf, t->name_off);
+ if (!value || !value[0]) {
+ btf_verifier_log_type(env, t, "Invalid value");
+ return -EINVAL;
+ }
+
+ if (btf_type_vlen(t)) {
+ btf_verifier_log_type(env, t, "vlen != 0");
+ return -EINVAL;
+ }
+
+ component_idx = btf_type_decl_tag(t)->component_idx;
+ if (component_idx < -1) {
+ btf_verifier_log_type(env, t, "Invalid component_idx");
+ return -EINVAL;
+ }
+
+ btf_verifier_log_type(env, t, NULL);
+
+ return meta_needed;
+}
+
+static int btf_decl_tag_resolve(struct btf_verifier_env *env,
+ const struct resolve_vertex *v)
+{
+ const struct btf_type *next_type;
+ const struct btf_type *t = v->t;
+ u32 next_type_id = t->type;
+ struct btf *btf = env->btf;
+ s32 component_idx;
+ u32 vlen;
+
+ next_type = btf_type_by_id(btf, next_type_id);
+ if (!next_type || !btf_type_is_decl_tag_target(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ if (!env_type_is_resolve_sink(env, next_type) &&
+ !env_type_is_resolved(env, next_type_id))
+ return env_stack_push(env, next_type, next_type_id);
+
+ component_idx = btf_type_decl_tag(t)->component_idx;
+ if (component_idx != -1) {
+ if (btf_type_is_var(next_type) || btf_type_is_typedef(next_type)) {
+ btf_verifier_log_type(env, v->t, "Invalid component_idx");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_struct(next_type)) {
+ vlen = btf_type_vlen(next_type);
+ } else {
+ /* next_type should be a function */
+ next_type = btf_type_by_id(btf, next_type->type);
+ vlen = btf_type_vlen(next_type);
+ }
+
+ if ((u32)component_idx >= vlen) {
+ btf_verifier_log_type(env, v->t, "Invalid component_idx");
+ return -EINVAL;
+ }
+ }
+
+ env_stack_pop_resolved(env, next_type_id, 0);
+
+ return 0;
+}
+
+static void btf_decl_tag_log(struct btf_verifier_env *env, const struct btf_type *t)
+{
+ btf_verifier_log(env, "type=%u component_idx=%d", t->type,
+ btf_type_decl_tag(t)->component_idx);
+}
+
+static const struct btf_kind_operations decl_tag_ops = {
+ .check_meta = btf_decl_tag_check_meta,
+ .resolve = btf_decl_tag_resolve,
+ .check_member = btf_df_check_member,
+ .check_kflag_member = btf_df_check_kflag_member,
+ .log_details = btf_decl_tag_log,
+ .show = btf_df_show,
+};
+
+static int btf_func_proto_check(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_type *ret_type;
+ const struct btf_param *args;
+ const struct btf *btf;
+ u16 nr_args, i;
+ int err;
+
+ btf = env->btf;
+ args = (const struct btf_param *)(t + 1);
+ nr_args = btf_type_vlen(t);
+
+ /* Check func return type which could be "void" (t->type == 0) */
+ if (t->type) {
+ u32 ret_type_id = t->type;
+
+ ret_type = btf_type_by_id(btf, ret_type_id);
+ if (!ret_type) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+
+ if (btf_type_is_resolve_source_only(ret_type)) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+
+ if (btf_type_needs_resolve(ret_type) &&
+ !env_type_is_resolved(env, ret_type_id)) {
+ err = btf_resolve(env, ret_type, ret_type_id);
+ if (err)
+ return err;
+ }
+
+ /* Ensure the return type is a type that has a size */
+ if (!btf_type_id_size(btf, &ret_type_id, NULL)) {
+ btf_verifier_log_type(env, t, "Invalid return type");
+ return -EINVAL;
+ }
+ }
+
+ if (!nr_args)
+ return 0;
+
+ /* Last func arg type_id could be 0 if it is a vararg */
+ if (!args[nr_args - 1].type) {
+ if (args[nr_args - 1].name_off) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u",
+ nr_args);
+ return -EINVAL;
+ }
+ nr_args--;
+ }
+
+ for (i = 0; i < nr_args; i++) {
+ const struct btf_type *arg_type;
+ u32 arg_type_id;
+
+ arg_type_id = args[i].type;
+ arg_type = btf_type_by_id(btf, arg_type_id);
+ if (!arg_type) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (btf_type_is_resolve_source_only(arg_type)) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (args[i].name_off &&
+ (!btf_name_offset_valid(btf, args[i].name_off) ||
+ !btf_name_valid_identifier(btf, args[i].name_off))) {
+ btf_verifier_log_type(env, t,
+ "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+
+ if (btf_type_needs_resolve(arg_type) &&
+ !env_type_is_resolved(env, arg_type_id)) {
+ err = btf_resolve(env, arg_type, arg_type_id);
+ if (err)
+ return err;
+ }
+
+ if (!btf_type_id_size(btf, &arg_type_id, NULL)) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int btf_func_check(struct btf_verifier_env *env,
+ const struct btf_type *t)
+{
+ const struct btf_type *proto_type;
+ const struct btf_param *args;
+ const struct btf *btf;
+ u16 nr_args, i;
+
+ btf = env->btf;
+ proto_type = btf_type_by_id(btf, t->type);
+
+ if (!proto_type || !btf_type_is_func_proto(proto_type)) {
+ btf_verifier_log_type(env, t, "Invalid type_id");
+ return -EINVAL;
+ }
+
+ args = (const struct btf_param *)(proto_type + 1);
+ nr_args = btf_type_vlen(proto_type);
+ for (i = 0; i < nr_args; i++) {
+ if (!args[i].name_off && args[i].type) {
+ btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = {
+ [BTF_KIND_INT] = &int_ops,
+ [BTF_KIND_PTR] = &ptr_ops,
+ [BTF_KIND_ARRAY] = &array_ops,
+ [BTF_KIND_STRUCT] = &struct_ops,
+ [BTF_KIND_UNION] = &struct_ops,
+ [BTF_KIND_ENUM] = &enum_ops,
+ [BTF_KIND_FWD] = &fwd_ops,
+ [BTF_KIND_TYPEDEF] = &modifier_ops,
+ [BTF_KIND_VOLATILE] = &modifier_ops,
+ [BTF_KIND_CONST] = &modifier_ops,
+ [BTF_KIND_RESTRICT] = &modifier_ops,
+ [BTF_KIND_FUNC] = &func_ops,
+ [BTF_KIND_FUNC_PROTO] = &func_proto_ops,
+ [BTF_KIND_VAR] = &var_ops,
+ [BTF_KIND_DATASEC] = &datasec_ops,
+ [BTF_KIND_FLOAT] = &float_ops,
+ [BTF_KIND_DECL_TAG] = &decl_tag_ops,
+ [BTF_KIND_TYPE_TAG] = &modifier_ops,
+ [BTF_KIND_ENUM64] = &enum64_ops,
+};
+
+static s32 btf_check_meta(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 meta_left)
+{
+ u32 saved_meta_left = meta_left;
+ s32 var_meta_size;
+
+ if (meta_left < sizeof(*t)) {
+ btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu",
+ env->log_type_id, meta_left, sizeof(*t));
+ return -EINVAL;
+ }
+ meta_left -= sizeof(*t);
+
+ if (t->info & ~BTF_INFO_MASK) {
+ btf_verifier_log(env, "[%u] Invalid btf_info:%x",
+ env->log_type_id, t->info);
+ return -EINVAL;
+ }
+
+ if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
+ BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
+ btf_verifier_log(env, "[%u] Invalid kind:%u",
+ env->log_type_id, BTF_INFO_KIND(t->info));
+ return -EINVAL;
+ }
+
+ if (!btf_name_offset_valid(env->btf, t->name_off)) {
+ btf_verifier_log(env, "[%u] Invalid name_offset:%u",
+ env->log_type_id, t->name_off);
+ return -EINVAL;
+ }
+
+ var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left);
+ if (var_meta_size < 0)
+ return var_meta_size;
+
+ meta_left -= var_meta_size;
+
+ return saved_meta_left - meta_left;
+}
+
+int btf_check_all_metas(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ struct btf_header *hdr;
+ void *cur, *end;
+
+ hdr = &btf->hdr;
+ cur = btf->nohdr_data + hdr->type_off;
+ end = cur + hdr->type_len;
+
+ env->log_type_id = btf->base_btf ? btf->start_id : 1;
+ while (cur < end) {
+ struct btf_type *t = cur;
+ s32 meta_size;
+
+ meta_size = btf_check_meta(env, t, end - cur);
+ if (meta_size < 0)
+ return meta_size;
+
+ btf_add_type(env, t);
+ cur += meta_size;
+ env->log_type_id++;
+ }
+
+ return 0;
+}
+
+static bool btf_resolve_valid(struct btf_verifier_env *env,
+ const struct btf_type *t,
+ u32 type_id)
+{
+ struct btf *btf = env->btf;
+
+ if (!env_type_is_resolved(env, type_id))
+ return false;
+
+ if (btf_type_is_struct(t) || btf_type_is_datasec(t))
+ return !btf_resolved_type_id(btf, type_id) &&
+ !btf_resolved_type_size(btf, type_id);
+
+ if (btf_type_is_decl_tag(t) || btf_type_is_func(t))
+ return btf_resolved_type_id(btf, type_id) &&
+ !btf_resolved_type_size(btf, type_id);
+
+ if (btf_type_is_modifier(t) || btf_type_is_ptr(t) ||
+ btf_type_is_var(t)) {
+ t = btf_type_id_resolve(btf, &type_id);
+ return t &&
+ !btf_type_is_modifier(t) &&
+ !btf_type_is_var(t) &&
+ !btf_type_is_datasec(t);
+ }
+
+ if (btf_type_is_array(t)) {
+ const struct btf_array *array = btf_type_array(t);
+ const struct btf_type *elem_type;
+ u32 elem_type_id = array->type;
+ u32 elem_size;
+
+ elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
+ return elem_type && !btf_type_is_modifier(elem_type) &&
+ (array->nelems * elem_size ==
+ btf_resolved_type_size(btf, type_id));
+ }
+
+ return false;
+}
+
+static int btf_resolve(struct btf_verifier_env *env,
+ const struct btf_type *t, u32 type_id)
+{
+ u32 save_log_type_id = env->log_type_id;
+ const struct resolve_vertex *v;
+ int err = 0;
+
+ env->resolve_mode = RESOLVE_TBD;
+ env_stack_push(env, t, type_id);
+ while (!err && (v = env_stack_peak(env))) {
+ env->log_type_id = v->type_id;
+ err = btf_type_ops(v->t)->resolve(env, v);
+ }
+
+ env->log_type_id = type_id;
+ if (err == -E2BIG) {
+ btf_verifier_log_type(env, t,
+ "Exceeded max resolving depth:%u",
+ MAX_RESOLVE_DEPTH);
+ } else if (err == -EEXIST) {
+ btf_verifier_log_type(env, t, "Loop detected");
+ }
+
+ /* Final sanity check */
+ if (!err && !btf_resolve_valid(env, t, type_id)) {
+ btf_verifier_log_type(env, t, "Invalid resolve state");
+ err = -EINVAL;
+ }
+
+ env->log_type_id = save_log_type_id;
+ return err;
+}
+
+static int btf_check_all_types(struct btf_verifier_env *env)
+{
+ struct btf *btf = env->btf;
+ const struct btf_type *t;
+ u32 type_id, i;
+ int err;
+
+ err = env_resolve_init(env);
+ if (err)
+ return err;
+
+ env->phase++;
+ for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
+ type_id = btf->start_id + i;
+ t = btf_type_by_id(btf, type_id);
+
+ env->log_type_id = type_id;
+ if (btf_type_needs_resolve(t) &&
+ !env_type_is_resolved(env, type_id)) {
+ err = btf_resolve(env, t, type_id);
+ if (err)
+ return err;
+ }
+
+ if (btf_type_is_func_proto(t)) {
+ err = btf_func_proto_check(env, t);
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+int btf_parse_type_sec(struct btf_verifier_env *env)
+{
+ const struct btf_header *hdr = &env->btf->hdr;
+ int err;
+
+ /* Type section must align to 4 bytes */
+ if (hdr->type_off & (sizeof(u32) - 1)) {
+ btf_verifier_log(env, "Unaligned type_off");
+ return -EINVAL;
+ }
+
+ if (!env->btf->base_btf && !hdr->type_len) {
+ btf_verifier_log(env, "No type found");
+ return -EINVAL;
+ }
+
+ err = btf_check_all_metas(env);
+ if (err)
+ return err;
+
+ return btf_check_all_types(env);
+}
+
+int btf_parse_str_sec(struct btf_verifier_env *env)
+{
+ const struct btf_header *hdr;
+ struct btf *btf = env->btf;
+ const char *start, *end;
+
+ hdr = &btf->hdr;
+ start = btf->nohdr_data + hdr->str_off;
+ end = start + hdr->str_len;
+
+ if (end != btf->data + btf->data_size) {
+ btf_verifier_log(env, "String section is not at the end");
+ return -EINVAL;
+ }
+
+ btf->strings = start;
+
+ if (btf->base_btf && !hdr->str_len)
+ return 0;
+ if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_NAME_OFFSET || end[-1]) {
+ btf_verifier_log(env, "Invalid string section");
+ return -EINVAL;
+ }
+ if (!btf->base_btf && start[0]) {
+ btf_verifier_log(env, "Invalid string section");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const size_t btf_sec_info_offset[] = {
+ offsetof(struct btf_header, type_off),
+ offsetof(struct btf_header, str_off),
+};
+
+static int btf_sec_info_cmp(const void *a, const void *b)
+{
+ const struct btf_sec_info *x = a;
+ const struct btf_sec_info *y = b;
+
+ return (int)(x->off - y->off) ? : (int)(x->len - y->len);
+}
+
+static int btf_check_sec_info(struct btf_verifier_env *env,
+ u32 btf_data_size)
+{
+ struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
+ u32 total, expected_total, i;
+ const struct btf_header *hdr;
+ const struct btf *btf;
+
+ btf = env->btf;
+ hdr = &btf->hdr;
+
+ /* Populate the secs from hdr */
+ for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++)
+ secs[i] = *(struct btf_sec_info *)((void *)hdr +
+ btf_sec_info_offset[i]);
+
+ sort(secs, ARRAY_SIZE(btf_sec_info_offset),
+ sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
+
+ /* Check for gaps and overlap among sections */
+ total = 0;
+ expected_total = btf_data_size - hdr->hdr_len;
+ for (i = 0; i < ARRAY_SIZE(btf_sec_info_offset); i++) {
+ if (expected_total < secs[i].off) {
+ btf_verifier_log(env, "Invalid section offset");
+ return -EINVAL;
+ }
+ if (total < secs[i].off) {
+ /* gap */
+ btf_verifier_log(env, "Unsupported section found");
+ return -EINVAL;
+ }
+ if (total > secs[i].off) {
+ btf_verifier_log(env, "Section overlap found");
+ return -EINVAL;
+ }
+ if (expected_total - total < secs[i].len) {
+ btf_verifier_log(env,
+ "Total section length too long");
+ return -EINVAL;
+ }
+ total += secs[i].len;
+ }
+
+ /* There is data other than hdr and known sections */
+ if (expected_total != total) {
+ btf_verifier_log(env, "Unsupported section found");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int btf_parse_hdr(struct btf_verifier_env *env)
+{
+ u32 hdr_len, hdr_copy, btf_data_size;
+ const struct btf_header *hdr;
+ struct btf *btf;
+
+ btf = env->btf;
+ btf_data_size = btf->data_size;
+
+ if (btf_data_size < offsetofend(struct btf_header, hdr_len)) {
+ btf_verifier_log(env, "hdr_len not found");
+ return -EINVAL;
+ }
+
+ hdr = btf->data;
+ hdr_len = hdr->hdr_len;
+ if (btf_data_size < hdr_len) {
+ btf_verifier_log(env, "btf_header not found");
+ return -EINVAL;
+ }
+
+ /* Ensure the unsupported header fields are zero */
+ if (hdr_len > sizeof(btf->hdr)) {
+ u8 *expected_zero = btf->data + sizeof(btf->hdr);
+ u8 *end = btf->data + hdr_len;
+
+ for (; expected_zero < end; expected_zero++) {
+ if (*expected_zero) {
+ btf_verifier_log(env, "Unsupported btf_header");
+ return -E2BIG;
+ }
+ }
+ }
+
+ hdr_copy = min_t(u32, hdr_len, sizeof(btf->hdr));
+ memcpy(&btf->hdr, btf->data, hdr_copy);
+
+ hdr = &btf->hdr;
+
+ btf_verifier_log_hdr(env, btf_data_size);
+
+ if (hdr->magic != BTF_MAGIC) {
+ btf_verifier_log(env, "Invalid magic");
+ return -EINVAL;
+ }
+
+ if (hdr->version != BTF_VERSION) {
+ btf_verifier_log(env, "Unsupported version");
+ return -ENOTSUPP;
+ }
+
+ if (hdr->flags) {
+ btf_verifier_log(env, "Unsupported flags");
+ return -ENOTSUPP;
+ }
+
+ if (!btf->base_btf && btf_data_size == hdr->hdr_len) {
+ btf_verifier_log(env, "No data");
+ return -EINVAL;
+ }
+
+ return btf_check_sec_info(env, btf_data_size);
+}
+
+
+int btf_check_type_tags(struct btf_verifier_env *env,
+ struct btf *btf, int start_id)
+{
+ int i, n, good_id = start_id - 1;
+ bool in_tags;
+
+ n = btf_nr_types(btf);
+ for (i = start_id; i < n; i++) {
+ const struct btf_type *t;
+ int chain_limit = 32;
+ u32 cur_id = i;
+
+ t = btf_type_by_id(btf, i);
+ if (!t)
+ return -EINVAL;
+ if (!btf_type_is_modifier(t))
+ continue;
+
+ cond_resched();
+
+ in_tags = btf_type_is_type_tag(t);
+ while (btf_type_is_modifier(t)) {
+ if (!chain_limit--) {
+ btf_verifier_log(env, "Max chain length or cycle detected");
+ return -ELOOP;
+ }
+ if (btf_type_is_type_tag(t)) {
+ if (!in_tags) {
+ btf_verifier_log(env, "Type tags don't precede modifiers");
+ return -EINVAL;
+ }
+ } else if (in_tags) {
+ in_tags = false;
+ }
+ if (cur_id <= good_id)
+ break;
+ /* Move to next type */
+ cur_id = t->type;
+ t = btf_type_by_id(btf, cur_id);
+ if (!t)
+ return -EINVAL;
+ }
+ good_id = i;
+ }
+ return 0;
+}
+
+
+struct btf *
+btf_parse_base(struct btf_verifier_env *env, const char *name,
+ void *data, unsigned int data_size)
+{
+ struct btf *btf = NULL;
+ int err;
+
+ if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
+ return ERR_PTR(-ENOENT);
+
+ btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
+ if (!btf) {
+ err = -ENOMEM;
+ goto errout;
+ }
+ env->btf = btf;
+
+ btf->data = data;
+ btf->data_size = data_size;
+ btf->kernel_btf = true;
+ btf->named_start_id = 0;
+ strscpy(btf->name, name);
+
+ err = btf_parse_hdr(env);
+ if (err)
+ goto errout;
+
+ btf->nohdr_data = btf->data + btf->hdr.hdr_len;
+
+ err = btf_parse_str_sec(env);
+ if (err)
+ goto errout;
+
+ err = btf_check_all_metas(env);
+ if (err)
+ goto errout;
+
+ err = btf_check_type_tags(env, btf, 1);
+ if (err)
+ goto errout;
+
+ btf_check_sorted(btf);
+ refcount_set(&btf->refcnt, 1);
+
+ return btf;
+
+errout:
+ if (btf) {
+ kvfree(btf->types);
+ kfree(btf);
+ }
+ return ERR_PTR(err);
+}
+
+
+static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
+ struct btf_show *show)
+{
+ const struct btf_type *t = btf_type_by_id(btf, type_id);
+
+ show->btf = btf;
+ memset(&show->state, 0, sizeof(show->state));
+ memset(&show->obj, 0, sizeof(show->obj));
+
+ btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
+}
+
+__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,
+ va_list args)
+{
+ seq_vprintf((struct seq_file *)show->target, fmt, args);
+}
+
+int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
+ void *obj, struct seq_file *m, u64 flags)
+{
+ struct btf_show sseq;
+
+ sseq.target = m;
+ sseq.showfn = btf_seq_show;
+ sseq.flags = flags;
+
+ btf_type_show(btf, type_id, obj, &sseq);
+
+ return sseq.state.status;
+}
+
+void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
+ struct seq_file *m)
+{
+ (void) btf_type_seq_show_flags(btf, type_id, obj, m,
+ BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
+ BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
+}
+
+struct btf_show_snprintf {
+ struct btf_show show;
+ int len_left; /* space left in string */
+ int len; /* length we would have written */
+};
+
+__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
+ va_list args)
+{
+ struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
+ int len;
+
+ len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
+
+ if (len < 0) {
+ ssnprintf->len_left = 0;
+ ssnprintf->len = len;
+ } else if (len >= ssnprintf->len_left) {
+ /* no space, drive on to get length we would have written */
+ ssnprintf->len_left = 0;
+ ssnprintf->len += len;
+ } else {
+ ssnprintf->len_left -= len;
+ ssnprintf->len += len;
+ show->target += len;
+ }
+}
+
+int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
+ char *buf, int len, u64 flags)
+{
+ struct btf_show_snprintf ssnprintf;
+
+ ssnprintf.show.target = buf;
+ ssnprintf.show.flags = flags;
+ ssnprintf.show.showfn = btf_snprintf_show;
+ ssnprintf.len_left = len;
+ ssnprintf.len = 0;
+
+ btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
+
+ /* If we encountered an error, return it. */
+ if (ssnprintf.show.state.status)
+ return ssnprintf.show.state.status;
+
+ /* Otherwise return length we would have written */
+ return ssnprintf.len;
+}
+
+
+bool btf_is_kernel(const struct btf *btf)
+{
+ return btf->kernel_btf;
+}
+
+bool btf_is_module(const struct btf *btf)
+{
+ return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
+}
+
+
+bool btf_param_match_suffix(const struct btf *btf,
+ const struct btf_param *arg,
+ const char *suffix)
+{
+ int suffix_len = strlen(suffix), len;
+ const char *param_name;
+
+ /* In the future, this can be ported to use BTF tagging */
+ param_name = btf_name_by_offset(btf, arg->name_off);
+ if (str_is_empty(param_name))
+ return false;
+ len = strlen(param_name);
+ if (len <= suffix_len)
+ return false;
+ param_name += len - suffix_len;
+ return !strncmp(param_name, suffix, suffix_len);
+}
diff --git a/kernel/btf/btf.h b/kernel/btf/btf.h
new file mode 100644
index 0000000000000..f66d9300adb31
--- /dev/null
+++ b/kernel/btf/btf.h
@@ -0,0 +1,135 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2018 Facebook */
+/* Internal BTF header for kernel/btf/ */
+
+#ifndef _KERNEL_BTF_BTF_H
+#define _KERNEL_BTF_BTF_H
+
+#include <linux/types.h>
+#include <linux/btf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/module.h>
+#include <linux/rculist.h>
+
+/* Forward declarations */
+struct btf_show;
+struct btf_struct_metas;
+
+/* btf_verifier_env and its dependencies (needed by both btf.c and bpf.c) */
+enum verifier_phase {
+ CHECK_META,
+ CHECK_TYPE,
+};
+
+struct resolve_vertex {
+ const struct btf_type *t;
+ u32 type_id;
+ u16 next_member;
+};
+
+enum resolve_mode {
+ RESOLVE_TBD, /* To Be Determined */
+ RESOLVE_PTR, /* Resolving for Pointer */
+ RESOLVE_STRUCT_OR_ARRAY, /* Resolving for struct/union
+ * or array
+ */
+};
+
+#define MAX_RESOLVE_DEPTH 32
+
+struct btf_verifier_env {
+ struct btf *btf;
+ u8 *visit_states;
+ struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
+ struct bpf_verifier_log log;
+ u32 log_type_id;
+ u32 top_stack;
+ enum verifier_phase phase;
+ enum resolve_mode resolve_mode;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+struct btf_kfunc_set_tab;
+struct btf_id_dtor_kfunc_tab;
+struct btf_struct_ops_tab;
+#endif
+
+struct btf {
+ void *data;
+ struct btf_type **types;
+ u32 *resolved_ids;
+ u32 *resolved_sizes;
+ const char *strings;
+ void *nohdr_data;
+ struct btf_header hdr;
+ u32 nr_types; /* includes VOID for base BTF */
+ u32 named_start_id;
+ u32 types_size;
+ u32 data_size;
+ refcount_t refcnt;
+#ifdef CONFIG_BPF_SYSCALL
+ u32 id;
+ struct rcu_head rcu;
+ struct btf_kfunc_set_tab *kfunc_set_tab;
+ struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
+ struct btf_struct_metas *struct_meta_tab;
+ struct btf_struct_ops_tab *struct_ops_tab;
+#endif
+
+ /* split BTF support */
+ struct btf *base_btf;
+ u32 start_id; /* first type ID in this BTF (0 for base BTF) */
+ u32 start_str_off; /* first string offset (0 for base BTF) */
+ char name[MODULE_NAME_LEN];
+ bool kernel_btf;
+ __u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
+};
+
+#define BITS_PER_BYTE_MASK (BITS_PER_BYTE - 1)
+#define BITS_PER_BYTE_MASKED(bits) ((bits) & BITS_PER_BYTE_MASK)
+#define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
+#define BITS_ROUNDUP_BYTES(bits) \
+ (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
+
+/* 16MB for 64k structs and each has 16 members and
+ * a few MB spaces for the string section.
+ * The hard limit is S32_MAX.
+ */
+#define BTF_MAX_SIZE (16 * 1024 * 1024)
+
+/*
+ * The suffix of a type that indicates it cannot alias another type when
+ * comparing BTF IDs for kfunc invocations.
+ */
+#define NOCAST_ALIAS_SUFFIX "___init"
+
+/* Core BTF functions needed by bpf.c */
+const char *__btf_name_by_offset(const struct btf *btf, u32 offset);
+const struct btf_type *
+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
+ u32 *type_size, const struct btf_type **elem_type,
+ u32 *elem_id, u32 *total_nelems, u32 *type_id);
+int btf_check_all_metas(struct btf_verifier_env *env);
+void btf_check_sorted(struct btf *btf);
+int btf_check_type_tags(struct btf_verifier_env *env,
+ struct btf *btf, int start_id);
+void btf_free(struct btf *btf);
+struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
+ void *data, unsigned int data_size);
+int btf_parse_hdr(struct btf_verifier_env *env);
+int btf_parse_str_sec(struct btf_verifier_env *env);
+int btf_parse_type_sec(struct btf_verifier_env *env);
+const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t);
+bool btf_type_has_size(const struct btf_type *t);
+bool btf_type_is_datasec(const struct btf_type *t);
+bool btf_type_is_decl_tag(const struct btf_type *t);
+bool btf_type_is_modifier(const struct btf_type *t);
+void btf_verifier_env_free(struct btf_verifier_env *env);
+
+extern const char * const btf_kind_str[NR_BTF_KINDS];
+
+/* Weak symbols - overridden by bpf.c when CONFIG_BPF_SYSCALL=y */
+void btf_free_bpf_data(struct btf *btf);
+void btf_put_bpf(struct btf *btf);
+
+#endif /* _KERNEL_BTF_BTF_H */
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 93f356d2b3d95..0009a7ac78462 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -395,14 +395,23 @@ config DEBUG_INFO_SPLIT
to know about the .dwo files and include them.
Incompatible with older versions of ccache.
-config DEBUG_INFO_BTF
- bool "Generate BTF type information"
+config BTF
+ bool "BTF type information support"
depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
- depends on BPF_SYSCALL
depends on PAHOLE_VERSION >= 122
# pahole uses elfutils, which does not have support for Hexagon relocations
depends on !HEXAGON
+ help
+ Enable the core BTF (BPF Type Format) infrastructure. This provides
+ BTF parsing, type lookup, and display functionality used by BPF and
+ other kernel subsystems.
+
+ This option is typically selected automatically by DEBUG_INFO_BTF.
+
+config DEBUG_INFO_BTF
+ bool "Generate BTF type information"
+ select BTF
help
Generate deduplicated BTF type information from DWARF debug info.
Turning this on requires pahole v1.22 or later, which will convert
--
2.51.0
^ permalink raw reply related
* Re: [RFC PATCH v1 1/1] This patch set introces a new action: DAMOS_COLLAPSE.
From: SeongJae Park @ 2026-03-25 1:29 UTC (permalink / raw)
To: Gutierrez Asier
Cc: SeongJae Park, artem.kuzin, stepanov.anatoly, wangkefeng.wang,
yanquanmin1, zuoze1, damon, akpm, ljs, Liam.Howlett, vbabka, rppt,
surenb, mhocko, corbet, skhan, linux-doc, linux-mm, linux-kernel
In-Reply-To: <6a07479d-0ddb-442e-bc1b-027f4c6d3b77@huawei-partners.com>
On Tue, 24 Mar 2026 18:01:23 +0300 Gutierrez Asier <gutierrez.asier@huawei-partners.com> wrote:
>
>
> On 3/24/2026 5:12 PM, SeongJae Park wrote:
> > On Tue, 24 Mar 2026 16:57:22 +0300 Gutierrez Asier <gutierrez.asier@huawei-partners.com> wrote:
> >
> >>
> >>
> >> On 3/24/2026 3:39 AM, SeongJae Park wrote:
> >>> Hello Asier,
> >>>
> >>> On Mon, 23 Mar 2026 14:56:45 +0000 <gutierrez.asier@huawei-partners.com> wrote:
> >>>
> >>>> From: Asier Gutierrez <gutierrez.asier@huawei-partners.com>
> >>>>
> >>>> For DAMOS_HUGEPAGE and DAMOS_NOHUGEPAGE to work, khugepaged should be
> >>>> working, since it relies on hugepage_madvise to add a new slot. This
> >>>> slot should be picked up by khugepaged and eventually collapse (or
> >>>> not, if we are using DAMOS_NOHUGEPAGE) the pages. If THP is not
> >>>> enabled, khugepaged will not be working, and therefore no collapse
> >>>> will happen.
> >>>>
> >>>> DAMOS_COLLAPSE eventually calls madvise_collapse, which will collapse
> >>>> the address range synchronously.
> >>>>
> >>>> This new action may be required to support autotuning with hugepage as
> >>>> a goal[1].
> >>>>
> >>>> [1]: https://lore.kernel.org/damon/20260313000816.79933-1-sj@kernel.org/
> >>>>
> >>>> ---------
> >>>> Benchmarks:
> >>>>
> >>>> T n: THP never
> >>>> T m: THP madvise
> >>>> D h: DAMON action hugepage
> >>>> D c: DAMON action collapse
> >>>>
> >>>> +------------------+----------+----------+----------+
> >>>> | | T n, D h | T m, D h | T n, D c |
> >>>> +------------------+----------+----------+----------+
> >>>> | Total memory use | 2.07 | 2.09 | 2.07 |
> >>>> | Huge pages | 0 | 1.3 | 1.25 |
> >>>> +------------------+----------+----------+----------+
> >>>
> >>> Thank you for sharing the benchmark results! But, I'm having a hard time to
> >>> understand what this really means. Could you please further clarify the setup
> >>> of the benchmarks and interpretation of the results?
> >> I will fix the cover in the next version, which I will submit soon.
> >>
> >> I tested the patch in a physical server with MariaDB 10.5. I run
> >> sysbench to load the server.
> >>
> >> I check 3 scenarios:
> >> - DAMON action hugepage for the database task, THP as never
> >> - DAMON action hugepage, THP madvise
> >> - DAMON action collapse, THP never
> >>
> >> I compared the memory consumption, both in overall in the server and
> >> anonymous huge page consumption. The results are in the table
> >>
> >> T n: THP never
> >> T m: THP madvise
> >> D h: DAMON action hugepage
> >> D c: DAMON action collapse
> >
> > Thank you for sharing the details of the setup, Asier.
> >
> >>
> >> +------------------+----------+----------+----------+
> >> | | T n, D h | T m, D h | T n, D c |
> >> +------------------+----------+----------+----------+
> >> | Total memory use | 2.07 | 2.09 | 2.07 |
> >> | Huge pages | 0 | 1.3 | 1.25 |
> >> +------------------+----------+----------+----------+
> >
> > Could you please further share interpretation of the results? What these
> > results mean? Does it show some benefit of DAMOS_COLLAPSE?
>
> In the first row is the total memory consumption in GB by the server
> and in the second row the huge page consumption.
>
> What this table shows is that DAMON action "hugepage" doesn't lead
> to any hugepage collapse unless THP is set to madvise. If DAMON
> action is set to "collapse", hugepage collapses happen, even when
> THP is disabled. The memory consumption for DAMON "collapse" is
> slightly lower than with "hugepage", since madvise applies to the
> entire server and other application may request to collapse pages
> through madvise.
>
> Regarding the performance, I saw improvement from no hugepage at all
> to use of hugepages. No significant difference between "hugepage" and
> "collapse" actions. I will have a look to the logs and publish it a bit
> later.
Makes sense, thank you for clarification Asier!
So this result shows the functionality well. Meanwhile, it doesn't clearly
show performance gain. Given the small size of the change and rationale of
MADV_COLLAPSE that I believe explored when it was introduced, I think this test
result is also good. I wouldn't strongly require more tests for this patch.
That said, if you find some more interesting thing from logs, please share, by
or after the next version of this patch.
Thanks,
SJ
[...]
^ permalink raw reply
* [PATCH v5 0/3] PCI Controller event and LTSSM tracepoint support
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
This patch-set adds new pci controller event and LTSSM tracepoint used by host drivers
which provide LTSSM trace functionality. The first user is pcie-dw-rockchip with a 256
Bytes FIFO for recording LTSSM transition.
Testing
=========
This series was tested on RK3588/RK3588s EVB1 with NVMe SSD connected to PCIe3 and PCIe2
root ports.
echo 1 > /sys/kernel/debug/tracing/events/pci_controller/pcie_ltssm_state_transition/enable
cat /sys/kernel/debug/tracing/trace_pipe
# tracer: nop
#
# entries-in-buffer/entries-written: 64/64 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
kworker/0:0-9 [000] ..... 5.600194: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_ACT rate: Unknown
kworker/0:0-9 [000] ..... 5.600198: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_WAIT rate: Unknown
kworker/0:0-9 [000] ..... 5.600199: pcie_ltssm_state_transition: dev: a40000000.pcie state: DETECT_ACT rate: Unknown
kworker/0:0-9 [000] ..... 5.600201: pcie_ltssm_state_transition: dev: a40000000.pcie state: POLL_ACTIVE rate: Unknown
kworker/0:0-9 [000] ..... 5.600202: pcie_ltssm_state_transition: dev: a40000000.pcie state: POLL_CONFIG rate: Unknown
kworker/0:0-9 [000] ..... 5.600204: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LINKWD_START rate: Unknown
kworker/0:0-9 [000] ..... 5.600206: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LINKWD_ACEPT rate: Unknown
kworker/0:0-9 [000] ..... 5.600207: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LANENUM_WAI rate: Unknown
kworker/0:0-9 [000] ..... 5.600208: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_LANENUM_ACEPT rate: Unknown
kworker/0:0-9 [000] ..... 5.600210: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_COMPLETE rate: Unknown
kworker/0:0-9 [000] ..... 5.600212: pcie_ltssm_state_transition: dev: a40000000.pcie state: CFG_IDLE rate: Unknown
kworker/0:0-9 [000] ..... 5.600213: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 2.5 GT/s
kworker/0:0-9 [000] ..... 5.600214: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: Unknown
kworker/0:0-9 [000] ..... 5.600216: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: Unknown
kworker/0:0-9 [000] ..... 5.600217: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_SPEED rate: Unknown
kworker/0:0-9 [000] ..... 5.600218: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: Unknown
kworker/0:0-9 [000] ..... 5.600220: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ1 rate: Unknown
kworker/0:0-9 [000] ..... 5.600221: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ2 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600222: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ3 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600224: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600225: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600226: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600227: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600228: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600229: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600231: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600232: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600233: pcie_ltssm_state_transition: dev: a40000000.pcie state: L123_SEND_EIDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600234: pcie_ltssm_state_transition: dev: a40000000.pcie state: L1_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600236: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_LOCK rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600237: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_RCVRCFG rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600238: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_IDLE rate: 8.0 GT/s
kworker/0:0-9 [000] ..... 5.600239: pcie_ltssm_state_transition: dev: a40000000.pcie state: L0 rate: 8.0 GT/s
Changes in v5:
- rebase
- use EM/EMe instead
- remove reg/unreg function and back to use TRACE_EVENT
- use trace_pcie_ltssm_state_transition_enabled()
Changes in v4:
- use TRACE_EVENT_FN to notify when to start and stop the tracepoint,
and export pci_ltssm_tp_enabled() for host drivers to use
- skip trace if pci_ltssm_tp_enabled() is false.(Steven)
- wrap into 80 columns(Bjorn)
Changes in v3:
- add TRACE_DEFINE_ENUM for all enums(Steven Rostedt)
- Add toctree entry in Documentation/trace/index.rst(Bagas Sanjaya)
- fix mismatch section underline length(Bagas Sanjaya)
- Make example snippets in code block(Bagas Sanjaya)
- warp context into 80 columns and fix the file name(Bjorn)
- reorder variables(Mani)
- rename loop to i; rename en to enable(Mani)
- use FIELD_GET(Mani)
- add comment about how the FIFO works(Mani)
Changes in v2:
- use tracepoint
Shawn Lin (3):
PCI: trace: Add PCI controller LTSSM transition tracepoint
Documentation: tracing: Add PCI controller event documentation
PCI: dw-rockchip: Add pcie_ltssm_state_transition trace support
Documentation/trace/events-pci-controller.rst | 42 ++++++++++
Documentation/trace/index.rst | 1 +
drivers/pci/controller/dwc/pcie-dw-rockchip.c | 111 ++++++++++++++++++++++++++
drivers/pci/trace.c | 1 +
include/trace/events/pci_controller.h | 58 ++++++++++++++
5 files changed, 213 insertions(+)
create mode 100644 Documentation/trace/events-pci-controller.rst
create mode 100644 include/trace/events/pci_controller.h
--
2.7.4
^ permalink raw reply
* [PATCH v5 1/3] PCI: trace: Add PCI controller LTSSM transition tracepoint
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
In-Reply-To: <1774403912-210670-1-git-send-email-shawn.lin@rock-chips.com>
Some platforms may provide LTSSM trace functionality, recording historical
LTSSM state transition information. This is very useful for debugging, such
as when certain devices cannot be recognized or link broken during test.
Implement the pci controller tracepoint for recording LTSSM and rate.
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
---
Changes in v5:
- use EM/EMe instead
- remove reg/unreg function and back to use TRACE_EVENT
Changes in v4:
- use TRACE_EVENT_FN to notify when to start and stop the tracepoint,
and export pci_ltssm_tp_enabled() for host drivers to use
Changes in v3:
- add TRACE_DEFINE_ENUM for all enums(Steven Rostedt)
Changes in v2: None
drivers/pci/trace.c | 1 +
include/trace/events/pci_controller.h | 58 +++++++++++++++++++++++++++++++++++
2 files changed, 59 insertions(+)
create mode 100644 include/trace/events/pci_controller.h
diff --git a/drivers/pci/trace.c b/drivers/pci/trace.c
index cf11abc..c1da9d3 100644
--- a/drivers/pci/trace.c
+++ b/drivers/pci/trace.c
@@ -9,3 +9,4 @@
#define CREATE_TRACE_POINTS
#include <trace/events/pci.h>
+#include <trace/events/pci_controller.h>
diff --git a/include/trace/events/pci_controller.h b/include/trace/events/pci_controller.h
new file mode 100644
index 0000000..a4b387c
--- /dev/null
+++ b/include/trace/events/pci_controller.h
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM pci_controller
+
+#if !defined(_TRACE_HW_EVENT_PCI_CONTROLLER_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_HW_EVENT_PCI_CONTROLLER_H
+
+#include <uapi/linux/pci_regs.h>
+#include <linux/tracepoint.h>
+
+#define RATE \
+ EM(PCIE_SPEED_2_5GT, "2.5 GT/s") \
+ EM(PCIE_SPEED_5_0GT, "5.0 GT/s") \
+ EM(PCIE_SPEED_8_0GT, "8.0 GT/s") \
+ EM(PCIE_SPEED_16_0GT, "16.0 GT/s") \
+ EM(PCIE_SPEED_32_0GT, "32.0 GT/s") \
+ EM(PCIE_SPEED_64_0GT, "64.0 GT/s") \
+ EMe(PCI_SPEED_UNKNOWN, "Unknown")
+
+
+#undef EM
+#undef EMe
+#define EM(a, b) TRACE_DEFINE_ENUM(a);
+#define EMe(a, b) TRACE_DEFINE_ENUM(a);
+
+RATE
+
+#undef EM
+#undef EMe
+#define EM(a, b) {a, b},
+#define EMe(a, b) {a, b}
+
+TRACE_EVENT(pcie_ltssm_state_transition,
+ TP_PROTO(const char *dev_name, const char *state, u32 rate),
+ TP_ARGS(dev_name, state, rate),
+
+ TP_STRUCT__entry(
+ __string(dev_name, dev_name)
+ __string(state, state)
+ __field(u32, rate)
+ ),
+
+ TP_fast_assign(
+ __assign_str(dev_name);
+ __assign_str(state);
+ __entry->rate = rate;
+ ),
+
+ TP_printk("dev: %s state: %s rate: %s",
+ __get_str(dev_name), __get_str(state),
+ __print_symbolic(__entry->rate, RATE)
+ )
+);
+
+#endif /* _TRACE_HW_EVENT_PCI_CONTROLLER_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
--
2.7.4
^ permalink raw reply related
* [PATCH v5 2/3] Documentation: tracing: Add PCI controller event documentation
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
In-Reply-To: <1774403912-210670-1-git-send-email-shawn.lin@rock-chips.com>
The available tracepoint, pcie_ltssm_state_transition, monitors the LTSSM
state transition for debugging purpose. Add description about it.
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
---
Changes in v5: None
Changes in v4: None
Changes in v3:
- Add toctree entry in Documentation/trace/index.rst(Bagas Sanjaya)
- fix mismatch section underline length(Bagas Sanjaya)
- Make example snippets in code block(Bagas Sanjaya)
- warp context into 80 columns and fix the file name(Bjorn)
Changes in v2: None
Documentation/trace/events-pci-controller.rst | 42 +++++++++++++++++++++++++++
Documentation/trace/index.rst | 1 +
2 files changed, 43 insertions(+)
create mode 100644 Documentation/trace/events-pci-controller.rst
diff --git a/Documentation/trace/events-pci-controller.rst b/Documentation/trace/events-pci-controller.rst
new file mode 100644
index 0000000..cb9f715
--- /dev/null
+++ b/Documentation/trace/events-pci-controller.rst
@@ -0,0 +1,42 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+======================================
+Subsystem Trace Points: PCI Controller
+======================================
+
+Overview
+========
+The PCI controller tracing system provides tracepoints to monitor controller
+level information for debugging purpose. The events normally show up here:
+
+ /sys/kernel/tracing/events/pci_controller
+
+Cf. include/trace/events/pci_controller.h for the events definitions.
+
+Available Tracepoints
+=====================
+
+pcie_ltssm_state_transition
+---------------------------
+
+Monitors PCIe LTSSM state transition including state and rate information
+::
+
+ pcie_ltssm_state_transition "dev: %s state: %s rate: %s\n"
+
+**Parameters**:
+
+* ``dev`` - PCIe controller instance
+* ``state`` - PCIe LTSSM state
+* ``rate`` - PCIe date rate
+
+**Example Usage**:
+
+.. code-block:: shell
+
+ # Enable the tracepoint
+ echo 1 > /sys/kernel/debug/tracing/events/pci_controller/pcie_ltssm_state_transition/enable
+
+ # Monitor events (the following output is generated when a device is linking)
+ cat /sys/kernel/debug/tracing/trace_pipe
+ kworker/0:0-9 [000] ..... 5.600221: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ2 rate: 8.0 GT/s
diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst
index 036db96..5d9bf469 100644
--- a/Documentation/trace/index.rst
+++ b/Documentation/trace/index.rst
@@ -55,6 +55,7 @@ applications.
events-nmi
events-msr
events-pci
+ events-pci-controller
boottime-trace
histogram
histogram-design
--
2.7.4
^ permalink raw reply related
* Re: [RFC] btf: split core BTF parsing out of BPF subsystem into kernel/btf/
From: bot+bpf-ci @ 2026-03-25 2:15 UTC (permalink / raw)
To: sashal, sashal
Cc: James.Bottomley, adobriyan, akpm, alexei.starovoitov, ast, bp,
bpf, corbet, dave.hansen, davidgow, deller, geert, gregkh, hpa,
jgross, jpoimboe, kees, laurent.pinchart, linux-doc, linux-kbuild,
linux-kernel, linux-modules, masahiroy, mcgrof, mingo, nathan,
nsc, peterz, petr.pavlu, pmladek, rdunlap, rostedt, tglx, vbabka,
x86, ast, andrii, daniel, martin.lau, eddyz87, yonghong.song, clm,
ihor.solodrai
In-Reply-To: <20260325011853.657295-1-sashal@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2607 bytes --]
> diff --git a/kernel/btf/bpf.c b/kernel/btf/bpf.c
> new file mode 100644
> index 0000000000000..ef973771bb2ab
> --- /dev/null
> +++ b/kernel/btf/bpf.c
[ ... ]
> +struct btf_struct_ops_tab {
> + u32 cnt;
> + u32 capacity;
> + struct bpf_struct_ops_desc ops[];
> +};
> + u32 id;
> + struct rcu_head rcu;
> + struct btf_kfunc_set_tab *kfunc_set_tab;
> + struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
> + struct btf_struct_metas *struct_meta_tab;
> + struct btf_struct_ops_tab *struct_ops_tab;
> +s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
These six declarations between the closing brace of struct
btf_struct_ops_tab and bpf_find_btf_id() appear to be orphaned
fields from the original struct btf definition that were left
behind during the split.
They are parsed as file-scope global variable definitions. With
-fno-common, the name "id" in particular is likely to collide at
link time. Were these meant to be removed here, given that they
already appear correctly inside struct btf in kernel/btf/btf.h
under the CONFIG_BPF_SYSCALL guard?
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 93f356d2b3d95..0009a7ac78462 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -395,14 +395,23 @@ config DEBUG_INFO_SPLIT
[ ... ]
> +config BTF
> + bool "BTF type information support"
> + depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
> + depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
> + depends on PAHOLE_VERSION >= 122
> + # pahole uses elfutils, which does not have support for Hexagon relocations
> + depends on !HEXAGON
[ ... ]
> +config DEBUG_INFO_BTF
> + bool "Generate BTF type information"
> + select BTF
The old DEBUG_INFO_BTF had its own depends on lines:
depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
depends on BPF_SYSCALL
depends on PAHOLE_VERSION >= 122
depends on !HEXAGON
Those constraints now live only on CONFIG_BTF, and DEBUG_INFO_BTF
uses "select BTF" with no depends of its own. Since select bypasses
the selected symbol's depends-on checks, DEBUG_INFO_BTF is now
visible and selectable even when BTF's dependencies are not met
(e.g. on Hexagon, or with PAHOLE_VERSION < 122).
Should DEBUG_INFO_BTF use "depends on BTF" instead of (or in
addition to) "select BTF", so that it inherits the same
constraints?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23520643445
^ permalink raw reply
* [PATCH v5 3/3] PCI: dw-rockchip: Add pcie_ltssm_state_transition trace support
From: Shawn Lin @ 2026-03-25 1:58 UTC (permalink / raw)
To: Manivannan Sadhasivam, Bjorn Helgaas
Cc: linux-rockchip, linux-pci, linux-trace-kernel, linux-doc,
Steven Rostedt, Shawn Lin
In-Reply-To: <1774403912-210670-1-git-send-email-shawn.lin@rock-chips.com>
Rockchip platforms provide a 64x4 bytes debug FIFO to trace the
LTSSM history. Any LTSSM change will be recorded. It's useful
for debug purpose, for example link failure, etc.
Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
---
Changes in v5:
- rebase
- use trace_pcie_ltssm_state_transition_enabled()
Changes in v4:
- skip trace if pci_ltssm_tp_enabled() is false.(Steven)
- wrap into 80 columns(Bjorn)
Changes in v3:
- reorder variables(Mani)
- rename loop to i; rename en to enable(Mani)
- use FIELD_GET(Mani)
- add comment about how the FIFO works(Mani)
Changes in v2:
- use tracepoint
drivers/pci/controller/dwc/pcie-dw-rockchip.c | 111 ++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
diff --git a/drivers/pci/controller/dwc/pcie-dw-rockchip.c b/drivers/pci/controller/dwc/pcie-dw-rockchip.c
index bb5d1a3..e737103 100644
--- a/drivers/pci/controller/dwc/pcie-dw-rockchip.c
+++ b/drivers/pci/controller/dwc/pcie-dw-rockchip.c
@@ -22,6 +22,8 @@
#include <linux/platform_device.h>
#include <linux/regmap.h>
#include <linux/reset.h>
+#include <linux/workqueue.h>
+#include <trace/events/pci_controller.h>
#include "../../pci.h"
#include "pcie-designware.h"
@@ -73,6 +75,20 @@
#define PCIE_CLIENT_CDM_RASDES_TBA_L1_1 BIT(4)
#define PCIE_CLIENT_CDM_RASDES_TBA_L1_2 BIT(5)
+/* Debug FIFO information */
+#define PCIE_CLIENT_DBG_FIFO_MODE_CON 0x310
+#define PCIE_CLIENT_DBG_EN 0xffff0007
+#define PCIE_CLIENT_DBG_DIS 0xffff0000
+#define PCIE_CLIENT_DBG_FIFO_PTN_HIT_D0 0x320
+#define PCIE_CLIENT_DBG_FIFO_PTN_HIT_D1 0x324
+#define PCIE_CLIENT_DBG_FIFO_TRN_HIT_D0 0x328
+#define PCIE_CLIENT_DBG_FIFO_TRN_HIT_D1 0x32c
+#define PCIE_CLIENT_DBG_TRANSITION_DATA 0xffff0000
+#define PCIE_CLIENT_DBG_FIFO_STATUS 0x350
+#define PCIE_DBG_FIFO_RATE_MASK GENMASK(22, 20)
+#define PCIE_DBG_FIFO_L1SUB_MASK GENMASK(10, 8)
+#define PCIE_DBG_LTSSM_HISTORY_CNT 64
+
/* Hot Reset Control Register */
#define PCIE_CLIENT_HOT_RESET_CTRL 0x180
#define PCIE_LTSSM_APP_DLY2_EN BIT(1)
@@ -98,6 +114,7 @@ struct rockchip_pcie {
struct irq_domain *irq_domain;
const struct rockchip_pcie_of_data *data;
bool supports_clkreq;
+ struct delayed_work trace_work;
};
struct rockchip_pcie_of_data {
@@ -208,6 +225,96 @@ static enum dw_pcie_ltssm rockchip_pcie_get_ltssm(struct dw_pcie *pci)
return rockchip_pcie_get_ltssm_reg(rockchip) & PCIE_LTSSM_STATUS_MASK;
}
+#ifdef CONFIG_TRACING
+static void rockchip_pcie_ltssm_trace_work(struct work_struct *work)
+{
+ struct rockchip_pcie *rockchip = container_of(work,
+ struct rockchip_pcie,
+ trace_work.work);
+ struct dw_pcie *pci = &rockchip->pci;
+ enum dw_pcie_ltssm state;
+ u32 i, l1ss, prev_val = DW_PCIE_LTSSM_UNKNOWN, rate, val;
+
+ if (!trace_pcie_ltssm_state_transition_enabled())
+ goto skip_trace;
+
+ for (i = 0; i < PCIE_DBG_LTSSM_HISTORY_CNT; i++) {
+ val = rockchip_pcie_readl_apb(rockchip,
+ PCIE_CLIENT_DBG_FIFO_STATUS);
+ rate = FIELD_GET(PCIE_DBG_FIFO_RATE_MASK, val);
+ l1ss = FIELD_GET(PCIE_DBG_FIFO_L1SUB_MASK, val);
+ val = FIELD_GET(PCIE_LTSSM_STATUS_MASK, val);
+
+ /*
+ * Hardware Mechanism: The ring FIFO employs two tracking
+ * counters:
+ * - 'last-read-point': maintains the user's last read position
+ * - 'last-valid-point': tracks the HW's last state update
+ *
+ * Software Handling: When two consecutive LTSSM states are
+ * identical, it indicates invalid subsequent data in the FIFO.
+ * In this case, we skip the remaining entries. The dual counter
+ * design ensures that on the next state transition, reading can
+ * resume from the last user position.
+ */
+ if ((i > 0 && val == prev_val) || val > DW_PCIE_LTSSM_RCVRY_EQ3)
+ break;
+
+ state = prev_val = val;
+ if (val == DW_PCIE_LTSSM_L1_IDLE) {
+ if (l1ss == 2)
+ state = DW_PCIE_LTSSM_L1_2;
+ else if (l1ss == 1)
+ state = DW_PCIE_LTSSM_L1_1;
+ }
+
+ trace_pcie_ltssm_state_transition(dev_name(pci->dev),
+ dw_pcie_ltssm_status_string(state),
+ ((rate + 1) > pci->max_link_speed) ?
+ PCI_SPEED_UNKNOWN : PCIE_SPEED_2_5GT + rate);
+ }
+
+skip_trace:
+ schedule_delayed_work(&rockchip->trace_work, msecs_to_jiffies(5000));
+}
+
+static void rockchip_pcie_ltssm_trace(struct rockchip_pcie *rockchip,
+ bool enable)
+{
+ if (enable) {
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_TRANSITION_DATA,
+ PCIE_CLIENT_DBG_FIFO_PTN_HIT_D0);
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_TRANSITION_DATA,
+ PCIE_CLIENT_DBG_FIFO_PTN_HIT_D1);
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_TRANSITION_DATA,
+ PCIE_CLIENT_DBG_FIFO_TRN_HIT_D0);
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_TRANSITION_DATA,
+ PCIE_CLIENT_DBG_FIFO_TRN_HIT_D1);
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_EN,
+ PCIE_CLIENT_DBG_FIFO_MODE_CON);
+
+ INIT_DELAYED_WORK(&rockchip->trace_work,
+ rockchip_pcie_ltssm_trace_work);
+ schedule_delayed_work(&rockchip->trace_work, 0);
+ } else {
+ rockchip_pcie_writel_apb(rockchip,
+ PCIE_CLIENT_DBG_DIS,
+ PCIE_CLIENT_DBG_FIFO_MODE_CON);
+ cancel_delayed_work_sync(&rockchip->trace_work);
+ }
+}
+#else
+static void rockchip_pcie_ltssm_trace(struct rockchip_pcie *rockchip,
+ bool enable)
+{
+}
+#endif
+
static void rockchip_pcie_enable_ltssm(struct rockchip_pcie *rockchip)
{
rockchip_pcie_writel_apb(rockchip, PCIE_CLIENT_ENABLE_LTSSM,
@@ -291,6 +398,9 @@ static int rockchip_pcie_start_link(struct dw_pcie *pci)
* 100us as we don't know how long should the device need to reset.
*/
msleep(PCIE_T_PVPERL_MS);
+
+ rockchip_pcie_ltssm_trace(rockchip, true);
+
gpiod_set_value_cansleep(rockchip->rst_gpio, 1);
return 0;
@@ -301,6 +411,7 @@ static void rockchip_pcie_stop_link(struct dw_pcie *pci)
struct rockchip_pcie *rockchip = to_rockchip_pcie(pci);
rockchip_pcie_disable_ltssm(rockchip);
+ rockchip_pcie_ltssm_trace(rockchip, false);
}
static int rockchip_pcie_host_init(struct dw_pcie_rp *pp)
--
2.7.4
^ permalink raw reply related
* Re: [PATCH v5 00/21] Virtual Swap Space
From: Askar Safin @ 2026-03-25 2:35 UTC (permalink / raw)
To: nphamcs
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, kasong,
kernel-team, lance.yang, lenb, linux-doc, linux-kernel, linux-mm,
linux-pm, lorenzo.stoakes, matthew.brost, mhocko, muchun.song,
npache, pavel, peterx, peterz, pfalcato, rafael, rakie.kim, riel,
roman.gushchin, rppt, ryan.roberts, ryncsn, shakeel.butt,
shikemeng, surenb, tglx, vbabka, weixugc, willy, ying.huang,
yosry.ahmed, yuanchu, zhengqi.arch, ziy
In-Reply-To: <CAKEwX=MgoPmiFdBQXK_4=XuR-8mVpGr+3Ku2MfjPmHCeuUdGJg@mail.gmail.com>
Nhat Pham <nphamcs@gmail.com>:
> I'm aware of that work. It's an improvement, but my understanding is:
Thank you for answer!
Also, is it possible to have checksummed swap?
I want to have checksummed swap to be protected from disk bit-rot
(I already have ECC memory, so RAM is protected).
And hibernation image should be protected, too.
I tried to put swap on top of dm-integrity, but this is
incompatible with hibernation in mainline kernel.
--
Askar Safin
^ permalink raw reply
* [PATCH v4 00/10] gpu: nova-core: convert registers to use the kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
nova-core carried its own helper macro to declare register types. Its
purpose was to be temporary since the beginning, and to serve as a
testbed to develop an equivalent that could be used kernel-wide.
That equivalent has now been merged, so it is time to retire the
nova-core local version.
The kernel register macro has evolved into something significantly
different from the one in nova-core, so it cannot be used as a drop-in
replacement. All declarations and sites using registers need to be
updated. No semantic change should happen as a result.
All the patches in this series could also be squashed into a single one
without altering their reviewability significantly.
I hope to be able to merge this quickly so we can rebase in-flight
series to use the updated register syntax.
This series is based on drm-rust-next as of 2026-03-25 and has been
tested on Turing.
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Changes in v4:
- Fix incorrect squash of `bounded_enum` code into the wrong patch.
- Move `bounded_enum` macro to `num` module and use it to generate
`Architecture.`
- Link to v3: https://patch.msgid.link/20260323-b4-nova-register-v3-0-ae2486ecef1b@nvidia.com
Changes in v3:
- Remove obsolete part of PMC commit message.
- Add { } brackets around const expression used in `Bounded`
constructor.
- Use `MAX..` arm to `bounded_enum` to catch missing values in the
required range of the `From` implementation. (thanks Gary!)
- Remove unneeded const asserts that are already caught by `Bounded::new`.
- Link to v2: https://patch.msgid.link/20260320-b4-nova-register-v2-0-88fcf103e8d4@nvidia.com
Changes in v2:
- Reorder register fields in descending order.
- Drop `nv_reg` macro and use the kernel `register` macro directly.
- Make `FUSE_UCODE1_VERSION` registers return a u16 directly.
- Use `into_raw` instead of accessing the inner value of registers directly.
- Use safer macro to generate the `From` and `TryFrom` implementations
of falcon enum types.
- Link to v1: https://patch.msgid.link/20260318-b4-nova-register-v1-0-22a358aa4c63@nvidia.com
---
Alexandre Courbot (10):
gpu: nova-core: introduce `bounded_enum` macro
gpu: nova-core: convert PMC registers to kernel register macro
gpu: nova-core: convert PBUS registers to kernel register macro
gpu: nova-core: convert PFB registers to kernel register macro
gpu: nova-core: convert GC6 registers to kernel register macro
gpu: nova-core: convert FUSE registers to kernel register macro
gpu: nova-core: convert PDISP registers to kernel register macro
gpu: nova-core: convert falcon registers to kernel register macro
gpu: nova-core: remove `io::` qualifier to register macro invocations
Documentation: nova: remove register abstraction task
Documentation/gpu/nova/core/todo.rst | 76 ---
drivers/gpu/nova-core/falcon.rs | 524 ++++++---------
drivers/gpu/nova-core/falcon/gsp.rs | 27 +-
drivers/gpu/nova-core/falcon/hal/ga102.rs | 70 +-
drivers/gpu/nova-core/falcon/hal/tu102.rs | 12 +-
drivers/gpu/nova-core/falcon/sec2.rs | 17 +-
drivers/gpu/nova-core/fb.rs | 6 +-
drivers/gpu/nova-core/fb/hal/ga100.rs | 37 +-
drivers/gpu/nova-core/fb/hal/ga102.rs | 7 +-
drivers/gpu/nova-core/fb/hal/tu102.rs | 17 +-
drivers/gpu/nova-core/firmware/fwsec/bootloader.rs | 19 +-
drivers/gpu/nova-core/gfw.rs | 11 +-
drivers/gpu/nova-core/gpu.rs | 53 +-
drivers/gpu/nova-core/gsp/boot.rs | 11 +-
drivers/gpu/nova-core/gsp/cmdq.rs | 9 +-
drivers/gpu/nova-core/nova_core.rs | 1 +
drivers/gpu/nova-core/num.rs | 80 +++
drivers/gpu/nova-core/regs.rs | 601 +++++++++--------
drivers/gpu/nova-core/regs/macros.rs | 739 ---------------------
19 files changed, 779 insertions(+), 1538 deletions(-)
---
base-commit: dff8302ca1d0e773c90dbeeb05e759f995c95482
change-id: 20260318-b4-nova-register-6908b5118552
Best regards,
--
Alexandre Courbot <acourbot@nvidia.com>
^ permalink raw reply
* [PATCH v4 01/10] gpu: nova-core: introduce `bounded_enum` macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Introduce a powered-up version of our ad-hoc `impl_from_enum_to_u8`
macro that allows the definition of an enum type associated to a
`Bounded` of a given width, and provides the `From` and `TryFrom`
implementations required to use that enum as a register field member.
This allows us to generate the required conversion implementations for
using the kernel register macro and skip some tedious boilerplate.
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/nova_core.rs | 1 +
drivers/gpu/nova-core/num.rs | 80 ++++++++++++++++++++++++++++++++++++++
2 files changed, 81 insertions(+)
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index ccd14b757b49..98675c69d2b7 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -21,6 +21,7 @@
mod gfw;
mod gpu;
mod gsp;
+#[macro_use]
mod num;
mod regs;
mod sbuffer;
diff --git a/drivers/gpu/nova-core/num.rs b/drivers/gpu/nova-core/num.rs
index c952a834e662..6c824b8d7b97 100644
--- a/drivers/gpu/nova-core/num.rs
+++ b/drivers/gpu/nova-core/num.rs
@@ -215,3 +215,83 @@ pub(crate) const fn [<$from _into_ $into>]<const N: $from>() -> $into {
impl_const_into!(u64 => { u8, u16, u32 });
impl_const_into!(u32 => { u8, u16 });
impl_const_into!(u16 => { u8 });
+
+/// Creates an enum type associated to a [`Bounded`](kernel::num::Bounded), with a [`From`]
+/// conversion to the associated `Bounded` and either a [`TryFrom`] or `From` conversion from the
+/// associated `Bounded`.
+// TODO[FPRI]: This is a temporary solution to be replaced with the corresponding derive macros
+// once they land.
+#[macro_export]
+macro_rules! bounded_enum {
+ (
+ $(#[$enum_meta:meta])*
+ $vis:vis enum $enum_type:ident with $from_impl:ident<Bounded<$width:ty, $length:literal>> {
+ $( $(#[doc = $variant_doc:expr])* $variant:ident = $value:expr),* $(,)*
+ }
+ ) => {
+ $(#[$enum_meta])*
+ $vis enum $enum_type {
+ $(
+ $(#[doc = $variant_doc])*
+ $variant = $value
+ ),*
+ }
+
+ impl core::convert::From<$enum_type> for kernel::num::Bounded<$width, $length> {
+ fn from(value: $enum_type) -> Self {
+ match value {
+ $($enum_type::$variant =>
+ kernel::num::Bounded::<$width, _>::new::<{ $value }>()),*
+ }
+ }
+ }
+
+ bounded_enum!(@impl_from $enum_type with $from_impl<Bounded<$width, $length>> {
+ $($variant = $value),*
+ });
+ };
+
+ // `TryFrom` implementation from associated `Bounded` to enum type.
+ (@impl_from $enum_type:ident with TryFrom<Bounded<$width:ty, $length:literal>> {
+ $($variant:ident = $value:expr),* $(,)*
+ }) => {
+ impl core::convert::TryFrom<kernel::num::Bounded<$width, $length>> for $enum_type {
+ type Error = kernel::error::Error;
+
+ fn try_from(
+ value: kernel::num::Bounded<$width, $length>
+ ) -> kernel::error::Result<Self> {
+ match value.get() {
+ $(
+ $value => Ok($enum_type::$variant),
+ )*
+ _ => Err(kernel::error::code::EINVAL),
+ }
+ }
+ }
+ };
+
+ // `From` implementation from associated `Bounded` to enum type. Triggers a build-time error if
+ // all possible values of the `Bounded` are not covered by the enum type.
+ (@impl_from $enum_type:ident with From<Bounded<$width:ty, $length:literal>> {
+ $($variant:ident = $value:expr),* $(,)*
+ }) => {
+ impl core::convert::From<kernel::num::Bounded<$width, $length>> for $enum_type {
+ fn from(value: kernel::num::Bounded<$width, $length>) -> Self {
+ const MAX: $width = 1 << $length;
+
+ // Makes the compiler optimizer aware of the possible range of values.
+ let value = value.get() & ((1 << $length) - 1);
+ match value {
+ $(
+ $value => $enum_type::$variant,
+ )*
+ // PANIC: we cannot reach this arm as all possible variants are handled by the
+ // match arms above. It is here to make the compiler complain if `$enum_type`
+ // does not cover all values of the `0..MAX` range.
+ MAX.. => unreachable!(),
+ }
+ }
+ }
+ }
+}
--
2.53.0
^ permalink raw reply related
* [PATCH v4 02/10] gpu: nova-core: convert PMC registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all PMC registers to use the kernel's register macro and update
the code accordingly.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/falcon.rs | 7 ++++--
drivers/gpu/nova-core/gpu.rs | 53 +++++++++++++----------------------------
drivers/gpu/nova-core/regs.rs | 50 ++++++++++++++++++++++++--------------
3 files changed, 53 insertions(+), 57 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon.rs b/drivers/gpu/nova-core/falcon.rs
index 5bf8da8760bf..123de6c55b45 100644
--- a/drivers/gpu/nova-core/falcon.rs
+++ b/drivers/gpu/nova-core/falcon.rs
@@ -13,7 +13,10 @@
DmaAddress,
DmaMask, //
},
- io::poll::read_poll_timeout,
+ io::{
+ poll::read_poll_timeout,
+ Io, //
+ },
prelude::*,
sync::aref::ARef,
time::Delta,
@@ -531,7 +534,7 @@ pub(crate) fn reset(&self, bar: &Bar0) -> Result {
self.hal.reset_wait_mem_scrubbing(bar)?;
regs::NV_PFALCON_FALCON_RM::default()
- .set_value(regs::NV_PMC_BOOT_0::read(bar).into())
+ .set_value(bar.read(regs::NV_PMC_BOOT_0).into())
.write(bar, &E::ID);
Ok(())
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 8579d632e717..0f6fe9a1b955 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -4,12 +4,15 @@
device,
devres::Devres,
fmt,
+ io::Io,
+ num::Bounded,
pci,
prelude::*,
sync::Arc, //
};
use crate::{
+ bounded_enum,
driver::Bar0,
falcon::{
gsp::Gsp as GspFalcon,
@@ -128,50 +131,26 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
}
}
-/// Enum representation of the GPU generation.
-///
-/// TODO: remove the `Default` trait implementation, and the `#[default]`
-/// attribute, once the register!() macro (which creates Architecture items) no
-/// longer requires it for read-only fields.
-#[derive(fmt::Debug, Default, Copy, Clone)]
-#[repr(u8)]
-pub(crate) enum Architecture {
- #[default]
- Turing = 0x16,
- Ampere = 0x17,
- Ada = 0x19,
-}
-
-impl TryFrom<u8> for Architecture {
- type Error = Error;
-
- fn try_from(value: u8) -> Result<Self> {
- match value {
- 0x16 => Ok(Self::Turing),
- 0x17 => Ok(Self::Ampere),
- 0x19 => Ok(Self::Ada),
- _ => Err(ENODEV),
- }
- }
-}
-
-impl From<Architecture> for u8 {
- fn from(value: Architecture) -> Self {
- // CAST: `Architecture` is `repr(u8)`, so this cast is always lossless.
- value as u8
+bounded_enum! {
+ /// Enum representation of the GPU generation.
+ #[derive(fmt::Debug, Copy, Clone)]
+ pub(crate) enum Architecture with TryFrom<Bounded<u32, 6>> {
+ Turing = 0x16,
+ Ampere = 0x17,
+ Ada = 0x19,
}
}
pub(crate) struct Revision {
- major: u8,
- minor: u8,
+ major: Bounded<u8, 4>,
+ minor: Bounded<u8, 4>,
}
impl From<regs::NV_PMC_BOOT_42> for Revision {
fn from(boot0: regs::NV_PMC_BOOT_42) -> Self {
Self {
- major: boot0.major_revision(),
- minor: boot0.minor_revision(),
+ major: boot0.major_revision().cast(),
+ minor: boot0.minor_revision().cast(),
}
}
}
@@ -208,13 +187,13 @@ fn new(dev: &device::Device, bar: &Bar0) -> Result<Spec> {
// from an earlier (pre-Fermi) era, and then using boot42 to precisely identify the GPU.
// Somewhere in the Rubin timeframe, boot0 will no longer have space to add new GPU IDs.
- let boot0 = regs::NV_PMC_BOOT_0::read(bar);
+ let boot0 = bar.read(regs::NV_PMC_BOOT_0);
if boot0.is_older_than_fermi() {
return Err(ENODEV);
}
- let boot42 = regs::NV_PMC_BOOT_42::read(bar);
+ let boot42 = bar.read(regs::NV_PMC_BOOT_42);
Spec::try_from(boot42).inspect_err(|_| {
dev_err!(dev, "Unsupported chipset: {}\n", boot42);
})
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 53f412f0ca32..58fb807605dd 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -8,6 +8,7 @@
pub(crate) mod macros;
use kernel::{
+ io,
prelude::*,
time, //
};
@@ -37,18 +38,38 @@
// PMC
-register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" {
- 3:0 minor_revision as u8, "Minor revision of the chip";
- 7:4 major_revision as u8, "Major revision of the chip";
- 8:8 architecture_1 as u8, "MSB of the architecture";
- 23:20 implementation as u8, "Implementation version of the architecture";
- 28:24 architecture_0 as u8, "Lower bits of the architecture";
-});
+io::register! {
+ /// Basic revision information about the GPU.
+ pub(crate) NV_PMC_BOOT_0(u32) @ 0x00000000 {
+ /// Lower bits of the architecture.
+ 28:24 architecture_0;
+ /// Implementation version of the architecture.
+ 23:20 implementation;
+ /// MSB of the architecture.
+ 8:8 architecture_1;
+ /// Major revision of the chip.
+ 7:4 major_revision;
+ /// Minor revision of the chip.
+ 3:0 minor_revision;
+ }
+
+ /// Extended architecture information.
+ pub(crate) NV_PMC_BOOT_42(u32) @ 0x00000a00 {
+ /// Architecture value.
+ 29:24 architecture ?=> Architecture;
+ /// Implementation version of the architecture.
+ 23:20 implementation;
+ /// Major revision of the chip.
+ 19:16 major_revision;
+ /// Minor revision of the chip.
+ 15:12 minor_revision;
+ }
+}
impl NV_PMC_BOOT_0 {
pub(crate) fn is_older_than_fermi(self) -> bool {
// From https://github.com/NVIDIA/open-gpu-doc/tree/master/manuals :
- const NV_PMC_BOOT_0_ARCHITECTURE_GF100: u8 = 0xc;
+ const NV_PMC_BOOT_0_ARCHITECTURE_GF100: u32 = 0xc;
// Older chips left arch1 zeroed out. That, combined with an arch0 value that is less than
// GF100, means "older than Fermi".
@@ -56,13 +77,6 @@ pub(crate) fn is_older_than_fermi(self) -> bool {
}
}
-register!(NV_PMC_BOOT_42 @ 0x00000a00, "Extended architecture information" {
- 15:12 minor_revision as u8, "Minor revision of the chip";
- 19:16 major_revision as u8, "Major revision of the chip";
- 23:20 implementation as u8, "Implementation version of the architecture";
- 29:24 architecture as u8 ?=> Architecture, "Architecture value";
-});
-
impl NV_PMC_BOOT_42 {
/// Combines `architecture` and `implementation` to obtain a code unique to the chipset.
pub(crate) fn chipset(self) -> Result<Chipset> {
@@ -76,8 +90,8 @@ pub(crate) fn chipset(self) -> Result<Chipset> {
/// Returns the raw architecture value from the register.
fn architecture_raw(self) -> u8 {
- ((self.0 >> Self::ARCHITECTURE_RANGE.start()) & ((1 << Self::ARCHITECTURE_RANGE.len()) - 1))
- as u8
+ ((self.into_raw() >> Self::ARCHITECTURE_RANGE.start())
+ & ((1 << Self::ARCHITECTURE_RANGE.len()) - 1)) as u8
}
}
@@ -86,7 +100,7 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
write!(
f,
"boot42 = 0x{:08x} (architecture 0x{:x}, implementation 0x{:x})",
- self.0,
+ self.inner,
self.architecture_raw(),
self.implementation()
)
--
2.53.0
^ permalink raw reply related
* [PATCH v4 03/10] gpu: nova-core: convert PBUS registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all PBUS registers to use the kernel's register macro and update
the code accordingly.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/gsp/boot.rs | 5 ++++-
drivers/gpu/nova-core/regs.rs | 12 +++++++-----
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/gsp/boot.rs b/drivers/gpu/nova-core/gsp/boot.rs
index e55210ebb6d1..3a0124818956 100644
--- a/drivers/gpu/nova-core/gsp/boot.rs
+++ b/drivers/gpu/nova-core/gsp/boot.rs
@@ -4,6 +4,7 @@
device,
dma::Coherent,
io::poll::read_poll_timeout,
+ io::Io,
pci,
prelude::*,
time::Delta, //
@@ -86,7 +87,9 @@ fn run_fwsec_frts(
}
// SCRATCH_E contains the error code for FWSEC-FRTS.
- let frts_status = regs::NV_PBUS_SW_SCRATCH_0E_FRTS_ERR::read(bar).frts_err_code();
+ let frts_status = bar
+ .read(regs::NV_PBUS_SW_SCRATCH_0E_FRTS_ERR)
+ .frts_err_code();
if frts_status != 0 {
dev_err!(
dev,
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 58fb807605dd..533d912659ba 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -109,12 +109,14 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
// PBUS
-register!(NV_PBUS_SW_SCRATCH @ 0x00001400[64] {});
+io::register! {
+ pub(crate) NV_PBUS_SW_SCRATCH(u32)[64] @ 0x00001400 {}
-register!(NV_PBUS_SW_SCRATCH_0E_FRTS_ERR => NV_PBUS_SW_SCRATCH[0xe],
- "scratch register 0xe used as FRTS firmware error code" {
- 31:16 frts_err_code as u16;
-});
+ /// Scratch register 0xe used as FRTS firmware error code.
+ pub(crate) NV_PBUS_SW_SCRATCH_0E_FRTS_ERR(u32) => NV_PBUS_SW_SCRATCH[0xe] {
+ 31:16 frts_err_code;
+ }
+}
// PFB
--
2.53.0
^ permalink raw reply related
* [PATCH v4 04/10] gpu: nova-core: convert PFB registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all PFB registers to use the kernel's register macro and update
the code accordingly.
NV_PGSP_QUEUE_HEAD was somehow caught in the PFB section, so move it to
its own section and convert it as well.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/fb/hal/ga100.rs | 34 +++++++++++++--------
drivers/gpu/nova-core/fb/hal/tu102.rs | 14 +++++----
drivers/gpu/nova-core/gsp/boot.rs | 6 ++--
drivers/gpu/nova-core/gsp/cmdq.rs | 9 +++---
drivers/gpu/nova-core/regs.rs | 57 ++++++++++++++++++++---------------
5 files changed, 70 insertions(+), 50 deletions(-)
diff --git a/drivers/gpu/nova-core/fb/hal/ga100.rs b/drivers/gpu/nova-core/fb/hal/ga100.rs
index e0acc41aa7cd..629588c75778 100644
--- a/drivers/gpu/nova-core/fb/hal/ga100.rs
+++ b/drivers/gpu/nova-core/fb/hal/ga100.rs
@@ -1,6 +1,10 @@
// SPDX-License-Identifier: GPL-2.0
-use kernel::prelude::*;
+use kernel::{
+ io::Io,
+ num::Bounded,
+ prelude::*, //
+};
use crate::{
driver::Bar0,
@@ -13,22 +17,26 @@
struct Ga100;
pub(super) fn read_sysmem_flush_page_ga100(bar: &Bar0) -> u64 {
- u64::from(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::read(bar).adr_39_08()) << FLUSH_SYSMEM_ADDR_SHIFT
- | u64::from(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI::read(bar).adr_63_40())
+ u64::from(bar.read(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR).adr_39_08()) << FLUSH_SYSMEM_ADDR_SHIFT
+ | u64::from(bar.read(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI).adr_63_40())
<< FLUSH_SYSMEM_ADDR_SHIFT_HI
}
pub(super) fn write_sysmem_flush_page_ga100(bar: &Bar0, addr: u64) {
- regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI::default()
- // CAST: `as u32` is used on purpose since the remaining bits are guaranteed to fit within
- // a `u32`.
- .set_adr_63_40((addr >> FLUSH_SYSMEM_ADDR_SHIFT_HI) as u32)
- .write(bar);
- regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::default()
- // CAST: `as u32` is used on purpose since we want to strip the upper bits that have been
- // written to `NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI`.
- .set_adr_39_08((addr >> FLUSH_SYSMEM_ADDR_SHIFT) as u32)
- .write(bar);
+ bar.write_reg(
+ regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI::zeroed().with_adr_63_40(
+ Bounded::<u64, _>::from(addr)
+ .shr::<FLUSH_SYSMEM_ADDR_SHIFT_HI, _>()
+ .cast(),
+ ),
+ );
+
+ bar.write_reg(
+ regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::zeroed()
+ // CAST: `as u32` is used on purpose since we want to strip the upper bits that have
+ // been written to `NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI`.
+ .with_adr_39_08((addr >> FLUSH_SYSMEM_ADDR_SHIFT) as u32),
+ );
}
pub(super) fn display_enabled_ga100(bar: &Bar0) -> bool {
diff --git a/drivers/gpu/nova-core/fb/hal/tu102.rs b/drivers/gpu/nova-core/fb/hal/tu102.rs
index eec984f4e816..515d50872224 100644
--- a/drivers/gpu/nova-core/fb/hal/tu102.rs
+++ b/drivers/gpu/nova-core/fb/hal/tu102.rs
@@ -1,6 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
-use kernel::prelude::*;
+use kernel::{
+ io::Io,
+ prelude::*, //
+};
use crate::{
driver::Bar0,
@@ -13,7 +16,7 @@
pub(super) const FLUSH_SYSMEM_ADDR_SHIFT: u32 = 8;
pub(super) fn read_sysmem_flush_page_gm107(bar: &Bar0) -> u64 {
- u64::from(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::read(bar).adr_39_08()) << FLUSH_SYSMEM_ADDR_SHIFT
+ u64::from(bar.read(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR).adr_39_08()) << FLUSH_SYSMEM_ADDR_SHIFT
}
pub(super) fn write_sysmem_flush_page_gm107(bar: &Bar0, addr: u64) -> Result {
@@ -21,9 +24,7 @@ pub(super) fn write_sysmem_flush_page_gm107(bar: &Bar0, addr: u64) -> Result {
u32::try_from(addr >> FLUSH_SYSMEM_ADDR_SHIFT)
.map_err(|_| EINVAL)
.map(|addr| {
- regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::default()
- .set_adr_39_08(addr)
- .write(bar)
+ bar.write_reg(regs::NV_PFB_NISO_FLUSH_SYSMEM_ADDR::zeroed().with_adr_39_08(addr))
})
}
@@ -32,7 +33,8 @@ pub(super) fn display_enabled_gm107(bar: &Bar0) -> bool {
}
pub(super) fn vidmem_size_gp102(bar: &Bar0) -> u64 {
- regs::NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE::read(bar).usable_fb_size()
+ bar.read(regs::NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE)
+ .usable_fb_size()
}
struct Tu102;
diff --git a/drivers/gpu/nova-core/gsp/boot.rs b/drivers/gpu/nova-core/gsp/boot.rs
index 3a0124818956..6f707b3d1a54 100644
--- a/drivers/gpu/nova-core/gsp/boot.rs
+++ b/drivers/gpu/nova-core/gsp/boot.rs
@@ -57,7 +57,7 @@ fn run_fwsec_frts(
) -> Result<()> {
// Check that the WPR2 region does not already exists - if it does, we cannot run
// FWSEC-FRTS until the GPU is reset.
- if regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI::read(bar).higher_bound() != 0 {
+ if bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound() != 0 {
dev_err!(
dev,
"WPR2 region already exists - GPU needs to be reset to proceed\n"
@@ -102,8 +102,8 @@ fn run_fwsec_frts(
// Check that the WPR2 region has been created as we requested.
let (wpr2_lo, wpr2_hi) = (
- regs::NV_PFB_PRI_MMU_WPR2_ADDR_LO::read(bar).lower_bound(),
- regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI::read(bar).higher_bound(),
+ bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_LO).lower_bound(),
+ bar.read(regs::NV_PFB_PRI_MMU_WPR2_ADDR_HI).higher_bound(),
);
match (wpr2_lo, wpr2_hi) {
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index c853be23e3a5..2224896ccc89 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -11,7 +11,10 @@
DmaAddress, //
},
dma_write,
- io::poll::read_poll_timeout,
+ io::{
+ poll::read_poll_timeout,
+ Io, //
+ },
new_mutex,
prelude::*,
sync::{
@@ -509,9 +512,7 @@ fn calculate_checksum<T: Iterator<Item = u8>>(it: T) -> u32 {
/// Notifies the GSP that we have updated the command queue pointers.
fn notify_gsp(bar: &Bar0) {
- regs::NV_PGSP_QUEUE_HEAD::default()
- .set_address(0)
- .write(bar);
+ bar.write_reg(regs::NV_PGSP_QUEUE_HEAD::zeroed().with_address(0u32));
}
/// Sends `command` to the GSP and waits for the reply.
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 533d912659ba..4f5cd64c2fce 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -120,26 +120,35 @@ fn fmt(&self, f: &mut kernel::fmt::Formatter<'_>) -> kernel::fmt::Result {
// PFB
-// The following two registers together hold the physical system memory address that is used by the
-// GPU to perform sysmembar operations (see `fb::SysmemFlush`).
+io::register! {
+ /// Low bits of the physical system memory address used by the GPU to perform sysmembar
+ /// operations (see [`crate::fb::SysmemFlush`]).
+ pub(crate) NV_PFB_NISO_FLUSH_SYSMEM_ADDR(u32) @ 0x00100c10 {
+ 31:0 adr_39_08;
+ }
-register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR @ 0x00100c10 {
- 31:0 adr_39_08 as u32;
-});
+ /// High bits of the physical system memory address used by the GPU to perform sysmembar
+ /// operations (see [`crate::fb::SysmemFlush`]).
+ pub(crate) NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI(u32) @ 0x00100c40 {
+ 23:0 adr_63_40;
+ }
-register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR_HI @ 0x00100c40 {
- 23:0 adr_63_40 as u32;
-});
+ pub(crate) NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE(u32) @ 0x00100ce0 {
+ 30:30 ecc_mode_enabled => bool;
+ 9:4 lower_mag;
+ 3:0 lower_scale;
+ }
-register!(NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE @ 0x00100ce0 {
- 3:0 lower_scale as u8;
- 9:4 lower_mag as u8;
- 30:30 ecc_mode_enabled as bool;
-});
+ pub(crate) NV_PFB_PRI_MMU_WPR2_ADDR_LO(u32) @ 0x001fa824 {
+ /// Bits 12..40 of the lower (inclusive) bound of the WPR2 region.
+ 31:4 lo_val;
+ }
-register!(NV_PGSP_QUEUE_HEAD @ 0x00110c00 {
- 31:0 address as u32;
-});
+ pub(crate) NV_PFB_PRI_MMU_WPR2_ADDR_HI(u32) @ 0x001fa828 {
+ /// Bits 12..40 of the higher (exclusive) bound of the WPR2 region.
+ 31:4 hi_val;
+ }
+}
impl NV_PFB_PRI_MMU_LOCAL_MEMORY_RANGE {
/// Returns the usable framebuffer size, in bytes.
@@ -156,10 +165,6 @@ pub(crate) fn usable_fb_size(self) -> u64 {
}
}
-register!(NV_PFB_PRI_MMU_WPR2_ADDR_LO@0x001fa824 {
- 31:4 lo_val as u32, "Bits 12..40 of the lower (inclusive) bound of the WPR2 region";
-});
-
impl NV_PFB_PRI_MMU_WPR2_ADDR_LO {
/// Returns the lower (inclusive) bound of the WPR2 region.
pub(crate) fn lower_bound(self) -> u64 {
@@ -167,10 +172,6 @@ pub(crate) fn lower_bound(self) -> u64 {
}
}
-register!(NV_PFB_PRI_MMU_WPR2_ADDR_HI@0x001fa828 {
- 31:4 hi_val as u32, "Bits 12..40 of the higher (exclusive) bound of the WPR2 region";
-});
-
impl NV_PFB_PRI_MMU_WPR2_ADDR_HI {
/// Returns the higher (exclusive) bound of the WPR2 region.
///
@@ -180,6 +181,14 @@ pub(crate) fn higher_bound(self) -> u64 {
}
}
+// PGSP
+
+io::register! {
+ pub(crate) NV_PGSP_QUEUE_HEAD(u32) @ 0x00110c00 {
+ 31:0 address;
+ }
+}
+
// PGC6 register space.
//
// `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except
--
2.53.0
^ permalink raw reply related
* [PATCH v4 05/10] gpu: nova-core: convert GC6 registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all GC6 registers to use the kernel's register macro and update
the code accordingly.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/falcon/gsp.rs | 7 ++--
drivers/gpu/nova-core/fb/hal/ga102.rs | 7 ++--
drivers/gpu/nova-core/gfw.rs | 11 ++++--
drivers/gpu/nova-core/regs.rs | 67 ++++++++++++++++++-----------------
4 files changed, 52 insertions(+), 40 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon/gsp.rs b/drivers/gpu/nova-core/falcon/gsp.rs
index 67edef3636c1..e52f57abc223 100644
--- a/drivers/gpu/nova-core/falcon/gsp.rs
+++ b/drivers/gpu/nova-core/falcon/gsp.rs
@@ -1,7 +1,10 @@
// SPDX-License-Identifier: GPL-2.0
use kernel::{
- io::poll::read_poll_timeout,
+ io::{
+ poll::read_poll_timeout,
+ Io, //
+ },
prelude::*,
time::Delta, //
};
@@ -47,7 +50,7 @@ pub(crate) fn clear_swgen0_intr(&self, bar: &Bar0) {
/// Checks if GSP reload/resume has completed during the boot process.
pub(crate) fn check_reload_completed(&self, bar: &Bar0, timeout: Delta) -> Result<bool> {
read_poll_timeout(
- || Ok(regs::NV_PGC6_BSI_SECURE_SCRATCH_14::read(bar)),
+ || Ok(bar.read(regs::NV_PGC6_BSI_SECURE_SCRATCH_14)),
|val| val.boot_stage_3_handoff(),
Delta::ZERO,
timeout,
diff --git a/drivers/gpu/nova-core/fb/hal/ga102.rs b/drivers/gpu/nova-core/fb/hal/ga102.rs
index 734605905031..4b9f0f74d0e7 100644
--- a/drivers/gpu/nova-core/fb/hal/ga102.rs
+++ b/drivers/gpu/nova-core/fb/hal/ga102.rs
@@ -1,6 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
-use kernel::prelude::*;
+use kernel::{
+ io::Io,
+ prelude::*, //
+};
use crate::{
driver::Bar0,
@@ -9,7 +12,7 @@
};
fn vidmem_size_ga102(bar: &Bar0) -> u64 {
- regs::NV_USABLE_FB_SIZE_IN_MB::read(bar).usable_fb_size()
+ bar.read(regs::NV_USABLE_FB_SIZE_IN_MB).usable_fb_size()
}
struct Ga102;
diff --git a/drivers/gpu/nova-core/gfw.rs b/drivers/gpu/nova-core/gfw.rs
index 9121f400046d..fb75dd10a172 100644
--- a/drivers/gpu/nova-core/gfw.rs
+++ b/drivers/gpu/nova-core/gfw.rs
@@ -19,7 +19,10 @@
//! Note that the devinit sequence also needs to run during suspend/resume.
use kernel::{
- io::poll::read_poll_timeout,
+ io::{
+ poll::read_poll_timeout,
+ Io, //
+ },
prelude::*,
time::Delta, //
};
@@ -58,9 +61,11 @@ pub(crate) fn wait_gfw_boot_completion(bar: &Bar0) -> Result {
Ok(
// Check that FWSEC has lowered its protection level before reading the GFW_BOOT
// status.
- regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK::read(bar)
+ bar.read(regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK)
.read_protection_level0()
- && regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT::read(bar).completed(),
+ && bar
+ .read(regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT)
+ .completed(),
)
},
|&gfw_booted| gfw_booted,
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 4f5cd64c2fce..6f49467e78ec 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -198,29 +198,41 @@ pub(crate) fn higher_bound(self) -> u64 {
// These scratch registers remain powered on even in a low-power state and have a designated group
// number.
-// Boot Sequence Interface (BSI) register used to determine
-// if GSP reload/resume has completed during the boot process.
-register!(NV_PGC6_BSI_SECURE_SCRATCH_14 @ 0x001180f8 {
- 26:26 boot_stage_3_handoff as bool;
-});
-
-// Privilege level mask register. It dictates whether the host CPU has privilege to access the
-// `PGC6_AON_SECURE_SCRATCH_GROUP_05` register (which it needs to read GFW_BOOT).
-register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128,
- "Privilege level mask register" {
- 0:0 read_protection_level0 as bool, "Set after FWSEC lowers its protection level";
-});
-
-// OpenRM defines this as a register array, but doesn't specify its size and only uses its first
-// element. Be conservative until we know the actual size or need to use more registers.
-register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05 @ 0x00118234[1] {});
-
-register!(
- NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT => NV_PGC6_AON_SECURE_SCRATCH_GROUP_05[0],
- "Scratch group 05 register 0 used as GFW boot progress indicator" {
- 7:0 progress as u8, "Progress of GFW boot (0xff means completed)";
+io::register! {
+ /// Boot Sequence Interface (BSI) register used to determine
+ /// if GSP reload/resume has completed during the boot process.
+ pub(crate) NV_PGC6_BSI_SECURE_SCRATCH_14(u32) @ 0x001180f8 {
+ 26:26 boot_stage_3_handoff => bool;
}
-);
+
+ /// Privilege level mask register. It dictates whether the host CPU has privilege to access the
+ /// `PGC6_AON_SECURE_SCRATCH_GROUP_05` register (which it needs to read GFW_BOOT).
+ pub(crate) NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK(u32) @ 0x00118128 {
+ /// Set after FWSEC lowers its protection level.
+ 0:0 read_protection_level0 => bool;
+ }
+
+ /// OpenRM defines this as a register array, but doesn't specify its size and only uses its
+ /// first element. Be conservative until we know the actual size or need to use more registers.
+ pub(crate) NV_PGC6_AON_SECURE_SCRATCH_GROUP_05(u32)[1] @ 0x00118234 {}
+
+ /// Scratch group 05 register 0 used as GFW boot progress indicator.
+ pub(crate) NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT(u32)
+ => NV_PGC6_AON_SECURE_SCRATCH_GROUP_05[0] {
+ /// Progress of GFW boot (0xff means completed).
+ 7:0 progress;
+ }
+
+ pub(crate) NV_PGC6_AON_SECURE_SCRATCH_GROUP_42(u32) @ 0x001183a4 {
+ 31:0 value;
+ }
+
+ /// Scratch group 42 register used as framebuffer size.
+ pub(crate) NV_USABLE_FB_SIZE_IN_MB(u32) => NV_PGC6_AON_SECURE_SCRATCH_GROUP_42 {
+ /// Usable framebuffer size, in megabytes.
+ 31:0 value;
+ }
+}
impl NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT {
/// Returns `true` if GFW boot is completed.
@@ -229,17 +241,6 @@ pub(crate) fn completed(self) -> bool {
}
}
-register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_42 @ 0x001183a4 {
- 31:0 value as u32;
-});
-
-register!(
- NV_USABLE_FB_SIZE_IN_MB => NV_PGC6_AON_SECURE_SCRATCH_GROUP_42,
- "Scratch group 42 register used as framebuffer size" {
- 31:0 value as u32, "Usable framebuffer size, in megabytes";
- }
-);
-
impl NV_USABLE_FB_SIZE_IN_MB {
/// Returns the usable framebuffer size, in bytes.
pub(crate) fn usable_fb_size(self) -> u64 {
--
2.53.0
^ permalink raw reply related
* [PATCH v4 06/10] gpu: nova-core: convert FUSE registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all FUSE registers to use the kernel's register macro and update
the code accordingly.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/falcon/hal/ga102.rs | 17 +++++++++----
drivers/gpu/nova-core/fb/hal/ga100.rs | 3 ++-
drivers/gpu/nova-core/fb/hal/tu102.rs | 3 ++-
drivers/gpu/nova-core/regs.rs | 40 +++++++++++++++++++------------
4 files changed, 41 insertions(+), 22 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon/hal/ga102.rs b/drivers/gpu/nova-core/falcon/hal/ga102.rs
index 8f62df10da0a..cbdf36bad633 100644
--- a/drivers/gpu/nova-core/falcon/hal/ga102.rs
+++ b/drivers/gpu/nova-core/falcon/hal/ga102.rs
@@ -4,7 +4,11 @@
use kernel::{
device,
- io::poll::read_poll_timeout,
+ io::{
+ poll::read_poll_timeout,
+ register::Array,
+ Io, //
+ },
prelude::*,
time::Delta, //
};
@@ -60,12 +64,15 @@ fn signature_reg_fuse_version_ga102(
// `ucode_idx` is guaranteed to be in the range [0..15], making the `read` calls provable valid
// at build-time.
- let reg_fuse_version = if engine_id_mask & 0x0001 != 0 {
- regs::NV_FUSE_OPT_FPF_SEC2_UCODE1_VERSION::read(bar, ucode_idx).data()
+ let reg_fuse_version: u16 = if engine_id_mask & 0x0001 != 0 {
+ bar.read(regs::NV_FUSE_OPT_FPF_SEC2_UCODE1_VERSION::at(ucode_idx))
+ .data()
} else if engine_id_mask & 0x0004 != 0 {
- regs::NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION::read(bar, ucode_idx).data()
+ bar.read(regs::NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION::at(ucode_idx))
+ .data()
} else if engine_id_mask & 0x0400 != 0 {
- regs::NV_FUSE_OPT_FPF_GSP_UCODE1_VERSION::read(bar, ucode_idx).data()
+ bar.read(regs::NV_FUSE_OPT_FPF_GSP_UCODE1_VERSION::at(ucode_idx))
+ .data()
} else {
dev_err!(dev, "unexpected engine_id_mask {:#x}\n", engine_id_mask);
return Err(EINVAL);
diff --git a/drivers/gpu/nova-core/fb/hal/ga100.rs b/drivers/gpu/nova-core/fb/hal/ga100.rs
index 629588c75778..1c03783cddef 100644
--- a/drivers/gpu/nova-core/fb/hal/ga100.rs
+++ b/drivers/gpu/nova-core/fb/hal/ga100.rs
@@ -40,7 +40,8 @@ pub(super) fn write_sysmem_flush_page_ga100(bar: &Bar0, addr: u64) {
}
pub(super) fn display_enabled_ga100(bar: &Bar0) -> bool {
- !regs::ga100::NV_FUSE_STATUS_OPT_DISPLAY::read(bar).display_disabled()
+ !bar.read(regs::ga100::NV_FUSE_STATUS_OPT_DISPLAY)
+ .display_disabled()
}
/// Shift applied to the sysmem address before it is written into
diff --git a/drivers/gpu/nova-core/fb/hal/tu102.rs b/drivers/gpu/nova-core/fb/hal/tu102.rs
index 515d50872224..281bb796e198 100644
--- a/drivers/gpu/nova-core/fb/hal/tu102.rs
+++ b/drivers/gpu/nova-core/fb/hal/tu102.rs
@@ -29,7 +29,8 @@ pub(super) fn write_sysmem_flush_page_gm107(bar: &Bar0, addr: u64) -> Result {
}
pub(super) fn display_enabled_gm107(bar: &Bar0) -> bool {
- !regs::gm107::NV_FUSE_STATUS_OPT_DISPLAY::read(bar).display_disabled()
+ !bar.read(regs::gm107::NV_FUSE_STATUS_OPT_DISPLAY)
+ .display_disabled()
}
pub(super) fn vidmem_size_gp102(bar: &Bar0) -> u64 {
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 6f49467e78ec..61a8dba22d88 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -270,17 +270,19 @@ pub(crate) fn vga_workspace_addr(self) -> Option<u64> {
pub(crate) const NV_FUSE_OPT_FPF_SIZE: usize = 16;
-register!(NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION @ 0x00824100[NV_FUSE_OPT_FPF_SIZE] {
- 15:0 data as u16;
-});
+io::register! {
+ pub(crate) NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION(u32)[NV_FUSE_OPT_FPF_SIZE] @ 0x00824100 {
+ 15:0 data => u16;
+ }
-register!(NV_FUSE_OPT_FPF_SEC2_UCODE1_VERSION @ 0x00824140[NV_FUSE_OPT_FPF_SIZE] {
- 15:0 data as u16;
-});
+ pub(crate) NV_FUSE_OPT_FPF_SEC2_UCODE1_VERSION(u32)[NV_FUSE_OPT_FPF_SIZE] @ 0x00824140 {
+ 15:0 data => u16;
+ }
-register!(NV_FUSE_OPT_FPF_GSP_UCODE1_VERSION @ 0x008241c0[NV_FUSE_OPT_FPF_SIZE] {
- 15:0 data as u16;
-});
+ pub(crate) NV_FUSE_OPT_FPF_GSP_UCODE1_VERSION(u32)[NV_FUSE_OPT_FPF_SIZE] @ 0x008241c0 {
+ 15:0 data => u16;
+ }
+}
// PFALCON
@@ -491,17 +493,25 @@ pub(crate) fn reset_engine<E: FalconEngine>(bar: &Bar0) {
// only be used in HAL modules.
pub(crate) mod gm107 {
+ use kernel::io;
+
// FUSE
- register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00021c04 {
- 0:0 display_disabled as bool;
- });
+ io::register! {
+ pub(crate) NV_FUSE_STATUS_OPT_DISPLAY(u32) @ 0x00021c04 {
+ 0:0 display_disabled => bool;
+ }
+ }
}
pub(crate) mod ga100 {
+ use kernel::io;
+
// FUSE
- register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00820c04 {
- 0:0 display_disabled as bool;
- });
+ io::register! {
+ pub(crate) NV_FUSE_STATUS_OPT_DISPLAY(u32) @ 0x00820c04 {
+ 0:0 display_disabled => bool;
+ }
+ }
}
--
2.53.0
^ permalink raw reply related
* [PATCH v4 07/10] gpu: nova-core: convert PDISP registers to kernel register macro
From: Alexandre Courbot @ 2026-03-25 2:46 UTC (permalink / raw)
To: Danilo Krummrich, Alice Ryhl, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann, Miguel Ojeda,
Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Trevor Gross
Cc: John Hubbard, Alistair Popple, Joel Fernandes, Timur Tabi,
Zhi Wang, Eliot Courtney, dri-devel, linux-kernel, linux-riscv,
linux-doc, rust-for-linux, Alexandre Courbot
In-Reply-To: <20260325-b4-nova-register-v4-0-bdf172f0f6ca@nvidia.com>
Convert all PDISP registers to use the kernel's register macro and
update the code accordingly.
Reviewed-by: Eliot Courtney <ecourtney@nvidia.com>
Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/fb.rs | 6 +++++-
drivers/gpu/nova-core/regs.rs | 12 ++++++++----
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/nova-core/fb.rs b/drivers/gpu/nova-core/fb.rs
index 6536d0035cb1..62fc90fa6a84 100644
--- a/drivers/gpu/nova-core/fb.rs
+++ b/drivers/gpu/nova-core/fb.rs
@@ -8,6 +8,7 @@
use kernel::{
device,
fmt,
+ io::Io,
prelude::*,
ptr::{
Alignable,
@@ -189,7 +190,10 @@ pub(crate) fn new(chipset: Chipset, bar: &Bar0, gsp_fw: &GspFirmware) -> Result<
let base = fb.end - NV_PRAMIN_SIZE;
if hal.supports_display(bar) {
- match regs::NV_PDISP_VGA_WORKSPACE_BASE::read(bar).vga_workspace_addr() {
+ match bar
+ .read(regs::NV_PDISP_VGA_WORKSPACE_BASE)
+ .vga_workspace_addr()
+ {
Some(addr) => {
if addr < base {
const VBIOS_WORKSPACE_SIZE: u64 = usize_as_u64(SZ_128K);
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 61a8dba22d88..b051d5568cd8 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -250,10 +250,14 @@ pub(crate) fn usable_fb_size(self) -> u64 {
// PDISP
-register!(NV_PDISP_VGA_WORKSPACE_BASE @ 0x00625f04 {
- 3:3 status_valid as bool, "Set if the `addr` field is valid";
- 31:8 addr as u32, "VGA workspace base address divided by 0x10000";
-});
+io::register! {
+ pub(crate) NV_PDISP_VGA_WORKSPACE_BASE(u32) @ 0x00625f04 {
+ /// VGA workspace base address divided by 0x10000.
+ 31:8 addr;
+ /// Set if the `addr` field is valid.
+ 3:3 status_valid => bool;
+ }
+}
impl NV_PDISP_VGA_WORKSPACE_BASE {
/// Returns the base address of the VGA workspace, or `None` if none exists.
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox