* [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs [not found] <d85ece6e-27f9-4d9f-83af-4101bc76fef6@app.fastmail.com> @ 2026-11-06 13:38 ` roblabla 2026-11-06 15:41 ` Paul Moore 0 siblings, 1 reply; 11+ messages in thread From: roblabla @ 2026-11-06 13:38 UTC (permalink / raw) To: Daan De Meyer, Paul Moore Cc: bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko Resubmitting as plain text. Sorry for the noise. Hello, First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. Those programs don't simply stop loading (which would be fine enough) - instead, they will misinterpret the argument from one type to another, which can cause them to behave erratically. This has already bitten me in the past, when inode_setattr got a similar argument change in Linux 6.9. This caused our eBPF LSM to misbehave and deny actions that should have been ordinarily accepted, leading to huge production outages on our customers' machines. Here's a simple reproducer that shows the problem (full repro with the loader, makefile and run.sh is at https://github.com/roblabla/kernel-7.4-bpf-bug-reproducer): ``` #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #include <bpf/bpf_core_read.h> char LICENSE[] SEC("license") = "GPL"; #define EPERM 1 #define NAME_LEN 32 static const char forbidden[] = "forbidden"; SEC("lsm/inode_create") int BPF_PROG(deny_forbidden, struct inode *dir, struct dentry *dentry) { char name[NAME_LEN] = {}; const unsigned char *name_ptr; unsigned long dir_ino; dir_ino = BPF_CORE_READ(dir, i_ino); name_ptr = BPF_CORE_READ(dentry, d_name.name); bpf_probe_read_kernel_str(name, sizeof(name), name_ptr); bpf_printk("inode_create: dir ino=%lu name='%s'", dir_ino, name); for (int i = 0; i < sizeof(forbidden); i++) if (name[i] != forbidden[i]) return 0; bpf_printk("inode_create: denied '%s'", name); return -EPERM; } ``` Compile it under a kernel before 7.4 release, then run it on mainline. Here's the output: - run on 7.1-rc7: touch forbidden: DENIED (dir ino=59 name='forbidden') - run on 7.4-rc1: touch forbidden: created (dir ino=0 name='') Before Linux 7.4, this program will properly deny the creation of any file named "forbidden". But in 7.4, the name_ptr BPF_CORE_READ will read garbage, and the check will fail. This is a bit of a contrived example, of course, but should show that there's a real problem in the way we handle eBPF loading in the face of changing signatures, that is prone to causing big compatibility issues. Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, such that eBPF written for an older version of the kernel do not start misinterpreting arguments? eBPFs don't care about having "too few" arguments in the hook, so this would solve the situation. Separately, I think the current lack of error handling in case of argument mismatch should really be improved. Could we perhaps have a mechanism such that the kernel (or maybe libbpf?) could check the arguments of the eBPF program, and refuse loading if it mismatches what the kernel expects? Thanks, roblabla ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-06 13:38 ` [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs roblabla @ 2026-11-06 15:41 ` Paul Moore 2026-11-06 19:33 ` Andrii Nakryiko 0 siblings, 1 reply; 11+ messages in thread From: Paul Moore @ 2026-11-06 15:41 UTC (permalink / raw) To: roblabla Cc: Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Fri, Nov 6, 2026 at 8:39 AM roblabla <linux@roblab.la> wrote: > > Hello, > First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. Hi, It looks like you sent this email to the right people/lists, well done on the first hurdle :) > In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. > > This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. ... > Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. Like kprobes, the LSM hooks are intentionally not part of any stable API promise. While we try not to needlessly churn the LSM hooks, we reserve the right to modify parameters, the LSM callbacks associated with a hook, and even the basic hook semantics from one kernel release to another. The reasoning is very similar to that of kprobes: the LSM hooks are linked to kernel internals and thus need the ability to change as the kernel changes. LSMs that need to provide a stable API must provide an abstraction layer between the LSM hook callbacks and their stable API. The LSM framework is not responsible for the stability of individual LSM APIs. > Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, I'm sorry, but I can't make that promise at the LSM framework layer. It is impossible to predict how the kernel internals will continue to evolve and thus I can't promise how the LSM hooks and callbacks will need to change as a result. Regarding the specifics of how the BPF LSM has dealt with this in the past, and any plans for the future, I'll let others speak to that; hopefully they will have some tricks you can use to make these transitions easier. -- paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-06 15:41 ` Paul Moore @ 2026-11-06 19:33 ` Andrii Nakryiko 2026-11-09 15:07 ` roblabla 0 siblings, 1 reply; 11+ messages in thread From: Andrii Nakryiko @ 2026-11-06 19:33 UTC (permalink / raw) To: Paul Moore, Daan De Meyer Cc: roblabla, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Fri, Nov 6, 2026 at 7:41 AM Paul Moore <paul@paul-moore.com> wrote: > > On Fri, Nov 6, 2026 at 8:39 AM roblabla <linux@roblab.la> wrote: > > > > Hello, > > First of all, I'm sorry if I'm sending this email to the wrong mailing lists, it's my first time reporting a linux issue ^^'. > > Hi, > > It looks like you sent this email to the right people/lists, well done > on the first hurdle :) > > > In commit 16959c469f232f5c76cfb628297cafbf6dbdba79 (lsm: expose mount idmaps to inode hooks), merged in 7.4-rc1, a new argument was added in position 0 of various inode-related `security_` functions. Specifically, it added this argument to inode_create, inode_link, inode_symlink, inode_mkdir, inode_mknod and inode_permission. > > > > This has the immediate effect of breaking eBPF LSM programs relying on these hooks that were compiled with BPF CO-RE and using BPF_CORE_READ. > > ... > > > Now, I'm not sure how the Linux Kernel treats eBPF LSM hooks - if they're considered part of the userspace that shouldn't be broken. My understanding is that kprobes are not part of the Linux Kernel stability guarantees, for rather obvious reasons (they allow watching the internals of the kernel) - but I'm not sure if this extends to the LSM Hooks. I assume not. Still, this kind of breakage is really painful for us, and I hope we can find a solution. > > Like kprobes, the LSM hooks are intentionally not part of any stable > API promise. While we try not to needlessly churn the LSM hooks, we > reserve the right to modify parameters, the LSM callbacks associated > with a hook, and even the basic hook semantics from one kernel release > to another. The reasoning is very similar to that of kprobes: the LSM > hooks are linked to kernel internals and thus need the ability to > change as the kernel changes. > > LSMs that need to provide a stable API must provide an abstraction > layer between the LSM hook callbacks and their stable API. The LSM > framework is not responsible for the stability of individual LSM APIs. > > > Could we fix this type of breakage by making sure new arguments are only added at the _end_ of the LSM hooks signatures, > > I'm sorry, but I can't make that promise at the LSM framework layer. > It is impossible to predict how the kernel internals will continue to > evolve and thus I can't promise how the LSM hooks and callbacks will > need to change as a result. Yeah, I think that's a reasonable stance and we went through similar discussion about tracepoints not being stable APIs, no questions about that. But I'm wondering if it's not too late (as far as I understand, this hasn't gotten into final release just yet), would it be ok to just move newly added argument to be the last ones? We had similar problem in not too distant past with one of scheduler tracepoints, where new argument was added in the middle of argument list, and that causes just unnecessary friction when using BPF with old and new kernels. We did move the argument to the end and resolved this (without really mandating any stability guarantees). So, in short, would it be possible to send a follow up to Daan's commit and move `struct mnt_idmap *idmap` to the end of LSM argument list? > > Regarding the specifics of how the BPF LSM has dealt with this in the > past, and any plans for the future, I'll let others speak to that; > hopefully they will have some tricks you can use to make these > transitions easier. > (this is answer to roblabla, I don't think you, Paul, should care) The answer here is nuanced. On one hand, libbpf and kernel (BPF verifier) don't really know exactly what BPF program assumes argument types are. libbpf-provided BPF_PROG() macro that everyone is using with BPF LSM just assumes user specifies correct sequence and types of arguments, and libbpf just provides a bit of syntactic sugar with casting untyped input arguments (they are just "u64" types) into expected types. So it's not easy to detect this automatically neither in libbpf nor in BPF verifier. Having said that, there is a way to handle this, although ugly and *everyone* using specific LSMs would need to do this on their own. Instead of using convenience BPF_PROG() macro to define BPF program, you just use raw `u64 ctx[]` access, and then cast ctx[0] to either `struct mnt_idmap *` on new kernel, or `struct dentry *` on old kernel. And you do that for all arguments. E.g., something like this SEC("...") int my_prog(u64 ctx[]) { bool new_kernel = ...; struct mnt_idmap *idmap = new_kernel ? bpf_core_cast(ctx[0], struct mnt_idmap) : NULL; struct dentry *old_dentry = bpf_core_cast(new_kernel ? ctx[1] : ctx[0]); ... and so on ... } How you determine "new_kernel" can be done in various ways as well, I don't want to overburden this email with details, but it's doable as well. so, tl;dr, it's possible to handle this LSM signature change, but it's painful and requires knowing about breakage and doing a bunch of extra plumbing. ideally, if it's just a new argument addition, it should go to the end of arg list and then all these issues go away > -- > paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-06 19:33 ` Andrii Nakryiko @ 2026-11-09 15:07 ` roblabla 2026-11-12 23:00 ` Andrii Nakryiko 2026-11-18 0:52 ` Ihor Solodrai 0 siblings, 2 replies; 11+ messages in thread From: roblabla @ 2026-11-09 15:07 UTC (permalink / raw) To: Andrii Nakryiko, Paul Moore, Daan De Meyer Cc: bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko Hi, Thanks for the replies! > LSMs that need to provide a stable API must provide an abstraction > layer between the LSM hook callbacks and their stable API. The LSM > framework is not responsible for the stability of individual LSM APIs. Understood, that lines up with my expectations. I think the biggest thing here would not necessarily be stability guarantees, but rather to make sure that the eBPFs fail to load when the compatibility is broken. But that's probably a problem for the eBPF side, and not LSM, I assume. > So, in short, would it be possible to send a follow up to Daan's > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > list? If this is possible, that'd be nice so that we avoid this breakage in the short term, since the change isn't in a released kernel yet. > Having said that, there is a way to handle this, although ugly and > *everyone* using specific LSMs would need to do this on their own. > Instead of using convenience BPF_PROG() macro to define BPF program, > you just use raw `u64 ctx[]` access, and then cast ctx[0] to either > `struct mnt_idmap *` on new kernel, or `struct dentry *` on old > kernel. So, we already do similar tricks in the software we ship (which uses various eBPF program types, not just LSM) to cover kernels 4.11 to 7.3, using a mixture of CO-RE tricks similar to what you describe and per-kernel builds selected at runtime. Here's the problem though: We ship to customers, and we don't control when they upgrade their kernel relative to our agent. When the kernel changes a hook signature, a program built before we could handle that new version still loads fine and just reads the wrong arguments, which causes some really gnarly bugs that can cause really bad breakage (such as accidentally denying events that should have been allowed, often resulting in machines left broken). So what would be really nice is a way to make eBPF code robust against kernel changes such that it either works, or fails to load, but doesn't do the halfway "loads but silently misbehaves". Here are two ideas that have been in the back of my mind that could help solve this issue (though I understand that they may be rather large changes, I'm putting them out there mostly as a conversation starter): 1. Could libbpf/the kernel have a mismatch check on hook arguments? I understand that libbpf and the verifier don't know the types a program expects, since BPF_PROG() only casts u64s. But the macro does know the types at compile time. Could it (or an opt-in variant) record the expected prototype in the program's BTF, so that libbpf can compare it against the LSM hook's prototype in vmlinux BTF and refuse to load on a mismatch? 2. Since kernel BTF has parameter names for the bpf_lsm_* functions, could we maybe have CO-RE-style relocation to resolve "the argument named dentry" to the argument index at load time? This way, adding a new argument could happen anywhere in the argument list without breaking eBPFs, and removing an argument will (correctly) cause our eBPF to stop loading. It does mean the eBPF programs must provide the same argument names, but maybe that's an acceptable tradeoff? Thanks, roblabla ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-09 15:07 ` roblabla @ 2026-11-12 23:00 ` Andrii Nakryiko 2026-11-13 2:41 ` Paul Moore 2026-11-18 0:52 ` Ihor Solodrai 1 sibling, 1 reply; 11+ messages in thread From: Andrii Nakryiko @ 2026-11-12 23:00 UTC (permalink / raw) To: roblabla Cc: Paul Moore, Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > Hi, > > Thanks for the replies! > > > LSMs that need to provide a stable API must provide an abstraction > > layer between the LSM hook callbacks and their stable API. The LSM > > framework is not responsible for the stability of individual LSM APIs. > > Understood, that lines up with my expectations. I think the biggest > thing here would not necessarily be stability guarantees, but rather to > make sure that the eBPFs fail to load when the compatibility is broken. > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > So, in short, would it be possible to send a follow up to Daan's > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > list? > > If this is possible, that'd be nice so that we avoid this breakage in > the short term, since the change isn't in a released kernel yet. can you please send a patch proposing that and we can continue discussion there? please cc me so I can follow along > > > Having said that, there is a way to handle this, although ugly and > > *everyone* using specific LSMs would need to do this on their own. > > Instead of using convenience BPF_PROG() macro to define BPF program, > > you just use raw `u64 ctx[]` access, and then cast ctx[0] to either > > `struct mnt_idmap *` on new kernel, or `struct dentry *` on old > > kernel. > > So, we already do similar tricks in the software we ship (which uses > various eBPF program types, not just LSM) to cover kernels 4.11 to 7.3, > using a mixture of CO-RE tricks similar to what you describe and > per-kernel builds selected at runtime. > > Here's the problem though: We ship to customers, and we don't control > when they upgrade their kernel relative to our agent. When the kernel > changes a hook signature, a program built before we could handle that > new version still loads fine and just reads the wrong arguments, which > causes some really gnarly bugs that can cause really bad breakage (such > as accidentally denying events that should have been allowed, often > resulting in machines left broken). > > So what would be really nice is a way to make eBPF code robust against > kernel changes such that it either works, or fails to load, but doesn't > do the halfway "loads but silently misbehaves". Here are two ideas that > have been in the back of my mind that could help solve this issue (though > I understand that they may be rather large changes, I'm putting them out > there mostly as a conversation starter): > > 1. Could libbpf/the kernel have a mismatch check on hook arguments? I > understand that libbpf and the verifier don't know the types a > program expects, since BPF_PROG() only casts u64s. But the macro > does know the types at compile time. Could it (or an opt-in variant) > record the expected prototype in the program's BTF, so that libbpf > can compare it against the LSM hook's prototype in vmlinux BTF and > refuse to load on a mismatch? > > 2. Since kernel BTF has parameter names for the bpf_lsm_* functions, > could we maybe have CO-RE-style relocation to resolve "the argument > named dentry" to the argument index at load time? This way, adding a > new argument could happen anywhere in the argument list without > breaking eBPFs, and removing an argument will (correctly) cause our > eBPF to stop loading. It does mean the eBPF programs must provide the > same argument names, but maybe that's an acceptable tradeoff? > All this is not really an LSM-specific problem, it's a more general tracing problem. And even if we have some custom hacky "solution" for LSM specifically (but also see my previous replies, it's not something that's easy to do either in libbpf nor kernel, IMO), the same problem is still present for tracepoints, among other use cases. It basically falls on BPF users to keep track of kernel evolution in terms of attach points they are depending on. Luckily breakage doesn't happen often and usually can be worked around or be adopted to. And so far kernel community has been pretty understanding in terms of evolution of some pre-existing hooks (like scheduler tracepoint I mentioned before, and hopefully in this case with new argument moved to the end of argument list). > Thanks, > roblabla ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-12 23:00 ` Andrii Nakryiko @ 2026-11-13 2:41 ` Paul Moore 2026-11-17 16:51 ` Andrii Nakryiko 0 siblings, 1 reply; 11+ messages in thread From: Paul Moore @ 2026-11-13 2:41 UTC (permalink / raw) To: Andrii Nakryiko, roblabla Cc: Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On Thu, Nov 12, 2026 at 6:00 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > > > Hi, > > > > Thanks for the replies! > > > > > LSMs that need to provide a stable API must provide an abstraction > > > layer between the LSM hook callbacks and their stable API. The LSM > > > framework is not responsible for the stability of individual LSM APIs. > > > > Understood, that lines up with my expectations. I think the biggest > > thing here would not necessarily be stability guarantees, but rather to > > make sure that the eBPFs fail to load when the compatibility is broken. > > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > > > So, in short, would it be possible to send a follow up to Daan's > > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > > list? > > > > If this is possible, that'd be nice so that we avoid this breakage in > > the short term, since the change isn't in a released kernel yet. > > can you please send a patch proposing that and we can continue > discussion there? please cc me so I can follow along Considering the original patch was submitted, reviewed, and merged with the BPF LSM being the only user (none of the existing in-tree LSMs use the idmap parameter at present), I'm a little surprised this wasn't brought up earlier. Beyond that, from a style perspective, I think I prefer the idmap at the front of the parameter list in most (all?) of the hooks where it was added. I'm not interested in moving it at this point in time. -- paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-13 2:41 ` Paul Moore @ 2026-11-17 16:51 ` Andrii Nakryiko 2026-11-17 19:30 ` Paul Moore 0 siblings, 1 reply; 11+ messages in thread From: Andrii Nakryiko @ 2026-11-17 16:51 UTC (permalink / raw) To: Paul Moore Cc: roblabla, Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko, Christian Brauner On Thu, Nov 12, 2026 at 6:41 PM Paul Moore <paul@paul-moore.com> wrote: > > On Thu, Nov 12, 2026 at 6:00 PM Andrii Nakryiko > <andrii.nakryiko@gmail.com> wrote: > > On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > > > > > Hi, > > > > > > Thanks for the replies! > > > > > > > LSMs that need to provide a stable API must provide an abstraction > > > > layer between the LSM hook callbacks and their stable API. The LSM > > > > framework is not responsible for the stability of individual LSM APIs. > > > > > > Understood, that lines up with my expectations. I think the biggest > > > thing here would not necessarily be stability guarantees, but rather to > > > make sure that the eBPFs fail to load when the compatibility is broken. > > > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > > > > > So, in short, would it be possible to send a follow up to Daan's > > > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > > > list? > > > > > > If this is possible, that'd be nice so that we avoid this breakage in > > > the short term, since the change isn't in a released kernel yet. > > > > can you please send a patch proposing that and we can continue > > discussion there? please cc me so I can follow along > > Considering the original patch was submitted, reviewed, and merged > with the BPF LSM being the only user (none of the existing in-tree > LSMs use the idmap parameter at present), I'm a little surprised this > wasn't brought up earlier. Yeah, unfortunately this was missed. Not sure it's surprising that it wasn't detected immediately, though, because unlikely someone tested an unreleased kernel in production. Luckily, this was still detected early enough before the final release and this gives us a chance to fix a mistake. > > Beyond that, from a style perspective, I think I prefer the idmap at > the front of the parameter list in most (all?) of the hooks where it > was added. I'm not interested in moving it at this point in time. So on one hand we have a real user pain which is easily fixable without burdening kernel and LSM subsystem in particular, right? On the other hand there is a minor issue of argument ordering with mnt_idmap preferably as a first argument. I get the preference, but I think fixing an unnecessary (but easily fixable!) user pain should come first, don't you agree? Can you please reconsider? > > -- > paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-17 16:51 ` Andrii Nakryiko @ 2026-11-17 19:30 ` Paul Moore 2026-11-17 21:44 ` Andrii Nakryiko 0 siblings, 1 reply; 11+ messages in thread From: Paul Moore @ 2026-11-17 19:30 UTC (permalink / raw) To: Andrii Nakryiko Cc: roblabla, Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko, Christian Brauner On Tue, Nov 17, 2026 at 11:51 AM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > On Thu, Nov 12, 2026 at 6:41 PM Paul Moore <paul@paul-moore.com> wrote: > > On Thu, Nov 12, 2026 at 6:00 PM Andrii Nakryiko > > <andrii.nakryiko@gmail.com> wrote: > > > On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > > > > > > > Hi, > > > > > > > > Thanks for the replies! > > > > > > > > > LSMs that need to provide a stable API must provide an abstraction > > > > > layer between the LSM hook callbacks and their stable API. The LSM > > > > > framework is not responsible for the stability of individual LSM APIs. > > > > > > > > Understood, that lines up with my expectations. I think the biggest > > > > thing here would not necessarily be stability guarantees, but rather to > > > > make sure that the eBPFs fail to load when the compatibility is broken. > > > > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > > > > > > > So, in short, would it be possible to send a follow up to Daan's > > > > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > > > > list? > > > > > > > > If this is possible, that'd be nice so that we avoid this breakage in > > > > the short term, since the change isn't in a released kernel yet. > > > > > > can you please send a patch proposing that and we can continue > > > discussion there? please cc me so I can follow along > > > > Considering the original patch was submitted, reviewed, and merged > > with the BPF LSM being the only user (none of the existing in-tree > > LSMs use the idmap parameter at present), I'm a little surprised this > > wasn't brought up earlier. > > Yeah, unfortunately this was missed. Not sure it's surprising that it > wasn't detected immediately, though, because unlikely someone tested > an unreleased kernel in production. Luckily, this was still detected > early enough before the final release and this gives us a chance to > fix a mistake. I've been surprised lately by apparently how little the BPF LSM program interface is tested with real BPF programs before landing in Linus' tree. I know the BPF folks have a pretty good set of CI tests, bots, etc. but there do seem to be gaps relating to real world usage. > > Beyond that, from a style perspective, I think I prefer the idmap at > > the front of the parameter list in most (all?) of the hooks where it > > was added. I'm not interested in moving it at this point in time. > > So on one hand we have a real user pain which is easily fixable > without burdening kernel and LSM subsystem in particular, right? Arguably changes that negatively impact code style and readability do affect developers and maintainers. > On > the other hand there is a minor issue of argument ordering with > mnt_idmap preferably as a first argument. I get the preference, but I > think fixing an unnecessary (but easily fixable!) user pain should > come first, don't you agree? > > Can you please reconsider? The user pain exists because there is no abstraction layer between the LSM hook callbacks and what the BPF LSM makes available to BPF programs. That's a choice made by the BPF LSM devs, and while it's a valid choice, it does have drawbacks like we are seeing here. Changes to the callback arguments and/or the semantic behavior of the LSM hooks are going to continue to happen regardless of what we do here. It's better if the BPF LSM figures out how to deal with problems like these rather than relying on a set of bandaids at the LSM framework layer. If the decision is to push the pain up to the user to keep the BPF program interface thin and flexible, that is their choice. -- paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-17 19:30 ` Paul Moore @ 2026-11-17 21:44 ` Andrii Nakryiko 2026-11-17 23:04 ` Paul Moore 0 siblings, 1 reply; 11+ messages in thread From: Andrii Nakryiko @ 2026-11-17 21:44 UTC (permalink / raw) To: Paul Moore Cc: roblabla, Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko, Christian Brauner On Tue, Nov 17, 2026 at 11:30 AM Paul Moore <paul@paul-moore.com> wrote: > > On Tue, Nov 17, 2026 at 11:51 AM Andrii Nakryiko > <andrii.nakryiko@gmail.com> wrote: > > On Thu, Nov 12, 2026 at 6:41 PM Paul Moore <paul@paul-moore.com> wrote: > > > On Thu, Nov 12, 2026 at 6:00 PM Andrii Nakryiko > > > <andrii.nakryiko@gmail.com> wrote: > > > > On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > > > > > > > > > Hi, > > > > > > > > > > Thanks for the replies! > > > > > > > > > > > LSMs that need to provide a stable API must provide an abstraction > > > > > > layer between the LSM hook callbacks and their stable API. The LSM > > > > > > framework is not responsible for the stability of individual LSM APIs. > > > > > > > > > > Understood, that lines up with my expectations. I think the biggest > > > > > thing here would not necessarily be stability guarantees, but rather to > > > > > make sure that the eBPFs fail to load when the compatibility is broken. > > > > > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > > > > > > > > > So, in short, would it be possible to send a follow up to Daan's > > > > > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > > > > > list? > > > > > > > > > > If this is possible, that'd be nice so that we avoid this breakage in > > > > > the short term, since the change isn't in a released kernel yet. > > > > > > > > can you please send a patch proposing that and we can continue > > > > discussion there? please cc me so I can follow along > > > > > > Considering the original patch was submitted, reviewed, and merged > > > with the BPF LSM being the only user (none of the existing in-tree > > > LSMs use the idmap parameter at present), I'm a little surprised this > > > wasn't brought up earlier. > > > > Yeah, unfortunately this was missed. Not sure it's surprising that it > > wasn't detected immediately, though, because unlikely someone tested > > an unreleased kernel in production. Luckily, this was still detected > > early enough before the final release and this gives us a chance to > > fix a mistake. > > I've been surprised lately by apparently how little the BPF LSM > program interface is tested with real BPF programs before landing in > Linus' tree. I know the BPF folks have a pretty good set of CI tests, > bots, etc. but there do seem to be gaps relating to real world usage. I think it's a hard problem, yes. BPF selftests test latest kernel, so this issue we are discussing here just won't come up there. Then, e.g., for Meta, we do have internal tests that validate existing BPF LSM programs across different kernel versions, but unless our internal BPF LSM programs utilize specific LSMs, this won't be caught either. > > > > Beyond that, from a style perspective, I think I prefer the idmap at > > > the front of the parameter list in most (all?) of the hooks where it > > > was added. I'm not interested in moving it at this point in time. > > > > So on one hand we have a real user pain which is easily fixable > > without burdening kernel and LSM subsystem in particular, right? > > Arguably changes that negatively impact code style and readability do > affect developers and maintainers. I agree, but argument order is not that big of an issue > > > On > > the other hand there is a minor issue of argument ordering with > > mnt_idmap preferably as a first argument. I get the preference, but I > > think fixing an unnecessary (but easily fixable!) user pain should > > come first, don't you agree? > > > > Can you please reconsider? > > The user pain exists because there is no abstraction layer between the > LSM hook callbacks and what the BPF LSM makes available to BPF > programs. That's a choice made by the BPF LSM devs, and while it's a > valid choice, it does have drawbacks like we are seeing here. Changes > to the callback arguments and/or the semantic behavior of the LSM > hooks are going to continue to happen regardless of what we do here. Yes, just like it will for tracepoints, kprobes, etc. Sometimes it's inevitable and BPF ecosystem does have tools to deal with. It's still a friction and pain and burden to do this, so if it's avoidable at low cost to the kernel, we should try to minimize unnecessary pain. That's really what I'm asking about here. We did do that with scheduling tracepoints, I think it's reasonable to do this here as well. > It's better if the BPF LSM figures out how to deal with problems like > these rather than relying on a set of bandaids at the LSM framework > layer. If the decision is to push the pain up to the user to keep the > BPF program interface thin and flexible, that is their choice. Yes, thin (and thus performant) and flexible is the design choice and I think it's fine. That's not really the crux of the problem here. As I mentioned, users have tools to work around changing unstable kernel interfaces that BPF programs rely on. But in this case we are discussing reordering a few arguments *before they even get into final release* just to avoid unnecessary pain and workaround. *Unnecessary* is the key here. I really don't think that's too much to ask in this situation. > > -- > paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-17 21:44 ` Andrii Nakryiko @ 2026-11-17 23:04 ` Paul Moore 0 siblings, 0 replies; 11+ messages in thread From: Paul Moore @ 2026-11-17 23:04 UTC (permalink / raw) To: Andrii Nakryiko Cc: roblabla, Daan De Meyer, bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko, Christian Brauner On Tue, Nov 17, 2026 at 4:44 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > On Tue, Nov 17, 2026 at 11:30 AM Paul Moore <paul@paul-moore.com> wrote: > > On Tue, Nov 17, 2026 at 11:51 AM Andrii Nakryiko > > <andrii.nakryiko@gmail.com> wrote: > > > On Thu, Nov 12, 2026 at 6:41 PM Paul Moore <paul@paul-moore.com> wrote: > > > > On Thu, Nov 12, 2026 at 6:00 PM Andrii Nakryiko > > > > <andrii.nakryiko@gmail.com> wrote: > > > > > On Mon, Nov 9, 2026 at 7:08 AM roblabla <linux@roblab.la> wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > Thanks for the replies! > > > > > > > > > > > > > LSMs that need to provide a stable API must provide an abstraction > > > > > > > layer between the LSM hook callbacks and their stable API. The LSM > > > > > > > framework is not responsible for the stability of individual LSM APIs. > > > > > > > > > > > > Understood, that lines up with my expectations. I think the biggest > > > > > > thing here would not necessarily be stability guarantees, but rather to > > > > > > make sure that the eBPFs fail to load when the compatibility is broken. > > > > > > But that's probably a problem for the eBPF side, and not LSM, I assume. > > > > > > > > > > > > > So, in short, would it be possible to send a follow up to Daan's > > > > > > > commit and move `struct mnt_idmap *idmap` to the end of LSM argument > > > > > > > list? > > > > > > > > > > > > If this is possible, that'd be nice so that we avoid this breakage in > > > > > > the short term, since the change isn't in a released kernel yet. > > > > > > > > > > can you please send a patch proposing that and we can continue > > > > > discussion there? please cc me so I can follow along > > > > > > > > Considering the original patch was submitted, reviewed, and merged > > > > with the BPF LSM being the only user (none of the existing in-tree > > > > LSMs use the idmap parameter at present), I'm a little surprised this > > > > wasn't brought up earlier. > > > > > > Yeah, unfortunately this was missed. Not sure it's surprising that it > > > wasn't detected immediately, though, because unlikely someone tested > > > an unreleased kernel in production. Luckily, this was still detected > > > early enough before the final release and this gives us a chance to > > > fix a mistake. > > > > I've been surprised lately by apparently how little the BPF LSM > > program interface is tested with real BPF programs before landing in > > Linus' tree. I know the BPF folks have a pretty good set of CI tests, > > bots, etc. but there do seem to be gaps relating to real world usage. > > I think it's a hard problem, yes. BPF selftests test latest kernel, so > this issue we are discussing here just won't come up there. Then, > e.g., for Meta, we do have internal tests that validate existing BPF > LSM programs across different kernel versions, but unless our internal > BPF LSM programs utilize specific LSMs, this won't be caught either. > > > > > > > Beyond that, from a style perspective, I think I prefer the idmap at > > > > the front of the parameter list in most (all?) of the hooks where it > > > > was added. I'm not interested in moving it at this point in time. > > > > > > So on one hand we have a real user pain which is easily fixable > > > without burdening kernel and LSM subsystem in particular, right? > > > > Arguably changes that negatively impact code style and readability do > > affect developers and maintainers. > > I agree, but argument order is not that big of an issue It can be for me, and in this case it is. > > > On > > > the other hand there is a minor issue of argument ordering with > > > mnt_idmap preferably as a first argument. I get the preference, but I > > > think fixing an unnecessary (but easily fixable!) user pain should > > > come first, don't you agree? > > > > > > Can you please reconsider? > > > > The user pain exists because there is no abstraction layer between the > > LSM hook callbacks and what the BPF LSM makes available to BPF > > programs. That's a choice made by the BPF LSM devs, and while it's a > > valid choice, it does have drawbacks like we are seeing here. Changes > > to the callback arguments and/or the semantic behavior of the LSM > > hooks are going to continue to happen regardless of what we do here. > > Yes, just like it will for tracepoints, kprobes, etc. Sometimes it's > inevitable and BPF ecosystem does have tools to deal with. It's still > a friction and pain and burden to do this, so if it's avoidable at low > cost to the kernel, we should try to minimize unnecessary pain. > > That's really what I'm asking about here. We did do that with > scheduling tracepoints, I think it's reasonable to do this here as > well. > > > It's better if the BPF LSM figures out how to deal with problems like > > these rather than relying on a set of bandaids at the LSM framework > > layer. If the decision is to push the pain up to the user to keep the > > BPF program interface thin and flexible, that is their choice. > > Yes, thin (and thus performant) and flexible is the design choice and > I think it's fine. That's not really the crux of the problem here. As > I mentioned, users have tools to work around changing unstable kernel > interfaces that BPF programs rely on. > > But in this case we are discussing reordering a few arguments *before > they even get into final release* just to avoid unnecessary pain and > workaround. *Unnecessary* is the key here. > > I really don't think that's too much to ask in this situation. As I said previously, the current ordering makes much more sense to me than moving it to the end of the list. I'm not interested in moving the parameter. -- paul-moore.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs 2026-11-09 15:07 ` roblabla 2026-11-12 23:00 ` Andrii Nakryiko @ 2026-11-18 0:52 ` Ihor Solodrai 1 sibling, 0 replies; 11+ messages in thread From: Ihor Solodrai @ 2026-11-18 0:52 UTC (permalink / raw) To: roblabla, Andrii Nakryiko, Paul Moore, Daan De Meyer Cc: bpf, linux-security-module, paul.mabileau, linux-kernel, Andrii Nakryiko On 11/9/26 8:07 AM, roblabla wrote: > Hi, > > Thanks for the replies! > >> [...] > > So what would be really nice is a way to make eBPF code robust against > kernel changes such that it either works, or fails to load, but doesn't > do the halfway "loads but silently misbehaves". Here are two ideas that > have been in the back of my mind that could help solve this issue (though > I understand that they may be rather large changes, I'm putting them out > there mostly as a conversation starter): > > 1. Could libbpf/the kernel have a mismatch check on hook arguments? I > understand that libbpf and the verifier don't know the types a > program expects, since BPF_PROG() only casts u64s. But the macro > does know the types at compile time. Could it (or an opt-in variant) > record the expected prototype in the program's BTF, so that libbpf > can compare it against the LSM hook's prototype in vmlinux BTF and > refuse to load on a mismatch? > > 2. Since kernel BTF has parameter names for the bpf_lsm_* functions, > could we maybe have CO-RE-style relocation to resolve "the argument > named dentry" to the argument index at load time? This way, adding a > new argument could happen anywhere in the argument list without > breaking eBPFs, and removing an argument will (correctly) cause our > eBPF to stop loading. It does mean the eBPF programs must provide the > same argument names, but maybe that's an acceptable tradeoff? Hi everyone. roblabla's comments got me curious, and I spent some time with my helpful assistants poking at the problem. There is another workaround in addition to what Andrii suggested. A program can already make itself fail to load on a signature mismatch: union security_list_options___mine { int (*inode_create)(struct inode *dir, struct dentry *dentry, umode_t mode); }; extern void lsm_proto_mismatch__inode_create(void) __ksym __weak; SEC("lsm/inode_create") int BPF_PROG(deny_create, struct inode *dir, struct dentry *dentry) { if (!bpf_core_type_matches(union security_list_options___mine)) lsm_proto_mismatch__inode_create(); ... } bpf_core_type_matches() checks the expected prototype against the hook's member of union security_list_options in kernel BTF. On a mismatch the call to the unresolved weak kfunc stays reachable, so the verifier rejects the program, and libbpf reports: kfunc 'lsm_proto_mismatch__inode_create' is referenced but wasn't resolved It's positional and needs the exact argument count, but it turns silent misbehavior into a load failure. The same should work for tp_btf with the btf_trace_<tp> typedefs. But of course it's just a workaround.. A bit more generally, expanding on roblabla's second idea, I think libbpf could resolve hook arguments by name with the existing CO-RE machinery, without kernel, verifier or LSM changes. The BPF side could look like this: SEC("lsm/inode_create") int BPF_PROG_ARGS(deny_create, struct inode *, dir, struct dentry *, dentry) { ... } BPF_PROG_ARGS() (name TBD) takes (type, name) pairs like BPF_PROG2(), but instead of casting ctx[0], ctx[1], ... it reads each argument through a local preserve_access_index-marked struct, so every argument access is a regular CO-RE field relocation: struct bpf_func_args___deny_create { const struct mnt_idmap *idmap; /* 7.4+ */ struct inode *dir; struct dentry *dentry; } __attribute__((preserve_access_index)); SEC("lsm/inode_create") int deny_create(u64 *ctx) { struct bpf_func_args___deny_create *args = (void *)ctx; if (bpf_core_field_exists(args->idmap)) /* use args->idmap */; return check(args->dir, args->dentry); } Nothing changes on the loader side: skel = deny_create__open_and_load(); if (!skel) /* mismatch: load fails, libbpf logs why */ return -errno; err = deny_create__attach(skel); To achieve this libbpf would need to do the following at load time: - treat bpf_func_args___* as a reserved type in CO-RE relocations; - find the program's attach target in kernel (or module) BTF: bpf_lsm_<hook>, the fentry/fexit target, or the tracepoint stub for tp_btf; - build a struct from the target's prototype: one member per parameter, named after it, at the parameter's ctx offset, plus 'ret' for fexit, fmod_ret and LSM; - resolve the relocations against it by name, with a strict type check (pointee types must match, qualifiers ignored), since CO-RE normally treats any two pointers as compatible. A missing argument or a type mismatch is a normal CO-RE relocation failure: the program fails to load with an error naming the argument, unless the access is guarded by bpf_core_field_exists(). And this is not LSM specific, fentry/fexit/fmod_ret and tp_btf would work the same way. It doesn't help if a hook keeps its signature but changes semantics, or if the attach point goes away (that already fails to attach). Here is a vibe-coded prototype of the above: https://github.com/kernel-patches/bpf/compare/bpf-next_base...theihor:bpf:bpf-named-args It's an entirely new libbpf feature though. Andrii, what do you think of it? --- And just my 2c re argument ordering: Paul, with all due respect, prioritizing code style and personal preferences over backwards compatibility is unreasonable even for unimportant applications. And here we are talking about apps implementing security policies, which many organizations (not just companies) in the world rely on. Given the recent flood of cybersecurity issues (in Linux and beyond), I don't see why would we, as in kernel devs, intentionally make the defenders' job harder. > > Thanks, > roblabla ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-11-18 0:52 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <d85ece6e-27f9-4d9f-83af-4101bc76fef6@app.fastmail.com>
2026-11-06 13:38 ` [BUG] bpf, lsm: new first argument in 7.4 inode hooks silently breaks BPF LSMs roblabla
2026-11-06 15:41 ` Paul Moore
2026-11-06 19:33 ` Andrii Nakryiko
2026-11-09 15:07 ` roblabla
2026-11-12 23:00 ` Andrii Nakryiko
2026-11-13 2:41 ` Paul Moore
2026-11-17 16:51 ` Andrii Nakryiko
2026-11-17 19:30 ` Paul Moore
2026-11-17 21:44 ` Andrii Nakryiko
2026-11-17 23:04 ` Paul Moore
2026-11-18 0:52 ` Ihor Solodrai
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.