Re: -next status as at v7.1-rc6

Re: -next status as at v7.1-rc6

[Paul Moore]

On Tue, Jun 2, 2026 at 4:20 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Mon, 1 Jun 2026 at 11:22, Mark Brown <broonie@kernel.org> wrote:
> >
> > We're reaching the peak of conflicts for this release cycle but
> > fundamentally everything is pretty quiet at the minute.
>
> So the "Hornet" LSM thing needs to bve removed from linux-next.
>
> I'm not ever going to pull it - it has been NAK'ed by the developers
> of the actual BPF code, and now it apparently causes merge problems
> too.

It's worth mentioning that resolving the merge issue was relatively
straightforward and we had a tested patch ready in a few hours; I was
in the process of merging it when your mail hit my inbox.  We see
cross-subsystem issues like this every couple of months. The good news
is that the linux-next process works well and we are usually able to
resolve the problems quickly.

I'll touch on the NACK below.

> The LSM people need to realize that they cannot override the people
> who actually write the real code.
>
> The security layer is not boss in the relationship. It's the
> subservient party. Security is important, but LSM's are not.

[NOTE: adding the LSM list to the CC line since I just realized you
didn't include it]

At this point I believe we are all aware of your dislike of LSMs, but
I once again feel compelled to speak out against the disrespect you
have shown towards the LSM developers and users.  Most (all?) of the
major Linux distributions rely on at least one LSM to meet the needs
of their users.  We've seen the importance of LSMs to Android, and the
real impact they have had on defending against vulnerabilities.  We
also know, oddly enough in this particular case, the importance of the
LSM framework to the BPF ecosystem; the BPF LSM attach points are
critical for many BPF use cases.

LSMs may not be important to you, but they are very important for a
very large number of Linux users.

> When the maintainer of a codebase NAK's a security model, and explains
> that the code has different needs and different security models, the
> LSM people don't just ignore that and go do their thing despite the
> NAK.

I think there may be some confusion about how Hornet works and
interacts with the existing BPF subsystem.  Hornet does not touch any
code inside the BPF subsystem, and outside of some minor PKCS7 patches
to enable some things from the PKCS specs, it doesn't touch any code
outside of security/.  Hornet's design criteria required it to work
within the existing LSM hooks and remain compatible with the existing
BPF signature verification code.  Hornet follows the traditional LSM
design pattern: it works within the LSM framework, building upon the
security functionality in other subsystems to satisfy user security
needs that have otherwise been ignored.

A BPF light skeleton signed with the Hornet tools passes both the
existing BPF signature verification code and Hornet's verification.
Hornet allows signature verification using arbitrary keys/keyrings,
just like the existing BPF signature code, enabling support for
dynamically generated/signed BPF programs.  In mixed environments,
Hornet can verify the loader portion of BPF light skeletons signed
with the existing BPF signing tools, distinguishing between existing
Alexei/KP signatures and a full Hornet signature.  Hornet also
supports unsigned BPF programs when needed.

As an LSM, Hornet can be enabled/disabled at build time, kernel boot,
and at runtime (subject to the LSM providing enforcement).  If a user
is required to run a specific kernel build, e.g. an "enterprise" Linux
distro support situation, the admin has multiple ways to disable
Hornet if it is not desired.  Any Hornet signed BPF light skeletons
will load without issue on a system without Hornet and Hornet's
presence does not block the existing BPF signature verification
mechanisms.

The BPF developers will quickly point out, if they haven't mentioned
it to you already, that Hornet calls into a 'bpf_map_ops' function.
This is true, Hornet calls the map_get_hash() method to get the hash
of a BPF map so it can be verified.  The BPF devs will argue this
presents a layering violation, but I would counter that several users
in the networking stack go much further than Hornet in their use of
'bpf_map_ops'; I find it difficult to see 'bpf_map_ops' as a private
API at this point.  It is also important to mention that Hornet does
not manipulate or modify the BPF program or map state in any way
beyond storing some state in a Hornet specific LSM blob, similar to
what other LSMs have done.

We've done this work, and tried collaborating with the BPF devs for
over a year, because we believe the kernel should verify the integrity
of both the BPF light skeleton loader and the associated maps.  While
the existing BPF signature verification scheme verifies the light
skeleton loader, it requires the loader to self-verify its associated
maps.  Relying on the loader to self-verify maps is problematic
because it adds an additional burden on system builders and admins who
must now also manage and verify the signature verification in every
BPF light skeleton loader on the system (how does one know if a
specific light skeleton loader suffers from a verification bug? what
additional steps need to be added to the deployment of third-party,
binary only BPF light skeletons?).  At the very least Hornet can
ensure the integrity of both the BPF light skeleton loader and the
signed maps without requiring prior analysis of the light skeleton
loader.  That is a big win for admins who care about what code is
loaded into their kernel.

Hornet lives entirely within the LSM framework, adds no additional LSM
hooks, remains compatible with the existing BPF signature verification
code, can be disabled in multiple ways if required, provides the
verification flexibility needed to support the existing BPF ecosystem,
and helps satisfy the needs of real users.  Why can't we support
Hornet alongside the existing BPF signature verification code and let
the users employ the mechanism that works best for them?

--
paul-moore.com

Re: -next status as at v7.1-rc6

[Linus Torvalds]

On Wed, 3 Jun 2026 at 17:04, Paul Moore <paul@paul-moore.com> wrote:
>
> It's worth mentioning that resolving the merge issue was relatively
> straightforward and we had a tested patch ready in a few hours

This is not the reason I'm not going to pull it - the merge issue was
just the reminder I got about an earlier email that I had dropped on
the floor.

No, the reason I won't pull it is that the main developer I pull bpf
code NAK'ed it.

Now, I will cdertainly sometimes override maintainers, so it's not
like a NAK is always some final thing.

I don't _like_ overriding developers, but I'll do it when I feel it is
necessary to make forward progress.

But I also have to  feel people have been unnecessarily difficult, and
I have been extensively informed about the decision and I feel like I
can make a reasonable judgement on it.

So it happens, but it happens with my explicit understanding.

My tree is *not* some kind of "we are bypassing developers by sending
a pull request directly to Linus" tree.

NEVER is that the way things get done.

[ Yes, that too has happened, and I have done that unwittingly because
I didn't realize what was going on ]

So I will not pull this tree. End of story.

The way to get me to override developers is to make me aware of the
conflict and convince me that yes, something needs overriding - but it
is typically not very easy to do with active developers.

And honestly, I also have two+ decades of history of "LSM people
cannot agree on a single thing".

That is _literally_ why the LSM layer exists in the first place.

So when LSM people then disagree with _other_ developers, quite
frankly my immediate and visceral reaction then is "oh, these people
who have decades of history of not being able to even agree amongst
themselves are now disagreeing with outsiders too".

Put another way: LSM people have a  higher barrier to convince me that
I should take their disagreements seriously.

And no, I'm afraid that may not be entirely fair.  But "history of
being disagreeable" is a thing.

              Linus

Re: -next status as at v7.1-rc6

[Paul Moore]

On Wed, Jun 3, 2026 at 8:32 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Wed, 3 Jun 2026 at 17:04, Paul Moore <paul@paul-moore.com> wrote:
> >
> > It's worth mentioning that resolving the merge issue was relatively
> > straightforward and we had a tested patch ready in a few hours
>
> This is not the reason I'm not going to pull it ...

I didn't assume the merge issue was the reason, but you felt it was
worth mentioning so I figured it deserved a response.

> ...  the merge issue was
> just the reminder I got about an earlier email that I had dropped on
> the floor.
>
> No, the reason I won't pull it is that the main developer I pull bpf
> code NAK'ed it.

I would like to clarify some things, not necessarily for Hornet's
sake, but because they have implications moving forward.  As much as I
*love* our little chats, I think we both would prefer not to repeat
this discussion in the future.

While you didn't reply to any of my comments explaining how Hornet
works, specifically how it ties into the kernel, I'm assuming you've
read the overview.  Can you help those of us in the LSM space
understand why a BPF dev's NACK on code that lives strictly under
security/ is sufficient grounds to reject an LSM patch?

Disagreement about what access controls an LSM is allowed to enforce?

Disagreement about using LSMs for additional signature verification,
despite compatibility (although one could argue we already do this
with IMA)?

Use of the bpf_map_ops::map_get_hash() method?

Other things ... ?  I'm genuinely not trying to be antagonistic; I'm
trying to understand your thinking in rejecting Hornet and take some
lessons away from it.  I'm currently stuck with (pardon the
paraphrase) "because Alexei said-so", and while I'm certain Alexei
would be happy to have that codified, I'd like to believe there is
more to your rejection than that.

> My tree is *not* some kind of "we are bypassing developers by sending
> a pull request directly to Linus" tree.

I've been sending you pull requests for a while now, and as you know
I'm always very upfront in the email if any of the commits contain a
NACK or even the absence of a cross-subsystem ACK.  My intention is
never to deceive anyone, my goal is always to make you aware of the
situation so that you can decide what to merge into your tree.

That was why I decided to merge Hornet into the LSM dev/next branch in
preparation for the upcoming merge window.  You had been CC'd on
several threads regarding the BPF signature verification work and
hadn't provided guidance on the cross-subsystem issues.  Even an
off-list email I sent you received what could best be described as a
shrug.  My plan was to present Hornet to you with an explanation,
likely similar to my last reply in this thread, detailing that we had
worked with the BPF devs for over a year and their solution actively
ignored our requirements (despite their claims otherwise).  Hornet,
while NACK'd by Alexei, did not touch any code under kernel/bpf/, was
compatible with the existing BPF verification code, and solved real
user problems.

While I was not looking forward to the discussion, I felt responsible
for trying one last time to bring this to your attention and make a
case for the signature verification requirements the BPF community has
chosen to reject.  It appears a private, off-list email from the BPF
devs preempted this, which is unfortunate regarding both visibility
and timing, but at least we are still having a discussion of sorts.

> And honestly, I also have two+ decades of history of "LSM people
> cannot agree on a single thing".

You've repeated this enough that I worry your opinion has ossified,
but I can promise you the reality is that the LSM maintainer community
is a reasonably tight knit group and has been for quite some time.
I'm laughing a bit as I write this (I know ...), but I would encourage
you to drop by the Linux Security Summit in Prague this fall; I
believe you'll already be in town for the Maintainer Summit and it
might be an opportunity to see some of us and realize we agree far
more than you envision.  You might particularly enjoy our "working
with Linus" support group that meets in the evening at a local bar ;)

-- 
paul-moore.com

Re: -next status as at v7.1-rc6

[Linus Torvalds]

On Thu, 4 Jun 2026 at 15:23, Paul Moore <paul@paul-moore.com> wrote:
>
> While you didn't reply to any of my comments explaining how Hornet
> works, specifically how it ties into the kernel, I'm assuming you've
> read the overview.  Can you help those of us in the LSM space
> understand why a BPF dev's NACK on code that lives strictly under
> security/ is sufficient grounds to reject an LSM patch?

Honestly, I'm not competent to make a judgment call between two
different models for hash chain verification, so I basically *have* to
go by maintainer opinions.

And the discussions I have been cc'd on have not been what I'd call
enlightening.

But people have pointed out that the LSM code mucks around with bpf
internals, and those NAK's have had reasons for them.

And honestly, I don't understand *why* Hornet does what it does - and
does it in ways that obviously annoy the bpf people. There is no
*reason* to look at the bpf maps that I can see, and from my
understanding of Alexei's arguments (which may be lacking), the fact
that Hornet does that is the main reason for the NAK.

But instead of working with the bpf people on coming up with some
model that does *not* do that, it all seems to have become a "we'll do
it anyway, despite maintainer complaints".

And I *did* see the bpf people pointing to "this would be an
acceptable alternative" with KP Singh outlining something that *had*
been discussed.

But I never actually saw anybody say "ok, we'll try that instead".

Maybe I missed it.

But from what I saw, it really looked like "I see NAK's from three
different bpf maintainers, with suggested alternate approaches". None
of which resulted in anything that looked like "ok, we'll try to
follow your guidance", only more of the same.

Why would *my* input then make any difference?

The bpf people's arguments resonated more with me. For example, the
whole "we need to know if it passed the bpf signature" seems to be
complate pointless silliness, and the bpf peoples responses to that
resonated with me. There's *no* point in any LSM check whether the
signature passed or not, since if it didn't pass, it's not getting
loaded.

So that's basically where I stand - I've seen disagreement, and I've
seen what looks to me like reasonable push-back, and I've not really
seen the LSM response as taking it into account.

           Linus

Re: -next status as at v7.1-rc6

[Paul Moore]

On Thu, Jun 4, 2026 at 7:19 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Thu, 4 Jun 2026 at 15:23, Paul Moore <paul@paul-moore.com> wrote:
> >
> > While you didn't reply to any of my comments explaining how Hornet
> > works, specifically how it ties into the kernel, I'm assuming you've
> > read the overview.  Can you help those of us in the LSM space
> > understand why a BPF dev's NACK on code that lives strictly under
> > security/ is sufficient grounds to reject an LSM patch?
>
> Honestly, I'm not competent to make a judgment call between two
> different models for hash chain verification, so I basically *have* to
> go by maintainer opinions.

I appreciate the explanation, thank you.

I'll admit it's not particularly satisfying, as it doesn't appear to
identify any specific failing other than two groups having differing
opinions.

> So that's basically where I stand - I've seen disagreement, and I've
> seen what looks to me like reasonable push-back, and I've not really
> seen the LSM response as taking it into account.

I would point out the several different attempts Blaise made to work
and compromise with the BPF devs before Hornet was even an idea.
Hornet came into existence only because the BPF devs refused to accept
any use cases other than their own.

Regardless, I think that's about it on this topic.  Thanks for the discussion.

... and of course the invitation to the security summit in Prague (or
any future instance for that matter) still stands.

-- 
paul-moore.com

Re: -next status as at v7.1-rc6

[Serge E. Hallyn]

On Thu, Jun 04, 2026 at 04:18:46PM -0700, Linus Torvalds wrote:
> On Thu, 4 Jun 2026 at 15:23, Paul Moore <paul@paul-moore.com> wrote:
> >
> > While you didn't reply to any of my comments explaining how Hornet
> > works, specifically how it ties into the kernel, I'm assuming you've
> > read the overview.  Can you help those of us in the LSM space
> > understand why a BPF dev's NACK on code that lives strictly under
> > security/ is sufficient grounds to reject an LSM patch?
> 
> Honestly, I'm not competent to make a judgment call between two
> different models for hash chain verification, so I basically *have* to
> go by maintainer opinions.
> 
> And the discussions I have been cc'd on have not been what I'd call
> enlightening.
> 
> But people have pointed out that the LSM code mucks around with bpf
> internals, and those NAK's have had reasons for them.
> 
> And honestly, I don't understand *why* Hornet does what it does - and
> does it in ways that obviously annoy the bpf people. There is no
> *reason* to look at the bpf maps that I can see, and from my
> understanding of Alexei's arguments (which may be lacking), the fact
> that Hornet does that is the main reason for the NAK.

The two most useful threads I believe were from a year ago,
20250502184421.1424368-1-bboscaccy@linux.microsoft.com
and
20250528215037.2081066-1-bboscaccy@linux.microsoft.com
which includes the proposal by the BPF side:
https://lore.kernel.org/linux-security-module/CACYkzJ6VQUExfyt0=-FmXz46GHJh3d=FXh5j4KfexcEFbHV-vg@mail.gmail.com/

There were 2 or three objections from each side iiuc, but the main ones
that stuck in my mind were

1. whether it is ok to rely on a signed userspace bpf verifier program to
   verify the signature.
2. objection by James Bottomley 
   (2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com)
   about the verifiability of the hash chain link.

> But instead of working with the bpf people on coming up with some
> model that does *not* do that, it all seems to have become a "we'll do
> it anyway, despite maintainer complaints".
> 
> And I *did* see the bpf people pointing to "this would be an
> acceptable alternative" with KP Singh outlining something that *had*
> been discussed.
> 
> But I never actually saw anybody say "ok, we'll try that instead".
> 
> Maybe I missed it.
> 
> But from what I saw, it really looked like "I see NAK's from three
> different bpf maintainers, with suggested alternate approaches". None
> of which resulted in anything that looked like "ok, we'll try to
> follow your guidance", only more of the same.
> 
> Why would *my* input then make any difference?
> 
> The bpf people's arguments resonated more with me. For example, the
> whole "we need to know if it passed the bpf signature" seems to be
> complate pointless silliness, and the bpf peoples responses to that
> resonated with me. There's *no* point in any LSM check whether the
> signature passed or not, since if it didn't pass, it's not getting
> loaded.
> 
> So that's basically where I stand - I've seen disagreement, and I've
> seen what looks to me like reasonable push-back, and I've not really
> seen the LSM response as taking it into account.
> 
>            Linus

Re: -next status as at v7.1-rc6

[Linus Torvalds]

On Fri, 5 Jun 2026 at 06:45, Serge E. Hallyn <serge@hallyn.com> wrote:
>
> The two most useful threads I believe were from a year ago,
> 20250502184421.1424368-1-bboscaccy@linux.microsoft.com
> and
> 20250528215037.2081066-1-bboscaccy@linux.microsoft.com
> which includes the proposal by the BPF side:
> https://lore.kernel.org/linux-security-module/CACYkzJ6VQUExfyt0=-FmXz46GHJh3d=FXh5j4KfexcEFbHV-vg@mail.gmail.com/

Ok, so that's not a thread I was cc'd on (and I'm not complaining -
this is not code that I *should* be cc'd on), but that does seem to
match what I saw in email discussions I've seen.

There clearly *have* been alternative proposals by the bpf people for
signed unprivileged eBPF.

And they have apparently been ignored for various reasons. And it's
not clear to me why LSM people think they can just ignore maintainers
in the subsystems they want to then check.

Now, again - I'm not claiming to be a subject expert, so I have to go
by maintainership trust. But as it stands now, I really don't see why
I would override the bpf maintainers' NAKs.

And that "as it stands now" obviously very much includes a "maybe in
the future I can be convinced that the bpf people are just being wrong
and overly difficult".

It certainly wouldn't be the first time kernel people have been ornery
and strongly opinionated and just loved arguing.

That's the kind of people we are.

                    Linus

Re: -next status as at v7.1-rc6

[James Bottomley]

On Fri, 2026-06-05 at 08:45 -0500, Serge E. Hallyn wrote:
> On Thu, Jun 04, 2026 at 04:18:46PM -0700, Linus Torvalds wrote:
> > On Thu, 4 Jun 2026 at 15:23, Paul Moore <paul@paul-moore.com>
> > wrote:
> > > 
> > > While you didn't reply to any of my comments explaining how
> > > Hornet works, specifically how it ties into the kernel, I'm
> > > assuming you've read the overview.  Can you help those of us in
> > > the LSM space understand why a BPF dev's NACK on code that lives
> > > strictly under security/ is sufficient grounds to reject an LSM
> > > patch?
> > 
> > Honestly, I'm not competent to make a judgment call between two
> > different models for hash chain verification, so I basically *have*
> > to go by maintainer opinions.
> > 
> > And the discussions I have been cc'd on have not been what I'd call
> > enlightening.
> > 
> > But people have pointed out that the LSM code mucks around with bpf
> > internals, and those NAK's have had reasons for them.
> > 
> > And honestly, I don't understand *why* Hornet does what it does -
> > and does it in ways that obviously annoy the bpf people. There is
> > no *reason* to look at the bpf maps that I can see, and from my
> > understanding of Alexei's arguments (which may be lacking), the
> > fact that Hornet does that is the main reason for the NAK.
> 
> The two most useful threads I believe were from a year ago,
> 20250502184421.1424368-1-bboscaccy@linux.microsoft.com
> and
> 20250528215037.2081066-1-bboscaccy@linux.microsoft.com
> which includes the proposal by the BPF side:
> https://lore.kernel.org/linux-security-module/CACYkzJ6VQUExfyt0=-FmXz46GHJh3d=FXh5j4KfexcEFbHV-vg@mail.gmail.com/
> 
> There were 2 or three objections from each side iiuc, but the main
> ones that stuck in my mind were
> 
> 1. whether it is ok to rely on a signed userspace bpf verifier
> program to
>    verify the signature.
> 2. objection by James Bottomley 
>   
> (2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com
> )
>    about the verifiability of the hash chain link.

My objection to the upstream signature scheme in that email was that if
you hand me a lskel consisting of a loader and a map and ask me to sign
it, how do I know the loader actually verifies the hash of the map
without having to disassemble and analyze it.

I think the second patch set (not yet upstream) makes that task easier
because it now calls out to the bpf_loader_verify_metadata() kfunc to
verify the map.  That means I can run the loader in a sandbox and see
if it makes the call, which is way easier than having to disassemble
and analyse its byte code.  Note: it's still not as easy as the hornet
method of simply having the map hash as a signed attribute I can
extract and verify, though.  However, the required task has gone from
pretty impossible to doable.

Regards,

James