* An opinion about Linux security
@ 2025-12-10 0:15 Timur Chernykh
2025-12-10 20:08 ` Casey Schaufler
2025-12-12 5:45 ` Dr. Greg
0 siblings, 2 replies; 13+ messages in thread
From: Timur Chernykh @ 2025-12-10 0:15 UTC (permalink / raw)
To: torvalds; +Cc: linux-security-module
Hello Linus,
I’m writing to ask for your opinion. What do you think about Linux’s
current readiness for security-focused commercial products? I’m
particularly interested in several areas.
First, in today’s 3rd-party (out-of-tree) EDR development — EDR being
the most common commercial class of security products — eBPF has
effectively become the main option. Yet eBPF is extremely restrictive.
It is not possible to write fully expressive real-time analysis code:
the verifier is overly strict, non-deterministic loops are not
allowed, and older kernels lack BTF support. These issues create real
limitations.
Second, the removal of the out-of-tree LSM API in the 4.x kernel
series caused significant problems for many AV/EDR vendors. I was
unable to find an explanation in the mailing lists that convincingly
justified that decision.
The next closest mechanism, fanotify, was a genuine improvement.
However, it does not allow an AV/EDR vendor to protect the integrity
of its own product. Is Linux truly expecting modern AV/EDR solutions
to rely on fanotify alone?
My main question is: what are the future plans? Linux provides very
few APIs for security and dynamic analysis. eBPF is still immature,
fanotify is insufficient, and driver workarounds that bypass kernel
restrictions are risky — they introduce both stability and security
problems. At the same time, properly implemented in-tree LSMs are not
inherently dangerous and remain the safer, supported path for
extending security functionality. Without safe, supported interfaces,
however, commercial products struggle to be competitive. At the
moment, macOS with its Endpoint Security Framework is significantly
ahead.
Yes, the kernel includes multiple in-tree LSM modules, but in practice
SELinux does not simplify operations — it often complicates them,
despite its long-standing presence. Many of the other LSMs are rarely
used in production. As an EDR developer, I seldom encounter them, and
when I do, they usually provide little practical value. Across
numerous real-world server intrusions, none of these LSM modules have
meaningfully prevented attacks, despite many years of kernel
development.
Perhaps it is time for Linux to focus on more than a theoretical model
of security.
P.S.
Everything above reflects only my personal opinion. I would greatly
appreciate your response and any criticism you may have.
Best regards,
Timur Chernykh
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-10 0:15 An opinion about Linux security Timur Chernykh
@ 2025-12-10 20:08 ` Casey Schaufler
2025-12-12 14:47 ` Timur Chernykh
2025-12-12 5:45 ` Dr. Greg
1 sibling, 1 reply; 13+ messages in thread
From: Casey Schaufler @ 2025-12-10 20:08 UTC (permalink / raw)
To: Timur Chernykh, torvalds; +Cc: linux-security-module, Casey Schaufler
On 12/9/2025 4:15 PM, Timur Chernykh wrote:
> Hello Linus,
>
> I’m writing to ask for your opinion. What do you think about Linux’s
> current readiness for security-focused commercial products? I’m
> particularly interested in several areas.
>
> First, in today’s 3rd-party (out-of-tree) EDR development — EDR being
> the most common commercial class of security products — eBPF has
> effectively become the main option. Yet eBPF is extremely restrictive.
eBPF has been accepted because of these restrictions. Without them
an eBPF program could very easily reek havoc on the system's behavior.
> It is not possible to write fully expressive real-time analysis code:
> the verifier is overly strict,
This is of necessity. It is the only protection against malicious and
badly written eBPF programs.
> non-deterministic loops are not
> allowed,
Without this restriction denial of service attacks become trivial.
> and older kernels lack BTF support.
Yes. That's true for many kernel features.
> These issues create real
> limitations.
>
> Second, the removal of the out-of-tree LSM API in the 4.x kernel
> series caused significant problems for many AV/EDR vendors. I was
> unable to find an explanation in the mailing lists that convincingly
> justified that decision.
To the best of my understanding, and I have been working with LSMs
and the infrastructure for quite some time, there has never been an
out-of-tree LSM API. We have been very clear that the LSM interfaces
are fluid. We have also been very clear that supporting out-of-tree
security modules is not a requirement. There are members of the
community who would prefer that they be completely prohibited.
> The next closest mechanism, fanotify, was a genuine improvement.
> However, it does not allow an AV/EDR vendor to protect the integrity
> of its own product. Is Linux truly expecting modern AV/EDR solutions
> to rely on fanotify alone?
You will need to provide more detail about why you believe that the
integrity of an AV/EDR product cannot be protected.
> My main question is: what are the future plans?
Security remains a dynamic technology. There are quite a few plans,
from a variety of sources, with no shortage of security goals. Trying
to keep up with the concern/crisis of the day is more than sufficient
to consume the resources committed to Linux security.
> Linux provides very
> few APIs for security and dynamic analysis.
What APIs do you want? It's possible that some exist that you haven't
discovered.
> eBPF is still immature,
And it will be for some time. That is, until it is mature.
> fanotify is insufficient,
Without specifics it is quite difficult to do anything about that.
And, as we like to say, patches are welcome.
> and driver workarounds that bypass kernel
> restrictions are risky — they introduce both stability and security
> problems.
Yes. Don't do that.
> At the same time, properly implemented in-tree LSMs are not
> inherently dangerous and remain the safer, supported path for
> extending security functionality. Without safe, supported interfaces,
> however, commercial products struggle to be competitive. At the
> moment, macOS with its Endpoint Security Framework is significantly
> ahead.
Please propose the interfaces you want. An API definition would be
good, and patches even better.
> Yes, the kernel includes multiple in-tree LSM modules, but in practice
> SELinux does not simplify operations — it often complicates them,
> despite its long-standing presence. Many of the other LSMs are rarely
> used in production. As an EDR developer, I seldom encounter them, and
> when I do, they usually provide little practical value. Across
> numerous real-world server intrusions, none of these LSM modules have
> meaningfully prevented attacks, despite many years of kernel
> development.
There are numerous cases where LSMs have mitigated attacks. One case:
https://mihail-milev.medium.com/mitigating-malware-risks-with-selinux-e37cf1db7c56
Your assertion that LSMs don't provide "real" value is not substantiated
in the literature.
> Perhaps it is time for Linux to focus on more than a theoretical model
> of security.
Each of the existing LSMs addresses one or more real world issues. I would
hardly call any of the Linux security mechanisms, from mode bits and setuid
through SELinux, landlock and EVM, "theoretical".
> P.S.
> Everything above reflects only my personal opinion. I would greatly
> appreciate your response and any criticism you may have.
We are always ready to improve the security infrastructure of Linux.
If it does not meet your needs, help us improve it. The best way to
accomplish this is to provide the changes required.
> Best regards,
> Timur Chernykh
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-10 0:15 An opinion about Linux security Timur Chernykh
2025-12-10 20:08 ` Casey Schaufler
@ 2025-12-12 5:45 ` Dr. Greg
2025-12-12 23:43 ` Casey Schaufler
1 sibling, 1 reply; 13+ messages in thread
From: Dr. Greg @ 2025-12-12 5:45 UTC (permalink / raw)
To: Timur Chernykh; +Cc: torvalds, linux-security-module
On Wed, Dec 10, 2025 at 03:15:39AM +0300, Timur Chernykh wrote:
Good morning Timur, I hope this note finds your week having gone well.
> Hello Linus,
>
> I'm writing to ask for your opinion. What do you think about Linux's
> current readiness for security-focused commercial products? I'm
> particularly interested in several areas.
I don't expect you will receive an answer.
Based on his previous comments and long standing position on this
issue, I believe it can be fairly stated that he looks at the LSM as
an unnecessary evil.
So in his absence, some 'in loco parentis' reflections on the issues
you raise.
I've been advised, more than once, that in this day and age, no one is
interested in reading more than a two sentence paragraph, so a short
response to your issues here and a bit more detail for anyone who
wants to read more, at the end.
There is active art available to address the shortcomings you outline
in your post below. Our TSEM LSM was designed to service the
realitities of the modern security environment and where it is going.
In a manner that doesn't provide any restrictions on how 'security'
can be implemented.
We've done four releases over three years and we believe an unbiased
observer would conclude they have received no substantive technical
review that would support interest in upstream integration.
The challenge is that the security gatekeepers view LSM submissions
through a lens of whether or not the LSM implements security
consistent with what they believe is security. Those views are
inconsistent with the realities of the modern security market, a
market that that is now predicated on detection rather than
enforcement. A trend that will only accelerate with advancements in
machine learning and AI.
It is worth noting that the history of the technology industry is
littered with examples of technology incumbents usually missing
disruptive innovation.
This restriction on suitability is actually inconsistent with Linus'
stated position on how Linux sub-systems can be used, as expressed in
his comment in the following post.
https://lore.kernel.org/lkml/CAHk-=wgLbz1Bm8QhmJ4dJGSmTuV5w_R0Gwvg5kHrYr4Ko9dUHQ@mail.gmail.com/
So the problem is not technical, it is a political eco-system problem.
So, big picture, that is the challenge facing resolution of your
concerns.
Apologies to everyone about the paragraph/sentence overflow and any
afront to sensibilities.
More detail below if anyone is interested.
> First, in today's 3rd-party (out-of-tree) EDR development EDR
> being the most common commercial class of security products eBPF
> has effectively become the main option. Yet eBPF is extremely
> restrictive. It is not possible to write fully expressive real-time
> analysis code: the verifier is overly strict, non-deterministic
> loops are not allowed, and older kernels lack BTF support. These
> issues create real limitations.
>
> Second, the removal of the out-of-tree LSM API in the 4.x kernel
> series caused significant problems for many AV/EDR vendors. I was
> unable to find an explanation in the mailing lists that convincingly
> justified that decision.
>
> The next closest mechanism, fanotify, was a genuine improvement.
> However, it does not allow an AV/EDR vendor to protect the integrity
> of its own product. Is Linux truly expecting modern AV/EDR solutions
> to rely on fanotify alone?
>
> My main question is: what are the future plans? Linux provides very
> few APIs for security and dynamic analysis. eBPF is still immature,
> fanotify is insufficient, and driver workarounds that bypass kernel
> restrictions are risky they introduce both stability and security
> problems. At the same time, properly implemented in-tree LSMs are not
> inherently dangerous and remain the safer, supported path for
> extending security functionality. Without safe, supported interfaces,
> however, commercial products struggle to be competitive. At the
> moment, macOS with its Endpoint Security Framework is significantly
> ahead.
>
> Yes, the kernel includes multiple in-tree LSM modules, but in
> practice SELinux does not simplify operations it often complicates
> them, despite its long-standing presence. Many of the other LSMs are
> rarely used in production. As an EDR developer, I seldom encounter
> them, and when I do, they usually provide little practical
> value. Across numerous real-world server intrusions, none of these
> LSM modules have meaningfully prevented attacks, despite many years
> of kernel development.
>
> Perhaps it is time for Linux to focus on more than a theoretical model
> of security.
The heart of the political eco-system challenge is best expressed by a
quote from Kyle Moffett, in which he stated that security should only
be developed and implemented by experts. Unfortunately that view is
inconsistent with the current state of the technology industry.
Classical security practititioners will defend complex subject/object
architectures with: "Google uses SeLinux for Android security".
Our response to that is that the world doesn't have a security problem
because Google lacks sufficient resources to implement anything it
desires to implement, regardless of the development and maintenance
input costs.
Unfortunately, that luxury is inconsistent with the rest of the
software development world that doesn't enjoy a 3.8 trillion dollar
market capitalization.
The world simply lacks enough experts to make the 'security only by
experts' model work.
Today, the fastest way to a product is to grab Linux and a development
team and write software for hardware that is now completely
commoditized. Everyone knows that security is not one of the
fundamental project predicates in this model.
Both NIST and DHS/CISA are officially on record as indicating that
security needs to start with and be baked in through the development
process. One of the objectives of TSEM was to provide a framework for
enabling this concept for the implementation of analysis and mandatory
behavior controls for software workloads.
A second fundamental problem is that the world has moved, in large
part, to containerized execution workloads. The Linux LSM, in its
current form, doesn't effectively support the application of workload
specific security policies.
Further complicating this issue is the fact that LSM 'stacking'
requires reasoning as to what a final security policy will be when
multiple different security architectures/policies get to decide on
the outcome of a security event/hook. The concept of least surprise
would suggest the need for stacking to have idempotency, in other
words, the order in which LSM event consumers are called shouldn't
influence the effective policy, but this is generally acknowledged as
not being the case with 'stacking'.
So we designed TSEM to provide an alternative, not a replacement, but
an alternative to how developers and system administrators can develop
and apply security policy, including integrity controls.
TSEM is an LSM that implements containerized security infrastructure
rather than security policy. It is designed around the concept of a
security orchestrator that can execute security isolated workloads and
receive the LSM events and their parameters from that workload and
process them in any manner it wishes.
For example: A Docker/Kubernetes container can be run and all of the
security events by that workload exported up into an OpenSearch or
ElasticSearch instance for anomaly detection and analysis.
So an EDR implemented on top of this has visibility into all of the
security events and their characteristics that are deemed security
relevant by the kernel developers.
One of the pushbacks is that this can lead to asynchronous security
decisions, but as you note, that is the model that the commercial
security industry and the consumers of its products has embraced,
particularly in light of the advancements coming out of the AI
industry, detection rather than enforcement.
If synchronous enforcement is required TSEM provides that as well,
including the use of standard kernel modules to implement analysis and
response to the LSM hooks. Internally we have implemented other LSM's
such as Tomoyo and IMA as loadable modules that can support multiple
and independent workload policies.
If you or other EDR vendors are interested, we would be more than
happy to engage in conversations as to how to improve the capabilities
of this type of architecture, as an alternative to what is currently
available in Linux, which as you note, has significant limitations.
> Everything above reflects only my personal opinion. I would greatly
> appreciate your response and any criticism you may have.
As I mentioned at the outset, you are unlikely to hear anything.
For the necessary Linux infrastructure improvements to emerge we
believe there is the need to develop and engage a community effort
that independently pursues the advancements that are necessary,
particularly those that enable Linux to implement first class AI based
security controls.
We believe that only this will result in sufficient 'market pull' at the
distribution level to help shape upstreaming decisions.
Absent that, it is likely that Linux will continue to implement what
has failed to work in the past in the hope that it will somehow work
in the future.
Comments and criticism welcome, we have had plenty of experience with
the latter.... :-)
> Best regards,
> Timur Chernykh
Best wishes for the success of your work and a pleasant holiday season.
As always,
Dr. Greg
The Quixote Project - Flailing at the Travails of Cybersecurity
https://github.com/Quixote-Project
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-10 20:08 ` Casey Schaufler
@ 2025-12-12 14:47 ` Timur Chernykh
2025-12-12 16:11 ` Stephen Smalley
2025-12-12 18:10 ` Dr. Greg
0 siblings, 2 replies; 13+ messages in thread
From: Timur Chernykh @ 2025-12-12 14:47 UTC (permalink / raw)
To: Casey Schaufler; +Cc: torvalds, linux-security-module, greg
Thanks for the answer!
> eBPF has been accepted because of these restrictions. Without them
> an eBPF program could very easily reek havoc on the system's behavior.
Unfortunately, these restrictions also make it impossible to use eBPF
effectively for security. We can certainly discuss the limitations
themselves if needed, but that’s not the main point I’m trying to make
here.
> Yes. That's true for many kernel features.
Right, and that’s a well-known issue that has no clean solution other
than backporting. I completely agree with that.
> To the best of my understanding, and I have been working with LSMs
> and the infrastructure for quite some time, there has never been an
> out-of-tree LSM API.
I was under the impression that there was something of that sort.
For example: https://elixir.bootlin.com/linux/v3.10/source/include/linux/security.h#L1688
It may not have been “officially” exported, but people who needed it
simply resolved the symbol through `kallsyms` or a kprobe and used it.
I’m not claiming that this is good practice, but it did work.
> You will need to provide more detail about why you believe that the
> integrity of an AV/EDR product cannot be protected.
Consider the case where an attacker already has root access. The first
thing they typically do is disable or tamper with security tools.
After that, they establish persistence. Access-control models alone
are not enough here — sometimes they need to be adjusted dynamically,
based on complex real-time analysis.
> What APIs do you want? It's possible that some exist that you haven't
> discovered.
I’m confident I’ve studied all the mechanisms available for real-time
system protection. I’ve even given talks and published material on
approaches to securing Linux.
I may not have covered every LSM module in depth, but in practice they
simply don’t provide the capabilities required for modern security
workloads.
And I don’t want to sound overly categorical, but the current path of
security development feels like a dead end to me, for several reasons.
Modern security relies on analyzing hundreds of thousands of events
per second. eBPF isn’t particularly fast to begin with, and on top of
that it imposes strict technological limitations that prevent
meaningful in-kernel analysis. Every EDR vendor ends up doing the same
thing: collecting raw data (which is already difficult) and pushing it
into userspace for processing. That adds transport overhead on top of
everything else.
Or, of course, writes a dirty and unsafe kernel module. For me in the
perfect world seems to it's better to provide an API for userspace.
It's safe enough and doesn't contain a restriction that eBPF has.
> Please propose the interfaces you want. An API definition would be
> good, and patches even better.
I’m lucky enough to have already built a working prototype, which I
once offered for review:
https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/include/uapi/linux/esf
Less lucky was the reaction I received. Paul Moore was strongly
opposed, as far as I remember. Dr. Greg once said that heat death of
the universe is more likely than this approach being accepted into the
kernel.
I don’t want to “poke old wounds”, but the design I proposed was
largely inspired by Apple’s Endpoint Security Framework (originally
“mac policies”):
https://developer.apple.com/documentation/endpointsecurity
In my view, it is an exemplary model. The kernel provides a consistent
stream of events to userspace, hides the complexity of data collection
and transport, and ensures fault-tolerance if userspace fails to
process events quickly enough.
Audit, for comparison, simply hangs the system if its buffer
overflows. ESF does not have that problem and even lets userspace
control the event stream.
> There are numerous cases where LSMs have mitigated attacks. One case:
And there are thousands of cases where LSMs did not. Most companies
never disclose successful breaches because of NDAs and fear of
investor backlash. The few disclosures we see typically appear only
when it’s safe for the company, or when they specifically want to
appear transparent.
> Your assertion that LSMs don't provide "real" value is not substantiated
> in the literature.
Which literature do you mean?
As I mentioned above, public reports about successful attacks are rare.
Maybe I poorly delivered my point. I'll try again.
The strongest protection on Linux today is provided by proprietary
security vendors. Unfortunately, they often achieve this using
kernel-module “hacks” rather than LSMs. It’s not because they prefer
hacks — it’s because Linux doesn’t offer proper APIs to implement the
required functionality.
> Each of the existing LSMs addresses one or more real world issues. I would
> hardly call any of the Linux security mechanisms, from mode bits and setuid
> through SELinux, landlock and EVM, "theoretical".
In practice, most LSM modules are either disabled or misconfigured on
real servers, and the responsibility falls on AV/EDR vendors. As
someone who has been building such systems professionally for years, I
can say with confidence: there is no clean, reliable mechanism
available today.
As much as I love Linux, its security and its friendliness toward
security vendors are its weak side.
In the real world, deploying a modern security product means dealing
with endless pain: missing kernel headers (sometimes there isn’t even
Internet access to download them), unstable kernel APIs, lack of BTF,
constant fights with the eBPF verifier, and all the other limitations
of eBPF.
To be honest, many developers — and I’m taking the liberty to speak
broadly here — would simply like to see something similar to Apple’s
ESF available on Linux.
Best regards,
Timur Chernykh
On Wed, Dec 10, 2025 at 11:08 PM Casey Schaufler <casey@schaufler-ca.com> wrote:
>
> On 12/9/2025 4:15 PM, Timur Chernykh wrote:
> > Hello Linus,
> >
> > I’m writing to ask for your opinion. What do you think about Linux’s
> > current readiness for security-focused commercial products? I’m
> > particularly interested in several areas.
> >
> > First, in today’s 3rd-party (out-of-tree) EDR development — EDR being
> > the most common commercial class of security products — eBPF has
> > effectively become the main option. Yet eBPF is extremely restrictive.
>
> eBPF has been accepted because of these restrictions. Without them
> an eBPF program could very easily reek havoc on the system's behavior.
>
> > It is not possible to write fully expressive real-time analysis code:
> > the verifier is overly strict,
>
> This is of necessity. It is the only protection against malicious and
> badly written eBPF programs.
>
> > non-deterministic loops are not
> > allowed,
>
> Without this restriction denial of service attacks become trivial.
>
> > and older kernels lack BTF support.
>
> Yes. That's true for many kernel features.
>
> > These issues create real
> > limitations.
> >
> > Second, the removal of the out-of-tree LSM API in the 4.x kernel
> > series caused significant problems for many AV/EDR vendors. I was
> > unable to find an explanation in the mailing lists that convincingly
> > justified that decision.
>
> To the best of my understanding, and I have been working with LSMs
> and the infrastructure for quite some time, there has never been an
> out-of-tree LSM API. We have been very clear that the LSM interfaces
> are fluid. We have also been very clear that supporting out-of-tree
> security modules is not a requirement. There are members of the
> community who would prefer that they be completely prohibited.
>
>
> > The next closest mechanism, fanotify, was a genuine improvement.
> > However, it does not allow an AV/EDR vendor to protect the integrity
> > of its own product. Is Linux truly expecting modern AV/EDR solutions
> > to rely on fanotify alone?
>
> You will need to provide more detail about why you believe that the
> integrity of an AV/EDR product cannot be protected.
>
> > My main question is: what are the future plans?
>
> Security remains a dynamic technology. There are quite a few plans,
> from a variety of sources, with no shortage of security goals. Trying
> to keep up with the concern/crisis of the day is more than sufficient
> to consume the resources committed to Linux security.
>
> > Linux provides very
> > few APIs for security and dynamic analysis.
>
> What APIs do you want? It's possible that some exist that you haven't
> discovered.
>
> > eBPF is still immature,
>
> And it will be for some time. That is, until it is mature.
>
> > fanotify is insufficient,
>
> Without specifics it is quite difficult to do anything about that.
> And, as we like to say, patches are welcome.
>
> > and driver workarounds that bypass kernel
> > restrictions are risky — they introduce both stability and security
> > problems.
>
> Yes. Don't do that.
>
> > At the same time, properly implemented in-tree LSMs are not
> > inherently dangerous and remain the safer, supported path for
> > extending security functionality. Without safe, supported interfaces,
> > however, commercial products struggle to be competitive. At the
> > moment, macOS with its Endpoint Security Framework is significantly
> > ahead.
>
> Please propose the interfaces you want. An API definition would be
> good, and patches even better.
>
> > Yes, the kernel includes multiple in-tree LSM modules, but in practice
> > SELinux does not simplify operations — it often complicates them,
> > despite its long-standing presence. Many of the other LSMs are rarely
> > used in production. As an EDR developer, I seldom encounter them, and
> > when I do, they usually provide little practical value. Across
> > numerous real-world server intrusions, none of these LSM modules have
> > meaningfully prevented attacks, despite many years of kernel
> > development.
>
> There are numerous cases where LSMs have mitigated attacks. One case:
>
> https://mihail-milev.medium.com/mitigating-malware-risks-with-selinux-e37cf1db7c56
>
> Your assertion that LSMs don't provide "real" value is not substantiated
> in the literature.
>
> > Perhaps it is time for Linux to focus on more than a theoretical model
> > of security.
>
> Each of the existing LSMs addresses one or more real world issues. I would
> hardly call any of the Linux security mechanisms, from mode bits and setuid
> through SELinux, landlock and EVM, "theoretical".
>
> > P.S.
> > Everything above reflects only my personal opinion. I would greatly
> > appreciate your response and any criticism you may have.
>
> We are always ready to improve the security infrastructure of Linux.
> If it does not meet your needs, help us improve it. The best way to
> accomplish this is to provide the changes required.
>
> > Best regards,
> > Timur Chernykh
> >
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 14:47 ` Timur Chernykh
@ 2025-12-12 16:11 ` Stephen Smalley
2025-12-12 17:06 ` Paul Moore
2025-12-12 18:10 ` Dr. Greg
1 sibling, 1 reply; 13+ messages in thread
From: Stephen Smalley @ 2025-12-12 16:11 UTC (permalink / raw)
To: Timur Chernykh; +Cc: Casey Schaufler, torvalds, linux-security-module, greg
On Fri, Dec 12, 2025 at 9:47 AM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
>
> Thanks for the answer!
>
> > eBPF has been accepted because of these restrictions. Without them
> > an eBPF program could very easily reek havoc on the system's behavior.
>
> Unfortunately, these restrictions also make it impossible to use eBPF
> effectively for security. We can certainly discuss the limitations
> themselves if needed, but that’s not the main point I’m trying to make
> here.
There was no shortage of talks about using eBPF for security at LPC this week,
https://lpc.events/event/19/sessions/235/#all
If eBPF can't do what you want today, I suspect it will in the
not-too-distant future.
The best way to make it so would be to try implementing your desired
functionality using it, find the cases where it is currently
insufficient and then take those to LPC or other venues to argue for
enhancements to enable it to do so. Doing so may require re-thinking
how you implement that functionality to fit within the required safety
constraints but that doesn't mean it is impossible.
> > Please propose the interfaces you want. An API definition would be
> > good, and patches even better.
>
> I’m lucky enough to have already built a working prototype, which I
> once offered for review:
>
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/include/uapi/linux/esf
>
> Less lucky was the reaction I received. Paul Moore was strongly
> opposed, as far as I remember. Dr. Greg once said that heat death of
> the universe is more likely than this approach being accepted into the
> kernel.
Not seeing an actual response from Paul in the archives, but did you
ever actually post patches to the list? Posting a GitHub URL is NOT a
real request for review.
> I don’t want to “poke old wounds”, but the design I proposed was
> largely inspired by Apple’s Endpoint Security Framework (originally
> “mac policies”):
> https://developer.apple.com/documentation/endpointsecurity
>
> In my view, it is an exemplary model. The kernel provides a consistent
> stream of events to userspace, hides the complexity of data collection
> and transport, and ensures fault-tolerance if userspace fails to
> process events quickly enough.
> Audit, for comparison, simply hangs the system if its buffer
> overflows. ESF does not have that problem and even lets userspace
> control the event stream.
>
> > There are numerous cases where LSMs have mitigated attacks. One case:
>
> And there are thousands of cases where LSMs did not. Most companies
> never disclose successful breaches because of NDAs and fear of
> investor backlash. The few disclosures we see typically appear only
> when it’s safe for the company, or when they specifically want to
> appear transparent.
>
> > Your assertion that LSMs don't provide "real" value is not substantiated
> > in the literature.
>
> Which literature do you mean?
> As I mentioned above, public reports about successful attacks are rare.
>
> Maybe I poorly delivered my point. I'll try again.
>
> The strongest protection on Linux today is provided by proprietary
> security vendors. Unfortunately, they often achieve this using
> kernel-module “hacks” rather than LSMs. It’s not because they prefer
> hacks — it’s because Linux doesn’t offer proper APIs to implement the
> required functionality.
>
> > Each of the existing LSMs addresses one or more real world issues. I would
> > hardly call any of the Linux security mechanisms, from mode bits and setuid
> > through SELinux, landlock and EVM, "theoretical".
>
> In practice, most LSM modules are either disabled or misconfigured on
> real servers, and the responsibility falls on AV/EDR vendors. As
> someone who has been building such systems professionally for years, I
> can say with confidence: there is no clean, reliable mechanism
> available today.
> As much as I love Linux, its security and its friendliness toward
> security vendors are its weak side.
>
> In the real world, deploying a modern security product means dealing
> with endless pain: missing kernel headers (sometimes there isn’t even
> Internet access to download them), unstable kernel APIs, lack of BTF,
> constant fights with the eBPF verifier, and all the other limitations
> of eBPF.
>
> To be honest, many developers — and I’m taking the liberty to speak
> broadly here — would simply like to see something similar to Apple’s
> ESF available on Linux.
>
> Best regards,
> Timur Chernykh
>
> On Wed, Dec 10, 2025 at 11:08 PM Casey Schaufler <casey@schaufler-ca.com> wrote:
> >
> > On 12/9/2025 4:15 PM, Timur Chernykh wrote:
> > > Hello Linus,
> > >
> > > I’m writing to ask for your opinion. What do you think about Linux’s
> > > current readiness for security-focused commercial products? I’m
> > > particularly interested in several areas.
> > >
> > > First, in today’s 3rd-party (out-of-tree) EDR development — EDR being
> > > the most common commercial class of security products — eBPF has
> > > effectively become the main option. Yet eBPF is extremely restrictive.
> >
> > eBPF has been accepted because of these restrictions. Without them
> > an eBPF program could very easily reek havoc on the system's behavior.
> >
> > > It is not possible to write fully expressive real-time analysis code:
> > > the verifier is overly strict,
> >
> > This is of necessity. It is the only protection against malicious and
> > badly written eBPF programs.
> >
> > > non-deterministic loops are not
> > > allowed,
> >
> > Without this restriction denial of service attacks become trivial.
> >
> > > and older kernels lack BTF support.
> >
> > Yes. That's true for many kernel features.
> >
> > > These issues create real
> > > limitations.
> > >
> > > Second, the removal of the out-of-tree LSM API in the 4.x kernel
> > > series caused significant problems for many AV/EDR vendors. I was
> > > unable to find an explanation in the mailing lists that convincingly
> > > justified that decision.
> >
> > To the best of my understanding, and I have been working with LSMs
> > and the infrastructure for quite some time, there has never been an
> > out-of-tree LSM API. We have been very clear that the LSM interfaces
> > are fluid. We have also been very clear that supporting out-of-tree
> > security modules is not a requirement. There are members of the
> > community who would prefer that they be completely prohibited.
> >
> >
> > > The next closest mechanism, fanotify, was a genuine improvement.
> > > However, it does not allow an AV/EDR vendor to protect the integrity
> > > of its own product. Is Linux truly expecting modern AV/EDR solutions
> > > to rely on fanotify alone?
> >
> > You will need to provide more detail about why you believe that the
> > integrity of an AV/EDR product cannot be protected.
> >
> > > My main question is: what are the future plans?
> >
> > Security remains a dynamic technology. There are quite a few plans,
> > from a variety of sources, with no shortage of security goals. Trying
> > to keep up with the concern/crisis of the day is more than sufficient
> > to consume the resources committed to Linux security.
> >
> > > Linux provides very
> > > few APIs for security and dynamic analysis.
> >
> > What APIs do you want? It's possible that some exist that you haven't
> > discovered.
> >
> > > eBPF is still immature,
> >
> > And it will be for some time. That is, until it is mature.
> >
> > > fanotify is insufficient,
> >
> > Without specifics it is quite difficult to do anything about that.
> > And, as we like to say, patches are welcome.
> >
> > > and driver workarounds that bypass kernel
> > > restrictions are risky — they introduce both stability and security
> > > problems.
> >
> > Yes. Don't do that.
> >
> > > At the same time, properly implemented in-tree LSMs are not
> > > inherently dangerous and remain the safer, supported path for
> > > extending security functionality. Without safe, supported interfaces,
> > > however, commercial products struggle to be competitive. At the
> > > moment, macOS with its Endpoint Security Framework is significantly
> > > ahead.
> >
> > Please propose the interfaces you want. An API definition would be
> > good, and patches even better.
> >
> > > Yes, the kernel includes multiple in-tree LSM modules, but in practice
> > > SELinux does not simplify operations — it often complicates them,
> > > despite its long-standing presence. Many of the other LSMs are rarely
> > > used in production. As an EDR developer, I seldom encounter them, and
> > > when I do, they usually provide little practical value. Across
> > > numerous real-world server intrusions, none of these LSM modules have
> > > meaningfully prevented attacks, despite many years of kernel
> > > development.
> >
> > There are numerous cases where LSMs have mitigated attacks. One case:
> >
> > https://mihail-milev.medium.com/mitigating-malware-risks-with-selinux-e37cf1db7c56
> >
> > Your assertion that LSMs don't provide "real" value is not substantiated
> > in the literature.
> >
> > > Perhaps it is time for Linux to focus on more than a theoretical model
> > > of security.
> >
> > Each of the existing LSMs addresses one or more real world issues. I would
> > hardly call any of the Linux security mechanisms, from mode bits and setuid
> > through SELinux, landlock and EVM, "theoretical".
> >
> > > P.S.
> > > Everything above reflects only my personal opinion. I would greatly
> > > appreciate your response and any criticism you may have.
> >
> > We are always ready to improve the security infrastructure of Linux.
> > If it does not meet your needs, help us improve it. The best way to
> > accomplish this is to provide the changes required.
> >
> > > Best regards,
> > > Timur Chernykh
> > >
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 16:11 ` Stephen Smalley
@ 2025-12-12 17:06 ` Paul Moore
2025-12-12 17:22 ` Timur Chernykh
0 siblings, 1 reply; 13+ messages in thread
From: Paul Moore @ 2025-12-12 17:06 UTC (permalink / raw)
To: Stephen Smalley
Cc: Timur Chernykh, Casey Schaufler, torvalds, linux-security-module,
greg
On Fri, Dec 12, 2025 at 11:12 AM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
> On Fri, Dec 12, 2025 at 9:47 AM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
> >
> > I’m lucky enough to have already built a working prototype, which I
> > once offered for review:
> >
> > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/include/uapi/linux/esf
> >
> > Less lucky was the reaction I received. Paul Moore was strongly
> > opposed, as far as I remember. Dr. Greg once said that heat death of
> > the universe is more likely than this approach being accepted into the
> > kernel.
>
> Not seeing an actual response from Paul in the archives, but did you
> ever actually post patches to the list?
I was wondering about this too. I searched through my sent mail and
while it's possible I'm missing some mail, the only conversation I see
with Timur is an off-list discussion from 2024 regarding changes in
the upstream kernel to support out-of-tree LSMs. While Timur was
added to the conversation by someone, I don't see any mail from him in
that thread. My comments in that thread are consistent with my
comments in on-list threads from around that same time when
out-of-tree code was discussed. Here is a snippet from one of my
responses, which still holds true as far as I'm concerned:
"As stated many times in the past, the LSM framework as
well as the Linux kernel in general, does not provide the
same level of consideration to out-of-tree code that it does
to upstream, mainline code. My policy on this remains the
same as last time we talked: while I have no goal to make
things difficult for out-of-tree code, I will not sacrifice
the continued development and maintenance of existing
upstream code in favor of out-of-tree code."
Searching for "Timur Chernykh" in all of the lore archives shows some
BPF related threads and the following LSM thread from June 2024:
https://lore.kernel.org/all/CABZOZnS13-KscVQY0YqqWZsBwmQaKyRO_G=kzCL8zc9jHxAC=A@mail.gmail.com
In this post Timur provides links to his ESF project on GitHub, but no
patches. I see comments from Stephen, Tetsuo, Casey, and Dr. Greg; I
did not comment on that thread.
Beyond that, I'm a bit lost. As far as I can remember, and both lore
and my own sent mail folder appear to support this, I've never
commented on ESF. At this point I think Timur may be mistaken
regarding my commenting on ESF, but if I am wrong please provide a
lore link so I can refresh my memory.
--
paul-moore.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 17:06 ` Paul Moore
@ 2025-12-12 17:22 ` Timur Chernykh
2025-12-12 17:37 ` Stephen Smalley
0 siblings, 1 reply; 13+ messages in thread
From: Timur Chernykh @ 2025-12-12 17:22 UTC (permalink / raw)
To: Paul Moore
Cc: Stephen Smalley, Casey Schaufler, torvalds, linux-security-module,
greg
> While Timur was
> added to the conversation by someone, I don't see any mail from him in
> that thread.
I probably missed this thread.
> Beyond that, I'm a bit lost. As far as I can remember, and both lore
> and my own sent mail folder appear to support this, I've never
> commented on ESF. At this point I think Timur may be mistaken
> regarding my commenting on ESF, but if I am wrong please provide a
> lore link so I can refresh my memory.
Sorry for misleading you. My mistake, I should have checked the thread
first instead of relying on my memory.
> In this post Timur provides links to his ESF project on GitHub, but no
> patches.
Am I correct in understanding that any proposals and questions I'd
like to discuss with the maintainers and the community should start
with patches? Even if the goal isn't to implement a change right away,
but merely to evaluate the idea.
When I proposed the prototype, it seemed excessive to me to prepare
patches for something that could be "finished" at the idea stage.
On Fri, Dec 12, 2025 at 8:06 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Fri, Dec 12, 2025 at 11:12 AM Stephen Smalley
> <stephen.smalley.work@gmail.com> wrote:
> > On Fri, Dec 12, 2025 at 9:47 AM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
> > >
> > > I’m lucky enough to have already built a working prototype, which I
> > > once offered for review:
> > >
> > > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> > > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/include/uapi/linux/esf
> > >
> > > Less lucky was the reaction I received. Paul Moore was strongly
> > > opposed, as far as I remember. Dr. Greg once said that heat death of
> > > the universe is more likely than this approach being accepted into the
> > > kernel.
> >
> > Not seeing an actual response from Paul in the archives, but did you
> > ever actually post patches to the list?
>
> I was wondering about this too. I searched through my sent mail and
> while it's possible I'm missing some mail, the only conversation I see
> with Timur is an off-list discussion from 2024 regarding changes in
> the upstream kernel to support out-of-tree LSMs. While Timur was
> added to the conversation by someone, I don't see any mail from him in
> that thread. My comments in that thread are consistent with my
> comments in on-list threads from around that same time when
> out-of-tree code was discussed. Here is a snippet from one of my
> responses, which still holds true as far as I'm concerned:
>
> "As stated many times in the past, the LSM framework as
> well as the Linux kernel in general, does not provide the
> same level of consideration to out-of-tree code that it does
> to upstream, mainline code. My policy on this remains the
> same as last time we talked: while I have no goal to make
> things difficult for out-of-tree code, I will not sacrifice
> the continued development and maintenance of existing
> upstream code in favor of out-of-tree code."
>
> Searching for "Timur Chernykh" in all of the lore archives shows some
> BPF related threads and the following LSM thread from June 2024:
>
> https://lore.kernel.org/all/CABZOZnS13-KscVQY0YqqWZsBwmQaKyRO_G=kzCL8zc9jHxAC=A@mail.gmail.com
>
> In this post Timur provides links to his ESF project on GitHub, but no
> patches. I see comments from Stephen, Tetsuo, Casey, and Dr. Greg; I
> did not comment on that thread.
>
> Beyond that, I'm a bit lost. As far as I can remember, and both lore
> and my own sent mail folder appear to support this, I've never
> commented on ESF. At this point I think Timur may be mistaken
> regarding my commenting on ESF, but if I am wrong please provide a
> lore link so I can refresh my memory.
>
> --
> paul-moore.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 17:22 ` Timur Chernykh
@ 2025-12-12 17:37 ` Stephen Smalley
2025-12-12 18:30 ` Paul Moore
0 siblings, 1 reply; 13+ messages in thread
From: Stephen Smalley @ 2025-12-12 17:37 UTC (permalink / raw)
To: Timur Chernykh
Cc: Paul Moore, Casey Schaufler, torvalds, linux-security-module,
greg
On Fri, Dec 12, 2025 at 12:22 PM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
>
> > While Timur was
> > added to the conversation by someone, I don't see any mail from him in
> > that thread.
>
> I probably missed this thread.
>
> > Beyond that, I'm a bit lost. As far as I can remember, and both lore
> > and my own sent mail folder appear to support this, I've never
> > commented on ESF. At this point I think Timur may be mistaken
> > regarding my commenting on ESF, but if I am wrong please provide a
> > lore link so I can refresh my memory.
>
> Sorry for misleading you. My mistake, I should have checked the thread
> first instead of relying on my memory.
>
> > In this post Timur provides links to his ESF project on GitHub, but no
> > patches.
>
> Am I correct in understanding that any proposals and questions I'd
> like to discuss with the maintainers and the community should start
> with patches? Even if the goal isn't to implement a change right away,
> but merely to evaluate the idea.
>
> When I proposed the prototype, it seemed excessive to me to prepare
> patches for something that could be "finished" at the idea stage.
It doesn't seem to require much more effort than creating the
prototype and publishing it on GitHub. "Write for maximum efficiency
of reading" includes avoiding the need to follow links to adequately
evaluate a proposal. Just provide enough code to show what it is you
want to do and why that can't be done (well) today.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 14:47 ` Timur Chernykh
2025-12-12 16:11 ` Stephen Smalley
@ 2025-12-12 18:10 ` Dr. Greg
1 sibling, 0 replies; 13+ messages in thread
From: Dr. Greg @ 2025-12-12 18:10 UTC (permalink / raw)
To: Timur Chernykh; +Cc: Casey Schaufler, torvalds, linux-security-module
On Fri, Dec 12, 2025 at 05:47:24PM +0300, Timur Chernykh wrote:
Good morning again to everyone.
> > eBPF has been accepted because of these restrictions. Without them
> > an eBPF program could very easily reek havoc on the system's behavior.
>
> Unfortunately, these restrictions also make it impossible to use eBPF
> effectively for security. We can certainly discuss the limitations
> themselves if needed, but that???s not the main point I???m trying to make
> here.
>
> > Yes. That's true for many kernel features.
>
> Right, and that's a well-known issue that has no clean solution other
> than backporting. I completely agree with that.
>
> > To the best of my understanding, and I have been working with LSMs
> > and the infrastructure for quite some time, there has never been an
> > out-of-tree LSM API.
>
> I was under the impression that there was something of that sort.
> For example: https://elixir.bootlin.com/linux/v3.10/source/include/linux/security.h#L1688
> It may not have been 'officially' exported, but people who needed it
> simply resolved the symbol through `kallsyms` or a kprobe and used it.
> I'm not claiming that this is good practice, but it did work.
>
> > You will need to provide more detail about why you believe that the
> > integrity of an AV/EDR product cannot be protected.
>
> Consider the case where an attacker already has root access. The first
> thing they typically do is disable or tamper with security tools.
> After that, they establish persistence. Access-control models alone
> are not enough here sometimes they need to be adjusted dynamically,
> based on complex real-time analysis.
>
> > What APIs do you want? It's possible that some exist that you haven't
> > discovered.
> I'm confident I've studied all the mechanisms available for
> real-time system protection. I've even given talks and published
> material on approaches to securing Linux. I may not have covered
> every LSM module in depth, but in practice they simply don't provide
> the capabilities required for modern security workloads.
>
> And I don't want to sound overly categorical, but the current path of
> security development feels like a dead end to me, for several reasons.
> Modern security relies on analyzing hundreds of thousands of events
> per second. eBPF isn't particularly fast to begin with, and on top of
> that it imposes strict technological limitations that prevent
> meaningful in-kernel analysis. Every EDR vendor ends up doing the same
> thing: collecting raw data (which is already difficult) and pushing it
> into userspace for processing. That adds transport overhead on top of
> everything else.
>
> Or, of course, writes a dirty and unsafe kernel module. For me in the
> perfect world seems to it's better to provide an API for userspace.
> It's safe enough and doesn't contain a restriction that eBPF has.
You may be cooking a response to my previous e-mail so a quick comment
to perhaps constructively shape forward conversations.
We looked at the ESF implementation and if we understand the Mac/Apple
approach an API is supplied to register for security events of
interest.
With TSEM we came at the problem from a framework of supporting the
mathematical modeling of security behavior, a 'y = mx + b' type of
approach. With an objective of providing complete flexibility with
respect to how the functional definition for security is generated.
The most universal method of doing this, at least in our view, was to
supply the stream of LSM events and their parameters and let a
modeling engine, either in the kernel or userspace, decide how to
filter for and process the events in order to create the functional
definition.
A userspace orchestrator receives a JSON encoded record for each LSM
event to do with as it pleases. If you feed that into something like
an Elastic or Open Search instance you have a complete compliance
record as to every event that the kernel deemed to have security
implications.
Obviously a lot of events can be generated. We currently support
policy specific screening of events with standard kernel modules and
doing this with eBPF is our next step. From an eBPF perspective this
is much more of a lightweight approach than attempting to code for
specific security analytics or behavior analysis.
For example, we implemented the rough equivalent of userspace IMA by
in kernel filtering for security_file_open events invoked with
administrative privileges.
The LSM API isn't considered stable but the reality of security
telemetry in this day and age is that security vendors have to code
for specific LTS kernel releases. Even with a 'stable' release there
are issues with telemetry acquisition, CrowdStike ended up blue
screening RHEL not too long ago on one of their kernel point releases.
We would be interested as to what you see to be the challenges, from
an EDR vendor perspective, with this approach.
FWIW. If you namespace the security event stream you end up with the
ability to much more precisely model whether or not a workload is
exhibiting aberrant behavior, while decreasing the event flux rate.
> > Please propose the interfaces you want. An API definition would be
> > good, and patches even better.
>
> I'm lucky enough to have already built a working prototype, which I
> once offered for review:
>
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/include/uapi/linux/esf
>
> Less lucky was the reaction I received. Paul Moore was strongly
> opposed, as far as I remember. Dr. Greg once said that heat death of
> the universe is more likely than this approach being accepted into the
> kernel.
> I don't want to 'poke old wounds', but the design I proposed was
> largely inspired by Apple's Endpoint Security Framework
> (originally 'mac policies'):
> https://developer.apple.com/documentation/endpointsecurity
> In my view, it is an exemplary model. The kernel provides a
> consistent stream of events to userspace, hides the complexity of
> data collection and transport, and ensures fault-tolerance if
> userspace fails to process events quickly enough. Audit, for
> comparison, simply hangs the system if its buffer overflows. ESF
> does not have that problem and even lets userspace control the event
> stream.
> > There are numerous cases where LSMs have mitigated attacks. One case:
>
> And there are thousands of cases where LSMs did not. Most companies
> never disclose successful breaches because of NDAs and fear of
> investor backlash. The few disclosures we see typically appear only
> when it???s safe for the company, or when they specifically want to
> appear transparent.
>
> > Your assertion that LSMs don't provide "real" value is not substantiated
> > in the literature.
>
> Which literature do you mean?
> As I mentioned above, public reports about successful attacks are rare.
>
> Maybe I poorly delivered my point. I'll try again.
>
> The strongest protection on Linux today is provided by proprietary
> security vendors. Unfortunately, they often achieve this using
> kernel-module ???hacks??? rather than LSMs. It???s not because they prefer
> hacks ??? it???s because Linux doesn???t offer proper APIs to implement the
> required functionality.
>
> > Each of the existing LSMs addresses one or more real world issues. I would
> > hardly call any of the Linux security mechanisms, from mode bits and setuid
> > through SELinux, landlock and EVM, "theoretical".
>
> In practice, most LSM modules are either disabled or misconfigured on
> real servers, and the responsibility falls on AV/EDR vendors. As
> someone who has been building such systems professionally for years, I
> can say with confidence: there is no clean, reliable mechanism
> available today.
> As much as I love Linux, its security and its friendliness toward
> security vendors are its weak side.
>
> In the real world, deploying a modern security product means dealing
> with endless pain: missing kernel headers (sometimes there isn???t even
> Internet access to download them), unstable kernel APIs, lack of BTF,
> constant fights with the eBPF verifier, and all the other limitations
> of eBPF.
>
> To be honest, many developers ??? and I???m taking the liberty to speak
> broadly here ??? would simply like to see something similar to Apple???s
> ESF available on Linux.
>
> Best regards,
> Timur Chernykh
>
> On Wed, Dec 10, 2025 at 11:08???PM Casey Schaufler <casey@schaufler-ca.com> wrote:
> >
> > On 12/9/2025 4:15 PM, Timur Chernykh wrote:
> > > Hello Linus,
> > >
> > > I???m writing to ask for your opinion. What do you think about Linux???s
> > > current readiness for security-focused commercial products? I???m
> > > particularly interested in several areas.
> > >
> > > First, in today???s 3rd-party (out-of-tree) EDR development ??? EDR being
> > > the most common commercial class of security products ??? eBPF has
> > > effectively become the main option. Yet eBPF is extremely restrictive.
> >
> > eBPF has been accepted because of these restrictions. Without them
> > an eBPF program could very easily reek havoc on the system's behavior.
> >
> > > It is not possible to write fully expressive real-time analysis code:
> > > the verifier is overly strict,
> >
> > This is of necessity. It is the only protection against malicious and
> > badly written eBPF programs.
> >
> > > non-deterministic loops are not
> > > allowed,
> >
> > Without this restriction denial of service attacks become trivial.
> >
> > > and older kernels lack BTF support.
> >
> > Yes. That's true for many kernel features.
> >
> > > These issues create real
> > > limitations.
> > >
> > > Second, the removal of the out-of-tree LSM API in the 4.x kernel
> > > series caused significant problems for many AV/EDR vendors. I was
> > > unable to find an explanation in the mailing lists that convincingly
> > > justified that decision.
> >
> > To the best of my understanding, and I have been working with LSMs
> > and the infrastructure for quite some time, there has never been an
> > out-of-tree LSM API. We have been very clear that the LSM interfaces
> > are fluid. We have also been very clear that supporting out-of-tree
> > security modules is not a requirement. There are members of the
> > community who would prefer that they be completely prohibited.
> >
> >
> > > The next closest mechanism, fanotify, was a genuine improvement.
> > > However, it does not allow an AV/EDR vendor to protect the integrity
> > > of its own product. Is Linux truly expecting modern AV/EDR solutions
> > > to rely on fanotify alone?
> >
> > You will need to provide more detail about why you believe that the
> > integrity of an AV/EDR product cannot be protected.
> >
> > > My main question is: what are the future plans?
> >
> > Security remains a dynamic technology. There are quite a few plans,
> > from a variety of sources, with no shortage of security goals. Trying
> > to keep up with the concern/crisis of the day is more than sufficient
> > to consume the resources committed to Linux security.
> >
> > > Linux provides very
> > > few APIs for security and dynamic analysis.
> >
> > What APIs do you want? It's possible that some exist that you haven't
> > discovered.
> >
> > > eBPF is still immature,
> >
> > And it will be for some time. That is, until it is mature.
> >
> > > fanotify is insufficient,
> >
> > Without specifics it is quite difficult to do anything about that.
> > And, as we like to say, patches are welcome.
> >
> > > and driver workarounds that bypass kernel
> > > restrictions are risky ??? they introduce both stability and security
> > > problems.
> >
> > Yes. Don't do that.
> >
> > > At the same time, properly implemented in-tree LSMs are not
> > > inherently dangerous and remain the safer, supported path for
> > > extending security functionality. Without safe, supported interfaces,
> > > however, commercial products struggle to be competitive. At the
> > > moment, macOS with its Endpoint Security Framework is significantly
> > > ahead.
> >
> > Please propose the interfaces you want. An API definition would be
> > good, and patches even better.
> >
> > > Yes, the kernel includes multiple in-tree LSM modules, but in practice
> > > SELinux does not simplify operations ??? it often complicates them,
> > > despite its long-standing presence. Many of the other LSMs are rarely
> > > used in production. As an EDR developer, I seldom encounter them, and
> > > when I do, they usually provide little practical value. Across
> > > numerous real-world server intrusions, none of these LSM modules have
> > > meaningfully prevented attacks, despite many years of kernel
> > > development.
> >
> > There are numerous cases where LSMs have mitigated attacks. One case:
> >
> > https://mihail-milev.medium.com/mitigating-malware-risks-with-selinux-e37cf1db7c56
> >
> > Your assertion that LSMs don't provide "real" value is not substantiated
> > in the literature.
> >
> > > Perhaps it is time for Linux to focus on more than a theoretical model
> > > of security.
> >
> > Each of the existing LSMs addresses one or more real world issues. I would
> > hardly call any of the Linux security mechanisms, from mode bits and setuid
> > through SELinux, landlock and EVM, "theoretical".
> >
> > > P.S.
> > > Everything above reflects only my personal opinion. I would greatly
> > > appreciate your response and any criticism you may have.
> >
> > We are always ready to improve the security infrastructure of Linux.
> > If it does not meet your needs, help us improve it. The best way to
> > accomplish this is to provide the changes required.
> >
> > > Best regards,
> > > Timur Chernykh
Have a good weekend.
As always,
Dr. Greg
The Quixote Project - Flailing at the Travails of Cybersecurity
https://github.com/Quixote-Project
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 17:37 ` Stephen Smalley
@ 2025-12-12 18:30 ` Paul Moore
0 siblings, 0 replies; 13+ messages in thread
From: Paul Moore @ 2025-12-12 18:30 UTC (permalink / raw)
To: Stephen Smalley
Cc: Timur Chernykh, Casey Schaufler, torvalds, linux-security-module,
greg
On Fri, Dec 12, 2025 at 12:37 PM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
> On Fri, Dec 12, 2025 at 12:22 PM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
> >
> > > While Timur was
> > > added to the conversation by someone, I don't see any mail from him in
> > > that thread.
> >
> > I probably missed this thread.
> >
> > > Beyond that, I'm a bit lost. As far as I can remember, and both lore
> > > and my own sent mail folder appear to support this, I've never
> > > commented on ESF. At this point I think Timur may be mistaken
> > > regarding my commenting on ESF, but if I am wrong please provide a
> > > lore link so I can refresh my memory.
> >
> > Sorry for misleading you. My mistake, I should have checked the thread
> > first instead of relying on my memory.
> >
> > > In this post Timur provides links to his ESF project on GitHub, but no
> > > patches.
> >
> > Am I correct in understanding that any proposals and questions I'd
> > like to discuss with the maintainers and the community should start
> > with patches? Even if the goal isn't to implement a change right away,
> > but merely to evaluate the idea.
> >
> > When I proposed the prototype, it seemed excessive to me to prepare
> > patches for something that could be "finished" at the idea stage.
>
> It doesn't seem to require much more effort than creating the
> prototype and publishing it on GitHub. "Write for maximum efficiency
> of reading" includes avoiding the need to follow links to adequately
> evaluate a proposal. Just provide enough code to show what it is you
> want to do and why that can't be done (well) today.
To add to what Stephen already said, if you are serious about
submitting a new LSM upstream, we do have some documented guidance
linked off the "SECURITY SUBSYSTEM" entry in the MAINTAINERS file. To
save some time/searching, the direct link is below:
https://github.com/LinuxSecurityModule/kernel/blob/main/README.md#new-lsms
--
paul-moore.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 5:45 ` Dr. Greg
@ 2025-12-12 23:43 ` Casey Schaufler
2025-12-15 4:55 ` Dr. Greg
0 siblings, 1 reply; 13+ messages in thread
From: Casey Schaufler @ 2025-12-12 23:43 UTC (permalink / raw)
To: Dr. Greg, Timur Chernykh; +Cc: torvalds, linux-security-module, Casey Schaufler
On 12/11/2025 9:45 PM, Dr. Greg wrote:
> On Wed, Dec 10, 2025 at 03:15:39AM +0300, Timur Chernykh wrote:
>
> Good morning Timur, I hope this note finds your week having gone well.
>
>> Hello Linus,
>>
>> I'm writing to ask for your opinion. What do you think about Linux's
>> current readiness for security-focused commercial products? I'm
>> particularly interested in several areas.
> I don't expect you will receive an answer.
>
> Based on his previous comments and long standing position on this
> issue, I believe it can be fairly stated that he looks at the LSM as
> an unnecessary evil.
>
> So in his absence, some 'in loco parentis' reflections on the issues
> you raise.
>
> I've been advised, more than once, that in this day and age, no one is
> interested in reading more than a two sentence paragraph, so a short
> response to your issues here and a bit more detail for anyone who
> wants to read more, at the end.
>
> There is active art available to address the shortcomings you outline
> in your post below. Our TSEM LSM was designed to service the
> realitities of the modern security environment and where it is going.
> In a manner that doesn't provide any restrictions on how 'security'
> can be implemented.
>
> We've done four releases over three years and we believe an unbiased
> observer would conclude they have received no substantive technical
> review that would support interest in upstream integration.
Stop. Really, I mean it. I put significant effort into trying to teach
you how to submit a patch set that could be reviewed. You ignored it.
I can't speak to what an "unbiased observer" would conclude because
your behavior has certainly left me with bias. Rather than writing
full length novels about why you submitted patches the way you've
done it you might consider heeding the advice. Grrr.
> The challenge is that the security gatekeepers view LSM submissions
> through a lens of whether or not the LSM implements security
> consistent with what they believe is security.
While there is some truth to that, we're really quite flexible.
What we need to see is that there is some sort of security model,
and that the proposed code implements it. We also need to be able
to examine the proposed code to see that it implements the model.
You have rejected all suggestions about how to make your proposal
reviewable.
> Those views are
> inconsistent with the realities of the modern security market,
Oh, stop it. Look at how many LSMs have been added over the past
few years. Sandboxing and application controlled security are
"modern" security concepts that were unheard of when the LSM was
introduced. As I said before, security is a dynamic technology.
If it were not, we'd have a Bell & LaPadula kernel config option
instead of an LSM infrastructure.
> a
> market that that is now predicated on detection rather than
> enforcement. A trend that will only accelerate with advancements in
> machine learning and AI.
Without the underlying access controls detection is rather pointless.
> It is worth noting that the history of the technology industry is
> littered with examples of technology incumbents usually missing
> disruptive innovation.
The technology industry is not unique on this. Acme Buggy Whips, inc.
> This restriction on suitability is actually inconsistent with Linus'
> stated position on how Linux sub-systems can be used, as expressed in
> his comment in the following post.
>
> https://lore.kernel.org/lkml/CAHk-=wgLbz1Bm8QhmJ4dJGSmTuV5w_R0Gwvg5kHrYr4Ko9dUHQ@mail.gmail.com/
>
> So the problem is not technical, it is a political eco-system problem.
>
> So, big picture, that is the challenge facing resolution of your
> concerns.
>
> Apologies to everyone about the paragraph/sentence overflow and any
> afront to sensibilities.
>
> More detail below if anyone is interested.
>
>> First, in today's 3rd-party (out-of-tree) EDR development EDR
>> being the most common commercial class of security products eBPF
>> has effectively become the main option. Yet eBPF is extremely
>> restrictive. It is not possible to write fully expressive real-time
>> analysis code: the verifier is overly strict, non-deterministic
>> loops are not allowed, and older kernels lack BTF support. These
>> issues create real limitations.
>>
>> Second, the removal of the out-of-tree LSM API in the 4.x kernel
>> series caused significant problems for many AV/EDR vendors. I was
>> unable to find an explanation in the mailing lists that convincingly
>> justified that decision.
>>
>> The next closest mechanism, fanotify, was a genuine improvement.
>> However, it does not allow an AV/EDR vendor to protect the integrity
>> of its own product. Is Linux truly expecting modern AV/EDR solutions
>> to rely on fanotify alone?
>>
>> My main question is: what are the future plans? Linux provides very
>> few APIs for security and dynamic analysis. eBPF is still immature,
>> fanotify is insufficient, and driver workarounds that bypass kernel
>> restrictions are risky they introduce both stability and security
>> problems. At the same time, properly implemented in-tree LSMs are not
>> inherently dangerous and remain the safer, supported path for
>> extending security functionality. Without safe, supported interfaces,
>> however, commercial products struggle to be competitive. At the
>> moment, macOS with its Endpoint Security Framework is significantly
>> ahead.
>>
>> Yes, the kernel includes multiple in-tree LSM modules, but in
>> practice SELinux does not simplify operations it often complicates
>> them, despite its long-standing presence. Many of the other LSMs are
>> rarely used in production. As an EDR developer, I seldom encounter
>> them, and when I do, they usually provide little practical
>> value. Across numerous real-world server intrusions, none of these
>> LSM modules have meaningfully prevented attacks, despite many years
>> of kernel development.
>>
>> Perhaps it is time for Linux to focus on more than a theoretical model
>> of security.
> The heart of the political eco-system challenge is best expressed by a
> quote from Kyle Moffett, in which he stated that security should only
> be developed and implemented by experts. Unfortunately that view is
> inconsistent with the current state of the technology industry.
Glad to hear I'm not an expert! - Not!
> Classical security practititioners will defend complex subject/object
> architectures with: "Google uses SeLinux for Android security".
Yea gads. Subject/Object is about as simple as it gets. Look at Smack.
> Our response to that is that the world doesn't have a security problem
> because Google lacks sufficient resources to implement anything it
> desires to implement, regardless of the development and maintenance
> input costs.
>
> Unfortunately, that luxury is inconsistent with the rest of the
> software development world that doesn't enjoy a 3.8 trillion dollar
> market capitalization.
>
> The world simply lacks enough experts to make the 'security only by
> experts' model work.
>
> Today, the fastest way to a product is to grab Linux and a development
> team and write software for hardware that is now completely
> commoditized. Everyone knows that security is not one of the
> fundamental project predicates in this model.
>
> Both NIST and DHS/CISA are officially on record as indicating that
> security needs to start with and be baked in through the development
> process. One of the objectives of TSEM was to provide a framework for
> enabling this concept for the implementation of analysis and mandatory
> behavior controls for software workloads.
>
> A second fundamental problem is that the world has moved, in large
> part, to containerized execution workloads. The Linux LSM, in its
> current form, doesn't effectively support the application of workload
> specific security policies.
>
> Further complicating this issue is the fact that LSM 'stacking'
> requires reasoning as to what a final security policy will be when
> multiple different security architectures/policies get to decide on
> the outcome of a security event/hook. The concept of least surprise
> would suggest the need for stacking to have idempotency, in other
> words, the order in which LSM event consumers are called shouldn't
> influence the effective policy, but this is generally acknowledged as
> not being the case with 'stacking'.
Any other approach, and they have been considered, fails miserably
and introduces a host of complications. Not to mention performance
de-optimization.
> So we designed TSEM to provide an alternative, not a replacement, but
> an alternative to how developers and system administrators can develop
> and apply security policy, including integrity controls.
>
> TSEM is an LSM that implements containerized security infrastructure
> rather than security policy. It is designed around the concept of a
> security orchestrator that can execute security isolated workloads and
> receive the LSM events and their parameters from that workload and
> process them in any manner it wishes.
I shan't repeat the objections that have been raised, but I will
point out that you have done nothing to address them.
> For example: A Docker/Kubernetes container can be run and all of the
> security events by that workload exported up into an OpenSearch or
> ElasticSearch instance for anomaly detection and analysis.
>
> So an EDR implemented on top of this has visibility into all of the
> security events and their characteristics that are deemed security
> relevant by the kernel developers.
>
> One of the pushbacks is that this can lead to asynchronous security
> decisions, but as you note, that is the model that the commercial
> security industry and the consumers of its products has embraced,
> particularly in light of the advancements coming out of the AI
> industry, detection rather than enforcement.
>
> If synchronous enforcement is required TSEM provides that as well,
> including the use of standard kernel modules to implement analysis and
> response to the LSM hooks. Internally we have implemented other LSM's
> such as Tomoyo and IMA as loadable modules that can support multiple
> and independent workload policies.
>
> If you or other EDR vendors are interested, we would be more than
> happy to engage in conversations as to how to improve the capabilities
> of this type of architecture, as an alternative to what is currently
> available in Linux, which as you note, has significant limitations.
>
>> Everything above reflects only my personal opinion. I would greatly
>> appreciate your response and any criticism you may have.
> As I mentioned at the outset, you are unlikely to hear anything.
>
> For the necessary Linux infrastructure improvements to emerge we
> believe there is the need to develop and engage a community effort
> that independently pursues the advancements that are necessary,
> particularly those that enable Linux to implement first class AI based
> security controls.
>
> We believe that only this will result in sufficient 'market pull' at the
> distribution level to help shape upstreaming decisions.
>
> Absent that, it is likely that Linux will continue to implement what
> has failed to work in the past in the hope that it will somehow work
> in the future.
Wow. calling Linux a failure is a bit of a stretch, don't you think?
> Comments and criticism welcome, we have had plenty of experience with
> the latter.... :-)
We've been over these issues many times. Go back and make some changes to
your approach.
>> Best regards,
>> Timur Chernykh
> Best wishes for the success of your work and a pleasant holiday season.
>
> As always,
> Dr. Greg
>
> The Quixote Project - Flailing at the Travails of Cybersecurity
> https://github.com/Quixote-Project
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-12 23:43 ` Casey Schaufler
@ 2025-12-15 4:55 ` Dr. Greg
2025-12-15 17:44 ` Casey Schaufler
0 siblings, 1 reply; 13+ messages in thread
From: Dr. Greg @ 2025-12-15 4:55 UTC (permalink / raw)
To: Casey Schaufler; +Cc: Timur Chernykh, torvalds, linux-security-module
On Fri, Dec 12, 2025 at 03:43:07PM -0800, Casey Schaufler wrote:
Good morning Casey, pleasant as always to hear from you.
> On 12/11/2025 9:45 PM, Dr. Greg wrote:
> > On Wed, Dec 10, 2025 at 03:15:39AM +0300, Timur Chernykh wrote:
> >
> > Good morning Timur, I hope this note finds your week having gone well.
> >
> >> Hello Linus,
> >>
> >> I'm writing to ask for your opinion. What do you think about Linux's
> >> current readiness for security-focused commercial products? I'm
> >> particularly interested in several areas.
> > I don't expect you will receive an answer.
> >
> > Based on his previous comments and long standing position on this
> > issue, I believe it can be fairly stated that he looks at the LSM as
> > an unnecessary evil.
> >
> > So in his absence, some 'in loco parentis' reflections on the issues
> > you raise.
> >
> > I've been advised, more than once, that in this day and age, no one is
> > interested in reading more than a two sentence paragraph, so a short
> > response to your issues here and a bit more detail for anyone who
> > wants to read more, at the end.
> >
> > There is active art available to address the shortcomings you outline
> > in your post below. Our TSEM LSM was designed to service the
> > realitities of the modern security environment and where it is going.
> > In a manner that doesn't provide any restrictions on how 'security'
> > can be implemented.
> >
> > We've done four releases over three years and we believe an unbiased
> > observer would conclude they have received no substantive technical
> > review that would support interest in upstream integration.
> Stop. Really, I mean it. I put significant effort into trying to teach
> you how to submit a patch set that could be reviewed. You ignored it.
> I can't speak to what an "unbiased observer" would conclude because
> your behavior has certainly left me with bias. Rather than writing
> full length novels about why you submitted patches the way you've
> done it you might consider heeding the advice. Grrr.
No, we are not going to stop, see immediately below.
> > The challenge is that the security gatekeepers view LSM submissions
> > through a lens of whether or not the LSM implements security
> > consistent with what they believe is security.
>
> While there is some truth to that, we're really quite flexible.
> What we need to see is that there is some sort of security model,
> and that the proposed code implements it. We also need to be able
> to examine the proposed code to see that it implements the model.
> You have rejected all suggestions about how to make your proposal
> reviewable.
No we didn't.
I've managed engineers before, I've never found a successful strategy
to be:
"Go off and do what I told you to do".
Without some kind of explanation and dialogue.
The Linux community has a process problem with the LSM and the
security sub-system that needs to be addressed. Personally, I now
have the cycles to address it.
We assume that if Timur elects to port the Apple security API (Timur's
ESF implementation) into the kernel, he will face the same issue that
concerns us.
When Paul Moore came on-board three years ago as the security
maintainer, he was immediately VERY prescriptive with respect to
establishing policies and guidelines that he wanted followed for LSM
submissions.
We see that he sent those guidelines to Timur in case he was
interested in submitting ESF as a new LSM. Here is the link for those
who haven't seen it.
https://github.com/LinuxSecurityModule/kernel/blob/main/README.md#new-lsms
What those guidelines are missing are prescriptive practices for
bringing a major body of virgin code into the security review process.
As we did with TSEM, and presumably Timur will have to do with ESF, if
he moves forward with that initiative.
For the benefit of those following this conversation and to close the
record in this thread from our perspective, as tedious as that might
be.
From an architectural perspective, TSEM consists of a series of
compilation units, each dedicated to a specific functional role needed
to support the LSM. We used a single include file that holds all of
the enumerations and structure definitions that are referenced by
every compilation unit.
We chose a submssion model of introducing this include file as a
single patch in the series, after a complete sub-system documentation
patch. Introduction of virgin sub-systems isn't really common but this
strategy was consistent with what we had previously observed.
In addition, we believe that Knuth was quoted at one time about an
understanding of a program flows from an understanding of its data
structures.
You reviewed about half of one compilation unit and said you didn't
have time for anymore. For the record, we made the changes that you
requested and responded to your other concerns for that unit.
Your primary issue, which you have repeated often, as you did here,
was with the fact that we had a single header file in the patch series
with all of the LSM global definitions. You indicated that the patch
series should be broken up in order to 'tell a story' by including the
structure and enumeration definitions with the code.
Fair enough, you are a valued voice in the Linux security community,
but your right pinky finger is not on the RETURN key that sends
security sub-system pull requests to Linus, Paul Moore's is.
If anyone chooses to review the new LSM guidelines, they will note
that it never makes mention of the fact that a patch series should
'tell a story' and/or how that should be done.
So, before we committed a ton of development time into what could very
well end up being a tag team match to try and please two maintainers,
we reached out to Paul for advice on how he would like to see a virgin
patch series structured.
His response was that TSEM had significant issues, the least of which
was the organization of the patch series. We repeatedly asked him to
detail what those problems were, as by his own admission he had only
read the documentation and not offered any review comments on the code
itself.
The only thing we received was that, in certain security behavior
modeling configurations enabled by TSEM, the security response would
be asynchronous from the event or events that triggered it.
We attempted to pursue further dialogue surrounding this. The last
response, at least a year ago now, was that he didn't think he needed
to respond to us because of our 'tone'. If anyone hasn't noticed, our
'tone' is driven by the fact that we tend to not be easily bul*ted.
So, in the interest of furthering Linux security development while at
the same time promoting homeostatic harmony in the community, we will
be sending out a request to the security list to formally address this
issue with new LSM's.
It obviously doesn't make sense to waste your or other people's time
addressing the other issues you have below, other than to agree that
we disagree on where security is headed and how Linux needs to
respond.
> > Those views are
> > inconsistent with the realities of the modern security market,
>
> Oh, stop it. Look at how many LSMs have been added over the past
> few years. Sandboxing and application controlled security are
> "modern" security concepts that were unheard of when the LSM was
> introduced. As I said before, security is a dynamic technology.
> If it were not, we'd have a Bell & LaPadula kernel config option
> instead of an LSM infrastructure.
>
> > a
> > market that that is now predicated on detection rather than
> > enforcement. A trend that will only accelerate with advancements in
> > machine learning and AI.
>
> Without the underlying access controls detection is rather pointless.
>
> > It is worth noting that the history of the technology industry is
> > littered with examples of technology incumbents usually missing
> > disruptive innovation.
>
> The technology industry is not unique on this. Acme Buggy Whips, inc.
>
> > This restriction on suitability is actually inconsistent with Linus'
> > stated position on how Linux sub-systems can be used, as expressed in
> > his comment in the following post.
> >
> > https://lore.kernel.org/lkml/CAHk-=wgLbz1Bm8QhmJ4dJGSmTuV5w_R0Gwvg5kHrYr4Ko9dUHQ@mail.gmail.com/
> >
> > So the problem is not technical, it is a political eco-system problem.
> >
> > So, big picture, that is the challenge facing resolution of your
> > concerns.
> >
> > Apologies to everyone about the paragraph/sentence overflow and any
> > afront to sensibilities.
> >
> > More detail below if anyone is interested.
> >
> >> First, in today's 3rd-party (out-of-tree) EDR development EDR
> >> being the most common commercial class of security products eBPF
> >> has effectively become the main option. Yet eBPF is extremely
> >> restrictive. It is not possible to write fully expressive real-time
> >> analysis code: the verifier is overly strict, non-deterministic
> >> loops are not allowed, and older kernels lack BTF support. These
> >> issues create real limitations.
> >>
> >> Second, the removal of the out-of-tree LSM API in the 4.x kernel
> >> series caused significant problems for many AV/EDR vendors. I was
> >> unable to find an explanation in the mailing lists that convincingly
> >> justified that decision.
> >>
> >> The next closest mechanism, fanotify, was a genuine improvement.
> >> However, it does not allow an AV/EDR vendor to protect the integrity
> >> of its own product. Is Linux truly expecting modern AV/EDR solutions
> >> to rely on fanotify alone?
> >>
> >> My main question is: what are the future plans? Linux provides very
> >> few APIs for security and dynamic analysis. eBPF is still immature,
> >> fanotify is insufficient, and driver workarounds that bypass kernel
> >> restrictions are risky they introduce both stability and security
> >> problems. At the same time, properly implemented in-tree LSMs are not
> >> inherently dangerous and remain the safer, supported path for
> >> extending security functionality. Without safe, supported interfaces,
> >> however, commercial products struggle to be competitive. At the
> >> moment, macOS with its Endpoint Security Framework is significantly
> >> ahead.
> >>
> >> Yes, the kernel includes multiple in-tree LSM modules, but in
> >> practice SELinux does not simplify operations it often complicates
> >> them, despite its long-standing presence. Many of the other LSMs are
> >> rarely used in production. As an EDR developer, I seldom encounter
> >> them, and when I do, they usually provide little practical
> >> value. Across numerous real-world server intrusions, none of these
> >> LSM modules have meaningfully prevented attacks, despite many years
> >> of kernel development.
> >>
> >> Perhaps it is time for Linux to focus on more than a theoretical model
> >> of security.
> > The heart of the political eco-system challenge is best expressed by a
> > quote from Kyle Moffett, in which he stated that security should only
> > be developed and implemented by experts. Unfortunately that view is
> > inconsistent with the current state of the technology industry.
>
> Glad to hear I'm not an expert! - Not!
>
> > Classical security practititioners will defend complex subject/object
> > architectures with: "Google uses SeLinux for Android security".
>
> Yea gads. Subject/Object is about as simple as it gets. Look at Smack.
>
> > Our response to that is that the world doesn't have a security problem
> > because Google lacks sufficient resources to implement anything it
> > desires to implement, regardless of the development and maintenance
> > input costs.
> >
> > Unfortunately, that luxury is inconsistent with the rest of the
> > software development world that doesn't enjoy a 3.8 trillion dollar
> > market capitalization.
> >
> > The world simply lacks enough experts to make the 'security only by
> > experts' model work.
> >
> > Today, the fastest way to a product is to grab Linux and a development
> > team and write software for hardware that is now completely
> > commoditized. Everyone knows that security is not one of the
> > fundamental project predicates in this model.
> >
> > Both NIST and DHS/CISA are officially on record as indicating that
> > security needs to start with and be baked in through the development
> > process. One of the objectives of TSEM was to provide a framework for
> > enabling this concept for the implementation of analysis and mandatory
> > behavior controls for software workloads.
> >
> > A second fundamental problem is that the world has moved, in large
> > part, to containerized execution workloads. The Linux LSM, in its
> > current form, doesn't effectively support the application of workload
> > specific security policies.
> >
> > Further complicating this issue is the fact that LSM 'stacking'
> > requires reasoning as to what a final security policy will be when
> > multiple different security architectures/policies get to decide on
> > the outcome of a security event/hook. The concept of least surprise
> > would suggest the need for stacking to have idempotency, in other
> > words, the order in which LSM event consumers are called shouldn't
> > influence the effective policy, but this is generally acknowledged as
> > not being the case with 'stacking'.
>
> Any other approach, and they have been considered, fails miserably
> and introduces a host of complications. Not to mention performance
> de-optimization.
>
> > So we designed TSEM to provide an alternative, not a replacement, but
> > an alternative to how developers and system administrators can develop
> > and apply security policy, including integrity controls.
> >
> > TSEM is an LSM that implements containerized security infrastructure
> > rather than security policy. It is designed around the concept of a
> > security orchestrator that can execute security isolated workloads and
> > receive the LSM events and their parameters from that workload and
> > process them in any manner it wishes.
>
> I shan't repeat the objections that have been raised, but I will
> point out that you have done nothing to address them.
>
> > For example: A Docker/Kubernetes container can be run and all of the
> > security events by that workload exported up into an OpenSearch or
> > ElasticSearch instance for anomaly detection and analysis.
> >
> > So an EDR implemented on top of this has visibility into all of the
> > security events and their characteristics that are deemed security
> > relevant by the kernel developers.
> >
> > One of the pushbacks is that this can lead to asynchronous security
> > decisions, but as you note, that is the model that the commercial
> > security industry and the consumers of its products has embraced,
> > particularly in light of the advancements coming out of the AI
> > industry, detection rather than enforcement.
> >
> > If synchronous enforcement is required TSEM provides that as well,
> > including the use of standard kernel modules to implement analysis and
> > response to the LSM hooks. Internally we have implemented other LSM's
> > such as Tomoyo and IMA as loadable modules that can support multiple
> > and independent workload policies.
> >
> > If you or other EDR vendors are interested, we would be more than
> > happy to engage in conversations as to how to improve the capabilities
> > of this type of architecture, as an alternative to what is currently
> > available in Linux, which as you note, has significant limitations.
> >
> >> Everything above reflects only my personal opinion. I would greatly
> >> appreciate your response and any criticism you may have.
> > As I mentioned at the outset, you are unlikely to hear anything.
> >
> > For the necessary Linux infrastructure improvements to emerge we
> > believe there is the need to develop and engage a community effort
> > that independently pursues the advancements that are necessary,
> > particularly those that enable Linux to implement first class AI based
> > security controls.
> >
> > We believe that only this will result in sufficient 'market pull' at the
> > distribution level to help shape upstreaming decisions.
> >
> > Absent that, it is likely that Linux will continue to implement what
> > has failed to work in the past in the hope that it will somehow work
> > in the future.
>
> Wow. calling Linux a failure is a bit of a stretch, don't you think?
>
> > Comments and criticism welcome, we have had plenty of experience with
> > the latter.... :-)
>
> We've been over these issues many times. Go back and make some changes to
> your approach.
>
> >> Best regards,
> >> Timur Chernykh
> > Best wishes for the success of your work and a pleasant holiday season.
> >
> > As always,
> > Dr. Greg
> >
> > The Quixote Project - Flailing at the Travails of Cybersecurity
> > https://github.com/Quixote-Project
> >
Best wishes for a productive week and a pleasant holiday season.
As always,
Dr. Greg
The Quixote Project - Flailing at the Travails of Cybersecurity
https://github.com/Quixote-Project
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: An opinion about Linux security
2025-12-15 4:55 ` Dr. Greg
@ 2025-12-15 17:44 ` Casey Schaufler
0 siblings, 0 replies; 13+ messages in thread
From: Casey Schaufler @ 2025-12-15 17:44 UTC (permalink / raw)
To: Dr. Greg; +Cc: Timur Chernykh, torvalds, linux-security-module
On 12/14/2025 8:55 PM, Dr. Greg wrote:
> On Fri, Dec 12, 2025 at 03:43:07PM -0800, Casey Schaufler wrote:
>
> Good morning Casey, pleasant as always to hear from you.
>
>> On 12/11/2025 9:45 PM, Dr. Greg wrote:
>>> On Wed, Dec 10, 2025 at 03:15:39AM +0300, Timur Chernykh wrote:
>>>
>>> Good morning Timur, I hope this note finds your week having gone well.
>>>
>>>> Hello Linus,
>>>>
>>>> I'm writing to ask for your opinion. What do you think about Linux's
>>>> current readiness for security-focused commercial products? I'm
>>>> particularly interested in several areas.
>>> I don't expect you will receive an answer.
>>>
>>> Based on his previous comments and long standing position on this
>>> issue, I believe it can be fairly stated that he looks at the LSM as
>>> an unnecessary evil.
>>>
>>> So in his absence, some 'in loco parentis' reflections on the issues
>>> you raise.
>>>
>>> I've been advised, more than once, that in this day and age, no one is
>>> interested in reading more than a two sentence paragraph, so a short
>>> response to your issues here and a bit more detail for anyone who
>>> wants to read more, at the end.
>>>
>>> There is active art available to address the shortcomings you outline
>>> in your post below. Our TSEM LSM was designed to service the
>>> realitities of the modern security environment and where it is going.
>>> In a manner that doesn't provide any restrictions on how 'security'
>>> can be implemented.
>>>
>>> We've done four releases over three years and we believe an unbiased
>>> observer would conclude they have received no substantive technical
>>> review that would support interest in upstream integration.
>> Stop. Really, I mean it. I put significant effort into trying to teach
>> you how to submit a patch set that could be reviewed. You ignored it.
>> I can't speak to what an "unbiased observer" would conclude because
>> your behavior has certainly left me with bias. Rather than writing
>> full length novels about why you submitted patches the way you've
>> done it you might consider heeding the advice. Grrr.
> No, we are not going to stop, see immediately below.
Rather than addressing the issues you again explain, in great detail,
why you're right about everything. And I never hit the enter key with my
pinky.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-12-15 17:55 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-12-10 0:15 An opinion about Linux security Timur Chernykh
2025-12-10 20:08 ` Casey Schaufler
2025-12-12 14:47 ` Timur Chernykh
2025-12-12 16:11 ` Stephen Smalley
2025-12-12 17:06 ` Paul Moore
2025-12-12 17:22 ` Timur Chernykh
2025-12-12 17:37 ` Stephen Smalley
2025-12-12 18:30 ` Paul Moore
2025-12-12 18:10 ` Dr. Greg
2025-12-12 5:45 ` Dr. Greg
2025-12-12 23:43 ` Casey Schaufler
2025-12-15 4:55 ` Dr. Greg
2025-12-15 17:44 ` Casey Schaufler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox