Yet another vision of Linux security | Endpoint Security Framework

Yet another vision of Linux security | Endpoint Security Framework

[Timur Chernykh]

Hello!

I'm here for yours' opinions.

The modern Linux, in fact, does not provide convenient options to
write AV/EDR software with real-time system events analysis as far as
unified system security events. For now we have only inconsistent
self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
cool technology, but even CO-RE sometimes works as not expected or
gets broken, for example:
https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
of a huge amount of the same hacks) looks a bit "crutchy".

As an EDR developer, I have an idea how to fix this situation, using a
unified endpoint security framework (ESF) placed in the kernel. Its
must:
- Provide unified and consistent security events;
- API to communicate with kernel:
 - To send program-defined events (as audit has);
 - Read events from kernel;
- Trusted agents delivery mechanisms (signing like kernel modules);
- Has a possibility to control what happens on system e.g block some
file operations, binary executions and so on;
- Has a portable and flexible events structure which doesn't get
broken from version to version;

For now I have PoC, which describes the concept in more detail:
GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
It contains all listed above points (maybe except portable event
structures)

There are an examples with:
- Security agent:
https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
- API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
- Event structures and types:
https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
- Main ESF source code:
https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf

Questions I'm interested in:
How does the community feel about this idea? Is it a viable concept?
If all is OK, what should I, as developer, do further? How much kernel
code outside the LSM module may be modified to keep further merge
acceptable? (currently not all LSM hooks meet to intercept all needed
data).

The general purpose is to make AV/EDR software development easier,
more convinient, and stable for Linux-based operating systems. This
PoC (as far as technology idea) is inspired by MacOS Endpoint Security
based on MAC policy.

Best regards,
Timur Chernykh,
Lead developer, F.A.C.C.T.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Stephen Smalley]

On Thu, Jun 20, 2024 at 9:40 AM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
>
> Hello!
>
> I'm here for yours' opinions.
>
> The modern Linux, in fact, does not provide convenient options to
> write AV/EDR software with real-time system events analysis as far as
> unified system security events. For now we have only inconsistent
> self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
> cool technology, but even CO-RE sometimes works as not expected or
> gets broken, for example:
> https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
> of a huge amount of the same hacks) looks a bit "crutchy".
>
> As an EDR developer, I have an idea how to fix this situation, using a
> unified endpoint security framework (ESF) placed in the kernel. Its
> must:
> - Provide unified and consistent security events;
> - API to communicate with kernel:
>  - To send program-defined events (as audit has);
>  - Read events from kernel;
> - Trusted agents delivery mechanisms (signing like kernel modules);
> - Has a possibility to control what happens on system e.g block some
> file operations, binary executions and so on;
> - Has a portable and flexible events structure which doesn't get
> broken from version to version;
>
> For now I have PoC, which describes the concept in more detail:
> GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
> It contains all listed above points (maybe except portable event
> structures)
>
> There are an examples with:
> - Security agent:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
> - API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
> - Event structures and types:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
> - Main ESF source code:
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
>
> Questions I'm interested in:
> How does the community feel about this idea? Is it a viable concept?
> If all is OK, what should I, as developer, do further? How much kernel
> code outside the LSM module may be modified to keep further merge
> acceptable? (currently not all LSM hooks meet to intercept all needed
> data).
>
> The general purpose is to make AV/EDR software development easier,
> more convinient, and stable for Linux-based operating systems. This
> PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> based on MAC policy.

I could be wrong, but I thought that supporting such solutions was one
of the reasons that fanotify was introduced, and that some combination
of LSM + fanotify + audit could fulfill most if not all of the
requirements. Can you explain which ones can't be met using those
today?

Note that a number of the LSM hooks are called from interrupt or while
holding locks and thus cannot sleep/block.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Tetsuo Handa]

On 2024/06/20 22:40, Timur Chernykh wrote:
> Questions I'm interested in:
> How does the community feel about this idea? Is it a viable concept?
> If all is OK, what should I, as developer, do further? How much kernel
> code outside the LSM module may be modified to keep further merge
> acceptable? (currently not all LSM hooks meet to intercept all needed
> data).
> 
> The general purpose is to make AV/EDR software development easier,
> more convinient, and stable for Linux-based operating systems. This
> PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> based on MAC policy.

I agree that security hooks for AV/EDR software are missing in Linux.
My experience says that customers cannot afford managing allowlist-based
access control mechanisms (such as SELinux and AppArmor) and they instead
choose AV/EDR software for their systems.

The LSM framework (which is using linked list for registering multiple
LSM modules) is about to be replaced with static calls (which reduces
overhead, at the cost of restricting at build time LSM modules which can
be registered). Use of static calls might make it possible to insert more
hooks into the Linux kernel because the overhead becomes negligible, but
kernel code used by AV/EDR software cannot be built into distributor
kernels due to support problem. Therefore, without ability to load
unlimited number of LSM modules after boot, AV/EDR software won't be
benefited with static calls. Such limitation will lead people to invent
a new set of security hooks (or resort to unofficial hacks such as
rewriting readonly data structure) rather than trying to utilize LSM
framework.

I prefer getting kernel code used by AV/EDR software reviewed (and get
their code tested by fuzzers), by allowing AV/EDR software developers
to submit their kernel code for upstream.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Stephen Smalley]

On Thu, Jun 20, 2024 at 10:28 AM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
>
> On Thu, Jun 20, 2024 at 9:40 AM Timur Chernykh <tim.cherry.co@gmail.com> wrote:
> >
> > Hello!
> >
> > I'm here for yours' opinions.
> >
> > The modern Linux, in fact, does not provide convenient options to
> > write AV/EDR software with real-time system events analysis as far as
> > unified system security events. For now we have only inconsistent
> > self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
> > cool technology, but even CO-RE sometimes works as not expected or
> > gets broken, for example:
> > https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
> > of a huge amount of the same hacks) looks a bit "crutchy".
> >
> > As an EDR developer, I have an idea how to fix this situation, using a
> > unified endpoint security framework (ESF) placed in the kernel. Its
> > must:
> > - Provide unified and consistent security events;
> > - API to communicate with kernel:
> >  - To send program-defined events (as audit has);
> >  - Read events from kernel;
> > - Trusted agents delivery mechanisms (signing like kernel modules);
> > - Has a possibility to control what happens on system e.g block some
> > file operations, binary executions and so on;
> > - Has a portable and flexible events structure which doesn't get
> > broken from version to version;
> >
> > For now I have PoC, which describes the concept in more detail:
> > GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
> > It contains all listed above points (maybe except portable event
> > structures)
> >
> > There are an examples with:
> > - Security agent:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
> > - API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
> > - Event structures and types:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
> > - Main ESF source code:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> >
> > Questions I'm interested in:
> > How does the community feel about this idea? Is it a viable concept?
> > If all is OK, what should I, as developer, do further? How much kernel
> > code outside the LSM module may be modified to keep further merge
> > acceptable? (currently not all LSM hooks meet to intercept all needed
> > data).
> >
> > The general purpose is to make AV/EDR software development easier,
> > more convinient, and stable for Linux-based operating systems. This
> > PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> > based on MAC policy.
>
> I could be wrong, but I thought that supporting such solutions was one
> of the reasons that fanotify was introduced, and that some combination
> of LSM + fanotify + audit could fulfill most if not all of the
> requirements. Can you explain which ones can't be met using those
> today?
>
> Note that a number of the LSM hooks are called from interrupt or while
> holding locks and thus cannot sleep/block.

Looking for a list of such hooks, it appears that the BPF LSM is
maintaining a list of sleepable LSM hooks in its sleepable_lsm_hooks
list in kernel/bpf/bpf_lsm.c.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Tetsuo Handa]

On 2024/06/20 23:28, Stephen Smalley wrote:
> Note that a number of the LSM hooks are called from interrupt or while
> holding locks and thus cannot sleep/block.

I think that that is because LSM hooks are intended for avoiding TOCTOU
problems. LSM hooks which TOMOYO checks permissions are allowed to
sleep/block, at the cost of giving up checking permissions for e.g.
sending signals.

Since I'm not an AV/EDR software developer, I can't say what hooks they
need. But history says that there was an attempt to use LSM hooks (and
the quality of an implementation which the source code is available was
too poor to recommend to customers). Analyzing vmcore suggests that recent
kernel code used by AV/EDR software tends to rewrite syscall tables.
I don't know the reason, but I think that that is because LSM framework
did not officially support loading LSM modules after boot, and LSM hooks
did not receive enough arguments needed by AV/EDR software. Thus, I guess
that adding security hooks into locations where locks are not yet held
would be helpful.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Timur Chernykh]

> I could be wrong, but I thought that supporting such solutions was one
> of the reasons that fanotify was introduced

You're right. Fanotify was invented to solve AV developers' problems,
but it gives only a filesystem monitor with, in my opinion, weak
instrumentary, when the modern EDR solutions do much more than just FS
control: firewall, fs protection, process monitoring, vulnerabilities
scanning, container security and so on.

> Can you explain which ones can't be met using those
> today?

Audit + fanotify doesn't give you control over the whole system, which
good AV/EDR should provide as a feature. For example, now you can not
"pause" the process execution and then "allow" it after scanning, like
Windows defender. You even cannot deny binary file or script
execution, like in my PoC here:
https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c#L543

Moreover, you can retrieve consistent information from just fanotify
and audit and correlate it because:
- They have a different format of events;
- They aren't synchronized with each other at all;
- Audit is not a reliable source of information, it has some sort of
events that differs or may be or may not be from system to system.

Of course, this is just "another programmers' task", but it could be
better if Linux provides it all from the box, like MacOS.

In my vision there should be only *one* source of security events for
the whole system. All events should be consistent and sequenced.

> Note that a number of the LSM hooks are called from interrupt or while
> holding locks and thus cannot sleep/block.

Yes, I know, thank you. This doesn't seem to be a big problem.

> I don't know the reason, but I think that that is because LSM framework
> did not officially support loading LSM modules after boot, and LSM hooks
> did not receive enough arguments needed by AV/EDR software.

Yep, I'm talking about moving all security logic to userspace, by
allowing some security agent(s) (like EDR or AV) to interact with
restricted API.
In this concept kernel just gives a guarantee of well-written code to
developers and delegates userspace protection to EDR/AV vendors. It's
like a loadable LSM module, but in userspace and not eBPF :)

I believe that there's no need in loadable LSM at all, just a security
agent, which communicates with the kernel. Therefore some of "hooks"
might not be LSM, it can be integrated in the kernel as part of
technology, as it is done with audit.

> I prefer getting kernel code used by AV/EDR software reviewed (and get
> their code tested by fuzzers), by allowing AV/EDR software developers
> to submit their kernel code for upstream.

Of course, it's an obvious thing, but at this step I haven't set up
any fuzzers or checks. It'll definitely be done in case this idea get
approve from community and I'll continue working on this PoC like on
new technology.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Casey Schaufler]

On 6/20/2024 6:40 AM, Timur Chernykh wrote:
> Hello!
>
> I'm here for yours' opinions.
>
> The modern Linux, in fact, does not provide convenient options to
> write AV/EDR software

Please don't assume that everyone knows your abbreviations.
Audio Visual / Editable Data Record.
Sure, "everyone knows" what AV/EDR is today. Except that many of
us also know what it was in 2010, 2000, 1990, and 1980. Confusion!

>  with real-time system events analysis as far as
> unified system security events. For now we have only inconsistent
> self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
> cool technology, but even CO-RE sometimes works as not expected or
> gets broken, for example:
> https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
> of a huge amount of the same hacks) looks a bit "crutchy".
>
> As an EDR developer, I have an idea how to fix this situation, using a
> unified endpoint security framework (ESF) placed in the kernel. Its
> must:
> - Provide unified and consistent security events;

"Consistent" is hardly a term you can use to describe Linux security events.
Is binding a socket to a port a security event? Where is the security event
when using io_uring?

> - API to communicate with kernel:
>  - To send program-defined events (as audit has);
>  - Read events from kernel;

A comprehensive set of events is likely to overwhelm any user space
collector. 

> - Trusted agents delivery mechanisms (signing like kernel modules);
> - Has a possibility to control what happens on system e.g block some
> file operations, binary executions and so on;

Performance concerns (locking, sleeping and a host of others) are
likely to prevent this.

> - Has a portable and flexible events structure which doesn't get
> broken from version to version;

If the kernel's policies, mechanisms and structures were more consistent
you might have a chance at this.

>
> For now I have PoC, which describes the concept in more detail:
> GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
> It contains all listed above points (maybe except portable event
> structures)
>
> There are an examples with:
> - Security agent:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
> - API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
> - Event structures and types:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
> - Main ESF source code:
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
>
> Questions I'm interested in:
> How does the community feel about this idea? Is it a viable concept?
> If all is OK, what should I, as developer, do further? How much kernel
> code outside the LSM module may be modified to keep further merge
> acceptable? (currently not all LSM hooks meet to intercept all needed
> data).

User-space arbitration of kernel events won't work. It's been tried
repeatedly. I started seeing it in the early 1980's. It always devolved
into a kernel bolt-on, with sub-optimal performance and versatility.

>
> The general purpose is to make AV/EDR software development easier,
> more convinient, and stable for Linux-based operating systems. This
> PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> based on MAC policy.
>
> Best regards,
> Timur Chernykh,
> Lead developer, F.A.C.C.T.
>

Re: Yet another vision of Linux security | Endpoint Security Framework

[Dr. Greg]

On Thu, Jun 20, 2024 at 04:40:09PM +0300, Timur Chernykh wrote:

> Hello!

Good afternoon Timur, I hope the week is going well for you.

> I'm here for yours' opinions.
> 
> The modern Linux, in fact, does not provide convenient options to
> write AV/EDR software with real-time system events analysis as far as
> unified system security events. For now we have only inconsistent
> self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
> cool technology, but even CO-RE sometimes works as not expected or
> gets broken, for example:
> https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
> of a huge amount of the same hacks) looks a bit "crutchy".
> 
> As an EDR developer, I have an idea how to fix this situation, using a
> unified endpoint security framework (ESF) placed in the kernel. Its
> must:
> - Provide unified and consistent security events;
> - API to communicate with kernel:
>  - To send program-defined events (as audit has);
>  - Read events from kernel;
> - Trusted agents delivery mechanisms (signing like kernel modules);
> - Has a possibility to control what happens on system e.g block some
> file operations, binary executions and so on;
> - Has a portable and flexible events structure which doesn't get
> broken from version to version;
> 
> For now I have PoC, which describes the concept in more detail:
> GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
> It contains all listed above points (maybe except portable event
> structures)
> 
> There are an examples with:
> - Security agent:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
> - API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
> - Event structures and types:
> https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
> - Main ESF source code:
> https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> 
> Questions I'm interested in:
> How does the community feel about this idea? Is it a viable concept?
> If all is OK, what should I, as developer, do further? How much kernel
> code outside the LSM module may be modified to keep further merge
> acceptable? (currently not all LSM hooks meet to intercept all needed
> data).
> 
> The general purpose is to make AV/EDR software development easier,
> more convinient, and stable for Linux-based operating systems. This
> PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> based on MAC policy.

I'm not sure how much you follow the Linux Security Mailing List so
you may have missed it.  I believe you will find that our Trusted
Security Event Modeling (TSEM) architecture, that we have submitted
for review and inclusion in the kernel, provides a superset of the
type of functionality that you and others in the industry are looking
for.

Here are links to the three releases:

V1
https://lore.kernel.org/linux-security-module/20230204050954.11583-1-greg@enjellic.com/T/#t

V2:
https://lore.kernel.org/linux-security-module/20230710102319.19716-1-greg@enjellic.com/T/#t

V3:
https://lore.kernel.org/linux-security-module/20240401105015.27614-1-greg@enjellic.com/T/#m1cc4ad76677d07517220f1fadc2303c36a75bab0

The first release was in February of 2023, a year and a half ago, we
are on schedule for the fourth release, which is now largely feature
complete, after the US Fourth of July holiday week.

We describe TSEM as a generic security modeling architecture for the
kernel.  It is based on the generation of security state coefficients
for each LSM based security event that occurs.  These coefficients can
be used to drive mathematical models that very precisely describe the
security state of either the platform at large or an isolated
workload.

The underlying model is capable of providing very precise detection of
attempts to implement concepts such as Living Off The Land, ie. Hello
Volt Typhoon.

The operative premise is that in all aspects of science and
engineering it is routine to build mathematical models for the
performance of things like airplanes, bridges, buildings, the
electronic state of molecular systems etc., why not the security
behavior of an operating system.

Obviously the current zest for machine learning/AI is an embodiment of
the principal of modeling.

Over the last year it has become increasingly apparent that TSEM can
play a very powerful role in endpoint security and anomaly detection
systems.  Particularly with respect to driving machine learning models
in environments where deterministic or quasi-deterministic models are
infeasible.

So TSEM now has an efficient export only mode that provides JSON
encoded description streams of almost all of the security relevant
events that the LSM can capture.  So it provides a framework for doing
everything from security reconaissance to enforcement.

Others have already commented on this, but the notion of having
'standardized' event descriptions for any system of this type is
problematic.  Given that it has been made clear that there is no
intention to have a stable API for the Linux security architecture.
FWIW, in practice we have not found this to be a significant
impediment to what can be implemented from a security perspective.

I believe we address the issue of security event handlers that run in
atomic context as best as can be expected.  If you look through the
e-mail threads surrounding our three releases you will see timing
information as to how fast our external modelers can react, on the
order of tens of micro-seconds, as compared to current agent based
systems that have measured reaction latencies in excess of multiple
seconds if not longer.

If external modeling is infeasible, and the default TSEM deterministic
model is insufficient, the v4 release TSEM has the ability to use
loadable modules to build customized in kernel modelers that can
detect and react at event time, regardless of process context.

We are always looking for thoughts and feedback from others on how to
enhance what we are working to deliver.

Our GitHub site has GIT trees for not only the TSEM kernel component
but also the Quixote userspace tools that make the kernel driver
useful.

The kernel sources include very extensive documentation describing the
design and motivation of the system, including the JSON encoding of
the event descriptions.

We are in the process of providing a public, MQTT based broker system,
that will provide access to event streams of some of our internal
systems if anyone wants to look at what the security event description
streams look like or if there would be a desire to experiment with
building models.

In closing, one of the increasingly important objectives with TSEM is
to provide kernel based security event description infrastructure that
reduces the potential supply chain attack scenarios that are possible,
given the amount of privilege being afforded to current agent systems.

> Best regards,
> Timur Chernykh,
> Lead developer, F.A.C.C.T.

We hope the above information is useful.

On behalf of the Quixote Team, have a good weekend.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: Yet another vision of Linux security | Endpoint Security Framework

[Timur Chernykh]

> Please don't assume that everyone knows your abbreviations.
> Audio Visual / Editable Data Record.

I'm sorry for that. Of course, AV - Antivirus, EDR - Endpoint
Detection and Response.

> "Consistent" is hardly a term you can use to describe Linux security events.
> Is binding a socket to a port a security event? Where is the security event
> when using io_uring?

Under "consistent" I mean "has a unified structure". Currently all
security mechanisms in Linux have a different events format and it's
an additional task to read, parse and transform it to a unified view.
The assumed list of security events can be found at
https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h#L73

> A comprehensive set of events is likely to overwhelm any user space
> collector.

I'm definitely agree. The kernel in my vision must define what
security events are, as it is done with LSM. Userspace should have a
restricted set of security events, it shouldn't know about any kernel
internals. Each event should be a kind of abstraction above what
really happens. There is a good example of real-life implementation:
https://developer.apple.com/documentation/endpointsecurity/event_types.
For my vision of linux security events in userspace, please, look the
link above thought.

> "Consistent" is hardly a term you can use to describe Linux security events.

Yes, and this is a bad thing, which I want to fix.

> Is binding a socket to a port a security event? Where is the security event
> when using io_uring?

Socket binding may not be a security event itself, but a listening
port for a network connection is. And it doesn't matter how many steps
are included to a port listening itself. Opening socket, setup,
binding should be converted to a single event, that says to userspace
agent: "Hey, this process is trying to listen to the socket for a
network connection. Do you want to allow this action?".

> Performance concerns (locking, sleeping and a host of others) are
> likely to prevent this.

It's not an unresolvable task. MacOS and Windows solved this issue.
Anyway, users should have a choice between security and performance.
All modern security solutions in any implementation already affect the
linux performance.

> If the kernel's policies, mechanisms and structures were more consistent
> you might have a chance at this.

I think that's a work I (and even probable we) should work for. As a
better OS Linux should provide userspace stable structured security
events.

> User-space arbitration of kernel events won't work. It's been tried
> repeatedly. I started seeing it in the early 1980's. It always devolved
> into a kernel bolt-on, with sub-optimal performance and versatility.

Can you give examples, please? I don't remember early '80, hah, I want
to get to know about previous attempts :)
I don't be sure if this is impossible. As I mentioned before, Windows
and MacOS are done with it. Anyway, I think that users must have a
choice.

Re: Yet another vision of Linux security | Endpoint Security Framework

[Dr. Greg]

On Thu, Jun 20, 2024 at 11:39:43PM +0900, Tetsuo Handa wrote:

Good morning, I hope the week is starting well for everyone.

> On 2024/06/20 22:40, Timur Chernykh wrote:
> > Questions I'm interested in:
> > How does the community feel about this idea? Is it a viable concept?
> > If all is OK, what should I, as developer, do further? How much kernel
> > code outside the LSM module may be modified to keep further merge
> > acceptable? (currently not all LSM hooks meet to intercept all needed
> > data).
> > 
> > The general purpose is to make AV/EDR software development easier,
> > more convinient, and stable for Linux-based operating systems. This
> > PoC (as far as technology idea) is inspired by MacOS Endpoint Security
> > based on MAC policy.

> I agree that security hooks for AV/EDR software are missing in
> Linux.

The issue isn't an insufficient number of security hooks, the issue is
that the detection of anomalous behavior is not being framed properly.

This is a modeling problem.  The challenge is that it is a modeling
problem over a range of potential model types, each with its own set
of advantages and disadvantages.  So the best thing that can be done
is to expose the basis set of information that the Linux development
community has indicated as relevant to the security posture of the
operating system.

That basis set is currently the LSM hooks.  Given the amount of
information that can be surfaced by these hooks, the security industry
hasn't even scratched the surface with respect to the available
detection limits.

> My experience says that customers cannot afford managing
> allowlist-based access control mechanisms (such as SELinux and
> AppArmor) and they instead choose AV/EDR software for their systems.

This issue, which may be an unpopular concept in some corners of the
this list, is true.  The market has spoken, for both technical and
political reasons, the future is going to involve host based modeling
and detection of unwanted platform or workload behaviors.

There are a myriad of technical reasons for this, the political reason
though is pretty sample, paying a vendor to be responsible for
detecting security intrusions allows companies, to at least
ostensibly, transfer the liability from themselves to another entity.

It is basically a manifestation of the long standing technology meme
of: "No one ever got fired for buying IBM".

> The LSM framework (which is using linked list for registering multiple
> LSM modules) is about to be replaced with static calls (which reduces
> overhead, at the cost of restricting at build time LSM modules which can
> be registered). Use of static calls might make it possible to insert more
> hooks into the Linux kernel because the overhead becomes negligible, but
> kernel code used by AV/EDR software cannot be built into distributor
> kernels due to support problem. Therefore, without ability to load
> unlimited number of LSM modules after boot, AV/EDR software won't be
> benefited with static calls. Such limitation will lead people to invent
> a new set of security hooks (or resort to unofficial hacks such as
> rewriting readonly data structure) rather than trying to utilize LSM
> framework.

The commentary above would thus suggest that the problem is not one of
an insufficient number of LSM hooks but rather the amount of
flexibility with respect to interpreting the existing hooks.

What we would suggest is needed is 2-fold.

First, there is not a need for more LSM's but rather an LSM that has
more flexibility with respect to the type of responses that can be
implemented for the existing hooks.  The BPF LSM is a partial step in
that direction, but there are challenges that you have previously
documented, with respect to what can be accomplished.

Secondly, we would posit that there is more interest from the security
industry in getting the information provided by the existing hooks out
of the kernel for analysis rather than implementing the classic
immediate response model of current mandatory access control systems.

We would further suggest that even casual observation of the security
industry would indicate, that like many other technology venues, it is
now completely enamored with the notion of the application of
AI/machine learning to it mission.  That is not going to happen inside
of the kernel, the question will be providing an optimized framework
for making this happen.

At the risk of being accussed of 'hijacking' a thread, we would posit
that we are addressing both issues with our TSEM implementation,
particularly in the V4 release that is right around the corner.

> I prefer getting kernel code used by AV/EDR software reviewed (and get
> their code tested by fuzzers), by allowing AV/EDR software developers
> to submit their kernel code for upstream.

We would also suggest that it is unlikely that security vendors are
going to be willing to submit much, if any, of their 'secret sauce'
for inclusion in the kernel.  Alexey Starovoitov aluded to this at his
BPF presentation at the European Security Summit.

At least on Linux, the current endpoint security agent architecture
that the industry is grappling with is a functional and security
nightmare waiting to happen.

The most positive security contributions will surround reducing the
security footprint and privilege level that the vendors need in order
to accomplish their objectives, along with increasing the simplicity
of access to the vast amount of information that is available from the
existing hooks/handlers.

Have a good week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: Yet another vision of Linux security | Endpoint Security Framework

[Dr. Greg]

On Thu, Jun 20, 2024 at 09:52:36AM -0700, Casey Schaufler wrote:

Good afternoon, I hope the week is going well for everyone.

> On 6/20/2024 6:40 AM, Timur Chernykh wrote:
> > Hello!
> >
> > I'm here for yours' opinions.
> >
> > The modern Linux, in fact, does not provide convenient options to
> > write AV/EDR software

> Please don't assume that everyone knows your abbreviations.  Audio
> Visual / Editable Data Record.  Sure, "everyone knows" what AV/EDR
> is today. Except that many of us also know what it was in 2010,
> 2000, 1990, and 1980. Confusion!

We will then take as standard practice for the Linux security mailing
list the dictum from academic publishing, for any phrases referenced
as acronyms, the expectation is to define the term with the acronym in
parenthesis and then use the acronym throughout the remainder of the
document in place of the term.

We could perhaps prevail upon Paul Moore to update the ANN document
accordingly to document correct practice... :-)

> >  with real-time system events analysis as far as
> > unified system security events. For now we have only inconsistent
> > self-made events from eBPF, audit, fanotify, etc. eBPF itself is a
> > cool technology, but even CO-RE sometimes works as not expected or
> > gets broken, for example:
> > https://github.com/aquasecurity/tracee/pull/3769/files. This hack (one
> > of a huge amount of the same hacks) looks a bit "crutchy".
> >
> > As an EDR developer, I have an idea how to fix this situation, using a
> > unified endpoint security framework (ESF) placed in the kernel. Its
> > must:
> > - Provide unified and consistent security events;

> "Consistent" is hardly a term you can use to describe Linux security
> events.  Is binding a socket to a port a security event? Where is
> the security event when using io_uring?

Linux security events grew organically, so that is what we have to
deal with.

The alternative is to implement what Timur is proposing.  A second set
of kernel infrastructure, parallel to the current LSM but with more
'consistency'.  A project whose arguments over purpose, and the
subsequent bike-shedding over implementation, will take until the heat
death of the Universe to resolve, if then.

Binding to a socket is absolutely a security event, if not there would
have presumably been no LSM security event handler defined for the
action.  Nearly every webshell or Remote Access Technology methodology
worth its weight as malware attempts to do so.

SMACK itself nods to the potential security relevance of binding a
socket, given that it provides a handler for security_socket_bind if
IPV6 port labeling is in effect.

With respect to io_uring functionality.

Based on the current 6.8 kernel sources, the security relevant events
for io_uring are defined as follows:

uring_override_creds

uring_sqpoll

uring_cmd

One of the challenges with defining what is a security relevant event
comes from the fact that security architectures have historically been
prospective, the notion of defining a set of rules that goven access
to security critical events.  In this Kantian formed model one has to
presume what security relevant events are.

In a retrospective or Hegelian model, where the desired security
behavior is derived from the security events that a workload is unit
tested to, this issue is decidedly more deterministic.  If you have a
basis set that describes the field of security relevant events any
departures from that basis represent a violation.

> > - API to communicate with kernel:
> >  - To send program-defined events (as audit has);
> >  - Read events from kernel;

> A comprehensive set of events is likely to overwhelm any user space
> collector.

A couple of responses to that:

First.

Given how busy you are it is unlikely that you would have had time to
patch a kernel with TSEM and setup the Quixote userspace utilities.
For anyone who would have time to do so, we believe that doing so
would provide a clear demonstration of the feasibility of capturing
detailed security event information from all of the LSM security event
handlers.

In round numbers.  Booting a minimal Debian 12 implementation to a
shell prompt with a 6.1 kernel implementation generates 81 megabytes
of JSON encoded LSM event information over 119,000 events which
represent 24,000 security unique events.

To extend further the tractability argument, let us assume that the
collector and/or enforcement system is running on hardware of similar
performance characteristics to the hardware that is generating the
event stream.

To suggest that the information could not be collected or acted on
without being overwhelmed would be to suggest that the existing Linux
mandatory access controls, acting on the same set of events at the
same rate, would be overwhelmed and thus incapable of being feasibly
implemented.

Secondly.

Current activity in the security industry would suggest that the
capture and interpretation of, at least what it considers relevant
security event information, by userspace is clearly viable.

Technical issues aside, CrowdStrike's current 93 billion dollar market
capitalization as of last Friday (06/21/2024) would suggest that the
concept is notionally, if not commercially viable.

Userspace capture and analysis of security event information is how
security is now being done and that will increasingly be the case in
the future.  Particularly if, as we believe, that access to high
resolution LSM event information provides an extremely strong basis
set for training machine learning models.

> > - Trusted agents delivery mechanisms (signing like kernel modules);
> > - Has a possibility to control what happens on system e.g block some
> > file operations, binary executions and so on;

> Performance concerns (locking, sleeping and a host of others) are
> likely to prevent this.

Once again, we believe that current art demonstrates that to not be
the case.

The locking and sleeping issues are only problematic in the context of
the current Kantian models that are attempting to respond with real
time deterministic security decisions.  As has been noted previously,
that is not how the industry is choosing to implement security.

All of the agent based systems are acting in a retrospective manner
where a determination is made that a system has exhibited security
anomalous behavior and an alert is generated and/or the system is
isolated.

> > - Has a portable and flexible events structure which doesn't get
> > broken from version to version;

> If the kernel's policies, mechanisms and structures were more
> consistent you might have a chance at this.

It is certainly the case that there is nothing on the horizon that
suggests the possibility of a portable or consistent event reporting
structure.  Paul Moore has consistently and strongly stated that there
can be no expectations for a stable security API on Linux.

That being said, this issue has to be taken in the context of how the
technology industry is operating.  As RedHat/IBM has clearly
demonstrated with their business model, there is generalized appetite
in industry to lock onto a particular version of an operating system
implementation for long periods of time.

This is even more so the case in the critical infrastructure industry.
The expectation there is for common infrastructure that will be
persisted for at least 15-20 year timeframes.

So security solutions, as do other application providers, simply
target the behavior of these standardized releases.

The BPF LSM has the same issues with respect to API stability and it
is not only being embraced by the security industry it is being
endorsed in some corners of the Linux community as the 'way forward'
for security implementations.

> > For now I have PoC, which describes the concept in more detail:
> > GH mirror: https://github.com/Linux-Endpoint-Security-Framework/linux.
> > It contains all listed above points (maybe except portable event
> > structures)
> >
> > There are an examples with:
> > - Security agent:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/samples/esf/agent.c
> > - API: https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/ctl.h
> > - Event structures and types:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/blob/esf/main/include/uapi/linux/esf/defs.h
> > - Main ESF source code:
> > https://github.com/Linux-Endpoint-Security-Framework/linux/tree/esf/main/security/esf
> >
> > Questions I'm interested in:
> > How does the community feel about this idea? Is it a viable concept?
> > If all is OK, what should I, as developer, do further? How much kernel
> > code outside the LSM module may be modified to keep further merge
> > acceptable? (currently not all LSM hooks meet to intercept all needed
> > data).

> User-space arbitration of kernel events won't work. It's been tried
> repeatedly. I started seeing it in the early 1980's. It always
> devolved into a kernel bolt-on, with sub-optimal performance and
> versatility.

Once again, we believe that current art suggests to the contrary.

Going to userspace for decision making always has performance impacts.
We have posted numbers that demonstrate that deterministic models,
mediated through userspace, are capable of responding with 20-40
micro-second decision latencies.

That may be considered a viable tradeoff if the alternative is to
spend 5-7 years attempting to get a kernel based solution first
accepted and then into the field.

As has been noted previously, the security industry is embracing a
model based approach to security.  Such systems are not going to go
into the kernel, not only because they can't, but also because doing
so would require solution providers to expose their intellectual
property.

The versatility argument is difficult to understand if userspace is
acting on the same information that an in kernel LSM event handler is
sampling and acting on.

A final comment on performance and versatility.  Those issue may be
relevant to kernel developers and perhaps the major technology
companies that are running their own cloud infrastructure, but they
are irrelevant to where most of Linux is going, general industry.

I can line up a dozen system administrators that will raise their hand
and swear on the good Book as to how badly performant and inherently
dangerous these current security solutions are.  The answer from
organizational leadership is always the same: 'you will run it without
questions'.

The challenge for the Linux security architecture, to remain relevant,
is to provide better infrastructure to implement these solutions given
the engineering constraints they impose.

Have a good remainder of the week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project