Cannot disable kernel's audit system via auditctl

Cannot disable kernel's audit system via auditctl

[Samuel Bahr]

[-- Attachment #1.1: Type: text/plain, Size: 3455 bytes --]

Hi linux-audit,

I'm running a fleet of Linux hosts with Red Canary Linux EDR (Endpoint
Detection and Response) which uses eBPF for gathering telemetry in service `
cfsvcd.service`. In an older configuration, it gathered data from the
kernel's audit system and everything was fine. However, when we switched
cfsvcd to gathering data from eBPF instead, we noticed that the kernel ring
buffer was flooded with audit messages. This is because cfsvcd.service now
stops auditd.service, but leaves the kernel audit system enabled.

I've mitigated this issue by manually running `# auditctl -e 0` on our
hosts (via Puppet). However, I'm running into a strange issue where _some_
hosts (~0.5%) are still logging all audit events to the kernel ring buffer
even after I have disabled the audit system via `# auditctl -e 0`. A `#
auditctl -s` run shows `enabled 0`, yet audit logs continue to flood the
kernel ring buffer.

I'm running Linux kernel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl
v2.8.2. `systemd-journald-audit.socket` is masked & inactive, `
auditd.service` is disabled & inactive.

I cannot entirely disable the audit system via a kernel parameter because
Red Canary Linux EDR fails to start cfsvcd.service as it fails to run some
auditctl command due to no audit support in the kernel:

Jun 28 20:41:04 systemd[1]: Started Canary Forwarder Service.
> Jun 28 20:41:04 cfsvcd[105781]: Found config file at path
> '"/opt/redcanary/config.json"', continuing execution...
> Jun 28 20:41:12 cfsvcd[105781]: component: Sentry
> Jun 28 20:41:12 cfsvcd[105781]:  Jun 28 20:41:12.055 INFO Initializing
> Sentry Config { version: "1.4.17.release.[...]", https_proxy: None,
> extra_tags: [("telemetry.source", "eBPF")], logging_enabled: true,
> metrics_enabled: true, [...]}
> Jun 28 20:41:12 cfsvcd[105781]: Failed to setup or configure host system:
> Linux Audit watcher failure: Netlink Error: IO Error: Protocol not
> supported (os error 93)
> Jun 28 20:41:12 cfsvcd[105781]: Received stop. Exiting
> unix::wait_for_signal. Signal: 0, done.is_signaled(): true
> Jun 28 20:41:12 cfsvcd[105781]: Joining async_agent_thread
> Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber (Ebpf).
> Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to termination signal
> [...]
> Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped.
> Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel
> Jun 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket
> Jun 28 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over,
> scheduling restart.
> Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart job, restart
> counter is at 301.
> Jun 28 20:41:15 systemd[1]: Stopped Canary Forwarder Service.
>

Here's the unit file for cfsvcd.service:

> [Unit]
> Description=Canary Forwarder Service
>
> DefaultDependencies=no
> After=local-fs.target systemd-tmpfiles-setup.service auditd.service
> Before=sysinit.target shutdown.target
> # Replace the auditd service if it is running
> Conflicts=auditd.service shutdown.target
>
> [Service]
> WorkingDirectory=/opt/redcanary
> ExecStart=/opt/redcanary/cfsvcd
> ExecStopPost=-/sbin/auditctl -D
> Restart=always
> TimeoutSec=15
>
> [Install]
> WantedBy=multi-user.target
>

Is this a known issue? Is there a workaround to stop the logging to the
kernel ring buffer? Is there any more information I can provide to help
debug?

Thanks,
Samuel Bahr
Pinterest Sr. Site Reliability Engineer

[-- Attachment #1.2: Type: text/html, Size: 4608 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Cannot disable kernel's audit system via auditctl

[Steve Grubb]

On Thursday, June 29, 2023 6:34:03 PM EDT Samuel Bahr wrote:
> Hi linux-audit,
> 
> I'm running a fleet of Linux hosts with Red Canary Linux EDR (Endpoint
> Detection and Response) which uses eBPF for gathering telemetry in service
> ` cfsvcd.service`. In an older configuration, it gathered data from the
> kernel's audit system and everything was fine. However, when we switched
> cfsvcd to gathering data from eBPF instead, we noticed that the kernel
> ring buffer was flooded with audit messages. This is because
> cfsvcd.service now stops auditd.service, but leaves the kernel audit
> system enabled.
> 
> I've mitigated this issue by manually running `# auditctl -e 0` on our
> hosts (via Puppet). However, I'm running into a strange issue where _some_
> hosts (~0.5%) are still logging all audit events to the kernel ring buffer
> even after I have disabled the audit system via `# auditctl -e 0`. A `#
> auditctl -s` run shows `enabled 0`, yet audit logs continue to flood the
> kernel ring buffer.
> 
> I'm running Linux kernel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl
> v2.8.2. `systemd-journald-audit.socket` is masked & inactive, `
> auditd.service` is disabled & inactive.
> 
> I cannot entirely disable the audit system via a kernel parameter because
> Red Canary Linux EDR fails to start cfsvcd.service as it fails to run some
> auditctl command due to no audit support in the kernel:
> 
> Jun 28 20:41:04 systemd[1]: Started Canary Forwarder Service.
> 
> > Jun 28 20:41:04 cfsvcd[105781]: Found config file at path
> > '"/opt/redcanary/config.json"', continuing execution...
> > Jun 28 20:41:12 cfsvcd[105781]: component: Sentry
> > Jun 28 20:41:12 cfsvcd[105781]:  Jun 28 20:41:12.055 INFO Initializing
> > Sentry Config { version: "1.4.17.release.[...]", https_proxy: None,
> > extra_tags: [("telemetry.source", "eBPF")], logging_enabled: true,
> > metrics_enabled: true, [...]}
> > Jun 28 20:41:12 cfsvcd[105781]: Failed to setup or configure host system:
> > Linux Audit watcher failure: Netlink Error: IO Error: Protocol not
> > supported (os error 93)
> > Jun 28 20:41:12 cfsvcd[105781]: Received stop. Exiting
> > unix::wait_for_signal. Signal: 0, done.is_signaled(): true
> > Jun 28 20:41:12 cfsvcd[105781]: Joining async_agent_thread
> > Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber (Ebpf).
> > Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to termination signal
> > [...]
> > Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped.
> > Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel
> > Jun 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket
> > Jun 28 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over,
> > scheduling restart.
> > Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart job,
> > restart
> > counter is at 301.
> > Jun 28 20:41:15 systemd[1]: Stopped Canary Forwarder Service.
> 
> Here's the unit file for cfsvcd.service:
> > [Unit]
> > Description=Canary Forwarder Service
> > 
> > DefaultDependencies=no
> > After=local-fs.target systemd-tmpfiles-setup.service auditd.service
> > Before=sysinit.target shutdown.target
> > # Replace the auditd service if it is running
> > Conflicts=auditd.service shutdown.target
> > 
> > [Service]
> > WorkingDirectory=/opt/redcanary
> > ExecStart=/opt/redcanary/cfsvcd
> > ExecStopPost=-/sbin/auditctl -D
> > Restart=always
> > TimeoutSec=15
> > 
> > [Install]
> > WantedBy=multi-user.target
> 
> Is this a known issue?

Not really. But this is a new capability. If you run auditctl -D, does it go 
away? Have any of the systems been setup with auditctl -e 2? This makes the 
configuration immutable.

-Steve

> Is there a workaround to stop the logging to the
> kernel ring buffer? Is there any more information I can provide to help
> debug?



--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Cannot disable kernel's audit system via auditctl

[Samuel Bahr]

[-- Attachment #1.1: Type: text/plain, Size: 4807 bytes --]

Hi Steve,

`auditctl -D` does not make it go away (outputs `No rules`). auditd isn't
running at all and this behavior is happening purely from the kernel. These
systems were never set to enabled 2 (locked).

I went ahead and filed a Github issue for this thread:
https://github.com/linux-audit/audit-kernel/issues/146

The maintainer there suggested it's too difficult to debug due to eBPF
programs + AWS's modified kernel. I've resigned to asking Red Canary to
support eBPF mode with `audit=0` kernel parameter in their Linux EDR. Let
me know if you have any other ideas.

Regards,
Samuel Bahr
Pinterest Sr. Site Reliability Engineer

On Sun, Jul 23, 2023 at 7:17 PM Steve Grubb <sgrubb@redhat.com> wrote:

> On Thursday, June 29, 2023 6:34:03 PM EDT Samuel Bahr wrote:
> > Hi linux-audit,
> >
> > I'm running a fleet of Linux hosts with Red Canary Linux EDR (Endpoint
> > Detection and Response) which uses eBPF for gathering telemetry in
> service
> > ` cfsvcd.service`. In an older configuration, it gathered data from the
> > kernel's audit system and everything was fine. However, when we switched
> > cfsvcd to gathering data from eBPF instead, we noticed that the kernel
> > ring buffer was flooded with audit messages. This is because
> > cfsvcd.service now stops auditd.service, but leaves the kernel audit
> > system enabled.
> >
> > I've mitigated this issue by manually running `# auditctl -e 0` on our
> > hosts (via Puppet). However, I'm running into a strange issue where
> _some_
> > hosts (~0.5%) are still logging all audit events to the kernel ring
> buffer
> > even after I have disabled the audit system via `# auditctl -e 0`. A `#
> > auditctl -s` run shows `enabled 0`, yet audit logs continue to flood the
> > kernel ring buffer.
> >
> > I'm running Linux kernel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl
> > v2.8.2. `systemd-journald-audit.socket` is masked & inactive, `
> > auditd.service` is disabled & inactive.
> >
> > I cannot entirely disable the audit system via a kernel parameter because
> > Red Canary Linux EDR fails to start cfsvcd.service as it fails to run
> some
> > auditctl command due to no audit support in the kernel:
> >
> > Jun 28 20:41:04 systemd[1]: Started Canary Forwarder Service.
> >
> > > Jun 28 20:41:04 cfsvcd[105781]: Found config file at path
> > > '"/opt/redcanary/config.json"', continuing execution...
> > > Jun 28 20:41:12 cfsvcd[105781]: component: Sentry
> > > Jun 28 20:41:12 cfsvcd[105781]:  Jun 28 20:41:12.055 INFO Initializing
> > > Sentry Config { version: "1.4.17.release.[...]", https_proxy: None,
> > > extra_tags: [("telemetry.source", "eBPF")], logging_enabled: true,
> > > metrics_enabled: true, [...]}
> > > Jun 28 20:41:12 cfsvcd[105781]: Failed to setup or configure host
> system:
> > > Linux Audit watcher failure: Netlink Error: IO Error: Protocol not
> > > supported (os error 93)
> > > Jun 28 20:41:12 cfsvcd[105781]: Received stop. Exiting
> > > unix::wait_for_signal. Signal: 0, done.is_signaled(): true
> > > Jun 28 20:41:12 cfsvcd[105781]: Joining async_agent_thread
> > > Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber (Ebpf).
> > > Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to termination signal
> > > [...]
> > > Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped.
> > > Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel
> > > Jun 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket
> > > Jun 28 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over,
> > > scheduling restart.
> > > Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart job,
> > > restart
> > > counter is at 301.
> > > Jun 28 20:41:15 systemd[1]: Stopped Canary Forwarder Service.
> >
> > Here's the unit file for cfsvcd.service:
> > > [Unit]
> > > Description=Canary Forwarder Service
> > >
> > > DefaultDependencies=no
> > > After=local-fs.target systemd-tmpfiles-setup.service auditd.service
> > > Before=sysinit.target shutdown.target
> > > # Replace the auditd service if it is running
> > > Conflicts=auditd.service shutdown.target
> > >
> > > [Service]
> > > WorkingDirectory=/opt/redcanary
> > > ExecStart=/opt/redcanary/cfsvcd
> > > ExecStopPost=-/sbin/auditctl -D
> > > Restart=always
> > > TimeoutSec=15
> > >
> > > [Install]
> > > WantedBy=multi-user.target
> >
> > Is this a known issue?
>
> Not really. But this is a new capability. If you run auditctl -D, does it
> go
> away? Have any of the systems been setup with auditctl -e 2? This makes
> the
> configuration immutable.
>
> -Steve
>
> > Is there a workaround to stop the logging to the
> > kernel ring buffer? Is there any more information I can provide to help
> > debug?
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 5886 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Cannot disable kernel's audit system via auditctl

[Steve Grubb]

On Monday, July 24, 2023 5:06:02 PM EDT Samuel Bahr wrote:
> `auditctl -D` does not make it go away (outputs `No rules`). auditd isn't
> running at all and this behavior is happening purely from the kernel. These
> systems were never set to enabled 2 (locked).
> 
> I went ahead and filed a Github issue for this thread:
> https://github.com/linux-audit/audit-kernel/issues/146
> 
> The maintainer there suggested it's too difficult to debug due to eBPF
> programs + AWS's modified kernel.

I think there is data that could help decide where the problem might be. On 
one of the systems that is still logging, try running an event type report:

aureport --start yesterday --event --summary -i

This should identify what kind of event is being emitted. Based on that, it 
might point to where the problem is.

> I've resigned to asking Red Canary to support eBPF mode with `audit=0`
> kernel parameter in their Linux EDR. Let me know if you have any other
> ideas.

I'd say collecting summary information about what kind of events are being 
logged would be a good start.

-Steve


--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Cannot disable kernel's audit system via auditctl

[Samuel Bahr]

[-- Attachment #1.1: Type: text/plain, Size: 2499 bytes --]

Hi Steve,

Since auditd.service is disabled nothing is writing to the audit log:

# aureport --start yesterday --event --summary -i
>
> Event Summary Report
> ======================
> total  type
> ======================
> <no events of interest were found>
>

However, I can parse the audit logs that do make it to the kernel ring
buffer (not suppressed by printk):

> $ dmesg | grep -Eo 'type=[0-9]+' | sort | uniq -c | sort -n
>    1198 type=1400
>

In the past, before landing on our current (partial) solution to disable
the kernel audit subsystem w/ `auditctl -e 0` we tried just filtering out
type 1400 messages w/ `auditctl -a exclude,always -F msgtype=1400`.
However, that only lead us to seeing other types of audit messages in the
kernel ring buffer:

> $ dmesg | grep -Eo 'type=[0-9]+' | sort | uniq -c | sort -n
> 7 type=1109
> 10 type=1110
> 12 type=1123
> 52 type=1112
> 91 type=1131
> 93 type=1130
> 268 type=1105
> 356 type=1104
> 357 type=1106
> 414 type=1006
> 454 type=1103
> 459 type=1101
>

Note, that was with the kernel audit subsystem enabled so perhaps the
behavior is different than now where it's disabled entirely.

Regards,
Samuel Bahr
Pinterest Sr. Site Reliability Engineer

On Tue, Jul 25, 2023 at 12:05 PM Steve Grubb <sgrubb@redhat.com> wrote:

> On Monday, July 24, 2023 5:06:02 PM EDT Samuel Bahr wrote:
> > `auditctl -D` does not make it go away (outputs `No rules`). auditd isn't
> > running at all and this behavior is happening purely from the kernel.
> These
> > systems were never set to enabled 2 (locked).
> >
> > I went ahead and filed a Github issue for this thread:
> > https://github.com/linux-audit/audit-kernel/issues/146
> >
> > The maintainer there suggested it's too difficult to debug due to eBPF
> > programs + AWS's modified kernel.
>
> I think there is data that could help decide where the problem might be.
> On
> one of the systems that is still logging, try running an event type report:
>
> aureport --start yesterday --event --summary -i
>
> This should identify what kind of event is being emitted. Based on that,
> it
> might point to where the problem is.
>
> > I've resigned to asking Red Canary to support eBPF mode with `audit=0`
> > kernel parameter in their Linux EDR. Let me know if you have any other
> > ideas.
>
> I'd say collecting summary information about what kind of events are being
> logged would be a good start.
>
> -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3546 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit