Re: Monitoring of linux containers

[Richard Guy Briggs <rgb@redhat.com>]

On 15/09/02, Roi Martin wrote:
> Hello,

Hi Roi,

> First, let me give you a bit of context about what we are trying to do.
> Basically, we are working on a monitoring solution for linux containers
> based on audit. The idea is logging all the syscalls executed within
> certain container and take metrics, that are compared with a baseline in
> order to find anomalies on the executed services. For instance, take as
> example a micro-service running within a container, if suddenly we see a
> clone, followed by an execve of "/bin/sh", this should be detected and
> actions would be taken automatically.
> 
> The thing is that to accomplish this, we need to create an audit rule that
> allows to filter only those system calls executed in the context of the
> monitored containers. For this, I can think on two possible ways:
> 1. By PID: Creating a rule like "filter all the syscalls executed by the
> process with PID X and all its current and future children".

A general problem with filtering on PID X is that currently is keyed on
a number, which can roll.  Reaping the rule when that process terminates
(and all its children) might be one way to do it, but that is not
currently implemented.

> 2. By namespace: It would be great to be able to create a rule that filters
> all the syscalls executed under a given list of namespaces, in this case,
> those corresponding to the monitored container.

Agreed!  A number of people have been thinking about this problem.  The 
problem is how to define a container and how to track a namespace.
There have been a number of patches submitted in the last couple of
years to try to address this, but discussion and development are still
on-going.

> But, after reading the audit's documentation, looks like right now any of
> these options are supported without applying some unofficial patch.
> 
> A work-around would be to automatically add a new rule per new child, but
> the new processes wouldn't be monitoring the time since its start until the
> creation the rule, causing race conditions (creation of a new child, that
> would not be detected, etc.)
> 
> Are you aware of some other option that would allow what we need?

Not yet, but that problem definitely needs to be solved, so please keep
the conversation going.

One goal is to be able to run an audit daemon in a container, isolated
from the master audit daemon on the system.  It would be able to monitor
activity in that container and any other containers spawned by it, but
not be able to change the configuration of any that are containing it.

> Thanks in advance.
> 
> Regards,
> Roi

> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit


- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545