Advice on enriching logs with user and group names before moving them to a central log repository

Advice on enriching logs with user and group names before moving them to a central log repository

[Burn Alting]

[-- Attachment #1.1: Type: text/plain, Size: 1412 bytes --]

Hi,

I have a scenario of a mixed collection of Linux systems, some that have
users authenticate via a central ldap, others have local (/etc/passwd)
authentication.
This means I cannot 100% depend that the user name say, fred, with uid
1000, has the same uid on every machine he has an account on.  Thus
before I send my logs to
a central server, I want to enrich them with user and group names I
validate at the local machine. That is, I want to change an event's ids
from

        .... uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=43
        sgid=43 fsgid=43 ....

to

        .... uid=1000(fred) gid=1000(prog) euid=1000(fred)
        suid=1000(fred) fsuid=1000(fred) egid=43(utmp) sgid=43(utmp)
        fsgid=43(utmp) ....


I BELIEVE my best approach is use the event multiplexor (audispd) to
convert raw logs via a child program, say based on the sample code,
audisp-example  (i.e. using the auparse library)
and send the output  of this audisp-example variant  to syslog to get
the event to a central repository.

Is this the best approach? 

Are there parameters I should consider for audisp.conf (e.g. q_depth =
99999)? Does such a configuration option in audisp.conf suggest I make
the buffer size set in audit.rules to something higher?

Is there any consideration to having auditd have a option to directly
generate user and group names in addition to uid and gids?

Thanks in advance

Burn

[-- Attachment #1.2: Type: text/html, Size: 1759 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[John Dennis]

On 08/02/2012 06:54 AM, Burn Alting wrote:
> Hi,
>
> I have a scenario of a mixed collection of Linux systems, some that have
> users authenticate via a central ldap, others have local (/etc/passwd)
> authentication.
> This means I cannot 100% depend that the user name say, fred, with uid
> 1000, has the same uid on every machine he has an account on.  Thus
> before I send my logs to
> a central server, I want to enrich them with user and group names I
> validate at the local machine. That is, I want to change an event's ids from
>
>     .... uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=43
>     sgid=43 fsgid=43 ....
>
> to
>
>     .... uid=1000(fred) gid=1000(prog) euid=1000(fred) suid=1000(fred)
>     fsuid=1000(fred) egid=43(utmp) sgid=43(utmp) fsgid=43(utmp) ....
>
>
> I BELIEVE my best approach is use the event multiplexor (audispd) to
> convert raw logs via a child program, say based on the sample code,
> audisp-example  (i.e. using the auparse library)
> and send the output  of this audisp-example variant  to syslog to get
> the event to a central repository.
>
> Is this the best approach?
>
> Are there parameters I should consider for audisp.conf (e.g. q_depth =
> 99999)? Does such a configuration option in audisp.conf suggest I make
> the buffer size set in audit.rules to something higher?
>
> Is there any consideration to having auditd have a option to directly
> generate user and group names in addition to uid and gids?

A while ago we were actively working on central log aggregation and ran 
into exactly this problem. There are a number of items in an audit log 
whose value can only be interpreted on the machine the event occurred on 
and at the moment the event occurs (or within a short duration).

There were plans to author a audit plugin that would augment the data 
items with their (interpreted) value. I'm not sure whatever happened to 
that plugin. Steve, can you elaborate?


-- 
John Dennis <jdennis@redhat.com>

Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Guillaume Destuynder]

I'm doing something similar on an audisp plugin as you mentionned. It's
part of a different plugin that changes the log format (to CEF) and does
a few other things, so unfortunately only some snippets would help you.

For user names eg:
        auid = auparse_find_field(au, "auid");
        if (auid) {	
         i = auparse_get_field_int(au);
          if (i != -1)
            if (getpwuid_r(i, &pwd, buf, bufsize, &result) == NULL)
               //too late

The functions available for the plugin interface really make making your
own plugins very easy :)

Works ok except for the ppid. Not sure how to get the ppid's process
name in userspace other than reading /proc and in any case it happens
the parent process died before you read the name. It would need to be
passed from the kernel to be more reliable.

Note that the same issue exists for uids, it's just that its a lot more
rare: user would need to be deleted between the uid audit message is
passed and the name lookup.

It might still be an idea to have auparse_get_uid(au) etc.


Guillaume

On 08/02/2012 06:54 AM, John Dennis wrote:
> On 08/02/2012 06:54 AM, Burn Alting wrote:
>> Hi,
>>
>> I have a scenario of a mixed collection of Linux systems, some that have
>> users authenticate via a central ldap, others have local (/etc/passwd)
>> authentication.
>> This means I cannot 100% depend that the user name say, fred, with uid
>> 1000, has the same uid on every machine he has an account on.  Thus
>> before I send my logs to
>> a central server, I want to enrich them with user and group names I
>> validate at the local machine. That is, I want to change an event's
>> ids from
>>
>>     .... uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=43
>>     sgid=43 fsgid=43 ....
>>
>> to
>>
>>     .... uid=1000(fred) gid=1000(prog) euid=1000(fred) suid=1000(fred)
>>     fsuid=1000(fred) egid=43(utmp) sgid=43(utmp) fsgid=43(utmp) ....
>>
>>
>> I BELIEVE my best approach is use the event multiplexor (audispd) to
>> convert raw logs via a child program, say based on the sample code,
>> audisp-example  (i.e. using the auparse library)
>> and send the output  of this audisp-example variant  to syslog to get
>> the event to a central repository.
>>
>> Is this the best approach?
>>
>> Are there parameters I should consider for audisp.conf (e.g. q_depth =
>> 99999)? Does such a configuration option in audisp.conf suggest I make
>> the buffer size set in audit.rules to something higher?
>>
>> Is there any consideration to having auditd have a option to directly
>> generate user and group names in addition to uid and gids?
> 
> A while ago we were actively working on central log aggregation and ran
> into exactly this problem. There are a number of items in an audit log
> whose value can only be interpreted on the machine the event occurred on
> and at the moment the event occurs (or within a short duration).
> 
> There were plans to author a audit plugin that would augment the data
> items with their (interpreted) value. I'm not sure whatever happened to
> that plugin. Steve, can you elaborate?
> 
> 

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Miloslav Trmac]

----- Original Message -----
> It might still be an idea to have auparse_get_uid(au) etc.

I'm not 100% sure what you mean, but is perhaps auparse_interpret_field what you are looking for?  It returns an "intepreted" (as opposed to "raw") version of the field, e.g. a name instead of an UID.
    Mirek

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[John Dennis]

On 08/02/2012 05:12 PM, Miloslav Trmac wrote:
> I'm not 100% sure what you mean, but is perhaps
> auparse_interpret_field what you are looking for?  It returns an
> "intepreted" (as opposed to "raw") version of the field, e.g. a name
> instead of an UID.

Yes, that's the correct function to call. However it should be done by a 
plugin which iterates over all the items and adds an interpreted result 
to the raw result. For long term detached audit purposes you need both 
the raw and interpreted value. The plugin then emits the augmented data 
containing both the raw and interpreted values.

-- 
John Dennis <jdennis@redhat.com>

Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Steve Grubb]

On Thursday, August 02, 2012 09:54:46 AM John Dennis wrote:
> There were plans to author a audit plugin that would augment the data 
> items with their (interpreted) value. I'm not sure whatever happened to 
> that plugin. Steve, can you elaborate?

This is a problem and I think about it every now and then. But there are 
bigger problems first.

-Steve

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Burn Alting]

[-- Attachment #1.1: Type: text/plain, Size: 1708 bytes --]

Steve,

I will go ahead with my audispd child program that enriches logs and use
rsyslog to get them to a central repository.
I also plan to concatenate all messages belonging to the same event (ie
time:event_id) and send this as one syslog message to the central
repository.
I'd rather do this on the client systems rather than at my central
repository, in order to gain benefits from effectively, distributed
processing.

I have some concerns though:
    - Does the concatenation of messages belonging to one event, outside
of bad code on my part, have some non-obvious risks (from those of you
who have done this?)
    - I intend that my code will have as small an overhead as I can, but
do I risk issues such as overruns of the audispd queue?
    - Do messages from different events ever get intermixed in the
output via audispd? And hence I need to cater for multiple simultaneous
events streaming in?


I will contribute my code to this list for what's it worth once I've
completed it ... perhaps it can be added to the contrib/plugin tree
given it passes this list's peer review.

Guillaume, 

One element of my central repository will take these 'enriched logs' and
map them into CEF also, so I'd be interested in any mappings you are
making.

Thanks in advance.
Burn

 On Mon, 2012-08-06 at 13:51 -0400, Steve Grubb wrote:

> On Thursday, August 02, 2012 09:54:46 AM John Dennis wrote:
> > There were plans to author a audit plugin that would augment the data 
> > items with their (interpreted) value. I'm not sure whatever happened to 
> > that plugin. Steve, can you elaborate?
> 
> This is a problem and I think about it every now and then. But there are 
> bigger problems first.
> 
> -Steve



[-- Attachment #1.2: Type: text/html, Size: 2147 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Michael Mather]

On Fri, 2012-08-10 at 19:51 +1000, Burn Alting wrote:
> Steve,
> 
> I will go ahead with my audispd child program that enriches logs and
> use rsyslog to get them to a central repository.
> I also plan to concatenate all messages belonging to the same event
> (ie time:event_id) and send this as one syslog message to the central
> repository.
> I'd rather do this on the client systems rather than at my central
> repository, in order to gain benefits from effectively, distributed
> processing.
> 

This sounds very useful, Burn.

In an EXECVE message there is something like:

args=2 a0="ls" a1="/etc"

It would be nice if this could be changed to something like

command="ls /etc".

One problem is that the shell script interprets wild cards before auditd
sees the command, and that can lead to long strings. So maybe that
situation could become something like:

something="ls /etc/aaa /etc/bbb /etc/ccc ..."

In most cases a human reader would recognise what is happening.

Also, sometimes the parameters are in hex instead of strings. For
example, when the parameter contains quotes.

Michael
-------

Re: Advice on enriching logs with user and group names before moving them to a central log repository

[Steve Grubb]

On Friday, August 10, 2012 07:51:29 PM Burn Alting wrote:
> Steve,
> 
> I will go ahead with my audispd child program that enriches logs and use
> rsyslog to get them to a central repository.
> I also plan to concatenate all messages belonging to the same event (ie
> time:event_id) and send this as one syslog message to the central
> repository.
> I'd rather do this on the client systems rather than at my central
> repository, in order to gain benefits from effectively, distributed
> processing.
> 
> I have some concerns though:
>     - Does the concatenation of messages belonging to one event, outside
> of bad code on my part, have some non-obvious risks (from those of you
> who have done this?)

The only problem might be that you will no longer be able to use any of the 
native reporting tools. If you don't use them anyways, then no problem.


>     - I intend that my code will have as small an overhead as I can, but
> do I risk issues such as overruns of the audispd queue?

Yes. You need to make it multi threaded if you do experience overflows with one 
thread dequeueing and another processing.


>     - Do messages from different events ever get intermixed in the
> output via audispd? And hence I need to cater for multiple simultaneous
> events streaming in?

Yes. This is a big problem. About 2 years ago I fixed this in ausearch/report. 
I started to fix this in libauparse but then I remembered it has this state 
machine in it to deal with the feed interface. I didn't write that code so it 
will take some time for me to figure out what it doing before fixing this 
problem. But basically you need a list of lists where each list is a 
collection of records that form one event.
 
 
> I will contribute my code to this list for what's it worth once I've
> completed it ... perhaps it can be added to the contrib/plugin tree
> given it passes this list's peer review.

I do plan to solve this problem at some point. Fixing the libauparse issue 
mentioned above is higher on my priority list.

-Steve