Audit dispatcher process

Audit dispatcher process

[James Antill]

[-- Attachment #1.1: Type: text/plain, Size: 2058 bytes --]


 Steve has asked me to write the audit dispatcher, and after talking
about it we already have some plans (as you'll see below :) but we would
welcome input from people on this list.
 First to bring you all up to speed with what we know:


. Development should be starting soon.

. It will, at least initially, be distributed as part of the audit
package.

. We are planning to have a usable version for Fedora 7.

. That initial version will be able to act as the dispatcher for auditd
and (re-)send those messages to multiple plugins.

. Those plugins can be shipped separately.


...and what seems very likely:


. The plugins will be external applications.

. The dispatcher itself will not be parsing audit messages and will be
designed as a kind of Publish/Subscribe daemon.

. In that vein, reuse of code from And-httpd/Vstr/etc.[1] is more than
very likely.

. The dispatcher will only be doing minimal content filtering for the
plugins (this kind of falls out from the minimal parsing).

. That message input will come from plugins, as well as the output.

. They'll be a mode for the plugin to run in where it speaks a
mini-protocol with the dispatcher, instead of just getting raw messages
from auditd.

. That the mini-protocol will allow "commands" to go back to the
dispatcher (think remote server says "out of disk space, do X" or IDS
says "attack happening from IP block X/y, do Z").

. The initial set of plugins will contain at least something to connect
the dispatcher to setroubleshootd and something for (secure) remote
logging.


 I've probably missed something already, so if there's anything you want
that isn't on the above list or anything that isn't clear and you want
to clarify ... just hit reply :).


[1] http://www.and.org/and-httpd/ and http://www.and.org/vstr/

-- 
James Antill - <james.antill@redhat.com>
setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, ...);
setsockopt(fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, ...);
setsockopt(fd, SOL_SOCKET,  SO_ATTACH_FILTER, ...);


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Audit dispatcher process

[Steve Grubb]

On Thursday 11 January 2007 00:58, James Antill wrote:
> It will, at least initially, be distributed as part of the audit
> package.

It will be distributed in the audit package as will a set of common plugins. 
Third parties can write their own just as pam modules can be distributed for 
the pam system.

> Those plugins can be shipped separately.

Well, a common set will be part of the audit system. See below.

> . The plugins will be external applications.

Yes, to be good selinux citizens, this will be necessary.

> . That message input will come from plugins, as well as the output.

The idea is that the dispatcher and plugins will be the transport mechanism 
for log aggregation. Most machines would have as sending plugin, while one 
machine would be receiving the events for logging. There is also a desire to 
be able to pull in iptables events for other plugins that would like to 
analyze these events for security purposes.

> . They'll be a mode for the plugin to run in where it speaks a
> mini-protocol with the dispatcher, instead of just getting raw messages
> from auditd.
>
> . That the mini-protocol will allow "commands" to go back to the
> dispatcher (think remote server says "out of disk space, do X" or IDS
> says "attack happening from IP block X/y, do Z").

Right. If we are doing remote logging and the remote machine runs out of disk 
space, the plugin will have to handle the admin selected action such as 
single user mode, halt, etc. I do not want a 2 way communication system with 
the audit daemon itself. This complicates its code for CAPP/LSPP analysis and 
testing. So, the receive plugin will need to ack everything its getting and 
the senders will need to wait for that ack. Also, need to consider what to do 
when reboot of the aggregator occurs. Should they spool, if so how much, etc.

> . The initial set of plugins will contain at least something to connect
> the dispatcher to setroubleshootd and something for (secure) remote
> logging.

Right, I see these as the initial common set:

1) setroubleshootd adapter to feed events to it.
2) remote logging plugin to send events to remote machine. This should use ssl 
to keep events private.
3) syslog plugin - simply takes events and writes to syslog
4) iptables event input plugin
5) remote logging receive plugin
6) possibly a filter plugin that can do matching for certain kinds of events 
and send notification when it sees that event. (not a high priority and could 
be added later depending on how long the others take.) Analogous to grep.


Other requirements:

plugin installation should be such that they are installed in a way that does 
not require editing the master config file. Perhaps each plugin is required 
to put a config file in /etc/audipsd.d directory which includes at a minimum 
path to binary and whether or not its enabled (like xinetd).

Dependencies on external libraries should be kept to a minimum and optional by 
configure options.

dispatcher interface with auditd has to conform to guidelines here:
http://people.redhat.com/sgrubb/audit/audit-rt-events.txt

config file parsing should use libaudit's parsing code. I'd like to keep a 
consistent style among pieces of the audit system.


If I think of other things I'll add them later.

Thanks,
-Steve

Re: Audit dispatcher process

[James Antill]

[-- Attachment #1.1: Type: text/plain, Size: 3421 bytes --]

On Thu, 2007-01-11 at 09:56 -0500, Steve Grubb wrote:

> So, the receive plugin will need to ack everything its getting and 
> the senders will need to wait for that ack. Also, need to consider what to do 
> when reboot of the aggregator occurs. Should they spool, if so how much, etc.

 This is all contained in the remote plugin, or do you want a general
ACK'ing process all the way back to any input plugins. Ie.

input plugin =>
 dispatcher =>
  remote logging plugin => 
   remote server =>
    disk "ACK" => 
   remote server <=
  remote logging plugin <=
 -- stop here? --
 dispatcher <=
 -- or stop here? --
input plugin <=
 -- or even stop here? --

...even if we just stop at the remote logging plugin level, are we
worried about duplicate entries on the logging server?
 I guess that's better than missing entries if the power fails on the
logging server (I just don't want to have to solve the two generals
problem :).

> Other requirements:
> 
> plugin installation should be such that they are installed in a way that does 
> not require editing the master config file. Perhaps each plugin is required 
> to put a config file in /etc/audipsd.d directory which includes at a minimum 
> path to binary and whether or not its enabled (like xinetd).

 Yeh, I'm meant to say that with the shipped separately point. One thing
I'm not sure about is if there's a requirement to pickup new
changes/plugins without having to "reboot" the dispatcher in some way.
 I'd also thought of having the auditd input be a plugin itself, this
way you could quickly reboot the dispatcher while the auditd input
plugin spooled the incoming data for a small period of time. This might
make signal handling simpler too, although it's certainly not in the
spirit of the dispatcher :).

> Dependencies on external libraries should be kept to a minimum and optional by 
> configure options.

 Well, if by optional you mean "something doesn't get built". Then, yeh,
I'd planned to do that ... but I'm not sure how useful it'll be, esp.
with regard to any deps needed for the dispatcher itself.

> dispatcher interface with auditd has to conform to guidelines here:
> http://people.redhat.com/sgrubb/audit/audit-rt-events.txt

 Ahh thanks, I'd looked at the libaudit.h header and worked out most of
the above but the clarification is appreciated.
 One question though: Should I just take the first 4 bytes, and compare
to zero then take the next 12 ... or just grab the first 16, I figured
from the .h file that the first 16 will always be there but the above
page suggests just checking the first 32 bits (note s/32 bytes/32 bits/
to fix the typo).

> config file parsing should use libaudit's parsing code. I'd like to keep a 
> consistent style among pieces of the audit system.

 From what I can see of audit.conf and that code, it would be extremely
hard to use for what we'd want to do with plugins. IMO the audit daemon
and the dispatcher do very different jobs, so a different config. file
syntax isn't a problem, and setroubleshoot already has a totally
different config. style.
 The conf code from And-httpd should be very easy to reuse for this
(it's used to configure similar types of applications), already supports
conf.d directories, already supports building paths from pieces of
dynamic information etc.

-- 
James Antill <jantill@redhat.com>

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Audit dispatcher process

[Steve Grubb]

On Thursday 11 January 2007 14:13, James Antill wrote:
> On Thu, 2007-01-11 at 09:56 -0500, Steve Grubb wrote:
> > So, the receive plugin will need to ack everything its getting and
> > the senders will need to wait for that ack. Also, need to consider what
> > to do when reboot of the aggregator occurs. Should they spool, if so how
> > much, etc.
>
>  This is all contained in the remote plugin, or do you want a general
> ACK'ing process all the way back to any input plugins. 

I'm thinking just the two that do the remote logging transfer. 

> ...even if we just stop at the remote logging plugin level, are we
> worried about duplicate entries on the logging server?

I'd rather not have them.

>  I guess that's better than missing entries if the power fails on the
> logging server (I just don't want to have to solve the two generals
> problem :).

True. But we don't want to make something too complicated either. Complicated 
makes it hard to analyze for failure modes and effects.

> > Other requirements:
> >
> > plugin installation should be such that they are installed in a way that
> > does not require editing the master config file. Perhaps each plugin is
> > required to put a config file in /etc/audipsd.d directory which includes
> > at a minimum path to binary and whether or not its enabled (like xinetd).
>
>  Yeh, I'm meant to say that with the shipped separately point. One thing
> I'm not sure about is if there's a requirement to pickup new
> changes/plugins without having to "reboot" the dispatcher in some way.

Yes. The desired way to do this is SIGHUPing the audit daemon. It will inform 
the dispatcher and it will do its magic.

>  I'd also thought of having the auditd input be a plugin itself, this
> way you could quickly reboot the dispatcher while the auditd input
> plugin spooled the incoming data for a small period of time.

No need for this. The audit daemon has to start the dispatcher so it will have 
a pipe between the processes. There's no way to insert a plugin between the 
two without making life complicated.

> > Dependencies on external libraries should be kept to a minimum and
> > optional by configure options.
>
>  Well, if by optional you mean "something doesn't get built". Then, yeh,

right.

> I'd planned to do that ... but I'm not sure how useful it'll be, esp.
> with regard to any deps needed for the dispatcher itself.

There should be no external libraries needed for it. Its too simple to need 
anything external. The plugins may need openssl or nss. Actually nss might be 
a better choice since its FIPS 140-2 certified. But we can talk about that 
when we get there.

> One question though: Should I just take the first 4 bytes, and compare
> to zero then take the next 12 ... or just grab the first 16, 

I'd make a pointer of uint32_t type and cast it to the data's pointer. Then 
you can check its contents and decide if you know how to handle data of that 
type. Odds are it will always be in sync because its being distributed with 
the audit daemon. The note in the specification is there mostly for the 
benefit of third party dispatchers since the audit daemon could have been 
upgraded. So, in reality, I don't think you have to worry about it so much. 
Any other author will, though.

> > config file parsing should use libaudit's parsing code. I'd like to keep
> > a consistent style among pieces of the audit system.
>
>  From what I can see of audit.conf and that code, it would be extremely
> hard to use for what we'd want to do with plugins. IMO the audit daemon
> and the dispatcher do very different jobs, 

Yes.

> so a different config. file syntax isn't a problem, and setroubleshoot
> already has a totally different config. style.

Well, that's a whole different package, though.

> The conf code from And-httpd should be very easy to reuse for this
> (it's used to configure similar types of applications), already supports
> conf.d directories, already supports building paths from pieces of
> dynamic information etc.

Hmm...I hadn't wanted to pull in dependencies due to needing to do external 
code reviews. What kind of syntax needs do you think we really need? I'm open 
to ideas. The plugins I would envision might need to read their own config 
files. They might have complicated needs, but I don't think the dispatcher 
would.

-Steve

RE: Audit dispatcher process

[Todd, Charles]

James,
Even though the xinetd model for plugins was probably a no-brainer
design decision, I still agree that it's an excellent choice.  Thanks
for using that model.  I don't know if you want to have signal (of some
kind) that must be issued in order for a new plugin to take effect.
Certainly, the addition of a new plugin should be an audited event, so
perhaps the CAPP-suggested list automatically includes a write/execute
watch on this directory.

I have a small suggestion for the local logging plugin that will
probably be put in by default: 
* I recommend a file naming convention similar to that used under
Solaris.  That is <init_datetime>.<close_datetime>.<host>
* <init_datetime> for 11Jan2007 at 3:46pm would be 20070111154632
* <close_datetime> is "not_terminated" when a file is currently being
written, although this is doesn't work when the dispatcher dies
prematurely
* <host> is usually not the FQDN, but allows lots of audit trail files
from lots of machines to be in one directory.
* Solaris administrators issues a "audit -n" to rotate the logs when
it's time, and this is occassionally quite handy when archiving logs 
* Real-time compression (gzip) is a GREAT idea.  If the binary file
written by the local logging plugin wrote whole records at a time, then
a leading bit could indicate compression.  Being able to turn off
compression might help in high-load situations.
* Also being able to set a max file size before rotation becomes
mandatory makes it easy to archive audit logs to fixed-size media.

A suggestion I have for the plugin architecture is to allow the plugin
to query the dispatcher for internal statistics and system call
number-to-name lookups.  Some internal statistics might be audit buffer
ring size, or other useful information to deal with high-load
environments.  Most everything else, the plugin can ask the OS (system
date/time, system load, etc.)

Is the dispatcher responsible for restarting a plugin if the
process-killer targets it?

A useful plugin for any authors might be a UDP transmitter, with PKE for
all packets.  This would allow a centralized repository in a trusted
environment.

Another plugin might be a real-time stream-to-tape so that audit trail
retention is as easy as replacing a tape when it's ejected because it
was full.  By maintaining an open file descriptor at all times, other
processes will have a tough time overwriting audit trails.

Should anyone approach the Webmin authors once the API and directory
structure is locked in so that they can write a module?

Of course, I don't expect both of those to make it into the standard
distro, but rather maybe spark an interest among the group.

Thanks,
Charlie Todd 
Ball Aerospace & Technologies Corp.  
Ctodd- at -ball.com 
 

> -----Original Message-----
> From: linux-audit-bounces@redhat.com 
> [mailto:linux-audit-bounces@redhat.com] On Behalf Of James Antill
> Sent: Thursday, January 11, 2007 12:59 AM
> To: linux-audit@redhat.com
> Subject: Audit dispatcher process
> 
> 
>  Steve has asked me to write the audit dispatcher, and after 
> talking about it we already have some plans (as you'll see 
> below :) but we would welcome input from people on this list.
>  First to bring you all up to speed with what we know:
> 
> 
> . Development should be starting soon.
> 
> . It will, at least initially, be distributed as part of the 
> audit package.
> 
> . We are planning to have a usable version for Fedora 7.
> 
> . That initial version will be able to act as the dispatcher 
> for auditd and (re-)send those messages to multiple plugins.
> 
> . Those plugins can be shipped separately.
> 
> 
> ...and what seems very likely:
> 
> 
> . The plugins will be external applications.
> 
> . The dispatcher itself will not be parsing audit messages 
> and will be designed as a kind of Publish/Subscribe daemon.
> 
> . In that vein, reuse of code from And-httpd/Vstr/etc.[1] is 
> more than very likely.
> 
> . The dispatcher will only be doing minimal content filtering 
> for the plugins (this kind of falls out from the minimal parsing).
> 
> . That message input will come from plugins, as well as the output.
> 
> . They'll be a mode for the plugin to run in where it speaks 
> a mini-protocol with the dispatcher, instead of just getting 
> raw messages from auditd.
> 
> . That the mini-protocol will allow "commands" to go back to 
> the dispatcher (think remote server says "out of disk space, 
> do X" or IDS says "attack happening from IP block X/y, do Z").
> 
> . The initial set of plugins will contain at least something 
> to connect the dispatcher to setroubleshootd and something 
> for (secure) remote logging.
> 
> 
>  I've probably missed something already, so if there's 
> anything you want that isn't on the above list or anything 
> that isn't clear and you want to clarify ... just hit reply :).
> 
> 
> [1] http://www.and.org/and-httpd/ and http://www.and.org/vstr/
> 
> --
> James Antill - <james.antill@redhat.com> setsockopt(fd, 
> IPPROTO_TCP, TCP_CONGESTION, ...); setsockopt(fd, 
> IPPROTO_TCP, TCP_DEFER_ACCEPT, ...); setsockopt(fd, 
> SOL_SOCKET,  SO_ATTACH_FILTER, ...);
> 
> 

Re: Audit dispatcher process

[Steve Grubb]

On Thursday 11 January 2007 16:03, Todd, Charles wrote:
> I don't know if you want to have signal (of some kind) that must be issued
> in order for a new plugin to take effect.

SIGHUP

> Certainly, the addition of a new plugin should be an audited event, so
> perhaps the CAPP-suggested list automatically includes a write/execute
> watch on this directory.

I have this feeling that the dispatcher may never be a part of a CAPP eval. 
That's why I separated them. But I think the startup code could log plugins 
and changes in config.

> I have a small suggestion for the local logging plugin that will
> probably be put in by default:
> * I recommend a file naming convention similar to that used under
> Solaris.  That is <init_datetime>.<close_datetime>.<host>
> * <init_datetime> for 11Jan2007 at 3:46pm would be 20070111154632
> * <close_datetime> is "not_terminated" when a file is currently being
> written, although this is doesn't work when the dispatcher dies
> prematurely

I'd rather keep the log names simple like it is. I've put some effort into 
making the tools tell you what's in each log file.

> * <host> is usually not the FQDN, but allows lots of audit trail files
> from lots of machines to be in one directory.

I had envisioned 1 unified log. No splitting per machine except by ausearch.

> * Solaris administrators issues a "audit -n" to rotate the logs when
> it's time, and this is occassionally quite handy when archiving logs

this audit system just needs "service auditd rotate"

> * Real-time compression (gzip) is a GREAT idea.

That's in the todo list and targeted around 1.3.3.

> A suggestion I have for the plugin architecture is to allow the plugin
> to query the dispatcher for internal statistics and system call
> number-to-name lookups.

The idea is not to complicate transport and archive with analysis, 
presentation, and display. I want to keep the plugins simple and lean. This 
is to make sure they are reliable. Third party's can go off and write 
something as complicated as they wish. But whatever is distributed in the 
audit package should be simple and robust.

> Some internal statistics might be audit buffer ring size, or other useful
> information to deal with high-load environments.

auditctl -s  ?

> Is the dispatcher responsible for restarting a plugin if the
> process-killer targets it?

Yes. It should keep them alive. Good point.

-Steve