Offline audit trail analysis

Offline audit trail analysis

[Todd, Charles]

[-- Attachment #1.1: Type: text/plain, Size: 1955 bytes --]

Thanks to Steve for being our biggest target for questions on this list!

 
Has anyone talked about sane ways to do offline analysis of Linux audit
logs?  Presumably, this would be on another Linux system, but maybe not
the same host, and probably not on the same release or with the same
username/IP address access.  Conceptually, ausearch would save and
optionally read a system's "configuration" to be saved for
interpretation later.  
 
My goal is central logging, but doing the reporting/analysis on the
central host.  That way, I can see a user across the Enterprise (or at
least in the Linux hosts), but with all the power of ausearch for
refining the report.  Ideally, I would do an ausearch -ts <date> -te
<date> --raw --config-to=<hostname.ausearch.config> and it would do
things like saving the syscall lookup table, lookup users referenced in
the reported audit trail, and resolve IP addresses references in the
reported audit trail.  Maybe one config file could be written for each
data type in an existing format (e.g. users in /etc/passwd format, hosts
in /etc/hosts format, etc.).  I'm mainly after whether or not anyone has
considered extending ausearch for this kind of processing?
 
This way, an archive of raw logs could be kept along with the exact
system configuration which allows offloading the audit trail analysis to
a trusted location, rather than risk side effects from a rootkit.
 
Charlie Todd 
Ball Aerospace & Technologies Corp.  




This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.

[-- Attachment #1.2: Type: text/html, Size: 3020 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Offline audit trail analysis

[Steve Grubb]

On Tuesday 11 September 2007 15:31:53 Todd, Charles wrote:
> Has anyone talked about sane ways to do offline analysis of Linux audit
> logs?  Presumably, this would be on another Linux system, but maybe not
> the same host, and probably not on the same release or with the same
> username/IP address access.  Conceptually, ausearch would save and
> optionally read a system's "configuration" to be saved for
> interpretation later.

ausearch uses getpw* and getgr* calls to resolve the uid & gid. Aside from 
that, I believe it is self contained. (I haven't dug too deep in answering 
this question.)


> My goal is central logging, but doing the reporting/analysis on the
> central host.  That way, I can see a user across the Enterprise (or at
> least in the Linux hosts), but with all the power of ausearch for
> refining the report.

That is in the works...

> Ideally, I would do an ausearch -ts <date> -te 
<date> --raw --config-to=<hostname.ausearch.config> and it would do
> things like saving the syscall lookup table,

This is actually inside libaudit for all arches.

> lookup users referenced in the reported audit trail, 

uid & gid are based on the host's /etc/nsswitch.conf settings. If file, then 
its the local passwd/group files. If not, you have a central uid database.

> and resolve IP addresses references in the reported audit trail.

Hmm. As long as systems don't get changed too much, this should be resolvable 
from anything that has DNS access.

> Maybe one config file could be written for each data type in an existing
> format (e.g. users in /etc/passwd format, hosts in /etc/hosts format, etc.). 
> I'm mainly after whether or not anyone has considered extending ausearch for
> this kind of processing?

Sort of. I am working towards central logging. Getting the new event 
dispatcher completed is the first step. But I haven't looked at storing user 
and gid away yet. The calls that I'm using don't really allow substituting a 
new passwd or group file. So, I'd have to change all that code. But if you 
have an enterprise setup, you shouldn't be deleting user IDs to keep from 
having a collision down the road.

> This way, an archive of raw logs could be kept along with the exact
> system configuration which allows offloading the audit trail analysis to
> a trusted location, rather than risk side effects from a rootkit.

Yep.

-Steve

RE: Offline audit trail analysis

[Wieprecht, Karen M.]

[-- Attachment #1.1: Type: text/plain, Size: 8948 bytes --]

This isn't probably as savvy as some of you would like, but I started
writing it before there were any auparse libraries to help with this
kind of stuff.    I've done similar stuff on Solaris, Mac OSX, irix,
etc.   My linux version scripting isn't finished (not enough hours in
the day), but it's still useful for my log reviews.  At some point I
hope to get out internal people to bless it enough to share with whoever
might be interested in the open source world,  but I'm not there yet.
here's the general thing I've  setting up:
 
I use scripting to do the following on each machine :
 
- rotate the log 
- (I want to try a script segment to make sure all related audit event
records are not interleaved)
- run an ausearch on it with some flags (interpret, all messages) and
some shell commands to get all related records for a particular event
onto a single line
- save this processed file (with user IDs, etc. particular to that
system) as something like audit.log.hostname.2006-09-10.ausearched
- Store the original and "ausearched" files in a central place so I have
them all gathered for my weekly log review
 
Then weekly,  I use a "reviewlogs" script to post process the
"ausearched" "single line" data into something more readable.   It also
filters some noise I don't care about.  The Chronological results look
like this from various test sessions:
 
karen    FAILED TO     mv /etc/nsswitch.conf /etc/nsswitch.conf.mine
on patton,      Permission denied       on 01/15/2007 14:05:16.582
 
dan      FAILED TO     console login   into patton on 01/17/2007
13:21:45.133
dan      successful      console login   into patton on 01/17/2007
13:21:59.152
dan      successful      console logout          from patton on
01/17/2007 13:22:26.059

karen    FAILED TO     ssh login       into patton from ? on 01/17/2007
13:34:26.517
karen    successful      ssh login       into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,
Permission denied       on 01/17/2007 13:34:57.759
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,
Permission denied       on 01/17/2007 13:34:57.759
karen    successful      SU LOGIN (PAM authentication)   as root
on patton on 01/17/2007 13:35:15.140
karen    successful      SU LOGIN (PAM accounting)       as root
on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM session open)     as root
on patton on 01/17/2007 13:35:15.141
 -- karen as root --     successful      SU LOGIN (PAM authentication)
as dan          on patton on 01/17/2007 13:35:31.364
 -- karen as root --     successful      SU LOGIN (PAM accounting)
as dan          on patton on 01/17/2007 13:35:31.366
 -- karen as root --     successful      SU LOGIN (PAM session open)
as dan          on patton on 01/17/2007 13:35:31.367
 -- karen as dan --      FAILED TO       rm /etc/hosts           on
patton,      Permission denied       on 01/17/2007 13:35:39.129
 -- karen as dan --      FAILED TO       rm /etc/hosts           on
patton,      Permission denied       on 01/17/2007 13:35:40.680
 -- karen as root --     successful      SU LOGIN (PAM setcred)
as dan          on patton on 01/17/2007 13:35:44.177
 -- karen as root --     successful      CLOSED SU as dan        on
patton on 01/17/2007 13:35:44.178
karen    successful      SU LOGIN (PAM setcred)          as root
on patton on 01/17/2007 13:35:46.089
karen    successful      CLOSED SU as root       on patton on 01/17/2007
13:35:46.089

Note that it's easy to spot users trying to make it look like other
users tried to do stuff.    When I review the logs, I usually sort the
reformatted lines  so that similar entries are all grouped (which makes
it really easy to let chunks of stuff I don't care about scroll by
quickly).  If I see something suspicious, I'll go back and look at the
stuff in chronological order to see what was going on.   Here's what the
above sample looks like sorted  (and if I had a lot of data, the chunks
of similar stuff would be more apparent):
 
 -- karen as dan --      FAILED TO       rm /etc/hosts           on
patton,      Permission denied       on 01/17/2007 13:35:39.129
 -- karen as dan --      FAILED TO       rm /etc/hosts           on
patton,      Permission denied       on 01/17/2007 13:35:40.680
 -- karen as root --     successful      CLOSED SU as dan        on
patton on 01/17/2007 13:35:44.178
 -- karen as root --     successful      SU LOGIN (PAM accounting)
as dan          on patton on 01/17/2007 13:35:31.366
 -- karen as root --     successful      SU LOGIN (PAM authentication)
as dan          on patton on 01/17/2007 13:35:31.364
 -- karen as root --     successful      SU LOGIN (PAM session open)
as dan          on patton on 01/17/2007 13:35:31.367
 -- karen as root --     successful      SU LOGIN (PAM setcred)
as dan          on patton on 01/17/2007 13:35:44.177
dan      FAILED TO     console login   into patton on 01/17/2007
13:21:45.133
dan      successful      console login   into patton on 01/17/2007
13:21:59.152
dan      successful      console logout          from patton on
01/17/2007 13:22:26.059
karen    FAILED TO     mv /etc/nsswitch.conf /etc/nsswitch.conf.mine
on patton,      Permission denied       on 01/15/2007 14:05:16.582
karen    FAILED TO     ssh login       into patton from ? on 01/17/2007
13:34:26.517
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,
Permission denied       on 01/17/2007 13:34:57.759
karen    FAILED TO     touch /etc/nsswitch.conf        on patton,
Permission denied       on 01/17/2007 13:34:57.759
karen    successful      CLOSED SU as root       on patton on 01/17/2007
13:35:46.089
karen    successful      SU LOGIN (PAM accounting)       as root
on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM authentication)   as root
on patton on 01/17/2007 13:35:15.140
karen    successful      SU LOGIN (PAM session open)     as root
on patton on 01/17/2007 13:35:15.141
karen    successful      SU LOGIN (PAM setcred)          as root
on patton on 01/17/2007 13:35:46.089
karen    successful      ssh login       into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387

 
We don't have a ton of Linux systems (yet), so this strategy works well
for reviewing a handful of systems from a central location.    The
scripts save me a lot of time interpretting any one given log entry as
well as keeping me from having to log into N systems one at a time  and
running M commands on each one each to decide if someone was messing
around on my machines or not. 
 
Karen Wieprecht

________________________________

From: linux-audit-bounces@redhat.com
[mailto:linux-audit-bounces@redhat.com] On Behalf Of Todd, Charles
Sent: Tuesday, September 11, 2007 3:32 PM
To: linux-audit@redhat.com
Subject: Offline audit trail analysis


Thanks to Steve for being our biggest target for questions on this list!

 
Has anyone talked about sane ways to do offline analysis of Linux audit
logs?  Presumably, this would be on another Linux system, but maybe not
the same host, and probably not on the same release or with the same
username/IP address access.  Conceptually, ausearch would save and
optionally read a system's "configuration" to be saved for
interpretation later.  
 
My goal is central logging, but doing the reporting/analysis on the
central host.  That way, I can see a user across the Enterprise (or at
least in the Linux hosts), but with all the power of ausearch for
refining the report.  Ideally, I would do an ausearch -ts <date> -te
<date> --raw --config-to=<hostname.ausearch.config> and it would do
things like saving the syscall lookup table, lookup users referenced in
the reported audit trail, and resolve IP addresses references in the
reported audit trail.  Maybe one config file could be written for each
data type in an existing format (e.g. users in /etc/passwd format, hosts
in /etc/hosts format, etc.).  I'm mainly after whether or not anyone has
considered extending ausearch for this kind of processing?
 
This way, an archive of raw logs could be kept along with the exact
system configuration which allows offloading the audit trail analysis to
a trusted location, rather than risk side effects from a rootkit.
 
Charlie Todd 
Ball Aerospace & Technologies Corp.  


This message and any enclosures are intended only for the addressee.
Please  
notify the sender by email if you are not the intended recipient.  If
you are  
not the intended recipient, you may not use, copy, disclose, or
distribute this  
message or its contents or enclosures to any other person and any such
actions  
may be unlawful.  Ball reserves the right to monitor and review all
messages  
and enclosures sent to or from this email address.

[-- Attachment #1.2: Type: text/html, Size: 17071 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

RE: Offline audit trail analysis

[Todd, Charles]

[-- Attachment #1.1: Type: text/plain, Size: 11370 bytes --]

Karen,
I'm most impressed.  I've been rewriting the audit trails as English
text, so I can shove them into syslog and a detailed review could have
more context.  I REALLY, REALLY like your --karen as root-- and --karen
as dan-- lines.  Very impressive.  I've also toyed in my head with
strings that might look like --karen as dan-- via
ssh@badhost.hacker.com, but so much state information has to be kept
that might not be available when things roll off the archive.  You've
obviously put some thought into usability.  I added a keyword such as
AUDIT_SRO to every record where I was testing access to
security-relevant objects which makes for an easy grep later.  I also
include the raw event ID for more details if I want them later.
 
My focus is the investigator looking at my audit trails four years from
now trying to see the kinds of things an individual did while on my
network.  Presumably, they've been identified as an insider threat and
an investigation is underway.  I'm certain to have changed IP address
resolution and user names around.  OS upgrades will certainly have
happened.  
 
Thanks for the reponse and continued ideas.  My idea for getting "the
boss" to let me share is to go to my security officer and say "if I
release this code, it will help increase uniformity in the community
will make the higher-echelon security folks feel like we know what we're
doing in our shop."  Once, they're on board with sharing, then your
management chain might have fewer arguments.  (we can take that
discussion offline if you want)

Charlie Todd 
Ball Aerospace & Technologies Corp.  


 


________________________________

	From: Wieprecht, Karen M. [mailto:Karen.Wieprecht@jhuapl.edu] 
	Sent: Tuesday, September 11, 2007 4:59 PM
	To: Todd, Charles; linux-audit@redhat.com
	Subject: RE: Offline audit trail analysis
	
	
	This isn't probably as savvy as some of you would like, but I
started writing it before there were any auparse libraries to help with
this kind of stuff.    I've done similar stuff on Solaris, Mac OSX,
irix, etc.   My linux version scripting isn't finished (not enough hours
in the day), but it's still useful for my log reviews.  At some point I
hope to get out internal people to bless it enough to share with whoever
might be interested in the open source world,  but I'm not there yet.
here's the general thing I've  setting up:
	 
	I use scripting to do the following on each machine :
	 
	- rotate the log 
	- (I want to try a script segment to make sure all related audit
event records are not interleaved)
	- run an ausearch on it with some flags (interpret, all
messages) and some shell commands to get all related records for a
particular event onto a single line
	- save this processed file (with user IDs, etc. particular to
that system) as something like audit.log.hostname.2006-09-10.ausearched
	- Store the original and "ausearched" files in a central place
so I have  them all gathered for my weekly log review
	 
	Then weekly,  I use a "reviewlogs" script to post process the
"ausearched" "single line" data into something more readable.   It also
filters some noise I don't care about.  The Chronological results look
like this from various test sessions:
	 
	karen    FAILED TO     mv /etc/nsswitch.conf
/etc/nsswitch.conf.mine   on patton,      Permission denied       on
01/15/2007 14:05:16.582
	 
	dan      FAILED TO     console login   into patton on 01/17/2007
13:21:45.133
	dan      successful      console login   into patton on
01/17/2007 13:21:59.152
	dan      successful      console logout          from patton on
01/17/2007 13:22:26.059
	
	karen    FAILED TO     ssh login       into patton from ? on
01/17/2007 13:34:26.517
	karen    successful      ssh login       into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387
	karen    FAILED TO     touch /etc/nsswitch.conf        on
patton,      Permission denied       on 01/17/2007 13:34:57.759
	karen    FAILED TO     touch /etc/nsswitch.conf        on
patton,      Permission denied       on 01/17/2007 13:34:57.759
	karen    successful      SU LOGIN (PAM authentication)   as root
on patton on 01/17/2007 13:35:15.140
	karen    successful      SU LOGIN (PAM accounting)       as root
on patton on 01/17/2007 13:35:15.141
	karen    successful      SU LOGIN (PAM session open)     as root
on patton on 01/17/2007 13:35:15.141
	 -- karen as root --     successful      SU LOGIN (PAM
authentication)   as dan          on patton on 01/17/2007 13:35:31.364
	 -- karen as root --     successful      SU LOGIN (PAM
accounting)       as dan          on patton on 01/17/2007 13:35:31.366
	 -- karen as root --     successful      SU LOGIN (PAM session
open)     as dan          on patton on 01/17/2007 13:35:31.367
	 -- karen as dan --      FAILED TO       rm /etc/hosts
on patton,      Permission denied       on 01/17/2007 13:35:39.129
	 -- karen as dan --      FAILED TO       rm /etc/hosts
on patton,      Permission denied       on 01/17/2007 13:35:40.680
	 -- karen as root --     successful      SU LOGIN (PAM setcred)
as dan          on patton on 01/17/2007 13:35:44.177
	 -- karen as root --     successful      CLOSED SU as dan
on patton on 01/17/2007 13:35:44.178
	karen    successful      SU LOGIN (PAM setcred)          as root
on patton on 01/17/2007 13:35:46.089
	karen    successful      CLOSED SU as root       on patton on
01/17/2007 13:35:46.089
	
	
	Note that it's easy to spot users trying to make it look like
other users tried to do stuff.    When I review the logs, I usually sort
the reformatted lines  so that similar entries are all grouped (which
makes it really easy to let chunks of stuff I don't care about scroll by
quickly).  If I see something suspicious, I'll go back and look at the
stuff in chronological order to see what was going on.   Here's what the
above sample looks like sorted  (and if I had a lot of data, the chunks
of similar stuff would be more apparent):
	 
	 -- karen as dan --      FAILED TO       rm /etc/hosts
on patton,      Permission denied       on 01/17/2007 13:35:39.129
	 -- karen as dan --      FAILED TO       rm /etc/hosts
on patton,      Permission denied       on 01/17/2007 13:35:40.680
	 -- karen as root --     successful      CLOSED SU as dan
on patton on 01/17/2007 13:35:44.178
	 -- karen as root --     successful      SU LOGIN (PAM
accounting)       as dan          on patton on 01/17/2007 13:35:31.366
	 -- karen as root --     successful      SU LOGIN (PAM
authentication)   as dan          on patton on 01/17/2007 13:35:31.364
	 -- karen as root --     successful      SU LOGIN (PAM session
open)     as dan          on patton on 01/17/2007 13:35:31.367
	 -- karen as root --     successful      SU LOGIN (PAM setcred)
as dan          on patton on 01/17/2007 13:35:44.177
	dan      FAILED TO     console login   into patton on 01/17/2007
13:21:45.133
	dan      successful      console login   into patton on
01/17/2007 13:21:59.152
	dan      successful      console logout          from patton on
01/17/2007 13:22:26.059
	karen    FAILED TO     mv /etc/nsswitch.conf
/etc/nsswitch.conf.mine   on patton,      Permission denied       on
01/15/2007 14:05:16.582
	karen    FAILED TO     ssh login       into patton from ? on
01/17/2007 13:34:26.517
	karen    FAILED TO     touch /etc/nsswitch.conf        on
patton,      Permission denied       on 01/17/2007 13:34:57.759
	karen    FAILED TO     touch /etc/nsswitch.conf        on
patton,      Permission denied       on 01/17/2007 13:34:57.759
	karen    successful      CLOSED SU as root       on patton on
01/17/2007 13:35:46.089
	karen    successful      SU LOGIN (PAM accounting)       as root
on patton on 01/17/2007 13:35:15.141
	karen    successful      SU LOGIN (PAM authentication)   as root
on patton on 01/17/2007 13:35:15.140
	karen    successful      SU LOGIN (PAM session open)     as root
on patton on 01/17/2007 13:35:15.141
	karen    successful      SU LOGIN (PAM setcred)          as root
on patton on 01/17/2007 13:35:46.089
	karen    successful      ssh login       into patton from
patton.jhuapl.edu on 01/17/2007 13:34:31.387
	
	 
	We don't have a ton of Linux systems (yet), so this strategy
works well for reviewing a handful of systems from a central location.
The scripts save me a lot of time interpretting any one given log entry
as well as keeping me from having to log into N systems one at a time
and running M commands on each one each to decide if someone was messing
around on my machines or not. 
	 
	Karen Wieprecht

________________________________

	From: linux-audit-bounces@redhat.com
[mailto:linux-audit-bounces@redhat.com] On Behalf Of Todd, Charles
	Sent: Tuesday, September 11, 2007 3:32 PM
	To: linux-audit@redhat.com
	Subject: Offline audit trail analysis
	
	
	Thanks to Steve for being our biggest target for questions on
this list!  
	 
	Has anyone talked about sane ways to do offline analysis of
Linux audit logs?  Presumably, this would be on another Linux system,
but maybe not the same host, and probably not on the same release or
with the same username/IP address access.  Conceptually, ausearch would
save and optionally read a system's "configuration" to be saved for
interpretation later.  
	 
	My goal is central logging, but doing the reporting/analysis on
the central host.  That way, I can see a user across the Enterprise (or
at least in the Linux hosts), but with all the power of ausearch for
refining the report.  Ideally, I would do an ausearch -ts <date> -te
<date> --raw --config-to=<hostname.ausearch.config> and it would do
things like saving the syscall lookup table, lookup users referenced in
the reported audit trail, and resolve IP addresses references in the
reported audit trail.  Maybe one config file could be written for each
data type in an existing format (e.g. users in /etc/passwd format, hosts
in /etc/hosts format, etc.).  I'm mainly after whether or not anyone has
considered extending ausearch for this kind of processing?
	 
	This way, an archive of raw logs could be kept along with the
exact system configuration which allows offloading the audit trail
analysis to a trusted location, rather than risk side effects from a
rootkit.
	 
	Charlie Todd 
	Ball Aerospace & Technologies Corp.  
	
	This message and any enclosures are intended only for the
addressee.  Please  
	notify the sender by email if you are not the intended
recipient.  If you are  
	not the intended recipient, you may not use, copy, disclose, or
distribute this  
	message or its contents or enclosures to any other person and
any such actions  
	may be unlawful.  Ball reserves the right to monitor and review
all messages  
	and enclosures sent to or from this email address.




This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.

[-- Attachment #1.2: Type: text/html, Size: 21240 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]