audit collector startup help

audit collector startup help

[LC Bruzenak]

DJ (or anyone) - 

Is there a HOWTO for activating the 1.7.5 aggregating feature?
My apologies if I missed this earlier.

I believe that the collector needs to uncomment the lines
in /etc/auditd/auditd.conf and the senders/clients need to set
active=yes, remote=<IP-address> in the  audisp-remote.conf file.

However, my collector auditd fails on start; it might be that I do not
have it configured correctly.

I have : audit-1.7.5-1.fc9.i386 

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> Is there a HOWTO for activating the 1.7.5 aggregating feature?

Just the man pages.

> I believe that the collector needs to uncomment the lines
> in /etc/auditd/auditd.conf and the senders/clients need to set
> active=yes, remote=<IP-address> in the  audisp-remote.conf file.

The collector needs the listener configured in /etc/audit/auditd.conf:

tcp_listen_port = 1237

The clients need the audisp-remote module enabled and configured:

/etc/audisp/plugins.d/au-remote.conf:
active = yes

/etc/audisp/audisp-remote.conf:
remote_server = 192.16.1.12   (your server's IP, not mine ;)
port = 1237  (or use some other port, up to you)
transport = tcp

Additional options:
format = managed
network_retry_time = 1
max_tries_per_record = 10
max_time_per_record = 7

You'll have to enable the connection through tcp_wrappers as well, if
you have that option enabled, as well as whatever firewall rules may
apply.

> However, my collector auditd fails on start;

Messages?

Re: audit collector startup help

[LC Bruzenak]

On Tue, 2008-09-09 at 14:36 -0400, DJ Delorie wrote:
> > Is there a HOWTO for activating the 1.7.5 aggregating feature?
> 
> Just the man pages.
> 
> > I believe that the collector needs to uncomment the lines
> > in /etc/auditd/auditd.conf and the senders/clients need to set
> > active=yes, remote=<IP-address> in the  audisp-remote.conf file.
> 
> The collector needs the listener configured in /etc/audit/auditd.conf:
> 
> tcp_listen_port = 1237
> 
> The clients need the audisp-remote module enabled and configured:
> 
> /etc/audisp/plugins.d/au-remote.conf:
> active = yes
> 
> /etc/audisp/audisp-remote.conf:
> remote_server = 192.16.1.12   (your server's IP, not mine ;)
> port = 1237  (or use some other port, up to you)
> transport = tcp
> 
> Additional options:
> format = managed
> network_retry_time = 1
> max_tries_per_record = 10
> max_time_per_record = 7
> 
> You'll have to enable the connection through tcp_wrappers as well, if
> you have that option enabled, as well as whatever firewall rules may
> apply.
> 

Thanks for the above.
I am only looking at the server/collector startup right now. 

> > However, my collector auditd fails on start;
> 
> Messages?

Not real helpful so far (/var/log/messages - any other place?):

Sep  9 13:41:15 fryspc auditd[3786]: Init complete, auditd 1.7.5
listening for events (startup state enable)
Sep  9 13:41:15 fryspc auditd[3786]: Cannot bind tcp listener socket to
port 1237
Sep  9 13:41:15 fryspc auditd[3786]: The audit daemon is exiting.


Thx!
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> Sep  9 13:41:15 fryspc auditd[3786]: Cannot bind tcp listener socket to
> port 1237

Have you tried other ports?  There might be something on that port
already.

Re: audit collector startup help

[LC Bruzenak]

On Tue, 2008-09-09 at 15:25 -0400, DJ Delorie wrote:
> > Sep  9 13:41:15 fryspc auditd[3786]: Cannot bind tcp listener socket to
> > port 1237
> 
> Have you tried other ports?  There might be something on that port
> already.

I looked (with lsof) and there was nothing.
I tried port 12037; same thing.

It is not a firewall issue:
[root@fryspc etc]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         


I do not any tcp_wrappers issues. 
I also accept prelude events on this machine.

This has to be something easy I've overlooked...I do appreciate the
suggestions.

Thx,
LCB.

I have other things listening away:
rpcbind   1845       rpc    8u     IPv4       6916                 TCP
*:sunrpc (LISTEN)
rpc.statd 1864   rpcuser    9u     IPv4       6988                 TCP
*:58971 (LISTEN)
sshd      2152      root    3u     IPv4       7975                 TCP
*:ssh (LISTEN)
sshd      2152      root    4u     IPv6       7977                 TCP
*:ssh (LISTEN)
mysqld    2246     mysql   10u     IPv4       8088                 TCP
localhost.localdomain:mysql (LISTEN)
prelude-m 2277      root    0u     IPv4       8201                 TCP
localhost.localdomain:prelude (LISTEN)
prelude-m 2277      root    2u     IPv4       8204                 TCP
fryspc:prelude (LISTEN)
sendmail  2301      root    4u     IPv4       8325                 TCP
localhost.localdomain:smtp (LISTEN)
httpd     2320      root    4u     IPv6       8381                 TCP
*:http (LISTEN)
cupsd     2375      root    4u     IPv4       8606                 TCP
localhost.localdomain:ipp (LISTEN)
httpd     6341    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6342    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6343    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6344    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6345    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6346    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6347    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)
httpd     6348    apache    4u     IPv6       8381                 TCP
*:http (LISTEN)


-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

Hmmm.... well, one option is to put a perror() after the bind() call
in src/auditd-listen.c and run auditd in the foreground (./auditd -f)
to see *why* it's failing.

Re: audit collector startup help

[LC Bruzenak]

On Tue, 2008-09-09 at 16:11 -0400, DJ Delorie wrote:
> Hmmm.... well, one option is to put a perror() after the bind() call
> in src/auditd-listen.c and run auditd in the foreground (./auditd -f)
> to see *why* it's failing.

Good suggestion...I did that and now the one I built doesn't fail.
Then I stopped the daemon, replaced the original one, restarted the
daemon. It works also! 

Only thing I did in between was load about 100 packages needed for the
rebuild. Is there any chance that one of these had some necessary magic
I was missing?

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[LC Bruzenak]

> 
> Only thing I did in between was load about 100 packages needed for the
> rebuild. Is there any chance that one of these had some necessary magic
> I was missing?

I guess the only way to know is remove all the packages and retry...

LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> Only thing I did in between was load about 100 packages needed for the
> rebuild. Is there any chance that one of these had some necessary magic
> I was missing?

More likely, something was holding the socket in CLOSE_WAIT or
something and happened to time out while you were updating everything.

Re: audit collector startup help

[LC Bruzenak]

On Tue, 2008-09-09 at 18:07 -0400, DJ Delorie wrote:
> > Only thing I did in between was load about 100 packages needed for the
> > rebuild. Is there any chance that one of these had some necessary magic
> > I was missing?
> 
> More likely, something was holding the socket in CLOSE_WAIT or
> something and happened to time out while you were updating everything.

Actually I believe one of the packages must installed my policy as
enforcing.

Thanks(!) to an excellent setroubleshoot pop-up I believe that was my
problem:

Source Context:  unconfined_u:system_r:auditd_t:s0
Target Context:  system_u:object_r:anon_inodefs_t:s0
Target Objects:  anon_inode [ file ]
Source:  auditdSource 
Path:  /sbin/auditd
Port:  <Unknown>
Host:  fryspc
Source RPM Packages:  audit-1.7.5-1.fc9
Target RPM Packages:  
Policy RPM:  selinux-policy-3.3.1-87.fc9
Selinux Enabled:  True
Policy Type:  targeted
MLS Enabled:  True
Enforcing Mode:  Enforcing
Plugin Name:  catchall_file
Host Name:  fryspc
Platform:  Linux fryspc 2.6.26.3-29.fc9.i686 #1 SMP Wed Sep 3 03:42:27 EDT 2008 i686 athlon
Alert Count:  1
First Seen:  Thu 11 Sep 2008 10:08:57 AM CDT
Last Seen:  Thu 11 Sep 2008 10:08:57 AM CDT
Local ID:  8b4ff486-ae1c-4448-bf38-9b56658ebc01
Line Numbers:  
Raw Audit Messages :
host=fryspc type=AVC msg=audit(1221145737.208:55): avc: denied { write } for pid=3280 comm="auditd" path="anon_inode:[eventfd]" dev=anon_inodefs ino=18 scontext=unconfined_u:system_r:auditd_t:s0 tcontext=system_u:object_r:anon_inodefs_t:s0 tclass=file 
host=fryspc type=SYSCALL msg=audit(1221145737.208:55): arch=40000003 syscall=4 success=no exit=-13 a0=8 a1=bfb98880 a2=8 a3=b7f6aab8 items=0 ppid=1 pid=3280 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="auditd" exe="/sbin/auditd" subj=unconfined_u:system_r:auditd_t:s0 key=(null) 

### note: I do not have an MLS policy on this machine (although the
setroubleshoot summary says I do) - and I didn't change any policy
defaults.

[lenny@fryspc ~]$ rpm -qa | grep policy
checkpolicy-2.0.16-3.fc9.i386
policycoreutils-2.0.52-8.fc9.i386
selinux-policy-targeted-3.3.1-87.fc9.noarch
selinux-policy-devel-3.3.1-87.fc9.noarch
policycoreutils-gui-2.0.52-8.fc9.i386
selinux-policy-3.3.1-87.fc9.noarch


Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

audit collector connect fails

[LC Bruzenak]

My sender fails to connect to my collector.

Is there any reason a MLS-policy F9 audisp-remote should be unable to
connect to a targeted-policy F9 auditd? I have no ipsec or anything else
involved...

I am looking for some hint as to why the connection is failing but I see
only this on the sender:

- lsof says I'm stuck on SYN_SENT
TCP comms:38827->192.168.30.120:tsdos390 (SYN_SENT)

- audit search on sender
ausearch -ts today -i -c audisp-remote:
...
----
type=SYSCALL msg=audit(09/11/2008 16:14:45.102:19013) : arch=x86_64
syscall=connect success=no exit=-110(Connection timed out) a0=3
a1=7f99ab0f20e0 a2=10 a3=7fffb289cf50 items=0 ppid=25435 pid=25436
auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root
sgid=root fsgid=root tty=(none) ses=61 comm=audisp-remote
exe=/sbin/audisp-remote
subj=system_u:system_r:audisp_remote_t:s15:c0.c1023 key=(null) 


Same audit versions on each (1.7.5-1).

On the sender, I can do a "newrole -l SystemHigh" and connect via
"telnet <collector> 1237", so I don't think it is the level giving me
any grief - sender is in permissive mode so there are AVCs but it should
work.

Eventually on the sender I get this:
Sep 11 16:57:12 comms audisp-remote: Error connecting to 192.168.30.120: Connection timed out - exiting
Sep 11 16:57:14 comms audispd: plugin /sbin/audisp-remote terminated unexpectedly

On the collector machine I see the listen socket open but I see no
denials in the messages log or the audit log.

Any suggestions?

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector connect fails

[DJ Delorie]

Have you tried killing the client auditd and manually restarting it
(i.e. /etc/rc.d/init.d/auditd restart) ?

I wonder if there's some catch-22 at system startup, relative to
network startup.

Re: audit collector connect fails

[LC Bruzenak]

On Thu, 2008-09-11 at 18:43 -0400, DJ Delorie wrote:
> Have you tried killing the client auditd and manually restarting it
> (i.e. /etc/rc.d/init.d/auditd restart) ?
> 
> I wonder if there's some catch-22 at system startup, relative to
> network startup.

I have, but actually I use :
run_init service auditd restart

otherwise my audit logs get created at SYSLO which is bad on an MLS
system.

I have had to restart often because while I am looking for a hint the
timeout happens and the audisp-remote process dies.

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[LC Bruzenak]

On Tue, 2008-09-09 at 14:36 -0400, DJ Delorie wrote:
> > Is there a HOWTO for activating the 1.7.5 aggregating feature?
> 
> Just the man pages.
> 
> > I believe that the collector needs to uncomment the lines
> > in /etc/auditd/auditd.conf and the senders/clients need to set
> > active=yes, remote=<IP-address> in the  audisp-remote.conf file.
> 
> The collector needs the listener configured in /etc/audit/auditd.conf:
> 
> tcp_listen_port = 1237
> 
> The clients need the audisp-remote module enabled and configured:
> 
> /etc/audisp/plugins.d/au-remote.conf:
> active = yes
> 
> /etc/audisp/audisp-remote.conf:
> remote_server = 192.16.1.12   (your server's IP, not mine ;)
> port = 1237  (or use some other port, up to you)
> transport = tcp
> 
> Additional options:
> format = managed
> network_retry_time = 1
> max_tries_per_record = 10
> max_time_per_record = 7

DJ,

Thanks for the above. The network_retry_time (et. al.) must be in the
later version.

I have: audispd-plugins-1.7.5-1.fc9.x86_64 ; there is no mention of that
one in the man page and I get this message on startup:

Sep 12 11:43:48 comms audisp-remote: Unknown keyword "network_retry_time" in line 14 of /etc/audisp/audisp-remote.conf
Sep 12 11:43:48 comms auditd[4411]: Init complete, auditd 1.7.5 listening for events (startup state enable)
Sep 12 11:43:48 comms audispd: plugin /sbin/audisp-remote terminated unexpectedly

So I Removed the timing parameters.

Now I get this:
...
Sep 12 11:46:20 comms audisp-remote: lost/losing sync, bad magic number
Sep 12 11:46:20 comms audisp-remote: lost/losing sync, bad magic number
...


I do not see any errors in the message log on the collector.

Any ideas?

Thx again!
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

You need 1.7.6 then, or SVN.  Coming soon to a theater near you ;-)

Re: audit collector startup help

[LC Bruzenak]

On Fri, 2008-09-12 at 13:14 -0400, DJ Delorie wrote:
> You need 1.7.6 then, or SVN.  Coming soon to a theater near you ;-)

So do you think that the "lost/losing sync, bad magic number" issue is
fixed in 1.7.6 or just the timing parameters?

LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> So do you think that the "lost/losing sync, bad magic number" issue is
> fixed in 1.7.6 or just the timing parameters?

What 1.7.6 adds is the ability to close the connection and start
again, without losing any audit records, if/when it detects any
communications problems.

Re: audit collector startup help

[LC Bruzenak]

On Fri, 2008-09-12 at 14:45 -0400, DJ Delorie wrote:
> > So do you think that the "lost/losing sync, bad magic number" issue is
> > fixed in 1.7.6 or just the timing parameters?
> 
> What 1.7.6 adds is the ability to close the connection and start
> again, without losing any audit records, if/when it detects any
> communications problems.

So any clue as to why I get a "bad magic number" on this version?

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> So any clue as to why I get a "bad magic number" on this version?

Nope.  It means the server sent back a header that was corrupt, or
sent back something other than a header.

The header is defined in lib/private.h

Re: audit collector startup help

[LC Bruzenak]

On Fri, 2008-09-12 at 16:33 -0400, DJ Delorie wrote:
> > So any clue as to why I get a "bad magic number" on this version?
> 
> Nope.  It means the server sent back a header that was corrupt, or
> sent back something other than a header.
> 
> The header is defined in lib/private.h

After looking at this I had a hunch - the collector machine is 32-bit,
the sender 64-bit. 

I reverse the sender/collector and I don't get this error anymore.
The machine architecture is not representative of the intended final
deployment, just the available machines I was using to test.

So now the bad news is that I still do not see events passing from
sender to collector. This may be to an incorrect assumption on my part.
I assume that all events on the sender make it to the collector. Is this
true always? I send in a forced event (on the sender):

[root@fryspc audisp]#  auditctl -m TEST
[root@fryspc audisp]# ausearch -ts recent -i | grep TEST
type=USER msg=audit(09/12/2008 18:34:34.930:126) : user pid=4866 uid=root auid=root ses=11 subj=unconfined_u:unconfined_r:auditctl_t:s0-s0:c0.c1023 msg='TEST: exe=/sbin/auditctl (hostname=?, addr=?, terminal=pts/2 res=success)' 

But I cannot see this event on the collector.

The good news is that I see the connection on both ends (using port 1237=tsdos, lsof results):
collector:
auditd     9422      root   11u     IPv4              55889                 TCP comms:tsdos390->fryspc:55303 (ESTABLISHED)
collector:
sender:
audisp-re 4846      root    3u     IPv4      26741                  TCP fryspc:55303->192.168.31.142:tsdos390 (ESTABLISHED)


Should I see this event, and if so, do you have any idea as to why I do
not? I also have the same thing from another 64-bit sender.

Thx,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: audit collector startup help

[DJ Delorie]

> After looking at this I had a hunch - the collector machine is 32-bit,
> the sender 64-bit. 

And the magic number has the high bit set.  I wonder if there's a sign
extension in there somewhere?

Can you try between two 32 bit hosts?

> I assume that all events on the sender make it to the collector. Is this
> true always?

I didn't add any filters - anything that makes it to audisp-remote
eventually gets queued in the server's event queue.

> But I cannot see this event on the collector.

All remote messages will have "node=" in them somewhere.  Can you grep
for that manually in your server's audit logs?  I wonder if ausearch
is skipping them.