Auditing network traffic

Auditing network traffic

[Lev Stipakov]

Hello,

I work on an audisp plugin which audits network traffic – what process 
has send/received data to/from what remote address. So far I see 2 ways 
of accomplishing that:

Hook syscalls. First, hook socket call with af_inet/inet6 to get pid and 
fd, then read/write/sendto/recvfrom filtered by pid and fd. I see few 
issues with this appoach:

1) Fd can be closed or duped, so I should probably hook close/dup2 calls 
too. Not sure, though, if socket could be closed by kernel without any 
syscall. As a workaroud, one can just hook read/write and check if fd is 
socket (S_ISSOCK) and also somehow filter out af_unix.

2) Getting saddr/daddr. Seems that dest addr could be obtained from 
connect call. However I am not sure what is the right way to get that - 
I got two records, first SYSCALL and then SOCKADDR. First one has an 
argument which points to memory location where sockaddr structure lays, 
and second one has ”saddr” field.  Latter looks good, but does SOCKADDR 
event type always follows SYSCALL for connect call? Same for sendto call.

Another way of getting network stats is the AUDIT target for netfilter. 
Looks good, no need to worry about fds/addrs. However there is no pid. 
What would be the ”best” way to get pid for those records? Anything else 
besides looking into /proc/net/tcp?

-Lev

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Auditing network traffic

[Steve Grubb]

On Wednesday, January 20, 2016 04:26:34 PM Lev Stipakov wrote:
> Hello,
> 
> I work on an audisp plugin which audits network traffic – what process
> has send/received data to/from what remote address. So far I see 2 ways
> of accomplishing that:
> 
> Hook syscalls. First, hook socket call with af_inet/inet6 to get pid and
> fd, then read/write/sendto/recvfrom filtered by pid and fd. I see few
> issues with this appoach:
> 
> 1) Fd can be closed or duped, so I should probably hook close/dup2 calls
> too.

It can also be passed to another process by sendmsg. The process can 
fork/clone changing the pid and then the child access the descriptor. It can 
also call sendfile and there are more writing/reading syscalls.


> Not sure, though, if socket could be closed by kernel without any
> syscall.

exit_group

> As a workaroud, one can just hook read/write and check if fd is
> socket (S_ISSOCK) and also somehow filter out af_unix.

what if its mmap'ed?


> 2) Getting saddr/daddr. Seems that dest addr could be obtained from
> connect call. However I am not sure what is the right way to get that -
> I got two records, first SYSCALL and then SOCKADDR. First one has an
> argument which points to memory location where sockaddr structure lays,
> and second one has ”saddr” field.  Latter looks good, but does SOCKADDR
> event type always follows SYSCALL for connect call? Same for sendto call.

Yes.
 
> Another way of getting network stats is the AUDIT target for netfilter.
> Looks good, no need to worry about fds/addrs. However there is no pid.

I am thinking that would be a good addition. However, there are times when 
there really is no pid. For example, it could be masquerading or doing SNAT.


> What would be the ”best” way to get pid for those records? Anything else
> besides looking into /proc/net/tcp?

That might be the best workaround right now. But adding the pid sounds 
reasonable to me. This way it can be correlated to other system activity. I'd 
have to ask Paul or Richard to comment on feasibility.

-Steve

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Auditing network traffic

[Steve Grubb]

On Wednesday, January 20, 2016 10:18:29 AM Steve Grubb wrote:
> > I work on an audisp plugin which audits network traffic – what process
> > has send/received data to/from what remote address. So far I see 2 ways
> > of accomplishing that:
> > 
> > Hook syscalls. First, hook socket call with af_inet/inet6 to get pid and
> > fd, then read/write/sendto/recvfrom filtered by pid and fd

One other thing, read and write will tell you that a read or write happened. 
It does not record what was read or written. If you need that, you will have 
to sniff network traffic. Audit won't be able to help much.

-Steve

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Auditing network traffic

[F Rafi]

[-- Attachment #1.1: Type: text/plain, Size: 1413 bytes --]

Perhaps this is of use. My goal was to restrict audit logs to outbound
connections only to reduce the amount of logs.

# Outbound connections could indicate exfiltration of data (connect vs
accept)
# Log 64 bit processes (a2!=6e filters local unix socket calls)

-a exit,always -F arch=b64 -S connect -F a2!=110 -k network_outbound64

# Log 32 bit processes (a0=3 means only outbound sys_connect calls)

-a exit,always -F arch=b32 -S socketcall -F a0=3 -k network_outbound32


-Farhan

PS: I'd appreciate if someone could poke holes in this.

On Wed, Jan 20, 2016 at 10:29 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, January 20, 2016 10:18:29 AM Steve Grubb wrote:
> > > I work on an audisp plugin which audits network traffic – what process
> > > has send/received data to/from what remote address. So far I see 2 ways
> > > of accomplishing that:
> > >
> > > Hook syscalls. First, hook socket call with af_inet/inet6 to get pid
> and
> > > fd, then read/write/sendto/recvfrom filtered by pid and fd
>
> One other thing, read and write will tell you that a read or write
> happened.
> It does not record what was read or written. If you need that, you will
> have
> to sniff network traffic. Audit won't be able to help much.
>
> -Steve
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>

[-- Attachment #1.2: Type: text/html, Size: 2073 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditing network traffic

[Steve Grubb]

On Wednesday, January 20, 2016 01:05:45 PM F Rafi wrote:
> Perhaps this is of use. My goal was to restrict audit logs to outbound
> connections only to reduce the amount of logs.
> 
> # Outbound connections could indicate exfiltration of data (connect vs
> accept)
> # Log 64 bit processes (a2!=6e filters local unix socket calls)
> 
> -a exit,always -F arch=b64 -S connect -F a2!=110 -k network_outbound64

This is good for TCP connections. There's always UDP where you would need 
sendto and sendmsg. Imagine someone exfiltrating on what seems to be DNS lookup 
requests.

The IPTables AUDIT target is what is really meant to audit information flow in 
or out of the system. I think this is the first discussion on the mail list 
where someone might be trying to use it. I'm hoping this leads to making it 
better.

-Steve


> # Log 32 bit processes (a0=3 means only outbound sys_connect calls)
> 
> -a exit,always -F arch=b32 -S socketcall -F a0=3 -k network_outbound32
> 
> 
> -Farhan
> 
> PS: I'd appreciate if someone could poke holes in this.
> 
> On Wed, Jan 20, 2016 at 10:29 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > On Wednesday, January 20, 2016 10:18:29 AM Steve Grubb wrote:
> > > > I work on an audisp plugin which audits network traffic – what process
> > > > has send/received data to/from what remote address. So far I see 2
> > > > ways
> > > > of accomplishing that:
> > > > 
> > > > Hook syscalls. First, hook socket call with af_inet/inet6 to get pid
> > 
> > and
> > 
> > > > fd, then read/write/sendto/recvfrom filtered by pid and fd
> > 
> > One other thing, read and write will tell you that a read or write
> > happened.
> > It does not record what was read or written. If you need that, you will
> > have
> > to sniff network traffic. Audit won't be able to help much.
> > 
> > -Steve
> > 
> > --
> > Linux-audit mailing list
> > Linux-audit@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit


--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Auditing network traffic

[Lev Stipakov]

Hi Steve,

Thank you for your comments! It seems that AUDIT target is better option 
than hooking syscalls and managing fds. I don't have to look inside 
traffic, just src/dest and bytes count is enough for me.

What would be the performance implications of that approach comparison 
to, say, libpcap option? Mostly I am concerned about logging part - 
seems that every packet produces NETFILTER_PKT record. I could not find 
any way to disable that, except probably disabling logging all together 
but that will break ausearch.

-Lev

Re: Auditing network traffic

[Steve Grubb]

On Thursday, January 21, 2016 11:49:13 AM Lev Stipakov wrote:
> Thank you for your comments! It seems that AUDIT target is better option
> than hooking syscalls and managing fds. I don't have to look inside
> traffic, just src/dest and bytes count is enough for me.
> 
> What would be the performance implications of that approach comparison
> to, say, libpcap option?

I'd say it would be better because you don't have to do nearly as much work.
The kernel takes care of all the heavy lifting and you just filter on
NETFILTER_PKT events.

> Mostly I am concerned about logging part - seems that every packet produces
> NETFILTER_PKT record. I could not find any way to disable that, except
> probably disabling logging all together but that will break ausearch.

There are plenty of examples of how to do logging of netfilter events. You can
just copy the examples and substitute AUDIT as the target (but you have to add
a --type argument after it). A couple examples I found after a quick search:

iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j AUDIT --type accept

To get any connection attempt:
iptables -I INPUT -p tcp --tcp-flags ALL SYN -j AUDIT --type accept

Of course any use of nmap -sS will flood the logs on this one. But you can
write any kind of netfilter rule. The examples in iptables-extensions man page
are quite limited when compared to what you can really do.

Maybe I should update an audit man page to show some real world examples. If
anyone would like to suggest a few examples, I'll see about adding them.

-Steve

Re: Auditing network traffic

[Lev Stipakov]

On 21.01.2016 18:50, Steve Grubb wrote:

> I'd say it would be better because you don't have to do nearly as much work.
> The kernel takes care of all the heavy lifting and you just filter on
> NETFILTER_PKT events.

Good to know, thanks!

> There are plenty of examples of how to do logging of netfilter events. You can
> just copy the examples and substitute AUDIT as the target (but you have to add
> a --type argument after it). A couple examples I found after a quick search:

Sorry, I probably was not clear here. I am able to catch packets by 
adding iptables rules like ones you've mentioned and process events 
(with record type AUDIT_NETFILTER_PKT) by code inside my plugin.

The problem is, I would prefer them not to be written to logfiles. My 
business logic does not require that (everything is handled by plugin 
code), and I noticed that logs are rotated quite fast (I capture all 
incoming/outgoing packets). So, is there any way to disable logging and 
make audit deliver those events to plugin only?

-Lev

Re: Auditing network traffic

[Steve Grubb]

On Thursday, January 21, 2016 10:49:37 PM Lev Stipakov wrote:
> Sorry, I probably was not clear here. I am able to catch packets by
> adding iptables rules like ones you've mentioned and process events
> (with record type AUDIT_NETFILTER_PKT) by code inside my plugin.
> 
> The problem is, I would prefer them not to be written to logfiles. My
> business logic does not require that (everything is handled by plugin
> code), and I noticed that logs are rotated quite fast (I capture all
> incoming/outgoing packets). So, is there any way to disable logging and
> make audit deliver those events to plugin only?

In /etc/audit/auditd.conf make log_firmat like this:

log_format = NOLOG

and auditd will not log anything to disk.

-Steve

Re: Auditing network traffic

[Paul Moore]

On Wed, Jan 20, 2016 at 9:26 AM, Lev Stipakov <lstipakov@gmail.com> wrote:
> Another way of getting network stats is the AUDIT target for netfilter.
> Looks good, no need to worry about fds/addrs. However there is no pid. What
> would be the ”best” way to get pid for those records? Anything else besides
> looking into /proc/net/tcp?

Linking a specific process/PID to a network packet is very difficult,
if not impossible, for the simple reason that the kernel doesn't track
the originating process, only the originating socket (which is an
unreliable way to determine the sending process).  Not to mention the
fact Steve already mentioned that some packets do not originate in
userspace; forwarded traffic, streaming protocol control messages,
ICMP error messages are all common examples of non-local userspace
generated messages.

-- 
paul moore
www.paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Auditing network traffic

[Peter Moody]

I actually did a bunch of work for this at a previous job. it was
supposed to be opensource'd, but then I switched jobs and the new
internal-maintainer never got around to opening it up :(

My short answer is that audit is probably the wrong tool for this,
especially for machines pushing a large amount of traffic.

the longer answer is that you can do it if you hook the discrete
events in the kernel and then correlate them later in your pipeline.
IIRC, the events you need are:

 * fork/exec/exit to correlate pid with proc information (exe, argv)
 * socket open/close & sock_sendmsg/sock_recvmsg to correlate socket
to pid (i think you might be able to get way without hooking socket
open/close and just hook sendmsg/recvmsg)
 * netfilters to correlate saddr/daddr + packet contents with sockets.

the linux sensor module (https://github.com/HoneProject/Linux-Sensor)
did most of this but there were a bunch of issues that made it
unsuitable for enterprise use (funding dried up so the folks at pnnl
who had done all of the heavy lifting weren't able to finish). the
internal fork fixed these issues but like I said, I left before I got
a chance to upstream the fixes.

you could also look at the sysdig module for this.

Cheers,
peter

On Wed, Jan 20, 2016 at 6:26 AM, Lev Stipakov <lstipakov@gmail.com> wrote:
> Hello,
>
> I work on an audisp plugin which audits network traffic – what process has
> send/received data to/from what remote address. So far I see 2 ways of
> accomplishing that:

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit