Decoding arguments passed to system calls

Decoding arguments passed to system calls

[Darryl Dixon - Winterhouse Consulting]

Hi List,

Forgive me if this isn't the correct forum for this, but I'd like to
present a scenario, outline my hypothetical solution to it, and then
solicit for feedback from the list on how to actually achieve it. I have
seen various discussions on the list around this topic (eg
http://www.redhat.com/archives/linux-audit/2005-March/msg00221.html, and
the current "Absolute path names in PATH records" thread), but
they all seemed intertwined with other things, so I am asking here to try
and pin down a firm answer :)

Scenario:
A very large filesystem with potentially millions of files in an ad-hoc,
unordered directory structure. The requirement is to be able to audit any
action on any file in this filesystem (moves, adds, changes, deletes,
etc). In auditfs terms, there is a requirement to have a 'watch' on every
single file (millions of files), and on any new file that is added.

Hypothetical solution:
Clearly, scanning the filesystem with `find` and adding calling auditctl
with the appropriate arguments to generate a watch on every singly file is
totally infeasible (find takes almost an hour to run, and in the meantime
stuff is potentially changing...). Instead, I envision it would make
better sense to simply audit every call to write(), open(), rename(), etc,
and then filter backwards from there with ausearch and a filter based on
the path. On Solaris with BSM, this is possible. My problem is that this
doesn't seem possible with the Linux Audit subsystem, as the arguments to
the system calls are not decoded (eg, the audit records for write()
include only an opaque filehandle and pointer to the written data, etc).

So, in terms of feedback:

1) Am I totally wrong and there's a method of getting this information
already that I have overlooked?

2) Knowing very little about the auditing subsystem, and the kernel
internals in general I envision that decoding the filehandle into a path
is something that would need to be done in the kernel, and is impossible
from userland. Is this the case?

3) How much work do you all estimate that it would actually take to be
able to generate this information? Is it even possible without a major
architectural overhaul of the audit subsystem?

Any and all feedback much appreciated.

many regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

Re: Decoding arguments passed to system calls

[Matthew Booth]

[-- Attachment #1.1: Type: text/plain, Size: 3443 bytes --]

On Tue, 2007-07-03 at 09:46 +1200, Darryl Dixon - Winterhouse Consulting
wrote:
> Scenario:
> A very large filesystem with potentially millions of files in an ad-hoc,
> unordered directory structure. The requirement is to be able to audit any
> action on any file in this filesystem (moves, adds, changes, deletes,
> etc). In auditfs terms, there is a requirement to have a 'watch' on every
> single file (millions of files), and on any new file that is added.
> 
> Hypothetical solution:
> Clearly, scanning the filesystem with `find` and adding calling auditctl
> with the appropriate arguments to generate a watch on every singly file is
> totally infeasible (find takes almost an hour to run, and in the meantime
> stuff is potentially changing...). Instead, I envision it would make
> better sense to simply audit every call to write(), open(), rename(), etc,
> and then filter backwards from there with ausearch and a filter based on
> the path. On Solaris with BSM, this is possible. My problem is that this
> doesn't seem possible with the Linux Audit subsystem, as the arguments to
> the system calls are not decoded (eg, the audit records for write()
> include only an opaque filehandle and pointer to the written data, etc).

This is entirely feasible, with the exception that you probably don't
want to audit write() calls generally. That would be a truly insane
amount of traffic. A highly effective heuristic is to look at the
arguments to the open() system call. If the file was opened for write
(a1 | 1 == 1 || a1 | 2 == 2), then assume that the file was altered.

Take the following example:

type=SYSCALL msg=audit(1183414428.656:2779): arch=40000003 syscall=5
success=yes exit=3 a0=91d4088 a1=8000 a2=0 a3=8000 items
=1 ppid=16166 pid=16270 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts2 comm="vi" exe="/bin/vi" sub
j=user_u:system_r:unconfined_t:s0 key=(null)

type=CWD msg=audit(1183414428.656:2779):  cwd="/root"

type=PATH msg=audit(1183414428.656:2779): item=0 name="install.log"
inode=720898 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00
:00 obj=root:object_r:user_home_t:s0

You can tell the filesystem from the dev field of the PATH record:
[mbooth@mbooth ~]$ ls -l /dev/mapper/vg_local-root 
brw-rw---- 1 root disk 253, 0 Jul  2 19:53 /dev/mapper/vg_local-root

253, 0 == fd:00

You can tell it was opened for read, because the last 2 bits of a1 in
the SYSCALL record are 00. I guess doing this automatically is the crux
of your question. For ease of path handling, you're also going to want a
solution to the absolute path question.

As an aside, adding the rules you mention above with a standard audit
configuration is likely to kill your system as soon as you put load on
it. The principal reason for this is the frequent flushing of the log
file. If integrity of the audit trail isn't that important, tone down
the flushing on the audit log and you'll find that it can sustain pretty
intense throughput. If you need both integrity and performance, the best
solution I've come up with is to send audit records to a remote host via
syslog in real time over a network which we trust not to drop packets,
and not write anything to disk.

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Decoding arguments passed to system calls

[Darryl Dixon - Winterhouse Consulting]

Hi Matt,

On Tue, July 3, 2007 10:27 am, Matthew Booth wrote:
> On Tue, 2007-07-03 at 09:46 +1200, Darryl Dixon - Winterhouse Consulting
> wrote:
>> Scenario:
>> A very large filesystem with potentially millions of files in an ad-hoc,
>> unordered directory structure. The requirement is to be able to audit
>> any
>> action on any file in this filesystem (moves, adds, changes, deletes,
>> etc). In auditfs terms, there is a requirement to have a 'watch' on
>> every
>> single file (millions of files), and on any new file that is added.
>>
>> Hypothetical solution:
>> Clearly, scanning the filesystem with `find` and adding calling auditctl
>> with the appropriate arguments to generate a watch on every singly file
>> is
>> totally infeasible (find takes almost an hour to run, and in the
>> meantime
>> stuff is potentially changing...). Instead, I envision it would make
>> better sense to simply audit every call to write(), open(), rename(),
>> etc,
>> and then filter backwards from there with ausearch and a filter based on
>> the path. On Solaris with BSM, this is possible. My problem is that this
>> doesn't seem possible with the Linux Audit subsystem, as the arguments
>> to
>> the system calls are not decoded (eg, the audit records for write()
>> include only an opaque filehandle and pointer to the written data, etc).
>
> This is entirely feasible, with the exception that you probably don't
> want to audit write() calls generally. That would be a truly insane
> amount of traffic. A highly effective heuristic is to look at the
> arguments to the open() system call. If the file was opened for write
> (a1 | 1 == 1 || a1 | 2 == 2), then assume that the file was altered.
>
> Take the following example:
>
> type=SYSCALL msg=audit(1183414428.656:2779): arch=40000003 syscall=5
> success=yes exit=3 a0=91d4088 a1=8000 a2=0 a3=8000 items
> =1 ppid=16166 pid=16270 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0
> egid=0 sgid=0 fsgid=0 tty=pts2 comm="vi" exe="/bin/vi" sub
> j=user_u:system_r:unconfined_t:s0 key=(null)
>
> type=CWD msg=audit(1183414428.656:2779):  cwd="/root"
>
> type=PATH msg=audit(1183414428.656:2779): item=0 name="install.log"
> inode=720898 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00
> :00 obj=root:object_r:user_home_t:s0
>
> You can tell the filesystem from the dev field of the PATH record:
> [mbooth@mbooth ~]$ ls -l /dev/mapper/vg_local-root
> brw-rw---- 1 root disk 253, 0 Jul  2 19:53 /dev/mapper/vg_local-root
>
> 253, 0 == fd:00
>
> You can tell it was opened for read, because the last 2 bits of a1 in
> the SYSCALL record are 00. I guess doing this automatically is the crux
> of your question. For ease of path handling, you're also going to want a
> solution to the absolute path question.
>
> As an aside, adding the rules you mention above with a standard audit
> configuration is likely to kill your system as soon as you put load on
> it. The principal reason for this is the frequent flushing of the log
> file. If integrity of the audit trail isn't that important, tone down
> the flushing on the audit log and you'll find that it can sustain pretty
> intense throughput. If you need both integrity and performance, the best
> solution I've come up with is to send audit records to a remote host via
> syslog in real time over a network which we trust not to drop packets,
> and not write anything to disk.
>
> Matt

Hi Matt,

Thank you for your very thorough response. What you say about not being
able to audit 'write()' is worrying to me. The problem with auditing write
by inference from open(), is that one doesn't know *when* the file was
written, or even if it really ever was at all (eg, was data written
continuously from open() to close(), or only sporadically over the course
of hours or days?). Auditing for actual alterations is definitely
something that we need to be able to track. Assuming for a moment that we
have beefy enough hardware ( heh ), can the path be extracted from write()
as with your example for open() above? My assumption would have been that
CWD reflected only where the exe was launched from, and not necessarily
where the write()-en file was located...

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

Re: Decoding arguments passed to system calls

[Matthew Booth]

[-- Attachment #1.1: Type: text/plain, Size: 1877 bytes --]

On Tue, 2007-07-03 at 10:48 +1200, Darryl Dixon - Winterhouse Consulting
wrote:
> Hi Matt,
> 
> Thank you for your very thorough response. What you say about not being
> able to audit 'write()' is worrying to me. The problem with auditing write
> by inference from open(), is that one doesn't know *when* the file was
> written, or even if it really ever was at all (eg, was data written
> continuously from open() to close(), or only sporadically over the course
> of hours or days?). Auditing for actual alterations is definitely
> something that we need to be able to track. Assuming for a moment that we
> have beefy enough hardware ( heh ), can the path be extracted from write()
> as with your example for open() above? My assumption would have been that
> CWD reflected only where the exe was launched from, and not necessarily
> where the write()-en file was located...

Well, you can audit write(). The question is whether you can handle the
resulting data volume.

read() and write() don't generate a PATH record. The only time at which
a path is relevant is at the time the file is opened. Once the file is
opened, it can have zero or more paths, which can all change without
affecting the open file. An open file is genuinely divorced from its
path.

That said, it's *not* divorced from its filesystem, which I understand
is what you're really looking for. Unfortunately that information
doesn't appear to live on its own in the audit trail, and it's not
available to filter on for these calls. This would leave you having to
audit all read() and write() calls and filtering them in
post-processing. I doubt this would be a practical solution.

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Decoding arguments passed to system calls

[Stephen Smalley]

On Tue, 2007-07-03 at 00:23 +0100, Matthew Booth wrote:
> On Tue, 2007-07-03 at 10:48 +1200, Darryl Dixon - Winterhouse Consulting
> wrote:
> > Hi Matt,
> > 
> > Thank you for your very thorough response. What you say about not being
> > able to audit 'write()' is worrying to me. The problem with auditing write
> > by inference from open(), is that one doesn't know *when* the file was
> > written, or even if it really ever was at all (eg, was data written
> > continuously from open() to close(), or only sporadically over the course
> > of hours or days?). Auditing for actual alterations is definitely
> > something that we need to be able to track. Assuming for a moment that we
> > have beefy enough hardware ( heh ), can the path be extracted from write()
> > as with your example for open() above? My assumption would have been that
> > CWD reflected only where the exe was launched from, and not necessarily
> > where the write()-en file was located...
> 
> Well, you can audit write(). The question is whether you can handle the
> resulting data volume.
> 
> read() and write() don't generate a PATH record. The only time at which
> a path is relevant is at the time the file is opened. Once the file is
> opened, it can have zero or more paths, which can all change without
> affecting the open file. An open file is genuinely divorced from its
> path.
> 
> That said, it's *not* divorced from its filesystem, which I understand
> is what you're really looking for. Unfortunately that information
> doesn't appear to live on its own in the audit trail, and it's not
> available to filter on for these calls. This would leave you having to
> audit all read() and write() calls and filtering them in
> post-processing. I doubt this would be a practical solution.

Another option might be to audit based on SELinux type, if the files in
question can have a different type than the rest of the files.  A
SELinux audit message on write would generate an AVC_PATH record.

Example:
$ vi myaudit.te
policy_module(myaudit, 1.0)
require {
	attribute domain;
	type etc_t;
}
auditallow domain etc_t:file write;
:q

$ make -f /usr/share/selinux/devel/Makefile myaudit.pp
$ su
# /usr/sbin/semodule -i myaudit.pp
# vi /etc/fstab
:wq

Audit log contains:
type=AVC msg=audit(1183463480.620:6351): avc:  granted  { write } for  pid=13773 comm="vi" name="fstab" dev=dm-0 ino=1280187 scontext=user_u:system_r:unconfined_t:s0 tcontext=user_u:object_r:etc_t:s0 tclass=file
type=SYSCALL msg=audit(1183463480.620:6351): arch=40000003 syscall=4 success=yes exit=767 a0=3 a1=9047510 a2=2ff a3=2ff items=0 ppid=13662 pid=13773 auid=4204 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 comm="vi" exe="/bin/vi" subj=user_u:system_r:unconfined_t:s0 key=(null)
type=AVC_PATH msg=audit(1183463480.620:6351):  path="/etc/fstab"


-- 
Stephen Smalley
National Security Agency

RE: Decoding arguments passed to system calls

[Wieprecht, Karen M.]

Not sure how Linux handles this, but on Irix, when I accidentally tried
to audit one of the "write"  audit record types,  it would crash the
machine.  If I still understand this correctly (it's been a few years),
the record I had selected for audit generated/collected an audit record
every time ANYTHING got written to, including terminal devices, not just
when you issued a "save" on a file,  so every character that I typed
created an audit record.  It was very ugly,  and definitely not what I
wanted (and definitely not anything anyone was requiring me to collect).


Food For thought,

Karen Wieprecht 

Re: Decoding arguments passed to system calls

[Stephen Smalley]

On Tue, 2007-07-03 at 07:57 -0400, Stephen Smalley wrote:
> On Tue, 2007-07-03 at 00:23 +0100, Matthew Booth wrote:
> > On Tue, 2007-07-03 at 10:48 +1200, Darryl Dixon - Winterhouse Consulting
> > wrote:
> > > Hi Matt,
> > > 
> > > Thank you for your very thorough response. What you say about not being
> > > able to audit 'write()' is worrying to me. The problem with auditing write
> > > by inference from open(), is that one doesn't know *when* the file was
> > > written, or even if it really ever was at all (eg, was data written
> > > continuously from open() to close(), or only sporadically over the course
> > > of hours or days?). Auditing for actual alterations is definitely
> > > something that we need to be able to track. Assuming for a moment that we
> > > have beefy enough hardware ( heh ), can the path be extracted from write()
> > > as with your example for open() above? My assumption would have been that
> > > CWD reflected only where the exe was launched from, and not necessarily
> > > where the write()-en file was located...
> > 
> > Well, you can audit write(). The question is whether you can handle the
> > resulting data volume.
> > 
> > read() and write() don't generate a PATH record. The only time at which
> > a path is relevant is at the time the file is opened. Once the file is
> > opened, it can have zero or more paths, which can all change without
> > affecting the open file. An open file is genuinely divorced from its
> > path.
> > 
> > That said, it's *not* divorced from its filesystem, which I understand
> > is what you're really looking for. Unfortunately that information
> > doesn't appear to live on its own in the audit trail, and it's not
> > available to filter on for these calls. This would leave you having to
> > audit all read() and write() calls and filtering them in
> > post-processing. I doubt this would be a practical solution.
> 
> Another option might be to audit based on SELinux type, if the files in
> question can have a different type than the rest of the files.  A
> SELinux audit message on write would generate an AVC_PATH record.
> 
> Example:
> $ vi myaudit.te
> policy_module(myaudit, 1.0)
> require {
> 	attribute domain;
> 	type etc_t;
> }
> auditallow domain etc_t:file write;
> :q
> 
> $ make -f /usr/share/selinux/devel/Makefile myaudit.pp
> $ su
> # /usr/sbin/semodule -i myaudit.pp
> # vi /etc/fstab
> :wq
> 
> Audit log contains:
> type=AVC msg=audit(1183463480.620:6351): avc:  granted  { write } for  pid=13773 comm="vi" name="fstab" dev=dm-0 ino=1280187 scontext=user_u:system_r:unconfined_t:s0 tcontext=user_u:object_r:etc_t:s0 tclass=file
> type=SYSCALL msg=audit(1183463480.620:6351): arch=40000003 syscall=4 success=yes exit=767 a0=3 a1=9047510 a2=2ff a3=2ff items=0 ppid=13662 pid=13773 auid=4204 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 comm="vi" exe="/bin/vi" subj=user_u:system_r:unconfined_t:s0 key=(null)
> type=AVC_PATH msg=audit(1183463480.620:6351):  path="/etc/fstab"

One caveat though - auditing of write() won't catch all possible ways of
modifying the file data, e.g. one could mmap() the file with MAP_SHARED
and then write to the memory, followed by msync or munmap.

-- 
Stephen Smalley
National Security Agency

Re: Decoding arguments passed to system calls

[Steve Grubb]

On Monday 02 July 2007 05:46:26 pm Darryl Dixon - Winterhouse Consulting 
wrote:
> Forgive me if this isn't the correct forum for this,

This is the correct forum for this.  :)

> Scenario:
> A very large filesystem with potentially millions of files in an ad-hoc,
> unordered directory structure. The requirement is to be able to audit any
> action on any file in this filesystem (moves, adds, changes, deletes,
> etc).

The new directory auditing capabilities should solve this problem. It will be 
in 2.6.23 and RHEL5.1 kernels.

> Hypothetical solution:
> Clearly, scanning the filesystem with `find` and adding calling auditctl
> with the appropriate arguments to generate a watch on every singly file is
> totally infeasible

That's where the directory auditing directives will help. You should only have 
to place and audit watch ob the directory while using audit package 1.5.4 or 
higher. The kernel will do the rest.

> (find takes almost an hour to run, and in the meantime 
> stuff is potentially changing...). Instead, I envision it would make
> better sense to simply audit every call to write(), open(), rename(), etc,

You could audit open, rename, symlink, unlink, etc. But don't audit write or 
read (it will fill your disk and won't cover the cases where the file is 
memory mapped). Assume that opening with write flags means it will be written 
to. You could do this and limit the auditing by dev major and dev minor so 
that you don't get too many records.

> My problem is that this doesn't seem possible with the Linux Audit
> subsystem, as the arguments to the system calls are not decoded (eg, the
> audit records for write() include only an opaque filehandle and pointer to
> the written data, etc). 

They are decoded if you pass the -i, but if the information is not collected, 
there is nothing to decode. In the case or write, the path is not collected. 
The path is collected at the open and the return code is the descriptor 
number if its non-negative.

> 1) Am I totally wrong and there's a method of getting this information
> already that I have overlooked?

Which kernel are you using? I think the directory auditing will solve your 
problem. But note what I was saying about reads & writes and mmap.

> 2) Knowing very little about the auditing subsystem, and the kernel
> internals in general I envision that decoding the filehandle into a path
> is something that would need to be done in the kernel, and is impossible
> from userland. Is this the case?

Its possible. you would have to do it yourself. I'd suggest using the auparse 
library.


> 3) How much work do you all estimate that it would actually take to be
> able to generate this information? 

probably a lot. The audit systems view of the word is similar to strace except 
from the kernel's PoV. There is no data retained between syscalls about 
syscalls.

> Is it even possible without a major architectural overhaul of the audit
> subsystem? 

You can certainly do it from user space.

-Steve

Re: Decoding arguments passed to system calls

[Steve Grubb]

On Monday 02 July 2007 06:48:23 pm Darryl Dixon - Winterhouse Consulting 
wrote:
> What you say about not being able to audit 'write()' is worrying to me. The
> problem with auditing write by inference from open(), is that one doesn't
> know *when* the file was written,

But you know who did it and when they started the process of writing to the 
file by opening with the intent to write. mmap is not covered by auditing for 
write, so you have a big hole anyways.

> My assumption would have been that CWD reflected only where the exe was
> launched from, and not necessarily where the write()-en file was located...

CWD is the processes' cwd. Some programs do a chdir("/") right after starting, 
so CWD would reflect wherever the app chdir'ed to.

-Steve

Re: Decoding arguments passed to system calls

[Steve Grubb]

On Tuesday 03 July 2007 10:38:05 am Stephen Smalley wrote:
> One caveat though - auditing of write() won't catch all possible ways of
> modifying the file data, e.g. one could mmap() the file with MAP_SHARED
> and then write to the memory, followed by msync or munmap.

Agreed. And another gotcha is programs that could pass a descriptor across 
af_unix sockets where it is then mmap'ed. There is also sendfile which could 
send the file away to be viewed by other people and there is splice() & 
tee(2). Don't forget the *at() syscalls, too. IOW, I think the problem is 
trickier than it might initially appear.

Based on your requirements, you might want to consider putting in place some 
SE Linux policy to control the different ways that a file can be accessed to 
keep apps honest. Then you don't need to worry about all the sneak paths that 
could subvert the audit system.

-Steve