[RFC] programmatic IDS routing

[RFC] programmatic IDS routing

[Steve Grubb]

Hi,

In working on the IDS piece of the audit system, there are 3 detections that I 
need to solve: watched files, watched execution, and making an executable. In 
this particular case, an admin may have watches and several things for 
archival purposes, but may want an alert if something changes in specific 
files. In order for the IDS system to be able to distinguish an open of a 
watched file from an open of a *special* watched file that an alert should be 
sent for, I'd like to propose a standard way of alerting the IDS that this 
record needs additional scrutiny.

The key field has been in all syscall records for at least 8 kernel releases, 
so I think it is well established that it could be used by this point. It is 
32 char's long. This is longer than what most people will ever need. What I'd 
like to do is to use this field for 2 purposes. One is to continue using it 
as is. The other could be to tell the IDS that this record needs review.

If a key field starts with "ids-" then we could parse it for a specific set of 
uses. We need to be able to distinguish: file, exec, and mkexe. Each of these 
would alert the IDS as to which alert category this fits under. Next, IDMEF 
has 4 severity ratings: info, low, medium, and high. These could be shortened 
to: inf, low, med, hi. This means you could tell the IDS that this is an 
execution attempt that means medium severity by this, "ids-exec-med".

To continue to be able to use the key field for searching, I would propose 
just adding a comma to separate the ids part from the searchable part. So, 
you could do something like this: "ids-file-med,shadow-password". auditctl 
can be fixed so that when it spots "ids-" in the key field, it will check 
that only valid types and severities are being used. This would not affect 
what auditd does in any way.

Thoughts? Comments?

-Steve

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> Hi,
> 
> In working on the IDS piece of the audit system, there are 3 detections that I 
> need to solve: watched files, watched execution, and making an executable. In 
> this particular case, an admin may have watches and several things for 
> archival purposes, but may want an alert if something changes in specific 
> files. In order for the IDS system to be able to distinguish an open of a 
> watched file from an open of a *special* watched file that an alert should be 
> sent for, I'd like to propose a standard way of alerting the IDS that this 
> record needs additional scrutiny.
> 
> The key field has been in all syscall records for at least 8 kernel releases, 
> so I think it is well established that it could be used by this point. It is 
> 32 char's long. This is longer than what most people will ever need. What I'd 
> like to do is to use this field for 2 purposes. One is to continue using it 
> as is. The other could be to tell the IDS that this record needs review.

Rather than using the key for two purposes and introducing special key
words, couldn't an admin just tell the IDS which he's are of interest?
And what the priority of each one is?

> If a key field starts with "ids-" then we could parse it for a specific set of 
> uses. We need to be able to distinguish: file, exec, and mkexe. Each of these 
> would alert the IDS as to which alert category this fits under. Next, IDMEF 
> has 4 severity ratings: info, low, medium, and high. These could be shortened 
> to: inf, low, med, hi. This means you could tell the IDS that this is an 
> execution attempt that means medium severity by this, "ids-exec-med".
> 
> To continue to be able to use the key field for searching, I would propose 
> just adding a comma to separate the ids part from the searchable part. So, 
> you could do something like this: "ids-file-med,shadow-password". auditctl 
> can be fixed so that when it spots "ids-" in the key field, it will check 
> that only valid types and severities are being used. This would not affect 
> what auditd does in any way.
> 
> Thoughts? Comments?

I think it makes more sense to leave the key alone and introduce
something that optionally associates keys with IDS properties.

-- ljk
> 
> -Steve
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 13:12:22 Linda Knippers wrote:
> Rather than using the key for two purposes and introducing special key
> words, couldn't an admin just tell the IDS which he's are of interest?
> And what the priority of each one is?

The problem is that you can tell the IDS that you want any reads 
of /opt/my-secrets, but unless you have a matching audit rule you will not 
get any records. This allows you to make sure you have a watch paired with 
its meaning.

-Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 13:40:21 Steve Grubb wrote:
> On Wednesday 19 March 2008 13:12:22 Linda Knippers wrote:
> > Rather than using the key for two purposes and introducing special key
> > words, couldn't an admin just tell the IDS which he's are of interest?
> > And what the priority of each one is?
>
> The problem is that you can tell the IDS that you want any reads
> of /opt/my-secrets, but unless you have a matching audit rule you will not
> get any records. This allows you to make sure you have a watch paired with
> its meaning.

And I should add, the IDS could run on each remote system, or off an 
aggregator. This means expressing rules gets more complicated when you have 
to express rules as on this particular host, I am looking for files in this 
location. To me, its just simpler and hopefully less error prone to use the 
key field like this.

-Steve

Re: [RFC] programmatic IDS routing

[Valdis.Kletnieks]

[-- Attachment #1.1: Type: text/plain, Size: 527 bytes --]

On Wed, 19 Mar 2008 13:02:48 EDT, Steve Grubb said:

> files. In order for the IDS system to be able to distinguish an open of a
> watched file from an open of a *special* watched file that an alert should be
> sent for, I'd like to propose a standard way of alerting the IDS that this
> record needs additional scrutiny.

Why do we need special handling for something the IDS should be able to do for
itself? If your IDS system doesn't already have a copy of the list of "special"
watched files, you have *bigger* problems.



[-- Attachment #1.2: Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC] programmatic IDS routing

[Valdis.Kletnieks]

[-- Attachment #1.1: Type: text/plain, Size: 1590 bytes --]

On Wed, 19 Mar 2008 13:40:21 EDT, Steve Grubb said:
> On Wednesday 19 March 2008 13:12:22 Linda Knippers wrote:
> > Rather than using the key for two purposes and introducing special key
> > words, couldn't an admin just tell the IDS which he's are of interest?
> > And what the priority of each one is?
> 
> The problem is that you can tell the IDS that you want any reads 
> of /opt/my-secrets, but unless you have a matching audit rule you will not 
> get any records. This allows you to make sure you have a watch paired with 
> its meaning.

You have this backwards.

If you have a "special" watch on /foo/bar, and you see an event arrive, the
IDS should already know that /foo/bar is special and handle it accordingly,
and it doesn't need to be told "this is special" in the audit record.  One
can make the case that it's *helpful* so that the IDS can note "Oh yes, this
is a special file, and the audit record says it's special, so it matches".

However, *no* amount of special tagging will allow the IDS to disambiguate
these two cases:

1) An audit rule was set, but no events generated because no activity matched.

2) An audit rule wasn't set at all.

"unless you have a matching audit rule you will not get any records" means
exactly that - so tagging the records you don't receive isn't useful.

There *is* the more general case of "I had a generic rule and a special watch
and *both* fired" - but that problem is in no way IDS specific, but applies
to *any* time that an event triggers more than one rule. We shouldn't be
coding IDS-specific solutions to the general problem.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 14:05:42 Valdis.Kletnieks@vt.edu wrote:
> On Wed, 19 Mar 2008 13:02:48 EDT, Steve Grubb said:
> > files. In order for the IDS system to be able to distinguish an open of a
> > watched file from an open of a *special* watched file that an alert
> > should be sent for, I'd like to propose a standard way of alerting the
> > IDS that this record needs additional scrutiny.
>
> Why do we need special handling for something the IDS should be able to do
> for itself? 

Something has to tell the IDS what to do for these 3 particular detections. It 
could come from a configuration file that it reads, or it could come from the 
event stream that it reads. Its just a matter of *where* the instructions 
come from.


> If your IDS system doesn't already have a copy of the list of 
> "special" watched files, you have *bigger* problems.

Not really, just think about the advantages of this approach. 

o You do not need to express host:file relationships if you are checking 
aggregate logs
o You always have a matching audit rule to make sure you get the data you need
o The event when recorded to disk has the meaning associated with it in case 
you wonder why something didn't get classified the way you thought it should
o This technique is higher performance since you do not need iterate across a 
list of all files for each incoming event.
o If the target file has a hardlink that is manipulated, the IDS won't be 
fooled because a different path showed up in the event stream. (This might 
even come into play for mount tricks that alter paths.)

These are the easy ones that I can think up easily. While we are at it, the 
disadvantages to using the IDS configuration file approach:

o The config file will become a mess when you get to this one entry that 
contains all file names one after another. Or you will have 2 config files 
one for the options and one for the files. You will of course need 2 more 
files for the other 2 detections, so now you have 4 config files to setup.
o No guarantee that any audit rules exist to give you the events you need.
o Lower performance
o Uses more system memory
o Susceptible to path name tricks
o Code will be far more complicated and harder to read or understand due to 
size.
o You have to be very careful to keep aggregate server rules in sync with 
remote system.

-Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 14:18:12 Valdis.Kletnieks@vt.edu wrote:
> However, *no* amount of special tagging will allow the IDS to disambiguate
> these two cases:
>
> 1) An audit rule was set, but no events generated because no activity
> matched.

In which case you have nothing to worry about.  :)


> 2) An audit rule wasn't set at all.

Again nothing to worry about since they haven't set the system up yet.


> "unless you have a matching audit rule you will not get any records" means
> exactly that - so tagging the records you don't receive isn't useful.

But if you don't receive any records, nothing happened. :)


> There *is* the more general case of "I had a generic rule and a special
> watch and *both* fired" - but that problem is in no way IDS specific,

Right, this *is* something to worry about. I was thinking that we could solve 
this by having an option that tells the kernel to evaluate all rules and not 
just first match.

I have also been wondering about detecting shadowed rules and warning when 
auditctl finishes a file.

-Steve

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 19 March 2008 14:05:42 Valdis.Kletnieks@vt.edu wrote:
>> On Wed, 19 Mar 2008 13:02:48 EDT, Steve Grubb said:
>>> files. In order for the IDS system to be able to distinguish an open of a
>>> watched file from an open of a *special* watched file that an alert
>>> should be sent for, I'd like to propose a standard way of alerting the
>>> IDS that this record needs additional scrutiny.
>> Why do we need special handling for something the IDS should be able to do
>> for itself? 
> 
> Something has to tell the IDS what to do for these 3 particular detections. It 
> could come from a configuration file that it reads, or it could come from the 
> event stream that it reads. Its just a matter of *where* the instructions 
> come from.
> 
> 
>> If your IDS system doesn't already have a copy of the list of 
>> "special" watched files, you have *bigger* problems.
> 
> Not really, just think about the advantages of this approach. 
> 
> o You do not need to express host:file relationships if you are checking 
> aggregate logs
> o You always have a matching audit rule to make sure you get the data you need
> o The event when recorded to disk has the meaning associated with it in case 
> you wonder why something didn't get classified the way you thought it should
> o This technique is higher performance since you do not need iterate across a 
> list of all files for each incoming event.
> o If the target file has a hardlink that is manipulated, the IDS won't be 
> fooled because a different path showed up in the event stream. (This might 
> even come into play for mount tricks that alter paths.)
> 
> These are the easy ones that I can think up easily. While we are at it, the 
> disadvantages to using the IDS configuration file approach:
> 
> o The config file will become a mess when you get to this one entry that 
> contains all file names one after another. Or you will have 2 config files 
> one for the options and one for the files. You will of course need 2 more 
> files for the other 2 detections, so now you have 4 config files to setup.
> o No guarantee that any audit rules exist to give you the events you need.
> o Lower performance
> o Uses more system memory
> o Susceptible to path name tricks
> o Code will be far more complicated and harder to read or understand due to 
> size.
> o You have to be very careful to keep aggregate server rules in sync with 
> remote system.

I'm not sure why all of the above apply.  If an IDS has a dependency on
audit and specific audit rules to get the information it needs, it can
use the information in its config file to construct the audit rules it
needs.  I don't think an IDS config file needs to be any more
complicated than an audit rules rules, and in fact should be simpler.
If the IDS is using the audit system, then I don't think the rest apply
either.

-- ljk
> 
> -Steve
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 15:04:57 Linda Knippers wrote:
> I'm not sure why all of the above apply.

Because this IDS is part of the audit system.

> If an IDS has a dependency on audit and specific audit rules to get the
> information it needs, it can use the information in its config file to
> construct the audit rules it needs.

Then you surely have duplicate rules controlled by 2 systems. The first rule 
in the audit.rules file is -D which would delete not only the audit event 
rules for archival purposes, but any IDS placed rules. There is not a simple 
way of deleting the rules placed by auditctl vs the ones placed by the IDS. 
The IDS system would also need to be prodded to reload its set of rules 
again.

> I don't think an IDS config file needs to be any more complicated than an
> audit rules, and in fact should be simpler.

I think it would be more complicated going down this path for a number of 
reasons.

-Steve

Re: [RFC] programmatic IDS routing

[Eric Paris]

On Wed, 2008-03-19 at 15:28 -0400, Steve Grubb wrote:
> On Wednesday 19 March 2008 15:04:57 Linda Knippers wrote:
> > I'm not sure why all of the above apply.
> 
> Because this IDS is part of the audit system.
> 
> > If an IDS has a dependency on audit and specific audit rules to get the
> > information it needs, it can use the information in its config file to
> > construct the audit rules it needs.
> 
> Then you surely have duplicate rules controlled by 2 systems. The first rule 
> in the audit.rules file is -D which would delete not only the audit event 
> rules for archival purposes, but any IDS placed rules. There is not a simple 
> way of deleting the rules placed by auditctl vs the ones placed by the IDS. 
> The IDS system would also need to be prodded to reload its set of rules 
> again.

If someone does -D they lose no matter what no matter how we solve
this  :)

I find it objectionable that they sysadmin has to learn some new
arbitrary key requirements.  Could the ids system parse its own
configuration file and automatically generating audit.rules.ids which is
just cat'ed onto the end of audit.rules for purposes of statup scripts
and things like that?  Then I wouldn't have a problem with it setting
its own key however it wanted the key.   Although admittedly I have no
idea what happens if you do

-a exit,always -S all -k hey2
-a exit,always -S all -k key2

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 19 March 2008 15:04:57 Linda Knippers wrote:
>> I'm not sure why all of the above apply.
> 
> Because this IDS is part of the audit system.

Is there something that describes what you're building so we can
have the right context to comment on this?  I assumed you were
building something that would be a dispatcher plug-in or something
rather than building something new into the core audit subsystem.

>> If an IDS has a dependency on audit and specific audit rules to get the
>> information it needs, it can use the information in its config file to
>> construct the audit rules it needs.
> 
> Then you surely have duplicate rules controlled by 2 systems. The first rule 
> in the audit.rules file is -D which would delete not only the audit event 
> rules for archival purposes, but any IDS placed rules. There is not a simple 
> way of deleting the rules placed by auditctl vs the ones placed by the IDS. 
> The IDS system would also need to be prodded to reload its set of rules 
> again.

An IDS should be able to be prodded to reload its rules.  And it should
do something if it sees audit being disabled.  If someone wants IDS
functionality, they'd probably be using the IDS to manage the files
they're watching so I don't think you'd have IDS watches and separate
audit watches.
> 
>> I don't think an IDS config file needs to be any more complicated than an
>> audit rules, and in fact should be simpler.
> 
> I think it would be more complicated going down this path for a number of 
> reasons.

To me it seems more complicated to bundle everything together and
overload the key.

-- ljk
> 
> -Steve

Re: [RFC] programmatic IDS routing

[Valdis.Kletnieks]

[-- Attachment #1.1: Type: text/plain, Size: 1212 bytes --]

On Wed, 19 Mar 2008 14:54:16 EDT, Steve Grubb said:

>> 2) An audit rule wasn't set at all.

> Again nothing to worry about since they haven't set the system up yet.

No - it's one of the failure modes you said you were worried about:

> The problem is that you can tell the IDS that you want any reads 
> of /opt/my-secrets, but unless you have a matching audit rule you will not 
> get any records. This allows you to make sure you have a watch paired with 
> its meaning.

Exactly - if you're missing the rule, you don't get records.

Determining whether it's a problem because a rule is missing, or not a
problem because "it's not set up yet", isn't anything the kernel should be
involved in - other than to maybe notify us "Hey dood, you have exactly zero
rules set, you might want to check what happened".

> I have also been wondering about detecting shadowed rules and warning when 
> auditctl finishes a file.

I wasn't even thinking about that - I was thinking of the ones that are like
the old SNL skit - a dessert topping *and* a floor wax.  Say, one rule triggered
on an event because it's an unsuccessful open, and another rule would have
triggered because it was a reference to a watched file....

[-- Attachment #1.2: Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 15:48:54 Eric Paris wrote:
> > Then you surely have duplicate rules controlled by 2 systems. The first
> > rule in the audit.rules file is -D which would delete not only the audit
> > event rules for archival purposes, but any IDS placed rules. There is not
> > a simple way of deleting the rules placed by auditctl vs the ones placed
> > by the IDS. The IDS system would also need to be prodded to reload its
> > set of rules again.
>
> If someone does -D they lose no matter what no matter how we solve
> this  :)

Well, in the way I propose, all the rest of the lines of audit.rules sets it 
back up.


> I find it objectionable that they sysadmin has to learn some new
> arbitrary key requirements.

Its not arbitrary if it follows a defined and agreed upon pattern.  ;)

> Could the ids system parse its own configuration file and automatically
> generating audit.rules.ids which is just cat'ed onto the end of audit.rules
> for purposes of statup scripts and things like that?

I suppose it could, but then what if you wanted to do something complicated 
like:

-a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k 
ids-file-med

or 

-a always,exit -F perms=wa -F subj_role=webadmin_r -F exit=-EPERM -k 
ids-file-med

In order to allow the expressiveness that auditctl rules could perform, you 
need to build this into the configuration that the IDS reads. As you add each 
capability, you suddenly realize you just wrote auditctl another way. So, its 
either do simplistic watches for the IDS or you wind up writing auditctl.

> Although admittedly I have no idea what happens if you do
>
> -a exit,always -S all -k hey2
> -a exit,always -S all -k key2

This would generate a lot of events, some would be trapped by the IDS, but 
none would fall into the watched file/exec/mkexe buckets of the IDS.

-Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 15:55:23 Linda Knippers wrote:
> > Because this IDS is part of the audit system.
>
> Is there something that describes what you're building so we can
> have the right context to comment on this?

I presented this:

http://people.redhat.com/sgrubb/audit/summit07_audit_ids.odp

at last year's Red Hat Summit. The idea is roughly the same, but the 
configuration is slightly different.

> I assumed you were building something that would be a dispatcher plug-in or
> something rather than building something new into the core audit subsystem.

It is. Its tightly integrated.

> >> If an IDS has a dependency on audit and specific audit rules to get the
> >> information it needs, it can use the information in its config file to
> >> construct the audit rules it needs.
> >
> > Then you surely have duplicate rules controlled by 2 systems. The first
> > rule in the audit.rules file is -D which would delete not only the audit
> > event rules for archival purposes, but any IDS placed rules. There is not
> > a simple way of deleting the rules placed by auditctl vs the ones placed
> > by the IDS. The IDS system would also need to be prodded to reload its
> > set of rules again.
>
> An IDS should be able to be prodded to reload its rules.

Sure, but you have the audit system loading CAPP rules with all those watches 
and then maybe the admin wants any write to shadow to be a high alert since 
he's the only user and won't be changing his password. We would either need 
to analyze the rules and make sure they are simplistic enough for the IDS to 
be guaranteed an event, or just add more rules to make sure we got 'em. In 
this case, we may windup creating records that the admin did not want on the 
disk. 

i just see this as progressing into a mess. We have 2 things that have 
different ideas about what needs to be tracked. Neither understanding why the 
other is doing something and not happy because either too much data is going 
to disk or not enough events to trap something important.

By using the key field, its in plain sight and done with purpose. Not enough 
events to trap something important, widen it in audit.rules and you also know 
that this will send more to disk. No suprises there.

-Steve

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 19 March 2008 15:55:23 Linda Knippers wrote:
>>> Because this IDS is part of the audit system.
>> Is there something that describes what you're building so we can
>> have the right context to comment on this?
> 
> I presented this:
> 
> http://people.redhat.com/sgrubb/audit/summit07_audit_ids.odp
> 
> at last year's Red Hat Summit. The idea is roughly the same, but the 
> configuration is slightly different.

To me, what's in the slides looks better.  There are lots of things
an IDS might care about, some of which can be expressed in audit
rules and some which can't.  I think the IDS should make sure that
the rules it cares about are there and it should react when there's
a change to the audit configuration.

>> I assumed you were building something that would be a dispatcher plug-in or
>> something rather than building something new into the core audit subsystem.
> 
> It is. Its tightly integrated.

Well, according to the slides its either layered or plugged in, which
I think is better than tightly integrated.  I don't think audit should
have special knowledge of the things that might be plugged into it.

> 
>>>> If an IDS has a dependency on audit and specific audit rules to get the
>>>> information it needs, it can use the information in its config file to
>>>> construct the audit rules it needs.
>>> Then you surely have duplicate rules controlled by 2 systems. The first
>>> rule in the audit.rules file is -D which would delete not only the audit
>>> event rules for archival purposes, but any IDS placed rules. There is not
>>> a simple way of deleting the rules placed by auditctl vs the ones placed
>>> by the IDS. The IDS system would also need to be prodded to reload its
>>> set of rules again.
>> An IDS should be able to be prodded to reload its rules.
> 
> Sure, but you have the audit system loading CAPP rules with all those watches 
> and then maybe the admin wants any write to shadow to be a high alert since 
> he's the only user and won't be changing his password. We would either need 
> to analyze the rules and make sure they are simplistic enough for the IDS to 
> be guaranteed an event, or just add more rules to make sure we got 'em. In 
> this case, we may windup creating records that the admin did not want on the 
> disk. 

If its a high alert, why wouldn't you also want it on the disk?  And
how would you specify that you do or don't want it written?  Isn't
the function of the auditd to "simply dequeue events from netlink
interface as fast and possible and log them to disk" (slide 17)?
I don't think it ought to be deciding which audit events go to disk
vs. go to specific dispatcher plug-ins.
> 
> i just see this as progressing into a mess. We have 2 things that have 
> different ideas about what needs to be tracked. Neither understanding why the 
> other is doing something and not happy because either too much data is going 
> to disk or not enough events to trap something important.

I don't see why this has to progress into a mess.  If someone wants
to use the audit rules to drive what the IDS is looking at, then they
could use existing audit rules and specify, for example, that any audit
event with key "CFG_shadow" are of interest with a specific priority
of alert.  The IDS could even check to see if there's a rule that
has that key, and if not, flag some kind of warning.
> 
> By using the key field, its in plain sight and done with purpose. Not enough 
> events to trap something important, widen it in audit.rules and you also know 
> that this will send more to disk. No suprises there.

I'm actually in favor of using the key, just in using it like its used
today.  All the capp watches have unique keys, and an admin could create
more/different rules with different keys.  I think that the IDS
should have different configuration information to tell it what to look
for and what to do with it, rather than embedding it into a short field
in the audit record.  What if other plug-ins also want to use that
field?

-- ljk
> 
> -Steve

Re: [RFC] programmatic IDS routing

[Eric Paris]

On Wed, 2008-03-19 at 17:31 -0400, Linda Knippers wrote:
> Steve Grubb wrote:

> > By using the key field, its in plain sight and done with purpose. Not enough 
> > events to trap something important, widen it in audit.rules and you also know 
> > that this will send more to disk. No suprises there.
> 
> I'm actually in favor of using the key, just in using it like its used
> today.  All the capp watches have unique keys, and an admin could create
> more/different rules with different keys.  I think that the IDS
> should have different configuration information to tell it what to look
> for and what to do with it, rather than embedding it into a short field
> in the audit record.  What if other plug-ins also want to use that
> field?

So maybe all we need is for the ids config file needs to be of the form

key type priority

so I can set up my audit rule however I want say

-a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k 500EPERM
-a always,exit -F perms=wa -F subj_role=webadmin_r -F exit=-EPERM -k webadminEPERM

And my ids config file would look like:

500EPERM	file	med
webadminEPERM	exec	high

And on startup the ids can easily look to see if 500EPERM and
webadminEPERM are actually keys to real rules just for sanity sake.  Is
the reverse mapping from key to ids action really so expensive that this
is unreasonable?

I tend to also agree with the part of the discussion which says that it
isn't audit's place to decide that some rules are meant for disk and
some rules aren't.  Unless we add some new explicit rule syntax for just
that case.....

-Eric

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 17:41:07 Eric Paris wrote:
> So maybe all we need is for the ids config file needs to be of the form
>
> key type priority

And hostname. Remember that this could be run from an aggregator.


> so I can set up my audit rule however I want say
>
> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k
> 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F exit=-EPERM
> -k webadminEPERM
>
> And my ids config file would look like:
>
> 500EPERM        file    med
> webadminEPERM   exec    high

This is pretty close to the idea that I started with. Then I thought, how do I 
make this engine run faster? How do I reduce memory consumption (since the 
keys have to be stored in memory)? How do I make sure that the keys are there 
and correct?


> And on startup the ids can easily look to see if 500EPERM and
> webadminEPERM are actually keys to real rules just for sanity sake.  

Sure...but audit rules are loaded after auditd starts so that we can record 
them being put into effect. So, you would have to wait for a a while after 
startup to do this.

> Is the reverse mapping from key to ids action really so expensive that this
> is unreasonable?

Consider a datacenter with many hosts that may want to run this off of the 
aggregator. There will be a high rate of incoming events and a bit of 
comparing to figure out if each event something we care about. 

With my proposal, I can tell with strncmp(key, "ids-", 4) if this is anything 
we need to pay attention to. So, inspection of 4 bytes let me decide yes/no. 
Its a finite amount of time and doesn't linearly slow down the system as more 
hosts and files of interest are configured. It scales well.


> I tend to also agree with the part of the discussion which says that it
> isn't audit's place to decide that some rules are meant for disk and
> some rules aren't. 

I agree and never proposed that.

-Steve

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 19 March 2008 17:41:07 Eric Paris wrote:
>> So maybe all we need is for the ids config file needs to be of the form
>>
>> key type priority
> 
> And hostname. Remember that this could be run from an aggregator.
> 
> 
>> so I can set up my audit rule however I want say
>>
>> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k
>> 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F exit=-EPERM
>> -k webadminEPERM
>>
>> And my ids config file would look like:
>>
>> 500EPERM        file    med
>> webadminEPERM   exec    high
> 
> This is pretty close to the idea that I started with. Then I thought, how do I 
> make this engine run faster? How do I reduce memory consumption (since the 
> keys have to be stored in memory)? 

How does overloading the key field help either of those?

> How do I make sure that the keys are there and correct?

That can be a startup check.
> 
>> And on startup the ids can easily look to see if 500EPERM and
>> webadminEPERM are actually keys to real rules just for sanity sake.  
> 
> Sure...but audit rules are loaded after auditd starts so that we can record 
> them being put into effect. So, you would have to wait for a a while after 
> startup to do this.

So the IDS starts after auditd, but it depends on auditd anyway.
> 
>> Is the reverse mapping from key to ids action really so expensive that this
>> is unreasonable?
> 
> Consider a datacenter with many hosts that may want to run this off of the 
> aggregator. There will be a high rate of incoming events and a bit of 
> comparing to figure out if each event something we care about. 

I think if someone is using audit as an IDS, they're going to care about
everything they're auditing or why are they auditing it?
> 
> With my proposal, I can tell with strncmp(key, "ids-", 4) if this is anything 
> we need to pay attention to. So, inspection of 4 bytes let me decide yes/no. 

Who is doing this, the auditd, the dispatcher or the plugin?
Couldn't you hash the key?

> Its a finite amount of time and doesn't linearly slow down the system as more 
> hosts and files of interest are configured. It scales well.
> 
> 
>> I tend to also agree with the part of the discussion which says that it
>> isn't audit's place to decide that some rules are meant for disk and
>> some rules aren't. 
> 
> I agree and never proposed that.

I guess Eric and I were both confused then by your comment about the
admin ending up with more audit events on disk than intended.

-- ljk
> 
> -Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 19:00:41 Linda Knippers wrote:
> Steve Grubb wrote:
> > On Wednesday 19 March 2008 17:41:07 Eric Paris wrote:
> >> So maybe all we need is for the ids config file needs to be of the form
> >>
> >> key type priority
> >
> > And hostname. Remember that this could be run from an aggregator.
> >
> >> so I can set up my audit rule however I want say
> >>
> >> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k
> >> 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F
> >> exit=-EPERM -k webadminEPERM
> >>
> >> And my ids config file would look like:
> >>
> >> 500EPERM        file    med
> >> webadminEPERM   exec    high
> >
> > This is pretty close to the idea that I started with. Then I thought, how
> > do I make this engine run faster? How do I reduce memory consumption
> > (since the keys have to be stored in memory)?
>
> How does overloading the key field help either of those?

I wouldn't call this overloading the key field. The key field's purpose is to 
allow searches for groups of audit events. You've traditionally used this 
from the command prompt with ausearch. Now with the auparse library, it can 
be used programmatically for searching by more utilities.

The way that it helps is that I can tell yes/no in 4 byte compares and without 
storing any linked lists, hashes, or binary trees.


> > How do I make sure that the keys are there and correct?
>
> That can be a startup check.

How do you do that remotely?


> >> Is the reverse mapping from key to ids action really so expensive that
> >> this is unreasonable?
> >
> > Consider a datacenter with many hosts that may want to run this off of
> > the aggregator. There will be a high rate of incoming events and a bit of
> > comparing to figure out if each event something we care about.
>
> I think if someone is using audit as an IDS, they're going to care about
> everything they're auditing or why are they auditing it?

That's not true. When it comes to IDS, the real question is how to effectively 
escalate events of interest without overloading the decision makers. 
Something that alerts once a second is of no use to anyone. You want it to 
pick out the things you really care about for realtime and save everything 
else for a daily report.


> > With my proposal, I can tell with strncmp(key, "ids-", 4) if this is
> > anything we need to pay attention to. So, inspection of 4 bytes let me
> > decide yes/no.
>
> Who is doing this, the auditd, the dispatcher or the plugin?

The IDS plugin.


> Couldn't you hash the key?

Yes. Takes more time than strncmp and then you may still have collisions where 
you walk a list of some kind. This also has the side effect of needing to set 
aside a good chunk of memory for the hashes - used or not.


> > Its a finite amount of time and doesn't linearly slow down the system as
> > more hosts and files of interest are configured. It scales well.
> >
> >> I tend to also agree with the part of the discussion which says that it
> >> isn't audit's place to decide that some rules are meant for disk and
> >> some rules aren't.
> >
> > I agree and never proposed that.
>
> I guess Eric and I were both confused then by your comment about the
> admin ending up with more audit events on disk than intended.

-a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F path=/etc/password

and

-w /etc/password

collect entirely different amounts of data.

-Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Wednesday 19 March 2008 17:31:49 Linda Knippers wrote:
> Steve Grubb wrote:
> > On Wednesday 19 March 2008 15:55:23 Linda Knippers wrote:
> >>> Because this IDS is part of the audit system.
> >>
> >> Is there something that describes what you're building so we can
> >> have the right context to comment on this?
> >
> > I presented this:
> >
> > http://people.redhat.com/sgrubb/audit/summit07_audit_ids.odp
> >
> > at last year's Red Hat Summit. The idea is roughly the same, but the
> > configuration is slightly different.
>
> To me, what's in the slides looks better.

I think that approach won't scale. I've been thinking about this and other 
surrounding issues for a long time now. Speed is very important in an IDS.


> There are lots of things an IDS might care about, some of which can be
> expressed in audit rules and some which can't.

Right, there are going to be about 15-20 different detections of potential 
problems. The only 3 I'm concerned with right now is escalation of watched 
files, executables, and attempts to make something an executable. The others 
are either solved or not in development right now.


> I think the IDS should make sure that the rules it cares about are there and
> it should react when there's a change to the audit configuration.

That is difficult. There's no way to tell when a new set of rules is about to 
be loaded that may shadow the IDS rules. It is far easier to just let the 
admin express in the rules which one he/she cares about. Easier also means 
that its more likely to work as advertised.


> >> I assumed you were building something that would be a dispatcher plug-in
> >> or something rather than building something new into the core audit
> >> subsystem.
> >
> > It is. Its tightly integrated.
>
> Well, according to the slides its either layered or plugged in, which
> I think is better than tightly integrated.  I don't think audit should
> have special knowledge of the things that might be plugged into it.

It doesn't. The admin does.


> >>>> If an IDS has a dependency on audit and specific audit rules to get
> >>>> the information it needs, it can use the information in its config
> >>>> file to construct the audit rules it needs.
> >>>
> >>> Then you surely have duplicate rules controlled by 2 systems. The first
> >>> rule in the audit.rules file is -D which would delete not only the
> >>> audit event rules for archival purposes, but any IDS placed rules.
> >>> There is not a simple way of deleting the rules placed by auditctl vs
> >>> the ones placed by the IDS. The IDS system would also need to be
> >>> prodded to reload its set of rules again.
> >>
> >> An IDS should be able to be prodded to reload its rules.
> >
> > Sure, but you have the audit system loading CAPP rules with all those
> > watches and then maybe the admin wants any write to shadow to be a high
> > alert since he's the only user and won't be changing his password. We
> > would either need to analyze the rules and make sure they are simplistic
> > enough for the IDS to be guaranteed an event, or just add more rules to
> > make sure we got 'em. In this case, we may windup creating records that
> > the admin did not want on the disk.
>
> If its a high alert, why wouldn't you also want it on the disk?

You do.


> And how would you specify that you do or don't want it written?

With my proposal, everything is written to disk. The difference is the admin 
is aware as he/she adds the rule of what consequences it has. They make the 
choice between fill my disk up and do I have enough events to catch what I 
want realtime notification of.


> Isn't the function of the auditd to "simply dequeue events from netlink
> interface as fast and possible and log them to disk" (slide 17)?

Yep.


> I don't think it ought to be deciding which audit events go to disk
> vs. go to specific dispatcher plug-ins.

It shouldn't.


> > i just see this as progressing into a mess. We have 2 things that have
> > different ideas about what needs to be tracked. Neither understanding why
> > the other is doing something and not happy because either too much data
> > is going to disk or not enough events to trap something important.
>
> I don't see why this has to progress into a mess.

Because the IDS will be insisting on widening the rules to get events and the 
admin will be trying to cut back disk usage. A program can't be smart about 
the rules it needs unless you let the admin better express what they want 
escalated. Perhaps the admin only cares about users >= 500 accessing a 
directory when they fail due to EPERM. You would have to add the ability to 
express that to the IDS configuration. 

If you continue down that path, the IDS system will have become auditctl 
except its working in a disjointed way. When audit rules are reloaded, there 
will be a lag between the rule being deleted and being able to get a new rule 
in that give you the events it needs. If not there is a huge amount of code 
that needs to be written to add rules in the audit.rules file.

The difference boils down to this. I can code up the detector this afternoon 
based on my proposal. Doing it in a more complicated way means taking a month 
or two to get it right.


> > By using the key field, its in plain sight and done with purpose. Not
> > enough events to trap something important, widen it in audit.rules and
> > you also know that this will send more to disk. No suprises there.
>
> I'm actually in favor of using the key, just in using it like its used
> today.  All the capp watches have unique keys, and an admin could create
> more/different rules with different keys.

Sure. This proposal doesn't affect CAPP at all.


> What if other plug-ins also want to use that field?

That would be fine by me. I invite that use as it means people are trying to 
do something useful with this subsystem.  :)

-Steve

Re: [RFC] programmatic IDS routing

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 19 March 2008 19:00:41 Linda Knippers wrote:
>> Steve Grubb wrote:
>>> On Wednesday 19 March 2008 17:41:07 Eric Paris wrote:
>>>> So maybe all we need is for the ids config file needs to be of the form
>>>>
>>>> key type priority
>>> And hostname. Remember that this could be run from an aggregator.
>>>
>>>> so I can set up my audit rule however I want say
>>>>
>>>> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k
>>>> 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F
>>>> exit=-EPERM -k webadminEPERM
>>>>
>>>> And my ids config file would look like:
>>>>
>>>> 500EPERM        file    med
>>>> webadminEPERM   exec    high
>>> This is pretty close to the idea that I started with. Then I thought, how
>>> do I make this engine run faster? How do I reduce memory consumption
>>> (since the keys have to be stored in memory)?
>> How does overloading the key field help either of those?
> 
> I wouldn't call this overloading the key field. The key field's purpose is to 
> allow searches for groups of audit events. You've traditionally used this 
> from the command prompt with ausearch. Now with the auparse library, it can 
> be used programmatically for searching by more utilities.
> 
> The way that it helps is that I can tell yes/no in 4 byte compares and without 
> storing any linked lists, hashes, or binary trees.

This optimization for a user-space component doesn't seem worth the hack.
You are overloading the field.  The manpage says its an arbitrary string of
text.

>>> How do I make sure that the keys are there and correct?
>> That can be a startup check.
> 
> How do you do that remotely?

Same way you would check that audit is running remotely.  You can either
check the state of the target system or not, and if you can, you can look
for rules just as well as any other state - like whether the dispatcher
is configured.  If you can't verify that remotely, then maybe you should
run your IDS locally.

> 
>>>> Is the reverse mapping from key to ids action really so expensive that
>>>> this is unreasonable?
>>> Consider a datacenter with many hosts that may want to run this off of
>>> the aggregator. There will be a high rate of incoming events and a bit of
>>> comparing to figure out if each event something we care about.
>> I think if someone is using audit as an IDS, they're going to care about
>> everything they're auditing or why are they auditing it?
> 
> That's not true. When it comes to IDS, the real question is how to effectively 
> escalate events of interest without overloading the decision makers. 
> Something that alerts once a second is of no use to anyone. You want it to 
> pick out the things you really care about for realtime and save everything 
> else for a daily report.

I think what Eric proposed will still let you do that.
> 
> 
>>> With my proposal, I can tell with strncmp(key, "ids-", 4) if this is
>>> anything we need to pay attention to. So, inspection of 4 bytes let me
>>> decide yes/no.
>> Who is doing this, the auditd, the dispatcher or the plugin?
> 
> The IDS plugin.
> 
> 
>> Couldn't you hash the key?
> 
> Yes. Takes more time than strncmp and then you may still have collisions where 
> you walk a list of some kind. This also has the side effect of needing to set 
> aside a good chunk of memory for the hashes - used or not.

But this is user space, right?

> 
>>> Its a finite amount of time and doesn't linearly slow down the system as
>>> more hosts and files of interest are configured. It scales well.
>>>
>>>> I tend to also agree with the part of the discussion which says that it
>>>> isn't audit's place to decide that some rules are meant for disk and
>>>> some rules aren't.
>>> I agree and never proposed that.
>> I guess Eric and I were both confused then by your comment about the
>> admin ending up with more audit events on disk than intended.
> 
> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F path=/etc/password
> 
> and
> 
> -w /etc/password
> 
> collect entirely different amounts of data.

So the admin would create the rule he wants, create a unique key and tell
the IDS what the key is and how it should react when it sees a record with
that key.

-- ljk
> 
> -Steve

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Thursday 20 March 2008 09:32:29 Linda Knippers wrote:
> Steve Grubb wrote:
> > I wouldn't call this overloading the key field. The key field's purpose
> > is to allow searches for groups of audit events. You've traditionally
> > used this from the command prompt with ausearch. Now with the auparse
> > library, it can be used programmatically for searching by more utilities.
> >
> > The way that it helps is that I can tell yes/no in 4 byte compares and
> > without storing any linked lists, hashes, or binary trees.
>
> This optimization for a user-space component doesn't seem worth the hack.
> You are overloading the field

How is this overloading the field? What is your definition of the field? What 
are its legitimate uses? Is the admin not allowed to put anything they want 
in it, include text they'd like picked out in realtime?

> > How do you do that remotely?
>
> Same way you would check that audit is running remotely.  You can either
> check the state of the target system or not, 

The answer is not. So, given that you can't probe the other end, you have no 
way to check the rules.


> >> I guess Eric and I were both confused then by your comment about the
> >> admin ending up with more audit events on disk than intended.
> >
> > -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F
> > path=/etc/password
> >
> > and
> >
> > -w /etc/password
> >
> > collect entirely different amounts of data.
>
> So the admin would create the rule he wants, create a unique key and tell
> the IDS what the key is and how it should react when it sees a record with
> that key.

Yes. What I want to do is also say that if you create your unique key a 
certain way, you get this additional benefit of realtime alerting or 
correlation. if you choose not to, then its treated as any other event.

-Steve

Re: [RFC] programmatic IDS routing

[Klaus Heinrich Kiwi]

> > So the admin would create the rule he wants, create a unique key and tell
> > the IDS what the key is and how it should react when it sees a record with
> > that key.
> 
> Yes. What I want to do is also say that if you create your unique key a 
> certain way, you get this additional benefit of realtime alerting or 
> correlation. if you choose not to, then its treated as any other event

I agree with Steve's compromise with speed, but maybe there's no way out
of using hashes or linked lists in the general case.

What if a message is important not only for the IDS plugin but also to a
fictional 'real-time compliance reporting' plugin - both wanting to use
the key field to carry special things, and in the same event:
type=USER_ACCT msg=... key=ids-file-high,sox-fault-med,actual_key

The plug-ins would need to check if their specific identifier is present
not only in the first 4 bytes, but after every comma.

If it's desirable to support the general case, instead of putting
everything in one single 'key' field, maybe having an index just like
execve arguments:
type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med
key[2]=actual_key

Still need to iterate through all keys in the worst case, but the
plugins could individually chose between having the rules hardcoded (for
speed) or configurable.

 -Klaus
-- 
Klaus Heinrich Kiwi
Security Development - IBM Linux Technology Center

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Friday 21 March 2008 06:28:42 Klaus Heinrich Kiwi wrote:
> > > So the admin would create the rule he wants, create a unique key and
> > > tell the IDS what the key is and how it should react when it sees a
> > > record with that key.
> >
> > Yes. What I want to do is also say that if you create your unique key a
> > certain way, you get this additional benefit of realtime alerting or
> > correlation. if you choose not to, then its treated as any other event
>
> I agree with Steve's compromise with speed, but maybe there's no way out
> of using hashes or linked lists in the general case.
>
> What if a message is important not only for the IDS plugin but also to a
> fictional 'real-time compliance reporting' plugin - both wanting to use
> the key field to carry special things, and in the same event:
> type=USER_ACCT msg=... key=ids-file-high,sox-fault-med,actual_key

Thanks Klaus, this is the conversation I really wanted to have.  :)

I think this is a valid concern and we should work out how to standardize this 
usage. I wouldn't be surprised if people are already doing this in their own 
deployments with tools they wrote.


> The plug-ins would need to check if their specific identifier is present
> not only in the first 4 bytes, but after every comma.

Sure, strchr is cheap, or strtok if you prefer. But I agree with this 
observation.


> If it's desirable to support the general case, instead of putting
> everything in one single 'key' field, maybe having an index just like
> execve arguments:
> type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med
> key[2]=actual_key

That's a thought. In a way, I'd rather stay in one key field so that we have a 
path forward right now. It won't complicate any existing kernel code. 
Besides, doing any kernel change means waiting about 3-4 months for 2.6.26 to 
be released. If we work out a standard that is backwards compatible, it 
should work all the way back to about 2.6.16.

I don't think every audit rule will get this treatment. In IDS, there are very 
few files that you'd want to have alerting on by default. So, maybe the best 
thing to do is bump up the key field size from 32 to 48 bytes to allow more 
room for this kind of thing? Or would 64 bytes be better? We should be able 
to make this change without too much worry. The kernel should truncate the 
key if we have new tools on old kernel and if old tools on new kernel, 
auditctl should complain if its too big. Either way, it should be apparent.

To improve support for this kind of usage, I was thinking that we could update 
auditctl so that it could selectively delete all by key and that it could 
list all rules by key. IOW,

auditctl -D -k ids-
auditctl -l -k ids-

Any other improvements/suggestions to aid support of this paradigm?


> Still need to iterate through all keys in the worst case, but the
> plugins could individually chose between having the rules hardcoded (for
> speed) or configurable.

Sure.

-Steve

Re: [RFC] programmatic IDS routing

[LC Bruzenak]

On Fri, 2008-03-21 at 08:50 -0400, Steve Grubb wrote:
> On Friday 21 March 2008 06:28:42 Klaus Heinrich Kiwi wrote:
...
> 
> 
> > If it's desirable to support the general case, instead of putting
> > everything in one single 'key' field, maybe having an index just like
> > execve arguments:
> > type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med
> > key[2]=actual_key
> 
> That's a thought. In a way, I'd rather stay in one key field so that we have a 
> path forward right now. It won't complicate any existing kernel code. 
> Besides, doing any kernel change means waiting about 3-4 months for 2.6.26 to 
> be released. If we work out a standard that is backwards compatible, it 
> should work all the way back to about 2.6.16.

I prefer this approach - NOT putting more than 1 key item in the key
field. I see order issues. I see myself repeating this phrase many
times, "No - the IDS part is first, then the comma, then the optional
extra part."
And given the data already proposed for the key in the IDS instance, I'd
say that I cannot think of what else we'd need in the key anyway.

> 
> I don't think every audit rule will get this treatment. In IDS, there are very 
> few files that you'd want to have alerting on by default. So, maybe the best 
> thing to do is bump up the key field size from 32 to 48 bytes to allow more 
> room for this kind of thing? Or would 64 bytes be better? We should be able 
> to make this change without too much worry. The kernel should truncate the 
> key if we have new tools on old kernel and if old tools on new kernel, 
> auditctl should complain if its too big. Either way, it should be apparent.
> 

Rather than make it a bigger monolithic size I'd prefer more keys
somehow. My first thought was to overload the key field based on the
event. For IDS events one would specify "-K" (for example) and the IDS
triple Steve proposed as appropriate in the 31-byte text area. For
another plugin need, choose a different constant - "-I" - or whatever.

This would make it not possible to have multiple plugin keys per event,
unless the overall key size were large enough to accommodate multiples.
But the important part to me is that the auditctl take care of any
ordering issues, rather than faulty people.

If many key-needing plugins are proposed, that to me suggests a dynamic
key length, which I doubt is preferable.

> To improve support for this kind of usage, I was thinking that we could update 
> auditctl so that it could selectively delete all by key and that it could 
> list all rules by key. IOW,
> 
> auditctl -D -k ids-
> auditctl -l -k ids-
> 
This sounds like a good idea.

> Any other improvements/suggestions to aid support of this paradigm?
> 
> 
> > Still need to iterate through all keys in the worst case, but the
> > plugins could individually chose between having the rules hardcoded (for
> > speed) or configurable.

I'd vote hardcoded not for speed because IIUC I believe it would
simplify administration.

Thanks,
LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: [RFC] programmatic IDS routing

[Steve Grubb]

On Friday 21 March 2008 10:14:04 LC Bruzenak wrote:
> On Fri, 2008-03-21 at 08:50 -0400, Steve Grubb wrote:
> > On Friday 21 March 2008 06:28:42 Klaus Heinrich Kiwi wrote:
> > > If it's desirable to support the general case, instead of putting
> > > everything in one single 'key' field, maybe having an index just like
> > > execve arguments:
> > > type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med
> > > key[2]=actual_key
> >
> > That's a thought. In a way, I'd rather stay in one key field so that we
> > have a path forward right now. It won't complicate any existing kernel
> > code. Besides, doing any kernel change means waiting about 3-4 months for
> > 2.6.26 to be released. If we work out a standard that is backwards
> > compatible, it should work all the way back to about 2.6.16.
>
> I prefer this approach - NOT putting more than 1 key item in the key
> field. I see order issues.

What if it were the duty of the plugins or even log analysis tools to be order 
independent? IOW, you can put things in any order you wish?


> My first thought was to overload the key field based on the
> event. For IDS events one would specify "-K" (for example) and the IDS
> triple Steve proposed as appropriate in the 31-byte text area. For
> another plugin need, choose a different constant - "-I" - or whatever.

I'd rather treat this like the -S option where it can be given multiple times 
if we go this route. Given the code in the kernel, having multiple key fields 
will require some significant patching.

If we felt like we wanted to keep this in one field, but thought 32 bytes was 
to restrictive we could easily extend the size.

struct audit_context {
...
  char *              filterkey;  /* key for rule that triggered record */

So, it is virtually unlimited in size since its allocated. The only 
restriction is a check in auditfilter.c.

 if (entry->rule.filterkey || f->val > AUDIT_MAX_KEY_LEN)
     goto exit_free;

So, it would be a simple matter of just changing the define from 32 to 48 or 
whatever.


> But the important part to me is that the auditctl take care of any
> ordering issues, rather than faulty people.

I could even fix auditctl to allow multiple -k fields, but glue them together 
with commas if that were helpful. I could event fix auditctl to split them 
back out for rule listing purposes. We could also fix auparse to be able to 
do the splitting in the key field too so that this paradigm is supported and 
enforced by the whole toolchain.

So, I could give the illusion of multiple key fields but without making any 
drastic kernel changes. Would this be acceptable?

-Steve

Re: [RFC] programmatic IDS routing

[LC Bruzenak]

On Fri, 2008-03-21 at 11:01 -0400, Steve Grubb wrote:
> On Friday 21 March 2008 10:14:04 LC Bruzenak wrote:
> > On Fri, 2008-03-21 at 08:50 -0400, Steve Grubb wrote:
> > > On Friday 21 March 2008 06:28:42 Klaus Heinrich Kiwi wrote:
> > > > If it's desirable to support the general case, instead of putting
> > > > everything in one single 'key' field, maybe having an index just like
> > > > execve arguments:
> > > > type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med
> > > > key[2]=actual_key
> > >
> > > That's a thought. In a way, I'd rather stay in one key field so that we
> > > have a path forward right now. It won't complicate any existing kernel
> > > code. Besides, doing any kernel change means waiting about 3-4 months for
> > > 2.6.26 to be released. If we work out a standard that is backwards
> > > compatible, it should work all the way back to about 2.6.16.
> >
> > I prefer this approach - NOT putting more than 1 key item in the key
> > field. I see order issues.
> 
> What if it were the duty of the plugins or even log analysis tools to be order 
> independent? IOW, you can put things in any order you wish?
> 

Works for me! As I said, I doubt that our system will add anything at
all to the key, since I believe there will be enough information already
in the key and the event. The less we (end-user administrators) need to
specify/enforce the better.

As far as the analysis tools, since the event is an IDS event, and
currently the prewikka tool is used to view, will there be enough text
viewable in that event to make sense of the severity & action to someone
not intimately familiar with the plugin?

If that answer is yes, I'm satisfied there as well.

> 
> > My first thought was to overload the key field based on the
> > event. For IDS events one would specify "-K" (for example) and the IDS
> > triple Steve proposed as appropriate in the 31-byte text area. For
> > another plugin need, choose a different constant - "-I" - or whatever.
> 
> I'd rather treat this like the -S option where it can be given multiple times 
> if we go this route. Given the code in the kernel, having multiple key fields 
> will require some significant patching.
> 
> If we felt like we wanted to keep this in one field, but thought 32 bytes was 
> to restrictive we could easily extend the size.
> 
> struct audit_context {
> ...
>   char *              filterkey;  /* key for rule that triggered record */
> 
> So, it is virtually unlimited in size since its allocated. The only 
> restriction is a check in auditfilter.c.
> 
>  if (entry->rule.filterkey || f->val > AUDIT_MAX_KEY_LEN)
>      goto exit_free;
> 
> So, it would be a simple matter of just changing the define from 32 to 48 or 
> whatever.

IMHO, the free-form key info is only useful when there is no plugin. I
cannot see using both a plugin key and our own key.

To me, if there is a plugin involved, it should provide all the key
information it needs relevant to the plugin, and so far I think it will.

When we use various keys now for watch events, the reason is because we
have no IDS plugin. The file watches, exec or mkexe events replace our
current ones, and are preferred since they are mainstream and part of
the IDS/audit package.
 
If you allow for some user space inside the plugin key, well, that is
generous and it may be used but honestly right now I cannot think of a
reason why.

> 
> 
> > But the important part to me is that the auditctl take care of any
> > ordering issues, rather than faulty people.
> 
> I could even fix auditctl to allow multiple -k fields, but glue them together 
> with commas if that were helpful. I could event fix auditctl to split them 
> back out for rule listing purposes. We could also fix auparse to be able to 
> do the splitting in the key field too so that this paradigm is supported and 
> enforced by the whole toolchain.
> 
> So, I could give the illusion of multiple key fields but without making any 
> drastic kernel changes. Would this be acceptable?

Yep, it would be acceptable and preferred.

In this case you would stretch the key size and warn/enforce size limits
right? Too many commas (or any 1-byte delimiter) start to matter with
~31-47 bytes. As it becomes crowded we start thinking about encoding the
entries, which is OK but opens up more issues. Again, if it is your (the
OS) part doing the work I'm happy! :)

LCB.

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: [RFC] programmatic IDS routing

[Klaus Heinrich Kiwi]

On Fri, 2008-03-21 at 11:01 -0400, Steve Grubb wrote:
> > My first thought was to overload the key field based on the
> > event. For IDS events one would specify "-K" (for example) and the IDS
> > triple Steve proposed as appropriate in the 31-byte text area. For
> > another plugin need, choose a different constant - "-I" - or whatever.
> 
> I'd rather treat this like the -S option where it can be given multiple times 
> if we go this route. Given the code in the kernel, having multiple key fields 
> will require some significant patching.
> 

I like the idea of having a stackable key field with tools and libraries
hiding the complexity of overloading the field, without deep changes to
the kernel.


> > But the important part to me is that the auditctl take care of any
> > ordering issues, rather than faulty people.
> 
> I could even fix auditctl to allow multiple -k fields, but glue them together 
> with commas if that were helpful. I could event fix auditctl to split them 
> back out for rule listing purposes. We could also fix auparse to be able to 
> do the splitting in the key field too so that this paradigm is supported and 
> enforced by the whole toolchain.
> 
> So, I could give the illusion of multiple key fields but without making any 
> drastic kernel changes. Would this be acceptable?

Yes, I assume it would. Maybe specialized interfaces (besides the legacy
ones) to add, remove and iterate through the keys would be desirable,
both to libauparse and auditctl.


 -Klaus


-- 
Klaus Heinrich Kiwi
Security Development - IBM Linux Technology Center