Re: missing avc message field names

Re: missing avc message field names

[Eamon Walsh]

Steve G wrote:
>> If you have to include code for parsing the current format, why the rush 
>> to change the kernel output?
>>     
>
> I was thinking that it should be done in near future so its not forgotten. But
> that is the only reason. It could be delayed for a while.
>
> But back to the original question, any preference for non-conflicting names? :)
>
>
>   
CC'ing linux-audit.

Some comments regarding userspace object managers and the userspace AVC: 
in general userspace object managers will introduce new fields to the 
AVC messages.  For example the AVC's generated by the X server have 
fields such as window=, property=, and extension= for X-specific things 
which do not appear in the kernel AVC's.  So it should be relatively 
easy to add new keywords to the dictionary, or even have the audit 
system gracefully accept keywords that are not in its dictionary.

If all of these keywords in the data dictionary have to be unique, I'm 
wondering if it might be useful to use a 3-tuple instead of a 
(name,value) pair.  The 3-tuple would consist of (namespace,name,value) 
with namespace coming from a defined list of subsystems.  So for example 
there would be an "SELinux" namespace encompassing all of the selinux 
keywords, so that the "result" and "perms" keywords from the previous 
example would not conflict with the "other" ones which would presumably 
be in a different namespace.  Or just prefix the names with "selinux-", 
"syscall-", etc.

Another request I have is that if this scheme moves forward, that we do 
away with the audit_log_user_avc_message(3) family of functions and 
instead introduce an interface that takes an array of name/value pairs 
making up the audit message, or a single string which it could parse as 
name/value pairs.  This would be one-size-fits-all and would avoid the 
10+ parameter situation with the current functions.
-- 

Eamon Walsh <ewalsh@tycho.nsa.gov>
National Security Agency

Re: missing avc message field names

[Karl MacMillan]

Eamon Walsh wrote:
> Steve G wrote:
>>> If you have to include code for parsing the current format, why the 
>>> rush to change the kernel output?
>>>     
>>
>> I was thinking that it should be done in near future so its not 
>> forgotten. But
>> that is the only reason. It could be delayed for a while.
>>
>> But back to the original question, any preference for non-conflicting 
>> names? :)
>>
>>
>>   
> CC'ing linux-audit.
> 
> Some comments regarding userspace object managers and the userspace AVC: 
> in general userspace object managers will introduce new fields to the 
> AVC messages.  For example the AVC's generated by the X server have 
> fields such as window=, property=, and extension= for X-specific things 
> which do not appear in the kernel AVC's.  So it should be relatively 
> easy to add new keywords to the dictionary, or even have the audit 
> system gracefully accept keywords that are not in its dictionary.
> 

Good point - as long as there are userspace generated audit messages it 
will be hard to enforce uniqueness.

> If all of these keywords in the data dictionary have to be unique, I'm 
> wondering if it might be useful to use a 3-tuple instead of a 
> (name,value) pair.  The 3-tuple would consist of (namespace,name,value) 
> with namespace coming from a defined list of subsystems.  So for example 
> there would be an "SELinux" namespace encompassing all of the selinux 
> keywords, so that the "result" and "perms" keywords from the previous 
> example would not conflict with the "other" ones which would presumably 
> be in a different namespace.  Or just prefix the names with "selinux-", 
> "syscall-", etc.
> 

The prefixes seems simpler and match with the current audit messages 
more closely.

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo-+05T5uksL2qpZYMLLGbcSA@public.gmane.org with
the words "unsubscribe selinux" without quotes as the message.

Re: missing avc message field names

[Eamon Walsh]

Karl MacMillan wrote:
> Eamon Walsh wrote:
>   
>> Steve G wrote:
>>     
>>>> If you have to include code for parsing the current format, why the 
>>>> rush to change the kernel output?
>>>>     
>>>>         
>>> I was thinking that it should be done in near future so its not 
>>> forgotten. But
>>> that is the only reason. It could be delayed for a while.
>>>
>>> But back to the original question, any preference for non-conflicting 
>>> names? :)
>>>
>>>
>>>   
>>>       
>> CC'ing linux-audit.
>>
>> Some comments regarding userspace object managers and the userspace AVC: 
>> in general userspace object managers will introduce new fields to the 
>> AVC messages.  For example the AVC's generated by the X server have 
>> fields such as window=, property=, and extension= for X-specific things 
>> which do not appear in the kernel AVC's.  So it should be relatively 
>> easy to add new keywords to the dictionary, or even have the audit 
>> system gracefully accept keywords that are not in its dictionary.
>>
>>     
>
> Good point - as long as there are userspace generated audit messages it 
> will be hard to enforce uniqueness.
>
>   
>> If all of these keywords in the data dictionary have to be unique, I'm 
>> wondering if it might be useful to use a 3-tuple instead of a 
>> (name,value) pair.  The 3-tuple would consist of (namespace,name,value) 
>> with namespace coming from a defined list of subsystems.  So for example 
>> there would be an "SELinux" namespace encompassing all of the selinux 
>> keywords, so that the "result" and "perms" keywords from the previous 
>> example would not conflict with the "other" ones which would presumably 
>> be in a different namespace.  Or just prefix the names with "selinux-", 
>> "syscall-", etc.
>>
>>     
>
> The prefixes seems simpler and match with the current audit messages 
> more closely.
>
> How about "prefix.keyword=value", selinux.result=denied, selinux.perms=foo,bar,baz.
>   


-- 
Eamon Walsh <ewalsh@tycho.nsa.gov>
National Security Agency

Re: missing avc message field names

[Steve Grubb]

On Monday 29 January 2007 14:22, Eamon Walsh wrote:
> Steve G wrote:
> > But back to the original question, any preference for non-conflicting
> > names? :)
>
> CC'ing linux-audit.
>
> Some comments regarding userspace object managers and the userspace AVC:
> in general userspace object managers will introduce new fields to the
> AVC messages.  

Which brings up the question of does everything have to be an AVC message 
type? If we ever want to have binary audit records, we need to think of these 
audit messages as if they were database tables - each type normalized. This 
is, I think, where we are ultimately heading.

> For example the AVC's generated by the X server have fields such as window=,
> property=, and extension= for X-specific things which do not appear in the
> kernel AVC's.  So it should be relatively easy to add new keywords to the
> dictionary, or even have the audit system gracefully accept keywords that
> are not in its dictionary.

The parsing API has a iterator approach and makes no assumptions about what is 
supposed to be there or not. So, it would accept new name/value pairs without 
any problem.

> If all of these keywords in the data dictionary have to be unique, I'm
> wondering if it might be useful to use a 3-tuple instead of a
> (name,value) pair. 

I just don't see audit growing that much bigger. Of course, I may be wrong.

> The 3-tuple would consist of (namespace,name,value) with namespace coming
> from a defined list of subsystems.  So for example there would be
> an "SELinux" namespace encompassing all of the selinux keywords, so that
> the "result" and "perms" keywords from the previous example would not
> conflict with the "other" ones which would presumably be in a different
> namespace.  Or just prefix the names with "selinux-", "syscall-", etc.

That would add a lot to disk space consumption in the current logging format. 
The real problem that we will have if there is a common name share among 
different records but with different meaning, is when people do a search. You 
should be able to say "I want all audit records with the perm field" and get 
records that are of the right type. Or another example, "I want records with 
perm == 0". The perm field in AVC's with get run through strtol which will 
yield 0 and then you have a match for an AVC when it should have been file 
related. We need unambiguous names to make searching work as expected.

> Another request I have is that if this scheme moves forward, that we do
> away with the audit_log_user_avc_message(3) family of functions and
> instead introduce an interface that takes an array of name/value pairs
> making up the audit message, or a single string which it could parse as
> name/value pairs.  This would be one-size-fits-all and would avoid the
> 10+ parameter situation with the current functions.

The one function you speak of was designed specifically for a python program. 
But otherwise the API is designed to enforce common format across a variety 
of applications. I do not see this family of functions growing much bigger at 
all. I think we have auditing in nearly all the trusted apps right now. But I 
also want to point out that if we ever want to have binary audit records, we 
would have to have a standard API that takes certain parameters and converts 
them to their binary equivalents. To allow a free style API would mean that 
it cannot be converted to binary format.

-Steve

Re: missing avc message field names

[Karl MacMillan]

Steve Grubb wrote:
> On Monday 29 January 2007 14:22, Eamon Walsh wrote:
>> Steve G wrote:
>>> But back to the original question, any preference for non-conflicting
>>> names? :)
>> CC'ing linux-audit.
>>
>> Some comments regarding userspace object managers and the userspace AVC:
>> in general userspace object managers will introduce new fields to the
>> AVC messages.  
> 
> Which brings up the question of does everything have to be an AVC message 
> type?

I don't know that it is necessary, but it is certainly preferable. All 
of these messages serve exactly the same purpose - splitting them would 
be a pain for any app that wants to process them (not to mention dealing 
with backwards compatibility).

For example, the policy generation tools that I am working on could 
process these X messages because they are AVC messages and I only 
require the core fields to be present (i.e., source / target types, 
granted / denied, class, and permissions). If they were split out I 
would a) have to process more message types and b) track which versions 
of the audit subsystem contain which message types.

  If we ever want to have binary audit records, we need to think of these
> audit messages as if they were database tables - each type normalized. This 
> is, I think, where we are ultimately heading.
> 

What problems do binary audit records solve? I think that binary records 
introduce significant problems and would need to have significant benefits.

The biggest issue, of course, is that it would prevent the use of any 
tools that process the files as text (grep, tail, awk, seaudit, 
setroubleshoot, etc., etc.). Some can be moved over to the audit 
library, but it is still useful to use grep, tail, and other shell tools 
to search logs. I know that you discourage that practice, but I believe 
that it is (and will remain) useful.

A related problem is that rigidly enforcing the formatting forces 
updates to the audit system for any slight change to the output. 
Tracking those changes is going to be incredibly difficult for tools 
that process audit data. The audit library will help, but won't 
alleviate all problems (like the introduction of new record types). 
Regular breakage of tools is just going to discourage adoption of the 
audit system.

>> For example the AVC's generated by the X server have fields such as window=,
>> property=, and extension= for X-specific things which do not appear in the
>> kernel AVC's.  So it should be relatively easy to add new keywords to the
>> dictionary, or even have the audit system gracefully accept keywords that
>> are not in its dictionary.
> 
> The parsing API has a iterator approach and makes no assumptions about what is 
> supposed to be there or not. So, it would accept new name/value pairs without 
> any problem.
> 

Great.

>> If all of these keywords in the data dictionary have to be unique, I'm
>> wondering if it might be useful to use a 3-tuple instead of a
>> (name,value) pair. 
> 
> I just don't see audit growing that much bigger. Of course, I may be wrong.
> 

Why? I hope there will be a growing list of selinux object managers. And 
what about third-party tools that are security critical?

>> The 3-tuple would consist of (namespace,name,value) with namespace coming
>> from a defined list of subsystems.  So for example there would be
>> an "SELinux" namespace encompassing all of the selinux keywords, so that
>> the "result" and "perms" keywords from the previous example would not
>> conflict with the "other" ones which would presumably be in a different
>> namespace.  Or just prefix the names with "selinux-", "syscall-", etc.
> 
> That would add a lot to disk space consumption in the current logging format. 
> The real problem that we will have if there is a common name share among 
> different records but with different meaning, is when people do a search. You 
> should be able to say "I want all audit records with the perm field" and get 
> records that are of the right type. Or another example, "I want records with 
> perm == 0". The perm field in AVC's with get run through strtol which will 
> yield 0 and then you have a match for an AVC when it should have been file 
> related. We need unambiguous names to make searching work as expected.
> 

I don't understand - Eamon is suggesting creating unique names across 
all message types by introducing a prefix, which seems like what you are 
saying is necessary. And prefixes don't have to take much room (e.g., 
sel instead of selinux-). The advantage to me is that it would be 
natural when there are conflicts (e.g., the sel prefix on selperms would 
let me know it would work for the selinux related message types).

Karl

Re: missing avc message field names

[Steve Grubb]

On Monday 29 January 2007 16:16, Karl MacMillan wrote:
> Steve Grubb wrote:
> > On Monday 29 January 2007 14:22, Eamon Walsh wrote:
> >> Steve G wrote:
> >>> But back to the original question, any preference for non-conflicting
> >>> names? :)
> >>
> >> CC'ing linux-audit.
> >>
> >> Some comments regarding userspace object managers and the userspace AVC:
> >> in general userspace object managers will introduce new fields to the
> >> AVC messages.
> >
> > Which brings up the question of does everything have to be an AVC message
> > type?
>
> I don't know that it is necessary, but it is certainly preferable. 

I'd also like to make sure that all AVCs really are AVCs. I think there are 
some error messages that come out as AVCs when they should be SE_ERR or 
USER_SE_ERR.

> All of these messages serve exactly the same purpose - splitting them would
> be a pain for any app that wants to process them (not to mention dealing
> with backwards compatibility).

You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.

> For example, the policy generation tools that I am working on could
> process these X messages because they are AVC messages and I only
> require the core fields to be present (i.e., source / target types,
> granted / denied, class, and permissions). If they were split out I
> would a) have to process more message types and b) track which versions
> of the audit subsystem contain which message types.

You already have them split out...or did you totally skip AUDIT_USER_AVC? So, 
you already have to solve the problem so why not generalize the solution? The 
parsing API should be able to help with this. If you tell it to get all the 
different types and they don't exist, you don't get a hit except the ones 
that do exist.

>   If we ever want to have binary audit records, we need to think of these
>
> > audit messages as if they were database tables - each type normalized.
> > This is, I think, where we are ultimately heading.
>
> What problems do binary audit records solve?

Speed and compression. People doing serious auditing have very large files. 
Compression is extremely important. Also when dealing with large files, any 
performance trick you can find helps immensely.

> The biggest issue, of course, is that it would prevent the use of any
> tools that process the files as text (grep, tail, awk, seaudit,
> setroubleshoot, etc., etc.). 

ausearch -m all --raw | grep anything you want

> Some can be moved over to the audit library, but it is still useful to use
> grep, tail, and other shell tools to search logs. I know that you discourage
> that practice, but I believe that it is (and will remain) useful.

What happens when the file format changes - zlib compression for example?

> >> If all of these keywords in the data dictionary have to be unique, I'm
> >> wondering if it might be useful to use a 3-tuple instead of a
> >> (name,value) pair.
> >
> > I just don't see audit growing that much bigger. Of course, I may be
> > wrong.
>
> Why? I hope there will be a growing list of selinux object managers

I doubt there very many security related pieces of information that is not 
already in the dictionary. There are a finite number of security objects and 
their attributes.

> And what about third-party tools that are security critical?

They have the TRUSTED_APP message type and they can put anything in that they 
want. I consider that one completely freestyle.

> I don't understand - Eamon is suggesting creating unique names across
> all message types by introducing a prefix, which seems like what you are
> saying is necessary.

Nearly all names are unique (and the problem should be solved when I code up 
seperm and seresults). Adding more letters to the name eats the diskspace. We 
only do it when necessary. That is what drives the choice of "res" instead 
of "results" for the audit system.

-Steve

Re: missing avc message field names

[Eamon Walsh]

>>>> Steve G wrote:
>>>>         
> I'd also like to make sure that all AVCs really are AVCs. I think there are 
> some error messages that come out as AVCs when they should be SE_ERR or 
> USER_SE_ERR.
>
>   
This is a separate issue, this results from the userspace AVC's logging 
callback not having a type field.  I'm aware of this problem and willing 
to fix it but I'd like to get a clear picture of what the final auditing 
format will be first, so I'm glad this discussion is taking place.
> You already have them split out...or did you totally skip AUDIT_USER_AVC? So, 
> you already have to solve the problem so why not generalize the solution? The 
> parsing API should be able to help with this. If you tell it to get all the 
> different types and they don't exist, you don't get a hit except the ones 
> that do exist.
>
>   
My point here was simply to warn in advance that userspace object 
managers will in general introduce widely varying fields to their AVC 
messages and that the auditing record format should be equipped to deal 
with that.  Who knows what types of objects will appear in these 
messages in the future: there might be SELinux-enhanced e-mail clients, 
office applications, file managers in the future.  Making the dictionary 
easily extensible now will likely pay off later and the prefix idea is a 
way to do that.

As far as binary records go, if that is the future direction then I 
don't see why the disk space argument is being made for not having a 
prefix.  Presumably in binary form the keywords will  be assigned 
numbers that are fixed size, so that the length of the string 
representation shouldn't matter.  And even in text form if the logs are 
compressed then the length issue will be mediated.
>>     
> I doubt there very many security related pieces of information that is not 
> already in the dictionary. There are a finite number of security objects and 
> their attributes.
>
>   
Again I think that the audit system should support future expansion if 
it's going to be the new way for handling AVC's and other security events.
>> And what about third-party tools that are security critical?
>>     
>
> They have the TRUSTED_APP message type and they can put anything in that they 
> want. I consider that one completely freestyle.
>
>   
How does this work with binary records?  How will these messages be 
searched?
> Nearly all names are unique (and the problem should be solved when I code up 
> seperm and seresults). Adding more letters to the name eats the diskspace. We 
> only do it when necessary. That is what drives the choice of "res" instead 
> of "results" for the audit system.
>   

I do think it should be selinux.perms, selinux.result and the other AVC 
stuff should be selinux.foo.  These are long names I concede but they 
are clearer and log size is already an issue anyway, moving to 
compressed or binary will solve the problem.
>   

-- 
Eamon Walsh <ewalsh@tycho.nsa.gov>
National Security Agency

Re: missing avc message field names

[Russell Coker]

On Tuesday 30 January 2007 10:48, Eamon Walsh <ewalsh-+05T5uksL2qpZYMLLGbcSA@public.gmane.org> wrote:
> there might be SELinux-enhanced e-mail clients,
> office applications, file managers in the future

Yes, we need all that.

There are some people interested in SE enhanced MUAs.

One issue is that SE-X is required for full functionality in this regard 
(let's assume for the sake of discussion that almost everyone who matters 
uses a GUI MUA).  Another issue is that the design of MUAs is tending towards 
greater integration with the desktop environment and larger more complex code 
bases.

I'm thinking of starting to attack this by developing a password sequestration 
system for MUAs.  The idea being that the MUA would run a SETGID program and 
request a POP connection, it would be returned a file handle for an 
authenticated connection but have no way of obtaining the password that was 
used.  This will offer significant security benefits in a non-SE environment 
and even better protection with SE Linux.  A compromised MUA would not be 
able to obtain a password list and send it to a hostile party (it would be 
able to proxy access to the POP server and to send copies of all stored 
messages).  Given the incidence of passwords being used for multiple 
functions this would significantly mitigate the risks of MUA based attacks.

The current situation is tending towards having an ever increasing amount of 
the practical system integrity dependant on the integrity of a single user 
account (in which all programs run with the same security context).

Getting upstream support for labelled email is going to be very difficult with 
the current client side security situation.

Now if we could just get web browsers to have their functionality split into 
multiple programs with different security contexts...

-- 
russell-YtRjSb8ePh30CCvOHzKKcA@public.gmane.org
http://etbe.blogspot.com/          My Blog

http://www.coker.com.au/sponsorship.html Sponsoring Free Software development

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo-+05T5uksL2qpZYMLLGbcSA@public.gmane.org with
the words "unsubscribe selinux" without quotes as the message.

Re: missing avc message field names

[Karl MacMillan]

Steve Grubb wrote:
> On Monday 29 January 2007 16:16, Karl MacMillan wrote:
>> Steve Grubb wrote:

<snip>

>> All of these messages serve exactly the same purpose - splitting them would
>> be a pain for any app that wants to process them (not to mention dealing
>> with backwards compatibility).
> 
> You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.
> 

Yes, but adding even more message types makes the problem worse. More 
importantly, what does splitting AVC messages even more accomplish? In 
fact, I would rather that AUDIT_AVC and AUDIT_USER_AVC were not split.

>> For example, the policy generation tools that I am working on could
>> process these X messages because they are AVC messages and I only
>> require the core fields to be present (i.e., source / target types,
>> granted / denied, class, and permissions). If they were split out I
>> would a) have to process more message types and b) track which versions
>> of the audit subsystem contain which message types.
> 
> You already have them split out...or did you totally skip AUDIT_USER_AVC? So, 
> you already have to solve the problem so why not generalize the solution? The 
> parsing API should be able to help with this. If you tell it to get all the 
> different types and they don't exist, you don't get a hit except the ones 
> that do exist.
> 

You keep talking about the future - which doesn't yet exist.

So today, if you add a message type I would have to somehow determine 
which audit version I was dealing with so that I didn't pass an invalid 
record type to ausearch (because it bails). Definitely problematic.

Using the audit library would be better if it ignores invalid record 
types, but it still requires tool updates.

>>   If we ever want to have binary audit records, we need to think of these
>>
>>> audit messages as if they were database tables - each type normalized.
>>> This is, I think, where we are ultimately heading.
>> What problems do binary audit records solve?
> 
> Speed and compression. People doing serious auditing have very large files. 
> Compression is extremely important. Also when dealing with large files, any 
> performance trick you can find helps immensely.
> 

gzip or bzip2 compression is not sufficient? Is audit searching 
performance the most important consideration?

Look, I'm not against binary logs, I'm just trying to point out that I 
think that the transition will not be as easy as you seem to suggest.

>> The biggest issue, of course, is that it would prevent the use of any
>> tools that process the files as text (grep, tail, awk, seaudit,
>> setroubleshoot, etc., etc.). 
> 
> ausearch -m all --raw | grep anything you want
> 

tail -f happens to be my favorite counter example, but I am certain 
there are other useful tricks for monitoring logs that will break. Not 
to mention the number of log monitoring and aggregation tools that 
assume text logs.

>> Some can be moved over to the audit library, but it is still useful to use
>> grep, tail, and other shell tools to search logs. I know that you discourage
>> that practice, but I believe that it is (and will remain) useful.
> 
> What happens when the file format changes - zlib compression for example?
> 

zcat :) Seriously, what you seem to be suggesting is that the presence 
of the audit library will make it possible to arbitrarily change the 
audit log format. I believe that the library is not sufficient - 
changing the format will continue to cause breakage and frustration. I'm 
concerned about that as someone that writes applications that are 
consumers of the audit system.

>>>> If all of these keywords in the data dictionary have to be unique, I'm
>>>> wondering if it might be useful to use a 3-tuple instead of a
>>>> (name,value) pair.
>>> I just don't see audit growing that much bigger. Of course, I may be
>>> wrong.
>> Why? I hope there will be a growing list of selinux object managers
> 
> I doubt there very many security related pieces of information that is not 
> already in the dictionary. There are a finite number of security objects and 
> their attributes.
> 

Almost by definition a userspace object manager adds new security 
critical objects. See Eamon's example.

>> And what about third-party tools that are security critical?
> 
> They have the TRUSTED_APP message type and they can put anything in that they 
> want. I consider that one completely freestyle.
> 

Why is it acceptable to have arbitrary name/value pairs in that message 
type and not in others?

Karl


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo-+05T5uksL2qpZYMLLGbcSA@public.gmane.org with
the words "unsubscribe selinux" without quotes as the message.

Re: missing avc message field names

[Joshua Brindle]

Karl MacMillan wrote:
>>> The biggest issue, of course, is that it would prevent the use of any
>>> tools that process the files as text (grep, tail, awk, seaudit,
>>> setroubleshoot, etc., etc.). 
>>
>> ausearch -m all --raw | grep anything you want
>>
>
> tail -f happens to be my favorite counter example, but I am certain 
> there are other useful tricks for monitoring logs that will break. Not 
> to mention the number of log monitoring and aggregation tools that 
> assume text logs.

This is fairly off topic here (selinux list) but I agree with Karl. As a 
recovering admin I think I can say that admins expect to be able to use 
various unix utilities to inspect log files, particularly tail -f. While 
I'm all for applications putting their data in private data formats and 
using tools and libraries to inspect them I think it is generally 
considered that everything in /var/log is fair game to inspect with 
anything available on systems (including perl, python, sed, awk, tail, 
grep, etc).

You will certainly be rubbing most admins the wrong way by forcing them 
through a different interface that won't support some common commands 
like tail -f.

There are probably hundreds of utilities that look through these files 
as well, what is going to happen when people try to add audit.log to a 
log watcher that emails logs to them? Huge binary dumps in email are 
going to make people turn off the audit daemon, not modify their apps to 
use different tools/libraries.

Re: missing avc message field names

[Valdis.Kletnieks]

[-- Attachment #1.1: Type: text/plain, Size: 748 bytes --]

On Tue, 30 Jan 2007 12:06:06 EST, Joshua Brindle said:
>
> This is fairly off topic here (selinux list) but I agree with Karl. As a 
> recovering admin I think I can say that admins expect to be able to use 
> various unix utilities to inspect log files, particularly tail -f.

As a counter-example - lastcomm and last.

If you want to use tail -f, don't run auditd, and use syslog-ng(*) or similar
to send the msgs you're interested in to a file that you can tail -f.

Or you *can* tail -f the file, just be ready to deal with the fact that it
contains binary data, same as the process accounting file and the last-login
file.

(*) syslog-ng can route to logfiles based on a regexp, so you don't have to
send all kernel output to the same file...

[-- Attachment #1.2: Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: missing avc message field names

[Casey Schaufler]

--- Joshua Brindle <jbrindle@tresys.com> wrote:


> This is fairly off topic here (selinux list) but I
> agree with Karl. As a 
> recovering admin I think I can say that admins
> expect to be able to use 
> various unix utilities to inspect log files,
> particularly tail -f. While 
> I'm all for applications putting their data in
> private data formats and 
> using tools and libraries to inspect them I think it
> is generally 
> considered that everything in /var/log is fair game
> to inspect with 
> anything available on systems (including perl,
> python, sed, awk, tail, 
> grep, etc).
> 
> You will certainly be rubbing most admins the wrong
> way by forcing them 
> through a different interface that won't support
> some common commands 
> like tail -f.
> 
> There are probably hundreds of utilities that look
> through these files 
> as well, what is going to happen when people try to
> add audit.log to a 
> log watcher that emails logs to them? Huge binary
> dumps in email are 
> going to make people turn off the audit daemon, not
> modify their apps to 
> use different tools/libraries.

Based on the Unix experience I find myself
agreeing with this assessment. Binary (or
compressed) audit logs don't get read very
often. A mechanism like audit_filters(5) from
Irix makes the problem more manageable, but
the truth is that humans like their information
human readable. Disk space used to be a major
problem, and I/O bandwidth still is (you can
overwhelm any system with too much audit no
matter how optimal your audit data) but the
cost of translation-on-read is going to stop
most humans from ever doing it.


Casey Schaufler
casey@schaufler-ca.com

Re: missing avc message field names

[Steve Grubb]

On Tuesday 30 January 2007 09:49, Karl MacMillan wrote:
> > You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.
>
> Yes, but adding even more message types makes the problem worse. More
> importantly, what does splitting AVC messages even more accomplish?

It allows users to tune the audit system to the kinds of messages they want to 
keep in their logs, makes analysis of the kinds of problems being reported 
easier, and allows normalization of records for binary use.

> In fact, I would rather that AUDIT_AVC and AUDIT_USER_AVC were not split.

They have to be, ironically, because of SE Linux.  :)

> > Speed and compression. People doing serious auditing have very large
> > files. Compression is extremely important. Also when dealing with large
> > files, any performance trick you can find helps immensely.
>
> gzip or bzip2 compression is not sufficient?

No, its an intermediate solution on the way to a better more difficult 
solution.

> Is audit searching performance the most important consideration?

Storage size is the more important followed by speed. As soon as you allow 
more records, they will have more to search.

> Look, I'm not against binary logs, I'm just trying to point out that I
> think that the transition will not be as easy as you seem to suggest.

No, I'm sure it will be time consuming and done in phases. The first step is 
providing the insulating layer so that apps do not have to care.

> >> The biggest issue, of course, is that it would prevent the use of any
> >> tools that process the files as text (grep, tail, awk, seaudit,
> >> setroubleshoot, etc., etc.).
> >
> > ausearch -m all --raw | grep anything you want
>
> tail -f happens to be my favorite counter example, but I am certain
> there are other useful tricks for monitoring logs that will break.

You can still use this on your system. What will be happening in the audit 
system is that there will be choices as to how the information will be 
stored. Text will still be an option. However, the portable way will be 
ausearch based.

> >> And what about third-party tools that are security critical?
> >
> > They have the TRUSTED_APP message type and they can put anything in that
> > they want. I consider that one completely freestyle.
>
> Why is it acceptable to have arbitrary name/value pairs in that message
> type and not in others?

Because by definition third party means we don't have the source to review or 
fix.

-Steve

Re: missing avc message field names

[James Antill]

On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote: 
> Steve Grubb wrote:
> > ausearch -m all --raw | grep anything you want
> 
> tail -f happens to be my favorite counter example, but I am certain 
> there are other useful tricks for monitoring logs that will break. Not 
> to mention the number of log monitoring and aggregation tools that 
> assume text logs.

 To be fair the new audit dispatcher already has a plugin that does the
same thing as "tail -f" without needing to call stat(), and that'll be
released before auditd has binary logs ... although one could certainly
argue that it's not as obvious, it seems like a small price.

-- 
James Antill <jantill@redhat.com>

Re: missing avc message field names

[Karl MacMillan]

James Antill wrote:
> On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote: 
>> Steve Grubb wrote:
>>> ausearch -m all --raw | grep anything you want
>> tail -f happens to be my favorite counter example, but I am certain 
>> there are other useful tricks for monitoring logs that will break. Not 
>> to mention the number of log monitoring and aggregation tools that 
>> assume text logs.
> 
>  To be fair the new audit dispatcher already has a plugin that does the
> same thing as "tail -f" without needing to call stat(), and that'll be
> released before auditd has binary logs ... although one could certainly
> argue that it's not as obvious, it seems like a small price.
> 

So you will have that wheel reinvented soon - that still leaves many, 
many more that you have no control over.

Karl

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo-+05T5uksL2qpZYMLLGbcSA@public.gmane.org with
the words "unsubscribe selinux" without quotes as the message.

Re: missing avc message field names

[Joshua Brindle]

Karl MacMillan wrote:
> James Antill wrote:
>> On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote:
>>> Steve Grubb wrote:
>>>> ausearch -m all --raw | grep anything you want
>>> tail -f happens to be my favorite counter example, but I am certain 
>>> there are other useful tricks for monitoring logs that will break. 
>>> Not to mention the number of log monitoring and aggregation tools 
>>> that assume text logs.
>>
>>  To be fair the new audit dispatcher already has a plugin that does the
>> same thing as "tail -f" without needing to call stat(), and that'll be
>> released before auditd has binary logs ... although one could certainly
>> argue that it's not as obvious, it seems like a small price.
>>
>
> So you will have that wheel reinvented soon - that still leaves many, 
> many more that you have no control over.

Even with a tail replacement there has to be thousands of internally 
written and maintained log monitoring and reporting apps that will 
break, this is a fundamental change in how logging works on linux, not 
something that can or should be changed on a whim (or otherwise).

Re: missing avc message field names

[Russell Coker]

On Wednesday 31 January 2007 16:29, Joshua Brindle <jbrindle-5TQdPaFcblfQT0dZR+AlfA@public.gmane.org> 
wrote:
> Even with a tail replacement there has to be thousands of internally
> written and maintained log monitoring and reporting apps that will
> break, this is a fundamental change in how logging works on linux, not
> something that can or should be changed on a whim (or otherwise).

Most such programs assume that log files keep the same name until a cron job 
renames them.  The current practice of auditd rotating it's log files has 
probably broken the majority of such programs already.

Also Steve Grubb suggested having a configuration option for plain-text files 
which will avoid the problems with binary files.

If we work with the assumption that indexed log files are required for sites 
with significant audit requirements due to the volume of logs and the need to 
get responses in a reasonable amount of time then we have two options.  One 
is a binary format, the other is to have index files along-side the text 
files.

Having separate index files introduces complications for renaming and other 
file management (complexity is bad for reliability), even without the issue 
of the sys-admin wanting to rename their own log files.

So it seems that the option of a binary log file is required.

Maybe there should be an option to have auditd write a binary log file as well 
as either a text log file or logging via syslog?  That way the admin could 
have the index benefits of a binary log as well as having text files.  If 
there were two log files then the second copy wouldn't need to be written 
synchronously so the IO load would not double.

-- 
russell-YtRjSb8ePh30CCvOHzKKcA@public.gmane.org
http://etbe.blogspot.com/          My Blog

http://www.coker.com.au/sponsorship.html Sponsoring Free Software development


--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo-+05T5uksL2qpZYMLLGbcSA@public.gmane.org with
the words "unsubscribe selinux" without quotes as the message.

Re: missing avc message field names

[Steve Grubb]

On Wednesday 31 January 2007 17:59, Russell Coker wrote:
> Maybe there should be an option to have auditd write a binary log file as
> well as either a text log file or logging via syslog?

This should  be possible. The audit event dispatcher typically has had a 
plugin that relays audit events to syslog. It will likely be a little while 
before there are binary formatted logs. I guess my message is really that you 
might not want to assume that the site will have text based logs for user 
support. Text logs are not being deprecated. Its that there will be more 
options soon.

-Steve