ABI guarantee for auditd

ABI guarantee for auditd

[hsultan]

Hi,

Sorry for the deluge of questions :)

Regarding auditd, what is the ABI guarantee ? Do you guarantee that the 
text contained in audit_reply->msg.data will always be the same format ? 
I imagine you reserve the right to add fields, but how about removing 
any or even reordering them ?

Or are people simply required to use auparse to guarantee they get 
records properly ?

Also, regarding 'unofficial' ABI compatibility, when has the 
audit_reply->msg.data format changed last ? Say these past 3-4 years, 
were there any changes in the format or could I use a faster, but 
specifically focused parser on the msgs when detecting older releases at 
least ?

Thanks,

Hassan

Re: ABI guarantee for auditd

[Steve Grubb]

On Thursday, January 15, 2015 12:24:38 PM hsultan@thefroid.net wrote:
> Regarding auditd, what is the ABI guarantee ? Do you guarantee that the
> text contained in audit_reply->msg.data will always be the same format ?
> I imagine you reserve the right to add fields, but how about removing
> any or even reordering them ?

Its happens on occasion. Requirements change, bugs are found, new features 
asked for.

> Or are people simply required to use auparse to guarantee they get
> records properly ?

Nobody is _required_ to do anything. :-)  But, if there are changes, auparse 
will definitely be updated because its used for a lot of purposes. I haven't 
found a problem yet that it couldn't handle. There are also plans to give it 
more capabilities later in the spring.

The intention of the auparse library is that anyone wanting to write an 
analytical application can use it to get something working without having to 
become an audit expert. You don't have to worry about where to lookup 
information to translate the fields from numbers to human readable form.


> Also, regarding 'unofficial' ABI compatibility, when has the
> audit_reply->msg.data format changed last ? Say these past 3-4 years,
> were there any changes in the format or could I use a faster, but
> specifically focused parser on the msgs when detecting older releases at
> least ?

The format of some events does change on occasion. Usually its after a problem 
is identified.

-Steve

Re: ABI guarantee for auditd

[hsultan]

On 2015-01-15 12:44, Steve Grubb wrote:
> On Thursday, January 15, 2015 12:24:38 PM hsultan@thefroid.net wrote:
>> Regarding auditd, what is the ABI guarantee ? Do you guarantee that 
>> the
>> text contained in audit_reply->msg.data will always be the same 
>> format ?
>> I imagine you reserve the right to add fields, but how about 
>> removing
>> any or even reordering them ?
>
> Its happens on occasion. Requirements change, bugs are found, new 
> features
> asked for.

Thanks, that tells me most of what I need, one last thing : do you 
happen to know if the 'big' distribs(Ubuntu,RH,CentOS,Debian...) ship 
those format changes only in new releases of their distribs, or do they 
include them in patches for existing releases as well ?

Thanks,

Hassan

Re: ABI guarantee for auditd

[Steve Grubb]

On Thursday, January 15, 2015 02:34:16 PM hsultan@thefroid.net wrote:
> On 2015-01-15 12:44, Steve Grubb wrote:
> > On Thursday, January 15, 2015 12:24:38 PM hsultan@thefroid.net wrote:
> >> Regarding auditd, what is the ABI guarantee ? Do you guarantee that
> >> the
> >> text contained in audit_reply->msg.data will always be the same
> >> format ?
> >> I imagine you reserve the right to add fields, but how about
> >> removing
> >> any or even reordering them ?
> > 
> > Its happens on occasion. Requirements change, bugs are found, new
> > features asked for.
> 
> Thanks, that tells me most of what I need, one last thing : do you
> happen to know if the 'big' distribs(Ubuntu,RH,CentOS,Debian...) ship
> those format changes only in new releases of their distribs, or do they
> include them in patches for existing releases as well ?

I can't speak for other distributions, but if I find a mistake in the audit 
records, I fix it to be right rather than hold ABI and stay forever wrong. This 
doesn't happen very often. The audit records are mostly stable. But there are 
155 different records.

-Steve

Re: ABI guarantee for auditd

[hsultan]

On 2015-01-15 14:59, Steve Grubb wrote:
> On Thursday, January 15, 2015 02:34:16 PM hsultan@thefroid.net wrote:
>> On 2015-01-15 12:44, Steve Grubb wrote:
>> > On Thursday, January 15, 2015 12:24:38 PM hsultan@thefroid.net 
>> wrote:
>> >> Regarding auditd, what is the ABI guarantee ? Do you guarantee 
>> that
>> >> the
>> >> text contained in audit_reply->msg.data will always be the same
>> >> format ?
>> >> I imagine you reserve the right to add fields, but how about
>> >> removing
>> >> any or even reordering them ?
>> >
>> > Its happens on occasion. Requirements change, bugs are found, new
>> > features asked for.
>>
>> Thanks, that tells me most of what I need, one last thing : do you
>> happen to know if the 'big' distribs(Ubuntu,RH,CentOS,Debian...) 
>> ship
>> those format changes only in new releases of their distribs, or do 
>> they
>> include them in patches for existing releases as well ?
>
> I can't speak for other distributions, but if I find a mistake in the 
> audit
> records, I fix it to be right rather than hold ABI and stay forever 
> wrong. This
> doesn't happen very often. The audit records are mostly stable. But 
> there are
> 155 different records.

Thanks for the info, so I tried using libauparse (again, Ubuntu 14.04 
LTS), however I'm hitting something truly weird: once I've adddd the 
event parsing code (taken from 
https://fedorahosted.org/audit/browser/trunk/contrib/plugin/audisp-example.c 
) and added -lauparse, what I get out of audit_get_reply now is 
mangled.
That clearly can't be a code mistake because I didn't touch the event 
retrieval code, it's totally separate and runs in a separate thread, 
dropping the messages retrieved in a queue that the parser picks from. 
Weird thing is, even if I comment out the parsing code, the problem 
remains. Then I even remove the libauparse lib from the link settings 
and rebuild from scratch, problem remains. BUT then I reboot the box, 
and the SAME PROCESS (no recompiling, just a reboot) now shows events 
properly again.

Is there a conflict or some specific setup between the 2 libraries I 
should know about ? Does libauparse configures the audit infrastructure 
in the kernel somehow ?

My libauparse version is 1:2.3.2-2ubuntu1 and from dpkg-query it lists 
:

Breaks: libaudit0, libaudit1 (<< 1:2.2.1-2)

My libaudit is :
Version: 1:2.3.2-2ubuntu1

Thanks,

Hassan

Re: ABI guarantee for auditd

[Hassan Sultan]

On Thu, 15 Jan 2015 18:20:41 -0800, <hsultan@thefroid.net> wrote:

> On 2015-01-15 14:59, Steve Grubb wrote:
>> On Thursday, January 15, 2015 02:34:16 PM hsultan@thefroid.net wrote:
>>> On 2015-01-15 12:44, Steve Grubb wrote:
>>> > On Thursday, January 15, 2015 12:24:38 PM hsultan@thefroid.net wrote:
>>> >> Regarding auditd, what is the ABI guarantee ? Do you guarantee that
>>> >> the
>>> >> text contained in audit_reply->msg.data will always be the same
>>> >> format ?
>>> >> I imagine you reserve the right to add fields, but how about
>>> >> removing
>>> >> any or even reordering them ?
>>> >
>>> > Its happens on occasion. Requirements change, bugs are found, new
>>> > features asked for.
>>>
>>> Thanks, that tells me most of what I need, one last thing : do you
>>> happen to know if the 'big' distribs(Ubuntu,RH,CentOS,Debian...) ship
>>> those format changes only in new releases of their distribs, or do they
>>> include them in patches for existing releases as well ?
>>
>> I can't speak for other distributions, but if I find a mistake in the  
>> audit
>> records, I fix it to be right rather than hold ABI and stay forever  
>> wrong. This
>> doesn't happen very often. The audit records are mostly stable. But  
>> there are
>> 155 different records.
>
> Thanks for the info, so I tried using libauparse (again, Ubuntu 14.04  
> LTS), however I'm hitting something truly weird: once I've adddd the  
> event parsing code (taken from  
> https://fedorahosted.org/audit/browser/trunk/contrib/plugin/audisp-example.c  
> ) and added -lauparse, what I get out of audit_get_reply now is mangled.
> That clearly can't be a code mistake because I didn't touch the event  
> retrieval code, it's totally separate and runs in a separate thread,  
> dropping the messages retrieved in a queue that the parser picks from.  
> Weird thing is, even if I comment out the parsing code, the problem  
> remains. Then I even remove the libauparse lib from the link settings  
> and rebuild from scratch, problem remains. BUT then I reboot the box,  
> and the SAME PROCESS (no recompiling, just a reboot) now shows events  
> properly again.
>
> Is there a conflict or some specific setup between the 2 libraries I  
> should know about ? Does libauparse configures the audit infrastructure  
> in the kernel somehow ?
>
> My libauparse version is 1:2.3.2-2ubuntu1 and from dpkg-query it lists :
>
> Breaks: libaudit0, libaudit1 (<< 1:2.2.1-2)
>
> My libaudit is :
> Version: 1:2.3.2-2ubuntu1

Ok ignore my message, I'm truly an idiot and should pay more attention  
before I send questions here.

Sorry,

Hassan

Re: ABI guarantee for auditd

[Steve Grubb]

On Thursday, January 15, 2015 06:20:41 PM hsultan@thefroid.net wrote:
> Thanks for the info, so I tried using libauparse (again, Ubuntu 14.04 
> LTS), however I'm hitting something truly weird: once I've adddd the 
> event parsing code (taken from 
> https://fedorahosted.org/audit/browser/trunk/contrib/plugin/audisp-example.c
>   ) and added -lauparse, what I get out of audit_get_reply now is
> mangled.

Why are you using that in an analytical program? That is a very low level 
function for getting events out of the kernel. You might want to have a look 
at this presentation to understand the audit architecture:

http://people.redhat.com/sgrubb/audit/audit_ids_2011.pdf

Auditd handles getting events from the kernel, passes them to audispd, you 
have a plugin to audispd and get the event in realtime. If you want events on 
disk, you just tell auparse_init that you want to use the logs as your source.

Libauparse handles events after they have been processed by auditd.


> That clearly can't be a code mistake because I didn't touch the event 
> retrieval code, 

It is a mistake. The example code works and demonstrates how to get events and 
iterate over the records and fields of the record. The presentation mentioned 
above also shows how to iterate over events, records, and fields. It also has a 
UML diagram to orient a developer to the data abstractions.


> Is there a conflict or some specific setup between the 2 libraries I 
> should know about ? 

No. Auparse needs to be linked against libaudit for syscall lookup functions 
and a couple other items.

> Does libauparse configures the audit infrastructure 
> in the kernel somehow ?

No. Its used for post-processing audit events. Its not meant for grabbing 
events out of the audit netlink socket. Its expects events that are properly 
formatted. 

-Steve

Re: ABI guarantee for auditd

[hsultan]

On 2015-01-16 05:48, Steve Grubb wrote:
> On Thursday, January 15, 2015 06:20:41 PM hsultan@thefroid.net wrote:
>> Thanks for the info, so I tried using libauparse (again, Ubuntu 
>> 14.04
>> LTS), however I'm hitting something truly weird: once I've adddd the
>> event parsing code (taken from
>> 
>> https://fedorahosted.org/audit/browser/trunk/contrib/plugin/audisp-example.c
>>   ) and added -lauparse, what I get out of audit_get_reply now is
>> mangled.
>
> Why are you using that in an analytical program? That is a very low 
> level
> function for getting events out of the kernel. You might want to have 
> a look
> at this presentation to understand the audit architecture:
>
> http://people.redhat.com/sgrubb/audit/audit_ids_2011.pdf
>
> Auditd handles getting events from the kernel, passes them to 
> audispd, you
> have a plugin to audispd and get the event in realtime. If you want 
> events on
> disk, you just tell auparse_init that you want to use the logs as 
> your source.
>
> Libauparse handles events after they have been processed by auditd.

I know. I sadly can't describe what I'm working on, however I have some 
stringent perf requirements. That's why I've been looking at doing 
custom parsing and that's why I'm bypassing the auditd daemon 
completely. I figured out how to recreate a msg that auparse likes from 
the output of audit_get_reply, and right now I'm planning on having both 
'modes' (fast using custom parsing/ slower but 'official' parsing) live 
in the binary, and simply have my process choose at start time after 
parsing some specifically generated audit msgs. If my custom parsing 
goes through properly, then I'll use my faster & custom parsing, 
otherwise I'll revert to the official but slower parsing (and patch 
appropriately to correct my custom parsing in the meantime).

Thanks,

Hassan