Troubleshooting Custom audispd Plugin

Troubleshooting Custom audispd Plugin

[Osama Elnaggar]

[-- Attachment #1.1: Type: text/plain, Size: 1733 bytes --]

Hi,

I'm working on a custom audispd plugin written in Python 3.  It’s a work in
progress and I’ve successfully run it numerous times as an audispd plugin.
However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog

Sep  6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep  6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...

This is repeated several times until audispd gives up and I see the
following message:

Sep  6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts

To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead.  I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).

Here is the code snippet (a colorized easier to read version is available
here - https://pastebin.com/84Nxu3Rp):

# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)

# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)

myfile = "/var/log/audit/audit.log"

while True:
    try:
        # we read in line by line from stdin
        for line in fileinput.input():
        #for line in fileinput.input(myfile):
            aup.feed(line)
    except:
        logger.error("Fatal error in while loop", exc_info=True)

# we flush the feed when we quit
aup.flush_feed()

Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog?  Thanks.

-- 
Osama Elnaggar

[-- Attachment #1.2: Type: text/html, Size: 4157 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Troubleshooting Custom audispd Plugin

[Steve Grubb]

On Friday, September 7, 2018 7:30:09 AM EDT Osama Elnaggar wrote:
> Hi,
> 
> I'm working on a custom audispd plugin written in Python 3.  It’s a work in
> progress and I’ve successfully run it numerous times as an audispd plugin.
> However, I sometimes make modifications that result in the audispd plugin
> failing and I end up with the following in /var/log/syslog
> 
> Sep  6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
> terminated unexpectedly
> Sep  6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
> restarted
> ...
> 
> This is repeated several times until audispd gives up and I see the
> following message:
> 
> Sep  6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
> exceeded max_restarts
> 
> To troubleshoot, I modify my code to read from /var/log/audit/audit.log
> instead.  I modify a single line (with fileinput.input() to read from
> myfile as shown in the commented line below).
> 
> Here is the code snippet (a colorized easier to read version is available
> here - https://pastebin.com/84Nxu3Rp):
> 
> # let us initialize the AuParser
> aup = auparse.AuParser(auparse.AUSOURCE_FEED)
> 
> # we initalize the callback to be fn_process_event
> aup.add_callback(fn_process_event, None, None)
> 
> myfile = "/var/log/audit/audit.log"
> 
> while True:
>     try:
>         # we read in line by line from stdin
>         for line in fileinput.input():
>         #for line in fileinput.input(myfile):
>             aup.feed(line)
>     except:
>         logger.error("Fatal error in while loop", exc_info=True)
> 
> # we flush the feed when we quit
> aup.flush_feed()
> 
> Any suggestions on how to troubleshoot these types of issues when reading
> from a file works fine without issue but running it as a plugin fails as
> shown in /var/log/syslog?  Thanks.

All plugins have a requirement to take events from stdin. As long as it 
expects strings (which is the way that auparse wants them), then all you have 
to do is:

ausearch --start boot --raw | ./plugin

You can also save raw logs with ausearch and cat them into the plugin. This 
is helpful when you get a problem down to a certain series of events and you 
don't want to go through a thousand events before the problem sequence.

-Steve




--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Troubleshooting Custom audispd Plugin

[Osama Elnaggar]

[-- Attachment #1.1: Type: text/plain, Size: 2854 bytes --]

Hi Steve,

I tried it but the problem still only shows up when it runs as a plugin.
Also, the script basically does some processing on the records and extracts
certain data from records of interest, so it should run fine regardless of
the input source.  It seems to fail immediately when run as a plugin.  Any
other suggestions on troubleshooting the discepancy?

PS.  I also read your very useful auditd tutorials over here -
https://security-plus-data-science.blogspot.com/ Thanks.

-- 
Osama Elnaggar

On September 7, 2018 at 10:57:05 PM, Steve Grubb (sgrubb@redhat.com) wrote:

On Friday, September 7, 2018 7:30:09 AM EDT Osama Elnaggar wrote:
> Hi,
>
> I'm working on a custom audispd plugin written in Python 3. It’s a work
in
> progress and I’ve successfully run it numerous times as an audispd
plugin.
> However, I sometimes make modifications that result in the audispd plugin
> failing and I end up with the following in /var/log/syslog
>
> Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
> terminated unexpectedly
> Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
> restarted
> ...
>
> This is repeated several times until audispd gives up and I see the
> following message:
>
> Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
> exceeded max_restarts
>
> To troubleshoot, I modify my code to read from /var/log/audit/audit.log
> instead. I modify a single line (with fileinput.input() to read from
> myfile as shown in the commented line below).
>
> Here is the code snippet (a colorized easier to read version is available
> here - https://pastebin.com/84Nxu3Rp):
>
> # let us initialize the AuParser
> aup = auparse.AuParser(auparse.AUSOURCE_FEED)
>
> # we initalize the callback to be fn_process_event
> aup.add_callback(fn_process_event, None, None)
>
> myfile = "/var/log/audit/audit.log"
>
> while True:
> try:
> # we read in line by line from stdin
> for line in fileinput.input():
> #for line in fileinput.input(myfile):
> aup.feed(line)
> except:
> logger.error("Fatal error in while loop", exc_info=True)
>
> # we flush the feed when we quit
> aup.flush_feed()
>
> Any suggestions on how to troubleshoot these types of issues when reading
> from a file works fine without issue but running it as a plugin fails as
> shown in /var/log/syslog? Thanks.

All plugins have a requirement to take events from stdin. As long as it
expects strings (which is the way that auparse wants them), then all you
have
to do is:

ausearch --start boot --raw | ./plugin

You can also save raw logs with ausearch and cat them into the plugin. This
is helpful when you get a problem down to a certain series of events and
you
don't want to go through a thousand events before the problem sequence.

-Steve

[-- Attachment #1.2: Type: text/html, Size: 4756 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Troubleshooting Custom audispd Plugin

[Steve Grubb]

Hello,

On Friday, September 7, 2018 9:19:34 AM EDT Osama Elnaggar wrote:
> I tried it but the problem still only shows up when it runs as a plugin.
> Also, the script basically does some processing on the records and extracts
> certain data from records of interest, so it should run fine regardless of
> the input source.  It seems to fail immediately when run as a plugin.  Any
> other suggestions on troubleshooting the discepancy?

I don't know if there are any permission restrictions by AppArmor or SE Linux 
if you have those running. I don't know if you are logging errors when they 
occur. But my guess would be something is throwing an uncaught exception. 
Which might be caused by MAC permissions. Just a guess.

-Steve

> PS.  I also read your very useful auditd tutorials over here -
> https://security-plus-data-science.blogspot.com/ Thanks.
> 
> > Hi,
> > 
> > I'm working on a custom audispd plugin written in Python 3. It’s a work
> 
> in
> 
> > progress and I’ve successfully run it numerous times as an audispd
> 
> plugin.
> 
> > However, I sometimes make modifications that result in the audispd plugin
> > failing and I end up with the following in /var/log/syslog
> > 
> > Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
> > terminated unexpectedly
> > Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
> > restarted
> > ...
> > 
> > This is repeated several times until audispd gives up and I see the
> > following message:
> > 
> > Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
> > exceeded max_restarts
> > 
> > To troubleshoot, I modify my code to read from /var/log/audit/audit.log
> > instead. I modify a single line (with fileinput.input() to read from
> > myfile as shown in the commented line below).
> > 
> > Here is the code snippet (a colorized easier to read version is available
> > here - https://pastebin.com/84Nxu3Rp):
> > 
> > # let us initialize the AuParser
> > aup = auparse.AuParser(auparse.AUSOURCE_FEED)
> > 
> > # we initalize the callback to be fn_process_event
> > aup.add_callback(fn_process_event, None, None)
> > 
> > myfile = "/var/log/audit/audit.log"
> > 
> > while True:
> > try:
> > # we read in line by line from stdin
> > for line in fileinput.input():
> > #for line in fileinput.input(myfile):
> > aup.feed(line)
> > except:
> > logger.error("Fatal error in while loop", exc_info=True)
> > 
> > # we flush the feed when we quit
> > aup.flush_feed()
> > 
> > Any suggestions on how to troubleshoot these types of issues when reading
> > from a file works fine without issue but running it as a plugin fails as
> > shown in /var/log/syslog? Thanks.
> 
> All plugins have a requirement to take events from stdin. As long as it
> expects strings (which is the way that auparse wants them), then all you
> have
> to do is:
> 
> ausearch --start boot --raw | ./plugin
> 
> You can also save raw logs with ausearch and cat them into the plugin. This
> is helpful when you get a problem down to a certain series of events and
> you
> don't want to go through a thousand events before the problem sequence.
> 
> -Steve





--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Troubleshooting Custom audispd Plugin

[Osama Elnaggar]

[-- Attachment #1.1: Type: text/plain, Size: 3647 bytes --]

Hi,

Just in case anyone runs into something similar in the future, it was due
to a Python library that I had added to my script and was installed was in
my normal user's environment (+ in sudo -s) but not in the system’s
environment used by audisp.  I discovered this by stracing the audisp
process.  Thank you for your help.

-- 
Osama Elnaggar

On September 7, 2018 at 11:42:27 PM, Steve Grubb (sgrubb@redhat.com) wrote:

Hello,

On Friday, September 7, 2018 9:19:34 AM EDT Osama Elnaggar wrote:
> I tried it but the problem still only shows up when it runs as a plugin.
> Also, the script basically does some processing on the records and
extracts
> certain data from records of interest, so it should run fine regardless
of
> the input source. It seems to fail immediately when run as a plugin. Any
> other suggestions on troubleshooting the discepancy?

I don't know if there are any permission restrictions by AppArmor or SE
Linux
if you have those running. I don't know if you are logging errors when they
occur. But my guess would be something is throwing an uncaught exception.
Which might be caused by MAC permissions. Just a guess.

-Steve

> PS. I also read your very useful auditd tutorials over here -
> https://security-plus-data-science.blogspot.com/ Thanks.
>
> > Hi,
> >
> > I'm working on a custom audispd plugin written in Python 3. It’s a work
>
> in
>
> > progress and I’ve successfully run it numerous times as an audispd
>
> plugin.
>
> > However, I sometimes make modifications that result in the audispd
plugin
> > failing and I end up with the following in /var/log/syslog
> >
> > Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
> > terminated unexpectedly
> > Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
> > restarted
> > ...
> >
> > This is repeated several times until audispd gives up and I see the
> > following message:
> >
> > Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
> > exceeded max_restarts
> >
> > To troubleshoot, I modify my code to read from /var/log/audit/audit.log
> > instead. I modify a single line (with fileinput.input() to read from
> > myfile as shown in the commented line below).
> >
> > Here is the code snippet (a colorized easier to read version is
available
> > here - https://pastebin.com/84Nxu3Rp):
> >
> > # let us initialize the AuParser
> > aup = auparse.AuParser(auparse.AUSOURCE_FEED)
> >
> > # we initalize the callback to be fn_process_event
> > aup.add_callback(fn_process_event, None, None)
> >
> > myfile = "/var/log/audit/audit.log"
> >
> > while True:
> > try:
> > # we read in line by line from stdin
> > for line in fileinput.input():
> > #for line in fileinput.input(myfile):
> > aup.feed(line)
> > except:
> > logger.error("Fatal error in while loop", exc_info=True)
> >
> > # we flush the feed when we quit
> > aup.flush_feed()
> >
> > Any suggestions on how to troubleshoot these types of issues when
reading
> > from a file works fine without issue but running it as a plugin fails
as
> > shown in /var/log/syslog? Thanks.
>
> All plugins have a requirement to take events from stdin. As long as it
> expects strings (which is the way that auparse wants them), then all you
> have
> to do is:
>
> ausearch --start boot --raw | ./plugin
>
> You can also save raw logs with ausearch and cat them into the plugin.
This
> is helpful when you get a problem down to a certain series of events and
> you
> don't want to go through a thousand events before the problem sequence.
>
> -Steve

[-- Attachment #1.2: Type: text/html, Size: 5341 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]