Weird issues in 2.6.5

Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 1804 bytes --]

Hi, I had some odd behavior to report.

I am running ubuntu 12.04.  Using the default auditd and audispd-plugins
packages for my release, I was able to get logs sent to local syslog and to
a remote auditd server (same basic configuration), but the entries were
being buffered somewhere (I think on the client side), and if the server
died reconnections didn't happen.

So, I wanted a more recent version, so I compiled audit-userspace from the
github src mirror,* trunk@1341.

When I did, I got some weird results.  For example, I expected got
something like this in my audit.log:

  node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
 cwd="/etc/audisp"

And that was as expected.  In syslog, I expected to get:

  Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"

But instead, I got:

  Jul 13 08:34:53 host audispd: type=CWD msg=node=host.loc.example.com
type=CWD msg=audit(1468363871.644

As you can see, the whole thing was prepended with "type=CWD msg=", and the
line was truncated.  Similarly, on the remote host, I got the same thing:

  type=CWD msg=node=host.loc.example.com type=CWD msg=audit(1468363871.644

I noticed that the most recent version of the src for ubuntu was 2.4.5, so
I grabbed the src tarball from packages.ubuntu and built it, and now
everything looks fine.  The exact same line I see in my audit.log shows up
in the remote audit.log, with no buffering.  When I restart the remote
auditd server or client, it reconnects.  syslog has same entry (prepended
with the timestamp etc.).  Everything seems happy now.


*For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I ran
`make` from the svn/git src.  I did not require this when building 2.4.5
from the ubuntu src.


--Chris

[-- Attachment #1.2: Type: text/html, Size: 2449 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> Hi, I had some odd behavior to report.
> 
> I am running ubuntu 12.04.  Using the default auditd and audispd-plugins
> packages for my release, I was able to get logs sent to local syslog and to
> a remote auditd server (same basic configuration), but the entries were
> being buffered somewhere (I think on the client side), and if the server
> died reconnections didn't happen.
> 
> So, I wanted a more recent version, so I compiled audit-userspace from the
> github src mirror,* trunk@1341.

The github repo is a mirror of svn and is not always up to date. The issue you 
are seeing is fixed in the next commit after the mirror stops.

https://fedorahosted.org/audit/changeset/1342

if you want the lastest you can:

svn co http://svn.fedorahosted.org/svn/audit/trunk

and then generate from there. I am planning to release audit-2.6.5 tomorrow. 
So, if anyone can test the current code, I'd really appreciate it. I'm hoping 
the next release settles down the audit code.


> When I did, I got some weird results.  For example, I expected got
> something like this in my audit.log:
> 
>   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
>  cwd="/etc/audisp"
> 
> And that was as expected.  In syslog, I expected to get:
> 
>   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> 
> But instead, I got:
> 
>   Jul 13 08:34:53 host audispd: type=CWD msg=node=host.loc.example.com
> type=CWD msg=audit(1468363871.644
> 
> As you can see, the whole thing was prepended with "type=CWD msg=", and the
> line was truncated.  Similarly, on the remote host, I got the same thing:
> 
>   type=CWD msg=node=host.loc.example.com type=CWD msg=audit(1468363871.644
> 
> I noticed that the most recent version of the src for ubuntu was 2.4.5, so
> I grabbed the src tarball from packages.ubuntu and built it, and now
> everything looks fine.  The exact same line I see in my audit.log shows up
> in the remote audit.log, with no buffering.  When I restart the remote
> auditd server or client, it reconnects.  syslog has same entry (prepended
> with the timestamp etc.).  Everything seems happy now.
> 
> 
> *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I ran
> `make` from the svn/git src.  I did not require this when building 2.4.5
> from the ubuntu src.

I think that should have been detected during configure.

-Steve

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 3429 bytes --]

Thanks, I'll try building with the actual latest in a bit.

Secondary question: the reason for what I'm working on is that we want to
be able to audit what folks do as root on our production hosts.  We're not
a bank, and a perfect solution is not required, but we do need to be able
to take reasonable steps to find out if people with access are doing bad
things.

Is this setup reasonable for that purpose?  I know that's a loaded question
and I can answer any questions anyone has that are necessary to figure this
out.  I am not asking so much about rules, but about architecture: logging
according to whatever rules we set up, to the local audit.log and
immediately to a remote using audisp-remote, so the log can't be easily
manipulated.


On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > Hi, I had some odd behavior to report.
> >
> > I am running ubuntu 12.04.  Using the default auditd and audispd-plugins
> > packages for my release, I was able to get logs sent to local syslog and
> to
> > a remote auditd server (same basic configuration), but the entries were
> > being buffered somewhere (I think on the client side), and if the server
> > died reconnections didn't happen.
> >
> > So, I wanted a more recent version, so I compiled audit-userspace from
> the
> > github src mirror,* trunk@1341.
>
> The github repo is a mirror of svn and is not always up to date. The issue
> you
> are seeing is fixed in the next commit after the mirror stops.
>
> https://fedorahosted.org/audit/changeset/1342
>
> if you want the lastest you can:
>
> svn co http://svn.fedorahosted.org/svn/audit/trunk
>
> and then generate from there. I am planning to release audit-2.6.5
> tomorrow.
> So, if anyone can test the current code, I'd really appreciate it. I'm
> hoping
> the next release settles down the audit code.
>
>
> > When I did, I got some weird results.  For example, I expected got
> > something like this in my audit.log:
> >
> >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> >  cwd="/etc/audisp"
> >
> > And that was as expected.  In syslog, I expected to get:
> >
> >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> >
> > But instead, I got:
> >
> >   Jul 13 08:34:53 host audispd: type=CWD msg=node=host.loc.example.com
> > type=CWD msg=audit(1468363871.644
> >
> > As you can see, the whole thing was prepended with "type=CWD msg=", and
> the
> > line was truncated.  Similarly, on the remote host, I got the same thing:
> >
> >   type=CWD msg=node=host.loc.example.com type=CWD
> msg=audit(1468363871.644
> >
> > I noticed that the most recent version of the src for ubuntu was 2.4.5,
> so
> > I grabbed the src tarball from packages.ubuntu and built it, and now
> > everything looks fine.  The exact same line I see in my audit.log shows
> up
> > in the remote audit.log, with no buffering.  When I restart the remote
> > auditd server or client, it reconnects.  syslog has same entry (prepended
> > with the timestamp etc.).  Everything seems happy now.
> >
> >
> > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I
> ran
> > `make` from the svn/git src.  I did not require this when building 2.4.5
> > from the ubuntu src.
>
> I think that should have been detected during configure.
>
> -Steve
>

[-- Attachment #1.2: Type: text/html, Size: 4755 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> Secondary question: the reason for what I'm working on is that we want to
> be able to audit what folks do as root on our production hosts.  We're not
> a bank, and a perfect solution is not required, but we do need to be able
> to take reasonable steps to find out if people with access are doing bad
> things.
> 
> Is this setup reasonable for that purpose?

Yes. You would want to do two things, first enable tty auditing. This is done 
by the pam_tty_audit module. Second consider adding the 32-power-abuse.rules 
to your rules.

> I know that's a loaded question
> and I can answer any questions anyone has that are necessary to figure this
> out.  I am not asking so much about rules, but about architecture: logging
> according to whatever rules we set up, to the local audit.log and
> immediately to a remote using audisp-remote, so the log can't be easily
> manipulated.

Remote logging is the defence against local log manipulation.

-Steve

> On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > > Hi, I had some odd behavior to report.
> > > 
> > > I am running ubuntu 12.04.  Using the default auditd and audispd-plugins
> > > packages for my release, I was able to get logs sent to local syslog and
> > 
> > to
> > 
> > > a remote auditd server (same basic configuration), but the entries were
> > > being buffered somewhere (I think on the client side), and if the server
> > > died reconnections didn't happen.
> > > 
> > > So, I wanted a more recent version, so I compiled audit-userspace from
> > 
> > the
> > 
> > > github src mirror,* trunk@1341.
> > 
> > The github repo is a mirror of svn and is not always up to date. The issue
> > you
> > are seeing is fixed in the next commit after the mirror stops.
> > 
> > https://fedorahosted.org/audit/changeset/1342
> > 
> > if you want the lastest you can:
> > 
> > svn co http://svn.fedorahosted.org/svn/audit/trunk
> > 
> > and then generate from there. I am planning to release audit-2.6.5
> > tomorrow.
> > So, if anyone can test the current code, I'd really appreciate it. I'm
> > hoping
> > the next release settles down the audit code.
> > 
> > > When I did, I got some weird results.  For example, I expected got
> > > 
> > > something like this in my audit.log:
> > >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> > >  cwd="/etc/audisp"
> > > 
> > > And that was as expected.  In syslog, I expected to get:
> > >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > > 
> > > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> > > 
> > > But instead, I got:
> > >   Jul 13 08:34:53 host audispd: type=CWD msg=node=host.loc.example.com
> > > 
> > > type=CWD msg=audit(1468363871.644
> > > 
> > > As you can see, the whole thing was prepended with "type=CWD msg=", and
> > 
> > the
> > 
> > > line was truncated.  Similarly, on the remote host, I got the same 
thing:
> > >   type=CWD msg=node=host.loc.example.com type=CWD
> > 
> > msg=audit(1468363871.644
> > 
> > > I noticed that the most recent version of the src for ubuntu was 2.4.5,
> > 
> > so
> > 
> > > I grabbed the src tarball from packages.ubuntu and built it, and now
> > > everything looks fine.  The exact same line I see in my audit.log shows
> > 
> > up
> > 
> > > in the remote audit.log, with no buffering.  When I restart the remote
> > > auditd server or client, it reconnects.  syslog has same entry
> > > (prepended
> > > with the timestamp etc.).  Everything seems happy now.
> > > 
> > > 
> > > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I
> > 
> > ran
> > 
> > > `make` from the svn/git src.  I did not require this when building 2.4.5
> > > from the ubuntu src.
> > 
> > I think that should have been detected during configure.
> > 
> > -Steve

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 12:32:55 PM EDT Steve Grubb wrote:
> On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > Secondary question: the reason for what I'm working on is that we want to
> > be able to audit what folks do as root on our production hosts.  We're not
> > a bank, and a perfect solution is not required, but we do need to be able
> > to take reasonable steps to find out if people with access are doing bad
> > things.
> > 
> > Is this setup reasonable for that purpose?
> 
> Yes. You would want to do two things, first enable tty auditing. This is
> done  by the pam_tty_audit module. Second consider adding the
> 32-power-abuse.rules to your rules.
> 
> > I know that's a loaded question
> > and I can answer any questions anyone has that are necessary to figure
> > this
> > out.  I am not asking so much about rules, but about architecture: logging
> > according to whatever rules we set up, to the local audit.log and
> > immediately to a remote using audisp-remote, so the log can't be easily
> > manipulated.
> 
> Remote logging is the defence against local log manipulation.

Another thing to consider is that the 2.6 version of the audit user space has 
a new logging format. You might consider going into auditd.conf and setting 
log_format = enriched. This resolves some information locally before sending 
it to the remote system.

-Steve

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 1628 bytes --]

Yeah, I saw that (and tried it out, but reverted when I noticed the
truncation issues).  Nice feature!

On Wed, Jul 13, 2016 at 9:42 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 12:32:55 PM EDT Steve Grubb wrote:
> > On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > > Secondary question: the reason for what I'm working on is that we want
> to
> > > be able to audit what folks do as root on our production hosts.  We're
> not
> > > a bank, and a perfect solution is not required, but we do need to be
> able
> > > to take reasonable steps to find out if people with access are doing
> bad
> > > things.
> > >
> > > Is this setup reasonable for that purpose?
> >
> > Yes. You would want to do two things, first enable tty auditing. This is
> > done  by the pam_tty_audit module. Second consider adding the
> > 32-power-abuse.rules to your rules.
> >
> > > I know that's a loaded question
> > > and I can answer any questions anyone has that are necessary to figure
> > > this
> > > out.  I am not asking so much about rules, but about architecture:
> logging
> > > according to whatever rules we set up, to the local audit.log and
> > > immediately to a remote using audisp-remote, so the log can't be easily
> > > manipulated.
> >
> > Remote logging is the defence against local log manipulation.
>
> Another thing to consider is that the 2.6 version of the audit user space
> has
> a new logging format. You might consider going into auditd.conf and setting
> log_format = enriched. This resolves some information locally before
> sending
> it to the remote system.
>
> -Steve
>

[-- Attachment #1.2: Type: text/html, Size: 2203 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 4413 bytes --]

I'll try that module and those rules out, thanks.

Also, do you forsee issues with using 2.6.x userspace on ubuntu12.04, which
is about four years old now?  Its kernel version 3.2.0-99-generic.

On Wed, Jul 13, 2016 at 9:32 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > Secondary question: the reason for what I'm working on is that we want to
> > be able to audit what folks do as root on our production hosts.  We're
> not
> > a bank, and a perfect solution is not required, but we do need to be able
> > to take reasonable steps to find out if people with access are doing bad
> > things.
> >
> > Is this setup reasonable for that purpose?
>
> Yes. You would want to do two things, first enable tty auditing. This is
> done
> by the pam_tty_audit module. Second consider adding the
> 32-power-abuse.rules
> to your rules.
>
> > I know that's a loaded question
> > and I can answer any questions anyone has that are necessary to figure
> this
> > out.  I am not asking so much about rules, but about architecture:
> logging
> > according to whatever rules we set up, to the local audit.log and
> > immediately to a remote using audisp-remote, so the log can't be easily
> > manipulated.
>
> Remote logging is the defence against local log manipulation.
>
> -Steve
>
> > On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > > On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > > > Hi, I had some odd behavior to report.
> > > >
> > > > I am running ubuntu 12.04.  Using the default auditd and
> audispd-plugins
> > > > packages for my release, I was able to get logs sent to local syslog
> and
> > >
> > > to
> > >
> > > > a remote auditd server (same basic configuration), but the entries
> were
> > > > being buffered somewhere (I think on the client side), and if the
> server
> > > > died reconnections didn't happen.
> > > >
> > > > So, I wanted a more recent version, so I compiled audit-userspace
> from
> > >
> > > the
> > >
> > > > github src mirror,* trunk@1341.
> > >
> > > The github repo is a mirror of svn and is not always up to date. The
> issue
> > > you
> > > are seeing is fixed in the next commit after the mirror stops.
> > >
> > > https://fedorahosted.org/audit/changeset/1342
> > >
> > > if you want the lastest you can:
> > >
> > > svn co http://svn.fedorahosted.org/svn/audit/trunk
> > >
> > > and then generate from there. I am planning to release audit-2.6.5
> > > tomorrow.
> > > So, if anyone can test the current code, I'd really appreciate it. I'm
> > > hoping
> > > the next release settles down the audit code.
> > >
> > > > When I did, I got some weird results.  For example, I expected got
> > > >
> > > > something like this in my audit.log:
> > > >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> > > >  cwd="/etc/audisp"
> > > >
> > > > And that was as expected.  In syslog, I expected to get:
> > > >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > > >
> > > > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> > > >
> > > > But instead, I got:
> > > >   Jul 13 08:34:53 host audispd: type=CWD msg=node=
> host.loc.example.com
> > > >
> > > > type=CWD msg=audit(1468363871.644
> > > >
> > > > As you can see, the whole thing was prepended with "type=CWD msg=",
> and
> > >
> > > the
> > >
> > > > line was truncated.  Similarly, on the remote host, I got the same
> thing:
> > > >   type=CWD msg=node=host.loc.example.com type=CWD
> > >
> > > msg=audit(1468363871.644
> > >
> > > > I noticed that the most recent version of the src for ubuntu was
> 2.4.5,
> > >
> > > so
> > >
> > > > I grabbed the src tarball from packages.ubuntu and built it, and now
> > > > everything looks fine.  The exact same line I see in my audit.log
> shows
> > >
> > > up
> > >
> > > > in the remote audit.log, with no buffering.  When I restart the
> remote
> > > > auditd server or client, it reconnects.  syslog has same entry
> > > > (prepended
> > > > with the timestamp etc.).  Everything seems happy now.
> > > >
> > > >
> > > > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when
> I
> > >
> > > ran
> > >
> > > > `make` from the svn/git src.  I did not require this when building
> 2.4.5
> > > > from the ubuntu src.
> > >
> > > I think that should have been detected during configure.
> > >
> > > -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6461 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 9:55:32 AM EDT Chris Nandor wrote:
> I'll try that module and those rules out, thanks.
> 
> Also, do you forsee issues with using 2.6.x userspace on ubuntu12.04, which
> is about four years old now?  Its kernel version 3.2.0-99-generic.

That's a tricky question for me to answer. You would probably be in a non-
supported position. I also don't know if they have made any patches for 
APP_ARMOR support in the audit system. There are a few things that auditctl 
might look for in newer kernels but it should gracefully handle it. It should 
be compatible with the rest of user space.

So, give it a try. It may work. It should be easy to go back if something is 
wrong.

-Steve

> On Wed, Jul 13, 2016 at 9:32 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > > Secondary question: the reason for what I'm working on is that we want
> > > to
> > > be able to audit what folks do as root on our production hosts.  We're
> > 
> > not
> > 
> > > a bank, and a perfect solution is not required, but we do need to be
> > > able
> > > to take reasonable steps to find out if people with access are doing bad
> > > things.
> > > 
> > > Is this setup reasonable for that purpose?
> > 
> > Yes. You would want to do two things, first enable tty auditing. This is
> > done
> > by the pam_tty_audit module. Second consider adding the
> > 32-power-abuse.rules
> > to your rules.
> > 
> > > I know that's a loaded question
> > > and I can answer any questions anyone has that are necessary to figure
> > 
> > this
> > 
> > > out.  I am not asking so much about rules, but about architecture:
> > logging
> > 
> > > according to whatever rules we set up, to the local audit.log and
> > > immediately to a remote using audisp-remote, so the log can't be easily
> > > manipulated.
> > 
> > Remote logging is the defence against local log manipulation.
> > 
> > -Steve
> > 
> > > On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > > > On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > > > > Hi, I had some odd behavior to report.
> > > > > 
> > > > > I am running ubuntu 12.04.  Using the default auditd and
> > 
> > audispd-plugins
> > 
> > > > > packages for my release, I was able to get logs sent to local syslog
> > 
> > and
> > 
> > > > to
> > > > 
> > > > > a remote auditd server (same basic configuration), but the entries
> > 
> > were
> > 
> > > > > being buffered somewhere (I think on the client side), and if the
> > 
> > server
> > 
> > > > > died reconnections didn't happen.
> > > > > 
> > > > > So, I wanted a more recent version, so I compiled audit-userspace
> > 
> > from
> > 
> > > > the
> > > > 
> > > > > github src mirror,* trunk@1341.
> > > > 
> > > > The github repo is a mirror of svn and is not always up to date. The
> > 
> > issue
> > 
> > > > you
> > > > are seeing is fixed in the next commit after the mirror stops.
> > > > 
> > > > https://fedorahosted.org/audit/changeset/1342
> > > > 
> > > > if you want the lastest you can:
> > > > 
> > > > svn co http://svn.fedorahosted.org/svn/audit/trunk
> > > > 
> > > > and then generate from there. I am planning to release audit-2.6.5
> > > > tomorrow.
> > > > So, if anyone can test the current code, I'd really appreciate it. I'm
> > > > hoping
> > > > the next release settles down the audit code.
> > > > 
> > > > > When I did, I got some weird results.  For example, I expected got
> > > > > 
> > > > > something like this in my audit.log:
> > > > >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> > > > >  cwd="/etc/audisp"
> > > > > 
> > > > > And that was as expected.  In syslog, I expected to get:
> > > > >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > > > > 
> > > > > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> > > > > 
> > > > > But instead, I got:
> > > > >   Jul 13 08:34:53 host audispd: type=CWD msg=node=
> > 
> > host.loc.example.com
> > 
> > > > > type=CWD msg=audit(1468363871.644
> > > > > 
> > > > > As you can see, the whole thing was prepended with "type=CWD msg=",
> > 
> > and
> > 
> > > > the
> > > > 
> > > > > line was truncated.  Similarly, on the remote host, I got the same
> > 
> > thing:
> > > > >   type=CWD msg=node=host.loc.example.com type=CWD
> > > > 
> > > > msg=audit(1468363871.644
> > > > 
> > > > > I noticed that the most recent version of the src for ubuntu was
> > 
> > 2.4.5,
> > 
> > > > so
> > > > 
> > > > > I grabbed the src tarball from packages.ubuntu and built it, and now
> > > > > everything looks fine.  The exact same line I see in my audit.log
> > 
> > shows
> > 
> > > > up
> > > > 
> > > > > in the remote audit.log, with no buffering.  When I restart the
> > 
> > remote
> > 
> > > > > auditd server or client, it reconnects.  syslog has same entry
> > > > > (prepended
> > > > > with the timestamp etc.).  Everything seems happy now.
> > > > > 
> > > > > 
> > > > > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when
> > 
> > I
> > 
> > > > ran
> > > > 
> > > > > `make` from the svn/git src.  I did not require this when building
> > 
> > 2.4.5
> > 
> > > > > from the ubuntu src.
> > > > 
> > > > I think that should have been detected during configure.
> > > > 
> > > > -Steve

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 6201 bytes --]

The only reason I am even upgrading is because of the issues with
audisp-remote, the not-reconnecting, and the apparent client-side
buffering, that went away with 2.4.x and 2.6.x.  So if we decide to ship
logs a different way than with audisp-remote, then it might be best to
stick with 1.7.x.

That said, so far I see no issues, so we're going to forge ahead and see
what happens.  I just need to keep in mind what our mitigation plan would
be if we do run into issues.

Thanks!


On Wed, Jul 13, 2016 at 10:07 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 9:55:32 AM EDT Chris Nandor wrote:
> > I'll try that module and those rules out, thanks.
> >
> > Also, do you forsee issues with using 2.6.x userspace on ubuntu12.04,
> which
> > is about four years old now?  Its kernel version 3.2.0-99-generic.
>
> That's a tricky question for me to answer. You would probably be in a non-
> supported position. I also don't know if they have made any patches for
> APP_ARMOR support in the audit system. There are a few things that auditctl
> might look for in newer kernels but it should gracefully handle it. It
> should
> be compatible with the rest of user space.
>
> So, give it a try. It may work. It should be easy to go back if something
> is
> wrong.
>
> -Steve
>
> > On Wed, Jul 13, 2016 at 9:32 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > > On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > > > Secondary question: the reason for what I'm working on is that we
> want
> > > > to
> > > > be able to audit what folks do as root on our production hosts.
> We're
> > >
> > > not
> > >
> > > > a bank, and a perfect solution is not required, but we do need to be
> > > > able
> > > > to take reasonable steps to find out if people with access are doing
> bad
> > > > things.
> > > >
> > > > Is this setup reasonable for that purpose?
> > >
> > > Yes. You would want to do two things, first enable tty auditing. This
> is
> > > done
> > > by the pam_tty_audit module. Second consider adding the
> > > 32-power-abuse.rules
> > > to your rules.
> > >
> > > > I know that's a loaded question
> > > > and I can answer any questions anyone has that are necessary to
> figure
> > >
> > > this
> > >
> > > > out.  I am not asking so much about rules, but about architecture:
> > > logging
> > >
> > > > according to whatever rules we set up, to the local audit.log and
> > > > immediately to a remote using audisp-remote, so the log can't be
> easily
> > > > manipulated.
> > >
> > > Remote logging is the defence against local log manipulation.
> > >
> > > -Steve
> > >
> > > > On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com>
> wrote:
> > > > > On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > > > > > Hi, I had some odd behavior to report.
> > > > > >
> > > > > > I am running ubuntu 12.04.  Using the default auditd and
> > >
> > > audispd-plugins
> > >
> > > > > > packages for my release, I was able to get logs sent to local
> syslog
> > >
> > > and
> > >
> > > > > to
> > > > >
> > > > > > a remote auditd server (same basic configuration), but the
> entries
> > >
> > > were
> > >
> > > > > > being buffered somewhere (I think on the client side), and if the
> > >
> > > server
> > >
> > > > > > died reconnections didn't happen.
> > > > > >
> > > > > > So, I wanted a more recent version, so I compiled audit-userspace
> > >
> > > from
> > >
> > > > > the
> > > > >
> > > > > > github src mirror,* trunk@1341.
> > > > >
> > > > > The github repo is a mirror of svn and is not always up to date.
> The
> > >
> > > issue
> > >
> > > > > you
> > > > > are seeing is fixed in the next commit after the mirror stops.
> > > > >
> > > > > https://fedorahosted.org/audit/changeset/1342
> > > > >
> > > > > if you want the lastest you can:
> > > > >
> > > > > svn co http://svn.fedorahosted.org/svn/audit/trunk
> > > > >
> > > > > and then generate from there. I am planning to release audit-2.6.5
> > > > > tomorrow.
> > > > > So, if anyone can test the current code, I'd really appreciate it.
> I'm
> > > > > hoping
> > > > > the next release settles down the audit code.
> > > > >
> > > > > > When I did, I got some weird results.  For example, I expected
> got
> > > > > >
> > > > > > something like this in my audit.log:
> > > > > >   node=host.example.com type=CWD
> msg=audit(1468363871.644:3279856):
> > > > > >  cwd="/etc/audisp"
> > > > > >
> > > > > > And that was as expected.  In syslog, I expected to get:
> > > > > >   Jul 13 08:34:53 host audispd: node=host.loc.example.com
> type=CWD
> > > > > >
> > > > > > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> > > > > >
> > > > > > But instead, I got:
> > > > > >   Jul 13 08:34:53 host audispd: type=CWD msg=node=
> > >
> > > host.loc.example.com
> > >
> > > > > > type=CWD msg=audit(1468363871.644
> > > > > >
> > > > > > As you can see, the whole thing was prepended with "type=CWD
> msg=",
> > >
> > > and
> > >
> > > > > the
> > > > >
> > > > > > line was truncated.  Similarly, on the remote host, I got the
> same
> > >
> > > thing:
> > > > > >   type=CWD msg=node=host.loc.example.com type=CWD
> > > > >
> > > > > msg=audit(1468363871.644
> > > > >
> > > > > > I noticed that the most recent version of the src for ubuntu was
> > >
> > > 2.4.5,
> > >
> > > > > so
> > > > >
> > > > > > I grabbed the src tarball from packages.ubuntu and built it, and
> now
> > > > > > everything looks fine.  The exact same line I see in my audit.log
> > >
> > > shows
> > >
> > > > > up
> > > > >
> > > > > > in the remote audit.log, with no buffering.  When I restart the
> > >
> > > remote
> > >
> > > > > > auditd server or client, it reconnects.  syslog has same entry
> > > > > > (prepended
> > > > > > with the timestamp etc.).  Everything seems happy now.
> > > > > >
> > > > > >
> > > > > > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell
> when
> > >
> > > I
> > >
> > > > > ran
> > > > >
> > > > > > `make` from the svn/git src.  I did not require this when
> building
> > >
> > > 2.4.5
> > >
> > > > > > from the ubuntu src.
> > > > >
> > > > > I think that should have been detected during configure.
> > > > >
> > > > > -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 9354 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 10:51:07 AM EDT Chris Nandor wrote:
> The only reason I am even upgrading is because of the issues with
> audisp-remote, the not-reconnecting, and the apparent client-side
> buffering, that went away with 2.4.x and 2.6.x.  So if we decide to ship
> logs a different way than with audisp-remote, then it might be best to
> stick with 1.7.x.

This sounds a lot like the idle detection is not set right. In audisp-
remote.conf there is a setting heartbeat_timeout. This should be set to 
something like 60 or 120. Then on the server in auditd.conf there is a setting 
tcp_client_max_idle which should be over twice as high as heartbeat_timeout. 
So, you'd set it to 180 or 300.
 
> That said, so far I see no issues, so we're going to forge ahead and see
> what happens.  I just need to keep in mind what our mitigation plan would
> be if we do run into issues.

Old utilities won't know what to do with enriched events. AFAICS, that would 
be the long term issue. You'll need to do aperl, awk, or cut command to trim 
off the unknown part of the event in your logs.

-Steve

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 1338 bytes --]

Since we're new to auditd, there's no old utilities so far that we're
using, so I don't see a problem there.  I think.

On Wed, Jul 13, 2016 at 11:38 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 10:51:07 AM EDT Chris Nandor wrote:
> > The only reason I am even upgrading is because of the issues with
> > audisp-remote, the not-reconnecting, and the apparent client-side
> > buffering, that went away with 2.4.x and 2.6.x.  So if we decide to ship
> > logs a different way than with audisp-remote, then it might be best to
> > stick with 1.7.x.
>
> This sounds a lot like the idle detection is not set right. In audisp-
> remote.conf there is a setting heartbeat_timeout. This should be set to
> something like 60 or 120. Then on the server in auditd.conf there is a
> setting
> tcp_client_max_idle which should be over twice as high as
> heartbeat_timeout.
> So, you'd set it to 180 or 300.
>
> > That said, so far I see no issues, so we're going to forge ahead and see
> > what happens.  I just need to keep in mind what our mitigation plan would
> > be if we do run into issues.
>
> Old utilities won't know what to do with enriched events. AFAICS, that
> would
> be the long term issue. You'll need to do aperl, awk, or cut command to
> trim
> off the unknown part of the event in your logs.
>
> -Steve
>

[-- Attachment #1.2: Type: text/html, Size: 1850 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 3136 bytes --]

I set heartbeat to 60 on the client and idle to 120 on the server.
Reconnects seem fine now, although I never did nail down the exact
conditions under which reconnects failed.

But I still have the problem of weird buffering on the client side.  If I
run `sudo ls` on the client side, locally I get something like:

    node=grax.sea.marchex.com type=SYSCALL
msg=audit(1468448156.161:3288765): arch=c000003e syscall=59 success=yes
exit=0 a0=8f81e8 a1=8f7578 a2=8febf0 a3=7ffd3d956370 items=2 ppid=19387
pid=19388 auid=2288 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=pts4 ses=399 comm="ls" exe="/bin/ls" key="rootcmd"
    node=grax.sea.marchex.com type=EXECVE
msg=audit(1468448156.161:3288765): argc=1 a0="ls"
    node=grax.sea.marchex.com type=CWD msg=audit(1468448156.161:3288765):
 cwd="/tmp"
    node=grax.sea.marchex.com type=PATH msg=audit(1468448156.161:3288765):
item=0 name="/bin/ls" inode=132438 dev=09:01 mode=0100755 ouid=0 ogid=0
rdev=00:00
    node=grax.sea.marchex.com type=PATH msg=audit(1468448156.161:3288765):
item=1 name=(null) inode=71179 dev=09:01 mode=0100755 ouid=0 ogid=0
rdev=00:00

But remotely, I just get:

    node=grax.sea.marchex.com type=SYSCALL
msg=audit(1468448156.161:3288765): arch=c000003e syscall=59 success=yes
exit=0 a0=8f81e8 a1=8f7578 a2=8febf0 a3=7ffd3d956370 items=2 ppid=19387
pid=19388 auid=2288 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=pts4 ses=399 comm="ls" exe="/bin/ls" key="rootcmd"

Just the first line of the audit record.  No matter how long I wait.  If I
then run `sudo ls` again, *then* the rest of the lines show up in the
server's log.

The buffering appears to be on the client side, because if I restart the
server's auditd, those lines are not lost: they still appear in the remote
log ... but not until the next time I run `sudo ls` on the client side.

This is on 1.7.x.  This does not happen on 2.4.x or 2.6.x.







On Wed, Jul 13, 2016 at 11:38 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 10:51:07 AM EDT Chris Nandor wrote:
> > The only reason I am even upgrading is because of the issues with
> > audisp-remote, the not-reconnecting, and the apparent client-side
> > buffering, that went away with 2.4.x and 2.6.x.  So if we decide to ship
> > logs a different way than with audisp-remote, then it might be best to
> > stick with 1.7.x.
>
> This sounds a lot like the idle detection is not set right. In audisp-
> remote.conf there is a setting heartbeat_timeout. This should be set to
> something like 60 or 120. Then on the server in auditd.conf there is a
> setting
> tcp_client_max_idle which should be over twice as high as
> heartbeat_timeout.
> So, you'd set it to 180 or 300.
>
> > That said, so far I see no issues, so we're going to forge ahead and see
> > what happens.  I just need to keep in mind what our mitigation plan would
> > be if we do run into issues.
>
> Old utilities won't know what to do with enriched events. AFAICS, that
> would
> be the long term issue. You'll need to do aperl, awk, or cut command to
> trim
> off the unknown part of the event in your logs.
>
> -Steve
>

[-- Attachment #1.2: Type: text/html, Size: 4318 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Steve Grubb]

On Wednesday, July 13, 2016 3:22:01 PM EDT Chris Nandor wrote:
> The buffering appears to be on the client side, because if I restart the
> server's auditd, those lines are not lost: they still appear in the remote
> log ... but not until the next time I run `sudo ls` on the client side.
> 
> This is on 1.7.x.  This does not happen on 2.4.x or 2.6.x.

Bugs were fixed. 1.7.x is very old.

-Steve

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 4615 bytes --]

Is there some documentation on how tty auditing works?  I've got it
enabled, but I can't tell when something is going to be logged.  It's not
immediate.  (Also, on ubuntu 12.04, our older kernel apparently doesn't
support the feature in the pam module that does *not* log passwords.  So
that's fun.  Maybe not the end of the world, if we get kerberos running for
the log transfer, but ... yuck.)

On Wed, Jul 13, 2016 at 9:32 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 9:22:57 AM EDT Chris Nandor wrote:
> > Secondary question: the reason for what I'm working on is that we want to
> > be able to audit what folks do as root on our production hosts.  We're
> not
> > a bank, and a perfect solution is not required, but we do need to be able
> > to take reasonable steps to find out if people with access are doing bad
> > things.
> >
> > Is this setup reasonable for that purpose?
>
> Yes. You would want to do two things, first enable tty auditing. This is
> done
> by the pam_tty_audit module. Second consider adding the
> 32-power-abuse.rules
> to your rules.
>
> > I know that's a loaded question
> > and I can answer any questions anyone has that are necessary to figure
> this
> > out.  I am not asking so much about rules, but about architecture:
> logging
> > according to whatever rules we set up, to the local audit.log and
> > immediately to a remote using audisp-remote, so the log can't be easily
> > manipulated.
>
> Remote logging is the defence against local log manipulation.
>
> -Steve
>
> > On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:
> > > On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > > > Hi, I had some odd behavior to report.
> > > >
> > > > I am running ubuntu 12.04.  Using the default auditd and
> audispd-plugins
> > > > packages for my release, I was able to get logs sent to local syslog
> and
> > >
> > > to
> > >
> > > > a remote auditd server (same basic configuration), but the entries
> were
> > > > being buffered somewhere (I think on the client side), and if the
> server
> > > > died reconnections didn't happen.
> > > >
> > > > So, I wanted a more recent version, so I compiled audit-userspace
> from
> > >
> > > the
> > >
> > > > github src mirror,* trunk@1341.
> > >
> > > The github repo is a mirror of svn and is not always up to date. The
> issue
> > > you
> > > are seeing is fixed in the next commit after the mirror stops.
> > >
> > > https://fedorahosted.org/audit/changeset/1342
> > >
> > > if you want the lastest you can:
> > >
> > > svn co http://svn.fedorahosted.org/svn/audit/trunk
> > >
> > > and then generate from there. I am planning to release audit-2.6.5
> > > tomorrow.
> > > So, if anyone can test the current code, I'd really appreciate it. I'm
> > > hoping
> > > the next release settles down the audit code.
> > >
> > > > When I did, I got some weird results.  For example, I expected got
> > > >
> > > > something like this in my audit.log:
> > > >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> > > >  cwd="/etc/audisp"
> > > >
> > > > And that was as expected.  In syslog, I expected to get:
> > > >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > > >
> > > > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> > > >
> > > > But instead, I got:
> > > >   Jul 13 08:34:53 host audispd: type=CWD msg=node=
> host.loc.example.com
> > > >
> > > > type=CWD msg=audit(1468363871.644
> > > >
> > > > As you can see, the whole thing was prepended with "type=CWD msg=",
> and
> > >
> > > the
> > >
> > > > line was truncated.  Similarly, on the remote host, I got the same
> thing:
> > > >   type=CWD msg=node=host.loc.example.com type=CWD
> > >
> > > msg=audit(1468363871.644
> > >
> > > > I noticed that the most recent version of the src for ubuntu was
> 2.4.5,
> > >
> > > so
> > >
> > > > I grabbed the src tarball from packages.ubuntu and built it, and now
> > > > everything looks fine.  The exact same line I see in my audit.log
> shows
> > >
> > > up
> > >
> > > > in the remote audit.log, with no buffering.  When I restart the
> remote
> > > > auditd server or client, it reconnects.  syslog has same entry
> > > > (prepended
> > > > with the timestamp etc.).  Everything seems happy now.
> > > >
> > > >
> > > > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when
> I
> > >
> > > ran
> > >
> > > > `make` from the svn/git src.  I did not require this when building
> 2.4.5
> > > > from the ubuntu src.
> > >
> > > I think that should have been detected during configure.
> > >
> > > -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6662 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Chris Nandor]

[-- Attachment #1.1: Type: text/plain, Size: 2790 bytes --]

I've been using the latest (@r1347) for the last couple of hours, and the
problems I saw have gone away.

On Wed, Jul 13, 2016 at 8:57 AM, Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, July 13, 2016 8:47:58 AM EDT Chris Nandor wrote:
> > Hi, I had some odd behavior to report.
> >
> > I am running ubuntu 12.04.  Using the default auditd and audispd-plugins
> > packages for my release, I was able to get logs sent to local syslog and
> to
> > a remote auditd server (same basic configuration), but the entries were
> > being buffered somewhere (I think on the client side), and if the server
> > died reconnections didn't happen.
> >
> > So, I wanted a more recent version, so I compiled audit-userspace from
> the
> > github src mirror,* trunk@1341.
>
> The github repo is a mirror of svn and is not always up to date. The issue
> you
> are seeing is fixed in the next commit after the mirror stops.
>
> https://fedorahosted.org/audit/changeset/1342
>
> if you want the lastest you can:
>
> svn co http://svn.fedorahosted.org/svn/audit/trunk
>
> and then generate from there. I am planning to release audit-2.6.5
> tomorrow.
> So, if anyone can test the current code, I'd really appreciate it. I'm
> hoping
> the next release settles down the audit code.
>
>
> > When I did, I got some weird results.  For example, I expected got
> > something like this in my audit.log:
> >
> >   node=host.example.com type=CWD msg=audit(1468363871.644:3279856):
> >  cwd="/etc/audisp"
> >
> > And that was as expected.  In syslog, I expected to get:
> >
> >   Jul 13 08:34:53 host audispd: node=host.loc.example.com type=CWD
> > msg=audit(1468363871.644:3279856):  cwd="/etc/audisp"
> >
> > But instead, I got:
> >
> >   Jul 13 08:34:53 host audispd: type=CWD msg=node=host.loc.example.com
> > type=CWD msg=audit(1468363871.644
> >
> > As you can see, the whole thing was prepended with "type=CWD msg=", and
> the
> > line was truncated.  Similarly, on the remote host, I got the same thing:
> >
> >   type=CWD msg=node=host.loc.example.com type=CWD
> msg=audit(1468363871.644
> >
> > I noticed that the most recent version of the src for ubuntu was 2.4.5,
> so
> > I grabbed the src tarball from packages.ubuntu and built it, and now
> > everything looks fine.  The exact same line I see in my audit.log shows
> up
> > in the remote audit.log, with no buffering.  When I restart the remote
> > auditd server or client, it reconnects.  syslog has same entry (prepended
> > with the timestamp etc.).  Everything seems happy now.
> >
> >
> > *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I
> ran
> > `make` from the svn/git src.  I did not require this when building 2.4.5
> > from the ubuntu src.
>
> I think that should have been detected during configure.
>
> -Steve
>

[-- Attachment #1.2: Type: text/html, Size: 4027 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Weird issues in 2.6.5

[Richard Guy Briggs]

On 2016-07-13 08:47, Chris Nandor wrote:
> *For some reason I had to define `CC_FOR_BUILD=gcc` in my shell when I ran
> `make` from the svn/git src.  I did not require this when building 2.4.5
> from the ubuntu src.

Ok, so it isn't just me.  I had a similar problem a couple of months ago
when building from current source on RHEL6.

> --Chris

- RGB

--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635