Auditd misses accept syscalls from sshd

Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 741 bytes --]

Auditd seems to miss accept syscalls from ssh on Ubuntu 14. I tried
versions 2.3.2 and 2.4.5 of the daemon with kernel versions 3.13.0-96 and
4.4.0-47. In all cases the accept syscall (43) failed to show up until
after I restarted the ssh daemon. It's especially weird because I don't see
this problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I am seeing
this or where to look?

I found a similar question in the archives, but it seems to do with the
architecture size and not OS versions:
https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html

I also posted this question on Stack Overflow:
http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-have-inconsistent-behavior-in-linux-audit-framework

[-- Attachment #1.2: Type: text/html, Size: 1073 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Steve Grubb]

On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> Auditd seems to miss accept syscalls from ssh on Ubuntu 14.

Its not auditd, the kernel does all the work. Auditd acts a lot like a
specialized syslog.  :-)


> I tried versions 2.3.2 and 2.4.5 of the daemon with kernel versions
> 3.13.0-96 and 4.4.0-47. In all cases the accept syscall (43) failed to show
> up until after I restarted the ssh daemon. It's especially weird because I
> don't see this problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I am
> seeing this or where to look?

It works fine on my 4.8 kernel:
# uname -r
4.8.10-200.fc24.x86_64

# auditctl -a always,exit -F arch=b64 -S accept,accept4 -F exe=/usr/sbin/sshd -F key=test

# ssh localhost
# exit

# ausearch --start recent -k test -i
----
type=CONFIG_CHANGE msg=audit(12/02/2016 15:53:00.297:917) : auid=sgrubb ses=5
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 op="add_rule" key=test
list=exit res=yes 
----
type=PROCTITLE msg=audit(12/02/2016 15:53:07.287:919) : proctitle=/usr/sbin/sshd 
type=SOCKADDR msg=audit(12/02/2016 15:53:07.287:919) : saddr={ fam=inet6 laddr=::1 lport=52740 } 
type=SYSCALL msg=audit(12/02/2016 15:53:07.287:919) : arch=x86_64
syscall=accept success=yes exit=5 a0=0x4 a1=0x7ffdd5bd06a0 a2=0x7ffdd5bd068c
a3=0x0 items=0 ppid=1 pid=1071 auid=unset uid=root gid=root euid=root
suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset
comm=sshd exe=/usr/sbin/sshd subj=system_u:system_r:sshd_t:s0-s0:c0.c1023
key=test 

I don't know if there were any bug fixes that made it start working. I also
think I was doing some testing on kernels close to when the audit by
executable code first went upstream and I remember not getting the results I
wanted. I had other things to do and when I came back to it I could not
replicate the missing events. I had upgraded the kernel in the mean time.

Does using a newer kernel fix it for you?

-Steve

> I found a similar question in the archives, but it seems to do with the
> architecture size and not OS versions:
> https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html
> 
> I also posted this question on Stack Overflow:
> http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-hav
> e-inconsistent-behavior-in-linux-audit-framework

Re: Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 2938 bytes --]

On Fri, Dec 2, 2016 at 4:09 PM Steve Grubb <sgrubb@redhat.com> wrote:

> On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> > Auditd seems to miss accept syscalls from ssh on Ubuntu 14.
>
> Its not auditd, the kernel does all the work. Auditd acts a lot like a
> specialized syslog.  :-)
>
>
> > I tried versions 2.3.2 and 2.4.5 of the daemon with kernel versions
> > 3.13.0-96 and 4.4.0-47. In all cases the accept syscall (43) failed to
> show
> > up until after I restarted the ssh daemon. It's especially weird because
> I
> > don't see this problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I
> am
> > seeing this or where to look?
>
> It works fine on my 4.8 kernel:
> # uname -r
> 4.8.10-200.fc24.x86_64
>
> # auditctl -a always,exit -F arch=b64 -S accept,accept4 -F
> exe=/usr/sbin/sshd -F key=test
>
> # ssh localhost
> # exit
>
> # ausearch --start recent -k test -i
> ----
> type=CONFIG_CHANGE msg=audit(12/02/2016 15:53:00.297:917) : auid=sgrubb
> ses=5
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 op="add_rule"
> key=test
> list=exit res=yes
> ----
> type=PROCTITLE msg=audit(12/02/2016 15:53:07.287:919) :
> proctitle=/usr/sbin/sshd
> type=SOCKADDR msg=audit(12/02/2016 15:53:07.287:919) : saddr={ fam=inet6
> laddr=::1 lport=52740 }
> type=SYSCALL msg=audit(12/02/2016 15:53:07.287:919) : arch=x86_64
> syscall=accept success=yes exit=5 a0=0x4 a1=0x7ffdd5bd06a0
> a2=0x7ffdd5bd068c
> a3=0x0 items=0 ppid=1 pid=1071 auid=unset uid=root gid=root euid=root
> suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset
> comm=sshd exe=/usr/sbin/sshd subj=system_u:system_r:sshd_t:s0-s0:c0.c1023
> key=test
>
> I don't know if there were any bug fixes that made it start working. I also
> think I was doing some testing on kernels close to when the audit by
> executable code first went upstream and I remember not getting the results
> I
> wanted. I had other things to do and when I came back to it I could not
> replicate the missing events. I had upgraded the kernel in the mean time.
>
> Does using a newer kernel fix it for you?
>
> -Steve
>
> > I found a similar question in the archives, but it seems to do with the
> > architecture size and not OS versions:
> > https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html
> >
> > I also posted this question on Stack Overflow:
> >
> http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-hav
> > e-inconsistent-behavior-in-linux-audit-framework
>

 I just tried again and had the same problem:

vagrant@vagrant:~$ uname -a
Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

That's a newer version than I have on my Ubuntu 16 VM, which does
demonstrate the problem. It's also strange that restarting ssh then makes
the accept syscall events show up. Other sshd syscalls show up in auditd
before and after the ssh restart.

[-- Attachment #1.2: Type: text/html, Size: 4729 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Steve Grubb]

Hello,

Addressing a couple obvious things here...

On Friday, December 2, 2016 9:55:17 PM EST Nathan Cooprider wrote:
> On Fri, Dec 2, 2016 at 4:09 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> > > Auditd seems to miss accept syscalls from ssh on Ubuntu 14.
> > 
> > Its not auditd, the kernel does all the work. Auditd acts a lot like a
> > specialized syslog.  :-)
> > 
> > > I tried versions 2.3.2 and 2.4.5 of the daemon

Support was not added until 2.5.

> > > with kernel versions 3.13.0-96

Definitely won't support it.

> > > and 4.4.0-47.

The feature landed in 4.3, so 4.4 should have it. However, you need audit 2.5 
or later to use the kernel feature.


>  I just tried again and had the same problem:
> 
> vagrant@vagrant:~$ uname -a
> Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30
> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Try pairing that with a newer auditd so that auditctl has the support to load 
the rule.

-Steve

> That's a newer version than I have on my Ubuntu 16 VM, which does
> demonstrate the problem. It's also strange that restarting ssh then makes
> the accept syscall events show up. Other sshd syscalls show up in auditd
> before and after the ssh restart.

Re: Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 2125 bytes --]

Hi! It sounds like I'm missing something obvious!

On Fri, Dec 2, 2016 at 5:13 PM Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> Addressing a couple obvious things here...
>
> On Friday, December 2, 2016 9:55:17 PM EST Nathan Cooprider wrote:
> > On Fri, Dec 2, 2016 at 4:09 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > > On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> > > > Auditd seems to miss accept syscalls from ssh on Ubuntu 14.
> > >
> > > Its not auditd, the kernel does all the work. Auditd acts a lot like a
> > > specialized syslog.  :-)
> > >
> > > > I tried versions 2.3.2 and 2.4.5 of the daemon
>
> Support was not added until 2.5.
>

Support for what? Auditing the accept syscall? What do you mean by
"support?" Those are auditd versions that I'm talking about. Is that what
you mean? Sorry if I was not clear. What did it do with accept syscalls
before then? I do not see this reflected in the changelog

https://people.redhat.com/sgrubb/audit/ChangeLog

> > > with kernel versions 3.13.0-96
>
> Definitely won't support it.
>

Support what?


> > > > and 4.4.0-47.
>
> The feature landed in 4.3, so 4.4 should have it. However, you need audit
> 2.5
> or later to use the kernel feature.
>

What feature are you talking about? This sounds like it could be the issue,
but I am not sure to what you are actually referring.


> >  I just tried again and had the same problem:
> >
> > vagrant@vagrant:~$ uname -a
> > Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30
> > UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> Try pairing that with a newer auditd so that auditctl has the support to
> load
> the rule.
>

I'll check this out. My initial attempts to compile more recent versions
than 2.4.5 on the newer kernel in Ubuntu 14 had issues, but those are
probably personal problems.


> -Steve
>
> > That's a newer version than I have on my Ubuntu 16 VM, which does
> > demonstrate the problem. It's also strange that restarting ssh then makes
> > the accept syscall events show up. Other sshd syscalls show up in auditd
> > before and after the ssh restart.
>

[-- Attachment #1.2: Type: text/html, Size: 4052 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Steve Grubb]

On Saturday, December 3, 2016 2:11:04 AM EST Nathan Cooprider wrote:
> Hi! It sounds like I'm missing something obvious!
> 
> On Fri, Dec 2, 2016 at 5:13 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > Hello,
> > 
> > Addressing a couple obvious things here...
> > 
> > On Friday, December 2, 2016 9:55:17 PM EST Nathan Cooprider wrote:
> > > On Fri, Dec 2, 2016 at 4:09 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > > > On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> > > > > Auditd seems to miss accept syscalls from ssh on Ubuntu 14.
> > > > 
> > > > Its not auditd, the kernel does all the work. Auditd acts a lot like a
> > > > specialized syslog.  :-)
> > > > 
> > > > > I tried versions 2.3.2 and 2.4.5 of the daemon
> > 
> > Support was not added until 2.5.
> 
> Support for what?

Audit by executable. In the example that I gave I showed the syntax for how 
you would audit accept only for sshd. I presume that you are not auditing 
accept across the whole system. What rule are you using to audit accept?


> Auditing the accept syscall? What do you mean by "support?" Those are auditd
> versions that I'm talking about. Is that what you mean? Sorry if I was not
> clear. What did it do with accept syscalls before then? I do not see this
> reflected in the changelog

Let's take a look at how you are auditing it and maybe that will explain a few 
things. Also, does Ubuntu 14 use upstart or systemd? And perhaps for good 
measure include just 1 event when it does work.

-Steve 

> https://people.redhat.com/sgrubb/audit/ChangeLog
> 
> > > > with kernel versions 3.13.0-96
> > 
> > Definitely won't support it.
> 
> Support what?
> 
> > > > > and 4.4.0-47.
> > 
> > The feature landed in 4.3, so 4.4 should have it. However, you need audit
> > 2.5
> > or later to use the kernel feature.
> 
> What feature are you talking about? This sounds like it could be the issue,
> but I am not sure to what you are actually referring.
> 
> > >  I just tried again and had the same problem:
> > > vagrant@vagrant:~$ uname -a
> > > Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24
> > > 19:22:30
> > > UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > Try pairing that with a newer auditd so that auditctl has the support to
> > load
> > the rule.
> 
> I'll check this out. My initial attempts to compile more recent versions
> than 2.4.5 on the newer kernel in Ubuntu 14 had issues, but those are
> probably personal problems.
> 
> > -Steve
> > 
> > > That's a newer version than I have on my Ubuntu 16 VM, which does
> > > demonstrate the problem. It's also strange that restarting ssh then
> > > makes
> > > the accept syscall events show up. Other sshd syscalls show up in auditd
> > > before and after the ssh restart.

Re: Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 8872 bytes --]

On Sat, Dec 3, 2016 at 12:47 PM Steve Grubb <sgrubb@redhat.com> wrote:

> On Saturday, December 3, 2016 2:11:04 AM EST Nathan Cooprider wrote:
> > Hi! It sounds like I'm missing something obvious!
> >
> > On Fri, Dec 2, 2016 at 5:13 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > > Hello,
> > >
> > > Addressing a couple obvious things here...
> > >
> > > On Friday, December 2, 2016 9:55:17 PM EST Nathan Cooprider wrote:
> > > > On Fri, Dec 2, 2016 at 4:09 PM Steve Grubb <sgrubb@redhat.com>
> wrote:
> > > > > On Friday, December 2, 2016 8:43:46 PM EST Nathan Cooprider wrote:
> > > > > > Auditd seems to miss accept syscalls from ssh on Ubuntu 14.
> > > > >
> > > > > Its not auditd, the kernel does all the work. Auditd acts a lot
> like a
> > > > > specialized syslog.  :-)
> > > > >
> > > > > > I tried versions 2.3.2 and 2.4.5 of the daemon
> > >
> > > Support was not added until 2.5.
> >
> > Support for what?
>
> Audit by executable. In the example that I gave I showed the syntax for how
> you would audit accept only for sshd. I presume that you are not auditing
> accept across the whole system. What rule are you using to audit accept?
>

Here's what I have:

vagrant@vagrant:~$ uname -a
Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
vagrant@vagrant:~$ sudo auditctl -l
No rules
vagrant@vagrant:~$ sudo auditctl -a exit,always -F arch=b64 -S accept
vagrant@vagrant:~$ sudo auditctl -l
LIST_RULES: exit,always arch=3221225534 (0xc000003e) syscall=accept

For my case, I am auditing accept syscalls across the whole system. I want
to look for when that syscall occurs in my log and alert on it.


> > Auditing the accept syscall? What do you mean by "support?" Those are
> auditd
> > versions that I'm talking about. Is that what you mean? Sorry if I was
> not
> > clear. What did it do with accept syscalls before then? I do not see this
> > reflected in the changelog
>
> Let's take a look at how you are auditing it and maybe that will explain a
> few
> things. Also, does Ubuntu 14 use upstart or systemd? And perhaps for good
> measure include just 1 event when it does work.
>

Ubuntu 14 uses upstart. Specifically, my VM is using upstart 1.12.1.

I should note that I tried this on CentOS 6 this morning and it did not
have this issue.

Here's the output from audit for my test, with <> tags indicating when
something happened in another terminal:

vagrant@vagrant:~$ sudo auditd -f
Config file /etc/audit/auditd.conf opened for parsing
log_file_parser called with: /var/log/audit/audit.log
log_format_parser called with: RAW
log_group_parser called with: root
priority_boost_parser called with: 4
flush_parser called with: INCREMENTAL
freq_parser called with: 20
num_logs_parser called with: 5
qos_parser called with: lossy
dispatch_parser called with: /sbin/audispd
name_format_parser called with: NONE
max_log_size_parser called with: 6
max_log_size_action_parser called with: ROTATE
space_left_parser called with: 75
space_action_parser called with: SYSLOG
action_mail_acct_parser called with: root
admin_space_left_parser called with: 50
admin_space_left_action_parser called with: SUSPEND
disk_full_action_parser called with: SUSPEND
disk_error_action_parser called with: SUSPEND
tcp_listen_queue_parser called with: 5
Listener support is not enabled, ignoring value at line 26
tcp_max_per_addr_parser called with: 1
Listener support is not enabled, ignoring value at line 27
tcp_client_max_idle_parser called with: 0
Listener support is not enabled, ignoring value at line 29
enable_krb5_parser called with: no
krb5_principal_parser called with: auditd
Started dispatcher: /sbin/audispd pid: 20106
type=DAEMON_START msg=audit(1480954770.929:9566): auditd start, ver=2.3.2
format=raw kernel=4.4.0-51-generic auid=1000 pid=20104 subj=unconfined
 res=success
config_manager init complete
Init complete, auditd 2.3.2 listening for events (startup state enable)
<An ssh log in before restarting ssh>
type=USER_LOGIN msg=audit(1480954826.056:416): pid=20108 uid=0
auid=4294967295 ses=4294967295 msg='op=login acct="vagrant"
exe="/usr/sbin/sshd" hostname=? addr=10.0.2.2 terminal=sshd res=failed'
type=USER_ACCT msg=audit(1480954826.064:417): pid=20108 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct="vagrant"
exe="/usr/sbin/sshd" hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh
res=success'
type=CRED_ACQ msg=audit(1480954826.068:418): pid=20108 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred acct="vagrant"
exe="/usr/sbin/sshd" hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh
res=success'
type=LOGIN msg=audit(1480954826.068:419): pid=20108 uid=0
old-auid=4294967295 auid=1000 old-ses=4294967295 ses=6 res=1
type=USER_START msg=audit(1480954826.404:420): pid=20108 uid=0 auid=1000
ses=6 msg='op=PAM:session_open acct="vagrant" exe="/usr/sbin/sshd"
hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh res=success'
type=CRED_ACQ msg=audit(1480954826.404:421): pid=20175 uid=0 auid=1000
ses=6 msg='op=PAM:setcred acct="vagrant" exe="/usr/sbin/sshd"
hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh res=success'
type=USER_LOGIN msg=audit(1480954826.404:422): pid=20108 uid=0 auid=1000
ses=6 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=10.0.2.2
addr=10.0.2.2 terminal=/dev/pts/1 res=success'
<Restarting ssh>
type=USER_START msg=audit(1480954850.688:423): pid=20190 uid=0 auid=1000
ses=6 msg='op=PAM:session_open acct="root" exe="/usr/bin/sudo" hostname=?
addr=? terminal=/dev/pts/1 res=success'
type=USER_END msg=audit(1480954850.720:424): pid=20190 uid=0 auid=1000
ses=6 msg='op=PAM:session_close acct="root" exe="/usr/bin/sudo" hostname=?
addr=? terminal=/dev/pts/1 res=success'
<After restarting ssh, so the accept event comes in now when I log in>
type=SYSCALL msg=audit(1480954895.508:425): arch=c000003e *syscall=43*
success=yes exit=5 a0=3 a1=7ffd736315c0 a2=7ffd736315ac a3=0 items=0 ppid=1
pid=20204 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
fsgid=0 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
key=(null)
type=SOCKADDR msg=audit(1480954895.508:425):
saddr=0200CE760A0002020000000000000000
type=UNKNOWN[1327] msg=audit(1480954895.508:425):
proctitle=2F7573722F7362696E2F73736864002D44
type=USER_LOGIN msg=audit(1480954895.524:426): pid=20206 uid=0
auid=4294967295 ses=4294967295 msg='op=login acct="vagrant"
exe="/usr/sbin/sshd" hostname=? addr=10.0.2.2 terminal=sshd res=failed'
type=USER_ACCT msg=audit(1480954895.532:427): pid=20206 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct="vagrant"
exe="/usr/sbin/sshd" hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh
res=success'
type=CRED_ACQ msg=audit(1480954895.536:428): pid=20206 uid=0
auid=4294967295 ses=4294967295 msg='op=PAM:setcred acct="vagrant"
exe="/usr/sbin/sshd" hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh
res=success'
type=LOGIN msg=audit(1480954895.536:429): pid=20206 uid=0
old-auid=4294967295 auid=1000 old-ses=4294967295 ses=7 res=1
type=USER_START msg=audit(1480954895.772:430): pid=20206 uid=0 auid=1000
ses=7 msg='op=PAM:session_open acct="vagrant" exe="/usr/sbin/sshd"
hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh res=success'
type=CRED_ACQ msg=audit(1480954895.772:431): pid=20270 uid=0 auid=1000
ses=7 msg='op=PAM:setcred acct="vagrant" exe="/usr/sbin/sshd"
hostname=10.0.2.2 addr=10.0.2.2 terminal=ssh res=success'
type=USER_LOGIN msg=audit(1480954895.772:432): pid=20206 uid=0 auid=1000
ses=7 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=10.0.2.2
addr=10.0.2.2 terminal=/dev/pts/3 res=success'

-Steve
>
> > https://people.redhat.com/sgrubb/audit/ChangeLog
> >
> > > > > with kernel versions 3.13.0-96
> > >
> > > Definitely won't support it.
> >
> > Support what?
> >
> > > > > > and 4.4.0-47.
> > >
> > > The feature landed in 4.3, so 4.4 should have it. However, you need
> audit
> > > 2.5
> > > or later to use the kernel feature.
> >
> > What feature are you talking about? This sounds like it could be the
> issue,
> > but I am not sure to what you are actually referring.
> >
> > > >  I just tried again and had the same problem:
> > > > vagrant@vagrant:~$ uname -a
> > > > Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24
> > > > 19:22:30
> > > > UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > Try pairing that with a newer auditd so that auditctl has the support
> to
> > > load
> > > the rule.
> >
> > I'll check this out. My initial attempts to compile more recent versions
> > than 2.4.5 on the newer kernel in Ubuntu 14 had issues, but those are
> > probably personal problems.
> >
> > > -Steve
> > >
> > > > That's a newer version than I have on my Ubuntu 16 VM, which does
> > > > demonstrate the problem. It's also strange that restarting ssh then
> > > > makes
> > > > the accept syscall events show up. Other sshd syscalls show up in
> auditd
> > > > before and after the ssh restart.
>

[-- Attachment #1.2: Type: text/html, Size: 12820 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Steve Grubb]

On Monday, December 5, 2016 4:42:14 PM EST Nathan Cooprider wrote:
> On Sat, Dec 3, 2016 at 12:47 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > > > Support was not added until 2.5.
> > > 
> > > Support for what?
> > 
> > Audit by executable. In the example that I gave I showed the syntax for
> > how you would audit accept only for sshd. I presume that you are not
> > auditing accept across the whole system. What rule are you using to audit
> > accept?
> 
> Here's what I have:
> 
> vagrant@vagrant:~$ uname -a
> Linux vagrant 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30
> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> vagrant@vagrant:~$ sudo auditctl -l
> No rules
> vagrant@vagrant:~$ sudo auditctl -a exit,always -F arch=b64 -S accept
> vagrant@vagrant:~$ sudo auditctl -l
> LIST_RULES: exit,always arch=3221225534 (0xc000003e) syscall=accept
> 
> For my case, I am auditing accept syscalls across the whole system. I want
> to look for when that syscall occurs in my log and alert on it.

OK. I was thinking that perhaps you had the rule qualified with -F auid>=500 -F 
auid!=-1 to detect user originating events and the restart (because its 
upstart) would put your auid into sshd's and then you were successful in 
auditing. If the above rule is in fact what you are auditing with, and you 
have auidit=1 on your grub kernel boot commandline, then I am out of guesses. 
Sounds like a problem unique to your kernel since you have found kernels that 
work fine.

-Steve

Re: Auditd misses accept syscalls from sshd

[Paul Moore]

On Fri, Dec 2, 2016 at 3:43 PM, Nathan Cooprider
<ncooprider@yankeehacker.com> wrote:
> Auditd seems to miss accept syscalls from ssh on Ubuntu 14. I tried versions
> 2.3.2 and 2.4.5 of the daemon with kernel versions 3.13.0-96 and 4.4.0-47.
> In all cases the accept syscall (43) failed to show up until after I
> restarted the ssh daemon. It's especially weird because I don't see this
> problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I am seeing this or
> where to look?
>
> I found a similar question in the archives, but it seems to do with the
> architecture size and not OS versions:
> https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html
>
> I also posted this question on Stack Overflow:
> http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-have-inconsistent-behavior-in-linux-audit-framework

I'm not really very aware of what Ubuntu is doing wrt to their default
audit configuration, but this really sounds like you need to add
'audit=1' to the kernel command line.

-- 
paul moore
www.paul-moore.com

Re: Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 1802 bytes --]

On Fri, Dec 2, 2016 at 4:26 PM Paul Moore <paul@paul-moore.com> wrote:

> On Fri, Dec 2, 2016 at 3:43 PM, Nathan Cooprider
> <ncooprider@yankeehacker.com> wrote:
> > Auditd seems to miss accept syscalls from ssh on Ubuntu 14. I tried
> versions
> > 2.3.2 and 2.4.5 of the daemon with kernel versions 3.13.0-96 and
> 4.4.0-47.
> > In all cases the accept syscall (43) failed to show up until after I
> > restarted the ssh daemon. It's especially weird because I don't see this
> > problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I am seeing this
> or
> > where to look?
> >
> > I found a similar question in the archives, but it seems to do with the
> > architecture size and not OS versions:
> > https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html
> >
> > I also posted this question on Stack Overflow:
> >
> http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-have-inconsistent-behavior-in-linux-audit-framework
>
> I'm not really very aware of what Ubuntu is doing wrt to their default
> audit configuration, but this really sounds like you need to add
> 'audit=1' to the kernel command line.
>
> --
> paul moore
> www.paul-moore.com


Thanks for the suggestion. I'm getting other audit events from sshd without
restarting ssh. It's just the accept syscalls that do not show up until
after I restart ssh:

type=SYSCALL msg=audit(1480714641.465:54): arch=c000003e syscall=43
success=yes exit=5 a0=3 a1=7ffce3b031b0 a2=7ffce3b0319c a3=0 items=0 ppid=1
pid=2602 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
fsgid=0 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
key=(null)

I think that indicates the kernel is sending up audit messages. My question
is why the above message fails to come up until after I've restarted ssh.

[-- Attachment #1.2: Type: text/html, Size: 3185 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Paul Moore]

On Fri, Dec 2, 2016 at 4:42 PM, Nathan Cooprider
<ncooprider@yankeehacker.com> wrote:
> On Fri, Dec 2, 2016 at 4:26 PM Paul Moore <paul@paul-moore.com> wrote:
>>
>> On Fri, Dec 2, 2016 at 3:43 PM, Nathan Cooprider
>> <ncooprider@yankeehacker.com> wrote:
>> > Auditd seems to miss accept syscalls from ssh on Ubuntu 14. I tried
>> > versions
>> > 2.3.2 and 2.4.5 of the daemon with kernel versions 3.13.0-96 and
>> > 4.4.0-47.
>> > In all cases the accept syscall (43) failed to show up until after I
>> > restarted the ssh daemon. It's especially weird because I don't see this
>> > problem on Ubuntu 16 (4.4.0-38). Any thoughts about why I am seeing this
>> > or
>> > where to look?
>> >
>> > I found a similar question in the archives, but it seems to do with the
>> > architecture size and not OS versions:
>> > https://www.redhat.com/archives/linux-audit/2015-January/msg00060.html
>> >
>> > I also posted this question on Stack Overflow:
>> >
>> > http://stackoverflow.com/questions/40940225/why-does-sshd-accept-syscall-have-inconsistent-behavior-in-linux-audit-framework
>>
>> I'm not really very aware of what Ubuntu is doing wrt to their default
>> audit configuration, but this really sounds like you need to add
>> 'audit=1' to the kernel command line.
>
> Thanks for the suggestion. I'm getting other audit events from sshd without
> restarting ssh. It's just the accept syscalls that do not show up until
> after I restart ssh:
>
> type=SYSCALL msg=audit(1480714641.465:54): arch=c000003e syscall=43
> success=yes exit=5 a0=3 a1=7ffce3b031b0 a2=7ffce3b0319c a3=0 items=0 ppid=1
> pid=2602 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
> fsgid=0 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
> key=(null)
>
> I think that indicates the kernel is sending up audit messages. My question
> is why the above message fails to come up until after I've restarted ssh.

If you haven't already, I would suggest opening an issue with
Ubuntu/Canonical; I'm not aware of any issues in current kernels that
would cause this and your testing on more modern Ubuntu flavors would
indicate current Ubuntu releases work correctly.

-- 
paul moore
www.paul-moore.com

Re: Auditd misses accept syscalls from sshd

[Hassan Sultan]

[-- Attachment #1.1: Type: text/plain, Size: 1281 bytes --]

On Fri, 02 Dec 2016 13:42:02 -0800, Nathan Cooprider  
<ncooprider@yankeehacker.com> wrote:
>
>
> Thanks for the suggestion. I'm getting other audit events from sshd  
> without restarting ssh. It's just the accept syscalls that do not show  
> up until after I >restart ssh:
>
> type=SYSCALL msg=audit(1480714641.465:54): arch=c000003e syscall=43  
> success=yes exit=5 a0=3 a1=7ffce3b031b0 a2=7ffce3b0319c a3=0 items=0  
> >ppid=1 pid=2602 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0  
> egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sshd"  
> exe="/usr/sbin/>sshd" key=(null)
>
> I think that indicates the kernel is sending up audit messages. My  
> question is why the above message fails to come up until after I've  
> restarted ssh.
>

(I was the person having that issue almost 2 years ago)

I never fully investigated it, but came up with one theory explaining it :

- accept is a blocking syscall , it might be that sshd started and the  
syscall was initiated before the audit rule was loaded. This would explain  
why you see the event when restarting sshd.

Don't use the tcp connection time to evaluate whether the auditing worked  
properly, but rather when the initial accept call was made, which  
basically amounts to when sshd is started.


Hassan

[-- Attachment #1.2.1: Type: text/html, Size: 1781 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Nathan Cooprider]

[-- Attachment #1.1: Type: text/plain, Size: 1714 bytes --]

On Fri, Dec 2, 2016 at 6:44 PM Hassan Sultan <hsultan@thefroid.net> wrote:

> On Fri, 02 Dec 2016 13:42:02 -0800, Nathan Cooprider <
> ncooprider@yankeehacker.com> wrote:
>
>
>
> Thanks for the suggestion. I'm getting other audit events from sshd
> without restarting ssh. It's just the accept syscalls that do not show up
> until after I restart ssh:
>
> type=SYSCALL msg=audit(1480714641.465:54): arch=c000003e syscall=43
> success=yes exit=5 a0=3 a1=7ffce3b031b0 a2=7ffce3b0319c a3=0 items=0 ppid=1
> pid=2602 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
> fsgid=0 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
> key=(null)
>
> I think that indicates the kernel is sending up audit messages. My
> question is why the above message fails to come up until after I've
> restarted ssh.
>
>
> (I was the person having that issue almost 2 years ago)
>
> I never fully investigated it, but came up with one theory explaining it :
>
> - accept is a blocking syscall , it might be that sshd started and the
> syscall was initiated before the audit rule was loaded. This would explain
> why you see the event when restarting sshd.
>
> Don't use the tcp connection time to evaluate whether the auditing worked
> properly, but rather when the initial accept call was made, which basically
> amounts to when sshd is started.
>
>
> Hassan
>

After I restart the ssh service I get an accept call every time I try to
ssh into the box, whereas before I restart the ssh service I get no events.
That does not conform with my mental model of how this would work, but I
wouldn't be asking this question if my mental model was correct!

Thanks for the hypothesis and tip, plus the original question!

[-- Attachment #1.2: Type: text/html, Size: 3200 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd misses accept syscalls from sshd

[Steve Grubb]

On Friday, December 2, 2016 3:44:35 PM EST Hassan Sultan wrote:
> On Fri, 02 Dec 2016 13:42:02 -0800, Nathan Cooprider
> 
> <ncooprider@yankeehacker.com> wrote:
> > Thanks for the suggestion. I'm getting other audit events from sshd
> > without restarting ssh. It's just the accept syscalls that do not show
> > up until after I >restart ssh:
> > 
> > type=SYSCALL msg=audit(1480714641.465:54): arch=c000003e syscall=43
> > success=yes exit=5 a0=3 a1=7ffce3b031b0 a2=7ffce3b0319c a3=0 items=0
> > 
> > >ppid=1 pid=2602 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0
> > 
> > egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sshd"
> > exe="/usr/sbin/>sshd" key=(null)
> > 
> > I think that indicates the kernel is sending up audit messages. My
> > question is why the above message fails to come up until after I've
> > restarted ssh.
> 
> (I was the person having that issue almost 2 years ago)
> 
> I never fully investigated it, but came up with one theory explaining it :
> 
> - accept is a blocking syscall , it might be that sshd started and the
> syscall was initiated before the audit rule was loaded. This would explain
> why you see the event when restarting sshd.

Because accept can block, sockets are almost always turned non-blocking when 
setting up the listen queue. Then you wait in select/poll/epoll for it to be 
ready to accept. And inspection of sshd's code and some stracing shows this to 
be the case.
 
> Don't use the tcp connection time to evaluate whether the auditing worked
> properly, but rather when the initial accept call was made, which
> basically amounts to when sshd is started.

On most systems, auditd starts just before or after syslog. Networking daemons 
usually come later in the boot process.

-Steve