Early processes (daemons) do not report audit events

Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 1457 bytes --]

Dear all,

We are developing custom user space audit agent to gather system wide system
call trace. While experimenting with various programs, we found out that
processes (daemons) that started early (along with the system bootstrapping) do
not report any audit events at all. These processes typically fall into PID
range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.

1. Reboot the system

2. Add and enable audit events
   # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
          -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
          -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
          -S listen -S socket -S socketpair
   # /sbin/auditctl -e1 -b 102400

3. Connect to the system via ssh
    Audit messages generated only from child processes and none are seen from
    the original daemon.

4. Restart sshd 
    # restart ssh

5. Connect again to the system via ssh
   Now, we see audit messages from both parent and child processes.

I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
version 3.13.0-58-generic).

I first wonder whether this is intended behavior of audit framework or
not. If it is intended, I also want to know how can we configure auditd
differently to capture system calls from all processes. 

Thanks a lot for your help in advance!

Regards, Kangkook


[-- Attachment #1.2: Type: text/html, Size: 2650 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 2268 bytes --]

Hi all,

I debugged a bit further to identify distributions that are affected by the issue.
I repeated the same experiment with sshd from 3 more distributions.

CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced

After all, Ubuntu family are affected by the issue and I could confirm that results are inconsistent 
across two different distribution families. 

If you can let us know how can we workaround the issue, it will be a great help.

Regards, Kangkook


> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
> 
> Dear all,
> 
> We are developing custom user space audit agent to gather system wide system
> call trace. While experimenting with various programs, we found out that
> processes (daemons) that started early (along with the system bootstrapping) do
> not report any audit events at all. These processes typically fall into PID
> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
> 
> 1. Reboot the system
> 
> 2. Add and enable audit events
>    # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>           -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>           -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>           -S listen -S socket -S socketpair
>    # /sbin/auditctl -e1 -b 102400
> 
> 3. Connect to the system via ssh
>     Audit messages generated only from child processes and none are seen from
>     the original daemon.
> 
> 4. Restart sshd 
>     # restart ssh
> 
> 5. Connect again to the system via ssh
>    Now, we see audit messages from both parent and child processes.
> 
> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
> version 3.13.0-58-generic).
> 
> I first wonder whether this is intended behavior of audit framework or
> not. If it is intended, I also want to know how can we configure auditd
> differently to capture system calls from all processes. 
> 
> Thanks a lot for your help in advance!
> 
> Regards, Kangkook
> 


[-- Attachment #1.2: Type: text/html, Size: 4184 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Early processes (daemons) do not report audit events

[Richard Guy Briggs]

On 15/09/10, Kangkook Jee wrote:
> Hi all,
> 
> I debugged a bit further to identify distributions that are affected by the issue.
> I repeated the same experiment with sshd from 3 more distributions.
> 
> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced

For each of these examples, what is the value of the kernel command line
"audit=<value>" if it is even present?  It is possible that the CentOS
examples include "audit=1" while Ubuntu omits the line.  "cat
/proc/cmdline" should tell you the answer.

> After all, Ubuntu family are affected by the issue and I could confirm
> that results are inconsistent across two different distribution
> families. 

I would be curious what your results are with a recent Debian and with a
recent Fedora.

> If you can let us know how can we workaround the issue, it will be a great help.
> 
> Regards, Kangkook
> 
> 
> > On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
> > 
> > Dear all,
> > 
> > We are developing custom user space audit agent to gather system wide system
> > call trace. While experimenting with various programs, we found out that
> > processes (daemons) that started early (along with the system bootstrapping) do
> > not report any audit events at all. These processes typically fall into PID
> > range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
> > 
> > 1. Reboot the system
> > 
> > 2. Add and enable audit events
> >    # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
> >           -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
> >           -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
> >           -S listen -S socket -S socketpair
> >    # /sbin/auditctl -e1 -b 102400
> > 
> > 3. Connect to the system via ssh
> >     Audit messages generated only from child processes and none are seen from
> >     the original daemon.
> > 
> > 4. Restart sshd 
> >     # restart ssh
> > 
> > 5. Connect again to the system via ssh
> >    Now, we see audit messages from both parent and child processes.
> > 
> > I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
> > version 3.13.0-58-generic).
> > 
> > I first wonder whether this is intended behavior of audit framework or
> > not. If it is intended, I also want to know how can we configure auditd
> > differently to capture system calls from all processes. 
> > 
> > Thanks a lot for your help in advance!
> > 
> > Regards, Kangkook
> > 
> 

> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit


- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 4553 bytes --]

Dear Richard,

Thanks a lot for your reply. Please refer to my inlined replies. 

Btw, we can stably re-produce the issue with most of Ubuntu distribution
families. I think we can ship you a VM image (via Amazon S3 or something) so as
to save your time debugging it.

Thanks a lot for your help again!

Regards, Kangkook

> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> 
> On 15/09/10, Kangkook Jee wrote:
>> Hi all,
>> 
>> I debugged a bit further to identify distributions that are affected by the issue.
>> I repeated the same experiment with sshd from 3 more distributions.
>> 
>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
> 
> For each of these examples, what is the value of the kernel command line
> "audit=<value>" if it is even present?  It is possible that the CentOS
> examples include "audit=1" while Ubuntu omits the line.  "cat
> /proc/cmdline" should tell you the answer.
> 

For all testcases, I used the same auditctl options to set up rules and I didn’t set any audit=<value> entries.
And options as same as I presented from the previous email. 

>>>   # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>          -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>          -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>          -S listen -S socket -S socketpair

And I personally don’t think audit=<value> would become an issue here, since
audit framework began to capture events from sshd daemon as soon as we restarted
it without making any changes. sshd was running with the same uid (or auid) and
the same permissions. 

>> After all, Ubuntu family are affected by the issue and I could confirm
>> that results are inconsistent across two different distribution
>> families. 
> 
> I would be curious what your results are with a recent Debian and with a
> recent Fedora.
> 

I will redo the same experiment with those distribution and let you know how it goes. 


>> If you can let us know how can we workaround the issue, it will be a great help.
>> 
>> Regards, Kangkook
>> 
>> 
>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
>>> 
>>> Dear all,
>>> 
>>> We are developing custom user space audit agent to gather system wide system
>>> call trace. While experimenting with various programs, we found out that
>>> processes (daemons) that started early (along with the system bootstrapping) do
>>> not report any audit events at all. These processes typically fall into PID
>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
>>> 
>>> 1. Reboot the system
>>> 
>>> 2. Add and enable audit events
>>>   # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>          -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>          -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>          -S listen -S socket -S socketpair
>>>   # /sbin/auditctl -e1 -b 102400
>>> 
>>> 3. Connect to the system via ssh
>>>    Audit messages generated only from child processes and none are seen from
>>>    the original daemon.
>>> 
>>> 4. Restart sshd 
>>>    # restart ssh
>>> 
>>> 5. Connect again to the system via ssh
>>>   Now, we see audit messages from both parent and child processes.
>>> 
>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
>>> version 3.13.0-58-generic).
>>> 
>>> I first wonder whether this is intended behavior of audit framework or
>>> not. If it is intended, I also want to know how can we configure auditd
>>> differently to capture system calls from all processes. 
>>> 
>>> Thanks a lot for your help in advance!
>>> 
>>> Regards, Kangkook
>>> 
>> 
> 
>> --
>> Linux-audit mailing list
>> Linux-audit@redhat.com <mailto:Linux-audit@redhat.com>
>> https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit>
> 
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545


[-- Attachment #1.2: Type: text/html, Size: 20605 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 4339 bytes --]

From the previous reply, I think I misunderstood your question regarding kernel command line. 
Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 

Ubuntu 14.04 (64-bit): 
    BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7

Ubuntu 12.04 (64-bit): 
    BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash

CentOS 7 (64-bit): 
    BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8

CentOS 6 (64-bit): 
    ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

I don’t see any audit=<value> entries from all examples above. 

/Kangkook

> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> 
> On 15/09/10, Kangkook Jee wrote:
>> Hi all,
>> 
>> I debugged a bit further to identify distributions that are affected by the issue.
>> I repeated the same experiment with sshd from 3 more distributions.
>> 
>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
> 
> For each of these examples, what is the value of the kernel command line
> "audit=<value>" if it is even present?  It is possible that the CentOS
> examples include "audit=1" while Ubuntu omits the line.  "cat
> /proc/cmdline" should tell you the answer.
> 
>> After all, Ubuntu family are affected by the issue and I could confirm
>> that results are inconsistent across two different distribution
>> families. 
> 
> I would be curious what your results are with a recent Debian and with a
> recent Fedora.
> 
>> If you can let us know how can we workaround the issue, it will be a great help.
>> 
>> Regards, Kangkook
>> 
>> 
>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
>>> 
>>> Dear all,
>>> 
>>> We are developing custom user space audit agent to gather system wide system
>>> call trace. While experimenting with various programs, we found out that
>>> processes (daemons) that started early (along with the system bootstrapping) do
>>> not report any audit events at all. These processes typically fall into PID
>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
>>> 
>>> 1. Reboot the system
>>> 
>>> 2. Add and enable audit events
>>>   # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>          -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>          -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>          -S listen -S socket -S socketpair
>>>   # /sbin/auditctl -e1 -b 102400
>>> 
>>> 3. Connect to the system via ssh
>>>    Audit messages generated only from child processes and none are seen from
>>>    the original daemon.
>>> 
>>> 4. Restart sshd 
>>>    # restart ssh
>>> 
>>> 5. Connect again to the system via ssh
>>>   Now, we see audit messages from both parent and child processes.
>>> 
>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
>>> version 3.13.0-58-generic).
>>> 
>>> I first wonder whether this is intended behavior of audit framework or
>>> not. If it is intended, I also want to know how can we configure auditd
>>> differently to capture system calls from all processes. 
>>> 
>>> Thanks a lot for your help in advance!
>>> 
>>> Regards, Kangkook
>>> 
>> 
> 
>> --
>> Linux-audit mailing list
>> Linux-audit@redhat.com <mailto:Linux-audit@redhat.com>
>> https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit>
> 
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545


[-- Attachment #1.2: Type: text/html, Size: 19940 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Early processes (daemons) do not report audit events

[Richard Guy Briggs]

On 15/09/11, Kangkook Jee wrote:
> From the previous reply, I think I misunderstood your question regarding kernel command line. 
> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 
> 
> Ubuntu 14.04 (64-bit): 
>     BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7
> 
> Ubuntu 12.04 (64-bit): 
>     BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash
> 
> CentOS 7 (64-bit): 
>     BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8
> 
> CentOS 6 (64-bit): 
>     ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
> 
> I don’t see any audit=<value> entries from all examples above. 

Yes, this is what I was seeking from you.  And you are correct, none of
them have audit=1 as I was hoping from at least CentOS.  There is a
chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
think that is a pretty small chance...

I'll have to look at this closer...  But any Debian and Fedora data
points that you can provide would certainly be useful.

> /Kangkook
> 
> > On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > 
> > On 15/09/10, Kangkook Jee wrote:
> >> Hi all,
> >> 
> >> I debugged a bit further to identify distributions that are affected by the issue.
> >> I repeated the same experiment with sshd from 3 more distributions.
> >> 
> >> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
> >> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
> >> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
> > 
> > For each of these examples, what is the value of the kernel command line
> > "audit=<value>" if it is even present?  It is possible that the CentOS
> > examples include "audit=1" while Ubuntu omits the line.  "cat
> > /proc/cmdline" should tell you the answer.
> > 
> >> After all, Ubuntu family are affected by the issue and I could confirm
> >> that results are inconsistent across two different distribution
> >> families. 
> > 
> > I would be curious what your results are with a recent Debian and with a
> > recent Fedora.
> > 
> >> If you can let us know how can we workaround the issue, it will be a great help.
> >> 
> >> Regards, Kangkook
> >> 
> >> 
> >>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
> >>> 
> >>> Dear all,
> >>> 
> >>> We are developing custom user space audit agent to gather system wide system
> >>> call trace. While experimenting with various programs, we found out that
> >>> processes (daemons) that started early (along with the system bootstrapping) do
> >>> not report any audit events at all. These processes typically fall into PID
> >>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
> >>> 
> >>> 1. Reboot the system
> >>> 
> >>> 2. Add and enable audit events
> >>>   # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
> >>>          -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
> >>>          -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
> >>>          -S listen -S socket -S socketpair
> >>>   # /sbin/auditctl -e1 -b 102400
> >>> 
> >>> 3. Connect to the system via ssh
> >>>    Audit messages generated only from child processes and none are seen from
> >>>    the original daemon.
> >>> 
> >>> 4. Restart sshd 
> >>>    # restart ssh
> >>> 
> >>> 5. Connect again to the system via ssh
> >>>   Now, we see audit messages from both parent and child processes.
> >>> 
> >>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
> >>> version 3.13.0-58-generic).
> >>> 
> >>> I first wonder whether this is intended behavior of audit framework or
> >>> not. If it is intended, I also want to know how can we configure auditd
> >>> differently to capture system calls from all processes. 
> >>> 
> >>> Thanks a lot for your help in advance!
> >>> 
> >>> Regards, Kangkook
> >>> 
> >> 
> > 
> >> --
> >> Linux-audit mailing list
> >> Linux-audit@redhat.com <mailto:Linux-audit@redhat.com>
> >> https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit>
> > 
> > 
> > - RGB
> > 
> > --
> > Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> > Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> > Remote, Ottawa, Canada
> > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
> 

- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 6011 bytes --]

Hi Richard,

I also did the same experiment for the latest distributions of Fedora core and Debian and here’s the results.

Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced.
Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced

Btw, Burn Alting (burn@swtf.dyndns.org) suggested me to append audit=1 to kernel flag. I added the option to boot-loader (grub) and problem went away. 

Regards, Kangkook


> On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> 
> On 15/09/11, Kangkook Jee wrote:
>> From the previous reply, I think I misunderstood your question regarding kernel command line. 
>> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 
>> 
>> Ubuntu 14.04 (64-bit): 
>>    BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7
>> 
>> Ubuntu 12.04 (64-bit): 
>>    BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash
>> 
>> CentOS 7 (64-bit): 
>>    BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8
>> 
>> CentOS 6 (64-bit): 
>>    ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>> 
>> I don’t see any audit=<value> entries from all examples above. 
> 
> Yes, this is what I was seeking from you.  And you are correct, none of
> them have audit=1 as I was hoping from at least CentOS.  There is a
> chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
> think that is a pretty small chance...
> 
> I'll have to look at this closer...  But any Debian and Fedora data
> points that you can provide would certainly be useful.
> 
>> /Kangkook
>> 
>>> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
>>> 
>>> On 15/09/10, Kangkook Jee wrote:
>>>> Hi all,
>>>> 
>>>> I debugged a bit further to identify distributions that are affected by the issue.
>>>> I repeated the same experiment with sshd from 3 more distributions.
>>>> 
>>>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
>>>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
>>>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
>>> 
>>> For each of these examples, what is the value of the kernel command line
>>> "audit=<value>" if it is even present?  It is possible that the CentOS
>>> examples include "audit=1" while Ubuntu omits the line.  "cat
>>> /proc/cmdline" should tell you the answer.
>>> 
>>>> After all, Ubuntu family are affected by the issue and I could confirm
>>>> that results are inconsistent across two different distribution
>>>> families. 
>>> 
>>> I would be curious what your results are with a recent Debian and with a
>>> recent Fedora.
>>> 
>>>> If you can let us know how can we workaround the issue, it will be a great help.
>>>> 
>>>> Regards, Kangkook
>>>> 
>>>> 
>>>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
>>>>> 
>>>>> Dear all,
>>>>> 
>>>>> We are developing custom user space audit agent to gather system wide system
>>>>> call trace. While experimenting with various programs, we found out that
>>>>> processes (daemons) that started early (along with the system bootstrapping) do
>>>>> not report any audit events at all. These processes typically fall into PID
>>>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
>>>>> 
>>>>> 1. Reboot the system
>>>>> 
>>>>> 2. Add and enable audit events
>>>>>  # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>>>         -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>>>         -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>>>         -S listen -S socket -S socketpair
>>>>>  # /sbin/auditctl -e1 -b 102400
>>>>> 
>>>>> 3. Connect to the system via ssh
>>>>>   Audit messages generated only from child processes and none are seen from
>>>>>   the original daemon.
>>>>> 
>>>>> 4. Restart sshd 
>>>>>   # restart ssh
>>>>> 
>>>>> 5. Connect again to the system via ssh
>>>>>  Now, we see audit messages from both parent and child processes.
>>>>> 
>>>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
>>>>> version 3.13.0-58-generic).
>>>>> 
>>>>> I first wonder whether this is intended behavior of audit framework or
>>>>> not. If it is intended, I also want to know how can we configure auditd
>>>>> differently to capture system calls from all processes. 
>>>>> 
>>>>> Thanks a lot for your help in advance!
>>>>> 
>>>>> Regards, Kangkook
>>>>> 
>>>> 
>>> 
>>>> --
>>>> Linux-audit mailing list
>>>> Linux-audit@redhat.com <mailto:Linux-audit@redhat.com> <mailto:Linux-audit@redhat.com <mailto:Linux-audit@redhat.com>>
>>>> https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit> <https://www.redhat.com/mailman/listinfo/linux-audit <https://www.redhat.com/mailman/listinfo/linux-audit>>
>>> 
>>> 
>>> - RGB
>>> 
>>> --
>>> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com> <mailto:rbriggs@redhat.com <mailto:rbriggs@redhat.com>>>
>>> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
>>> Remote, Ottawa, Canada
>>> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545
>> 
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545


[-- Attachment #1.2: Type: text/html, Size: 20857 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Early processes (daemons) do not report audit events

[Richard Guy Briggs]

On 15/09/11, Kangkook Jee wrote:
> Hi Richard,

Hi Kangkook,

> I also did the same experiment for the latest distributions of Fedora
> core and Debian and here’s the results.
> 
> Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced.
> Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced
> 
> Btw, Burn Alting (burn@swtf.dyndns.org) suggested me to append audit=1
> to kernel flag. I added the option to boot-loader (grub) and problem
> went away. 

On all systems?  This is expected behaviour.  Sorry I was not more
explicit in asking you to test that.  I guess it was implied by asking
what the settings for the kernel command line were.

Now the surprising bit is that CentOS does not demonstrate the problem
without audit=1 in the command line, which leads me to wonder if they
have set "u32             audit_enabled = 1;" around line 83 of
kernel/audit.c in their kernel source.  It would surprise me if they
did, but it would not be completely unreasonable.

> Regards, Kangkook
> 
> > On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > On 15/09/11, Kangkook Jee wrote:
> >> From the previous reply, I think I misunderstood your question regarding kernel command line. 
> >> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 
> >> 
> >> Ubuntu 14.04 (64-bit): 
> >>    BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7
> >> 
> >> Ubuntu 12.04 (64-bit): 
> >>    BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash
> >> 
> >> CentOS 7 (64-bit): 
> >>    BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8
> >> 
> >> CentOS 6 (64-bit): 
> >>    ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
> >> 
> >> I don’t see any audit=<value> entries from all examples above. 
> > 
> > Yes, this is what I was seeking from you.  And you are correct, none of
> > them have audit=1 as I was hoping from at least CentOS.  There is a
> > chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
> > think that is a pretty small chance...
> > 
> > I'll have to look at this closer...  But any Debian and Fedora data
> > points that you can provide would certainly be useful.
> > 
> >> /Kangkook
> >> 
> >>> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> >>> 
> >>> On 15/09/10, Kangkook Jee wrote:
> >>>> Hi all,
> >>>> 
> >>>> I debugged a bit further to identify distributions that are affected by the issue.
> >>>> I repeated the same experiment with sshd from 3 more distributions.
> >>>> 
> >>>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
> >>>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
> >>>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
> >>> 
> >>> For each of these examples, what is the value of the kernel command line
> >>> "audit=<value>" if it is even present?  It is possible that the CentOS
> >>> examples include "audit=1" while Ubuntu omits the line.  "cat
> >>> /proc/cmdline" should tell you the answer.
> >>> 
> >>>> After all, Ubuntu family are affected by the issue and I could confirm
> >>>> that results are inconsistent across two different distribution
> >>>> families. 
> >>> 
> >>> I would be curious what your results are with a recent Debian and with a
> >>> recent Fedora.
> >>> 
> >>>> If you can let us know how can we workaround the issue, it will be a great help.
> >>>> 
> >>>> Regards, Kangkook
> >>>> 
> >>>> 
> >>>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com> wrote:
> >>>>> 
> >>>>> Dear all,
> >>>>> 
> >>>>> We are developing custom user space audit agent to gather system wide system
> >>>>> call trace. While experimenting with various programs, we found out that
> >>>>> processes (daemons) that started early (along with the system bootstrapping) do
> >>>>> not report any audit events at all. These processes typically fall into PID
> >>>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
> >>>>> 
> >>>>> 1. Reboot the system
> >>>>> 
> >>>>> 2. Add and enable audit events
> >>>>>  # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
> >>>>>         -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
> >>>>>         -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
> >>>>>         -S listen -S socket -S socketpair
> >>>>>  # /sbin/auditctl -e1 -b 102400
> >>>>> 
> >>>>> 3. Connect to the system via ssh
> >>>>>   Audit messages generated only from child processes and none are seen from
> >>>>>   the original daemon.
> >>>>> 
> >>>>> 4. Restart sshd 
> >>>>>   # restart ssh
> >>>>> 
> >>>>> 5. Connect again to the system via ssh
> >>>>>  Now, we see audit messages from both parent and child processes.
> >>>>> 
> >>>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
> >>>>> version 3.13.0-58-generic).
> >>>>> 
> >>>>> I first wonder whether this is intended behavior of audit framework or
> >>>>> not. If it is intended, I also want to know how can we configure auditd
> >>>>> differently to capture system calls from all processes. 
> >>>>> 
> >>>>> Thanks a lot for your help in advance!
> >>>>> 
> >>>>> Regards, Kangkook
> >>> 
> >>> - RGB
> > 
> > - RGB

- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: Early processes (daemons) do not report audit events

[Kangkook Jee]

[-- Attachment #1.1: Type: text/plain, Size: 7172 bytes --]

Dear Richard,

Thanks a lot for your help. I agree that the inconsistency seems a bit strange but 
I also doubt that different distributions make different changes to kernel/audit.c. This 
may be just a timing issue with regard to  startup orderings. 

I’d like to append Burn’s message that solved my issue for future reference.

>>> 
>>> Kangkook,
>>> 
>>> Perhaps you can re-test, but modify the kernel boot parameters to
>>> include audit=1 as an additional argument.
>>> 
>>> Reading the auditd(8) manual, one sees
>>> 
>>>       A boot param of audit=1 should be added to ensure that all
>>>       processes that run before the audit daemon starts is marked as
>>>       auditable by the kernel. Not doing that will make a few
>>>       processes impossible to properly audit.


Thanks again for your help!

Regards, Kangkook


> On Sep 13, 2015, at 11:58 AM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
> 
> On 15/09/11, Kangkook Jee wrote:
>> Hi Richard,
> 
> Hi Kangkook,
> 
>> I also did the same experiment for the latest distributions of Fedora
>> core and Debian and here’s the results.
>> 
>> Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced.
>> Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced
>> 
>> Btw, Burn Alting (burn@swtf.dyndns.org <mailto:burn@swtf.dyndns.org>) suggested me to append audit=1
>> to kernel flag. I added the option to boot-loader (grub) and problem
>> went away. 
> 
> On all systems?  This is expected behaviour.  Sorry I was not more
> explicit in asking you to test that.  I guess it was implied by asking
> what the settings for the kernel command line were.
> 
> Now the surprising bit is that CentOS does not demonstrate the problem
> without audit=1 in the command line, which leads me to wonder if they
> have set "u32             audit_enabled = 1;" around line 83 of
> kernel/audit.c in their kernel source.  It would surprise me if they
> did, but it would not be completely unreasonable.
> 
>> Regards, Kangkook
>> 
>>> On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
>>> On 15/09/11, Kangkook Jee wrote:
>>>> From the previous reply, I think I misunderstood your question regarding kernel command line. 
>>>> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 
>>>> 
>>>> Ubuntu 14.04 (64-bit): 
>>>>   BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7
>>>> 
>>>> Ubuntu 12.04 (64-bit): 
>>>>   BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash
>>>> 
>>>> CentOS 7 (64-bit): 
>>>>   BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8
>>>> 
>>>> CentOS 6 (64-bit): 
>>>>   ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>>>> 
>>>> I don’t see any audit=<value> entries from all examples above. 
>>> 
>>> Yes, this is what I was seeking from you.  And you are correct, none of
>>> them have audit=1 as I was hoping from at least CentOS.  There is a
>>> chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
>>> think that is a pretty small chance...
>>> 
>>> I'll have to look at this closer...  But any Debian and Fedora data
>>> points that you can provide would certainly be useful.
>>> 
>>>> /Kangkook
>>>> 
>>>>> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
>>>>> 
>>>>> On 15/09/10, Kangkook Jee wrote:
>>>>>> Hi all,
>>>>>> 
>>>>>> I debugged a bit further to identify distributions that are affected by the issue.
>>>>>> I repeated the same experiment with sshd from 3 more distributions.
>>>>>> 
>>>>>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
>>>>>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
>>>>>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
>>>>> 
>>>>> For each of these examples, what is the value of the kernel command line
>>>>> "audit=<value>" if it is even present?  It is possible that the CentOS
>>>>> examples include "audit=1" while Ubuntu omits the line.  "cat
>>>>> /proc/cmdline" should tell you the answer.
>>>>> 
>>>>>> After all, Ubuntu family are affected by the issue and I could confirm
>>>>>> that results are inconsistent across two different distribution
>>>>>> families. 
>>>>> 
>>>>> I would be curious what your results are with a recent Debian and with a
>>>>> recent Fedora.
>>>>> 
>>>>>> If you can let us know how can we workaround the issue, it will be a great help.
>>>>>> 
>>>>>> Regards, Kangkook
>>>>>> 
>>>>>> 
>>>>>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com <mailto:aixer77@gmail.com>> wrote:
>>>>>>> 
>>>>>>> Dear all,
>>>>>>> 
>>>>>>> We are developing custom user space audit agent to gather system wide system
>>>>>>> call trace. While experimenting with various programs, we found out that
>>>>>>> processes (daemons) that started early (along with the system bootstrapping) do
>>>>>>> not report any audit events at all. These processes typically fall into PID
>>>>>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
>>>>>>> 
>>>>>>> 1. Reboot the system
>>>>>>> 
>>>>>>> 2. Add and enable audit events
>>>>>>> # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>>>>>        -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>>>>>        -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>>>>>        -S listen -S socket -S socketpair
>>>>>>> # /sbin/auditctl -e1 -b 102400
>>>>>>> 
>>>>>>> 3. Connect to the system via ssh
>>>>>>>  Audit messages generated only from child processes and none are seen from
>>>>>>>  the original daemon.
>>>>>>> 
>>>>>>> 4. Restart sshd 
>>>>>>>  # restart ssh
>>>>>>> 
>>>>>>> 5. Connect again to the system via ssh
>>>>>>> Now, we see audit messages from both parent and child processes.
>>>>>>> 
>>>>>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
>>>>>>> version 3.13.0-58-generic).
>>>>>>> 
>>>>>>> I first wonder whether this is intended behavior of audit framework or
>>>>>>> not. If it is intended, I also want to know how can we configure auditd
>>>>>>> differently to capture system calls from all processes. 
>>>>>>> 
>>>>>>> Thanks a lot for your help in advance!
>>>>>>> 
>>>>>>> Regards, Kangkook
>>>>> 
>>>>> - RGB
>>> 
>>> - RGB
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545


[-- Attachment #1.2: Type: text/html, Size: 25300 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]