Re: Early processes (daemons) do not report audit events

[Kangkook Jee <aixer77@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 7172 bytes --]

Dear Richard,

Thanks a lot for your help. I agree that the inconsistency seems a bit strange but 
I also doubt that different distributions make different changes to kernel/audit.c. This 
may be just a timing issue with regard to  startup orderings. 

I’d like to append Burn’s message that solved my issue for future reference.

>>> 
>>> Kangkook,
>>> 
>>> Perhaps you can re-test, but modify the kernel boot parameters to
>>> include audit=1 as an additional argument.
>>> 
>>> Reading the auditd(8) manual, one sees
>>> 
>>>       A boot param of audit=1 should be added to ensure that all
>>>       processes that run before the audit daemon starts is marked as
>>>       auditable by the kernel. Not doing that will make a few
>>>       processes impossible to properly audit.


Thanks again for your help!

Regards, Kangkook


> On Sep 13, 2015, at 11:58 AM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
> 
> On 15/09/11, Kangkook Jee wrote:
>> Hi Richard,
> 
> Hi Kangkook,
> 
>> I also did the same experiment for the latest distributions of Fedora
>> core and Debian and here’s the results.
>> 
>> Fedora-22 (64-bit, 4.0.4-301.fc22.x86_64): Problem reproduced.
>> Debian-8 (64-bit, 3.16.0-4-amd64): Problem reproduced
>> 
>> Btw, Burn Alting (burn@swtf.dyndns.org <mailto:burn@swtf.dyndns.org>) suggested me to append audit=1
>> to kernel flag. I added the option to boot-loader (grub) and problem
>> went away. 
> 
> On all systems?  This is expected behaviour.  Sorry I was not more
> explicit in asking you to test that.  I guess it was implied by asking
> what the settings for the kernel command line were.
> 
> Now the surprising bit is that CentOS does not demonstrate the problem
> without audit=1 in the command line, which leads me to wonder if they
> have set "u32             audit_enabled = 1;" around line 83 of
> kernel/audit.c in their kernel source.  It would surprise me if they
> did, but it would not be completely unreasonable.
> 
>> Regards, Kangkook
>> 
>>> On Sep 11, 2015, at 12:24 PM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
>>> On 15/09/11, Kangkook Jee wrote:
>>>> From the previous reply, I think I misunderstood your question regarding kernel command line. 
>>>> Here’s "cat /proc/cmdline” results for distributions that I’ve experimented. 
>>>> 
>>>> Ubuntu 14.04 (64-bit): 
>>>>   BOOT_IMAGE=/boot/vmlinuz-3.13.0-58-generic root=UUID=7505f862-ce46-49e5-9d1c-e4e307844889 ro text quiet splash vt.handoff=7
>>>> 
>>>> Ubuntu 12.04 (64-bit): 
>>>>   BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=5be789be-9b0c-463e-bd18-42bfa79fb24c ro quiet splash
>>>> 
>>>> CentOS 7 (64-bit): 
>>>>   BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet LANG=en_US.UTF-8
>>>> 
>>>> CentOS 6 (64-bit): 
>>>>   ro root=UUID=a7d44560-adcc-4000-9584-8b9fcf2afd74 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>>>> 
>>>> I don’t see any audit=<value> entries from all examples above. 
>>> 
>>> Yes, this is what I was seeking from you.  And you are correct, none of
>>> them have audit=1 as I was hoping from at least CentOS.  There is a
>>> chance that the CentOS kernel was compiled with audit=1 hardcoded, but I
>>> think that is a pretty small chance...
>>> 
>>> I'll have to look at this closer...  But any Debian and Fedora data
>>> points that you can provide would certainly be useful.
>>> 
>>>> /Kangkook
>>>> 
>>>>> On Sep 11, 2015, at 5:50 AM, Richard Guy Briggs <rgb@redhat.com <mailto:rgb@redhat.com>> wrote:
>>>>> 
>>>>> On 15/09/10, Kangkook Jee wrote:
>>>>>> Hi all,
>>>>>> 
>>>>>> I debugged a bit further to identify distributions that are affected by the issue.
>>>>>> I repeated the same experiment with sshd from 3 more distributions.
>>>>>> 
>>>>>> CentOS Linux release 7.1.1503 (64-bit, 3.10.0-229.el7.x86_64): Problem NOT reproduced
>>>>>> CentOS release 6.6 (64-bit, 2.6.32-504.el6.x86_64): Problem NOT reproduced
>>>>>> Ubuntu 12.04.5 LTS (64-bit, 3.13.0-32-generic): Problem reproduced
>>>>> 
>>>>> For each of these examples, what is the value of the kernel command line
>>>>> "audit=<value>" if it is even present?  It is possible that the CentOS
>>>>> examples include "audit=1" while Ubuntu omits the line.  "cat
>>>>> /proc/cmdline" should tell you the answer.
>>>>> 
>>>>>> After all, Ubuntu family are affected by the issue and I could confirm
>>>>>> that results are inconsistent across two different distribution
>>>>>> families. 
>>>>> 
>>>>> I would be curious what your results are with a recent Debian and with a
>>>>> recent Fedora.
>>>>> 
>>>>>> If you can let us know how can we workaround the issue, it will be a great help.
>>>>>> 
>>>>>> Regards, Kangkook
>>>>>> 
>>>>>> 
>>>>>>> On Sep 9, 2015, at 11:50 PM, Kangkook Jee <aixer77@gmail.com <mailto:aixer77@gmail.com>> wrote:
>>>>>>> 
>>>>>>> Dear all,
>>>>>>> 
>>>>>>> We are developing custom user space audit agent to gather system wide system
>>>>>>> call trace. While experimenting with various programs, we found out that
>>>>>>> processes (daemons) that started early (along with the system bootstrapping) do
>>>>>>> not report any audit events at all. These processes typically fall into PID
>>>>>>> range of less than 2000. Here’s how I reproduced the symptom with sshd daemon.
>>>>>>> 
>>>>>>> 1. Reboot the system
>>>>>>> 
>>>>>>> 2. Add and enable audit events
>>>>>>> # /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup
>>>>>>>        -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat 
>>>>>>>        -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect 
>>>>>>>        -S listen -S socket -S socketpair
>>>>>>> # /sbin/auditctl -e1 -b 102400
>>>>>>> 
>>>>>>> 3. Connect to the system via ssh
>>>>>>>  Audit messages generated only from child processes and none are seen from
>>>>>>>  the original daemon.
>>>>>>> 
>>>>>>> 4. Restart sshd 
>>>>>>>  # restart ssh
>>>>>>> 
>>>>>>> 5. Connect again to the system via ssh
>>>>>>> Now, we see audit messages from both parent and child processes.
>>>>>>> 
>>>>>>> I did the experiment from Ubuntu 14.04.2 LTS distribution (64-bit, kernel
>>>>>>> version 3.13.0-58-generic).
>>>>>>> 
>>>>>>> I first wonder whether this is intended behavior of audit framework or
>>>>>>> not. If it is intended, I also want to know how can we configure auditd
>>>>>>> differently to capture system calls from all processes. 
>>>>>>> 
>>>>>>> Thanks a lot for your help in advance!
>>>>>>> 
>>>>>>> Regards, Kangkook
>>>>> 
>>>>> - RGB
>>> 
>>> - RGB
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com <mailto:rbriggs@redhat.com>>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545


[-- Attachment #1.2: Type: text/html, Size: 25300 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]