[RFC] audit support for BPF notification

[RFC] audit support for BPF notification

[Jiri Olsa]

hi,
I posted initial change that allows auditd to log BPF program
load/unload events, it's in here:
  https://github.com/linux-audit/audit-userspace/pull/104

We tried to push pure AUDIT interface for BPF program notification,
but it was denied, the discussion is in here:
  https://marc.info/?t=153866123200003&r=1&w=2

The outcome of the discussion was to use perf event interface
for BPF notification and use it in some deamon.. audit was our
first choice.

thoughts?

thanks,
jirka

Re: [RFC] audit support for BPF notification

[Steve Grubb]

Hello,

On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> I posted initial change that allows auditd to log BPF program
> load/unload events, it's in here:
>   https://github.com/linux-audit/audit-userspace/pull/104

Thanks for the patch...but we probably should have talked a bit more before 
undertaking this effort. We normally do not audit from user space what happens 
in the kernel. Doing this can be racy and it keeps auditd from doing the one 
job it has - which is to grab events and record them to disk and send them 
out the realtime interface.


> We tried to push pure AUDIT interface for BPF program notification,
> but it was denied, the discussion is in here:
>   https://marc.info/?t=153866123200003&r=1&w=2

Hmm. The email I remember was here:
https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html

and was only 2 emails long with no answer to my question. :-)

> The outcome of the discussion was to use perf event interface
> for BPF notification and use it in some deamon.. audit was our
> first choice.
> 
> thoughts?

I'd like to understand what the basic problem is that needs to be solved.

-Steve

Re: [RFC] audit support for BPF notification

[Jiri Olsa]

On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> Hello,
> 
> On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > I posted initial change that allows auditd to log BPF program
> > load/unload events, it's in here:
> >   https://github.com/linux-audit/audit-userspace/pull/104
> 
> Thanks for the patch...but we probably should have talked a bit more before 
> undertaking this effort. We normally do not audit from user space what happens 
> in the kernel. Doing this can be racy and it keeps auditd from doing the one 
> job it has - which is to grab events and record them to disk and send them 
> out the realtime interface.
> 
> 
> > We tried to push pure AUDIT interface for BPF program notification,
> > but it was denied, the discussion is in here:
> >   https://marc.info/?t=153866123200003&r=1&w=2
> 
> Hmm. The email I remember was here:
> https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> 
> and was only 2 emails long with no answer to my question. :-)

oops, sry about that, your question was:

	> I'm not sure exactly what the issue is. You can audit for specific syscall 
	> and argument. So, if you want to see loads, then you can make a rule like:

	> -a always,exit -F arch=b64 -S bpf -F a0=5


The problem with above for us is that we also:

  - need to log also other properties of the BPF program,
    which are not visible from BPF syscall arguments, like
    its ID, JIT status or license info

  - need to see BPF program UNLOAD, which is not done
    via syscall, so those would be unvisible at all

> 
> > The outcome of the discussion was to use perf event interface
> > for BPF notification and use it in some deamon.. audit was our
> > first choice.
> > 
> > thoughts?
> 
> I'd like to understand what the basic problem is that needs to be solved.

we need a way for administrators to see the history of loaded BPF
programs, to help investigating issues related to BPF.. and the
only BPF interface for this data is through perf ring buffer

jirka

Re: [RFC] audit support for BPF notification

[Steve Grubb]

On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > Hello,
> > 
> > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > I posted initial change that allows auditd to log BPF program
> > > 
> > > load/unload events, it's in here:
> > >   https://github.com/linux-audit/audit-userspace/pull/104
> > 
> > Thanks for the patch...but we probably should have talked a bit more
> > before undertaking this effort. We normally do not audit from user space
> > what happens in the kernel. Doing this can be racy and it keeps auditd
> > from doing the one job it has - which is to grab events and record them
> > to disk and send them out the realtime interface.
> > 
> > > We tried to push pure AUDIT interface for BPF program notification,
> > > 
> > > but it was denied, the discussion is in here:
> > >   https://marc.info/?t=153866123200003&r=1&w=2
> > 
> > Hmm. The email I remember was here:
> > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > 
> > and was only 2 emails long with no answer to my question. :-)
> 
> oops, sry about that, your question was:
> 	> I'm not sure exactly what the issue is. You can audit for specific
> 	> syscall
> 	> and argument. So, if you want to see loads, then you can make a rule
> 	> like:
> 	> 
> 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> 
> The problem with above for us is that we also:
> 
>   - need to log also other properties of the BPF program,
>     which are not visible from BPF syscall arguments, like
>     its ID, JIT status 

The way this is normally done is to add a supplemental record. For example, 
when auditing the open syscall, we also get CWD & PATH supplemental records. 
When auditing connect, we get a SOCKADDR supplemental record. We have 
requirements around selective audit whereby the admin is in control of what 
is selected for audit via audit rules. So, what one could do is set a rule 
for the bpf syscall and then when it triggers, we get these other records 
added to the bpf syscall event.

>     or license info

This ^^ is not a security issue.


>   - need to see BPF program UNLOAD, which is not done
>     via syscall, so those would be unvisible at all

Is there a place in the kernel where this happens? I could see abnormal 
termination being something we might want. Does the program go through 
something like an exit syscall internally?

 
> > > The outcome of the discussion was to use perf event interface
> > > for BPF notification and use it in some deamon.. audit was our
> > > first choice.
> > > 
> > > thoughts?
> > 
> > I'd like to understand what the basic problem is that needs to be solved.
> 
> we need a way for administrators to see the history of loaded BPF
> programs, to help investigating issues related to BPF.. and the
> only BPF interface for this data is through perf ring buffer

That is really not the audit way. Let's keep talking to see where this ends 
up.

Thanks,
-Steve

Re: [RFC] audit support for BPF notification

[Jiri Olsa]

On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:
> On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > > Hello,
> > > 
> > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > > I posted initial change that allows auditd to log BPF program
> > > > 
> > > > load/unload events, it's in here:
> > > >   https://github.com/linux-audit/audit-userspace/pull/104
> > > 
> > > Thanks for the patch...but we probably should have talked a bit more
> > > before undertaking this effort. We normally do not audit from user space
> > > what happens in the kernel. Doing this can be racy and it keeps auditd
> > > from doing the one job it has - which is to grab events and record them
> > > to disk and send them out the realtime interface.
> > > 
> > > > We tried to push pure AUDIT interface for BPF program notification,
> > > > 
> > > > but it was denied, the discussion is in here:
> > > >   https://marc.info/?t=153866123200003&r=1&w=2
> > > 
> > > Hmm. The email I remember was here:
> > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > 
> > > and was only 2 emails long with no answer to my question. :-)
> > 
> > oops, sry about that, your question was:
> > 	> I'm not sure exactly what the issue is. You can audit for specific
> > 	> syscall
> > 	> and argument. So, if you want to see loads, then you can make a rule
> > 	> like:
> > 	> 
> > 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> > 
> > The problem with above for us is that we also:
> > 
> >   - need to log also other properties of the BPF program,
> >     which are not visible from BPF syscall arguments, like
> >     its ID, JIT status 
> 
> The way this is normally done is to add a supplemental record. For example, 
> when auditing the open syscall, we also get CWD & PATH supplemental records. 
> When auditing connect, we get a SOCKADDR supplemental record. We have 
> requirements around selective audit whereby the admin is in control of what 
> is selected for audit via audit rules. So, what one could do is set a rule 
> for the bpf syscall and then when it triggers, we get these other records 
> added to the bpf syscall event.

right, that was the initial plan, but BPF guys wanted just
single notification system without specific hooks for audit,
so we ended up with perf specific interface

> >     or license info
> 
> This ^^ is not a security issue.
> 
> 
> >   - need to see BPF program UNLOAD, which is not done
> >     via syscall, so those would be unvisible at all
> 
> Is there a place in the kernel where this happens? I could see abnormal 
> termination being something we might want. Does the program go through 
> something like an exit syscall internally?

it's happening in here (kernel/bpf/syscall.c):

	bpf_prog_put
	  __bpf_prog_put
	  {
		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
			...
	  }

BPF program is released when it drops the reference count,
which is normally when its file descriptor is closed.

However it might get pinned and stay alive even when the initial
file descriptor is closed.. and then there's the networking world,
which might have some other specific ways.. but it all ends up
in bpf_prog_put and zero reference count.

> > > > The outcome of the discussion was to use perf event interface
> > > > for BPF notification and use it in some deamon.. audit was our
> > > > first choice.
> > > > 
> > > > thoughts?
> > > 
> > > I'd like to understand what the basic problem is that needs to be solved.
> > 
> > we need a way for administrators to see the history of loaded BPF
> > programs, to help investigating issues related to BPF.. and the
> > only BPF interface for this data is through perf ring buffer
> 
> That is really not the audit way. Let's keep talking to see where this ends 
> up.

Would you see some other auditing daemon/app in place for this kind of data?

thanks,
jirka

Re: [RFC] audit support for BPF notification

[Jiri Olsa]

hi,
Adding Vlad Dronov to the loop, because he asked
about this functionality some time ago.

Vlad, the full thread can be found in here:
  https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html

Any thoughts on this?

thanks,
jirka

On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:
> On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:
> > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > > > Hello,
> > > > 
> > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > > > I posted initial change that allows auditd to log BPF program
> > > > > 
> > > > > load/unload events, it's in here:
> > > > >   https://github.com/linux-audit/audit-userspace/pull/104
> > > > 
> > > > Thanks for the patch...but we probably should have talked a bit more
> > > > before undertaking this effort. We normally do not audit from user space
> > > > what happens in the kernel. Doing this can be racy and it keeps auditd
> > > > from doing the one job it has - which is to grab events and record them
> > > > to disk and send them out the realtime interface.
> > > > 
> > > > > We tried to push pure AUDIT interface for BPF program notification,
> > > > > 
> > > > > but it was denied, the discussion is in here:
> > > > >   https://marc.info/?t=153866123200003&r=1&w=2
> > > > 
> > > > Hmm. The email I remember was here:
> > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > 
> > > > and was only 2 emails long with no answer to my question. :-)
> > > 
> > > oops, sry about that, your question was:
> > > 	> I'm not sure exactly what the issue is. You can audit for specific
> > > 	> syscall
> > > 	> and argument. So, if you want to see loads, then you can make a rule
> > > 	> like:
> > > 	> 
> > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> > > 
> > > The problem with above for us is that we also:
> > > 
> > >   - need to log also other properties of the BPF program,
> > >     which are not visible from BPF syscall arguments, like
> > >     its ID, JIT status 
> > 
> > The way this is normally done is to add a supplemental record. For example, 
> > when auditing the open syscall, we also get CWD & PATH supplemental records. 
> > When auditing connect, we get a SOCKADDR supplemental record. We have 
> > requirements around selective audit whereby the admin is in control of what 
> > is selected for audit via audit rules. So, what one could do is set a rule 
> > for the bpf syscall and then when it triggers, we get these other records 
> > added to the bpf syscall event.
> 
> right, that was the initial plan, but BPF guys wanted just
> single notification system without specific hooks for audit,
> so we ended up with perf specific interface
> 
> > >     or license info
> > 
> > This ^^ is not a security issue.
> > 
> > 
> > >   - need to see BPF program UNLOAD, which is not done
> > >     via syscall, so those would be unvisible at all
> > 
> > Is there a place in the kernel where this happens? I could see abnormal 
> > termination being something we might want. Does the program go through 
> > something like an exit syscall internally?
> 
> it's happening in here (kernel/bpf/syscall.c):
> 
> 	bpf_prog_put
> 	  __bpf_prog_put
> 	  {
> 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> 			...
> 	  }
> 
> BPF program is released when it drops the reference count,
> which is normally when its file descriptor is closed.
> 
> However it might get pinned and stay alive even when the initial
> file descriptor is closed.. and then there's the networking world,
> which might have some other specific ways.. but it all ends up
> in bpf_prog_put and zero reference count.
> 
> > > > > The outcome of the discussion was to use perf event interface
> > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > first choice.
> > > > > 
> > > > > thoughts?
> > > > 
> > > > I'd like to understand what the basic problem is that needs to be solved.
> > > 
> > > we need a way for administrators to see the history of loaded BPF
> > > programs, to help investigating issues related to BPF.. and the
> > > only BPF interface for this data is through perf ring buffer
> > 
> > That is really not the audit way. Let's keep talking to see where this ends 
> > up.
> 
> Would you see some other auditing daemon/app in place for this kind of data?
> 
> thanks,
> jirka

Re: [RFC] audit support for BPF notification

[Jiri Olsa]

cc-ing Petr Matousek

jirka

On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote:
> hi,
> Adding Vlad Dronov to the loop, because he asked
> about this functionality some time ago.
> 
> Vlad, the full thread can be found in here:
>   https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html
> 
> Any thoughts on this?
> 
> thanks,
> jirka
> 
> On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:
> > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:
> > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > > > > Hello,
> > > > > 
> > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > > > > I posted initial change that allows auditd to log BPF program
> > > > > > 
> > > > > > load/unload events, it's in here:
> > > > > >   https://github.com/linux-audit/audit-userspace/pull/104
> > > > > 
> > > > > Thanks for the patch...but we probably should have talked a bit more
> > > > > before undertaking this effort. We normally do not audit from user space
> > > > > what happens in the kernel. Doing this can be racy and it keeps auditd
> > > > > from doing the one job it has - which is to grab events and record them
> > > > > to disk and send them out the realtime interface.
> > > > > 
> > > > > > We tried to push pure AUDIT interface for BPF program notification,
> > > > > > 
> > > > > > but it was denied, the discussion is in here:
> > > > > >   https://marc.info/?t=153866123200003&r=1&w=2
> > > > > 
> > > > > Hmm. The email I remember was here:
> > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > > 
> > > > > and was only 2 emails long with no answer to my question. :-)
> > > > 
> > > > oops, sry about that, your question was:
> > > > 	> I'm not sure exactly what the issue is. You can audit for specific
> > > > 	> syscall
> > > > 	> and argument. So, if you want to see loads, then you can make a rule
> > > > 	> like:
> > > > 	> 
> > > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> > > > 
> > > > The problem with above for us is that we also:
> > > > 
> > > >   - need to log also other properties of the BPF program,
> > > >     which are not visible from BPF syscall arguments, like
> > > >     its ID, JIT status 
> > > 
> > > The way this is normally done is to add a supplemental record. For example, 
> > > when auditing the open syscall, we also get CWD & PATH supplemental records. 
> > > When auditing connect, we get a SOCKADDR supplemental record. We have 
> > > requirements around selective audit whereby the admin is in control of what 
> > > is selected for audit via audit rules. So, what one could do is set a rule 
> > > for the bpf syscall and then when it triggers, we get these other records 
> > > added to the bpf syscall event.
> > 
> > right, that was the initial plan, but BPF guys wanted just
> > single notification system without specific hooks for audit,
> > so we ended up with perf specific interface
> > 
> > > >     or license info
> > > 
> > > This ^^ is not a security issue.
> > > 
> > > 
> > > >   - need to see BPF program UNLOAD, which is not done
> > > >     via syscall, so those would be unvisible at all
> > > 
> > > Is there a place in the kernel where this happens? I could see abnormal 
> > > termination being something we might want. Does the program go through 
> > > something like an exit syscall internally?
> > 
> > it's happening in here (kernel/bpf/syscall.c):
> > 
> > 	bpf_prog_put
> > 	  __bpf_prog_put
> > 	  {
> > 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> > 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> > 			...
> > 	  }
> > 
> > BPF program is released when it drops the reference count,
> > which is normally when its file descriptor is closed.
> > 
> > However it might get pinned and stay alive even when the initial
> > file descriptor is closed.. and then there's the networking world,
> > which might have some other specific ways.. but it all ends up
> > in bpf_prog_put and zero reference count.
> > 
> > > > > > The outcome of the discussion was to use perf event interface
> > > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > > first choice.
> > > > > > 
> > > > > > thoughts?
> > > > > 
> > > > > I'd like to understand what the basic problem is that needs to be solved.
> > > > 
> > > > we need a way for administrators to see the history of loaded BPF
> > > > programs, to help investigating issues related to BPF.. and the
> > > > only BPF interface for this data is through perf ring buffer
> > > 
> > > That is really not the audit way. Let's keep talking to see where this ends 
> > > up.
> > 
> > Would you see some other auditing daemon/app in place for this kind of data?
> > 
> > thanks,
> > jirka

Re: [RFC] audit support for BPF notification

[Jiri Benc]

Seems there have been no reply to this... 

Jiri, what is the current status?

Vlad, what is the Product Security's view on this? Is the audit support
for bpf programs loading/unloading a requirement for full support of
eBPF (as opposed to tech preview)?

Thanks,

 Jiri

On Tue, 20 Aug 2019 15:54:53 +0200, Jiri Olsa wrote:
> cc-ing Petr Matousek
> 
> jirka
> 
> On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote:
> > hi,
> > Adding Vlad Dronov to the loop, because he asked
> > about this functionality some time ago.
> > 
> > Vlad, the full thread can be found in here:
> >   https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html
> > 
> > Any thoughts on this?
> > 
> > thanks,
> > jirka
> > 
> > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:  
> > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:  
> > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:  
> > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:  
> > > > > > Hello,
> > > > > > 
> > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:  
> > > > > > > I posted initial change that allows auditd to log BPF program
> > > > > > > 
> > > > > > > load/unload events, it's in here:
> > > > > > >   https://github.com/linux-audit/audit-userspace/pull/104  
> > > > > > 
> > > > > > Thanks for the patch...but we probably should have talked a bit more
> > > > > > before undertaking this effort. We normally do not audit from user space
> > > > > > what happens in the kernel. Doing this can be racy and it keeps auditd
> > > > > > from doing the one job it has - which is to grab events and record them
> > > > > > to disk and send them out the realtime interface.
> > > > > >   
> > > > > > > We tried to push pure AUDIT interface for BPF program notification,
> > > > > > > 
> > > > > > > but it was denied, the discussion is in here:
> > > > > > >   https://marc.info/?t=153866123200003&r=1&w=2  
> > > > > > 
> > > > > > Hmm. The email I remember was here:
> > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > > > 
> > > > > > and was only 2 emails long with no answer to my question. :-)  
> > > > > 
> > > > > oops, sry about that, your question was:  
> > > > > 	> I'm not sure exactly what the issue is. You can audit for specific
> > > > > 	> syscall
> > > > > 	> and argument. So, if you want to see loads, then you can make a rule
> > > > > 	> like:
> > > > > 	> 
> > > > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5  
> > > > > 
> > > > > The problem with above for us is that we also:
> > > > > 
> > > > >   - need to log also other properties of the BPF program,
> > > > >     which are not visible from BPF syscall arguments, like
> > > > >     its ID, JIT status   
> > > > 
> > > > The way this is normally done is to add a supplemental record. For example, 
> > > > when auditing the open syscall, we also get CWD & PATH supplemental records. 
> > > > When auditing connect, we get a SOCKADDR supplemental record. We have 
> > > > requirements around selective audit whereby the admin is in control of what 
> > > > is selected for audit via audit rules. So, what one could do is set a rule 
> > > > for the bpf syscall and then when it triggers, we get these other records 
> > > > added to the bpf syscall event.  
> > > 
> > > right, that was the initial plan, but BPF guys wanted just
> > > single notification system without specific hooks for audit,
> > > so we ended up with perf specific interface
> > >   
> > > > >     or license info  
> > > > 
> > > > This ^^ is not a security issue.
> > > > 
> > > >   
> > > > >   - need to see BPF program UNLOAD, which is not done
> > > > >     via syscall, so those would be unvisible at all  
> > > > 
> > > > Is there a place in the kernel where this happens? I could see abnormal 
> > > > termination being something we might want. Does the program go through 
> > > > something like an exit syscall internally?  
> > > 
> > > it's happening in here (kernel/bpf/syscall.c):
> > > 
> > > 	bpf_prog_put
> > > 	  __bpf_prog_put
> > > 	  {
> > > 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> > > 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> > > 			...
> > > 	  }
> > > 
> > > BPF program is released when it drops the reference count,
> > > which is normally when its file descriptor is closed.
> > > 
> > > However it might get pinned and stay alive even when the initial
> > > file descriptor is closed.. and then there's the networking world,
> > > which might have some other specific ways.. but it all ends up
> > > in bpf_prog_put and zero reference count.
> > >   
> > > > > > > The outcome of the discussion was to use perf event interface
> > > > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > > > first choice.
> > > > > > > 
> > > > > > > thoughts?  
> > > > > > 
> > > > > > I'd like to understand what the basic problem is that needs to be solved.  
> > > > > 
> > > > > we need a way for administrators to see the history of loaded BPF
> > > > > programs, to help investigating issues related to BPF.. and the
> > > > > only BPF interface for this data is through perf ring buffer  
> > > > 
> > > > That is really not the audit way. Let's keep talking to see where this ends 
> > > > up.  
> > > 
> > > Would you see some other auditing daemon/app in place for this kind of data?
> > > 
> > > thanks,
> > > jirka  

Re: [RFC] audit support for BPF notification

[Jiri Olsa]

On Mon, Nov 04, 2019 at 02:05:18PM +0100, Jiri Benc wrote:
> Seems there have been no reply to this... 
> 
> Jiri, what is the current status?

same, there's still no answer from Product Security

jirka

> 
> Vlad, what is the Product Security's view on this? Is the audit support
> for bpf programs loading/unloading a requirement for full support of
> eBPF (as opposed to tech preview)?
> 
> Thanks,
> 
>  Jiri
> 
> On Tue, 20 Aug 2019 15:54:53 +0200, Jiri Olsa wrote:
> > cc-ing Petr Matousek
> > 
> > jirka
> > 
> > On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote:
> > > hi,
> > > Adding Vlad Dronov to the loop, because he asked
> > > about this functionality some time ago.
> > > 
> > > Vlad, the full thread can be found in here:
> > >   https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html
> > > 
> > > Any thoughts on this?
> > > 
> > > thanks,
> > > jirka
> > > 
> > > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:  
> > > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:  
> > > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:  
> > > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:  
> > > > > > > Hello,
> > > > > > > 
> > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:  
> > > > > > > > I posted initial change that allows auditd to log BPF program
> > > > > > > > 
> > > > > > > > load/unload events, it's in here:
> > > > > > > >   https://github.com/linux-audit/audit-userspace/pull/104  
> > > > > > > 
> > > > > > > Thanks for the patch...but we probably should have talked a bit more
> > > > > > > before undertaking this effort. We normally do not audit from user space
> > > > > > > what happens in the kernel. Doing this can be racy and it keeps auditd
> > > > > > > from doing the one job it has - which is to grab events and record them
> > > > > > > to disk and send them out the realtime interface.
> > > > > > >   
> > > > > > > > We tried to push pure AUDIT interface for BPF program notification,
> > > > > > > > 
> > > > > > > > but it was denied, the discussion is in here:
> > > > > > > >   https://marc.info/?t=153866123200003&r=1&w=2  
> > > > > > > 
> > > > > > > Hmm. The email I remember was here:
> > > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > > > > 
> > > > > > > and was only 2 emails long with no answer to my question. :-)  
> > > > > > 
> > > > > > oops, sry about that, your question was:  
> > > > > > 	> I'm not sure exactly what the issue is. You can audit for specific
> > > > > > 	> syscall
> > > > > > 	> and argument. So, if you want to see loads, then you can make a rule
> > > > > > 	> like:
> > > > > > 	> 
> > > > > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5  
> > > > > > 
> > > > > > The problem with above for us is that we also:
> > > > > > 
> > > > > >   - need to log also other properties of the BPF program,
> > > > > >     which are not visible from BPF syscall arguments, like
> > > > > >     its ID, JIT status   
> > > > > 
> > > > > The way this is normally done is to add a supplemental record. For example, 
> > > > > when auditing the open syscall, we also get CWD & PATH supplemental records. 
> > > > > When auditing connect, we get a SOCKADDR supplemental record. We have 
> > > > > requirements around selective audit whereby the admin is in control of what 
> > > > > is selected for audit via audit rules. So, what one could do is set a rule 
> > > > > for the bpf syscall and then when it triggers, we get these other records 
> > > > > added to the bpf syscall event.  
> > > > 
> > > > right, that was the initial plan, but BPF guys wanted just
> > > > single notification system without specific hooks for audit,
> > > > so we ended up with perf specific interface
> > > >   
> > > > > >     or license info  
> > > > > 
> > > > > This ^^ is not a security issue.
> > > > > 
> > > > >   
> > > > > >   - need to see BPF program UNLOAD, which is not done
> > > > > >     via syscall, so those would be unvisible at all  
> > > > > 
> > > > > Is there a place in the kernel where this happens? I could see abnormal 
> > > > > termination being something we might want. Does the program go through 
> > > > > something like an exit syscall internally?  
> > > > 
> > > > it's happening in here (kernel/bpf/syscall.c):
> > > > 
> > > > 	bpf_prog_put
> > > > 	  __bpf_prog_put
> > > > 	  {
> > > > 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> > > > 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> > > > 			...
> > > > 	  }
> > > > 
> > > > BPF program is released when it drops the reference count,
> > > > which is normally when its file descriptor is closed.
> > > > 
> > > > However it might get pinned and stay alive even when the initial
> > > > file descriptor is closed.. and then there's the networking world,
> > > > which might have some other specific ways.. but it all ends up
> > > > in bpf_prog_put and zero reference count.
> > > >   
> > > > > > > > The outcome of the discussion was to use perf event interface
> > > > > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > > > > first choice.
> > > > > > > > 
> > > > > > > > thoughts?  
> > > > > > > 
> > > > > > > I'd like to understand what the basic problem is that needs to be solved.  
> > > > > > 
> > > > > > we need a way for administrators to see the history of loaded BPF
> > > > > > programs, to help investigating issues related to BPF.. and the
> > > > > > only BPF interface for this data is through perf ring buffer  
> > > > > 
> > > > > That is really not the audit way. Let's keep talking to see where this ends 
> > > > > up.  
> > > > 
> > > > Would you see some other auditing daemon/app in place for this kind of data?
> > > > 
> > > > thanks,
> > > > jirka  
> 

Re: [RFC] audit support for BPF notification

[Vladis Dronov]

Hello, Jiri,

Unfortunately, I'm not with the Product Security anymore and cannot speak in
ProdSec name. My personal view is that we/customers must be able to log eBPF
program load/unload events in some way (not necessary by audit).

Best regards,
Vladis Dronov | Red Hat, Inc. | The Core Kernel | Senior Software Engineer

----- Original Message -----
> From: "Jiri Benc" <jbenc@redhat.com>
> To: "Jiri Olsa" <jolsa@redhat.com>
> Cc: "Steve Grubb" <sgrubb@redhat.com>, linux-audit@redhat.com, "Stanislav Kozina" <skozina@redhat.com>, "Yauheni
> Kaliuta" <yauheni.kaliuta@redhat.com>, "Toke Høiland-Jørgensen" <toke@redhat.com>, "Arnaldo Carvalho de Melo"
> <acme@redhat.com>, "Jesper Dangaard Brouer" <brouer@redhat.com>, "Vlad Dronov" <vdronov@redhat.com>, "Petr Matousek"
> <pmatouse@redhat.com>, "Rashid Khan" <rkhan@redhat.com>
> Sent: Monday, November 4, 2019 2:05:18 PM
> Subject: Re: [RFC] audit support for BPF notification
> 
> Seems there have been no reply to this...
> 
> Jiri, what is the current status?
> 
> Vlad, what is the Product Security's view on this? Is the audit support
> for bpf programs loading/unloading a requirement for full support of
> eBPF (as opposed to tech preview)?
> 
> Thanks,
> 
>  Jiri
> 
> On Tue, 20 Aug 2019 15:54:53 +0200, Jiri Olsa wrote:
> > cc-ing Petr Matousek
> > 
> > jirka
> > 
> > On Wed, Aug 14, 2019 at 09:33:34AM +0200, Jiri Olsa wrote:
> > > hi,
> > > Adding Vlad Dronov to the loop, because he asked
> > > about this functionality some time ago.
> > > 
> > > Vlad, the full thread can be found in here:
> > >   https://www.redhat.com/archives/linux-audit/2019-August/msg00004.html
> > > 
> > > Any thoughts on this?
> > > 
> > > thanks,
> > > jirka
> > > 
> > > On Mon, Aug 12, 2019 at 04:33:10PM +0200, Jiri Olsa wrote:
> > > > On Mon, Aug 12, 2019 at 09:49:43AM -0400, Steve Grubb wrote:
> > > > > On Monday, August 12, 2019 3:59:22 AM EDT Jiri Olsa wrote:
> > > > > > On Fri, Aug 09, 2019 at 01:45:21PM -0400, Steve Grubb wrote:
> > > > > > > Hello,
> > > > > > > 
> > > > > > > On Friday, August 9, 2019 10:18:31 AM EDT Jiri Olsa wrote:
> > > > > > > > I posted initial change that allows auditd to log BPF program
> > > > > > > > 
> > > > > > > > load/unload events, it's in here:
> > > > > > > >   https://github.com/linux-audit/audit-userspace/pull/104
> > > > > > > 
> > > > > > > Thanks for the patch...but we probably should have talked a bit
> > > > > > > more
> > > > > > > before undertaking this effort. We normally do not audit from
> > > > > > > user space
> > > > > > > what happens in the kernel. Doing this can be racy and it keeps
> > > > > > > auditd
> > > > > > > from doing the one job it has - which is to grab events and
> > > > > > > record them
> > > > > > > to disk and send them out the realtime interface.
> > > > > > >   
> > > > > > > > We tried to push pure AUDIT interface for BPF program
> > > > > > > > notification,
> > > > > > > > 
> > > > > > > > but it was denied, the discussion is in here:
> > > > > > > >   https://marc.info/?t=153866123200003&r=1&w=2
> > > > > > > 
> > > > > > > Hmm. The email I remember was here:
> > > > > > > https://www.redhat.com/archives/linux-audit/2018-October/msg00053.html
> > > > > > > 
> > > > > > > and was only 2 emails long with no answer to my question. :-)
> > > > > > 
> > > > > > oops, sry about that, your question was:
> > > > > > 	> I'm not sure exactly what the issue is. You can audit for
> > > > > > 	> specific
> > > > > > 	> syscall
> > > > > > 	> and argument. So, if you want to see loads, then you can make a
> > > > > > 	> rule
> > > > > > 	> like:
> > > > > > 	> 
> > > > > > 	> -a always,exit -F arch=b64 -S bpf -F a0=5
> > > > > > 
> > > > > > The problem with above for us is that we also:
> > > > > > 
> > > > > >   - need to log also other properties of the BPF program,
> > > > > >     which are not visible from BPF syscall arguments, like
> > > > > >     its ID, JIT status
> > > > > 
> > > > > The way this is normally done is to add a supplemental record. For
> > > > > example,
> > > > > when auditing the open syscall, we also get CWD & PATH supplemental
> > > > > records.
> > > > > When auditing connect, we get a SOCKADDR supplemental record. We have
> > > > > requirements around selective audit whereby the admin is in control
> > > > > of what
> > > > > is selected for audit via audit rules. So, what one could do is set a
> > > > > rule
> > > > > for the bpf syscall and then when it triggers, we get these other
> > > > > records
> > > > > added to the bpf syscall event.
> > > > 
> > > > right, that was the initial plan, but BPF guys wanted just
> > > > single notification system without specific hooks for audit,
> > > > so we ended up with perf specific interface
> > > >   
> > > > > >     or license info
> > > > > 
> > > > > This ^^ is not a security issue.
> > > > > 
> > > > >   
> > > > > >   - need to see BPF program UNLOAD, which is not done
> > > > > >     via syscall, so those would be unvisible at all
> > > > > 
> > > > > Is there a place in the kernel where this happens? I could see
> > > > > abnormal
> > > > > termination being something we might want. Does the program go
> > > > > through
> > > > > something like an exit syscall internally?
> > > > 
> > > > it's happening in here (kernel/bpf/syscall.c):
> > > > 
> > > > 	bpf_prog_put
> > > > 	  __bpf_prog_put
> > > > 	  {
> > > > 		    if (atomic_dec_and_test(&prog->aux->refcnt)) {
> > > > 			perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_UNLOAD, 0);
> > > > 			...
> > > > 	  }
> > > > 
> > > > BPF program is released when it drops the reference count,
> > > > which is normally when its file descriptor is closed.
> > > > 
> > > > However it might get pinned and stay alive even when the initial
> > > > file descriptor is closed.. and then there's the networking world,
> > > > which might have some other specific ways.. but it all ends up
> > > > in bpf_prog_put and zero reference count.
> > > >   
> > > > > > > > The outcome of the discussion was to use perf event interface
> > > > > > > > for BPF notification and use it in some deamon.. audit was our
> > > > > > > > first choice.
> > > > > > > > 
> > > > > > > > thoughts?
> > > > > > > 
> > > > > > > I'd like to understand what the basic problem is that needs to be
> > > > > > > solved.
> > > > > > 
> > > > > > we need a way for administrators to see the history of loaded BPF
> > > > > > programs, to help investigating issues related to BPF.. and the
> > > > > > only BPF interface for this data is through perf ring buffer
> > > > > 
> > > > > That is really not the audit way. Let's keep talking to see where
> > > > > this ends
> > > > > up.
> > > > 
> > > > Would you see some other auditing daemon/app in place for this kind of
> > > > data?
> > > > 
> > > > thanks,
> > > > jirka
> 
> 

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: [RFC] audit support for BPF notification

[Vladis Dronov]

[an addition]

I also believe that this log entry should include program source and/or bytecode
checksum so customer/our support can verify that exactly this eBPF program was
loaded/unloaded and not the program that someone states that it was loaded.

Best regards,
Vladis Dronov | Red Hat, Inc. | The Core Kernel | Senior Software Engineer

----- Original Message -----
> From: "Jiri Benc" <jbenc@redhat.com>
> To: "Jiri Olsa" <jolsa@redhat.com>
> Cc: "Steve Grubb" <sgrubb@redhat.com>, linux-audit@redhat.com, "Stanislav Kozina" <skozina@redhat.com>, "Yauheni
> Kaliuta" <yauheni.kaliuta@redhat.com>, "Toke Høiland-Jørgensen" <toke@redhat.com>, "Arnaldo Carvalho de Melo"
> <acme@redhat.com>, "Jesper Dangaard Brouer" <brouer@redhat.com>, "Vlad Dronov" <vdronov@redhat.com>, "Petr Matousek"
> <pmatouse@redhat.com>, "Rashid Khan" <rkhan@redhat.com>
> Sent: Monday, November 4, 2019 2:05:18 PM
> Subject: Re: [RFC] audit support for BPF notification
> 
> Seems there have been no reply to this...
> 
> Jiri, what is the current status?
> 
> Vlad, what is the Product Security's view on this? Is the audit support
> for bpf programs loading/unloading a requirement for full support of
> eBPF (as opposed to tech preview)?
> 
> Thanks,
> 
>  Jiri

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

Re: [RFC] audit support for BPF notification

[Paul Moore]

On Mon, Nov 4, 2019 at 9:41 AM Jiri Benc <jbenc@redhat.com> wrote:
> Seems there have been no reply to this...
>
> Jiri, what is the current status?
>
> Vlad, what is the Product Security's view on this? Is the audit support
> for bpf programs loading/unloading a requirement for full support of
> eBPF (as opposed to tech preview)?

As a reminder, this is a public mailing list that focuses on the
technical development of the upstream Linux audit project; Red Hat
product decisions should not be discussed here.

-- 
paul moore
www.paul-moore.com