* Re: [GIT PULL] Audit patches for v4.10
From: Paul Moore @ 2016-12-14 21:14 UTC (permalink / raw)
To: linux-audit
In-Reply-To: <1522672.4tJvQfugPF@sifl>
On Wed, Dec 14, 2016 at 1:27 PM, Paul Moore <pmoore@redhat.com> wrote:
> Hi Linus,
>
> After the small number of patches for v4.9, we've got a much bigger pile for
> v4.10 ...
[NOTE: dropped everyone except for the linux-audit list]
Due to the large number of changes in audit#next for v4.10 I've
decided to not do the usual audit#next rebase following the
pull-request to Linus; the next branch will remain based on v4.8 for
the v4.11 development cycle. I've also merged the two patches in my
next-queue branch into audit#next.
As a reminder, the normal process is documented here:
* http://www.paul-moore.com/blog/d/2016/03/kernel-repo-process.html
--
paul moore
www.paul-moore.com
^ permalink raw reply
* [GIT PULL] Audit patches for v4.10
From: Paul Moore @ 2016-12-14 18:27 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux-audit, linux-kernel
Hi Linus,
After the small number of patches for v4.9, we've got a much bigger pile for
v4.10.
The bulk of these patches involve a rework of the audit backlog queue to
enable us to move the netlink multicasting out of the task/thread that
generates the audit record and into the kernel thread that emits the record
(just like we do for the audit unicast to auditd). While we were playing
with the backlog queue(s) we fixed a number of other little problems with
the code, and from all the testing so far things look to be in much better
shape now. Doing this also allowed us to re-enable disabling IRQs for some
netns operations ("netns: avoid disabling irq for netns id"). The remaining
patches fix some small problems that are well documented in the commit
descriptions, as well as adding session ID filtering support.
You will likely hit two merge conflicts, one in net/core/net_namespace.c and
one in include/uapi/linux/audit.h, both are easily resolved so I won't
bother you with that here. If you have questions, you know how to find me.
Thanks,
-Paul
---
The following changes since commit c8d2bc9bc39ebea8437fd974fdbc21847bb897a3:
Linux 4.8 (2016-10-02 16:24:33 -0700)
are available in the git repository at:
git://git.infradead.org/users/pcmoore/audit stable-4.10
for you to fetch changes up to 533c7b69c764ad5febb3e716899f43a75564fcab:
audit: use proper refcount locking on audit_sock
(2016-12-14 13:06:04 -0500)
----------------------------------------------------------------
Alexey Dobriyan (1):
audit: less stack usage for /proc/*/loginuid
Paul Moore (9):
audit: fixup audit_init()
audit: queue netlink multicast sends just like we do for unicast sends
audit: rename the queues and kauditd related functions
audit: rework the audit queue handling
audit: rework audit_log_start()
audit: wake up kauditd_thread after auditd registers
audit: handle a clean auditd shutdown with grace
audit: don't ever sleep on a command record/message
netns: avoid disabling irq for netns id
Richard Guy Briggs (5):
audit: tame initialization warning len_abuf in audit_log_execve_info
audit: skip sessionid sentinel value when auto-incrementing
audit: add support for session ID user filter
audit: move kaudit thread start from auditd registration to
kaudit init (#2)
audit: use proper refcount locking on audit_sock
Steve Grubb (1):
audit: fix formatting of AUDIT_CONFIG_CHANGE events
fs/proc/base.c | 2 +-
include/uapi/linux/audit.h | 5 +-
kernel/audit.c | 532 ++++++++++++++++++++++++---------------
kernel/audit_fsnotify.c | 5 +-
kernel/audit_tree.c | 3 +-
kernel/audit_watch.c | 5 +-
kernel/auditfilter.c | 5 +-
kernel/auditsc.c | 12 +-
net/core/net_namespace.c | 35 ++-
9 files changed, 361 insertions(+), 243 deletions(-)
--
paul moore
security @ redhat
^ permalink raw reply
* Re: [RFC PATCH v3] audit: use proper refcount locking on audit_sock
From: Cong Wang @ 2016-12-14 5:36 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: Linux Kernel Network Developers, LKML, Eric Dumazet, linux-audit,
Dmitry Vyukov
In-Reply-To: <20161214040005.GL22660@madcap2.tricolour.ca>
On Tue, Dec 13, 2016 at 8:00 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> On 2016-12-13 16:19, Cong Wang wrote:
>> On Tue, Dec 13, 2016 at 7:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
>> > @@ -1283,8 +1299,10 @@ static void __net_exit audit_net_exit(struct net *net)
>> > {
>> > struct audit_net *aunet = net_generic(net, audit_net_id);
>> > struct sock *sock = aunet->nlsk;
>> > + mutex_lock(&audit_cmd_mutex);
>> > if (sock == audit_sock)
>> > auditd_reset();
>> > + mutex_unlock(&audit_cmd_mutex);
>>
>> This still doesn't look correct to me, b/c here we release the audit_sock
>> refcnt twice:
>>
>> 1) inside audit_reset()
>
> The audit_reset() refcount decrement corresponds to a setting of
> audit_sock only if audit_sock is still non-NULL.
>
Hmm, thinking about it again, looks like the sock == audit_sock
and audit_sock != NULL checks can guarantee we are safe. So,
Reviewed-by: Cong Wang <xiyou.wangcong@gmail.com>
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Richard Guy Briggs @ 2016-12-14 4:17 UTC (permalink / raw)
To: Cong Wang
Cc: Herbert Xu, Johannes Berg, netdev, Florian Westphal, LKML,
Eric Dumazet, linux-audit, syzkaller, David Miller, Dmitry Vyukov
In-Reply-To: <CAM_iQpVPEJ2t29ENpT4qcBznwE83w_PEBOxStwyzDH27Si2Ppw@mail.gmail.com>
On 2016-12-13 16:17, Cong Wang wrote:
> On Tue, Dec 13, 2016 at 2:52 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > It is actually the audit_pid and audit_nlk_portid that I care about
> > more. The audit daemon could vanish or close the socket while the
> > kernel sock to which it was attached is still quite valid. Accessing
> > the set of three atomically is the urge. I wonder if it makes more
> > sense to test for the presence of auditd using audit_sock rather than
> > audit_pid, but still keep audit_pid for our reporting and replacement
> > strategy. Another idea would be to put the three in one struct.
>
> Note, the process has audit_pid should hold a refcnt to the netns too,
> so the netns can't be gone until that process is gone.
I noted that. I did wonder if there might be a problem if all the
processes were moved to another netns with the struct sock stuck in the
now process-void netns.
This is alluded-to in 6f285b19d09f ("audit: Send replies in the proper
network namespace.").
> > Can someone explain how they think the original test was able to trigger
> > this GPF? Network namespace shutdown while something pretended to set
> > up a new auditd? That's impressive for a fuzzer if that's the case...
> > Is there an strace? I guess it is all in test().
>
> I am surprised you still don't get the race condition even when you
> are now working on v2...
>
> The race happens in this scenarios :
>
> 1) Create a new netns
>
> 2) In the new netns, communicate with kauditd to set audit_sock
>
> 3) Generate some audit messages, so kauditd will keep sending them
> via audit_sock
>
> 4) exit the netns
>
> 5) the previous audit_sock is now going away, but kaudit_sock could still
> access it in this small window.
Ah ok that fits...
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: [RFC PATCH v3] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-14 4:00 UTC (permalink / raw)
To: Cong Wang
Cc: Linux Kernel Network Developers, LKML, Eric Dumazet, linux-audit,
Dmitry Vyukov
In-Reply-To: <CAM_iQpVFkNdEirvBDi8wV=iExt9BnCm3KU7+Q8oqhrJJtcnu9Q@mail.gmail.com>
On 2016-12-13 16:19, Cong Wang wrote:
> On Tue, Dec 13, 2016 at 7:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > @@ -1283,8 +1299,10 @@ static void __net_exit audit_net_exit(struct net *net)
> > {
> > struct audit_net *aunet = net_generic(net, audit_net_id);
> > struct sock *sock = aunet->nlsk;
> > + mutex_lock(&audit_cmd_mutex);
> > if (sock == audit_sock)
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
>
> This still doesn't look correct to me, b/c here we release the audit_sock
> refcnt twice:
>
> 1) inside audit_reset()
The audit_reset() refcount decrement corresponds to a setting of
audit_sock only if audit_sock is still non-NULL.
> 2) netlink_kernel_release()
This refcount decrement corresponds to netlink_kernel_create().
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: [RFC PATCH v3] audit: use proper refcount locking on audit_sock
From: Cong Wang @ 2016-12-14 0:19 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: Linux Kernel Network Developers, LKML, linux-audit, Dmitry Vyukov,
Eric Dumazet, Eric Paris, Paul Moore, sgrubb
In-Reply-To: <61c37ca790bc11bc023aea8f9b70ab3098aa30f5.1481626466.git.rgb@redhat.com>
On Tue, Dec 13, 2016 at 7:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> @@ -1283,8 +1299,10 @@ static void __net_exit audit_net_exit(struct net *net)
> {
> struct audit_net *aunet = net_generic(net, audit_net_id);
> struct sock *sock = aunet->nlsk;
> + mutex_lock(&audit_cmd_mutex);
> if (sock == audit_sock)
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
This still doesn't look correct to me, b/c here we release the audit_sock
refcnt twice:
1) inside audit_reset()
2) netlink_kernel_release()
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Cong Wang @ 2016-12-14 0:17 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: Herbert Xu, Johannes Berg, netdev, Florian Westphal, LKML,
Eric Dumazet, linux-audit, syzkaller, David Miller, Dmitry Vyukov
In-Reply-To: <20161213105233.GG1305@madcap2.tricolour.ca>
On Tue, Dec 13, 2016 at 2:52 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> It is actually the audit_pid and audit_nlk_portid that I care about
> more. The audit daemon could vanish or close the socket while the
> kernel sock to which it was attached is still quite valid. Accessing
> the set of three atomically is the urge. I wonder if it makes more
> sense to test for the presence of auditd using audit_sock rather than
> audit_pid, but still keep audit_pid for our reporting and replacement
> strategy. Another idea would be to put the three in one struct.
Note, the process has audit_pid should hold a refcnt to the netns too,
so the netns can't be gone until that process is gone.
>
> Can someone explain how they think the original test was able to trigger
> this GPF? Network namespace shutdown while something pretended to set
> up a new auditd? That's impressive for a fuzzer if that's the case...
> Is there an strace? I guess it is all in test().
>
I am surprised you still don't get the race condition even when you
are now working on v2...
The race happens in this scenarios :
1) Create a new netns
2) In the new netns, communicate with kauditd to set audit_sock
3) Generate some audit messages, so kauditd will keep sending them
via audit_sock
4) exit the netns
5) the previous audit_sock is now going away, but kaudit_sock could still
access it in this small window.
^ permalink raw reply
* Re: [RFC PATCH v3] audit: use proper refcount locking on audit_sock
From: Paul Moore @ 2016-12-13 20:50 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: netdev, linux-kernel, edumazet, linux-audit, xiyou.wangcong,
dvyukov
In-Reply-To: <61c37ca790bc11bc023aea8f9b70ab3098aa30f5.1481626466.git.rgb@redhat.com>
On Tue, Dec 13, 2016 at 10:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> Resetting audit_sock appears to be racy.
>
> audit_sock was being copied and dereferenced without using a refcount on
> the source sock.
>
> Bump the refcount on the underlying sock when we store a refrence in
> audit_sock and release it when we reset audit_sock. audit_sock
> modification needs the audit_cmd_mutex.
>
> See: https://lkml.org/lkml/2016/11/26/232
>
> Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> <xiyou.wangcong@gmail.com> on ideas how to fix it.
>
> Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> ---
> There has been a lot of change in the audit code that is about to go
> upstream to address audit queue issues. This patch is based on the
> source tree: git://git.infradead.org/users/pcmoore/audit#next
> ---
> kernel/audit.c | 28 +++++++++++++++++++++++-----
> 1 files changed, 23 insertions(+), 5 deletions(-)
This looks more reasonable. I still wonder about synchronization
between threads changing the audit_* connection variables and the
kauditd_thread, but I guess we can treat that as another issue; this
patch fixes a bug and is worth merging now.
I'm building a test kernel right now, assuming nothing blows up I'll
push this patch with the rest of the audit patches tomorrow; if
something bad happens, this is going to miss the first audit pull
request.
> diff --git a/kernel/audit.c b/kernel/audit.c
> index f20eee0..3bb4126 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -446,14 +446,19 @@ static void kauditd_retry_skb(struct sk_buff *skb)
> * Description:
> * Break the auditd/kauditd connection and move all the records in the retry
> * queue into the hold queue in case auditd reconnects.
> + * The audit_cmd_mutex must be held when calling this function.
> */
Don't resend, but in the future please start comments like this on the
previous line.
^ permalink raw reply
* [RFC PATCH v3] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-13 15:03 UTC (permalink / raw)
To: netdev, linux-kernel, linux-audit
Cc: Richard Guy Briggs, dvyukov, xiyou.wangcong, edumazet, eparis,
pmoore, sgrubb
In-Reply-To: <20161212100215.GA1305@madcap2.tricolour.ca>
Resetting audit_sock appears to be racy.
audit_sock was being copied and dereferenced without using a refcount on
the source sock.
Bump the refcount on the underlying sock when we store a refrence in
audit_sock and release it when we reset audit_sock. audit_sock
modification needs the audit_cmd_mutex.
See: https://lkml.org/lkml/2016/11/26/232
Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
<xiyou.wangcong@gmail.com> on ideas how to fix it.
Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
---
There has been a lot of change in the audit code that is about to go
upstream to address audit queue issues. This patch is based on the
source tree: git://git.infradead.org/users/pcmoore/audit#next
---
kernel/audit.c | 28 +++++++++++++++++++++++-----
1 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/kernel/audit.c b/kernel/audit.c
index f20eee0..3bb4126 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -446,14 +446,19 @@ static void kauditd_retry_skb(struct sk_buff *skb)
* Description:
* Break the auditd/kauditd connection and move all the records in the retry
* queue into the hold queue in case auditd reconnects.
+ * The audit_cmd_mutex must be held when calling this function.
*/
static void auditd_reset(void)
{
struct sk_buff *skb;
/* break the connection */
+ if (audit_sock) {
+ sock_put(audit_sock);
+ audit_sock = NULL;
+ }
audit_pid = 0;
- audit_sock = NULL;
+ audit_nlk_portid = 0;
/* flush all of the retry queue to the hold queue */
while ((skb = skb_dequeue(&audit_retry_queue)))
@@ -579,7 +584,9 @@ static int kauditd_thread(void *dummy)
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
}
} else
@@ -594,7 +601,9 @@ static int kauditd_thread(void *dummy)
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
kauditd_hold_skb(skb);
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
} else
/* temporary problem (we hope), queue
@@ -623,7 +632,9 @@ quick_loop:
if (rc) {
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
}
@@ -1010,11 +1021,16 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
}
if (audit_enabled != AUDIT_OFF)
audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
- audit_pid = new_pid;
- audit_nlk_portid = NETLINK_CB(skb).portid;
- audit_sock = skb->sk;
- if (!new_pid)
+ if (new_pid) {
+ if (audit_sock)
+ sock_put(audit_sock);
+ audit_pid = new_pid;
+ audit_nlk_portid = NETLINK_CB(skb).portid;
+ sock_hold(skb->sk);
+ audit_sock = skb->sk;
+ } else {
auditd_reset();
+ }
wake_up_interruptible(&kauditd_wait);
}
if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
@@ -1283,8 +1299,10 @@ static void __net_exit audit_net_exit(struct net *net)
{
struct audit_net *aunet = net_generic(net, audit_net_id);
struct sock *sock = aunet->nlsk;
+ mutex_lock(&audit_cmd_mutex);
if (sock == audit_sock)
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
RCU_INIT_POINTER(aunet->nlsk, NULL);
synchronize_net();
--
1.7.1
^ permalink raw reply related
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-13 15:01 UTC (permalink / raw)
To: Paul Moore
Cc: netdev, linux-kernel, linux-audit, edumazet, xiyou.wangcong,
dvyukov
In-Reply-To: <20161213051028.GE1305@madcap2.tricolour.ca>
On 2016-12-13 00:10, Richard Guy Briggs wrote:
> On 2016-12-12 15:18, Paul Moore wrote:
> > On Mon, Dec 12, 2016 at 5:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > > Resetting audit_sock appears to be racy.
> > >
> > > audit_sock was being copied and dereferenced without using a refcount on
> > > the source sock.
> > >
> > > Bump the refcount on the underlying sock when we store a refrence in
> > > audit_sock and release it when we reset audit_sock. audit_sock
> > > modification needs the audit_cmd_mutex.
> > >
> > > See: https://lkml.org/lkml/2016/11/26/232
> > >
> > > Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> > > <xiyou.wangcong@gmail.com> on ideas how to fix it.
> > >
> > > Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> > > ---
> > > There has been a lot of change in the audit code that is about to go
> > > upstream to address audit queue issues. This patch is based on the
> > > source tree: git://git.infradead.org/users/pcmoore/audit#next
> > > ---
> > > kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> > > 1 files changed, 28 insertions(+), 6 deletions(-)
> > > diff --git a/kernel/audit.c b/kernel/audit.c
> > > index f20eee0..439f7f3 100644
> > > --- a/kernel/audit.c
> > > +++ b/kernel/audit.c
> > > @@ -452,7 +452,9 @@ static void auditd_reset(void)
> > > struct sk_buff *skb;
> > >
> > > /* break the connection */
> > > + sock_put(audit_sock);
> > > audit_pid = 0;
> > > + audit_nlk_portid = 0;
> > > audit_sock = NULL;
> > >
> > > /* flush all of the retry queue to the hold queue */
> > > @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> > > if (rc >= 0) {
> > > consume_skb(skb);
> > > rc = 0;
> > > + } else {
> > > + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
> >
> > I dislike the way you wrote this because instead of simply looking at
> > this to see if it correct I need to sort out all the bits and find out
> > if there are other error codes that could run afoul of this check ...
> > make it simple, e.g. (rc == -ENOMEM || rc == -EPERM || ...).
> > Actually, since EPERM is 1, -EPERM (-1 in two's compliment is
> > 0xffffffff) is going to cause this to be true for pretty much any
> > value of rc, yes?
>
> Yes, you are correct. We need there a logical or on the results of each
> comparison to the return code rather than bit-wise or-ing the result
> codes together first to save a step.
>
> > > + mutex_lock(&audit_cmd_mutex);
> > > + auditd_reset();
> > > + mutex_unlock(&audit_cmd_mutex);
> > > + }
> >
> > The code in audit#next handles netlink_unicast() errors in
> > kauditd_thread() and you are adding error handling code here in
> > kauditd_send_unicast_skb() ... that's messy. I don't care too much
> > where the auditd_reset() call is made, but let's only do it in one
> > function; FWIW, I originally put the error handling code in
> > kauditd_thread() because there was other error handling code that
> > needed to done in that scope so it resulted in cleaner code.
>
> Hmmm, I seem to remember it not returning the return code and I thought
> I had changed it to do so, but I see now that it was already there.
> Agreed, I needlessly duplicated that error handling.
>
> > Related, I see you are now considering ENOMEM to be a fatal condition,
> > that differs from the AUDITD_BAD macro in kauditd_thread(); this
> > difference needs to be reconciled.
>
> Also correct about -EPERM now that I check back to the intent of commit
> 32a1dbaece7e ("audit: try harder to send to auditd upon netlink
> failure")
>
> > Finally, you should update the comment header block for auditd_reset()
> > that it needs to be called with the audit_cmd_mutex held.
>
> Yup.
>
> > > @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> > > return -EACCES;
> > > }
> > > if (audit_pid && new_pid &&
> > > - audit_replace(requesting_pid) != -ECONNREFUSED) {
> > > + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
> >
> > Do we simply want to treat any error here as fatal, and not just
> > ECONN/EPERM/ENOMEM? If not, let's come up with a single macro to
> > handle the fatal netlink_unicast() return codes so we have some chance
> > to keep things consistent in the future.
>
> I'll work through this before I post another patch...
Ok, I've gone back to look at the reasoning in commit 133e1e5acd4a
("audit: stop an old auditd being starved out by a new auditd") which
suggests only ECONNREFUSED can cause an audit_pid replace, so I've
returned it to its original state.
I'll post another tested patch, but I'm still not that happy that it
does not proactively reset audit_pid, audit_nlk_portid and audit_sock
when auditd's socket has a problem. I'll leave the test run overnight.
> > paul moore
>
> - RGB
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-13 14:55 UTC (permalink / raw)
To: Cong Wang
Cc: Linux Kernel Network Developers, LKML, linux-audit, Dmitry Vyukov,
Eric Dumazet, Eric Paris, Paul Moore, sgrubb
In-Reply-To: <CAM_iQpXAZOOn7G-EbEy1T11w6uoqwx5M8jVt=iHfOj4TJYsqpA@mail.gmail.com>
On 2016-12-12 15:58, Cong Wang wrote:
> On Mon, Dec 12, 2016 at 2:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > Resetting audit_sock appears to be racy.
> >
> > audit_sock was being copied and dereferenced without using a refcount on
> > the source sock.
> >
> > Bump the refcount on the underlying sock when we store a refrence in
> > audit_sock and release it when we reset audit_sock. audit_sock
> > modification needs the audit_cmd_mutex.
> >
> > See: https://lkml.org/lkml/2016/11/26/232
> >
> > Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> > <xiyou.wangcong@gmail.com> on ideas how to fix it.
> >
> > Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> > ---
> > There has been a lot of change in the audit code that is about to go
> > upstream to address audit queue issues. This patch is based on the
> > source tree: git://git.infradead.org/users/pcmoore/audit#next
> > ---
> > kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> > 1 files changed, 28 insertions(+), 6 deletions(-)
> >
> > diff --git a/kernel/audit.c b/kernel/audit.c
> > index f20eee0..439f7f3 100644
> > --- a/kernel/audit.c
> > +++ b/kernel/audit.c
> > @@ -452,7 +452,9 @@ static void auditd_reset(void)
> > struct sk_buff *skb;
> >
> > /* break the connection */
> > + sock_put(audit_sock);
>
> Why audit_sock can't be NULL here?
Fixed.
> > audit_pid = 0;
> > + audit_nlk_portid = 0;
> > audit_sock = NULL;
> >
> > /* flush all of the retry queue to the hold queue */
> > @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> > if (rc >= 0) {
> > consume_skb(skb);
> > rc = 0;
> > + } else {
> > + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
>
> Are these errno's bits??
No, I've fixed this silly error.
> > + mutex_lock(&audit_cmd_mutex);
> > + auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > + }
> > }
> >
> > return rc;
> > @@ -579,7 +587,9 @@ static int kauditd_thread(void *dummy)
> >
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > }
> > } else
> > @@ -594,7 +604,9 @@ static int kauditd_thread(void *dummy)
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > kauditd_hold_skb(skb);
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > } else
> > /* temporary problem (we hope), queue
> > @@ -623,7 +635,9 @@ quick_loop:
> > if (rc) {
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > }
> >
> > @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> > return -EACCES;
> > }
> > if (audit_pid && new_pid &&
> > - audit_replace(requesting_pid) != -ECONNREFUSED) {
> > + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
> > audit_log_config_change("audit_pid", new_pid, audit_pid, 0);
> > return -EEXIST;
> > }
> > if (audit_enabled != AUDIT_OFF)
> > audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> > - audit_pid = new_pid;
> > - audit_nlk_portid = NETLINK_CB(skb).portid;
> > - audit_sock = skb->sk;
> > - if (!new_pid)
> > + if (new_pid) {
> > + if (audit_sock)
> > + sock_put(audit_sock);
> > + audit_pid = new_pid;
> > + audit_nlk_portid = NETLINK_CB(skb).portid;
> > + sock_hold(skb->sk);
>
> Why refcnt is still needed here? I need it because I removed the code
> in net exit code path.
Because there is a chance that auditd exits abnormally and no message is
send from the kauditd thread to discover it has gone.
> > + audit_sock = skb->sk;
> > + } else {
> > auditd_reset();
> > + }
> > wake_up_interruptible(&kauditd_wait);
> > }
> > if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> > @@ -1283,8 +1302,11 @@ static void __net_exit audit_net_exit(struct net *net)
> > {
> > struct audit_net *aunet = net_generic(net, audit_net_id);
> > struct sock *sock = aunet->nlsk;
> > - if (sock == audit_sock)
> > + if (sock == audit_sock) {
> > + mutex_lock(&audit_cmd_mutex);
>
> You need to put the if check inside the mutex too. Again, this could be
> removed if you use refcnt.
Ok, right, fixed.
That last patch was a bit of a mess! Thanks for your patience in
checking it...
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > + }
> >
> > RCU_INIT_POINTER(aunet->nlsk, NULL);
> > synchronize_net();
> > --
> > 1.7.1
> >
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Richard Guy Briggs @ 2016-12-13 10:52 UTC (permalink / raw)
To: Cong Wang
Cc: linux-audit, Paul Moore, Dmitry Vyukov, David Miller,
Johannes Berg, Florian Westphal, Eric Dumazet, Herbert Xu, netdev,
LKML, syzkaller
In-Reply-To: <CAM_iQpUejChnsMWx3Z59GTuxZMeBjgrs2rhd7gsQAKoxE8YuDg@mail.gmail.com>
On 2016-12-12 16:10, Cong Wang wrote:
> On Mon, Dec 12, 2016 at 2:02 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > On 2016-12-09 20:13, Cong Wang wrote:
> >> Netlink notifier can safely be converted to blocking one, I will send
> >> a patch.
> >
> > I had a quick look at how that might happen. The netlink notifier chain
> > is atomic. Would the registered callback funciton need to spawn a
> > one-time thread to avoid blocking?
>
> It is already non-atomic now:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=efa172f42836477bf1ac3c9a3053140df764699c
Ok, that is recent... It is still less attractive as you point out due
to the overhead, but still worth considering if we can't find another
way.
> > I had a look at your patch. It looks attractively simple. The audit
> > next tree has patches queued that add an audit_reset function that will
> > require more work. I still see some potential gaps.
> >
> > - If the process messes up (or the sock lookup messes up) it is reset
> > in the kauditd thread under the audit_cmd_mutex.
> >
> > - If the process exits normally or is replaced due to an audit_replace
> > error, it is reset from audit_receive_skb under the audit_cmd_mutex.
> >
> > - If the process dies before the kauditd thread notices, either reap it
> > via notifier callback or it needs a check on net exit to reset. This
> > last one appears necessary to decrement the sock refcount so the sock
> > can be released in netlink_kernel_release().
> >
> > If we want to be proactive and use the netlink notifier, we assume the
> > overhead of adding to the netlink notifier chain and eliminate all the
> > other reset calls under the kauditd thread. If we are ok being
> > reactionary, then we'll at least need the net exit check on audit_sock.
>
> I don't see why we need to check it in net exit if we use refcnt,
> because we have two different users of audit_sock: kauditd and
> netns, if both take care of refcnt properly, we don't need to worry
> about who is the last, no matter what failures occur in what order.
It is actually the audit_pid and audit_nlk_portid that I care about
more. The audit daemon could vanish or close the socket while the
kernel sock to which it was attached is still quite valid. Accessing
the set of three atomically is the urge. I wonder if it makes more
sense to test for the presence of auditd using audit_sock rather than
audit_pid, but still keep audit_pid for our reporting and replacement
strategy. Another idea would be to put the three in one struct.
Can someone explain how they think the original test was able to trigger
this GPF? Network namespace shutdown while something pretended to set
up a new auditd? That's impressive for a fuzzer if that's the case...
Is there an strace? I guess it is all in test().
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Richard Guy Briggs @ 2016-12-13 8:28 UTC (permalink / raw)
To: Cong Wang
Cc: Herbert Xu, Johannes Berg, netdev, Florian Westphal, LKML,
Eric Dumazet, linux-audit, syzkaller, David Miller, Dmitry Vyukov
In-Reply-To: <20161213075117.GH22660@madcap2.tricolour.ca>
On 2016-12-13 02:51, Richard Guy Briggs wrote:
> On 2016-12-09 23:40, Cong Wang wrote:
> > On Fri, Dec 9, 2016 at 8:13 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> > > On Fri, Dec 9, 2016 at 3:01 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > >> On 2016-12-08 22:57, Cong Wang wrote:
> > >>> On Thu, Dec 8, 2016 at 10:02 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > >>> > I also tried to extend Cong Wang's idea to attempt to proactively respond to a
> > >>> > NETLINK_URELEASE on the audit_sock and reset it, but ran into a locking error
> > >>> > stack dump using mutex_lock(&audit_cmd_mutex) in the notifier callback.
> > >>> > Eliminating the lock since the sock is dead anways eliminates the error.
> > >>> >
> > >>> > Is it safe? I'll resubmit if this looks remotely sane. Meanwhile I'll try to
> > >>> > get the test case to compile.
> > >>>
> > >>> It doesn't look safe, because 'audit_sock', 'audit_nlk_portid' and 'audit_pid'
> > >>> are updated as a whole and race between audit_receive_msg() and
> > >>> NETLINK_URELEASE.
> > >>
> > >> This is what I expected and why I originally added the mutex lock in the
> > >> callback... The dumps I got were bare with no wrapper identifying the
> > >> process context or specific error, so I'm at a bit of a loss how to
> > >> solve this (without thinking more about it) other than instinctively
> > >> removing the mutex.
> > >
> > > Netlink notifier can safely be converted to blocking one, I will send
> > > a patch.
> > >
> > > But I seriously doubt you really need NETLINK_URELEASE here,
> > > it adds nothing but overhead, b/c the netlink notifier is called on
> > > every netlink socket in the system, but for net exit path, that is
> > > relatively a slow path.
> > >
> > > Also, kauditd_send_skb() needs audit_cmd_mutex too.
> >
> > Please let me know what you think about the attached patch?
> >
> > Thanks!
>
> > commit a12b43ee814625933ff155c20dc863c59cfcf240
> > Author: Cong Wang <xiyou.wangcong@gmail.com>
> > Date: Fri Dec 9 17:56:42 2016 -0800
> >
> > audit: close a race condition on audit_sock
> >
> > Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
> >
> > diff --git a/kernel/audit.c b/kernel/audit.c
> > index f1ca116..ab947d8 100644
> > --- a/kernel/audit.c
> > +++ b/kernel/audit.c
> > @@ -423,6 +423,8 @@ static void kauditd_send_skb(struct sk_buff *skb)
> > snprintf(s, sizeof(s), "audit_pid=%d reset", audit_pid);
> > audit_log_lost(s);
> > audit_pid = 0;
> > + audit_nlk_portid = 0;
> > + sock_put(audit_sock);
> > audit_sock = NULL;
> > } else {
> > pr_warn("re-scheduling(#%d) write to audit_pid=%d\n",
> > @@ -899,6 +901,9 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> > audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> > audit_pid = new_pid;
> > audit_nlk_portid = NETLINK_CB(skb).portid;
> > + sock_hold(skb->sk);
> > + if (audit_sock)
> > + sock_put(audit_sock);
> > audit_sock = skb->sk;
> > }
> > if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> > @@ -1167,10 +1172,6 @@ static void __net_exit audit_net_exit(struct net *net)
> > {
> > struct audit_net *aunet = net_generic(net, audit_net_id);
> > struct sock *sock = aunet->nlsk;
> > - if (sock == audit_sock) {
> > - audit_pid = 0;
> > - audit_sock = NULL;
> > - }
>
> So how does this not leak memory leaving the sock refcount incremented
> by the registered audit daemon when that daemon shuts down normally?
Sorry, that should have been: How does it not leak if auditd exits
abnormally without sending a shutdown message, but no message is sent on
the queue to trigger an error before the net namespace exits?
> - RGB
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Richard Guy Briggs @ 2016-12-13 7:51 UTC (permalink / raw)
To: Cong Wang
Cc: Herbert Xu, Johannes Berg, netdev, Florian Westphal, LKML,
Eric Dumazet, linux-audit, syzkaller, David Miller, Dmitry Vyukov
In-Reply-To: <CAM_iQpVcHGywXn90EpiSz-LsUDgKVqs-7BY-L7UBCu2VxkC31Q@mail.gmail.com>
On 2016-12-09 23:40, Cong Wang wrote:
> On Fri, Dec 9, 2016 at 8:13 PM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> > On Fri, Dec 9, 2016 at 3:01 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> >> On 2016-12-08 22:57, Cong Wang wrote:
> >>> On Thu, Dec 8, 2016 at 10:02 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> >>> > I also tried to extend Cong Wang's idea to attempt to proactively respond to a
> >>> > NETLINK_URELEASE on the audit_sock and reset it, but ran into a locking error
> >>> > stack dump using mutex_lock(&audit_cmd_mutex) in the notifier callback.
> >>> > Eliminating the lock since the sock is dead anways eliminates the error.
> >>> >
> >>> > Is it safe? I'll resubmit if this looks remotely sane. Meanwhile I'll try to
> >>> > get the test case to compile.
> >>>
> >>> It doesn't look safe, because 'audit_sock', 'audit_nlk_portid' and 'audit_pid'
> >>> are updated as a whole and race between audit_receive_msg() and
> >>> NETLINK_URELEASE.
> >>
> >> This is what I expected and why I originally added the mutex lock in the
> >> callback... The dumps I got were bare with no wrapper identifying the
> >> process context or specific error, so I'm at a bit of a loss how to
> >> solve this (without thinking more about it) other than instinctively
> >> removing the mutex.
> >
> > Netlink notifier can safely be converted to blocking one, I will send
> > a patch.
> >
> > But I seriously doubt you really need NETLINK_URELEASE here,
> > it adds nothing but overhead, b/c the netlink notifier is called on
> > every netlink socket in the system, but for net exit path, that is
> > relatively a slow path.
> >
> > Also, kauditd_send_skb() needs audit_cmd_mutex too.
>
> Please let me know what you think about the attached patch?
>
> Thanks!
> commit a12b43ee814625933ff155c20dc863c59cfcf240
> Author: Cong Wang <xiyou.wangcong@gmail.com>
> Date: Fri Dec 9 17:56:42 2016 -0800
>
> audit: close a race condition on audit_sock
>
> Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
>
> diff --git a/kernel/audit.c b/kernel/audit.c
> index f1ca116..ab947d8 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -423,6 +423,8 @@ static void kauditd_send_skb(struct sk_buff *skb)
> snprintf(s, sizeof(s), "audit_pid=%d reset", audit_pid);
> audit_log_lost(s);
> audit_pid = 0;
> + audit_nlk_portid = 0;
> + sock_put(audit_sock);
> audit_sock = NULL;
> } else {
> pr_warn("re-scheduling(#%d) write to audit_pid=%d\n",
> @@ -899,6 +901,9 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> audit_pid = new_pid;
> audit_nlk_portid = NETLINK_CB(skb).portid;
> + sock_hold(skb->sk);
> + if (audit_sock)
> + sock_put(audit_sock);
> audit_sock = skb->sk;
> }
> if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> @@ -1167,10 +1172,6 @@ static void __net_exit audit_net_exit(struct net *net)
> {
> struct audit_net *aunet = net_generic(net, audit_net_id);
> struct sock *sock = aunet->nlsk;
> - if (sock == audit_sock) {
> - audit_pid = 0;
> - audit_sock = NULL;
> - }
So how does this not leak memory leaving the sock refcount incremented
by the registered audit daemon when that daemon shuts down normally?
- RGB
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-13 5:10 UTC (permalink / raw)
To: Paul Moore
Cc: netdev, linux-kernel, linux-audit, edumazet, xiyou.wangcong,
dvyukov
In-Reply-To: <CAHC9VhSjU+-KDSUcjHZAPEwCDbyLYGvkmXYcUqSfSsa1+=DsRw@mail.gmail.com>
On 2016-12-12 15:18, Paul Moore wrote:
> On Mon, Dec 12, 2016 at 5:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > Resetting audit_sock appears to be racy.
> >
> > audit_sock was being copied and dereferenced without using a refcount on
> > the source sock.
> >
> > Bump the refcount on the underlying sock when we store a refrence in
> > audit_sock and release it when we reset audit_sock. audit_sock
> > modification needs the audit_cmd_mutex.
> >
> > See: https://lkml.org/lkml/2016/11/26/232
> >
> > Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> > <xiyou.wangcong@gmail.com> on ideas how to fix it.
> >
> > Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> > ---
> > There has been a lot of change in the audit code that is about to go
> > upstream to address audit queue issues. This patch is based on the
> > source tree: git://git.infradead.org/users/pcmoore/audit#next
> > ---
> > kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> > 1 files changed, 28 insertions(+), 6 deletions(-)
> > diff --git a/kernel/audit.c b/kernel/audit.c
> > index f20eee0..439f7f3 100644
> > --- a/kernel/audit.c
> > +++ b/kernel/audit.c
> > @@ -452,7 +452,9 @@ static void auditd_reset(void)
> > struct sk_buff *skb;
> >
> > /* break the connection */
> > + sock_put(audit_sock);
> > audit_pid = 0;
> > + audit_nlk_portid = 0;
> > audit_sock = NULL;
> >
> > /* flush all of the retry queue to the hold queue */
> > @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> > if (rc >= 0) {
> > consume_skb(skb);
> > rc = 0;
> > + } else {
> > + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
>
> I dislike the way you wrote this because instead of simply looking at
> this to see if it correct I need to sort out all the bits and find out
> if there are other error codes that could run afoul of this check ...
> make it simple, e.g. (rc == -ENOMEM || rc == -EPERM || ...).
> Actually, since EPERM is 1, -EPERM (-1 in two's compliment is
> 0xffffffff) is going to cause this to be true for pretty much any
> value of rc, yes?
Yes, you are correct. We need there a logical or on the results of each
comparison to the return code rather than bit-wise or-ing the result
codes together first to save a step.
> > + mutex_lock(&audit_cmd_mutex);
> > + auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > + }
>
> The code in audit#next handles netlink_unicast() errors in
> kauditd_thread() and you are adding error handling code here in
> kauditd_send_unicast_skb() ... that's messy. I don't care too much
> where the auditd_reset() call is made, but let's only do it in one
> function; FWIW, I originally put the error handling code in
> kauditd_thread() because there was other error handling code that
> needed to done in that scope so it resulted in cleaner code.
Hmmm, I seem to remember it not returning the return code and I thought
I had changed it to do so, but I see now that it was already there.
Agreed, I needlessly duplicated that error handling.
> Related, I see you are now considering ENOMEM to be a fatal condition,
> that differs from the AUDITD_BAD macro in kauditd_thread(); this
> difference needs to be reconciled.
Also correct about -EPERM now that I check back to the intent of commit
32a1dbaece7e ("audit: try harder to send to auditd upon netlink
failure")
> Finally, you should update the comment header block for auditd_reset()
> that it needs to be called with the audit_cmd_mutex held.
Yup.
> > @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> > return -EACCES;
> > }
> > if (audit_pid && new_pid &&
> > - audit_replace(requesting_pid) != -ECONNREFUSED) {
> > + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
>
> Do we simply want to treat any error here as fatal, and not just
> ECONN/EPERM/ENOMEM? If not, let's come up with a single macro to
> handle the fatal netlink_unicast() return codes so we have some chance
> to keep things consistent in the future.
I'll work through this before I post another patch...
> paul moore
- RGB
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-13 4:49 UTC (permalink / raw)
To: Paul Moore
Cc: netdev, linux-kernel, linux-audit, edumazet, xiyou.wangcong,
dvyukov
In-Reply-To: <CAHC9VhTt5Pbw+LzVwRV1E==drwrH0ihUJvJkWwDgOA35OUFV2g@mail.gmail.com>
On 2016-12-12 12:10, Paul Moore wrote:
> On Mon, Dec 12, 2016 at 5:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > Resetting audit_sock appears to be racy.
> >
> > audit_sock was being copied and dereferenced without using a refcount on
> > the source sock.
> >
> > Bump the refcount on the underlying sock when we store a refrence in
> > audit_sock and release it when we reset audit_sock. audit_sock
> > modification needs the audit_cmd_mutex.
> >
> > See: https://lkml.org/lkml/2016/11/26/232
> >
> > Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> > <xiyou.wangcong@gmail.com> on ideas how to fix it.
> >
> > Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> > ---
> > There has been a lot of change in the audit code that is about to go
> > upstream to address audit queue issues. This patch is based on the
> > source tree: git://git.infradead.org/users/pcmoore/audit#next
> > ---
> > kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> > 1 files changed, 28 insertions(+), 6 deletions(-)
>
> This is coming in pretty late for the v4.10 merge window, much later
> than I would usually take things, but this is arguably important, and
> (at first glance) relatively low risk - what testing have you done on
> this?
At this point, compile and boot, and I'm able to compile and run the
supplied test code without any issues.
> > diff --git a/kernel/audit.c b/kernel/audit.c
> > index f20eee0..439f7f3 100644
> > --- a/kernel/audit.c
> > +++ b/kernel/audit.c
> > @@ -452,7 +452,9 @@ static void auditd_reset(void)
> > struct sk_buff *skb;
> >
> > /* break the connection */
> > + sock_put(audit_sock);
> > audit_pid = 0;
> > + audit_nlk_portid = 0;
> > audit_sock = NULL;
> >
> > /* flush all of the retry queue to the hold queue */
> > @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> > if (rc >= 0) {
> > consume_skb(skb);
> > rc = 0;
> > + } else {
> > + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
> > + mutex_lock(&audit_cmd_mutex);
> > + auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > + }
> > }
> >
> > return rc;
> > @@ -579,7 +587,9 @@ static int kauditd_thread(void *dummy)
> >
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > }
> > } else
> > @@ -594,7 +604,9 @@ static int kauditd_thread(void *dummy)
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > kauditd_hold_skb(skb);
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > } else
> > /* temporary problem (we hope), queue
> > @@ -623,7 +635,9 @@ quick_loop:
> > if (rc) {
> > auditd = 0;
> > if (AUDITD_BAD(rc, reschedule)) {
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > reschedule = 0;
> > }
> >
> > @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> > return -EACCES;
> > }
> > if (audit_pid && new_pid &&
> > - audit_replace(requesting_pid) != -ECONNREFUSED) {
> > + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
> > audit_log_config_change("audit_pid", new_pid, audit_pid, 0);
> > return -EEXIST;
> > }
> > if (audit_enabled != AUDIT_OFF)
> > audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> > - audit_pid = new_pid;
> > - audit_nlk_portid = NETLINK_CB(skb).portid;
> > - audit_sock = skb->sk;
> > - if (!new_pid)
> > + if (new_pid) {
> > + if (audit_sock)
> > + sock_put(audit_sock);
> > + audit_pid = new_pid;
> > + audit_nlk_portid = NETLINK_CB(skb).portid;
> > + sock_hold(skb->sk);
> > + audit_sock = skb->sk;
> > + } else {
> > auditd_reset();
> > + }
> > wake_up_interruptible(&kauditd_wait);
> > }
> > if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> > @@ -1283,8 +1302,11 @@ static void __net_exit audit_net_exit(struct net *net)
> > {
> > struct audit_net *aunet = net_generic(net, audit_net_id);
> > struct sock *sock = aunet->nlsk;
> > - if (sock == audit_sock)
> > + if (sock == audit_sock) {
> > + mutex_lock(&audit_cmd_mutex);
> > auditd_reset();
> > + mutex_unlock(&audit_cmd_mutex);
> > + }
> >
> > RCU_INIT_POINTER(aunet->nlsk, NULL);
> > synchronize_net();
> > --
> > 1.7.1
> >
> > --
> > Linux-audit mailing list
> > Linux-audit@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit
>
>
>
> --
> paul moore
> www.paul-moore.com
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Cong Wang @ 2016-12-13 0:10 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: linux-audit, Paul Moore, Dmitry Vyukov, David Miller,
Johannes Berg, Florian Westphal, Eric Dumazet, Herbert Xu, netdev,
LKML, syzkaller
In-Reply-To: <20161212100215.GA1305@madcap2.tricolour.ca>
On Mon, Dec 12, 2016 at 2:02 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> On 2016-12-09 20:13, Cong Wang wrote:
>> Netlink notifier can safely be converted to blocking one, I will send
>> a patch.
>
> I had a quick look at how that might happen. The netlink notifier chain
> is atomic. Would the registered callback funciton need to spawn a
> one-time thread to avoid blocking?
It is already non-atomic now:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=efa172f42836477bf1ac3c9a3053140df764699c
> I had a look at your patch. It looks attractively simple. The audit
> next tree has patches queued that add an audit_reset function that will
> require more work. I still see some potential gaps.
>
> - If the process messes up (or the sock lookup messes up) it is reset
> in the kauditd thread under the audit_cmd_mutex.
>
> - If the process exits normally or is replaced due to an audit_replace
> error, it is reset from audit_receive_skb under the audit_cmd_mutex.
>
> - If the process dies before the kauditd thread notices, either reap it
> via notifier callback or it needs a check on net exit to reset. This
> last one appears necessary to decrement the sock refcount so the sock
> can be released in netlink_kernel_release().
>
> If we want to be proactive and use the netlink notifier, we assume the
> overhead of adding to the netlink notifier chain and eliminate all the
> other reset calls under the kauditd thread. If we are ok being
> reactionary, then we'll at least need the net exit check on audit_sock.
>
I don't see why we need to check it in net exit if we use refcnt,
because we have two different users of audit_sock: kauditd and
netns, if both take care of refcnt properly, we don't need to worry
about who is the last, no matter what failures occur in what order.
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Cong Wang @ 2016-12-12 23:58 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: Linux Kernel Network Developers, LKML, linux-audit, Dmitry Vyukov,
Eric Dumazet, Eric Paris, Paul Moore, sgrubb
In-Reply-To: <5714bd7468cfec225407a6c367e658478d590495.1481534171.git.rgb@redhat.com>
On Mon, Dec 12, 2016 at 2:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> Resetting audit_sock appears to be racy.
>
> audit_sock was being copied and dereferenced without using a refcount on
> the source sock.
>
> Bump the refcount on the underlying sock when we store a refrence in
> audit_sock and release it when we reset audit_sock. audit_sock
> modification needs the audit_cmd_mutex.
>
> See: https://lkml.org/lkml/2016/11/26/232
>
> Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> <xiyou.wangcong@gmail.com> on ideas how to fix it.
>
> Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> ---
> There has been a lot of change in the audit code that is about to go
> upstream to address audit queue issues. This patch is based on the
> source tree: git://git.infradead.org/users/pcmoore/audit#next
> ---
> kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> 1 files changed, 28 insertions(+), 6 deletions(-)
>
> diff --git a/kernel/audit.c b/kernel/audit.c
> index f20eee0..439f7f3 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -452,7 +452,9 @@ static void auditd_reset(void)
> struct sk_buff *skb;
>
> /* break the connection */
> + sock_put(audit_sock);
Why audit_sock can't be NULL here?
> audit_pid = 0;
> + audit_nlk_portid = 0;
> audit_sock = NULL;
>
> /* flush all of the retry queue to the hold queue */
> @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> if (rc >= 0) {
> consume_skb(skb);
> rc = 0;
> + } else {
> + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
Are these errno's bits??
> + mutex_lock(&audit_cmd_mutex);
> + auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> + }
> }
>
> return rc;
> @@ -579,7 +587,9 @@ static int kauditd_thread(void *dummy)
>
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> }
> } else
> @@ -594,7 +604,9 @@ static int kauditd_thread(void *dummy)
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> kauditd_hold_skb(skb);
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> } else
> /* temporary problem (we hope), queue
> @@ -623,7 +635,9 @@ quick_loop:
> if (rc) {
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> }
>
> @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> return -EACCES;
> }
> if (audit_pid && new_pid &&
> - audit_replace(requesting_pid) != -ECONNREFUSED) {
> + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
> audit_log_config_change("audit_pid", new_pid, audit_pid, 0);
> return -EEXIST;
> }
> if (audit_enabled != AUDIT_OFF)
> audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> - audit_pid = new_pid;
> - audit_nlk_portid = NETLINK_CB(skb).portid;
> - audit_sock = skb->sk;
> - if (!new_pid)
> + if (new_pid) {
> + if (audit_sock)
> + sock_put(audit_sock);
> + audit_pid = new_pid;
> + audit_nlk_portid = NETLINK_CB(skb).portid;
> + sock_hold(skb->sk);
Why refcnt is still needed here? I need it because I removed the code
in net exit code path.
> + audit_sock = skb->sk;
> + } else {
> auditd_reset();
> + }
> wake_up_interruptible(&kauditd_wait);
> }
> if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> @@ -1283,8 +1302,11 @@ static void __net_exit audit_net_exit(struct net *net)
> {
> struct audit_net *aunet = net_generic(net, audit_net_id);
> struct sock *sock = aunet->nlsk;
> - if (sock == audit_sock)
> + if (sock == audit_sock) {
> + mutex_lock(&audit_cmd_mutex);
You need to put the if check inside the mutex too. Again, this could be
removed if you use refcnt.
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> + }
>
> RCU_INIT_POINTER(aunet->nlsk, NULL);
> synchronize_net();
> --
> 1.7.1
>
^ permalink raw reply
* Re: [RFC][PATCH] audit: add feature audit_lost reset
From: Paul Moore @ 2016-12-12 20:53 UTC (permalink / raw)
To: Steve Grubb; +Cc: Richard Guy Briggs, linux-audit
In-Reply-To: <2846309.PeHxrRBX1K@x2>
On Sat, Dec 10, 2016 at 3:40 PM, Steve Grubb <sgrubb@redhat.com> wrote:
> On Friday, December 9, 2016 6:46:43 PM EST Paul Moore wrote:
>> > I would suggest that the return value (presuming it was reset when
>> > non-zero) or the audit record generated reporting the lost value
>> > reset would be sufficient confirmation that the feature exists on the
>> > running kernel and the addition to the feature bitmap is not strictly
>> > necessary, but you only find this out upon attempting that lost reset.
>> >
>> > Well, we haven't used much of that bitmap space and if it isn't to be
>> > used when needed, why is it there? If there is a relatively simple
>> > alternate non-destructive way to discover the presence of a feature use
>> > of the bitmap isn't necessary.
>>
>> My concern isn't the absolute consumption of the bitmap, but rather
>> the rate of the consumption.
>
> I'm not concerned much about it. There are very few more RFE's that are either
> in the pipeline or something I can think of that we need.
We always need to plan on more features, regardless of what you know
about today, something (many somethings actually) is almost certain to
come up in the future.
--
paul moore
www.paul-moore.com
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Paul Moore @ 2016-12-12 20:18 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: netdev, linux-kernel, linux-audit, edumazet, xiyou.wangcong,
dvyukov
In-Reply-To: <5714bd7468cfec225407a6c367e658478d590495.1481534171.git.rgb@redhat.com>
On Mon, Dec 12, 2016 at 5:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> Resetting audit_sock appears to be racy.
>
> audit_sock was being copied and dereferenced without using a refcount on
> the source sock.
>
> Bump the refcount on the underlying sock when we store a refrence in
> audit_sock and release it when we reset audit_sock. audit_sock
> modification needs the audit_cmd_mutex.
>
> See: https://lkml.org/lkml/2016/11/26/232
>
> Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> <xiyou.wangcong@gmail.com> on ideas how to fix it.
>
> Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> ---
> There has been a lot of change in the audit code that is about to go
> upstream to address audit queue issues. This patch is based on the
> source tree: git://git.infradead.org/users/pcmoore/audit#next
> ---
> kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> 1 files changed, 28 insertions(+), 6 deletions(-)
My previous question about testing still stands, but I took a closer
look and have some additional comments, see below ...
> diff --git a/kernel/audit.c b/kernel/audit.c
> index f20eee0..439f7f3 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -452,7 +452,9 @@ static void auditd_reset(void)
> struct sk_buff *skb;
>
> /* break the connection */
> + sock_put(audit_sock);
> audit_pid = 0;
> + audit_nlk_portid = 0;
> audit_sock = NULL;
>
> /* flush all of the retry queue to the hold queue */
> @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> if (rc >= 0) {
> consume_skb(skb);
> rc = 0;
> + } else {
> + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
I dislike the way you wrote this because instead of simply looking at
this to see if it correct I need to sort out all the bits and find out
if there are other error codes that could run afoul of this check ...
make it simple, e.g. (rc == -ENOMEM || rc == -EPERM || ...).
Actually, since EPERM is 1, -EPERM (-1 in two's compliment is
0xffffffff) is going to cause this to be true for pretty much any
value of rc, yes?
> + mutex_lock(&audit_cmd_mutex);
> + auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> + }
The code in audit#next handles netlink_unicast() errors in
kauditd_thread() and you are adding error handling code here in
kauditd_send_unicast_skb() ... that's messy. I don't care too much
where the auditd_reset() call is made, but let's only do it in one
function; FWIW, I originally put the error handling code in
kauditd_thread() because there was other error handling code that
needed to done in that scope so it resulted in cleaner code.
Related, I see you are now considering ENOMEM to be a fatal condition,
that differs from the AUDITD_BAD macro in kauditd_thread(); this
difference needs to be reconciled.
Finally, you should update the comment header block for auditd_reset()
that it needs to be called with the audit_cmd_mutex held.
> @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> return -EACCES;
> }
> if (audit_pid && new_pid &&
> - audit_replace(requesting_pid) != -ECONNREFUSED) {
> + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
Do we simply want to treat any error here as fatal, and not just
ECONN/EPERM/ENOMEM? If not, let's come up with a single macro to
handle the fatal netlink_unicast() return codes so we have some chance
to keep things consistent in the future.
--
paul moore
www.paul-moore.com
^ permalink raw reply
* Re: [PATCH v2] audit: use proper refcount locking on audit_sock
From: Paul Moore @ 2016-12-12 17:10 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: netdev, linux-kernel, edumazet, linux-audit, xiyou.wangcong,
dvyukov
In-Reply-To: <5714bd7468cfec225407a6c367e658478d590495.1481534171.git.rgb@redhat.com>
On Mon, Dec 12, 2016 at 5:03 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> Resetting audit_sock appears to be racy.
>
> audit_sock was being copied and dereferenced without using a refcount on
> the source sock.
>
> Bump the refcount on the underlying sock when we store a refrence in
> audit_sock and release it when we reset audit_sock. audit_sock
> modification needs the audit_cmd_mutex.
>
> See: https://lkml.org/lkml/2016/11/26/232
>
> Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
> <xiyou.wangcong@gmail.com> on ideas how to fix it.
>
> Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> ---
> There has been a lot of change in the audit code that is about to go
> upstream to address audit queue issues. This patch is based on the
> source tree: git://git.infradead.org/users/pcmoore/audit#next
> ---
> kernel/audit.c | 34 ++++++++++++++++++++++++++++------
> 1 files changed, 28 insertions(+), 6 deletions(-)
This is coming in pretty late for the v4.10 merge window, much later
than I would usually take things, but this is arguably important, and
(at first glance) relatively low risk - what testing have you done on
this?
> diff --git a/kernel/audit.c b/kernel/audit.c
> index f20eee0..439f7f3 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -452,7 +452,9 @@ static void auditd_reset(void)
> struct sk_buff *skb;
>
> /* break the connection */
> + sock_put(audit_sock);
> audit_pid = 0;
> + audit_nlk_portid = 0;
> audit_sock = NULL;
>
> /* flush all of the retry queue to the hold queue */
> @@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
> if (rc >= 0) {
> consume_skb(skb);
> rc = 0;
> + } else {
> + if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
> + mutex_lock(&audit_cmd_mutex);
> + auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> + }
> }
>
> return rc;
> @@ -579,7 +587,9 @@ static int kauditd_thread(void *dummy)
>
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> }
> } else
> @@ -594,7 +604,9 @@ static int kauditd_thread(void *dummy)
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> kauditd_hold_skb(skb);
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> } else
> /* temporary problem (we hope), queue
> @@ -623,7 +635,9 @@ quick_loop:
> if (rc) {
> auditd = 0;
> if (AUDITD_BAD(rc, reschedule)) {
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> reschedule = 0;
> }
>
> @@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
> return -EACCES;
> }
> if (audit_pid && new_pid &&
> - audit_replace(requesting_pid) != -ECONNREFUSED) {
> + (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
> audit_log_config_change("audit_pid", new_pid, audit_pid, 0);
> return -EEXIST;
> }
> if (audit_enabled != AUDIT_OFF)
> audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
> - audit_pid = new_pid;
> - audit_nlk_portid = NETLINK_CB(skb).portid;
> - audit_sock = skb->sk;
> - if (!new_pid)
> + if (new_pid) {
> + if (audit_sock)
> + sock_put(audit_sock);
> + audit_pid = new_pid;
> + audit_nlk_portid = NETLINK_CB(skb).portid;
> + sock_hold(skb->sk);
> + audit_sock = skb->sk;
> + } else {
> auditd_reset();
> + }
> wake_up_interruptible(&kauditd_wait);
> }
> if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
> @@ -1283,8 +1302,11 @@ static void __net_exit audit_net_exit(struct net *net)
> {
> struct audit_net *aunet = net_generic(net, audit_net_id);
> struct sock *sock = aunet->nlsk;
> - if (sock == audit_sock)
> + if (sock == audit_sock) {
> + mutex_lock(&audit_cmd_mutex);
> auditd_reset();
> + mutex_unlock(&audit_cmd_mutex);
> + }
>
> RCU_INIT_POINTER(aunet->nlsk, NULL);
> synchronize_net();
> --
> 1.7.1
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
--
paul moore
www.paul-moore.com
^ permalink raw reply
* Re: netlink: GPF in sock_sndtimeo
From: Dmitry Vyukov @ 2016-12-12 10:07 UTC (permalink / raw)
To: syzkaller
Cc: Richard Guy Briggs, linux-audit, Paul Moore, David Miller,
Johannes Berg, Florian Westphal, Eric Dumazet, Herbert Xu, netdev,
LKML
In-Reply-To: <CAM_iQpVcHGywXn90EpiSz-LsUDgKVqs-7BY-L7UBCu2VxkC31Q@mail.gmail.com>
On Sat, Dec 10, 2016 at 8:40 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>>> On 2016-12-08 22:57, Cong Wang wrote:
>>>> On Thu, Dec 8, 2016 at 10:02 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
>>>> > I also tried to extend Cong Wang's idea to attempt to proactively respond to a
>>>> > NETLINK_URELEASE on the audit_sock and reset it, but ran into a locking error
>>>> > stack dump using mutex_lock(&audit_cmd_mutex) in the notifier callback.
>>>> > Eliminating the lock since the sock is dead anways eliminates the error.
>>>> >
>>>> > Is it safe? I'll resubmit if this looks remotely sane. Meanwhile I'll try to
>>>> > get the test case to compile.
>>>>
>>>> It doesn't look safe, because 'audit_sock', 'audit_nlk_portid' and 'audit_pid'
>>>> are updated as a whole and race between audit_receive_msg() and
>>>> NETLINK_URELEASE.
>>>
>>> This is what I expected and why I originally added the mutex lock in the
>>> callback... The dumps I got were bare with no wrapper identifying the
>>> process context or specific error, so I'm at a bit of a loss how to
>>> solve this (without thinking more about it) other than instinctively
>>> removing the mutex.
>>
>> Netlink notifier can safely be converted to blocking one, I will send
>> a patch.
>>
>> But I seriously doubt you really need NETLINK_URELEASE here,
>> it adds nothing but overhead, b/c the netlink notifier is called on
>> every netlink socket in the system, but for net exit path, that is
>> relatively a slow path.
>>
>> Also, kauditd_send_skb() needs audit_cmd_mutex too.
>
> Please let me know what you think about the attached patch?
Applied the patch locally and have not seen the bug since then (~24
hours of testing).
^ permalink raw reply
* [PATCH v2] audit: use proper refcount locking on audit_sock
From: Richard Guy Briggs @ 2016-12-12 10:03 UTC (permalink / raw)
To: netdev, linux-kernel, linux-audit
Cc: Richard Guy Briggs, dvyukov, xiyou.wangcong, edumazet, eparis,
pmoore, sgrubb
In-Reply-To: <20161212100215.GA1305@madcap2.tricolour.ca>
Resetting audit_sock appears to be racy.
audit_sock was being copied and dereferenced without using a refcount on
the source sock.
Bump the refcount on the underlying sock when we store a refrence in
audit_sock and release it when we reset audit_sock. audit_sock
modification needs the audit_cmd_mutex.
See: https://lkml.org/lkml/2016/11/26/232
Thanks to Eric Dumazet <edumazet@google.com> and Cong Wang
<xiyou.wangcong@gmail.com> on ideas how to fix it.
Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
---
There has been a lot of change in the audit code that is about to go
upstream to address audit queue issues. This patch is based on the
source tree: git://git.infradead.org/users/pcmoore/audit#next
---
kernel/audit.c | 34 ++++++++++++++++++++++++++++------
1 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/kernel/audit.c b/kernel/audit.c
index f20eee0..439f7f3 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -452,7 +452,9 @@ static void auditd_reset(void)
struct sk_buff *skb;
/* break the connection */
+ sock_put(audit_sock);
audit_pid = 0;
+ audit_nlk_portid = 0;
audit_sock = NULL;
/* flush all of the retry queue to the hold queue */
@@ -478,6 +480,12 @@ static int kauditd_send_unicast_skb(struct sk_buff *skb)
if (rc >= 0) {
consume_skb(skb);
rc = 0;
+ } else {
+ if (rc & (-ENOMEM|-EPERM|-ECONNREFUSED)) {
+ mutex_lock(&audit_cmd_mutex);
+ auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
+ }
}
return rc;
@@ -579,7 +587,9 @@ static int kauditd_thread(void *dummy)
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
}
} else
@@ -594,7 +604,9 @@ static int kauditd_thread(void *dummy)
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
kauditd_hold_skb(skb);
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
} else
/* temporary problem (we hope), queue
@@ -623,7 +635,9 @@ quick_loop:
if (rc) {
auditd = 0;
if (AUDITD_BAD(rc, reschedule)) {
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
reschedule = 0;
}
@@ -1004,17 +1018,22 @@ static int audit_receive_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
return -EACCES;
}
if (audit_pid && new_pid &&
- audit_replace(requesting_pid) != -ECONNREFUSED) {
+ (audit_replace(requesting_pid) & (-ECONNREFUSED|-EPERM|-ENOMEM))) {
audit_log_config_change("audit_pid", new_pid, audit_pid, 0);
return -EEXIST;
}
if (audit_enabled != AUDIT_OFF)
audit_log_config_change("audit_pid", new_pid, audit_pid, 1);
- audit_pid = new_pid;
- audit_nlk_portid = NETLINK_CB(skb).portid;
- audit_sock = skb->sk;
- if (!new_pid)
+ if (new_pid) {
+ if (audit_sock)
+ sock_put(audit_sock);
+ audit_pid = new_pid;
+ audit_nlk_portid = NETLINK_CB(skb).portid;
+ sock_hold(skb->sk);
+ audit_sock = skb->sk;
+ } else {
auditd_reset();
+ }
wake_up_interruptible(&kauditd_wait);
}
if (s.mask & AUDIT_STATUS_RATE_LIMIT) {
@@ -1283,8 +1302,11 @@ static void __net_exit audit_net_exit(struct net *net)
{
struct audit_net *aunet = net_generic(net, audit_net_id);
struct sock *sock = aunet->nlsk;
- if (sock == audit_sock)
+ if (sock == audit_sock) {
+ mutex_lock(&audit_cmd_mutex);
auditd_reset();
+ mutex_unlock(&audit_cmd_mutex);
+ }
RCU_INIT_POINTER(aunet->nlsk, NULL);
synchronize_net();
--
1.7.1
^ permalink raw reply related
* Re: netlink: GPF in sock_sndtimeo
From: Richard Guy Briggs @ 2016-12-12 10:02 UTC (permalink / raw)
To: Cong Wang
Cc: linux-audit, Paul Moore, Dmitry Vyukov, David Miller,
Johannes Berg, Florian Westphal, Eric Dumazet, Herbert Xu, netdev,
LKML, syzkaller
In-Reply-To: <CAM_iQpV2GuKhR_1tD5jjACeD+pajJLws08CLmeYAo+rsjxvB0A@mail.gmail.com>
On 2016-12-09 20:13, Cong Wang wrote:
> On Fri, Dec 9, 2016 at 3:01 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > On 2016-12-08 22:57, Cong Wang wrote:
> >> On Thu, Dec 8, 2016 at 10:02 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
> >> > I also tried to extend Cong Wang's idea to attempt to proactively respond to a
> >> > NETLINK_URELEASE on the audit_sock and reset it, but ran into a locking error
> >> > stack dump using mutex_lock(&audit_cmd_mutex) in the notifier callback.
> >> > Eliminating the lock since the sock is dead anways eliminates the error.
> >> >
> >> > Is it safe? I'll resubmit if this looks remotely sane. Meanwhile I'll try to
> >> > get the test case to compile.
> >>
> >> It doesn't look safe, because 'audit_sock', 'audit_nlk_portid' and 'audit_pid'
> >> are updated as a whole and race between audit_receive_msg() and
> >> NETLINK_URELEASE.
> >
> > This is what I expected and why I originally added the mutex lock in the
> > callback... The dumps I got were bare with no wrapper identifying the
> > process context or specific error, so I'm at a bit of a loss how to
> > solve this (without thinking more about it) other than instinctively
> > removing the mutex.
>
> Netlink notifier can safely be converted to blocking one, I will send
> a patch.
I had a quick look at how that might happen. The netlink notifier chain
is atomic. Would the registered callback funciton need to spawn a
one-time thread to avoid blocking?
> But I seriously doubt you really need NETLINK_URELEASE here,
> it adds nothing but overhead, b/c the netlink notifier is called on
> every netlink socket in the system, but for net exit path, that is
> relatively a slow path.
I was a bit concerned about its overhead, but was hoping to update
audit_sock more quickly in the case of a sock shutting down for any
reason.
> Also, kauditd_send_skb() needs audit_cmd_mutex too.
Agreed.
> I will send a formal patch.
I had a look at your patch. It looks attractively simple. The audit
next tree has patches queued that add an audit_reset function that will
require more work. I still see some potential gaps.
- If the process messes up (or the sock lookup messes up) it is reset
in the kauditd thread under the audit_cmd_mutex.
- If the process exits normally or is replaced due to an audit_replace
error, it is reset from audit_receive_skb under the audit_cmd_mutex.
- If the process dies before the kauditd thread notices, either reap it
via notifier callback or it needs a check on net exit to reset. This
last one appears necessary to decrement the sock refcount so the sock
can be released in netlink_kernel_release().
If we want to be proactive and use the netlink notifier, we assume the
overhead of adding to the netlink notifier chain and eliminate all the
other reset calls under the kauditd thread. If we are ok being
reactionary, then we'll at least need the net exit check on audit_sock.
Have I understood this correctly?
I'll follow with a patch based on audit#next
There will be an upstream merge conflict between audit#next and net#next
due to the removal of:
RCU_INIT_POINTER(aunet->nlsk, NULL);
synchronize_net();
from the end of audit_net_exit(). This patch should probably go through
the audit maintainer due to the other anticipated merge conflicts.
> Thanks.
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Kernel Security Engineering, Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
* Re: [RFC][PATCH] audit: add feature audit_lost reset
From: Steve Grubb @ 2016-12-10 20:40 UTC (permalink / raw)
To: linux-audit; +Cc: Richard Guy Briggs
In-Reply-To: <CAHC9VhS-roP2arsepAFPzRdbfrADTp6i=mp0FAOGvT33GNCg7A@mail.gmail.com>
On Friday, December 9, 2016 6:46:43 PM EST Paul Moore wrote:
> > I would suggest that the return value (presuming it was reset when
> > non-zero) or the audit record generated reporting the lost value
> > reset would be sufficient confirmation that the feature exists on the
> > running kernel and the addition to the feature bitmap is not strictly
> > necessary, but you only find this out upon attempting that lost reset.
> >
> > Well, we haven't used much of that bitmap space and if it isn't to be
> > used when needed, why is it there? If there is a relatively simple
> > alternate non-destructive way to discover the presence of a feature use
> > of the bitmap isn't necessary.
>
> My concern isn't the absolute consumption of the bitmap, but rather
> the rate of the consumption.
I'm not concerned much about it. There are very few more RFE's that are either
in the pipeline or something I can think of that we need.
-Steve
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox