All of lore.kernel.org
 help / color / mirror / Atom feed
From: Eric Paris <eparis@redhat.com>
To: Richard Guy Briggs <rgb@redhat.com>
Cc: linux-audit@redhat.com, selinux@tycho.nsa.gov,
	linux-kernel@vger.kernel.org
Subject: Re: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
Date: Thu, 18 Dec 2014 12:50:16 -0500	[thread overview]
Message-ID: <1418925016.5162.21.camel@localhost> (raw)
In-Reply-To: <20141218174629.GB29827@madcap2.tricolour.ca>

On Thu, 2014-12-18 at 12:46 -0500, Richard Guy Briggs wrote:
> On 14/12/18, Eric Paris wrote:
> > On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks@vt.edu wrote:
> > > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > > Spotted these two while booting single-user on 20141216.  20141208
> > > > doesn't throw these, so it's something in the last week or so..
> > > 
> > > Gaah!  Turns out that 20141208 *is* susceptible - it had been booting
> > > just fine for several days, but it went around the bend, apparently due
> > > to a userspace or initrd change.
> > 
> > $5 says you updated systemd?
> > 
> > Richard?
> 
> Ok, so if you are correct, then either we justify dropping the lock (I
> assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
> or we make yet another queue were were hoping to avoid...
> 
> It would also be good to narrow it down to a rule that triggers this.

I thought the first message was enough to find the problem, but:

static void kauditd_send_multicast_skb(struct sk_buff *skb)
{
...
        nlmsg_multicast(sock, copy, 0, AUDIT_NLGRP_READLOG, GFP_KERNEL);
...
}

Since kauditd_send_multicast_skb() gets called in audit_log_end(), which
can come from any context (aka even a sleeping context) you can't use
GFP_KERNEL.  The audit_buffer know what context it should use.  So pass
that down and use that.

-Eric

> 
> > > egrep 'BUG|Linux vers' from my syslog:
> > > 
> > > Dec  9 12:19:53 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> ...
> > > Dec 12 19:42:30 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping function called from invalid context at mm/slab.c:2849
> ...
> > > Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG: sleeping function called from invalid context at mm/slab.c:2849
> > > Dec 12 20:51:33 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping function called from invalid context at mm/slab.c:2849
> ...
> > > I need to figure out what changed around 7:30PM on the 12th.
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

WARNING: multiple messages have this Message-ID (diff)
From: Eric Paris <eparis@redhat.com>
To: Richard Guy Briggs <rgb@redhat.com>
Cc: Valdis.Kletnieks@vt.edu, Paul Moore <paul@paul-moore.com>,
	linux-kernel@vger.kernel.org, selinux@tycho.nsa.gov,
	linux-audit@redhat.com
Subject: Re: linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849
Date: Thu, 18 Dec 2014 12:50:16 -0500	[thread overview]
Message-ID: <1418925016.5162.21.camel@localhost> (raw)
In-Reply-To: <20141218174629.GB29827@madcap2.tricolour.ca>

On Thu, 2014-12-18 at 12:46 -0500, Richard Guy Briggs wrote:
> On 14/12/18, Eric Paris wrote:
> > On Thu, 2014-12-18 at 11:45 -0500, Valdis.Kletnieks@vt.edu wrote:
> > > On Tue, 16 Dec 2014 20:09:54 -0500, Valdis Kletnieks said:
> > > > Spotted these two while booting single-user on 20141216.  20141208
> > > > doesn't throw these, so it's something in the last week or so..
> > > 
> > > Gaah!  Turns out that 20141208 *is* susceptible - it had been booting
> > > just fine for several days, but it went around the bend, apparently due
> > > to a userspace or initrd change.
> > 
> > $5 says you updated systemd?
> > 
> > Richard?
> 
> Ok, so if you are correct, then either we justify dropping the lock (I
> assume the one commone to both BUG reports [sig->cred_guard_mutex] ),
> or we make yet another queue were were hoping to avoid...
> 
> It would also be good to narrow it down to a rule that triggers this.

I thought the first message was enough to find the problem, but:

static void kauditd_send_multicast_skb(struct sk_buff *skb)
{
...
        nlmsg_multicast(sock, copy, 0, AUDIT_NLGRP_READLOG, GFP_KERNEL);
...
}

Since kauditd_send_multicast_skb() gets called in audit_log_end(), which
can come from any context (aka even a sleeping context) you can't use
GFP_KERNEL.  The audit_buffer know what context it should use.  So pass
that down and use that.

-Eric

> 
> > > egrep 'BUG|Linux vers' from my syslog:
> > > 
> > > Dec  9 12:19:53 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> ...
> > > Dec 12 19:42:30 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 20:00:39 turing-police kernel: [ 1109.635328] BUG: sleeping function called from invalid context at mm/slab.c:2849
> ...
> > > Dec 12 20:42:47 turing-police kernel: [ 3633.863552] BUG: sleeping function called from invalid context at mm/slab.c:2849
> > > Dec 12 20:51:33 turing-police kernel: [    0.000000] Linux version 3.18.0-next-20141208 (source@turing-police.cc.vt.edu) (gcc version 4.9.2 20141101 (Red Hat 4.9.2-1) (GCC) ) #27 SMP PREEMPT Mon Dec 8 22:20:07 EST 2014
> > > Dec 12 21:51:04 turing-police kernel: [ 3587.132867] BUG: sleeping function called from invalid context at mm/slab.c:2849
> ...
> > > I need to figure out what changed around 7:30PM on the 12th.
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
> Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545



  reply	other threads:[~2014-12-18 17:50 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-12-17  1:09 linux-next 20141216 BUG: sleeping function called from invalid context at mm/slab.c:2849 Valdis Kletnieks
2014-12-17  1:21 ` Eric Paris
2014-12-17  1:21   ` Eric Paris
2014-12-17 22:44   ` Richard Guy Briggs
2014-12-17 22:44     ` Richard Guy Briggs
2014-12-18 16:45 ` Valdis.Kletnieks
2014-12-18 16:50   ` Eric Paris
2014-12-18 16:50     ` Eric Paris
2014-12-18 17:46     ` Richard Guy Briggs
2014-12-18 17:46       ` Richard Guy Briggs
2014-12-18 17:50       ` Eric Paris [this message]
2014-12-18 17:50         ` Eric Paris
2014-12-18 18:44         ` Richard Guy Briggs
2014-12-18 18:44           ` Richard Guy Briggs
2014-12-18 19:04           ` Eric Paris
2014-12-18 19:04             ` Eric Paris
2014-12-18 20:03             ` Paul Moore
2014-12-18 20:03               ` Paul Moore
2014-12-18 19:21     ` Valdis.Kletnieks
2014-12-18 19:21       ` Valdis.Kletnieks
2014-12-18 19:24       ` Richard Guy Briggs
2014-12-18 19:24         ` Richard Guy Briggs

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1418925016.5162.21.camel@localhost \
    --to=eparis@redhat.com \
    --cc=linux-audit@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rgb@redhat.com \
    --cc=selinux@tycho.nsa.gov \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.