Kernel panic - not syncing: Fatal exception in interrupt (file_free_rcu+0x14)

Kernel panic - not syncing: Fatal exception in interrupt (file_free_rcu+0x14)

[Kyle Sanderson]

Not sure if the stack is crap or not, but this looks like an RCU crash?

https://i.imgur.com/sBnNe1p.jpg

Kyle.

FileServer ~ # uname -a
Linux FileServer.OpenWRT.local 4.12.5-gentoo #1 SMP PREEMPT Fri Aug 18
17:23:00 PDT 2017 x86_64 Intel(R) Atom(TM) CPU 330 @ 1.60GHz
GenuineIntel GNU/Linux
FileServer ~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 28
model name      : Intel(R) Atom(TM) CPU  330   @ 1.60GHz
stepping        : 2
microcode       : 0x20d
cpu MHz         : 1999.917
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm
constant_tsc arch_perfmon pebs bts nopl cpuid aperfmperf pni dtes64
monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm
bugs            :
bogomips        : 3999.83
clflush size    : 64
cache_alignment : 64
address sizes   : 32 bits physical, 48 bits virtual
power management:

Re: Kernel panic - not syncing: Fatal exception in interrupt (file_free_rcu+0x14)

[Linus Torvalds]

On Thu, Sep 28, 2017 at 8:32 PM, Kyle Sanderson <kyle.leet@gmail.com> wrote:
> Not sure if the stack is crap or not, but this looks like an RCU crash?
>
> https://i.imgur.com/sBnNe1p.jpg

Hmm. Not the clearest picture, and the "Code:" line in particular is
missing the interesting part, but at a guess it's taking a fault in
put_cred(), which inlines to

        if (atomic_dec_and_test(&(cred)->usage))
                __put_cred(cred);

and I think it's that "cred" pointer that may be NULL, which makes
"&(cred)->usage" be a NULL pointer too, and you get a page fault when
it tries to decrement the usage count.

Now, it goes without saying that the cred pointer should never *be*
NULL on a filp that is on the RCU freeing list, because we always
initialize file->f_cred when we allocate a file to the current creds.

So there's something odd going on. Possibly entirely unrelated memory
corruption.

Nothing obvious stands out, I think we'd need to see more of a pattern
of the problem to see what is up.

                 Linus

Re: Kernel panic - not syncing: Fatal exception in interrupt (file_free_rcu+0x14)

[Paul E. McKenney]

On Thu, Sep 28, 2017 at 09:00:52PM -0700, Linus Torvalds wrote:
> On Thu, Sep 28, 2017 at 8:32 PM, Kyle Sanderson <kyle.leet@gmail.com> wrote:
> > Not sure if the stack is crap or not, but this looks like an RCU crash?
> >
> > https://i.imgur.com/sBnNe1p.jpg
> 
> Hmm. Not the clearest picture, and the "Code:" line in particular is
> missing the interesting part, but at a guess it's taking a fault in
> put_cred(), which inlines to
> 
>         if (atomic_dec_and_test(&(cred)->usage))
>                 __put_cred(cred);
> 
> and I think it's that "cred" pointer that may be NULL, which makes
> "&(cred)->usage" be a NULL pointer too, and you get a page fault when
> it tries to decrement the usage count.
> 
> Now, it goes without saying that the cred pointer should never *be*
> NULL on a filp that is on the RCU freeing list, because we always
> initialize file->f_cred when we allocate a file to the current creds.
> 
> So there's something odd going on. Possibly entirely unrelated memory
> corruption.
> 
> Nothing obvious stands out, I think we'd need to see more of a pattern
> of the problem to see what is up.

Kyle, if this is reproducible, please build your kernel with
CONFIG_DEBUG_OBJECTS_RCU_HEAD=y.  This can detect the call_rcu()
equivalent of a double free, and these double frees have sometimes
resulted in output looking something like the .jpg along with the
NULL-pointer problem that Linus suspects.

							Thanx, Paul