Top kernel oopses/warnings for the week of May 16th 2008

Top kernel oopses/warnings for the week of May 16th 2008

[Arjan van de Ven]

The http://www.kerneloops.org website collects kernel oops and
warning reports from various mailing lists and bugzillas as well as
with a client users can install to auto-submit oopses.
Below is a top 10 list of the oopses collected in the last 7 days.
(Reports prior to 2.6.23 have been omitted in collecting the top 10)

This week, a total of 1617 oopses and warnings have been reported,
compared to 452 reports in the previous week. This sharp increase
is due to Fedora 9 being released, which includes the automatic
collection client.


Per file statistics
-------------------
743	kernel/sysctl.c
113	fs/buffer.c
76	fs/sysfs/dir.c
38	kernel/spinlock.c
22	fs/inotify.c
21	kernel/sysctl_check.c (P)
21	net/core/sock.c
18	fs/file_table.c
17	mm/page_alloc.c
15	lib/iomap.c


Bug of the week
---------------
Not in the top 10 (but barely not so), but upcoming fast is a bug that has a very
distinct pattern.
The backtraces are at http://www.kerneloops.org/searchweek.php?search=fput

The pattern is that the kernel gets an invalid pointer passed to fput(),
coming down from a select() system call done by the "wpa_supplicant" program.
The fact that it is ONLY wpa_supplicant implicates the wireless/network stack.
Another observation is that this only happens with 64 bit kernels, even though
a large portion of the users uses 32 bit kernels. This implies that this is a 64-bit
type of bug. It appears that the top 32 bit of the pointers is getting corrupted
(the bottom part at least looks valid).



Top 10 reported bugs
--------------------

Rank 1: __register_sysctl_paths
	Reported 741 times (1254 total reports)
	Duplicate /proc registration. Bugs in madwifi but also in the parport driver
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.25-rc3.
	More info: http://www.kerneloops.org/searchweek.php?search=__register_sysctl_paths

Rank 2: mark_buffer_dirty
	Reported 110 times (306 total reports)
	EXT3 bug while hot-removing a USB device
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.24-rc6.
	More info: http://www.kerneloops.org/searchweek.php?search=mark_buffer_dirty

Rank 3: sysfs_add_one
	Reported 67 times (272 total reports)
	Adding duplicate sysfs files... seems to be mostly around USB so all in GregKH's park
	This oops was last seen in version 2.6.26-rc2, and first seen in 2.6.24-rc6.
	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one

Rank 4: _spin_unlock_irqrestore
	Reported 38 times (130 total reports)
	Mostly "softlockups" coming out of idle. This could well be mostly hardware issues;
	idle is the most harsh thing in terms of voltage/current swings.
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.22-rc1.
	More info: http://www.kerneloops.org/searchweek.php?search=_spin_unlock_irqrestore

Rank 5: ieee80211_stop_tx_ba_session
	Reported 36 times (56 total reports)
	Seems to be caused by the 4965 driver
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.25-rc7-git6.
	More info: http://www.kerneloops.org/searchweek.php?search=ieee80211_stop_tx_ba_session

Rank 6: nouveau_gpuobj_ref_del
	Reported 22 times (40 total reports)
	Bug in the out-of-tree nouveau driver
	This oops was last seen in version 2.6.25, and first seen in 2.6.25-rc4.
	More info: http://www.kerneloops.org/searchweek.php?search=nouveau_gpuobj_ref_del

Rank 7: set_dentry_child_flags
	Reported 22 times (741 total reports)
	Bug in the 2.6.24 inotify code that got exposed by KDE4 and got fixed in 2.6.25
	This oops was last seen in version 2.6.24.4, and first seen in 2.6.24-rc8-git4.
	More info: http://www.kerneloops.org/searchweek.php?search=set_dentry_child_flags

Rank 8: sysctl_check_lookup
	Reported 21 times (239 total reports)
	Bug in the proprietary madwifi driver
	Oops only shows up in tainted kernels
	This oops was last seen in version 2.6.24.5, and first seen in 2.6.24-rc5.
	More info: http://www.kerneloops.org/searchweek.php?search=sysctl_check_lookup

Rank 9: sk_free
	Reported 19 times (80 total reports)
	VMWare driver bug
	Oops only shows up in tainted kernels
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.23.9.
	More info: http://www.kerneloops.org/searchweek.php?search=sk_free

Rank 10: __alloc_pages
	Reported 16 times (31 total reports)
	Sleeping allocation in interrupt context, some in netlink, some in the nv sata driver
	This oops was last seen in version 2.6.25.3, and first seen in 2.6.18-rc1.
	More info: http://www.kerneloops.org/searchweek.php?search=__alloc_pages

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Evgeniy Polyakov]

On Fri, May 16, 2008 at 09:41:31AM -0700, Arjan van de Ven (arjan@linux.intel.com) wrote:
> Rank 10: __alloc_pages
> 	Reported 16 times (31 total reports)
> 	Sleeping allocation in interrupt context, some in netlink, some in 
> 	the nv sata driver
> 	This oops was last seen in version 2.6.25.3, and first seen in 
> 	2.6.18-rc1.
> 	More info: 
> 	http://www.kerneloops.org/searchweek.php?search=__alloc_pages

Number of them from via-velocity driver should be fixed by attached
patch (added Francois Romieu <romieu@fr.zoreil.com> to copy), but
frankly that looks really bad. Allocations are protected by lock, which
is used for interrupts, but that is safe, since device is turned off,
but also for suspend (which can free them again, btw), mii register dump
(will break without lock) and something else, which should be fine
though because of rtnl. What we could do better, is to allocate new
rings in advance, and only substitue pointers and write registers under
the lock, Francois?

diff --git a/drivers/net/via-velocity.c b/drivers/net/via-velocity.c
index 6b8d882..d6b7972 100644
--- a/drivers/net/via-velocity.c
+++ b/drivers/net/via-velocity.c
@@ -1251,7 +1251,7 @@ static int velocity_init_rd_ring(struct velocity_info *vptr)
 	vptr->rx_buf_sz = (mtu <= ETH_DATA_LEN) ? PKT_BUF_SZ : mtu + 32;
 
 	vptr->rd_info = kcalloc(vptr->options.numrx,
-				sizeof(struct velocity_rd_info), GFP_KERNEL);
+				sizeof(struct velocity_rd_info), GFP_ATOMIC);
 	if (!vptr->rd_info)
 		return -ENOMEM;
 
@@ -1324,7 +1324,7 @@ static int velocity_init_td_ring(struct velocity_info *vptr)
 
 		vptr->td_infos[j] = kcalloc(vptr->options.numtx,
 					    sizeof(struct velocity_td_info),
-					    GFP_KERNEL);
+					    GFP_ATOMIC);
 		if (!vptr->td_infos[j])	{
 			while(--j >= 0)
 				kfree(vptr->td_infos[j]);


-- 
	Evgeniy Polyakov

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Adrian Bunk]

On Fri, May 16, 2008 at 09:41:31AM -0700, Arjan van de Ven wrote:
>...
> Bug of the week
> ---------------
> Not in the top 10 (but barely not so), but upcoming fast is a bug that has a very
> distinct pattern.
> The backtraces are at http://www.kerneloops.org/searchweek.php?search=fput
>
> The pattern is that the kernel gets an invalid pointer passed to fput(),
> coming down from a select() system call done by the "wpa_supplicant" program.
> The fact that it is ONLY wpa_supplicant implicates the wireless/network stack.
> Another observation is that this only happens with 64 bit kernels, even though
> a large portion of the users uses 32 bit kernels. This implies that this is a 64-bit
> type of bug. It appears that the top 32 bit of the pointers is getting corrupted
> (the bottom part at least looks valid).
>...

Unless I misunderstand your webinterface another pattern is a "fc9" in 
the version string.

My first guess would be that it might be a problem in some code that is 
only in Fedora kernels?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Arjan van de Ven]

Adrian Bunk wrote:
> On Fri, May 16, 2008 at 09:41:31AM -0700, Arjan van de Ven wrote:
>> ...
>> Bug of the week
>> ---------------
>> Not in the top 10 (but barely not so), but upcoming fast is a bug that has a very
>> distinct pattern.
>> The backtraces are at http://www.kerneloops.org/searchweek.php?search=fput
>>
>> The pattern is that the kernel gets an invalid pointer passed to fput(),
>> coming down from a select() system call done by the "wpa_supplicant" program.
>> The fact that it is ONLY wpa_supplicant implicates the wireless/network stack.
>> Another observation is that this only happens with 64 bit kernels, even though
>> a large portion of the users uses 32 bit kernels. This implies that this is a 64-bit
>> type of bug. It appears that the top 32 bit of the pointers is getting corrupted
>> (the bottom part at least looks valid).
>> ...
> 
> Unless I misunderstand your webinterface another pattern is a "fc9" in 
> the version string.

that's because fc9 is the only OS that currently ships the client by default,
which means that it's a statistical thing where 90%+ of the reports come from
Fedora kernels, just because that's where the data is mined.

> 
> My first guess would be that it might be a problem in some code that is 
> only in Fedora kernels?

that may or may not be true, but we can't conclude that right now.

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Robert Hancock]

Arjan van de Ven wrote:
> Rank 10: __alloc_pages
>     Reported 16 times (31 total reports)
>     Sleeping allocation in interrupt context, some in netlink, some in 
> the nv sata driver
>     This oops was last seen in version 2.6.25.3, and first seen in 
> 2.6.18-rc1.
>     More info: 
> http://www.kerneloops.org/searchweek.php?search=__alloc_pages

In the case of the sata_nv error, it appears this is happening now 
because blk_queue_bounce_limit is initializing emergency ISA pools which 
can't be done under spinlock. This is happening because the code in 
blk_queue_bounce_limit now thinks that a 32-bit DMA mask requires 
allocating with GFP_DMA. This is only needed for a DMA mask less than 
32-bit, which is what the original code did. It looks like this was 
broken by this commit:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=00d61e3e8c12d5f395b167856d2b3c430816afb0

author	Andrea Arcangeli <andrea@qumranet.com>
	Wed, 2 Apr 2008 07:06:44 +0000 (09:06 +0200)
committer	Jens Axboe <jens.axboe@oracle.com>
	Wed, 2 Apr 2008 07:06:44 +0000 (09:06 +0200)

Fix bounce setting for 64-bit

Not sure what this was intended to fix, but I don't think it's right..

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Andrea Arcangeli]

On Fri, May 16, 2008 at 07:55:39PM -0600, Robert Hancock wrote:
> Arjan van de Ven wrote:
>> Rank 10: __alloc_pages
>>     Reported 16 times (31 total reports)
>>     Sleeping allocation in interrupt context, some in netlink, some in the 
>> nv sata driver
>>     This oops was last seen in version 2.6.25.3, and first seen in 
>> 2.6.18-rc1.
>>     More info: 
>> http://www.kerneloops.org/searchweek.php?search=__alloc_pages
>
> In the case of the sata_nv error, it appears this is happening now because 
> blk_queue_bounce_limit is initializing emergency ISA pools which can't be 
> done under spinlock. This is happening because the code in 
> blk_queue_bounce_limit now thinks that a 32-bit DMA mask requires 
> allocating with GFP_DMA. This is only needed for a DMA mask less than 
> 32-bit, which is what the original code did. It looks like this was broken 
> by this commit:

Looks like or you're certain? I ask because I had your exact same
problem with a regression introduced in 2.6.25-rc, and my patch
attempted to fix it. It looks like it wasn't enough to fix all of it,
but at least it looked like to improve things a bit to reduce the
regression impact without introducing any other problem compared to
the previous 2.6.25-rc code.

> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=00d61e3e8c12d5f395b167856d2b3c430816afb0
>
> author	Andrea Arcangeli <andrea@qumranet.com>
> 	Wed, 2 Apr 2008 07:06:44 +0000 (09:06 +0200)
> committer	Jens Axboe <jens.axboe@oracle.com>
> 	Wed, 2 Apr 2008 07:06:44 +0000 (09:06 +0200)
>
> Fix bounce setting for 64-bit
>
> Not sure what this was intended to fix, but I don't think it's right..

The reason I touched that code, is that a change introduced during
2.6.25-rc initialized the isa dma pool even if not necessary and that
broke the reserved-ram patch that requires no __GFP_DMA
allocations. There was no crash in 2.6.24 based kernels, the
regression started in 2.6.25-rc.

I think my patch isn't enough yet as I had another crash for the same
reason but it doesn't seem to trigger with all controllers, simulated
ata under kvm looked ok so I thought the regression was fixed after
the problem was gone under kvm, but it seems other hardware
configuration can still trigger the same problem. At least my patch
reduced the impact of the regression.

Please try to backout my patch, I suspect it'll make thing worse for
you and no btter. You'll have to backout the other change as well to
get back to 2.6.24 correct behavior like I still have too.

Then the fact that the isa pool may be initialized under spinlock that
seems another orthogonal problem, the reason I noticed this wasn't
because of some debug check but because there are no __GFP_DMA pages
in my boot.

Can't work on this right now, but I'm confident my change only
improved things, and the real trouble was with the previous commit.

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Arjan van de Ven]

Andrea Arcangeli wrote:
> The reason I touched that code, is that a change introduced during
> 2.6.25-rc initialized the isa dma pool even if not necessary and that
> broke the reserved-ram patch that requires no __GFP_DMA
> allocations. There was no crash in 2.6.24 based kernels, the
> regression started in 2.6.25-rc.

I'd not really call "breaks external patch" a regression ;)


What we really ought to be doing is always initialize the pool, from
the right process context. However, we need to make it such that we
can detect that there is zero __GFP_DMA memory in the system, and bail
out in that case. Doing it on-demand is just not going to fly; by that
time it's just too late (the pool may have been eaten already, the context
might be nasty to do allocations from etc etc).

the sata_nv driver has to do it on finding a cdrom; afaik it has something
like a different DMA mask for disks and cdroms, and it scales down once you
insert a CD.

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Robert Hancock]

Andrea Arcangeli wrote:
> On Fri, May 16, 2008 at 07:55:39PM -0600, Robert Hancock wrote:
>> Arjan van de Ven wrote:
>>> Rank 10: __alloc_pages
>>>     Reported 16 times (31 total reports)
>>>     Sleeping allocation in interrupt context, some in netlink, some in the 
>>> nv sata driver
>>>     This oops was last seen in version 2.6.25.3, and first seen in 
>>> 2.6.18-rc1.
>>>     More info: 
>>> http://www.kerneloops.org/searchweek.php?search=__alloc_pages
>> In the case of the sata_nv error, it appears this is happening now because 
>> blk_queue_bounce_limit is initializing emergency ISA pools which can't be 
>> done under spinlock. This is happening because the code in 
>> blk_queue_bounce_limit now thinks that a 32-bit DMA mask requires 
>> allocating with GFP_DMA. This is only needed for a DMA mask less than 
>> 32-bit, which is what the original code did. It looks like this was broken 
>> by this commit:
> 
> Looks like or you're certain? I ask because I had your exact same
> problem with a regression introduced in 2.6.25-rc, and my patch
> attempted to fix it. It looks like it wasn't enough to fix all of it,
> but at least it looked like to improve things a bit to reduce the
> regression impact without introducing any other problem compared to
> the previous 2.6.25-rc code.

What was the original patch that you were trying to fix? If it's this 
one, it does seem to be wrong:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=419c434c35614609fd0c79d335c134bf4b88b30b

author	Yang Shi <yang.shi@windriver.com>
	Tue, 4 Mar 2008 10:20:51 +0000 (11:20 +0100)
committer	Jens Axboe <jens.axboe@oracle.com>
	Tue, 4 Mar 2008 10:20:51 +0000 (11:20 +0100)

Fix DMA access of block device in 64-bit kernel on some non-x86 systems 
with 4GB or upper 4GB memory

Originally it was using the DMA path only for DMA masks of less than 
32-bit. This change made it use that path for 32-bit or less. Looking 
more closely it doesn't seem like your patch is really harmful, it just 
doesn't completely repair the damage from the first one.

IMO both of these patches should just be reverted. The commit 
description doesn't specify what arch the first one was trying to fix 
but it seems to break x86_64 anyway..

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Andrea Arcangeli]

On Sat, May 17, 2008 at 07:57:26AM -0700, Arjan van de Ven wrote:
> Andrea Arcangeli wrote:
>> The reason I touched that code, is that a change introduced during
>> 2.6.25-rc initialized the isa dma pool even if not necessary and that
>> broke the reserved-ram patch that requires no __GFP_DMA
>> allocations. There was no crash in 2.6.24 based kernels, the
>> regression started in 2.6.25-rc.
>
> I'd not really call "breaks external patch" a regression ;)

The external patch only allowed me to notice the regression when
nobody else noticed it. For mainline the regression was to put ram
into the bounce buffer pool even if no dma could ever require the
bounce buffering. There's no point to initialize the pool when total
ram < highest dma address. That is the regression. My patch turned the
regression from a waste of ram, to a kernel crash at boot. That's the
only relation between the reserved-ram patch this bug.

I assume Robert has a similar issue with some debugging code checking
for GFP_KERNEL allocations inside atomic context or similar, I assume
his driver has a bug and calls that function in the wrong context. But
if this didn't happen in 2.4.24, it means such bug has nothing to do
with the bug in blk-settings.c. It's just that such a bug or the
reserved-ram patches are required to notice the regression in
blk-settings.c.

rolling back to the code before this commit solves the problem for
me. The weird thing is that I'm quite sure I verified my patch
incremental with the below solved the crash at boot when run in qemu
but I also noticed later that sata didn't fix it.

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=419c434c35614609fd0c79d335c134bf4b88b30b

So there are two possibilities:

1) the current code in blk-settings is just fine after my last fix,
but there's yet another bug in this area in sata code unrelated to my
last fix

2) my last fix only fixed part of the problem and it only solved the
issue for ata or similar (my kvm setup where I tested my fix didn't
have any sata, but without my fix not even ata was working correctly)
and another fix is needed

> What we really ought to be doing is always initialize the pool, from
> the right process context. However, we need to make it such that we

Yes, that's the bug that probably Robert found, and that's also what
allowed him to notice we still have some regression in blk-settings.c
compared to 2.6.24, just like I could notice it thanks to the
reserved-ram patch.

> can detect that there is zero __GFP_DMA memory in the system, and bail
> out in that case. Doing it on-demand is just not going to fly; by that
> time it's just too late (the pool may have been eaten already, the context
> might be nasty to do allocations from etc etc).

That surely sounds nice, and it's yet another problem. But let's say
this last problem is low priority to solve because on any system where
we want to reserve the ram for the guest to run pci-passthrough dma on
hardware without VT-d, we can assume all devices are pci64 capable and
the current reserved-ram patch doesn't allow to reserve more than 2G
of the guest anyway because the host kernel has to run between -2GUL
and -1UL (included) so running on hosts with >4G of ram with pci32
devices attached isn't a top priority at the moment. Infact if
something there should also be a pool from the 32bit zone and not only
from the 24bit one.

> the sata_nv driver has to do it on finding a cdrom; afaik it has something
> like a different DMA mask for disks and cdroms, and it scales down once you
> insert a CD.

So that would surely give problems with reserved-ram. We'll sort this
out later. So for sata_nv it would be right to call the bounce
buffering, but in my hardware test the value was like 1000 and 1001,
with driver claiming 1001 required and b_pfn set to 1000 or similar,
so it was an off by one thing again with sata, and not a cdrom real
low dma mask.

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Robert Hancock]

Andrea Arcangeli wrote:
> On Sat, May 17, 2008 at 07:57:26AM -0700, Arjan van de Ven wrote:
>> Andrea Arcangeli wrote:
>>> The reason I touched that code, is that a change introduced during
>>> 2.6.25-rc initialized the isa dma pool even if not necessary and that
>>> broke the reserved-ram patch that requires no __GFP_DMA
>>> allocations. There was no crash in 2.6.24 based kernels, the
>>> regression started in 2.6.25-rc.
>> I'd not really call "breaks external patch" a regression ;)
> 
> The external patch only allowed me to notice the regression when
> nobody else noticed it. For mainline the regression was to put ram
> into the bounce buffer pool even if no dma could ever require the
> bounce buffering. There's no point to initialize the pool when total
> ram < highest dma address. That is the regression. My patch turned the
> regression from a waste of ram, to a kernel crash at boot. That's the
> only relation between the reserved-ram patch this bug.
> 
> I assume Robert has a similar issue with some debugging code checking
> for GFP_KERNEL allocations inside atomic context or similar, I assume
> his driver has a bug and calls that function in the wrong context. But
> if this didn't happen in 2.4.24, it means such bug has nothing to do
> with the bug in blk-settings.c. It's just that such a bug or the
> reserved-ram patches are required to notice the regression in
> blk-settings.c.

Well, it's not really documented what the locking semantics are supposed 
to be for blk_queue_bounce_limit. Based on the implementation, though, 
it's OK to call it under spin_lock_irqsave (it only sets some variables) 
unless you hit the case where dma is set to 1 and we do 
init_emergency_isa_pool. That's the problem, that case should not be hit 
with a DMA mask of 32-bit, but with Yang Shi's change to blk-settings.c, 
now we are.

The code in that function seems rather hackish, actually. It seems like 
a lot of those assumptions it's making should be in 
architecture-specific code..

Re: Top kernel oopses/warnings for the week of May 16th 2008

[Dave Jones]

On Fri, May 16, 2008 at 09:04:26PM +0300, Adrian Bunk wrote:
 > On Fri, May 16, 2008 at 09:41:31AM -0700, Arjan van de Ven wrote:
 > >...
 > > Bug of the week
 > > ---------------
 > > Not in the top 10 (but barely not so), but upcoming fast is a bug that has a very
 > > distinct pattern.
 > > The backtraces are at http://www.kerneloops.org/searchweek.php?search=fput
 > >
 > > The pattern is that the kernel gets an invalid pointer passed to fput(),
 > > coming down from a select() system call done by the "wpa_supplicant" program.
 > > The fact that it is ONLY wpa_supplicant implicates the wireless/network stack.
 > > Another observation is that this only happens with 64 bit kernels, even though
 > > a large portion of the users uses 32 bit kernels. This implies that this is a 64-bit
 > > type of bug. It appears that the top 32 bit of the pointers is getting corrupted
 > > (the bottom part at least looks valid).
 > >...
 > 
 > Unless I misunderstand your webinterface another pattern is a "fc9" in 
 > the version string.

Unsurprising really given we just did a release, and not many other distros
are enabling kerneloops by default yet.

 > My first guess would be that it might be a problem in some code that is 
 > only in Fedora kernels?

Very likely, though it's worth noting that all the wireless patches we have
in f9 are from wireless.git, so they're valid 2.6.26-rc bugs 

	Dave

-- 
http://www.codemonkey.org.uk