Oops/Warning report for the week of September 10th, 2008

Oops/Warning report for the week of September 10th, 2008

[Arjan van de Ven]

The http://www.kerneloops.org website collects kernel oops and
warning reports from various mailing lists and bugzillas as well as
with a client users can install to auto-submit oopses.
Below is a top 13 list of the traces collected in the last 7 days.
(Reports prior to 2.6.23 have been omitted in collecting the top 10)

This week, a total of 3106 oopses and warnings have been reported,
compared to 1758 reports in the previous week.


Per file statistics
295	fs/jbd/journal.c
226	fs/sysfs/dir.c
179	drivers/parport/procfs.c
145	external/fireglx/binary (P)
131	drivers/ssb/main.c
126	net/core/dev.c
110	kernel/timer/tglx.c
65	fs/ext3/super.c
62	net/sched/sch_generic.c
60	kernel/time/tick-broadcast.c
49	drivers/base/power/main.c
45	drivers/gpu/drm/drm_drv.c


Rank 1: journal_update_superblock (warning)
	Reported 287 times (3559 total reports)
	Likely caused by the user removing a USB stick while mounted
	This warning was last seen in version 2.6.27-rc5-git9, and first seen in 2.6.24-rc6-git1.
	More info: http://www.kerneloops.org/searchweek.php?search=journal_update_superblock

Rank 2: sysfs_add_one (warning)
	Reported 215 times (3751 total reports)
	Duplicate sysfs registration; mostly in USB audio
	This warning was last seen in version 2.6.27-rc5, and first seen in 2.6.24-rc6.
	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one

Rank 3: parport_device_proc_register (warning)
	Reported 179 times (3764 total reports)
	Duplicate /proc registration in the parport driver
	This warning was last seen in version 2.6.26.3, and first seen in 2.6.24-rc5.
	More info: http://www.kerneloops.org/searchweek.php?search=parport_device_proc_register

Rank 4: firegl_ioctl (oops)
	Reported 139 times (2360 total reports)
	[external] Bug in the proprietary fireglx driver
	oops only shows up in tainted kernels
	This warning was last seen in version 2.6.26, and first seen in 2.6.25.
	More info: http://www.kerneloops.org/searchweek.php?search=firegl_ioctl

Rank 5: ssb_tmslow_reject_bitmask (warning)
	New entry due to the Debian kernel having this bug, and Debian now including
	the data collection client
	Reported 131 times (770 total reports)
	This warning was last seen in version 2.6.26, and first seen in 2.6.24.
	More info: http://www.kerneloops.org/searchweek.php?search=ssb_tmslow_reject_bitmask

Rank 6: tick_dev_program_event (warning)
	Reported 110 times
	[debug] temporary WARN_ON() inserted in -rc5 for diagnostics purposes, removed for -rc6
	This warning was last seen in version 2.6.27-rc5-git10, and first seen in 2.6.27-rc5.
	More info: http://www.kerneloops.org/searchweek.php?search=tick_dev_program_event

Rank 7: __netif_schedule (warning)
	Reported 104 times (501 total reports)
	[fixed] issue as a result of the multiqueue feature, fixed for 2.6.27-rc1
	This warning was last seen in version 2.6.27-rc0-git11, and first seen in 2.6.26-rc8-git.
	More info: http://www.kerneloops.org/searchweek.php?search=__netif_schedule

Rank 8: ext3_commit_super (warning)
	Reported 61 times (928 total reports)
	Likely caused by the user removing a USB stick while mounted
	This warning was last seen in version 2.6.27-rc2-git1, and first seen in 2.6.24.
	More info: http://www.kerneloops.org/searchweek.php?search=ext3_commit_super

Rank 9: dev_watchdog (warning)
	Network card transmit timeout; several sis900's and others. Dave has a simple patched queued
	up for 2.6.28 that will include the driver name in the warning, to help diagnose this issue
	better.
	Reported 61 times (241 total reports)
	This warning was last seen in version 2.6.27-rc5-git7, and first seen in 2.6.26-rc3.
	More info: http://www.kerneloops.org/searchweek.php?search=dev_watchdog

Rank 10: tick_broadcast_oneshot_control (oops)
	Reported 58 times (1331 total reports)
	Some interaction betwee tickless and system with an AMD CPU and an ATI chipset
	This oops was last seen in version 2.6.25.14, and first seen in 2.6.24-rc4.
	More info: http://www.kerneloops.org/searchweek.php?search=tick_broadcast_oneshot_control

Rank 11: drm_lastclose (oops)
	Reported 45 times (127 total reports)
	[fedora] Bug in the Fedora DRM patches
	This oops was last seen in version 2.6.25.14, and first seen in 2.6.25.
	More info: http://www.kerneloops.org/searchweek.php?search=drm_lastclose

Rank 12: ata_hsm_move (oops)
	Reported 39 times (660 total reports)
	Long standing libata bug
	This oops was last seen in version 2.6.25.14, and first seen in 2.6.25-rc9-git1.
	More info: http://www.kerneloops.org/searchweek.php?search=ata_hsm_move

Rank 13: rs_get_rate (warning)
	Reported 38 times (284 total reports)
	Bug in the Intel IWL wireless drivers
	This warning was last seen in version 2.6.27-rc1-git5, and first seen in 2.6.25-rc2-git5.
	More info: http://www.kerneloops.org/searchweek.php?search=rs_get_rate

Rank 14: empress_querycap (oops)
	Reported 28 times (261 total reports)
	Null pointer access in the V4L layer
	This oops was last seen in version 2.6.25.14, and first seen in 2.6.25-rc8-git4.
	More info: http://www.kerneloops.org/searchweek.php?search=empress_querycap

Re: Oops/Warning report for the week of September 10th, 2008

[Thomas Gleixner]

> 
> Rank 6: tick_dev_program_event (warning)
> 	Reported 110 times
> 	[debug] temporary WARN_ON() inserted in -rc5 for diagnostics purposes,
> removed for -rc6
> 	This warning was last seen in version 2.6.27-rc5-git10, and first seen
> in 2.6.27-rc5.
> 	More info:
> http://www.kerneloops.org/searchweek.php?search=tick_dev_program_event

WARN_ON() is removed in 2.6.27-rc6. It served it's purpose.
 
> Rank 10: tick_broadcast_oneshot_control (oops)
> 	Reported 58 times (1331 total reports)
> 	Some interaction betwee tickless and system with an AMD CPU and an ATI
> chipset
> 	This oops was last seen in version 2.6.25.14, and first seen in
> 2.6.24-rc4.
> 	More info:
> http://www.kerneloops.org/searchweek.php?search=tick_broadcast_oneshot_control

Fixed by the latest updates/fixes to clock events core and hpet
code. Working on a backport to 25/26 right now.

Thanks,

	tglx

Re: Oops/Warning report for the week of September 10th, 2008

[Greg KH]

On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>
> Rank 2: sysfs_add_one (warning)
> 	Reported 215 times (3751 total reports)
> 	Duplicate sysfs registration; mostly in USB audio
> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
> 2.6.24-rc6.
> 	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one

This one is going to be a bit tough to track over time as it is the low
level sysfs code complaining that a user of it is doing something wrong.
So you can get all sorts of different callers causing the same warning,
we just have to look at the backtrace to get a hint of who is causing
the problem.

Is there any way to break this one down further by caller so we can try
to narrow it down?  I thought you did that for other types of warnings
in the past (may_sleep(), etc.)

thanks,

greg k-h

Re: Oops/Warning report for the week of September 10th, 2008

[Arjan van de Ven]

Greg KH wrote:
> On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>> Rank 2: sysfs_add_one (warning)
>> 	Reported 215 times (3751 total reports)
>> 	Duplicate sysfs registration; mostly in USB audio
>> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
>> 2.6.24-rc6.
>> 	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
> 
> This one is going to be a bit tough to track over time as it is the low
> level sysfs code complaining that a user of it is doing something wrong.
> So you can get all sorts of different callers causing the same warning,
> we just have to look at the backtrace to get a hint of who is causing
> the problem.
> 
> Is there any way to break this one down further by caller so we can try
> to narrow it down?  I thought you did that for other types of warnings
> in the past (may_sleep(), etc.)
> 

it's.. a harder one than that.
All the cases so far were "if it's <this function> for <this class>, go one
down in the stacktrace", which I've done table driven.

For sysfs_add_one() it seems to be "if it is sysfs_add_one, dive into the
stacktrace to pick the one below device_add except if that is device_register
or device_create, because then you pick the one below that instead"

Not impossible, just going to take me more than the 3 minutes it normally takes ;-)

(but please correct me if my description of the heuristic is wrong)

Re: Oops/Warning report for the week of September 10th, 2008

[Arjan van de Ven]

Greg KH wrote:
> On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>> Rank 2: sysfs_add_one (warning)
>> 	Reported 215 times (3751 total reports)
>> 	Duplicate sysfs registration; mostly in USB audio
>> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
>> 2.6.24-rc6.
>> 	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
> 
> 
> Is there any way to break this one down further by caller so we can try
> to narrow it down?  I thought you did that for other types of warnings
> in the past (may_sleep(), etc.)

ok I think I got this one sorted;
your entry is now replaced by

Rank 4: snd_card_register (warning)
	Reported 146 times (2594 total reports)
	This warning was last seen in version 2.6.25.14, and first seen in 2.6.24.
	More info: http://www.kerneloops.org/searchweek.php?search=snd_card_register

which is ALL usb audio... (so I might have to also skip snd_card_register in the backtrace, but I'll leave that between you and the alsa guys)

Re: Oops/Warning report for the week of September 10th, 2008

[Greg KH]

On Thu, Sep 11, 2008 at 06:56:25AM -0700, Arjan van de Ven wrote:
> Greg KH wrote:
>> On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>>> Rank 2: sysfs_add_one (warning)
>>> 	Reported 215 times (3751 total reports)
>>> 	Duplicate sysfs registration; mostly in USB audio
>>> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
>>> 2.6.24-rc6.
>>> 	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
>> Is there any way to break this one down further by caller so we can try
>> to narrow it down?  I thought you did that for other types of warnings
>> in the past (may_sleep(), etc.)
>
> ok I think I got this one sorted;
> your entry is now replaced by
>
> Rank 4: snd_card_register (warning)
> 	Reported 146 times (2594 total reports)
> 	This warning was last seen in version 2.6.25.14, and first seen in 2.6.24.
> 	More info: 
> http://www.kerneloops.org/searchweek.php?search=snd_card_register
>
> which is ALL usb audio... (so I might have to also skip snd_card_register 
> in the backtrace, but I'll leave that between you and the alsa guys)

Yes, that's much better thanks.  I think that has now been fixed by
the ALSA people, as seen by no reports 2.6.26 and onward.

thanks,

greg k-h

Re: Oops/Warning report for the week of September 10th, 2008

[Arjan van de Ven]

Greg KH wrote:
> On Thu, Sep 11, 2008 at 06:56:25AM -0700, Arjan van de Ven wrote:
>> Greg KH wrote:
>>> On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>>>> Rank 2: sysfs_add_one (warning)
>>>> 	Reported 215 times (3751 total reports)
>>>> 	Duplicate sysfs registration; mostly in USB audio
>>>> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
>>>> 2.6.24-rc6.
>>>> 	More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
>>> Is there any way to break this one down further by caller so we can try
>>> to narrow it down?  I thought you did that for other types of warnings
>>> in the past (may_sleep(), etc.)
>> ok I think I got this one sorted;
>> your entry is now replaced by
>>
>> Rank 4: snd_card_register (warning)
>> 	Reported 146 times (2594 total reports)
>> 	This warning was last seen in version 2.6.25.14, and first seen in 2.6.24.
>> 	More info: 
>> http://www.kerneloops.org/searchweek.php?search=snd_card_register
>>
>> which is ALL usb audio... (so I might have to also skip snd_card_register 
>> in the backtrace, but I'll leave that between you and the alsa guys)
> 
> Yes, that's much better thanks.  I think that has now been fixed by
> the ALSA people, as seen by no reports 2.6.26 and onward.

sounds like a -stable candidate to me

(also be careful with the "2.6.26+" comment; most test data is from 2.6.25 still
until Fedora pushes their 2.6.26 kernel)

Re: Oops/Warning report for the week of September 10th, 2008

[Greg KH]

On Thu, Sep 11, 2008 at 01:54:36PM -0700, Arjan van de Ven wrote:
> Greg KH wrote:
>> On Thu, Sep 11, 2008 at 06:56:25AM -0700, Arjan van de Ven wrote:
>>> Greg KH wrote:
>>>> On Wed, Sep 10, 2008 at 12:16:48PM -0700, Arjan van de Ven wrote:
>>>>> Rank 2: sysfs_add_one (warning)
>>>>> 	Reported 215 times (3751 total reports)
>>>>> 	Duplicate sysfs registration; mostly in USB audio
>>>>> 	This warning was last seen in version 2.6.27-rc5, and first seen in 
>>>>> 2.6.24-rc6.
>>>>> 	More info: 
>>>>> http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
>>>> Is there any way to break this one down further by caller so we can try
>>>> to narrow it down?  I thought you did that for other types of warnings
>>>> in the past (may_sleep(), etc.)
>>> ok I think I got this one sorted;
>>> your entry is now replaced by
>>>
>>> Rank 4: snd_card_register (warning)
>>> 	Reported 146 times (2594 total reports)
>>> 	This warning was last seen in version 2.6.25.14, and first seen in 
>>> 2.6.24.
>>> 	More info: 
>>> http://www.kerneloops.org/searchweek.php?search=snd_card_register
>>>
>>> which is ALL usb audio... (so I might have to also skip snd_card_register 
>>> in the backtrace, but I'll leave that between you and the alsa guys)
>> Yes, that's much better thanks.  I think that has now been fixed by
>> the ALSA people, as seen by no reports 2.6.26 and onward.
>
> sounds like a -stable candidate to me

Probably is, I'll dig into it after next week...

thanks,

greg k-h