2.6.29-rc4: Reported regressions from 2.6.28

2.6.29-rc4: Reported regressions from 2.6.28

[Rafael J. Wysocki]

This message contains a list of some regressions from 2.6.28, for which there
are no fixes in the mainline I know of.  If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.28, please let me know
either and I'll add them to the list.  Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2009-02-08       82       45          36
  2009-02-04       66       51          39
  2009-01-20       38       35          27
  2009-01-11       13       13          10


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12671
Subject		: uvc_status_cleanup(): undefined reference to `input_unregister_device'
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-02-08 14:58 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123410529909318&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12670
Subject		: BUG: unable to handle kernel paging request at pin_to_kill+0x21
Submitter	: Alessandro Bono <alessandro.bono@gmail.com>
Date		: 2009-02-08 11:04 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123409113223833&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12669
Subject		: 2.6.28.4 regression: mmap fails if mlockall used
Submitter	: Sami Farin <safari-kernel@safari.iki.fi>
Date		: 2009-02-08 10:52 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123409037423068&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
Subject		: USB flash disk surprise disconnect
Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
Date		: 2009-02-08 10:21 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
Submitter	: Paul Collins <paul@burly.ondioline.org>
Date		: 2009-01-21 7:15 (19 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12666
Subject		: Build failure with latest -git: btrfs on ppc64
Submitter	: Chuck Ebbert <cebbert@redhat.com>
Date		: 2009-02-07 20:50 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123404002114219&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12664
Subject		: viafb triggers BUG at mm/vmalloc.c:294
Submitter	: wixor <wixorpeek@gmail.com>
Date		: 2009-02-07 19:44 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123403591109515&w=4
Handled-By	: Alexey Dobriyan <adobriyan@gmail.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12661
Subject		: commit 64ff3b938ec6782e6585a83d5459b98b0c3f6eb8 breaks rlogin
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-02-05 14:41 (4 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=64ff3b938ec6782e6585a83d5459b98b0c3f6eb8
References	: http://marc.info/?l=linux-kernel&m=123384496107361&w=4
Handled-By	: Herbert Xu <herbert@gondor.apana.org.au>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
Submitter	: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date		: 2009-02-04 21:11 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
Handled-By	: Ingo Molnar <mingo@elte.hu>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12659
Subject		: Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).
Submitter	: 2.6.29-rc3-git3 -- Failure to resume two Sandisk USB flash drives
Date		: 2009-02-03 4:58 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=123363718614763&w=4
Handled-By	: Rafael J. Wysocki <rjw@sisk.pl>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12656
Subject		: iwl3945 broken after hibernation: Wait for START_ALIVE timeout after 2000ms.
Submitter	: Alex Riesen <fork0@users.sf.net>
Date		: 2009-02-08 01:52 (1 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart@free.fr>
Date		: 2009-01-20 16:25 (20 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12649
Subject		: Early crash with 2.6.29-rc3
Submitter	: Andrew McMillan <andrew@morphoss.com>
Date		: 2009-02-04 20:03 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=123377840816097&w=4
Handled-By	: Len Brown <lenb@kernel.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12617
Subject		: unable to compile e100 firmware into kernel
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-31 15:59 (9 days old)
References	: http://marc.info/?l=linux-kernel&m=123341764915181&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
Subject		: boot hangs while bringing up gianfar ethernet
Submitter	: Ira Snyder <iws@ovro.caltech.edu>
Date		: 2009-01-29 19:41 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
Subject		: [Suspend regression][DRM, RADEON]
Submitter	: etienne <etienne.basset@numericable.fr>
Date		: 2009-01-28 22:00 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
Subject		: sync-Regression in 2.6.28.2?
Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>
Date		: 2009-01-27 9:35 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12608
Subject		: 2.6.29-rc powerpc G5 Xorg legacy_mem regression
Submitter	: Hugh Dickins <hugh@veritas.com>
Date		: 2009-01-21 21:12 (19 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d3a54014e2a94bd37b7dee5e76e03f7bc4fab49a
References	: http://marc.info/?l=linux-kernel&m=123257250431870&w=4
Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
Submitter	: Jan Kara <jack@suse.cz>
Date		: 2009-01-30 1:23 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12600
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject		: possible circular locking dependency detected
Submitter	: Michael S. Tsirkin <m.s.tsirkin@gmail.com>
Date		: 2009-01-29 11:35 (11 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12571
Subject		: Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*
Submitter	: Ross Boswell <drb@med.co.nz>
Date		: 2009-01-29 03:46 (11 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12551
Subject		: end_request: I/O error, dev cciss/c0d0, sector 87435720
Submitter	: Ralf Hildebrandt <ralf.hildebrandt@charite.de>
Date		: 2009-01-27 06:51 (13 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
Subject		: 2.6.29-rc2 dies on startup
Submitter	: Ferenc Wagner <wferi@niif.hu>
Date		: 2009-01-19 12:53 (21 days old)
References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12508
Subject		: "powerpc/pci: Reserve legacy regions on PCI" broke my G3
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2009-01-17 22:46 (23 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c1f343028d35ba4e88cd4a3c44e0d8b8a84264ee
References	: http://marc.info/?l=linux-kernel&m=123223247325452&w=4
Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12503
Subject		: [slab corruption] BUG key_jar: Poison overwritten
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-15 18:16 (25 days old)
References	: http://marc.info/?l=linux-kernel&m=123204353425825&w=4
Handled-By	: David Howells <dhowells@redhat.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12502
Subject		: pipe_read oops on sh
Submitter	: Adrian McMenamin <lkmladrian@gmail.com>
Date		: 2009-01-15 9:48 (25 days old)
References	: http://marc.info/?l=linux-kernel&m=123201298005600&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12501
Subject		: build bug in eeepc-laptop.c
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-14 17:25 (26 days old)
References	: http://lkml.org/lkml/2009/1/14/315


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12499
Subject		: Problem with using bluetooth adaper connected to usb port
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2009-01-13 18:34 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123187185426236&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12497
Subject		: new barrier warnings in 2.6.29-rc1
Submitter	: Christoph Hellwig <hch@lst.de>
Date		: 2009-01-12 15:46 (28 days old)
References	: http://marc.info/?l=linux-kernel&m=123177528217154&w=4
Handled-By	: Jens Axboe <jens.axboe@oracle.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
Subject		: Sony backlight regression from 2.6.28 to 29-rc
Submitter	: Norbert Preining <preining@logic.at>
Date		: 2009-01-19 8:14 (21 days old)
References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
Handled-By	: Mattia Dongili <malattia@linux.it>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12490
Subject		: ath5k related kernel panic in 2.6.29-rc1
Submitter	: Sergey S. Kostyliov <rathamahata@gmail.com>
Date		: 2009-01-12 7:38 (28 days old)
References	: http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By	: Bob Copeland <me@bobcopeland.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12444
Subject		: X hangs following switch from radeonfb console - Bisected
Submitter	: Graham Murray <graham@gmurray.org.uk>
Date		: 2009-01-13 14:03 (27 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
Subject		: possible circular locking dependency on i915 dma
Submitter	: Wang Chen <wangchen@cn.fujitsu.com>
Date		: 2009-01-08 14:11 (32 days old)
References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12418
Subject		: Repeated ioctl(4, 0x40046445, ..) loop in glxgears
Submitter	: Pallipadi, Venkatesh <venkatesh.pallipadi@intel.com>
Date		: 2009-01-07 22:43 (33 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a
References	: http://marc.info/?l=linux-kernel&m=123136836213319&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12414
Subject		: iwl4965 cannot use "ap auto" on latest 2.6.28/29?
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-01-05 4:13 (35 days old)
References	: http://marc.info/?l=linux-kernel&m=123112882127823&w=4


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12663
Subject		: Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume
Submitter	: Jiri Slaby <jirislaby@gmail.com>
Date		: 2009-02-07 18:40 (2 days old)
References	: http://lkml.org/lkml/2009/2/7/100
Handled-By	: Brian Gerst <brgerst@gmail.com>
Patch		: http://lkml.org/lkml/2009/2/7/100
		  http://lkml.org/lkml/2009/2/8/59


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12662
Subject		: linux 2.6.29-rc3 kernel failure with mptsas
Submitter	: Morten P.D. Stevens <mstevens@win-professional.com>
Date		: 2009-02-05 22:29 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=123387302729741&w=4
Handled-By	: Andrew Morton <akpm@linux-foundation.org>
Patch		: http://marc.info/?l=linux-kernel&m=123396429501103&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12618
Subject		: hackbench [pthread mode] regression with 2.6.29-rc3
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-02-01 7:30 (8 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=490dea45d00f01847ebebd007685d564aaf2cd98
References	: http://marc.info/?l=linux-kernel&m=123347347726527&w=4
Handled-By	: Peter Zijlstra <peterz@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123383366122146&w=4
		  http://marc.info/?l=linux-kernel&m=123383366222152&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
Submitter	: Larry Finger <Larry.Finger@lwfinger.net>
Date		: 2009-01-23 23:52 (17 days old)
References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
Handled-By	: Mikael Pettersson <mikpe@it.uu.se>
		  Hugh Dickins <hugh@veritas.com>
		  Sergei Shtylyov <sshtylyov@ru.mvista.com>
Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12606
Subject		: fb_mmap: circular locking dependency on hibernation
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-27 18:37 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=123308162731408&w=4
Handled-By	: Andrea Righi <righi.andrea@gmail.com>
Patch		: http://marc.info/?l=linux-kernel&m=123365581406194&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
Subject		: swsusp cannot find resume device (sometimes)
Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
Date		: 2009-01-09 12:24 (31 days old)
References	: http://marc.info/?l=linux-kernel&m=123150395731165&w=4
Handled-By	: Arjan van de Ven <arjan@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123156441218358&w=4
		  http://marc.info/?t=123156453100002&r=1&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
Handled-By	: Roland Dreier <rolandd@cisco.com>
Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12441
Subject		: Xorg can't use dri on radeon X1950 AGP
Submitter	: Daniel Vetter <daniel@ffwll.ch>
Date		: 2009-01-13 05:20 (27 days old)
Handled-By	: Dave Airlie <airlied@linux.ie>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=20032&action=view


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12415
Subject		: WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:689
Submitter	: Christian Borntraeger <borntraeger@de.ibm.com>
Date		: 2009-01-05 10:36 (35 days old)
References	: http://marc.info/?l=linux-wireless&m=123115178019082&w=4
Handled-By	: Reinette Chatre <reinette.chatre@intel.com>
Patch		: http://marc.info/?l=linux-wireless&m=123316414222201&w=2


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.28,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=12398

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael

[Bug #12414] iwl4965 cannot use "ap auto" on latest 2.6.28/29?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12414
Subject		: iwl4965 cannot use "ap auto" on latest 2.6.28/29?
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-01-05 4:13 (35 days old)
References	: http://marc.info/?l=linux-kernel&m=123112882127823&w=4

[Bug #12418] Repeated ioctl(4, 0x40046445, ..) loop in glxgears

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12418
Subject		: Repeated ioctl(4, 0x40046445, ..) loop in glxgears
Submitter	: Pallipadi, Venkatesh <venkatesh.pallipadi@intel.com>
Date		: 2009-01-07 22:43 (33 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a
References	: http://marc.info/?l=linux-kernel&m=123136836213319&w=4

[Bug #12415] WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:689

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12415
Subject		: WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:689
Submitter	: Christian Borntraeger <borntraeger@de.ibm.com>
Date		: 2009-01-05 10:36 (35 days old)
References	: http://marc.info/?l=linux-wireless&m=123115178019082&w=4
Handled-By	: Reinette Chatre <reinette.chatre@intel.com>
Patch		: http://marc.info/?l=linux-wireless&m=123316414222201&w=2

[Bug #12441] Xorg can't use dri on radeon X1950 AGP

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12441
Subject		: Xorg can't use dri on radeon X1950 AGP
Submitter	: Daniel Vetter <daniel@ffwll.ch>
Date		: 2009-01-13 05:20 (27 days old)
Handled-By	: Dave Airlie <airlied@linux.ie>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=20032&action=view

[Bug #12419] possible circular locking dependency on i915 dma

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
Subject		: possible circular locking dependency on i915 dma
Submitter	: Wang Chen <wangchen@cn.fujitsu.com>
Date		: 2009-01-08 14:11 (32 days old)
References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4

[Bug #12490] ath5k related kernel panic in 2.6.29-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12490
Subject		: ath5k related kernel panic in 2.6.29-rc1
Submitter	: Sergey S. Kostyliov <rathamahata@gmail.com>
Date		: 2009-01-12 7:38 (28 days old)
References	: http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By	: Bob Copeland <me@bobcopeland.com>

[Bug #12491] i915 lockdep warning

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
Handled-By	: Roland Dreier <rolandd@cisco.com>
Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2

[Bug #12444] X hangs following switch from radeonfb console - Bisected

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12444
Subject		: X hangs following switch from radeonfb console - Bisected
Submitter	: Graham Murray <graham@gmurray.org.uk>
Date		: 2009-01-13 14:03 (27 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a

[Bug #12496] swsusp cannot find resume device (sometimes)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
Subject		: swsusp cannot find resume device (sometimes)
Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
Date		: 2009-01-09 12:24 (31 days old)
References	: http://marc.info/?l=linux-kernel&m=123150395731165&w=4
Handled-By	: Arjan van de Ven <arjan@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123156441218358&w=4
		  http://marc.info/?t=123156453100002&r=1&w=4

Re: [Bug #12496] swsusp cannot find resume device (sometimes)

[Arjan van de Ven]

On Sun,  8 Feb 2009 20:21:39 +0100 (CET)
"Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me
> know (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
> Subject		: swsusp cannot find resume device (sometimes)
> Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
> Date		: 2009-01-09 12:24 (31 days old)
> References	:
> http://marc.info/?l=linux-kernel&m=123150395731165&w=4
> Handled-By	: Arjan van de Ven <arjan@infradead.org>
> Patch		:
> http://marc.info/?l=linux-kernel&m=123156441218358&w=4
> http://marc.info/?t=123156453100002&r=1&w=4
> 
> 

the patches are in -mm waiting for gregkh to pick them up and push...



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [Bug #12496] swsusp cannot find resume device (sometimes)

[Greg KH]

On Sun, Feb 08, 2009 at 04:38:17PM -0800, Arjan van de Ven wrote:
> On Sun,  8 Feb 2009 20:21:39 +0100 (CET)
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me
> > know (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
> > Subject		: swsusp cannot find resume device (sometimes)
> > Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
> > Date		: 2009-01-09 12:24 (31 days old)
> > References	:
> > http://marc.info/?l=linux-kernel&m=123150395731165&w=4
> > Handled-By	: Arjan van de Ven <arjan@infradead.org>
> > Patch		:
> > http://marc.info/?l=linux-kernel&m=123156441218358&w=4
> > http://marc.info/?t=123156453100002&r=1&w=4
> > 
> > 
> 
> the patches are in -mm waiting for gregkh to pick them up and push...

Ok, I'm totally confused now, care to point out exactly which ones I
should be picking up and pushing?

thanks,

greg "i'm slow, humor me" k-h

Re: [Bug #12496] swsusp cannot find resume device (sometimes)

[Rafael J. Wysocki]

On Monday 09 February 2009, Greg KH wrote:
> On Sun, Feb 08, 2009 at 04:38:17PM -0800, Arjan van de Ven wrote:
> > On Sun,  8 Feb 2009 20:21:39 +0100 (CET)
> > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > 
> > > This message has been generated automatically as a part of a report
> > > of recent regressions.
> > > 
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.28.  Please verify if it still should be listed and let me
> > > know (either way).
> > > 
> > > 
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
> > > Subject		: swsusp cannot find resume device (sometimes)
> > > Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
> > > Date		: 2009-01-09 12:24 (31 days old)
> > > References	:
> > > http://marc.info/?l=linux-kernel&m=123150395731165&w=4
> > > Handled-By	: Arjan van de Ven <arjan@infradead.org>
> > > Patch		:
> > > http://marc.info/?l=linux-kernel&m=123156441218358&w=4
> > > http://marc.info/?t=123156453100002&r=1&w=4
> > > 
> > > 
> > 
> > the patches are in -mm waiting for gregkh to pick them up and push...
> 
> Ok, I'm totally confused now, care to point out exactly which ones I
> should be picking up and pushing?

Patches from -mm called:

drivers-consolidate-driver_probe_done-loops-into-one-place.patch
drivers-consolidate-driver_probe_done-loops-into-one-place-checkpatch-fixes.patch
resume-wait-for-device-probing-to-finish.patch
drivers-consolidate-driver_probe_done-loops-into-one-place-fix.patch

in this order.

Thanks,
Rafael

[Bug #12494] Sony backlight regression from 2.6.28 to 29-rc

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
Subject		: Sony backlight regression from 2.6.28 to 29-rc
Submitter	: Norbert Preining <preining@logic.at>
Date		: 2009-01-19 8:14 (21 days old)
References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
Handled-By	: Mattia Dongili <malattia@linux.it>

[Bug #12497] new barrier warnings in 2.6.29-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12497
Subject		: new barrier warnings in 2.6.29-rc1
Submitter	: Christoph Hellwig <hch@lst.de>
Date		: 2009-01-12 15:46 (28 days old)
References	: http://marc.info/?l=linux-kernel&m=123177528217154&w=4
Handled-By	: Jens Axboe <jens.axboe@oracle.com>

[Bug #12503] [slab corruption] BUG key_jar: Poison overwritten

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12503
Subject		: [slab corruption] BUG key_jar: Poison overwritten
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-15 18:16 (25 days old)
References	: http://marc.info/?l=linux-kernel&m=123204353425825&w=4
Handled-By	: David Howells <dhowells@redhat.com>

Re: [Bug #12503] [slab corruption] BUG key_jar: Poison overwritten

[Ingo Molnar]

* Rafael J. Wysocki <rjw@sisk.pl> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12503
> Subject		: [slab corruption] BUG key_jar: Poison overwritten
> Submitter	: Ingo Molnar <mingo@elte.hu>
> Date		: 2009-01-15 18:16 (25 days old)
> References	: http://marc.info/?l=linux-kernel&m=123204353425825&w=4
> Handled-By	: David Howells <dhowells@redhat.com>

This slab corruption has not triggered again. It's either a very rare thing (in 
which case there's no realistic way to debug it via this angle), or it got fixed 
meanwhile.

Please close it. Thanks,

	Ingo

Re: [Bug #12503] [slab corruption] BUG key_jar: Poison overwritten

[Rafael J. Wysocki]

On Sunday 08 February 2009, Ingo Molnar wrote:
> 
> * Rafael J. Wysocki <rjw@sisk.pl> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12503
> > Subject		: [slab corruption] BUG key_jar: Poison overwritten
> > Submitter	: Ingo Molnar <mingo@elte.hu>
> > Date		: 2009-01-15 18:16 (25 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123204353425825&w=4
> > Handled-By	: David Howells <dhowells@redhat.com>
> 
> This slab corruption has not triggered again. It's either a very rare thing (in 
> which case there's no realistic way to debug it via this angle), or it got fixed 
> meanwhile.
> 
> Please close it. Thanks,

Closed as "unreproducible".

Thanks,
Rafael

[Bug #12499] Problem with using bluetooth adaper connected to usb port

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12499
Subject		: Problem with using bluetooth adaper connected to usb port
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2009-01-13 18:34 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123187185426236&w=4

[Bug #12502] pipe_read oops on sh

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12502
Subject		: pipe_read oops on sh
Submitter	: Adrian McMenamin <lkmladrian@gmail.com>
Date		: 2009-01-15 9:48 (25 days old)
References	: http://marc.info/?l=linux-kernel&m=123201298005600&w=4

[Bug #12501] build bug in eeepc-laptop.c

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12501
Subject		: build bug in eeepc-laptop.c
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-14 17:25 (26 days old)
References	: http://lkml.org/lkml/2009/1/14/315

[Bug #12510] 2.6.29-rc2 dies on startup

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
Subject		: 2.6.29-rc2 dies on startup
Submitter	: Ferenc Wagner <wferi@niif.hu>
Date		: 2009-01-19 12:53 (21 days old)
References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4

[Bug #12571] Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12571
Subject		: Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*
Submitter	: Ross Boswell <drb@med.co.nz>
Date		: 2009-01-29 03:46 (11 days old)

[Bug #12508] "powerpc/pci: Reserve legacy regions on PCI" broke my G3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12508
Subject		: "powerpc/pci: Reserve legacy regions on PCI" broke my G3
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2009-01-17 22:46 (23 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c1f343028d35ba4e88cd4a3c44e0d8b8a84264ee
References	: http://marc.info/?l=linux-kernel&m=123223247325452&w=4
Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Re: [Bug #12508] "powerpc/pci: Reserve legacy regions on PCI" broke my G3

[Mikael Pettersson]

Rafael J. Wysocki wrote:
>This message has been generated automatically as a part of a report
>of recent regressions.
>
>The following bug entry is on the current list of known regressions
>from 2.6.28.  Please verify if it still should be listed and let me know
>(either way).
>
>
>Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=3D12508
>Subject		: "powerpc/pci: Reserve legacy regions on PCI" broke my G3
>Submitter	: Mikael Pettersson <mikpe@it.uu.se>
>Date		: 2009-01-17 22:46 (23 days old)
>First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
>x-2.6.git;a=3Dcommit;h=3Dc1f343028d35ba4e88cd4a3c44e0d8b8a84264ee
>References	: http://marc.info/?l=3Dlinux-kernel&m=3D123223247325452&w=3D4
>Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Fixed in 2.6.29-rc4.
See <http://marc.info/?l=linux-kernel&m=123394660404763&w=2> (you were CC:d).

Re: [Bug #12508] "powerpc/pci: Reserve legacy regions on PCI" broke my G3

[Rafael J. Wysocki]

On Monday 09 February 2009, Mikael Pettersson wrote:
> Rafael J. Wysocki wrote:
> >This message has been generated automatically as a part of a report
> >of recent regressions.
> >
> >The following bug entry is on the current list of known regressions
> >from 2.6.28.  Please verify if it still should be listed and let me know
> >(either way).
> >
> >
> >Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=3D12508
> >Subject		: "powerpc/pci: Reserve legacy regions on PCI" broke my G3
> >Submitter	: Mikael Pettersson <mikpe@it.uu.se>
> >Date		: 2009-01-17 22:46 (23 days old)
> >First-Bad-Commit: http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linu=
> >x-2.6.git;a=3Dcommit;h=3Dc1f343028d35ba4e88cd4a3c44e0d8b8a84264ee
> >References	: http://marc.info/?l=3Dlinux-kernel&m=3D123223247325452&w=3D4
> >Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> 
> Fixed in 2.6.29-rc4.

Thanks, closed.

> See <http://marc.info/?l=linux-kernel&m=123394660404763&w=2> (you were CC:d).

Yes, but I wasn't quite sure if that was a mainline commit or a commit from
the Ben's tree.

Thanks,
Rafael

[Bug #12551] end_request: I/O error, dev cciss/c0d0, sector 87435720

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12551
Subject		: end_request: I/O error, dev cciss/c0d0, sector 87435720
Submitter	: Ralf Hildebrandt <ralf.hildebrandt@charite.de>
Date		: 2009-01-27 06:51 (13 days old)

[Bug #12600] i915 lockdep warning

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12600
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4

Re: [Bug #12600] i915 lockdep warning

[Roland Dreier]

 > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12600
 > Subject		: i915 lockdep warning
 > Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
 > Date		: 2009-01-13 23:17 (27 days old)
 > References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4

Looks like a dup of 12491 to me.

 - R.

Re: [Bug #12600] i915 lockdep warning

[Bob Copeland]

On Sun, 08 Feb 2009 17:12:21 -0800, Roland Dreier wrote
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12600
>  > Subject		: i915 lockdep warning
>  > Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
>  > Date		: 2009-01-13 23:17 (27 days old)
>  > References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
> 
> Looks like a dup of 12491 to me.
> 
>  - R.

I can also confirm that Roland's patch fixes it for me.

-- 
Bob Copeland %% www.bobcopeland.com

[Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
Submitter	: Jan Kara <jack@suse.cz>
Date		: 2009-01-30 1:23 (10 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Jan Kara]

On Sun 08-02-09 20:21:42, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
  Yes, I've verified with latest git and the regression is still there.

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
> Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
> Submitter	: Jan Kara <jack@suse.cz>
> Date		: 2009-01-30 1:23 (10 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
> References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4

									Honza

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Nick Piggin]

On Tue, Feb 10, 2009 at 05:28:30PM +0100, Jan Kara wrote:
> On Sun 08-02-09 20:21:42, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
>   Yes, I've verified with latest git and the regression is still there.

I'm working on this FWIW...

> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
> > Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
> > Submitter	: Jan Kara <jack@suse.cz>
> > Date		: 2009-01-30 1:23 (10 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
> > References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4
> 
> 									Honza

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Linus Torvalds]

On Thu, 12 Feb 2009, Nick Piggin wrote:

> On Tue, Feb 10, 2009 at 05:28:30PM +0100, Jan Kara wrote:
> > On Sun 08-02-09 20:21:42, Rafael J. Wysocki wrote:
> > > This message has been generated automatically as a part of a report
> > > of recent regressions.
> > > 
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.28.  Please verify if it still should be listed and let me know
> > > (either way).
> >   Yes, I've verified with latest git and the regression is still there.
> 
> I'm working on this FWIW...

Shouldn't we just revert it? The code does look to be broken.

It also looks like the interaction with that ever-buggy "nr_to_write" 
thing are totally wrong. I can see that whole

	if (!cycled) {
		..
		index = 0;
		goto retry
	}

doing all the wrong things: if we ever hit the combination of 
"!cycled + nr_to-write==0", we're always screwed. It simply _cannot_ do 
the right thing.

I dunno. That whole piece-of-sh*t function has been incredibly buggy this 
release. The code is an unreadable mess, and I think that "cyclic" stuff 
is part of the reason for it being messy and buggy. Please convince me 
it's worth saving, or let me revert the whole stinking pile of crud?

Please?

			Linus

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Nick Piggin]

On Wed, Feb 11, 2009 at 06:02:19PM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 12 Feb 2009, Nick Piggin wrote:
> 
> > On Tue, Feb 10, 2009 at 05:28:30PM +0100, Jan Kara wrote:
> > > On Sun 08-02-09 20:21:42, Rafael J. Wysocki wrote:
> > > > This message has been generated automatically as a part of a report
> > > > of recent regressions.
> > > > 
> > > > The following bug entry is on the current list of known regressions
> > > > from 2.6.28.  Please verify if it still should be listed and let me know
> > > > (either way).
> > >   Yes, I've verified with latest git and the regression is still there.
> > 
> > I'm working on this FWIW...
> 
> Shouldn't we just revert it? The code does look to be broken.
> 
> It also looks like the interaction with that ever-buggy "nr_to_write" 
> thing are totally wrong. I can see that whole
> 
> 	if (!cycled) {
> 		..
> 		index = 0;
> 		goto retry
> 	}
> 
> doing all the wrong things: if we ever hit the combination of 
> "!cycled + nr_to-write==0", we're always screwed. It simply _cannot_ do 
> the right thing.

Doh, I think you spotted the bug. I was going off on the wrong track.

 
> I dunno. That whole piece-of-sh*t function has been incredibly buggy this 
> release. The code is an unreadable mess, and I think that "cyclic" stuff 
> is part of the reason for it being messy and buggy. Please convince me 
> it's worth saving, or let me revert the whole stinking pile of crud?

Well... the cyclic stuff apparently gets used by some filesystems to do
data integrity stuff, so I think the problem addressed by my broken
commit maybe shouldn't be ignored.

Maybe you're being harsh on this function. It has been two thinko bugs
so far. Before this release cycle it seemed to have the record for the
most data integrity bugs in a single function...

How's this? Jan, can you test with the bdb workload please?

--

A bug was introduced into write_cache_pages cyclic writeout by commit
31a12666d8f0c22235297e1c1575f82061480029. The intention (and comments)
is that we should cycle back and look for more dirty pages at the
beginning of the file if there is no more work to be done.

But the !done condition was dropped from the test. This means that any
time the page writeout loop breaks (eg. due to nr_to_write == 0), we
will set index to 0, then goto again. This will set done_index to index,
then find done is set, so will proceed to the end of the function. When
updating mapping->writeback_index for cyclic writeout, we now use
done_index == 0, so we're always cycling back to 0.

This seemed to be causing random mmap writes (slapadd and iozone) to
start writing more pages from the LRU and writeout would slowdown.

With this patch, iozone random write performance is increased nearly
5x on my system (iozone -B -r 4k -s 64k -s 512m -s 1200m on ext2). 

Signed-off-by: Nick Piggin <npiggin@suse.de>
---
Index: linux-2.6/mm/page-writeback.c
===================================================================
--- linux-2.6.orig/mm/page-writeback.c	2009-02-12 13:30:42.000000000 +1100
+++ linux-2.6/mm/page-writeback.c	2009-02-12 14:16:28.000000000 +1100
@@ -1079,7 +1079,7 @@ continue_unlock:
 		pagevec_release(&pvec);
 		cond_resched();
 	}
-	if (!cycled) {
+	if (!cycled && !done) {
 		/*
 		 * range_cyclic:
 		 * We hit the last page and there is more work to be done: wrap

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Jan Kara]

On Thu 12-02-09 04:34:23, Nick Piggin wrote:
> On Wed, Feb 11, 2009 at 06:02:19PM -0800, Linus Torvalds wrote:
> > 
> > 
> > On Thu, 12 Feb 2009, Nick Piggin wrote:
> > 
> > > On Tue, Feb 10, 2009 at 05:28:30PM +0100, Jan Kara wrote:
> > > > On Sun 08-02-09 20:21:42, Rafael J. Wysocki wrote:
> > > > > This message has been generated automatically as a part of a report
> > > > > of recent regressions.
> > > > > 
> > > > > The following bug entry is on the current list of known regressions
> > > > > from 2.6.28.  Please verify if it still should be listed and let me know
> > > > > (either way).
> > > >   Yes, I've verified with latest git and the regression is still there.
> > > 
> > > I'm working on this FWIW...
> > 
> > Shouldn't we just revert it? The code does look to be broken.
> > 
> > It also looks like the interaction with that ever-buggy "nr_to_write" 
> > thing are totally wrong. I can see that whole
> > 
> > 	if (!cycled) {
> > 		..
> > 		index = 0;
> > 		goto retry
> > 	}
> > 
> > doing all the wrong things: if we ever hit the combination of 
> > "!cycled + nr_to-write==0", we're always screwed. It simply _cannot_ do 
> > the right thing.
> 
> Doh, I think you spotted the bug. I was going off on the wrong track.
> 
>  
> > I dunno. That whole piece-of-sh*t function has been incredibly buggy this 
> > release. The code is an unreadable mess, and I think that "cyclic" stuff 
> > is part of the reason for it being messy and buggy. Please convince me 
> > it's worth saving, or let me revert the whole stinking pile of crud?
> 
> Well... the cyclic stuff apparently gets used by some filesystems to do
> data integrity stuff, so I think the problem addressed by my broken
> commit maybe shouldn't be ignored.
> 
> Maybe you're being harsh on this function. It has been two thinko bugs
> so far. Before this release cycle it seemed to have the record for the
> most data integrity bugs in a single function...
> 
> How's this? Jan, can you test with the bdb workload please?
  Yes, this patch returns write performance of BDB workload back to
original numbers. Thanks for the fix.

								Honza

> --
> 
> A bug was introduced into write_cache_pages cyclic writeout by commit
> 31a12666d8f0c22235297e1c1575f82061480029. The intention (and comments)
> is that we should cycle back and look for more dirty pages at the
> beginning of the file if there is no more work to be done.
> 
> But the !done condition was dropped from the test. This means that any
> time the page writeout loop breaks (eg. due to nr_to_write == 0), we
> will set index to 0, then goto again. This will set done_index to index,
> then find done is set, so will proceed to the end of the function. When
> updating mapping->writeback_index for cyclic writeout, we now use
> done_index == 0, so we're always cycling back to 0.
> 
> This seemed to be causing random mmap writes (slapadd and iozone) to
> start writing more pages from the LRU and writeout would slowdown.
> 
> With this patch, iozone random write performance is increased nearly
> 5x on my system (iozone -B -r 4k -s 64k -s 512m -s 1200m on ext2). 
> 
> Signed-off-by: Nick Piggin <npiggin@suse.de>
> ---
> Index: linux-2.6/mm/page-writeback.c
> ===================================================================
> --- linux-2.6.orig/mm/page-writeback.c	2009-02-12 13:30:42.000000000 +1100
> +++ linux-2.6/mm/page-writeback.c	2009-02-12 14:16:28.000000000 +1100
> @@ -1079,7 +1079,7 @@ continue_unlock:
>  		pagevec_release(&pvec);
>  		cond_resched();
>  	}
> -	if (!cycled) {
> +	if (!cycled && !done) {
>  		/*
>  		 * range_cyclic:
>  		 * We hit the last page and there is more work to be done: wrap
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Theodore Tso]

On Sun, Feb 08, 2009 at 08:21:42PM +0100, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
> Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
> Submitter	: Jan Kara <jack@suse.cz>
> Date		: 2009-01-30 1:23 (10 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
> References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4
> 

Note: this has been fixed in mainline as of 2.6.29-rc5, in commit
3a4c6800.  So this bug can be closed now...

						- Ted

P.S.  One good thing about the commit 31a12666 was that made an
otherwise very hard to hit potential deadlock condition in ext4 very
easy to hit, and thus for us to detect and to fix.  So, thanks.  :-)

Re: [Bug #12604] Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB

[Rafael J. Wysocki]

On Saturday 14 February 2009, Theodore Tso wrote:
> On Sun, Feb 08, 2009 at 08:21:42PM +0100, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12604
> > Subject		: Commit 31a12666d8f0c22235297e1c1575f82061480029 slows down Berkeley DB
> > Submitter	: Jan Kara <jack@suse.cz>
> > Date		: 2009-01-30 1:23 (10 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=31a12666d8f0c22235297e1c1575f82061480029
> > References	: http://marc.info/?l=linux-kernel&m=123327862901800&w=4
> > 
> 
> Note: this has been fixed in mainline as of 2.6.29-rc5, in commit
> 3a4c6800.  So this bug can be closed now...

The bug has been closed already.

Thanks,
Rafael

[Bug #12574] possible circular locking dependency detected

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject		: possible circular locking dependency detected
Submitter	: Michael S. Tsirkin <m.s.tsirkin@gmail.com>
Date		: 2009-01-29 11:35 (11 days old)

Re: [Bug #12574] possible circular locking dependency detected

[Michael S. Tsirkin]

I have verified that this still occurs on 2.6.29-rc4.

On Sun, Feb 8, 2009 at 9:21 PM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12574
> Subject         : possible circular locking dependency detected
> Submitter       : Michael S. Tsirkin <m.s.tsirkin@gmail.com>
> Date            : 2009-01-29 11:35 (11 days old)
>
>
>

Re: [Bug #12574] possible circular locking dependency detected

[Michael S. Tsirkin]

Dave, dri guys,
Could you take a look at this circular dependency please (below)?  I
observe it when suspending laptop with radeon drm loaded and with
lockdep enabled. It seems that the root of the problem is that
various vm ops such as drm_vm_open, drm_mmap) are called with mm
semaphore taken, and take dev->struct_mutex. On the other hand,
drm_rmmap_locked is called with dev->struct_mutex, and calls mtrr_del
which depends on mm semaphore indirectly.

What do you think?

Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject         : possible circular locking dependency detected
Submitter       : Michael S. Tsirkin <m.s.tsirkin@gmail.com>
Date            : 2009-01-29 11:35 (11 days old)

/var/log/message dump below.


 =======================================================
 [ INFO: possible circular locking dependency detected ]
 2.6.29-rc4-mst-debug #95
 -------------------------------------------------------
 sleep.sh/6730 is trying to acquire lock:
  (&per_cpu(cpu_policy_rwsem, cpu)){----}, at: [<c02c0da1>] lock_policy_rwsem_write+0x31/0x70
 
 but task is already holding lock:
  (&cpu_hotplug.lock){--..}, at: [<c012d89a>] cpu_hotplug_begin+0x1a/0x50
 
 which lock already depends on the new lock.
 
 
 the existing dependency chain (in reverse order) is:
 
 -> #6 (&cpu_hotplug.lock){--..}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c0366c5d>] mutex_lock_nested+0x9d/0x2e0
        [<c012d8fc>] get_online_cpus+0x2c/0x40
        [<c010d96f>] mtrr_del_page+0x2f/0x160
        [<c010dada>] mtrr_del+0x3a/0x50
        [<f851a342>] drm_rmmap_locked+0xc2/0x180 [drm]
        [<f8521d31>] drm_master_destroy+0x151/0x160 [drm]
        [<c022a37c>] kref_put+0x2c/0x80
        [<f8521af2>] drm_master_put+0x12/0x20 [drm]
        [<f851dd1b>] drm_release+0x25b/0x4a0 [drm]
        [<c019781d>] __fput+0xbd/0x1d0
        [<c0197c09>] fput+0x19/0x20
        [<c0194a47>] filp_close+0x47/0x70
        [<c0194ada>] sys_close+0x6a/0xc0
        [<c0103215>] sysenter_do_call+0x12/0x35
        [<ffffffff>] 0xffffffff
 
 -> #5 (&dev->struct_mutex){--..}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c0366c5d>] mutex_lock_nested+0x9d/0x2e0
        [<f8522add>] drm_vm_open+0x2d/0x50 [drm]
        [<c012a397>] dup_mm+0x227/0x310
        [<c012b22f>] copy_process+0xd7f/0x1020
        [<c012b5e8>] do_fork+0x78/0x320
        [<c01017ef>] sys_clone+0x2f/0x40
        [<c0103215>] sysenter_do_call+0x12/0x35
        [<ffffffff>] 0xffffffff
 
 -> #4 (&mm->mmap_sem/1){--..}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c01441e8>] down_write_nested+0x48/0x70
        [<c012a238>] dup_mm+0xc8/0x310
        [<c012b22f>] copy_process+0xd7f/0x1020
        [<c012b5e8>] do_fork+0x78/0x320
        [<c01017ef>] sys_clone+0x2f/0x40
        [<c0103292>] syscall_call+0x7/0xb
        [<ffffffff>] 0xffffffff
 
 -> #3 (&mm->mmap_sem){----}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c0183d73>] might_fault+0x73/0x90
        [<c022f633>] copy_to_user+0x33/0x60
        [<c01a3975>] filldir64+0xb5/0xe0
        [<c01e0c2f>] sysfs_readdir+0x11f/0x1f0
        [<c01a3b0d>] vfs_readdir+0x8d/0xb0
        [<c01a3b99>] sys_getdents64+0x69/0xc0
        [<c0103292>] syscall_call+0x7/0xb
        [<ffffffff>] 0xffffffff
 
 -> #2 (sysfs_mutex){--..}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c0366c5d>] mutex_lock_nested+0x9d/0x2e0
        [<c01e0f0c>] sysfs_addrm_start+0x2c/0xb0
        [<c01e14a0>] create_dir+0x40/0x90
        [<c01e1556>] sysfs_create_subdir+0x16/0x20
        [<c01e2770>] internal_create_group+0x50/0x1a0
        [<c01e28ec>] sysfs_create_group+0xc/0x10
        [<f81674fc>] cpufreq_stat_notifier_policy+0x9c/0x230 [cpufreq_stats]
        [<c036b007>] notifier_call_chain+0x37/0x80
        [<c0144d24>] __blocking_notifier_call_chain+0x44/0x60
        [<c0144d5a>] blocking_notifier_call_chain+0x1a/0x20
        [<c02c0226>] __cpufreq_set_policy+0xd6/0x230
        [<c02c14a8>] cpufreq_add_dev+0x4e8/0x6b0
        [<c029d5a5>] sysdev_driver_register+0x75/0x130
        [<c02bff55>] cpufreq_register_driver+0xb5/0x1c0
        [<f808b0bd>] uinput_destroy_device+0x4d/0x60 [uinput]
        [<c010111a>] do_one_initcall+0x2a/0x160
        [<c015bdf5>] sys_init_module+0x85/0x1b0
        [<c0103215>] sysenter_do_call+0x12/0x35
        [<ffffffff>] 0xffffffff
 
 -> #1 ((cpufreq_policy_notifier_list).rwsem){----}:
        [<c0152221>] validate_chain+0xb51/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c0367441>] down_read+0x41/0x60
        [<c0144d0a>] __blocking_notifier_call_chain+0x2a/0x60
        [<c0144d5a>] blocking_notifier_call_chain+0x1a/0x20
        [<c02c1165>] cpufreq_add_dev+0x1a5/0x6b0
        [<c029d5a5>] sysdev_driver_register+0x75/0x130
        [<c02bff55>] cpufreq_register_driver+0xb5/0x1c0
        [<f808b0bd>] uinput_destroy_device+0x4d/0x60 [uinput]
        [<c010111a>] do_one_initcall+0x2a/0x160
        [<c015bdf5>] sys_init_module+0x85/0x1b0
        [<c0103215>] sysenter_do_call+0x12/0x35
        [<ffffffff>] 0xffffffff
 
 -> #0 (&per_cpu(cpu_policy_rwsem, cpu)){----}:
        [<c0151cbb>] validate_chain+0x5eb/0x1150
        [<c0152a66>] __lock_acquire+0x246/0xa50
        [<c01532d0>] lock_acquire+0x60/0x80
        [<c03674a1>] down_write+0x41/0x60
        [<c02c0da1>] lock_policy_rwsem_write+0x31/0x70
        [<c03655a5>] cpufreq_cpu_callback+0x45/0x80
        [<c036b007>] notifier_call_chain+0x37/0x80
        [<c0144b49>] __raw_notifier_call_chain+0x19/0x20
        [<c03574c9>] _cpu_down+0x79/0x280
        [<c012da5c>] disable_nonboot_cpus+0x7c/0x100
        [<c015cac5>] suspend_devices_and_enter+0xd5/0x170
        [<c015cd40>] enter_state+0x1b0/0x1c0
        [<c015cddf>] state_store+0x8f/0xd0
        [<c0228cf4>] kobj_attr_store+0x24/0x30
        [<c01e02d2>] sysfs_write_file+0xa2/0x100
        [<c0196e89>] vfs_write+0x99/0x130
        [<c01973cd>] sys_write+0x3d/0x70
        [<c0103215>] sysenter_do_call+0x12/0x35
        [<ffffffff>] 0xffffffff
 
 other info that might help us debug this:
 
 4 locks held by sleep.sh/6730:
  #0:  (&buffer->mutex){--..}, at: [<c01e025b>] sysfs_write_file+0x2b/0x100
  #1:  (pm_mutex){--..}, at: [<c015cbdb>] enter_state+0x4b/0x1c0
  #2:  (cpu_add_remove_lock){--..}, at: [<c012d83f>] cpu_maps_update_begin+0xf/0x20
  #3:  (&cpu_hotplug.lock){--..}, at: [<c012d89a>] cpu_hotplug_begin+0x1a/0x50
 
 stack backtrace:
 Pid: 6730, comm: sleep.sh Not tainted 2.6.29-rc4-mst-debug #95
 Call Trace:
  [<c015166c>] print_circular_bug_tail+0x7c/0xe0
  [<c0151cbb>] validate_chain+0x5eb/0x1150
  [<c0152a66>] __lock_acquire+0x246/0xa50
  [<c013cf1e>] ? __cancel_work_timer+0x2e/0x190
  [<c01532d0>] lock_acquire+0x60/0x80
  [<c02c0da1>] ? lock_policy_rwsem_write+0x31/0x70
  [<c03674a1>] down_write+0x41/0x60
  [<c02c0da1>] ? lock_policy_rwsem_write+0x31/0x70
  [<c02c0da1>] lock_policy_rwsem_write+0x31/0x70
  [<c03655a5>] cpufreq_cpu_callback+0x45/0x80
  [<c036b007>] notifier_call_chain+0x37/0x80
  [<c0144b49>] __raw_notifier_call_chain+0x19/0x20
  [<c03574c9>] _cpu_down+0x79/0x280
  [<c012d83f>] ? cpu_maps_update_begin+0xf/0x20
  [<c012da5c>] disable_nonboot_cpus+0x7c/0x100
  [<c02531cb>] ? acpi_disable_all_gpes+0x25/0x2a
  [<c015cac5>] suspend_devices_and_enter+0xd5/0x170
  [<c015cd40>] enter_state+0x1b0/0x1c0
  [<c015cddf>] state_store+0x8f/0xd0
  [<c015cd50>] ? state_store+0x0/0xd0
  [<c0228cf4>] kobj_attr_store+0x24/0x30
  [<c01e02d2>] sysfs_write_file+0xa2/0x100
  [<c0196e89>] vfs_write+0x99/0x130
  [<c0103247>] ? sysenter_exit+0xf/0x18
  [<c01e0230>] ? sysfs_write_file+0x0/0x100
  [<c01973cd>] sys_write+0x3d/0x70
  [<c0103215>] sysenter_do_call+0x12/0x35

-- 
MST

Re: [Bug #12574] possible circular locking dependency detected

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Wed, 2009-02-11 at 00:37 +0200, Michael S. Tsirkin wrote:
> Dave, dri guys,
> Could you take a look at this circular dependency please (below)?  I
> observe it when suspending laptop with radeon drm loaded and with
> lockdep enabled. It seems that the root of the problem is that
> various vm ops such as drm_vm_open, drm_mmap) are called with mm
> semaphore taken, and take dev->struct_mutex. On the other hand,
> drm_rmmap_locked is called with dev->struct_mutex, and calls mtrr_del
> which depends on mm semaphore indirectly.
> 
> What do you think?

Yes, there are real lock inversions now due to the GTT mmap code.  It's
going to be a pain to fix (I tried getting the mmap_sem -> struct_mutex
path to go away, but the fact that mmap_sem is held over the fault
handler pretty much kills that).  It's high on the list, though.

-- 
Eric Anholt
eric@anholt.net                         eric.anholt@intel.com



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Bug #12574] possible circular locking dependency detected

[Thomas Hellström]

Eric Anholt wrote:
> On Wed, 2009-02-11 at 00:37 +0200, Michael S. Tsirkin wrote:
>   
>> Dave, dri guys,
>> Could you take a look at this circular dependency please (below)?  I
>> observe it when suspending laptop with radeon drm loaded and with
>> lockdep enabled. It seems that the root of the problem is that
>> various vm ops such as drm_vm_open, drm_mmap) are called with mm
>> semaphore taken, and take dev->struct_mutex. On the other hand,
>> drm_rmmap_locked is called with dev->struct_mutex, and calls mtrr_del
>> which depends on mm semaphore indirectly.
>>
>> What do you think?
>>     
>
> Yes, there are real lock inversions now due to the GTT mmap code.  It's
> going to be a pain to fix (I tried getting the mmap_sem -> struct_mutex
> path to go away, but the fact that mmap_sem is held over the fault
> handler pretty much kills that).  It's high on the list, though.
>   
Shouldn't it be possible to relax this given that in the fault() 
handler, the mmap_sem() is taken in read mode only. This means that it 
should be deadlock-safe (but still a little ugly) to take the mmap_sem() 
in read mode when the struct mutex is held.

Another quick way to fix potential deadlocks, would be to take the 
struct mutex in the following way in fault():

if (!mutex_trylock(&dev->struct_mutex)) {
      needs_resched();
      return VM_FAULT_NOPAGE;
}

/Thomas

>   
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
> software. With Adobe AIR, Ajax developers can use existing skills and code to
> build responsive, highly engaging applications that combine the power of local
> resources and data with the reach of the web. Download the Adobe AIR SDK and
> Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
> ------------------------------------------------------------------------
>
> --
> _______________________________________________
> Dri-devel mailing list
> Dri-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dri-devel
>   

[Bug #12606] fb_mmap: circular locking dependency on hibernation

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12606
Subject		: fb_mmap: circular locking dependency on hibernation
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-27 18:37 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=123308162731408&w=4
Handled-By	: Andrea Righi <righi.andrea@gmail.com>
Patch		: http://marc.info/?l=linux-kernel&m=123365581406194&w=2

Re: [Bug #12606] fb_mmap: circular locking dependency on hibernation

[Andrea Righi]

On 2009-02-08 20:21, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12606
> Subject		: fb_mmap: circular locking dependency on hibernation
> Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
> Date		: 2009-01-27 18:37 (13 days old)
> References	: http://marc.info/?l=linux-kernel&m=123308162731408&w=4
> Handled-By	: Andrea Righi <righi.andrea@gmail.com>
> Patch		: http://marc.info/?l=linux-kernel&m=123365581406194&w=2

This is fixed by:

commit 1f5e31d7e55ac7fbd4ec5e5b20c8868b0e4564c9
Author: Andrea Righi <righi.andrea@gmail.com>
Date:   Wed Feb 4 15:12:03 2009 -0800

    fbmem: don't call copy_from/to_user() with mutex held

    Avoid calling copy_from/to_user() with fb_info->lock mutex held in fbmem
    ioctl().

    fb_mmap() is called under mm->mmap_sem (A) held, that also acquires
    fb_info->lock (B); fb_ioctl() takes fb_info->lock (B) and does
    copy_from/to_user() that might acquire mm->mmap_sem (A), causing a
    deadlock.

    NOTE: it doesn't push down the fb_info->lock in each own driver's
    fb_ioctl(), so there are still potential deadlocks elsewhere.

    Signed-off-by: Andrea Righi <righi.andrea@gmail.com>
    Cc: Dave Jones <davej@redhat.com>
    Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
    Cc: Johannes Weiner <hannes@saeurebad.de>
    Cc: Krzysztof Helt <krzysztof.h1@wp.pl>
    Cc: Harvey Harrison <harvey.harrison@gmail.com>
    Cc: Stefan Richter <stefanr@s5r6.in-berlin.de>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Re: [Bug #12606] fb_mmap: circular locking dependency on hibernation

[Rafael J. Wysocki]

On Sunday 08 February 2009, Andrea Righi wrote:
> On 2009-02-08 20:21, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12606
> > Subject		: fb_mmap: circular locking dependency on hibernation
> > Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
> > Date		: 2009-01-27 18:37 (13 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123308162731408&w=4
> > Handled-By	: Andrea Righi <righi.andrea@gmail.com>
> > Patch		: http://marc.info/?l=linux-kernel&m=123365581406194&w=2
> 
> This is fixed by:
> 
> commit 1f5e31d7e55ac7fbd4ec5e5b20c8868b0e4564c9
> Author: Andrea Righi <righi.andrea@gmail.com>
> Date:   Wed Feb 4 15:12:03 2009 -0800
> 
>     fbmem: don't call copy_from/to_user() with mutex held

Thanks, closed.

Rafael

[Bug #12608] 2.6.29-rc powerpc G5 Xorg legacy_mem regression

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12608
Subject		: 2.6.29-rc powerpc G5 Xorg legacy_mem regression
Submitter	: Hugh Dickins <hugh@veritas.com>
Date		: 2009-01-21 21:12 (19 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d3a54014e2a94bd37b7dee5e76e03f7bc4fab49a
References	: http://marc.info/?l=linux-kernel&m=123257250431870&w=4
Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Re: [Bug #12608] 2.6.29-rc powerpc G5 Xorg legacy_mem regression

[Hugh Dickins]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12608
> Subject		: 2.6.29-rc powerpc G5 Xorg legacy_mem regression
> Submitter	: Hugh Dickins <hugh@veritas.com>
> Date		: 2009-01-21 21:12 (19 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d3a54014e2a94bd37b7dee5e76e03f7bc4fab49a
> References	: http://marc.info/?l=linux-kernel&m=123257250431870&w=4
> Handled-By	: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Please still list it for now, but I expect Ben's fix to be in -rc5.

Hugh

[Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
Submitter	: Larry Finger <Larry.Finger@lwfinger.net>
Date		: 2009-01-23 23:52 (17 days old)
References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
Handled-By	: Mikael Pettersson <mikpe@it.uu.se>
		  Hugh Dickins <hugh@veritas.com>
		  Sergei Shtylyov <sshtylyov@ru.mvista.com>
Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4

Re: [Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Hugh Dickins]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
> Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
> Submitter	: Larry Finger <Larry.Finger@lwfinger.net>
> Date		: 2009-01-23 23:52 (17 days old)
> References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
> 		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
> Handled-By	: Mikael Pettersson <mikpe@it.uu.se>
> 		  Hugh Dickins <hugh@veritas.com>
> 		  Sergei Shtylyov <sshtylyov@ru.mvista.com>
> Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4

Please still list it for now, but I expect Sergei's and Alan's fixes
to be in -rc5.

Hugh

[Bug #12610] sync-Regression in 2.6.28.2?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
Subject		: sync-Regression in 2.6.28.2?
Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>
Date		: 2009-01-27 9:35 (13 days old)
References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4

[Bug #12613] [Suspend regression][DRM, RADEON]

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
Subject		: [Suspend regression][DRM, RADEON]
Submitter	: etienne <etienne.basset@numericable.fr>
Date		: 2009-01-28 22:00 (12 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[etienne]

Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
> Subject		: [Suspend regression][DRM, RADEON]
> Submitter	: etienne <etienne.basset@numericable.fr>
> Date		: 2009-01-28 22:00 (12 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
> References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
> 		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
>
>
>   
hello,
yes it's still present in -rc4
But I noticed that when I switch off KDE4.2 desktop effects, suspend to 
ram is 100% reliable with 2.6.29-rc4
With 2.6.28, STR is 100% reliable with or without desktop effects

hope this helps,
Etienne

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[Rafael J. Wysocki]

On Sunday 08 February 2009, etienne wrote:
> Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
> > Subject		: [Suspend regression][DRM, RADEON]
> > Submitter	: etienne <etienne.basset@numericable.fr>
> > Date		: 2009-01-28 22:00 (12 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
> > References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
> > 		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
> >
> >
> >   
> hello,
> yes it's still present in -rc4
> But I noticed that when I switch off KDE4.2 desktop effects, suspend to 
> ram is 100% reliable with 2.6.29-rc4
> With 2.6.28, STR is 100% reliable with or without desktop effects

Thanks for the update.

Rafael

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[Dave Airlie]

On Mon, Feb 9, 2009 at 8:07 AM, etienne <etienne.basset@numericable.fr> wrote:
> Rafael J. Wysocki wrote:
>>
>> This message has been generated automatically as a part of a report
>> of recent regressions.
>>
>> The following bug entry is on the current list of known regressions
>> from 2.6.28.  Please verify if it still should be listed and let me know
>> (either way).
>>
>>
>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12613
>> Subject         : [Suspend regression][DRM, RADEON]
>> Submitter       : etienne <etienne.basset@numericable.fr>
>> Date            : 2009-01-28 22:00 (12 days old)
>> First-Bad-Commit:
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
>> References      : http://marc.info/?l=linux-kernel&m=123318030419558&w=4
>>                  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
>>
>>
>>
>
> hello,
> yes it's still present in -rc4
> But I noticed that when I switch off KDE4.2 desktop effects, suspend to ram
> is 100% reliable with 2.6.29-rc4
> With 2.6.28, STR is 100% reliable with or without desktop effects
>

Hi Etienne,

Can you try commenting out the calls to the radeon_suspend and
radeon_resume hooks in radeon_drv.c?

Dave.

> hope this helps,
> Etienne
>
>
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[etienne]

Dave Airlie wrote:
> On Mon, Feb 9, 2009 at 8:07 AM, etienne <etienne.basset@numericable.fr> wrote:
>   
>> Rafael J. Wysocki wrote:
>>     
>>> This message has been generated automatically as a part of a report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.28.  Please verify if it still should be listed and let me know
>>> (either way).
>>>
>>>
>>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12613
>>> Subject         : [Suspend regression][DRM, RADEON]
>>> Submitter       : etienne <etienne.basset@numericable.fr>
>>> Date            : 2009-01-28 22:00 (12 days old)
>>> First-Bad-Commit:
>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
>>> References      : http://marc.info/?l=linux-kernel&m=123318030419558&w=4
>>>                  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
>>>
>>>
>>>
>>>       
>> hello,
>> yes it's still present in -rc4
>> But I noticed that when I switch off KDE4.2 desktop effects, suspend to ram
>> is 100% reliable with 2.6.29-rc4
>> With 2.6.28, STR is 100% reliable with or without desktop effects
>>
>>     
>
> Hi Etienne,
>
> Can you try commenting out the calls to the radeon_suspend and
> radeon_resume hooks in radeon_drv.c?
>
> Dave.
>   
Hi Dave,

I just tested that and I didn't change anything (cannot STR/resume with 
desktop effect, STR/resume OK without desktop effects)

thanks,
Etienne

the patch i used, to verify i understood you ;)

diff --git a/drivers/gpu/drm/radeon/radeon_drv.c 
b/drivers/gpu/drm/radeon/radeon
index fef2078..805b367 100644
--- a/drivers/gpu/drm/radeon/radeon_drv.c
+++ b/drivers/gpu/drm/radeon/radeon_drv.c
@@ -90,8 +90,8 @@ static struct drm_driver driver = {
        .postclose = radeon_driver_postclose,
        .lastclose = radeon_driver_lastclose,
        .unload = radeon_driver_unload,
-       .suspend = radeon_suspend,
-       .resume = radeon_resume,
+/*     .suspend = radeon_suspend,
+       .resume = radeon_resume, */
        .get_vblank_counter = radeon_get_vblank_counter,
        .enable_vblank = radeon_enable_vblank,
        .disable_vblank = radeon_disable_vblank,

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[etienne]

Dave Airlie wrote:
> On Mon, Feb 9, 2009 at 8:07 AM, etienne <etienne.basset@numericable.fr> wrote:
>   
>> Rafael J. Wysocki wrote:
>>     
>>> This message has been generated automatically as a part of a report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.28.  Please verify if it still should be listed and let me know
>>> (either way).
>>>
>>>
>>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12613
>>> Subject         : [Suspend regression][DRM, RADEON]
>>> Submitter       : etienne <etienne.basset@numericable.fr>
>>> Date            : 2009-01-28 22:00 (12 days old)
>>> First-Bad-Commit:
>>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
>>> References      : http://marc.info/?l=linux-kernel&m=123318030419558&w=4
>>>                  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
>>>
>>>
>>>
>>>       
>> hello,
>> yes it's still present in -rc4
>> But I noticed that when I switch off KDE4.2 desktop effects, suspend to ram
>> is 100% reliable with 2.6.29-rc4
>> With 2.6.28, STR is 100% reliable with or without desktop effects
>>
>>     
>
> Hi Etienne,
>
> Can you try commenting out the calls to the radeon_suspend and
> radeon_resume hooks in radeon_drv.c?
>
> Dave.
>   
Hi Dave,

I created the following "shot in the dark" patch that solves my problem!

I looked at the change between 2.6.28 and .29rc, and only the  
radeon_cp.c:radeon_cp_init_ring_buffer changes stroke my eyes (cause 
it's called by radeon_cp_resume indirectedly)

I don't understand what i did and how this works, but it works for me
regards
Etienne

Signed-off-by: etienne <etienne.basset@numericable.fr>

diff --git a/drivers/gpu/drm/radeon/radeon_cp.c 
b/drivers/gpu/drm/radeon/radeon_cp.c
index 63212d7..fc6e134 
100644                                                      
--- 
a/drivers/gpu/drm/radeon/radeon_cp.c                                            

+++ 
b/drivers/gpu/drm/radeon/radeon_cp.c                                            

@@ -557,7 +557,8 @@ static int radeon_do_engine_reset(struct drm_device 
* dev)     
 }                                                                                  

                                                                                    

 static void radeon_cp_init_ring_buffer(struct drm_device * 
dev,                   
-                                      drm_radeon_private_t * 
dev_priv)            
+                                      drm_radeon_private_t * 
dev_priv,            
+                                      struct drm_radeon_master_private 
*master)   
 {                                                                                                                                                                                  

        u32 ring_start, 
cur_read_ptr;                                                                                                                                               

        u32 
tmp;                                                                                                                                                                    

@@ -668,13 +669,13 @@ static void radeon_cp_init_ring_buffer(struct 
drm_device * 
dev,                                                                                               

                RADEON_WRITE(RADEON_BUS_CNTL, 
tmp);                                                                                                                                 

        } /* PCIE cards appears to not need this 
*/                                                                                                                                 

                                                                                                                                                                                    

-       dev_priv->scratch[0] = 
0;                                                                                                                                                   

+       master->sarea_priv->last_frame = dev_priv->scratch[0] = 0;
        RADEON_WRITE(RADEON_LAST_FRAME_REG, 0);

-       dev_priv->scratch[1] = 0;
+       master->sarea_priv->last_dispatch = dev_priv->scratch[1] = 0;
        RADEON_WRITE(RADEON_LAST_DISPATCH_REG, 0);

-       dev_priv->scratch[2] = 0;
+       master->sarea_priv->last_clear = dev_priv->scratch[2] = 0;
        RADEON_WRITE(RADEON_LAST_CLEAR_REG, 0);

        radeon_do_wait_for_idle(dev_priv);
@@ -1215,7 +1216,7 @@ static int radeon_do_init_cp(struct drm_device 
*dev, drm_radeon_init_t *init,
        }

        radeon_cp_load_microcode(dev_priv);
-       radeon_cp_init_ring_buffer(dev, dev_priv);
+       radeon_cp_init_ring_buffer(dev, dev_priv, master_priv);

        dev_priv->last_buf = 0;

@@ -1281,9 +1282,11 @@ static int radeon_do_cleanup_cp(struct drm_device 
* dev)
  *
  * Charl P. Botha <http://cpbotha.net>
  */
-static int radeon_do_resume_cp(struct drm_device * dev)
+static int radeon_do_resume_cp(struct drm_device * dev,
+                              struct drm_file * file_priv)
 {
        drm_radeon_private_t *dev_priv = dev->dev_private;
+       struct drm_radeon_master_private * master_priv = 
file_priv->master->driver_priv;

        if (!dev_priv) {
                DRM_ERROR("Called with no initialization\n");
@@ -1304,7 +1307,7 @@ static int radeon_do_resume_cp(struct drm_device * 
dev)
        }

        radeon_cp_load_microcode(dev_priv);
-       radeon_cp_init_ring_buffer(dev, dev_priv);
+       radeon_cp_init_ring_buffer(dev, dev_priv, master_priv);

        radeon_do_engine_reset(dev);
        radeon_irq_set_state(dev, RADEON_SW_INT_ENABLE, 1);
@@ -1480,7 +1483,7 @@ int radeon_cp_idle(struct drm_device *dev, void 
*data, struct drm_file *file_pri
 int radeon_cp_resume(struct drm_device *dev, void *data, struct 
drm_file *file_priv)
 {

-       return radeon_do_resume_cp(dev);
+       return radeon_do_resume_cp(dev, file_priv);
 }

 int radeon_engine_reset(struct drm_device *dev, void *data, struct 
drm_file *file_priv)

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[Soeren Sonnenburg]

On Sun, 2009-02-08 at 23:07 +0100, etienne wrote:
> Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
> > Subject		: [Suspend regression][DRM, RADEON]
> > Submitter	: etienne <etienne.basset@numericable.fr>
> > Date		: 2009-01-28 22:00 (12 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
> > References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
> > 		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
> >
> >
> >   
> hello,
> yes it's still present in -rc4
> But I noticed that when I switch off KDE4.2 desktop effects, suspend to 
> ram is 100% reliable with 2.6.29-rc4
> With 2.6.28, STR is 100% reliable with or without desktop effects

For me it is reliable as soon as compositing/dri is disabled on 2.6.29.
On 2.6.28 it worked with compositing/dri *sometimes* - so 2.6.29 is the
first reliable kernel for me (when not using 3d).

Soeren

[Bug #12615] boot hangs while bringing up gianfar ethernet

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
Subject		: boot hangs while bringing up gianfar ethernet
Submitter	: Ira Snyder <iws@ovro.caltech.edu>
Date		: 2009-01-29 19:41 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4

[Bug #12617] unable to compile e100 firmware into kernel

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12617
Subject		: unable to compile e100 firmware into kernel
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-31 15:59 (9 days old)
References	: http://marc.info/?l=linux-kernel&m=123341764915181&w=4

[Bug #12656] iwl3945 broken after hibernation: Wait for START_ALIVE timeout after 2000ms.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12656
Subject		: iwl3945 broken after hibernation: Wait for START_ALIVE timeout after 2000ms.
Submitter	: Alex Riesen <fork0@users.sf.net>
Date		: 2009-02-08 01:52 (1 days old)

[Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart@free.fr>
Date		: 2009-01-20 16:25 (20 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

[Bug #12649] Early crash with 2.6.29-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12649
Subject		: Early crash with 2.6.29-rc3
Submitter	: Andrew McMillan <andrew@morphoss.com>
Date		: 2009-02-04 20:03 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=123377840816097&w=4
Handled-By	: Len Brown <lenb@kernel.org>

[Bug #12618] hackbench [pthread mode] regression with 2.6.29-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12618
Subject		: hackbench [pthread mode] regression with 2.6.29-rc3
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-02-01 7:30 (8 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=490dea45d00f01847ebebd007685d564aaf2cd98
References	: http://marc.info/?l=linux-kernel&m=123347347726527&w=4
Handled-By	: Peter Zijlstra <peterz@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123383366122146&w=4
		  http://marc.info/?l=linux-kernel&m=123383366222152&w=4

[Bug #12661] commit 64ff3b938ec6782e6585a83d5459b98b0c3f6eb8 breaks rlogin

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12661
Subject		: commit 64ff3b938ec6782e6585a83d5459b98b0c3f6eb8 breaks rlogin
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-02-05 14:41 (4 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=64ff3b938ec6782e6585a83d5459b98b0c3f6eb8
References	: http://marc.info/?l=linux-kernel&m=123384496107361&w=4
Handled-By	: Herbert Xu <herbert@gondor.apana.org.au>

Re: [Bug #12661] commit 64ff3b938ec6782e6585a83d5459b98b0c3f6eb8 breaks rlogin

[Herbert Xu]

On Sun, Feb 08, 2009 at 08:21:46PM +0100, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).

The patch in question has already been reverted.  So while we
still need to track down the underlying problem with rlogin, it's
no longer a kernel regression.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Bug #12661] commit 64ff3b938ec6782e6585a83d5459b98b0c3f6eb8 breaks rlogin

[Rafael J. Wysocki]

On Monday 09 February 2009, Herbert Xu wrote:
> On Sun, Feb 08, 2009 at 08:21:46PM +0100, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> The patch in question has already been reverted.  So while we
> still need to track down the underlying problem with rlogin, it's
> no longer a kernel regression.

I've closed the bug, sorry for the slow response.

Thanks,
Rafael

[Bug #12660] Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
Submitter	: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date		: 2009-02-04 21:11 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
Handled-By	: Ingo Molnar <mingo@elte.hu>

[Bug #12659] Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12659
Subject		: Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).
Submitter	: 2.6.29-rc3-git3 -- Failure to resume two Sandisk USB flash drives
Date		: 2009-02-03 4:58 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=123363718614763&w=4
Handled-By	: Rafael J. Wysocki <rjw@sisk.pl>

[Bug #12662] linux 2.6.29-rc3 kernel failure with mptsas

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12662
Subject		: linux 2.6.29-rc3 kernel failure with mptsas
Submitter	: Morten P.D. Stevens <mstevens@win-professional.com>
Date		: 2009-02-05 22:29 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=123387302729741&w=4
Handled-By	: Andrew Morton <akpm@linux-foundation.org>
Patch		: http://marc.info/?l=linux-kernel&m=123396429501103&w=4

[Bug #12668] USB flash disk surprise disconnect

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
Subject		: USB flash disk surprise disconnect
Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
Date		: 2009-02-08 10:21 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4

[Bug #12663] Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12663
Subject		: Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume
Submitter	: Jiri Slaby <jirislaby@gmail.com>
Date		: 2009-02-07 18:40 (2 days old)
References	: http://lkml.org/lkml/2009/2/7/100
Handled-By	: Brian Gerst <brgerst@gmail.com>
Patch		: http://lkml.org/lkml/2009/2/7/100
		  http://lkml.org/lkml/2009/2/8/59

[Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
Submitter	: Paul Collins <paul@burly.ondioline.org>
Date		: 2009-01-21 7:15 (19 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Paul Collins]

"Rafael J. Wysocki" <rjw@sisk.pl> writes:

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
> Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
> Submitter	: Paul Collins <paul@burly.ondioline.org>
> Date		: 2009-01-21 7:15 (19 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
> References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4

I've just done ten suspend/resume cycles with 2.6.29-rc4 and this
message was not logged, so it's either gone away or else become much
more difficult to reproduce.

-- 
Paul Collins
Wellington, New Zealand

Dag vijandelijk luchtschip de huismeester is dood

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Ingo Molnar]

* Paul Collins <paul@burly.ondioline.org> wrote:

> "Rafael J. Wysocki" <rjw@sisk.pl> writes:
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
> > Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
> > Submitter	: Paul Collins <paul@burly.ondioline.org>
> > Date		: 2009-01-21 7:15 (19 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
> > References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4
> 
> I've just done ten suspend/resume cycles with 2.6.29-rc4 and this
> message was not logged, so it's either gone away or else become much
> more difficult to reproduce.

Various suspend handlers were enabling irqs spuriously - fixed by recent patches in 
-rc4. The warning above is consistent with the timekeeping code being surprised with 
a premature irqs-enable.

	Ingo

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Rafael J. Wysocki]

On Monday 09 February 2009, Paul Collins wrote:
> "Rafael J. Wysocki" <rjw@sisk.pl> writes:
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
> > Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
> > Submitter	: Paul Collins <paul@burly.ondioline.org>
> > Date		: 2009-01-21 7:15 (19 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
> > References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4
> 
> I've just done ten suspend/resume cycles with 2.6.29-rc4 and this
> message was not logged, so it's either gone away or else become much
> more difficult to reproduce.

Thanks, I've closed the bug.

Sorry for the slow response.

Rafael

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Paul Collins]

"Rafael J. Wysocki" <rjw@sisk.pl> writes:

> On Monday 09 February 2009, Paul Collins wrote:
>> "Rafael J. Wysocki" <rjw@sisk.pl> writes:
>> 
>> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
>> > Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
>> > Submitter	: Paul Collins <paul@burly.ondioline.org>
>> > Date		: 2009-01-21 7:15 (19 days old)
>> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496
>> > References	: http://marc.info/?l=linux-kernel&m=123252215315106&w=4
>> 
>> I've just done ten suspend/resume cycles with 2.6.29-rc4 and this
>> message was not logged, so it's either gone away or else become much
>> more difficult to reproduce.
>
> Thanks, I've closed the bug.

It turns out I didn't test this properly.  The warning is only triggered
when I close and open the lid, not when I run 'snooze' to suspend and
hit Return to resume, as I did for my so-called testing of 2.6.29-rc4.

Whatever is triggering the warning is still present in 2.6.29-rc5.

-- 
Paul Collins
Wellington, New Zealand

Dag vijandelijk luchtschip de huismeester is dood

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Benjamin Herrenschmidt]

On Mon, 2009-02-16 at 20:17 +1300, Paul Collins wrote:
> It turns out I didn't test this properly.  The warning is only
> triggered
> when I close and open the lid, not when I run 'snooze' to suspend and
> hit Return to resume, as I did for my so-called testing of 2.6.29-rc4.
> 
> Whatever is triggering the warning is still present in 2.6.29-rc5.

Right, we probably need to stop sending input events from the PMU driver
when it's "suspended".

Ping me if I don't produce a patch tomorrow (ie, I forgot :-)

Cheers,
Ben.

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Paul Collins]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> On Mon, 2009-02-16 at 20:17 +1300, Paul Collins wrote:
>> It turns out I didn't test this properly.  The warning is only
>> triggered
>> when I close and open the lid, not when I run 'snooze' to suspend and
>> hit Return to resume, as I did for my so-called testing of 2.6.29-rc4.
>> 
>> Whatever is triggering the warning is still present in 2.6.29-rc5.
>
> Right, we probably need to stop sending input events from the PMU driver
> when it's "suspended".
>
> Ping me if I don't produce a patch tomorrow (ie, I forgot :-)

Just for laughs I slapped together the following, which seems to do the
job, although not especially tidily.


diff --git a/drivers/macintosh/via-pmu-event.c b/drivers/macintosh/via-pmu-event.c
index 25cd565..33c4b45 100644
--- a/drivers/macintosh/via-pmu-event.c
+++ b/drivers/macintosh/via-pmu-event.c
@@ -23,6 +23,7 @@
 #include <linux/input.h>
 #include <linux/adb.h>
 #include <linux/pmu.h>
+#include <linux/bug.h>
 #include "via-pmu-event.h"
 
 static struct input_dev *pmu_input_dev;
@@ -56,24 +57,62 @@ static int __init via_pmu_event_init(void)
 	return err;
 }
 
+static bool lid_event_pending;
+static int lid_event_pending_down;
+static bool power_button_event_pending;
+static int power_button_event_pending_down;
+
 void via_pmu_event(int key, int down)
 {
 
 	if (unlikely(!pmu_input_dev))
 		return;
 
-	switch (key) {
-	case PMU_EVT_POWER:
-		input_report_key(pmu_input_dev, KEY_POWER, down);
-		break;
-	case PMU_EVT_LID:
-		input_report_switch(pmu_input_dev, SW_LID, down);
-		break;
-	default:
-		/* no such key handled */
-		return;
-	}
+	if (!pmu_sys_suspended) {
+		switch (key) {
+		case PMU_EVT_POWER:
+			input_report_key(pmu_input_dev, KEY_POWER, down);
+			break;
+		case PMU_EVT_LID:
+			input_report_switch(pmu_input_dev, SW_LID, down);
+			break;
+		default:
+			/* no such key handled */
+			return;
+		}
+
+		input_sync(pmu_input_dev);
+	} else {
+		switch (key) {
+		case PMU_EVT_POWER:
+			power_button_event_pending = true;
+			power_button_event_pending_down = down;
+			break;
+		case PMU_EVT_LID:
+			lid_event_pending = true;
+			lid_event_pending_down = down;
+			break;
+		default:
+			/* no such key handled */
+			return;
+		}
+	}		
+}
 
+void via_pmu_event_resume(void)
+{
+	WARN_ON(pmu_sys_suspended);
+
+	if (power_button_event_pending) {
+		input_report_key(pmu_input_dev, KEY_POWER,
+				 power_button_event_pending_down);
+		power_button_event_pending = false;
+	}
+	if (lid_event_pending) {
+		input_report_switch(pmu_input_dev, SW_LID,
+				    lid_event_pending_down);
+		lid_event_pending = false;
+	}
 	input_sync(pmu_input_dev);
 }
 
diff --git a/drivers/macintosh/via-pmu-event.h b/drivers/macintosh/via-pmu-event.h
index 72c54de..2754adc 100644
--- a/drivers/macintosh/via-pmu-event.h
+++ b/drivers/macintosh/via-pmu-event.h
@@ -4,5 +4,6 @@
 #define PMU_EVT_POWER	0
 #define PMU_EVT_LID	1
 extern void via_pmu_event(int key, int down);
+extern void via_pmu_event_resume(void);
 
 #endif /* __VIA_PMU_EVENT_H */
diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c
index b40fb9b..01b8689 100644
--- a/drivers/macintosh/via-pmu.c
+++ b/drivers/macintosh/via-pmu.c
@@ -2479,6 +2479,8 @@ static int pmu_sys_resume(struct sys_device *sysdev)
 	pmu_resume();
 	pmu_sys_suspended = 0;
 
+	via_pmu_event_resume();
+
 	return 0;
 }
 


-- 
Paul Collins
Wellington, New Zealand

Dag vijandelijk luchtschip de huismeester is dood

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Paul Collins]

Paul Collins <paul@burly.ondioline.org> writes:

> Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
>
>> On Mon, 2009-02-16 at 20:17 +1300, Paul Collins wrote:
>>> It turns out I didn't test this properly.  The warning is only
>>> triggered
>>> when I close and open the lid, not when I run 'snooze' to suspend and
>>> hit Return to resume, as I did for my so-called testing of 2.6.29-rc4.
>>> 
>>> Whatever is triggering the warning is still present in 2.6.29-rc5.
>>
>> Right, we probably need to stop sending input events from the PMU driver
>> when it's "suspended".
>>
>> Ping me if I don't produce a patch tomorrow (ie, I forgot :-)
>
> Just for laughs I slapped together the following, which seems to do the
> job, although not especially tidily.

And it doesn't even do the job.  Judging by this new trace, submitting
input events from the via-pmu resume function is still too early.

NIP [c0053b4c] getnstimeofday+0x24/0x188
LR [c0053ccc] do_gettimeofday+0x1c/0x58
Call Trace:
[eed09cb0] [c0053ccc] do_gettimeofday+0x1c/0x58
[eed09ce0] [c03492cc] evdev_event+0x28/0x158
[eed09d10] [c0341748] input_pass_event+0xac/0xb0
[eed09d30] [c03444a8] input_event+0x80/0x98
[eed09d50] [c02f7668] via_pmu_event_resume+0x60/0xb8 <-- the function I added
[eed09d60] [c02f725c] pmu_sys_resume+0x5c/0x74
[eed09de0] [c02d6420] __sysdev_resume+0x64/0x84
[eed09e00] [c02d6498] sysdev_resume+0x58/0xa4
[eed09e20] [c02dd39c] device_power_up+0x18/0x38
[eed09e40] [c00609d4] suspend_devices_and_enter+0x11c/0x180
[eed09e60] [c0060be8] enter_state+0x11c/0x160
[eed09e80] [c02f4d6c] pmu_ioctl+0x15c/0x24c
[eed09e90] [c00bad34] vfs_ioctl+0x8c/0x90
[eed09ea0] [c00bade8] do_vfs_ioctl+0x8c/0x70c
[eed09f10] [c00bb504] sys_ioctl+0x9c/0xa4
[eed09f40] [c0012eb8] ret_from_syscall+0x0/0x38

-- 
Paul Collins
Wellington, New Zealand

Dag vijandelijk luchtschip de huismeester is dood

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Benjamin Herrenschmidt]

On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > Just for laughs I slapped together the following, which seems to do
> the
> > job, although not especially tidily.
> 
> And it doesn't even do the job.  Judging by this new trace, submitting
> input events from the via-pmu resume function is still too early.
> 
What's up Thomas ? We can't call gettimeofday() from a sysdev
suspend/resume ? That's a little bit too harsh no ?

Cheers,
Ben.

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Rafael J. Wysocki]

On Thursday 19 February 2009, Benjamin Herrenschmidt wrote:
> On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > > Just for laughs I slapped together the following, which seems to do
> > the
> > > job, although not especially tidily.
> > 
> > And it doesn't even do the job.  Judging by this new trace, submitting
> > input events from the via-pmu resume function is still too early.
> > 
> What's up Thomas ? We can't call gettimeofday() from a sysdev
> suspend/resume ? That's a little bit too harsh no ?

Perhaps the ordering is wrong (ie. via-pmu resume happens bevore timekeeping
resume)?

Rafael

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Benjamin Herrenschmidt]

On Thu, 2009-02-19 at 14:00 +0100, Rafael J. Wysocki wrote:
> On Thursday 19 February 2009, Benjamin Herrenschmidt wrote:
> > On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > > > Just for laughs I slapped together the following, which seems to do
> > > the
> > > > job, although not especially tidily.
> > > 
> > > And it doesn't even do the job.  Judging by this new trace, submitting
> > > input events from the via-pmu resume function is still too early.
> > > 
> > What's up Thomas ? We can't call gettimeofday() from a sysdev
> > suspend/resume ? That's a little bit too harsh no ?
> 
> Perhaps the ordering is wrong (ie. via-pmu resume happens bevore timekeeping
> resume)?

In this case, maybe gtod should just return the frozen time (ie, last
time at the time of suspend) rather than WARN ?

Ben.

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Rafael J. Wysocki]

On Thursday 19 February 2009, Benjamin Herrenschmidt wrote:
> On Thu, 2009-02-19 at 14:00 +0100, Rafael J. Wysocki wrote:
> > On Thursday 19 February 2009, Benjamin Herrenschmidt wrote:
> > > On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > > > > Just for laughs I slapped together the following, which seems to do
> > > > the
> > > > > job, although not especially tidily.
> > > > 
> > > > And it doesn't even do the job.  Judging by this new trace, submitting
> > > > input events from the via-pmu resume function is still too early.
> > > > 
> > > What's up Thomas ? We can't call gettimeofday() from a sysdev
> > > suspend/resume ? That's a little bit too harsh no ?
> > 
> > Perhaps the ordering is wrong (ie. via-pmu resume happens bevore timekeeping
> > resume)?
> 
> In this case, maybe gtod should just return the frozen time (ie, last
> time at the time of suspend) rather than WARN ?

This might work, but there seem to be more problems like this (cpufreq vs
timekeeping for example).

I think we need a more general approach.

Thanks,
Rafael

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Thomas Gleixner]

On Thu, 19 Feb 2009, Benjamin Herrenschmidt wrote:

> On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > > Just for laughs I slapped together the following, which seems to do
> > the
> > > job, although not especially tidily.
> > 
> > And it doesn't even do the job.  Judging by this new trace, submitting
> > input events from the via-pmu resume function is still too early.
> > 
> What's up Thomas ? We can't call gettimeofday() from a sysdev
> suspend/resume ? That's a little bit too harsh no ?

Well, harsh or not is not the question here. 

Fact is that you call gettimeofday() _before_ the timekeeping code has
resumed.

That's a simple ordering problem. timekeeping is in the sysdev class
as well and it's not the only sysdev which has explicit ordering
requirements.

Thanks,

	tglx

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Rafael J. Wysocki]

On Thursday 19 February 2009, Thomas Gleixner wrote:
> On Thu, 19 Feb 2009, Benjamin Herrenschmidt wrote:
> 
> > On Thu, 2009-02-19 at 21:27 +1300, Paul Collins wrote:
> > > > Just for laughs I slapped together the following, which seems to do
> > > the
> > > > job, although not especially tidily.
> > > 
> > > And it doesn't even do the job.  Judging by this new trace, submitting
> > > input events from the via-pmu resume function is still too early.
> > > 
> > What's up Thomas ? We can't call gettimeofday() from a sysdev
> > suspend/resume ? That's a little bit too harsh no ?
> 
> Well, harsh or not is not the question here. 
> 
> Fact is that you call gettimeofday() _before_ the timekeeping code has
> resumed.
> 
> That's a simple ordering problem. timekeeping is in the sysdev class
> as well and it's not the only sysdev which has explicit ordering
> requirements.

Do we need suspend-resume priorities for sysdevs?  Such that sysdevs
with a higher priority will always be suspended earlier and resumed later
than sysdevs with lower priority (or the other way around)?

Rafael

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Benjamin Herrenschmidt]

On Thu, 2009-02-19 at 21:17 +0100, Thomas Gleixner wrote:
> 
> Well, harsh or not is not the question here. 
> 
> Fact is that you call gettimeofday() _before_ the timekeeping code has
> resumed.
> 
> That's a simple ordering problem. timekeeping is in the sysdev class
> as well and it's not the only sysdev which has explicit ordering
> requirements.

And how do I control that ordering ?

I find that a bit fishy ... What about making gettimeofday() in the
timekeeping code work, just return a frozen snapshot of the value on
suspend instead ?

Ben.

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Thomas Gleixner]

On Fri, 20 Feb 2009, Benjamin Herrenschmidt wrote:
> On Thu, 2009-02-19 at 21:17 +0100, Thomas Gleixner wrote:
> > 
> > Well, harsh or not is not the question here. 
> > 
> > Fact is that you call gettimeofday() _before_ the timekeeping code has
> > resumed.
> > 
> > That's a simple ordering problem. timekeeping is in the sysdev class
> > as well and it's not the only sysdev which has explicit ordering
> > requirements.
> 
> And how do I control that ordering ?
> 
> I find that a bit fishy ... What about making gettimeofday() in the
> timekeeping code work, just return a frozen snapshot of the value on
> suspend instead ?

We had problems in the past where we just returned frozen time and the
calling code got surprised when the time jumped 5 hours ahead just a
few microseconds later.

What I find more fishy is the fact that the lid switch needs to be a
sysdev. It's a simple input event, which causes the user space code to
trigger the suspend sequence when the lid is shut.

Thanks,

	tglx

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Benjamin Herrenschmidt]

On Sun, 2009-02-22 at 20:31 +0100, Thomas Gleixner wrote:
> We had problems in the past where we just returned frozen time and the
> calling code got surprised when the time jumped 5 hours ahead just a
> few microseconds later.
> 
> What I find more fishy is the fact that the lid switch needs to be a
> sysdev. It's a simple input event, which causes the user space code to
> trigger the suspend sequence when the lid is shut.
> 
Well, the PMU is a sysdev for lots of good reasons.. then it -also-
happens to provide us with the lid switch event.

I'm a bit surprised by the explanation about the code that is surprised
to see the time jump. IE. Code -will- see the time jump anyway. IE. I
don't see how making gtod blow instead of returning a frozen time will
make any difference whatsoever for code who get the time some time
before suspend and then some time after resume and see a 5 hours gap...

Cheers,
Ben.

Re: [Bug #12667] Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)

[Thomas Gleixner]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12667
> Subject		: Badness at kernel/time/timekeeping.c:98 in pmud (timekeeping_suspended)
> Submitter	: Paul Collins <paul@burly.ondioline.org>
> Date		: 2009-01-21 7:15 (19 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1c5745aa380efb6417b5681104b007c8612fb496

First bad commit ? The commit adds a WARN_ON to getnstimeofday() when
it is called while timekeeping is suspended. So instead of pointing to
that commit can we please figure out WTF this happens?

Looking at the call stack this is pretty obvious:

[ee4fdc80] [c0053b10] do_gettimeofday+0x1c/0x58
[ee4fdcb0] [c0348868] evdev_event+0x28/0x158
[ee4fdce0] [c0340ce4] input_pass_event+0xac/0xb0
[ee4fdd00] [c0343a44] input_event+0x80/0x98
[ee4fdd20] [c02f6d24] via_pmu_event+0x88/0x8c
[ee4fdd30] [c02f4f60] via_pmu_interrupt+0x6e0/0xb2c
[ee4fdd90] [c02f5664] pmu_wait_complete+0x50/0x84
[ee4fddb0] [c02f6764] powerbook_sleep+0x9f0/0xb24
[ee4fde40] [c00607f8] suspend_devices_and_enter+0x10c/0x180
[ee4fde60] [c0060a1c] enter_state+0x11c/0x160
[ee4fde80] [c02f4404] pmu_ioctl+0x15c/0x24c
[ee4fde90] [c00bab04] vfs_ioctl+0x8c/0x90
[ee4fdea0] [c00babb8] do_vfs_ioctl+0x8c/0x70c
[ee4fdf10] [c00bb2d4] sys_ioctl+0x9c/0xa4
[ee4fdf40] [c0012eb8] ret_from_syscall+0x0/0x38

powerbook_sleep() calls into the event interface which calls
gettimeofday() _AFTER_ timekeeping was suspended already so the
WARN_ON triggers.

Thanks,

	tglx

[Bug #12666] Build failure with latest -git: btrfs on ppc64

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12666
Subject		: Build failure with latest -git: btrfs on ppc64
Submitter	: Chuck Ebbert <cebbert@redhat.com>
Date		: 2009-02-07 20:50 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123404002114219&w=4

Re: [Bug #12666] Build failure with latest -git: btrfs on ppc64

[Kyle McMartin]

On Sun, Feb 08, 2009 at 08:21:49PM +0100, Rafael J. Wysocki wrote:
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12666
> Subject		: Build failure with latest -git: btrfs on ppc64
> Submitter	: Chuck Ebbert <cebbert@redhat.com>
> Date		: 2009-02-07 20:50 (2 days old)
> References	: http://marc.info/?l=linux-kernel&m=123404002114219&w=4
> 

Just fyi, patch here:
Subject: x86: spinlocks: define dummy __raw_spin_is_contended                   
Date: Sun, 8 Feb 2009 13:03:41 -0500                                            
Message-ID: <20090208180341.GC11398@bombadil.infradead.org>    

cheers, Kyle

Re: [Bug #12666] Build failure with latest -git: btrfs on ppc64

[Rafael J. Wysocki]

On Sunday 08 February 2009, Kyle McMartin wrote:
> On Sun, Feb 08, 2009 at 08:21:49PM +0100, Rafael J. Wysocki wrote:
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12666
> > Subject		: Build failure with latest -git: btrfs on ppc64
> > Submitter	: Chuck Ebbert <cebbert@redhat.com>
> > Date		: 2009-02-07 20:50 (2 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123404002114219&w=4
> > 
> 
> Just fyi, patch here:
> Subject: x86: spinlocks: define dummy __raw_spin_is_contended                   
> Date: Sun, 8 Feb 2009 13:03:41 -0500                                            
> Message-ID: <20090208180341.GC11398@bombadil.infradead.org>    

Thanks, updated the bug entry with this patch.

Rafael

Re: [Bug #12666] Build failure with latest -git: btrfs on ppc64

[Chuck Ebbert]

On Sun, 8 Feb 2009 23:08:13 +0100
"Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Sunday 08 February 2009, Kyle McMartin wrote:
> > On Sun, Feb 08, 2009 at 08:21:49PM +0100, Rafael J. Wysocki wrote:
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12666
> > > Subject		: Build failure with latest -git: btrfs on ppc64
> > > Submitter	: Chuck Ebbert <cebbert@redhat.com>
> > > Date		: 2009-02-07 20:50 (2 days old)
> > > References	: http://marc.info/?l=linux-kernel&m=123404002114219&w=4
> > > 
> > 
> > Just fyi, patch here:
> > Subject: x86: spinlocks: define dummy __raw_spin_is_contended                   
> > Date: Sun, 8 Feb 2009 13:03:41 -0500                                            
> > Message-ID: <20090208180341.GC11398@bombadil.infradead.org>    
> 

Patch was merged, commit a5ef7ca0e2636bad0ccd07b996d775348ae2b65e

[Bug #12664] viafb triggers BUG at mm/vmalloc.c:294

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12664
Subject		: viafb triggers BUG at mm/vmalloc.c:294
Submitter	: wixor <wixorpeek@gmail.com>
Date		: 2009-02-07 19:44 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123403591109515&w=4
Handled-By	: Alexey Dobriyan <adobriyan@gmail.com>

Re: [Bug #12664] viafb triggers BUG at mm/vmalloc.c:294

[wixor]

There was no support for Chrome9 framebuffer before 2.6.28, so that is
a progress. However now that we have it, it doesn't work (at least for
me).... The BUG appearing seemed to be an mm-issue also reported by
others (see http://marc.info/?l=linux-kernel&m=123415925903748&w=2). I
don't know if it was there before 2.6.28. It doesn't look like any
solution has been merged to mainline.

It is certainly an issue, but I'm not sure if it's also a regression
from 2.6.28.

--
wixor

Re: [Bug #12664] viafb triggers BUG at mm/vmalloc.c:294

[Rafael J. Wysocki]

On Monday 09 February 2009, wixor wrote:
> There was no support for Chrome9 framebuffer before 2.6.28, so that is
> a progress. However now that we have it, it doesn't work (at least for
> me).... The BUG appearing seemed to be an mm-issue also reported by
> others (see http://marc.info/?l=linux-kernel&m=123415925903748&w=2). I
> don't know if it was there before 2.6.28. It doesn't look like any
> solution has been merged to mainline.
> 
> It is certainly an issue, but I'm not sure if it's also a regression
> from 2.6.28.

I've dropped it from the list of recent regressions, sorry for the slow
response.

Thanks,
Rafael

[Bug #12671] uvc_status_cleanup(): undefined reference to `input_unregister_device'

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12671
Subject		: uvc_status_cleanup(): undefined reference to `input_unregister_device'
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-02-08 14:58 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123410529909318&w=4

[Bug #12669] 2.6.28.4 regression: mmap fails if mlockall used

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12669
Subject		: 2.6.28.4 regression: mmap fails if mlockall used
Submitter	: Sami Farin <safari-kernel@safari.iki.fi>
Date		: 2009-02-08 10:52 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123409037423068&w=4

Re: [Bug #12669] 2.6.28.4 regression: mmap fails if mlockall used

[Hugh Dickins]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12669
> Subject		: 2.6.28.4 regression: mmap fails if mlockall used
> Submitter	: Sami Farin <safari-kernel@safari.iki.fi>
> Date		: 2009-02-08 10:52 (1 days old)
> References	: http://marc.info/?l=linux-kernel&m=123409037423068&w=4

Please still list it for now, but I expect
d5b562330ec766292a3ac54ae5e0673610bd5b3d
mm: fix error case in mlock downgrade reversion
(which went in just after -rc4) to have fixed it:
let's wait for confirmation from Sami before removing.

Hugh

[Bug #12670] BUG: unable to handle kernel paging request at pin_to_kill+0x21

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12670
Subject		: BUG: unable to handle kernel paging request at pin_to_kill+0x21
Submitter	: Alessandro Bono <alessandro.bono@gmail.com>
Date		: 2009-02-08 11:04 (1 days old)
References	: http://marc.info/?l=linux-kernel&m=123409113223833&w=4

Re: 2.6.29-rc4: Reported regressions from 2.6.28

[Linus Torvalds]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12669
> Subject		: 2.6.28.4 regression: mmap fails if mlockall used
> Submitter	: Sami Farin <safari-kernel@safari.iki.fi>
> Date		: 2009-02-08 10:52 (1 days old)
> References	: http://marc.info/?l=linux-kernel&m=123409037423068&w=4

Oh, Hugh just sent a patch, and it's now commit 
d5b562330ec766292a3ac54ae5e0673610bd5b3d. Sadly this just missed -rc4 by 
something like 20 minutes..

		Linus

Re: 2.6.29-rc4: Reported regressions from 2.6.28

[Rafael J. Wysocki]

On Sunday 08 February 2009, Linus Torvalds wrote:
> 
> On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12669
> > Subject		: 2.6.28.4 regression: mmap fails if mlockall used
> > Submitter	: Sami Farin <safari-kernel@safari.iki.fi>
> > Date		: 2009-02-08 10:52 (1 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123409037423068&w=4
> 
> Oh, Hugh just sent a patch, and it's now commit 
> d5b562330ec766292a3ac54ae5e0673610bd5b3d. Sadly this just missed -rc4 by 
> something like 20 minutes..

Yes, I saw the Hugh's message right after sending the report.

Bug closed.

Thanks,
Rafael

Re: [linux-pm] 2.6.29-rc4: Reported regressions from 2.6.28

[Alan Stern]

On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
> Subject		: USB flash disk surprise disconnect
> Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
> Date		: 2009-02-08 10:21 (1 days old)
> References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4

Is there any reason to think this is a regression rather than a 
one-time fluke?

If it really is a regression, additional debugging info would help.  
The log excerpt in the email report doesn't show anything that happened
prior to the disconnect.

Alan Stern

Re: [linux-pm] 2.6.29-rc4: Reported regressions from 2.6.28

[Rafael J. Wysocki]

On Sunday 08 February 2009, Alan Stern wrote:
> On Sun, 8 Feb 2009, Rafael J. Wysocki wrote:
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
> > Subject		: USB flash disk surprise disconnect
> > Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
> > Date		: 2009-02-08 10:21 (1 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4
> 
> Is there any reason to think this is a regression rather than a 
> one-time fluke?

Well, Vegard is the right person to ask this question.

I saw the message, so I assumed it was sent as a regression report.

Thanks,
Rafael

Re: 2.6.29-rc4: Reported regressions from 2.6.28

[Stefan Richter]

Rafael J. Wysocki wrote:
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12600
> Subject		: i915 lockdep warning
> Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
> Date		: 2009-01-13 23:17 (27 days old)
> References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4

Now closed as duplicate of

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
> Subject		: i915 lockdep warning
> Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
> Date		: 2009-01-13 23:17 (27 days old)
> References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
> Handled-By	: Roland Dreier <rolandd@cisco.com>
> Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2
-- 
Stefan Richter
-=====-==--= --=- -=--=
http://arcgraph.de/sr/

2.6.29-rc5: Reported regressions from 2.6.28

[Rafael J. Wysocki]

This message contains a list of some regressions from 2.6.28, for which there
are no fixes in the mainline I know of.  If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.28, please let me know
either and I'll add them to the list.  Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2009-02-14       85       33          27
  2009-02-08       82       45          36
  2009-02-04       66       51          39
  2009-01-20       38       35          27
  2009-01-11       13       13          10


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12706
Subject		: Oopses and ACPI problems (Linus 2.6.29-rc4)
Submitter	: Darren Salt <linux@youmustbejoking.demon.co.uk>
Date		: 2009-02-09 18:26 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=123420431709877&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12705
Subject		: X200: Brightness broken since 2.6.29-rc4-58-g4c098bc
Submitter	: Nico Schottelius <nico-linux-20090213@schottelius.org>
Date		: 2009-02-13 9:33 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123451768406825&w=4
Handled-By	: Len Brown <lenb@kernel.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12681
Subject		: s2ram: fails to wake up on Acer Extensa 4220 (SMP disabled)
Submitter	: Orivej Desh <smpuj@bk.ru>
Date		: 2009-02-09 13:01 (6 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1cfe62c8010ac56e1bd3827e30386a87cc2f3594


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12680
Subject		: Not having a VIA PadLock hardware incurs a long delay in probing on modules insertion attempt.
Submitter	: Valentin QUEQUET <v.quequet-techniques@orange.fr>
Date		: 2009-02-09 09:12 (6 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12671
Subject		: uvc_status_cleanup(): undefined reference to `input_unregister_device'
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-02-08 14:58 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123410529909318&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12670
Subject		: BUG: unable to handle kernel paging request at pin_to_kill+0x21
Submitter	: Alessandro Bono <alessandro.bono@gmail.com>
Date		: 2009-02-08 11:04 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123409113223833&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
Subject		: USB flash disk surprise disconnect
Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
Date		: 2009-02-08 10:21 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
Submitter	: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date		: 2009-02-04 21:11 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
Handled-By	: Ingo Molnar <mingo@elte.hu>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart@free.fr>
Date		: 2009-01-20 16:25 (26 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12617
Subject		: unable to compile e100 firmware into kernel
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-31 15:59 (15 days old)
References	: http://marc.info/?l=linux-kernel&m=123341764915181&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
Subject		: boot hangs while bringing up gianfar ethernet
Submitter	: Ira Snyder <iws@ovro.caltech.edu>
Date		: 2009-01-29 19:41 (17 days old)
References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
Subject		: [Suspend regression][DRM, RADEON]
Submitter	: etienne <etienne.basset@numericable.fr>
Date		: 2009-01-28 22:00 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
		  http://lkml.org/lkml/2009/2/8/203


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
Subject		: sync-Regression in 2.6.28.2?
Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>
Date		: 2009-01-27 9:35 (19 days old)
References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject		: possible circular locking dependency detected
Submitter	: Michael S. Tsirkin <m.s.tsirkin@gmail.com>
Date		: 2009-01-29 11:35 (17 days old)
References	: http://lkml.org/lkml/2009/2/9/205


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12571
Subject		: Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*
Submitter	: Ross Boswell <drb@med.co.nz>
Date		: 2009-01-29 03:46 (17 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12551
Subject		: end_request: I/O error, dev cciss/c0d0, sector 87435720
Submitter	: Ralf Hildebrandt <ralf.hildebrandt@charite.de>
Date		: 2009-01-27 06:51 (19 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
Subject		: 2.6.29-rc2 dies on startup
Submitter	: Ferenc Wagner <wferi@niif.hu>
Date		: 2009-01-19 12:53 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12502
Subject		: pipe_read oops on sh
Submitter	: Adrian McMenamin <lkmladrian@gmail.com>
Date		: 2009-01-15 9:48 (31 days old)
References	: http://marc.info/?l=linux-kernel&m=123201298005600&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12501
Subject		: build bug in eeepc-laptop.c
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-14 17:25 (32 days old)
References	: http://lkml.org/lkml/2009/1/14/315


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12499
Subject		: Problem with using bluetooth adaper connected to usb port
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2009-01-13 18:34 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123187185426236&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12497
Subject		: new barrier warnings in 2.6.29-rc1
Submitter	: Christoph Hellwig <hch@lst.de>
Date		: 2009-01-12 15:46 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123177528217154&w=4
Handled-By	: Jens Axboe <jens.axboe@oracle.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
Subject		: Sony backlight regression from 2.6.28 to 29-rc
Submitter	: Norbert Preining <preining@logic.at>
Date		: 2009-01-19 8:14 (27 days old)
References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
Handled-By	: Mattia Dongili <malattia@linux.it>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12490
Subject		: ath5k related kernel panic in 2.6.29-rc1
Submitter	: Sergey S. Kostyliov <rathamahata@gmail.com>
Date		: 2009-01-12 7:38 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By	: Bob Copeland <me@bobcopeland.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12444
Subject		: X hangs following switch from radeonfb console - Bisected
Submitter	: Graham Murray <graham@gmurray.org.uk>
Date		: 2009-01-13 14:03 (33 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
Subject		: possible circular locking dependency on i915 dma
Submitter	: Wang Chen <wangchen@cn.fujitsu.com>
Date		: 2009-01-08 14:11 (38 days old)
References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12418
Subject		: Repeated ioctl(4, 0x40046445, ..) loop in glxgears
Submitter	: Pallipadi, Venkatesh <venkatesh.pallipadi@intel.com>
Date		: 2009-01-07 22:43 (39 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a
References	: http://marc.info/?l=linux-kernel&m=123136836213319&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12414
Subject		: iwl4965 cannot use "ap auto" on latest 2.6.28/29?
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-01-05 4:13 (41 days old)
References	: http://marc.info/?l=linux-kernel&m=123112882127823&w=4


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12663
Subject		: Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume
Submitter	: Jiri Slaby <jirislaby@gmail.com>
Date		: 2009-02-07 18:40 (8 days old)
References	: http://lkml.org/lkml/2009/2/7/100
Handled-By	: Brian Gerst <brgerst@gmail.com>
Patch		: http://lkml.org/lkml/2009/2/7/100
		  http://lkml.org/lkml/2009/2/8/59


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12659
Subject		: Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).
Submitter	: Miles Lane <miles.lane@gmail.com>
Date		: 2009-02-03 4:58 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=123363718614763&w=4
Handled-By	: Rafael J. Wysocki <rjw@sisk.pl>
Patch		: http://marc.info/?l=linux-kernel&m=123456499020575&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12618
Subject		: hackbench [pthread mode] regression with 2.6.29-rc3
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-02-01 7:30 (14 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=490dea45d00f01847ebebd007685d564aaf2cd98
References	: http://marc.info/?l=linux-kernel&m=123347347726527&w=4
Handled-By	: Peter Zijlstra <peterz@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123383366122146&w=4
		  http://marc.info/?l=linux-kernel&m=123383366222152&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
Submitter	: Larry Finger <Larry.Finger@lwfinger.net>
Date		: 2009-01-23 23:52 (23 days old)
References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
Handled-By	: Mikael Pettersson <mikpe@it.uu.se>
		  Hugh Dickins <hugh@veritas.com>
		  Sergei Shtylyov <sshtylyov@ru.mvista.com>
Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
Subject		: swsusp cannot find resume device (sometimes)
Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
Date		: 2009-01-09 12:24 (37 days old)
References	: http://marc.info/?l=linux-kernel&m=123150395731165&w=4
Handled-By	: Arjan van de Ven <arjan@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123156441218358&w=4
		  http://marc.info/?t=123156453100002&r=1&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
Handled-By	: Roland Dreier <rolandd@cisco.com>
Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.28,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=12398

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael

[Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart@free.fr>
Date		: 2009-01-20 16:25 (26 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello,

> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
> Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
> Submitter	: Damien Wyart <damien.wyart@free.fr>
> Date		: 2009-01-20 16:25 (26 days old)
> References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

The problem is still there in 2.6.29-rc5, nothing has changed. I am
surprised noone has looked at it as it is trivial to reproduce for me.

Load average is between 0.30 and 0.60 when the machine is idle, the two
ksoftirqd threads get a very high total running time in top. This is on
a P4 machine. On a recent laptop, the load is not as high, but the
ksoftirqd threads also get a very high total running time.

-- 
Damien Wyart

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> Hello,
> 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
> > Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
> > Submitter	: Damien Wyart <damien.wyart@free.fr>
> > Date		: 2009-01-20 16:25 (26 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2
> 
> The problem is still there in 2.6.29-rc5, nothing has changed. I am
> surprised noone has looked at it as it is trivial to reproduce for me.
> 
> Load average is between 0.30 and 0.60 when the machine is idle, the two
> ksoftirqd threads get a very high total running time in top. This is on
> a P4 machine. On a recent laptop, the load is not as high, but the
> ksoftirqd threads also get a very high total running time.

Mind having a look at this anomaly with the function graph tracer? We are
interested in a representative trace that shows the weird ksoftirq activities.
[ which trace could possibly pinpoint their origin. ]

Here's the tracing quickstart:

   http://redhat.com/~mingo/tip.git/tracing-quickstart.txt

Also attached below. Let me know if you have trouble getting a good trace,
or if any of the steps were non-intuitive or burdensome to you.

Thanks,

	Ingo

------------------------------>

You probably came here because:

- you have a weird latency somewhere that you'd like to debug?
- you have some weird kernel behavior that you'd like to see in detail?
- you are curious how Linux kernel internals look like on a real system?

A good generic tool for that is the built-in function graph tracer (available
in v2.6.28 and later Linux kernels), which will provide system-wide tracing
output of any workload you are interested in.

Sample output:

 # tracer: function_graph
 #
 # CPU DURATION        FUNCTION CALLS
 # |    |   |          |   |   |   |

   1) + 39.757 us   |    }
   1) + 41.593 us   |  }
   1)               |  sys_rt_sigprocmask() {
   1)   0.399 us    |    _spin_lock_irq();
   1)   1.209 us    |  }
   1)               |  sys_read() {
   1)   0.425 us    |    fget_light();
   1)               |    vfs_read() {
   1)               |      rw_verify_area() {
   1)               |        security_file_permission() {
   1)   0.377 us    |          cap_file_permission();
   1)   1.048 us    |        }
   1)   1.803 us    |      }
   1)               |      tty_read() {
   1)   0.476 us    |        tty_paranoia_check();
   1)               |        tty_ldisc_ref_wait() {
   1)               |          tty_ldisc_try() {
   1)   0.365 us    |            _spin_lock_irqsave();
   1)   0.426 us    |            _spin_unlock_irqrestore();
   1)   1.837 us    |          }
   1)   2.544 us    |        }

To get the very latest version of the tracer, pick up the -tip tree (which,
amongst other trees, also includes the latest tracing tree and Linus's latest
upstream -git tree):

   http://people.redhat.com/mingo/tip.git/README

And enable the following .config options:

  CONFIG_DEBUG_KERNEL=y
  CONFIG_FUNCTION_TRACER=y
  CONFIG_FUNCTION_GRAPH_TRACER=y
  CONFIG_DYNAMIC_FTRACE=y

Boot into the new kernel and mount debugfs (if you dont have it mounted
already):

  mkdir /debug
  mount -t debugfs nodev /debug

That's all - now you can use the tracer. There's no user-space utilities
needed, everything is built into the kernel and all functionality can
be accessed via the /debug/tracing/* special files.

For example, the following commands in a shell prompt will capture a
1-second trace of whatever happens on your box right now:

  cd /debug/tracing/
  echo function_graph_tracer > current_tracer
  echo 1 > tracing_enabled
  sleep 1
  echo 0 > tracing_enabled
  cat trace > /tmp/trace.txt

/tmp/trace.txt will contain the trace. If you are trying to debug a bug,
send that file to kernel developers :)

Tracing can be expensive, especially if every single kernel function is
traced (which is the default).

To solve that problem you can limit the scope of traced functions via the
function filter:

  echo ""              > set_ftrace_filter        # clear all previous filters
  echo "schedule*"    >> set_ftrace_filter        # trace schedule functions
  echo "*switch_to*"  >> set_ftrace_filter        # trace context switches
  echo "*wake_up*"    >> set_ftrace_filter        # trace task wakeups

Note that if you want the rules to be additive then filters have to be
appended via '>>'. Using '>' will override all previous filters again.

Use "cat set_ftrace_filter" to see the currently traced functions. (if the
file is empty then it means 'all') Use "cat available_filter_functions" for
all kernel functions that can be traced. On a typical system there will
be tens of thousands to pick from (!).

Besides the default output you can find additional attributes in the
'trace_options' file. For example:

  echo funcgraph-proc > trace_options

Will instruct the tracer to output which process/PID generated a trace
entry:

  0)   bash-2623    |               |  sys_dup2() {
  0)   bash-2623    |               |    sys_dup3() {
  0)   bash-2623    |   0.262 us    |      _spin_lock();
  0)   bash-2623    |   0.290 us    |      expand_files();
  0)   bash-2623    |               |      filp_close() {
  0)   bash-2623    |   0.260 us    |        dnotify_flush();
  0)   bash-2623    |   0.305 us    |        locks_remove_posix();
  0)   bash-2623    |               |        fput() {
  0)   bash-2623    |               |          __fput() {
  0)   bash-2623    |   0.393 us    |            _cond_resched();

Have fun tracing, but beware, if there's even just a little bit of
a kernel developer in you then it can be highly addictive! ;-)

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Load average is between 0.30 and 0.60 when the machine is idle, the
> > two ksoftirqd threads get a very high total running time in top.
> > This is on a P4 machine. On a recent laptop, the load is not as
> > high, but the ksoftirqd threads also get a very high total running
> > time.

> Mind having a look at this anomaly with the function graph tracer? We are
> interested in a representative trace that shows the weird ksoftirq activities.
> [ which trace could possibly pinpoint their origin. ]

> Here's the tracing quickstart:

>    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt

> Also attached below. Let me know if you have trouble getting a good trace,
> or if any of the steps were non-intuitive or burdensome to you.

Thanks for your feedback & explanations; I will try to have a look at
all this material asap, I think tomorrow.

Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> > > Load average is between 0.30 and 0.60 when the machine is idle, the
> > > two ksoftirqd threads get a very high total running time in top.
> > > This is on a P4 machine. On a recent laptop, the load is not as
> > > high, but the ksoftirqd threads also get a very high total running
> > > time.
> 
> > Mind having a look at this anomaly with the function graph tracer? We are
> > interested in a representative trace that shows the weird ksoftirq activities.
> > [ which trace could possibly pinpoint their origin. ]
> 
> > Here's the tracing quickstart:
> 
> >    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
> 
> > Also attached below. Let me know if you have trouble getting a good trace,
> > or if any of the steps were non-intuitive or burdensome to you.
> 
> Thanks for your feedback & explanations; I will try to have a look at
> all this material asap, I think tomorrow.

Note that if the box you test this on is multi-core or HT, then interpreting
traces is easier if there's just a single CPU to look at. In that case i'd 
suggest to reproduce with just a single core, by turning the second one off:

   echo 0 > /sys/devices/system/cpu/cpu1/online

Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

   echo 2 > /debug/tracing/tracing_cpumask 

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo@elte.hu> [2009-02-15 11:13]:
> Note that if the box you test this on is multi-core or HT, then interpreting
> traces is easier if there's just a single CPU to look at. In that case i'd 
> suggest to reproduce with just a single core, by turning the second one off:

>    echo 0 > /sys/devices/system/cpu/cpu1/online

> Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

>    echo 2 > /debug/tracing/tracing_cpumask 

The box I test on is HT, so I tried the first suggestion and it made the
problem much less visible (but not completely absent).

So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
HT mode and then it made the problem much more visible on CPU#1:
ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
average is about 0.80 and the total running time for ksoftirqd/1 is
almost one minute (and I booted on rc5 ten minutes ago)!

So I followed the tracing steps in the tutorial (with the 1 sec sleep),
which gave me this:
http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

As I will be away until tomorrow, I did this on vanilla rc5 to get
something out today, and if tip is really needed, I will work on it
tomorrow. But maybe this vanilla trace will be helpful to you...

Do not hesitate to ask for further tests or info.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> Do not hesitate to ask for further tests or info.

I really must leave now, but please note the dmesg and config were
attached to my very first report. Will send new ones this evening (with
rc5) if you need them).

-- 
Damien Wyart

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Note that if the box you test this on is multi-core or HT, then interpreting
> > traces is easier if there's just a single CPU to look at. In that case i'd 
> > suggest to reproduce with just a single core, by turning the second one off:

> >    echo 0 > /sys/devices/system/cpu/cpu1/online

> > Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

> >    echo 2 > /debug/tracing/tracing_cpumask 

> The box I test on is HT, so I tried the first suggestion and it made the
> problem much less visible (but not completely absent).

> So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
> HT mode and then it made the problem much more visible on CPU#1:
> ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
> average is about 0.80 and the total running time for ksoftirqd/1 is
> almost one minute (and I booted on rc5 ten minutes ago)!

> So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> which gave me this:
> http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

Of course, I used your first suggestion (tracing on CPU#1) to get this
trace.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Damien Wyart <damien.wyart@free.fr> [2009-02-15 11:42]:
> > > Note that if the box you test this on is multi-core or HT, then interpreting
> > > traces is easier if there's just a single CPU to look at. In that case i'd 
> > > suggest to reproduce with just a single core, by turning the second one off:

> > >    echo 0 > /sys/devices/system/cpu/cpu1/online

> > > Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

> > >    echo 2 > /debug/tracing/tracing_cpumask 

> > The box I test on is HT, so I tried the first suggestion and it made the
> > problem much less visible (but not completely absent).

> > So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
> > HT mode and then it made the problem much more visible on CPU#1:
> > ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
> > average is about 0.80 and the total running time for ksoftirqd/1 is
> > almost one minute (and I booted on rc5 ten minutes ago)!

> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

> Of course, I used your first suggestion (tracing on CPU#1) to get this
                         ^^^^^
                         second !

> trace.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> which gave me this:
> http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

thanks. There's definitely some weirdness visible in the trace,
for example:

  0)    gpm-1879    =>   ksoftir-4   
  ------------------------------------------

  0)   0.964 us    |    finish_task_switch();
  0) ! 1768184 us |  }
  0)               |  do_softirq() {
  0)               |    __do_softirq() {
  0)               |      rcu_process_callbacks() {

the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
ksoftirqd _really_ took that much time to execute.

One possibility for these symptoms would be broken scheduler timestamps.
Could you enable absolute timestamp printing via:

    echo funcgraph-abstime  > trace_options

Also, my guess is that if you boot via idle=poll, the symptoms go away.
This would strengthen the suspicion that it's scheduler-clock troubles.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Sun, Feb 15, 2009 at 12:01:04PM +0100, Ingo Molnar wrote:
> 
> * Damien Wyart <damien.wyart@free.fr> wrote:
> 
> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz
> 
> thanks. There's definitely some weirdness visible in the trace,
> for example:
> 
>   0)    gpm-1879    =>   ksoftir-4   
>   ------------------------------------------
> 
>   0)   0.964 us    |    finish_task_switch();
>   0) ! 1768184 us |  }
>   0)               |  do_softirq() {
>   0)               |    __do_softirq() {
>   0)               |      rcu_process_callbacks() {
> 
> the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> ksoftirqd _really_ took that much time to execute.
> 
> One possibility for these symptoms would be broken scheduler timestamps.
> Could you enable absolute timestamp printing via:
> 
>     echo funcgraph-abstime  > trace_options
> 
> Also, my guess is that if you boot via idle=poll, the symptoms go away.
> This would strengthen the suspicion that it's scheduler-clock troubles.
> 
> 	Ingo

Looking at the following ksoftirqd quantum, there are some things I don't understand:

 ------------------------------------------
 0)    gpm-1879    =>   ksoftir-4   
 ------------------------------------------

 0)   0.681 us    |    finish_task_switch();
 0) ! 7670.083 us |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          cpu_quiet() {
 0)   0.509 us    |            _spin_lock_irqsave();
 0)               |            cpu_quiet_msk() {
 0)   0.503 us    |              _spin_unlock_irqrestore();
 0)   1.546 us    |            }
 0)   3.617 us    |          }
 0)   4.804 us    |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.496 us    |          force_quiescent_state();
 0)   1.608 us    |        }
 0)   8.043 us    |      }
 0)   0.533 us    |      _local_bh_enable();
 0) + 10.190 us   |    }
 0) + 11.337 us   |  }
 0)   0.537 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.516 us    |    _spin_lock_irq();
 0)   0.586 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.519 us    |            calc_delta_mine();
 0)   1.603 us    |          }
 0)   2.667 us    |        }
 0)   3.662 us    |      }
 0)   4.629 us    |    }
 0)   0.937 us    |    find_busiest_group();
 0)   0.493 us    |    msecs_to_jiffies();
 0)   0.501 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.501 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_rt();
 0)   0.494 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_idle();
 0)   4.591 us    |    }
 0)   0.501 us    |    __lock_text_start();
 0)   0.699 us    |    finish_task_switch();
 0) ! 289.895 us  |  }


The two above is the internal of context_switch, which means it entered
there in schedule():

next = pick_next_task(rq, prev);
if (likely(prev != next)) {
	sched_info_switch(prev, next);
	perf_counter_task_sched_out(prev, cpu);

	rq->nr_switches++;
	rq->curr = next;
	++*switch_count;

	context_switch(rq, prev, next); /* unlocks the rq */
	/*
	 * the context switch might have flipped the stack from under
	 * us, hence refresh the local variables.
	 */
	cpu = smp_processor_id();
	rq = cpu_rq(cpu);
}

But after that we are still in ksoftirqd. As if it bypassed the if (prev != next)
and rescheduled itself.

Well, I guess it's a bit off topic here. Whatever if this a bug in the sched_clock(),
it reminds me a possible problem in sched_clock() I saw with tracers...



 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          force_quiescent_state() {
 0)   0.478 us    |            __lock_text_start();
 0)   0.489 us    |            _spin_lock();
 0)               |            rcu_process_dyntick() {
 0)   0.496 us    |              _spin_lock_irqsave();
 0)   0.571 us    |              dyntick_save_progress_counter();
 0)               |              cpu_quiet_msk() {
 0)               |                rcu_start_gp() {
 0)   0.496 us    |                  _spin_unlock_irqrestore();
 0)   1.533 us    |                }
 0)   2.588 us    |              }
 0)   5.784 us    |            }
 0)   0.493 us    |            _spin_lock();
 0)   0.501 us    |            _spin_unlock_irqrestore();
 0) + 10.959 us   |          }
 0)               |          file_free_rcu() {
 0)   0.551 us    |            kmem_cache_free();
 0)   1.613 us    |          }
 0) + 14.486 us   |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.458 us    |          force_quiescent_state();
 0)   1.573 us    |        }
 0) + 17.661 us   |      }
 0)   0.486 us    |      _local_bh_enable();
 0) + 20.014 us   |    }
 0) + 21.094 us   |  }
 0)   0.546 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.514 us    |    _spin_lock_irq();
 0)   0.596 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.500 us    |            calc_delta_mine();
 0)   1.570 us    |          }
 0)   2.632 us    |        }
 0)   3.632 us    |      }
 0)   4.594 us    |    }
 0)   0.846 us    |    find_busiest_group();
 0)   0.471 us    |    msecs_to_jiffies();
 0)   0.506 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.486 us    |      pick_next_task_fair();
 0)   0.481 us    |      pick_next_task_rt();
 0)   0.484 us    |      pick_next_task_fair();
 0)   0.493 us    |      pick_next_task_idle();
 0)   4.544 us    |    }
 0)   0.506 us    |    __lock_text_start();
 0)   0.554 us    |    finish_task_switch();
 0) + 30.202 us   |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          cpu_quiet() {
 0)   0.493 us    |            _spin_lock_irqsave();
 0)               |            cpu_quiet_msk() {
 0)   0.504 us    |              _spin_unlock_irqrestore();
 0)   1.515 us    |            }
 0)   3.535 us    |          }
 0)   4.692 us    |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.480 us    |          force_quiescent_state();
 0)   1.593 us    |        }
 0)   7.878 us    |      }
 0)   0.491 us    |      _local_bh_enable();
 0)   9.934 us    |    }
 0) + 11.037 us   |  }
 0)   0.529 us    |  _cond_resched();
 0)   0.506 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.499 us    |    _spin_lock_irq();
 0)   0.581 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.493 us    |            calc_delta_mine();
 0)   1.548 us    |          }
 0)   2.583 us    |        }
 0)   3.559 us    |      }
 0)   4.514 us    |    }
 0)   0.797 us    |    find_busiest_group();
 0)   0.481 us    |    msecs_to_jiffies();
 0)   0.499 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.481 us    |      pick_next_task_fair();
 0)   0.479 us    |      pick_next_task_rt();
 0)   0.483 us    |      pick_next_task_fair();
 0)   0.481 us    |      pick_next_task_idle();
 0)   4.488 us    |    }
 0)   0.506 us    |    __lock_text_start();
 0)   0.672 us    |    finish_task_switch();
 0) ! 928.567 us  |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          force_quiescent_state() {
 0)   0.484 us    |            __lock_text_start();
 0)   0.511 us    |            _spin_lock();
 0)               |            rcu_process_dyntick() {
 0)   0.511 us    |              _spin_lock_irqsave();
 0)   0.564 us    |              dyntick_save_progress_counter();
 0)               |              cpu_quiet_msk() {
 0)               |                rcu_start_gp() {
 0)   0.503 us    |                  _spin_unlock_irqrestore();
 0)   1.548 us    |                }
 0)   2.582 us    |              }
 0)   6.102 us    |            }
 0)   0.501 us    |            _spin_lock();
 0)   0.501 us    |            _spin_unlock_irqrestore();
 0) + 11.265 us   |          }
 0)               |          file_free_rcu() {
 0)   0.579 us    |            kmem_cache_free();
 0)   1.643 us    |          }
 0)               |          file_free_rcu() {
 0)   0.549 us    |            kmem_cache_free();
 0)   1.588 us    |          }
 0) + 16.827 us   |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.479 us    |          force_quiescent_state();
 0)   1.593 us    |        }
 0) + 20.024 us   |      }
 0)   0.500 us    |      _local_bh_enable();
 0) + 22.081 us   |    }
 0) + 23.163 us   |  }
 0)   0.523 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.506 us    |    _spin_lock_irq();
 0)   0.586 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.516 us    |            calc_delta_mine();
 0)   1.578 us    |          }
 0)   2.628 us    |        }
 0)   3.622 us    |      }
 0)   4.604 us    |    }
 0)   0.917 us    |    find_busiest_group();
 0)   0.484 us    |    msecs_to_jiffies();
 0)   0.501 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.486 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_rt();
 0)               |      pick_next_task_fair() {
 0)   0.486 us    |      pick_next_task_idle();
 0)   4.581 us    |    }
 0)   0.496 us    |    __lock_text_start();

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

> thanks. There's definitely some weirdness visible in the trace,
> for example:

>   0)    gpm-1879    =>   ksoftir-4   
>   ------------------------------------------

>   0)   0.964 us    |    finish_task_switch();
>   0) ! 1768184 us |  }
>   0)               |  do_softirq() {
>   0)               |    __do_softirq() {
>   0)               |      rcu_process_callbacks() {

> the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> ksoftirqd _really_ took that much time to execute.

> One possibility for these symptoms would be broken scheduler timestamps.
> Could you enable absolute timestamp printing via:

>     echo funcgraph-abstime  > trace_options

Mmm, seems I do not have this option recognized in rc5, so could not
test. Will retry all this with tip tomorrow...

> Also, my guess is that if you boot via idle=poll, the symptoms go away.
> This would strengthen the suspicion that it's scheduler-clock troubles.

In fact, with idle=poll, the symptoms do not go away, they are much
stronger: without it, ksotirqd have a few % of CPU in top output; with
it, they have 20 or 30% and the global average is not far from 1.

On my laptop (I do not have it at hand today), with rc3-gitX (did not
retest with rc5), the load avg was ok, but I saw that after boot,
ksoftird threads had a quite higher running time in top than with
2.6.28. I am surprised nobody reported this yet...

I attach to this mail config and dmesg if needed (this is rc5 without
idle=poll).

If a trace with funcgraph-abstime is interesting for you with tip,
I will do this tomorrow.

-- 
Damien Wyart

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 42926 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.29-rc5
# Sun Feb 15 11:07:36 2009
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
# CONFIG_CLASSIC_RCU is not set
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
# CONFIG_FAIR_GROUP_SCHED is not set
# CONFIG_RT_GROUP_SCHED is not set
# CONFIG_USER_SCHED is not set
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CGROUP_NS is not set
# CONFIG_CGROUP_FREEZER is not set
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_CPUSETS is not set
# CONFIG_CGROUP_CPUACCT is not set
# CONFIG_RESOURCE_COUNTERS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
# CONFIG_RELAY is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
# CONFIG_BLK_DEV_INITRD is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBD=y
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
# CONFIG_FREEZER is not set

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
# CONFIG_X86_RDC321X is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_CYRIX_32=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR_32=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
# CONFIG_HPET_TIMER is not set
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
# CONFIG_IOMMU_API is not set
CONFIG_NR_CPUS=2
CONFIG_SCHED_SMT=y
# CONFIG_SCHED_MC is not set
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_UNEVICTABLE_LRU=y
CONFIG_HIGHPTE=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW_64K=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
# CONFIG_ACPI_FAN is not set
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_COMPAT_NET_DEV_OPS=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_IRC=m
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_NF_CT_NETLINK is not set
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
# CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
# CONFIG_NF_NAT_SIP is not set
CONFIG_IP_NF_MANGLE=m
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
CONFIG_NET_CLS_FW=m
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
CONFIG_NET_CLS_FLOW=m
# CONFIG_NET_CLS_CGROUP is not set
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_PHONET is not set
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_BLK_DEV_HD is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
# CONFIG_SCSI_DH is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
# CONFIG_SATA_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
CONFIG_E100=y
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SMSC9420 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_NET_POCKET is not set
# CONFIG_ATL2 is not set
# CONFIG_NETDEV_1000 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
# CONFIG_SPI is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_HWMON is not set
CONFIG_THERMAL=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_REGULATOR is not set

#
# Multimedia devices
#

#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set

#
# Multimedia drivers
#
# CONFIG_DAB is not set

#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_HRTIMER is not set
CONFIG_SND_RTCTIMER=y
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_AC97_CODEC=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
# CONFIG_SND_PORTMAN2X4 is not set
# CONFIG_SND_AC97_POWER_SAVE is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
CONFIG_SND_EMU10K1=y
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SIS7019 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_USB is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set

#
# Special HID drivers
#
CONFIG_HID_COMPAT=y
CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_GYRATION=m
CONFIG_HID_LOGITECH=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_HID_NTRIG=m
CONFIG_HID_PANTHERLORD=m
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_SUNPLUS=m
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_TOPSEED=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_MON is not set
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# Enable Host or Gadget support to see Inventra options
#

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed;
#

#
# see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set
# CONFIG_X86_PLATFORM_DEVICES is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_DMIID is not set
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
CONFIG_EXT4_FS=m
CONFIG_EXT4DEV_COMPAT=y
CONFIG_EXT4_FS_XATTR=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
# CONFIG_EXT4_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=m
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_FILE_LOCKING=y
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set
CONFIG_EXPORTFS=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_HW_BRANCH_TRACER=y
CONFIG_RING_BUFFER=y
CONFIG_TRACING=y

#
# Tracers
#
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
# CONFIG_BOOT_TRACER is not set
# CONFIG_TRACE_BRANCH_PROFILING is not set
# CONFIG_POWER_TRACER is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_HW_BRANCH_TRACER is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_PRINTK_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_4KSTACKS=y
CONFIG_DOUBLEFAULT=y
# CONFIG_MMIOTRACE is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_586 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_586 is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=m
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y

[-- Attachment #3: dmesg --]
[-- Type: text/plain, Size: 23572 bytes --]

Initializing cgroup subsys cpu
Linux version 2.6.29-rc5 (root@brouette) (gcc version 4.3.3 (Debian 4.3.3-3) ) #1 SMP Sun Feb 15 11:15:54 CET 2009
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  NSC Geode by NSC
  Cyrix CyrixInstead
  Centaur CentaurHauls
  Transmeta GenuineTMx86
  Transmeta TransmetaCPU
  UMC UMC UMC UMC
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff74000 (usable)
 BIOS-e820: 000000007ff74000 - 000000007ff76000 (ACPI NVS)
 BIOS-e820: 000000007ff76000 - 000000007ff97000 (ACPI data)
 BIOS-e820: 000000007ff97000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fecf0000 - 00000000fecf1000 (reserved)
 BIOS-e820: 00000000fed20000 - 00000000fed90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
DMI 2.3 present.
last_pfn = 0x7ff74 max_arch_pfn = 0x100000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
kernel direct mapping tables up to 377fe000 @ 7000-c000
ACPI: RSDP 000FEB90, 0014 (r0 DELL  )
ACPI: RSDT 000FD1CA, 0034 (r1 DELL    8300           8 ASL        61)
ACPI: FACP 000FD1FE, 0074 (r1 DELL    8300           8 ASL        61)
FADT: X_PM1a_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
ACPI: DSDT FFFC6E75, 2426 (r1   DELL    dt_ex     1000 MSFT  100000D)
ACPI: FACS 7FF74000, 0040
ACPI: SSDT FFFC93D8, 00BA (r1   DELL    st_ex     1000 MSFT  100000D)
ACPI: APIC 000FD272, 006C (r1 DELL    8300           8 ASL        61)
ACPI: BOOT 000FD2DE, 0028 (r1 DELL    8300           8 ASL        61)
ACPI: Local APIC address 0xfee00000
1159MB HIGHMEM available.
887MB LOWMEM available.
  mapped low ram: 0 - 377fe000
  low ram: 00000000 - 377fe000
  bootmap 00008000 - 0000ef00
(8 early reservations) ==> bootmem [0000000000 - 00377fe000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000001000 - 0000002000]    EX TRAMPOLINE ==> [0000001000 - 0000002000]
  #2 [0000006000 - 0000007000]       TRAMPOLINE ==> [0000006000 - 0000007000]
  #3 [0000100000 - 0000500640]    TEXT DATA BSS ==> [0000100000 - 0000500640]
  #4 [0000501000 - 0000504000]    INIT_PG_TABLE ==> [0000501000 - 0000504000]
  #5 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
  #6 [0000007000 - 0000008000]          PGTABLE ==> [0000007000 - 0000008000]
  #7 [0000008000 - 000000f000]          BOOTMAP ==> [0000008000 - 000000f000]
found SMP MP-table at [c00fe710] 000fe710
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  Normal   0x00001000 -> 0x000377fe
  HighMem  0x000377fe -> 0x0007ff74
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x000000a0
    0: 0x00000100 -> 0x0007ff74
On node 0 totalpages: 524052
free_area_init_node: node 0, pgdat c044ed80, node_mem_map c1000000
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 3968 pages, LIFO batch:0
  Normal zone: 1744 pages used for memmap
  Normal zone: 221486 pages, LIFO batch:31
  HighMem zone: 2319 pages used for memmap
  HighMem zone: 294503 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
4 Processors exceeds NR_CPUS limit of 2
SMP: Allowing 2 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 24
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Allocating 36864 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 519957
Kernel command line: root=/dev/sdb2 ro vga=0x307 selinux=0 elevator=cfq video=vesafb:mtrr:3 
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
Experimental hierarchical RCU implementation.
Experimental hierarchical RCU init done.
CPU 0 irqstacks, hard=c04ae000 soft=c04ac000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Fast TSC calibration using PIT
Detected 2992.546 MHz processor.
Console: colour dummy device 80x25
console [tty0] enabled
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2073812k/2096592k available (2449k kernel code, 21420k reserved, 1034k data, 248k init, 1187288k highmem)
virtual kernel memory layout:
    fixmap  : 0xfff9f000 - 0xfffff000   ( 384 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf7ffe000 - 0xff7fe000   ( 120 MB)
    lowmem  : 0xc0000000 - 0xf77fe000   ( 887 MB)
      .init : 0xc046b000 - 0xc04a9000   ( 248 kB)
      .data : 0xc0364487 - 0xc0466e40   (1034 kB)
      .text : 0xc0100000 - 0xc0364487   (2449 kB)
Checking if this processor honours the WP bit even in supervisor mode...Ok.
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 5985.09 BogoMIPS (lpj=2992546)
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
using mwait in idle threads.
Checking 'hlt' instruction... OK.
ACPI: Core revision 20081204
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 13232 entries in 52 pages
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
CPU 1 irqstacks, hard=c04af000 soft=c04ad000
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5983.76 BogoMIPS (lpj=2991881)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Brought up 2 CPUs
Total of 2 processors activated (11968.85 BogoMIPS).
net_namespace: 504 bytes
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbb30, last bus=2
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:00.0: reg 10 32bit mmio: [0xe8000000-0xefffffff]
pci 0000:00:1d.0: reg 20 io port: [0xff80-0xff9f]
pci 0000:00:1d.1: reg 20 io port: [0xff60-0xff7f]
pci 0000:00:1d.2: reg 20 io port: [0xff40-0xff5f]
pci 0000:00:1d.3: reg 20 io port: [0xff20-0xff3f]
pci 0000:00:1d.7: reg 10 32bit mmio: [0xffa80800-0xffa80bff]
pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.7: PME# disabled
pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
pci 0000:00:1f.0: quirk: region 0880-08bf claimed by ICH4 GPIO
pci 0000:00:1f.1: reg 10 io port: [0x1f0-0x1f7]
pci 0000:00:1f.1: reg 14 io port: [0x3f4-0x3f7]
pci 0000:00:1f.1: reg 18 io port: [0x170-0x177]
pci 0000:00:1f.1: reg 1c io port: [0x374-0x377]
pci 0000:00:1f.1: reg 20 io port: [0xffa0-0xffaf]
pci 0000:00:1f.1: reg 24 32bit mmio: [0xfebffc00-0xfebfffff]
pci 0000:00:1f.2: reg 10 io port: [0xfe00-0xfe07]
pci 0000:00:1f.2: reg 14 io port: [0xfe10-0xfe13]
pci 0000:00:1f.2: reg 18 io port: [0xfe20-0xfe27]
pci 0000:00:1f.2: reg 1c io port: [0xfe30-0xfe33]
pci 0000:00:1f.2: reg 20 io port: [0xfea0-0xfeaf]
pci 0000:00:1f.3: reg 20 io port: [0xefe0-0xefff]
pci 0000:01:00.0: reg 10 32bit mmio: [0xfd000000-0xfdffffff]
pci 0000:01:00.0: reg 14 32bit mmio: [0xf0000000-0xf7ffffff]
pci 0000:01:00.0: reg 30 32bit mmio: [0xfea00000-0xfea1ffff]
pci 0000:00:01.0: bridge 32bit mmio: [0xfd000000-0xfeafffff]
pci 0000:00:01.0: bridge 32bit mmio pref: [0xf0000000-0xf7ffffff]
pci 0000:02:00.0: reg 10 io port: [0xdf20-0xdf3f]
pci 0000:02:00.0: supports D1 D2
pci 0000:02:00.1: reg 10 io port: [0xdf18-0xdf1f]
pci 0000:02:00.1: supports D1 D2
pci 0000:02:08.0: reg 10 32bit mmio: [0xfcfff000-0xfcffffff]
pci 0000:02:08.0: reg 14 io port: [0xdf40-0xdf7f]
pci 0000:02:08.0: supports D1 D2
pci 0000:02:08.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:02:08.0: PME# disabled
pci 0000:00:1e.0: transparent bridge
pci 0000:00:1e.0: bridge io port: [0xd000-0xdfff]
pci 0000:00:1e.0: bridge 32bit mmio: [0xfcf00000-0xfcffffff]
pci_bus 0000:00: on NUMA node 0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *4 5 6 7 9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs *3 4 5 6 7 9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 7 9 10 11 12 15)
SCSI subsystem initialized
libata version 3.00 loaded.
PCI: Using ACPI for IRQ routing
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:07: io resource (0x800-0x85f) overlaps 0000:00:1f.0 BAR 7 (0x800-0x87f), disabling
pnp 00:07: io resource (0x860-0x8ff) overlaps 0000:00:1f.0 BAR 7 (0x800-0x87f), disabling
pnp: PnP ACPI: found 8 devices
ACPI: ACPI bus type pnp unregistered
system 00:00: iomem range 0x0-0x9ffff could not be reserved
system 00:00: iomem range 0x100000-0xffffff could not be reserved
system 00:00: iomem range 0x1000000-0x7ff73fff could not be reserved
system 00:00: iomem range 0xc0000-0xfffff could not be reserved
system 00:00: iomem range 0xfec00000-0xfec0ffff has been reserved
system 00:00: iomem range 0xfee00000-0xfee0ffff has been reserved
system 00:00: iomem range 0xfed20000-0xfed8ffff has been reserved
system 00:00: iomem range 0xfecf0000-0xfecf0fff has been reserved
system 00:00: iomem range 0xffb00000-0xffbfffff has been reserved
system 00:00: iomem range 0xffc00000-0xffffffff has been reserved
system 00:07: ioport range 0xc00-0xc7f has been reserved
pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
pci 0000:00:01.0:   IO window: disabled
pci 0000:00:01.0:   MEM window: 0xfd000000-0xfeafffff
pci 0000:00:01.0:   PREFETCH window: 0x000000f0000000-0x000000f7ffffff
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:02
pci 0000:00:1e.0:   IO window: 0xd000-0xdfff
pci 0000:00:1e.0:   MEM window: 0xfcf00000-0xfcffffff
pci 0000:00:1e.0:   PREFETCH window: disabled
pci 0000:00:1e.0: setting latency timer to 64
pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffff]
pci_bus 0000:01: resource 0 mem: [0x0-0x0]
pci_bus 0000:01: resource 1 mem: [0xfd000000-0xfeafffff]
pci_bus 0000:01: resource 2 mem: [0xf0000000-0xf7ffffff]
pci_bus 0000:01: resource 3 mem: [0x0-0x0]
pci_bus 0000:02: resource 0 io:  [0xd000-0xdfff]
pci_bus 0000:02: resource 1 mem: [0xfcf00000-0xfcffffff]
pci_bus 0000:02: resource 2 mem: [0x0-0x0]
pci_bus 0000:02: resource 3 io:  [0x00-0xffff]
pci_bus 0000:02: resource 4 mem: [0x000000-0xffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
SGI XFS with security attributes, large block/inode numbers, no debug enabled
msgmni has been set to 1733
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
pci 0000:01:00.0: Boot video device
pci 0000:02:08.0: Firmware left e100 interrupts enabled; disabling
vesafb: framebuffer at 0xf0000000, mapped to 0xf8080000, using 2560k, total 131072k
vesafb: mode is 1280x1024x8, linelength=1280, pages=1
vesafb: protected mode interface info at c000:f080
vesafb: pmi: set display start = c00cf0b6, set palette = c00cf120
vesafb: pmi: ports = 3b4 3b5 3ba 3c0 3c1 3c4 3c5 3c6 3c7 3c8 3c9 3cc 3ce 3cf 3d0 3d1 3d2 3d3 3d4 3d5 3da 
vesafb: scrolling: redraw
vesafb: Pseudocolor: size=8:8:8:8, shift=0:0:0:0
Console: switching to colour frame buffer device 160x64
fb0: VESA VGA frame buffer device
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input1
ACPI: Power Button (CM) [VBTN]
processor ACPI_CPU:00: registered as cooling_device0
processor ACPI_CPU:01: registered as cooling_device1
Real Time Clock Driver v1.12b
e100: Intel(R) PRO/100 Network Driver, 3.5.23-k6-NAPI
e100: Copyright(c) 1999-2006 Intel Corporation
e100 0000:02:08.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
e100 0000:02:08.0: PME# disabled
e100: eth0: e100_probe: addr 0xfcfff000, irq 20, MAC addr 00:0c:f1:b6:ba:54
Driver 'sd' needs updating - please use bus_type methods
ata_piix 0000:00:1f.1: version 2.12
ata_piix 0000:00:1f.1: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ata_piix 0000:00:1f.1: setting latency timer to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
ata1: port disabled. ignoring.
ata2.00: ATAPI: SAMSUNG DVD-ROM SD-616T, F310, max UDMA/33
ata2.01: ATAPI: SAMSUNG CD-R/RW SW-252S, R901, max UDMA/33
ata2.00: configured for UDMA/33
ata2.01: configured for UDMA/33
scsi 1:0:0:0: CD-ROM            SAMSUNG  DVD-ROM SD-616T  F310 PQ: 0 ANSI: 5
scsi 1:0:0:0: Attached scsi generic sg0 type 5
scsi 1:0:1:0: CD-ROM            SAMSUNG  CD-R/RW SW-252S  R901 PQ: 0 ANSI: 5
scsi 1:0:1:0: Attached scsi generic sg1 type 5
ata_piix 0000:00:1f.2: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
ata_piix 0000:00:1f.2: setting latency timer to 64
scsi2 : ata_piix
scsi3 : ata_piix
ata3: SATA max UDMA/133 cmd 0xfe00 ctl 0xfe10 bmdma 0xfea0 irq 18
ata4: SATA max UDMA/133 cmd 0xfe20 ctl 0xfe30 bmdma 0xfea8 irq 18
ata3.00: ATA-6: WDC WD740GD-75FLA0, 21.08U21, max UDMA/133
ata3.00: 144531250 sectors, multi 8: LBA48 
ata3.00: applying bridge limits
ata3.00: configured for UDMA/100
scsi 2:0:0:0: Direct-Access     ATA      WDC WD740GD-75FL 21.0 PQ: 0 ANSI: 5
sd 2:0:0:0: [sda] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sda] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 < sda5 >
sd 2:0:0:0: [sda] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg2 type 0
Switched to high resolution mode on CPU 1
Switched to high resolution mode on CPU 0
ata4.00: ATA-6: WDC WD740GD-75FLA0, 21.08U21, max UDMA/133
ata4.00: 144531250 sectors, multi 8: LBA48 
ata4.00: applying bridge limits
ata4.00: configured for UDMA/100
scsi 3:0:0:0: Direct-Access     ATA      WDC WD740GD-75FL 21.0 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdb] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdb] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2 sdb3 < sdb5 sdb6 sdb7 sdb8 >
sd 3:0:0:0: [sdb] Attached SCSI disk
sd 3:0:0:0: Attached scsi generic sg3 type 0
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
cpuidle: using governor ladder
cpuidle: using governor menu
Advanced Linux Sound Architecture Driver Version 1.0.18a.
EMU10K1_Audigy 0000:02:00.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
Control name 'Sigmatel Surround Phase Inversion Playback Switch' truncated to 'Sigmatel Surround Phase Inversion Playback '
ALSA device list:
  #0: SB Live! [Unknown] (rev.10, serial:0x80671102) at 0xdf20, irq 21
TCP cubic registered
NET: Registered protocol family 17
Using IPI No-Shortcut mode
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly on device 8:18.
Freeing unused kernel memory: 248k freed
parport_pc 00:06: reported by Plug and Play ACPI
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
uhci_hcd: USB Universal Host Controller Interface driver
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.0: setting latency timer to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 16, io base 0x0000ff80
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1d.1: setting latency timer to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000ff60
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: setting latency timer to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000ff40
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.3: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.3: setting latency timer to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.3: irq 16, io base 0x0000ff20
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
Driver 'sr' needs updating - please use bus_type methods
sr0: scsi3-mmc drive: 1x/48x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Warning! ehci_hcd should always be loaded before uhci_hcd and ohci_hcd, not after
sr1: scsi3-mmc drive: 48x/16x writer cd/rw xa/form2 cdda tray
sr 1:0:1:0: Attached scsi CD-ROM sr1
ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.7: setting latency timer to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: cache line size of 128 is not supported
ehci_hcd 0000:00:1d.7: irq 23, io mem 0xffa80800
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
usb usb5: configuration #1 chosen from 1 choice
input: PC Speaker as /devices/platform/pcspkr/input/input2
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
usb 1-2: new full speed USB device using uhci_hcd and address 2
Adding 1212896k swap on /dev/sdb7.  Priority:-1 extents:1 across:1212896k 
usb 1-2: configuration #1 chosen from 1 choice
usb 2-1: new low speed USB device using uhci_hcd and address 2
EXT3 FS on sdb2, internal journal
usb 2-1: configuration #1 chosen from 1 choice
usb 3-2: new low speed USB device using uhci_hcd and address 2
generic-usb 0003:056D:0002.0001: claimed by neither input, hiddev nor hidraw
usb 3-2: configuration #1 chosen from 1 choice
input: HID 05af:0310 as /devices/pci0000:00/0000:00:1d.1/usb2/2-1/2-1:1.0/input/input3
generic-usb 0003:05AF:0310.0002: input: USB HID v1.10 Keyboard [HID 05af:0310] on usb-0000:00:1d.1-1/input0
input: HID 05af:0310 as /devices/pci0000:00/0000:00:1d.1/usb2/2-1/2-1:1.1/input/input4
generic-usb 0003:05AF:0310.0003: input: USB HID v1.10 Device [HID 05af:0310] on usb-0000:00:1d.1-1/input1
input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.2/usb3/3-2/3-2:1.0/input/input5
generic-usb 0003:046D:C51B.0004: input: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:1d.2-2/input0
generic-usb 0003:046D:C51B.0005: claimed by neither input, hiddev nor hidraw
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
XFS mounting filesystem sdb5
Ending clean XFS mount for filesystem: sdb5
XFS mounting filesystem sdb6
Ending clean XFS mount for filesystem: sdb6
XFS mounting filesystem sdb8
Ending clean XFS mount for filesystem: sdb8
XFS mounting filesystem sda5
Ending clean XFS mount for filesystem: sda5
e100: eth0 NIC Link is Up 100 Mbps Full Duplex
lp0: using parport0 (interrupt-driven).
warning: `ntpd' uses 32-bit capabilities (legacy support in use)
CPU 1 is now offline
SMP alternatives: switching to UP code
SMP alternatives: switching to SMP code
CPU 1 irqstacks, hard=c04af000 soft=c04ad000
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5984.41 BogoMIPS (lpj=2992209)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Switched to high resolution mode on CPU 1
Warning: Processor Platform Limit event detected, but not handled.
Consider compiling CPUfreq support into your kernel.

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Also, my guess is that if you boot via idle=poll, the symptoms go away.
> > This would strengthen the suspicion that it's scheduler-clock troubles.

> In fact, with idle=poll, the symptoms do not go away, they are much
> stronger: without it, ksotirqd have a few % of CPU in top output; with
> it, they have 20 or 30% and the global average is not far from 1.

Not true each time, in fact. Another try with this option gave same
symptoms as without it, not stronger. In any case, they never go away.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> > > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > > which gave me this:
> > > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz
> 
> > thanks. There's definitely some weirdness visible in the trace,
> > for example:
> 
> >   0)    gpm-1879    =>   ksoftir-4   
> >   ------------------------------------------
> 
> >   0)   0.964 us    |    finish_task_switch();
> >   0) ! 1768184 us |  }
> >   0)               |  do_softirq() {
> >   0)               |    __do_softirq() {
> >   0)               |      rcu_process_callbacks() {
> 
> > the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> > ksoftirqd _really_ took that much time to execute.
> 
> > One possibility for these symptoms would be broken scheduler timestamps.
> > Could you enable absolute timestamp printing via:
> 
> >     echo funcgraph-abstime  > trace_options
> 
> Mmm, seems I do not have this option recognized in rc5, so could not
> test. Will retry all this with tip tomorrow...

Yeah, it got renamed in -tip - in rc5 it's iter_ctrl.

> > Also, my guess is that if you boot via idle=poll, the symptoms go away.
> > This would strengthen the suspicion that it's scheduler-clock troubles.
> 
> In fact, with idle=poll, the symptoms do not go away, they are much
> stronger: without it, ksotirqd have a few % of CPU in top output; with
> it, they have 20 or 30% and the global average is not far from 1.
> 
> On my laptop (I do not have it at hand today), with rc3-gitX (did not
> retest with rc5), the load avg was ok, but I saw that after boot,
> ksoftird threads had a quite higher running time in top than with
> 2.6.28. I am surprised nobody reported this yet...
> 
> I attach to this mail config and dmesg if needed (this is rc5 without
> idle=poll).
> 
> If a trace with funcgraph-abstime is interesting for you with tip,
> I will do this tomorrow.

Yes, an abstime trace would be useful.

> checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> Switched to high resolution mode on CPU 1

Lets double-check your scheduler clock first. Without being able to
trust the clock we cannot trust the task stats nor the trace output.

What does this check display:

  http://people.redhat.com/mingo/time-warp-test/time-warp-test.c

Does it find any TSC time warps?

Also, could you send the output of:

  http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

Run it while you can see the ksoftirqd anomaly.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello,

Ok, I've redone the tests under tip from this morning (Paris time).
Everything is in http://damien.wyart.free.fr/ksoftirqd_pb/

* Ingo Molnar <mingo@elte.hu> [2009-02-15 20:31]:
> Yes, an abstime trace would be useful.

The corresponding file is trace_tip_2009.02.16_ksoftirqd_pb_abstime.txt.gz
and there is also a trace without abstime:
trace_tip_2009.02.16_ksoftirqd_pb.txt.gz

> > checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> > Switched to high resolution mode on CPU 1

> Lets double-check your scheduler clock first. Without being able to
> trust the clock we cannot trust the task stats nor the trace output.

> What does this check display:

>   http://people.redhat.com/mingo/time-warp-test/time-warp-test.c

The file is time-warp-test_result.txt
I've let it run for a few tens of minutes; the first number varies
slightly sometimes. The second one stays at 0.

> Does it find any TSC time warps?

Seems not.

> Also, could you send the output of:

>   http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

> Run it while you can see the ksoftirqd anomaly.

In fact I see it all the time when the machine is idle. When something
runs (spamd for example), the running time of ksoftirqd stops
increasing, and it goes back to increasing like crazy when idle state
comes back.

The corresponding file is cfs-debug-info-2009.02.16-08.09.17.gz


Hope this will be useful; do not hesitate to ask for further info. Now
that I have tip, I guess it will be easier.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> > Does it find any TSC time warps?
> 
> Seems not.

ok, it's indeed pretty conclusive:

|  2 CPUs, running 2 parallel test-tasks.
|  checking for time-warps via:
|  - read time stamp counter (RDTSC) instruction (cycle resolution)
| 
|   | TSC: 0.32us, fail:0 |

i suspect you ran it for at least a couple of minutes, and the ksoftirqd 
anomaly occured during it, right?

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > > Does it find any TSC time warps?

> > Seems not.

> ok, it's indeed pretty conclusive:

> |  2 CPUs, running 2 parallel test-tasks.
> |  checking for time-warps via:
> |  - read time stamp counter (RDTSC) instruction (cycle resolution)
> | 
> |   | TSC: 0.32us, fail:0 |

> i suspect you ran it for at least a couple of minutes, and the ksoftirqd 
> anomaly occured during it, right?

Yes, that's right. I ran it for a few ten of minutes and the ksoftirqd
anomaly was occuring.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> > Also, could you send the output of:
> 
> >   http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
> 
> > Run it while you can see the ksoftirqd anomaly.
> 
> In fact I see it all the time when the machine is idle. When something 
> runs (spamd for example), the running time of ksoftirqd stops 
> increasing, and it goes back to increasing like crazy when idle state 
> comes back.

here's the ksoftirqd stats from your data:

top - 08:09:20 up 2 min,  1 user,  load average: 0.95, 0.35, 0.13
Tasks: 100 total,   1 running,  99 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.3%sy,  0.0%ni, 89.2%id,  0.0%wa,  0.0%hi,  9.0%si,  0.0%st
Mem:   2074980k total,   236544k used,  1838436k free,     5032k buffers
Swap:  1212896k total,        0k used,  1212896k free,    66016k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2280 root      15  -5     0    0    0 S   22  0.0   0:06.39 [ksoftirqd/1]      
 2452 root      20   0  2400 1132  888 R    1  0.1   0:00.02 top -H -c -b -d 1 -
    1 root      20   0  2100  688  588 S    0  0.0   0:00.48 init [2]           
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kthreadd]         

ksoftirqd really seems to be active (according to scheduler stats), and 
has gathered more than 6 seconds of runtime after 2 minutes of uptime - 
that is clearly anomalous.

The main method how softirqs get generated are hardirqs, but the hardirq 
is low on your box (as expected):

-- interrupts: --
           CPU0       CPU1       
  0:        155          0   IO-APIC-edge      timer
  1:          2          0   IO-APIC-edge      i8042
  7:          0          0   IO-APIC-edge      parport0
  8:          2          0   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          4          0   IO-APIC-edge      i8042
 14:          0          0   IO-APIC-edge      ata_piix
 15:        132          0   IO-APIC-edge      ata_piix
 16:        213          0   IO-APIC-fasteoi   uhci_hcd:usb2, uhci_hcd:usb5
 18:       6908          0   IO-APIC-fasteoi   ata_piix, uhci_hcd:usb4
 19:        221          0   IO-APIC-fasteoi   uhci_hcd:usb3
 20:       1072          0   IO-APIC-fasteoi   eth0
 21:          0          0   IO-APIC-fasteoi   EMU10K1
 23:          4          0   IO-APIC-fasteoi   ehci_hcd:usb1
NMI:          0          0   Non-maskable interrupts
LOC:      79306      74184   Local timer interrupts
CNT:          0          0   Performance counter interrupts
RES:      32672      13541   Rescheduling interrupts
CAL:          8         40   Function call interrupts
TLB:        235        591   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0

about 200K irqs - most of which are lapic timer irqs. Thus 6 seconds of 
ksoftirqd runtime is way too much.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

scheduler clock deltas look healthy at first sight:

$ grep clock-delta cfs-debug-info-2009.02.16-08.09.17

clock-delta                        :                  285
clock-delta                        :                  313
clock-delta                        :                  286
clock-delta                        :                  311
clock-delta                        :                  348
clock-delta                        :                  301
clock-delta                        :                  386
clock-delta                        :                  318
clock-delta                        :                  275
clock-delta                        :                  283
clock-delta                        :                  343
clock-delta                        :                  341
clock-delta                        :                  318
clock-delta                        :                  295
clock-delta                        :                  330
clock-delta                        :                  311
clock-delta                        :                  286
clock-delta                        :                  333
clock-delta                        :                  309
clock-delta                        :                  369
clock-delta                        :                 1102
clock-delta                        :                  323
clock-delta                        :                  283
clock-delta                        :                  333
clock-delta                        :                  316
clock-delta                        :                  300
clock-delta                        :                  278
clock-delta                        :                  286
clock-delta                        :                  364
clock-delta                        :                  298
clock-delta                        :                  326
clock-delta                        :                  353
clock-delta                        :                  323
clock-delta                        :                  286
clock-delta                        :                  286
clock-delta                        :                  256
clock-delta                        :                  343
clock-delta                        :                  305
clock-delta                        :                  363
clock-delta                        :                  283
clock-delta                        :                  280
clock-delta                        :                  301
clock-delta                        :                  283
clock-delta                        :                  308
clock-delta                        :                  316
clock-delta                        :                  315
clock-delta                        :                  318
clock-delta                        :                  311
clock-delta                        :                  298
clock-delta                        :                  343
clock-delta                        :                  276
clock-delta                        :                  324
clock-delta                        :                  285
clock-delta                        :                  268
clock-delta                        :                  281
clock-delta                        :                  280
clock-delta                        :                  338
clock-delta                        :                 1060
clock-delta                        :                  315
clock-delta                        :                  315
clock-delta                        :                  298
clock-delta                        :                  308
clock-delta                        :                  281
clock-delta                        :                  323
clock-delta                        :                  321
clock-delta                        :                  389
clock-delta                        :                  288
clock-delta                        :                  355
clock-delta                        :                  295
clock-delta                        :                  284
clock-delta                        :                 1037
clock-delta                        :                  298
clock-delta                        :                  348
clock-delta                        :                  328
clock-delta                        :                  305
clock-delta                        :                  358
clock-delta                        :                  340
clock-delta                        :                  406
clock-delta                        :                  323
clock-delta                        :                  306
clock-delta                        :                  295
clock-delta                        :                  294
clock-delta                        :                  288
clock-delta                        :                  363
clock-delta                        :                  296
clock-delta                        :                  321
clock-delta                        :                  278
clock-delta                        :                  338
clock-delta                        :                  291
clock-delta                        :                  318
clock-delta                        :                  331
clock-delta                        :                  305
clock-delta                        :                  353
clock-delta                        :                  319
clock-delta                        :                  338
clock-delta                        :                  286
clock-delta                        :                  309
clock-delta                        :                  298
clock-delta                        :                  316

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

Ok, the ksoftirqd/1 stats contain a smoking gun:

ksoftirqd/1 (2280, #threads: 1)
---------------------------------------------------------
se.exec_start                      :        152642.614531
se.vruntime                        :         57051.648140
se.sum_exec_runtime                :          5741.506722
se.avg_overlap                     :             0.000000
se.avg_wakeup                      :            10.000000
se.wait_start                      :             0.000000
se.sleep_start                     :        152642.623948
se.block_start                     :             0.000000
se.sleep_max                       :         22662.455146
se.block_max                       :          4096.496408
se.exec_max                        :             0.016849
se.slice_max                       :             0.000000
se.wait_max                        :             0.053862
se.wait_sum                        :          7899.023463
se.wait_count                      :              4235574
sched_info.bkl_count               :                    0
se.nr_migrations                   :                    1
se.nr_migrations_cold              :                    0
se.nr_failed_migrations_affine     :                   32
se.nr_failed_migrations_running    :                   47
se.nr_failed_migrations_hot        :                   23
se.nr_forced_migrations            :                    0
se.nr_forced2_migrations           :                    5
se.nr_wakeups                      :              4235539
se.nr_wakeups_sync                 :                    0
se.nr_wakeups_migrate              :                   11
se.nr_wakeups_local                :              4235506
se.nr_wakeups_remote               :                   34
se.nr_wakeups_affine               :                    4
se.nr_wakeups_affine_attempts      :                   44
se.nr_wakeups_passive              :                    7
se.nr_wakeups_idle                 :                    0
avg_atom                           :             0.001355
avg_per_cpu                        :          5741.514564
nr_switches                        :              4235404
nr_voluntary_switches              :              4235389
nr_involuntary_switches            :                   18
se.load.weight                     :                 3121
policy                             :                    0
prio                               :                  115
clock-delta                        :                  295

these bits:

 se.sum_exec_runtime                :          5741.506722
 nr_switches                        :              4235404
 nr_voluntary_switches              :              4235389
 nr_involuntary_switches            :                   18

mean that ksoftirqd _really_ ran more than 4 million times since bootup 
- that is _highly_ anomalous. It means that scheduler clock is fine, and 
that your box is really running a lot of softirq workload.

Here is how it should look like normally. A 16-way testbox with almost 
an hour of uptime, running high load with a lot of networking. Its CPU#0 
ksoftirqd [the busiest one] has these stats:

 se.sum_exec_runtime                :             0.523552
 nr_switches                        :                   42
 nr_voluntary_switches              :                   42
 nr_involuntary_switches            :                    0

it ran only 42 times. That is a normal ksoftirqd pattern.

I'll check your traces as the next step.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> Hello,
> 
> Ok, I've redone the tests under tip from this morning (Paris time).
> Everything is in http://damien.wyart.free.fr/ksoftirqd_pb/
> 
> * Ingo Molnar <mingo@elte.hu> [2009-02-15 20:31]:
> > Yes, an abstime trace would be useful.
> 
> The corresponding file is trace_tip_2009.02.16_ksoftirqd_pb_abstime.txt.gz
> and there is also a trace without abstime:
> trace_tip_2009.02.16_ksoftirqd_pb.txt.gz

hm, we need a trace with both abstime and process information included:

  echo funcgraph-proc     > trace_options
  echo funcgraph-abstime  > trace_options

Also, at 140 msecs the duration is a bit short - could you please make a 
1-2 seconds capture? You can do that by increasing the number in 
buffer_size_kb 10-fold:

   echo 14100 > buffer_size_kb 

(your defaults might be different)

You can see the duration of the trace by looking at the first timestamp 
and the last one:

  310.846260 |   0)   2.380 us    |                            }
  [...]
  457.712729 |   1)               |        dequeue_task() {

Hm ... even with this limited trace, there's an unusually high amount of 
RCU activities. Each activity goes like this:

  457.680976 |   1)               |  do_softirq() {
  457.680976 |   1)               |    __do_softirq() {
  457.680977 |   1)               |      rcu_process_callbacks() {
  457.680977 |   1)               |        __rcu_process_callbacks() {
4  457.680978 |   1)   0.478 us    |          force_quiescent_state();
  457.680979 |   1)   1.591 us    |        }
  457.680979 |   1)               |        __rcu_process_callbacks() {
  457.680980 |   1)   0.478 us    |          force_quiescent_state();
  457.680981 |   1)               |          cpu_quiet() {
  457.680981 |   1)   0.506 us    |            _spin_lock_irqsave();
  457.680982 |   1)   0.496 us    |            _spin_unlock_irqrestore();
  457.680983 |   1)   2.545 us    |          }
  457.680984 |   1)   4.626 us    |        }
  457.680985 |   1)   7.823 us    |      }
  457.680985 |   1)   0.496 us    |      _local_bh_enable();
  457.680986 |   1)   9.845 us    |    }
  457.680987 |   1) + 10.962 us   |  }

I've Cc:-ed Paul, as you have tree-RCU enabled, which is a new feature 
in v2.6.29:

 #
 # RCU Subsystem
 #
 # CONFIG_CLASSIC_RCU is not set
 CONFIG_TREE_RCU=y
 # CONFIG_PREEMPT_RCU is not set
 # CONFIG_RCU_TRACE is not set
 CONFIG_RCU_FANOUT=32
 # CONFIG_RCU_FANOUT_EXACT is not set
 # CONFIG_TREE_RCU_TRACE is not set
 # CONFIG_PREEMPT_RCU_TRACE is not set

Damien, as an experiment, if you change your config to 
CONFIG_CLASSIC_RCU=y, does the ksoftirqd problem go away?

On the other hand ... the softirq processing there looks anomalous in 
itself, and might be due to some compiler bug perhaps. Could you try the 
debug patch below please (you'll get it automatically if you update to 
latest -tip), and redo the trace - the ftrace_printk() info should now 
be embedded in the trace. The expected result would be for the printed 
ot value to be non-zero at the #1 point, and zero at the #2 point.

Note: if it's a compiler optimization bug then this patch might make the 
whole problem go away.

	Ingo

------------------>
>From 2d7cf65eec92937bff1073311f6843aa7189bff2 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@elte.hu>
Date: Mon, 16 Feb 2009 10:48:37 +0100
Subject: [PATCH] softirq: debug

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/softirq.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 3dd0d13..110cad0 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -196,7 +196,9 @@ asmlinkage void __do_softirq(void)
 	cpu = smp_processor_id();
 restart:
 	/* Reset the pending bitmask before enabling irqs */
+	ftrace_printk("#1 softirq pending: %08x\n", local_softirq_pending());
 	set_softirq_pending(0);
+	ftrace_printk("#2 softirq pending: %08x\n", local_softirq_pending());
 
 	local_irq_enable();
 

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo@elte.hu> [090216 10:50]:
> hm, we need a trace with both abstime and process information included:

>   echo funcgraph-proc     > trace_options
>   echo funcgraph-abstime  > trace_options

> Also, at 140 msecs the duration is a bit short - could you please make a 
> 1-2 seconds capture? You can do that by increasing the number in 
> buffer_size_kb 10-fold:

>    echo 14100 > buffer_size_kb 

Ok, I've redone a trace with these options enabled. The file is here:
http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz

> Hm ... even with this limited trace, there's an unusually high amount of 
> RCU activities. Each activity goes like this:

>   457.680976 |   1)               |  do_softirq() {
>   457.680976 |   1)               |    __do_softirq() {
>   457.680977 |   1)               |      rcu_process_callbacks() {
>   457.680977 |   1)               |        __rcu_process_callbacks() {
> 4  457.680978 |   1)   0.478 us    |          force_quiescent_state();
>   457.680979 |   1)   1.591 us    |        }
>   457.680979 |   1)               |        __rcu_process_callbacks() {
>   457.680980 |   1)   0.478 us    |          force_quiescent_state();
>   457.680981 |   1)               |          cpu_quiet() {
>   457.680981 |   1)   0.506 us    |            _spin_lock_irqsave();
>   457.680982 |   1)   0.496 us    |            _spin_unlock_irqrestore();
>   457.680983 |   1)   2.545 us    |          }
>   457.680984 |   1)   4.626 us    |        }
>   457.680985 |   1)   7.823 us    |      }
>   457.680985 |   1)   0.496 us    |      _local_bh_enable();
>   457.680986 |   1)   9.845 us    |    }
>   457.680987 |   1) + 10.962 us   |  }

> I've Cc:-ed Paul, as you have tree-RCU enabled, which is a new feature 
> in v2.6.29:

>  #
>  # RCU Subsystem
>  #
>  # CONFIG_CLASSIC_RCU is not set
>  CONFIG_TREE_RCU=y
>  # CONFIG_PREEMPT_RCU is not set
>  # CONFIG_RCU_TRACE is not set
>  CONFIG_RCU_FANOUT=32
>  # CONFIG_RCU_FANOUT_EXACT is not set
>  # CONFIG_TREE_RCU_TRACE is not set
>  # CONFIG_PREEMPT_RCU_TRACE is not set

> Damien, as an experiment, if you change your config to 
> CONFIG_CLASSIC_RCU=y, does the ksoftirqd problem go away?

Yes, in that case the problem completely goes away. Nice catch!

> On the other hand ... the softirq processing there looks anomalous in
> itself, and might be due to some compiler bug perhaps. Could you try
> the debug patch below please (you'll get it automatically if you
> update to latest -tip), and redo the trace - the ftrace_printk() info
> should now be embedded in the trace. The expected result would be for
> the printed ot value to be non-zero at the #1 point, and zero at the
> #2 point.

> Note: if it's a compiler optimization bug then this patch might make the 
> whole problem go away.

> 	Ingo

> ------------------>
> >From 2d7cf65eec92937bff1073311f6843aa7189bff2 Mon Sep 17 00:00:00 2001
> From: Ingo Molnar <mingo@elte.hu>
> Date: Mon, 16 Feb 2009 10:48:37 +0100
> Subject: [PATCH] softirq: debug

> Signed-off-by: Ingo Molnar <mingo@elte.hu>
> ---
>  kernel/softirq.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)

> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index 3dd0d13..110cad0 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -196,7 +196,9 @@ asmlinkage void __do_softirq(void)
>  	cpu = smp_processor_id();
>  restart:
>  	/* Reset the pending bitmask before enabling irqs */
> +	ftrace_printk("#1 softirq pending: %08x\n", local_softirq_pending());
>  	set_softirq_pending(0);
> +	ftrace_printk("#2 softirq pending: %08x\n", local_softirq_pending());

>  	local_irq_enable();

I've done my new trace after recompiling very latest tip, so this should
have been taken into account.

So this seems RCU-related... If more tests are needed, do not hesitate
to ask. Btw, on the laptop which also shows ksoftirqd problems, I also
have CONFIG_TREE_RCU=y.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> * Ingo Molnar <mingo@elte.hu> [090216 10:50]:
> > hm, we need a trace with both abstime and process information included:
> 
> >   echo funcgraph-proc     > trace_options
> >   echo funcgraph-abstime  > trace_options
> 
> > Also, at 140 msecs the duration is a bit short - could you please make a 
> > 1-2 seconds capture? You can do that by increasing the number in 
> > buffer_size_kb 10-fold:
> 
> >    echo 14100 > buffer_size_kb 
> 
> Ok, I've redone a trace with these options enabled. The file is here:
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz

ok, here's the new annotated trace:

  799.555279 |   1)  ksoftir-2324  |               |  do_softirq() {
  799.555279 |   1)  ksoftir-2324  |               |    __do_softirq() {
  799.555280 |   1)  ksoftir-2324  |               |      /* #1 softirq pending: 00000100 */
  799.555281 |   1)  ksoftir-2324  |               |      /* #2 softirq pending: 00000000 */
  799.555282 |   1)  ksoftir-2324  |               |      rcu_process_callbacks() {
  799.555282 |   1)  ksoftir-2324  |               |        __rcu_process_callbacks() {
  799.555283 |   1)  ksoftir-2324  |   0.479 us    |          force_quiescent_state();
  799.555284 |   1)  ksoftir-2324  |   1.576 us    |        }
  799.555284 |   1)  ksoftir-2324  |               |        __rcu_process_callbacks() {
  799.555285 |   1)  ksoftir-2324  |               |          force_quiescent_state() {
  799.555286 |   1)  ksoftir-2324  |               |          cpu_quiet() {
  799.555286 |   1)  ksoftir-2324  |   0.518 us    |            _spin_lock_irqsave();
  799.555287 |   1)  ksoftir-2324  |   0.506 us    |            _spin_unlock_irqrestore();
  799.555288 |   1)  ksoftir-2324  |   2.563 us    |          }
  799.555289 |   1)  ksoftir-2324  |   4.624 us    |        }
  799.555289 |   1)  ksoftir-2324  |   7.836 us    |      }
  799.555290 |   1)  ksoftir-2324  |   0.495 us    |      _local_bh_enable();
  799.555291 |   1)  ksoftir-2324  | + 11.550 us   |    }
  799.555291 |   1)  ksoftir-2324  | + 12.713 us   |  }
  799.555292 |   1)  ksoftir-2324  |   0.524 us    |  _cond_resched();

We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
this indeed seems to be a CONFIG_TREE_RCU=y bug.

What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
no raise_softirq() calls. Could you please do a two-CPU trace too via:

   echo 3 > /debug/tracing/tracing_cpumask

So that we can see what's happening on the other CPU?

Also, could you please apply the debug patch below (or update to the 
very latest -tip tree), so that we get trace entries of softirq triggers 
too?

Thanks,

	Ingo

-------------->

>From 6876d5d56716427f9bbe3af7e4e9c06cb760ae0c Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@elte.hu>
Date: Mon, 16 Feb 2009 13:23:36 +0100
Subject: [PATCH] softirq: debug #2

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 include/linux/interrupt.h |    3 ++-
 kernel/softirq.c          |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index e7bcfd7..cc1f529 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -271,7 +271,8 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
-#define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+#define ___raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+extern void __raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 110cad0..431cb4f 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -302,6 +302,13 @@ void irq_exit(void)
 	preempt_enable_no_resched();
 }
 
+void __raise_softirq_irqoff(unsigned int nr)
+{
+	ftrace_printk("nr: %d\n", nr);
+	or_softirq_pending(1UL << nr);
+}
+EXPORT_SYMBOL_GPL(__raise_softirq_irqoff);
+
 /*
  * This function must run with irqs disabled!
  */

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> this indeed seems to be a CONFIG_TREE_RCU=y bug.

> What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> no raise_softirq() calls. Could you please do a two-CPU trace too via:

>    echo 3 > /debug/tracing/tracing_cpumask

> So that we can see what's happening on the other CPU?

> Also, could you please apply the debug patch below (or update to the 
> very latest -tip tree), so that we get trace entries of softirq triggers 
> too?

Ok, the new trace with these additional modifications is here:
http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> 
> > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> 
> >    echo 3 > /debug/tracing/tracing_cpumask
> 
> > So that we can see what's happening on the other CPU?
> 
> > Also, could you please apply the debug patch below (or update to the 
> > very latest -tip tree), so that we get trace entries of softirq triggers 
> > too?
> 
> Ok, the new trace with these additional modifications is here:
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz

thanks.

This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:

  136.255963 |   0)   sleep-2345   |               |          update_process_times() {
  136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
  136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
  136.255966 |   0)   sleep-2345   |   2.262 us    |            }
  136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
  136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
  136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
  136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
  136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
  136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
  136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
  136.255974 |   0)   sleep-2345   |   3.832 us    |                }
  136.255975 |   0)   sleep-2345   |   5.491 us    |              }
  136.255976 |   0)   sleep-2345   |   8.667 us    |            }
  136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
  136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
  136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
  136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
  136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
  136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
  136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
  136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
  136.255984 |   0)   sleep-2345   |   3.059 us    |                }
  136.255985 |   0)   sleep-2345   |   4.594 us    |              }
  136.255986 |   0)   sleep-2345   |   7.800 us    |            }
  136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();

again and again.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> 
> * Damien Wyart <damien.wyart@free.fr> wrote:
> 
> > * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > 
> > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > 
> > >    echo 3 > /debug/tracing/tracing_cpumask
> > 
> > > So that we can see what's happening on the other CPU?
> > 
> > > Also, could you please apply the debug patch below (or update to the 
> > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > too?
> > 
> > Ok, the new trace with these additional modifications is here:
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> 
> thanks.
> 
> This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> 
>   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
>   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
>   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
>   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
>   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
>   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
>   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
>   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
>   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
>   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
>   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
>   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
>   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
>   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
>   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
>   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
>   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
>   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
>   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
>   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
>   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
>   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
>   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
>   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
>   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
>   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> 
> again and again.

Interesting...

I will take a look!

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > 
> > * Damien Wyart <damien.wyart@free.fr> wrote:
> > 
> > > * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > 
> > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > 
> > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > 
> > > > So that we can see what's happening on the other CPU?
> > > 
> > > > Also, could you please apply the debug patch below (or update to the 
> > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > too?
> > > 
> > > Ok, the new trace with these additional modifications is here:
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > 
> > thanks.
> > 
> > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > 
> >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > 
> > again and again.
> 
> Interesting...
> 
> I will take a look!

The above sequence is more or less normal behavior -- the RCU softirq
handler rcu_process_callbacks() is being invoked once per tick, which
appears to be HZ=1000 or thereabouts.  The system appears to be mostly
idle during this time period.

One oddity is that the _bh call to __rcu_process_callbacks() is invoking
force_quiescent_state() each time, and force_quiescent_state() isn't
doing anything.  This is a possible mismatch between the conditions in
rcu_pending() and force_quiescent_state(), and I will look into this.

However, this sequence is consuming less than 10 microseconds per
millisecond, so cannot be the main cause of the softirq issues you
are seeing, though if there really is a mismatch, it needs to be fixed,
and I will attend to this.

The interesting portion of the trace is later on:

  137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
  137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
  137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
  137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
  137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
  137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
  137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
  137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
  137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
  137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
  137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
  137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
  137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
  137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
  137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
  137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
  137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
  137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
  137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
  137.897007 |   1)  ksoftir-2302  |               |  schedule() {
  137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
  137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
  137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
  137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
  137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
  137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
  137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
  137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
  137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
  137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
  137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
  137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
  137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
  137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
  137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
  137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
  137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
  137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
  137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
  137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
  137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
  137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
  137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
  137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
  137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
  137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
  137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
  137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
  137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
  137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
  137.897045 |   ------------------------------------------
 1)  ksoftir-2302  =>    <idle>-0   
 ------------------------------------------

 1)    <idle>-0    |               |  /* nr: 8 */
 ------------------------------------------
 1)    <idle>-0    =>  ksoftir-2302 
 ------------------------------------------

  137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
  137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
  137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
  137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
  137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
  137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
  137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
  137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
  137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
  137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
  137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
  137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
  137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
  137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
  137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
  137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
  137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
  137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
  137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
  137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
  137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
  137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
  137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
  137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }

Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
so that this function is consuming more than 10% of a CPU.  The strange
thing is that I don't see a raise_softirq() in between, though perhaps
it gets inlined or something that makes it invisible to ftrace.
Certainly rcu_process_callbacks() can re-invoke itself, for example,
when a large number of RCU callbacks has piled up.  However, there are
only 29 calls to __call_rcu() over the entire time period, so that does
not appear to be the cause.  Strangely enough, there appear to be no
calls to rcu_do_batch() over the full trace, but this is invoked
unconditionally from __rcu_process_callbacks().  So perhaps the trace
wasn't covering that function?

Whatever, this pattern continues for more than 300 milliseconds(!).

Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
reproduce this and send the output of the debugfs files rcu/rcudata
and rcu/rcuhier?  The commands for this would be:

	mkdir /debug || :
	mount -t debugfs debugfs /debug
	cat /debug/rcu/rcuhier
	cat /debug/rcu/rcudata

I will try to reproduce locally as well.

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > 
> > > * Damien Wyart <damien.wyart@free.fr> wrote:
> > > 
> > > > * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > 
> > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > 
> > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > 
> > > > > So that we can see what's happening on the other CPU?
> > > > 
> > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > too?
> > > > 
> > > > Ok, the new trace with these additional modifications is here:
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > 
> > > thanks.
> > > 
> > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > 
> > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > 
> > > again and again.
> > 
> > Interesting...
> > 
> > I will take a look!
> 
> The above sequence is more or less normal behavior -- the RCU softirq
> handler rcu_process_callbacks() is being invoked once per tick, which
> appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> idle during this time period.
> 
> One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> force_quiescent_state() each time, and force_quiescent_state() isn't
> doing anything.  This is a possible mismatch between the conditions in
> rcu_pending() and force_quiescent_state(), and I will look into this.
> 
> However, this sequence is consuming less than 10 microseconds per
> millisecond, so cannot be the main cause of the softirq issues you
> are seeing, though if there really is a mismatch, it needs to be fixed,
> and I will attend to this.
> 
> The interesting portion of the trace is later on:
> 
>   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
>   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
>   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
>   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
>   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
>   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
>   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
>   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
>   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
>   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
>   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
>   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
>   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
>   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
>   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
>   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
>   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
>   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
>   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
>   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
>   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
>   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
>   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
>   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
>   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
>   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
>   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
>   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
>   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
>   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
>   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
>   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
>   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
>   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
>   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
>   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
>   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
>   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
>   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
>   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
>   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
>   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
>   137.897045 |   ------------------------------------------
>  1)  ksoftir-2302  =>    <idle>-0   
>  ------------------------------------------
> 
>  1)    <idle>-0    |               |  /* nr: 8 */
>  ------------------------------------------
>  1)    <idle>-0    =>  ksoftir-2302 
>  ------------------------------------------
> 
>   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
>   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
>   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
>   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
>   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
>   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
>   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
>   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
>   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
>   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
>   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
>   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
>   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
>   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
>   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
>   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> 
> Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> so that this function is consuming more than 10% of a CPU.  The strange
> thing is that I don't see a raise_softirq() in between, though perhaps
> it gets inlined or something that makes it invisible to ftrace.


More likely it's a bug in the function graph tracer.
I will try to fix it.


> Certainly rcu_process_callbacks() can re-invoke itself, for example,
> when a large number of RCU callbacks has piled up.  However, there are
> only 29 calls to __call_rcu() over the entire time period, so that does
> not appear to be the cause.  Strangely enough, there appear to be no
> calls to rcu_do_batch() over the full trace, but this is invoked
> unconditionally from __rcu_process_callbacks().  So perhaps the trace
> wasn't covering that function?


Hmm. I have to check that too. I'm building CONFIG_RCU_TREE.
Will try to find what happens.


> 
> Whatever, this pattern continues for more than 300 milliseconds(!).
> 
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier?  The commands for this would be:
> 
> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata
> 
> I will try to reproduce locally as well.
> 
> 							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > 
> > > * Damien Wyart <damien.wyart@free.fr> wrote:
> > > 
> > > > * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > 
> > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > 
> > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > 
> > > > > So that we can see what's happening on the other CPU?
> > > > 
> > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > too?
> > > > 
> > > > Ok, the new trace with these additional modifications is here:
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > 
> > > thanks.
> > > 
> > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > 
> > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > 
> > > again and again.
> > 
> > Interesting...
> > 
> > I will take a look!
> 
> The above sequence is more or less normal behavior -- the RCU softirq
> handler rcu_process_callbacks() is being invoked once per tick, which
> appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> idle during this time period.
> 
> One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> force_quiescent_state() each time, and force_quiescent_state() isn't
> doing anything.  This is a possible mismatch between the conditions in
> rcu_pending() and force_quiescent_state(), and I will look into this.
> 
> However, this sequence is consuming less than 10 microseconds per
> millisecond, so cannot be the main cause of the softirq issues you
> are seeing, though if there really is a mismatch, it needs to be fixed,
> and I will attend to this.
> 
> The interesting portion of the trace is later on:
> 
>   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
>   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
>   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
>   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
>   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
>   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
>   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
>   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
>   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
>   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
>   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
>   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
>   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
>   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
>   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
>   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
>   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
>   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
>   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
>   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
>   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
>   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
>   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
>   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
>   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
>   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
>   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
>   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
>   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
>   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
>   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
>   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
>   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
>   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
>   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
>   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
>   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
>   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
>   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
>   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
>   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
>   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
>   137.897045 |   ------------------------------------------
>  1)  ksoftir-2302  =>    <idle>-0   
>  ------------------------------------------
> 
>  1)    <idle>-0    |               |  /* nr: 8 */
>  ------------------------------------------
>  1)    <idle>-0    =>  ksoftir-2302 
>  ------------------------------------------
> 
>   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
>   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
>   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
>   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
>   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
>   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
>   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
>   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
>   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
>   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
>   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
>   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
>   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
>   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
>   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
>   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> 
> Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> so that this function is consuming more than 10% of a CPU.  The strange
> thing is that I don't see a raise_softirq() in between, though perhaps
> it gets inlined or something that makes it invisible to ftrace.
> Certainly rcu_process_callbacks() can re-invoke itself, for example,
> when a large number of RCU callbacks has piled up.  However, there are
> only 29 calls to __call_rcu() over the entire time period, so that does
> not appear to be the cause.  Strangely enough, there appear to be no
> calls to rcu_do_batch() over the full trace, but this is invoked
> unconditionally from __rcu_process_callbacks().  So perhaps the trace
> wasn't covering that function?


I just checked an assembly dump of my vmlinux, and rcu_do_batch()
has been inlined. I don't understand why, this is a wide function.

 
> Whatever, this pattern continues for more than 300 milliseconds(!).
> 
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier?  The commands for this would be:
> 
> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata
> 
> I will try to reproduce locally as well.
> 
> 							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 09:02:18PM +0100, Frederic Weisbecker wrote:
> On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Damien Wyart <damien.wyart@free.fr> wrote:
> > > > 
> > > > > * Ingo Molnar <mingo@elte.hu> [090216 13:26]:
> > > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > > 
> > > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > > 
> > > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > > 
> > > > > > So that we can see what's happening on the other CPU?
> > > > > 
> > > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > > too?
> > > > > 
> > > > > Ok, the new trace with these additional modifications is here:
> > > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > > 
> > > > thanks.
> > > > 
> > > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > > 
> > > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > > 
> > > > again and again.
> > > 
> > > Interesting...
> > > 
> > > I will take a look!
> > 
> > The above sequence is more or less normal behavior -- the RCU softirq
> > handler rcu_process_callbacks() is being invoked once per tick, which
> > appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> > idle during this time period.
> > 
> > One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> > force_quiescent_state() each time, and force_quiescent_state() isn't
> > doing anything.  This is a possible mismatch between the conditions in
> > rcu_pending() and force_quiescent_state(), and I will look into this.
> > 
> > However, this sequence is consuming less than 10 microseconds per
> > millisecond, so cannot be the main cause of the softirq issues you
> > are seeing, though if there really is a mismatch, it needs to be fixed,
> > and I will attend to this.
> > 
> > The interesting portion of the trace is later on:
> > 
> >   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
> >   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
> >   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
> >   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
> >   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
> >   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
> >   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
> >   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
> >   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
> >   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
> >   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
> >   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
> >   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
> >   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
> >   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
> >   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
> >   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
> >   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
> >   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
> >   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
> >   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
> >   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
> >   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
> >   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
> >   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
> >   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
> >   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
> >   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
> >   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
> >   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
> >   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
> >   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
> >   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
> >   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
> >   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
> >   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
> >   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
> >   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
> >   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
> >   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
> >   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
> >   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
> >   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
> >   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
> >   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
> >   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
> >   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
> >   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
> >   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
> >   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
> >   137.897045 |   ------------------------------------------
> >  1)  ksoftir-2302  =>    <idle>-0   
> >  ------------------------------------------
> > 
> >  1)    <idle>-0    |               |  /* nr: 8 */
> >  ------------------------------------------
> >  1)    <idle>-0    =>  ksoftir-2302 
> >  ------------------------------------------
> > 
> >   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
> >   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
> >   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
> >   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
> >   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
> >   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
> >   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
> >   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
> >   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
> >   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
> >   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
> >   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
> >   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
> >   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
> >   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
> >   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
> >   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
> >   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
> >   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
> >   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
> >   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
> >   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
> >   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
> >   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> > 
> > Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> > so that this function is consuming more than 10% of a CPU.  The strange
> > thing is that I don't see a raise_softirq() in between, though perhaps
> > it gets inlined or something that makes it invisible to ftrace.
> > Certainly rcu_process_callbacks() can re-invoke itself, for example,
> > when a large number of RCU callbacks has piled up.  However, there are
> > only 29 calls to __call_rcu() over the entire time period, so that does
> > not appear to be the cause.  Strangely enough, there appear to be no
> > calls to rcu_do_batch() over the full trace, but this is invoked
> > unconditionally from __rcu_process_callbacks().  So perhaps the trace
> > wasn't covering that function?
> 
> I just checked an assembly dump of my vmlinux, and rcu_do_batch()
> has been inlined. I don't understand why, this is a wide function.

Yow!!!  ;-)

> > Whatever, this pattern continues for more than 300 milliseconds(!).
> > 
> > Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> > reproduce this and send the output of the debugfs files rcu/rcudata
> > and rcu/rcuhier?  The commands for this would be:
> > 
> > 	mkdir /debug || :
> > 	mount -t debugfs debugfs /debug
> > 	cat /debug/rcu/rcuhier
> > 	cat /debug/rcu/rcudata
> > 
> > I will try to reproduce locally as well.

No luck thus far, perhaps because I first tried x86_64 and Damien's run
was on 32-bit x86.

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:

> Here the calls to rcu_process_callbacks() are only 75 
> microseconds apart, so that this function is consuming more 
> than 10% of a CPU.  The strange thing is that I don't see a 
> raise_softirq() in between, though perhaps it gets inlined or 
> something that makes it invisible to ftrace.

look at the latest trace please, that has even the most inline 
raise-softirq method instrumented, so all the raising is 
visible.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> 
> > Here the calls to rcu_process_callbacks() are only 75 
> > microseconds apart, so that this function is consuming more 
> > than 10% of a CPU.  The strange thing is that I don't see a 
> > raise_softirq() in between, though perhaps it gets inlined or 
> > something that makes it invisible to ftrace.
> 
> look at the latest trace please, that has even the most inline 
> raise-softirq method instrumented, so all the raising is 
> visible.

Ah, my apologies!  This time looking at:

http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz


  799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
  799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
  799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
  799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
[ . . . ]

Yikes!!!

Why is rcu_check_callbacks() being invoked so often?  It should be called
but once per jiffy, and here it is called no less than 22 times in about
3.5 milliseconds, meaning one call every 160 microseconds or so.

Hmmm...

Looks like we never return from:

  799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {

Perhaps we are taking an interrupt immediately after the
local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
And then deciding to go back into nohz mode at 799.521326, 117
microseconds later, after which we re-invoke rcu_check_callbacks(),
which again raises RCU's softirq.

And the reason we are invoking rcu_check_callbacks() so often appears
to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
which explains my failure to reproduce on a 64-bit system:

	void cpu_idle(void)
	{
		int cpu = smp_processor_id();

		current_thread_info()->status |= TS_POLLING;

		/* endless idle loop with no priority at all */
		while (1) {
			tick_nohz_stop_sched_tick(1);
			while (!need_resched()) {

				check_pgt_cache();
				rmb();

				if (rcu_pending(cpu))
					rcu_check_callbacks(cpu, 0);

				if (cpu_is_offline(cpu))
					play_dead();

				local_irq_disable();
				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
				/* Don't trace irqs off for idle */
				stop_critical_timings();
				pm_idle();
				start_critical_timings();
			}
			tick_nohz_restart_sched_tick();
			preempt_enable_no_resched();
			schedule();
			preempt_disable();
		}
	}

If we go in and out of nohz mode quickly, we will invoke rcu_pending()
each time.  I would expect rcu_pending() to return 0 most of the time,
but that apparently isn't the case with treercu...

What is the easiest way for me to make it easy to trace the return path
from __rcu_pending()?  Make each return path call an empty function
located off where the compiler cannot see it, I guess...  Diagnostic
patch along these lines below.  Frederic, Damien, could you please give
it a go?  (And of course please let me know if something else is
needed.)

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---

 rcupdate.c |   23 +++++++++++++++++++++++
 rcutree.c  |   31 +++++++++++++++++++++++++------
 2 files changed, 48 insertions(+), 6 deletions(-)

diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
index d92a76a..42bbf03 100644
--- a/kernel/rcupdate.c
+++ b/kernel/rcupdate.c
@@ -175,3 +175,26 @@ void __init rcu_init(void)
 	__rcu_init();
 }
 
+void __rcu_pending_qs_pending(void)
+{
+}
+
+void __rcu_pending_callbacks_ready(void)
+{
+}
+
+void __rcu_pending_needs_gp(void)
+{
+}
+
+void __rcu_pending_new_completed(void)
+{
+}
+
+void __rcu_pending_new_gp(void)
+{
+}
+
+void __rcu_pending_fqs(void)
+{
+}
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..e2d72c3 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
 }
 EXPORT_SYMBOL_GPL(call_rcu_bh);
 
+extern void __rcu_pending_qs_pending(void);
+extern void __rcu_pending_callbacks_ready(void);
+extern void __rcu_pending_needs_gp(void);
+extern void __rcu_pending_new_completed(void);
+extern void __rcu_pending_new_gp(void);
+extern void __rcu_pending_fqs(void);
+
 /*
  * Check to see if there is any immediate RCU-related work to be done
  * by the current CPU, for the specified type of RCU, returning 1 if so.
@@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 	check_cpu_stall(rsp, rdp);
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
-	if (rdp->qs_pending)
+	if (rdp->qs_pending) {
+		__rcu_pending_qs_pending();
 		return 1;
+	}
 
 	/* Does this CPU have callbacks ready to invoke? */
-	if (cpu_has_callbacks_ready_to_invoke(rdp))
+	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
+		__rcu_pending_callbacks_ready();
 		return 1;
+	}
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
-	if (cpu_needs_another_gp(rsp, rdp))
+	if (cpu_needs_another_gp(rsp, rdp)) {
+		__rcu_pending_needs_gp();
 		return 1;
+	}
 
 	/* Has another RCU grace period completed?  */
-	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
+	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
+		__rcu_pending_new_completed();
 		return 1;
+	}
 
 	/* Has a new RCU grace period started? */
-	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
+	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
+		__rcu_pending_new_gp();
 		return 1;
+	}
 
 	/* Has an RCU GP gone long enough to send resched IPIs &c? */
 	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
 	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
-	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
+	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
+		__rcu_pending_fqs();
 		return 1;
+	}
 
 	/* nothing to do */
 	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > 
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > 
> > > Here the calls to rcu_process_callbacks() are only 75 
> > > microseconds apart, so that this function is consuming more 
> > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > raise_softirq() in between, though perhaps it gets inlined or 
> > > something that makes it invisible to ftrace.
> > 
> > look at the latest trace please, that has even the most inline 
> > raise-softirq method instrumented, so all the raising is 
> > visible.
> 
> Ah, my apologies!  This time looking at:
> 
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> 
> 
>   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
>   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> [ . . . ]
> 
> Yikes!!!
> 
> Why is rcu_check_callbacks() being invoked so often?  It should be called
> but once per jiffy, and here it is called no less than 22 times in about
> 3.5 milliseconds, meaning one call every 160 microseconds or so.

BTW, the other question I have is "why do we need to call rcu_pending()
and rcu_check_callbacks() from the idle loop of 32-bit x86, especially
given that no other architecture does this?".  Don't get me wrong, it
would be good to get rcutree's rcu_pending() to avoid spuriously saying
that rcu_check_callbacks() should be invoked, so I would still like the
trace with my patch, but...

							Thanx, Paul

> Hmmm...
> 
> Looks like we never return from:
> 
>   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> 
> Perhaps we are taking an interrupt immediately after the
> local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> And then deciding to go back into nohz mode at 799.521326, 117
> microseconds later, after which we re-invoke rcu_check_callbacks(),
> which again raises RCU's softirq.
> 
> And the reason we are invoking rcu_check_callbacks() so often appears
> to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> which explains my failure to reproduce on a 64-bit system:
> 
> 	void cpu_idle(void)
> 	{
> 		int cpu = smp_processor_id();
> 
> 		current_thread_info()->status |= TS_POLLING;
> 
> 		/* endless idle loop with no priority at all */
> 		while (1) {
> 			tick_nohz_stop_sched_tick(1);
> 			while (!need_resched()) {
> 
> 				check_pgt_cache();
> 				rmb();
> 
> 				if (rcu_pending(cpu))
> 					rcu_check_callbacks(cpu, 0);
> 
> 				if (cpu_is_offline(cpu))
> 					play_dead();
> 
> 				local_irq_disable();
> 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> 				/* Don't trace irqs off for idle */
> 				stop_critical_timings();
> 				pm_idle();
> 				start_critical_timings();
> 			}
> 			tick_nohz_restart_sched_tick();
> 			preempt_enable_no_resched();
> 			schedule();
> 			preempt_disable();
> 		}
> 	}
> 
> If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> each time.  I would expect rcu_pending() to return 0 most of the time,
> but that apparently isn't the case with treercu...
> 
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> 
>  rcupdate.c |   23 +++++++++++++++++++++++
>  rcutree.c  |   31 +++++++++++++++++++++++++------
>  2 files changed, 48 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
> index d92a76a..42bbf03 100644
> --- a/kernel/rcupdate.c
> +++ b/kernel/rcupdate.c
> @@ -175,3 +175,26 @@ void __init rcu_init(void)
>  	__rcu_init();
>  }
>  
> +void __rcu_pending_qs_pending(void)
> +{
> +}
> +
> +void __rcu_pending_callbacks_ready(void)
> +{
> +}
> +
> +void __rcu_pending_needs_gp(void)
> +{
> +}
> +
> +void __rcu_pending_new_completed(void)
> +{
> +}
> +
> +void __rcu_pending_new_gp(void)
> +{
> +}
> +
> +void __rcu_pending_fqs(void)
> +{
> +}
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..e2d72c3 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
>  }
>  EXPORT_SYMBOL_GPL(call_rcu_bh);
>  
> +extern void __rcu_pending_qs_pending(void);
> +extern void __rcu_pending_callbacks_ready(void);
> +extern void __rcu_pending_needs_gp(void);
> +extern void __rcu_pending_new_completed(void);
> +extern void __rcu_pending_new_gp(void);
> +extern void __rcu_pending_fqs(void);
> +
>  /*
>   * Check to see if there is any immediate RCU-related work to be done
>   * by the current CPU, for the specified type of RCU, returning 1 if so.
> @@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
>  	check_cpu_stall(rsp, rdp);
>  
>  	/* Is the RCU core waiting for a quiescent state from this CPU? */
> -	if (rdp->qs_pending)
> +	if (rdp->qs_pending) {
> +		__rcu_pending_qs_pending();
>  		return 1;
> +	}
>  
>  	/* Does this CPU have callbacks ready to invoke? */
> -	if (cpu_has_callbacks_ready_to_invoke(rdp))
> +	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
> +		__rcu_pending_callbacks_ready();
>  		return 1;
> +	}
>  
>  	/* Has RCU gone idle with this CPU needing another grace period? */
> -	if (cpu_needs_another_gp(rsp, rdp))
> +	if (cpu_needs_another_gp(rsp, rdp)) {
> +		__rcu_pending_needs_gp();
>  		return 1;
> +	}
>  
>  	/* Has another RCU grace period completed?  */
> -	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
> +		__rcu_pending_new_completed();
>  		return 1;
> +	}
>  
>  	/* Has a new RCU grace period started? */
> -	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
> +		__rcu_pending_new_gp();
>  		return 1;
> +	}
>  
>  	/* Has an RCU GP gone long enough to send resched IPIs &c? */
>  	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
>  	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
> -	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
> +	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
> +		__rcu_pending_fqs();
>  		return 1;
> +	}
>  
>  	/* nothing to do */
>  	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:

> On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > 
> > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > microseconds apart, so that this function is consuming more 
> > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > something that makes it invisible to ftrace.
> > > 
> > > look at the latest trace please, that has even the most inline 
> > > raise-softirq method instrumented, so all the raising is 
> > > visible.
> > 
> > Ah, my apologies!  This time looking at:
> > 
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > 
> > 
> >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > [ . . . ]
> > 
> > Yikes!!!
> > 
> > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > but once per jiffy, and here it is called no less than 22 times in about
> > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> 
> BTW, the other question I have is "why do we need to call 
> rcu_pending() and rcu_check_callbacks() from the idle loop of 
> 32-bit x86, especially given that no other architecture does 
> this?".  Don't get me wrong, it would be good to get rcutree's 
> rcu_pending() to avoid spuriously saying that 
> rcu_check_callbacks() should be invoked, so I would still like 
> the trace with my patch, but...

There's no strong reason - we've been back and forth about RCU 
in the dynticks code. Mind sending a test patch for Damien to 
try?

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 10:46:57AM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > 
> > BTW, the other question I have is "why do we need to call 
> > rcu_pending() and rcu_check_callbacks() from the idle loop of 
> > 32-bit x86, especially given that no other architecture does 
> > this?".  Don't get me wrong, it would be good to get rcutree's 
> > rcu_pending() to avoid spuriously saying that 
> > rcu_check_callbacks() should be invoked, so I would still like 
> > the trace with my patch, but...
> 
> There's no strong reason - we've been back and forth about RCU 
> in the dynticks code. Mind sending a test patch for Damien to 
> try?

But of course!  ;-)

The following patch removes the call to rcu_pending() and
rcu_check_callbacks() from the x86 32-bit idle loop in order to
reduce the softirq load on idle systems.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---

 process_32.c |    3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index a546f55..bd4da2a 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -104,9 +104,6 @@ void cpu_idle(void)
 			check_pgt_cache();
 			rmb();
 
-			if (rcu_pending(cpu))
-				rcu_check_callbacks(cpu, 0);
-
 			if (cpu_is_offline(cpu))
 				play_dead();
 

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello Paul,

> > There's no strong reason - we've been back and forth about RCU in
> > the dynticks code. Mind sending a test patch for Damien to try?

> But of course!  ;-)

With this patch, the problem goes away and system activity seems normal,
both on the P4 with high load and on the recent laptop.

Btw, could you explain briefly why, without this patch, a kernel
enabling classical RCU doesn't show the ksoftirqd problem at all?


Damien

> The following patch removes the call to rcu_pending() and
> rcu_check_callbacks() from the x86 32-bit idle loop in order to
> reduce the softirq load on idle systems.

> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---

>  process_32.c |    3 ---
>  1 file changed, 3 deletions(-)

> diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
> index a546f55..bd4da2a 100644
> --- a/arch/x86/kernel/process_32.c
> +++ b/arch/x86/kernel/process_32.c
> @@ -104,9 +104,6 @@ void cpu_idle(void)
>  			check_pgt_cache();
>  			rmb();

> -			if (rcu_pending(cpu))
> -				rcu_check_callbacks(cpu, 0);
> -
>  			if (cpu_is_offline(cpu))
>  				play_dead();

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 04:39:25PM +0100, Damien Wyart wrote:
> Hello Paul,
> 
> > > There's no strong reason - we've been back and forth about RCU in
> > > the dynticks code. Mind sending a test patch for Damien to try?
> 
> > But of course!  ;-)
> 
> With this patch, the problem goes away and system activity seems normal,
> both on the P4 with high load and on the recent laptop.
> 
> Btw, could you explain briefly why, without this patch, a kernel
> enabling classical RCU doesn't show the ksoftirqd problem at all?

Classic RCU's rcu_pending() can afford to be much more conservative,
due to the easy availability of global information.  In contrast,
Hierarchical RCU will invoke the softirq in cases where the information
required to make an exact decision is off somewhere else in the tree.

There is also a possibility that Hierarchical RCU is for some reason
failing to detect that its rcu_check_callbacks() is being invoked from
the idle loop -- which would need to be fixed if it is really happening.
But given that the two rcu_check_callbacks() implementations are nearly
identical, I cannot see how this is happening.  (Wouldn't be the first
time that I failed to see how something was happening, though!)

							Thanx, Paul

> Damien
> 
> > The following patch removes the call to rcu_pending() and
> > rcu_check_callbacks() from the x86 32-bit idle loop in order to
> > reduce the softirq load on idle systems.
> 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> 
> >  process_32.c |    3 ---
> >  1 file changed, 3 deletions(-)
> 
> > diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
> > index a546f55..bd4da2a 100644
> > --- a/arch/x86/kernel/process_32.c
> > +++ b/arch/x86/kernel/process_32.c
> > @@ -104,9 +104,6 @@ void cpu_idle(void)
> >  			check_pgt_cache();
> >  			rmb();
> 
> > -			if (rcu_pending(cpu))
> > -				rcu_check_callbacks(cpu, 0);
> > -
> >  			if (cpu_is_offline(cpu))
> >  				play_dead();
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart@free.fr> wrote:

> Hello Paul,
> 
> > > There's no strong reason - we've been back and forth about RCU in
> > > the dynticks code. Mind sending a test patch for Damien to try?
> 
> > But of course!  ;-)
> 
> With this patch, the problem goes away and system activity 
> seems normal, both on the P4 with high load and on the recent 
> laptop.

Applied to tip:x86/urgent, thanks a lot for the patient testing 
and tracing!

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > 
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > 
> > > Here the calls to rcu_process_callbacks() are only 75 
> > > microseconds apart, so that this function is consuming more 
> > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > raise_softirq() in between, though perhaps it gets inlined or 
> > > something that makes it invisible to ftrace.
> > 
> > look at the latest trace please, that has even the most inline 
> > raise-softirq method instrumented, so all the raising is 
> > visible.
> 
> Ah, my apologies!  This time looking at:
> 
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> 
> 
>   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
>   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> [ . . . ]
> 
> Yikes!!!
> 
> Why is rcu_check_callbacks() being invoked so often?  It should be called
> but once per jiffy, and here it is called no less than 22 times in about
> 3.5 milliseconds, meaning one call every 160 microseconds or so.
> 
> Hmmm...
> 
> Looks like we never return from:
> 
>   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> 
> Perhaps we are taking an interrupt immediately after the
> local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> And then deciding to go back into nohz mode at 799.521326, 117
> microseconds later, after which we re-invoke rcu_check_callbacks(),
> which again raises RCU's softirq.
> 
> And the reason we are invoking rcu_check_callbacks() so often appears
> to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> which explains my failure to reproduce on a 64-bit system:
> 
> 	void cpu_idle(void)
> 	{
> 		int cpu = smp_processor_id();
> 
> 		current_thread_info()->status |= TS_POLLING;
> 
> 		/* endless idle loop with no priority at all */
> 		while (1) {
> 			tick_nohz_stop_sched_tick(1);
> 			while (!need_resched()) {
> 
> 				check_pgt_cache();
> 				rmb();
> 
> 				if (rcu_pending(cpu))
> 					rcu_check_callbacks(cpu, 0);
> 
> 				if (cpu_is_offline(cpu))
> 					play_dead();
> 
> 				local_irq_disable();
> 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> 				/* Don't trace irqs off for idle */
> 				stop_critical_timings();
> 				pm_idle();
> 				start_critical_timings();
> 			}
> 			tick_nohz_restart_sched_tick();
> 			preempt_enable_no_resched();
> 			schedule();
> 			preempt_disable();
> 		}
> 	}
> 
> If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> each time.  I would expect rcu_pending() to return 0 most of the time,
> but that apparently isn't the case with treercu...
> 
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)


No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
inside the functions, ie:

__rcu_pending() {
	 /* pending_qs */
}

I've converted your below patch with ftrace_printks and tested it under an old P2
with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
lucky :-) 
I guess I successfully reproduced the softirq/rcu overhead.
Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
When I will have more time, I will fix that.

The trace is here http://dl.free.fr/uyWGgCbx4

It looks like it mostly returns 1 because of the waiting for quiescent state:

$ cat rcutrace | grep "/* pending_none" | wc -l
221
$ cat rcutrace | grep "/* pending_qs" | wc -l
248
$ cat rcutrace | grep "/* pending" | wc -l
469


diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..c9e78f6 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -45,6 +45,7 @@
 #include <linux/cpu.h>
 #include <linux/mutex.h>
 #include <linux/time.h>
+#include <linux/ftrace.h>
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 static struct lock_class_key rcu_lock_key;
@@ -1249,31 +1250,44 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 	check_cpu_stall(rsp, rdp);
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
-	if (rdp->qs_pending)
+	if (rdp->qs_pending) {
+		ftrace_printk("pending_qs\n");
 		return 1;
+	}
 
 	/* Does this CPU have callbacks ready to invoke? */
-	if (cpu_has_callbacks_ready_to_invoke(rdp))
+	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
+		ftrace_printk("pending_ready_invoke\n");
 		return 1;
+	}
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
-	if (cpu_needs_another_gp(rsp, rdp))
+	if (cpu_needs_another_gp(rsp, rdp)) {
+		ftrace_printk("pending_gp\n");
 		return 1;
+	}
 
 	/* Has another RCU grace period completed?  */
-	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
+	if (ACCESS_ONCE(rsp->completed) != rdp->completed) {/* outside of lock */
+		ftrace_printk("pending_gp_completed\n");
 		return 1;
+	}
 
 	/* Has a new RCU grace period started? */
-	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
+	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) { /* outside of lock */
+		ftrace_printk("pending_gp_new_started\n");
 		return 1;
+	}
 
 	/* Has an RCU GP gone long enough to send resched IPIs &c? */
 	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
 	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
-	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
+	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
+		ftrace_printk("pending_ipi\n");
 		return 1;
+	}
 
+	ftrace_printk("pending_none\n");
 	/* nothing to do */
 	return 0;
 }

 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> 
>  rcupdate.c |   23 +++++++++++++++++++++++
>  rcutree.c  |   31 +++++++++++++++++++++++++------
>  2 files changed, 48 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
> index d92a76a..42bbf03 100644
> --- a/kernel/rcupdate.c
> +++ b/kernel/rcupdate.c
> @@ -175,3 +175,26 @@ void __init rcu_init(void)
>  	__rcu_init();
>  }
>  
> +void __rcu_pending_qs_pending(void)
> +{
> +}
> +
> +void __rcu_pending_callbacks_ready(void)
> +{
> +}
> +
> +void __rcu_pending_needs_gp(void)
> +{
> +}
> +
> +void __rcu_pending_new_completed(void)
> +{
> +}
> +
> +void __rcu_pending_new_gp(void)
> +{
> +}
> +
> +void __rcu_pending_fqs(void)
> +{
> +}
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..e2d72c3 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
>  }
>  EXPORT_SYMBOL_GPL(call_rcu_bh);
>  
> +extern void __rcu_pending_qs_pending(void);
> +extern void __rcu_pending_callbacks_ready(void);
> +extern void __rcu_pending_needs_gp(void);
> +extern void __rcu_pending_new_completed(void);
> +extern void __rcu_pending_new_gp(void);
> +extern void __rcu_pending_fqs(void);
> +
>  /*
>   * Check to see if there is any immediate RCU-related work to be done
>   * by the current CPU, for the specified type of RCU, returning 1 if so.
> @@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
>  	check_cpu_stall(rsp, rdp);
>  
>  	/* Is the RCU core waiting for a quiescent state from this CPU? */
> -	if (rdp->qs_pending)
> +	if (rdp->qs_pending) {
> +		__rcu_pending_qs_pending();
>  		return 1;
> +	}
>  
>  	/* Does this CPU have callbacks ready to invoke? */
> -	if (cpu_has_callbacks_ready_to_invoke(rdp))
> +	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
> +		__rcu_pending_callbacks_ready();
>  		return 1;
> +	}
>  
>  	/* Has RCU gone idle with this CPU needing another grace period? */
> -	if (cpu_needs_another_gp(rsp, rdp))
> +	if (cpu_needs_another_gp(rsp, rdp)) {
> +		__rcu_pending_needs_gp();
>  		return 1;
> +	}
>  
>  	/* Has another RCU grace period completed?  */
> -	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
> +		__rcu_pending_new_completed();
>  		return 1;
> +	}
>  
>  	/* Has a new RCU grace period started? */
> -	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
> +		__rcu_pending_new_gp();
>  		return 1;
> +	}
>  
>  	/* Has an RCU GP gone long enough to send resched IPIs &c? */
>  	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
>  	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
> -	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
> +	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
> +		__rcu_pending_fqs();
>  		return 1;
> +	}
>  
>  	/* nothing to do */
>  	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > 
> > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > microseconds apart, so that this function is consuming more 
> > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > something that makes it invisible to ftrace.
> > > 
> > > look at the latest trace please, that has even the most inline 
> > > raise-softirq method instrumented, so all the raising is 
> > > visible.
> > 
> > Ah, my apologies!  This time looking at:
> > 
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > 
> > 
> >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > [ . . . ]
> > 
> > Yikes!!!
> > 
> > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > but once per jiffy, and here it is called no less than 22 times in about
> > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > 
> > Hmmm...
> > 
> > Looks like we never return from:
> > 
> >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > 
> > Perhaps we are taking an interrupt immediately after the
> > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > And then deciding to go back into nohz mode at 799.521326, 117
> > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > which again raises RCU's softirq.
> > 
> > And the reason we are invoking rcu_check_callbacks() so often appears
> > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > which explains my failure to reproduce on a 64-bit system:
> > 
> > 	void cpu_idle(void)
> > 	{
> > 		int cpu = smp_processor_id();
> > 
> > 		current_thread_info()->status |= TS_POLLING;
> > 
> > 		/* endless idle loop with no priority at all */
> > 		while (1) {
> > 			tick_nohz_stop_sched_tick(1);
> > 			while (!need_resched()) {
> > 
> > 				check_pgt_cache();
> > 				rmb();
> > 
> > 				if (rcu_pending(cpu))
> > 					rcu_check_callbacks(cpu, 0);
> > 
> > 				if (cpu_is_offline(cpu))
> > 					play_dead();
> > 
> > 				local_irq_disable();
> > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > 				/* Don't trace irqs off for idle */
> > 				stop_critical_timings();
> > 				pm_idle();
> > 				start_critical_timings();
> > 			}
> > 			tick_nohz_restart_sched_tick();
> > 			preempt_enable_no_resched();
> > 			schedule();
> > 			preempt_disable();
> > 		}
> > 	}
> > 
> > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > each time.  I would expect rcu_pending() to return 0 most of the time,
> > but that apparently isn't the case with treercu...
> > 
> > What is the easiest way for me to make it easy to trace the return path
> > from __rcu_pending()?  Make each return path call an empty function
> > located off where the compiler cannot see it, I guess...  Diagnostic
> > patch along these lines below.  Frederic, Damien, could you please give
> > it a go?  (And of course please let me know if something else is
> > needed.)
> 
> 
> No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> inside the functions, ie:
> 
> __rcu_pending() {
> 	 /* pending_qs */
> }

Ah!!!  So if I were to put ftrace_printk() calls at strategic points
in the RCU code, that would be a good thing?

> I've converted your below patch with ftrace_printks and tested it under an old P2
> with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> lucky :-) 
> I guess I successfully reproduced the softirq/rcu overhead.
> Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> When I will have more time, I will fix that.
> 
> The trace is here http://dl.free.fr/uyWGgCbx4
> 
> It looks like it mostly returns 1 because of the waiting for quiescent state:
> 
> $ cat rcutrace | grep "/* pending_none" | wc -l
> 221
> $ cat rcutrace | grep "/* pending_qs" | wc -l
> 248
> $ cat rcutrace | grep "/* pending" | wc -l
> 469

Hmmm...  This looks like normal behavior.  Though I wonder if
rcu_check_callbacks() is recognizing that we are in the idle loop given
the large number of "pending_qs" entries.  To that end, would you be
willing to try the attached patch (on top of your ftrace_printk() patch)?

Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
determine when RCU has detected a quiescent state due to interrupting
from within it.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---

 rcutree.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..fa14a0f 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
 
 		rcu_qsctr_inc(cpu);
 		rcu_bh_qsctr_inc(cpu);
+		ftrace_printk("rcu user/idle");
 
 	} else if (!in_softirq()) {
 
@@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
 		 */
 
 		rcu_bh_qsctr_inc(cpu);
+		ftrace_printk("rcu !softirq");
 	}
 	raise_softirq(RCU_SOFTIRQ);
 }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > 
> > > Hmmm...
> > > 
> > > Looks like we never return from:
> > > 
> > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > 
> > > Perhaps we are taking an interrupt immediately after the
> > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > And then deciding to go back into nohz mode at 799.521326, 117
> > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > which again raises RCU's softirq.
> > > 
> > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > which explains my failure to reproduce on a 64-bit system:
> > > 
> > > 	void cpu_idle(void)
> > > 	{
> > > 		int cpu = smp_processor_id();
> > > 
> > > 		current_thread_info()->status |= TS_POLLING;
> > > 
> > > 		/* endless idle loop with no priority at all */
> > > 		while (1) {
> > > 			tick_nohz_stop_sched_tick(1);
> > > 			while (!need_resched()) {
> > > 
> > > 				check_pgt_cache();
> > > 				rmb();
> > > 
> > > 				if (rcu_pending(cpu))
> > > 					rcu_check_callbacks(cpu, 0);
> > > 
> > > 				if (cpu_is_offline(cpu))
> > > 					play_dead();
> > > 
> > > 				local_irq_disable();
> > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > 				/* Don't trace irqs off for idle */
> > > 				stop_critical_timings();
> > > 				pm_idle();
> > > 				start_critical_timings();
> > > 			}
> > > 			tick_nohz_restart_sched_tick();
> > > 			preempt_enable_no_resched();
> > > 			schedule();
> > > 			preempt_disable();
> > > 		}
> > > 	}
> > > 
> > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > but that apparently isn't the case with treercu...
> > > 
> > > What is the easiest way for me to make it easy to trace the return path
> > > from __rcu_pending()?  Make each return path call an empty function
> > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > patch along these lines below.  Frederic, Damien, could you please give
> > > it a go?  (And of course please let me know if something else is
> > > needed.)
> > 
> > 
> > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > inside the functions, ie:
> > 
> > __rcu_pending() {
> > 	 /* pending_qs */
> > }
> 
> Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> in the RCU code, that would be a good thing?


Only when you are doing some debugging yes. But it is not a good thing to put an ftrace_printk
for code that has to be officially released since it adds a small overhead.
And actually ftrace_printk() is only for casual debugging, IMHO we shoudn't find any ftrace_printk
on the mainline code.

Instead, if you need some constant and defined probe inside your code, it's better to use
tracepoints, since they only add the overhead of a single branch check when they are off.


> > I've converted your below patch with ftrace_printks and tested it under an old P2
> > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > lucky :-) 
> > I guess I successfully reproduced the softirq/rcu overhead.
> > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > When I will have more time, I will fix that.
> > 
> > The trace is here http://dl.free.fr/uyWGgCbx4
> > 
> > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > 
> > $ cat rcutrace | grep "/* pending_none" | wc -l
> > 221
> > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > 248
> > $ cat rcutrace | grep "/* pending" | wc -l
> > 469
> 
> Hmmm...  This looks like normal behavior.  Though I wonder if
> rcu_check_callbacks() is recognizing that we are in the idle loop given
> the large number of "pending_qs" entries.  To that end, would you be
> willing to try the attached patch (on top of your ftrace_printk() patch)?
> 
> Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> determine when RCU has detected a quiescent state due to interrupting
> from within it.


Ok. I'm just fixing the orphans comments on the function graph tracer (the init_tasks
were not traced) and I test it.


> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> 
>  rcutree.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..fa14a0f 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
>  
>  		rcu_qsctr_inc(cpu);
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu user/idle");
>  
>  	} else if (!in_softirq()) {
>  
> @@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
>  		 */
>  
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu !softirq");
>  	}
>  	raise_softirq(RCU_SOFTIRQ);
>  }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > 
> > > Hmmm...
> > > 
> > > Looks like we never return from:
> > > 
> > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > 
> > > Perhaps we are taking an interrupt immediately after the
> > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > And then deciding to go back into nohz mode at 799.521326, 117
> > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > which again raises RCU's softirq.
> > > 
> > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > which explains my failure to reproduce on a 64-bit system:
> > > 
> > > 	void cpu_idle(void)
> > > 	{
> > > 		int cpu = smp_processor_id();
> > > 
> > > 		current_thread_info()->status |= TS_POLLING;
> > > 
> > > 		/* endless idle loop with no priority at all */
> > > 		while (1) {
> > > 			tick_nohz_stop_sched_tick(1);
> > > 			while (!need_resched()) {
> > > 
> > > 				check_pgt_cache();
> > > 				rmb();
> > > 
> > > 				if (rcu_pending(cpu))
> > > 					rcu_check_callbacks(cpu, 0);
> > > 
> > > 				if (cpu_is_offline(cpu))
> > > 					play_dead();
> > > 
> > > 				local_irq_disable();
> > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > 				/* Don't trace irqs off for idle */
> > > 				stop_critical_timings();
> > > 				pm_idle();
> > > 				start_critical_timings();
> > > 			}
> > > 			tick_nohz_restart_sched_tick();
> > > 			preempt_enable_no_resched();
> > > 			schedule();
> > > 			preempt_disable();
> > > 		}
> > > 	}
> > > 
> > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > but that apparently isn't the case with treercu...
> > > 
> > > What is the easiest way for me to make it easy to trace the return path
> > > from __rcu_pending()?  Make each return path call an empty function
> > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > patch along these lines below.  Frederic, Damien, could you please give
> > > it a go?  (And of course please let me know if something else is
> > > needed.)
> > 
> > 
> > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > inside the functions, ie:
> > 
> > __rcu_pending() {
> > 	 /* pending_qs */
> > }
> 
> Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> in the RCU code, that would be a good thing?
> 
> > I've converted your below patch with ftrace_printks and tested it under an old P2
> > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > lucky :-) 
> > I guess I successfully reproduced the softirq/rcu overhead.
> > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > When I will have more time, I will fix that.
> > 
> > The trace is here http://dl.free.fr/uyWGgCbx4
> > 
> > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > 
> > $ cat rcutrace | grep "/* pending_none" | wc -l
> > 221
> > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > 248
> > $ cat rcutrace | grep "/* pending" | wc -l
> > 469
> 
> Hmmm...  This looks like normal behavior.  Though I wonder if
> rcu_check_callbacks() is recognizing that we are in the idle loop given
> the large number of "pending_qs" entries.  To that end, would you be
> willing to try the attached patch (on top of your ftrace_printk() patch)?
> 
> Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> determine when RCU has detected a quiescent state due to interrupting
> from within it.


Do you still need this trace even if your solution were applied on -tip ?

 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> 
>  rcutree.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..fa14a0f 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
>  
>  		rcu_qsctr_inc(cpu);
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu user/idle");
>  
>  	} else if (!in_softirq()) {
>  
> @@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
>  		 */
>  
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu !softirq");
>  	}
>  	raise_softirq(RCU_SOFTIRQ);
>  }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 11:37:42PM +0100, Frederic Weisbecker wrote:
> On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> > On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > > 
> > > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > > > 
> > > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > > microseconds apart, so that this function is consuming more 
> > > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > > something that makes it invisible to ftrace.
> > > > > 
> > > > > look at the latest trace please, that has even the most inline 
> > > > > raise-softirq method instrumented, so all the raising is 
> > > > > visible.
> > > > 
> > > > Ah, my apologies!  This time looking at:
> > > > 
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > > 
> > > > 
> > > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > > [ . . . ]
> > > > 
> > > > Yikes!!!
> > > > 
> > > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > > but once per jiffy, and here it is called no less than 22 times in about
> > > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > > 
> > > > Hmmm...
> > > > 
> > > > Looks like we never return from:
> > > > 
> > > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > > 
> > > > Perhaps we are taking an interrupt immediately after the
> > > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > > And then deciding to go back into nohz mode at 799.521326, 117
> > > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > > which again raises RCU's softirq.
> > > > 
> > > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > > which explains my failure to reproduce on a 64-bit system:
> > > > 
> > > > 	void cpu_idle(void)
> > > > 	{
> > > > 		int cpu = smp_processor_id();
> > > > 
> > > > 		current_thread_info()->status |= TS_POLLING;
> > > > 
> > > > 		/* endless idle loop with no priority at all */
> > > > 		while (1) {
> > > > 			tick_nohz_stop_sched_tick(1);
> > > > 			while (!need_resched()) {
> > > > 
> > > > 				check_pgt_cache();
> > > > 				rmb();
> > > > 
> > > > 				if (rcu_pending(cpu))
> > > > 					rcu_check_callbacks(cpu, 0);
> > > > 
> > > > 				if (cpu_is_offline(cpu))
> > > > 					play_dead();
> > > > 
> > > > 				local_irq_disable();
> > > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > > 				/* Don't trace irqs off for idle */
> > > > 				stop_critical_timings();
> > > > 				pm_idle();
> > > > 				start_critical_timings();
> > > > 			}
> > > > 			tick_nohz_restart_sched_tick();
> > > > 			preempt_enable_no_resched();
> > > > 			schedule();
> > > > 			preempt_disable();
> > > > 		}
> > > > 	}
> > > > 
> > > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > > but that apparently isn't the case with treercu...
> > > > 
> > > > What is the easiest way for me to make it easy to trace the return path
> > > > from __rcu_pending()?  Make each return path call an empty function
> > > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > > patch along these lines below.  Frederic, Damien, could you please give
> > > > it a go?  (And of course please let me know if something else is
> > > > needed.)
> > > 
> > > 
> > > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > > inside the functions, ie:
> > > 
> > > __rcu_pending() {
> > > 	 /* pending_qs */
> > > }
> > 
> > Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> > in the RCU code, that would be a good thing?
> > 
> > > I've converted your below patch with ftrace_printks and tested it under an old P2
> > > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > > lucky :-) 
> > > I guess I successfully reproduced the softirq/rcu overhead.
> > > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > > When I will have more time, I will fix that.
> > > 
> > > The trace is here http://dl.free.fr/uyWGgCbx4
> > > 
> > > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > > 
> > > $ cat rcutrace | grep "/* pending_none" | wc -l
> > > 221
> > > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > > 248
> > > $ cat rcutrace | grep "/* pending" | wc -l
> > > 469
> > 
> > Hmmm...  This looks like normal behavior.  Though I wonder if
> > rcu_check_callbacks() is recognizing that we are in the idle loop given
> > the large number of "pending_qs" entries.  To that end, would you be
> > willing to try the attached patch (on top of your ftrace_printk() patch)?
> > 
> > Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> > determine when RCU has detected a quiescent state due to interrupting
> > from within it.
> 
> Do you still need this trace even if your solution were applied on -tip ?

No, it was my confusion -- I later realized that your data above meant
that the force-quiescent-state code path was not being heavily exercised.
So no need for this trace!

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:

> No, it was my confusion -- I later realized that your data 
> above meant that the force-quiescent-state code path was not 
> being heavily exercised. So no need for this trace!

Do you have any theory for why RCU was activated every 100-200 
microseconds, resulting in 20% ksoftirqd CPU use - and why the 
problem went away with classic-rcu?

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Wed, Feb 18, 2009 at 01:38:01AM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> 
> > No, it was my confusion -- I later realized that your data 
> > above meant that the force-quiescent-state code path was not 
> > being heavily exercised. So no need for this trace!
> 
> Do you have any theory for why RCU was activated every 100-200 
> microseconds, resulting in 20% ksoftirqd CPU use - and why the 
> problem went away with classic-rcu?

RCU was activated every 100-200 microseconds because the x86 32-bit
idle loop would call rcu_pending() and rcu_check_callbacks() in a tight
loop under some conditions.  This was happening to both classic and
tree RCU, but classic RCU has a more exact rcu_pending() check, and so
classic RCU's rcu_pending() always returns false, so that classic RCU's
rcu_check_callbacks() was never invoked, so that the raise_softirq()
is never called, so that control never passed to ksoftirqd, so that
things like "uptime" could not see the activity.

But the activity was occurring with classic RCU nevertheless.

</useful information>

<aside>

Interestingly enough, this is actually a symptom of a theoretical bug
in classic RCU (noted by Manfred Spraul some months ago).  Classic RCU
assumes that interrupts from dynticks idle mode don't run long enough
to run through a full grace period (which, in absence of truly broken
driver code, they do not).  Therefore, classic RCU removes all dynticks
idle CPUs from considation at the beginning of each grace period, so
that classic RCU's rcu_pending() doesn't have to concern itself with
other dyntick-idle CPUs.

When rcu_pending() is invoked once per jiffy, the additional checking
that tree RCU must do is in the noise, but not so when called repeatedly
from the idle loop.

</aside>

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> [2009-02-16 14:39]:
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)

As Frederic already sent a trace (made with another method), I'll
consider for now mine would not be very useful at this stage, so I'm
waiting for further instructions.

If you need further testing for me, do not hesitate to ask...

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 07:11:42AM +0100, Damien Wyart wrote:
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> [2009-02-16 14:39]:
> > What is the easiest way for me to make it easy to trace the return path
> > from __rcu_pending()?  Make each return path call an empty function
> > located off where the compiler cannot see it, I guess...  Diagnostic
> > patch along these lines below.  Frederic, Damien, could you please give
> > it a go?  (And of course please let me know if something else is
> > needed.)
> 
> As Frederic already sent a trace (made with another method), I'll
> consider for now mine would not be very useful at this stage, so I'm
> waiting for further instructions.
> 
> If you need further testing for me, do not hesitate to ask...

I would be very interested to hear whether or not my patch removing
rcu_pending() and rcu_check_callbacks() from the x86_32 idle loop
fixes the problem for you.

						Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> [2009-02-16 10:56]:
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier? The commands for this would be:

> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata

Here are the two files (stored while the ksoftirqd problem occurs)
attached.

-- 
Damien

[-- Attachment #2: rcuhier.txt --]
[-- Type: text/plain, Size: 170 bytes --]

rcu:
c=6148 g=6149 s=2 jfq=3 j=31ba nfqs=62392/nfqsng=3(62389) fqlh=7
1/3 0:1 ^0    
rcu_bh:
c=-296 g=-296 s=2 jfq=-93079 j=31ba nfqs=6/nfqsng=0(6) fqlh=0
0/3 0:1 ^0    

[-- Attachment #3: rcudata.txt --]
[-- Type: text/plain, Size: 417 bytes --]

rcu:
  0 c=6163 g=6164 pq=0 pqc=6162 qp=1 rpfq=2 rp=a249 dt=729511/1 dn=0 df=10 of=0 ri=38915 ql=5 b=10
  1 c=6163 g=6164 pq=1 pqc=6163 qp=0 rpfq=0 rp=d580 dt=34418019/1 dn=0 df=374 of=0 ri=19636 ql=0 b=10
rcu_bh:
  0 c=-296 g=-296 pq=1 pqc=-296 qp=0 rpfq=-19494 rp=e947 dt=729511/1 dn=0 df=0 of=0 ri=0 ql=0 b=10
  1 c=-296 g=-296 pq=1 pqc=-296 qp=1 rpfq=-16839253 rp=2918 dt=34418019/1 dn=0 df=0 of=0 ri=2 ql=0 b=10

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Christian Kujau]

On Sun, 15 Feb 2009, Ingo Molnar wrote:
>    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
>    http://people.redhat.com/mingo/tip.git/README

Hm, both URLs gave me a 404, .../tip.git/ seems empty - could these 
documents find a nice place under Documentation/ perhaps?

Thanks,
Christian.
-- 
BOFH excuse #321:

Scheduled global CPU outage

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Christian Kujau <lists@nerdbynature.de> wrote:

> On Sun, 15 Feb 2009, Ingo Molnar wrote:
> >    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
> >    http://people.redhat.com/mingo/tip.git/README
> 
> Hm, both URLs gave me a 404, .../tip.git/ seems empty - could these 
> documents find a nice place under Documentation/ perhaps?

Hm, there's a people.redhat.com outage today - find below that file
in plaintext too.

	Ingo

# --------------{ tip.git instructions }---------->

mkdir linux.trees.git || exit -1
cd linux.trees.git

git init
# Add Linus's tree as a remote
git remote add linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# Add the -tip tree as a remote
git remote add tip git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git

#
# With that setup, just run the following to get any changes you
# don't have.  It will also notice any new branches Ingo/Linus
# add to their repo.  Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
#
# This will take a lot of time initially (it downloads the ~150MB
# repository), but will be much faster subsequently as it only
# does delta updates. Note that it may warn you about no common
# commits but you can ignore that:
# 
git remote update

#
# Check out the latest -tip branch to a local branch
# in this example, we create a branch called 'tip-latest'
# You can pick whatever name suits you.
#
git checkout -b tip-latest tip/master

#
# if you need to do bisection of the -tip tree, then do:
# (but first check that linus/master is indeed a 'good' kernel :-)
#
git bisect start
git bisect good linus/master
git bisect bad tip/master

#
# If you want to help out with cleanups, and want to pick some
# low hanging fruits, do this:
#
wget http://redhat.com/~mingo/tip.git/code-quality
chmod +x code-quality

./code-quality `find kernel/ -name '*.c'` | tee quality.txt

#
# Pick the file that looks most interesting to you:
#
sort -n -k 4 quality.txt

#
# and if you do some work based on tip.git, in particular when
# you change x86 specific bits, feel free to talk to the
# maintainers about it (especially if you are about to
# do some bigger chunk of work and think that you'd like to
# ask whether it makes sense or whether anyone else is working
# on it):
#
#  X86 ARCHITECTURE (32-BIT AND 64-BIT)
#  P:      Thomas Gleixner
#  M:      tglx@linutronix.de
#  P:      Ingo Molnar
#  M:      mingo@redhat.com
#  P:      H. Peter Anvin
#  M:      hpa@zytor.com
#
# And this is the mailing list to send patches to:
#
#  L:      linux-kernel@vger.kernel.org
#
# When sending arch/x86 patches, please try to use the following
# subject line format (sample):
#
# Subject: [patch] x86: fix typo in ...
#