[Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org>
Date		: 2009-01-20 16:25 (20 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

2.6.29-rc5: Reported regressions from 2.6.28

[Rafael J. Wysocki]

This message contains a list of some regressions from 2.6.28, for which there
are no fixes in the mainline I know of.  If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.28, please let me know
either and I'll add them to the list.  Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2009-02-14       85       33          27
  2009-02-08       82       45          36
  2009-02-04       66       51          39
  2009-01-20       38       35          27
  2009-01-11       13       13          10


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12706
Subject		: Oopses and ACPI problems (Linus 2.6.29-rc4)
Submitter	: Darren Salt <linux@youmustbejoking.demon.co.uk>
Date		: 2009-02-09 18:26 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=123420431709877&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12705
Subject		: X200: Brightness broken since 2.6.29-rc4-58-g4c098bc
Submitter	: Nico Schottelius <nico-linux-20090213@schottelius.org>
Date		: 2009-02-13 9:33 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123451768406825&w=4
Handled-By	: Len Brown <lenb@kernel.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12681
Subject		: s2ram: fails to wake up on Acer Extensa 4220 (SMP disabled)
Submitter	: Orivej Desh <smpuj@bk.ru>
Date		: 2009-02-09 13:01 (6 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1cfe62c8010ac56e1bd3827e30386a87cc2f3594


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12680
Subject		: Not having a VIA PadLock hardware incurs a long delay in probing on modules insertion attempt.
Submitter	: Valentin QUEQUET <v.quequet-techniques@orange.fr>
Date		: 2009-02-09 09:12 (6 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12671
Subject		: uvc_status_cleanup(): undefined reference to `input_unregister_device'
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-02-08 14:58 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123410529909318&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12670
Subject		: BUG: unable to handle kernel paging request at pin_to_kill+0x21
Submitter	: Alessandro Bono <alessandro.bono@gmail.com>
Date		: 2009-02-08 11:04 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123409113223833&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
Subject		: USB flash disk surprise disconnect
Submitter	: Vegard Nossum <vegard.nossum@gmail.com>
Date		: 2009-02-08 10:21 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
Submitter	: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date		: 2009-02-04 21:11 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
Handled-By	: Ingo Molnar <mingo@elte.hu>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart@free.fr>
Date		: 2009-01-20 16:25 (26 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12617
Subject		: unable to compile e100 firmware into kernel
Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
Date		: 2009-01-31 15:59 (15 days old)
References	: http://marc.info/?l=linux-kernel&m=123341764915181&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
Subject		: boot hangs while bringing up gianfar ethernet
Submitter	: Ira Snyder <iws@ovro.caltech.edu>
Date		: 2009-01-29 19:41 (17 days old)
References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
Subject		: [Suspend regression][DRM, RADEON]
Submitter	: etienne <etienne.basset@numericable.fr>
Date		: 2009-01-28 22:00 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
		  http://lkml.org/lkml/2009/2/8/203


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
Subject		: sync-Regression in 2.6.28.2?
Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>
Date		: 2009-01-27 9:35 (19 days old)
References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject		: possible circular locking dependency detected
Submitter	: Michael S. Tsirkin <m.s.tsirkin@gmail.com>
Date		: 2009-01-29 11:35 (17 days old)
References	: http://lkml.org/lkml/2009/2/9/205


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12571
Subject		: Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*
Submitter	: Ross Boswell <drb@med.co.nz>
Date		: 2009-01-29 03:46 (17 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12551
Subject		: end_request: I/O error, dev cciss/c0d0, sector 87435720
Submitter	: Ralf Hildebrandt <ralf.hildebrandt@charite.de>
Date		: 2009-01-27 06:51 (19 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
Subject		: 2.6.29-rc2 dies on startup
Submitter	: Ferenc Wagner <wferi@niif.hu>
Date		: 2009-01-19 12:53 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12502
Subject		: pipe_read oops on sh
Submitter	: Adrian McMenamin <lkmladrian@gmail.com>
Date		: 2009-01-15 9:48 (31 days old)
References	: http://marc.info/?l=linux-kernel&m=123201298005600&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12501
Subject		: build bug in eeepc-laptop.c
Submitter	: Ingo Molnar <mingo@elte.hu>
Date		: 2009-01-14 17:25 (32 days old)
References	: http://lkml.org/lkml/2009/1/14/315


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12499
Subject		: Problem with using bluetooth adaper connected to usb port
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2009-01-13 18:34 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123187185426236&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12497
Subject		: new barrier warnings in 2.6.29-rc1
Submitter	: Christoph Hellwig <hch@lst.de>
Date		: 2009-01-12 15:46 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123177528217154&w=4
Handled-By	: Jens Axboe <jens.axboe@oracle.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
Subject		: Sony backlight regression from 2.6.28 to 29-rc
Submitter	: Norbert Preining <preining@logic.at>
Date		: 2009-01-19 8:14 (27 days old)
References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
Handled-By	: Mattia Dongili <malattia@linux.it>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12490
Subject		: ath5k related kernel panic in 2.6.29-rc1
Submitter	: Sergey S. Kostyliov <rathamahata@gmail.com>
Date		: 2009-01-12 7:38 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By	: Bob Copeland <me@bobcopeland.com>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12444
Subject		: X hangs following switch from radeonfb console - Bisected
Submitter	: Graham Murray <graham@gmurray.org.uk>
Date		: 2009-01-13 14:03 (33 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
Subject		: possible circular locking dependency on i915 dma
Submitter	: Wang Chen <wangchen@cn.fujitsu.com>
Date		: 2009-01-08 14:11 (38 days old)
References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12418
Subject		: Repeated ioctl(4, 0x40046445, ..) loop in glxgears
Submitter	: Pallipadi, Venkatesh <venkatesh.pallipadi@intel.com>
Date		: 2009-01-07 22:43 (39 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a
References	: http://marc.info/?l=linux-kernel&m=123136836213319&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12414
Subject		: iwl4965 cannot use "ap auto" on latest 2.6.28/29?
Submitter	: Jeff Chua <jeff.chua.linux@gmail.com>
Date		: 2009-01-05 4:13 (41 days old)
References	: http://marc.info/?l=linux-kernel&m=123112882127823&w=4


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12663
Subject		: Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume
Submitter	: Jiri Slaby <jirislaby@gmail.com>
Date		: 2009-02-07 18:40 (8 days old)
References	: http://lkml.org/lkml/2009/2/7/100
Handled-By	: Brian Gerst <brgerst@gmail.com>
Patch		: http://lkml.org/lkml/2009/2/7/100
		  http://lkml.org/lkml/2009/2/8/59


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12659
Subject		: Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).
Submitter	: Miles Lane <miles.lane@gmail.com>
Date		: 2009-02-03 4:58 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=123363718614763&w=4
Handled-By	: Rafael J. Wysocki <rjw@sisk.pl>
Patch		: http://marc.info/?l=linux-kernel&m=123456499020575&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12618
Subject		: hackbench [pthread mode] regression with 2.6.29-rc3
Submitter	: Zhang, Yanmin <yanmin_zhang@linux.intel.com>
Date		: 2009-02-01 7:30 (14 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=490dea45d00f01847ebebd007685d564aaf2cd98
References	: http://marc.info/?l=linux-kernel&m=123347347726527&w=4
Handled-By	: Peter Zijlstra <peterz@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123383366122146&w=4
		  http://marc.info/?l=linux-kernel&m=123383366222152&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
Submitter	: Larry Finger <Larry.Finger@lwfinger.net>
Date		: 2009-01-23 23:52 (23 days old)
References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
Handled-By	: Mikael Pettersson <mikpe@it.uu.se>
		  Hugh Dickins <hugh@veritas.com>
		  Sergei Shtylyov <sshtylyov@ru.mvista.com>
Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
Subject		: swsusp cannot find resume device (sometimes)
Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
Date		: 2009-01-09 12:24 (37 days old)
References	: http://marc.info/?l=linux-kernel&m=123150395731165&w=4
Handled-By	: Arjan van de Ven <arjan@infradead.org>
Patch		: http://marc.info/?l=linux-kernel&m=123156441218358&w=4
		  http://marc.info/?t=123156453100002&r=1&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg@intel.com>
Date		: 2009-01-13 23:17 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
Handled-By	: Roland Dreier <rolandd@cisco.com>
Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.28,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=12398

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael

[Bug #12414] iwl4965 cannot use "ap auto" on latest 2.6.28/29?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12414
Subject		: iwl4965 cannot use "ap auto" on latest 2.6.28/29?
Submitter	: Jeff Chua <jeff.chua.linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-05 4:13 (41 days old)
References	: http://marc.info/?l=linux-kernel&m=123112882127823&w=4

[Bug #12444] X hangs following switch from radeonfb console - Bisected

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12444
Subject		: X hangs following switch from radeonfb console - Bisected
Submitter	: Graham Murray <graham-0mDQKtL6jrwDXYZnReoRVg@public.gmane.org>
Date		: 2009-01-13 14:03 (33 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a

[Bug #12490] ath5k related kernel panic in 2.6.29-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12490
Subject		: ath5k related kernel panic in 2.6.29-rc1
Submitter	: Sergey S. Kostyliov <rathamahata-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-12 7:38 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By	: Bob Copeland <me-aXfl/3sk2vNUbtYUoyoikg@public.gmane.org>

[Bug #12418] Repeated ioctl(4, 0x40046445, ..) loop in glxgears

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12418
Subject		: Repeated ioctl(4, 0x40046445, ..) loop in glxgears
Submitter	: Pallipadi, Venkatesh <venkatesh.pallipadi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-07 22:43 (39 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=7c1c2871a6a3a114853ec6836e9035ac1c0c7f7a
References	: http://marc.info/?l=linux-kernel&m=123136836213319&w=4

[Bug #12419] possible circular locking dependency on i915 dma

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
Subject		: possible circular locking dependency on i915 dma
Submitter	: Wang Chen <wangchen-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
Date		: 2009-01-08 14:11 (38 days old)
References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4

[Bug #12491] i915 lockdep warning

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12491
Subject		: i915 lockdep warning
Submitter	: Brandeburg, Jesse <jesse.brandeburg-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-13 23:17 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123188898423532&w=4
Handled-By	: Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=123378709730700&w=2

[Bug #12494] Sony backlight regression from 2.6.28 to 29-rc

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
Subject		: Sony backlight regression from 2.6.28 to 29-rc
Submitter	: Norbert Preining <preining-DX+603jRYB8@public.gmane.org>
Date		: 2009-01-19 8:14 (27 days old)
References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
Handled-By	: Mattia Dongili <malattia-k2GhghHVRtY@public.gmane.org>

[Bug #12496] swsusp cannot find resume device (sometimes)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
Subject		: swsusp cannot find resume device (sometimes)
Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Date		: 2009-01-09 12:24 (37 days old)
References	: http://marc.info/?l=linux-kernel&m=123150395731165&w=4
Handled-By	: Arjan van de Ven <arjan-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=123156441218358&w=4
		  http://marc.info/?t=123156453100002&r=1&w=4

[Bug #12499] Problem with using bluetooth adaper connected to usb port

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12499
Subject		: Problem with using bluetooth adaper connected to usb port
Submitter	: Maciej Rutecki <maciej.rutecki-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-13 18:34 (33 days old)
References	: http://marc.info/?l=linux-kernel&m=123187185426236&w=4

[Bug #12497] new barrier warnings in 2.6.29-rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12497
Subject		: new barrier warnings in 2.6.29-rc1
Submitter	: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Date		: 2009-01-12 15:46 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=123177528217154&w=4
Handled-By	: Jens Axboe <jens.axboe-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

[Bug #12551] end_request: I/O error, dev cciss/c0d0, sector 87435720

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12551
Subject		: end_request: I/O error, dev cciss/c0d0, sector 87435720
Submitter	: Ralf Hildebrandt <ralf.hildebrandt-jq1tPX9l7E6ELgA04lAiVw@public.gmane.org>
Date		: 2009-01-27 06:51 (19 days old)

[Bug #12510] 2.6.29-rc2 dies on startup

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
Subject		: 2.6.29-rc2 dies on startup
Submitter	: Ferenc Wagner <wferi-eEbw3PyuezQ@public.gmane.org>
Date		: 2009-01-19 12:53 (27 days old)
References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4

[Bug #12502] pipe_read oops on sh

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12502
Subject		: pipe_read oops on sh
Submitter	: Adrian McMenamin <lkmladrian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-15 9:48 (31 days old)
References	: http://marc.info/?l=linux-kernel&m=123201298005600&w=4

[Bug #12501] build bug in eeepc-laptop.c

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12501
Subject		: build bug in eeepc-laptop.c
Submitter	: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
Date		: 2009-01-14 17:25 (32 days old)
References	: http://lkml.org/lkml/2009/1/14/315

[Bug #12571] Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12571
Subject		: Suspend-resume on Dell Latitude D410 newly broken in 2.6.29-rc*
Submitter	: Ross Boswell <drb-KpQn0vuWxAmlVyrhU4qvOw@public.gmane.org>
Date		: 2009-01-29 03:46 (17 days old)

[Bug #12574] possible circular locking dependency detected

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12574
Subject		: possible circular locking dependency detected
Submitter	: Michael S. Tsirkin <m.s.tsirkin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-01-29 11:35 (17 days old)
References	: http://lkml.org/lkml/2009/2/9/205

[Bug #12610] sync-Regression in 2.6.28.2?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
Subject		: sync-Regression in 2.6.28.2?
Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt-jq1tPX9l7E6ELgA04lAiVw@public.gmane.org>
Date		: 2009-01-27 9:35 (19 days old)
References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4

[Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
Submitter	: Larry Finger <Larry.Finger-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
Date		: 2009-01-23 23:52 (23 days old)
References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
Handled-By	: Mikael Pettersson <mikpe-1zs4UD6AkMk@public.gmane.org>
		  Hugh Dickins <hugh-DTz5qymZ9yRBDgjK7y7TUQ@public.gmane.org>
		  Sergei Shtylyov <sshtylyov-hkdhdckH98+B+jHODAdFcQ@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4

[Bug #12615] boot hangs while bringing up gianfar ethernet

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
Subject		: boot hangs while bringing up gianfar ethernet
Submitter	: Ira Snyder <iws-lulEs6mt1IksTUYHLfqkUA@public.gmane.org>
Date		: 2009-01-29 19:41 (17 days old)
References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4

[Bug #12613] [Suspend regression][DRM, RADEON]

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
Subject		: [Suspend regression][DRM, RADEON]
Submitter	: etienne <etienne.basset-Bf/eaXMDFuuXqB7oj33eUg@public.gmane.org>
Date		: 2009-01-28 22:00 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
		  http://lkml.org/lkml/2009/2/8/203

[Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
Submitter	: Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org>
Date		: 2009-01-20 16:25 (26 days old)
References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

[Bug #12659] Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12659
Subject		: Failure to resume two Sandisk USB flash drives attached to a Belkin USB Busport Mobile (F5U022).
Submitter	: Miles Lane <miles.lane-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-02-03 4:58 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=123363718614763&w=4
Handled-By	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=123456499020575&w=4

[Bug #12617] unable to compile e100 firmware into kernel

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12617
Subject		: unable to compile e100 firmware into kernel
Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
Date		: 2009-01-31 15:59 (15 days old)
References	: http://marc.info/?l=linux-kernel&m=123341764915181&w=4

[Bug #12618] hackbench [pthread mode] regression with 2.6.29-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12618
Subject		: hackbench [pthread mode] regression with 2.6.29-rc3
Submitter	: Zhang, Yanmin <yanmin_zhang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Date		: 2009-02-01 7:30 (14 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=490dea45d00f01847ebebd007685d564aaf2cd98
References	: http://marc.info/?l=linux-kernel&m=123347347726527&w=4
Handled-By	: Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=123383366122146&w=4
		  http://marc.info/?l=linux-kernel&m=123383366222152&w=4

[Bug #12660] Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
Submitter	: Mathieu Desnoyers <mathieu.desnoyers-scC8bbJcJLCw5LPnMra/2Q@public.gmane.org>
Date		: 2009-02-04 21:11 (11 days old)
References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
Handled-By	: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>

[Bug #12663] Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12663
Subject		: Commit 8c7e58e690ae60ab4215b025f433ed4af261e103 breaks resume
Submitter	: Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-02-07 18:40 (8 days old)
References	: http://lkml.org/lkml/2009/2/7/100
Handled-By	: Brian Gerst <brgerst-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Patch		: http://lkml.org/lkml/2009/2/7/100
		  http://lkml.org/lkml/2009/2/8/59

[Bug #12668] USB flash disk surprise disconnect

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12668
Subject		: USB flash disk surprise disconnect
Submitter	: Vegard Nossum <vegard.nossum-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-02-08 10:21 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123408851821292&w=4

[Bug #12671] uvc_status_cleanup(): undefined reference to `input_unregister_device'

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12671
Subject		: uvc_status_cleanup(): undefined reference to `input_unregister_device'
Submitter	: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
Date		: 2009-02-08 14:58 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123410529909318&w=4

[Bug #12670] BUG: unable to handle kernel paging request at pin_to_kill+0x21

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12670
Subject		: BUG: unable to handle kernel paging request at pin_to_kill+0x21
Submitter	: Alessandro Bono <alessandro.bono-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2009-02-08 11:04 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=123409113223833&w=4

[Bug #12680] Not having a VIA PadLock hardware incurs a long delay in probing on modules insertion attempt.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12680
Subject		: Not having a VIA PadLock hardware incurs a long delay in probing on modules insertion attempt.
Submitter	: Valentin QUEQUET <v.quequet-techniques-1tsiiZ//OF9QFI55V6+gNQ@public.gmane.org>
Date		: 2009-02-09 09:12 (6 days old)

[Bug #12681] s2ram: fails to wake up on Acer Extensa 4220 (SMP disabled)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12681
Subject		: s2ram: fails to wake up on Acer Extensa 4220 (SMP disabled)
Submitter	: Orivej Desh <smpuj-5URONGGNgjI@public.gmane.org>
Date		: 2009-02-09 13:01 (6 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1cfe62c8010ac56e1bd3827e30386a87cc2f3594

[Bug #12706] Oopses and ACPI problems (Linus 2.6.29-rc4)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12706
Subject		: Oopses and ACPI problems (Linus 2.6.29-rc4)
Submitter	: Darren Salt <linux-Ym6Ivm/Ks7B65+NM0QMiFAbYiX8G1TQY9dF7HbQ/qKg@public.gmane.org>
Date		: 2009-02-09 18:26 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=123420431709877&w=4

[Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.28.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12705
Subject		: X200: Brightness broken since 2.6.29-rc4-58-g4c098bc
Submitter	: Nico Schottelius <nico-linux-20090213-xuaVFQXs+5hIG4jRRZ66WA@public.gmane.org>
Date		: 2009-02-13 9:33 (2 days old)
References	: http://marc.info/?l=linux-kernel&m=123451768406825&w=4
Handled-By	: Len Brown <lenb-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

Re: [Bug #12660] Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p

[Mathieu Desnoyers]

* Rafael J. Wysocki (rjw@sisk.pl) wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12660
> Subject		: Linux 2.6.28.3 freezing on a 32-bits x86 Thinkpad T43p
> Submitter	: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> Date		: 2009-02-04 21:11 (11 days old)
> References	: http://marc.info/?l=linux-kernel&m=123378196022258&w=4
> Handled-By	: Ingo Molnar <mingo@elte.hu>
> 

Yes, still applies to 2.6.28.4 and 2.6.28.5.

Mathieu

> 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [Bug #12496] swsusp cannot find resume device (sometimes)

[Arjan van de Ven]

On Sat, 14 Feb 2009 21:38:16 +0100 (CET)
"Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me
> know (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
> Subject		: swsusp cannot find resume device (sometimes)
> Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> Date		: 2009-01-09 12:24 (37 days old)
> References	:
> http://marc.info/?l=linux-kernel&m=123150395731165&w=4
> Handled-By	: Arjan van de Ven <arjan-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
> Patch		:
> http://marc.info/?l=linux-kernel&m=123156441218358&w=4
> http://marc.info/?t=123156453100002&r=1&w=4
> 
> 

it is sad that a patch for a regression that has been available for
this long still has not been merged.

Sending me email over and over again does not make the maintainers
who have the patch in their queues get off their sit-organs any
faster....


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [Bug #12502] pipe_read oops on sh

[Adrian McMenamin]

2009/2/14 Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>:
> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12502
> Subject         : pipe_read oops on sh
> Submitter       : Adrian McMenamin <lkmladrian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Date            : 2009-01-15 9:48 (31 days old)
> References      : http://marc.info/?l=linux-kernel&m=123201298005600&w=4
>
>
>

No, it was caused by a race condition that has now been fixed.

Re: [Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Larry Finger]

Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
> Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
> Submitter	: Larry Finger <Larry.Finger-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
> Date		: 2009-01-23 23:52 (23 days old)
> References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
> 		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
> Handled-By	: Mikael Pettersson <mikpe-1zs4UD6AkMk@public.gmane.org>
> 		  Hugh Dickins <hugh-DTz5qymZ9yRBDgjK7y7TUQ@public.gmane.org>
> 		  Sergei Shtylyov <sshtylyov-hkdhdckH98+B+jHODAdFcQ@public.gmane.org>
> Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4

This problem is not fixed as of 2.6.29-rc5.

Larry

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello,

> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
> Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
> Submitter	: Damien Wyart <damien.wyart@free.fr>
> Date		: 2009-01-20 16:25 (26 days old)
> References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2

The problem is still there in 2.6.29-rc5, nothing has changed. I am
surprised noone has looked at it as it is trivial to reproduce for me.

Load average is between 0.30 and 0.60 when the machine is idle, the two
ksoftirqd threads get a very high total running time in top. This is on
a P4 machine. On a recent laptop, the load is not as high, but the
ksoftirqd threads also get a very high total running time.

-- 
Damien Wyart

Re: [Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Jeff Garzik]

Larry Finger wrote:
> Rafael J. Wysocki wrote:
>> This message has been generated automatically as a part of a report
>> of recent regressions.
>>
>> The following bug entry is on the current list of known regressions
>> from 2.6.28.  Please verify if it still should be listed and let me know
>> (either way).
>>
>>
>> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
>> Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
>> Submitter	: Larry Finger <Larry.Finger-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
>> Date		: 2009-01-23 23:52 (23 days old)
>> References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
>> 		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
>> Handled-By	: Mikael Pettersson <mikpe-1zs4UD6AkMk@public.gmane.org>
>> 		  Hugh Dickins <hugh-DTz5qymZ9yRBDgjK7y7TUQ@public.gmane.org>
>> 		  Sergei Shtylyov <sshtylyov-hkdhdckH98+B+jHODAdFcQ@public.gmane.org>
>> Patch		: http://marc.info/?l=linux-kernel&m=123412278730735&w=4
> 
> This problem is not fixed as of 2.6.29-rc5.

Sergei, what is the latest version of your patch?   According to my 
notes, you and Hugh were going back and forth trading patches, and I 
never saw a final version...

	Jeff

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> Hello,
> 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12650
> > Subject		: Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1
> > Submitter	: Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org>
> > Date		: 2009-01-20 16:25 (26 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123246937207675&w=2
> 
> The problem is still there in 2.6.29-rc5, nothing has changed. I am
> surprised noone has looked at it as it is trivial to reproduce for me.
> 
> Load average is between 0.30 and 0.60 when the machine is idle, the two
> ksoftirqd threads get a very high total running time in top. This is on
> a P4 machine. On a recent laptop, the load is not as high, but the
> ksoftirqd threads also get a very high total running time.

Mind having a look at this anomaly with the function graph tracer? We are
interested in a representative trace that shows the weird ksoftirq activities.
[ which trace could possibly pinpoint their origin. ]

Here's the tracing quickstart:

   http://redhat.com/~mingo/tip.git/tracing-quickstart.txt

Also attached below. Let me know if you have trouble getting a good trace,
or if any of the steps were non-intuitive or burdensome to you.

Thanks,

	Ingo

------------------------------>

You probably came here because:

- you have a weird latency somewhere that you'd like to debug?
- you have some weird kernel behavior that you'd like to see in detail?
- you are curious how Linux kernel internals look like on a real system?

A good generic tool for that is the built-in function graph tracer (available
in v2.6.28 and later Linux kernels), which will provide system-wide tracing
output of any workload you are interested in.

Sample output:

 # tracer: function_graph
 #
 # CPU DURATION        FUNCTION CALLS
 # |    |   |          |   |   |   |

   1) + 39.757 us   |    }
   1) + 41.593 us   |  }
   1)               |  sys_rt_sigprocmask() {
   1)   0.399 us    |    _spin_lock_irq();
   1)   1.209 us    |  }
   1)               |  sys_read() {
   1)   0.425 us    |    fget_light();
   1)               |    vfs_read() {
   1)               |      rw_verify_area() {
   1)               |        security_file_permission() {
   1)   0.377 us    |          cap_file_permission();
   1)   1.048 us    |        }
   1)   1.803 us    |      }
   1)               |      tty_read() {
   1)   0.476 us    |        tty_paranoia_check();
   1)               |        tty_ldisc_ref_wait() {
   1)               |          tty_ldisc_try() {
   1)   0.365 us    |            _spin_lock_irqsave();
   1)   0.426 us    |            _spin_unlock_irqrestore();
   1)   1.837 us    |          }
   1)   2.544 us    |        }

To get the very latest version of the tracer, pick up the -tip tree (which,
amongst other trees, also includes the latest tracing tree and Linus's latest
upstream -git tree):

   http://people.redhat.com/mingo/tip.git/README

And enable the following .config options:

  CONFIG_DEBUG_KERNEL=y
  CONFIG_FUNCTION_TRACER=y
  CONFIG_FUNCTION_GRAPH_TRACER=y
  CONFIG_DYNAMIC_FTRACE=y

Boot into the new kernel and mount debugfs (if you dont have it mounted
already):

  mkdir /debug
  mount -t debugfs nodev /debug

That's all - now you can use the tracer. There's no user-space utilities
needed, everything is built into the kernel and all functionality can
be accessed via the /debug/tracing/* special files.

For example, the following commands in a shell prompt will capture a
1-second trace of whatever happens on your box right now:

  cd /debug/tracing/
  echo function_graph_tracer > current_tracer
  echo 1 > tracing_enabled
  sleep 1
  echo 0 > tracing_enabled
  cat trace > /tmp/trace.txt

/tmp/trace.txt will contain the trace. If you are trying to debug a bug,
send that file to kernel developers :)

Tracing can be expensive, especially if every single kernel function is
traced (which is the default).

To solve that problem you can limit the scope of traced functions via the
function filter:

  echo ""              > set_ftrace_filter        # clear all previous filters
  echo "schedule*"    >> set_ftrace_filter        # trace schedule functions
  echo "*switch_to*"  >> set_ftrace_filter        # trace context switches
  echo "*wake_up*"    >> set_ftrace_filter        # trace task wakeups

Note that if you want the rules to be additive then filters have to be
appended via '>>'. Using '>' will override all previous filters again.

Use "cat set_ftrace_filter" to see the currently traced functions. (if the
file is empty then it means 'all') Use "cat available_filter_functions" for
all kernel functions that can be traced. On a typical system there will
be tens of thousands to pick from (!).

Besides the default output you can find additional attributes in the
'trace_options' file. For example:

  echo funcgraph-proc > trace_options

Will instruct the tracer to output which process/PID generated a trace
entry:

  0)   bash-2623    |               |  sys_dup2() {
  0)   bash-2623    |               |    sys_dup3() {
  0)   bash-2623    |   0.262 us    |      _spin_lock();
  0)   bash-2623    |   0.290 us    |      expand_files();
  0)   bash-2623    |               |      filp_close() {
  0)   bash-2623    |   0.260 us    |        dnotify_flush();
  0)   bash-2623    |   0.305 us    |        locks_remove_posix();
  0)   bash-2623    |               |        fput() {
  0)   bash-2623    |               |          __fput() {
  0)   bash-2623    |   0.393 us    |            _cond_resched();

Have fun tracing, but beware, if there's even just a little bit of
a kernel developer in you then it can be highly addictive! ;-)

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Load average is between 0.30 and 0.60 when the machine is idle, the
> > two ksoftirqd threads get a very high total running time in top.
> > This is on a P4 machine. On a recent laptop, the load is not as
> > high, but the ksoftirqd threads also get a very high total running
> > time.

> Mind having a look at this anomaly with the function graph tracer? We are
> interested in a representative trace that shows the weird ksoftirq activities.
> [ which trace could possibly pinpoint their origin. ]

> Here's the tracing quickstart:

>    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt

> Also attached below. Let me know if you have trouble getting a good trace,
> or if any of the steps were non-intuitive or burdensome to you.

Thanks for your feedback & explanations; I will try to have a look at
all this material asap, I think tomorrow.

Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Christian Kujau]

On Sun, 15 Feb 2009, Ingo Molnar wrote:
>    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
>    http://people.redhat.com/mingo/tip.git/README

Hm, both URLs gave me a 404, .../tip.git/ seems empty - could these 
documents find a nice place under Documentation/ perhaps?

Thanks,
Christian.
-- 
BOFH excuse #321:

Scheduled global CPU outage

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> > > Load average is between 0.30 and 0.60 when the machine is idle, the
> > > two ksoftirqd threads get a very high total running time in top.
> > > This is on a P4 machine. On a recent laptop, the load is not as
> > > high, but the ksoftirqd threads also get a very high total running
> > > time.
> 
> > Mind having a look at this anomaly with the function graph tracer? We are
> > interested in a representative trace that shows the weird ksoftirq activities.
> > [ which trace could possibly pinpoint their origin. ]
> 
> > Here's the tracing quickstart:
> 
> >    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
> 
> > Also attached below. Let me know if you have trouble getting a good trace,
> > or if any of the steps were non-intuitive or burdensome to you.
> 
> Thanks for your feedback & explanations; I will try to have a look at
> all this material asap, I think tomorrow.

Note that if the box you test this on is multi-core or HT, then interpreting
traces is easier if there's just a single CPU to look at. In that case i'd 
suggest to reproduce with just a single core, by turning the second one off:

   echo 0 > /sys/devices/system/cpu/cpu1/online

Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

   echo 2 > /debug/tracing/tracing_cpumask 

	Ingo

Re: [Bug #12613] [Suspend regression][DRM, RADEON]

[etienne]

etienne wrote:
> Rafael J. Wysocki wrote:
>> This message has been generated automatically as a part of a report
>> of recent regressions.
>>
>> The following bug entry is on the current list of known regressions
>> from 2.6.28.  Please verify if it still should be listed and let me know
>> (either way).
>>
>>
>> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12613
>> Subject		: [Suspend regression][DRM, RADEON]
>> Submitter	: etienne <etienne.basset-Bf/eaXMDFuuXqB7oj33eUg@public.gmane.org>
>> Date		: 2009-01-28 22:00 (18 days old)
>> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a9d51a5ad1154b5b20add1e8d30a5564f8aabbe9
>> References	: http://marc.info/?l=linux-kernel&m=123318030419558&w=4
>> 		  http://marc.info/?l=linux-kernel&m=123334865404574&w=4
>> 		  http://lkml.org/lkml/2009/2/8/203
>>
>>
>>
hello,

yes it is still present in -rc5;
the following unreviewed patch fixes it for me;

regards,
Etienne, who should learn to send correct emails

Signed-off-by: <etienne.basset-Bf/eaXMDFuuXqB7oj33eUg@public.gmane.org>
---
diff --git a/drivers/gpu/drm/radeon/radeon_cp.c b/drivers/gpu/drm/radeon/radeon_cp.c
index df4cf97..6554adf 100644
--- a/drivers/gpu/drm/radeon/radeon_cp.c
+++ b/drivers/gpu/drm/radeon/radeon_cp.c
@@ -557,7 +557,8 @@ static int radeon_do_engine_reset(struct drm_device * dev)
 }
 
 static void radeon_cp_init_ring_buffer(struct drm_device * dev,
-				       drm_radeon_private_t * dev_priv)
+				       drm_radeon_private_t * dev_priv,
+				       struct drm_radeon_master_private *master)
 {
 	u32 ring_start, cur_read_ptr;
 	u32 tmp;
@@ -668,13 +669,13 @@ static void radeon_cp_init_ring_buffer(struct drm_device * dev,
 		RADEON_WRITE(RADEON_BUS_CNTL, tmp);
 	} /* PCIE cards appears to not need this */
 
-	dev_priv->scratch[0] = 0;
+	master->sarea_priv->last_frame = dev_priv->scratch[0] = 0;
 	RADEON_WRITE(RADEON_LAST_FRAME_REG, 0);
 
-	dev_priv->scratch[1] = 0;
+	master->sarea_priv->last_dispatch = dev_priv->scratch[1] = 0;
 	RADEON_WRITE(RADEON_LAST_DISPATCH_REG, 0);
 
-	dev_priv->scratch[2] = 0;
+	master->sarea_priv->last_clear = dev_priv->scratch[2] = 0;
 	RADEON_WRITE(RADEON_LAST_CLEAR_REG, 0);
 
 	radeon_do_wait_for_idle(dev_priv);
@@ -1215,7 +1216,7 @@ static int radeon_do_init_cp(struct drm_device *dev, drm_radeon_init_t *init,
 	}
 
 	radeon_cp_load_microcode(dev_priv);
-	radeon_cp_init_ring_buffer(dev, dev_priv);
+	radeon_cp_init_ring_buffer(dev, dev_priv, master_priv);
 
 	dev_priv->last_buf = 0;
 
@@ -1281,9 +1282,11 @@ static int radeon_do_cleanup_cp(struct drm_device * dev)
  *
  * Charl P. Botha <http://cpbotha.net>
  */
-static int radeon_do_resume_cp(struct drm_device * dev)
+static int radeon_do_resume_cp(struct drm_device * dev,
+			       struct drm_file * file_priv)
 {
 	drm_radeon_private_t *dev_priv = dev->dev_private;
+	struct drm_radeon_master_private * master_priv = file_priv->master->driver_priv;
 
 	if (!dev_priv) {
 		DRM_ERROR("Called with no initialization\n");
@@ -1304,7 +1307,7 @@ static int radeon_do_resume_cp(struct drm_device * dev)
 	}
 
 	radeon_cp_load_microcode(dev_priv);
-	radeon_cp_init_ring_buffer(dev, dev_priv);
+	radeon_cp_init_ring_buffer(dev, dev_priv, master_priv);
 
 	radeon_do_engine_reset(dev);
 	radeon_irq_set_state(dev, RADEON_SW_INT_ENABLE, 1);
@@ -1480,7 +1483,7 @@ int radeon_cp_idle(struct drm_device *dev, void *data, struct drm_file *file_pri
 int radeon_cp_resume(struct drm_device *dev, void *data, struct drm_file *file_priv)
 {
 
-	return radeon_do_resume_cp(dev);
+	return radeon_do_resume_cp(dev, file_priv);
 }
 
 int radeon_engine_reset(struct drm_device *dev, void *data, struct drm_file *file_priv)

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [2009-02-15 11:13]:
> Note that if the box you test this on is multi-core or HT, then interpreting
> traces is easier if there's just a single CPU to look at. In that case i'd 
> suggest to reproduce with just a single core, by turning the second one off:

>    echo 0 > /sys/devices/system/cpu/cpu1/online

> Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

>    echo 2 > /debug/tracing/tracing_cpumask 

The box I test on is HT, so I tried the first suggestion and it made the
problem much less visible (but not completely absent).

So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
HT mode and then it made the problem much more visible on CPU#1:
ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
average is about 0.80 and the total running time for ksoftirqd/1 is
almost one minute (and I booted on rc5 ten minutes ago)!

So I followed the tracing steps in the tutorial (with the 1 sec sleep),
which gave me this:
http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

As I will be away until tomorrow, I did this on vanilla rc5 to get
something out today, and if tip is really needed, I will work on it
tomorrow. But maybe this vanilla trace will be helpful to you...

Do not hesitate to ask for further tests or info.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> Do not hesitate to ask for further tests or info.

I really must leave now, but please note the dmesg and config were
attached to my very first report. Will send new ones this evening (with
rc5) if you need them).

-- 
Damien Wyart

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Note that if the box you test this on is multi-core or HT, then interpreting
> > traces is easier if there's just a single CPU to look at. In that case i'd 
> > suggest to reproduce with just a single core, by turning the second one off:

> >    echo 0 > /sys/devices/system/cpu/cpu1/online

> > Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

> >    echo 2 > /debug/tracing/tracing_cpumask 

> The box I test on is HT, so I tried the first suggestion and it made the
> problem much less visible (but not completely absent).

> So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
> HT mode and then it made the problem much more visible on CPU#1:
> ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
> average is about 0.80 and the total running time for ksoftirqd/1 is
> almost one minute (and I booted on rc5 ten minutes ago)!

> So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> which gave me this:
> http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

Of course, I used your first suggestion (tracing on CPU#1) to get this
trace.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> [2009-02-15 11:42]:
> > > Note that if the box you test this on is multi-core or HT, then interpreting
> > > traces is easier if there's just a single CPU to look at. In that case i'd 
> > > suggest to reproduce with just a single core, by turning the second one off:

> > >    echo 0 > /sys/devices/system/cpu/cpu1/online

> > > Or, if the problem only occurs with two cpus, restrict tracing to CPU#1:

> > >    echo 2 > /debug/tracing/tracing_cpumask 

> > The box I test on is HT, so I tried the first suggestion and it made the
> > problem much less visible (but not completely absent).

> > So I used "echo 1 > /sys/devices/system/cpu/cpu1/online" to go back to
> > HT mode and then it made the problem much more visible on CPU#1:
> > ksoftirqd/1 is running a lot and ksoftirqd/0 is almost normal. The load
> > average is about 0.80 and the total running time for ksoftirqd/1 is
> > almost one minute (and I booted on rc5 ten minutes ago)!

> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

> Of course, I used your first suggestion (tracing on CPU#1) to get this
                         ^^^^^
                         second !

> trace.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Christian Kujau <lists-AanptEQQ3TL9uQeqpI+JUg@public.gmane.org> wrote:

> On Sun, 15 Feb 2009, Ingo Molnar wrote:
> >    http://redhat.com/~mingo/tip.git/tracing-quickstart.txt
> >    http://people.redhat.com/mingo/tip.git/README
> 
> Hm, both URLs gave me a 404, .../tip.git/ seems empty - could these 
> documents find a nice place under Documentation/ perhaps?

Hm, there's a people.redhat.com outage today - find below that file
in plaintext too.

	Ingo

# --------------{ tip.git instructions }---------->

mkdir linux.trees.git || exit -1
cd linux.trees.git

git init
# Add Linus's tree as a remote
git remote add linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# Add the -tip tree as a remote
git remote add tip git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git

#
# With that setup, just run the following to get any changes you
# don't have.  It will also notice any new branches Ingo/Linus
# add to their repo.  Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
#
# This will take a lot of time initially (it downloads the ~150MB
# repository), but will be much faster subsequently as it only
# does delta updates. Note that it may warn you about no common
# commits but you can ignore that:
# 
git remote update

#
# Check out the latest -tip branch to a local branch
# in this example, we create a branch called 'tip-latest'
# You can pick whatever name suits you.
#
git checkout -b tip-latest tip/master

#
# if you need to do bisection of the -tip tree, then do:
# (but first check that linus/master is indeed a 'good' kernel :-)
#
git bisect start
git bisect good linus/master
git bisect bad tip/master

#
# If you want to help out with cleanups, and want to pick some
# low hanging fruits, do this:
#
wget http://redhat.com/~mingo/tip.git/code-quality
chmod +x code-quality

./code-quality `find kernel/ -name '*.c'` | tee quality.txt

#
# Pick the file that looks most interesting to you:
#
sort -n -k 4 quality.txt

#
# and if you do some work based on tip.git, in particular when
# you change x86 specific bits, feel free to talk to the
# maintainers about it (especially if you are about to
# do some bigger chunk of work and think that you'd like to
# ask whether it makes sense or whether anyone else is working
# on it):
#
#  X86 ARCHITECTURE (32-BIT AND 64-BIT)
#  P:      Thomas Gleixner
#  M:      tglx-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org
#  P:      Ingo Molnar
#  M:      mingo-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
#  P:      H. Peter Anvin
#  M:      hpa-YMNOUZJC4hwAvxtiuMwx3w@public.gmane.org
#
# And this is the mailing list to send patches to:
#
#  L:      linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
#
# When sending arch/x86 patches, please try to use the following
# subject line format (sample):
#
# Subject: [patch] x86: fix typo in ...
#

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> which gave me this:
> http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

thanks. There's definitely some weirdness visible in the trace,
for example:

  0)    gpm-1879    =>   ksoftir-4   
  ------------------------------------------

  0)   0.964 us    |    finish_task_switch();
  0) ! 1768184 us |  }
  0)               |  do_softirq() {
  0)               |    __do_softirq() {
  0)               |      rcu_process_callbacks() {

the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
ksoftirqd _really_ took that much time to execute.

One possibility for these symptoms would be broken scheduler timestamps.
Could you enable absolute timestamp printing via:

    echo funcgraph-abstime  > trace_options

Also, my guess is that if you boot via idle=poll, the symptoms go away.
This would strengthen the suspicion that it's scheduler-clock troubles.

	Ingo

Re: [Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Sergei Shtylyov]

Hello.

Jeff Garzik wrote:

>>> This message has been generated automatically as a part of a report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.28.  Please verify if it still should be listed and let me 
>>> know
>>> (either way).
>>>
>>>
>>> Bug-Entry    : http://bugzilla.kernel.org/show_bug.cgi?id=12609
>>> Subject        : v2.6.29-rc2 libata sff 32bit PIO regression
>>> Submitter    : Larry Finger <Larry.Finger-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
>>> Date        : 2009-01-23 23:52 (23 days old)
>>> References    : http://marc.info/?l=linux-kernel&m=123275478111406&w=4
>>>           http://marc.info/?l=linux-kernel&m=123254501314058&w=4
>>> Handled-By    : Mikael Pettersson <mikpe-1zs4UD6AkMk@public.gmane.org>
>>>           Hugh Dickins <hugh-DTz5qymZ9yRBDgjK7y7TUQ@public.gmane.org>
>>>           Sergei Shtylyov <sshtylyov-hkdhdckH98+B+jHODAdFcQ@public.gmane.org>
>>> Patch        : http://marc.info/?l=linux-kernel&m=123412278730735&w=4
>>
>> This problem is not fixed as of 2.6.29-rc5.
>
> Sergei, what is the latest version of your patch?   According to my 
> notes, you and Hugh were going back and forth trading patches, and I 
> never saw a final version...

   I've been excessively busy again. Will post today.

>     Jeff

WBR, Sergei

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Matthew Garrett]

On Sat, Feb 14, 2009 at 09:38:23PM +0100, Rafael J. Wysocki wrote:

> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).

I think Eric sent a test patch for this. Did that get pushed?

-- 
Matthew Garrett | mjg59-1xO5oi07KQx4cg9Nei1l7Q@public.gmane.org

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Sun, Feb 15, 2009 at 12:01:04PM +0100, Ingo Molnar wrote:
> 
> * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> 
> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz
> 
> thanks. There's definitely some weirdness visible in the trace,
> for example:
> 
>   0)    gpm-1879    =>   ksoftir-4   
>   ------------------------------------------
> 
>   0)   0.964 us    |    finish_task_switch();
>   0) ! 1768184 us |  }
>   0)               |  do_softirq() {
>   0)               |    __do_softirq() {
>   0)               |      rcu_process_callbacks() {
> 
> the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> ksoftirqd _really_ took that much time to execute.
> 
> One possibility for these symptoms would be broken scheduler timestamps.
> Could you enable absolute timestamp printing via:
> 
>     echo funcgraph-abstime  > trace_options
> 
> Also, my guess is that if you boot via idle=poll, the symptoms go away.
> This would strengthen the suspicion that it's scheduler-clock troubles.
> 
> 	Ingo

Looking at the following ksoftirqd quantum, there are some things I don't understand:

 ------------------------------------------
 0)    gpm-1879    =>   ksoftir-4   
 ------------------------------------------

 0)   0.681 us    |    finish_task_switch();
 0) ! 7670.083 us |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          cpu_quiet() {
 0)   0.509 us    |            _spin_lock_irqsave();
 0)               |            cpu_quiet_msk() {
 0)   0.503 us    |              _spin_unlock_irqrestore();
 0)   1.546 us    |            }
 0)   3.617 us    |          }
 0)   4.804 us    |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.496 us    |          force_quiescent_state();
 0)   1.608 us    |        }
 0)   8.043 us    |      }
 0)   0.533 us    |      _local_bh_enable();
 0) + 10.190 us   |    }
 0) + 11.337 us   |  }
 0)   0.537 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.516 us    |    _spin_lock_irq();
 0)   0.586 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.519 us    |            calc_delta_mine();
 0)   1.603 us    |          }
 0)   2.667 us    |        }
 0)   3.662 us    |      }
 0)   4.629 us    |    }
 0)   0.937 us    |    find_busiest_group();
 0)   0.493 us    |    msecs_to_jiffies();
 0)   0.501 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.501 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_rt();
 0)   0.494 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_idle();
 0)   4.591 us    |    }
 0)   0.501 us    |    __lock_text_start();
 0)   0.699 us    |    finish_task_switch();
 0) ! 289.895 us  |  }


The two above is the internal of context_switch, which means it entered
there in schedule():

next = pick_next_task(rq, prev);
if (likely(prev != next)) {
	sched_info_switch(prev, next);
	perf_counter_task_sched_out(prev, cpu);

	rq->nr_switches++;
	rq->curr = next;
	++*switch_count;

	context_switch(rq, prev, next); /* unlocks the rq */
	/*
	 * the context switch might have flipped the stack from under
	 * us, hence refresh the local variables.
	 */
	cpu = smp_processor_id();
	rq = cpu_rq(cpu);
}

But after that we are still in ksoftirqd. As if it bypassed the if (prev != next)
and rescheduled itself.

Well, I guess it's a bit off topic here. Whatever if this a bug in the sched_clock(),
it reminds me a possible problem in sched_clock() I saw with tracers...



 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          force_quiescent_state() {
 0)   0.478 us    |            __lock_text_start();
 0)   0.489 us    |            _spin_lock();
 0)               |            rcu_process_dyntick() {
 0)   0.496 us    |              _spin_lock_irqsave();
 0)   0.571 us    |              dyntick_save_progress_counter();
 0)               |              cpu_quiet_msk() {
 0)               |                rcu_start_gp() {
 0)   0.496 us    |                  _spin_unlock_irqrestore();
 0)   1.533 us    |                }
 0)   2.588 us    |              }
 0)   5.784 us    |            }
 0)   0.493 us    |            _spin_lock();
 0)   0.501 us    |            _spin_unlock_irqrestore();
 0) + 10.959 us   |          }
 0)               |          file_free_rcu() {
 0)   0.551 us    |            kmem_cache_free();
 0)   1.613 us    |          }
 0) + 14.486 us   |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.458 us    |          force_quiescent_state();
 0)   1.573 us    |        }
 0) + 17.661 us   |      }
 0)   0.486 us    |      _local_bh_enable();
 0) + 20.014 us   |    }
 0) + 21.094 us   |  }
 0)   0.546 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.514 us    |    _spin_lock_irq();
 0)   0.596 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.500 us    |            calc_delta_mine();
 0)   1.570 us    |          }
 0)   2.632 us    |        }
 0)   3.632 us    |      }
 0)   4.594 us    |    }
 0)   0.846 us    |    find_busiest_group();
 0)   0.471 us    |    msecs_to_jiffies();
 0)   0.506 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.486 us    |      pick_next_task_fair();
 0)   0.481 us    |      pick_next_task_rt();
 0)   0.484 us    |      pick_next_task_fair();
 0)   0.493 us    |      pick_next_task_idle();
 0)   4.544 us    |    }
 0)   0.506 us    |    __lock_text_start();
 0)   0.554 us    |    finish_task_switch();
 0) + 30.202 us   |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          cpu_quiet() {
 0)   0.493 us    |            _spin_lock_irqsave();
 0)               |            cpu_quiet_msk() {
 0)   0.504 us    |              _spin_unlock_irqrestore();
 0)   1.515 us    |            }
 0)   3.535 us    |          }
 0)   4.692 us    |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.480 us    |          force_quiescent_state();
 0)   1.593 us    |        }
 0)   7.878 us    |      }
 0)   0.491 us    |      _local_bh_enable();
 0)   9.934 us    |    }
 0) + 11.037 us   |  }
 0)   0.529 us    |  _cond_resched();
 0)   0.506 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.499 us    |    _spin_lock_irq();
 0)   0.581 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.493 us    |            calc_delta_mine();
 0)   1.548 us    |          }
 0)   2.583 us    |        }
 0)   3.559 us    |      }
 0)   4.514 us    |    }
 0)   0.797 us    |    find_busiest_group();
 0)   0.481 us    |    msecs_to_jiffies();
 0)   0.499 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.481 us    |      pick_next_task_fair();
 0)   0.479 us    |      pick_next_task_rt();
 0)   0.483 us    |      pick_next_task_fair();
 0)   0.481 us    |      pick_next_task_idle();
 0)   4.488 us    |    }
 0)   0.506 us    |    __lock_text_start();
 0)   0.672 us    |    finish_task_switch();
 0) ! 928.567 us  |  }
 0)               |  do_softirq() {
 0)               |    __do_softirq() {
 0)               |      rcu_process_callbacks() {
 0)               |        __rcu_process_callbacks() {
 0)               |          force_quiescent_state() {
 0)   0.484 us    |            __lock_text_start();
 0)   0.511 us    |            _spin_lock();
 0)               |            rcu_process_dyntick() {
 0)   0.511 us    |              _spin_lock_irqsave();
 0)   0.564 us    |              dyntick_save_progress_counter();
 0)               |              cpu_quiet_msk() {
 0)               |                rcu_start_gp() {
 0)   0.503 us    |                  _spin_unlock_irqrestore();
 0)   1.548 us    |                }
 0)   2.582 us    |              }
 0)   6.102 us    |            }
 0)   0.501 us    |            _spin_lock();
 0)   0.501 us    |            _spin_unlock_irqrestore();
 0) + 11.265 us   |          }
 0)               |          file_free_rcu() {
 0)   0.579 us    |            kmem_cache_free();
 0)   1.643 us    |          }
 0)               |          file_free_rcu() {
 0)   0.549 us    |            kmem_cache_free();
 0)   1.588 us    |          }
 0) + 16.827 us   |        }
 0)               |        __rcu_process_callbacks() {
 0)   0.479 us    |          force_quiescent_state();
 0)   1.593 us    |        }
 0) + 20.024 us   |      }
 0)   0.500 us    |      _local_bh_enable();
 0) + 22.081 us   |    }
 0) + 23.163 us   |  }
 0)   0.523 us    |  _cond_resched();
 0)   0.536 us    |  kthread_should_stop();
 0)               |  schedule() {
 0)   0.506 us    |    _spin_lock_irq();
 0)   0.586 us    |    update_rq_clock();
 0)               |    deactivate_task() {
 0)               |      dequeue_task() {
 0)               |        dequeue_task_fair() {
 0)               |          update_curr() {
 0)   0.516 us    |            calc_delta_mine();
 0)   1.578 us    |          }
 0)   2.628 us    |        }
 0)   3.622 us    |      }
 0)   4.604 us    |    }
 0)   0.917 us    |    find_busiest_group();
 0)   0.484 us    |    msecs_to_jiffies();
 0)   0.501 us    |    put_prev_task_fair();
 0)               |    pick_next_task() {
 0)   0.486 us    |      pick_next_task_fair();
 0)   0.491 us    |      pick_next_task_rt();
 0)               |      pick_next_task_fair() {
 0)   0.486 us    |      pick_next_task_idle();
 0)   4.581 us    |    }
 0)   0.496 us    |    __lock_text_start();

Re: [Bug #12496] swsusp cannot find resume device (sometimes)

[Rafael J. Wysocki]

On Sunday 15 February 2009, Arjan van de Ven wrote:
> On Sat, 14 Feb 2009 21:38:16 +0100 (CET)
> "Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me
> > know (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12496
> > Subject		: swsusp cannot find resume device (sometimes)
> > Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> > Date		: 2009-01-09 12:24 (37 days old)
> > References	:
> > http://marc.info/?l=linux-kernel&m=123150395731165&w=4
> > Handled-By	: Arjan van de Ven <arjan-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
> > Patch		:
> > http://marc.info/?l=linux-kernel&m=123156441218358&w=4
> > http://marc.info/?t=123156453100002&r=1&w=4
> > 
> > 
> 
> it is sad that a patch for a regression that has been available for
> this long still has not been merged.
> 
> Sending me email over and over again does not make the maintainers
> who have the patch in their queues get off their sit-organs any
> faster....

Sorry, this is a automated thing.  Please just discard the messages about
this bug in the future.

The patch has been sent to Len and hopefully it will be merged shortly.

Thanks,
Rafael

Re: [Bug #12502] pipe_read oops on sh

[Rafael J. Wysocki]

On Sunday 15 February 2009, Adrian McMenamin wrote:
> 2009/2/14 Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12502
> > Subject         : pipe_read oops on sh
> > Submitter       : Adrian McMenamin <lkmladrian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> > Date            : 2009-01-15 9:48 (31 days old)
> > References      : http://marc.info/?l=linux-kernel&m=123201298005600&w=4
> >
> >
> >
> 
> No, it was caused by a race condition that has now been fixed.

Thanks, bug closed.

Rafael

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Rafael J. Wysocki]

On Sunday 15 February 2009, Matthew Garrett wrote:
> On Sat, Feb 14, 2009 at 09:38:23PM +0100, Rafael J. Wysocki wrote:
> 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> I think Eric sent a test patch for this. Did that get pushed?

I haven't found it.

Rafael

Re: [Bug #12615] boot hangs while bringing up gianfar ethernet

[Peter Korsgaard]

>>>>> "Rafael" == Rafael J Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> writes:

 Rafael> This message has been generated automatically as a part of a report
 Rafael> of recent regressions.

 Rafael> The following bug entry is on the current list of known
 Rafael> regressions from 2.6.28.  Please verify if it still should be
 Rafael> listed and let me know (either way).

As already mentioned, the fix works for me so I think this can be
closed.

 Rafael> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
 Rafael> Subject		: boot hangs while bringing up gianfar ethernet
 Rafael> Submitter	: Ira Snyder <iws-lulEs6mt1IksTUYHLfqkUA@public.gmane.org>
 Rafael> Date		: 2009-01-29 19:41 (17 days old)
 Rafael> References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4

-- 
Bye, Peter Korsgaard

Re: [Bug #12609] v2.6.29-rc2 libata sff 32bit PIO regression

[Hugh Dickins]

On Sun, 15 Feb 2009, Jeff Garzik wrote:
> Larry Finger wrote:
> > Rafael J. Wysocki wrote:
> > >
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12609
> > > Subject		: v2.6.29-rc2 libata sff 32bit PIO regression
> > > Submitter	: Larry Finger <Larry.Finger-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
> > > Date		: 2009-01-23 23:52 (23 days old)
> > > References	: http://marc.info/?l=linux-kernel&m=123275478111406&w=4
> > > 		  http://marc.info/?l=linux-kernel&m=123254501314058&w=4
> > > Handled-By	: Mikael Pettersson <mikpe-1zs4UD6AkMk@public.gmane.org>
> > >     Hugh Dickins <hugh-DTz5qymZ9yRBDgjK7y7TUQ@public.gmane.org>
> > >     Sergei Shtylyov <sshtylyov-hkdhdckH98+B+jHODAdFcQ@public.gmane.org>
> > > Patch		:
> > > http://marc.info/?l=linux-kernel&m=123412278730735&w=4
> > 
> > This problem is not fixed as of 2.6.29-rc5.
> 
> Sergei, what is the latest version of your patch?   According to my notes, you
> and Hugh were going back and forth trading patches, and I never saw a final
> version...

I was happy with the patch Sergei posted a week ago, and said so -
except for the question of whether to say "unlikely(slop)": and
that's a very silly little issue which should never have delayed
fixing a significant regression, I'm sorry I ever raised it.

But I was expecting Alan to gather the fixes together and send them
to you: on 26 Jan he posted two other patches, one perhaps just a
cleanup, but the other he said required for correct behaviour on
AMD devices: I thought all three patches would be in by now.

Hugh

Re: [Bug #12617] unable to compile e100 firmware into kernel

[David Woodhouse]

On Sat, 2009-02-14 at 21:38 +0100, Rafael J. Wysocki wrote:
>  This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12617
> Subject         : unable to compile e100 firmware into kernel
> Submitter       : Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
> Date            : 2009-01-31 15:59 (15 days old)
> References      : http://marc.info/?l=linux-kernel&m=123341764915181&w=4

I thought this was resolved?

-- 
dwmw2

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

> > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > which gave me this:
> > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz

> thanks. There's definitely some weirdness visible in the trace,
> for example:

>   0)    gpm-1879    =>   ksoftir-4   
>   ------------------------------------------

>   0)   0.964 us    |    finish_task_switch();
>   0) ! 1768184 us |  }
>   0)               |  do_softirq() {
>   0)               |    __do_softirq() {
>   0)               |      rcu_process_callbacks() {

> the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> ksoftirqd _really_ took that much time to execute.

> One possibility for these symptoms would be broken scheduler timestamps.
> Could you enable absolute timestamp printing via:

>     echo funcgraph-abstime  > trace_options

Mmm, seems I do not have this option recognized in rc5, so could not
test. Will retry all this with tip tomorrow...

> Also, my guess is that if you boot via idle=poll, the symptoms go away.
> This would strengthen the suspicion that it's scheduler-clock troubles.

In fact, with idle=poll, the symptoms do not go away, they are much
stronger: without it, ksotirqd have a few % of CPU in top output; with
it, they have 20 or 30% and the global average is not far from 1.

On my laptop (I do not have it at hand today), with rc3-gitX (did not
retest with rc5), the load avg was ok, but I saw that after boot,
ksoftird threads had a quite higher running time in top than with
2.6.28. I am surprised nobody reported this yet...

I attach to this mail config and dmesg if needed (this is rc5 without
idle=poll).

If a trace with funcgraph-abstime is interesting for you with tip,
I will do this tomorrow.

-- 
Damien Wyart

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 42926 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.29-rc5
# Sun Feb 15 11:07:36 2009
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
# CONFIG_CLASSIC_RCU is not set
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
# CONFIG_FAIR_GROUP_SCHED is not set
# CONFIG_RT_GROUP_SCHED is not set
# CONFIG_USER_SCHED is not set
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
# CONFIG_CGROUP_NS is not set
# CONFIG_CGROUP_FREEZER is not set
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_CPUSETS is not set
# CONFIG_CGROUP_CPUACCT is not set
# CONFIG_RESOURCE_COUNTERS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
# CONFIG_RELAY is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
# CONFIG_BLK_DEV_INITRD is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBD=y
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=m
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
# CONFIG_FREEZER is not set

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
# CONFIG_X86_RDC321X is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_CYRIX_32=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR_32=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
# CONFIG_HPET_TIMER is not set
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
# CONFIG_IOMMU_API is not set
CONFIG_NR_CPUS=2
CONFIG_SCHED_SMT=y
# CONFIG_SCHED_MC is not set
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_UNEVICTABLE_LRU=y
CONFIG_HIGHPTE=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW_64K=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
# CONFIG_ACPI_FAN is not set
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_COMPAT_NET_DEV_OPS=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_IRC=m
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_NF_CT_NETLINK is not set
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
# CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
# CONFIG_NF_NAT_SIP is not set
CONFIG_IP_NF_MANGLE=m
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
CONFIG_NET_CLS_FW=m
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
CONFIG_NET_CLS_FLOW=m
# CONFIG_NET_CLS_CGROUP is not set
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_PHONET is not set
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_BLK_DEV_HD is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
# CONFIG_SCSI_DH is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
# CONFIG_SATA_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
CONFIG_E100=y
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SMSC9420 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_NET_POCKET is not set
# CONFIG_ATL2 is not set
# CONFIG_NETDEV_1000 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
# CONFIG_SPI is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_HWMON is not set
CONFIG_THERMAL=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_REGULATOR is not set

#
# Multimedia devices
#

#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set

#
# Multimedia drivers
#
# CONFIG_DAB is not set

#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_HRTIMER is not set
CONFIG_SND_RTCTIMER=y
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_AC97_CODEC=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
# CONFIG_SND_PORTMAN2X4 is not set
# CONFIG_SND_AC97_POWER_SAVE is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
CONFIG_SND_EMU10K1=y
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SIS7019 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_USB is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set

#
# Special HID drivers
#
CONFIG_HID_COMPAT=y
CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_GYRATION=m
CONFIG_HID_LOGITECH=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_HID_NTRIG=m
CONFIG_HID_PANTHERLORD=m
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_SUNPLUS=m
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_TOPSEED=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_MON is not set
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# Enable Host or Gadget support to see Inventra options
#

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed;
#

#
# see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set
# CONFIG_X86_PLATFORM_DEVICES is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_DMIID is not set
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
CONFIG_EXT4_FS=m
CONFIG_EXT4DEV_COMPAT=y
CONFIG_EXT4_FS_XATTR=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
# CONFIG_EXT4_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=m
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_FILE_LOCKING=y
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set
CONFIG_EXPORTFS=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_HW_BRANCH_TRACER=y
CONFIG_RING_BUFFER=y
CONFIG_TRACING=y

#
# Tracers
#
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
# CONFIG_BOOT_TRACER is not set
# CONFIG_TRACE_BRANCH_PROFILING is not set
# CONFIG_POWER_TRACER is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_HW_BRANCH_TRACER is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_PRINTK_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_4KSTACKS=y
CONFIG_DOUBLEFAULT=y
# CONFIG_MMIOTRACE is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_586 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_586 is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=m
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y

[-- Attachment #3: dmesg --]
[-- Type: text/plain, Size: 23572 bytes --]

Initializing cgroup subsys cpu
Linux version 2.6.29-rc5 (root@brouette) (gcc version 4.3.3 (Debian 4.3.3-3) ) #1 SMP Sun Feb 15 11:15:54 CET 2009
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  NSC Geode by NSC
  Cyrix CyrixInstead
  Centaur CentaurHauls
  Transmeta GenuineTMx86
  Transmeta TransmetaCPU
  UMC UMC UMC UMC
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff74000 (usable)
 BIOS-e820: 000000007ff74000 - 000000007ff76000 (ACPI NVS)
 BIOS-e820: 000000007ff76000 - 000000007ff97000 (ACPI data)
 BIOS-e820: 000000007ff97000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fecf0000 - 00000000fecf1000 (reserved)
 BIOS-e820: 00000000fed20000 - 00000000fed90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
DMI 2.3 present.
last_pfn = 0x7ff74 max_arch_pfn = 0x100000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
kernel direct mapping tables up to 377fe000 @ 7000-c000
ACPI: RSDP 000FEB90, 0014 (r0 DELL  )
ACPI: RSDT 000FD1CA, 0034 (r1 DELL    8300           8 ASL        61)
ACPI: FACP 000FD1FE, 0074 (r1 DELL    8300           8 ASL        61)
FADT: X_PM1a_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
ACPI: DSDT FFFC6E75, 2426 (r1   DELL    dt_ex     1000 MSFT  100000D)
ACPI: FACS 7FF74000, 0040
ACPI: SSDT FFFC93D8, 00BA (r1   DELL    st_ex     1000 MSFT  100000D)
ACPI: APIC 000FD272, 006C (r1 DELL    8300           8 ASL        61)
ACPI: BOOT 000FD2DE, 0028 (r1 DELL    8300           8 ASL        61)
ACPI: Local APIC address 0xfee00000
1159MB HIGHMEM available.
887MB LOWMEM available.
  mapped low ram: 0 - 377fe000
  low ram: 00000000 - 377fe000
  bootmap 00008000 - 0000ef00
(8 early reservations) ==> bootmem [0000000000 - 00377fe000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000001000 - 0000002000]    EX TRAMPOLINE ==> [0000001000 - 0000002000]
  #2 [0000006000 - 0000007000]       TRAMPOLINE ==> [0000006000 - 0000007000]
  #3 [0000100000 - 0000500640]    TEXT DATA BSS ==> [0000100000 - 0000500640]
  #4 [0000501000 - 0000504000]    INIT_PG_TABLE ==> [0000501000 - 0000504000]
  #5 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
  #6 [0000007000 - 0000008000]          PGTABLE ==> [0000007000 - 0000008000]
  #7 [0000008000 - 000000f000]          BOOTMAP ==> [0000008000 - 000000f000]
found SMP MP-table at [c00fe710] 000fe710
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  Normal   0x00001000 -> 0x000377fe
  HighMem  0x000377fe -> 0x0007ff74
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x000000a0
    0: 0x00000100 -> 0x0007ff74
On node 0 totalpages: 524052
free_area_init_node: node 0, pgdat c044ed80, node_mem_map c1000000
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 3968 pages, LIFO batch:0
  Normal zone: 1744 pages used for memmap
  Normal zone: 221486 pages, LIFO batch:31
  HighMem zone: 2319 pages used for memmap
  HighMem zone: 294503 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
4 Processors exceeds NR_CPUS limit of 2
SMP: Allowing 2 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 24
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Allocating 36864 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 519957
Kernel command line: root=/dev/sdb2 ro vga=0x307 selinux=0 elevator=cfq video=vesafb:mtrr:3 
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
Experimental hierarchical RCU implementation.
Experimental hierarchical RCU init done.
CPU 0 irqstacks, hard=c04ae000 soft=c04ac000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Fast TSC calibration using PIT
Detected 2992.546 MHz processor.
Console: colour dummy device 80x25
console [tty0] enabled
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2073812k/2096592k available (2449k kernel code, 21420k reserved, 1034k data, 248k init, 1187288k highmem)
virtual kernel memory layout:
    fixmap  : 0xfff9f000 - 0xfffff000   ( 384 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf7ffe000 - 0xff7fe000   ( 120 MB)
    lowmem  : 0xc0000000 - 0xf77fe000   ( 887 MB)
      .init : 0xc046b000 - 0xc04a9000   ( 248 kB)
      .data : 0xc0364487 - 0xc0466e40   (1034 kB)
      .text : 0xc0100000 - 0xc0364487   (2449 kB)
Checking if this processor honours the WP bit even in supervisor mode...Ok.
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 5985.09 BogoMIPS (lpj=2992546)
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
using mwait in idle threads.
Checking 'hlt' instruction... OK.
ACPI: Core revision 20081204
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 13232 entries in 52 pages
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
CPU 1 irqstacks, hard=c04af000 soft=c04ad000
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5983.76 BogoMIPS (lpj=2991881)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Brought up 2 CPUs
Total of 2 processors activated (11968.85 BogoMIPS).
net_namespace: 504 bytes
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbb30, last bus=2
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:00.0: reg 10 32bit mmio: [0xe8000000-0xefffffff]
pci 0000:00:1d.0: reg 20 io port: [0xff80-0xff9f]
pci 0000:00:1d.1: reg 20 io port: [0xff60-0xff7f]
pci 0000:00:1d.2: reg 20 io port: [0xff40-0xff5f]
pci 0000:00:1d.3: reg 20 io port: [0xff20-0xff3f]
pci 0000:00:1d.7: reg 10 32bit mmio: [0xffa80800-0xffa80bff]
pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.7: PME# disabled
pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
pci 0000:00:1f.0: quirk: region 0880-08bf claimed by ICH4 GPIO
pci 0000:00:1f.1: reg 10 io port: [0x1f0-0x1f7]
pci 0000:00:1f.1: reg 14 io port: [0x3f4-0x3f7]
pci 0000:00:1f.1: reg 18 io port: [0x170-0x177]
pci 0000:00:1f.1: reg 1c io port: [0x374-0x377]
pci 0000:00:1f.1: reg 20 io port: [0xffa0-0xffaf]
pci 0000:00:1f.1: reg 24 32bit mmio: [0xfebffc00-0xfebfffff]
pci 0000:00:1f.2: reg 10 io port: [0xfe00-0xfe07]
pci 0000:00:1f.2: reg 14 io port: [0xfe10-0xfe13]
pci 0000:00:1f.2: reg 18 io port: [0xfe20-0xfe27]
pci 0000:00:1f.2: reg 1c io port: [0xfe30-0xfe33]
pci 0000:00:1f.2: reg 20 io port: [0xfea0-0xfeaf]
pci 0000:00:1f.3: reg 20 io port: [0xefe0-0xefff]
pci 0000:01:00.0: reg 10 32bit mmio: [0xfd000000-0xfdffffff]
pci 0000:01:00.0: reg 14 32bit mmio: [0xf0000000-0xf7ffffff]
pci 0000:01:00.0: reg 30 32bit mmio: [0xfea00000-0xfea1ffff]
pci 0000:00:01.0: bridge 32bit mmio: [0xfd000000-0xfeafffff]
pci 0000:00:01.0: bridge 32bit mmio pref: [0xf0000000-0xf7ffffff]
pci 0000:02:00.0: reg 10 io port: [0xdf20-0xdf3f]
pci 0000:02:00.0: supports D1 D2
pci 0000:02:00.1: reg 10 io port: [0xdf18-0xdf1f]
pci 0000:02:00.1: supports D1 D2
pci 0000:02:08.0: reg 10 32bit mmio: [0xfcfff000-0xfcffffff]
pci 0000:02:08.0: reg 14 io port: [0xdf40-0xdf7f]
pci 0000:02:08.0: supports D1 D2
pci 0000:02:08.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:02:08.0: PME# disabled
pci 0000:00:1e.0: transparent bridge
pci 0000:00:1e.0: bridge io port: [0xd000-0xdfff]
pci 0000:00:1e.0: bridge 32bit mmio: [0xfcf00000-0xfcffffff]
pci_bus 0000:00: on NUMA node 0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *4 5 6 7 9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs *3 4 5 6 7 9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 7 9 10 11 12 15)
SCSI subsystem initialized
libata version 3.00 loaded.
PCI: Using ACPI for IRQ routing
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:07: io resource (0x800-0x85f) overlaps 0000:00:1f.0 BAR 7 (0x800-0x87f), disabling
pnp 00:07: io resource (0x860-0x8ff) overlaps 0000:00:1f.0 BAR 7 (0x800-0x87f), disabling
pnp: PnP ACPI: found 8 devices
ACPI: ACPI bus type pnp unregistered
system 00:00: iomem range 0x0-0x9ffff could not be reserved
system 00:00: iomem range 0x100000-0xffffff could not be reserved
system 00:00: iomem range 0x1000000-0x7ff73fff could not be reserved
system 00:00: iomem range 0xc0000-0xfffff could not be reserved
system 00:00: iomem range 0xfec00000-0xfec0ffff has been reserved
system 00:00: iomem range 0xfee00000-0xfee0ffff has been reserved
system 00:00: iomem range 0xfed20000-0xfed8ffff has been reserved
system 00:00: iomem range 0xfecf0000-0xfecf0fff has been reserved
system 00:00: iomem range 0xffb00000-0xffbfffff has been reserved
system 00:00: iomem range 0xffc00000-0xffffffff has been reserved
system 00:07: ioport range 0xc00-0xc7f has been reserved
pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
pci 0000:00:01.0:   IO window: disabled
pci 0000:00:01.0:   MEM window: 0xfd000000-0xfeafffff
pci 0000:00:01.0:   PREFETCH window: 0x000000f0000000-0x000000f7ffffff
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:02
pci 0000:00:1e.0:   IO window: 0xd000-0xdfff
pci 0000:00:1e.0:   MEM window: 0xfcf00000-0xfcffffff
pci 0000:00:1e.0:   PREFETCH window: disabled
pci 0000:00:1e.0: setting latency timer to 64
pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffff]
pci_bus 0000:01: resource 0 mem: [0x0-0x0]
pci_bus 0000:01: resource 1 mem: [0xfd000000-0xfeafffff]
pci_bus 0000:01: resource 2 mem: [0xf0000000-0xf7ffffff]
pci_bus 0000:01: resource 3 mem: [0x0-0x0]
pci_bus 0000:02: resource 0 io:  [0xd000-0xdfff]
pci_bus 0000:02: resource 1 mem: [0xfcf00000-0xfcffffff]
pci_bus 0000:02: resource 2 mem: [0x0-0x0]
pci_bus 0000:02: resource 3 io:  [0x00-0xffff]
pci_bus 0000:02: resource 4 mem: [0x000000-0xffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
SGI XFS with security attributes, large block/inode numbers, no debug enabled
msgmni has been set to 1733
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
pci 0000:01:00.0: Boot video device
pci 0000:02:08.0: Firmware left e100 interrupts enabled; disabling
vesafb: framebuffer at 0xf0000000, mapped to 0xf8080000, using 2560k, total 131072k
vesafb: mode is 1280x1024x8, linelength=1280, pages=1
vesafb: protected mode interface info at c000:f080
vesafb: pmi: set display start = c00cf0b6, set palette = c00cf120
vesafb: pmi: ports = 3b4 3b5 3ba 3c0 3c1 3c4 3c5 3c6 3c7 3c8 3c9 3cc 3ce 3cf 3d0 3d1 3d2 3d3 3d4 3d5 3da 
vesafb: scrolling: redraw
vesafb: Pseudocolor: size=8:8:8:8, shift=0:0:0:0
Console: switching to colour frame buffer device 160x64
fb0: VESA VGA frame buffer device
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input1
ACPI: Power Button (CM) [VBTN]
processor ACPI_CPU:00: registered as cooling_device0
processor ACPI_CPU:01: registered as cooling_device1
Real Time Clock Driver v1.12b
e100: Intel(R) PRO/100 Network Driver, 3.5.23-k6-NAPI
e100: Copyright(c) 1999-2006 Intel Corporation
e100 0000:02:08.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
e100 0000:02:08.0: PME# disabled
e100: eth0: e100_probe: addr 0xfcfff000, irq 20, MAC addr 00:0c:f1:b6:ba:54
Driver 'sd' needs updating - please use bus_type methods
ata_piix 0000:00:1f.1: version 2.12
ata_piix 0000:00:1f.1: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ata_piix 0000:00:1f.1: setting latency timer to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
ata1: port disabled. ignoring.
ata2.00: ATAPI: SAMSUNG DVD-ROM SD-616T, F310, max UDMA/33
ata2.01: ATAPI: SAMSUNG CD-R/RW SW-252S, R901, max UDMA/33
ata2.00: configured for UDMA/33
ata2.01: configured for UDMA/33
scsi 1:0:0:0: CD-ROM            SAMSUNG  DVD-ROM SD-616T  F310 PQ: 0 ANSI: 5
scsi 1:0:0:0: Attached scsi generic sg0 type 5
scsi 1:0:1:0: CD-ROM            SAMSUNG  CD-R/RW SW-252S  R901 PQ: 0 ANSI: 5
scsi 1:0:1:0: Attached scsi generic sg1 type 5
ata_piix 0000:00:1f.2: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ata_piix 0000:00:1f.2: MAP [ P0 -- P1 -- ]
ata_piix 0000:00:1f.2: setting latency timer to 64
scsi2 : ata_piix
scsi3 : ata_piix
ata3: SATA max UDMA/133 cmd 0xfe00 ctl 0xfe10 bmdma 0xfea0 irq 18
ata4: SATA max UDMA/133 cmd 0xfe20 ctl 0xfe30 bmdma 0xfea8 irq 18
ata3.00: ATA-6: WDC WD740GD-75FLA0, 21.08U21, max UDMA/133
ata3.00: 144531250 sectors, multi 8: LBA48 
ata3.00: applying bridge limits
ata3.00: configured for UDMA/100
scsi 2:0:0:0: Direct-Access     ATA      WDC WD740GD-75FL 21.0 PQ: 0 ANSI: 5
sd 2:0:0:0: [sda] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sda] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 < sda5 >
sd 2:0:0:0: [sda] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg2 type 0
Switched to high resolution mode on CPU 1
Switched to high resolution mode on CPU 0
ata4.00: ATA-6: WDC WD740GD-75FLA0, 21.08U21, max UDMA/133
ata4.00: 144531250 sectors, multi 8: LBA48 
ata4.00: applying bridge limits
ata4.00: configured for UDMA/100
scsi 3:0:0:0: Direct-Access     ATA      WDC WD740GD-75FL 21.0 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdb] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdb] 144531250 512-byte hardware sectors: (74.0 GB/68.9 GiB)
sd 3:0:0:0: [sdb] Write Protect is off
sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2 sdb3 < sdb5 sdb6 sdb7 sdb8 >
sd 3:0:0:0: [sdb] Attached SCSI disk
sd 3:0:0:0: Attached scsi generic sg3 type 0
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
cpuidle: using governor ladder
cpuidle: using governor menu
Advanced Linux Sound Architecture Driver Version 1.0.18a.
EMU10K1_Audigy 0000:02:00.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
Control name 'Sigmatel Surround Phase Inversion Playback Switch' truncated to 'Sigmatel Surround Phase Inversion Playback '
ALSA device list:
  #0: SB Live! [Unknown] (rev.10, serial:0x80671102) at 0xdf20, irq 21
TCP cubic registered
NET: Registered protocol family 17
Using IPI No-Shortcut mode
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly on device 8:18.
Freeing unused kernel memory: 248k freed
parport_pc 00:06: reported by Plug and Play ACPI
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
uhci_hcd: USB Universal Host Controller Interface driver
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.0: setting latency timer to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 16, io base 0x0000ff80
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1d.1: setting latency timer to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000ff60
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: setting latency timer to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000ff40
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.3: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.3: setting latency timer to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.3: irq 16, io base 0x0000ff20
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
Driver 'sr' needs updating - please use bus_type methods
sr0: scsi3-mmc drive: 1x/48x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Warning! ehci_hcd should always be loaded before uhci_hcd and ohci_hcd, not after
sr1: scsi3-mmc drive: 48x/16x writer cd/rw xa/form2 cdda tray
sr 1:0:1:0: Attached scsi CD-ROM sr1
ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.7: setting latency timer to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: cache line size of 128 is not supported
ehci_hcd 0000:00:1d.7: irq 23, io mem 0xffa80800
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
usb usb5: configuration #1 chosen from 1 choice
input: PC Speaker as /devices/platform/pcspkr/input/input2
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
usb 1-2: new full speed USB device using uhci_hcd and address 2
Adding 1212896k swap on /dev/sdb7.  Priority:-1 extents:1 across:1212896k 
usb 1-2: configuration #1 chosen from 1 choice
usb 2-1: new low speed USB device using uhci_hcd and address 2
EXT3 FS on sdb2, internal journal
usb 2-1: configuration #1 chosen from 1 choice
usb 3-2: new low speed USB device using uhci_hcd and address 2
generic-usb 0003:056D:0002.0001: claimed by neither input, hiddev nor hidraw
usb 3-2: configuration #1 chosen from 1 choice
input: HID 05af:0310 as /devices/pci0000:00/0000:00:1d.1/usb2/2-1/2-1:1.0/input/input3
generic-usb 0003:05AF:0310.0002: input: USB HID v1.10 Keyboard [HID 05af:0310] on usb-0000:00:1d.1-1/input0
input: HID 05af:0310 as /devices/pci0000:00/0000:00:1d.1/usb2/2-1/2-1:1.1/input/input4
generic-usb 0003:05AF:0310.0003: input: USB HID v1.10 Device [HID 05af:0310] on usb-0000:00:1d.1-1/input1
input: Logitech USB Receiver as /devices/pci0000:00/0000:00:1d.2/usb3/3-2/3-2:1.0/input/input5
generic-usb 0003:046D:C51B.0004: input: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:1d.2-2/input0
generic-usb 0003:046D:C51B.0005: claimed by neither input, hiddev nor hidraw
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
XFS mounting filesystem sdb5
Ending clean XFS mount for filesystem: sdb5
XFS mounting filesystem sdb6
Ending clean XFS mount for filesystem: sdb6
XFS mounting filesystem sdb8
Ending clean XFS mount for filesystem: sdb8
XFS mounting filesystem sda5
Ending clean XFS mount for filesystem: sda5
e100: eth0 NIC Link is Up 100 Mbps Full Duplex
lp0: using parport0 (interrupt-driven).
warning: `ntpd' uses 32-bit capabilities (legacy support in use)
CPU 1 is now offline
SMP alternatives: switching to UP code
SMP alternatives: switching to SMP code
CPU 1 irqstacks, hard=c04af000 soft=c04ad000
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5984.41 BogoMIPS (lpj=2992209)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 03
checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Switched to high resolution mode on CPU 1
Warning: Processor Platform Limit event detected, but not handled.
Consider compiling CPUfreq support into your kernel.

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > Also, my guess is that if you boot via idle=poll, the symptoms go away.
> > This would strengthen the suspicion that it's scheduler-clock troubles.

> In fact, with idle=poll, the symptoms do not go away, they are much
> stronger: without it, ksotirqd have a few % of CPU in top output; with
> it, they have 20 or 30% and the global average is not far from 1.

Not true each time, in fact. Another try with this option gave same
symptoms as without it, not stronger. In any case, they never go away.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> > > So I followed the tracing steps in the tutorial (with the 1 sec sleep),
> > > which gave me this:
> > > http://damien.wyart.free.fr/trace_2.6.29-rc5_ksoftirqd_prob.txt.gz
> 
> > thanks. There's definitely some weirdness visible in the trace,
> > for example:
> 
> >   0)    gpm-1879    =>   ksoftir-4   
> >   ------------------------------------------
> 
> >   0)   0.964 us    |    finish_task_switch();
> >   0) ! 1768184 us |  }
> >   0)               |  do_softirq() {
> >   0)               |    __do_softirq() {
> >   0)               |      rcu_process_callbacks() {
> 
> > the 1.7 seconds 'overhead' there must be a fluke - you'd notice it if
> > ksoftirqd _really_ took that much time to execute.
> 
> > One possibility for these symptoms would be broken scheduler timestamps.
> > Could you enable absolute timestamp printing via:
> 
> >     echo funcgraph-abstime  > trace_options
> 
> Mmm, seems I do not have this option recognized in rc5, so could not
> test. Will retry all this with tip tomorrow...

Yeah, it got renamed in -tip - in rc5 it's iter_ctrl.

> > Also, my guess is that if you boot via idle=poll, the symptoms go away.
> > This would strengthen the suspicion that it's scheduler-clock troubles.
> 
> In fact, with idle=poll, the symptoms do not go away, they are much
> stronger: without it, ksotirqd have a few % of CPU in top output; with
> it, they have 20 or 30% and the global average is not far from 1.
> 
> On my laptop (I do not have it at hand today), with rc3-gitX (did not
> retest with rc5), the load avg was ok, but I saw that after boot,
> ksoftird threads had a quite higher running time in top than with
> 2.6.28. I am surprised nobody reported this yet...
> 
> I attach to this mail config and dmesg if needed (this is rc5 without
> idle=poll).
> 
> If a trace with funcgraph-abstime is interesting for you with tip,
> I will do this tomorrow.

Yes, an abstime trace would be useful.

> checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> Switched to high resolution mode on CPU 1

Lets double-check your scheduler clock first. Without being able to
trust the clock we cannot trust the task stats nor the trace output.

What does this check display:

  http://people.redhat.com/mingo/time-warp-test/time-warp-test.c

Does it find any TSC time warps?

Also, could you send the output of:

  http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

Run it while you can see the ksoftirqd anomaly.

	Ingo

Re: [Bug #12617] unable to compile e100 firmware into kernel

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1684 bytes --]

On 15 of February 2009 20:38:15 David Woodhouse wrote:
> On Sat, 2009-02-14 at 21:38 +0100, Rafael J. Wysocki wrote:
> >  This message has been generated automatically as a part of a
> > report of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me
> > know (either way).
> >
> >
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12617
> > Subject         : unable to compile e100 firmware into kernel
> > Submitter       : Andrey Borzenkov <arvidjaar@mail.ru>
> > Date            : 2009-01-31 15:59 (15 days old)
> > References      :
> > http://marc.info/?l=linux-kernel&m=123341764915181&w=4
>
> I thought this was resolved?

I guess it has to be closed. This was caused by config options mismatch 
and I do not see how we can programmatically express "this device could 
be required before any (even early) user space is available, so make 
sure everything that this device needs is built in".

One possibility is following:

--- drivers/base/Kconfig.orig 2009-01-17 14:46:28.000000000 +0300
+++ drivers/base/Kconfig        2009-02-15 22:57:02.000000000 +0300
@@ -37,7 +37,7 @@ config FW_LOADER

 config FIRMWARE_IN_KERNEL
        bool "Include in-kernel firmware blobs in kernel binary"
-       depends on FW_LOADER
+       depends on FW_LOADER = y
        default y
        help
          The kernel source tree includes a number of firmware 'blobs'

On assumption that compiling firmware in kernel makes no sense as long 
as kernel does not even have facility to actually access it.

Does it make sense?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Bug #12615] boot hangs while bringing up gianfar ethernet

[Rafael J. Wysocki]

On Sunday 15 February 2009, Peter Korsgaard wrote:
> >>>>> "Rafael" == Rafael J Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> writes:
> 
>  Rafael> This message has been generated automatically as a part of a report
>  Rafael> of recent regressions.
> 
>  Rafael> The following bug entry is on the current list of known
>  Rafael> regressions from 2.6.28.  Please verify if it still should be
>  Rafael> listed and let me know (either way).
> 
> As already mentioned, the fix works for me so I think this can be
> closed.
> 
>  Rafael> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
>  Rafael> Subject		: boot hangs while bringing up gianfar ethernet
>  Rafael> Submitter	: Ira Snyder <iws-lulEs6mt1IksTUYHLfqkUA@public.gmane.org>
>  Rafael> Date		: 2009-01-29 19:41 (17 days old)
>  Rafael> References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4

Thanks, closed.

Rafael

Re: [Bug #12617] unable to compile e100 firmware into kernel

[Rafael J. Wysocki]

On Sunday 15 February 2009, Andrey Borzenkov wrote:
> On 15 of February 2009 20:38:15 David Woodhouse wrote:
> > On Sat, 2009-02-14 at 21:38 +0100, Rafael J. Wysocki wrote:
> > >  This message has been generated automatically as a part of a
> > > report of recent regressions.
> > >
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.28.  Please verify if it still should be listed and let me
> > > know (either way).
> > >
> > >
> > > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=12617
> > > Subject         : unable to compile e100 firmware into kernel
> > > Submitter       : Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
> > > Date            : 2009-01-31 15:59 (15 days old)
> > > References      :
> > > http://marc.info/?l=linux-kernel&m=123341764915181&w=4
> >
> > I thought this was resolved?
> 
> I guess it has to be closed. This was caused by config options mismatch 
> and I do not see how we can programmatically express "this device could 
> be required before any (even early) user space is available, so make 
> sure everything that this device needs is built in".
> 
> One possibility is following:
> 
> --- drivers/base/Kconfig.orig 2009-01-17 14:46:28.000000000 +0300
> +++ drivers/base/Kconfig        2009-02-15 22:57:02.000000000 +0300
> @@ -37,7 +37,7 @@ config FW_LOADER
> 
>  config FIRMWARE_IN_KERNEL
>         bool "Include in-kernel firmware blobs in kernel binary"
> -       depends on FW_LOADER
> +       depends on FW_LOADER = y
>         default y
>         help
>           The kernel source tree includes a number of firmware 'blobs'
> 
> On assumption that compiling firmware in kernel makes no sense as long 
> as kernel does not even have facility to actually access it.
> 
> Does it make sense?

Guess so.

I've closed the bug.

Thanks,
Rafael

Re: [Bug #12419] possible circular locking dependency on i915 dma

[Wang Chen]

Rafael J. Wysocki said the following on 2009-2-15 4:38:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12419
> Subject		: possible circular locking dependency on i915 dma
> Submitter	: Wang Chen <wangchen-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
> Date		: 2009-01-08 14:11 (38 days old)
> References	: http://marc.info/?l=linux-kernel&m=123142399720125&w=4
> 
> 

Yes. It's still there in mainline.
I think the commit 546b0974c39657017407c86fe79811100b60700d
"i915: Use struct_mutex to protect ring in GEM mode." brought this regression.

The lockdep problem is as following:
thread-1
i915_cmdbuffer()
      |
      ---> lock(drm_device->struct_mutex)
                   |
		   V
	i915_dispatch_cmdbuffer()
		   |
		   ---->i915_emit_box()
                             |
                             ----->copy_from_user()
					|
					-----might_fault()
						|
						--->lock(mm->mmap_sem)

thread-2
dup_mm()
   |
   --->lock(mm->mmap_sem)
           |
	   V
	drm_vm_open()
	   |
	   -------> lock(drm_device->struct_mutex)

The different order to lock "mmap_sem" and "drm_dev->struct_mutex" introduces the problem.
But it seems no way to reverse the lock order in i915.
So how about refine the lock granularity of drm_dev->struct_mutex and exclude the mmap_sem
lock/unlock out of the drm_dev->struct_mutex lock/unlock range?

Re: 2.6.29-rc5: Reported regressions from 2.6.28

[Jarek Poplawski]

On 14-02-2009 21:35, Rafael J. Wysocki wrote:
> This message contains a list of some regressions from 2.6.28, for which there
> are no fixes in the mainline I know of.  If any of them have been fixed already,
> please let me know.
...
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
> Subject		: boot hangs while bringing up gianfar ethernet
> Submitter	: Ira Snyder <iws@ovro.caltech.edu>
> Date		: 2009-01-29 19:41 (17 days old)
> References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4

Fixed by:
commit	8707bdd48ab705a459ac1b12014075a139d1d4f9
gianfar: Fix boot hangs while bringing up gianfar ethernet
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=8707bdd48ab705a459ac1b12014075a139d1d4f9

Cheers,
Jarek P.

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello,

Ok, I've redone the tests under tip from this morning (Paris time).
Everything is in http://damien.wyart.free.fr/ksoftirqd_pb/

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [2009-02-15 20:31]:
> Yes, an abstime trace would be useful.

The corresponding file is trace_tip_2009.02.16_ksoftirqd_pb_abstime.txt.gz
and there is also a trace without abstime:
trace_tip_2009.02.16_ksoftirqd_pb.txt.gz

> > checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> > Switched to high resolution mode on CPU 1

> Lets double-check your scheduler clock first. Without being able to
> trust the clock we cannot trust the task stats nor the trace output.

> What does this check display:

>   http://people.redhat.com/mingo/time-warp-test/time-warp-test.c

The file is time-warp-test_result.txt
I've let it run for a few tens of minutes; the first number varies
slightly sometimes. The second one stays at 0.

> Does it find any TSC time warps?

Seems not.

> Also, could you send the output of:

>   http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

> Run it while you can see the ksoftirqd anomaly.

In fact I see it all the time when the machine is idle. When something
runs (spamd for example), the running time of ksoftirqd stops
increasing, and it goes back to increasing like crazy when idle state
comes back.

The corresponding file is cfs-debug-info-2009.02.16-08.09.17.gz


Hope this will be useful; do not hesitate to ask for further info. Now
that I have tip, I guess it will be easier.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> > Does it find any TSC time warps?
> 
> Seems not.

ok, it's indeed pretty conclusive:

|  2 CPUs, running 2 parallel test-tasks.
|  checking for time-warps via:
|  - read time stamp counter (RDTSC) instruction (cycle resolution)
| 
|   | TSC: 0.32us, fail:0 |

i suspect you ran it for at least a couple of minutes, and the ksoftirqd 
anomaly occured during it, right?

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> > Also, could you send the output of:
> 
> >   http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
> 
> > Run it while you can see the ksoftirqd anomaly.
> 
> In fact I see it all the time when the machine is idle. When something 
> runs (spamd for example), the running time of ksoftirqd stops 
> increasing, and it goes back to increasing like crazy when idle state 
> comes back.

here's the ksoftirqd stats from your data:

top - 08:09:20 up 2 min,  1 user,  load average: 0.95, 0.35, 0.13
Tasks: 100 total,   1 running,  99 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.3%sy,  0.0%ni, 89.2%id,  0.0%wa,  0.0%hi,  9.0%si,  0.0%st
Mem:   2074980k total,   236544k used,  1838436k free,     5032k buffers
Swap:  1212896k total,        0k used,  1212896k free,    66016k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2280 root      15  -5     0    0    0 S   22  0.0   0:06.39 [ksoftirqd/1]      
 2452 root      20   0  2400 1132  888 R    1  0.1   0:00.02 top -H -c -b -d 1 -
    1 root      20   0  2100  688  588 S    0  0.0   0:00.48 init [2]           
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kthreadd]         

ksoftirqd really seems to be active (according to scheduler stats), and 
has gathered more than 6 seconds of runtime after 2 minutes of uptime - 
that is clearly anomalous.

The main method how softirqs get generated are hardirqs, but the hardirq 
is low on your box (as expected):

-- interrupts: --
           CPU0       CPU1       
  0:        155          0   IO-APIC-edge      timer
  1:          2          0   IO-APIC-edge      i8042
  7:          0          0   IO-APIC-edge      parport0
  8:          2          0   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          4          0   IO-APIC-edge      i8042
 14:          0          0   IO-APIC-edge      ata_piix
 15:        132          0   IO-APIC-edge      ata_piix
 16:        213          0   IO-APIC-fasteoi   uhci_hcd:usb2, uhci_hcd:usb5
 18:       6908          0   IO-APIC-fasteoi   ata_piix, uhci_hcd:usb4
 19:        221          0   IO-APIC-fasteoi   uhci_hcd:usb3
 20:       1072          0   IO-APIC-fasteoi   eth0
 21:          0          0   IO-APIC-fasteoi   EMU10K1
 23:          4          0   IO-APIC-fasteoi   ehci_hcd:usb1
NMI:          0          0   Non-maskable interrupts
LOC:      79306      74184   Local timer interrupts
CNT:          0          0   Performance counter interrupts
RES:      32672      13541   Rescheduling interrupts
CAL:          8         40   Function call interrupts
TLB:        235        591   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0

about 200K irqs - most of which are lapic timer irqs. Thus 6 seconds of 
ksoftirqd runtime is way too much.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

scheduler clock deltas look healthy at first sight:

$ grep clock-delta cfs-debug-info-2009.02.16-08.09.17

clock-delta                        :                  285
clock-delta                        :                  313
clock-delta                        :                  286
clock-delta                        :                  311
clock-delta                        :                  348
clock-delta                        :                  301
clock-delta                        :                  386
clock-delta                        :                  318
clock-delta                        :                  275
clock-delta                        :                  283
clock-delta                        :                  343
clock-delta                        :                  341
clock-delta                        :                  318
clock-delta                        :                  295
clock-delta                        :                  330
clock-delta                        :                  311
clock-delta                        :                  286
clock-delta                        :                  333
clock-delta                        :                  309
clock-delta                        :                  369
clock-delta                        :                 1102
clock-delta                        :                  323
clock-delta                        :                  283
clock-delta                        :                  333
clock-delta                        :                  316
clock-delta                        :                  300
clock-delta                        :                  278
clock-delta                        :                  286
clock-delta                        :                  364
clock-delta                        :                  298
clock-delta                        :                  326
clock-delta                        :                  353
clock-delta                        :                  323
clock-delta                        :                  286
clock-delta                        :                  286
clock-delta                        :                  256
clock-delta                        :                  343
clock-delta                        :                  305
clock-delta                        :                  363
clock-delta                        :                  283
clock-delta                        :                  280
clock-delta                        :                  301
clock-delta                        :                  283
clock-delta                        :                  308
clock-delta                        :                  316
clock-delta                        :                  315
clock-delta                        :                  318
clock-delta                        :                  311
clock-delta                        :                  298
clock-delta                        :                  343
clock-delta                        :                  276
clock-delta                        :                  324
clock-delta                        :                  285
clock-delta                        :                  268
clock-delta                        :                  281
clock-delta                        :                  280
clock-delta                        :                  338
clock-delta                        :                 1060
clock-delta                        :                  315
clock-delta                        :                  315
clock-delta                        :                  298
clock-delta                        :                  308
clock-delta                        :                  281
clock-delta                        :                  323
clock-delta                        :                  321
clock-delta                        :                  389
clock-delta                        :                  288
clock-delta                        :                  355
clock-delta                        :                  295
clock-delta                        :                  284
clock-delta                        :                 1037
clock-delta                        :                  298
clock-delta                        :                  348
clock-delta                        :                  328
clock-delta                        :                  305
clock-delta                        :                  358
clock-delta                        :                  340
clock-delta                        :                  406
clock-delta                        :                  323
clock-delta                        :                  306
clock-delta                        :                  295
clock-delta                        :                  294
clock-delta                        :                  288
clock-delta                        :                  363
clock-delta                        :                  296
clock-delta                        :                  321
clock-delta                        :                  278
clock-delta                        :                  338
clock-delta                        :                  291
clock-delta                        :                  318
clock-delta                        :                  331
clock-delta                        :                  305
clock-delta                        :                  353
clock-delta                        :                  319
clock-delta                        :                  338
clock-delta                        :                  286
clock-delta                        :                  309
clock-delta                        :                  298
clock-delta                        :                  316

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

Ok, the ksoftirqd/1 stats contain a smoking gun:

ksoftirqd/1 (2280, #threads: 1)
---------------------------------------------------------
se.exec_start                      :        152642.614531
se.vruntime                        :         57051.648140
se.sum_exec_runtime                :          5741.506722
se.avg_overlap                     :             0.000000
se.avg_wakeup                      :            10.000000
se.wait_start                      :             0.000000
se.sleep_start                     :        152642.623948
se.block_start                     :             0.000000
se.sleep_max                       :         22662.455146
se.block_max                       :          4096.496408
se.exec_max                        :             0.016849
se.slice_max                       :             0.000000
se.wait_max                        :             0.053862
se.wait_sum                        :          7899.023463
se.wait_count                      :              4235574
sched_info.bkl_count               :                    0
se.nr_migrations                   :                    1
se.nr_migrations_cold              :                    0
se.nr_failed_migrations_affine     :                   32
se.nr_failed_migrations_running    :                   47
se.nr_failed_migrations_hot        :                   23
se.nr_forced_migrations            :                    0
se.nr_forced2_migrations           :                    5
se.nr_wakeups                      :              4235539
se.nr_wakeups_sync                 :                    0
se.nr_wakeups_migrate              :                   11
se.nr_wakeups_local                :              4235506
se.nr_wakeups_remote               :                   34
se.nr_wakeups_affine               :                    4
se.nr_wakeups_affine_attempts      :                   44
se.nr_wakeups_passive              :                    7
se.nr_wakeups_idle                 :                    0
avg_atom                           :             0.001355
avg_per_cpu                        :          5741.514564
nr_switches                        :              4235404
nr_voluntary_switches              :              4235389
nr_involuntary_switches            :                   18
se.load.weight                     :                 3121
policy                             :                    0
prio                               :                  115
clock-delta                        :                  295

these bits:

 se.sum_exec_runtime                :          5741.506722
 nr_switches                        :              4235404
 nr_voluntary_switches              :              4235389
 nr_involuntary_switches            :                   18

mean that ksoftirqd _really_ ran more than 4 million times since bootup 
- that is _highly_ anomalous. It means that scheduler clock is fine, and 
that your box is really running a lot of softirq workload.

Here is how it should look like normally. A 16-way testbox with almost 
an hour of uptime, running high load with a lot of networking. Its CPU#0 
ksoftirqd [the busiest one] has these stats:

 se.sum_exec_runtime                :             0.523552
 nr_switches                        :                   42
 nr_voluntary_switches              :                   42
 nr_involuntary_switches            :                    0

it ran only 42 times. That is a normal ksoftirqd pattern.

I'll check your traces as the next step.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> Hello,
> 
> Ok, I've redone the tests under tip from this morning (Paris time).
> Everything is in http://damien.wyart.free.fr/ksoftirqd_pb/
> 
> * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [2009-02-15 20:31]:
> > Yes, an abstime trace would be useful.
> 
> The corresponding file is trace_tip_2009.02.16_ksoftirqd_pb_abstime.txt.gz
> and there is also a trace without abstime:
> trace_tip_2009.02.16_ksoftirqd_pb.txt.gz

hm, we need a trace with both abstime and process information included:

  echo funcgraph-proc     > trace_options
  echo funcgraph-abstime  > trace_options

Also, at 140 msecs the duration is a bit short - could you please make a 
1-2 seconds capture? You can do that by increasing the number in 
buffer_size_kb 10-fold:

   echo 14100 > buffer_size_kb 

(your defaults might be different)

You can see the duration of the trace by looking at the first timestamp 
and the last one:

  310.846260 |   0)   2.380 us    |                            }
  [...]
  457.712729 |   1)               |        dequeue_task() {

Hm ... even with this limited trace, there's an unusually high amount of 
RCU activities. Each activity goes like this:

  457.680976 |   1)               |  do_softirq() {
  457.680976 |   1)               |    __do_softirq() {
  457.680977 |   1)               |      rcu_process_callbacks() {
  457.680977 |   1)               |        __rcu_process_callbacks() {
4  457.680978 |   1)   0.478 us    |          force_quiescent_state();
  457.680979 |   1)   1.591 us    |        }
  457.680979 |   1)               |        __rcu_process_callbacks() {
  457.680980 |   1)   0.478 us    |          force_quiescent_state();
  457.680981 |   1)               |          cpu_quiet() {
  457.680981 |   1)   0.506 us    |            _spin_lock_irqsave();
  457.680982 |   1)   0.496 us    |            _spin_unlock_irqrestore();
  457.680983 |   1)   2.545 us    |          }
  457.680984 |   1)   4.626 us    |        }
  457.680985 |   1)   7.823 us    |      }
  457.680985 |   1)   0.496 us    |      _local_bh_enable();
  457.680986 |   1)   9.845 us    |    }
  457.680987 |   1) + 10.962 us   |  }

I've Cc:-ed Paul, as you have tree-RCU enabled, which is a new feature 
in v2.6.29:

 #
 # RCU Subsystem
 #
 # CONFIG_CLASSIC_RCU is not set
 CONFIG_TREE_RCU=y
 # CONFIG_PREEMPT_RCU is not set
 # CONFIG_RCU_TRACE is not set
 CONFIG_RCU_FANOUT=32
 # CONFIG_RCU_FANOUT_EXACT is not set
 # CONFIG_TREE_RCU_TRACE is not set
 # CONFIG_PREEMPT_RCU_TRACE is not set

Damien, as an experiment, if you change your config to 
CONFIG_CLASSIC_RCU=y, does the ksoftirqd problem go away?

On the other hand ... the softirq processing there looks anomalous in 
itself, and might be due to some compiler bug perhaps. Could you try the 
debug patch below please (you'll get it automatically if you update to 
latest -tip), and redo the trace - the ftrace_printk() info should now 
be embedded in the trace. The expected result would be for the printed 
ot value to be non-zero at the #1 point, and zero at the #2 point.

Note: if it's a compiler optimization bug then this patch might make the 
whole problem go away.

	Ingo

------------------>
From 2d7cf65eec92937bff1073311f6843aa7189bff2 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
Date: Mon, 16 Feb 2009 10:48:37 +0100
Subject: [PATCH] softirq: debug

Signed-off-by: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
---
 kernel/softirq.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 3dd0d13..110cad0 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -196,7 +196,9 @@ asmlinkage void __do_softirq(void)
 	cpu = smp_processor_id();
 restart:
 	/* Reset the pending bitmask before enabling irqs */
+	ftrace_printk("#1 softirq pending: %08x\n", local_softirq_pending());
 	set_softirq_pending(0);
+	ftrace_printk("#2 softirq pending: %08x\n", local_softirq_pending());
 
 	local_irq_enable();
 

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

> > > Does it find any TSC time warps?

> > Seems not.

> ok, it's indeed pretty conclusive:

> |  2 CPUs, running 2 parallel test-tasks.
> |  checking for time-warps via:
> |  - read time stamp counter (RDTSC) instruction (cycle resolution)
> | 
> |   | TSC: 0.32us, fail:0 |

> i suspect you ran it for at least a couple of minutes, and the ksoftirqd 
> anomaly occured during it, right?

Yes, that's right. I ran it for a few ten of minutes and the ksoftirqd
anomaly was occuring.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 10:50]:
> hm, we need a trace with both abstime and process information included:

>   echo funcgraph-proc     > trace_options
>   echo funcgraph-abstime  > trace_options

> Also, at 140 msecs the duration is a bit short - could you please make a 
> 1-2 seconds capture? You can do that by increasing the number in 
> buffer_size_kb 10-fold:

>    echo 14100 > buffer_size_kb 

Ok, I've redone a trace with these options enabled. The file is here:
http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz

> Hm ... even with this limited trace, there's an unusually high amount of 
> RCU activities. Each activity goes like this:

>   457.680976 |   1)               |  do_softirq() {
>   457.680976 |   1)               |    __do_softirq() {
>   457.680977 |   1)               |      rcu_process_callbacks() {
>   457.680977 |   1)               |        __rcu_process_callbacks() {
> 4  457.680978 |   1)   0.478 us    |          force_quiescent_state();
>   457.680979 |   1)   1.591 us    |        }
>   457.680979 |   1)               |        __rcu_process_callbacks() {
>   457.680980 |   1)   0.478 us    |          force_quiescent_state();
>   457.680981 |   1)               |          cpu_quiet() {
>   457.680981 |   1)   0.506 us    |            _spin_lock_irqsave();
>   457.680982 |   1)   0.496 us    |            _spin_unlock_irqrestore();
>   457.680983 |   1)   2.545 us    |          }
>   457.680984 |   1)   4.626 us    |        }
>   457.680985 |   1)   7.823 us    |      }
>   457.680985 |   1)   0.496 us    |      _local_bh_enable();
>   457.680986 |   1)   9.845 us    |    }
>   457.680987 |   1) + 10.962 us   |  }

> I've Cc:-ed Paul, as you have tree-RCU enabled, which is a new feature 
> in v2.6.29:

>  #
>  # RCU Subsystem
>  #
>  # CONFIG_CLASSIC_RCU is not set
>  CONFIG_TREE_RCU=y
>  # CONFIG_PREEMPT_RCU is not set
>  # CONFIG_RCU_TRACE is not set
>  CONFIG_RCU_FANOUT=32
>  # CONFIG_RCU_FANOUT_EXACT is not set
>  # CONFIG_TREE_RCU_TRACE is not set
>  # CONFIG_PREEMPT_RCU_TRACE is not set

> Damien, as an experiment, if you change your config to 
> CONFIG_CLASSIC_RCU=y, does the ksoftirqd problem go away?

Yes, in that case the problem completely goes away. Nice catch!

> On the other hand ... the softirq processing there looks anomalous in
> itself, and might be due to some compiler bug perhaps. Could you try
> the debug patch below please (you'll get it automatically if you
> update to latest -tip), and redo the trace - the ftrace_printk() info
> should now be embedded in the trace. The expected result would be for
> the printed ot value to be non-zero at the #1 point, and zero at the
> #2 point.

> Note: if it's a compiler optimization bug then this patch might make the 
> whole problem go away.

> 	Ingo

> ------------------>
> >From 2d7cf65eec92937bff1073311f6843aa7189bff2 Mon Sep 17 00:00:00 2001
> From: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
> Date: Mon, 16 Feb 2009 10:48:37 +0100
> Subject: [PATCH] softirq: debug

> Signed-off-by: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
> ---
>  kernel/softirq.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)

> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index 3dd0d13..110cad0 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -196,7 +196,9 @@ asmlinkage void __do_softirq(void)
>  	cpu = smp_processor_id();
>  restart:
>  	/* Reset the pending bitmask before enabling irqs */
> +	ftrace_printk("#1 softirq pending: %08x\n", local_softirq_pending());
>  	set_softirq_pending(0);
> +	ftrace_printk("#2 softirq pending: %08x\n", local_softirq_pending());

>  	local_irq_enable();

I've done my new trace after recompiling very latest tip, so this should
have been taken into account.

So this seems RCU-related... If more tests are needed, do not hesitate
to ask. Btw, on the laptop which also shows ksoftirqd problems, I also
have CONFIG_TREE_RCU=y.

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 10:50]:
> > hm, we need a trace with both abstime and process information included:
> 
> >   echo funcgraph-proc     > trace_options
> >   echo funcgraph-abstime  > trace_options
> 
> > Also, at 140 msecs the duration is a bit short - could you please make a 
> > 1-2 seconds capture? You can do that by increasing the number in 
> > buffer_size_kb 10-fold:
> 
> >    echo 14100 > buffer_size_kb 
> 
> Ok, I've redone a trace with these options enabled. The file is here:
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz

ok, here's the new annotated trace:

  799.555279 |   1)  ksoftir-2324  |               |  do_softirq() {
  799.555279 |   1)  ksoftir-2324  |               |    __do_softirq() {
  799.555280 |   1)  ksoftir-2324  |               |      /* #1 softirq pending: 00000100 */
  799.555281 |   1)  ksoftir-2324  |               |      /* #2 softirq pending: 00000000 */
  799.555282 |   1)  ksoftir-2324  |               |      rcu_process_callbacks() {
  799.555282 |   1)  ksoftir-2324  |               |        __rcu_process_callbacks() {
  799.555283 |   1)  ksoftir-2324  |   0.479 us    |          force_quiescent_state();
  799.555284 |   1)  ksoftir-2324  |   1.576 us    |        }
  799.555284 |   1)  ksoftir-2324  |               |        __rcu_process_callbacks() {
  799.555285 |   1)  ksoftir-2324  |               |          force_quiescent_state() {
  799.555286 |   1)  ksoftir-2324  |               |          cpu_quiet() {
  799.555286 |   1)  ksoftir-2324  |   0.518 us    |            _spin_lock_irqsave();
  799.555287 |   1)  ksoftir-2324  |   0.506 us    |            _spin_unlock_irqrestore();
  799.555288 |   1)  ksoftir-2324  |   2.563 us    |          }
  799.555289 |   1)  ksoftir-2324  |   4.624 us    |        }
  799.555289 |   1)  ksoftir-2324  |   7.836 us    |      }
  799.555290 |   1)  ksoftir-2324  |   0.495 us    |      _local_bh_enable();
  799.555291 |   1)  ksoftir-2324  | + 11.550 us   |    }
  799.555291 |   1)  ksoftir-2324  | + 12.713 us   |  }
  799.555292 |   1)  ksoftir-2324  |   0.524 us    |  _cond_resched();

We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
this indeed seems to be a CONFIG_TREE_RCU=y bug.

What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
no raise_softirq() calls. Could you please do a two-CPU trace too via:

   echo 3 > /debug/tracing/tracing_cpumask

So that we can see what's happening on the other CPU?

Also, could you please apply the debug patch below (or update to the 
very latest -tip tree), so that we get trace entries of softirq triggers 
too?

Thanks,

	Ingo

-------------->

From 6876d5d56716427f9bbe3af7e4e9c06cb760ae0c Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
Date: Mon, 16 Feb 2009 13:23:36 +0100
Subject: [PATCH] softirq: debug #2

Signed-off-by: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
---
 include/linux/interrupt.h |    3 ++-
 kernel/softirq.c          |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index e7bcfd7..cc1f529 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -271,7 +271,8 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
-#define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+#define ___raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+extern void __raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 110cad0..431cb4f 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -302,6 +302,13 @@ void irq_exit(void)
 	preempt_enable_no_resched();
 }
 
+void __raise_softirq_irqoff(unsigned int nr)
+{
+	ftrace_printk("nr: %d\n", nr);
+	or_softirq_pending(1UL << nr);
+}
+EXPORT_SYMBOL_GPL(__raise_softirq_irqoff);
+
 /*
  * This function must run with irqs disabled!
  */

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> this indeed seems to be a CONFIG_TREE_RCU=y bug.

> What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> no raise_softirq() calls. Could you please do a two-CPU trace too via:

>    echo 3 > /debug/tracing/tracing_cpumask

> So that we can see what's happening on the other CPU?

> Also, could you please apply the debug patch below (or update to the 
> very latest -tip tree), so that we get trace entries of softirq triggers 
> too?

Ok, the new trace with these additional modifications is here:
http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> 
> > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> 
> >    echo 3 > /debug/tracing/tracing_cpumask
> 
> > So that we can see what's happening on the other CPU?
> 
> > Also, could you please apply the debug patch below (or update to the 
> > very latest -tip tree), so that we get trace entries of softirq triggers 
> > too?
> 
> Ok, the new trace with these additional modifications is here:
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz

thanks.

This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:

  136.255963 |   0)   sleep-2345   |               |          update_process_times() {
  136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
  136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
  136.255966 |   0)   sleep-2345   |   2.262 us    |            }
  136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
  136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
  136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
  136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
  136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
  136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
  136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
  136.255974 |   0)   sleep-2345   |   3.832 us    |                }
  136.255975 |   0)   sleep-2345   |   5.491 us    |              }
  136.255976 |   0)   sleep-2345   |   8.667 us    |            }
  136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
  136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
  136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
  136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
  136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
  136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
  136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
  136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
  136.255984 |   0)   sleep-2345   |   3.059 us    |                }
  136.255985 |   0)   sleep-2345   |   4.594 us    |              }
  136.255986 |   0)   sleep-2345   |   7.800 us    |            }
  136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();

again and again.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> 
> * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> 
> > * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > 
> > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > 
> > >    echo 3 > /debug/tracing/tracing_cpumask
> > 
> > > So that we can see what's happening on the other CPU?
> > 
> > > Also, could you please apply the debug patch below (or update to the 
> > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > too?
> > 
> > Ok, the new trace with these additional modifications is here:
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> 
> thanks.
> 
> This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> 
>   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
>   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
>   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
>   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
>   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
>   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
>   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
>   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
>   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
>   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
>   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
>   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
>   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
>   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
>   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
>   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
>   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
>   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
>   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
>   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
>   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
>   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
>   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
>   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
>   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
>   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> 
> again and again.

Interesting...

I will take a look!

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > 
> > * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> > 
> > > * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > 
> > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > 
> > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > 
> > > > So that we can see what's happening on the other CPU?
> > > 
> > > > Also, could you please apply the debug patch below (or update to the 
> > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > too?
> > > 
> > > Ok, the new trace with these additional modifications is here:
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > 
> > thanks.
> > 
> > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > 
> >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > 
> > again and again.
> 
> Interesting...
> 
> I will take a look!

The above sequence is more or less normal behavior -- the RCU softirq
handler rcu_process_callbacks() is being invoked once per tick, which
appears to be HZ=1000 or thereabouts.  The system appears to be mostly
idle during this time period.

One oddity is that the _bh call to __rcu_process_callbacks() is invoking
force_quiescent_state() each time, and force_quiescent_state() isn't
doing anything.  This is a possible mismatch between the conditions in
rcu_pending() and force_quiescent_state(), and I will look into this.

However, this sequence is consuming less than 10 microseconds per
millisecond, so cannot be the main cause of the softirq issues you
are seeing, though if there really is a mismatch, it needs to be fixed,
and I will attend to this.

The interesting portion of the trace is later on:

  137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
  137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
  137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
  137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
  137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
  137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
  137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
  137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
  137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
  137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
  137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
  137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
  137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
  137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
  137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
  137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
  137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
  137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
  137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
  137.897007 |   1)  ksoftir-2302  |               |  schedule() {
  137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
  137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
  137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
  137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
  137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
  137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
  137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
  137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
  137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
  137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
  137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
  137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
  137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
  137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
  137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
  137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
  137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
  137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
  137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
  137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
  137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
  137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
  137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
  137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
  137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
  137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
  137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
  137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
  137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
  137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
  137.897045 |   ------------------------------------------
 1)  ksoftir-2302  =>    <idle>-0   
 ------------------------------------------

 1)    <idle>-0    |               |  /* nr: 8 */
 ------------------------------------------
 1)    <idle>-0    =>  ksoftir-2302 
 ------------------------------------------

  137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
  137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
  137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
  137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
  137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
  137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
  137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
  137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
  137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
  137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
  137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
  137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
  137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
  137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
  137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
  137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
  137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
  137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
  137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
  137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
  137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
  137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
  137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
  137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
  137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }

Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
so that this function is consuming more than 10% of a CPU.  The strange
thing is that I don't see a raise_softirq() in between, though perhaps
it gets inlined or something that makes it invisible to ftrace.
Certainly rcu_process_callbacks() can re-invoke itself, for example,
when a large number of RCU callbacks has piled up.  However, there are
only 29 calls to __call_rcu() over the entire time period, so that does
not appear to be the cause.  Strangely enough, there appear to be no
calls to rcu_do_batch() over the full trace, but this is invoked
unconditionally from __rcu_process_callbacks().  So perhaps the trace
wasn't covering that function?

Whatever, this pattern continues for more than 300 milliseconds(!).

Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
reproduce this and send the output of the debugfs files rcu/rcudata
and rcu/rcuhier?  The commands for this would be:

	mkdir /debug || :
	mount -t debugfs debugfs /debug
	cat /debug/rcu/rcuhier
	cat /debug/rcu/rcudata

I will try to reproduce locally as well.

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > 
> > > * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> > > 
> > > > * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > 
> > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > 
> > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > 
> > > > > So that we can see what's happening on the other CPU?
> > > > 
> > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > too?
> > > > 
> > > > Ok, the new trace with these additional modifications is here:
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > 
> > > thanks.
> > > 
> > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > 
> > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > 
> > > again and again.
> > 
> > Interesting...
> > 
> > I will take a look!
> 
> The above sequence is more or less normal behavior -- the RCU softirq
> handler rcu_process_callbacks() is being invoked once per tick, which
> appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> idle during this time period.
> 
> One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> force_quiescent_state() each time, and force_quiescent_state() isn't
> doing anything.  This is a possible mismatch between the conditions in
> rcu_pending() and force_quiescent_state(), and I will look into this.
> 
> However, this sequence is consuming less than 10 microseconds per
> millisecond, so cannot be the main cause of the softirq issues you
> are seeing, though if there really is a mismatch, it needs to be fixed,
> and I will attend to this.
> 
> The interesting portion of the trace is later on:
> 
>   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
>   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
>   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
>   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
>   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
>   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
>   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
>   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
>   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
>   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
>   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
>   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
>   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
>   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
>   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
>   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
>   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
>   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
>   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
>   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
>   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
>   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
>   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
>   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
>   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
>   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
>   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
>   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
>   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
>   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
>   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
>   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
>   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
>   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
>   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
>   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
>   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
>   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
>   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
>   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
>   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
>   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
>   137.897045 |   ------------------------------------------
>  1)  ksoftir-2302  =>    <idle>-0   
>  ------------------------------------------
> 
>  1)    <idle>-0    |               |  /* nr: 8 */
>  ------------------------------------------
>  1)    <idle>-0    =>  ksoftir-2302 
>  ------------------------------------------
> 
>   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
>   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
>   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
>   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
>   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
>   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
>   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
>   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
>   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
>   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
>   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
>   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
>   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
>   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
>   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
>   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> 
> Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> so that this function is consuming more than 10% of a CPU.  The strange
> thing is that I don't see a raise_softirq() in between, though perhaps
> it gets inlined or something that makes it invisible to ftrace.


More likely it's a bug in the function graph tracer.
I will try to fix it.


> Certainly rcu_process_callbacks() can re-invoke itself, for example,
> when a large number of RCU callbacks has piled up.  However, there are
> only 29 calls to __call_rcu() over the entire time period, so that does
> not appear to be the cause.  Strangely enough, there appear to be no
> calls to rcu_do_batch() over the full trace, but this is invoked
> unconditionally from __rcu_process_callbacks().  So perhaps the trace
> wasn't covering that function?


Hmm. I have to check that too. I'm building CONFIG_RCU_TREE.
Will try to find what happens.


> 
> Whatever, this pattern continues for more than 300 milliseconds(!).
> 
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier?  The commands for this would be:
> 
> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata
> 
> I will try to reproduce locally as well.
> 
> 							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > 
> > > * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> > > 
> > > > * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > 
> > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > 
> > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > 
> > > > > So that we can see what's happening on the other CPU?
> > > > 
> > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > too?
> > > > 
> > > > Ok, the new trace with these additional modifications is here:
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > 
> > > thanks.
> > > 
> > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > 
> > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > 
> > > again and again.
> > 
> > Interesting...
> > 
> > I will take a look!
> 
> The above sequence is more or less normal behavior -- the RCU softirq
> handler rcu_process_callbacks() is being invoked once per tick, which
> appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> idle during this time period.
> 
> One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> force_quiescent_state() each time, and force_quiescent_state() isn't
> doing anything.  This is a possible mismatch between the conditions in
> rcu_pending() and force_quiescent_state(), and I will look into this.
> 
> However, this sequence is consuming less than 10 microseconds per
> millisecond, so cannot be the main cause of the softirq issues you
> are seeing, though if there really is a mismatch, it needs to be fixed,
> and I will attend to this.
> 
> The interesting portion of the trace is later on:
> 
>   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
>   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
>   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
>   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
>   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
>   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
>   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
>   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
>   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
>   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
>   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
>   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
>   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
>   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
>   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
>   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
>   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
>   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
>   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
>   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
>   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
>   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
>   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
>   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
>   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
>   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
>   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
>   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
>   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
>   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
>   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
>   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
>   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
>   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
>   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
>   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
>   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
>   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
>   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
>   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
>   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
>   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
>   137.897045 |   ------------------------------------------
>  1)  ksoftir-2302  =>    <idle>-0   
>  ------------------------------------------
> 
>  1)    <idle>-0    |               |  /* nr: 8 */
>  ------------------------------------------
>  1)    <idle>-0    =>  ksoftir-2302 
>  ------------------------------------------
> 
>   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
>   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
>   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
>   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
>   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
>   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
>   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
>   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
>   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
>   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
>   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
>   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
>   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
>   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
>   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
>   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
>   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
>   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
>   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
>   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
>   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
>   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
>   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
>   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> 
> Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> so that this function is consuming more than 10% of a CPU.  The strange
> thing is that I don't see a raise_softirq() in between, though perhaps
> it gets inlined or something that makes it invisible to ftrace.
> Certainly rcu_process_callbacks() can re-invoke itself, for example,
> when a large number of RCU callbacks has piled up.  However, there are
> only 29 calls to __call_rcu() over the entire time period, so that does
> not appear to be the cause.  Strangely enough, there appear to be no
> calls to rcu_do_batch() over the full trace, but this is invoked
> unconditionally from __rcu_process_callbacks().  So perhaps the trace
> wasn't covering that function?


I just checked an assembly dump of my vmlinux, and rcu_do_batch()
has been inlined. I don't understand why, this is a wide function.

 
> Whatever, this pattern continues for more than 300 milliseconds(!).
> 
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier?  The commands for this would be:
> 
> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata
> 
> I will try to reproduce locally as well.
> 
> 							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:

> Here the calls to rcu_process_callbacks() are only 75 
> microseconds apart, so that this function is consuming more 
> than 10% of a CPU.  The strange thing is that I don't see a 
> raise_softirq() in between, though perhaps it gets inlined or 
> something that makes it invisible to ftrace.

look at the latest trace please, that has even the most inline 
raise-softirq method instrumented, so all the raising is 
visible.

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

* Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> [2009-02-16 10:56]:
> Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> reproduce this and send the output of the debugfs files rcu/rcudata
> and rcu/rcuhier? The commands for this would be:

> 	mkdir /debug || :
> 	mount -t debugfs debugfs /debug
> 	cat /debug/rcu/rcuhier
> 	cat /debug/rcu/rcudata

Here are the two files (stored while the ksoftirqd problem occurs)
attached.

-- 
Damien

[-- Attachment #2: rcuhier.txt --]
[-- Type: text/plain, Size: 170 bytes --]

rcu:
c=6148 g=6149 s=2 jfq=3 j=31ba nfqs=62392/nfqsng=3(62389) fqlh=7
1/3 0:1 ^0    
rcu_bh:
c=-296 g=-296 s=2 jfq=-93079 j=31ba nfqs=6/nfqsng=0(6) fqlh=0
0/3 0:1 ^0    

[-- Attachment #3: rcudata.txt --]
[-- Type: text/plain, Size: 417 bytes --]

rcu:
  0 c=6163 g=6164 pq=0 pqc=6162 qp=1 rpfq=2 rp=a249 dt=729511/1 dn=0 df=10 of=0 ri=38915 ql=5 b=10
  1 c=6163 g=6164 pq=1 pqc=6163 qp=0 rpfq=0 rp=d580 dt=34418019/1 dn=0 df=374 of=0 ri=19636 ql=0 b=10
rcu_bh:
  0 c=-296 g=-296 pq=1 pqc=-296 qp=0 rpfq=-19494 rp=e947 dt=729511/1 dn=0 df=0 of=0 ri=0 ql=0 b=10
  1 c=-296 g=-296 pq=1 pqc=-296 qp=1 rpfq=-16839253 rp=2918 dt=34418019/1 dn=0 df=0 of=0 ri=2 ql=0 b=10

Re: [Bug #12510] 2.6.29-rc2 dies on startup

[Ferenc Wagner]

"Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> writes:

> This message has been generated automatically as a part of a report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
>
>
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
> Subject		: 2.6.29-rc2 dies on startup
> Submitter	: Ferenc Wagner <wferi-eEbw3PyuezQ@public.gmane.org>
> Date		: 2009-01-19 12:53 (27 days old)
> References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4

This is fixed in 2.6.29-rc5 by

commit 5701c0519b7a357a602fda5c96f26197ecfc4c85
Author: Arve Hjønnevåg <arve-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org>
Date:   Fri Jan 30 20:21:09 2009 -0800

    Staging: android: ram_console: Disable ECC when early init is
    enabled and validate buffer size
-- 
Thanks,
Feri.

Re: 2.6.29-rc5: Reported regressions from 2.6.28

[Rafael J. Wysocki]

On Monday 16 February 2009, Jarek Poplawski wrote:
> On 14-02-2009 21:35, Rafael J. Wysocki wrote:
> > This message contains a list of some regressions from 2.6.28, for which there
> > are no fixes in the mainline I know of.  If any of them have been fixed already,
> > please let me know.
> ...
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12615
> > Subject		: boot hangs while bringing up gianfar ethernet
> > Submitter	: Ira Snyder <iws@ovro.caltech.edu>
> > Date		: 2009-01-29 19:41 (17 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123325817201665&w=4
> 
> Fixed by:
> commit	8707bdd48ab705a459ac1b12014075a139d1d4f9
> gianfar: Fix boot hangs while bringing up gianfar ethernet
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=8707bdd48ab705a459ac1b12014075a139d1d4f9

Thanks, closed.

Rafael

Re: [Bug #12510] 2.6.29-rc2 dies on startup

[Rafael J. Wysocki]

On Monday 16 February 2009, Ferenc Wagner wrote:
> "Rafael J. Wysocki" <rjw@sisk.pl> writes:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> >
> >
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12510
> > Subject		: 2.6.29-rc2 dies on startup
> > Submitter	: Ferenc Wagner <wferi@niif.hu>
> > Date		: 2009-01-19 12:53 (27 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123236969321703&w=4
> 
> This is fixed in 2.6.29-rc5 by
> 
> commit 5701c0519b7a357a602fda5c96f26197ecfc4c85
> Author: Arve Hjønnevåg <arve@android.com>
> Date:   Fri Jan 30 20:21:09 2009 -0800
> 
>     Staging: android: ram_console: Disable ECC when early init is
>     enabled and validate buffer size

Thanks, closed.

Rafael

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 09:02:18PM +0100, Frederic Weisbecker wrote:
> On Mon, Feb 16, 2009 at 10:56:16AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 08:06:13AM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 02:21:51PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:
> > > > 
> > > > > * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> [090216 13:26]:
> > > > > > We do get 0x100 which is 1 << RCU_SOFTIRQ, i.e. the RCU softirq. Paul, 
> > > > > > this indeed seems to be a CONFIG_TREE_RCU=y bug.
> > > > > 
> > > > > > What is weird is that RCU_SOFTIRQ gets set again and again - but there's 
> > > > > > no raise_softirq() calls. Could you please do a two-CPU trace too via:
> > > > > 
> > > > > >    echo 3 > /debug/tracing/tracing_cpumask
> > > > > 
> > > > > > So that we can see what's happening on the other CPU?
> > > > > 
> > > > > > Also, could you please apply the debug patch below (or update to the 
> > > > > > very latest -tip tree), so that we get trace entries of softirq triggers 
> > > > > > too?
> > > > > 
> > > > > Ok, the new trace with these additional modifications is here:
> > > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_1300_ksoftirqd_pb_abstime_proc_mask3.txt.gz
> > > > 
> > > > thanks.
> > > > 
> > > > This confirms that SOFTIRQ_RCU gets raised here in the timer IRQ:
> > > > 
> > > >   136.255963 |   0)   sleep-2345   |               |          update_process_times() {
> > > >   136.255964 |   0)   sleep-2345   |               |            account_process_tick() {
> > > >   136.255965 |   0)   sleep-2345   |   0.779 us    |              account_system_time();
> > > >   136.255966 |   0)   sleep-2345   |   2.262 us    |            }
> > > >   136.255967 |   0)   sleep-2345   |               |            run_local_timers() {
> > > >   136.255968 |   0)   sleep-2345   |   0.802 us    |              hrtimer_run_queues();
> > > >   136.255969 |   0)   sleep-2345   |               |              raise_softirq() {
> > > >   136.255970 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > > >   136.255971 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > > >   136.255972 |   0)   sleep-2345   |               |                    /* nr: 1 */
> > > >   136.255973 |   0)   sleep-2345   |   2.194 us    |                  }
> > > >   136.255974 |   0)   sleep-2345   |   3.832 us    |                }
> > > >   136.255975 |   0)   sleep-2345   |   5.491 us    |              }
> > > >   136.255976 |   0)   sleep-2345   |   8.667 us    |            }
> > > >   136.255976 |   0)   sleep-2345   |   0.792 us    |            rcu_pending();
> > > >   136.255978 |   0)   sleep-2345   |               |            rcu_check_callbacks() {
> > > >   136.255979 |   0)   sleep-2345   |   0.781 us    |              idle_cpu();
> > > >   136.255981 |   0)   sleep-2345   |               |              raise_softirq() {
> > > >   136.255981 |   0)   sleep-2345   |               |                raise_softirq_irqoff() {
> > > >   136.255982 |   0)   sleep-2345   |               |                  __raise_softirq_irqoff() {
> > > >   136.255983 |   0)   sleep-2345   |               |                    /* nr: 8 */
> > > >   136.255984 |   0)   sleep-2345   |   1.555 us    |                  }
> > > >   136.255984 |   0)   sleep-2345   |   3.059 us    |                }
> > > >   136.255985 |   0)   sleep-2345   |   4.594 us    |              }
> > > >   136.255986 |   0)   sleep-2345   |   7.800 us    |            }
> > > >   136.255987 |   0)   sleep-2345   |   0.737 us    |            printk_tick();
> > > > 
> > > > again and again.
> > > 
> > > Interesting...
> > > 
> > > I will take a look!
> > 
> > The above sequence is more or less normal behavior -- the RCU softirq
> > handler rcu_process_callbacks() is being invoked once per tick, which
> > appears to be HZ=1000 or thereabouts.  The system appears to be mostly
> > idle during this time period.
> > 
> > One oddity is that the _bh call to __rcu_process_callbacks() is invoking
> > force_quiescent_state() each time, and force_quiescent_state() isn't
> > doing anything.  This is a possible mismatch between the conditions in
> > rcu_pending() and force_quiescent_state(), and I will look into this.
> > 
> > However, this sequence is consuming less than 10 microseconds per
> > millisecond, so cannot be the main cause of the softirq issues you
> > are seeing, though if there really is a mismatch, it needs to be fixed,
> > and I will attend to this.
> > 
> > The interesting portion of the trace is later on:
> > 
> >   137.896992 |   1)  ksoftir-2302  |               |  do_softirq() {
> >   137.896993 |   1)  ksoftir-2302  |               |    __do_softirq() {
> >   137.896993 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
> >   137.896994 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
> >   137.896995 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
> >   137.896995 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.896996 |   1)  ksoftir-2302  |   0.498 us    |          force_quiescent_state();
> >   137.896997 |   1)  ksoftir-2302  |   1.588 us    |        }
> >   137.896997 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.896998 |   1)  ksoftir-2302  |   0.475 us    |          force_quiescent_state();
> >   137.896999 |   1)  ksoftir-2302  |               |          cpu_quiet() {
> >   137.896999 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
> >   137.897000 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
> >   137.897001 |   1)  ksoftir-2302  |   2.528 us    |          }
> >   137.897002 |   1)  ksoftir-2302  |   4.607 us    |        }
> >   137.897002 |   1)  ksoftir-2302  |   7.825 us    |      }
> >   137.897003 |   1)  ksoftir-2302  |   0.498 us    |      _local_bh_enable();
> >   137.897004 |   1)  ksoftir-2302  | + 11.430 us   |    }
> >   137.897005 |   1)  ksoftir-2302  | + 12.572 us   |  }
> >   137.897005 |   1)  ksoftir-2302  |   0.549 us    |  _cond_resched();
> >   137.897006 |   1)  ksoftir-2302  |   0.541 us    |  kthread_should_stop();
> >   137.897007 |   1)  ksoftir-2302  |               |  schedule() {
> >   137.897008 |   1)  ksoftir-2302  |               |    __schedule() {
> >   137.897008 |   1)  ksoftir-2302  |   0.514 us    |      _spin_lock_irq();
> >   137.897009 |   1)  ksoftir-2302  |   0.594 us    |      update_rq_clock();
> >   137.897011 |   1)  ksoftir-2302  |               |      deactivate_task() {
> >   137.897011 |   1)  ksoftir-2302  |               |        dequeue_task() {
> >   137.897011 |   1)  ksoftir-2302  |               |          dequeue_task_fair() {
> >   137.897012 |   1)  ksoftir-2302  |               |            update_curr() {
> >   137.897012 |   1)  ksoftir-2302  |               |              calc_delta_fair() {
> >   137.897013 |   1)  ksoftir-2302  |   0.506 us    |                calc_delta_mine();
> >   137.897014 |   1)  ksoftir-2302  |   1.528 us    |              }
> >   137.897015 |   1)  ksoftir-2302  |   2.563 us    |            }
> >   137.897015 |   1)  ksoftir-2302  |   0.513 us    |            hrtick_start_fair();
> >   137.897019 |   1)  ksoftir-2302  |   4.662 us    |          }
> >   137.897019 |   1)  ksoftir-2302  |   8.213 us    |        }
> >   137.897020 |   1)  ksoftir-2302  |   9.195 us    |      }
> >   137.897020 |   1)  ksoftir-2302  |   0.960 us    |      find_busiest_group();
> >   137.897022 |   1)  ksoftir-2302  |   0.493 us    |      msecs_to_jiffies();
> >   137.897023 |   1)  ksoftir-2302  |   0.511 us    |      put_prev_task_fair();
> >   137.897024 |   1)  ksoftir-2302  |               |      pick_next_task() {
> >   137.897024 |   1)  ksoftir-2302  |   0.481 us    |        pick_next_task_fair();
> >   137.897025 |   1)  ksoftir-2302  |   0.491 us    |        pick_next_task_rt();
> >   137.897026 |   1)  ksoftir-2302  |   0.474 us    |        pick_next_task_fair();
> >   137.897027 |   1)  ksoftir-2302  |   0.480 us    |        pick_next_task_idle();
> >   137.897028 |   1)  ksoftir-2302  |   4.516 us    |      }
> >   137.897029 |   1)  ksoftir-2302  |               |      perf_counter_task_sched_out() {
> >   137.897029 |   1)  ksoftir-2302  |               |        __perf_counter_sched_out() {
> >   137.897030 |   1)  ksoftir-2302  |   0.516 us    |          _spin_lock();
> >   137.897031 |   1)  ksoftir-2302  |   1.486 us    |        }
> >   137.897031 |   1)  ksoftir-2302  |   2.462 us    |      }
> >   137.897032 |   1)  ksoftir-2302  |   0.516 us    |      __lock_text_start();
> >   137.897045 |   ------------------------------------------
> >  1)  ksoftir-2302  =>    <idle>-0   
> >  ------------------------------------------
> > 
> >  1)    <idle>-0    |               |  /* nr: 8 */
> >  ------------------------------------------
> >  1)    <idle>-0    =>  ksoftir-2302 
> >  ------------------------------------------
> > 
> >   137.897064 |   1)  ksoftir-2302  |               |      finish_task_switch() {
> >   137.897064 |   1)  ksoftir-2302  |               |        perf_counter_task_sched_in() {
> >   137.897065 |   1)  ksoftir-2302  |   0.508 us    |          _spin_lock();
> >   137.897066 |   1)  ksoftir-2302  |   1.525 us    |        }
> >   137.897066 |   1)  ksoftir-2302  |   2.617 us    |      }
> >   137.897067 |   1)  ksoftir-2302  | + 58.928 us   |    }
> >   137.897067 |   1)  ksoftir-2302  | + 59.926 us   |  }
> >   137.897068 |   1)  ksoftir-2302  |               |  do_softirq() {
> >   137.897068 |   1)  ksoftir-2302  |               |    __do_softirq() {
> >   137.897069 |   1)  ksoftir-2302  |               |      /* #1 softirq pending: 00000100 */
> >   137.897070 |   1)  ksoftir-2302  |               |      /* #2 softirq pending: 00000000 */
> >   137.897070 |   1)  ksoftir-2302  |               |      rcu_process_callbacks() {
> >   137.897071 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.897071 |   1)  ksoftir-2302  |               |          force_quiescent_state() {
> >   137.897073 |   1)  ksoftir-2302  |   1.575 us    |        }
> >   137.897073 |   1)  ksoftir-2302  |               |        __rcu_process_callbacks() {
> >   137.897074 |   1)  ksoftir-2302  |   0.474 us    |          force_quiescent_state();
> >   137.897075 |   1)  ksoftir-2302  |               |          cpu_quiet() {
> >   137.897075 |   1)  ksoftir-2302  |   0.526 us    |            _spin_lock_irqsave();
> >   137.897076 |   1)  ksoftir-2302  |   0.511 us    |            _spin_unlock_irqrestore();
> >   137.897077 |   1)  ksoftir-2302  |   2.532 us    |          }
> >   137.897078 |   1)  ksoftir-2302  |   4.632 us    |        }
> >   137.897078 |   1)  ksoftir-2302  |   7.815 us    |      }
> >   137.897079 |   1)  ksoftir-2302  |   0.501 us    |      _local_bh_enable();
> >   137.897080 |   1)  ksoftir-2302  | + 11.405 us   |    }
> >   137.897080 |   1)  ksoftir-2302  | + 12.542 us   |  }
> > 
> > Here the calls to rcu_process_callbacks() are only 75 microseconds apart,
> > so that this function is consuming more than 10% of a CPU.  The strange
> > thing is that I don't see a raise_softirq() in between, though perhaps
> > it gets inlined or something that makes it invisible to ftrace.
> > Certainly rcu_process_callbacks() can re-invoke itself, for example,
> > when a large number of RCU callbacks has piled up.  However, there are
> > only 29 calls to __call_rcu() over the entire time period, so that does
> > not appear to be the cause.  Strangely enough, there appear to be no
> > calls to rcu_do_batch() over the full trace, but this is invoked
> > unconditionally from __rcu_process_callbacks().  So perhaps the trace
> > wasn't covering that function?
> 
> I just checked an assembly dump of my vmlinux, and rcu_do_batch()
> has been inlined. I don't understand why, this is a wide function.

Yow!!!  ;-)

> > Whatever, this pattern continues for more than 300 milliseconds(!).
> > 
> > Would you be willing to enable CONFIG_RCU_TRACE and CONFIG_TREE_RCU,
> > reproduce this and send the output of the debugfs files rcu/rcudata
> > and rcu/rcuhier?  The commands for this would be:
> > 
> > 	mkdir /debug || :
> > 	mount -t debugfs debugfs /debug
> > 	cat /debug/rcu/rcuhier
> > 	cat /debug/rcu/rcudata
> > 
> > I will try to reproduce locally as well.

No luck thus far, perhaps because I first tried x86_64 and Damien's run
was on 32-bit x86.

							Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> 
> > Here the calls to rcu_process_callbacks() are only 75 
> > microseconds apart, so that this function is consuming more 
> > than 10% of a CPU.  The strange thing is that I don't see a 
> > raise_softirq() in between, though perhaps it gets inlined or 
> > something that makes it invisible to ftrace.
> 
> look at the latest trace please, that has even the most inline 
> raise-softirq method instrumented, so all the raising is 
> visible.

Ah, my apologies!  This time looking at:

http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz


  799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
  799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
  799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
  799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
  799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
[ . . . ]

Yikes!!!

Why is rcu_check_callbacks() being invoked so often?  It should be called
but once per jiffy, and here it is called no less than 22 times in about
3.5 milliseconds, meaning one call every 160 microseconds or so.

Hmmm...

Looks like we never return from:

  799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {

Perhaps we are taking an interrupt immediately after the
local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
And then deciding to go back into nohz mode at 799.521326, 117
microseconds later, after which we re-invoke rcu_check_callbacks(),
which again raises RCU's softirq.

And the reason we are invoking rcu_check_callbacks() so often appears
to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
which explains my failure to reproduce on a 64-bit system:

	void cpu_idle(void)
	{
		int cpu = smp_processor_id();

		current_thread_info()->status |= TS_POLLING;

		/* endless idle loop with no priority at all */
		while (1) {
			tick_nohz_stop_sched_tick(1);
			while (!need_resched()) {

				check_pgt_cache();
				rmb();

				if (rcu_pending(cpu))
					rcu_check_callbacks(cpu, 0);

				if (cpu_is_offline(cpu))
					play_dead();

				local_irq_disable();
				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
				/* Don't trace irqs off for idle */
				stop_critical_timings();
				pm_idle();
				start_critical_timings();
			}
			tick_nohz_restart_sched_tick();
			preempt_enable_no_resched();
			schedule();
			preempt_disable();
		}
	}

If we go in and out of nohz mode quickly, we will invoke rcu_pending()
each time.  I would expect rcu_pending() to return 0 most of the time,
but that apparently isn't the case with treercu...

What is the easiest way for me to make it easy to trace the return path
from __rcu_pending()?  Make each return path call an empty function
located off where the compiler cannot see it, I guess...  Diagnostic
patch along these lines below.  Frederic, Damien, could you please give
it a go?  (And of course please let me know if something else is
needed.)

Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---

 rcupdate.c |   23 +++++++++++++++++++++++
 rcutree.c  |   31 +++++++++++++++++++++++++------
 2 files changed, 48 insertions(+), 6 deletions(-)

diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
index d92a76a..42bbf03 100644
--- a/kernel/rcupdate.c
+++ b/kernel/rcupdate.c
@@ -175,3 +175,26 @@ void __init rcu_init(void)
 	__rcu_init();
 }
 
+void __rcu_pending_qs_pending(void)
+{
+}
+
+void __rcu_pending_callbacks_ready(void)
+{
+}
+
+void __rcu_pending_needs_gp(void)
+{
+}
+
+void __rcu_pending_new_completed(void)
+{
+}
+
+void __rcu_pending_new_gp(void)
+{
+}
+
+void __rcu_pending_fqs(void)
+{
+}
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..e2d72c3 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
 }
 EXPORT_SYMBOL_GPL(call_rcu_bh);
 
+extern void __rcu_pending_qs_pending(void);
+extern void __rcu_pending_callbacks_ready(void);
+extern void __rcu_pending_needs_gp(void);
+extern void __rcu_pending_new_completed(void);
+extern void __rcu_pending_new_gp(void);
+extern void __rcu_pending_fqs(void);
+
 /*
  * Check to see if there is any immediate RCU-related work to be done
  * by the current CPU, for the specified type of RCU, returning 1 if so.
@@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 	check_cpu_stall(rsp, rdp);
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
-	if (rdp->qs_pending)
+	if (rdp->qs_pending) {
+		__rcu_pending_qs_pending();
 		return 1;
+	}
 
 	/* Does this CPU have callbacks ready to invoke? */
-	if (cpu_has_callbacks_ready_to_invoke(rdp))
+	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
+		__rcu_pending_callbacks_ready();
 		return 1;
+	}
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
-	if (cpu_needs_another_gp(rsp, rdp))
+	if (cpu_needs_another_gp(rsp, rdp)) {
+		__rcu_pending_needs_gp();
 		return 1;
+	}
 
 	/* Has another RCU grace period completed?  */
-	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
+	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
+		__rcu_pending_new_completed();
 		return 1;
+	}
 
 	/* Has a new RCU grace period started? */
-	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
+	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
+		__rcu_pending_new_gp();
 		return 1;
+	}
 
 	/* Has an RCU GP gone long enough to send resched IPIs &c? */
 	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
 	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
-	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
+	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
+		__rcu_pending_fqs();
 		return 1;
+	}
 
 	/* nothing to do */
 	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > 
> > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > 
> > > Here the calls to rcu_process_callbacks() are only 75 
> > > microseconds apart, so that this function is consuming more 
> > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > raise_softirq() in between, though perhaps it gets inlined or 
> > > something that makes it invisible to ftrace.
> > 
> > look at the latest trace please, that has even the most inline 
> > raise-softirq method instrumented, so all the raising is 
> > visible.
> 
> Ah, my apologies!  This time looking at:
> 
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> 
> 
>   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
>   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> [ . . . ]
> 
> Yikes!!!
> 
> Why is rcu_check_callbacks() being invoked so often?  It should be called
> but once per jiffy, and here it is called no less than 22 times in about
> 3.5 milliseconds, meaning one call every 160 microseconds or so.

BTW, the other question I have is "why do we need to call rcu_pending()
and rcu_check_callbacks() from the idle loop of 32-bit x86, especially
given that no other architecture does this?".  Don't get me wrong, it
would be good to get rcutree's rcu_pending() to avoid spuriously saying
that rcu_check_callbacks() should be invoked, so I would still like the
trace with my patch, but...

							Thanx, Paul

> Hmmm...
> 
> Looks like we never return from:
> 
>   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> 
> Perhaps we are taking an interrupt immediately after the
> local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> And then deciding to go back into nohz mode at 799.521326, 117
> microseconds later, after which we re-invoke rcu_check_callbacks(),
> which again raises RCU's softirq.
> 
> And the reason we are invoking rcu_check_callbacks() so often appears
> to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> which explains my failure to reproduce on a 64-bit system:
> 
> 	void cpu_idle(void)
> 	{
> 		int cpu = smp_processor_id();
> 
> 		current_thread_info()->status |= TS_POLLING;
> 
> 		/* endless idle loop with no priority at all */
> 		while (1) {
> 			tick_nohz_stop_sched_tick(1);
> 			while (!need_resched()) {
> 
> 				check_pgt_cache();
> 				rmb();
> 
> 				if (rcu_pending(cpu))
> 					rcu_check_callbacks(cpu, 0);
> 
> 				if (cpu_is_offline(cpu))
> 					play_dead();
> 
> 				local_irq_disable();
> 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> 				/* Don't trace irqs off for idle */
> 				stop_critical_timings();
> 				pm_idle();
> 				start_critical_timings();
> 			}
> 			tick_nohz_restart_sched_tick();
> 			preempt_enable_no_resched();
> 			schedule();
> 			preempt_disable();
> 		}
> 	}
> 
> If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> each time.  I would expect rcu_pending() to return 0 most of the time,
> but that apparently isn't the case with treercu...
> 
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)
> 
> Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> ---
> 
>  rcupdate.c |   23 +++++++++++++++++++++++
>  rcutree.c  |   31 +++++++++++++++++++++++++------
>  2 files changed, 48 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
> index d92a76a..42bbf03 100644
> --- a/kernel/rcupdate.c
> +++ b/kernel/rcupdate.c
> @@ -175,3 +175,26 @@ void __init rcu_init(void)
>  	__rcu_init();
>  }
>  
> +void __rcu_pending_qs_pending(void)
> +{
> +}
> +
> +void __rcu_pending_callbacks_ready(void)
> +{
> +}
> +
> +void __rcu_pending_needs_gp(void)
> +{
> +}
> +
> +void __rcu_pending_new_completed(void)
> +{
> +}
> +
> +void __rcu_pending_new_gp(void)
> +{
> +}
> +
> +void __rcu_pending_fqs(void)
> +{
> +}
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..e2d72c3 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
>  }
>  EXPORT_SYMBOL_GPL(call_rcu_bh);
>  
> +extern void __rcu_pending_qs_pending(void);
> +extern void __rcu_pending_callbacks_ready(void);
> +extern void __rcu_pending_needs_gp(void);
> +extern void __rcu_pending_new_completed(void);
> +extern void __rcu_pending_new_gp(void);
> +extern void __rcu_pending_fqs(void);
> +
>  /*
>   * Check to see if there is any immediate RCU-related work to be done
>   * by the current CPU, for the specified type of RCU, returning 1 if so.
> @@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
>  	check_cpu_stall(rsp, rdp);
>  
>  	/* Is the RCU core waiting for a quiescent state from this CPU? */
> -	if (rdp->qs_pending)
> +	if (rdp->qs_pending) {
> +		__rcu_pending_qs_pending();
>  		return 1;
> +	}
>  
>  	/* Does this CPU have callbacks ready to invoke? */
> -	if (cpu_has_callbacks_ready_to_invoke(rdp))
> +	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
> +		__rcu_pending_callbacks_ready();
>  		return 1;
> +	}
>  
>  	/* Has RCU gone idle with this CPU needing another grace period? */
> -	if (cpu_needs_another_gp(rsp, rdp))
> +	if (cpu_needs_another_gp(rsp, rdp)) {
> +		__rcu_pending_needs_gp();
>  		return 1;
> +	}
>  
>  	/* Has another RCU grace period completed?  */
> -	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
> +		__rcu_pending_new_completed();
>  		return 1;
> +	}
>  
>  	/* Has a new RCU grace period started? */
> -	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
> +		__rcu_pending_new_gp();
>  		return 1;
> +	}
>  
>  	/* Has an RCU GP gone long enough to send resched IPIs &c? */
>  	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
>  	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
> -	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
> +	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
> +		__rcu_pending_fqs();
>  		return 1;
> +	}
>  
>  	/* nothing to do */
>  	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > 
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > 
> > > Here the calls to rcu_process_callbacks() are only 75 
> > > microseconds apart, so that this function is consuming more 
> > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > raise_softirq() in between, though perhaps it gets inlined or 
> > > something that makes it invisible to ftrace.
> > 
> > look at the latest trace please, that has even the most inline 
> > raise-softirq method instrumented, so all the raising is 
> > visible.
> 
> Ah, my apologies!  This time looking at:
> 
> http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> 
> 
>   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
>   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
>   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
>   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> [ . . . ]
> 
> Yikes!!!
> 
> Why is rcu_check_callbacks() being invoked so often?  It should be called
> but once per jiffy, and here it is called no less than 22 times in about
> 3.5 milliseconds, meaning one call every 160 microseconds or so.
> 
> Hmmm...
> 
> Looks like we never return from:
> 
>   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> 
> Perhaps we are taking an interrupt immediately after the
> local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> And then deciding to go back into nohz mode at 799.521326, 117
> microseconds later, after which we re-invoke rcu_check_callbacks(),
> which again raises RCU's softirq.
> 
> And the reason we are invoking rcu_check_callbacks() so often appears
> to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> which explains my failure to reproduce on a 64-bit system:
> 
> 	void cpu_idle(void)
> 	{
> 		int cpu = smp_processor_id();
> 
> 		current_thread_info()->status |= TS_POLLING;
> 
> 		/* endless idle loop with no priority at all */
> 		while (1) {
> 			tick_nohz_stop_sched_tick(1);
> 			while (!need_resched()) {
> 
> 				check_pgt_cache();
> 				rmb();
> 
> 				if (rcu_pending(cpu))
> 					rcu_check_callbacks(cpu, 0);
> 
> 				if (cpu_is_offline(cpu))
> 					play_dead();
> 
> 				local_irq_disable();
> 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> 				/* Don't trace irqs off for idle */
> 				stop_critical_timings();
> 				pm_idle();
> 				start_critical_timings();
> 			}
> 			tick_nohz_restart_sched_tick();
> 			preempt_enable_no_resched();
> 			schedule();
> 			preempt_disable();
> 		}
> 	}
> 
> If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> each time.  I would expect rcu_pending() to return 0 most of the time,
> but that apparently isn't the case with treercu...
> 
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)


No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
inside the functions, ie:

__rcu_pending() {
	 /* pending_qs */
}

I've converted your below patch with ftrace_printks and tested it under an old P2
with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
lucky :-) 
I guess I successfully reproduced the softirq/rcu overhead.
Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
When I will have more time, I will fix that.

The trace is here http://dl.free.fr/uyWGgCbx4

It looks like it mostly returns 1 because of the waiting for quiescent state:

$ cat rcutrace | grep "/* pending_none" | wc -l
221
$ cat rcutrace | grep "/* pending_qs" | wc -l
248
$ cat rcutrace | grep "/* pending" | wc -l
469


diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..c9e78f6 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -45,6 +45,7 @@
 #include <linux/cpu.h>
 #include <linux/mutex.h>
 #include <linux/time.h>
+#include <linux/ftrace.h>
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 static struct lock_class_key rcu_lock_key;
@@ -1249,31 +1250,44 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 	check_cpu_stall(rsp, rdp);
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
-	if (rdp->qs_pending)
+	if (rdp->qs_pending) {
+		ftrace_printk("pending_qs\n");
 		return 1;
+	}
 
 	/* Does this CPU have callbacks ready to invoke? */
-	if (cpu_has_callbacks_ready_to_invoke(rdp))
+	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
+		ftrace_printk("pending_ready_invoke\n");
 		return 1;
+	}
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
-	if (cpu_needs_another_gp(rsp, rdp))
+	if (cpu_needs_another_gp(rsp, rdp)) {
+		ftrace_printk("pending_gp\n");
 		return 1;
+	}
 
 	/* Has another RCU grace period completed?  */
-	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
+	if (ACCESS_ONCE(rsp->completed) != rdp->completed) {/* outside of lock */
+		ftrace_printk("pending_gp_completed\n");
 		return 1;
+	}
 
 	/* Has a new RCU grace period started? */
-	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
+	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) { /* outside of lock */
+		ftrace_printk("pending_gp_new_started\n");
 		return 1;
+	}
 
 	/* Has an RCU GP gone long enough to send resched IPIs &c? */
 	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
 	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
-	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
+	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
+		ftrace_printk("pending_ipi\n");
 		return 1;
+	}
 
+	ftrace_printk("pending_none\n");
 	/* nothing to do */
 	return 0;
 }

 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> 
>  rcupdate.c |   23 +++++++++++++++++++++++
>  rcutree.c  |   31 +++++++++++++++++++++++++------
>  2 files changed, 48 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
> index d92a76a..42bbf03 100644
> --- a/kernel/rcupdate.c
> +++ b/kernel/rcupdate.c
> @@ -175,3 +175,26 @@ void __init rcu_init(void)
>  	__rcu_init();
>  }
>  
> +void __rcu_pending_qs_pending(void)
> +{
> +}
> +
> +void __rcu_pending_callbacks_ready(void)
> +{
> +}
> +
> +void __rcu_pending_needs_gp(void)
> +{
> +}
> +
> +void __rcu_pending_new_completed(void)
> +{
> +}
> +
> +void __rcu_pending_new_gp(void)
> +{
> +}
> +
> +void __rcu_pending_fqs(void)
> +{
> +}
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..e2d72c3 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1234,6 +1234,13 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
>  }
>  EXPORT_SYMBOL_GPL(call_rcu_bh);
>  
> +extern void __rcu_pending_qs_pending(void);
> +extern void __rcu_pending_callbacks_ready(void);
> +extern void __rcu_pending_needs_gp(void);
> +extern void __rcu_pending_new_completed(void);
> +extern void __rcu_pending_new_gp(void);
> +extern void __rcu_pending_fqs(void);
> +
>  /*
>   * Check to see if there is any immediate RCU-related work to be done
>   * by the current CPU, for the specified type of RCU, returning 1 if so.
> @@ -1249,30 +1256,42 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
>  	check_cpu_stall(rsp, rdp);
>  
>  	/* Is the RCU core waiting for a quiescent state from this CPU? */
> -	if (rdp->qs_pending)
> +	if (rdp->qs_pending) {
> +		__rcu_pending_qs_pending();
>  		return 1;
> +	}
>  
>  	/* Does this CPU have callbacks ready to invoke? */
> -	if (cpu_has_callbacks_ready_to_invoke(rdp))
> +	if (cpu_has_callbacks_ready_to_invoke(rdp)) {
> +		__rcu_pending_callbacks_ready();
>  		return 1;
> +	}
>  
>  	/* Has RCU gone idle with this CPU needing another grace period? */
> -	if (cpu_needs_another_gp(rsp, rdp))
> +	if (cpu_needs_another_gp(rsp, rdp)) {
> +		__rcu_pending_needs_gp();
>  		return 1;
> +	}
>  
>  	/* Has another RCU grace period completed?  */
> -	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->completed) != rdp->completed) /* outside of lock */ {
> +		__rcu_pending_new_completed();
>  		return 1;
> +	}
>  
>  	/* Has a new RCU grace period started? */
> -	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */
> +	if (ACCESS_ONCE(rsp->gpnum) != rdp->gpnum) /* outside of lock */ {
> +		__rcu_pending_new_gp();
>  		return 1;
> +	}
>  
>  	/* Has an RCU GP gone long enough to send resched IPIs &c? */
>  	if (ACCESS_ONCE(rsp->completed) != ACCESS_ONCE(rsp->gpnum) &&
>  	    ((long)(ACCESS_ONCE(rsp->jiffies_force_qs) - jiffies) < 0 ||
> -	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0))
> +	     (rdp->n_rcu_pending_force_qs - rdp->n_rcu_pending) < 0)) {
> +		__rcu_pending_fqs();
>  		return 1;
> +	}
>  
>  	/* nothing to do */
>  	return 0;

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

* Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> [2009-02-16 14:39]:
> What is the easiest way for me to make it easy to trace the return path
> from __rcu_pending()?  Make each return path call an empty function
> located off where the compiler cannot see it, I guess...  Diagnostic
> patch along these lines below.  Frederic, Damien, could you please give
> it a go?  (And of course please let me know if something else is
> needed.)

As Frederic already sent a trace (made with another method), I'll
consider for now mine would not be very useful at this stage, so I'm
waiting for further instructions.

If you need further testing for me, do not hesitate to ask...

-- 
Damien

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:

> On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > 
> > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > 
> > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > microseconds apart, so that this function is consuming more 
> > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > something that makes it invisible to ftrace.
> > > 
> > > look at the latest trace please, that has even the most inline 
> > > raise-softirq method instrumented, so all the raising is 
> > > visible.
> > 
> > Ah, my apologies!  This time looking at:
> > 
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > 
> > 
> >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > [ . . . ]
> > 
> > Yikes!!!
> > 
> > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > but once per jiffy, and here it is called no less than 22 times in about
> > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> 
> BTW, the other question I have is "why do we need to call 
> rcu_pending() and rcu_check_callbacks() from the idle loop of 
> 32-bit x86, especially given that no other architecture does 
> this?".  Don't get me wrong, it would be good to get rcutree's 
> rcu_pending() to avoid spuriously saying that 
> rcu_check_callbacks() should be invoked, so I would still like 
> the trace with my patch, but...

There's no strong reason - we've been back and forth about RCU 
in the dynticks code. Mind sending a test patch for Damien to 
try?

	Ingo

Re: [Bug #12494] Sony backlight regression from 2.6.28 to 29-rc

[Norbert Preining]

On Sa, 14 Feb 2009, Rafael J. Wysocki wrote:
> The following bug entry is on the current list of known regressions
> from 2.6.28.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12494
> Subject		: Sony backlight regression from 2.6.28 to 29-rc
> Submitter	: Norbert Preining <preining-DX+603jRYB8@public.gmane.org>
> Date		: 2009-01-19 8:14 (27 days old)
> References	: http://marc.info/?l=linux-acpi&m=123235286829512&w=4
> Handled-By	: Mattia Dongili <malattia-k2GhghHVRtY@public.gmane.org>

2.6.29-rc5 activates config_fb automatically, so that should be fixed.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <preining-DX+603jRYB8@public.gmane.org>        Vienna University of Technology
Debian Developer <preining-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org>                         Debian TeX Group
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
The suit into which the man's body had been stuffed looked
as if it's only purpose in life was to demonstrate how
difficult it was to get this sort of body into a suit.
                 --- Douglas Adams, The Hitchhikers Guide to the Galaxy

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 10:46:57AM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> 
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > 
> > BTW, the other question I have is "why do we need to call 
> > rcu_pending() and rcu_check_callbacks() from the idle loop of 
> > 32-bit x86, especially given that no other architecture does 
> > this?".  Don't get me wrong, it would be good to get rcutree's 
> > rcu_pending() to avoid spuriously saying that 
> > rcu_check_callbacks() should be invoked, so I would still like 
> > the trace with my patch, but...
> 
> There's no strong reason - we've been back and forth about RCU 
> in the dynticks code. Mind sending a test patch for Damien to 
> try?

But of course!  ;-)

The following patch removes the call to rcu_pending() and
rcu_check_callbacks() from the x86 32-bit idle loop in order to
reduce the softirq load on idle systems.

Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---

 process_32.c |    3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index a546f55..bd4da2a 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -104,9 +104,6 @@ void cpu_idle(void)
 			check_pgt_cache();
 			rmb();
 
-			if (rcu_pending(cpu))
-				rcu_check_callbacks(cpu, 0);
-
 			if (cpu_is_offline(cpu))
 				play_dead();
 

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > 
> > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > 
> > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > microseconds apart, so that this function is consuming more 
> > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > something that makes it invisible to ftrace.
> > > 
> > > look at the latest trace please, that has even the most inline 
> > > raise-softirq method instrumented, so all the raising is 
> > > visible.
> > 
> > Ah, my apologies!  This time looking at:
> > 
> > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > 
> > 
> >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > [ . . . ]
> > 
> > Yikes!!!
> > 
> > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > but once per jiffy, and here it is called no less than 22 times in about
> > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > 
> > Hmmm...
> > 
> > Looks like we never return from:
> > 
> >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > 
> > Perhaps we are taking an interrupt immediately after the
> > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > And then deciding to go back into nohz mode at 799.521326, 117
> > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > which again raises RCU's softirq.
> > 
> > And the reason we are invoking rcu_check_callbacks() so often appears
> > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > which explains my failure to reproduce on a 64-bit system:
> > 
> > 	void cpu_idle(void)
> > 	{
> > 		int cpu = smp_processor_id();
> > 
> > 		current_thread_info()->status |= TS_POLLING;
> > 
> > 		/* endless idle loop with no priority at all */
> > 		while (1) {
> > 			tick_nohz_stop_sched_tick(1);
> > 			while (!need_resched()) {
> > 
> > 				check_pgt_cache();
> > 				rmb();
> > 
> > 				if (rcu_pending(cpu))
> > 					rcu_check_callbacks(cpu, 0);
> > 
> > 				if (cpu_is_offline(cpu))
> > 					play_dead();
> > 
> > 				local_irq_disable();
> > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > 				/* Don't trace irqs off for idle */
> > 				stop_critical_timings();
> > 				pm_idle();
> > 				start_critical_timings();
> > 			}
> > 			tick_nohz_restart_sched_tick();
> > 			preempt_enable_no_resched();
> > 			schedule();
> > 			preempt_disable();
> > 		}
> > 	}
> > 
> > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > each time.  I would expect rcu_pending() to return 0 most of the time,
> > but that apparently isn't the case with treercu...
> > 
> > What is the easiest way for me to make it easy to trace the return path
> > from __rcu_pending()?  Make each return path call an empty function
> > located off where the compiler cannot see it, I guess...  Diagnostic
> > patch along these lines below.  Frederic, Damien, could you please give
> > it a go?  (And of course please let me know if something else is
> > needed.)
> 
> 
> No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> inside the functions, ie:
> 
> __rcu_pending() {
> 	 /* pending_qs */
> }

Ah!!!  So if I were to put ftrace_printk() calls at strategic points
in the RCU code, that would be a good thing?

> I've converted your below patch with ftrace_printks and tested it under an old P2
> with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> lucky :-) 
> I guess I successfully reproduced the softirq/rcu overhead.
> Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> When I will have more time, I will fix that.
> 
> The trace is here http://dl.free.fr/uyWGgCbx4
> 
> It looks like it mostly returns 1 because of the waiting for quiescent state:
> 
> $ cat rcutrace | grep "/* pending_none" | wc -l
> 221
> $ cat rcutrace | grep "/* pending_qs" | wc -l
> 248
> $ cat rcutrace | grep "/* pending" | wc -l
> 469

Hmmm...  This looks like normal behavior.  Though I wonder if
rcu_check_callbacks() is recognizing that we are in the idle loop given
the large number of "pending_qs" entries.  To that end, would you be
willing to try the attached patch (on top of your ftrace_printk() patch)?

Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
determine when RCU has detected a quiescent state due to interrupting
from within it.

Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---

 rcutree.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b2fd602..fa14a0f 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
 
 		rcu_qsctr_inc(cpu);
 		rcu_bh_qsctr_inc(cpu);
+		ftrace_printk("rcu user/idle");
 
 	} else if (!in_softirq()) {
 
@@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
 		 */
 
 		rcu_bh_qsctr_inc(cpu);
+		ftrace_printk("rcu !softirq");
 	}
 	raise_softirq(RCU_SOFTIRQ);
 }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 07:11:42AM +0100, Damien Wyart wrote:
> * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> [2009-02-16 14:39]:
> > What is the easiest way for me to make it easy to trace the return path
> > from __rcu_pending()?  Make each return path call an empty function
> > located off where the compiler cannot see it, I guess...  Diagnostic
> > patch along these lines below.  Frederic, Damien, could you please give
> > it a go?  (And of course please let me know if something else is
> > needed.)
> 
> As Frederic already sent a trace (made with another method), I'll
> consider for now mine would not be very useful at this stage, so I'm
> waiting for further instructions.
> 
> If you need further testing for me, do not hesitate to ask...

I would be very interested to hear whether or not my patch removing
rcu_pending() and rcu_check_callbacks() from the x86_32 idle loop
fixes the problem for you.

						Thanx, Paul

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Damien Wyart]

Hello Paul,

> > There's no strong reason - we've been back and forth about RCU in
> > the dynticks code. Mind sending a test patch for Damien to try?

> But of course!  ;-)

With this patch, the problem goes away and system activity seems normal,
both on the P4 with high load and on the recent laptop.

Btw, could you explain briefly why, without this patch, a kernel
enabling classical RCU doesn't show the ksoftirqd problem at all?


Damien

> The following patch removes the call to rcu_pending() and
> rcu_check_callbacks() from the x86 32-bit idle loop in order to
> reduce the softirq load on idle systems.

> Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> ---

>  process_32.c |    3 ---
>  1 file changed, 3 deletions(-)

> diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
> index a546f55..bd4da2a 100644
> --- a/arch/x86/kernel/process_32.c
> +++ b/arch/x86/kernel/process_32.c
> @@ -104,9 +104,6 @@ void cpu_idle(void)
>  			check_pgt_cache();
>  			rmb();

> -			if (rcu_pending(cpu))
> -				rcu_check_callbacks(cpu, 0);
> -
>  			if (cpu_is_offline(cpu))
>  				play_dead();

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > 
> > > Hmmm...
> > > 
> > > Looks like we never return from:
> > > 
> > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > 
> > > Perhaps we are taking an interrupt immediately after the
> > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > And then deciding to go back into nohz mode at 799.521326, 117
> > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > which again raises RCU's softirq.
> > > 
> > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > which explains my failure to reproduce on a 64-bit system:
> > > 
> > > 	void cpu_idle(void)
> > > 	{
> > > 		int cpu = smp_processor_id();
> > > 
> > > 		current_thread_info()->status |= TS_POLLING;
> > > 
> > > 		/* endless idle loop with no priority at all */
> > > 		while (1) {
> > > 			tick_nohz_stop_sched_tick(1);
> > > 			while (!need_resched()) {
> > > 
> > > 				check_pgt_cache();
> > > 				rmb();
> > > 
> > > 				if (rcu_pending(cpu))
> > > 					rcu_check_callbacks(cpu, 0);
> > > 
> > > 				if (cpu_is_offline(cpu))
> > > 					play_dead();
> > > 
> > > 				local_irq_disable();
> > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > 				/* Don't trace irqs off for idle */
> > > 				stop_critical_timings();
> > > 				pm_idle();
> > > 				start_critical_timings();
> > > 			}
> > > 			tick_nohz_restart_sched_tick();
> > > 			preempt_enable_no_resched();
> > > 			schedule();
> > > 			preempt_disable();
> > > 		}
> > > 	}
> > > 
> > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > but that apparently isn't the case with treercu...
> > > 
> > > What is the easiest way for me to make it easy to trace the return path
> > > from __rcu_pending()?  Make each return path call an empty function
> > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > patch along these lines below.  Frederic, Damien, could you please give
> > > it a go?  (And of course please let me know if something else is
> > > needed.)
> > 
> > 
> > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > inside the functions, ie:
> > 
> > __rcu_pending() {
> > 	 /* pending_qs */
> > }
> 
> Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> in the RCU code, that would be a good thing?


Only when you are doing some debugging yes. But it is not a good thing to put an ftrace_printk
for code that has to be officially released since it adds a small overhead.
And actually ftrace_printk() is only for casual debugging, IMHO we shoudn't find any ftrace_printk
on the mainline code.

Instead, if you need some constant and defined probe inside your code, it's better to use
tracepoints, since they only add the overhead of a single branch check when they are off.


> > I've converted your below patch with ftrace_printks and tested it under an old P2
> > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > lucky :-) 
> > I guess I successfully reproduced the softirq/rcu overhead.
> > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > When I will have more time, I will fix that.
> > 
> > The trace is here http://dl.free.fr/uyWGgCbx4
> > 
> > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > 
> > $ cat rcutrace | grep "/* pending_none" | wc -l
> > 221
> > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > 248
> > $ cat rcutrace | grep "/* pending" | wc -l
> > 469
> 
> Hmmm...  This looks like normal behavior.  Though I wonder if
> rcu_check_callbacks() is recognizing that we are in the idle loop given
> the large number of "pending_qs" entries.  To that end, would you be
> willing to try the attached patch (on top of your ftrace_printk() patch)?
> 
> Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> determine when RCU has detected a quiescent state due to interrupting
> from within it.


Ok. I'm just fixing the orphans comments on the function graph tracer (the init_tasks
were not traced) and I test it.


> Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> ---
> 
>  rcutree.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..fa14a0f 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
>  
>  		rcu_qsctr_inc(cpu);
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu user/idle");
>  
>  	} else if (!in_softirq()) {
>  
> @@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
>  		 */
>  
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu !softirq");
>  	}
>  	raise_softirq(RCU_SOFTIRQ);
>  }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 04:39:25PM +0100, Damien Wyart wrote:
> Hello Paul,
> 
> > > There's no strong reason - we've been back and forth about RCU in
> > > the dynticks code. Mind sending a test patch for Damien to try?
> 
> > But of course!  ;-)
> 
> With this patch, the problem goes away and system activity seems normal,
> both on the P4 with high load and on the recent laptop.
> 
> Btw, could you explain briefly why, without this patch, a kernel
> enabling classical RCU doesn't show the ksoftirqd problem at all?

Classic RCU's rcu_pending() can afford to be much more conservative,
due to the easy availability of global information.  In contrast,
Hierarchical RCU will invoke the softirq in cases where the information
required to make an exact decision is off somewhere else in the tree.

There is also a possibility that Hierarchical RCU is for some reason
failing to detect that its rcu_check_callbacks() is being invoked from
the idle loop -- which would need to be fixed if it is really happening.
But given that the two rcu_check_callbacks() implementations are nearly
identical, I cannot see how this is happening.  (Wouldn't be the first
time that I failed to see how something was happening, though!)

							Thanx, Paul

> Damien
> 
> > The following patch removes the call to rcu_pending() and
> > rcu_check_callbacks() from the x86 32-bit idle loop in order to
> > reduce the softirq load on idle systems.
> 
> > Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> > ---
> 
> >  process_32.c |    3 ---
> >  1 file changed, 3 deletions(-)
> 
> > diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
> > index a546f55..bd4da2a 100644
> > --- a/arch/x86/kernel/process_32.c
> > +++ b/arch/x86/kernel/process_32.c
> > @@ -104,9 +104,6 @@ void cpu_idle(void)
> >  			check_pgt_cache();
> >  			rmb();
> 
> > -			if (rcu_pending(cpu))
> > -				rcu_check_callbacks(cpu, 0);
> > -
> >  			if (cpu_is_offline(cpu))
> >  				play_dead();
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Damien Wyart <damien.wyart-GANU6spQydw@public.gmane.org> wrote:

> Hello Paul,
> 
> > > There's no strong reason - we've been back and forth about RCU in
> > > the dynticks code. Mind sending a test patch for Damien to try?
> 
> > But of course!  ;-)
> 
> With this patch, the problem goes away and system activity 
> seems normal, both on the P4 with high load and on the recent 
> laptop.

Applied to tip:x86/urgent, thanks a lot for the patient testing 
and tracing!

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Frederic Weisbecker]

On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > 
> > > Hmmm...
> > > 
> > > Looks like we never return from:
> > > 
> > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > 
> > > Perhaps we are taking an interrupt immediately after the
> > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > And then deciding to go back into nohz mode at 799.521326, 117
> > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > which again raises RCU's softirq.
> > > 
> > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > which explains my failure to reproduce on a 64-bit system:
> > > 
> > > 	void cpu_idle(void)
> > > 	{
> > > 		int cpu = smp_processor_id();
> > > 
> > > 		current_thread_info()->status |= TS_POLLING;
> > > 
> > > 		/* endless idle loop with no priority at all */
> > > 		while (1) {
> > > 			tick_nohz_stop_sched_tick(1);
> > > 			while (!need_resched()) {
> > > 
> > > 				check_pgt_cache();
> > > 				rmb();
> > > 
> > > 				if (rcu_pending(cpu))
> > > 					rcu_check_callbacks(cpu, 0);
> > > 
> > > 				if (cpu_is_offline(cpu))
> > > 					play_dead();
> > > 
> > > 				local_irq_disable();
> > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > 				/* Don't trace irqs off for idle */
> > > 				stop_critical_timings();
> > > 				pm_idle();
> > > 				start_critical_timings();
> > > 			}
> > > 			tick_nohz_restart_sched_tick();
> > > 			preempt_enable_no_resched();
> > > 			schedule();
> > > 			preempt_disable();
> > > 		}
> > > 	}
> > > 
> > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > but that apparently isn't the case with treercu...
> > > 
> > > What is the easiest way for me to make it easy to trace the return path
> > > from __rcu_pending()?  Make each return path call an empty function
> > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > patch along these lines below.  Frederic, Damien, could you please give
> > > it a go?  (And of course please let me know if something else is
> > > needed.)
> > 
> > 
> > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > inside the functions, ie:
> > 
> > __rcu_pending() {
> > 	 /* pending_qs */
> > }
> 
> Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> in the RCU code, that would be a good thing?
> 
> > I've converted your below patch with ftrace_printks and tested it under an old P2
> > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > lucky :-) 
> > I guess I successfully reproduced the softirq/rcu overhead.
> > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > When I will have more time, I will fix that.
> > 
> > The trace is here http://dl.free.fr/uyWGgCbx4
> > 
> > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > 
> > $ cat rcutrace | grep "/* pending_none" | wc -l
> > 221
> > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > 248
> > $ cat rcutrace | grep "/* pending" | wc -l
> > 469
> 
> Hmmm...  This looks like normal behavior.  Though I wonder if
> rcu_check_callbacks() is recognizing that we are in the idle loop given
> the large number of "pending_qs" entries.  To that end, would you be
> willing to try the attached patch (on top of your ftrace_printk() patch)?
> 
> Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> determine when RCU has detected a quiescent state due to interrupting
> from within it.


Do you still need this trace even if your solution were applied on -tip ?

 
> Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> ---
> 
>  rcutree.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index b2fd602..fa14a0f 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -966,6 +966,7 @@ void rcu_check_callbacks(int cpu, int user)
>  
>  		rcu_qsctr_inc(cpu);
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu user/idle");
>  
>  	} else if (!in_softirq()) {
>  
> @@ -977,6 +978,7 @@ void rcu_check_callbacks(int cpu, int user)
>  		 */
>  
>  		rcu_bh_qsctr_inc(cpu);
> +		ftrace_printk("rcu !softirq");
>  	}
>  	raise_softirq(RCU_SOFTIRQ);
>  }

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Tue, Feb 17, 2009 at 11:37:42PM +0100, Frederic Weisbecker wrote:
> On Tue, Feb 17, 2009 at 07:10:46AM -0800, Paul E. McKenney wrote:
> > On Tue, Feb 17, 2009 at 05:34:23AM +0100, Frederic Weisbecker wrote:
> > > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > > 
> > > > > * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> > > > > 
> > > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > > microseconds apart, so that this function is consuming more 
> > > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > > something that makes it invisible to ftrace.
> > > > > 
> > > > > look at the latest trace please, that has even the most inline 
> > > > > raise-softirq method instrumented, so all the raising is 
> > > > > visible.
> > > > 
> > > > Ah, my apologies!  This time looking at:
> > > > 
> > > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > > 
> > > > 
> > > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > > [ . . . ]
> > > > 
> > > > Yikes!!!
> > > > 
> > > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > > but once per jiffy, and here it is called no less than 22 times in about
> > > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > > > 
> > > > Hmmm...
> > > > 
> > > > Looks like we never return from:
> > > > 
> > > >   799.521142 |   1)    <idle>-0    |          | tick_nohz_stop_sched_tick() {
> > > > 
> > > > Perhaps we are taking an interrupt immediately after the
> > > > local_irq_restore()?  And at 799.521209 deciding to exit nohz mode.
> > > > And then deciding to go back into nohz mode at 799.521326, 117
> > > > microseconds later, after which we re-invoke rcu_check_callbacks(),
> > > > which again raises RCU's softirq.
> > > > 
> > > > And the reason we are invoking rcu_check_callbacks() so often appears
> > > > to be in in arch/x86/kernel/process_32.c cpu_idle() near line 107,
> > > > which explains my failure to reproduce on a 64-bit system:
> > > > 
> > > > 	void cpu_idle(void)
> > > > 	{
> > > > 		int cpu = smp_processor_id();
> > > > 
> > > > 		current_thread_info()->status |= TS_POLLING;
> > > > 
> > > > 		/* endless idle loop with no priority at all */
> > > > 		while (1) {
> > > > 			tick_nohz_stop_sched_tick(1);
> > > > 			while (!need_resched()) {
> > > > 
> > > > 				check_pgt_cache();
> > > > 				rmb();
> > > > 
> > > > 				if (rcu_pending(cpu))
> > > > 					rcu_check_callbacks(cpu, 0);
> > > > 
> > > > 				if (cpu_is_offline(cpu))
> > > > 					play_dead();
> > > > 
> > > > 				local_irq_disable();
> > > > 				__get_cpu_var(irq_stat).idle_timestamp = jiffies;
> > > > 				/* Don't trace irqs off for idle */
> > > > 				stop_critical_timings();
> > > > 				pm_idle();
> > > > 				start_critical_timings();
> > > > 			}
> > > > 			tick_nohz_restart_sched_tick();
> > > > 			preempt_enable_no_resched();
> > > > 			schedule();
> > > > 			preempt_disable();
> > > > 		}
> > > > 	}
> > > > 
> > > > If we go in and out of nohz mode quickly, we will invoke rcu_pending()
> > > > each time.  I would expect rcu_pending() to return 0 most of the time,
> > > > but that apparently isn't the case with treercu...
> > > > 
> > > > What is the easiest way for me to make it easy to trace the return path
> > > > from __rcu_pending()?  Make each return path call an empty function
> > > > located off where the compiler cannot see it, I guess...  Diagnostic
> > > > patch along these lines below.  Frederic, Damien, could you please give
> > > > it a go?  (And of course please let me know if something else is
> > > > needed.)
> > > 
> > > 
> > > No, you don't need that, you can use ftrace_printk, it will generate a C-comment like
> > > inside the functions, ie:
> > > 
> > > __rcu_pending() {
> > > 	 /* pending_qs */
> > > }
> > 
> > Ah!!!  So if I were to put ftrace_printk() calls at strategic points
> > in the RCU code, that would be a good thing?
> > 
> > > I've converted your below patch with ftrace_printks and tested it under an old P2
> > > with rcu_tree and 1000 Hz. I made a trace during an idle state, and well, looks like I'm
> > > lucky :-) 
> > > I guess I successfully reproduced the softirq/rcu overhead.
> > > Please find the below patch to trace the rcu_pending return path, as well as the trace I made.
> > > Sorry, the trace is a bit buggy with sometimes flying orphans C like comments.
> > > When I will have more time, I will fix that.
> > > 
> > > The trace is here http://dl.free.fr/uyWGgCbx4
> > > 
> > > It looks like it mostly returns 1 because of the waiting for quiescent state:
> > > 
> > > $ cat rcutrace | grep "/* pending_none" | wc -l
> > > 221
> > > $ cat rcutrace | grep "/* pending_qs" | wc -l
> > > 248
> > > $ cat rcutrace | grep "/* pending" | wc -l
> > > 469
> > 
> > Hmmm...  This looks like normal behavior.  Though I wonder if
> > rcu_check_callbacks() is recognizing that we are in the idle loop given
> > the large number of "pending_qs" entries.  To that end, would you be
> > willing to try the attached patch (on top of your ftrace_printk() patch)?
> > 
> > Add ftrace_printk() to rcu_check_callbacks() to allow ftrace to
> > determine when RCU has detected a quiescent state due to interrupting
> > from within it.
> 
> Do you still need this trace even if your solution were applied on -tip ?

No, it was my confusion -- I later realized that your data above meant
that the force-quiescent-state code path was not being heavily exercised.
So no need for this trace!

							Thanx, Paul

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Sun, 2009-02-15 at 13:43 +0000, Matthew Garrett wrote:
> On Sat, Feb 14, 2009 at 09:38:23PM +0100, Rafael J. Wysocki wrote:
> 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> I think Eric sent a test patch for this. Did that get pushed?

I haven't done anything in the area of brightness control.

-- 
Eric Anholt
eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org                         eric.anholt-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Matthew Garrett]

On Tue, Feb 17, 2009 at 03:05:25PM -0800, Eric Anholt wrote:
> On Sun, 2009-02-15 at 13:43 +0000, Matthew Garrett wrote:
> > On Sat, Feb 14, 2009 at 09:38:23PM +0100, Rafael J. Wysocki wrote:
> > 
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.28.  Please verify if it still should be listed and let me know
> > > (either way).
> > 
> > I think Eric sent a test patch for this. Did that get pushed?
> 
> I haven't done anything in the area of brightness control.

This was drm failing to initialise because of incorrect mtrr setup and 
opregion not working as a result.

-- 
Matthew Garrett | mjg59-1xO5oi07KQx4cg9Nei1l7Q@public.gmane.org

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Jesse Barnes]

On Tuesday, February 17, 2009 3:13 pm Matthew Garrett wrote:
> On Tue, Feb 17, 2009 at 03:05:25PM -0800, Eric Anholt wrote:
> > On Sun, 2009-02-15 at 13:43 +0000, Matthew Garrett wrote:
> > > On Sat, Feb 14, 2009 at 09:38:23PM +0100, Rafael J. Wysocki wrote:
> > > > The following bug entry is on the current list of known regressions
> > > > from 2.6.28.  Please verify if it still should be listed and let me
> > > > know (either way).
> > >
> > > I think Eric sent a test patch for this. Did that get pushed?
> >
> > I haven't done anything in the area of brightness control.
>
> This was drm failing to initialise because of incorrect mtrr setup and
> opregion not working as a result.

I sent one out; it just made the ioremap in i915_initialize into an ioremap_wc 
for consistency.

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Ingo Molnar]

* Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:

> No, it was my confusion -- I later realized that your data 
> above meant that the force-quiescent-state code path was not 
> being heavily exercised. So no need for this trace!

Do you have any theory for why RCU was activated every 100-200 
microseconds, resulting in 20% ksoftirqd CPU use - and why the 
problem went away with classic-rcu?

	Ingo

Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

[Paul E. McKenney]

On Wed, Feb 18, 2009 at 01:38:01AM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org> wrote:
> 
> > No, it was my confusion -- I later realized that your data 
> > above meant that the force-quiescent-state code path was not 
> > being heavily exercised. So no need for this trace!
> 
> Do you have any theory for why RCU was activated every 100-200 
> microseconds, resulting in 20% ksoftirqd CPU use - and why the 
> problem went away with classic-rcu?

RCU was activated every 100-200 microseconds because the x86 32-bit
idle loop would call rcu_pending() and rcu_check_callbacks() in a tight
loop under some conditions.  This was happening to both classic and
tree RCU, but classic RCU has a more exact rcu_pending() check, and so
classic RCU's rcu_pending() always returns false, so that classic RCU's
rcu_check_callbacks() was never invoked, so that the raise_softirq()
is never called, so that control never passed to ksoftirqd, so that
things like "uptime" could not see the activity.

But the activity was occurring with classic RCU nevertheless.

</useful information>

<aside>

Interestingly enough, this is actually a symptom of a theoretical bug
in classic RCU (noted by Manfred Spraul some months ago).  Classic RCU
assumes that interrupts from dynticks idle mode don't run long enough
to run through a full grace period (which, in absence of truly broken
driver code, they do not).  Therefore, classic RCU removes all dynticks
idle CPUs from considation at the beginning of each grace period, so
that classic RCU's rcu_pending() doesn't have to concern itself with
other dyntick-idle CPUs.

When rcu_pending() is invoked once per jiffy, the additional checking
that tree RCU must do is in the noise, but not so when called repeatedly
from the idle loop.

</aside>

							Thanx, Paul

Re: [Bug #12705] X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

Jesse Barnes [Tue, Feb 17, 2009 at 03:23:57PM -0800]:
> On Tuesday, February 17, 2009 3:13 pm Matthew Garrett wrote:
> > This was drm failing to initialise because of incorrect mtrr setup and
> > opregion not working as a result.
> 
> I sent one out; it just made the ioremap in i915_initialize into an ioremap_wc 
> for consistency.

If you refer to <200902131104.04318.jbarnes@virtuousgeek.org>,
as stated in <20090216131351.GA4839@denkbrett.schottelius.org>, it does not
fix the problem here.

Sincerly,

Nico

-- 
Think about Free and Open Source Software (FOSS).
http://nico.schottelius.org/documentations/foss/the-term-foss/

PGP: BFE4 C736 ABE5 406F 8F42  F7CF B8BE F92A 9885 188C

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: X200: Brightness broken since 2.6.29-rc4-58-g4c098bc

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

Issue persisting on 2.6.29-rc5-ikn-00168-gba95fd4

Nico

-- 
Think about Free and Open Source Software (FOSS).
http://nico.schottelius.org/documentations/foss/the-term-foss/

PGP: BFE4 C736 ABE5 406F 8F42  F7CF B8BE F92A 9885 188C

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Bug #12610] sync-Regression in 2.6.28.2?

[Theodore Tso]

On Sat, Feb 14, 2009 at 09:38:18PM +0100, Rafael J. Wysocki wrote:
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
> Subject		: sync-Regression in 2.6.28.2?
> Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt-jq1tPX9l7E6ELgA04lAiVw@public.gmane.org>
> Date		: 2009-01-27 9:35 (19 days old)
> References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4

This fix for this has landed in mainline post 2.6.29-rc5, as commit
2acf2c.  The deadlock is technically not a regression but it was made
*much* more likely to show up because of commit 31a1266: ("mm:
write_cache_pages cyclic fix, which show up in 2.6.28.1").  

Commit 3a4c68 in mainline backs up the change made in 31a1266, so you
probably won't see this much after 2.6.28.6 (when 3a4c68 was
backported to 2.6.28.y), but we should get commit 2acf2c pushed to
2.6.28.x and 2.6.27.y to completely solve the deadlock problem.

						- Ted

Re: [Bug #12610] sync-Regression in 2.6.28.2?

[Rafael J. Wysocki]

On Saturday 21 February 2009, Theodore Tso wrote:
> On Sat, Feb 14, 2009 at 09:38:18PM +0100, Rafael J. Wysocki wrote:
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
> > Subject		: sync-Regression in 2.6.28.2?
> > Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt-jq1tPX9l7E6ELgA04lAiVw@public.gmane.org>
> > Date		: 2009-01-27 9:35 (19 days old)
> > References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4
> 
> This fix for this has landed in mainline post 2.6.29-rc5, as commit
> 2acf2c.  The deadlock is technically not a regression but it was made
> *much* more likely to show up because of commit 31a1266: ("mm:
> write_cache_pages cyclic fix, which show up in 2.6.28.1").  
> 
> Commit 3a4c68 in mainline backs up the change made in 31a1266, so you
> probably won't see this much after 2.6.28.6 (when 3a4c68 was
> backported to 2.6.28.y), but we should get commit 2acf2c pushed to
> 2.6.28.x and 2.6.27.y to completely solve the deadlock problem.

Thanks for the update.

I'll close the bug since it's fixed in the mainline.  Please tell Greg which
patches to put into the -stable kernels.

Thanks,
Rafael


> 
> 						- Ted
> 
> 


-- 
Everyone knows that debugging is twice as hard as writing a program
in the first place.  So if you're as clever as you can be when you write it,
how will you ever debug it? --- Brian Kernighan

Re: [Bug #12610] sync-Regression in 2.6.28.2?

[Greg KH]

On Sun, Feb 22, 2009 at 11:02:09AM +0100, Rafael J. Wysocki wrote:
> On Saturday 21 February 2009, Theodore Tso wrote:
> > On Sat, Feb 14, 2009 at 09:38:18PM +0100, Rafael J. Wysocki wrote:
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12610
> > > Subject		: sync-Regression in 2.6.28.2?
> > > Submitter	: Ralf Hildebrandt <Ralf.Hildebrandt-jq1tPX9l7E6ELgA04lAiVw@public.gmane.org>
> > > Date		: 2009-01-27 9:35 (19 days old)
> > > References	: http://marc.info/?l=linux-kernel&m=123304977706620&w=4
> > 
> > This fix for this has landed in mainline post 2.6.29-rc5, as commit
> > 2acf2c.  The deadlock is technically not a regression but it was made
> > *much* more likely to show up because of commit 31a1266: ("mm:
> > write_cache_pages cyclic fix, which show up in 2.6.28.1").  
> > 
> > Commit 3a4c68 in mainline backs up the change made in 31a1266, so you
> > probably won't see this much after 2.6.28.6 (when 3a4c68 was
> > backported to 2.6.28.y), but we should get commit 2acf2c pushed to
> > 2.6.28.x and 2.6.27.y to completely solve the deadlock problem.
> 
> Thanks for the update.
> 
> I'll close the bug since it's fixed in the mainline.  Please tell Greg which
> patches to put into the -stable kernels.

Ted, can you forward the needed patches to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, or just
tell us to take those git commits directly?

thanks,

greg k-h

Re: [Bug #12610] sync-Regression in 2.6.28.2?

[Theodore Tso]

On Sun, Feb 22, 2009 at 08:35:36PM -0800, Greg KH wrote:
> 
> Ted, can you forward the needed patches to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, or just
> tell us to take those git commits directly?
> 

Already in the for-stable and for-stable-2.6.27 branches of the ext4
git tree.  I was going to see if some folks would do some quick
testing (even though it it was mostely a straightforward git
cherry-pick) before sending it to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; you should see
them in 2-3 days or so.  If you need them sooner, I can accelerate
when I push them to you....

					- Ted

Re: [stable] [Bug #12610] sync-Regression in 2.6.28.2?

[Greg KH]

On Mon, Feb 23, 2009 at 12:37:36AM -0500, Theodore Tso wrote:
> On Sun, Feb 22, 2009 at 08:35:36PM -0800, Greg KH wrote:
> > 
> > Ted, can you forward the needed patches to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, or just
> > tell us to take those git commits directly?
> > 
> 
> Already in the for-stable and for-stable-2.6.27 branches of the ext4
> git tree.  I was going to see if some folks would do some quick
> testing (even though it it was mostely a straightforward git
> cherry-pick) before sending it to stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; you should see
> them in 2-3 days or so.  If you need them sooner, I can accelerate
> when I push them to you....

Sooner would be good, want to get another review cycle started tomorrow.

But if you miss it, it'll only be a week or so until the next cycle :)

thanks,

greg k-h