[Bug 9745] New: CPU is waking up too often

[Bug 9745] New: CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745

           Summary: CPU is waking up too often
           Product: Power Management
           Version: 2.5
     KernelVersion: 2.6.24-rc7
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: cpufreq
        AssignedTo: cpufreq@www.linux.org.uk
        ReportedBy: schoenfeld@in-medias-res.com


Latest working kernel version: 2.6.22
Earliest failing kernel version: 2.6.24-rc7
Distribution: Debian GNU/Linux Sid (Unstable)
Hardware Environment:
CPU: Mobile AMD Sempron(tm) Processor 3300+
System is a Fujitsu Siemens K7610 mobile system.
lspci:
00:00.0 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:00.1 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:00.2 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:00.3 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:00.4 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:00.7 Host bridge: VIA Technologies, Inc. K8M800 Host Bridge
00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800/K8T890
South]
00:09.0 CardBus bridge: ENE Technology Inc CB1410 Cardbus Controller (rev 01)
00:0a.0 Ethernet controller: Atheros Communications, Inc. AR5005G 802.11abg NIC
(rev 01)
00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 80)
00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 80)
00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 80)
00:10.3 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 82)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8235 ISA Bridge
00:11.1 IDE interface: VIA Technologies, Inc.
VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237
AC97 Audio Controller (rev 50)
00:11.6 Communication controller: VIA Technologies, Inc. AC'97 Modem Controller
(rev 80)
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 74)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address
Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
01:00.0 VGA compatible controller: VIA Technologies, Inc. S3 Unichrome Pro VGA
Adapter (rev 01)

Software Environment:
Debian Sid, Xorg with openchrome drivers from openchrome.org, Software typical
for a Desktop system

Problem Description:
With 2.6.22 according to powertop my system is waking up between 55times a
second and 200 times a second with an average at about 80 times a second.
When I now boot 2.6.24rc7 this number increases to an average of 2500
wakeups/second with (seldom but happening) tops above the 10000 wake-ups/second
mark (to top values of 28000 wake-ups/second!!). This way the CPU is always
running at C0 (99,2%).

Top causes for wakeups (according to powertop):
  42.6% ( 17.6)       <interrupt> : extra timer interrupt 
  21.8% (  9.0)       <interrupt> : ide1
   4.8% (  2.0)       <interrupt> : acpi
   4.1% (  1.7)           xfsbufd : schedule_timeout (process_timeout)
   3.4% (  1.4)              Xorg : do_setitimer (it_real_fn)
   2.4% (  1.0)            dhcdbd : schedule_timeout (process_timeout)
   2.4% (  1.0)         nm-applet : schedule_timeout (process_timeout)
   1.5% (  0.6)     <kernel core> : neigh_table_init_no_netlink
(neigh_periodic_timer)
   1.2% (  0.5)          ifconfig : __netdev_watchdog_up (dev_watchdog)
   1.2% (  0.5)   hald-addon-stor : schedule_timeout (process_timeout)
   1.2% (  0.5)    NetworkManager : schedule_timeout (process_timeout)
   1.2% (  0.5)       gnome-panel : schedule_timeout (process_timeout)
   1.2% (  0.5)   <kernel module> : neigh_table_init_no_netlink
(neigh_periodic_timer)
   1.2% (  0.5)     <kernel core> : queue_delayed_work_on
(delayed_work_timer_fn)
   0.7% (  0.3)       <interrupt> : eth0
   0.7% (  0.3)     <kernel core> : neigh_update (neigh_timer_handler)
   0.7% (  0.3)   <kernel module> : register_sound_midi
(irlmp_discovery_timer_expired)
   0.7% (  0.3)   gnome-cups-icon : schedule_timeout (process_timeout)
   0.7% (  0.3)   gnome-power-man : schedule_timeout (process_timeout)
   0.5% (  0.2)       <interrupt> : ide0
   0.5% (  0.2)    mapping-daemon : schedule_timeout (process_timeout)
   0.5% (  0.2)   gnome-cups-icon : futex_wait (hrtimer_wakeup)
   0.5% (  0.2)              init : schedule_timeout (process_timeout)
   0.5% (  0.2)             cupsd : schedule_timeout (process_timeout)
   0.5% (  0.2)          nautilus : schedule_timeout (process_timeout)
   0.5% (  0.2)   gnome-settings- : schedule_timeout (process_timeout)
   0.5% (  0.2)   <kernel module> : ide_do_rw_disk (ledtrig_ide_timerfunc)
   0.2% (  0.1)            urxvtd : schedule_timeout (process_timeout)
   0.2% (  0.1)           syslogd : do_setitimer (it_real_fn)
   0.2% (  0.1)          gconfd-2 : schedule_timeout (process_timeout)
   0.2% (  0.1)     <kernel core> : page_writeback_init (wb_timer_fn)

Steps to reproduce:

Boot system with 2.6.24-rc7


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745


trenn@suse.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tglx@linutronix.de,
                   |                            |trenn@suse.de, mingo@elte.hu




------- Comment #1 from trenn@suse.de  2008-01-14 05:12 -------
This has nothing to do with cpufreq, but with timers.
I also saw a similar bugreport somewhere else.

I cannot help here (but would like to learn how to debug this...), if you have
some luck Ingo Molnar or Thomas Gleixner can give you some hints how to debug
this.

I expect this has been introduced (just a guess, git should help) by the
clockevents/highres/tickless timer patches. If you could prove this with git,
you should get some help from the author of the patchsets...


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #2 from tglx@linutronix.de  2008-01-14 05:32 -------
Patrick, can you please upload your .config, a full boot log and the output of
/proc/timer_list ?

Thanks,
       tglx


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #3 from schoenfeld@in-medias-res.com  2008-01-14 05:45 -------
Created an attachment (id=14450)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14450&action=view)
kernel config


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #4 from schoenfeld@in-medias-res.com  2008-01-14 05:45 -------
Created an attachment (id=14451)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14451&action=view)
bootlog


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #5 from schoenfeld@in-medias-res.com  2008-01-14 05:46 -------
Created an attachment (id=14452)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14452&action=view)
output of cat /proc/timer_list


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #6 from schoenfeld@in-medias-res.com  2008-01-14 05:48 -------
Thomas Renninger,
yeah, you are right. It first looked like cpufreq scaling wouldn't happen and I
didn't really knew which category would fit. Neither do I do really know now.
(Subsections of Timers is quiet confusing for me).

Thomas Gleixner,
I'm not exactly sure what do you mean by "full boot log"? A full boot log of
the problematic kernel or a log including boots of working kernels? I've added
the part of /var/log/kern.log where I booted the kernel the last time.

Hope that helps.

Regards,
Patrick


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #7 from tglx@linutronix.de  2008-01-14 06:01 -------
> I'm not exactly sure what do you mean by "full boot log"? A full boot log of
> the problematic kernel or a log including boots of working kernels? I've added
> the part of /var/log/kern.log where I booted the kernel the last time.

boot log of the problematic kernel. kern.log is fine. The output of
dmesg is easier to read though.

According to the output of /proc/timer_list there is no fast rearming
timer, so it's likely to be something else.

Can you please do the following:

# cat /proc/interrupts >irqs.txt; sleep 10; cat /proc/interrupts >>irqs.txt

and upload irqs.txt 

Thanks,
        tglx


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #8 from schoenfeld@in-medias-res.com  2008-01-14 06:06 -------
Created an attachment (id=14453)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14453&action=view)
output file of at /proc/interrupts >irqs.txt; sleep 10; cat /proc/interrupts
>>irqs.txt

Here you are.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745


tglx@linutronix.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |arjan@infradead.org




------- Comment #9 from tglx@linutronix.de  2008-01-14 06:27 -------
Hmm. The deltas are:                                                            

   0:   742                                                                     
  14:     4                                                                     
  15:    90                                                                     
  18:     1                                                                     
 LOC:   724                                                                     
-----------                                                                     
       1561                                                                     

1561/10 ~= 156 interrupts/sec                                                   

Not amazing, but not really in the range of 2500+.                              


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #10 from schoenfeld@in-medias-res.com  2008-01-14 06:38 -------
Hmm. So probably this really is a powertop issue. However why isn't it
triggered by 2.6.22? Btw. you can see [1] for an example on a powertop output,
where the value is beyond 16000.

[1] http://just-imho.net/16000wakeupspersecond.png


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #11 from trenn@suse.de  2008-01-14 06:46 -------
Maybe this is related to:
https://bugzilla.novell.com/show_bug.cgi?id=335086
and
https://bugzilla.novell.com/show_bug.cgi?id=334170 (there only after s2ram
case)
and
https://bugs.launchpad.net/ubuntu/+bug/145377

there also powertop shows extrem interrupt activity and blacklisting
yenta_socket driver helps.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #12 from schoenfeld@in-medias-res.com  2008-01-14 06:51 -------
Yeah, indeed it seems related to 335086 @ Novell Bugzilla. If I do rmmod
processor (i need to force it but that just besides) then the situation is
relative normal with an average of 100-200 wakeups. Thats better then 2500, but
still worse as with 2.6.22 (there the average is 80 and sometimes less then 50)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #13 from trenn@suse.de  2008-01-14 07:01 -------
Then a workaround for you should be processor.max_cstate=2 boot param, if this
does not help processor.max_cstate=1.
But it would be great to know more about the root cause (and get this fixed to
fully support the C-states).
Could you try to keep processor driver without max_cstate workaround and
blacklist yenta_socket, e.g. add a line:
install yenta_socket /bin/echo "Yenta socket blacklisted, it will not get
loaded"
to /etc/modprobe.conf
I already asked Arjan about the yenta_socket problem and he said it would be
the yenta driver's fault... As there seem to be quite some people affected, it
would be great to bring them together and find out a bit more about the problem
(Did this machine ever worked correctly with C-states? If yes, one might want
to start a git bisect...).


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #14 from schoenfeld@in-medias-res.com  2008-01-14 07:19 -------
Thomas Renninger,

the current workaround for me is to use 2.6.22 which seems not to be affected
by the problem. However in my case it does help nothing to blacklist
yenta_socket.
Well, I haven't yet tried the max_cstate thing as it requires me to reboot
again.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #15 from tglx@linutronix.de  2008-01-14 07:40 -------
> the current workaround for me is to use 2.6.22 which seems not to be affected
> by the problem. However in my case it does help nothing to blacklist
> yenta_socket.
> Well, I haven't yet tried the max_cstate thing as it requires me to reboot
> again.

If you decide to reboot, could you please try to capture something,
which shows us the large number of interrupts (via the shell command I
gave you before).

I looked at your .config again. That's the .22 one. Is the .23
significantly different? Can you please upload it as well.

Thanks,
        tglx


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #16 from schoenfeld@in-medias-res.com  2008-01-14 08:42 -------
(In reply to comment #15)
> If you decide to reboot, could you please try to capture something,
> which shows us the large number of interrupts (via the shell command I
> gave you before).

You mean With those above stated workarounds?

> I looked at your .config again. That's the .22 one. Is the .23
> significantly different? Can you please upload it as well.

Oh. You are right. I'll attach the one for .24. Sorry, my mistake.
However I don't have a .23 version ;)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #17 from schoenfeld@in-medias-res.com  2008-01-14 08:43 -------
Created an attachment (id=14454)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14454&action=view)
kernel config (the right one :o)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745


schoenfeld@in-medias-res.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #14450|0                           |1
        is obsolete|                            |




-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #18 from schoenfeld@in-medias-res.com  2008-01-14 23:58 -------
Okay I did some additional testing. With the cstate options from above I
experience the following result (NOK = NOT OK)

max_cstate = 2 = NOK
max_cstate = 1 = NOK (increased (!) value in powertop!)

I have created files with the cat /proc interupts.. command again and will
attach them in a minute.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #19 from schoenfeld@in-medias-res.com  2008-01-14 23:59 -------
Created an attachment (id=14460)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14460&action=view)
ircqs with max_cstate=1


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #20 from schoenfeld@in-medias-res.com  2008-01-15 00:00 -------
Created an attachment (id=14461)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=14461&action=view)
irqs with max_cstate=2


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #21 from schoenfeld@in-medias-res.com  2008-01-15 00:08 -------
BTW. interesting: If I do cat /proc.. on a 2.6.22 system the interrupts/sec is
even higher. So is this probably really a bug in powertop?


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745





------- Comment #22 from arjan@infradead.org  2008-01-15 06:35 -------
max_cstate=1 is a red herring btw
the kernel doesn't count wakeups for C1 so powertop doesnt get to see them
either; that makes it a counting game not something reflecting what is actually
happening on the system


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745


protasnb@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |protasnb@gmail.com




------- Comment #23 from protasnb@gmail.com  2008-06-03 22:28 -------
What is the status on this problem, still exists with recent kernel?
Thanks.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 9745] CPU is waking up too often

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=9745


davej@codemonkey.org.uk changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|cpufreq@www.linux.org.uk    |platform_i386@kernel-
                   |                            |bugs.osdl.org
          Component|cpufreq                     |i386
            Product|Power Management            |Platform Specific/Hardware




-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
You are the assignee for the bug, or are watching the assignee.