2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Mark Knecht]

Hi,
   I've been running 2.6.21.3-rt9 and a couple of -rc predecessors for
a few weeks and this minor problem has been consistent on all versions
I've run. On this AMD64 machine I still have 2.6.17-rt8 and 2.6.18-rt7
and neither of those versions cause the problem. I am currently
building the matching vanilla kernel as well as the 2.6.22-rc3 kernel
to see if it's in those versions.

   I see today that the same problem has been reported by another
AMD64 user on a distribution mail list so I think the problem isn't
specific to my hardware anymore. I had previously thought so and
therefore didn't report this right away thinking I'd do more debug.

http://search.gmane.org/?query=IRQ+23&group=gmane.linux.distributions.64studio

   Since my IRQ23 is shared on my machine by Ethernet and USB I have
tried detaching all network cables and turning off all USB devices but
the problem persists.

   I'll report back what happens with the vanilla-kernels. If a binary
regression search is required of everything between 2.6.18 and 2.6.21
to look for where the problem crept in that will take some time so any
guidance on where to start would be helpful.

   If more info is desired please don't hesitate to ask.

Cheers,
Mark


mark@lightning ~ $ uname -a
Linux lightning 2.6.21.3-rt9 #1 PREEMPT RT Sat Jun 9 09:49:32 PDT 2007
x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
mark@lightning ~ $


mark@lightning ~ $ top

top - 10:02:17 up 3 min,  2 users,  load average: 0.00, 0.00, 0.00
Tasks:  88 total,   2 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3%us,  0.0%sy,  0.0%ni, 95.7%id,  0.0%wa,  3.0%hi,  0.0%si,  0.0%st
Mem:   1026612k total,   190316k used,   836296k free,     8124k buffers
Swap:  2008084k total,        0k used,  2008084k free,    74616k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 947 root     -76  -5     0    0    0 S  2.7  0.0   0:05.58 IRQ-23
 9533 root      15   0  324m  13m 7420 S  1.3  1.4   0:03.24 X
   1 root      15   0  2668  588  500 S  0.0  0.1   0:00.27 init
   2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 posix_cpu_timer
   3 root     -51   0     0    0    0 S  0.0  0.0   0:00.00 softirq-high/0



mark@lightning ~ $ cat /proc/interrupts
          CPU0
 0:        212   IO-APIC-edge      timer
 1:        296   IO-APIC-edge      i8042
 6:          3   IO-APIC-edge      floppy
 7:          0   IO-APIC-edge      lpptest
 8:          0   IO-APIC-edge      rtc
 9:          0   IO-APIC-fasteoi   acpi
 12:       8690   IO-APIC-edge      i8042
 14:         62   IO-APIC-edge      ide0
 16:          0   IO-APIC-fasteoi   hdsp
 18:      14496   IO-APIC-fasteoi   ohci1394, radeon@pci:0000:01:00.0
 20:          2   IO-APIC-fasteoi   ehci_hcd:usb1
 21:       7724   IO-APIC-fasteoi   libata
 22:          0   IO-APIC-fasteoi   libata, NVidia CK804
 23:    1739277   IO-APIC-fasteoi   ohci_hcd:usb2, eth0
NMI:          0
LOC:      59688
ERR:          0
mark@lightning ~ $


mark@lightning ~ $ uname -a
Linux lightning 2.6.18-rt7 #4 PREEMPT Sun Apr 8 17:32:31 PDT 2007
x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
mark@lightning ~ $


mark@lightning ~ $ top

top - 10:09:51 up 1 min,  2 users,  load average: 0.00, 0.00, 0.00
Tasks:  83 total,   1 running,  82 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.3%sy,  0.0%ni, 95.4%id,  2.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1026840k total,   185688k used,   841152k free,     7904k buffers
Swap:  2008084k total,        0k used,  2008084k free,    74172k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9024 root      15   0  324m  13m 7072 S  1.7  1.3   0:01.00 X
   7 root      -2   0     0    0    0 S  0.3  0.0   0:01.02 softirq-block/0
 9912 mark      15   0 10600 1304  960 R  0.3  0.1   0:00.01 top
   1 root      15   0  2676  588  500 S  0.0  0.1   0:00.31 init
   2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 posix_cpu_timer
   3 root      -2   0     0    0    0 S  0.0  0.0   0:00.00 softirq-high/0


lightning ~ # lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97
Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B60
[Radeon X300 (PCIE)]
01:00.1 Display controller: ATI Technologies Inc RV370 [Radeon X300SE]
05:06.0 Multimedia audio controller: Xilinx Corporation RME Hammerfall
DSP (rev 68)
05:08.0 FireWire (IEEE 1394): Texas Instruments TSB82AA2 IEEE-1394b
Link Layer Controller (rev 01)
lightning ~ #

lightning ~ # lsmod
Module                  Size  Used by
it87                   22032  0
hwmon_vid               2880  1 it87
i2c_isa                 6144  1 it87
eeprom                  7248  0
snd_seq_midi            7296  0
snd_pcm_oss            40160  0
snd_mixer_oss          15168  1 snd_pcm_oss
snd_seq_dummy           3460  0
snd_seq_oss            29696  0
snd_seq_midi_event      7040  2 snd_seq_midi,snd_seq_oss
snd_seq                48000  6
snd_seq_midi,snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
realtime                9416  0
sbp2                   22084  0
radeon                115488  2
drm                    84392  3 radeon
snd_hdsp               47812  0
snd_rawmidi            21216  2 snd_seq_midi,snd_hdsp
snd_seq_device          7252  5
snd_seq_midi,snd_seq_dummy,snd_seq_oss,snd_seq,snd_rawmidi
snd_hwdep               8456  1 snd_hdsp
ohci1394               31368  0
ieee1394               91024  2 sbp2,ohci1394
snd_intel8x0           32296  1
snd_ac97_codec        107288  1 snd_intel8x0
ac97_bus                3712  1 snd_ac97_codec
snd_pcm                74440  4 snd_pcm_oss,snd_hdsp,snd_intel8x0,snd_ac97_codec
snd_timer              20424  2 snd_seq,snd_pcm
snd                    52264  14
snd_pcm_oss,snd_mixer_oss,snd_seq_oss,snd_seq,snd_hdsp,snd_rawmidi,snd_seq_device,snd_hwdep,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
snd_page_alloc          8208  3 snd_hdsp,snd_intel8x0,snd_pcm
k8temp                  5888  0
i2c_nforce2             6016  0
i2c_core               21056  4 it87,i2c_isa,eeprom,i2c_nforce2
lightning ~ #

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Ingo Molnar]

* Mark Knecht <markknecht@gmail.com> wrote:

> 23:    1739277   IO-APIC-fasteoi   ohci_hcd:usb2, eth0

> 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)

ok, that's forcedeth. Does that behavior go away if you revert the 
change below, via patch -p0 -R ?

but in general, -rt will track IRQ cpu usage accurately, while mainline 
does not measure it at all when in idle. So a frequent IRQ on mainline 
might be using up similar amounts of of CPU time as well, but it's not 
reported.

	Ingo

---
 drivers/net/forcedeth.c |   12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

Index: linux-rt.q/drivers/net/forcedeth.c
===================================================================
--- linux-rt.q.orig/drivers/net/forcedeth.c
+++ linux-rt.q/drivers/net/forcedeth.c
@@ -800,7 +800,7 @@ struct fe_priv {
  * Maximum number of loops until we assume that a bit in the irq mask
  * is stuck. Overridable with module param.
  */
-static int max_interrupt_work = 5;
+static int max_interrupt_work = 50;
 
 /*
  * Optimization can be either throuput mode or cpu mode
@@ -812,7 +812,7 @@ enum {
 	NV_OPTIMIZATION_MODE_THROUGHPUT,
 	NV_OPTIMIZATION_MODE_CPU
 };
-static int optimization_mode = NV_OPTIMIZATION_MODE_THROUGHPUT;
+static int optimization_mode = NV_OPTIMIZATION_MODE_CPU;
 
 /*
  * Poll interval for timer irq
@@ -3108,9 +3108,17 @@ static int nv_napi_poll(struct net_devic
 	int retcode;
 
 	if (np->desc_ver == DESC_VER_1 || np->desc_ver == DESC_VER_2) {
+		spin_lock_irqsave(&np->lock, flags);
+		nv_tx_done(dev);
+		spin_unlock_irqrestore(&np->lock, flags);
+
 		pkts = nv_rx_process(dev, limit);
 		retcode = nv_alloc_rx(dev);
 	} else {
+		spin_lock_irqsave(&np->lock, flags);
+		nv_tx_done_optimized(dev, np->tx_ring_size);
+		spin_unlock_irqrestore(&np->lock, flags);
+
 		pkts = nv_rx_process_optimized(dev, limit);
 		retcode = nv_alloc_rx_optimized(dev);
 	}

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Mark Knecht]

On 6/10/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Mark Knecht <markknecht@gmail.com> wrote:
>
> > 23:    1739277   IO-APIC-fasteoi   ohci_hcd:usb2, eth0
>
> > 00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
>
> ok, that's forcedeth. Does that behavior go away if you revert the
> change below, via patch -p0 -R ?
>
> but in general, -rt will track IRQ cpu usage accurately, while mainline
> does not measure it at all when in idle. So a frequent IRQ on mainline
> might be using up similar amounts of of CPU time as well, but it's not
> reported.
>
>         Ingo
>

Ingo,
   GMail is being kranky. Can you please send the patch as a zipped attachment?

Thanks,
Mark

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Ingo Molnar]

* Mark Knecht <markknecht@gmail.com> wrote:

>   GMail is being kranky. Can you please send the patch as a zipped 
>   attachment?

you can pick it up from:

   http://people.redhat.com/mingo/realtime-preempt/testing/forcedeth-rt-tweak.patch

	Ingo

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Mark Knecht]

On 6/10/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Mark Knecht <markknecht@gmail.com> wrote:
>
> >   GMail is being kranky. Can you please send the patch as a zipped
> >   attachment?
>
> you can pick it up from:
>
>    http://people.redhat.com/mingo/realtime-preempt/testing/forcedeth-rt-tweak.patch
>
>         Ingo
>

Hi Ingo,
   Yep, that solved it on my machine. I did have to patch using 'patch
-p1 -R' I think that's because the paths in your patch file are maybe
based on your work setup? I don't know. I'm amazed most of the time
that I can even do this stuff but with your help it seems to work out.
Thanks.

   Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do
see a little bit of CPU  consumption from X and firefox but that seems
pretty reasonable.

   If there is more you'd like me to look at or do on my end please let me know.

Cheers,
Mark

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Ingo Molnar]

* Mark Knecht <markknecht@gmail.com> wrote:

> Hi Ingo,
>   Yep, that solved it on my machine. I did have to patch using 'patch 
> -p1 -R' I think that's because the paths in your patch file are maybe 
> based on your work setup? I don't know. I'm amazed most of the time 
> that I can even do this stuff but with your help it seems to work out. 
> Thanks.
> 
>   Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do 
> see a little bit of CPU consumption from X and firefox but that seems 
> pretty reasonable.

ok, thanks - i took that change out from -rt.

	Ingo

Re: 2.6.21-rt9 - IRQ23 consuming a steady 2.7% of CPU

[Mark Knecht]

On 6/11/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Mark Knecht <markknecht@gmail.com> wrote:
>
> > Hi Ingo,
> >   Yep, that solved it on my machine. I did have to patch using 'patch
> > -p1 -R' I think that's because the paths in your patch file are maybe
> > based on your work setup? I don't know. I'm amazed most of the time
> > that I can even do this stuff but with your help it seems to work out.
> > Thanks.
> >
> >   Anyway, top no longer reports the 2.7% CPU usage from IRQ23. I do
> > see a little bit of CPU consumption from X and firefox but that seems
> > pretty reasonable.
>
> ok, thanks - i took that change out from -rt.
>
>         Ingo
>
Thanks Ingo. Have a great week.

- Mark