* arch/i386/Kconfig: CONFIG_IRQBALANCE Description @ 2004-03-22 23:02 Rusty Russell 2004-03-23 17:16 ` John Stoffel 0 siblings, 1 reply; 6+ messages in thread From: Rusty Russell @ 2004-03-22 23:02 UTC (permalink / raw) To: lkml - Kernel Mailing List; +Cc: Ingo Molnar I came across this description: config IRQBALANCE bool "Enable kernel irq balancing" depends on SMP && X86_IO_APIC default y help The default yes will allow the kernel to do irq load balancing. Saying no will keep the kernel from doing irq load balancing. Holy shades of preempt! Help messages should describe the advantages and disadvantages of an option: little more disclosure would really help here. IMHO, if you're having trouble describing the pros and cons of an option to a user of average technical ability (think sysadmin), it's not a good config option. Rusty. -- Anyone who quotes me in their signature is an idiot -- Rusty Russell ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: arch/i386/Kconfig: CONFIG_IRQBALANCE Description 2004-03-22 23:02 arch/i386/Kconfig: CONFIG_IRQBALANCE Description Rusty Russell @ 2004-03-23 17:16 ` John Stoffel 2004-03-23 21:30 ` Miquel van Smoorenburg 0 siblings, 1 reply; 6+ messages in thread From: John Stoffel @ 2004-03-23 17:16 UTC (permalink / raw) To: Rusty Russell; +Cc: lkml - Kernel Mailing List, Ingo Molnar And hey, under 2.6.5-rc2-mm1, it doens't seem to do anything: > cat /proc/version Linux version 2.6.5-rc2-mm1 (john@jfsnew) (gcc version 3.3.3 (Debian 20040314)) #3 SMP Sun Mar 21 15:26:27 EST 2004 > zcat /proc/config.gz | grep IRQ CONFIG_IRQBALANCE=y CONFIG_IDEPCI_SHARE_IRQ=y > cat /proc/interrupts CPU0 CPU1 0: 46272316 487 IO-APIC-edge timer 1: 376 0 IO-APIC-edge i8042 2: 0 0 XT-PIC cascade 4: 4147 1 IO-APIC-edge serial 7: 2 0 IO-APIC-edge parport0 8: 4 0 IO-APIC-edge rtc 11: 95936 1 IO-APIC-edge Cyclom-Y 12: 677 0 IO-APIC-edge i8042 14: 87 0 IO-APIC-edge ide0 16: 46770 3 IO-APIC-level ide2, ide3, ehci_hcd 17: 307832 1 IO-APIC-level eth0 18: 118258 1 IO-APIC-level aic7xxx, aic7xxx, ohci_hcd 19: 0 0 IO-APIC-level ohci_hcd NMI: 0 0 LOC: 46279245 46279281 ERR: 0 MIS: 417 John ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: arch/i386/Kconfig: CONFIG_IRQBALANCE Description 2004-03-23 17:16 ` John Stoffel @ 2004-03-23 21:30 ` Miquel van Smoorenburg 2004-03-24 14:04 ` John Stoffel 2004-03-25 19:07 ` Jamie Lokier 0 siblings, 2 replies; 6+ messages in thread From: Miquel van Smoorenburg @ 2004-03-23 21:30 UTC (permalink / raw) To: linux-kernel In article <16480.28882.388997.71072@gargle.gargle.HOWL>, John Stoffel <stoffel@lucent.com> wrote: > >And hey, under 2.6.5-rc2-mm1, it doens't seem to do anything: > > > zcat /proc/config.gz | grep IRQ > CONFIG_IRQBALANCE=y > CONFIG_IDEPCI_SHARE_IRQ=y > > > cat /proc/interrupts > CPU0 CPU1 > 0: 46272316 487 IO-APIC-edge timer > 1: 376 0 IO-APIC-edge i8042 > 16: 46770 3 IO-APIC-level ide2, ide3, ehci_hcd > 17: 307832 1 IO-APIC-level eth0 > 18: 118258 1 IO-APIC-level aic7xxx, aic7xxx, ohci_hcd > LOC: 46279245 46279281 Is that real SMP, or hyperthreading? If it's hyperthreading, then it makes sense that the IRQs are not balanced. In fact I have a server on which the IRQ balancing code does balance over the 2 virtual CPUs by accident (still have to debug what goes wrong and file a proper bug report) and as a result performance sucked until I turned it off. Mike. -- Netu, v qba'g yvxr gur cynvagrkg :) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: arch/i386/Kconfig: CONFIG_IRQBALANCE Description 2004-03-23 21:30 ` Miquel van Smoorenburg @ 2004-03-24 14:04 ` John Stoffel 2004-03-25 19:07 ` Jamie Lokier 1 sibling, 0 replies; 6+ messages in thread From: John Stoffel @ 2004-03-24 14:04 UTC (permalink / raw) To: Miquel van Smoorenburg; +Cc: linux-kernel >>>>> "Miquel" == Miquel van Smoorenburg <miquels@cistron.nl> writes: Miquel> In article <16480.28882.388997.71072@gargle.gargle.HOWL>, Miquel> John Stoffel <stoffel@lucent.com> wrote: >> >> And hey, under 2.6.5-rc2-mm1, it doens't seem to do anything: >> >> > zcat /proc/config.gz | grep IRQ >> CONFIG_IRQBALANCE=y >> CONFIG_IDEPCI_SHARE_IRQ=y >> >> > cat /proc/interrupts >> CPU0 CPU1 >> 0: 46272316 487 IO-APIC-edge timer >> 1: 376 0 IO-APIC-edge i8042 >> 16: 46770 3 IO-APIC-level ide2, ide3, ehci_hcd >> 17: 307832 1 IO-APIC-level eth0 >> 18: 118258 1 IO-APIC-level aic7xxx, aic7xxx, ohci_hcd >> LOC: 46279245 46279281 Miquel> Is that real SMP, or hyperthreading? If it's hyperthreading, Miquel> then it makes sense that the IRQs are not balanced. It's dual Xeon PIII 550mhz, in a Dell Precision 610MT workstation. Intel GX chipset. None of that fancy HT stuff here! > cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 7 model name : Pentium III (Katmai) stepping : 3 cpu MHz : 547.343 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1077.24 processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 7 model name : Pentium III (Katmai) stepping : 3 cpu MHz : 547.343 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1093.63 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: arch/i386/Kconfig: CONFIG_IRQBALANCE Description 2004-03-23 21:30 ` Miquel van Smoorenburg 2004-03-24 14:04 ` John Stoffel @ 2004-03-25 19:07 ` Jamie Lokier 2004-03-26 10:39 ` Miquel van Smoorenburg 1 sibling, 1 reply; 6+ messages in thread From: Jamie Lokier @ 2004-03-25 19:07 UTC (permalink / raw) To: Miquel van Smoorenburg; +Cc: linux-kernel Miquel van Smoorenburg wrote: > Is that real SMP, or hyperthreading? If it's hyperthreading, then > it makes sense that the IRQs are not balanced. That's unfair to the two tasks which might be running on each virtual CPU: one of the tasks is interrupted often. > In fact I have a server on which the IRQ balancing code does > balance over the 2 virtual CPUs by accident (still have to debug > what goes wrong and file a proper bug report) and as a result > performance sucked until I turned it off. What caused the suckage? Obviously there's a small time spend doing the work of rebalancing, but there is no cache hit from moving an interrupt between virtual CPUs, unlike with SMP, so why did that make performance suck? -- Jamie ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: arch/i386/Kconfig: CONFIG_IRQBALANCE Description 2004-03-25 19:07 ` Jamie Lokier @ 2004-03-26 10:39 ` Miquel van Smoorenburg 0 siblings, 0 replies; 6+ messages in thread From: Miquel van Smoorenburg @ 2004-03-26 10:39 UTC (permalink / raw) To: Jamie Lokier; +Cc: Miquel van Smoorenburg, linux-kernel On Thu, 25 Mar 2004 20:07:18, Jamie Lokier wrote: > Miquel van Smoorenburg wrote: > > Is that real SMP, or hyperthreading? If it's hyperthreading, then > > it makes sense that the IRQs are not balanced. > > That's unfair to the two tasks which might be running on each virtual > CPU: one of the tasks is interrupted often. > > > In fact I have a server on which the IRQ balancing code does > > balance over the 2 virtual CPUs by accident (still have to debug > > what goes wrong and file a proper bug report) and as a result > > performance sucked until I turned it off. > > What caused the suckage? Obviously there's a small time spend doing > the work of rebalancing, but there is no cache hit from moving an > interrupt between virtual CPUs, unlike with SMP, so why did that make > performance suck? I have no idea. I have a transit NNTP server, newsgate.cistron.nl, that has a acenic GigE card, 1 Maxtor ATA 80 GB for the O/S, and 4 Maxtor SATA 80 GB for database and storage. Sustained input is 100 mbit/sec, sustained output is 250-300 mbit/sec. It rans fine with 2.6.0-test11, but not any later kernels. I tried 2.6.2, 2.6.3 etc but somehow the output wouldn't get above 100 mbit/sec. Then I noticed that with the 2.6.0-test11 kernel IRQs weren't balanced over the 2 SMT cores while with later kernels they were. So I installed a 2.6.4-rc2 kernel. Bad performance. Did a "echo 1 > /proc/irq/XX/smp_affinity" for the NIC and IO interrupts, and performance went bang right back to what it was before. Yesterday I rebooted with 2.6.4-rc2, but with the "noirqbalance" option. That didn't really perform well. So I rebooted again with 2.6.5-rc2 without the "noirqbalance" option. It appeared to run better, but not quite up to par. Then I did the "echo 1 > /proc/irq/XX/smp_affinity" for the NIC and IO interrupts again. Output traffic peaked again. If you look at http://newsgate.cistron.nl/lkml/stats.cistron.nl/mrtg/html/quantum.html you can see the effect on the bandwidth stats: 25-03 before 14:00 2.6.4-rc2 with "echo 1 > /proc/irq/XX/smp_affinity 25-03 16:00 2.6.4-rc2 with noirqbalance 26-03 01:00 2.6.5-rc2 irq balancing enabled 26-03 10:30 2.6.5-rc2 with "echo 1 > /proc/irq/XX/smp_affinity In my case, it looks like the box runs best with only IRQ balancing for the timer interrupt over the 2 SMT cores, and no IRQ balancing for all the other hardware. I have no idea _why_ this affects throughput so much - the box itself doesn't "feel" any different wrt latency on the shell, or load average. It's just throughput, and I don't even know if this is disk controller or NIC related. Now since the box was down for 2 hours yesterday, it also has a large backlog to process. I really have to retry this once it has been running for a few days in a stable state, that's why I haven't really dug into it yet, circumstances have been changing too much. Mike. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2004-03-26 10:39 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2004-03-22 23:02 arch/i386/Kconfig: CONFIG_IRQBALANCE Description Rusty Russell 2004-03-23 17:16 ` John Stoffel 2004-03-23 21:30 ` Miquel van Smoorenburg 2004-03-24 14:04 ` John Stoffel 2004-03-25 19:07 ` Jamie Lokier 2004-03-26 10:39 ` Miquel van Smoorenburg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox