Re: Bad network performance over 2Gbps

Re: Bad network performance over 2Gbps

[H. Willstrand]

[Changed mail list]

On Tue, Apr 15, 2008 at 8:06 PM, Anton Titov <a.titov@host.bg> wrote:
> I use Linux for serving a huge amount of static web on few servers. When
>  network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but
>  every time just one) starts using exactly 100% CPU time and packet
>  packet loss starts preventing traffic from going up. When the network
>  traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
>
>  Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm
>  with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
>  switch. Machine is with two quad core Intel Xeons @2.33GHz.
>
>  Here goes a screen snapshot of "top" command. The described behavior
>  have nothing to do with 13% io-wait. It happens even if it is 0%
>  io-wait.
>  http://www.titov.net/misc/top-snap.png
>
>  kernel configuration:
>  http://www.titov.net/misc/config.gz
>
>  /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
>  uname -a:
>  http://www.titov.net/misc/misc.txt.gz
>
>  Is it a Linux bug or some hardware limitation?
>
>  Regards,
>  Anton Titov
>
>  --
>  To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>  the body of a message to majordomo@vger.kernel.org
>  More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Bad network performance over 2Gbps

[Kok, Auke]

H. Willstrand wrote:
> [Changed mail list]
> 
> On Tue, Apr 15, 2008 at 8:06 PM, Anton Titov <a.titov@host.bg> wrote:
>> I use Linux for serving a huge amount of static web on few servers. When
>>  network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but
>>  every time just one) starts using exactly 100% CPU time and packet
>>  packet loss starts preventing traffic from going up. When the network
>>  traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
>>
>>  Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm
>>  with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
>>  switch. Machine is with two quad core Intel Xeons @2.33GHz.
>>
>>  Here goes a screen snapshot of "top" command. The described behavior
>>  have nothing to do with 13% io-wait. It happens even if it is 0%
>>  io-wait.
>>  http://www.titov.net/misc/top-snap.png
>>
>>  kernel configuration:
>>  http://www.titov.net/misc/config.gz
>>
>>  /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
>>  uname -a:
>>  http://www.titov.net/misc/misc.txt.gz
>>
>>  Is it a Linux bug or some hardware limitation?

I'm wondering if this is not a classical demonstration of the NAPI-irq trap where
after migration all the interrupts from the various cards are migrated to a single
CPU, and because of NAPI once they're busy polling won't ever migrate away from
that CPU again.

Have you looked at `cat /proc/interrupts` before and after this happens?

My guess is that your specific situation can benefit from setting smp_affinity and
forcing the NIC irq's so that you're at least occupying the load over multiple
CPU's (but preferably ones that use the same cache!) will help relieve the situation.

alternatively you might even see an improvement by disabling NAPI. depending on
the driver that you're using this might be possible.

I actually don't know much about bonding and how this affects everything, but my
guess is that that's a less important factor in this issue.

Cheers,

Auke

Re: Bad network performance over 2Gbps

[Chris Snook]

Kok, Auke wrote:
> H. Willstrand wrote:
>> [Changed mail list]
>>
>> On Tue, Apr 15, 2008 at 8:06 PM, Anton Titov <a.titov@host.bg> wrote:
>>> I use Linux for serving a huge amount of static web on few servers. When
>>>  network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but
>>>  every time just one) starts using exactly 100% CPU time and packet
>>>  packet loss starts preventing traffic from going up. When the network
>>>  traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
>>>
>>>  Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm
>>>  with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
>>>  switch. Machine is with two quad core Intel Xeons @2.33GHz.
>>>
>>>  Here goes a screen snapshot of "top" command. The described behavior
>>>  have nothing to do with 13% io-wait. It happens even if it is 0%
>>>  io-wait.
>>>  http://www.titov.net/misc/top-snap.png
>>>
>>>  kernel configuration:
>>>  http://www.titov.net/misc/config.gz
>>>
>>>  /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
>>>  uname -a:
>>>  http://www.titov.net/misc/misc.txt.gz
>>>
>>>  Is it a Linux bug or some hardware limitation?
> 
> I'm wondering if this is not a classical demonstration of the NAPI-irq trap where
> after migration all the interrupts from the various cards are migrated to a single
> CPU, and because of NAPI once they're busy polling won't ever migrate away from
> that CPU again.
> 
> Have you looked at `cat /proc/interrupts` before and after this happens?
> 
> My guess is that your specific situation can benefit from setting smp_affinity and
> forcing the NIC irq's so that you're at least occupying the load over multiple
> CPU's (but preferably ones that use the same cache!) will help relieve the situation.
> 
> alternatively you might even see an improvement by disabling NAPI. depending on
> the driver that you're using this might be possible.
> 
> I actually don't know much about bonding and how this affects everything, but my
> guess is that that's a less important factor in this issue.
> 
> Cheers,
> 
> Auke

I'm not sure that spreading IRQs out completely is necessarily a good 
idea, due to cache line ping-pong.  I suspect you'll get optimal 
performance by assigning the six IRQs to two cores that share an L2 cache.

Still, I think you're on to something here.  Disabling NAPI and instead 
tuning the cards' interrupt coalescing settings might allow irqbalance 
to do a better job than it is currently.

-- Chris

Re: Bad network performance over 2Gbps

[Kok, Auke]

Chris Snook wrote:
> Kok, Auke wrote:
>> H. Willstrand wrote:
>>> [Changed mail list]
>>>
>>> On Tue, Apr 15, 2008 at 8:06 PM, Anton Titov <a.titov@host.bg> wrote:
>>>> I use Linux for serving a huge amount of static web on few servers.
>>>> When
>>>>  network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5,
>>>> but
>>>>  every time just one) starts using exactly 100% CPU time and packet
>>>>  packet loss starts preventing traffic from going up. When the network
>>>>  traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
>>>>
>>>>  Uplink is 6 gigabit Intel cards bonded together using 802.3ad
>>>> algorithm
>>>>  with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
>>>>  switch. Machine is with two quad core Intel Xeons @2.33GHz.
>>>>
>>>>  Here goes a screen snapshot of "top" command. The described behavior
>>>>  have nothing to do with 13% io-wait. It happens even if it is 0%
>>>>  io-wait.
>>>>  http://www.titov.net/misc/top-snap.png
>>>>
>>>>  kernel configuration:
>>>>  http://www.titov.net/misc/config.gz
>>>>
>>>>  /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
>>>>  uname -a:
>>>>  http://www.titov.net/misc/misc.txt.gz
>>>>
>>>>  Is it a Linux bug or some hardware limitation?
>>
>> I'm wondering if this is not a classical demonstration of the NAPI-irq
>> trap where
>> after migration all the interrupts from the various cards are migrated
>> to a single
>> CPU, and because of NAPI once they're busy polling won't ever migrate
>> away from
>> that CPU again.
>>
>> Have you looked at `cat /proc/interrupts` before and after this happens?
>>
>> My guess is that your specific situation can benefit from setting
>> smp_affinity and
>> forcing the NIC irq's so that you're at least occupying the load over
>> multiple
>> CPU's (but preferably ones that use the same cache!) will help relieve
>> the situation.
>>
>> alternatively you might even see an improvement by disabling NAPI.
>> depending on
>> the driver that you're using this might be possible.
>>
>> I actually don't know much about bonding and how this affects
>> everything, but my
>> guess is that that's a less important factor in this issue.
>>
>> Cheers,
>>
>> Auke
> 
> I'm not sure that spreading IRQs out completely is necessarily a good
> idea, due to cache line ping-pong.  I suspect you'll get optimal
> performance by assigning the six IRQs to two cores that share an L2 cache.
> 
> Still, I think you're on to something here.  Disabling NAPI and instead
> tuning the cards' interrupt coalescing settings might allow irqbalance
> to do a better job than it is currently.

well I posted another reply to him after I looked at the debug output he posted
and it appeats that the in-kernel irqbalance is the culprit and the 100% softirqd
is because his interrupts are being balanced across all cores, therefore pretty
much guaranteeing him full cache misses on every single receive, not to mention
unneeded migration of tasks.

I definately think we should disable the in-kernel irqbalance option by default :)

as to what the best solution is with 6 adapters is not clear as they're all on the
same bridge, but hard setting the affinity to two cores that share an L2 seems to
give the best results quickly.

Cheers,

Auke

Re: Bad network performance over 2Gbps

[Anton Titov]

On Tue, 2008-04-15 at 16:59 -0400, Chris Snook wrote:
> Still, I think you're on to something here.  Disabling NAPI and instead 
> tuning the cards' interrupt coalescing settings might allow irqbalance 
> to do a better job than it is currently.

Disabling NAPI allowed me to push as much as 3.5Gbit out of the same
server with ~ 20% of time CPUs doing software interrupts.

Regards,
Anton Titov

[PATCH] Re: Bad network performance over 2Gbps

[Kok, Auke]

Anton Titov wrote:
> On Tue, 2008-04-15 at 16:59 -0400, Chris Snook wrote:
>> Still, I think you're on to something here.  Disabling NAPI and instead 
>> tuning the cards' interrupt coalescing settings might allow irqbalance 
>> to do a better job than it is currently.
> 
> Disabling NAPI allowed me to push as much as 3.5Gbit out of the same
> server with ~ 20% of time CPUs doing software interrupts.

yes, I really don't see this is such an amazing discovery - the in-kernel
irqbalance code is totally wrong for network interrupts (and probably for most
interrupts).

on your system with 6 network interrupts it blows chunks and it's not NAPI that is
the issue - NAPI will work just fine on it's own. By disabling NAPI and reverting
to the in-driver irq moderation code you've effectively put the in-kernel
irqbalance code to the sideline and this is what makes it work again.

It's not the right solution.

We keep seing this exact issue pop up everywhere - especially with e1000(e)
datacenter users - this code _has_ to go or be fixed. Since there is a perfectly
viable solution, I strongly suggest disabling it.

This is not the first time I've sent this patch out in some form...

Auke


---
[X86] IRQBALANCE: Mark as BROKEN and disable by default

The IRQBALANCE option causes interrupts to bounce all around on SMP systems
quickly burying the CPU in migration cost and cache misses. Mainly affected are
network interrupts and this results in one CPU pegged in softirqd completely.

Disable this option and provide documentation to a better solution (userspace
irqbalance daemon does overall the best job to begin with and only manual setting
of smp_affinity will beat it).

Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>

---

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 6c70fed..956aa22 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1026,13 +1026,17 @@ config EFI
   	platforms.

 config IRQBALANCE
-	def_bool y
+	def_bool n
 	prompt "Enable kernel irq balancing"
-	depends on X86_32 && SMP && X86_IO_APIC
+	depends on X86_32 && SMP && X86_IO_APIC && BROKEN
 	help
 	  The default yes will allow the kernel to do irq load balancing.
 	  Saying no will keep the kernel from doing irq load balancing.

+	  This option is known to cause performance issues on SMP
+	  systems. The preferred method is to use the userspace
+	  'irqbalance' daemon instead. See http://irqbalance.org/.
+
 config SECCOMP
 	def_bool y
 	prompt "Enable seccomp to safely compute untrusted bytecode"

Re: [PATCH] Re: Bad network performance over 2Gbps

[Denys Fedoryshchenko]

By default also without IRQBALANCE enabled in kernel, APIC or someone else distributing interrupts over processors too.
There is no irqbalance daemon or whatever.

For example:
Router-KARAM ~ # cat /proc/interrupts
           CPU0       CPU1
  0:   87956938 1403052485   IO-APIC-edge      timer
  1:          0          2   IO-APIC-edge      i8042
  9:          0          0   IO-APIC-fasteoi   acpi
 19:        140       5714   IO-APIC-fasteoi   ohci_hcd:usb1, ohci_hcd:usb2
 24:  675673280 1186506694   IO-APIC-fasteoi   eth2
 26:  717865662 2201633562   IO-APIC-fasteoi   eth0
 27:    1869190   23075556   IO-APIC-fasteoi   eth1
NMI:          0          0   Non-maskable interrupts
LOC: 1403052485   87956683   Local timer interrupts
RES:      75059      25408   Rescheduling interrupts
CAL:      99542         83   function call interrupts
TLB:        616        200   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0

sunfire-1 ~ # cat config|grep -i irq
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
# CONFIG_IRQBALANCE is not set
CONFIG_HT_IRQ=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_DEBUG_SHIRQ is not set

Is it harmful too?

On Thursday 17 April 2008 20:37, Kok, Auke wrote:
> Anton Titov wrote:
> > On Tue, 2008-04-15 at 16:59 -0400, Chris Snook wrote:
> >> Still, I think you're on to something here.  Disabling NAPI and instead 
> >> tuning the cards' interrupt coalescing settings might allow irqbalance 
> >> to do a better job than it is currently.
> > 
> > Disabling NAPI allowed me to push as much as 3.5Gbit out of the same
> > server with ~ 20% of time CPUs doing software interrupts.
> 
> yes, I really don't see this is such an amazing discovery - the in-kernel
> irqbalance code is totally wrong for network interrupts (and probably for most
> interrupts).
> 
> on your system with 6 network interrupts it blows chunks and it's not NAPI that is
> the issue - NAPI will work just fine on it's own. By disabling NAPI and reverting
> to the in-driver irq moderation code you've effectively put the in-kernel
> irqbalance code to the sideline and this is what makes it work again.
> 
> It's not the right solution.
> 
> We keep seing this exact issue pop up everywhere - especially with e1000(e)
> datacenter users - this code _has_ to go or be fixed. Since there is a perfectly
> viable solution, I strongly suggest disabling it.
> 
> This is not the first time I've sent this patch out in some form...
> 
> Auke
> 
> 
> ---
> [X86] IRQBALANCE: Mark as BROKEN and disable by default
> 
> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
> quickly burying the CPU in migration cost and cache misses. Mainly affected are
> network interrupts and this results in one CPU pegged in softirqd completely.
> 
> Disable this option and provide documentation to a better solution (userspace
> irqbalance daemon does overall the best job to begin with and only manual setting
> of smp_affinity will beat it).
> 
> Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>
> 
> ---
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 6c70fed..956aa22 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1026,13 +1026,17 @@ config EFI
>    	platforms.
> 
>  config IRQBALANCE
> -	def_bool y
> +	def_bool n
>  	prompt "Enable kernel irq balancing"
> -	depends on X86_32 && SMP && X86_IO_APIC
> +	depends on X86_32 && SMP && X86_IO_APIC && BROKEN
>  	help
>  	  The default yes will allow the kernel to do irq load balancing.
>  	  Saying no will keep the kernel from doing irq load balancing.
> 
> +	  This option is known to cause performance issues on SMP
> +	  systems. The preferred method is to use the userspace
> +	  'irqbalance' daemon instead. See http://irqbalance.org/.
> +
>  config SECCOMP
>  	def_bool y
>  	prompt "Enable seccomp to safely compute untrusted bytecode"
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
------
Technical Manager
Virtual ISP S.A.L.
Lebanon

Re: [PATCH] Re: Bad network performance over 2Gbps

[Pavel Machek]

Hi!

> [X86] IRQBALANCE: Mark as BROKEN and disable by default
> 
> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
> quickly burying the CPU in migration cost and cache misses. Mainly affected are
> network interrupts and this results in one CPU pegged in softirqd completely.
> 
> Disable this option and provide documentation to a better solution (userspace
> irqbalance daemon does overall the best job to begin with and only manual setting
> of smp_affinity will beat it).
> 
> Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>
> 
> ---
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 6c70fed..956aa22 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1026,13 +1026,17 @@ config EFI
>    	platforms.
> 
>  config IRQBALANCE
> -	def_bool y
> +	def_bool n

ACK.
>  	prompt "Enable kernel irq balancing"
> -	depends on X86_32 && SMP && X86_IO_APIC
> +	depends on X86_32 && SMP && X86_IO_APIC && BROKEN

This is wrong. irqbalance works, there's nothing wrong with it; but it
has nasty sideffects.

>  	help
>  	  The default yes will allow the kernel to do irq load balancing.
>  	  Saying no will keep the kernel from doing irq load balancing.
> 
> +	  This option is known to cause performance issues on SMP
> +	  systems. The preferred method is to use the userspace
> +	  'irqbalance' daemon instead. See http://irqbalance.org/.
> +

ACK.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH] Re: Bad network performance over 2Gbps

[Kok, Auke]

Pavel Machek wrote:
> Hi!
> 
>> [X86] IRQBALANCE: Mark as BROKEN and disable by default
>>
>> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
>> quickly burying the CPU in migration cost and cache misses. Mainly affected are
>> network interrupts and this results in one CPU pegged in softirqd completely.
>>
>> Disable this option and provide documentation to a better solution (userspace
>> irqbalance daemon does overall the best job to begin with and only manual setting
>> of smp_affinity will beat it).
>>
>> Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>
>>
>> ---
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 6c70fed..956aa22 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -1026,13 +1026,17 @@ config EFI
>>    	platforms.
>>
>>  config IRQBALANCE
>> -	def_bool y
>> +	def_bool n
> 
> ACK.
>>  	prompt "Enable kernel irq balancing"
>> -	depends on X86_32 && SMP && X86_IO_APIC
>> +	depends on X86_32 && SMP && X86_IO_APIC && BROKEN
> 
> This is wrong. irqbalance works, there's nothing wrong with it; but it
> has nasty sideffects.

ok, I'm fine with taking that part out of the patch.

Ingo, want me to send an updated patch?


> 
>>  	help
>>  	  The default yes will allow the kernel to do irq load balancing.
>>  	  Saying no will keep the kernel from doing irq load balancing.
>>
>> +	  This option is known to cause performance issues on SMP
>> +	  systems. The preferred method is to use the userspace
>> +	  'irqbalance' daemon instead. See http://irqbalance.org/.
>> +
> 
> ACK.
> 

Re: [PATCH] Re: Bad network performance over 2Gbps

[Ingo Molnar]

* Kok, Auke <auke-jan.h.kok@intel.com> wrote:

> We keep seing this exact issue pop up everywhere - especially with 
> e1000(e) datacenter users - this code _has_ to go or be fixed. Since 
> there is a perfectly viable solution, I strongly suggest disabling it.

strongly agreed. Thanks Auke, applied.

	Ingo

Re: [PATCH] Re: Bad network performance over 2Gbps

[Kok, Auke]

Ingo Molnar wrote:
> * Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> 
>> We keep seing this exact issue pop up everywhere - especially with 
>> e1000(e) datacenter users - this code _has_ to go or be fixed. Since 
>> there is a perfectly viable solution, I strongly suggest disabling it.
> 
> strongly agreed. Thanks Auke, applied.
> 
> 	Ingo


excellent, ignore my other reply to Pavel - I didn't see this reply yet :)

Thanks Ingo


Auke

Re: [PATCH] Re: Bad network performance over 2Gbps

[Andi Kleen]

"Kok, Auke" <auke-jan.h.kok@intel.com> writes:

> Ingo Molnar wrote:
>> * Kok, Auke <auke-jan.h.kok@intel.com> wrote:
>> 
>>> We keep seing this exact issue pop up everywhere - especially with 
>>> e1000(e) datacenter users - this code _has_ to go or be fixed. Since 
>>> there is a perfectly viable solution, I strongly suggest disabling it.
>> 
>> strongly agreed. Thanks Auke, applied.
>> 
>> 	Ingo
>
>
> excellent, ignore my other reply to Pavel - I didn't see this reply yet :)

Shouldn't you just add it to the FeatureRemoval list too and remove it 
then quickly? No need to keep disabled and known to be wrong code around.

-Andi

Re: [PATCH] Re: Bad network performance over 2Gbps

[Bill Fink]

On Thu, 17 Apr 2008, Kok, Auke wrote:

> [X86] IRQBALANCE: Mark as BROKEN and disable by default
> 
> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
> quickly burying the CPU in migration cost and cache misses. Mainly affected are
> network interrupts and this results in one CPU pegged in softirqd completely.
> 
> Disable this option and provide documentation to a better solution (userspace
> irqbalance daemon does overall the best job to begin with and only manual setting
> of smp_affinity will beat it).
> 
> Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>
> 
> ---
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 6c70fed..956aa22 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1026,13 +1026,17 @@ config EFI
>    	platforms.
> 
>  config IRQBALANCE
> -	def_bool y
> +	def_bool n
>  	prompt "Enable kernel irq balancing"
> -	depends on X86_32 && SMP && X86_IO_APIC
> +	depends on X86_32 && SMP && X86_IO_APIC && BROKEN
>  	help
>  	  The default yes will allow the kernel to do irq load balancing.
>  	  Saying no will keep the kernel from doing irq load balancing.

Since you're changing the default setting, shouldn't the above be
changed to:

 	  Saying yes will allow the kernel to do irq load balancing.
 	  The default no will keep the kernel from doing irq load balancing.

> +	  This option is known to cause performance issues on SMP
> +	  systems. The preferred method is to use the userspace
> +	  'irqbalance' daemon instead. See http://irqbalance.org/.
> +
>  config SECCOMP
>  	def_bool y
>  	prompt "Enable seccomp to safely compute untrusted bytecode"

						-Bill

Re: [PATCH] Re: Bad network performance over 2Gbps

[Bodo Eggert]

Kok, Auke <auke-jan.h.kok@intel.com> wrote:

> [X86] IRQBALANCE: Mark as BROKEN and disable by default
> 
> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
> quickly burying the CPU in migration cost and cache misses. Mainly affected
> are network interrupts and this results in one CPU pegged in softirqd
> completely.

If this is the problem, maybe it would help to only balance the IRQs each
e.g. ten seconds? Unfortunately I have no SMP system to try it out.

Re: [PATCH] Re: Bad network performance over 2Gbps

[Stephen Hemminger]

Bodo Eggert wrote:
> Kok, Auke <auke-jan.h.kok@intel.com> wrote:
>
>   
>> [X86] IRQBALANCE: Mark as BROKEN and disable by default
>>
>> The IRQBALANCE option causes interrupts to bounce all around on SMP systems
>> quickly burying the CPU in migration cost and cache misses. Mainly affected
>> are network interrupts and this results in one CPU pegged in softirqd
>> completely.
>>     
>
> If this is the problem, maybe it would help to only balance the IRQs each
> e.g. ten seconds? Unfortunately I have no SMP system to try it out.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   
The kernel level IRQBALANCE is useless. The userlevel irqbalance does 
the right thing,
it handles multi-core, and network devices, and all the other special cases.
*Don't use kernel level irqbalance*

Re: [PATCH] Re: Bad network performance over 2Gbps

[Rick Jones]

Bodo Eggert wrote:
> Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> 
> 
>>[X86] IRQBALANCE: Mark as BROKEN and disable by default
>>
>>The IRQBALANCE option causes interrupts to bounce all around on SMP systems
>>quickly burying the CPU in migration cost and cache misses. Mainly affected
>>are network interrupts and this results in one CPU pegged in softirqd
>>completely.
> 
> 
> If this is the problem, maybe it would help to only balance the IRQs each
> e.g. ten seconds? Unfortunately I have no SMP system to try it out.

Be it kernel or user space, for consistent benchmark results it needs to 
be able to be turned-off without turning the code.  That leaves me in 
agreement with Stephen that if it must exist, the user space one would 
be preferable.  It can be easily terminated with extreme prejudice.

rick jones

Re: [PATCH] Re: Bad network performance over 2Gbps

[Bodo Eggert]

On Mon, 21 Apr 2008, Rick Jones wrote:
> Bodo Eggert wrote:
> > Kok, Auke <auke-jan.h.kok@intel.com> wrote:

> > > [X86] IRQBALANCE: Mark as BROKEN and disable by default
> > > 
> > > The IRQBALANCE option causes interrupts to bounce all around on SMP
> > > systems
> > > quickly burying the CPU in migration cost and cache misses. Mainly
> > > affected
> > > are network interrupts and this results in one CPU pegged in softirqd
> > > completely.
> > 
> > 
> > If this is the problem, maybe it would help to only balance the IRQs each
> > e.g. ten seconds? Unfortunately I have no SMP system to try it out.
> 
> Be it kernel or user space, for consistent benchmark results it needs to be
> able to be turned-off without turning the code.  That leaves me in agreement
> with Stephen that if it must exist, the user space one would be preferable.
> It can be easily terminated with extreme prejudice.

I agree that having a full-featured userspace balancer daemon with lots of 
intelligence will be theoretically better, but if you can have a simple
daemon doing OK on many machines for less than the userspace daemon's
kernel stack, why not?
-- 
Funny quotes:
31. Why do "overlook" and "oversee" mean opposite things?

Re: [PATCH] Re: Bad network performance over 2Gbps

[Rick Jones]

Bodo Eggert wrote:
> On Mon, 21 Apr 2008, Rick Jones wrote:
>>Be it kernel or user space, for consistent benchmark results it needs to be
>>able to be turned-off without turning the code.  That leaves me in agreement
>>with Stephen that if it must exist, the user space one would be preferable.
>>It can be easily terminated with extreme prejudice.
> 
> 
> I agree that having a full-featured userspace balancer daemon with lots of 
> intelligence will be theoretically better, but if you can have a simple
> daemon doing OK on many machines for less than the userspace daemon's
> kernel stack, why not?

Perhaps my judgement is too colored by benchmark(et)ing, and desires to 
have repeatable results on things like neperf, but I very much like to 
know where my interrupts are going and don't like them moving around. 
That is why I am not particularly fond of either flavor of irq balancing.

That being the case, whatever is out there aught to be able to be 
disabled on a running system without having to roll bits or reboot.

rick jones

Re: [PATCH] Re: Bad network performance over 2Gbps

[Bodo Eggert]

On Mon, 21 Apr 2008, Rick Jones wrote:
> Bodo Eggert wrote:
> > On Mon, 21 Apr 2008, Rick Jones wrote:

> > > Be it kernel or user space, for consistent benchmark results it needs to
> > > be
> > > able to be turned-off without turning the code.  That leaves me in
> > > agreement
> > > with Stephen that if it must exist, the user space one would be
> > > preferable.
> > > It can be easily terminated with extreme prejudice.
> > 
> > 
> > I agree that having a full-featured userspace balancer daemon with lots of
> > intelligence will be theoretically better, but if you can have a simple
> > daemon doing OK on many machines for less than the userspace daemon's
> > kernel stack, why not?
> 
> Perhaps my judgement is too colored by benchmark(et)ing, and desires to have
> repeatable results on things like neperf, but I very much like to know where
> my interrupts are going and don't like them moving around. That is why I am
> not particularly fond of either flavor of irq balancing.
> 
> That being the case, whatever is out there aught to be able to be disabled on
> a running system without having to roll bits or reboot.

Adding a "module" parameter to disable it should be cheap, isn't it?
-- 
Top 100 things you don't want the sysadmin to say:
34. The network's down, but we're working on it. Come back after diner.
    (Usually said at 2200 the night before thesis deadline... )

Re: [PATCH] Re: Bad network performance over 2Gbps

[Chris Snook]

Bodo Eggert wrote:
> On Mon, 21 Apr 2008, Rick Jones wrote:
>> Bodo Eggert wrote:
>>> On Mon, 21 Apr 2008, Rick Jones wrote:
> 
>>>> Be it kernel or user space, for consistent benchmark results it needs to
>>>> be
>>>> able to be turned-off without turning the code.  That leaves me in
>>>> agreement
>>>> with Stephen that if it must exist, the user space one would be
>>>> preferable.
>>>> It can be easily terminated with extreme prejudice.
>>>
>>> I agree that having a full-featured userspace balancer daemon with lots of
>>> intelligence will be theoretically better, but if you can have a simple
>>> daemon doing OK on many machines for less than the userspace daemon's
>>> kernel stack, why not?
>> Perhaps my judgement is too colored by benchmark(et)ing, and desires to have
>> repeatable results on things like neperf, but I very much like to know where
>> my interrupts are going and don't like them moving around. That is why I am
>> not particularly fond of either flavor of irq balancing.
>>
>> That being the case, whatever is out there aught to be able to be disabled on
>> a running system without having to roll bits or reboot.
> 
> Adding a "module" parameter to disable it should be cheap, isn't it?

Except the irq balancing is system-wide.  Adding per-device exemptions to an 
obsolete feature seems like the wrong way to go.

-- Chris

Re: [PATCH] Re: Bad network performance over 2Gbps

[Bodo Eggert]

On Mon, 21 Apr 2008, Chris Snook wrote:
> Bodo Eggert wrote:
> > On Mon, 21 Apr 2008, Rick Jones wrote:
> >> Bodo Eggert wrote:
> >>> On Mon, 21 Apr 2008, Rick Jones wrote:

> >>>> Be it kernel or user space, for consistent benchmark results it needs to
> >>>> be
> >>>> able to be turned-off without turning the code.  That leaves me in
> >>>> agreement
> >>>> with Stephen that if it must exist, the user space one would be
> >>>> preferable.
> >>>> It can be easily terminated with extreme prejudice.
> >>>
> >>> I agree that having a full-featured userspace balancer daemon with lots of
> >>> intelligence will be theoretically better, but if you can have a simple
> >>> daemon doing OK on many machines for less than the userspace daemon's
> >>> kernel stack, why not?
> >> Perhaps my judgement is too colored by benchmark(et)ing, and desires to have
> >> repeatable results on things like neperf, but I very much like to know where
> >> my interrupts are going and don't like them moving around. That is why I am
> >> not particularly fond of either flavor of irq balancing.
> >>
> >> That being the case, whatever is out there aught to be able to be disabled on
> >> a running system without having to roll bits or reboot.
> > 
> > Adding a "module" parameter to disable it should be cheap, isn't it?
> 
> Except the irq balancing is system-wide.  Adding per-device exemptions to an 
> obsolete feature seems like the wrong way to go.

No, not a per-device-exemption. My reasoning was: If the IRQ balancer 
bounces the IRQ too often, doing it less often seems to be the correct 
solution. One cache miss each ten seconds sounds like it should be OK.
As said before, I can't verify this theory.

Re: [PATCH] Re: Bad network performance over 2Gbps

[Kok, Auke]

Bodo Eggert wrote:
> On Mon, 21 Apr 2008, Chris Snook wrote:
>> Bodo Eggert wrote:
>>> On Mon, 21 Apr 2008, Rick Jones wrote:
>>>> Bodo Eggert wrote:
>>>>> On Mon, 21 Apr 2008, Rick Jones wrote:
> 
>>>>>> Be it kernel or user space, for consistent benchmark results it needs to
>>>>>> be
>>>>>> able to be turned-off without turning the code.  That leaves me in
>>>>>> agreement
>>>>>> with Stephen that if it must exist, the user space one would be
>>>>>> preferable.
>>>>>> It can be easily terminated with extreme prejudice.
>>>>> I agree that having a full-featured userspace balancer daemon with lots of
>>>>> intelligence will be theoretically better, but if you can have a simple
>>>>> daemon doing OK on many machines for less than the userspace daemon's
>>>>> kernel stack, why not?
>>>> Perhaps my judgement is too colored by benchmark(et)ing, and desires to have
>>>> repeatable results on things like neperf, but I very much like to know where
>>>> my interrupts are going and don't like them moving around. That is why I am
>>>> not particularly fond of either flavor of irq balancing.
>>>>
>>>> That being the case, whatever is out there aught to be able to be disabled on
>>>> a running system without having to roll bits or reboot.
>>> Adding a "module" parameter to disable it should be cheap, isn't it?
>> Except the irq balancing is system-wide.  Adding per-device exemptions to an 
>> obsolete feature seems like the wrong way to go.
> 
> No, not a per-device-exemption. My reasoning was: If the IRQ balancer 
> bounces the IRQ too often, doing it less often seems to be the correct 
> solution. One cache miss each ten seconds sounds like it should be OK.
> As said before, I can't verify this theory.

this is exaclty what the userspace irqbalance does and it's even optimized to not
do those migrations once every 10 seconds if things look OK. from that
perspective, it's definately more mature and it's maintained as well.

Auke