Re: e1000 softirq load balancing

Re: e1000 softirq load balancing

[David Miller]

From: Don Porter <porterde@cs.utexas.edu>
Date: Tue, 14 Oct 2008 14:05:34 -0500

> It seems to me that with 4 independent NICs and plenty of CPUs to
> spare, I ought to be able to assign one softirq daemon to each NIC
> rather than funnelling all of the traffic through 1 or 2.

Traffic doesn't get distributed unless the NIC has support
for RX flow seperation and PCI MSI-X interrupts.  Your NICs
do not.

So no matter how hard you try, each NIC is going to have it's
packets processed essentially on one cpu.

Re: e1000 softirq load balancing

[Don Porter]

Thanks David.

Would you mind giving me a bit of intuition why I can't have a 1:1 
mapping of CPUs to NICs? 

I am a bit out of my depth here, but I'd like to learn.

Best,
Don

David Miller wrote:
> From: Don Porter <porterde@cs.utexas.edu>
> Date: Tue, 14 Oct 2008 14:05:34 -0500
>
>   
>> It seems to me that with 4 independent NICs and plenty of CPUs to
>> spare, I ought to be able to assign one softirq daemon to each NIC
>> rather than funnelling all of the traffic through 1 or 2.
>>     
>
> Traffic doesn't get distributed unless the NIC has support
> for RX flow seperation and PCI MSI-X interrupts.  Your NICs
> do not.
>
> So no matter how hard you try, each NIC is going to have it's
> packets processed essentially on one cpu.
>   

Re: e1000 softirq load balancing

[David Miller]

From: Don Porter <porterde@cs.utexas.edu>
Date: Tue, 14 Oct 2008 18:46:11 -0500

> Would you mind giving me a bit of intuition why I can't have a 1:1
> mapping of CPUs to NICs?

I didn't say that.

I said that without HW flow seperation support, you can only
expect N cpus to be busy where N is the number of NICs you
have.

Re: e1000 softirq load balancing

[Donald Porter]

Ok.  That seems very reasonable.

So the behavior I am seeing is that I have 4 NICs, but all of the  
traffic is being funneled to 1-2 softirq handlers, despite the fact  
that the hardware interrupts are being delivered to 4 different CPUs.

Any tips on how to debug this?  Or perhaps there is some configuration  
step I am missing?

Thanks,
Don

On Oct 14, 2008, at 6:51 PM, David Miller wrote:

> From: Don Porter <porterde@cs.utexas.edu>
> Date: Tue, 14 Oct 2008 18:46:11 -0500
>
>> Would you mind giving me a bit of intuition why I can't have a 1:1
>> mapping of CPUs to NICs?
>
> I didn't say that.
>
> I said that without HW flow seperation support, you can only
> expect N cpus to be busy where N is the number of NICs you
> have.