[Bridge] Lowering CPU utilisation

[Bridge] Lowering CPU utilisation

[Leigh Sharpe]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Hi All, 
 I have a box with 12 bridges on it, each of which has 2 ports.
When the traffic level on the box reaches around 15-20Mbps on each
bridge, my CPU utilisation gets to about 90-95%.
I would like to avoid working the CPU that hard. Is there anything which
can be done to get the CPU utlilisation down a bit? I have plans for an
even bigger bridge, but these results indicate that it may not scale
quite that well.
 
Regards,
             Leigh
 
 

[-- Attachment #2: Type: text/html, Size: 1267 bytes --]

Re: [Bridge] Lowering CPU utilisation

[Stephen Hemminger]

On Thu, 1 Nov 2007 09:00:07 +1100
"Leigh Sharpe" <lsharpe@pacificwireless.com.au> wrote:

> Hi All, 
>  I have a box with 12 bridges on it, each of which has 2 ports.
> When the traffic level on the box reaches around 15-20Mbps on each
> bridge, my CPU utilisation gets to about 90-95%.
> I would like to avoid working the CPU that hard. Is there anything which
> can be done to get the CPU utlilisation down a bit? I have plans for an
> even bigger bridge, but these results indicate that it may not scale
> quite that well.
>  
> Regards,
>              Leigh
>  
>  

Learn how to use oprofile and figure out where the time is going.
You might be getting hit by the choice of network interface.


-- 
Stephen Hemminger <shemminger@linux-foundation.org>

RE: [Bridge] Lowering CPU utilisation

[Leigh Sharpe]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

I'm using e1000 cards.
According to oprofile, around 50% of the time is spent in the e1000
driver. 
Given that my load average is still under 0.5, I suspect I've got plenty
of room for the driver to handle more traffic before I need to worry too
much.
 

________________________________

From: Vaidas M [mailto:admin@vdx.lt] 
Sent: Tuesday, 6 November 2007 9:32 PM
To: Leigh Sharpe
Subject: RE: [Bridge] Lowering CPU utilisation



What kind of NIC you are using?

I suggest you to use cards with NAPI featured in driver.

 

_      _     _    _   _  _ _ __ ___ ____ _____

Vaidas M. 

 

From: bridge-bounces@lists.linux-foundation.org
[mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Leigh
Sharpe
Sent: Thursday, November 01, 2007 12:00 AM
To: bridge@lists.linux-foundation.org
Subject: [Bridge] Lowering CPU utilisation

 

Hi All, 

 I have a box with 12 bridges on it, each of which has 2 ports.

When the traffic level on the box reaches around 15-20Mbps on each
bridge, my CPU utilisation gets to about 90-95%.

I would like to avoid working the CPU that hard. Is there anything which
can be done to get the CPU utlilisation down a bit? I have plans for an
even bigger bridge, but these results indicate that it may not scale
quite that well.

 

Regards,
             Leigh

 

 


[-- Attachment #2: Type: text/html, Size: 6050 bytes --]

RE: [Bridge] Lowering CPU utilisation

[Brad Dameron]

On Wed, 2007-11-07 at 08:51 +1100, Leigh Sharpe wrote:
> I'm using e1000 cards.
> According to oprofile, around 50% of the time is spent in the e1000
> driver. 
> Given that my load average is still under 0.5, I suspect I've got
> plenty of room for the driver to handle more traffic before I need to
> worry too much.
>  
> 
> 
> ______________________________________________________________________
> From: Vaidas M [mailto:admin@vdx.lt] 
> Sent: Tuesday, 6 November 2007 9:32 PM
> To: Leigh Sharpe
> Subject: RE: [Bridge] Lowering CPU utilisation
> 
> 
> 
> What kind of NIC you are using?
> 
> I suggest you to use cards with NAPI featured in driver.
> 
>  
> 
>                         _      _     _    _   _  _ _ __ ___ ____ _____
> 
>                                                             Vaidas M. 
> 
> 
>  
> 
> From: bridge-bounces@lists.linux-foundation.org
> [mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Leigh
> Sharpe
> Sent: Thursday, November 01, 2007 12:00 AM
> To: bridge@lists.linux-foundation.org
> Subject: [Bridge] Lowering CPU utilisation
> 
> 
>  
> 
> Hi All, 
> 
> 
>  I have a box with 12 bridges on it, each of which has 2 ports.
> 
> 
> When the traffic level on the box reaches around 15-20Mbps on each
> bridge, my CPU utilisation gets to about 90-95%.
> 
> 
> I would like to avoid working the CPU that hard. Is there anything
> which can be done to get the CPU utlilisation down a bit? I have plans
> for an even bigger bridge, but these results indicate that it may not
> scale quite that well.
> 
> 
>  
> 
> 
> Regards,
>              Leigh
> 


I assume you are using Linux. Try setting these in your /etc/sysctl.conf
file.

# increase TCP max buffer size
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.ipv4.tcp_rmem = 4096 1048576 8388608
net.ipv4.tcp_wmem = 4096 65535 8388608
net.core.optmem_max = 40960


Then load them with sysctl -p command.

Hope this helps. 


Brad Dameron
Senior Systems Engineer
Clearw're

Re: [Bridge] Lowering CPU utilisation

[Stephen Hemminger]

On Wed, 7 Nov 2007 08:51:41 +1100
"Leigh Sharpe" <lsharpe@pacificwireless.com.au> wrote:

> I'm using e1000 cards.
> According to oprofile, around 50% of the time is spent in the e1000
> driver. 
> Given that my load average is still under 0.5, I suspect I've got plenty
> of room for the driver to handle more traffic before I need to worry too
> much.
>  
>
Did you remember to configure NAPI for the e1000 driver (it is optional).

-- 
Stephen Hemminger <shemminger@linux-foundation.org>

RE: [Bridge] Lowering CPU utilisation

[Brad Dameron]

On Tue, 2007-11-06 at 14:13 -0800, Brad Dameron wrote:
> On Wed, 2007-11-07 at 08:51 +1100, Leigh Sharpe wrote:
> > I'm using e1000 cards.
> > According to oprofile, around 50% of the time is spent in the e1000
> > driver. 
> > Given that my load average is still under 0.5, I suspect I've got
> > plenty of room for the driver to handle more traffic before I need to
> > worry too much.
> >  
> > 
> > 
> > ______________________________________________________________________
> > From: Vaidas M [mailto:admin@vdx.lt] 
> > Sent: Tuesday, 6 November 2007 9:32 PM
> > To: Leigh Sharpe
> > Subject: RE: [Bridge] Lowering CPU utilisation
> > 
> > 
> > 
> > What kind of NIC you are using?
> > 
> > I suggest you to use cards with NAPI featured in driver.
> > 
> >  
> > 
> >                         _      _     _    _   _  _ _ __ ___ ____ _____
> > 
> >                                                             Vaidas M. 
> > 
> > 
> >  
> > 
> > From: bridge-bounces@lists.linux-foundation.org
> > [mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Leigh
> > Sharpe
> > Sent: Thursday, November 01, 2007 12:00 AM
> > To: bridge@lists.linux-foundation.org
> > Subject: [Bridge] Lowering CPU utilisation
> > 
> > 
> >  
> > 
> > Hi All, 
> > 
> > 
> >  I have a box with 12 bridges on it, each of which has 2 ports.
> > 
> > 
> > When the traffic level on the box reaches around 15-20Mbps on each
> > bridge, my CPU utilisation gets to about 90-95%.
> > 
> > 
> > I would like to avoid working the CPU that hard. Is there anything
> > which can be done to get the CPU utlilisation down a bit? I have plans
> > for an even bigger bridge, but these results indicate that it may not
> > scale quite that well.
> > 
> > 
> >  
> > 
> > 
> > Regards,
> >              Leigh
> > 
> 
> 
> I assume you are using Linux. Try setting these in your /etc/sysctl.conf
> file.
> 
> # increase TCP max buffer size
> net.core.rmem_max = 8388608
> net.core.wmem_max = 8388608
> # increase Linux autotuning TCP buffer limits
> # min, default, and max number of bytes to use
> net.ipv4.tcp_mem = 8388608 8388608 8388608
> net.ipv4.tcp_rmem = 4096 1048576 8388608
> net.ipv4.tcp_wmem = 4096 65535 8388608
> net.core.optmem_max = 40960
> 
> 
> Then load them with sysctl -p command.
> 
> Hope this helps. 

Oops. Too many lists. Of course your using Linux.


Brad Dameron
Senior Systems Engineer
Clearw're

RE: [Bridge] Lowering CPU utilisation

[Leigh Sharpe]

> Did you remember to configure NAPI for the e1000 driver (it is
optional).

As far as I can tell, it's on by default. A 'make menuconfig' shows "Use
Rx Polling (NAPI)" as being enabled.
 

  

-----Original Message-----
From: Stephen Hemminger [mailto:shemminger@linux-foundation.org] 
Sent: Wednesday, 7 November 2007 9:19 AM
To: Leigh Sharpe
Cc: Vaidas M; bridge@lists.linux-foundation.org
Subject: Re: [Bridge] Lowering CPU utilisation

On Wed, 7 Nov 2007 08:51:41 +1100
"Leigh Sharpe" <lsharpe@pacificwireless.com.au> wrote:

> I'm using e1000 cards.
> According to oprofile, around 50% of the time is spent in the e1000
> driver. 
> Given that my load average is still under 0.5, I suspect I've got
plenty
> of room for the driver to handle more traffic before I need to worry
too
> much.
>  
>
Did you remember to configure NAPI for the e1000 driver (it is
optional).

-- 
Stephen Hemminger <shemminger@linux-foundation.org>