gigabit ethernet power consumption

gigabit ethernet power consumption

[Pavel Machek]

Hi!

I've found that gbit vs. 100mbit power consumption difference is about
1W -- pretty significant. (Maybe powertop should include it in the
tips section? :).

Energy Star people insist that machines should switch down to 100mbit
when network is idle, and I guess that makes a lot of sense -- you
save 1W locally and 1W on the router.

Question is, how to implement it correctly? Daemon that would watch
data rates and switch speeds using mii-tool would be simple, but is
that enough?
									Pavel 
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: gigabit ethernet power consumption

[Kok, Auke]

Pavel Machek wrote:
> Hi!
> 
> I've found that gbit vs. 100mbit power consumption difference is about
> 1W -- pretty significant. (Maybe powertop should include it in the
> tips section? :).
> 
> Energy Star people insist that machines should switch down to 100mbit
> when network is idle, and I guess that makes a lot of sense -- you
> save 1W locally and 1W on the router.
> 
> Question is, how to implement it correctly? Daemon that would watch
> data rates and switch speeds using mii-tool would be simple, but is
> that enough?

you most certainly want to do this in userspace I think.

One of the biggest problems is that link negotiation can take a significant amount
of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
having your ethernet connection go offline for 3 seconds may not be the desired
effect for when you want to get more bandwidth in the first place.

However, when a laptop is in battery mode, switching down from gigabit to 100mbit
makes a lot more sense, so this is something I would recommend. This can be as
easy as changing the advertisement mask of the interface and renegotiating the
link. Userspace could handle that very easily.

Auke

Re: gigabit ethernet power consumption

[Willy Tarreau]

Hi Auke,

On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
> Pavel Machek wrote:
> > Hi!
> > 
> > I've found that gbit vs. 100mbit power consumption difference is about
> > 1W -- pretty significant. (Maybe powertop should include it in the
> > tips section? :).
> > 
> > Energy Star people insist that machines should switch down to 100mbit
> > when network is idle, and I guess that makes a lot of sense -- you
> > save 1W locally and 1W on the router.
> > 
> > Question is, how to implement it correctly? Daemon that would watch
> > data rates and switch speeds using mii-tool would be simple, but is
> > that enough?
> 
> you most certainly want to do this in userspace I think.
> 
> One of the biggest problems is that link negotiation can take a significant amount
> of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
> having your ethernet connection go offline for 3 seconds may not be the desired
> effect for when you want to get more bandwidth in the first place.
> 
> However, when a laptop is in battery mode, switching down from gigabit to 100mbit
> makes a lot more sense, so this is something I would recommend. This can be as
> easy as changing the advertisement mask of the interface and renegotiating the
> link. Userspace could handle that very easily.

If something does that, it must *only* be in userspace so that we can
easily disable it. It's amazing how many laptops consider that you
don't want any performance when you run off batteries. I've seen a
2GHz laptop falling back to 600 MHz when running on batteries, which
was very inconvenient when the laptop in question was used to go
sniffing gigabit traffic in datacenters... I would even go as far
as to say that my notebook runs lowpower only when it's plugged into
the wall because it's when I'm typing or doing low activity things.

In my opinion, battery != low power, battery == mobility. It's user's
choice which must imply low power, so that must be done with a dedicated
daemon.

Regards,
Willy

Re: gigabit ethernet power consumption

[Lennart Sorensen]

On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
> you most certainly want to do this in userspace I think.
> 
> One of the biggest problems is that link negotiation can take a significant amount
> of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
> having your ethernet connection go offline for 3 seconds may not be the desired
> effect for when you want to get more bandwidth in the first place.
> 
> However, when a laptop is in battery mode, switching down from gigabit to 100mbit
> makes a lot more sense, so this is something I would recommend. This can be as
> easy as changing the advertisement mask of the interface and renegotiating the
> link. Userspace could handle that very easily.

Now if you were trying to transfer a lot of data to the laptop, would it
be more power efficient to do it at gigabit speeds so you can finish
sooner and shut down the machine entirely, or to slow to 100mbit and
take longer to do it, and hence spend more time powering the cpu and
ram?

--
Len Sorensen

Re: gigabit ethernet power consumption

[Kok, Auke]

Lennart Sorensen wrote:
> On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
>> you most certainly want to do this in userspace I think.
>>
>> One of the biggest problems is that link negotiation can take a significant amount
>> of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
>> having your ethernet connection go offline for 3 seconds may not be the desired
>> effect for when you want to get more bandwidth in the first place.
>>
>> However, when a laptop is in battery mode, switching down from gigabit to 100mbit
>> makes a lot more sense, so this is something I would recommend. This can be as
>> easy as changing the advertisement mask of the interface and renegotiating the
>> link. Userspace could handle that very easily.
> 
> Now if you were trying to transfer a lot of data to the laptop, would it
> be more power efficient to do it at gigabit speeds so you can finish
> sooner and shut down the machine entirely, or to slow to 100mbit and
> take longer to do it, and hence spend more time powering the cpu and
> ram?

my suspicion is that the cost of switching is much higher than what you would
consume running at 100mbit, even if the amount of data is quite large. going
offline to renegotiate the link would already cost you 3W typically.

I definately think that userspace is the right field to solve this problem: let
the users decide how to use the available power on their sytems through a decent
power profile tool (perhaps gnome-power-manager or something like that). This way
the user can choose.

Auke

Re: gigabit ethernet power consumption

[Mark Gross]

On Tue, Oct 09, 2007 at 11:41:17AM -0700, Kok, Auke wrote:
> Lennart Sorensen wrote:
> > On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
> >> you most certainly want to do this in userspace I think.
> >>
> >> One of the biggest problems is that link negotiation can take a significant amount
> >> of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
> >> having your ethernet connection go offline for 3 seconds may not be the desired
> >> effect for when you want to get more bandwidth in the first place.
> >>
> >> However, when a laptop is in battery mode, switching down from gigabit to 100mbit
> >> makes a lot more sense, so this is something I would recommend. This can be as
> >> easy as changing the advertisement mask of the interface and renegotiating the
> >> link. Userspace could handle that very easily.
> > 
> > Now if you were trying to transfer a lot of data to the laptop, would it
> > be more power efficient to do it at gigabit speeds so you can finish
> > sooner and shut down the machine entirely, or to slow to 100mbit and
> > take longer to do it, and hence spend more time powering the cpu and
> > ram?
> 
> my suspicion is that the cost of switching is much higher than what you would
> consume running at 100mbit, even if the amount of data is quite large. going
> offline to renegotiate the link would already cost you 3W typically.
> 
> I definately think that userspace is the right field to solve this problem: let
> the users decide how to use the available power on their sytems through a decent
> power profile tool (perhaps gnome-power-manager or something like that). This way
> the user can choose.
> 

Auke, 
I was wondering if we could use PM-QOS to have the driver drop to the
100Mb speed, when requests for bandwidth and latency are not in effect?

--mgross

Re: gigabit ethernet power consumption

[Kok, Auke]

Mark Gross wrote:
> On Tue, Oct 09, 2007 at 11:41:17AM -0700, Kok, Auke wrote:
>> Lennart Sorensen wrote:
>>> On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
>>>> you most certainly want to do this in userspace I think.
>>>>
>>>> One of the biggest problems is that link negotiation can take a significant amount
>>>> of time, well over several seconds (1 to 3 seconds typical) with gigabit, and
>>>> having your ethernet connection go offline for 3 seconds may not be the desired
>>>> effect for when you want to get more bandwidth in the first place.
>>>>
>>>> However, when a laptop is in battery mode, switching down from gigabit to 100mbit
>>>> makes a lot more sense, so this is something I would recommend. This can be as
>>>> easy as changing the advertisement mask of the interface and renegotiating the
>>>> link. Userspace could handle that very easily.
>>> Now if you were trying to transfer a lot of data to the laptop, would it
>>> be more power efficient to do it at gigabit speeds so you can finish
>>> sooner and shut down the machine entirely, or to slow to 100mbit and
>>> take longer to do it, and hence spend more time powering the cpu and
>>> ram?
>> my suspicion is that the cost of switching is much higher than what you would
>> consume running at 100mbit, even if the amount of data is quite large. going
>> offline to renegotiate the link would already cost you 3W typically.
>>
>> I definately think that userspace is the right field to solve this problem: let
>> the users decide how to use the available power on their sytems through a decent
>> power profile tool (perhaps gnome-power-manager or something like that). This way
>> the user can choose.
>>
> 
> Auke, 
> I was wondering if we could use PM-QOS to have the driver drop to the
> 100Mb speed, when requests for bandwidth and latency are not in effect?

sure, that would probably work just fine. Technically this is not that hard anyway

I suppose that HAL integration is lacking a bit and missing ethtool ioctl code for
this which would be another solution - and would work without any driver changes.

Auke

Re: gigabit ethernet power consumption

[K.Prasad]

On Wed, 10 Oct 2007 00:11:17 +0530, Kok, Auke <auke-jan.h.kok@intel.com>  
wrote:

> Lennart Sorensen wrote:
>> On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
>>> you most certainly want to do this in userspace I think.
>>>
>>> One of the biggest problems is that link negotiation can take a  
>>> significant amount
>>> of time, well over several seconds (1 to 3 seconds typical) with  
>>> gigabit, and
>>> having your ethernet connection go offline for 3 seconds may not be  
>>> the desired
>>> effect for when you want to get more bandwidth in the first place.
>>>
>>> However, when a laptop is in battery mode, switching down from gigabit  
>>> to 100mbit
>>> makes a lot more sense, so this is something I would recommend. This  
>>> can be as
>>> easy as changing the advertisement mask of the interface and  
>>> renegotiating the
>>> link. Userspace could handle that very easily.
>>
>> Now if you were trying to transfer a lot of data to the laptop, would it
>> be more power efficient to do it at gigabit speeds so you can finish
>> sooner and shut down the machine entirely, or to slow to 100mbit and
>> take longer to do it, and hence spend more time powering the cpu and
>> ram?
>
> my suspicion is that the cost of switching is much higher than what you  
> would
> consume running at 100mbit, even if the amount of data is quite large.  
> going
> offline to renegotiate the link would already cost you 3W typically.
>
> I definately think that userspace is the right field to solve this  
> problem: let
> the users decide how to use the available power on their sytems through  
> a decent
> power profile tool (perhaps gnome-power-manager or something like that).  
> This way
> the user can choose.
>
> Auke
> -

Perhaps interrupt moderation could be of help here (say - switch to lesser  
interrupts per unit of time when running on battery), which I find that  
the e1000 driver doesn't employ in the kernel presently. (For interrupt  
moderation, refer:  
http://download.intel.com/design/network/applnots/ap450.pdf.)

Without the side-effect of experiencing a link-flap when switching to a  
lower-speed (with its toll in terms of down-time for auto-negotiation,  
STP, etc), the Interrupt Moderation Algorithm dynamically adjusts the  
number of interrupts based on traffic - and presumably consume less power.  
For an "Optimise for Power" kind of profile - the driver can be loaded  
with a higher throttle rate during boot-time.

Thanks,
K.Prasad

Re: gigabit ethernet power consumption

[Kok, Auke]

K.Prasad wrote:
> On Wed, 10 Oct 2007 00:11:17 +0530, Kok, Auke <auke-jan.h.kok@intel.com>
> wrote:
> 
>> Lennart Sorensen wrote:
>>> On Mon, Oct 08, 2007 at 03:31:51PM -0700, Kok, Auke wrote:
>>>> you most certainly want to do this in userspace I think.
>>>>
>>>> One of the biggest problems is that link negotiation can take a
>>>> significant amount
>>>> of time, well over several seconds (1 to 3 seconds typical) with
>>>> gigabit, and
>>>> having your ethernet connection go offline for 3 seconds may not be
>>>> the desired
>>>> effect for when you want to get more bandwidth in the first place.
>>>>
>>>> However, when a laptop is in battery mode, switching down from
>>>> gigabit to 100mbit
>>>> makes a lot more sense, so this is something I would recommend. This
>>>> can be as
>>>> easy as changing the advertisement mask of the interface and
>>>> renegotiating the
>>>> link. Userspace could handle that very easily.
>>>
>>> Now if you were trying to transfer a lot of data to the laptop, would it
>>> be more power efficient to do it at gigabit speeds so you can finish
>>> sooner and shut down the machine entirely, or to slow to 100mbit and
>>> take longer to do it, and hence spend more time powering the cpu and
>>> ram?
>>
>> my suspicion is that the cost of switching is much higher than what
>> you would
>> consume running at 100mbit, even if the amount of data is quite large.
>> going
>> offline to renegotiate the link would already cost you 3W typically.
>>
>> I definately think that userspace is the right field to solve this
>> problem: let
>> the users decide how to use the available power on their sytems
>> through a decent
>> power profile tool (perhaps gnome-power-manager or something like
>> that). This way
>> the user can choose.
>>
>> Auke
>> -
> 
> Perhaps interrupt moderation could be of help here (say - switch to
> lesser interrupts per unit of time when running on battery), which I
> find that the e1000 driver doesn't employ in the kernel presently. (For
> interrupt moderation, refer:
> http://download.intel.com/design/network/applnots/ap450.pdf.)

Not true, interrupt moderation has always been part of the e1000 driver in one
form or another. In the basic form, you have always been able to set a manual
rate, which might help somewhat because it reduces CPU wakeups on mostly-idle
systems. Refer to the InterruptThrottleRate parameter of the e1000 module.

recent version of the driver actually adjust the irq rate dynamically.

> Without the side-effect of experiencing a link-flap when switching to a
> lower-speed (with its toll in terms of down-time for auto-negotiation,
> STP, etc), the Interrupt Moderation Algorithm dynamically adjusts the
> number of interrupts based on traffic - and presumably consume less
> power. For an "Optimise for Power" kind of profile - the driver can be
> loaded with a higher throttle rate during boot-time.

We're changing this to be run-time adjustable in newer drivers.

However, the power consumed by your nic staying in gigabit mode is much greater in
the long run then what you can save by trying to scrounge for milliwatts reducing
interrupts generated by the nic. By default it already moderates them somewhat.
Practically this feature is really not useful for powersaving, it just won't add
up to actual benefits in a real life situtation I think.

Auke

Re: gigabit ethernet power consumption

[Oliver Neukum]

Am Dienstag 09 Oktober 2007 schrieb Lennart Sorensen:

> Now if you were trying to transfer a lot of data to the laptop, would it
> be more power efficient to do it at gigabit speeds so you can finish
> sooner and shut down the machine entirely, or to slow to 100mbit and
> take longer to do it, and hence spend more time powering the cpu and
> ram?

There's an amount of data which would be the cutoff point.

But I don't think you can by default make machines slower. So as soon as
the link is saturated you have to switch to the higher speed. To avoid a
ping-pong effect I'd use a timer for bringing the link speed down again.

	Regards
		Oliver

Re: gigabit ethernet power consumption

[Chris Snook]

Pavel Machek wrote:
> Hi!
> 
> I've found that gbit vs. 100mbit power consumption difference is about
> 1W -- pretty significant. (Maybe powertop should include it in the
> tips section? :).
> 
> Energy Star people insist that machines should switch down to 100mbit
> when network is idle, and I guess that makes a lot of sense -- you
> save 1W locally and 1W on the router.
> 
> Question is, how to implement it correctly? Daemon that would watch
> data rates and switch speeds using mii-tool would be simple, but is
> that enough?

I believe you misspelled "ethtool".

While you're at it, why stop at 100Mb?  I believe you save even more power at 
10Mb, which is why WOL puts the card in 10Mb mode.  In my experience, you 
generally want either the maximum setting or the minimum setting when going for 
power savings, because of the race-to-idle effect.  Workloads that have a 
sustained fractional utilization are rare.  Right now I'm at home, hooked up to 
a cable modem, so anything over 4Mb is wasted, unless I'm talking to the box 
across the room, which is rare.

Talk to the NetworkManager folks.  This is right up their alley.

	-- Chris

Re: gigabit ethernet power consumption

[Oliver Neukum]

Am Dienstag 09 Oktober 2007 schrieb Pavel Machek:
> Question is, how to implement it correctly? Daemon that would watch
> data rates and switch speeds using mii-tool would be simple, but is
> that enough?

Do you only want to affect true ethernet devices this way? It seems
to me that the savings for wireless devices could be larger and we don't
want a separate mechanism for each type of network device. So I think
you need to tell the kernel why you want to reduce the link speed.

	Regards
		Oliver

Re: gigabit ethernet power consumption

[Bodo Eggert]

Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> K.Prasad wrote:

>> Without the side-effect of experiencing a link-flap when switching to a
>> lower-speed (with its toll in terms of down-time for auto-negotiation,
>> STP, etc), the Interrupt Moderation Algorithm dynamically adjusts the
>> number of interrupts based on traffic - and presumably consume less
>> power. For an "Optimise for Power" kind of profile - the driver can be
>> loaded with a higher throttle rate during boot-time.
> 
> We're changing this to be run-time adjustable in newer drivers.
> 
> However, the power consumed by your nic staying in gigabit mode is much
> greater in the long run then what you can save by trying to scrounge for
> milliwatts reducing interrupts generated by the nic. By default it already
> moderates them somewhat. Practically this feature is really not useful for
> powersaving, it just won't add up to actual benefits in a real life situtation
> I think.

Just a thought:
How much power does a non-connected NIC consume, and can you save power
by forcing 10 MBit until a link is detected (doubling negotiation time)?
-- 
Top 100 things you don't want the sysadmin to say:
22. hey, what does mkfs do?

Friß, Spammer: 9nnbse9@lgrrp.7eggert.dyndns.org

Re: gigabit ethernet power consumption

[Kok, Auke]

Bodo Eggert wrote:
> Kok, Auke <auke-jan.h.kok@intel.com> wrote:
>> K.Prasad wrote:
> 
>>> Without the side-effect of experiencing a link-flap when switching to a
>>> lower-speed (with its toll in terms of down-time for auto-negotiation,
>>> STP, etc), the Interrupt Moderation Algorithm dynamically adjusts the
>>> number of interrupts based on traffic - and presumably consume less
>>> power. For an "Optimise for Power" kind of profile - the driver can be
>>> loaded with a higher throttle rate during boot-time.
>> We're changing this to be run-time adjustable in newer drivers.
>>
>> However, the power consumed by your nic staying in gigabit mode is much
>> greater in the long run then what you can save by trying to scrounge for
>> milliwatts reducing interrupts generated by the nic. By default it already
>> moderates them somewhat. Practically this feature is really not useful for
>> powersaving, it just won't add up to actual benefits in a real life situtation
>> I think.
> 
> Just a thought:
> How much power does a non-connected NIC consume, and can you save power
> by forcing 10 MBit until a link is detected (doubling negotiation time)?

no, the PHY consumes a minimal amount of energy when not connected, regardless of
whether it is advertising 10, 100 or 1000mbit.

Cheers,

Auke

Re: gigabit ethernet power consumption

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

On Fri, 12 Oct 2007 09:35:15 PDT, "Kok, Auke" said:

> > How much power does a non-connected NIC consume, and can you save power
> > by forcing 10 MBit until a link is detected (doubling negotiation time)?
> 
> no, the PHY consumes a minimal amount of energy when not connected, regardless of
> whether it is advertising 10, 100 or 1000mbit.

Is that true for essentially all the chipsets we support (or at least "the vast
majority of the ones currently found in common machines")?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: gigabit ethernet power consumption

[Kok, Auke]

Valdis.Kletnieks@vt.edu wrote:
> On Fri, 12 Oct 2007 09:35:15 PDT, "Kok, Auke" said:
> 
>>> How much power does a non-connected NIC consume, and can you save power
>>> by forcing 10 MBit until a link is detected (doubling negotiation time)?
>> no, the PHY consumes a minimal amount of energy when not connected, regardless of
>> whether it is advertising 10, 100 or 1000mbit.
> 
> Is that true for essentially all the chipsets we support (or at least "the vast
> majority of the ones currently found in common machines")?

I would assume that that is true for all PHY's - if there is no link to keep the
carrier active on I would think that the power consumption is nominal across the
board. Once the PHY detects link pulses it should obviously use different power
levels to negotiate the link for each speed.

PHY autonegotiation just works the same across the board - the PHY's send out
little pulses (iow nominal power consumption) to detect a link partner, and only
when one is found do they engage in more aggressive conversation to establish
capabilities across the link.


Auke

Re: gigabit ethernet power consumption

[Lennart Sorensen]

On Fri, Oct 12, 2007 at 01:46:33PM -0700, Kok, Auke wrote:
> I would assume that that is true for all PHY's - if there is no link to keep the
> carrier active on I would think that the power consumption is nominal across the
> board. Once the PHY detects link pulses it should obviously use different power
> levels to negotiate the link for each speed.
> 
> PHY autonegotiation just works the same across the board - the PHY's send out
> little pulses (iow nominal power consumption) to detect a link partner, and only
> when one is found do they engage in more aggressive conversation to establish
> capabilities across the link.

Too bad not much can be done about fibre ports, but then again you can't
even change speeds on those.  Then again those are not on laptops or
other low power devices in the first place.

--
Len Sorensen

Re: gigabit ethernet power consumption

[Kok, Auke]

Lennart Sorensen wrote:
> On Fri, Oct 12, 2007 at 01:46:33PM -0700, Kok, Auke wrote:
>> I would assume that that is true for all PHY's - if there is no link to keep the
>> carrier active on I would think that the power consumption is nominal across the
>> board. Once the PHY detects link pulses it should obviously use different power
>> levels to negotiate the link for each speed.
>>
>> PHY autonegotiation just works the same across the board - the PHY's send out
>> little pulses (iow nominal power consumption) to detect a link partner, and only
>> when one is found do they engage in more aggressive conversation to establish
>> capabilities across the link.
> 
> Too bad not much can be done about fibre ports, but then again you can't
> even change speeds on those.  Then again those are not on laptops or
> other low power devices in the first place.

the only thing we do for fiber ports is power off the laser if the user explicitly
powers them down at shutdown. Note that WoL needs to be disabled for that.

Auke