Performance/resume issues on Toshiba NB305

Performance/resume issues on Toshiba NB305

[Seth Forshee]

I've been looking into a couple of problems with this machine that have
me a bit stumped at the moment. The two problems may or may not be
related, so I'm including details about both issues below. If anyone has
any ideas, I'd love to hear them. Note that these are not recent
regressions; they've been around since at least 2.6.32.

The CPU in this machine is an Atom N450.

When booted normally the performance on this machine is very poor, but
when booted with any of nohz=off, nolapic, or nohpet it improves
significantly. The performance also improves if I use the patch below to
force the hpet to remain in periodic mode (with hpet=periodic on the
command line).

One other thing I noticed when I had added some logging related to hpet
rearming is 3-5 second periods of no log activity occurring fairly
frequently, whereas such inactive periods are infrequent when
performance is good and are also infrequent on another machine with very
similar hardware and no performance issues.

The machine also hangs for 5 minutes during resume, unless booted with
both nohz=off and highres=off, or with hpet=periodic using the patch
below. I've traced this down to hanging in an SMI handler during the
ACPI _WAK method execution. The 5 minutes corresponds to how long it
takes for the low 32 bits of the hpet to wrap in this machine, and since
the options that eliminate the hang result in the hpet being in periodic
mode during _WAK method execution I suspect that the SMI handler is
hanging until a timer interrupt happens.

One possible explanation here is that the performance problems are also
related to hangs in SMI handlers until there's a timer interrupt,
although I don't know how that explains why some of the command line
options eliminate the performance issues.

Note also that I've already looked into the triggering for IRQ0, and it
seems to be correct.

Thanks,
Seth


diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
index 4ff5968..ab10012 100644
--- a/arch/x86/kernel/hpet.c
+++ b/arch/x86/kernel/hpet.c
@@ -87,6 +87,7 @@ static inline void hpet_clear_mapping(void)
 static int boot_hpet_disable;
 int hpet_force_user;
 static int hpet_verbose;
+static int hpet_oneshot_disable;
 
 static int __init hpet_setup(char *str)
 {
@@ -97,6 +98,8 @@ static int __init hpet_setup(char *str)
 			hpet_force_user = 1;
 		if (!strncmp("verbose", str, 7))
 			hpet_verbose = 1;
+		if (!strncmp("periodic", str, 8))
+			hpet_oneshot_disable = 1;
 	}
 	return 1;
 }
@@ -306,6 +309,10 @@ static void hpet_legacy_clockevent_register(void)
 	hpet_clockevent.min_delta_ns = clockevent_delta2ns(HPET_MIN_PROG_DELTA,
 							   &hpet_clockevent);
 
+	/* Disable one-shot mode if requested */
+	if (hpet_oneshot_disable)
+		hpet_clockevent.features &= ~CLOCK_EVT_FEAT_ONESHOT;
+
 	/*
 	 * Start hpet with the boot cpu mask and make it
 	 * global after the IO_APIC has been initialized.
diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index 48b2761..8e84c11 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
 
 	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 
-	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	bc = tick_broadcast_device.evtdev;
+	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
+		goto unlock;
+
+	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
 	if (bc)
 		tick_broadcast_setup_oneshot(bc);
+
+unlock:
 	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
 }
 

Re: Performance/resume issues on Toshiba NB305

[Steven Rostedt]

Would be helpful to Cc the people that may help you.


On Fri, Feb 25, 2011 at 10:42:39AM -0600, Seth Forshee wrote:
> I've been looking into a couple of problems with this machine that have
> me a bit stumped at the moment. The two problems may or may not be
> related, so I'm including details about both issues below. If anyone has
> any ideas, I'd love to hear them. Note that these are not recent
> regressions; they've been around since at least 2.6.32.
> 
> The CPU in this machine is an Atom N450.
> 
> When booted normally the performance on this machine is very poor, but
> when booted with any of nohz=off, nolapic, or nohpet it improves
> significantly. The performance also improves if I use the patch below to
> force the hpet to remain in periodic mode (with hpet=periodic on the
> command line).
> 
> One other thing I noticed when I had added some logging related to hpet
> rearming is 3-5 second periods of no log activity occurring fairly
> frequently, whereas such inactive periods are infrequent when
> performance is good and are also infrequent on another machine with very
> similar hardware and no performance issues.
> 
> The machine also hangs for 5 minutes during resume, unless booted with
> both nohz=off and highres=off, or with hpet=periodic using the patch
> below. I've traced this down to hanging in an SMI handler during the
> ACPI _WAK method execution. The 5 minutes corresponds to how long it
> takes for the low 32 bits of the hpet to wrap in this machine, and since
> the options that eliminate the hang result in the hpet being in periodic
> mode during _WAK method execution I suspect that the SMI handler is
> hanging until a timer interrupt happens.
> 
> One possible explanation here is that the performance problems are also
> related to hangs in SMI handlers until there's a timer interrupt,
> although I don't know how that explains why some of the command line
> options eliminate the performance issues.

Sounds like a buggy BIOS to me. Is there upgrades available for this
box?

-- Steve

> 
> Note also that I've already looked into the triggering for IRQ0, and it
> seems to be correct.
> 
> Thanks,
> Seth
> 
> 
> diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
> index 4ff5968..ab10012 100644
> --- a/arch/x86/kernel/hpet.c
> +++ b/arch/x86/kernel/hpet.c
> @@ -87,6 +87,7 @@ static inline void hpet_clear_mapping(void)
>  static int boot_hpet_disable;
>  int hpet_force_user;
>  static int hpet_verbose;
> +static int hpet_oneshot_disable;
>  
>  static int __init hpet_setup(char *str)
>  {
> @@ -97,6 +98,8 @@ static int __init hpet_setup(char *str)
>  			hpet_force_user = 1;
>  		if (!strncmp("verbose", str, 7))
>  			hpet_verbose = 1;
> +		if (!strncmp("periodic", str, 8))
> +			hpet_oneshot_disable = 1;
>  	}
>  	return 1;
>  }
> @@ -306,6 +309,10 @@ static void hpet_legacy_clockevent_register(void)
>  	hpet_clockevent.min_delta_ns = clockevent_delta2ns(HPET_MIN_PROG_DELTA,
>  							   &hpet_clockevent);
>  
> +	/* Disable one-shot mode if requested */
> +	if (hpet_oneshot_disable)
> +		hpet_clockevent.features &= ~CLOCK_EVT_FEAT_ONESHOT;
> +
>  	/*
>  	 * Start hpet with the boot cpu mask and make it
>  	 * global after the IO_APIC has been initialized.
> diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
> index 48b2761..8e84c11 100644
> --- a/kernel/time/tick-broadcast.c
> +++ b/kernel/time/tick-broadcast.c
> @@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
>  
>  	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
>  
> -	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
>  	bc = tick_broadcast_device.evtdev;
> +	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
> +		goto unlock;
> +
> +	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
>  	if (bc)
>  		tick_broadcast_setup_oneshot(bc);
> +
> +unlock:
>  	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
>  }
>  

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 03:17:47PM -0500, Steven Rostedt wrote:
> Would be helpful to Cc the people that may help you.
> 
> 
> On Fri, Feb 25, 2011 at 10:42:39AM -0600, Seth Forshee wrote:
> > I've been looking into a couple of problems with this machine that have
> > me a bit stumped at the moment. The two problems may or may not be
> > related, so I'm including details about both issues below. If anyone has
> > any ideas, I'd love to hear them. Note that these are not recent
> > regressions; they've been around since at least 2.6.32.
> > 
> > The CPU in this machine is an Atom N450.
> > 
> > When booted normally the performance on this machine is very poor, but
> > when booted with any of nohz=off, nolapic, or nohpet it improves
> > significantly. The performance also improves if I use the patch below to
> > force the hpet to remain in periodic mode (with hpet=periodic on the
> > command line).
> > 
> > One other thing I noticed when I had added some logging related to hpet
> > rearming is 3-5 second periods of no log activity occurring fairly
> > frequently, whereas such inactive periods are infrequent when
> > performance is good and are also infrequent on another machine with very
> > similar hardware and no performance issues.
> > 
> > The machine also hangs for 5 minutes during resume, unless booted with
> > both nohz=off and highres=off, or with hpet=periodic using the patch
> > below. I've traced this down to hanging in an SMI handler during the
> > ACPI _WAK method execution. The 5 minutes corresponds to how long it
> > takes for the low 32 bits of the hpet to wrap in this machine, and since
> > the options that eliminate the hang result in the hpet being in periodic
> > mode during _WAK method execution I suspect that the SMI handler is
> > hanging until a timer interrupt happens.
> > 
> > One possible explanation here is that the performance problems are also
> > related to hangs in SMI handlers until there's a timer interrupt,
> > although I don't know how that explains why some of the command line
> > options eliminate the performance issues.
> 
> Sounds like a buggy BIOS to me. Is there upgrades available for this
> box?

It does sound like a buggy BIOS. I've already flashed the latest BIOS
though, so no luck there.

Re: Performance/resume issues on Toshiba NB305

[Thomas Gleixner]

On Fri, 25 Feb 2011, Seth Forshee wrote:

Please always Cc the relevant maintainers. I just got notified by
someone who accidentaly stumbled over that.

> I've been looking into a couple of problems with this machine that have
> me a bit stumped at the moment. The two problems may or may not be
> related, so I'm including details about both issues below. If anyone has
> any ideas, I'd love to hear them. Note that these are not recent
> regressions; they've been around since at least 2.6.32.
> 
> The CPU in this machine is an Atom N450.
> 
> When booted normally the performance on this machine is very poor, but
> when booted with any of nohz=off, nolapic, or nohpet it improves
> significantly. The performance also improves if I use the patch below to
> force the hpet to remain in periodic mode (with hpet=periodic on the
> command line).
> 
> One other thing I noticed when I had added some logging related to hpet
> rearming is 3-5 second periods of no log activity occurring fairly
> frequently, whereas such inactive periods are infrequent when
> performance is good and are also infrequent on another machine with very
> similar hardware and no performance issues.

That seems to be related to low power states. When the machine goes
idle we switch into lower power states and that requires to use the
hpet instead of the local apic timer as that one stops.

You could verify that theory by booting with processor.max_cstate=1

> The machine also hangs for 5 minutes during resume, unless booted with
> both nohz=off and highres=off, or with hpet=periodic using the patch
> below. I've traced this down to hanging in an SMI handler during the
> ACPI _WAK method execution. The 5 minutes corresponds to how long it
> takes for the low 32 bits of the hpet to wrap in this machine, and since
> the options that eliminate the hang result in the hpet being in periodic
> mode during _WAK method execution I suspect that the SMI handler is
> hanging until a timer interrupt happens.
> 
> One possible explanation here is that the performance problems are also
> related to hangs in SMI handlers until there's a timer interrupt,
> although I don't know how that explains why some of the command line
> options eliminate the performance issues.

Well, that's simple. If you disable NOHZ then we never go into deep
idle because the next timer interrupt will arrive in a very short time
span. nohpet makes it use the PIT which has a very short (25ms)
maximum oneshot time. nolapic disables a lot of the functionality as
well.
 
> --- a/kernel/time/tick-broadcast.c
> +++ b/kernel/time/tick-broadcast.c
> @@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
>  
>  	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
>  
> -	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
>  	bc = tick_broadcast_device.evtdev;
> +	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
> +		goto unlock;
> +
> +	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
>  	if (bc)
>  		tick_broadcast_setup_oneshot(bc);
> +
> +unlock:
>  	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
>  }

Why would you need that? We should not call that when the broadcast
device does not have TICKDEV_MODE_ONESHOT. If we do the bug is
somewhere else.

Thanks,

	tglx

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 09:37:39PM +0100, Thomas Gleixner wrote:
> On Fri, 25 Feb 2011, Seth Forshee wrote:
> 
> Please always Cc the relevant maintainers. I just got notified by
> someone who accidentaly stumbled over that.

I'll be sure to remember that in the future.

> > I've been looking into a couple of problems with this machine that have
> > me a bit stumped at the moment. The two problems may or may not be
> > related, so I'm including details about both issues below. If anyone has
> > any ideas, I'd love to hear them. Note that these are not recent
> > regressions; they've been around since at least 2.6.32.
> > 
> > The CPU in this machine is an Atom N450.
> > 
> > When booted normally the performance on this machine is very poor, but
> > when booted with any of nohz=off, nolapic, or nohpet it improves
> > significantly. The performance also improves if I use the patch below to
> > force the hpet to remain in periodic mode (with hpet=periodic on the
> > command line).
> > 
> > One other thing I noticed when I had added some logging related to hpet
> > rearming is 3-5 second periods of no log activity occurring fairly
> > frequently, whereas such inactive periods are infrequent when
> > performance is good and are also infrequent on another machine with very
> > similar hardware and no performance issues.
> 
> That seems to be related to low power states. When the machine goes
> idle we switch into lower power states and that requires to use the
> hpet instead of the local apic timer as that one stops.
> 
> You could verify that theory by booting with processor.max_cstate=1

This fixes the performance in combination with intel_idle.max_cstate=0.
Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.

> > The machine also hangs for 5 minutes during resume, unless booted with
> > both nohz=off and highres=off, or with hpet=periodic using the patch
> > below. I've traced this down to hanging in an SMI handler during the
> > ACPI _WAK method execution. The 5 minutes corresponds to how long it
> > takes for the low 32 bits of the hpet to wrap in this machine, and since
> > the options that eliminate the hang result in the hpet being in periodic
> > mode during _WAK method execution I suspect that the SMI handler is
> > hanging until a timer interrupt happens.
> > 
> > One possible explanation here is that the performance problems are also
> > related to hangs in SMI handlers until there's a timer interrupt,
> > although I don't know how that explains why some of the command line
> > options eliminate the performance issues.
> 
> Well, that's simple. If you disable NOHZ then we never go into deep
> idle because the next timer interrupt will arrive in a very short time
> span. nohpet makes it use the PIT which has a very short (25ms)
> maximum oneshot time. nolapic disables a lot of the functionality as
> well.

Makes sense.

Is the answer to quirk the machine to avoid deep C-states, or is there
some better way I can fix this?

> > --- a/kernel/time/tick-broadcast.c
> > +++ b/kernel/time/tick-broadcast.c
> > @@ -565,10 +565,15 @@ void tick_broadcast_switch_to_oneshot(void)
> >  
> >  	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
> >  
> > -	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
> >  	bc = tick_broadcast_device.evtdev;
> > +	if (bc && !(bc->features & CLOCK_EVT_FEAT_ONESHOT))
> > +		goto unlock;
> > +
> > +	tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
> >  	if (bc)
> >  		tick_broadcast_setup_oneshot(bc);
> > +
> > +unlock:
> >  	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
> >  }
> 
> Why would you need that? We should not call that when the broadcast
> device does not have TICKDEV_MODE_ONESHOT. If we do the bug is
> somewhere else.

Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the
HPET without this change the HPET got put into oneshot mode. The local
tick device is checked before switching to nohz, but not the broadcast
device. This change was just a quick hack to get around that and test my
theory.

Re: Performance/resume issues on Toshiba NB305

[Thomas Gleixner]

On Fri, 25 Feb 2011, Seth Forshee wrote:
> On Fri, Feb 25, 2011 at 09:37:39PM +0100, Thomas Gleixner wrote:
> > On Fri, 25 Feb 2011, Seth Forshee wrote:
> > That seems to be related to low power states. When the machine goes
> > idle we switch into lower power states and that requires to use the
> > hpet instead of the local apic timer as that one stops.
> > 
> > You could verify that theory by booting with processor.max_cstate=1
> 
> This fixes the performance in combination with intel_idle.max_cstate=0.
> Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.

That was expected :)

> Is the answer to quirk the machine to avoid deep C-states, or is there
> some better way I can fix this?

Let's wait for the intel and acpi folks. It would be interesting what
the new intel toy says to your BIOS.

    http://biosbits.org/

> > > +unlock:
> > >  	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
> > >  }
> > 
> > Why would you need that? We should not call that when the broadcast
> > device does not have TICKDEV_MODE_ONESHOT. If we do the bug is
> > somewhere else.
> 
> Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the
> HPET without this change the HPET got put into oneshot mode. The local
> tick device is checked before switching to nohz, but not the broadcast
> device. This change was just a quick hack to get around that and test my
> theory.

Indeed. The patch below should cure that.

Thanks,

	tglx

----------->
Subject: clockevents-fix-broadcast.patch
From: Thomas Gleixner <tglx@linutronix.de>
Date: Fri, 25 Feb 2011 22:34:23 +0100

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 kernel/time/tick-broadcast.c |   10 ++++++++++
 kernel/time/tick-common.c    |    6 +++++-
 kernel/time/tick-internal.h  |    3 +++
 3 files changed, 18 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/time/tick-broadcast.c
===================================================================
--- linux-2.6.orig/kernel/time/tick-broadcast.c
+++ linux-2.6/kernel/time/tick-broadcast.c
@@ -600,4 +600,14 @@ int tick_broadcast_oneshot_active(void)
 	return tick_broadcast_device.mode == TICKDEV_MODE_ONESHOT;
 }
 
+/*
+ * Check whether the broadcast device supports oneshot.
+ */
+bool tick_broadcast_oneshot_available(void)
+{
+	struct clock_event_device *bc = tick_broadcast_device.evtdev;
+
+	return bc ? bc->features & CLOCK_EVT_FEAT_ONESHOT : false;
+}
+
 #endif
Index: linux-2.6/kernel/time/tick-common.c
===================================================================
--- linux-2.6.orig/kernel/time/tick-common.c
+++ linux-2.6/kernel/time/tick-common.c
@@ -51,7 +51,11 @@ int tick_is_oneshot_available(void)
 {
 	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
 
-	return dev && (dev->features & CLOCK_EVT_FEAT_ONESHOT);
+	if (!dev || !(dev->features & CLOCK_EVT_FEAT_ONESHOT))
+		return 0;
+	if (!(dev->features & CLOCK_EVT_FEAT_C3STOP))
+		return 1;
+	return tick_broadcast_oneshot_available();
 }
 
 /*
Index: linux-2.6/kernel/time/tick-internal.h
===================================================================
--- linux-2.6.orig/kernel/time/tick-internal.h
+++ linux-2.6/kernel/time/tick-internal.h
@@ -36,6 +36,7 @@ extern void tick_shutdown_broadcast_ones
 extern int tick_resume_broadcast_oneshot(struct clock_event_device *bc);
 extern int tick_broadcast_oneshot_active(void);
 extern void tick_check_oneshot_broadcast(int cpu);
+bool tick_broadcast_oneshot_available(void);
 # else /* BROADCAST */
 static inline void tick_broadcast_setup_oneshot(struct clock_event_device *bc)
 {
@@ -46,6 +47,7 @@ static inline void tick_broadcast_switch
 static inline void tick_shutdown_broadcast_oneshot(unsigned int *cpup) { }
 static inline int tick_broadcast_oneshot_active(void) { return 0; }
 static inline void tick_check_oneshot_broadcast(int cpu) { }
+static inline bool tick_broadcast_oneshot_available(void) { return true; }
 # endif /* !BROADCAST */
 
 #else /* !ONESHOT */
@@ -76,6 +78,7 @@ static inline int tick_resume_broadcast_
 	return 0;
 }
 static inline int tick_broadcast_oneshot_active(void) { return 0; }
+static inline bool tick_broadcast_oneshot_available(void) { return true; }
 #endif /* !TICK_ONESHOT */
 
 /*

Re: Performance/resume issues on Toshiba NB305

[Thomas Gleixner]

On Fri, 25 Feb 2011, Seth Forshee wrote:
> On Fri, Feb 25, 2011 at 03:17:47PM -0500, Steven Rostedt wrote:
> > > One possible explanation here is that the performance problems are also
> > > related to hangs in SMI handlers until there's a timer interrupt,
> > > although I don't know how that explains why some of the command line
> > > options eliminate the performance issues.
> > 
> > Sounds like a buggy BIOS to me. Is there upgrades available for this
> > box?
> 
> It does sound like a buggy BIOS. I've already flashed the latest BIOS
> though, so no luck there.

Has the BIOS some knob to disable that feature^Wcrap? It's often
hidden behind some "$manufacturer super extra mode".

Thanks,

	tglx

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 10:47:16PM +0100, Thomas Gleixner wrote:
> Let's wait for the intel and acpi folks. It would be interesting what
> the new intel toy says to your BIOS.
> 
>     http://biosbits.org/

Not much because it doesn't know about my processor. About all I could
get out of it is that my MSRs are inconsistent, SMI latency is bad, and
the round-trip latency via MWAIT test gives elapsed time = 285ms with
229 iterations/ms.

> > Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the
> > HPET without this change the HPET got put into oneshot mode. The local
> > tick device is checked before switching to nohz, but not the broadcast
> > device. This change was just a quick hack to get around that and test my
> > theory.
> 
> Indeed. The patch below should cure that.

It works. I was wondering whether or not I should put the broadcast
device in periodic mode with the local ones in nohz; I guess your patch
answers my question.

> 
> Thanks,
> 
> 	tglx
> 
> ----------->
> Subject: clockevents-fix-broadcast.patch
> From: Thomas Gleixner <tglx@linutronix.de>
> Date: Fri, 25 Feb 2011 22:34:23 +0100
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Tested-by: Seth Forshee <seth.forshee@canonical.com>

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 11:11:31PM +0100, Thomas Gleixner wrote:
> On Fri, 25 Feb 2011, Seth Forshee wrote:
> > On Fri, Feb 25, 2011 at 03:17:47PM -0500, Steven Rostedt wrote:
> > > > One possible explanation here is that the performance problems are also
> > > > related to hangs in SMI handlers until there's a timer interrupt,
> > > > although I don't know how that explains why some of the command line
> > > > options eliminate the performance issues.
> > > 
> > > Sounds like a buggy BIOS to me. Is there upgrades available for this
> > > box?
> > 
> > It does sound like a buggy BIOS. I've already flashed the latest BIOS
> > though, so no luck there.
> 
> Has the BIOS some knob to disable that feature^Wcrap? It's often
> hidden behind some "$manufacturer super extra mode".

You mean the "hang in SMI until a timer interrupt fires" feature? ;)

There's nothing in the BIOS settings that looks relevant to either of
the topics at hand.

Re: Performance/resume issues on Toshiba NB305

[Thomas Gleixner]

On Fri, 25 Feb 2011, Seth Forshee wrote:

> On Fri, Feb 25, 2011 at 10:47:16PM +0100, Thomas Gleixner wrote:
> > Let's wait for the intel and acpi folks. It would be interesting what
> > the new intel toy says to your BIOS.
> > 
> >     http://biosbits.org/
> 
> Not much because it doesn't know about my processor. About all I could
> get out of it is that my MSRs are inconsistent, SMI latency is bad, and
> the round-trip latency via MWAIT test gives elapsed time = 285ms with
> 229 iterations/ms.

Ouch.
 
> > > Then there must be a bug. When I cleared CLOCK_EVT_FEAT_ONESHOT for the
> > > HPET without this change the HPET got put into oneshot mode. The local
> > > tick device is checked before switching to nohz, but not the broadcast
> > > device. This change was just a quick hack to get around that and test my
> > > theory.
> > 
> > Indeed. The patch below should cure that.
> 
> It works. I was wondering whether or not I should put the broadcast
> device in periodic mode with the local ones in nohz; I guess your patch
> answers my question.

Well, there is no point having the local ones in nohz mode when
broadcast one does not support it. You only get power saving when your
box stays in deep power modes for a long time. So with the BC periodic
it will try to go into deep power states (not knowing about the
periodic BC issue) and pop out of it in the same way as you do with
nohz disabled. So the power saving effect is approx. zero.

Thanks,

	tglx

Re: Performance/resume issues on Toshiba NB305

[Burt Triplett]

On 02/25/2011 02:40 PM, Thomas Gleixner wrote:
> On Fri, 25 Feb 2011, Seth Forshee wrote:
>> On Fri, Feb 25, 2011 at 10:47:16PM +0100, Thomas Gleixner wrote:
>>> Let's wait for the intel and acpi folks. It would be interesting what
>>> the new intel toy says to your BIOS.
>>>
>>>      http://biosbits.org/
>>
>> Not much because it doesn't know about my processor. About all I could
>> get out of it is that my MSRs are inconsistent, SMI latency is bad, and
>> the round-trip latency via MWAIT test gives elapsed time = 285ms with
>> 229 iterations/ms.
>
> Ouch.

Looks like the Toshiba NB305 has an Intel Atom processor in it. 
bits-327, released today, now detects Intel Atom processors, and no 
longer shows them as unknown.  We don't have any processor-specific 
tests for Atom yet, though we may add some in the future.  All of the 
general tests should work, however.

Can you provide more details on which MSRs show up as inconsistent on 
your system?  They might represent actual bugs in your BIOS, but we 
might just need to add a few more entries to the Atom CPU-specific MSR 
blacklist as expected inconsistencies.

- Burt

[tip:timers/urgent] clockevents: Prevent oneshot mode when broadcast device is periodic

[tip-bot for Thomas Gleixner]

Commit-ID:  3a142a0672b48a853f00af61f184c7341ac9c99d
Gitweb:     http://git.kernel.org/tip/3a142a0672b48a853f00af61f184c7341ac9c99d
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Fri, 25 Feb 2011 22:34:23 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 26 Feb 2011 09:45:28 +0100

clockevents: Prevent oneshot mode when broadcast device is periodic

When the per cpu timer is marked CLOCK_EVT_FEAT_C3STOP, then we only
can switch into oneshot mode, when the backup broadcast device
supports oneshot mode as well. Otherwise we would try to switch the
broadcast device into an unsupported mode unconditionally. This went
unnoticed so far as the current available broadcast devices support
oneshot mode. Seth unearthed this problem while debugging and working
around an hpet related BIOS wreckage.

Add the necessary check to tick_is_oneshot_available().

Reported-and-tested-by: Seth Forshee <seth.forshee@canonical.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
LKML-Reference: <alpine.LFD.2.00.1102252231200.2701@localhost6.localdomain6>
Cc: stable@kernel.org # .21 ->
---
 kernel/time/tick-broadcast.c |   10 ++++++++++
 kernel/time/tick-common.c    |    6 +++++-
 kernel/time/tick-internal.h  |    3 +++
 3 files changed, 18 insertions(+), 1 deletions(-)

diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index 48b2761..a3b5aff 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -600,4 +600,14 @@ int tick_broadcast_oneshot_active(void)
 	return tick_broadcast_device.mode == TICKDEV_MODE_ONESHOT;
 }
 
+/*
+ * Check whether the broadcast device supports oneshot.
+ */
+bool tick_broadcast_oneshot_available(void)
+{
+	struct clock_event_device *bc = tick_broadcast_device.evtdev;
+
+	return bc ? bc->features & CLOCK_EVT_FEAT_ONESHOT : false;
+}
+
 #endif
diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index 051bc80..ed228ef 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -51,7 +51,11 @@ int tick_is_oneshot_available(void)
 {
 	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
 
-	return dev && (dev->features & CLOCK_EVT_FEAT_ONESHOT);
+	if (!dev || !(dev->features & CLOCK_EVT_FEAT_ONESHOT))
+		return 0;
+	if (!(dev->features & CLOCK_EVT_FEAT_C3STOP))
+		return 1;
+	return tick_broadcast_oneshot_available();
 }
 
 /*
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 290eefb..f65d3a7 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -36,6 +36,7 @@ extern void tick_shutdown_broadcast_oneshot(unsigned int *cpup);
 extern int tick_resume_broadcast_oneshot(struct clock_event_device *bc);
 extern int tick_broadcast_oneshot_active(void);
 extern void tick_check_oneshot_broadcast(int cpu);
+bool tick_broadcast_oneshot_available(void);
 # else /* BROADCAST */
 static inline void tick_broadcast_setup_oneshot(struct clock_event_device *bc)
 {
@@ -46,6 +47,7 @@ static inline void tick_broadcast_switch_to_oneshot(void) { }
 static inline void tick_shutdown_broadcast_oneshot(unsigned int *cpup) { }
 static inline int tick_broadcast_oneshot_active(void) { return 0; }
 static inline void tick_check_oneshot_broadcast(int cpu) { }
+static inline bool tick_broadcast_oneshot_available(void) { return true; }
 # endif /* !BROADCAST */
 
 #else /* !ONESHOT */
@@ -76,6 +78,7 @@ static inline int tick_resume_broadcast_oneshot(struct clock_event_device *bc)
 	return 0;
 }
 static inline int tick_broadcast_oneshot_active(void) { return 0; }
+static inline bool tick_broadcast_oneshot_available(void) { return false; }
 #endif /* !TICK_ONESHOT */
 
 /*

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 09:58:25PM -0800, Burt Triplett wrote:
> Looks like the Toshiba NB305 has an Intel Atom processor in it.
> bits-327, released today, now detects Intel Atom processors, and no
> longer shows them as unknown.  We don't have any processor-specific
> tests for Atom yet, though we may add some in the future.  All of
> the general tests should work, however.

The new release detects the Atom processor okay, but in bits-327 the
system resets during the SMI frequency/latency test, whereas it didn't
with bits-316.

> Can you provide more details on which MSRs show up as inconsistent
> on your system?  They might represent actual bugs in your BIOS, but
> we might just need to add a few more entries to the Atom
> CPU-specific MSR blacklist as expected inconsistencies.

Here's what it says:

  (MSR 0x39 consistent): FAIL
  (MSR 0x199 consistent): FAIL
  (MSR 0x1a0 consistent): FAIL
  Summary: 7917 passed, 3 failed

I have 3 systems with this same processor, and they all report the same
thing here. 

Seth

Re: Performance/resume issues on Toshiba NB305

[Burt Triplett]

On 02/26/2011 07:00 AM, Seth Forshee wrote:
> On Fri, Feb 25, 2011 at 09:58:25PM -0800, Burt Triplett wrote:
>> Looks like the Toshiba NB305 has an Intel Atom processor in it.
>> bits-327, released today, now detects Intel Atom processors, and no
>> longer shows them as unknown.  We don't have any processor-specific
>> tests for Atom yet, though we may add some in the future.  All of
>> the general tests should work, however.
>
> The new release detects the Atom processor okay, but in bits-327 the
> system resets during the SMI frequency/latency test, whereas it didn't
> with bits-316.

The SMI Frequency and Latency test should be fixed in bits-329.

>> Can you provide more details on which MSRs show up as inconsistent
>> on your system?  They might represent actual bugs in your BIOS, but
>> we might just need to add a few more entries to the Atom
>> CPU-specific MSR blacklist as expected inconsistencies.
>
> Here's what it says:
>
>    (MSR 0x39 consistent): FAIL
>    (MSR 0x199 consistent): FAIL
>    (MSR 0x1a0 consistent): FAIL
>    Summary: 7917 passed, 3 failed
>
> I have 3 systems with this same processor, and they all report the same
> thing here.

To get more information on how the MSRs differ between CPUs, you can 
turn on verbose mode.  Go to the GRUB command line by hitting 'c', and 
enter "test_options -v 2".  Then hit Esc to go back to the menu, and 
re-run the MSR consistency check.  It will now show you the value of the 
MSR on each CPU.

MSR 0x1a0 is IA32_MISC_ENABLE, and generally any inconsistency in that 
MSR represents a BIOS bug: it didn't enable features identically on all 
CPUs.  Once you see which bits differ, you can look at the Intel 
Software Developer's Manual (specifically volume 3B) to find out what 
those bits mean and which specific features the BIOS inconsistently enabled.

MSR 0x199 is IA32_PERF_CTL.  A difference in that MSR usually means the 
BIOS left CPUs in different P-states when booting, which it really 
shouldn't do.  Again, you can decode the result with the Intel SDM.

Looking into MSR 0x39.

- Burt Triplett

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Sun, Feb 27, 2011 at 12:17:29PM -0800, Burt Triplett wrote:
> >The new release detects the Atom processor okay, but in bits-327 the
> >system resets during the SMI frequency/latency test, whereas it didn't
> >with bits-316.
> 
> The SMI Frequency and Latency test should be fixed in bits-329.

Yes, it's fixed now, thanks.

> To get more information on how the MSRs differ between CPUs, you can
> turn on verbose mode.  Go to the GRUB command line by hitting 'c',
> and enter "test_options -v 2".  Then hit Esc to go back to the menu,
> and re-run the MSR consistency check.  It will now show you the
> value of the MSR on each CPU.
> 
> MSR 0x1a0 is IA32_MISC_ENABLE, and generally any inconsistency in
> that MSR represents a BIOS bug: it didn't enable features
> identically on all CPUs.  Once you see which bits differ, you can
> look at the Intel Software Developer's Manual (specifically volume
> 3B) to find out what those bits mean and which specific features the
> BIOS inconsistently enabled.

 (MSR 0x1a0 consistent) FAIL (2 different values)
 1 CPUs 0x0000000364950089: 0x0
 1 CPUs 0x0000000364950081: 0x1

 Bit 3: Automatic Thermal Control Circuit Enable

I'm not sure of the impact of this, since there's really only one core
in the CPU. One of the others differs has bit 3 inconsistent as well. On
the third bit 3 is the same but bit 0 (fast-strings enable) is
inconsistent.

The three machines have further differences in this MSR, but all in bits
documented as being reserved.

> MSR 0x199 is IA32_PERF_CTL.  A difference in that MSR usually means
> the BIOS left CPUs in different P-states when booting, which it
> really shouldn't do.  Again, you can decode the result with the
> Intel SDM.

 (MSR 0x199 consistent): FAIL (2 different values)
 1 CPUs 0x0000000000000a1f: 0x0
 1 CPUs 0x0000000000000613: 0x1

So yes, the cores are in different P-states. The results from the other
machines are similar, but the values for CPU1 are slightly different for
each machine. CPU1 has the same value on all three machines.

> Looking into MSR 0x39.

Yeah, this one isn't in the SDM at all. Here are the values I got:

 (MSR 0x39 consistent): FAIL (2 different values)
 1 CPUs 0x0000000000000000: 0x0
 1 CPUs 0x0000000000000001: 0x1

Values are identical for all three machines.

I'm not really getting the idea that any of these is contributing to the
problems I'm seeing with this machine though.

Thanks,
Seth

Re: Performance/resume issues on Toshiba NB305

[Seth Forshee]

On Fri, Feb 25, 2011 at 10:47:16PM +0100, Thomas Gleixner wrote:
> On Fri, 25 Feb 2011, Seth Forshee wrote:
> > On Fri, Feb 25, 2011 at 09:37:39PM +0100, Thomas Gleixner wrote:
> > > On Fri, 25 Feb 2011, Seth Forshee wrote:
> > > That seems to be related to low power states. When the machine goes
> > > idle we switch into lower power states and that requires to use the
> > > hpet instead of the local apic timer as that one stops.
> > > 
> > > You could verify that theory by booting with processor.max_cstate=1
> > 
> > This fixes the performance in combination with intel_idle.max_cstate=0.
> > Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.
> 
> That was expected :)
> 
> > Is the answer to quirk the machine to avoid deep C-states, or is there
> > some better way I can fix this?
> 
> Let's wait for the intel and acpi folks. It would be interesting what
> the new intel toy says to your BIOS.

Since the discussion on this issue died without really getting anywhere,
I went ahead and threw together the patch below to disable anything
deeper than C1 for this machine. I hope a better solution can be found,
but if not would something like this be an acceptable workaround?

As for the hangs during resume, unless someone has a better suggestion I
guess I'll start looking into forcing the HPET to remain in periodic
mode throughout suspend.

Thanks,
Seth

---

diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c
index 1fa091e..7d88540 100644
--- a/drivers/idle/intel_idle.c
+++ b/drivers/idle/intel_idle.c
@@ -61,6 +61,7 @@
 #include <linux/sched.h>
 #include <linux/notifier.h>
 #include <linux/cpu.h>
+#include <linux/dmi.h>
 #include <asm/mwait.h>
 
 #define INTEL_IDLE_VERSION "0.4"
@@ -441,6 +442,32 @@ static int intel_idle_cpuidle_devices_init(void)
 }
 
 
+/*
+ * dmi_disable_cstates()
+ * Disable states beyond C1 for broken machines
+ */
+static int __init dmi_disable_cstates(const struct dmi_system_id *d)
+{
+	if (max_cstate == MWAIT_MAX_NUM_CSTATES - 1) {
+		pr_notice(PREFIX "%s detected, disabling C-states past C1\n",
+			  d->ident);
+		max_cstate = 1;
+	}
+}
+
+/* List of systems with known idle problems */
+static struct dmi_system_id __initdata bad_cstate_dmi_table[] = {
+	{
+		.callback	= dmi_disable_cstates,
+		.ident		= "Toshiba NB305",
+		.matches	= {
+			DMI_MATCH(DMI_BOARD_VENDOR, "TOSHIBA"),
+			DMI_MATCH(DMI_BOARD_NAME, "NPVAA"),
+		},
+	},
+	{}
+};
+
 static int __init intel_idle_init(void)
 {
 	int retval;
@@ -449,6 +476,9 @@ static int __init intel_idle_init(void)
 	if (boot_option_idle_override != IDLE_NO_OVERRIDE)
 		return -ENODEV;
 
+	/* Check for known-bad hardware */
+	dmi_check_system(bad_cstate_dmi_table);
+
 	retval = intel_idle_probe();
 	if (retval)
 		return retval;
-- 
1.7.4.1

Re: Performance/resume issues on Toshiba NB305

[Thomas Gleixner]

On Tue, 1 Mar 2011, Seth Forshee wrote:
> On Fri, Feb 25, 2011 at 10:47:16PM +0100, Thomas Gleixner wrote:
> > On Fri, 25 Feb 2011, Seth Forshee wrote:
> > > On Fri, Feb 25, 2011 at 09:37:39PM +0100, Thomas Gleixner wrote:
> > > > On Fri, 25 Feb 2011, Seth Forshee wrote:
> > > > That seems to be related to low power states. When the machine goes
> > > > idle we switch into lower power states and that requires to use the
> > > > hpet instead of the local apic timer as that one stops.
> > > > 
> > > > You could verify that theory by booting with processor.max_cstate=1
> > > 
> > > This fixes the performance in combination with intel_idle.max_cstate=0.
> > > Alternately, intel_idle.max_cstate=1 works. But the resume still hangs.
> > 
> > That was expected :)
> > 
> > > Is the answer to quirk the machine to avoid deep C-states, or is there
> > > some better way I can fix this?
> > 
> > Let's wait for the intel and acpi folks. It would be interesting what
> > the new intel toy says to your BIOS.
> 
> Since the discussion on this issue died without really getting anywhere,
> I went ahead and threw together the patch below to disable anything
> deeper than C1 for this machine. I hope a better solution can be found,
> but if not would something like this be an acceptable workaround?
> 
> As for the hangs during resume, unless someone has a better suggestion I
> guess I'll start looking into forcing the HPET to remain in periodic
> mode throughout suspend.

The problem with such DMI quirks is that it's hard to get rid of them
when a fixed BIOS version comes out while simply adding nohpet to the
kernel command line, is a non permanent, but sensible workaround.

Thanks,

	tglx

Re: Performance/resume issues on Toshiba NB305

[Burt Triplett]

On 02/28/2011 07:10 AM, Seth Forshee wrote:
> On Sun, Feb 27, 2011 at 12:17:29PM -0800, Burt Triplett wrote:
>>> The new release detects the Atom processor okay, but in bits-327 the
>>> system resets during the SMI frequency/latency test, whereas it didn't
>>> with bits-316.
>>
>> The SMI Frequency and Latency test should be fixed in bits-329.
>
> Yes, it's fixed now, thanks.

Glad to hear it.

>> To get more information on how the MSRs differ between CPUs, you can
>> turn on verbose mode.  Go to the GRUB command line by hitting 'c',
>> and enter "test_options -v 2".  Then hit Esc to go back to the menu,
>> and re-run the MSR consistency check.  It will now show you the
>> value of the MSR on each CPU.
>>
>> MSR 0x1a0 is IA32_MISC_ENABLE, and generally any inconsistency in
>> that MSR represents a BIOS bug: it didn't enable features
>> identically on all CPUs.  Once you see which bits differ, you can
>> look at the Intel Software Developer's Manual (specifically volume
>> 3B) to find out what those bits mean and which specific features the
>> BIOS inconsistently enabled.
>
>   (MSR 0x1a0 consistent) FAIL (2 different values)
>   1 CPUs 0x0000000364950089: 0x0
>   1 CPUs 0x0000000364950081: 0x1
>
>   Bit 3: Automatic Thermal Control Circuit Enable
>
> I'm not sure of the impact of this, since there's really only one core
> in the CPU. One of the others differs has bit 3 inconsistent as well. On
> the third bit 3 is the same but bit 0 (fast-strings enable) is
> inconsistent.

0x1a0 bit 3 definitely shouldn't differ between CPUs; that's a BIOS bug.

Fast-strings should *always* be turned on, and should always be 
consistent across processors.  That's also a BIOS bug, which will cause 
suboptimal performance of the string instructions on any CPU with that 
bit disabled.

>> MSR 0x199 is IA32_PERF_CTL.  A difference in that MSR usually means
>> the BIOS left CPUs in different P-states when booting, which it
>> really shouldn't do.  Again, you can decode the result with the
>> Intel SDM.
>
>   (MSR 0x199 consistent): FAIL (2 different values)
>   1 CPUs 0x0000000000000a1f: 0x0
>   1 CPUs 0x0000000000000613: 0x1
>
> So yes, the cores are in different P-states. The results from the other
> machines are similar, but the values for CPU1 are slightly different for
> each machine. CPU1 has the same value on all three machines.

Intel recommends that BIOSes set all CPUs to the same P-state before 
booting (partly for the case where the OS never touches P-states 
itself).  So, that's a BIOS bug, though it'll go away the moment Linux 
starts using P-states itself.

>> Looking into MSR 0x39.
>
> Yeah, this one isn't in the SDM at all. Here are the values I got:
>
>   (MSR 0x39 consistent): FAIL (2 different values)
>   1 CPUs 0x0000000000000000: 0x0
>   1 CPUs 0x0000000000000001: 0x1
>
> Values are identical for all three machines.

After investigating this MSR, I've blacklisted it from the MSR 
consistency check as of bits-332.  This isn't a BIOS bug, and the 
difference is expected.

> I'm not really getting the idea that any of these is contributing to the
> problems I'm seeing with this machine though.

No, none of the issues BITS has detected on your system should affect 
the original problem you reported.  Thanks for providing the additional 
information, though.

- Burt Triplett