How is user space notified of CPU speed changes?

How is user space notified of CPU speed changes?

[Lee Revell]

This issue came up on the JACK (http://jackit.sf.net) mailing list. 
Google was not helpful so I ask here.

JACK needs to know the CPU speed, in order to calculate the DSP load
among other things.  It used to be a valid assumption that you could
calculate it on startup and it would not change.

Now with frequency scaling (apparently desktops do this as well as
laptops) this doesn't work anymore.  Is there a sane way for jackd to be
notified when the CPU speed changes?  Polling a file in /sys is not good
enough, the overhead is unacceptable and we need to know _now_.

Is this the kind of thing that would require the new kernel event
interface?

Lee

Re: How is user space notified of CPU speed changes?

[Alan Cox]

On Gwe, 2004-10-22 at 00:01, Lee Revell wrote:
> This issue came up on the JACK (http://jackit.sf.net) mailing list. 
> Google was not helpful so I ask here.
> 
> JACK needs to know the CPU speed, in order to calculate the DSP load
> among other things.  It used to be a valid assumption that you could
> calculate it on startup and it would not change.

No it did not. It has never been a safe assumption. Even my old PC110
does APM non-linux assisted shifts between 8 16 and 33Mhz. In addition
there are boxes with dual CPU's and different multipliers - dual 
300/450's were not uncommon.

And thats before we even mention such things at hyped-threading.

Re: How is user space notified of CPU speed changes?

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 97 bytes --]

Alan Cox wrote:
> And thats before we even mention such things at hyped-threading.
[sic]...

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Fri, 2004-10-22 at 07:23, Alan Cox wrote:
> On Gwe, 2004-10-22 at 00:01, Lee Revell wrote:
> > This issue came up on the JACK (http://jackit.sf.net) mailing list. 
> > Google was not helpful so I ask here.
> > 
> > JACK needs to know the CPU speed, in order to calculate the DSP load
> > among other things.  It used to be a valid assumption that you could
> > calculate it on startup and it would not change.
> 
> No it did not. It has never been a safe assumption. Even my old PC110
> does APM non-linux assisted shifts between 8 16 and 33Mhz. In addition
> there are boxes with dual CPU's and different multipliers - dual 
> 300/450's were not uncommon.
> 
> And thats before we even mention such things at hyped-threading.

Seems like you are implying that any userspace app that needs to know
the CPU speed is broken.  Is this correct?

Lee

Re: How is user space notified of CPU speed changes?

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Oct 22, 2004  14:05 -0400, Lee Revell wrote:
> On Fri, 2004-10-22 at 07:23, Alan Cox wrote:
> > No it did not. It has never been a safe assumption. Even my old PC110
> > does APM non-linux assisted shifts between 8 16 and 33Mhz. In addition
> > there are boxes with dual CPU's and different multipliers - dual 
> > 300/450's were not uncommon.
> > 
> > And thats before we even mention such things at hyped-threading.
> 
> Seems like you are implying that any userspace app that needs to know
> the CPU speed is broken.  Is this correct?

Sadly, if I boot my laptop on battery and then plug it in, all of my
MP3s run twice (or whatever) as fast (XMMS on 2.4), making them sound like
the Chipmunks on crack.  I have to reboot my system while plugged in
to listen to music.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: How is user space notified of CPU speed changes?

[Chris Friesen]

Lee Revell wrote:

> Seems like you are implying that any userspace app that needs to know
> the CPU speed is broken.  Is this correct?

No, we're saying that Intel's tsc implementation is broken.  <grin>

x86 really could use an on-die register that increments at 1GHz independent of 
clock speed and is synchronized across all CPUs in an SMP box.

Chris

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Fri, 2004-10-22 at 15:40, Chris Friesen wrote:
> Lee Revell wrote:
> 
> > Seems like you are implying that any userspace app that needs to know
> > the CPU speed is broken.  Is this correct?
> 
> No, we're saying that Intel's tsc implementation is broken.  <grin>
> 
> x86 really could use an on-die register that increments at 1GHz independent of 
> clock speed and is synchronized across all CPUs in an SMP box.

Well, it looks like we can just fix JACK to not use the TSC for timing. 
Seems like the current design is fundamentally incompatible with CPU
scaling.

Lee

Re: How is user space notified of CPU speed changes?

[Alan Cox]

On Gwe, 2004-10-22 at 20:40, Chris Friesen wrote:
> x86 really could use an on-die register that increments at 1GHz independent of 
> clock speed and is synchronized across all CPUs in an SMP box.

HPET sort of is this but at chipset level

Re: How is user space notified of CPU speed changes?

[Chris Friesen]

Alan Cox wrote:
> On Gwe, 2004-10-22 at 20:40, Chris Friesen wrote:
> 
>>x86 really could use an on-die register that increments at 1GHz independent of 
>>clock speed and is synchronized across all CPUs in an SMP box.

> HPET sort of is this but at chipset level

Right.  So you still have to go across the cpu bus to get it.

Chris

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Fri, 2004-10-22 at 13:40 -0600, Chris Friesen wrote:
> Lee Revell wrote:
> 
> > Seems like you are implying that any userspace app that needs to know
> > the CPU speed is broken.  Is this correct?
> 
> No, we're saying that Intel's tsc implementation is broken.  <grin>
> 
> x86 really could use an on-die register that increments at 1GHz independent of 
> clock speed and is synchronized across all CPUs in an SMP box.

Like this? (posted to jackit-devel):

On Fri, 2004-10-22 at 18:20 -0500, Jack O'Quin wrote: 

> On PowerPC, JACK uses the lower half of the 64-bit Timebase register,
> which is accessible from user mode.  This is better then the i386
> cycle counter, I believe.  See: `config/cpu/powerpc/cycles.h', the
> comment about only using it for SMP is bogus.  We use this for both
> MacOS X and Linux.

-- 
Lee Revell <rlrevell@joe-job.com>

Re: How is user space notified of CPU speed changes?

[Chris Friesen]

Lee Revell wrote:
> On Fri, 2004-10-22 at 13:40 -0600, Chris Friesen wrote:
>>x86 really could use an on-die register that increments at 1GHz independent of 
>>clock speed and is synchronized across all CPUs in an SMP box.
> 
> 
> Like this? (posted to jackit-devel):
> 
> On Fri, 2004-10-22 at 18:20 -0500, Jack O'Quin wrote: 
>>On PowerPC, JACK uses the lower half of the 64-bit Timebase register,
>>which is accessible from user mode.  This is better then the i386
>>cycle counter, I believe.

Yes, ppc tbr is nice.  It's actually lower resolution than the x86 one, but it 
might be better for smp and freq changes--not sure.

Chris

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Fri, 2004-10-22 at 12:23 +0100, Alan Cox wrote:
> On Gwe, 2004-10-22 at 00:01, Lee Revell wrote:
> > JACK needs to know the CPU speed, in order to calculate the DSP load
> > among other things.  It used to be a valid assumption that you could
> > calculate it on startup and it would not change.
> 
> No it did not. It has never been a safe assumption.

OK, thanks.  Still no answer to my original question though.

JACK makes extensive use of microsecond-level timers.  These must be
calibrated at startup, and recalibrated when the CPU speed changes.  How
does JACK register with the kernel to be notified when the CPU speed
changes?

Lee

Re: How is user space notified of CPU speed changes?

[Matthew Garrett]

Lee Revell <rlrevell@joe-job.com> wrote:
> On Fri, 2004-10-22 at 12:23 +0100, Alan Cox wrote:
>> No it did not. It has never been a safe assumption.
> 
> OK, thanks.  Still no answer to my original question though.
> 
> JACK makes extensive use of microsecond-level timers.  These must be
> calibrated at startup, and recalibrated when the CPU speed changes.  How
> does JACK register with the kernel to be notified when the CPU speed
> changes?

The kernel does not always know when the CPU speed changes. This makes
notification somewhat harder.

-- 
Matthew Garrett | mjg59-chiark.mail.linux-rutgers.kernel@srcf.ucam.org

Re: How is user space notified of CPU speed changes?

[Alan Cox]

On Sad, 2004-10-23 at 06:10, Lee Revell wrote:
> JACK makes extensive use of microsecond-level timers.  These must be
> calibrated at startup, and recalibrated when the CPU speed changes.  How
> does JACK register with the kernel to be notified when the CPU speed
> changes?

It did

- The kernel doesn't always know
- CPU speed is meaningless in hyper-threading since performance is not
x2 for two cores but instead varies
- It doesn't handle split CPU speed SMP - where CPU speeds vary
- God help you if virtualised

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Sat, 2004-10-23 at 22:19 +0100, Alan Cox wrote:
> On Sad, 2004-10-23 at 06:10, Lee Revell wrote:
> > JACK makes extensive use of microsecond-level timers.  These must be
> > calibrated at startup, and recalibrated when the CPU speed changes.  How
> > does JACK register with the kernel to be notified when the CPU speed
> > changes?
> 
> It did
> 
> - The kernel doesn't always know

OK, this one seems like the hard one.  Wouldn't this cause weird
behavior though?  For example Linux only calculates the delay loop once,
at boot time.  Does this render *delay() useless?

> - CPU speed is meaningless in hyper-threading since performance is not
> x2 for two cores but instead varies

Doesn't matter.  As long as they are both the same speed, JACK's
calculations will be correct.  We calculate the CPU speed at startup.
Then we just read the TSC and do the math.

> - It doesn't handle split CPU speed SMP - where CPU speeds vary

We don't have to support this.  If someone wants to do it they will have
to bind jackd to one CPU or the other.

> - God help you if virtualised

We don't have to support this.

Does anyone know how OSX/CoreAudio handles the situation?  Apparently
realtime apps work flawlessly on speed scaling laptops under OSX.

Lee

Re: How is user space notified of CPU speed changes?

[Jon Masters]

On Sat, 23 Oct 2004 18:42:13 -0400, Lee Revell <rlrevell@joe-job.com> wrote:

> Does anyone know how OSX/CoreAudio handles the situation?  Apparently
> realtime apps work flawlessly on speed scaling laptops under OSX.

The difference in implementation between the Intel TSC and PowerPC
TB[LU] has been mentioned previously in this thread.

Jon.

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Sun, 2004-10-24 at 00:17 +0100, Jon Masters wrote:
> On Sat, 23 Oct 2004 18:42:13 -0400, Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Does anyone know how OSX/CoreAudio handles the situation?  Apparently
> > realtime apps work flawlessly on speed scaling laptops under OSX.
> 
> The difference in implementation between the Intel TSC and PowerPC
> TB[LU] has been mentioned previously in this thread.

Well, it was mentioned that there is a difference...

http://www-106.ibm.com/developerworks/eserver/pdfs/archpub2.pdf

pg. 39, "Non-constant update frequency".

"Each time the update frequency changes, the system software is notified
of the change via an interrupt, or the change was instigated by the
system software itself.  At each such change, the system software must
compute the current time of day using the old update frequency, compute
a new value of ticks_per_sec for the new frequency, and save the time of
day, Time Base value, and the tick rate"

This was what I wanted to know: does Linux and/or the x86 have a similar
mechanism?  Looks like the answer is no...

Lee

Re: How is user space notified of CPU speed changes?

[Alan Cox]

On Sad, 2004-10-23 at 23:42, Lee Revell wrote:
> > - The kernel doesn't always know
> 
> OK, this one seems like the hard one.  Wouldn't this cause weird
> behavior though?  For example Linux only calculates the delay loop once,
> at boot time.  Does this render *delay() useless?

Such systems you do need to run with notsc - although 2.6 autodetects
this prints complaints and does the job itself.

> > - CPU speed is meaningless in hyper-threading since performance is not
> > x2 for two cores but instead varies
> 
> Doesn't matter.  As long as they are both the same speed, JACK's
> calculations will be correct.  We calculate the CPU speed at startup.
> Then we just read the TSC and do the math.

Are you trying to use tsc for delays or measure CPU speed. The original
question you asked was about CPU speed and the two are very different.

> > - It doesn't handle split CPU speed SMP - where CPU speeds vary
> 
> We don't have to support this.  If someone wants to do it they will have
> to bind jackd to one CPU or the other.

If you are trying to do tsc stuff remember tsc synchronization between
processors is not guaranteed and works on most boards today as a quirk
of design.

> Does anyone know how OSX/CoreAudio handles the situation?  Apparently
> realtime apps work flawlessly on speed scaling laptops under OSX.

Presumably they use the other timers that Apple designed into their
hardware as they control both ends. You've got a good 48Khz or so clock
in the audio device too so many games clock off the audio clock anyway.

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Sun, 2004-10-24 at 15:02 +0100, Alan Cox wrote:
> Are you trying to use tsc for delays or measure CPU speed. The original
> question you asked was about CPU speed and the two are very different.

No we are only using it as a cheap way to do microsecond level timing.
Are you saying we should just use gettimeofday() instead?
jack_get_microseconds() is called at least twice per period which can be
several thousand times per second.  Is the overhead of a system call
really low enough that this should work?  rdtsc is definitely cheap
enough.

Lee

Re: How is user space notified of CPU speed changes?

[Lee Revell]

On Sun, 2004-10-24 at 15:02 +0100, Alan Cox wrote:
> You've got a good 48Khz or so clock
> in the audio device too so many games clock off the audio clock anyway.

This is OK for a game or mplayer but this is not 100% reliable, we need
to know if we missed an interrupt or didn't get scheduled in time.  If
it were there would be no such thing as an xrun.  For serious audio work
we want an xrun to be a fatal error.

For 2.6 it looks like sched_clock, the HPET, and maybe the PM timer may
do what we want, if the syscall overhead is tolerable.

Lee 

Re: How is user space notified of CPU speed changes?

[Jon Masters]

On Sat, 23 Oct 2004 22:19:28 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> - It doesn't handle split CPU speed SMP - where CPU speeds vary

Alan,

Out of sheer interest, you said you had an example box which did this.
I've never actually seen a modern SMP setup with different cock
frequencies (even accepting it's possible) - can you give me a more
modern example? I'm sure they're out there, I've just missed it, and I
have to confess to not being aware that Linux supported this kind of
setup.

Cheers,

Jon.

Re: How is user space notified of CPU speed changes?

[Jon Masters]

On Sun, 24 Oct 2004 00:35:46 +0100, Jon Masters <jonmasters@gmail.com> wrote:

> Out of sheer interest, you said you had an example box which did this.
> I've never actually seen a modern SMP setup with different cock
> frequencies (even accepting it's possible) - can you give me a more
> modern example? I'm sure they're out there, I've just missed it, and I
> have to confess to not being aware that Linux supported this kind of
> setup.

Apologies for the unfortunate typo in the above. I'll refrain from
emphasing it :-)

Jon.

Re: How is user space notified of CPU speed changes?

[Alan Cox]

On Sul, 2004-10-24 at 00:35, Jon Masters wrote:
> Out of sheer interest, you said you had an example box which did this.
> I've never actually seen a modern SMP setup with different cock
> frequencies (even accepting it's possible) - can you give me a more
> modern example? I'm sure they're out there, I've just missed it, and I
> have to confess to not being aware that Linux supported this kind of
> setup.

The classic setups were dual PIII type systems - common FSB clock but
different multipliers for the processors. A lot of dual slot 1 boards
ended up with weird processor combinations.

Modern systems that don't tie the tsc include the big IBM 440 series
systems.

Alan

Re: How is user space notified of CPU speed changes?

[Joel Becker]

On Sun, Oct 24, 2004 at 12:35:46AM +0100, Jon Masters wrote:
> Out of sheer interest, you said you had an example box which did this.
> I've never actually seen a modern SMP setup with different cock
> frequencies (even accepting it's possible) - can you give me a more
> modern example? I'm sure they're out there, I've just missed it, and I
> have to confess to not being aware that Linux supported this kind of
> setup.

	I have a dual celeron with a 433 CPU and a 466 CPU in the slots.
It works, as long as you don't rely on TSC synchronicity.  Also, IBM
x440 boxes don't sync the TSCs between each group of 4 CPUs (that is,
each group of 4 is synced internally, but different from each other
group of 4).

Joel

-- 

"Vote early and vote often." 
        - Al Capone

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: How is user space notified of CPU speed changes?

[Robert Love]

On Sat, 2004-10-23 at 01:10 -0400, Lee Revell wrote:

> OK, thanks.  Still no answer to my original question though.
> 
> JACK makes extensive use of microsecond-level timers.  These must be
> calibrated at startup, and recalibrated when the CPU speed changes.  How
> does JACK register with the kernel to be notified when the CPU speed
> changes?

Ignoring all of these meta-issues like whether or not JACK actually
should be checking the CPU speed: Yes, I think that doing a kevent tied
to the processor object when the speed changes is an absolutely ideal
use of the kernel event layer.

HAL would use it, too.

	Robert Love

Re: How is user space notified of CPU speed changes?

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> It did
> 
> - The kernel doesn't always know
> - CPU speed is meaningless in hyper-threading since performance is not
> x2 for two cores but instead varies
> - It doesn't handle split CPU speed SMP - where CPU speeds vary
> - God help you if virtualised

Even without virtualization it doesn't make much sense to know the
exact CPU speed for an user process: it never knows how much CPU time
it will get (even with real time scheduling there could be even higher
priority processes)

The solution is to call gettimeofday frequently to resychronize.
It should be cheap enough that it isn't a big issue.

-Andi