fscked clock sources revisited

fscked clock sources revisited

[jamal]

Folks,

I have posted this before but got no good response and i havent had time
to chase it. 
While doing some batching tests with pktgen and then with a simple
client server app with udp, it does appear that the clock source used
matters. Here are some basic runs with plain vanilla 2.6.22-rc4 runs
with udp. There are five runs per clock source and each result is in
Mbps.

acpi_pm: 108, 110, 111,  91, 108
tsc:     143, 108, 161, 129, 108
jiffies: 132, 138, 132, 146, 150

jiffies produces better results than tsc which produces better results
than acpi_pm. 

Second issue: I cant consistently get two reboots to always boot with
the same clock source.

Any ideas whats going on here? I havent tried with latest tree.

PS:- These results are very easy to validate with iperf or netperf.

cheers,
jamal

Re: fscked clock sources revisited

[Arjan van de Ven]

On Mon, 2007-07-30 at 21:10 -0400, jamal wrote:
> Folks,
> 
> I have posted this before but got no good response and i havent had time
> to chase it. 
> While doing some batching tests with pktgen and then with a simple
> client server app with udp, it does appear that the clock source used
> matters. Here are some basic runs with plain vanilla 2.6.22-rc4 runs
> with udp. There are five runs per clock source and each result is in
> Mbps.

can you make sure hpet is enabled as well?

Re: fscked clock sources revisited

[jamal]

On Mon, 2007-30-07 at 18:10 -0700, Arjan van de Ven wrote:

> can you make sure hpet is enabled as well?

Will do next opportunity. Actually iirc, hpet is not even enabled in the
kernel - What are you expecting to see?

cheers,
jamal

Re: fscked clock sources revisited

[jamal]

On Mon, 2007-30-07 at 21:17 -0400, jamal wrote:

>  Actually iirc, hpet is not even enabled in the
> kernel -

Sorry, i lied - the config file is on my laptop - it is enabled.

cheers,
jamal

Re: fscked clock sources revisited

[Arjan van de Ven]

On Mon, 2007-07-30 at 21:23 -0400, jamal wrote:
> On Mon, 2007-30-07 at 21:17 -0400, jamal wrote:
> 
> >  Actually iirc, hpet is not even enabled in the
> > kernel -
> 
> Sorry, i lied - the config file is on my laptop - it is enabled.

is it also on in the bios? (and if not, can you grab the patches to
force enable it instead?)
(they're in the -hrt patchkit and iirc also in -mm)
-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: fscked clock sources revisited

[David Miller]

From: jamal <hadi@cyberus.ca>
Date: Mon, 30 Jul 2007 21:10:39 -0400

> acpi_pm: 108, 110, 111,  91, 108
> tsc:     143, 108, 161, 129, 108
> jiffies: 132, 138, 132, 146, 150
> 
> jiffies produces better results than tsc which produces better results
> than acpi_pm. 

Relatively speaking, the acpi_pm numbers are at least consistent and
about as much as so as jiffies.  Jiffies numbers are also possibly
better, at least in part, because of the decreased accuracy and errors
propagating.

tsc acts as expected, since every time your cpu changes power
management state (which it is going to do dynamically) the TSC
rates change and thus the accuracy goes into outer-space.

There really isn't much that can be done by any of this.  These issues
exist because of hardware limitations, nobody bothered to build
x86/x86_64 systems with a system wide TICK register that is both
impervious to cpu frequence scaling and also cheap to access.

So the above is what we basically have to live with :-)

Re: fscked clock sources revisited

[Arjan van de Ven]

On Mon, 2007-07-30 at 18:37 -0700, David Miller wrote:

> There really isn't much that can be done by any of this.  These issues
> exist because of hardware limitations, nobody bothered to build
> x86/x86_64 systems with a system wide TICK register that is both
> impervious to cpu frequence scaling and also cheap to access.

actually the tsc is clock speed invariant on pretty much all cpus these
days. The problem is that it stops in deeper idle states... (and that it
may be out of sync on an multi-socket system) not that it keeps changing
frequency ;(

(so it still sucks, just differently)

Re: fscked clock sources revisited

[jamal]

On Mon, 2007-30-07 at 18:37 -0700, David Miller wrote:

> Relatively speaking, the acpi_pm numbers are at least consistent and
> about as much as so as jiffies.  Jiffies numbers are also possibly
> better, at least in part, because of the decreased accuracy and errors
> propagating.

I am not sure jiffies gives innacurate results. It gives the best
performance on the dual xeon i tested on, all the time. Very consistent.
On its accuracy: I was able to validate it With pktgen generating
traffic and some external hardware tool capturing the interpacket gaps
etc.

> tsc acts as expected, since every time your cpu changes power
> management state (which it is going to do dynamically) the TSC
> rates change and thus the accuracy goes into outer-space.

Robert was saying he had even more bizare results with opteron given the
NUMA nature.

> There really isn't much that can be done by any of this.  These issues
> exist because of hardware limitations, nobody bothered to build
> x86/x86_64 systems with a system wide TICK register that is both
> impervious to cpu frequence scaling and also cheap to access.
> 
> So the above is what we basically have to live with :-)

That is a bummer. I am going to test with hpet when i get the chance
and perhaps turn off all the other sources if nothing good comes out; i
need my numbers ;->

cheers,
jamal

Re: fscked clock sources revisited

[jamal]

On Mon, 2007-30-07 at 22:14 -0400, jamal wrote:

> I am going to test with hpet when i get the chance

Couldnt figure how to turn on/off hpet, so didnt test.

> and perhaps turn off all the other sources if nothing good comes out; i
> need my numbers ;->

Here are some numbers that make the mystery even more interesting. This
is with kernel 2.6.22-rc4. Repeating with kernel 2.6.23-rc1 didnt show
anything different. I went back to test on 2.6.22-rc4 because it is the 
base for my batching patches - and since those drove me to this test, i
wanted something that reduces variables when comparing with batching.

I picked udp for this test because i can select different packet sizes.
i used iperf. The sender is a dual opteron with tg3. The receiver is a
dual xeon.

The default HZ is 250. Each packet size was run 3 times with different
clock sources. The experiment made sure that the receiver wasnt a
bottleneck (increased socket buffer sizes etc)

Packet | jiffies (1/250) |      tsc      |    acpi_pm
-------------------------|---------------|---------------
  64   |  141, 145, 142  | 131, 136, 130 | 103, 104, 110
 128   |  256, 256, 256  | 274, 260, 269 | 216, 206, 220
 512   |  513, 513, 513  | 886, 886, 886 | 828, 814, 806
1280   |  684, 684, 684  | 951, 951, 951 | 951, 951, 951

So i was wrong to declare jiffies as being good. The last batch of
experiments were based on only 64 byte UDP. Clearly as packet size goes
up, the results are worse with jiffies.
At this point, i decided to recompile the kernel with HZ=1000 and the
observations show that the jiffies results are improved.

Packet | jiffies (1/250) |      tsc      |    acpi_pm
-------------------------|---------------|---------------
  64   |  145, 135, 135  | 131, 137, 139 | 110, 110, 108
 128   |  257, 257, 257  | 270, 264, 250 | 218, 216, 217
 512   |  819, 776, 819  | 886, 886, 886 | 841, 824, 846
1280   |  855, 855, 855  | 951, 950, 951 | 951, 951, 951

Still not as good as the other two at large packet sizes.
For this machine: The ideal clock source would be jiffies with
HZ=1000 upto about 100 bytes then change to tsc. Of course i could pick
tsc but people have dissed it so far - i probably didnt hit the
condition where it goes into deep slumber.

Any insights? This makes it hard to quantify batching experimental
improvements as i feel it could be architecture or worse machine
dependent.

cheers,
jamal