smp cputime issues

smp cputime issues

[Steinar Hauan]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1196 bytes --]

hello,

  we are encountering some weird timing behaviour on our linux cluster.

  specifically: when running 2 copies of selected programs on a
  dual-cpu system, the cputime reported for each process is up to 25%
  higher than when the processes are run on their own. however, if running
  two different jobs on the same machine, both complete with a cputime
  equal to when run individually. sample timing output attached.

  profiling confirms that everything slows down approximately to scale.
  the results reproduce on a range of different machines (see below).

additional specifications:
  - kernel version 2.4.16 (with apic enabled)
  - chipsets: apollo pro 133, apollo pro 266,
              intel i860, serverworks LE
  - all jobs requires less than 1/10 of physical memory
  - no significant disk i/o takes place
  - timing with dtime(), /usr/bin/time and shell built-in time
  - this behavior is NOT seen for all applications. the worst
    "offender" spends most of its time doing linear algebra.

  ideas or info-pointers appreciated. more specs available on request.

regards,
--
  Steinar Hauan, dept of ChemE  --  hauan@cmu.edu
  Carnegie Mellon University, Pittsburgh PA, USA

[-- Attachment #2: Type: TEXT/PLAIN, Size: 936 bytes --]

output from running a single image copy

[reported by dtime()]

CPU seconds spent in IPOPT and function evaluations =     131.9999982

[reported by /usr/bin/time -v ]

	Command being timed: "./ipopt robot_2000.nl"
	User time (seconds): 134.01
	System time (seconds): 0.36
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:14.42
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 0
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 293
	Minor (reclaiming a frame) page faults: 23352
	Voluntary context switches: 0
	Involuntary context switches: 0
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

[-- Attachment #3: Type: TEXT/PLAIN, Size: 940 bytes --]

output from running two images simultaneously

[reported by dtime()]

CPU seconds spent in IPOPT and function evaluations =     157.7000024

[reported by /usr/bin/time -v ]
	Command being timed: "./ipopt robot_2000.nl"
	User time (seconds): 159.81
	System time (seconds): 0.50
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:40.41
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 0
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 293
	Minor (reclaiming a frame) page faults: 23352
	Voluntary context switches: 0
	Involuntary context switches: 0
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

RE: smp cputime issues

[M. Edward Borasky]

The obvious question is: how do the printed *elapsed* (wall clock) times
compare with a stopwatch timing of the same run??

--
M. Edward Borasky

znmeb@borasky-research.net
http://www.borasky-research.net

RE: smp cputime issues

[Steinar Hauan]

On Tue, 1 Jan 2002, M. Edward Borasky wrote:
> The obvious question is: how do the printed *elapsed* (wall clock) times
> compare with a stopwatch timing of the same run??

sorry,

  should have included that all timings are consistent.
  (usr/sys vs reported wall clock time vs external stop watch time)

  for reference: the effect arises for a several different memory types
  (pc133, pc133 ecc, pc133 reg ecc, pc2100) and the impact is similar.
  thus if it was only a memory bandwidth issue, i would expect
  the results to depend more on the memory/chipset in question.

regards,
--
  Steinar Hauan, dept of ChemE  --  hauan@cmu.edu
  Carnegie Mellon University, Pittsburgh PA, USA

Re: smp cputime issues (patch request ?)

[J.A. Magallon]

On 20020102 Steinar Hauan wrote:
>hello,
>
>  we are encountering some weird timing behaviour on our linux cluster.
>
>  specifically: when running 2 copies of selected programs on a
>  dual-cpu system, the cputime reported for each process is up to 25%
>  higher than when the processes are run on their own. however, if running
>  two different jobs on the same machine, both complete with a cputime
>  equal to when run individually. sample timing output attached.
>

Cache pollution problems ? 

As I understand, your job does not use too much memory, does no IO,
just linear algebra (ie, matrix-times-vector or vector-plus-vector
operations). That implies sequential access to matrix rows and vectors.

I will try to guess...

Problem with linux scheduler is that processes are bounced from one CPU
to the other, they are not tied to one, nor try to stay in the one they
start, even if there is no need for the cpu to do any other job.
On an UP box, the cache is useful to speed up your matrix-vector ops.
One process on a 2-way box, just bounces from one cpu to the other,
and both caches are filled with the same data. Two processes on two
cpus, and everytime they 'swap' between cpus they trash the previous
cache for the other job, so when it returs it has no data cached.

Solutions:
- cpu affinity patch: manually tie processes to cpus
- new scheduler: a patch for the scheduler that tries to
  keep processes on the cpu they start was talked about on the list.

I would prefer the second option. I think it is named something like
'multiqueue scheduler', and its 'father' could be (AFAIR) Davide Libezni.
Look for that on the list archives. Problem: I think the patch only
exists for 2.5.

Request: a version for 2.4.17+ ?? (plz)

Disclaimer: of course, all the previous discussion can be crap.

Good luck. I am also interested in this problem.

-- 
J.A. Magallon                           #  Let the source be with you...        
mailto:jamagallon@able.es
Mandrake Linux release 8.2 (Cooker) for i586
Linux werewolf 2.4.18-pre1-beo #3 SMP Thu Dec 27 10:15:27 CET 2001 i686

Re: smp cputime issues (patch request ?)

[Davide Libenzi]

On Thu, 3 Jan 2002, J.A. Magallon wrote:

>
> On 20020102 Steinar Hauan wrote:
> >hello,
> >
> >  we are encountering some weird timing behaviour on our linux cluster.
> >
> >  specifically: when running 2 copies of selected programs on a
> >  dual-cpu system, the cputime reported for each process is up to 25%
> >  higher than when the processes are run on their own. however, if running
> >  two different jobs on the same machine, both complete with a cputime
> >  equal to when run individually. sample timing output attached.
> >
>
> Cache pollution problems ?
>
> As I understand, your job does not use too much memory, does no IO,
> just linear algebra (ie, matrix-times-vector or vector-plus-vector
> operations). That implies sequential access to matrix rows and vectors.
>
> I will try to guess...
>
> Problem with linux scheduler is that processes are bounced from one CPU
> to the other, they are not tied to one, nor try to stay in the one they
> start, even if there is no need for the cpu to do any other job.
> On an UP box, the cache is useful to speed up your matrix-vector ops.
> One process on a 2-way box, just bounces from one cpu to the other,
> and both caches are filled with the same data. Two processes on two
> cpus, and everytime they 'swap' between cpus they trash the previous
> cache for the other job, so when it returs it has no data cached.
>
> Solutions:
> - cpu affinity patch: manually tie processes to cpus
> - new scheduler: a patch for the scheduler that tries to
>   keep processes on the cpu they start was talked about on the list.
>
> I would prefer the second option. I think it is named something like
> 'multiqueue scheduler', and its 'father' could be (AFAIR) Davide Libezni.
> Look for that on the list archives. Problem: I think the patch only
> exists for 2.5.

The patch is here :

http://www.xmailserver.org/linux-patches/xsched-2.5.2-pre4-0.58.diff

I did not read the whole thread but if your two tasks are strictly cpu
bound and you've two cpus, you should not have problems even with the
current scheduler.




- Davide

Re: smp cputime issues (patch request ?)

[Steinar Hauan]

On Thu, 3 Jan 2002, J.A. Magallon wrote:
> Cache pollution problems ?
>
> As I understand, your job does not use too much memory, does no IO,
> just linear algebra (ie, matrix-times-vector or vector-plus-vector
> operations). That implies sequential access to matrix rows and vectors.

very correct.

> Problem with linux scheduler is that processes are bounced from one CPU
> to the other, they are not tied to one, nor try to stay in the one they
> start, even if there is no need for the cpu to do any other job.

one of the tips received was to set the penalty for cpu switch, i.e. set

  linux/include/asm/smp.h:#define PROC_CHANGE_PENALTY   15

to a much higher value (50). this had no effect on the results.

> On an UP box, the cache is useful to speed up your matrix-vector ops.
> One process on a 2-way box, just bounces from one cpu to the other,
> and both caches are filled with the same data. Two processes on two
> cpus, and everytime they 'swap' between cpus they trash the previous
> cache for the other job, so when it returs it has no data cached.

this would be an issue, agreed, but cache invalidation by cpu bounces
should also affect one-cpu jobs? thus is does not explain why this
effect should be (much) worse with 2 jobs.

regards,
--
  Steinar Hauan, dept of ChemE  --  hauan@cmu.edu
  Carnegie Mellon University, Pittsburgh PA, USA

Re: smp cputime issues (patch request ?)

[John Alvord]

On Sat, 5 Jan 2002 00:21:44 -0500 (EST), Steinar Hauan <hauan@cmu.edu>
wrote:

>On Thu, 3 Jan 2002, J.A. Magallon wrote:
>> Cache pollution problems ?
>>
>> As I understand, your job does not use too much memory, does no IO,
>> just linear algebra (ie, matrix-times-vector or vector-plus-vector
>> operations). That implies sequential access to matrix rows and vectors.
>
>very correct.
>
>> Problem with linux scheduler is that processes are bounced from one CPU
>> to the other, they are not tied to one, nor try to stay in the one they
>> start, even if there is no need for the cpu to do any other job.
>
>one of the tips received was to set the penalty for cpu switch, i.e. set
>
>  linux/include/asm/smp.h:#define PROC_CHANGE_PENALTY   15
>
>to a much higher value (50). this had no effect on the results.
>
>> On an UP box, the cache is useful to speed up your matrix-vector ops.
>> One process on a 2-way box, just bounces from one cpu to the other,
>> and both caches are filled with the same data. Two processes on two
>> cpus, and everytime they 'swap' between cpus they trash the previous
>> cache for the other job, so when it returs it has no data cached.
>
>this would be an issue, agreed, but cache invalidation by cpu bounces
>should also affect one-cpu jobs? thus is does not explain why this
>effect should be (much) worse with 2 jobs.

One factor to consider is that to see it bounce, you need to be
running an observation process like top, or if it is a GUI display two
processes (application and X). Those observing processes will
continuosly bump aside the calcuation processes, causing a bouncing
effect.

john