Problem: CPU sleep when calling a function in another object for the first time.

Problem: CPU sleep when calling a function in another object for the first time.

[John Marconi]

Hello,

I have been seeing a problem in which one of my CPUs goes to sleep for
40ms in the middle of running a user-space program.  The problem occurs
when I have a function in object file X call another function in object
file Y.  If I add a gettimeofday right before the function call occurs,
and then also right at the beginning of the called function, I see a
40ms delay.  This only occurs on the first call to any function in file
Y.  Once a function in file Y is called, all other calls to functions in
this file are immediate.

I have added a kernel profiler to the timer interrupt of this CPU - and
find that during this 40ms period, the CPU is in r4k_wait - which looks
like cpu_idle is running.  It almost seems like file Y is being loaded
into memory to be used.

The program in question is using pthreads and each thread is on a
separate CPU.  Each CPU is dedicated to running a single thread and
nothing else.  The files X and Y are two object files linked into a
single ELF binary.  The kernel version is 2.6.14 and this is a MIPS
based processor.

Does anyone have any ideas why this delay might be occurring?

Thank you in advance,
John

Re: Problem: CPU sleep when calling a function in another object for the first time.

[Phillip Susi]

John Marconi wrote:
> I have added a kernel profiler to the timer interrupt of this CPU - and
> find that during this 40ms period, the CPU is in r4k_wait - which looks
> like cpu_idle is running.  It almost seems like file Y is being loaded
> into memory to be used.

That is exactly what is happening.  Pages in the code segment are only 
loaded on demand.

Re: Problem: CPU sleep when calling a function in another object for the first time.

[Robert Hancock]

John Marconi wrote:
> Hello,
> 
> I have been seeing a problem in which one of my CPUs goes to sleep for
> 40ms in the middle of running a user-space program.  The problem occurs
> when I have a function in object file X call another function in object
> file Y.  If I add a gettimeofday right before the function call occurs,
> and then also right at the beginning of the called function, I see a
> 40ms delay.  This only occurs on the first call to any function in file
> Y.  Once a function in file Y is called, all other calls to functions in
> this file are immediate.
> 
> I have added a kernel profiler to the timer interrupt of this CPU - and
> find that during this 40ms period, the CPU is in r4k_wait - which looks
> like cpu_idle is running.  It almost seems like file Y is being loaded
> into memory to be used.
> 
> The program in question is using pthreads and each thread is on a
> separate CPU.  Each CPU is dedicated to running a single thread and
> nothing else.  The files X and Y are two object files linked into a
> single ELF binary.  The kernel version is 2.6.14 and this is a MIPS
> based processor.
> 
> Does anyone have any ideas why this delay might be occurring?

Binaries get loaded into memory on demand, so it's quite possible that 
that part of the executable hasn't been loaded until something calls a 
function within it.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/