[linux-lvm] Daemon or Thread with 5 minute interval

[linux-lvm] Daemon or Thread with 5 minute interval

[Howard Rifkin]

I have been working on a project that analyzes "noise" in Linux clusters 
running large parallel jobs. Where noise is caused by daemons or kernel 
threads periodically waking up and slowing down a process of a parallel 
job enough so that is misses a rendezvous with the other processes and 
so slows down the entire job. Using an internal tool I have been able to 
track down several daemons that have been a large source of noise but, 
after turning them all off I am left with a large noise spike that 
occurs every 5 minutes. I have tried turning off most of the most common 
daemons such as cron, ntp, syslog, ect. but, I am still left with this 
spike. Not being a kernel expert myself I have not been able to 
experiment with kernel threads. So, my question is does anyone know of a 
kernel thread or daemon in the 2.4 kernel that wakes up every 5 minutes?

Thanks

Howard

Re: [linux-lvm] Daemon or Thread with 5 minute interval

[johnpaul]

Hi

Can you find the new process's name by getting the ps ax command  or Stop
the crond and see what happens

Regards
John

----- Original Message ----- 
From: "Howard Rifkin" <howard.rifkin@hp.com>
To: <linux-lvm@redhat.com>
Sent: Thursday, May 26, 2005 5:20 PM
Subject: [linux-lvm] Daemon or Thread with 5 minute interval


> I have been working on a project that analyzes "noise" in Linux clusters
> running large parallel jobs. Where noise is caused by daemons or kernel
> threads periodically waking up and slowing down a process of a parallel
> job enough so that is misses a rendezvous with the other processes and
> so slows down the entire job. Using an internal tool I have been able to
> track down several daemons that have been a large source of noise but,
> after turning them all off I am left with a large noise spike that
> occurs every 5 minutes. I have tried turning off most of the most common
> daemons such as cron, ntp, syslog, ect. but, I am still left with this
> spike. Not being a kernel expert myself I have not been able to
> experiment with kernel threads. So, my question is does anyone know of a
> kernel thread or daemon in the 2.4 kernel that wakes up every 5 minutes?
>
> Thanks
>
> Howard
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[linux-lvm] Re: Daemon or Thread with 5 minute interval

[Bill Mair]

Howard Rifkin wrote:
>  So, my question is does anyone know of a
> kernel thread or daemon in the 2.4 kernel that wakes up every 5 minutes?

[kupdated] can cause massive CPU load if your kernel does not have the IDE 
chipset support for your system. I think it flushed any data in the HD
buffers to disk and if the chipset support isn't there, then the main CPU has
to do this using PIO.

--
Bill

Re: [linux-lvm] Re: Daemon or Thread with 5 minute interval

[Howard Rifkin]

We do have the right kernel support for our didsks. But, you do not need 
a lot of CPU activity to generate noise. Just enough to disrupt the 
cashe. Does kupdated run every 5 minutes? If so it could still be the 
culprit.

Thanks

Howard


Bill Mair wrote:
> Howard Rifkin wrote:
> 
>>  So, my question is does anyone know of a
>> kernel thread or daemon in the 2.4 kernel that wakes up every 5 minutes?
> 
> 
> [kupdated] can cause massive CPU load if your kernel does not have the 
> IDE chipset support for your system. I think it flushed any data in the HD
> buffers to disk and if the chipset support isn't there, then the main 
> CPU has
> to do this using PIO.
> 
> -- 
> Bill
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[linux-lvm] Re: Daemon or Thread with 5 minute interval

[Bill Mair]

Howard Rifkin wrote:
> 
> We do have the right kernel support for our didsks. But, you do not need 
> a lot of CPU activity to generate noise. Just enough to disrupt the 
> cashe. Does kupdated run every 5 minutes? If so it could still be the 
> culprit.

kupdated - journal update
kswapd - swapper
bdflush - flush IO

All/any of them could cause problems fir what you are doing, but I don't think
LVM is your problem.

--
Bill

[linux-lvm] Re: Daemon or Thread with 5 minute interval

[bill.mair]

Howard Rifkin wrote:

 >> So, my question is does anyone know of a
 >> kernel thread or daemon in the 2.4 kernel that wakes up every 5 minutes?
 >
 >[kupdated] can cause massive CPU load if your kernel does not have the IDE 
 >chipset support for your system. I think it flushed any data in the HD
 >buffers to disk and if the chipset support isn't there, then the main CPU has
 >to do this using PIO.
 >
 >--
 >Bill


** In addition to my post I might add that kswapd and bdflushd also deal with
IDE drives.

**  Correcting my initial reply, kupdated deals with "journal" flushing to the
HD, which may also be relevant on your cluster.

--
Bill