Re: Clock has stopped (time/date looping over 5 seconds), things are broken - what to check to debug?

[Roger Heflin <rogerheflin@gmail.com>]

Joel K. Greene wrote:
> Hi Roger,
> 
> Does this sound familiar:
> 
> http://lkml.org/lkml/2008/3/14/178

That sounds like it matches what I have.

> 
> 
> We've been chasing this for quite a while. Our PIC gets in a bad state
> where it thinks the CPU is in the ISR, and so won't give another int. We
> haven't much of an idea of how we get in that state other than that
> HZ=1000 makes it happen faster and HZ=100 causes it less often.

I do have HZ=1000 set, Pavel mentions setting it the =4000 to make it happen 
faster, I will try that, I am rebuilding 2.4.24.4 with =4000 in the .config 
file, and will verify after it is up that 4000 is running on it.

My machine does have a fair amount of cpu usage (transcoding video), and has a 
fair amount of interrupt handling (5 disks, and 3 TV recording cards).

> 
> I think that if you look at jiffies you will see it is not incrementing.
> The 4 second loop seems to be in the conversion from jiffies to wall
> time.

I did check the counter in /proc/timer_list under (now at) and it was looping too.

> 
> 
> It _appears_ that there is a race in the kernel that can be triggered by
> any number of hardware issues. There's another thread by Gregory Stark
> with the same symptoms - he thinks his was fixed by replacing a bad
> DIMM.

I don't think I have bad HW, I will run a test job for a few hours that checks 
its results and make sure that the proper answers are coming back, and it is not 
crashing.

I do have a couple of disks (on a SIL controller) that every so often appear to 
give funny errors, but recover and continue on.

> 
> Note that we first saw this on 2.6.16, and Gregory found it on 2.6.5.
> We've seen systems run for a couple of months before seeing this, so
> it's a bear to debug.
> 
> How often is this happening for you? How repeatable?

14-30 days, I don't know if it always happens or not, I don't have exact enough 
data, but I don't think the machine has made it past 30 days in the last 6 
months, if I go back far enough though, I believe it was stable, before I added 
a couple of TV recording cards (PVR150, HD5500), and a disk controller (SIL) to 
it.

> 
> What hardware are you running on?

AMD-754 Sempron64 processor.

ASUS K8V-SE Deluxe MB (VT8385/VT8387 Chipset), so very different HW that the 
Serverworks-P3 that you have.