rtc: lost some interrupts at 256Hz

rtc: lost some interrupts at 256Hz

[Michael Monnerie]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

When you google for such messages, you can find a lot of people asking, 
but nobody seems to have an answer. That's why I ask this list, where 
the Godfathers Of Linux reside, and maybe someone hears my prayer and 
could explain us sheep what you should do in such a case. Increase the 
HZ from 250 to 1000, or decrease to 100? Or maybe setting the 
preemption model from server to voluntary or preemptible? Or is that 
whining to be ignored, and if yes, what is this message for at all?

Please give us wisdom, and we will spread your word. Amen.

Answers please per PM, I'm not on this list.

mfg zmi *or could you ask in a nicer way?*
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660/4156531                          .network.your.ideas.
// PGP Key:   "lynx -source http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2  9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net                 Key-ID: 0x55CBA4EE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: rtc: lost some interrupts at 256Hz

[linux-os (Dick Johnson)]

On Thu, 20 Apr 2006, Michael Monnerie wrote:

> When you google for such messages, you can find a lot of people asking,
> but nobody seems to have an answer. That's why I ask this list, where
> the Godfathers Of Linux reside, and maybe someone hears my prayer and
> could explain us sheep what you should do in such a case. Increase the
> HZ from 250 to 1000, or decrease to 100? Or maybe setting the
> preemption model from server to voluntary or preemptible? Or is that
> whining to be ignored, and if yes, what is this message for at all?
>
> Please give us wisdom, and we will spread your word. Amen.
>
> Answers please per PM, I'm not on this list.
>

If you are losing interrupts at 256 Hz, you have either/or:
(1) Some very BAD driver that is disabling interrupts for way too long.
(2) Some very slow CPU (like 40 Mhz) that is being overwhelmed by a lot of
network interrupt activity.

The last time I measured, with a 400 MHz, '486 machine, Linux could
handle 50,000 interrupts per second off the printer port, with the
lowest priority interrupt (IRQ7). The work, within the interrupt was
to toggle a bit in memory and write it back to the printer data port.

This was used to show latency (with a scope) and log any missed
interrupts of which there were none.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.89 BogoMips).
Warning : 98.36% of all statistics are fiction, book release in April.
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: rtc: lost some interrupts at 256Hz

[Lee Revell]

On Thu, 2006-04-20 at 22:37 +0200, Michael Monnerie wrote:
> When you google for such messages, you can find a lot of people asking, 
> but nobody seems to have an answer. That's why I ask this list, where 
> the Godfathers Of Linux reside, and maybe someone hears my prayer and 
> could explain us sheep what you should do in such a case. Increase the 
> HZ from 250 to 1000, or decrease to 100? Or maybe setting the 
> preemption model from server to voluntary or preemptible? Or is that 
> whining to be ignored, and if yes, what is this message for at all?
> 
> Please give us wisdom, and we will spread your word. Amen.
> 
> Answers please per PM, I'm not on this list.
> 
> mfg zmi *or could you ask in a nicer way?*

Changing the preemption model to voluntary or full preemption could
certainly help.  What app is using the RTC, mplayer?

Lee

Re: rtc: lost some interrupts at 256Hz

[Florian Schmidt]

On Thu, 20 Apr 2006 17:03:02 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> Changing the preemption model to voluntary or full preemption could
> certainly help.  What app is using the RTC, mplayer?

I wonder what the "lost ticks" message really means. That the tick
wasn't consumed by a read on the device file? If so, setting niceness or
rt prio for the consuming task might help. 

Or does it mean that the kernel failed in some way keeping up with the
rtc?

Flo

-- 
Palimm Palimm!
http://tapas.affenbande.org

Re: rtc: lost some interrupts at 256Hz

[Jan Engelhardt]

On Apr 20 2006 16:57, linux-os (Dick Johnson) wrote:
>On Thu, 20 Apr 2006, Michael Monnerie wrote:
>
>> When you google for such messages, you can find a lot of people asking,
>> but nobody seems to have an answer. That's why I ask this list, where
>> the Godfathers Of Linux reside, and maybe someone hears my prayer and
>> could explain us sheep what you should do in such a case. Increase the
>> HZ from 250 to 1000, or decrease to 100? Or maybe setting the
>> preemption model from server to voluntary or preemptible? Or is that
>> whining to be ignored, and if yes, what is this message for at all?
>>
>> Please give us wisdom, and we will spread your word. Amen.
>>
>> Answers please per PM, I'm not on this list.
>
>If you are losing interrupts at 256 Hz, you have either/or:
>(1) Some very BAD driver that is disabling interrupts for way too long.
>(2) Some very slow CPU (like 40 Mhz) that is being overwhelmed by a lot of
>network interrupt activity.

On Apr 20 2006 17:03, Lee Revell wrote:
>Date: Thu, 20 Apr 2006 17:03:02 -0400
>Changing the preemption model to voluntary or full preemption could
>certainly help.  What app is using the RTC, mplayer?


I see this message too. (And I've got some more details).

Whenever I switch from console to X, I get that message. I presume it is 
due the binary nvidia blob I have loaded, because when switching to X, 
even the sound gets distorted for a microsecond (the well-known the 
soundcard replays the sound buffer - no new data transferred to it).
Yeah, looks like something's keeping IRQ disabled for quite a while.
(My normal freq is 100 Hz + no preemption, if that makes any difference.)

I have not yet seen this on the free nv driver for Xorg or the (free) 
ati/Xorg. Same configuration (100 Hz, no preempt), though!

Does not really matter what userspace app is running. If it's mplayer the 
message is "rtc: lost some interrupts at 1024Hz.", and if it's VMware it's 
whatever the Guest OS requires, mostly 2000 or 200 Hz.


Jan Engelhardt
-- 

Re: rtc: lost some interrupts at 256Hz

[Lee Revell]

On Fri, 2006-04-21 at 00:18 +0200, Jan Engelhardt wrote:
> Does not really matter what userspace app is running. If it's mplayer
> the message is "rtc: lost some interrupts at 1024Hz.", and if it's
> VMware it's whatever the Guest OS requires, mostly 2000 or 200 Hz. 

I asked about the app because I was wondering whether he had a server or
a desktop.

Lee

Re: rtc: lost some interrupts at 256Hz

[Michael Monnerie]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

On Freitag, 21. April 2006 00:22 Lee Revell wrote:
> I asked about the app because I was wondering whether he had a server
> or a desktop.

It's all servers. The Hz rate can change over time:

Mar  3 02:04:41 baum kernel: rtc: lost some interrupts at 256Hz.
Mar  3 02:04:53 baum kernel: rtc: lost some interrupts at 256Hz.
Mar  3 02:04:55 baum kernel: rtc: lost some interrupts at 256Hz.
Mar  3 02:38:19 baum kernel: rtc: lost some interrupts at 256Hz.
Mar  3 02:42:22 baum kernel: rtc: lost some interrupts at 256Hz.
Mar 16 15:53:50 baum kernel: rtc: lost some interrupts at 2048Hz.
Mar 16 15:54:01 baum kernel: rtc: lost some interrupts at 2048Hz.
Mar 16 16:29:33 baum kernel: rtc: lost some interrupts at 2048Hz.
Mar 29 23:01:36 baum kernel: warning: many lost ticks.

Interesting may be the fact that VMware runs on at least some hosts. I 
didn't check if all servers have it, but I found a good article on 
vmware support:

http://www.vmware.com/support/kb/enduser/std_adp.php?p_sid=tthOqz5i&p_lva=&p_faqid=1420&p_created=1093994398&p_sp=cF9zcmNoPTEmcF9ncmlkc29ydD0mcF9yb3dfY250PTE3JnBfc2VhcmNoX3RleHQ9bG9zdCBzb21lIGludGVycnJ1cHRzJnBfc2VhcmNoX3R5cGU9NyZwX3Byb2RfbHZsMT1_YW55fiZwX3Byb2RfbHZsMj1_YW55fiZwX3NvcnRfYnk9ZGZsdCZwX3BhZ2U9MQ**&p_li=

So as a summary:
- Lee Revell said changing preemption model from server to voluntary or 
preemption could help
- increasing HZ on the vm-host, decreasing in the vm-clients should 
help, according to VMware support

It would be good if this information is spread on the support lists, I 
saw that question a lot. Also, a short text under Documentation in the 
kernel source could be nice, explaining why this happens and what to do 
then.

I will try the suggestion from the vmware support link I posted above, 
and will report back.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660/4156531                          .network.your.ideas.
// PGP Key:   "lynx -source http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2  9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net                 Key-ID: 0x55CBA4EE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: rtc: lost some interrupts at 256Hz

[Michael Monnerie]

[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]

On Freitag, 21. April 2006 00:18 Jan Engelhardt wrote:
> Does not really matter what userspace app is running. If it's mplayer
> the message is "rtc: lost some interrupts at 1024Hz.", and if it's
> VMware it's whatever the Guest OS requires, mostly 2000 or 200 Hz.

The tipps from vmware support at
http://www.vmware.com/support/kb/enduser/std_adp.php?p_sid=tthOqz5i&p_lva=&p_faqid=1420&p_created=1093994398&p_sp=cF9zcmNoPTEmcF9ncmlkc29ydD0mcF9yb3dfY250PTE3JnBfc2VhcmNoX3RleHQ9bG9zdCBzb21lIGludGVycnJ1cHRzJnBfc2VhcmNoX3R5cGU9NyZwX3Byb2RfbHZsMT1_YW55fiZwX3Byb2RfbHZsMj1_YW55fiZwX3NvcnRfYnk9ZGZsdCZwX3BhZ2U9MQ**&p_li=
seem to help, didn't get a "lost some interrupts" warning during the 
last 2 days. But, I got one "many lost ticks" warning:

Apr 23 23:06:51 baum kernel: warning: many lost ticks.
Apr 23 23:06:51 baum kernel: Your time source seems to be instable or 
some driver is hogging interupts
Apr 23 23:06:51 baum kernel: rip __do_softirq+0x4f/0xd0

What I did was
- setup VM host OS to 1000HZ, preemptible kernel
- VM clients use 100HZ, and kernel options "clock=pit nosmp noapic 
nolapic"
- VM client kernels are all non-SMP compiled now.

Still, some kind of documentation in the Linux source tree would be 
nice, explaining what that message means and how to fix it.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660/4156531                          .network.your.ideas.
// PGP Key:   "lynx -source http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2  9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net                 Key-ID: 0x55CBA4EE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: rtc: lost some interrupts at 256Hz

[Lee Revell]

On Tue, 2006-04-25 at 12:32 +0200, Michael Monnerie wrote:
> Still, some kind of documentation in the Linux source tree would be 
> nice, explaining what that message means and how to fix it.
> 

It means exactly what the printk says:

"Your time source seems to be instable or 
some driver is hogging interupts"

Lee