why guests show "Clocksource tsc unstable" on bootup?

why guests show "Clocksource tsc unstable" on bootup?

[Tomasz Chmielewski]

Why do my guests show "Clocksource tsc unstable" on bootup?

dmesg shows that kvm-clock was set as the primary cpu clock.
Yet a bit later kernel says "Clocksource tsc unstable".

Is it something to worry about, or perhaps calculating tsc is hardcoded?
And as such, will be always checked?
Or, is it host CPU related?

[    0.000000] kvm-clock: cpu 0, msr 0:3baf81, boot clock
[    0.000000] kvm-clock: cpu 0, msr 0:1208f81, primary cpu clock
[    0.000000] Kernel command line: root=/dev/vda1 ro quiet clocksource=kvm-clock
[    1.253602] rtc_cmos 00:01: setting system clock to 2009-03-09 11:41:30 UTC (1236598890)
[   41.500623] Clocksource tsc unstable (delta = -153498948 ns)


-- 
Tomasz Chmielewski
http://wpkg.org

Re: why guests show "Clocksource tsc unstable" on bootup?

[Avi Kivity]

Tomasz Chmielewski wrote:
> Why do my guests show "Clocksource tsc unstable" on bootup?
>

Linux expects the tsc to be monotonic and to have a 1:1 correspondence 
with real time, which isn't easy to achieve with virtualization.

> dmesg shows that kvm-clock was set as the primary cpu clock.
> Yet a bit later kernel says "Clocksource tsc unstable".
>
> Is it something to worry about, or perhaps calculating tsc is hardcoded?
> And as such, will be always checked?
> Or, is it host CPU related?
>
> [    0.000000] kvm-clock: cpu 0, msr 0:3baf81, boot clock
> [    0.000000] kvm-clock: cpu 0, msr 0:1208f81, primary cpu clock
> [    0.000000] Kernel command line: root=/dev/vda1 ro quiet 
> clocksource=kvm-clock
> [    1.253602] rtc_cmos 00:01: setting system clock to 2009-03-09 
> 11:41:30 UTC (1236598890)
> [   41.500623] Clocksource tsc unstable (delta = -153498948 ns)
>
>

What host cpu and kvm version are you using?


-- 
error compiling committee.c: too many arguments to function

Re: why guests show "Clocksource tsc unstable" on bootup?

[Tomasz Chmielewski]

Avi Kivity schrieb:
> Tomasz Chmielewski wrote:
>> Why do my guests show "Clocksource tsc unstable" on bootup?
>>
> 
> Linux expects the tsc to be monotonic and to have a 1:1 correspondence 
> with real time, which isn't easy to achieve with virtualization.

But the clocksource is kvm-clock, so why does the guest probe tsc at all?


>> dmesg shows that kvm-clock was set as the primary cpu clock.
>> Yet a bit later kernel says "Clocksource tsc unstable".
>>
>> Is it something to worry about, or perhaps calculating tsc is hardcoded?
>> And as such, will be always checked?
>> Or, is it host CPU related?
>>
>> [    0.000000] kvm-clock: cpu 0, msr 0:3baf81, boot clock
>> [    0.000000] kvm-clock: cpu 0, msr 0:1208f81, primary cpu clock
>> [    0.000000] Kernel command line: root=/dev/vda1 ro quiet 
>> clocksource=kvm-clock
>> [    1.253602] rtc_cmos 00:01: setting system clock to 2009-03-09 
>> 11:41:30 UTC (1236598890)
>> [   41.500623] Clocksource tsc unstable (delta = -153498948 ns)
>>
>>
> 
> What host cpu and kvm version are you using?

I pasted a part of /proc/cpuinfo below.
I saw these with kvm-83 and kvm-84 (with cpufreq disabled, as it perhaps 
can matter).

processor       : 3
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 2212
stepping        : 2
cpu MHz         : 2000.000
cache size      : 1024 KB
physical id     : 1
siblings        : 2
core id         : 1
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy 
svm extapic cr8_legacy
bogomips        : 3993.03
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc



-- 
Tomasz Chmielewski
http://wpkg.org

Re: why guests show "Clocksource tsc unstable" on bootup?

[Michael Tokarev]

Tomasz Chmielewski wrote:
> Avi Kivity schrieb:
>> Tomasz Chmielewski wrote:
>>> Why do my guests show "Clocksource tsc unstable" on bootup?

The same happens here as well.

>> Linux expects the tsc to be monotonic and to have a 1:1 correspondence 
>> with real time, which isn't easy to achieve with virtualization.
> 
> But the clocksource is kvm-clock, so why does the guest probe tsc at all?
> 
> 
>>> dmesg shows that kvm-clock was set as the primary cpu clock.
>>> Yet a bit later kernel says "Clocksource tsc unstable".
>>>
>>> Is it something to worry about, or perhaps calculating tsc is hardcoded?
>>> And as such, will be always checked?
>>> Or, is it host CPU related?
>>>
>>> [    0.000000] kvm-clock: cpu 0, msr 0:3baf81, boot clock
>>> [    0.000000] kvm-clock: cpu 0, msr 0:1208f81, primary cpu clock
>>> [    0.000000] Kernel command line: root=/dev/vda1 ro quiet 
>>> clocksource=kvm-clock
>>> [    1.253602] rtc_cmos 00:01: setting system clock to 2009-03-09 
>>> 11:41:30 UTC (1236598890)
>>> [   41.500623] Clocksource tsc unstable (delta = -153498948 ns)
>>>
>>
>> What host cpu and kvm version are you using?
> 
> I pasted a part of /proc/cpuinfo below.
> I saw these with kvm-83 and kvm-84 (with cpufreq disabled, as it perhaps 
> can matter).
> 
> processor       : 3
> vendor_id       : AuthenticAMD
> cpu family      : 15
> model           : 65
> model name      : Dual-Core AMD Opteron(tm) Processor 2212
> stepping        : 2
> cpu MHz         : 2000.000
> cache size      : 1024 KB
> physical id     : 1
> siblings        : 2
> core id         : 1
> cpu cores       : 2
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
> fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy 
> svm extapic cr8_legacy
> bogomips        : 3993.03
> TLB size        : 1024 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc

Here, it shows unstable tsc on this machine since kvm-72 or even earlier,
up to current kvm-84.

The CPU on the host is this one (4 cores):

vendor_id	: AuthenticAMD
cpu family	: 16
model		: 2
model name	: AMD Phenom(tm) 9750 Quad-Core Processor
stepping	: 3
cpu MHz		: 2400.000
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good pni monitor cx16 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips	: 4812.60
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

Host kernel is 2.6.28.8 amd64, guest is running 2.6.27.19 i686,
kvm-84 userspace, kvm modules are from host kernel.

Here how it looks like from the guest:

# dmesg | egrep -i 'clock|tsc|time'
kvm-clock: cpu 0, msr 0:3cb001, boot clock
ACPI: PM-Timer IO Port: 0xb008
kvm-clock: cpu 0, msr 0:1409001, primary cpu clock
TSC: Unable to calibrate against PIT
TSC: using PMTIMER reference calibration
Calibrating delay loop (skipped), value calculated using timer frequency.. 4800.87 BogoMIPS (lpj=24004390)
..TIMER: vector=0x31 apic1=0 pin1=0 apic2=-1 pin2=-1
Machine check exception polling timer started.
Real Time Clock Driver v1.12ac
Clocksource tsc unstable (delta = -69824578 ns)

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock acpi_pm jiffies tsc

Thanks!

/mjt

Re: why guests show "Clocksource tsc unstable" on bootup?

[Avi Kivity]

Tomasz Chmielewski wrote:
> Avi Kivity schrieb:
>> Tomasz Chmielewski wrote:
>>> Why do my guests show "Clocksource tsc unstable" on bootup?
>>>
>>
>> Linux expects the tsc to be monotonic and to have a 1:1 
>> correspondence with real time, which isn't easy to achieve with 
>> virtualization.
>
> But the clocksource is kvm-clock, so why does the guest probe tsc at all?
>

Good question.  I don't know what the answer is.  Maybe the scheduler 
uses tsc for its internal clock.

>
> I pasted a part of /proc/cpuinfo below.
> I saw these with kvm-83 and kvm-84 (with cpufreq disabled, as it 
> perhaps can matter).
>
> processor       : 3
> vendor_id       : AuthenticAMD
> cpu family      : 15
> model           : 65
> model name      : Dual-Core AMD Opteron(tm) Processor 2212
> stepping        : 2
> cpu MHz         : 2000.000
> cache size      : 1024 KB
> physical id     : 1
> siblings        : 2
> core id         : 1
> cpu cores       : 2
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
> fxsr_opt rdtscp lm 3dnowext 3dnow rep_good pni cx16 lahf_lm cmp_legacy 
> svm extapic cr8_legacy
> bogomips        : 3993.03
> TLB size        : 1024 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc
>

This cpu indeed has unstable tsc, but disabling cpufreq should have 
fixed it.

Maybe we need to disable C-states as well?


-- 
error compiling committee.c: too many arguments to function

Re: why guests show "Clocksource tsc unstable" on bootup?

[Avi Kivity]

Avi Kivity wrote:
>
> This cpu indeed has unstable tsc, but disabling cpufreq should have 
> fixed it.
>
> Maybe we need to disable C-states as well?
>
>

Can you try idle=poll on the host kernel command line?  Note that this 
will run the cpu hot, so remove it after the test.

-- 
error compiling committee.c: too many arguments to function