TSC Problems (warp between CPUs)

TSC Problems (warp between CPUs)

[Alex]

Hi There,

Firstly, apologies for the length of this post, however there is a bit 
of information I need to give so it is clear to everyone
what is happening, what I have tried, and what I am hoping to achieve.

I am having a problem with getting the TSC clocksource to work on my 
new system. I have been trying to work with my motherboard manufacturer 
(gigabyte)
to try and alert them to a possible BIOS bug but I am not getting 
anywhere with them (replies in broken english, problem not being 
understood
by their support etc).

CPU: Intel i7-4930K
Motherboard: Gigabyte GA-X79-UP4 with latest bios.

Some info on the problem (various outputs of shell commands):
-------------------------------------------------------------

alex@desktop:~$ uname -a
Linux desktop 3.12.5-custom #1 SMP PREEMPT Sat Dec 21 17:28:12 EST 2013 
x86_64 x86_64 x86_64 GNU/Linux

alex@desktop:~$ dmesg | grep -i tsc
tsc: Fast TSC calibration using PIT
tsc: Detected 3400.159 MHz processor
TSC deadline timer enabled
TSC synchronization [CPU#0 -> CPU#1]:
Measured 6618476436 cycles TSC warp between CPUs, turning off TSC 
clock.
tsc: Marking TSC unstable due to check_tsc_sync_source failed

alex@desktop:~$ cat 
/sys/devices/system/clocksource/clocksource0/available_clocksource
hpet acpi_pm

alex@desktop:~$ cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

alex@desktop:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
stepping        : 4
microcode       : 0x416
cpu MHz         : 3400.159
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good 
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 
x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat 
epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid 
fsgsbase smep erms
bogomips        : 6800.31
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 1

<and this continues for processor id's up to 11>

------------------------

As you can see "nonstop_tsc" is supported.

What I have tried doing to address the issue:
---------------------------------------------

* Tried disabling all power/energy saving functions in the CPU cores
* CPU Eist/freqency Scaling is disabled.
* Nothing is overclocked.
* No CPU turbo function enabled.

None of the above have helped. Some digging around on the net has led 
me back to the BIOS being the issue, in that it is using an MSR to write 
to the TSC and leaving it in an inconsistent state.


An interesting quote I found online, apparently from a linux kernel 
dev:
------------------------------------------------------------------------

so the way the hardware works is that there is 1 "master" tsc in the 
CPU package, that gets started when the cpu package comes out of reset. 
all logical cpus keep an offset value from that, which starts at 0, and 
the "master + offset" value is what gets returned on rdtsc. if someone 
writes to the tsc (using an MSR), what actually happens is that the 
master tsc does not change, only the per logical cpu offset gets 
changed.

Linux does not write to the TSC since quite a while... which means the 
BIOS is doing that. It really should not.
---------------------------

What I am wanting to know, is whether there is any way I can work 
around what is likely to be a BIOS bug by having the kernel 
intentionally reset the TSC.

I saw a patch floating around on the net that does something like this 
(for tsc-sync.c):

+       wrmsrl(MSR_IA32_TSC, 0);
         rdtsc_barrier();
         start = get_cycles();
         rdtsc_barrier();

Is there any safe patch to force the TSC to be reset/reinitialized that 
I can add to the kernel?


I have a number of applications that will benefit from TSC timing 
rather than HPET and would really like to try and get TSC to work.

Kind Regards,
Alex.

Re: TSC Problems (warp between CPUs)

[Mike Galbraith]

On Sat, 2013-12-28 at 13:24 +1100, Alex wrote:

> Linux does not write to the TSC since quite a while... which means the 
> BIOS is doing that. It really should not.

I saw the same on an Intel server recently, poor thing was stuck with
using HPET.

-Mike

Re: TSC Problems (warp between CPUs)

[Alex]

Just as a follow up to this, I tried to reset the TSC in tsc-sync.c 
with "wrmsrl(MSR_IA32_TSC, 0);" code looks like this:

static void check_tsc_warp(unsigned int timeout)
{
         cycles_t start, now, prev, end;
         int i;

         wrmsrl(MSR_IA32_TSC, 0);
         rdtsc_barrier();
         start = get_cycles();
         rdtsc_barrier();


Now i see this during boot:

alex@desktop:~$ dmesg | grep -i tsc
tsc: Fast TSC calibration using PIT
tsc: Detected 3400.348 MHz processor
TSC deadline timer enabled
TSC synchronization [CPU#0 -> CPU#3]:
Measured 56 cycles TSC warp between CPUs, turning off TSC clock.

56 cycles... a lot lower than 6618476436 cycles.

I read a post by an intel engineer on the TSC:

Quote:
------
The time-stamp counter on recent Intel processors is reset to zero each 
time the processor package has RESET asserted. From that point onwards 
the invariant TSC will continue to tick constantly across frequency 
changes, turbo mode and ACPI C-states. All parts that see RESET 
synchronously will have their TSC's completely synchronized. This 
synchronous distribution of RESET is required for all sockets connected 
to a single PCH. For multi-node systems RESET might not be synchronous.

The biggest issue with TSC synchronization across multiple 
threads/cores/packages is the ability for software to write the TSC. The 
TSC is exposed as MSR 0x10. Software is able to use WRMSR 0x10 to set 
the TSC. However, as the TSC continues as a moving target, writing it is 
not guaranteed to be precise. For example a SMI (System Management 
Interrupt) could interrupt the software flow that is attempting to write 
the time-stamp counter immediately prior to the WRMSR. This could mean 
the value written to the TSC could vary by thousands to millions of 
clocks.

------------ end quote ----------

I suspect the TSC cannot be reset in the manner I just attempted (given 
what was just said above)?  I gather this means I am out of luck and 
this is impossible to fix (short of a miracle from my motherboard 
manufacturer).

Alex.



On 2013-12-28 13:24, Alex wrote:
> Hi There,
>
> Firstly, apologies for the length of this post, however there is a
> bit of information I need to give so it is clear to everyone
> what is happening, what I have tried, and what I am hoping to 
> achieve.
>
> I am having a problem with getting the TSC clocksource to work on my
> new system. I have been trying to work with my motherboard
> manufacturer (gigabyte)
> to try and alert them to a possible BIOS bug but I am not getting
> anywhere with them (replies in broken english, problem not being
> understood
> by their support etc).
>
> CPU: Intel i7-4930K
> Motherboard: Gigabyte GA-X79-UP4 with latest bios.
>
> Some info on the problem (various outputs of shell commands):
> -------------------------------------------------------------
>
> alex@desktop:~$ uname -a
> Linux desktop 3.12.5-custom #1 SMP PREEMPT Sat Dec 21 17:28:12 EST
> 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> alex@desktop:~$ dmesg | grep -i tsc
> tsc: Fast TSC calibration using PIT
> tsc: Detected 3400.159 MHz processor
> TSC deadline timer enabled
> TSC synchronization [CPU#0 -> CPU#1]:
> Measured 6618476436 cycles TSC warp between CPUs, turning off TSC 
> clock.
> tsc: Marking TSC unstable due to check_tsc_sync_source failed
>
> alex@desktop:~$ cat
> /sys/devices/system/clocksource/clocksource0/available_clocksource
> hpet acpi_pm
>
> alex@desktop:~$ cat
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> hpet
>
> alex@desktop:~$ cat /proc/cpuinfo
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 62
> model name      : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
> stepping        : 4
> microcode       : 0x416
> cpu MHz         : 3400.159
> cache size      : 12288 KB
> physical id     : 0
> siblings        : 12
> core id         : 0
> cpu cores       : 6
> apicid          : 0
> initial apicid  : 0
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm 
> pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
> dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid dca 
> sse4_1
> sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand
> lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority
> ept vpid fsgsbase smep erms
> bogomips        : 6800.31
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 46 bits physical, 48 bits virtual
> power management:
>
> processor       : 1
>
> <and this continues for processor id's up to 11>
>
> ------------------------
>
> As you can see "nonstop_tsc" is supported.
>
> What I have tried doing to address the issue:
> ---------------------------------------------
>
> * Tried disabling all power/energy saving functions in the CPU cores
> * CPU Eist/freqency Scaling is disabled.
> * Nothing is overclocked.
> * No CPU turbo function enabled.
>
> None of the above have helped. Some digging around on the net has led
> me back to the BIOS being the issue, in that it is using an MSR to
> write to the TSC and leaving it in an inconsistent state.
>
>
> An interesting quote I found online, apparently from a linux kernel 
> dev:
> 
> ------------------------------------------------------------------------
>
> so the way the hardware works is that there is 1 "master" tsc in the
> CPU package, that gets started when the cpu package comes out of
> reset. all logical cpus keep an offset value from that, which starts
> at 0, and the "master + offset" value is what gets returned on rdtsc.
> if someone writes to the tsc (using an MSR), what actually happens is
> that the master tsc does not change, only the per logical cpu offset
> gets changed.
>
> Linux does not write to the TSC since quite a while... which means
> the BIOS is doing that. It really should not.
> ---------------------------
>
> What I am wanting to know, is whether there is any way I can work
> around what is likely to be a BIOS bug by having the kernel
> intentionally reset the TSC.
>
> I saw a patch floating around on the net that does something like
> this (for tsc-sync.c):
>
> +       wrmsrl(MSR_IA32_TSC, 0);
>         rdtsc_barrier();
>         start = get_cycles();
>         rdtsc_barrier();
>
> Is there any safe patch to force the TSC to be reset/reinitialized
> that I can add to the kernel?
>
>
> I have a number of applications that will benefit from TSC timing
> rather than HPET and would really like to try and get TSC to work.
>
> Kind Regards,
> Alex.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Kind Regards,
Alex.

Re: TSC Problems (warp between CPUs)

[el_es]

Alex <xor <at> xor.bz> writes:
[snip quotes]
> 
> I suspect the TSC cannot be reset in the manner I just attempted (given 
> what was just said above)?  I gather this means I am out of luck and 
> this is impossible to fix (short of a miracle from my motherboard 
> manufacturer).
> 
> Alex.
> 
Not being a specialist here at all, so, sorry if it's a lame question,

why would the whole calibration routine be only ran once anyway?
Maybe a re-run (or a few) would eventually miss the SMI and 
the warp be gone? (and could be detected and added as a quirk/
workaround for buggy bios(es)/hw too maybe?)

(example from real life: guitar tuners [people that is] can go over 
the entire tuning procedure a few times before achieving perfection,
as sometimes tuning one string affects all the others)

-L.

Re: TSC Problems (warp between CPUs)

[One Thousand Gnomes]

> not guaranteed to be precise. For example a SMI (System Management 
> Interrupt) could interrupt the software flow that is attempting to write 
> the time-stamp counter immediately prior to the WRMSR. This could mean 
> the value written to the TSC could vary by thousands to millions of 
> clocks.

Yes SMI is a disaster area for any real time activity (and many other
things ;) ), but many systems actually make little use of it, especially
once the USB is owned by the OS.

For synchronization you can retry the sync if it isn't within an
acceptable range. The odds of getting an SMI mid sync setup should be
very very low, so the odds of repeating the failure several times should
be negligible and after a few tries you could give up and assume the
hardware is buggered then fall back to HPET.

Alan

Re: TSC Problems (warp between CPUs)

[Alex]

On 2013-12-29 01:35, One Thousand Gnomes wrote:
>> not guaranteed to be precise. For example a SMI (System Management
>> Interrupt) could interrupt the software flow that is attempting to 
>> write
>> the time-stamp counter immediately prior to the WRMSR. This could 
>> mean
>> the value written to the TSC could vary by thousands to millions of
>> clocks.
>
> Yes SMI is a disaster area for any real time activity (and many other
> things ;) ), but many systems actually make little use of it, 
> especially
> once the USB is owned by the OS.
>
> For synchronization you can retry the sync if it isn't within an
> acceptable range. The odds of getting an SMI mid sync setup should be
> very very low, so the odds of repeating the failure several times 
> should
> be negligible and after a few tries you could give up and assume the
> hardware is buggered then fall back to HPET.

Hi Alan,

The sync retry sounds like an interesting strategy but is outside my 
knowledge
scope as a regular end user (I do have a reasonable understanding of C 
and can patch+
recompile) but I lack the architecture knowledge to implement a patch 
to do this.

Any suggestions?

-- 
Kind Regards,
Alex.

Re: TSC Problems (warp between CPUs)

[Alex]

A Happy new year to all :)

Just wondering whether anyone has any ideas on how I could force the 
TSC to resync?

Starting to get a bit desperate. The motherboard manufacturers support 
is useless.... They keep telling me to install Windows *groan*. I dont 
think there is any easy way to expose what clock source the OS is using 
even in windows.

I even tried firing an email at the AmiBios guys. I think I have 
exhausted every avenue of help as far as the hardware goes.

Any suggestions?

Kind regards,
Alex

On 2013-12-29 11:21, Alex wrote:
> On 2013-12-29 01:35, One Thousand Gnomes wrote:
>>> not guaranteed to be precise. For example a SMI (System Management
>>> Interrupt) could interrupt the software flow that is attempting to 
>>> write
>>> the time-stamp counter immediately prior to the WRMSR. This could 
>>> mean
>>> the value written to the TSC could vary by thousands to millions of
>>> clocks.
>>
>> Yes SMI is a disaster area for any real time activity (and many 
>> other
>> things ;) ), but many systems actually make little use of it, 
>> especially
>> once the USB is owned by the OS.
>>
>> For synchronization you can retry the sync if it isn't within an
>> acceptable range. The odds of getting an SMI mid sync setup should 
>> be
>> very very low, so the odds of repeating the failure several times 
>> should
>> be negligible and after a few tries you could give up and assume the
>> hardware is buggered then fall back to HPET.
>
> Hi Alan,
>
> The sync retry sounds like an interesting strategy but is outside my 
> knowledge
> scope as a regular end user (I do have a reasonable understanding of
> C and can patch+
> recompile) but I lack the architecture knowledge to implement a patch
> to do this.
>
> Any suggestions?

-- 
Kind Regards,
Alex.

Re: TSC Problems (warp between CPUs)

[Adrian Vasile]

Alex,

Have you managed to fix this issue? I'm seeing the same on an Asus X99
Deluxe board with both 5930K and 5960X and the latest BIOS as of today
(0904).
I had some hopes of linux-3.17 working around this but it did not help.
Asus support is disappointing, I don't think they will ever fix this.

I'm wondering if it's worth trying a different motherboard, any
recommendations?

Thank you.

Regards,
Adrian

On 31/12/13 22:24, Alex wrote:
> A Happy new year to all :)
>
> Just wondering whether anyone has any ideas on how I could force the
> TSC to resync?
>
> Starting to get a bit desperate. The motherboard manufacturers support
> is useless.... They keep telling me to install Windows *groan*. I dont
> think there is any easy way to expose what clock source the OS is
> using even in windows.
>
> I even tried firing an email at the AmiBios guys. I think I have
> exhausted every avenue of help as far as the hardware goes.
>
> Any suggestions?
>
> Kind regards,
> Alex
>
> On 2013-12-29 11:21, Alex wrote:
>> On 2013-12-29 01:35, One Thousand Gnomes wrote:
>>>> not guaranteed to be precise. For example a SMI (System Management
>>>> Interrupt) could interrupt the software flow that is attempting to
>>>> write
>>>> the time-stamp counter immediately prior to the WRMSR. This could mean
>>>> the value written to the TSC could vary by thousands to millions of
>>>> clocks.
>>>
>>> Yes SMI is a disaster area for any real time activity (and many other
>>> things ;) ), but many systems actually make little use of it,
>>> especially
>>> once the USB is owned by the OS.
>>>
>>> For synchronization you can retry the sync if it isn't within an
>>> acceptable range. The odds of getting an SMI mid sync setup should be
>>> very very low, so the odds of repeating the failure several times
>>> should
>>> be negligible and after a few tries you could give up and assume the
>>> hardware is buggered then fall back to HPET.
>>
>> Hi Alan,
>>
>> The sync retry sounds like an interesting strategy but is outside my
>> knowledge
>> scope as a regular end user (I do have a reasonable understanding of
>> C and can patch+
>> recompile) but I lack the architecture knowledge to implement a patch
>> to do this.
>>
>> Any suggestions?
>

________________________________

The information in this e-mail is intended only for the person or entity to which it is addressed.

It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it.

If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments.

Messages and attachments are scanned for all known viruses. Always scan attachments before opening them.