[v1 0/9] Early boot time stamps for x86

[v1 0/9] Early boot time stamps for x86

[Pavel Tatashin]

Last week I sent out patches to enable early boot time stamp on SPARC, that
work is now under review:
http://www.spinics.net/lists/sparclinux/msg17372.html

This is continuation for that work, but adding early boot time stamps
support for x86 machines.

Here is example how this information is useful:

Before:
https://hastebin.com/zofiqazuze.scala

After:
https://hastebin.com/otayoliruc.scala

If you take a look at the before case, it seems that this particular x86
machine took only 2 minutes to boot. Which, while can be improved, is not
too bad considering that this machine has 192CPUs and 2.2T of memory.

But, as it can be seen in the fix case,
the time is much longer:
- early boot time stamps account for another 80s!
- early tsc offset is 549s, so it took over 9 minutes to get through POST,
  and GRUB before starting linux

Now, the total boot time is 12m52s, which is really slow.

Pavel Tatashin (9):
  sched/clock: broken stable to unstable transfer
  sched/clock: interface to allow timestamps early in boot
  x86/cpu: determining x86 vendor early
  x86/tsc: early MSR-based CPU/TSC frequency discovery
  x86/tsc: disable early messages from quick_pit_calibrate
  x86/tsc: use cpuid to determine TSC frequency
  x86/tsc: use cpuid to determine CPU frequency
  x86/tsc: tsc early
  x86/tsc: use tsc early

 arch/x86/include/asm/processor.h |    1 +
 arch/x86/include/asm/tsc.h       |    5 +
 arch/x86/kernel/cpu/common.c     |   36 ++++++++
 arch/x86/kernel/head64.c         |    1 +
 arch/x86/kernel/time.c           |    1 +
 arch/x86/kernel/tsc.c            |  166 +++++++++++++++++++++++++++++++++-----
 arch/x86/kernel/tsc_msr.c        |   38 ++++++---
 include/linux/sched/clock.h      |    4 +
 kernel/sched/clock.c             |   70 +++++++++++++++-
 9 files changed, 284 insertions(+), 38 deletions(-)

[v1 1/9] sched/clock: broken stable to unstable transfer

[Pavel Tatashin]

When it is determined that clock is actually unstable, and we switch from
stable to unstable the following function is eventually called:

__clear_sched_clock_stable()

In this function we set gtod_offset so the following holds true:

sched_clock() + raw_offset == ktime_get_ns() + gtod_offset

But instead of getting the latest timestamps, we use the last values
from scd, so instead of sched_clock() we use scd->tick_raw, and instead of
ktime_get_ns() we use scd->tick_gtod.

However, later, when we use gtod_offset sched_clock_local() we do not add
it to scd->tick_gtod to calculate the correct clock value when we determine
the boundaries for min/max clocks.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 kernel/sched/clock.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index a08795e..2bc1090 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -214,7 +214,7 @@ static inline u64 wrap_max(u64 x, u64 y)
  */
 static u64 sched_clock_local(struct sched_clock_data *scd)
 {
-	u64 now, clock, old_clock, min_clock, max_clock;
+	u64 now, clock, old_clock, min_clock, max_clock, gtod;
 	s64 delta;
 
 again:
@@ -231,9 +231,10 @@ static u64 sched_clock_local(struct sched_clock_data *scd)
 	 *		      scd->tick_gtod + TICK_NSEC);
 	 */
 
-	clock = scd->tick_gtod + gtod_offset + delta;
-	min_clock = wrap_max(scd->tick_gtod, old_clock);
-	max_clock = wrap_max(old_clock, scd->tick_gtod + TICK_NSEC);
+	gtod = scd->tick_gtod + gtod_offset;
+	clock = gtod + delta;
+	min_clock = wrap_max(gtod, old_clock);
+	max_clock = wrap_max(old_clock, gtod + TICK_NSEC);
 
 	clock = wrap_max(clock, min_clock);
 	clock = wrap_min(clock, max_clock);
-- 
1.7.1

[tip:sched/urgent] sched/clock: Fix broken stable to unstable transfer

[tip-bot for Pavel Tatashin]

Commit-ID:  7b09cc5a9debc86c903c2eff8f8a1fdef773c649
Gitweb:     http://git.kernel.org/tip/7b09cc5a9debc86c903c2eff8f8a1fdef773c649
Author:     Pavel Tatashin <pasha.tatashin@oracle.com>
AuthorDate: Wed, 22 Mar 2017 16:24:17 -0400
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 27 Mar 2017 10:23:48 +0200

sched/clock: Fix broken stable to unstable transfer

When it is determined that the clock is actually unstable, and
we switch from stable to unstable, the __clear_sched_clock_stable()
function is eventually called.

In this function we set gtod_offset so the following holds true:

  sched_clock() + raw_offset == ktime_get_ns() + gtod_offset

But instead of getting the latest timestamps, we use the last values
from scd, so instead of sched_clock() we use scd->tick_raw, and
instead of ktime_get_ns() we use scd->tick_gtod.

However, later, when we use gtod_offset sched_clock_local() we do not
add it to scd->tick_gtod to calculate the correct clock value when we
determine the boundaries for min/max clocks.

This can result in tick granularity sched_clock() values, so fix it.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: hpa@zytor.com
Fixes: 5680d8094ffa ("sched/clock: Provide better clock continuity")
Link: http://lkml.kernel.org/r/1490214265-899964-2-git-send-email-pasha.tatashin@oracle.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/clock.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index 24a3e01..00a45c4 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -221,7 +221,7 @@ static inline u64 wrap_max(u64 x, u64 y)
  */
 static u64 sched_clock_local(struct sched_clock_data *scd)
 {
-	u64 now, clock, old_clock, min_clock, max_clock;
+	u64 now, clock, old_clock, min_clock, max_clock, gtod;
 	s64 delta;
 
 again:
@@ -238,9 +238,10 @@ again:
 	 *		      scd->tick_gtod + TICK_NSEC);
 	 */
 
-	clock = scd->tick_gtod + __gtod_offset + delta;
-	min_clock = wrap_max(scd->tick_gtod, old_clock);
-	max_clock = wrap_max(old_clock, scd->tick_gtod + TICK_NSEC);
+	gtod = scd->tick_gtod + __gtod_offset;
+	clock = gtod + delta;
+	min_clock = wrap_max(gtod, old_clock);
+	max_clock = wrap_max(old_clock, gtod + TICK_NSEC);
 
 	clock = wrap_max(clock, min_clock);
 	clock = wrap_min(clock, max_clock);

[v1 2/9] sched/clock: interface to allow timestamps early in boot

[Pavel Tatashin]

In Linux printk() can output timestamps next to every line.  This is very
useful for tracking regressions, and finding places that can be optimized.
However, the timestamps are available only later in boot. On smaller
machines it is insignificant amount of time, but on larger it can be many
seconds or even minutes into the boot process.

This patch adds an interface for platforms with unstable sched clock to
show timestamps early in boot. In order to get this functionality a
platform must do:

- Implement u64 sched_clock_early()
  Clock that returns monotonic time

- Call sched_clock_early_init()
  Tells sched clock that the early clock can be used

- Call sched_clock_early_fini()
  Tells sched clock that the early clock is finished, and sched clock
  should hand over the operation to permanent clock.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 include/linux/sched/clock.h |    4 +++
 kernel/sched/clock.c        |   61 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 0 deletions(-)

diff --git a/include/linux/sched/clock.h b/include/linux/sched/clock.h
index 4a68c67..9b06b3a 100644
--- a/include/linux/sched/clock.h
+++ b/include/linux/sched/clock.h
@@ -67,6 +67,10 @@ static inline u64 local_clock(void)
 extern void sched_clock_idle_sleep_event(void);
 extern void sched_clock_idle_wakeup_event(u64 delta_ns);
 
+void sched_clock_early_init(void);
+void sched_clock_early_fini(void);
+u64 sched_clock_early(void);
+
 /*
  * As outlined in clock.c, provides a fast, high resolution, nanosecond
  * time source that is monotonic per cpu argument and has bounded drift
diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index 2bc1090..6a6d57b 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -79,6 +79,15 @@ unsigned long long __weak sched_clock(void)
 
 __read_mostly int sched_clock_running;
 
+/*
+ * We start with sched clock early static branch enabled, and global status
+ * disabled.  Early in boot it is decided whether to enable the global
+ * status as well (set sched_clock_early_running to true), and later, when
+ * early clock is no longer needed, the static branch is disabled.
+ */
+static DEFINE_STATIC_KEY_TRUE(__use_sched_clock_early);
+static bool __read_mostly sched_clock_early_running;
+
 void sched_clock_init(void)
 {
 	sched_clock_running = 1;
@@ -188,6 +197,12 @@ void sched_clock_init_late(void)
 	 */
 	smp_mb(); /* matches {set,clear}_sched_clock_stable() */
 
+	/*
+	 * It is guaranteed early clock is not running anymore. This function is
+	 * called from sched_init_smp(), and early clock must finish before smp.
+	 */
+	static_branch_disable(&__use_sched_clock_early);
+
 	if (__sched_clock_stable_early)
 		__set_sched_clock_stable();
 }
@@ -320,6 +335,11 @@ u64 sched_clock_cpu(int cpu)
 	if (sched_clock_stable())
 		return sched_clock() + raw_offset;
 
+	if (static_branch_unlikely(&__use_sched_clock_early)) {
+		if (sched_clock_early_running)
+			return sched_clock_early();
+	}
+
 	if (unlikely(!sched_clock_running))
 		return 0ull;
 
@@ -379,6 +399,47 @@ void sched_clock_idle_wakeup_event(u64 delta_ns)
 }
 EXPORT_SYMBOL_GPL(sched_clock_idle_wakeup_event);
 
+u64 __weak sched_clock_early(void)
+{
+	return 0;
+}
+
+/*
+ * Is called when sched_clock_early() is about to be finished, notifies sched
+ * clock that after this call sched_clock_early() can't be used.
+ *
+ * Must be called before smp_init() as sched_clock_early() is supposed to be
+ * used only during early boot, and are not guarantee to handle multi-CPU
+ * environment properly.
+ */
+void __init sched_clock_early_fini(void)
+{
+	struct sched_clock_data *scd = this_scd();
+	u64 now_early = sched_clock_early();
+	u64 now_sched = sched_clock();
+
+	/*
+	 * Set both: gtod_offset and raw_offset because we could switch to
+	 * either unstable clock or stable clock.
+	 */
+	gtod_offset = now_early - scd->tick_gtod;
+	raw_offset = now_early - now_sched;
+
+	sched_clock_early_running = false;
+}
+
+/*
+ * Notifies sched clock that early boot clocksource is available, it means that
+ * the current platform has implemented sched_clock_early().
+ *
+ * The early clock is running until we switch to a stable clock, or when we
+ * learn that the stable clock is not available.
+ */
+void __init sched_clock_early_init(void)
+{
+	sched_clock_early_running = true;
+}
+
 #else /* CONFIG_HAVE_UNSTABLE_SCHED_CLOCK */
 
 u64 sched_clock_cpu(int cpu)
-- 
1.7.1

[v1 3/9] x86/cpu: determining x86 vendor early

[Pavel Tatashin]

In order to support early time stamps we must know the vendor id of the
chip early in boot. This patch implements it by getting vendor string from
cpuid, and comparing it against the known to Linux x86 vendors.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/include/asm/processor.h |    1 +
 arch/x86/kernel/cpu/common.c     |   36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index f385eca..a4128d1 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -184,6 +184,7 @@ enum cpuid_regs_idx {
 
 extern void cpu_detect(struct cpuinfo_x86 *c);
 
+int get_x86_vendor_early(void);
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 58094a1..9c02851 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -847,6 +847,42 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	fpu__init_system(c);
 }
 
+/*
+ * Returns x86_vendor early, before other platform data structures being
+ * initialized.
+ */
+int get_x86_vendor_early(void)
+{
+	int x86_vendor = X86_VENDOR_UNKNOWN;
+	const struct cpu_dev *const *cdev;
+	char vendor_str[16];
+	int count = 0;
+
+	cpuid(0, (unsigned int *)&vendor_str[12],
+	      (unsigned int *)&vendor_str[0],
+	      (unsigned int *)&vendor_str[8],
+	      (unsigned int *)&vendor_str[4]);
+	vendor_str[12] = '\0';
+
+	for (cdev = __x86_cpu_dev_start; cdev < __x86_cpu_dev_end; cdev++) {
+		const struct cpu_dev *cpudev = *cdev;
+		const char *str1 = cpudev->c_ident[0];
+		const char *str2 = cpudev->c_ident[1];
+
+		if (count == X86_VENDOR_NUM)
+			break;
+
+		if ((!strcmp(vendor_str, str1)) ||
+		    (str2 && !strcmp(vendor_str, str2))) {
+			x86_vendor = cpudev->c_x86_vendor;
+			break;
+		}
+		count++;
+	}
+
+	return x86_vendor;
+}
+
 void __init early_cpu_init(void)
 {
 	const struct cpu_dev *const *cdev;
-- 
1.7.1

[v1 4/9] x86/tsc: early MSR-based CPU/TSC frequency discovery

[Pavel Tatashin]

Allow discovering MSR-based CPU/TSC frequency early in boot. This method
works only for some Intel CPUs.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/include/asm/tsc.h |    1 +
 arch/x86/kernel/tsc_msr.c  |   38 +++++++++++++++++++++++++-------------
 2 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/arch/x86/include/asm/tsc.h b/arch/x86/include/asm/tsc.h
index f5e6f1c..893de0c 100644
--- a/arch/x86/include/asm/tsc.h
+++ b/arch/x86/include/asm/tsc.h
@@ -61,6 +61,7 @@ static inline void check_tsc_sync_target(void) { }
 extern void tsc_save_sched_clock_state(void);
 extern void tsc_restore_sched_clock_state(void);
 
+unsigned long cpu_khz_from_msr_early(int vendor, int family, int model);
 unsigned long cpu_khz_from_msr(void);
 
 #endif /* _ASM_X86_TSC_H */
diff --git a/arch/x86/kernel/tsc_msr.c b/arch/x86/kernel/tsc_msr.c
index 19afdbd..6f5f617 100644
--- a/arch/x86/kernel/tsc_msr.c
+++ b/arch/x86/kernel/tsc_msr.c
@@ -62,22 +62,20 @@ static int match_cpu(u8 family, u8 model)
 #define id_to_freq(cpu_index, freq_id) \
 	(freq_desc_tables[cpu_index].freqs[freq_id])
 
-/*
- * MSR-based CPU/TSC frequency discovery for certain CPUs.
- *
- * Set global "lapic_timer_frequency" to bus_clock_cycles/jiffy
- * Return processor base frequency in KHz, or 0 on failure.
- */
-unsigned long cpu_khz_from_msr(void)
+ /*
+  * Get CPU/TSC frequency early in boot.
+  * Set global "lapic_timer_frequency" to bus_clock_cycles/jiffy
+  * Return processor base frequency in KHz, or 0 on failure.
+  */
+unsigned long cpu_khz_from_msr_early(int vendor, int family, int model)
 {
 	u32 lo, hi, ratio, freq_id, freq;
-	unsigned long res;
 	int cpu_index;
 
-	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+	if (vendor != X86_VENDOR_INTEL)
 		return 0;
 
-	cpu_index = match_cpu(boot_cpu_data.x86, boot_cpu_data.x86_model);
+	cpu_index = match_cpu(family, model);
 	if (cpu_index < 0)
 		return 0;
 
@@ -94,13 +92,27 @@ unsigned long cpu_khz_from_msr(void)
 	freq_id = lo & 0x7;
 	freq = id_to_freq(cpu_index, freq_id);
 
-	/* TSC frequency = maximum resolved freq * maximum resolved bus ratio */
-	res = freq * ratio;
-
 #ifdef CONFIG_X86_LOCAL_APIC
 	lapic_timer_frequency = (freq * 1000) / HZ;
 #endif
 
+	/* TSC frequency = maximum resolved freq * maximum resolved bus ratio */
+	return freq * ratio;
+}
+
+/*
+ * MSR-based CPU/TSC frequency discovery for certain CPUs.
+ */
+unsigned long cpu_khz_from_msr(void)
+{
+	unsigned long res;
+
+	res = cpu_khz_from_msr_early(boot_cpu_data.x86_vendor,
+				     boot_cpu_data.x86,
+				     boot_cpu_data.x86_model);
+	if (res ==  0)
+		return 0;
+
 	/*
 	 * TSC frequency determined by MSR is always considered "known"
 	 * because it is reported by HW.
-- 
1.7.1

[v1 5/9] x86/tsc: disable early messages from quick_pit_calibrate

[Pavel Tatashin]

quick_pit_calibrate() is another method that can determine the frequency
of TSC.  However, this function by default outputs some information
messages. Allow to disable these messages if this function is called early
so early time stamps can be initialized before anything is printed.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/kernel/tsc.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index c73a7f9..5add503 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -580,7 +580,7 @@ static inline int pit_expect_msb(unsigned char val, u64 *tscp, unsigned long *de
 #define MAX_QUICK_PIT_MS 50
 #define MAX_QUICK_PIT_ITERATIONS (MAX_QUICK_PIT_MS * PIT_TICK_RATE / 1000 / 256)
 
-static unsigned long quick_pit_calibrate(void)
+static unsigned long quick_pit_calibrate(bool early_boot)
 {
 	int i;
 	u64 tsc, delta;
@@ -645,7 +645,8 @@ static unsigned long quick_pit_calibrate(void)
 			goto success;
 		}
 	}
-	pr_info("Fast TSC calibration failed\n");
+	if (!early_boot)
+		pr_info("Fast TSC calibration failed\n");
 	return 0;
 
 success:
@@ -664,7 +665,8 @@ static unsigned long quick_pit_calibrate(void)
 	 */
 	delta *= PIT_TICK_RATE;
 	do_div(delta, i*256*1000);
-	pr_info("Fast TSC calibration using PIT\n");
+	if (!early_boot)
+		pr_info("Fast TSC calibration using PIT\n");
 	return delta;
 }
 
@@ -764,7 +766,7 @@ unsigned long native_calibrate_cpu(void)
 		return fast_calibrate;
 
 	local_irq_save(flags);
-	fast_calibrate = quick_pit_calibrate();
+	fast_calibrate = quick_pit_calibrate(false);
 	local_irq_restore(flags);
 	if (fast_calibrate)
 		return fast_calibrate;
-- 
1.7.1

[v1 6/9] x86/tsc: use cpuid to determine TSC frequency

[Pavel Tatashin]

One of the newer methods to determine TSC frequency, is to use one of cpuid
extensions to get TSC/Crystal ratio. This method is preferred on CPUs that
implements it. This patch adds a new function calibrate_tsc_early() that
can be called early in boot to determine the TSC by using this method.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/kernel/tsc.c |   39 ++++++++++++++++++++++++++-------------
 1 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index 5add503..1c9fc23 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -26,6 +26,9 @@
 #include <asm/apic.h>
 #include <asm/intel-family.h>
 
+/* CPUID 15H TSC/Crystal ratio, plus optionally Crystal Hz */
+#define CPUID_TSC_LEAF		0x15
+
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
 
@@ -671,24 +674,16 @@ static unsigned long quick_pit_calibrate(bool early_boot)
 }
 
 /**
- * native_calibrate_tsc
- * Determine TSC frequency via CPUID, else return 0.
+ * The caller already checked that TSC leaf capability can be read from cpuid
  */
-unsigned long native_calibrate_tsc(void)
+static unsigned long calibrate_tsc_early(int model)
 {
 	unsigned int eax_denominator, ebx_numerator, ecx_hz, edx;
 	unsigned int crystal_khz;
 
-	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
-		return 0;
-
-	if (boot_cpu_data.cpuid_level < 0x15)
-		return 0;
-
 	eax_denominator = ebx_numerator = ecx_hz = edx = 0;
 
-	/* CPUID 15H TSC/Crystal ratio, plus optionally Crystal Hz */
-	cpuid(0x15, &eax_denominator, &ebx_numerator, &ecx_hz, &edx);
+	cpuid(CPUID_TSC_LEAF, &eax_denominator, &ebx_numerator, &ecx_hz, &edx);
 
 	if (ebx_numerator == 0 || eax_denominator == 0)
 		return 0;
@@ -696,7 +691,7 @@ unsigned long native_calibrate_tsc(void)
 	crystal_khz = ecx_hz / 1000;
 
 	if (crystal_khz == 0) {
-		switch (boot_cpu_data.x86_model) {
+		switch (model) {
 		case INTEL_FAM6_SKYLAKE_MOBILE:
 		case INTEL_FAM6_SKYLAKE_DESKTOP:
 		case INTEL_FAM6_KABYLAKE_MOBILE:
@@ -713,6 +708,24 @@ unsigned long native_calibrate_tsc(void)
 		}
 	}
 
+	return crystal_khz * ebx_numerator / eax_denominator;
+}
+
+/**
+ * native_calibrate_tsc
+ * Determine TSC frequency via CPUID, else return 0.
+ */
+unsigned long native_calibrate_tsc(void)
+{
+	unsigned int tsc_khz;
+
+	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+		return 0;
+
+	if (boot_cpu_data.cpuid_level < CPUID_TSC_LEAF)
+		return 0;
+
+	tsc_khz = calibrate_tsc_early(boot_cpu_data.x86_model);
 	/*
 	 * TSC frequency determined by CPUID is a "hardware reported"
 	 * frequency and is the most accurate one so far we have. This
@@ -727,7 +740,7 @@ unsigned long native_calibrate_tsc(void)
 	if (boot_cpu_data.x86_model == INTEL_FAM6_ATOM_GOLDMONT)
 		setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);
 
-	return crystal_khz * ebx_numerator / eax_denominator;
+	return tsc_khz;
 }
 
 static unsigned long cpu_khz_from_cpuid(void)
-- 
1.7.1

[v1 7/9] x86/tsc: use cpuid to determine CPU frequency

[Pavel Tatashin]

Newer processors implement cpuid extension to determine CPU frequency
from cpuid. This patch adds a function that can do this early in boot.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/kernel/tsc.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index 1c9fc23..58bd575 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -743,14 +743,14 @@ unsigned long native_calibrate_tsc(void)
 	return tsc_khz;
 }
 
-static unsigned long cpu_khz_from_cpuid(void)
+static unsigned long cpu_khz_from_cpuid_early(int vendor, int cpuid_level)
 {
 	unsigned int eax_base_mhz, ebx_max_mhz, ecx_bus_mhz, edx;
 
-	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+	if (vendor != X86_VENDOR_INTEL)
 		return 0;
 
-	if (boot_cpu_data.cpuid_level < 0x16)
+	if (cpuid_level < 0x16)
 		return 0;
 
 	eax_base_mhz = ebx_max_mhz = ecx_bus_mhz = edx = 0;
@@ -770,7 +770,9 @@ unsigned long native_calibrate_cpu(void)
 	unsigned long flags, latch, ms, fast_calibrate;
 	int hpet = is_hpet_enabled(), i, loopmin;
 
-	fast_calibrate = cpu_khz_from_cpuid();
+	fast_calibrate = cpu_khz_from_cpuid_early(boot_cpu_data.x86_vendor,
+						  boot_cpu_data.cpuid_level);
+
 	if (fast_calibrate)
 		return fast_calibrate;
 
-- 
1.7.1

[v1 8/9] x86/tsc: tsc early

[Pavel Tatashin]

tsc_early_init():
Use verious methods to determine the availability of TSC feature and its
frequency early in boot, and if that is possible initialize TSC and also
call sched_clock_early_init() to be able to get timestamps early in boot.

tsc_early_fini()
Implement the finish part of early tsc feature, print message about the
offset, which can be useful to findout how much time was spent in post and
boot manager, and also call sched_clock_early_fini() to let sched clock
know that

sched_clock_early():
TSC based implementation of weak function that is defined in sched clock.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/include/asm/tsc.h |    4 ++
 arch/x86/kernel/tsc.c      |  107 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 111 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/tsc.h b/arch/x86/include/asm/tsc.h
index 893de0c..a8c7f2e 100644
--- a/arch/x86/include/asm/tsc.h
+++ b/arch/x86/include/asm/tsc.h
@@ -50,11 +50,15 @@ static inline cycles_t get_cycles(void)
 extern void tsc_verify_tsc_adjust(bool resume);
 extern void check_tsc_sync_source(int cpu);
 extern void check_tsc_sync_target(void);
+void tsc_early_init(void);
+void tsc_early_fini(void);
 #else
 static inline bool tsc_store_and_check_tsc_adjust(bool bootcpu) { return false; }
 static inline void tsc_verify_tsc_adjust(bool resume) { }
 static inline void check_tsc_sync_source(int cpu) { }
 static inline void check_tsc_sync_target(void) { }
+static inline void tsc_early_init(void) { }
+static inline void tsc_early_fini(void) { }
 #endif
 
 extern int notsc_setup(char *);
diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index 58bd575..67ecddf 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -25,10 +25,13 @@
 #include <asm/geode.h>
 #include <asm/apic.h>
 #include <asm/intel-family.h>
+#include <asm/cpu.h>
 
 /* CPUID 15H TSC/Crystal ratio, plus optionally Crystal Hz */
 #define CPUID_TSC_LEAF		0x15
 
+static struct cyc2ns_data __read_mostly cyc2ns_early;
+
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
 
@@ -290,6 +293,16 @@ static void set_cyc2ns_scale(unsigned long khz, int cpu)
 	sched_clock_idle_wakeup_event(0);
 	local_irq_restore(flags);
 }
+
+u64 sched_clock_early(void)
+{
+	u64 ns;
+
+	ns = mul_u64_u32_shr(rdtsc(), cyc2ns_early.cyc2ns_mul,
+			     cyc2ns_early.cyc2ns_shift);
+	return ns + cyc2ns_early.cyc2ns_offset;
+}
+
 /*
  * Scheduler clock - returns current time in nanosec units.
  */
@@ -1365,6 +1378,100 @@ static int __init init_tsc_clocksource(void)
  */
 device_initcall(init_tsc_clocksource);
 
+#ifdef CONFIG_X86_TSC
+
+/* Determine if tsc is invariant early in boot */
+static bool __init tsc_invariant_early(void)
+{
+	unsigned int ext_cpuid_level, tsc_flag;
+
+	/* Get extended CPUID level */
+	ext_cpuid_level = cpuid_eax(0x80000000);
+	if (ext_cpuid_level < 0x80000007)
+		return false;
+
+	/* get field with invariant TSC flag */
+	tsc_flag = cpuid_edx(0x80000007);
+	if (!(tsc_flag & (1 << 8)))
+		return false;
+
+	return true;
+}
+
+/*
+ * Determine if we can use TSC early in boot. On larger machines early boot can
+ * take a significant amount of time, therefore, for observability reasons, and
+ * also to avoid regressions it is important to have timestamps during the whole
+ * boot process.
+ */
+void __init tsc_early_init(void)
+{
+	int vendor, model, family, cpuid_level;
+	unsigned int sig, khz;
+	u64 tsc_now;
+
+	/*
+	 * Should we disable early timestamps on platforms without invariant
+	 * TSC?
+	 *
+	 * On the one hand invariant TSC guarantees, that early timestamps run
+	 * only on the latest hardware (Nehalem and later), but on the other
+	 * hand accuracy wise, non-invariant timestamps should be OK,
+	 * because during early boot power management features are not used.
+	 * ---
+	 * For now we disable invariant TSC for early boot.
+	 */
+	if (!tsc_invariant_early())
+		return;
+
+	cpuid_level = cpuid_eax(0);
+	sig = cpuid_eax(1);
+	model = x86_model(sig);
+	family = x86_family(sig);
+	vendor = get_x86_vendor_early();
+
+	/*
+	 * Try several methods to get TSC frequency, if fail, return false
+	 * otherwise setup mult and shift values to convert ticks to nanoseconds
+	 * efficiently.
+	 */
+	khz = 0;
+	if (vendor == X86_VENDOR_INTEL && cpuid_level >= CPUID_TSC_LEAF)
+		khz = calibrate_tsc_early(model);
+
+	if (khz == 0)
+		khz = cpu_khz_from_cpuid_early(vendor, cpuid_level);
+
+	if (khz == 0)
+		khz = cpu_khz_from_msr_early(vendor, family, model);
+
+	if (khz == 0)
+		khz = quick_pit_calibrate(true);
+
+	if (khz == 0)
+		return;
+
+	tsc_now = rdtsc();
+	clocks_calc_mult_shift(&cyc2ns_early.cyc2ns_mul,
+			       &cyc2ns_early.cyc2ns_shift,
+			       khz, NSEC_PER_MSEC, 0);
+	cyc2ns_early.cyc2ns_offset = -sched_clock_early();
+	sched_clock_early_init();
+}
+
+void __init tsc_early_fini(void)
+{
+	/* We did not have early sched clock if multiplier is 0 */
+	if (cyc2ns_early.cyc2ns_mul == 0)
+		return;
+
+	sched_clock_early_fini();
+	pr_info("sched clock early is finished, offset [%lld.%09llds]\n",
+		-cyc2ns_early.cyc2ns_offset / NSEC_PER_SEC,
+		-cyc2ns_early.cyc2ns_offset % NSEC_PER_SEC);
+}
+#endif /* CONFIG_X86_TSC */
+
 void __init tsc_init(void)
 {
 	u64 lpj;
-- 
1.7.1

Re: [v1 8/9] x86/tsc: tsc early

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

Hi Pavel,

[auto build test ERROR on tip/sched/core]
[also build test ERROR on v4.11-rc3 next-20170323]
[cannot apply to tip/x86/core]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Pavel-Tatashin/Early-boot-time-stamps-for-x86/20170324-114338
config: i386-randconfig-s0-201712 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   arch/x86/built-in.o: In function `tsc_early_fini':
>> (.init.text+0x8665): undefined reference to `__umoddi3'
   arch/x86/built-in.o: In function `tsc_early_fini':
>> (.init.text+0x8679): undefined reference to `__udivdi3'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 24315 bytes --]

[v1 9/9] x86/tsc: use tsc early

[Pavel Tatashin]

Call tsc_early_init() to initialize early boot time stamps functionality on
the supported x86 platforms, and call tsc_early_fini() to finish this
feature after permanent tsc has been initialized.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
---
 arch/x86/kernel/head64.c |    1 +
 arch/x86/kernel/time.c   |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index b5785c1..1068a56 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -157,6 +157,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 	clear_bss();
 
 	clear_page(init_level4_pgt);
+	tsc_early_init();
 
 	kasan_early_init();
 
diff --git a/arch/x86/kernel/time.c b/arch/x86/kernel/time.c
index d39c091..2d691eb 100644
--- a/arch/x86/kernel/time.c
+++ b/arch/x86/kernel/time.c
@@ -85,6 +85,7 @@ static __init void x86_late_time_init(void)
 {
 	x86_init.timers.timer_init();
 	tsc_init();
+	tsc_early_fini();
 }
 
 /*
-- 
1.7.1

Re: [v1 0/9] Early boot time stamps for x86

[Peter Zijlstra]

On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote:
> Last week I sent out patches to enable early boot time stamp on SPARC, that
> work is now under review:
> http://www.spinics.net/lists/sparclinux/msg17372.html
> 
> This is continuation for that work, but adding early boot time stamps
> support for x86 machines.

*groan*, so we get an even bigger trainwreck when TSC is fscked?

Re: [v1 0/9] Early boot time stamps for x86

[Pasha Tatashin]

On 2017-03-22 16:27, Peter Zijlstra wrote:
> On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote:
>> Last week I sent out patches to enable early boot time stamp on SPARC, that
>> work is now under review:
>> http://www.spinics.net/lists/sparclinux/msg17372.html
>>
>> This is continuation for that work, but adding early boot time stamps
>> support for x86 machines.
>
> *groan*, so we get an even bigger trainwreck when TSC is fscked?
>

Hi Peter,

I am not sure what you mean. If tsc is not reliable, the worst case is 
that during early boot the timestamps won't be as accurate as the 
timestamps that are enabled once we initialize some other clock sources.

In either way, if you look at the way I enabled early timestamps, it is 
really narrowed down to work only on modern hardware with stable tsc. 
(Nehalem and later)

Thank you,
Pasha

Re: [v1 0/9] Early boot time stamps for x86

[Peter Zijlstra]

On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote:
> - early tsc offset is 549s, so it took over 9 minutes to get through POST,
>   and GRUB before starting linux

Lol, how cute. You assume TSC starts at 0 on reset.

Re: [v1 0/9] Early boot time stamps for x86

[Pasha Tatashin]

Hi Peter,

Thank you for looking at this patchset.

Yes, I am certain it is 0 or  near 0 on reset on this machine. Because, 
I actually wondered about it, and used stop watch as an alternative way 
to verify the result, twice.

While, I suspect it is often the case that on reset tsc is 0, it is not 
really import, as the offset value is not the important part of the 
project. The important part is that we can cover the whole Linux boot 
process with timestamps, and know where spend time, and avoid 
regressions in the future.

Thank you,
Pasha

On 2017-03-22 16:28, Peter Zijlstra wrote:
> On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote:
>> - early tsc offset is 549s, so it took over 9 minutes to get through POST,
>>   and GRUB before starting linux
>
> Lol, how cute. You assume TSC starts at 0 on reset.
>

Re: [v1 0/9] Early boot time stamps for x86

[Thomas Gleixner]

On Wed, 22 Mar 2017, Pasha Tatashin wrote:
> Yes, I am certain it is 0 or  near 0 on reset on this machine. Because, I

Emphasis on "this machine'

It's not guaranteed especially not on reboot and not with creative BIOSes
fiddling with the TSC_ADJUST value. 

 - It CANNOT be used to measure BIOS boot time reliably

 - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the
   window once the kernel sanitized it.

See other mail.

Thanks,

	tglx

Re: [v1 0/9] Early boot time stamps for x86

[Pasha Tatashin]

Hi Thomas,

Thank you very much for looking at this patchset. Comments below:

On 03/23/2017 06:56 AM, Thomas Gleixner wrote:
> On Wed, 22 Mar 2017, Pasha Tatashin wrote:
>> Yes, I am certain it is 0 or  near 0 on reset on this machine. Because, I
>
> Emphasis on "this machine'
>
> It's not guaranteed especially not on reboot and not with creative BIOSes
> fiddling with the TSC_ADJUST value.
>
>  - It CANNOT be used to measure BIOS boot time reliably

Yes, understood, I will remove comment about BIOS time from the next 
cover letter.

However, I think the pr_info() with offset is still useful at least for 
those whose BIOS does not alter TSC_ADJUST, also it is consisten with 
every other clocksource in linux where offset is printed in pr_info().

 From Intel PRM 2016/12:
The time-stamp counter (as implemented in the P6 family, Pentium, 
Pentium M, Pentium 4, Intel Xeon, Intel Core Solo and Intel Core Duo 
processors and later processors) is a 64-bit counter that is set to 0 
following a RESET of the processor

Since early boot time stamps feature target processors that are later 
than "Pentium 4" because invariant TSC flag is checked, it is safe to 
assume that offset is going to be valid on power-on if TSC_ADJUST  was 
not altered

>
>  - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the
>    window once the kernel sanitized it.

I will add a condition to tsc_early_init() to check for TSC_ADJUST if it 
is not 0, disable early TSC feature. Does this sound OK?

Thank you,
Pasha

Re: [v1 0/9] Early boot time stamps for x86

[Thomas Gleixner]

On Thu, 23 Mar 2017, Pasha Tatashin wrote:
> I will add a condition to tsc_early_init() to check for TSC_ADJUST if it is
> not 0, disable early TSC feature. Does this sound OK?

Not really.

I have strong objections against how this is crammed into the early boot
process along with the code duplication and the extra magic which is
caused by this.

The early boot process is fragile enough, so we really only want to have
code there which is absolutely required. That timestamp feature does not
qualify for that at all.

I need some quiet time to look into that, so please don't waste too much
time on refactoring that patch set.

Thanks,

	tglx