* Re: [PATCH] timekeeping: Align tick_sched_timer() with the HZ tick. -- regression report
2023-06-13 22:59 ` [PATCH] timekeeping: Align tick_sched_timer() with the HZ tick. -- regression report Mathias Krause
@ 2023-06-15 9:03 ` Richard W.M. Jones
2023-06-15 11:40 ` Mathias Krause
2023-06-15 9:22 ` Sebastian Andrzej Siewior
2023-06-16 18:53 ` [tip: timers/urgent] tick/common: Align tick period during sched_timer setup tip-bot2 for Thomas Gleixner
2 siblings, 1 reply; 5+ messages in thread
From: Richard W.M. Jones @ 2023-06-15 9:03 UTC (permalink / raw)
To: Mathias Krause
Cc: Sebastian Andrzej Siewior, Thomas Gleixner, Peter Zijlstra,
linux-kernel, Ingo Molnar, Gusenleitner Klaus,
Frederic Weisbecker, stable
[-- Attachment #1: Type: text/plain, Size: 690 bytes --]
On Wed, Jun 14, 2023 at 12:59:46AM +0200, Mathias Krause wrote:
> This patch causes VM boot hangs for us. It took a while to identify as
> the boot hangs were only ~1 out of 30 but it's clearly it. Reverting
> the commit got me 100 boots in a row without any issue.
FWIW I have quite a nice test program for catching these sorts of boot
hangs, see attached. You need to change the VMLINUX define to point
to your vmlinux or vmlinuz file.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
[-- Attachment #2: bootbootboot.c --]
[-- Type: text/plain, Size: 3144 bytes --]
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <error.h>
#include <unistd.h>
#include <sys/wait.h>
#include <pthread.h>
#define NR_ITERATIONS 10000
#define MAX_THREADS 8
#define MAX_TIME 60 /* max time to wait for qemu to complete */
//#define VMLINUX "/home/rjones/d/linux/vmlinux"
#define VMLINUX "vmlinux"
//#define QEMU "/home/rjones/d/qemu/build/qemu-system-x86_64"
#define QEMU "qemu-system-x86_64"
#define QEMU_COMMAND \
QEMU " -no-user-config -nodefaults -display none " \
"-machine accel=kvm:tcg,graphics=off -cpu max,la57=off -m 1280 " \
"-no-reboot " \
"-rtc driftfix=slew -no-hpet -global kvm-pit.lost_tick_policy=discard " \
"-kernel " VMLINUX " " \
"-serial stdio " \
"-append \"panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 selinux=0 TERM=xterm-256color\""
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
static unsigned iterations;
static void *start_thread (void *);
int
main (int argc, char *argv[])
{
long n, i;
int r;
pthread_t thread[MAX_THREADS];
n = sysconf (_SC_NPROCESSORS_ONLN);
if (n == -1) error (EXIT_FAILURE, errno, "sysconf");
if (n > MAX_THREADS)
n = MAX_THREADS;
for (i = 0; i < n; ++i) {
r = pthread_create (&thread[i], NULL, start_thread, NULL);
if (r != 0) error (EXIT_FAILURE, r, "pthread_create");
}
for (i = 0; i < n; ++i) {
r = pthread_join (thread[i], NULL);
if (r != 0) error (EXIT_FAILURE, r, "pthread_join");
}
printf ("\n");
printf ("\n");
printf ("test ok\n");
exit (EXIT_SUCCESS);
}
static void *
start_thread (void *vp)
{
pid_t pid;
char tmp[] = "/tmp/kernel.out.XXXXXX";
char cmd[1024];
int i, r, status;
if (mkstemp (tmp) == -1)
error (EXIT_FAILURE, errno, "mkstemp: %s", tmp);
snprintf (cmd, sizeof cmd, QEMU_COMMAND " >& %s", tmp);
/* This basically runs a loop starting qemu. */
for (;;) {
pthread_mutex_lock (&lock);
if (iterations >= NR_ITERATIONS) {
pthread_mutex_unlock (&lock);
return NULL;
}
if (iterations <= MAX_THREADS) { // stagger the start times
pthread_mutex_unlock (&lock);
usleep (rand () % 3000000);
pthread_mutex_lock (&lock);
}
iterations++;
printf ("%d... ", iterations); fflush (stdout);
pthread_mutex_unlock (&lock);
pid = fork ();
if (pid == -1) error (EXIT_FAILURE, errno, "fork");
if (pid == 0) {
/* Child process, run qemu and wait. */
if (system (cmd) != 0)
_exit (EXIT_FAILURE);
else
_exit (EXIT_SUCCESS);
}
/* In the parent wait up to MAX_TIME seconds. */
for (i = 0; i < MAX_TIME; ++i) {
r = waitpid (pid, &status, WNOHANG);
if (r == -1) error (EXIT_FAILURE, errno, "waitpid");
if (r > 0) break;
sleep (1);
}
if (i == MAX_TIME || status != 0) {
/* Something failed in qemu (or it didn't
* exit), dump the whole log and exit with
* error.
*/
printf ("\n");
printf ("\n");
snprintf (cmd, sizeof cmd, "tail -20 %s", tmp);
system (cmd);
fprintf (stderr, "*** ERROR OR HANG ***\n");
exit (EXIT_FAILURE);
}
}
unlink (tmp);
}
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] timekeeping: Align tick_sched_timer() with the HZ tick. -- regression report
2023-06-13 22:59 ` [PATCH] timekeeping: Align tick_sched_timer() with the HZ tick. -- regression report Mathias Krause
2023-06-15 9:03 ` Richard W.M. Jones
@ 2023-06-15 9:22 ` Sebastian Andrzej Siewior
2023-06-16 18:53 ` [tip: timers/urgent] tick/common: Align tick period during sched_timer setup tip-bot2 for Thomas Gleixner
2 siblings, 0 replies; 5+ messages in thread
From: Sebastian Andrzej Siewior @ 2023-06-15 9:22 UTC (permalink / raw)
To: Mathias Krause
Cc: Thomas Gleixner, Peter Zijlstra, linux-kernel, Ingo Molnar,
Gusenleitner Klaus, Frederic Weisbecker, stable
On 2023-06-14 00:59:46 [+0200], Mathias Krause wrote:
> Hi Sebastian,
Hi Mathias,
> this change doesn't seem to achieve that goal, unfortunately. Quite the
> opposite. It makes the (boot) clock run faster and, because of the per-
> boot different offset, differently fast for each boot. Up to the point
> where it's running too fast to make any progress at all.
I meant to reply to this thread but took the other by accident. Sorry.
This should be addressed by
https://lore.kernel.org/20230615091830.RxMV2xf_@linutronix.de
Thank you for the report.
> Thanks,
> Mathias
Sebastian
^ permalink raw reply [flat|nested] 5+ messages in thread
* [tip: timers/urgent] tick/common: Align tick period during sched_timer setup
2023-06-13 22:59 ` [PATCH] timekeeping: Align tick_sched_timer() with the HZ tick. -- regression report Mathias Krause
2023-06-15 9:03 ` Richard W.M. Jones
2023-06-15 9:22 ` Sebastian Andrzej Siewior
@ 2023-06-16 18:53 ` tip-bot2 for Thomas Gleixner
2 siblings, 0 replies; 5+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2023-06-16 18:53 UTC (permalink / raw)
To: linux-tip-commits
Cc: Mathias Krause, Bhatnagar, Rishabh, Thomas Gleixner,
Sebastian Andrzej Siewior, Richard W.M. Jones, SeongJae Park,
stable, x86, linux-kernel
The following commit has been merged into the timers/urgent branch of tip:
Commit-ID: 13bb06f8dd42071cb9a49f6e21099eea05d4b856
Gitweb: https://git.kernel.org/tip/13bb06f8dd42071cb9a49f6e21099eea05d4b856
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Thu, 15 Jun 2023 11:18:30 +02:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Fri, 16 Jun 2023 20:45:28 +02:00
tick/common: Align tick period during sched_timer setup
The tick period is aligned very early while the first clock_event_device is
registered. At that point the system runs in periodic mode and switches
later to one-shot mode if possible.
The next wake-up event is programmed based on the aligned value
(tick_next_period) but the delta value, that is used to program the
clock_event_device, is computed based on ktime_get().
With the subtracted offset, the device fires earlier than the exact time
frame. With a large enough offset the system programs the timer for the
next wake-up and the remaining time left is too small to make any boot
progress. The system hangs.
Move the alignment later to the setup of tick_sched timer. At this point
the system switches to oneshot mode and a high resolution clocksource is
available. At this point it is safe to align tick_next_period because
ktime_get() will now return accurate (not jiffies based) time.
[bigeasy: Patch description + testing].
Fixes: e9523a0d81899 ("tick/common: Align tick period with the HZ tick.")
Reported-by: Mathias Krause <minipli@grsecurity.net>
Reported-by: "Bhatnagar, Rishabh" <risbhat@amazon.com>
Suggested-by: Mathias Krause <minipli@grsecurity.net>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Tested-by: Mathias Krause <minipli@grsecurity.net>
Acked-by: SeongJae Park <sj@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/5a56290d-806e-b9a5-f37c-f21958b5a8c0@grsecurity.net
Link: https://lore.kernel.org/12c6f9a3-d087-b824-0d05-0d18c9bc1bf3@amazon.com
Link: https://lore.kernel.org/r/20230615091830.RxMV2xf_@linutronix.de
---
kernel/time/tick-common.c | 13 +------------
kernel/time/tick-sched.c | 13 ++++++++++++-
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index 65b8658..e9138cd 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -218,19 +218,8 @@ static void tick_setup_device(struct tick_device *td,
* this cpu:
*/
if (tick_do_timer_cpu == TICK_DO_TIMER_BOOT) {
- ktime_t next_p;
- u32 rem;
-
tick_do_timer_cpu = cpu;
-
- next_p = ktime_get();
- div_u64_rem(next_p, TICK_NSEC, &rem);
- if (rem) {
- next_p -= rem;
- next_p += TICK_NSEC;
- }
-
- tick_next_period = next_p;
+ tick_next_period = ktime_get();
#ifdef CONFIG_NO_HZ_FULL
/*
* The boot CPU may be nohz_full, in which case set
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 5225467..42c0be3 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -161,8 +161,19 @@ static ktime_t tick_init_jiffy_update(void)
raw_spin_lock(&jiffies_lock);
write_seqcount_begin(&jiffies_seq);
/* Did we start the jiffies update yet ? */
- if (last_jiffies_update == 0)
+ if (last_jiffies_update == 0) {
+ u32 rem;
+
+ /*
+ * Ensure that the tick is aligned to a multiple of
+ * TICK_NSEC.
+ */
+ div_u64_rem(tick_next_period, TICK_NSEC, &rem);
+ if (rem)
+ tick_next_period += TICK_NSEC - rem;
+
last_jiffies_update = tick_next_period;
+ }
period = last_jiffies_update;
write_seqcount_end(&jiffies_seq);
raw_spin_unlock(&jiffies_lock);
^ permalink raw reply related [flat|nested] 5+ messages in thread