[PATCH] io_uring/wait: don't sleep on an already expired timeout

[Ioannis Coward <ioanniscrd@gmail.com>]

When io_cqring_wait() gets to the point of sleeping and the wait's timeout
has already passed, io_cqring_schedule_timeout() still arms an hrtimer
for it and calls schedule(). The timer fires right away, but only after
the task has switched to idle, so getting -ETIME back costs a round trip
through idle plus a timer interrupt.

NAPI busy polling hits this on every wait that times out.
__io_napi_busy_loop() caps the busy poll at the remaining timeout, so when
the loop runs to its end without a completion, the timeout has expired by
the time we get here. The task then stops polling for several usec, and
packets that arrive meanwhile are left to the device interrupt. This
affects one of our production workloads, which currently works around it
by having another thread signal an eventfd on the ring before the timeout
expires.

A wait with a zero timeout, or one that runs out before the task gets to
sleep, pays the same cost. io_uring_wait_cqe_timeout() with nothing to
reap and a zero timeout takes ~11 usec and a voluntary context switch on
a Zen 5 desktop.

Return -ETIME without arming the timer once the timeout has passed, as
ep_poll() already does with ep_schedule_timeout(). A passed timeout ends
the wait with -ETIME whether or not a min_timeout is set, so the check
covers both. It costs one io_get_time() per wait that is about to sleep
with a timeout.

Signed-off-by: Ioannis Coward <ioanniscrd@gmail.com>
---
Compile-tested only. The zero timeout case can be reproduced with:

	// io_uring_wait_cqe_timeout() with nothing to reap and a timeout that has already
	// expired (here zero) should return -ETIME at once. Count how often it sleeps instead.
	// gcc -O2 expired_wait.c -o expired_wait -luring
	#define _GNU_SOURCE
	#include <liburing.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <errno.h>
	#include <time.h>
	#include <sys/resource.h>

	int main(int argc, char **argv)
	{
		int n = argc > 1 ? atoi(argv[1]) : 100000;
		struct __kernel_timespec ts = { 0 };
		struct io_uring_cqe *cqe;
		struct io_uring ring;
		struct rusage r0, r1;
		struct timespec t0, t1;

		if (io_uring_queue_init(8, &ring, 0))
			return 1;
		getrusage(RUSAGE_THREAD, &r0);
		clock_gettime(CLOCK_MONOTONIC, &t0);
		for (int i = 0; i < n; i++)
			if (io_uring_wait_cqe_timeout(&ring, &cqe, &ts) != -ETIME)
				return 1;
		clock_gettime(CLOCK_MONOTONIC, &t1);
		getrusage(RUSAGE_THREAD, &r1);
		printf("%.2f us per wait, %.3f voluntary context switches per wait\n",
		       ((t1.tv_sec - t0.tv_sec) * 1e9 + t1.tv_nsec - t0.tv_nsec) / n / 1e3,
		       (double)(r1.ru_nvcsw - r0.ru_nvcsw) / n);
		return 0;
	}

 io_uring/wait.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/io_uring/wait.c b/io_uring/wait.c
index fb1c08299..edc589302 100644
--- a/io_uring/wait.c
+++ b/io_uring/wait.c
@@ -165,6 +165,16 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx,
 {
 	int ret = 0;
 
+	/*
+	 * Don't arm a timer for a timeout that has already passed: it would
+	 * fire right away, but only after we've gone to sleep, costing an idle
+	 * round trip and a timer interrupt just to return -ETIME. NAPI busy
+	 * polling gets here like this every time it spins up to the timeout.
+	 */
+	if (iowq->timeout != KTIME_MAX &&
+	    ktime_compare(io_get_time(ctx), iowq->timeout) >= 0)
+		return -ETIME;
+
 	/*
 	 * Mark us as being in io_wait if we have pending requests, so cpufreq
 	 * can take into account that the task is waiting for IO - turns out
-- 
2.54.0