Re: [PATCH i-g-t v2] tests/kms_setmode: Relax vblank timing tolerance from 1 scanline to 1%

["Ville Syrjälä" <ville.syrjala@linux.intel.com>]

On Thu, Apr 16, 2026 at 10:45:28PM +0800, Jason-JH Lin wrote:
> The original check_timings() validation required vblank intervals to be
> within 1 scanline (~15us for 1080p60) which is overly strict and causes
> false failures due to userspace measurement overhead.
> 
> Changes:
> - Keep original strict checks (1 scanline, 1.718 sigma) as warnings only
> - Add relaxed validation with 1% relative tolerance for pass/fail
> - Use 3 sigma threshold (99.7% confidence) instead of 1.718 sigma (90%)
> 
> Rationale:
> Hardware specs (VESA/HDMI/DP) allow ±0.5% pixel clock tolerance, which
> translates to ±0.5% frame time due to inverse relationship. However,
> userspace measurement via read() syscall adds ~0.3-0.5% overhead from:

read() overhead is irrelevant here. This is measuring the accuracy
of the kernel generated vblank event timestamps. If you are having
problems with this then that would indicate your kernel driver isn't
generating particularly good timestamps, or it's using a dotclock that
doesn't quite match what userspace specified originally.

> - Interrupt latency and scheduling delays (5-50 us)
> - System call overhead (1-5 us)
> - Variable delays depending on system load
> 
> Total tolerance: ±0.5% (hardware) + ~0.5% (software) = ±1.0%
> 
> This makes the test ~11x more tolerant while still detecting real timing
> issues and aligning with industry hardware specifications.
> 
> Signed-off-by: Jason-JH Lin <jason-jh.lin@mediatek.com>
> Suggested-by: Jeevain B <jeevan.b@intel.com>
> ---
>  tests/kms_setmode.c | 71 +++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 59 insertions(+), 12 deletions(-)
> 
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c
> index 1f2849bc26cf..b452f8a9e997 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -495,6 +495,7 @@ static bool check_timings(int crtc_idx, const drmModeModeInfo *kmode)
>  	double mean;
>  	double stddev;
>  	int n;
> +	bool strict_failed = false;
>  
>  	memset(&wait, 0, sizeof(wait));
>  	wait.request.type = kmstest_get_vbl_flag(crtc_idx);
> @@ -563,10 +564,9 @@ static bool check_timings(int crtc_idx, const drmModeModeInfo *kmode)
>  
>  	/* 99.7% samples within one scanline on each side of mean */
>  	if (accuracy >= line_time(kmode)) {
> -		igt_info("vblank accuracy (%.3fus, %.1f%%) worse than a scanline (%.3fus)\n",
> -			  accuracy, 100 * accuracy / mean, line_time(kmode));
> -
> -		return false;
> +		igt_warn("vblank accuracy (%.3fus, %.1f%%) worse than a scanline (%.3fus)\n",
> +			 accuracy, 100 * accuracy / mean, line_time(kmode));
> +		strict_failed = true;
>  	}
>  
>  	/* At least 90% of frame times fall within the one scanline on each
> @@ -591,14 +591,61 @@ static bool check_timings(int crtc_idx, const drmModeModeInfo *kmode)
>  	 * https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data
>  	 */
>  	if (fabs(mean - expected) >= max(line_time(kmode), 1.718 * stddev)) {
> -		igt_info("vblank interval differs from modeline! expected %.1fus, "
> -			 "measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma, %.1f scanlines)\n",
> -			  expected, mean, stddev,
> -			  fabs(mean - expected),
> -			  fabs(mean - expected) / stddev,
> -			  fabs(mean - expected) / line_time(kmode));
> -
> -		return false;
> +		igt_warn("vblank interval differs from modeline! expected %.1fus, "
> +			 "measured %1.fus +- %.3fus, difference %.1fus "
> +			 "(%.1f sigma, %.1f scanlines)\n",
> +			 expected, mean, stddev,
> +			 fabs(mean - expected),
> +			 fabs(mean - expected) / stddev,
> +			 fabs(mean - expected) / line_time(kmode));
> +		strict_failed = true;
> +	}
> +
> +	if (strict_failed) {
> +		/* Relaxed validation: 1% relative tolerance for accuracy and deviation
> +		 *
> +		 * Hardware specifications (VESA/HDMI/DP) allow ±0.5% pixel clock tolerance.
> +		 * Since frame_time = (htotal × vtotal) / pixel_clock, the relationship is
> +		 * inverse: ±0.5% pixel clock → ±0.5% frame time (in opposite direction).
> +		 *
> +		 * However, userspace measurement introduces additional overhead (~0.3-0.5%):
> +		 *   - Interrupt latency and scheduling delays (5-50 μs)
> +		 *   - System call overhead for read() (1-5 μs)
> +		 *   - Variable delays depending on system load
> +		 *
> +		 * Total tolerance: ±0.5% (hardware) + ~0.5% (software) ≈ ±1.0%
> +		 *
> +		 * For stricter validation matching hardware spec only, use 0.005 (0.5%),
> +		 * but expect potential false failures due to measurement noise.
> +		 */
> +		double tolerance_pct = 0.01;  /* 1% tolerance */
> +		double tolerance_us = expected * tolerance_pct;
> +		double relative_accuracy = accuracy / expected;
> +
> +		/* Check 1: 99.7% samples within 1% of the mean */
> +		if (accuracy >= tolerance_us) {
> +			igt_info("FAIL: vblank accuracy (%.3fus, %.2f%%) exceeds 1%% tolerance "
> +				 "(%.3fus)\n",
> +				 accuracy, 100 * relative_accuracy, tolerance_us);
> +			return false;
> +		}
> +
> +		/* Check 2: Mean within 1% of expected, or within 3 sigma (99.7% confidence) */
> +		if (fabs(mean - expected) >= max(tolerance_us, 3.0 * stddev)) {
> +			igt_info("FAIL: vblank interval differs from expected! expected %.1fus, "
> +				 "measured %.1fus +- %.3fus, difference %.1fus "
> +				 "(%.1f sigma, %.2f%%), "
> +				 "exceeds 1%% tolerance or 3-sigma threshold\n",
> +				 expected, mean, stddev,
> +				 fabs(mean - expected),
> +				 fabs(mean - expected) / stddev,
> +				 100 * fabs(mean - expected) / expected);
> +			return false;
> +		}
> +
> +		igt_info("Passed relaxed validation (strict failed)\n");
> +	} else {
> +		igt_info("Passed strict validation\n");
>  	}
>  
>  	return true;
> -- 
> 2.43.0

-- 
Ville Syrjälä
Intel