[PATCH v4] hwlatdetect: Add MTBF calculation

[PATCH v4] hwlatdetect: Add MTBF calculation

[Costa Shulyupin]

Hwlatdetect reports the number of latency spikes but provides no
information about their frequency distribution over time. This makes it
difficult to compare results - a test with 10 spikes over 1 hour is very
different from 10 spikes over 24 hours, but both show 'spikes = 10'.

Add Mean Time Between Failures (MTBF) calculation to quantify spike
frequency.

By definition MTBF = total operating time / number of failures.

When the failure interval is large relative to test duration, this
formula is biased.  For example, imagine stable periodic failures.  The
total operating time will include time before the first failure and
after the last failure.  These intervals are determined by when the test
starts and stops, not by the system's failure behavior, which adds
measurement bias.  The resulting MTBF will vary between runs even for
stable periodic failures.

To reduce this bias, calculate MTBF using only the time between the
first and last failure, divided by the number of intervals (failures
minus one).

Additionally, hwlatdetect only samples during the 'width' period within
each 'window' cycle. The non-sampling periods contribute to the error
margin. The MTBF is therefore adjusted by multiplying it by the ratio
of window to width:

MTBF = (timestamp of last failure - timestamp of first failure) * window
/ ((number of failures - 1) * width)

In hwlatdetect, the failures are called samples. The failure count is
the sum of counters from all samples.

This metric enables meaningful comparison of hardware latency across
different test runs, hardware configurations, and kernel versions.  It
can be considered a KPI for real-time stability, relevant for
certification and SLA evaluation.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
---

v4: Adjust MTBF by multiplying by the ratio of window to width
v3:
- Fix formatting
- Make first and last instance variables
v2:
- Use another more stable calculation of MTBF
---
 src/hwlatdetect/hwlatdetect.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/hwlatdetect/hwlatdetect.py b/src/hwlatdetect/hwlatdetect.py
index 38671724f3e3..abfbb954fe75 100755
--- a/src/hwlatdetect/hwlatdetect.py
+++ b/src/hwlatdetect/hwlatdetect.py
@@ -306,6 +306,8 @@     def __init__(self):
             raise DetectorNotAvailable("hwlat", "hwlat tracer not available")
         self.type = "tracer"
         self.samples = []
+        self.first = None
+        self.last = None
         self.set("enable", 0)
         self.set('current_tracer', 'hwlat')
 
@@ -338,6 +340,8 @@     def detect(self):
                 pollcnt += 1
                 val = self.get_sample()
                 while val:
+                    self.first = self.first or val.timestamp
+                    self.last = val.timestamp
                     self.samples.append(val)
                     if watch:
                         val.display()
@@ -559,6 +563,11 @@     def cleanup(self):
     exceeding = detect.get("count")
     info(f"Samples exceeding threshold: {exceeding}")
 
+    if exceeding > 1:
+        mtbf = ((float(detect.last) - float(detect.first)) * int(detect.get('window'))
+                / ((exceeding - 1) * int(detect.get('width'))))
+        info(f"MTBF: {mtbf:.3f} seconds")
+
     if detect.have_msr:
         finishsmi = detect.getsmicounts()
         total_smis = 0
-- 
2.53.0

Re: [PATCH v4] hwlatdetect: Add MTBF calculation

[John Kacur]

On Wed, 25 Feb 2026, Costa Shulyupin wrote:

> Hwlatdetect reports the number of latency spikes but provides no
> information about their frequency distribution over time. This makes it
> difficult to compare results - a test with 10 spikes over 1 hour is very
> different from 10 spikes over 24 hours, but both show 'spikes = 10'.
> 
> Add Mean Time Between Failures (MTBF) calculation to quantify spike
> frequency.
> 
> By definition MTBF = total operating time / number of failures.
> 
> When the failure interval is large relative to test duration, this
> formula is biased.  For example, imagine stable periodic failures.  The
> total operating time will include time before the first failure and
> after the last failure.  These intervals are determined by when the test
> starts and stops, not by the system's failure behavior, which adds
> measurement bias.  The resulting MTBF will vary between runs even for
> stable periodic failures.
> 
> To reduce this bias, calculate MTBF using only the time between the
> first and last failure, divided by the number of intervals (failures
> minus one).
> 
> Additionally, hwlatdetect only samples during the 'width' period within
> each 'window' cycle. The non-sampling periods contribute to the error
> margin. The MTBF is therefore adjusted by multiplying it by the ratio
> of window to width:
> 
> MTBF = (timestamp of last failure - timestamp of first failure) * window
> / ((number of failures - 1) * width)
> 
> In hwlatdetect, the failures are called samples. The failure count is
> the sum of counters from all samples.
> 
> This metric enables meaningful comparison of hardware latency across
> different test runs, hardware configurations, and kernel versions.  It
> can be considered a KPI for real-time stability, relevant for
> certification and SLA evaluation.
> 
> Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
> ---
> 
> v4: Adjust MTBF by multiplying by the ratio of window to width
> v3:
> - Fix formatting
> - Make first and last instance variables
> v2:
> - Use another more stable calculation of MTBF
> ---
>  src/hwlatdetect/hwlatdetect.py | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/src/hwlatdetect/hwlatdetect.py b/src/hwlatdetect/hwlatdetect.py
> index 38671724f3e3..abfbb954fe75 100755
> --- a/src/hwlatdetect/hwlatdetect.py
> +++ b/src/hwlatdetect/hwlatdetect.py
> @@ -306,6 +306,8 @@     def __init__(self):
>              raise DetectorNotAvailable("hwlat", "hwlat tracer not available")
>          self.type = "tracer"
>          self.samples = []
> +        self.first = None
> +        self.last = None
>          self.set("enable", 0)
>          self.set('current_tracer', 'hwlat')
>  
> @@ -338,6 +340,8 @@     def detect(self):
>                  pollcnt += 1
>                  val = self.get_sample()
>                  while val:
> +                    self.first = self.first or val.timestamp
> +                    self.last = val.timestamp
>                      self.samples.append(val)
>                      if watch:
>                          val.display()
> @@ -559,6 +563,11 @@     def cleanup(self):
>      exceeding = detect.get("count")
>      info(f"Samples exceeding threshold: {exceeding}")
>  
> +    if exceeding > 1:
> +        mtbf = ((float(detect.last) - float(detect.first)) * int(detect.get('window'))
> +                / ((exceeding - 1) * int(detect.get('width'))))
> +        info(f"MTBF: {mtbf:.3f} seconds")
> +
>      if detect.have_msr:
>          finishsmi = detect.getsmicounts()
>          total_smis = 0
> -- 
> 2.53.0
> 
> 
> 
Signed-off-by: John Kacur <jkacur@redhat.com>