[PATCH v4] hwlatdetect: Add MTBF calculation

[Costa Shulyupin <costa.shul@redhat.com>]

Hwlatdetect reports the number of latency spikes but provides no
information about their frequency distribution over time. This makes it
difficult to compare results - a test with 10 spikes over 1 hour is very
different from 10 spikes over 24 hours, but both show 'spikes = 10'.

Add Mean Time Between Failures (MTBF) calculation to quantify spike
frequency.

By definition MTBF = total operating time / number of failures.

When the failure interval is large relative to test duration, this
formula is biased.  For example, imagine stable periodic failures.  The
total operating time will include time before the first failure and
after the last failure.  These intervals are determined by when the test
starts and stops, not by the system's failure behavior, which adds
measurement bias.  The resulting MTBF will vary between runs even for
stable periodic failures.

To reduce this bias, calculate MTBF using only the time between the
first and last failure, divided by the number of intervals (failures
minus one).

Additionally, hwlatdetect only samples during the 'width' period within
each 'window' cycle. The non-sampling periods contribute to the error
margin. The MTBF is therefore adjusted by multiplying it by the ratio
of window to width:

MTBF = (timestamp of last failure - timestamp of first failure) * window
/ ((number of failures - 1) * width)

In hwlatdetect, the failures are called samples. The failure count is
the sum of counters from all samples.

This metric enables meaningful comparison of hardware latency across
different test runs, hardware configurations, and kernel versions.  It
can be considered a KPI for real-time stability, relevant for
certification and SLA evaluation.

Signed-off-by: Costa Shulyupin <costa.shul@redhat.com>
---

v4: Adjust MTBF by multiplying by the ratio of window to width
v3:
- Fix formatting
- Make first and last instance variables
v2:
- Use another more stable calculation of MTBF
---
 src/hwlatdetect/hwlatdetect.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/hwlatdetect/hwlatdetect.py b/src/hwlatdetect/hwlatdetect.py
index 38671724f3e3..abfbb954fe75 100755
--- a/src/hwlatdetect/hwlatdetect.py
+++ b/src/hwlatdetect/hwlatdetect.py
@@ -306,6 +306,8 @@     def __init__(self):
             raise DetectorNotAvailable("hwlat", "hwlat tracer not available")
         self.type = "tracer"
         self.samples = []
+        self.first = None
+        self.last = None
         self.set("enable", 0)
         self.set('current_tracer', 'hwlat')
 
@@ -338,6 +340,8 @@     def detect(self):
                 pollcnt += 1
                 val = self.get_sample()
                 while val:
+                    self.first = self.first or val.timestamp
+                    self.last = val.timestamp
                     self.samples.append(val)
                     if watch:
                         val.display()
@@ -559,6 +563,11 @@     def cleanup(self):
     exceeding = detect.get("count")
     info(f"Samples exceeding threshold: {exceeding}")
 
+    if exceeding > 1:
+        mtbf = ((float(detect.last) - float(detect.first)) * int(detect.get('window'))
+                / ((exceeding - 1) * int(detect.get('width'))))
+        info(f"MTBF: {mtbf:.3f} seconds")
+
     if detect.have_msr:
         finishsmi = detect.getsmicounts()
         total_smis = 0
-- 
2.53.0