[LTP] [RFC] kirk/json2logs: "warn" treated as failure + robustness improvements

[LTP] [RFC] kirk/json2logs: "warn" treated as failure + robustness improvements

[Michael Menasherov via ltp]

  Hi,

  While investigating test results reported as failures, I found that
  tools/kirk/kirk-src/utils/json2logs.py treats any test status that
  is not "pass" or "conf" as a failure. This includes "warn" (TWARN),
  which generates .fail.log files and appends to fails.log.

  TWARN typically indicates a non-critical issue (e.g., cleanup warning),
  not a test failure. I think this is an issue because it creates noise
  in failure reports and can mask real failures.

  Beyond the missing "warn" exclusion, I think there are a few other
  issues in the script:

  1. Fragile failure logic (line 46): the check uses a blocklist
  (not in ("pass", "conf")) instead of an allowlist. I think this
  is an issue because any new status kirk adds in the future would
  silently become a "failure". An explicit allowlist would be safer:

  current (blocklist):

  1. if status not in ("pass", "conf"):

  proposed (allowlist):

  1. if status in ("fail", "brok"):
  2. Path injection in output filename (line 48): the test FQN is
  used directly in f"{fqn}.fail.log". I think this is an issue
  because if the FQN contains "/", the file is written to an
  unexpected directory.
  3. Empty fails.log always created (line 74): even when all tests
  pass, an empty fails.log is written. I think this is an issue
  because downstream tooling that checks for the file's existence
  could interpret this as "failures occurred".

  Questions:

  - Is treating "warn" as failure intentional — to ensure warn results
  get visibility — or a bug?
  - Should the tests producing TWARN be investigated as well, or is
  fixing the reporter sufficient?
  - Would you accept a patch series addressing all three issues, or
  prefer the "warn" fix alone first?

  Thanks,
  Michael Menasherov

-- 

Michael Menasherov

Software Quality Engineer - Automotive Kernel

Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] kirk/json2logs: "warn" treated as failure + robustness improvements

[Andrea Cervesato via ltp]

Hi Michael,

if you have any suggestion, please send a PR to the kirk repo, I will
take a look and discuss it there.

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp