Re: [PATCH i-g-t] RFC: runner/results: Allow generate partial json

[Gustavo Sousa <gustavo.sousa@intel.com>]

Quoting Kamil Konieczny (2025-12-18 14:46:50-03:00)
>When json-c hits 2GB barrier it fails to generate results.json:
>
>resultgen: Failed to create json representation of the results.
>           This usually means that the results are too big
>
>Allow to select start and end folder for generation, so it could
>be done if a few steps, for example when there are 0...5432
>output folders, it could be done with:
>
>./igt_results /path/to/results 0 3000
>mv /path/to/results/results.json /path/to/results/results-0-3000.json
>
>./igt_results /path/to/results 3001 5432
>mv /path/to/results/results.json /path/to/results/results-3001-5432.json
>
>Cc: Mateusz Grabski <mateusz.grabski@intel.com>
>Cc: Ryszard Knop <ryszard.knop@intel.com>
>Cc: Ryszard Knop <rk@dragonic.eu>
>Signed-off-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>

I find it odd that we are generating a full string representation
in memory to then only be used to dump it to a file.  Doesn't json-c
allow streaming the JSON object directly into a file?  If not, I wonder
if there is another lib out there that allows that.

An alternative appraoch would be to open the output file for writing and
"help" the library by doing part of the top level entries of the JSON by
ourselves.  Something along the lines of:

  * Write "{";
  * Iterate the top level properties (key, value) of the root object:
      * Write the object key;
      * If the key is "tests": recurse;
        else: use json_object_to_json_string() on the value.
  * Write "}";

I think this partial open-coding would be preferable than breaking the
output into multiple files because of a limitation of the library we are
using.

--
Gustavo Sousa

>---
> runner/resultgen.c | 25 +++++++++++++++++++++----
> runner/resultgen.h |  2 +-
> runner/results.c   | 31 +++++++++++++++++++++++++++++--
> 3 files changed, 51 insertions(+), 7 deletions(-)
>
>diff --git a/runner/resultgen.c b/runner/resultgen.c
>index 1f7e79f24..5e0f5d320 100644
>--- a/runner/resultgen.c
>+++ b/runner/resultgen.c
>@@ -47,6 +47,9 @@ struct results
>         struct json_object *runtimes;
> };
> 
>+static int r_start = -1;
>+static int r_end = -1;
>+
> static void add_dynamic_subtest(struct subtest *subtest, char *dynamic)
> {
>         size_t len = strlen(dynamic);
>@@ -2309,6 +2312,7 @@ struct json_object *generate_results_json(int dirfd)
>         struct results results;
>         int testdirfd, fd;
>         size_t i;
>+        size_t tests_count;
> 
>         init_settings(&settings);
>         init_job_list(&job_list);
>@@ -2379,7 +2383,17 @@ struct json_object *generate_results_json(int dirfd)
>          * - options
>          */
> 
>-        for (i = 0; i < job_list.size; i++) {
>+        i = r_start == -1 ? 0 : (size_t)r_start;
>+
>+        tests_count = r_end == -1 ? job_list.size : (size_t)r_end;

I didn't spend much on this, but here it appears r_end (and hence "end"
argument of the CLI) is actually the maximum number of tests to be
processed starting from r_start?

--
Gustavo Sousa

>+        if (tests_count > job_list.size)
>+                tests_count = job_list.size;
>+
>+        fprintf(stderr, "Number of tests for parsing: %ld\n", job_list.size);
>+        if (r_start != -1 || r_end != -1)
>+                fprintf(stderr, "Parsing only partial results, from %ld to %ld\n", i, tests_count);
>+
>+        for (; i < tests_count; i++) {
>                 char name[16];
> 
>                 snprintf(name, 16, "%zd", i);
>@@ -2434,12 +2448,15 @@ struct json_object *generate_results_json(int dirfd)
>         return obj;
> }
> 
>-bool generate_results(int dirfd)
>+bool generate_results(int dirfd, int start, int end)
> {
>-        struct json_object *obj = generate_results_json(dirfd);
>+        struct json_object *obj;
>         const char *json_string;
>         int resultsfd;
> 
>+        r_start = start;
>+        r_end = end;
>+        obj = generate_results_json(dirfd);
>         if (obj == NULL)
>                 return false;
> 
>@@ -2475,5 +2492,5 @@ bool generate_results_path(char *resultspath)
>         if (dirfd < 0)
>                 return false;
> 
>-        return generate_results(dirfd);
>+        return generate_results(dirfd, -1, -1);
> }
>diff --git a/runner/resultgen.h b/runner/resultgen.h
>index ab0bf1082..14ae96a54 100644
>--- a/runner/resultgen.h
>+++ b/runner/resultgen.h
>@@ -3,7 +3,7 @@
> 
> #include <stdbool.h>
> 
>-bool generate_results(int dirfd);
>+bool generate_results(int dirfd, int start, int end);
> bool generate_results_path(char *resultspath);
> 
> struct json_object *generate_results_json(int dirfd);
>diff --git a/runner/results.c b/runner/results.c
>index 3eb7cb155..62ae8a1c0 100644
>--- a/runner/results.c
>+++ b/runner/results.c
>@@ -9,6 +9,7 @@
> int main(int argc, char **argv)
> {
>         int dirfd;
>+        int start, end;
> 
>         if (argc < 2)
>                 exit(1);
>@@ -17,10 +18,36 @@ int main(int argc, char **argv)
>         if (dirfd < 0)
>                 exit(1);
> 
>-        if (generate_results(dirfd)) {
>-                printf("Results generated\n");
>+        if (argc == 3) {
>+                start = 0;
>+                end = atoi(argv[2]);
>+                if (end >= 0)
>+                        ++end;
>+        } else if (argc == 4) {
>+                start = atoi(argv[2]);
>+                end = atoi(argv[3]);
>+                if (end >= 0)
>+                        ++end;
>+        } else {
>+                start = -1;
>+                end = -1;
>+        }
>+
>+        if (start < -1 || end < -1) {
>+                printf("Error: start and end options should be numbers >= 0\n");
>+                start = -1;
>+                end = -1;
>+        }
>+
>+        if (generate_results(dirfd, start, end)) {
>+                if (start >= 0)
>+                        printf("Results generated from %d to %d\n", start, end);
>+                else
>+                        printf("Results generated\n");
>                 exit(0);
>         }
> 
>+        if (start >= 0)
>+                printf("Error: cannot generate results from %d to %d\n", start, end);
>         exit(1);
> }
>-- 
>2.52.0
>