* [PATCH i-g-t] RFC: runner/results: Allow generate partial json
@ 2025-12-18 17:46 Kamil Konieczny
2025-12-24 18:15 ` Gustavo Sousa
0 siblings, 1 reply; 2+ messages in thread
From: Kamil Konieczny @ 2025-12-18 17:46 UTC (permalink / raw)
To: igt-dev; +Cc: Kamil Konieczny, Mateusz Grabski, Ryszard Knop, Ryszard Knop
When json-c hits 2GB barrier it fails to generate results.json:
resultgen: Failed to create json representation of the results.
This usually means that the results are too big
Allow to select start and end folder for generation, so it could
be done if a few steps, for example when there are 0...5432
output folders, it could be done with:
./igt_results /path/to/results 0 3000
mv /path/to/results/results.json /path/to/results/results-0-3000.json
./igt_results /path/to/results 3001 5432
mv /path/to/results/results.json /path/to/results/results-3001-5432.json
Cc: Mateusz Grabski <mateusz.grabski@intel.com>
Cc: Ryszard Knop <ryszard.knop@intel.com>
Cc: Ryszard Knop <rk@dragonic.eu>
Signed-off-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
---
runner/resultgen.c | 25 +++++++++++++++++++++----
runner/resultgen.h | 2 +-
runner/results.c | 31 +++++++++++++++++++++++++++++--
3 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/runner/resultgen.c b/runner/resultgen.c
index 1f7e79f24..5e0f5d320 100644
--- a/runner/resultgen.c
+++ b/runner/resultgen.c
@@ -47,6 +47,9 @@ struct results
struct json_object *runtimes;
};
+static int r_start = -1;
+static int r_end = -1;
+
static void add_dynamic_subtest(struct subtest *subtest, char *dynamic)
{
size_t len = strlen(dynamic);
@@ -2309,6 +2312,7 @@ struct json_object *generate_results_json(int dirfd)
struct results results;
int testdirfd, fd;
size_t i;
+ size_t tests_count;
init_settings(&settings);
init_job_list(&job_list);
@@ -2379,7 +2383,17 @@ struct json_object *generate_results_json(int dirfd)
* - options
*/
- for (i = 0; i < job_list.size; i++) {
+ i = r_start == -1 ? 0 : (size_t)r_start;
+
+ tests_count = r_end == -1 ? job_list.size : (size_t)r_end;
+ if (tests_count > job_list.size)
+ tests_count = job_list.size;
+
+ fprintf(stderr, "Number of tests for parsing: %ld\n", job_list.size);
+ if (r_start != -1 || r_end != -1)
+ fprintf(stderr, "Parsing only partial results, from %ld to %ld\n", i, tests_count);
+
+ for (; i < tests_count; i++) {
char name[16];
snprintf(name, 16, "%zd", i);
@@ -2434,12 +2448,15 @@ struct json_object *generate_results_json(int dirfd)
return obj;
}
-bool generate_results(int dirfd)
+bool generate_results(int dirfd, int start, int end)
{
- struct json_object *obj = generate_results_json(dirfd);
+ struct json_object *obj;
const char *json_string;
int resultsfd;
+ r_start = start;
+ r_end = end;
+ obj = generate_results_json(dirfd);
if (obj == NULL)
return false;
@@ -2475,5 +2492,5 @@ bool generate_results_path(char *resultspath)
if (dirfd < 0)
return false;
- return generate_results(dirfd);
+ return generate_results(dirfd, -1, -1);
}
diff --git a/runner/resultgen.h b/runner/resultgen.h
index ab0bf1082..14ae96a54 100644
--- a/runner/resultgen.h
+++ b/runner/resultgen.h
@@ -3,7 +3,7 @@
#include <stdbool.h>
-bool generate_results(int dirfd);
+bool generate_results(int dirfd, int start, int end);
bool generate_results_path(char *resultspath);
struct json_object *generate_results_json(int dirfd);
diff --git a/runner/results.c b/runner/results.c
index 3eb7cb155..62ae8a1c0 100644
--- a/runner/results.c
+++ b/runner/results.c
@@ -9,6 +9,7 @@
int main(int argc, char **argv)
{
int dirfd;
+ int start, end;
if (argc < 2)
exit(1);
@@ -17,10 +18,36 @@ int main(int argc, char **argv)
if (dirfd < 0)
exit(1);
- if (generate_results(dirfd)) {
- printf("Results generated\n");
+ if (argc == 3) {
+ start = 0;
+ end = atoi(argv[2]);
+ if (end >= 0)
+ ++end;
+ } else if (argc == 4) {
+ start = atoi(argv[2]);
+ end = atoi(argv[3]);
+ if (end >= 0)
+ ++end;
+ } else {
+ start = -1;
+ end = -1;
+ }
+
+ if (start < -1 || end < -1) {
+ printf("Error: start and end options should be numbers >= 0\n");
+ start = -1;
+ end = -1;
+ }
+
+ if (generate_results(dirfd, start, end)) {
+ if (start >= 0)
+ printf("Results generated from %d to %d\n", start, end);
+ else
+ printf("Results generated\n");
exit(0);
}
+ if (start >= 0)
+ printf("Error: cannot generate results from %d to %d\n", start, end);
exit(1);
}
--
2.52.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH i-g-t] RFC: runner/results: Allow generate partial json
2025-12-18 17:46 [PATCH i-g-t] RFC: runner/results: Allow generate partial json Kamil Konieczny
@ 2025-12-24 18:15 ` Gustavo Sousa
0 siblings, 0 replies; 2+ messages in thread
From: Gustavo Sousa @ 2025-12-24 18:15 UTC (permalink / raw)
To: Kamil Konieczny, igt-dev
Cc: Kamil Konieczny, Mateusz Grabski, Ryszard Knop, Ryszard Knop
Quoting Kamil Konieczny (2025-12-18 14:46:50-03:00)
>When json-c hits 2GB barrier it fails to generate results.json:
>
>resultgen: Failed to create json representation of the results.
> This usually means that the results are too big
>
>Allow to select start and end folder for generation, so it could
>be done if a few steps, for example when there are 0...5432
>output folders, it could be done with:
>
>./igt_results /path/to/results 0 3000
>mv /path/to/results/results.json /path/to/results/results-0-3000.json
>
>./igt_results /path/to/results 3001 5432
>mv /path/to/results/results.json /path/to/results/results-3001-5432.json
>
>Cc: Mateusz Grabski <mateusz.grabski@intel.com>
>Cc: Ryszard Knop <ryszard.knop@intel.com>
>Cc: Ryszard Knop <rk@dragonic.eu>
>Signed-off-by: Kamil Konieczny <kamil.konieczny@linux.intel.com>
I find it odd that we are generating a full string representation
in memory to then only be used to dump it to a file. Doesn't json-c
allow streaming the JSON object directly into a file? If not, I wonder
if there is another lib out there that allows that.
An alternative appraoch would be to open the output file for writing and
"help" the library by doing part of the top level entries of the JSON by
ourselves. Something along the lines of:
* Write "{";
* Iterate the top level properties (key, value) of the root object:
* Write the object key;
* If the key is "tests": recurse;
else: use json_object_to_json_string() on the value.
* Write "}";
I think this partial open-coding would be preferable than breaking the
output into multiple files because of a limitation of the library we are
using.
--
Gustavo Sousa
>---
> runner/resultgen.c | 25 +++++++++++++++++++++----
> runner/resultgen.h | 2 +-
> runner/results.c | 31 +++++++++++++++++++++++++++++--
> 3 files changed, 51 insertions(+), 7 deletions(-)
>
>diff --git a/runner/resultgen.c b/runner/resultgen.c
>index 1f7e79f24..5e0f5d320 100644
>--- a/runner/resultgen.c
>+++ b/runner/resultgen.c
>@@ -47,6 +47,9 @@ struct results
> struct json_object *runtimes;
> };
>
>+static int r_start = -1;
>+static int r_end = -1;
>+
> static void add_dynamic_subtest(struct subtest *subtest, char *dynamic)
> {
> size_t len = strlen(dynamic);
>@@ -2309,6 +2312,7 @@ struct json_object *generate_results_json(int dirfd)
> struct results results;
> int testdirfd, fd;
> size_t i;
>+ size_t tests_count;
>
> init_settings(&settings);
> init_job_list(&job_list);
>@@ -2379,7 +2383,17 @@ struct json_object *generate_results_json(int dirfd)
> * - options
> */
>
>- for (i = 0; i < job_list.size; i++) {
>+ i = r_start == -1 ? 0 : (size_t)r_start;
>+
>+ tests_count = r_end == -1 ? job_list.size : (size_t)r_end;
I didn't spend much on this, but here it appears r_end (and hence "end"
argument of the CLI) is actually the maximum number of tests to be
processed starting from r_start?
--
Gustavo Sousa
>+ if (tests_count > job_list.size)
>+ tests_count = job_list.size;
>+
>+ fprintf(stderr, "Number of tests for parsing: %ld\n", job_list.size);
>+ if (r_start != -1 || r_end != -1)
>+ fprintf(stderr, "Parsing only partial results, from %ld to %ld\n", i, tests_count);
>+
>+ for (; i < tests_count; i++) {
> char name[16];
>
> snprintf(name, 16, "%zd", i);
>@@ -2434,12 +2448,15 @@ struct json_object *generate_results_json(int dirfd)
> return obj;
> }
>
>-bool generate_results(int dirfd)
>+bool generate_results(int dirfd, int start, int end)
> {
>- struct json_object *obj = generate_results_json(dirfd);
>+ struct json_object *obj;
> const char *json_string;
> int resultsfd;
>
>+ r_start = start;
>+ r_end = end;
>+ obj = generate_results_json(dirfd);
> if (obj == NULL)
> return false;
>
>@@ -2475,5 +2492,5 @@ bool generate_results_path(char *resultspath)
> if (dirfd < 0)
> return false;
>
>- return generate_results(dirfd);
>+ return generate_results(dirfd, -1, -1);
> }
>diff --git a/runner/resultgen.h b/runner/resultgen.h
>index ab0bf1082..14ae96a54 100644
>--- a/runner/resultgen.h
>+++ b/runner/resultgen.h
>@@ -3,7 +3,7 @@
>
> #include <stdbool.h>
>
>-bool generate_results(int dirfd);
>+bool generate_results(int dirfd, int start, int end);
> bool generate_results_path(char *resultspath);
>
> struct json_object *generate_results_json(int dirfd);
>diff --git a/runner/results.c b/runner/results.c
>index 3eb7cb155..62ae8a1c0 100644
>--- a/runner/results.c
>+++ b/runner/results.c
>@@ -9,6 +9,7 @@
> int main(int argc, char **argv)
> {
> int dirfd;
>+ int start, end;
>
> if (argc < 2)
> exit(1);
>@@ -17,10 +18,36 @@ int main(int argc, char **argv)
> if (dirfd < 0)
> exit(1);
>
>- if (generate_results(dirfd)) {
>- printf("Results generated\n");
>+ if (argc == 3) {
>+ start = 0;
>+ end = atoi(argv[2]);
>+ if (end >= 0)
>+ ++end;
>+ } else if (argc == 4) {
>+ start = atoi(argv[2]);
>+ end = atoi(argv[3]);
>+ if (end >= 0)
>+ ++end;
>+ } else {
>+ start = -1;
>+ end = -1;
>+ }
>+
>+ if (start < -1 || end < -1) {
>+ printf("Error: start and end options should be numbers >= 0\n");
>+ start = -1;
>+ end = -1;
>+ }
>+
>+ if (generate_results(dirfd, start, end)) {
>+ if (start >= 0)
>+ printf("Results generated from %d to %d\n", start, end);
>+ else
>+ printf("Results generated\n");
> exit(0);
> }
>
>+ if (start >= 0)
>+ printf("Error: cannot generate results from %d to %d\n", start, end);
> exit(1);
> }
>--
>2.52.0
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2025-12-24 18:15 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-12-18 17:46 [PATCH i-g-t] RFC: runner/results: Allow generate partial json Kamil Konieczny
2025-12-24 18:15 ` Gustavo Sousa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox