* [LTP] [PATCH 0/3] Shell test library v3
@ 2024-07-31 9:20 Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 1/3] Add support for mixing C and shell code Cyril Hrubis
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Cyril Hrubis @ 2024-07-31 9:20 UTC (permalink / raw)
To: ltp
Changes againts RFC:
- The metadata in tests are encoded in JSON now, there is a proper
parser used to decode them and construct the tst_test accordingly.
Most of the C library features are now available in shell tests as
well.
- The shell tests use a script to bootstrap themselves so they can be
just executed normally see tst_loader.sh
- Many small fixes that were pointed out for the RFC
Cyril Hrubis (3):
Add support for mixing C and shell code
libs: Vendor ujson library
testcaes/lib: Add shell loader
include/tst_test.h | 40 +-
include/ujson.h | 13 +
include/ujson_common.h | 69 ++
include/ujson_reader.h | 539 ++++++++
include/ujson_utf.h | 168 +++
include/ujson_writer.h | 224 ++++
lib/tst_test.c | 51 +
libs/ujson/Makefile | 12 +
libs/ujson/ujson_common.c | 38 +
libs/ujson/ujson_reader.c | 1081 +++++++++++++++++
libs/ujson/ujson_utf.c | 105 ++
libs/ujson/ujson_writer.c | 491 ++++++++
testcases/lib/.gitignore | 2 +
testcases/lib/Makefile | 8 +-
testcases/lib/run_tests.sh | 20 +
testcases/lib/tests/.gitignore | 6 +
testcases/lib/tests/Makefile | 11 +
testcases/lib/tests/shell_loader.sh | 15 +
.../lib/tests/shell_loader_all_filesystems.sh | 24 +
.../lib/tests/shell_loader_filesystems.sh | 30 +
.../tests/shell_loader_invalid_metadata.sh | 12 +
.../lib/tests/shell_loader_no_metadata.sh | 8 +
.../lib/tests/shell_loader_supported_archs.sh | 9 +
.../lib/tests/shell_loader_wrong_metadata.sh | 12 +
testcases/lib/tests/shell_test01.c | 17 +
testcases/lib/tests/shell_test02.c | 18 +
testcases/lib/tests/shell_test03.c | 25 +
testcases/lib/tests/shell_test04.c | 18 +
testcases/lib/tests/shell_test05.c | 27 +
testcases/lib/tests/shell_test06.c | 16 +
testcases/lib/tests/shell_test_brk.sh | 6 +
testcases/lib/tests/shell_test_check_argv.sh | 23 +
testcases/lib/tests/shell_test_checkpoint.sh | 7 +
testcases/lib/tests/shell_test_pass.sh | 6 +
testcases/lib/tst_env.sh | 25 +
testcases/lib/tst_loader.sh | 11 +
testcases/lib/tst_res_.c | 58 +
testcases/lib/tst_run_shell.c | 378 ++++++
38 files changed, 3620 insertions(+), 3 deletions(-)
create mode 100644 include/ujson.h
create mode 100644 include/ujson_common.h
create mode 100644 include/ujson_reader.h
create mode 100644 include/ujson_utf.h
create mode 100644 include/ujson_writer.h
create mode 100644 libs/ujson/Makefile
create mode 100644 libs/ujson/ujson_common.c
create mode 100644 libs/ujson/ujson_reader.c
create mode 100644 libs/ujson/ujson_utf.c
create mode 100644 libs/ujson/ujson_writer.c
create mode 100755 testcases/lib/run_tests.sh
create mode 100644 testcases/lib/tests/.gitignore
create mode 100644 testcases/lib/tests/Makefile
create mode 100755 testcases/lib/tests/shell_loader.sh
create mode 100755 testcases/lib/tests/shell_loader_all_filesystems.sh
create mode 100755 testcases/lib/tests/shell_loader_filesystems.sh
create mode 100755 testcases/lib/tests/shell_loader_invalid_metadata.sh
create mode 100755 testcases/lib/tests/shell_loader_no_metadata.sh
create mode 100755 testcases/lib/tests/shell_loader_supported_archs.sh
create mode 100755 testcases/lib/tests/shell_loader_wrong_metadata.sh
create mode 100644 testcases/lib/tests/shell_test01.c
create mode 100644 testcases/lib/tests/shell_test02.c
create mode 100644 testcases/lib/tests/shell_test03.c
create mode 100644 testcases/lib/tests/shell_test04.c
create mode 100644 testcases/lib/tests/shell_test05.c
create mode 100644 testcases/lib/tests/shell_test06.c
create mode 100755 testcases/lib/tests/shell_test_brk.sh
create mode 100755 testcases/lib/tests/shell_test_check_argv.sh
create mode 100755 testcases/lib/tests/shell_test_checkpoint.sh
create mode 100755 testcases/lib/tests/shell_test_pass.sh
create mode 100644 testcases/lib/tst_env.sh
create mode 100644 testcases/lib/tst_loader.sh
create mode 100644 testcases/lib/tst_res_.c
create mode 100644 testcases/lib/tst_run_shell.c
--
2.44.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH 1/3] Add support for mixing C and shell code
2024-07-31 9:20 [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
@ 2024-07-31 9:20 ` Cyril Hrubis
2024-08-13 15:35 ` Richard Palethorpe
2024-07-31 9:20 ` [LTP] [PATCH 2/3] libs: Vendor ujson library Cyril Hrubis
` (2 subsequent siblings)
3 siblings, 1 reply; 10+ messages in thread
From: Cyril Hrubis @ 2024-07-31 9:20 UTC (permalink / raw)
To: ltp
This is a proof of a concept of a seamless C and shell integration. The
idea is that with this you can mix shell and C code as much as as you
wish to get the best of the two worlds.
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
include/tst_test.h | 38 +++++++++++++
lib/tst_test.c | 51 +++++++++++++++++
testcases/lib/.gitignore | 1 +
testcases/lib/Makefile | 4 +-
testcases/lib/run_tests.sh | 11 ++++
testcases/lib/tests/.gitignore | 6 ++
testcases/lib/tests/Makefile | 11 ++++
testcases/lib/tests/shell_test01.c | 17 ++++++
testcases/lib/tests/shell_test02.c | 18 ++++++
testcases/lib/tests/shell_test03.c | 25 +++++++++
testcases/lib/tests/shell_test04.c | 18 ++++++
testcases/lib/tests/shell_test05.c | 27 +++++++++
testcases/lib/tests/shell_test06.c | 16 ++++++
testcases/lib/tests/shell_test_brk.sh | 6 ++
testcases/lib/tests/shell_test_check_argv.sh | 23 ++++++++
testcases/lib/tests/shell_test_checkpoint.sh | 7 +++
testcases/lib/tests/shell_test_pass.sh | 6 ++
testcases/lib/tst_env.sh | 21 +++++++
testcases/lib/tst_res_.c | 58 ++++++++++++++++++++
19 files changed, 362 insertions(+), 2 deletions(-)
create mode 100755 testcases/lib/run_tests.sh
create mode 100644 testcases/lib/tests/.gitignore
create mode 100644 testcases/lib/tests/Makefile
create mode 100644 testcases/lib/tests/shell_test01.c
create mode 100644 testcases/lib/tests/shell_test02.c
create mode 100644 testcases/lib/tests/shell_test03.c
create mode 100644 testcases/lib/tests/shell_test04.c
create mode 100644 testcases/lib/tests/shell_test05.c
create mode 100644 testcases/lib/tests/shell_test06.c
create mode 100755 testcases/lib/tests/shell_test_brk.sh
create mode 100755 testcases/lib/tests/shell_test_check_argv.sh
create mode 100755 testcases/lib/tests/shell_test_checkpoint.sh
create mode 100755 testcases/lib/tests/shell_test_pass.sh
create mode 100644 testcases/lib/tst_env.sh
create mode 100644 testcases/lib/tst_res_.c
diff --git a/include/tst_test.h b/include/tst_test.h
index 6c76f043d..a334195ac 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -331,6 +331,8 @@ struct tst_fs {
* @child_needs_reinit: Has to be set if the test needs to call tst_reinit()
* from a process started by exec().
*
+ * @runs_script: Implies child_needs_reinit and forks_child at the moment.
+ *
* @needs_devfs: If set the devfs is mounted at tst_test.mntpoint. This is
* needed for tests that need to create device files since tmpfs
* at /tmp is usually mounted with 'nodev' option.
@@ -518,6 +520,7 @@ struct tst_fs {
unsigned int mount_device:1;
unsigned int needs_rofs:1;
unsigned int child_needs_reinit:1;
+ unsigned int runs_script:1;
unsigned int needs_devfs:1;
unsigned int restore_wallclock:1;
@@ -526,6 +529,8 @@ struct tst_fs {
unsigned int skip_in_lockdown:1;
unsigned int skip_in_secureboot:1;
unsigned int skip_in_compat:1;
+
+
int needs_abi_bits;
unsigned int needs_hugetlbfs:1;
@@ -611,6 +616,39 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
*/
void tst_reinit(void);
+/**
+ * tst_run_shell() - Prepare the environment and execute a shell script.
+ *
+ * @script_name: A filename of the script.
+ * @params: A NULL terminated array of shell script parameters, pass NULL if
+ * none are needed. This what is passed starting from argv[1].
+ *
+ * The shell script is executed with LTP_IPC_PATH in environment so that the
+ * binary helpers such as tst_res_ or tst_checkpoint work properly when executed
+ * from the script. This also means that the tst_test.runs_script flag needs to
+ * be set.
+ *
+ * The shell script itself has to source the tst_env.sh shell script at the
+ * start and after that it's free to use tst_res in the same way C code would
+ * use.
+ *
+ * Example shell script that reports success::
+ *
+ * #!/bin/sh
+ * . tst_env.sh
+ *
+ * tst_res TPASS "Example test works"
+ *
+ * The call returns a pid in a case that you want to examine the return value
+ * of the script yourself. If you do not need to check the return value
+ * yourself you can use tst_reap_children() to wait for the completion. Or let
+ * the test library collect the child automatically, just be wary that the
+ * script and the test both runs concurently at the same time in this case.
+ *
+ * Return: A pid of the shell process.
+ */
+int tst_run_shell(char *script_name, char *const params[]);
+
unsigned int tst_multiply_timeout(unsigned int timeout);
/*
diff --git a/lib/tst_test.c b/lib/tst_test.c
index e5bc5bf4d..7e1075fdf 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -4,6 +4,8 @@
* Copyright (c) Linux Test Project, 2016-2024
*/
+#define _GNU_SOURCE
+
#include <limits.h>
#include <stdio.h>
#include <stdarg.h>
@@ -173,6 +175,50 @@ void tst_reinit(void)
SAFE_CLOSE(fd);
}
+extern char **environ;
+
+static unsigned int params_array_len(char *const array[])
+{
+ unsigned int ret = 0;
+
+ if (!array)
+ return 0;
+
+ while (*(array++))
+ ret++;
+
+ return ret;
+}
+
+int tst_run_shell(char *script_name, char *const params[])
+{
+ int pid;
+ unsigned int i, params_len = params_array_len(params);
+ char *argv[params_len + 2];
+
+ if (!tst_test->runs_script)
+ tst_brk(TBROK, "runs_script flag must be set!");
+
+ argv[0] = script_name;
+
+ if (params) {
+ for (i = 0; i < params_len; i++)
+ argv[i+1] = params[i];
+ }
+
+ argv[params_len+1] = NULL;
+
+ pid = SAFE_FORK();
+ if (pid)
+ return pid;
+
+ execvpe(script_name, argv, environ);
+
+ tst_brk(TBROK | TERRNO, "execvpe(%s, ...) failed!", script_name);
+
+ return -1;
+}
+
static void update_results(int ttype)
{
if (!results)
@@ -1224,6 +1270,11 @@ static void do_setup(int argc, char *argv[])
tdebug = 1;
}
+ if (tst_test->runs_script) {
+ tst_test->child_needs_reinit = 1;
+ tst_test->forks_child = 1;
+ }
+
if (tst_test->needs_kconfigs && tst_kconfig_check(tst_test->needs_kconfigs))
tst_brk(TCONF, "Aborting due to unsuitable kernel config, see above!");
diff --git a/testcases/lib/.gitignore b/testcases/lib/.gitignore
index e8afd06f3..d0dacf62a 100644
--- a/testcases/lib/.gitignore
+++ b/testcases/lib/.gitignore
@@ -23,3 +23,4 @@
/tst_sleep
/tst_supported_fs
/tst_timeout_kill
+/tst_res_
diff --git a/testcases/lib/Makefile b/testcases/lib/Makefile
index 990b46089..928d76d62 100644
--- a/testcases/lib/Makefile
+++ b/testcases/lib/Makefile
@@ -13,6 +13,6 @@ MAKE_TARGETS := tst_sleep tst_random tst_checkpoint tst_rod tst_kvcmp\
tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port\
tst_get_median tst_hexdump tst_get_free_pids tst_timeout_kill\
tst_check_kconfigs tst_cgctl tst_fsfreeze tst_ns_create tst_ns_exec\
- tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled
+ tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled tst_res_
-include $(top_srcdir)/include/mk/generic_leaf_target.mk
+include $(top_srcdir)/include/mk/generic_trunk_target.mk
diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
new file mode 100755
index 000000000..60e7d1bcf
--- /dev/null
+++ b/testcases/lib/run_tests.sh
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+testdir=$(realpath $(dirname $0))
+export PATH=$PATH:$testdir:$testdir/tests/
+
+for i in `seq -w 01 06`; do
+ echo
+ echo "*** Running shell_test$i ***"
+ echo
+ ./tests/shell_test$i
+done
diff --git a/testcases/lib/tests/.gitignore b/testcases/lib/tests/.gitignore
new file mode 100644
index 000000000..da967c4d6
--- /dev/null
+++ b/testcases/lib/tests/.gitignore
@@ -0,0 +1,6 @@
+shell_test01
+shell_test02
+shell_test03
+shell_test04
+shell_test05
+shell_test06
diff --git a/testcases/lib/tests/Makefile b/testcases/lib/tests/Makefile
new file mode 100644
index 000000000..5a5cf5310
--- /dev/null
+++ b/testcases/lib/tests/Makefile
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (C) 2009, Cisco Systems Inc.
+# Ngie Cooper, August 2009
+
+top_srcdir ?= ../../..
+
+include $(top_srcdir)/include/mk/testcases.mk
+
+INSTALL_TARGETS=
+
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/testcases/lib/tests/shell_test01.c b/testcases/lib/tests/shell_test01.c
new file mode 100644
index 000000000..53c092783
--- /dev/null
+++ b/testcases/lib/tests/shell_test01.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ tst_run_shell("shell_test_pass.sh", NULL);
+ tst_res(TINFO, "C test exits now");
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test02.c b/testcases/lib/tests/shell_test02.c
new file mode 100644
index 000000000..1bc300ed3
--- /dev/null
+++ b/testcases/lib/tests/shell_test02.c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ tst_run_shell("shell_test_pass.sh", NULL);
+ tst_reap_children();
+ tst_res(TINFO, "Shell test has finished at this point!");
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test03.c b/testcases/lib/tests/shell_test03.c
new file mode 100644
index 000000000..2faa5e84c
--- /dev/null
+++ b/testcases/lib/tests/shell_test03.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include <sys/wait.h>
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ int pid, status;
+
+ pid = tst_run_shell("shell_test_pass.sh", NULL);
+
+ tst_res(TINFO, "Waiting for the pid %i", pid);
+
+ waitpid(pid, &status, 0);
+
+ tst_res(TINFO, "Shell test has %s", tst_strstatus(status));
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test04.c b/testcases/lib/tests/shell_test04.c
new file mode 100644
index 000000000..beb4d783d
--- /dev/null
+++ b/testcases/lib/tests/shell_test04.c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ char *const params[] = {"param1", "param2", NULL};
+
+ tst_run_shell("shell_test_check_argv.sh", params);
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test05.c b/testcases/lib/tests/shell_test05.c
new file mode 100644
index 000000000..c6b446c76
--- /dev/null
+++ b/testcases/lib/tests/shell_test05.c
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ int pid;
+
+ pid = tst_run_shell("shell_test_checkpoint.sh", NULL);
+
+ tst_res(TINFO, "Waiting for shell to sleep on checkpoint!");
+
+ TST_PROCESS_STATE_WAIT(pid, 'S', 10000);
+
+ tst_res(TINFO, "Waking shell child!");
+
+ TST_CHECKPOINT_WAKE(0);
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .needs_checkpoints = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test06.c b/testcases/lib/tests/shell_test06.c
new file mode 100644
index 000000000..d7f0ce946
--- /dev/null
+++ b/testcases/lib/tests/shell_test06.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test example.
+ */
+
+#include "tst_test.h"
+
+static void run_test(void)
+{
+ tst_run_shell("shell_test_brk.sh", NULL);
+}
+
+static struct tst_test test = {
+ .runs_script = 1,
+ .test_all = run_test,
+};
diff --git a/testcases/lib/tests/shell_test_brk.sh b/testcases/lib/tests/shell_test_brk.sh
new file mode 100755
index 000000000..f266dc3fe
--- /dev/null
+++ b/testcases/lib/tests/shell_test_brk.sh
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+. tst_env.sh
+
+tst_brk TCONF "This exits test and the next message should not be reached"
+tst_res TFAIL "If you see this the test failed"
diff --git a/testcases/lib/tests/shell_test_check_argv.sh b/testcases/lib/tests/shell_test_check_argv.sh
new file mode 100755
index 000000000..ce357027d
--- /dev/null
+++ b/testcases/lib/tests/shell_test_check_argv.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+. tst_env.sh
+
+tst_res TINFO "argv = $@"
+
+if [ $# -ne 2 ]; then
+ tst_res TFAIL "Wrong number of parameters got $# expected 2"
+else
+ tst_res TPASS "Got 2 parameters"
+fi
+
+if [ "$1" != "param1" ]; then
+ tst_res TFAIL "First parameter is $1 expected param1"
+else
+ tst_res TPASS "First parameter is $1"
+fi
+
+if [ "$2" != "param2" ]; then
+ tst_res TFAIL "Second parameter is $2 expected param2"
+else
+ tst_res TPASS "Second parameter is $2"
+fi
diff --git a/testcases/lib/tests/shell_test_checkpoint.sh b/testcases/lib/tests/shell_test_checkpoint.sh
new file mode 100755
index 000000000..0ceb7cf66
--- /dev/null
+++ b/testcases/lib/tests/shell_test_checkpoint.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+. tst_env.sh
+
+tst_res TINFO "Waiting for a checkpoint 0"
+tst_checkpoint wait 10000 0
+tst_res TPASS "Continuing after checkpoint"
diff --git a/testcases/lib/tests/shell_test_pass.sh b/testcases/lib/tests/shell_test_pass.sh
new file mode 100755
index 000000000..fd0684eb2
--- /dev/null
+++ b/testcases/lib/tests/shell_test_pass.sh
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+. tst_env.sh
+
+tst_res TPASS "This is called from the shell script!"
+tst_sleep 100ms
diff --git a/testcases/lib/tst_env.sh b/testcases/lib/tst_env.sh
new file mode 100644
index 000000000..948bc5024
--- /dev/null
+++ b/testcases/lib/tst_env.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+tst_script_name=$(basename $0)
+
+if [ -z "$LTP_IPC_PATH" ]; then
+ echo "This script has to be executed from a LTP loader!"
+ exit 1
+fi
+
+tst_brk_()
+{
+ tst_res_ "$@"
+
+ case "$3" in
+ "TBROK") exit 2;;
+ *) exit 0;;
+ esac
+}
+
+alias tst_res="tst_res_ $tst_script_name \$LINENO"
+alias tst_brk="tst_brk_ $tst_script_name \$LINENO"
diff --git a/testcases/lib/tst_res_.c b/testcases/lib/tst_res_.c
new file mode 100644
index 000000000..a43920f36
--- /dev/null
+++ b/testcases/lib/tst_res_.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (c) 2024 Cyril Hrubis <chrubis@suse.cz>
+ */
+
+#define TST_NO_DEFAULT_MAIN
+#include "tst_test.h"
+
+static void print_help(void)
+{
+ printf("Usage: tst_res_ filename lineno [TPASS|TFAIL|TCONF|TINFO|TDEBUG] 'A short description'\n");
+}
+
+int main(int argc, char *argv[])
+{
+ int type, i;
+
+ if (argc < 5)
+ goto help;
+
+ if (!strcmp(argv[3], "TPASS"))
+ type = TPASS;
+ else if (!strcmp(argv[3], "TFAIL"))
+ type = TFAIL;
+ else if (!strcmp(argv[3], "TCONF"))
+ type = TCONF;
+ else if (!strcmp(argv[3], "TINFO"))
+ type = TINFO;
+ else if (!strcmp(argv[3], "TDEBUG"))
+ type = TDEBUG;
+ else
+ goto help;
+
+ size_t len = 0;
+
+ for (i = 4; i < argc; i++)
+ len += strlen(argv[i]) + 1;
+
+ char *msg = SAFE_MALLOC(len);
+ char *msgp = msg;
+
+ for (i = 4; i < argc; i++) {
+ msgp = strcpy(msgp, argv[i]);
+ msgp += strlen(argv[i]);
+ *(msgp++) = ' ';
+ }
+
+ *(msgp - 1) = 0;
+
+ tst_reinit();
+
+ tst_res_(argv[1], atoi(argv[2]), type, "%s", msg);
+
+ return 0;
+help:
+ print_help();
+ return 1;
+}
--
2.44.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH 2/3] libs: Vendor ujson library
2024-07-31 9:20 [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 1/3] Add support for mixing C and shell code Cyril Hrubis
@ 2024-07-31 9:20 ` Cyril Hrubis
2024-08-13 15:38 ` Richard Palethorpe
2024-07-31 9:20 ` [LTP] [PATCH 3/3] testcaes/lib: Add shell loader Cyril Hrubis
2024-07-31 10:06 ` [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
3 siblings, 1 reply; 10+ messages in thread
From: Cyril Hrubis @ 2024-07-31 9:20 UTC (permalink / raw)
To: ltp
See: https://github.com/metan-ucw/ujson
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
include/ujson.h | 13 +
include/ujson_common.h | 69 +++
include/ujson_reader.h | 539 ++++++++++++++++++
include/ujson_utf.h | 168 ++++++
include/ujson_writer.h | 224 ++++++++
libs/ujson/Makefile | 12 +
libs/ujson/ujson_common.c | 38 ++
libs/ujson/ujson_reader.c | 1081 +++++++++++++++++++++++++++++++++++++
libs/ujson/ujson_utf.c | 105 ++++
libs/ujson/ujson_writer.c | 491 +++++++++++++++++
10 files changed, 2740 insertions(+)
create mode 100644 include/ujson.h
create mode 100644 include/ujson_common.h
create mode 100644 include/ujson_reader.h
create mode 100644 include/ujson_utf.h
create mode 100644 include/ujson_writer.h
create mode 100644 libs/ujson/Makefile
create mode 100644 libs/ujson/ujson_common.c
create mode 100644 libs/ujson/ujson_reader.c
create mode 100644 libs/ujson/ujson_utf.c
create mode 100644 libs/ujson/ujson_writer.c
diff --git a/include/ujson.h b/include/ujson.h
new file mode 100644
index 000000000..8faeb18f0
--- /dev/null
+++ b/include/ujson.h
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+#ifndef UJSON_H
+#define UJSON_H
+
+#include <ujson_common.h>
+#include <ujson_reader.h>
+#include <ujson_writer.h>
+
+#endif /* UJSON_H */
diff --git a/include/ujson_common.h b/include/ujson_common.h
new file mode 100644
index 000000000..ed31c090d
--- /dev/null
+++ b/include/ujson_common.h
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+/**
+ * @file ujson_common.h
+ * @brief Common JSON reader/writer definitions.
+ */
+
+#ifndef UJSON_COMMON_H
+#define UJSON_COMMON_H
+
+/** @brief Maximal error message length. */
+#define UJSON_ERR_MAX 128
+/** @brief Maximal id string lenght including terminating null element. */
+#define UJSON_ID_MAX 64
+/** @brief Maximal recursion depth allowed. */
+#define UJSON_RECURSION_MAX 128
+
+#define UJSON_ERR_PRINT ujson_err_handler
+#define UJSON_ERR_PRINT_PRIV stderr
+
+/**
+ * @brief A JSON data type.
+ */
+enum ujson_type {
+ /** @brief No type. Returned when parser finishes. */
+ UJSON_VOID = 0,
+ /** @brief An integer. */
+ UJSON_INT,
+ /** @brief A floating point. */
+ UJSON_FLOAT,
+ /** @brief A boolean. */
+ UJSON_BOOL,
+ /** @brief NULL */
+ UJSON_NULL,
+ /** @brief A string. */
+ UJSON_STR,
+ /** @brief A JSON object. */
+ UJSON_OBJ,
+ /** @brief A JSON array. */
+ UJSON_ARR,
+};
+
+/**
+ * @brief Returns type name.
+ *
+ * @param type A json type.
+ * @return A type name.
+ */
+const char *ujson_type_name(enum ujson_type type);
+
+/**
+ * @brief Default error print handler.
+ *
+ * @param print_priv A json buffer print_priv pointer.
+ * @param line A line of text to be printed.
+ */
+void ujson_err_handler(void *print_priv, const char *line);
+
+typedef struct ujson_reader ujson_reader;
+typedef struct ujson_writer ujson_writer;
+typedef struct ujson_val ujson_val;
+
+/** @brief An array size macro. */
+#define UJSON_ARRAY_SIZE(array) (sizeof(array) / sizeof(*array))
+
+#endif /* UJSON_COMMON_H */
diff --git a/include/ujson_reader.h b/include/ujson_reader.h
new file mode 100644
index 000000000..9f105af65
--- /dev/null
+++ b/include/ujson_reader.h
@@ -0,0 +1,539 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+/**
+ * @file ujson_reader.h
+ * @brief A recursive descend JSON parser.
+ *
+ * All the function that parse JSON return zero on success and non-zero on a
+ * failure. Once an error has happened all subsequent attempts to parse more
+ * return with non-zero exit status immediatelly. This is designed so that we
+ * can parse several values without checking each return value and only check
+ * if error has happened at the end of the sequence.
+ */
+
+#ifndef UJSON_READER_H
+#define UJSON_READER_H
+
+#include <stdio.h>
+#include <ujson_common.h>
+
+/**
+ * @brief An ujson_reader initializer with default values.
+ *
+ * @param buf A pointer to a buffer with JSON data.
+ * @param buf_len A JSON data buffer lenght.
+ * @param rflags enum ujson_reader_flags.
+ *
+ * @return An ujson_reader initialized with default values.
+ */
+#define UJSON_READER_INIT(buf, buf_len, rflags) { \
+ .max_depth = UJSON_RECURSION_MAX, \
+ .err_print = UJSON_ERR_PRINT, \
+ .err_print_priv = UJSON_ERR_PRINT_PRIV, \
+ .json = buf, \
+ .len = buf_len, \
+ .flags = rflags \
+}
+
+/** @brief Reader flags. */
+enum ujson_reader_flags {
+ /** @brief If set warnings are treated as errors. */
+ UJSON_READER_STRICT = 0x01,
+};
+
+/**
+ * @brief A JSON parser internal state.
+ */
+struct ujson_reader {
+ /** Pointer to a null terminated JSON string */
+ const char *json;
+ /** A length of the JSON string */
+ size_t len;
+ /** A current offset into the JSON string */
+ size_t off;
+ /** An offset to the start of the last array or object */
+ size_t sub_off;
+ /** Recursion depth increased when array/object is entered decreased on leave */
+ unsigned int depth;
+ /** Maximal recursion depth */
+ unsigned int max_depth;
+
+ /** Reader flags. */
+ enum ujson_reader_flags flags;
+
+ /** Handler to print errors and warnings */
+ void (*err_print)(void *err_print_priv, const char *line);
+ void *err_print_priv;
+
+ char err[UJSON_ERR_MAX];
+ char buf[];
+};
+
+/**
+ * @brief An ujson_val initializer.
+ *
+ * @param sbuf A pointer to a buffer used for string values.
+ * @param sbuf_size A length of the buffer used for string values.
+ *
+ * @return An ujson_val initialized with default values.
+ */
+#define UJSON_VAL_INIT(sbuf, sbuf_size) { \
+ .buf = sbuf, \
+ .buf_size = sbuf_size, \
+}
+
+/**
+ * @brief A parsed JSON key value pair.
+ */
+struct ujson_val {
+ /**
+ * @brief A value type
+ *
+ * UJSON_VALUE_VOID means that no value was parsed.
+ */
+ enum ujson_type type;
+
+ /** An user supplied buffer and size to store a string values to. */
+ char *buf;
+ size_t buf_size;
+
+ /**
+ * @brief An index to attribute list.
+ *
+ * This is set by the ujson_obj_first_filter() and
+ * ujson_obj_next_filter() functions.
+ */
+ size_t idx;
+
+ /** An union to store the parsed value into. */
+ union {
+ /** @brief A boolean value. */
+ int val_bool;
+ /** @brief An integer value. */
+ long long val_int;
+ /** @brief A string value. */
+ const char *val_str;
+ };
+
+ /**
+ * @brief A floating point value.
+ *
+ * Since integer values are subset of floating point values val_float
+ * is always set when val_int was set.
+ */
+ double val_float;
+
+ /** @brief An ID for object values */
+ char id[UJSON_ID_MAX];
+
+ char buf__[];
+};
+
+/**
+ * @brief Allocates a JSON value.
+ *
+ * @param buf_size A maximal buffer size for a string value, pass 0 for default.
+ * @return A newly allocated JSON value.
+ */
+ujson_val *ujson_val_alloc(size_t buf_size);
+
+/**
+ * @brief Frees a JSON value.
+ *
+ * @param self A JSON value previously allocated by ujson_val_alloc().
+ */
+void ujson_val_free(ujson_val *self);
+
+/**
+ * @brief Checks is result has valid type.
+ *
+ * @param res An ujson value.
+ * @return Zero if result is not valid, non-zero otherwise.
+ */
+static inline int ujson_val_valid(struct ujson_val *res)
+{
+ return !!res->type;
+}
+
+/**
+ * @brief Fills the reader error.
+ *
+ * Once buffer error is set all parsing functions return immediatelly with type
+ * set to UJSON_VOID.
+ *
+ * @param self An ujson_reader
+ * @param fmt A printf like format string
+ * @param ... A printf like parameters
+ */
+void ujson_err(ujson_reader *self, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+/**
+ * @brief Prints error stored in the buffer.
+ *
+ * The error takes into consideration the current offset in the buffer and
+ * prints a few preceding lines along with the exact position of the error.
+ *
+ * The error is passed to the err_print() handler.
+ *
+ * @param self A ujson_reader
+ */
+void ujson_err_print(ujson_reader *self);
+
+/**
+ * @brief Prints a warning.
+ *
+ * Uses the print handler in the buffer to print a warning along with a few
+ * lines of context from the JSON at the current position.
+ *
+ * @param self A ujson_reader
+ * @param fmt A printf-like error string.
+ * @param ... A printf-like parameters.
+ */
+void ujson_warn(ujson_reader *self, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+/**
+ * @brief Returns true if error was encountered.
+ *
+ * @param self A ujson_reader
+ * @return True if error was encountered false otherwise.
+ */
+static inline int ujson_reader_err(ujson_reader *self)
+{
+ return !!self->err[0];
+}
+
+/**
+ * @brief Returns the type of next element in buffer.
+ *
+ * @param self An ujson_reader
+ * @return A type of next element in the buffer.
+ */
+enum ujson_type ujson_next_type(ujson_reader *self);
+
+/**
+ * @brief Returns if first element in JSON is object or array.
+ *
+ * @param self A ujson_reader
+ * @return On success returns UJSON_OBJ or UJSON_ARR. On failure UJSON_VOID.
+ */
+enum ujson_type ujson_reader_start(ujson_reader *self);
+
+/**
+ * @brief Starts parsing of a JSON object.
+ *
+ * @param self An ujson_reader
+ * @param res An ujson_val to store the parsed value to.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_first(ujson_reader *self, struct ujson_val *res);
+
+/**
+ * @brief Parses next value from a JSON object.
+ *
+ * If the res->type is UJSON_OBJ or UJSON_ARR it has to be parsed or skipped
+ * before next call to this function.
+ *
+ * @param self An ujson_reader.
+ * @param res A ujson_val to store the parsed value to.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_next(ujson_reader *self, struct ujson_val *res);
+
+/**
+ * @brief A loop over a JSON object.
+ *
+ * @code
+ * UJSON_OBJ_FOREACH(reader, val) {
+ * printf("Got value id '%s' type '%s'", val->id, ujson_type_name(val->type));
+ * ...
+ * }
+ * @endcode
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the next parsed value to.
+ */
+#define UJSON_OBJ_FOREACH(self, res) \
+ for (ujson_obj_first(self, res); ujson_val_valid(res); ujson_obj_next(self, res))
+
+/**
+ * @brief Utility function for log(n) lookup in a sorted array.
+ *
+ * @param list Analphabetically sorted array.
+ * @param list_len Array length.
+ *
+ * @return An array index or (size_t)-1 if key wasn't found.
+ */
+size_t ujson_lookup(const void *arr, size_t memb_size, size_t list_len,
+ const char *key);
+
+/**
+ * @brief A JSON object attribute description i.e. key and type.
+ */
+typedef struct ujson_obj_attr {
+ /** @brief A JSON object key name. */
+ const char *key;
+ /**
+ * @brief A JSON object value type.
+ *
+ * Note that because integer numbers are subset of floating point
+ * numbers if requested type was UJSON_FLOAT it will match if parsed
+ * type was UJSON_INT and the val_float will be set in addition to
+ * val_int.
+ */
+ enum ujson_type type;
+} ujson_obj_attr;
+
+/** @brief A JSON object description */
+typedef struct ujson_obj {
+ /**
+ * @brief A list of attributes.
+ *
+ * Attributes we are looking for, the parser sets the val->idx for these.
+ */
+ const ujson_obj_attr *attrs;
+ /** @brief A size of attrs array. */
+ size_t attr_cnt;
+} ujson_obj;
+
+static inline size_t ujson_obj_lookup(const ujson_obj *obj, const char *key)
+{
+ return ujson_lookup(obj->attrs, sizeof(*obj->attrs), obj->attr_cnt, key);
+}
+
+/** @brief An ujson_obj_attr initializer. */
+#define UJSON_OBJ_ATTR(keyv, typev) \
+ {.key = keyv, .type = typev}
+
+/**
+ * @brief Starts parsing of a JSON object with attribute lists.
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the parsed value to.
+ * @param obj An ujson_obj object description.
+ * @param ign A list of keys to ignore.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_first_filter(ujson_reader *self, struct ujson_val *res,
+ const struct ujson_obj *obj, const struct ujson_obj *ign);
+
+/**
+ * @brief An empty object attribute list.
+ *
+ * To be passed to UJSON_OBJ_FOREACH_FITLER() as ignore list.
+ */
+extern const struct ujson_obj *ujson_empty_obj;
+
+/**
+ * @brief Parses next value from a JSON object with attribute lists.
+ *
+ * If the res->type is UJSON_OBJ or UJSON_ARR it has to be parsed or skipped
+ * before next call to this function.
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the parsed value to.
+ * @param obj An ujson_obj object description.
+ * @param ign A list of keys to ignore. If set to NULL all unknown keys are
+ * ignored, if set to ujson_empty_obj all unknown keys produce warnings.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_next_filter(ujson_reader *self, struct ujson_val *res,
+ const struct ujson_obj *obj, const struct ujson_obj *ign);
+
+/**
+ * @brief A loop over a JSON object with a pre-defined list of expected attributes.
+ *
+ * @code
+ * static struct ujson_obj_attr attrs[] = {
+ * UJSON_OBJ_ATTR("bool", UJSON_BOOL),
+ * UJSON_OBJ_ATTR("number", UJSON_INT),
+ * };
+ *
+ * static struct ujson_obj obj = {
+ * .attrs = filter_attrs,
+ * .attr_cnt = UJSON_ARRAY_SIZE(filter_attrs)
+ * };
+ *
+ * UJSON_OBJ_FOREACH_FILTER(reader, val, &obj, NULL) {
+ * printf("Got value id '%s' type '%s'",
+ * attrs[val->idx].id, ujson_type_name(val->type));
+ * ...
+ * }
+ * @endcode
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the next parsed value to.
+ * @param obj An ujson_obj with a description of attributes to parse.
+ * @param ign An ujson_obj with a description of attributes to ignore.
+ */
+#define UJSON_OBJ_FOREACH_FILTER(self, res, obj, ign) \
+ for (ujson_obj_first_filter(self, res, obj, ign); \
+ ujson_val_valid(res); \
+ ujson_obj_next_filter(self, res, obj, ign))
+
+/**
+ * @brief Skips parsing of a JSON object.
+ *
+ * @param self An ujson_reader.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_skip(ujson_reader *self);
+
+/**
+ * @brief Starts parsing of a JSON array.
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the parsed value to.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_arr_first(ujson_reader *self, struct ujson_val *res);
+
+/**
+ * @brief Parses next value from a JSON array.
+ *
+ * If the res->type is UJSON_OBJ or UJSON_ARR it has to be parsed or skipped
+ * before next call to this function.
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the parsed value to.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_arr_next(ujson_reader *self, struct ujson_val *res);
+
+/**
+ * @brief A loop over a JSON array.
+ *
+ * @code
+ * UJSON_ARR_FOREACH(reader, val) {
+ * printf("Got value type '%s'", ujson_type_name(val->type));
+ * ...
+ * }
+ * @endcode
+ *
+ * @param self An ujson_reader.
+ * @param res An ujson_val to store the next parsed value to.
+ */
+#define UJSON_ARR_FOREACH(self, res) \
+ for (ujson_arr_first(self, res); ujson_val_valid(res); ujson_arr_next(self, res))
+
+/**
+ * @brief Skips parsing of a JSON array.
+ *
+ * @param self A ujson_reader.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_arr_skip(ujson_reader *self);
+
+/**
+ * @brief A JSON reader state.
+ */
+typedef struct ujson_reader_state {
+ size_t off;
+ unsigned int depth;
+} ujson_reader_state;
+
+/**
+ * @brief Returns a parser state at the start of current object/array.
+ *
+ * This function could be used for the parser to return to the start of the
+ * currently parsed object or array.
+ *
+ * @param self A ujson_reader
+ * @return A state that points to a start of the last object or array.
+ */
+static inline ujson_reader_state ujson_reader_state_save(ujson_reader *self)
+{
+ struct ujson_reader_state ret = {
+ .off = self->sub_off,
+ .depth = self->depth,
+ };
+
+ return ret;
+}
+
+/**
+ * @brief Returns the parser to a saved state.
+ *
+ * This function could be used for the parser to return to the start of
+ * object or array saved by t the ujson_reader_state_get() function.
+ *
+ * @param self A ujson_reader
+ * @param state An parser state as returned by the ujson_reader_state_get().
+ */
+static inline void ujson_reader_state_load(ujson_reader *self, ujson_reader_state state)
+{
+ if (ujson_reader_err(self))
+ return;
+
+ self->off = state.off;
+ self->sub_off = state.off;
+ self->depth = state.depth;
+}
+
+/**
+ * @brief Resets the parser to a start.
+ *
+ * @param self A ujson_reader
+ */
+static inline void ujson_reader_reset(ujson_reader *self)
+{
+ self->off = 0;
+ self->sub_off = 0;
+ self->depth = 0;
+ self->err[0] = 0;
+}
+
+/**
+ * @brief Loads a file into an ujson_reader buffer.
+ *
+ * The reader has to be later freed by ujson_reader_free().
+ *
+ * @param path A path to a file.
+ * @return A ujson_reader or NULL in a case of a failure.
+ */
+ujson_reader *ujson_reader_load(const char *path);
+
+/**
+ * @brief Frees an ujson_reader buffer.
+ *
+ * @param self A ujson_reader allocated by ujson_reader_load() function.
+ */
+void ujson_reader_free(ujson_reader *self);
+
+/**
+ * @brief Prints errors and warnings at the end of parsing.
+ *
+ * Checks if self->err is set and prints the error with ujson_reader_err()
+ *
+ * Checks if there is any text left after the parser has finished with
+ * ujson_reader_consumed() and prints a warning if there were any non-whitespace
+ * characters left.
+ *
+ * @param self A ujson_reader
+ */
+void ujson_reader_finish(ujson_reader *self);
+
+/**
+ * @brief Returns non-zero if whole buffer has been consumed.
+ *
+ * @param self A ujson_reader.
+ * @return Non-zero if whole buffer was consumed.
+ */
+static inline int ujson_reader_consumed(ujson_reader *self)
+{
+ return self->off >= self->len;
+}
+
+#endif /* UJSON_H */
diff --git a/include/ujson_utf.h b/include/ujson_utf.h
new file mode 100644
index 000000000..f939fbe8c
--- /dev/null
+++ b/include/ujson_utf.h
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2022-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+/**
+ * @file ujson_utf.h
+ * @brief Unicode helper macros and functions.
+ */
+
+#ifndef UJSON_UTF_H
+#define UJSON_UTF_H
+
+#include <stdint.h>
+#include <stddef.h>
+
+/** Returns true if unicode byte is ASCII */
+#define UJSON_UTF8_IS_ASCII(ch) (!((ch) & 0x80))
+/** Returns true if we have first unicode byte of single byte sequence */
+#define UJSON_UTF8_IS_NBYTE(ch) (((ch) & 0xc0) == 0x80)
+/** Returns true if we have first unicode byte of two byte sequence */
+#define UJSON_UTF8_IS_2BYTE(ch) (((ch) & 0xe0) == 0xc0)
+/** Returns true if we have first unicode byte of three byte sequence */
+#define UJSON_UTF8_IS_3BYTE(ch) (((ch) & 0xf0) == 0xe0)
+/** Returns true if we have first unicode byte of four byte sequence */
+#define UJSON_UTF8_IS_4BYTE(ch) (((ch) & 0xf8) == 0xf0)
+
+#define UJSON_UTF8_NBYTE_MASK 0x3f
+
+/**
+ * @brief Parses next unicode character in UTF-8 string.
+ * @param str A pointer to the C string.
+ * @return A unicode character or 0 on error or end of the string.
+ */
+static inline uint32_t ujson_utf8_next(const char **str)
+{
+ uint32_t s0 = *str[0];
+
+ (*str)++;
+
+ if (UJSON_UTF8_IS_ASCII(s0))
+ return s0;
+
+ uint32_t s1 = *str[0];
+
+ if (!UJSON_UTF8_IS_NBYTE(s1))
+ return 0;
+
+ s1 &= UJSON_UTF8_NBYTE_MASK;
+
+ (*str)++;
+
+ if (UJSON_UTF8_IS_2BYTE(s0))
+ return (s0 & 0x1f)<<6 | s1;
+
+ uint32_t s2 = *str[0];
+
+ if (!UJSON_UTF8_IS_NBYTE(s2))
+ return 0;
+
+ s2 &= UJSON_UTF8_NBYTE_MASK;
+
+ (*str)++;
+
+ if (UJSON_UTF8_IS_3BYTE(s0))
+ return (s0 & 0x0f)<<12 | s1<<6 | s2;
+
+ (*str)++;
+
+ uint32_t s3 = *str[0];
+
+ if (!UJSON_UTF8_IS_NBYTE(s2))
+ return 0;
+
+ s3 &= UJSON_UTF8_NBYTE_MASK;
+
+ if (UJSON_UTF8_IS_4BYTE(s0))
+ return (s0 & 0x07)<<18 | s1<<12 | s2<<6 | s3;
+
+ return 0;
+}
+
+/**
+ * @brief Returns number of bytes next character is occupying in an UTF-8 string.
+ *
+ * @param str A pointer to a string.
+ * @param off An offset into the string, must point to a valid multibyte boundary.
+ * @return Number of bytes next character occupies, zero on string end and -1 on failure.
+ */
+int8_t ujson_utf8_next_chsz(const char *str, size_t off);
+
+/**
+ * @brief Returns number of bytes previous character is occupying in an UTF-8 string.
+ *
+ * @param str A pointer to a string.
+ * @param off An offset into the string, must point to a valid multibyte boundary.
+ * @return Number of bytes previous character occupies, and -1 on failure.
+ */
+int8_t ujson_utf8_prev_chsz(const char *str, size_t off);
+
+/**
+ * @brief Returns a number of characters in UTF-8 string.
+ *
+ * Returns number of characters in an UTF-8 string, which may be less or equal
+ * to what strlen() reports.
+ *
+ * @param str An UTF-8 string.
+ * @return Number of characters in the string.
+ */
+size_t ujson_utf8_strlen(const char *str);
+
+/**
+ * @brief Returns a number of bytes needed to store unicode character into UTF-8.
+ *
+ * @param unicode A unicode character.
+ * @return Number of utf8 bytes required to store a unicode character.
+ */
+static inline unsigned int ujson_utf8_bytes(uint32_t unicode)
+{
+ if (unicode < 0x0080)
+ return 1;
+
+ if (unicode < 0x0800)
+ return 2;
+
+ if (unicode < 0x10000)
+ return 3;
+
+ return 4;
+}
+
+/**
+ * @brief Writes an unicode character into a UTF-8 buffer.
+ *
+ * The buffer _must_ be large enough!
+ *
+ * @param unicode A unicode character.
+ * @param buf A byte buffer.
+ * @return A number of bytes written.
+ */
+static inline int ujson_to_utf8(uint32_t unicode, char *buf)
+{
+ if (unicode < 0x0080) {
+ buf[0] = unicode & 0x007f;
+ return 1;
+ }
+
+ if (unicode < 0x0800) {
+ buf[0] = 0xc0 | (0x1f & (unicode>>6));
+ buf[1] = 0x80 | (0x3f & unicode);
+ return 2;
+ }
+
+ if (unicode < 0x10000) {
+ buf[0] = 0xe0 | (0x0f & (unicode>>12));
+ buf[1] = 0x80 | (0x3f & (unicode>>6));
+ buf[2] = 0x80 | (0x3f & unicode);
+ return 3;
+ }
+
+ buf[0] = 0xf0 | (0x07 & (unicode>>18));
+ buf[1] = 0x80 | (0x3f & (unicode>>12));
+ buf[2] = 0x80 | (0x3f & (unicode>>6));
+ buf[3] = 0x80 | (0x3f & unicode);
+ return 4;
+}
+
+#endif /* UJSON_UTF_H */
diff --git a/include/ujson_writer.h b/include/ujson_writer.h
new file mode 100644
index 000000000..dfcc95053
--- /dev/null
+++ b/include/ujson_writer.h
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+/**
+ * @file ujson_writer.h
+ * @brief A JSON writer.
+ *
+ * All the function that add values return zero on success and non-zero on a
+ * failure. Once an error has happened all subsequent attempts to add more
+ * values return with non-zero exit status immediatelly. This is designed
+ * so that we can add several values without checking each return value
+ * and only check if error has happened at the end of the sequence.
+ *
+ * Failures may occur:
+ * - if we call the functions out of order, e.g. attempt to finish array when
+ * we are not writing out an array.
+ * - if we run out of recursion stack
+ * - may be propagated from the writer function, e.g. allocation failure, no
+ * space on disk, etc.
+ */
+
+#ifndef UJSON_WRITER_H
+#define UJSON_WRITER_H
+
+#include <ujson_common.h>
+
+/** @brief A JSON writer */
+struct ujson_writer {
+ unsigned int depth;
+ char depth_type[UJSON_RECURSION_MAX/8];
+ char depth_first[UJSON_RECURSION_MAX/8];
+
+ /** Handler to print errors and warnings */
+ void (*err_print)(void *err_print_priv, const char *line);
+ void *err_print_priv;
+ char err[UJSON_ERR_MAX];
+
+ /** Handler to produce JSON output */
+ int (*out)(struct ujson_writer *self, const char *buf, size_t buf_size);
+ void *out_priv;
+};
+
+/**
+ * @brief An ujson_writer initializer with default values.
+ *
+ * @param vout A pointer to function to write out the data.
+ * @param vout_priv An user pointer passed to the out function.
+ *
+ * @return An ujson_writer initialized with default values.
+ */
+#define UJSON_WRITER_INIT(vout, vout_priv) { \
+ .err_print = UJSON_ERR_PRINT, \
+ .err_print_priv = UJSON_ERR_PRINT_PRIV, \
+ .out = vout, \
+ .out_priv = vout_priv \
+}
+
+/**
+ * @brief Allocates a JSON file writer.
+ *
+ * The call may fail either when file cannot be opened for writing or if
+ * allocation has failed. In all cases errno should be set correctly.
+ *
+ * @param path A path to the file, file is opened for writing and created if it
+ * does not exist.
+ *
+ * @return A ujson_writer pointer or NULL in a case of failure.
+ */
+ujson_writer *ujson_writer_file_open(const char *path);
+
+/**
+ * @brief Closes and frees a JSON file writer.
+ *
+ * @param self A ujson_writer file writer.
+ *
+ * @return Zero on success, non-zero on a failure and errno is set.
+ */
+int ujson_writer_file_close(ujson_writer *self);
+
+/**
+ * @brief Returns true if writer error happened.
+ *
+ * @param self A JSON writer.
+ *
+ * @return True if error has happened.
+ */
+static inline int ujson_writer_err(ujson_writer *self)
+{
+ return !!self->err[0];
+}
+
+/**
+ * @brief Starts a JSON object.
+ *
+ * For a top level object the id must be NULL, every other object has to have
+ * non-NULL id. The call will also fail if maximal recursion depth
+ * UJSON_RECURSION_MAX has been reached.
+ *
+ * @param self A JSON writer.
+ * @param id An object name.
+ *
+ * @return Zero on a success, non-zero otherwise.
+ */
+int ujson_obj_start(ujson_writer *self, const char *id);
+
+/**
+ * @brief Finishes a JSON object.
+ *
+ * The call will fail if we are currenlty not writing out an object.
+ *
+ * @param self A JSON writer.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_obj_finish(ujson_writer *self);
+
+/**
+ * @brief Starts a JSON array.
+ *
+ * For a top level array the id must be NULL, every other array has to have
+ * non-NULL id. The call will also fail if maximal recursion depth
+ * UJSON_RECURSION_MAX has been reached.
+ *
+ * @param self A JSON writer.
+ * @param id An array name.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_arr_start(ujson_writer *self, const char *id);
+
+/**
+ * @brief Finishes a JSON array.
+ *
+ * The call will fail if we are currenlty not writing out an array.
+ *
+ * @param self A JSON writer.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_arr_finish(ujson_writer *self);
+
+/**
+ * @brief Adds a null value.
+ *
+ * The id must be NULL inside of an array, and must be non-NULL inside of an
+ * object.
+ *
+ * @param self A JSON writer.
+ * @param id A null value name.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_null_add(ujson_writer *self, const char *id);
+
+/**
+ * @brief Adds an integer value.
+ *
+ * The id must be NULL inside of an array, and must be non-NULL inside of an
+ * object.
+ *
+ * @param self A JSON writer.
+ * @param id An integer value name.
+ * @param val An integer value.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_int_add(ujson_writer *self, const char *id, long val);
+
+/**
+ * @brief Adds a bool value.
+ *
+ * The id must be NULL inside of an array, and must be non-NULL inside of an
+ * object.
+ *
+ * @param self A JSON writer.
+ * @param id An boolean value name.
+ * @param val A boolean value.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_bool_add(ujson_writer *self, const char *id, int val);
+
+/**
+ * @brief Adds a float value.
+ *
+ * The id must be NULL inside of an array, and must be non-NULL inside of an
+ * object.
+ *
+ * @param self A JSON writer.
+ * @param id A floating point value name.
+ * @param val A floating point value.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_float_add(ujson_writer *self, const char *id, double val);
+
+/**
+ * @brief Adds a string value.
+ *
+ * The id must be NULL inside of an array, and must be non-NULL inside of an
+ * object.
+ *
+ * @param self A JSON writer.
+ * @param id A string value name.
+ * @param str An UTF8 string value.
+ *
+ * @return Zero on success, non-zero otherwise.
+ */
+int ujson_str_add(ujson_writer *self, const char *id, const char *str);
+
+/**
+ * @brief Finalizes json writer.
+ *
+ * Finalizes the json writer, throws possible errors through the error printing
+ * function.
+ *
+ * @param self A JSON writer.
+ * @return Overall error value.
+ */
+int ujson_writer_finish(ujson_writer *self);
+
+#endif /* UJSON_WRITER_H */
diff --git a/libs/ujson/Makefile b/libs/ujson/Makefile
new file mode 100644
index 000000000..4c8508010
--- /dev/null
+++ b/libs/ujson/Makefile
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright (C) Cyril Hrubis <chrubis@suse.cz>
+
+top_srcdir ?= ../..
+
+include $(top_srcdir)/include/mk/env_pre.mk
+
+INTERNAL_LIB := libujson.a
+
+include $(top_srcdir)/include/mk/lib.mk
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/libs/ujson/ujson_common.c b/libs/ujson/ujson_common.c
new file mode 100644
index 000000000..c9cada9a9
--- /dev/null
+++ b/libs/ujson/ujson_common.c
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+#include <stdio.h>
+#include "ujson_common.h"
+
+void ujson_err_handler(void *err_print_priv, const char *line)
+{
+ fputs(line, err_print_priv);
+ putc('\n', err_print_priv);
+}
+
+const char *ujson_type_name(enum ujson_type type)
+{
+ switch (type) {
+ case UJSON_VOID:
+ return "void";
+ case UJSON_INT:
+ return "integer";
+ case UJSON_FLOAT:
+ return "float";
+ case UJSON_BOOL:
+ return "boolean";
+ case UJSON_NULL:
+ return "null";
+ case UJSON_STR:
+ return "string";
+ case UJSON_OBJ:
+ return "object";
+ case UJSON_ARR:
+ return "array";
+ default:
+ return "invalid";
+ }
+}
+
diff --git a/libs/ujson/ujson_reader.c b/libs/ujson/ujson_reader.c
new file mode 100644
index 000000000..d508f00d3
--- /dev/null
+++ b/libs/ujson/ujson_reader.c
@@ -0,0 +1,1081 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <stdint.h>
+
+#include "ujson_utf.h"
+#include "ujson_reader.h"
+
+static const struct ujson_obj empty = {};
+const struct ujson_obj *ujson_empty_obj = ∅
+
+static inline int buf_empty(ujson_reader *buf)
+{
+ return buf->off >= buf->len;
+}
+
+static int eatws(ujson_reader *buf)
+{
+ while (!buf_empty(buf)) {
+ switch (buf->json[buf->off]) {
+ case ' ':
+ case '\t':
+ case '\n':
+ case '\r':
+ break;
+ default:
+ goto ret;
+ }
+
+ buf->off += 1;
+ }
+ret:
+ return buf_empty(buf);
+}
+
+static char getb(ujson_reader *buf)
+{
+ if (buf_empty(buf))
+ return 0;
+
+ return buf->json[buf->off++];
+}
+
+static char peekb_off(ujson_reader *buf, size_t off)
+{
+ if (buf->off + off >= buf->len)
+ return 0;
+
+ return buf->json[buf->off + off];
+}
+
+static char peekb(ujson_reader *buf)
+{
+ if (buf_empty(buf))
+ return 0;
+
+ return buf->json[buf->off];
+}
+
+static int eatb(ujson_reader *buf, char ch)
+{
+ if (peekb(buf) != ch)
+ return 0;
+
+ getb(buf);
+ return 1;
+}
+
+static int eatb2(ujson_reader *buf, char ch1, char ch2)
+{
+ if (peekb(buf) != ch1 && peekb(buf) != ch2)
+ return 0;
+
+ getb(buf);
+ return 1;
+}
+
+static int eatstr(ujson_reader *buf, const char *str)
+{
+ while (*str) {
+ if (!eatb(buf, *str))
+ return 0;
+ str++;
+ }
+
+ return 1;
+}
+
+static int hex2val(unsigned char b)
+{
+ switch (b) {
+ case '0' ... '9':
+ return b - '0';
+ case 'a' ... 'f':
+ return b - 'a' + 10;
+ case 'A' ... 'F':
+ return b - 'A' + 10;
+ default:
+ return -1;
+ }
+}
+
+static int32_t parse_ucode_cp(ujson_reader *buf)
+{
+ int ret = 0, v, i;
+
+ for (i = 0; i < 4; i++) {
+ if ((v = hex2val(getb(buf))) < 0)
+ goto err;
+ ret *= 16;
+ ret += v;
+ }
+
+ return ret;
+err:
+ ujson_err(buf, "Expected four hexadecimal digits");
+ return -1;
+}
+
+static unsigned int parse_ucode_esc(ujson_reader *buf, char *str,
+ size_t off, size_t len)
+{
+ int32_t ucode = parse_ucode_cp(buf);
+
+ if (ucode < 0)
+ return 0;
+
+ if (!str)
+ return ucode;
+
+ if (ujson_utf8_bytes(ucode) + 1 >= len - off) {
+ ujson_err(buf, "String buffer too short!");
+ return 0;
+ }
+
+ return ujson_to_utf8(ucode, str+off);
+}
+
+static int copy_str(ujson_reader *buf, char *str, size_t len)
+{
+ size_t pos = 0;
+ int esc = 0;
+ unsigned int l;
+
+ eatb(buf, '"');
+
+ for (;;) {
+ if (buf_empty(buf)) {
+ ujson_err(buf, "Unterminated string");
+ return 1;
+ }
+
+ if (!esc && eatb(buf, '"')) {
+ if (str)
+ str[pos] = 0;
+ return 0;
+ }
+
+ unsigned char b = getb(buf);
+
+ if (b < 0x20) {
+ if (!peekb(buf))
+ ujson_err(buf, "Unterminated string");
+ else
+ ujson_err(buf, "Invalid string character 0x%02x", b);
+ return 1;
+ }
+
+ if (!esc && b == '\\') {
+ esc = 1;
+ continue;
+ }
+
+ if (esc) {
+ switch (b) {
+ case '"':
+ case '\\':
+ case '/':
+ break;
+ case 'b':
+ b = '\b';
+ break;
+ case 'f':
+ b = '\f';
+ break;
+ case 'n':
+ b = '\n';
+ break;
+ case 'r':
+ b = '\r';
+ break;
+ case 't':
+ b = '\t';
+ break;
+ case 'u':
+ if (!(l = parse_ucode_esc(buf, str, pos, len)))
+ return 1;
+ pos += l;
+ b = 0;
+ break;
+ default:
+ ujson_err(buf, "Invalid escape \\%c", b);
+ return 1;
+ }
+ esc = 0;
+ }
+
+ if (str && b) {
+ if (pos + 1 >= len) {
+ ujson_err(buf, "String buffer too short!");
+ return 1;
+ }
+
+ str[pos++] = b;
+ }
+ }
+
+ return 1;
+}
+
+static int copy_id_str(ujson_reader *buf, char *str, size_t len)
+{
+ size_t pos = 0;
+
+ if (eatws(buf))
+ goto err0;
+
+ if (!eatb(buf, '"'))
+ goto err0;
+
+ for (;;) {
+ if (buf_empty(buf)) {
+ ujson_err(buf, "Unterminated ID string");
+ return 1;
+ }
+
+ if (eatb(buf, '"')) {
+ str[pos] = 0;
+ break;
+ }
+
+ if (pos >= len-1) {
+ ujson_err(buf, "ID string too long");
+ return 1;
+ }
+
+ str[pos++] = getb(buf);
+ }
+
+ if (eatws(buf))
+ goto err1;
+
+ if (!eatb(buf, ':'))
+ goto err1;
+
+ return 0;
+err0:
+ ujson_err(buf, "Expected ID string");
+ return 1;
+err1:
+ ujson_err(buf, "Expected ':' after ID string");
+ return 1;
+}
+
+static int is_digit(char b)
+{
+ switch (b) {
+ case '0' ... '9':
+ return 1;
+ default:
+ return 0;
+ }
+}
+
+static int get_int(ujson_reader *buf, struct ujson_val *res)
+{
+ long val = 0;
+ int sign = 1;
+
+ if (eatb(buf, '-')) {
+ sign = -1;
+ if (!is_digit(peekb(buf))) {
+ ujson_err(buf, "Expected digit(s)");
+ return 1;
+ }
+ }
+
+ if (peekb(buf) == '0' && is_digit(peekb_off(buf, 1))) {
+ ujson_err(buf, "Leading zero in number!");
+ return 1;
+ }
+
+ while (is_digit(peekb(buf))) {
+ val *= 10;
+ val += getb(buf) - '0';
+ //TODO: overflow?
+ }
+
+ if (sign < 0)
+ val = -val;
+
+ res->val_int = val;
+ res->val_float = val;
+
+ return 0;
+}
+
+static int eat_digits(ujson_reader *buf)
+{
+ if (!is_digit(peekb(buf))) {
+ ujson_err(buf, "Expected digit(s)");
+ return 1;
+ }
+
+ while (is_digit(peekb(buf)))
+ getb(buf);
+
+ return 0;
+}
+
+static int get_float(ujson_reader *buf, struct ujson_val *res)
+{
+ off_t start = buf->off;
+
+ eatb(buf, '-');
+
+ if (peekb(buf) == '0' && is_digit(peekb_off(buf, 1))) {
+ ujson_err(buf, "Leading zero in float");
+ return 1;
+ }
+
+ if (eat_digits(buf))
+ return 1;
+
+ switch (getb(buf)) {
+ case '.':
+ if (eat_digits(buf))
+ return 1;
+
+ if (!eatb2(buf, 'e', 'E'))
+ break;
+
+ /* fallthrough */
+ case 'e':
+ case 'E':
+ eatb2(buf, '+', '-');
+
+ if (eat_digits(buf))
+ return 1;
+ break;
+ }
+
+ size_t len = buf->off - start;
+ char tmp[len+1];
+
+ memcpy(tmp, buf->json + start, len);
+
+ tmp[len] = 0;
+
+ res->val_float = strtod(tmp, NULL);
+
+ return 0;
+}
+
+static int get_bool(ujson_reader *buf, struct ujson_val *res)
+{
+ switch (peekb(buf)) {
+ case 'f':
+ if (!eatstr(buf, "false")) {
+ ujson_err(buf, "Expected 'false'");
+ return 1;
+ }
+
+ res->val_bool = 0;
+ break;
+ case 't':
+ if (!eatstr(buf, "true")) {
+ ujson_err(buf, "Expected 'true'");
+ return 1;
+ }
+
+ res->val_bool = 1;
+ break;
+ }
+
+ return 0;
+}
+
+static int get_null(ujson_reader *buf)
+{
+ if (!eatstr(buf, "null")) {
+ ujson_err(buf, "Expected 'null'");
+ return 1;
+ }
+
+ return 0;
+}
+
+int ujson_obj_skip(ujson_reader *buf)
+{
+ struct ujson_val res = {};
+
+ UJSON_OBJ_FOREACH(buf, &res) {
+ switch (res.type) {
+ case UJSON_OBJ:
+ if (ujson_obj_skip(buf))
+ return 1;
+ break;
+ case UJSON_ARR:
+ if (ujson_arr_skip(buf))
+ return 1;
+ break;
+ default:
+ break;
+ }
+ }
+
+ return 0;
+}
+
+int ujson_arr_skip(ujson_reader *buf)
+{
+ struct ujson_val res = {};
+
+ UJSON_ARR_FOREACH(buf, &res) {
+ switch (res.type) {
+ case UJSON_OBJ:
+ if (ujson_obj_skip(buf))
+ return 1;
+ break;
+ case UJSON_ARR:
+ if (ujson_arr_skip(buf))
+ return 1;
+ break;
+ default:
+ break;
+ }
+ }
+
+ return 0;
+}
+
+static enum ujson_type next_num_type(ujson_reader *buf)
+{
+ size_t off = 0;
+
+ for (;;) {
+ char b = peekb_off(buf, off++);
+
+ switch (b) {
+ case 0:
+ case ',':
+ return UJSON_INT;
+ case '.':
+ case 'e':
+ case 'E':
+ return UJSON_FLOAT;
+ }
+ }
+
+ return UJSON_VOID;
+}
+
+enum ujson_type ujson_next_type(ujson_reader *buf)
+{
+ if (eatws(buf)) {
+ ujson_err(buf, "Unexpected end");
+ return UJSON_VOID;
+ }
+
+ char b = peekb(buf);
+
+ switch (b) {
+ case '{':
+ return UJSON_OBJ;
+ case '[':
+ return UJSON_ARR;
+ case '"':
+ return UJSON_STR;
+ case '-':
+ case '0' ... '9':
+ return next_num_type(buf);
+ case 'f':
+ case 't':
+ return UJSON_BOOL;
+ break;
+ case 'n':
+ return UJSON_NULL;
+ break;
+ default:
+ ujson_err(buf, "Expected object, array, number or string");
+ return UJSON_VOID;
+ }
+}
+
+enum ujson_type ujson_reader_start(ujson_reader *buf)
+{
+ enum ujson_type type = ujson_next_type(buf);
+
+ switch (type) {
+ case UJSON_ARR:
+ case UJSON_OBJ:
+ case UJSON_VOID:
+ break;
+ default:
+ ujson_err(buf, "JSON can start only with array or object");
+ type = UJSON_VOID;
+ break;
+ }
+
+ return type;
+}
+
+static int get_value(ujson_reader *buf, struct ujson_val *res)
+{
+ int ret = 0;
+
+ res->type = ujson_next_type(buf);
+
+ switch (res->type) {
+ case UJSON_STR:
+ if (copy_str(buf, res->buf, res->buf_size)) {
+ res->type = UJSON_VOID;
+ return 0;
+ }
+ res->val_str = res->buf;
+ return 1;
+ case UJSON_INT:
+ ret = get_int(buf, res);
+ break;
+ case UJSON_FLOAT:
+ ret = get_float(buf, res);
+ break;
+ case UJSON_BOOL:
+ ret = get_bool(buf, res);
+ break;
+ case UJSON_NULL:
+ ret = get_null(buf);
+ break;
+ case UJSON_VOID:
+ return 0;
+ case UJSON_ARR:
+ case UJSON_OBJ:
+ buf->sub_off = buf->off;
+ return 1;
+ }
+
+ if (ret) {
+ res->type = UJSON_VOID;
+ return 0;
+ }
+
+ return 1;
+}
+
+static int pre_next(ujson_reader *buf, struct ujson_val *res)
+{
+ if (!eatb(buf, ',')) {
+ ujson_err(buf, "Expected ','");
+ res->type = UJSON_VOID;
+ return 1;
+ }
+
+ if (eatws(buf)) {
+ ujson_err(buf, "Unexpected end");
+ res->type = UJSON_VOID;
+ return 1;
+ }
+
+ return 0;
+}
+
+static int check_end(ujson_reader *buf, struct ujson_val *res, char b)
+{
+ if (eatws(buf)) {
+ ujson_err(buf, "Unexpected end");
+ return 1;
+ }
+
+ if (eatb(buf, b)) {
+ res->type = UJSON_VOID;
+ eatws(buf);
+ eatb(buf, 0);
+ buf->depth--;
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * This is supposed to return a pointer to a string stored as a first member of
+ * a structure given an array.
+ *
+ * e.g.
+ *
+ * struct foo {
+ * const char *key;
+ * ...
+ * };
+ *
+ * const struct foo bar[10] = {...};
+ *
+ * // Returns a pointer to the key string in a second structure in bar[].
+ * const char *key = list_elem(bar, sizeof(struct foo), 1);
+ */
+static inline const char *list_elem(const void *arr, size_t memb_size, size_t idx)
+{
+ return *(const char**)(arr + idx * memb_size);
+}
+
+size_t ujson_lookup(const void *arr, size_t memb_size, size_t list_len,
+ const char *key)
+{
+ size_t l = 0;
+ size_t r = list_len-1;
+ size_t mid = -1;
+
+ if (!list_len)
+ return (size_t)-1;
+
+ while (r - l > 1) {
+ mid = (l+r)/2;
+
+ int ret = strcmp(list_elem(arr, memb_size, mid), key);
+ if (!ret)
+ return mid;
+
+ if (ret < 0)
+ l = mid;
+ else
+ r = mid;
+ }
+
+ if (r != mid && !strcmp(list_elem(arr, memb_size, r), key))
+ return r;
+
+ if (l != mid && !strcmp(list_elem(arr, memb_size, l), key))
+ return l;
+
+ return -1;
+}
+
+static int skip_obj_val(ujson_reader *buf)
+{
+ struct ujson_val dummy = {};
+
+ if (!get_value(buf, &dummy))
+ return 0;
+
+ switch (dummy.type) {
+ case UJSON_OBJ:
+ return !ujson_obj_skip(buf);
+ case UJSON_ARR:
+ return !ujson_arr_skip(buf);
+ default:
+ return 1;
+ }
+}
+
+static int obj_next(ujson_reader *buf, struct ujson_val *res)
+{
+ if (copy_id_str(buf, res->id, sizeof(res->id)))
+ return 0;
+
+ return get_value(buf, res);
+}
+
+static int obj_pre_next(ujson_reader *buf, struct ujson_val *res)
+{
+ if (ujson_reader_err(buf))
+ return 1;
+
+ if (check_end(buf, res, '}'))
+ return 1;
+
+ if (pre_next(buf, res))
+ return 1;
+
+ return 0;
+}
+
+static int obj_next_filter(ujson_reader *buf, struct ujson_val *res,
+ const struct ujson_obj *obj, const struct ujson_obj *ign)
+{
+ const struct ujson_obj_attr *attr;
+
+ for (;;) {
+ if (copy_id_str(buf, res->id, sizeof(res->id)))
+ return 0;
+
+ res->idx = obj ? ujson_obj_lookup(obj, res->id) : (size_t)-1;
+
+ if (res->idx != (size_t)-1) {
+ if (!get_value(buf, res))
+ return 0;
+
+ attr = &obj->attrs[res->idx];
+
+ if (attr->type == UJSON_VOID)
+ return 1;
+
+ if (attr->type == res->type)
+ return 1;
+
+ if (attr->type == UJSON_FLOAT &&
+ res->type == UJSON_INT)
+ return 1;
+
+ ujson_warn(buf, "Wrong '%s' type expected %s",
+ attr->key, ujson_type_name(attr->type));
+ } else {
+ if (!skip_obj_val(buf))
+ return 0;
+
+ if (ign && ujson_obj_lookup(ign, res->id) == (size_t)-1)
+ ujson_warn(buf, "Unexpected key '%s'", res->id);
+ }
+
+ if (obj_pre_next(buf, res))
+ return 0;
+ }
+}
+
+static int check_err(ujson_reader *buf, struct ujson_val *res)
+{
+ if (ujson_reader_err(buf)) {
+ res->type = UJSON_VOID;
+ return 1;
+ }
+
+ return 0;
+}
+
+int ujson_obj_next_filter(ujson_reader *buf, struct ujson_val *res,
+ const struct ujson_obj *obj, const struct ujson_obj *ign)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (obj_pre_next(buf, res))
+ return 0;
+
+ return obj_next_filter(buf, res, obj, ign);
+}
+
+int ujson_obj_next(ujson_reader *buf, struct ujson_val *res)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (obj_pre_next(buf, res))
+ return 0;
+
+ return obj_next(buf, res);
+}
+
+static int any_first(ujson_reader *buf, char b)
+{
+ if (eatws(buf)) {
+ ujson_err(buf, "Unexpected end");
+ return 1;
+ }
+
+ if (!eatb(buf, b)) {
+ ujson_err(buf, "Expected '%c'", b);
+ return 1;
+ }
+
+ buf->depth++;
+
+ if (buf->depth > buf->max_depth) {
+ ujson_err(buf, "Recursion too deep");
+ return 1;
+ }
+
+ return 0;
+}
+
+int ujson_obj_first_filter(ujson_reader *buf, struct ujson_val *res,
+ const struct ujson_obj *obj, const struct ujson_obj *ign)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (any_first(buf, '{'))
+ return 0;
+
+ if (check_end(buf, res, '}'))
+ return 0;
+
+ return obj_next_filter(buf, res, obj, ign);
+}
+
+int ujson_obj_first(ujson_reader *buf, struct ujson_val *res)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (any_first(buf, '{'))
+ return 0;
+
+ if (check_end(buf, res, '}'))
+ return 0;
+
+ return obj_next(buf, res);
+}
+
+static int arr_next(ujson_reader *buf, struct ujson_val *res)
+{
+ return get_value(buf, res);
+}
+
+int ujson_arr_first(ujson_reader *buf, struct ujson_val *res)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (any_first(buf, '['))
+ return 0;
+
+ if (check_end(buf, res, ']'))
+ return 0;
+
+ return arr_next(buf, res);
+}
+
+int ujson_arr_next(ujson_reader *buf, struct ujson_val *res)
+{
+ if (check_err(buf, res))
+ return 0;
+
+ if (check_end(buf, res, ']'))
+ return 0;
+
+ if (pre_next(buf, res))
+ return 0;
+
+ return arr_next(buf, res);
+}
+
+static void ujson_err_va(ujson_reader *buf, const char *fmt, va_list va)
+{
+ vsnprintf(buf->err, UJSON_ERR_MAX, fmt, va);
+}
+
+void ujson_err(ujson_reader *buf, const char *fmt, ...)
+{
+ va_list va;
+
+ va_start(va, fmt);
+ ujson_err_va(buf, fmt, va);
+ va_end(va);
+}
+
+static void vprintf_line(ujson_reader *buf, const char *fmt, va_list va)
+{
+ char line[UJSON_ERR_MAX+1];
+
+ vsnprintf(line, sizeof(line), fmt, va);
+
+ line[UJSON_ERR_MAX] = 0;
+
+ buf->err_print(buf->err_print_priv, line);
+}
+
+static void printf_line(ujson_reader *buf, const char *fmt, ...)
+{
+ va_list va;
+
+ va_start(va, fmt);
+ vprintf_line(buf, fmt, va);
+ va_end(va);
+}
+
+static void printf_json_line(ujson_reader *buf, size_t line_nr, const char *buf_pos)
+{
+ char line[UJSON_ERR_MAX+1];
+ size_t plen, i;
+
+ plen = sprintf(line, "%03zu: ", line_nr);
+
+ for (i = 0; i < UJSON_ERR_MAX-plen && buf_pos[i] && buf_pos[i] != '\n'; i++)
+ line[i+plen] = buf_pos[i];
+
+ line[i+plen] = 0;
+
+ buf->err_print(buf->err_print_priv, line);
+}
+
+static void print_arrow(ujson_reader *buf, const char *buf_pos, size_t count)
+{
+ char line[count + 7];
+ size_t i;
+
+ /* The '000: ' prefix */
+ for (i = 0; i <= 5; i++)
+ line[i] = ' ';
+
+ for (i = 0; i < count; i++)
+ line[i+5] = buf_pos[i] == '\t' ? '\t' : ' ';
+
+ line[count+5] = '^';
+ line[count+6] = 0;
+
+ buf->err_print(buf->err_print_priv, line);
+}
+
+#define ERR_LINES 10
+
+#define MIN(A, B) ((A < B) ? (A) : (B))
+
+static void print_snippet(ujson_reader *buf, const char *type)
+{
+ ssize_t i;
+ const char *lines[ERR_LINES] = {};
+ size_t cur_line = 0;
+ size_t cur_off = 0;
+ size_t last_off = buf->off;
+
+ for (;;) {
+ lines[(cur_line++) % ERR_LINES] = buf->json + cur_off;
+
+ while (cur_off < buf->len && buf->json[cur_off] != '\n')
+ cur_off++;
+
+ if (cur_off >= buf->off)
+ break;
+
+ cur_off++;
+ last_off = buf->off - cur_off;
+ }
+
+ printf_line(buf, "%s at line %03zu", type, cur_line);
+ buf->err_print(buf->err_print_priv, "");
+
+ size_t idx = 0;
+
+ for (i = MIN(ERR_LINES, cur_line); i > 0; i--) {
+ idx = (cur_line - i) % ERR_LINES;
+ printf_json_line(buf, cur_line - i + 1, lines[idx]);
+ }
+
+ print_arrow(buf, lines[idx], last_off);
+}
+
+void ujson_err_print(ujson_reader *buf)
+{
+ if (!buf->err_print)
+ return;
+
+ print_snippet(buf, "Parse error");
+ buf->err_print(buf->err_print_priv, buf->err);
+}
+
+void ujson_warn(ujson_reader *buf, const char *fmt, ...)
+{
+ va_list va;
+
+ if (buf->flags & UJSON_READER_STRICT) {
+ va_start(va, fmt);
+ ujson_err_va(buf, fmt, va);
+ va_end(va);
+ return;
+ }
+
+ if (!buf->err_print)
+ return;
+
+ print_snippet(buf, "Warning");
+
+ va_start(va, fmt);
+ vprintf_line(buf, fmt, va);
+ va_end(va);
+}
+
+void ujson_print(void *err_print_priv, const char *line)
+{
+ fputs(line, err_print_priv);
+ putc('\n', err_print_priv);
+}
+
+ujson_reader *ujson_reader_load(const char *path)
+{
+ int fd = open(path, O_RDONLY);
+ ujson_reader *ret;
+ ssize_t res;
+ off_t len, off = 0;
+
+ if (fd < 0)
+ return NULL;
+
+ len = lseek(fd, 0, SEEK_END);
+ if (len == (off_t)-1) {
+ fprintf(stderr, "lseek() failed\n");
+ goto err0;
+ }
+
+ if (lseek(fd, 0, SEEK_SET) == (off_t)-1) {
+ fprintf(stderr, "lseek() failed\n");
+ goto err0;
+ }
+
+ ret = malloc(sizeof(ujson_reader) + len + 1);
+ if (!ret) {
+ fprintf(stderr, "malloc() failed\n");
+ goto err0;
+ }
+
+ memset(ret, 0, sizeof(*ret));
+
+ ret->buf[len] = 0;
+ ret->len = len;
+ ret->max_depth = UJSON_RECURSION_MAX;
+ ret->json = ret->buf;
+ ret->err_print = UJSON_ERR_PRINT;
+ ret->err_print_priv = UJSON_ERR_PRINT_PRIV;
+
+ while (off < len) {
+ res = read(fd, ret->buf + off, len - off);
+ if (res < 0) {
+ fprintf(stderr, "read() failed\n");
+ goto err1;
+ }
+
+ off += res;
+ }
+
+ close(fd);
+
+ return ret;
+err1:
+ free(ret);
+err0:
+ close(fd);
+ return NULL;
+}
+
+void ujson_reader_finish(ujson_reader *self)
+{
+ if (ujson_reader_err(self))
+ ujson_err_print(self);
+ else if (!ujson_reader_consumed(self))
+ ujson_warn(self, "Garbage after JSON string!");
+}
+
+void ujson_reader_free(ujson_reader *buf)
+{
+ free(buf);
+}
+
+ujson_val *ujson_val_alloc(size_t buf_size)
+{
+ buf_size = buf_size == 0 ? 4096 : buf_size;
+ ujson_val *ret;
+
+ ret = malloc(sizeof(ujson_val) + buf_size);
+ if (!ret)
+ return NULL;
+
+ memset(ret, 0, sizeof(ujson_val) + buf_size);
+
+ ret->buf = ret->buf__;
+ ret->buf_size = buf_size;
+
+ return ret;
+}
+
+void ujson_val_free(ujson_val *self)
+{
+ free(self);
+}
diff --git a/libs/ujson/ujson_utf.c b/libs/ujson/ujson_utf.c
new file mode 100644
index 000000000..2c08a39a8
--- /dev/null
+++ b/libs/ujson/ujson_utf.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2022-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+#include <stddef.h>
+#include <ujson_utf.h>
+
+int8_t ujson_utf8_next_chsz(const char *str, size_t off)
+{
+ char ch = str[off];
+ uint8_t len = 0;
+
+ if (!ch)
+ return 0;
+
+ if (UJSON_UTF8_IS_ASCII(ch))
+ return 1;
+
+ if (UJSON_UTF8_IS_2BYTE(ch)) {
+ len = 2;
+ goto ret;
+ }
+
+ if (UJSON_UTF8_IS_3BYTE(ch)) {
+ len = 3;
+ goto ret;
+ }
+
+ if (UJSON_UTF8_IS_4BYTE(ch)) {
+ len = 4;
+ goto ret;
+ }
+
+ return -1;
+ret:
+ if (!UJSON_UTF8_IS_NBYTE(str[off+1]))
+ return -1;
+
+ if (len > 2 && !UJSON_UTF8_IS_NBYTE(str[off+2]))
+ return -1;
+
+ if (len > 3 && !UJSON_UTF8_IS_NBYTE(str[off+3]))
+ return -1;
+
+ return len;
+}
+
+int8_t ujson_utf8_prev_chsz(const char *str, size_t off)
+{
+ char ch;
+
+ if (!off)
+ return 0;
+
+ ch = str[--off];
+
+ if (UJSON_UTF8_IS_ASCII(ch))
+ return 1;
+
+ if (!UJSON_UTF8_IS_NBYTE(ch))
+ return -1;
+
+ if (off < 1)
+ return -1;
+
+ ch = str[--off];
+
+ if (UJSON_UTF8_IS_2BYTE(ch))
+ return 2;
+
+ if (!UJSON_UTF8_IS_NBYTE(ch))
+ return -1;
+
+ if (off < 1)
+ return -1;
+
+ ch = str[--off];
+
+ if (UJSON_UTF8_IS_3BYTE(ch))
+ return 3;
+
+ if (!UJSON_UTF8_IS_NBYTE(ch))
+ return -1;
+
+ if (off < 1)
+ return -1;
+
+ ch = str[--off];
+
+ if (UJSON_UTF8_IS_4BYTE(ch))
+ return 4;
+
+ return -1;
+}
+
+size_t ujson_utf8_strlen(const char *str)
+{
+ size_t cnt = 0;
+
+ while (ujson_utf8_next(&str))
+ cnt++;
+
+ return cnt;
+}
diff --git a/libs/ujson/ujson_writer.c b/libs/ujson/ujson_writer.c
new file mode 100644
index 000000000..6275be1ff
--- /dev/null
+++ b/libs/ujson/ujson_writer.c
@@ -0,0 +1,491 @@
+// SPDX-License-Identifier: LGPL-2.1-or-later
+/*
+ * Copyright (C) 2021-2024 Cyril Hrubis <metan@ucw.cz>
+ */
+
+#include <string.h>
+#include <stdarg.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+#include "ujson_utf.h"
+#include "ujson_writer.h"
+
+static inline int get_depth_bit(ujson_writer *self, char *mask)
+{
+ int depth = self->depth - 1;
+
+ if (depth < 0)
+ return -1;
+
+ return !!(mask[depth/8] & (1<<(depth%8)));
+}
+
+static inline void set_depth_bit(ujson_writer *self, int val)
+{
+ if (val)
+ self->depth_type[self->depth/8] |= (1<<(self->depth%8));
+ else
+ self->depth_type[self->depth/8] &= ~(1<<(self->depth%8));
+
+ self->depth_first[self->depth/8] |= (1<<(self->depth%8));
+
+ self->depth++;
+}
+
+static inline void clear_depth_bit(ujson_writer *self)
+{
+ self->depth--;
+}
+
+static inline int in_arr(ujson_writer *self)
+{
+ return !get_depth_bit(self, self->depth_type);
+}
+
+static inline int in_obj(ujson_writer *self)
+{
+ return get_depth_bit(self, self->depth_type);
+}
+
+static inline void clear_depth_first(ujson_writer *self)
+{
+ int depth = self->depth - 1;
+
+ self->depth_first[depth/8] &= ~(1<<(depth%8));
+}
+
+static inline int is_first(ujson_writer *self)
+{
+ int ret = get_depth_bit(self, self->depth_first);
+
+ if (ret == 1)
+ clear_depth_first(self);
+
+ return ret;
+}
+
+static inline void err(ujson_writer *buf, const char *fmt, ...)
+{
+ va_list va;
+
+ va_start(va, fmt);
+ vsnprintf(buf->err, UJSON_ERR_MAX, fmt, va);
+ va_end(va);
+}
+
+static inline int is_err(ujson_writer *buf)
+{
+ return buf->err[0];
+}
+
+static inline int out(ujson_writer *self, const char *buf, size_t len)
+{
+ return self->out(self, buf, len);
+}
+
+static inline int out_str(ujson_writer *self, const char *str)
+{
+ return out(self, str, strlen(str));
+}
+
+static inline int out_ch(ujson_writer *self, char ch)
+{
+ return out(self, &ch, 1);
+}
+
+#define ESC_FLUSH(esc_char) do {\
+ out(self, val, i); \
+ val += i + 1; \
+ i = 0; \
+ out_str(self, esc_char); \
+} while (0)
+
+static inline int out_esc_str(ujson_writer *self, const char *val)
+{
+ if (out_ch(self, '"'))
+ return 1;
+
+ size_t i = 0;
+ int8_t next_chsz;
+
+ do {
+ next_chsz = ujson_utf8_next_chsz(val, i);
+
+ if (next_chsz == 1) {
+ switch (val[i]) {
+ case '\"':
+ ESC_FLUSH("\\\"");
+ break;
+ case '\\':
+ ESC_FLUSH("\\\\");
+ break;
+ case '/':
+ ESC_FLUSH("\\/");
+ break;
+ case '\b':
+ ESC_FLUSH("\\b");
+ break;
+ case '\f':
+ ESC_FLUSH("\\f");
+ break;
+ case '\n':
+ ESC_FLUSH("\\n");
+ break;
+ case '\r':
+ ESC_FLUSH("\\r");
+ break;
+ case '\t':
+ ESC_FLUSH("\\t");
+ break;
+ default:
+ i += next_chsz;
+ }
+ } else {
+ i += next_chsz;
+ }
+ } while (next_chsz);
+
+ if (i) {
+ if (out(self, val, i))
+ return 1;
+ }
+
+ if (out_ch(self, '"'))
+ return 1;
+
+ return 0;
+}
+
+static int do_padd(ujson_writer *self)
+{
+ unsigned int i;
+
+ for (i = 0; i < self->depth; i++) {
+ if (out_ch(self, ' '))
+ return 1;
+ }
+
+ return 0;
+}
+
+static int newline(ujson_writer *self)
+{
+ if (out_ch(self, '\n'))
+ return 0;
+
+ if (do_padd(self))
+ return 1;
+
+ return 0;
+}
+
+static int add_common(ujson_writer *self, const char *id)
+{
+ if (is_err(self))
+ return 1;
+
+ if (!self->depth) {
+ err(self, "Object/Array has to be started first");
+ return 1;
+ }
+
+ if (in_arr(self)) {
+ if (id) {
+ err(self, "Array entries can't have id");
+ return 1;
+ }
+ } else {
+ if (!id) {
+ err(self, "Object entries must have id");
+ return 1;
+ }
+ }
+
+ if (!is_first(self) && out_ch(self, ','))
+ return 1;
+
+ if (self->depth && newline(self))
+ return 1;
+
+ if (id) {
+ if (out_esc_str(self, id))
+ return 1;
+
+ if (out_str(self, ": "))
+ return 1;
+ }
+
+ return 0;
+}
+
+int ujson_obj_start(ujson_writer *self, const char *id)
+{
+ if (self->depth >= UJSON_RECURSION_MAX)
+ return 1;
+
+ if (!self->depth && id) {
+ err(self, "Top level object cannot have id");
+ return 1;
+ }
+
+ if (self->depth && add_common(self, id))
+ return 1;
+
+ if (out_ch(self, '{'))
+ return 1;
+
+ set_depth_bit(self, 1);
+
+ return 0;
+}
+
+int ujson_obj_finish(ujson_writer *self)
+{
+ if (is_err(self))
+ return 1;
+
+ if (!in_obj(self)) {
+ err(self, "Not in object!");
+ return 1;
+ }
+
+ int first = is_first(self);
+
+ clear_depth_bit(self);
+
+ if (!first)
+ newline(self);
+
+ return out_ch(self, '}');
+}
+
+int ujson_arr_start(ujson_writer *self, const char *id)
+{
+ if (self->depth >= UJSON_RECURSION_MAX) {
+ err(self, "Recursion too deep");
+ return 1;
+ }
+
+ if (!self->depth && id) {
+ err(self, "Top level array cannot have id");
+ return 1;
+ }
+
+ if (self->depth && add_common(self, id))
+ return 1;
+
+ if (out_ch(self, '['))
+ return 1;
+
+ set_depth_bit(self, 0);
+
+ return 0;
+}
+
+int ujson_arr_finish(ujson_writer *self)
+{
+ if (is_err(self))
+ return 1;
+
+ if (!in_arr(self)) {
+ err(self, "Not in array!");
+ return 1;
+ }
+
+ int first = is_first(self);
+
+ clear_depth_bit(self);
+
+ if (!first)
+ newline(self);
+
+ return out_ch(self, ']');
+}
+
+int ujson_null_add(ujson_writer *self, const char *id)
+{
+ if (add_common(self, id))
+ return 1;
+
+ return out_str(self, "null");
+}
+
+int ujson_int_add(ujson_writer *self, const char *id, long val)
+{
+ char buf[64];
+
+ if (add_common(self, id))
+ return 1;
+
+ snprintf(buf, sizeof(buf), "%li", val);
+
+ return out_str(self, buf);
+}
+
+int ujson_bool_add(ujson_writer *self, const char *id, int val)
+{
+ if (add_common(self, id))
+ return 1;
+
+ if (val)
+ return out_str(self, "true");
+ else
+ return out_str(self, "false");
+}
+
+int ujson_str_add(ujson_writer *self, const char *id, const char *val)
+{
+ if (add_common(self, id))
+ return 1;
+
+ if (out_esc_str(self, val))
+ return 1;
+
+ return 0;
+}
+
+int ujson_float_add(ujson_writer *self, const char *id, double val)
+{
+ char buf[64];
+
+ if (add_common(self, id))
+ return 1;
+
+ snprintf(buf, sizeof(buf), "%lg", val);
+
+ return out_str(self, buf);
+}
+
+int ujson_writer_finish(ujson_writer *self)
+{
+ if (is_err(self))
+ goto err;
+
+ if (self->depth) {
+ err(self, "Objects and/or Arrays not finished");
+ goto err;
+ }
+
+ if (newline(self))
+ return 1;
+
+ return 0;
+err:
+ if (self->err_print)
+ self->err_print(self->err_print_priv, self->err);
+
+ return 1;
+}
+
+struct json_writer_file {
+ int fd;
+ size_t buf_used;
+ char buf[1024];
+};
+
+static int out_writer_file_write(ujson_writer *self, int fd, const char *buf, ssize_t buf_len)
+{
+ do {
+ ssize_t ret = write(fd, buf, buf_len);
+ if (ret <= 0) {
+ err(self, "Failed to write to a file");
+ return 1;
+ }
+
+ if (ret > buf_len) {
+ err(self, "Wrote more bytes than requested?!");
+ return 1;
+ }
+
+ buf_len -= ret;
+ } while (buf_len);
+
+ return 0;
+}
+
+static int out_writer_file(ujson_writer *self, const char *buf, size_t buf_len)
+{
+ struct json_writer_file *writer_file = self->out_priv;
+ size_t buf_size = sizeof(writer_file->buf);
+ size_t buf_avail = buf_size - writer_file->buf_used;
+
+ if (buf_len > buf_size/4)
+ return out_writer_file_write(self, writer_file->fd, buf, buf_len);
+
+ if (buf_len >= buf_avail) {
+ if (out_writer_file_write(self, writer_file->fd,
+ writer_file->buf, writer_file->buf_used))
+ return 1;
+
+ memcpy(writer_file->buf, buf, buf_len);
+ writer_file->buf_used = buf_len;
+ return 0;
+ }
+
+ memcpy(writer_file->buf + writer_file->buf_used, buf, buf_len);
+ writer_file->buf_used += buf_len;
+
+ return 0;
+}
+
+int ujson_writer_file_close(ujson_writer *self)
+{
+ struct json_writer_file *writer_file = self->out_priv;
+ int saved_errno = 0;
+
+ if (writer_file->buf_used) {
+ if (out_writer_file_write(self, writer_file->fd,
+ writer_file->buf, writer_file->buf_used))
+
+ saved_errno = errno;
+ }
+
+ if (close(writer_file->fd)) {
+ if (!saved_errno)
+ saved_errno = errno;
+ }
+
+ free(self);
+
+ if (saved_errno) {
+ errno = saved_errno;
+ return 1;
+ }
+
+ return 0;
+}
+
+ujson_writer *ujson_writer_file_open(const char *path)
+{
+ ujson_writer *ret;
+ struct json_writer_file *writer_file;
+
+ ret = malloc(sizeof(ujson_writer) + sizeof(struct json_writer_file));
+ if (!ret)
+ return NULL;
+
+ writer_file = (void*)ret + sizeof(ujson_writer);
+
+ writer_file->fd = open(path, O_CREAT | O_WRONLY | O_TRUNC, 0664);
+ if (!writer_file->fd) {
+ free(ret);
+ return NULL;
+ }
+
+ writer_file->buf_used = 0;
+
+ memset(ret, 0, sizeof(*ret));
+
+ ret->err_print = UJSON_ERR_PRINT;
+ ret->err_print_priv = UJSON_ERR_PRINT_PRIV;
+ ret->out = out_writer_file;
+ ret->out_priv = writer_file;
+
+ return ret;
+}
+
+
--
2.44.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [LTP] [PATCH 3/3] testcaes/lib: Add shell loader
2024-07-31 9:20 [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 1/3] Add support for mixing C and shell code Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 2/3] libs: Vendor ujson library Cyril Hrubis
@ 2024-07-31 9:20 ` Cyril Hrubis
2024-08-13 16:15 ` Richard Palethorpe
2024-07-31 10:06 ` [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
3 siblings, 1 reply; 10+ messages in thread
From: Cyril Hrubis @ 2024-07-31 9:20 UTC (permalink / raw)
To: ltp
This commit implements a shell loader so that we don't have to write a C
loader for each LTP shell test. The idea is simple, the loader parses
the shell test and prepares the tst_test structure accordingly, then
runs the actual shell test.
The format for the metadata in the shell test was choosen to be JSON
because:
- I didn't want to invent an adhoc format and JSON is perfect for
serializing data structures
- The metadata parser for shell test will be trivial, it will just pick
the JSON from the comment, no parsing will be required
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
include/tst_test.h | 2 +-
testcases/lib/.gitignore | 1 +
testcases/lib/Makefile | 6 +-
testcases/lib/run_tests.sh | 9 +
testcases/lib/tests/shell_loader.sh | 15 +
.../lib/tests/shell_loader_all_filesystems.sh | 24 ++
.../lib/tests/shell_loader_filesystems.sh | 30 ++
.../tests/shell_loader_invalid_metadata.sh | 12 +
.../lib/tests/shell_loader_no_metadata.sh | 8 +
.../lib/tests/shell_loader_supported_archs.sh | 9 +
.../lib/tests/shell_loader_wrong_metadata.sh | 12 +
testcases/lib/tst_env.sh | 4 +
testcases/lib/tst_loader.sh | 11 +
testcases/lib/tst_run_shell.c | 378 ++++++++++++++++++
14 files changed, 519 insertions(+), 2 deletions(-)
create mode 100755 testcases/lib/tests/shell_loader.sh
create mode 100755 testcases/lib/tests/shell_loader_all_filesystems.sh
create mode 100755 testcases/lib/tests/shell_loader_filesystems.sh
create mode 100755 testcases/lib/tests/shell_loader_invalid_metadata.sh
create mode 100755 testcases/lib/tests/shell_loader_no_metadata.sh
create mode 100755 testcases/lib/tests/shell_loader_supported_archs.sh
create mode 100755 testcases/lib/tests/shell_loader_wrong_metadata.sh
create mode 100644 testcases/lib/tst_loader.sh
create mode 100644 testcases/lib/tst_run_shell.c
diff --git a/include/tst_test.h b/include/tst_test.h
index a334195ac..c04498937 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -274,7 +274,7 @@ struct tst_fs {
const char *const *mkfs_opts;
const char *mkfs_size_opt;
- const unsigned int mnt_flags;
+ unsigned int mnt_flags;
const void *mnt_data;
};
diff --git a/testcases/lib/.gitignore b/testcases/lib/.gitignore
index d0dacf62a..385f3c3ca 100644
--- a/testcases/lib/.gitignore
+++ b/testcases/lib/.gitignore
@@ -24,3 +24,4 @@
/tst_supported_fs
/tst_timeout_kill
/tst_res_
+/tst_run_shell
diff --git a/testcases/lib/Makefile b/testcases/lib/Makefile
index 928d76d62..b3a9181c1 100644
--- a/testcases/lib/Makefile
+++ b/testcases/lib/Makefile
@@ -4,6 +4,9 @@
top_srcdir ?= ../..
+LTPLIBS = ujson
+tst_run_shell: LTPLDLIBS = -lujson
+
include $(top_srcdir)/include/mk/testcases.mk
INSTALL_TARGETS := *.sh
@@ -13,6 +16,7 @@ MAKE_TARGETS := tst_sleep tst_random tst_checkpoint tst_rod tst_kvcmp\
tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port\
tst_get_median tst_hexdump tst_get_free_pids tst_timeout_kill\
tst_check_kconfigs tst_cgctl tst_fsfreeze tst_ns_create tst_ns_exec\
- tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled tst_res_
+ tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled tst_res_\
+ tst_run_shell
include $(top_srcdir)/include/mk/generic_trunk_target.mk
diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
index 60e7d1bcf..9857f5f82 100755
--- a/testcases/lib/run_tests.sh
+++ b/testcases/lib/run_tests.sh
@@ -9,3 +9,12 @@ for i in `seq -w 01 06`; do
echo
./tests/shell_test$i
done
+
+for i in shell_loader.sh shell_loader_all_filesystems.sh shell_loader_no_metadata.sh \
+ shell_loader_wrong_metadata.sh shell_loader_invalid_metadata.sh\
+ shell_loader_supported_archs.sh shell_loader_filesystems.sh; do
+ echo
+ echo "*** Running $i ***"
+ echo
+ $i
+done
diff --git a/testcases/lib/tests/shell_loader.sh b/testcases/lib/tests/shell_loader.sh
new file mode 100755
index 000000000..642ffe97b
--- /dev/null
+++ b/testcases/lib/tests/shell_loader.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+#
+# TEST = {
+# "needs_tmpdir": true
+# }
+
+. tst_loader.sh
+
+tst_res TPASS "Shell loader works fine!"
+case "$PWD" in
+ /tmp/*)
+ tst_res TPASS "We are running in temp directory in $PWD";;
+ *)
+ tst_res TFAIL "We are not running in temp directory but $PWD";;
+esac
diff --git a/testcases/lib/tests/shell_loader_all_filesystems.sh b/testcases/lib/tests/shell_loader_all_filesystems.sh
new file mode 100755
index 000000000..8432b4b3d
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_all_filesystems.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# TEST = {
+# "needs_root": true,
+# "mount_device": true,
+# "all_filesystems": true,
+# "mntpoint": "ltp_mntpoint"
+# }
+
+. tst_loader.sh
+
+tst_res TINFO "In shell"
+
+mntpath=$(realpath ltp_mntpoint)
+mounted=$(grep $mntpath /proc/mounts)
+
+if [ -n "$mounted" ]; then
+ device=$(echo $mounted |cut -d' ' -f 1)
+ path=$(echo $mounted |cut -d' ' -f 2)
+
+ tst_res TPASS "$device mounted at $path"
+else
+ tst_res TFAIL "Device not mounted!"
+fi
diff --git a/testcases/lib/tests/shell_loader_filesystems.sh b/testcases/lib/tests/shell_loader_filesystems.sh
new file mode 100755
index 000000000..ede1a8fe9
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_filesystems.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+#
+# TEST = {
+# "mount_device": true,
+# "mntpoint": "ltp_mntpoint",
+# "filesystems": [
+# {
+# "type": "btrfs"
+# },
+# {
+# "type": "xfs",
+# "mkfs_opts": ["-m", "reflink=1"]
+# }
+# ]
+# }
+
+. tst_loader.sh
+
+tst_res TINFO "In shell"
+
+mntpoint=$(realpath ltp_mntpoint)
+mounted=$(grep $mntpoint /proc/mounts)
+
+if [ -n "$mounted" ]; then
+ fs=$(echo $mounted |cut -d' ' -f 3)
+
+ tst_res TPASS "Mounted device formatted with $fs"
+else
+ tst_res TFAIL "Device not mounted!"
+fi
diff --git a/testcases/lib/tests/shell_loader_invalid_metadata.sh b/testcases/lib/tests/shell_loader_invalid_metadata.sh
new file mode 100755
index 000000000..265be6f36
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_invalid_metadata.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+#
+# This test has wrong metadata and should not be run
+#
+# TEST = {
+# {"needs_tmpdir": 42,
+# }
+#
+
+. tst_loader.sh
+
+tst_res TFAIL "Shell loader should TBROK the test"
diff --git a/testcases/lib/tests/shell_loader_no_metadata.sh b/testcases/lib/tests/shell_loader_no_metadata.sh
new file mode 100755
index 000000000..60ba8b889
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_no_metadata.sh
@@ -0,0 +1,8 @@
+#!/bin/sh
+#
+# This test has no metadata and should not be executed
+#
+
+. tst_loader.sh
+
+tst_res TFAIL "Shell loader should TBROK the test"
diff --git a/testcases/lib/tests/shell_loader_supported_archs.sh b/testcases/lib/tests/shell_loader_supported_archs.sh
new file mode 100755
index 000000000..d5c6c648b
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_supported_archs.sh
@@ -0,0 +1,9 @@
+#!/bin/sh
+#
+# TEST = {
+# "supported_archs": ["x86", "ppc64", "x86_64"]
+# }
+
+. tst_loader.sh
+
+tst_res TPASS "We are running on supported architecture"
diff --git a/testcases/lib/tests/shell_loader_wrong_metadata.sh b/testcases/lib/tests/shell_loader_wrong_metadata.sh
new file mode 100755
index 000000000..f29b9308f
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_wrong_metadata.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+#
+# This test has wrong metadata and should not be run
+#
+# TEST = {
+# "needs_tmpdir": 42,
+# }
+#
+
+. tst_loader.sh
+
+tst_res TFAIL "Shell loader should TBROK the test"
diff --git a/testcases/lib/tst_env.sh b/testcases/lib/tst_env.sh
index 948bc5024..67ba80744 100644
--- a/testcases/lib/tst_env.sh
+++ b/testcases/lib/tst_env.sh
@@ -1,4 +1,8 @@
#!/bin/sh
+#
+# This is a minimal test environment for a shell scripts executed from C by
+# tst_run_shell() function. Shell tests must use the tst_loader.sh instead!
+#
tst_script_name=$(basename $0)
diff --git a/testcases/lib/tst_loader.sh b/testcases/lib/tst_loader.sh
new file mode 100644
index 000000000..5ac095e44
--- /dev/null
+++ b/testcases/lib/tst_loader.sh
@@ -0,0 +1,11 @@
+#!/bin/sh
+#
+# This is a loader for shell tests that use the C test library.
+#
+
+if [ -z "$LTP_IPC_PATH" ]; then
+ tst_run_shell $(basename "$0")
+ exit $?
+else
+ . tst_env.sh
+fi
diff --git a/testcases/lib/tst_run_shell.c b/testcases/lib/tst_run_shell.c
new file mode 100644
index 000000000..96827e1cc
--- /dev/null
+++ b/testcases/lib/tst_run_shell.c
@@ -0,0 +1,378 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (c) 2024 Cyril Hrubis <chrubis@suse.cz>
+ */
+#include <sys/mount.h>
+
+#define TST_NO_DEFAULT_MAIN
+#include "tst_test.h"
+#include "tst_safe_stdio.h"
+#include "ujson.h"
+
+static char *shell_filename;
+
+static void run_shell(void)
+{
+ tst_run_shell(shell_filename, NULL);
+}
+
+struct tst_test test = {
+ .test_all = run_shell,
+ .runs_script = 1,
+};
+
+static void print_help(void)
+{
+ printf("Usage: tst_shell_loader ltp_shell_test.sh ...");
+}
+
+static char *metadata;
+static size_t metadata_size;
+static size_t metadata_used;
+
+static void metadata_append(const char *line)
+{
+ size_t linelen = strlen(line);
+
+ if (metadata_size - metadata_used < linelen + 1) {
+ metadata_size += 128;
+ metadata = SAFE_REALLOC(metadata, metadata_size);
+ }
+
+ strcpy(metadata + metadata_used, line);
+ metadata_used += linelen;
+}
+
+static ujson_obj_attr test_attrs[] = {
+ UJSON_OBJ_ATTR("all_filesystems", UJSON_BOOL),
+ UJSON_OBJ_ATTR("dev_min_size", UJSON_INT),
+ UJSON_OBJ_ATTR("filesystems", UJSON_ARR),
+ UJSON_OBJ_ATTR("format_device", UJSON_BOOL),
+ UJSON_OBJ_ATTR("min_cpus", UJSON_INT),
+ UJSON_OBJ_ATTR("min_mem_avail", UJSON_INT),
+ UJSON_OBJ_ATTR("min_kver", UJSON_STR),
+ UJSON_OBJ_ATTR("min_swap_avail", UJSON_INT),
+ UJSON_OBJ_ATTR("mntpoint", UJSON_STR),
+ UJSON_OBJ_ATTR("mount_device", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_abi_bits", UJSON_INT),
+ UJSON_OBJ_ATTR("needs_devfs", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_device", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_hugetlbfs", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_rofs", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_root", UJSON_BOOL),
+ UJSON_OBJ_ATTR("needs_tmpdir", UJSON_BOOL),
+ UJSON_OBJ_ATTR("restore_wallclock", UJSON_BOOL),
+ UJSON_OBJ_ATTR("skip_filesystems", UJSON_ARR),
+ UJSON_OBJ_ATTR("skip_in_compat", UJSON_BOOL),
+ UJSON_OBJ_ATTR("skip_in_lockdown", UJSON_BOOL),
+ UJSON_OBJ_ATTR("skip_in_secureboot", UJSON_BOOL),
+ UJSON_OBJ_ATTR("supported_archs", UJSON_ARR),
+};
+
+static ujson_obj test_obj = {
+ .attrs = test_attrs,
+ .attr_cnt = UJSON_ARRAY_SIZE(test_attrs),
+};
+
+/* Must match the order of test_attrs. */
+enum test_attr_ids {
+ ALL_FILESYSTEMS,
+ DEV_MIN_SIZE,
+ FILESYSTEMS,
+ FORMAT_DEVICE,
+ MIN_CPUS,
+ MIN_MEM_AVAIL,
+ MIN_KVER,
+ MIN_SWAP_AVAIL,
+ MNTPOINT,
+ MOUNT_DEVICE,
+ NEEDS_ABI_BITS,
+ NEEDS_DEVFS,
+ NEEDS_DEVICE,
+ NEEDS_HUGETLBFS,
+ NEEDS_ROFS,
+ NEEDS_ROOT,
+ NEEDS_TMPDIR,
+ RESTORE_WALLCLOCK,
+ SKIP_FILESYSTEMS,
+ SKIP_IN_COMPAT,
+ SKIP_IN_LOCKDOWN,
+ SKIP_IN_SECUREBOOT,
+ SUPPORTED_ARCHS,
+};
+
+static const char *const *parse_strarr(ujson_reader *reader, ujson_val *val)
+{
+ unsigned int cnt = 0, i = 0;
+ char **ret;
+
+ ujson_reader_state state = ujson_reader_state_save(reader);
+
+ UJSON_ARR_FOREACH(reader, val) {
+ if (val->type != UJSON_STR) {
+ ujson_err(reader, "Expected string!");
+ return NULL;
+ }
+
+ cnt++;
+ }
+
+ ujson_reader_state_load(reader, state);
+
+ ret = SAFE_MALLOC(sizeof(char*) * (cnt + 1));
+
+ UJSON_ARR_FOREACH(reader, val) {
+ ret[i++] = strdup(val->val_str);
+ }
+
+ ret[i] = NULL;
+
+ return (const char *const *)ret;
+}
+
+static ujson_obj_attr fs_attrs[] = {
+ UJSON_OBJ_ATTR("mkfs_opts", UJSON_ARR),
+ UJSON_OBJ_ATTR("mkfs_size_opt", UJSON_STR),
+ UJSON_OBJ_ATTR("mnt_flags", UJSON_ARR),
+ UJSON_OBJ_ATTR("type", UJSON_STR),
+};
+
+static ujson_obj fs_obj = {
+ .attrs = fs_attrs,
+ .attr_cnt = UJSON_ARRAY_SIZE(fs_attrs),
+};
+
+/* Must match the order of fs_attrs. */
+enum fs_ids {
+ MKFS_OPTS,
+ MKFS_SIZE_OPT,
+ MNT_FLAGS,
+ TYPE,
+};
+
+static int parse_mnt_flags(ujson_reader *reader, ujson_val *val)
+{
+ int ret = 0;
+
+ UJSON_ARR_FOREACH(reader, val) {
+ if (val->type != UJSON_STR) {
+ ujson_err(reader, "Expected string!");
+ return ret;
+ }
+
+ if (!strcmp(val->val_str, "RDONLY"))
+ ret |= MS_RDONLY;
+ else if (!strcmp(val->val_str, "NOATIME"))
+ ret |= MS_NOATIME;
+ else if (!strcmp(val->val_str, "NOEXEC"))
+ ret |= MS_NOEXEC;
+ else if (!strcmp(val->val_str, "NOSUID"))
+ ret |= MS_NOSUID;
+ else
+ ujson_err(reader, "Invalid mount flag");
+ }
+
+ return ret;
+}
+
+struct tst_fs *parse_filesystems(ujson_reader *reader, ujson_val *val)
+{
+ unsigned int i = 0, cnt = 0;
+ struct tst_fs *ret;
+
+ ujson_reader_state state = ujson_reader_state_save(reader);
+
+ UJSON_ARR_FOREACH(reader, val) {
+ if (val->type != UJSON_OBJ) {
+ ujson_err(reader, "Expected object!");
+ return NULL;
+ }
+ ujson_obj_skip(reader);
+ cnt++;
+ }
+
+ ujson_reader_state_load(reader, state);
+
+ ret = SAFE_MALLOC(sizeof(struct tst_fs) * (cnt + 1));
+ memset(ret, 0, sizeof(*ret) * (cnt+1));
+
+ UJSON_ARR_FOREACH(reader, val) {
+ UJSON_OBJ_FOREACH_FILTER(reader, val, &fs_obj, ujson_empty_obj) {
+ switch ((enum fs_ids)val->idx) {
+ case MKFS_OPTS:
+ ret[i].mkfs_opts = parse_strarr(reader, val);
+ break;
+ case MKFS_SIZE_OPT:
+ ret[i].mkfs_size_opt = strdup(val->val_str);
+ break;
+ case MNT_FLAGS:
+ ret[i].mnt_flags = parse_mnt_flags(reader, val);
+ break;
+ case TYPE:
+ ret[i].type = strdup(val->val_str);
+ break;
+ }
+
+ }
+
+ i++;
+ }
+
+ return ret;
+}
+
+static void parse_metadata(void)
+{
+ ujson_reader reader = UJSON_READER_INIT(metadata, metadata_used, UJSON_READER_STRICT);
+ char str_buf[128];
+ ujson_val val = UJSON_VAL_INIT(str_buf, sizeof(str_buf));
+
+ UJSON_OBJ_FOREACH_FILTER(&reader, &val, &test_obj, ujson_empty_obj) {
+ switch ((enum test_attr_ids)val.idx) {
+ case ALL_FILESYSTEMS:
+ test.all_filesystems = val.val_bool;
+ break;
+ case DEV_MIN_SIZE:
+ if (val.val_int <= 0)
+ ujson_err(&reader, "Device size must be > 0");
+ else
+ test.dev_min_size = val.val_int;
+ break;
+ case FILESYSTEMS:
+ test.filesystems = parse_filesystems(&reader, &val);
+ break;
+ case FORMAT_DEVICE:
+ test.format_device = val.val_bool;
+ break;
+ case MIN_CPUS:
+ if (val.val_int <= 0)
+ ujson_err(&reader, "Minimal number of cpus must be > 0");
+ else
+ test.min_cpus = val.val_int;
+ break;
+ case MIN_MEM_AVAIL:
+ if (val.val_int <= 0)
+ ujson_err(&reader, "Minimal available memory size must be > 0");
+ else
+ test.min_mem_avail = val.val_int;
+ break;
+ case MIN_KVER:
+ test.min_kver = strdup(val.val_str);
+ break;
+ case MIN_SWAP_AVAIL:
+ if (val.val_int <= 0)
+ ujson_err(&reader, "Minimal available swap size must be > 0");
+ else
+ test.min_swap_avail = val.val_int;
+ break;
+ case MNTPOINT:
+ test.mntpoint = strdup(val.val_str);
+ break;
+ case MOUNT_DEVICE:
+ test.mount_device = val.val_bool;
+ break;
+ case NEEDS_ABI_BITS:
+ if (val.val_int == 32 || val.val_int == 64)
+ test.needs_abi_bits = val.val_int;
+ else
+ ujson_err(&reader, "ABI bits must be 32 or 64");
+ break;
+ case NEEDS_DEVFS:
+ test.needs_devfs = val.val_bool;
+ break;
+ case NEEDS_DEVICE:
+ test.needs_device = val.val_bool;
+ break;
+ case NEEDS_HUGETLBFS:
+ test.needs_hugetlbfs = val.val_bool;
+ break;
+ case NEEDS_ROFS:
+ test.needs_rofs = val.val_bool;
+ break;
+ case NEEDS_ROOT:
+ test.needs_root = val.val_bool;
+ break;
+ case NEEDS_TMPDIR:
+ test.needs_tmpdir = val.val_bool;
+ break;
+ case RESTORE_WALLCLOCK:
+ test.restore_wallclock = val.val_bool;
+ break;
+ case SKIP_FILESYSTEMS:
+ test.skip_filesystems = parse_strarr(&reader, &val);
+ break;
+ case SKIP_IN_COMPAT:
+ test.skip_in_compat = val.val_bool;
+ break;
+ case SKIP_IN_LOCKDOWN:
+ test.skip_in_lockdown = val.val_bool;
+ break;
+ case SKIP_IN_SECUREBOOT:
+ test.skip_in_secureboot = val.val_bool;
+ break;
+ case SUPPORTED_ARCHS:
+ test.supported_archs = parse_strarr(&reader, &val);
+ break;
+ }
+ }
+
+ ujson_reader_finish(&reader);
+
+ if (ujson_reader_err(&reader))
+ tst_brk(TBROK, "Invalid metadata");
+}
+
+static void extract_metadata(void)
+{
+ FILE *f;
+ char line[4096];
+ char path[4096];
+ int in_json = 0;
+
+ if (tst_get_path(shell_filename, path, sizeof(path)) == -1)
+ tst_brk(TBROK, "Failed to find %s in $PATH", shell_filename);
+
+ f = SAFE_FOPEN(path, "r");
+
+ while (fgets(line, sizeof(line), f)) {
+ if (in_json)
+ metadata_append(line + 2);
+
+ if (in_json) {
+ if (!strcmp(line, "# }\n"))
+ in_json = 0;
+ } else {
+ if (!strcmp(line, "# TEST = {\n")) {
+ metadata_append("{\n");
+ in_json = 1;
+ }
+ }
+ }
+
+ fclose(f);
+}
+
+static void prepare_test_struct(void)
+{
+ extract_metadata();
+
+ if (metadata)
+ parse_metadata();
+ else
+ tst_brk(TBROK, "No metadata found!");
+}
+
+int main(int argc, char *argv[])
+{
+ if (argc < 2)
+ goto help;
+
+ shell_filename = argv[1];
+
+ prepare_test_struct();
+
+ tst_run_tcases(argc - 1, argv + 1, &test);
+help:
+ print_help();
+ return 1;
+}
--
2.44.2
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 0/3] Shell test library v3
2024-07-31 9:20 [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
` (2 preceding siblings ...)
2024-07-31 9:20 ` [LTP] [PATCH 3/3] testcaes/lib: Add shell loader Cyril Hrubis
@ 2024-07-31 10:06 ` Cyril Hrubis
3 siblings, 0 replies; 10+ messages in thread
From: Cyril Hrubis @ 2024-07-31 10:06 UTC (permalink / raw)
To: ltp
Hi!
And I've forget to amend the commit, so there is no support for tags
etc. please ignore this, will send v2.
--
Cyril Hrubis
chrubis@suse.cz
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 1/3] Add support for mixing C and shell code
2024-07-31 9:20 ` [LTP] [PATCH 1/3] Add support for mixing C and shell code Cyril Hrubis
@ 2024-08-13 15:35 ` Richard Palethorpe
2024-08-26 14:51 ` Cyril Hrubis
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2024-08-13 15:35 UTC (permalink / raw)
To: Cyril Hrubis; +Cc: ltp
Cyril Hrubis <chrubis@suse.cz> writes:
Hello Cyril,
In general I like the idea except that it will encourage more shell
usage, but I'm guessing that is a battle that has already been
lost. Also if a suitable embedded scripting language were found this is
a starting point for that.
> This is a proof of a concept of a seamless C and shell integration. The
> idea is that with this you can mix shell and C code as much as as you
> wish to get the best of the two worlds.
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
> include/tst_test.h | 38 +++++++++++++
> lib/tst_test.c | 51 +++++++++++++++++
> testcases/lib/.gitignore | 1 +
> testcases/lib/Makefile | 4 +-
> testcases/lib/run_tests.sh | 11 ++++
> testcases/lib/tests/.gitignore | 6 ++
> testcases/lib/tests/Makefile | 11 ++++
> testcases/lib/tests/shell_test01.c | 17 ++++++
> testcases/lib/tests/shell_test02.c | 18 ++++++
> testcases/lib/tests/shell_test03.c | 25 +++++++++
> testcases/lib/tests/shell_test04.c | 18 ++++++
> testcases/lib/tests/shell_test05.c | 27 +++++++++
> testcases/lib/tests/shell_test06.c | 16 ++++++
> testcases/lib/tests/shell_test_brk.sh | 6 ++
> testcases/lib/tests/shell_test_check_argv.sh | 23 ++++++++
> testcases/lib/tests/shell_test_checkpoint.sh | 7 +++
> testcases/lib/tests/shell_test_pass.sh | 6 ++
> testcases/lib/tst_env.sh | 21 +++++++
> testcases/lib/tst_res_.c | 58 ++++++++++++++++++++
> 19 files changed, 362 insertions(+), 2 deletions(-)
> create mode 100755 testcases/lib/run_tests.sh
> create mode 100644 testcases/lib/tests/.gitignore
> create mode 100644 testcases/lib/tests/Makefile
> create mode 100644 testcases/lib/tests/shell_test01.c
> create mode 100644 testcases/lib/tests/shell_test02.c
> create mode 100644 testcases/lib/tests/shell_test03.c
> create mode 100644 testcases/lib/tests/shell_test04.c
> create mode 100644 testcases/lib/tests/shell_test05.c
> create mode 100644 testcases/lib/tests/shell_test06.c
> create mode 100755 testcases/lib/tests/shell_test_brk.sh
> create mode 100755 testcases/lib/tests/shell_test_check_argv.sh
> create mode 100755 testcases/lib/tests/shell_test_checkpoint.sh
> create mode 100755 testcases/lib/tests/shell_test_pass.sh
> create mode 100644 testcases/lib/tst_env.sh
> create mode 100644 testcases/lib/tst_res_.c
>
> diff --git a/include/tst_test.h b/include/tst_test.h
> index 6c76f043d..a334195ac 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -331,6 +331,8 @@ struct tst_fs {
> * @child_needs_reinit: Has to be set if the test needs to call tst_reinit()
> * from a process started by exec().
> *
> + * @runs_script: Implies child_needs_reinit and forks_child at the moment.
> + *
> * @needs_devfs: If set the devfs is mounted at tst_test.mntpoint. This is
> * needed for tests that need to create device files since tmpfs
> * at /tmp is usually mounted with 'nodev' option.
> @@ -518,6 +520,7 @@ struct tst_fs {
> unsigned int mount_device:1;
> unsigned int needs_rofs:1;
> unsigned int child_needs_reinit:1;
> + unsigned int runs_script:1;
This could be a string constant instead of a flag if you want to future
proof against multiple scripting languages or you could change it to runs_shell.
> unsigned int needs_devfs:1;
> unsigned int restore_wallclock:1;
>
> @@ -526,6 +529,8 @@ struct tst_fs {
> unsigned int skip_in_lockdown:1;
> unsigned int skip_in_secureboot:1;
> unsigned int skip_in_compat:1;
> +
> +
> int needs_abi_bits;
>
> unsigned int needs_hugetlbfs:1;
> @@ -611,6 +616,39 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
> */
> void tst_reinit(void);
>
> +/**
> + * tst_run_shell() - Prepare the environment and execute a shell script.
> + *
> + * @script_name: A filename of the script.
> + * @params: A NULL terminated array of shell script parameters, pass NULL if
> + * none are needed. This what is passed starting from argv[1].
> + *
> + * The shell script is executed with LTP_IPC_PATH in environment so that the
> + * binary helpers such as tst_res_ or tst_checkpoint work properly when executed
> + * from the script. This also means that the tst_test.runs_script flag needs to
> + * be set.
> + *
> + * The shell script itself has to source the tst_env.sh shell script at the
> + * start and after that it's free to use tst_res in the same way C code would
> + * use.
> + *
> + * Example shell script that reports success::
> + *
> + * #!/bin/sh
> + * . tst_env.sh
> + *
> + * tst_res TPASS "Example test works"
> + *
> + * The call returns a pid in a case that you want to examine the return value
> + * of the script yourself. If you do not need to check the return value
> + * yourself you can use tst_reap_children() to wait for the completion. Or let
> + * the test library collect the child automatically, just be wary that the
> + * script and the test both runs concurently at the same time in this case.
> + *
> + * Return: A pid of the shell process.
> + */
> +int tst_run_shell(char *script_name, char *const params[]);
> +
> unsigned int tst_multiply_timeout(unsigned int timeout);
>
> /*
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index e5bc5bf4d..7e1075fdf 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -4,6 +4,8 @@
> * Copyright (c) Linux Test Project, 2016-2024
> */
>
> +#define _GNU_SOURCE
> +
> #include <limits.h>
> #include <stdio.h>
> #include <stdarg.h>
> @@ -173,6 +175,50 @@ void tst_reinit(void)
> SAFE_CLOSE(fd);
> }
>
> +extern char **environ;
> +
> +static unsigned int params_array_len(char *const array[])
> +{
> + unsigned int ret = 0;
> +
> + if (!array)
> + return 0;
> +
> + while (*(array++))
> + ret++;
> +
> + return ret;
> +}
> +
> +int tst_run_shell(char *script_name, char *const params[])
script_name should really be const as this is an API
> +{
> + int pid;
> + unsigned int i, params_len = params_array_len(params);
> + char *argv[params_len + 2];
> +
> + if (!tst_test->runs_script)
> + tst_brk(TBROK, "runs_script flag must be set!");
> +
> + argv[0] = script_name;
maybe you'd have to cast it here or something, but IMO worth it
Reviewed-by: Richard Palethorpe <io@richiejp.com>
--
Thank you,
Richard,
richiejp.com.
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 2/3] libs: Vendor ujson library
2024-07-31 9:20 ` [LTP] [PATCH 2/3] libs: Vendor ujson library Cyril Hrubis
@ 2024-08-13 15:38 ` Richard Palethorpe
0 siblings, 0 replies; 10+ messages in thread
From: Richard Palethorpe @ 2024-08-13 15:38 UTC (permalink / raw)
To: Cyril Hrubis; +Cc: ltp
I spent quite some time in the past looking at this lib and don't think
I found any issues that I haven't already mentioned to you, so:
Acked-by: Richard Palethorpe <io@richiejp.com>
--
Thank you,
Richard,
richiejp.com.
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 3/3] testcaes/lib: Add shell loader
2024-07-31 9:20 ` [LTP] [PATCH 3/3] testcaes/lib: Add shell loader Cyril Hrubis
@ 2024-08-13 16:15 ` Richard Palethorpe
2024-08-26 15:10 ` Cyril Hrubis
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2024-08-13 16:15 UTC (permalink / raw)
To: Cyril Hrubis; +Cc: ltp
Hello Cyril Hrubis <chrubis@suse.cz> writes:
> This commit implements a shell loader so that we don't have to write a C
> loader for each LTP shell test. The idea is simple, the loader parses
> the shell test and prepares the tst_test structure accordingly, then
> runs the actual shell test.
>
> The format for the metadata in the shell test was choosen to be JSON
> because:
>
> - I didn't want to invent an adhoc format and JSON is perfect for
> serializing data structures
> - The metadata parser for shell test will be trivial, it will just pick
> the JSON from the comment, no parsing will be required
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
> include/tst_test.h | 2 +-
> testcases/lib/.gitignore | 1 +
> testcases/lib/Makefile | 6 +-
> testcases/lib/run_tests.sh | 9 +
> testcases/lib/tests/shell_loader.sh | 15 +
> .../lib/tests/shell_loader_all_filesystems.sh | 24 ++
> .../lib/tests/shell_loader_filesystems.sh | 30 ++
> .../tests/shell_loader_invalid_metadata.sh | 12 +
> .../lib/tests/shell_loader_no_metadata.sh | 8 +
> .../lib/tests/shell_loader_supported_archs.sh | 9 +
> .../lib/tests/shell_loader_wrong_metadata.sh | 12 +
> testcases/lib/tst_env.sh | 4 +
> testcases/lib/tst_loader.sh | 11 +
> testcases/lib/tst_run_shell.c | 378 ++++++++++++++++++
> 14 files changed, 519 insertions(+), 2 deletions(-)
> create mode 100755 testcases/lib/tests/shell_loader.sh
> create mode 100755 testcases/lib/tests/shell_loader_all_filesystems.sh
> create mode 100755 testcases/lib/tests/shell_loader_filesystems.sh
> create mode 100755 testcases/lib/tests/shell_loader_invalid_metadata.sh
> create mode 100755 testcases/lib/tests/shell_loader_no_metadata.sh
> create mode 100755 testcases/lib/tests/shell_loader_supported_archs.sh
> create mode 100755 testcases/lib/tests/shell_loader_wrong_metadata.sh
> create mode 100644 testcases/lib/tst_loader.sh
> create mode 100644 testcases/lib/tst_run_shell.c
>
> diff --git a/include/tst_test.h b/include/tst_test.h
> index a334195ac..c04498937 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -274,7 +274,7 @@ struct tst_fs {
> const char *const *mkfs_opts;
> const char *mkfs_size_opt;
>
> - const unsigned int mnt_flags;
> + unsigned int mnt_flags;
> const void *mnt_data;
> };
>
> diff --git a/testcases/lib/.gitignore b/testcases/lib/.gitignore
> index d0dacf62a..385f3c3ca 100644
> --- a/testcases/lib/.gitignore
> +++ b/testcases/lib/.gitignore
> @@ -24,3 +24,4 @@
> /tst_supported_fs
> /tst_timeout_kill
> /tst_res_
> +/tst_run_shell
> diff --git a/testcases/lib/Makefile b/testcases/lib/Makefile
> index 928d76d62..b3a9181c1 100644
> --- a/testcases/lib/Makefile
> +++ b/testcases/lib/Makefile
> @@ -4,6 +4,9 @@
>
> top_srcdir ?= ../..
>
> +LTPLIBS = ujson
> +tst_run_shell: LTPLDLIBS = -lujson
> +
> include $(top_srcdir)/include/mk/testcases.mk
>
> INSTALL_TARGETS := *.sh
> @@ -13,6 +16,7 @@ MAKE_TARGETS := tst_sleep tst_random tst_checkpoint tst_rod tst_kvcmp\
> tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port\
> tst_get_median tst_hexdump tst_get_free_pids tst_timeout_kill\
> tst_check_kconfigs tst_cgctl tst_fsfreeze tst_ns_create tst_ns_exec\
> - tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled tst_res_
> + tst_ns_ifmove tst_lockdown_enabled tst_secureboot_enabled tst_res_\
> + tst_run_shell
>
> include $(top_srcdir)/include/mk/generic_trunk_target.mk
> diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
> index 60e7d1bcf..9857f5f82 100755
> --- a/testcases/lib/run_tests.sh
> +++ b/testcases/lib/run_tests.sh
> @@ -9,3 +9,12 @@ for i in `seq -w 01 06`; do
> echo
> ./tests/shell_test$i
> done
> +
> +for i in shell_loader.sh shell_loader_all_filesystems.sh shell_loader_no_metadata.sh \
> + shell_loader_wrong_metadata.sh shell_loader_invalid_metadata.sh\
> + shell_loader_supported_archs.sh shell_loader_filesystems.sh; do
> + echo
> + echo "*** Running $i ***"
> + echo
> + $i
> +done
> diff --git a/testcases/lib/tests/shell_loader.sh b/testcases/lib/tests/shell_loader.sh
> new file mode 100755
> index 000000000..642ffe97b
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader.sh
> @@ -0,0 +1,15 @@
> +#!/bin/sh
> +#
> +# TEST = {
> +# "needs_tmpdir": true
> +# }
> +
> +. tst_loader.sh
> +
> +tst_res TPASS "Shell loader works fine!"
> +case "$PWD" in
> + /tmp/*)
> + tst_res TPASS "We are running in temp directory in $PWD";;
> + *)
> + tst_res TFAIL "We are not running in temp directory but $PWD";;
> +esac
> diff --git a/testcases/lib/tests/shell_loader_all_filesystems.sh b/testcases/lib/tests/shell_loader_all_filesystems.sh
> new file mode 100755
> index 000000000..8432b4b3d
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_all_filesystems.sh
> @@ -0,0 +1,24 @@
> +#!/bin/sh
> +#
> +# TEST = {
> +# "needs_root": true,
> +# "mount_device": true,
> +# "all_filesystems": true,
> +# "mntpoint": "ltp_mntpoint"
> +# }
> +
> +. tst_loader.sh
> +
> +tst_res TINFO "In shell"
> +
> +mntpath=$(realpath ltp_mntpoint)
> +mounted=$(grep $mntpath /proc/mounts)
> +
> +if [ -n "$mounted" ]; then
> + device=$(echo $mounted |cut -d' ' -f 1)
> + path=$(echo $mounted |cut -d' ' -f 2)
> +
> + tst_res TPASS "$device mounted at $path"
> +else
> + tst_res TFAIL "Device not mounted!"
> +fi
> diff --git a/testcases/lib/tests/shell_loader_filesystems.sh b/testcases/lib/tests/shell_loader_filesystems.sh
> new file mode 100755
> index 000000000..ede1a8fe9
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_filesystems.sh
> @@ -0,0 +1,30 @@
> +#!/bin/sh
> +#
> +# TEST = {
> +# "mount_device": true,
> +# "mntpoint": "ltp_mntpoint",
> +# "filesystems": [
> +# {
> +# "type": "btrfs"
> +# },
> +# {
> +# "type": "xfs",
> +# "mkfs_opts": ["-m", "reflink=1"]
> +# }
> +# ]
> +# }
> +
> +. tst_loader.sh
> +
> +tst_res TINFO "In shell"
> +
> +mntpoint=$(realpath ltp_mntpoint)
> +mounted=$(grep $mntpoint /proc/mounts)
> +
> +if [ -n "$mounted" ]; then
> + fs=$(echo $mounted |cut -d' ' -f 3)
> +
> + tst_res TPASS "Mounted device formatted with $fs"
> +else
> + tst_res TFAIL "Device not mounted!"
> +fi
> diff --git a/testcases/lib/tests/shell_loader_invalid_metadata.sh b/testcases/lib/tests/shell_loader_invalid_metadata.sh
> new file mode 100755
> index 000000000..265be6f36
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_invalid_metadata.sh
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +#
> +# This test has wrong metadata and should not be run
> +#
> +# TEST = {
> +# {"needs_tmpdir": 42,
> +# }
> +#
> +
> +. tst_loader.sh
> +
> +tst_res TFAIL "Shell loader should TBROK the test"
> diff --git a/testcases/lib/tests/shell_loader_no_metadata.sh b/testcases/lib/tests/shell_loader_no_metadata.sh
> new file mode 100755
> index 000000000..60ba8b889
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_no_metadata.sh
> @@ -0,0 +1,8 @@
> +#!/bin/sh
> +#
> +# This test has no metadata and should not be executed
> +#
> +
> +. tst_loader.sh
> +
> +tst_res TFAIL "Shell loader should TBROK the test"
> diff --git a/testcases/lib/tests/shell_loader_supported_archs.sh b/testcases/lib/tests/shell_loader_supported_archs.sh
> new file mode 100755
> index 000000000..d5c6c648b
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_supported_archs.sh
> @@ -0,0 +1,9 @@
> +#!/bin/sh
> +#
> +# TEST = {
> +# "supported_archs": ["x86", "ppc64", "x86_64"]
> +# }
> +
> +. tst_loader.sh
> +
> +tst_res TPASS "We are running on supported architecture"
> diff --git a/testcases/lib/tests/shell_loader_wrong_metadata.sh b/testcases/lib/tests/shell_loader_wrong_metadata.sh
> new file mode 100755
> index 000000000..f29b9308f
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_wrong_metadata.sh
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +#
> +# This test has wrong metadata and should not be run
> +#
> +# TEST = {
> +# "needs_tmpdir": 42,
> +# }
> +#
> +
> +. tst_loader.sh
> +
> +tst_res TFAIL "Shell loader should TBROK the test"
> diff --git a/testcases/lib/tst_env.sh b/testcases/lib/tst_env.sh
> index 948bc5024..67ba80744 100644
> --- a/testcases/lib/tst_env.sh
> +++ b/testcases/lib/tst_env.sh
> @@ -1,4 +1,8 @@
> #!/bin/sh
> +#
> +# This is a minimal test environment for a shell scripts executed from C by
> +# tst_run_shell() function. Shell tests must use the tst_loader.sh instead!
> +#
>
> tst_script_name=$(basename $0)
>
> diff --git a/testcases/lib/tst_loader.sh b/testcases/lib/tst_loader.sh
> new file mode 100644
> index 000000000..5ac095e44
> --- /dev/null
> +++ b/testcases/lib/tst_loader.sh
> @@ -0,0 +1,11 @@
> +#!/bin/sh
> +#
> +# This is a loader for shell tests that use the C test library.
> +#
> +
> +if [ -z "$LTP_IPC_PATH" ]; then
> + tst_run_shell $(basename "$0")
> + exit $?
> +else
> + . tst_env.sh
> +fi
> diff --git a/testcases/lib/tst_run_shell.c b/testcases/lib/tst_run_shell.c
> new file mode 100644
> index 000000000..96827e1cc
> --- /dev/null
> +++ b/testcases/lib/tst_run_shell.c
> @@ -0,0 +1,378 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * Copyright (c) 2024 Cyril Hrubis <chrubis@suse.cz>
> + */
> +#include <sys/mount.h>
> +
> +#define TST_NO_DEFAULT_MAIN
> +#include "tst_test.h"
> +#include "tst_safe_stdio.h"
> +#include "ujson.h"
> +
> +static char *shell_filename;
> +
> +static void run_shell(void)
> +{
> + tst_run_shell(shell_filename, NULL);
> +}
> +
> +struct tst_test test = {
> + .test_all = run_shell,
> + .runs_script = 1,
> +};
> +
> +static void print_help(void)
> +{
> + printf("Usage: tst_shell_loader ltp_shell_test.sh ...");
> +}
> +
> +static char *metadata;
> +static size_t metadata_size;
> +static size_t metadata_used;
> +
> +static void metadata_append(const char *line)
> +{
> + size_t linelen = strlen(line);
> +
> + if (metadata_size - metadata_used < linelen + 1) {
> + metadata_size += 128;
This seems like a very small amount to bother allocating.
> + metadata = SAFE_REALLOC(metadata, metadata_size);
> + }
> +
> + strcpy(metadata + metadata_used, line);
> + metadata_used += linelen;
> +}
> +
> +static ujson_obj_attr test_attrs[] = {
> + UJSON_OBJ_ATTR("all_filesystems", UJSON_BOOL),
> + UJSON_OBJ_ATTR("dev_min_size", UJSON_INT),
> + UJSON_OBJ_ATTR("filesystems", UJSON_ARR),
> + UJSON_OBJ_ATTR("format_device", UJSON_BOOL),
> + UJSON_OBJ_ATTR("min_cpus", UJSON_INT),
> + UJSON_OBJ_ATTR("min_mem_avail", UJSON_INT),
> + UJSON_OBJ_ATTR("min_kver", UJSON_STR),
> + UJSON_OBJ_ATTR("min_swap_avail", UJSON_INT),
> + UJSON_OBJ_ATTR("mntpoint", UJSON_STR),
> + UJSON_OBJ_ATTR("mount_device", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_abi_bits", UJSON_INT),
> + UJSON_OBJ_ATTR("needs_devfs", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_device", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_hugetlbfs", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_rofs", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_root", UJSON_BOOL),
> + UJSON_OBJ_ATTR("needs_tmpdir", UJSON_BOOL),
> + UJSON_OBJ_ATTR("restore_wallclock", UJSON_BOOL),
> + UJSON_OBJ_ATTR("skip_filesystems", UJSON_ARR),
> + UJSON_OBJ_ATTR("skip_in_compat", UJSON_BOOL),
> + UJSON_OBJ_ATTR("skip_in_lockdown", UJSON_BOOL),
> + UJSON_OBJ_ATTR("skip_in_secureboot", UJSON_BOOL),
> + UJSON_OBJ_ATTR("supported_archs", UJSON_ARR),
> +};
> +
> +static ujson_obj test_obj = {
> + .attrs = test_attrs,
> + .attr_cnt = UJSON_ARRAY_SIZE(test_attrs),
> +};
> +
> +/* Must match the order of test_attrs. */
You could use the index syntax like [ALL_FILESYSTEMS] = UJASON_OBJ_ATTR...
IIRC.
Then the order can't be messed up
> +enum test_attr_ids {
> + ALL_FILESYSTEMS,
> + DEV_MIN_SIZE,
> + FILESYSTEMS,
> + FORMAT_DEVICE,
> + MIN_CPUS,
> + MIN_MEM_AVAIL,
> + MIN_KVER,
> + MIN_SWAP_AVAIL,
> + MNTPOINT,
> + MOUNT_DEVICE,
> + NEEDS_ABI_BITS,
> + NEEDS_DEVFS,
> + NEEDS_DEVICE,
> + NEEDS_HUGETLBFS,
> + NEEDS_ROFS,
> + NEEDS_ROOT,
> + NEEDS_TMPDIR,
> + RESTORE_WALLCLOCK,
> + SKIP_FILESYSTEMS,
> + SKIP_IN_COMPAT,
> + SKIP_IN_LOCKDOWN,
> + SKIP_IN_SECUREBOOT,
> + SUPPORTED_ARCHS,
> +};
> +
> +static const char *const *parse_strarr(ujson_reader *reader, ujson_val *val)
> +{
> + unsigned int cnt = 0, i = 0;
> + char **ret;
> +
> + ujson_reader_state state = ujson_reader_state_save(reader);
> +
> + UJSON_ARR_FOREACH(reader, val) {
> + if (val->type != UJSON_STR) {
> + ujson_err(reader, "Expected string!");
> + return NULL;
> + }
> +
> + cnt++;
> + }
> +
> + ujson_reader_state_load(reader, state);
> +
> + ret = SAFE_MALLOC(sizeof(char*) * (cnt + 1));
> +
> + UJSON_ARR_FOREACH(reader, val) {
> + ret[i++] = strdup(val->val_str);
> + }
> +
> + ret[i] = NULL;
> +
> + return (const char *const *)ret;
> +}
> +
> +static ujson_obj_attr fs_attrs[] = {
> + UJSON_OBJ_ATTR("mkfs_opts", UJSON_ARR),
> + UJSON_OBJ_ATTR("mkfs_size_opt", UJSON_STR),
> + UJSON_OBJ_ATTR("mnt_flags", UJSON_ARR),
> + UJSON_OBJ_ATTR("type", UJSON_STR),
> +};
> +
> +static ujson_obj fs_obj = {
> + .attrs = fs_attrs,
> + .attr_cnt = UJSON_ARRAY_SIZE(fs_attrs),
> +};
> +
> +/* Must match the order of fs_attrs. */
> +enum fs_ids {
> + MKFS_OPTS,
> + MKFS_SIZE_OPT,
> + MNT_FLAGS,
> + TYPE,
> +};
> +
> +static int parse_mnt_flags(ujson_reader *reader, ujson_val *val)
> +{
> + int ret = 0;
> +
> + UJSON_ARR_FOREACH(reader, val) {
> + if (val->type != UJSON_STR) {
> + ujson_err(reader, "Expected string!");
> + return ret;
> + }
> +
> + if (!strcmp(val->val_str, "RDONLY"))
> + ret |= MS_RDONLY;
> + else if (!strcmp(val->val_str, "NOATIME"))
> + ret |= MS_NOATIME;
> + else if (!strcmp(val->val_str, "NOEXEC"))
> + ret |= MS_NOEXEC;
> + else if (!strcmp(val->val_str, "NOSUID"))
> + ret |= MS_NOSUID;
> + else
> + ujson_err(reader, "Invalid mount flag");
> + }
> +
> + return ret;
> +}
> +
> +struct tst_fs *parse_filesystems(ujson_reader *reader, ujson_val *val)
> +{
> + unsigned int i = 0, cnt = 0;
> + struct tst_fs *ret;
> +
> + ujson_reader_state state = ujson_reader_state_save(reader);
> +
> + UJSON_ARR_FOREACH(reader, val) {
> + if (val->type != UJSON_OBJ) {
> + ujson_err(reader, "Expected object!");
> + return NULL;
> + }
> + ujson_obj_skip(reader);
> + cnt++;
> + }
> +
> + ujson_reader_state_load(reader, state);
> +
> + ret = SAFE_MALLOC(sizeof(struct tst_fs) * (cnt + 1));
> + memset(ret, 0, sizeof(*ret) * (cnt+1));
> +
> + UJSON_ARR_FOREACH(reader, val) {
> + UJSON_OBJ_FOREACH_FILTER(reader, val, &fs_obj, ujson_empty_obj) {
> + switch ((enum fs_ids)val->idx) {
> + case MKFS_OPTS:
> + ret[i].mkfs_opts = parse_strarr(reader, val);
> + break;
> + case MKFS_SIZE_OPT:
> + ret[i].mkfs_size_opt = strdup(val->val_str);
> + break;
> + case MNT_FLAGS:
> + ret[i].mnt_flags = parse_mnt_flags(reader, val);
> + break;
> + case TYPE:
> + ret[i].type = strdup(val->val_str);
> + break;
> + }
> +
> + }
> +
> + i++;
> + }
> +
> + return ret;
> +}
> +
> +static void parse_metadata(void)
> +{
> + ujson_reader reader = UJSON_READER_INIT(metadata, metadata_used, UJSON_READER_STRICT);
> + char str_buf[128];
> + ujson_val val = UJSON_VAL_INIT(str_buf, sizeof(str_buf));
> +
> + UJSON_OBJ_FOREACH_FILTER(&reader, &val, &test_obj, ujson_empty_obj) {
> + switch ((enum test_attr_ids)val.idx) {
> + case ALL_FILESYSTEMS:
> + test.all_filesystems = val.val_bool;
> + break;
> + case DEV_MIN_SIZE:
> + if (val.val_int <= 0)
> + ujson_err(&reader, "Device size must be > 0");
> + else
> + test.dev_min_size = val.val_int;
> + break;
> + case FILESYSTEMS:
> + test.filesystems = parse_filesystems(&reader, &val);
> + break;
> + case FORMAT_DEVICE:
> + test.format_device = val.val_bool;
> + break;
> + case MIN_CPUS:
> + if (val.val_int <= 0)
> + ujson_err(&reader, "Minimal number of cpus must be > 0");
> + else
> + test.min_cpus = val.val_int;
> + break;
> + case MIN_MEM_AVAIL:
> + if (val.val_int <= 0)
> + ujson_err(&reader, "Minimal available memory size must be > 0");
> + else
> + test.min_mem_avail = val.val_int;
> + break;
> + case MIN_KVER:
> + test.min_kver = strdup(val.val_str);
> + break;
> + case MIN_SWAP_AVAIL:
> + if (val.val_int <= 0)
> + ujson_err(&reader, "Minimal available swap size must be > 0");
> + else
> + test.min_swap_avail = val.val_int;
> + break;
> + case MNTPOINT:
> + test.mntpoint = strdup(val.val_str);
> + break;
> + case MOUNT_DEVICE:
> + test.mount_device = val.val_bool;
> + break;
> + case NEEDS_ABI_BITS:
> + if (val.val_int == 32 || val.val_int == 64)
> + test.needs_abi_bits = val.val_int;
> + else
> + ujson_err(&reader, "ABI bits must be 32 or 64");
> + break;
> + case NEEDS_DEVFS:
> + test.needs_devfs = val.val_bool;
> + break;
> + case NEEDS_DEVICE:
> + test.needs_device = val.val_bool;
> + break;
> + case NEEDS_HUGETLBFS:
> + test.needs_hugetlbfs = val.val_bool;
> + break;
> + case NEEDS_ROFS:
> + test.needs_rofs = val.val_bool;
> + break;
> + case NEEDS_ROOT:
> + test.needs_root = val.val_bool;
> + break;
> + case NEEDS_TMPDIR:
> + test.needs_tmpdir = val.val_bool;
> + break;
> + case RESTORE_WALLCLOCK:
> + test.restore_wallclock = val.val_bool;
> + break;
> + case SKIP_FILESYSTEMS:
> + test.skip_filesystems = parse_strarr(&reader, &val);
> + break;
> + case SKIP_IN_COMPAT:
> + test.skip_in_compat = val.val_bool;
> + break;
> + case SKIP_IN_LOCKDOWN:
> + test.skip_in_lockdown = val.val_bool;
> + break;
> + case SKIP_IN_SECUREBOOT:
> + test.skip_in_secureboot = val.val_bool;
> + break;
> + case SUPPORTED_ARCHS:
> + test.supported_archs = parse_strarr(&reader, &val);
> + break;
> + }
> + }
> +
> + ujson_reader_finish(&reader);
> +
> + if (ujson_reader_err(&reader))
> + tst_brk(TBROK, "Invalid metadata");
> +}
> +
> +static void extract_metadata(void)
> +{
> + FILE *f;
> + char line[4096];
> + char path[4096];
> + int in_json = 0;
> +
> + if (tst_get_path(shell_filename, path, sizeof(path)) == -1)
> + tst_brk(TBROK, "Failed to find %s in $PATH", shell_filename);
> +
> + f = SAFE_FOPEN(path, "r");
> +
> + while (fgets(line, sizeof(line), f)) {
> + if (in_json)
> + metadata_append(line + 2);
> +
> + if (in_json) {
> + if (!strcmp(line, "# }\n"))
> + in_json = 0;
> + } else {
> + if (!strcmp(line, "# TEST = {\n")) {
> + metadata_append("{\n");
> + in_json = 1;
> + }
> + }
This is maybe a little bit too rigid, even if you want this exact
formatting to be the only valid one. People will get frustrated when
parsing fails due to whitespace and don't see a clear indication why
it failed.
Alternativey you could adopt the standard markdown meta-data format
where the meta-data is enclosed with three dashes like
---
"meta": "data"
---
at the top of the file. I suppose you could change it to have comments
if you want the script to run outside LTP's runner.
In any case using three dots removes any whitespace between tokens which
is harder to trim.
The whole patchset looks good though.
Reviewed-by: Richard Palethorpe <io@richiejp.com>
--
Thank you,
Richard,
richiejp.com.
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 1/3] Add support for mixing C and shell code
2024-08-13 15:35 ` Richard Palethorpe
@ 2024-08-26 14:51 ` Cyril Hrubis
0 siblings, 0 replies; 10+ messages in thread
From: Cyril Hrubis @ 2024-08-26 14:51 UTC (permalink / raw)
To: Richard Palethorpe; +Cc: ltp
Hi!
> In general I like the idea except that it will encourage more shell
> usage, but I'm guessing that is a battle that has already been
> lost.
Indeed. However messy shell is we are not going to get rid of it.
> Also if a suitable embedded scripting language were found this is
> a starting point for that.
My hopes are not that high, LTP targets embedded, so we are quite
limited in what we can do here and having python or perl as a dependency
is a no-go. Perhaps there is a minimalistic scripting language that
integrated into UNIX well out there, but I'm not aware of any. So as far
as I can tell we are stuck with shell for the foreseeable future.
> > This is a proof of a concept of a seamless C and shell integration. The
> > idea is that with this you can mix shell and C code as much as as you
> > wish to get the best of the two worlds.
> >
> > Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> > ---
> > include/tst_test.h | 38 +++++++++++++
> > lib/tst_test.c | 51 +++++++++++++++++
> > testcases/lib/.gitignore | 1 +
> > testcases/lib/Makefile | 4 +-
> > testcases/lib/run_tests.sh | 11 ++++
> > testcases/lib/tests/.gitignore | 6 ++
> > testcases/lib/tests/Makefile | 11 ++++
> > testcases/lib/tests/shell_test01.c | 17 ++++++
> > testcases/lib/tests/shell_test02.c | 18 ++++++
> > testcases/lib/tests/shell_test03.c | 25 +++++++++
> > testcases/lib/tests/shell_test04.c | 18 ++++++
> > testcases/lib/tests/shell_test05.c | 27 +++++++++
> > testcases/lib/tests/shell_test06.c | 16 ++++++
> > testcases/lib/tests/shell_test_brk.sh | 6 ++
> > testcases/lib/tests/shell_test_check_argv.sh | 23 ++++++++
> > testcases/lib/tests/shell_test_checkpoint.sh | 7 +++
> > testcases/lib/tests/shell_test_pass.sh | 6 ++
> > testcases/lib/tst_env.sh | 21 +++++++
> > testcases/lib/tst_res_.c | 58 ++++++++++++++++++++
> > 19 files changed, 362 insertions(+), 2 deletions(-)
> > create mode 100755 testcases/lib/run_tests.sh
> > create mode 100644 testcases/lib/tests/.gitignore
> > create mode 100644 testcases/lib/tests/Makefile
> > create mode 100644 testcases/lib/tests/shell_test01.c
> > create mode 100644 testcases/lib/tests/shell_test02.c
> > create mode 100644 testcases/lib/tests/shell_test03.c
> > create mode 100644 testcases/lib/tests/shell_test04.c
> > create mode 100644 testcases/lib/tests/shell_test05.c
> > create mode 100644 testcases/lib/tests/shell_test06.c
> > create mode 100755 testcases/lib/tests/shell_test_brk.sh
> > create mode 100755 testcases/lib/tests/shell_test_check_argv.sh
> > create mode 100755 testcases/lib/tests/shell_test_checkpoint.sh
> > create mode 100755 testcases/lib/tests/shell_test_pass.sh
> > create mode 100644 testcases/lib/tst_env.sh
> > create mode 100644 testcases/lib/tst_res_.c
> >
> > diff --git a/include/tst_test.h b/include/tst_test.h
> > index 6c76f043d..a334195ac 100644
> > --- a/include/tst_test.h
> > +++ b/include/tst_test.h
> > @@ -331,6 +331,8 @@ struct tst_fs {
> > * @child_needs_reinit: Has to be set if the test needs to call tst_reinit()
> > * from a process started by exec().
> > *
> > + * @runs_script: Implies child_needs_reinit and forks_child at the moment.
> > + *
> > * @needs_devfs: If set the devfs is mounted at tst_test.mntpoint. This is
> > * needed for tests that need to create device files since tmpfs
> > * at /tmp is usually mounted with 'nodev' option.
> > @@ -518,6 +520,7 @@ struct tst_fs {
> > unsigned int mount_device:1;
> > unsigned int needs_rofs:1;
> > unsigned int child_needs_reinit:1;
> > + unsigned int runs_script:1;
>
> This could be a string constant instead of a flag if you want to future
> proof against multiple scripting languages or you could change it to runs_shell.
Heh, Martin Doucha requtested runs_shell to be changed to runs_script,
and that was a rightful request, since there is actually nothing shell
specific in the test library, it just sets up env variables and passes
command line option to something that is executed by execvpe(). All that
is missing for any other scripting language is a bit of glue code that
would interface the shared memory passed down in LTP_IPC_ENV.
So quite contrary the tst_run_shell() function should be renamed to
tst_run_script() as well, which I will do in v3.
> > + int pid;
> > + unsigned int i, params_len = params_array_len(params);
> > + char *argv[params_len + 2];
> > +
> > + if (!tst_test->runs_script)
> > + tst_brk(TBROK, "runs_script flag must be set!");
> > +
> > + argv[0] = script_name;
> maybe you'd have to cast it here or something, but IMO worth it
Hmm, I was expecting that we would have to copy the data, but casting
seems to work, will do.
--
Cyril Hrubis
chrubis@suse.cz
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [LTP] [PATCH 3/3] testcaes/lib: Add shell loader
2024-08-13 16:15 ` Richard Palethorpe
@ 2024-08-26 15:10 ` Cyril Hrubis
0 siblings, 0 replies; 10+ messages in thread
From: Cyril Hrubis @ 2024-08-26 15:10 UTC (permalink / raw)
To: Richard Palethorpe; +Cc: ltp
Hi!
> > +static void metadata_append(const char *line)
> > +{
> > + size_t linelen = strlen(line);
> > +
> > + if (metadata_size - metadata_used < linelen + 1) {
> > + metadata_size += 128;
>
> This seems like a very small amount to bother allocating.
That is what I have left there after the testing phase where I wanted to
trigger the reallocation. I will change this to 4k.
> > + metadata = SAFE_REALLOC(metadata, metadata_size);
> > + }
> > +
> > + strcpy(metadata + metadata_used, line);
> > + metadata_used += linelen;
> > +}
> > +
> > +static ujson_obj_attr test_attrs[] = {
> > + UJSON_OBJ_ATTR("all_filesystems", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("dev_min_size", UJSON_INT),
> > + UJSON_OBJ_ATTR("filesystems", UJSON_ARR),
> > + UJSON_OBJ_ATTR("format_device", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("min_cpus", UJSON_INT),
> > + UJSON_OBJ_ATTR("min_mem_avail", UJSON_INT),
> > + UJSON_OBJ_ATTR("min_kver", UJSON_STR),
> > + UJSON_OBJ_ATTR("min_swap_avail", UJSON_INT),
> > + UJSON_OBJ_ATTR("mntpoint", UJSON_STR),
> > + UJSON_OBJ_ATTR("mount_device", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_abi_bits", UJSON_INT),
> > + UJSON_OBJ_ATTR("needs_devfs", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_device", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_hugetlbfs", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_rofs", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_root", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("needs_tmpdir", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("restore_wallclock", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("skip_filesystems", UJSON_ARR),
> > + UJSON_OBJ_ATTR("skip_in_compat", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("skip_in_lockdown", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("skip_in_secureboot", UJSON_BOOL),
> > + UJSON_OBJ_ATTR("supported_archs", UJSON_ARR),
> > +};
> > +
> > +static ujson_obj test_obj = {
> > + .attrs = test_attrs,
> > + .attr_cnt = UJSON_ARRAY_SIZE(test_attrs),
> > +};
> > +
> > +/* Must match the order of test_attrs. */
>
> You could use the index syntax like [ALL_FILESYSTEMS] = UJASON_OBJ_ATTR...
> IIRC.
>
> Then the order can't be messed up
Good idea, will do. Maybe it makes sense to embedded this into the
UJSON_OBJ_ATTR() macro as a second parameter so that we have better
syntax too.
> > + while (fgets(line, sizeof(line), f)) {
> > + if (in_json)
> > + metadata_append(line + 2);
> > +
> > + if (in_json) {
> > + if (!strcmp(line, "# }\n"))
> > + in_json = 0;
> > + } else {
> > + if (!strcmp(line, "# TEST = {\n")) {
> > + metadata_append("{\n");
> > + in_json = 1;
> > + }
> > + }
>
> This is maybe a little bit too rigid, even if you want this exact
> formatting to be the only valid one. People will get frustrated when
> parsing fails due to whitespace and don't see a clear indication why
> it failed.
>
> Alternativey you could adopt the standard markdown meta-data format
> where the meta-data is enclosed with three dashes like
>
> ---
> "meta": "data"
> ---
That may be better, but we would need some markers for the asciidoc
documentation as well, so we need some kind of identifier too.
So maybe:
# ---
# doc
#
# [Description]
#
# This test does bla bla.
# ---
#
# ---
# env
#
# {
# "needs_root": 1,
# }
# ---
Getting the delimiters right when embedding data is
hard...
> at the top of the file. I suppose you could change it to have comments
> if you want the script to run outside LTP's runner.
We actually need it to be in the comments, because we run the script by
bash first, which bootstraps the enviroment from the shell script we
source first, then reexecutes the script in the right environment.
> In any case using three dots removes any whitespace between tokens which
> is harder to trim.
>
> The whole patchset looks good though.
Thanks a lot for the review, I will work on the small bits and send v3.
--
Cyril Hrubis
chrubis@suse.cz
--
Mailing list info: https://lists.linux.it/listinfo/ltp
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-08-26 15:11 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-07-31 9:20 [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 1/3] Add support for mixing C and shell code Cyril Hrubis
2024-08-13 15:35 ` Richard Palethorpe
2024-08-26 14:51 ` Cyril Hrubis
2024-07-31 9:20 ` [LTP] [PATCH 2/3] libs: Vendor ujson library Cyril Hrubis
2024-08-13 15:38 ` Richard Palethorpe
2024-07-31 9:20 ` [LTP] [PATCH 3/3] testcaes/lib: Add shell loader Cyril Hrubis
2024-08-13 16:15 ` Richard Palethorpe
2024-08-26 15:10 ` Cyril Hrubis
2024-07-31 10:06 ` [LTP] [PATCH 0/3] Shell test library v3 Cyril Hrubis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox