[LTP] [PATCH v5 0/3] Automatically generate syscalls.h

[LTP] [PATCH v5 0/3] Automatically generate syscalls.h

[Andrea Cervesato]

The problem we have at the moment is that all syscalls input files must
be changed by hand each time we need to test a new syscall. The idea is
to simplify this process, providing a script that is able to read from
kernel sources and to generate all syscalls files we need.

This patch set adds a new command inside the syscalls folder and
it refactors the previous regen.sh code.
The new way we have to generate the syscalls.h file is the following:

	include/lapi/syscalls/generate_arch.sh 		path/to/kernel/source
	include/lapi/syscalls/generate_syscalls.sh 	path/to/syscalls.h

Scripts are independent and they can be run separately.
generate_syscalls.sh is the equivalent of regen.sh, but its code has
been cleaned up.

Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
Changes in v5:
- license SPDX inside scripts files
- quote shell variables inside scripts
- some generic code cleanup
- Link to v4: https://lore.kernel.org/r/20241009-generate_syscalls-v4-0-5328a785bbad@suse.com

Changes in v4:
- remove blacklist file
- update syscalls files
- Link to v3: https://lore.kernel.org/r/20240930-generate_syscalls-v3-0-e56ede4bc7f5@suse.com

Changes in v3:
- fix generate_arch.sh to work properly for all architectures
- rename some arch .in files and handle renaming inside
  generate_syscalls.sh using the right gcc directives
- Link to v2: https://lore.kernel.org/r/20240927-generate_syscalls-v2-0-108f5f2ad318@suse.com

Changes in v2:
- refactor regen.sh
- split syscalls.h and syscalls .in files creation
- updated syscalls .in files
- Link to v1: https://lore.kernel.org/r/20240924-generate_syscalls-v1-0-941507a9cdac@suse.com

---
Andrea Cervesato (3):
      Refactor regen.sh script to generate syscalls
      Add script to generate arch(s) dependant syscalls
      Delete obsolete strip_syscall.awk file

 configure.ac                                       |   2 +-
 include/lapi/syscalls/{aarch64.in => arm64.in}     |   0
 include/lapi/syscalls/generate_arch.sh             | 211 +++++++++++++++++++++
 include/lapi/syscalls/generate_syscalls.sh         | 115 +++++++++++
 .../lapi/syscalls/{loongarch.in => loongarch64.in} |   0
 include/lapi/syscalls/{mips_n64.in => mips64.in}   |   0
 .../lapi/syscalls/{mips_n32.in => mips64n32.in}    |   0
 include/lapi/syscalls/{mips_o32.in => mipso32.in}  |   0
 include/lapi/syscalls/{hppa.in => parisc.in}       |   0
 include/lapi/syscalls/regen.sh                     | 129 -------------
 include/lapi/syscalls/strip_syscall.awk            |  19 --
 .../lapi/syscalls/{order => supported-arch.txt}    |  12 +-
 12 files changed, 333 insertions(+), 155 deletions(-)
---
base-commit: 34e6dd2d233abcebae435d34d793b6a49d08c190
change-id: 20240923-generate_syscalls-780dc2227bdb

Best regards,
-- 
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

Rename regen.sh into a more meaningful generate_syscalls.sh name, rename
order into a more meaningful supported-syscalls.txt name and rewrite
part of the regen.sh script code in order to execute it from anywhere in
the filesystem, without need to be in its own folder. The new code is
also more clear and concise, using native sh features which are
simplifying the code.

Reviewed-by: Li Wang <liwang@redhat.com>
Reviewed-by: Petr Vorel <pvorel@suse.cz>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 configure.ac                                       |   2 +-
 include/lapi/syscalls/generate_syscalls.sh         | 115 ++++++++++++++++++
 include/lapi/syscalls/regen.sh                     | 129 ---------------------
 .../lapi/syscalls/{order => supported-arch.txt}    |   0
 4 files changed, 116 insertions(+), 130 deletions(-)

diff --git a/configure.ac b/configure.ac
index d327974efa71f263d7f7f5aec9d2c5831d53dd0e..cd1233d19fad376973fc880d6689859845613fb0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -386,7 +386,7 @@ else
     AC_SUBST([WITH_REALTIME_TESTSUITE],["no"])
 fi
 
-AC_CONFIG_COMMANDS([syscalls.h], [cd ${ac_top_srcdir}/include/lapi/syscalls; ./regen.sh; cd - >/dev/null])
+AC_CONFIG_COMMANDS([syscalls.h], [cd ${ac_top_srcdir}/include/lapi/syscalls; ./generate_syscalls.sh ../syscalls.h; cd - >/dev/null])
 
 # custom functions
 # NOTE: don't create custom functions for simple checks, put them into this file
diff --git a/include/lapi/syscalls/generate_syscalls.sh b/include/lapi/syscalls/generate_syscalls.sh
new file mode 100755
index 0000000000000000000000000000000000000000..2bcb10b5d1bb67704f51e1a81ec979725353b692
--- /dev/null
+++ b/include/lapi/syscalls/generate_syscalls.sh
@@ -0,0 +1,115 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Generate the syscalls.h file, merging all architectures syscalls input file
+# which are in the current folder and defined inside supported-arch.txt file.
+
+SYSCALLS_FILE="$1"
+
+if [ -z "${SYSCALLS_FILE}" ]; then
+	echo "Please provide the syscalls.h directory:"
+	echo ""
+	echo "$0 path/of/syscalls.h"
+	echo ""
+	exit 1
+fi
+
+SCRIPT_DIR="$(realpath $(dirname "$0"))"
+SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
+
+echo '
+/************************************************
+* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
+*  change your arch specific .in file instead  *
+************************************************/
+
+/*
+* Here we stick all the ugly *fallback* logic for linux
+* system call numbers (those __NR_ thingies).
+*
+* Licensed under the GPLv2 or later, see the COPYING file.
+*/
+
+#ifndef LAPI_SYSCALLS_H__
+#define LAPI_SYSCALLS_H__
+
+#include <errno.h>
+#include <sys/syscall.h>
+#include <asm/unistd.h>
+
+#ifdef TST_TEST_H__
+#define TST_SYSCALL_BRK__(NR, SNR) ({ \
+tst_brk(TCONF, \
+	"syscall(%d) " SNR " not supported on your arch", NR); \
+})
+#else
+inline static void dummy_cleanup(void) {}
+
+#define TST_SYSCALL_BRK__(NR, SNR) ({ \
+tst_brkm(TCONF, dummy_cleanup, \
+	"syscall(%d) " SNR " not supported on your arch", NR); \
+})
+#endif
+
+#define tst_syscall(NR, ...) ({ \
+intptr_t tst_ret; \
+if (NR == __LTP__NR_INVALID_SYSCALL) { \
+	errno = ENOSYS; \
+	tst_ret = -1; \
+} else { \
+	tst_ret = syscall(NR, ##__VA_ARGS__); \
+} \
+if (tst_ret == -1 && errno == ENOSYS) { \
+	TST_SYSCALL_BRK__(NR, #NR); \
+} \
+tst_ret; \
+})
+
+#define __LTP__NR_INVALID_SYSCALL -1' >${SYSCALLS_FILE}
+
+while IFS= read -r arch; do
+	(
+		echo
+		case ${arch} in
+		sparc64) echo "#if defined(__sparc__) && defined(__arch64__)" ;;
+		sparc) echo "#if defined(__sparc__) && !defined(__arch64__)" ;;
+		s390) echo "#if defined(__s390__) && !defined(__s390x__)" ;;
+		mips64n32) echo "#if defined(__mips__) && defined(_ABIN32)" ;;
+		mips64) echo "#if defined(__mips__) && defined(_ABI64)" ;;
+		mipso32) echo "#if defined(__mips__) && defined(_ABIO32) && _MIPS_SZLONG == 32" ;;
+		parisc) echo "#ifdef __hppa__" ;;
+		loongarch64) echo "#ifdef __loongarch__" ;;
+		arm64) echo "#ifdef __aarch64__" ;;
+		*) echo "#ifdef __${arch}__" ;;
+		esac
+
+		while read -r line; do
+			set -- ${line}
+			syscall_nr="__NR_$1"
+			shift
+
+			echo "# ifndef ${syscall_nr}"
+			echo "#  define ${syscall_nr} $*"
+			echo "# endif"
+		done <"${SCRIPT_DIR}/${arch}.in"
+		echo "#endif"
+		echo
+	) >>${SYSCALLS_FILE}
+done <${SUPPORTED_ARCH}
+
+(
+	echo
+	echo "/* Common stubs */"
+	while IFS= read -r arch; do
+		while IFS= read -r line; do
+			set -- ${line}
+			syscall_nr="__NR_$1"
+			shift
+
+			echo "# ifndef ${syscall_nr}"
+			echo "#  define ${syscall_nr} __LTP__NR_INVALID_SYSCALL"
+			echo "# endif"
+		done <"${SCRIPT_DIR}/${arch}.in"
+	done <${SUPPORTED_ARCH}
+	echo "#endif"
+) >>${SYSCALLS_FILE}
diff --git a/include/lapi/syscalls/regen.sh b/include/lapi/syscalls/regen.sh
deleted file mode 100755
index 663ce4458bbc67c3a9e8073dfe8359164a9da0ee..0000000000000000000000000000000000000000
--- a/include/lapi/syscalls/regen.sh
+++ /dev/null
@@ -1,129 +0,0 @@
-#!/bin/sh
-
-output="syscalls.h"
-rm -f "${output}".[1-9]*
-output_pid="${output}.$$"
-
-max_jobs=$(getconf _NPROCESSORS_ONLN 2>/dev/null)
-: ${max_jobs:=1}
-
-srcdir=${0%/*}
-
-err() {
-	echo "$*" 1>&2
-	exit 1
-}
-
-cat << EOF > "${output_pid}"
-/************************************************
- * GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
- *  change your arch specific .in file instead  *
- ************************************************/
-
-/*
- * Here we stick all the ugly *fallback* logic for linux
- * system call numbers (those __NR_ thingies).
- *
- * Licensed under the GPLv2 or later, see the COPYING file.
- */
-
-#ifndef LAPI_SYSCALLS_H__
-#define LAPI_SYSCALLS_H__
-
-#include <errno.h>
-#include <sys/syscall.h>
-#include <asm/unistd.h>
-
-#ifdef TST_TEST_H__
-#define TST_SYSCALL_BRK__(NR, SNR) ({ \\
-	tst_brk(TCONF, \\
-		"syscall(%d) " SNR " not supported on your arch", NR); \\
-})
-#else
-inline static void dummy_cleanup(void) {}
-
-#define TST_SYSCALL_BRK__(NR, SNR) ({ \\
-	tst_brkm(TCONF, dummy_cleanup, \\
-		"syscall(%d) " SNR " not supported on your arch", NR); \\
-})
-#endif
-
-#define tst_syscall(NR, ...) ({ \\
-	intptr_t tst_ret; \\
-	if (NR == __LTP__NR_INVALID_SYSCALL) { \\
-		errno = ENOSYS; \\
-		tst_ret = -1; \\
-	} else { \\
-		tst_ret = syscall(NR, ##__VA_ARGS__); \\
-	} \\
-	if (tst_ret == -1 && errno == ENOSYS) { \\
-		TST_SYSCALL_BRK__(NR, #NR); \\
-	} \\
-	tst_ret; \\
-})
-
-EOF
-
-jobs=0
-for arch in $(cat "${srcdir}/order") ; do
-	(
-	echo "Generating data for arch $arch ... "
-
-	(
-	echo
-	case ${arch} in
-		sparc64) echo "#if defined(__sparc__) && defined(__arch64__)" ;;
-		sparc) echo "#if defined(__sparc__) && !defined(__arch64__)" ;;
-		s390) echo "#if defined(__s390__) && !defined(__s390x__)" ;;
-		mips_n32) echo "#if defined(__mips__) && defined(_ABIN32)" ;;
-		mips_n64) echo "#if defined(__mips__) && defined(_ABI64)" ;;
-		mips_o32) echo "#if defined(__mips__) && defined(_ABIO32) && _MIPS_SZLONG == 32" ;;
-		*) echo "#ifdef __${arch}__" ;;
-	esac
-	while read line ; do
-		set -- ${line}
-		nr="__NR_$1"
-		shift
-		if [ $# -eq 0 ] ; then
-			err "invalid line found: $line"
-		fi
-		echo "# ifndef ${nr}"
-		echo "#  define ${nr} $*"
-		echo "# endif"
-	done < "${srcdir}/${arch}.in"
-	echo "#endif"
-	echo
-	) >> "${output_pid}.${arch}"
-
-	) &
-
-	jobs=$(( jobs + 1 ))
-	if [ ${jobs} -ge ${max_jobs} ] ; then
-		wait || exit 1
-		jobs=0
-	fi
-done
-
-echo "Generating stub list ... "
-(
-echo
-echo "/* Common stubs */"
-echo "#define __LTP__NR_INVALID_SYSCALL -1" >> "${output_pid}"
-for nr in $(awk '{print $1}' "${srcdir}/"*.in | sort -u) ; do
-	nr="__NR_${nr}"
-	echo "# ifndef ${nr}"
-	echo "#  define ${nr} __LTP__NR_INVALID_SYSCALL"
-	echo "# endif"
-done
-echo "#endif"
-) >> "${output_pid}._footer"
-
-wait || exit 1
-
-printf "Combining them all ... "
-for arch in $(cat "${srcdir}/order") _footer ; do
-	cat "${output_pid}.${arch}"
-done >> "${output_pid}"
-mv "${output_pid}" "../${output}"
-rm -f "${output_pid}"*
-echo "OK!"
diff --git a/include/lapi/syscalls/order b/include/lapi/syscalls/supported-arch.txt
similarity index 100%
rename from include/lapi/syscalls/order
rename to include/lapi/syscalls/supported-arch.txt

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Cyril Hrubis]

Hi!
> +(
> +	echo
> +	echo "/* Common stubs */"
> +	while IFS= read -r arch; do
> +		while IFS= read -r line; do
> +			set -- ${line}
> +			syscall_nr="__NR_$1"
> +			shift
> +
> +			echo "# ifndef ${syscall_nr}"
> +			echo "#  define ${syscall_nr} __LTP__NR_INVALID_SYSCALL"
> +			echo "# endif"
> +		done <"${SCRIPT_DIR}/${arch}.in"
> +	done <${SUPPORTED_ARCH}
> +	echo "#endif"
> +) >>${SYSCALLS_FILE}

This will generate multiple fallback definitions, one per architecture
for common syscall, there was a good reason why these files were parsed
by awk with sort -u

> -echo "#define __LTP__NR_INVALID_SYSCALL -1" >> "${output_pid}"
> -for nr in $(awk '{print $1}' "${srcdir}/"*.in | sort -u) ; do

This is the important part that IMHO should stay.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Andrea Cervesato via ltp]

Hi Cyril,

On 10/30/24 14:47, Cyril Hrubis wrote:
> Hi!
>> +(
>> +	echo
>> +	echo "/* Common stubs */"
>> +	while IFS= read -r arch; do
>> +		while IFS= read -r line; do
>> +			set -- ${line}
>> +			syscall_nr="__NR_$1"
>> +			shift
>> +
>> +			echo "# ifndef ${syscall_nr}"
>> +			echo "#  define ${syscall_nr} __LTP__NR_INVALID_SYSCALL"
>> +			echo "# endif"
>> +		done <"${SCRIPT_DIR}/${arch}.in"
>> +	done <${SUPPORTED_ARCH}
>> +	echo "#endif"
>> +) >>${SYSCALLS_FILE}
> This will generate multiple fallback definitions, one per architecture
> for common syscall, there was a good reason why these files were parsed
> by awk with sort -u
I wrote the code a few weeks ago, so I needed to read it again to 
understand the reason.

If you take a look at the drop_bad_entries() and generate_tables() you 
will see that `sort` command is used to remove duplicates and to sort 
the entire list per architecture.

Or am I missing something?

>> -echo "#define __LTP__NR_INVALID_SYSCALL -1" >> "${output_pid}"
>> -for nr in $(awk '{print $1}' "${srcdir}/"*.in | sort -u) ; do
> This is the important part that IMHO should stay.
>
Andrea

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Cyril Hrubis]

Hi!
> > This will generate multiple fallback definitions, one per architecture
> > for common syscall, there was a good reason why these files were parsed
> > by awk with sort -u
> I wrote the code a few weeks ago, so I needed to read it again to 
> understand the reason.
> 
> If you take a look at the drop_bad_entries() and generate_tables() you 
> will see that `sort` command is used to remove duplicates and to sort 
> the entire list per architecture.

No no, this is the code that generates the actuall LTP header and what
you do here is:

	for each architecture:
		for each syscall in architecture:
			generate fallback with invalid value


And what you need to do is:

	read all syscall names for all architectures and do sort -u then
	generate fallback with invalid value


-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Petr Vorel]

Hi Andrea,

> +++ b/include/lapi/syscalls/generate_syscalls.sh
> @@ -0,0 +1,115 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Generate the syscalls.h file, merging all architectures syscalls input file
> +# which are in the current folder and defined inside supported-arch.txt file.
> +
> +SYSCALLS_FILE="$1"
> +
> +if [ -z "${SYSCALLS_FILE}" ]; then
nit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} =>
$SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use
${...} in whole script.

The same applies to include/lapi/syscalls/generate_arch.sh in the second commit.

> +	echo "Please provide the syscalls.h directory:"
> +	echo ""
> +	echo "$0 path/of/syscalls.h"
> +	echo ""
> +	exit 1
> +fi
> +
> +SCRIPT_DIR="$(realpath $(dirname "$0"))"
> +SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
> +
> +echo '
> +/************************************************
> +* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
> +*  change your arch specific .in file instead  *
> +************************************************/
> +
> +/*
> +* Here we stick all the ugly *fallback* logic for linux
> +* system call numbers (those __NR_ thingies).
> +*
> +* Licensed under the GPLv2 or later, see the COPYING file.
> +*/


nit: fixing space in /* */ comments (correct in the original regen.sh)
+ replacing old note about COPYING with SPDX in generated header:

+++ include/lapi/syscalls/generate_syscalls.sh
@@ -17,18 +17,16 @@ fi
 SCRIPT_DIR="$(realpath $(dirname "$0"))"
 SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
 
-echo '
+echo '// SPDX-License-Identifier: GPL-2.0-or-later
 /************************************************
-* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
-*  change your arch specific .in file instead  *
-************************************************/
+ * GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
+ *  change your arch specific .in file instead  *
+ ************************************************/
 
 /*
-* Here we stick all the ugly *fallback* logic for linux
-* system call numbers (those __NR_ thingies).
-*
-* Licensed under the GPLv2 or later, see the COPYING file.
-*/
+ * Here we stick all the ugly *fallback* logic for linux
+ * system call numbers (those __NR_ thingies).
+ */
 
 #ifndef LAPI_SYSCALLS_H__
 #define LAPI_SYSCALLS_H__
---

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Andrea Cervesato via ltp]

Hi Petr,

On 10/30/24 15:49, Petr Vorel wrote:
> Hi Andrea,
>
>> +++ b/include/lapi/syscalls/generate_syscalls.sh
>> @@ -0,0 +1,115 @@
>> +#!/bin/sh
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +#
>> +# Generate the syscalls.h file, merging all architectures syscalls input file
>> +# which are in the current folder and defined inside supported-arch.txt file.
>> +
>> +SYSCALLS_FILE="$1"
>> +
>> +if [ -z "${SYSCALLS_FILE}" ]; then
> nit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} =>
> $SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use
> ${...} in whole script.
>
> The same applies to include/lapi/syscalls/generate_arch.sh in the second commit.

This is a bit debatable. The reason why using ${VAR} over $VAR is the 
readability in the first place (i.e. when you have `${PATH}/bin` instead 
`$PATH/bin` or when you use characters which could mix with PATH var 
name and it might create debug issues), but also the fact it provides 
${VAR:-default} syntax which can always added afterwards.

I usually prefer this method for these reasons.

>
>> +	echo "Please provide the syscalls.h directory:"
>> +	echo ""
>> +	echo "$0 path/of/syscalls.h"
>> +	echo ""
>> +	exit 1
>> +fi
>> +
>> +SCRIPT_DIR="$(realpath $(dirname "$0"))"
>> +SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
>> +
>> +echo '
>> +/************************************************
>> +* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
>> +*  change your arch specific .in file instead  *
>> +************************************************/
>> +
>> +/*
>> +* Here we stick all the ugly *fallback* logic for linux
>> +* system call numbers (those __NR_ thingies).
>> +*
>> +* Licensed under the GPLv2 or later, see the COPYING file.
>> +*/
>
> nit: fixing space in /* */ comments (correct in the original regen.sh)
> + replacing old note about COPYING with SPDX in generated header:
Thanks you seen that. Now I know I can't fully trust auto-refactoring.
>
> +++ include/lapi/syscalls/generate_syscalls.sh
> @@ -17,18 +17,16 @@ fi
>   SCRIPT_DIR="$(realpath $(dirname "$0"))"
>   SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
>   
> -echo '
> +echo '// SPDX-License-Identifier: GPL-2.0-or-later
>   /************************************************
> -* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
> -*  change your arch specific .in file instead  *
> -************************************************/
> + * GENERATED FILE: DO NOT EDIT/PATCH THIS FILE  *
> + *  change your arch specific .in file instead  *
> + ************************************************/
>   
>   /*
> -* Here we stick all the ugly *fallback* logic for linux
> -* system call numbers (those __NR_ thingies).
> -*
> -* Licensed under the GPLv2 or later, see the COPYING file.
> -*/
> + * Here we stick all the ugly *fallback* logic for linux
> + * system call numbers (those __NR_ thingies).
> + */
>   
>   #ifndef LAPI_SYSCALLS_H__
>   #define LAPI_SYSCALLS_H__
> ---
>
> Kind regards,
> Petr
Andrea

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls

[Petr Vorel]

> Hi Petr,

> On 10/30/24 15:49, Petr Vorel wrote:
> > Hi Andrea,

> > > +++ b/include/lapi/syscalls/generate_syscalls.sh
> > > @@ -0,0 +1,115 @@
> > > +#!/bin/sh
> > > +# SPDX-License-Identifier: GPL-2.0-or-later
> > > +#
> > > +# Generate the syscalls.h file, merging all architectures syscalls input file
> > > +# which are in the current folder and defined inside supported-arch.txt file.
> > > +
> > > +SYSCALLS_FILE="$1"
> > > +
> > > +if [ -z "${SYSCALLS_FILE}" ]; then
> > nit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} =>
> > $SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use
> > ${...} in whole script.

> > The same applies to include/lapi/syscalls/generate_arch.sh in the second commit.

> This is a bit debatable. The reason why using ${VAR} over $VAR is the
> readability in the first place (i.e. when you have `${PATH}/bin` instead
> `$PATH/bin` or when you use characters which could mix with PATH var name
> and it might create debug issues), but also the fact it provides
> ${VAR:-default} syntax which can always added afterwards.

> I usually prefer this method for these reasons.

Sure, it's matter of preference. At least LTP shell library tst_test.sh
is happy with $VAR instead of ${VAR}. But as generate_arch.sh origin is in the
separate project, and this is a separate shell script (not part of shell API)
let's ignore my suggestion.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

Add generate_arch.sh script which can be used to generate arch(s)
dependant syscalls file. The way it works is pretty simple: for each
architecture defined into supported-arch.txt, compile kernel headers,
extract the list of syscalls and generate a .in file containing all of
them, associated with their own syscall's number.
The way syscalls files are generated, passes through a C application
which is automatically checking the availability of the syscalls in
the user space environment.

Reviewed-by: Li Wang <liwang@redhat.com>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 include/lapi/syscalls/{aarch64.in => arm64.in}     |   0
 include/lapi/syscalls/generate_arch.sh             | 211 +++++++++++++++++++++
 .../lapi/syscalls/{loongarch.in => loongarch64.in} |   0
 include/lapi/syscalls/{mips_n64.in => mips64.in}   |   0
 .../lapi/syscalls/{mips_n32.in => mips64n32.in}    |   0
 include/lapi/syscalls/{mips_o32.in => mipso32.in}  |   0
 include/lapi/syscalls/{hppa.in => parisc.in}       |   0
 include/lapi/syscalls/supported-arch.txt           |  12 +-
 8 files changed, 217 insertions(+), 6 deletions(-)

diff --git a/include/lapi/syscalls/aarch64.in b/include/lapi/syscalls/arm64.in
similarity index 100%
rename from include/lapi/syscalls/aarch64.in
rename to include/lapi/syscalls/arm64.in
diff --git a/include/lapi/syscalls/generate_arch.sh b/include/lapi/syscalls/generate_arch.sh
new file mode 100755
index 0000000000000000000000000000000000000000..c88ca945127c3c26162a48babcd6b0f8ca552311
--- /dev/null
+++ b/include/lapi/syscalls/generate_arch.sh
@@ -0,0 +1,211 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# This is an adaptation of the update-tables.sh script, included in the
+# syscalls-table project (https://github.com/hrw/syscalls-table) and released
+# under the MIT license.
+#
+# Author: Andrea Cervesato <andrea.cervesato@suse.com>
+
+KERNELSRC="$1"
+
+# to keep sorting in order
+export LC_ALL=C
+
+if [ -z "$KERNELSRC" ]; then
+	echo "Please provide kernel sources:"
+	echo ""
+	echo "$0 path/to/Linux/kernel/sources"
+	echo ""
+	exit 1
+fi
+
+if [ ! -d "$KERNELSRC" ]; then
+	echo "$KERNELSRC is not a directory"
+	exit 1
+fi
+
+if [ ! -e "${KERNELSRC}/Makefile" ]; then
+	echo "No Makefile in $KERNELSRC directory"
+	exit 1
+fi
+
+TEMP="$(mktemp -d)"
+KVER="$(make -C ${KERNELSRC} kernelversion -s)"
+
+SCRIPT_DIR="$(realpath $(dirname "$0"))"
+SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
+LINUX_HEADERS="${TEMP}/headers"
+
+grab_syscall_names_from_tables()
+{
+	for tbl_file in $(find ${KERNELSRC}/arch -name syscall*.tbl); do
+		grep -E -v "(^#|^$|sys_ni_syscall)" $tbl_file |
+			awk '{ print $3 }' >> ${TEMP}/syscall-names.tosort
+	done
+
+	drop_bad_entries
+}
+
+grab_syscall_names_from_unistd_h()
+{
+	grep -E -h "^#define __NR_" \
+		${LINUX_HEADERS}/usr/include/asm/unistd*.h \
+		${LINUX_HEADERS}/usr/include/asm-generic/unistd.h \
+		>${TEMP}/syscall-names.tosort
+
+	drop_bad_entries
+}
+
+drop_bad_entries()
+{
+	grep -E -v "(unistd.h|NR3264|__NR_syscall|__SC_COMP|__NR_.*Linux|__NR_FAST)" \
+		${TEMP}/syscall-names.tosort |
+		grep -E -v "(__SYSCALL|SYSCALL_BASE|SYSCALL_MASK)" |
+		sed -e "s/#define\s*__NR_//g" -e "s/\s.*//g" |
+		sort -u >${TEMP}/syscall-names.txt
+}
+
+generate_table()
+{
+	echo "- $arch"
+
+	if [ $bits == 32 ]; then
+		extraflags="${extraflags} -D__BITS_PER_LONG=32"
+	fi
+
+	local uppercase_arch=$(echo "$arch" | tr '[:lower:]' '[:upper:]')
+
+	gcc ${TEMP}/list-syscalls.c -U__LP64__ -U__ILP32__ -U__i386__ \
+		-D${uppercase_arch} \
+		-D__${arch}__ ${extraflags} \
+		-I ${LINUX_HEADERS}/usr/include/ \
+		-o ${TEMP}/list-syscalls &>/dev/null
+
+	${TEMP}/list-syscalls > "${TEMP}/${arch}.in.tosort"
+
+	sort -k2,2n "${TEMP}/${arch}.in.tosort" > "${TEMP}/${arch}.in"
+}
+
+generate_list_syscalls_c()
+{
+	(
+		echo
+		echo "
+		#include <stdio.h>
+		#include <asm/unistd.h>
+
+		int main(void)
+		{
+		"
+		for syscall in $(cat ${TEMP}/syscall-names.txt); do
+			echo "
+		#ifdef __NR_$syscall
+			printf(\"$syscall %d\\n\", __NR_$syscall);
+		#endif
+		"
+		done
+		echo " return 0;
+		}"
+	) > ${TEMP}/list-syscalls.c
+}
+
+export_headers()
+{
+	make -s -C ${KERNELSRC} ARCH=${arch} O=${LINUX_HEADERS} \
+		headers_install &>/dev/null
+}
+
+do_all_tables()
+{
+	for archdir in ${KERNELSRC}/arch/*; do
+		arch=$(basename $archdir)
+
+		bits=64
+		extraflags=
+
+		case ${arch} in
+		Kconfig)
+			continue
+			;;
+		um)
+			continue
+			;;
+		esac
+
+		export_headers
+		grab_syscall_names_from_unistd_h
+
+		case ${arch} in
+		arm)
+			bits=32
+			arch=armoabi extraflags= generate_table
+			arch=arm extraflags=-D__ARM_EABI__ generate_table
+			;;
+		loongarch)
+			# 32-bit variant of loongarch may appear
+			arch=loongarch64 extraflags=-D_LOONGARCH_SZLONG=64 generate_table
+			;;
+		mips)
+			arch=mips64 extraflags=-D_MIPS_SIM=_MIPS_SIM_ABI64 generate_table
+			bits=32
+			arch=mipso32 extraflags=-D_MIPS_SIM=_MIPS_SIM_ABI32 generate_table
+			arch=mips64n32 extraflags=-D_MIPS_SIM=_MIPS_SIM_NABI32 generate_table
+			;;
+		powerpc)
+			generate_table
+			arch=powerpc64 generate_table
+			;;
+		riscv)
+			arch=riscv64 extraflags=-D__LP64__ generate_table
+			bits=32
+			arch=riscv32 extraflags=-D__SIZEOF_POINTER__=4 generate_table
+			;;
+		s390)
+			bits=32
+			generate_table
+			bits=64
+			arch=s390x generate_table
+			;;
+		sparc)
+			bits=32
+			extraflags=-D__32bit_syscall_numbers__ generate_table
+			bits=64
+			arch=sparc64 extraflags=-D__arch64__ generate_table
+			;;
+		x86)
+			arch=x86_64 extraflags=-D__LP64__ generate_table
+			bits=32
+			arch=i386 generate_table
+			arch=x32 extraflags=-D__ILP32__ generate_table
+			;;
+		arc | csky | hexagon | m68k | microblaze | nios2 | openrisc | sh | xtensa)
+			bits=32 generate_table
+			;;
+		*)
+			generate_table
+			;;
+		esac
+	done
+}
+
+copy_supported_arch()
+{
+	while IFS= read -r arch; do
+		if [ -f "${TEMP}/${arch}.in" ]; then
+			echo "- ${arch}"
+			cp "${TEMP}/${arch}.in" "${SCRIPT_DIR}/${arch}.in"
+		fi
+	done < ${SUPPORTED_ARCH}
+}
+
+echo "Temporary directory ${TEMP}"
+echo "Extracting syscalls"
+
+grab_syscall_names_from_tables
+generate_list_syscalls_c
+
+do_all_tables
+
+echo "Copying supported syscalls"
+copy_supported_arch
diff --git a/include/lapi/syscalls/loongarch.in b/include/lapi/syscalls/loongarch64.in
similarity index 100%
rename from include/lapi/syscalls/loongarch.in
rename to include/lapi/syscalls/loongarch64.in
diff --git a/include/lapi/syscalls/mips_n64.in b/include/lapi/syscalls/mips64.in
similarity index 100%
rename from include/lapi/syscalls/mips_n64.in
rename to include/lapi/syscalls/mips64.in
diff --git a/include/lapi/syscalls/mips_n32.in b/include/lapi/syscalls/mips64n32.in
similarity index 100%
rename from include/lapi/syscalls/mips_n32.in
rename to include/lapi/syscalls/mips64n32.in
diff --git a/include/lapi/syscalls/mips_o32.in b/include/lapi/syscalls/mipso32.in
similarity index 100%
rename from include/lapi/syscalls/mips_o32.in
rename to include/lapi/syscalls/mipso32.in
diff --git a/include/lapi/syscalls/hppa.in b/include/lapi/syscalls/parisc.in
similarity index 100%
rename from include/lapi/syscalls/hppa.in
rename to include/lapi/syscalls/parisc.in
diff --git a/include/lapi/syscalls/supported-arch.txt b/include/lapi/syscalls/supported-arch.txt
index c18aa38cf4546cdf3ac8c89a45bd1b202ffa7711..c5c5191ac08482d89f0a8c39dfae936538f58f7f 100644
--- a/include/lapi/syscalls/supported-arch.txt
+++ b/include/lapi/syscalls/supported-arch.txt
@@ -1,13 +1,13 @@
-aarch64
 arc
+arm64
 arm
-hppa
 i386
 ia64
-loongarch
-mips_n32
-mips_n64
-mips_o32
+loongarch64
+mips64n32
+mips64
+mipso32
+parisc
 powerpc64
 powerpc
 s390x

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls

[Cyril Hrubis]

Hi!
Uff what a monostrosity, one would have expected that this could be done
easier...

Anyways I do not see anything wrong here, so:

Reviewed-by: Cyril Hrubis <chrubis@suse.cz>

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls

[Petr Vorel]

Hi Andrea,


> From: Andrea Cervesato <andrea.cervesato@suse.com>

> Add generate_arch.sh script which can be used to generate arch(s)
> dependant syscalls file. The way it works is pretty simple: for each
> architecture defined into supported-arch.txt, compile kernel headers,
> extract the list of syscalls and generate a .in file containing all of
> them, associated with their own syscall's number.
> The way syscalls files are generated, passes through a C application
> which is automatically checking the availability of the syscalls in
> the user space environment.

Thanks for this!

...
> --- /dev/null
> +++ b/include/lapi/syscalls/generate_arch.sh
> @@ -0,0 +1,211 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# This is an adaptation of the update-tables.sh script, included in the
> +# syscalls-table project (https://github.com/hrw/syscalls-table) and released
> +# under the MIT license.
I was surprised you use shell instead of python, but I see the original author
also use shell [1]. I wish he would have used python for this part, but good
that you reuse what was available.

Reviewed-by: Petr Vorel <pvorel@suse.cz>

...
> +++ b/include/lapi/syscalls/supported-arch.txt
> @@ -1,13 +1,13 @@
> -aarch64
>  arc
> +arm64
>  arm
> -hppa
>  i386
>  ia64
> -loongarch
> -mips_n32
> -mips_n64
> -mips_o32
> +loongarch64
> +mips64n32
> +mips64
> +mipso32
> +parisc
>  powerpc64
>  powerpc
>  s390x

Interesting enough how many archs the original project supports [2], which makes
sense for it as he products HTML table as reference [3].

Kind regards,
Petr

[1] https://github.com/hrw/syscalls-table/blob/master/scripts/update-tables.sh
[2] https://github.com/hrw/syscalls-table/blob/master/system_calls/architectures_in_kernel.py
[3] https://gpages.juszkiewicz.com.pl/syscalls-table/syscalls.html

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls

[Andrea Cervesato via ltp]

Hi,

On 10/30/24 16:15, Petr Vorel wrote
> ...
>> --- /dev/null
>> +++ b/include/lapi/syscalls/generate_arch.sh
>> @@ -0,0 +1,211 @@
>> +#!/bin/sh
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +#
>> +# This is an adaptation of the update-tables.sh script, included in the
>> +# syscalls-table project (https://github.com/hrw/syscalls-table) and released
>> +# under the MIT license.
> I was surprised you use shell instead of python, but I see the original author
> also use shell [1]. I wish he would have used python for this part, but good
> that you reuse what was available.

One of the option was to use the syscalls-table project in order to 
generate tables, but we needed something out of the box, without complex 
scenarios using `pip`, which might be a bit unfamiliar for some people.

Andrea

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v5 3/3] Delete obsolete strip_syscall.awk file

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

Reviewed-by: Li Wang <liwang@redhat.com>
Reviewed-by: Petr Vorel <pvorel@suse.cz>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 include/lapi/syscalls/strip_syscall.awk | 19 -------------------
 1 file changed, 19 deletions(-)

diff --git a/include/lapi/syscalls/strip_syscall.awk b/include/lapi/syscalls/strip_syscall.awk
deleted file mode 100755
index e8dff422e2667745c144ed984a4d66461fcce0c6..0000000000000000000000000000000000000000
--- a/include/lapi/syscalls/strip_syscall.awk
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/usr/bin/awk -f
-#
-# Dumb script that can be used to strip all of the syscall information from
-# the arch-respective unistd*.h.
-#
-# Examples:
-#
-# 1. Grab the i386 32-bit syscalls from unistd_32.h and put them in i386.in
-# strip_syscall.awk arch/x86/include/asm/unistd_32.h > i386.in
-#
-
-/^#define[[:space:]]+__NR_[0-9a-z]+/ {
-
-	sub (/#define[[:space:]]+__NR_/, "", $0);
-	sub (/[[:space:]]*(\/\*.*)/, "", $0);
-	sub (/[[:space:]]+/, " ", $0);
-
-	print
-}

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 3/3] Delete obsolete strip_syscall.awk file

[Cyril Hrubis]

Hi!
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp