[RFC PATCH 0/5] Add dead syscalls elimination support

[RFC PATCH 0/5] Add dead syscalls elimination support

[Zhangjin Wu]

Hi, all

CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION allows to eliminate dead code
and data, this patchset allows to further eliminate dead syscalls which
are not used in target system.

It includes 5 patches:

- syscall: Allow configure used system calls

  This adds a new CONFIG_SYSCALLS_USED option to allow users or tools to
  tell kernel what syscalls are used in target system. A list of
  used syscalls can be passed to it.

- MIPS: Add dead syscalls elimination support

  Add CONFIG_SYSCALLS_USED support for mips, it simply adds a 'used'
  variant for the syscall*.tbl and accordingly updates the kernel apis
  and eventually the sys_call_table. The unused ones in the table are
  replaced by sys_ni_syscall and therefore if they are also not used by
  kernel itself, they will be eliminated by gc-sections.

  The old architectures use syscall*.tbl, so, they can use this method.

- RISC-V: Enable dead code elimination

  Select HAVE_LD_DEAD_CODE_DATA_ELIMINATION for riscv.

- RISC-V: Add dead syscalls elimination support

  Add CONFIG_SYSCALLS_USED support for riscv, it simply adds a 'used'
  variant for the *syscall_table.c and eventually the sys_call_table.

  The new architectures use generic unistd.h, they can use this method.

- nolibc: Record used syscalls in their own sections

  This allows to record syscalls used by a nolibc based application. It
  is possible to eliminate dead syscalls automatically while building
  the monolithic kernel+nolibc software.

Testing shows, on both mips and riscv, with a small config, gc-sections
shrinks ~10% and syscalls_used shrinks another ~5%.

This patchset is only a prototype implementation, welcome your feedback
and suggestion, Thanks.

Related emails:

- Re: Re: Kernel-only deployments
  https://lore.kernel.org/lkml/20230216130935.37976-1-falcon@tinylab.org/

- Re: Re: RISC-V: Enable dead code elimination
  https://lore.kernel.org/linux-riscv/Y+qSBu3YZH0JPY4I@spud/T/#t

Best Regards,
- Zhangjin Wu

---

Zhangjin Wu (5):
  syscall: Allow configure used system calls
  MIPS: Add dead syscalls elimination support
  RISC-V: Enable dead code elimination
  RISC-V: Add dead syscalls elimination support
  nolibc: Record used syscalls in their own sections

 arch/mips/Kconfig                   |  1 +
 arch/mips/kernel/syscalls/Makefile  | 24 ++++++++-
 arch/riscv/Kconfig                  |  2 +
 arch/riscv/kernel/Makefile          |  5 +-
 arch/riscv/kernel/syscalls/Makefile | 38 ++++++++++++++
 arch/riscv/kernel/vmlinux.lds.S     |  2 +-
 init/Kconfig                        | 22 +++++++++
 tools/include/nolibc/Makefile       |  2 +-
 tools/include/nolibc/arch-aarch64.h | 17 ++++---
 tools/include/nolibc/arch-arm.h     | 15 +++---
 tools/include/nolibc/arch-i386.h    | 17 ++++---
 tools/include/nolibc/arch-mips.h    | 15 +++---
 tools/include/nolibc/arch-riscv.h   | 17 ++++---
 tools/include/nolibc/arch-x86_64.h  | 17 ++++---
 tools/include/nolibc/arch.h         |  2 +
 tools/include/nolibc/record.h       | 77 +++++++++++++++++++++++++++++
 16 files changed, 226 insertions(+), 47 deletions(-)
 create mode 100644 arch/riscv/kernel/syscalls/Makefile
 create mode 100644 tools/include/nolibc/record.h

-- 
2.25.1

[RFC PATCH 1/5] syscall: Allow configure used system calls

[Zhangjin Wu]

A new CONFIG_SYSCALLS_USED is added to configure the system calls used
in a target system (especially for the ultra small embedded systems),
the other system calls will be disabled.

If further enabling CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION, the other
ones (if also not used by kernel itself) will be removed automatically.

The used system calls should be listed one by one like this:

    write exit reboot

The architecture should implement it and then select the
HAVE_SYSCALLS_USED option.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 init/Kconfig | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index 44e90b28a30f..2c4b8b234168 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1502,6 +1502,10 @@ config BPF
 	bool
 	select CRYPTO_LIB_SHA1
 
+config HAVE_SYSCALLS_USED
+	bool
+	default n
+
 menuconfig EXPERT
 	bool "Configure standard kernel features (expert users)"
 	# Unhide debug options, to make the on-by-default options visible
@@ -1728,6 +1732,24 @@ config MEMBARRIER
 
 	  If unsure, say Y.
 
+config SYSCALLS_USED
+	string "Configure used syscalls (EXPERIMENTAL)"  if EXPERT
+	depends on HAVE_SYSCALLS_USED
+	default ""
+	help
+          This option allows to configure the system calls used in a target
+          system, the other system calls will be disabled.
+
+          If further enabling CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION, the
+          other ones (if also not used by kernel itself) will be removed
+          automatically.
+
+          The used system calls should be listed one by one like this:
+
+              write exit reboot
+
+          If unsure, keep this empty.
+
 config KALLSYMS
 	bool "Load all symbols for debugging/ksymoops" if EXPERT
 	default y
-- 
2.25.1

[RFC PATCH 2/5] MIPS: Add dead syscalls elimination support

[Zhangjin Wu]

By enabling CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION and setting
CONFIG_SYSCALLS_USED, It is able to remove the left 'dead' syscalls.

For example, if setting CONFIG_SYSCALLS_USED="write exit reboot", a
'used' variant of the *.tbl will be generated, accordingly, the kernel
api unistd_nr_*.h and syscall_table_*.h will be generated from the
'used' *tbl variant. the user api version of unistd_*.h is reserved
as-is.

Here is a test result on qemu with a minimal malta config.

                    | mipsel malta    | config
    ----------------|-----------------|-------------------
            vmlinux | 5041628         | https://pastebin.com/0bE2ibLD
      + gc-sections | 4474060 (-11.2%)| CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
    + syscalls_used | 4265280 (-4.67%)| CONFIG_SYSCALLS_USED="_newselect"
    + syscalls_used | 4274364 (-4.46%)| CONFIG_SYSCALLS_USED="write exit reboot"

notes:

- The shrink ratios of the syscalls_used lines are based on the
  gc-sections line.

- "write exit reboot" are used by a hello.c to simply print "Hello,
   World!", exit and shutdown qemu.

- "_newselect" is used by rcutorture to do a long-time sleep.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 arch/mips/Kconfig                  |  1 +
 arch/mips/kernel/syscalls/Makefile | 24 ++++++++++++++++++++++--
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 15cb692b0a09..868d9a871b3e 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -102,6 +102,7 @@ config MIPS
 	select TRACE_IRQFLAGS_SUPPORT
 	select ARCH_HAS_ELFCORE_COMPAT
 	select HAVE_ARCH_KCSAN if 64BIT
+	select HAVE_SYSCALLS_USED
 
 config MIPS_FIXUP_BIGPHYS_ADDR
 	bool
diff --git a/arch/mips/kernel/syscalls/Makefile b/arch/mips/kernel/syscalls/Makefile
index e6b21de65cca..8ffba5301cf0 100644
--- a/arch/mips/kernel/syscalls/Makefile
+++ b/arch/mips/kernel/syscalls/Makefile
@@ -26,10 +26,30 @@ sysnr_pfx_unistd_nr_n32 := N32
 sysnr_pfx_unistd_nr_n64 := 64
 sysnr_pfx_unistd_nr_o32 := O32
 
-$(kapi)/unistd_nr_%.h: $(src)/syscall_%.tbl $(sysnr) FORCE
+ifdef CONFIG_SYSCALLS_USED
+syscalls_used := $(shell echo $(CONFIG_SYSCALLS_USED) | tr -s ' ' | tr ' ' '|')
+endif
+
+ifneq ($(syscalls_used),)
+utbl := arch/$(SRCARCH)/include/generated/tbl
+_tbl := $(src)/syscall_%.tbl
+ tbl := $(utbl)/syscall_used_%.tbl
+
+$(shell mkdir -p $(utbl))
+
+quiet_cmd_used = USED    $@
+      cmd_used = sed -E -e "/^[0-9]*[[:space:]]/{/(^($(syscalls_used))[[:space:]]|[[:space:]]($(syscalls_used))[[:space:]]|[[:space:]]($(syscalls_used))$$)/!{s/^/\#/g}}" $< > $@;
+
+$(tbl): $(_tbl) $(objtree)/.config
+	$(call cmd,used)
+else
+tbl := $(src)/syscall_%.tbl
+endif
+
+$(kapi)/unistd_nr_%.h: $(tbl) $(sysnr) FORCE
 	$(call if_changed,sysnr)
 
-$(kapi)/syscall_table_%.h: $(src)/syscall_%.tbl $(systbl) FORCE
+$(kapi)/syscall_table_%.h: $(tbl) $(systbl) FORCE
 	$(call if_changed,systbl)
 
 uapisyshdr-y		+= unistd_n32.h			\
-- 
2.25.1

[RFC PATCH 3/5] RISC-V: Enable dead code elimination

[Zhangjin Wu]

Select CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION for RISC-V, allowing
the user to enable dead code elimination. In order for this to work,
ensure that we keep the alternative table by annotating them with KEEP.

This boots well on qemu with both rv32_defconfig & rv64 defconfig, but
it only shrinks their builds by ~1%, a smaller config is thereforce
customized to test this feature:

          | rv32                   | rv64
  --------|------------------------|---------------------
   No DCE | 4460684                | 4893488
      DCE | 3986716                | 4376400
   Shrink |  473968 (~10.6%)       |  517088 (~10.5%)

The config used above only reserves necessary options to boot on qemu
with serial console, more like the size-critical embedded scenes:

  - rv64 config: https://pastebin.com/crz82T0s
  - rv32 config: rv64 config + 32-bit.config

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 arch/riscv/Kconfig              | 1 +
 arch/riscv/kernel/vmlinux.lds.S | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index e2b656043abf..8a73d7180cb8 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -103,6 +103,7 @@ config RISCV
 	select HAVE_KPROBES_ON_FTRACE if !XIP_KERNEL
 	select HAVE_KRETPROBES if !XIP_KERNEL
 	select HAVE_RETHOOK if !XIP_KERNEL
+	select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
 	select HAVE_MOVE_PMD
 	select HAVE_MOVE_PUD
 	select HAVE_PCI
diff --git a/arch/riscv/kernel/vmlinux.lds.S b/arch/riscv/kernel/vmlinux.lds.S
index 4e6c88aa4d87..51218cfe1ee4 100644
--- a/arch/riscv/kernel/vmlinux.lds.S
+++ b/arch/riscv/kernel/vmlinux.lds.S
@@ -101,7 +101,7 @@ SECTIONS
 	. = ALIGN(8);
 	.alternative : {
 		__alt_start = .;
-		*(.alternative)
+		KEEP(*(.alternative))
 		__alt_end = .;
 	}
 	__init_end = .;
-- 
2.25.1

[RFC PATCH 4/5] RISC-V: Add dead syscalls elimination support

[Zhangjin Wu]

By enabling CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION and setting
CONFIG_SYSCALLS_USED, It is able to remove the left 'dead' syscalls.

For example, if setting CONFIG_SYSCALLS_USED="write exit reboot", a
'used' variant of the *syscall_table.c will be generated.

Here is a test result on qemu with a minimal rv64 config.

                    | rv64            | config
    ----------------|-----------------|-------------------
            vmlinux | 4893488         | https://pastebin.com/crz82T0s
      + gc-sections | 4376400 (-10.5%)| CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
    + syscalls_used | 4172112 (-4.67%)| CONFIG_SYSCALLS_USED="pselect6"
    + syscalls_used | 4172848 (-4.65%)| CONFIG_SYSCALLS_USED="write exit reboot"

notes:

- The shrink ratios of the syscalls_used lines are based on the
  gc-sections line.

- "write exit reboot" are used by a hello.c to simply print "Hello,
   World!", exit and shutdown qemu.

- "pselect6" is used by rcutorture to do a long-time sleep.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 arch/riscv/Kconfig                  |  1 +
 arch/riscv/kernel/Makefile          |  5 ++--
 arch/riscv/kernel/syscalls/Makefile | 46 +++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/kernel/syscalls/Makefile

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 8a73d7180cb8..f78cc6b2413f 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -140,6 +140,7 @@ config RISCV
 	select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_TRACER if !XIP_KERNEL
+	select HAVE_SYSCALLS_USED
 
 config ARCH_MMAP_RND_BITS_MIN
 	default 18 if 64BIT
diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
index 4cf303a779ab..fd716d5ffda5 100644
--- a/arch/riscv/kernel/Makefile
+++ b/arch/riscv/kernel/Makefile
@@ -8,7 +8,8 @@ CFLAGS_REMOVE_ftrace.o	= $(CC_FLAGS_FTRACE)
 CFLAGS_REMOVE_patch.o	= $(CC_FLAGS_FTRACE)
 CFLAGS_REMOVE_sbi.o	= $(CC_FLAGS_FTRACE)
 endif
-CFLAGS_syscall_table.o	+= $(call cc-option,-Wno-override-init,)
+
+obj-y += syscalls/
 
 ifdef CONFIG_KEXEC
 AFLAGS_kexec_relocate.o := -mcmodel=medany $(call cc-option,-mno-relax)
@@ -42,7 +43,6 @@ obj-y	+= ptrace.o
 obj-y	+= reset.o
 obj-y	+= setup.o
 obj-y	+= signal.o
-obj-y	+= syscall_table.o
 obj-y	+= sys_riscv.o
 obj-y	+= time.o
 obj-y	+= traps.o
@@ -86,6 +86,5 @@ obj-$(CONFIG_CRASH_CORE)	+= crash_core.o
 obj-$(CONFIG_JUMP_LABEL)	+= jump_label.o
 
 obj-$(CONFIG_EFI)		+= efi.o
-obj-$(CONFIG_COMPAT)		+= compat_syscall_table.o
 obj-$(CONFIG_COMPAT)		+= compat_signal.o
 obj-$(CONFIG_COMPAT)		+= compat_vdso/
diff --git a/arch/riscv/kernel/syscalls/Makefile b/arch/riscv/kernel/syscalls/Makefile
new file mode 100644
index 000000000000..7bd327331a98
--- /dev/null
+++ b/arch/riscv/kernel/syscalls/Makefile
@@ -0,0 +1,46 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Copyright (C) 2023 Zhangjin Wu <falcon@tinylab.org>
+#
+
+CFLAGS_syscall_table_used.o	+= $(call cc-option,-Wno-override-init,)
+obj-y				+= syscall_table_used.o
+obj-$(CONFIG_COMPAT)		+= compat_syscall_table_used.o
+
+ifdef CONFIG_SYSCALLS_USED
+syscalls_used := $(shell echo $(CONFIG_SYSCALLS_USED) | tr -s ' ' | tr ' ' '|')
+endif
+
+ifneq ($(syscalls_used),)
+
+quiet_cmd_calc = CALC    $@
+      cmd_calc = sed -n -e '/^\[([0-9 +]*)\] = /{s/.*\[\(.*\)\] = .*/sed \\"s%^\\\\[\1\\\\] = %[$$((\1))] = %g\\" -i /gp}' $@ | xargs -I{} echo {} $@ | sh;
+
+quiet_cmd_used = USED    $@
+      cmd_used = sed -E -e '/^\[[0-9]*\] = /{/(^\[($(syscalls_used))\] *=|= *\((sys_)*($(syscalls_used))\),)/!{s%^%// %g}}' -i $@;
+
+$(obj)/syscall_table.i: $(src)/../syscall_table.c $(objtree)/.config FORCE
+	$(call if_changed_dep,cpp_i_c)
+	$(call cmd,calc)
+	$(call cmd,used)
+
+$(obj)/syscall_table_used.o: $(obj)/syscall_table.i FORCE
+	$(call if_changed,cc_o_c)
+
+$(obj)/compat_syscall_table.i: $(src)/../compat_syscall_table.c $(objtree)/.config FORCE
+	$(call if_changed_dep,cpp_i_c)
+	$(call cmd,calc)
+	$(call cmd,used)
+
+$(obj)/compat_syscall_table_used.o: $(obj)/compat_syscall_table.i FORCE
+	$(call if_changed,cc_o_c)
+
+else
+
+$(obj)/syscall_table_used.o: $(src)/../syscall_table.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+$(obj)/compat_syscall_table_used.o: $(src)/../compat_syscall_table.c FORCE
+	$(call if_changed_rule,cc_o_c)
+
+endif
-- 
2.25.1

[RFC PATCH 5/5] nolibc: Record used syscalls in their own sections

[Zhangjin Wu]

A new section is added for every system call, the section is encoded
with the syscall name and syscall number. for example:

    .rodata.syscall.__NR_exit.4001
    .rodata.syscall.__NR_getpid.(4000+20)
    .rodata.syscall.__NR_kill.(4000+37)
    .rodata.syscall.__NR_write.(4000+4)
    .rodata.syscall.__NR_reboot.(4000+88)

Both such sections and the unused syscalls can be printed by the
-Wl,--print-gc-sections option of ld (with -ffunction-sections
-fdata-sections and -Wl,--gc-sections together):

    removing unused section '.text.sys_getpid' in file '/tmp/ccbRltF4.o'
    removing unused section '.text.sys_kill' in file '/tmp/ccbRltF4.o'
    removing unused section '.rodata.syscall.__NR_exit.4001' in file '/tmp/cc0vNiof.o'
    removing unused section '.rodata.syscall.__NR_getpid.(4000+20)' in file '/tmp/ccbRltF4.o'
    removing unused section '.rodata.syscall.__NR_kill.(4000+37)' in file '/tmp/ccbRltF4.o'
    removing unused section '.rodata.syscall.__NR_write.(4000+4)' in file '/tmp/ccbRltF4.o'
    removing unused section '.rodata.syscall.__NR_reboot.(4000+88)' in file '/tmp/ccbRltF4.o'

To get the used syscalls, we can use:

    the group of '.rodata.syscall.*' - the group of '.text.sys_*'

At last, we get:

    __NR_exit.4001
    __NR_write.(4000+4)
    __NR_reboot.(4000+88)

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 tools/include/nolibc/Makefile       |  2 +-
 tools/include/nolibc/arch-aarch64.h | 17 ++++---
 tools/include/nolibc/arch-arm.h     | 15 +++---
 tools/include/nolibc/arch-i386.h    | 17 ++++---
 tools/include/nolibc/arch-mips.h    | 15 +++---
 tools/include/nolibc/arch-riscv.h   | 17 ++++---
 tools/include/nolibc/arch-x86_64.h  | 17 ++++---
 tools/include/nolibc/arch.h         |  2 +
 tools/include/nolibc/record.h       | 77 +++++++++++++++++++++++++++++
 9 files changed, 138 insertions(+), 41 deletions(-)
 create mode 100644 tools/include/nolibc/record.h

diff --git a/tools/include/nolibc/Makefile b/tools/include/nolibc/Makefile
index cfd06764b5ae..f06cbb24124e 100644
--- a/tools/include/nolibc/Makefile
+++ b/tools/include/nolibc/Makefile
@@ -26,7 +26,7 @@ endif
 nolibc_arch := $(patsubst arm64,aarch64,$(ARCH))
 arch_file := arch-$(nolibc_arch).h
 all_files := ctype.h errno.h nolibc.h signal.h std.h stdio.h stdlib.h string.h \
-             sys.h time.h types.h unistd.h
+             sys.h time.h types.h unistd.h record.h
 
 # install all headers needed to support a bare-metal compiler
 all: headers
diff --git a/tools/include/nolibc/arch-aarch64.h b/tools/include/nolibc/arch-aarch64.h
index f68baf8f395f..9f7f5f662cd3 100644
--- a/tools/include/nolibc/arch-aarch64.h
+++ b/tools/include/nolibc/arch-aarch64.h
@@ -62,7 +62,7 @@ struct sys_stat_struct {
  */
 #define __ARCH_WANT_SYS_PSELECT6
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0");                                   \
@@ -76,7 +76,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -91,7 +91,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -107,7 +107,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -124,7 +124,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -142,7 +142,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -161,7 +161,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                  \
+#define _my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                 \
 ({                                                                            \
 	register long _num  __asm__ ("x8") = (num);                           \
 	register long _arg1 __asm__ ("x0") = (long)(arg1);                    \
@@ -181,6 +181,8 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
+#include "record.h"
+
 /* startup code */
 __asm__ (".section .text\n"
     ".weak _start\n"
@@ -194,6 +196,7 @@ __asm__ (".section .text\n"
     "bl main\n"                   // main() returns the status code, we'll exit with it.
     "mov x8, 93\n"                // NR_exit == 93
     "svc #0\n"
+    asm_record_exit(93)
     "");
 
 #endif // _NOLIBC_ARCH_AARCH64_H
diff --git a/tools/include/nolibc/arch-arm.h b/tools/include/nolibc/arch-arm.h
index f31be8e967d6..7e9f190b8d48 100644
--- a/tools/include/nolibc/arch-arm.h
+++ b/tools/include/nolibc/arch-arm.h
@@ -75,7 +75,7 @@ struct sys_stat_struct {
  */
 #define __ARCH_WANT_SYS_OLD_SELECT
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0");                                   \
@@ -89,7 +89,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0") = (long)(arg1);                    \
@@ -104,7 +104,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0") = (long)(arg1);                    \
@@ -120,7 +120,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0") = (long)(arg1);                    \
@@ -137,7 +137,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0") = (long)(arg1);                    \
@@ -155,7 +155,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	register long _num __asm__ ("r7") = (num);                            \
 	register long _arg1 __asm__ ("r0") = (long)(arg1);                    \
@@ -174,6 +174,8 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
+#include "record.h"
+
 /* startup code */
 __asm__ (".section .text\n"
     ".weak _start\n"
@@ -199,6 +201,7 @@ __asm__ (".section .text\n"
     "bl main\n"                   // main() returns the status code, we'll exit with it.
     "movs r7, $1\n"               // NR_exit == 1
     "svc $0x00\n"
+    asm_record_exit(1)
     "");
 
 #endif // _NOLIBC_ARCH_ARM_H
diff --git a/tools/include/nolibc/arch-i386.h b/tools/include/nolibc/arch-i386.h
index d7e7212346e2..de8ea6b32cc4 100644
--- a/tools/include/nolibc/arch-i386.h
+++ b/tools/include/nolibc/arch-i386.h
@@ -63,7 +63,7 @@ struct sys_stat_struct {
  */
 #define __ARCH_WANT_SYS_OLD_SELECT
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -77,7 +77,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -93,7 +93,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -110,7 +110,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -128,7 +128,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -147,7 +147,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num __asm__ ("eax") = (num);                           \
@@ -167,7 +167,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)	\
+#define _my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)	\
 ({								\
 	long _eax  = (long)(num);				\
 	long _arg6 = (long)(arg6); /* Always in memory */	\
@@ -190,6 +190,8 @@ struct sys_stat_struct {
 	_eax;							\
 })
 
+#include "record.h"
+
 /* startup code */
 /*
  * i386 System V ABI mandates:
@@ -214,6 +216,7 @@ __asm__ (".section .text\n"
     "movl $1, %eax\n"           // NR_exit == 1
     "int $0x80\n"               // exit now
     "hlt\n"                     // ensure it does not
+    asm_record_exit(1)
     "");
 
 #endif // _NOLIBC_ARCH_I386_H
diff --git a/tools/include/nolibc/arch-mips.h b/tools/include/nolibc/arch-mips.h
index 7380093ba9e7..719eef58b187 100644
--- a/tools/include/nolibc/arch-mips.h
+++ b/tools/include/nolibc/arch-mips.h
@@ -67,7 +67,7 @@ struct sys_stat_struct {
  *     don't have to experience issues with register constraints.
  */
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	register long _num __asm__ ("v0") = (num);                            \
 	register long _arg4 __asm__ ("a3");                                   \
@@ -84,7 +84,7 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	register long _num __asm__ ("v0") = (num);                            \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -103,7 +103,7 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	register long _num __asm__ ("v0") = (num);                            \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -123,7 +123,7 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	register long _num __asm__ ("v0")  = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -144,7 +144,7 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	register long _num __asm__ ("v0") = (num);                            \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -165,7 +165,7 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	register long _num __asm__ ("v0") = (num);                            \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -188,6 +188,8 @@ struct sys_stat_struct {
 	_arg4 ? -_num : _num;                                                 \
 })
 
+#include "record.h"
+
 /* startup code, note that it's called __start on MIPS */
 __asm__ (".section .text\n"
     ".weak __start\n"
@@ -212,6 +214,7 @@ __asm__ (".section .text\n"
     "syscall\n"
     ".end __start\n"
     ".set pop\n"
+    asm_record_exit(4001)
     "");
 
 #endif // _NOLIBC_ARCH_MIPS_H
diff --git a/tools/include/nolibc/arch-riscv.h b/tools/include/nolibc/arch-riscv.h
index a3bdd9803f8c..700d02860ecd 100644
--- a/tools/include/nolibc/arch-riscv.h
+++ b/tools/include/nolibc/arch-riscv.h
@@ -64,7 +64,7 @@ struct sys_stat_struct {
  */
 #define __ARCH_WANT_SYS_PSELECT6
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0");                                   \
@@ -78,7 +78,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);		      \
@@ -92,7 +92,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -108,7 +108,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -125,7 +125,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -143,7 +143,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -162,7 +162,7 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
-#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                  \
+#define _my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                 \
 ({                                                                            \
 	register long _num  __asm__ ("a7") = (num);                           \
 	register long _arg1 __asm__ ("a0") = (long)(arg1);                    \
@@ -182,6 +182,8 @@ struct sys_stat_struct {
 	_arg1;                                                                \
 })
 
+#include "record.h"
+
 /* startup code */
 __asm__ (".section .text\n"
     ".weak _start\n"
@@ -199,6 +201,7 @@ __asm__ (".section .text\n"
     "call  main\n"               // main() returns the status code, we'll exit with it.
     "li a7, 93\n"                // NR_exit == 93
     "ecall\n"
+    asm_record_exit(93)
     "");
 
 #endif // _NOLIBC_ARCH_RISCV_H
diff --git a/tools/include/nolibc/arch-x86_64.h b/tools/include/nolibc/arch-x86_64.h
index 0e1e9eb8545d..5628b27a6f25 100644
--- a/tools/include/nolibc/arch-x86_64.h
+++ b/tools/include/nolibc/arch-x86_64.h
@@ -65,7 +65,7 @@ struct sys_stat_struct {
  *
  */
 
-#define my_syscall0(num)                                                      \
+#define _my_syscall0(num)                                                     \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -79,7 +79,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall1(num, arg1)                                                \
+#define _my_syscall1(num, arg1)                                               \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -95,7 +95,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall2(num, arg1, arg2)                                          \
+#define _my_syscall2(num, arg1, arg2)                                         \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -112,7 +112,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall3(num, arg1, arg2, arg3)                                    \
+#define _my_syscall3(num, arg1, arg2, arg3)                                   \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -130,7 +130,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall4(num, arg1, arg2, arg3, arg4)                              \
+#define _my_syscall4(num, arg1, arg2, arg3, arg4)                             \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -149,7 +149,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                        \
+#define _my_syscall5(num, arg1, arg2, arg3, arg4, arg5)                       \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -169,7 +169,7 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
-#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                  \
+#define _my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)                 \
 ({                                                                            \
 	long _ret;                                                            \
 	register long _num  __asm__ ("rax") = (num);                          \
@@ -190,6 +190,8 @@ struct sys_stat_struct {
 	_ret;                                                                 \
 })
 
+#include "record.h"
+
 /* startup code */
 /*
  * x86-64 System V ABI mandates:
@@ -210,6 +212,7 @@ __asm__ (".section .text\n"
     "mov $60, %eax\n"           // NR_exit == 60
     "syscall\n"                 // really exit
     "hlt\n"                     // ensure it does not return
+    asm_record_exit(60)
     "");
 
 #endif // _NOLIBC_ARCH_X86_64_H
diff --git a/tools/include/nolibc/arch.h b/tools/include/nolibc/arch.h
index 4c6992321b0d..e0552934eb77 100644
--- a/tools/include/nolibc/arch.h
+++ b/tools/include/nolibc/arch.h
@@ -29,4 +29,6 @@
 #include "arch-riscv.h"
 #endif
 
+#include "record.h"
+
 #endif /* _NOLIBC_ARCH_H */
diff --git a/tools/include/nolibc/record.h b/tools/include/nolibc/record.h
new file mode 100644
index 000000000000..95e02092cecf
--- /dev/null
+++ b/tools/include/nolibc/record.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * Copyright (C) 2023 Zhangjin Wu <falcon@tinylab.org>
+ */
+
+#ifndef _NOLIBC_RECORD_H
+#define _NOLIBC_RECORD_H
+
+/* To record syscalls used, please pass -DRECORD_SYSCALL to gcc */
+#ifdef RECORD_SYSCALL
+#define __asm_record_syscall(name, val) 				\
+	".pushsection .rodata.syscall." name "." #val ",\"a\"\n"	\
+	".word (" #val ")\n"     		              		\
+	".popsection\n"
+
+#define asm_record_syscall(num) __asm_record_syscall(#num, num)
+#define asm_record_exit(num) __asm_record_syscall("__NR_exit", num)
+
+#define record_syscall(name, val)	 				\
+({									\
+	__asm__ volatile (						\
+		__asm_record_syscall(name, val)				\
+	); 								\
+})
+
+#else /* RECORD_SYSCALL */
+#define asm_record_exit(num) ""
+#define record_syscall(name, val) do { } while (0)
+#endif /* !RECORD_SYSCALL */
+
+#define my_syscall0(num)						\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall0(num);						\
+})
+
+#define my_syscall1(num, arg1)						\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall1(num, arg1);					\
+})
+
+#define my_syscall2(num, arg1, arg2)					\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall2(num, arg1, arg2);					\
+})
+
+#define my_syscall3(num, arg1, arg2, arg3)				\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall3(num, arg1, arg2, arg3);				\
+})
+
+#define my_syscall4(num, arg1, arg2, arg3, arg4)			\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall4(num, arg1, arg2, arg3, arg4);			\
+})
+
+#ifdef _my_syscall5
+#define my_syscall5(num, arg1, arg2, arg3, arg4, arg5)			\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall5(num, arg1, arg2, arg3, arg4, arg5);		\
+})
+#endif
+
+#ifdef _my_syscall6
+#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6)		\
+({									\
+	record_syscall(#num, num);					\
+	_my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6);		\
+})
+#endif
+
+#endif /* _NOLIBC_RECORD_H */
-- 
2.25.1