[PATCH 0/4] tools/nolibc: pass mode to open syscall if O_TMPFILE is set

[PATCH 0/4] tools/nolibc: pass mode to open syscall if O_TMPFILE is set

[Thomas Weißschuh]

When O_TMPFILE is set, the open mode needs to be passed to the kernel as
per the documentation.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
Thomas Weißschuh (4):
      tools/nolibc: split implicit open flags into a macro
      tools/nolibc: split open mode handling into a macro
      tools/nolibc: pass mode to open syscall if O_TMPFILE is set
      selftests/nolibc: test open mode handling

 tools/include/nolibc/fcntl.h                 | 46 ++++++++++++----------------
 tools/testing/selftests/nolibc/nolibc-test.c | 23 ++++++++++++++
 2 files changed, 43 insertions(+), 26 deletions(-)
---
base-commit: 266df86c4893fed1a7f027e767fe1c7f6456b100
change-id: 20260429-nolibc-open-tmpfile-19e677e33c8f

Best regards,
--  
Thomas Weißschuh <linux@weissschuh.net>

[PATCH 1/4] tools/nolibc: split implicit open flags into a macro

[Thomas Weißschuh]

This logic is duplicated and its current form will be in the way of some
upcoming simplificiations.

Move it into a macro to avoid the duplication and enable some cleanups.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/include/nolibc/fcntl.h | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/tools/include/nolibc/fcntl.h b/tools/include/nolibc/fcntl.h
index 014910a8e928..46f591cf82fd 100644
--- a/tools/include/nolibc/fcntl.h
+++ b/tools/include/nolibc/fcntl.h
@@ -14,6 +14,8 @@
 #include "types.h"
 #include "sys.h"
 
+#define __nolibc_open_flags(_flags) ((_flags) | O_LARGEFILE)
+
 /*
  * int openat(int dirfd, const char *path, int flags[, mode_t mode]);
  */
@@ -29,8 +31,6 @@ int openat(int dirfd, const char *path, int flags, ...)
 {
 	mode_t mode = 0;
 
-	flags |= O_LARGEFILE;
-
 	if (flags & O_CREAT) {
 		va_list args;
 
@@ -39,7 +39,7 @@ int openat(int dirfd, const char *path, int flags, ...)
 		va_end(args);
 	}
 
-	return __sysret(_sys_openat(dirfd, path, flags, mode));
+	return __sysret(_sys_openat(dirfd, path, __nolibc_open_flags(flags), mode));
 }
 
 /*
@@ -57,8 +57,6 @@ int open(const char *path, int flags, ...)
 {
 	mode_t mode = 0;
 
-	flags |= O_LARGEFILE;
-
 	if (flags & O_CREAT) {
 		va_list args;
 
@@ -67,7 +65,7 @@ int open(const char *path, int flags, ...)
 		va_end(args);
 	}
 
-	return __sysret(_sys_open(path, flags, mode));
+	return __sysret(_sys_open(path, __nolibc_open_flags(flags), mode));
 }
 
 /*

-- 
2.54.0

[PATCH 2/4] tools/nolibc: split open mode handling into a macro

[Thomas Weißschuh]

This logic is duplicated and some upcoming extensions would require even
more duplicated logic.

Move it into a macro to avoid the duplication and allow cleaner changes.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/include/nolibc/fcntl.h | 40 ++++++++++++++++++----------------------
 1 file changed, 18 insertions(+), 22 deletions(-)

diff --git a/tools/include/nolibc/fcntl.h b/tools/include/nolibc/fcntl.h
index 46f591cf82fd..d7ea02e1332d 100644
--- a/tools/include/nolibc/fcntl.h
+++ b/tools/include/nolibc/fcntl.h
@@ -16,6 +16,21 @@
 
 #define __nolibc_open_flags(_flags) ((_flags) | O_LARGEFILE)
 
+#define __nolibc_open_mode(_flags)							\
+({											\
+	mode_t _mode = 0;								\
+											\
+	if ((_flags) & O_CREAT) {							\
+		va_list args;								\
+											\
+		va_start(args, (_flags));						\
+		_mode = va_arg(args, mode_t);						\
+		va_end(args);								\
+	}										\
+											\
+	_mode;										\
+})
+
 /*
  * int openat(int dirfd, const char *path, int flags[, mode_t mode]);
  */
@@ -29,17 +44,8 @@ int _sys_openat(int dirfd, const char *path, int flags, mode_t mode)
 static __attribute__((unused))
 int openat(int dirfd, const char *path, int flags, ...)
 {
-	mode_t mode = 0;
-
-	if (flags & O_CREAT) {
-		va_list args;
-
-		va_start(args, flags);
-		mode = va_arg(args, mode_t);
-		va_end(args);
-	}
-
-	return __sysret(_sys_openat(dirfd, path, __nolibc_open_flags(flags), mode));
+	return __sysret(_sys_openat(dirfd, path, __nolibc_open_flags(flags),
+						 __nolibc_open_mode(flags)));
 }
 
 /*
@@ -55,17 +61,7 @@ int _sys_open(const char *path, int flags, mode_t mode)
 static __attribute__((unused))
 int open(const char *path, int flags, ...)
 {
-	mode_t mode = 0;
-
-	if (flags & O_CREAT) {
-		va_list args;
-
-		va_start(args, flags);
-		mode = va_arg(args, mode_t);
-		va_end(args);
-	}
-
-	return __sysret(_sys_open(path, __nolibc_open_flags(flags), mode));
+	return __sysret(_sys_open(path, __nolibc_open_flags(flags), __nolibc_open_mode(flags)));
 }
 
 /*

-- 
2.54.0

[PATCH 3/4] tools/nolibc: pass mode to open syscall if O_TMPFILE is set

[Thomas Weißschuh]

When O_TMPFILE is set, the open mode needs to be passed to the kernel as
per the documentation.

Fixes: a7604ba149e7 ("tools/nolibc/sys: make open() take a vararg on the 3rd argument")
Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/include/nolibc/fcntl.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/include/nolibc/fcntl.h b/tools/include/nolibc/fcntl.h
index d7ea02e1332d..c401419e12b1 100644
--- a/tools/include/nolibc/fcntl.h
+++ b/tools/include/nolibc/fcntl.h
@@ -20,7 +20,7 @@
 ({											\
 	mode_t _mode = 0;								\
 											\
-	if ((_flags) & O_CREAT) {							\
+	if (((_flags) & O_CREAT) || (((_flags) & O_TMPFILE) == O_TMPFILE)) {		\
 		va_list args;								\
 											\
 		va_start(args, (_flags));						\

-- 
2.54.0

[PATCH 4/4] selftests/nolibc: test open mode handling

[Thomas Weißschuh]

Add a selftest for the new O_TMPFILE open mode handling.
While O_CREAT or openat() are not tested, the code is the same,
so assume these also work.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
 tools/testing/selftests/nolibc/nolibc-test.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index 08610cacf030..6a03781fffab 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -1300,6 +1300,28 @@ int test_openat(void)
 	return 0;
 }
 
+int test_open_mode(void)
+{
+	const mode_t mode = 0444;
+	struct stat stat_buf;
+	int fd, ret;
+
+	fd = open("/tmp", O_TMPFILE | O_RDWR, mode);
+	if (fd == -1)
+		return -1;
+
+	ret = fstat(fd, &stat_buf);
+	close(fd);
+
+	if (ret == -1)
+		return -1;
+
+	if ((stat_buf.st_mode & 0777) != mode)
+		return -1;
+
+	return 0;
+}
+
 int test_nolibc_enosys(void)
 {
 	if (true)
@@ -1535,6 +1557,7 @@ int run_syscall(int min, int max)
 		CASE_TEST(open_tty);          EXPECT_SYSNE(1, tmp = open("/dev/null", O_RDONLY), -1); if (tmp != -1) close(tmp); break;
 		CASE_TEST(open_blah);         EXPECT_SYSER(1, tmp = open("/proc/self/blah", O_RDONLY), -1, ENOENT); if (tmp != -1) close(tmp); break;
 		CASE_TEST(openat_dir);        EXPECT_SYSZR(1, test_openat()); break;
+		CASE_TEST(open_mode);         EXPECT_SYSZR(1, test_open_mode()); break;
 		CASE_TEST(pipe);              EXPECT_SYSZR(1, test_pipe()); break;
 		CASE_TEST(poll_null);         EXPECT_SYSZR(1, poll(NULL, 0, 0)); break;
 		CASE_TEST(poll_stdout);       EXPECT_SYSNE(1, ({ struct pollfd fds = { 1, POLLOUT, 0}; poll(&fds, 1, 0); }), -1); break;

-- 
2.54.0

Re: [PATCH 0/4] tools/nolibc: pass mode to open syscall if O_TMPFILE is set

[Willy Tarreau]

Hi Thomas,

On Wed, Apr 29, 2026 at 04:45:08PM +0200, Thomas Weißschuh wrote:
> When O_TMPFILE is set, the open mode needs to be passed to the kernel as
> per the documentation.
> 
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> ---
> Thomas Weißschuh (4):
>       tools/nolibc: split implicit open flags into a macro
>       tools/nolibc: split open mode handling into a macro
>       tools/nolibc: pass mode to open syscall if O_TMPFILE is set
>       selftests/nolibc: test open mode handling
> 
>  tools/include/nolibc/fcntl.h                 | 46 ++++++++++++----------------
>  tools/testing/selftests/nolibc/nolibc-test.c | 23 ++++++++++++++
>  2 files changed, 43 insertions(+), 26 deletions(-)

This is a clever approach, and it can definitely work.

Acked-by: Willy Tarreau <w@1wt.eu>

I'm wondering why we're looking at the mode only for certain flags
though. After all, the kernel is supposed to look at it when it needs.
Why wouldn't we just always pass the va_args and let the syscall refer
to it when it needs ? If it's a register, it's implicitly passed anyway,
and if it's in the stack, it's only one entry, it's the same as if someone
calls open() with one of these flags without passing the argument. Any
garbage it contains should be ignored by the kernel when not needed, so
I don't see why it would be a problem.

Cheers,
Willy

Re: [PATCH 0/4] tools/nolibc: pass mode to open syscall if O_TMPFILE is set

[Thomas Weißschuh]

On 2026-05-01 10:08:45+0200, Willy Tarreau wrote:
> Hi Thomas,
> 
> On Wed, Apr 29, 2026 at 04:45:08PM +0200, Thomas Weißschuh wrote:
> > When O_TMPFILE is set, the open mode needs to be passed to the kernel as
> > per the documentation.
> > 
> > Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> > ---
> > Thomas Weißschuh (4):
> >       tools/nolibc: split implicit open flags into a macro
> >       tools/nolibc: split open mode handling into a macro
> >       tools/nolibc: pass mode to open syscall if O_TMPFILE is set
> >       selftests/nolibc: test open mode handling
> > 
> >  tools/include/nolibc/fcntl.h                 | 46 ++++++++++++----------------
> >  tools/testing/selftests/nolibc/nolibc-test.c | 23 ++++++++++++++
> >  2 files changed, 43 insertions(+), 26 deletions(-)
> 
> This is a clever approach, and it can definitely work.
> 
> Acked-by: Willy Tarreau <w@1wt.eu>

Thanks!

> I'm wondering why we're looking at the mode only for certain flags
> though. After all, the kernel is supposed to look at it when it needs.
> Why wouldn't we just always pass the va_args and let the syscall refer
> to it when it needs ? If it's a register, it's implicitly passed anyway,
> and if it's in the stack, it's only one entry, it's the same as if someone
> calls open() with one of these flags without passing the argument. Any
> garbage it contains should be ignored by the kernel when not needed, so
> I don't see why it would be a problem.

I think there are some architectures with trap representations for
register values. There that aproach could fail. However these should be
irrelevant for nolibc. I'll try your suggestion for the next revision.


Thomas