[PATCH] selftests/core: fix unshare_test with large fs.nr_open

[PATCH] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko]

The test assumes fs.nr_open is close to the default 1048576, but some
systems set it much higher (e.g. 1073741816). This is systemd's doing:
since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
their largest possible values on boot, as file descriptors are already
accounted for by memcg [1].

In that case, dup2() to nr_open + 64 requires the kernel to allocate a
file descriptor table with ~1 billion entries, which fails with ENOMEM.

Cap the nr_open value used for the test's own arithmetic to a known
reasonable base value (1048576) and restore the true original value
once the test has completed.

[1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
    ("main: bump fs.nr_open + fs.max-file to their largest possible values")

Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
Signed-off-by: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
---
 tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/core/unshare_test.c b/tools/testing/selftests/core/unshare_test.c
index ffce75a6c228..d40e963dd520 100644
--- a/tools/testing/selftests/core/unshare_test.c
+++ b/tools/testing/selftests/core/unshare_test.c
@@ -40,6 +40,14 @@ TEST(unshare_EMFILE)
 
 	ASSERT_EQ(sscanf(buf, "%d", &nr_open), 1);
 
+	/*
+	 * Cap nr_open for the duration of the test to avoid ENOMEM from a
+	 * huge fd table allocation; buf/n keep the real original value so
+	 * fs.nr_open can be restored to it once the test is done.
+	 */
+	if (nr_open > 1024 * 1024)
+		nr_open = 1024 * 1024;
+
 	ASSERT_EQ(0, getrlimit(RLIMIT_NOFILE, &rlimit));
 
 	/* bump fs.nr_open */
@@ -73,10 +81,13 @@ TEST(unshare_EMFILE)
 
 	if (pid == 0) {
 		int err;
+		char buf3[32];
+		ssize_t n3;
 
-		/* restore fs.nr_open */
+		/* restore fs.nr_open to the (possibly capped) test baseline */
+		n3 = sprintf(buf3, "%d\n", nr_open);
 		lseek(fd, 0, SEEK_SET);
-		write(fd, buf, n);
+		write(fd, buf3, n3);
 		/* ... and now unshare(CLONE_FILES) must fail with EMFILE */
 		err = unshare(CLONE_FILES);
 		EXPECT_EQ(err, -1)
@@ -89,6 +100,10 @@ TEST(unshare_EMFILE)
 	EXPECT_EQ(waitpid(pid, &status, 0), pid);
 	EXPECT_EQ(true, WIFEXITED(status));
 	EXPECT_EQ(0, WEXITSTATUS(status));
+
+	/* restore the real fs.nr_open value */
+	lseek(fd, 0, SEEK_SET);
+	write(fd, buf, n);
 }
 
 TEST_HARNESS_MAIN
-- 
2.47.1

Re: [PATCH] selftests/core: fix unshare_test with large fs.nr_open

[Andrew Morton]

On Fri, 14 Aug 2026 18:57:09 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:

> The test assumes fs.nr_open is close to the default 1048576, but some
> systems set it much higher (e.g. 1073741816). This is systemd's doing:
> since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
> their largest possible values on boot, as file descriptors are already
> accounted for by memcg [1].
> 
> In that case, dup2() to nr_open + 64 requires the kernel to allocate a
> file descriptor table with ~1 billion entries, which fails with ENOMEM.

2018.  Can you suggest why we (or I) haven't heard about this?

> Cap the nr_open value used for the test's own arithmetic to a known
> reasonable base value (1048576) and restore the true original value
> once the test has completed.
> 
> [1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
>     ("main: bump fs.nr_open + fs.max-file to their largest possible values")

Well, we do want selftests to run well on the kernel with which they
are shipped, so I'm thinking we should backport this into -stable
kernels.  We can probably skip the Fixes:, but a cc:stable should be
added to capture this.

Thoughts?

Re: [PATCH] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko]

On 8/30/26 03:26, Andrew Morton wrote:
> On Fri, 14 Aug 2026 18:57:09 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
> 
>> The test assumes fs.nr_open is close to the default 1048576, but some
>> systems set it much higher (e.g. 1073741816). This is systemd's doing:
>> since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
>> their largest possible values on boot, as file descriptors are already
>> accounted for by memcg [1].
>>
>> In that case, dup2() to nr_open + 64 requires the kernel to allocate a
>> file descriptor table with ~1 billion entries, which fails with ENOMEM.
> 
> 2018.  Can you suggest why we (or I) haven't heard about this?

The 2018 date only says when systemd started doing this, not when it began
to matter. Laid out as a timeline:

  Oct 2018   systemd a8b627aaed409 ("main: bump fs.nr_open + fs.max-file
             to their largest possible values"), released in v240. From
             here on PID 1 raises fs.nr_open to 1073741816 on boot.
             Harmless in itself: nothing was asking for an fd table
             anywhere near that size.

  Aug 2024   611fbeb44a777 ("selftests:core: test coverage for dup_fd()
             failure handling in unshare_fd()"), v6.12-rc1. The test is
             added, and it is the first thing in tree to deliberately
             dup2() to a descriptor just past fs.nr_open. This is the
             point where the two meet, so the exposure window is about
             two years, not eight.

  Jun 2025   04a2c4b4511d1 ("fs: Prevent file descriptor table
             allocations exceeding INT_MAX"), v6.17-rc1, cc:stable.
             Somebody did hit it and did fix it - but on the kernel
             side. Its changelog explicitly names
             tools/testing/selftests/core/unshare_test.c as a way to
             reach the WARNING.

So the report exists and the kernel half is fixed; what nobody did was go
back and fix the test that provoked it, which is what this patch is for.

As for why it is not seen more widely, distributions differ on whether the
bump happens at all. The systemd option is 'bump-proc-sys-fs-nr-open',
declared as a boolean with no explicit value in meson_options.txt, so it
defaults to true.
i know for sure Ubuntu passes -Dbump-proc-sys-fs-nr-open=false in rules,
so on Ubuntu nr_open stays at the kernel default of 1048576 and the test
passes.

On the other hand i tried Almalinux 10 (fs.nr_open = 1073741816) + latest
mainstream kernel and the testcase still fails there.

One correction to my own changelog while we are here: on a kernel that
already carries 04a2c4b4511d1, dup2() no longer fails with ENOMEM. The
allocation is now rejected up front and the caller gets EMFILE instead,
without the WARNING, but the test still fails.


>> Cap the nr_open value used for the test's own arithmetic to a known
>> reasonable base value (1048576) and restore the true original value
>> once the test has completed.
>>
>> [1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
>>     ("main: bump fs.nr_open + fs.max-file to their largest possible values")
> 
> Well, we do want selftests to run well on the kernel with which they
> are shipped, so I'm thinking we should backport this into -stable
> kernels.  We can probably skip the Fixes:, but a cc:stable should be
> added to capture this.
> 
> Thoughts?