[PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Hajime Tazaki]

The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
expose under !MMU case but used it wit the default vaule
DEFAULT_MAX_MAP_COUNT as a limit of count.  This is currently used when
a vma entry is split into two chunks (split_vma()) but not used when
allocated (do_mmap()).  As a result, even if users request a large
number of allocate memory, it will keep allocating until OOM happens.

This commit introduces a check at the begining of do_mmap in nommu.c to
prevent this situation.

This is detected with a LTP (Linux Test Project) test, which linked
below.

Link:
https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/munmap/munmap04.c
Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
---
 mm/nommu.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/nommu.c b/mm/nommu.c
index 5d461ddc3405..815ebefddfae 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1035,6 +1035,9 @@ unsigned long do_mmap(struct file *file,
 	if (ret < 0)
 		return ret;
 
+	if (current->mm->map_count >= get_sysctl_max_map_count())
+		return -ENOMEM;
+
 	/* we ignore the address hint */
 	addr = 0;
 	len = PAGE_ALIGN(len);
-- 
2.43.0

Re: [PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Andrew Morton]

On Thu,  2 Jul 2026 10:28:30 +0900 Hajime Tazaki <thehajime@gmail.com> wrote:

> The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
> expose under !MMU case but used it wit the default vaule
> DEFAULT_MAX_MAP_COUNT as a limit of count.

This sounds like an oversight.  I see no reason why nommu users cannot
alter max_map_count.

> This is currently used when
> a vma entry is split into two chunks (split_vma()) but not used when
> allocated (do_mmap()).  As a result, even if users request a large
> number of allocate memory, it will keep allocating until OOM happens.
> 
> This commit introduces a check at the begining of do_mmap in nommu.c to
> prevent this situation.
> 
> ...
>
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -1035,6 +1035,9 @@ unsigned long do_mmap(struct file *file,
>  	if (ret < 0)
>  		return ret;
>  
> +	if (current->mm->map_count >= get_sysctl_max_map_count())
> +		return -ENOMEM;
> +
>  	/* we ignore the address hint */
>  	addr = 0;
>  	len = PAGE_ALIGN(len);

Seems sensible, thanks.  That's what the !NOMMU code does.

Re: [PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Andrew Morton]

On Thu,  2 Jul 2026 10:28:30 +0900 Hajime Tazaki <thehajime@gmail.com> wrote:

> The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
> expose under !MMU case but used it wit the default vaule
> DEFAULT_MAX_MAP_COUNT as a limit of count.  This is currently used when
> a vma entry is split into two chunks (split_vma()) but not used when
> allocated (do_mmap()).  As a result, even if users request a large
> number of allocate memory, it will keep allocating until OOM happens.
> 
> This commit introduces a check at the begining of do_mmap in nommu.c to
> prevent this situation.
> 
> This is detected with a LTP (Linux Test Project) test, which linked
> below.

AI review flagged a couple of issues in the current code, one quite
serious:

https://sashiko.dev/#/patchset/20260702012830.667205-1-thehajime@gmail.com

Re: [PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Hajime Tazaki]

On Mon, 06 Jul 2026 07:15:12 +0900,
Andrew Morton wrote:
> 
> On Thu,  2 Jul 2026 10:28:30 +0900 Hajime Tazaki <thehajime@gmail.com> wrote:
> 
> > The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
> > expose under !MMU case but used it wit the default vaule
> > DEFAULT_MAX_MAP_COUNT as a limit of count.  This is currently used when
> > a vma entry is split into two chunks (split_vma()) but not used when
> > allocated (do_mmap()).  As a result, even if users request a large
> > number of allocate memory, it will keep allocating until OOM happens.
> > 
> > This commit introduces a check at the begining of do_mmap in nommu.c to
> > prevent this situation.
> > 
> > This is detected with a LTP (Linux Test Project) test, which linked
> > below.
> 
> AI review flagged a couple of issues in the current code, one quite
> serious:
> 
> https://sashiko.dev/#/patchset/20260702012830.667205-1-thehajime@gmail.com

I will follow up with another patch for the issue detected here.

btw, I found (and sashiko also detected) several typos for this
particular patch.  Can I send v2 patch with the additional fix, or do
you prefer that I'll only send the additional patch ?

anyway, thanks for your time to look at this.

-- Hajime

Re: [PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Andrew Morton]

On Mon, 06 Jul 2026 09:19:02 +0900 Hajime Tazaki <thehajime@gmail.com> wrote:

> 
> On Mon, 06 Jul 2026 07:15:12 +0900,
> Andrew Morton wrote:
> > 
> > On Thu,  2 Jul 2026 10:28:30 +0900 Hajime Tazaki <thehajime@gmail.com> wrote:
> > 
> > > The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
> > > expose under !MMU case but used it wit the default vaule
> > > DEFAULT_MAX_MAP_COUNT as a limit of count.  This is currently used when
> > > a vma entry is split into two chunks (split_vma()) but not used when
> > > allocated (do_mmap()).  As a result, even if users request a large
> > > number of allocate memory, it will keep allocating until OOM happens.
> > > 
> > > This commit introduces a check at the begining of do_mmap in nommu.c to
> > > prevent this situation.
> > > 
> > > This is detected with a LTP (Linux Test Project) test, which linked
> > > below.
> > 
> > AI review flagged a couple of issues in the current code, one quite
> > serious:
> > 
> > https://sashiko.dev/#/patchset/20260702012830.667205-1-thehajime@gmail.com
> 
> I will follow up with another patch for the issue detected here.

Great, thanks.

> btw, I found (and sashiko also detected) several typos for this
> particular patch.  Can I send v2 patch with the additional fix, or do
> you prefer that I'll only send the additional patch ?

Ah.  Well.  Confession.  I actually quietly fed your changelog through
Gemini to fix up a few things, resulting in

: The sysctl variable vm.max_map_count (sysctl_max_map_count) is not exposed
: under !MMU configurations, but its default value (DEFAULT_MAX_MAP_COUNT)
: is still used as a allocation limit.  Currently, this limit is enforced
: when a VMA entry is split into two chunks (split_vma()), but it is not
: checked during initial allocation (do_mmap()).  As a result, if a user
: requests a large number of memory allocations, the system will continue
: allocating until it hits an Out-Of-Memory (OOM) condition.
: 
: This commit introduces a check at the beginning of do_mmap() in nommu.c to
: prevent this situation.
: 
: This issue was detected using the Linux Test Project (LTP) test linked
: below.

I did this "quietly" because it feels a bit rude telling people that
their English isn't great ;)

Lots of non-English speakers appear to be using LLMs on their
changelogs nowadays and as long as they double-check the result, I
think it's working very well!

Re: [PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Hajime Tazaki]

On Mon, 06 Jul 2026 09:44:43 +0900,
Andrew Morton wrote:
> > 
> > I will follow up with another patch for the issue detected here.
> 
> Great, thanks.
> 
> > btw, I found (and sashiko also detected) several typos for this
> > particular patch.  Can I send v2 patch with the additional fix, or do
> > you prefer that I'll only send the additional patch ?
> 
> Ah.  Well.  Confession.  I actually quietly fed your changelog through
> Gemini to fix up a few things, resulting in
> 
> : The sysctl variable vm.max_map_count (sysctl_max_map_count) is not exposed
> : under !MMU configurations, but its default value (DEFAULT_MAX_MAP_COUNT)
> : is still used as a allocation limit.  Currently, this limit is enforced
> : when a VMA entry is split into two chunks (split_vma()), but it is not
> : checked during initial allocation (do_mmap()).  As a result, if a user
> : requests a large number of memory allocations, the system will continue
> : allocating until it hits an Out-Of-Memory (OOM) condition.
> : 
> : This commit introduces a check at the beginning of do_mmap() in nommu.c to
> : prevent this situation.
> : 
> : This issue was detected using the Linux Test Project (LTP) test linked
> : below.

thanks, this is perfect.

> I did this "quietly" because it feels a bit rude telling people that
> their English isn't great ;)
> 
> Lots of non-English speakers appear to be using LLMs on their
> changelogs nowadays and as long as they double-check the result, I
> think it's working very well!

thanks for handling this and a kind way to fix my mistakes.
yes, the AI era.

I will send a follow up patch for the additional issue.

-- Hajime