[LTP] [PATCH] oom03: use size_t for memory length to fix 32-bit overflow

[LTP] [PATCH] oom03: use size_t for memory length to fix 32-bit overflow

[Ben Copeland]

The alloc_mem() function is supposed to test out of memory conditions.
How it works is it calls mmap() on a giant chunk of memory.
It's either LENGTH (2GB) or "TESTMEM * 2 + TST_MB" (3GB) bytes.
This mmap() is generally supposed to succeed.  Then at the bottom of
the alloc_mem() function when we actually try to use all the memory,
the thread is supposed to die with a SIGKILL.

The problem is that length is signed so on a 32-bit system it will be
negative. That means that at the bottom of the function when we loop
through the memory, the for loop is a no-op and there is no SIGKILL.
Fix this by changing the type to size_t which is unsigned.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>
---
 testcases/kernel/mem/oom/oom.h | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/testcases/kernel/mem/oom/oom.h b/testcases/kernel/mem/oom/oom.h
index 41cc681f9..42ed181b0 100644
--- a/testcases/kernel/mem/oom/oom.h
+++ b/testcases/kernel/mem/oom/oom.h
@@ -62,13 +62,14 @@ static inline void set_global_mempolicy(int mempolicy)
 static void set_global_mempolicy(int mempolicy LTP_ATTRIBUTE_UNUSED) { }
 #endif
 
-static int alloc_mem(long int length, int testcase)
+static int alloc_mem(size_t length, int testcase)
 {
 	char *s;
-	long i, pagesz = getpagesize();
+	size_t i;
+	long pagesz = getpagesize();
 	int loop = 10;
 
-	tst_res(TINFO, "thread (%lx), allocating %ld bytes.",
+	tst_res(TINFO, "thread (%lx), allocating %zu bytes.",
 		(unsigned long) pthread_self(), length);
 
 	s = mmap(NULL, length, PROT_READ | PROT_WRITE,
@@ -111,7 +112,7 @@ static void child_alloc(int testcase, int lite, int threads)
 	pthread_t *th;
 
 	if (lite) {
-		int ret = alloc_mem(TESTMEM * 2 + TST_MB, testcase);
+		int ret = alloc_mem((size_t)TESTMEM * 2 + TST_MB, testcase);
 		exit(ret);
 	}
 
-- 
2.51.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] oom03: use size_t for memory length to fix 32-bit overflow

[Li Wang via ltp]

On Thu, Oct 16, 2025 at 10:03 PM Ben Copeland <ben.copeland@linaro.org>
wrote:

> The alloc_mem() function is supposed to test out of memory conditions.
> How it works is it calls mmap() on a giant chunk of memory.
> It's either LENGTH (2GB) or "TESTMEM * 2 + TST_MB" (3GB) bytes.
> This mmap() is generally supposed to succeed.  Then at the bottom of
> the alloc_mem() function when we actually try to use all the memory,
> the thread is supposed to die with a SIGKILL.
>
> The problem is that length is signed so on a 32-bit system it will be
> negative. That means that at the bottom of the function when we loop
> through the memory, the for loop is a no-op and there is no SIGKILL.
> Fix this by changing the type to size_t which is unsigned.
>
> Signed-off-by: Ben Copeland <ben.copeland@linaro.org>
>

Reviewed-by: Li Wang <liwang@redhat.com>

---
>  testcases/kernel/mem/oom/oom.h | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/testcases/kernel/mem/oom/oom.h
> b/testcases/kernel/mem/oom/oom.h
> index 41cc681f9..42ed181b0 100644
> --- a/testcases/kernel/mem/oom/oom.h
> +++ b/testcases/kernel/mem/oom/oom.h
> @@ -62,13 +62,14 @@ static inline void set_global_mempolicy(int mempolicy)
>  static void set_global_mempolicy(int mempolicy LTP_ATTRIBUTE_UNUSED) { }
>  #endif
>
> -static int alloc_mem(long int length, int testcase)
> +static int alloc_mem(size_t length, int testcase)
>  {
>         char *s;
> -       long i, pagesz = getpagesize();
> +       size_t i;
> +       long pagesz = getpagesize();
>         int loop = 10;
>
> -       tst_res(TINFO, "thread (%lx), allocating %ld bytes.",
> +       tst_res(TINFO, "thread (%lx), allocating %zu bytes.",
>                 (unsigned long) pthread_self(), length);
>
>         s = mmap(NULL, length, PROT_READ | PROT_WRITE,
> @@ -111,7 +112,7 @@ static void child_alloc(int testcase, int lite, int
> threads)
>         pthread_t *th;
>
>         if (lite) {
> -               int ret = alloc_mem(TESTMEM * 2 + TST_MB, testcase);
> +               int ret = alloc_mem((size_t)TESTMEM * 2 + TST_MB,
> testcase);
>                 exit(ret);
>         }
>
> --
> 2.51.0
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] oom03: use size_t for memory length to fix 32-bit overflow

[Petr Vorel]

Hi all,

> The alloc_mem() function is supposed to test out of memory conditions.
> How it works is it calls mmap() on a giant chunk of memory.
> It's either LENGTH (2GB) or "TESTMEM * 2 + TST_MB" (3GB) bytes.
> This mmap() is generally supposed to succeed.  Then at the bottom of
> the alloc_mem() function when we actually try to use all the memory,
> the thread is supposed to die with a SIGKILL.

> The problem is that length is signed so on a 32-bit system it will be
> negative. That means that at the bottom of the function when we loop
> through the memory, the for loop is a no-op and there is no SIGKILL.
> Fix this by changing the type to size_t which is unsigned.

I dared to change subject to oom.h and merged.
Thanks!

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp