[PATCH v2 0/2] New selftest for mm

[PATCH v2 0/2] New selftest for mm

[Breno Leitao]

This is a simple test case that reproduces an mm problem[1], where a page
fault races with madvise(), and it is not trivial to reproduce and
debug.

This test-case aims to avoid such race problems from happening again,
impacting workloads that leverages external allocators, such as
tcmalloc, jemalloc, etc.

Changelog:
---------

V2:
 * Remove some unnecessary checks
 * Skip the test instead of failing in some cases

[1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r

Breno Leitao (2):
  selftests/mm: export get_free_hugepages()
  selftests/mm: Add a new test for madv and hugetlb

 tools/testing/selftests/mm/Makefile           |  1 +
 tools/testing/selftests/mm/hugetlb-madvise.c  | 19 -----
 .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |  4 +
 tools/testing/selftests/mm/vm_util.c          | 19 +++++
 tools/testing/selftests/mm/vm_util.h          |  1 +
 6 files changed, 98 insertions(+), 19 deletions(-)
 create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c

-- 
2.34.1

[PATCH v2 1/2] selftests/mm: export get_free_hugepages()

[Breno Leitao]

get_free_hugepages() is helpful for other hugepage tests. Export it to
the common file (vm_util.c) to be reused.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 tools/testing/selftests/mm/hugetlb-madvise.c | 19 -------------------
 tools/testing/selftests/mm/vm_util.c         | 19 +++++++++++++++++++
 tools/testing/selftests/mm/vm_util.h         |  1 +
 3 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/tools/testing/selftests/mm/hugetlb-madvise.c b/tools/testing/selftests/mm/hugetlb-madvise.c
index d55322df4b73..f32d99565c5e 100644
--- a/tools/testing/selftests/mm/hugetlb-madvise.c
+++ b/tools/testing/selftests/mm/hugetlb-madvise.c
@@ -36,25 +36,6 @@
 unsigned long huge_page_size;
 unsigned long base_page_size;
 
-unsigned long get_free_hugepages(void)
-{
-	unsigned long fhp = 0;
-	char *line = NULL;
-	size_t linelen = 0;
-	FILE *f = fopen("/proc/meminfo", "r");
-
-	if (!f)
-		return fhp;
-	while (getline(&line, &linelen, f) > 0) {
-		if (sscanf(line, "HugePages_Free:      %lu", &fhp) == 1)
-			break;
-	}
-
-	free(line);
-	fclose(f);
-	return fhp;
-}
-
 void write_fault_pages(void *addr, unsigned long nr_pages)
 {
 	unsigned long i;
diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c
index 558c9cd8901c..3082b40492dd 100644
--- a/tools/testing/selftests/mm/vm_util.c
+++ b/tools/testing/selftests/mm/vm_util.c
@@ -269,3 +269,22 @@ int uffd_unregister(int uffd, void *addr, uint64_t len)
 
 	return ret;
 }
+
+unsigned long get_free_hugepages(void)
+{
+	unsigned long fhp = 0;
+	char *line = NULL;
+	size_t linelen = 0;
+	FILE *f = fopen("/proc/meminfo", "r");
+
+	if (!f)
+		return fhp;
+	while (getline(&line, &linelen, f) > 0) {
+		if (sscanf(line, "HugePages_Free:      %lu", &fhp) == 1)
+			break;
+	}
+
+	free(line);
+	fclose(f);
+	return fhp;
+}
diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h
index c7fa61f0dff8..c02990bbd56f 100644
--- a/tools/testing/selftests/mm/vm_util.h
+++ b/tools/testing/selftests/mm/vm_util.h
@@ -51,6 +51,7 @@ int uffd_register(int uffd, void *addr, uint64_t len,
 int uffd_unregister(int uffd, void *addr, uint64_t len);
 int uffd_register_with_ioctls(int uffd, void *addr, uint64_t len,
 			      bool miss, bool wp, bool minor, uint64_t *ioctls);
+unsigned long get_free_hugepages(void);
 
 /*
  * On ppc64 this will only work with radix 2M hugepage size
-- 
2.34.1

[PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Breno Leitao]

Create a selftest that exercises the race between page faults and
madvise(MADV_DONTNEED) in the same huge page. Do it by running two
threads that touches the huge page and madvise(MADV_DONTNEED) at the same
time.

In case of a SIGBUS coming at pagefault, the test should fail, since we
hit the bug.

The test doesn't have a signal handler, and if it fails, it fails like
the following

  ----------------------------------
  running ./hugetlb_fault_after_madv
  ----------------------------------
  ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
  [FAIL]

This selftest goes together with the fix of the bug[1] itself.

[1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 tools/testing/selftests/mm/Makefile           |  1 +
 .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |  4 +
 3 files changed, 78 insertions(+)
 create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c

diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index 6a9fc5693145..e71ec9910c62 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -68,6 +68,7 @@ TEST_GEN_FILES += split_huge_page_test
 TEST_GEN_FILES += ksm_tests
 TEST_GEN_FILES += ksm_functional_tests
 TEST_GEN_FILES += mdwe_test
+TEST_GEN_FILES += hugetlb_fault_after_madv
 
 ifneq ($(ARCH),arm64)
 TEST_GEN_PROGS += soft-dirty
diff --git a/tools/testing/selftests/mm/hugetlb_fault_after_madv.c b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
new file mode 100644
index 000000000000..73b81c632366
--- /dev/null
+++ b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "vm_util.h"
+#include "../kselftest.h"
+
+#define MMAP_SIZE (1 << 21)
+#define INLOOP_ITER 100
+
+char *huge_ptr;
+
+/* Touch the memory while it is being madvised() */
+void *touch(void *unused)
+{
+	char *ptr = (char *)huge_ptr;
+
+	for (int i = 0; i < INLOOP_ITER; i++)
+		ptr[0] = '.';
+
+	return NULL;
+}
+
+void *madv(void *unused)
+{
+	usleep(rand() % 10);
+
+	for (int i = 0; i < INLOOP_ITER; i++)
+		madvise(huge_ptr, MMAP_SIZE, MADV_DONTNEED);
+
+	return NULL;
+}
+
+int main(void)
+{
+	unsigned long free_hugepages;
+	pthread_t thread1, thread2;
+	/*
+	 * On kernel 6.4, we are able to reproduce the problem with ~1000
+	 * interactions
+	 */
+	int max = 10000;
+
+	srand(getpid());
+
+	free_hugepages = get_free_hugepages();
+	if (free_hugepages != 1) {
+		ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
+			       free_hugepages);
+	}
+
+	while (max--) {
+		huge_ptr = mmap(NULL, MMAP_SIZE, PROT_READ | PROT_WRITE,
+				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
+				-1, 0);
+
+		if ((unsigned long)huge_ptr == -1)
+			ksft_exit_skip("Failed to allocated huge page\n");
+
+		pthread_create(&thread1, NULL, madv, NULL);
+		pthread_create(&thread2, NULL, touch, NULL);
+
+		pthread_join(thread1, NULL);
+		pthread_join(thread2, NULL);
+		munmap(huge_ptr, MMAP_SIZE);
+	}
+
+	return KSFT_PASS;
+}
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 3e2bc818d566..9f53f7318a38 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -221,6 +221,10 @@ CATEGORY="hugetlb" run_test ./hugepage-mremap
 CATEGORY="hugetlb" run_test ./hugepage-vmemmap
 CATEGORY="hugetlb" run_test ./hugetlb-madvise
 
+# For this test, we need one and just one huge page
+echo 1 > /proc/sys/vm/nr_hugepages
+CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
+
 if test_selected "hugetlb"; then
 	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
 	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
-- 
2.34.1

Re: [PATCH v2 1/2] selftests/mm: export get_free_hugepages()

[Rik van Riel]

On Thu, 2023-10-05 at 09:39 -0700, Breno Leitao wrote:
> get_free_hugepages() is helpful for other hugepage tests. Export it
> to
> the common file (vm_util.c) to be reused.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> 
Reviewed-by: Rik van Riel <riel@surriel.com>

-- 
All Rights Reversed.

Re: [PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Rik van Riel]

On Thu, 2023-10-05 at 09:39 -0700, Breno Leitao wrote:
> Create a selftest that exercises the race between page faults and
> madvise(MADV_DONTNEED) in the same huge page. Do it by running two
> threads that touches the huge page and madvise(MADV_DONTNEED) at the
> same
> time.
> 

>   ----------------------------------
>   ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
>   [FAIL]
> 
> This selftest goes together with the fix of the bug[1] itself.
> 
> [1]
> https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
Tested-and-reviewed-by: Rik van Riel <riel@surriel.com>

-- 
All Rights Reversed.

Re: [PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Ryan Roberts]

Hi Breno,


On 05/10/2023 17:39, Breno Leitao wrote:
> Create a selftest that exercises the race between page faults and
> madvise(MADV_DONTNEED) in the same huge page. Do it by running two
> threads that touches the huge page and madvise(MADV_DONTNEED) at the same
> time.
> 
> In case of a SIGBUS coming at pagefault, the test should fail, since we
> hit the bug.
> 
> The test doesn't have a signal handler, and if it fails, it fails like
> the following
> 
>   ----------------------------------
>   running ./hugetlb_fault_after_madv
>   ----------------------------------
>   ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
>   [FAIL]
> 
> This selftest goes together with the fix of the bug[1] itself.
> 
> [1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
>  tools/testing/selftests/mm/Makefile           |  1 +
>  .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
>  tools/testing/selftests/mm/run_vmtests.sh     |  4 +
>  3 files changed, 78 insertions(+)
>  create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> 
> diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
> index 6a9fc5693145..e71ec9910c62 100644
> --- a/tools/testing/selftests/mm/Makefile
> +++ b/tools/testing/selftests/mm/Makefile
> @@ -68,6 +68,7 @@ TEST_GEN_FILES += split_huge_page_test
>  TEST_GEN_FILES += ksm_tests
>  TEST_GEN_FILES += ksm_functional_tests
>  TEST_GEN_FILES += mdwe_test
> +TEST_GEN_FILES += hugetlb_fault_after_madv
>  
>  ifneq ($(ARCH),arm64)
>  TEST_GEN_PROGS += soft-dirty
> diff --git a/tools/testing/selftests/mm/hugetlb_fault_after_madv.c b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> new file mode 100644
> index 000000000000..73b81c632366
> --- /dev/null
> +++ b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> @@ -0,0 +1,73 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <pthread.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <sys/mman.h>
> +#include <sys/types.h>
> +#include <unistd.h>
> +
> +#include "vm_util.h"
> +#include "../kselftest.h"
> +
> +#define MMAP_SIZE (1 << 21)
> +#define INLOOP_ITER 100
> +
> +char *huge_ptr;
> +
> +/* Touch the memory while it is being madvised() */
> +void *touch(void *unused)
> +{
> +	char *ptr = (char *)huge_ptr;
> +
> +	for (int i = 0; i < INLOOP_ITER; i++)
> +		ptr[0] = '.';
> +
> +	return NULL;
> +}
> +
> +void *madv(void *unused)
> +{
> +	usleep(rand() % 10);
> +
> +	for (int i = 0; i < INLOOP_ITER; i++)
> +		madvise(huge_ptr, MMAP_SIZE, MADV_DONTNEED);
> +
> +	return NULL;
> +}
> +
> +int main(void)
> +{
> +	unsigned long free_hugepages;
> +	pthread_t thread1, thread2;
> +	/*
> +	 * On kernel 6.4, we are able to reproduce the problem with ~1000
> +	 * interactions
> +	 */
> +	int max = 10000;
> +
> +	srand(getpid());
> +
> +	free_hugepages = get_free_hugepages();
> +	if (free_hugepages != 1) {
> +		ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
> +			       free_hugepages);
> +	}
> +
> +	while (max--) {
> +		huge_ptr = mmap(NULL, MMAP_SIZE, PROT_READ | PROT_WRITE,
> +				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
> +				-1, 0);
> +
> +		if ((unsigned long)huge_ptr == -1)
> +			ksft_exit_skip("Failed to allocated huge page\n");
> +
> +		pthread_create(&thread1, NULL, madv, NULL);
> +		pthread_create(&thread2, NULL, touch, NULL);
> +
> +		pthread_join(thread1, NULL);
> +		pthread_join(thread2, NULL);
> +		munmap(huge_ptr, MMAP_SIZE);
> +	}
> +
> +	return KSFT_PASS;
> +}
> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
> index 3e2bc818d566..9f53f7318a38 100755
> --- a/tools/testing/selftests/mm/run_vmtests.sh
> +++ b/tools/testing/selftests/mm/run_vmtests.sh
> @@ -221,6 +221,10 @@ CATEGORY="hugetlb" run_test ./hugepage-mremap
>  CATEGORY="hugetlb" run_test ./hugepage-vmemmap
>  CATEGORY="hugetlb" run_test ./hugetlb-madvise
>  
> +# For this test, we need one and just one huge page
> +echo 1 > /proc/sys/vm/nr_hugepages

I've noticed that this change breaks some of the uffd-stress tests further down
the file, because you have freed previously reserved hugepages that the test
requires to run. I notice that the patch is already in mm-stable, so perhaps its
possible to submit a patch that does a save and restore?

Although I'm not sure if that might be tricky because the previous reservation
is per-size and per-node (our CI does this on the kernel command line), and I
suspect if you want just 1 huge page in the entire system you won't be able to
get back to the previous state by just restoring this value?

These are the failing tests for reference:

# ------------------------------------
# running ./uffd-stress hugetlb 128 32
# ------------------------------------
# nr_pages: 64, nr_pages_per_cpu: 8
# ERROR: context init failed (errno=12, @uffd-stress.c:254)
# [FAIL]
# --------------------------------------------
# running ./uffd-stress hugetlb-private 128 32
# --------------------------------------------
# nr_pages: 64, nr_pages_per_cpu: 8
# ERROR: context init failed (errno=12, @uffd-stress.c:254)
# [FAIL]

Thanks,
Ryan



> +CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
> +
>  if test_selected "hugetlb"; then
>  	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
>  	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"

Re: [PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Ryan Roberts]

On 02/11/2023 12:24, Ryan Roberts wrote:
> Hi Breno,
> 
> 
> On 05/10/2023 17:39, Breno Leitao wrote:
>> Create a selftest that exercises the race between page faults and
>> madvise(MADV_DONTNEED) in the same huge page. Do it by running two
>> threads that touches the huge page and madvise(MADV_DONTNEED) at the same
>> time.
>>
>> In case of a SIGBUS coming at pagefault, the test should fail, since we
>> hit the bug.
>>
>> The test doesn't have a signal handler, and if it fails, it fails like
>> the following
>>
>>   ----------------------------------
>>   running ./hugetlb_fault_after_madv
>>   ----------------------------------
>>   ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
>>   [FAIL]
>>
>> This selftest goes together with the fix of the bug[1] itself.
>>
>> [1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r
>>
>> Signed-off-by: Breno Leitao <leitao@debian.org>
>> ---
>>  tools/testing/selftests/mm/Makefile           |  1 +
>>  .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
>>  tools/testing/selftests/mm/run_vmtests.sh     |  4 +
>>  3 files changed, 78 insertions(+)
>>  create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>>
>> diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
>> index 6a9fc5693145..e71ec9910c62 100644
>> --- a/tools/testing/selftests/mm/Makefile
>> +++ b/tools/testing/selftests/mm/Makefile
>> @@ -68,6 +68,7 @@ TEST_GEN_FILES += split_huge_page_test
>>  TEST_GEN_FILES += ksm_tests
>>  TEST_GEN_FILES += ksm_functional_tests
>>  TEST_GEN_FILES += mdwe_test
>> +TEST_GEN_FILES += hugetlb_fault_after_madv
>>  
>>  ifneq ($(ARCH),arm64)
>>  TEST_GEN_PROGS += soft-dirty
>> diff --git a/tools/testing/selftests/mm/hugetlb_fault_after_madv.c b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>> new file mode 100644
>> index 000000000000..73b81c632366
>> --- /dev/null
>> +++ b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>> @@ -0,0 +1,73 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +#include <pthread.h>
>> +#include <stdio.h>
>> +#include <stdlib.h>
>> +#include <sys/mman.h>
>> +#include <sys/types.h>
>> +#include <unistd.h>
>> +
>> +#include "vm_util.h"
>> +#include "../kselftest.h"
>> +
>> +#define MMAP_SIZE (1 << 21)
>> +#define INLOOP_ITER 100
>> +
>> +char *huge_ptr;
>> +
>> +/* Touch the memory while it is being madvised() */
>> +void *touch(void *unused)
>> +{
>> +	char *ptr = (char *)huge_ptr;
>> +
>> +	for (int i = 0; i < INLOOP_ITER; i++)
>> +		ptr[0] = '.';
>> +
>> +	return NULL;
>> +}
>> +
>> +void *madv(void *unused)
>> +{
>> +	usleep(rand() % 10);
>> +
>> +	for (int i = 0; i < INLOOP_ITER; i++)
>> +		madvise(huge_ptr, MMAP_SIZE, MADV_DONTNEED);
>> +
>> +	return NULL;
>> +}
>> +
>> +int main(void)
>> +{
>> +	unsigned long free_hugepages;
>> +	pthread_t thread1, thread2;
>> +	/*
>> +	 * On kernel 6.4, we are able to reproduce the problem with ~1000
>> +	 * interactions
>> +	 */
>> +	int max = 10000;
>> +
>> +	srand(getpid());
>> +
>> +	free_hugepages = get_free_hugepages();
>> +	if (free_hugepages != 1) {
>> +		ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
>> +			       free_hugepages);
>> +	}
>> +
>> +	while (max--) {
>> +		huge_ptr = mmap(NULL, MMAP_SIZE, PROT_READ | PROT_WRITE,
>> +				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
>> +				-1, 0);
>> +
>> +		if ((unsigned long)huge_ptr == -1)
>> +			ksft_exit_skip("Failed to allocated huge page\n");
>> +
>> +		pthread_create(&thread1, NULL, madv, NULL);
>> +		pthread_create(&thread2, NULL, touch, NULL);
>> +
>> +		pthread_join(thread1, NULL);
>> +		pthread_join(thread2, NULL);
>> +		munmap(huge_ptr, MMAP_SIZE);
>> +	}
>> +
>> +	return KSFT_PASS;
>> +}
>> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
>> index 3e2bc818d566..9f53f7318a38 100755
>> --- a/tools/testing/selftests/mm/run_vmtests.sh
>> +++ b/tools/testing/selftests/mm/run_vmtests.sh
>> @@ -221,6 +221,10 @@ CATEGORY="hugetlb" run_test ./hugepage-mremap
>>  CATEGORY="hugetlb" run_test ./hugepage-vmemmap
>>  CATEGORY="hugetlb" run_test ./hugetlb-madvise
>>  
>> +# For this test, we need one and just one huge page
>> +echo 1 > /proc/sys/vm/nr_hugepages
> 
> I've noticed that this change breaks some of the uffd-stress tests further down
> the file, because you have freed previously reserved hugepages that the test
> requires to run. I notice that the patch is already in mm-stable, so perhaps its
> possible to submit a patch that does a save and restore?
> 
> Although I'm not sure if that might be tricky because the previous reservation
> is per-size and per-node (our CI does this on the kernel command line), and I
> suspect if you want just 1 huge page in the entire system you won't be able to
> get back to the previous state by just restoring this value?

Actually on closer inspection, I don't think this will be a problem; simply
saving and restoring the value around the test will be sufficient.

I also notice that the binary for the new test is not added to the .gitignore,
which is a minor annoyance.

Thanks,
Ryan

> 
> These are the failing tests for reference:
> 
> # ------------------------------------
> # running ./uffd-stress hugetlb 128 32
> # ------------------------------------
> # nr_pages: 64, nr_pages_per_cpu: 8
> # ERROR: context init failed (errno=12, @uffd-stress.c:254)
> # [FAIL]
> # --------------------------------------------
> # running ./uffd-stress hugetlb-private 128 32
> # --------------------------------------------
> # nr_pages: 64, nr_pages_per_cpu: 8
> # ERROR: context init failed (errno=12, @uffd-stress.c:254)
> # [FAIL]
> 
> Thanks,
> Ryan
> 
> 
> 
>> +CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
>> +
>>  if test_selected "hugetlb"; then
>>  	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
>>  	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
> 

Re: [PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Breno Leitao]

Hello Ryan,

On Thu, Nov 02, 2023 at 12:29:54PM +0000, Ryan Roberts wrote:
> On 02/11/2023 12:24, Ryan Roberts wrote:
> > On 05/10/2023 17:39, Breno Leitao wrote:
> >> Create a selftest that exercises the race between page faults and
> >> madvise(MADV_DONTNEED) in the same huge page. Do it by running two
> >> threads that touches the huge page and madvise(MADV_DONTNEED) at the same
> >> time.
> >>
> >> In case of a SIGBUS coming at pagefault, the test should fail, since we
> >> hit the bug.
> >>
> >> The test doesn't have a signal handler, and if it fails, it fails like
> >> the following
> >>
> >>   ----------------------------------
> >>   running ./hugetlb_fault_after_madv
> >>   ----------------------------------
> >>   ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
> >>   [FAIL]
> >>
> >> This selftest goes together with the fix of the bug[1] itself.
> >>
> >> [1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r
> >>
> >> Signed-off-by: Breno Leitao <leitao@debian.org>
> >> ---
> >>  tools/testing/selftests/mm/Makefile           |  1 +
> >>  .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
> >>  tools/testing/selftests/mm/run_vmtests.sh     |  4 +
> >>  3 files changed, 78 insertions(+)
> >>  create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> >>
> >> diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
> >> index 6a9fc5693145..e71ec9910c62 100644
> >> --- a/tools/testing/selftests/mm/Makefile
> >> +++ b/tools/testing/selftests/mm/Makefile
> >> @@ -68,6 +68,7 @@ TEST_GEN_FILES += split_huge_page_test
> >>  TEST_GEN_FILES += ksm_tests
> >>  TEST_GEN_FILES += ksm_functional_tests
> >>  TEST_GEN_FILES += mdwe_test
> >> +TEST_GEN_FILES += hugetlb_fault_after_madv
> >>  
> >>  ifneq ($(ARCH),arm64)
> >>  TEST_GEN_PROGS += soft-dirty
> >> diff --git a/tools/testing/selftests/mm/hugetlb_fault_after_madv.c b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> >> new file mode 100644
> >> index 000000000000..73b81c632366
> >> --- /dev/null
> >> +++ b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
> >> @@ -0,0 +1,73 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +#include <pthread.h>
> >> +#include <stdio.h>
> >> +#include <stdlib.h>
> >> +#include <sys/mman.h>
> >> +#include <sys/types.h>
> >> +#include <unistd.h>
> >> +
> >> +#include "vm_util.h"
> >> +#include "../kselftest.h"
> >> +
> >> +#define MMAP_SIZE (1 << 21)
> >> +#define INLOOP_ITER 100
> >> +
> >> +char *huge_ptr;
> >> +
> >> +/* Touch the memory while it is being madvised() */
> >> +void *touch(void *unused)
> >> +{
> >> +	char *ptr = (char *)huge_ptr;
> >> +
> >> +	for (int i = 0; i < INLOOP_ITER; i++)
> >> +		ptr[0] = '.';
> >> +
> >> +	return NULL;
> >> +}
> >> +
> >> +void *madv(void *unused)
> >> +{
> >> +	usleep(rand() % 10);
> >> +
> >> +	for (int i = 0; i < INLOOP_ITER; i++)
> >> +		madvise(huge_ptr, MMAP_SIZE, MADV_DONTNEED);
> >> +
> >> +	return NULL;
> >> +}
> >> +
> >> +int main(void)
> >> +{
> >> +	unsigned long free_hugepages;
> >> +	pthread_t thread1, thread2;
> >> +	/*
> >> +	 * On kernel 6.4, we are able to reproduce the problem with ~1000
> >> +	 * interactions
> >> +	 */
> >> +	int max = 10000;
> >> +
> >> +	srand(getpid());
> >> +
> >> +	free_hugepages = get_free_hugepages();
> >> +	if (free_hugepages != 1) {
> >> +		ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
> >> +			       free_hugepages);
> >> +	}
> >> +
> >> +	while (max--) {
> >> +		huge_ptr = mmap(NULL, MMAP_SIZE, PROT_READ | PROT_WRITE,
> >> +				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
> >> +				-1, 0);
> >> +
> >> +		if ((unsigned long)huge_ptr == -1)
> >> +			ksft_exit_skip("Failed to allocated huge page\n");
> >> +
> >> +		pthread_create(&thread1, NULL, madv, NULL);
> >> +		pthread_create(&thread2, NULL, touch, NULL);
> >> +
> >> +		pthread_join(thread1, NULL);
> >> +		pthread_join(thread2, NULL);
> >> +		munmap(huge_ptr, MMAP_SIZE);
> >> +	}
> >> +
> >> +	return KSFT_PASS;
> >> +}
> >> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
> >> index 3e2bc818d566..9f53f7318a38 100755
> >> --- a/tools/testing/selftests/mm/run_vmtests.sh
> >> +++ b/tools/testing/selftests/mm/run_vmtests.sh
> >> @@ -221,6 +221,10 @@ CATEGORY="hugetlb" run_test ./hugepage-mremap
> >>  CATEGORY="hugetlb" run_test ./hugepage-vmemmap
> >>  CATEGORY="hugetlb" run_test ./hugetlb-madvise
> >>  
> >> +# For this test, we need one and just one huge page
> >> +echo 1 > /proc/sys/vm/nr_hugepages
> > 
> > I've noticed that this change breaks some of the uffd-stress tests further down
> > the file, because you have freed previously reserved hugepages that the test
> > requires to run. I notice that the patch is already in mm-stable, so perhaps its
> > possible to submit a patch that does a save and restore?
> > 
> > Although I'm not sure if that might be tricky because the previous reservation
> > is per-size and per-node (our CI does this on the kernel command line), and I
> > suspect if you want just 1 huge page in the entire system you won't be able to
> > get back to the previous state by just restoring this value?
> 
> Actually on closer inspection, I don't think this will be a problem; simply
> saving and restoring the value around the test will be sufficient.

Thanks for checking it, I will prepare a patch that will restore the number
of huge pages allocated after the test.

> I also notice that the binary for the new test is not added to the .gitignore,
> which is a minor annoyance.

I will add the file to .gitignore also.

Thanks for the heads-up.

Re: [PATCH v2 2/2] selftests/mm: Add a new test for madv and hugetlb

[Ryan Roberts]

On 03/11/2023 13:59, Breno Leitao wrote:
> Hello Ryan,
> 
> On Thu, Nov 02, 2023 at 12:29:54PM +0000, Ryan Roberts wrote:
>> On 02/11/2023 12:24, Ryan Roberts wrote:
>>> On 05/10/2023 17:39, Breno Leitao wrote:
>>>> Create a selftest that exercises the race between page faults and
>>>> madvise(MADV_DONTNEED) in the same huge page. Do it by running two
>>>> threads that touches the huge page and madvise(MADV_DONTNEED) at the same
>>>> time.
>>>>
>>>> In case of a SIGBUS coming at pagefault, the test should fail, since we
>>>> hit the bug.
>>>>
>>>> The test doesn't have a signal handler, and if it fails, it fails like
>>>> the following
>>>>
>>>>   ----------------------------------
>>>>   running ./hugetlb_fault_after_madv
>>>>   ----------------------------------
>>>>   ./run_vmtests.sh: line 186: 595563 Bus error    (core dumped) "$@"
>>>>   [FAIL]
>>>>
>>>> This selftest goes together with the fix of the bug[1] itself.
>>>>
>>>> [1] https://lore.kernel.org/all/20231001005659.2185316-1-riel@surriel.com/#r
>>>>
>>>> Signed-off-by: Breno Leitao <leitao@debian.org>
>>>> ---
>>>>  tools/testing/selftests/mm/Makefile           |  1 +
>>>>  .../selftests/mm/hugetlb_fault_after_madv.c   | 73 +++++++++++++++++++
>>>>  tools/testing/selftests/mm/run_vmtests.sh     |  4 +
>>>>  3 files changed, 78 insertions(+)
>>>>  create mode 100644 tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>>>>
>>>> diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
>>>> index 6a9fc5693145..e71ec9910c62 100644
>>>> --- a/tools/testing/selftests/mm/Makefile
>>>> +++ b/tools/testing/selftests/mm/Makefile
>>>> @@ -68,6 +68,7 @@ TEST_GEN_FILES += split_huge_page_test
>>>>  TEST_GEN_FILES += ksm_tests
>>>>  TEST_GEN_FILES += ksm_functional_tests
>>>>  TEST_GEN_FILES += mdwe_test
>>>> +TEST_GEN_FILES += hugetlb_fault_after_madv
>>>>  
>>>>  ifneq ($(ARCH),arm64)
>>>>  TEST_GEN_PROGS += soft-dirty
>>>> diff --git a/tools/testing/selftests/mm/hugetlb_fault_after_madv.c b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>>>> new file mode 100644
>>>> index 000000000000..73b81c632366
>>>> --- /dev/null
>>>> +++ b/tools/testing/selftests/mm/hugetlb_fault_after_madv.c
>>>> @@ -0,0 +1,73 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +#include <pthread.h>
>>>> +#include <stdio.h>
>>>> +#include <stdlib.h>
>>>> +#include <sys/mman.h>
>>>> +#include <sys/types.h>
>>>> +#include <unistd.h>
>>>> +
>>>> +#include "vm_util.h"
>>>> +#include "../kselftest.h"
>>>> +
>>>> +#define MMAP_SIZE (1 << 21)
>>>> +#define INLOOP_ITER 100
>>>> +
>>>> +char *huge_ptr;
>>>> +
>>>> +/* Touch the memory while it is being madvised() */
>>>> +void *touch(void *unused)
>>>> +{
>>>> +	char *ptr = (char *)huge_ptr;
>>>> +
>>>> +	for (int i = 0; i < INLOOP_ITER; i++)
>>>> +		ptr[0] = '.';
>>>> +
>>>> +	return NULL;
>>>> +}
>>>> +
>>>> +void *madv(void *unused)
>>>> +{
>>>> +	usleep(rand() % 10);
>>>> +
>>>> +	for (int i = 0; i < INLOOP_ITER; i++)
>>>> +		madvise(huge_ptr, MMAP_SIZE, MADV_DONTNEED);
>>>> +
>>>> +	return NULL;
>>>> +}
>>>> +
>>>> +int main(void)
>>>> +{
>>>> +	unsigned long free_hugepages;
>>>> +	pthread_t thread1, thread2;
>>>> +	/*
>>>> +	 * On kernel 6.4, we are able to reproduce the problem with ~1000
>>>> +	 * interactions
>>>> +	 */
>>>> +	int max = 10000;
>>>> +
>>>> +	srand(getpid());
>>>> +
>>>> +	free_hugepages = get_free_hugepages();
>>>> +	if (free_hugepages != 1) {
>>>> +		ksft_exit_skip("This test needs one and only one page to execute. Got %lu\n",
>>>> +			       free_hugepages);
>>>> +	}
>>>> +
>>>> +	while (max--) {
>>>> +		huge_ptr = mmap(NULL, MMAP_SIZE, PROT_READ | PROT_WRITE,
>>>> +				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
>>>> +				-1, 0);
>>>> +
>>>> +		if ((unsigned long)huge_ptr == -1)
>>>> +			ksft_exit_skip("Failed to allocated huge page\n");
>>>> +
>>>> +		pthread_create(&thread1, NULL, madv, NULL);
>>>> +		pthread_create(&thread2, NULL, touch, NULL);
>>>> +
>>>> +		pthread_join(thread1, NULL);
>>>> +		pthread_join(thread2, NULL);
>>>> +		munmap(huge_ptr, MMAP_SIZE);
>>>> +	}
>>>> +
>>>> +	return KSFT_PASS;
>>>> +}
>>>> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
>>>> index 3e2bc818d566..9f53f7318a38 100755
>>>> --- a/tools/testing/selftests/mm/run_vmtests.sh
>>>> +++ b/tools/testing/selftests/mm/run_vmtests.sh
>>>> @@ -221,6 +221,10 @@ CATEGORY="hugetlb" run_test ./hugepage-mremap
>>>>  CATEGORY="hugetlb" run_test ./hugepage-vmemmap
>>>>  CATEGORY="hugetlb" run_test ./hugetlb-madvise
>>>>  
>>>> +# For this test, we need one and just one huge page
>>>> +echo 1 > /proc/sys/vm/nr_hugepages
>>>
>>> I've noticed that this change breaks some of the uffd-stress tests further down
>>> the file, because you have freed previously reserved hugepages that the test
>>> requires to run. I notice that the patch is already in mm-stable, so perhaps its
>>> possible to submit a patch that does a save and restore?
>>>
>>> Although I'm not sure if that might be tricky because the previous reservation
>>> is per-size and per-node (our CI does this on the kernel command line), and I
>>> suspect if you want just 1 huge page in the entire system you won't be able to
>>> get back to the previous state by just restoring this value?
>>
>> Actually on closer inspection, I don't think this will be a problem; simply
>> saving and restoring the value around the test will be sufficient.
> 
> Thanks for checking it, I will prepare a patch that will restore the number
> of huge pages allocated after the test.
> 
>> I also notice that the binary for the new test is not added to the .gitignore,
>> which is a minor annoyance.
> 
> I will add the file to .gitignore also.

Excellent - thanks!

> 
> Thanks for the heads-up.