[BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Zhouping Liu]

Hello All,

When I did some testing to check thp's performance, the following
strange action occurred:

when a process try to allocate 500+(or other large value)
anonymous hugepage, the 'khugepaged' thread will stop to
scan vma. the testing system has 2Gb RAM, and the thp
enabled value is 'always', set 0 to 'scan_sleep_millisecs'

you can use the following steps to confirm the issue:

---------------- example code ------------
/* file test_thp.c */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------

executing './test_thp 500' in a system with 2GB RAM, the values in
/sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.

as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
and found the 'khugepaged' is stopped:
# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                               ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Wanpeng Li]

On Sun, Apr 07, 2013 at 02:29:30AM -0400, Zhouping Liu wrote:
>Hello All,
>
>When I did some testing to check thp's performance, the following
>strange action occurred:
>
>when a process try to allocate 500+(or other large value)
>anonymous hugepage, the 'khugepaged' thread will stop to
>scan vma. the testing system has 2Gb RAM, and the thp
>enabled value is 'always', set 0 to 'scan_sleep_millisecs'
>
>you can use the following steps to confirm the issue:
>
>---------------- example code ------------
>/* file test_thp.c */
>
>#include <stdio.h>
>#include <stdlib.h>
>#include <string.h>
>#include <sys/mman.h>
>
>int main(int argc, char *argv[])
>{
>	int nr_thps = 1000, ret = 0;
>	unsigned long hugepagesize, size;
>	void *addr;
>
>	hugepagesize = (1UL << 21);
>
>	if (argc == 2)
>		nr_thps = atoi(argv[1]);
>
>	printf("try to allocate %d transparent hugepages\n", nr_thps);
>	size = (unsigned long)nr_thps * hugepagesize;
>
>	ret = posix_memalign(&addr, hugepagesize, size);
>	if (ret != 0) {
>		printf("posix_memalign failed\n");
>		return ret;
>	}
>
>	memset (addr, 10, size);
>
>	sleep(50);
>
>	return ret;
>}
>-------- end example code -----------
>
>executing './test_thp 500' in a system with 2GB RAM, the values in
>/sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
>you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
>
>as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
>the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
>but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
>and found the 'khugepaged' is stopped:
># ps aux | grep -i hugepaged
>root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
>                                               ^^ 
>also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
>will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
>comments?

khugepaged will preallocate one hugepage in NUMA case or alloc one 
hugepage before collapse in UMA case. If the memory is serious 
fragmentation and can't successfully allocate hugepage, khugepaged 
will go to sleep one minute. scan_sleep_millisecs determines how 
many milliseconds to wait in khugepaged between each pass, however, 
alloc_sleep_millisecs(default value is one minute) determines how 
many milliseconds to wait in khugepaged if there's an hugepage 
allocation failure to throttle the next allocation attempt.

Regards,
Wanpeng Li 

>
>-- 
>Thanks,
>Zhouping
>
>--
>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>the body to majordomo@kvack.org.  For more info on Linux MM,
>see: http://www.linux-mm.org/ .
>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Wanpeng Li]

On Sun, Apr 07, 2013 at 02:29:30AM -0400, Zhouping Liu wrote:
>Hello All,
>
>When I did some testing to check thp's performance, the following
>strange action occurred:
>
>when a process try to allocate 500+(or other large value)
>anonymous hugepage, the 'khugepaged' thread will stop to
>scan vma. the testing system has 2Gb RAM, and the thp
>enabled value is 'always', set 0 to 'scan_sleep_millisecs'
>
>you can use the following steps to confirm the issue:
>
>---------------- example code ------------
>/* file test_thp.c */
>
>#include <stdio.h>
>#include <stdlib.h>
>#include <string.h>
>#include <sys/mman.h>
>
>int main(int argc, char *argv[])
>{
>	int nr_thps = 1000, ret = 0;
>	unsigned long hugepagesize, size;
>	void *addr;
>
>	hugepagesize = (1UL << 21);
>
>	if (argc == 2)
>		nr_thps = atoi(argv[1]);
>
>	printf("try to allocate %d transparent hugepages\n", nr_thps);
>	size = (unsigned long)nr_thps * hugepagesize;
>
>	ret = posix_memalign(&addr, hugepagesize, size);
>	if (ret != 0) {
>		printf("posix_memalign failed\n");
>		return ret;
>	}
>
>	memset (addr, 10, size);
>
>	sleep(50);
>
>	return ret;
>}
>-------- end example code -----------
>
>executing './test_thp 500' in a system with 2GB RAM, the values in
>/sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
>you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
>
>as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
>the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
>but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
>and found the 'khugepaged' is stopped:
># ps aux | grep -i hugepaged
>root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
>                                               ^^ 
>also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
>will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
>comments?

khugepaged will preallocate one hugepage in NUMA case or alloc one 
hugepage before collapse in UMA case. If the memory is serious 
fragmentation and can't successfully allocate hugepage, khugepaged 
will go to sleep one minute. scan_sleep_millisecs determines how 
many milliseconds to wait in khugepaged between each pass, however, 
alloc_sleep_millisecs(default value is one minute) determines how 
many milliseconds to wait in khugepaged if there's an hugepage 
allocation failure to throttle the next allocation attempt.

Regards,
Wanpeng Li 

>
>-- 
>Thanks,
>Zhouping
>
>--
>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>the body to majordomo@kvack.org.  For more info on Linux MM,
>see: http://www.linux-mm.org/ .
>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Zhouping Liu]

On 04/07/2013 03:10 PM, Wanpeng Li wrote:
> On Sun, Apr 07, 2013 at 02:29:30AM -0400, Zhouping Liu wrote:
>> Hello All,
>>
>> When I did some testing to check thp's performance, the following
>> strange action occurred:
>>
>> when a process try to allocate 500+(or other large value)
>> anonymous hugepage, the 'khugepaged' thread will stop to
>> scan vma. the testing system has 2Gb RAM, and the thp
>> enabled value is 'always', set 0 to 'scan_sleep_millisecs'
>>
>> you can use the following steps to confirm the issue:
>>
>> ---------------- example code ------------
>> /* file test_thp.c */
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> #include <sys/mman.h>
>>
>> int main(int argc, char *argv[])
>> {
>> 	int nr_thps = 1000, ret = 0;
>> 	unsigned long hugepagesize, size;
>> 	void *addr;
>>
>> 	hugepagesize = (1UL << 21);
>>
>> 	if (argc == 2)
>> 		nr_thps = atoi(argv[1]);
>>
>> 	printf("try to allocate %d transparent hugepages\n", nr_thps);
>> 	size = (unsigned long)nr_thps * hugepagesize;
>>
>> 	ret = posix_memalign(&addr, hugepagesize, size);
>> 	if (ret != 0) {
>> 		printf("posix_memalign failed\n");
>> 		return ret;
>> 	}
>>
>> 	memset (addr, 10, size);
>>
>> 	sleep(50);
>>
>> 	return ret;
>> }
>> -------- end example code -----------
>>
>> executing './test_thp 500' in a system with 2GB RAM, the values in
>> /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
>> you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
>>
>> as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
>> the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
>> but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
>> and found the 'khugepaged' is stopped:
>> # ps aux | grep -i hugepaged
>> root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
>>                                                ^^
>> also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
>> will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
>> comments?
> khugepaged will preallocate one hugepage in NUMA case or alloc one
> hugepage before collapse in UMA case. If the memory is serious
> fragmentation and can't successfully allocate hugepage, khugepaged
> will go to sleep one minute. scan_sleep_millisecs determines how
> many milliseconds to wait in khugepaged between each pass, however,
> alloc_sleep_millisecs(default value is one minute) determines how
> many milliseconds to wait in khugepaged if there's an hugepage
> allocation failure to throttle the next allocation attempt.

I see, I only set scan_sleep_millisecs, didn't care 
alloc_sleep_millisecs value.
yes, you are right, allocating memory failure made khugepaged stopped in 
my case.

Thanks very much :)

Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Wanpeng Li]

On Sun, Apr 07, 2013 at 03:10:47PM +0800, Wanpeng Li wrote:
>On Sun, Apr 07, 2013 at 02:29:30AM -0400, Zhouping Liu wrote:
>>Hello All,
>>
>>When I did some testing to check thp's performance, the following
>>strange action occurred:
>>
>>when a process try to allocate 500+(or other large value)
>>anonymous hugepage, the 'khugepaged' thread will stop to
>>scan vma. the testing system has 2Gb RAM, and the thp
>>enabled value is 'always', set 0 to 'scan_sleep_millisecs'
>>
>>you can use the following steps to confirm the issue:
>>
>>---------------- example code ------------
>>/* file test_thp.c */
>>
>>#include <stdio.h>
>>#include <stdlib.h>
>>#include <string.h>
>>#include <sys/mman.h>
>>
>>int main(int argc, char *argv[])
>>{
>>	int nr_thps = 1000, ret = 0;
>>	unsigned long hugepagesize, size;
>>	void *addr;
>>
>>	hugepagesize = (1UL << 21);
>>
>>	if (argc == 2)
>>		nr_thps = atoi(argv[1]);
>>
>>	printf("try to allocate %d transparent hugepages\n", nr_thps);
>>	size = (unsigned long)nr_thps * hugepagesize;
>>
>>	ret = posix_memalign(&addr, hugepagesize, size);
>>	if (ret != 0) {
>>		printf("posix_memalign failed\n");
>>		return ret;
>>	}
>>
>>	memset (addr, 10, size);
>>
>>	sleep(50);
>>
>>	return ret;
>>}
>>-------- end example code -----------
>>
>>executing './test_thp 500' in a system with 2GB RAM, the values in
>>/sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
>>you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
>>
>>as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
>>the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
>>but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
>>and found the 'khugepaged' is stopped:
>># ps aux | grep -i hugepaged
>>root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
>>                                               ^^ 
>>also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
>>will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
>>comments?
>
>khugepaged will preallocate one hugepage in NUMA case or alloc one 
>hugepage before collapse in UMA case. If the memory is serious

Sorry, it should be reverse. khugepaged will preallocate one hugepage 
in UMA case and alloc one hugepage before collapse in NUMA case.

Regards,
Wanpeng Li 

>fragmentation and can't successfully allocate hugepage, khugepaged 
>will go to sleep one minute. scan_sleep_millisecs determines how 
>many milliseconds to wait in khugepaged between each pass, however, 
>alloc_sleep_millisecs(default value is one minute) determines how 
>many milliseconds to wait in khugepaged if there's an hugepage 
>allocation failure to throttle the next allocation attempt.
>
>Regards,
>Wanpeng Li 
>
>>
>>-- 
>>Thanks,
>>Zhouping
>>
>>--
>>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>the body to majordomo@kvack.org.  For more info on Linux MM,
>>see: http://www.linux-mm.org/ .
>>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
>--
>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>the body to majordomo@kvack.org.  For more info on Linux MM,
>see: http://www.linux-mm.org/ .
>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG?] thp: too much anonymous hugepage caused 'khugepaged' thread stopped

[Wanpeng Li]

On Sun, Apr 07, 2013 at 03:10:47PM +0800, Wanpeng Li wrote:
>On Sun, Apr 07, 2013 at 02:29:30AM -0400, Zhouping Liu wrote:
>>Hello All,
>>
>>When I did some testing to check thp's performance, the following
>>strange action occurred:
>>
>>when a process try to allocate 500+(or other large value)
>>anonymous hugepage, the 'khugepaged' thread will stop to
>>scan vma. the testing system has 2Gb RAM, and the thp
>>enabled value is 'always', set 0 to 'scan_sleep_millisecs'
>>
>>you can use the following steps to confirm the issue:
>>
>>---------------- example code ------------
>>/* file test_thp.c */
>>
>>#include <stdio.h>
>>#include <stdlib.h>
>>#include <string.h>
>>#include <sys/mman.h>
>>
>>int main(int argc, char *argv[])
>>{
>>	int nr_thps = 1000, ret = 0;
>>	unsigned long hugepagesize, size;
>>	void *addr;
>>
>>	hugepagesize = (1UL << 21);
>>
>>	if (argc == 2)
>>		nr_thps = atoi(argv[1]);
>>
>>	printf("try to allocate %d transparent hugepages\n", nr_thps);
>>	size = (unsigned long)nr_thps * hugepagesize;
>>
>>	ret = posix_memalign(&addr, hugepagesize, size);
>>	if (ret != 0) {
>>		printf("posix_memalign failed\n");
>>		return ret;
>>	}
>>
>>	memset (addr, 10, size);
>>
>>	sleep(50);
>>
>>	return ret;
>>}
>>-------- end example code -----------
>>
>>executing './test_thp 500' in a system with 2GB RAM, the values in
>>/sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
>>you can  repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
>>
>>as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs,
>>the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least,
>>but the actual is opposite, the value is never change, so I checked 'khugepaged' thread,
>>and found the 'khugepaged' is stopped:
>># ps aux | grep -i hugepaged
>>root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
>>                                               ^^ 
>>also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
>>will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
>>comments?
>
>khugepaged will preallocate one hugepage in NUMA case or alloc one 
>hugepage before collapse in UMA case. If the memory is serious

Sorry, it should be reverse. khugepaged will preallocate one hugepage 
in UMA case and alloc one hugepage before collapse in NUMA case.

Regards,
Wanpeng Li 

>fragmentation and can't successfully allocate hugepage, khugepaged 
>will go to sleep one minute. scan_sleep_millisecs determines how 
>many milliseconds to wait in khugepaged between each pass, however, 
>alloc_sleep_millisecs(default value is one minute) determines how 
>many milliseconds to wait in khugepaged if there's an hugepage 
>allocation failure to throttle the next allocation attempt.
>
>Regards,
>Wanpeng Li 
>
>>
>>-- 
>>Thanks,
>>Zhouping
>>
>>--
>>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>the body to majordomo@kvack.org.  For more info on Linux MM,
>>see: http://www.linux-mm.org/ .
>>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
>--
>To unsubscribe, send a message with 'unsubscribe linux-mm' in
>the body to majordomo@kvack.org.  For more info on Linux MM,
>see: http://www.linux-mm.org/ .
>Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>