From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx166.postini.com [74.125.245.166]) by kanga.kvack.org (Postfix) with SMTP id 84C9A6B0002 for ; Mon, 1 Apr 2013 04:00:29 -0400 (EDT) Date: Mon, 1 Apr 2013 04:00:27 -0400 (EDT) From: Zhouping Liu Message-ID: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: <1207916095.642011.1364800448075.JavaMail.root@redhat.com> Subject: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrea Arcangeli , Hugh Dickins , Mel Gorman , David Rientjes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Hi all, I found THP can't correctly distinguish one anonymous hugepage map. 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less. Testing code: ---- snip -------- unsigned long hugepagesize = (1UL << 21); int main() { void *addr; int i; printf("pid is %d\n", getpid()); for (i = 0; i < 5; i++) { addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return -1; } memset(addr, i, hugepagesize); } sleep(50); return 0; } ------ snip ---------- the /proc/[pid]/smaps show that Anonymous is 10240kB but AnonHugePages is 8192Kb, one THP less: ----- snip -------- 7f59ccc01000-7f59cd601000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 10240 kB Pss: 10240 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 10240 kB Referenced: 10240 kB Anonymous: 10240 kB AnonHugePages: 8192 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac hg ------- sinp --------- 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size: Testing code: -------- snip -------- unsigned long hugepagesize = (1UL << 21); int main() { void *addr; int i; printf("pid is %d\n", getpid()); for (i = 0; i < 5; i++) { addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return -1; } if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { perror("madvise"); return -1; } memset(addr, i, hugepagesize); } sleep(50); return 0; } --------- snip ---------- The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : -------------- snip ------- 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 10240 kB Pss: 10240 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 10240 kB Referenced: 10240 kB Anonymous: 10240 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac ----------- snip ---------- 3. when I made the address aligned with HUGEPAGESIZE using 'posix_memalign()' instead of mmap(), THP perform good, and can distinguish all anonymous huge pages. my question is: 1. all the above behaviour is right? 2. why THP can't distinguish one naturally aligned huge page(generated by mmap())? -- Thanks, Zhouping -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx170.postini.com [74.125.245.170]) by kanga.kvack.org (Postfix) with SMTP id 879186B0002 for ; Mon, 1 Apr 2013 18:23:36 -0400 (EDT) Received: by mail-pa0-f41.google.com with SMTP id kx1so1538944pab.28 for ; Mon, 01 Apr 2013 15:23:35 -0700 (PDT) Date: Mon, 1 Apr 2013 15:23:33 -0700 (PDT) From: David Rientjes Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <383590596.664138.1364803227470.JavaMail.root@redhat.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Zhouping Liu Cc: Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On Mon, 1 Apr 2013, Zhouping Liu wrote: > Hi all, > > I found THP can't correctly distinguish one anonymous hugepage map. > > 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the > amount of THP always is one less. > It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time. > 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't > distinguish any one anonymous hugepage size: > > Testing code: > -------- snip -------- > unsigned long hugepagesize = (1UL << 21); > > int main() > { > void *addr; > int i; > > printf("pid is %d\n", getpid()); > > for (i = 0; i < 5; i++) { > addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); > > if (addr == MAP_FAILED) { > perror("mmap"); > return -1; > } > > if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { > perror("madvise"); > return -1; > } > > memset(addr, i, hugepagesize); > } > > sleep(50); > > return 0; > } > --------- snip ---------- > > The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : > -------------- snip ------- > 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 > Size: 10240 kB > Rss: 10240 kB > Pss: 10240 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 10240 kB > Referenced: 10240 kB > Anonymous: 10240 kB > AnonHugePages: 0 kB > Swap: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me ac "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx129.postini.com [74.125.245.129]) by kanga.kvack.org (Postfix) with SMTP id 4811B6B005A for ; Mon, 1 Apr 2013 23:11:05 -0400 (EDT) Message-ID: <515A4BD1.1020407@redhat.com> Date: Tue, 02 Apr 2013 11:09:05 +0800 From: Zhouping Liu MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' could you please explain more details about 'only 1/512th of the time'? > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? I think it's the same issue as the above, according to the sample code, it does mmap() five times in total, and each time it map 2MB anonymous maps, as all the maps maybe aren't 2MB aligned, so "AnonHugePages" show 0 kB, and no "hg" VmFalgs. so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? Thanks, Zhouping -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx195.postini.com [74.125.245.195]) by kanga.kvack.org (Postfix) with SMTP id F0F976B0002 for ; Mon, 1 Apr 2013 23:38:34 -0400 (EDT) Message-ID: <515A5329.6020805@cn.fujitsu.com> Date: Tue, 02 Apr 2013 11:40:25 +0800 From: Lin Feng MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> In-Reply-To: <515A4BD1.1020407@redhat.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Zhouping Liu Cc: David Rientjes , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Hi Zhouping, On 04/02/2013 11:09 AM, Zhouping Liu wrote: > I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' > could you please explain more details about 'only 1/512th of the time'? IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). Since your test code is not 2M aligned(not using posix_memalign()) so the start address of the mapped vma will be random, such as 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities. The only chance you get thp happens when the first map just starts at 2M*i, and the consequent maps also benefit from this. -------- snip -------- > > so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), > make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? We may gain performance improving from this. thanks, linfeng -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx111.postini.com [74.125.245.111]) by kanga.kvack.org (Postfix) with SMTP id 673496B0002 for ; Mon, 1 Apr 2013 23:44:37 -0400 (EDT) Received: by mail-pa0-f50.google.com with SMTP id bg2so49356pad.23 for ; Mon, 01 Apr 2013 20:44:36 -0700 (PDT) Date: Mon, 1 Apr 2013 20:44:34 -0700 (PDT) From: David Rientjes Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <515A5329.6020805@cn.fujitsu.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> <515A5329.6020805@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Lin Feng Cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On Tue, 2 Apr 2013, Lin Feng wrote: > > so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), > > make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? > > We may gain performance improving from this. > To attain the maximum number of hugepages, you would naturally want to ensure that the mappings are done aligned to 2MB; for very large allocations, missing one or two hugepages typically won't hurt performance much. posix_memalign() is the best way of doing this which just wraps mmap() for the needed alignment. More interesting is creating your own custom malloc() that allocates in 2MB aligned chunks, if possible, and uses 2MB aligned arenas for its own metadata. If you do that for malloc(), then you'll only need to make code that does its own mmap()s to use posix_memalign(). -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx106.postini.com [74.125.245.106]) by kanga.kvack.org (Postfix) with SMTP id 00C576B0036 for ; Mon, 1 Apr 2013 23:50:32 -0400 (EDT) Message-ID: <515A5506.1000809@redhat.com> Date: Tue, 02 Apr 2013 11:48:22 +0800 From: Zhouping Liu MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> <515A5329.6020805@cn.fujitsu.com> In-Reply-To: <515A5329.6020805@cn.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Lin Feng Cc: David Rientjes , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On 04/02/2013 11:40 AM, Lin Feng wrote: > Hi Zhouping, > > On 04/02/2013 11:09 AM, Zhouping Liu wrote: >> I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' >> could you please explain more details about 'only 1/512th of the time'? > IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). > Since your test code is not 2M aligned(not using posix_memalign()) so > the start address of the mapped vma will be random, such as > 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities. > > The only chance you get thp happens when the first map just starts at 2M*i, > and the consequent maps also benefit from this. Feng, it's easy to understand now, thanks for your detailed explanation :) Zhouping -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx160.postini.com [74.125.245.160]) by kanga.kvack.org (Postfix) with SMTP id 04D196B0002 for ; Tue, 2 Apr 2013 08:23:45 -0400 (EDT) Received: by mail-qc0-f177.google.com with SMTP id u28so132154qcs.22 for ; Tue, 02 Apr 2013 05:23:45 -0700 (PDT) Message-ID: <515ACDC9.2090506@gmail.com> Date: Tue, 02 Apr 2013 20:23:37 +0800 From: Simon Jeons MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Hi David, On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code Both thp and hugetlb pages should be 2MB aligned, correct? > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx122.postini.com [74.125.245.122]) by kanga.kvack.org (Postfix) with SMTP id 0CD3D6B0027 for ; Tue, 2 Apr 2013 08:26:54 -0400 (EDT) Received: by mail-qa0-f48.google.com with SMTP id hu16so188243qab.7 for ; Tue, 02 Apr 2013 05:26:54 -0700 (PDT) Message-ID: <515ACE87.8070005@gmail.com> Date: Tue, 02 Apr 2013 20:26:47 +0800 From: Simon Jeons MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code Btw, why need 2MB aligned? Does it has relationship with tlb? > > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx159.postini.com [74.125.245.159]) by kanga.kvack.org (Postfix) with SMTP id DF20B6B0002 for ; Tue, 2 Apr 2013 14:09:03 -0400 (EDT) Received: by mail-pa0-f42.google.com with SMTP id kq13so435251pab.15 for ; Tue, 02 Apr 2013 11:09:03 -0700 (PDT) Date: Tue, 2 Apr 2013 11:09:01 -0700 (PDT) From: David Rientjes Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <515ACDC9.2090506@gmail.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515ACDC9.2090506@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Simon Jeons Cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong On Tue, 2 Apr 2013, Simon Jeons wrote: > Both thp and hugetlb pages should be 2MB aligned, correct? > To answer this question and your followup reply at the same time: they come from one level higher in the page table so they will naturally need to be 2MB aligned. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx116.postini.com [74.125.245.116]) by kanga.kvack.org (Postfix) with SMTP id 9EEEA6B003D for ; Tue, 2 Apr 2013 19:58:32 -0400 (EDT) Received: by mail-qa0-f42.google.com with SMTP id bv4so1778918qab.15 for ; Tue, 02 Apr 2013 16:58:31 -0700 (PDT) Message-ID: <515B70A1.3040200@gmail.com> Date: Wed, 03 Apr 2013 07:58:25 +0800 From: Simon Jeons MIME-Version: 1.0 Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515ACDC9.2090506@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Hi David, On 04/03/2013 02:09 AM, David Rientjes wrote: > On Tue, 2 Apr 2013, Simon Jeons wrote: > >> Both thp and hugetlb pages should be 2MB aligned, correct? >> > To answer this question and your followup reply at the same time: they > come from one level higher in the page table so they will naturally need > to be 2MB aligned. When I hacking arch/x86/mm/hugetlbpage.c like this, diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index ae1aa71..87f34ee 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -354,14 +354,13 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, #endif /*HAVE_ARCH_HUGETLB_UNMAPPED_AREA*/ -#ifdef CONFIG_X86_64 static __init int setup_hugepagesz(char *opt) { unsigned long ps = memparse(opt, &opt); if (ps == PMD_SIZE) { hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT); - } else if (ps == PUD_SIZE && cpu_has_gbpages) { - hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT); + } else if (ps == PUD_SIZE) { + hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT+4); } else { printk(KERN_ERR "hugepagesz: Unsupported page size %lu M\n", ps >> 20); I set boot=hugepagesz=1G hugepages=10, then I got 10 32MB huge pages. What's the difference between these pages which I hacking and normal huge pages? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756897Ab3DAIBN (ORCPT ); Mon, 1 Apr 2013 04:01:13 -0400 Received: from mx4-phx2.redhat.com ([209.132.183.25]:41513 "EHLO mx4-phx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751281Ab3DAIBM (ORCPT ); Mon, 1 Apr 2013 04:01:12 -0400 Date: Mon, 1 Apr 2013 04:00:27 -0400 (EDT) From: Zhouping Liu To: Andrea Arcangeli , Hugh Dickins , Mel Gorman , David Rientjes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Message-ID: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: <1207916095.642011.1364800448075.JavaMail.root@redhat.com> Subject: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.5.82.12] X-Mailer: Zimbra 8.0.3_GA_5664 (ZimbraWebClient - FF17 (Linux)/8.0.3_GA_5664) Thread-Topic: AnonHugePages in /proc/[pid]/smaps is correct or not? Thread-Index: XwswPDG4DyQ4aShL0QozGDI+OoiLOQ== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, I found THP can't correctly distinguish one anonymous hugepage map. 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less. Testing code: ---- snip -------- unsigned long hugepagesize = (1UL << 21); int main() { void *addr; int i; printf("pid is %d\n", getpid()); for (i = 0; i < 5; i++) { addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return -1; } memset(addr, i, hugepagesize); } sleep(50); return 0; } ------ snip ---------- the /proc/[pid]/smaps show that Anonymous is 10240kB but AnonHugePages is 8192Kb, one THP less: ----- snip -------- 7f59ccc01000-7f59cd601000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 10240 kB Pss: 10240 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 10240 kB Referenced: 10240 kB Anonymous: 10240 kB AnonHugePages: 8192 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac hg ------- sinp --------- 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size: Testing code: -------- snip -------- unsigned long hugepagesize = (1UL << 21); int main() { void *addr; int i; printf("pid is %d\n", getpid()); for (i = 0; i < 5; i++) { addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return -1; } if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { perror("madvise"); return -1; } memset(addr, i, hugepagesize); } sleep(50); return 0; } --------- snip ---------- The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : -------------- snip ------- 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 10240 kB Pss: 10240 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 10240 kB Referenced: 10240 kB Anonymous: 10240 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac ----------- snip ---------- 3. when I made the address aligned with HUGEPAGESIZE using 'posix_memalign()' instead of mmap(), THP perform good, and can distinguish all anonymous huge pages. my question is: 1. all the above behaviour is right? 2. why THP can't distinguish one naturally aligned huge page(generated by mmap())? -- Thanks, Zhouping From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759225Ab3DAWYC (ORCPT ); Mon, 1 Apr 2013 18:24:02 -0400 Received: from mail-da0-f53.google.com ([209.85.210.53]:65025 "EHLO mail-da0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758530Ab3DAWXg (ORCPT ); Mon, 1 Apr 2013 18:23:36 -0400 Date: Mon, 1 Apr 2013 15:23:33 -0700 (PDT) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Zhouping Liu cc: Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <383590596.664138.1364803227470.JavaMail.root@redhat.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 1 Apr 2013, Zhouping Liu wrote: > Hi all, > > I found THP can't correctly distinguish one anonymous hugepage map. > > 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the > amount of THP always is one less. > It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time. > 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't > distinguish any one anonymous hugepage size: > > Testing code: > -------- snip -------- > unsigned long hugepagesize = (1UL << 21); > > int main() > { > void *addr; > int i; > > printf("pid is %d\n", getpid()); > > for (i = 0; i < 5; i++) { > addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); > > if (addr == MAP_FAILED) { > perror("mmap"); > return -1; > } > > if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { > perror("madvise"); > return -1; > } > > memset(addr, i, hugepagesize); > } > > sleep(50); > > return 0; > } > --------- snip ---------- > > The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : > -------------- snip ------- > 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 > Size: 10240 kB > Rss: 10240 kB > Pss: 10240 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 10240 kB > Referenced: 10240 kB > Anonymous: 10240 kB > AnonHugePages: 0 kB > Swap: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me ac "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759892Ab3DBDLt (ORCPT ); Mon, 1 Apr 2013 23:11:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:8173 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759360Ab3DBDLs (ORCPT ); Mon, 1 Apr 2013 23:11:48 -0400 Message-ID: <515A4BD1.1020407@redhat.com> Date: Tue, 02 Apr 2013 11:09:05 +0800 From: Zhouping Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: David Rientjes CC: Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' could you please explain more details about 'only 1/512th of the time'? > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? I think it's the same issue as the above, according to the sample code, it does mmap() five times in total, and each time it map 2MB anonymous maps, as all the maps maybe aren't 2MB aligned, so "AnonHugePages" show 0 kB, and no "hg" VmFalgs. so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? Thanks, Zhouping From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760047Ab3DBDid (ORCPT ); Mon, 1 Apr 2013 23:38:33 -0400 Received: from cn.fujitsu.com ([222.73.24.84]:12670 "EHLO song.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-OK) by vger.kernel.org with ESMTP id S1759535Ab3DBDic (ORCPT ); Mon, 1 Apr 2013 23:38:32 -0400 X-IronPort-AV: E=Sophos;i="4.87,391,1363104000"; d="scan'208";a="6987181" Message-ID: <515A5329.6020805@cn.fujitsu.com> Date: Tue, 02 Apr 2013 11:40:25 +0800 From: Lin Feng User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Zhouping Liu CC: David Rientjes , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> In-Reply-To: <515A4BD1.1020407@redhat.com> X-MIMETrack: Itemize by SMTP Server on mailserver/fnst(Release 8.5.3|September 15, 2011) at 2013/04/02 11:37:30, Serialize by Router on mailserver/fnst(Release 8.5.3|September 15, 2011) at 2013/04/02 11:37:31, Serialize complete at 2013/04/02 11:37:31 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Zhouping, On 04/02/2013 11:09 AM, Zhouping Liu wrote: > I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' > could you please explain more details about 'only 1/512th of the time'? IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). Since your test code is not 2M aligned(not using posix_memalign()) so the start address of the mapped vma will be random, such as 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities. The only chance you get thp happens when the first map just starts at 2M*i, and the consequent maps also benefit from this. -------- snip -------- > > so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), > make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? We may gain performance improving from this. thanks, linfeng From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759709Ab3DBDoi (ORCPT ); Mon, 1 Apr 2013 23:44:38 -0400 Received: from mail-pa0-f41.google.com ([209.85.220.41]:39418 "EHLO mail-pa0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757413Ab3DBDoh (ORCPT ); Mon, 1 Apr 2013 23:44:37 -0400 Date: Mon, 1 Apr 2013 20:44:34 -0700 (PDT) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Lin Feng cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <515A5329.6020805@cn.fujitsu.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> <515A5329.6020805@cn.fujitsu.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2 Apr 2013, Lin Feng wrote: > > so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), > > make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it? > > We may gain performance improving from this. > To attain the maximum number of hugepages, you would naturally want to ensure that the mappings are done aligned to 2MB; for very large allocations, missing one or two hugepages typically won't hurt performance much. posix_memalign() is the best way of doing this which just wraps mmap() for the needed alignment. More interesting is creating your own custom malloc() that allocates in 2MB aligned chunks, if possible, and uses 2MB aligned arenas for its own metadata. If you do that for malloc(), then you'll only need to make code that does its own mmap()s to use posix_memalign(). From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760062Ab3DBDvQ (ORCPT ); Mon, 1 Apr 2013 23:51:16 -0400 Received: from mx1.redhat.com ([209.132.183.28]:28456 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758500Ab3DBDvP (ORCPT ); Mon, 1 Apr 2013 23:51:15 -0400 Message-ID: <515A5506.1000809@redhat.com> Date: Tue, 02 Apr 2013 11:48:22 +0800 From: Zhouping Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Lin Feng CC: David Rientjes , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515A4BD1.1020407@redhat.com> <515A5329.6020805@cn.fujitsu.com> In-Reply-To: <515A5329.6020805@cn.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/02/2013 11:40 AM, Lin Feng wrote: > Hi Zhouping, > > On 04/02/2013 11:09 AM, Zhouping Liu wrote: >> I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' >> could you please explain more details about 'only 1/512th of the time'? > IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). > Since your test code is not 2M aligned(not using posix_memalign()) so > the start address of the mapped vma will be random, such as > 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities. > > The only chance you get thp happens when the first map just starts at 2M*i, > and the consequent maps also benefit from this. Feng, it's easy to understand now, thanks for your detailed explanation :) Zhouping From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760019Ab3DBMXq (ORCPT ); Tue, 2 Apr 2013 08:23:46 -0400 Received: from mail-qc0-f171.google.com ([209.85.216.171]:33981 "EHLO mail-qc0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754107Ab3DBMXp (ORCPT ); Tue, 2 Apr 2013 08:23:45 -0400 Message-ID: <515ACDC9.2090506@gmail.com> Date: Tue, 02 Apr 2013 20:23:37 +0800 From: Simon Jeons User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: David Rientjes CC: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi David, On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code Both thp and hugetlb pages should be 2MB aligned, correct? > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760226Ab3DBM07 (ORCPT ); Tue, 2 Apr 2013 08:26:59 -0400 Received: from mail-qe0-f51.google.com ([209.85.128.51]:39779 "EHLO mail-qe0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1760043Ab3DBM06 (ORCPT ); Tue, 2 Apr 2013 08:26:58 -0400 Message-ID: <515ACE87.8070005@gmail.com> Date: Tue, 02 Apr 2013 20:26:47 +0800 From: Simon Jeons User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: David Rientjes CC: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/02/2013 06:23 AM, David Rientjes wrote: > On Mon, 1 Apr 2013, Zhouping Liu wrote: > >> Hi all, >> >> I found THP can't correctly distinguish one anonymous hugepage map. >> >> 1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the >> amount of THP always is one less. >> > It's not a problem with identifying an anonymous mapping as a hugepage, > setting thp enabled to "always" does not guarantee that they will always > be allocatable or that your mmap() will be 2MB aligned. Your sample code Btw, why need 2MB aligned? Does it has relationship with tlb? > > is using mmap() instead of posix_memalign() so you'll probably only get > 100% hugepages only 1/512th of the time. > >> 2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't >> distinguish any one anonymous hugepage size: >> >> Testing code: >> -------- snip -------- >> unsigned long hugepagesize = (1UL << 21); >> >> int main() >> { >> void *addr; >> int i; >> >> printf("pid is %d\n", getpid()); >> >> for (i = 0; i < 5; i++) { >> addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0); >> >> if (addr == MAP_FAILED) { >> perror("mmap"); >> return -1; >> } >> >> if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) { >> perror("madvise"); >> return -1; >> } >> >> memset(addr, i, hugepagesize); >> } >> >> sleep(50); >> >> return 0; >> } >> --------- snip ---------- >> >> The result is that it can't find any AnonHugePages from /proc/[pid]/smaps : >> -------------- snip ------- >> 7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0 >> Size: 10240 kB >> Rss: 10240 kB >> Pss: 10240 kB >> Shared_Clean: 0 kB >> Shared_Dirty: 0 kB >> Private_Clean: 0 kB >> Private_Dirty: 10240 kB >> Referenced: 10240 kB >> Anonymous: 10240 kB >> AnonHugePages: 0 kB >> Swap: 0 kB >> KernelPageSize: 4 kB >> MMUPageSize: 4 kB >> Locked: 0 kB >> VmFlags: rd wr mr mw me ac > "hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are > you sure this is the right vma? > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932751Ab3DBSJF (ORCPT ); Tue, 2 Apr 2013 14:09:05 -0400 Received: from mail-da0-f53.google.com ([209.85.210.53]:59365 "EHLO mail-da0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759845Ab3DBSJD (ORCPT ); Tue, 2 Apr 2013 14:09:03 -0400 Date: Tue, 2 Apr 2013 11:09:01 -0700 (PDT) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Simon Jeons cc: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? In-Reply-To: <515ACDC9.2090506@gmail.com> Message-ID: References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515ACDC9.2090506@gmail.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2 Apr 2013, Simon Jeons wrote: > Both thp and hugetlb pages should be 2MB aligned, correct? > To answer this question and your followup reply at the same time: they come from one level higher in the page table so they will naturally need to be 2MB aligned. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758675Ab3DBX6d (ORCPT ); Tue, 2 Apr 2013 19:58:33 -0400 Received: from mail-qe0-f54.google.com ([209.85.128.54]:54560 "EHLO mail-qe0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756205Ab3DBX6c (ORCPT ); Tue, 2 Apr 2013 19:58:32 -0400 Message-ID: <515B70A1.3040200@gmail.com> Date: Wed, 03 Apr 2013 07:58:25 +0800 From: Simon Jeons User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: David Rientjes CC: Zhouping Liu , Andrea Arcangeli , Hugh Dickins , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Amos Kong Subject: Re: THP: AnonHugePages in /proc/[pid]/smaps is correct or not? References: <383590596.664138.1364803227470.JavaMail.root@redhat.com> <515ACDC9.2090506@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi David, On 04/03/2013 02:09 AM, David Rientjes wrote: > On Tue, 2 Apr 2013, Simon Jeons wrote: > >> Both thp and hugetlb pages should be 2MB aligned, correct? >> > To answer this question and your followup reply at the same time: they > come from one level higher in the page table so they will naturally need > to be 2MB aligned. When I hacking arch/x86/mm/hugetlbpage.c like this, diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index ae1aa71..87f34ee 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -354,14 +354,13 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, #endif /*HAVE_ARCH_HUGETLB_UNMAPPED_AREA*/ -#ifdef CONFIG_X86_64 static __init int setup_hugepagesz(char *opt) { unsigned long ps = memparse(opt, &opt); if (ps == PMD_SIZE) { hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT); - } else if (ps == PUD_SIZE && cpu_has_gbpages) { - hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT); + } else if (ps == PUD_SIZE) { + hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT+4); } else { printk(KERN_ERR "hugepagesz: Unsupported page size %lu M\n", ps >> 20); I set boot=hugepagesz=1G hugepages=10, then I got 10 32MB huge pages. What's the difference between these pages which I hacking and normal huge pages?