[PATCH] mm: make transparent hugepage size public

[PATCH] mm: make transparent hugepage size public

[Hugh Dickins]

Test programs want to know the size of a transparent hugepage.
While it is commonly the same as the size of a hugetlbfs page
(shown as Hugepagesize in /proc/meminfo), that is not always so:
powerpc implements transparent hugepages in a different way from
hugetlbfs pages, so it's coincidence when their sizes are the same;
and x86 and others can support more than one hugetlbfs page size.

Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
THP size in bytes - it's the same for Anonymous and Shmem hugepages.
Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
in case some transparent support for pud and pgd pages is added later.

Signed-off-by: Hugh Dickins <hughd@google.com>
---

 Documentation/vm/transhuge.txt |    5 +++++
 mm/huge_memory.c               |   10 ++++++++++
 2 files changed, 15 insertions(+)

--- 4.9-rc8/Documentation/vm/transhuge.txt	2016-10-02 16:24:33.000000000 -0700
+++ linux/Documentation/vm/transhuge.txt	2016-12-05 20:55:12.142578631 -0800
@@ -136,6 +136,11 @@ or enable it back by writing 1:
 echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 
+Some userspace (such as a test program, or an optimized memory allocation
+library) may want to know the size (in bytes) of a transparent hugepage:
+
+cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
+
 khugepaged will be automatically started when
 transparent_hugepage/enabled is set to "always" or "madvise, and it'll
 be automatically shutdown if it's set to "never".
--- 4.9-rc8/mm/huge_memory.c	2016-12-04 16:42:39.881703357 -0800
+++ linux/mm/huge_memory.c	2016-12-05 20:58:19.953010005 -0800
@@ -285,6 +285,15 @@ static ssize_t use_zero_page_store(struc
 }
 static struct kobj_attribute use_zero_page_attr =
 	__ATTR(use_zero_page, 0644, use_zero_page_show, use_zero_page_store);
+
+static ssize_t hpage_pmd_size_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", HPAGE_PMD_SIZE);
+}
+static struct kobj_attribute hpage_pmd_size_attr =
+	__ATTR_RO(hpage_pmd_size);
+
 #ifdef CONFIG_DEBUG_VM
 static ssize_t debug_cow_show(struct kobject *kobj,
 				struct kobj_attribute *attr, char *buf)
@@ -307,6 +316,7 @@ static struct attribute *hugepage_attr[]
 	&enabled_attr.attr,
 	&defrag_attr.attr,
 	&use_zero_page_attr.attr,
+	&hpage_pmd_size_attr.attr,
 #if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE)
 	&shmem_enabled_attr.attr,
 #endif

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Aneesh Kumar K.V]

Hugh Dickins <hughd@google.com> writes:

> Test programs want to know the size of a transparent hugepage.
> While it is commonly the same as the size of a hugetlbfs page
> (shown as Hugepagesize in /proc/meminfo), that is not always so:
> powerpc implements transparent hugepages in a different way from
> hugetlbfs pages, so it's coincidence when their sizes are the same;
> and x86 and others can support more than one hugetlbfs page size.
>
> Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
> THP size in bytes - it's the same for Anonymous and Shmem hugepages.
> Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
> in case some transparent support for pud and pgd pages is added later.

We have in /proc/meminfo

Hugepagesize:       2048 kB

Does it makes it easy for application to find THP page size also there ?

-aneesh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Dave Hansen]

On 12/06/2016 01:07 AM, Aneesh Kumar K.V wrote:
> Hugh Dickins <hughd@google.com> writes:
> 
>> Test programs want to know the size of a transparent hugepage.
>> While it is commonly the same as the size of a hugetlbfs page
>> (shown as Hugepagesize in /proc/meminfo), that is not always so:
>> powerpc implements transparent hugepages in a different way from
>> hugetlbfs pages, so it's coincidence when their sizes are the same;
>> and x86 and others can support more than one hugetlbfs page size.
>>
>> Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
>> THP size in bytes - it's the same for Anonymous and Shmem hugepages.
>> Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
>> in case some transparent support for pud and pgd pages is added later.
> 
> We have in /proc/meminfo
> 
> Hugepagesize:       2048 kB
> 
> Does it makes it easy for application to find THP page size also there ?

Nope.  That's the default hugetlbfs page size.  Even on x86, that can be
changed and _could_ be 1G.  If hugetlbfs is configured out, you also
won't get this in meminfo.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Kirill A. Shutemov]

On Tue, Dec 06, 2016 at 07:14:50AM -0800, Dave Hansen wrote:
> On 12/06/2016 01:07 AM, Aneesh Kumar K.V wrote:
> > Hugh Dickins <hughd@google.com> writes:
> > 
> >> Test programs want to know the size of a transparent hugepage.
> >> While it is commonly the same as the size of a hugetlbfs page
> >> (shown as Hugepagesize in /proc/meminfo), that is not always so:
> >> powerpc implements transparent hugepages in a different way from
> >> hugetlbfs pages, so it's coincidence when their sizes are the same;
> >> and x86 and others can support more than one hugetlbfs page size.
> >>
> >> Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
> >> THP size in bytes - it's the same for Anonymous and Shmem hugepages.
> >> Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
> >> in case some transparent support for pud and pgd pages is added later.
> > 
> > We have in /proc/meminfo
> > 
> > Hugepagesize:       2048 kB
> > 
> > Does it makes it easy for application to find THP page size also there ?
> 
> Nope.  That's the default hugetlbfs page size.  Even on x86, that can be
> changed and _could_ be 1G.  If hugetlbfs is configured out, you also
> won't get this in meminfo.

I think Aneesh propose to add one more line into the file.

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Dave Hansen]

On 12/06/2016 09:19 AM, Kirill A. Shutemov wrote:
>>> > > We have in /proc/meminfo
>>> > > 
>>> > > Hugepagesize:       2048 kB
>>> > > 
>>> > > Does it makes it easy for application to find THP page size also there ?
>> > 
>> > Nope.  That's the default hugetlbfs page size.  Even on x86, that can be
>> > changed and _could_ be 1G.  If hugetlbfs is configured out, you also
>> > won't get this in meminfo.
> I think Aneesh propose to add one more line into the file.

Ahhh, ok...

Personally, I think Hugh did the right things.  There's no reason to
waste cycles sticking a number in meminfo that never changes.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Andrew Morton]

On Tue, 06 Dec 2016 14:37:54 +0530 "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> wrote:

> Hugh Dickins <hughd@google.com> writes:
> 
> > Test programs want to know the size of a transparent hugepage.
> > While it is commonly the same as the size of a hugetlbfs page
> > (shown as Hugepagesize in /proc/meminfo), that is not always so:
> > powerpc implements transparent hugepages in a different way from
> > hugetlbfs pages, so it's coincidence when their sizes are the same;
> > and x86 and others can support more than one hugetlbfs page size.
> >
> > Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
> > THP size in bytes - it's the same for Anonymous and Shmem hugepages.
> > Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
> > in case some transparent support for pud and pgd pages is added later.
> 
> We have in /proc/meminfo
> 
> Hugepagesize:       2048 kB
> 
> Does it makes it easy for application to find THP page size also there ?
> 

Probably that would be more logical.  But I'm a bit concerned about
adding more stuff to /proc/meminfo from a performance point of view -
that file gets read from quite frequently and we've already put some
quite obscure things in there.  Probably we whould be careful about
this.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Hugh Dickins]

On Tue, 6 Dec 2016, Dave Hansen wrote:
> On 12/06/2016 09:19 AM, Kirill A. Shutemov wrote:
> >>> > > We have in /proc/meminfo
> >>> > > 
> >>> > > Hugepagesize:       2048 kB
> >>> > > 
> >>> > > Does it makes it easy for application to find THP page size also there ?
> >> > 
> >> > Nope.  That's the default hugetlbfs page size.  Even on x86, that can be
> >> > changed and _could_ be 1G.  If hugetlbfs is configured out, you also
> >> > won't get this in meminfo.
> > I think Aneesh propose to add one more line into the file.
> 
> Ahhh, ok...
> 
> Personally, I think Hugh did the right things.  There's no reason to
> waste cycles sticking a number in meminfo that never changes.

Thanks, yes, that was my feeling: I prefer not to clutter /proc/meminfo
with constants - especially not a unit, or granularity, as this is
(too late to stop Hugepagesize of course, but I wish it weren't there,
and those HugePages_ numbers in kB).

On top of that, /proc/meminfo is the mm admin's "front page", whereas
this is a low-level detail: it somewhat goes against our claim of
"transparent" hugepages to post it up there, even though a few tests
etc may find the value useful: I'd rather bury it among the tunables.
(My guess is that Andrea felt rather the same, in choosing not to
publish it six years ago.)

But of course, that's just my/our opinion: any strong preferences?

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Michal Hocko]

On Mon 05-12-16 22:05:22, Hugh Dickins wrote:
> Test programs want to know the size of a transparent hugepage.
> While it is commonly the same as the size of a hugetlbfs page
> (shown as Hugepagesize in /proc/meminfo), that is not always so:
> powerpc implements transparent hugepages in a different way from
> hugetlbfs pages, so it's coincidence when their sizes are the same;
> and x86 and others can support more than one hugetlbfs page size.
> 
> Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
> THP size in bytes - it's the same for Anonymous and Shmem hugepages.
> Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
> in case some transparent support for pud and pgd pages is added later.

Definitely much better than cluttering /proc/meminfo even more.

> Signed-off-by: Hugh Dickins <hughd@google.com>

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
> 
>  Documentation/vm/transhuge.txt |    5 +++++
>  mm/huge_memory.c               |   10 ++++++++++
>  2 files changed, 15 insertions(+)
> 
> --- 4.9-rc8/Documentation/vm/transhuge.txt	2016-10-02 16:24:33.000000000 -0700
> +++ linux/Documentation/vm/transhuge.txt	2016-12-05 20:55:12.142578631 -0800
> @@ -136,6 +136,11 @@ or enable it back by writing 1:
>  echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
>  echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
>  
> +Some userspace (such as a test program, or an optimized memory allocation
> +library) may want to know the size (in bytes) of a transparent hugepage:
> +
> +cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
> +
>  khugepaged will be automatically started when
>  transparent_hugepage/enabled is set to "always" or "madvise, and it'll
>  be automatically shutdown if it's set to "never".
> --- 4.9-rc8/mm/huge_memory.c	2016-12-04 16:42:39.881703357 -0800
> +++ linux/mm/huge_memory.c	2016-12-05 20:58:19.953010005 -0800
> @@ -285,6 +285,15 @@ static ssize_t use_zero_page_store(struc
>  }
>  static struct kobj_attribute use_zero_page_attr =
>  	__ATTR(use_zero_page, 0644, use_zero_page_show, use_zero_page_store);
> +
> +static ssize_t hpage_pmd_size_show(struct kobject *kobj,
> +		struct kobj_attribute *attr, char *buf)
> +{
> +	return sprintf(buf, "%lu\n", HPAGE_PMD_SIZE);
> +}
> +static struct kobj_attribute hpage_pmd_size_attr =
> +	__ATTR_RO(hpage_pmd_size);
> +
>  #ifdef CONFIG_DEBUG_VM
>  static ssize_t debug_cow_show(struct kobject *kobj,
>  				struct kobj_attribute *attr, char *buf)
> @@ -307,6 +316,7 @@ static struct attribute *hugepage_attr[]
>  	&enabled_attr.attr,
>  	&defrag_attr.attr,
>  	&use_zero_page_attr.attr,
> +	&hpage_pmd_size_attr.attr,
>  #if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE)
>  	&shmem_enabled_attr.attr,
>  #endif
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: make transparent hugepage size public

[Vlastimil Babka]

[+CC linux-api]

On 12/07/2016 08:50 AM, Michal Hocko wrote:
> On Mon 05-12-16 22:05:22, Hugh Dickins wrote:
>> Test programs want to know the size of a transparent hugepage.
>> While it is commonly the same as the size of a hugetlbfs page
>> (shown as Hugepagesize in /proc/meminfo), that is not always so:
>> powerpc implements transparent hugepages in a different way from
>> hugetlbfs pages, so it's coincidence when their sizes are the same;
>> and x86 and others can support more than one hugetlbfs page size.
>>
>> Add /sys/kernel/mm/transparent_hugepage/hpage_pmd_size to show the
>> THP size in bytes - it's the same for Anonymous and Shmem hugepages.
>> Call it hpage_pmd_size (after HPAGE_PMD_SIZE) rather than hpage_size,
>> in case some transparent support for pud and pgd pages is added later.

Let's hope no arch is crazy enough to add intermediate sizes for THP,
like they have for hugetlbfs? (See Dave's "proc: mm: export PTE sizes
directly in smaps" threads).

> 
> Definitely much better than cluttering /proc/meminfo even more.

Agreed.

>> Signed-off-by: Hugh Dickins <hughd@google.com>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> 
> Acked-by: Michal Hocko <mhocko@suse.com>
> 
>> ---
>>
>>  Documentation/vm/transhuge.txt |    5 +++++
>>  mm/huge_memory.c               |   10 ++++++++++
>>  2 files changed, 15 insertions(+)
>>
>> --- 4.9-rc8/Documentation/vm/transhuge.txt	2016-10-02 16:24:33.000000000 -0700
>> +++ linux/Documentation/vm/transhuge.txt	2016-12-05 20:55:12.142578631 -0800
>> @@ -136,6 +136,11 @@ or enable it back by writing 1:
>>  echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
>>  echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
>>  
>> +Some userspace (such as a test program, or an optimized memory allocation
>> +library) may want to know the size (in bytes) of a transparent hugepage:
>> +
>> +cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
>> +
>>  khugepaged will be automatically started when
>>  transparent_hugepage/enabled is set to "always" or "madvise, and it'll
>>  be automatically shutdown if it's set to "never".
>> --- 4.9-rc8/mm/huge_memory.c	2016-12-04 16:42:39.881703357 -0800
>> +++ linux/mm/huge_memory.c	2016-12-05 20:58:19.953010005 -0800
>> @@ -285,6 +285,15 @@ static ssize_t use_zero_page_store(struc
>>  }
>>  static struct kobj_attribute use_zero_page_attr =
>>  	__ATTR(use_zero_page, 0644, use_zero_page_show, use_zero_page_store);
>> +
>> +static ssize_t hpage_pmd_size_show(struct kobject *kobj,
>> +		struct kobj_attribute *attr, char *buf)
>> +{
>> +	return sprintf(buf, "%lu\n", HPAGE_PMD_SIZE);
>> +}
>> +static struct kobj_attribute hpage_pmd_size_attr =
>> +	__ATTR_RO(hpage_pmd_size);
>> +
>>  #ifdef CONFIG_DEBUG_VM
>>  static ssize_t debug_cow_show(struct kobject *kobj,
>>  				struct kobj_attribute *attr, char *buf)
>> @@ -307,6 +316,7 @@ static struct attribute *hugepage_attr[]
>>  	&enabled_attr.attr,
>>  	&defrag_attr.attr,
>>  	&use_zero_page_attr.attr,
>> +	&hpage_pmd_size_attr.attr,
>>  #if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE)
>>  	&shmem_enabled_attr.attr,
>>  #endif
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>