All of lore.kernel.org
 help / color / mirror / Atom feed
From: Mike Kravetz <mike.kravetz@oracle.com>
To: "Jörn Engel" <joern@purestorage.com>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
	linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: hugetlb pages not accounted for in rss
Date: Tue, 28 Jul 2015 14:15:36 -0700	[thread overview]
Message-ID: <55B7F0F8.8080909@oracle.com> (raw)
In-Reply-To: <20150728183248.GB1406@Sligo.logfs.org>

On 07/28/2015 11:32 AM, Jorn Engel wrote:
> On Mon, Jul 27, 2015 at 04:26:47PM -0700, Mike Kravetz wrote:
>> I started looking at the hugetlb self tests.  The test hugetlbfstest
>> expects hugetlb pages to be accounted for in rss.  However, there is
>> no code in the kernel to do this accounting.
>>
>> It looks like there was an effort to add the accounting back in 2013.
>> The test program made it into tree, but the accounting code did not.
>
> My apologies.  Upstream work always gets axed first when I run out of
> time - which happens more often than not.

No worries, I just noticed the inconsistency of the test program and
no supporting code in the kernel.

>> The easiest way to resolve this issue would be to remove the test and
>> perhaps document that hugetlb pages are not accounted for in rss.
>> However, it does seem like a big oversight that hugetlb pages are not
>> accounted for in rss.  From a quick scan of the code it appears THP
>> pages are properly accounted for.
>>
>> Thoughts?
>
> Unsurprisingly I agree that hugepages should count towards rss.  Keeping
> the test in keeps us honest.  Actually fixing the issue would make us
> honest and correct.
>
> Increasingly we have tiny processes (by rss) that actually consume large
> fractions of total memory.  Makes rss somewhat useless as a measure of
> anything.

I'll take a look at what it would take to get the accounting in place.
-- 
Mike Kravetz

>
> Jorn
>
> --
> Consensus is no proof!
> -- John Naisbitt
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

WARNING: multiple messages have this Message-ID (diff)
From: Mike Kravetz <mike.kravetz@oracle.com>
To: "Jörn Engel" <joern@purestorage.com>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
	linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: hugetlb pages not accounted for in rss
Date: Tue, 28 Jul 2015 14:15:36 -0700	[thread overview]
Message-ID: <55B7F0F8.8080909@oracle.com> (raw)
In-Reply-To: <20150728183248.GB1406@Sligo.logfs.org>

On 07/28/2015 11:32 AM, Jörn Engel wrote:
> On Mon, Jul 27, 2015 at 04:26:47PM -0700, Mike Kravetz wrote:
>> I started looking at the hugetlb self tests.  The test hugetlbfstest
>> expects hugetlb pages to be accounted for in rss.  However, there is
>> no code in the kernel to do this accounting.
>>
>> It looks like there was an effort to add the accounting back in 2013.
>> The test program made it into tree, but the accounting code did not.
>
> My apologies.  Upstream work always gets axed first when I run out of
> time - which happens more often than not.

No worries, I just noticed the inconsistency of the test program and
no supporting code in the kernel.

>> The easiest way to resolve this issue would be to remove the test and
>> perhaps document that hugetlb pages are not accounted for in rss.
>> However, it does seem like a big oversight that hugetlb pages are not
>> accounted for in rss.  From a quick scan of the code it appears THP
>> pages are properly accounted for.
>>
>> Thoughts?
>
> Unsurprisingly I agree that hugepages should count towards rss.  Keeping
> the test in keeps us honest.  Actually fixing the issue would make us
> honest and correct.
>
> Increasingly we have tiny processes (by rss) that actually consume large
> fractions of total memory.  Makes rss somewhat useless as a measure of
> anything.

I'll take a look at what it would take to get the accounting in place.
-- 
Mike Kravetz

>
> Jörn
>
> --
> Consensus is no proof!
> -- John Naisbitt
>

  reply	other threads:[~2015-07-28 21:15 UTC|newest]

Thread overview: 142+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-07-27 23:26 hugetlb pages not accounted for in rss Mike Kravetz
2015-07-27 23:26 ` Mike Kravetz
2015-07-28 18:32 ` Jörn Engel
2015-07-28 18:32   ` Jörn Engel
2015-07-28 21:15   ` Mike Kravetz [this message]
2015-07-28 21:15     ` Mike Kravetz
2015-07-28 22:15     ` David Rientjes
2015-07-28 22:15       ` David Rientjes
2015-07-28 22:26       ` Jörn Engel
2015-07-28 22:26         ` Jörn Engel
2015-07-28 23:30         ` David Rientjes
2015-07-28 23:30           ` David Rientjes
2015-07-29  0:53           ` Jörn Engel
2015-07-29  0:53             ` Jörn Engel
2015-07-29 19:08             ` David Rientjes
2015-07-29 19:08               ` David Rientjes
2015-07-29 23:20               ` Mike Kravetz
2015-07-29 23:20                 ` Mike Kravetz
2015-07-30 21:34                 ` Jörn Engel
2015-07-30 21:34                   ` Jörn Engel
2015-07-31 21:09                   ` David Rientjes
2015-07-31 21:09                     ` David Rientjes
2015-08-04  2:55                 ` Naoya Horiguchi
2015-08-04  2:55                   ` Naoya Horiguchi
2015-08-04  5:13                   ` [PATCH] smaps: fill missing fields for vma(VM_HUGETLB) Naoya Horiguchi
2015-08-04  5:13                     ` Naoya Horiguchi
2015-08-04 18:21                     ` Jörn Engel
2015-08-04 18:21                       ` Jörn Engel
2015-08-06  2:18                       ` David Rientjes
2015-08-06  2:18                         ` David Rientjes
2015-08-06  7:44                         ` Naoya Horiguchi
2015-08-06  7:44                           ` Naoya Horiguchi
2015-08-07  7:24                           ` [PATCH v2 0/2] hugetlb: display per-process/per-vma usage Naoya Horiguchi
2015-08-07  7:24                             ` Naoya Horiguchi
2015-08-07  7:24                             ` [PATCH v2 2/2] mm: hugetlb: add VmHugetlbRSS: field in /proc/pid/status Naoya Horiguchi
2015-08-07  7:24                               ` Naoya Horiguchi
2015-08-07 22:55                               ` Andrew Morton
2015-08-07 22:55                                 ` Andrew Morton
2015-08-10  0:47                                 ` Naoya Horiguchi
2015-08-10  0:47                                   ` Naoya Horiguchi
2015-08-10  0:47                                   ` [PATCH v3 1/3] smaps: fill missing fields for vma(VM_HUGETLB) Naoya Horiguchi
2015-08-10  0:47                                     ` Naoya Horiguchi
2015-08-10  0:47                                   ` [PATCH v3 3/3] Documentation/filesystems/proc.txt: document hugetlb RSS Naoya Horiguchi
2015-08-10  0:47                                     ` Naoya Horiguchi
2015-08-11  0:44                                     ` David Rientjes
2015-08-11  0:44                                       ` David Rientjes
2015-08-12  0:03                                       ` Naoya Horiguchi
2015-08-12  0:03                                         ` Naoya Horiguchi
2015-08-12  7:45                                         ` [PATCH v4 1/2] mm: hugetlb: proc: add HugetlbPages field to /proc/PID/smaps Naoya Horiguchi
2015-08-12  7:45                                           ` Naoya Horiguchi
2015-08-12  7:45                                           ` [PATCH v4 2/2] mm: hugetlb: proc: add HugetlbPages field to /proc/PID/status Naoya Horiguchi
2015-08-12  7:45                                             ` Naoya Horiguchi
2015-08-12 20:30                                             ` David Rientjes
2015-08-12 20:30                                               ` David Rientjes
2015-08-13  0:45                                               ` Naoya Horiguchi
2015-08-13  0:45                                                 ` Naoya Horiguchi
2015-08-13 21:14                                                 ` Jörn Engel
2015-08-13 21:14                                                   ` Jörn Engel
2015-08-13 21:13                                               ` Jörn Engel
2015-08-13 21:13                                                 ` Jörn Engel
2015-08-17 21:28                                               ` David Rientjes
2015-08-17 21:28                                                 ` David Rientjes
2015-08-12 20:25                                           ` [PATCH v4 1/2] mm: hugetlb: proc: add HugetlbPages field to /proc/PID/smaps David Rientjes
2015-08-12 20:25                                             ` David Rientjes
2015-08-13 21:14                                             ` Jörn Engel
2015-08-13 21:14                                               ` Jörn Engel
2015-08-20  8:26                                         ` [PATCH v5 0/2] hugetlb: display per-process/per-vma usage Naoya Horiguchi
2015-08-20  8:26                                           ` Naoya Horiguchi
2015-08-20  8:26                                           ` [PATCH v5 1/2] mm: hugetlb: proc: add HugetlbPages field to /proc/PID/smaps Naoya Horiguchi
2015-08-20  8:26                                             ` Naoya Horiguchi
2015-08-20 10:49                                             ` Michal Hocko
2015-08-20 10:49                                               ` Michal Hocko
2015-08-20 23:20                                               ` Naoya Horiguchi
2015-08-20 23:20                                                 ` Naoya Horiguchi
2015-08-21  6:33                                                 ` Michal Hocko
2015-08-21  6:33                                                   ` Michal Hocko
2015-09-07  1:29                                             ` Pádraig Brady
2015-09-07  1:29                                               ` Pádraig Brady
2015-09-07  2:23                                               ` Naoya Horiguchi
2015-09-07  2:23                                                 ` Naoya Horiguchi
2015-09-07  6:46                                                 ` Naoya Horiguchi
2015-09-07  6:46                                                   ` Naoya Horiguchi
2015-09-07  9:52                                                   ` Pádraig Brady
2015-09-07  9:52                                                     ` Pádraig Brady
2015-09-07 10:52                                                     ` Pádraig Brady
2015-09-07 10:52                                                       ` Pádraig Brady
2015-09-17  9:39                                                       ` Naoya Horiguchi
2015-09-17  9:39                                                         ` Naoya Horiguchi
2015-09-09 15:12                                             ` Vlastimil Babka
2015-09-09 15:12                                               ` Vlastimil Babka
2015-09-09 22:14                                               ` David Rientjes
2015-09-09 22:14                                                 ` David Rientjes
2015-08-20  8:26                                           ` [PATCH v5 2/2] mm: hugetlb: proc: add HugetlbPages field to /proc/PID/status Naoya Horiguchi
2015-08-20  8:26                                             ` Naoya Horiguchi
2015-08-20 11:00                                             ` Michal Hocko
2015-08-20 11:00                                               ` Michal Hocko
2015-08-20 19:49                                               ` David Rientjes
2015-08-20 19:49                                                 ` David Rientjes
2015-08-21  6:32                                                 ` Michal Hocko
2015-08-21  6:32                                                   ` Michal Hocko
2015-08-21 16:38                                                   ` Jörn Engel
2015-08-21 16:38                                                     ` Jörn Engel
2015-08-20 23:34                                               ` Naoya Horiguchi
2015-08-20 23:34                                                 ` Naoya Horiguchi
2015-08-21  6:53                                                 ` Michal Hocko
2015-08-21  6:53                                                   ` Michal Hocko
2015-08-21 16:30                                                   ` Jörn Engel
2015-08-21 16:30                                                     ` Jörn Engel
2015-08-24  8:51                                                     ` Michal Hocko
2015-08-24  8:51                                                       ` Michal Hocko
2015-08-25 23:23                                                       ` David Rientjes
2015-08-25 23:23                                                         ` David Rientjes
2015-08-26  6:38                                                         ` Michal Hocko
2015-08-26  6:38                                                           ` Michal Hocko
2015-08-26 22:02                                                           ` David Rientjes
2015-08-26 22:02                                                             ` David Rientjes
2015-08-27  6:48                                                             ` Michal Hocko
2015-08-27  6:48                                                               ` Michal Hocko
2015-08-27 17:23                                                               ` Jörn Engel
2015-08-27 17:23                                                                 ` Jörn Engel
2015-08-27 20:44                                                                 ` David Rientjes
2015-08-27 20:44                                                                   ` David Rientjes
2015-08-31  9:12                                                                 ` Michal Hocko
2015-08-31  9:12                                                                   ` Michal Hocko
2015-09-16  0:21                                         ` [PATCH v1] mm: migrate: hugetlb: putback destination hugepage to active list Naoya Horiguchi
2015-09-16  0:21                                           ` Naoya Horiguchi
2015-09-16  2:53                                           ` Naoya Horiguchi
2015-09-16  2:53                                             ` Naoya Horiguchi
2015-08-10  0:47                                   ` [PATCH v3 2/3] mm: hugetlb: add VmHugetlbRSS: field in /proc/pid/status Naoya Horiguchi
2015-08-10  0:47                                     ` Naoya Horiguchi
2015-08-10  1:16                                     ` Naoya Horiguchi
2015-08-10  1:16                                       ` Naoya Horiguchi
2015-08-07  7:24                             ` [PATCH v2 1/2] smaps: fill missing fields for vma(VM_HUGETLB) Naoya Horiguchi
2015-08-07  7:24                               ` Naoya Horiguchi
2015-08-07 22:50                               ` Andrew Morton
2015-08-07 22:50                                 ` Andrew Morton
2015-08-11  0:37                               ` David Rientjes
2015-08-11  0:37                                 ` David Rientjes
2015-08-11 23:32                                 ` Naoya Horiguchi
2015-08-11 23:32                                   ` Naoya Horiguchi
2015-08-11 23:48                                   ` David Rientjes
2015-08-11 23:48                                     ` David Rientjes

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=55B7F0F8.8080909@oracle.com \
    --to=mike.kravetz@oracle.com \
    --cc=joern@purestorage.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.