[PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Fengguang Wu]

The "proportional set size" (PSS) of a process is the count of pages it has in
memory, where each page is divided by the number of processes sharing it. So if
a process has 1000 pages all to itself, and 1000 shared with one other process,
its PSS will be 1500.
               - lwn.net: "ELC: How much memory are applications really using?"

The PSS proposed by Matt Mackall is a very nice metic for measuring an process's
memory footprint. So collect and export it via /proc/<pid>/smaps.

Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
They are comprehensive tools. But for PSS, let's do it in the simple way. 

Cc: John Berthels <jjberthels@gmail.com>
Cc: Balbir Singh <balbir@linux.vnet.ibm.com>
Cc: Denys Vlasenko <vda.linux@googlemail.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/proc/task_mmu.c |   29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

--- linux-2.6.23-rc4-mm1.orig/fs/proc/task_mmu.c
+++ linux-2.6.23-rc4-mm1/fs/proc/task_mmu.c
@@ -324,6 +324,27 @@ struct mem_size_stats
 	unsigned long private_clean;
 	unsigned long private_dirty;
 	unsigned long referenced;
+
+	/*
+	 * Proportional Set Size(PSS): my share of RSS.
+	 *
+	 * PSS of a process is the count of pages it has in memory, where each
+	 * page is divided by the number of processes sharing it.  So if a
+	 * process has 1000 pages all to itself, and 1000 shared with one other
+	 * process, its PSS will be 1500.               - Matt Mackall, lwn.net
+	 */
+	u64 	      pss;
+	/*
+	 * To keep (accumulated) division errors low, we adopt 64bit pss and
+	 * use some low bits for division errors. So (pss >> PSS_DIV_BITS)
+	 * would be the real byte count.
+	 *
+	 * A shift of 12 before division means(assuming 4K page size):
+	 * 	- 1M 3-user-pages add up to 8KB errors;
+	 * 	- supports mapcount up to 2^24, or 16M;
+	 * 	- supports PSS up to 2^52 bytes, or 4PB.
+	 */
+#define PSS_DIV_BITS	12
 };
 
 struct smaps_arg
@@ -341,6 +362,7 @@ static int smaps_pte_range(pmd_t *pmd, u
 	pte_t *pte, ptent;
 	spinlock_t *ptl;
 	struct page *page;
+	int mapcount;
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	for (; addr != end; pte++, addr += PAGE_SIZE) {
@@ -357,16 +379,19 @@ static int smaps_pte_range(pmd_t *pmd, u
 		/* Accumulate the size in pages that have been accessed. */
 		if (pte_young(ptent) || PageReferenced(page))
 			mss->referenced += PAGE_SIZE;
-		if (page_mapcount(page) >= 2) {
+		mapcount = page_mapcount(page);
+		if (mapcount >= 2) {
 			if (pte_dirty(ptent))
 				mss->shared_dirty += PAGE_SIZE;
 			else
 				mss->shared_clean += PAGE_SIZE;
+			mss->pss += (PAGE_SIZE << PSS_DIV_BITS) / mapcount;
 		} else {
 			if (pte_dirty(ptent))
 				mss->private_dirty += PAGE_SIZE;
 			else
 				mss->private_clean += PAGE_SIZE;
+			mss->pss += (PAGE_SIZE << PSS_DIV_BITS);
 		}
 	}
 	pte_unmap_unlock(pte - 1, ptl);
@@ -395,6 +420,7 @@ static int show_smap(struct seq_file *m,
 	seq_printf(m,
 		   "Size:           %8lu kB\n"
 		   "Rss:            %8lu kB\n"
+		   "Pss:            %8lu kB\n"
 		   "Shared_Clean:   %8lu kB\n"
 		   "Shared_Dirty:   %8lu kB\n"
 		   "Private_Clean:  %8lu kB\n"
@@ -402,6 +428,7 @@ static int show_smap(struct seq_file *m,
 		   "Referenced:     %8lu kB\n",
 		   (vma->vm_end - vma->vm_start) >> 10,
 		   sarg.mss.resident >> 10,
+		   (unsigned long)(sarg.mss.pss >> (10 + PSS_DIV_BITS)),
 		   sarg.mss.shared_clean  >> 10,
 		   sarg.mss.shared_dirty  >> 10,
 		   sarg.mss.private_clean >> 10,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Andrew Morton]

On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:

> Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
> They are comprehensive tools. But for PSS, let's do it in the simple way. 

right.  I'm rather reluctant to merge anything which could have been done from
userspace via the maps2 interfaces.

See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
we had that, you might have implemented this as a little proglet which parses
the maps2 files.  But we don't have that, so you ended up doing it in-kernel.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Userspace tools (was Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps)

[Balbir Singh]

Andrew Morton wrote:
> On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> 
>> Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
>> They are comprehensive tools. But for PSS, let's do it in the simple way. 
> 
> right.  I'm rather reluctant to merge anything which could have been done from
> userspace via the maps2 interfaces.
> 
> See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
> we had that, you might have implemented this as a little proglet which parses
> the maps2 files.  But we don't have that, so you ended up doing it in-kernel.

Andrew, I second the userspace-tools idea. I would also add an FAQ in
that directory, explaining what problem each tool solves. I think your
page cache control program would be a great example of something to put
in there.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Userspace tools (was Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps)

[Christoph Hellwig]

On Mon, Sep 17, 2007 at 12:38:50PM +0530, Balbir Singh wrote:
> Andrew Morton wrote:
> > On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> > 
> >> Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
> >> They are comprehensive tools. But for PSS, let's do it in the simple way. 
> > 
> > right.  I'm rather reluctant to merge anything which could have been done from
> > userspace via the maps2 interfaces.
> > 
> > See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
> > we had that, you might have implemented this as a little proglet which parses
> > the maps2 files.  But we don't have that, so you ended up doing it in-kernel.
> 
> Andrew, I second the userspace-tools idea. I would also add an FAQ in
> that directory, explaining what problem each tool solves. I think your
> page cache control program would be a great example of something to put
> in there.

It's called the util-linux package.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Userspace tools (was Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps)

[Balbir Singh]

Christoph Hellwig wrote:
> On Mon, Sep 17, 2007 at 12:38:50PM +0530, Balbir Singh wrote:
>> Andrew Morton wrote:
>>> On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
>>>
>>>> Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
>>>> They are comprehensive tools. But for PSS, let's do it in the simple way. 
>>> right.  I'm rather reluctant to merge anything which could have been done from
>>> userspace via the maps2 interfaces.
>>>
>>> See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
>>> we had that, you might have implemented this as a little proglet which parses
>>> the maps2 files.  But we don't have that, so you ended up doing it in-kernel.
>> Andrew, I second the userspace-tools idea. I would also add an FAQ in
>> that directory, explaining what problem each tool solves. I think your
>> page cache control program would be a great example of something to put
>> in there.
> 
> It's called the util-linux package.

I am looking at http://www.kernel.org/pub/linux/utils/util-linux/ and
the last release happened in September 2005. May be I am looking in
the wrong place.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Userspace tools (was Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps)

[Christoph Hellwig]

On Mon, Sep 17, 2007 at 02:41:31PM +0530, Balbir Singh wrote:
> > It's called the util-linux package.
> 
> I am looking at http://www.kernel.org/pub/linux/utils/util-linux/ and
> the last release happened in September 2005. May be I am looking in
> the wrong place.

http://userweb.kernel.org/~kzak/util-linux-ng/

Karel,
	any chance to put the new util-linux release into the place above
	aswell.  (And maybe strip the silly -ng postfix)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Userspace tools

[Karel Zak]

On Mon, Sep 17, 2007 at 10:15:27AM +0100, Christoph Hellwig wrote:
> On Mon, Sep 17, 2007 at 02:41:31PM +0530, Balbir Singh wrote:
> > > It's called the util-linux package.
> > 
> > I am looking at http://www.kernel.org/pub/linux/utils/util-linux/ and
> > the last release happened in September 2005. May be I am looking in
> > the wrong place.
> 
> http://userweb.kernel.org/~kzak/util-linux-ng/
> 
> Karel,
> 	any chance to put the new util-linux release into the place above
> 	aswell.  (And maybe strip the silly -ng postfix)

 Ask Adrian Bunk who ignores all mails about util-linux in last two
 years. For more details see thread:

    http://lkml.org/lkml/2006/12/20/130

 ... it's old and stupid story. Maybe we can try to restart this
 discussion. I have not a clue, how deep is Adrian's hibernation...

    Karel

-- 
 Karel Zak  <kzak@redhat.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Userspace tools (was Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps)

[Andrew Morton]

On Mon, 17 Sep 2007 10:00:57 +0100 Christoph Hellwig <hch@infradead.org> wrote:

> On Mon, Sep 17, 2007 at 12:38:50PM +0530, Balbir Singh wrote:
> > Andrew Morton wrote:
> > > On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> > > 
> > >> Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
> > >> They are comprehensive tools. But for PSS, let's do it in the simple way. 
> > > 
> > > right.  I'm rather reluctant to merge anything which could have been done from
> > > userspace via the maps2 interfaces.
> > > 
> > > See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
> > > we had that, you might have implemented this as a little proglet which parses
> > > the maps2 files.  But we don't have that, so you ended up doing it in-kernel.
> > 
> > Andrew, I second the userspace-tools idea. I would also add an FAQ in
> > that directory, explaining what problem each tool solves. I think your
> > page cache control program would be a great example of something to put
> > in there.
> 
> It's called the util-linux package.

That's the theory.  I'd be interested in seeing someone test it out: submit
something and see if/when it goes in and gets redistributed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Fengguang Wu]

On Sun, Sep 16, 2007 at 11:51:20PM -0700, Andrew Morton wrote:
> On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> 
> > Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
> > They are comprehensive tools. But for PSS, let's do it in the simple way. 
> 
> right.  I'm rather reluctant to merge anything which could have been done from
> userspace via the maps2 interfaces.

Agreed.  My thought was that PSS will be used far more widely than
other maps2 memory analysis tools. Providing PSS in smaps could
possibly help simplify many ps/top like tools.

> See, this is why I think the kernel needs a ./userspace-tools/ directory.  If
> we had that, you might have implemented this as a little proglet which parses
> the maps2 files.  But we don't have that, so you ended up doing it in-kernel.

Because Matt didn't put it there ;-)

I wholly agree that it is a good practice to carry some user space
tools with the kernel as the reference implementations. We already
have some in the Document/ and usr/ directories.

Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Matt Mackall]

On Sun, Sep 16, 2007 at 11:51:20PM -0700, Andrew Morton wrote:
> On Mon, 17 Sep 2007 10:40:54 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn> wrote:
> 
> > Matt Mackall's pagemap/kpagemap and John Berthels's exmap can also do the job.
> > They are comprehensive tools. But for PSS, let's do it in the simple way. 
> 
> right.  I'm rather reluctant to merge anything which could have been done from
> userspace via the maps2 interfaces.

It can be done via maps2, but it's considerably less efficient. So if
this particular metric is a useful one, we ought to consider putting
it in-kernel.

Now is this PSS metric that useful? I'd argue yes. By comparison, the
RSS and VSS numbers are of basically no help in answering the most
common question about memory usage: "how much memory is my process
actually using?" Most people fall back to RSS and a bunch of
hand-waving, but that tends to fall over because typically sum(RSS) >
available RAM (sometimes by an order of magnitude), leaving users
completely confused. This was perhaps the biggest complaint from one
of the people on the Kernel Summit user panel, by the way. The PSS
numbers always sum to used RAM and give a fairly intuitive accounting
of shared mappings.

The big downside to PSS is that it's expensive to track. We have to
either visit each page when we report the count or we have to update
each PSS counter when we change the use count on a shared page. There
might be some tricks we can pull here but RSS and VSS, on the other
hand, are effectively O(1). An efficient in-kernel PSS calculator
might be a little painful if used in something like top(1), but the
map2 approach definitely won't be fast enough here.

Also, there's a second number we should be reporting at the same
time, which I've been calling USS (unique or unshared set size), which
is the size of the unshared pages. This is, for example, the amount of
memory that will get freed when you kill one of 20 Apache threads, or,
alternately, the amount of memory that adding another one will consume.

-- 
Mathematics is the supreme nostalgia of our time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Denys Vlasenko]

Hi Matt,

On Monday 17 September 2007 17:10, Matt Mackall wrote:
> Also, there's a second number we should be reporting at the same
> time, which I've been calling USS (unique or unshared set size), which
> is the size of the unshared pages. This is, for example, the amount of
> memory that will get freed when you kill one of 20 Apache threads, or,
> alternately, the amount of memory that adding another one will consume.

USS is already there, smaps already gives you that.

If you read entire smaps "file" and sum up all numbers there:

Shared_Clean: N
Shared_Dirty: N
Private_Clean: N
Private_Dirty: N

Then you can calculate the following (among other useful things):

rss_sh   - sum of (shared_clean + shared_dirty)
uss      - sum of (private_clean + private_dirty) <=== HERE
rss      - uss + rss_sh

PSS, on the other hand, cannot be inferred from this data,
so please push it into mainline.
--
vda

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[John Berthels]

On 17/09/2007, Matt Mackall <mpm@selenic.com> wrote:
> The big downside to PSS is that it's expensive to track. We have to
> either visit each page when we report the count or we have to update
> each PSS counter when we change the use count on a shared page. There
> might be some tricks we can pull here but RSS and VSS, on the other
> hand, are effectively O(1). An efficient in-kernel PSS calculator
> might be a little painful if used in something like top(1), but the
> map2 approach definitely won't be fast enough here.

This is the advantage of exmap/pagemap. You don't pay any runtime penalty.

Personally, I'd probably rather make do with RSS and VSS to see which
processes are 'large'. There usually aren't many.

If there's a significant/measurable penalty to tracking PSS in kernel
I'd say why bother? It's a very useful number *when you are interested
in it*. On those occasions you can fire up the more complex tool and
pay the time to walk the page tables. It's not a very dynamic
quantity, so a snapshotting approach works well.

Also exmap (I don't know if pagemap does this) grovels through ELF and
/proc/<pid>/maps so you can see which section+symbol of your shared
lib is hurting you. You're generally going to want this info in order
to do anything about bad PSS numbers, so I'm not sure raw PSS numbers
are directly useful.

Is map2 -mm tree only (I didn't get anything on a grep of mainline
2.6.22.6)? Sorry, I'm a bit out of touch. If I could drop the kernel
module from exmap and use an existing interface that would be great.

jb

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH][RESEND] maps: PSS(proportional set size) accounting in smaps

[Fengguang Wu]

On Wed, Sep 19, 2007 at 09:37:21AM +0100, John Berthels wrote:
[...]
> Also exmap (I don't know if pagemap does this) grovels through ELF and
> /proc/<pid>/maps so you can see which section+symbol of your shared
> lib is hurting you. You're generally going to want this info in order
> to do anything about bad PSS numbers, so I'm not sure raw PSS numbers
> are directly useful.

Basically,
- getting the list of used/unused symbols are great for developers;
- getting the list of applications with their PSS numbers are good for users.

One is for 'analysis' and another is about 'accounting'.

> Is map2 -mm tree only (I didn't get anything on a grep of mainline
> 2.6.22.6)? Sorry, I'm a bit out of touch. If I could drop the kernel
> module from exmap and use an existing interface that would be great.

Yes.
It could be the right time for an early tryout and early feedbacks ;-)

Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>