How do you accurately determine a process' RAM usage?

How do you accurately determine a process' RAM usage?

[P]

I wrote a tool to report how much RAM a
particular program (apache for e.g.) was using:
http://www.pixelbeat.org/scripts/ps_mem.py

I was then pointed at the following:
http://wiki.apache.org/spamassassin/TopSharedMemoryBug
which describes how copy-on-write pages are
not counted as shared since 2.6.

So how can one determine how much RAM a process is using?
Seems like a fundamental requirement to me.
Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?

Here's a little python script to demo the problem:

#!/usr/bin/python
import os, sys, time
a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
     time.sleep(1)
     sys.exit()
sizel=map(float,open("/proc/"+str(pid)+"/statm").readlines()[0].split()[1:3])
print "only %.1f%% reported as shared" % ((sizel[1]/sizel[0])*100)
os.wait()

-- 
Pádraig Brady - http://www.pixelbeat.org
--

Re: How do you accurately determine a process' RAM usage?

[Andrew Morton]

P@draigBrady.com wrote:
>
> I wrote a tool to report how much RAM a
>  particular program (apache for e.g.) was using:
>  http://www.pixelbeat.org/scripts/ps_mem.py
> 
>  I was then pointed at the following:
>  http://wiki.apache.org/spamassassin/TopSharedMemoryBug
>  which describes how copy-on-write pages are
>  not counted as shared since 2.6.
> 
>  So how can one determine how much RAM a process is using?
>  Seems like a fundamental requirement to me.
>  Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?

Calculating this stuff accurately is very expensive.  You'll get a better
answer using proc-pid-smaps.patch from -mm, but even that won't tell you
things about sharing levels of the pages.

Re: How do you accurately determine a process' RAM usage?

[P]

Andrew Morton wrote:
> Calculating this stuff accurately is very expensive.  You'll get a better
> answer using proc-pid-smaps.patch from -mm, but even that won't tell you
> things about sharing levels of the pages.

Great, thanks! I'll play around with this:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch
Looks like it's been stable for 4 months?

Given that it's an independent /proc/$pid/smaps file,
it only needs to be queried when required and so
I wouldn't worry too much about cost. `top` wouldn't use it
for e.g., but specialised tools like mine would.

-- 
Pádraig Brady - http://www.pixelbeat.org
--

Re: How do you accurately determine a process' RAM usage?

[Andrew Morton]

P@draigBrady.com wrote:
>
> Andrew Morton wrote:
> > Calculating this stuff accurately is very expensive.  You'll get a better
> > answer using proc-pid-smaps.patch from -mm, but even that won't tell you
> > things about sharing levels of the pages.
> 
> Great, thanks! I'll play around with this:
> http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch

OK, please let us know how it goes.

> Looks like it's been stable for 4 months?

yup, although I don't think it's been used much.

> Given that it's an independent /proc/$pid/smaps file,
> it only needs to be queried when required and so
> I wouldn't worry too much about cost. `top` wouldn't use it
> for e.g., but specialised tools like mine would.

I agree, but people get upset ;)

Plus some userspace tool developer might see it and start using it without
knowing the cost on big iron.

Re: How do you accurately determine a process' RAM usage?

[Hugh Dickins]

On Thu, 7 Jul 2005, Andrew Morton wrote:
> P@draigBrady.com wrote:
> 
> > Looks like it's been stable for 4 months?
> 
> yup, although I don't think it's been used much.

Just a sidenote to say I should be sending you an update to it
(the /proc/$pid/smaps code) in the next couple of days, but merely
cosmetic ("map" -> "vma", cond_resched_lock, p?d_addr_next ptwalking).

I'm still pretty sceptical about it, but it's probably a useful
framework for people to hack on, to report whatever kind of page
numbers they're interested in for this or that investigation.

Hugh

Re: How do you accurately determine a process' RAM usage?

[P]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

Andrew Morton wrote:
> OK, please let us know how it goes.

It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py

Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps
but there can be large discrepancies.
In the real world you can see this with a newly started apache.
On my system statm reported that apache was using 35MB,
whereas smaps reported the correct amount of 11MB.
These numbers will converge over time as the unused pages
of the apache processes were swapped out.
Attached is a mem.py script that triggers and reports
the memory discrepancy on an smaps enabled system.

As for the overhead of calling smaps, well comparing to
/proc/$$/maps which does essentially the same thing only
not walking mem, on my 1.6GHz P4, 384MB laptop, gives:

time read_10K_smaps

real    0m3.323s
user    0m0.636s
sys     0m2.660s

time read_10K_maps

real    0m1.742s
user    0m0.300s
sys     0m1.428s

I've also attached a patch to make smaps output match the maps output,
as smaps was not showing info like which map was heap and stack etc.
The code is smaller also, as show_smap() now just calls show_map().
Note this patch is to be applied _after_ Hugh's changes.

-- 
Pádraig Brady - http://www.pixelbeat.org
--

[-- Attachment #2: mem.py --]
[-- Type: text/x-python, Size: 894 bytes --]

#!/usr/bin/python
import os, sys, time
PAGESIZE=os.sysconf("SC_PAGE_SIZE")/1024 #KiB

#stat = {Shared,Rss}
#file = {smaps,statm}
def getMemStat(pid,stat,file):
    if file=="smaps":
        shared_lines=[line
                      for line in open("/proc/"+str(pid)+"/smaps").readlines()
                      if line.find(stat)!=-1]
        return sum([int(line.split()[1]) for line in shared_lines])
    else:
        if stat == "Shared": stat_index=2
        elif stat == "Rss": stat_index=1
        return int(open("/proc/"+str(pid)+"/statm").readline().split()[stat_index])*PAGESIZE

a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
     time.sleep(1)
     sys.exit()

for file in ["statm", "smaps"]:
    rss=float(getMemStat(pid,"Rss",file))
    shr=float(getMemStat(pid,"Shared",file))
    print "/proc/%d/%s reports %.1f%% shared" % (pid,file,((shr/rss)*100))

os.wait()

[-- Attachment #3: linux-2.6.13-rc2-mm1-hd-smaps.diff --]
[-- Type: application/x-texinfo, Size: 1468 bytes --]

Re: How do you accurately determine a process' RAM usage?

[Mauricio Lin]

Hi,

On 7/12/05, P@draigbrady.com <P@draigbrady.com> wrote:
> Andrew Morton wrote:
> > OK, please let us know how it goes.
> 
> It went very well. I could find no problems at all.
> I've updated my script to use the new method, so please merge smaps :)
> http://www.pixelbeat.org/scripts/ps_mem.py
> 
> Usually the shared mem reported by /proc/$$/statm
> is the same as summing all the shared values in in /proc/$$/smaps
> but there can be large discrepancies.

Have you checked how the statm shared is calculated? I guess it does
something like:
shared = mm->rss - mm->anon_rss

But in smaps output you can have anonymous area like:

b6e0e000-b6e13000 rw-p
Size:                20 KB
Rss:                  4 KB
Shared_Clean:         0 KB
Shared_Dirty:         4 KB
Private_Clean:        0 KB
Private_Dirty:        0 KB

Look that it presents 4 KB of shared value in area considered anonymous.

ANDREW: anon_rss is the rss for anonymous area, right?

> In the real world you can see this with a newly started apache.
> On my system statm reported that apache was using 35MB,
> whereas smaps reported the correct amount of 11MB.

How dou you know that 11MB is the correct shared value  and the 35MB
is the wrong value?

BR,

Mauricio Lin.

Re: How do you accurately determine a process' RAM usage?

[P]

Mauricio Lin wrote:
> Hi,
> 
> On 7/12/05, P@draigbrady.com <P@draigbrady.com> wrote:
> 
>>Andrew Morton wrote:
>>
>>>OK, please let us know how it goes.
>>
>>It went very well. I could find no problems at all.
>>I've updated my script to use the new method, so please merge smaps :)
>>http://www.pixelbeat.org/scripts/ps_mem.py
>>
>>Usually the shared mem reported by /proc/$$/statm
>>is the same as summing all the shared values in in /proc/$$/smaps
>>but there can be large discrepancies.
> 
> 
> Have you checked how the statm shared is calculated? I guess it does
> something like:
> shared = mm->rss - mm->anon_rss

yes

> But in smaps output you can have anonymous area like:
> 
> b6e0e000-b6e13000 rw-p
> Size:                20 KB
> Rss:                  4 KB
> Shared_Clean:         0 KB
> Shared_Dirty:         4 KB
> Private_Clean:        0 KB
> Private_Dirty:        0 KB
> 
> Look that it presents 4 KB of shared value in area considered anonymous.
> 
> ANDREW: anon_rss is the rss for anonymous area, right?

I see your point and I'm not sure.
The following shell gets the shared values for the
first httpd process:

FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '`
HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`)
HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' 
| cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)
 

This shows that "smaps" reports 3060 KB more shared mem than "statm".
However adding up all the anon sections in smaps only gives 2456 KB?

When doing this I also noticed that there are duplicate
entries in smaps. Any ideas why?

grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c

       2 b7f7d000-b7f7e000 r-xp 00000000 03:05 246646 
/usr/lib/httpd/modules/mod_auth_anon.so
       2 b7f7e000-b7f7f000 rwxp 00000000 03:05 246646 
/usr/lib/httpd/modules/mod_auth_anon.so
       2 b7f7f000-b7f81000 r-xp 00000000 03:05 246645 
/usr/lib/httpd/modules/mod_auth.so
       2 b7f81000-b7f82000 rwxp 00001000 03:05 246645 
/usr/lib/httpd/modules/mod_auth.so
       2 b7f82000-b7f84000 r-xp 00000000 03:05 246641 
/usr/lib/httpd/modules/mod_access.so
       2 b7f84000-b7f85000 rwxp 00001000 03:05 246641 
/usr/lib/httpd/modules/mod_access.so
       2 b7f85000-b7f9a000 r-xp 00000000 03:05 361234     /lib/ld-2.3.3.so
       2 b7f9a000-b7f9b000 r-xp 00014000 03:05 361234     /lib/ld-2.3.3.so
       2 b7f9b000-b7f9c000 rwxp 00015000 03:05 361234     /lib/ld-2.3.3.so
       2 bfb85000-bfb9a000 rw-p bfb85000 00:00 0          [stack]
       2 ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]

If you factor that in, it means that smaps will report 156 KB
too much shared mem in this example.

>>In the real world you can see this with a newly started apache.
>>On my system statm reported that apache was using 35MB,
>>whereas smaps reported the correct amount of 11MB.
> 
> 
> How dou you know that 11MB is the correct shared value  and the 35MB
> is the wrong value?

Well I'm quite sure that COW pages ar not accounted for
in the statm shared value, which can be easily seen with
my previously posted test program. Also putting the machine
into swap, and then using httpd again causes the value reported
to be very close to 11MB (i.e. the unused pages are not swapped
back in).

-- 
Pádraig Brady - http://www.pixelbeat.org
--

Re: How do you accurately determine a process' RAM usage?

[Mauricio Lin]

Hi Brady,

On 7/20/05, P@draigbrady.com <P@draigbrady.com> wrote:
> Mauricio Lin wrote:
> > Hi,
> >
> > On 7/12/05, P@draigbrady.com <P@draigbrady.com> wrote:
> >
> >>Andrew Morton wrote:
> >>
> >>>OK, please let us know how it goes.
> >>
> >>It went very well. I could find no problems at all.
> >>I've updated my script to use the new method, so please merge smaps :)
> >>http://www.pixelbeat.org/scripts/ps_mem.py
> >>
> >>Usually the shared mem reported by /proc/$$/statm
> >>is the same as summing all the shared values in in /proc/$$/smaps
> >>but there can be large discrepancies.
> >
> >
> > Have you checked how the statm shared is calculated? I guess it does
> > something like:
> > shared = mm->rss - mm->anon_rss
> 
> yes
> 
> > But in smaps output you can have anonymous area like:
> >
> > b6e0e000-b6e13000 rw-p
> > Size:                20 KB
> > Rss:                  4 KB
> > Shared_Clean:         0 KB
> > Shared_Dirty:         4 KB
> > Private_Clean:        0 KB
> > Private_Dirty:        0 KB
> >
> > Look that it presents 4 KB of shared value in area considered anonymous.
> >
> > ANDREW: anon_rss is the rss for anonymous area, right?
> 
> I see your point and I'm not sure.
> The following shell gets the shared values for the
> first httpd process:
> 
> FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '`
> HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`)
> HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' '
> | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)
> 
> 
> This shows that "smaps" reports 3060 KB more shared mem than "statm".
> However adding up all the anon sections in smaps only gives 2456 KB?

You are adding up all Shared_Clean and Shared_Dirty as Shared, right?

> 
> When doing this I also noticed that there are duplicate
> entries in smaps. Any ideas why?

Each pair of address per line indicates the start and end address of a
memory area (VMA) such as:

b7f7d000-b7f7e000 

This means that an specific memory area start on virtual address 
b7f7d000 and end on b7f7e000 .

An mapped file like /lib/ld-2.3.3.so is organized in different memory
areas. The memory area can be a text section, data section or bss. So
it is normal you find the same filename mapped in more than one memory
area.

You can find more information about VMA on Linux Kernel Development
book (chapter 14) written by Robert Love.

For instance:

> grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c
> 
>        2 b7f7d000-b7f7e000 r-xp 00000000 03:05 246646
> /usr/lib/httpd/modules/mod_auth_anon.so
This is a text section.

>        2 b7f7e000-b7f7f000 rwxp 00000000 03:05 246646
> /usr/lib/httpd/modules/mod_auth_anon.so
This should be a data section.

IMHO, bss section corresponds to the anonymous area where the mapping
is not backed by a file.

BR,

Mauricio Lin.

Re: How do you accurately determine a process' RAM usage?

[P]

Mauricio Lin wrote:
> Hi Brady,
> 
> On 7/20/05, P@draigbrady.com <P@draigbrady.com> wrote:
> 
>>The following shell gets the shared values for the
>>first httpd process:
>>
>>FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '`
>>HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`)
>>HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' '
>>| cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)
>>
>>
>>This shows that "smaps" reports 3060 KB more shared mem than "statm".
>>However adding up all the anon sections in smaps only gives 2456 KB?
> 
> 
> You are adding up all Shared_Clean and Shared_Dirty as Shared, right?

yes. Look at the command I posted.

>>When doing this I also noticed that there are duplicate
>>entries in smaps. Any ideas why?
> 
> 
> Each pair of address per line indicates the start and end address of a
> memory area (VMA) such as:
> 
> b7f7d000-b7f7e000 
> 
> This means that an specific memory area start on virtual address 
> b7f7d000 and end on b7f7e000 .
> 
> An mapped file like /lib/ld-2.3.3.so is organized in different memory
> areas. The memory area can be a text section, data section or bss. So
> it is normal you find the same filename mapped in more than one memory
> area.

yes. Look at the command I posted.
There are multiple entries with the _same addresses_

>>grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c

-- 
Pádraig Brady - http://www.pixelbeat.org
--