Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Eric Valette]

Hi,

I'm currently trying use mlockall for a soft RT application and was 
concerned by the memory usage of my (big) RT process. So I stated to 
look at /proc/<PID>/status and /proc/<PID>/statm and then to the 
documentation of statm (Documentation/filesystem/proc.txt) because 
unexplained values are rather useless.

The doc currently says :
-------------------------------------------------------------------------
  Field    Content
  size     total program size (pages)		(same as VmSize in status)
  resident size of memory portions (pages)	(same as VmRSS in status)
  shared   number of pages that are shared	(i.e. backed by a file)
  trs      number of pages that are 'code'	(not including libs; broken,
							includes data segment)
  lrs      number of pages of library		(always 0 on 2.6)
  drs      number of pages of data/stack		(including libs; broken,
							includes library text)
  dt       number of dirty pages			(always 0 on 2.6)
---------------------------------------------------------------------------

But when I did cat /proc/<pid>/status and /proc/<pid>/statm  the result 
where rather different (and not only due to size units). Looking at 
/usr/src/linux/fs/proc/task_mmu.c It is not surpising.

May I suggest :
	- To use consistent memory size units between status and statm,
	- To have VmSize and statm return the same global memory size,
	- To eventually display mm->reserved_vm in status and put a word about 
it in the doc,

Of course I can send a patch but I would rather let you decide the way 
you want to go...


Thanks,

-- 
    __
   /  `                   	Eric Valette
  /--   __  o _.          	6 rue Paul Le Flem
(___, / (_(_(__         	35740 Pace

Tel: +33 (0)2 99 85 26 76	Fax: +33 (0)2 99 85 26 76
E-mail: eric.valette@free.fr

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Albert Cahalan]

Eric Valette writes:

> I'm currently trying use mlockall for a soft RT application and was 
> concerned by the memory usage of my (big) RT process. So I stated to
> look at /proc/<PID>/status and /proc/<PID>/statm and then to the  
> documentation of statm (Documentation/filesystem/proc.txt) because
> unexplained values are rather useless.
>
> The doc currently says :
> -------------------------------------------------------------------------
>   Field    Content
>   size     total program size (pages)  (same as VmSize in status)

The documentation is incorrect. It was written to match a buggy
implementation in early 2.6.x kernels.

VmSize is the address space occupied, excluding memory-mapped IO.
The statm value is the address space occupied.

> May I suggest :
>  - To use consistent memory size units between status and statm,

No way. This would instantly break the "top" program.

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Eric Valette]

Albert Cahalan wrote:

> The documentation is incorrect. It was written to match a buggy
> implementation in early 2.6.x kernels.

Well the Documentation is said to matches 2.6.8-rc3 and is only 5 weeks 
old according to bitkeeper changesets... So at least the doc should be 
fixed.

> VmSize is the address space occupied, excluding memory-mapped IO.
> The statm value is the address space occupied.

Why removing memory-mapped IO in one case (status) and not the other 
(statm)? Memory mapped IO, may of course reserve some physical memory 
pages for establishing the mmu->phys adress translation table (if any) 
but not really the amount of space mapped.

>>May I suggest :
>> - To use consistent memory size units between status and statm,

> No way. This would instantly break the "top" program.

OK. Too bad because statm is hardly readable but I guess it is not for 
human then...

Thanks for responding,

-- eric

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Albert Cahalan]

On Sat, 2004-10-09 at 10:08, Eric Valette wrote:
> Albert Cahalan wrote:
> 
> > The documentation is incorrect. It was written to match a buggy
> > implementation in early 2.6.x kernels.
> 
> Well the Documentation is said to matches 2.6.8-rc3 and is only 5 weeks 
> old according to bitkeeper changesets... So at least the doc should be 
> fixed.

Removal would be simpler.

> > VmSize is the address space occupied, excluding memory-mapped IO.
> > The statm value is the address space occupied.
> 
> Why removing memory-mapped IO in one case (status) and not the other 
> (statm)? Memory mapped IO, may of course reserve some physical memory 
> pages for establishing the mmu->phys adress translation table (if any) 
> but not really the amount of space mapped.

First of all, because that's the way it's been done.
This interface goes back to the Linux 1.0.xx days.

Second of all, because you get more information this
way. You can subtract to determine the address space
used for IO alone.

I could go for another number: available address space.
Then I could display percent used.

> >>May I suggest :
> >> - To use consistent memory size units between status and statm,
> 
> > No way. This would instantly break the "top" program.
> 
> OK. Too bad because statm is hardly readable but I guess it is not for 
> human then...

Even the other files are only partly for humans.
Minor changes will cause many tools to break.

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Eric Valette]

Albert Cahalan wrote:

>>Well the Documentation is said to matches 2.6.8-rc3 and is only 5 weeks 
>>old according to bitkeeper changesets... So at least the doc should be 
>>fixed.
> 
> 
> Removal would be simpler.

I beg to disagree. statm catche eyes when you want to know precisely 
your memory usage or do you consider to be the single statm user via procps?

> Second of all, because you get more information this
> way. You can subtract to determine the address space
> used for IO alone.

Sure but displaying it via status information would be much more simple 
furthermore I think it is incorrectly computed.

> I could go for another number: available address space.
> Then I could display percent used.

The free command does provide the information I think so it must be 
somewhere else...

> Even the other files are only partly for humans.
> Minor changes will cause many tools to break.

Sure but many people in the embedded wolrd need to know precisley 
process memory usage and possibly inside the program itsel not via 
top/ps/free/...

And if tools display wrong information for ages, then it can still be 
fixed...

-- 
    __
   /  `                   	Eric Valette
  /--   __  o _.          	6 rue Paul Le Flem
(___, / (_(_(__         	35740 Pace

Tel: +33 (0)2 99 85 26 76	Fax: +33 (0)2 99 85 26 76
E-mail: eric.valette@free.fr

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Albert Cahalan]

On Sat, 2004-10-09 at 12:46, Eric Valette wrote:
> Albert Cahalan wrote:
> 
> >>Well the Documentation is said to matches 2.6.8-rc3 and is only 5 weeks 
> >>old according to bitkeeper changesets... So at least the doc should be 
> >>fixed.
> > 
> > 
> > Removal would be simpler.
> 
> I beg to disagree. statm catche eyes when you want to know precisely 
> your memory usage or do you consider to be the single statm user via procps?

Not quite, but close. Those that know are willing
to read the kernel code, including that of past
releases, can certainly use the statm file.

> > I could go for another number: available address space.
> > Then I could display percent used.
> 
> The free command does provide the information I think so it must be 
> somewhere else...

Not at all. The free command is not per-process,
and doesn't ever deal with address space. It deals
with RAM.

For example, on a typical i386 box, you have almost
3 GB of address space available. (minus a bit at the
top for syscall handling, and perhaps minus a page at
address 0) The portion of this in use is of interest.

> > Even the other files are only partly for humans.
> > Minor changes will cause many tools to break.
> 
> Sure but many people in the embedded wolrd need to know precisley 
> process memory usage and possibly inside the program itsel not via 
> top/ps/free/...

For that, go directly to /proc/*/maps and be happy.
You may also want the RSS from /proc/*/status.

Re: Inconsistancies in /proc (status vs statm) leading to wrong documentation (proc.txt)

[Eric Valette]

Albert Cahalan wrote:

> Not quite, but close. Those that know are willing
> to read the kernel code, including that of past
> releases, can certainly use the statm file.

Sure but it while doing that after readding the official documentation 
that the trouble began...


> For that, go directly to /proc/*/maps and be happy.

That's a good point. Very usefull for knowing which dynamic library 
account for what size. I start wondering if dynamic libraries and 
mlockall works well toghether..

> You may also want the RSS from /proc/*/status.

Well RSS = VmSize after mlockall I guess

Thanks for the maps hint,

-- eric