Need assistance in determining memory usage

Need assistance in determining memory usage

[Thomas Schenk]

Greetings all.

I have been asked a question by some of the developers in my
organization and after searching Google, scouring the Linux newsgroups,
and searching as many mailing list archives and howtos as I could find,
I still cannot find a satisfactory answer to the following question:

Q. How can you determine how much memory a process is using at a given
point in time?  Specifically, I want to know of a method or tool that
will tell me how much total memory a process is using, how much of that
total is shared with other processes, how much is resident, and how much
is swapped out.

Please don't say to just use ps or top, because if either of these tools
was adequate, I wouldn't be asking here and every reference I could find
indicates that this is not a trivial problem.  There were also
indications I found while searching that these tools do not always
report memory numbers accurately.  If there is a way to determine this
information using /proc, this would be ideal, since I could then
conceivably create a script or simple program that could determine the
answer given the process ID, which is what the developers here really
want.

If you can assist me in determining the answer to this question, I would
greatly appreciate it.  Please note that I am far from being a kernel
expert (being just a lowly systems administrator), so please keep make
your explanations as detailed as possible.

Tom S.

-- 
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| Tom Schenk      | A positive attitude may not solve all your    |
| Online Ops      | problems, but it will annoy enough people to  |
| tschenk@ea.com  | make it worth the effort. -- Herm Albright    |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

Re: Need assistance in determining memory usage

[bert hubert]

On Mon, Nov 04, 2002 at 12:11:07PM -0600, Thomas Schenk wrote:

> Please don't say to just use ps or top, because if either of these tools
> was adequate, I wouldn't be asking here and every reference I could find

(...)

> expert (being just a lowly systems administrator), so please keep make
> your explanations as detailed as possible.

If you could state your question as detailed as possible. Please explain how
'ps' and 'top' are inadequate.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: Need assistance in determining memory usage

[Ketil Froyn]

On 4 Nov 2002, Thomas Schenk wrote:

> Q. How can you determine how much memory a process is using at a given
> point in time?  Specifically, I want to know of a method or tool that
> will tell me how much total memory a process is using, how much of that
> total is shared with other processes, how much is resident, and how much
> is swapped out.

/proc/<pid>/{stat,statm,status} are probably helpful. There's some info on
this in Documentation/filesystems/proc.txt

Ketil

Re: Need assistance in determining memory usage

[Richard B. Johnson]

In /proc there are a bunch of directories that are numbered.
These numbers correspond to the PID of each process.
In each of these directories is a file called "status".

This file contains most all the information you need just
in case you don't like `ps` or `top`.

Name:	init
State:	S (sleeping)
Tgid:	1
Pid:	1
PPid:	0
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	32
Groups:	
VmSize:	     224 kB              Total virtual memory accessed so far.
VmLck:	       0 kB              Locks held
VmRSS:	     148 kB              Resident virtual memory (not swapped)
VmData:	      28 kB              Data virtual memory
VmStk:	       4 kB              Stack virtual memory
VmExe:	     184 kB              Executable virtual memory
VmLib:	       0 kB              Shared library
SigPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000057f0defc
SigCgt:	00000000280b2003
CapInh:	0000000000000000
CapPrm:	00000000ffffffff
CapEff:	00000000fffffeff


Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
   Bush : The Fourth Reich of America

Re: Need assistance in determining memory usage

[Alan Cox]

On Mon, 2002-11-04 at 18:11, Thomas Schenk wrote:
> was adequate, I wouldn't be asking here and every reference I could find
> indicates that this is not a trivial problem.  There were also
> indications I found while searching that these tools do not always
> report memory numbers accurately.  If there is a way to determine this
> information using /proc, this would be ideal, since I could then
> conceivably create a script or simple program that could determine the
> answer given the process ID, which is what the developers here really
> want.

Neither the question nor the answer are trivial. What are you trying to
do with the data may be the most relevant question

Re: Need assistance in determining memory usage

[Thomas Schenk]

On Mon, 2002-11-04 at 13:01, Alan Cox wrote:
> On Mon, 2002-11-04 at 18:11, Thomas Schenk wrote:
> > was adequate, I wouldn't be asking here and every reference I could find
> > indicates that this is not a trivial problem.  There were also
> > indications I found while searching that these tools do not always
> > report memory numbers accurately.  If there is a way to determine this
> > information using /proc, this would be ideal, since I could then
> > conceivably create a script or simple program that could determine the
> > answer given the process ID, which is what the developers here really
> > want.
> 
> Neither the question nor the answer are trivial. What are you trying to
> do with the data may be the most relevant question

This situation is this:

We are building an online game system.  On some of the systems, there
are simulator processes running that each service a player.  There may
be up to 200 or more of these processes running at any given time and
each uses a fairly large amount of memory (as reported by ps).  Part of
this is due to the fact that the processes have not been optimized to
make the most efficient use of memory.  When the simulator processes
start swapping, then the systems are becoming unstable, performance goes
all to hell and sometimes the systems totally hang.  It would be useful
for us to be able to monitor as closely as possible the amount of memory
each processes is using and especially to be notified when these
processes start using significant amounts of swap, so that we can be
prepared to react before the situation gets out of hand.  The other
reason why we want to collect this data is so that the developers can
analyze the process when it starts to swap and determine if there
optimizations that they can make that will improve the memory
utilization of these processes so that more processes can be run on the
same box and that swap usage is minimized.  

A couple of points that may be useful (or not):

1.  These systems are based on RedHat 7.2, but are running a kernel
built from the kernel source tree for Advanced Server 2.1 (as obtained
from the kernel source RPM for the 2.4.9.e-3 kernel).  Originally, they
were running on the 2.4.18 kernel from RedHat 7.3, but in our particular
situation, the 2.4.9 Advanced Server kernel was found to have better
performance characteristics.

2.  Each of the systems running the simulator processes consist of the
following:

    Dual P4 Xeon 2.5 GHz processors (hyperthreading is enabled in the
      motherboard BIOS setup)
    4 Gigs of RAM
    3Ware RAID controller
    2 x 40 GB disks in RAID 1 configuration
    2 x E1000 NICS

3.  The only modifications that were made to the 2.4.9 AS kernel was to
update to the latest version of the E1000 driver from Intel, since the
one in the AS 2.1 kernel source tree didn't work with the systems.  The
reason why the kernel was compiled locally was to remove unwanted
options, such as USB and sound support, and to eliminate the need for an
initrd.

If further data is required, I can provide it.  Thanks for all who have
responded thus far.

Tom S. 
    
 
-- 
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| Tom Schenk      | A positive attitude may not solve all your    |
| Online Ops      | problems, but it will annoy enough people to  |
| tschenk@ea.com  | make it worth the effort. -- Herm Albright    |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

Re: Need assistance in determining memory usage

[Dan Kegel]

  Thomas Schenk <tschenk@origin.ea.com> wrote:
 > We are building an online game system.  On some of
 > the systems, there are simulator processes running
 > that each service a player.  There may be up to
 > 200 or more of these processes running at any given
 > time and each uses a fairly large amount of memory
 > ...  When the simulator processes start swapping,
 > the systems are becoming unstable, performance goes
 > all to hell...  It would
 > be useful for us to be able to monitor as closely as
 > possible the amount of memory each processes is using
 > and especially to be notified when these processes
 > start using significant amounts of swap, so that we
 > can be prepared to react before the situation gets
 > out of hand.  The other reason why we want to collect
 > this data is so that the developers can analyze the
 > process when it starts to swap ...

A few things you might try:

1. Set RLIMIT_DATA or RLIMIT_AS for your processes
using ulimit.  That should cause malloc() and the
like to return NULL or throw an exception if you go
over the limit.  (RLIMIT_AS doesn't work too well
on heavily multithreaded programs, though, because
of the many stacks.)

2. Try the no-overcommit patch.  That will fail
allocation requests that might conceivably cause swapping
later.  Harsh, but you did want early notification :-)

3. Start reading the kernel source to see how it calculates
memory use and enforces the above limits.  That helped
clarify things for me a bit.

4. If you're feeling really nasty, you could try applying
my patch that implements RLIMIT_RSS by killing the process.
That's pretty immediate feedback, too :-)

- Dan

Re: Need assistance in determining memory usage

[Andy Isaacson]

On Mon, Nov 04, 2002 at 01:22:44PM -0600, Thomas Schenk wrote:
> On Mon, 2002-11-04 at 13:01, Alan Cox wrote:
> > On Mon, 2002-11-04 at 18:11, Thomas Schenk wrote:
> > > was adequate, I wouldn't be asking here and every reference I could find
> > > indicates that this is not a trivial problem.  There were also
> > > indications I found while searching that these tools do not always
> > > report memory numbers accurately.  If there is a way to determine this
> > > information using /proc, this would be ideal, since I could then
> > > conceivably create a script or simple program that could determine the
> > > answer given the process ID, which is what the developers here really
> > > want.
> > 
> > Neither the question nor the answer are trivial. What are you trying to
> > do with the data may be the most relevant question
> 
> This situation is this:
> 
> We are building an online game system.  On some of the systems, there
> are simulator processes running that each service a player.  There may
> be up to 200 or more of these processes running at any given time and
> each uses a fairly large amount of memory (as reported by ps).  Part of
> this is due to the fact that the processes have not been optimized to
> make the most efficient use of memory.  When the simulator processes
> start swapping, then the systems are becoming unstable, performance goes
> all to hell and sometimes the systems totally hang.  It would be useful
> for us to be able to monitor as closely as possible the amount of memory
> each processes is using and especially to be notified when these
> processes start using significant amounts of swap, so that we can be
> prepared to react before the situation gets out of hand.

I do not believe that the kernel exports the information "what processes
are using swap?".  You can answer some of your questions by using my
pmap program; it's in at least some recent procps packages, or download
the source:
http://web.hexapodia.org/~adi/pmap.c

-andy