Newbie question on malloc()

Newbie question on malloc()

[Wen Guangcheng]

Hello Gurus,
I have made a daemon in which dynamic memory is gotten
by malloc(). Does the memory get free automatically without
free() by the deamon  when the daemon process is killed?
Thanks in advance.

--Wen

Re: Newbie question on malloc()

[John T. Williams]

Since no one else has answered yet.  

As I understand it, that is entirely up to the operation system.  Linux
and NT Kernel systems do reclaim unfreed memory, however I believe one
the major problems with Win98 was that it did not.

Anyone with more information feel free to correct me

On Wed, 2004-06-02 at 07:48, Wen Guangcheng wrote:
> Hello Gurus,
> I have made a daemon in which dynamic memory is gotten
> by malloc(). Does the memory get free automatically without
> free() by the deamon  when the daemon process is killed?
> Thanks in advance.
> 
> --Wen
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Newbie question on malloc()

[Glynn Clements]

Wen Guangcheng wrote:

> I have made a daemon in which dynamic memory is gotten
> by malloc(). Does the memory get free automatically without
> free() by the deamon  when the daemon process is killed?

All memory which has been allocated to a process is released when the
process terminates.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[Glynn Clements]

John T. Williams wrote:

> As I understand it, that is entirely up to the operation system.  Linux
> and NT Kernel systems do reclaim unfreed memory, however I believe one
> the major problems with Win98 was that it did not.
> 
> Anyone with more information feel free to correct me

I very much doubt that the above is accurate.

It may be that Win98 had some specific memory leaks, but any OS which,
in the general case, failed to recover a process' memory upon
termination would run out of memory very quickly.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[John T. Williams]

What you never had win98 crash on you after running for an hour? ;) 

May statement is based on something I remember vaguely from my OS class
2 years ago, so I'm quite happy to believe I'm wrong.  I remember there
was some major issue with how win98 handled dynamic allocated memory,
but I can't exactly remember what.  

After googling a little I seems that win95 had some problems with
Garbage Collection not win98.
 

On Wed, 2004-06-02 at 13:41, Glynn Clements wrote:
> John T. Williams wrote:
> 
> > As I understand it, that is entirely up to the operation system.  Linux
> > and NT Kernel systems do reclaim unfreed memory, however I believe one
> > the major problems with Win98 was that it did not.
> > 
> > Anyone with more information feel free to correct me
> 
> I very much doubt that the above is accurate.
> 
> It may be that Win98 had some specific memory leaks, but any OS which,
> in the general case, failed to recover a process' memory upon
> termination would run out of memory very quickly.

Re: Newbie question on malloc()

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Wed, 2004-06-02 18:41:03 +0100, Glynn Clements <glynn.clements@virgin.net>
wrote in message <16574.4399.288522.256729@cerise.nosuchdomain.co.uk>:
> John T. Williams wrote:
> > As I understand it, that is entirely up to the operation system.  Linux
> > and NT Kernel systems do reclaim unfreed memory, however I believe one
> > the major problems with Win98 was that it did not.
> > 
> > Anyone with more information feel free to correct me
> 
> I very much doubt that the above is accurate.
> 
> It may be that Win98 had some specific memory leaks, but any OS which,
> in the general case, failed to recover a process' memory upon
> termination would run out of memory very quickly.

Well, that all depends on the definition of "OS". Any recent 32+ bit
operating system with virtual memory capabilities and multi-processing
will reclaim malloc()ed RAM upon process termination.

However, there are some minor operating systems (as I said, depends on
definition) out there under which processes *need* to free() their
memory. But if we talk about Linux, *BSD (right, and even Windows
systems starting from Win 3.1) will reclaim memory:)

However, if your program is designed so that you can easily track all
allocations, it doesn't harm if you free everything before exit...

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Newbie question on malloc()

[Micha Feigin]

On Wed, Jun 02, 2004 at 08:48:07PM +0900, Wen Guangcheng wrote:
> Hello Gurus,
> I have made a daemon in which dynamic memory is gotten
> by malloc(). Does the memory get free automatically without
> free() by the deamon  when the daemon process is killed?
> Thanks in advance.
> 

It does, but in general its not in good practice to count on process
exit for freeing memory (a good way to get memory leaks).

> --Wen
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>  +++++++++++++++++++++++++++++++++++++++++++
>  This Mail Was Scanned By Mail-seCure System
>  at the Tel-Aviv University CC.
> 

Re: Newbie question on malloc()

[Micha Feigin]

On Wed, Jun 02, 2004 at 08:37:34PM +0200, Jan-Benedict Glaw wrote:
> On Wed, 2004-06-02 18:41:03 +0100, Glynn Clements <glynn.clements@virgin.net>
> wrote in message <16574.4399.288522.256729@cerise.nosuchdomain.co.uk>:
> > John T. Williams wrote:
> > > As I understand it, that is entirely up to the operation system.  Linux
> > > and NT Kernel systems do reclaim unfreed memory, however I believe one
> > > the major problems with Win98 was that it did not.
> > > 
> > > Anyone with more information feel free to correct me
> > 
> > I very much doubt that the above is accurate.
> > 
> > It may be that Win98 had some specific memory leaks, but any OS which,
> > in the general case, failed to recover a process' memory upon
> > termination would run out of memory very quickly.
> 
> Well, that all depends on the definition of "OS". Any recent 32+ bit
> operating system with virtual memory capabilities and multi-processing
> will reclaim malloc()ed RAM upon process termination.

It has nothing to do with virtual memory. In any multi-processing OS
(any modern one probably) memory allocations are done by the kernel
since the kernel is the one managing the system's memory.

If the kernel doesn't have a bug, this memory is released on process
exit (unless it is still used by another process such as with fork with
virtual memory where it is marked copy on write).

> 
> However, there are some minor operating systems (as I said, depends on
> definition) out there under which processes *need* to free() their
> memory. But if we talk about Linux, *BSD (right, and even Windows
> systems starting from Win 3.1) will reclaim memory:)
> 
> However, if your program is designed so that you can easily track all
> allocations, it doesn't harm if you free everything before exit...
> 
> MfG, JBG
> 
> -- 
>    Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
>    "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
>     fuer einen Freien Staat voll Freier B?rger" | im Internet! |   im Irak!
>    ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

Re: Newbie question on malloc()

[Luciano Moreira - igLnx]

My suggests (I expect that it's commom):

If you are making a app for your personal brief use (like a brief tool), 
OK !!
    You can leave OS to control your auto-"freed" memory.
else
    You need to free your allocated memory by yorself (either by itself 
<---- your app).

Luciano.
----------
- Try to use "free()" or "delete".
- app = (your) application/software.



Micha Feigin wrote:

>On Wed, Jun 02, 2004 at 08:48:07PM +0900, Wen Guangcheng wrote:
>  
>
>>Hello Gurus,
>>I have made a daemon in which dynamic memory is gotten
>>by malloc(). Does the memory get free automatically without
>>free() by the deamon  when the daemon process is killed?
>>Thanks in advance.
>>
>>    
>>
>
>It does, but in general its not in good practice to count on process
>exit for freeing memory (a good way to get memory leaks).
>
>  
>
>>--Wen
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>> +++++++++++++++++++++++++++++++++++++++++++
>> This Mail Was Scanned By Mail-seCure System
>> at the Tel-Aviv University CC.
>>
>>    
>>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: Newbie question on malloc()

[Glynn Clements]

John T. Williams wrote:

> > > As I understand it, that is entirely up to the operation system.  Linux
> > > and NT Kernel systems do reclaim unfreed memory, however I believe one
> > > the major problems with Win98 was that it did not.
> > > 
> > > Anyone with more information feel free to correct me
> > 
> > I very much doubt that the above is accurate.
> > 
> > It may be that Win98 had some specific memory leaks, but any OS which,
> > in the general case, failed to recover a process' memory upon
> > termination would run out of memory very quickly.
> 
> What you never had win98 crash on you after running for an hour? ;) 

Sure. I've also had it crash after running for less than a minute. And
I've also had it run for hours without crashing.

However, if it didn't reclaim a process' memory upon termination, the
scenario where you run a short-lived but memory-hungry program many
times simply wouldn't be possible.

Bear in mind that many (possibly most) programs don't explicitly free
dynamically-allocated memory blocks upon termination; they just
terminate.

This is recommended practice. Explicitly freeing memory blocks is a
waste of time: you're basically just modifying regions of the process'
memory, which is about to be returned to the OS, whereupon its
contents become irrelevant.

[It may also make the program substantially more complex, as you would
need to keep track of such blocks.]

Note that free() doesn't generally return memory to the OS; it usually
just records the fact that the specified region of memory is available
for re-use by subsequent calls to malloc() etc.

Similarly, malloc() doesn't necessary result in any additional memory
being obtained from the OS; frequently it will just return a region of
memory which has already been allocated to the process but isn't being
used.

> May statement is based on something I remember vaguely from my OS class
> 2 years ago, so I'm quite happy to believe I'm wrong.  I remember there
> was some major issue with how win98 handled dynamic allocated memory,
> but I can't exactly remember what.  

The main reason for the unreliability of Win95/98/ME is that, for
compatibility with earlier versions, certain regions of memory are
unprotected. So, if an application crashes, it can corrupt data
belonging to other applications or the OS.

[The details relate to the fact that the 8086 doesn't have an MMU, so
there is a single, global address space shared by all processes (and
the OS). Originally, interprocess communication made extensive use of
passing around handles to memory blocks, relying upon the fact that
any process could read (and write) any part of memory. While Windows
3.1 and later require at least a 286, the behaviour was retained for
compatibility.]

> After googling a little I seems that win95 had some problems with
> Garbage Collection not win98.

The differences between 95 and 98 are essentially superficial. Win98
includes IE, and probably a significant number of bug fixes, which may
include bugs in memory allocation. However, the underlying
architecture is unchanged, and that includes the way in which memory
management works (other than any outright bugs which may have been
fixed).

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[Glynn Clements]

Micha Feigin wrote:

> > I have made a daemon in which dynamic memory is gotten
> > by malloc(). Does the memory get free automatically without
> > free() by the deamon  when the daemon process is killed?
> > Thanks in advance.
> 
> It does, but in general its not in good practice to count on process
> exit for freeing memory (a good way to get memory leaks).

This is incorrect.

You shouldn't make an effort to return memory to the process' heap (by
calling free()) if the program is about to terminate.

Doing so consumes CPU time (and probably disk bandwidth, given that
some of that memory will probably be swapped out) and doesn't provide
any benefit (free() won't usually return memory to the OS and, in any
case, all of the process' memory will be returned to the OS upon
exit).

The only valid reason for free()ing memory blocks upon termination is
if the structure of the program is such that it isn't practical to
avoid doing so. E.g. if data structures have matched setup/cleanup
routines, and you need to call the cleanup code for other reasons, and
the cleanup code will free the memory anyway.

Similarly, you don't need to explicitly close() file descriptors, or
release file locks (those obtained with flock/lockf/fcntl) as the OS
will do that.

The sort of actions which might make sense to perform before calling
exit() are:

+ Writing a log entry reporting shutdown
+ Deleting temporary files
+ Deleting any SysV IPC structures which aren't shared with other
  processes.
+ Informing other networked processes of termination, e.g. sending a
  "QUIT" command.
+ Killing child processes.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[Micha Feigin]

On Thu, Jun 03, 2004 at 08:41:33AM +0100, Glynn Clements wrote:
> 
> John T. Williams wrote:
> 
> > > > As I understand it, that is entirely up to the operation system.  Linux
> > > > and NT Kernel systems do reclaim unfreed memory, however I believe one
> > > > the major problems with Win98 was that it did not.
> > > > 
> > > > Anyone with more information feel free to correct me
> > > 
> > > I very much doubt that the above is accurate.
> > > 
> > > It may be that Win98 had some specific memory leaks, but any OS which,
> > > in the general case, failed to recover a process' memory upon
> > > termination would run out of memory very quickly.
> > 
> > What you never had win98 crash on you after running for an hour? ;) 
> 
> Sure. I've also had it crash after running for less than a minute. And
> I've also had it run for hours without crashing.
> 
> However, if it didn't reclaim a process' memory upon termination, the
> scenario where you run a short-lived but memory-hungry program many
> times simply wouldn't be possible.
> 
> Bear in mind that many (possibly most) programs don't explicitly free
> dynamically-allocated memory blocks upon termination; they just
> terminate.
> 
> This is recommended practice. Explicitly freeing memory blocks is a
> waste of time: you're basically just modifying regions of the process'
> memory, which is about to be returned to the OS, whereupon its
> contents become irrelevant.
> 
> [It may also make the program substantially more complex, as you would
> need to keep track of such blocks.]
> 
> Note that free() doesn't generally return memory to the OS; it usually
> just records the fact that the specified region of memory is available
> for re-use by subsequent calls to malloc() etc.
> 

That would depend on the memory block size. Small memory blocks are
held as malloc allocates whole pages at a time, large memory blocks are
returned. Depending on size the OS may then also store them on caches
in case a memory block of the same size is requested again.

Otherwise in a long running program that allocates a lot of memory on
the start and then frees most of it would still be hogging all that
memory.

As for not freeing memory on program exit thats another issue since you
are saving malloc's internal work (which especially with large memory
chunks will probably mostly be a system call letting the kernel do most
of the work, for small chunks there is much more overhead).

> Similarly, malloc() doesn't necessary result in any additional memory
> being obtained from the OS; frequently it will just return a region of
> memory which has already been allocated to the process but isn't being
> used.
> 
> > May statement is based on something I remember vaguely from my OS class
> > 2 years ago, so I'm quite happy to believe I'm wrong.  I remember there
> > was some major issue with how win98 handled dynamic allocated memory,
> > but I can't exactly remember what.  
> 
> The main reason for the unreliability of Win95/98/ME is that, for
> compatibility with earlier versions, certain regions of memory are
> unprotected. So, if an application crashes, it can corrupt data
> belonging to other applications or the OS.
> 
> [The details relate to the fact that the 8086 doesn't have an MMU, so
> there is a single, global address space shared by all processes (and
> the OS). Originally, interprocess communication made extensive use of
> passing around handles to memory blocks, relying upon the fact that
> any process could read (and write) any part of memory. While Windows
> 3.1 and later require at least a 286, the behaviour was retained for
> compatibility.]
> 
> > After googling a little I seems that win95 had some problems with
> > Garbage Collection not win98.
> 
> The differences between 95 and 98 are essentially superficial. Win98
> includes IE, and probably a significant number of bug fixes, which may
> include bugs in memory allocation. However, the underlying
> architecture is unchanged, and that includes the way in which memory
> management works (other than any outright bugs which may have been
> fixed).
> 
> -- 
> Glynn Clements <glynn.clements@virgin.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>  +++++++++++++++++++++++++++++++++++++++++++
>  This Mail Was Scanned By Mail-seCure System
>  at the Tel-Aviv University CC.
> 

Re: Newbie question on malloc()

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

On Thu, 2004-06-03 04:34:27 +0300, Micha Feigin <michf@post.tau.ac.il>
wrote in message <20040603013427.GD2562@luna.mooo.com>:
> On Wed, Jun 02, 2004 at 08:37:34PM +0200, Jan-Benedict Glaw wrote:
> > Well, that all depends on the definition of "OS". Any recent 32+ bit
> > operating system with virtual memory capabilities and multi-processing
> > will reclaim malloc()ed RAM upon process termination.
> 
> It has nothing to do with virtual memory. In any multi-processing OS
> (any modern one probably) memory allocations are done by the kernel
> since the kernel is the one managing the system's memory.
> 
> If the kernel doesn't have a bug, this memory is released on process
> exit (unless it is still used by another process such as with fork with
> virtual memory where it is marked copy on write).

Read again. There ARE (or at least: were) operating systems that
intentionally did *not* implicitely free memory. If an application
missed to free all allocated memory, it's lost. Forever.

But modern OSes (those with at least have the capabilities mentioned
above) will automatically reclaim all of a exit'ed process' memory.

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Newbie question on malloc()

[John T. Williams]

I think this is may be a somewhat misleading statement. Not freeing
memory before normal termination decreases re-usability of your code.
You never know when your main function might be reused verbatim as
another sub function. Failing to free memory before all reference leave
scope can lead to massive memory leeks. Further, the more memory your
program is holding at any given moment of operation the more pages it
holds, the more pages it holds the more likely any particular call to a
memory location will lead to a page fault, and page faults are
expensive. Freeing memory allows malloc to reassign that memory next
time its called. Now I know that this was a question about freeing
memory before termination, and therefore unless we are talking about a
situation where your code is being reused my last two arguments hold no
weight, but I believe that it is simply good practice to always be aware
of memory you have dynamically allocated and always free it after you no
more use for it. In this way you don't get used to saying well if I miss
it the operating system will clean it up for me. Please note I do not
suggest that one should go to out of the way to free memory on abnormal
termination, but its simply good practice to free memory when it is no
longer being used in the normal behavior of your code. 



On Thu, 2004-06-03 at 03:59, Glynn Clements wrote:
> Micha Feigin wrote:
> 
> > > I have made a daemon in which dynamic memory is gotten
> > > by malloc(). Does the memory get free automatically without
> > > free() by the deamon  when the daemon process is killed?
> > > Thanks in advance.
> > 
> > It does, but in general its not in good practice to count on process
> > exit for freeing memory (a good way to get memory leaks).
> 
> This is incorrect.
> 
> You shouldn't make an effort to return memory to the process' heap (by
> calling free()) if the program is about to terminate.
> 
> Doing so consumes CPU time (and probably disk bandwidth, given that
> some of that memory will probably be swapped out) and doesn't provide
> any benefit (free() won't usually return memory to the OS and, in any
> case, all of the process' memory will be returned to the OS upon
> exit).
> 
> The only valid reason for free()ing memory blocks upon termination is
> if the structure of the program is such that it isn't practical to
> avoid doing so. E.g. if data structures have matched setup/cleanup
> routines, and you need to call the cleanup code for other reasons, and
> the cleanup code will free the memory anyway.
> 
> Similarly, you don't need to explicitly close() file descriptors, or
> release file locks (those obtained with flock/lockf/fcntl) as the OS
> will do that.
> 
> The sort of actions which might make sense to perform before calling
> exit() are:
> 
> + Writing a log entry reporting shutdown
> + Deleting temporary files
> + Deleting any SysV IPC structures which aren't shared with other
>   processes.
> + Informing other networked processes of termination, e.g. sending a
>   "QUIT" command.
> + Killing child processes.

Re: Newbie question on malloc()

[Paul Gimpelj]

hi,
pardon me for patching to this thread,

but, following this conversation,
if files are left open at exit, does the operating system (linux) flush the
i/o buffers to disk before closing files?
This is not the case with windows 95/98  and dos.
OR, does the _exit() function in the c runtime do it.
Thanks.

thanks

regards,
Paul


----- Original Message -----
From: "John T. Williams" <jowillia@vt.edu>
To: "Glynn Clements" <glynn.clements@virgin.net>
Cc: "Micha Feigin" <michf@post.tau.ac.il>; "linux-c-programming"
<linux-c-programming@vger.kernel.org>
Sent: Thursday, June 03, 2004 6:25 PM
Subject: Re: Newbie question on malloc()


>
> I think this is may be a somewhat misleading statement. Not freeing
> memory before normal termination decreases re-usability of your code.
> You never know when your main function might be reused verbatim as
> another sub function. Failing to free memory before all reference leave
> scope can lead to massive memory leeks. Further, the more memory your
> program is holding at any given moment of operation the more pages it
> holds, the more pages it holds the more likely any particular call to a
> memory location will lead to a page fault, and page faults are
> expensive. Freeing memory allows malloc to reassign that memory next
> time its called. Now I know that this was a question about freeing
> memory before termination, and therefore unless we are talking about a
> situation where your code is being reused my last two arguments hold no
> weight, but I believe that it is simply good practice to always be aware
> of memory you have dynamically allocated and always free it after you no
> more use for it. In this way you don't get used to saying well if I miss
> it the operating system will clean it up for me. Please note I do not
> suggest that one should go to out of the way to free memory on abnormal
> termination, but its simply good practice to free memory when it is no
> longer being used in the normal behavior of your code.
>
>
>
> On Thu, 2004-06-03 at 03:59, Glynn Clements wrote:
> > Micha Feigin wrote:
> >
> > > > I have made a daemon in which dynamic memory is gotten
> > > > by malloc(). Does the memory get free automatically without
> > > > free() by the deamon  when the daemon process is killed?
> > > > Thanks in advance.
> > >
> > > It does, but in general its not in good practice to count on process
> > > exit for freeing memory (a good way to get memory leaks).
> >
> > This is incorrect.
> >
> > You shouldn't make an effort to return memory to the process' heap (by
> > calling free()) if the program is about to terminate.
> >
> > Doing so consumes CPU time (and probably disk bandwidth, given that
> > some of that memory will probably be swapped out) and doesn't provide
> > any benefit (free() won't usually return memory to the OS and, in any
> > case, all of the process' memory will be returned to the OS upon
> > exit).
> >
> > The only valid reason for free()ing memory blocks upon termination is
> > if the structure of the program is such that it isn't practical to
> > avoid doing so. E.g. if data structures have matched setup/cleanup
> > routines, and you need to call the cleanup code for other reasons, and
> > the cleanup code will free the memory anyway.
> >
> > Similarly, you don't need to explicitly close() file descriptors, or
> > release file locks (those obtained with flock/lockf/fcntl) as the OS
> > will do that.
> >
> > The sort of actions which might make sense to perform before calling
> > exit() are:
> >
> > + Writing a log entry reporting shutdown
> > + Deleting temporary files
> > + Deleting any SysV IPC structures which aren't shared with other
> >   processes.
> > + Informing other networked processes of termination, e.g. sending a
> >   "QUIT" command.
> > + Killing child processes.
>
> -
> To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Newbie question on malloc()

[Micha Feigin]

On Thu, Jun 03, 2004 at 09:42:51PM +0200, Jan-Benedict Glaw wrote:
> On Thu, 2004-06-03 04:34:27 +0300, Micha Feigin <michf@post.tau.ac.il>
> wrote in message <20040603013427.GD2562@luna.mooo.com>:
> > On Wed, Jun 02, 2004 at 08:37:34PM +0200, Jan-Benedict Glaw wrote:
> > > Well, that all depends on the definition of "OS". Any recent 32+ bit
> > > operating system with virtual memory capabilities and multi-processing
> > > will reclaim malloc()ed RAM upon process termination.
> > 
> > It has nothing to do with virtual memory. In any multi-processing OS
> > (any modern one probably) memory allocations are done by the kernel
> > since the kernel is the one managing the system's memory.
> > 
> > If the kernel doesn't have a bug, this memory is released on process
> > exit (unless it is still used by another process such as with fork with
> > virtual memory where it is marked copy on write).
> 
> Read again. There ARE (or at least: were) operating systems that
> intentionally did *not* implicitely free memory. If an application
> missed to free all allocated memory, it's lost. Forever.
> 

Single process operating systems could get away with doing that. Multi
process operating systems (and most modern ones are) can't afford that
luxury as it will allow one misbehaving process to kill the system
(maybe win 3 could get away with that as it was a cooperative multi
tasking system as opposed to a properly scheduled one and dynamic memory
there was a very sick thing using handles)

Maybe some special purpose embedded systems will do that to reduce
overhead but they will be hard to find probably.

> But modern OSes (those with at least have the capabilities mentioned
> above) will automatically reclaim all of a exit'ed process' memory.
> 
> MfG, JBG
> 
> -- 
>    Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
>    "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
>     fuer einen Freien Staat voll Freier B?rger" | im Internet! |   im Irak!
>    ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

Re: Newbie question on malloc()

[Glynn Clements]

John T. Williams wrote:

> but I believe that it is simply good practice to always be aware
> of memory you have dynamically allocated and always free it after you no
> more use for it.

As an "all other factors being equal ..." argument, I would agree with
that. But, most of the time, all other factors aren't equal.

There are many situations where keeping track of allocated memory
blocks would add significant complexity. E.g. when using heavily
linked data structures, it's common to copy pointers, so there may be
many pointers to the same block of memory.

In that situation, you can't just traverse the data structure, freeing
all of the pointers which you encounter; doing so would result not
only in freeing blocks multiple times, but quite possibly freeing
blocks which you will subsequently need to traverse into.

Another example: if you are storing strings in dynamically allocated
memory, you can "discard" some bytes from the beginning of the string
by advancing the pointer. However, you can't then pass the modified
pointer to free(); for that purpose, you would have to keep the
original pointer around as well.

In either case, you essentially have to add an additional layer of
memory management, so that allocating a block adds it to the set of
allocated blocks while freeing the block removes it.

For a short-lived program (i.e. a typical Unix command-line utility),
there is often no need to do this; often you can just allocate memory
as you need it and not to bother explicitly freeing anything.

Essentially, memory leaks are only an issue for programs which either
run indefinitely (e.g. daemons or interactive applications) or which
process large amounts of data in a linear sequence (i.e. you wouldn't
want "grep" to allocate a new block of memory for every line).

> In this way you don't get used to saying well if I miss
> it the operating system will clean it up for me. Please note I do not
> suggest that one should go to out of the way to free memory on abnormal
> termination, but its simply good practice to free memory when it is no
> longer being used in the normal behavior of your code. 

But that assumes that you know when memory is no longer being used. In
non-trivial programs, determining "liveness" can itself be
non-trivial.

When you release a structure which contains pointers to other
structures, you can only release any referenced structures when you
remove the last remaining reference. And it may not be straightforward
to determine whether other references exist.

There are a number of solutions, all of which have some drawbacks:

1. Reference counting. Not only does this requires significant
discipline (ensuring that you always increment/decrement reference
counts at the right points), but it doesn't cope with circular
references: if A references B and B references A, the reference counts
will be non-zero, so the structures won't be released even if nothing
else references either A or B.

2. Mark/Scan. This requires that you can reliably enumerate all
referencess to dynamic memory blocks. Unlike reference counting, this
scheme handles circular references correctly, but the effort involved
tends to make this approach impractical for anything other than
language interpreters or projects of such complexity that no other
solution is practical.

3. Copy everything; i.e. never duplicate or modify pointers. If you
need to copy data, copy the entire structure (and anything which it
references) rather than just the pointer. If you need a substring,
copy the relevant portion into a newly-allocated memory block. Not
only is this approach only feasible with "lightly" linked data
structures, it can easily result in significant bloat.

And, to keep to the original point (freeing memory upon termination),
if you're going to add your own memory management layer, one obvious
feature would be:

	extern int shutting_down;

	void my_free(void *p)
	{
		if (!shutting_down)
			free(p);
	}

I.e. explicitly skip calling free() if you know that you're about to
quit.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[John T. Williams]

According to the glibc documentation, it is not guaranteed to do so

http://www.gnu.org/software/libc/manual/html_node/Termination-Internals.html#Termination%20Internals






On Thu, 2004-06-03 at 19:24, Paul Gimpelj wrote:
> hi,
> pardon me for patching to this thread,
> 
> but, following this conversation,
> if files are left open at exit, does the operating system (linux) flush the
> i/o buffers to disk before closing files?
> This is not the case with windows 95/98  and dos.
> OR, does the _exit() function in the c runtime do it.
> Thanks.
> 
> thanks
> 
> regards,
> Paul
> 
> 
> ----- Original Message -----
> From: "John T. Williams" <jowillia@vt.edu>
> To: "Glynn Clements" <glynn.clements@virgin.net>
> Cc: "Micha Feigin" <michf@post.tau.ac.il>; "linux-c-programming"
> <linux-c-programming@vger.kernel.org>
> Sent: Thursday, June 03, 2004 6:25 PM
> Subject: Re: Newbie question on malloc()
> 
> 
> >
> > I think this is may be a somewhat misleading statement. Not freeing
> > memory before normal termination decreases re-usability of your code.
> > You never know when your main function might be reused verbatim as
> > another sub function. Failing to free memory before all reference leave
> > scope can lead to massive memory leeks. Further, the more memory your
> > program is holding at any given moment of operation the more pages it
> > holds, the more pages it holds the more likely any particular call to a
> > memory location will lead to a page fault, and page faults are
> > expensive. Freeing memory allows malloc to reassign that memory next
> > time its called. Now I know that this was a question about freeing
> > memory before termination, and therefore unless we are talking about a
> > situation where your code is being reused my last two arguments hold no
> > weight, but I believe that it is simply good practice to always be aware
> > of memory you have dynamically allocated and always free it after you no
> > more use for it. In this way you don't get used to saying well if I miss
> > it the operating system will clean it up for me. Please note I do not
> > suggest that one should go to out of the way to free memory on abnormal
> > termination, but its simply good practice to free memory when it is no
> > longer being used in the normal behavior of your code.
> >
> >
> >
> > On Thu, 2004-06-03 at 03:59, Glynn Clements wrote:
> > > Micha Feigin wrote:
> > >
> > > > > I have made a daemon in which dynamic memory is gotten
> > > > > by malloc(). Does the memory get free automatically without
> > > > > free() by the deamon  when the daemon process is killed?
> > > > > Thanks in advance.
> > > >
> > > > It does, but in general its not in good practice to count on process
> > > > exit for freeing memory (a good way to get memory leaks).
> > >
> > > This is incorrect.
> > >
> > > You shouldn't make an effort to return memory to the process' heap (by
> > > calling free()) if the program is about to terminate.
> > >
> > > Doing so consumes CPU time (and probably disk bandwidth, given that
> > > some of that memory will probably be swapped out) and doesn't provide
> > > any benefit (free() won't usually return memory to the OS and, in any
> > > case, all of the process' memory will be returned to the OS upon
> > > exit).
> > >
> > > The only valid reason for free()ing memory blocks upon termination is
> > > if the structure of the program is such that it isn't practical to
> > > avoid doing so. E.g. if data structures have matched setup/cleanup
> > > routines, and you need to call the cleanup code for other reasons, and
> > > the cleanup code will free the memory anyway.
> > >
> > > Similarly, you don't need to explicitly close() file descriptors, or
> > > release file locks (those obtained with flock/lockf/fcntl) as the OS
> > > will do that.
> > >
> > > The sort of actions which might make sense to perform before calling
> > > exit() are:
> > >
> > > + Writing a log entry reporting shutdown
> > > + Deleting temporary files
> > > + Deleting any SysV IPC structures which aren't shared with other
> > >   processes.
> > > + Informing other networked processes of termination, e.g. sending a
> > >   "QUIT" command.
> > > + Killing child processes.
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> linux-c-programming" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Newbie question on malloc()

[Glynn Clements]

Micha Feigin wrote:

> > Note that free() doesn't generally return memory to the OS; it usually
> > just records the fact that the specified region of memory is available
> > for re-use by subsequent calls to malloc() etc.
> 
> That would depend on the memory block size. Small memory blocks are
> held as malloc allocates whole pages at a time, large memory blocks are
> returned. Depending on size the OS may then also store them on caches
> in case a memory block of the same size is requested again.

That's what I meant by "generally".

Some implementations *may* return freed memory to the OS. GNU libc 2.x
does so for blocks above a certain size (the default is 128kb,
configurable via environment variables and mallopt()).

However, there may also be a limit on the number of such blocks (due
to kernel limits on the number of memory mappings), so you can't
drastically reduce the threshold.

Other implementations may or may not provide such a feature.

Also, kernel patches are available which disable this feature. Memory
which is obtained from anonymous mmap() (rather than brk()) isn't
restricted by setrlimit(RLIMIT_DATA), which makes it difficult to
enforce resource limits.

> Otherwise in a long running program that allocates a lot of memory on
> the start and then frees most of it would still be hogging all that
> memory.

That may still happen if the memory is allocated in many small chunks
rather than one large chunk.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[Glynn Clements]

Paul Gimpelj wrote:

> pardon me for patching to this thread,
> 
> but, following this conversation,
> if files are left open at exit, does the operating system (linux) flush the
> i/o buffers to disk before closing files?

It depends upon what you mean by "buffers" and "disk".

The OS (kernel) doesn't know anything about user-space buffers (as
used by the ANSI stdio functions), so it won't flush those.

However, the exit() function will flush any user-space buffers (to the
kernel's buffer cache). Also, returning from main() performs an
implicit call to exit(), so the same applies there. To exit without
flushing any buffers, you need to call _exit() instead.

Needless to say, if a program terminates abnormally (due to a fatal
signal), buffers aren't flushed.

OTOH, the kernel does know about the kernel buffer cache, which is
where write() etc store their data. This is often inadvertently
referred to as "disk", on the assumption that write() etc actually
write directly to disk (which was true for e.g. MS-DOS).

The buffer cache is shared between processes, and is flushed to disk
as and when the kernel deems it appropriate, or if a process uses
sync/fsync.

However, once a process has sent the data to the kernel (i.e. the
buffer cache), it doesn't make any difference what the process does
(call exit(), call _exit(), crash, or whatever). The kernel ensures
that the buffer cache will eventually be flushed to disk
(notwithstanding exceptional circumstances, e.g. hardware or power
failure).

> This is not the case with windows 95/98  and dos.

ANSI C dictates that both exit() and returning from main() will flush
any buffers. Ultimately, it's up to the C library which you use to
implement this correctly, but I don't recall any problems with either
Borland or Watcom under DOS, or either Watcom or MSVC under Windows.

Terminating by other means (e.g. TerminateProcess()) may or may not
flush buffers.

Sometimes, it's necessary to terminate a process immediately, without
any unwanted side-effects. E.g. if you catch SIGSEGV or SIGBUS,
there's a good chance that user-space memory has become corrupted. Any
attempt to call functions which access that memory (e.g. fflush()) may
result in further exceptions.

> OR, does the _exit() function in the c runtime do it.

_exit() specifically does not flush the stdio buffers.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Newbie question on malloc()

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Fri, 2004-06-04 02:44:57 +0300, Micha Feigin <michf@post.tau.ac.il>
wrote in message <20040603234457.GK2562@luna.mooo.com>:
> On Thu, Jun 03, 2004 at 09:42:51PM +0200, Jan-Benedict Glaw wrote:
> > Read again. There ARE (or at least: were) operating systems that
> > intentionally did *not* implicitely free memory. If an application
> > missed to free all allocated memory, it's lost. Forever.
> 
> Single process operating systems could get away with doing that. Multi
> process operating systems (and most modern ones are) can't afford that
> luxury as it will allow one misbehaving process to kill the system
> (maybe win 3 could get away with that as it was a cooperative multi
> tasking system as opposed to a properly scheduled one and dynamic memory
> there was a very sick thing using handles)
> 
> Maybe some special purpose embedded systems will do that to reduce
> overhead but they will be hard to find probably.

Right you are, as I am:) I don't claim that today's most common OSes
don't reclaim memory - they do. But there have been (and for sure
they're still used) special-purpose operating systems that don't do
that. It's like "all cars to have four wheels". At least in Italia,
you'll commonly see cars with three wheels:) Special purpose, but they
exist...

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Newbie question on malloc()

[Micha Feigin]

On Fri, Jun 04, 2004 at 03:11:50AM +0100, Glynn Clements wrote:
> 
> Micha Feigin wrote:
> 
> > > Note that free() doesn't generally return memory to the OS; it usually
> > > just records the fact that the specified region of memory is available
> > > for re-use by subsequent calls to malloc() etc.
> > 
> > That would depend on the memory block size. Small memory blocks are
> > held as malloc allocates whole pages at a time, large memory blocks are
> > returned. Depending on size the OS may then also store them on caches
> > in case a memory block of the same size is requested again.
> 
> That's what I meant by "generally".
> 
> Some implementations *may* return freed memory to the OS. GNU libc 2.x
> does so for blocks above a certain size (the default is 128kb,
> configurable via environment variables and mallopt()).
> 
> However, there may also be a limit on the number of such blocks (due
> to kernel limits on the number of memory mappings), so you can't
> drastically reduce the threshold.
> 
> Other implementations may or may not provide such a feature.
> 
> Also, kernel patches are available which disable this feature. Memory
> which is obtained from anonymous mmap() (rather than brk()) isn't
> restricted by setrlimit(RLIMIT_DATA), which makes it difficult to
> enforce resource limits.
> 
> > Otherwise in a long running program that allocates a lot of memory on
> > the start and then frees most of it would still be hogging all that
> > memory.
> 
> That may still happen if the memory is allocated in many small chunks
> rather than one large chunk.
> 

It may be close to the truth if the program allocates a lot of small
chunks of variable sizes. IIRC with malloc (on glibc) for each chunk
size for small sizes malloc will allocate a page (or group of pages)
and split them up to equal sized chunks.

Different sized chunks sit on different such chains.

This can take more memory but saves on fragmentation and easier
maintenance. For efficiency it may store the last free chunk or so
(don't remember) as cache, but it should store all freed memory (the
doubling technique for dynamic buffers). You add more memory if you
need more then you have, and free some up if you drop bellow a
threshold.

It would be grossly inefficient on system resources otherwise on not
much more efficient in time. Programs that need a memory cache should
handle it themselves.

> -- 
> Glynn Clements <glynn.clements@virgin.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>  +++++++++++++++++++++++++++++++++++++++++++
>  This Mail Was Scanned By Mail-seCure System
>  at the Tel-Aviv University CC.
>