Linux processes, tempfs and programs

Linux processes, tempfs and programs

[Stephen Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


This is not really PPC specific, but it is an embedded PPC system
that I'm working with, so...

My embedded system is structured such that the main user-mode
processes that are being run are downloaded and executed on demand.
I'm currently downloading the executable to an ext3 system on the
CompactFlash, but there is really no reason to use non-volatile
memory so I'm thinking to download to a tempfs directory and
execute from there.

But if I do that, I want to remove the program from the directory
after I start it, so that the file does not take up ram space. Will
that actually work? I'm using exec(2) to execute the program file
wherever it is downloaded. Will a subsequent unlink of the file
have a result, or will the file continue to take up space as
backing store for the executable?
- --
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCFSrcrPt1Sc2b3ikRApXCAKDJpcfBMIGj7aOQ46hh19o0eOV2XgCgnprQ
rIAecQJg+VpdFbo+gWXwWKk=
=p475
-----END PGP SIGNATURE-----

Re: Linux processes, tempfs and programs

[Eugene Surovegin]

On Thu, Feb 17, 2005 at 03:38:05PM -0800, Stephen Williams wrote:
> My embedded system is structured such that the main user-mode
> processes that are being run are downloaded and executed on demand.
> I'm currently downloading the executable to an ext3 system on the
> CompactFlash, but there is really no reason to use non-volatile
> memory so I'm thinking to download to a tempfs directory and
> execute from there.
> 
> But if I do that, I want to remove the program from the directory
> after I start it, so that the file does not take up ram space. Will
> that actually work? I'm using exec(2) to execute the program file
> wherever it is downloaded. Will a subsequent unlink of the file
> have a result, or will the file continue to take up space as
> backing store for the executable?

I think unlink will remove the file from directory (so you won't be
able to see it with ls), but it will still continue to to take space -  
you're right it will be used as backing store, at least for read-only 
segments, which can be discarded if memory is tight. Even if you mlock 
all executable in memory, I think there will be still at least one 
reference to this file, which will prevent freeing tmpfs memory.

--
Eugene

Re: Linux processes, tempfs and programs

[Stephen Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eugene Surovegin wrote:
| On Thu, Feb 17, 2005 at 03:38:05PM -0800, Stephen Williams wrote:
|>But if I do that, I want to remove the program from the directory
|>after I start it, so that the file does not take up ram space. Will
|>that actually work? I'm using exec(2) to execute the program file
|>wherever it is downloaded. Will a subsequent unlink of the file
|>have a result, or will the file continue to take up space as
|>backing store for the executable?
|
|
| I think unlink will remove the file from directory (so you won't be
| able to see it with ls), but it will still continue to to take space -
| you're right it will be used as backing store, at least for read-only
| segments, which can be discarded if memory is tight. Even if you mlock
| all executable in memory, I think there will be still at least one
| reference to this file, which will prevent freeing tmpfs memory.

Which is what I thought, and why I didn't do it that way in the
first place:-( That's 128+ Kbytes I'd rather have holding image
data:-(((

I've got shared libraries on the CF disk, it's fine (preferable) if
it pages out of them, but I don't want the program itself to reside
anywhere but in its memory image. (And I don't want to go writing to
the CF disk except for upgrades or sys admin stuff.)
- --
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCFTxvrPt1Sc2b3ikRAuy4AKCidKG/LYPDo9thwZag9S7JXLfSRACfUen6
N8ICW/n6BCR6OgJ5NMlW99Q=
=Fjct
-----END PGP SIGNATURE-----

Re: Linux processes, tempfs and programs

[Eugene Surovegin]

On Thu, Feb 17, 2005 at 04:43:27PM -0800, Eugene Surovegin wrote:
> On Thu, Feb 17, 2005 at 03:38:05PM -0800, Stephen Williams wrote:
> > My embedded system is structured such that the main user-mode
> > processes that are being run are downloaded and executed on demand.
> > I'm currently downloading the executable to an ext3 system on the
> > CompactFlash, but there is really no reason to use non-volatile
> > memory so I'm thinking to download to a tempfs directory and
> > execute from there.
> > 
> > But if I do that, I want to remove the program from the directory
> > after I start it, so that the file does not take up ram space. Will
> > that actually work? I'm using exec(2) to execute the program file
> > wherever it is downloaded. Will a subsequent unlink of the file
> > have a result, or will the file continue to take up space as
> > backing store for the executable?
> 
> I think unlink will remove the file from directory (so you won't be
> able to see it with ls), but it will still continue to to take space -  
> you're right it will be used as backing store, at least for read-only 
> segments, which can be discarded if memory is tight. Even if you mlock 
> all executable in memory, I think there will be still at least one 
> reference to this file, which will prevent freeing tmpfs memory.

A little correction, according to tmpfs doc, it lives completely in 
page cache, so I think memory is not wasted for unmodified sections of 
the loaded file (e.g. a second copy, when file is executed and loaded 
into user-space, isn't being made).

But as usual, make some measurements first :)

--
Eugene.

Re: Linux processes, tempfs and programs

[Stephen Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eugene Surovegin wrote:
| On Thu, Feb 17, 2005 at 04:43:27PM -0800, Eugene Surovegin wrote:
|
|>On Thu, Feb 17, 2005 at 03:38:05PM -0800, Stephen Williams wrote:

|>>But if I do that, I want to remove the program from the directory
|>>after I start it, so that the file does not take up ram space. Will
|>>that actually work? I'm using exec(2) to execute the program file
|>>wherever it is downloaded. Will a subsequent unlink of the file
|>>have a result, or will the file continue to take up space as
|>>backing store for the executable?

|>I think unlink will remove the file from directory (so you won't be
|>able to see it with ls), but it will still continue to to take space -
|>you're right it will be used as backing store, at least for read-only
|>segments, which can be discarded if memory is tight. Even if you mlock
|>all executable in memory, I think there will be still at least one
|>reference to this file, which will prevent freeing tmpfs memory.
|
|
| A little correction, according to tmpfs doc, it lives completely in
| page cache, so I think memory is not wasted for unmodified sections of
| the loaded file (e.g. a second copy, when file is executed and loaded
| into user-space, isn't being made).
|
| But as usual, make some measurements first :)

So far as I can see in my simplistic tests, this still takes twice
as much memory as leaving the executable on a disk somewhere. This
seems to be an inevitable consequence of execve on Linux 2.4.x. That's
a bummer. Any other suggestions?


- --
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCFjrjrPt1Sc2b3ikRAjwvAKCogiZ5i+OHdsGskhsfC97+jxl/ygCgqAql
P77LUWxMW87T7ReXVg27fj8=
=j9KI
-----END PGP SIGNATURE-----