Security question: "Text file busy" overwriting executables but not shared libraries?

Security question: "Text file busy" overwriting executables but not shared libraries?

[Rob Landley]

Anybody want to venture an opinion why overwriting executable files that are 
currently in use gives you a "text file busy" error, but overwriting shared 
libraries that are in use apparently works just fine (modulo a core dump if 
you aren't subtle about your run-time patching)?

Permissions are still enforced, but it seems to me somebody who cracks root 
on a system could potentially modify the behavior of important system daemons 
without changing their process ID numbers.

Did I miss something somewhere?

Rob

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Alexander Viro]

On Tue, 2 Oct 2001, Rob Landley wrote:

> Anybody want to venture an opinion why overwriting executable files that are 
> currently in use gives you a "text file busy" error, but overwriting shared 
> libraries that are in use apparently works just fine (modulo a core dump if 
> you aren't subtle about your run-time patching)?
> 
> Permissions are still enforced, but it seems to me somebody who cracks root 
> on a system could potentially modify the behavior of important system daemons 
> without changing their process ID numbers.
> 
> Did I miss something somewhere?

Somebody who cracks root can attach gdb to a daemon, modify the contents of
its text segment and detach.  No need to change any files...

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jesse Pollard]

Alexander Viro <viro@math.psu.edu>:
> On Tue, 2 Oct 2001, Rob Landley wrote:
> 
> > Anybody want to venture an opinion why overwriting executable files that are 
> > currently in use gives you a "text file busy" error, but overwriting shared 
> > libraries that are in use apparently works just fine (modulo a core dump if 
> > you aren't subtle about your run-time patching)?
> > 
> > Permissions are still enforced, but it seems to me somebody who cracks root 
> > on a system could potentially modify the behavior of important system daemons 
> > without changing their process ID numbers.
> > 
> > Did I miss something somewhere?
> 
> Somebody who cracks root can attach gdb to a daemon, modify the contents of
> its text segment and detach.  No need to change any files...

True, but the original problem still appears to be a bug.

Even the owner of the file should not be able to write to a busy executable,
whether it is a shared library, or an executable image. Remove it, yes.
Create a new one (in a different inode) -  yes.

But not modify a busy executable.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Jesse Pollard <pollard@tomcat.admin.navo.hpc.mil> writes:

> Alexander Viro <viro@math.psu.edu>:
> > On Tue, 2 Oct 2001, Rob Landley wrote:
> > 
> > > Anybody want to venture an opinion why overwriting executable files that are
> 
> > > currently in use gives you a "text file busy" error, but overwriting shared
> 
> > > libraries that are in use apparently works just fine (modulo a core dump if
> 
> > > you aren't subtle about your run-time patching)?
> > > 
> > > Permissions are still enforced, but it seems to me somebody who cracks root
> 
> > > on a system could potentially modify the behavior of important system
> daemons
> 
> > > without changing their process ID numbers.
> > > 
> > > Did I miss something somewhere?
> > 
> > Somebody who cracks root can attach gdb to a daemon, modify the contents of
> > its text segment and detach.  No need to change any files...
> 
> True, but the original problem still appears to be a bug.
> 
> Even the owner of the file should not be able to write to a busy executable,
> whether it is a shared library, or an executable image. Remove it, yes.
> Create a new one (in a different inode) -  yes.
> 
> But not modify a busy executable.

Have ld-linux.so set the MAP_DENYWRITE bit when it is mapping
the library.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Rob Landley]

On Wednesday 03 October 2001 14:06, Eric W. Biederman wrote:

> > But not modify a busy executable.
>
> Have ld-linux.so set the MAP_DENYWRITE bit when it is mapping
> the library.

And of course since the FSF wrote it, it's not quite that simple...

>/* The right way to map in the shared library files is MAP_COPY, which
>   makes a virtual copy of the data at the time of the mmap call; this
>   guarantees the mapped pages will be consistent even if the file is
>   overwritten.  Some losing VM systems like Linux's lack MAP_COPY.  All we
>   get is MAP_PRIVATE, which copies each page when it is modified; this
>   means if the file is overwritten, we may at some point get some pages
>   from the new version after starting with pages from the old version.  */

I.E. it seems like they go out of their way to ALLOW writing to the libaries. 
 (I assume they KNOW the difference between MAP_DENYWRITE, MAP_COPY, and 
MAP_PRIVATE...?)

This look right to anybody else?  Or am I about to wander into weird 
side-effect land?  (Is there a reason they DON'T want a read-only mapping?  
Are they writing data into those pages, perhaps doing the linking fixup 
stuff?  What?)

--- elf/dl-load.bak Wed Oct  3 18:53:37 2001
+++ elf/dl-load.c   Wed Oct  3 18:55:57 2001
@@ -48,7 +48,7 @@
    means if the file is overwritten, we may at some point get some pages
    from the new version after starting with pages from the old version.  */
 #ifndef MAP_COPY
-# define MAP_COPY      MAP_PRIVATE
+# define MAP_COPY      MAP_DENYWRITE
 #endif
 
 /* Some systems link their relocatable objects for another base address

I should just try this and see what it does.  On a machine I don't mind 
reinstalling from scratch.  Which means I need to dig up a spare keyboard for 
my junk machine...  (And figure out how to get glibc's ./configure script to 
realise that linuxthreads is, in fact, there in the source directory.  It's 
right there.  Use it.  Don't yell at me it's not there.  I didn't make this 
SRPM, I changed one line...  Sigh...)

In the morning...

Rob

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Rob Landley <landley@trommello.org> writes:

> On Wednesday 03 October 2001 14:06, Eric W. Biederman wrote:
> 
> > > But not modify a busy executable.
> >
> > Have ld-linux.so set the MAP_DENYWRITE bit when it is mapping
> > the library.
> 
> And of course since the FSF wrote it, it's not quite that simple...
> 
> >/* The right way to map in the shared library files is MAP_COPY, which
> >   makes a virtual copy of the data at the time of the mmap call; this
> >   guarantees the mapped pages will be consistent even if the file is
> >   overwritten.  Some losing VM systems like Linux's lack MAP_COPY.  All we
> >   get is MAP_PRIVATE, which copies each page when it is modified; this
> >   means if the file is overwritten, we may at some point get some pages
> >   from the new version after starting with pages from the old version.  */
> 
> I.E. it seems like they go out of their way to ALLOW writing to the libaries. 
>  (I assume they KNOW the difference between MAP_DENYWRITE, MAP_COPY, and 
> MAP_PRIVATE...?)
> 
> This look right to anybody else?  Or am I about to wander into weird 
> side-effect land?  (Is there a reason they DON'T want a read-only mapping?  
> Are they writing data into those pages, perhaps doing the linking fixup 
> stuff?  What?)

You definentily need to do some writing to do the fixups.
The deny write solves the problem of somone potentially writing to the
file at a later date.
Probably what is needed is:

#ifndef MAP_COPY
# ifdef MAP_DENYWRITE
#  define MAP_COPY (MAP_PRIVATE | MAP_DENYWRITE)
# else
#  define MAP_COPY MAP_PRIVATE
# endif
#endif

> 
> --- elf/dl-load.bak Wed Oct  3 18:53:37 2001
> +++ elf/dl-load.c   Wed Oct  3 18:55:57 2001
> @@ -48,7 +48,7 @@
>     means if the file is overwritten, we may at some point get some pages
>     from the new version after starting with pages from the old version.  */
>  #ifndef MAP_COPY
> -# define MAP_COPY      MAP_PRIVATE
> +# define MAP_COPY      MAP_DENYWRITE
>  #endif
>  
>  /* Some systems link their relocatable objects for another base address
> 
> I should just try this and see what it does.  On a machine I don't mind 
> reinstalling from scratch.  Which means I need to dig up a spare keyboard for 
> my junk machine...  (And figure out how to get glibc's ./configure script to 
> realise that linuxthreads is, in fact, there in the source directory.  It's 
> right there.  Use it.  Don't yell at me it's not there.  I didn't make this 
> SRPM, I changed one line...  Sigh...)
> 
> In the morning...

For testing you can do ./ld-linux.so program to run a program under to
see if it actually works. 

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Alexander Viro]

On 3 Oct 2001, Eric W. Biederman quoted:

> > >/* The right way to map in the shared library files is MAP_COPY, which
> > >   makes a virtual copy of the data at the time of the mmap call; this
> > >   guarantees the mapped pages will be consistent even if the file is
> > >   overwritten.  Some losing VM systems like Linux's lack MAP_COPY.  All we
> > >   get is MAP_PRIVATE, which copies each page when it is modified; this
> > >   means if the file is overwritten, we may at some point get some pages
> > >   from the new version after starting with pages from the old version.  */

IMO it needs a slight correction.

+ /* Unfortunately, that is not an option, since losing bloatware like GNU's
+    relies heavily on equally bloated shared libraries and use of MAP_COPY
+    would eat memory with no mercy.  OTOH, implementing it might be a good
+    idea, since results would force people to switch to something less obese */

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

In article <01100319203903.00728@localhost.localdomain>,
Rob Landley  <landley@trommello.org> wrote:
>
>I.E. it seems like they go out of their way to ALLOW writing to the libaries. 
> (I assume they KNOW the difference between MAP_DENYWRITE, MAP_COPY, and 
>MAP_PRIVATE...?)

Note that the kernel will refuse to honour MAP_DENYWRITE from user
space, so I'm afraid that changing ld.so won't do a thing.

The reason the kernel refuses to honour it, is that MAP_DENYWRITE is an
excellent DoS-vehicle - you just mmap("/etc/passwd") with MAP_DENYWRITE,
and even root cannot write to it.. Vary nasty.

Which is why the kernel only allows it when the binary loader itself
sets the flag, because security-conscious application writers are
already aware of the "oh, a running binary may not be writable" issues.

So sorry..

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Alexander Viro]

On Thu, 4 Oct 2001, Linus Torvalds wrote:

> The reason the kernel refuses to honour it, is that MAP_DENYWRITE is an
> excellent DoS-vehicle - you just mmap("/etc/passwd") with MAP_DENYWRITE,
> and even root cannot write to it.. Vary nasty.

<nit>
I _really_ doubt that something does write() on /etc/passwd.  Create a
file and rename it over the thing - sure, but that's it.
</nit>

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On Thu, 4 Oct 2001, Alexander Viro wrote:
> <nit>
> I _really_ doubt that something does write() on /etc/passwd.  Create a
> file and rename it over the thing - sure, but that's it.
> </nit>

Well, yeah, bad choice. Can you believe /var/run/utmp or similar?

And yes, we could add checks for the thing being executable before we
accept MAP_DENYWRITE instead of just ignoring the flag from user space.
Nobody has cared enough to make the effort.

Until now?

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Richard Gooch]

Alexander Viro writes:
> <nit>
> I _really_ doubt that something does write() on /etc/passwd.  Create a
> file and rename it over the thing - sure, but that's it.
> </nit>

# vi /etc/passwd

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Alexander Viro <viro@math.psu.edu> writes:

> On 3 Oct 2001, Eric W. Biederman quoted:
> 
> > > >/* The right way to map in the shared library files is MAP_COPY, which
> > > >   makes a virtual copy of the data at the time of the mmap call; this
> > > >   guarantees the mapped pages will be consistent even if the file is
> > > > overwritten.  Some losing VM systems like Linux's lack MAP_COPY.  All we
> 
> > > >   get is MAP_PRIVATE, which copies each page when it is modified; this
> > > >   means if the file is overwritten, we may at some point get some pages
> > > > from the new version after starting with pages from the old version.  */
> 
> 
> IMO it needs a slight correction.
> 
> + /* Unfortunately, that is not an option, since losing bloatware like GNU's
> +    relies heavily on equally bloated shared libraries and use of MAP_COPY
> +    would eat memory with no mercy.  OTOH, implementing it might be a good
> + idea, since results would force people to switch to something less obese */

Hmm. Perhaps. But if we went there we would need to add something like.

/* But finding a less obese platform to run these less obese libraries is a
   challenge.  Unix clones like UZI have been shown to run a complete system
   including user space binaries in just 64KB of RAM, on systems
   originally designed to run CPM.  But today you can't find a general
   purpose kernel whose binary much less it footprint fits in 256KB.
   It seems bloatware is everywhere.
 */

I have days when I'm frustrated by the size of both glibc and the
linux kernel.  stripped both the linux kernel and glibc are comparable
in size.  Though I think the 400KB of compressed glibc-2.1.2 is
actually smaller than the kernel for the most part.  I have to strip
off practically everthing to get a useable bzImage under 400KB.

So any good ideas on how to get the size of linux down?

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[George Greer]

On Thu, 4 Oct 2001, Linus Torvalds wrote:

>Which is why the kernel only allows it when the binary loader itself
>sets the flag, because security-conscious application writers are
>already aware of the "oh, a running binary may not be writable" issues.

One of the methods I tried to use to stop a fork()-bomb was to zero the
executable in question to force it to crash. No such luck, reboot it was.
Not that I can think of any other useful application of said behavior.

-- 
George Greer, greerga@m-l.org
http://www.m-l.org/~greerga/

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[CaT]

On Thu, Oct 04, 2001 at 12:15:01AM -0600, Eric W. Biederman wrote:
> I have days when I'm frustrated by the size of both glibc and the
> linux kernel.  stripped both the linux kernel and glibc are comparable
> in size.  Though I think the 400KB of compressed glibc-2.1.2 is
> actually smaller than the kernel for the most part.  I have to strip
> off practically everthing to get a useable bzImage under 400KB.
> 
> So any good ideas on how to get the size of linux down?

Mind if I ask why you need a bzimage under 400kb? Just curious as I've
never had the need. (And I can see needing it less then 1.4meg - are you
trying to get a kernel AND a ramdisk on the one floppy?)

-- 
CaT        "As you can expect it's really affecting my sex life. I can't help
           it. Each time my wife initiates sex, these ejaculating hippos keep
           floating through my mind."
                - Mohd. Binatang bin Goncang, Singapore Zoological Gardens

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Ville Herva]

On Thu, Oct 04, 2001 at 12:15:01AM -0600, you [Eric W. Biederman] claimed:
> 
> <snip size of glibc>

Where size is an issue, diet libc might be an alternative:

http://www.fefe.de/dietlibc/

(287kB statically linked zsh is not too shabby, I reckon.)

That and things like busybox:

http://busybox.lineo.com/

I have no suggestion wrt the kernel, though.


-- v --

v@iki.fi

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[john slee]

[ cc list trimmed ]

On Thu, Oct 04, 2001 at 06:21:27PM +1000, CaT wrote:
> On Thu, Oct 04, 2001 at 12:15:01AM -0600, Eric W. Biederman wrote:
> > I have days when I'm frustrated by the size of both glibc and the
> > linux kernel.  stripped both the linux kernel and glibc are comparable
> > in size.  Though I think the 400KB of compressed glibc-2.1.2 is
> > actually smaller than the kernel for the most part.  I have to strip
> > off practically everthing to get a useable bzImage under 400KB.
> > 
> > So any good ideas on how to get the size of linux down?
> 
> Mind if I ask why you need a bzimage under 400kb? Just curious as I've
> never had the need. (And I can see needing it less then 1.4meg - are you
> trying to get a kernel AND a ramdisk on the one floppy?)

plenty of reasons.  i'm building a compactflash-based linux router which
will only have 16mb of flash for the entire system... saving 100kb means
you can fit a few extra userspace tools in there...

-rwxr-xr-x    1 indigoid indigoid    54444 Oct  4 18:30 boa*

j.

-- 
R N G G   "Well, there it goes again... And we just sit 
 I G G G   here without opposable thumbs." -- gary larson

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[CaT]

On Thu, Oct 04, 2001 at 06:35:07PM +1000, john slee wrote:
> > Mind if I ask why you need a bzimage under 400kb? Just curious as I've
> > never had the need. (And I can see needing it less then 1.4meg - are you
> > trying to get a kernel AND a ramdisk on the one floppy?)
> 
> plenty of reasons.  i'm building a compactflash-based linux router which
> will only have 16mb of flash for the entire system... saving 100kb means
> you can fit a few extra userspace tools in there...
> 
> -rwxr-xr-x    1 indigoid indigoid    54444 Oct  4 18:30 boa*

Well, duh. :)

Thanks. :)

-- 
CaT        "As you can expect it's really affecting my sex life. I can't help
           it. Each time my wife initiates sex, these ejaculating hippos keep
           floating through my mind."
                - Mohd. Binatang bin Goncang, Singapore Zoological Gardens

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Andreas Schwab]

ebiederm@xmission.com (Eric W. Biederman) writes:

|> So any good ideas on how to get the size of linux down?

How about linux-0.01?

SCNR.

Andreas.

-- 
Andreas Schwab                                  "And now for something
Andreas.Schwab@suse.de				completely different."
SuSE Labs, SuSE GmbH, Schanzäckerstr. 10, D-90443 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Erik Andersen]

On Thu Oct 04, 2001 at 11:30:19AM +0300, Ville Herva wrote:
> On Thu, Oct 04, 2001 at 12:15:01AM -0600, you [Eric W. Biederman] claimed:
> > 
> > <snip size of glibc>
> 
> Where size is an issue, diet libc might be an alternative:
> 
> http://www.fefe.de/dietlibc/
> 
> (287kB statically linked zsh is not too shabby, I reckon.)

uClibc is also a nice alternative.  Works just great and uses glibc
header files.  I only fully support shared libs on x86 and arm
at the moment.  

http://cvs.uclinux.org/uClibc.html

(I need to update the webpage sometime)

> That and things like busybox:
> 
> http://busybox.lineo.com/

Why thanks.  I've sure worked hard to make it be nice and small...

 -Erik

--
Erik B. Andersen   email:  andersee@debian.org, formerly of Lineo
--This message was written using 73% post-consumer electrons--

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[John Levon]

On Thu, Oct 04, 2001 at 05:38:12AM +0000, Linus Torvalds wrote:

> The reason the kernel refuses to honour it, is that MAP_DENYWRITE is an
> excellent DoS-vehicle - you just mmap("/etc/passwd") with MAP_DENYWRITE,
> and even root cannot write to it.. Vary nasty.

why is MAP_EXECUTABLE dealt with in the same way then ?

john

-- 
" It is quite humbling to realize that the storage occupied by the longest line
from a typical Usenet posting is sufficient to provide a state space so vast
that all the computation power in the world can not conquer it."
	- Dave Wallace

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

CaT <cat@zip.com.au> writes:

> On Thu, Oct 04, 2001 at 12:15:01AM -0600, Eric W. Biederman wrote:
> > I have days when I'm frustrated by the size of both glibc and the
> > linux kernel.  stripped both the linux kernel and glibc are comparable
> > in size.  Though I think the 400KB of compressed glibc-2.1.2 is
> > actually smaller than the kernel for the most part.  I have to strip
> > off practically everthing to get a useable bzImage under 400KB.
> > 
> > So any good ideas on how to get the size of linux down?
> 
> Mind if I ask why you need a bzimage under 400kb? Just curious as I've
> never had the need. (And I can see needing it less then 1.4meg - are you
> trying to get a kernel AND a ramdisk on the one floppy?)

floppies have lots of room.

I'd like to get a kernel, ramdisk, and some hw initialization code all
on a 256KB ROM.  I have my ramdisk down to about 14KB compressed.  I
have my hw initialization code down to 32KB uncompressed (and I might
be able to reduce that further). So I want something like a 192KB
(compressed) linux kernel.   

If I had that some of the hard problems of with linuxBIOS would just
drop away.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Andreas Schwab <schwab@suse.de> writes:

> ebiederm@xmission.com (Eric W. Biederman) writes:
> 
> |> So any good ideas on how to get the size of linux down?
> 
> How about linux-0.01?

There might be some fodder there, but I doubt it.  I have
played with linux-lite-v1.00 (which is something like linux-1.09).
And couldn't get any really compelling results.  Plus for it to be
useful I would still need to backport all of the driver API's from
2.4.x.

Just for a note, UZI has been ported to x86 as UZIX so a 32KB kernel
(without a network stack is achievable on x86).  If I can get a core
kernel size with no drivers down to 64KB or less I would be happy.

So far I haven't been able to come up with anything satisfying.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:

> On Thu, 4 Oct 2001, Alexander Viro wrote:
> > <nit>
> > I _really_ doubt that something does write() on /etc/passwd.  Create a
> > file and rename it over the thing - sure, but that's it.
> > </nit>
> 
> Well, yeah, bad choice. Can you believe /var/run/utmp or similar?
> 
> And yes, we could add checks for the thing being executable before we
> accept MAP_DENYWRITE instead of just ignoring the flag from user space.
> Nobody has cared enough to make the effort.
> 
> Until now?

Hmm.  Before I volunteer I need to think this thing out.  I orginally
missed the clearing of MAP_DENYWRITE in the arch specific code.  

First what user space really wants is the MAP_COPY.  Which is
MAP_PRIVATE with the guarantee that they don't see anyone else's changes.  

Just skimming /lib most libraries are only rw by root so the case we
are protecting ourselves against is fumble fingered administrators.
The two fumbles in particular are fumbling the permissions, and
accidentaly writing to a shared library.

given that MAP_DENYWRITE does remove unlink permission for most uses it
can be worked around.  

We already allow user space applications to make arbitrary files
MAP_DENYWRITE simply by executing them, and the only restriction is
that MAP_DENYWRITE only persists while user space has the file open.
So I guess allowing it in mmap is not actually a problem, as we can
already do that. 

At the same time there are cases where it is unacceptable to stop
people from writing to a file just because you have read access to it,
and you open the file.  Even having write access to a file isn't
enough.  So you really need to have execute and read permissions on a
file for this to be reasonable.

The one downside of requiring libraries to be executable is that
tricks like preventing. /lib/ld-linux.so.2 /mnt/noexec/bin/true is
a little harder.  

A remaining question was for newer kernels should MAP_DENYWRITE fail
if you don't have execute permissions, or should it just be a strong
hint.

Having MAP_DENYWRITE fail on filesystems that are mounted noexec and
having a dynamic loader that tests looks like it would be easier to
enforce a noexec policy for untrusted mounts.

So the code will need to look something like.
if (flags & MAP_DENYWRITE) {
	struct inode *inode = file->f_dentry->inode;
	if (IS_NOEXEC(inode) || !ISREG(inode->i_mode) || 
		(permission(inode, MAY_EXEC) != 0)) {
		return -EACCESS;
        }
}


Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On 4 Oct 2001, Eric W. Biederman wrote:
>
> First what user space really wants is the MAP_COPY.  Which is
> MAP_PRIVATE with the guarantee that they don't see anyone else's changes.

Which is a completely idiotic idea, and which is only just another example
of how absolutely and stunningly _stupid_ Hurd is.

The thing with MAP_COPY is that how do you efficiently _detect_ somebody
elses changes on a page that you haven't even read in yet?

So you have a few choices, all bad:

 - immediately reading in everything, basically turning the mmap() into a
   read. Obviously a bad idea.

 - mark the inode as a "copy" inode, and whenever somebody writes to it,
   you not only make sure that you do copy-on-write on the page cache page
   (which, btw, is pretty much impossible - how did you intend to find all
   the other _non_COPY_ users that _want_ coherency).

   You also have to make sure that if somebody changes the page, you have
   to read in the old contents first (not normally needed for most
   changes that write over at least a full block), but you also have to
   save the old page somewhere so that the mapping can use it if it faults
   it in later. And how the hell do you do THAT? Especially as you can
   have multiple generations of inodes with different sets of "MAP_COPY"
   on different contents..

   In short, now you need filesystem versioning at a per-page level etc.

Trust me. The people who came up with MAP_COPY were stupid. Really. It's
an idiotic concept, and it's not worth implementing.

And this all for what is a administration bug in the first place.

In short: just say NO TO DRUGS, and maybe you won't end up like the Hurd
people.

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Richard Gooch]

Linus Torvalds writes:
> 
> On 4 Oct 2001, Eric W. Biederman wrote:
> >
> > First what user space really wants is the MAP_COPY.  Which is
> > MAP_PRIVATE with the guarantee that they don't see anyone else's changes.
> 
> Which is a completely idiotic idea, and which is only just another
> example of how absolutely and stunningly _stupid_ Hurd is.

Indeed. If you're updated a shared library, why not *create a new
file* and then rename it?!? That lets running programmes work fine,
and new programmes will get the new library. Also, the following
construct makes a lot of sense:
	ld -shared -o libfred.so *.o || mv libfred.so /usr/local/lib

Why? Because if ld(1) fails for some reason, and ends up writing a
short file, *you don't want to install the bloody thing*!!! Any new
user would be stuffed (no way around that, even with MAP_COPY).
I don't want to install/upgrade to a half-working library. What's the
point in that?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Alexander Viro]

On Thu, 4 Oct 2001, Linus Torvalds wrote:

>    In short, now you need filesystem versioning at a per-page level etc.

*ding* *ding* *ding* we have a near winner.  Remember, folks, Hurd had been
started by people who not only don't understand UNIX, but detest it.
ITS/TWENEX refugees.  And semantics in question comes from there -
they had "open and make sure that anyone who tries to modify will get
a new version, leaving one we'd opened unchanged".

> Trust me. The people who came up with MAP_COPY were stupid. Really. It's
> an idiotic concept, and it's not worth implementing.

Well, actually that's a concept that made sense on system we got mmap from[1]
They just want infection to be complete.

[1] cue Tom Lehrer singing "I got it from Agnes, she got it from Jim"

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Andreas Schwab]

Richard Gooch <rgooch@ras.ucalgary.ca> writes:

|> Linus Torvalds writes:
|> > 
|> > On 4 Oct 2001, Eric W. Biederman wrote:
|> > >
|> > > First what user space really wants is the MAP_COPY.  Which is
|> > > MAP_PRIVATE with the guarantee that they don't see anyone else's changes.
|> > 
|> > Which is a completely idiotic idea, and which is only just another
|> > example of how absolutely and stunningly _stupid_ Hurd is.
|> 
|> Indeed. If you're updated a shared library, why not *create a new
|> file* and then rename it?!? That lets running programmes work fine,
|> and new programmes will get the new library. Also, the following
|> construct makes a lot of sense:
|> 	ld -shared -o libfred.so *.o || mv libfred.so /usr/local/lib
                                     ^^

That || should be &&, otherwise you are doing exactly the opposite of what
you want.

Andreas.

-- 
Andreas Schwab                                  "And now for something
Andreas.Schwab@suse.de				completely different."
SuSE Labs, SuSE GmbH, Schanzäckerstr. 10, D-90443 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Richard Gooch]

Andreas Schwab writes:
> Richard Gooch <rgooch@ras.ucalgary.ca> writes:
> 
> |> Linus Torvalds writes:
> |> > 
> |> > On 4 Oct 2001, Eric W. Biederman wrote:
> |> > >
> |> > > First what user space really wants is the MAP_COPY.  Which is
> |> > > MAP_PRIVATE with the guarantee that they don't see anyone else's changes.
> |> > 
> |> > Which is a completely idiotic idea, and which is only just another
> |> > example of how absolutely and stunningly _stupid_ Hurd is.
> |> 
> |> Indeed. If you're updated a shared library, why not *create a new
> |> file* and then rename it?!? That lets running programmes work fine,
> |> and new programmes will get the new library. Also, the following
> |> construct makes a lot of sense:
> |> 	ld -shared -o libfred.so *.o || mv libfred.so /usr/local/lib
> 
> That || should be &&, otherwise you are doing exactly the opposite
> of what you want.

Yeah. Of course. Brain fart. Fingers faster than brain syndrome...

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Wow what a wild spin off of my stream of consciousness.

Linus Torvalds <torvalds@transmeta.com> writes:

> On 4 Oct 2001, Eric W. Biederman wrote:
> >
> > First what user space really wants is the MAP_COPY.  Which is
> > MAP_PRIVATE with the guarantee that they don't see anyone else's changes.
> 
> Which is a completely idiotic idea, and which is only just another example
> of how absolutely and stunningly _stupid_ Hurd is.

Well in this case it is Mach not Hurd, and I wouldn't be suprised if
you could find MAP_COPY in selected BSDs.  The semantics wanted are
very reasonable.  You only want to see your changes to a given file. 
In practice there is no reason that anyone needs to actually change
the file so MAP_PRIVATE | MAP_DENYWRITE is much more sensible (at
least implementation wise).

> The thing with MAP_COPY is that how do you efficiently _detect_ somebody
> elses changes on a page that you haven't even read in yet?

Definentily.

> Trust me. The people who came up with MAP_COPY were stupid. Really. It's
> an idiotic concept, and it's not worth implementing.

I quite agree that MAP_COPY is not worth implementing.  And I never
said otherwise.  I only mentioned so I could think what the
alternatives where and what it's benefits really were.

> And this all for what is a administration bug in the first place.

Probably.  I can't think of any other cases where this could trigger.

However I think is is sensible to export MAP_DENYWRITE to user space.
It cheaply closes the administration bug, it is partially exported
already, and can even allow dynamic loaders to follow the kernel
noexec policy.  

The latter looks like an actual advantage over MAP_COPY.  Though
somehow MAP_EXECUTABLE sounds like a better name...

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Linus Torvalds wrote:
> Trust me. The people who came up with MAP_COPY were stupid. Really. It's
> an idiotic concept, and it's not worth implementing.

I can think of an efficiency-related use for MAP_COPY, and it has
nothing to do with shared libraries:

 - An editor using mmap() to read a file.

The existing semantics require that you either call read() and waste
(potentially shared) memory to do this, or use MAP_PRIVATE and then
deliberately page in and dirty all of the file's pages.

Neither of these seem to be the most efficient way to launch an editor.

cheers,
-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On Sat, 13 Oct 2001, Jamie Lokier wrote:
>
> I can think of an efficiency-related use for MAP_COPY, and it has
> nothing to do with shared libraries:
>
>  - An editor using mmap() to read a file.

No, you're thinking the wrong way.

Trust me, MAP_COPY really _is_ stupid, and the Hurd is a piece of crap.

People who think MAP_COPY is a good idea are people who cannot think about
the implications of it, and cannot think about the alternatives.

In particular, you claim that you could use "mmap()" for "read()", and
speed up the application that way. Ok, fair enough.

Now, somebody who _isn't_ stupid (and that, of course, is me), immediately
goes "well, _duh_, why don't you speed up read() instead?".

The fact is, all the problems that "MAP_COPY" has just go away if you
instead of thinking about a mmap(), you think about doing a "read()" and
just marking the pages PAGE_COPY if they are exclusive.

In short: MAP_COPY is braindamaged, because it doesn't have enough
information at the right level to do a reasonable job of it. What people
want to use it for is really to emulate "read()" efficiently using mmap,
and _nothing_ else. That is the only reason for it ever existing, and the
fact is, that clearly shows just how _stupid_ the whole thing is.

You migth as well just do a read() in the first place.

Your arguments are
 - read() implies a memcpy()
 - read() dirties pages and causes more memory pressure

but you don't actually _question_ those arguments.

I will tell you that doing a read() that _acts_ like the MAP_COPY you so
want is a LOT easier than doing MAP_COPY in the first place.

Why?

 - a read() call doesn't have any "history" - it doesn't leave (bogus)
   VM data around like MAP_COPY does. MAP_COPY says "I want these pages to
   have the contents they did _when_I_did_the_mapping_", which is a
   temporal shift that just doesn't make sense in any sane VM model, and
   which inherently implies versioning.

 - a read() can fairly easily just do the optimization

	(a) if we're reading a large area
	(b) if the offset and the destination are page-aligned
	(c) if the page is exclusive (ie no existing other owners)
		then
	just do the page move instead of the copy, and mark the page as
	PAGE_COPY

   Every other use of the page that can change it (ie a shared writable
   mapping, or a "write()" call) will now check the PAGE_COPY bit on the
   _page_, and just say "ok, I'll allocate a new page, and atomically
   switch the ones, and leave the old page untouched and remove it from
   the page cache"

   (And the swap-out logic has to turn a PAGE_COPY page into a swap-cache
   page - this is the real downside, because it implies that we will have
   to write it out to swap if we're low on memory, unlike a real mmap)

Notice? Same as MAP_COPY, but without any global state.

And notice how this is actually conceptually much closer to what you
actually _want_ to use MAP_COPY for.

Could we implement MAP_COPY as such a read()? Yes, sure. But that's just
confusing the issue - why call it a mmap() at all, when it isn't. The day
when Hurd is so common that we want to emulate its braindamages is not
going to be in my life-time, I suspect.

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Rik van Riel]

On Sat, 13 Oct 2001, Linus Torvalds wrote:

> Trust me, MAP_COPY really _is_ stupid, and the Hurd is a piece of crap.

Isn't it a Mach thing ?

> People who think MAP_COPY is a good idea are people who cannot think about
> the implications of it, and cannot think about the alternatives.

I guess thinking about the implications will come when
the Hurd people seriously start porting their beast to
other microkernels, say L4 ;)

This should be a spectacle worth watching (from a safe
distance).

cheers,

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/  (volunteers needed)

http://www.surriel.com/		http://distro.conectiva.com/

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Pablo Alcaraz]

Whatever will be the chosen solution, it would have to allow to 
overwrite all the executables and libraries files (if we have enough 
permissions).

Because:

- If I overwrite a shared library and then one running program crash, it 
will be my fault (as system administrator) or mistake.. ;-)

- It is probable that one file library is updated within one more global 
update, then probably I restart later the new demon or program. So if 
the program crash I'll fix the problem eventually.

- The previous version of a file library that I am replacing can depend 
on another file that the installer of the new version of the program 
simply erases it. For example:

a.so depends of b.so

but

a_new_version.so does not depend of b.so.

When I or an installer install the new program version, me or the 
installer erase b.so because the new version doesn't use it.

So, that it matters if a program can or can't access to the old version 
of a.so if b.so was erased?

And eventually, if I decide to update a library, I would have to do it 
(I suspect it would be the same case with executables files). It doesn't 
the matter if the change implies a fault in a running program.


It can be that this serves so that a hacker can attack the system... or 
I could hang a program when this is not my objective. Maybe a flag in 
/proc/somewhere would be am useful thing:

- if it's 1, I can overwrite all the libraries and executables files (If 
I've permission, etc.);

- if it's 0, I can not overwrite anything If it's in use.

I only want that everybody respect my right to do the wrong or stupid 
thing. This is an system administrator right :-)


Pablo

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Linus Torvalds wrote:
> On Sat, 13 Oct 2001, Jamie Lokier wrote:
> > I can think of an efficiency-related use for MAP_COPY, and it has
> > nothing to do with shared libraries:
> >
> >  - An editor using mmap() to read a file.
> 
> No, you're thinking the wrong way.
...
> People who think MAP_COPY is a good idea are people who cannot think about
> the implications of it, and cannot think about the alternatives.
:-)
> In particular, you claim that you could use "mmap()" for "read()", and
> speed up the application that way. Ok, fair enough.
> 
> Now, somebody who _isn't_ stupid (and that, of course, is me), immediately
> goes "well, _duh_, why don't you speed up read() instead?".

Thanks Linus.  You are right, speeding up read() is the right thing to do.

In fact it was proposed here on this list years ago, and I think you
argued against it (TLB flush costs).  The costs and kernel
infrastructure have changed and maybe the idea could be revisited now.

Sometimes one has to be an idiot and explore an application of MAP_COPY
to get someone looking at sensible old ideas again :-)

have a nice day,
-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Pablo Alcaraz wrote:
> Whatever will be the chosen solution, it would have to allow to 
> overwrite all the executables and libraries files (if we have enough 
> permissions).

Pablo, there's no need for this.  Upgrades to libraries are done by
removing the old file's name from its parent directory and placing the
new file at that name, perhaps using an atomic rename.  The old _file_
continues to exist even though its name has been deleted, until the last
program using the library finishes using it, even though the old file
does not have a name any more.

New programs that open the library will find the new file.  Both files
exist until the old file finally disappears.  At no point is any file's
contents overwritten.

This is why you can upgrade a running linux system including critical
system libraries, and nothing crashes.  Usually ;-)

-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On Sat, 13 Oct 2001, Jamie Lokier wrote:
>
> In fact it was proposed here on this list years ago, and I think you
> argued against it (TLB flush costs).  The costs and kernel
> infrastructure have changed and maybe the idea could be revisited now.

It's still not entirely unlikely that doing VM mappings is simply more
expensive than just doing a memcpy. The TLB invalidate is only part of the
issue - you also have the page table walk, the VM lock, and the fact that
PAGE_COPY itself ends up being overhead.

Which is why the PAGE_COPY kind of read() optimization is _probably_ only
worth it if the user asks for it directly (or automatically only for large
reads together with single-threaded applications).

The explicit flag is probably a good idea also because of usage patterns
(PAGE_COPY is a slowdown _if_ the file is actually written to or even
mapped shared).

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Linus Torvalds wrote:
> > In fact it was proposed here on this list years ago, and I think you
> > argued against it (TLB flush costs).  The costs and kernel
> > infrastructure have changed and maybe the idea could be revisited now.
> 
> It's still not entirely unlikely that doing VM mappings is simply more
> expensive than just doing a memcpy. The TLB invalidate is only part of the
> issue - you also have the page table walk, the VM lock, and the fact that
> PAGE_COPY itself ends up being overhead.

There are applications (GCC comes to mind) which are using mmap() to
read files now because it is measurably faster than read(), for
sufficiently large source files.

I don't know where the optimal costs lie.

-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Aaron Lehmann]

On Sat, Oct 13, 2001 at 09:46:03PM +0200, Jamie Lokier wrote:
> There are applications (GCC comes to mind) which are using mmap() to
> read files now because it is measurably faster than read(), for
> sufficiently large source files.

But it does have the advantage of allowing the sharing of memory, does
it not?

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On Sat, 13 Oct 2001, Jamie Lokier wrote:
>
> There are applications (GCC comes to mind) which are using mmap() to
> read files now because it is measurably faster than read(), for
> sufficiently large source files.
>
> I don't know where the optimal costs lie.

The gcc people tested it, and their cut-off point is at 30kB or so.
Anything smaller than that is faster to just "read()".

Now, that's a traditional mmap(), though, which has more overhead than a
"read-with-PAGE_COPY" would have. The pure mmap() approach has the actual
page fault overhead too, along with having to do "fstat()" and "munmap()".

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

In article <Pine.LNX.4.33.0110131219520.8900-100000@penguin.transmeta.com>,
Linus Torvalds  <torvalds@transmeta.com> wrote:
>
>The explicit flag is probably a good idea also because of usage patterns
>(PAGE_COPY is a slowdown _if_ the file is actually written to or even
>mapped shared).

Actually, I missed the obvious case: quite often when you do a "read()",
the reader itself will end up writing to the area read into.  In which
case doing the PAGE_COPY would also slow down measurably, due to the
extra overhead of the copy-on-write fault (which not just does the copy
that we tried to avoid, but will take a fault and more VM locks). 

So if we want to do this optimization, we _definitely_ want it to be
explicitly controlled by a flag, like O_DIRECT is.  There are just too
many cases where it's a pessimization, and while the user can often tell
before-hand, the kernel simply cannot. 

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Aaron Lehmann <aaronl@vitelus.com> writes:

> On Sat, Oct 13, 2001 at 09:46:03PM +0200, Jamie Lokier wrote:
> > There are applications (GCC comes to mind) which are using mmap() to
> > read files now because it is measurably faster than read(), for
> > sufficiently large source files.
> 
> But it does have the advantage of allowing the sharing of memory, does
> it not?

Only if you are going to write to the data.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Richard Gooch]

Jamie Lokier writes:
> There are applications (GCC comes to mind) which are using mmap() to
> read files now because it is measurably faster than read(), for
> sufficiently large source files.

So? MAP_PRIVATE is just fine for these. The simple solution if you
care about an edit in the middle of a compile is to have your editor
write a new file and do an atomic rename. No half-and-half data
problems, and the VM logic is kept simple (well, relative to what we
have now;-).

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Aaron Lehmann]

On Sat, Oct 13, 2001 at 04:27:48PM -0600, Eric W. Biederman wrote:
> > But it does have the advantage of allowing the sharing of memory, does
> > it not?
> 
> Only if you are going to write to the data.

Why? If gcc and another application read the source file with an
mmap() with the right parameters (read-only), it would only be shared
between them, as I understand it. If they both read() the file into
private buffers those can not be shared.

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:

> On Sat, 13 Oct 2001, Jamie Lokier wrote:
> >
> > There are applications (GCC comes to mind) which are using mmap() to
> > read files now because it is measurably faster than read(), for
> > sufficiently large source files.
> >
> > I don't know where the optimal costs lie.
> 
> The gcc people tested it, and their cut-off point is at 30kB or so.
> Anything smaller than that is faster to just "read()".
> 
> Now, that's a traditional mmap(), though, which has more overhead than a
> "read-with-PAGE_COPY" would have. The pure mmap() approach has the actual
> page fault overhead too, along with having to do "fstat()" and "munmap()".

Hmm.  read-with-PAGE_COPY may not be any faster than read as you still
read all of the data into memory, so you have almost the same latency.
mmap might work better because of better overlapping of I/O and cpu
processing.

Also read-with-PAGE_COPY has some really interesting implications for the
page out routines.  Because anytime you start the page out you have to
copy the page.  Not exactly when you want to increase the memory presure.
And not at all suitable for shared libraries.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Xavier Bestel]

le dim 14-10-2001 à 08:49, Eric W. Biederman a écrit :
> Also read-with-PAGE_COPY has some really interesting implications for the
> page out routines.  Because anytime you start the page out you have to
> copy the page.  Not exactly when you want to increase the memory presure.
> And not at all suitable for shared libraries.

I didn't understood very well why you would want to swap out a page
marked PAGE_COPY ? Doesn't it make sense to special-case it and just
leave it "in the file", as long as it's untouched ?

	Xav

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Linus Torvalds]

On 14 Oct 2001, Eric W. Biederman wrote:
>
> Hmm.  read-with-PAGE_COPY may not be any faster than read as you still
> read all of the data into memory, so you have almost the same latency.
> mmap might work better because of better overlapping of I/O and cpu
> processing.

Most of the time, you either have the IO overhead (and whether you use
read or mmap won't matter all that much, because you're IO limited), or
the thing is cached.

For gcc, it's cached 99% of the time, because most of the IO ends up being
header files (this is, of course, assuming that you're compiling a big
project, but if you're not, the big overhead is in loading _gcc_, not in
the pages it reads).

> Also read-with-PAGE_COPY has some really interesting implications for the
> page out routines.  Because anytime you start the page out you have to
> copy the page.  Not exactly when you want to increase the memory presure.

No no. Read my thing again. On swap-out, you just move the thing to the
swap cache.

Sure, that removes it from the regular cache, and that's possibly a
performance problem. But

> And not at all suitable for shared libraries.

No. Why would you "read" shared libraries? read is read, mmap is mmap. If
you want mmap, use mmap. Don't mess it up with MAP_COPY, which is not mmap
at all.

		Linus

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:

> On 14 Oct 2001, Eric W. Biederman wrote:
> >
> > Hmm.  read-with-PAGE_COPY may not be any faster than read as you still
> > read all of the data into memory, so you have almost the same latency.
> > mmap might work better because of better overlapping of I/O and cpu
> > processing.
> 
> Most of the time, you either have the IO overhead (and whether you use
> read or mmap won't matter all that much, because you're IO limited), or
> the thing is cached.

Thanks that makes sense for where you are targeting the performance
improvement.

> For gcc, it's cached 99% of the time, because most of the IO ends up being
> header files (this is, of course, assuming that you're compiling a big
> project, but if you're not, the big overhead is in loading _gcc_, not in
> the pages it reads).

O.k.  So the case that matters is when you are repeatedly reading from
the cache.

> > Also read-with-PAGE_COPY has some really interesting implications for the
> > page out routines.  Because anytime you start the page out you have to
> > copy the page.  Not exactly when you want to increase the memory presure.
> 
> No no. Read my thing again. On swap-out, you just move the thing to the
> swap cache.
> 
> Sure, that removes it from the regular cache, and that's possibly a
> performance problem. But

On swap-out the optimization to steal the page from the page cache
makes it much less of a problem.  But you still have to be prepared to
do the copy.  As there might be multiple users of the page, in which
case you can't steal the page cache copy. 

> > And not at all suitable for shared libraries.
> 
> No. Why would you "read" shared libraries? read is read, mmap is mmap. If
> you want mmap, use mmap. Don't mess it up with MAP_COPY, which is not mmap
> at all.

Linus I'm sure you realized that.  I'm not certain the whole rest of the world
did.  And the shared library topic is what got this discussion going.

Hmm.  So what you are looking at is something similiar to O_DIRECT,
but that will instead of doing the I/O and bypassing the page cache,
will instead borrow the page cache copies pages.

Eric

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Mark H. Wood]

You could also look at TOPS-20 for things they did well or unwell.  All
disk I/O in TOPS-20 is done by the VM code; the funky SIN%/SOUT% etc.
simply adjust the mapping window and copy stuff, letting the VM subsystem
schedule I/O as needed.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Aaron Lehmann wrote:
> > > But it does have the advantage of allowing the sharing of memory, does
> > > it not?
> > 
> > Only if you are going to write to the data.
> 
> Why? If gcc and another application read the source file with an
> mmap() with the right parameters (read-only), it would only be shared
> between them, as I understand it. If they both read() the file into
> private buffers those can not be shared.

And furthermore, the private buffers cannot be shared with the
filesystem's page cache.

-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Richard Gooch wrote:
> > There are applications (GCC comes to mind) which are using mmap() to
> > read files now because it is measurably faster than read(), for
> > sufficiently large source files.
> 
> So? MAP_PRIVATE is just fine for these. The simple solution if you
> care about an edit in the middle of a compile is to have your editor
> write a new file and do an atomic rename. No half-and-half data
> problems, and the VM logic is kept simple (well, relative to what we
> have now;-).

This does not work.  Example:

  1. JamieEmacs loads file using MAP_PRIVATE.
  2. Something else writes to the file.
  3. Scroll to the bottom of the file in JamieEmacs.  It displays some
     of the newly written data, though not all of it.

--> Wrong editor semantics.

Note that the something else which modifies the file in step 2 is not an
editor written especially to cooperate with JamieEmacs.  So it does not
do renaming -- why should it?  You might have just loaded
/var/log/messages into JamieEmacs, for example, and syslog is the
program in step 2.

What you need is read() or an equivalent.  I don't know of a
memory-efficient equivalent to read.  MAP_PRIVATE doesn't do it because
you have to dirty every page before you can be sure that file
modifications won't change your view of the data, and the dirtying
creates just as many page duplicates as read() does.

cheers,
-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Linus Torvalds wrote:
> Now, that's a traditional mmap(), though, which has more overhead than a
> "read-with-PAGE_COPY" would have. The pure mmap() approach has the actual
> page fault overhead too, along with having to do "fstat()" and "munmap()".

read() into a freshly allocated buffer (as you do for any large file)
has page fault overhead too -- to allocate zero pages.  It may be a
greater overhead, because the pages are unnecessarily zeroed.

read-with-PAGE_COPY may eliminate both of these overheads.

But then, even without PAGE_COPY, a read() which looks at the receiving
process' page tables may be able to eliminate the page faults, by simply
allocating (without zeroing) pages in kernel context prior to copying
the data there.

-- Jamie

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Alexander Viro]

On Mon, 15 Oct 2001, Jamie Lokier wrote:

> This does not work.  Example:
> 
>   1. JamieEmacs loads file using MAP_PRIVATE.
>   2. Something else writes to the file.
>   3. Scroll to the bottom of the file in JamieEmacs.  It displays some
>      of the newly written data, though not all of it.
> 
> --> Wrong editor semantics.

--> Wrong permissions or hopelessly crappy source control system.

At point 2 you are _already_ screwed.  Depending on who hits (hell,
what's the equivalent of :x in Emacsese?) first, one of you is
going to lose results of editing.  Doctor, it hurts when I do it...

If you want versioning - use source control system.  Or go play
with DEC cra^WOSes.  In RSX that "feature" sucked (and so did
editor semantics, but that's a separate story).

Without versioning - see above.

Re: Security question: "Text file busy" overwriting executables but not shared libraries?

[Jamie Lokier]

Alexander Viro wrote:
> > This does not work.  Example:
> > 
> >   1. JamieEmacs loads file using MAP_PRIVATE.
> >   2. Something else writes to the file.
> >   3. Scroll to the bottom of the file in JamieEmacs.  It displays some
> >      of the newly written data, though not all of it.
> > 
> > --> Wrong editor semantics.
> 
> --> Wrong permissions or hopelessly crappy source control system.
> 
> At point 2 you are _already_ screwed.  Depending on who hits (hell,
> what's the equivalent of :x in Emacsese?) first, one of you is
> going to lose results of editing.  Doctor, it hurts when I do it...

I am _not_ saving anything.  Viewing
/home/web/automatically_generated_every_hour.html from a particular
moment is a perfectly reasonable thing to do in Emacs, and it's a
perfectly reasonable thing to do in Less and Midnight Commander and
Mozilla for that matter.

_If_ I hit :x (in Vi-mode in Emacs ;-) then I expect the editor to warn
me that the file was updated by some other program.  Some editors will
warn before that.  Some will reload the file automatically if I haven't
made changed within the editor.

However, at all times I expect a consistent display of the file either
from read time, or from the current time.  _Never_ some unparsable,
invalid, mixed up combination of pages.

> If you want versioning - use source control system.  Or go play
> with DEC cra^WOSes.  In RSX that "feature" sucked (and so did
> editor semantics, but that's a separate story).

I do _not_ want versioning.  I want to load a file into an editor and
look at _that_ snapshot, at my leisure.  (Almost) every editor ever
written works this way, and I am quite happy with it.

read() gives the correct semantics.

There is potential to make read() more efficient, both in execution time
and in memory consumption.

Enjoy :-)

-- Jamie