Re: VFS locking: f_pos thread-safe ?

Re: VFS locking: f_pos thread-safe ?

[Matt]

> Werner Almesberger <wa@almesberger.net> wrote:
>>
>> "[...] read( ) [...] shall be atomic with respect to each other
>>   in the effects specified in IEEE Std. 1003.1-200x when they
>>   operate on regular files. If two threads each call one of these
>>   functions, each call shall either see all of the specified
>>   effects of the other call, or none of them."

> Whichever thread finishes its read last gets to update f_pos.

> I'm struggling a bit to understand what they're calling for there.  If
> thread A enters a read and then shortly afterwards thread B enters the
> read, does thread B see an f_pos which starts out at the beginning of A's
> read, or the end of it?

> Similar questions apply as the threads exit their read()s.

> Either way, there's no way in which we should serialise concurrent readers.
> That would really suck for sensible apps which are using pread64().

Surely, we can just serialise read() (and related) calls that modify f_pos?
Since pread() doesn't modify f_pos we shouldn't need to serialise those calls
no? Also doesn't spec make the same claims about other calls that modify
f_pos such as write()?

	Matt

Re: VFS locking: f_pos thread-safe ?

[Andrew Morton]

Matt <dirtbird@ntlworld.com> wrote:
>
> > Werner Almesberger <wa@almesberger.net> wrote:
> >>
> >> "[...] read( ) [...] shall be atomic with respect to each other
> >>   in the effects specified in IEEE Std. 1003.1-200x when they
> >>   operate on regular files. If two threads each call one of these
> >>   functions, each call shall either see all of the specified
> >>   effects of the other call, or none of them."
> 
> > Whichever thread finishes its read last gets to update f_pos.
> 
> > I'm struggling a bit to understand what they're calling for there.  If
> > thread A enters a read and then shortly afterwards thread B enters the
> > read, does thread B see an f_pos which starts out at the beginning of A's
> > read, or the end of it?
> 
> > Similar questions apply as the threads exit their read()s.
> 
> > Either way, there's no way in which we should serialise concurrent readers.
> > That would really suck for sensible apps which are using pread64().
> 
> Surely, we can just serialise read() (and related) calls that modify f_pos?
> Since pread() doesn't modify f_pos we shouldn't need to serialise those calls
> no? Also doesn't spec make the same claims about other calls that modify
> f_pos such as write()?

We could do somethnig like that.

But is there any application in which two threads simultaneously perform
read() against the same fd which is not already buggy?

Re: VFS locking: f_pos thread-safe ?

[Matt]

Andrew Morton wrote:

>Matt <dirtbird@ntlworld.com> wrote:
>  
>
>>>Werner Almesberger <wa@almesberger.net> wrote:
>>>      
>>>
>>>>"[...] read( ) [...] shall be atomic with respect to each other
>>>>  in the effects specified in IEEE Std. 1003.1-200x when they
>>>>  operate on regular files. If two threads each call one of these
>>>>  functions, each call shall either see all of the specified
>>>>  effects of the other call, or none of them."
>>>>        
>>>>
>>>Whichever thread finishes its read last gets to update f_pos.
>>>      
>>>
>>>I'm struggling a bit to understand what they're calling for there.  If
>>>thread A enters a read and then shortly afterwards thread B enters the
>>>read, does thread B see an f_pos which starts out at the beginning of A's
>>>read, or the end of it?
>>>      
>>>
>>>Similar questions apply as the threads exit their read()s.
>>>      
>>>
>>>Either way, there's no way in which we should serialise concurrent readers.
>>>That would really suck for sensible apps which are using pread64().
>>>      
>>>
>>Surely, we can just serialise read() (and related) calls that modify f_pos?
>>Since pread() doesn't modify f_pos we shouldn't need to serialise those calls
>>no? Also doesn't spec make the same claims about other calls that modify
>>f_pos such as write()?
>>    
>>
>
>We could do somethnig like that.
>
>But is there any application in which two threads simultaneously perform
>read() against the same fd which is not already buggy?
>
>
>  
>
touché :) but still we should do what we can.. want me to make a patch?

    Matt

Re: VFS locking: f_pos thread-safe ?

[Andrew Morton]

Matt <dirtbird@ntlworld.com> wrote:
>
>  >But is there any application in which two threads simultaneously perform
>  >read() against the same fd which is not already buggy?
>  >
>  >
>  >  
>  >
>  touché :) but still we should do what we can.. want me to make a patch?

Not unless we can think of a way in which it actually matters, thanks.

Re: VFS locking: f_pos thread-safe ?

[Matthias Urlichs]

Hi, Andrew Morton wrote:
>>  touché :) but still we should do what we can.. want me to make a patch?
> 
> Not unless we can think of a way in which it actually matters, thanks.
> 
I've written one of those... a migration program from an old
fixed-record-size database to SQL. Somebody had the brilliant idea to let
two threads read the file concurrently, postprocess the thing (which took
a variable amount of time depending on what was in the record) and then
feed it to the database (ditto).

When that idea didn't work, I used a separate reader thread and
coordinated buffer usage with semaphores or whatever, making the thing a
whole lot more complicated in the process. :-/

So count me as one of those people who think that it does matter -- if N
threads read a file of M bytes, they should collectively get M bytes from
it (possibly out of order, of course, but that's a different problem).
Anything else would be inconsistent.

Same thing goes for writing (in append mode as well as otherwise).

-- 
Matthias Urlichs

Re: VFS locking: f_pos thread-safe ?

[viro]

On Fri, Feb 06, 2004 at 11:19:58AM +0100, Matthias Urlichs wrote:
> I've written one of those... a migration program from an old
> fixed-record-size database to SQL. Somebody had the brilliant idea to let
> two threads read the file concurrently, postprocess the thing (which took
> a variable amount of time depending on what was in the record) and then
> feed it to the database (ditto).

"Somebody made a guess about undefined behaviour"

> When that idea didn't work, I used a separate reader thread and
> coordinated buffer usage with semaphores or whatever, making the thing a
> whole lot more complicated in the process. :-/

"Guess what? It didn't work".
 
> So count me as one of those people who think that it does matter -- if N
> threads read a file of M bytes, they should collectively get M bytes from
> it (possibly out of order, of course, but that's a different problem).
> Anything else would be inconsistent.

"Please, make that undefined behaviour conform to the guess made by that
somebody.  Whaddya mean, 'non-portable' and 'broken application'?!?!?".

> Same thing goes for writing (in append mode as well as otherwise).

Look for ->i_sem use in there.

Re: VFS locking: f_pos thread-safe ?

[Matthias Urlichs]

Hi, viro wrote:

> "Somebody made a guess about undefined behaviour"

Guess what? The manpage says that read(2) return N bytes and advances the
file pointer by N bytes. It doesn't talk, much less caution, about threads.

YOU may immediately know, based on your kernel knowledge or whatever, that
things get somewhat undefined when two threads do that at the same time,
but it's NOT AT ALL obvious to a "normal" application programmer. There's
plenty of system calls that CAN be done concurrently, after all.

So we tried the simple idea, it didn't work, we grumbled a bit ;-) and
then did something else instead. Not a big deal.

Please save your "translations" for stupid ideas that are obviously so
without in-depth kernel knowledge or equivalent.

Thank you.

-- 
Matthias Urlichs

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

Matthias Urlichs wrote:
> but it's NOT AT ALL obvious to a "normal" application programmer.

It gets worse. From the same draft (perhaps someone who has the final
version could comment ?), in the rationale for read():

| The standard developers considered adding atomicity requirements to a
| pipe or FIFO, but recognized that due to the nature of pipes and FIFOs
| there could be no guarantee of atomicity of reads of {PIPE_BUF} or any
| other size that would be an aid to applications portability.

But then

| I/O is intended to be atomic to ordinary files and pipes and FIFOs.

Now, what exactly does "intended" mean ?

Of course, in this part, they only talk about data staying together,
not whether it can get duplicated, or effects on f_pos.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: VFS locking: f_pos thread-safe ?

[Kai Henningsen]

smurf@smurf.noris.de (Matthias Urlichs)  wrote on 06.02.04 in <pan.2004.02.06.18.59.44.936432@smurf.noris.de>:

> Hi, viro wrote:
>
> > "Somebody made a guess about undefined behaviour"
>
> Guess what? The manpage says that read(2) return N bytes and advances the
> file pointer by N bytes. It doesn't talk, much less caution, about threads.

That's a defect in the man page, then.

POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/ 
read.html>:

[...]
DESCRIPTION

The read() function shall attempt to read nbyte bytes from the file
associated with the open file descriptor, fildes, into the buffer pointed
to by buf. The behavior of multiple concurrent reads on the same pipe,
FIFO, or terminal device is unspecified.
[...]

That's a pretty explicit warning exactly where one would expect it.

If Linux read(2) doesn't say something like that, I'll consider that a  
serious bug.

OTOH ...

> YOU may immediately know, based on your kernel knowledge or whatever, that
> things get somewhat undefined when two threads do that at the same time,
> but it's NOT AT ALL obvious to a "normal" application programmer. There's
> plenty of system calls that CAN be done concurrently, after all.

... these days, programmers really *should* know enough to consult a free,  
current, and easily readable version of the relevant standard!

(If you forget where it is, it's easy to find from <http://www.unix- 
systems.org/>.)

> Please save your "translations" for stupid ideas that are obviously so
> without in-depth kernel knowledge or equivalent.

... such as the current case.

MfG Kai

Re: VFS locking: f_pos thread-safe ?

[Pavel Machek]

Hi!

> POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/ 
> read.html>:
> 
> [...]
> DESCRIPTION
> 
> The read() function shall attempt to read nbyte bytes from the file
> associated with the open file descriptor, fildes, into the buffer pointed
> to by buf. The behavior of multiple concurrent reads on the same pipe,
> FIFO, or terminal device is unspecified.

...which implies that concurrent reads on regular files are okay :-P.

				Pavel

-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: VFS locking: f_pos thread-safe ?

[Matthias Urlichs]

Hi,

Stephen Hemminger:
> 
> Ever hear of pread/pwrite?

Sure I have, but the general idea had been to leave the work entirely to
the file system...

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

Andrew Morton wrote:
> Not unless we can think of a way in which it actually matters, thanks.

What I'm struggling with when reading that POSIX draft is to
understand whether CLONE_FILES is appropriate or not for
pthread_create.

If this is unspecified, the f_pos issue becomes largely academic,
although one might argue that read() should behave either one
way or the other.

And no, luckily, I don't have a real-life application for this
either :-)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: VFS locking: f_pos thread-safe ?

[viro]

On Fri, Feb 06, 2004 at 10:50:09AM -0300, Werner Almesberger wrote:
> Andrew Morton wrote:
> > Not unless we can think of a way in which it actually matters, thanks.
> 
> What I'm struggling with when reading that POSIX draft is to
> understand whether CLONE_FILES is appropriate or not for
> pthread_create.
> 
> If this is unspecified, the f_pos issue becomes largely academic,
> although one might argue that read() should behave either one
> way or the other.
> 
> And no, luckily, I don't have a real-life application for this
> either :-)

WTF does that have to CLONE_FILES?  Whether you share descriptor table
or get an independent copy, pointers to struct file are the same.

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

viro@parcelfarce.linux.theplanet.co.uk wrote:
> WTF does that have to CLONE_FILES?  Whether you share descriptor table
> or get an independent copy, pointers to struct file are the same.

Err, right, of course. Sorry, I shouldn't post before the caffeine
breakfast ...

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

VFS locking: f_pos thread-safe ?

[Werner Almesberger]

I'm trying to figure out how all the locking in VFS and friends
works, and I can't quite explain to myself how f_pos is kept
consistent with concurrent readers.

In fact, there might be a violation of atomicity requirements:
e.g. if we take the route sys_read -> vfs_read ->
generic_file_read -> __generic_file_aio_read ->
do_generic_file_read -> do_generic_mapping_read, we don't seem
to be holding any locks. So if I have two threads that start
reading the same fd at the same time, they could retrieve the
same data.

Section 2.9.7 of the "Austin" draft of IEEE Std. 1003.1-200x,
28-JUL-2000, says:

"[...] read( ) [...] shall be atomic with respect to each other
 in the effects specified in IEEE Std. 1003.1-200x when they
 operate on regular files. If two threads each call one of these
 functions, each call shall either see all of the specified
 effects of the other call, or none of them."

I've written a little test program with concurrent readers that
seems to support this observation, i.e. given the following
pseudo-code:

static void *reader(...)
{
    while (read(0,buffer,PAGE_SIZE));
    ...
}

...
    for (...)
	pthread_create(...,reader...);
...

More than one reader may obtain a given page.
The full test program is at
http://www.almesberger.net/misc/tt.tar.gz

Is this a real bug or am I just confused ?

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: VFS locking: f_pos thread-safe ?

[Andrew Morton]

Werner Almesberger <wa@almesberger.net> wrote:
>
> "[...] read( ) [...] shall be atomic with respect to each other
>   in the effects specified in IEEE Std. 1003.1-200x when they
>   operate on regular files. If two threads each call one of these
>   functions, each call shall either see all of the specified
>   effects of the other call, or none of them."

Whichever thread finishes its read last gets to update f_pos.

I'm struggling a bit to understand what they're calling for there.  If
thread A enters a read and then shortly afterwards thread B enters the
read, does thread B see an f_pos which starts out at the beginning of A's
read, or the end of it?

Similar questions apply as the threads exit their read()s.

Either way, there's no way in which we should serialise concurrent readers.
That would really suck for sensible apps which are using pread64().

Re: VFS locking: f_pos thread-safe ?

[Joel Becker]

On Fri, Feb 06, 2004 at 04:12:24AM -0300, Werner Almesberger wrote:
> Section 2.9.7 of the "Austin" draft of IEEE Std. 1003.1-200x,
> 28-JUL-2000, says:
> 
> "[...] read( ) [...] shall be atomic with respect to each other
>  in the effects specified in IEEE Std. 1003.1-200x when they
>  operate on regular files. If two threads each call one of these
>  functions, each call shall either see all of the specified
>  effects of the other call, or none of them."

	This reads:  "all of the specified effects of the other call,
or none of them."  If I read that correctly, if f_pos is at N, and
threads A and B concurrently read M bytes, then each thread's read()
must either start at f_pos = N or f_pos = N+M, but never at N < f_pos <
N+M.  So as long as our code doesn't partially update f_pos, it is
valid.  
	Of course, that doesn't change the possible race updating
f_pos at the end of each thread's call.

Joel

-- 

"In a crisis, don't hide behind anything or anybody. They're going
 to find you anyway."
	- Paul "Bear" Bryant

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: VFS locking: f_pos thread-safe ?

[Matthias Urlichs]

Hi, Joel Becker wrote:

> This reads:  "all of the specified effects of the other call,
> or none of them."  If I read that correctly, if f_pos is at N, and
> threads A and B concurrently read M bytes, then each thread's read()
> must either start at f_pos = N or f_pos = N+M, but never at N < f_pos <
> N+M.  So as long as our code doesn't partially update f_pos, it is
> valid.

Umm, strictly speaking there are three possible valid cases:

thread A reads M @N   thread B reads M @N     file pointer ends up as N+M
thread A reads M @N   thread B reads M @M+N   file pointer ends up as N+2M
thread A reads M @M+N thread B reads M @N     file pointer ends up as N+2M

With your description,
thread A reads M @M+N thread B reads M @M+N   file pointer ends up as N+2M

would be equally valid, which I'd declare buggy ^W non-conforming.

-- 
Matthias Urlichs

Re: VFS locking: f_pos thread-safe ?

[Joel Becker]

On Fri, Feb 06, 2004 at 08:05:48PM +0100, Matthias Urlichs wrote:
> thread A reads M @N   thread B reads M @N     file pointer ends up as N+M
> thread A reads M @N   thread B reads M @M+N   file pointer ends up as N+2M
> thread A reads M @M+N thread B reads M @N     file pointer ends up as N+2M
> 
> With your description,
> thread A reads M @M+N thread B reads M @M+N   file pointer ends up as N+2M

	I didn't mean the last.  I was just writing English, and figured
everyone would know what I meant.  Obviously, the first read() in starts
@M.

Joel

-- 

Life's Little Instruction Book #198

	"Feed a stranger's expired parking meter."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

Joel Becker wrote:
> 	This reads:  "all of the specified effects of the other call,
> or none of them."
[...]
> 	Of course, that doesn't change the possible race updating
> f_pos at the end of each thread's call.

Hmm, "all, but the f_pos read-modify-write" sounds more like how
an insurance company would define "all" :-)

What's puzzling here is that the standard would introduce such an
important concept in the discussion of threads.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: VFS locking: f_pos thread-safe ?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Fri, 06 Feb 2004 17:09:17 -0300, Werner Almesberger <wa@almesberger.net>  said:

> Hmm, "all, but the f_pos read-modify-write" sounds more like how
> an insurance company would define "all" :-)
> 
> What's puzzling here is that the standard would introduce such an
> important concept in the discussion of threads.

Well... it's the sort of problem that's very Schrodenger unless you have
another thread/process observing.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

Valdis.Kletnieks@vt.edu wrote:
> Well... it's the sort of problem that's very Schrodenger unless you have
> another thread/process observing.

In a way :-) What I meant was that the section on read() is worded
carefully to avoid making such assurances, but then we find such a
broad statement at the end of a section discussing specifically
threads (and not even concurrency in general, and after pointing
out that threads are optional).

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: VFS locking: f_pos thread-safe ?

[Andries Brouwer]

On Fri, Feb 06, 2004 at 04:12:24AM -0300, Werner Almesberger wrote:

> I'm trying to figure out how all the locking in VFS and friends
> works, and I can't quite explain to myself how f_pos is kept
> consistent with concurrent readers.
> 
> Section 2.9.7 of the "Austin" draft of IEEE Std. 1003.1-200x,
> 28-JUL-2000, says:
> 
> "[...] read( ) [...] shall be atomic with respect to each other
>  in the effects specified in IEEE Std. 1003.1-200x when they
>  operate on regular files. If two threads each call one of these
>  functions, each call shall either see all of the specified
>  effects of the other call, or none of them."

The 2003 version can be found at

http://www.opengroup.org/onlinepubs/007904975/toc.htm
http://www.opengroup.org/onlinepubs/007904975/functions/xsh_chap02_09.html

Andries

Re: VFS locking: f_pos thread-safe ?

[Werner Almesberger]

Andries Brouwer wrote:
> The 2003 version can be found at
> 
> http://www.opengroup.org/onlinepubs/007904975/toc.htm
> http://www.opengroup.org/onlinepubs/007904975/functions/xsh_chap02_09.html

Thanks a lot !

The wording is the same, so it seems that we don't comply with
that part, at least not in the letter :-(

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/