POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[David Rolenc]

I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
something?  I want to read data from disk with readahead and drop the 
data from the page cache as soon as I am done with it. Do I have to call 
fadvise with POSIX_FADV_DONTNEED after every read?

Thanks,

David Rolenc
RT Logic!
www.rtlogic.com 

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[Robert Love]

On Thu, 2004-08-26 at 11:39 -0600, David Rolenc wrote:
> I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
> 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
> not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
> something?

Yes, they are currently the same.  Their difference in theory is subtle:
NOREUSE says "I will use this once in the future" and WILLNEED says "I
will use this [some number of times] in the future".

So for both of them, you want to keep the data in the page cache.  But
for NOREUSE you could do drop-behind of the pages.  But Linux does not
currently do drop-behind, so we have them both do the same thing.

>  I want to read data from disk with readahead and drop the 
> data from the page cache as soon as I am done with it. Do I have to call 
> fadvise with POSIX_FADV_DONTNEED after every read?

Yes, you do.  Or every couple of reads.

The sort of applications that need DONTNEED already are setup in a way
to make that easy.  You either have a streaming loop, like:

	do {
		read
		process
	} while (repeat);

Or you are readings lots of files (like updatedb).

In the first case, just stick the fadvise call after the processing.  In
the latter case, call fadvise before closing the file.

You don't _have_ to call fadvise after every reading.  Just whenever you
want to drop the pages.  Whenever it makes sense.

	Robert Love

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[David Rolenc]

Is there any technical reason that we couldn't set a flag in the struct 
file* and put code in the read/write calls to handle shrinking the page 
cache? This is similar to what was done with O_STREAMING in 2.4. It 
seems like it would conform more to the behavior expected from 
POSIX_FADV_NOREUSE, and I wouldn't have to make a bunch of system calls :-)

-Dave


Robert Love wrote:

>On Thu, 2004-08-26 at 11:39 -0600, David Rolenc wrote:
>  
>
>>I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
>>2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
>>not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
>>something?
>>    
>>
>
>Yes, they are currently the same.  Their difference in theory is subtle:
>NOREUSE says "I will use this once in the future" and WILLNEED says "I
>will use this [some number of times] in the future".
>
>So for both of them, you want to keep the data in the page cache.  But
>for NOREUSE you could do drop-behind of the pages.  But Linux does not
>currently do drop-behind, so we have them both do the same thing.
>
>  
>
>> I want to read data from disk with readahead and drop the 
>>data from the page cache as soon as I am done with it. Do I have to call 
>>fadvise with POSIX_FADV_DONTNEED after every read?
>>    
>>
>
>Yes, you do.  Or every couple of reads.
>
>The sort of applications that need DONTNEED already are setup in a way
>to make that easy.  You either have a streaming loop, like:
>
>	do {
>		read
>		process
>	} while (repeat);
>
>Or you are readings lots of files (like updatedb).
>
>In the first case, just stick the fadvise call after the processing.  In
>the latter case, call fadvise before closing the file.
>
>You don't _have_ to call fadvise after every reading.  Just whenever you
>want to drop the pages.  Whenever it makes sense.
>
>	Robert Love
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[David Greaves]

David Rolenc wrote:

> I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
> 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
> not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
> something?  I want to read data from disk with readahead and drop the 
> data from the page cache as soon as I am done with it. Do I have to 
> call fadvise with POSIX_FADV_DONTNEED after every read?

And will this work over nfs?

David

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[Trond Myklebust]

På to , 26/08/2004 klokka 14:38, skreiv David Greaves:
> David Rolenc wrote:
> 
> > I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
> > 2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
> > not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
> > something?  I want to read data from disk with readahead and drop the 
> > data from the page cache as soon as I am done with it. Do I have to 
> > call fadvise with POSIX_FADV_DONTNEED after every read?
> 
> And will this work over nfs?

What do you mean?

The client will of course respect fadvise() if the generic VM code
supports it, but there is no NFS protocol support for this, so the
client is not able to communicate your fadvise call on to the server.

Cheers,
  Trond

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[David Greaves]

Trond Myklebust wrote:

>På to , 26/08/2004 klokka 14:38, skreiv David Greaves:
>  
>
>>David Rolenc wrote:
>>
>>    
>>
>>>I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior in 
>>>2.6 and just a glance at fadvise.c suggests that POSIX_FADV_NOREUSE is 
>>>not implemented any differently than POSIX_FADV_WILLNEED. Am I missing 
>>>something?  I want to read data from disk with readahead and drop the 
>>>data from the page cache as soon as I am done with it. Do I have to 
>>>call fadvise with POSIX_FADV_DONTNEED after every read?
>>>      
>>>
>>And will this work over nfs?
>>    
>>
>
>What do you mean?
>
>The client will of course respect fadvise() if the generic VM code
>supports it, but there is no NFS protocol support for this, so the
>client is not able to communicate your fadvise call on to the server.
>  
>
Perfect answer
thank you

I want my nfs client to communicate this to my nfs server. Thus avoiding 
my nfs server having a cache of useless video.
I can see this becoming an important benefit for video distribution (an 
area linux is likely to see more of)

David

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[Trond Myklebust]

På fr , 27/08/2004 klokka 04:25, skreiv David Greaves:

> I want my nfs client to communicate this to my nfs server. Thus avoiding 
> my nfs server having a cache of useless video.
> I can see this becoming an important benefit for video distribution (an 
> area linux is likely to see more of)

Then you are probably going to have to invent some out-of-band protocol
in order to do so. Creating such a protocol shouldn't be a hard task,
but getting all those server manufacturers to adopt it afterwards
probably will be be. ;-)

Alternatively, if you can make a good case (and write a decent RFC), you
might be able to persuade the IETF working group to add this feature
into a future minor version of NFSv4.

Cheers,
  Trond

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[Trond Myklebust]

På fr , 27/08/2004 klokka 04:25, skreiv David Greaves:
> I want my nfs client to communicate this to my nfs server. Thus avoiding 
> my nfs server having a cache of useless video.
> I can see this becoming an important benefit for video distribution (an 
> area linux is likely to see more of)

Then that will have to be done either through some out-of-band protocol,
or as part of some future minor revision of NFSv4 (assuming that you are
able to make a good case for it in an RFC and present it to the IETF's
NFS working group).

Cheers,
  Trond

Re: POSIX_FADV_NOREUSE and O_STREAMING behavior in 2.6.7

[David Rolenc]

David Greaves wrote:

> David Rolenc wrote:
>
>> I am trying to get O_STREAMING (Robert Love patch for 2.4) behavior 
>> in 2.6 and just a glance at fadvise.c suggests that 
>> POSIX_FADV_NOREUSE is not implemented any differently than 
>> POSIX_FADV_WILLNEED. Am I missing something?  I want to read data 
>> from disk with readahead and drop the data from the page cache as 
>> soon as I am done with it. Do I have to call fadvise with 
>> POSIX_FADV_DONTNEED after every read?
>
>
> And will this work over nfs?

The system will not be using nfs at all.

>
> David
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>