Fwd: Adaptive read-ahead V12

Fwd: Adaptive read-ahead V12

[Wu Fengguang]

----- Forwarded message from Iozone <capps@iozone.org> -----

Subject: Adaptive read-ahead V12
From: Iozone <capps@iozone.org>
To: Wu Fengguang <wfg@mail.ustc.edu.cn>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500

Wu Fengguang,

       I see that Andrew M. is giving you some pushback.... 
   His argument is that the application could do a better job
   of scheduling its own read-ahead.  ( I've heard this one 
   before)

   My thoughts on this argument would be along the 
   lines of:

   Indeed the application might be able to do a better
   job, however expecting, or demanding, the rewrite
   of all applications to behave better might be an unreasonable
   expectation.   Rewriting all the weather codes, all the 
   structrual analysis codes, all the data-bases, video streaming
   and so on, would solve the problem but the timeline for
   such an re-codification may approach infinity, while
   systems that have the ability to silently increase 
   performance maintain an advantage in the market.
   Note: Many vendors already have sophisticated A.I. read-ahead
   algorithms. Examples: Microsoft, Veritas HP, and Isolon.

   Linux already does read-ahead that does not require
   the application to be involved. It just does it in
   a simpler way that could be improved for handling
   a wider range of applications. This new technology is
   a simple evolutionary step in that direction. Linux read-ahead
   and is currently 8 years behind existing technologies in this area. 
   Isn't it time to catch up a bit ?

   Note: You probably will need to reword this before replying
   to Andrew, as I tend to be somewhat blunt. :-) 

   Hint: I took me a while to convince the kernel folks in my
   company to accept such a technology. But when the 
   prototype accelerated one of their key partner's code by
   30 % wall clock, and another showed 50X speedup in
   file I/O across a wide stripe,  that did the trick. 

Enjoy,
Don Capps

[...]
----- End forwarded message -----

Fwd: Adaptive read-ahead V12

[Wu Fengguang]

----- Forwarded message from Iozone <capps@iozone.org> -----

Subject: Adaptive read-ahead V12
From: Iozone <capps@iozone.org>
To: Wu Fengguang <wfg@mail.ustc.edu.cn>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500

Wu Fengguang,

       I see that Andrew M. is giving you some pushback.... 
   His argument is that the application could do a better job
   of scheduling its own read-ahead.  ( I've heard this one 
   before)

   My thoughts on this argument would be along the 
   lines of:

   Indeed the application might be able to do a better
   job, however expecting, or demanding, the rewrite
   of all applications to behave better might be an unreasonable
   expectation.   Rewriting all the weather codes, all the 
   structrual analysis codes, all the data-bases, video streaming
   and so on, would solve the problem but the timeline for
   such an re-codification may approach infinity, while
   systems that have the ability to silently increase 
   performance maintain an advantage in the market.
   Note: Many vendors already have sophisticated A.I. read-ahead
   algorithms. Examples: Microsoft, Veritas HP, and Isolon.

   Linux already does read-ahead that does not require
   the application to be involved. It just does it in
   a simpler way that could be improved for handling
   a wider range of applications. This new technology is
   a simple evolutionary step in that direction. Linux read-ahead
   and is currently 8 years behind existing technologies in this area. 
   Isn't it time to catch up a bit ?

   Note: You probably will need to reword this before replying
   to Andrew, as I tend to be somewhat blunt. :-) 

   Hint: I took me a while to convince the kernel folks in my
   company to accept such a technology. But when the 
   prototype accelerated one of their key partner's code by
   30 % wall clock, and another showed 50X speedup in
   file I/O across a wide stripe,  that did the trick. 

Enjoy,
Don Capps

[...]
----- End forwarded message -----

Re: Fwd: Adaptive read-ahead V12

[Bill Davidsen]

Wu Fengguang wrote:
> ----- Forwarded message from Iozone <capps@iozone.org> -----
> 
> Subject: Adaptive read-ahead V12
> From: Iozone <capps@iozone.org>
> To: Wu Fengguang <wfg@mail.ustc.edu.cn>
> X-Mailer: Microsoft Outlook Express 6.00.2900.2670
> Date: Thu, 25 May 2006 11:44:37 -0500
> 
> Wu Fengguang,
> 
>        I see that Andrew M. is giving you some pushback.... 
>    His argument is that the application could do a better job
>    of scheduling its own read-ahead.  ( I've heard this one 
>    before)
> 
>    My thoughts on this argument would be along the 
>    lines of:
> 
>    Indeed the application might be able to do a better
>    job, however expecting, or demanding, the rewrite
>    of all applications to behave better might be an unreasonable
>    expectation. 

A reasonable expectation would be that the application would have a way 
to tell the kernel to ignore readahead for a given file, other than 
changing the behavior of the kernel as a whole for all processes on the 
machine. This smart application could then use aio or some other similar 
method to do preread itself.

My personal opinion is that the kernel only does a good job reading 
ahead for sequential access, and since that's the common case only a 
means of preventing that effort need be provided.

I can think of several ways the kernel might be smarter about the way it 
reads (or not) random access, but since I have no way to test them on 
applications other than my own I won't belabor them here.

-- 
Bill Davidsen <davidsen@tmr.com>
   Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.

Re: Fwd: Adaptive read-ahead V12

[Wu Fengguang]

On Wed, May 31, 2006 at 04:17:58PM -0400, Bill Davidsen wrote:
> Wu Fengguang wrote:
> >----- Forwarded message from Iozone <capps@iozone.org> -----
> >
> >Subject: Adaptive read-ahead V12
> >From: Iozone <capps@iozone.org>
> >To: Wu Fengguang <wfg@mail.ustc.edu.cn>
> >X-Mailer: Microsoft Outlook Express 6.00.2900.2670
> >Date: Thu, 25 May 2006 11:44:37 -0500
> >
> >Wu Fengguang,
> >
> >       I see that Andrew M. is giving you some pushback.... 
> >   His argument is that the application could do a better job
> >   of scheduling its own read-ahead.  ( I've heard this one 
> >   before)
> >
> >   My thoughts on this argument would be along the 
> >   lines of:
> >
> >   Indeed the application might be able to do a better
> >   job, however expecting, or demanding, the rewrite
> >   of all applications to behave better might be an unreasonable
> >   expectation. 
> 
> A reasonable expectation would be that the application would have a way 
> to tell the kernel to ignore readahead for a given file, other than 
> changing the behavior of the kernel as a whole for all processes on the 
> machine. This smart application could then use aio or some other similar 
> method to do preread itself.

Sure. The adaptive readahead works fine with user level readahead via
the readahead() or posix_fadvise() syscall. I.e. if a program do
necessary readahead() calls that can cover all the file pages
requested by the following read() calls, it avoids triggering
unnecessary kernel readaheads.

> My personal opinion is that the kernel only does a good job reading 
> ahead for sequential access, and since that's the common case only a 
> means of preventing that effort need be provided.

posix_fadvise(fd, ..., POSIX_FADV_RANDOM) can do the trick for a file.
blockdev --setra 0 /dev/hda  does so for a device.

Wu

Re: Fwd: Adaptive read-ahead V12

[Wu Fengguang]

On Wed, May 31, 2006 at 04:17:58PM -0400, Bill Davidsen wrote:
> Wu Fengguang wrote:
> >----- Forwarded message from Iozone <capps@iozone.org> -----
> >
> >Subject: Adaptive read-ahead V12
> >From: Iozone <capps@iozone.org>
> >To: Wu Fengguang <wfg@mail.ustc.edu.cn>
> >X-Mailer: Microsoft Outlook Express 6.00.2900.2670
> >Date: Thu, 25 May 2006 11:44:37 -0500
> >
> >Wu Fengguang,
> >
> >       I see that Andrew M. is giving you some pushback.... 
> >   His argument is that the application could do a better job
> >   of scheduling its own read-ahead.  ( I've heard this one 
> >   before)
> >
> >   My thoughts on this argument would be along the 
> >   lines of:
> >
> >   Indeed the application might be able to do a better
> >   job, however expecting, or demanding, the rewrite
> >   of all applications to behave better might be an unreasonable
> >   expectation. 
> 
> A reasonable expectation would be that the application would have a way 
> to tell the kernel to ignore readahead for a given file, other than 
> changing the behavior of the kernel as a whole for all processes on the 
> machine. This smart application could then use aio or some other similar 
> method to do preread itself.

Sure. The adaptive readahead works fine with user level readahead via
the readahead() or posix_fadvise() syscall. I.e. if a program do
necessary readahead() calls that can cover all the file pages
requested by the following read() calls, it avoids triggering
unnecessary kernel readaheads.

> My personal opinion is that the kernel only does a good job reading 
> ahead for sequential access, and since that's the common case only a 
> means of preventing that effort need be provided.

posix_fadvise(fd, ..., POSIX_FADV_RANDOM) can do the trick for a file.
blockdev --setra 0 /dev/hda  does so for a device.

Wu