Streaming disk I/O kills file buffering and makes Linux unusable

Streaming disk I/O kills file buffering and makes Linux unusable

[Benno Senoner]

Hi,
I'm doing experiments of high performance data streaming from disk,
in my case streaming 60 audio tracks (44khz mono 16bit)  from disk.

This works quite well, but it has one BIG DRAWBACK: 
it makes the rest of  the system unusable.
If I run this app which reads huge files from disks continuously,
all other regular files and executables buffered in the file buffer cache,
are evicted from the filebuffer, and every time I want to launch an executable,
which was previously launched, the kernel has to load the entire executable
from disk, since it's not longer buffered and therefore it degrades the
performance of my streaming app too.
I could use a second disk dedicated for streaming, but the first disk
(on which the Linux distro and apps do reside), will become very slow,
since it will pratically RUN IN AN UNBUFFERED fashion all the time,
since the LRU buffering algorithm will remove all buffered regular
files and executable because the kernel caches the huge streaming files.

Is there a way to limit filebuffering on a per-process basis ?
For example let's say, I want that my streaming process can't use more than 5MB
of the kernel filebuffers, so that regular files/executables can be buffered
witout being removed from the buffers due to the huge filestreaming activity.
SGI's had the O_DIRECT flag to pass to the open() call to avoid buffering.
But linux is missing this flag , and IMHO  O_SYNC would not help, since it syncs
only on disk writes, disk reads will still be cached.

I found nothing usable in the /proc/sys/*  tuneable values, and setlimit()
has no tunable buffers in it's structure.
The setbuffer() of stdio seems not to solve the problem since the kernel uses
his own buffering.
Is there a solution for this buffer cache problem ?

Will future kernels ave an O_DIRECT like flag to avoid caching ?
I think without some buffer-cache usage limiting, or buffering disabling,
Linux is actually UNSUITABLE for streaming applications which do run
concurrently with other apps.

comments ?

PS: the easiest way to test this bad behavior is to run in background a script
which copies 5-10 files of 50-100MB in size simultaneously, let it run a while
and then try to open and xterm .. it will take SEVERAL seconds.

regards,
Benno.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Peter Monta]

> Will future kernels ave an O_DIRECT like flag to avoid caching ?

Stephen Tweedie has written a patch for unbuffered I/O; the URL I have
from some time ago is

ftp://ftp.linux.org.uk/pub/linux/sct/fs/raw-io

though this host doesn't seem to respond at the moment.  I believe
Stephen has mentioned that this will go into the main tree at some point.
(Possibly it's already in 2.3.x---I haven't looked recently.)

> I think without some buffer-cache usage limiting, or buffering disabling,
> Linux is actually UNSUITABLE for streaming applications which do run
> concurrently with other apps.

I agree; the raw-io system will make this much more pleasant.

Cheers,
Peter Monta   pmonta@imedia.com
Imedia Corp.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Linus Torvalds]

On Mon, 23 Aug 1999, Peter Monta wrote:
> 
> > I think without some buffer-cache usage limiting, or buffering disabling,
> > Linux is actually UNSUITABLE for streaming applications which do run
> > concurrently with other apps.
> 
> I agree; the raw-io system will make this much more pleasant.

Why do you think so?

IO is IO, whether it is raw or not. And it will eat up resources from
everybody else trying to do IO.

I think this is a case of believing in magic cures and supernatural
beings.

		Linus

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Peter Monta]

> > I agree; the raw-io system will make this much more pleasant.
>
> Why do you think so?
>
> IO is IO, whether it is raw or not. And it will eat up resources from
> everybody else trying to do IO.

True; raw I/O won't magically increase resources.  But it offers
a valuable way to tell the kernel something it can't know/infer: a good
buffering policy for streaming data.  The existing cache mechanism
for normal I/O works fine but is inappropriate for this case; indeed
perfectly good working sets are messed up by buffering the streaming data,
as Benno points out.

Cheers,
Peter

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Benno Senoner]

On Mon, 23 Aug 1999, Linus Torvalds wrote:
> On Mon, 23 Aug 1999, Peter Monta wrote:
> > 
> > > I think without some buffer-cache usage limiting, or buffering disabling,
> > > Linux is actually UNSUITABLE for streaming applications which do run
> > > concurrently with other apps.
> > 
> > I agree; the raw-io system will make this much more pleasant.
> 
> Why do you think so?
> 
> IO is IO, whether it is raw or not. And it will eat up resources from
> everybody else trying to do IO.
> 
> I think this is a case of believing in magic cures and supernatural
> beings.
> 
> 		Linus

IO is IO, BUT:   buffering algorithms do work well almost all the time
under _NORMAL_ conditions , but
streaming hundreds of MBytes from disk (reading) , allocates all buffers
for these files (which you read only once, therefore buffering is not required),
and all other apps and files ( your browser , windowmanager, X server xterm ,
bash etc) are evicted from the buffer cache, and every time (with maybe 10-30
secs in between) you launch a new xterm, click to the KDE's application bar,
the disk has to reload the page for the executables, which makes you run
the disk as it were not buffered.

Is it hard to implement unbuffered disk I/O in Linux (at least for reading) on
per file basis so that reading large files doesn't monopolizes the buffer cache
decreasing the overall caching effectiveness ?

Another solution could be limiting file buffering on a per-process basis,
If the streaming app can tell to the kernel "give me only 4MB of your buffer
cache", then the rest of the apps will still get a chunk of the buffer cache so
that launching my xterm multiple times will work nicely without reading from
disk every time.
Is the second solution maybe easier to implement ?

I'm afraid, if we want that Linux will be a good multimedia OS, this is a
_STRONGLY_NEEDED_  feature.

The user wants to playback his video/audio from disk, and still be able
to launch his apps, without waiting 10 secs for loading a simple xterm.

The SCT's  raw-io patches are nice, but not very suitable in a general purpose
multimedia enviroment, since you can't tell to the user to store his videos on a
raw partition.
  
regards,
Benno.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Alan Cox]

> I'm afraid, if we want that Linux will be a good multimedia OS, this is a
> _STRONGLY_NEEDED_  feature.
> 
> The user wants to playback his video/audio from disk, and still be able
> to launch his apps, without waiting 10 secs for loading a simple xterm.
> 
> The SCT's  raw-io patches are nice, but not very suitable in a general purpose
> multimedia enviroment, since you can't tell to the user to store his videos on a
> raw partition.

Stephens patches are basis of raw I/O on files in a filesystem too.

However what you are saying and raw-io don't neccessarily tally. You are
actually saying "there is a bug in the current page cache handling for
this kind of operation". Far better therefore to fix the heuristic used.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Chuck Lever]

On Mon, 23 Aug 1999, Peter Monta wrote:
> > Will future kernels ave an O_DIRECT like flag to avoid caching ?
> 
> Stephen Tweedie has written a patch for unbuffered I/O; the URL I have
> from some time ago is
> 
> ftp://ftp.linux.org.uk/pub/linux/sct/fs/raw-io
> 
> though this host doesn't seem to respond at the moment.  I believe
> Stephen has mentioned that this will go into the main tree at some point.
> (Possibly it's already in 2.3.x---I haven't looked recently.)
> 
> > I think without some buffer-cache usage limiting, or buffering disabling,
> > Linux is actually UNSUITABLE for streaming applications which do run
> > concurrently with other apps.
> 
> I agree; the raw-io system will make this much more pleasant.

seems to me what you really want is an application-specific file system
that is designed to manage streamed data on disks.  i've been very
impressed with the Sony MiniDisc file system -- it's a simplified file
system that transparently manages data on MO MiniDiscs.

the name space is flat -- you can have 1 to 255 separate "tracks", and
i've seen this system used for monaural or up to 8 concurrent channels of
recording, on 128M MO disks.  i'm no expert, but i'd bet a faster, larger
disk could be used to boost the number of concurrent channels.  it's
probably just a matter of how many channels can be multiplexed into a
track's block stream.

a simple TOC-based block-allocation system is used.  the TOC is stored in
the disk writer's volatile memory, and written out when the disk is
ejected; this eliminates the extra seeks involved with metadata updates.
new blocks are allocated sequentially from free space at the end of the
disk, or eventually from blocks freed by erasing tracks of previously
recorded material. a text area of about 2K per disk is reserved for
tagging each track with "title" data.

for managing a large disk, one might extend such a file system to handle
more tracks, more blocks, and generate a periodic TOC update that writes
the TOC into special areas allocated across the entire disk to minimize
seek latency.

this kind of system would eliminate the need for indirect blocks or even
extent-based allocation, and keep metadata updates very cheap.  it would
also make it easy to combine, divide, and otherwise edit the data in the
tracks.

another example of an application-specific file system was early Dyaxis
gear. the Dyaxis editing suite ran on a Mac II, but used separate disks
and some proprietary form of HFS, i think, to record and edit audio data.

but it might be cool to have a special simplified file system that was
designed specifically to manage time-critical streamed data like
multitrack audio or video.

	- Chuck Lever
--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

The Linux Scalability project:
	http://www.citi.umich.edu/projects/linux-scalability/

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Andrea Arcangeli]

On Mon, 23 Aug 1999, Benno Senoner wrote:

>No , I'm not saying that there is a bug, I'm saying that the
>filebuffering works very well in almost all cases , except of reading large
>files from disk continuously.

Could you check `vmstat 1` and verify if you have continous
swapin/swapout? (I bet yes) If I am right then you'll probably want to
apply this my patch and try again and notice the performance difference
with eyes.

	ftp://ftp.suse.com/pub/people/andrea/kernel-patches/my-2.2.12-final3/no-swapout-2.2.10-B

Andrea

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Benno Senoner]

On Mon, 23 Aug 1999, Mark Hahn wrote:

> > Is it hard to implement unbuffered disk I/O in Linux (at least for reading) on
> 
> this is basically an oxymoron: apps read only from buffers.  the issue is 
> getting the kernel to realize that you're never going to read old buffers
> again.  if you want to help, read the code, don't just complain.

Unfortunately I don't (yet) started kernel hacking, therefore the only thing
I can to now is testing/benchmarking using userspace apps.

> 
> > Another solution could be limiting file buffering on a per-process basis,
> > If the streaming app can tell to the kernel "give me only 4MB of your buffer
> > cache", then the rest of the apps will still get a chunk of the buffer cache so
> 
> yuck!  this would be a grotesque tuning nightmare, and would hold the promise
> of utterly trashing performance in unexpected ways.

This was only an idea I had, it may be dirty but it would work, without
trashing.

 > 
> > I'm afraid, if we want that Linux will be a good multimedia OS, this is a
> > _STRONGLY_NEEDED_  feature.
> 
> debatable.  first, you mean "media _producer_ OS", since media clients
> do not generally mess with this kind of bandwidth.

Ok, playing streaming an AVI video is low-bandwidth , and will not
trash performance of the linux file buffering system.
But what about streaming a DVD movie from the harddisk ?
A DVD movie is made up of GBs of data, and playing back a DVD is not
a media_producer_OS task, it will be very common on every desktop.

>  next, you're also 
> assuming that media production apps should be naive about how they do IO.
> for instance, have you tried using mmap, with explicit munmap's on data
> that you're done with?  or played with some of the parameters on the 
> current code that attempts to prevent cache flushing?

Ah ! thank you for this suggestion,  I will try this mmap() method,
but it is not a very portable way to do disk I/O and makes things a bit more
complicated.
I'm predicting that I will not get much benefits of mmap() due to Murphy's law.
:-) 

Which parameters do you mean ?  /proc/sys/* ?
I did not found anything useful here.

> 
> in short, cache flushing is well-understood and media production is just
> one minor case where it comes up.  hinting (via O_ flags, madvise, ioctl)
> are the right way to turn up the agressiveness of the existing free-behind
> and pre-fetching mechanisms.

I agree, for special situations like this you have to tell the kernel your
intentions, since it's not easy to find out for the kernel through heuristics.
When SCT will release his per-file raw i/o patches then I will rerun my tests.
In the meantime I will play around with mmap().

regards,
Benno.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Benno Senoner]

On Mon, 23 Aug 1999, Andrea Arcangeli wrote:
> On Mon, 23 Aug 1999, Benno Senoner wrote:
> 
> >No , I'm not saying that there is a bug, I'm saying that the
> >filebuffering works very well in almost all cases , except of reading large
> >files from disk continuously.
> 
> Could you check `vmstat 1` and verify if you have continous
> swapin/swapout? (I bet yes) If I am right then you'll probably want to
> apply this my patch and try again and notice the performance difference
> with eyes.
> 
> 	ftp://ftp.suse.com/pub/people/andrea/kernel-patches/my-2.2.12-final3/no-swapout-2.2.10-B
> 
> Andrea

No , have very little swapin/swapouts (only occasionally) when clicking on the
xterm icon. 
Since I run on a 256MB RAM box, I have to wait 30-40sec for the xterm
executable disappearing from buffers ( since I stream about 5MB/sec).
After this time when I must wait up to 5-6sec from clicking to the icon to the
actual launch of the xterm.
Of course this is because the disk is under high load , and loads the
executable slowly, but if I I click on the icon again after only a few secs
(instead of waiting 30-40secs) the 2nd xterm starts up almost immediately since
it's still cached,  200MB filebuffer buffer/5MB/sec = takes about 40secs to fill
all buffers.

Browsing the web while runnig the streaming app, becomes frustrating because 
you have no caching at all while working on a very slow disk (due to disk I/O).
IMHO with direct I/O the rest of the system will remain quite snappy.

regards,
Benno.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Benno Senoner]

On Mon, 23 Aug 1999, Alan Cox wrote:
> > I'm afraid, if we want that Linux will be a good multimedia OS, this is a
> > _STRONGLY_NEEDED_  feature.
> > 
> > The user wants to playback his video/audio from disk, and still be able
> > to launch his apps, without waiting 10 secs for loading a simple xterm.
> > 
> > The SCT's  raw-io patches are nice, but not very suitable in a general purpose
> > multimedia enviroment, since you can't tell to the user to store his videos on a
> > raw partition.
> 
> Stephens patches are basis of raw I/O on files in a filesystem too.

That sounds nice to me !
Any idea when some patches ( for raw file I/O in a filesystem) will be available
? 
> 
> However what you are saying and raw-io don't neccessarily tally. You are
> actually saying "there is a bug in the current page cache handling for
> this kind of operation". Far better therefore to fix the heuristic used.

No , I'm not saying that there is a bug, I'm saying that the
filebuffering works very well in almost all cases , except of reading large
files from disk continuously.
This is of course quite logical since the kernel makes assumptions that you
will read this large file again, and therefore the kernel has to put it into
the buffer cache.

In this case I don't need raw I/O to get fast response or higher throughput,
but only  to avoid that the streaming apps do not monopolizes the buffer, which
is very bad.

will the SCT's patches add an O_DIRECT -like flag ala SGI ?

regards,
Benno.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Andrea Arcangeli]

On Mon, 23 Aug 1999, Benno Senoner wrote:

>No , have very little swapin/swapouts (only occasionally) when clicking on the
>xterm icon. 

As expected.

Incidentally that's exactly the _thing_ you report to be slow just in the
line below. Maybe it's only an unlucky coincidence? ;)

>After this time when I must wait up to 5-6sec from clicking to the icon to the
>actual launch of the xterm.

>Of course this is because the disk is under high load , and loads the

The swapin slowdown things (still more if the disk is under load).

Andrea

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Dave Mielke]

On Mon, 23 Aug 1999, Benno Senoner wrote:
>In this case I don't need raw I/O to get fast response or higher throughput,
>but only  to avoid that the streaming apps do not monopolizes the buffer, which
>is very bad.

You probably don't want strict raw I/O, as there would be considerable benefit
in some sort of read-ahead buffering. What you want is an open/fcntl attribute
which would instruct the kernel that you don't intend to do a seek, in which
case it can free each buffer as soon as it has been fully read.

-- 
Dave Mielke           | 856 Grenon Avenue | I believe that the Bible is the
Phone: 1-613-726-0014 | Ottawa, Ontario   | Word of God. Please contact me
EMail: dave@mielke.cc | Canada  K2B 6G3   | if you're concerned about Hell.

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Gerard Roudier]

If you are in a hurry but are a courageous guy ;-), you can, for now, add
some file control (fcntl) option that tells the kernel to discard the
pages of data of this file once read by user (or just to set the pages at
the end of the LRU in recent kernels), then make appropriate changes in
the kernel and in the application. The kernel will still prefetch file
data on read, allowing read streaming, without having to multithread IOs
at application level.

Sorry if this has been already suggested.

Gérard.

On Mon, 23 Aug 1999, Benno Senoner wrote:

> On Mon, 23 Aug 1999, Alan Cox wrote:
> > > I'm afraid, if we want that Linux will be a good multimedia OS, this is a
> > > _STRONGLY_NEEDED_  feature.
> > > 
> > > The user wants to playback his video/audio from disk, and still be able
> > > to launch his apps, without waiting 10 secs for loading a simple xterm.
> > > 
> > > The SCT's  raw-io patches are nice, but not very suitable in a general purpose
> > > multimedia enviroment, since you can't tell to the user to store his videos on a
> > > raw partition.
> > 
> > Stephens patches are basis of raw I/O on files in a filesystem too.
> 
> That sounds nice to me !
> Any idea when some patches ( for raw file I/O in a filesystem) will be available
> ? 
> > 
> > However what you are saying and raw-io don't neccessarily tally. You are
> > actually saying "there is a bug in the current page cache handling for
> > this kind of operation". Far better therefore to fix the heuristic used.
> 
> No , I'm not saying that there is a bug, I'm saying that the
> filebuffering works very well in almost all cases , except of reading large
> files from disk continuously.
> This is of course quite logical since the kernel makes assumptions that you
> will read this large file again, and therefore the kernel has to put it into
> the buffer cache.
> 
> In this case I don't need raw I/O to get fast response or higher throughput,
> but only  to avoid that the streaming apps do not monopolizes the buffer, which
> is very bad.
> 
> will the SCT's patches add an O_DIRECT -like flag ala SGI ?
> 
> regards,
> Benno.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.rutgers.edu
> Please read the FAQ at http://www.tux.org/lkml/

Re: Streaming disk I/O kills file buffering and makes Linux unusable

[Gerard Roudier]

On Tue, 24 Aug 1999, Gerard Roudier wrote:

> If you are in a hurry but are a courageous guy ;-), you can, for now, add
> some file control (fcntl) option that tells the kernel to discard the
> pages of data of this file once read by user 

> (or just to set the pages at the end of the LRU in recent kernels),

This suggestion anticipates too much about page map LRU inclusion in the
kernel main stream and just makes the suggestion just broken). Without
Andrea's page LRU, it may happen that useful page of data will still be
thrown away after a shrink_mmap revolution.
By the way, if 2.4 does not include Andrea's page LRU, I know of people 
who will be very disappointed.

> then make appropriate changes in
> the kernel and in the application. The kernel will still prefetch file
> data on read, allowing read streaming, without having to multithread IOs
> at application level.
> 
> Sorry if this has been already suggested.
> 
> Gérard.
> 
> On Mon, 23 Aug 1999, Benno Senoner wrote:
> 
> > On Mon, 23 Aug 1999, Alan Cox wrote:
> > > > I'm afraid, if we want that Linux will be a good multimedia OS, this is a
> > > > _STRONGLY_NEEDED_  feature.
> > > > 
> > > > The user wants to playback his video/audio from disk, and still be able
> > > > to launch his apps, without waiting 10 secs for loading a simple xterm.
> > > > 
> > > > The SCT's  raw-io patches are nice, but not very suitable in a general purpose
> > > > multimedia enviroment, since you can't tell to the user to store his videos on a
> > > > raw partition.
> > > 
> > > Stephens patches are basis of raw I/O on files in a filesystem too.
> > 
> > That sounds nice to me !
> > Any idea when some patches ( for raw file I/O in a filesystem) will be available
> > ? 
> > > 
> > > However what you are saying and raw-io don't neccessarily tally. You are
> > > actually saying "there is a bug in the current page cache handling for
> > > this kind of operation". Far better therefore to fix the heuristic used.
> > 
> > No , I'm not saying that there is a bug, I'm saying that the
> > filebuffering works very well in almost all cases , except of reading large
> > files from disk continuously.
> > This is of course quite logical since the kernel makes assumptions that you
> > will read this large file again, and therefore the kernel has to put it into
> > the buffer cache.
> > 
> > In this case I don't need raw I/O to get fast response or higher throughput,
> > but only  to avoid that the streaming apps do not monopolizes the buffer, which
> > is very bad.
> > 
> > will the SCT's patches add an O_DIRECT -like flag ala SGI ?
> > 
> > regards,
> > Benno.
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.rutgers.edu
> > Please read the FAQ at http://www.tux.org/lkml/
> 
>