More on O_STREAMING (goodby read pauses)

More on O_STREAMING (goodby read pauses)

[J.A. Magallon]

Hi all...

I have discovered one other (good) side effect of O_STREAMING.

I tested it on my duap PII@400, ~900Mb RAM.

Timings:

Without O_STREAMING:
  0.00user 10.13system 0:41.16elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (81major+2574minor)pagefaults 0swaps
With O_STREAMING:
  0.00user 11.03system 0:40.06elapsed 27%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (81major+2574minor)pagefaults 0swaps

Memory:

Before
        total:    used:    free:  shared: buffers:  cached:
Mem:  926797824 228376576 698421248        0 20729856 95637504
Swap: 353644544   114688 353529856
MemTotal:       905076 kB
MemFree:        682052 kB
MemShared:           0 kB
Buffers:         20244 kB
Cached:          93284 kB
SwapCached:        112 kB
Active:          77284 kB
Inactive:        97160 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       905076 kB
LowFree:        682052 kB
SwapTotal:      345356 kB
SwapFree:       345244 kB
BigFree:             0 kB

After test without O_STREAMING:
        total:    used:    free:  shared: buffers:  cached:
Mem:  926797824 910376960 16420864        0 13369344 788623360
Swap: 353644544  1077248 352567296
MemTotal:       905076 kB
MemFree:         16036 kB
MemShared:           0 kB
Buffers:         13056 kB
Cached:         769088 kB
SwapCached:       1052 kB
Active:          83672 kB
Inactive:       759388 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       905076 kB
LowFree:         16036 kB
SwapTotal:      345356 kB
SwapFree:       344304 kB
BigFree:             0 kB

After test with O_STREAMING:
        total:    used:    free:  shared: buffers:  cached:
Mem:  926797824 228241408 698556416        0 20647936 95596544
Swap: 353644544   114688 353529856
MemTotal:       905076 kB
MemFree:        682184 kB
MemShared:           0 kB
Buffers:         20164 kB
Cached:          93244 kB
SwapCached:        112 kB
Active:          77200 kB
Inactive:        97112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       905076 kB
LowFree:        682184 kB
SwapTotal:      345356 kB
SwapFree:       345244 kB
BigFree:             0 kB

But I did the test with an addition: read a 1Gb file and print an '*'
after every 10M. Without O_STREAMING, when memory fills, the 'progress
bar' stalls for a few seconds while pages are sent to disk.
So the patch also favours a constant sustained rate of read from the
disk. Very interesting for things like video edition and so on.
I like it ;).

Thanks.

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.1 (Cooker) for i586
Linux 2.4.20-pre10-jam1 (gcc 3.2 (Mandrake Linux 9.0 3.2-2mdk))

Re: More on O_STREAMING (goodby read pauses)

[Robert Love]

On Wed, 2002-10-09 at 18:23, J.A. Magallon wrote:

> But I did the test with an addition: read a 1Gb file and print an '*'
> after every 10M. Without O_STREAMING, when memory fills, the 'progress
> bar' stalls for a few seconds while pages are sent to disk.
> So the patch also favours a constant sustained rate of read from the
> disk. Very interesting for things like video edition and so on.
> I like it ;).

This is 100% the point of the patch and hopefully the point I proved
when I first posted it.

	Robert Love

Re: More on O_STREAMING (goodby read pauses)

[Mark Mielke]

On Wed, Oct 09, 2002 at 06:43:52PM -0400, Robert Love wrote:
> On Wed, 2002-10-09 at 18:23, J.A. Magallon wrote:
> > But I did the test with an addition: read a 1Gb file and print an '*'
> > after every 10M. Without O_STREAMING, when memory fills, the 'progress
> > bar' stalls for a few seconds while pages are sent to disk.
> > So the patch also favours a constant sustained rate of read from the
> > disk. Very interesting for things like video edition and so on.
> > I like it ;).
> This is 100% the point of the patch and hopefully the point I proved
> when I first posted it.

I assume the stall is not 'while pages are sent to disk', but rather
until kswapd gets around to freeing enough pages to allow memory to
fill again. The stall is due to the pages being fully analyzed to
determine which ones should go, and which ones shouldn't. O_STREAMING
removes the pages ahead of time, so no analysis is ever required.

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: More on O_STREAMING (goodby read pauses)

[J.A. Magallon]

On 2002.10.10 Mark Mielke wrote:
>On Wed, Oct 09, 2002 at 06:43:52PM -0400, Robert Love wrote:
>> On Wed, 2002-10-09 at 18:23, J.A. Magallon wrote:
>> > But I did the test with an addition: read a 1Gb file and print an '*'
>> > after every 10M. Without O_STREAMING, when memory fills, the 'progress
>> > bar' stalls for a few seconds while pages are sent to disk.
>> > So the patch also favours a constant sustained rate of read from the
>> > disk. Very interesting for things like video edition and so on.
>> > I like it ;).
>> This is 100% the point of the patch and hopefully the point I proved
>> when I first posted it.
>
>I assume the stall is not 'while pages are sent to disk', but rather
>until kswapd gets around to freeing enough pages to allow memory to
>fill again. The stall is due to the pages being fully analyzed to
>determine which ones should go, and which ones shouldn't. O_STREAMING
>removes the pages ahead of time, so no analysis is ever required.
>

I can _hear_ the disk activity when the stall happens, so selecting what
to drop is fast, but then you have to write it...

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.1 (Cooker) for i586
Linux 2.4.20-pre10-jam1 (gcc 3.2 (Mandrake Linux 9.0 3.2-2mdk))

Re: More on O_STREAMING (goodby read pauses)

[Mark Mielke]

On Thu, Oct 10, 2002 at 04:39:27PM +0200, J.A. Magallon wrote:
> On 2002.10.10 Mark Mielke wrote:
> >I assume the stall is not 'while pages are sent to disk', but rather
> >until kswapd gets around to freeing enough pages to allow memory to
> >fill again. The stall is due to the pages being fully analyzed to
> >determine which ones should go, and which ones shouldn't. O_STREAMING
> >removes the pages ahead of time, so no analysis is ever required.
> I can _hear_ the disk activity when the stall happens, so selecting what
> to drop is fast, but then you have to write it...

I don't think this is right. Prove me wrong by explaining how kswapd works,
but if a page isn't dirty, there is no need to write it out to disk.

My (perhaps incorrect) assumption is that kswapd prefers to swap on clean
pages over dirty pages. If your pages are mostly clean, there is nothing
to write to disk the clear majority of the time.

Clean read-only pages should *never* be written to swap. They can be re-read
from their source.

I _think_ what you are seeing is that kswapd is not cleaning pages out
fast enough, which means that *other* tasks executing need to have their
*swapped out* pages *read* from disk. I.e. the churning you hear is probably
mostly reads - not writes.

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: More on O_STREAMING (goodby read pauses)

[J.A. Magallon]

On 2002.10.10 Mark Mielke wrote:
>On Thu, Oct 10, 2002 at 04:39:27PM +0200, J.A. Magallon wrote:
>> On 2002.10.10 Mark Mielke wrote:
>> >I assume the stall is not 'while pages are sent to disk', but rather
>> >until kswapd gets around to freeing enough pages to allow memory to
>> >fill again. The stall is due to the pages being fully analyzed to
>> >determine which ones should go, and which ones shouldn't. O_STREAMING
>> >removes the pages ahead of time, so no analysis is ever required.
>> I can _hear_ the disk activity when the stall happens, so selecting what
>> to drop is fast, but then you have to write it...
>
>I don't think this is right. Prove me wrong by explaining how kswapd works,
>but if a page isn't dirty, there is no need to write it out to disk.
>

You suppose it is a page of code ? What about data that programs malloc() ???
You can also send data memory to swap. If you do not write it on disk,
how do you recover it ???

>My (perhaps incorrect) assumption is that kswapd prefers to swap on clean
>pages over dirty pages. If your pages are mostly clean, there is nothing
>to write to disk the clear majority of the time.
>
>Clean read-only pages should *never* be written to swap. They can be re-read
>from their source.

That is your fault, <read-only>. Pages maped read-only are those from 
binary executables or shared libraries, but, again, what about data ?

>
>I _think_ what you are seeing is that kswapd is not cleaning pages out
>fast enough, which means that *other* tasks executing need to have their
>*swapped out* pages *read* from disk. I.e. the churning you hear is probably
>mostly reads - not writes.
>

I look at gnome system monitor graph for mem. I start with a tiny amount of
used memory. Start the 1Gb read without O_STREAM, the blue area in monitor
starts to grow linearly in time, stars (*) from the reader appear at a
given rate, and as soon as it touches the top limit the stars stop, the disk
begins to thrash, and swap space used grows. After a 2-4 seconds, the stars
go again with the same rate. Tell me what is this but swapper writing pages,
and reading the new pages for my giga.

With O_STREAM, the 'blue bar' does not move from its place, and star rate
(ie, read rate from disk), stays uniform.


-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.1 (Cooker) for i586
Linux 2.4.20-pre10-jam1 (gcc 3.2 (Mandrake Linux 9.0 3.2-2mdk))

Re: More on O_STREAMING (goodby read pauses)

[Andrew Morton]

"J.A. Magallon" wrote:
> 
> ...
> I look at gnome system monitor graph for mem. I start with a tiny amount of
> used memory. Start the 1Gb read without O_STREAM, the blue area in monitor
> starts to grow linearly in time, stars (*) from the reader appear at a
> given rate, and as soon as it touches the top limit the stars stop, the disk
> begins to thrash, and swap space used grows. After a 2-4 seconds, the stars
> go again with the same rate. Tell me what is this but swapper writing pages,
> and reading the new pages for my giga.
> 

That's fairly rude behaviour for a 2.4 kernel.  Sounds like 2.5 ;)

What kernel is that?

Re: More on O_STREAMING (goodby read pauses)

[Mark Mielke]

On Fri, Oct 11, 2002 at 12:50:52AM +0200, J.A. Magallon wrote:
> On 2002.10.10 Mark Mielke wrote:
> >I don't think this is right. Prove me wrong by explaining how kswapd works,
> >but if a page isn't dirty, there is no need to write it out to disk.
> You suppose it is a page of code ? What about data that programs malloc() ???
> You can also send data memory to swap. If you do not write it on disk,
> how do you recover it ???

Unless you are telling me that you have 1Gb of dirty pages in your
swap, it doesn't matter. The majority of the pages are not dirty, and
they are only in use because the 1Gb+ file was read from front to
back.

> >Clean read-only pages should *never* be written to swap. They can be re-read
> >from their source.
> That is your fault, <read-only>. Pages maped read-only are those from 
> binary executables or shared libraries, but, again, what about data ?

I shouldn't have said read only. The real qualifier is the 'clean' adjective.

It doesn't matter if a page is read-write or read-only, if the page
came directly from disk, and no modifications have been made to the
page, the page can safely be turfed and should _never_ be written to
swap.

> >I _think_ what you are seeing is that kswapd is not cleaning pages out
> >fast enough, which means that *other* tasks executing need to have their
> >*swapped out* pages *read* from disk. I.e. the churning you hear is probably
> >mostly reads - not writes.
> I look at gnome system monitor graph for mem. I start with a tiny amount of
> used memory. Start the 1Gb read without O_STREAM, the blue area in monitor
> starts to grow linearly in time, stars (*) from the reader appear at a
> given rate, and as soon as it touches the top limit the stars stop, the disk
> begins to thrash, and swap space used grows. After a 2-4 seconds, the stars
> go again with the same rate. Tell me what is this but swapper writing pages,
> and reading the new pages for my giga.
> With O_STREAM, the 'blue bar' does not move from its place, and star rate
> (ie, read rate from disk), stays uniform.

It sounds like your kernel is broken.

mark

-- 
mark@mielke.cc/markm@ncf.ca/markm@nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

Re: More on O_STREAMING (goodby read pauses)

[Samium Gromoff]

   So i suspect this O_STREAMING will revive the disk->disk
 data copying transfer rate for large files like these bloody
 mpeg4 .avi`s ;)

   Disk transfers depends on the drive being constantly busy
 and this new report from the battlefield brings a new hope.

   The only thing left is to adopt the userland tools
 like cp and mc...

---
cheers,
   Samium Gromoff
_________________________________
_____________________________________