public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* Re: kernel 2.6.13 buffer strangeness
@ 2005-09-09  9:11 Anthony Wesley
  2005-09-09 15:09 ` Roger Heflin
  0 siblings, 1 reply; 17+ messages in thread
From: Anthony Wesley @ 2005-09-09  9:11 UTC (permalink / raw)
  To: linux-kernel

Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.

regards, Anthony

-- 
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

Anthony.Wesley@acquerra.com.au
Phone: (02) 62595404 or 0419409836


^ permalink raw reply	[flat|nested] 17+ messages in thread
* kernel 2.6.13 buffer strangeness
@ 2005-09-09  8:14 Anthony Wesley
  2005-09-09  8:24 ` David Lang
  0 siblings, 1 reply; 17+ messages in thread
From: Anthony Wesley @ 2005-09-09  8:14 UTC (permalink / raw)
  To: linux-kernel

Hi everyone. I'm seeing something very Odd with 2.6.11 / 2.6.12 and 2.6.13, it has me stumped so I want to describe my 
problem here and hopefully some kind soul can verify that what I am seeing is a Feature and NotABug :-)

It it's a Feature, then I'll find some way to work around it, but it seems to me that it's a Bug :-)

I have a digital video stream coming in at 25Mb/s over a firewire connector, and this data is being written out to a 
usb2 hard drive as it's captured. The hard drive can only sustain about 17Mb/s write speed, so I am in deficit by 7Mb 
per second.

I'm using a Dell Inspiron laptop with a 2GHz p4 and 1.5Gb of physical RAM, running a FC3 based distro but with a number 
of custom kernels, including 2.6.11, 2.6.12 and 2.6.13. I see the same issue on all of them.

Apart from X and my capture application (Coriander) there is nothing else happening on the machine, and so I see about 
1.3Gb of physical ram is available for buffering this data.

Okay, so what I *expect* to see is this: The video will stream in and out of Coriander at 25Mb/s until all the free RAM 
is full of dirty data waiting to be written to disk. At a deficit of 7Mb/s this should take about 3 minutes to fill 
1.3Gb. When this point is reached I would see my write speed suddenly drop from 25Mb/s to 17Mb/s because there is 
effectively no more RAM to allocate for buffers.

*But* what I actually see is that I hit this point after only 50 seconds. This is very puzzling. After 50 seconds of 
operation I see my video write speed suddenly drop to 17Mb/s.

Coincidentally (or not...) this is about how long it takes the kernel to allocate all of the 1.3Gb of RAM to disk 
buffers. If I watch "free" while I am running I can see the amount of allocated buffers growing by 25Mb/s until it is 
all used, and then my write speed immediately drops.

It's as if the kernel isn't bothering to write out any of my data until it runs out of RAM.

Now, I'm pretty sure that the data is streaming out to the disk, cause I can see the disk activity light solidly lit on 
the usb2 drive (very scientific). There is a short delay after I start streaming before it comes on, maybe 4 or 5 
seconds. I've tinkered with the VM settings in /proc/sys/vm and I can make the kernel start writing data faster by 
lowering dirty_ratio, but this does not change the symptoms that I am seeing.

When all 1.3Gb has been allocated to buffers after 50 seconds, 900M of that should have become free to be recycled as it 
has been written out to the usb2 drive at 17Mb/s - Somehow this doesn't seem to be happening.

The result that I am seeing could be explained if the kernel was not releasing dirty pages after they have been written 
to disk, but I know this doesn't make sense.

Can anyone make sense of this and let me know whether this is expected behaviour or not?

I'd prefer to read any responses on the list - if you email me directly then you'll have to get past my spam filter.

Thanks in advance,

Anthony

-- 
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

Anthony.Wesley@acquerra.com.au
Phone: (02) 62595404 or 0419409836


^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2005-09-10 11:52 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-09-09  9:11 kernel 2.6.13 buffer strangeness Anthony Wesley
2005-09-09 15:09 ` Roger Heflin
2005-09-09 21:39   ` Nate Diller
2005-09-10  0:16     ` Anthony Wesley
2005-09-10  0:35       ` Nate Diller
2005-09-10  1:07         ` Anthony Wesley
2005-09-10  1:47           ` Nate Diller
2005-09-10  2:23             ` Anthony Wesley
     [not found]               ` <5c49b0ed05090922021b8f8112@mail.gmail.com>
2005-09-10  5:32                 ` Anthony Wesley
2005-09-10  6:02                 ` kernel 2.6.13 buffer strangeness - FIXED Anthony Wesley
2005-09-10 10:23                 ` kernel 2.6.13 buffer strangeness - ext2/3/reiser4/xfs comparison Anthony Wesley
2005-09-10 11:42                   ` Andrew Morton
2005-09-10 11:56                     ` Anthony Wesley
2005-09-10  0:50     ` kernel 2.6.13 buffer strangeness Anthony Wesley
2005-09-10  5:41       ` Andrew Morton
  -- strict thread matches above, loose matches on Subject: below --
2005-09-09  8:14 Anthony Wesley
2005-09-09  8:24 ` David Lang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox