Re: kernel 2.6.13 buffer strangeness

[Anthony Wesley <awesley@acquerra.com.au>]

Hi Nate and Roger,

First off, thanks fo taking some time to answer my call for help :-)

I've already spent some time fiddling with dirty_ratio before I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.

Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.

Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.

Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.

So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.

Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.

So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.

Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes. 

Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??

regards, Anthony

Nate Diller wrote:

> On 9/9/05, Roger Heflin <rheflin@atipa.com> wrote:
> 
>>I saw it mentioned before that the kernel only allows a certain
>>percentage of total memory to be dirty, I thought the number was
>>around 40%, and I have seen machines with large amounts of ram,
>>hit the 40% and then put the writing application into disk wait
>>until certain amounts of things are written out, and then take
>>it out of disk wait, and repeat when it again hits 40%, given your
>>rate different it would be close to 40% in 50seconds.
>>
> 
> yes, on 2.6 there are two tunables which are important here. 
> dirty_background_ratio is the threshold where the kernel will begin
> flushing dirty buffers, so it should change how soon the disk becomes
> active.  dirty_ratio changes when the write-throttling code kicks in,
> which is what Anthony is seeing.  The purpose of the write throttling
> code is to limit the dirtying process to disk bandwidth, so that is a
> Feature.  Anthony, try *increasing* dirty_ratio, you can go up to 100,
> but you could trigger an OOM if you let it get too high, so maybe try
> setting it at 85 or so.  This should effectively disable the write
> throttling and give you the bandwidth you want.
> 
> NATE
> 
> 
>>And I think that you mean MB(yte) not Mb(it).
>>
>>                           Roger
>>
>>
>>>-----Original Message-----
>>>From: linux-kernel-owner@vger.kernel.org 
>>>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of 
>>>Anthony Wesley
>>>Sent: Friday, September 09, 2005 4:11 AM
>>>To: linux-kernel@vger.kernel.org
>>>Subject: Re: kernel 2.6.13 buffer strangeness
>>>
>>>Thanks David, but if you read my original post in full you'll 
>>>see that I've tried that, and while I can start the write out 
>>>sooner by lowering /proc/sys/vm/dirty_ratio , it makes no 
>>>difference to the results that I am getting. I still seem to 
>>>run out of steam after only 50 seconds where it should take 
>>>about 3 minutes.
>>>
>>>regards, Anthony
>>>
>>>--
>>>Anthony Wesley
>>>Director and IT/Network Consultant
>>>Smart Networks Pty Ltd
>>>Acquerra Pty Ltd
>>>
>>>Anthony.Wesley@acquerra.com.au
>>>Phone: (02) 62595404 or 0419409836
>>>
>>>-
>>>To unsubscribe from this list: send the line "unsubscribe 
>>>linux-kernel" in the body of a message to 
>>>majordomo@vger.kernel.org More majordomo info at  
>>>http://vger.kernel.org/majordomo-info.html
>>>Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>>
> 
> 

-- 
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd

Anthony.Wesley@acquerra.com.au
Phone: (02) 62595404 or 0419409836