XFS RAID only writing half the time

XFS RAID only writing half the time

[AndyLiebman]

A puzzle. 

I am writing uncompressed video (18 MB/sec) to a RAID 5 array through a 
Gigabit network. I am using the xfs filesystem on the array with a block size of 
4096 and an internal log. The RAID has a chunk size of 128K. Otherwise, all 
settings are the default (i.e., left symetric algorithm). 

I have a P4-3.06 with 1 GB RAM. I have hyperthreading enabled -- and Mandrake 
9.2 (2.4.22 kernel) shows that I have 2 CPUs. 

While capturing video, I am running a monitoring program, XOSVIEW -- which 
shows usage of memory, cpu, ints (interrupts?), and disk read/write/idle 
activity. 

Although data is coming into my Linux machine at a rate of 18 MB/sec, I don't 
see any disk activity for as much as 30 or 40 seconds after I begin recording 
video. Looking at XOSVIEW, I can see on the memory display that the cache 
fills up until my memory is almost entirely full, and ONLY THEN does the disk 
writing activity begin. 

Usually this is fine, but once in a while -- after capturing video for 20 
minutes or so, the system will have a "hiccup" and fail to write data fast 
enough. My video editing application will then stop capturing. I don'

It seems to me that there must be a way to tell my system not to cache so 
much data before beginning to write so that there is more margin for delays. 

It also seems there should be a way to tell my system to spend more time 
writing to the RAID array. 

When I test out my RAID with Bonnie++, it looks as if sequential writes are 
about 30 MB/sec -- so the RAID itself is fast enough to keep up with the video. 
And so is the gigabit network. 

Does anybody have any ideas? 

Re: XFS RAID only writing half the time

[Hendrik Visage]

On Tue, Nov 25, 2003 at 10:07:56AM -0500, AndyLiebman@aol.com wrote:
> A puzzle. 
> I have a P4-3.06 with 1 GB RAM. I have hyperthreading enabled -- and Mandrake 
> 9.2 (2.4.22 kernel) shows that I have 2 CPUs. 

I've seen similar troubles with the SuSE kernels. I believe it has
something to do with some of the desktop/interactive patches that fills
up memory too much before flushing caches.

> Although data is coming into my Linux machine at a rate of 18 MB/sec, I don't 
> see any disk activity for as much as 30 or 40 seconds after I begin recording 
> video. Looking at XOSVIEW, I can see on the memory display that the cache 
> fills up until my memory is almost entirely full, and ONLY THEN does the disk 
> writing activity begin. 
> 
> Usually this is fine, but once in a while -- after capturing video for 20 
> minutes or so, the system will have a "hiccup" and fail to write data fast 
> enough. My video editing application will then stop capturing. I don'

Play with the settings of: 
sysctl -a | grep vm.bdflush
vm.bdflush = 30 500     0       0       500     3000    60      20      0

I've found the right settings would alleviate this hiccup issue.

HEndrik

Re: XFS RAID only writing half the time

[Net Llama!]

On Tue, 25 Nov 2003, Hendrik Visage wrote:
> On Tue, Nov 25, 2003 at 10:07:56AM -0500, AndyLiebman@aol.com wrote:
> > A puzzle.
> > I have a P4-3.06 with 1 GB RAM. I have hyperthreading enabled -- and Mandrake
> > 9.2 (2.4.22 kernel) shows that I have 2 CPUs.
>
> I've seen similar troubles with the SuSE kernels. I believe it has
> something to do with some of the desktop/interactive patches that fills
> up memory too much before flushing caches.
>
> > Although data is coming into my Linux machine at a rate of 18 MB/sec, I don't
> > see any disk activity for as much as 30 or 40 seconds after I begin recording
> > video. Looking at XOSVIEW, I can see on the memory display that the cache
> > fills up until my memory is almost entirely full, and ONLY THEN does the disk
> > writing activity begin.
> >
> > Usually this is fine, but once in a while -- after capturing video for 20
> > minutes or so, the system will have a "hiccup" and fail to write data fast
> > enough. My video editing application will then stop capturing. I don'
>
> Play with the settings of:
> sysctl -a | grep vm.bdflush
> vm.bdflush = 30 500     0       0       500     3000    60      20      0
>
> I've found the right settings would alleviate this hiccup issue.

Is there any guidance or documentation on how to set those values?

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lonni J Friedman				netllama@linux-sxs.org
Linux Step-by-step & TyGeMo		     http://netllama.ipfox.com

Re: XFS RAID only writing half the time

[AndyLiebman]

Hendrik, 

Thanks for that suggestion. I tried it. Went to Google and under "vm.bdflush" 
found some documentation (http://www.faqs.org/docs/securing/chap6sec68.html) 
that explained what the various numbers mean. I didn't understand them all, 
but when I experimented and put in: 

10  1200  0  0   500     3000    60      20      0  (Just changed the first 
two figures) 

I noticed when looking at XOSVIEW that now I was writing to the disks about 
two or three times more often, and I guess writing smaller amounts of 
information each time. And the video capture lasted much longer than it did with the 
default Mandrake 9.2 setting (exactly the numbers you typed below, by the way). 

So now, do you know of any logical way to tweak these different values? I'm 
not sure that the second value of 1200 makes too much sense for what I'm trying 
to do. It seems like the higher the number in that position, the more 
"bursty" the writes are. 

I'm going to want to try to come up with some settings that allow capturing 
uncompressed video (18 MB/sec) while one or two other users are trying to read 
DV video (3.5 MB/sec) from the same drives. I guess that means I will want get 
the cache written to disk as soon as possible (leaving margin for difficult 
moments) while allowing other users to read from the disk on demand. In some 
ways, reading video files is more demanding than writing -- because what is read 
MUST arrive at the client computer exactly when needed. 

Also, do you know how Samba settings and TCP/IP settings could interact with 
this bdflush setting? I am using Samba 3.0.0 by the way. 

It's exciting that you can do all this tweaking in Linux. I'm a total newbie. 
But it's also a little daunting. 

Thanks again
Andy Liebman




> Although data is coming into my Linux machine at a rate of 18 MB/sec, I 
don't 
> see any disk activity for as much as 30 or 40 seconds after I begin 
recording 
> video. Looking at XOSVIEW, I can see on the memory display that the cache 
> fills up until my memory is almost entirely full, and ONLY THEN does the 
disk 
> writing activity begin. 
> 
> Usually this is fine, but once in a while -- after capturing video for 20 
> minutes or so, the system will have a "hiccup" and fail to write data fast 
> enough. My video editing application will then stop capturing. I don'

Play with the settings of: 
sysctl -a | grep vm.bdflush
vm.bdflush = 30 500    0       0       500     3000    60      20      0    
I've found the right settings would alleviate this hiccup issue.

HEndrik

Re: XFS RAID only writing half the time

[Hendrik Visage]

On Tue, Nov 25, 2003 at 12:41:23PM -0500, AndyLiebman@aol.com wrote:
> Hendrik, 
> 
> Thanks for that suggestion. I tried it. Went to Google and under "vm.bdflush" 
> found some documentation (http://www.faqs.org/docs/securing/chap6sec68.html) 
> that explained what the various numbers mean. I didn't understand them all, 
> but when I experimented and put in: 
> 
> 10  1200  0  0   500     3000    60      20      0  (Just changed the first 
> two figures) 

More Documentation is in /usr/src/linux/Documentation/filesystems/proc.txt,
look for "Table 2-2: Parameters in /proc/sys/vm/bdflush "

 I recall also fixing some values in the middle..

HEndrik

Re: XFS RAID only writing half the time

[Dirk Hufnagel]

AndyLiebman@aol.com wrote:
> Hendrik, 

<snip>

> I'm going to want to try to come up with some settings that allow capturing 
> uncompressed video (18 MB/sec) while one or two other users are trying to read 
> DV video (3.5 MB/sec) from the same drives. I guess that means I will want get 
> the cache written to disk as soon as possible (leaving margin for difficult 
> moments) while allowing other users to read from the disk on demand. In some 
> ways, reading video files is more demanding than writing -- because what is read 
> MUST arrive at the client computer exactly when needed. 

Could be problematic since you can't just add bandwidth requierements
and compare it to the maximum bandwidth. Multiple active disk access
jobs means that the disk will be seeking, which means that the read
accesses will interfere with your write stream, possible to a point
where the writing will skip.

If you are not writing 24/7 and it's in your budget, I would put two
RAID arrays into the machine, one to write to and the other one
for reading. You can synchronize them whenever you are not writing.
Of course, that only works if you don't need to read the written
data immediatly. For the read array I would use Raid5 for the data
security and for the write array you can either use Raid5 or Raid0.
Raid0 is faster compared to Raid5, especially for writes, but it
would mean your data isn't secure until it has been copied to the
other Raid5 array.

	Dirk