Poor RAID6 performance on frankenstein server

Poor RAID6 performance on frankenstein server

[Trevor Cordes]

I'm not sure if this is a kernel-ide-disk issue or a raid issue, so I'm 
trying here first:

I've built a frankenstein file and mythtv server with RAID 6 8x250GB + 
2x300GB disks for 2TB usable space.  I used 4 generic PCI (Promise, 
2xSyba(SiI_IDE), C-Media) and onboard(Intel)+onboard(Promise) PATA 
controllers.  (The 2 bigger 300 disks also house my root/boot/swap RAID1 
partitions.)  I know, cheesy, but this is a SUPER cheap way to get GOBS of 
storage space... those IDE cards are cheap and most of my money goes to 
drives.

Works great... except for during heavy disk access my MythTV recordings
will get somewhat glitchy audio.  Stop the disk access and the audio
returns to normal.  Even with moderate disk access it's ok.  It's just
when it's heavy.  And the glitches are usually every few seconds unless
it's really heavy where the audio gets nearly unintelligible for 10 secs
or so.

The audio, and the video, come through USB2 from a Plextor hardware MPEG4
encoder.  The video +audio stream is not disk intensive, only about
600k/sec.  However, the way the Plextor works is the audio is
software-encoded into MP3 on the fly (ie: not in the Plextor like the
video).  My video almost never glitches -- it's just the audio.

If I run some simple tests I can make it glitch recording some TV while 
doing something like copying a duplicating a large TV file with cp.  The 
CPU load never gets above 34% and bounces between 25-35%.

I have done all the hdparm tweaks I can think of including: DMA (-d1), 
unmasqirq (-u1), lookahead, multiple io 16, 32bit transfers.  I tried 
using an SMP kernel (it's a UP system) for enhanced APIC.  All drives are 
running as master with NO slaves except for 1 (unused during the tests) 
optical drive.  I did used to have 2 drives as slaves and no unmasqirq and 
it was really bad, and things did improve when I moved to all-master and 
enabled unmasqirq.  It's like if I could tweak it just a little more I'd 
have it perfect.

The CPU is a not-slow Pentium 4 2.4/400 on Intel 845D chipset.  RAM is 1GB 
ECC.

Anything else I can try tuning to fix this nagging issue?  Any ideas on 
what's biting me here?

Thanks!  And Merry Christmas to all!

Re: Poor RAID6 performance on frankenstein server

[Max Waterman]

Trevor Cordes wrote:
> ...However, the way the Plextor works is the audio is
> software-encoded into MP3 on the fly (ie: not in the Plextor like the
> video).

I would be tempted to look at the path the audio takes from the plextor 
to the disk. Clearly it goes via the CPU. Seems like there's a 
bottleneck there somewhere - some bus bandwidth issues (does USB share 
the same bus as the disk [ie PCI] at any point?) - I don't know that 
chipset (Intel 845D). Does it depend on which disk array you copy to/from?

...but, no, I don't really know what I'm talking about...so take it for 
what it's worth.

Max.

Re: Poor RAID6 performance on frankenstein server

[Andrew Burgess]

>I'm not sure if this is a kernel-ide-disk issue or a raid issue, so I'm 
>trying here first:

Its an audio issue :-)

>Works great... except for during heavy disk access my MythTV recordings
>will get somewhat glitchy audio.  Stop the disk access and the audio
>returns to normal.  Even with moderate disk access it's ok.  It's just
>when it's heavy.  And the glitches are usually every few seconds unless
>it's really heavy where the audio gets nearly unintelligible for 10 secs
>or so.

What does the audio do exactly? Popping, crackles, dropouts or...
Does MythTV report XRUNS? Using ALSA or OSS?
Can you increase audio buffer sizes in MythTV?

>Anything else I can try tuning to fix this nagging issue?  Any ideas on 
>what's biting me here?

You can try pci latency. Turn it way up for the audio device, way down for
everything else.

I use this at bootup (edited a bit so I hope I didn't break it :-)

# make default 16
latency=10 # hex
echo set default pci latency to 0x$latency
setpci -d *:* latency_timer=$latency # expects hex

# find multimedia devices
pcis=$(lspci -v | grep Multimedia | awk '{ print $1; }')
if test -z "$pcis"; then
  echo WARNING: no multimedia devices found on pci bus
else
  for p in $pcis; do
    echo set latency timer for $p
    setpci -s $p latency_timer=ff
  done
fi

This reduced some audio pops for me but didn't eliminate them...

You can google for pci latency but basically it controls how long a device
holds the bus when another device requests it. If your disk controllers
don't let go soon enough then streaming devices lose.