Software RAID & Filesystem Cache

[Philip Molter <philip@corp.texas.net>]

What is the relationship to software RAID, filesystem caching, and I/O? 
  Here's why I ask:

I have a software application that uses RRD files to monitor numerous 
devices.  All told, we're talking about 21GB of RRD files, about 75% of 
which are being updated every 5 minutes.  Each RRD file is about 3MB in 
size, and updating an RRD file involves a lot of seeking, and writing 
small bits of data to very small parts.  The application should never be 
reading the entire file into memory.  On a properly running system, 
reads are done almost entirely out of filesystem cache on a 4GB box. 
Checking iostat, there is very little read util (<10KB/s over 5 minutes).

Now, I have a box with 12 37GB Raptor 10000RPM drives hooked up to a 
3ware 8500-12 controller.  In software RAID10 under kernel 2.4 (1024k 
chunks), the system works as expected.  Read load is light.  The box has 
plenty of I/O to spare (which it should; those drives are monsters).  In 
software RAID10 under kernel 2.6 (1024k chunks), read load is 
excessively heavy.  Each *drive* reports anywhere from 8-12MB/s (5 
minute average).  The RAID10 device itself reports 95-120MB/s (5 minute 
average).  There is no way the application is actually requesting to 
read that much data.  I trust the figures reported by iostat, though, 
because the application cannot keep up and the system feels excessively 
heavy.  In software RAID10 under kernel 2.6 (32k chunks), the read load 
is lighter, on the order of 1MB/s per disk and 10-12MB/s for the RAID10 
device.  If I convert the box to hardware RAID, the box functions 
normally.  Read levels are even better than 2.4 software RAID 
configuration (literally, less than 1KB/s over a 5 minute average). 
Each configuration is doing exactly the same work, with the exact same 
code and data.  The only difference is the kernel and software RAID 
configurations.

It's almost as if every read request to software RAID under 2.6 is 
bypassing the filesystem cache and furthermore, reading the entire 
chunk, not just the bit of information it needs (that's the only way I 
can explain the drop in read activity between 1024k and 32k).

Kernels are:
2.4.20-31.9 (roughtly 2.4.22)
2.6.7-1.494.2.2 (roughtly 2.6.8-rc2)

Filesystem is ext3

The RAIDs are setup as 6 RAID1 mirrors (2 drives each), striped together.

Any insight is greatly appreciated.
Philip