Re: Slow disks.

[Rogier Wolff <R.E.Wolff@BitWizard.nl>]

On Thu, Dec 23, 2010 at 09:40:54AM -0500, Jeff Moyer wrote:
> > In my performance calculations, 10ms average seek (should be around
> > 7), 4ms average rotational latency for a total of 14ms. This would
> > degrade for read-modify-write to 10+4+8 = 22ms. Still 10 times better
> > than what we observe: service times on the order of 200-300ms. 
> 
> I didn't say it would account for all of your degradation, just that it
> could affect performance.  I'm sorry if I wasn't clear on that.

We can live with a "2x performance degradation" due to stupid
configuration. But not with the 10x -30x that we're seeing now. 

> >  > md1 : active raid5 sda2[0] sdd2[3](S) sdb2[1] sdc2[4]
> >> >       39067648 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]
> >> > [UUU]
> >> 
> >> A 512KB raid5 chunk with 4KB I/Os?  That is a recipe for inefficiency.
> >> Again, blktrace data would be helpful.
> >
> > Where did you get the 4kb IOs from? You mean from the iostat -x
> > output?
> 
> Yes, since that's all I have to go on at the moment.
> 
> > The system/filesystem decided to do those small IOs. With the
> > throughput we're getting on the filesystem, it better not try to write
> > larger chuncks...
> 
> Your logic is a bit flawed, for so many reasons I'm not even going to
> try to enumerate them here.  Anyway, I'll continue to sound like a
> broken record and ask for blktrace data.

Here it is. 

http://prive.bitwizard.nl/blktrace.log

I can't read those yet... Manual is unclear. 

I'd guess that "D" means "submitted to driver". and "C" means
"completed". I very often see a D followed VERY shortly by a C. Also I
see more C's than "D"s.

Anohter way of looking at it, was to sort on the "ID" field. I would
expect each "transaction" to follow similar steps. But many IDs only
occur twice, and not the same for each. 

> > I have benchmarked my own "high bandwidth" raid arrays. I benchmarked
> > them with 128k, 256, 512 and 1024k blocksize. I got the best
> > throughput (for my benchmark: dd if=/dev/md0 of=/dev/null bs=1024k)
> > with 512k blocksize. (and yes that IS a valid benchmark for my
> > usage of the array.)
> 
> Sorry, I'm not sure I understand how this is relevant.  I thought we
> were troubleshooting a problem on someone else's system.  Further, the
> window into the workload we saw via iostat definitely shows that smaller
> I/Os are issued.

My friend confessed to me today that he determined the "optimal" RAID
block size with the exact same test as I had done, and reached the
same conclusion. So that explains his raid blocksize of 512k. 

The system is a mailserver running on a raid on three of the disks.
most of the IOs are generated by the mail server software through the
FS driver, and the raid system. It's not that we're running a database
that inherently requires 4k IOs. Apparently what the
system needs are those small IOs. 

	Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ