Raid 1 (non) performance

Raid 1 (non) performance

[Magnus Naeslund(k)]

Hello I might have a strange raid setup that results in bad raid 1 
performance. I would be very happy if someone have any pointers about 
this so that I can get some decent performance out of my box...

This is an mailserver running postfix and dovecot (imap/pop). It's running Ubuntu Dapper amd64 with an 2.6.17.6 since the original dapper kernel doesn't support the sata controller.
The motherboard is some kind of HT1000 based thing with a single opteron 146 processor with 2gb ram, the sata driver I use is sata_svw. The two disks forming the raid1 set are Maxtor 6V250F0 sata disks.
The /var filesystem has some (+j) attributes set on the postfix directories.

I'm not sure what more information is needed, please point out if something is missing.

The raid setup:

# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda3[0] sdb3[1]
      236725696 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      4192896 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      4192832 blocks [2/2] [UU]

unused devices: <none>


Partitions:

# mount|grep ext
/dev/md1 on / type ext3 (rw,noatime,errors=remount-ro)
/dev/md2 on /var type ext3 (rw,noatime)

The tune2fs options and journal sizes on the filesystems:

/dev/md1: Journal size: 67108864 (64mb)
	ext_attr, resize_inode, dir_index, filetype, sparse_super, large_file

/dev/md2: Journal size: 402653184 (384mb)
	resize_inode, dir_index, filetype, sparse_super, large_file

Swaps:

# swapon -s
Filename                                Type            Size    Used 
Priority
/dev/md0                                partition       4192824 116     -1


I've tried to modify the queuing by doing this, to disable the write cache and enable CFQ. The CFQ choice is rather random.

for disk in sda sdb; do
  blktool /dev/$disk wcache off
  hdparm -q -W 0 /dev/$disk
  echo cfq > /sys/block/$disk/queue/scheduler
done

When I search a imap account having 5 gb (approx 500000 mail) worth of mail the box seems to block alot, somehow processes monopolize the IO bandwidth, not like the many other systems I'm running.
The raid1 should be an improvement over using the disks directly on the read side atleast.

Any ideas as of why the performance/fairness is out the window?
Do you need any more info?

Regards,
Magnus

Re: Raid 1 (non) performance

[dean gaudet]

On Wed, 15 Nov 2006, Magnus Naeslund(k) wrote:

> # cat /proc/mdstat
> Personalities : [raid1]
> md2 : active raid1 sda3[0] sdb3[1]
>       236725696 blocks [2/2] [UU]
> 
> md1 : active raid1 sda2[0] sdb2[1]
>       4192896 blocks [2/2] [UU]
> 
> md0 : active raid1 sda1[0] sdb1[1]
>       4192832 blocks [2/2] [UU]

i see you have split /var and / on the same spindle... if your /home is on 
/ then you're causing extra seek action by having two active filesystems 
on the same spindles.  another option to consider is to make / small and 
mostly read-only and move /home to /var/home (and use a symlink or mount 
--bind to place it at /home).

or just put everything in one big / filesystem.

hopefully your swap isn't being used much anyhow.

try "iostat -kx /dev/sd* 5" and see if the split is causing you troubles 
-- i/o activity on more than one partition at once.


> I've tried to modify the queuing by doing this, to disable the write cache and enable CFQ. The CFQ choice is rather random.
> 
> for disk in sda sdb; do
>   blktool /dev/$disk wcache off
>   hdparm -q -W 0 /dev/$disk

turning off write caching is a recipe for disasterous performance on most 
ata disks... unfortunately.  better to buy a UPS and set up nut or apcupsd 
or something to handle shutdown.  or just take your chances.

-dean

Re: Raid 1 (non) performance

[Magnus Naeslund(k)]

dean gaudet wrote:
> 
> i see you have split /var and / on the same spindle... if your /home is on 
> / then you're causing extra seek action by having two active filesystems 
> on the same spindles.  another option to consider is to make / small and 
> mostly read-only and move /home to /var/home (and use a symlink or mount 
> --bind to place it at /home).
> 

Yes I have it like this now:
	/var/bind/home on /home type bind (rw,bind,noatime)

> hopefully your swap isn't being used much anyhow.
> 

Zero used, infact.

> try "iostat -kx /dev/sd* 5" and see if the split is causing you troubles 
> -- i/o activity on more than one partition at once.
> 

It seems to be roughly the same numbers on the disks.

> turning off write caching is a recipe for disasterous performance on most 
> ata disks... unfortunately.  better to buy a UPS and set up nut or apcupsd 
> or something to handle shutdown.  or just take your chances.
> 

This is a mail server, I would like it to not lose mail if there is an power outage.
Is there anything one can do without buying more hardware?


Thanks for the feedback,
Magnus

Re: Raid 1 (non) performance

[Bill Davidsen]

Magnus Naeslund(k) wrote:
> dean gaudet wrote:
>   
>> i see you have split /var and / on the same spindle... if your /home is on 
>> / then you're causing extra seek action by having two active filesystems 
>> on the same spindles.  another option to consider is to make / small and 
>> mostly read-only and move /home to /var/home (and use a symlink or mount 
>> --bind to place it at /home).
>>
>>     
>
> Yes I have it like this now:
> 	/var/bind/home on /home type bind (rw,bind,noatime)
>
>   
>> hopefully your swap isn't being used much anyhow.
>>
>>     
>
> Zero used, infact.
>
>   
>> try "iostat -kx /dev/sd* 5" and see if the split is causing you troubles 
>> -- i/o activity on more than one partition at once.
>>
>>     
>
> It seems to be roughly the same numbers on the disks.
>
>   
>> turning off write caching is a recipe for disasterous performance on most 
>> ata disks... unfortunately.  better to buy a UPS and set up nut or apcupsd 
>> or something to handle shutdown.  or just take your chances.
>>
>>     
>
> This is a mail server, I would like it to not lose mail if there is an power outage.
> Is there anything one can do without buying more hardware?

Nothing you can do will prevent loss of mail locally if the power goes 
out when the mail has gotten to memory and not disk, or the metadata has 
not been updated on the disk. Unless you want to hack your SMTP 
interface to not return an "accepted" status until you have flushed 
everything, get a UPS.

Hint, I looked at this for an ISP internal mailer, and performance 
trying to flush is so bad it's only justified for a very low volume of 
mail. As in legal or financial stuff. If your mail is that critical you 
buy expensive hardware when justified. Otherwise spend a few hundred 
bucks on a good UPS and install the software.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979