One Large md or Many Smaller md for Better Peformance?

One Large md or Many Smaller md for Better Peformance?

[Moshe Yudkowsky]

Question: with the same number of physical drives,  do I get better 
performance with one large md-based drive, or do I get better 
performance if I have several smaller md-based drives?

Situation: dual CPU, 4 drives (which I will set up as RAID-1 after being 
terrorized by the anti-RAID-5 polemics included in the Debian distro of 
mdadm).

I've two choices:

1. Allocate all the drive space into a single large partition, place 
into a single RAID array (either 10 or 1 + LVM, a separate question).

2. Allocate each drive into several smaller partitions. Make each set of 
smaller partitions into a separate RAID 1 array and use separate RAID md 
drives for the various file systems.

Example use case:

While working other problems, I download a large torrent in the 
background. The torrent writes to its own, separate file system called 
/foo. If /foo is mounted on its own RAID 10 or 1-LVM array, will that 
help or hinder overall system responsiveness?

It would seem a "no brainer" that giving each major filesystem its own 
array would allow for better threading and responsiveness, but I'm 
picking up hints in various piece of documentation that the performance 
can be counter-intuitive. I've even considered the possibility of giving 
/var and /usr separate RAID arrays (data vs. executables).

If an expert could chime in, I'd appreciate it a great deal.


-- 
Moshe Yudkowsky * moshe@pobox.com * www.pobox.com/~moshe
  "There are more ways to skin a cat than nuking it from orbit
     -- but it's the only way to be sure."
     				-- Eliezer Yudkowsky

Re: One Large md or Many Smaller md for Better Peformance?

[Iustin Pop]

On Sun, Jan 20, 2008 at 02:24:46PM -0600, Moshe Yudkowsky wrote:
> Question: with the same number of physical drives,  do I get better  
> performance with one large md-based drive, or do I get better  
> performance if I have several smaller md-based drives?

No expert here, but my opinion:
  - md code works better if it's only one array per physical drive,
    because it keeps statistics per array (like last accessed sector,
    etc.) and if you combine two arrays on the same drive these
    statistics are not exactly true anymore
  - simply separating 'application work areas' into different
    filesystems is IMHO enogh, no need to separate the raid arrays too
  - if you download torrents, fragmentation is a real problem, so use a
    filesystem that knows how to preallocate space (XFS and maybe ext4;
    for XFS use xfs_io to set a bigger extend size for where you
    download)

regards,
iustin

Re: One Large md or Many Smaller md for Better Peformance?

[Bill Davidsen]

Moshe Yudkowsky wrote:
> Question: with the same number of physical drives,  do I get better 
> performance with one large md-based drive, or do I get better 
> performance if I have several smaller md-based drives?
>
> Situation: dual CPU, 4 drives (which I will set up as RAID-1 after 
> being terrorized by the anti-RAID-5 polemics included in the Debian 
> distro of mdadm).
>
> I've two choices:
>
> 1. Allocate all the drive space into a single large partition, place 
> into a single RAID array (either 10 or 1 + LVM, a separate question).
>
One partitionable RAID-10, perhaps, then partition as needed. Read the 
discussion here about performance of LVM and RAID. I personally don't do 
LVM unless I know I will have to have great flexibility of configuration 
and can give up performance to get it. Other report different results, 
so make up your own mind.
> 2. Allocate each drive into several smaller partitions. Make each set 
> of smaller partitions into a separate RAID 1 array and use separate 
> RAID md drives for the various file systems.
>
> Example use case:
>
> While working other problems, I download a large torrent in the 
> background. The torrent writes to its own, separate file system called 
> /foo. If /foo is mounted on its own RAID 10 or 1-LVM array, will that 
> help or hinder overall system responsiveness?
>
> It would seem a "no brainer" that giving each major filesystem its own 
> array would allow for better threading and responsiveness, but I'm 
> picking up hints in various piece of documentation that the 
> performance can be counter-intuitive. I've even considered the 
> possibility of giving /var and /usr separate RAID arrays (data vs. 
> executables).
>
> If an expert could chime in, I'd appreciate it a great deal.
>
>


-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: One Large md or Many Smaller md for Better Performance?

[Moshe Yudkowsky]

Bill Davidsen wrote:

> One partitionable RAID-10, perhaps, then partition as needed. Read the 
> discussion here about performance of LVM and RAID. I personally don't do 
> LVM unless I know I will have to have great flexibility of configuration 
> and can give up performance to get it. Other report different results, 
> so make up your own mind.

I've used Google to search (again) through the archives of the 
newsgroup, and "performance lvm" turns up relevant discussions back in 
2004 or so, but nothing very recent. Am I missing some other location 
for discussions of this question, or perhaps I'm looking in the wrong 
places? (The Wiki didn't help either.)

-- 
Moshe Yudkowsky * moshe@pobox.com * www.pobox.com/~moshe
"Rumor is information distilled so finely that it can filter through 
anything."
          --  Terry Pratchet, _Feet of Clay_

Re: One Large md or Many Smaller md for Better Peformance?

[Moshe Yudkowsky]

Thanks for the tips, and in particular:

Iustin Pop wrote:

>   - if you download torrents, fragmentation is a real problem, so use a
>     filesystem that knows how to preallocate space (XFS and maybe ext4;
>     for XFS use xfs_io to set a bigger extend size for where you
>     download)

That's a very interesting idea; it also gives me an opportunity to 
experiment with XFS. I had been avoiding it because of possible 
power-failure issues on writes.

-- 
Moshe Yudkowsky * moshe@pobox.com * www.pobox.com/~moshe
  "She will have fun who knows when to work
   and when not to work."
    			-- Segami

Re: One Large md or Many Smaller md for Better Peformance?

[Ask Bjørn Hansen]

On Jan 20, 2008, at 2:18 PM, Bill Davidsen wrote:

> One partitionable RAID-10, perhaps, then partition as needed. Read  
> the discussion here about performance of LVM and RAID. I personally  
> don't do LVM unless I know I will have to have great flexibility of  
> configuration and can give up performance to get it. Other report  
> different results, so make up your own mind.


On MySQL servers I always always use LVM, even if performance is  
critical there: Snapshots!  They make it easy and efficient to do an  
online snapshot after just freezing the database for a second or three.


  - ask

[1] http://lenz.homelinux.org/mylvmbackup/

-- 
http://develooper.com/ - http://askask.com/

Re: One Large md or Many Smaller md for Better Peformance?

[Carlos Carvalho]

Moshe Yudkowsky (moshe@pobox.com) wrote on 20 January 2008 21:19:
 >Thanks for the tips, and in particular:
 >
 >Iustin Pop wrote:
 >
 >>   - if you download torrents, fragmentation is a real problem, so use a
 >>     filesystem that knows how to preallocate space (XFS and maybe ext4;
 >>     for XFS use xfs_io to set a bigger extend size for where you
 >>     download)
 >
 >That's a very interesting idea; it also gives me an opportunity to 
 >experiment with XFS. I had been avoiding it because of possible 
 >power-failure issues on writes.

I use reiser3 and xfs. reiser3 is very good with many small files. A
simple test shows interactively perceptible results: removing large
files is faster with xfs, removing large directories (ex. the kernel
tree) is faster with reiser3.

Re: One Large md or Many Smaller md for Better Peformance?

[Moshe Yudkowsky]

Carlos Carvalho wrote:

> I use reiser3 and xfs. reiser3 is very good with many small files. A
> simple test shows interactively perceptible results: removing large
> files is faster with xfs, removing large directories (ex. the kernel
> tree) is faster with reiser3.

My current main concern about XFS and reiser3 is writebacks. The default 
mode for ext3 is "journal," which in case of power failure is more 
robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm 
given to understand.

On the other hand, I have a UPS and it should shut down gracefully 
regardless if there's a power failure. I wonder if I'm being too cautious?


-- 
Moshe Yudkowsky * moshe@pobox.com * www.pobox.com/~moshe
  "Keep some secrets/Never tell,
   And they will keep you very well."
    					-- Michelle Shocked

Re: One Large md or Many Smaller md for Better Peformance?

[Tomasz Chmielewski]

Moshe Yudkowsky schrieb:
> Carlos Carvalho wrote:
> 
>> I use reiser3 and xfs. reiser3 is very good with many small files. A
>> simple test shows interactively perceptible results: removing large
>> files is faster with xfs, removing large directories (ex. the kernel
>> tree) is faster with reiser3.
> 
> My current main concern about XFS and reiser3 is writebacks. The default 
> mode for ext3 is "journal," which in case of power failure is more 
> robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm 
> given to understand.

Also, barriers ("barrier=1" option for ext3) are not supported on 
filesystems placed on md/dm, it's a bit of a pain.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: One Large md or Many Smaller md for Better Peformance?

[Bill Davidsen]

Moshe Yudkowsky wrote:
> Carlos Carvalho wrote:
>
>> I use reiser3 and xfs. reiser3 is very good with many small files. A
>> simple test shows interactively perceptible results: removing large
>> files is faster with xfs, removing large directories (ex. the kernel
>> tree) is faster with reiser3.
>
> My current main concern about XFS and reiser3 is writebacks. The 
> default mode for ext3 is "journal," which in case of power failure is 
> more robust than the writeback modes of XFS, reiser3, or JFS -- or so 
> I'm given to understand.
>
> On the other hand, I have a UPS and it should shut down gracefully 
> regardless if there's a power failure. I wonder if I'm being too 
> cautious?
>
No.

If you haven't actually *tested* the UPS failover code to be sure your 
system is talking to the UPS properly, and that the UPS is able to hold 
up power long enough for a shutdown after the system detects the 
problem, then you don't know if you actually have protection.  Even 
then, if you don't proactively replace batteries on schedule, etc, then 
you aren't as protected as you might wish to be.

And CPU fans fail, capacitors pop, power supplies fail, etc. These are 
things which have happened here in the last ten years. I also had a 
charging circuit in a UPS half-fail (from full wave rectifier to half). 
So the UPS would discharge until it ran out of power, then the system 
would fail hard. By the time I got on site and rebooted the UPS had 
trickle charged and would run the system. After replacing two 
"intermittent power supplies" in the system, the UPS was swapped on 
general principles and the real problem was isolated.

Shit happens, don't rely on graceful shutdowns (or recovery, have backups).

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: One Large md or Many Smaller md for Better Peformance?

[Iustin Pop]

On Tue, Jan 22, 2008 at 05:34:14AM -0600, Moshe Yudkowsky wrote:
> Carlos Carvalho wrote:
>
>> I use reiser3 and xfs. reiser3 is very good with many small files. A
>> simple test shows interactively perceptible results: removing large
>> files is faster with xfs, removing large directories (ex. the kernel
>> tree) is faster with reiser3.
>
> My current main concern about XFS and reiser3 is writebacks. The default  
> mode for ext3 is "journal," which in case of power failure is more  
> robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm  
> given to understand.
>
> On the other hand, I have a UPS and it should shut down gracefully  
> regardless if there's a power failure. I wonder if I'm being too 
> cautious?

I'm not sure what your actual worry is. It's not like XFS loses
*commited* data on power failure. It may lose data that was never
required to go to disk via fsync()/fdatasync()/sync. If someone is
losing data on power failure is the unprotected write cache of the
harddrive.

If you have properly-behaved applications, then they know when to do an
fsync and if XFS returns success on fsync and your linux is properly
configured (no write-back caches on drives that are not backed by NVRAM,
etc.) then you won't lose data.

regards,
iustin