Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Alex Madarasz]

We're building a new Fedora 8.0.1 Linux system to stream data from a
250Msps ADC to disk, and want to start tuning the system configuration
for maximum XFS write performance.  To date, without any significant
effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
push the tuning as far as we can go with this architecture before we
start looking at other hardware options.  Looking at various other
tuning pages on the Web finds few that are interested in maxing out
sequential writes to very large arrays while using SAS HW RAID with big
fast SAS drives too.


HW: HP ML350 G5
 - 2 x Dual-Core Intel 5160 Xeon (3GHz, 1333 FSB) CPUs
 - 8GB dual-interleaved memory
 - HP Smart Array E200i (cciss) embedded PCI-E RAID ctrlr
   - 128MB cache
   - 2 x WD RE2 500GB 7200RPM SATA drives
   - HW RAID 1
   - ext3 root LVM and Linux system partitions
 - HP Smart Array P400 (cciss) PCI-E x8 RAID ctrlr
   - 512MB cache
   - 8 x HP/Seagate 300GB 15000RPM SAS drives
   - HW RAID 0 logical disk
   - Dedicated xfs data partition
 - 250Msps ADC on 64-bit/100MHz PCI-X


XFS Tuning Options?

- HW RAID0:
  - Array/logical disk HW RAID stripe size?
  - Cache enabled (some reports that cache s/b turned off?)?
  - xfs mkfs / mount options?

- External Log?
  - How big a partition on the E200i?
  - mkfs / mount options?

...?

Thanks for any tips,

-- 
Alex

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Eric Sandeen]

Alex Madarasz wrote:
> We're building a new Fedora 8.0.1 Linux system to stream data from a
> 250Msps ADC to disk, and want to start tuning the system configuration
> for maximum XFS write performance.  To date, without any significant
> effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
> with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
> push the tuning as far as we can go with this architecture before we
> start looking at other hardware options.  Looking at various other
> tuning pages on the Web finds few that are interested in maxing out
> sequential writes to very large arrays while using SAS HW RAID with big
> fast SAS drives too.

...

> XFS Tuning Options?
> 
> - HW RAID0:
>   - Array/logical disk HW RAID stripe size?

At any rate you'll want to match xfs's geometry with the raid geometry.

>   - Cache enabled (some reports that cache s/b turned off?)?

If it's battery-backed cache, leave it on, and disable barriers in xfs
(it's a mount option)

>   - xfs mkfs / mount options?

David mentioned these before as a generic place to start:

# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
<dev>
# mount -o logbsize=256k <dev> <mtpt>

and that those would be upcoming new defaults for mkfs.

4 ags may not be what you want for a ~2T filesystem.

> - External Log?
>   - How big a partition on the E200i?
>   - mkfs / mount options?

not sure if an external log would be beneficial or not.

I'm sure others will have more concrete suggestions.  It might be
interesting to post any performance you're getting which does not meet
your expectations...

-Eric

> 
> ...?
> 
> Thanks for any tips,
> 

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Justin Piszcz]

On Sat, 15 Dec 2007, Eric Sandeen wrote:

> Alex Madarasz wrote:
>> We're building a new Fedora 8.0.1 Linux system to stream data from a
>> 250Msps ADC to disk, and want to start tuning the system configuration
>> for maximum XFS write performance.  To date, without any significant
>> effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
>> with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
>> push the tuning as far as we can go with this architecture before we
>> start looking at other hardware options.  Looking at various other
>> tuning pages on the Web finds few that are interested in maxing out
>> sequential writes to very large arrays while using SAS HW RAID with big
>> fast SAS drives too.
>
> ...
>
>> XFS Tuning Options?
>>
>> - HW RAID0:
>>   - Array/logical disk HW RAID stripe size?
>
> At any rate you'll want to match xfs's geometry with the raid geometry.
>
>>   - Cache enabled (some reports that cache s/b turned off?)?
>
> If it's battery-backed cache, leave it on, and disable barriers in xfs
> (it's a mount option)
>
>>   - xfs mkfs / mount options?
>
> David mentioned these before as a generic place to start:
>
> # mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
> <dev>
> # mount -o logbsize=256k <dev> <mtpt>
>
> and that those would be upcoming new defaults for mkfs.
>
> 4 ags may not be what you want for a ~2T filesystem.
>
>> - External Log?
>>   - How big a partition on the E200i?
>>   - mkfs / mount options?
>
> not sure if an external log would be beneficial or not.
>
> I'm sure others will have more concrete suggestions.  It might be
> interesting to post any performance you're getting which does not meet
> your expectations...
>
> -Eric
>
>>
>> ...?
>>
>> Thanks for any tips,
>>
>
>

I have a question, if he exported all of the HDD as JBOD and then made a 
software raid, took those software raid creation parameters and then 
re-made the HW raid, would they still apply to the HW raid?

Justin.

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[David Chinner]

On Sat, Dec 15, 2007 at 10:35:20PM -0600, Eric Sandeen wrote:
> Alex Madarasz wrote:
> > We're building a new Fedora 8.0.1 Linux system to stream data from a
> > 250Msps ADC to disk, and want to start tuning the system configuration
> > for maximum XFS write performance.  To date, without any significant
> > effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
> > with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
> > push the tuning as far as we can go with this architecture before we
> > start looking at other hardware options.  Looking at various other
> > tuning pages on the Web finds few that are interested in maxing out
> > sequential writes to very large arrays while using SAS HW RAID with big
> > fast SAS drives too.
> 
> ...
> 
> > XFS Tuning Options?
> > 
> > - HW RAID0:
> >   - Array/logical disk HW RAID stripe size?
> 
> At any rate you'll want to match xfs's geometry with the raid geometry.
> 
> >   - Cache enabled (some reports that cache s/b turned off?)?
> 
> If it's battery-backed cache, leave it on, and disable barriers in xfs
> (it's a mount option)
> 
> >   - xfs mkfs / mount options?
> 
> David mentioned these before as a generic place to start:
> 
> # mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
> <dev>
> # mount -o logbsize=256k <dev> <mtpt>
> 
> and that those would be upcoming new defaults for mkfs.
> 
> 4 ags may not be what you want for a ~2T filesystem.

Right - the 4 AG tuning is effectively for single disk configurations to
limit parallelism and therefore keep seeks between AGs down. When you
have multiple disks, the [new] mkfs defaults should be just fine (i.e.
just drop the agcount suggestion).

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Justin Piszcz]

On Mon, 17 Dec 2007, David Chinner wrote:

> On Sat, Dec 15, 2007 at 10:35:20PM -0600, Eric Sandeen wrote:
>> Alex Madarasz wrote:
>>> We're building a new Fedora 8.0.1 Linux system to stream data from a
>>> 250Msps ADC to disk, and want to start tuning the system configuration
>>> for maximum XFS write performance.  To date, without any significant
>>> effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
>>> with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
>>> push the tuning as far as we can go with this architecture before we
>>> start looking at other hardware options.  Looking at various other
>>> tuning pages on the Web finds few that are interested in maxing out
>>> sequential writes to very large arrays while using SAS HW RAID with big
>>> fast SAS drives too.
>>
>> ...
>>
>>> XFS Tuning Options?
>>>
>>> - HW RAID0:
>>>   - Array/logical disk HW RAID stripe size?
>>
>> At any rate you'll want to match xfs's geometry with the raid geometry.
>>
>>>   - Cache enabled (some reports that cache s/b turned off?)?
>>
>> If it's battery-backed cache, leave it on, and disable barriers in xfs
>> (it's a mount option)
>>
>>>   - xfs mkfs / mount options?
>>
>> David mentioned these before as a generic place to start:
>>
>> # mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
>> <dev>
>> # mount -o logbsize=256k <dev> <mtpt>
>>
>> and that those would be upcoming new defaults for mkfs.
>>
>> 4 ags may not be what you want for a ~2T filesystem.
>
> Right - the 4 AG tuning is effectively for single disk configurations to
> limit parallelism and therefore keep seeks between AGs down. When you
> have multiple disks, the [new] mkfs defaults should be just fine (i.e.
> just drop the agcount suggestion).
>
> Cheers,
>
> Dave.
> -- 
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
>
>

Dave,

The mkfs.xfs defaults will be just fine for a HW raid device?

Justin.

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[David Chinner]

On Sun, Dec 16, 2007 at 06:47:26PM -0500, Justin Piszcz wrote:
> 
> The mkfs.xfs defaults will be just fine for a HW raid device?

If you get a mkfs.xfs from CVS. When barry releases an updated xfsprogs
tarball the defaults from there will be as we've been talking about.

You may still need to tweak sunit/swidth for hardware raid (depending
on you config) but that is no different to now....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Changliang Chen]

Hi
  We have use xfs for storage's filesystem which has reach 4T per
node.inour mkfs option,we always use agsize=
4g.Is this any problem, and any suggest?
  By the way,how many agcounts the [new] mkfs defaults make?


2007/12/17, David Chinner <dgc@sgi.com>:
>
> On Sat, Dec 15, 2007 at 10:35:20PM -0600, Eric Sandeen wrote:
> > Alex Madarasz wrote:
> > > We're building a new Fedora 8.0.1 Linux system to stream data from a
> > > 250Msps ADC to disk, and want to start tuning the system configuration
> > > for maximum XFS write performance.  To date, without any significant
> > > effort at tuning our Fedora 7 dev system, we're seeing 250MBps write
> > > with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to
> > > push the tuning as far as we can go with this architecture before we
> > > start looking at other hardware options.  Looking at various other
> > > tuning pages on the Web finds few that are interested in maxing out
> > > sequential writes to very large arrays while using SAS HW RAID with
> big
> > > fast SAS drives too.
> >
> > ...
> >
> > > XFS Tuning Options?
> > >
> > > - HW RAID0:
> > >   - Array/logical disk HW RAID stripe size?
> >
> > At any rate you'll want to match xfs's geometry with the raid geometry.
> >
> > >   - Cache enabled (some reports that cache s/b turned off?)?
> >
> > If it's battery-backed cache, leave it on, and disable barriers in xfs
> > (it's a mount option)
> >
> > >   - xfs mkfs / mount options?
> >
> > David mentioned these before as a generic place to start:
> >
> > # mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4
> > <dev>
> > # mount -o logbsize=256k <dev> <mtpt>
> >
> > and that those would be upcoming new defaults for mkfs.
> >
> > 4 ags may not be what you want for a ~2T filesystem.
>
> Right - the 4 AG tuning is effectively for single disk configurations to
> limit parallelism and therefore keep seeks between AGs down. When you
> have multiple disks, the [new] mkfs defaults should be just fine (i.e.
> just drop the agcount suggestion).
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
>
>
>


-- 
Best Regards


[[HTML alternate version deleted]]

Re: Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?

[Alex Madarasz]

On Sat, 15 Dec 2007 22:35:20 -0600, "Eric Sandeen"
<sandeen@sandeen.net> said:
> Alex Madarasz wrote:
> > We're building a new Fedora 8.0.1 Linux system to stream data from a
> > 250Msps ADC to disk, and want to start tuning the system
> > configuration for maximum XFS write performance.  To date, without
> > any significant effort at tuning our Fedora 7 dev system, we're
> > seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-
> > bit samples. We want to push the tuning as far as we can go with
> > this architecture before we start looking at other hardware options.
> > Looking at various other tuning pages on the Web finds few that are
> > interested in maxing out sequential writes to very large arrays
> > while using SAS HW RAID with big fast SAS drives too.
> ...
>
> > XFS Tuning Options?
> >
> > - HW RAID0:
> >   - Array/logical disk HW RAID stripe size?
>
> At any rate you'll want to match xfs's geometry with the raid
> geometry.

The HP P400 stripe size defaults to 128K and can go up to 256K (and down
to 64K, 32K, ...). Any reason, with big sequential XFS write loads, I 
shouldn't just max it out at 256K and move on?

I understand Linux XFS block size is limited to 4KiB max on Intel 32-bit
architectures ... including PAE? 


> >   - Cache enabled (some reports that cache s/b turned off?)?
>
> If it's battery-backed cache, leave it on, and disable barriers in xfs
> (it's a mount option)

It is indeed 512M of HP Battery-Backed Write Cache (BBWC).


> >   - xfs mkfs / mount options?
>
> David mentioned these before as a generic place to start:
>
> # mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d
> # agcount=4
> <dev>
> # mount -o logbsize=256k <dev> <mtpt>
>
> and that those would be upcoming new defaults for mkfs.
> 
> 4 ags may not be what you want for a ~2T filesystem.


> > - External Log?
> >   - How big a partition on the E200i?
> >   - mkfs / mount options?
>
> not sure if an external log would be beneficial or not.