Re: allocsize mount option

[Gim Leong Chin <chingimleong@yahoo.com.sg>]

Hi,


The application is ANSYS, which writes 128 GB files.  The existing computer with SUSE Linux Enterprise Desktop 11 which is used for running ANSYS, has two software RAID 0 devices made up of five 1 TB drives.  The /home partition is 4.5 T, and it is now 4 TB full.  I see a fragmentation > 19%.


I have just set up a new computer with 16 WD Cavair Black 1 TB drives connected to an Areca 1680ix-16 RAID with 4 GB cache.  14 of these drives are in RAID 6 with 128 kB stripes.  The OS is also SLED 11.  The system has 16 GB memory, and AMD Phenom II X4 965 CPU.

I have done tests writing 100 30 MB files and 1 GB, 10 GB and 20 GB files, with single instance and multiple instances.

There is a big difference in writing speed when writing 20 GB files when using allocsize=1g and not using the option.  That is without the inode64 option, which gives further speed gains.

I use dd for writing the 1 GB, 10 GB and 20 GB files.

mkfs.xfs -f -b size=4k -d agcount=32,su=128k,sw=12 -i size=256,align=1,attr=2 -l version=2,su=128k,lazy-count=1 -n version=2 -s size=512 -L /data /dev/sdb1


defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc

The start of the partition has been set to LBA 3072 using GPT Fdisk to align the stripes.

The dd command is:

chingl@tsunami:/data/test/t2> dd if=/dev/zero of=bigfile20GB bs=1073741824 count=20

Single instance of 20 GB dd repeats were 214, 221, 123 MB/s with allocsize=1g, compared to 94, 126 MB/s without.

Two instances of 20 GB dd repeats were aggregate 331, 372 MB/s with allocsize=1g, compared to 336, 296 MB/s without.

Three instances of 20 GB dd was aggregate 400 MB/s with, 326 MB/s without.

Six instances of 20 GB dd was 606 MB/s with, 473 MB/s without.


My production configuration is

defaults,nobarrier,usrquota,grpquota,noatime,nodiratime,allocsize=1g,logbufs=8,logbsize=256k,largeio,swalloc,inode64

for which I got up to 297 MB/s for single instance 20 GB dd.



Chin Gim Leong


--- On Tue, 12/1/10, Eric Sandeen <sandeen@sandeen.net> wrote:

> From: Eric Sandeen <sandeen@sandeen.net>
> Subject: Re: allocsize mount option
> To: "Gim Leong Chin" <chingimleong@yahoo.com.sg>
> Cc: xfs@oss.sgi.com
> Date: Tuesday, 12 January, 2010, 2:16 AM
> Gim Leong Chin wrote:
> > Hi,
> > 
> > Mount options for xfs allocsize=size Sets  the
> buffered I/O
> > end-of-file preallocation size when doing delayed
> allocation writeout
> > (default size is 64KiB).
> > 
> > 
> > I read that setting allocsize to a big value can be
> used to combat
> > filesystem fragmentation when writing big files.
> 
> That's not universally necessary though, depending on how
> you are
> writing them.  I've only used it in the very specific
> case of mythtv
> calling "sync" every couple seconds, and defeating
> delalloc.
> 
> > I do not understand how allocsize works.  Say I
> set allocsize=1g, but
> > my file size is only 1 MB or even smaller.  Will
> the rest of the 1 GB
> > file extent be allocated, resulting in wasted space
> and even file
> > fragmentation problem?
> 
> possibly :)  It's only speculatively allocated,
> though, so you won't
> have 1g for every file; when it's closed the preallocation
> goes
> away, IIRC.
> 
> > Does setting allocsize to a big value result in
> performance gain when
> > writing big files?  Is performance hurt by a big
> value setting when
> > writing files smaller than the allocsize value?
> > 
> > I am setting up a system for HPC, where two different
> applications
> > have different file size characteristics, one writes
> files of GBs and
> > even 128 GB, the other is in MBs to tens of MBs.
> 
> We should probably back up and say:  are you seeing
> fragmentation
> problems -without- the mount option, and if so, what is
> your write pattern?
> 
> -Eric
> 
> > I am not able to find documentation on the behaviour
> of allocsize
> > mount option.
> > 
> > Thank you.
> > 
> > 
> > Chin Gim Leong
> > 
> > 
> > New Email names for you! Get the Email name you've
> always wanted
> > on the new @ymail and @rocketmail. Hurry before
> someone else does! 
> > http://mail.promotions.yahoo.com/newdomains/sg/
> > 
> > _______________________________________________ xfs
> mailing list 
> > xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs
> > 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>



      New Email addresses available on Yahoo!
Get the Email name you&#39;ve always wanted on the new @ymail and @rocketmail. 
Hurry before someone else does!
http://mail.promotions.yahoo.com/newdomains/sg/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs