Re: Fragmentation Issue We Are Having

Dave Chinner
2012-04-19T23:13:22Z

On Wed, Apr 18, 2012 at 10:00:51AM +0100, Brian Candler wrote:
> On Wed, Apr 18, 2012 at 11:36:07AM +1000, Dave Chinner wrote:
> > And it assumes that inode32 cannot do locality of
> > files at all, when in fact it has tunable locality through a syctl.
> >
> > Indeed, here's some the performance enhancing games SGI play that
> > can only be achieved by using the inode32 allocator:
> >
> > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=linux&db=bks&fname=/SGI_Admin/LX_XFS_AG/ch07.html
>
> Ah, that's new to me. So with inode32 and
> sysctl fs.xfs.rotorstep=255
> you can get roughly the same locality benefit for sequentially-written files
> as inode64? (Aside: if you have two processes writing files to two different
> directories, will they end up mixing their files in the same AG? That
> could hurt performance at readback time if reading them sequentially)

As Richard pointed out, the filestreams allocator was designed
specifically to avoid this problem (mainly for file-per-frame
realtima, concurrent e ingest/playout of multiple 2k and 4k HD
uncompressed video streams on a single SAN.

> I'm not really complaining about anything here except the dearth of
> readily-accessible information. If I download the whole admin book:
> http://techpubs.sgi.com/library/manuals/4000/007-4273-004/pdf/007-4273-004.pdf
> I see the option inode64 only mentioned once in passing (as being
> incompatible with ibound). So if there's detailled information on
> what exactly inode64 does and when to use it, it must be somewhere else.

It's scattered around the place. The man pages, the FAQ, kernel
documentation, techpubs, user documentation, in the heads of
developers and experienced admins, etc. There's no one
real definitive source of the information you were looking for...

> Here's my user story. As a newbie, my first test was to make a 3TB
> filesystem on a single drive, and I was doing a simple workload consisting
> of writing 1000 files per directory sequentially. I could achieve a
> sequential write speed of 75MB/s but only a sequential read speed of 25MB/s.
> After questioning this on the list and eventually finding that files were
> scattered around the disk (thanks to xfs_bmap). I was pointed to the
> inode64 option, which I had seen in the FAQ but hadn't realised how big a
> performance difference it would make.
>
> This wasn't just an idle benchmark: my main application is creating a corpus
> of files and then processing that corpus of files (either sequentially, or
> with multiple processes each working sequentially through subsections of the
> corpus)

What you are learning is the reason why we say "use the defaults".
The process oflearning what all the different options do is
difficult because it requires understanding how the internals of XFS
work. i.e. You need to understand what your application does (which
you obviously do) and what the filesystem tunables do before really
being able to understand why it helps or hinders.

Given all the issues you had finding out this information, the most
valuable thing you could do right now is update the FAQ on the wiki
to correct the obvious problems, and maybe even add a new "what are
the differences between inode32 and inode64?" page to the wiki to
summarise what we've talked about in this thread. That will make it
much easier for others in future to find the same information....

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com