Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Martin Knoblauch]

----- Original Message ----

> From: Dave Chinner <david@fromorbit.com>
> To: Szabolcs Szakacsits <szaka@ntfs-3g.org>; Andrew Morton <akpm@linux-foundation.org>; linux-fsdevel@vger.kernel.org; linux-kernel@vger.kernel.org; xfs@oss.sgi.com
> Sent: Thursday, August 21, 2008 10:25:32 AM
> Subject: Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)
> 
> On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 03:15:08PM +1000, Dave Chinner wrote:
> > > On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits wrote:
> > > > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > > Everything is default.
> > > > 
> > > >   % rpm -qf =mkfs.xfs
> > > >   xfsprogs-2.9.8-7.1 
> > > > 
> > > > which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is the 
> > > > latest stable mkfs.xfs. Its output is
> > > > 
> > > > meta-data=/dev/sda8              isize=256    agcount=4, agsize=1221440 
> blks
> > > >          =                       sectsz=512   attr=2
> > > > data     =                       bsize=4096   blocks=4885760, imaxpct=25
> > > >          =                       sunit=0      swidth=0 blks
> > > > naming   =version 2              bsize=4096  
> > > > log      =internal log           bsize=4096   blocks=2560, version=2
> > > >          =                       sectsz=512   sunit=0 blks, lazy-count=0
> > > > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > > 
> > > Ok, I thought it might be the tiny log, but it didn't improve anything
> > > here when increased the log size, or the log buffer size.
> > 
> > One thing I just found out - my old *laptop* is 4-5x faster than the
> > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > if the long delays in dispatch is caused by an interaction with CTQ
> > but I can't change it on the cciss raid controllers. Are you using
> > ctq/ncq on your machine?  If so, can you reduce the depth to
> > something less than 4 and see what difference that makes?
> 
> Just to point out - this is not a new problem - I can reproduce
> it on 2.6.24 as well as 2.6.26. Likewise, my laptop shows XFS
> being faster than ext3 on both 2.6.24 and 2.6.26. So the difference
> is something related to the disk subsystem on the server....
> 
Hi Dave,

 just curious - which CCISS controller and and what kind of disk configuration are you using.

Cheers
Martin

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 04:05:21AM -0700, Martin Knoblauch wrote:
>  just curious - which CCISS controller and and what kind of disk configuration are you using.

Smart Array 5300, using a single 36GB 10krpm U3SCSI disk.

Looks like most of the problems were caused by the firmware,
and are not driver related. An upgrade to a newer (2002!) firmware
appears to have fixed the worst of the problems.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits wrote:
> On Thu, 21 Aug 2008, Dave Chinner wrote:
> > On Wed, Aug 20, 2008 at 02:39:16PM -0700, Andrew Morton wrote:
> > > On Thu, 21 Aug 2008 00:25:55 +0300 (MET DST)
> > > Szabolcs Szakacsits <szaka@ntfs-3g.org> wrote:
> > > > I ran compilebench on kernel 2.6.26 with freshly formatted volumes. 
> > > > The behavior of NILFS2 was interesting.
> > > > 
> > > > Its peformance rapidly degrades to the lowest ever measured level 
> > > > (< 1 MB/s) but after a while it recovers and gives consistent numbers.
> > > > However it's still very far from the current unstable btrfs performance. 
> > > > The results are reproducible.
> > > > 
> > > >                     MB/s    Runtime (s)
> > > >                    -----    -----------
> > > >   btrfs unstable   17.09        572
> > > >   ext3             13.24        877
> > > >   btrfs 0.16       12.33        793
> > > >   nilfs2 2nd+ runs 11.29        674
> > > >   ntfs-3g           8.55        865
> > > >   reiserfs          8.38        966
> > > >   nilfs2 1st run    4.95       3800
> > > >   xfs               1.88       3901
> > > 
> > > err, what the heck happened to xfs?  Is this usual?
> > 
> > No, definitely not usual. I suspect it's from an old mkfs and
> > barriers being used.  What is the output of the xfs.mkfs when
> > you make the filesystem and what mount options being used?
> 
> Everything is default.
> 
>   % rpm -qf =mkfs.xfs
>   xfsprogs-2.9.8-7.1 
> 
> which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is the 
> latest stable mkfs.xfs. Its output is
> 
> meta-data=/dev/sda8              isize=256    agcount=4, agsize=1221440 blks
>          =                       sectsz=512   attr=2
> data     =                       bsize=4096   blocks=4885760, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096  
> log      =internal log           bsize=4096   blocks=2560, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=0
> realtime =none                   extsz=4096   blocks=0, rtextents=0

Ok, I thought it might be the tiny log, but it didn't improve anything
here when increased the log size, or the log buffer size.

Looking at the block trace, I think elevator merging is somewhat busted. I'm
seeing adjacent I/Os being dispatched without having been merged.  e.g:

104,48   1     2139     4.803090086  4175  Q   W 18540712 + 8 [pdflush]
104,48   1     2140     4.803092492  4175  G   W 18540712 + 8 [pdflush]
104,48   1     2141     4.803094875  4175  P   N [pdflush]
104,48   1     2142     4.803096205  4175  I   W 18540712 + 8 [pdflush]
104,48   1     2143     4.803160324  4175  Q   W 18540720 + 40 [pdflush]
104,48   1     2144     4.803162724  4175  M   W 18540720 + 40 [pdflush]
104,48   1     2145     4.803231701  4175  Q   W 18540760 + 48 [pdflush]
104,48   1     2146     4.803234223  4175  M   W 18540760 + 48 [pdflush]
.....
104,48   1     2163     4.803844214  4175  Q   W 18541032 + 56 [pdflush]
104,48   1     2164     4.803846694  4175  M   W 18541032 + 56 [pdflush]
104,48   1     2165     4.803932321  4175  Q   W 18541088 + 48 [pdflush]
104,48   1     2166     4.803937177  4175  G   W 18541088 + 48 [pdflush]
104,48   1     2167     4.803940416  4175  I   W 18541088 + 48 [pdflush]
104,48   1     2168     4.804005265  4175  Q   W 18541136 + 24 [pdflush]
104,48   1     2169     4.804007664  4175  M   W 18541136 + 24 [pdflush]
.....
104,48   1     2183     4.804518129  4175  D   W 18540712 + 376 [pdflush]
104,48   1     2184     4.804537981  4175  D   W 18541088 + 248 [pdflush]

In entry 2165, a new request is made rather than merging the
existing, adjacent request that is already open. The result is we
then dispatch two I/Os instead of one.

Also, CFQ appears to not be merging WRITE_SYNC bios or issuing them
with any urgency.  The result of this is that it stalls the XFS
transaction subsystem by capturing all the log buffers in the
elevator and not issuing them. e.g.:

104,48   0      149     0.107856547  4160  Q  WS 35624860 + 128 [pdflush]
104,48   0      150     0.107861855  4160  G  WS 35624860 + 128 [pdflush]
104,48   0      151     0.107865332  4160  I   W 35624860 + 128 [pdflush]
...
104,48   0      162     0.120791581  4159  Q  WS 35624988 + 128 [python]
104,48   0      163     0.120805714  4159  G  WS 35624988 + 128 [python]
104,48   0      164     0.120813427  4159  I   W 35624988 + 128 [python]
104,48   0      165     0.132109889  4159  Q  WS 35625116 + 128 [python]
104,48   0      166     0.132128642  4159  G  WS 35625116 + 128 [python]
104,48   0      167     0.132132988  4159  I   W 35625116 + 128 [python]
104,48   0      168     0.143612843  4159  Q  WS 35625244 + 128 [python]
104,48   0      169     0.143640248  4159  G  WS 35625244 + 128 [python]
104,48   0      170     0.143644697  4159  I   W 35625244 + 128 [python]
104,48   0      171     0.158243553  4159  Q  WS 35625372 + 128 [python]
104,48   0      172     0.158261652  4159  G  WS 35625372 + 128 [python]
104,48   0      173     0.158266233  4159  I   W 35625372 + 128 [python]
104,48   0      174     0.171342555  4159  Q  WS 35625500 + 128 [python]
104,48   0      175     0.171360707  4159  G  WS 35625500 + 128 [python]
104,48   0      176     0.171365036  4159  I   W 35625500 + 128 [python]
104,48   0      177     0.183936429  4159  Q  WS 35625628 + 128 [python]
104,48   0      178     0.183955172  4159  G  WS 35625628 + 128 [python]
104,48   0      179     0.183959726  4159  I   W 35625628 + 128 [python]
...
104,48   0      180     0.194008953  4159  Q  WS 35625756 + 128 [python]
104,48   0      181     0.194027120  4159  G  WS 35625756 + 128 [python]
104,48   0      182     0.194031311  4159  I   W 35625756 + 128 [python]
...
104,48   0      191     0.699915104     0  D   W 35624860 + 128 [swapper]
...
104,48   0      196     0.700513279     0  C   W 35624860 + 128 [0]
...
104,48   0      198     0.711808579  4159  Q  WS 35625884 + 128 [python]
104,48   0      199     0.711826259  4159  G  WS 35625884 + 128 [python]
104,48   0      200     0.711830589  4159  I   W 35625884 + 128 [python]
104,48   0      201     0.711848493  4159  D   W 35624988 + 128 [python]
104,48   0      202     0.711861868  4159  D   W 35625116 + 128 [python]\x02
104,48   0      203     0.711868531  4159  D   W 35625244 + 128 [python]
104,48   0      204     0.711874967  4159  D   W 35625372 + 128 [python]
....
104,48   1       72     0.900288147     0  D   W 35625500 + 128 [swapper]
104,48   1       73     0.900296058     0  D   W 35625628 + 128 [swapper]
104,48   1       74     0.900302401     0  D   W 35625756 + 128 [swapper]
104,48   1       75     0.900308516     0  D   W 35625884 + 128 [swapper]
.....

here we see all 8 log buffers written and queued in ~95ms. At this point
(0.194s into the trace) the log stalls because we've used all the log
buffers and have to wait for I/O to complete. The filesystem effectively
sits idle now for half a second waiting for I/O to be dispatched.

At 0.699s, we have a single buffer issued and it completes in 500
*microseconds* (NVRAM on raid controller). We do completion
processing, fill and dispatch that buffer in under 10ms (on a 1GHz
P3) at which point we dispatch the 4 oldest remaining buffers. 200ms
later, we dispatch the remainder.

Effectively, the elevator has stalled all transactions in the
filesystem for close to 700ms by not dispatching the SYNC_WRITE
buffers, and all the bios could have been merged into a single 512k
I/O when they were to be dispatched. I guess the only way to prevent
this really is to issue explicit unplugs....

On 2.6.24:

104,48   0      975     1.707253442  2761  Q  WS 35753545 + 128 [python]
104,48   0      976     1.707268811  2761  G  WS 35753545 + 128 [python]
104,48   0      977     1.707275455  2761  I   W 35753545 + 128 [python]
104,48   0      978     1.728703316  2761  Q  WS 35753673 + 128 [python]
104,48   0      979     1.728714289  2761  M  WS 35753673 + 128 [python]
104,48   0      980     1.761603632  2761  Q  WS 35753801 + 128 [python]
104,48   0      981     1.761614498  2761  M  WS 35753801 + 128 [python]
104,48   0      982     1.784522988  2761  Q  WS 35753929 + 128 [python]
104,48   0      983     1.784533351  2761  M  WS 35753929 + 128 [python]
....
104,48   0     1125     2.475132431     0  D   W 35753545 + 512 [swapper]

The I/Os are merged, but there's still that 700ms delay before dispatch.
i was looking at this a while back but didn't get to finishing it off.
i.e.:

http://oss.sgi.com/archives/xfs/2008-01/msg00151.html
http://oss.sgi.com/archives/xfs/2008-01/msg00152.html

I'll have a bit more of a look at this w.r.t to compilebench performance,
because it seems like a similar set of problems that I was seeing back
then...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[gus3]

--- On Wed, 8/20/08, Dave Chinner <david@fromorbit.com> wrote:

> Ok, I thought it might be the tiny log, but it didn't
> improve anything
> here when increased the log size, or the log buffer size.
> 
> Looking at the block trace, I think elevator merging is
> somewhat busted. I'm
> seeing adjacent I/Os being dispatched without having been
> merged.  e.g:

[snip]

> Also, CFQ appears to not be merging WRITE_SYNC bios or
> issuing them
> with any urgency.  The result of this is that it stalls the
> XFS
> transaction subsystem by capturing all the log buffers in
> the
> elevator and not issuing them. e.g.:

[snip]

> The I/Os are merged, but there's still that 700ms delay
> before dispatch.
> i was looking at this a while back but didn't get to
> finishing it off.
> i.e.:
> 
> http://oss.sgi.com/archives/xfs/2008-01/msg00151.html
> http://oss.sgi.com/archives/xfs/2008-01/msg00152.html
> 
> I'll have a bit more of a look at this w.r.t to
> compilebench performance,
> because it seems like a similar set of problems that I was
> seeing back
> then...

I concur your observation, esp. w.r.t. XFS and CFQ clashing:

http://gus3.typepad.com/i_am_therefore_i_think/2008/07/finding-the-fas.html

CFQ is the default on most Linux systems AFAIK; for decent XFS performance one needs to switch to "noop" or "deadline". I wasn't sure if it was broken code, or simply base assumptions in conflict (XFS vs. CFQ). Your log output sheds light on the matter for me, thanks.

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Wed, Aug 20, 2008 at 11:00:07PM -0700, gus3 wrote:
> --- On Wed, 8/20/08, Dave Chinner <david@fromorbit.com> wrote:
> 
> > Ok, I thought it might be the tiny log, but it didn't improve
> > anything here when increased the log size, or the log buffer
> > size.
> > 
> > Looking at the block trace, I think elevator merging is somewhat
> > busted. I'm seeing adjacent I/Os being dispatched without having
> > been merged.  e.g:
> 
> [snip]
> 
> > Also, CFQ appears to not be merging WRITE_SYNC bios or issuing
> > them with any urgency.  The result of this is that it stalls the
> > XFS transaction subsystem by capturing all the log buffers in
> > the elevator and not issuing them. e.g.:
> 
> [snip]
> 
> > The I/Os are merged, but there's still that 700ms delay before
> > dispatch.  i was looking at this a while back but didn't get to
> > finishing it off.  i.e.:
> > 
> > http://oss.sgi.com/archives/xfs/2008-01/msg00151.html
> > http://oss.sgi.com/archives/xfs/2008-01/msg00152.html
> > 
> > I'll have a bit more of a look at this w.r.t to compilebench
> > performance, because it seems like a similar set of problems
> > that I was seeing back then...
> 
> I concur your observation, esp. w.r.t. XFS and CFQ clashing:
> 
> http://gus3.typepad.com/i_am_therefore_i_think/2008/07/finding-the-fas.html
> 
> CFQ is the default on most Linux systems AFAIK; for decent XFS
> performance one needs to switch to "noop" or "deadline". I wasn't
> sure if it was broken code, or simply base assumptions in conflict
> (XFS vs. CFQ). Your log output sheds light on the matter for me,
> thanks.

I'm wondering if these elevators are just getting too smart for
their own good. w.r.t to the above test, deadline was about twice
as slow as CFQ - it does immediate dispatch on SYNC_WRITE bios and
so caused more seeks that CFQ and hence went slower. noop had
similar dispatch latency problems to CFQ, so it wasn't any
faster either.

I think that we need to issue explicit unplugs to get the log I/O
dispatched the way we want on all elevators and stop trying to
give elevators implicit hints by abusing the bio types and hoping
they do the right thing....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 05:00:39PM +1000, Nick Piggin wrote:
> On Thursday 21 August 2008 16:14, Dave Chinner wrote:
> 
> > I think that we need to issue explicit unplugs to get the log I/O
> > dispatched the way we want on all elevators and stop trying to
> > give elevators implicit hints by abusing the bio types and hoping
> > they do the right thing....
> 
> FWIW, my explicit plugging idea is still hanging around in one of
> Jens' block trees (actually he refreshed it a couple of months ago).
> 
> It provides an API for VM or filesystems to plug and unplug
> requests coming out of the current process, and it can reduce the
> need to idle the queue. Needs more performance analysis and tuning
> though.

We've already got plenty of explicit unplugs in XFS to get stuff
moving quickly - I'll just have to add another....

> But existing plugging is below the level of the elevators, and should
> only kick in for at most tens of ms at queue idle events, so it sounds
> like it may not be your problem. Elevators will need some hint to give
> priority to specific requests -- either via the current threads's io
> priority, or information attached to bios.

It's getting too bloody complex, IMO. What is right for one elevator
is wrong for another, so as a filesystem developer I have to pick
one to target. With the way the elevators have been regressing,
improving and changing behaviour, I am starting to think that I
should be picking the noop scheduler. Any 'advanced' scheduler that
is slower than the same test on the noop scheduler needs fixing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Chris Mason]

On Thu, 2008-08-21 at 18:53 +1000, Dave Chinner wrote:
> On Thu, Aug 21, 2008 at 05:00:39PM +1000, Nick Piggin wrote:
> > On Thursday 21 August 2008 16:14, Dave Chinner wrote:
> > 
> > > I think that we need to issue explicit unplugs to get the log I/O
> > > dispatched the way we want on all elevators and stop trying to
> > > give elevators implicit hints by abusing the bio types and hoping
> > > they do the right thing....
> > 
> > FWIW, my explicit plugging idea is still hanging around in one of
> > Jens' block trees (actually he refreshed it a couple of months ago).
> > 
> > It provides an API for VM or filesystems to plug and unplug
> > requests coming out of the current process, and it can reduce the
> > need to idle the queue. Needs more performance analysis and tuning
> > though.
> 
> We've already got plenty of explicit unplugs in XFS to get stuff
> moving quickly - I'll just have to add another....
> 

I did some compilebench runs with xfs this morning, creating 30 kernel
trees on the same machine I posted btrfs and xfs numbers with last week.
Btrfs gets between 60 and 75MB/s average depending on the mount options
used, ext4 gets around 60MB/s

This is a single sata drive that can run at 100MB/s streaming writes.
The numbers show XFS is largely log bound, and that turning off barriers
makes a huge difference.  I'd be happy to try another run with explicit
unplugging somewhere in the transaction commit path.

I think the most relevant number is the count of MB written at the end
of blkparse. I'm not sure why the 4ag XFS writes less, but the numbers
do include calling sync at the end.  None of the filesystems were doing
barriers in these numbers:

Ext4                                9036MiB
Btrfs metadata dup                  9190MiB
Btrfs metadata dup no inline files 10280MiB
XFS 4ag, nobarrier                 14299MiB
XFS 1ag, nobarrier                 17836MiB

This is a long way of saying the xfs log isn't optimal for these kinds
of operations, which isn't really news.  I'm not ripping on xfs here,
this is just one tiny benchmark.

I uploaded some graphs of the IO here:

http://oss.oracle.com/~mason/seekwatcher/compilebench-30/xfs


XFS:

*** 4ag, 128m log, logbsize=256k
intial create total runs 30 avg 7.48 MB/s (user 0.52s sys 1.04s)

*** 4ag, 128m log, logbsize=256k, nobarrier
intial create total runs 30 avg 21.58 MB/s (user 0.51s sys 1.04s)
http://oss.oracle.com/~mason/seekwatcher/compilebench-30/xfs/xfs-4ag-nobarrier.png

*** 1ag, 128m log, logbsize=256k, nobarrier
intial create total runs 30 avg 26.28 MB/s (user 0.50s sys 1.15s)
http://oss.oracle.com/~mason/seekwatcher/compilebench-30/xfs/xfs-nobarrier-1ag.png

It is hard to see in the graph, but it looks like the log is in the
first 128MB of the drive.  If we give XFS an external log device:

*** 1ag 128m external log, logbsize=256k, nobarrier
intial create total runs 30 avg 38.44 MB/s (user 0.51s sys 1.09s)

This graph shows that log is running more or less seek free between
30-60MB/s for the whole run.  I'd expect the explicit unplugging to help
the most in this config?

http://oss.oracle.com/~mason/seekwatcher/compilebench-30/xfs/xfs-external-log-disk.png

Here is the main disk during the run:
http://oss.oracle.com/~mason/seekwatcher/compilebench-30/xfs/xfs-external-log-main-disk.png


*** 1ag 128m external log, logbsize=256k, nobarrier, deadline
intial create total runs 30 avg 34.00 MB/s (user 0.51s sys 1.07s)

Deadline didn't help on this box.

-chris

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 07:33:34PM +1000, Nick Piggin wrote:
> > > But existing plugging is below the level of the elevators, and should
> > > only kick in for at most tens of ms at queue idle events, so it sounds
> > > like it may not be your problem. Elevators will need some hint to give
> > > priority to specific requests -- either via the current threads's io
> > > priority, or information attached to bios.
> >
> > It's getting too bloody complex, IMO. What is right for one elevator
> > is wrong for another, so as a filesystem developer I have to pick
> > one to target.
> 
> I don't really see it as too complex. If you know how you want the
> request to be handled, then it should be possible to implement.

That is the problem in a nutshell. Nobody can keep up with all
the shiny new stuff that is being implemented,let alone the
subtle behavioural differences that accumulate through such
change...

> > With the way the elevators have been regressing, 
> > improving and changing behaviour,
> 
> AFAIK deadline, AS, and noop haven't significantly changed for years.

Yet they've regularly shown performance regressions because other
stuff has been changing around them, right?

> > I am starting to think that I 
> > should be picking the noop scheduler.
> > Any 'advanced' scheduler that 
> > is slower than the same test on the noop scheduler needs fixing...
> 
> I disagree. On devices with no seek penalty or their own queueing,
> noop is often the best choice. Same for specialized apps that do
> their own disk scheduling.

A filesystem is nothing but a complex disk scheduler that
has to handle vastly larger queues than an elevator. Іf the
filesystem doesn't get it's disk scheduling right, then the
elevator is irrelevant because nothing will fix the I/O
problems in the filesystem algorithms.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Fri, Aug 22, 2008 at 12:29:10PM +1000, Nick Piggin wrote:
> On Friday 22 August 2008 03:08, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 07:33:34PM +1000, Nick Piggin wrote:
> 
> > > I don't really see it as too complex. If you know how you want the
> > > request to be handled, then it should be possible to implement.
> >
> > That is the problem in a nutshell. Nobody can keep up with all
> > the shiny new stuff that is being implemented,let alone the
> > subtle behavioural differences that accumulate through such
> > change...
> 
> I'm not sure exactly what you mean.. I certainly have not been keeping
> up with all the changes here as I'm spending most of my time on other
> things lately...
> 
> But from what I see, you've got a fairly good handle on analysing the
> elevator behaviour (if only the end result).

Only from having to do this analysis over and over again trying to
understand what has changed in the elevator that has negated the
effect of some previous optimisation....

> So if you were to tell
> Jens that "these blocks" need more priority, or not to contribute to
> a process's usage quota, etc. then I'm sure improvements could be
> made.

It's exactly this sort of complexity that is the problem. When the
behaviour of such things change, filesystems that are optimised for
the previous behaviour are not updated - we're not even aware that
the elevator has been changed in some subtle manner that breaks
the optimisations that have been done.

To keep on top of this, we keep adding new variations and types and
expect the filesystems to make best use of them (without
documentation) to optimise for certain situations. Example - the
new(ish) BIO_META tag that only CFQ understands. I can change the
way XFS issues bios to use this tag to make CFQ behave the same way
it used to w.r.t. metadata I/O from XFS, but then the deadline and
AS will probably regress because they don't understand that tag and
still need the old optimisations that just got removed. Ditto for
prioritised bio dispatch - CFQ supports it but none of the others
do.

IOWs, I am left with a choice - optimise for a specific elevator
(CFQ) to the detriment of all others (noop, as, deadline), or make
the filesystem work best with the simple elevator (noop) and
consider the smarter schedulers deficient if they are slower than
the noop elevator....

> Or am I completely misunderstanding you? :)

You're suggesting that I add complexity to solve the too much complexity
problem.... ;)

> > > > With the way the elevators have been regressing,
> > > > improving and changing behaviour,
> > >
> > > AFAIK deadline, AS, and noop haven't significantly changed for years.
> >
> > Yet they've regularly shown performance regressions because other
> > stuff has been changing around them, right?
> 
> Is this rhetorical? Because I don't see how *they* could be showing
> regular performance regressions.

I get private email fairly often asking questions as to why XFS is
slower going from, say, 2.6.23 to 2.6.24 and then speeds back up in
2.6.25. I seen a number of cases where the answer to this was that
elevator 'x' with XFS in 2.6.x because for some reason it is much,
much slower than the others on that workload on that hardware.

As seen earlier in this thread, this can be caused by a problem with
the hardware, firmware, configuration, driver bugs, etc - there are
so many combinations of variables that can cause performance issues
that often the only 'macro' level change that you can make to avoid
them is to switch schedulers. IOWs, while a specific scheduler has
not changed, the code around it has changed sufficiently for a
specific elevator to show a regression compared to the otherr
elevators.....

Basically, the complexity of the interactions between the
filesystems, elevators and the storage devices is such that there
are transient second order effects occurring that are not reported
widely because they are easily worked around by switching elevators.

> Deadline literally had its last
> behaviour change nearly a year ago, and before that was before
> recorded (git) history.
> 
> AS hasn't changed much more frequently, although I will grant that it
> and CFS add a lot more complexity. So I would always compare results
> with deadline or noop.

Which can still change by things like changing merging behaviour.
Granted, it is less complex, but still we can have subtle changes
having major impact in less commonly run workloads...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Jamie Lokier]

Dave Chinner wrote:
> To keep on top of this, we keep adding new variations and types and
> expect the filesystems to make best use of them (without
> documentation) to optimise for certain situations. Example - the
> new(ish) BIO_META tag that only CFQ understands. I can change the
> way XFS issues bios to use this tag to make CFQ behave the same way
> it used to w.r.t. metadata I/O from XFS, but then the deadline and
> AS will probably regress because they don't understand that tag and
> still need the old optimisations that just got removed. Ditto for
> prioritised bio dispatch - CFQ supports it but none of the others
> do.


There's nothing wrong with adding BIO_META (for example) and other
hints in _principle_.  You should be able to ignore it with no adverse
effects.  If its not used by a filesystem (and there's nothing else
competing to use the same disk), I would hope to see the same
performance as other kernels which don't have it.

If the elevators are being changed in such a way that old filesystem
code which doesn't use new hint bits is running significantly slower,
surely that's blatant elevator regression, and that's where the bugs
should be reported and fixed?

-- Jamie

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Mon, Aug 25, 2008 at 01:01:47PM +0100, Jamie Lokier wrote:
> Dave Chinner wrote:
> > To keep on top of this, we keep adding new variations and types and
> > expect the filesystems to make best use of them (without
> > documentation) to optimise for certain situations. Example - the
> > new(ish) BIO_META tag that only CFQ understands. I can change the
> > way XFS issues bios to use this tag to make CFQ behave the same way
> > it used to w.r.t. metadata I/O from XFS, but then the deadline and
> > AS will probably regress because they don't understand that tag and
> > still need the old optimisations that just got removed. Ditto for
> > prioritised bio dispatch - CFQ supports it but none of the others
> > do.
> 
> There's nothing wrong with adding BIO_META (for example) and other
> hints in _principle_.  You should be able to ignore it with no adverse
> effects.  If its not used by a filesystem (and there's nothing else
> competing to use the same disk), I would hope to see the same
> performance as other kernels which don't have it.

Right, but it's what we need to do to make use of that optimisation
that is the problem. For XFS, it needs to replace the current
BIO_SYNC hints we use (even for async I/O) to get metadata
dispatched quickly. i.e. CFQ looks at the sync flag first then the
meta flag.  Hence to take advantage of it, we need to remove the
BIO_SYNC hints we currently use which will change the behaviour on
all other elevators as a side effect.

This is the optimisation problem I'm refering to - the BIO_SYNC
usage was done years ago to get metadata dispatched quickly because
that is what all the elevators did with sync I/O. Now to optimise
for CFQ we need to remove that BIO_SYNC optimisation which is still
valid for the other elevators....

> If the elevators are being changed in such a way that old filesystem
> code which doesn't use new hint bits is running significantly slower,
> surely that's blatant elevator regression, and that's where the bugs
> should be reported and fixed?

Sure, but in reality getting ppl to go through the pain of triage is
extremely rare because it only takes 10s to change elevators and 
make the problem go away...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[david]

On Tue, 26 Aug 2008, Dave Chinner wrote:

> 
> On Mon, Aug 25, 2008 at 01:01:47PM +0100, Jamie Lokier wrote:
>> Dave Chinner wrote:
>>> To keep on top of this, we keep adding new variations and types and
>>> expect the filesystems to make best use of them (without
>>> documentation) to optimise for certain situations. Example - the
>>> new(ish) BIO_META tag that only CFQ understands. I can change the
>>> way XFS issues bios to use this tag to make CFQ behave the same way
>>> it used to w.r.t. metadata I/O from XFS, but then the deadline and
>>> AS will probably regress because they don't understand that tag and
>>> still need the old optimisations that just got removed. Ditto for
>>> prioritised bio dispatch - CFQ supports it but none of the others
>>> do.
>>
>> There's nothing wrong with adding BIO_META (for example) and other
>> hints in _principle_.  You should be able to ignore it with no adverse
>> effects.  If its not used by a filesystem (and there's nothing else
>> competing to use the same disk), I would hope to see the same
>> performance as other kernels which don't have it.
>
> Right, but it's what we need to do to make use of that optimisation
> that is the problem. For XFS, it needs to replace the current
> BIO_SYNC hints we use (even for async I/O) to get metadata
> dispatched quickly. i.e. CFQ looks at the sync flag first then the
> meta flag.  Hence to take advantage of it, we need to remove the
> BIO_SYNC hints we currently use which will change the behaviour on
> all other elevators as a side effect.
>
> This is the optimisation problem I'm refering to - the BIO_SYNC
> usage was done years ago to get metadata dispatched quickly because
> that is what all the elevators did with sync I/O. Now to optimise
> for CFQ we need to remove that BIO_SYNC optimisation which is still
> valid for the other elevators....
>
>> If the elevators are being changed in such a way that old filesystem
>> code which doesn't use new hint bits is running significantly slower,
>> surely that's blatant elevator regression, and that's where the bugs
>> should be reported and fixed?
>
> Sure, but in reality getting ppl to go through the pain of triage is
> extremely rare because it only takes 10s to change elevators and
> make the problem go away...

it sounds as if the various flag definitions have been evolving, would it 
be worthwhile to sep back and try to get the various filesystem folks to 
brainstorm together on what types of hints they would _like_ to see 
supported?

it sounds like you are using 'sync' for things where you really should be 
saying 'metadata' (or 'journal contents'), it's happened to work well 
enough in the past, but it's forcing you to keep tweaking the filesystems. 
it may be better to try and define things from the filesystem point of 
view and let the elevators do the tweaking.

basicly I'm proposing a complete rethink of the filesyste <-> elevator 
interface.

David Lang

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
> it sounds as if the various flag definitions have been evolving, would it 
> be worthwhile to sep back and try to get the various filesystem folks to  
> brainstorm together on what types of hints they would _like_ to see  
> supported?

Three types:

	1. immediate dispatch - merge first with adjacent requests
	   then dispatch
	2. delayed dispatch - queue for a short while to allow
	   merging of requests from above
	3. bulk data - queue and merge. dispatch is completely
	   controlled by the elevator

Basically most metadata and log writes would fall into category 2,
which every logbufs/2 log writes or every log force using a category
1 to prevent log I/O from being stalled too long by other I/O.

Data writes from the filesystem would appear as category 3 (read and write)
and are subject to the specific elevator scheduling. That is, things
like the CFQ ionice throttling would work on the bulk data queue,
but not the other queues that the filesystem is using for metadata.

Tagging the I/O as a sync I/O can still be done, but that only
affects category 3 scheduling - category 1 or 2 would do the same
thing whether sync or async....

> it sounds like you are using 'sync' for things where you really should be 
> saying 'metadata' (or 'journal contents'), it's happened to work well  
> enough in the past, but it's forcing you to keep tweaking the 
> filesystems.

Right, because there was no 'metadata' tagging, and 'sync' happened
to do exactly what we needed on all elevators at the time.

> it may be better to try and define things from the 
> filesystem point of view and let the elevators do the tweaking.
>
> basicly I'm proposing a complete rethink of the filesyste <-> elevator  
> interface.

Yeah, I've been saying that for a while w.r.t. the filesystem/block
layer interfaces, esp. now with discard requests, data integrity,
device alignment information, barriers, etc being exposed by the
layers below the filesystem, but with no interface for filesystems
to be able to access that information...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[david]

On Wed, 27 Aug 2008, Dave Chinner wrote:

> On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
>> it sounds as if the various flag definitions have been evolving, would it
>> be worthwhile to sep back and try to get the various filesystem folks to
>> brainstorm together on what types of hints they would _like_ to see
>> supported?
>
> Three types:
>
> 	1. immediate dispatch - merge first with adjacent requests
> 	   then dispatch
> 	2. delayed dispatch - queue for a short while to allow
> 	   merging of requests from above
> 	3. bulk data - queue and merge. dispatch is completely
> 	   controlled by the elevator

does this list change if you consider the fact that there may be a raid 
array or some more complex structure for the block device instead of a 
simple single disk partition?

since I am suggesting re-thinking the filesystem <-> elevator interface, 
is there anything you need to have the elevator tell the filesystem? (I'm 
thinking that this may be the path for the filesystem to learn things 
about the block device that's under it, is it a raid array, a solid-state 
drive, etc)

David Lang

> Basically most metadata and log writes would fall into category 2,
> which every logbufs/2 log writes or every log force using a category
> 1 to prevent log I/O from being stalled too long by other I/O.
>
> Data writes from the filesystem would appear as category 3 (read and write)
> and are subject to the specific elevator scheduling. That is, things
> like the CFQ ionice throttling would work on the bulk data queue,
> but not the other queues that the filesystem is using for metadata.
>
> Tagging the I/O as a sync I/O can still be done, but that only
> affects category 3 scheduling - category 1 or 2 would do the same
> thing whether sync or async....
>
>> it sounds like you are using 'sync' for things where you really should be
>> saying 'metadata' (or 'journal contents'), it's happened to work well
>> enough in the past, but it's forcing you to keep tweaking the
>> filesystems.
>
> Right, because there was no 'metadata' tagging, and 'sync' happened
> to do exactly what we needed on all elevators at the time.
>
>> it may be better to try and define things from the
>> filesystem point of view and let the elevators do the tweaking.
>>
>> basicly I'm proposing a complete rethink of the filesyste <-> elevator
>> interface.
>
> Yeah, I've been saying that for a while w.r.t. the filesystem/block
> layer interfaces, esp. now with discard requests, data integrity,
> device alignment information, barriers, etc being exposed by the
> layers below the filesystem, but with no interface for filesystems
> to be able to access that information...
>
> Cheers,
>
> Dave.
>

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Wed, Aug 27, 2008 at 02:54:28PM -0700, david@lang.hm wrote:
> On Wed, 27 Aug 2008, Dave Chinner wrote:
>
>> On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
>>> it sounds as if the various flag definitions have been evolving, would it
>>> be worthwhile to sep back and try to get the various filesystem folks to
>>> brainstorm together on what types of hints they would _like_ to see
>>> supported?
>>
>> Three types:
>>
>> 	1. immediate dispatch - merge first with adjacent requests
>> 	   then dispatch
>> 	2. delayed dispatch - queue for a short while to allow
>> 	   merging of requests from above
>> 	3. bulk data - queue and merge. dispatch is completely
>> 	   controlled by the elevator
>
> does this list change if you consider the fact that there may be a raid  
> array or some more complex structure for the block device instead of a  
> simple single disk partition?

No. The whole point of immediate dispatch is that those I/Os are
extremely latency sensitive (i.e. whole fs can stall waiting or
them), so it doesn't matter what the end target is. The faster the
storage subsystem, the more important it is to dispatch those
I/Os immediately to keep the pipes filled...

> since I am suggesting re-thinking the filesystem <-> elevator interface,  
> is there anything you need to have the elevator tell the filesystem? (I'm 
> thinking that this may be the path for the filesystem to learn things  
> about the block device that's under it, is it a raid array, a solid-state 
> drive, etc)

Not so much the elevator, but the block layer in general. That is:

	- capability reporting
		- barriers and type
		- discard support
		- integrity support
		- maximum number of I/Os that can be in flight
		  before congestion occurs
	- geometry of the underlying storage
		- independent domains within the device (e.g. boundaries
		  of linear concatentations)
		- stripe unit/width per domain
		- optimal I/O size per domain
		- latency characteristics per domain
	- notifiers to indicate change of status due to device
	  hotplug back up to the filesystem
		- barrier status change
		- geometry changes due to on-line volume modification
		  (e.g. raid5/6 rebuild after adding a new disk,
		   added another disk to a linear concat, etc)

I'm sure there's more, but that's the list quickly off the top of
my head.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 03:15:08PM +1000, Dave Chinner wrote:
> On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits wrote:
> > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > Everything is default.
> > 
> >   % rpm -qf =mkfs.xfs
> >   xfsprogs-2.9.8-7.1 
> > 
> > which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is the 
> > latest stable mkfs.xfs. Its output is
> > 
> > meta-data=/dev/sda8              isize=256    agcount=4, agsize=1221440 blks
> >          =                       sectsz=512   attr=2
> > data     =                       bsize=4096   blocks=4885760, imaxpct=25
> >          =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096  
> > log      =internal log           bsize=4096   blocks=2560, version=2
> >          =                       sectsz=512   sunit=0 blks, lazy-count=0
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> Ok, I thought it might be the tiny log, but it didn't improve anything
> here when increased the log size, or the log buffer size.

One thing I just found out - my old *laptop* is 4-5x faster than the
10krpm scsi disk behind an old cciss raid controller.  I'm wondering
if the long delays in dispatch is caused by an interaction with CTQ
but I can't change it on the cciss raid controllers. Are you using
ctq/ncq on your machine?  If so, can you reduce the depth to
something less than 4 and see what difference that makes?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Aaron Carroll]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

Dave Chinner wrote:
> One thing I just found out - my old *laptop* is 4-5x faster than the
> 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> if the long delays in dispatch is caused by an interaction with CTQ
> but I can't change it on the cciss raid controllers. Are you using
> ctq/ncq on your machine?  If so, can you reduce the depth to
> something less than 4 and see what difference that makes?

I've been benchmarking on a cciss card, and patched the driver to
control the queue depth via sysfs.  Maybe you'll find it useful...

The original patch was for 2.6.24, but that won't apply on git head.
I fixed it for 2.6.27, and it seems to work fine.  Both are attached.


   -- Aaron


[-- Attachment #2: cciss_qdepth-2.6.24.patch --]
[-- Type: text/plain, Size: 4025 bytes --]

diff --git a/drivers/block/cciss.c b/drivers/block/cciss.c
index 55bd35c..709c419 100644
--- a/drivers/block/cciss.c
+++ b/drivers/block/cciss.c
@@ -474,7 +474,7 @@ static CommandList_struct *cmd_alloc(ctlr_info_t *h, int get_from_pool)
 
 		do {
 			i = find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds);
-			if (i == h->nr_cmds)
+			if (i >= h->qdepth_max)
 				return NULL;
 		} while (test_and_set_bit
 			 (i & (BITS_PER_LONG - 1),
@@ -1257,7 +1257,7 @@ static void cciss_check_queues(ctlr_info_t *h)
 	 * in case the interrupt we serviced was from an ioctl and did not
 	 * free any new commands.
 	 */
-	if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) == h->nr_cmds)
+	if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) >= h->qdepth_max)
 		return;
 
 	/* We have room on the queue for more commands.  Now we need to queue
@@ -1276,7 +1276,7 @@ static void cciss_check_queues(ctlr_info_t *h)
 		/* check to see if we have maxed out the number of commands
 		 * that can be placed on the queue.
 		 */
-		if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) == h->nr_cmds) {
+		if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) >= h->qdepth_max) {
 			if (curr_queue == start_queue) {
 				h->next_to_run =
 				    (start_queue + 1) % (h->highest_lun + 1);
@@ -3075,6 +3075,7 @@ static int __devinit cciss_pci_init(ctlr_info_t *c, struct pci_dev *pdev)
 			c->product_name = products[i].product_name;
 			c->access = *(products[i].access);
 			c->nr_cmds = products[i].nr_cmds;
+			c->qdepth_max = products[i].nr_cmds;
 			break;
 		}
 	}
@@ -3095,6 +3096,7 @@ static int __devinit cciss_pci_init(ctlr_info_t *c, struct pci_dev *pdev)
 			c->product_name = products[i-1].product_name;
 			c->access = *(products[i-1].access);
 			c->nr_cmds = products[i-1].nr_cmds;
+			c->qdepth_max = products[i-1].nr_cmds;
 			printk(KERN_WARNING "cciss: This is an unknown "
 				"Smart Array controller.\n"
 				"cciss: Please update to the latest driver "
@@ -3346,6 +3348,44 @@ static void free_hba(int i)
 	kfree(p);
 }
 
+static inline ctlr_info_t *cciss_get_ctlr_info(struct device *dev)
+{
+	struct pci_dev *pdev = container_of(dev, struct pci_dev, dev);
+	return pci_get_drvdata(pdev);
+}
+
+static ssize_t cciss_show_queue_depth(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	ctlr_info_t *ctlr = cciss_get_ctlr_info(dev);
+	BUG_ON(!ctlr);
+
+	return sprintf(buf, "%u\n", ctlr->qdepth_max);
+}
+
+static ssize_t cciss_store_queue_depth(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	ctlr_info_t *ctlr = cciss_get_ctlr_info(dev);
+	unsigned long qdepth_max;
+
+	BUG_ON(!ctlr);
+	qdepth_max = simple_strtoul(buf, NULL, 10);
+
+	if (qdepth_max < 1)
+		qdepth_max = 1;
+	else if (qdepth_max > ctlr->nr_cmds)
+		qdepth_max = ctlr->nr_cmds;
+
+	ctlr->qdepth_max = (unsigned)qdepth_max;
+	return count;
+}
+
+static struct device_attribute cciss_queue_depth =
+		__ATTR(queue_depth, S_IRUGO | S_IWUSR,
+			&cciss_show_queue_depth,
+			&cciss_store_queue_depth);
+
 /*
  *  This is it.  Find all the controllers and register them.  I really hate
  *  stealing all these major device numbers.
@@ -3450,6 +3490,11 @@ static int __devinit cciss_init_one(struct pci_dev *pdev,
 	       ((hba[i]->nr_cmds + BITS_PER_LONG -
 		 1) / BITS_PER_LONG) * sizeof(unsigned long));
 
+	/* Setup queue_depth sysfs entry */
+	rc = device_create_file(&pdev->dev, &cciss_queue_depth);
+	if (rc)
+		goto clean4;
+
 #ifdef CCISS_DEBUG
 	printk(KERN_DEBUG "Scanning for drives on controller cciss%d\n", i);
 #endif				/* CCISS_DEBUG */
diff --git a/drivers/block/cciss.h b/drivers/block/cciss.h
index b70988d..6a4a38a 100644
--- a/drivers/block/cciss.h
+++ b/drivers/block/cciss.h
@@ -60,6 +60,7 @@ struct ctlr_info
 	void __iomem *vaddr;
 	unsigned long paddr;
 	int 	nr_cmds; /* Number of commands allowed on this controller */
+	unsigned qdepth_max;  /* userspace queue depth limit */
 	CfgTable_struct __iomem *cfgtable;
 	int	interrupts_enabled;
 	int	major;

[-- Attachment #3: cciss_qdepth-2.6.27.patch --]
[-- Type: text/plain, Size: 3974 bytes --]

diff --git a/drivers/block/cciss.c b/drivers/block/cciss.c
index b73116e..066577f 100644
--- a/drivers/block/cciss.c
+++ b/drivers/block/cciss.c
@@ -480,7 +480,7 @@ static CommandList_struct *cmd_alloc(ctlr_info_t *h, int get_from_pool)
 
 		do {
 			i = find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds);
-			if (i == h->nr_cmds)
+			if (i >= h->qdepth_max)
 				return NULL;
 		} while (test_and_set_bit
 			 (i & (BITS_PER_LONG - 1),
@@ -1259,7 +1259,7 @@ static void cciss_check_queues(ctlr_info_t *h)
 	 * in case the interrupt we serviced was from an ioctl and did not
 	 * free any new commands.
 	 */
-	if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) == h->nr_cmds)
+	if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) >= h->qdepth_max)
 		return;
 
 	/* We have room on the queue for more commands.  Now we need to queue
@@ -1278,7 +1278,7 @@ static void cciss_check_queues(ctlr_info_t *h)
 		/* check to see if we have maxed out the number of commands
 		 * that can be placed on the queue.
 		 */
-		if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) == h->nr_cmds) {
+		if ((find_first_zero_bit(h->cmd_pool_bits, h->nr_cmds)) >= h->qdepth_max) {
 			if (curr_queue == start_queue) {
 				h->next_to_run =
 				    (start_queue + 1) % (h->highest_lun + 1);
@@ -3253,6 +3253,7 @@ static int __devinit cciss_pci_init(ctlr_info_t *c, struct pci_dev *pdev)
 			c->product_name = products[i].product_name;
 			c->access = *(products[i].access);
 			c->nr_cmds = c->max_commands - 4;
+			c->qdepth_max = c->nr_cmds;
 			break;
 		}
 	}
@@ -3273,6 +3274,7 @@ static int __devinit cciss_pci_init(ctlr_info_t *c, struct pci_dev *pdev)
 			c->product_name = products[i-1].product_name;
 			c->access = *(products[i-1].access);
 			c->nr_cmds = c->max_commands - 4;
+			c->qdepth_max = c->nr_cmds;
 			printk(KERN_WARNING "cciss: This is an unknown "
 				"Smart Array controller.\n"
 				"cciss: Please update to the latest driver "
@@ -3392,6 +3394,44 @@ static void free_hba(int i)
 	kfree(p);
 }
 
+static inline ctlr_info_t *cciss_get_ctlr_info(struct device *dev)
+{
+	struct pci_dev *pdev = container_of(dev, struct pci_dev, dev);
+	return pci_get_drvdata(pdev);
+}
+
+static ssize_t cciss_show_queue_depth(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	ctlr_info_t *ctlr = cciss_get_ctlr_info(dev);
+	BUG_ON(!ctlr);
+
+	return sprintf(buf, "%u\n", ctlr->qdepth_max);
+}
+
+static ssize_t cciss_store_queue_depth(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	ctlr_info_t *ctlr = cciss_get_ctlr_info(dev);
+	unsigned long qdepth_max;
+
+	BUG_ON(!ctlr);
+	qdepth_max = simple_strtoul(buf, NULL, 10);
+
+	if (qdepth_max < 1)
+		qdepth_max = 1;
+	else if (qdepth_max > ctlr->nr_cmds)
+		qdepth_max = ctlr->nr_cmds;
+
+	ctlr->qdepth_max = (unsigned)qdepth_max;
+	return count;
+}
+
+static struct device_attribute cciss_queue_depth =
+		__ATTR(queue_depth, S_IRUGO | S_IWUSR,
+			&cciss_show_queue_depth,
+			&cciss_store_queue_depth);
+
 /*
  *  This is it.  Find all the controllers and register them.  I really hate
  *  stealing all these major device numbers.
@@ -3496,6 +3536,11 @@ static int __devinit cciss_init_one(struct pci_dev *pdev,
 	       ((hba[i]->nr_cmds + BITS_PER_LONG -
 		 1) / BITS_PER_LONG) * sizeof(unsigned long));
 
+	/* Setup queue_depth sysfs entry */
+	rc = device_create_file(&pdev->dev, &cciss_queue_depth);
+	if (rc)
+		goto clean4;
+
 	hba[i]->num_luns = 0;
 	hba[i]->highest_lun = -1;
 	for (j = 0; j < CISS_MAX_LUN; j++) {
diff --git a/drivers/block/cciss.h b/drivers/block/cciss.h
index 24a7efa..91dcac6 100644
--- a/drivers/block/cciss.h
+++ b/drivers/block/cciss.h
@@ -62,6 +62,7 @@ struct ctlr_info
 	void __iomem *vaddr;
 	unsigned long paddr;
 	int 	nr_cmds; /* Number of commands allowed on this controller */
+	unsigned qdepth_max;  /* userspace queue depth limit */
 	CfgTable_struct __iomem *cfgtable;
 	int	interrupts_enabled;
 	int	major;

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> On Thu, Aug 21, 2008 at 03:15:08PM +1000, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits wrote:
> > > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > Everything is default.
> > > 
> > >   % rpm -qf =mkfs.xfs
> > >   xfsprogs-2.9.8-7.1 
> > > 
> > > which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is the 
> > > latest stable mkfs.xfs. Its output is
> > > 
> > > meta-data=/dev/sda8              isize=256    agcount=4, agsize=1221440 blks
> > >          =                       sectsz=512   attr=2
> > > data     =                       bsize=4096   blocks=4885760, imaxpct=25
> > >          =                       sunit=0      swidth=0 blks
> > > naming   =version 2              bsize=4096  
> > > log      =internal log           bsize=4096   blocks=2560, version=2
> > >          =                       sectsz=512   sunit=0 blks, lazy-count=0
> > > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > 
> > Ok, I thought it might be the tiny log, but it didn't improve anything
> > here when increased the log size, or the log buffer size.
> 
> One thing I just found out - my old *laptop* is 4-5x faster than the
> 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> if the long delays in dispatch is caused by an interaction with CTQ
> but I can't change it on the cciss raid controllers. Are you using
> ctq/ncq on your machine?  If so, can you reduce the depth to
> something less than 4 and see what difference that makes?

Just to point out - this is not a new problem - I can reproduce
it on 2.6.24 as well as 2.6.26. Likewise, my laptop shows XFS
being faster than ext3 on both 2.6.24 and 2.6.26. So the difference
is something related to the disk subsystem on the server....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Martin Steigerwald]

Am Donnerstag 21 August 2008 schrieb Dave Chinner:
> On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 03:15:08PM +1000, Dave Chinner wrote:
> > > On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits wrote:
> > > > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > > Everything is default.
> > > >
> > > >   % rpm -qf =mkfs.xfs
> > > >   xfsprogs-2.9.8-7.1
> > > >
> > > > which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is
> > > > the latest stable mkfs.xfs. Its output is
> > > >
> > > > meta-data=/dev/sda8              isize=256    agcount=4,
> > > > agsize=1221440 blks =                       sectsz=512   attr=2
> > > > data     =                       bsize=4096   blocks=4885760,
> > > > imaxpct=25 =                       sunit=0      swidth=0 blks
> > > > naming   =version 2              bsize=4096
> > > > log      =internal log           bsize=4096   blocks=2560,
> > > > version=2 =                       sectsz=512   sunit=0 blks,
> > > > lazy-count=0 realtime =none                   extsz=4096  
> > > > blocks=0, rtextents=0
> > >
> > > Ok, I thought it might be the tiny log, but it didn't improve
> > > anything here when increased the log size, or the log buffer size.
> >
> > One thing I just found out - my old *laptop* is 4-5x faster than the
> > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > if the long delays in dispatch is caused by an interaction with CTQ
> > but I can't change it on the cciss raid controllers. Are you using
> > ctq/ncq on your machine?  If so, can you reduce the depth to
> > something less than 4 and see what difference that makes?
>
> Just to point out - this is not a new problem - I can reproduce
> it on 2.6.24 as well as 2.6.26. Likewise, my laptop shows XFS
> being faster than ext3 on both 2.6.24 and 2.6.26. So the difference
> is something related to the disk subsystem on the server....

Interesting. I switched from cfq to deadline some time ago, due to abysmal 
XFS performance on parallel IO - aptitude upgrade and doing desktop 
stuff. Just my subjective perception, but I have seen it crawl, even 
stall for 5-10 seconds easily at times. I found deadline to be way faster 
initially, but then it rarely happened that IO for desktop tasks is 
basically stalled for even longer, say 15 seconds or more, on parallel 
IO. However I can't remember having this problem with the last kernel 
2.6.26.2.

I am now testing with cfq again. On a ThinkPad T42 internal 160 GB 
harddisk with barriers enabled. But you tell, it only happens on certain 
servers, so I might have seen something different.

Thus I had the rough feeling that something is wrong with at least CFQ and 
XFS together, but I couldn't prove it back then. I have no idea how to 
easily do a reproducable test case. Maybe having a script that unpacks 
kernel source archives while I try to use the desktop...

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 6210 bytes --]

Am Donnerstag 21 August 2008 schrieb Martin Steigerwald:
> Am Donnerstag 21 August 2008 schrieb Dave Chinner:
> > On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > > On Thu, Aug 21, 2008 at 03:15:08PM +1000, Dave Chinner wrote:
> > > > On Thu, Aug 21, 2008 at 05:46:00AM +0300, Szabolcs Szakacsits 
wrote:
> > > > > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > > > Everything is default.
> > > > >
> > > > >   % rpm -qf =mkfs.xfs
> > > > >   xfsprogs-2.9.8-7.1
> > > > >
> > > > > which, according to ftp://oss.sgi.com/projects/xfs/cmd_tars, is
> > > > > the latest stable mkfs.xfs. Its output is
> > > > >
> > > > > meta-data=/dev/sda8              isize=256    agcount=4,
> > > > > agsize=1221440 blks =                       sectsz=512   attr=2
> > > > > data     =                       bsize=4096   blocks=4885760,
> > > > > imaxpct=25 =                       sunit=0      swidth=0 blks
> > > > > naming   =version 2              bsize=4096
> > > > > log      =internal log           bsize=4096   blocks=2560,
> > > > > version=2 =                       sectsz=512   sunit=0 blks,
> > > > > lazy-count=0 realtime =none                   extsz=4096
> > > > > blocks=0, rtextents=0
> > > >
> > > > Ok, I thought it might be the tiny log, but it didn't improve
> > > > anything here when increased the log size, or the log buffer
> > > > size.
> > >
> > > One thing I just found out - my old *laptop* is 4-5x faster than
> > > the 10krpm scsi disk behind an old cciss raid controller.  I'm
> > > wondering if the long delays in dispatch is caused by an
> > > interaction with CTQ but I can't change it on the cciss raid
> > > controllers. Are you using ctq/ncq on your machine?  If so, can you
> > > reduce the depth to something less than 4 and see what difference
> > > that makes?
> >
> > Just to point out - this is not a new problem - I can reproduce
> > it on 2.6.24 as well as 2.6.26. Likewise, my laptop shows XFS
> > being faster than ext3 on both 2.6.24 and 2.6.26. So the difference
> > is something related to the disk subsystem on the server....
>
> Interesting. I switched from cfq to deadline some time ago, due to
> abysmal XFS performance on parallel IO - aptitude upgrade and doing
> desktop stuff. Just my subjective perception, but I have seen it crawl,
> even stall for 5-10 seconds easily at times. I found deadline to be way
> faster initially, but then it rarely happened that IO for desktop tasks
> is basically stalled for even longer, say 15 seconds or more, on
> parallel IO. However I can't remember having this problem with the last
> kernel 2.6.26.2.
>
> I am now testing with cfq again. On a ThinkPad T42 internal 160 GB
> harddisk with barriers enabled. But you tell, it only happens on
> certain servers, so I might have seen something different.
>
> Thus I had the rough feeling that something is wrong with at least CFQ
> and XFS together, but I couldn't prove it back then. I have no idea how
> to easily do a reproducable test case. Maybe having a script that
> unpacks kernel source archives while I try to use the desktop...

Okay, some numbers attached:

- On XFS: Barrier versus Nobarrier makes quite a difference with 
compilebench. Also on rm -rf'ing the large directory tree it leaves 
behind. While I did not measure the first barrier related compilebench 
directory deletion I am pretty sure it took way longer. Also vmstat 
throughput it higher without nobarriers.
 
- On XFS: CFQ versus NOOP does not seem to make that much of a difference, 
at least not with barriers enabled (didn't test without). With NOOP 
responsiveness was even weaker than with CFQ. Opening a context menu on a 
webpage link displayed in Konqueror could take easily a minute or more. I 
think it shall never ever take that long for the OS to respond to user 
input.

- Ext3, NILFS, BTRFS with CFQ: Perform quite well. Especially btrfs. nilfs 
text isn't complete, cause likely due to checkpoints those 4G I dedicated 
to it were not enough for the compilebench test to complete.

So at least here performance degration with XFS seems more related to 
barriers than scheduler decision - least when it comes to the two choices 
CFQ and NOOP. But no, I won't switch barriers off permanently on my 
laptop. ;) Would be fine if performance impact of barriers could be 
reduced a bit tough.

At last I appear to see something different than the I/O scheduler issue 
discussed here.

Anyway subjectively I am quite happy with XFS performance nonetheless. But 
then since I can't switch from XFS to ext3 or btrfs in a second I can't 
really compare subjective impressions. Maybe desktop would respond faster 
with ext3 or btrfs? Who knows?

I think a script which does extensive automated testing would be fine:

- have some basic settings like

SCRATCH_DEV=/dev/sda8 (this should be a real partition in order to be able 
to test barriers which do not work over LVM / device mapper)

SCRATCH_MNT=/mnt/test

- have an array of pre-pre-test setups like

[ echo "cfq" >/sys/block/sda/queue/scheduler ]
[ echo "deadline" >/sys/block/sda/queue/scheduler ]
[ echo "anticipatory" >/sys/block/sda/queue/scheduler ]
[ echo "noop" >/sys/block/sda/queue/scheduler ]

- have an array of pre-test setups like

[ mkfs.xfs -f $SCRATCH_DEV
mount $SCRATCH_DEV $SCRATCH_MNT ]
[ mkfs.xfs -f $SCRATCH_DEV
mount -o nobarrier $SCRATCH_DEV $SCRATCH_MNT ]
[ mkfs.xfs -f $SCRATCH_DEV
mount -o logbsize=256k $SCRATCH_DEV $SCRATCH_MNT ]
[ mkfs.btrfs $SCRATCH_DEV
mount $SCRATCH_DEV $SCRATCH_MNT ]

- have an array of tests like

[ ./compilebench -D /mnt/zeit-btrfs -i 5 -r 10 ]
[ postmark whatever ]
[ iozone whatever ]

- and let it run every combination of those array elements unattended 
(over night;-)

- have any results collected with settings for each patch and basic 
machine info in one easy to share text file

- then as additional feature let it test responsiveness during each 
running test. Let it makes sure there are some files that are not in the 
cache and let it access one of those files once in a while and measure 
how long it takes the filesystem to respond

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: filesystem-benchmarks-compilebench-2008-08-21.txt --]
[-- Type: text/plain, Size: 20700 bytes --]



martin@shambhala:~> date
Do 21. Aug 13:27:49 CEST 2008

shambhala:~> cat /proc/version
Linux version 2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch
(martin@shambala) (gcc version 4.3.1 (Debian 4.3.1-8) ) #1 PREEMPT Wed
Aug 13 10:10:11 CEST 2008

shambhala:~> apt-show-versions | egrep "(btrfs|nilfs)"
btrfs-modules-2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch 0.15-1+1
installed: No available version in archive
btrfs-source/lenny uptodate 0.15-1
btrfs-tools/lenny uptodate 0.15-2
nilfs2-modules-2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch 2.0.4-1+1
installed: No available version in archive
nilfs2-source/sid uptodate 2.0.4-1
nilfs2-tools/lenny uptodate 2.0.5-1

shambhala:~> cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 13
model name      : Intel(R) Pentium(R) M processor 1.80GHz
stepping        : 6
cpu MHz         : 600.000
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov
pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2
bogomips        : 1197.54
clflush size    : 64
power management:


martin@shambhala:~> cat /proc/mounts | tail -4
/dev/mapper/shambala-ext3 /mnt/zeit-ext3 ext3
rw,errors=continue,data=ordered 0 0
/dev/mapper/shambala-nilfs /mnt/zeit-nilfs2 nilfs2 rw 0 0
/dev/mapper/shambala-btrfs /mnt/zeit-btrfs btrfs rw 0 0
/dev/mapper/shambala-xfs /mnt/zeit-xfs xfs
rw,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0
martin@shambhala:~> df -hT | tail -8
/dev/mapper/shambala-ext3
              ext3    4,0G  137M  3,7G   4% /mnt/zeit-ext3
/dev/mapper/shambala-nilfs
            nilfs2    4,0G   16M  3,8G   1% /mnt/zeit-nilfs2
/dev/mapper/shambala-btrfs
             btrfs    4,0G   40K  4,0G   1% /mnt/zeit-btrfs
/dev/mapper/shambala-xfs
               xfs    4,0G  4,2M  4,0G   1% /mnt/zeit-xfs

shambhala:~> xfs_info /mnt/zeit-xfs
meta-data=/dev/mapper/shambala-xfs isize=256    agcount=4, agsize=262144
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=1048576, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

martin@shambhala:~> cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]


XFS without barriers, since device mapper doesn't support barrier
requests (http://bugzilla.kernel.org/show_bug.cgi?id=9554):


shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-xfs -i 5 -r
10
using working directory /mnt/zeit-xfs, 5 intial dirs 10 runs
native unpatched native-0 222MB in 25.88 seconds (8.59 MB/s)
native patched native-0 109MB in 5.82 seconds (18.84 MB/s)
native patched compiled native-0 691MB in 33.69 seconds (20.53 MB/s)
create dir kernel-0 222MB in 20.38 seconds (10.91 MB/s)
create dir kernel-1 222MB in 27.27 seconds (8.15 MB/s)
create dir kernel-2 222MB in 26.69 seconds (8.33 MB/s)
create dir kernel-3 222MB in 25.17 seconds (8.83 MB/s)
create dir kernel-4 222MB in 29.52 seconds (7.53 MB/s)
patch dir kernel-2 109MB in 38.54 seconds (2.85 MB/s)
compile dir kernel-2 691MB in 41.60 seconds (16.62 MB/s)
compile dir kernel-4 680MB in 49.46 seconds (13.76 MB/s)
patch dir kernel-4 691MB in 118.19 seconds (5.85 MB/s)
read dir kernel-4 in 77.09 11.89 MB/s
read dir kernel-3 in 30.91 7.19 MB/s
create dir kernel-3116 222MB in 42.73 seconds (5.20 MB/s)
clean kernel-4 691MB in 6.48 seconds (106.73 MB/s)
read dir kernel-1 in 32.08 6.93 MB/s
stat dir kernel-0 in 6.94 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 8.75 MB/s (user 2.05s sys 3.72s)
create total runs 1 avg 5.20 MB/s (user 2.40s sys 5.34s)
patch total runs 2 avg 4.35 MB/s (user 0.83s sys 3.93s)
compile total runs 2 avg 15.19 MB/s (user 0.56s sys 2.90s)
clean total runs 1 avg 106.73 MB/s (user 0.07s sys 0.40s)
read tree total runs 2 avg 7.06 MB/s (user 1.93s sys 3.94s)
read compiled tree total runs 1 avg 11.89 MB/s (user 2.29s sys 6.22s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 6.94 seconds (user 1.13s sys 0.94s)
no runs for stat compiled tree


With barriers on an already heavily populated filesystem - I don't have
an empty one on a raw partition at hand at the moment and I for sure
won't empty this one:

martin@shambhala:~> df -hT | grep /home
/dev/sda5      xfs    112G  104G  8,2G  93% /home

shambhala:~> df -hiT | grep /home
/dev/sda5      xfs       34M    751K     33M    3% /home

shambhala:~> xfs_db -rx /dev/sda5
xfs_db> frag
actual 726986, ideal 703687, fragmentation factor 3.20%
xfs_db> quit
shambhala:~>

martin@shambhala:~> cat /proc/mounts | grep "/home "
/dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0
0

shambhala:~> xfs_info /home
meta-data=/dev/sda5              isize=256    agcount=6, agsize=4883256
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=29299536,
imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D
/home/martin/Zeit/compilebench -i 5 -r 10
using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10
runs
native unpatched native-0 222MB in 117.37 seconds (1.89 MB/s)
native patched native-0 109MB in 27.46 seconds (3.99 MB/s)
native patched compiled native-0 691MB in 48.03 seconds (14.40 MB/s)
create dir kernel-0 222MB in 83.55 seconds (2.66 MB/s)
create dir kernel-1 222MB in 86.01 seconds (2.59 MB/s)
create dir kernel-2 222MB in 71.61 seconds (3.11 MB/s)
create dir kernel-3 222MB in 71.73 seconds (3.10 MB/s)
create dir kernel-4 222MB in 61.61 seconds (3.61 MB/s)
patch dir kernel-2 109MB in 63.14 seconds (1.74 MB/s)
compile dir kernel-2 691MB in 45.61 seconds (15.16 MB/s)
compile dir kernel-4 680MB in 50.13 seconds (13.58 MB/s)
patch dir kernel-4 691MB in 154.38 seconds (4.48 MB/s)
read dir kernel-4 in 95.04 9.65 MB/s
read dir kernel-3 in 49.49 4.49 MB/s
create dir kernel-3116 222MB in 79.44 seconds (2.80 MB/s)
clean kernel-4 691MB in 8.64 seconds (80.05 MB/s)
read dir kernel-1 in 71.40 3.11 MB/s
stat dir kernel-0 in 14.44 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 3.01 MB/s (user 2.34s sys 4.30s)
create total runs 1 avg 2.80 MB/s (user 2.36s sys 4.12s)
patch total runs 2 avg 3.11 MB/s (user 0.91s sys 4.07s)
compile total runs 2 avg 14.37 MB/s (user 0.60s sys 2.76s)
clean total runs 1 avg 80.05 MB/s (user 0.09s sys 0.45s)
read tree total runs 2 avg 3.80 MB/s (user 2.00s sys 4.05s)
read compiled tree total runs 1 avg 9.65 MB/s (user 2.36s sys 6.42s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 14.44 seconds (user 1.17s sys 1.07s)
no runs for stat compiled tree

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> rm -rf /home/martin/Zeit/compilebench

I didn't measure it, but it took *ages* while rm -rf was mostly in D
state. According to harddisk noise a lot of seeks where involved.

vmstat 1 during the rm -rf:

 0  0   2784 748048     20 247160    0    0   160  4628  352 1224 15 14
71  0
 0  0   2784 748056     20 247308    0    0   148  3848  298  442 11 10
79  0
 0  0   2784 747996     20 247428    0    0   120  3377  260  449  9  9
82  0
 0  0   2784 747764     20 247580    0    0   152  4364  324 1094 20 10
70  0
 1  0   2784 747452     20 247736    0    0   156  4356  279  814 15 11
74  0
 0  0   2784 747408     20 247900    0    0   164  4112  360 1131 13 13
74  0
 0  0   2784 747136     20 248064    0    0   164  5128  318  855 16 10
74  0
 0  0   2784 746780     20 248208    0    0   144  4353  305 1066 20 12
68  0
 0  0   2784 746204     20 248336    0    0   128  5388  275  966 14 11
75  0
 1  0   2784 748352     20 248468    0    0   132  5384  314 1234 22 11
67  0
 0  0   2784 748104     20 248604    0    0   136  4873  284  807 16 11
73  0


Same game on same productively used partition, but now without barriers:

shambhala:~> mount -o remount,nobarrier /home
shambhala:~> cat /proc/mounts | grep "/home "
/dev/sda5 /home xfs
rw,relatime,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> mkdir /home/martin/Zeit/compilebench

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D
/home/martin/Zeit/compilebench -i 5 -r 10
using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10
runs
native unpatched native-0 222MB in 51.44 seconds (4.32 MB/s)
native patched native-0 109MB in 12.69 seconds (8.64 MB/s)
native patched compiled native-0 691MB in 51.75 seconds (13.36 MB/s)
create dir kernel-0 222MB in 47.64 seconds (4.67 MB/s)
create dir kernel-1 222MB in 53.40 seconds (4.16 MB/s)
create dir kernel-2 222MB in 48.04 seconds (4.63 MB/s)
create dir kernel-3 222MB in 38.26 seconds (5.81 MB/s)
create dir kernel-4 222MB in 34.15 seconds (6.51 MB/s)
patch dir kernel-2 109MB in 50.61 seconds (2.17 MB/s)
compile dir kernel-2 691MB in 37.94 seconds (18.23 MB/s)
compile dir kernel-4 680MB in 45.32 seconds (15.02 MB/s)
patch dir kernel-4 691MB in 107.27 seconds (6.45 MB/s)
read dir kernel-4 in 82.18 11.16 MB/s
read dir kernel-3 in 42.35 5.25 MB/s
create dir kernel-3116 222MB in 38.27 seconds (5.81 MB/s)
clean kernel-4 691MB in 5.92 seconds (116.82 MB/s)
read dir kernel-1 in 73.63 3.02 MB/s
stat dir kernel-0 in 13.77 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 5.16 MB/s (user 2.21s sys 4.23s)
create total runs 1 avg 5.81 MB/s (user 2.18s sys 4.89s)
patch total runs 2 avg 4.31 MB/s (user 0.90s sys 4.05s)
compile total runs 2 avg 16.62 MB/s (user 0.59s sys 3.05s)
clean total runs 1 avg 116.82 MB/s (user 0.09s sys 0.41s)
read tree total runs 2 avg 4.14 MB/s (user 1.90s sys 4.02s)
read compiled tree total runs 1 avg 11.16 MB/s (user 2.28s sys 6.36s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 13.77 seconds (user 1.19s sys 1.01s)
no runs for stat compiled tree


Not as fast as on the clean XFS LV, but still almost everytime almost
twice as fast as with barriers.


shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> time rm -rf
/home/martin/Zeit/compilebench
rm -rf /home/martin/Zeit/compilebench  0,32s user 19,19s system 15% cpu
2:09,79 total

This is definately faster than before. I didn't measure exact time on
first occasion, but it took ages.

vmstat 1 during the rm -rf indicated much higher metadata throughput:

 3  0   2780 827696     20 162492    0    0   280 11109  449  865 31 15
52  2
 0  0   2780 827304     20 162816    0    0   324  6656  468 1009 57  8 
7 28
 2  0   2636 828992     20 163364    0    0   540  5317  350  545 30 10
30 31
 2  1   2636 837488     20 164020    0    0   656  7691  394  650 39 12 
0 49
 0  0   2224 960360     20 164516    0    0   496 12060  420  549 13 26
56  5
 0  0   2224 959988     20 164904    0    0   388 13704  425  792 16 23
61  0
 0  0   2224 959864     20 165128    0    0   224  6209  363  503 12 10
78  0
 1  0   2224 959376     20 165540    0    0   412 14886  392  513 12 22
66  0


Now with barriers again, but with "noop" as scheduler:

shambhala:~> mount -o remount,barrier /home
shambhala:~> cat /proc/mounts | grep /home
/dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0
0
shambhala:~> echo "noop" >/sys/block/sda/queue/scheduler
shambhala:~> cat /sys/block/sda/queue/scheduler
[noop] anticipatory deadline cfq

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> mkdir /home/martin/Zeit/compilebench

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D
/home/martin/Zeit/compilebench -i 5 -r 10
using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10
runs
native unpatched native-0 222MB in 97.42 seconds (2.28 MB/s)
native patched native-0 109MB in 20.72 seconds (5.29 MB/s)
native patched compiled native-0 691MB in 46.37 seconds (14.91 MB/s)
create dir kernel-0 222MB in 84.12 seconds (2.64 MB/s)
create dir kernel-1 222MB in 95.18 seconds (2.34 MB/s)
create dir kernel-2 222MB in 74.57 seconds (2.98 MB/s)
create dir kernel-3 222MB in 71.81 seconds (3.10 MB/s)
create dir kernel-4 222MB in 64.77 seconds (3.43 MB/s)
patch dir kernel-2 109MB in 81.22 seconds (1.35 MB/s)
compile dir kernel-2 691MB in 41.87 seconds (16.52 MB/s)
compile dir kernel-4 680MB in 50.35 seconds (13.52 MB/s)
patch dir kernel-4 691MB in 151.03 seconds (4.58 MB/s)
read dir kernel-4 in 82.83 11.07 MB/s
read dir kernel-3 in 48.49 4.59 MB/s
create dir kernel-3116 222MB in 79.43 seconds (2.80 MB/s)
clean kernel-4 691MB in 15.51 seconds (44.59 MB/s)
read dir kernel-1 in 75.36 2.95 MB/s
stat dir kernel-0 in 14.65 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 2.90 MB/s (user 2.35s sys 4.56s)
create total runs 1 avg 2.80 MB/s (user 2.18s sys 3.92s)
patch total runs 2 avg 2.96 MB/s (user 0.87s sys 4.07s)
compile total runs 2 avg 15.02 MB/s (user 0.60s sys 2.73s)
clean total runs 1 avg 44.59 MB/s (user 0.07s sys 0.44s)
read tree total runs 2 avg 3.77 MB/s (user 2.03s sys 3.82s)
read compiled tree total runs 1 avg 11.07 MB/s (user 2.29s sys 6.24s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 14.65 seconds (user 1.12s sys 1.00s)
no runs for stat compiled tree

Some tests run a bit faster, but on cost of responsiveness to out of
line I/Os (opening a new webpage in Konqueror). Some do not run faster
at all.

Seems that write barriers on/off make the bigger difference here.

As last XFS thing:

vmstat 1 during a rm -rf while switching of XFS from nobarrier to
barrier:

 0  0   1976 422236   1784 516840    0    0   508 17160  410  540  7 23
70  0
 1  0   1976 420624   1784 517576    0    0   736 26904  539 1032 14 35
51  0
 0  0   1976 419176   1784 518152    0    0   576 23842  486 1060 17 33
50  0
 0  0   1976 418316   1784 518460    0    0   308 12812  317  552  6 18
76  0
 2  0   1976 417392   1784 518776    0    0   316 16689  360  882  2 23
75  0
 8  0   1976 432948   1784 519252    0    0   476 16710  452  630  8 39
53  0
 0  0   1976 432892   1784 519392    0    0   140  4146  371 1564 14 26
60  0
 0  0   1976 432628   1784 519572    0    0   180  3844  340  660 11 10
79  0
 0  0   1976 432496   1784 519736    0    0   164  3852  328  534  9  8
83  0
 0  0   1976 432372   1784 519920    0    0   176  4100  359  788 19 11
70  0

Its obvious, where it was switched to barrier ;)



Now the other filesystems with CFQ enabled.

Ext3:

shambhala:~> echo "cfq" >/sys/block/sda/queue/scheduler
shambhala:~> cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-ext3 -i 5
-r 10
using working directory /mnt/zeit-ext3, 5 intial dirs 10 runs
native unpatched native-0 222MB in 16.90 seconds (13.16 MB/s)
native patched native-0 109MB in 4.63 seconds (23.69 MB/s)
native patched compiled native-0 691MB in 39.78 seconds (17.39 MB/s)
create dir kernel-0 222MB in 12.24 seconds (18.17 MB/s)
create dir kernel-1 222MB in 16.71 seconds (13.31 MB/s)
create dir kernel-2 222MB in 18.50 seconds (12.02 MB/s)
create dir kernel-3 222MB in 18.25 seconds (12.18 MB/s)
create dir kernel-4 222MB in 27.24 seconds (8.16 MB/s)
patch dir kernel-2 109MB in 29.26 seconds (3.75 MB/s)
compile dir kernel-2 691MB in 53.41 seconds (12.95 MB/s)
compile dir kernel-4 680MB in 55.24 seconds (12.32 MB/s)
patch dir kernel-4 691MB in 108.66 seconds (6.36 MB/s)
read dir kernel-4 in 79.38 11.55 MB/s
read dir kernel-3 in 21.65 10.27 MB/s
create dir kernel-3116 222MB in 28.22 seconds (7.88 MB/s)
clean kernel-4 691MB in 17.05 seconds (40.56 MB/s)
read dir kernel-1 in 23.67 9.39 MB/s
stat dir kernel-0 in 9.63 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 12.77 MB/s (user 1.96s sys 3.24s)
create total runs 1 avg 7.88 MB/s (user 1.57s sys 2.39s)
patch total runs 2 avg 5.06 MB/s (user 0.78s sys 3.92s)
compile total runs 2 avg 12.64 MB/s (user 0.54s sys 3.75s)
clean total runs 1 avg 40.56 MB/s (user 0.08s sys 0.36s)
read tree total runs 2 avg 9.83 MB/s (user 1.82s sys 4.32s)
read compiled tree total runs 1 avg 11.55 MB/s (user 2.32s sys 7.02s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 9.63 seconds (user 1.11s sys 0.89s)
no runs for stat compiled tree


nilfs2:

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-nilfs2 -i 5
-r 10
using working directory /mnt/zeit-nilfs2, 5 intial dirs 10 runs
native unpatched native-0 222MB in 20.28 seconds (10.97 MB/s)
native patched native-0 109MB in 8.83 seconds (12.42 MB/s)
native patched compiled native-0 691MB in 42.44 seconds (16.30 MB/s)
create dir kernel-0 222MB in 20.89 seconds (10.65 MB/s)
create dir kernel-1 222MB in 21.13 seconds (10.52 MB/s)
create dir kernel-2 222MB in 20.22 seconds (11.00 MB/s)
create dir kernel-3 222MB in 21.60 seconds (10.30 MB/s)
create dir kernel-4 222MB in 20.63 seconds (10.78 MB/s)
patch dir kernel-2 109MB in 20.97 seconds (5.23 MB/s)
compile dir kernel-2 691MB in 44.40 seconds (15.58 MB/s)
Traceback (most recent call last):
  File "./compilebench", line 631, in <module>
    total_runs += func(dset, rnd)
  File "./compilebench", line 368, in compile_one_dir
    mbs = run_directory(ch[0], dir, "compile dir")
  File "./compilebench", line 241, in run_directory
    fp.write(buf[:cur])
IOError: [Errno 28] No space left on device


Okay, possibly due to those 11 checkpoints it stored. Seems I would need
more than 4 GB for the test to complete. But enough testing for today
;).


btrfs 0.15:

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-btrfs -i 5
-r 10
using working directory /mnt/zeit-btrfs, 5 intial dirs 10 runs
native unpatched native-0 222MB in 13.61 seconds (16.34 MB/s)
native patched native-0 109MB in 3.12 seconds (35.15 MB/s)
native patched compiled native-0 691MB in 28.84 seconds (23.98 MB/s)
create dir kernel-0 222MB in 10.99 seconds (20.23 MB/s)
create dir kernel-1 222MB in 13.95 seconds (15.94 MB/s)
create dir kernel-2 222MB in 14.99 seconds (14.83 MB/s)
create dir kernel-3 222MB in 15.00 seconds (14.82 MB/s)
create dir kernel-4 222MB in 16.16 seconds (13.76 MB/s)
patch dir kernel-2 109MB in 30.09 seconds (3.64 MB/s)
compile dir kernel-2 691MB in 58.05 seconds (11.91 MB/s)
compile dir kernel-4 680MB in 55.23 seconds (12.32 MB/s)
patch dir kernel-4 691MB in 134.20 seconds (5.15 MB/s)
read dir kernel-4 in 108.58 8.44 MB/s
read dir kernel-3 in 43.47 5.12 MB/s
create dir kernel-3116 222MB in 27.81 seconds (8.00 MB/s)
clean kernel-4 691MB in 17.63 seconds (39.23 MB/s)
read dir kernel-1 in 70.31 3.16 MB/s
stat dir kernel-0 in 32.85 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 15.92 MB/s (user 1.06s sys 5.43s)
create total runs 1 avg 8.00 MB/s (user 1.17s sys 7.41s)
patch total runs 2 avg 4.40 MB/s (user 0.88s sys 10.55s)
compile total runs 2 avg 12.12 MB/s (user 0.56s sys 5.34s)
clean total runs 1 avg 39.23 MB/s (user 0.05s sys 2.30s)
read tree total runs 2 avg 4.14 MB/s (user 1.85s sys 10.00s)
read compiled tree total runs 1 avg 8.44 MB/s (user 2.19s sys 16.50s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 32.85 seconds (user 1.01s sys 3.35s)
no runs for stat compiled tree

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Szabolcs Szakacsits]

On Thu, 21 Aug 2008, Dave Chinner wrote:
> On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > 
> > One thing I just found out - my old *laptop* is 4-5x faster than the
> > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > if the long delays in dispatch is caused by an interaction with CTQ
> > but I can't change it on the cciss raid controllers. Are you using
> > ctq/ncq on your machine?  

It's a laptop and has NCQ. It makes no difference if NCQ is enabled or 
disabled. The problem seems to be XFS only.

> > If so, can you reduce the depth to something less than 4 and see what 
> > difference that makes?
> 
> Just to point out - this is not a new problem - I can reproduce
> it on 2.6.24 as well as 2.6.26. Likewise, my laptop shows XFS
> being faster than ext3 on both 2.6.24 and 2.6.26. So the difference
> is something related to the disk subsystem on the server....

XFS definitely stalls somewhere: stats show virtually no CPU usage and no 
time spent waiting for IO. No file system produces similar output.

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 3146180   7848 600868    0    0     0  4128  790  549  0  2 98  0
 0  0      0 3145200   7848 601524    0    0     0  2372  766  516  0  2 98  0
 1  0      0 3144328   7848 602260    0    0     0  2924  792  542  1  2 98  0
 0  1      0 3143824   7856 602664    0    0     0  4116  732  426  0  2 53 45
 1  0      0 3143068   7856 603136    0    0     0  4676  756  534  0  3 95  1
 0  0      0 3142652   7856 603540    0    0     0  6577  756  436  0  0 100  0
 0  0      0 3141952   7856 604100    0    0     0  5840  764  498  1  3 96  0
 0  0      0 3141424   7856 604544    0    0     0  4752  761  386  0  0 99  0
 0  0      0 3140860   7856 604916    0    0     0  6477  785  495  0  1 98  0
 0  0      0 3139980   7856 605468    0    0     0  2840  743  370  1  2 97  0
 0  0      0 3138464   7856 606884    0    0     0  4902  795  421  0  4 96  0
 0  0      0 3137636   7856 607696    0    0     0  4364  739  395  0  1 99  0
 0  0      0 3136520   7856 608220    0    0     0  6160  774  566  0  2 97  0

	Szaka

-- 
NTFS-3G:  http://ntfs-3g.org

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Szabolcs Szakacsits]

On Thu, 21 Aug 2008, Szabolcs Szakacsits wrote:
> On Thu, 21 Aug 2008, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > > 
> > > One thing I just found out - my old *laptop* is 4-5x faster than the
> > > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > > if the long delays in dispatch is caused by an interaction with CTQ
> > > but I can't change it on the cciss raid controllers. Are you using
> > > ctq/ncq on your machine?  
> 
> It's a laptop and has NCQ. It makes no difference if NCQ is enabled or 
> disabled. The problem seems to be XFS only.

The 'nobarrier' mount option made a big improvement:

                    MB/s    Runtime (s)
                   -----    -----------
  btrfs unstable   17.09        572
  ext3             13.24        877
  btrfs 0.16       12.33        793
  nilfs2 2nd+ runs 11.29        674
  ntfs-3g           8.55        865
  reiserfs          8.38        966
  xfs nobarrier     7.89        949
  nilfs2 1st run    4.95       3800
  xfs               1.88       3901

	Szaka
 
-- 
NTFS-3G:  http://ntfs-3g.org

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 08:33:50PM +0300, Szabolcs Szakacsits wrote:
> 
> On Thu, 21 Aug 2008, Szabolcs Szakacsits wrote:
> > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > > > 
> > > > One thing I just found out - my old *laptop* is 4-5x faster than the
> > > > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > > > if the long delays in dispatch is caused by an interaction with CTQ
> > > > but I can't change it on the cciss raid controllers. Are you using
> > > > ctq/ncq on your machine?  
> > 
> > It's a laptop and has NCQ. It makes no difference if NCQ is enabled or 
> > disabled. The problem seems to be XFS only.
> 
> The 'nobarrier' mount option made a big improvement:
> 
>                     MB/s    Runtime (s)
>                    -----    -----------
>   btrfs unstable   17.09        572
>   ext3             13.24        877
>   btrfs 0.16       12.33        793
>   nilfs2 2nd+ runs 11.29        674
>   ntfs-3g           8.55        865
>   reiserfs          8.38        966
>   xfs nobarrier     7.89        949
>   nilfs2 1st run    4.95       3800
>   xfs               1.88       3901

INteresting. Barriers make only a little difference on my laptop;
10-20% slower. But yes, barriers will have this effect on XFS.

If you've got NCQ, then you'd do better to turn off write caching
on the drive, turn off barriers and use NCQ to give you back the
performance that the write cache used to. That is, of course,
assuming the NCQ implementation doesn't suck....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Martin Steigerwald]

Am Freitag 22 August 2008 schrieb Dave Chinner:
> On Thu, Aug 21, 2008 at 08:33:50PM +0300, Szabolcs Szakacsits wrote:
> > On Thu, 21 Aug 2008, Szabolcs Szakacsits wrote:
> > > On Thu, 21 Aug 2008, Dave Chinner wrote:
> > > > On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > > > > One thing I just found out - my old *laptop* is 4-5x faster
> > > > > than the 10krpm scsi disk behind an old cciss raid controller. 
> > > > > I'm wondering if the long delays in dispatch is caused by an
> > > > > interaction with CTQ but I can't change it on the cciss raid
> > > > > controllers. Are you using ctq/ncq on your machine?
> > >
> > > It's a laptop and has NCQ. It makes no difference if NCQ is enabled
> > > or disabled. The problem seems to be XFS only.
> >
> > The 'nobarrier' mount option made a big improvement:
> >
> >                     MB/s    Runtime (s)
> >                    -----    -----------
> >   btrfs unstable   17.09        572
> >   ext3             13.24        877
> >   btrfs 0.16       12.33        793
> >   nilfs2 2nd+ runs 11.29        674
> >   ntfs-3g           8.55        865
> >   reiserfs          8.38        966
> >   xfs nobarrier     7.89        949
> >   nilfs2 1st run    4.95       3800
> >   xfs               1.88       3901
>
> INteresting. Barriers make only a little difference on my laptop;
> 10-20% slower. But yes, barriers will have this effect on XFS.
>
> If you've got NCQ, then you'd do better to turn off write caching
> on the drive, turn off barriers and use NCQ to give you back the
> performance that the write cache used to. That is, of course,
> assuming the NCQ implementation doesn't suck....

See my other post with performance numbers:

Barriers appear to make more than 50% difference on my laptop for some 
operations on some other operations it hardly makes a difference at all - 
I bet it goes slow mainly when creating or deleting lots of small files. 
Looking at vmstat 1 during a rm -rf of a compilebench leftover directory 
while switching off barriers shows a difference of even more than 50% in 
metadata throughput.

It has this controller

00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller 
(rev 01)

and this drive

---------------------------------------------------------------------
shambhala:~> hdparm -I /dev/sda

/dev/sda:

ATA device, with non-removable media
        Model Number:       Hitachi HTS541616J9AT00
        Serial Number:      SB0442SJDVDDHH
        Firmware Revision:  SB4OA70H
Standards:
        Used: ATA/ATAPI-7 T13 1532D revision 1
        Supported: 7 6 5 4
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  312581808
        device size with M = 1024*1024:      152627 MBytes
        device size with M = 1000*1000:      160041 MBytes (160 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        Standby timer values: spec'd by Vendor, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: 254
        Recommended acoustic management value: 128, current value: 128
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=240ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
           *    Advanced Power Management feature set
                Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
                Address Offset Reserved Area Boot
           *    SET_MAX security extension
           *    Automatic Acoustic Management feature set
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
           *    IDLE_IMMEDIATE with UNLOAD
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
        not     supported: enhanced erase
        82min for SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000cca525da17b6
        NAA             : 5
        IEEE OUI        : cca
        Unique ID       : 525da17b6
HW reset results:
        CBLID- above Vih
        Device num = 0 determined by the jumper
---------------------------------------------------------------------

with libata driver which doesn't use FUA while its advertised above:

---------------------------------------------------------------------
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't 
support DPO or FUA
sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't 
support DPO or FUA
sd 0:0:0:0: [sda] Starting disk
---------------------------------------------------------------------

So AFAIK that should be without NCQ since its not a SATA drive and 
apparently its also without FUA (maybe due to controller?). Maybe the bad 
results are due to lack of NCQ and FUA?

Here the relevant parts from my other mail:

---------------------------------------------------------------------
With barriers on an already heavily populated filesystem - I don't have
an empty one on a raw partition at hand at the moment and I for sure
won't empty this one:

martin@shambhala:~> df -hT | grep /home
/dev/sda5      xfs    112G  104G  8,2G  93% /home

shambhala:~> df -hiT | grep /home
/dev/sda5      xfs       34M    751K     33M    3% /home

shambhala:~> xfs_db -rx /dev/sda5
xfs_db> frag
actual 726986, ideal 703687, fragmentation factor 3.20%
xfs_db> quit
shambhala:~>

martin@shambhala:~> cat /proc/mounts | grep "/home "
/dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0
0

shambhala:~> xfs_info /home
meta-data=/dev/sda5              isize=256    agcount=6, agsize=4883256
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=29299536,
imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D
/home/martin/Zeit/compilebench -i 5 -r 10
using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10
runs
native unpatched native-0 222MB in 117.37 seconds (1.89 MB/s)
native patched native-0 109MB in 27.46 seconds (3.99 MB/s)
native patched compiled native-0 691MB in 48.03 seconds (14.40 MB/s)
create dir kernel-0 222MB in 83.55 seconds (2.66 MB/s)
create dir kernel-1 222MB in 86.01 seconds (2.59 MB/s)
create dir kernel-2 222MB in 71.61 seconds (3.11 MB/s)
create dir kernel-3 222MB in 71.73 seconds (3.10 MB/s)
create dir kernel-4 222MB in 61.61 seconds (3.61 MB/s)
patch dir kernel-2 109MB in 63.14 seconds (1.74 MB/s)
compile dir kernel-2 691MB in 45.61 seconds (15.16 MB/s)
compile dir kernel-4 680MB in 50.13 seconds (13.58 MB/s)
patch dir kernel-4 691MB in 154.38 seconds (4.48 MB/s)
read dir kernel-4 in 95.04 9.65 MB/s
read dir kernel-3 in 49.49 4.49 MB/s
create dir kernel-3116 222MB in 79.44 seconds (2.80 MB/s)
clean kernel-4 691MB in 8.64 seconds (80.05 MB/s)
read dir kernel-1 in 71.40 3.11 MB/s
stat dir kernel-0 in 14.44 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 3.01 MB/s (user 2.34s sys 4.30s)
create total runs 1 avg 2.80 MB/s (user 2.36s sys 4.12s)
patch total runs 2 avg 3.11 MB/s (user 0.91s sys 4.07s)
compile total runs 2 avg 14.37 MB/s (user 0.60s sys 2.76s)
clean total runs 1 avg 80.05 MB/s (user 0.09s sys 0.45s)
read tree total runs 2 avg 3.80 MB/s (user 2.00s sys 4.05s)
read compiled tree total runs 1 avg 9.65 MB/s (user 2.36s sys 6.42s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 14.44 seconds (user 1.17s sys 1.07s)
no runs for stat compiled tree

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> rm -rf /home/martin/Zeit/compilebench

I didn't measure it, but it took *ages* while rm -rf was mostly in D
state. According to harddisk noise a lot of seeks where involved.

vmstat 1 during the rm -rf:

 0  0   2784 748048     20 247160    0    0   160  4628  352 1224 15 14
71  0
 0  0   2784 748056     20 247308    0    0   148  3848  298  442 11 10
79  0
 0  0   2784 747996     20 247428    0    0   120  3377  260  449  9  9
82  0
 0  0   2784 747764     20 247580    0    0   152  4364  324 1094 20 10
70  0
 1  0   2784 747452     20 247736    0    0   156  4356  279  814 15 11
74  0
 0  0   2784 747408     20 247900    0    0   164  4112  360 1131 13 13
74  0
 0  0   2784 747136     20 248064    0    0   164  5128  318  855 16 10
74  0
 0  0   2784 746780     20 248208    0    0   144  4353  305 1066 20 12
68  0
 0  0   2784 746204     20 248336    0    0   128  5388  275  966 14 11
75  0
 1  0   2784 748352     20 248468    0    0   132  5384  314 1234 22 11
67  0
 0  0   2784 748104     20 248604    0    0   136  4873  284  807 16 11
73  0

Same game on same productively used partition, but now without barriers:

shambhala:~> mount -o remount,nobarrier /home
shambhala:~> cat /proc/mounts | grep "/home "
/dev/sda5 /home xfs
rw,relatime,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> mkdir /home/martin/Zeit/compilebench

shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> ./compilebench -D
/home/martin/Zeit/compilebench -i 5 -r 10
using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10
runs
native unpatched native-0 222MB in 51.44 seconds (4.32 MB/s)
native patched native-0 109MB in 12.69 seconds (8.64 MB/s)
native patched compiled native-0 691MB in 51.75 seconds (13.36 MB/s)
create dir kernel-0 222MB in 47.64 seconds (4.67 MB/s)
create dir kernel-1 222MB in 53.40 seconds (4.16 MB/s)
create dir kernel-2 222MB in 48.04 seconds (4.63 MB/s)
create dir kernel-3 222MB in 38.26 seconds (5.81 MB/s)
create dir kernel-4 222MB in 34.15 seconds (6.51 MB/s)
patch dir kernel-2 109MB in 50.61 seconds (2.17 MB/s)
compile dir kernel-2 691MB in 37.94 seconds (18.23 MB/s)
compile dir kernel-4 680MB in 45.32 seconds (15.02 MB/s)
patch dir kernel-4 691MB in 107.27 seconds (6.45 MB/s)
read dir kernel-4 in 82.18 11.16 MB/s
read dir kernel-3 in 42.35 5.25 MB/s
create dir kernel-3116 222MB in 38.27 seconds (5.81 MB/s)
clean kernel-4 691MB in 5.92 seconds (116.82 MB/s)
read dir kernel-1 in 73.63 3.02 MB/s
stat dir kernel-0 in 13.77 seconds

run complete:
========================================================================
==
intial create total runs 5 avg 5.16 MB/s (user 2.21s sys 4.23s)
create total runs 1 avg 5.81 MB/s (user 2.18s sys 4.89s)
patch total runs 2 avg 4.31 MB/s (user 0.90s sys 4.05s)
compile total runs 2 avg 16.62 MB/s (user 0.59s sys 3.05s)
clean total runs 1 avg 116.82 MB/s (user 0.09s sys 0.41s)
read tree total runs 2 avg 4.14 MB/s (user 1.90s sys 4.02s)
read compiled tree total runs 1 avg 11.16 MB/s (user 2.28s sys 6.36s)
no runs for delete tree
no runs for delete compiled tree
stat tree total runs 1 avg 13.77 seconds (user 1.19s sys 1.01s)
no runs for stat compiled tree


Not as fast as on the clean XFS LV, but still almost everytime almost
twice as fast as with barriers.


shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/
compilebench/compilebench-0.6> time rm -rf
/home/martin/Zeit/compilebench
rm -rf /home/martin/Zeit/compilebench  0,32s user 19,19s system 15% cpu
2:09,79 total

This is definately faster than before. I didn't measure exact time on
first occasion, but it took ages.

vmstat 1 during the rm -rf indicated much higher metadata throughput:

 3  0   2780 827696     20 162492    0    0   280 11109  449  865 31 15
52  2
 0  0   2780 827304     20 162816    0    0   324  6656  468 1009 57  8 
7 28
 2  0   2636 828992     20 163364    0    0   540  5317  350  545 30 10
30 31
 2  1   2636 837488     20 164020    0    0   656  7691  394  650 39 12 
0 49
 0  0   2224 960360     20 164516    0    0   496 12060  420  549 13 26
56  5
 0  0   2224 959988     20 164904    0    0   388 13704  425  792 16 23
61  0
 0  0   2224 959864     20 165128    0    0   224  6209  363  503 12 10
78  0
 1  0   2224 959376     20 165540    0    0   412 14886  392  513 12 22
66  0

[...]

As last XFS thing:

vmstat 1 during a rm -rf while switching of XFS from nobarrier to
barrier:

 0  0   1976 422236   1784 516840    0    0   508 17160  410  540  7 23
70  0
 1  0   1976 420624   1784 517576    0    0   736 26904  539 1032 14 35
51  0
 0  0   1976 419176   1784 518152    0    0   576 23842  486 1060 17 33
50  0
 0  0   1976 418316   1784 518460    0    0   308 12812  317  552  6 18
76  0
 2  0   1976 417392   1784 518776    0    0   316 16689  360  882  2 23
75  0
 8  0   1976 432948   1784 519252    0    0   476 16710  452  630  8 39
53  0
 0  0   1976 432892   1784 519392    0    0   140  4146  371 1564 14 26
60  0
 0  0   1976 432628   1784 519572    0    0   180  3844  340  660 11 10
79  0
 0  0   1976 432496   1784 519736    0    0   164  3852  328  534  9  8
83  0
 0  0   1976 432372   1784 519920    0    0   176  4100  359  788 19 11
70  0

Its obvious, where it was switched to barrier ;)
---------------------------------------------------------------------

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Szabolcs Szakacsits]

On Fri, 22 Aug 2008, Dave Chinner wrote:
> On Thu, Aug 21, 2008 at 08:33:50PM +0300, Szabolcs Szakacsits wrote:
>
> > The 'nobarrier' mount option made a big improvement:
> 
> INteresting. Barriers make only a little difference on my laptop;
> 10-20% slower. But yes, barriers will have this effect on XFS.
> 
> If you've got NCQ, then you'd do better to turn off write caching
> on the drive, turn off barriers and use NCQ to give you back the
> performance that the write cache used to. That is, of course,
> assuming the NCQ implementation doesn't suck....

Write cache off, nobarrier and AHCI NCQ lowered the XFS result:

                               MB/s    Runtime (s)
                              -----    -----------
  btrfs unstable              17.09        572
  ext3                        13.24        877
  btrfs 0.16                  12.33        793
  ntfs-3g unstable            11.52        673
  nilfs2 2nd+ runs            11.29        674
  reiserfs                     8.38        966
  xfs nobarrier                7.89        949
  nilfs2 1st run               4.95       3800
  xfs nobarrier, ncq, wc off   3.81       1973
  xfs                          1.88       3901

	Szaka

-- 
NTFS-3G:  http://ntfs-3g.org

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Szabolcs Szakacsits]

On Fri, 22 Aug 2008, Szabolcs Szakacsits wrote:
> On Fri, 22 Aug 2008, Dave Chinner wrote:
> > On Thu, Aug 21, 2008 at 08:33:50PM +0300, Szabolcs Szakacsits wrote:
> >
> > > The 'nobarrier' mount option made a big improvement:
> > 
> > INteresting. Barriers make only a little difference on my laptop;
> > 10-20% slower. But yes, barriers will have this effect on XFS.
> > 
> > If you've got NCQ, then you'd do better to turn off write caching
> > on the drive, turn off barriers and use NCQ to give you back the
> > performance that the write cache used to. That is, of course,
> > assuming the NCQ implementation doesn't suck....
> 
> Write cache off, nobarrier and AHCI NCQ lowered the XFS result:
> 
>                                MB/s    Runtime (s)
>                               -----    -----------
>   btrfs unstable              17.09        572
>   ext3                        13.24        877
>   btrfs 0.16                  12.33        793
>   ntfs-3g unstable            11.52        673
>   nilfs2 2nd+ runs            11.29        674
>   reiserfs                     8.38        966
>   xfs nobarrier                7.89        949
>   nilfs2 1st run               4.95       3800
>   xfs nobarrier, ncq, wc off   3.81       1973
>   xfs                          1.88       3901

Retested with a different disk, SATA-II, NCQ, capable of 70-110 MB/s 
read/write:

                               MB/s    Runtime (s)
                              -----    -----------
  btrfs unstable, no dup      51.42        168
  btrfs unstable              42.67        197
  ext4 2.6.26                 35.63        245
  nilfs2 2nd+ runs            26.43        287
  ntfs-3g unstable            21.41        370
  ext3                        19.92        559
  xfs nobarrier               14.17        562
  reiserfs                    13.11        595
  nilfs2 1st run              12.06       3719
  xfs nobarrier, ncq, wc off   6.89       1070
  xfs                          1.95       3786

	Szaka

-- 
NTFS-3G:  http://ntfs-3g.org

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Matthew Wilcox]

On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> One thing I just found out - my old *laptop* is 4-5x faster than the
> 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> if the long delays in dispatch is caused by an interaction with CTQ
> but I can't change it on the cciss raid controllers. Are you using
> ctq/ncq on your machine?  If so, can you reduce the depth to
> something less than 4 and see what difference that makes?

I don't think that's going to make a difference when using CFQ.  I did
some tests that showed that CFQ would never issue more than one IO at a
time to a drive.  This was using sixteen userspace threads, each doing a
4k direct I/O to the same location.  When using noop, I would get 70k
IOPS and when using CFQ I'd get around 40k IOPS.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: XFS vs Elevators (was Re: [PATCH RFC] nilfs2: continuous snapshotting file system)

[Dave Chinner]

On Thu, Aug 21, 2008 at 05:53:10AM -0600, Matthew Wilcox wrote:
> On Thu, Aug 21, 2008 at 04:04:18PM +1000, Dave Chinner wrote:
> > One thing I just found out - my old *laptop* is 4-5x faster than the
> > 10krpm scsi disk behind an old cciss raid controller.  I'm wondering
> > if the long delays in dispatch is caused by an interaction with CTQ
> > but I can't change it on the cciss raid controllers. Are you using
> > ctq/ncq on your machine?  If so, can you reduce the depth to
> > something less than 4 and see what difference that makes?
> 
> I don't think that's going to make a difference when using CFQ.  I did
> some tests that showed that CFQ would never issue more than one IO at a
> time to a drive.  This was using sixteen userspace threads, each doing a
> 4k direct I/O to the same location.  When using noop, I would get 70k
> IOPS and when using CFQ I'd get around 40k IOPS.

Not obviously the same sort of issue. The traces clearly show
multiple nested dispatches and completions so CTQ is definitely
active...

Anyway, after a teeth-pulling equivalent exercise of finding the
latest firmware for the machine in a format I could apply, I
upgraded the firmware throughout the machine (disks, raid
controller, system, etc) and XFS is a *lot* faster. In fact -
mostly back to +/- a small amount compared to ext3.

run complete:
==========================================================================
				  avg MB/s       user       sys
			runs	 xfs   ext3    xfs ext3    xfs ext3
intial create total      30	6.36   6.29   4.48 3.79   7.03 5.22
create total              7	5.20   5.68   4.47 3.69   7.34 5.23
patch total               6	4.53   5.87   2.26 1.96   6.27 4.86
compile total             9    16.46   9.61   1.74 1.72   9.02 9.74
clean total               4   478.50 553.22   0.09 0.06   0.92 0.70
read tree total           2    13.07  15.62   2.39 2.19   3.68 3.44
read compiled tree        1    53.94  60.91   2.57 2.71   7.35 7.27
delete tree total         3    15.94s  6.82s  1.38 1.06   4.10 1.49
delete compiled tree      1    24.07s  8.70s  1.58 1.18   5.56 2.30
stat tree total           5	3.30s  3.22s  1.09 1.07   0.61 0.53
stat compiled tree total  3	2.93s  3.85s  1.17 1.22   0.59 0.55


The blocktrace looks very regular, too. All the big bursts of
dispatch and completion are gone as are the latencies on
log I/Os. It would appear that ext3 is not sensitive to
concurrent I/O latency like XFS is...

At this point, I'm still interested to know if the original
results were had ctq/ncq enabled and if it is whether it is
introducing latencies are not.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com