Re: btrfs kernel workqueues performance regression

[Dave Chinner <david@fromorbit.com>]

On Tue, Jul 15, 2014 at 01:39:11PM -0400, Chris Mason wrote:
> On 07/15/2014 11:26 AM, Morten Stevens wrote:
> > Hi,
> > 
> > I see that btrfs is using kernel workqueues since linux 3.15. After
> > some tests I noticed performance regressions with fs_mark.
> > 
> > mount options: rw,relatime,compress=lzo,space_cache
> > 
> > fs_mark on Kernel 3.14.9:
> > 
> > # fs_mark  -d  /mnt/btrfs/fsmark  -D  512  -t  16  -n  4096  -s  51200  -L5  -S0
> > FSUse%        Count         Size    Files/sec     App Overhead
> >      1        65536        51200      17731.4           723894
> >      1       131072        51200      16832.6           685444
> >      1       196608        51200      19604.5           652294
> >      1       262144        51200      18663.6           630067
> >      1       327680        51200      20112.2           692769
> > 
> > The results are really nice! compress=lzo performs very good.
> > 
> > fs_mark after upgrading to Kernel 3.15.4:
> > 
> > # fs_mark  -d  /mnt/btrfs/fsmark  -D  512  -t  16  -n  4096  -s  51200  -L5  -S0
> > FSUse%        Count         Size    Files/sec     App Overhead
> >      0        65536        51200      10718.1           749540
> >      0       131072        51200       8601.2           853050
> >      0       196608        51200      11623.2           558546
> >      0       262144        51200      11534.2           536342
> >      0       327680        51200      11167.4           578562
> > 
> > That's really a big performance regression :(
> > 
> > What do you think? It's easy to reproduce with fs_mark.
> 
> I wasn't able to trigger regressions here when we first merged it, but I
> was sure that something would pop up.  fs_mark is sensitive to a few
> different factors outside just the worker threads, so it could easily be
> another change as well.
> 
> With 16 threads, the btree locking also has a huge impact, and we've
> made change there too.

FWIW, I ran my usual 16-way fsmark test last week on my sparse 500TB
perf test rig on btrfs. It sucked, big time, much worse than it's
sucked in the past. It didn't scale past a single thread - 1 thread
got 24,000 files/s, 2 threads got 25,000 files/s 16 threads got
22,000 files/s.

$ ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  32  -d /mnt/scratch/0
....
FSUse%        Count         Size    Files/sec     App Overhead
     0       100000            0      24808.8           686583
....
$ ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  32  -d /mnt/scratch/0  -d  /mnt/scratch/1  -d  /mnt/scratch/2  -d /mnt/scratch/3  -d  /mnt/scratch/4  -d  /mnt/scratch/5  -d /mnt/scratch/6  -d  /mnt/scratch/7  -d  /mnt/scratch/8  -d /mnt/scratch/9  -d  /mnt/scratch/10  -d  /mnt/scratch/11  -d /mnt/scratch/12  -d  /mnt/scratch/13  -d  /mnt/scratch/14  -d /mnt/scratch/15
....
FSUse%        Count         Size    Files/sec     App Overhead
     0      1600000            0      23599.7         38047237

Last time I ran this (probably about 3.12 - btrfs was simply too
broken when I last tried on 3.14) I got about 80,000 files/s so this
is a pretty significant regression.

The 16-way run consumed most of the 16 CPUs in the system, and the
perf top output showed this:

+  44.48%  [kernel]  [k] _raw_spin_unlock_irqrestore
+  28.60%  [kernel]  [k] queue_read_lock_slowpath
+  14.34%  [kernel]  [k] queue_write_lock_slowpath
+   1.91%  [kernel]  [k] _raw_spin_unlock_irq
+   0.85%  [kernel]  [k] __do_softirq
+   0.45%  [kernel]  [k] do_raw_read_lock
+   0.43%  [kernel]  [k] do_raw_read_unlock
+   0.42%  [kernel]  [k] btrfs_search_slot
+   0.40%  [kernel]  [k] do_raw_spin_lock
+   0.35%  [kernel]  [k] btrfs_tree_read_unlock
+   0.33%  [kernel]  [k] do_raw_write_lock
+   0.30%  [kernel]  [k] btrfs_clear_lock_blocking_rw
+   0.29%  [kernel]  [k] btrfs_tree_read_lock

All the CPU time is basically spend in locking functions.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com