[regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU

[regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU

[Dave Chinner]

Hi Peter,

I'm guessing that x86 vdso problems are in your area of expertise,
if not can you point me at the right person to bug?

I just spun up my XFS performance tests on 3.15-rc1, the first of
which runs a concurrent fsmark workload. fsmark uses gettimeofday to
do per-operation timing and performance is horrendous. 3.14.0 on this
workload does around 300,000 file creates per second. 3.15-rc1
does:

#  ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  32  -d  /mnt/scratch/0  -d  /mnt/scratch/1  -d  /mnt/scratch/2  -d  /mnt/scratch/3  -d  /mnt/scratch/4  -d  /mnt/scratch/5  -d  /mnt/scratch/6  -d  /mnt/scratch/7  -d  /mnt/scratch/8  -d  /mnt/scratch/9  -d  /mnt/scratch/10  -d  /mnt/scratch/11  -d  /mnt/scratch/12  -d  /mnt/scratch/13  -d  /mnt/scratch/14  -d  /mnt/scratch/15
#       Version 3.3, 16 thread(s) starting at Tue Apr 15 17:37:20 2014
#       Sync method: NO SYNC: Test does not issue sync() or fsync() calls.
#       Directories:  Time based hash between directories across 10000 subdirectories with 180 seconds per subdirectory.
#       File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
#       Files info: size 0 bytes, written with an IO size of 16384 bytes per write
#       App overhead is time in microseconds spent in the test not doing file writing related system calls.

FSUse%        Count         Size    Files/sec     App Overhead
     0      1600000            0      70950.5        155219877
     0      3200000            0      71832.4        150138649
     0      4800000            0      71195.5        150935842
     0      6400000            0      71215.4        150906768
     0      8000000            0      63707.7        166569461
.....

about 70,000 files/sec, so it's way, way down on 3.14. The load has
pegged all 16 CPUs in the VM, and perf top tells me:

-  63.11%  [vdso]              [.] __vdso_gettimeofday
   - __vdso_gettimeofday
        44.24% stop
        28.18% do_run
        13.98% write_file
        13.60% setup_file_name

that it's the gettimeofday calls that are causing the excessive CPU
load.

I haven't changed anything in userspace for the past couple of
months, the only thing that has changed is the kernel. I haven't
looked at this any further and have no idea what to do to debug it
further, so let me know if you need more information....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU

[Dave Chinner]

On Tue, Apr 15, 2014 at 05:53:36PM +1000, Dave Chinner wrote:
> Hi Peter,
> 
> I'm guessing that x86 vdso problems are in your area of expertise,
> if not can you point me at the right person to bug?

And you can ignore it. The VM wasn't running the kernel I thought it
was - it was an old kernel from half way through last week. A
3.15-rc1 kernel, which I'm now testing, doesn't show the regression:

FSUse%        Count         Size    Files/sec     App Overhead
     0      1600000            0     281779.0          9964735
     0      3200000            0     280343.0         10174512
     0      4800000            0     268615.2         11330160
     0      6400000            0     264782.4         11072193
.....

It shows all sorts of interesting new inode cache reclaim imbalances,
but other than that the peak performance is back to normal...

Sorry for the noise.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com