Re: CFQ slower than NOOP with pgbench

[Jan Kara <jack@suse.cz>]

On Thu 11-02-10 09:40:33, Nikanth Karthikesan wrote:
> On Thursday 11 February 2010 04:02:55 Jan Kara wrote:
> >   Hi,
> > 
> >   I was playing with a pgbench benchmark - it runs a series of operations
> > on top of PostgreSQL database. I was using:
> >   pgbench -c 8 -t 2000 pgbench
> > which runs 8 threads and each thread does 2000 transactions over the
> > database. The funny thing is that the benchmark does ~70 tps (transactions
> > per second) with CFQ and ~90 tps with a NOOP io scheduler. This is with
> > 2.6.32 kernel.
> >   The load on the IO subsystem basically looks like lots of random reads
> > interleaved with occasional short synchronous sequential writes (the
> > database does write immediately followed by fdatasync) to the database
> > logs. I was pondering for quite some time why CFQ is slower and I've tried
> > tuning it in various ways without success. What I found is that with NOOP
> > scheduler, the fdatasync is like 20-times faster on average than with CFQ.
> > Looking at the block traces (available on request) this is usually because
> > when fdatasync is called, it takes time before the timeslice of the process
> > doing the sync comes (other processes are using their timeslices for reads)
> > and writes are dispatched... The question is: Can we do something about
> > that? Because I'm currently out of ideas except for hacks like "run this
> > queue immediately if it's fsync" or such...
> 
> I guess, noop would be hurting those reads which is also a synchronous 
> operation like fsync. But it doesn't seem to have a huge negative impact on 
> the pgbench. Is it because reads are random in this benchmark and delaying 
> them might even help by getting new requests for sectors in between two random 
> reads? If that is the case, I dont think fsync should be given higher priority 
> than reads based on this benchmark.
> 
> Can you make the blktrace available?
  OK, traces are available from:
http://beta.suse.com/private/jack/pgbench-cfq-noop/pgbench-blktrace.tar.gz

  I've tried also two tests: I've run the database with LD_PRELOAD so that
fdatasync does
a) nothing
b) calls sync_file_range(fd, 0, LLONG_MAX, SYNC_FILE_RANGE_WRITE)
c) calls posix_fadvise(fd, 0, LLONG_MAX, POSIX_FADV_DONTNEED)
   - it does filemap_flush() which was my main aim..

  The results (CFQ as a IO scheduler) are interesting. In a) the performance
was slightly higher than with NOOP scheduler and fully functional fdatasync.
Not surprising - we spend only like 2 s (out of ~200) in fdatasync with NOOP
scheduler.
  In b) the performance was only about 2% better than with full fdatasync
(with NOOP scheduler, it's ~20% better). Looking at the strace
output, it seems sync_file_range() takes as long as fdatasync() took -
probably because we are waiting for PageWriteback or lock_page.
  In c) the performance was ~11% better - fadvise calls seem to be quite
quick - comparable times between CFQ and NOOP. So higher latency of fdatasync
seems to be at least part of a problem...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR