Re: Poor interactive performance with I/O loads with fsync()ing

[Ben Gamari <bgamari.foss@gmail.com>]

On Wed, 17 Mar 2010 10:37:04 +0100, Ingo Molnar <mingo@elte.hu> wrote:
> * Nick Piggin <npiggin@suse.de> wrote:
> 
> > Hi,
> > 
> > On Tue, Mar 16, 2010 at 08:31:12AM -0700, Ben Gamari wrote:
> > > Hey all,
> > > 
> > > Recently I started using the Xapian-based notmuch mail client for everyday
> > > use.  One of the things I was quite surprised by after the switch was the
> > > incredible hit in interactive performance that is observed during database
> > > updates. Things are particularly bad during runs of 'notmuch new,' which scans
> > > the file system looking for new messages and adds them to the database.
> > > Specifically, the worst of the performance hit appears to occur when the
> > > database is being updated.
> > > 
> > > During these periods, even small chunks of I/O can become minute-long ordeals.
> > > It is common for latencytop to show 30 second long latencies for page faults
> > > and writing pages.  Interactive performance is absolutely abysmal, with other
> > > unrelated processes feeling horrible latencies, causing media players,
> > > editors, and even terminals to grind to a halt.
> > > 
> > > Despite the system being clearly I/O bound, iostat shows pitiful disk
> > > throughput (700kByte/second read, 300 kByte/second write). Certainly this poor
> > > performance can, at least to some degree, be attributable to the fact that
> > > Xapian uses fdatasync() to ensure data consistency. That being said, it seems
> > > like Xapian's page usage causes horrible thrashing, hence the performance hit
> > > on unrelated processes.
> > 
> > Where are the unrelated processes waiting? Can you get a sample of several 
> > backtraces?  (/proc/<pid>/stack should do it)
> 
> A call-graph profile will show the precise reason for IO latencies, and their 
> relatively likelihood.
> 
Are these backtraces at all useful? I've received no feedback thusfar, so I can
only that either,
  a) there is insufficient data to draw any conclusions and there is no
     interest in pursuing this further, or
  b) nobody has looked at the backtraces

As I've said in the past, I am very interested in seeing this problem looked at and
would love to contribute whatever I can to that effort. However, without knowing what
information is necessary, I can be of only very limited use in my own debugging
efforts. Thanks,

- Ben