Re: Hot rb_next, setup_cluster_no_bitmap

[Simon Kirby <sim@hostway.ca>]

On Thu, Aug 04, 2011 at 10:04:29AM -0400, Chris Mason wrote:

> Excerpts from Simon Kirby's message of 2011-08-03 21:32:10 -0400:
> > Perhaps as a further clue as to what is going on, on this same backup box
> > after all of the rsyncs are finished/killed and a good amount of time has
> > passed (no cleaner processes running in the background or anything),
> > "sync" is still consistently takes ~4 minutes to run, and pushes out a
> > lot to disk every time it is run. Example:
> > 
> > echo 3 > /proc/sys/vm/drop_caches
> > sync
> > echo 3 > /proc/sys/vm/drop_caches
> > sync
> > vmstat 1 &
> > time sync
> 
> Just to confirm, the profiling during this time shows your system time
> is all in rb_next?

Correct, rb_next() called from setup_cluster_no_bitmap(). Also, much of the
other CPU time is spent spinning (on refill_lock, I think).

It's in this state again now (just one overnight backup run does it). It
doesn't seem to happen for the first few hours, but shows up by the next
morning.

Simon-