Re: long sleep_on_page delays writing to slow storage

[Josh Boyer <jwboyer@redhat.com>]

On Fri, Nov 18, 2011 at 11:11:40AM +0000, Mel Gorman wrote:
> On Thu, Nov 17, 2011 at 02:47:20PM -0500, Dave Jones wrote:
> > On Tue, Nov 15, 2011 at 10:13:13AM +0000, Mel Gorman wrote:
> >  
> >  > If they are still experiencing major stalls, I have an experimental
> >  > script that may be able to capture stack traces of processes stalled
> >  > for more than 1 second. I've had some success with it locally so
> >  > maybe they could try it out to identify if it's THP or something else.
> >  
> > I'm not sure if it's the same problem, but I'd be interested in trying
> > that script.
> > 
> 
> Monitor script is attached as watch-dstate.pl. Run it as
> 
> watch-dstate.pl -o logfile
> 
> I'm also attaching a post-processing script stap-dstate-frequency.
> 
> cat logfile | stap-dstate-frequency
> 
> will report on unique stack traces, what got stuck in them and
> for how long. Unfortunately, this does require a working systemtap
> installation because it had to work on systems without ftrace. Usually
> systemtap is a case of installing debugging symbols and its package
> but milage varies.
> 
> I ran this for a few days on my own desktop but found that the worst
> stalls for firefox and evolution were in futex_wait with the second
> worst in
> 
> [<ffffffffa018e3c5>] ext4_sync_file+0x225/0x290 [ext4]
> [<ffffffff81178250>] do_fsync+0x50/0x80
> [<ffffffff8117852e>] sys_fdatasync+0xe/0x20
> [<ffffffff81448592>] system_call_fastpath+0x16/0x1b
> 
> The stall timing is approximate at best. If you find the stall figures
> are way too high or unrealistic, try running with --accurate-stall. The
> stall figures will be much more accurate but depending on your
> kernel version, the stack traces may be one line long ending with
> kretprobe_trampoline.
> 
> > When I build a kernel on my laptop, when it gets to the final link stage,
> > and there's a ton of IO, my entire X session wedges for a few seconds.
> > This may be unrelated, because this is on an SSD, which shouldn't suffer
> > from the slow IO of the USB devices mentioned in this thread.
> > 
> 
> I have a vague suspicion that there are some interactivity issues
> around SSDs but I don't know why that is. I'm basing this on some
> complaints of audio skipping with heavy kernel compiles on machines
> very similar to my own other than mine uses rotary storage. It's on
> the Christmas list to by myself a SSD to take a closer look.

I see similar pauses on my laptop that doesn't have an SSD, so I don't
think it's drive related.  Certainly could be I/O scheduling or ext4
sync related, but probably not a specific drive technology.

I also wonder if the dirty_background_ratio and dirty_ratio might be too
high on this machine, and the flush of the page cache is causing
excessive I/O.

> > (This is even with that patch applied btw, perhaps adding further fuel to
> > the idea that it's unrelated).
> > 
> 
> I suspect it's not compaction related in that case but the script may be
> able to tell for sure. If it does not catch anything, alter this line to
> have a smaller threshold
> 
> global stall_threshold = 1000

I'll give this a try as well.