Re: [PATCH 4/5] writeback: avoid livelocking WB_SYNC_ALL writeback

[Jan Kara <jack@suse.cz>]

On Tue 09-11-10 14:43:46, Andrew Morton wrote:
> On Tue, 09 Nov 2010 07:09:20 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > From: Jan Kara <jack@suse.cz>
> > 
> > When wb_writeback() is called in WB_SYNC_ALL mode, work->nr_to_write is
> > usually set to LONG_MAX. The logic in wb_writeback() then calls
> > __writeback_inodes_sb() with nr_to_write == MAX_WRITEBACK_PAGES and thus
> > we easily end up with negative nr_to_write after the function returns.
> 
> No, nr_to_write can only be negative if the filesystem wrote back more
> pages than requested.
  Since some time, write_cache_pages() does not stop when nr_to_write
<= 0 in WB_SYNC_ALL mode as that is a possible data-integrity issue (we
could have written newly created pages but not the ones written before
sync was called). So nr_to_write gets negative rather easily in
WB_SYNC_ALL mode.
 
> > wb_writeback() then decides we need another round of writeback but this
> > is wrong in some cases! For example when a single large file is
> > continuously dirtied, we would never finish syncing it because each pass
> > would be able to write MAX_WRITEBACK_PAGES and inode dirty timestamp
> > never gets updated (as inode is never completely clean).
> 
> Well we shouldn't have asked the function to write LONG_MAX pages then!
> 
> The way this used to work was to try to write back N=(total dirty pages
> + total unstable pages + various fudge factors) to each superblock.  So
> each superblock will get fully written back unless someone is madly
> writing to it.  If that _is_ happening then we'll write a large amount
> of data to it and will then give up and move onto the next superblock.
> 
> But the "large amount of data" is constrained to a sane upper limit:
> total amount of dirty memory plus fudge factors.  Increasing that sane
> upper limit to an insane 2^63-1 pages will *of course* cause sync() to
> livelock.
> 
> Why was that sane->insane change made?
  Note that we are speaking about WB_SYNC_ALL mode and for above mentioned
data integrity reason any finite nr_to_write is just wrong... That's why we
do all that complex page tagging livelock avoidance thing in
write_cache_pages().

> > Fix the issue by setting nr_to_write to LONG_MAX in WB_SYNC_ALL mode. We
> > do not need nr_to_write in WB_SYNC_ALL mode anyway since livelock
> > avoidance is done differently for it.
> 
> Here the changelog should spell out what "done differently" means. 
> Because I really am unsure what is begin referred to.
> 
> I don't really see how this patch changes anything.  For WB_SYNC_ALL
> requests the code will still try to write out 2^63 pages, only it does
> it all in a single writeback_inodes_wb() call.  What prevents that call
> itself from getting livelocked?
  I'm referring to the livelock avoidance using page tagging. Fengguang
actually added a note about this into a comment in the code but it's not
in the changelog. And you're right it should be here.

> Perhaps the unmentioned problem here is that each call to
> writeback_inodes_wb(MAX_WRITEBACK_PAGES) will restart its walk across
> the inode lists.  So instead of giving up on a being-written-to-file,
> we continuously revisit it again and again and again.
> 
> Correct?  If so, please add the description.  If incorrect, please add
> the description as well ;)
  Yes, that's the problem.

> Root cause time: it's those damn per-sb inode lists *again*.  They're
> just awful.  We need some data structure there which is more amenable
> to being iterated over.  Something against which we can store cursors,
> for a start.
  This would be definitely nice. But in this particular case, since we have
that page tagging livelock avoidance, we can just do all we need in a one
big sweep so we are OK.

Suggestion for the new changelog:
When wb_writeback() is called in WB_SYNC_ALL mode, work->nr_to_write is
usually set to LONG_MAX. The logic in wb_writeback() then calls
__writeback_inodes_sb() with nr_to_write == MAX_WRITEBACK_PAGES and
we easily end up with negative nr_to_write after the function returns.
This is because write_cache_pages() does not stop writing when
nr_to_write drops to zero in WB_SYNC_ALL mode.

When nr_to_write is <= 0 wb_writeback() decides we need another round of
writeback but this is wrong in some cases! For example when a single large
file is continuously dirtied, we would never finish syncing it because each
pass would be able to write MAX_WRITEBACK_PAGES and inode dirty timestamp
never gets updated (as inode is never completely clean). Thus
__writeback_inodes_sb() would write the redirtied inode again and again.

Fix the issue by setting nr_to_write to LONG_MAX in WB_SYNC_ALL mode.  We
do not need nr_to_write in WB_SYNC_ALL mode anyway since
write_cache_pages() does livelock avoidance using page tagging in
WB_SYNC_ALL mode.

After this patch, program from http://lkml.org/lkml/2010/10/24/154 is no
longer able to stall sync forever.
-

Is this better?

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR