Re: HUGE slowdown when doing dpkg with ext4 over nbd

[Dave Chinner <david@fromorbit.com>]

On Fri, Dec 09, 2016 at 01:28:05PM -0700, Andreas Dilger wrote:
> On Dec 8, 2016, at 6:25 PM, Dave Chinner <david@fromorbit.com> wrote:
> > 
> > On Wed, Dec 07, 2016 at 07:34:17PM +0100, Sven Joachim wrote:
> >> On 2016-12-07 11:16 -0700, Andreas Dilger wrote:
> >> 
> >>> Add debian-dpkg mailing list to CC.
> >>> 
> >>> On Dec 7, 2016, at 10:58 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> >>>> 
> >>>> On Dec 7, 2016, at 2:52 AM, Renaud Mariana <rmariana@online.net> wrote:
> >>>>> 
> >>>>> Here are my answers, hope it will help solve this issue, thanks.
> >>>>> 
> >>>>> Recap:
> >>>>> dpkg kibana on ext4 over a nbd device takes 10 minutes
> >>>>> with xfs it's only 30s.
> >>>>> with ext4 no extends only 30s.
> >>>>> 
> >>>>> 
> >>>>> kernels :
> >>>>> 4.5.7 has this issue as older kernel like 4.4.34
> >>>>> The issue is also when nbd client & server run on same host
> >>>>> 
> >>>>> 
> >>>>> How small are the files?
> >>>>> here is the histogram of file sizes : http://pasteboard.co/6HC3nKyk2.png
> >>>>> We can see 5000 files around 512 Bytes.
> >>>> 
> >>>> Definitely there is no value to use fallocate for 512-byte files, or any
> >>>> of the files that can be written in a single write() syscall.  I'd expect
> >>>> any reasonable tool to be using a write buffer of at least 2-4MB these
> >>>> days to get good performance, so writes below the buffer size shouldn't
> >>>> use fallocate() at all.
> >> 
> >> It should be noted that the latest dpkg (1.18.15) only uses fallocate
> >> for files which are at least 16 KiB in size[1], so it would be nice if
> >> Renaud could recheck with that version, or cherry-pick the patch into
> >> whatever version he uses.
> > 
> > The fallocate() call should be removed completely. Applications
> > should not be attempting to control file allocation like this as it
> > defeats all the optimisations that filesystems use to optimise IO
> > patterns and minimise filesystem fragmentation (e.g. delayed
> > allocation).
> > 
> > There is /rarely/ a need for applications to use fallocate() to
> > manage fragmentation - especailly as excessive use of fallocate()
> > actively harms performance and accelerates filesystem aging.
> > 
> > Unless an application has a specific, repeatable performance problem
> > due to file fragmentation, it should not be using fallocate() to
> > allocate file space.
> 
> I'm not sure I'd go so far as to say that fallocate() should be removed
> completely.  Isn't that the best (only) way for an application to tell
> the filesystem that it is about to write a file of X size

That's most definitely not what preallocation is for. Filesystems
optimise the "growing file via sequential writes at EOF" case just
fine - using fallocate for this sort of thing is simply defeats all
the writeback optimisations and improvements we've developed over
the past 20 years for this /very common/ workload...

> and try to
> find a suitable amount of free space for it?

fallocate() does give a guarantee than a subsequent write won't
ENOSPC, but "suitable" is very dependent on context. This contenxt
is something
applications don't have - they have no idea what allocation
optimisations are required to provide fast, efficient IO, and have
no idea that different filesystems will require /different
optimisations/. 

e.g. btrfs will probably also suffer horribly under fallocate usage
like what dpkg is doing, and I can tell you for certain it will make
a mess of XFS filesystems, too....

> Otherwise, if the file
> is large and/or written slowly and/or the system has memory pressure
> the filesystem (even with delalloc) can't make a good decision about
> allocation.

None of which are the case for dpkg. Nor is it the case for /most
applications/. And fallocate actually makes memory pressure
problems worse, because it defeats writeback optimisations to
maximise dirty page cleaning rates...

Preallocation is *not a general purpose tool*. It's for applications
that have performance problems caused by known, repeatable
fragmentation or file layout issue.

> However, fallocate() won't really help if the file size
> is small (e.g. a few MB) since that can easily fit into RAM and will
> be written to disk in a single chunk.

In my experience, the list of "where fallocate is harmful" is quite
a bit larger than the list of "where fallocate is beneficial". This
is just one example of where it's harmful. 

-Dave.
-- 
Dave Chinner
david@fromorbit.com