Re: Files appear too big in `du`

Matthias Schniedermeyer
2011-05-10T15:33:19Z

On 10.05.2011 23:17, Dave Chinner wrote:
> On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> > Hi
> >
> >
> > Since a few weeks i'm experiencing an annoying 'thing' where files are
> > often too big in `du` and directory totals are to high in `ls -l`.
> >
> > I appears that files, which are in the process of beeing
> > copied/downloaded/whatever, grow in large chunks ahead of time, while
> > the actual file-content is beeing copied into the files.
>
> It's supposed to work like this. It's called speculative allocation
> beyond end of file. XFS has always done this, but we've recently
> made it more aggressive to prevent excessive fragmentation on
> concurrent large file workloads when there is lots of disk space
> free.

OK.

> > And then it
> > appears that the last chunk isn't shrunk after the process is finished.
>
> It should be truncated away when the file descriptor is closed and
> the last reference goes away.
>
> > Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the
> > extent that compromises the actual file-content.
>
> what is the output of xfs_bmap -vvp on a file that apparently hasn't
> been shrunk? How do you know it hasn't been shrunk? Does it persist

du

> forever in this state, or does doing something like dropping caches
> (echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
> preallocation to disappear?

This works:
sync ; echo 3 > /proc/sys/vm/drop_caches

At least in several tries the `du` output shrunk to the size of the
original.

> > Any idea how to debug this, or is this a known bug and waiting a few
> > days for 2.6.39 should fix this?
>
> It doesn't appear to be doing anything wrong from your description.
> Remember that XFS is optimised for high end storage and server
> configurations and workloads, not typical desktop usage...

I would call it a regression.
I reguarly follow copying/downloading with `du`, the speculative
preallocation makes that more or less useless. Especially downloading
someting big from the internet which @ 231kb/s isn't exactly fast and
shows identical `du`s for increasingly longer periods of time.
(Or "--apparent-size" should be made default, but that falls short with
sparse-files)

IMHO `du`/`ls -l` should not be able to 'see' the speculative
preallocation.