Re: Files appear too big in `du`

Dave Chinner
2011-05-10T13:17:12Z

On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> Hi
>
>
> Since a few weeks i'm experiencing an annoying 'thing' where files are
> often too big in `du` and directory totals are to high in `ls -l`.
>
> I appears that files, which are in the process of beeing
> copied/downloaded/whatever, grow in large chunks ahead of time, while
> the actual file-content is beeing copied into the files.

It's supposed to work like this. It's called speculative allocation
beyond end of file. XFS has always done this, but we've recently
made it more aggressive to prevent excessive fragmentation on
concurrent large file workloads when there is lots of disk space
free.

> And then it
> appears that the last chunk isn't shrunk after the process is finished.

It should be truncated away when the file descriptor is closed and
the last reference goes away.

> Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the
> extent that compromises the actual file-content.

what is the output of xfs_bmap -vvp on a file that apparently hasn't
been shrunk? How do you know it hasn't been shrunk? Does it persist
forever in this state, or does doing something like dropping caches
(echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
preallocation to disappear?

> Any idea how to debug this, or is this a known bug and waiting a few
> days for 2.6.39 should fix this?

It doesn't appear to be doing anything wrong from your description.
Remember that XFS is optimised for high end storage and server
configurations and workloads, not typical desktop usage...

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com