Files appear too big in `du`

Files appear too big in `du`

[Matthias Schniedermeyer]

Hi


Since a few weeks i'm experiencing an annoying 'thing' where files are 
often too big in `du` and directory totals are to high in `ls -l`.

I appears that files, which are in the process of beeing 
copied/downloaded/whatever, grow in large chunks ahead of time, while 
the actual file-content is beeing copied into the files. And then it 
appears that the last chunk isn't shrunk after the process is finished.
Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the 
extent that compromises the actual file-content.

I've noticed this at least with:
- "git gc"
- cp -a
- rsync
- downloads with firefox (technically Iceweasel)

Kernel is currently 2.6.38.5, Distribution is an up-to-date Debian-SID.
mount is with default-parameters, except "noatime".

Any idea how to debug this, or is this a known bug and waiting a few 
days for 2.6.39 should fix this?




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Files appear too big in `du`

[Dave Chinner]

On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> Hi
> 
> 
> Since a few weeks i'm experiencing an annoying 'thing' where files are 
> often too big in `du` and directory totals are to high in `ls -l`.
> 
> I appears that files, which are in the process of beeing 
> copied/downloaded/whatever, grow in large chunks ahead of time, while 
> the actual file-content is beeing copied into the files.

It's supposed to work like this. It's called speculative allocation
beyond end of file. XFS has always done this, but we've recently
made it more aggressive to prevent excessive fragmentation on
concurrent large file workloads when there is lots of disk space
free.

> And then it 
> appears that the last chunk isn't shrunk after the process is finished.

It should be truncated away when the file descriptor is closed and
the last reference goes away.

> Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the 
> extent that compromises the actual file-content.

what is the output of xfs_bmap -vvp on a file that apparently hasn't
been shrunk? How do you know it hasn't been shrunk? Does it persist
forever in this state, or does doing something like dropping caches
(echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
preallocation to disappear?

> Any idea how to debug this, or is this a known bug and waiting a few 
> days for 2.6.39 should fix this?

It doesn't appear to be doing anything wrong from your description.
Remember that XFS is optimised for high end storage and server
configurations and workloads, not typical desktop usage...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Files appear too big in `du`

[Matthias Schniedermeyer]

On 10.05.2011 23:17, Dave Chinner wrote:
> On Tue, May 10, 2011 at 12:57:00PM +0200, Matthias Schniedermeyer wrote:
> > Hi
> > 
> > 
> > Since a few weeks i'm experiencing an annoying 'thing' where files are 
> > often too big in `du` and directory totals are to high in `ls -l`.
> > 
> > I appears that files, which are in the process of beeing 
> > copied/downloaded/whatever, grow in large chunks ahead of time, while 
> > the actual file-content is beeing copied into the files.
> 
> It's supposed to work like this. It's called speculative allocation
> beyond end of file. XFS has always done this, but we've recently
> made it more aggressive to prevent excessive fragmentation on
> concurrent large file workloads when there is lots of disk space
> free.

OK.

> > And then it 
> > appears that the last chunk isn't shrunk after the process is finished.
> 
> It should be truncated away when the file descriptor is closed and
> the last reference goes away.
>
> > Neither xfs_bmap (Version 3.1.5) nor filefrag show anything beyond the 
> > extent that compromises the actual file-content.
> 
> what is the output of xfs_bmap -vvp on a file that apparently hasn't
> been shrunk? How do you know it hasn't been shrunk? Does it persist

du

> forever in this state, or does doing something like dropping caches
> (echo 3 > /proc/sys/vm/drop_caches) cause the specualtive
> preallocation to disappear?

This works:
sync ; echo 3 > /proc/sys/vm/drop_caches

At least in several tries the `du` output shrunk to the size of the 
original.

> > Any idea how to debug this, or is this a known bug and waiting a few 
> > days for 2.6.39 should fix this?
> 
> It doesn't appear to be doing anything wrong from your description.
> Remember that XFS is optimised for high end storage and server
> configurations and workloads, not typical desktop usage...

I would call it a regression.
I reguarly follow copying/downloading with `du`, the speculative
preallocation makes that more or less useless. Especially downloading 
someting big from the internet which @ 231kb/s isn't exactly fast and 
shows identical `du`s for increasingly longer periods of time.
(Or "--apparent-size" should be made default, but that falls short with 
sparse-files)

IMHO `du`/`ls -l` should not be able to 'see' the speculative 
preallocation.




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Files appear too big in `du`

[Matthias Schniedermeyer]

On 10.05.2011 17:33, Matthias Schniedermeyer wrote:
> 
> > > Any idea how to debug this, or is this a known bug and waiting a few 
> > > days for 2.6.39 should fix this?
> > 
> > It doesn't appear to be doing anything wrong from your description.
> > Remember that XFS is optimised for high end storage and server
> > configurations and workloads, not typical desktop usage...
> 
> I would call it a regression.
> I reguarly follow copying/downloading with `du`, the speculative
> preallocation makes that more or less useless. Especially downloading 
> someting big from the internet which @ 231kb/s isn't exactly fast and 
> shows identical `du`s for increasingly longer periods of time.
> (Or "--apparent-size" should be made default, but that falls short with 
> sparse-files)
> 
> IMHO `du`/`ls -l` should not be able to 'see' the speculative 
> preallocation.


After digging into the log of v2.6.37..v2.6.38 i stumbled upon:

- snip -
    The allocsize mount option turns off the dynamic behaviour and fixes
    the prealloc size to whatever the mount option specifies. i.e. the
    behaviour is unchanged.
- snip -

I think Documentation/filesystems/xfs.txt is in need of an update. All 
that information in the commit-log is a little "out-of-reach" for most 
people.






Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Files appear too big in `du`

[Dave Chinner]

On Thu, May 12, 2011 at 12:01:53PM +0200, Matthias Schniedermeyer wrote:
> On 10.05.2011 17:33, Matthias Schniedermeyer wrote:
> > 
> > > > Any idea how to debug this, or is this a known bug and waiting a few 
> > > > days for 2.6.39 should fix this?
> > > 
> > > It doesn't appear to be doing anything wrong from your description.
> > > Remember that XFS is optimised for high end storage and server
> > > configurations and workloads, not typical desktop usage...
> > 
> > I would call it a regression.
> > I reguarly follow copying/downloading with `du`, the speculative
> > preallocation makes that more or less useless.

There's no guarantee that what du reports is in any way relevant to
the size of the file that is being written. e.g. the filesystem
might be compressing the file, doing inline deduplication,
speculative preallocation, filling holes with preallocated space,
etc. Indeed, XFS reports delayed allocation reservations in du -
blocks that haven't even been allocated yet - but it's always done
that and the behaviour you describe is what you always seen when
using the allocsize mount option....

In essence, a filesystem is free to allocate blocks in any way it
desires - you cannot rely on different filesystems to behave the
same way, and even different releases of the same filesystem to
behave the same way. It's just a bad assumption to make.

> > Especially downloading 
> > someting big from the internet which @ 231kb/s isn't exactly fast and 
> > shows identical `du`s for increasingly longer periods of time.
> > (Or "--apparent-size" should be made default, but that falls short with 
> > sparse-files)

Use an application that shows download progress?

> > IMHO `du`/`ls -l` should not be able to 'see' the speculative 
> > preallocation.

ls -l cannot see speculative preallocation beyond EOF of any kind.
Never has, never will - it only reports the file size, not the
number of blocks allocated.  du _should_ report it because it is
supposed to report the number of blocks allocated to the inode.
IOWs, they report two different things, so they should have
different behaviour....

> After digging into the log of v2.6.37..v2.6.38 i stumbled upon:
> 
> - snip -
>     The allocsize mount option turns off the dynamic behaviour and fixes
>     the prealloc size to whatever the mount option specifies. i.e. the
>     behaviour is unchanged.
> - snip -
> 
> I think Documentation/filesystems/xfs.txt is in need of an update. All 
> that information in the commit-log is a little "out-of-reach" for most 
> people.

Patches welcome.

> -- 
> Real Programmers consider "what you see is what you get" to be just as 
> bad a concept in Text Editors as it is in women. No, the Real Programmer
> wants a "you asked for it, you got it" text editor -- complicated, 
> cryptic, powerful, unforgiving, dangerous.

s/text editor/file system/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs