du accuracy

du accuracy

[Russell Coker]

I'm copying an image of a 2G microSD card to a BTRFS filesystem.  The SD card 
is from an Android phone and is almost full of Android packages and photos, 
there's very little space on the filesystem and what space there is probably 
isn't filled with zeros.

# ls -l
total 888000
-rw-r--r--. 1 root root 1043529728 Apr 10 11:20 2gsd

While doing it I run ls commands to see the progress and get output like the 
above.  Why is the file size according to ls not matching the total used?

# du -h
886M    .

When I ran du immediately afterwards it also reported that there was less than 
1G used.

# du -h
1.2G    .
# du -h
1.2G    .
# du -h
1.1G    .
# du -h
1.1G    .
# du -h
1.1G    .
# du -h
1.2G    .
# du -h
1.2G    .

Above are some consecutive du runs.  Why does the space used go from 1.2G to 
1.1G before going up again?  The file was created by "cat /dev/sde > 2gsd" so 
it definitely wasn't getting smaller.

What's going on here?

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: du accuracy

[Liu Bo]

On Fri, Apr 10, 2015 at 11:25:57AM +1000, Russell Coker wrote:
> I'm copying an image of a 2G microSD card to a BTRFS filesystem.  The SD card 
> is from an Android phone and is almost full of Android packages and photos, 
> there's very little space on the filesystem and what space there is probably 
> isn't filled with zeros.
> 
> # ls -l
> total 888000
> -rw-r--r--. 1 root root 1043529728 Apr 10 11:20 2gsd
> 
> While doing it I run ls commands to see the progress and get output like the 
> above.  Why is the file size according to ls not matching the total used?
> 
> # du -h
> 886M    .
> 
> When I ran du immediately afterwards it also reported that there was less than 
> 1G used.
> 
> # du -h
> 1.2G    .
> # du -h
> 1.2G    .
> # du -h
> 1.1G    .
> # du -h
> 1.1G    .
> # du -h
> 1.1G    .
> # du -h
> 1.2G    .
> # du -h
> 1.2G    .
> 
> Above are some consecutive du runs.  Why does the space used go from 1.2G to 
> 1.1G before going up again?  The file was created by "cat /dev/sde > 2gsd" so 
> it definitely wasn't getting smaller.
> 
> What's going on here?

What's your mount options?  with autodefrag or compression?

Thanks,

-liubo
> 
> -- 
> My Main Blog         http://etbe.coker.com.au/
> My Documents Blog    http://doc.coker.com.au/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: du accuracy

[Russell Coker]

On Fri, 10 Apr 2015, Liu Bo <bo.li.liu@oracle.com> wrote:
> > Above are some consecutive du runs.  Why does the space used go from 1.2G
> > to  1.1G before going up again?  The file was created by "cat /dev/sde >
> > 2gsd" so it definitely wasn't getting smaller.
> >
> > 
> >
> > What's going on here?
> 
> What's your mount options?  with autodefrag or compression?

Linux server 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt4-2 (2015-01-27) x86_64 
GNU/Linux

Above is the kernel I'm using, one of the recent ones from Debian/Jessie.

UUID=XXXX /big btrfs skip_balance,relatime,nosuid,nodev 0 0

Above is the /etc/fstab line.

/dev/sdb2 /big btrfs 
rw,seclabel,nosuid,nodev,relatime,space_cache,skip_balance 0 0

Above is the /proc/mounts line.  I'm not doing anything noteworthy here.  

# du -h tmp|tail -1 ; sleep 10 ; du -h tmp|tail -1 
1.6G    tmp
1.4G    tmp

I've just had something similar happen when rsyncing a directory full of files 
to the server and running du to check on the progress.  It's probably possible 
that a rename could happen at the wrong time to cause a file to be missed in 
the du count.  So this could be legit.  But the case described in my previous 
message concerned a single file that was being extended.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/