something's using space, can't figure out what - how to debug?

something's using space, can't figure out what - how to debug?

[Tomasz Chmielewski]

Here is one of my ext4 filesystems, which uses 2x more space as it
should:

# df -h /srv
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdf             194G  134G   51G  73% /srv

# du -sh /srv/
60G     /srv/

# lsof -n|grep /srv
# 


So df says 134 GB is used, while du says 60 GB is used.
No removed but used files, according to lsof.

How do I best debug what happened to ~70 GB?

This is EC2 kernel, nothing interesting in dmesg:

# uname -a
Linux ec2.agenteimovel.com.br 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7
18:19:02 UTC 2011 x86_64 GNU/Linux


-- 
Tomasz Chmielewski
http://www.ptraveler.com

Re: something's using space, can't figure out what - how to debug?

[Theodore Ts'o]

On Wed, Apr 10, 2013 at 07:56:06PM +0900, Tomasz Chmielewski wrote:
> # lsof -n|grep /srv
> # 
> 
> 
> So df says 134 GB is used, while du says 60 GB is used.
> No removed but used files, according to lsof.
> 
> How do I best debug what happened to ~70 GB?

Well, nothing is using /srv, so why don't you unmount it and run
"e2fsck -f /dev/xvdf", and see e2fsck reports in terms of the number
of blocks in use, i.e:

/dev/lambda/backup: 581320/4915200 files (0.1% non-contiguous), 6837767/19660800 blocks

		    		   	       - Ted

Re: something's using space, can't figure out what - how to debug?

[Tomasz Chmielewski]

On Wed, 10 Apr 2013 20:26:43 -0400
Theodore Ts'o <tytso@mit.edu> wrote:

> On Wed, Apr 10, 2013 at 07:56:06PM +0900, Tomasz Chmielewski wrote:
> > # lsof -n|grep /srv
> > # 
> > 
> > 
> > So df says 134 GB is used, while du says 60 GB is used.
> > No removed but used files, according to lsof.
> > 
> > How do I best debug what happened to ~70 GB?
> 
> Well, nothing is using /srv, so why don't you unmount it and run
> "e2fsck -f /dev/xvdf", and see e2fsck reports in terms of the number
> of blocks in use, i.e:
> 
> /dev/lambda/backup: 581320/4915200 files (0.1% non-contiguous),
> 6837767/19660800 blocks

If it was only so easy!

# lsof -n|grep /srv

# fuser /srv

# ls /sys/block/xvdf/holders/

# umount /srv
umount: /srv: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Yet after reboot, back to normal:

root@ec2:~# df -h /srv
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdf             194G   62G  123G  34% /srv


-- 
Tomasz Chmielewski
http://www.ptraveler.com