Re: Phantom full ext4 root filesystems on 4.1 through 4.14 kernels

[Jan Kara <jack@suse.cz>]

On Wed 26-06-19 11:17:54, Thomas Walker wrote:
> Sorry to revive a rather old thread, but Elana mentioned that there might
> have been a related fix recently?  Possibly something to do with
> truncate?  A quick scan of the last month or so turned up
> https://www.spinics.net/lists/linux-ext4/msg65772.html but none of these
> seemed obviously applicable to me.  We do still experience this phantom
> space usage quite frequently (although the remount workaround below has
> lowered the priority). 

I don't recall any fix for this. But seeing that remount "fixes" the issue
for you can you try whether one of the following has a similar effect?

1) Try "sync"
2) Try "fsfreeze -f / && fsfreeze -u /"
3) Try "echo 3 >/proc/sys/vm/drop_caches"

Also what is the contents of
/sys/fs/ext4/<problematic-device>/delayed_allocation_blocks
when the issue happens?

								Honza

> 
> On Wed, Jan 23, 2019 at 02:59:22PM -0500, Thomas Walker wrote:
> > Unfortunately this still continues to be a persistent problem for us.  On another example system:
> > 
> > # uname -a
> > Linux <hostname> 4.14.67-ts1 #1 SMP Wed Aug 29 13:28:25 UTC 2018 x86_64 GNU/Linux
> > 
> > # df -h /
> > Filesystem                                              Size  Used Avail Use% Mounted on
> > /dev/disk/by-uuid/<uuid>                                 50G   46G  1.1G  98% /
> > 
> > # df -hi /
> > Filesystem                                             Inodes IUsed IFree IUse% Mounted on
> > /dev/disk/by-uuid/<uuid>                                 3.2M  306K  2.9M   10% /
> > 
> > # du -hsx  /
> > 14G     /
> > 
> > And confirmed not to be due to sparse files or deleted but still open files.
> > 
> > The most interesting thing that I've been able to find so far is this:
> > 
> > # mount -o remount,ro /
> > mount: / is busy
> > # df -h /
> > Filesystem                                              Size  Used Avail Use% Mounted on
> > /dev/disk/by-uuid/<uuid>                                 50G   14G   33G  30% /
> > 
> > Something about attempting (and failing) to remount read-only frees up all of the phantom space usage.
> > Curious whether that sparks ideas in anyone's mind?
> > 
> > I've tried all manner of other things without success.  Unmounting all of the overlays.  Killing off virtually all of usersapce (dropping to single user).  Dropping page/inode/dentry caches.Nothing else (short of a reboot) seems to give us the space back.
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR