large directory entry

large directory entry

[Christian Kujau]

Hi,

I have a somewhat larger "directory entry" on this ext4 filesystem, and it 
takes quite some time to list its contents:

------------------------------------
# time ls -lah /var/lib/php5
total 18M
drwxrwx--T.  6 root     www-data  18M Apr 28 04:41 .
drwxr-xr-x. 48 root     root     4.0K Apr 28 06:03 ..
drwxr-xr-x.  5 root     root     4.0K Apr 28 04:41 modules
drwxr-xr-x.  2 www-data www-data 4.0K Oct  9  2014 owncloud-51d9c764244bc
drwxr-xr-x.  2 www-data www-data 4.0K Jan  6 23:27 owncloud-oc55674097b9
drwx-wx-wt.  2 root     root      92K May  1 01:14 sessions

real    0m39.292s
user    0m0.000s
sys     0m3.964s
------------------------------------

Notice the "18M" entry for "." - I realize this is a directory for 
temporary files, meaning that lots of files are generated here, but
the server is not _that_ busy; according to lsof(8) no files are
currently open in /var/lib/php5 and only the "sessions" directory
contains 100 files, there are far less files below the other directories.

Once the above is run, the next run of "ls" is fast, of course:

------------------------------------
# time ls -Rl /var/lib/php5 | wc -l
119

real    0m0.017s
user    0m0.004s
sys     0m0.008s
------------------------------------

There isn't really much content here, copying the structure to a new 
directory results in ~450 KB of data.

I feel like this is some kind of FAQ, but I could not find anything 
similar for Ext4. The only similar thing I could think of is an older 
bug[0] in JFS, where a directory slowly grows when many files are 
generated & deleted, but I could not reproduce this for Ext4.

The /var/lib/php5 directory from above is located on an Ext4 filesystem 
from 2010, now running a 3.16 kernel (Debian/jessie, amd64), with the 
following features as per tune2fs(8):

  Filesystem features:      has_journal ext_attr resize_inode dir_index
       filetype needs_recovery extent sparse_super large_file uninit_bg
  Filesystem flags:         signed_directory_hash

Mount options:

  /dev/sda1 on / type ext4 (rw,relatime,data=ordered)


Any pointers what's going on here?

Thanks,
Christian.

[0] https://bugzilla.kernel.org/show_bug.cgi?id=15154
-- 
BOFH excuse #268:

Neutrino overload on the nameserver

Re: large directory entry

[Theodore Ts'o]

On Thu, Apr 30, 2015 at 10:02:58PM -0700, Christian Kujau wrote:
> 
> I have a somewhat larger "directory entry" on this ext4 filesystem, and it 
> takes quite some time to list its contents:
> 
> ------------------------------------
> # time ls -lah /var/lib/php5
> total 18M
> drwxrwx--T.  6 root     www-data  18M Apr 28 04:41 .
> drwxr-xr-x. 48 root     root     4.0K Apr 28 06:03 ..
> drwxr-xr-x.  5 root     root     4.0K Apr 28 04:41 modules
> drwxr-xr-x.  2 www-data www-data 4.0K Oct  9  2014 owncloud-51d9c764244bc
> drwxr-xr-x.  2 www-data www-data 4.0K Jan  6 23:27 owncloud-oc55674097b9
> drwx-wx-wt.  2 root     root      92K May  1 01:14 sessions
> 
> real    0m39.292s
> user    0m0.000s
> sys     0m3.964s
> ------------------------------------
> 
> Notice the "18M" entry for "." - I realize this is a directory for 
> temporary files, meaning that lots of files are generated here, but
> the server is not _that_ busy; according to lsof(8) no files are
> currently open in /var/lib/php5 and only the "sessions" directory
> contains 100 files, there are far less files below the other directories.

That's simply because shrinking directories while the file system is
mounted is... tricky.  At some point we might try to get this
supported, but until we do, there are two workarounds:

1)  Recreate the directory, i.e.,

    mkdir /var/lib/php5.new
    chown root:www-data /var/lib/php5.new
    chmod 1770 /var/lib/lib/php5.new
    mv /var/lib/php5/* /var/lib/php5.new ; mv /var/lib/php5 /var/lib/php5.old  ; mv /var/lib/php5.new /var/lib/php5
    rmdir /var/lib/php5

2) Run "e2fsck -fD /dev/sdXX" on the device containing /var while the file system is unmounted.

Obviously, both of these will require temporarily pausing your web
server, and the second will probably require going to single user
mode or booting from a rescue CD.

> I feel like this is some kind of FAQ, but I could not find anything 
> similar for Ext4. The only similar thing I could think of is an older 
> bug[0] in JFS, where a directory slowly grows when many files are 
> generated & deleted, but I could not reproduce this for Ext4.


I'm not sure I would call this a bug; it's a long standing proper of
ext2/3/4, the BSD FFS, etc., which is directories can't get shurnk by
the file system.  At some point the directory had enough files in it
that it grew to that size, and once having grown to that size, it
won't shirnk on its own.

It would be possible to enhance ext4 to be able to shrink directories
on-line, but it would require adding a new file system (to allow
sparse directories), getting a new version of e2fsck, and then writing
a bunch of new code, and it's just one of those things we've never
gotten around to doing, mainly because for most use cases it rarely
hits people, and the fix should be realtive well understood, as
various Linux/Unix systems have suffered from this for decades.

I'd accept patches to address this, and would even sketch out the
design to some interested summer student interested in a Google Summer
of Code project (for example), but it just hasn't happened yet.

Regards,

						- Ted

Re: large directory entry

[Christian Kujau]

On Fri, 1 May 2015 at 10:31, Theodore Ts'o wrote:
> That's simply because shrinking directories while the file system is
> mounted is... tricky.  At some point we might try to get this
> supported, but until we do, there are two workarounds:
> 
> 1)  Recreate the directory, i.e.,

Yes, I did that and the copy is only 450 KB in size, so this works.

> 2) Run "e2fsck -fD /dev/sdXX" on the device containing /var while
> the file system is unmounted.

My /var is on the root disk and a normal fsck has been run during 
bootup just a few days ago, but without -D, of course.

> I'm not sure I would call this a bug; it's a long standing proper of
> ext2/3/4, the BSD FFS, etc., which is directories can't get shurnk by
> the file system.  At some point the directory had enough files in it
> that it grew to that size, and once having grown to that size, it
> won't shirnk on its own.

Yes, I noticed that with smaller sized directory, but somehow this 18 MB 
directory with "nothing" inside tipped me off.

> It would be possible to enhance ext4 to be able to shrink directories
> on-line, but it would require adding a new file system (to allow
> sparse directories), getting a new version of e2fsck, and then writing
> a bunch of new code, and it's just one of those things we've never
> gotten around to doing, mainly because for most use cases it rarely
> hits people, and the fix should be realtive well understood, as
> various Linux/Unix systems have suffered from this for decades.

Thanks for the explanation, Ted. I hope the next one asking this will find 
this in the archives, I somehow didn't :-\

Christian.
-- 
BOFH excuse #226:

A star wars satellite accidently blew up the WAN.

Re: large directory entry

[Dave Chinner]

On Fri, May 01, 2015 at 10:31:47AM -0400, Theodore Ts'o wrote:
> On Thu, Apr 30, 2015 at 10:02:58PM -0700, Christian Kujau wrote:
> > Notice the "18M" entry for "." - I realize this is a directory for 
> > temporary files, meaning that lots of files are generated here, but
> > the server is not _that_ busy; according to lsof(8) no files are
> > currently open in /var/lib/php5 and only the "sessions" directory
> > contains 100 files, there are far less files below the other directories.
> 
> That's simply because shrinking directories while the file system is
> mounted is... tricky.  At some point we might try to get this
> supported, but until we do, there are two workarounds:
.....
> I'm not sure I would call this a bug; it's a long standing proper of
> ext2/3/4, the BSD FFS, etc., which is directories can't get shurnk by
> the file system.  At some point the directory had enough files in it
> that it grew to that size, and once having grown to that size, it
> won't shirnk on its own.

Interesting - I didn't realise that ext4 had this property. I'm so
used to XFS where the directory structure grows and shrinks
according to it's contents. I learnt something new about ext4 today!

> It would be possible to enhance ext4 to be able to shrink directories
> on-line, but it would require adding a new file system (to allow
> sparse directories),

Yeah, XFS directories are sparsely mapped files with internal
indexes to maintain constant offset mappings and index free space.
It is rather complex... :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com