Re: Packed small files

[Hugo Mills <hugo@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]

On Tue, Jan 31, 2012 at 08:21:30AM -0800, Hans Stimer wrote:
> How space efficient are packed small files? For instance, if I have a
> 10,000 16 byte files with 16 byte file names, how much disk space will
> they use? Are there any caveats or gotchas around using btrfs to store
> millions of small (size<64bytes) files?

   For small files, the file content may be kept inline in the btree
blocks of the extent tree. The data is packed in alongside the
associated keys and inline extent metadata, and the Btree
implementation can be somewhat variable in its usage as well, so it's
hard to put fixed values on these things.

   There's metadata overhead on storing files as well -- each file
will be represented by a number of keys (17 bytes each) and metadata
structures to go with them. In the FS tree:

   DIR_ITEM      13
   DIR_INDEX     13
   INODE_ITEM   162
   INODE_REF     10 + size(name)
   EXTENT_DATA   52
   Keys          85
                ---
                335 + size(name)

   And in the extent tree:

   EXTENT_ITEM        24 + 17 for key
   extent_inline_ref   9
   extent_data_ref    28
                     ---
                      78

   So you're looking at a minimum of 413 bytes of metadata overhead
for an inline file, plus the length of the filename.

   Also note that the file is stored in the metadata, so by default
it's stored with DUP or RAID-1 replication (even if data is set to be
"single"). This means that you'll actually use up twice this amount of
space on the disks, unless you create the FS with metadata set to
"single".

   I don't know how these figures compare with other filesystems. My
entirely uneducated guess is that they're probably comparable, with
the exception of the DUP effect.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                   --- I know of three kinds: hot, ---                   
                cool,  and what-time-does-the-tune-start?                

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]