Re: Split some metadata onto separate device?

[Reinoud Zandijk <reinoud-qavaossjCcEdnm+yROfE0A@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 2526 bytes --]

Hi Dexen, hi Ryusuke,

been a while since i was active on the list or active with NiLFS in general
but the last vacation did remind me why i started working on NiLFS for NetBSD
again! Using FFS, even with logging, UDF or msdosfs just doesn't work well on
Flash media ;)

On Wed, Jun 08, 2011 at 10:49:06PM +0900, Ryusuke Konishi wrote:
> > > > Yes, but extents will decrease the amount of metadata, and this has
> > > > potential for speed up.
> > > > 
> > > > NILFS2 uses 32 bytes metadata per disk block at present.  I guess you
> > > > know that the number of DAT blocks are actually indispensable through
> > > > analysis using the dumpseg tool.
> > > 
> > > Oh, looks confusing.  I meant the amount of DAT blocks is not
> > > negligible.
> > 
> > Understood ;-)
> > 
> > Anyway, isn't some fragmentation avoidance necessary to profit from extents 
> > (that we hope we'll use at some point)?
> 
> I guess it doesn't become a big issue unless we do many small random
> writes like database.
> 
> > AFAIK, an extent is only good to describe file blocks if they are laid out 
> > continuous on the block device.
> 
> Yep.  And, we know files are enough continuous in most cases.
> 
> We use at least 16 bytes per 4 kiro-bytes disk block to point the
> block (16KB for 8MB segment); btrees of nilfs2 use an 8 bytes key and
> an 8 bytes pointer for a disk block.
> 
> In typical case, this can be reduced to 16 bytes per file (16 bytes
> for 8MB segment) with the extents.
> 
> IMO, 64-bit filesystems have a reasonable reason to adopt extents
> though nilfs2 did not apply it.
> 
> I guess applying extents to the DAT file is especially effective.

I've used extents a lot in UDF since its all-extent based, and yes, it CAN be
a lot smaller indeed :) BUT only when files are NOT sparse and NOT fragmented.

Anyway, my sugestion is an intermediate one since it doesn't change the disc
format: on garbage collection, make an extra pass to seek (very) fragmented
extents in the DAT, how much might be a parameter, read it all in and then
relocate/write out a whole (or partial) segment with it. This could also be
done with the CP and SU files although those files are relatively tiny. This
will leave the DAT file in just a few segments when the filesystem has been
garbage collected and (hopefully) most references only within the segment also
easing cachability.

Other files could also be un-sparsified this way but i think the DAT will have
the most benefit and it's relatively easy to do this way.

With regards,
Reinoud


[-- Attachment #2: Type: application/pgp-signature, Size: 487 bytes --]