Re: Partial Allocation

[Josef Bacik <jbacik@redhat.com>]

On Wed, Dec 03, 2008 at 12:32:56AM +0000, Oliver Mattos wrote:
> Hi,
> 
> I presume that in the design of BTRFS, like most other filesystems, a
> block on the underlying storage is either allocated (ie. to store
> metadata or file data), or deallocated (possibly blank or containing
> garbage left over, but the contents are irrelivant).
> 
> Does BTRFS have any system that could allow adding at a later point in
> time a feature which would allow "weak" allocation of blocks, by which I
> mean the block is allocated (ie. storing useful data), but if another
> file needs to be written which has a higher priority and there are not
> many free blocks left, then the data could be replaced.
> 
> I could forsee uses for features like that as a cache - for example my
> web browsing cache is not vital data, and as such doesn't need to use up
> disk space, but it might as well use up any disk space that would
> otherwise go unused.  The cache data can always be regenerated, so
> loosing the data isn't a problem.
> 
> Other uses of the feature could be for persistant network caches (ie. to
> store copies of remote files on the network can they can be accessed
> faster locally), but again the cache data isn't critical to the
> operation of the system, so could be stored in "weakly allocated"
> blocks.  Further uses could be caches of compressed files (decompressed
> versions of the same files are also saved in other blocks, and depending
> on IO and CPU load either the compressed or decompressed version is
> used).
> 
> >From a user-land perspective, these files could be created with a
> special flag which specifies they are only "weakly allocated", which
> means any time the file has no open file descriptors it could "vanish"
> if the underlying filesystem wants to use the space it occupies for
> something else.  A file could have a "priority" value which specifies
> how important it is, and therefore how likely it is to be erased if a
> new block needs to be allocated.  The block allocator would use this
> information, together with the physical layout of the data to decide
> where to place new data to avoid fragmentation while retaining possibly
> useful data for future use.
> 
> Let me know your ideas on this - at the moment it's only an idea, but
> I'm interested to know if a) it would be possible to implement it into a
> complex filesystem like btrfs, and b) if it would prove useful if
> implemented.
> 
> Thanks
> Oliver.
> 
> PS.  I realise this could be implemented with a user space daemon which
> polls available disk space and deletes caches when disk space gets low
> (as windows does with shadow copies), but that hardly seems ideal, since
> it can't intelligently choose which caches to delete to reduce
> fragmentation, and large sudden disk allocations will fail.
>

So one thing you could do was wire this in with the reserve allocation stuff.
You can do a btrfs_reserve_extent which will just make the allocation and hold
it until you decide to allocate it.  Then you can just keep this in a list and
if things get tough you can go through and start reaping them in order to make
space for things that are actually going to allocate the space and use it.  The
reservations wouldn't be persistent across unmounts, but I think thats for the
best for something like this.  All of the hard work is done, all that would need
to be done is to have an interface wired up for it.  Thanks,

Josef