Re: More eager discard behaviour

[Chris Webb <chris@arachsys.com>]

Kent Overstreet <kent.overstreet@gmail.com> writes:

> On Sat, Nov 06, 2021 at 05:11:56PM +0000, Chris Webb wrote:
> > How practical would it be either to more-greedily wake the allocator thread
> > and reclaim buckets, or to detect buckets available to discard earlier in
> > their lifetime?
> 
> It's on the todo list, it'll be part of the grand allocator rework I have
> planned - there's a lot that needs to be done.

I read your discussion on the status update and in the related write-up on
the accounting code. It makes a lot of sense to me, given the tried and
tested btree layer is there to be used.

> We've got to rework the state transitions, and differentiate between buckets
> with cached data, buckets that need a journal commit before we can write to them
> again, buckets that need discarding, and buckets that are ready for use - and we
> need to start storing all that persistently, as well as build some new data
> structures to get rid of the scanning over every bucket we currently have to do
> in various places.
> 
> Step one is going to be creating a persistent LRU. It'll be another btree,
> sorted by the order in which we want to reuse buckets - a combination of the
> amount of live data in the bucket and the time since it was last read.
> 
> If you're interested in tackling this, I can sketch it out for you :)

I'm definitely interested in helping out if I can, although there's a slight
risk of biting off more than I can chew: I probably understand even less
about the design and structure of your fs at the moment than you think!

I don't want to be a slow-moving obstacle on your critical path or more of a
nuisance than a help, but equally, getting stuck in is clearly the best way
to figure out how it fits together and become more useful. So happy to give
anything a go that you think is reasonable for a newcomer to take a crack
at. :) 

Best wishes,

Chris.