Re: [RFC 0/4] Object reclaim via the slab allocator V1

[Marcelo Tosatti <marcelo@kvack.org>]

On Mon, Jul 03, 2006 at 06:46:55PM -0700, Christoph Lameter wrote:
> On Mon, 3 Jul 2006, Marcelo Tosatti wrote:
> 
> > > I think is pretty obvious. With atomic refcounters you can simply scan
> > > a slab for unused objects without any callbacks. Needing a callback for 
> > > every single object is a waste of resources and will limit reclaim 
> > > efficiency. You would have to do 120 callbacks on some slabs just to 
> > > figure out that it is worth trying to free objects in that 
> > > particular slab block.
> > 
> > Inline the callbacks into a per-cache kmem_cache_reclaim ?
> 
> You mean the user writes the check functions? Can you give an example how 
> inlining is supposed to work here?

I meant per-cache kmem_cache_reclaim (user-provided function). "Inline"
is confusing, sorry.

> > > Cannot see a valid reason so far to draw that conclusion. With the right 
> > > convention the atomic refcounter can be used as an indicator that the 
> > > object is being freed (refcnt = 0), not in use (refcnt = 1) or in active 
> > > use (refcnt=2). The easy and efficient access to this kind of knowledge 
> > > about an object is essential for reclaim.
> > 
> > But the assumption that "refcnt = 1 implies unused object" is too weak.
> > 
> > For example, 
> > 
> > struct dentry {
> >         atomic_t d_count;
> >         unsigned int d_flags;           /* protected by d_lock */
> > 
> > d_count can be higher than one _and_ the object freeable. Think of
> > workloads operating on a large number of directories.
> 
> The scheme that I proposed implies that the refcount handling is changed.
> It must be uniform for all object types that use reclaim. 
>
> If used for the dcache then dentry handling must be changed so that the 
> refcount at the beginnDing of the slab is 1 if the object is reclaimable 
> and the higher refcount needs to be an indicator that the object is in 
> use. I am not saying that existing use gets us there. Maybe we need to 
> call this a reclaim flag instead of a refcount?

I think the cache has to decide...

> > Andrew mentioned:
> > 
> > "That seems like quite a drawback. A single refcount=2 object on the
> > page means that nothing gets freed from that page at all. It'd be easy
> > (especially with dcache) to do tons of work without achieving anything."
> 
> We can check for a single high count object in a slab and then call
> the destructor if we feel that is warranted. The refcount is an 
> indicator to the slab of the reclaim status of the object.
> 
> > > Ok. I will have a look at that. But these callbacks are too heavy for my 
> > > taste. A refcounter could avoid all of that.
> > 
> > Inline them.
> 
> "Inline" seem to be way to say that the user has to provide the function.

Yes.

> > > Of course there is the challenge of preserving the LRU like behavior using 
> > > the slab lists. But I think it may be sufficiently approximated by the 
> > > LRU ness of the used slab list and the push back to the partial lists 
> > > whenever we touch a slab during reclaim (we free some objects so the slab 
> > > has to move).
> > 
> > Well, individual object usage is not reflected at all in the slab lists,
> > is it?
> 
> Correct. We would have to treat objects in a slab all the same. We could 
> just kick out some if we find the slab at the end of the list and see how 
> things develop. Pretty bare hacky LRU but it may be better than going 
> through huge lists of small objects on a LRU lists.

Yep... Did you try this?