Re: percpu allocator vs reclaim

[Peter Zijlstra <peterz@infradead.org>]

On Thu, 2009-03-05 at 01:40 +0900, Tejun Heo wrote:
> Peter Zijlstra wrote:
> > Hi Tejun,
> > 
> > Thomas hit the below on recent -tip kernels.
> > 
> > Which basically states that we could deadlock due to reclaim lock
> > recursion.
> > 
> > Looking at the code I don't see a quick solution, other than using
> > GFP_NOFS, which is a bit of a bother (as I suspect it might easily grow
> > __GFP_IO inversion too, if it doesn't already have it).
> 
> Ah... maybe percpu allocator should just swallow @gfp.  Any better
> ideas? :-(

Could you somehow break that lock so that you get something like:

  alloc_lock
    kmalloc(GFP_KERNEL)
    free_lock

Where

 percpu_free()
   lock(free_lock)
     put area on free list

 percpu_alloc()
   lock(free_list)
     collect free list
   kmalloc()

Then the free code can be used from reclaim, because there's never an
allocation done while holding it, and the alloc path can first
check/cleanup whatever mess the last free left behind before trying an
allocation.