Re: [PATCH UPDATED 2/2] mempool: fix first round failure behavior

[Vivek Goyal <vgoyal@redhat.com>]

On Wed, Dec 21, 2011 at 05:23:12PM -0800, Tejun Heo wrote:
> Hello, Andrew.
> 
> On Wed, Dec 21, 2011 at 05:09:19PM -0800, Andrew Morton wrote:
> > If the pool is empty and the memory allocator is down into its
> > emergency reserves then we have:
> > 
> > Old behaviour: Wait for someone to return an item, then retry
> > 
> > New behaviour: enable page reclaim in gfp_mask, retry a
> >                single time then wait for someone to return an item.
> > 
> > So what we can expect to see is that in this low-memory situation,
> > mempool_alloc() will perform a lot more page reclaim, and more mempool
> > items will be let loose into the kernel.
> > 
> > I'm not sure what the effects of this will be.  I can't immediately
> > point at any bad ones.  Probably not much, as the mempool_alloc()
> > caller will probably be doing other allocations, using the
> > reclaim-permitting gfp_mask.
> > 
> > But I have painful memories of us (me and Jens, iirc) churning this
> > code over and over again until it stopped causing problems.  Some were
> > subtle and nasty.  Much dumpster diving into the pre-git changelogs
> > should be done before changing it, lest we rediscover long-fixed
> > problems :(
> 
> I see.  It just seemed like a weird behavior and looking at the commit
> log, there was originally code to kick reclaim there, so the sequence
> made sense - first try w/o reclaim, look at the mempool, kick reclaim
> and retry w/ GFP_WAIT and then wait for someone else to free.  That
> part was removed by 20a77776c24 "[PATCH] mempool: simplify alloc" back
> in 05.  In the process, it also lost retry w/ reclaim before waiting
> for mempool reserves.
> 
> I was trying to add percpu mempool and this bit me as percpu allocator
> can't to NOIO and the above delayed retry logic ended up adding random
> 5s delay (or until the next free).
> 

Tejun,

For blkio to use these percpu mempool, I guess we shall have to change this
notion that an allocated object from mempool is returned to the pool soon.
We might not be returning these per cpu stat objects to mempool for a very
long time (These will be returned to pool only when somebody deletes the
cgroup).

Thanks
Vivek