Re: slow performance on disk/network i/o full speed after drop_caches

[Mel Gorman <mel@csn.ul.ie>]

On Thu, Sep 01, 2011 at 12:14:58PM +0800, Wu Fengguang wrote:
> Hi Stefan,
> 
> On Wed, Aug 31, 2011 at 03:11:02PM +0800, Stefan Priebe - Profihost AG wrote:
> > Hi Fengguang,
> > Hi Yanhai,
> > 
> > > you're abssolutely corect zone_reclaim_mode is on - but why?
> > > There must be some linux software which switches it on.
> > >
> > > ~# grep 'zone_reclaim_mode' /etc/sysctl.* -r -i
> > > ~#
> > >
> > > also
> > > ~# grep 'zone_reclaim_mode' /etc/sysctl.* -r -i
> > > ~#
> > >
> > > tells us nothing.
> > >
> > > I've then read this:
> > >
> > > "zone_reclaim_mode is set during bootup to 1 if it is determined that
> > > pages from remote zones will cause a measurable performance reduction.
> > > The page allocator will then reclaim easily reusable pages (those page
> > > cache pages that are currently not used) before allocating off node pages."
> > >
> > > Why does the kernel do that here in our case on these machines.
> > 
> > Can nobody help why the kernel in this case set it to 1?
> 
> It's determined by RECLAIM_DISTANCE.
> 
> build_zonelists():
> 
>                 /*
>                  * If another node is sufficiently far away then it is better
>                  * to reclaim pages in a zone before going off node.
>                  */
>                 if (distance > RECLAIM_DISTANCE)
>                         zone_reclaim_mode = 1;
> 
> Since Linux v3.0 RECLAIM_DISTANCE is increased from 20 to 30 by this commit.
> It may well help your case, too.
> 

Even with that, it's known that zone_reclaim() can be a disaster when
it runs into problems. This should be fixed in 3.1 by the following
commits;

[cd38b115 mm: page allocator: initialise ZLC for first zone eligible for zone_reclaim]
[76d3fbf8 mm: page allocator: reconsider zones for allocation after direct reclaim]

The description in cd38b115 has the interesting details.

-- 
Mel Gorman
SUSE Labs