Re: SMP performance problem in 2.4 (was: Athlon spinlock performance)

[Con Kolivas <kernel@kolivas.org>]

Hi Scott

On Sun, 3 Aug 2003 06:03, Scott L. Burson wrote:
> In one approximately 60-second period with the problematic workload
> running, `try_to_free_pages' was called 511 times.  It made 2597 calls to
> `shrink_caches', which made 2592 calls to `shrink_cache' (i.e. it was very
> rare for `kmem_cache_reap' to release enough pages itself).  The main loop
> of `shrink_cache' was executed -- brace yourselves -- 189 million times!
> During that time it called `page_cache_release' on only 31265 pages.

I noticed a curly section of the vm code when I was playing around with some 
hacks that are in the -ck kernel and this section might be helpful as it 
wasn't a hack so much as a fix in mm/vmscan.c around line 600. The problem
is when the priority drops to 1 it should do the most cache reaping but
instead bypasses some of it.
You could try this modification and see if it helps.

This isn't a real patch but you should get the idea. 

Con

 	nr_pages -= kmem_cache_reap(gfp_mask);
-	if (nr_pages <= 0)
-		return 0;
+	if (nr_pages < 1)
+		goto shrinkcheck;

 	nr_pages = chunk_size;
 	/* try to keep the active list 2/3 of the size of the cache */
 	ratio = (unsigned long) nr_pages * nr_active_pages / ((nr_inactive_pages + 1) * 2);
 	refill_inactive(ratio);
 
 	nr_pages = shrink_cache(nr_pages, classzone, gfp_mask, priority);
-	if (nr_pages <= 0)
-		return 0;
+	/*
+	 * Will return if nr_pages have been freed unless the
+	 * priority managed to reach 1. If the vm is under this much
+	 * pressure then shrink the d/i/dqcaches regardless. CK 2003
+	 */
+shrinkcheck:
+	if (nr_pages < 1){
+		if (priority > 1)
+			return 0;
+		else
+			nr_pages = 0;
+		}
+

 	shrink_dcache_memory(priority, gfp_mask);
 	shrink_icache_memory(priority, gfp_mask);