Re: Add prefetch switch stack hook in scheduler function

[Ingo Molnar <mingo@elte.hu>]

* Ingo Molnar <mingo@elte.hu> wrote:

> > Sorry, this is not enough.  Switch stack on ia64 is 528 bytes.  We 
> > need to prefetch 5 lines.  It probably should use prefetch_range().  
> 
> ok, how about the additional patch below? Does this do the trick on 
> ia64? It makes complete sense on every architecture to prefetch from 
> below the current kernel stack, in the expectation of the next task 
> touching the stack. The only difference is that for ia64 the 'expected 
> minimum stack footprint' is larger, due to the switch_stack.

the patch below unrolls the prefetch_range() loop manually, for up to 5 
cachelines prefetched. This patch, ontop of the 4 previous patches, 
should generate similar code to the assembly code in your original 
patch. The full patch-series is:

 patches/prefetch-next.patch
 patches/prefetch-mm.patch
 patches/prefetch-kstack-size.patch
 patches/prefetch-unroll.patch

	Ingo

---------
unroll prefetch_range() loops manually.

Signed-off-by: Ingo Molnar <mingo@elte.hu>

 include/linux/prefetch.h |   31 +++++++++++++++++++++++++++++--
 1 files changed, 29 insertions(+), 2 deletions(-)

Index: linux/include/linux/prefetch.h
===================================================================
--- linux.orig/include/linux/prefetch.h
+++ linux/include/linux/prefetch.h
@@ -58,11 +58,38 @@ static inline void prefetchw(const void 
 static inline void prefetch_range(void *addr, size_t len)
 {
 #ifdef ARCH_HAS_PREFETCH
-	char *cp;
+	char *cp = addr;
 	char *end = addr + len;
 
-	for (cp = addr; cp < end; cp += PREFETCH_STRIDE)
+	/*
+	 * Unroll agressively:
+	 */
+	if (len <= PREFETCH_STRIDE)
 		prefetch(cp);
+	else if (len <= 2*PREFETCH_STRIDE) {
+		prefetch(cp);
+		prefetch(cp + PREFETCH_STRIDE);
+	}
+	else if (len <= 3*PREFETCH_STRIDE) {
+		prefetch(cp);
+		prefetch(cp + PREFETCH_STRIDE);
+		prefetch(cp + 2*PREFETCH_STRIDE);
+	}
+	else if (len <= 4*PREFETCH_STRIDE) {
+		prefetch(cp);
+		prefetch(cp + PREFETCH_STRIDE);
+		prefetch(cp + 2*PREFETCH_STRIDE);
+		prefetch(cp + 3*PREFETCH_STRIDE);
+	}
+	else if (len <= 5*PREFETCH_STRIDE) {
+		prefetch(cp);
+		prefetch(cp + PREFETCH_STRIDE);
+		prefetch(cp + 2*PREFETCH_STRIDE);
+		prefetch(cp + 3*PREFETCH_STRIDE);
+		prefetch(cp + 4*PREFETCH_STRIDE);
+	} else
+		for (; cp < end; cp += PREFETCH_STRIDE)
+			prefetch(cp);
 #endif
 }