Re: Add prefetch switch stack hook in scheduler function

Ingo Molnar
2005-07-29T07:07:02Z

* Ingo Molnar <mingo@elte.hu> wrote:

> > Sorry, this is not enough. Switch stack on ia64 is 528 bytes. We
> > need to prefetch 5 lines. It probably should use prefetch_range().
>
> ok, how about the additional patch below? Does this do the trick on
> ia64? It makes complete sense on every architecture to prefetch from
> below the current kernel stack, in the expectation of the next task
> touching the stack. The only difference is that for ia64 the 'expected
> minimum stack footprint' is larger, due to the switch_stack.

the patch below unrolls the prefetch_range() loop manually, for up to 5
cachelines prefetched. This patch, ontop of the 4 previous patches,
should generate similar code to the assembly code in your original
patch. The full patch-series is:

patches/prefetch-next.patch
patches/prefetch-mm.patch
patches/prefetch-kstack-size.patch
patches/prefetch-unroll.patch

Ingo

---------
unroll prefetch_range() loops manually.

Signed-off-by: Ingo Molnar <mingo@elte.hu>

include/linux/prefetch.h | 31 +++++++++++++++++++++++++++++--
1 files changed, 29 insertions(+), 2 deletions(-)

Index: linux/include/linux/prefetch.h
=================================--- linux.orig/include/linux/prefetch.h
+++ linux/include/linux/prefetch.h
@@ -58,11 +58,38 @@ static inline void prefetchw(const void
static inline void prefetch_range(void *addr, size_t len)
{
#ifdef ARCH_HAS_PREFETCH
- char *cp;
+ char *cp = addr;
char *end = addr + len;

- for (cp = addr; cp < end; cp += PREFETCH_STRIDE)
+ /*
+ * Unroll agressively:
+ */
+ if (len <= PREFETCH_STRIDE)
prefetch(cp);
+ else if (len <= 2*PREFETCH_STRIDE) {
+ prefetch(cp);
+ prefetch(cp + PREFETCH_STRIDE);
+ }
+ else if (len <= 3*PREFETCH_STRIDE) {
+ prefetch(cp);
+ prefetch(cp + PREFETCH_STRIDE);
+ prefetch(cp + 2*PREFETCH_STRIDE);
+ }
+ else if (len <= 4*PREFETCH_STRIDE) {
+ prefetch(cp);
+ prefetch(cp + PREFETCH_STRIDE);
+ prefetch(cp + 2*PREFETCH_STRIDE);
+ prefetch(cp + 3*PREFETCH_STRIDE);
+ }
+ else if (len <= 5*PREFETCH_STRIDE) {
+ prefetch(cp);
+ prefetch(cp + PREFETCH_STRIDE);
+ prefetch(cp + 2*PREFETCH_STRIDE);
+ prefetch(cp + 3*PREFETCH_STRIDE);
+ prefetch(cp + 4*PREFETCH_STRIDE);
+ } else
+ for (; cp < end; cp += PREFETCH_STRIDE)
+ prefetch(cp);
#endif
}