Prefetch mmap_sem in ia64_do_page_fault()

Prefetch mmap_sem in ia64_do_page_fault()

[Christoph Lameter]

Take a hint from an x86_64 optimization by Arjan van de Ven and use it
for ia64.

See http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h©ba9a3b3897561d01e04cd21433746df46548c0

Prefetch the mmap_sem, which is critical for the performance of the page fault
handler. Maybe we can offset the damage done by the kprobes notifier?

Note: mm may be NULL but I guess that is safe. See 
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;hE8f935527372499b714bf4f8e646a68bb0f52e3

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.16/arch/ia64/mm/fault.c
=================================--- linux-2.6.16.orig/arch/ia64/mm/fault.c	2006-03-19 21:53:29.000000000 -0800
+++ linux-2.6.16/arch/ia64/mm/fault.c	2006-03-28 22:44:01.000000000 -0800
@@ -60,6 +60,9 @@ ia64_do_page_fault (unsigned long addres
 	struct siginfo si;
 	unsigned long mask;
 
+	/* mmap_sem is performance critical.... */
+	prefetchw(&mm->mmap_sem);
+
 	/*
 	 * If we're in an interrupt or have no user context, we must not take the fault..
 	 */

RE: Prefetch mmap_sem in ia64_do_page_fault()

[Chen, Kenneth W]

Christoph Lameter wrote on Tuesday, March 28, 2006 10:55 PM
> Take a hint from an x86_64 optimization by Arjan van de Ven and use it
> for ia64.
> 
> ...
> 
> Prefetch the mmap_sem, which is critical for the performance of the page fault
> handler. Maybe we can offset the damage done by the kprobes notifier?
> 
> Note: mm may be NULL but I guess that is safe. See 
> 
> +	/* mmap_sem is performance critical.... */
> +	prefetchw(&mm->mmap_sem);
> +

I would say push that a couple of lines down after checking !mm

        if (in_atomic() || !mm)
                goto no_context;

If you insist on prefetching mmap_sem at the very beginning of the
function, then use speculative load, it will "nat" right away with
null pointer without generating any exception at all.  It is super
fast because of nat page.

- Ken

RE: Prefetch mmap_sem in ia64_do_page_fault()

[Christoph Lameter]

On Tue, 28 Mar 2006, Chen, Kenneth W wrote:

> > +	/* mmap_sem is performance critical.... */
> > +	prefetchw(&mm->mmap_sem);
> > +
> 
> I would say push that a couple of lines down after checking !mm
> 
>         if (in_atomic() || !mm)
>                 goto no_context;

I initially also thought that would be better.
 
> If you insist on prefetching mmap_sem at the very beginning of the
> function, then use speculative load, it will "nat" right away with
> null pointer without generating any exception at all.  It is super
> fast because of nat page.

I saw David Mosberger's patch from last year

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;hE8f935527372499b714bf4f8e646a68bb0f52e3

and it seems that he implemented fast handling of these NULL cases?

mm = NULL is very rare so I think we can try to prefetch as 
early as possible.

RE: Prefetch mmap_sem in ia64_do_page_fault()

[Chen, Kenneth W]

Christoph Lameter wrote on Tuesday, March 28, 2006 11:16 PM
> > If you insist on prefetching mmap_sem at the very beginning of the
> > function, then use speculative load, it will "nat" right away with
> > null pointer without generating any exception at all.  It is super
> > fast because of nat page.
> 
> I saw David Mosberger's patch from last year
> 
> http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;hE8f935527372499b714bf4f8e646a68bb0f52e3
> 
> and it seems that he implemented fast handling of these NULL cases?

I know about that patch and I'm referring in the context of that commit:

ld8.s r1=[r0]      - immediate nat generation, no fault
lfetch.fault [r0]  - generate nat page consumption because lfetch is not
                     considered as a speculative access.

The nat page consumption fault handler should be rather quick, though cpu
still have to fault into the handler, muck around with ipsr.ed bit and
then rfi.  What I'm saying is there is an even *faster* way of doing
prefetch.

> mm = NULL is very rare so I think we can try to prefetch as 
> early as possible.

Yeah, sure. Looks good!

- Ken

RE: Prefetch mmap_sem in ia64_do_page_fault()

[Chen, Kenneth W]

Chen, Kenneth W wrote on Tuesday, March 28, 2006 11:32 PM
> Christoph Lameter wrote on Tuesday, March 28, 2006 11:16 PM
> > 
> > I saw David Mosberger's patch from last year
> > 
> > and it seems that he implemented fast handling of these NULL cases?
> 
> I know about that patch and I'm referring in the context of that commit:
> 
> ld8.s r1=[r0]      - immediate nat generation, no fault
> lfetch.fault [r0]  - generate nat page consumption because lfetch is not
>                      considered as a speculative access.
> 
> The nat page consumption fault handler should be rather quick, though cpu
> still have to fault into the handler, muck around with ipsr.ed bit and
> then rfi.  What I'm saying is there is an even *faster* way of doing
> prefetch.

One other caveat with this is that if you have a tlb miss on mm->mmap_sem,
it won't prefetch anything because alt_dtlb_miss handler will nuke the
ld8.s or lfetch instruction :-(

- Ken

RE: Prefetch mmap_sem in ia64_do_page_fault()

[Christoph Lameter]

On Tue, 28 Mar 2006, Chen, Kenneth W wrote:

> One other caveat with this is that if you have a tlb miss on mm->mmap_sem,
> it won't prefetch anything because alt_dtlb_miss handler will nuke the
> ld8.s or lfetch instruction :-(

Still no harm done. It is likely (especially in contended situations) that 
we have the tlb already. 

Maybe we could modify the patch to prefetch the tlb somehow?