Re: page fault fastpath: Increasing SMP scalability by introducing pte

Re: page fault fastpath: Increasing SMP scalability by introducing pte

[Manfred Spraul]

Christoph wrote:

>Well this is more an idea than a real patch yet. The page_table_lock
>becomes a bottleneck if more than 4 CPUs are rapidly allocating and using
>memory. "pft" is a program that measures the performance of page faults on
>SMP system. It allocates memory simultaneously in multiple threads thereby
>causing lots of page faults for anonymous pages.
>
>  
>
Very odd. Why do you see a problem with the page_table_lock but no 
problem from the mmap semaphore?
The page fault codepath acquires both.
How often is the page table lock acquired per page fault? Just once or 
multiple spin_lock calls per page fault? Is the problem contention or 
cache line trashing?

Do you have profile/lockmeter output? Is the down_read() in 
do_page_fault() a hot spot, too?

--
    Manfred

Re: page fault fastpath: Increasing SMP scalability by introducing pte

[William Lee Irwin III]

Christoph wrote:
>> Well this is more an idea than a real patch yet. The page_table_lock
>> becomes a bottleneck if more than 4 CPUs are rapidly allocating and using
>> memory. "pft" is a program that measures the performance of page faults on
>> SMP system. It allocates memory simultaneously in multiple threads thereby
>> causing lots of page faults for anonymous pages.

On Sun, Aug 15, 2004 at 04:17:11PM +0200, Manfred Spraul wrote:
> Very odd. Why do you see a problem with the page_table_lock but no 
> problem from the mmap semaphore?
> The page fault codepath acquires both.
> How often is the page table lock acquired per page fault? Just once or 
> multiple spin_lock calls per page fault? Is the problem contention or 
> cache line trashing?
> Do you have profile/lockmeter output? Is the down_read() in 
> do_page_fault() a hot spot, too?

->mmap_sem would likely be useful to address as well. A different
structure for vma trees, more amenable to finegrained locking or
lockfree algorithms, would likely be useful there.


-- wli

Re: page fault fastpath: Increasing SMP scalability by introducing pte

[Christoph Lameter]

On Sun, 15 Aug 2004, Manfred Spraul wrote:

> Very odd. Why do you see a problem with the page_table_lock but no
> problem from the mmap semaphore?

I will have a look at that...

> The page fault codepath acquires both.
> How often is the page table lock acquired per page fault? Just once or
> multiple spin_lock calls per page fault? Is the problem contention or
> cache line trashing?

The page table lock seems to be acquired at least 2 times and up to 5
times per page fault.

Re: page fault fastpath: Increasing SMP scalability by introducing pte

[Christoph Lameter]

On Sun, 15 Aug 2004, Manfred Spraul wrote:

> Very odd. Why do you see a problem with the page_table_lock but no
> problem from the mmap semaphore?

Because there is a only a down_read() call for the mmap semaphore before
invoking handle_mm_fault. This is a rw semaphore which means that multiple
processors/processes may be entering handle_mm_fault with a read lock on
the mmap semaphore.

Re: page fault fastpath: Increasing SMP scalability by introducing pte

[Manfred Spraul]

Christoph Lameter wrote:

>On Sun, 15 Aug 2004, Manfred Spraul wrote:
>
>  
>
>>Very odd. Why do you see a problem with the page_table_lock but no
>>problem from the mmap semaphore?
>>    
>>
>
>Because there is a only a down_read() call for the mmap semaphore before
>invoking handle_mm_fault. This is a rw semaphore which means that multiple
>processors/processes may be entering handle_mm_fault with a read lock on
>the mmap semaphore.
>  
>
As far as I can see the actual hold times on the page_table_lock are not 
that long and down_read() causes a cache line transfer, too.
But you have answered my question in your first mail: There are 2-5 
spin_lock calls per page fault but only one down_read call.

--
    Manfred