[LSF/MM TOPIC] Addressing mmap_sem contention

[LSF/MM TOPIC] Addressing mmap_sem contention

[Laurent Dufour]

Hi,

I would like to talk about the way to remove the mmap_sem contention we
could see on large threaded systems.

I already resurrected the Speculative Page Fault patchset from Peter
Zijlstra [1]. This series allows concurrency between page fault handler and
the other thread's activity. Running a massively threaded benchmark like
ebizzy [2] on top of this kernel shows that there is an opportunity to
scale far better on large systems (x2). But the SPF series is addressing
only one part of the issue, and there is a need to address the other part
of picture.

There have been some discussions last year about the range locking but this
has been put in hold, especially because this implies huge change in the
kernel as the mmap_sem is used to protect so many resources (should we need
to protect the process command line with the mmap_sem ?), and sometimes the
assumption is made that the mmap_sem is protecting code against concurrency
while it is not dealing clearly with the mmap_sem.

This will be a massive change and rebasing such a series will be hard, so
it may be far better to first agreed on best options to improve mmap_sem's
performance and scalability. There are several additional options on the
table, including range locking,    multiple fine-grained locks, etc...
In addition, I would like to discuss the options and the best way to make
the move smooth in breaking or replacing the mmap_sem.

Peoples (sorry if I missed someone) :
    Andrea Arcangeli
    Davidlohr Bueso
    Michal Hocko
    Anshuman Khandual
    Andi Kleen
    Andrew Morton
    Matthew Wilcox
    Peter Zijlstra

Thanks,
Laurent
[1] https://lkml.org/lkml/2018/1/12/515
[2] http://ebizzy.sourceforge.net/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM TOPIC] Addressing mmap_sem contention

[Davidlohr Bueso]

On Mon, 29 Jan 2018, Laurent Dufour wrote:

>Hi,
>
>I would like to talk about the way to remove the mmap_sem contention we
>could see on large threaded systems.

'cause what's lsfmm without mmap_sem, right? ;)

>
>I already resurrected the Speculative Page Fault patchset from Peter
>Zijlstra [1]. This series allows concurrency between page fault handler and
>the other thread's activity. Running a massively threaded benchmark like
>ebizzy [2] on top of this kernel shows that there is an opportunity to
>scale far better on large systems (x2). But the SPF series is addressing
>only one part of the issue, and there is a need to address the other part
>of picture.
>
>There have been some discussions last year about the range locking but this
>has been put in hold, especially because this implies huge change in the
>kernel as the mmap_sem is used to protect so many resources (should we need
>to protect the process command line with the mmap_sem ?), and sometimes the
>assumption is made that the mmap_sem is protecting code against concurrency
>while it is not dealing clearly with the mmap_sem.
>
>This will be a massive change and rebasing such a series will be hard, so
>it may be far better to first agreed on best options to improve mmap_sem's
>performance and scalability. There are several additional options on the
>table, including range locking,    multiple fine-grained locks, etc...
>In addition, I would like to discuss the options and the best way to make
>the move smooth in breaking or replacing the mmap_sem.

I'd also like to discuss this stuff. In particular I've been focusing on
range locking the mm. With the range_lock primitive ready by now (including
rbtree optimizations), my priority as been converting mmap_sem and getting
adequate performance data for the worst case scenario (full range).

Also, fyi recently by means of auditing handle_mm_fault() and gup family,
two new naughty users were found that were doing gup() without mmap_sem:

https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/commit/?h=wip/dl-for-next&id=487f6683f1b738e40aca2386b9f73da4ebb8223d
https://lkml.org/lkml/2018/1/22/640

With a from-scratch conversion, it's been mostly pretty straightforward
although I've done some hacks along the way. In particular avoiding having
to teach file_operations about mmrange, which would be a gazillion times
more changes than what we already have. So removing the is_locked() check
for calls like zap_pmd_range(), thp (pmd_trans_huge_lock()),
vm_insert_page() (which I audited and all ->fault() users seem to correctly
set VM_MIXEDMAP, so we might be able to get rid of it, dunno).

All this said, yes, I hope to have the patches and numbers asap (way
before lsfmm).

>Peoples (sorry if I missed someone) :
>    Andrea Arcangeli
>    Davidlohr Bueso
>    Michal Hocko
>    Anshuman Khandual
>    Andi Kleen
>    Andrew Morton
>    Matthew Wilcox
>    Peter Zijlstra

I'd also add Kirill A. Shutemov.

Thanks,
Davidlohr

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>