Re: mmap/munmap latency on multithreaded apps, because pagefaults hold mmap_sem during disk read

["Török Edwin" <edwintorok@gmail.com>]

On 2008-09-12 22:35, Mike Waychison wrote:
> Török Edwin wrote:
>> On 2008-09-08 22:10, Török Edwin wrote:
>>> [snip]
>>> There is however a problem with mmap [mmap with N threads is as slow as
>>> mmap with 1 thread, i.e. it is sequential :(], pagefaults and disk I/O,
>>> I think I am hitting the problem described in this thread (2 years
>>> ago!)
>>> http://lwn.net/Articles/200215/
>>> http://lkml.org/lkml/2006/9/19/260
>>>
>>> It looks like such a patch is still not part of 2.6.27, what
>>> happened to it?
>>> I will see if that patch applies to 2.6.27, and will rerun my test with
>>> that patch applied too.
>>>   
>>
>> The patch doesn't apply to 2.6.27-rc6, I tried manually applying the
>> patch.
>> There have been many changes since 2.6.18 (like replacing find_get_page
>> with find_lock_page, filemap returning VM_FAULT codes, etc.).
>> I have probably done something wrong, because the resulting kernel won't
>> boot: I  get abnormal exits and random sigbus during boot.
>>
>> Can you please help porting the patch to 2.6.27-rc6? I have attached my
>> 2 attempts at the end of this mail.
>
> I actually have to forward port this and a bunch of other mm speed-ups
> in the coming two weeks, though they'll be ports from 2.6.18 to
> 2.6.26.  I'll be sending them out to linux-mm once I've done so and
> completed some testing.
>


That would be great, thanks!

>>
>> Also it looks like the original patch just releases the mmap_sem if
>> there is lock contention on the page, but keeps mmap_sem during read?
>> I would like mmap_sem be released during disk I/O too.
>
> The 'lock'ing of the page is the part that waits for the read to
> finish, and is the part that is contentious.

Didn't know that, thanks for explaining.

>
>>
>> I also tried changing i_mmap_lock into a semaphore, however I that won't
>> work since some users of i_mmap_lock can't sleep.
>> Taking the i_mmap_lock spinlock in filemap fault is also not possible,
>> since we would sleep while holding a spinlock.
>
> You shouldn't be seeing much contention on the i_mmap_lock, at least
> not in the fault path (it's mostly just painful when you have a lot of
> threads in direct reclaim and you have a large file mmaped).

I was thinking of using i_mmap_lock as an alternative to holding
mmap_sem, but it didn't seem like a good idea after all.

Best regards,
--Edwin