Re: Re: Lock overhead in shrink_inactive_list / Slow page reclamation

[Kirill Tkhai <ktkhai@virtuozzo.com>]

On 14.01.2019 02:12, Baptiste Lepers wrote:
> On Sat, Jan 12, 2019 at 4:53 AM Daniel Jordan
> <daniel.m.jordan@oracle.com> wrote:
>>
>> On Fri, Jan 11, 2019 at 02:59:38PM +0100, Michal Hocko wrote:
>>> On Fri 11-01-19 16:52:17, Baptiste Lepers wrote:
>>>> Hello,
>>>>
>>>> We have a performance issue with the page cache. One of our workload
>>>> spends more than 50% of it's time in the lru_locks called by
>>>> shrink_inactive_list in mm/vmscan.c.
>>>
>>> Who does contend on the lock? Are there direct reclaimers or is it
>>> solely kswapd with paths that are faulting the new page cache in?
>>
>> Yes, and could you please post your performance data showing the time in
>> lru_lock?  Whatever you have is fine, but using perf with -g would give
>> callstacks and help answer Michal's question about who's contending.
> 
> Thanks for the quick answer.
> 
> The time spent in the lru_lock is mainly due to direct reclaimers
> (reading an mmaped page that causes some readahead to happen). We have
> tried to play with readahead values, but it doesn't change performance
> a lot. We have disabled swap on the machine, so kwapd doesn't run.
> 
> Our programs run in memory cgroups, but I don't think that the issue
> directly comes from cgroups (I might be wrong though).
> 
> Here is the callchain that I have using perf report --no-children;
> (Paste here https://pastebin.com/151x4QhR )
> 
>     44.30%  swapper      [kernel.vmlinux]  [k] intel_idle
>     # The machine is idle mainly because it waits in that lru_locks,
> which is the 2nd function in the report:
>     10.98%  testradix    [kernel.vmlinux]  [k] native_queued_spin_lock_slowpath
>                |--10.33%--_raw_spin_lock_irq
>                |          |
>                |           --10.12%--shrink_inactive_list
>                |                     shrink_node_memcg
>                |                     shrink_node
>                |                     do_try_to_free_pages
>                |                     try_to_free_mem_cgroup_pages
>                |                     try_charge
>                |                     mem_cgroup_try_charge
>                |                     __add_to_page_cache_locked
>                |                     add_to_page_cache_lru
>                |                     |
>                |                     |--5.39%--ext4_mpage_readpages
>                |                     |          ext4_readpages
>                |                     |          __do_page_cache_readahead
>                |                     |          |
>                |                     |           --5.37%--ondemand_readahead
>                |                     |
> page_cache_async_readahead

Does MADV_RANDOM make the trace better or worse?

>                |                     |                     filemap_fault
>                |                     |                     ext4_filemap_fault
>                |                     |                     __do_fault
>                |                     |                     handle_pte_fault
>                |                     |                     __handle_mm_fault
>                |                     |                     handle_mm_fault
>                |                     |                     __do_page_fault
>                |                     |                     do_page_fault
>                |                     |                     page_fault
>                |                     |                     |
>                |                     |                     |--4.23%-- <our app>
> 
> 
> Thanks,
> 
> Baptiste.
> 
> 
> 
> 
> 
> 
>>
>> Happy to help profile and debug offline.
>