Re: memcgroup lruvec_lru_size scaling issue

[Tim Chen <tim.c.chen@linux.intel.com>]

On 10/14/19 10:59 AM, Michal Hocko wrote:
> On Mon 14-10-19 10:49:49, Dave Hansen wrote:
>> On 10/14/19 10:37 AM, Michal Hocko wrote:
>>>> for_each_possible_cpu(cpu)
>>>> 	x += per_cpu(pn->lruvec_stat_local->count[idx], cpu);
>>>>
>>>> It is costly looping through all the cpus to get the lru vec size info.
>>>> And doing this on our workload with 96 cpu threads and 500 mem cgroups
>>>> makes things much worse.  We might end up having 96 cpus * 500 cgroups * 2 (main) LRUs pagevecs,
>>>> which is a lot of data structures to be running through all the time.
>>> Why does the number of cgroup matter?
>>
>> I was thinking purely of the cache footprint.  If it's reading
>> pn->lruvec_stat_local->count[idx] is three separate cachelines, so 192
>> bytes of cache *96 CPUs = 18k of data, mostly read-only.  1 cgroup would
>> be 18k of data for the whole system and the caching would be pretty
>> efficient and all 18k would probably survive a tight page fault loop in
>> the L1.  500 cgroups would be ~90k of data per CPU thread which doesn't
>> fit in the L1 and probably wouldn't survive a tight page fault loop if
>> both logical threads were banging on different cgroups.
>>
>> It's just a theory, but it's why I noted the number of cgroups when I
>> initially saw this show up in profiles.
> 
> Yes, the cache traffic might be really high but I still find it a bit
> surprising that it makes such a large footprint because this should be
> mostly called from slow paths (reclaim) and the real work done should
> just be larger - at least that's my intuition which might be quite off
> here. How much is that 25% of the system time in the total time btw?
> 

About 7% of total cpu cycles.

Tim