btrfs and numa - needing drop_caches to keep speed up

btrfs and numa - needing drop_caches to keep speed up

[Stefan Priebe - Profihost AG]

Hello list,

while running the same workload on two machines (single xeon and a dual
xeon) both with 64GB RAM.

I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to
keep the speed as good as on the non numa system. I'm not sure whether
this is related to numa.

Is there any sysctl parameter to tune?

Tested with vanilla v4.8.1

Greets,
Stefan

Re: btrfs and numa - needing drop_caches to keep speed up

[Julian Taylor]

On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> while running the same workload on two machines (single xeon and a dual
> xeon) both with 64GB RAM.
>
> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to
> keep the speed as good as on the non numa system. I'm not sure whether
> this is related to numa.
>
> Is there any sysctl parameter to tune?
>
> Tested with vanilla v4.8.1
>
> Greets,
> Stefan

hi,
why do you think this is related to btrfs?

The only known issue that has this type of workaround that I know of are 
transparent huge pages.
This is easy to diagnose but recording some kernel stacks during the 
problem with perf.
If there is very high system cpu usage in a spinlock called by 
compaction functions during page faults it is the synchronous memory 
defragmentation needed for thp.
Should that be the case the better workaround is disabling it in 
/sys/kernel/mm/transparent_hugepage/defrag

cheers,
Julian

Re: btrfs and numa - needing drop_caches to keep speed up

[Stefan Priebe - Profihost AG]

Dear julian,

Am 14.10.2016 um 14:26 schrieb Julian Taylor:
> On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> while running the same workload on two machines (single xeon and a dual
>> xeon) both with 64GB RAM.
>>
>> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to
>> keep the speed as good as on the non numa system. I'm not sure whether
>> this is related to numa.
>>
>> Is there any sysctl parameter to tune?
>>
>> Tested with vanilla v4.8.1
>>
>> Greets,
>> Stefan
> 
> hi,
> why do you think this is related to btrfs?

was just an idea as i couldn't find any other difference between those
systems.

> This is easy to diagnose but recording some kernel stacks during the >
problem with perf.

you just mean perf top? Does it also show locking problems? As i see not
much CPU usage in that case.

> The only known issue that has this type of workaround that I know of are
> transparent huge pages.

I already disabled thp by:
echo never > /sys/kernel/mm/transparent_hugepage/enabled

cat /proc/meminfo says:
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0



Greets,
Stefan

> 
> cheers,
> Julian

Re: btrfs and numa - needing drop_caches to keep speed up

[Stefan Priebe - Profihost AG]

Hi,
Am 14.10.2016 um 15:19 schrieb Stefan Priebe - Profihost AG:
> Dear julian,
> 
> Am 14.10.2016 um 14:26 schrieb Julian Taylor:
>> On 10/14/2016 08:28 AM, Stefan Priebe - Profihost AG wrote:
>>> Hello list,
>>>
>>> while running the same workload on two machines (single xeon and a dual
>>> xeon) both with 64GB RAM.
>>>
>>> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to
>>> keep the speed as good as on the non numa system. I'm not sure whether
>>> this is related to numa.
>>>
>>> Is there any sysctl parameter to tune?
>>>
>>> Tested with vanilla v4.8.1
>>>
>>> Greets,
>>> Stefan
>>
>> hi,
>> why do you think this is related to btrfs?
> 
> was just an idea as i couldn't find any other difference between those
> systems.
> 
>> This is easy to diagnose but recording some kernel stacks during the >
> problem with perf.
> 
> you just mean perf top? Does it also show locking problems? As i see not
> much CPU usage in that case.


perf top looks like this:
   5,46%  libc-2.19.so           [.] memset
   5,26%  [kernel]               [k] page_fault
   3,63%  [kernel]               [k] clear_page_c_e
   1,38%  [kernel]               [k] _raw_spin_lock
   1,06%  [kernel]               [k] get_page_from_freelist
   0,83%  [kernel]               [k] copy_user_enhanced_fast_string
   0,79%  [kernel]               [k] release_pages
   0,68%  [kernel]               [k] handle_mm_fault
   0,57%  [kernel]               [k] free_hot_cold_page
   0,55%  [kernel]               [k] handle_pte_fault
   0,54%  [kernel]               [k] __pagevec_lru_add_fn
   0,45%  [kernel]               [k] unmap_page_range
   0,45%  [kernel]               [k] __mod_zone_page_state
   0,43%  [kernel]               [k] page_add_new_anon_rmap
   0,38%  [kernel]               [k] free_pcppages_bulk

> 
>> The only known issue that has this type of workaround that I know of are
>> transparent huge pages.
> 
> I already disabled thp by:
> echo never > /sys/kernel/mm/transparent_hugepage/enabled
> 
> cat /proc/meminfo says:
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> 
> 
> 
> Greets,
> Stefan
> 
>>
>> cheers,
>> Julian

Re: btrfs and numa - needing drop_caches to keep speed up

[Austin S. Hemmelgarn]

On 2016-10-14 02:28, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> while running the same workload on two machines (single xeon and a dual
> xeon) both with 64GB RAM.
>
> I need to run echo 3 >/proc/sys/vm/drop_caches every 15-30 minutes to
> keep the speed as good as on the non numa system. I'm not sure whether
> this is related to numa.
>
> Is there any sysctl parameter to tune?
>
> Tested with vanilla v4.8.1
This may sound odd, but does echoing 1 or 2 to /proc/sys/vm/drop_caches 
help at all, or is it just 3?   The value itself is actually a bit-field 
with just two bits defined.  1 just drops the page-cache, while 2 just 
drops freeable SLAB objects, and 3 drops both.  The thing is, both have 
an impact when dealing with filesystems (page-cache contains cached file 
contents, while freeable SLAB objects includes cached dentries and 
inodes), so knowing whether one or the other or only both is what's 
helping things can help diagnose this further.

Regardless of that, you might try adjusting vm.vfs_cache_pressure. 
Increasing that will make the page-cache reclaim more aggressive, while 
decreasing it will make it less aggressive.  It defaults to 100, and I 
wouldn't suggest setting it below 50 or above about 150.  Keep in mind 
that increasing that will mean you're likely to put more load on the 
storage devices.