API/syscall to alleviate page/memory problem when quickly accessing memory?

[Levo D <l-asm@mail9fcb1a.bolinlang.com>]

I optimized and profile my program to a point that it seems like it's spending more time in kernel than in userspace (likely not true but I'll explain).

Here's one run. I spawn many threads (6 at minimum, more depending on flags). As you can see more than half of the total time is in sys. Is the kernel running on multiple cores simultaneously to give my program pages?

real	0m0.954s
user	0m6.442s
sys 	0m0.607s

The test below is using -test-flags which gets me these numbers, sys is 51% of total time

real	0m0.733s
user	0m3.476s
sys 	0m0.378s


perf record -F 5000 ./myapp -test-flags shows me 61% of the app is in my biggest function and 6% is in `clear_page_rep`. When I record cache misses using `perf record -F 5000 --call-graph=fp -e cache-misses ./myapp -test-flags` I can see that

clear_page_rep takes 40%
clear_huge_page takes 1.2%
My big function self is 8%, while total is 25.5%. the remaining is mostly asm_exc_page_fault (12%) and asm_sysvec_apic_timer_interrupt (2.7%)
That's about 56% (of all misses and waiting) in the kernel

I believe if I can reduce work being done in the kernel and have pages be ready before I fault I'll have less cache misses in my large function and I could be significantly faster. I measured how long my large function takes in single threaded compared to multi. Multithreaded at minimum is 1.5x slower to 2x slower. I spawn 1 thread per core (I'm testing on a zen2, it has 6cores with 12threads, spawning more than 6 threads slow the program down). Each thread is using <100MB.

Is there an API I should look into? What can I do here?