Re: [External] - Re: Conflicting EVL Processing Loops

[Philippe Gerum <rpm@xenomai.org>]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Philippe,
>
> An update and question for you.
>
> First, as you know, we found that the built-in heap management in Xenomai
> 4/EVL was causing us substantial performance problems, due to the need to
> perform locking on all memory allocations. The nolock API is not an option
> for us since we perform memory operations in most of our threads. We also
> tried the TLSF manager, but saw similar performance issues.
>
> The good news is that we've adapted the mimalloc memory management library,
> which I believe implements something like the fast bins you mention in your
> earlier email. The performance of mimalloc looks to be very good, and we are
> able to get our dual processing loop system running within our real-time
> constraints. The current implementation is still a bit "hackish", and we're
> continuing to test it and clean it up. I am hoping to get you the specifics
> about what we had to do to implement it, since I think it could be a good
> option for other X4/EVL users. At a high level, it is essentially a
> "go-between" with the Xenomai heap management at the bottom layer, replacing
> the low-level sbrk() used to get memory in a Linux run-time environment.
>
> One nagging problem is that we're still plagued by occasional page faults.
> We have tried to prefault everything we can think of, but we're obviously
> missing something. I go through each accessible section in the
> /proc/self/maps file, prefaulting each one (this includes all code and data
> segments). I have also added a hack to the kernel so that when the "switched
> inband (fault)" occurs, the faulting address is displayed in dmesg. So far,
> all of the runtime page faults we see are in the heap section, which I have
> attempted to prefault completely, even doing it multiple times during
> startup, since the heap section seems to be growing as we start up our real
> time threads. 
>
> So, I'm looking at one of 2 possibilities:
> 1. My prefaulting code, which touches one memory location in each page, is
> not actually doing what it is supposed to. I've declared variables in the
> prefaulting function to be "volatile" so that they don't get optimized out.
> But I don't know any way to really verify that the pages are being mapped in
> and locked.
> 2. A kernel bug, where the pages are not, in fact, being locked into memory.
> We're calling "mlockall(MCL_CURRENT | MCL_FUTURE)", so, even if the heap is
> growing, I don't understand why any future pages are not being populated and
> locked into memory at the very beginning. And, the kernel should not be
> unmapping any of our pages, but perhaps it is?
>
> I know this isn't likely a problem with the EVL code, but we're just about
> out of ideas for how to find and kill this problem. I'm not much of a kernel
> expert. If you have any ideas for how to isolate this problem, especially if
> there's a way to verify whether our process pages are really locked or not,
> they would be greatly appreciated.
>
> [[End of S/MIME Signed Part]]

Mlocked pages may be migrated (Documentation/vm/unevictable-lru.txt
gives details about this), in which case such pages would not be immune
from minor faults when accessed anew after migration, which may be the
events the core detects.

NUMA support and transparent huge pages are the usual suspects in this
case (CONFIG_NUMA, CONFIG_TRANSPARENT_HUGEPAGE), and/or memory
compaction (CONFIG_COMPACTION).

If turning these off is not a suitable option, you could try to fiddle
with the related runtime settings to see what helps,

e.g.

sysctl -w kernel.numa_balancing=0
echo 0 > /proc/sys/vm/compact_unevictable_allowed

Bottom line is that you would need to stop vmscan from invalidating the
page table entries of the mlocked pages (at the expense of less
flexibility in memory management, but that may not be the most important
issue at hand in this case).

-- 
Philippe.