Re: Unexpected switches to in-band

[Philippe Gerum <rpm@xenomai.org>]

Philippe Gerum <rpm@xenomai.org> writes:

> Philippe Gerum <rpm@xenomai.org> writes:
>
>> Giulio Moro <giulio@bela.io> writes:
>>
>>>> It seems like latmus is trying at some point access some evicted from
>>>> cache memory page...
>>>
>>>> In my case I do use two simple test programs to allocate C++ <vector> -
>>>
>>> Thanks, that put me on a path to reliably reproduce this.
>>>
>>> Swap is disabled. I, then set up the system so that the oom-killer has is easy: it kills the allocating process. This is much faster at executing than going through the list of programs and picking the one with the worst score.
>>>
>>> echo 2 | sudo tee /proc/sys/vm/overcommit_memory
>>> echo 0 | sudo tee /proc/sys/vm/overcommit_ratio
>>> echo 1 | sudo tee /proc/sys/vm/oom_kill_allocating_task
>>>
>>> I then have a C++ program allocating 50MiB, and I run 4 or more instances of it, one per core:
>>>  while sleep 0.1; do ./alloc& ./alloc& ./alloc& ./alloc; done
>>>
>>> Furthermore, I have four instance of dd in the background:
>>>
>>> dd if=/dev/zero of=/dev/null
>>>
>>> With that, I can trigger latmus's inband switch pretty reliably within seconds (e.g.: latmus -m -K -p 360). If instead of latmus I run our application, it seems to be even faster and more reliable at triggering an in-band switch (once I set T_WOSS | T_WOLI | T_WOSX | T_HMSIG for the thread), and sigdebug_marked() confirms it is marked as sigdebug. While running it inside gdb I can inspect the backtrace upon receving the signal and it seems to be happening in seemingly harmless places. Most of the time it happens at some depth inside evl_usleep(), sometimes it happens inside libc's sinf(), sometimes somewhere else in our rt thread. I'd guess I just see it happen at random places, so the fact that it happens more often in evl_usleep() it's just because the thread spends 85% of the time in it. Note that our application never uses raw_copy_from_user(): the only call into the kernel from the real-time thread is via evl_usleep()
>>>
>>> It may be of interest that if I disabled (T_WOSS | T_WOLI | T_WOSX | T_HMSIG) in our application and thus it's free to keep running when receiving an ISW, I can see the number of ISW grows quickly in the first few seconds of execution to something like 20 but then remains constant . Similarly, latmus with -K seems to accumulate several (5 to 10) ISW at the beginning and then proceed without any further ISW for several minutes. They eventually occasionally occur again, but much more sparingly than in the first few seconds.
>>>
>>> For completeness, here's the C++ program I use for testing. I attempt to allocate memory in smaller chunks and get close as close as I can to filling up system memory across the four processes before the oom kills one of them.
>>>
>>> #include <vector>
>>>
>>> int main()
>>> {
>>> 	std::vector<std::vector<char>> all;
>>> 	for(unsigned int n = 0; n < 5; ++n)
>>> 	{
>>> 		all.emplace_back();
>>> 		all.back().resize(10 * 1024 * 1024);
>>> 	}
>>> 	return 0;
>>> }
>>
>> Ok, so it looks like both of you observe the same issue, which seems to
>> be arch-independent. Checking the code which takes care of preventing
>> COW for oob-enabled processes (a behavior which may cause unwanted minor
>> faults in a tricky way), I stumbled upon a really bad bug. Could you
>> check whether the patch below helps?
>>
>> diff --git a/include/linux/sched/coredump.h b/include/linux/sched/coredump.h
>> index 73de18353e79..c6b1efcbd833 100644
>> --- a/include/linux/sched/coredump.h
>> +++ b/include/linux/sched/coredump.h
>> @@ -91,7 +91,7 @@ static inline int get_dumpable(struct mm_struct *mm)
>>  
>>  #define MMF_VM_MERGE_ANY	30
>>  #define MMF_VM_MERGE_ANY_MASK	(1 << MMF_VM_MERGE_ANY)
>> -#define MMF_DOVETAILED		31	/* mm belongs to a dovetailed process */
>> +#define MMF_DOVETAILED		18	/* mm belongs to a dovetailed process */
>>  
>>  #define MMF_TOPDOWN		31	/* mm searches top down by default */
>>  #define MMF_TOPDOWN_MASK	(1 << MMF_TOPDOWN)
>
> Note: this issue only affects kernels from v6.10 and on

Actually, this issue only affects v6.12 among the kernel releases we
currently maintain, v6.16 is fine.

-- 
Philippe.