Re: Unexpected switches to in-band

["Łukasz Majewski" <lukma@nabladev.com>]

Hi Giulio,

> > It seems like latmus is trying at some point access some evicted
> > from cache memory page...  
> 
> > In my case I do use two simple test programs to allocate C++
> > <vector> -  
> 
> Thanks, that put me on a path to reliably reproduce this.
> 
> Swap is disabled. I, then set up the system so that the oom-killer
> has is easy: it kills the allocating process. This is much faster at
> executing than going through the list of programs and picking the one
> with the worst score.
> 
> echo 2 | sudo tee /proc/sys/vm/overcommit_memory
> echo 0 | sudo tee /proc/sys/vm/overcommit_ratio
> echo 1 | sudo tee /proc/sys/vm/oom_kill_allocating_task
> 
> I then have a C++ program allocating 50MiB, and I run 4 or more
> instances of it, one per core: 
> while sleep 0.1; do ./alloc& ./alloc& ./alloc& ./alloc; done
> 
> Furthermore, I have four instance of dd in the background:
> 
> dd if=/dev/zero of=/dev/null
> 
> With that, I can trigger latmus's inband switch pretty reliably
> within seconds (e.g.: latmus -m -K -p 360). If instead of latmus I
> run our application, it seems to be even faster and more reliable at
> triggering an in-band switch (once I set T_WOSS | T_WOLI | T_WOSX |
> T_HMSIG for the thread), and sigdebug_marked() confirms it is marked
> as sigdebug. While running it inside gdb I can inspect the backtrace
> upon receving the signal and it seems to be happening in seemingly
> harmless places. Most of the time it happens at some depth inside
> evl_usleep(), sometimes it happens inside libc's sinf(), sometimes
> somewhere else in our rt thread. I'd guess I just see it happen at
> random places, so the fact that it happens more often in evl_usleep()
> it's just because the thread spends 85% of the time in it. Note that
> our application never uses raw_copy_from_user(): the only call into
> the kernel from the real-time thread is via evl_usleep()
> 

I can also confirm that the issue appears at random places. Apparently
this is determined by the affected pages content.

Another viable observation is that:

1. This issue was not observed with Xenomai 3

2. After the "first" occurrence of the in-band switch in latmus, when
it is started again, the issue cannot be reproduced any more. The
conclusion here is that the issue is with "startup" or "fresh"
configuration of memory subsystem with dovetail.

> It may be of interest that if I disabled (T_WOSS | T_WOLI | T_WOSX |
> T_HMSIG) in our application and thus it's free to keep running when
> receiving an ISW, I can see the number of ISW grows quickly in the
> first few seconds of execution to something like 20 but then remains
> constant . Similarly, latmus with -K seems to accumulate several (5
> to 10) ISW at the beginning and then proceed without any further ISW
> for several minutes. They eventually occasionally occur again, but
> much more sparingly than in the first few seconds.
> 
> For completeness, here's the C++ program I use for testing. I attempt
> to allocate memory in smaller chunks and get close as close as I can
> to filling up system memory across the four processes before the oom
> kills one of them.
> 
> #include <vector>
> 
> int main()
> {
> 	std::vector<std::vector<char>> all;
> 	for(unsigned int n = 0; n < 5; ++n)
> 	{
> 		all.emplace_back();
> 		all.back().resize(10 * 1024 * 1024);
> 	}
> 	return 0;
> }



-- 
Best regards,

Lukasz Majewski

--
Nabla Software Engineering GmbH
HRB 40522 Augsburg
Phone: +49 821 45592596
E-Mail: office@nabladev.com
Geschftsfhrer : Stefano Babic