Re: Unexpected switches to in-band

["Łukasz Majewski" <lukma@nabladev.com>]

Hi Philippe, Giulio,

> Giulio Moro <giulio@bela.io> writes:
> 
> > Łukasz Majewski wrote on 20/10/2025 02:47:
> >  
> >> Could you share which version of libevl do you use?  
> >
> > I was using the latest release of libevl that was compatible with
> > the kernel UAPI. Sorry I haven't provided more details on the
> > issue; I am focusing on building an image around 6.1 for a deadline
> > coming up next month so I haven't been able to get into tracing yet.
> >
> > The only additional finding I have so far is that there seems to be
> > something on the Linux side that "breaks real-time" which affects
> > both linux and evl. In the below list, "evl bad" means that the
> > above mentioned ISW are observed. "Linux bad" means that I see a
> > disproportionate number of underruns under stress on a Linux program
> > running with SCHED_FIFO and priority 95 with a period of 360us. I
> > understand "disproportionate" is a very subjective term but, to give
> > an idea, over a 10 minutes test I get a couple of underruns with
> > "linux good" and I get hundreds of underruns with "linux good"
> >
> > v6.12.y-evl-rebase: evl bad, linux bad

The evl ps -l -> gives value of ISW 0 all the time.

> > v6.11.y-evl-rebase: evl bad, linux untested
> > v6.10.y-evl-rebase: evl bad, linux untested
> > v6.9.y-evl-rebase: evl bad, linux bad
> > v6.6.y-evl-rebase: evl bad, linux bad
> > v6.3.y-evl-rebase: evl bad, linux bad on startup only
> > v6.2.y-evl-rebase: libevl r42, evl good, linux good
> > v6.1.y-cip-evl-rebase: libevl master, evl good, linux good
> >
> > Not sure if this is of any help; I hope to be able to get back on
> > this soon. Best,
> > Giulio  

When switching to 6.12.y-evl-rebase [1] (including the MMF_DOVETAIL bit
fix with libevl r55) I cannot catch the issue anymore. For comparison -
I've fallback to 6.6 (with r50 libevl) with the same setup and I can
catch it.

Giulio, on what system do you run the test setup:

echo 2 | sudo tee /proc/sys/vm/overcommit_memory
echo 0 | sudo tee /proc/sys/vm/overcommit_ratio
echo 1 | sudo tee /proc/sys/vm/oom_kill_allocating_task

I then have a C++ program allocating 50MiB, and I run 4 or more
instances of it, one per core:
  
while sleep 0.1; do ./alloc& ./alloc& ./alloc& ./alloc; done

Furthermore, I have four instance of dd in the background:

dd if=/dev/zero of=/dev/null

With that, I can trigger latmus's inband switch pretty reliably within
seconds (e.g.: latmus -m -K -p 360)

As on mine the write to:
echo 0 | sudo tee /proc/sys/vm/overcommit_ratio

just makes the system unusable:
./run_tst.sh: fork: Cannot allocate memory



> 
> If someone could send me the relevant portion of a trace file with a
> 'latspot' tracepoint triggered on a latmus run, I could investigate
> this issue. I'd need the function tracer active on all CPUs, with all
> traces dumped to a single trace file ('evl trace -ef' should do).
> 

I strive to have the issue reproductible on 6.12.

Links:
[1] -
https://gitlab.com/Xenomai/xenomai4/linux-evl/-/commits/v6.12.y-evl-rebase?ref_type=heads

-- 
Best regards,

Lukasz Majewski

--
Nabla Software Engineering GmbH
HRB 40522 Augsburg
Phone: +49 821 45592596
E-Mail: office@nabladev.com
Geschftsfhrer : Stefano Babic