Re: Unexpected switches to in-band

["Łukasz Majewski" <lukma@nabladev.com>]

Hi Philippe,

> Giulio Moro <giulio@bela.io> writes:
> 
> > Łukasz Majewski wrote on 20/10/2025 02:47:
> >  
> >> Could you share which version of libevl do you use?  
> >
> > I was using the latest release of libevl that was compatible with
> > the kernel UAPI. Sorry I haven't provided more details on the
> > issue; I am focusing on building an image around 6.1 for a deadline
> > coming up next month so I haven't been able to get into tracing yet.
> >
> > The only additional finding I have so far is that there seems to be
> > something on the Linux side that "breaks real-time" which affects
> > both linux and evl. In the below list, "evl bad" means that the
> > above mentioned ISW are observed. "Linux bad" means that I see a
> > disproportionate number of underruns under stress on a Linux program
> > running with SCHED_FIFO and priority 95 with a period of 360us. I
> > understand "disproportionate" is a very subjective term but, to give
> > an idea, over a 10 minutes test I get a couple of underruns with
> > "linux good" and I get hundreds of underruns with "linux good"
> >
> > v6.12.y-evl-rebase: evl bad, linux bad
> > v6.11.y-evl-rebase: evl bad, linux untested
> > v6.10.y-evl-rebase: evl bad, linux untested
> > v6.9.y-evl-rebase: evl bad, linux bad
> > v6.6.y-evl-rebase: evl bad, linux bad
> > v6.3.y-evl-rebase: evl bad, linux bad on startup only
> > v6.2.y-evl-rebase: libevl r42, evl good, linux good
> > v6.1.y-cip-evl-rebase: libevl master, evl good, linux good
> >
> > Not sure if this is of any help; I hope to be able to get back on
> > this soon. Best,
> > Giulio  
> 
> If someone could send me the relevant portion of a trace file with a
> 'latspot' tracepoint triggered on a latmus run, I could investigate
> this issue. I'd need the function tracer active on all CPUs, with all
> traces dumped to a single trace file ('evl trace -ef' should do).
> 

Please find tar'ed output for the trace(s).

Please, however be aware that - I've fall back to 6.6 (as it is the
version in which I can reproduce the issue in the fastest way).

Customer also reported, that they can reproduce with their SW stack the
issue on 6.1-slts and 6.12, but it takes considerably longer than for
6.6 (in which I can use simple programs to "allocate" memory).

I've used pretty standard set of ftrace CONFIG_* options enabled.

However, it seems like there is a "hole" around the time when in-band
switch has been reported (in dmesg) and in ftrace output.

I'm going to do the same with all available tracers enabled.

Last but not least, the latspot event is not present in my ftrace
output (although I've enabled all the CONFIG_EVL*DEBUG options).

Is there any special set of options to required for EVL tracig?

Tars with logs:
https://nextcloud.swupdate.org/index.php/s/FgiMsHG9xG8frk3
https://nextcloud.swupdate.org/index.php/s/XcW75xsQPMXm3zg


-- 
Best regards,

Lukasz Majewski

--
Nabla Software Engineering GmbH
HRB 40522 Augsburg
Phone: +49 821 45592596
E-Mail: office@nabladev.com
Geschftsfhrer : Stefano Babic