Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

["Jorge Ramirez-Ortiz, Gmail" <jorge.ramirez.ortiz@gmail.com>]

On 03/09/25 07:29:07, Jan Kiszka wrote:
> On 01.11.24 17:55, Jorge Ramirez-Ortiz, Gmail wrote:
> > On 27/10/24 20:13:08, Jorge Ramirez-Ortiz, Gmail wrote:
> >> On 24/10/24 20:00:12, Schaffner, Tobias wrote:
> >>> Hi Philippe, hi Jorge,
> >>>
> >>> On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
> >>>> Hey Jorge,
> >>>>
> >>>> On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> >>>>> On 13/06/24 19:27:10, Philippe Gerum wrote:
> >>>>>>
> >>>>>> Philippe Gerum <rpm@xenomai.org> writes:
> >>>>>>
> >>>>>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> >>>>>>>
> >>>>>>>> Stress evl with a load command while running the tests. The
> >>>>>>>> load command
> >>>>>>>> may be set to an empty string to run tests without stressing
> >>>>>>>> the system.
> >>>>>>>>
> >>>>>>>> To align with xeno-test the -l command line argument was used
> >>>>>>>> for the load
> >>>>>>>> command. Listing of the unittests can now be done with --list
> >>>>>>>> and --List.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Nak. Compat with xeno-test is definitely not a requirement if
> >>>>>>> this means
> >>>>>>> breaking the existing evl command line usage. Besides, --List
> >>>>>>> looks
> >>>>>>> strange as an option. Let's pick a different option for the
> >>>>>>> load command
> >>>>>>> instead.
> >>>>>>>
> >>>>>>
> >>>>>> We may want to implement the stress utility as a separate evl-
> >>>>>> stress
> >>>>>> command, keeping evl-test for unit testing which has a different
> >>>>>> purpose. evl-stress as a script could generate the load (dohell)
> >>>>>> and
> >>>>>> possibly check the latency figures reported by latmus (using the
> >>>>>> -K and
> >>>>>> -A switches to detect weird behavior).
> >>>>>>
> >>>>>> PS: Jorge is going to look into leveraging stress-ng for load
> >>>>>> generation.
> >>>>>
> >>>>> yes I am going to start looking into it. I think that maintaining
> >>>>> Linux
> >>>>> stressors should not be Xenomai's business. If we can't replicate
> >>>>> the
> >>>>> load with todays stress-ng perhaps we could just usptream what is
> >>>>> needed (stress-ng is well maintained)
> >>>>
> >>>
> >>> may I bring this up again? Any new developments or opinions on this?
> >>> I would like to see a solution that allows to compare xenomai 3 and
> >>> EVL. Reliable numbers are necessary to make an informed decision if EVL
> >>> is on par or even better than xenomai 3. Imo we are comparing apples
> >>> and oranges as long as we are not using the same tooling on both
> >>> platforms.
> >>
> >> Sorry about this, it is being a bit of a crazy year for me at a personal
> >> levely. I am start on it this week.
> >>
> >> My setup uses some Zephyr code executing on frdmk64 I wrote years ago to
> >> benchmark evl
> >>
> >> https://github.com/ldts/libevl/commit/905d1946ca117e3889c53d1150a1334cc91033a8
> >>
> >> I will load my evl system using stress-ng and measure latency figures
> >> and then replicate with dohell loading tools. I should have something to
> >> share by next friday.
> >>
> >> feel free to ping me anytime. In IRC you can find me in LiberaChat
> >> #u-boot (nick ldts) and since recently in OFTC #edk2
> > 
> > I was finally able to get some numbers on imx8mm (an Arduino Pro with
> > the Portenta breakout board recently contributed to my lab by Wim
> > Taymans for audio testing of the Xenomai4 Pipewire-plugin[0] for
> > realtime audio)
> > 
> > I am using dohell/rt-tools/ltp and stress-ng on EVL - havent tested
> > Xenomai3.
> > 
> > My test bench runs with the help of an external FRDMK64F running Zephyr
> > very similar to what is described in the latmus/latmon combo
> > documentation [1]. Given how cheap this board is perhaps it should be
> > more widely used (?).
> > 
> > [RFC]
> > I would like to update latmon[2] to the tip of Zephyr - it has fallen
> > behind - I'd like then to propose it upstream (if accepted I would like
> > to propose that it is removed from EVL). Anyone disagrees?
> > 
> > IMO We should also take latmus to its own separate tree dedicated to RT
> > benchmarking and extend it to support scheduling latency metrics based
> > on GPIO data.
> > [\]
> > 
> > So:
> > 
> > * stress-ng --vm 2 --vm-bytes 1G --mmap 2 --mmap-bytes 1G --page-in \
> >   --matrix 0 --matrix-size 64 --tz -t 60
> > 
> > This simple command (VM and cpu load for a 2GB DDR system) hits the
> > system very quick and my externally measured GPIO IRQ latency peaks
> > within seconds.
> > 
> > stress-ng has FPU, VM, CPU all sort of stressors but I didnt need to go
> > into those just yet.
> > 
> > I found some info that RH [3] is providing interesting - still the
> > stress-ng code is simple enough to work with if needed (really good
> > encapsulation and code base)
> > 
> > * dohell -l /opt/ltp -b /bin/hackbench -m /home/root
> > 
> > After five minutes of run the latency is still contained (far from
> > peaking, 50% below the peak) but I assume this is because LTP takes a
> > while to run all tests. Without going into more details, this difference
> > in the invested time seems enough for me to switch to stress-ng.
> > 
> 
> I think we should split these activities into two streams:
>  - short-term: setup of a well-defined but not necessarily 100%
>    finalized test environment for EVL so that we can run that
>    automatically in CI

100%. Do you think it would be at all possible to setup a simple LAVA
lab with some reference hardware - unless Xenomai has one already?

I am not sure how it would be funded - do you know ? - but it could then
also be connected to KernelCI so that Xenomai enabled systems avoid
regressions.

https://docs.kernelci.org/intro/platform-testing/

Is this something at all possible you think?


>  - long-term: evaluation of background load profiles and tests to
>    trigger maximal latencies, with EVL, Xenomai 3 and may even
>    PREEMPT_RT
> 
> With the first item solved, the second could even be automated and may
> not need some poor intern to press buttons and record numbers on a piece
> of paper ;).
> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center