[LTP] [kirk] Remove support for external testing suite

[LTP] [kirk] Remove support for external testing suite

[Andrea Cervesato via ltp]

Hi all,

As you may know, Kirk supports the execution of certain kselftests and 
liburing testing suites. It was initially created as an experiment.

We have made attempts to implement these features, but it seems we have 
not succeeded. They appear to be largely unused by most users and are 
creating a bottleneck for new Kirk implementations, such as the 
automatic installation of LTP inside the SUT. This feature would be 
particularly beneficial for QEMU vanilla images. We had it in the past 
already and probably it's time to get this feature back. And we should 
probably implement it for other testing suites due to the current API: 
something that we really don't want to do (see eBPF kselftests compile 
madness).

Another concern is the maintainability of supporting for additional 
testing suites. While they are interesting, the implementation is 
somewhat flawed. Each testing suite has its own 
input/output/configuration/compilation methods, leading to confusion and 
over-engineered solutions. While we understand that no solution is 
perfect, it seems we have reached a point where Kirk = LTP. Therefore, 
we should reconsider maintaining features that require updates due to 
changes in external testing suites.

For these reasons, I propose that we remove support for the additional 
testing suites and focus exclusively on LTP for Kirk maintenance.

What do you think?

Best regards,
Andrea Cervesato

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [kirk] Remove support for external testing suite

[Cyril Hrubis]

Hi!
> As you may know, Kirk supports the execution of certain kselftests and 
> liburing testing suites. It was initially created as an experiment.
> 
> We have made attempts to implement these features, but it seems we have 
> not succeeded. They appear to be largely unused by most users and are 
> creating a bottleneck for new Kirk implementations, such as the 
> automatic installation of LTP inside the SUT. This feature would be 
> particularly beneficial for QEMU vanilla images. We had it in the past 
> already and probably it's time to get this feature back. And we should 
> probably implement it for other testing suites due to the current API: 
> something that we really don't want to do (see eBPF kselftests compile 
> madness).
> 
> Another concern is the maintainability of supporting for additional 
> testing suites. While they are interesting, the implementation is 
> somewhat flawed. Each testing suite has its own 
> input/output/configuration/compilation methods, leading to confusion and 
> over-engineered solutions. While we understand that no solution is 
> perfect, it seems we have reached a point where Kirk = LTP. Therefore, 
> we should reconsider maintaining features that require updates due to 
> changes in external testing suites.
> 
> For these reasons, I propose that we remove support for the additional 
> testing suites and focus exclusively on LTP for Kirk maintenance.
> 
> What do you think?

Sounds reasonable to me. There is quite a bit of work to be done in
order to implement all the advanced features in kirk and it does not
help if we get distracted by side quests.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [kirk] Remove support for external testing suite

[Andrea Cervesato via ltp]

Hi Cyril,

Thanks for your reply. I sent a pull request to fix this:

https://github.com/linux-test-project/kirk/pull/44

Andrea

On 3/31/25 18:17, Cyril Hrubis wrote:
> Hi!
>> As you may know, Kirk supports the execution of certain kselftests and
>> liburing testing suites. It was initially created as an experiment.
>>
>> We have made attempts to implement these features, but it seems we have
>> not succeeded. They appear to be largely unused by most users and are
>> creating a bottleneck for new Kirk implementations, such as the
>> automatic installation of LTP inside the SUT. This feature would be
>> particularly beneficial for QEMU vanilla images. We had it in the past
>> already and probably it's time to get this feature back. And we should
>> probably implement it for other testing suites due to the current API:
>> something that we really don't want to do (see eBPF kselftests compile
>> madness).
>>
>> Another concern is the maintainability of supporting for additional
>> testing suites. While they are interesting, the implementation is
>> somewhat flawed. Each testing suite has its own
>> input/output/configuration/compilation methods, leading to confusion and
>> over-engineered solutions. While we understand that no solution is
>> perfect, it seems we have reached a point where Kirk = LTP. Therefore,
>> we should reconsider maintaining features that require updates due to
>> changes in external testing suites.
>>
>> For these reasons, I propose that we remove support for the additional
>> testing suites and focus exclusively on LTP for Kirk maintenance.
>>
>> What do you think?
> Sounds reasonable to me. There is quite a bit of work to be done in
> order to implement all the advanced features in kirk and it does not
> help if we get distracted by side quests.
>

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp