Re: [PATCH RFC net-next v4 2/2] selftests/dpll: add DPLL system integration selftests

[Jakub Kicinski <kuba@kernel.org>]

On Fri, 1 Dec 2023 18:33:11 +0000 Michalik, Michal wrote:
> That looks really promising - great idea. I tried only vmtest today, and my tests
> work kind of flawless with my own built kernel (nested VMs):
>   $ vmtest -k /home/net-next/vmlinux "modprobe netdevsim && KSRC=/home/net-next/ pytest"
>   => vmlinux
>   ===> Booting
>   ===> Setting up VM
>   ===> Running command  
>   ============================= test session starts ==============================
>   platform linux -- Python 3.9.16, pytest-7.4.3, pluggy-1.3.0
>   rootdir: /home/net-next/tools/testing/selftests/drivers/net/netdevsim/dpll
>   collected 91 items  
> 
>   test_dpll.py ........................................................... [ 64%]
>   ................................                                         [100%]
> 
>   ============================= 91 passed in 10.54s ==============================
> 
> I will try to take a look at virtme-ng next week, but to be frank I already like
> the vmtest.

Hm, FWIW I manged to get virtme-ng to work (I was pointing it at a
vmlinux not bzImage which it expects). But vmtest is still unhappy.

$ vmtest -k build/vmlinux "echo Running!"
=> vmlinux
===> Booting
Failed to connect QGA

Caused by:
    Timed out waiting for QGA connection


Are you on Ubuntu? I'm on Fedora. Maybe it has some distro deps :(

> >> To be honest I did not see that. I agree that this is a simpler solution, but I am
> >> not sure if that is not "too simple". What I mean, I'm not sure who wrote the sdsi
> >> tests, but maybe they were not aware about the Python best practices? Python used
> >> to be my first language, and I would vote for using the venvs if you asked me.
> >> I understand that it haven't been done before, but we are here to try to improve
> >> the things, yes?   
> > 
> > I think I already asked how long the setup takes but my only concern 
> > is that the setup will be slower, and less useful during development.
> 
> I wanted for "run_dpll_test.sh" to be userfriendly even for people who does not
> have a clue how python/pytest works. If somebody is developing tests, I assume
> he/she knows what she is doing and is using own environment either way, like
> venvs with additional Python debug tools and direct pytest in tests directory:
>   KSRC=<KERNEL SRC> pytest

Fair point.

> I don't feel like it is slowing anybody down. But since vmtest looks promising,
> maybe I can prepare a reverse logic. What I mean is I will prepare script which
> helps prepare the environment, but the default will be to use "locally installed
> stuff" when people just run "make -C tools/testing/selftests".

Let's keep it as is. 10sec for automated run is fine.