Re: [RFC PATCH 00/23] NXP i.MX RT595, ARM SVD and device model unit tests

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Aug 12, 2024 at 05:10:52PM +0100, Peter Maydell wrote:
> On Mon, 5 Aug 2024 at 21:17, Octavian Purdila <tavip@google.com> wrote:
> > It also introduces unit tests for device models. To allow accessing
> > registers from unit tests a system bus mock is created. The main
> > advantage of unit tests for device models over QTest is that device
> > models can be tested in isolation and do not require a full qemu
> > machine.
> 
> On the other hand the disadvantage is that you need to add a
> bunch of extra code to mock the interfaces that the device
> connects to, and then you compile into the test binary a
> subset of C files which weren't written with the expectation
> that they got compiled into tests like that, so it feels a
> bit brittle. The nice thing about qtest is that it doesn't
> need you to do any of that -- you just run the QEMU machine
> model and prod the devices it already has.

Yep, mocking often ends up being rather a double-edged sword.
You can do some really powerful things with it, and in particular
it can enable you test scenarios / code paths that are hard or
impractical to exercise with regular functional tests where fault
injection is difficult/impossible.

The cost of this though is that the mocks can pose a significant
ongoing maint burden on anyone who refactors code in future, as
the mocks need to evolve in lockstep with the code refactoring.

12 months down the line, it can be hard for another maintainer
to understand the full purpose of the mocks & thus even harder
to know what todo as they refactor the code. The future burden
of mocks rarely falls on the person who creates them in the
first place.

> Do you have examples in this series of tests which you
> were able to write with this unit test setup that you
> wouldn't have been able to write equivalents of with the
> qtest framework ?



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|