Re: [RFC] make sandbox UT more generic

[AKASHI Takahiro <takahiro.akashi@linaro.org>]

Hi Simon,

On Thu, Aug 31, 2023 at 09:04:43AM -0600, Simon Glass wrote:
> Hi,
> 
> On Wed, 30 Aug 2023 at 23:28, AKASHI Takahiro
> <takahiro.akashi@linaro.org> wrote:
> >
> > Hi Simon,
> >
> > On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> > > Hi,
> > >
> > > On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> > > <takahiro.akashi@linaro.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > > > and want to re-use sandbox UT to test the code. However,
> > > > It is somehow sandbox-specific (with additional DT nodes).
> > > > How can/should we make it more generic for other targets/drivers
> > > > rather than just by copying the test code?
> > > > (I have already created a test for pinmux since there is only
> > > > one existing scenario, but gpio test has many.)
> > > >
> > > > Even if I say 'generic', my case may be special since real
> > > > hardware (device drivers) cannot always run all the test cases,
> > > > while SCMI-based drivers potentially can with a dummy SCMI server
> > > > for sandbox.
> > > > See:
> > > >     drivers/firmware/scmi/sandbox-scmi_agent.c
> > >
> > > We don't have a good way to test drivers that talk to hardware, in general.
> > >
> > > For I2C, SPI and some PCI devices you can sometimes write an emulator
> > > for the chip and then your driver can talk to the emulator as if it
> > > were talking to the hardware. Sandbox does actually support that with
> > > memory-mapped I/O too, although it is fairly rarely used.
> >
> > Well, I don't want or need to emulate some *real* hardware.
> > Instead, I would like to emulate what the current sandbox drivers
> > (pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
> > (some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).
> >
> > As you might know, SCMI protocol with associated drivers on U-Boot is
> > so generic that it would be able to talk to any of real pinctrl/gpio
> > drivers/firmware (say, run on OPTEE or SCP).
> > By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
> > SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
> > similar to what sandbox does.
> 
> I actually know almost nothing about SCMI.
> 
> >
> > I have already implemented pinmux test with some tweaks by copying
> > test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
> > in test.dts.
> > But I'm looking for any other means without test code duplication.
> >
> > Did I clarify my question a bit?
> 
> Well you should be able to factor out the test code into a function,
> then call it from two places with the two different devices (or other
> params) that are needed.
> 
> For the DT, copying a few nodes is not the end of the world, IMO.
> 
> BTW have you seen this talk? [2] It seems that you are moving pieces
> into firmware which should be OS drivers?
> 
> Anyway, if you place a sandbox pinmux device under the SCMI node in
> the DT, then you should end up with a pinmux device you can use likely
> normal. Then if that device uses the sandbox emulator, you can run the
> existing tests on it with little modification, I suspect.
> 
> But if I am still missing the point, a diagram or patch might help me
> understand!

I just posted my RFC for supporting SCMI pinctrl protocol[1],
hoping it will help you understand what I'm planning to do regarding
test methodology, in particular by looking at patch#5 and #6.

[1] https://lists.denx.de/pipermail/u-boot/2023-September/529765.html

Thanks,
-Takahiro Akashi

> Regards,
> Simon
> 
> 
> >
> > -Takahiro Akashi
> >
> >
> > > We have done this a lot with Zephyr, as well[1] and achieved 90% code
> > > coverage on some boards.
> > >
> > > But I'm not quite sure I am answering the right question, so I will stop here.
> > >
> > > Regards,
> > > Simon
> > >
> > > [1] https://www.youtube.com/watch?v=usXCAXR2G_c
> 
> [2] https://www.usenix.org/conference/osdi21/presentation/fri-keynote