[RFC] make sandbox UT more generic

[RFC] make sandbox UT more generic

[AKASHI Takahiro]

Hi,

I'm working on implementing SCMI-based pinctrl/gpio driver,
and want to re-use sandbox UT to test the code. However,
It is somehow sandbox-specific (with additional DT nodes).
How can/should we make it more generic for other targets/drivers
rather than just by copying the test code?
(I have already created a test for pinmux since there is only
one existing scenario, but gpio test has many.)

Even if I say 'generic', my case may be special since real
hardware (device drivers) cannot always run all the test cases,
while SCMI-based drivers potentially can with a dummy SCMI server
for sandbox.
See:
    drivers/firmware/scmi/sandbox-scmi_agent.c

Thanks,
-Takahiro Akashi

Re: [RFC] make sandbox UT more generic

[Simon Glass]

Hi,

On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
<takahiro.akashi@linaro.org> wrote:
>
> Hi,
>
> I'm working on implementing SCMI-based pinctrl/gpio driver,
> and want to re-use sandbox UT to test the code. However,
> It is somehow sandbox-specific (with additional DT nodes).
> How can/should we make it more generic for other targets/drivers
> rather than just by copying the test code?
> (I have already created a test for pinmux since there is only
> one existing scenario, but gpio test has many.)
>
> Even if I say 'generic', my case may be special since real
> hardware (device drivers) cannot always run all the test cases,
> while SCMI-based drivers potentially can with a dummy SCMI server
> for sandbox.
> See:
>     drivers/firmware/scmi/sandbox-scmi_agent.c

We don't have a good way to test drivers that talk to hardware, in general.

For I2C, SPI and some PCI devices you can sometimes write an emulator
for the chip and then your driver can talk to the emulator as if it
were talking to the hardware. Sandbox does actually support that with
memory-mapped I/O too, although it is fairly rarely used.

We have done this a lot with Zephyr, as well[1] and achieved 90% code
coverage on some boards.

But I'm not quite sure I am answering the right question, so I will stop here.

Regards,
Simon

[1] https://www.youtube.com/watch?v=usXCAXR2G_c

Re: [RFC] make sandbox UT more generic

[AKASHI Takahiro]

Hi Simon,

On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> Hi,
> 
> On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> <takahiro.akashi@linaro.org> wrote:
> >
> > Hi,
> >
> > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > and want to re-use sandbox UT to test the code. However,
> > It is somehow sandbox-specific (with additional DT nodes).
> > How can/should we make it more generic for other targets/drivers
> > rather than just by copying the test code?
> > (I have already created a test for pinmux since there is only
> > one existing scenario, but gpio test has many.)
> >
> > Even if I say 'generic', my case may be special since real
> > hardware (device drivers) cannot always run all the test cases,
> > while SCMI-based drivers potentially can with a dummy SCMI server
> > for sandbox.
> > See:
> >     drivers/firmware/scmi/sandbox-scmi_agent.c
> 
> We don't have a good way to test drivers that talk to hardware, in general.
> 
> For I2C, SPI and some PCI devices you can sometimes write an emulator
> for the chip and then your driver can talk to the emulator as if it
> were talking to the hardware. Sandbox does actually support that with
> memory-mapped I/O too, although it is fairly rarely used.

Well, I don't want or need to emulate some *real* hardware.
Instead, I would like to emulate what the current sandbox drivers
(pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
(some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).

As you might know, SCMI protocol with associated drivers on U-Boot is
so generic that it would be able to talk to any of real pinctrl/gpio
drivers/firmware (say, run on OPTEE or SCP).
By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
similar to what sandbox does.

I have already implemented pinmux test with some tweaks by copying
test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
in test.dts.
But I'm looking for any other means without test code duplication.

Did I clarify my question a bit?

-Takahiro Akashi


> We have done this a lot with Zephyr, as well[1] and achieved 90% code
> coverage on some boards.
> 
> But I'm not quite sure I am answering the right question, so I will stop here.
> 
> Regards,
> Simon
> 
> [1] https://www.youtube.com/watch?v=usXCAXR2G_c

Re: [RFC] make sandbox UT more generic

[Simon Glass]

Hi,

On Wed, 30 Aug 2023 at 23:28, AKASHI Takahiro
<takahiro.akashi@linaro.org> wrote:
>
> Hi Simon,
>
> On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> > Hi,
> >
> > On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> > <takahiro.akashi@linaro.org> wrote:
> > >
> > > Hi,
> > >
> > > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > > and want to re-use sandbox UT to test the code. However,
> > > It is somehow sandbox-specific (with additional DT nodes).
> > > How can/should we make it more generic for other targets/drivers
> > > rather than just by copying the test code?
> > > (I have already created a test for pinmux since there is only
> > > one existing scenario, but gpio test has many.)
> > >
> > > Even if I say 'generic', my case may be special since real
> > > hardware (device drivers) cannot always run all the test cases,
> > > while SCMI-based drivers potentially can with a dummy SCMI server
> > > for sandbox.
> > > See:
> > >     drivers/firmware/scmi/sandbox-scmi_agent.c
> >
> > We don't have a good way to test drivers that talk to hardware, in general.
> >
> > For I2C, SPI and some PCI devices you can sometimes write an emulator
> > for the chip and then your driver can talk to the emulator as if it
> > were talking to the hardware. Sandbox does actually support that with
> > memory-mapped I/O too, although it is fairly rarely used.
>
> Well, I don't want or need to emulate some *real* hardware.
> Instead, I would like to emulate what the current sandbox drivers
> (pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
> (some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).
>
> As you might know, SCMI protocol with associated drivers on U-Boot is
> so generic that it would be able to talk to any of real pinctrl/gpio
> drivers/firmware (say, run on OPTEE or SCP).
> By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
> SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
> similar to what sandbox does.

I actually know almost nothing about SCMI.

>
> I have already implemented pinmux test with some tweaks by copying
> test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
> in test.dts.
> But I'm looking for any other means without test code duplication.
>
> Did I clarify my question a bit?

Well you should be able to factor out the test code into a function,
then call it from two places with the two different devices (or other
params) that are needed.

For the DT, copying a few nodes is not the end of the world, IMO.

BTW have you seen this talk? [2] It seems that you are moving pieces
into firmware which should be OS drivers?

Anyway, if you place a sandbox pinmux device under the SCMI node in
the DT, then you should end up with a pinmux device you can use likely
normal. Then if that device uses the sandbox emulator, you can run the
existing tests on it with little modification, I suspect.

But if I am still missing the point, a diagram or patch might help me
understand!

Regards,
Simon


>
> -Takahiro Akashi
>
>
> > We have done this a lot with Zephyr, as well[1] and achieved 90% code
> > coverage on some boards.
> >
> > But I'm not quite sure I am answering the right question, so I will stop here.
> >
> > Regards,
> > Simon
> >
> > [1] https://www.youtube.com/watch?v=usXCAXR2G_c

[2] https://www.usenix.org/conference/osdi21/presentation/fri-keynote

Re: [RFC] make sandbox UT more generic

[AKASHI Takahiro]

Hi Simon,

On Thu, Aug 31, 2023 at 09:04:43AM -0600, Simon Glass wrote:
> Hi,
> 
> On Wed, 30 Aug 2023 at 23:28, AKASHI Takahiro
> <takahiro.akashi@linaro.org> wrote:
> >
> > Hi Simon,
> >
> > On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> > > Hi,
> > >
> > > On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> > > <takahiro.akashi@linaro.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > > > and want to re-use sandbox UT to test the code. However,
> > > > It is somehow sandbox-specific (with additional DT nodes).
> > > > How can/should we make it more generic for other targets/drivers
> > > > rather than just by copying the test code?
> > > > (I have already created a test for pinmux since there is only
> > > > one existing scenario, but gpio test has many.)
> > > >
> > > > Even if I say 'generic', my case may be special since real
> > > > hardware (device drivers) cannot always run all the test cases,
> > > > while SCMI-based drivers potentially can with a dummy SCMI server
> > > > for sandbox.
> > > > See:
> > > >     drivers/firmware/scmi/sandbox-scmi_agent.c
> > >
> > > We don't have a good way to test drivers that talk to hardware, in general.
> > >
> > > For I2C, SPI and some PCI devices you can sometimes write an emulator
> > > for the chip and then your driver can talk to the emulator as if it
> > > were talking to the hardware. Sandbox does actually support that with
> > > memory-mapped I/O too, although it is fairly rarely used.
> >
> > Well, I don't want or need to emulate some *real* hardware.
> > Instead, I would like to emulate what the current sandbox drivers
> > (pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
> > (some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).
> >
> > As you might know, SCMI protocol with associated drivers on U-Boot is
> > so generic that it would be able to talk to any of real pinctrl/gpio
> > drivers/firmware (say, run on OPTEE or SCP).
> > By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
> > SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
> > similar to what sandbox does.
> 
> I actually know almost nothing about SCMI.
> 
> >
> > I have already implemented pinmux test with some tweaks by copying
> > test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
> > in test.dts.
> > But I'm looking for any other means without test code duplication.
> >
> > Did I clarify my question a bit?
> 
> Well you should be able to factor out the test code into a function,
> then call it from two places with the two different devices (or other
> params) that are needed.
> 
> For the DT, copying a few nodes is not the end of the world, IMO.
> 
> BTW have you seen this talk? [2] It seems that you are moving pieces
> into firmware which should be OS drivers?
> 
> Anyway, if you place a sandbox pinmux device under the SCMI node in
> the DT, then you should end up with a pinmux device you can use likely
> normal. Then if that device uses the sandbox emulator, you can run the
> existing tests on it with little modification, I suspect.
> 
> But if I am still missing the point, a diagram or patch might help me
> understand!

I just posted my RFC for supporting SCMI pinctrl protocol[1],
hoping it will help you understand what I'm planning to do regarding
test methodology, in particular by looking at patch#5 and #6.

[1] https://lists.denx.de/pipermail/u-boot/2023-September/529765.html

Thanks,
-Takahiro Akashi

> Regards,
> Simon
> 
> 
> >
> > -Takahiro Akashi
> >
> >
> > > We have done this a lot with Zephyr, as well[1] and achieved 90% code
> > > coverage on some boards.
> > >
> > > But I'm not quite sure I am answering the right question, so I will stop here.
> > >
> > > Regards,
> > > Simon
> > >
> > > [1] https://www.youtube.com/watch?v=usXCAXR2G_c
> 
> [2] https://www.usenix.org/conference/osdi21/presentation/fri-keynote

Re: [RFC] make sandbox UT more generic

[Simon Glass]

Hi,

On Tue, 5 Sept 2023 at 21:00, AKASHI Takahiro
<takahiro.akashi@linaro.org> wrote:
>
> Hi Simon,
>
> On Thu, Aug 31, 2023 at 09:04:43AM -0600, Simon Glass wrote:
> > Hi,
> >
> > On Wed, 30 Aug 2023 at 23:28, AKASHI Takahiro
> > <takahiro.akashi@linaro.org> wrote:
> > >
> > > Hi Simon,
> > >
> > > On Wed, Aug 30, 2023 at 08:49:05PM -0600, Simon Glass wrote:
> > > > Hi,
> > > >
> > > > On Wed, 30 Aug 2023 at 18:38, AKASHI Takahiro
> > > > <takahiro.akashi@linaro.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm working on implementing SCMI-based pinctrl/gpio driver,
> > > > > and want to re-use sandbox UT to test the code. However,
> > > > > It is somehow sandbox-specific (with additional DT nodes).
> > > > > How can/should we make it more generic for other targets/drivers
> > > > > rather than just by copying the test code?
> > > > > (I have already created a test for pinmux since there is only
> > > > > one existing scenario, but gpio test has many.)
> > > > >
> > > > > Even if I say 'generic', my case may be special since real
> > > > > hardware (device drivers) cannot always run all the test cases,
> > > > > while SCMI-based drivers potentially can with a dummy SCMI server
> > > > > for sandbox.
> > > > > See:
> > > > >     drivers/firmware/scmi/sandbox-scmi_agent.c
> > > >
> > > > We don't have a good way to test drivers that talk to hardware, in general.
> > > >
> > > > For I2C, SPI and some PCI devices you can sometimes write an emulator
> > > > for the chip and then your driver can talk to the emulator as if it
> > > > were talking to the hardware. Sandbox does actually support that with
> > > > memory-mapped I/O too, although it is fairly rarely used.
> > >
> > > Well, I don't want or need to emulate some *real* hardware.
> > > Instead, I would like to emulate what the current sandbox drivers
> > > (pinctrl-sandbox.c and gpio/sandbox.c) emulate so that we can re-use
> > > (some portion of) test cases for sandbox (test/dm/pinmux.c and gpio.c).
> > >
> > > As you might know, SCMI protocol with associated drivers on U-Boot is
> > > so generic that it would be able to talk to any of real pinctrl/gpio
> > > drivers/firmware (say, run on OPTEE or SCP).
> > > By implementing/mimicking protocol messages in sandbox-scmi_agent.c,
> > > SCMI drivers are expected to provide *virtual* pinctrl/gpio devices
> > > similar to what sandbox does.
> >
> > I actually know almost nothing about SCMI.
> >
> > >
> > > I have already implemented pinmux test with some tweaks by copying
> > > test/dm/pinmux.c and duplicating almost the same DT nodes as "pinctrl-gpio"
> > > in test.dts.
> > > But I'm looking for any other means without test code duplication.
> > >
> > > Did I clarify my question a bit?
> >
> > Well you should be able to factor out the test code into a function,
> > then call it from two places with the two different devices (or other
> > params) that are needed.
> >
> > For the DT, copying a few nodes is not the end of the world, IMO.
> >
> > BTW have you seen this talk? [2] It seems that you are moving pieces
> > into firmware which should be OS drivers?
> >
> > Anyway, if you place a sandbox pinmux device under the SCMI node in
> > the DT, then you should end up with a pinmux device you can use likely
> > normal. Then if that device uses the sandbox emulator, you can run the
> > existing tests on it with little modification, I suspect.
> >
> > But if I am still missing the point, a diagram or patch might help me
> > understand!
>
> I just posted my RFC for supporting SCMI pinctrl protocol[1],
> hoping it will help you understand what I'm planning to do regarding
> test methodology, in particular by looking at patch#5 and #6.

OK...I am mostly out this week but will take a look next week.


>
> [1] https://lists.denx.de/pipermail/u-boot/2023-September/529765.html
>
> Thanks,
> -Takahiro Akashi
>
> > Regards,
> > Simon
> >
> >
> > >
> > > -Takahiro Akashi
> > >
> > >
> > > > We have done this a lot with Zephyr, as well[1] and achieved 90% code
> > > > coverage on some boards.
> > > >
> > > > But I'm not quite sure I am answering the right question, so I will stop here.
> > > >
> > > > Regards,
> > > > Simon
> > > >
> > > > [1] https://www.youtube.com/watch?v=usXCAXR2G_c
> >
> > [2] https://www.usenix.org/conference/osdi21/presentation/fri-keynote

Regards,
Simon