Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Damien Hedde <damien.hedde@greensocs.com>]

On 7/3/19 11:29 AM, Stefan Hajnoczi wrote:
> On Mon, Jul 01, 2019 at 12:16:44PM +0200, Philippe Mathieu-Daudé wrote:
>> On 7/1/19 10:37 AM, Stefan Hajnoczi wrote:
>>> On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
>>>> This series adds a python framework aiming to provide some ways to do fault
>>>> injection in a running vm. In its current state, it allows to easily interact
>>>> with memory, change gpios and qom properties.
>>>>
>>>> The framework consists in a python script based on the qmp existing module
>>>> which allows to interact with the vm.
>>>
>>> How does this compare to qtest?  There seems to be a lot of overlap
>>> between them.
>>>
>>> Why is it called "fault injection"?  The commands seem to be
>>> general-purpose device testing functions (like qtest and libqos), not
>>> functions for testing error code paths as would be expected from a fault
>>> injection framework.
>>
>> I understand qtest is to test QEMU, while this framework/command is to
>> test how the guest react to an hardware faults.
> 
> The commands seems to be equivalent to qtest commands, just implemented
> as QMP commands.
> 
> Damien: Can you explain the use case more and show some example test
> cases?

The goal is to test and validate the software running on the vp. We want
to generate some fault to test if the software behave correctly. We
target corner cases that do not happen otherwise on the vp. Basically we
would like, using some scripts, to run some specific scenarios and check
that the expected behavior happens.

Regarding qtest, I was not aware that it provided such commands. I'm
sorry I've missed that. Just checked after reading your feedback,
commands seem indeed equivalent. I don't know if running the vp with
qtest enabled has some hidden drawbacks. But if that's not the case, we
can work to extend the existing qtest commands (or switch some of them
to QMP like Philippe proposed, I don't know what's best).

> 
>> To use the qtest_mem commands you need to run QEMU with the qtest
>> chardev backend, while this series expose a QMP interface.
>>
>> To avoid the overlap, a cleaner follow up might be to have qtest wrap
>> these QMP commands (mostly like HMP commands do).
>>
>> Another note while looking at a glance, qtest uses the 1st cpu address
>> space view, this series allow to select a specific cpu.
>>
>> It makes sense to me to be able to select address spaces by name (more
>> generic, not restricted to a cpu view, since one might want to inject
>> fault in a device ram not always mapped to a cpu: dma, emac desc).

Good point.

> 
> The naming issue still stands: none of the commands actually perform
> fault injection.  They can be used for other types of testing or even
> non-testing purposes.
> 
> Fault injection commands would be "make the next watchdog expiry fail",
> "return error code X on the next DMA request", "report an AHCI link
> failure", etc.
> 
> These commands are lower-level.  Therefore, I think "fault injection
> framework" is a misnomer and will age poorly if this API is extended in
> the future.

The only fault injection naming was for the python module. I suppose
that if we just extend qtest, there is no need for a new module or
documentation file.

Thanks,

Damien