Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers

[Fam Zheng <famz@redhat.com>]

On Mon, 03/12 18:17, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Mar 12, 2018 at 6:11 PM, Fam Zheng <famz@redhat.com> wrote:
> > On Mon, 03/12 15:11, Marc-André Lureau wrote:
> >> Hi Fam,
> >>
> >> It would be great if patchew could run the debug/sanitizer build.
> >> Unfortunately, except x86 target, the run is full of ASAN leaks or
> >> warnings.
> >>
> >> Do you think patchew could learn to do regression testing? By that I
> >> mean that patchew could compare a patchset output with the current
> >> master (that output should be cached for multiple tests). The output
> >> to compare here would be the errors reported by ASAN during the build.
> >> (eventually, this could be used for other metrics some day). If so,
> >> would you be willing to help me implementing it?
> >
> > The question is can we simply rely on the exit code of cmp or diff for that kind
> > of comparison? If there is any out-of-order texts, that simple diff won't work.
> 
> If the tests are run sequentially, i think it should produce the same
> output when run multiple times (to be verified).

In that case the test will last for a long time. Currently patchew isn't very
good as multi-tasking. But that is one option we can explore (like we can devote
one VM for asan test).

> 
> However, if there are leaks already, and the address of allocation or
> symbols change, this may produce diff on existing lines.

Maybe we can come up with a 'make docker-test-debug-filtered@fedora' or a
relatively simple 'make docker-test-debug@fedora | sed -e "..."' command that
can be fed to diff.

> 
> What I would like to see is the diff of added (or removed) lines.
> 
> I suppose that should be possible, I'll do some experiments.
> 
> > Anything more intelligent/sensible should go as a policy, and patchew should
> > continue focus on mechanisms.
> 
> hmm ok :)

Like said above, I think it's possible to postprocess the output to make it easy
to diff/cmp. By mechanism/policy I mean the place to draw a line to code
Patchew.  However patchew has a database where we can write test script snippets
(as configurations) that is "the policy".

Fam