Re: [PATCH RFC 10/10] tests/migration-tests: Add test case for responsive CPU throttle

[Peter Xu <peterx@redhat.com>]

On Thu, Sep 12, 2024 at 04:14:20PM +0100, Peter Maydell wrote:
> On Thu, 12 Sept 2024 at 16:09, Peter Xu <peterx@redhat.com> wrote:
> >
> > On Thu, Sep 12, 2024 at 09:13:16AM +0100, Peter Maydell wrote:
> > > On Wed, 11 Sept 2024 at 22:26, Fabiano Rosas <farosas@suse.de> wrote:
> > > > I don't think we're discussing total CI time at this point, so the math
> > > > doesn't really add up. We're not looking into making the CI finish
> > > > faster. We're looking into making migration-test finish faster. That
> > > > would reduce timeouts in CI, speed-up make check and reduce the chance
> > > > of random race conditions* affecting other people/staging runs.
> > >
> > > Right. The reason migration-test appears on my radar is because
> > > it is very frequently the thing that shows up as "this sometimes
> > > just fails or just times out and if you hit retry it goes away
> > > again". That might not be migration-test's fault specifically,
> > > because those retries tend to be certain CI configs (s390,
> > > the i686-tci one), and I have some theories about what might be
> > > causing it (e.g. build system runs 4 migration-tests in parallel,
> > > which means 8 QEMU processes which is too many for the number
> > > of host CPUs). But right now I look at CI job failures and my reaction
> > > is "oh, it's the migration-test failing yet again" :-(
> > >
> > > For some examples from this week:
> > >
> > > https://gitlab.com/qemu-project/qemu/-/jobs/7802183144
> > > https://gitlab.com/qemu-project/qemu/-/jobs/7799842373  <--------[1]
> > > https://gitlab.com/qemu-project/qemu/-/jobs/7786579152  <--------[2]
> > > https://gitlab.com/qemu-project/qemu/-/jobs/7786579155
> >
> > Ah right, the TIMEOUT is unfortunate, especially if tests can be run in
> > parallel.  It indeed sounds like no good way to finally solve.. I don't
> > also see how speeding up / reducing tests in migration test would help, as
> > that's (from some degree..) is the same as tuning the timeout value bigger.
> > When the tests are less it'll fit into 480s window, but maybe it's too
> > quick now we wonder whether we should shrink it to e.g. 90s, but then it
> > can timeout again when on a busy host with less capability of concurrency.
> 
> For the TIMEOUT part on cross-i686-tci I plan to try this patch:
> https://patchew.org/QEMU/20240912151003.2045031-1-peter.maydell@linaro.org/
> which makes 'make check' single-threaded; that will help to see
> if the parallelism is a problem. (If it is then we might want
> to do a more generalised approach rather than just for that one
> CI job.)

Sounds good.

> 
> > But indeed there're two ERRORs ([1,2] above)..  I collected some more info
> > here before the log expires:
> 
> > So.. it's the same test (multifd/tcp/plain/cancel) that is failing on
> > different host / arch being tested.  What is more weird is the two failures
> > are different, the 2nd failure throw out a TLS error even though the test
> > doesn't yet have tls involved.
> >
> > Fabiano, is this the issue you're looking at?
> >
> > Peter, do you think it'll be helpful if we temporarily mark this test as
> > "slow" too so it's not run in CI (so we still run it ourselves when prepare
> > migration PR, with the hope that it can reproduce)?
> 
> If you think that specific test is flaky then I think that's
> probably a good idea. As usual with this kind of thing,
> probably best to have a comment next to the test noting
> why and with a URL to a gitlab issue for it, so we don't
> forget why we disabled the test.

Looks like Fabiano root-caused the issue.  We'll see how that goes, or we
can prepare a patch to make it optional with the comments in place.

Thanks,

-- 
Peter Xu