Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Peter Maydell]

We've made pretty good progress on transitioning our pre-merge CI
from running ad-hoc on machines the person doing the merge has access to
to all CI being driven by the Gitlab CI infrastructure. For this (7.1) release
cycle I think ideally we should try to get rid of the last few bits
of ad-hoc CI so that for 7.2 we are using only the gitlab CI. (This
will help in handing over merge request management to Stefan for 7.2.)

I think the last setups I have been using ad-hoc scripting for are:

(1) PPC64 big-endian Linux
(2) NetBSD (x86)
(3) OpenBSD (x86)

I think we can get away with just dropping ppc64be -- we have
coverage for it as a cross-compile setup, and hopefully the
s390x CI runner will catch the various "fails tests on big-endian host"
issues. (Alternatively if anybody has a ppc64be machine they'd like
to let us run a gitlab CI runner on, we could do that :-))

For the BSDs, the ad-hoc CI is just running the tests/vm
"make vm-build-netbsd" etc. Is there some way we can get
coverage of this into the gitlab CI setup? (I think we
have FreeBSD via Cirrus CI, so I have not listed that one.)

thanks
-- PMM

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Cédric Le Goater]

On 5/16/22 14:43, Peter Maydell wrote:
> We've made pretty good progress on transitioning our pre-merge CI
> from running ad-hoc on machines the person doing the merge has access to
> to all CI being driven by the Gitlab CI infrastructure. For this (7.1) release
> cycle I think ideally we should try to get rid of the last few bits
> of ad-hoc CI so that for 7.2 we are using only the gitlab CI. (This
> will help in handing over merge request management to Stefan for 7.2.)
> 
> I think the last setups I have been using ad-hoc scripting for are:
> 
> (1) PPC64 big-endian Linux
> (2) NetBSD (x86)
> (3) OpenBSD (x86)
> 
> I think we can get away with just dropping ppc64be -- we have
> coverage for it as a cross-compile setup, and hopefully the
> s390x CI runner will catch the various "fails tests on big-endian host"
> issues. (Alternatively if anybody has a ppc64be machine they'd like
> to let us run a gitlab CI runner on, we could do that :-))

No recent HW (P8 and above) would run a PPC64 BE distro if LE is
supported by HW.
  
The only BE-only HW would be a G5 (970) or a P7 (with OPAL).
These are really scarce now and I doubt they would be accessible
for external jobs.

The simplest would be to run a pseries KVM guest or PowerVM LPAR
with a debian sid, which still supports BE. Where is the question.

C.

> 
> For the BSDs, the ad-hoc CI is just running the tests/vm
> "make vm-build-netbsd" etc. Is there some way we can get
> coverage of this into the gitlab CI setup? (I think we
> have FreeBSD via Cirrus CI, so I have not listed that one.)
> 
> thanks
> -- PMM
> 

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Peter Maydell]

On Mon, 16 May 2022 at 14:51, Cédric Le Goater <clg@kaod.org> wrote:
>
> On 5/16/22 14:43, Peter Maydell wrote:
> > I think we can get away with just dropping ppc64be -- we have
> > coverage for it as a cross-compile setup, and hopefully the
> > s390x CI runner will catch the various "fails tests on big-endian host"
> > issues. (Alternatively if anybody has a ppc64be machine they'd like
> > to let us run a gitlab CI runner on, we could do that :-))
>
> No recent HW (P8 and above) would run a PPC64 BE distro if LE is
> supported by HW.

FWIW, the machine I use for ad-hoc CI is one in the gcc compile
farm, which is supposedly a "IBM POWER8 8284-22A", running Debian sid.

If BE PPC is fading away then that's another argument for
living with the loss of CI coverage, I guess.

thanks
-- PMM

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Paolo Bonzini]

On 5/16/22 14:43, Peter Maydell wrote:
> For the BSDs, the ad-hoc CI is just running the tests/vm
> "make vm-build-netbsd" etc. Is there some way we can get
> coverage of this into the gitlab CI setup? (I think we
> have FreeBSD via Cirrus CI, so I have not listed that one.)

We have vm-build-netbsd and vm-build-openbsd in Cirrus CI as well. 
However they are disabled by default due to the limit on the # of jobs 
in unpaid plans; some jobs will time out if you exceed the limit, 
because they are placed on a waiting list.

Paolo

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Thomas Huth]

On 16/05/2022 14.43, Peter Maydell wrote:
> We've made pretty good progress on transitioning our pre-merge CI
> from running ad-hoc on machines the person doing the merge has access to
> to all CI being driven by the Gitlab CI infrastructure. For this (7.1) release
> cycle I think ideally we should try to get rid of the last few bits
> of ad-hoc CI so that for 7.2 we are using only the gitlab CI. (This
> will help in handing over merge request management to Stefan for 7.2.)
> 
> I think the last setups I have been using ad-hoc scripting for are:
> 
> (1) PPC64 big-endian Linux
> (2) NetBSD (x86)
> (3) OpenBSD (x86)
> 
> I think we can get away with just dropping ppc64be -- we have
> coverage for it as a cross-compile setup, and hopefully the
> s390x CI runner will catch the various "fails tests on big-endian host"
> issues. (Alternatively if anybody has a ppc64be machine they'd like
> to let us run a gitlab CI runner on, we could do that :-))

Ack, that should cover most scenarios already. (tcg backend is the only one 
that would not get any coverage anymore)

> For the BSDs, the ad-hoc CI is just running the tests/vm
> "make vm-build-netbsd" etc. Is there some way we can get
> coverage of this into the gitlab CI setup? (I think we
> have FreeBSD via Cirrus CI, so I have not listed that one.)

A simple setup is already there, running NetBSD and OpenBSD via KVM on the 
Cirrus-CI, see e.g.:

  https://gitlab.com/thuth/qemu/-/jobs/2411943817#L1973

Caveats:
- The jobs are currently marked as "manual only" since the double
   indirect setup (via cirrus-run and KVM) is not that reliable.
   Also we can not run that many cirrus-ci jobs in parallel, so
   we likely don't want to enable these by default.
- Compilation is not very fast, the jobs often run longer than
   1h, though the --target-list is very short already.

Anyway, this should show that running NetBSD and OpenBSD is very well 
possible in our CI - we just need a more powerful x86 host with KVM enabled 
for this.

  Thomas

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Peter Maydell]

On Mon, 16 May 2022 at 15:30, Thomas Huth <thuth@redhat.com> wrote:
>
> On 16/05/2022 14.43, Peter Maydell wrote:
> > For the BSDs, the ad-hoc CI is just running the tests/vm
> > "make vm-build-netbsd" etc. Is there some way we can get
> > coverage of this into the gitlab CI setup? (I think we
> > have FreeBSD via Cirrus CI, so I have not listed that one.)
>
> A simple setup is already there, running NetBSD and OpenBSD via KVM on the
> Cirrus-CI, see e.g.:
>
>   https://gitlab.com/thuth/qemu/-/jobs/2411943817#L1973
>
> Caveats:
> - The jobs are currently marked as "manual only" since the double
>    indirect setup (via cirrus-run and KVM) is not that reliable.
>    Also we can not run that many cirrus-ci jobs in parallel, so
>    we likely don't want to enable these by default.
> - Compilation is not very fast, the jobs often run longer than
>    1h, though the --target-list is very short already.
>
> Anyway, this should show that running NetBSD and OpenBSD is very well
> possible in our CI - we just need a more powerful x86 host with KVM enabled
> for this.

Yes, if we have an x86 machine we can use as a private CI runner
for these jobs that would work.

-- PMM

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Cédric Le Goater]

On 5/16/22 15:55, Peter Maydell wrote:
> On Mon, 16 May 2022 at 14:51, Cédric Le Goater <clg@kaod.org> wrote:
>>
>> On 5/16/22 14:43, Peter Maydell wrote:
>>> I think we can get away with just dropping ppc64be -- we have
>>> coverage for it as a cross-compile setup, and hopefully the
>>> s390x CI runner will catch the various "fails tests on big-endian host"
>>> issues. (Alternatively if anybody has a ppc64be machine they'd like
>>> to let us run a gitlab CI runner on, we could do that :-))
>>
>> No recent HW (P8 and above) would run a PPC64 BE distro if LE is
>> supported by HW.
> 
> FWIW, the machine I use for ad-hoc CI is one in the gcc compile
> farm, which is supposedly a "IBM POWER8 8284-22A", running Debian sid.

Since the P8 have been around (~2014), the focus is really on LE.
I think debian is the last distro still providing BE binaries.
But no iso, you have to start with a debian 1O and do the upgrade.

> If BE PPC is fading away then that's another argument for
> living with the loss of CI coverage, I guess.

yes.

It would good to keep a BE host for test coverage. It doesn't have
to be ppc64be if it is too complex to maintain.

I can help on setting up a debian BE sid vm on the above IBM POWER8
8284-22A system if you need to. I have an image ready to use.

Thanks,

C.

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

On Mon, May 16, 2022 at 03:35:25PM +0100, Peter Maydell wrote:
> On Mon, 16 May 2022 at 15:30, Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 16/05/2022 14.43, Peter Maydell wrote:
> > > For the BSDs, the ad-hoc CI is just running the tests/vm
> > > "make vm-build-netbsd" etc. Is there some way we can get
> > > coverage of this into the gitlab CI setup? (I think we
> > > have FreeBSD via Cirrus CI, so I have not listed that one.)
> >
> > A simple setup is already there, running NetBSD and OpenBSD via KVM on the
> > Cirrus-CI, see e.g.:
> >
> >   https://gitlab.com/thuth/qemu/-/jobs/2411943817#L1973
> >
> > Caveats:
> > - The jobs are currently marked as "manual only" since the double
> >    indirect setup (via cirrus-run and KVM) is not that reliable.
> >    Also we can not run that many cirrus-ci jobs in parallel, so
> >    we likely don't want to enable these by default.
> > - Compilation is not very fast, the jobs often run longer than
> >    1h, though the --target-list is very short already.
> >
> > Anyway, this should show that running NetBSD and OpenBSD is very well
> > possible in our CI - we just need a more powerful x86 host with KVM enabled
> > for this.
> 
> Yes, if we have an x86 machine we can use as a private CI runner
> for these jobs that would work.

Hi Cleber,
I think there was a Fosshost x86 machine that is currently idle? Does it
support nested virt?

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Getting rid of the last bits of QEMU's 'ad-hoc CI' for merges

[Daniel P. Berrangé]

On Mon, May 16, 2022 at 04:50:29PM +0200, Cédric Le Goater wrote:
> On 5/16/22 15:55, Peter Maydell wrote:
> > On Mon, 16 May 2022 at 14:51, Cédric Le Goater <clg@kaod.org> wrote:
> > > 
> > > On 5/16/22 14:43, Peter Maydell wrote:
> > > > I think we can get away with just dropping ppc64be -- we have
> > > > coverage for it as a cross-compile setup, and hopefully the
> > > > s390x CI runner will catch the various "fails tests on big-endian host"
> > > > issues. (Alternatively if anybody has a ppc64be machine they'd like
> > > > to let us run a gitlab CI runner on, we could do that :-))
> > > 
> > > No recent HW (P8 and above) would run a PPC64 BE distro if LE is
> > > supported by HW.
> > 
> > FWIW, the machine I use for ad-hoc CI is one in the gcc compile
> > farm, which is supposedly a "IBM POWER8 8284-22A", running Debian sid.
> 
> Since the P8 have been around (~2014), the focus is really on LE.
> I think debian is the last distro still providing BE binaries.
> But no iso, you have to start with a debian 1O and do the upgrade.

If even Debian 11 has stopped providing ppc64be images, then
the end really is nigh. We still support Debian 10 as a target
in our platform support matrix until Debin 12 comes out in
mid/late 2023.

So IMHO we've got at most 1 year left of needing to worry
about ppc64be targets.

> > If BE PPC is fading away then that's another argument for
> > living with the loss of CI coverage, I guess.
> 
> yes.
> 
> It would good to keep a BE host for test coverage. It doesn't have
> to be ppc64be if it is too complex to maintain.

IIUC, we've got s390x giving BE coverage right now, at least
for certain configure option combinations.
 
> I can help on setting up a debian BE sid vm on the above IBM POWER8
> 8284-22A system if you need to. I have an image ready to use.

If we have a means to run CI for the next ~1 year great, but it
doesn't seem like worth investing masses of resources into it.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|