[Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Niels de Vos]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

Hi,

it seems we broke the block/gluster.c functionality with a recent patch
in upstream Gluster. In order to prevent this from happening in the
future, I would like to setup a Jenkins job that installs a plan CentOS
with its version of QEMU, and nightly builds of upstream Gluster.
Getting a notification about breakage the day after a patch got merged
seems like a reasonable approach.

The test should at least boot the generic CentOS cloud image (slightly
modified with libguestfs) and return a success/fail. I am wondering if
there are automated tests like this already, and if I could (re)use some
of the scripts for it. At the moment, I am thinking to so it like this:
 - download the image [1]
 - set kernel parameters to output on the serial console
 - add a auto-login user/script
 - have the script write "bootup complete" or something
 - have the script poweroff the VM
 - script that started the VM checks for the "bootup complete" message
 - return success/fail

Ideas and suggestions for running more heavy I/O in the VM are welcome
too.

Thanks,
Niels


1. http://cloud.centos.org/centos/7/images/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Vasiliy Tolstov]

I'm recommend to use packer for this,it able to run via qemu VM,run scripts
and output artifacts.
28 Июн 2016 г. 12:10 пользователь "Niels de Vos" <ndevos@redhat.com>
написал:

> Hi,
>
> it seems we broke the block/gluster.c functionality with a recent patch
> in upstream Gluster. In order to prevent this from happening in the
> future, I would like to setup a Jenkins job that installs a plan CentOS
> with its version of QEMU, and nightly builds of upstream Gluster.
> Getting a notification about breakage the day after a patch got merged
> seems like a reasonable approach.
>
> The test should at least boot the generic CentOS cloud image (slightly
> modified with libguestfs) and return a success/fail. I am wondering if
> there are automated tests like this already, and if I could (re)use some
> of the scripts for it. At the moment, I am thinking to so it like this:
>  - download the image [1]
>  - set kernel parameters to output on the serial console
>  - add a auto-login user/script
>  - have the script write "bootup complete" or something
>  - have the script poweroff the VM
>  - script that started the VM checks for the "bootup complete" message
>  - return success/fail
>
> Ideas and suggestions for running more heavy I/O in the VM are welcome
> too.
>
> Thanks,
> Niels
>
>
> 1. http://cloud.centos.org/centos/7/images/
>

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Niels de Vos]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

On Tue, Jun 28, 2016 at 12:41:28PM +0300, Vasiliy Tolstov wrote:
> I'm recommend to use packer for this,it able to run via qemu VM,run scripts
> and output artifacts.

I'm not familiar with packer, but it seems very similar to virt-builder.
It does not look to be available in standard CentOS repositories.
Because the tests will run in the CentOS CI, I'd prefer to use as few
external tools as possible.

Thanks for the idea,
Niels


> 28 Июн 2016 г. 12:10 пользователь "Niels de Vos" <ndevos@redhat.com>
> написал:
> 
> > Hi,
> >
> > it seems we broke the block/gluster.c functionality with a recent patch
> > in upstream Gluster. In order to prevent this from happening in the
> > future, I would like to setup a Jenkins job that installs a plan CentOS
> > with its version of QEMU, and nightly builds of upstream Gluster.
> > Getting a notification about breakage the day after a patch got merged
> > seems like a reasonable approach.
> >
> > The test should at least boot the generic CentOS cloud image (slightly
> > modified with libguestfs) and return a success/fail. I am wondering if
> > there are automated tests like this already, and if I could (re)use some
> > of the scripts for it. At the moment, I am thinking to so it like this:
> >  - download the image [1]
> >  - set kernel parameters to output on the serial console
> >  - add a auto-login user/script
> >  - have the script write "bootup complete" or something
> >  - have the script poweroff the VM
> >  - script that started the VM checks for the "bootup complete" message
> >  - return success/fail
> >
> > Ideas and suggestions for running more heavy I/O in the VM are welcome
> > too.
> >
> > Thanks,
> > Niels
> >
> >
> > 1. http://cloud.centos.org/centos/7/images/
> >

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Vasiliy Tolstov]

2016-06-28 13:27 GMT+03:00 Niels de Vos <ndevos@redhat.com>:
> I'm not familiar with packer, but it seems very similar to virt-builder.
> It does not look to be available in standard CentOS repositories.
> Because the tests will run in the CentOS CI, I'd prefer to use as few
> external tools as possible.


packer is the static binary so you can download it if you ci allow out
coming connections...
If you took some tools, please provide info about you test environment
with tools in this email

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@yoctocloud.net

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

Am 28.06.2016 um 11:02 hat Niels de Vos geschrieben:
> Hi,
> 
> it seems we broke the block/gluster.c functionality with a recent patch
> in upstream Gluster. In order to prevent this from happening in the
> future, I would like to setup a Jenkins job that installs a plan CentOS
> with its version of QEMU, and nightly builds of upstream Gluster.
> Getting a notification about breakage the day after a patch got merged
> seems like a reasonable approach.
> 
> The test should at least boot the generic CentOS cloud image (slightly
> modified with libguestfs) and return a success/fail. I am wondering if
> there are automated tests like this already, and if I could (re)use some
> of the scripts for it. At the moment, I am thinking to so it like this:
>  - download the image [1]
>  - set kernel parameters to output on the serial console
>  - add a auto-login user/script
>  - have the script write "bootup complete" or something
>  - have the script poweroff the VM
>  - script that started the VM checks for the "bootup complete" message
>  - return success/fail

Sounds like something that Avocado should be able (or actually is
designed) to do. I can't tell you the details of how to write the test
case for it, but I'm adding a CC to Lukáš who probably can (and I think
it shouldn't be hard anyway).

Kevin

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Lukáš Doktor]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

Dne 28.6.2016 v 16:10 Kevin Wolf napsal(a):
> Am 28.06.2016 um 11:02 hat Niels de Vos geschrieben:
>> Hi,
>>
>> it seems we broke the block/gluster.c functionality with a recent patch
>> in upstream Gluster. In order to prevent this from happening in the
>> future, I would like to setup a Jenkins job that installs a plan CentOS
>> with its version of QEMU, and nightly builds of upstream Gluster.
>> Getting a notification about breakage the day after a patch got merged
>> seems like a reasonable approach.
>>
>> The test should at least boot the generic CentOS cloud image (slightly
>> modified with libguestfs) and return a success/fail. I am wondering if
>> there are automated tests like this already, and if I could (re)use some
>> of the scripts for it. At the moment, I am thinking to so it like this:
>>  - download the image [1]
>>  - set kernel parameters to output on the serial console
>>  - add a auto-login user/script
>>  - have the script write "bootup complete" or something
>>  - have the script poweroff the VM
>>  - script that started the VM checks for the "bootup complete" message
>>  - return success/fail
>
> Sounds like something that Avocado should be able (or actually is
> designed) to do. I can't tell you the details of how to write the test
> case for it, but I'm adding a CC to Lukáš who probably can (and I think
> it shouldn't be hard anyway).
>
> Kevin
>

Hello guys,

yes, Avocado is designed to do this and I believe it even contain quite 
a few Gluster tests. You can look for them in avocado-vt or ping our QA 
folks who might give you some pointers (cc Xu nad Hao).

Regarding the building the CI I use the combination of Jenkins, Jenkins 
job builder and Avocado (avocado-vt) to check power/arm 
weekly/per-package-update. Jenkins even supports github and other 
triggers if you decide you have enough resources to check each 
PR/commit. It all depends on what HW you have available.

Regards,
Lukáš


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Niels de Vos]

[-- Attachment #1: Type: text/plain, Size: 2813 bytes --]

On Tue, Jun 28, 2016 at 05:20:03PM +0200, Lukáš Doktor wrote:
> Dne 28.6.2016 v 16:10 Kevin Wolf napsal(a):
> > Am 28.06.2016 um 11:02 hat Niels de Vos geschrieben:
> > > Hi,
> > > 
> > > it seems we broke the block/gluster.c functionality with a recent patch
> > > in upstream Gluster. In order to prevent this from happening in the
> > > future, I would like to setup a Jenkins job that installs a plan CentOS
> > > with its version of QEMU, and nightly builds of upstream Gluster.
> > > Getting a notification about breakage the day after a patch got merged
> > > seems like a reasonable approach.
> > > 
> > > The test should at least boot the generic CentOS cloud image (slightly
> > > modified with libguestfs) and return a success/fail. I am wondering if
> > > there are automated tests like this already, and if I could (re)use some
> > > of the scripts for it. At the moment, I am thinking to so it like this:
> > >  - download the image [1]
> > >  - set kernel parameters to output on the serial console
> > >  - add a auto-login user/script
> > >  - have the script write "bootup complete" or something
> > >  - have the script poweroff the VM
> > >  - script that started the VM checks for the "bootup complete" message
> > >  - return success/fail
> > 
> > Sounds like something that Avocado should be able (or actually is
> > designed) to do. I can't tell you the details of how to write the test
> > case for it, but I'm adding a CC to Lukáš who probably can (and I think
> > it shouldn't be hard anyway).
> > 
> > Kevin
> > 
> 
> Hello guys,
> 
> yes, Avocado is designed to do this and I believe it even contain quite a
> few Gluster tests. You can look for them in avocado-vt or ping our QA folks
> who might give you some pointers (cc Xu nad Hao).
> 
> Regarding the building the CI I use the combination of Jenkins, Jenkins job
> builder and Avocado (avocado-vt) to check power/arm
> weekly/per-package-update. Jenkins even supports github and other triggers
> if you decide you have enough resources to check each PR/commit. It all
> depends on what HW you have available.

That looks promising! Its a bit more complex (or at least 'new' for me)
than that I was hoping. There is Gluster support in there, I found a
description of it here:
  http://avocado-vt.readthedocs.io/en/latest/GlusterFs.html
  http://avocado-vt.readthedocs.io/en/latest/RunQemuUnittests.html

Browsing through the docs does not really explain me how to put a
configuration file together that runs the QEMU tests with a VM image on
Gluster though. I probably need to read much more, but a pointer or very
minimal example would be much appreciated.

When I'm able to run avocado-vt, it should be trivial to put that in a
Jenkins job :)

Many thanks,
Niels

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Lukáš Doktor]

[-- Attachment #1: Type: text/plain, Size: 4947 bytes --]

Dne 28.6.2016 v 17:56 Niels de Vos napsal(a):
> On Tue, Jun 28, 2016 at 05:20:03PM +0200, Lukáš Doktor wrote:
>> Dne 28.6.2016 v 16:10 Kevin Wolf napsal(a):
>>> Am 28.06.2016 um 11:02 hat Niels de Vos geschrieben:
>>>> Hi,
>>>>
>>>> it seems we broke the block/gluster.c functionality with a recent patch
>>>> in upstream Gluster. In order to prevent this from happening in the
>>>> future, I would like to setup a Jenkins job that installs a plan CentOS
>>>> with its version of QEMU, and nightly builds of upstream Gluster.
>>>> Getting a notification about breakage the day after a patch got merged
>>>> seems like a reasonable approach.
>>>>
>>>> The test should at least boot the generic CentOS cloud image (slightly
>>>> modified with libguestfs) and return a success/fail. I am wondering if
>>>> there are automated tests like this already, and if I could (re)use some
>>>> of the scripts for it. At the moment, I am thinking to so it like this:
>>>>  - download the image [1]
>>>>  - set kernel parameters to output on the serial console
>>>>  - add a auto-login user/script
>>>>  - have the script write "bootup complete" or something
>>>>  - have the script poweroff the VM
>>>>  - script that started the VM checks for the "bootup complete" message
>>>>  - return success/fail
>>>
>>> Sounds like something that Avocado should be able (or actually is
>>> designed) to do. I can't tell you the details of how to write the test
>>> case for it, but I'm adding a CC to Lukáš who probably can (and I think
>>> it shouldn't be hard anyway).
>>>
>>> Kevin
>>>
>>
>> Hello guys,
>>
>> yes, Avocado is designed to do this and I believe it even contain quite a
>> few Gluster tests. You can look for them in avocado-vt or ping our QA folks
>> who might give you some pointers (cc Xu nad Hao).
>>
>> Regarding the building the CI I use the combination of Jenkins, Jenkins job
>> builder and Avocado (avocado-vt) to check power/arm
>> weekly/per-package-update. Jenkins even supports github and other triggers
>> if you decide you have enough resources to check each PR/commit. It all
>> depends on what HW you have available.
>
> That looks promising! Its a bit more complex (or at least 'new' for me)
> than that I was hoping. There is Gluster support in there, I found a
> description of it here:
>   http://avocado-vt.readthedocs.io/en/latest/GlusterFs.html
>   http://avocado-vt.readthedocs.io/en/latest/RunQemuUnittests.html
>
> Browsing through the docs does not really explain me how to put a
> configuration file together that runs the QEMU tests with a VM image on
> Gluster though. I probably need to read much more, but a pointer or very
> minimal example would be much appreciated.
>
> When I'm able to run avocado-vt, it should be trivial to put that in a
> Jenkins job :)
>
> Many thanks,
> Niels
>

Hello Niels,

yep, it should be quite simple, but I don't want to break my setup just 
to try it out. Hopefully you'll manage to do it yourself. Anyway few 
pointers...

Install avocado:

 
http://avocado-vt.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado

it should be straight forward so I expect you're able to run `avocado 
boot` already. There are several backends so can run the same tests on 
top of them. Let's assume you're using `qemu`, which is the default. The 
difference is the backend configuration location and some details 
regarding importing images and so on. Qemu is the simplest for me as it 
does not require anything.

Now regarding the GlusterFS. By default avocado uses `only 
(image_backend=filesystem)` hardcoded in 
`/usr/share/avocado_vt/backends/qemu/cfg/tests.cfg`. This is because 
wast majority of people don't want to change it and if they do they 
usually use custom config files. I don't think you'd like to go that way 
so let's just patch that file and change it to `only 
(image_backend=gluster)`.

Then you might take a look into 
`/usr/share/avocado_vt/shared/cfg/guest-hw.cfg` where you can find the 
`variants image_backend:` and several profiles there including 
`gluster`. You can specify the `gluster_brick` and other options.

When you modify everything to your needs you should re-run `avocado 
vt-bootstrap` to update the configs and you should be ready to run. Any 
test should then use the `gluster` image instead of file-based image.


As for the kvm-unit-test, the documentation is seriously outdated and 
new version is in progress. You can use the pure avocado for running 
kvm-unit-test, see 
https://github.com/avocado-framework/avocado/pull/1280 for details. I'll 
update the avocado-vt documentation when the script is merged.


In the end I'd recommend using the `--xunit` to produce junit results 
and you can import them in jenkins including per-subtest-statuses. Let 
me send you my setup in PM for inspiration...


Regards,
Lukáš


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] Automated testing of block/gluster.c with upstream Gluster

[Niels de Vos]

[-- Attachment #1: Type: text/plain, Size: 5508 bytes --]

On Wed, Jun 29, 2016 at 09:39:22AM +0200, Lukáš Doktor wrote:
> Dne 28.6.2016 v 17:56 Niels de Vos napsal(a):
> > On Tue, Jun 28, 2016 at 05:20:03PM +0200, Lukáš Doktor wrote:
> > > Dne 28.6.2016 v 16:10 Kevin Wolf napsal(a):
> > > > Am 28.06.2016 um 11:02 hat Niels de Vos geschrieben:
> > > > > Hi,
> > > > > 
> > > > > it seems we broke the block/gluster.c functionality with a recent patch
> > > > > in upstream Gluster. In order to prevent this from happening in the
> > > > > future, I would like to setup a Jenkins job that installs a plan CentOS
> > > > > with its version of QEMU, and nightly builds of upstream Gluster.
> > > > > Getting a notification about breakage the day after a patch got merged
> > > > > seems like a reasonable approach.
> > > > > 
> > > > > The test should at least boot the generic CentOS cloud image (slightly
> > > > > modified with libguestfs) and return a success/fail. I am wondering if
> > > > > there are automated tests like this already, and if I could (re)use some
> > > > > of the scripts for it. At the moment, I am thinking to so it like this:
> > > > >  - download the image [1]
> > > > >  - set kernel parameters to output on the serial console
> > > > >  - add a auto-login user/script
> > > > >  - have the script write "bootup complete" or something
> > > > >  - have the script poweroff the VM
> > > > >  - script that started the VM checks for the "bootup complete" message
> > > > >  - return success/fail
> > > > 
> > > > Sounds like something that Avocado should be able (or actually is
> > > > designed) to do. I can't tell you the details of how to write the test
> > > > case for it, but I'm adding a CC to Lukáš who probably can (and I think
> > > > it shouldn't be hard anyway).
> > > > 
> > > > Kevin
> > > > 
> > > 
> > > Hello guys,
> > > 
> > > yes, Avocado is designed to do this and I believe it even contain quite a
> > > few Gluster tests. You can look for them in avocado-vt or ping our QA folks
> > > who might give you some pointers (cc Xu nad Hao).
> > > 
> > > Regarding the building the CI I use the combination of Jenkins, Jenkins job
> > > builder and Avocado (avocado-vt) to check power/arm
> > > weekly/per-package-update. Jenkins even supports github and other triggers
> > > if you decide you have enough resources to check each PR/commit. It all
> > > depends on what HW you have available.
> > 
> > That looks promising! Its a bit more complex (or at least 'new' for me)
> > than that I was hoping. There is Gluster support in there, I found a
> > description of it here:
> >   http://avocado-vt.readthedocs.io/en/latest/GlusterFs.html
> >   http://avocado-vt.readthedocs.io/en/latest/RunQemuUnittests.html
> > 
> > Browsing through the docs does not really explain me how to put a
> > configuration file together that runs the QEMU tests with a VM image on
> > Gluster though. I probably need to read much more, but a pointer or very
> > minimal example would be much appreciated.
> > 
> > When I'm able to run avocado-vt, it should be trivial to put that in a
> > Jenkins job :)
> > 
> > Many thanks,
> > Niels
> > 
> 
> Hello Niels,
> 
> yep, it should be quite simple, but I don't want to break my setup just to
> try it out. Hopefully you'll manage to do it yourself. Anyway few
> pointers...
> 
> Install avocado:
> 
> 
> http://avocado-vt.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
> 
> it should be straight forward so I expect you're able to run `avocado boot`
> already. There are several backends so can run the same tests on top of
> them. Let's assume you're using `qemu`, which is the default. The difference
> is the backend configuration location and some details regarding importing
> images and so on. Qemu is the simplest for me as it does not require
> anything.
> 
> Now regarding the GlusterFS. By default avocado uses `only
> (image_backend=filesystem)` hardcoded in
> `/usr/share/avocado_vt/backends/qemu/cfg/tests.cfg`. This is because wast
> majority of people don't want to change it and if they do they usually use
> custom config files. I don't think you'd like to go that way so let's just
> patch that file and change it to `only (image_backend=gluster)`.
> 
> Then you might take a look into
> `/usr/share/avocado_vt/shared/cfg/guest-hw.cfg` where you can find the
> `variants image_backend:` and several profiles there including `gluster`.
> You can specify the `gluster_brick` and other options.
> 
> When you modify everything to your needs you should re-run `avocado
> vt-bootstrap` to update the configs and you should be ready to run. Any test
> should then use the `gluster` image instead of file-based image.
> 
> 
> As for the kvm-unit-test, the documentation is seriously outdated and new
> version is in progress. You can use the pure avocado for running
> kvm-unit-test, see https://github.com/avocado-framework/avocado/pull/1280
> for details. I'll update the avocado-vt documentation when the script is
> merged.
> 
> 
> In the end I'd recommend using the `--xunit` to produce junit results and
> you can import them in jenkins including per-subtest-statuses. Let me send
> you my setup in PM for inspiration...

Thanks! This really helps me a lot. When I have my Jenkins job and
scripts ready, I'll share them here as well. It is not high on my
priority list, but I try to get it up and running before the end of next
week.

Niels

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]