* [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack
2016-11-14 12:33 [OSSTEST PATCH v7 0/3] Have OpenStack tested on top of xen's master and libvirt's master Anthony PERARD
@ 2016-11-14 12:33 ` Anthony PERARD
2016-11-15 21:40 ` Konrad Rzeszutek Wilk
2016-11-18 18:51 ` Ian Jackson
2016-11-14 12:33 ` [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack Anthony PERARD
2016-11-14 12:33 ` [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt Anthony PERARD
2 siblings, 2 replies; 10+ messages in thread
From: Anthony PERARD @ 2016-11-14 12:33 UTC (permalink / raw)
To: xen-devel; +Cc: Anthony PERARD, Ian Jackson
This script installs any necessary packages and clones all of the OpenStack
trees which are used by devstack to deploy OpenStack.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
Changes in V7:
- reindent
- fix style
- remove many workaround, there are not needed anymore.
- create bug report upstream
Changes in V6:
- rebased
- fix issues due to new debian and newer devstack:
- add missing libvirt group
- switch back to old nova-network instead of neutron
- have devstack use 'service' instead of 'systemctl' to restart
services
Only change in V5:
- edit stackrc from devstack file to change the hardcoded path DEST
No change in V4:
- acked
Change in V3:
- Use host as argument to run the job.
- Use selectjob() and get rid of the unused $gho.
- Use target_jobdir() instead of builddirsprops().
- Remove GIT_BASE from devstack config.
- Rename the script to ts-openstack-deploy (from ts-openstack-devstack).
---
sg-run-job | 5 ++
ts-openstack-deploy | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 137 insertions(+)
create mode 100755 ts-openstack-deploy
diff --git a/sg-run-job b/sg-run-job
index 9c3e0ce..7b9296c 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -474,6 +474,11 @@ proc run-job/test-rumprun {} {
ts-guest-destroy-hard host $g +
}
+proc need-hosts/test-devstack {} { return host }
+proc run-job/test-devstack {} {
+ run-ts . = ts-openstack-deploy host
+}
+
if {[file exists sg-run-job-adhoc]} {
source sg-run-job-adhoc
}
diff --git a/ts-openstack-deploy b/ts-openstack-deploy
new file mode 100755
index 0000000..f26963a
--- /dev/null
+++ b/ts-openstack-deploy
@@ -0,0 +1,132 @@
+#!/usr/bin/perl
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho = selecthost($whhost);
+our $builddir = target_jobdir($ho);
+
+sub packages () {
+ target_install_packages($ho, qw(git sudo));
+}
+sub checkout () {
+ prepbuilddirs();
+ build_clone($ho, 'cinder', $builddir, 'cinder');
+ build_clone($ho, 'devstack', $builddir, 'devstack');
+ build_clone($ho, 'glance', $builddir, 'glance');
+ build_clone($ho, 'keystone', $builddir, 'keystone');
+ build_clone($ho, 'nova', $builddir, 'nova');
+ build_clone($ho, 'requirements', $builddir, 'requirements');
+ build_clone($ho, 'tempest', $builddir, 'tempest');
+
+ my $vg = target_choose_vg($ho, 10*1024); # 10GB
+ target_putfilecontents_stash($ho, 60, <<END, $builddir.'/devstack/local.conf');
+[[local|localrc]]
+# Everything should be cloned by osstest, so devstack don't have to do it
+ERROR_ON_CLONE=True
+USE_SCREEN=False
+ADMIN_PASSWORD=secretadmin
+DATABASE_PASSWORD=secretdatabase
+RABBIT_PASSWORD=secretrabbit
+SERVICE_PASSWORD=secretservice
+SERVICE_TOKEN=atokenserviced
+# make it small because there is no way to not have this lvm volume created
+VOLUME_BACKING_FILE_SIZE=50M
+CINDER_LVM_TYPE=default
+DEST=$builddir
+DATA_DIR=\$DEST/data
+SERVICE_DIR=\$DEST/status
+SUBUNIT_OUTPUT=\$DEST/devstack.subunit
+LOGFILE=\$DEST/logs/stack.sh.log
+LOG_COLOR=False
+LIBVIRT_TYPE=xen
+disable_service horizon
+disable_service n-novnc
+disable_service dstat
+enable_service n-obj
+# Disable neutron and switch back to nova-network
+disable_service q-svc
+disable_service q-dhcp
+disable_service q-meta
+disable_service q-agt
+disable_service q-l3
+enable_service n-net
+[[post-config|\$CINDER_CONF]]
+[lvmdriver-1]
+volume_group = $vg
+END
+
+ # libvirt is already installed, but not as a package, so avoid installation of
+ # the libvirt package with devstack
+ # https://bugs.launchpad.net/devstack/+bug/1641144
+ target_editfile($ho, "$builddir/devstack/files/debs/nova", sub {
+ while (<EI>) {
+ next if m/.*libvirt.*/;
+ print EO or die $!;
+ }
+ });
+ target_editfile($ho, "$builddir/devstack/lib/nova_plugins/functions-libvirt", sub {
+ while (<EI>) {
+ next if m/install_package.*libvirt.*/;
+ print EO or die $!;
+ }
+ });
+
+ # devstack blindly assume that systemd is used if systemctl is present
+ # https://bugs.launchpad.net/devstack/+bug/1641112
+ target_editfile($ho, "$builddir/devstack/functions-common", sub {
+ while (<EI>) {
+ if (m#\[ -x /bin/systemctl#) {
+ s#\[ -x /bin/systemctl \]#false#
+ }
+ print EO or die $!;
+ }
+ });
+
+ # OpenStack needs access to libvirt from a user.
+ target_cmd_root($ho, <<END);
+ addgroup --system libvirt
+ cat >> /etc/libvirt/libvirtd.conf <<EOF
+unix_sock_group = "libvirt"
+unix_sock_ro_perms = "0777"
+unix_sock_rw_perms = "0770"
+EOF
+ service libvirtd restart
+END
+
+ # devstack is going to setup the host, install some dependency.
+ target_putfilecontents_root_stash($ho, 100, <<END,"/etc/sudoers.d/devstack");
+osstest ALL=(ALL) NOPASSWD:ALL
+END
+}
+
+sub deploy() {
+ target_cmd($ho, <<END, 1800);
+ cd $builddir/devstack
+ ./stack.sh
+END
+}
+
+packages();
+checkout();
+deploy();
--
Anthony PERARD
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack
2016-11-14 12:33 ` [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack Anthony PERARD
@ 2016-11-15 21:40 ` Konrad Rzeszutek Wilk
2016-11-18 18:51 ` Ian Jackson
1 sibling, 0 replies; 10+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-11-15 21:40 UTC (permalink / raw)
To: Anthony PERARD; +Cc: Ian Jackson, xen-devel
On Mon, Nov 14, 2016 at 12:33:32PM +0000, Anthony PERARD wrote:
> This script installs any necessary packages and clones all of the OpenStack
> trees which are used by devstack to deploy OpenStack.
>
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
.. snip..
> diff --git a/ts-openstack-deploy b/ts-openstack-deploy
> new file mode 100755
> index 0000000..f26963a
> --- /dev/null
> +++ b/ts-openstack-deploy
> @@ -0,0 +1,132 @@
> +#!/usr/bin/perl
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2015 Citrix Inc.
2016.
> +#
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU Affero General Public License for more details.
> +#
> +# You should have received a copy of the GNU Affero General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +use strict qw(vars);
> +use Osstest;
> +use Osstest::TestSupport;
> +use Osstest::BuildSupport;
> +
> +tsreadconfig();
> +our ($whhost) = @ARGV;
> +$whhost ||= 'host';
> +our $ho = selecthost($whhost);
> +our $builddir = target_jobdir($ho);
> +
> +sub packages () {
> + target_install_packages($ho, qw(git sudo));
> +}
> +sub checkout () {
> + prepbuilddirs();
> + build_clone($ho, 'cinder', $builddir, 'cinder');
> + build_clone($ho, 'devstack', $builddir, 'devstack');
> + build_clone($ho, 'glance', $builddir, 'glance');
> + build_clone($ho, 'keystone', $builddir, 'keystone');
> + build_clone($ho, 'nova', $builddir, 'nova');
> + build_clone($ho, 'requirements', $builddir, 'requirements');
> + build_clone($ho, 'tempest', $builddir, 'tempest');
> +
> + my $vg = target_choose_vg($ho, 10*1024); # 10GB
> + target_putfilecontents_stash($ho, 60, <<END, $builddir.'/devstack/local.conf');
> +[[local|localrc]]
> +# Everything should be cloned by osstest, so devstack don't have to do it
> +ERROR_ON_CLONE=True
> +USE_SCREEN=False
> +ADMIN_PASSWORD=secretadmin
> +DATABASE_PASSWORD=secretdatabase
> +RABBIT_PASSWORD=secretrabbit
> +SERVICE_PASSWORD=secretservice
> +SERVICE_TOKEN=atokenserviced
> +# make it small because there is no way to not have this lvm volume created
> +VOLUME_BACKING_FILE_SIZE=50M
> +CINDER_LVM_TYPE=default
> +DEST=$builddir
> +DATA_DIR=\$DEST/data
> +SERVICE_DIR=\$DEST/status
> +SUBUNIT_OUTPUT=\$DEST/devstack.subunit
> +LOGFILE=\$DEST/logs/stack.sh.log
> +LOG_COLOR=False
> +LIBVIRT_TYPE=xen
> +disable_service horizon
> +disable_service n-novnc
> +disable_service dstat
> +enable_service n-obj
> +# Disable neutron and switch back to nova-network
> +disable_service q-svc
> +disable_service q-dhcp
> +disable_service q-meta
> +disable_service q-agt
> +disable_service q-l3
> +enable_service n-net
> +[[post-config|\$CINDER_CONF]]
> +[lvmdriver-1]
> +volume_group = $vg
> +END
> +
> + # libvirt is already installed, but not as a package, so avoid installation of
> + # the libvirt package with devstack
> + # https://bugs.launchpad.net/devstack/+bug/1641144
> + target_editfile($ho, "$builddir/devstack/files/debs/nova", sub {
> + while (<EI>) {
> + next if m/.*libvirt.*/;
> + print EO or die $!;
> + }
> + });
> + target_editfile($ho, "$builddir/devstack/lib/nova_plugins/functions-libvirt", sub {
> + while (<EI>) {
> + next if m/install_package.*libvirt.*/;
> + print EO or die $!;
> + }
> + });
> +
> + # devstack blindly assume that systemd is used if systemctl is present
> + # https://bugs.launchpad.net/devstack/+bug/1641112
Would it make sense to put xen-devel as an CC on those bugs so when
there is an resolution these changes would be removed?
Or would we keep them forever (which looks to be OK).
> + target_editfile($ho, "$builddir/devstack/functions-common", sub {
> + while (<EI>) {
> + if (m#\[ -x /bin/systemctl#) {
> + s#\[ -x /bin/systemctl \]#false#
> + }
> + print EO or die $!;
> + }
> + });
> +
> + # OpenStack needs access to libvirt from a user.
> + target_cmd_root($ho, <<END);
> + addgroup --system libvirt
> + cat >> /etc/libvirt/libvirtd.conf <<EOF
> +unix_sock_group = "libvirt"
> +unix_sock_ro_perms = "0777"
> +unix_sock_rw_perms = "0770"
> +EOF
> + service libvirtd restart
> +END
> +
> + # devstack is going to setup the host, install some dependency.
You mean give it access to install whatever it wants.
> + target_putfilecontents_root_stash($ho, 100, <<END,"/etc/sudoers.d/devstack");
> +osstest ALL=(ALL) NOPASSWD:ALL
> +END
> +}
> +
> +sub deploy() {
> + target_cmd($ho, <<END, 1800);
> + cd $builddir/devstack
> + ./stack.sh
> +END
> +}
> +
> +packages();
> +checkout();
> +deploy();
> --
> Anthony PERARD
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack
2016-11-14 12:33 ` [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack Anthony PERARD
2016-11-15 21:40 ` Konrad Rzeszutek Wilk
@ 2016-11-18 18:51 ` Ian Jackson
1 sibling, 0 replies; 10+ messages in thread
From: Ian Jackson @ 2016-11-18 18:51 UTC (permalink / raw)
To: Anthony PERARD; +Cc: xen-devel
Anthony PERARD writes ("[OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack"):
> This script installs any necessary packages and clones all of the OpenStack
> trees which are used by devstack to deploy OpenStack.
Thanks. I have only trivial comments.
> +proc need-hosts/test-devstack {} { return host }
> +proc run-job/test-devstack {} {
> + run-ts . = ts-openstack-deploy host
^
Should have a `+' here I think. Sorry for not spotting this before.
> + # libvirt is already installed, but not as a package, so avoid installation of
Comment has too-long lines.
> + # the libvirt package with devstack
> + # https://bugs.launchpad.net/devstack/+bug/1641144
> + target_editfile($ho, "$builddir/devstack/files/debs/nova", sub {
> + while (<EI>) {
> + next if m/.*libvirt.*/;
> + print EO or die $!;
> + }
> + });
> + target_editfile($ho, "$builddir/devstack/lib/nova_plugins/functions-libvirt", sub {
Code has too-long lines.
> + service libvirtd restart
> +END
> +
> + # devstack is going to setup the host, install some dependency.
> + target_putfilecontents_root_stash($ho, 100, <<END,"/etc/sudoers.d/devstack");
Too-long line.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack
2016-11-14 12:33 [OSSTEST PATCH v7 0/3] Have OpenStack tested on top of xen's master and libvirt's master Anthony PERARD
2016-11-14 12:33 ` [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack Anthony PERARD
@ 2016-11-14 12:33 ` Anthony PERARD
2016-11-15 21:42 ` Konrad Rzeszutek Wilk
2016-11-14 12:33 ` [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt Anthony PERARD
2 siblings, 1 reply; 10+ messages in thread
From: Anthony PERARD @ 2016-11-14 12:33 UTC (permalink / raw)
To: xen-devel; +Cc: Anthony PERARD, Ian Jackson
This script runs the OpenStack integration test suite, Tempest.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in V7:
- reindent
- acked
No change in V5
Change in V4:
- use \Q\E for tests names
- write the full name of the tests to skip
- use push @ignored_test then join()
- use variables to store common prefix
- rewrite comments
Change in V3:
- Use host as argument of the script.
- Use selecthost() and get rid of $gho.
- Use target_jobdir() instead of builddirsprops().
- Put the ignored Tempest tests into a var and try to explain why there are
skip.
---
---
sg-run-job | 1 +
ts-openstack-tempest | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 65 insertions(+)
create mode 100755 ts-openstack-tempest
diff --git a/sg-run-job b/sg-run-job
index 7b9296c..f086e44 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -477,6 +477,7 @@ proc run-job/test-rumprun {} {
proc need-hosts/test-devstack {} { return host }
proc run-job/test-devstack {} {
run-ts . = ts-openstack-deploy host
+ run-ts . = ts-openstack-tempest host
}
if {[file exists sg-run-job-adhoc]} {
diff --git a/ts-openstack-tempest b/ts-openstack-tempest
new file mode 100755
index 0000000..8cc993a
--- /dev/null
+++ b/ts-openstack-tempest
@@ -0,0 +1,64 @@
+#!/usr/bin/perl
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho = selecthost($whhost);
+our $builddir = target_jobdir($ho);
+
+sub tempest() {
+ my @ignored_tests;
+ my $scenario = 'tempest.scenario';
+ my $volume_boot_pattern =
+ "$scenario.test_volume_boot_pattern.TestVolumeBootPattern";
+ my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance";
+
+ # Ignore tests which try to boot a guest with /dev/vda as boot device name.
+ push @ignored_tests,
+ "^\Q$volume_boot_pattern.test_volume_boot_pattern\E";
+ push @ignored_tests,
+ "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E";
+ push @ignored_tests,
+ "^\Q$shelve_instance.test_shelve_volume_backed_instance\E";
+
+ # Those tests access a volume through iSCSI. This does not work when both
+ # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the
+ # first Linux to have a fix.
+ push @ignored_tests,
+ "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E";
+ push @ignored_tests,
+ "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E";
+
+ # This regex below select the tests to run and exclude the ones marked as
+ # slow as well as the explicit tests listed above. It is based on the one
+ # that can be found in tempest.git/tox.ini in the section [testenv:full].
+ my $ignored_tests = join("|", @ignored_tests);
+ my $regex =
+ "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";
+
+ target_cmd($ho, <<END, 7200);
+ $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
+END
+}
+
+tempest();
--
Anthony PERARD
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack
2016-11-14 12:33 ` [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack Anthony PERARD
@ 2016-11-15 21:42 ` Konrad Rzeszutek Wilk
2016-11-21 17:50 ` Anthony PERARD
0 siblings, 1 reply; 10+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-11-15 21:42 UTC (permalink / raw)
To: Anthony PERARD; +Cc: Ian Jackson, xen-devel
On Mon, Nov 14, 2016 at 12:33:33PM +0000, Anthony PERARD wrote:
> This script runs the OpenStack integration test suite, Tempest.
>
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> ---
> Changes in V7:
> - reindent
> - acked
>
> No change in V5
>
> Change in V4:
> - use \Q\E for tests names
> - write the full name of the tests to skip
> - use push @ignored_test then join()
> - use variables to store common prefix
> - rewrite comments
>
> Change in V3:
> - Use host as argument of the script.
> - Use selecthost() and get rid of $gho.
> - Use target_jobdir() instead of builddirsprops().
> - Put the ignored Tempest tests into a var and try to explain why there are
> skip.
>
> ---
> ---
> sg-run-job | 1 +
> ts-openstack-tempest | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 65 insertions(+)
> create mode 100755 ts-openstack-tempest
>
> diff --git a/sg-run-job b/sg-run-job
> index 7b9296c..f086e44 100755
> --- a/sg-run-job
> +++ b/sg-run-job
> @@ -477,6 +477,7 @@ proc run-job/test-rumprun {} {
> proc need-hosts/test-devstack {} { return host }
> proc run-job/test-devstack {} {
> run-ts . = ts-openstack-deploy host
> + run-ts . = ts-openstack-tempest host
> }
>
> if {[file exists sg-run-job-adhoc]} {
> diff --git a/ts-openstack-tempest b/ts-openstack-tempest
> new file mode 100755
> index 0000000..8cc993a
> --- /dev/null
> +++ b/ts-openstack-tempest
> @@ -0,0 +1,64 @@
> +#!/usr/bin/perl
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2015 Citrix Inc.
2016.
> +#
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU Affero General Public License for more details.
> +#
> +# You should have received a copy of the GNU Affero General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +use strict qw(vars);
> +use Osstest;
> +use Osstest::TestSupport;
> +use Osstest::BuildSupport;
> +
> +tsreadconfig();
> +our ($whhost) = @ARGV;
> +$whhost ||= 'host';
> +our $ho = selecthost($whhost);
> +our $builddir = target_jobdir($ho);
> +
> +sub tempest() {
> + my @ignored_tests;
> + my $scenario = 'tempest.scenario';
> + my $volume_boot_pattern =
> + "$scenario.test_volume_boot_pattern.TestVolumeBootPattern";
> + my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance";
> +
> + # Ignore tests which try to boot a guest with /dev/vda as boot device name.
> + push @ignored_tests,
> + "^\Q$volume_boot_pattern.test_volume_boot_pattern\E";
> + push @ignored_tests,
> + "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E";
> + push @ignored_tests,
> + "^\Q$shelve_instance.test_shelve_volume_backed_instance\E";
> +
> + # Those tests access a volume through iSCSI. This does not work when both
> + # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the
> + # first Linux to have a fix.
What fix? Would it make sense to point this out? Also I am going to
assume the 'server' and 'client' are guests, and 'server' is not dom0?
> + push @ignored_tests,
> + "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E";
> + push @ignored_tests,
> + "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E";
> +
> + # This regex below select the tests to run and exclude the ones marked as
> + # slow as well as the explicit tests listed above. It is based on the one
> + # that can be found in tempest.git/tox.ini in the section [testenv:full].
> + my $ignored_tests = join("|", @ignored_tests);
> + my $regex =
> + "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";
> +
> + target_cmd($ho, <<END, 7200);
> + $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
How come you are hardcoding 2?
> +END
> +}
> +
> +tempest();
> --
> Anthony PERARD
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack
2016-11-15 21:42 ` Konrad Rzeszutek Wilk
@ 2016-11-21 17:50 ` Anthony PERARD
0 siblings, 0 replies; 10+ messages in thread
From: Anthony PERARD @ 2016-11-21 17:50 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: Ian Jackson, xen-devel
On Tue, Nov 15, 2016 at 04:42:50PM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 14, 2016 at 12:33:33PM +0000, Anthony PERARD wrote:
> > + # Those tests access a volume through iSCSI. This does not work when both
> > + # the server and client of iSCSI are on the same Xen host, Linux 4.0 is the
> > + # first Linux to have a fix.
>
> What fix? Would it make sense to point this out? Also I am going to
> assume the 'server' and 'client' are guests, and 'server' is not dom0?
It was a patch series, I don't know if I can find it again. I thought
pointing to a release was enough.
I think iSCSI have other terms for 'server' and 'client', maybe 'target'
and something else. But those two sides are running in dom0, in this
test.
> > + push @ignored_tests,
> > + "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E";
> > + push @ignored_tests,
> > + "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E";
> > +
> > + # This regex below select the tests to run and exclude the ones marked as
> > + # slow as well as the explicit tests listed above. It is based on the one
> > + # that can be found in tempest.git/tox.ini in the section [testenv:full].
> > + my $ignored_tests = join("|", @ignored_tests);
> > + my $regex =
> > + "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";
> > +
> > + target_cmd($ho, <<END, 7200);
> > + $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
>
>
> How come you are hardcoding 2?
By default, this number is equal to the number of cpus. I think it would
be too much, and increase the failure rate of openstack it self, which I
don't think it would be usefull to test.
For their CI loop, I think this is equal to half the number of cpus. We
could do that but I don't know if it's going to be usefull.
We do use 2 on the XenProject OpenStack CI loop, because I don't think
it can do more than that.
--
Anthony PERARD
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt
2016-11-14 12:33 [OSSTEST PATCH v7 0/3] Have OpenStack tested on top of xen's master and libvirt's master Anthony PERARD
2016-11-14 12:33 ` [OSSTEST PATCH v7 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack Anthony PERARD
2016-11-14 12:33 ` [OSSTEST PATCH v7 2/3] ts-openstack-tempest: Run Tempest to check OpenStack Anthony PERARD
@ 2016-11-14 12:33 ` Anthony PERARD
2016-11-18 18:57 ` Ian Jackson
2 siblings, 1 reply; 10+ messages in thread
From: Anthony PERARD @ 2016-11-14 12:33 UTC (permalink / raw)
To: xen-devel; +Cc: Anthony PERARD, Ian Jackson
This patch should create a flight "openstack-nova", with those jobs:
build-amd64
build-amd64-xsm
build-amd64-pvops
build-amd64-libvirt
test-amd64-amd64-devstack
test-amd64-amd64-devstack-xsm
About the runvars revision_* of test-*-*-devstack:
only REVISION_OPENSTACK_NOVA is set, the others are unset.
Empty revision_* runvar would clone the default branch, which should
be master for every openstack repos.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
Changes in V7:
- metaprograming
- reindent
Change in V6:
- rebased
Change in V5:
- rename flight openstack to openstack-nova
- add -xsm variant of the test
- run test-devstack only on openstack-nova flight
Change in V4:
- also skip build-*-oldkern in make flight
- fix select_xenbranch
- set revision_*=$REVISION_OPENSTACK_* in make-flight
(was revision_*=master before)
only REVISION_OPENSTACK_NOVA is set, the others are unset.
empty revision_* runvar would clone the default branch, which should
be master for every openstack repos
Change in V3:
- Switch to "track" Nova tree instead of devstack.
Nova is the service we care about from a Xen point of view.
Also it is updated much more often than devstack.
- Use TREE_OPENSTACK_ as prefix for all trees variables.
- Change the filter, keep only *-devstack jobs.
- Add stuff into ./ap-push
- Add stuff into ./cr-daily-branch.
- Add 'openstack' into ./cr-for-branches.
---
ap-common | 18 ++++++++++++++++++
ap-fetch-version | 4 ++++
ap-fetch-version-old | 5 +++++
ap-print-url | 3 +++
ap-push | 5 +++++
cr-daily-branch | 8 ++++++++
cr-for-branches | 2 +-
cri-common | 1 +
make-flight | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
9 files changed, 93 insertions(+), 1 deletion(-)
diff --git a/ap-common b/ap-common
index cbb815c..26534a5 100644
--- a/ap-common
+++ b/ap-common
@@ -54,6 +54,23 @@
: ${PUSH_TREE_OVMF:=$XENBITS:/home/xen/git/osstest/ovmf.git}
: ${BASE_TREE_OVMF:=git://xenbits.xen.org/osstest/ovmf.git}
+define_openstack_trees() {
+ local openstack_trees=(cinder glance keystone nova requirements tempest)
+ local tree
+ local url
+
+ : ${GIT_OPENSTACK_ORG:=git://git.openstack.org}
+ : ${TREE_OPENSTACK_DEVSTACK:=$GIT_OPENSTACK_ORG/openstack-dev/devstack.git}
+ for tree in ${openstack_trees[@]}; do
+ url=$GIT_OPENSTACK_ORG/openstack/$tree.git
+ eval ": \${TREE_OPENSTACK_${tree^^}:=$url}"
+ done
+}
+
+define_openstack_trees
+: ${PUSH_TREE_OPENSTACK_NOVA:=$XENBITS:/home/xen/git/osstest/openstack-nova.git}
+: ${BASE_TREE_OPENSTACK_NOVA:=git://xenbits.xen.org/osstest/openstack-nova.git}
+
: ${TREE_LINUXFIRMWARE:=git://xenbits.xen.org/osstest/linux-firmware.git}
: ${PUSH_TREE_LINUXFIRMWARE:=$XENBITS:/home/osstest/ext/linux-firmware.git}
: ${UPSTREAM_TREE_LINUXFIRMWARE:=$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git}
@@ -82,6 +99,7 @@ fi
: ${LOCALREV_SEABIOS:=daily-cron.$branch}
: ${LOCALREV_OVMF:=daily-cron.$branch}
: ${LOCALREV_XTF:=daily-cron.$branch}
+: ${LOCALREV_OPENSTACK_NOVA:=daily-cron.$branch}
: ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
diff --git a/ap-fetch-version b/ap-fetch-version
index a107c93..a714ee2 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -106,6 +106,10 @@ ovmf)
repo_tree_rev_fetch_git ovmf \
$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
;;
+openstack-nova)
+ repo_tree_rev_fetch_git openstack-nova \
+ $TREE_OPENSTACK_NOVA master $LOCALREV_OPENSTACK_NOVA
+ ;;
osstest)
if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
git update-ref -m "Arranging to test HEAD" \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3cbc176..6dddbb7 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -35,6 +35,7 @@ check_ap_fetch_placeholders
: ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
: ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
: ${BASE_TAG_LIBVIRT:=xen-tested-master}
+: ${BASE_LOCALREV_OPENSTACK_NOVA:=daily-cron.$branch.old}
if info_linux_tree "$branch"; then
repo_tree_rev_fetch_git linux \
@@ -114,6 +115,10 @@ ovmf)
repo_tree_rev_fetch_git ovmf \
$BASE_TREE_OVMF xen-tested-master $BASE_LOCALREV_OVMF
;;
+openstack-nova)
+ repo_tree_rev_fetch_git openstack-nova \
+ $BASE_TREE_OPENSTACK_NOVA xen-tested-master $BASE_LOCALREV_OPENSTACK_NOVA
+ ;;
osstest)
if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
git fetch -f $HOME/testing.git production:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index 93c14b3..6f4e6b1 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -67,6 +67,9 @@ ovmf)
osstest)
echo none:;
;;
+openstack-nova)
+ echo $TREE_OPENSTACK_NOVA
+ ;;
*)
echo >&2 "branch $branch ?"
exit 1
diff --git a/ap-push b/ap-push
index a27ccc2..136d1b6 100755
--- a/ap-push
+++ b/ap-push
@@ -41,6 +41,7 @@ TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
TREE_SEABIOS=$PUSH_TREE_SEABIOS
TREE_OVMF=$PUSH_TREE_OVMF
TREE_XTF=$PUSH_TREE_XTF
+TREE_OPENSTACK_NOVA=$PUSH_TREE_OPENSTACK_NOVA
if info_linux_tree "$branch"; then
cd $repos/linux
@@ -129,6 +130,10 @@ ovmf)
cd $repos/ovmf
git push $TREE_OVMF $revision:refs/heads/xen-tested-master
;;
+openstack-nova)
+ cd $repos/openstack-nova
+ git push $TREE_OPENSTACK_NOVA $revision:refs/heads/xen-tested-master
+ ;;
osstest)
git push $HOME/testing.git $revision:production
if [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
diff --git a/cr-daily-branch b/cr-daily-branch
index 158554d..45a2f69 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -207,6 +207,10 @@ if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
export REVISION_LINUXFIRMWARE
fi
+if [ "x$REVISION_OPENSTACK_NOVA" = x ]; then
+ determine_version REVISION_OPENSTACK_NOVA openstack-nova OPENSTACK_NOVA
+ export REVISION_OPENSTACK_NOVA
+fi
if [ "x$branch" != "xxen-unstable" ]; then
export REVISION_LINUX_OLD=disable
@@ -260,6 +264,10 @@ ovmf)
realtree=ovmf
NEW_REVISION=$REVISION_OVMF
;;
+openstack-nova)
+ realtree=openstack-nova
+ NEW_REVISION=$REVISION_OPENSTACK_NOVA
+ ;;
*)
NEW_REVISION=''
wantpush=false
diff --git a/cr-for-branches b/cr-for-branches
index 5e8b1a4..ebe8986 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
LOGFILE=tmp/cr-for-branches.log
export LOGFILE
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf openstack-nova ${EXTRA_BRANCHES}}
export BRANCHES
fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index ac360de..f48dfa5 100644
--- a/cri-common
+++ b/cri-common
@@ -82,6 +82,7 @@ select_xenbranch () {
ovmf) tree=ovmf; xenbranch=xen-unstable ;;
distros-*) tree=none; xenbranch=xen-unstable ;;
osstest) tree=osstest; xenbranch=xen-unstable ;;
+ openstack-nova) tree=openstack-nova; xenbranch=xen-unstable ;;
esac
if [ "x$tree" = xlinux ]; then
linuxbranch=$branch
diff --git a/make-flight b/make-flight
index a374884..344e9e6 100755
--- a/make-flight
+++ b/make-flight
@@ -167,6 +167,17 @@ job_create_test_filter_callback () {
*) return 1;;
esac
;;
+ openstack-nova)
+ case "$job" in
+ *-devstack) ;;
+ *-devstack-xsm) ;;
+ *) return 1;;
+ esac
+ case $dom0arch in
+ amd64) ;;
+ *) return 1;;
+ esac
+ ;;
*)
case "$job" in
*-qemuu-*)
@@ -198,6 +209,12 @@ arch_branch_filter_callback () {
rumprun) return 1;;
seabios) return 1;;
ovmf) return 1;;
+ openstack-nova) return 1;;
+ esac
+ ;;
+ i386)
+ case "$branch" in
+ openstack-nova) return 1;;
esac
;;
i386|amd64)
@@ -606,6 +623,35 @@ do_pv_debian_tests () {
$endfmt$dom0arch
}
+do_openstack_tests () {
+ local xsms=$(xenbranch_xsm_variants)
+ local openstack_trees=(cinder devstack glance keystone nova
+ requirements tempest)
+
+
+ if [ $dom0arch != amd64 ]; then
+ return
+ fi
+ if [ $branch != openstack-nova ]; then
+ return
+ fi
+
+ local os_runvars=""
+ for tree in ${openstack_trees[@]}; do
+ eval "os_runvars+=\" tree_$tree=\${TREE_OPENSTACK_${tree^^}}\""
+ eval "os_runvars+=\" revision_$tree=\${REVISION_OPENSTACK_${tree^^}}\""
+ done
+
+ for xsm in $xsms ; do
+ job_create_test test-$xenarch$kern-$dom0arch-devstack \
+ test-devstack libvirt $xenarch $dom0arch \
+ $os_runvars \
+ dom0_mem=4000 \
+ enable_xsm=$xsm \
+ all_hostflags=$most_hostflags
+ done
+}
+
test_matrix_do_one () {
do_pv_debian_tests
@@ -753,6 +799,8 @@ test_matrix_do_one () {
do_pvgrub_tests
do_xtf_tests
+
+ do_openstack_tests
}
if [ x$buildflight = x ]; then
--
Anthony PERARD
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt
2016-11-14 12:33 ` [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt Anthony PERARD
@ 2016-11-18 18:57 ` Ian Jackson
2016-11-18 19:51 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 10+ messages in thread
From: Ian Jackson @ 2016-11-18 18:57 UTC (permalink / raw)
To: Anthony PERARD; +Cc: Ian Jackson, xen-devel
Anthony PERARD writes ("[Xen-devel] [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt"):
> This patch should create a flight "openstack-nova", with those jobs:
Did I already ask for a diff of the output of
standalone-generate-dump-flight-runvars
before and after ?
You probably want
AP_FETCH_PLACEHOLDERS=y eatmydata ./standalone-generate-dump-flight-runvars
Your new shell code is mostly good. I have a style comment:
> +define_openstack_trees() {
> + local openstack_trees=(cinder glance keystone nova requirements tempest)
The use of an array variable here is not necessary since none of these
contain spaces. I don't object, except that:
> + for tree in ${openstack_trees[@]}; do
This sets off my style alarm because of the lack of "" around a
${something[@]} (which off course doesn't matter here.
Can you either add the " ", or switch to a plain (space-separated)
variable.
> + for tree in ${openstack_trees[@]}; do
Here too.
Aside from that, and assuming the runvars diff is good, this is
looking good.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt
2016-11-18 18:57 ` Ian Jackson
@ 2016-11-18 19:51 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 10+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-11-18 19:51 UTC (permalink / raw)
To: Ian Jackson; +Cc: Anthony PERARD, xen-devel
On Fri, Nov 18, 2016 at 06:57:52PM +0000, Ian Jackson wrote:
> Anthony PERARD writes ("[Xen-devel] [OSSTEST PATCH v7 3/3] Create a flight to test OpenStack with xen-unstable and libvirt"):
> > This patch should create a flight "openstack-nova", with those jobs:
>
> Did I already ask for a diff of the output of
> standalone-generate-dump-flight-runvars
> before and after ?
You asked that on my patch set :-)
Perhaps it should be part of the README.devel?
>
> You probably want
> AP_FETCH_PLACEHOLDERS=y eatmydata ./standalone-generate-dump-flight-runvars
>
> Your new shell code is mostly good. I have a style comment:
>
> > +define_openstack_trees() {
> > + local openstack_trees=(cinder glance keystone nova requirements tempest)
>
> The use of an array variable here is not necessary since none of these
> contain spaces. I don't object, except that:
>
> > + for tree in ${openstack_trees[@]}; do
>
> This sets off my style alarm because of the lack of "" around a
> ${something[@]} (which off course doesn't matter here.
>
> Can you either add the " ", or switch to a plain (space-separated)
> variable.
>
> > + for tree in ${openstack_trees[@]}; do
>
> Here too.
>
> Aside from that, and assuming the runvars diff is good, this is
> looking good.
I concur. I used his patchset to grok the OSSTest.
>
> Ian.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread