Re: [PATCH OSSTEST v3 2/3] ts-openstack-tempest: Run Tempest to check OpenStack

[Ian Campbell <ian.campbell@citrix.com>]

On Tue, 2015-09-29 at 18:15 +0100, Anthony PERARD wrote:
> On Tue, Sep 29, 2015 at 04:43:50PM +0100, Ian Campbell wrote:
> > On Mon, 2015-09-28 at 16:56 +0100, Anthony PERARD wrote:
> > 
> > > +  # Ignore these tests:
> > > +  #
> > > tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_
> > > volume_boot_pattern
> > > +  #   It try to start a guest with /dev/vda as boot device name.
> > > +  $ignored_tests .=
> > > '|.*TestVolumeBootPattern.test_volume_boot_pattern';
> > 
> > Don't you need to escape the second "." for the regex context in the
> > tempest script? \Q and \E might be helpful for this?
> 
> Yes, I should probably escape the dot. \Q and \E is for perl, isn't it?
> The regex is going to be interpreted by python.

$foo = "\Qtest.case\E" ends up with $foo having the a literal value which
is escaped i.e. "test\.case", which then gets passed to whichever regex
(including Pythons) and does the right thing.

e.g. 

$ perl -e 'print "\Qtest.case\E\n";'
test\.case
$ perl -e 'print "\Qtest.case*\E\n";'
test\.case\*

There is a subtle reliance on the Perl quoting rules and the Python quoting
rules being similar enough, but since these are regexp's I think that is
safe enough. We also assume Perl quoting is compatible with Shell quoting
in various places.

> > Also, you actually ignore any test ending with that string, not just
> > the
> > exact one in the comment, is that deliberate? Also any test with a
> > longer
> > name which has this substring will also be ignored.
> > 
> > If not then you could put the common prefix into a variable to avoid
> > repeating it (and the very long lines) and maybe add some anchors?
> 
> I'll clean that up, and use a prefix.
> 
> What are you speaking about when you say "anchors"?

I mean anchoring the match to the start of the string with ^ or to the end
with $. i.e. /^foo$/ only matches "foo" and not "foobar" or "bazfoo" etc.


> > > +  #
> > > tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_
> > > volume_backed_instance
> > > +  #   It try to start a guest with /dev/vda as boot device name.
> > > +  $ignored_tests .=
> > > '|.*TestShelveInstance.test_shelve_volume_backed_instance';
> > > +  #
> > > scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume
> > > _boot_pattern
> > > +  #   It access a volume through iSCSI. This does not work when both
> > > the
> > > +  #   server and client of iSCSI are on the same Xen host, Linux 4.0
> > > is the
> > > +  #   first Linux to have a fix.
> > > +  $ignored_tests .=
> > > '|.*TestVolumeBootPatternV2.test_volume_boot_pattern';
> > > +
> > > +  # Adding the tests to skip to the default regex.
> > > +  my $regex =
> > > "(?!.*\\[.*\\bslow\\b.*\\]$ignored_tests)(^tempest\\.(api|scenario|th
> > > irdparty))";
> > 
> > Where the default regex is:
> >     "(?!.*\\[.*\\bslow\\b.*\\])(^tempest\\.(api|scenario|thirdparty))";
> > ?
> 
> This regex of tests to execute can be found in tempest.git/tox.ini, in
> the
> section "[testenv:full]". That what's been used the OpenStack gate.
> 
> I'll add a comment about it in the next patch version.

Given that the upstream default might change under our feet maybe we should
just consider this to be our own decision to make, even if it happens to
match the default today?

> > This is a pretty rubbish interface which tempest has given you. What a
> > pain!
> 
> Yeah, that the only way to ignore a test on a specific third-party
> openstack CI loop...

Yuck. Oh well.

Ian.