Re: [PATCH OSSTEST v3 2/3] ts-openstack-tempest: Run Tempest to check OpenStack

[Anthony PERARD <anthony.perard@citrix.com>]

On Tue, Sep 29, 2015 at 04:43:50PM +0100, Ian Campbell wrote:
> On Mon, 2015-09-28 at 16:56 +0100, Anthony PERARD wrote:
> 
> > +  # Ignore these tests:
> > +  # tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
> > +  #   It try to start a guest with /dev/vda as boot device name.
> > +  $ignored_tests .= '|.*TestVolumeBootPattern.test_volume_boot_pattern';
> 
> Don't you need to escape the second "." for the regex context in the
> tempest script? \Q and \E might be helpful for this?

Yes, I should probably escape the dot. \Q and \E is for perl, isn't it?
The regex is going to be interpreted by python.

> Also, you actually ignore any test ending with that string, not just the
> exact one in the comment, is that deliberate? Also any test with a longer
> name which has this substring will also be ignored.
> 
> If not then you could put the common prefix into a variable to avoid
> repeating it (and the very long lines) and maybe add some anchors?

I'll clean that up, and use a prefix.

What are you speaking about when you say "anchors"?

> You might also want to consider
> 
>     push @ignored_tests, "^$prefix\QTestVolumeBootPattern.test_volume_boot_pattern$";
> 
> Then later:
>     $ignored_tests = join("|", @ignored_tests);

I'll look into that.

> (I'm not sure, but I don't think you want qr{} [0] here, I might be wrong
> though)
> 
> > +  # tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance
> > +  #   It try to start a guest with /dev/vda as boot device name.
> > +  $ignored_tests .= '|.*TestShelveInstance.test_shelve_volume_backed_instance';
> > +  # scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
> > +  #   It access a volume through iSCSI. This does not work when both the
> > +  #   server and client of iSCSI are on the same Xen host, Linux 4.0 is the
> > +  #   first Linux to have a fix.
> > +  $ignored_tests .= '|.*TestVolumeBootPatternV2.test_volume_boot_pattern';
> > +
> > +  # Adding the tests to skip to the default regex.
> > +  my $regex = "(?!.*\\[.*\\bslow\\b.*\\]$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";
> 
> Where the default regex is:
>     "(?!.*\\[.*\\bslow\\b.*\\])(^tempest\\.(api|scenario|thirdparty))";
> ?

This regex of tests to execute can be found in tempest.git/tox.ini, in the
section "[testenv:full]". That what's been used the OpenStack gate.

I'll add a comment about it in the next patch version.

> This is a pretty rubbish interface which tempest has given you. What a
> pain!

Yeah, that the only way to ignore a test on a specific third-party
openstack CI loop...

-- 
Anthony PERARD