[Qemu-devel] Question for iotests 188, 189 and 087

[Qemu-devel] Question for iotests 188, 189 and 087

[Jing Liu]

Hi all,

Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu 
upsteam?

I just wondered if it has something wrong with my test machines because 
I have different results with two machines.


Thanks.

Jing

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On 07/17/2017 10:01 PM, Jing Liu wrote:
> Hi all,
> 
> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
> upsteam?
> 
> I just wondered if it has something wrong with my test machines because
> I have different results with two machines.

It might help to show the ./check command line you are using, as well as
the errors you are getting.  Right now, master is failing on 51, 140,
and 143 when I test across './check -raw', './check -qcow2', and
'./check nbd', but not the three you mention.  But the iotests framework
allows a lot more combinations of testing (and we probably have some
lurking testsuite bugs on some of the more exotic combinations).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[John Snow]

On 07/17/2017 11:01 PM, Jing Liu wrote:
> Hi all,
> 
> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
> upsteam?
> 
> I just wondered if it has something wrong with my test machines because
> I have different results with two machines.
> 
> 
> Thanks.
> 
> Jing
> 
> 

Presumably you are missing aio=native support for 087,

and 188/189 I believe require compression and/or LUKS libraries.

These are false positives caused by missing libraries. We should
probably work out a proper solution to meaningfully skipping tests we
didn't compile support for.

--js

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 3033 bytes --]



On 07/18/2017 02:07 PM, John Snow wrote:
> 
> 
> On 07/17/2017 11:01 PM, Jing Liu wrote:
>> Hi all,
>>
>> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
>> upsteam?
>>
>> I just wondered if it has something wrong with my test machines because
>> I have different results with two machines.
>>
>>
>> Thanks.
>>
>> Jing
>>
>>
> 
> Presumably you are missing aio=native support for 087,
> 

I see 6 different "tests" as part of 087, and only one them requires AIO
support ("aio=native without O_DIRECT"), which can fail like this:

--- /home/cleber/src/qemu/tests/qemu-iotests/087.out    2017-07-17
19:33:26.409758360 -0400
+++ 087.out.bad 2017-07-18 17:01:37.736038689 -0400
@@ -27,7 +27,7 @@
 Testing:
 QMP_VERSION
 {"return": {}}
-{"error": {"class": "GenericError", "desc": "aio=native was specified,
but it requires cache.direct=on, which was not specified."}}
+{"error": {"class": "GenericError", "desc": "aio=native was specified,
but is not supported in this build."}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
"event": "SHUTDOWN", "data": {"guest": false}}

Failures: 087
Failed 1 of 1 tests

When either "--disable-linux-aio" is given or when libaio headers are
missing, which in the end means that CONFIG_LINUX_AIO won't be set in
config-host.mak or config-host.h.

> and 188/189 I believe require compression and/or LUKS libraries.
> 

I did not check if/which optional configure time option would end up
disabling LUKS support and affecting those tests, but it's possible that
something similar exists.

> These are false positives caused by missing libraries. We should
> probably work out a proper solution to meaningfully skipping tests we
> didn't compile support for.
> 
> --js
> 

I see issues here:

1) The qemu-iotests "runner", that is, "./check", understands that a
file number is a test.  No further granularity is (currently?)
achievable here.

The easy solution would be to split tests which depend on optional
features to separate test files.  In the 087 test given as example,
the "=== aio=native without O_DIRECT ===" block would become, say, 190.

2) Skip tests based on features more easily.  There's already support
for skipping tests on various situations.  From 087:

...
_supported_fmt qcow2
_supported_proto file
_supported_os Linux
...

It's trivial to also have access to other compile time settings.  I did
a quick experiment that would add a "_required_feature" function to
"common.rc".  For 087 (or 190?) it would look like:

...
_supported_fmt qcow2
_supported_proto file
_supported_os Linux
_required_feature CONFIG_LINUX_AIO
...

Does it make sense?  Would a patch series along those lines would be
welcomed by reviews?

Thanks!

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[John Snow]

On 07/18/2017 05:22 PM, Cleber Rosa wrote:
> 
> 
> On 07/18/2017 02:07 PM, John Snow wrote:
>>
>>
>> On 07/17/2017 11:01 PM, Jing Liu wrote:
>>> Hi all,
>>>
>>> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
>>> upsteam?
>>>
>>> I just wondered if it has something wrong with my test machines because
>>> I have different results with two machines.
>>>
>>>
>>> Thanks.
>>>
>>> Jing
>>>
>>>
>>
>> Presumably you are missing aio=native support for 087,
>>
> 
> I see 6 different "tests" as part of 087, and only one them requires AIO
> support ("aio=native without O_DIRECT"), which can fail like this:
> 
> --- /home/cleber/src/qemu/tests/qemu-iotests/087.out    2017-07-17
> 19:33:26.409758360 -0400
> +++ 087.out.bad 2017-07-18 17:01:37.736038689 -0400
> @@ -27,7 +27,7 @@
>  Testing:
>  QMP_VERSION
>  {"return": {}}
> -{"error": {"class": "GenericError", "desc": "aio=native was specified,
> but it requires cache.direct=on, which was not specified."}}
> +{"error": {"class": "GenericError", "desc": "aio=native was specified,
> but is not supported in this build."}}
>  {"return": {}}
>  {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
> "event": "SHUTDOWN", "data": {"guest": false}}
> 
> Failures: 087
> Failed 1 of 1 tests
> 
> When either "--disable-linux-aio" is given or when libaio headers are
> missing, which in the end means that CONFIG_LINUX_AIO won't be set in
> config-host.mak or config-host.h.
> 
>> and 188/189 I believe require compression and/or LUKS libraries.
>>
> 
> I did not check if/which optional configure time option would end up
> disabling LUKS support and affecting those tests, but it's possible that
> something similar exists.
> 
>> These are false positives caused by missing libraries. We should
>> probably work out a proper solution to meaningfully skipping tests we
>> didn't compile support for.
>>
>> --js
>>
> 
> I see issues here:
> 
> 1) The qemu-iotests "runner", that is, "./check", understands that a
> file number is a test.  No further granularity is (currently?)
> achievable here.
> 

Yes.

> The easy solution would be to split tests which depend on optional
> features to separate test files.  In the 087 test given as example,
> the "=== aio=native without O_DIRECT ===" block would become, say, 190.
> 

Are we married to the numbers? Can we do 087a, 087b? Or even start
naming them so we can organize them in some meaningful sense?

> 2) Skip tests based on features more easily.  There's already support
> for skipping tests on various situations.  From 087:
> 
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> ...
> 
> It's trivial to also have access to other compile time settings.  I did
> a quick experiment that would add a "_required_feature" function to
> "common.rc".  For 087 (or 190?) it would look like:
> 
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> _required_feature CONFIG_LINUX_AIO
> ...
> 
> Does it make sense?  Would a patch series along those lines would be
> welcomed by reviews?
> 
> Thanks!
> 

Makes a lot of sense to me, but it would be nice to differentiate the
output from the ./check script into some broad categories:

(1) We ran the test(s), and it/they passed!
(2) We ran the test(s), and it/they failed.
(3) We didn't run the test, but that's OK (e.g. ./check -v -raw for a
test that only supports qcow2; that is, it's normal/expected for this
test to have been skipped in this way)
(4) We didn't run the test, and this is a coverage gap (e.g. libaio
support missing, so we cannot meaningfully test.)


Importantly, 3 and 4 are very different and we only support 1-3 today.
Most people run ./check -qcow2 or ./check -raw (if they run iotests at
all) but that still leaves some protocols and formats in the lurch, but
they're fairly regularly skipped regardless.

We have no "run EVERYTHING!" invocation that does raw, qcow2, luks (and
everything!!) all at once, so "SKIPS" currently have the semantic meaning:

"I didn't run this, but it's not a big deal, really."

If we start skipping tests because of missing libraries or compile
options, some skips take on the meaning:

"We didn't run this, but that means you didn't test some stuff that you
really ought to have."

I guess the key difference here is:

(1) The first skip mode doesn't change depending on what your
environment is, it only changes based on what you ask it to run, but

(2) These skips are environmentally dependent, and should be visually
identified as failures of the test runner to even attempt the test,
which is semantically rather distinct from (1) above

Clear as mud?

Great!

--js

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

On 07/18/2017 04:22 PM, Cleber Rosa wrote:

> +++ 087.out.bad 2017-07-18 17:01:37.736038689 -0400
> @@ -27,7 +27,7 @@
>  Testing:
>  QMP_VERSION
>  {"return": {}}
> -{"error": {"class": "GenericError", "desc": "aio=native was specified,
> but it requires cache.direct=on, which was not specified."}}
> +{"error": {"class": "GenericError", "desc": "aio=native was specified,
> but is not supported in this build."}}
>  {"return": {}}
>  {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
> "event": "SHUTDOWN", "data": {"guest": false}}
> 
> Failures: 087
> Failed 1 of 1 tests
> 
> When either "--disable-linux-aio" is given or when libaio headers are
> missing, which in the end means that CONFIG_LINUX_AIO won't be set in
> config-host.mak or config-host.h.

Or, instead of splitting the test, we could just add another post-test
filter that does sed 's/"aio=native[^"]*/"aio=native missing/' and
update the expected output accordingly.

> 2) Skip tests based on features more easily.  There's already support
> for skipping tests on various situations.  From 087:
> 
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> ...

There's also _notrun (for example, see test 183, which checks whether
'migrate -b' works before getting on to the rest of the test).

> 
> It's trivial to also have access to other compile time settings.  I did
> a quick experiment that would add a "_required_feature" function to
> "common.rc".  For 087 (or 190?) it would look like:
> 
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> _required_feature CONFIG_LINUX_AIO
> ...
> 
> Does it make sense?  Would a patch series along those lines would be
> welcomed by reviews?

Seems reasonable, if it makes it easier to mark particular tests as
requiring certain subsets of configure options.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Jing Liu]

Hi Eric,


On 2017/7/18 下午10:57, Eric Blake wrote:
> On 07/17/2017 10:01 PM, Jing Liu wrote:
>> Hi all,
>>
>> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
>> upsteam?
>>
>> I just wondered if it has something wrong with my test machines because
>> I have different results with two machines.
> It might help to show the ./check command line you are using, as well as
> the errors you are getting.  Right now, master is failing on 51, 140,
> and 143 when I test across './check -raw', './check -qcow2', and
> './check nbd',
I used ./check -qcow2 to do it.
I tested on different machines and got different results. One is failed 
on 140 and 143. The other one told
me 087, 140, 143, 188, 189.
I guess it is because some configuration or libraries, and will also 
refer to other replies from the mail.
I'll check the details.

Thank you so much.

Jing


> but not the three you mention.  But the iotests framework
> allows a lot more combinations of testing (and we probably have some
> lurking testsuite bugs on some of the more exotic combinations).
>

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Jing Liu]

On 2017/7/19 上午2:07, John Snow wrote:
>
> On 07/17/2017 11:01 PM, Jing Liu wrote:
>> Hi all,
>>
>> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
>> upsteam?
>>
>> I just wondered if it has something wrong with my test machines because
>> I have different results with two machines.
>>
>>
>> Thanks.
>>
>> Jing
>>
>>
> Presumably you are missing aio=native support for 087,
>
> and 188/189 I believe require compression and/or LUKS libraries.
Hi John,

Yes, I indeed miss some crypto libraries.
Thank you for your support. Now all of them are fixed.

Jing
> These are false positives caused by missing libraries. We should
> probably work out a proper solution to meaningfully skipping tests we
> didn't compile support for.
>
> --js
>

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 8844 bytes --]



On 07/18/2017 05:47 PM, John Snow wrote:
> 
> 
> On 07/18/2017 05:22 PM, Cleber Rosa wrote:
>>
>>
>> On 07/18/2017 02:07 PM, John Snow wrote:
>>>
>>>
>>> On 07/17/2017 11:01 PM, Jing Liu wrote:
>>>> Hi all,
>>>>
>>>> Do you anybody have iotests failure: 188, 189 and 087 of the latest qemu
>>>> upsteam?
>>>>
>>>> I just wondered if it has something wrong with my test machines because
>>>> I have different results with two machines.
>>>>
>>>>
>>>> Thanks.
>>>>
>>>> Jing
>>>>
>>>>
>>>
>>> Presumably you are missing aio=native support for 087,
>>>
>>
>> I see 6 different "tests" as part of 087, and only one them requires AIO
>> support ("aio=native without O_DIRECT"), which can fail like this:
>>
>> --- /home/cleber/src/qemu/tests/qemu-iotests/087.out    2017-07-17
>> 19:33:26.409758360 -0400
>> +++ 087.out.bad 2017-07-18 17:01:37.736038689 -0400
>> @@ -27,7 +27,7 @@
>>  Testing:
>>  QMP_VERSION
>>  {"return": {}}
>> -{"error": {"class": "GenericError", "desc": "aio=native was specified,
>> but it requires cache.direct=on, which was not specified."}}
>> +{"error": {"class": "GenericError", "desc": "aio=native was specified,
>> but is not supported in this build."}}
>>  {"return": {}}
>>  {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP},
>> "event": "SHUTDOWN", "data": {"guest": false}}
>>
>> Failures: 087
>> Failed 1 of 1 tests
>>
>> When either "--disable-linux-aio" is given or when libaio headers are
>> missing, which in the end means that CONFIG_LINUX_AIO won't be set in
>> config-host.mak or config-host.h.
>>
>>> and 188/189 I believe require compression and/or LUKS libraries.
>>>
>>
>> I did not check if/which optional configure time option would end up
>> disabling LUKS support and affecting those tests, but it's possible that
>> something similar exists.
>>
>>> These are false positives caused by missing libraries. We should
>>> probably work out a proper solution to meaningfully skipping tests we
>>> didn't compile support for.
>>>
>>> --js
>>>
>>
>> I see issues here:
>>
>> 1) The qemu-iotests "runner", that is, "./check", understands that a
>> file number is a test.  No further granularity is (currently?)
>> achievable here.
>>
> 
> Yes.
> 
>> The easy solution would be to split tests which depend on optional
>> features to separate test files.  In the 087 test given as example,
>> the "=== aio=native without O_DIRECT ===" block would become, say, 190.
>>
> 
> Are we married to the numbers? Can we do 087a, 087b? Or even start
> naming them so we can organize them in some meaningful sense?
> 

Yes, check only understands numbered tests.  Naming a test "087a"
currently makes it run "087" instead.  And asking it run "abc" results
in "000 - unknown test, ignored".

Of course the limitation could be lifted, but there's at least one
feature that I'm aware that would be impacted: running test ranges (say,
012-021).  With test being sequentially added, and with the existence of
groups, I really wonder if ranges still make any sense.

>> 2) Skip tests based on features more easily.  There's already support
>> for skipping tests on various situations.  From 087:
>>
>> ...
>> _supported_fmt qcow2
>> _supported_proto file
>> _supported_os Linux
>> ...
>>
>> It's trivial to also have access to other compile time settings.  I did
>> a quick experiment that would add a "_required_feature" function to
>> "common.rc".  For 087 (or 190?) it would look like:
>>
>> ...
>> _supported_fmt qcow2
>> _supported_proto file
>> _supported_os Linux
>> _required_feature CONFIG_LINUX_AIO
>> ...
>>
>> Does it make sense?  Would a patch series along those lines would be
>> welcomed by reviews?
>>
>> Thanks!
>>
> 
> Makes a lot of sense to me, but it would be nice to differentiate the
> output from the ./check script into some broad categories:
> 
> (1) We ran the test(s), and it/they passed!
> (2) We ran the test(s), and it/they failed.
> (3) We didn't run the test, but that's OK (e.g. ./check -v -raw for a
> test that only supports qcow2; that is, it's normal/expected for this
> test to have been skipped in this way)
> (4) We didn't run the test, and this is a coverage gap (e.g. libaio
> support missing, so we cannot meaningfully test.)
> 
> 

I understand how you see #3 and #4 differently when a "test coverage"
policy is applied to the scene.  Writing tests that can adapt to
different policies is possible, but the policy itself should really live
elsewhere.  This simplifies the test itself, and makes #3 and #4
basically the same thing.

For instance, let's suppose "./check" gets a "-skip-features=" option,
which, to implement the default policy of complete coverage, defaults to
an empty list.  The test itself could contain something like:

  skipIf("libaio" in skip_features)

The default policy on a platform that lacks libaio, would be one with
'-skip-features=libaio'.

On scenarios where complete coverage is *expected* ("skip_features" is
empty), the lack of libaio on such a test *demands* an ERROR.  On test
frameworks that distinguish between ERRORs and FAILures, this means that
the test code itself could not be run successfully and this demands an
action from whoever is running the test (installing libaio and/or
running configure with --enable-linux-aio).

> Importantly, 3 and 4 are very different and we only support 1-3 today.
> Most people run ./check -qcow2 or ./check -raw (if they run iotests at
> all) but that still leaves some protocols and formats in the lurch, but
> they're fairly regularly skipped regardless.
> 

Right.  Combining different parameters and thus expanding test coverage
(leaving nothing, or as little as possible, in the lurch) is a very
interesting topic.  My quick words about it is that it's possible to use
tools to combine protocols, formats and what else automatically, while
still making obvious exclusions.

> We have no "run EVERYTHING!" invocation that does raw, qcow2, luks (and
> everything!!) all at once, so "SKIPS" currently have the semantic meaning:
> 

Using a tools I'm familiar with, I create a
"$QEMU_SRC/tests/qemu-io-tests/complete-coverate.yaml" with the
following content (really a copy and paste from ./check --help):

---
format: !mux
  raw:
  boochs:
  cloop:
  parallels:
  qcow:
  qcow2:
  qed:
  vdi:
  vpc:
  vhdx:
  vmdk:
  luks:

protocol: !mux
  file:
  rbd:
  sheepdog:
    !filter-out : /run/format/cloop
  nbd:
  ssh:
  nfs:
  vxhs:
---

This "policy" combines all formats with all protocols, but excludes
cloop+sheepdog, which (let's suppose), are known to be incompatible.
Again, using the tool I'm familiar with, I'd run my "complete coverage" job:

 $ avocado run 001 002 -m complete-coverage.yaml
 ...
 (001/166) 001;raw-file-4e7c:  PASS (0.01 s)
 (002/166) 001;raw-rbd-c698:  PASS (0.01 s)
 (003/166) 001;raw-sheepdog-9705:  PASS (0.01 s)
 (004/166) 001;raw-nbd-1420:  PASS (0.01 s)
 (005/166) 001;raw-ssh-ed68:  PASS (0.01 s)
 (006/166) 001;raw-nfs-ce6b:  PASS (0.01 s)
 (007/166) 001;raw-vxhs-7003:  PASS (0.01 s)
 (008/166) 001;boochs-file-efcb:  PASS (0.01 s)
 ...
 (166/166) 002;luks-vxhs-e2cf:  PASS (0.02 s)
 ...

Rest assured that neither "001;cloop-sheepdog-XXXX" or
"002;cloop-sheepdog-XXXX" are in the resulting test list.

> "I didn't run this, but it's not a big deal, really."
> 

Again, IMO, SKIPs should be OK if they are intentional.  Otherwise,
there should be test ERRORs.  And, using the example given above, it'd
be possible to not even attempt to run tests that are *known* to not be
compatible with a given environment.

> If we start skipping tests because of missing libraries or compile
> options, some skips take on the meaning:
> 
> "We didn't run this, but that means you didn't test some stuff that you
> really ought to have."
> 
> I guess the key difference here is:
> 
> (1) The first skip mode doesn't change depending on what your
> environment is, it only changes based on what you ask it to run, but
> 
> (2) These skips are environmentally dependent, and should be visually
> identified as failures of the test runner to even attempt the test,
> which is semantically rather distinct from (1) above
> 

Again, I think what's missing is:

1) a layer above the test itself to determine what should be run
2) a better differentiation between a test SKIP/ERROR/FAIL.

> Clear as mud?
> 

A lot clearer... now I hand the same question back to you!

- Cleber.

> Great!
> 
> --js
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Jing Liu]

Hi Cleber Rosa,


On 2017/7/19 上午5:22, Cleber Rosa wrote:
>
> On 07/18/2017 02:07 PM, John Snow wrote:
>>
>> On 07/17/2017 11:01 PM, Jing Liu wrote:
>>
[...]
> I see issues here:
>
> 1) The qemu-iotests "runner", that is, "./check", understands that a
> file number is a test.  No further granularity is (currently?)
> achievable here.
>
> The easy solution would be to split tests which depend on optional
> features to separate test files.  In the 087 test given as example,
> the "=== aio=native without O_DIRECT ===" block would become, say, 190.
>
> 2) Skip tests based on features more easily.  There's already support
> for skipping tests on various situations.  From 087:
>
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> ...
>
> It's trivial to also have access to other compile time settings.  I did
> a quick experiment that would add a "_required_feature" function to
> "common.rc".
This seems a good way to solve this problem.
I failed for 087 just because there is no CONFIG_NETTLE_KDF.

I wonder how can common.rc detect if there's CONFIG_XXX in config-host.mak ?

Thanks.
Jing
>   For 087 (or 190?) it would look like:
>
> ...
> _supported_fmt qcow2
> _supported_proto file
> _supported_os Linux
> _required_feature CONFIG_LINUX_AIO
> ...
>
> Does it make sense?  Would a patch series along those lines would be
> welcomed by reviews?
>
> Thanks!
>

Re: [Qemu-devel] Question for iotests 188, 189 and 087

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2522 bytes --]



On 07/20/2017 04:37 AM, Jing Liu wrote:
> Hi Cleber Rosa,
> 
> 
> On 2017/7/19 上午5:22, Cleber Rosa wrote:
>>
>> On 07/18/2017 02:07 PM, John Snow wrote:
>>>
>>> On 07/17/2017 11:01 PM, Jing Liu wrote:
>>>
> [...]
>> I see issues here:
>>
>> 1) The qemu-iotests "runner", that is, "./check", understands that a
>> file number is a test.  No further granularity is (currently?)
>> achievable here.
>>
>> The easy solution would be to split tests which depend on optional
>> features to separate test files.  In the 087 test given as example,
>> the "=== aio=native without O_DIRECT ===" block would become, say, 190.
>>
>> 2) Skip tests based on features more easily.  There's already support
>> for skipping tests on various situations.  From 087:
>>
>> ...
>> _supported_fmt qcow2
>> _supported_proto file
>> _supported_os Linux
>> ...
>>
>> It's trivial to also have access to other compile time settings.  I did
>> a quick experiment that would add a "_required_feature" function to
>> "common.rc".
> This seems a good way to solve this problem.
> I failed for 087 just because there is no CONFIG_NETTLE_KDF.
> 

Good.  This actually raises a question about requirements.  I haven't
checked thoroughly, but it looks like either gnutls or nettle can be
used as crypto libraries.  I'll take a better look at that.

> I wonder how can common.rc detect if there's CONFIG_XXX in
> config-host.mak ?
> 

My experiment was based on a standalone tool that can be used like this:

$ ./scripts/buildconf.py CONFIG_NETTLE_KDF
False
$ echo $?
255

And similarly, but considering a specific target configuration:

$ ./scripts/buildconf.py CONFIG_MARVELL_88W8618 arm-softmmu
True
$ ./scripts/buildconf.py CONFIG_MARVELL_88W8618 x86_64-softmmu
False

And a 'common.rc' function that makes use of it.  I also have use cases
for doing something similar from Python code, so it should also be
possible to use it as a (Python) module.

- Cleber.

> Thanks.
> Jing
>>   For 087 (or 190?) it would look like:
>>
>> ...
>> _supported_fmt qcow2
>> _supported_proto file
>> _supported_os Linux
>> _required_feature CONFIG_LINUX_AIO
>> ...
>>
>> Does it make sense?  Would a patch series along those lines would be
>> welcomed by reviews?
>>
>> Thanks!
>>
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]