Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

["Alex Bennée" <alex.bennee@linaro.org>]

Cleber Rosa <crosa@redhat.com> writes:

> On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 09/10/2018 06:18, Cleber Rosa wrote:
>>> This enables the execution of the acceptance tests on Travis.
>>
>> Did you test this? =)
>>
>
> I did have some jobs on Travis that looked promising.  But yeah, later I
> found some issues. :/
>
>>>
>>> Because the Travis environment is based on Ubuntu Trusty, it requires
>>> the python3-pip.
>>>
>>> Note: while another supposedely required component on newer versions
>>> (such as on Bionic) split the Python 3 installation further on the
>>> python3-venv package.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  .travis.yml | 6 ++++++
>>>  1 file changed, 6 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 95be6ec59f..db1a31ea51 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -36,6 +36,7 @@ addons:
>>>        - liburcu-dev
>>>        - libusb-1.0-0-dev
>>>        - libvte-2.90-dev
>>> +      - python3-pip
>>>        - sparse
>>>        - uuid-dev
>>>        - gcovr
>>> @@ -117,6 +118,11 @@ matrix:
>>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>>        python:
>>>          - "3.6"
>>> +    # Acceptance (Functional) tests
>>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>>> +           TEST_CMD="make check-acceptance"
>>> +      python:
>>> +        - "3.6"
>>>      # Using newer GCC with sanitizers
>>>      - addons:
>>>          apt:
>>>
>>
>> Using the following patch:
>>
>> -- >8 --
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..87e0c9a13f 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -118,4 +118,15 @@ matrix:
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +           # sudo rm /usr/local/bin/pip*
>> +      python:
>> +        - "3.6"
>> +      addons:
>> +        apt:
>> +          packages:
>> +            - python3-pip
>> +            - python3.4-venv
>
> This patch revealed to me that, even though we're asking Travis for a
> Python 3.6 stack, we're manually pointing to the primary Python (3.4)
> installation.  This is in itself a source of problems.
>
> Read on...
>
>>      # Using newer GCC with sanitizers
>>      - addons:
>> ---
>>
>> I got some improvements until:
>>
>>   VENV    /home/travis/build/philmd/qemu/tests/venv
>>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
>> Exception:
>> Traceback (most recent call last):
>>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
>> main
>>     status = self.run(options, args)
>>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
>> 283, in run
>>     requirement_set.install(install_options, global_options,
>> root=options.root_path)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>>     requirement.install(install_options, global_options, *args, **kwargs)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>>     self.move_wheel_files(self.source_dir, root=root)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
>> move_wheel_files
>>     pycompile=self.pycompile,
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
>> move_wheel_files
>>     clobber(source, lib_dir, True)
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>>     os.makedirs(destsubdir)
>>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>>     mkdir(name, mode)
>> PermissionError: [Errno 13] Permission denied:
>> '/usr/local/lib/python3.4/dist-packages/avocado'
>>
>
> This looks to me like it's caused by the use of the system wide Python
> 3.4 installation, as mentioned above.  That, and possibly the
> "--system-wide-packages" flag to the venv.
>
> So, in order to understand if, in those environments, we should let
> "configure" pick up the right Python, I did the following experiment:
>
>  $ python3 -m venv /tmp/py3-build
>  $ . /tmp/py3-build/bin/activate
>  $ which python
>  /tmp/py3-build/bin/python
>  $ /tmp/py3-build/bin/python --version
>  Python 3.6.6
>
> This shows that when a venv is activated, the "right" Python should be
> available in the $PATH.  So, building QEMU without pointint to a
> specific Python binary (when inside a venv) gives you:
>
>  $ mkdir -p /tmp/qemu-build
>  $ cd /tmp/qemu-build
>  $ ~/src/qemu/configure --target-list="x86_64-softmmu"
>  $ make print-PYTHON
>  PYTHON=python -B
>  $ which python
>  /tmp/py3-build/bin/python
>
> The whole question now is how/where Travis puts the requested Python
> version (python: - "3.6").  Is it on a venv?  Is it on a specific
> location? Is that reliable enough?
>
> I'm going to look for those answers now.

You can experiment with the Travis environment on your own system using:

  make docker-test-build@travis DEBUG=1

>
> - Cleber.
>
>> See: https://travis-ci.org/philmd/qemu/jobs/439138706
>>


--
Alex Bennée