Re: [Qemu-devel] [PATCH] iotests: Use configured python

[Max Reitz <mreitz@redhat.com>]

On 15.05.2014 19:35, Markus Armbruster wrote:
> Max Reitz <mreitz@redhat.com> writes:
>
>> On 15.05.2014 08:52, Markus Armbruster wrote:
>>> Max Reitz <mreitz@redhat.com> writes:
>>>
>>>> On 14.05.2014 14:33, Markus Armbruster wrote:
>>>>> Max Reitz <mreitz@redhat.com> writes:
>>>>>
>>>>>> Currently, QEMU's iotests rely on /usr/bin/env to start the correct
>>>>>> Python (that is, at least Python 2.4, but not 3). On systems where
>>>>>> Python 3 is the default, the user has no clean way of making the iotests
>>>>>> use the correct binary.
>>>>>>
>>>>>> This commit makes the iotests use the Python selected by configure.
>>>>>>
>>>>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>>>> I'm afraid this broke qemu-iotests in a separate build tree:
>>>>>
>>>>>        ./check: line 38: ./common.env: No such file or directory
>>>> As I already replied to Peter, I see two (or maybe three) ways to fix this:
>>>>
>>>> The first is, we use the correct path to common.env. This would
>>>> however result in modification of the source tree although this is
>>>> probably not what the user intends with an out-of-tree build. On the
>>>> other hand, this would just work.
>>> Writing to the source tree works only when the write is exactly the same
>>> for every build tree.  Example: autoconf writing configure.
>>>
>>> Modifying files under git-control is right out.
>>>
>>>> The second is, we do not create common.env for out-of-tree builds and
>>>> add a default common.env to the repository ("PYTHON = python" should
>>>> probably suffice). This would not introduce any regressions, however,
>>>> the iotests would remain problematic on systems with Python 3 being
>>>> the default when using out-of-tree builds.
>>> A difference between an in-tree and an out-of-tree build is a bug.
>>>
>>> If plain "python" is good enough for out-of-tree builds, it should be
>>> good enough for in-tree builds.
>>>
>>>>                                              As I guess that out-of-tree
>>>> builds are actually recommended,
>>> Correct.
>>>
>>>>                                    this would mean that the better
>>>> solution might be to revert my patch and instead change every "python"
>>>> occurrence in the iotests' Shebangs to "python2", as kind of a third
>>>> way to go. However, for this I'm not sure whether all systems which
>>>> are supposed to be supported by qemu actually have a "python2"
>>>> executable/symlink. I guess so, but then again...
>>> I don't know.  Try and find out :)
>> Okay, so there is a Python Enhancement Proposal, which suggests having
>> the symlink python2:
>>
>> http://legacy.python.org/dev/peps/pep-0394/
>>
>> However, at least Debian seems to ignore this (or at least some Debian
>> versions do).
> Fools :)
>
>> I think I'll go with Fam's proposal, which is making common.config
>> look for python itself, which then can be overwritten by an
>> environment variable.
> Namely "tell ./check the path to common.env with an env var, like how we
> tell ./check the path to qemu-img with QEMU_IMG_PROG".  I don't like how
> we tell check where to find the stuff we build.  But you can declare
> that a separate problem, and fix your "where is common.env" problem with
> the current method, as Fam suggests.
>
>>>> So, either we fix it but try to write to the source tree without the
>>>> user intending to modify it; or we fix it for in-tree builds only; or
>>>> we drop the magic and just use "python2" in the iotests' Shebangs.
>>> The problem is including generated bits, namely results of configure,
>>> into source files.
>>>
>>> The Autoconf way is to substitute placeholders in FOO.in producing FOO.
>>>
>>> When you want to limit .in contents as much as possible, you factor out
>>> the stuff that needs substitutions into some SUB.in, which you then
>>> include into FOO.  Requires a suitable include mechanism.  In shell,
>>> builtin source.
>>>
>>> But then you need to find SUB from FOO.  I've seen two ways used:
>>>
>>> 1. Assume SUB is in the current directory.  Link FOO into the build tree
>>> to make it so.
>>>
>>> 2. Require FOO to be run in a way that lets it find its source
>>> directory.  If whatever runs FOO puts the full path into argv[0], you
>>> can use that.  Else, require a suitable argument or environment
>>> variable.
>> Hm, doing this would probably be even more magic than my previous
>> patch – which already had too much magic for myself to handle and
>> therefore broke. I hope finding the correct python to use in
>> tests/qemu-iotests/common.config will work out without too much
>> hassle.
> It's low-level magic at most :)
>
> The root stupid idea is to run stuff in the source tree.  Since a source
> tree can have many build trees, finding the correct build tree can't be
> automated.
>
> If you run stuff in the build tree, there is exactly one source tree,
> and putting a pointer to it in the build tree is trivial.  In fact, we
> already have one: try "make -pn | grep ^SRC_PATH".
>
> I'd generate a suitable script into the build tree that sets up
> necessary variables, then sources $SRC_PATH/qemu-iotests/check.

And here I was, just not wanting to do "rm /usr/bin/python; ln -s 
python2 /usr/bin/python" anymore...

Okay, as this coincides mostly with what Peter says, I'll try to allow 
the iotests to be executed from the build tree (while hopefully keeping 
compatibility with the current situation).

Max