[PATCH 1/2] scripts/lib/compatlayer: detect_layers always use realpath's

[PATCH 1/2] scripts/lib/compatlayer: detect_layers always use realpath's

[Aníbal Limón]

If you are using relative paths and change to other folder for
execution it will fail, so use realpaths always.

[YOCTO #11164]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/lib/compatlayer/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 58f94b7..087ac14 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -112,6 +112,7 @@ def detect_layers(layer_directories, no_auto):
     layers = []
 
     for directory in layer_directories:
+        directory = os.path.realpath(directory)
         if directory[-1] == '/':
             directory = directory[0:-1]
 
-- 
2.1.4

[PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Aníbal Limón]

This script will be used to create it's own environment to make
runs of yocto-compat-layer.py againts layers isolated.

It clones the oe-core and bitbake repository in a temporary folder
and then calls yocto-compat-layer.py with specified LAYERS_DIRs to
test.

Example:

$ ./scripts/yocto-compat-layer-wrapper LAYER_DIR LAYER_DIR_N

[YOCTO #11164]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/yocto-compat-layer-wrapper | 68 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100755 scripts/yocto-compat-layer-wrapper

diff --git a/scripts/yocto-compat-layer-wrapper b/scripts/yocto-compat-layer-wrapper
new file mode 100755
index 0000000..6776f64
--- /dev/null
+++ b/scripts/yocto-compat-layer-wrapper
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+# Yocto Project compatibility layer tool wrapper
+#
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+oe_core_repo="git://git.openembedded.org/openembedded-core"
+bitbake_repo="git://git.openembedded.org/bitbake"
+
+show_help() {
+	printf "Usage: %s [-o output_log] [-h] LAYER_DIR ...\n" $0
+}
+
+output_log=''
+layer_dirs=""
+while getopts o:h arg
+do
+	case $arg in
+	o)
+		output_log=$(realpath "$OPTARG");;
+	h)
+		show_help
+		exit 0;;
+	?)
+		show_help
+		exit 2;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [[ -z $* ]]; then
+	show_help
+	exit 1
+else
+	for layer_dir in $(realpath $*); do
+		layer_dirs="$layer_dirs $layer_dir"
+	done
+fi
+
+env_dir=$(mktemp -d -t yocto-compat-XXXX)
+echo "The environment will be setup at $env_dir"
+echo ""
+
+echo "Cloning oe-core..."
+git clone $oe_core_repo $env_dir
+if [ $? -ne 0 ]; then
+	echo "Failed to clone oe-core repository"
+	exit 1
+fi
+
+echo "Cloning bitbake..."
+git clone $bitbake_repo $env_dir/bitbake
+if [ $? -ne 0 ]; then
+	echo "Failed to clone bitbake repository"
+	exit 1
+fi
+
+echo ""
+echo "Changing directory to $env_dir"
+cd $env_dir
+source oe-init-build-env
+if [[ -z $output_log ]]; then
+	echo "Running yocto-compat-layer.py $layer_dirs"
+	yocto-compat-layer.py $layer_dirs
+else
+	echo "Running yocto-compat-layer.py -o $output_log $layer_dirs"
+	yocto-compat-layer.py -o $output_log $layer_dirs
+fi
-- 
2.1.4

Re: [PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

Sending to oe-core ml, i used wrong oe-core mail address, :/,

	Anibal



On 03/30/2017 10:03 AM, Aníbal Limón wrote:
> 
> 
> On 03/30/2017 12:02 AM, Patrick Ohly wrote:
>> On Wed, 2017-03-29 at 15:44 -0600, Aníbal Limón wrote:
>> ...
>>> +show_help() {
>>> +	printf "Usage: %s [-o output_log] [-h] LAYER_DIR ...\n" $0
>>> +}
>>> +
>> ...
>>> +env_dir=$(mktemp -d -t yocto-compat-XXXX)
>>> +echo "The environment will be setup at $env_dir"
>>> +echo ""
>>
>> The directory gets created, but not removed.
> 
> I didn't remove the temp directory because may be the user wants to
> access the dir after the check.
> 
>>
>>> +echo "Cloning oe-core..."
>>> +git clone $oe_core_repo $env_dir
>>> +if [ $? -ne 0 ]; then
>>> +	echo "Failed to clone oe-core repository"
>>> +	exit 1
>>> +fi
>>> +
>>> +echo "Cloning bitbake..."
>>> +git clone $bitbake_repo $env_dir/bitbake
>>> +if [ $? -ne 0 ]; then
>>> +	echo "Failed to clone bitbake repository"
>>> +	exit 1
>>> +fi
>>
>> Cloning bitbake and OE-core each time the script runs will be fairly
>> slow. There's also a chicken-and-egg problem: if you don't have bitbake,
>> where's the script?
>>
>> I'd prefer to use an existing checkout of both, just as for the layers
>> which are to be tested.
> 
> I choose to clone the oe-core/bitbake to ensure there are a clean
> environment, without any previous layer added. I could add an option to
> specify a oe-core/bitbake dir if isn't set then clone.
> 
>>
>>> +cd $env_dir
>>> +source oe-init-build-env
>>> +if [[ -z $output_log ]]; then
>>> +	echo "Running yocto-compat-layer.py $layer_dirs"
>>> +	yocto-compat-layer.py $layer_dirs
>>> +else
>>> +	echo "Running yocto-compat-layer.py -o $output_log $layer_dirs"
>>> +	yocto-compat-layer.py -o $output_log $layer_dirs
>>> +fi
>>
>> It would be useful to accept and pass through all yocto-compat-layer.py
>> parameters, ideally without having to repeat a full list of them in the
>> argument parsing of the wrapper.
> 
> Agree,
> 
> 	Anibal
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Patrick Ohly]

On Thu, 2017-03-30 at 10:03 -0600, Aníbal Limón wrote:
> 
> On 03/30/2017 12:02 AM, Patrick Ohly wrote:
> > On Wed, 2017-03-29 at 15:44 -0600, Aníbal Limón wrote:
> > ...
> >> +show_help() {
> >> +	printf "Usage: %s [-o output_log] [-h] LAYER_DIR ...\n" $0
> >> +}
> >> +
> > ...
> >> +env_dir=$(mktemp -d -t yocto-compat-XXXX)
> >> +echo "The environment will be setup at $env_dir"
> >> +echo ""
> > 
> > The directory gets created, but not removed.
> 
> I didn't remove the temp directory because may be the user wants to
> access the dir after the check.

I think that this should be something that the user explicitly needs to
request. As it is now, accumulating directories in /tmp when invoking
the script is just unexpected and not the normal behavior of tools
creating something in /tmp.

> >> +echo "Cloning oe-core..."
> >> +git clone $oe_core_repo $env_dir
> >> +if [ $? -ne 0 ]; then
> >> +	echo "Failed to clone oe-core repository"
> >> +	exit 1
> >> +fi
> >> +
> >> +echo "Cloning bitbake..."
> >> +git clone $bitbake_repo $env_dir/bitbake
> >> +if [ $? -ne 0 ]; then
> >> +	echo "Failed to clone bitbake repository"
> >> +	exit 1
> >> +fi
> > 
> > Cloning bitbake and OE-core each time the script runs will be fairly
> > slow. There's also a chicken-and-egg problem: if you don't have bitbake,
> > where's the script?
> > 
> > I'd prefer to use an existing checkout of both, just as for the layers
> > which are to be tested.
> 
> I choose to clone the oe-core/bitbake to ensure there are a clean
> environment, without any previous layer added.

I don't quite get that argument. When using existing bitbake and OE-core
directories, how can they have layers added to them? I understand that
the existing checkout might contain additional modifications and/or
might not match current master, but I consider that a feature. As it
stands now, the caller of the script cannot test against Yocto 2.3 once
it is released, because the script will always check out master.

Control over that can of course also be added to the script, but I just
don't see the need for all this additional complexity. Just my 2 cents.

> I could add an option to
> specify a oe-core/bitbake dir if isn't set then clone.

At the very least please do that.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]



On 03/30/2017 10:13 AM, Patrick Ohly wrote:
> On Thu, 2017-03-30 at 10:03 -0600, Aníbal Limón wrote:
>>
>> On 03/30/2017 12:02 AM, Patrick Ohly wrote:
>>> On Wed, 2017-03-29 at 15:44 -0600, Aníbal Limón wrote:
>>> ...
>>>> +show_help() {
>>>> +	printf "Usage: %s [-o output_log] [-h] LAYER_DIR ...\n" $0
>>>> +}
>>>> +
>>> ...
>>>> +env_dir=$(mktemp -d -t yocto-compat-XXXX)
>>>> +echo "The environment will be setup at $env_dir"
>>>> +echo ""
>>>
>>> The directory gets created, but not removed.
>>
>> I didn't remove the temp directory because may be the user wants to
>> access the dir after the check.
> 
> I think that this should be something that the user explicitly needs to
> request. As it is now, accumulating directories in /tmp when invoking
> the script is just unexpected and not the normal behavior of tools
> creating something in /tmp.

Ok, i will add an option to don't delete and change the default behavior
to delete after check.


> 
>>>> +echo "Cloning oe-core..."
>>>> +git clone $oe_core_repo $env_dir
>>>> +if [ $? -ne 0 ]; then
>>>> +	echo "Failed to clone oe-core repository"
>>>> +	exit 1
>>>> +fi
>>>> +
>>>> +echo "Cloning bitbake..."
>>>> +git clone $bitbake_repo $env_dir/bitbake
>>>> +if [ $? -ne 0 ]; then
>>>> +	echo "Failed to clone bitbake repository"
>>>> +	exit 1
>>>> +fi
>>>
>>> Cloning bitbake and OE-core each time the script runs will be fairly
>>> slow. There's also a chicken-and-egg problem: if you don't have bitbake,
>>> where's the script?
>>>
>>> I'd prefer to use an existing checkout of both, just as for the layers
>>> which are to be tested.
>>
>> I choose to clone the oe-core/bitbake to ensure there are a clean
>> environment, without any previous layer added.
> 
> I don't quite get that argument. When using existing bitbake and OE-core
> directories, how can they have layers added to them? I understand that
> the existing checkout might contain additional modifications and/or
> might not match current master, but I consider that a feature. As it
> stands now, the caller of the script cannot test against Yocto 2.3 once
> it is released, because the script will always check out master.
> 
> Control over that can of course also be added to the script, but I just
> don't see the need for all this additional complexity. Just my 2 cents.

That's the original issue to create this script, if a user tries to
validate a layer and has modifications in configuration (bblayers,
local, auto), the check will be contaminated in some how.

If you want to run in some previous cloned repository you can use
directly the yocto-compat-layer.py without the wrapper.

> 
>> I could add an option to
>> specify a oe-core/bitbake dir if isn't set then clone.
> 
> At the very least please do that.

Yes i will do.

	Anibal
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Joshua Lock]

On Thu, 2017-03-30 at 10:43 -0600, Aníbal Limón wrote:
> 
> On 03/30/2017 10:13 AM, Patrick Ohly wrote:
> > On Thu, 2017-03-30 at 10:03 -0600, Aníbal Limón wrote:
> > > 
> > > On 03/30/2017 12:02 AM, Patrick Ohly wrote:
> > > > On Wed, 2017-03-29 at 15:44 -0600, Aníbal Limón wrote:
> > > > ...
> > > > > +show_help() {
> > > > > +	printf "Usage: %s [-o output_log] [-h] LAYER_DIR
> > > > > ...\n" $0
> > > > > +}
> > > > > +
> > > > 
> > > > ...
> > > > > +env_dir=$(mktemp -d -t yocto-compat-XXXX)
> > > > > +echo "The environment will be setup at $env_dir"
> > > > > +echo ""
> > > > 
> > > > The directory gets created, but not removed.
> > > 
> > > I didn't remove the temp directory because may be the user wants
> > > to
> > > access the dir after the check.
> > 
> > I think that this should be something that the user explicitly
> > needs to
> > request. As it is now, accumulating directories in /tmp when
> > invoking
> > the script is just unexpected and not the normal behavior of tools
> > creating something in /tmp.
> 
> Ok, i will add an option to don't delete and change the default
> behavior
> to delete after check.
> 
> 
> > 
> > > > > +echo "Cloning oe-core..."
> > > > > +git clone $oe_core_repo $env_dir
> > > > > +if [ $? -ne 0 ]; then
> > > > > +	echo "Failed to clone oe-core repository"
> > > > > +	exit 1
> > > > > +fi
> > > > > +
> > > > > +echo "Cloning bitbake..."
> > > > > +git clone $bitbake_repo $env_dir/bitbake
> > > > > +if [ $? -ne 0 ]; then
> > > > > +	echo "Failed to clone bitbake repository"
> > > > > +	exit 1
> > > > > +fi
> > > > 
> > > > Cloning bitbake and OE-core each time the script runs will be
> > > > fairly
> > > > slow. There's also a chicken-and-egg problem: if you don't have
> > > > bitbake,
> > > > where's the script?
> > > > 
> > > > I'd prefer to use an existing checkout of both, just as for the
> > > > layers
> > > > which are to be tested.
> > > 
> > > I choose to clone the oe-core/bitbake to ensure there are a clean
> > > environment, without any previous layer added.
> > 
> > I don't quite get that argument. When using existing bitbake and
> > OE-core
> > directories, how can they have layers added to them? I understand
> > that
> > the existing checkout might contain additional modifications and/or
> > might not match current master, but I consider that a feature. As
> > it
> > stands now, the caller of the script cannot test against Yocto 2.3
> > once
> > it is released, because the script will always check out master.
> > 
> > Control over that can of course also be added to the script, but I
> > just
> > don't see the need for all this additional complexity. Just my 2
> > cents.
> 
> That's the original issue to create this script, if a user tries to
> validate a layer and has modifications in configuration (bblayers,
> local, auto), the check will be contaminated in some how.

Those are all build-directory state, not items which are tracked in the
git repository.

Rather than cloning repositories to ensure clean state your script
could ensure a unique/new build directory by sourcing oe-init-build-env 
with a non-standard build directory, perhaps checking whether the
directory exists first and adding random characters until you get
something new?

Joshua

Re: [PATCH 2/2] scripts: Add yocto-compat-layer-wrapper

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 3508 bytes --]



On 03/30/2017 10:58 AM, Joshua Lock wrote:
> On Thu, 2017-03-30 at 10:43 -0600, Aníbal Limón wrote:
>>
>> On 03/30/2017 10:13 AM, Patrick Ohly wrote:
>>> On Thu, 2017-03-30 at 10:03 -0600, Aníbal Limón wrote:
>>>>
>>>> On 03/30/2017 12:02 AM, Patrick Ohly wrote:
>>>>> On Wed, 2017-03-29 at 15:44 -0600, Aníbal Limón wrote:
>>>>> ...
>>>>>> +show_help() {
>>>>>> +	printf "Usage: %s [-o output_log] [-h] LAYER_DIR
>>>>>> ...\n" $0
>>>>>> +}
>>>>>> +
>>>>>
>>>>> ...
>>>>>> +env_dir=$(mktemp -d -t yocto-compat-XXXX)
>>>>>> +echo "The environment will be setup at $env_dir"
>>>>>> +echo ""
>>>>>
>>>>> The directory gets created, but not removed.
>>>>
>>>> I didn't remove the temp directory because may be the user wants
>>>> to
>>>> access the dir after the check.
>>>
>>> I think that this should be something that the user explicitly
>>> needs to
>>> request. As it is now, accumulating directories in /tmp when
>>> invoking
>>> the script is just unexpected and not the normal behavior of tools
>>> creating something in /tmp.
>>
>> Ok, i will add an option to don't delete and change the default
>> behavior
>> to delete after check.
>>
>>
>>>
>>>>>> +echo "Cloning oe-core..."
>>>>>> +git clone $oe_core_repo $env_dir
>>>>>> +if [ $? -ne 0 ]; then
>>>>>> +	echo "Failed to clone oe-core repository"
>>>>>> +	exit 1
>>>>>> +fi
>>>>>> +
>>>>>> +echo "Cloning bitbake..."
>>>>>> +git clone $bitbake_repo $env_dir/bitbake
>>>>>> +if [ $? -ne 0 ]; then
>>>>>> +	echo "Failed to clone bitbake repository"
>>>>>> +	exit 1
>>>>>> +fi
>>>>>
>>>>> Cloning bitbake and OE-core each time the script runs will be
>>>>> fairly
>>>>> slow. There's also a chicken-and-egg problem: if you don't have
>>>>> bitbake,
>>>>> where's the script?
>>>>>
>>>>> I'd prefer to use an existing checkout of both, just as for the
>>>>> layers
>>>>> which are to be tested.
>>>>
>>>> I choose to clone the oe-core/bitbake to ensure there are a clean
>>>> environment, without any previous layer added.
>>>
>>> I don't quite get that argument. When using existing bitbake and
>>> OE-core
>>> directories, how can they have layers added to them? I understand
>>> that
>>> the existing checkout might contain additional modifications and/or
>>> might not match current master, but I consider that a feature. As
>>> it
>>> stands now, the caller of the script cannot test against Yocto 2.3
>>> once
>>> it is released, because the script will always check out master.
>>>
>>> Control over that can of course also be added to the script, but I
>>> just
>>> don't see the need for all this additional complexity. Just my 2
>>> cents.
>>
>> That's the original issue to create this script, if a user tries to
>> validate a layer and has modifications in configuration (bblayers,
>> local, auto), the check will be contaminated in some how.
> 
> Those are all build-directory state, not items which are tracked in the
> git repository.
> 
> Rather than cloning repositories to ensure clean state your script
> could ensure a unique/new build directory by sourcing oe-init-build-env 
> with a non-standard build directory, perhaps checking whether the
> directory exists first and adding random characters until you get
> something new?

That's a very good idea, i can generate tmp build directory and in this
way we can delete the logic for clone a new repository, so if anyone
reject, i'll do that.

	Anibal

> 
> Joshua
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]