[PATCH] Support kmeta directory usage with devtool modify/finish

[PATCH] Support kmeta directory usage with devtool modify/finish

[Jaewon Lee]

When using Kmeta directories, devtool finish will add every single file
in the directory to the bbappend. This is because in the current
implementation, the get_recipe_local_files function treats the kmeta
directory like a file. Modifying the function to loop through the
provided directories and return all included files instead of just the
top level directory. This will enable correct file to file comparison
when determing which files are new/changed and need to be added to the
bbappend.

Adding an extra check in devtool-source.bbclass to not copy the cfg file
if its already included somewhere in the kmeta directory

Also during 'modify', when moving necessary files in the kmeta directory
from the workdir to oe-local-files, the dangling parent directories are
left behind.  This in itself is not an issue as the temporary devtool
workspace is automatically deleted, but this causes an incorrect include
directory to be added in kernel-yocto.bbclass.  Changing the order of
the if statements to catch the correct conditional. This is safe to do
as when not in the devtool context, there will be no oe-local-files
directory.

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 meta/classes/devtool-source.bbclass | 6 ++++--
 meta/classes/kernel-yocto.bbclass   | 4 ++--
 meta/lib/oe/recipeutils.py          | 9 ++++++++-
 3 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 1372e32..a811000 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -103,8 +103,10 @@ python devtool_post_unpack() {
                 for l in sccfile:
                     line = l.split()
                     if line and line[0] in ('kconf', 'patch'):
-                        local_files[line[-1]] = os.path.join(os.path.dirname(local_files[key]), line[-1])
-                        shutil.copy2(os.path.join(os.path.dirname(local_files[key]), line[-1]), workdir)
+                        cfg = os.path.join(os.path.dirname(local_files[key]), line[-1])
+                        if not cfg in local_files.values():
+                            local_files[line[-1]] = cfg
+                            shutil.copy2(cfg, workdir)
                 sccfile.close()
 
     # Ignore local files with subdir={BP}
diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 496c8a7..2f556ca 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -138,10 +138,10 @@ do_kernel_metadata() {
 	for f in ${feat_dirs}; do
 		if [ -d "${WORKDIR}/$f/meta" ]; then
 			includes="$includes -I${WORKDIR}/$f/kernel-meta"
-	        elif [ -d "${WORKDIR}/$f" ]; then
-			includes="$includes -I${WORKDIR}/$f"
 		elif [ -d "${WORKDIR}/../oe-local-files/$f" ]; then
 			includes="$includes -I${WORKDIR}/../oe-local-files/$f"
+	        elif [ -d "${WORKDIR}/$f" ]; then
+			includes="$includes -I${WORKDIR}/$f"
 		fi
 	done
 	for s in ${sccs} ${patches}; do
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 9c99164..e63f7ae 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -480,7 +480,14 @@ def get_recipe_local_files(d, patches=False, archives=False):
                     unpack = fetch.ud[uri].parm.get('unpack', True)
                     if unpack:
                         continue
-            ret[fname] = localpath
+            if os.path.isdir(localpath):
+                for root, dirs, files in os.walk(localpath):
+                    for fname in files:
+                        fileabspath = os.path.join(root,fname)
+                        srcdir = os.path.dirname(localpath)
+                        ret[os.path.relpath(fileabspath,srcdir)] = fileabspath
+            else:
+                ret[fname] = localpath
     return ret
 
 
-- 
2.7.5

✗ patchtest: failure for Support kmeta directory usage with devtool modify/finish

[Patchwork]

== Series Details ==

Series: Support kmeta directory usage with devtool modify/finish
Revision: 1
URL   : https://patchwork.openembedded.org/series/15361/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            Support kmeta directory usage with devtool modify/finish
 Issue             Shortlog does not follow expected format [test_shortlog_format] 
  Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] Support kmeta directory usage with devtool modify/finish

[Jaewon Lee]

ping

-----Original Message-----
From: Jaewon Lee [mailto:jaewon.lee@xilinx.com] 
Sent: Friday, December 14, 2018 4:39 PM
To: openembedded-core@lists.openembedded.org
Cc: Jaewon Lee <JAEWON@xilinx.com>; Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Subject: [OE-core][PATCH] Support kmeta directory usage with devtool modify/finish

When using Kmeta directories, devtool finish will add every single file in the directory to the bbappend. This is because in the current implementation, the get_recipe_local_files function treats the kmeta directory like a file. Modifying the function to loop through the provided directories and return all included files instead of just the top level directory. This will enable correct file to file comparison when determing which files are new/changed and need to be added to the bbappend.

Adding an extra check in devtool-source.bbclass to not copy the cfg file if its already included somewhere in the kmeta directory

Also during 'modify', when moving necessary files in the kmeta directory from the workdir to oe-local-files, the dangling parent directories are left behind.  This in itself is not an issue as the temporary devtool workspace is automatically deleted, but this causes an incorrect include directory to be added in kernel-yocto.bbclass.  Changing the order of the if statements to catch the correct conditional. This is safe to do as when not in the devtool context, there will be no oe-local-files directory.

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 meta/classes/devtool-source.bbclass | 6 ++++--
 meta/classes/kernel-yocto.bbclass   | 4 ++--
 meta/lib/oe/recipeutils.py          | 9 ++++++++-
 3 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 1372e32..a811000 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -103,8 +103,10 @@ python devtool_post_unpack() {
                 for l in sccfile:
                     line = l.split()
                     if line and line[0] in ('kconf', 'patch'):
-                        local_files[line[-1]] = os.path.join(os.path.dirname(local_files[key]), line[-1])
-                        shutil.copy2(os.path.join(os.path.dirname(local_files[key]), line[-1]), workdir)
+                        cfg = os.path.join(os.path.dirname(local_files[key]), line[-1])
+                        if not cfg in local_files.values():
+                            local_files[line[-1]] = cfg
+                            shutil.copy2(cfg, workdir)
                 sccfile.close()
 
     # Ignore local files with subdir={BP} diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 496c8a7..2f556ca 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -138,10 +138,10 @@ do_kernel_metadata() {
 	for f in ${feat_dirs}; do
 		if [ -d "${WORKDIR}/$f/meta" ]; then
 			includes="$includes -I${WORKDIR}/$f/kernel-meta"
-	        elif [ -d "${WORKDIR}/$f" ]; then
-			includes="$includes -I${WORKDIR}/$f"
 		elif [ -d "${WORKDIR}/../oe-local-files/$f" ]; then
 			includes="$includes -I${WORKDIR}/../oe-local-files/$f"
+	        elif [ -d "${WORKDIR}/$f" ]; then
+			includes="$includes -I${WORKDIR}/$f"
 		fi
 	done
 	for s in ${sccs} ${patches}; do
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index 9c99164..e63f7ae 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -480,7 +480,14 @@ def get_recipe_local_files(d, patches=False, archives=False):
                     unpack = fetch.ud[uri].parm.get('unpack', True)
                     if unpack:
                         continue
-            ret[fname] = localpath
+            if os.path.isdir(localpath):
+                for root, dirs, files in os.walk(localpath):
+                    for fname in files:
+                        fileabspath = os.path.join(root,fname)
+                        srcdir = os.path.dirname(localpath)
+                        ret[os.path.relpath(fileabspath,srcdir)] = fileabspath
+            else:
+                ret[fname] = localpath
     return ret
 
 
--
2.7.5

Re: [PATCH] Support kmeta directory usage with devtool modify/finish

[Jaewon Lee]

ping

-----Original Message-----
From: Jaewon Lee 
Sent: Monday, January 28, 2019 3:18 PM
To: Jaewon Lee <JAEWON@xilinx.com>; openembedded-core@lists.openembedded.org
Cc: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Subject: RE: [OE-core][PATCH] Support kmeta directory usage with devtool modify/finish

ping

-----Original Message-----
From: Jaewon Lee [mailto:jaewon.lee@xilinx.com] 
Sent: Friday, December 14, 2018 4:39 PM
To: openembedded-core@lists.openembedded.org
Cc: Jaewon Lee <JAEWON@xilinx.com>; Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Subject: [OE-core][PATCH] Support kmeta directory usage with devtool modify/finish

When using Kmeta directories, devtool finish will add every single file in the directory to the bbappend. This is because in the current implementation, the get_recipe_local_files function treats the kmeta directory like a file. Modifying the function to loop through the provided directories and return all included files instead of just the top level directory. This will enable correct file to file comparison when determing which files are new/changed and need to be added to the bbappend.

Adding an extra check in devtool-source.bbclass to not copy the cfg file if its already included somewhere in the kmeta directory

Also during 'modify', when moving necessary files in the kmeta directory from the workdir to oe-local-files, the dangling parent directories are left behind.  This in itself is not an issue as the temporary devtool workspace is automatically deleted, but this causes an incorrect include directory to be added in kernel-yocto.bbclass.  Changing the order of the if statements to catch the correct conditional. This is safe to do as when not in the devtool context, there will be no oe-local-files directory.

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 meta/classes/devtool-source.bbclass | 6 ++++--
 meta/classes/kernel-yocto.bbclass   | 4 ++--
 meta/lib/oe/recipeutils.py          | 9 ++++++++-
 3 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 1372e32..a811000 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -103,8 +103,10 @@ python devtool_post_unpack() {
                 for l in sccfile:
                     line = l.split()
                     if line and line[0] in ('kconf', 'patch'):
-                        local_files[line[-1]] = os.path.join(os.path.dirname(local_files[key]), line[-1])
-                        shutil.copy2(os.path.join(os.path.dirname(local_files[key]), line[-1]), workdir)
+                        cfg = os.path.join(os.path.dirname(local_files[key]), line[-1])
+                        if not cfg in local_files.values():
+                            local_files[line[-1]] = cfg
+                            shutil.copy2(cfg, workdir)
                 sccfile.close()
 
     # Ignore local files with subdir={BP} diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 496c8a7..2f556ca 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -138,10 +138,10 @@ do_kernel_metadata() {
 	for f in ${feat_dirs}; do
 		if [ -d "${WORKDIR}/$f/meta" ]; then
 			includes="$includes -I${WORKDIR}/$f/kernel-meta"
-	        elif [ -d "${WORKDIR}/$f" ]; then
-			includes="$includes -I${WORKDIR}/$f"
 		elif [ -d "${WORKDIR}/../oe-local-files/$f" ]; then
 			includes="$includes -I${WORKDIR}/../oe-local-files/$f"
+	        elif [ -d "${WORKDIR}/$f" ]; then
+			includes="$includes -I${WORKDIR}/$f"
 		fi
 	done
 	for s in ${sccs} ${patches}; do
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index 9c99164..e63f7ae 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -480,7 +480,14 @@ def get_recipe_local_files(d, patches=False, archives=False):
                     unpack = fetch.ud[uri].parm.get('unpack', True)
                     if unpack:
                         continue
-            ret[fname] = localpath
+            if os.path.isdir(localpath):
+                for root, dirs, files in os.walk(localpath):
+                    for fname in files:
+                        fileabspath = os.path.join(root,fname)
+                        srcdir = os.path.dirname(localpath)
+                        ret[os.path.relpath(fileabspath,srcdir)] = fileabspath
+            else:
+                ret[fname] = localpath
     return ret
 
 
--
2.7.5

Re: [PATCH] Support kmeta directory usage with devtool modify/finish

[Paul Eggleton]

On Saturday, 15 December 2018 1:39:22 PM NZDT Jaewon Lee wrote:
> When using Kmeta directories, devtool finish will add every single file
> in the directory to the bbappend. This is because in the current
> implementation, the get_recipe_local_files function treats the kmeta
> directory like a file. Modifying the function to loop through the
> provided directories and return all included files instead of just the
> top level directory. This will enable correct file to file comparison
> when determing which files are new/changed and need to be added to the
> bbappend.
> 
> Adding an extra check in devtool-source.bbclass to not copy the cfg file
> if its already included somewhere in the kmeta directory
> 
> Also during 'modify', when moving necessary files in the kmeta directory
> from the workdir to oe-local-files, the dangling parent directories are
> left behind.  This in itself is not an issue as the temporary devtool
> workspace is automatically deleted, but this causes an incorrect include
> directory to be added in kernel-yocto.bbclass.  Changing the order of
> the if statements to catch the correct conditional. This is safe to do
> as when not in the devtool context, there will be no oe-local-files
> directory.

Looks good to me. My apologies for the extreme delay in reviewing.

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>


-- 

Paul Eggleton
Intel Open Source Technology Centre