[PATCH] python3: Improve logging capabilities for do_create_manifest

[PATCH] python3: Improve logging capabilities for do_create_manifest

[Alejandro Enedino Hernandez Samaniego]

Adds a couple of prints to get a nicer log, and creates a
small summary or report after checking every module, it
makes it more feasible for adoption, easier to debug why
a module ended at a certain package and see how the
manifest was created.

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 .../python/python3/create_manifest3.py             | 34 +++++++++++++++++-----
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-devtools/python/python3/create_manifest3.py b/meta/recipes-devtools/python/python3/create_manifest3.py
index 212ddd4..44f3454 100644
--- a/meta/recipes-devtools/python/python3/create_manifest3.py
+++ b/meta/recipes-devtools/python/python3/create_manifest3.py
@@ -80,7 +80,7 @@ with open('python3-manifest.json') as manifest:

 # First pass to get core-package functionality, because we base everything on the fact that core is actually working
 # Not exactly the same so it should not be a function
-print ('Getting dependencies for core package:')
+print ('Getting dependencies for package: core')

 # Special call to check for core package
 output = subprocess.check_output([sys.executable, 'get_module_deps3.py', 'python-core-package']).decode('utf8')
@@ -128,6 +128,7 @@ for value in old_manifest['core']['files']:
   # Each module will only import what is necessary for it to work in specific
   print ('Getting dependencies for module: %s' % value)
   output = subprocess.check_output([sys.executable, 'get_module_deps3.py', '%s' % value]).decode('utf8')
+  print ('The following dependencies were found for module %s:\n' % value)
   print (output)
   for item in output.split():
     # We append it so it doesnt hurt what we currently have:
@@ -170,7 +171,10 @@ for key in old_manifest:

     # Handle special cases, we assume that when they were manually added
     # to the manifest we knew what we were doing.
+    print('\n')
+    print('--------------------------')
     print ('Handling package %s' % key)
+    print('--------------------------')
     special_packages=['misc', 'modules', 'dev']
     if key in special_packages or 'staticdev' in key:
         print('Passing %s package directly' % key)
@@ -219,11 +223,16 @@ for key in old_manifest:

         # Launch separate task for each module for deterministic behavior
         # Each module will only import what is necessary for it to work in specific
-        print ('Getting dependencies for module: %s' % value)
+        print ('\nGetting dependencies for module: %s' % value)
         output = subprocess.check_output([sys.executable, 'get_module_deps3.py', '%s' % value]).decode('utf8')
         # We can print dependencies for debugging purposes
+        print ('The following dependencies were found for module %s:\n' % value)
         print (output)
         # Output will have all dependencies
+
+        reportFILES = []
+        reportRDEPS = []
+
         for item in output.split():

             # Warning: This first part is ugly
@@ -255,7 +264,7 @@ for key in old_manifest:
                             #print('Checking folder %s on package %s' % (item,keyfolder))
                             for file_folder in old_manifest[keyfolder]['files'] or file_folder in old_manifest[keyfolder]['cached']:
                                 if file_folder==folder:
-                                    print ('%s found in %s' % (folder, keyfolder))
+                                    print ('%s folder found in %s' % (folder, keyfolder))
                                     folderFound = True
                                     if keyfolder not in new_manifest[key]['rdepends'] and keyfolder != key:
                                         new_manifest[key]['rdepends'].append(keyfolder)
@@ -267,6 +276,7 @@ for key in old_manifest:
             if inFolders:
                 continue

+
             # We might already have it on the dictionary since it could depend on a (previously checked) module
             if item not in new_manifest[key]['files'] and item not in new_manifest[key]['cached']:
                 # Handle core as a special package, we already did it so we pass it to NEW data structure directly
@@ -287,6 +297,7 @@ for key in old_manifest:

                 else:

+
                     # Check if this dependency is already contained on another package, so we add it
                     # as an RDEPENDS, or if its not, it means it should be contained on the current
                     # package, so we should add it to FILES
@@ -298,19 +309,19 @@ for key in old_manifest:
                                 if(newkey!=key):
                                     if newkey not in new_manifest[key]['rdepends']:
                                        # Add it to the new manifest data struct
-                                       # Debug
-                                       print('Adding %s to %s RDEPENDS, because it contains %s' % (newkey, key, item))
+                                       reportRDEPS.append('Adding %s to %s RDEPENDS, because it contains %s\n' % (newkey, key, item))
                                        new_manifest[key]['rdepends'].append(newkey)
                                     break
                     else:
                       # A module shouldn't contain itself (${libdir}/python3/sqlite3 shouldnt be on sqlite3 files)
                       if os.path.basename(item) != key:
-                        print('Adding %s to %s FILES' % (item, key))
+                        reportFILES.append(('Adding %s to %s FILES\n' % (item, key)))
                         # Since it wasnt found on another package, its not an RDEP, so add it to FILES for this package
                         if isCached(item):
                             new_manifest[key]['cached'].append(item)
                         else:
                             new_manifest[key]['files'].append(item)
+
                         if item.endswith('*'):
                             wildcards.append(item)
                         if item not in allfiles:
@@ -318,6 +329,15 @@ for key in old_manifest:
                         else:
                             repeated.append(item)

+        print('\n')
+        print('#################################')
+        print('Summary for module %s' % value)
+        print('FILES found for module %s:' % value)
+        print(''.join(reportFILES))
+        print('RDEPENDS found for module %s:' % value)
+        print(''.join(reportRDEPS))
+        print('#################################')
+
 print ('The following files are repeated (contained in more than one package), please check which package should get it:')
 print (repeated)
 print('The following files contain wildcards, please check they are necessary')
@@ -334,4 +354,4 @@ for key in new_manifest:
 # Create the manifest from the data structure that was built
 with open('python3-manifest.json.new','w') as outfile:
     json.dump(new_manifest,outfile,sort_keys=True, indent=4)
-    outfile.write("\n")
+    outfile.write('\n')
--
2.7.4

This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

✗ patchtest: failure for python3: Improve logging capabilities for do_create_manifest

[Patchwork]

== Series Details ==

Series: python3: Improve logging capabilities for do_create_manifest
Revision: 1
URL   : https://patchwork.openembedded.org/series/11647/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 4cedddb836)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] python3: Improve logging capabilities for do_create_manifest

[Richard Purdie]

On Mon, 2018-04-02 at 13:08 -0700, Alejandro Enedino Hernandez
Samaniego wrote:
> Adds a couple of prints to get a nicer log, and creates a
> small summary or report after checking every module, it
> makes it more feasible for adoption, easier to debug why
> a module ended at a certain package and see how the
> manifest was created.
> 
> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx
> .com>

Can you rebase against master please as this clashes with one of your
other changes which already merged!

Cheers,

Richard

Re: [PATCH] python3: Improve logging capabilities for do_create_manifest

[Richard Purdie]

On Tue, 2018-04-03 at 11:04 +0100, Richard Purdie wrote:
> On Mon, 2018-04-02 at 13:08 -0700, Alejandro Enedino Hernandez
> Samaniego wrote:
> > 
> > Adds a couple of prints to get a nicer log, and creates a
> > small summary or report after checking every module, it
> > makes it more feasible for adoption, easier to debug why
> > a module ended at a certain package and see how the
> > manifest was created.
> > 
> > Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xili
> > nx
> > .com>
> Can you rebase against master please as this clashes with one of your
> other changes which already merged!

Actually, I have it applied now, thanks.

Richard

Re: [PATCH] python3: Improve logging capabilities for do_create_manifest

[Alejandro Enedino Hernandez Samaniego]

Hey Richard,

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: Tuesday, April 3, 2018 3:09 AM
To: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] python3: Improve logging capabilities for do_create_manifest

On Tue, 2018-04-03 at 11:04 +0100, Richard Purdie wrote:
> On Mon, 2018-04-02 at 13:08 -0700, Alejandro Enedino Hernandez 
> Samaniego wrote:
> > 
> > Adds a couple of prints to get a nicer log, and creates a small 
> > summary or report after checking every module, it makes it more 
> > feasible for adoption, easier to debug why a module ended at a 
> > certain package and see how the manifest was created.
> > 
> > Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xili 
> > nx .com>
> Can you rebase against master please as this clashes with one of your 
> other changes which already merged!

Actually, I have it applied now, thanks.

Richard

Thanks, I think my email server is to blame for this errors.