[layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

[layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

[Robert Yang]

Hi Paul,

This email contains two parts:
- The first 2 are bug fixes
- The other 3 are performance improvements, I have 124 layers,
  the "update.py -b <branch>" needs about 9m20s to finish the running when everything is
  update, and we have several branches, we need them to run in periodically
  (e.g., per 30 mins, the short the better), so one branch 9m20s is really a problem.
  Only about 1m43s are needed with these improvements when set PARALLEL_JOBS to 10:
  update.py: fetch repos parallelly # Saved 2 mins
  update_layer.py: use it as a module # Saved 5 mins
  update_layer.py: only call init_parser when needed # Saved 1 min

// Robert

The following changes since commit fa0b01082fb0596959841e95af8c516ae3f267a7:

  admin: fix dependency display and search (2017-12-21 16:14:17 +1300)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/LI
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/LI

Robert Yang (5):
  utils.py: fix remove obsolete dependencies
  update.py: make sure oe-core is fetched
  update.py: fetch repos parallelly
  update_layer.py: use it as a module
  update_layer.py: only call init_parser when needed

 layerindex/update.py       | 176 +++++++++++++++++++++++----------------------
 layerindex/update_layer.py | 146 +++++++++++--------------------------
 layerindex/utils.py        |   6 +-
 settings.py                |   2 +
 4 files changed, 137 insertions(+), 193 deletions(-)

-- 
2.7.4

Re: [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

[Robert Yang]

Hi Paul,

Sorry, this is the cover letter, I should reply to this one just now.

// Robert

On 01/03/2018 01:42 PM, Robert Yang wrote:
> Hi Paul,
> 
> This email contains two parts:
> - The first 2 are bug fixes
> - The other 3 are performance improvements, I have 124 layers,
>    the "update.py -b <branch>" needs about 9m20s to finish the running when everything is
>    update, and we have several branches, we need them to run in periodically
>    (e.g., per 30 mins, the short the better), so one branch 9m20s is really a problem.
>    Only about 1m43s are needed with these improvements when set PARALLEL_JOBS to 10:
>    update.py: fetch repos parallelly # Saved 2 mins
>    update_layer.py: use it as a module # Saved 5 mins
>    update_layer.py: only call init_parser when needed # Saved 1 min
> 
> // Robert
> 
> The following changes since commit fa0b01082fb0596959841e95af8c516ae3f267a7:
> 
>    admin: fix dependency display and search (2017-12-21 16:14:17 +1300)
> 
> are available in the git repository at:
> 
>    git://git.pokylinux.org/poky-contrib rbt/LI
>    http://git.pokylinux.org/cgit.cgi//log/?h=rbt/LI
> 
> Robert Yang (5):
>    utils.py: fix remove obsolete dependencies
>    update.py: make sure oe-core is fetched
>    update.py: fetch repos parallelly
>    update_layer.py: use it as a module
>    update_layer.py: only call init_parser when needed
> 
>   layerindex/update.py       | 176 +++++++++++++++++++++++----------------------
>   layerindex/update_layer.py | 146 +++++++++++--------------------------
>   layerindex/utils.py        |   6 +-
>   settings.py                |   2 +
>   4 files changed, 137 insertions(+), 193 deletions(-)
> 

Re: [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

[Robert Yang]

Hi Paul,

Any comments, please ?

// Robert

On 01/03/2018 01:42 PM, Robert Yang wrote:
> Hi Paul,
> 
> This email contains two parts:
> - The first 2 are bug fixes
> - The other 3 are performance improvements, I have 124 layers,
>    the "update.py -b <branch>" needs about 9m20s to finish the running when everything is
>    update, and we have several branches, we need them to run in periodically
>    (e.g., per 30 mins, the short the better), so one branch 9m20s is really a problem.
>    Only about 1m43s are needed with these improvements when set PARALLEL_JOBS to 10:
>    update.py: fetch repos parallelly # Saved 2 mins
>    update_layer.py: use it as a module # Saved 5 mins
>    update_layer.py: only call init_parser when needed # Saved 1 min
> 
> // Robert
> 
> The following changes since commit fa0b01082fb0596959841e95af8c516ae3f267a7:
> 
>    admin: fix dependency display and search (2017-12-21 16:14:17 +1300)
> 
> are available in the git repository at:
> 
>    git://git.pokylinux.org/poky-contrib rbt/LI
>    http://git.pokylinux.org/cgit.cgi//log/?h=rbt/LI
> 
> Robert Yang (5):
>    utils.py: fix remove obsolete dependencies
>    update.py: make sure oe-core is fetched
>    update.py: fetch repos parallelly
>    update_layer.py: use it as a module
>    update_layer.py: only call init_parser when needed
> 
>   layerindex/update.py       | 176 +++++++++++++++++++++++----------------------
>   layerindex/update_layer.py | 146 +++++++++++--------------------------
>   layerindex/utils.py        |   6 +-
>   settings.py                |   2 +
>   4 files changed, 137 insertions(+), 193 deletions(-)
> 

Re: [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

[Paul Eggleton]

Hi Robert

On Thursday, 1 February 2018 5:59:42 PM NZDT Robert Yang wrote:
> On 01/03/2018 01:42 PM, Robert Yang wrote:
> > This email contains two parts:
> > - The first 2 are bug fixes
> > - The other 3 are performance improvements, I have 124 layers,
> >    the "update.py -b <branch>" needs about 9m20s to finish the running
> >    when everything is
> >    update, and we have several branches, we need them to run in
> >    periodically
> >    (e.g., per 30 mins, the short the better), so one branch 9m20s is
> >    really a problem.
> >    Only about 1m43s are needed with these improvements when set
> >    PARALLEL_JOBS to 10:
> >    update.py: fetch repos parallelly # Saved 2 mins
> >    update_layer.py: use it as a module # Saved 5 mins
> >    update_layer.py: only call init_parser when needed # Saved 1 min
>
> Any comments, please ?

My apologies for the delay - I had concerns about the update_layer.py
module stuff but hadn't made the time to review it properly until now.

Aside from that the other patches are fine so I've merged those (with
some minor tweaks to the commit messages).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre