[PATCH] package_manager: Run createrepo sequentially

[PATCH] package_manager: Run createrepo sequentially

[Ed Bartosh]

Running multiple createrepo in parallel can cause raice conditions
when creating/accessing rpm database. Createrepo fails with the
error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
in such cases.

Running createrepo sequentially should fix the race as rpm database
will not be accessed by multiple createrepo at the same time.

[YOCTO #6571]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index c9a8084..c956116 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
         archs = archs.union(set(sdk_pkg_archs))
 
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
-        index_cmds = []
         rpm_dirs_found = False
         for arch in archs:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
-
             rpm_dirs_found = True
 
+            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            if result:
+                bb.fatal(result)
+
         if not rpm_dirs_found:
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
-        result = oe.utils.multiprocess_exec(index_cmds, create_index)
-        if result:
-            bb.fatal('%s' % ('\n'.join(result)))
-
 
 class OpkgIndexer(Indexer):
     def write_index(self):
-- 
2.1.4

Re: [PATCH] package_manager: Run createrepo sequentially

[Richard Purdie]

On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> Running multiple createrepo in parallel can cause raice conditions
> when creating/accessing rpm database. Createrepo fails with the
> error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> in such cases.
> 
> Running createrepo sequentially should fix the race as rpm database
> will not be accessed by multiple createrepo at the same time.
> 
> [YOCTO #6571]
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oe/package_manager.py | 11 ++++-------
>  1 file changed, 4 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index c9a8084..c956116 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
>          archs = archs.union(set(sdk_pkg_archs))
>  
>          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> -        index_cmds = []
>          rpm_dirs_found = False
>          for arch in archs:
>              arch_dir = os.path.join(self.deploy_dir, arch)
>              if not os.path.isdir(arch_dir):
>                  continue
>  
> -            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> -
>              rpm_dirs_found = True
>  
> +            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> +            if result:
> +                bb.fatal(result)
> +
>          if not rpm_dirs_found:
>              bb.note("There are no packages in %s" % self.deploy_dir)
>              return
>  
> -        result = oe.utils.multiprocess_exec(index_cmds, create_index)
> -        if result:
> -            bb.fatal('%s' % ('\n'.join(result)))
> -
>  
>  class OpkgIndexer(Indexer):
>      def write_index(self):

Running these in parallel gives us a significant speed improvement.
Could we instead point them at separate directories for the RPM database
(within WORKDIR to address the previous issue too)?

Cheers,

Richard

Re: [PATCH] package_manager: Run createrepo sequentially

[Ed Bartosh]

On Mon, Mar 30, 2015 at 10:11:13AM +0100, Richard Purdie wrote:
> On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> > Running multiple createrepo in parallel can cause raice conditions
> > when creating/accessing rpm database. Createrepo fails with the
> > error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> > in such cases.
> > 
> > Running createrepo sequentially should fix the race as rpm database
> > will not be accessed by multiple createrepo at the same time.
> > 
> > [YOCTO #6571]
> > 
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> >  meta/lib/oe/package_manager.py | 11 ++++-------
> >  1 file changed, 4 insertions(+), 7 deletions(-)
> > 
> > diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> > index c9a8084..c956116 100644
> > --- a/meta/lib/oe/package_manager.py
> > +++ b/meta/lib/oe/package_manager.py
> > @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
> >          archs = archs.union(set(sdk_pkg_archs))
> >  
> >          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> > -        index_cmds = []
> >          rpm_dirs_found = False
> >          for arch in archs:
> >              arch_dir = os.path.join(self.deploy_dir, arch)
> >              if not os.path.isdir(arch_dir):
> >                  continue
> >  
> > -            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > -
> >              rpm_dirs_found = True
> >  
> > +            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > +            if result:
> > +                bb.fatal(result)
> > +
> >          if not rpm_dirs_found:
> >              bb.note("There are no packages in %s" % self.deploy_dir)
> >              return
> >  
> > -        result = oe.utils.multiprocess_exec(index_cmds, create_index)
> > -        if result:
> > -            bb.fatal('%s' % ('\n'.join(result)))
> > -
> >  
> >  class OpkgIndexer(Indexer):
> >      def write_index(self):
> 
> Running these in parallel gives us a significant speed improvement.
> Could we instead point them at separate directories for the RPM database
> (within WORKDIR to address the previous issue too)?
> 
I didn't see any noticeable performance gain at least for minimal image. I measured the time of
bitbake build before and after this change. It was practically the same.

Regards,
Ed

Re: [PATCH] package_manager: Run createrepo sequentially

[Richard Purdie]

On Mon, 2015-03-30 at 12:17 +0300, Ed Bartosh wrote:
> On Mon, Mar 30, 2015 at 10:11:13AM +0100, Richard Purdie wrote:
> > On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> > > Running multiple createrepo in parallel can cause raice conditions
> > > when creating/accessing rpm database. Createrepo fails with the
> > > error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> > > in such cases.
> > > 
> > > Running createrepo sequentially should fix the race as rpm database
> > > will not be accessed by multiple createrepo at the same time.
> > > 
> > > [YOCTO #6571]
> > > 
> > > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > > ---
> > >  meta/lib/oe/package_manager.py | 11 ++++-------
> > >  1 file changed, 4 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> > > index c9a8084..c956116 100644
> > > --- a/meta/lib/oe/package_manager.py
> > > +++ b/meta/lib/oe/package_manager.py
> > > @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
> > >          archs = archs.union(set(sdk_pkg_archs))
> > >  
> > >          rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> > > -        index_cmds = []
> > >          rpm_dirs_found = False
> > >          for arch in archs:
> > >              arch_dir = os.path.join(self.deploy_dir, arch)
> > >              if not os.path.isdir(arch_dir):
> > >                  continue
> > >  
> > > -            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > > -
> > >              rpm_dirs_found = True
> > >  
> > > +            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > > +            if result:
> > > +                bb.fatal(result)
> > > +
> > >          if not rpm_dirs_found:
> > >              bb.note("There are no packages in %s" % self.deploy_dir)
> > >              return
> > >  
> > > -        result = oe.utils.multiprocess_exec(index_cmds, create_index)
> > > -        if result:
> > > -            bb.fatal('%s' % ('\n'.join(result)))
> > > -
> > >  
> > >  class OpkgIndexer(Indexer):
> > >      def write_index(self):
> > 
> > Running these in parallel gives us a significant speed improvement.
> > Could we instead point them at separate directories for the RPM database
> > (within WORKDIR to address the previous issue too)?
> > 
> I didn't see any noticeable performance gain at least for minimal image. I measured the time of
> bitbake build before and after this change. It was practically the same.

For minimal, no, you wouldn't. For something like meta-toolchain-sdk
there are multiple large package feeds and it should show more of a
change.

The best benchmark for this is:

"time bitbake package-index"

having cleaned out the indexes from the package feeds by hand (having
built a large feed such as sdk images).

Cheers,

Richard