* [PATCH] package_manager: Run createrepo sequentially
@ 2015-03-30 9:00 Ed Bartosh
2015-03-30 9:11 ` Richard Purdie
0 siblings, 1 reply; 4+ messages in thread
From: Ed Bartosh @ 2015-03-30 9:00 UTC (permalink / raw)
To: openembedded-core
Running multiple createrepo in parallel can cause raice conditions
when creating/accessing rpm database. Createrepo fails with the
error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
in such cases.
Running createrepo sequentially should fix the race as rpm database
will not be accessed by multiple createrepo at the same time.
[YOCTO #6571]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/lib/oe/package_manager.py | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index c9a8084..c956116 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
archs = archs.union(set(sdk_pkg_archs))
rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
- index_cmds = []
rpm_dirs_found = False
for arch in archs:
arch_dir = os.path.join(self.deploy_dir, arch)
if not os.path.isdir(arch_dir):
continue
- index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
-
rpm_dirs_found = True
+ result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
+ if result:
+ bb.fatal(result)
+
if not rpm_dirs_found:
bb.note("There are no packages in %s" % self.deploy_dir)
return
- result = oe.utils.multiprocess_exec(index_cmds, create_index)
- if result:
- bb.fatal('%s' % ('\n'.join(result)))
-
class OpkgIndexer(Indexer):
def write_index(self):
--
2.1.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] package_manager: Run createrepo sequentially
2015-03-30 9:00 [PATCH] package_manager: Run createrepo sequentially Ed Bartosh
@ 2015-03-30 9:11 ` Richard Purdie
2015-03-30 9:17 ` Ed Bartosh
0 siblings, 1 reply; 4+ messages in thread
From: Richard Purdie @ 2015-03-30 9:11 UTC (permalink / raw)
To: Ed Bartosh; +Cc: openembedded-core
On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> Running multiple createrepo in parallel can cause raice conditions
> when creating/accessing rpm database. Createrepo fails with the
> error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> in such cases.
>
> Running createrepo sequentially should fix the race as rpm database
> will not be accessed by multiple createrepo at the same time.
>
> [YOCTO #6571]
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
> meta/lib/oe/package_manager.py | 11 ++++-------
> 1 file changed, 4 insertions(+), 7 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index c9a8084..c956116 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
> archs = archs.union(set(sdk_pkg_archs))
>
> rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> - index_cmds = []
> rpm_dirs_found = False
> for arch in archs:
> arch_dir = os.path.join(self.deploy_dir, arch)
> if not os.path.isdir(arch_dir):
> continue
>
> - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> -
> rpm_dirs_found = True
>
> + result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> + if result:
> + bb.fatal(result)
> +
> if not rpm_dirs_found:
> bb.note("There are no packages in %s" % self.deploy_dir)
> return
>
> - result = oe.utils.multiprocess_exec(index_cmds, create_index)
> - if result:
> - bb.fatal('%s' % ('\n'.join(result)))
> -
>
> class OpkgIndexer(Indexer):
> def write_index(self):
Running these in parallel gives us a significant speed improvement.
Could we instead point them at separate directories for the RPM database
(within WORKDIR to address the previous issue too)?
Cheers,
Richard
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] package_manager: Run createrepo sequentially
2015-03-30 9:11 ` Richard Purdie
@ 2015-03-30 9:17 ` Ed Bartosh
2015-03-30 9:20 ` Richard Purdie
0 siblings, 1 reply; 4+ messages in thread
From: Ed Bartosh @ 2015-03-30 9:17 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-core
On Mon, Mar 30, 2015 at 10:11:13AM +0100, Richard Purdie wrote:
> On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> > Running multiple createrepo in parallel can cause raice conditions
> > when creating/accessing rpm database. Createrepo fails with the
> > error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> > in such cases.
> >
> > Running createrepo sequentially should fix the race as rpm database
> > will not be accessed by multiple createrepo at the same time.
> >
> > [YOCTO #6571]
> >
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> > meta/lib/oe/package_manager.py | 11 ++++-------
> > 1 file changed, 4 insertions(+), 7 deletions(-)
> >
> > diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> > index c9a8084..c956116 100644
> > --- a/meta/lib/oe/package_manager.py
> > +++ b/meta/lib/oe/package_manager.py
> > @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
> > archs = archs.union(set(sdk_pkg_archs))
> >
> > rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> > - index_cmds = []
> > rpm_dirs_found = False
> > for arch in archs:
> > arch_dir = os.path.join(self.deploy_dir, arch)
> > if not os.path.isdir(arch_dir):
> > continue
> >
> > - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > -
> > rpm_dirs_found = True
> >
> > + result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > + if result:
> > + bb.fatal(result)
> > +
> > if not rpm_dirs_found:
> > bb.note("There are no packages in %s" % self.deploy_dir)
> > return
> >
> > - result = oe.utils.multiprocess_exec(index_cmds, create_index)
> > - if result:
> > - bb.fatal('%s' % ('\n'.join(result)))
> > -
> >
> > class OpkgIndexer(Indexer):
> > def write_index(self):
>
> Running these in parallel gives us a significant speed improvement.
> Could we instead point them at separate directories for the RPM database
> (within WORKDIR to address the previous issue too)?
>
I didn't see any noticeable performance gain at least for minimal image. I measured the time of
bitbake build before and after this change. It was practically the same.
Regards,
Ed
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] package_manager: Run createrepo sequentially
2015-03-30 9:17 ` Ed Bartosh
@ 2015-03-30 9:20 ` Richard Purdie
0 siblings, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2015-03-30 9:20 UTC (permalink / raw)
To: ed.bartosh; +Cc: openembedded-core
On Mon, 2015-03-30 at 12:17 +0300, Ed Bartosh wrote:
> On Mon, Mar 30, 2015 at 10:11:13AM +0100, Richard Purdie wrote:
> > On Mon, 2015-03-30 at 12:00 +0300, Ed Bartosh wrote:
> > > Running multiple createrepo in parallel can cause raice conditions
> > > when creating/accessing rpm database. Createrepo fails with the
> > > error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected"
> > > in such cases.
> > >
> > > Running createrepo sequentially should fix the race as rpm database
> > > will not be accessed by multiple createrepo at the same time.
> > >
> > > [YOCTO #6571]
> > >
> > > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > > ---
> > > meta/lib/oe/package_manager.py | 11 ++++-------
> > > 1 file changed, 4 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> > > index c9a8084..c956116 100644
> > > --- a/meta/lib/oe/package_manager.py
> > > +++ b/meta/lib/oe/package_manager.py
> > > @@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
> > > archs = archs.union(set(sdk_pkg_archs))
> > >
> > > rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
> > > - index_cmds = []
> > > rpm_dirs_found = False
> > > for arch in archs:
> > > arch_dir = os.path.join(self.deploy_dir, arch)
> > > if not os.path.isdir(arch_dir):
> > > continue
> > >
> > > - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > > -
> > > rpm_dirs_found = True
> > >
> > > + result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
> > > + if result:
> > > + bb.fatal(result)
> > > +
> > > if not rpm_dirs_found:
> > > bb.note("There are no packages in %s" % self.deploy_dir)
> > > return
> > >
> > > - result = oe.utils.multiprocess_exec(index_cmds, create_index)
> > > - if result:
> > > - bb.fatal('%s' % ('\n'.join(result)))
> > > -
> > >
> > > class OpkgIndexer(Indexer):
> > > def write_index(self):
> >
> > Running these in parallel gives us a significant speed improvement.
> > Could we instead point them at separate directories for the RPM database
> > (within WORKDIR to address the previous issue too)?
> >
> I didn't see any noticeable performance gain at least for minimal image. I measured the time of
> bitbake build before and after this change. It was practically the same.
For minimal, no, you wouldn't. For something like meta-toolchain-sdk
there are multiple large package feeds and it should show more of a
change.
The best benchmark for this is:
"time bitbake package-index"
having cleaned out the indexes from the package feeds by hand (having
built a large feed such as sdk images).
Cheers,
Richard
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-03-30 9:20 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-03-30 9:00 [PATCH] package_manager: Run createrepo sequentially Ed Bartosh
2015-03-30 9:11 ` Richard Purdie
2015-03-30 9:17 ` Ed Bartosh
2015-03-30 9:20 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.