Openembedded Core Discussions
 help / color / mirror / Atom feed
* opkg-make-index cache failing
@ 2011-11-25 15:38 Koen Kooi
  2011-11-25 16:53 ` Richard Purdie
  0 siblings, 1 reply; 7+ messages in thread
From: Koen Kooi @ 2011-11-25 15:38 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

Hi,

In OE-classic the opkg-make-index cache was working pretty well, do_rootfs only spent a few seconds doing opkg-make-index on incremental builds. In the OE-core world the situation is different, opkg-make-index will reindex every package on each do_rootfs run, so after a while building images starts taking *really* long, on my systemd (work/ on ssd, deploy on rotating media) it now takes ±7minutes just do refresh the opkg indices.

Since I can't blame sstate for this I suspect that pseudo gets in the way of the naive caching logic. Is there a simple way to turn off pseudo when running opkg-make-index?

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2011-11-25 15:38 opkg-make-index cache failing Koen Kooi
@ 2011-11-25 16:53 ` Richard Purdie
  2011-11-25 18:15   ` Koen Kooi
  0 siblings, 1 reply; 7+ messages in thread
From: Richard Purdie @ 2011-11-25 16:53 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
> In OE-classic the opkg-make-index cache was working pretty well,
> do_rootfs only spent a few seconds doing opkg-make-index on
> incremental builds. In the OE-core world the situation is different,
> opkg-make-index will reindex every package on each do_rootfs run, so
> after a while building images starts taking *really* long, on my
> systemd (work/ on ssd, deploy on rotating media) it now takes
> ±7minutes just do refresh the opkg indices.
> 
> Since I can't blame sstate for this I suspect that pseudo gets in the
> way of the naive caching logic. Is there a simple way to turn off
> pseudo when running opkg-make-index?

You could try:

PSEUDO_DISABLED=1 opkg-make-index

or even

PSEUDO_UNLOAD=1 opkg-make-index

although the latter is a sledge hammer to crack a nut approach...

Cheers,

Richard




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2011-11-25 16:53 ` Richard Purdie
@ 2011-11-25 18:15   ` Koen Kooi
  2011-11-25 19:59     ` Koen Kooi
  0 siblings, 1 reply; 7+ messages in thread
From: Koen Kooi @ 2011-11-25 18:15 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]


Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:

> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
>> In OE-classic the opkg-make-index cache was working pretty well,
>> do_rootfs only spent a few seconds doing opkg-make-index on
>> incremental builds. In the OE-core world the situation is different,
>> opkg-make-index will reindex every package on each do_rootfs run, so
>> after a while building images starts taking *really* long, on my
>> systemd (work/ on ssd, deploy on rotating media) it now takes
>> ±7minutes just do refresh the opkg indices.
>> 
>> Since I can't blame sstate for this I suspect that pseudo gets in the
>> way of the naive caching logic. Is there a simple way to turn off
>> pseudo when running opkg-make-index?

With a relatively small deploy:

real	6m14.611s
user	2m45.909s
sys	0m43.674s

> You could try:
> 
> PSEUDO_DISABLED=1 opkg-make-index

real	5m25.920s
user	2m41.279s
sys	0m41.681s

> or even
> 
> PSEUDO_UNLOAD=1 opkg-make-index

real	5m5.699s
user	2m37.283s
sys	0m37.508s

Now I need to double check to see if  the patches are really having effect on the opkg caching instead of just not having pseudo overhead.

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2011-11-25 18:15   ` Koen Kooi
@ 2011-11-25 19:59     ` Koen Kooi
  2011-11-29 11:40       ` Martin Jansa
  0 siblings, 1 reply; 7+ messages in thread
From: Koen Kooi @ 2011-11-25 19:59 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]


Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven:

> 
> Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:
> 
>> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
>>> In OE-classic the opkg-make-index cache was working pretty well,
>>> do_rootfs only spent a few seconds doing opkg-make-index on
>>> incremental builds. In the OE-core world the situation is different,
>>> opkg-make-index will reindex every package on each do_rootfs run, so
>>> after a while building images starts taking *really* long, on my
>>> systemd (work/ on ssd, deploy on rotating media) it now takes
>>> ±7minutes just do refresh the opkg indices.
>>> 
>>> Since I can't blame sstate for this I suspect that pseudo gets in the
>>> way of the naive caching logic. Is there a simple way to turn off
>>> pseudo when running opkg-make-index?
> 
> With a relatively small deploy:
> 
> real	6m14.611s
> user	2m45.909s
> sys	0m43.674s
> 
>> You could try:
>> 
>> PSEUDO_DISABLED=1 opkg-make-index
> 
> real	5m25.920s
> user	2m41.279s
> sys	0m41.681s
> 
>> or even
>> 
>> PSEUDO_UNLOAD=1 opkg-make-index
> 
> real	5m5.699s
> user	2m37.283s
> sys	0m37.508s
> 
> Now I need to double check to see if  the patches are really having effect on the opkg caching instead of just not having pseudo overhead.

Retesting without PSEUDO changes:

real    5m18.412s
user    2m40.736s
sys     0m41.374s

So the PSEUDO changes don't seem to do much :(

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2011-11-25 19:59     ` Koen Kooi
@ 2011-11-29 11:40       ` Martin Jansa
  2012-03-29  7:58         ` Martin Jansa
  0 siblings, 1 reply; 7+ messages in thread
From: Martin Jansa @ 2011-11-29 11:40 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 6641 bytes --]

On Fri, Nov 25, 2011 at 08:59:06PM +0100, Koen Kooi wrote:
> 
> Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven:
> 
> > 
> > Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:
> > 
> >> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
> >>> In OE-classic the opkg-make-index cache was working pretty well,
> >>> do_rootfs only spent a few seconds doing opkg-make-index on
> >>> incremental builds. In the OE-core world the situation is different,
> >>> opkg-make-index will reindex every package on each do_rootfs run, so
> >>> after a while building images starts taking *really* long, on my
> >>> systemd (work/ on ssd, deploy on rotating media) it now takes
> >>> ±7minutes just do refresh the opkg indices.
> >>> 
> >>> Since I can't blame sstate for this I suspect that pseudo gets in the
> >>> way of the naive caching logic. Is there a simple way to turn off
> >>> pseudo when running opkg-make-index?
> > 
> > With a relatively small deploy:
> > 
> > real	6m14.611s
> > user	2m45.909s
> > sys	0m43.674s
> > 
> >> You could try:
> >> 
> >> PSEUDO_DISABLED=1 opkg-make-index
> > 
> > real	5m25.920s
> > user	2m41.279s
> > sys	0m41.681s
> > 
> >> or even
> >> 
> >> PSEUDO_UNLOAD=1 opkg-make-index
> > 
> > real	5m5.699s
> > user	2m37.283s
> > sys	0m37.508s
> > 
> > Now I need to double check to see if  the patches are really having effect on the opkg caching instead of just not having pseudo overhead.
> 
> Retesting without PSEUDO changes:
> 
> real    5m18.412s
> user    2m40.736s
> sys     0m41.374s
> 
> So the PSEUDO changes don't seem to do much :(

I can confirm this change here too:
OE om-gta02@shr ~/shr-core $ time /usr/bin/python2.7 \
/OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-core/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-core/tmp/deploy/ipk/armv4t/


real    11m47.949s
user    4m13.848s
sys     0m17.917s

It hurts even more when someone is building ie for spitz and
linux-kexecboot depends on initramfs image which calls another 
opkg-make-index with every image build.

If I replace opkg-make-index from oe-core with old oe-classic version, I
get a lot of messages like:
"Lost field license, MIT-X"

but the time is similar
real    10m23.953s
user    4m10.940s
sys     0m15.833s

And if I do the same completely in shr-unstable (based on OE-classic)
OE om-gta02@shr ~/shr-unstable $ time /usr/bin/python2.7 \
/OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-unstable/tmp/deploy/ipk/armv4t/


real    0m6.469s
user    0m6.056s
sys     0m0.400s

Another difference seems to be in Packages.filelist which were empty in
OE-classic:

OE om-gta02@shr ~/shr-core $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
1173 tmp/deploy/ipk/palmpre2/Packages.filelist
1254 tmp/deploy/ipk/palmpre/Packages.filelist
107346 tmp/deploy/ipk/armv4t/Packages.filelist
110557 tmp/deploy/ipk/armv7a-vfp-neon/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
1365 tmp/deploy/ipk/nokia900/Packages.filelist
812 tmp/deploy/ipk/all/Packages.filelist
1643 tmp/deploy/ipk/om_gta02/Packages.filelist
OE om-gta02@shr ~/shr-core $ cd ../shr-unstable/
OE om-gta02@shr ~/shr-unstable $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
0 tmp/deploy/ipk/armv7a/Packages.filelist
0 tmp/deploy/ipk/om-gta01/Packages.filelist
0 tmp/deploy/ipk/om-gta02/Packages.filelist
0 tmp/deploy/ipk/palmpre2/Packages.filelist
0 tmp/deploy/ipk/armv6/Packages.filelist
0 tmp/deploy/ipk/palmpre/Packages.filelist
0 tmp/deploy/ipk/armv4t/Packages.filelist
0 tmp/deploy/ipk/armv4/Packages.filelist
0 tmp/deploy/ipk/armv6-novfp/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
0 tmp/deploy/ipk/htcdream/Packages.filelist
0 tmp/deploy/ipk/nokia900/Packages.filelist
0 tmp/deploy/ipk/all/Packages.filelist
0 tmp/deploy/ipk/armv7/Packages.filelist

But that's not the cause, because even without Packages.filelist
generation it takes about 3 min (better but it still writes info for
every package).

Weird is that there is only small diff between new and old opkg.py/opkg-make-index
--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg.py  2011-05-27 18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg.py      2011-09-01 16:36:35.000000000 +0200
@@ -145,6 +145,7 @@
         self.priority = None
         self.tags = None
         self.fn = fn
+        self.license = None

         if fn:
             # see if it is deb format
@@ -319,6 +320,12 @@
     def get_section(self, section):
         return self.section

+    def set_license(self, license):
+        self.license = license
+
+    def get_license(self, license):
+        return self.license
+
     def get_file_list(self):
         if not self.fn:
             return []
@@ -425,6 +432,7 @@
         if self.description: out = out + "Description: %s\n" % (self.description)
         if self.oe: out = out + "OE: %s\n" % (self.oe)
         if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
+        if self.license: out = out + "License: %s\n" % (self.license)
         if self.priority: out = out + "Priority: %s\n" % (self.priority)
         if self.tags: out = out + "Tags: %s\n" % (self.tags)
         out = out + "\n"
(reverse-i-search)`uNr': diff -^Cr ~/common/conf/shr-core/ conf/

--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index  2011-05-27 18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index      2011-11-29 12:32:39.000000000 +0100
@@ -197,7 +197,15 @@
 names = packages.packages.keys()
 names.sort()
 for name in names:
-     for fn in packages[name].get_file_list():
+     try:
+          fnlist = packages[name].get_file_list()
+     except OSError, e:
+          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          continue
+     except IOError, e:
+          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          continue
+     for fn in fnlist:
           (h,t) = os.path.split(fn)
           if not t: continue
           if not files.has_key(t): files[t] = name+':'+fn


Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2011-11-29 11:40       ` Martin Jansa
@ 2012-03-29  7:58         ` Martin Jansa
  2012-03-29  8:59           ` Richard Purdie
  0 siblings, 1 reply; 7+ messages in thread
From: Martin Jansa @ 2012-03-29  7:58 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 8268 bytes --]

On Tue, Nov 29, 2011 at 12:40:52PM +0100, Martin Jansa wrote:
> On Fri, Nov 25, 2011 at 08:59:06PM +0100, Koen Kooi wrote:
> > 
> > Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven:
> > 
> > > 
> > > Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:
> > > 
> > >> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
> > >>> In OE-classic the opkg-make-index cache was working pretty well,
> > >>> do_rootfs only spent a few seconds doing opkg-make-index on
> > >>> incremental builds. In the OE-core world the situation is different,
> > >>> opkg-make-index will reindex every package on each do_rootfs run, so
> > >>> after a while building images starts taking *really* long, on my
> > >>> systemd (work/ on ssd, deploy on rotating media) it now takes
> > >>> ±7minutes just do refresh the opkg indices.
> > >>> 
> > >>> Since I can't blame sstate for this I suspect that pseudo gets in the
> > >>> way of the naive caching logic. Is there a simple way to turn off
> > >>> pseudo when running opkg-make-index?
> > > 
> > > With a relatively small deploy:
> > > 
> > > real	6m14.611s
> > > user	2m45.909s
> > > sys	0m43.674s
> > > 
> > >> You could try:
> > >> 
> > >> PSEUDO_DISABLED=1 opkg-make-index
> > > 
> > > real	5m25.920s
> > > user	2m41.279s
> > > sys	0m41.681s
> > > 
> > >> or even
> > >> 
> > >> PSEUDO_UNLOAD=1 opkg-make-index
> > > 
> > > real	5m5.699s
> > > user	2m37.283s
> > > sys	0m37.508s
> > > 
> > > Now I need to double check to see if  the patches are really having effect on the opkg caching instead of just not having pseudo overhead.
> > 
> > Retesting without PSEUDO changes:
> > 
> > real    5m18.412s
> > user    2m40.736s
> > sys     0m41.374s
> > 
> > So the PSEUDO changes don't seem to do much :(
> 
> I can confirm this change here too:
> OE om-gta02@shr ~/shr-core $ time /usr/bin/python2.7 \
> /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
> -r /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
> -p /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
> -l /OE/shr-core/tmp/deploy/ipk/armv4t/Packages.filelist \
> -m /OE/shr-core/tmp/deploy/ipk/armv4t/
> 
> 
> real    11m47.949s
> user    4m13.848s
> sys     0m17.917s
> 
> It hurts even more when someone is building ie for spitz and
> linux-kexecboot depends on initramfs image which calls another 
> opkg-make-index with every image build.
> 
> If I replace opkg-make-index from oe-core with old oe-classic version, I
> get a lot of messages like:
> "Lost field license, MIT-X"
> 
> but the time is similar
> real    10m23.953s
> user    4m10.940s
> sys     0m15.833s
> 
> And if I do the same completely in shr-unstable (based on OE-classic)
> OE om-gta02@shr ~/shr-unstable $ time /usr/bin/python2.7 \
> /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
> -r /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
> -p /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
> -l /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages.filelist \
> -m /OE/shr-unstable/tmp/deploy/ipk/armv4t/
> 
> 
> real    0m6.469s
> user    0m6.056s
> sys     0m0.400s
> 
> Another difference seems to be in Packages.filelist which were empty in
> OE-classic:
> 
> OE om-gta02@shr ~/shr-core $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
> 1173 tmp/deploy/ipk/palmpre2/Packages.filelist
> 1254 tmp/deploy/ipk/palmpre/Packages.filelist
> 107346 tmp/deploy/ipk/armv4t/Packages.filelist
> 110557 tmp/deploy/ipk/armv7a-vfp-neon/Packages.filelist
> 0 tmp/deploy/ipk/Packages.filelist
> 1365 tmp/deploy/ipk/nokia900/Packages.filelist
> 812 tmp/deploy/ipk/all/Packages.filelist
> 1643 tmp/deploy/ipk/om_gta02/Packages.filelist
> OE om-gta02@shr ~/shr-core $ cd ../shr-unstable/
> OE om-gta02@shr ~/shr-unstable $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
> 0 tmp/deploy/ipk/armv7a/Packages.filelist
> 0 tmp/deploy/ipk/om-gta01/Packages.filelist
> 0 tmp/deploy/ipk/om-gta02/Packages.filelist
> 0 tmp/deploy/ipk/palmpre2/Packages.filelist
> 0 tmp/deploy/ipk/armv6/Packages.filelist
> 0 tmp/deploy/ipk/palmpre/Packages.filelist
> 0 tmp/deploy/ipk/armv4t/Packages.filelist
> 0 tmp/deploy/ipk/armv4/Packages.filelist
> 0 tmp/deploy/ipk/armv6-novfp/Packages.filelist
> 0 tmp/deploy/ipk/Packages.filelist
> 0 tmp/deploy/ipk/htcdream/Packages.filelist
> 0 tmp/deploy/ipk/nokia900/Packages.filelist
> 0 tmp/deploy/ipk/all/Packages.filelist
> 0 tmp/deploy/ipk/armv7/Packages.filelist
> 
> But that's not the cause, because even without Packages.filelist
> generation it takes about 3 min (better but it still writes info for
> every package).
> 
> Weird is that there is only small diff between new and old opkg.py/opkg-make-index
> --- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg.py  2011-05-27 18:35:03.000000000 +0200
> +++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg.py      2011-09-01 16:36:35.000000000 +0200
> @@ -145,6 +145,7 @@
>          self.priority = None
>          self.tags = None
>          self.fn = fn
> +        self.license = None
> 
>          if fn:
>              # see if it is deb format
> @@ -319,6 +320,12 @@
>      def get_section(self, section):
>          return self.section
> 
> +    def set_license(self, license):
> +        self.license = license
> +
> +    def get_license(self, license):
> +        return self.license
> +
>      def get_file_list(self):
>          if not self.fn:
>              return []
> @@ -425,6 +432,7 @@
>          if self.description: out = out + "Description: %s\n" % (self.description)
>          if self.oe: out = out + "OE: %s\n" % (self.oe)
>          if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
> +        if self.license: out = out + "License: %s\n" % (self.license)
>          if self.priority: out = out + "Priority: %s\n" % (self.priority)
>          if self.tags: out = out + "Tags: %s\n" % (self.tags)
>          out = out + "\n"
> (reverse-i-search)`uNr': diff -^Cr ~/common/conf/shr-core/ conf/
> 
> --- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index  2011-05-27 18:35:03.000000000 +0200
> +++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index      2011-11-29 12:32:39.000000000 +0100
> @@ -197,7 +197,15 @@
>  names = packages.packages.keys()
>  names.sort()
>  for name in names:
> -     for fn in packages[name].get_file_list():
> +     try:
> +          fnlist = packages[name].get_file_list()
> +     except OSError, e:
> +          sys.stderr.write("Package %s disappeared on us!\n" % (name))
> +          continue
> +     except IOError, e:
> +          sys.stderr.write("Package %s disappeared on us!\n" % (name))
> +          continue
> +     for fn in fnlist:
>            (h,t) = os.path.split(fn)
>            if not t: continue
>            if not files.has_key(t): files[t] = name+':'+fn
> 
> 
> Regards,
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Today I did some more tests and started using simple bash script to
reindex all Packages files just once (instead of rebuilding common arch
many times with bitbake package-index).

OPKG_MAKE_INDEX=~/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/opkg-make-index
DEPLOY_IPK=~/shr-core/tmp-eglibc/deploy/ipk/
for PACKAGE_FILE in `find ${DEPLOY_IPK} -name Packages`; do
  PACKAGE_DIR=`dirname ${PACKAGE_FILE}`
  touch ${PACKAGE_FILE}
  echo "Regenerating ${PACKAGE_FILE}"
  time flock ${PACKAGE_FILE}.flock -c "${OPKG_MAKE_INDEX} -r ${PACKAGE_FILE} -p ${PACKAGE_FILE} -l ${PACKAGE_FILE}.filelist -m ${PACKAGE_DIR}/"
done

And on my machine it takes really long to finish
real    26m40.108s
user    23m6.673s
sys     0m14.464s

So I'll look into opkg-make-index changes again and to make it easier I would like 
to move all oe-core local patches:
http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/opkg-utils/opkg-utils
to yocto git repo:
http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/
should I send pull-request for opkg-utils repo here or somewhere else? e.g. yocto ML?

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: opkg-make-index cache failing
  2012-03-29  7:58         ` Martin Jansa
@ 2012-03-29  8:59           ` Richard Purdie
  0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2012-03-29  8:59 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

On Thu, 2012-03-29 at 09:58 +0200, Martin Jansa wrote:
> Today I did some more tests and started using simple bash script to
> reindex all Packages files just once (instead of rebuilding common arch
> many times with bitbake package-index).
> 
> OPKG_MAKE_INDEX=~/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/opkg-make-index
> DEPLOY_IPK=~/shr-core/tmp-eglibc/deploy/ipk/
> for PACKAGE_FILE in `find ${DEPLOY_IPK} -name Packages`; do
>   PACKAGE_DIR=`dirname ${PACKAGE_FILE}`
>   touch ${PACKAGE_FILE}
>   echo "Regenerating ${PACKAGE_FILE}"
>   time flock ${PACKAGE_FILE}.flock -c "${OPKG_MAKE_INDEX} -r ${PACKAGE_FILE} -p ${PACKAGE_FILE} -l ${PACKAGE_FILE}.filelist -m ${PACKAGE_DIR}/"
> done
> 
> And on my machine it takes really long to finish
> real    26m40.108s
> user    23m6.673s
> sys     0m14.464s
> 
> So I'll look into opkg-make-index changes again and to make it easier I would like 
> to move all oe-core local patches:
> http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/opkg-utils/opkg-utils
> to yocto git repo:
> http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/
> should I send pull-request for opkg-utils repo here or somewhere else? e.g. yocto ML?

I've been advising people to do that kind of thing on the Yocto mailing
list since there is already lots of patch traffic on OE-Core...

Cheers,

Richard




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2012-03-29  9:08 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-11-25 15:38 opkg-make-index cache failing Koen Kooi
2011-11-25 16:53 ` Richard Purdie
2011-11-25 18:15   ` Koen Kooi
2011-11-25 19:59     ` Koen Kooi
2011-11-29 11:40       ` Martin Jansa
2012-03-29  7:58         ` Martin Jansa
2012-03-29  8:59           ` Richard Purdie

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox