All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Joshua Watt" <JPEWhacker@gmail.com>
To: bitbake-devel@lists.openembedded.org
Cc: Joshua Watt <JPEWhacker@gmail.com>
Subject: [bitbake-devel][PATCH v2 6/8] bitbake: cache: Cache size optimization
Date: Mon,  1 Jun 2020 16:49:43 -0500	[thread overview]
Message-ID: <20200601214945.32481-7-JPEWhacker@gmail.com> (raw)
In-Reply-To: <20200601214945.32481-1-JPEWhacker@gmail.com>

Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Only the base
configuration and the specific multiconfig that the cache is tracking is
required. This significantly reduces the number of files parsed and the
size of each cache when there are multiple multiconfigs enabled.

This still doesn't get quite to the ideal case, since each multiconfig
cache still has a independent copy of the base cache, but removing this
is more difficult and invasive.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 bitbake/lib/bb/cache.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index b34bfa9b5a..3683ac6246 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -328,7 +328,7 @@ class NoCache(object):
         bb_data = self.load_bbfile(virtualfn, appends, virtonly=True)
         return bb_data[virtual]
 
-    def load_bbfile(self, bbfile, appends, virtonly = False):
+    def load_bbfile(self, bbfile, appends, virtonly = False, mclist = None):
         """
         Load and parse one .bb build file
         Return the data and whether parsing resulted in the file being skipped
@@ -344,7 +344,10 @@ class NoCache(object):
         bb_data = self.data.createCopy()
         datastores = parse_recipe(bb_data, bbfile, appends)
 
-        for mc in self.databuilder.mcdata:
+        if mclist is None:
+            mclist = self.databuilder.mcdata.keys()
+
+        for mc in mclist:
             if not mc:
                 continue
             bb_data = self.databuilder.mcdata[mc].createCopy()
@@ -500,7 +503,7 @@ class Cache(NoCache):
         """Parse the specified filename, returning the recipe information"""
         self.logger.debug(1, "Parsing %s", filename)
         infos = []
-        datastores = self.load_bbfile(filename, appends)
+        datastores = self.load_bbfile(filename, appends, mclist=(self.mc,))
         depends = []
         variants = []
         # Process the "real" fn last so we can store variants list
-- 
2.26.2


  parent reply	other threads:[~2020-06-01 21:49 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-01 20:27 [bitbake-devel][PATCH 0/8] Add support for per-multiconfig BBMASK Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 1/8] bitbake: cooker: Split file collections per multiconfig Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 2/8] bitbake: cache: Use multiconfig aware caches Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 3/8] bitbake: lib: Add support for Logging Adapters Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 4/8] bitbake: lib: Add PrefixLoggerAdapter helper Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 5/8] bitbake: cache: Improve logging Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 6/8] bitbake: cache: Cache size optimization Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 7/8] bitbake: tests: Add tests for BBMASK in multiconfig Joshua Watt
2020-06-01 20:28 ` [bitbake-devel][PATCH 8/8] bitbake: command: Move split_mc_pn to runqueue Joshua Watt
2020-06-01 21:49 ` [bitbake-devel][PATCH v2 0/8] Add support for per-multiconfig BBMASK Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 1/8] bitbake: cooker: Split file collections per multiconfig Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 2/8] bitbake: cache: Use multiconfig aware caches Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 3/8] bitbake: lib: Add support for Logging Adapters Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 4/8] bitbake: lib: Add PrefixLoggerAdapter helper Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 5/8] bitbake: cache: Improve logging Joshua Watt
2020-06-01 21:49   ` Joshua Watt [this message]
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 7/8] bitbake: tests: Add tests for BBMASK in multiconfig Joshua Watt
2020-06-01 21:49   ` [bitbake-devel][PATCH v2 8/8] bitbake: command: Move split_mc_pn to runqueue Joshua Watt
2020-06-03  2:53 ` [OE-core][PATCH v3 0/8] Add support for per-multiconfig BBMASK Joshua Watt
2020-06-03  2:53   ` [OE-core][PATCH v3 1/8] bitbake: cooker: Split file collections per multiconfig Joshua Watt
2020-06-03  2:53   ` [OE-core][PATCH v3 2/8] bitbake: cache: Use multiconfig aware caches Joshua Watt
2020-06-03  2:54   ` [OE-core][PATCH v3 3/8] bitbake: lib: Add support for Logging Adapters Joshua Watt
2020-06-03  2:54   ` [OE-core][PATCH v3 4/8] bitbake: lib: Add PrefixLoggerAdapter helper Joshua Watt
2020-06-04 21:00     ` Richard Purdie
2020-06-03  2:54   ` [OE-core][PATCH v3 5/8] bitbake: cache: Improve logging Joshua Watt
2020-06-03  2:54   ` [OE-core][PATCH v3 6/8] bitbake: cache: Cache size optimization Joshua Watt
2020-06-03  2:54   ` [OE-core][PATCH v3 7/8] bitbake: tests: Add tests for BBMASK in multiconfig Joshua Watt
2020-06-03  2:54   ` [OE-core][PATCH v3 8/8] bitbake: command: Move split_mc_pn to runqueue Joshua Watt
2020-06-05  1:53   ` [bitbake-devel][PATCH v4 0/8] Add support for per-multiconfig BBMASK Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 1/8] bitbake: cooker: Split file collections per multiconfig Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 2/8] bitbake: cache: Use multiconfig aware caches Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 3/8] bitbake: lib: Add support for Logging Adapters Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 4/8] bitbake: lib: Add PrefixLoggerAdapter helper Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 5/8] bitbake: cache: Improve logging Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 6/8] bitbake: cache: Cache size optimization Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 7/8] bitbake: tests: Add tests for BBMASK in multiconfig Joshua Watt
2020-06-05  1:53     ` [bitbake-devel][PATCH v4 8/8] bitbake: command: Move split_mc_pn to runqueue Joshua Watt
2020-06-05  7:29     ` [bitbake-devel][PATCH v4 0/8] Add support for per-multiconfig BBMASK Alejandro Hernandez
2020-06-05 15:22       ` Joshua Watt
2020-06-05 20:05     ` Richard Purdie
2020-06-06  3:15     ` [bitbake-devel][PATCH v5 " Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 1/8] bitbake: cooker: Split file collections per multiconfig Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 2/8] bitbake: cache: Use multiconfig aware caches Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 3/8] bitbake: lib: Add support for Logging Adapters Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 4/8] bitbake: lib: Add PrefixLoggerAdapter helper Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 5/8] bitbake: cache: Improve logging Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 6/8] bitbake: cache: Cache size optimization Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 7/8] bitbake: tests: Add tests for BBMASK in multiconfig Joshua Watt
2020-06-06  3:15       ` [bitbake-devel][PATCH v5 8/8] bitbake: command: Move split_mc_pn to runqueue Joshua Watt
2020-06-08 20:32       ` [bitbake-devel][PATCH v5 0/8] Add support for per-multiconfig BBMASK Richard Purdie
2020-06-03  3:02 ` ✗ patchtest: failure for " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200601214945.32481-7-JPEWhacker@gmail.com \
    --to=jpewhacker@gmail.com \
    --cc=bitbake-devel@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.