* [RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set @ 2013-06-28 15:05 Joe MacDonald 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors Joe MacDonald 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 2/2] sstate.bbclass: " Joe MacDonald 0 siblings, 2 replies; 7+ messages in thread From: Joe MacDonald @ 2013-06-28 15:05 UTC (permalink / raw) To: bitbake-devel My objective here is to support a very specific scenario. I have a well-fed sstate cache on my internal network. I want to use it for accelerating builds across a variety of machines where NFS is not really an option. I don't want bitbake to ever attempt to fetch sources for packages, though, so I'm always using BB_NO_NETWORK. The proposal I have is adding a new local.conf variable that will allow cache fetches only to be exempt from BB_NO_NETWORK restrictions. If it is set and BB_NO_NETWORK is not set, there's no change in behaviour. If it is not set and BB_NO_NETWORK is set, again, no change in today's behaviour. If both are set, however, then any of the regular fetch2 fetchers will get an "okay" from check_network_access() if they're invoked by a sstate cache fetch attempt. Any other fetch attempt will still hit raise a NetworkAccess exception. -- -Joe MacDonald. :wq ^ permalink raw reply [flat|nested] 7+ messages in thread
* [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors 2013-06-28 15:05 [RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set Joe MacDonald @ 2013-06-28 15:05 ` Joe MacDonald 2013-06-28 16:56 ` Richard Purdie 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 2/2] sstate.bbclass: " Joe MacDonald 1 sibling, 1 reply; 7+ messages in thread From: Joe MacDonald @ 2013-06-28 15:05 UTC (permalink / raw) To: bitbake-devel BB_NO_NETWORK disables any fetching, however if we're using an external sstate cache, we may want to be able to fetch those objects even if we are not fetching the upstream sources. Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> --- bitbake/lib/bb/fetch2/__init__.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 6211cd7..edd290b3 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None): """ log remote network access, and error if BB_NO_NETWORK is set """ - if d.getVar("BB_NO_NETWORK", True) == "1": - raise NetworkAccess(url, info) + if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True: + logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info) else: - logger.debug(1, "Fetcher accessed the network with the command %s" % info) + if d.getVar("BB_NO_NETWORK", True) == "1": + raise NetworkAccess(url, info) + else: + logger.debug(1, "Fetcher accessed the network with the command %s" % info) def build_mirroruris(origud, mirrors, ld): uris = [] @@ -1301,7 +1304,7 @@ class FetchMethod(object): return "%s-%s" % (key, d.getVar("PN", True) or "") class Fetch(object): - def __init__(self, urls, d, cache = True, localonly = False): + def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False): if localonly and cache: raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time") @@ -1310,6 +1313,8 @@ class Fetch(object): self.urls = urls self.d = d self.ud = {} + self.sstate_cache = sstate_cache + d.setVar("LOCAL_SSTATE_CACHE", sstate_cache) fn = d.getVar('FILE', True) if cache and fn and fn in urldata_cache: @@ -1354,7 +1359,10 @@ class Fetch(object): if len(urls) == 0: urls = self.urls - network = self.d.getVar("BB_NO_NETWORK", True) + if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True: + network = True + else: + network = self.d.getVar("BB_NO_NETWORK", True) premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1") for u in urls: -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors Joe MacDonald @ 2013-06-28 16:56 ` Richard Purdie 2013-06-28 17:00 ` Mark Hatle 0 siblings, 1 reply; 7+ messages in thread From: Richard Purdie @ 2013-06-28 16:56 UTC (permalink / raw) To: Joe MacDonald; +Cc: bitbake-devel On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote: > BB_NO_NETWORK disables any fetching, however if we're using an external > sstate cache, we may want to be able to fetch those objects even if we are > not fetching the upstream sources. > > Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> > --- > bitbake/lib/bb/fetch2/__init__.py | 18 +++++++++++++----- > 1 file changed, 13 insertions(+), 5 deletions(-) Bitbake's fetch module knows nothing about sstate right now and I don't see any reason to teach it about it. Why can't you just change BB_NO_NETWORK in sstate.bbclass before we call into the fetcher? Cheers, Richard > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py > index 6211cd7..edd290b3 100644 > --- a/bitbake/lib/bb/fetch2/__init__.py > +++ b/bitbake/lib/bb/fetch2/__init__.py > @@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None): > """ > log remote network access, and error if BB_NO_NETWORK is set > """ > - if d.getVar("BB_NO_NETWORK", True) == "1": > - raise NetworkAccess(url, info) > + if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True: > + logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info) > else: > - logger.debug(1, "Fetcher accessed the network with the command %s" % info) > + if d.getVar("BB_NO_NETWORK", True) == "1": > + raise NetworkAccess(url, info) > + else: > + logger.debug(1, "Fetcher accessed the network with the command %s" % info) > > def build_mirroruris(origud, mirrors, ld): > uris = [] > @@ -1301,7 +1304,7 @@ class FetchMethod(object): > return "%s-%s" % (key, d.getVar("PN", True) or "") > > class Fetch(object): > - def __init__(self, urls, d, cache = True, localonly = False): > + def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False): > if localonly and cache: > raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time") > > @@ -1310,6 +1313,8 @@ class Fetch(object): > self.urls = urls > self.d = d > self.ud = {} > + self.sstate_cache = sstate_cache > + d.setVar("LOCAL_SSTATE_CACHE", sstate_cache) > > fn = d.getVar('FILE', True) > if cache and fn and fn in urldata_cache: > @@ -1354,7 +1359,10 @@ class Fetch(object): > if len(urls) == 0: > urls = self.urls > > - network = self.d.getVar("BB_NO_NETWORK", True) > + if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True: > + network = True > + else: > + network = self.d.getVar("BB_NO_NETWORK", True) > premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1") > > for u in urls: ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors 2013-06-28 16:56 ` Richard Purdie @ 2013-06-28 17:00 ` Mark Hatle 2013-06-28 17:04 ` Richard Purdie 0 siblings, 1 reply; 7+ messages in thread From: Mark Hatle @ 2013-06-28 17:00 UTC (permalink / raw) To: bitbake-devel On 6/28/13 11:56 AM, Richard Purdie wrote: > On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote: >> BB_NO_NETWORK disables any fetching, however if we're using an external >> sstate cache, we may want to be able to fetch those objects even if we are >> not fetching the upstream sources. >> >> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> >> --- >> bitbake/lib/bb/fetch2/__init__.py | 18 +++++++++++++----- >> 1 file changed, 13 insertions(+), 5 deletions(-) > > > Bitbake's fetch module knows nothing about sstate right now and I don't > see any reason to teach it about it. Why can't you just change > BB_NO_NETWORK in sstate.bbclass before we call into the fetcher? We considered that, but won't that then change the value potentially for other things that could be running in parallel? Or is the namespace protection good enough for that? I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)? --Mark > Cheers, > > Richard > >> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py >> index 6211cd7..edd290b3 100644 >> --- a/bitbake/lib/bb/fetch2/__init__.py >> +++ b/bitbake/lib/bb/fetch2/__init__.py >> @@ -714,10 +714,13 @@ def check_network_access(d, info = "", url = None): >> """ >> log remote network access, and error if BB_NO_NETWORK is set >> """ >> - if d.getVar("BB_NO_NETWORK", True) == "1": >> - raise NetworkAccess(url, info) >> + if d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == "1" and d.getVar("LOCAL_SSTATE_CACHE", True) == True: >> + logger.debug(1, "Fetcher accessed the network for sstate cache data with the command %s" % info) >> else: >> - logger.debug(1, "Fetcher accessed the network with the command %s" % info) >> + if d.getVar("BB_NO_NETWORK", True) == "1": >> + raise NetworkAccess(url, info) >> + else: >> + logger.debug(1, "Fetcher accessed the network with the command %s" % info) >> >> def build_mirroruris(origud, mirrors, ld): >> uris = [] >> @@ -1301,7 +1304,7 @@ class FetchMethod(object): >> return "%s-%s" % (key, d.getVar("PN", True) or "") >> >> class Fetch(object): >> - def __init__(self, urls, d, cache = True, localonly = False): >> + def __init__(self, urls, d, cache = True, localonly = False, sstate_cache = False): >> if localonly and cache: >> raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time") >> >> @@ -1310,6 +1313,8 @@ class Fetch(object): >> self.urls = urls >> self.d = d >> self.ud = {} >> + self.sstate_cache = sstate_cache >> + d.setVar("LOCAL_SSTATE_CACHE", sstate_cache) >> >> fn = d.getVar('FILE', True) >> if cache and fn and fn in urldata_cache: >> @@ -1354,7 +1359,10 @@ class Fetch(object): >> if len(urls) == 0: >> urls = self.urls >> >> - network = self.d.getVar("BB_NO_NETWORK", True) >> + if self.sstate_cache == True and self.d.getVar("SSTATE_MIRROR_ALLOW_NETWORK", True) == True: >> + network = True >> + else: >> + network = self.d.getVar("BB_NO_NETWORK", True) >> premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY", True) == "1") >> >> for u in urls: > > > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors 2013-06-28 17:00 ` Mark Hatle @ 2013-06-28 17:04 ` Richard Purdie 2013-06-28 19:19 ` Joe MacDonald 0 siblings, 1 reply; 7+ messages in thread From: Richard Purdie @ 2013-06-28 17:04 UTC (permalink / raw) To: Mark Hatle; +Cc: bitbake-devel On Fri, 2013-06-28 at 12:00 -0500, Mark Hatle wrote: > On 6/28/13 11:56 AM, Richard Purdie wrote: > > On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote: > >> BB_NO_NETWORK disables any fetching, however if we're using an external > >> sstate cache, we may want to be able to fetch those objects even if we are > >> not fetching the upstream sources. > >> > >> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> > >> --- > >> bitbake/lib/bb/fetch2/__init__.py | 18 +++++++++++++----- > >> 1 file changed, 13 insertions(+), 5 deletions(-) > > > > > > Bitbake's fetch module knows nothing about sstate right now and I don't > > see any reason to teach it about it. Why can't you just change > > BB_NO_NETWORK in sstate.bbclass before we call into the fetcher? > > We considered that, but won't that then change the value potentially for other > things that could be running in parallel? Or is the namespace protection good > enough for that? > > I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)? If you look carefully you'll see sstate.bbclass does a bb.data.createCopy() of the datastore and only changes that before calling into the fetcher module. The change is only local to that copy and the copy only lives as long as its referenced so it should all be fine. Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors 2013-06-28 17:04 ` Richard Purdie @ 2013-06-28 19:19 ` Joe MacDonald 0 siblings, 0 replies; 7+ messages in thread From: Joe MacDonald @ 2013-06-28 19:19 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2025 bytes --] [Re: [bitbake-devel] [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors] On 13.06.28 (Fri 18:04) Richard Purdie wrote: > On Fri, 2013-06-28 at 12:00 -0500, Mark Hatle wrote: > > On 6/28/13 11:56 AM, Richard Purdie wrote: > > > On Fri, 2013-06-28 at 11:05 -0400, Joe MacDonald wrote: > > >> BB_NO_NETWORK disables any fetching, however if we're using an external > > >> sstate cache, we may want to be able to fetch those objects even if we are > > >> not fetching the upstream sources. > > >> > > >> Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> > > >> --- > > >> bitbake/lib/bb/fetch2/__init__.py | 18 +++++++++++++----- > > >> 1 file changed, 13 insertions(+), 5 deletions(-) > > > > > > > > > Bitbake's fetch module knows nothing about sstate right now and I don't > > > see any reason to teach it about it. Why can't you just change > > > BB_NO_NETWORK in sstate.bbclass before we call into the fetcher? > > > > We considered that, but won't that then change the value potentially for other > > things that could be running in parallel? Or is the namespace protection good > > enough for that? > > > > I.e. in sstate.bbclass do a d.setVar ... call the fetch .. d.setVar (restore)? > > If you look carefully you'll see sstate.bbclass does a > bb.data.createCopy() of the datastore and only changes that before > calling into the fetcher module. The change is only local to that copy > and the copy only lives as long as its referenced so it should all be > fine. Okay, thanks for the feedback, guys. I've re-worked this into a purely sstate.bbclass change and verified it is still behaving nicely, I'll mail that out to openembedded-core@lists.openembedded.org now. -J. > > Cheers, > > Richard > > > > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel -- -Joe MacDonald. :wq [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* [bitbake][oe-core][RFC PATCH 2/2] sstate.bbclass: Add an extra check for sstate_mirrors 2013-06-28 15:05 [RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set Joe MacDonald 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors Joe MacDonald @ 2013-06-28 15:05 ` Joe MacDonald 1 sibling, 0 replies; 7+ messages in thread From: Joe MacDonald @ 2013-06-28 15:05 UTC (permalink / raw) To: bitbake-devel BB_NO_NETWORK disables any fetching, however if we're using an external sstate cache, we may want to be able to fetch those objects even if we are not fetching the upstream sources. Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com> --- meta/classes/sstate.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 28dc312..759fb93 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -515,7 +515,7 @@ def pstaging_fetch(sstatefetch, sstatepkg, d): 'file://{0}.siginfo'.format(sstatefetch)]: localdata.setVar('SRC_URI', srcuri) try: - fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False) + fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False, sstate_cache=True) fetcher.download() # Need to optimise this, if using file:// urls, the fetcher just changes the local path @@ -632,7 +632,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): bb.debug(2, "SState: Attempting to fetch %s" % srcuri) try: - fetcher = bb.fetch2.Fetch(srcuri.split(), localdata) + fetcher = bb.fetch2.Fetch(srcuri.split(), localdata, sstate_cache=True) fetcher.checkstatus() bb.debug(2, "SState: Successful fetch test for %s" % srcuri) ret.append(task) -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-06-28 19:19 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-06-28 15:05 [RFC PATCH] Allow remote sstate cache fetches when BB_NO_NETWORK set Joe MacDonald 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 1/2] fetch2: Add an extra check for sstate_mirrors Joe MacDonald 2013-06-28 16:56 ` Richard Purdie 2013-06-28 17:00 ` Mark Hatle 2013-06-28 17:04 ` Richard Purdie 2013-06-28 19:19 ` Joe MacDonald 2013-06-28 15:05 ` [bitbake][oe-core][RFC PATCH 2/2] sstate.bbclass: " Joe MacDonald
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.