Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Evade Flow]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

This might be an 'X-Y' problem, so I should explain what I'm trying to
achieve. We have a somewhat beefy build server with 32 cores that most of
my team likes to `ssh` into to do builds of our Yocto/OE-based BSP. But a
few of us build on different machines often enough that I recently spun up
a web server—on the 32-core build machine—to act as a 'pre-mirror' to
statically serve  our dependencies for remote builds. The web server is
serving the exact same folder that I've been pointing `SOURCE_MIRROR_URL`
at, as described at:
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

When I went to integrate support for the HTTP pre-mirror, I ran into the
following problem: if I assign `SOURCE_MIRROR_URL` and the specified folder
doesn't exist (as will be the case for remote builds), the build *fails*. I
had hoped that bitbake would 'fall through' to searching other pre-mirrors,
but that doesn't appear to be the case(?)

IIs there some way I can use a specific (and *invariant*) local source
mirror folder *only if it exists*, and use our internal HTTP pre-mirror
otherwise?

```
$ bitbake --version
BitBake Build Tool Core version 1.28.0
```

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

Re: Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Evade Flow]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

> Is there some way I can use a specific (and *invariant*) local source
mirror folder *only if it exists*, and use our internal HTTP pre-mirror
otherwise?

Apologies, it seems I may have posted this to the 'wrong' list. Anyway, I
see that both poky and oe-core have an 'own-mirrors' class. Since I'm using
a Yocto-based distro... I guess I should have posted to
https://lists.yoctoproject.org/listinfo/poky...

On Thu, Sep 1, 2016 at 11:29 AM, Evade Flow <evadeflow@gmail.com> wrote:

> This might be an 'X-Y' problem, so I should explain what I'm trying to
> achieve. We have a somewhat beefy build server with 32 cores that most of
> my team likes to `ssh` into to do builds of our Yocto/OE-based BSP. But a
> few of us build on different machines often enough that I recently spun up
> a web server—on the 32-core build machine—to act as a 'pre-mirror' to
> statically serve  our dependencies for remote builds. The web server is
> serving the exact same folder that I've been pointing `SOURCE_MIRROR_URL`
> at, as described at: https://wiki.yoctoproject.org/
> wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
>
> When I went to integrate support for the HTTP pre-mirror, I ran into the
> following problem: if I assign `SOURCE_MIRROR_URL` and the specified folder
> doesn't exist (as will be the case for remote builds), the build *fails*. I
> had hoped that bitbake would 'fall through' to searching other pre-mirrors,
> but that doesn't appear to be the case(?)
>
> IIs there some way I can use a specific (and *invariant*) local source
> mirror folder *only if it exists*, and use our internal HTTP pre-mirror
> otherwise?
>
> ```
> $ bitbake --version
> BitBake Build Tool Core version 1.28.0
> ```
>
>

[-- Attachment #2: Type: text/html, Size: 2348 bytes --]

Re: Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Richard Purdie]

On Thu, 2016-09-01 at 11:29 -0400, Evade Flow wrote:
> This might be an 'X-Y' problem, so I should explain what I'm trying
> to achieve. We have a somewhat beefy build server with 32 cores that
> most of my team likes to `ssh` into to do builds of our Yocto/OE
> -based BSP. But a few of us build on different machines often enough
> that I recently spun up a web server—on the 32-core build machine—to
> act as a 'pre-mirror' to statically serve  our dependencies for
> remote builds. The web server is serving the exact same folder that
> I've been pointing `SOURCE_MIRROR_URL` at, as described at: https://w
> iki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_d
> ownload_mirror_.3F
> 
> When I went to integrate support for the HTTP pre-mirror, I ran into
> the following problem: if I assign `SOURCE_MIRROR_URL` and the
> specified folder doesn't exist (as will be the case for remote
> builds), the build *fails*. I had hoped that bitbake would 'fall
> through' to searching other pre-mirrors, but that doesn't appear to
> be the case(?)
> 
> IIs there some way I can use a specific (and *invariant*) local
> source mirror folder *only if it exists*, and use our internal HTTP
> pre-mirror otherwise?

You could do something like:

def testislocal(d):
    if os.path.exists("xxxx")
        return "local"
 
  return ""

LOCALBUILD = "${@testislocal(d)}"

SOURCE_MIRROR_URL ?= "{@base_contains('LOCALBUILD', 'local', 'file://xxx', 'http://xxx', d)}"

I'm sure there are ways to neaten this up but you get the idea...

Cheers,

Richard

Re: Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Evade Flow]

[-- Attachment #1: Type: text/plain, Size: 2253 bytes --]

> You could do something like:
...
>
SOURCE_MIRROR_URL ?= "{@base_contains('LOCALBUILD', 'local', 'file://xxx', '
http://xxx', d)}"


That's exactly what I was looking for, thanks. I gather I can't put
functions in local.conf, which is where our current build is setting
SOURCE_MIRROR_URL. Can I trouble you to recommend a home for such a
function? It 'feels weird' to put it in a recipe, is there some other,
semi-standard place to stuff helper functions?

On Fri, Sep 2, 2016 at 7:57 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2016-09-01 at 11:29 -0400, Evade Flow wrote:
> > This might be an 'X-Y' problem, so I should explain what I'm trying
> > to achieve. We have a somewhat beefy build server with 32 cores that
> > most of my team likes to `ssh` into to do builds of our Yocto/OE
> > -based BSP. But a few of us build on different machines often enough
> > that I recently spun up a web server—on the 32-core build machine—to
> > act as a 'pre-mirror' to statically serve  our dependencies for
> > remote builds. The web server is serving the exact same folder that
> > I've been pointing `SOURCE_MIRROR_URL` at, as described at: https://w
> > iki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_d
> > ownload_mirror_.3F
> >
> > When I went to integrate support for the HTTP pre-mirror, I ran into
> > the following problem: if I assign `SOURCE_MIRROR_URL` and the
> > specified folder doesn't exist (as will be the case for remote
> > builds), the build *fails*. I had hoped that bitbake would 'fall
> > through' to searching other pre-mirrors, but that doesn't appear to
> > be the case(?)
> >
> > IIs there some way I can use a specific (and *invariant*) local
> > source mirror folder *only if it exists*, and use our internal HTTP
> > pre-mirror otherwise?
>
> You could do something like:
>
> def testislocal(d):
>     if os.path.exists("xxxx")
>         return "local"
>
>   return ""
>
> LOCALBUILD = "${@testislocal(d)}"
>
> SOURCE_MIRROR_URL ?= "{@base_contains('LOCALBUILD', 'local', 'file://xxx',
> 'http://xxx', d)}"
>
> I'm sure there are ways to neaten this up but you get the idea...
>
> Cheers,
>
> Richard
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3252 bytes --]

Re: Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Richard Purdie]

On Fri, 2016-09-02 at 15:26 -0400, Evade Flow wrote:
> > You could do something like:
> ...
> > 
> SOURCE_MIRROR_URL ?= "{@base_contains('LOCALBUILD', 'local', '
> file://xxx', 'http://xxx', d)}"
> 
> 
> That's exactly what I was looking for, thanks. I gather I can't put
> functions in local.conf, which is where our current build is setting
> SOURCE_MIRROR_URL. Can I trouble you to recommend a home for such a
> function? It 'feels weird' to put it in a recipe, is there some
> other, semi-standard place to stuff helper functions?

It is an annoying and somewhat artificial limitation. I have been known
to work around it by adding a xxx.bbclass file in a classes directory
and then INHERIT += "xxx".

Cheers,

Richard

Re: Conditionally inherit own-mirrors and set SOURCE_MIRROR_URL?

[Evade Flow]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

> It is an annoying and somewhat artificial limitation. I have been known
> to work around it by adding a xxx.bbclass file in a classes directory
> and then INHERIT += "xxx".

That's a really handy trick to know about, thanks! I will probably switch
to that approach soon since—now that I see the possibilities—I anticipate
I'll be adding more conditional config items so my teammates don't have to
worry about it.

The solution I managed to finger-mumble my way to in the meantime is a
little scary-looking:

SOURCE_MIRROR_URL = "${@'file:///demo/source-mirror' if
os.path.exists('/demo/source-mirror') else
'http://mycompany.com:1234/source-mirror'}"
INHERIT += "own-mirrors"


About the best thing I can say about it is: it *does* work. And it's
actually pretty readable, aside from the line length being so long it won't
fit in a tweet(!)

On Fri, Sep 2, 2016 at 5:46 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2016-09-02 at 15:26 -0400, Evade Flow wrote:
> > > You could do something like:
> > ...
> > >
> > SOURCE_MIRROR_URL ?= "{@base_contains('LOCALBUILD', 'local', '
> > file://xxx', 'http://xxx', d)}"
> >
> >
> > That's exactly what I was looking for, thanks. I gather I can't put
> > functions in local.conf, which is where our current build is setting
> > SOURCE_MIRROR_URL. Can I trouble you to recommend a home for such a
> > function? It 'feels weird' to put it in a recipe, is there some
> > other, semi-standard place to stuff helper functions?
>
> It is an annoying and somewhat artificial limitation. I have been known
> to work around it by adding a xxx.bbclass file in a classes directory
> and then INHERIT += "xxx".
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]