[Buildroot] how to point at a local archive of S/W tarballs?

[Buildroot] how to point at a local archive of S/W tarballs?

[Robert P. J. Day]

  [apologies if you've seen this before -- i posted it yesterday but
it doesn't seem to show up in the ML archives.]

  if i'm starting a new build, how can i take advantage of the
gigabytes of S/W tarballs i've collected for doing OE/yocto project
builds?

  i want to set the equivalent of a local mirror -- would i set a
value in the "Primary download site" config option, with the format
"file:///pathname/to/all/my/tarballs"?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

[Buildroot] how to point at a local archive of S/W tarballs?

[Samuel Martin]

Hi Robert,

On Thu, Apr 16, 2015 at 2:21 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>   [apologies if you've seen this before -- i posted it yesterday but
> it doesn't seem to show up in the ML archives.]
>
>   if i'm starting a new build, how can i take advantage of the
> gigabytes of S/W tarballs i've collected for doing OE/yocto project
> builds?
>
>   i want to set the equivalent of a local mirror -- would i set a
> value in the "Primary download site" config option, with the format
> "file:///pathname/to/all/my/tarballs"?

You certainly want to set BR2_DL_DIR [1].


[1] http://nightly.buildroot.org/#env-vars

Regards,

-- 
Samuel

[Buildroot] how to point at a local archive of S/W tarballs?

[Robert P. J. Day]

On Thu, 16 Apr 2015, Samuel Martin wrote:

> Hi Robert,
>
> On Thu, Apr 16, 2015 at 2:21 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> >
> >   [apologies if you've seen this before -- i posted it yesterday but
> > it doesn't seem to show up in the ML archives.]
> >
> >   if i'm starting a new build, how can i take advantage of the
> > gigabytes of S/W tarballs i've collected for doing OE/yocto project
> > builds?
> >
> >   i want to set the equivalent of a local mirror -- would i set a
> > value in the "Primary download site" config option, with the format
> > "file:///pathname/to/all/my/tarballs"?
>
> You certainly want to set BR2_DL_DIR [1].
>
> [1] http://nightly.buildroot.org/#env-vars

  i'm not convinced that's what i want ... AIUI, that variable defines
where the downloaded SW tarballs will be *placed*, and i'm happy to
leave that where it is as a local dl/ directory for the build.

  what i want is to define a local (on-host) mirror for tarballs that
will be searched before going out to the net. i definitely don't want
the fetch process to try to *place* anything in that local mirror,
only to *fetch* from it.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

[Buildroot] how to point at a local archive of S/W tarballs?

[Samuel Martin]

On Thu, Apr 16, 2015 at 2:40 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> On Thu, 16 Apr 2015, Samuel Martin wrote:
>
>> Hi Robert,
>>
>> On Thu, Apr 16, 2015 at 2:21 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>> >
>> >   [apologies if you've seen this before -- i posted it yesterday but
>> > it doesn't seem to show up in the ML archives.]
>> >
>> >   if i'm starting a new build, how can i take advantage of the
>> > gigabytes of S/W tarballs i've collected for doing OE/yocto project
>> > builds?
>> >
>> >   i want to set the equivalent of a local mirror -- would i set a
>> > value in the "Primary download site" config option, with the format
>> > "file:///pathname/to/all/my/tarballs"?
>>
>> You certainly want to set BR2_DL_DIR [1].
>>
>> [1] http://nightly.buildroot.org/#env-vars
>
>   i'm not convinced that's what i want ... AIUI, that variable defines
> where the downloaded SW tarballs will be *placed*, and i'm happy to
> leave that where it is as a local dl/ directory for the build.
>
>   what i want is to define a local (on-host) mirror for tarballs that
> will be searched before going out to the net. i definitely don't want
> the fetch process to try to *place* anything in that local mirror,
> only to *fetch* from it.

For alternative mirror, there is a couple of options in the menuconfig:
Build options ---> Mirrors and Download locations


-- 
Samuel

[Buildroot] how to point at a local archive of S/W tarballs?

[Robert P. J. Day]

On Thu, 16 Apr 2015, Samuel Martin wrote:

> On Thu, Apr 16, 2015 at 2:40 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> > On Thu, 16 Apr 2015, Samuel Martin wrote:
> >
> >> Hi Robert,
> >>
> >> On Thu, Apr 16, 2015 at 2:21 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> >> >
> >> >   [apologies if you've seen this before -- i posted it yesterday but
> >> > it doesn't seem to show up in the ML archives.]
> >> >
> >> >   if i'm starting a new build, how can i take advantage of the
> >> > gigabytes of S/W tarballs i've collected for doing OE/yocto project
> >> > builds?
> >> >
> >> >   i want to set the equivalent of a local mirror -- would i set a
> >> > value in the "Primary download site" config option, with the format
> >> > "file:///pathname/to/all/my/tarballs"?
> >>
> >> You certainly want to set BR2_DL_DIR [1].
> >>
> >> [1] http://nightly.buildroot.org/#env-vars
> >
> >   i'm not convinced that's what i want ... AIUI, that variable defines
> > where the downloaded SW tarballs will be *placed*, and i'm happy to
> > leave that where it is as a local dl/ directory for the build.
> >
> >   what i want is to define a local (on-host) mirror for tarballs that
> > will be searched before going out to the net. i definitely don't want
> > the fetch process to try to *place* anything in that local mirror,
> > only to *fetch* from it.
>
> For alternative mirror, there is a couple of options in the menuconfig:
> Build options ---> Mirrors and Download locations

  i know, i referred to that very config option in my original
posting. what i'm wondering is if that config option -- "Primary
download site" -- can be set to point at a local directory on my build
host that's already loaded with S/W tarballs i've built up over the
months for use with my OE and yocto builds.

  the help for that option says only that, "Valid URIs are URIs
recognized by $(WGET) and scp URIs of the form
scp://[user@]host:path." last time i looked, wget doesn't support the
"file://" form of URI, so i can't use that.

  is there a standard way to do this?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

[Buildroot] how to point at a local archive of S/W tarballs?

[Yann E. MORIN]

Robert, All,

On 2015-04-16 09:17 -0400, Robert P. J. Day spake thusly:
[--SNIP--]
>   i know, i referred to that very config option in my original
> posting. what i'm wondering is if that config option -- "Primary
> download site" -- can be set to point at a local directory on my build
> host that's already loaded with S/W tarballs i've built up over the
> months for use with my OE and yocto builds.
> 
>   the help for that option says only that, "Valid URIs are URIs
> recognized by $(WGET) and scp URIs of the form
> scp://[user@]host:path." last time i looked, wget doesn't support the
> "file://" form of URI, so i can't use that.
> 
>   is there a standard way to do this?

Short answer: no.

Long answer:

BR2_DL_DIR is used both as a way to look for existing download, and as
the location where new downloads will be stored.

So, if you set (in the menuconfig or on the command line)
BR2_DL_DIR=/path/to/your/local/cache it will be looked into for existing
archives, and Buildroot will also store the ones it downlaods on its on
there.

If you really, really wish no to shared the download location, you could
get to run a simple webserver locally that you would use as a primary.
But then Buildroot would also store the archives it downloads from the
primary in its own dl/ dir (hence duplication in that case).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] how to point at a local archive of S/W tarballs?

[Robert P. J. Day]

On Thu, 16 Apr 2015, Yann E. MORIN wrote:

> Robert, All,
>
> On 2015-04-16 09:17 -0400, Robert P. J. Day spake thusly:
> [--SNIP--]
> >   i know, i referred to that very config option in my original
> > posting. what i'm wondering is if that config option -- "Primary
> > download site" -- can be set to point at a local directory on my build
> > host that's already loaded with S/W tarballs i've built up over the
> > months for use with my OE and yocto builds.
> >
> >   the help for that option says only that, "Valid URIs are URIs
> > recognized by $(WGET) and scp URIs of the form
> > scp://[user@]host:path." last time i looked, wget doesn't support the
> > "file://" form of URI, so i can't use that.
> >
> >   is there a standard way to do this?
>
> Short answer: no.
>
> Long answer:
>
> BR2_DL_DIR is used both as a way to look for existing download, and as
> the location where new downloads will be stored.
>
> So, if you set (in the menuconfig or on the command line)
> BR2_DL_DIR=/path/to/your/local/cache it will be looked into for existing
> archives, and Buildroot will also store the ones it downlaods on its on
> there.
>
> If you really, really wish no to shared the download location, you could
> get to run a simple webserver locally that you would use as a primary.
> But then Buildroot would also store the archives it downloads from the
> primary in its own dl/ dir (hence duplication in that case).

  ok, that answers the question, thanks very much.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

[Buildroot] how to point at a local archive of S/W tarballs?

[Thomas De Schampheleire]

On April 16, 2015 7:07:32 PM CEST, "Robert P. J. Day" <rpjday@crashcourse.ca> wrote:
>On Thu, 16 Apr 2015, Yann E. MORIN wrote:
>
>> Robert, All,
>>
>> On 2015-04-16 09:17 -0400, Robert P. J. Day spake thusly:
>> [--SNIP--]
>> >   i know, i referred to that very config option in my original
>> > posting. what i'm wondering is if that config option -- "Primary
>> > download site" -- can be set to point at a local directory on my
>build
>> > host that's already loaded with S/W tarballs i've built up over the
>> > months for use with my OE and yocto builds.
>> >
>> >   the help for that option says only that, "Valid URIs are URIs
>> > recognized by $(WGET) and scp URIs of the form
>> > scp://[user@]host:path." last time i looked, wget doesn't support
>the
>> > "file://" form of URI, so i can't use that.
>> >
>> >   is there a standard way to do this?
>>
>> Short answer: no.
>>
>> Long answer:
>>
>> BR2_DL_DIR is used both as a way to look for existing download, and
>as
>> the location where new downloads will be stored.
>>
>> So, if you set (in the menuconfig or on the command line)
>> BR2_DL_DIR=/path/to/your/local/cache it will be looked into for
>existing
>> archives, and Buildroot will also store the ones it downlaods on its
>on
>> there.
>>
>> If you really, really wish no to shared the download location, you
>could
>> get to run a simple webserver locally that you would use as a
>primary.
>> But then Buildroot would also store the archives it downloads from
>the
>> primary in its own dl/ dir (hence duplication in that case).
>
>  ok, that answers the question, thanks very much.

Using BR2_PRIMARY_SITE with scp to localhost is normally more easy than 
setting up a web server, assuming an ssh server is already running.

That said, adding support for file:// in BR2_PRIMARY_SITE is trivial to add. In 
fact, I used to have such a patch in a much older buildroot version.

Best regards,
Thomas

[Buildroot] how to point at a local archive of S/W tarballs?

[Robert P. J. Day]

On Thu, 16 Apr 2015, Thomas De Schampheleire wrote:

> On April 16, 2015 7:07:32 PM CEST, "Robert P. J. Day" <rpjday@crashcourse.ca> wrote:
> >On Thu, 16 Apr 2015, Yann E. MORIN wrote:
> >
> >> Robert, All,
> >>
> >> On 2015-04-16 09:17 -0400, Robert P. J. Day spake thusly:
> >> [--SNIP--]
> >> >   i know, i referred to that very config option in my original
> >> > posting. what i'm wondering is if that config option -- "Primary
> >> > download site" -- can be set to point at a local directory on my
> >build
> >> > host that's already loaded with S/W tarballs i've built up over the
> >> > months for use with my OE and yocto builds.
> >> >
> >> >   the help for that option says only that, "Valid URIs are URIs
> >> > recognized by $(WGET) and scp URIs of the form
> >> > scp://[user@]host:path." last time i looked, wget doesn't support
> >the
> >> > "file://" form of URI, so i can't use that.
> >> >
> >> >   is there a standard way to do this?
> >>
> >> Short answer: no.
> >>
> >> Long answer:
> >>
> >> BR2_DL_DIR is used both as a way to look for existing download, and
> >as
> >> the location where new downloads will be stored.
> >>
> >> So, if you set (in the menuconfig or on the command line)
> >> BR2_DL_DIR=/path/to/your/local/cache it will be looked into for
> >existing
> >> archives, and Buildroot will also store the ones it downlaods on its
> >on
> >> there.
> >>
> >> If you really, really wish no to shared the download location, you
> >could
> >> get to run a simple webserver locally that you would use as a
> >primary.
> >> But then Buildroot would also store the archives it downloads from
> >the
> >> primary in its own dl/ dir (hence duplication in that case).
> >
> >  ok, that answers the question, thanks very much.
>
> Using BR2_PRIMARY_SITE with scp to localhost is normally more easy than
> setting up a web server, assuming an ssh server is already running.
>
> That said, adding support for file:// in BR2_PRIMARY_SITE is trivial to add. In
> fact, I used to have such a patch in a much older buildroot version.

  i think that would be useful as i'm in the situation of wanting to
have a massive archive of S/W tarballs available for students in
various embedded linux classes that they can take advantage of for
their builds, but i (unsurprisingly) want that to be a read-only
directory so they can't make a mess of it or arbitrarily add anything
to it.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

[Buildroot] how to point at a local archive of S/W tarballs?

[Yann E. MORIN]

Robert, All,

On 2015-04-16 13:33 -0400, Robert P. J. Day spake thusly:
> On Thu, 16 Apr 2015, Thomas De Schampheleire wrote:
[--SNIP--]
> > Using BR2_PRIMARY_SITE with scp to localhost is normally more easy than
> > setting up a web server, assuming an ssh server is already running.
> >
> > That said, adding support for file:// in BR2_PRIMARY_SITE is trivial to add. In
> > fact, I used to have such a patch in a much older buildroot version.
> 
>   i think that would be useful as i'm in the situation of wanting to
> have a massive archive of S/W tarballs available for students in
> various embedded linux classes that they can take advantage of for
> their builds, but i (unsurprisingly) want that to be a read-only
> directory so they can't make a mess of it or arbitrarily add anything
> to it.

Well, how would those students reach that location? NFS?

Would it not make much more sense to share that location via HTTP and
use that as the primary, so you know they won;t be able to write
anything in there?

I think this is typically a case where the primary (over HTTP) is indeed
a good solution. And as I replied to Thomas, running a *simple* webserver
is pretty easy. Granted, there's no access control, no fancy stuff but
directory listing and file download (no fancy PUT, POST COL and stuff),
but that's basically just what you need in that case...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] how to point at a local archive of S/W tarballs?

[Yann E. MORIN]

Thomas, All,

On 2015-04-16 19:27 +0200, Thomas De Schampheleire spake thusly:
[--SNIP--]
> Using BR2_PRIMARY_SITE with scp to localhost is normally more easy than 
> setting up a web server, assuming an ssh server is already running.

Well, what about:

    $ python -m SimpleHTTPServer 8181

which will server the current directory via HTTP?

Do you want something even simpler? Hehe! ;-)

> That said, adding support for file:// in BR2_PRIMARY_SITE is trivial to add. In 
> fact, I used to have such a patch in a much older buildroot version.

Well, I was thinking of this, too, but I really wonder if that makes
sense: archives are copied from the primary, so you'd end up with
duplication locally.

Unless we're playing it smart and make symlinks.

But then what would happen if the archive disapear from primary?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] how to point at a local archive of S/W tarballs?

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

>> That said, adding support for file:// in BR2_PRIMARY_SITE is trivial to add. In 
 >> fact, I used to have such a patch in a much older buildroot version.

 > Well, I was thinking of this, too, but I really wonder if that makes
 > sense: archives are copied from the primary, so you'd end up with
 > duplication locally.

 > Unless we're playing it smart and make symlinks.

 > But then what would happen if the archive disapear from primary?

I would prefer to keep it dumb and simple. The typical usecase for
BR2_PRIMARY_SITE is imho just to use to local mirror for download speed
/ reliability reasons (that's atleast what I have used it for).

I don't feel strongly for/against file:// support, but if there's a
desire to add it ..

-- 
Bye, Peter Korsgaard