Stable source repository

Stable source repository

[Esben Haabendal]

Hi

For some reason, some OSS projects does not offer persistent access to
old releases,
which continously give problems for OE and other similar distributions.

With the org.openembedded.stable branch, this problem is considerably worse.
The usual fix in .dev is to upgrade the recipes to a current release,
which of-course
is not what we want for a stable branch.

Take for example
http://bugs.openembedded.net/show_bug.cgi?id=4227

I think it is worth considering a permanent solution to this kind of problems.
We should have a download server which carries all sources referred to
by http and ftp URI's in org.openembedded.stable recipes.
Why not setup ftp://downloads.openembedded.org/org.openmbedded.stable
to mirror a copy of all sources referred to by current .stable recipes, and
when a bug is filed with a missing source, the solution is simply to change
the SRC_URI to use the ftp://downloads.openembedded.org URI instead.

Of-course, unless continous manual labor is to be required, we need to
write a script or something like that to maintain the mirror repository.

And we need a server with enough storage and a big fat pipe to the
Internet.

What do you think?

Best regards,
Esben Haabendal

Re: Stable source repository

[Javi Roman]

On Tue, Jul 22, 2008 at 10:11 AM, Esben Haabendal <esbenhaabendal@gmail.com>
wrote:

> Hi
>
> For some reason, some OSS projects does not offer persistent access to
> old releases,
> which continously give problems for OE and other similar distributions.
>
> With the org.openembedded.stable branch, this problem is considerably
> worse.
> The usual fix in .dev is to upgrade the recipes to a current release,
> which of-course
> is not what we want for a stable branch.
>
> Take for example
> http://bugs.openembedded.net/show_bug.cgi?id=4227
>
> I think it is worth considering a permanent solution to this kind of
> problems.
> We should have a download server which carries all sources referred to
> by http and ftp URI's in org.openembedded.stable recipes.
> Why not setup ftp://downloads.openembedded.org/org.openmbedded.stable
> to mirror a copy of all sources referred to by current .stable recipes, and
> when a bug is filed with a missing source, the solution is simply to change
> the SRC_URI to use the ftp://downloads.openembedded.org URI instead.
>
> Of-course, unless continous manual labor is to be required, we need to
> write a script or something like that to maintain the mirror repository.
>
> And we need a server with enough storage and a big fat pipe to the
> Internet.
>
> What do you think?
>
> Best regards,
> Esben Haabendal
>
>
I was thinking to automatize the search for old software versions by means
of http://www.filewatcher.com/ or something similar. If the original SRC_URI
fails, then OE could automatically look for the source file with searching
engines.

Regards.

-- 
Javi Roman

Re: Stable source repository

[Leon Woestenberg]

Esben,

On Tue, Jul 22, 2008 at 10:11 AM, Esben Haabendal
<esbenhaabendal@gmail.com> wrote:
>
> Of-course, unless continous manual labor is to be required, we need to
> write a script or something like that to maintain the mirror repository.
>
I have written a "where is that package at" ("witpa") tool to help me
find old archives.

I will submit it to OE.

Regards,
-- 
Leon

Re: Stable source repository

[Koen Kooi]

Esben Haabendal wrote:
> Hi
>
> For some reason, some OSS projects does not offer persistent access to
> old releases,
> which continously give problems for OE and other similar distributions.

Angstrom already provides a sourcemirror for exactly this reason.

>
> With the org.openembedded.stable branch, this problem is considerably worse.
> The usual fix in .dev is to upgrade the recipes to a current release,
> which of-course
> is not what we want for a stable branch.
>
> Take for example
> http://bugs.openembedded.net/show_bug.cgi?id=4227

Fakeroot is present on the angstrom sourcemirror.

> I think it is worth considering a permanent solution to this kind of problems.
> We should have a download server which carries all sources referred to
> by http and ftp URI's in org.openembedded.stable recipes.
> Why not setup ftp://downloads.openembedded.org/org.openmbedded.stable
> to mirror a copy of all sources referred to by current .stable recipes, and
> when a bug is filed with a missing source, the solution is simply to change
> the SRC_URI to use the ftp://downloads.openembedded.org URI instead.
>
> Of-course, unless continous manual labor is to be required, we need to
> write a script or something like that to maintain the mirror repository.

The angstrom autobuilder updates the sourcemirror already with 
distributable sources.

> And we need a server with enough storage and a big fat pipe to the
> Internet.
>
> What do you think?

If you use a sane distro like angstrom, you will automagically have a 
source mirror.

regards,

Koen

Re: Stable source repository

[Rich Pixley]

Locally, we have a comparable problem.  Not only can we not rely on 
external repositories for archiving code, but given the volume of 
downloads we'd be placing, we can't really even rely on external 
repositories for current versions due to bandwidth constraints, (most 
facilities would resent the load our company would place on their servers).

What's been done locally is to place copies of the tarchives we use into 
our "sources" directory in version control.  So when we check out our 
local copy of OE metadata, we also get a huge, prepopulated sources 
directory.  I've also hacked all of fetching methods in our local copy 
of bitbake such that any attempt to reach outside our company for source 
produces a fatal error.

I don't like our solution because it make "mrproper" a problem and we 
have no "mrproper" of our own.  I think a better solution would have 
been for us to create a local repository aside from our source control 
system.  Then people would check out OE metadata and in the process of 
building, they'd fetch from the local repository.  But that would still 
require hijacking the component URL's, which is ugly.

As a general solution, I think OE needs to support a chain of such 
repositories and I think the OE project itself needs to create and 
maintain one.  The only way to be sure that a tarchive will be available 
is if you own and manage the repository yourself.  There's precedent for 
this - debian source packages encapsulate the upstream source so that 
the source archive is complete on it's own.  This means that the debian 
repositories each contain copies of the upstream tarchives and access to 
the upstream repositories only need to be accessed by debian maintainers 
when creating packaging for a new version.

--rich

Re: Stable source repository

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]



Rich Pixley wrote:

> As a general solution, I think OE needs to support a chain of such 
> repositories and I think the OE project itself needs to create and 
> maintain one.  The only way to be sure that a tarchive will be available 
> is if you own and manage the repository yourself.  There's precedent for 
> this - debian source packages encapsulate the upstream source so that 
> the source archive is complete on it's own.  This means that the debian 
> repositories each contain copies of the upstream tarchives and access to 
> the upstream repositories only need to be accessed by debian maintainers 
> when creating packaging for a new version.

I suspect that the OE project maintaining a source mirror would be out 
of scope for the OE project. However, as noted by Koen, people creating 
distributions for OE should maintain source mirrors. (This is needed for 
long term GPL compliance).

The point that OE should make it easy to create and maintain source 
mirrors is a very good point. What support exist in OE today? (I use 
Angstrom, so it just works for me :)

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: Stable source repository

[Esben Haabendal]

On Tue, Jul 22, 2008 at 10:45 AM, Leon Woestenberg
<leon.woestenberg@gmail.com> wrote:
> On Tue, Jul 22, 2008 at 10:11 AM, Esben Haabendal
> <esbenhaabendal@gmail.com> wrote:
>>
>> Of-course, unless continous manual labor is to be required, we need to
>> write a script or something like that to maintain the mirror repository.
>>
> I have written a "where is that package at" ("witpa") tool to help me
> find old archives.
>
> I will submit it to OE.

Hi Leon,

Do you try to verify the validity of the new location of the
package/archive?  Like comparing with a stored md5 from the file from
the original/previous location?

/Esben

Re: Stable source repository

[Esben Haabendal]

On Tue, Jul 22, 2008 at 1:56 PM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> Esben Haabendal wrote:
>>
>> Hi
>>
>> For some reason, some OSS projects does not offer persistent access to
>> old releases,
>> which continously give problems for OE and other similar distributions.
>
> Angstrom already provides a sourcemirror for exactly this reason.

I see.  But I guess it is still valid to use OE without using Angstrom?

>> With the org.openembedded.stable branch, this problem is considerably
>> worse.
>> The usual fix in .dev is to upgrade the recipes to a current release,
>> which of-course
>> is not what we want for a stable branch.
>>
>> Take for example
>> http://bugs.openembedded.net/show_bug.cgi?id=4227
>
> Fakeroot is present on the angstrom sourcemirror.

I know.  I am just trying to find out if a better permanent solution
for OE in general could be established.

>> I think it is worth considering a permanent solution to this kind of
>> problems.
>> We should have a download server which carries all sources referred to
>> by http and ftp URI's in org.openembedded.stable recipes.
>> Why not setup ftp://downloads.openembedded.org/org.openmbedded.stable
>> to mirror a copy of all sources referred to by current .stable recipes,
>> and
>> when a bug is filed with a missing source, the solution is simply to
>> change
>> the SRC_URI to use the ftp://downloads.openembedded.org URI instead.
>>
>> Of-course, unless continous manual labor is to be required, we need to
>> write a script or something like that to maintain the mirror repository.
>
> The angstrom autobuilder updates the sourcemirror already with distributable
> sources.

It would be nice if someone knowing how to operate the angstrom
autobuilder could describe how to re-use that for other distros.  We
really should have an easy way to do the same for any OE distro.

>> And we need a server with enough storage and a big fat pipe to the
>> Internet.
>>
>> What do you think?
>
> If you use a sane distro like angstrom, you will automagically have a source
> mirror.

Sorry, I seem to be using a non-sane distro then. I am just seeking a
way to sanitize it.

/Esben

Re: Stable source repository

[Esben Haabendal]

On Tue, Jul 22, 2008 at 9:28 PM, Philip Balister <philip@balister.org> wrote:
> The point that OE should make it easy to create and maintain source mirrors
> is a very good point. What support exist in OE today? (I use Angstrom, so it
> just works for me :)

Perhaps the Angstrom autobuilder could be stripped down to
sourcemirror tool, which could then be used for easy setup of
sourcemirrors for any OE based distro wanting a stable development
environment.

/Esben

Re: Stable source repository

[Leon Woestenberg]

Hello Esben,

On Tue, Jul 22, 2008 at 11:07 PM, Esben Haabendal
<esbenhaabendal@gmail.com> wrote:
> On Tue, Jul 22, 2008 at 10:45 AM, Leon Woestenberg
> <leon.woestenberg@gmail.com> wrote:
>> On Tue, Jul 22, 2008 at 10:11 AM, Esben Haabendal
>> <esbenhaabendal@gmail.com> wrote:
>>>
>>> Of-course, unless continous manual labor is to be required, we need to
>>> write a script or something like that to maintain the mirror repository.
>>>
>> I have written a "where is that package at" ("witpa") tool to help me
>> find old archives.
>>
> Hi Leon,
>
> Do you try to verify the validity of the new location of the
> package/archive?  Like comparing with a stored md5 from the file from
> the original/previous location?
>
Yes, I do.

Let me get that thing online.

I'll post to this list soon.
-- 
Leon

Re: Stable source repository

[Leon Woestenberg]

Hello,

On Wed, Jul 30, 2008 at 5:07 PM, Leon Woestenberg
<leon.woestenberg@gmail.com> wrote:
> On Tue, Jul 22, 2008 at 11:07 PM, Esben Haabendal
> <esbenhaabendal@gmail.com> wrote:
>> On Tue, Jul 22, 2008 at 10:45 AM, Leon Woestenberg
>> <leon.woestenberg@gmail.com> wrote:
>>> On Tue, Jul 22, 2008 at 10:11 AM, Esben Haabendal
>>> <esbenhaabendal@gmail.com> wrote:
>>>>
>>>> Of-course, unless continous manual labor is to be required, we need to
>>>> write a script or something like that to maintain the mirror repository.
>>>>
>>> I have written a "where is that package at" ("witpa") tool to help me
>>> find old archives.
>>>
>> Hi Leon,
>>
>> Do you try to verify the validity of the new location of the
>> package/archive?  Like comparing with a stored md5 from the file from
>> the original/previous location?
>>
> Yes, I do.
>
> Let me get that thing online.
>
> I'll post to this list soon.
> --

Done. See the new thread on OE-devel.

Regards,
-- 
Leon