* Fetch failure for source at kernel.org @ 2011-09-21 19:38 Philip Balister 2011-09-21 19:43 ` Mark Hatle 0 siblings, 1 reply; 13+ messages in thread From: Philip Balister @ 2011-09-21 19:38 UTC (permalink / raw) To: openembedded-core I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails. What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core. Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance? Thanks, Philip ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-21 19:38 Fetch failure for source at kernel.org Philip Balister @ 2011-09-21 19:43 ` Mark Hatle 2011-09-21 19:49 ` Koen Kooi 0 siblings, 1 reply; 13+ messages in thread From: Mark Hatle @ 2011-09-21 19:43 UTC (permalink / raw) To: openembedded-core On 9/21/11 2:38 PM, Philip Balister wrote: > I'm finally trying to get started with oe-core. I'm using a setup > created with the setup-script created by Koen. Everything looks ok until > it tried to build cpufrequtils. The recipe tries to fetch the source > from a git repo at kernel.org. Obviously, the fetch fails. > > What is odd, in the "good old days" I would expect this config to fall > back and use the Angstrom source mirror, but apparently this is not the > case with oe-core. > > Is this a bug or a feature? If this is a bug, is there a plan to fix it? > If it is a feature, how are we supposed to do builds if servers go away? > What do we do about GPL compliance? This is a feature, as people did not want oe-core tied to a given mirror. Myself, I use: PREMIRRORS ?= "\ bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n" MIRRORS =+ "\ ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n" PREMIRRORS are tried before the upstream source, MIRRORS are tried only after a failure. (Please correct me someone if I have that wrong.) --Mark > Thanks, > > Philip > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-21 19:43 ` Mark Hatle @ 2011-09-21 19:49 ` Koen Kooi 2011-09-22 0:26 ` Richard Purdie 0 siblings, 1 reply; 13+ messages in thread From: Koen Kooi @ 2011-09-21 19:49 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: > On 9/21/11 2:38 PM, Philip Balister wrote: >> I'm finally trying to get started with oe-core. I'm using a setup >> created with the setup-script created by Koen. Everything looks ok until >> it tried to build cpufrequtils. The recipe tries to fetch the source >> from a git repo at kernel.org. Obviously, the fetch fails. >> >> What is odd, in the "good old days" I would expect this config to fall >> back and use the Angstrom source mirror, but apparently this is not the >> case with oe-core. >> >> Is this a bug or a feature? If this is a bug, is there a plan to fix it? >> If it is a feature, how are we supposed to do builds if servers go away? >> What do we do about GPL compliance? > > This is a feature, as people did not want oe-core tied to a given mirror. This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors. regards, Koen ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-21 19:49 ` Koen Kooi @ 2011-09-22 0:26 ` Richard Purdie 2011-09-22 6:40 ` Koen Kooi 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2011-09-22 0:26 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: > Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: > > > On 9/21/11 2:38 PM, Philip Balister wrote: > >> I'm finally trying to get started with oe-core. I'm using a setup > >> created with the setup-script created by Koen. Everything looks ok until > >> it tried to build cpufrequtils. The recipe tries to fetch the source > >> from a git repo at kernel.org. Obviously, the fetch fails. > >> > >> What is odd, in the "good old days" I would expect this config to fall > >> back and use the Angstrom source mirror, but apparently this is not the > >> case with oe-core. > >> > >> Is this a bug or a feature? If this is a bug, is there a plan to fix it? > >> If it is a feature, how are we supposed to do builds if servers go away? > >> What do we do about GPL compliance? > > > > This is a feature, as people did not want oe-core tied to a given mirror. > > This is the "OE core doesn't generate versioned git tarballs" problem > people keep talking about. I know I can edit each and every recipe to > add ';rebasable=true', but I'd like a global solution before the GPL > police starts looking at the angstrom source mirrors. Is this a versioned tarball problem or just the tarballs not being generated at all? Are you setting BB_GENERATE_MIRROR_TARBALLS? Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 0:26 ` Richard Purdie @ 2011-09-22 6:40 ` Koen Kooi 2011-09-22 7:03 ` Richard Purdie 0 siblings, 1 reply; 13+ messages in thread From: Koen Kooi @ 2011-09-22 6:40 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: > On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: >> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: >> >>> On 9/21/11 2:38 PM, Philip Balister wrote: >>>> I'm finally trying to get started with oe-core. I'm using a setup >>>> created with the setup-script created by Koen. Everything looks ok until >>>> it tried to build cpufrequtils. The recipe tries to fetch the source >>>> from a git repo at kernel.org. Obviously, the fetch fails. >>>> >>>> What is odd, in the "good old days" I would expect this config to fall >>>> back and use the Angstrom source mirror, but apparently this is not the >>>> case with oe-core. >>>> >>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? >>>> If it is a feature, how are we supposed to do builds if servers go away? >>>> What do we do about GPL compliance? >>> >>> This is a feature, as people did not want oe-core tied to a given mirror. >> >> This is the "OE core doesn't generate versioned git tarballs" problem >> people keep talking about. I know I can edit each and every recipe to >> add ';rebasable=true', but I'd like a global solution before the GPL >> police starts looking at the angstrom source mirrors. > > Is this a versioned tarball problem or just the tarballs not being > generated at all? versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 6:40 ` Koen Kooi @ 2011-09-22 7:03 ` Richard Purdie 2011-09-22 8:28 ` Koen Kooi 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2011-09-22 7:03 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: > Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: > > > On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: > >> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: > >> > >>> On 9/21/11 2:38 PM, Philip Balister wrote: > >>>> I'm finally trying to get started with oe-core. I'm using a setup > >>>> created with the setup-script created by Koen. Everything looks ok until > >>>> it tried to build cpufrequtils. The recipe tries to fetch the source > >>>> from a git repo at kernel.org. Obviously, the fetch fails. > >>>> > >>>> What is odd, in the "good old days" I would expect this config to fall > >>>> back and use the Angstrom source mirror, but apparently this is not the > >>>> case with oe-core. > >>>> > >>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? > >>>> If it is a feature, how are we supposed to do builds if servers go away? > >>>> What do we do about GPL compliance? > >>> > >>> This is a feature, as people did not want oe-core tied to a given mirror. > >> > >> This is the "OE core doesn't generate versioned git tarballs" problem > >> people keep talking about. I know I can edit each and every recipe to > >> add ';rebasable=true', but I'd like a global solution before the GPL > >> police starts looking at the angstrom source mirrors. > > > > Is this a versioned tarball problem or just the tarballs not being > > generated at all? > > versioned tarball problem, since the source mirror won't overwrite > files with the same name, so unversioned ones are both useless and > bandwidth wasting. They seem to be working perfectly well for Yocto... They're also not bandwidth wasting since the fetcher is efficient and will incrementally update something that is incomplete. Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always). Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 7:03 ` Richard Purdie @ 2011-09-22 8:28 ` Koen Kooi 2011-09-22 8:45 ` Martin Jansa 2011-09-22 10:20 ` Richard Purdie 0 siblings, 2 replies; 13+ messages in thread From: Koen Kooi @ 2011-09-22 8:28 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven: > On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: >> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: >> >>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: >>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: >>>> >>>>> On 9/21/11 2:38 PM, Philip Balister wrote: >>>>>> I'm finally trying to get started with oe-core. I'm using a setup >>>>>> created with the setup-script created by Koen. Everything looks ok until >>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source >>>>>> from a git repo at kernel.org. Obviously, the fetch fails. >>>>>> >>>>>> What is odd, in the "good old days" I would expect this config to fall >>>>>> back and use the Angstrom source mirror, but apparently this is not the >>>>>> case with oe-core. >>>>>> >>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? >>>>>> If it is a feature, how are we supposed to do builds if servers go away? >>>>>> What do we do about GPL compliance? >>>>> >>>>> This is a feature, as people did not want oe-core tied to a given mirror. >>>> >>>> This is the "OE core doesn't generate versioned git tarballs" problem >>>> people keep talking about. I know I can edit each and every recipe to >>>> add ';rebasable=true', but I'd like a global solution before the GPL >>>> police starts looking at the angstrom source mirrors. >>> >>> Is this a versioned tarball problem or just the tarballs not being >>> generated at all? >> >> versioned tarball problem, since the source mirror won't overwrite >> files with the same name, so unversioned ones are both useless and >> bandwidth wasting. > > They seem to be working perfectly well for Yocto... > > They're also not bandwidth wasting They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic. The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. > since the fetcher is efficient and > will incrementally update something that is incomplete. Which won't work currently since kernel.org is down. > Anyhow, this is the first time someone has explained that the mirror > tarball improvements that were made to bitbake have not done everything > they desire. If you want someone on the Yocto team to look at this I'd > suggest a bug report or I'll take patches to bitbake (as always). But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev. regards, Koen ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 8:28 ` Koen Kooi @ 2011-09-22 8:45 ` Martin Jansa 2011-09-22 9:01 ` Koen Kooi 2011-09-22 10:20 ` Richard Purdie 1 sibling, 1 reply; 13+ messages in thread From: Martin Jansa @ 2011-09-22 8:45 UTC (permalink / raw) To: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 3502 bytes --] On Thu, Sep 22, 2011 at 10:28:02AM +0200, Koen Kooi wrote: > > Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven: > > > On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: > >> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: > >> > >>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: > >>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: > >>>> > >>>>> On 9/21/11 2:38 PM, Philip Balister wrote: > >>>>>> I'm finally trying to get started with oe-core. I'm using a setup > >>>>>> created with the setup-script created by Koen. Everything looks ok until > >>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source > >>>>>> from a git repo at kernel.org. Obviously, the fetch fails. > >>>>>> > >>>>>> What is odd, in the "good old days" I would expect this config to fall > >>>>>> back and use the Angstrom source mirror, but apparently this is not the > >>>>>> case with oe-core. > >>>>>> > >>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? > >>>>>> If it is a feature, how are we supposed to do builds if servers go away? > >>>>>> What do we do about GPL compliance? > >>>>> > >>>>> This is a feature, as people did not want oe-core tied to a given mirror. > >>>> > >>>> This is the "OE core doesn't generate versioned git tarballs" problem > >>>> people keep talking about. I know I can edit each and every recipe to > >>>> add ';rebasable=true', but I'd like a global solution before the GPL > >>>> police starts looking at the angstrom source mirrors. > >>> > >>> Is this a versioned tarball problem or just the tarballs not being > >>> generated at all? > >> > >> versioned tarball problem, since the source mirror won't overwrite > >> files with the same name, so unversioned ones are both useless and > >> bandwidth wasting. > > > > They seem to be working perfectly well for Yocto... > > > > They're also not bandwidth wasting > > They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic. > > The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. > > > since the fetcher is efficient and > > will incrementally update something that is incomplete. > > Which won't work currently since kernel.org is down. > > > Anyhow, this is the first time someone has explained that the mirror > > tarball improvements that were made to bitbake have not done everything > > they desire. If you want someone on the Yocto team to look at this I'd > > suggest a bug report or I'll take patches to bitbake (as always). > > But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev. FWIW: I would also like to have versioned git checkouts in fetcher2 as it was in old fetcher. It would be more consitent with svn versioned checkouts which are still created. Regards, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 8:45 ` Martin Jansa @ 2011-09-22 9:01 ` Koen Kooi 0 siblings, 0 replies; 13+ messages in thread From: Koen Kooi @ 2011-09-22 9:01 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 22 sep. 2011, om 10:45 heeft Martin Jansa het volgende geschreven: > On Thu, Sep 22, 2011 at 10:28:02AM +0200, Koen Kooi wrote: >> >> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven: >> >>> On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: >>>> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: >>>> >>>>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: >>>>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: >>>>>> >>>>>>> On 9/21/11 2:38 PM, Philip Balister wrote: >>>>>>>> I'm finally trying to get started with oe-core. I'm using a setup >>>>>>>> created with the setup-script created by Koen. Everything looks ok until >>>>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source >>>>>>>> from a git repo at kernel.org. Obviously, the fetch fails. >>>>>>>> >>>>>>>> What is odd, in the "good old days" I would expect this config to fall >>>>>>>> back and use the Angstrom source mirror, but apparently this is not the >>>>>>>> case with oe-core. >>>>>>>> >>>>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? >>>>>>>> If it is a feature, how are we supposed to do builds if servers go away? >>>>>>>> What do we do about GPL compliance? >>>>>>> >>>>>>> This is a feature, as people did not want oe-core tied to a given mirror. >>>>>> >>>>>> This is the "OE core doesn't generate versioned git tarballs" problem >>>>>> people keep talking about. I know I can edit each and every recipe to >>>>>> add ';rebasable=true', but I'd like a global solution before the GPL >>>>>> police starts looking at the angstrom source mirrors. >>>>> >>>>> Is this a versioned tarball problem or just the tarballs not being >>>>> generated at all? >>>> >>>> versioned tarball problem, since the source mirror won't overwrite >>>> files with the same name, so unversioned ones are both useless and >>>> bandwidth wasting. >>> >>> They seem to be working perfectly well for Yocto... >>> >>> They're also not bandwidth wasting >> >> They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic. >> >> The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. >> >>> since the fetcher is efficient and >>> will incrementally update something that is incomplete. >> >> Which won't work currently since kernel.org is down. >> >>> Anyhow, this is the first time someone has explained that the mirror >>> tarball improvements that were made to bitbake have not done everything >>> they desire. If you want someone on the Yocto team to look at this I'd >>> suggest a bug report or I'll take patches to bitbake (as always). >> >> But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev. > > FWIW: I would also like to have versioned git checkouts in fetcher2 as > it was in old fetcher. It would be more consitent with svn versioned > checkouts which are still created. http://patches.openembedded.org/patch/11971/ adds a global flag to generate such tarballs, but I don't know if the mirroring logic will actually try to fetch those or if fetch2 will actually try to unpack them when present, but at least we can generate the tarballs now :) regards, Koen ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 8:28 ` Koen Kooi 2011-09-22 8:45 ` Martin Jansa @ 2011-09-22 10:20 ` Richard Purdie 2011-09-22 11:27 ` Koen Kooi 1 sibling, 1 reply; 13+ messages in thread From: Richard Purdie @ 2011-09-22 10:20 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote: > Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven: > > On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: > >> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven: > >>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: > >>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven: > >>>> > >>>>> On 9/21/11 2:38 PM, Philip Balister wrote: > >>>>>> I'm finally trying to get started with oe-core. I'm using a setup > >>>>>> created with the setup-script created by Koen. Everything looks ok until > >>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source > >>>>>> from a git repo at kernel.org. Obviously, the fetch fails. > >>>>>> > >>>>>> What is odd, in the "good old days" I would expect this config to fall > >>>>>> back and use the Angstrom source mirror, but apparently this is not the > >>>>>> case with oe-core. > >>>>>> > >>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? > >>>>>> If it is a feature, how are we supposed to do builds if servers go away? > >>>>>> What do we do about GPL compliance? > >>>>> > >>>>> This is a feature, as people did not want oe-core tied to a given mirror. > >>>> > >>>> This is the "OE core doesn't generate versioned git tarballs" problem > >>>> people keep talking about. I know I can edit each and every recipe to > >>>> add ';rebasable=true', but I'd like a global solution before the GPL > >>>> police starts looking at the angstrom source mirrors. > >>> > >>> Is this a versioned tarball problem or just the tarballs not being > >>> generated at all? > >> > >> versioned tarball problem, since the source mirror won't overwrite > >> files with the same name, so unversioned ones are both useless and > >> bandwidth wasting. > > > > They seem to be working perfectly well for Yocto... > > > > They're also not bandwidth wasting > > They are, since for the kernel you need to re-upload the 300MB tarball > to the source mirror for mirroring to work, even if there were no real > changes but bitbake repacked it. I know that versioned tarballs aren't > terribly efficient either, but at least they are deterministic. Well, it depends on your perspective, I was looking at this from a user's point of view. The server's bandwidth is nothing compared to the users pulling the data. > The angstrom source mirror checks for duplicate filenames to avoid > introducing checksum changes due to upstream stupidity or corrupted > downloads. Fair enough but you could make an exception for the git scm tarballs (which follow a fairly identifiable pattern). > > since the fetcher is efficient and > > will incrementally update something that is incomplete. > > Which won't work currently since kernel.org is down. This just means there aren't appropriate git:// -> git:// mappings in people's mirrors files (including Yocto). If git:// -> git:// mappings don't work, we should fix that but they should. > > Anyhow, this is the first time someone has explained that the mirror > > tarball improvements that were made to bitbake have not done everything > > they desire. If you want someone on the Yocto team to look at this I'd > > suggest a bug report or I'll take patches to bitbake (as always). > > But you do acknowledge the problem? FWIW, I've been arguing for a > global method of enabled versioned tarballs since I first started > using oe-core (called differently back then). I know linux-yocto > stretches the fetcher to its limits, but I'd like this change for all > the recipes using a single git tree with a single srcrev. Now someone has explained the issue, yes, I can see why it is causing problems. I do think some of them are of people's own choosing but I'm trying to be flexible and not make judgement on that. So we can do something along the lines of what you after but there are the following considerations: a) The versioned tarballs do need to be different to what we had before in that the tarball will be of a .git directory containing the git metadata, not just a checkout. Why? This is so we can incrementally update a git repo if the user changes the source revision as one example. We're dealing with the git fetcher and limiting ourselves to a flat one dimension doesn't make sense. b) We need to agree what order the tarballs on the server should be searched for. The "obvious" order would be: 1. Specific Revision X tarball 2. generic mirror tarball 3. git clone The problems come about when we have an existing git checkout which doesn't contain the revision we want. Currently we can just update and assume that will bring in the revision. If we now support people doing rebases and other things, we now at least in theory have to: 1. See if the revision exists locally 2. Try pulling 3. If that doesn't work look for a versioned tarball 4. Blow away all fetcher git status from DL_DIR 5. Extract the tarball My big concern with this is rather than having some general archictecture and standard way of doing this, the fetcher once again becomes a set of special case if A do B but if C do D first then do E but don't forget to do A if Z happens type code. So what I'm saying is I don't want the fetcher to become the unmaintainable mess the that the code paths in fetch1 were. I don't honestly know how to add a code path like the above and avoid this :( The patch on the bitbake list doesn't consider any of this complexity or the corner cases :( Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 10:20 ` Richard Purdie @ 2011-09-22 11:27 ` Koen Kooi 2011-09-22 11:42 ` Richard Purdie 0 siblings, 1 reply; 13+ messages in thread From: Koen Kooi @ 2011-09-22 11:27 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven: > On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote: >> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende >> geschreven: >>> On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote: >>>> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende >>>> geschreven: >>>>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote: >>>>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende >>>>>> geschreven: >>>>>> >>>>>>> On 9/21/11 2:38 PM, Philip Balister wrote: >>>>>>>> I'm finally trying to get started with oe-core. I'm using a >>>>>>>> setup >>>>>>>> created with the setup-script created by Koen. Everything >>>>>>>> looks ok until >>>>>>>> it tried to build cpufrequtils. The recipe tries to fetch the >>>>>>>> source >>>>>>>> from a git repo at kernel.org. Obviously, the fetch fails. >>>>>>>> >>>>>>>> What is odd, in the "good old days" I would expect this >>>>>>>> config to fall >>>>>>>> back and use the Angstrom source mirror, but apparently this >>>>>>>> is not the >>>>>>>> case with oe-core. >>>>>>>> >>>>>>>> Is this a bug or a feature? If this is a bug, is there a plan >>>>>>>> to fix it? >>>>>>>> If it is a feature, how are we supposed to do builds if >>>>>>>> servers go away? >>>>>>>> What do we do about GPL compliance? >>>>>>> >>>>>>> This is a feature, as people did not want oe-core tied to a >>>>>>> given mirror. >>>>>> >>>>>> This is the "OE core doesn't generate versioned git tarballs" >>>>>> problem >>>>>> people keep talking about. I know I can edit each and every >>>>>> recipe to >>>>>> add ';rebasable=true', but I'd like a global solution before >>>>>> the GPL >>>>>> police starts looking at the angstrom source mirrors. >>>>> >>>>> Is this a versioned tarball problem or just the tarballs not being >>>>> generated at all? >>>> >>>> versioned tarball problem, since the source mirror won't overwrite >>>> files with the same name, so unversioned ones are both useless and >>>> bandwidth wasting. >>> >>> They seem to be working perfectly well for Yocto... >>> >>> They're also not bandwidth wasting >> >> They are, since for the kernel you need to re-upload the 300MB >> tarball >> to the source mirror for mirroring to work, even if there were no >> real >> changes but bitbake repacked it. I know that versioned tarballs >> aren't >> terribly efficient either, but at least they are deterministic. > > Well, it depends on your perspective, I was looking at this from a > user's point of view. The server's bandwidth is nothing compared to > the > users pulling the data. > >> The angstrom source mirror checks for duplicate filenames to avoid >> introducing checksum changes due to upstream stupidity or corrupted >> downloads. > > Fair enough but you could make an exception for the git scm tarballs > (which follow a fairly identifiable pattern). That still opens the door for people to upload "old" tarballs with less revs in them. No solution is perfect, though. >>> since the fetcher is efficient and >>> will incrementally update something that is incomplete. >> >> Which won't work currently since kernel.org is down. > > This just means there aren't appropriate git:// -> git:// mappings in > people's mirrors files (including Yocto). If git:// -> git:// mappings > don't work, we should fix that but they should. You mean something like git.kernel.org/scm/torvalds/linux-2.6.git -> github.com/torvalds/linux.git ? >>> Anyhow, this is the first time someone has explained that the mirror >>> tarball improvements that were made to bitbake have not done >>> everything >>> they desire. If you want someone on the Yocto team to look at this >>> I'd >>> suggest a bug report or I'll take patches to bitbake (as always). >> >> But you do acknowledge the problem? FWIW, I've been arguing for a >> global method of enabled versioned tarballs since I first started >> using oe-core (called differently back then). I know linux-yocto >> stretches the fetcher to its limits, but I'd like this change for all >> the recipes using a single git tree with a single srcrev. > > Now someone has explained the issue, yes, I can see why it is causing > problems. I do think some of them are of people's own choosing but I'm > trying to be flexible and not make judgement on that. > > So we can do something along the lines of what you after but there are > the following considerations: > > a) The versioned tarballs do need to be different to what we had > before > in that the tarball will be of a .git directory containing the git > metadata, not just a checkout. Why? This is so we can incrementally > update a git repo if the user changes the source revision as one > example. We're dealing with the git fetcher and limiting ourselves > to a > flat one dimension doesn't make sense. Agreed on that. The bitbake patch I sent does at least do that :) > b) We need to agree what order the tarballs on the server should be > searched for. The "obvious" order would be: > > 1. Specific Revision X tarball > 2. generic mirror tarball > 3. git clone > > The problems come about when we have an existing git checkout which > doesn't contain the revision we want. There currently is a bug in the fetcher that makes it hard to use now that kernel.org is down. The situation: I have a recipe that pulls from git (e.g. cpufrequtils.git from kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - c fetch -f it will try to do an update and fail since kernel.org is down. I would have expected it to check the existing bare clone for deadbeef first and not try updating if it is present. Anyway, I agree on the search order above. > Currently we can just update and > assume that will bring in the revision. If we now support people doing > rebases and other things, we now at least in theory have to: > > 1. See if the revision exists locally > 2. Try pulling > 3. If that doesn't work look for a versioned tarball > 4. Blow away all fetcher git status from DL_DIR > 5. Extract the tarball Since git supports local fetches the above can be: 1) see if rev exists 2) try pulling 3) try versioned tarball 4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo That would stop things from being blown away. It does allow the versioned snapshots to differ, but they will always have the rev they were versioned with. > My big concern with this is rather than having some general > archictecture and standard way of doing this, the fetcher once again > becomes a set of special case if A do B but if C do D first then do E > but don't forget to do A if Z happens type code. I know virtually nothing of the fetcher code, but I think treating every tarball as a local mirror that gets fetched into DL_DIR/git2/ repodir would avoid a lot of special casing. > So what I'm saying is I don't want the fetcher to become the > unmaintainable mess the that the code paths in fetch1 were. I don't > honestly know how to add a code path like the above and avoid this :( > > The patch on the bitbake list doesn't consider any of this > complexity or > the corner cases Sadly it's the best thing I can accomplish with my python skills :( It does solve my #1 need, which is an easy way (ls | grep) to check for GPL compliance. regards, Koen ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 11:27 ` Koen Kooi @ 2011-09-22 11:42 ` Richard Purdie 2011-09-22 12:13 ` Koen Kooi 0 siblings, 1 reply; 13+ messages in thread From: Richard Purdie @ 2011-09-22 11:42 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Thu, 2011-09-22 at 13:27 +0200, Koen Kooi wrote: > Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven: > > > On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote: > >> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende > > This just means there aren't appropriate git:// -> git:// mappings in > > people's mirrors files (including Yocto). If git:// -> git:// mappings > > don't work, we should fix that but they should. > > You mean something like git.kernel.org/scm/torvalds/linux-2.6.git -> > github.com/torvalds/linux.git ? Yes. Thinking about this, the current fetcher might not be optimised in making this as efficient as possible but it should work and if it doesn't, we should make it. > > Now someone has explained the issue, yes, I can see why it is causing > > problems. I do think some of them are of people's own choosing but I'm > > trying to be flexible and not make judgement on that. > > > > So we can do something along the lines of what you after but there are > > the following considerations: > > > > a) The versioned tarballs do need to be different to what we had > > before > > in that the tarball will be of a .git directory containing the git > > metadata, not just a checkout. Why? This is so we can incrementally > > update a git repo if the user changes the source revision as one > > example. We're dealing with the git fetcher and limiting ourselves > > to a > > flat one dimension doesn't make sense. > > Agreed on that. The bitbake patch I sent does at least do that :) Right :) I just want to be clear we're doing something different to what was there before (and why). > > b) We need to agree what order the tarballs on the server should be > > searched for. The "obvious" order would be: > > > > 1. Specific Revision X tarball > > 2. generic mirror tarball > > 3. git clone > > > > The problems come about when we have an existing git checkout which > > doesn't contain the revision we want. > > There currently is a bug in the fetcher that makes it hard to use now > that kernel.org is down. The situation: > > I have a recipe that pulls from git (e.g. cpufrequtils.git from > kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ > git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - > c fetch -f it will try to do an update and fail since kernel.org is > down. I would have expected it to check the existing bare clone for > deadbeef first and not try updating if it is present. Agreed, it shouldn't be hitting the network unless the revision is floating. That is bad :/. > Anyway, I agree on the search order above. > > > Currently we can just update and > > assume that will bring in the revision. If we now support people doing > > rebases and other things, we now at least in theory have to: > > > > 1. See if the revision exists locally > > 2. Try pulling > > 3. If that doesn't work look for a versioned tarball > > 4. Blow away all fetcher git status from DL_DIR > > 5. Extract the tarball > > Since git supports local fetches the above can be: > > 1) see if rev exists > 2) try pulling > 3) try versioned tarball > 4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo > > That would stop things from being blown away. It does allow the > versioned snapshots to differ, but they will always have the rev they > were versioned with. This is a much better approach. Its quite a big difference from anything we've done before but the approach should fit in better. I need to think on it more... > > My big concern with this is rather than having some general > > archictecture and standard way of doing this, the fetcher once again > > becomes a set of special case if A do B but if C do D first then do E > > but don't forget to do A if Z happens type code. > > I know virtually nothing of the fetcher code, but I think treating > every tarball as a local mirror that gets fetched into DL_DIR/git2/ > repodir would avoid a lot of special casing. Yes, its nicer. We keep falling into the trap of thinking about the non-distributed SCM cases but we have more capability with git and it makes sense to use it. > > So what I'm saying is I don't want the fetcher to become the > > unmaintainable mess the that the code paths in fetch1 were. I don't > > honestly know how to add a code path like the above and avoid this :( > > > > The patch on the bitbake list doesn't consider any of this > > complexity or > > the corner cases > > Sadly it's the best thing I can accomplish with my python skills :( It > does solve my #1 need, which is an easy way (ls | grep) to check for > GPL compliance. Ok. Can you at least file a bug in the yocto bugzilla with a summary of this discussion please? :) That should ensure the issue gets more attention since with the best intentions, I can't remember everything. Cheers, Richard ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fetch failure for source at kernel.org 2011-09-22 11:42 ` Richard Purdie @ 2011-09-22 12:13 ` Koen Kooi 0 siblings, 0 replies; 13+ messages in thread From: Koen Kooi @ 2011-09-22 12:13 UTC (permalink / raw) To: Patches and discussions about the oe-core layer Op 22 sep. 2011, om 13:42 heeft Richard Purdie het volgende geschreven: [..] > Ok. Can you at least file a bug in the yocto bugzilla with a summary of > this discussion please? :) That should ensure the issue gets more > attention since with the best intentions, I can't remember everything. I filed it as http://bugzilla.pokylinux.org/show_bug.cgi?id=1511 ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-09-22 12:19 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-09-21 19:38 Fetch failure for source at kernel.org Philip Balister 2011-09-21 19:43 ` Mark Hatle 2011-09-21 19:49 ` Koen Kooi 2011-09-22 0:26 ` Richard Purdie 2011-09-22 6:40 ` Koen Kooi 2011-09-22 7:03 ` Richard Purdie 2011-09-22 8:28 ` Koen Kooi 2011-09-22 8:45 ` Martin Jansa 2011-09-22 9:01 ` Koen Kooi 2011-09-22 10:20 ` Richard Purdie 2011-09-22 11:27 ` Koen Kooi 2011-09-22 11:42 ` Richard Purdie 2011-09-22 12:13 ` Koen Kooi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.