Bitbake fetchers

Bitbake fetchers

[Richard Purdie]

Hi,

I've talked with various people about the bitbake fetchers and I think
the time has come for an overhaul. I'm going to take the opportunity to
write down all the various issues I've seen with the current approach so
we can then come up with a plan and some changes to address this.

Some things that currently bother me:

a) For a git checkout, we clone the repository, make a checkout, tar
this up, then do_unpack task untars it. This is inefficient to say the
least.

b) For git controlled sources, we "lose" the .git directory in the
workdir. For any git checkout, I'd like this to be available.

c) Even when git recipes have the SRCREV specified to a fixed value, the
recipe always gets reparsed as there is no way to tell if its locked
down or could float (with AUTOREV). We shouldn't be reparsing.

d) There is no way to set two SRCREV values for different branches of
the same git repository without two entries in SRC_URI. Knowing when to
update a multi revision bare clone git repository is hard with the
information spread over two SRC_URI entries.

e) Its hard to configure bitbake to be "networkless", or to turn off the
default SRC_URI entries and force bitbake to only use a mirror, or a
local directory.

f) The whole of the fetcher code has grown orgnaically and doesn't have
an overall design or a sensible API for accessing it.

g) SRCREV has to be set in the configuration space, not in recipe space.
We should support it in recipe space but this will mean caching values
and providing them to the main configuration space. This is a bitbake
parsing/data caching issue rather than a fetcher one.

h) The error handling and propagation of errors appears to have issues
in places.

i) Its hard to enable/disable SCM mirror tarballs at present (this
should be optional as they'd really become unneeded).

From a design standpoint I'd therefore like to create a "fetch2"
directory in bitbake and try and redesign the fetchers there with a
sensible API learning from the codebase we already have. In places this
will no doubt use the same code but I'd like to take a step back and
restructure it.

To address some of the problems above, I'd like to see the do_unpack
task that Poky/OE have, call into the fetcher rather than have the code
in Poky/OE clases. This means the likes of the git checkouts can be
optimised and a checkout into WORKDIR can just be a git clone by
reference (see man git-clone). This probably needs to be through
symlinks and not hardlinks as DL_DIR and WORKDIR can be on different
filesystems.

There are some complexities in the fetcher code:

a) There is the issue of caching, we try and cache the parsed SRC_URI
rather than reparse it multiple times.

b) SRCREV = "${AUTOREV}" introduces a lot of complexity as it has to
inject the git revision into the PV variable which is widely used. This
is partly the reason a) above is needed. The value for PV needs to be
computed at parse time, not task execution, further complicating the
model.

c) Sometimes, we need to "fetch" different data so we change the values
of SRC_URI, the mirrors and DL_DIR. These things therefore need to be
configurable. An example user is the sstate fetcher code. Another
example is the checkuri task. It would be nice if for a given recipe,
the fetcher could give feedback about whether the main SRC_URI works
with no mirror and which mirrors had the file availability.

These complexities need to be addressed in the rewrite to ensure
features still work.

I'm sure there are other things I'm missing on these lists but hopefully
this is a good start at documenting them. If anyone has further issues
to add, I'd be interested to hear about them.

Cheers,

Richard

Re: Bitbake fetchers

[Mark Hatle]

I was talking to someone a few weeks ago about the SRC_URI entries.  They were 
wondering if the Apache Maven stuff had been looked at:

http://maven.apache.org/scm/scms-overview.html

basically it's an attempt at standardizing the scm URIs.  It's certainly 
different then the current bitbake methods, so it's unclear to me if this would 
be at all useful for the work you are suggesting.. but it might be work 
exploring as an alternative way to secify the scm info.

--Mark

On 11/8/10 9:38 PM, Richard Purdie wrote:
> Hi,
>
> I've talked with various people about the bitbake fetchers and I think
> the time has come for an overhaul. I'm going to take the opportunity to
> write down all the various issues I've seen with the current approach so
> we can then come up with a plan and some changes to address this.
>
> Some things that currently bother me:
>
> a) For a git checkout, we clone the repository, make a checkout, tar
> this up, then do_unpack task untars it. This is inefficient to say the
> least.
>
> b) For git controlled sources, we "lose" the .git directory in the
> workdir. For any git checkout, I'd like this to be available.
>
> c) Even when git recipes have the SRCREV specified to a fixed value, the
> recipe always gets reparsed as there is no way to tell if its locked
> down or could float (with AUTOREV). We shouldn't be reparsing.
>
> d) There is no way to set two SRCREV values for different branches of
> the same git repository without two entries in SRC_URI. Knowing when to
> update a multi revision bare clone git repository is hard with the
> information spread over two SRC_URI entries.
>
> e) Its hard to configure bitbake to be "networkless", or to turn off the
> default SRC_URI entries and force bitbake to only use a mirror, or a
> local directory.
>
> f) The whole of the fetcher code has grown orgnaically and doesn't have
> an overall design or a sensible API for accessing it.
>
> g) SRCREV has to be set in the configuration space, not in recipe space.
> We should support it in recipe space but this will mean caching values
> and providing them to the main configuration space. This is a bitbake
> parsing/data caching issue rather than a fetcher one.
>
> h) The error handling and propagation of errors appears to have issues
> in places.
>
> i) Its hard to enable/disable SCM mirror tarballs at present (this
> should be optional as they'd really become unneeded).
>
>> From a design standpoint I'd therefore like to create a "fetch2"
> directory in bitbake and try and redesign the fetchers there with a
> sensible API learning from the codebase we already have. In places this
> will no doubt use the same code but I'd like to take a step back and
> restructure it.
>
> To address some of the problems above, I'd like to see the do_unpack
> task that Poky/OE have, call into the fetcher rather than have the code
> in Poky/OE clases. This means the likes of the git checkouts can be
> optimised and a checkout into WORKDIR can just be a git clone by
> reference (see man git-clone). This probably needs to be through
> symlinks and not hardlinks as DL_DIR and WORKDIR can be on different
> filesystems.
>
> There are some complexities in the fetcher code:
>
> a) There is the issue of caching, we try and cache the parsed SRC_URI
> rather than reparse it multiple times.
>
> b) SRCREV = "${AUTOREV}" introduces a lot of complexity as it has to
> inject the git revision into the PV variable which is widely used. This
> is partly the reason a) above is needed. The value for PV needs to be
> computed at parse time, not task execution, further complicating the
> model.
>
> c) Sometimes, we need to "fetch" different data so we change the values
> of SRC_URI, the mirrors and DL_DIR. These things therefore need to be
> configurable. An example user is the sstate fetcher code. Another
> example is the checkuri task. It would be nice if for a given recipe,
> the fetcher could give feedback about whether the main SRC_URI works
> with no mirror and which mirrors had the file availability.
>
> These complexities need to be addressed in the rewrite to ensure
> features still work.
>
> I'm sure there are other things I'm missing on these lists but hopefully
> this is a good start at documenting them. If anyone has further issues
> to add, I'd be interested to hear about them.
>
> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Bitbake fetchers

[Frans Meulenbroeks]

2010/11/9 Richard Purdie <rpurdie@linux.intel.com>:
> Hi,
>
> I've talked with various people about the bitbake fetchers and I think
> the time has come for an overhaul. I'm going to take the opportunity to
> write down all the various issues I've seen with the current approach so
> we can then come up with a plan and some changes to address this.
>
> Some things that currently bother me:
[...]

Interesting ideas & plans Richard!

What I miss is the possibility for a fetcher or a specific SRC_URI
entry to disable storing the results in downloads.
This typically seems to be an issue wrt cvs, where a change in the cvs
repo is not detected.
(or I am simply doing something wrong).

When rewriting perhaps adding a disable-save-local-copy flag would be nice.

Frans