Re: [PATCH 7/8] Fetcher: break the "SRCREVINACTION" deadlock

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2011-01-06 at 11:06 +0000, Richard Purdie wrote:
> On Mon, 2010-12-27 at 21:58 +0800, Yu Ke wrote:
> > Current fetcher has annoying "SRCREVINACTION" deadlock,
> > which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
> > get_srcrev()->setup_localpath()->srcrev_internal_helper()
> > ->evaluate SRCREV->get_srcrev()
> > 
> > current fetcher resolve the deadlock by introducing a
> > "SRCREVINACTION" condition check. Althoguh it works, it is
> > indeed not clean.
> > 
> > This patch use antoehr idea to break the deadlock: break
> > the dependency among SRCREV and get_srcrev(), i.e. assign
> > a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
> > this keyword, it will check and set the latest revision to
> > urldata.revision. get_srcrev later can use the urldata.revision
> > for value evaluation(SRCPV etc). In this case, SRCREV no longer
> > depends on get_srcrev, and there is not deadlock anymore.
> > 
> > In implementation side, several things are done:
> > a) set the revision in FetchData:__init__, so that
> >    urldata.revision is ready when get_srcrev() need it
> > b) init fetch method specific data in Fetch:supports, so that
> >    Fetch.latest_revision() can be conducted when FetchData:
> >    __init__ need it to achieve a)
> 
> I do think we need to make this change. 
> 
> Since we're not doing this in a fetcher "version 2" currently there is
> the backwards compatibility question though. What does this code do if
> the bitbake.conf change isn't made? Can we detect the old situation and
> error at least? This change is going to need a little discussion on the
> bitbake-dev list...
> 
> The answer may be that we need to start a fetch2 codebase to carry on
> with this kind of cleanup.

I've given this further thought and lets go with the original plan,
create a fetch2 and then go and really clean it up.

We therefore need to copy the fetch directory to fetch2, then start
applying patches against fetch2. We can update poky to use bb.fetch2
instead of bb.fetch. This should also give OE a way to transition too.

I made a quick grep over bitbake itself and there are some fetch
references we'll need to resolve to switch between the two fetch
modules:

lib/bb/cooker.py:        bb.fetch.persistent_database_connection = {}
lib/bb/cooker.py:        bb.fetch.persistent_database_connection = {}

The above two references no longer do anything as the
persistent_database code was removed in bitbake upstream and for now
poky has followed that although I would like to revisit it.

lib/bb/parse/parse_py/BBHandler.py:import bb.fetch, bb.build, bb.utils

I doubt BBHandler still needs this reference

lib/bb/cooker.py:            bb.fetch.fetcher_init(self.configuration.data)
lib/bb/command.py:        if bb.fetch.fetcher_compare_revisions(command.cooker.configuration.data):

These two are important and we'll need to have a way to switch between
the fetchers for these.

Cheers,

Richard