[PATCH] fetch2: try PREMIRRORS when calculating latest_revision

[PATCH] fetch2: try PREMIRRORS when calculating latest_revision

[Enrico Scholz]

Since support for native git mirrors was added in recent bitbake, it
is possible to use local mirrors for calculating ${AUTOREV}.  This is
very useful for development: recipes SRCREV can be overridden locally
by 'SRCREV_pn-<recipe> = ${AUTOREV}' and a PREMIRROR which refers to a
local repository can be added for the SRC_URI .

Patch checks PREMIRRORS only but not normal MIRRORS because:

a) it can break things when upstream is temporarily unavailable while
   calling latest_revision() (--> MIRRORS will be consulted) but comes
   back when fetching source.

b) it makes the error reporting more complicated (only errors for
   upstream should be shown to user, not the MIRROR's ones).

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 lib/bb/fetch2/__init__.py |   25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index e3e03e5..2da5e99 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -616,6 +616,29 @@ def try_mirrors(d, origud, mirrors, check = False):
             return ret
     return None
 
+def latest_revision(ud, d, name):
+    mirrors = mirror_from_string(d.getVar('PREMIRRORS', True))
+    ld = d.createCopy()
+    rev = None
+    uris, uds = build_mirroruris(ud, mirrors, ld)
+
+    for index, uri in enumerate(uris):
+        if not hasattr(uds[index].method, '_latest_revision'):
+            continue
+
+        try:
+            rev = uds[index].method.latest_revision(uri, uds[index], d, name)
+            break
+        except bb.fetch2.NetworkAccess:
+            pass
+        except bb.fetch2.BBFetchException as e:
+            pass
+
+    if rev == None:
+        rev = ud.method.latest_revision(ud.url, ud, d, name)
+
+    return rev
+
 def srcrev_internal_helper(ud, d, name):
     """
     Return:
@@ -643,7 +666,7 @@ def srcrev_internal_helper(ud, d, name):
     if rev == "INVALID":
         raise FetchError("Please set SRCREV to a valid value", ud.url)
     if rev == "AUTOINC":
-        rev = ud.method.latest_revision(ud.url, ud, d, name)
+        rev = latest_revision(ud, d, name)
 
     return rev
 
-- 
1.7.10.2

Re: [PATCH] fetch2: try PREMIRRORS when calculating latest_revision

[Richard Purdie]

On Sun, 2012-07-01 at 11:52 +0200, Enrico Scholz wrote:
> Since support for native git mirrors was added in recent bitbake, it
> is possible to use local mirrors for calculating ${AUTOREV}.  This is
> very useful for development: recipes SRCREV can be overridden locally
> by 'SRCREV_pn-<recipe> = ${AUTOREV}' and a PREMIRROR which refers to a
> local repository can be added for the SRC_URI .
> 
> Patch checks PREMIRRORS only but not normal MIRRORS because:
> 
> a) it can break things when upstream is temporarily unavailable while
>    calling latest_revision() (--> MIRRORS will be consulted) but comes
>    back when fetching source.
> 
> b) it makes the error reporting more complicated (only errors for
>    upstream should be shown to user, not the MIRROR's ones).
> 
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>  lib/bb/fetch2/__init__.py |   25 ++++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)

I'm torn over this one. If you want to refer to local changes wouldn't
it make more sense just to totally override SRC_URI in cases like this,
then things are specific.

I'm worried that here, upstream revisions may just get ignored for
example with just a premirror copy being consulted and this wouldn't be
the expected behaviour a user would want.

Cheers,

Richard

Re: [PATCH] fetch2: try PREMIRRORS when calculating latest_revision

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> Since support for native git mirrors was added in recent bitbake, it
>> is possible to use local mirrors for calculating ${AUTOREV}.
>
> I'm torn over this one. If you want to refer to local changes wouldn't
> it make more sense just to totally override SRC_URI in cases

Modifying SRC_URI is a pain (split it into components, find the matching
one and replace it with the new one; managing the .bbappend file with
this code).  A declarative syntax like for PREMIRRORS is much easier too
handle and maintain.

Before native git mirrors were supported I solved it by 'url.<new>.insteadOf'
config option. But, this requires either a very recent git to set the '-c'
option in FETCHCMD_git, or the modification of global ~/.gitconfig (which
will cause conflicts and bad surprises when having multiple projects).


> I'm worried that here, upstream revisions may just get ignored for
> example with just a premirror copy being consulted and this wouldn't
> be the expected behaviour a user would want.

That's already the case with recent PREMIRRORs.  People will get the
version which is in PREMIRROR, not the upstream one.

Difference is that after applying my patch, the early recipe-parsing
will not fail when upstream unavailable.


Enrico