[PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[fabien.proriolpatch]

From: Fabien Proriol <fabien.proriol@jdsu.com>

This patch add the possibility to bitbake to avoid git protocol to fetch sources.
This is usefull in some network with firewall blocking git port.

When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is used to find the new protocol (http or https) and the new host (if different) to used.

BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted. This is usefull for exemple, if we use local git repository inside the network.

Exemple usage:
    # Avoid all git protocol
    BB_GIT_PROTOCOL_FIREWALL = "1"

    # Avoid git protocol, except for srv1 and srv2 in local network
    BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"

Signed-off-by: Fabien Proriol <fabien.proriol@jdsu.com>
---
 bitbake/lib/bb/fetch2/git.py | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 526668b..27dde60 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -76,6 +76,20 @@ from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
 
+PROTOCOL_MIRRORS = {
+    "sourceware.org": {"protocol":"http"},
+    "github.com": {"protocol":"https"},
+    "git.sv.gnu.org": {"protocol":"https", "host":"git.savannah.gnu.org/r"},
+    "anongit.freedesktop.org": {"protocol":"http"},
+    "anonscm.debian.org": {"protocol":"https", "host":"anonscm.debian.org/git"},
+    "git.gnome.org": {"protocol":"https", "host":"git.gnome.org/browse"},
+    "git.yoctoproject.org": {"protocol":"http", "host":"git.yoctoproject.org/git"},
+    "git.kernel.org": {"protocol":"https"},
+    "git.denx.de": {"protocol":"http"},
+    "git.lttng.org": {"protocol":"http"},
+    "git.infradead.org":  {"protocol":"https", "host":"github.com/eva-oss"},
+}
+
 class Git(FetchMethod):
     """Class to fetch a module or modules from git repositories"""
     def init(self, d):
@@ -102,6 +116,12 @@ class Git(FetchMethod):
         else:
             ud.proto = "git"
 
+        firewall = d.getVar("BB_GIT_PROTOCOL_FIREWALL", True)
+        if firewall:
+            self.git_firewall = firewall.split(";")
+        else:
+            self.git_firewall = None
+
         if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
             raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
 
@@ -315,6 +335,16 @@ class Git(FetchMethod):
             username = ud.user + '@'
         else:
             username = ""
+
+        if (ud.proto == "git") and (self.git_firewall):
+            if not ud.host in self.git_firewall:
+                if ud.host.strip() in PROTOCOL_MIRRORS.keys():
+                    ud.proto = PROTOCOL_MIRRORS[ud.host]["protocol"]
+                    if "host" in PROTOCOL_MIRRORS[ud.host].keys():
+                        ud.host = PROTOCOL_MIRRORS[ud.host]["host"]
+                else:
+                    raise bb.fetch2.FetchError("Unknown protocol mirror for %s (%s)" % (ud.host.strip(), ud.path))
+
         return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path)
 
     def _revision_key(self, ud, d, name):
-- 
2.7.3

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Olof Johansson]

On 16-05-31 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
> From: Fabien Proriol <fabien.proriol@jdsu.com>
> 
> This patch add the possibility to bitbake to avoid git protocol to fetch sources.
> This is usefull in some network with firewall blocking git port.
> 
> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is used to find the new protocol (http or https) and the new host (if different) to used.
> 
> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted. This is usefull for exemple, if we use local git repository inside the network.

Can't you use PREMIRRORS instead? Something along the lines:

 PREMIRRORS = "git://.*/.* https://mirrorhost.example.com/"

There's even a variable, BB_FETCH_PREMIRRORONLY, for when the
original host shouldn't be contacted at all.

-- 
olofjn

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Richard Purdie]

On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
> From: Fabien Proriol <fabien.proriol@jdsu.com>
> 
> This patch add the possibility to bitbake to avoid git protocol to
> fetch sources.
> This is usefull in some network with firewall blocking git port.
> 
> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
> used to find the new protocol (http or https) and the new host (if
> different) to used.
> 
> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
> This is usefull for exemple, if we use local git repository inside
> the network.
> 
> Exemple usage:
>     # Avoid all git protocol
>     BB_GIT_PROTOCOL_FIREWALL = "1"
> 
>     # Avoid git protocol, except for srv1 and srv2 in local network
>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"

Why not use the more generic MIRRORS and PREMIRRORS for this? If they
can't support this, we likely should figure out a way to make them work
for it rather than adding new fetcher specific variables.

Cheers,

Richard

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Andre McCurdy]

On Tue, May 31, 2016 at 6:50 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
>> From: Fabien Proriol <fabien.proriol@jdsu.com>
>>
>> This patch add the possibility to bitbake to avoid git protocol to
>> fetch sources.
>> This is usefull in some network with firewall blocking git port.
>>
>> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
>> used to find the new protocol (http or https) and the new host (if
>> different) to used.
>>
>> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
>> This is usefull for exemple, if we use local git repository inside
>> the network.
>>
>> Exemple usage:
>>     # Avoid all git protocol
>>     BB_GIT_PROTOCOL_FIREWALL = "1"
>>
>>     # Avoid git protocol, except for srv1 and srv2 in local network
>>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"
>
> Why not use the more generic MIRRORS and PREMIRRORS for this? If they
> can't support this, we likely should figure out a way to make them work
> for it rather than adding new fetcher specific variables.

Sorry for resurrecting such an old thread, but I'm wondering if
there's been any more work on an upstreamable solution?

The problem being that recipes which define SRC_URI to git://* with
protocol=git (or with protocol unspecified, since git is the default)
fail during fetching for users working behind a firewall which blocks
the native git protocol.

Although MIRRORS and PREMIRRORS allow mapping from one host to another
there's no support for mapping one git fetch protocol to another and
it looks like extending the MIRRORS syntax to support that would be a
non-trivial change...

> Cheers,
>
> Richard
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Mark Hatle]

On 7/24/17 4:04 PM, Andre McCurdy wrote:
> On Tue, May 31, 2016 at 6:50 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
>>> From: Fabien Proriol <fabien.proriol@jdsu.com>
>>>
>>> This patch add the possibility to bitbake to avoid git protocol to
>>> fetch sources.
>>> This is usefull in some network with firewall blocking git port.
>>>
>>> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
>>> used to find the new protocol (http or https) and the new host (if
>>> different) to used.
>>>
>>> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
>>> This is usefull for exemple, if we use local git repository inside
>>> the network.
>>>
>>> Exemple usage:
>>>     # Avoid all git protocol
>>>     BB_GIT_PROTOCOL_FIREWALL = "1"
>>>
>>>     # Avoid git protocol, except for srv1 and srv2 in local network
>>>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"
>>
>> Why not use the more generic MIRRORS and PREMIRRORS for this? If they
>> can't support this, we likely should figure out a way to make them work
>> for it rather than adding new fetcher specific variables.
> 
> Sorry for resurrecting such an old thread, but I'm wondering if
> there's been any more work on an upstreamable solution?
> 
> The problem being that recipes which define SRC_URI to git://* with
> protocol=git (or with protocol unspecified, since git is the default)
> fail during fetching for users working behind a firewall which blocks
> the native git protocol.
> 
> Although MIRRORS and PREMIRRORS allow mapping from one host to another
> there's no support for mapping one git fetch protocol to another and
> it looks like extending the MIRRORS syntax to support that would be a
> non-trivial change...

Won't the following work:

PREMIRRORS_append = " \
     git://.*/.* git://HOST/PATH;protocol=https \n \
     git://.*/.* git://HOST/PATH;protocol=http \n \
"

Note: I've not actually tried the above, but I expect it will work.  We do
something similar to provide local (on-disk) git caches using:

git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file

--Mark

>> Cheers,
>>
>> Richard
>> --
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Andre McCurdy]

On Tue, Jul 25, 2017 at 7:43 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 7/24/17 4:04 PM, Andre McCurdy wrote:
>> On Tue, May 31, 2016 at 6:50 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>> On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
>>>> From: Fabien Proriol <fabien.proriol@jdsu.com>
>>>>
>>>> This patch add the possibility to bitbake to avoid git protocol to
>>>> fetch sources.
>>>> This is usefull in some network with firewall blocking git port.
>>>>
>>>> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
>>>> used to find the new protocol (http or https) and the new host (if
>>>> different) to used.
>>>>
>>>> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
>>>> This is usefull for exemple, if we use local git repository inside
>>>> the network.
>>>>
>>>> Exemple usage:
>>>>     # Avoid all git protocol
>>>>     BB_GIT_PROTOCOL_FIREWALL = "1"
>>>>
>>>>     # Avoid git protocol, except for srv1 and srv2 in local network
>>>>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"
>>>
>>> Why not use the more generic MIRRORS and PREMIRRORS for this? If they
>>> can't support this, we likely should figure out a way to make them work
>>> for it rather than adding new fetcher specific variables.
>>
>> Sorry for resurrecting such an old thread, but I'm wondering if
>> there's been any more work on an upstreamable solution?
>>
>> The problem being that recipes which define SRC_URI to git://* with
>> protocol=git (or with protocol unspecified, since git is the default)
>> fail during fetching for users working behind a firewall which blocks
>> the native git protocol.
>>
>> Although MIRRORS and PREMIRRORS allow mapping from one host to another
>> there's no support for mapping one git fetch protocol to another and
>> it looks like extending the MIRRORS syntax to support that would be a
>> non-trivial change...
>
> Won't the following work:
>
> PREMIRRORS_append = " \
>      git://.*/.* git://HOST/PATH;protocol=https \n \
>      git://.*/.* git://HOST/PATH;protocol=http \n \
> "

Thanks! Yes, that does work. It's the first time I've ever seen HOST
or PATH being used as mirror replacement patterns. Useful to know.

Would a patch adding something similar to the default MIRRORS (ie try
http[s] as a fallback if the git protocol fails) be acceptable in
upstream oe-core?

> Note: I've not actually tried the above, but I expect it will work.  We do
> something similar to provide local (on-disk) git caches using:
>
> git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file
>
> --Mark
>
>>> Cheers,
>>>
>>> Richard
>>> --
>>> _______________________________________________
>>> bitbake-devel mailing list
>>> bitbake-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Richard Purdie]

On Tue, 2017-07-25 at 12:30 -0700, Andre McCurdy wrote:
> On Tue, Jul 25, 2017 at 7:43 AM, Mark Hatle <mark.hatle@windriver.com
> > wrote:
> > Won't the following work:
> > 
> > PREMIRRORS_append = " \
> >      git://.*/.* git://HOST/PATH;protocol=https \n \
> >      git://.*/.* git://HOST/PATH;protocol=http \n \
> > "
> Thanks! Yes, that does work. It's the first time I've ever seen HOST
> or PATH being used as mirror replacement patterns. Useful to know.
> 
> Would a patch adding something similar to the default MIRRORS (ie try
> http[s] as a fallback if the git protocol fails) be acceptable in
> upstream oe-core?

Yes, I'd take such a patch. I'd love to have this better documented.

Cheers,

Richard

Re: [PATCH] bb.fetch.git: add a way to avoid git protocol, and force http or https mirrors

[Mark Hatle]

On 7/25/17 2:30 PM, Andre McCurdy wrote:
> On Tue, Jul 25, 2017 at 7:43 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> On 7/24/17 4:04 PM, Andre McCurdy wrote:
>>> On Tue, May 31, 2016 at 6:50 AM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>>> On Tue, 2016-05-31 at 15:18 +0200, fabien.proriolpatch@kazoe.org wrote:
>>>>> From: Fabien Proriol <fabien.proriol@jdsu.com>
>>>>>
>>>>> This patch add the possibility to bitbake to avoid git protocol to
>>>>> fetch sources.
>>>>> This is usefull in some network with firewall blocking git port.
>>>>>
>>>>> When BB_GIT_PROTOCOL_FIREWALL is set, the PROTOCOL_MIRRORS table is
>>>>> used to find the new protocol (http or https) and the new host (if
>>>>> different) to used.
>>>>>
>>>>> BB_GIT_PROTOCOL_FIREWALL can also contains a list of host accepted.
>>>>> This is usefull for exemple, if we use local git repository inside
>>>>> the network.
>>>>>
>>>>> Exemple usage:
>>>>>     # Avoid all git protocol
>>>>>     BB_GIT_PROTOCOL_FIREWALL = "1"
>>>>>
>>>>>     # Avoid git protocol, except for srv1 and srv2 in local network
>>>>>     BB_GIT_PROTOCOL_FIREWALL = "srv1.mydomain.com;srv2.mydomain.com"
>>>>
>>>> Why not use the more generic MIRRORS and PREMIRRORS for this? If they
>>>> can't support this, we likely should figure out a way to make them work
>>>> for it rather than adding new fetcher specific variables.
>>>
>>> Sorry for resurrecting such an old thread, but I'm wondering if
>>> there's been any more work on an upstreamable solution?
>>>
>>> The problem being that recipes which define SRC_URI to git://* with
>>> protocol=git (or with protocol unspecified, since git is the default)
>>> fail during fetching for users working behind a firewall which blocks
>>> the native git protocol.
>>>
>>> Although MIRRORS and PREMIRRORS allow mapping from one host to another
>>> there's no support for mapping one git fetch protocol to another and
>>> it looks like extending the MIRRORS syntax to support that would be a
>>> non-trivial change...
>>
>> Won't the following work:
>>
>> PREMIRRORS_append = " \
>>      git://.*/.* git://HOST/PATH;protocol=https \n \
>>      git://.*/.* git://HOST/PATH;protocol=http \n \
>> "
> 
> Thanks! Yes, that does work. It's the first time I've ever seen HOST
> or PATH being used as mirror replacement patterns. Useful to know.

As far as I know the following are defined:

TYPE, HOST, PATH, BASENAME, MIRRORNAME

The premirror pattern matching is broken up into three parts, type, host, path.
I.e.:

<pattern>://<pattern>/<pattern>

Thus the first three replacements, TYPE, HOST, PATH.

The BASENAME is simply the item the last '/' in the PATH.

MIRRORNAME is more complicated, but matches the name set in the DL_DIR.

(What has never been clear to me is how the mirroring handles the parameters,
items after the ';'.  I assume it just preserves the existing parameters and
then if you specify any additional ones adds them to the end of the list
effectively overriding any prior settings.)

I've not seen it specifically documented this way -- but I've been using this
approach now for a few years so it's possible it has been documented and I've
just missed it.

--Mark

> Would a patch adding something similar to the default MIRRORS (ie try
> http[s] as a fallback if the git protocol fails) be acceptable in
> upstream oe-core?
> 
>> Note: I've not actually tried the above, but I expect it will work.  We do
>> something similar to provide local (on-disk) git caches using:
>>
>> git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file
>>
>> --Mark
>>
>>>> Cheers,
>>>>
>>>> Richard
>>>> --
>>>> _______________________________________________
>>>> bitbake-devel mailing list
>>>> bitbake-devel@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>