Git and http problem.

Git and http problem.

[Francois P. Gallichand]

[-- Attachment #1.1: Type: text/plain, Size: 975 bytes --]

Hello,

I'm working behind a firewall, on which I have absolutely no control, and it does not allows the git protocol to go through. So I patch the recipes to replace the git protocol by the http protocol and it works most of the time. However one specific package, linux-yocto, does not seem to work even If I use the http protocol. It either times-out during the fetch, of the fetch goes on forever.

Frustrated with that situation, I tried various things to realize that if I tried to manually clone the git repository, it doesn't work with a "git clone git://...;protocol=http;..." but if I use "git clone http://...;protocol=http; ...",it is working fine. But the  http:// form is not supported by bitbake as if I use it in a recipe as the SRC_URI, bitbake gives me an error.

Is it a bug with bitbake or am I doing something wrong?

Regards,

François Gallichand
Engineer - Software Group
ABB Group
Analytical Business
(418)877-2944 x268

[ABB logo]


[-- Attachment #1.2: Type: text/html, Size: 3687 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 3279 bytes --]

Re: Git and http problem.

[Paul Eggleton]

Hi Francois,

On Monday 13 April 2015 11:53:25 Francois P. Gallichand wrote:
> I'm working behind a firewall, on which I have absolutely no control, and it
> does not allows the git protocol to go through. So I patch the recipes to
> replace the git protocol by the http protocol and it works most of the
> time. However one specific package, linux-yocto, does not seem to work even
> If I use the http protocol. It either times-out during the fetch, of the
> fetch goes on forever.

Assuming there's no infrastructure issue on our end, it sounds a bit like your 
firewall might be proxying the http fetch completely, and the fetch takes 
longer than the configured timeout.

You could set up a local mirror to work around this and set up the system to 
point to it using PREMIRRORS. Note that you can also use PREMIRRORS to 
effectively add ;protocol=http to git URIs as well, you shouldn't need to patch 
each recipe.
 
> Frustrated with that situation, I tried various things to realize that if I
> tried to manually clone the git repository, it doesn't work with a "git
> clone git://...;protocol=http;..." but if I use "git clone
> http://...;protocol=http; ...",it is working fine. But the  http:// form is
> not supported by bitbake as if I use it in a recipe as the SRC_URI, bitbake
> gives me an error.
> 
> Is it a bug with bitbake or am I doing something wrong?

It's not a bug - git and bitbake's fetcher expect different URI formats - 
bitbake has support for multiple fetch types and therefore needs a hint to 
tell it which fetcher module should handle the URI and that's the prefix, so 
anything that should be fetched by the git fetcher module needs to be prefixed 
with git:// even if the actual protocol ends up being http (in which case it 
would be git://...;protocol=http). If you are then going back to git on the 
command line you'd need to translate that back into http://.. form (note that 
protocol= is something our git fetcher understands; git itself does not so you 
would not specify that on the git command line).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Git and http problem.

[Nikolay Dimitrov]

Hi Francois,

On 04/13/2015 03:43 PM, Paul Eggleton wrote:
> Hi Francois,
>
> On Monday 13 April 2015 11:53:25 Francois P. Gallichand wrote:
>> I'm working behind a firewall, on which I have absolutely no
>> control, and it does not allows the git protocol to go through. So
>> I patch the recipes to replace the git protocol by the http
>> protocol and it works most of the time. However one specific
>> package, linux-yocto, does not seem to work even If I use the http
>> protocol. It either times-out during the fetch, of the fetch goes
>> on forever.
>
> Assuming there's no infrastructure issue on our end, it sounds a bit
> like your firewall might be proxying the http fetch completely, and
> the fetch takes longer than the configured timeout.
>
> You could set up a local mirror to work around this and set up the
> system to point to it using PREMIRRORS. Note that you can also use
> PREMIRRORS to effectively add ;protocol=http to git URIs as well,
> you shouldn't need to patch each recipe.
>
>> Frustrated with that situation, I tried various things to realize
>> that if I tried to manually clone the git repository, it doesn't
>> work with a "git clone git://...;protocol=http;..." but if I use
>> "git clone http://...;protocol=http; ...",it is working fine. But
>> the  http:// form is not supported by bitbake as if I use it in a
>> recipe as the SRC_URI, bitbake gives me an error.
>>
>> Is it a bug with bitbake or am I doing something wrong?
>
> It's not a bug - git and bitbake's fetcher expect different URI
> formats - bitbake has support for multiple fetch types and therefore
> needs a hint to tell it which fetcher module should handle the URI
> and that's the prefix, so anything that should be fetched by the git
> fetcher module needs to be prefixed with git:// even if the actual
> protocol ends up being http (in which case it would be
> git://...;protocol=http). If you are then going back to git on the
> command line you'd need to translate that back into http://.. form
> (note that protocol= is something our git fetcher understands; git
> itself does not so you would not specify that on the git command
> line).

The reason for this behavior is that the git and http contents are
served via different servers - git vs apache+cgit. So the URLs are not
always 1:1 as in this specific case, and you need to change more then
the URL scheme:

git://git.yoctoproject.org/linux-yocto-3.14
http://git.yoctoproject.org/git/linux-yocto-3.14

Please note the additional "git/" in the 2nd URL.

Regards,
Nikolay

Re: Git and http problem.

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

On Mon, Apr 13, 2015 at 1:53 PM, Francois P. Gallichand <
francois.p.gallichand@ca.abb.com> wrote:

> I'm working behind a firewall, on which I have absolutely no control, and
> it does not allows the git protocol to go through. So I patch the recipes
> to replace the git protocol by the http protocol and it works most of the
> time. However one specific package, linux-yocto, does not seem to work even
> If I use the http protocol. It either times-out during the fetch, of the
> fetch goes on forever.
>
>
>
> Frustrated with that situation, I tried various things to realize that if
> I tried to manually clone the git repository, it doesn't work with a "git
> clone git://…;protocol=http;…" but if I use "git clone http://...;protocol=http;
> …",it is working fine. But the  http:// form is not supported by bitbake
> as if I use it in a recipe as the SRC_URI, bitbake gives me an error.
>
>
>
> Is it a bug with bitbake or am I doing something wrong?
>


not a direct answer to your question.. but I would recommend to look into
using something like this:

https://wiki.yoctoproject.org/wiki/Working_Behind_a_Network_Proxy
http://gitolite.com/git-over-proxy.html

I have used that a lot in the past, when I had to deal with corporate
firewalls, and it doesn't require that you patch any recipe.

[-- Attachment #2: Type: text/html, Size: 2040 bytes --]