Yocto git server via http and mirror tarballs

Yocto git server via http and mirror tarballs

[Pascal Bach]

Hi

Because of firewall restrictions I'm not always not able to access the git.yoctoproject.org via the git protocol but I'm required to use http.

By default most of the recipes in Poky recipes use the git protocol not the http protocol. For example yocto-kernel-tools points to repositories "git://git.yoctoproject.org/yocto-kernel-tools" instead of "http://git.yoctoproject.org/git/yocto-kernel-tools".

To work around this issue I added a mirror rules for rewriting the protocol like this:

MIRRORS += "\
    git://git.yoctoproject.org/.*                               git://git.yoctoproject.org/git/PATH;protocol=http \n \
    "

This seems to work and I'm able to fetch the sources even when git access is not possible.

But when I then enable BB_GENERATE_MIRROR_TARBALLS = "1" I'm starting to get different tarball names depending on what protocol was used.
For example "git2_git.yoctoproject.org.yocto-kernel-tools.git.tar.gz" vs "git2_git.yoctoproject.org.git.yocto-kernel-tools.git.tar.gz".

I assume this happens because the first tarball was create while git access was possible, while the second one was created when only http was possible:
git.yoctoproject.org/git => git.yoctoproject.org.git
git.yoctoproject.org     => git.yoctoproject.org

Is there any way to prevent the tarballs from having different names? For example derive the name always from the original URL not the mirror one?

Or is there a better way to achieve what we are doing with MIRROR?

Thanks for your help.

Pascal

Re: Yocto git server via http and mirror tarballs

[Woronicz, Bartosz (Nokia - PL/Wroclaw)]

I set in .gitconfig to always use https:// (don't forget 'git' at the end of path) instead git:// for yoctoproject url

[url "https://git.yoctoproject.org/git/"]
        insteadOf = git://git.yoctoproject.org/

Kind regards,
Bartosz Woronicz
Engineer, Software Configuration (SCM)
Nokia Networks - PL/Wroclaw

-----Original Message-----
From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Pascal Bach
Sent: Wednesday, June 15, 2016 12:51 PM
To: yocto@yoctoproject.org
Subject: [yocto] Yocto git server via http and mirror tarballs

Hi

Because of firewall restrictions I'm not always not able to access the git.yoctoproject.org via the git protocol but I'm required to use http.

By default most of the recipes in Poky recipes use the git protocol not the http protocol. For example yocto-kernel-tools points to repositories "git://git.yoctoproject.org/yocto-kernel-tools" instead of "http://git.yoctoproject.org/git/yocto-kernel-tools".

To work around this issue I added a mirror rules for rewriting the protocol like this:

MIRRORS += "\
    git://git.yoctoproject.org/.*                               git://git.yoctoproject.org/git/PATH;protocol=http \n \
    "

This seems to work and I'm able to fetch the sources even when git access is not possible.

But when I then enable BB_GENERATE_MIRROR_TARBALLS = "1" I'm starting to get different tarball names depending on what protocol was used.
For example "git2_git.yoctoproject.org.yocto-kernel-tools.git.tar.gz" vs "git2_git.yoctoproject.org.git.yocto-kernel-tools.git.tar.gz".

I assume this happens because the first tarball was create while git access was possible, while the second one was created when only http was possible:
git.yoctoproject.org/git => git.yoctoproject.org.git
git.yoctoproject.org     => git.yoctoproject.org

Is there any way to prevent the tarballs from having different names? For example derive the name always from the original URL not the mirror one?

Or is there a better way to achieve what we are doing with MIRROR?

Thanks for your help.

Pascal
-- 
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: Yocto git server via http and mirror tarballs

[Pascal Bach]

> I set in .gitconfig to always use https:// (don't forget 'git' at the end of path) instead git:// for yoctoproject url
>
> [url "https://git.yoctoproject.org/git/"]
>         insteadOf = git://git.yoctoproject.org/

I would pefer a solution that can be handled at the distro/site.conf level so that not very developer has to change their .gitconfig.
However this is a good work around, thanks.

>
> Kind regards,
> Bartosz Woronicz
> Engineer, Software Configuration (SCM)
> Nokia Networks - PL/Wroclaw
>
> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Pascal Bach
> Sent: Wednesday, June 15, 2016 12:51 PM
> To: yocto@yoctoproject.org
> Subject: [yocto] Yocto git server via http and mirror tarballs
>
> Hi
>
> Because of firewall restrictions I'm not always not able to access the git.yoctoproject.org via the git protocol but I'm required to use http.
>
> By default most of the recipes in Poky recipes use the git protocol not the http protocol. For example yocto-kernel-tools points to repositories "git://git.yoctoproject.org/yocto-kernel-tools" instead of "http://git.yoctoproject.org/git/yocto-kernel-tools".
>
> To work around this issue I added a mirror rules for rewriting the protocol like this:
>
> MIRRORS += "\
>     git://git.yoctoproject.org/.*                               git://git.yoctoproject.org/git/PATH;protocol=http \n \
>     "
>
> This seems to work and I'm able to fetch the sources even when git access is not possible.
>
> But when I then enable BB_GENERATE_MIRROR_TARBALLS = "1" I'm starting to get different tarball names depending on what protocol was used.
> For example "git2_git.yoctoproject.org.yocto-kernel-tools.git.tar.gz" vs "git2_git.yoctoproject.org.git.yocto-kernel-tools.git.tar.gz".
>
> I assume this happens because the first tarball was create while git access was possible, while the second one was created when only http was possible:
> git.yoctoproject.org/git => git.yoctoproject.org.git
> git.yoctoproject.org     => git.yoctoproject.org
>
> Is there any way to prevent the tarballs from having different names? For example derive the name always from the original URL not the mirror one?
>
> Or is there a better way to achieve what we are doing with MIRROR?
>
> Thanks for your help.
>
> Pascal

Re: Yocto git server via http and mirror tarballs

[Woronicz, Bartosz ( NSN - PL/Wroclaw)]

Maybe try to prepend PREMIRRORS , then possibly everytime it will use 
HTTP/HTTPS protocol instead git (because it will be the very first match).

http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-PREMIRRORS

Kind regards,
Bartosz Woronicz
Engineer, Software Configuration (SCM)
Nokia Networks - PL/Wroclaw

On 15.06.2016 13:33, Pascal Bach wrote:
>> I set in .gitconfig to always use https:// (don't forget 'git' at the end of path) instead git:// for yoctoproject url
>>
>> [url "https://git.yoctoproject.org/git/"]
>>          insteadOf = git://git.yoctoproject.org/
> I would pefer a solution that can be handled at the distro/site.conf level so that not very developer has to change their .gitconfig.
> However this is a good work around, thanks.
>
>> Kind regards,
>> Bartosz Woronicz
>> Engineer, Software Configuration (SCM)
>> Nokia Networks - PL/Wroclaw
>>
>> -----Original Message-----
>> From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Pascal Bach
>> Sent: Wednesday, June 15, 2016 12:51 PM
>> To: yocto@yoctoproject.org
>> Subject: [yocto] Yocto git server via http and mirror tarballs
>>
>> Hi
>>
>> Because of firewall restrictions I'm not always not able to access the git.yoctoproject.org via the git protocol but I'm required to use http.
>>
>> By default most of the recipes in Poky recipes use the git protocol not the http protocol. For example yocto-kernel-tools points to repositories "git://git.yoctoproject.org/yocto-kernel-tools" instead of "http://git.yoctoproject.org/git/yocto-kernel-tools".
>>
>> To work around this issue I added a mirror rules for rewriting the protocol like this:
>>
>> MIRRORS += "\
>>      git://git.yoctoproject.org/.*                               git://git.yoctoproject.org/git/PATH;protocol=http \n \
>>      "
>>
>> This seems to work and I'm able to fetch the sources even when git access is not possible.
>>
>> But when I then enable BB_GENERATE_MIRROR_TARBALLS = "1" I'm starting to get different tarball names depending on what protocol was used.
>> For example "git2_git.yoctoproject.org.yocto-kernel-tools.git.tar.gz" vs "git2_git.yoctoproject.org.git.yocto-kernel-tools.git.tar.gz".
>>
>> I assume this happens because the first tarball was create while git access was possible, while the second one was created when only http was possible:
>> git.yoctoproject.org/git => git.yoctoproject.org.git
>> git.yoctoproject.org     => git.yoctoproject.org
>>
>> Is there any way to prevent the tarballs from having different names? For example derive the name always from the original URL not the mirror one?
>>
>> Or is there a better way to achieve what we are doing with MIRROR?
>>
>> Thanks for your help.
>>
>> Pascal