Introduction and Request for HTTP Git Access

Introduction and Request for HTTP Git Access

[Ryan_D_Phillips]

Good Day!

My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux Developer, currently a software developer architecting an embedded firmware solution on OE for my employer. With the release of Yocto, we are extremely interested in trying it out and possibly contributing to the project. My cohort working with me is Sean Hudson (irc: darknighte), and is someone who you may see contributions from as well.

A major stumbling block we must work around is the restrictive firewall and git checkouts. We have found out the git-http-backend works extremely well, and are requesting this support on git.pokylinux.org. We suspect other people within larger corporations would have a similar problem, and by adding this support would help the community at large.

I can help out in any way possible.

Regards,
Ryan Phillips

Re: Introduction and Request for HTTP Git Access

[Denys Dmytriyenko]

On Tue, Nov 09, 2010 at 10:31:37AM -0600, Ryan_D_Phillips@Dell.com wrote:
> Good Day!
> 
> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux Developer, 
> currently a software developer architecting an embedded firmware solution on 
> OE for my employer. With the release of Yocto, we are extremely interested 
> in trying it out and possibly contributing to the project. My cohort working 
> with me is Sean Hudson (irc: darknighte), and is someone who you may see 
> contributions from as well.
> 
> A major stumbling block we must work around is the restrictive firewall and 
> git checkouts. We have found out the git-http-backend works extremely well, 
> and are requesting this support on git.pokylinux.org. We suspect other 
> people within larger corporations would have a similar problem, and by 
> adding this support would help the community at large.

Ryan,

It's nice to see another Gentooist!

Have you looked at core.gitProxy setting or GIT_PROXY_COMMAND environment in 
git-config(1)? People behind strict firewalls been using that for years to 
tunnel their git access through proxies. Used together with things like 
netcat, socat or corkscrew work miracles even with HTTP-only proxies!

-- 
Denys

Re: Introduction and Request for HTTP Git Access

[Darren Hart]

On 11/09/2010 09:02 AM, Denys Dmytriyenko wrote:
> On Tue, Nov 09, 2010 at 10:31:37AM -0600, Ryan_D_Phillips@Dell.com wrote:
>> Good Day!
>>
>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux Developer,
>> currently a software developer architecting an embedded firmware solution on
>> OE for my employer. With the release of Yocto, we are extremely interested
>> in trying it out and possibly contributing to the project. My cohort working
>> with me is Sean Hudson (irc: darknighte), and is someone who you may see
>> contributions from as well.
>>
>> A major stumbling block we must work around is the restrictive firewall and
>> git checkouts. We have found out the git-http-backend works extremely well,
>> and are requesting this support on git.pokylinux.org. We suspect other
>> people within larger corporations would have a similar problem, and by
>> adding this support would help the community at large.
>
> Ryan,
>
> It's nice to see another Gentooist!
>
> Have you looked at core.gitProxy setting or GIT_PROXY_COMMAND environment in
> git-config(1)? People behind strict firewalls been using that for years to
> tunnel their git access through proxies. Used together with things like
> netcat, socat or corkscrew work miracles even with HTTP-only proxies!

It is certainly sub-ideal (but so is git over http), but it should 
provide the functionality you need. Please give the above a shot and if 
for some reason it is insufficient, please let us know.

-- 
Darren Hart
Yocto Linux Kernel

Re: Introduction and Request for HTTP Git Access

[Ryan D Phillips]

On 11/9/2010 11:22 AM, Darren Hart wrote:
> On 11/09/2010 09:02 AM, Denys Dmytriyenko wrote:
>> On Tue, Nov 09, 2010 at 10:31:37AM -0600, Ryan_D_Phillips@Dell.com wrote:
>>> Good Day!
>>>
>>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux Developer,
>>> currently a software developer architecting an embedded firmware solution on
>>> OE for my employer. With the release of Yocto, we are extremely interested
>>> in trying it out and possibly contributing to the project. My cohort working
>>> with me is Sean Hudson (irc: darknighte), and is someone who you may see
>>> contributions from as well.
>>>
>>> A major stumbling block we must work around is the restrictive firewall and
>>> git checkouts. We have found out the git-http-backend works extremely well,
>>> and are requesting this support on git.pokylinux.org. We suspect other
>>> people within larger corporations would have a similar problem, and by
>>> adding this support would help the community at large.
>>
>> Ryan,
>>
>> It's nice to see another Gentooist!
>>
>> Have you looked at core.gitProxy setting or GIT_PROXY_COMMAND environment in
>> git-config(1)? People behind strict firewalls been using that for years to
>> tunnel their git access through proxies. Used together with things like
>> netcat, socat or corkscrew work miracles even with HTTP-only proxies!
>
> It is certainly sub-ideal (but so is git over http), but it should
> provide the functionality you need. Please give the above a shot and if
> for some reason it is insufficient, please let us know.
>

I don't understand the comment that git over http is sub-ideal (see 
github using https for git by default), but the issue we have is the 
proxy server restricts outbound ports to 80 and 443, and doesn't allow 
CONNECT.

Regards,
Ryan

Re: Introduction and Request for HTTP Git Access

[Scott Garman]

On 11/09/2010 08:31 AM, Ryan_D_Phillips@Dell.com wrote:
> Good Day!
>
> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux
> Developer, currently a software developer architecting an embedded
> firmware solution on OE for my employer. With the release of Yocto,
> we are extremely interested in trying it out and possibly
> contributing to the project. My cohort working with me is Sean Hudson
> (irc: darknighte), and is someone who you may see contributions from
> as well.
>
> A major stumbling block we must work around is the restrictive
> firewall and git checkouts. We have found out the git-http-backend
> works extremely well, and are requesting this support on
> git.pokylinux.org. We suspect other people within larger corporations
> would have a similar problem, and by adding this support would help
> the community at large.
>
> I can help out in any way possible.

Hi Ryan,

I just added a page to our wiki that details instructions for getting up
and running with Poky behind a SOCKS proxy server:

https://wiki.yoctoproject.org/wiki/Working_Behind_a_Network_Proxy

Scott Garman

Re: Introduction and Request for HTTP Git Access

[Denys Dmytriyenko]

On Tue, Nov 09, 2010 at 11:34:48AM -0600, Ryan D Phillips wrote:
> On 11/9/2010 11:22 AM, Darren Hart wrote:
>> On 11/09/2010 09:02 AM, Denys Dmytriyenko wrote:
>>> On Tue, Nov 09, 2010 at 10:31:37AM -0600, Ryan_D_Phillips@Dell.com wrote:
>>>> Good Day!
>>>>
>>>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux 
>>>> Developer,
>>>> currently a software developer architecting an embedded firmware 
>>>> solution on
>>>> OE for my employer. With the release of Yocto, we are extremely 
>>>> interested
>>>> in trying it out and possibly contributing to the project. My cohort 
>>>> working
>>>> with me is Sean Hudson (irc: darknighte), and is someone who you may see
>>>> contributions from as well.
>>>>
>>>> A major stumbling block we must work around is the restrictive firewall 
>>>> and
>>>> git checkouts. We have found out the git-http-backend works extremely 
>>>> well,
>>>> and are requesting this support on git.pokylinux.org. We suspect other
>>>> people within larger corporations would have a similar problem, and by
>>>> adding this support would help the community at large.
>>>
>>> Ryan,
>>>
>>> It's nice to see another Gentooist!
>>>
>>> Have you looked at core.gitProxy setting or GIT_PROXY_COMMAND environment 
>>> in
>>> git-config(1)? People behind strict firewalls been using that for years 
>>> to
>>> tunnel their git access through proxies. Used together with things like
>>> netcat, socat or corkscrew work miracles even with HTTP-only proxies!
>>
>> It is certainly sub-ideal (but so is git over http), but it should
>> provide the functionality you need. Please give the above a shot and if
>> for some reason it is insufficient, please let us know.
>>
>
> I don't understand the comment that git over http is sub-ideal (see github 
> using https for git by default), but the issue we have is the proxy server 
> restricts outbound ports to 80 and 443, and doesn't allow CONNECT.

git-over-http used to be sub-ideal until version 1.6.6 of git, as it only 
provided "dumb" read-only access. Now, with git-http-backend(1) there is a 
"smart" transport available, which can also handle pushing over https...

http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html
http://progit.org/2010/03/04/smart-http.html

-- 
Denys

Re: Introduction and Request for HTTP Git Access

[Denys Dmytriyenko]

On Tue, Nov 09, 2010 at 10:05:29AM -0800, Scott Garman wrote:
> On 11/09/2010 08:31 AM, Ryan_D_Phillips@Dell.com wrote:
>> Good Day!
>>
>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux
>> Developer, currently a software developer architecting an embedded
>> firmware solution on OE for my employer. With the release of Yocto,
>> we are extremely interested in trying it out and possibly
>> contributing to the project. My cohort working with me is Sean Hudson
>> (irc: darknighte), and is someone who you may see contributions from
>> as well.
>>
>> A major stumbling block we must work around is the restrictive
>> firewall and git checkouts. We have found out the git-http-backend
>> works extremely well, and are requesting this support on
>> git.pokylinux.org. We suspect other people within larger corporations
>> would have a similar problem, and by adding this support would help
>> the community at large.
>>
>> I can help out in any way possible.
>
> Hi Ryan,
>
> I just added a page to our wiki that details instructions for getting up
> and running with Poky behind a SOCKS proxy server:
>
> https://wiki.yoctoproject.org/wiki/Working_Behind_a_Network_Proxy

Hmm, quite interesting. Looks like Richard finally saw the light at the end of 
the tunnel, so to speak, :) and you guys started using standard environment 
variables - http_proxy, ftp_proxy, no_proxy and GIT_PROXY_COMMAND, passing 
them through bitbake to the external fetchers. At least it appears this way 
from reading the above Wiki instructions... As we have almost the same exact 
instructions to work inside TI.

2 years ago I tried convincing Richard to switch from non-standard HTTP_PROXY* 
and FTP_PROXY* bitbake variables to the standard ones, but wasn't successful[1].
So I had to fork bitbake and make the changes myself for our Arago Project, 
rebasing them later against the final 1.8.18 version of bitbake[2].

Richard, that's what I mentioned during OEDEM couple weeks ago. I was going to 
squash my changes, rebase them against 1.10 or master and post them to the 
bitbake mailing list...

[1] http://thread.gmane.org/gmane.comp.handhelds.openembedded/19716
[2] http://arago-project.org/git/?p=arago-bitbake.git;a=summary

-- 
Denys

Re: Introduction and Request for HTTP Git Access

[Darren Hart]

On 11/09/2010 10:06 AM, Denys Dmytriyenko wrote:
> On Tue, Nov 09, 2010 at 11:34:48AM -0600, Ryan D Phillips wrote:
>> On 11/9/2010 11:22 AM, Darren Hart wrote:
>>> On 11/09/2010 09:02 AM, Denys Dmytriyenko wrote:
>>>> On Tue, Nov 09, 2010 at 10:31:37AM -0600, Ryan_D_Phillips@Dell.com wrote:
>>>>> Good Day!
>>>>>
>>>>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux
>>>>> Developer,
>>>>> currently a software developer architecting an embedded firmware
>>>>> solution on
>>>>> OE for my employer. With the release of Yocto, we are extremely
>>>>> interested
>>>>> in trying it out and possibly contributing to the project. My cohort
>>>>> working
>>>>> with me is Sean Hudson (irc: darknighte), and is someone who you may see
>>>>> contributions from as well.
>>>>>
>>>>> A major stumbling block we must work around is the restrictive firewall
>>>>> and
>>>>> git checkouts. We have found out the git-http-backend works extremely
>>>>> well,
>>>>> and are requesting this support on git.pokylinux.org. We suspect other
>>>>> people within larger corporations would have a similar problem, and by
>>>>> adding this support would help the community at large.
>>>>
>>>> Ryan,
>>>>
>>>> It's nice to see another Gentooist!
>>>>
>>>> Have you looked at core.gitProxy setting or GIT_PROXY_COMMAND environment
>>>> in
>>>> git-config(1)? People behind strict firewalls been using that for years
>>>> to
>>>> tunnel their git access through proxies. Used together with things like
>>>> netcat, socat or corkscrew work miracles even with HTTP-only proxies!
>>>
>>> It is certainly sub-ideal (but so is git over http), but it should
>>> provide the functionality you need. Please give the above a shot and if
>>> for some reason it is insufficient, please let us know.
>>>
>>
>> I don't understand the comment that git over http is sub-ideal (see github
>> using https for git by default), but the issue we have is the proxy server
>> restricts outbound ports to 80 and 443, and doesn't allow CONNECT.
>
> git-over-http used to be sub-ideal until version 1.6.6 of git, as it only
> provided "dumb" read-only access. Now, with git-http-backend(1) there is a
> "smart" transport available, which can also handle pushing over https...
>
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html
> http://progit.org/2010/03/04/smart-http.html

Aha, I didn't realize that. My concern would be how that would tie in 
with the ssh key based user auth we currently use. I'll have to leave 
that to Richard or perhaps someone else that knows the backend setup 
better than I do.


-- 
Darren Hart
Embedded Linux Kernel

Re: Introduction and Request for HTTP Git Access

[Ryan D Phillips]

On 11/9/2010 12:31 PM, Denys Dmytriyenko wrote:
> On Tue, Nov 09, 2010 at 10:05:29AM -0800, Scott Garman wrote:
>> On 11/09/2010 08:31 AM, Ryan_D_Phillips@Dell.com wrote:
>>> Good Day!
>>>
>>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux
>>> Developer, currently a software developer architecting an embedded
>>> firmware solution on OE for my employer. With the release of Yocto,
>>> we are extremely interested in trying it out and possibly
>>> contributing to the project. My cohort working with me is Sean Hudson
>>> (irc: darknighte), and is someone who you may see contributions from
>>> as well.
>>>
>>> A major stumbling block we must work around is the restrictive
>>> firewall and git checkouts. We have found out the git-http-backend
>>> works extremely well, and are requesting this support on
>>> git.pokylinux.org. We suspect other people within larger corporations
>>> would have a similar problem, and by adding this support would help
>>> the community at large.
>>>
>

Hmm. I didn't mean to start a firestorm. Read only git-http-backend 
access is something that we would find useful. As an alternative, we can 
create repo.or.cz mirrors, but I'm not all that keen on this.

-ryan

Re: Introduction and Request for HTTP Git Access

[Richard Purdie]

On Tue, 2010-11-09 at 15:46 -0600, Ryan D Phillips wrote:
> On 11/9/2010 12:31 PM, Denys Dmytriyenko wrote:
> > On Tue, Nov 09, 2010 at 10:05:29AM -0800, Scott Garman wrote:
> >> On 11/09/2010 08:31 AM, Ryan_D_Phillips@Dell.com wrote:
> >>> Good Day!
> >>>
> >>> My name is Ryan Phillips (irc: rphillips), a former Gentoo Linux
> >>> Developer, currently a software developer architecting an embedded
> >>> firmware solution on OE for my employer. With the release of Yocto,
> >>> we are extremely interested in trying it out and possibly
> >>> contributing to the project. My cohort working with me is Sean Hudson
> >>> (irc: darknighte), and is someone who you may see contributions from
> >>> as well.
> >>>
> >>> A major stumbling block we must work around is the restrictive
> >>> firewall and git checkouts. We have found out the git-http-backend
> >>> works extremely well, and are requesting this support on
> >>> git.pokylinux.org. We suspect other people within larger corporations
> >>> would have a similar problem, and by adding this support would help
> >>> the community at large.
> >>>
> >
> 
> Hmm. I didn't mean to start a firestorm. Read only git-http-backend 
> access is something that we would find useful. As an alternative, we can 
> create repo.or.cz mirrors, but I'm not all that keen on this.

I'm travelling at the moment so I'll just keep this brief.

I'm more than happy to try and find a way to share git.yoctoproject.org
over http for cloning, that shouldn't be too hard. I'll see if I can get
something working next week although I'm meant to be taking some
vacation too.

For push access over https, that is a lot more difficult as it is
unlikely to integrate well with our access control system. I'd need to
do more reading before I could really comment on that. It appears
changes in git support it better at least.

Cheers,

Richard