RFC: Add http_proxy and ftp_proxy to BitBake white list

RFC: Add http_proxy and ftp_proxy to BitBake white list

[Denys Dmytriyenko]

Hi,

I'd like to propose to add http_proxy and ftp_proxy environment variables to 
BitBake's white list of preserved environment variables. They are usually set 
to be used by fetching tools like wget for users behind a firewall/proxy. 
Thanks.

-- 
Denys

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Cliff Brake]

On Thu, Oct 23, 2008 at 7:09 PM, Denys Dmytriyenko <denis@denix.org> wrote:
> Hi,
>
> I'd like to propose to add http_proxy and ftp_proxy environment variables to
> BitBake's white list of preserved environment variables. They are usually set
> to be used by fetching tools like wget for users behind a firewall/proxy.
> Thanks.

In my latest tests, it does not seem like http_proxy is used by the bb
fetcher even when it is included in BB_ENV_EXTRAWHITE.  Has anyone
else seen this?

Thanks,
Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Richard Purdie]

On Fri, 2008-10-24 at 09:38 -0400, Cliff Brake wrote:
> On Thu, Oct 23, 2008 at 7:09 PM, Denys Dmytriyenko <denis@denix.org> wrote:
> > Hi,
> >
> > I'd like to propose to add http_proxy and ftp_proxy environment variables to
> > BitBake's white list of preserved environment variables. They are usually set
> > to be used by fetching tools like wget for users behind a firewall/proxy.
> > Thanks.
> 
> In my latest tests, it does not seem like http_proxy is used by the bb
> fetcher even when it is included in BB_ENV_EXTRAWHITE.  Has anyone
> else seen this?

Yes, I've just added some extra support for this to the bitbake in Poky.
I'll aim to get that (and some other changes) pushed upstream this
weekend.

Cheers,

Richard

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Cliff Brake]

In Fri, Oct 24, 2008 at 10:17 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
>
> On Fri, 2008-10-24 at 09:38 -0400, Cliff Brake wrote:
>> On Thu, Oct 23, 2008 at 7:09 PM, Denys Dmytriyenko <denis@denix.org> wrote:
>> > Hi,
>> >
>> > I'd like to propose to add http_proxy and ftp_proxy environment variables to
>> > BitBake's white list of preserved environment variables. They are usually set
>> > to be used by fetching tools like wget for users behind a firewall/proxy.
>> > Thanks.
>>
>> In my latest tests, it does not seem like http_proxy is used by the bb
>> fetcher even when it is included in BB_ENV_EXTRAWHITE.  Has anyone
>> else seen this?
>
> Yes, I've just added some extra support for this to the bitbake in Poky.
> I'll aim to get that (and some other changes) pushed upstream this
> weekend.

Richard,

I've attempted to merge your patches and test, but the http_proxy is
still not preserved for wget.  A few notes:

wget does not pass through runfetchcmd() so exportvars stuff never
gets called there.

try_mirror also has some http_proxy stuff, but it does not appear to
be used by wget.

wget.py looks at HTTP_PROXY, but why HTTP_PROXY vs http_proxy?  If I
set HTTP_PROXY in local.conf, it then works.

So, is the idea the user has to set HTTP_PROXY in a conf file for wget
proxy to work?  What are the downsides to using http_proxy from the
environment?  At any rate, it seems like we have some inconsistencies
in the various fetcher methods.

Thanks,
Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Cliff Brake]

On Fri, Nov 7, 2008 at 2:57 PM, Cliff Brake <cliff.brake@gmail.com> wrote:

One additional note,

If http_proxy is in BB_ENV_EXTRAWHITE, the value of http_proxy from
the env does _not_ get passed to wget.  If I set BB_PRESERVE_ENV, it
does.  This is a bit puzzling as looking at the code in utils.py --
with both methods, the environment includes http_proxy after
clean_environment() is called, and this is the only place
BB_ENV_EXTRAWHITE is used -- very interesting.  And in both cases
"bitbake -e" lists it in the environment.

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Richard Purdie]

Hi Cliff,

On Fri, 2008-11-07 at 14:57 -0500, Cliff Brake wrote:
> I've attempted to merge your patches and test, but the http_proxy is
> still not preserved for wget.  A few notes:
> 
> wget does not pass through runfetchcmd() so exportvars stuff never
> gets called there.
> 
> try_mirror also has some http_proxy stuff, but it does not appear to
> be used by wget.
> 
> wget.py looks at HTTP_PROXY, but why HTTP_PROXY vs http_proxy?  If I
> set HTTP_PROXY in local.conf, it then works.

For various reasons I've now had some experience of using bitbake behind
some pretty strict proxy setups. The problem is that its not as simple
as just setting http_proxy since some addresses need to go through the
proxy but things like a local source mirror might be on an internal
network and therefore shouldn't go through the proxy.

I hence reworked things so you set HTTP_PROXY but can also set
HTTP_PROXY_IGNORE which is a list of base urls which should not use the
proxy. The fetcher code will only set http_proxy to HTTP_PROXY if the
url is not in the ignore list.

FTP_PROXY and ftp_proxy works in a similar way.

> So, is the idea the user has to set HTTP_PROXY in a conf file for wget
> proxy to work?  What are the downsides to using http_proxy from the
> environment?  At any rate, it seems like we have some inconsistencies
> in the various fetcher methods.

The idea is the user sets HTTP_PROXY and optionally HTTP_PROXY_IGNORE to
make http proxying work. I'm planning to make bitbake ignore http_proxy
from the environment due to the problems is causes with mirror servers
behind the proxy.

Hopefully this clears up the intent of the patches, the improvements are
based on real world usage problems.

Cheers,

Richard

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Denys Dmytriyenko]

On Sun, Nov 09, 2008 at 01:22:14PM +0000, Richard Purdie wrote:
> On Fri, 2008-11-07 at 14:57 -0500, Cliff Brake wrote:
> > I've attempted to merge your patches and test, but the http_proxy is
> > still not preserved for wget.  A few notes:
> > 
> > wget does not pass through runfetchcmd() so exportvars stuff never
> > gets called there.
> > 
> > try_mirror also has some http_proxy stuff, but it does not appear to
> > be used by wget.
> > 
> > wget.py looks at HTTP_PROXY, but why HTTP_PROXY vs http_proxy?  If I
> > set HTTP_PROXY in local.conf, it then works.

Richard,

Thanks for your work.

> For various reasons I've now had some experience of using bitbake behind
> some pretty strict proxy setups. The problem is that its not as simple
> as just setting http_proxy since some addresses need to go through the
> proxy but things like a local source mirror might be on an internal
> network and therefore shouldn't go through the proxy.

Why not use the standard $no_proxy environment variable to list addresses 
inside the firewall?

> I hence reworked things so you set HTTP_PROXY but can also set
> HTTP_PROXY_IGNORE which is a list of base urls which should not use the
> proxy. The fetcher code will only set http_proxy to HTTP_PROXY if the
> url is not in the ignore list.
> 
> FTP_PROXY and ftp_proxy works in a similar way.

Looks redundant, as it can be achieved like this:

$ export http_proxy=proxy.company.com:80
$ export no_proxy=company.com,interweb.net

> > So, is the idea the user has to set HTTP_PROXY in a conf file for wget
> > proxy to work?  What are the downsides to using http_proxy from the
> > environment?  At any rate, it seems like we have some inconsistencies
> > in the various fetcher methods.
> 
> The idea is the user sets HTTP_PROXY and optionally HTTP_PROXY_IGNORE to
> make http proxying work. I'm planning to make bitbake ignore http_proxy
> from the environment due to the problems is causes with mirror servers
> behind the proxy.
> 
> Hopefully this clears up the intent of the patches, the improvements are
> based on real world usage problems.

Even though I'd prefer it to be fixed properly with 15-year-old http_proxy and 
no_proxy environment variables, it does not matter too much, as long as any 
kind of http/ftp proxy support is merged into upstream BitBake, please. :) 
Thanks.

-- 
Denys

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Denys Dmytriyenko]

On Wed, Nov 12, 2008 at 01:44:16PM -0500, Denys Dmytriyenko wrote:
> On Sun, Nov 09, 2008 at 01:22:14PM +0000, Richard Purdie wrote:
> > For various reasons I've now had some experience of using bitbake behind
> > some pretty strict proxy setups. The problem is that its not as simple
> > as just setting http_proxy since some addresses need to go through the
> > proxy but things like a local source mirror might be on an internal
> > network and therefore shouldn't go through the proxy.
> 
> Why not use the standard $no_proxy environment variable to list addresses 
> inside the firewall?
> 
> > I hence reworked things so you set HTTP_PROXY but can also set
> > HTTP_PROXY_IGNORE which is a list of base urls which should not use the
> > proxy. The fetcher code will only set http_proxy to HTTP_PROXY if the
> > url is not in the ignore list.
> > 
> > FTP_PROXY and ftp_proxy works in a similar way.
> 
> Looks redundant, as it can be achieved like this:
> 
> $ export http_proxy=proxy.company.com:80
> $ export no_proxy=company.com,interweb.net
> 
> > > So, is the idea the user has to set HTTP_PROXY in a conf file for wget
> > > proxy to work?  What are the downsides to using http_proxy from the
> > > environment?  At any rate, it seems like we have some inconsistencies
> > > in the various fetcher methods.
> > 
> > The idea is the user sets HTTP_PROXY and optionally HTTP_PROXY_IGNORE to
> > make http proxying work. I'm planning to make bitbake ignore http_proxy
> > from the environment due to the problems is causes with mirror servers
> > behind the proxy.
> > 
> > Hopefully this clears up the intent of the patches, the improvements are
> > based on real world usage problems.
> 
> Even though I'd prefer it to be fixed properly with 15-year-old http_proxy and 
> no_proxy environment variables, it does not matter too much, as long as any 
> kind of http/ftp proxy support is merged into upstream BitBake, please. :) 
> Thanks.

And speaking of proxies - since recently you removed GIT_PROXY_COMMAND from 
the list of exported variables, what's the correct method of using Git behind 
the proxy with BitBake?

-- 
Denys

Re: RFC: Add http_proxy and ftp_proxy to BitBake white list

[Cliff Brake]

On Sun, Nov 9, 2008 at 8:22 AM, Richard Purdie <rpurdie@rpsys.net> wrote:

> Hopefully this clears up the intent of the patches, the improvements are
> based on real world usage problems.

Thanks for explaining.  I'll do some more testing and would be glad to
help with documentation once things are merged to upstream bitbake.

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com