[Buildroot] [PATCH 1/1] Add proxy download support

[Buildroot] [PATCH 1/1] Add proxy download support

[Karoly Kasza]

This patch adds support for downloading sources over a http proxy.
Supports wget with http, https & ftp targets, bzr, hg, svn.
Git does work with forcing https instead of git protocol.
Cvs is not supported, but Buildroot does not currently have any
packages relying on cvs repos (plus this would fall back to the
Buildroot mirror over http).

Signed-off-by: Karoly Kasza <kaszak@gmail.com>
---
 Config.in                |   25 +++++++++++++++++++++++++
 Makefile                 |    9 +++++++++
 support/download/wrapper |   26 ++++++++++++++++++++++++++
 3 files changed, 60 insertions(+)

diff --git a/Config.in b/Config.in
index 9cefcbc..2e2b493 100644
--- a/Config.in
+++ b/Config.in
@@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR
 
 endmenu
 
+config BR2_PROXY
+	bool "Use http proxy"
+	help
+	  Use a http proxy when downloading sources with various protocols.
+
+if BR2_PROXY
+
+config BR2_PROXY_URL
+	string "Http proxy URL"
+	help
+	  Http proxy URL in the form of <IP or FQDN>:<port>.
+
+config BR2_PROXY_USER
+	string "Http proxy username"
+	help
+	  Http proxy username (if needed).
+
+config BR2_PROXY_PASS
+	string "Http proxy password"
+	help
+	  Http proxy password (if needed).
+	  PLEASE NOTE! This password will be saved in the .config file!
+
+endif
+
 config BR2_JLEVEL
 	int "Number of jobs to run simultaneously (0 for auto)"
 	default "0"
diff --git a/Makefile b/Makefile
index 907a0fc..5dd40e8 100644
--- a/Makefile
+++ b/Makefile
@@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton
 # should not be used as the root filesystem.
 TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
 
+ifeq ($(BR2_PROXY),y)
+BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL))
+export BR_PROXY_URL
+BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER))
+export BR_PROXY_USER
+BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS))
+export BR_PROXY_PASS
+endif
+
 ifeq ($(BR2_CCACHE),y)
 CCACHE := $(HOST_DIR)/usr/bin/ccache
 BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))
diff --git a/support/download/wrapper b/support/download/wrapper
index 8ae2797..b224474 100755
--- a/support/download/wrapper
+++ b/support/download/wrapper
@@ -44,6 +44,32 @@ tmpf="${tmpd}/output"
 # Doing the 'cd' here rather than in all helpers is easier.
 cd "${tmpd}"
 
+# Set the proxy environment variables.
+# cvs doesn't support proxies OOB, but there is currently no cvs
+# based package in Buildroot. Also, in that case after the download
+# fails, the wget method would be used from sources.buildroot.net.
+if [ -n "${BR_PROXY_URL}" ]; then
+# SVN does not support http_proxy environment variables, so we use
+# it's commandline arguments
+proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1`
+proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2`
+SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \
+--config-option servers:global:http-proxy-port=${proxy_port}"
+if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then
+SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \
+--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}"
+proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}"
+else
+proxy="http://${BR_PROXY_URL}"
+fi
+export http_proxy=${proxy}
+export https_proxy=${proxy}
+export ftp_proxy=${proxy}
+# While git does support http_proxy environment variables, we need to use
+# the config option below to force proxying over https instead of git
+GIT="${GIT} -c url.https://.insteadOf=git://"
+fi
+
 # If the helper fails, we can just remove the temporary directory to
 # remove all the cruft it may have left behind. Then we just exit in
 # error too.
-- 
1.7.10.4

[Buildroot] [PATCH 1/1] Add proxy download support

[Yann E. MORIN]

Karoly, All,

On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
> This patch adds support for downloading sources over a http proxy.
> Supports wget with http, https & ftp targets, bzr, hg, svn.
> Git does work with forcing https instead of git protocol.
> Cvs is not supported, but Buildroot does not currently have any
> packages relying on cvs repos (plus this would fall back to the
> Buildroot mirror over http).
> 
> Signed-off-by: Karoly Kasza <kaszak@gmail.com>

NAK.

This should belong to the user's envirnment.

I once tried to handle proxy support in crosstool-NG, and after trying
to handle the exponentially growing cases (http proxy, with or without
authentication, socks proxy...)

Also, this solution is not bullet proof, because it leaves alone
non-http protocols: cvs, git, svn, scp...

Actually, it does take care of git, but some proxies do not allow the
CONNECT to anything else than 443, so we're back to square one for the
git protocol anyway.

If we were to add a proxy setting, then users would expect it to work
for all kind of downloads, which is absolutely impossible to achieve.

So, we do not have a 100% coverage, and quite some is left out.

So, I prefer we not have any of this proxy settings in Buildroot at all,
and just rely on the user's environment to be properly set up.

Regards,
Yann E. MORIN.

> ---
>  Config.in                |   25 +++++++++++++++++++++++++
>  Makefile                 |    9 +++++++++
>  support/download/wrapper |   26 ++++++++++++++++++++++++++
>  3 files changed, 60 insertions(+)
> 
> diff --git a/Config.in b/Config.in
> index 9cefcbc..2e2b493 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR
>  
>  endmenu
>  
> +config BR2_PROXY
> +	bool "Use http proxy"
> +	help
> +	  Use a http proxy when downloading sources with various protocols.
> +
> +if BR2_PROXY
> +
> +config BR2_PROXY_URL
> +	string "Http proxy URL"
> +	help
> +	  Http proxy URL in the form of <IP or FQDN>:<port>.
> +
> +config BR2_PROXY_USER
> +	string "Http proxy username"
> +	help
> +	  Http proxy username (if needed).
> +
> +config BR2_PROXY_PASS
> +	string "Http proxy password"
> +	help
> +	  Http proxy password (if needed).
> +	  PLEASE NOTE! This password will be saved in the .config file!
> +
> +endif
> +
>  config BR2_JLEVEL
>  	int "Number of jobs to run simultaneously (0 for auto)"
>  	default "0"
> diff --git a/Makefile b/Makefile
> index 907a0fc..5dd40e8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton
>  # should not be used as the root filesystem.
>  TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
>  
> +ifeq ($(BR2_PROXY),y)
> +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL))
> +export BR_PROXY_URL
> +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER))
> +export BR_PROXY_USER
> +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS))
> +export BR_PROXY_PASS
> +endif
> +
>  ifeq ($(BR2_CCACHE),y)
>  CCACHE := $(HOST_DIR)/usr/bin/ccache
>  BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))
> diff --git a/support/download/wrapper b/support/download/wrapper
> index 8ae2797..b224474 100755
> --- a/support/download/wrapper
> +++ b/support/download/wrapper
> @@ -44,6 +44,32 @@ tmpf="${tmpd}/output"
>  # Doing the 'cd' here rather than in all helpers is easier.
>  cd "${tmpd}"
>  
> +# Set the proxy environment variables.
> +# cvs doesn't support proxies OOB, but there is currently no cvs
> +# based package in Buildroot. Also, in that case after the download
> +# fails, the wget method would be used from sources.buildroot.net.
> +if [ -n "${BR_PROXY_URL}" ]; then
> +# SVN does not support http_proxy environment variables, so we use
> +# it's commandline arguments
> +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1`
> +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2`
> +SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \
> +--config-option servers:global:http-proxy-port=${proxy_port}"
> +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then
> +SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \
> +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}"
> +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}"
> +else
> +proxy="http://${BR_PROXY_URL}"
> +fi
> +export http_proxy=${proxy}
> +export https_proxy=${proxy}
> +export ftp_proxy=${proxy}
> +# While git does support http_proxy environment variables, we need to use
> +# the config option below to force proxying over https instead of git
> +GIT="${GIT} -c url.https://.insteadOf=git://"
> +fi
> +
>  # If the helper fails, we can just remove the temporary directory to
>  # remove all the cruft it may have left behind. Then we just exit in
>  # error too.
> -- 
> 1.7.10.4
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/1] Add proxy download support

[Károly Kasza]

>
>
> NAK.
>
> This should belong to the user's envirnment.
>
> OK, but then I think this should be stated in the user manual.


> I once tried to handle proxy support in crosstool-NG, and after trying
> to handle the exponentially growing cases (http proxy, with or without
> authentication, socks proxy...)
>
This handles standard http proxies (hence the name) with or without auth.
Most proxies (and even corporate ones) fall into this category (like Squid,
MS Forefront, etc).


>
> Also, this solution is not bullet proof, because it leaves alone
> non-http protocols: cvs, git, svn, scp...
>
Not true. Only CVS won't work, but:
A, There is currently no CVS based package in Buildroot
B, If there would, it would fall back the sources.buildroot.com, which is
http
C, I may have mislooked and CVS can be proxied - if You allow me to send a
V2, I will introduce that too

All other download methods work with the distributed packages.
Should at least, I only downloaded a few packages from every possible type.

AFAIK SCP & SSH are not standard download methods in the packaging system,
they are
added functionality for custom packages.


>
> Actually, it does take care of git, but some proxies do not allow the
> CONNECT to anything else than 443, so we're back to square one for the
> git protocol anyway.

Git always falls back back to https (the "instead of" directive) and the
standard 443,
because git:// protocol simply cannot be proxied.


> If we were to add a proxy setting, then users would expect it to work
> for all kind of downloads, which is absolutely impossible to achieve.
>
It does work with all existing packages (again, I found no CVS, SCP or SSH
based packages in the distribution).
I can proxy scp or ssh over http, but that relies on the user's proxy
server - there are proxies which won't allow a
CONNECT method for ssh.


>
> So, we do not have a 100% coverage, and quite some is left out.
>
Please name, I will try my best, to send a V2 covering those methods.
Currently it support the 5 types that
are used in the packages.


>
> So, I prefer we not have any of this proxy settings in Buildroot at all,
> and just rely on the user's environment to be properly set up.
>
I think setting up all methods in the user environment is possible, but
sometimes can be quite complicated.

Thank you for your opinion, could you state in which case would you accept
the patch?
I think I can add CVS support, and ssh/scp support - which means ALL 8
download helpers would be able to
use http proxies.
And again, this is a http proxy patch, I didn't aim to support all proxy
types in the world, as that would be
probably pointless.

Best regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141021/32f00315/attachment.html>

[Buildroot] [PATCH 1/1] Add proxy download support

[Yann E. MORIN]

K?roly, All,

On 2014-10-21 20:23 +0200, K?roly Kasza spake thusly:
> > NAK.
> > This should belong to the user's envirnment.
> >
> OK, but then I think this should be stated in the user manual.

This is by far not specific to Buildroot at all, but we can add a blurb
in the manual, telling the user that Buildroot uses the user's settings.

> > I once tried to handle proxy support in crosstool-NG, and after trying
> > to handle the exponentially growing cases (http proxy, with or without
> > authentication, socks proxy...)
> >
> This handles standard http proxies (hence the name) with or without auth.
> Most proxies (and even corporate ones) fall into this category (like Squid,
> MS Forefront, etc).

That's just am incorrect assumption. I know first-hand of network setups
that use much complex proxy setups...

> > Also, this solution is not bullet proof, because it leaves alone
> > non-http protocols: cvs, git, svn, scp...
> >
> Not true. Only CVS won't work, but:
> A, There is currently no CVS based package in Buildroot
> B, If there would, it would fall back the sources.buildroot.com, which is
> http

The initial reason for sources.buildroot.com is to be a fallback location
in case an upstream disapears after we cut a release, so we ensure that
releases continue to work for the foreseeable future.

> C, I may have mislooked and CVS can be proxied - if You allow me to send a
> V2, I will introduce that too

Well, it seems it is possible to proxy cvs through an http proxy, but
that proxy must allow the CONNECT to the CVS port, which a lot of
enterprise proxies just disallow.

For example, my company's proxy only allow CONNECT to the port 443, and
blocks anything else. Which means I can not use it to connect to:
  - a git server using the git protocol
  - a SVN server using the svn protocol
  - a CVS server
  - an ssh server

> All other download methods work with the distributed packages.
> Should at least, I only downloaded a few packages from every possible type.
> 
> AFAIK SCP & SSH are not standard download methods in the packaging system,
> they are added functionality for custom packages.

True, but nothing would prevent a user to add a custom package that
fetches via scp from a remote server outside the enterprise network
(e.g. to retrieve a package from a subsidiary network, for example).

> > Actually, it does take care of git, but some proxies do not allow the
> > CONNECT to anything else than 443, so we're back to square one for the
> > git protocol anyway.
> 
> Git always falls back back to https (the "instead of" directive) and the
> standard 443, because git:// protocol simply cannot be proxied.

Both assumptions are wrong. There are git servers that serves only via
the git protocol, so there is no possible fallback to http.

And even if there were, the relative paths of the two URLs may be
different between the two schemes, so just using http:// instead of
git:// is not sufficient.

And third, the git protocol *can* be proxified, even through an http
proxy, but it requires that proxy to be configured to allow the CONNECT
method to any port, not only 443.

> > If we were to add a proxy setting, then users would expect it to work
> > for all kind of downloads, which is absolutely impossible to achieve.
> >
> It does work with all existing packages (again, I found no CVS, SCP or SSH
> based packages in the distribution).

Yes, they are not used in Buildroot itself. But if we do changes to the
download infra, all our download helpers should behave the same.

> I can proxy scp or ssh over http, but that relies on the user's proxy
> server - there are proxies which won't allow a CONNECT method for ssh.

Exactly, so goes for CVS and GIT.

> > So, we do not have a 100% coverage, and quite some is left out.
> >
> Please name, I will try my best, to send a V2 covering those methods.
> Currently it support the 5 types that are used in the packages.

Well, I agree that it does cover all the methods Buildroot uses, but not
all the methods Buildroot uses.

What are we gonna explain our users that the proxy settings he put in
the menuconfig are not being used, because a package in its BR2_EXTERNAL
tree uses a download method that we do not know how to proxify?

> > So, I prefer we not have any of this proxy settings in Buildroot at all,
> > and just rely on the user's environment to be properly set up.
> >
> I think setting up all methods in the user environment is possible, but
> sometimes can be quite complicated.

Well, it is really easy:

    export http-proxy='http://user:pass at proxy:port/'
    export https_proxy="${http_proxy}"
    export ftp_proxy="${http_proxy}"
    . tsocks -on

Et voil?, all things are proxified (one just has to edit /etc/tsocks.conf
but that is something we really can do ourselves anyway.... Yes, I know
we can use TSOCKS_CONF_FILE=/path/to/my/tsocks.conf)

> Thank you for your opinion, could you state in which case would you accept
> the patch?
> I think I can add CVS support, and ssh/scp support - which means ALL 8
> download helpers would be able to
> use http proxies.
> And again, this is a http proxy patch, I didn't aim to support all proxy
> types in the world, as that would be
> probably pointless.

I still believe we should not have this in Buildroot at all. That is my
opinion, others may disagree with me. Let them speak up! ;-)

I know this can be a bit frustrating, but I can assure you I have
already gone down this road in the past, and it is not a road I would
like to go down again...

To me, the best would be to just document that Buildroot uses whatever
settings are set in the user's system.

BTW, I should know about proxies: absolutely everything I do on my home
machine *is* proxified via either an http or a socks proxy. Even this
very mail will leave my home network via a proxy... ;-]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/1] Add proxy download support

[Arnout Vandecappelle]

On 21/10/14 22:25, Yann E. MORIN wrote:
> I still believe we should not have this in Buildroot at all. That is my
> opinion, others may disagree with me. Let them speak up! ;-)

 Sorry, Yann, I won't speak up because I completely agree with you :-)

 Regards,
 Arnout



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 1/1] Add proxy download support

[Károly Kasza]

Hi Yann,

>
> > > I once tried to handle proxy support in crosstool-NG, and after trying
> > > to handle the exponentially growing cases (http proxy, with or without
> > > authentication, socks proxy...)
> > >
> > This handles standard http proxies (hence the name) with or without auth.
> > Most proxies (and even corporate ones) fall into this category (like
> Squid,
> > MS Forefront, etc).
>
> That's just am incorrect assumption. I know first-hand of network setups
> that use much complex proxy setups...
>
I know some enterprises with tens of thousands computers who use http
proxies
from the users point of view (US & German multinationals). Using anything
else than http proxies are overcomplicated for them and can not be
authorized
reliably (even using SSO). They of course use strict security settings,
IPS/IDS chains,
antivirus and antispam filters, but from the end user perspective they are
all just http proxies...
NAT with L7 filter or transparent proxying are rare, using socks in a
corporate is
even more rare (in my experience at least) - even if it would be available.
I would still say that http proxies are the most common solution, and
that's not just an
assumption. See
https://www.websense.com/assets/reports/report-gartner-magic-quadrant-for-security-web-gateway-2011-en.pdf


> > > Also, this solution is not bullet proof, because it leaves alone
> > > non-http protocols: cvs, git, svn, scp...
> > >
> > Not true. Only CVS won't work, but:
> > A, There is currently no CVS based package in Buildroot
> > B, If there would, it would fall back the sources.buildroot.com, which
> is
> > http
>
> The initial reason for sources.buildroot.com is to be a fallback location
> in case an upstream disapears after we cut a release, so we ensure that
> releases continue to work for the foreseeable future.
>
Aye, but this does not make it unusable if the user falls back there from
another protocol.


>
> > C, I may have mislooked and CVS can be proxied - if You allow me to send
> a
> > V2, I will introduce that too
>
> Well, it seems it is possible to proxy cvs through an http proxy, but
> that proxy must allow the CONNECT to the CVS port, which a lot of
> enterprise proxies just disallow.
>
Very true. In that case you simply cannot use a proxy, you won't be able to
use one.
The proxies' security settings are of course outside of the scope of
Buildroot.


> For example, my company's proxy only allow CONNECT to the port 443, and
> blocks anything else. Which means I can not use it to connect to:
>   - a git server using the git protocol
>
I did not find any information about git allowed over proxy without 3rd
parties (socat, etc).
The insteadof method is the standard for git.


>   - a SVN server using the svn protocol
>   - a CVS server
>   - an ssh server
>
All true. In these cases you simply won't be able to use a proxy. Not from
Buildroot, nor
from the user's environment. Again, security settings can prevent anything.


>
> > All other download methods work with the distributed packages.
> > Should at least, I only downloaded a few packages from every possible
> type.
> >
> > AFAIK SCP & SSH are not standard download methods in the packaging
> system,
> > they are added functionality for custom packages.
>
> True, but nothing would prevent a user to add a custom package that
> fetches via scp from a remote server outside the enterprise network
> (e.g. to retrieve a package from a subsidiary network, for example).
>
Aye, in that case I could still add some logic to the wrapper, like using
netcat or corkscrew
(which are both 3rd party ofc).


> > > Actually, it does take care of git, but some proxies do not allow the
> > > CONNECT to anything else than 443, so we're back to square one for the
> > > git protocol anyway.
> >
> > Git always falls back back to https (the "instead of" directive) and the
> > standard 443, because git:// protocol simply cannot be proxied.
>
> Both assumptions are wrong. There are git servers that serves only via
> the git protocol, so there is no possible fallback to http.
>
I'm not saying that it's falling back by standard. It falls back because of
the directive.
Ofc servers who don't serve over https are can not be proxied this way -
only with socat.


> And even if there were, the relative paths of the two URLs may be
> different between the two schemes, so just using http:// instead of
> git:// is not sufficient.
>
In that case this is a server based problem.


> And third, the git protocol *can* be proxified, even through an http
> proxy, but it requires that proxy to be configured to allow the CONNECT
> method to any port, not only 443.
>
But only with a 3rd party software, not by default. The default recommended
method is
falling back.


>
> > > If we were to add a proxy setting, then users would expect it to work
> > > for all kind of downloads, which is absolutely impossible to achieve.
> > >
> > It does work with all existing packages (again, I found no CVS, SCP or
> SSH
> > based packages in the distribution).
>
> Yes, they are not used in Buildroot itself. But if we do changes to the
> download infra, all our download helpers should behave the same.
>
You are completely right.


>
> > I can proxy scp or ssh over http, but that relies on the user's proxy
> > server - there are proxies which won't allow a CONNECT method for ssh.
>
> Exactly, so goes for CVS and GIT.
>
Aye, in this case the users won't be able to proxy from their environments.


>
> > > So, we do not have a 100% coverage, and quite some is left out.
> > >
> > Please name, I will try my best, to send a V2 covering those methods.
> > Currently it support the 5 types that are used in the packages.
>
> Well, I agree that it does cover all the methods Buildroot uses, but not
> all the methods Buildroot uses.
>
You mean Buildroot could use?


>
> What are we gonna explain our users that the proxy settings he put in
> the menuconfig are not being used, because a package in its BR2_EXTERNAL
> tree uses a download method that we do not know how to proxify?
>
Exactly. Buildroot would support using http proxies (if allowed by the
proxy administrator)
with download methods that are pre-packaged in Buildroot. I still think
that's fair, more than
telling to solve it for yourself.


>
> > > So, I prefer we not have any of this proxy settings in Buildroot at
> all,
> > > and just rely on the user's environment to be properly set up.
> > >
> > I think setting up all methods in the user environment is possible, but
> > sometimes can be quite complicated.
>
> Well, it is really easy:
>
>     export http-proxy='http://user:pass at proxy:port/'
>     export https_proxy="${http_proxy}"
>     export ftp_proxy="${http_proxy}"
>     . tsocks -on
>

> Et voil?, all things are proxified (one just has to edit /etc/tsocks.conf
> but that is something we really can do ourselves anyway.... Yes, I know
> we can use TSOCKS_CONF_FILE=/path/to/my/tsocks.conf)
>
And nope - just as you said, this would work with wget, bzr, hg, and would
not with
cvs, git://, scp & ssh. For those the user would have to setup individual
startup configs.
Also, while I personally like Socks, I find it a very rarely used protocol
outside of socks
through ssh.


>
> > Thank you for your opinion, could you state in which case would you
> accept
> > the patch?
> > I think I can add CVS support, and ssh/scp support - which means ALL 8
> > download helpers would be able to
> > use http proxies.
> > And again, this is a http proxy patch, I didn't aim to support all proxy
> > types in the world, as that would be
> > probably pointless.
>
> I still believe we should not have this in Buildroot at all. That is my
> opinion, others may disagree with me. Let them speak up! ;-)
>
I can accept it (even if I don't agree with it).

Than again, from another point of view, if somebody can not solve
him/herself the
using of a proxy, why would he/she want to compile a complete embedded linux
system from sources...


>
> I know this can be a bit frustrating, but I can assure you I have
> already gone down this road in the past, and it is not a road I would
> like to go down again...
>
OK.


>
> To me, the best would be to just document that Buildroot uses whatever
> settings are set in the user's system.
>
> BTW, I should know about proxies: absolutely everything I do on my home
> machine *is* proxified via either an http or a socks proxy. Even this
> very mail will leave my home network via a proxy... ;-]
>
:D

Best regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141022/eb10635b/attachment.html>

[Buildroot] [PATCH 1/1] Add proxy download support

[Yann E. MORIN]

Karoly, All,

On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
> This patch adds support for downloading sources over a http proxy.
> Supports wget with http, https & ftp targets, bzr, hg, svn.
> Git does work with forcing https instead of git protocol.
> Cvs is not supported, but Buildroot does not currently have any
> packages relying on cvs repos (plus this would fall back to the
> Buildroot mirror over http).
> 
> Signed-off-by: Karoly Kasza <kaszak@gmail.com>

After speaking with Peter, and given the feedback from both Arnout and
I, we believe this belongs to the user's settings, and has no place in
Buildroot.

So, I've marked this patch as Rejected in patchwork.

Regards,
Yann E. MORIN.

> ---
>  Config.in                |   25 +++++++++++++++++++++++++
>  Makefile                 |    9 +++++++++
>  support/download/wrapper |   26 ++++++++++++++++++++++++++
>  3 files changed, 60 insertions(+)
> 
> diff --git a/Config.in b/Config.in
> index 9cefcbc..2e2b493 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR
>  
>  endmenu
>  
> +config BR2_PROXY
> +	bool "Use http proxy"
> +	help
> +	  Use a http proxy when downloading sources with various protocols.
> +
> +if BR2_PROXY
> +
> +config BR2_PROXY_URL
> +	string "Http proxy URL"
> +	help
> +	  Http proxy URL in the form of <IP or FQDN>:<port>.
> +
> +config BR2_PROXY_USER
> +	string "Http proxy username"
> +	help
> +	  Http proxy username (if needed).
> +
> +config BR2_PROXY_PASS
> +	string "Http proxy password"
> +	help
> +	  Http proxy password (if needed).
> +	  PLEASE NOTE! This password will be saved in the .config file!
> +
> +endif
> +
>  config BR2_JLEVEL
>  	int "Number of jobs to run simultaneously (0 for auto)"
>  	default "0"
> diff --git a/Makefile b/Makefile
> index 907a0fc..5dd40e8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton
>  # should not be used as the root filesystem.
>  TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
>  
> +ifeq ($(BR2_PROXY),y)
> +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL))
> +export BR_PROXY_URL
> +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER))
> +export BR_PROXY_USER
> +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS))
> +export BR_PROXY_PASS
> +endif
> +
>  ifeq ($(BR2_CCACHE),y)
>  CCACHE := $(HOST_DIR)/usr/bin/ccache
>  BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))
> diff --git a/support/download/wrapper b/support/download/wrapper
> index 8ae2797..b224474 100755
> --- a/support/download/wrapper
> +++ b/support/download/wrapper
> @@ -44,6 +44,32 @@ tmpf="${tmpd}/output"
>  # Doing the 'cd' here rather than in all helpers is easier.
>  cd "${tmpd}"
>  
> +# Set the proxy environment variables.
> +# cvs doesn't support proxies OOB, but there is currently no cvs
> +# based package in Buildroot. Also, in that case after the download
> +# fails, the wget method would be used from sources.buildroot.net.
> +if [ -n "${BR_PROXY_URL}" ]; then
> +# SVN does not support http_proxy environment variables, so we use
> +# it's commandline arguments
> +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1`
> +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2`
> +SVN="${SVN} --config-option servers:global:http-proxy-host=${proxy_host} \
> +--config-option servers:global:http-proxy-port=${proxy_port}"
> +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then
> +SVN="${SVN} --config-option servers:global:http-proxy-username=${BR_PROXY_USER} \
> +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}"
> +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}"
> +else
> +proxy="http://${BR_PROXY_URL}"
> +fi
> +export http_proxy=${proxy}
> +export https_proxy=${proxy}
> +export ftp_proxy=${proxy}
> +# While git does support http_proxy environment variables, we need to use
> +# the config option below to force proxying over https instead of git
> +GIT="${GIT} -c url.https://.insteadOf=git://"
> +fi
> +
>  # If the helper fails, we can just remove the temporary directory to
>  # remove all the cruft it may have left behind. Then we just exit in
>  # error too.
> -- 
> 1.7.10.4
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/1] Add proxy download support

[Károly Kasza]

Ok, I understand.

Best regards,
Karoly
On 25 Oct 2014 01:24, "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Karoly, All,
>
> On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
> > This patch adds support for downloading sources over a http proxy.
> > Supports wget with http, https & ftp targets, bzr, hg, svn.
> > Git does work with forcing https instead of git protocol.
> > Cvs is not supported, but Buildroot does not currently have any
> > packages relying on cvs repos (plus this would fall back to the
> > Buildroot mirror over http).
> >
> > Signed-off-by: Karoly Kasza <kaszak@gmail.com>
>
> After speaking with Peter, and given the feedback from both Arnout and
> I, we believe this belongs to the user's settings, and has no place in
> Buildroot.
>
> So, I've marked this patch as Rejected in patchwork.
>
> Regards,
> Yann E. MORIN.
>
> > ---
> >  Config.in                |   25 +++++++++++++++++++++++++
> >  Makefile                 |    9 +++++++++
> >  support/download/wrapper |   26 ++++++++++++++++++++++++++
> >  3 files changed, 60 insertions(+)
> >
> > diff --git a/Config.in b/Config.in
> > index 9cefcbc..2e2b493 100644
> > --- a/Config.in
> > +++ b/Config.in
> > @@ -253,6 +253,31 @@ config BR2_CPAN_MIRROR
> >
> >  endmenu
> >
> > +config BR2_PROXY
> > +     bool "Use http proxy"
> > +     help
> > +       Use a http proxy when downloading sources with various protocols.
> > +
> > +if BR2_PROXY
> > +
> > +config BR2_PROXY_URL
> > +     string "Http proxy URL"
> > +     help
> > +       Http proxy URL in the form of <IP or FQDN>:<port>.
> > +
> > +config BR2_PROXY_USER
> > +     string "Http proxy username"
> > +     help
> > +       Http proxy username (if needed).
> > +
> > +config BR2_PROXY_PASS
> > +     string "Http proxy password"
> > +     help
> > +       Http proxy password (if needed).
> > +       PLEASE NOTE! This password will be saved in the .config file!
> > +
> > +endif
> > +
> >  config BR2_JLEVEL
> >       int "Number of jobs to run simultaneously (0 for auto)"
> >       default "0"
> > diff --git a/Makefile b/Makefile
> > index 907a0fc..5dd40e8 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -338,6 +338,15 @@ TARGET_SKELETON = $(TOPDIR)/system/skeleton
> >  # should not be used as the root filesystem.
> >  TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
> >
> > +ifeq ($(BR2_PROXY),y)
> > +BR_PROXY_URL = $(call qstrip,$(BR2_PROXY_URL))
> > +export BR_PROXY_URL
> > +BR_PROXY_USER = $(call qstrip,$(BR2_PROXY_USER))
> > +export BR_PROXY_USER
> > +BR_PROXY_PASS = $(call qstrip,$(BR2_PROXY_PASS))
> > +export BR_PROXY_PASS
> > +endif
> > +
> >  ifeq ($(BR2_CCACHE),y)
> >  CCACHE := $(HOST_DIR)/usr/bin/ccache
> >  BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR))
> > diff --git a/support/download/wrapper b/support/download/wrapper
> > index 8ae2797..b224474 100755
> > --- a/support/download/wrapper
> > +++ b/support/download/wrapper
> > @@ -44,6 +44,32 @@ tmpf="${tmpd}/output"
> >  # Doing the 'cd' here rather than in all helpers is easier.
> >  cd "${tmpd}"
> >
> > +# Set the proxy environment variables.
> > +# cvs doesn't support proxies OOB, but there is currently no cvs
> > +# based package in Buildroot. Also, in that case after the download
> > +# fails, the wget method would be used from sources.buildroot.net.
> > +if [ -n "${BR_PROXY_URL}" ]; then
> > +# SVN does not support http_proxy environment variables, so we use
> > +# it's commandline arguments
> > +proxy_host=`echo $BR_PROXY_URL | cut -d":" -f1`
> > +proxy_port=`echo $BR_PROXY_URL | cut -d":" -f2`
> > +SVN="${SVN} --config-option
> servers:global:http-proxy-host=${proxy_host} \
> > +--config-option servers:global:http-proxy-port=${proxy_port}"
> > +if [ -n "${BR_PROXY_USER}" ] && [ -n "${BR_PROXY_PASS}" ]; then
> > +SVN="${SVN} --config-option
> servers:global:http-proxy-username=${BR_PROXY_USER} \
> > +--config-option servers:global:http-proxy-password=${BR_PROXY_PASS}"
> > +proxy="http://${BR_PROXY_USER}:${BR_PROXY_PASS}@${BR_PROXY_URL}"
> > +else
> > +proxy="http://${BR_PROXY_URL}"
> > +fi
> > +export http_proxy=${proxy}
> > +export https_proxy=${proxy}
> > +export ftp_proxy=${proxy}
> > +# While git does support http_proxy environment variables, we need to
> use
> > +# the config option below to force proxying over https instead of git
> > +GIT="${GIT} -c url.https://.insteadOf=git://"
> > +fi
> > +
> >  # If the helper fails, we can just remove the temporary directory to
> >  # remove all the cruft it may have left behind. Then we just exit in
> >  # error too.
> > --
> > 1.7.10.4
> >
>
> --
>
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
>      |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There
> is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
>  conspiracy.  |
>
> '------------------------------^-------^------------------^--------------------'
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141025/5018238a/attachment.html>

[Buildroot] [PATCH 1/1] Add proxy download support

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Fri, 24 Oct 2014 16:23:55 -0700, Yann E. MORIN wrote:

> On 2014-10-21 16:53 +0200, Karoly Kasza spake thusly:
> > This patch adds support for downloading sources over a http proxy.
> > Supports wget with http, https & ftp targets, bzr, hg, svn.
> > Git does work with forcing https instead of git protocol.
> > Cvs is not supported, but Buildroot does not currently have any
> > packages relying on cvs repos (plus this would fall back to the
> > Buildroot mirror over http).
> > 
> > Signed-off-by: Karoly Kasza <kaszak@gmail.com>
> 
> After speaking with Peter, and given the feedback from both Arnout and
> I, we believe this belongs to the user's settings, and has no place in
> Buildroot.

I agree with this decision, but I certainly wouldn't be opposed to an
addition to the manual that explains what to use Buildroot through
proxies. At least the relevant pointers to the appropriate environment
variables, some hints, etc.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com