[PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

[PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

[Mingrui Ren]

From d228100b7551b2327d66072b03f2a809744e8f52 Mon Sep 17 00:00:00 2001
From: Mingrui Ren <jiladahe1997@gmail.com>
Date: Thu, 8 Jul 2021 23:44:53 +0800
Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

The enviroment variables used by Fetcher are hard-coded, and are obtained
from HOST env instead of bitbake datastore.
This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 
'BB_ENV_EXTRAWHITE', trying to fix the problems above.

Signed-off-by: Mingrui Ren <jiladahe1997@gmail.com>
---
The situation I encountered was that I tried to use a custom git-proxy by
setting GIT_PROXY_COMMAND and SOCK_PROXY enviroment variables. However, 
I found that even if I set BB_ENV_EXTRAWHITE, the SOCK_PROXY variable does
not work.

 bitbake/lib/bb/fetch2/__init__.py | 16 +---------------
 bitbake/lib/bb/utils.py           |  5 +++++
 scripts/oe-buildenv-internal      |  7 +++++++
 3 files changed, 13 insertions(+), 15 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index c8e91262a9..7cdc6c5ee8 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -820,21 +820,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
     # rather than host provided
     # Also include some other variables.
     # FIXME: Should really include all export varaiables?
-    exportvars = ['HOME', 'PATH',
-                  'HTTP_PROXY', 'http_proxy',
-                  'HTTPS_PROXY', 'https_proxy',
-                  'FTP_PROXY', 'ftp_proxy',
-                  'FTPS_PROXY', 'ftps_proxy',
-                  'NO_PROXY', 'no_proxy',
-                  'ALL_PROXY', 'all_proxy',
-                  'GIT_PROXY_COMMAND',
-                  'GIT_SSH',
-                  'GIT_SSL_CAINFO',
-                  'GIT_SMART_HTTP',
-                  'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
-                  'SOCKS5_USER', 'SOCKS5_PASSWD',
-                  'DBUS_SESSION_BUS_ADDRESS',
-                  'P4CONFIG']
+    exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split()

     if not cleanup:
         cleanup = []
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 6ba1d2a376..6676584845 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -585,6 +585,7 @@ def preserved_envvars():
         'BB_PRESERVE_ENV',
         'BB_ENV_WHITELIST',
         'BB_ENV_EXTRAWHITE',
+        'FETCH_ENV_WHITELIST',
     ]
     return v + preserved_envvars_exported()

@@ -630,6 +631,10 @@ def approved_variables():
         approved.extend(os.environ['BB_ENV_EXTRAWHITE'].split())
         if 'BB_ENV_EXTRAWHITE' not in approved:
             approved.extend(['BB_ENV_EXTRAWHITE'])
+    if 'FETCH_ENV_WHITELIST' in os.environ:
+        approved.extend(os.environ['FETCH_ENV_WHITELIST'].split())
+        if 'FETCH_ENV_WHITELIST' not in approved:
+            approved.extend(['FETCH_ENV_WHITELIST'])
     return approved

 def clean_environment():
diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index e0d920f2fc..39aa2f2444 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -106,6 +106,13 @@ unset BITBAKEDIR newpath
 export BUILDDIR
 export PATH

+FETCH_ENV_WHITELIST="HOME PATH HTTP_PROXY http_proxy HTTPS_PROXY https_proxy \
+FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy NO_PROXY no_proxy ALL_PROXY all_proxy \
+GIT_PROXY_COMMAND GIT_SSH GIT_SSL_CAINFO GIT_SMART_HTTP SSH_AUTH_SOCK SSH_AGENT_PID \
+SOCKS5_USER SOCKS5_PASSWD DBUS_SESSION_BUS_ADDRESS P4CONFIG"
+
+export FETCH_ENV_WHITELIST
+
 BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \
 HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \
 all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \
--
2.20.1

Re: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

[Richard Purdie]

On Fri, 2021-07-09 at 09:44 +0800, jiladahe1997@gmail.com wrote:
> From d228100b7551b2327d66072b03f2a809744e8f52 Mon Sep 17 00:00:00 2001
> From: Mingrui Ren <jiladahe1997@gmail.com>
> Date: Thu, 8 Jul 2021 23:44:53 +0800
> Subject: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'
> 
> The enviroment variables used by Fetcher are hard-coded, and are obtained
> from HOST env instead of bitbake datastore.
> This patch adds a new variable 'FETCH_ENV_WHITELIST' similar to 
> 'BB_ENV_EXTRAWHITE', trying to fix the problems above.
> 
> Signed-off-by: Mingrui Ren <jiladahe1997@gmail.com>
> ---
> The situation I encountered was that I tried to use a custom git-proxy by
> setting GIT_PROXY_COMMAND and SOCK_PROXY enviroment variables. However, 
> I found that even if I set BB_ENV_EXTRAWHITE, the SOCK_PROXY variable does
> not work.
> 
>  bitbake/lib/bb/fetch2/__init__.py | 16 +---------------
>  bitbake/lib/bb/utils.py           |  5 +++++
>  scripts/oe-buildenv-internal      |  7 +++++++
>  3 files changed, 13 insertions(+), 15 deletions(-)

The patch looks like it was generated against poky since that script is in
openembedded-core, not bitbake.

For the patch itself, I'd much rather add something to extend the list
rather than replace it entirely which would make backwards compatibility
easier.

There is also an issue with the variable naming. We have adopted a policy
around inclusive langauage so new variables need to be mindful of that.
We're in the process of trying to work out what to do with some of our
older variables names.

https://wiki.yoctoproject.org/wiki/Inclusive_language

Whatever we do to address this issue really needs to match how we fix the
existing BB_ENV_EXTRAWHITE.

Also, since fetching happens much later, we should be able to set this
variable from config such as bitbake.conf or local.conf and there shouldn't
be any need to export it to the environment.

Cheers,

Richard

Re: [PATCH] Add a new variable:'FETCH_ENV_WHITELIST'

[Mingrui Ren]

Thanks for you reply,


在 2021/7/9 16:50, Richard Purdie 写道:

> On Fri, 2021-07-09 at 09:44 +0800, jiladahe1997@gmail.com wrote:
> The patch looks like it was generated against poky since that script is in
> openembedded-core, not bitbake.
> Whatever we do to address this issue really needs to match how we fix the
> existing BB_ENV_EXTRAWHITE.
>
> Also, since fetching happens much later, we should be able to set this
> variable from config such as bitbake.conf or local.conf and there 
> shouldn't
> be any need to export it to the environment.
Yes, BB_ENV_EXTRAWHITE must be modified to solve this problem gracefully, it
seems that needs patching both of poky and bitbake.
Should I send an e-mail to both poky and bitbake mailling list?


> For the patch itself, I'd much rather add something to extend the list
> rather than replace it entirely which would make backwards compatibility
> easier.
About backwards compatibility, I have considered it, But I'm not sure if you
prefer cleaner code or backward compatibility.

For example:

FETCH_ENV_WHITELIST = "HOME PATH HTTP_PROXY http_proxy ..."
exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split()

This code is cleaner, and better for forwards compatibility.

FETCH_ENV_WHITELIST = ""
exportvars = (d.getVar("FETCH_ENV_WHITELIST") or "").split().extend 
"HOME PATH
HTTP_PROXY http_proxy ..."

This code is better for backwards compatibility.


> There is also an issue with the variable naming. We have adopted a policy
> around inclusive langauage so new variables need to be mindful of that.
> We're in the process of trying to work out what to do with some of our
> older variables names.
>
> https://wiki.yoctoproject.org/wiki/Inclusive_language
I'll fix it.


thanks,
Mingrui Ren