Re: [meta-virtualization] [[PATCH] botocore: Fix rejecting URLs with unsafe characters in is_valid_endpoint_url()

[Bruce Ashfield <bruce.ashfield@gmail.com>]

merged.

Bruce

In message: [meta-virtualization] [[PATCH] botocore: Fix rejecting URLs with unsafe characters in is_valid_endpoint_url()
on 21/03/2023 Wentao Zhang wrote:

> The function is_valid_endpoint_url() in botocore is designed to validate
> endpoint URLs, but it fails to detect unsafe characters with Python 3.9.5+
> and other versions carrying bpo-43882 fix. The issue is caused by urlsplit()
> silently stripping LF, CR, and HT characters while splitting the URL,
> which disarms the validator in botocore.
> 
> This patch detects unsafe characters in is_valid_endpoint_url() and
> is_valid_ipv6_endpoint_url() early, in order to fix rejecting invalid URLs
> with unsafe characters.
> 
> Signed-off-by: Wentao Zhang <wentao.zhang@windriver.com>
> ---
>  ...Ls-with-unsafe-characters-in-is_vali.patch | 58 +++++++++++++++++++
>  .../python/python3-botocore_1.20.51.bb        |  2 +
>  2 files changed, 60 insertions(+)
>  create mode 100644 recipes-devtools/python/python3-botocore/0001-Fix-rejecting-URLs-with-unsafe-characters-in-is_vali.patch
> 
> diff --git a/recipes-devtools/python/python3-botocore/0001-Fix-rejecting-URLs-with-unsafe-characters-in-is_vali.patch b/recipes-devtools/python/python3-botocore/0001-Fix-rejecting-URLs-with-unsafe-characters-in-is_vali.patch
> new file mode 100644
> index 0000000..6a43608
> --- /dev/null
> +++ b/recipes-devtools/python/python3-botocore/0001-Fix-rejecting-URLs-with-unsafe-characters-in-is_vali.patch
> @@ -0,0 +1,58 @@
> +From 370cdf7d708c92bf21a42f15392f7be330cf8f80 Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?= <mgorny@gentoo.org>
> +Date: Fri, 7 May 2021 19:54:16 +0200
> +Subject: [PATCH] Fix rejecting URLs with unsafe characters in
> + is_valid_endpoint_url() (#2381)
> +
> +Detect unsafe characters in is_valid_endpoint_url()
> +and is_valid_ipv6_endpoint_url() early, in order to fix rejecting
> +invalid URLs with Python 3.9.5+ and other versions carrying bpo-43882
> +fix.  In these versions, urlsplit() silently strips LF, CR and HT
> +characters while splitting the URL, effectively disarming the validator
> +in botocore.
> +
> +The solution is based on a similar fix in Django.
> +
> +Fixes #2377
> +---
> + botocore/utils.py | 10 ++++++++++
> + 1 file changed, 10 insertions(+)
> +
> +diff --git a/botocore/utils.py b/botocore/utils.py
> +index 378972248..d35dd64bb 100644
> +--- a/botocore/utils.py
> ++++ b/botocore/utils.py
> +@@ -173,6 +173,10 @@ ZONE_ID_PAT = "(?:%25|%)(?:[" + UNRESERVED_PAT + "]|%[a-fA-F0-9]{2})+"
> + IPV6_ADDRZ_PAT = r"\[" + IPV6_PAT + r"(?:" + ZONE_ID_PAT + r")?\]"
> + IPV6_ADDRZ_RE = re.compile("^" + IPV6_ADDRZ_PAT + "$")
> + 
> ++# These are the characters that are stripped by post-bpo-43882 urlparse().
> ++UNSAFE_URL_CHARS = frozenset('\t\r\n')
> ++
> ++
> + def ensure_boolean(val):
> +     """Ensures a boolean value if a string or boolean is provided
> + 
> +@@ -977,6 +981,8 @@ class ArgumentGenerator(object):
> + 
> + 
> + def is_valid_ipv6_endpoint_url(endpoint_url):
> ++    if UNSAFE_URL_CHARS.intersection(endpoint_url):
> ++        return False
> +     netloc = urlparse(endpoint_url).netloc
> +     return IPV6_ADDRZ_RE.match(netloc) is not None
> + 
> +@@ -990,6 +996,10 @@ def is_valid_endpoint_url(endpoint_url):
> +     :return: True if the endpoint url is valid. False otherwise.
> + 
> +     """
> ++    # post-bpo-43882 urlsplit() strips unsafe characters from URL, causing
> ++    # it to pass hostname validation below.  Detect them early to fix that.
> ++    if UNSAFE_URL_CHARS.intersection(endpoint_url):
> ++        return False
> +     parts = urlsplit(endpoint_url)
> +     hostname = parts.hostname
> +     if hostname is None:
> +-- 
> +2.25.1
> +
> diff --git a/recipes-devtools/python/python3-botocore_1.20.51.bb b/recipes-devtools/python/python3-botocore_1.20.51.bb
> index ca506f6..f71db1f 100644
> --- a/recipes-devtools/python/python3-botocore_1.20.51.bb
> +++ b/recipes-devtools/python/python3-botocore_1.20.51.bb
> @@ -8,3 +8,5 @@ SRC_URI[sha256sum] = "c853d6c2321e2f2328282c7d49d7b1a06201826ba0e7049c6975ab5f22
>  inherit pypi setuptools3
>  
>  RDEPENDS:${PN} += "python3-jmespath python3-dateutil python3-logging"
> +
> +SRC_URI += "file://0001-Fix-rejecting-URLs-with-unsafe-characters-in-is_vali.patch"
> -- 
> 2.25.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#7950): https://lists.yoctoproject.org/g/meta-virtualization/message/7950
> Mute This Topic: https://lists.yoctoproject.org/mt/97749704/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>