[OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Zhang, Qiang]

From: Zqiang <qiang.zhang@windriver.com>

The timeout for threading.Lock, threading.Condition, etc, is not using
a monotonic clock, it is affected if the system time (realtime clock)
is set.

This patch will make condvar use monotonic clock.
Refence: https://bugs.python.org/issue41710

Signed-off-by: Zqiang <qiang.zhang@windriver.com>
---
 meta/recipes-devtools/python/python3_3.9.5.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-devtools/python/python3_3.9.5.bb b/meta/recipes-devtools/python/python3_3.9.5.bb
index 82177f4a18..f3f0237c78 100644
--- a/meta/recipes-devtools/python/python3_3.9.5.bb
+++ b/meta/recipes-devtools/python/python3_3.9.5.bb
@@ -73,6 +73,9 @@ DEPENDS = "bzip2-replacement-native libffi bzip2 openssl sqlite3 zlib virtual/li
 DEPENDS_append_class-target = " python3-native"
 DEPENDS_append_class-nativesdk = " python3-native"
 
+# force to use the mutex+cond implementation
+CFLAGS += "-DHAVE_BROKEN_POSIX_SEMAPHORES"
+
 EXTRA_OECONF = " --without-ensurepip --enable-shared --with-platlibdir=${baselib}"
 EXTRA_OECONF_append_class-native = " --bindir=${bindir}/${PN}"
 
-- 
2.17.1

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Steve Sakoman]

On Mon, Aug 2, 2021 at 6:41 PM Zhang, Qiang <qiang.zhang@windriver.com> wrote:
>
> From: Zqiang <qiang.zhang@windriver.com>
>
> The timeout for threading.Lock, threading.Condition, etc, is not using
> a monotonic clock, it is affected if the system time (realtime clock)
> is set.
>
> This patch will make condvar use monotonic clock.
> Refence: https://bugs.python.org/issue41710

Should this be backported to the 3.8.10 and 3.8.11 recipes in dunfell?

Steve

> Signed-off-by: Zqiang <qiang.zhang@windriver.com>
> ---
>  meta/recipes-devtools/python/python3_3.9.5.bb | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/meta/recipes-devtools/python/python3_3.9.5.bb b/meta/recipes-devtools/python/python3_3.9.5.bb
> index 82177f4a18..f3f0237c78 100644
> --- a/meta/recipes-devtools/python/python3_3.9.5.bb
> +++ b/meta/recipes-devtools/python/python3_3.9.5.bb
> @@ -73,6 +73,9 @@ DEPENDS = "bzip2-replacement-native libffi bzip2 openssl sqlite3 zlib virtual/li
>  DEPENDS_append_class-target = " python3-native"
>  DEPENDS_append_class-nativesdk = " python3-native"
>
> +# force to use the mutex+cond implementation
> +CFLAGS += "-DHAVE_BROKEN_POSIX_SEMAPHORES"
> +
>  EXTRA_OECONF = " --without-ensurepip --enable-shared --with-platlibdir=${baselib}"
>  EXTRA_OECONF_append_class-native = " --bindir=${bindir}/${PN}"
>
> --
> 2.17.1
>
>
> 
>

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Ross Burton]

On Tue, 3 Aug 2021 at 05:41, Zhang, Qiang <qiang.zhang@windriver.com> wrote:
> The timeout for threading.Lock, threading.Condition, etc, is not using
> a monotonic clock, it is affected if the system time (realtime clock)
> is set.
>
> This patch will make condvar use monotonic clock.
> Refence: https://bugs.python.org/issue41710

That doesn't quite make sense to me.

As referenced in that bug report,
https://github.com/python/cpython/commit/001fee14e0f2ba5f41fb733adc69d5965925a094
uses monotonic clocks by default.  That was in 3.8.0 onwards.

So, why is this change to pretend that the semaphore is broken needed?

Ross

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

I too do not think this is sufficiently explained. All of python ptests
pass, so there needs to be a demonstrator of incorrect behavior, or let's
just revert it.

Alex

On Mon, 9 Aug 2021 at 19:53, Ross Burton <ross@burtonini.com> wrote:

> On Tue, 3 Aug 2021 at 05:41, Zhang, Qiang <qiang.zhang@windriver.com>
> wrote:
> > The timeout for threading.Lock, threading.Condition, etc, is not using
> > a monotonic clock, it is affected if the system time (realtime clock)
> > is set.
> >
> > This patch will make condvar use monotonic clock.
> > Refence: https://bugs.python.org/issue41710
>
> That doesn't quite make sense to me.
>
> As referenced in that bug report,
>
> https://github.com/python/cpython/commit/001fee14e0f2ba5f41fb733adc69d5965925a094
> uses monotonic clocks by default.  That was in 3.8.0 onwards.
>
> So, why is this change to pretend that the semaphore is broken needed?
>
> Ross
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1593 bytes --]

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Mike Crowe]

On Wednesday 11 August 2021 at 13:36:23 +0200, Alexander Kanavin wrote:
> I too do not think this is sufficiently explained. All of python ptests
> pass, so there needs to be a demonstrator of incorrect behavior, or let's
> just revert it.

I agree regarding the lack of explanation. However, even if the problem is
real (which it looks like it is based on https://bugs.python.org/issue41710)
then it wouldn't be expected to cause test failures unless the system clock
was being warped at inconvenient times during the tests.

Thanks.

Mike.

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

On Wed, 11 Aug 2021 at 18:14, Mike Crowe <mac@mcrowe.com> wrote:

> I agree regarding the lack of explanation. However, even if the problem is
> real (which it looks like it is based on
> https://bugs.python.org/issue41710)
> then it wouldn't be expected to cause test failures unless the system clock
> was being warped at inconvenient times during the tests.
>

So can we just patch python to use the right glibc function with the right
clock type, and send the patch upstream then? Much better than the
workaround that was merged.

Alex

[-- Attachment #2: Type: text/html, Size: 962 bytes --]

Re: [OE-core] [PATCH] python3: use monotonic clock for condvar if possible

[Mike Crowe]

On Wednesday 11 August 2021 at 18:32:31 +0200, Alexander Kanavin wrote:
> On Wed, 11 Aug 2021 at 18:14, Mike Crowe <mac@mcrowe.com> wrote:
> > I agree regarding the lack of explanation. However, even if the problem is
> > real (which it looks like it is based on
> > https://bugs.python.org/issue41710)
> > then it wouldn't be expected to cause test failures unless the system clock
> > was being warped at inconvenient times during the tests.
>
> So can we just patch python to use the right glibc function with the right
> clock type, and send the patch upstream then? Much better than the
> workaround that was merged.

In theory, yes. I've no idea how hard that is though. It probably requires
adding build-time detection of whether the new functions are available to
avoid breaking other/older target operating systems.

Mike.