bitbake-server does not exit after build ends

bitbake-server does not exit after build ends

[Böszörményi Zoltán]

Hi,

I am not sure what's the actual problem here.

FWIW, I have recently upgraded my laptop to Fedora 39
where I test recipe changes first.

Now, whenever a build ends (no matter if successfully or
if there was a recipe build failure) two bitbake-server
processes stay running.

As a result, the next bitbake run spits out messages about
the previous instance holding bitbake.lock and retries.

Fedora 39 has Python 3.12 but maybe an outdated python
module is the issue.

I am happy to try any ideas.

Thanks in advance,
Zoltán

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

2023. 11. 27. 10:52 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:
> Hi,
>
> I am not sure what's the actual problem here.
>
> FWIW, I have recently upgraded my laptop to Fedora 39
> where I test recipe changes first.
>
> Now, whenever a build ends (no matter if successfully or
> if there was a recipe build failure) two bitbake-server
> processes stay running.

Builds for mickledore and nanbield both show the same issue.

>
> As a result, the next bitbake run spits out messages about
> the previous instance holding bitbake.lock and retries.
>
> Fedora 39 has Python 3.12 but maybe an outdated python
> module is the issue.
>
> I am happy to try any ideas.
>
> Thanks in advance,
> Zoltán
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191263): https://lists.openembedded.org/g/openembedded-core/message/191263
> Mute This Topic: https://lists.openembedded.org/mt/102826186/3617728
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [zboszor@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bitbake-server does not exit after build ends

[Alexander Kanavin]

Fedora 39 isn't yet supported or tested, so it's almost certainly
you've hit a python 3.12 compatibility issue. If you can look into,
that would be very appreciated.

Alex

On Mon, 27 Nov 2023 at 10:52, Zoltan Boszormenyi <zboszor@gmail.com> wrote:
>
> Hi,
>
> I am not sure what's the actual problem here.
>
> FWIW, I have recently upgraded my laptop to Fedora 39
> where I test recipe changes first.
>
> Now, whenever a build ends (no matter if successfully or
> if there was a recipe build failure) two bitbake-server
> processes stay running.
>
> As a result, the next bitbake run spits out messages about
> the previous instance holding bitbake.lock and retries.
>
> Fedora 39 has Python 3.12 but maybe an outdated python
> module is the issue.
>
> I am happy to try any ideas.
>
> Thanks in advance,
> Zoltán
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191263): https://lists.openembedded.org/g/openembedded-core/message/191263
> Mute This Topic: https://lists.openembedded.org/mt/102826186/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bitbake-server does not exit after build ends

[Richard Purdie]

On Mon, 2023-11-27 at 10:52 +0100, Zoltan Boszormenyi wrote:
> Hi,
> 
> I am not sure what's the actual problem here.
> 
> FWIW, I have recently upgraded my laptop to Fedora 39
> where I test recipe changes first.
> 
> Now, whenever a build ends (no matter if successfully or
> if there was a recipe build failure) two bitbake-server
> processes stay running.
> 
> As a result, the next bitbake run spits out messages about
> the previous instance holding bitbake.lock and retries.
> 
> Fedora 39 has Python 3.12 but maybe an outdated python
> module is the issue.
> 
> I am happy to try any ideas.

Is there anything interesting in the bitbake-cookerdaemon.log file?

Cheers,

Richard

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

2023. 11. 27. 11:52 keltezéssel, Richard Purdie írta:
> Is there anything interesting in the bitbake-cookerdaemon.log file?

Actually, yes.

This is obviously the most suspicious:

/usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process 
(pid=1605202) is multi-threaded, use of fork() may lead to deadlocks in the child.
   self.pid = os.fork()
/home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210: DeprecationWarning: This process 
(pid=1605590) is multi-threaded, use of fork() may lead to deadlocks in the child.
   pid = os.fork()
/home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210: DeprecationWarning: This process 
(pid=1607833) is multi-threaded, use of fork() may lead to deadlocks in the child.
   pid = os.fork()

There are a few other deprecation warnings, too,
which are not relevant to this particular problem
but may be interesting for the future, nevertheless:

/home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:260: DeprecationWarning: ast.Str is 
deprecated and will be removed in Python 3.14; use ast.Constant instead
   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
/home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:263: DeprecationWarning: Attribute 
s is deprecated and will be removed in Python 3.14; use value instead
   self.contains[varname].add(node.args[1].s)

ruby.bbclass has a lot of these at different line numbers:

/home/zozo/test-yocto-4.3/conf/../layers/meta-cloud-services/classes/ruby.bbclass:28: 
SyntaxWarning: invalid escape sequence '\.'
   r = re.compile("ruby ([0-9]+\.[0-9]+\.[0-9]+)*")

These lines occur at the end (probably normal):

1605202 10:39:58.854287 Parse cache valid
1605202 10:40:13.098694 Registering idle function <function 
BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7fa3e22e79c0>
1605202 10:40:13.098736 Removing idle function <bound method Command.runAsyncCommand of 
<bb.command.Command object at 0x7fa41259d580>>
1605202 12:17:02.357378 Running command ['stateShutdown']
1605202 12:17:02.357622 Sending reply (None, None)
1605202 12:17:02.357819 Command Completed (socket: True)

And the remaining child processes:

$ ps auxw | grep bitb | grep -v grep
zozo     1605202  2.3  3.5 1333384 1151808 ?     Sl   10:39   2:29 /usr/bin/python3 
/home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5 
/home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log /home/zozo/test-yocto-4.3/bitbake.lock 
/home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0
zozo     1605208  0.0  1.4 559372 466260 ?       S    10:39   0:00 /usr/bin/python3 
/home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5 
/home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log /home/zozo/test-yocto-4.3/bitbake.lock 
/home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0

Re: [OE-core] bitbake-server does not exit after build ends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3574 bytes --]

I can confirm the same on gentoo with python-3.12.

On Mon, Nov 27, 2023 at 12:29 PM Zoltan Boszormenyi <zboszor@gmail.com>
wrote:

> 2023. 11. 27. 11:52 keltezéssel, Richard Purdie írta:
> > Is there anything interesting in the bitbake-cookerdaemon.log file?
>
> Actually, yes.
>
> This is obviously the most suspicious:
>
> /usr/lib64/python3.12/multiprocessing/popen_fork.py:66:
> DeprecationWarning: This process
> (pid=1605202) is multi-threaded, use of fork() may lead to deadlocks in
> the child.
>    self.pid = os.fork()
> /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210:
> DeprecationWarning: This process
> (pid=1605590) is multi-threaded, use of fork() may lead to deadlocks in
> the child.
>    pid = os.fork()
> /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210:
> DeprecationWarning: This process
> (pid=1607833) is multi-threaded, use of fork() may lead to deadlocks in
> the child.
>    pid = os.fork()
>
> There are a few other deprecation warnings, too,
> which are not relevant to this particular problem
> but may be interesting for the future, nevertheless:
>
> /home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:260:
> DeprecationWarning: ast.Str is
> deprecated and will be removed in Python 3.14; use ast.Constant instead
>    if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
> /home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:263:
> DeprecationWarning: Attribute
> s is deprecated and will be removed in Python 3.14; use value instead
>    self.contains[varname].add(node.args[1].s)
>
> ruby.bbclass has a lot of these at different line numbers:
>
> /home/zozo/test-yocto-4.3/conf/../layers/meta-cloud-services/classes/ruby.bbclass:28:
>
> SyntaxWarning: invalid escape sequence '\.'
>    r = re.compile("ruby ([0-9]+\.[0-9]+\.[0-9]+)*")
>
> These lines occur at the end (probably normal):
>
> 1605202 10:39:58.854287 Parse cache valid
> 1605202 10:40:13.098694 Registering idle function <function
> BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7fa3e22e79c0>
> 1605202 10:40:13.098736 Removing idle function <bound method
> Command.runAsyncCommand of
> <bb.command.Command object at 0x7fa41259d580>>
> 1605202 12:17:02.357378 Running command ['stateShutdown']
> 1605202 12:17:02.357622 Sending reply (None, None)
> 1605202 12:17:02.357819 Command Completed (socket: True)
>
> And the remaining child processes:
>
> $ ps auxw | grep bitb | grep -v grep
> zozo     1605202  2.3  3.5 1333384 1151808 ?     Sl   10:39   2:29
> /usr/bin/python3
> /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5
> /home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log
> /home/zozo/test-yocto-4.3/bitbake.lock
> /home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0
> zozo     1605208  0.0  1.4 559372 466260 ?       S    10:39   0:00
> /usr/bin/python3
> /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5
> /home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log
> /home/zozo/test-yocto-4.3/bitbake.lock
> /home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191294):
> https://lists.openembedded.org/g/openembedded-core/message/191294
> Mute This Topic: https://lists.openembedded.org/mt/102826186/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4512 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

So it's not just Fedora, but Python 3.12 in general.

Googling for problem with threads + fork reveals that
it is indeed an issue even in C / POSIX: only the thread
executing fork() will become part of the new process,
but not the main process starting the thread or any
other threads from the original process.

The warning is justified, and apparently the issue
not occurring in bitbake with previous Python versions
is pure luck.

2023. 11. 27. 14:14 keltezéssel, Martin Jansa írta:
> I can confirm the same on gentoo with python-3.12.
>
> On Mon, Nov 27, 2023 at 12:29 PM Zoltan Boszormenyi <zboszor@gmail.com> wrote:
>
>     2023. 11. 27. 11:52 keltezéssel, Richard Purdie írta:
>     > Is there anything interesting in the bitbake-cookerdaemon.log file?
>
>     Actually, yes.
>
>     This is obviously the most suspicious:
>
>     /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This
>     process
>     (pid=1605202) is multi-threaded, use of fork() may lead to deadlocks in the child.
>        self.pid = os.fork()
>     /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210: DeprecationWarning: This
>     process
>     (pid=1605590) is multi-threaded, use of fork() may lead to deadlocks in the child.
>        pid = os.fork()
>     /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-worker:210: DeprecationWarning: This
>     process
>     (pid=1607833) is multi-threaded, use of fork() may lead to deadlocks in the child.
>        pid = os.fork()
>
>     There are a few other deprecation warnings, too,
>     which are not relevant to this particular problem
>     but may be interesting for the future, nevertheless:
>
>     /home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:260: DeprecationWarning:
>     ast.Str is
>     deprecated and will be removed in Python 3.14; use ast.Constant instead
>        if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>     /home/zozo/test-yocto-4.3/bitbake/lib/bb/codeparser.py:263: DeprecationWarning:
>     Attribute
>     s is deprecated and will be removed in Python 3.14; use value instead
>        self.contains[varname].add(node.args[1].s)
>
>     ruby.bbclass has a lot of these at different line numbers:
>
>     /home/zozo/test-yocto-4.3/conf/../layers/meta-cloud-services/classes/ruby.bbclass:28:
>     SyntaxWarning: invalid escape sequence '\.'
>        r = re.compile("ruby ([0-9]+\.[0-9]+\.[0-9]+)*")
>
>     These lines occur at the end (probably normal):
>
>     1605202 10:39:58.854287 Parse cache valid
>     1605202 10:40:13.098694 Registering idle function <function
>     BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7fa3e22e79c0>
>     1605202 10:40:13.098736 Removing idle function <bound method Command.runAsyncCommand of
>     <bb.command.Command object at 0x7fa41259d580>>
>     1605202 12:17:02.357378 Running command ['stateShutdown']
>     1605202 12:17:02.357622 Sending reply (None, None)
>     1605202 12:17:02.357819 Command Completed (socket: True)
>
>     And the remaining child processes:
>
>     $ ps auxw | grep bitb | grep -v grep
>     zozo     1605202  2.3  3.5 1333384 1151808 ?     Sl   10:39 2:29 /usr/bin/python3
>     /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5
>     /home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log
>     /home/zozo/test-yocto-4.3/bitbake.lock
>     /home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0
>     zozo     1605208  0.0  1.4 559372 466260 ?       S    10:39 0:00 /usr/bin/python3
>     /home/zozo/test-yocto-4.3/bitbake/bin/bitbake-server decafbad 3 5
>     /home/zozo/test-yocto-4.3/bitbake-cookerdaemon.log
>     /home/zozo/test-yocto-4.3/bitbake.lock
>     /home/zozo/test-yocto-4.3/bitbake.sock 0 0 None 0
>
>
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>     View/Reply Online (#191294):
>     https://lists.openembedded.org/g/openembedded-core/message/191294
>     Mute This Topic: https://lists.openembedded.org/mt/102826186/3617156
>     Group Owner: openembedded-core+owner@lists.openembedded.org
>     <mailto:openembedded-core%2Bowner@lists.openembedded.org>
>     Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
>     [martin.jansa@gmail.com]
>     -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bitbake-server does not exit after build ends

[Richard Purdie]

On Tue, 2023-11-28 at 10:26 +0100, Böszörményi Zoltán wrote:
> So it's not just Fedora, but Python 3.12 in general.
> 
> Googling for problem with threads + fork reveals that
> it is indeed an issue even in C / POSIX: only the thread
> executing fork() will become part of the new process,
> but not the main process starting the thread or any
> other threads from the original process.
> 
> The warning is justified, and apparently the issue
> not occurring in bitbake with previous Python versions
> is pure luck.

I'm not sure that is entirely true/fair. The way bitbake has executed
processes and handled threads was very carefully written deliberately
to avoid some of these kinds of issues so it wasn't "pure luck". What I
don't know is what changed in 3.12 to mean this now is potentially
breaking.

Cheers,

Richard

Re: [OE-core] bitbake-server does not exit after build ends

[Alexander Kanavin]

On Tue, 28 Nov 2023 at 11:00, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I'm not sure that is entirely true/fair. The way bitbake has executed
> processes and handled threads was very carefully written deliberately
> to avoid some of these kinds of issues so it wasn't "pure luck". What I
> don't know is what changed in 3.12 to mean this now is potentially
> breaking.

With my python 3.12 recipe I'm getting a different issue when that
recipe is used to produce python in buildtools, and much sooner:
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8593/steps/13/logs/stdio

I've reproduced it locally, and looking into it. It seems as though
hash server is not creating the unix socket file, but isn't failing to
start either.

Alex

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

2023. 11. 28. 11:00 keltezéssel, Richard Purdie írta:
> On Tue, 2023-11-28 at 10:26 +0100, Böszörményi Zoltán wrote:
>> So it's not just Fedora, but Python 3.12 in general.
>>
>> Googling for problem with threads + fork reveals that
>> it is indeed an issue even in C / POSIX: only the thread
>> executing fork() will become part of the new process,
>> but not the main process starting the thread or any
>> other threads from the original process.
>>
>> The warning is justified, and apparently the issue
>> not occurring in bitbake with previous Python versions
>> is pure luck.
> I'm not sure that is entirely true/fair.

Maybe not fair. Still, true. See this piece:

http://www.doublersolutions.com/docs/dce/osfdocs/htmls/develop/appdev/Appde193.htm

Other discussions, like this one confirm the same:
https://stackoverflow.com/questions/39890363/what-happens-when-a-thread-forks

>   The way bitbake has executed
> processes and handled threads was very carefully written deliberately
> to avoid some of these kinds of issues so it wasn't "pure luck". What I
> don't know is what changed in 3.12 to mean this now is potentially
> breaking.

The warning only occurs if os.fork() is called when
there are already Python threads.

I guess the general performance uplift in Python 3.12
can trigger a pre-existing issue in bitbake more easily.

It is entirely possible that the threads bitbake creates do not
exist yet with Python 3.11 and earlier at the time of executing
os.fork(). ¯\_(ツ)_/¯

Re: [OE-core] bitbake-server does not exit after build ends

[Alexander Kanavin]

On Tue, 28 Nov 2023 at 11:31, Zoltan Boszormenyi <zboszor@gmail.com> wrote:
> The warning only occurs if os.fork() is called when
> there are already Python threads.
>
> I guess the general performance uplift in Python 3.12
> can trigger a pre-existing issue in bitbake more easily.
>
> It is entirely possible that the threads bitbake creates do not
> exist yet with Python 3.11 and earlier at the time of executing
> os.fork(). ¯\_(ツ)_/¯

I think you might be too quick to jump to conclusions here. I've fixed
the async issue I've been observing (backporting
https://github.com/python/cpython/commit/2e5d4e24ebcd671915427b6fd35a5b0eea412c4a
), and when running bitbake with 3.12 from buildtools-tarball,
everything works fine, and I'm not seeing any leftover processes. Poky
revision is current master:
https://git.yoctoproject.org/poky/commit/?h=master&id=3e50e45917831d9da2d86e69fb908a1a78483b62

Alex

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

2023. 11. 28. 15:50 keltezéssel, Alexander Kanavin írta:
> On Tue, 28 Nov 2023 at 11:31, Zoltan Boszormenyi <zboszor@gmail.com> wrote:
>> The warning only occurs if os.fork() is called when
>> there are already Python threads.
>>
>> I guess the general performance uplift in Python 3.12
>> can trigger a pre-existing issue in bitbake more easily.
>>
>> It is entirely possible that the threads bitbake creates do not
>> exist yet with Python 3.11 and earlier at the time of executing
>> os.fork(). ¯\_(ツ)_/¯
> I think you might be too quick to jump to conclusions here.

Indeed.
I should have added "assuming Python is not the buggy party here".
Sorry.

>   I've fixed
> the async issue I've been observing (backporting
> https://github.com/python/cpython/commit/2e5d4e24ebcd671915427b6fd35a5b0eea412c4a
> ), and when running bitbake with 3.12 from buildtools-tarball,
> everything works fine, and I'm not seeing any leftover processes. Poky
> revision is current master:
> https://git.yoctoproject.org/poky/commit/?h=master&id=3e50e45917831d9da2d86e69fb908a1a78483b62

This means that 3.12.1 should fix this problem, and that release
is supposedly "imminent": 3.12.0 was released on 2023-10-02 and
subsequent bugfix releases are scheduled every two months.

Re: [OE-core] bitbake-server does not exit after build ends

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On 2023-11-28 4:26 a.m., Zoltan Boszormenyi via lists.openembedded.org 
wrote:
> So it's not just Fedora, but Python 3.12 in general.
>
> Googling for problem with threads + fork reveals that
> it is indeed an issue even in C / POSIX: only the thread
> executing fork() will become part of the new process,
> but not the main process starting the thread or any
> other threads from the original process.
>
> The warning is justified, and apparently the issue
> not occurring in bitbake with previous Python versions
> is pure luck.

FYI: Fixed in Fedora 39 if you update to get python-3.12.1-1 or later.


-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 1158 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Thu, Dec 21, 2023 at 7:47 PM Randy MacLeod <randy.macleod@windriver.com>
wrote:

> On 2023-11-28 4:26 a.m., Zoltan Boszormenyi via lists.openembedded.org
> wrote:
>
> So it's not just Fedora, but Python 3.12 in general.
>
> Googling for problem with threads + fork reveals that
> it is indeed an issue even in C / POSIX: only the thread
> executing fork() will become part of the new process,
> but not the main process starting the thread or any
> other threads from the original process.
>
> The warning is justified, and apparently the issue
> not occurring in bitbake with previous Python versions
> is pure luck.
>
> FYI: Fixed in Fedora 39 if you update to get python-3.12.1-1 or later.
>

But this fixes it only for bitbake master, right?

With bitbake 2.0 used in kirkstone I still see bitbake processes still
running after the build is finished.

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On 2023-12-21 1:56 p.m., Martin Jansa wrote:
> On Thu, Dec 21, 2023 at 7:47 PM Randy MacLeod 
> <randy.macleod@windriver.com> wrote:
>
>     On 2023-11-28 4:26 a.m., Zoltan Boszormenyi via
>     lists.openembedded.org
>     <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!dmJtTKioa9DzcUCj9Bnd0SAhV_iwzhIPFdUGgYGvfCcHJTBSqwE1j07pX4x-RmoIfylxSh1gB9shlAshIeRlbYI0P5Q$>
>     wrote:
>>     So it's not just Fedora, but Python 3.12 in general.
>>
>>     Googling for problem with threads + fork reveals that
>>     it is indeed an issue even in C / POSIX: only the thread
>>     executing fork() will become part of the new process,
>>     but not the main process starting the thread or any
>>     other threads from the original process.
>>
>>     The warning is justified, and apparently the issue
>>     not occurring in bitbake with previous Python versions
>>     is pure luck.
>
>     FYI: Fixed in Fedora 39 if you update to get python-3.12.1-1 or later.
>
>
> But this fixes it only for bitbake master, right?
Even master might still have some problems.
I can do a simple build of a package but in doing:

$ bitbake -k world

I see:
NOTE: No reply from server in 30s (for command ping at 17:50:35.510798)

After some time.

I'm not going to try to debug it at this point in the day/year!

>
> With bitbake 2.0 used in kirkstone I still see bitbake processes still 
> running after the build is finished.

I didn't try kirkstone. Maybe Steve has that on his list of things to do 
in his spare time! ;-)

../Randy


-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2: Type: text/html, Size: 3285 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Alexander Kanavin]

For what it's worth, I am not seeing any of these issues (hanging
processes after bitbake exit, timeouts on world builds with all
meta-oe-layers) with python 3.12 provided by buildtools-tarball. The
issues are probably real, but it's also possible python 3.12 per se is
not to blame.

It would probably help if you check if what you're seeing occurs with
python 3.12 from the tarball, which you can build from this branch:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/py3.12
(the log in the web ui isn't updated yet, I just pushed)

Alex

On Fri, 22 Dec 2023 at 00:06, Randy MacLeod via lists.openembedded.org
<randy.macleod=windriver.com@lists.openembedded.org> wrote:
>
> On 2023-12-21 1:56 p.m., Martin Jansa wrote:
>
> On Thu, Dec 21, 2023 at 7:47 PM Randy MacLeod <randy.macleod@windriver.com> wrote:
>>
>> On 2023-11-28 4:26 a.m., Zoltan Boszormenyi via lists.openembedded.org wrote:
>>
>> So it's not just Fedora, but Python 3.12 in general.
>>
>> Googling for problem with threads + fork reveals that
>> it is indeed an issue even in C / POSIX: only the thread
>> executing fork() will become part of the new process,
>> but not the main process starting the thread or any
>> other threads from the original process.
>>
>> The warning is justified, and apparently the issue
>> not occurring in bitbake with previous Python versions
>> is pure luck.
>>
>> FYI: Fixed in Fedora 39 if you update to get python-3.12.1-1 or later.
>
>
> But this fixes it only for bitbake master, right?
>
> Even master might still have some problems.
> I can do a simple build of a package but in doing:
>
> $ bitbake -k world
>
> I see:
> NOTE: No reply from server in 30s (for command ping at 17:50:35.510798)
>
> After some time.
>
> I'm not going to try to debug it at this point in the day/year!
>
>
> With bitbake 2.0 used in kirkstone I still see bitbake processes still running after the build is finished.
>
> I didn't try kirkstone. Maybe Steve has that on his list of things to do in his spare time! ;-)
>
> ../Randy
>
>
> --
> # Randy MacLeod
> # Wind River Linux
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192854): https://lists.openembedded.org/g/openembedded-core/message/192854
> Mute This Topic: https://lists.openembedded.org/mt/102826186/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bitbake-server does not exit after build ends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 19152 bytes --]

FWIW: I see the processes running after bitbake exit only in kirkstone and
only with
PRSERV_HOST = "localhost:0"
if I drop PRSERV_HOST then all 3 bitbake-server processes exist together
with bitbake itself.

luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
luneos-kirkstone $ ps aux | grep luneos-kirkstone
martin   4184043  0.0  0.0   9124  2048 pts/4    S+   16:47   0:00 grep
--colour=auto luneos-kirkstone
luneos-kirkstone $ bitbake -k zlib-native
NOTE: Started PRServer with DBfile:
/OE/build/luneos-kirkstone/cache/prserv.sqlite3, Address: 127.0.0.1:35811,
PID: 4184924
Loading cache: 100%
|##########################################################################################################################################################################################################|
Time: 0:00:04
Loaded 4570 entries from dependency cache.
...
Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100%
match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be rerun
and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 15 seconds
luneos-kirkstone $ ps aux | grep luneos-kirkstone
martin     51324  0.0  0.0   9124  2048 pts/4    S+   16:54   0:00 grep
--colour=auto luneos-kirkstone
martin   4184584  8.9  0.4 668196 536892 ?       Sl   16:47   0:37
bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
/OE/build/luneos-kirkstone/bitbake.lock
/OE/build/luneos-kirkstone/bitbake.sock 0 None 0
martin   4184924  0.0  0.1 340280 196428 ?       S    16:47   0:00
bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
/OE/build/luneos-kirkstone/bitbake.lock
/OE/build/luneos-kirkstone/bitbake.sock 0 None 0
martin   4184927  0.0  0.1 340328 194440 ?       S    16:47   0:00
bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
/OE/build/luneos-kirkstone/bitbake.lock
/OE/build/luneos-kirkstone/bitbake.sock 0 None 0

luneos-kirkstone $ cat bitbake-cookerdaemon.log
4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at
2024-01-05 16:47:38.367526 ---
4184584 16:47:38.370315 Started bitbake server pid 4184584
4184584 16:47:38.370831 Entering server connection loop
4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1,
proto=0, laddr=bitbake.sock>] ([])
4184584 16:47:38.376431 Processing Client
4184584 16:47:38.376633 Connecting Client
4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
4184584 16:47:38.379355 Command Completed
4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False,
'force': False, 'invalidate_stamp': None, 'dry_run': False,
'dump_signatures': [], 'extra_assume_provided': [], 'profile': False,
'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False,
'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly':
None, 'writeeventlog': None, 'build_verbose_shell': False,
'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains':
{}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS': 'MACHINE DISTRO
TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy
SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS
PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER
WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos',
'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID':
'1894', 'PWD': '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME':
'/home/martin-gentoo', 'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH':
'/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin',
'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER':
'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838',
'WINDOWID': '35651587', 'COLORTERM': 'truecolor',
'CSF_MDTVTexturesDirectory': '/usr/share/opencascade/resources/Textures',
'MACHINES': 'qemux86-64', 'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M
--shift 5', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1',
'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME':
'/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK':
'/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf
/etc/ca-certificates.conf /etc/revdep-rebuild', 'TERMCAP':
'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual
terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:',
'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES':
'/usr/share/opencascade/resources/StdResource/MigrationSheet.txt',
'WINDOW': '3', 'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce',
'CSF_STEPDefaults': '/usr/share/opencascade/resources/XSTEPResource',
'EDITOR': 'vim', 'GTK_MODULES': 'canberra-gtk-module', 'XDG_SEAT': 'seat0',
'CONFIG_PROTECT': '/var/bind /usr/share/gnupg/qualified.txt
/usr/share/easy-rsa /usr/share/config
/usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE',
'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH':
'/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man',
'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults':
'/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia',
'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8',
'LS_COLORS':
'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:',
'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr':
'/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11',
'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0',
'QT_GRAPHICSSYSTEM': 'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory':
'/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource':
'/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH':
'/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info',
'MOZ_GMP_PATH':
'/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed',
'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC':
'/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS':
'user', 'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME':
'/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color',
'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s',
'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults':
'/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults':
'/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager',
'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0',
'CSF_XCAFDefaults': '/usr/share/opencascade/resources/StdResource',
'SHLVL': '2', 'PAGER': '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2',
'CSF_PluginDefaults': '/usr/share/opencascade/resources/StdResource',
'XDG_SESSION_ID': '2', 'CASROOT': '/usr', 'XDG_RUNTIME_DIR':
'/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND': 'dconf', 'PS1':
'\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w
\\$\\[\\033[00m\\] ', 'CSF_XSMessage':
'/usr/share/opencascade/resources/XSMessage', 'XDG_DATA_DIRS':
'/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop',
'CSF_UnitsDefinition':
'/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY': '2067.x',
'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS':
'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone',
'MAIL': '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_':
'/OE/build/luneos-kirkstone/bitbake/bin/bitbake'},
['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
DeprecationWarning: ast.Str is deprecated and will be removed in Python
3.14; use ast.Constant instead
  if isinstance(node.args[0], ast.Str):
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
DeprecationWarning: Attribute s is deprecated and will be removed in Python
3.14; use value instead
  varname = node.args[0].s
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
DeprecationWarning: ast.Str is deprecated and will be removed in Python
3.14; use ast.Constant instead
  if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
DeprecationWarning: Attribute s is deprecated and will be removed in Python
3.14; use value instead
  self.contains[varname].add(node.args[1].s)
/usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning:
This process (pid=4184584) is multi-threaded, use of fork() may lead to
deadlocks in the child.
  self.pid = os.fork()
4184584 16:47:45.108449 Command Completed
4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
4184584 16:47:45.113080 Command Completed
4184584 16:47:45.113399 Running command ['getVariable',
'BBINCLUDELOGS_LINES']
4184584 16:47:45.113925 Command Completed
4184584 16:47:45.114218 Running command ['getSetVariable', 'BB_CONSOLELOG']
4184584 16:47:45.114956 Command Completed
4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
4184584 16:47:45.116590 Command Completed
4184584 16:47:45.122232 Running command ['getUIHandlerNum']
4184584 16:47:45.122805 Command Completed
4184584 16:47:45.123194 Running command ['setEventMask', 1, 20,
{'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19},
['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY',
'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase',
'bb.event.ParseStarted', 'bb.event.ParseProgress',
'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted',
'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted',
'bb.command.CommandFailed', 'bb.command.CommandExit',
'bb.command.CommandCompleted', 'bb.cooker.CookerExit',
'bb.event.MultipleProviders', 'bb.event.NoProvider',
'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted',
'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed',
'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded',
'bb.build.TaskFailedSilent', 'bb.build.TaskProgress',
'bb.event.ProcessStarted', 'bb.event.ProcessProgress',
'bb.event.ProcessFinished']]
4184584 16:47:45.123558 Command Completed
4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
4184584 16:47:45.124228 Command Completed
4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
4184584 16:47:45.124858 Command Completed
4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'],
'build']
4184584 16:47:45.125294 Command Completed
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
DeprecationWarning: ast.Str is deprecated and will be removed in Python
3.14; use ast.Constant instead
  if isinstance(node.args[0], ast.Str):
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
DeprecationWarning: Attribute s is deprecated and will be removed in Python
3.14; use value instead
  varname = node.args[0].s
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
DeprecationWarning: ast.Str is deprecated and will be removed in Python
3.14; use ast.Constant instead
  if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
DeprecationWarning: Attribute s is deprecated and will be removed in Python
3.14; use value instead
  self.contains[varname].add(node.args[1].s)
/usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed
<StreamWriter transport=<_SelectorSocketTransport closing fd=18 read=idle
write=<idle, bufsize=0>> reader=<StreamReader
transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle,
bufsize=0>>>>
  warnings.warn(f"unclosed {self!r}", ResourceWarning)
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
DeprecationWarning: ast.Str is deprecated and will be removed in Python
3.14; use ast.Constant instead
  if isinstance(node.args[0], ast.Str):
/OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
DeprecationWarning: Attribute s is deprecated and will be removed in Python
3.14; use value instead
  varname = node.args[0].s
4184584 16:48:37.909973 Processing Client
4184584 16:48:37.910125 Disconnecting Client
4184584 16:48:37.910729 No timeout, exiting.
4184584 16:48:38.011341 Exiting

It is reproducible with nanbield as well and fixed in scarthgap, will
bisect it when time allows.

Regards,

On Sun, Dec 31, 2023 at 11:03 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> For what it's worth, I am not seeing any of these issues (hanging
> processes after bitbake exit, timeouts on world builds with all
> meta-oe-layers) with python 3.12 provided by buildtools-tarball. The
> issues are probably real, but it's also possible python 3.12 per se is
> not to blame.
>
> It would probably help if you check if what you're seeing occurs with
> python 3.12 from the tarball, which you can build from this branch:
> https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/py3.12
> (the log in the web ui isn't updated yet, I just pushed)
>
> Alex
>
> On Fri, 22 Dec 2023 at 00:06, Randy MacLeod via lists.openembedded.org
> <randy.macleod=windriver.com@lists.openembedded.org> wrote:
> >
> > On 2023-12-21 1:56 p.m., Martin Jansa wrote:
> >
> > On Thu, Dec 21, 2023 at 7:47 PM Randy MacLeod <
> randy.macleod@windriver.com> wrote:
> >>
> >> On 2023-11-28 4:26 a.m., Zoltan Boszormenyi via lists.openembedded.org
> wrote:
> >>
> >> So it's not just Fedora, but Python 3.12 in general.
> >>
> >> Googling for problem with threads + fork reveals that
> >> it is indeed an issue even in C / POSIX: only the thread
> >> executing fork() will become part of the new process,
> >> but not the main process starting the thread or any
> >> other threads from the original process.
> >>
> >> The warning is justified, and apparently the issue
> >> not occurring in bitbake with previous Python versions
> >> is pure luck.
> >>
> >> FYI: Fixed in Fedora 39 if you update to get python-3.12.1-1 or later.
> >
> >
> > But this fixes it only for bitbake master, right?
> >
> > Even master might still have some problems.
> > I can do a simple build of a package but in doing:
> >
> > $ bitbake -k world
> >
> > I see:
> > NOTE: No reply from server in 30s (for command ping at 17:50:35.510798)
> >
> > After some time.
> >
> > I'm not going to try to debug it at this point in the day/year!
> >
> >
> > With bitbake 2.0 used in kirkstone I still see bitbake processes still
> running after the build is finished.
> >
> > I didn't try kirkstone. Maybe Steve has that on his list of things to do
> in his spare time! ;-)
> >
> > ../Randy
> >
> >
> > --
> > # Randy MacLeod
> > # Wind River Linux
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#192854):
> https://lists.openembedded.org/g/openembedded-core/message/192854
> > Mute This Topic: https://lists.openembedded.org/mt/102826186/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 23116 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 17248 bytes --]

On Fri, Jan 5, 2024 at 5:06 PM Martin Jansa via lists.openembedded.org
<martin.jansa=gmail.com@lists.openembedded.org> wrote:

> FWIW: I see the processes running after bitbake exit only in kirkstone and
> only with
> PRSERV_HOST = "localhost:0"
> if I drop PRSERV_HOST then all 3 bitbake-server processes exist together
> with bitbake itself.
>
> luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
> luneos-kirkstone $ ps aux | grep luneos-kirkstone
> martin   4184043  0.0  0.0   9124  2048 pts/4    S+   16:47   0:00 grep
> --colour=auto luneos-kirkstone
> luneos-kirkstone $ bitbake -k zlib-native
> NOTE: Started PRServer with DBfile:
> /OE/build/luneos-kirkstone/cache/prserv.sqlite3, Address: 127.0.0.1:35811,
> PID: 4184924
> Loading cache: 100%
> |##########################################################################################################################################################################################################|
> Time: 0:00:04
> Loaded 4570 entries from dependency cache.
> ...
> Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100%
> match, 100% complete)
> NOTE: Executing Tasks
> NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be
> rerun and all succeeded.
> NOTE: Writing buildhistory
> NOTE: Writing buildhistory took: 15 seconds
> luneos-kirkstone $ ps aux | grep luneos-kirkstone
> martin     51324  0.0  0.0   9124  2048 pts/4    S+   16:54   0:00 grep
> --colour=auto luneos-kirkstone
> martin   4184584  8.9  0.4 668196 536892 ?       Sl   16:47   0:37
> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
> /OE/build/luneos-kirkstone/bitbake.lock
> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
> martin   4184924  0.0  0.1 340280 196428 ?       S    16:47   0:00
> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
> /OE/build/luneos-kirkstone/bitbake.lock
> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
> martin   4184927  0.0  0.1 340328 194440 ?       S    16:47   0:00
> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
> /OE/build/luneos-kirkstone/bitbake.lock
> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>
> luneos-kirkstone $ cat bitbake-cookerdaemon.log
> 4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at
> 2024-01-05 16:47:38.367526 ---
> 4184584 16:47:38.370315 Started bitbake server pid 4184584
> 4184584 16:47:38.370831 Entering server connection loop
> 4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1,
> proto=0, laddr=bitbake.sock>] ([])
> 4184584 16:47:38.376431 Processing Client
> 4184584 16:47:38.376633 Connecting Client
> 4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
> 4184584 16:47:38.379355 Command Completed
> 4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False,
> 'force': False, 'invalidate_stamp': None, 'dry_run': False,
> 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False,
> 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False,
> 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly':
> None, 'writeeventlog': None, 'build_verbose_shell': False,
> 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains':
> {}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS': 'MACHINE DISTRO
> TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy
> SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS
> PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER
> WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos',
> 'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID':
> '1894', 'PWD': '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME':
> '/home/martin-gentoo', 'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH':
> '/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin',
> 'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838',
> 'WINDOWID': '35651587', 'COLORTERM': 'truecolor',
> 'CSF_MDTVTexturesDirectory': '/usr/share/opencascade/resources/Textures',
> 'MACHINES': 'qemux86-64', 'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M
> --shift 5', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1',
> 'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME':
> '/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK':
> '/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf
> /etc/ca-certificates.conf /etc/revdep-rebuild', 'TERMCAP':
> 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual
> terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:',
> 'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES':
> '/usr/share/opencascade/resources/StdResource/MigrationSheet.txt',
> 'WINDOW': '3', 'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce',
> 'CSF_STEPDefaults': '/usr/share/opencascade/resources/XSTEPResource',
> 'EDITOR': 'vim', 'GTK_MODULES': 'canberra-gtk-module', 'XDG_SEAT': 'seat0',
> 'CONFIG_PROTECT': '/var/bind /usr/share/gnupg/qualified.txt
> /usr/share/easy-rsa /usr/share/config
> /usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE',
> 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH':
> '/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man',
> 'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults':
> '/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia',
> 'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8',
> 'LS_COLORS':
> 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:',
> 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr':
> '/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11',
> 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0',
> 'QT_GRAPHICSSYSTEM': 'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory':
> '/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource':
> '/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH':
> '/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info',
> 'MOZ_GMP_PATH':
> '/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed',
> 'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC':
> '/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS':
> 'user', 'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME':
> '/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color',
> 'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s',
> 'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults':
> '/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults':
> '/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager',
> 'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0',
> 'CSF_XCAFDefaults': '/usr/share/opencascade/resources/StdResource',
> 'SHLVL': '2', 'PAGER': '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2',
> 'CSF_PluginDefaults': '/usr/share/opencascade/resources/StdResource',
> 'XDG_SESSION_ID': '2', 'CASROOT': '/usr', 'XDG_RUNTIME_DIR':
> '/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND': 'dconf', 'PS1':
> '\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w
> \\$\\[\\033[00m\\] ', 'CSF_XSMessage':
> '/usr/share/opencascade/resources/XSMessage', 'XDG_DATA_DIRS':
> '/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop',
> 'CSF_UnitsDefinition':
> '/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY': '2067.x',
> 'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS':
> 'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone',
> 'MAIL': '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_':
> '/OE/build/luneos-kirkstone/bitbake/bin/bitbake'},
> ['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
> DeprecationWarning: ast.Str is deprecated and will be removed in Python
> 3.14; use ast.Constant instead
>   if isinstance(node.args[0], ast.Str):
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
> DeprecationWarning: Attribute s is deprecated and will be removed in Python
> 3.14; use value instead
>   varname = node.args[0].s
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
> DeprecationWarning: ast.Str is deprecated and will be removed in Python
> 3.14; use ast.Constant instead
>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
> DeprecationWarning: Attribute s is deprecated and will be removed in Python
> 3.14; use value instead
>   self.contains[varname].add(node.args[1].s)
> /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning:
> This process (pid=4184584) is multi-threaded, use of fork() may lead to
> deadlocks in the child.
>   self.pid = os.fork()
> 4184584 16:47:45.108449 Command Completed
> 4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
> 4184584 16:47:45.113080 Command Completed
> 4184584 16:47:45.113399 Running command ['getVariable',
> 'BBINCLUDELOGS_LINES']
> 4184584 16:47:45.113925 Command Completed
> 4184584 16:47:45.114218 Running command ['getSetVariable', 'BB_CONSOLELOG']
> 4184584 16:47:45.114956 Command Completed
> 4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
> 4184584 16:47:45.116590 Command Completed
> 4184584 16:47:45.122232 Running command ['getUIHandlerNum']
> 4184584 16:47:45.122805 Command Completed
> 4184584 16:47:45.123194 Running command ['setEventMask', 1, 20,
> {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19},
> ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY',
> 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase',
> 'bb.event.ParseStarted', 'bb.event.ParseProgress',
> 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted',
> 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted',
> 'bb.command.CommandFailed', 'bb.command.CommandExit',
> 'bb.command.CommandCompleted', 'bb.cooker.CookerExit',
> 'bb.event.MultipleProviders', 'bb.event.NoProvider',
> 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted',
> 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed',
> 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded',
> 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress',
> 'bb.event.ProcessStarted', 'bb.event.ProcessProgress',
> 'bb.event.ProcessFinished']]
> 4184584 16:47:45.123558 Command Completed
> 4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
> 4184584 16:47:45.124228 Command Completed
> 4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
> 4184584 16:47:45.124858 Command Completed
> 4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'],
> 'build']
> 4184584 16:47:45.125294 Command Completed
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
> DeprecationWarning: ast.Str is deprecated and will be removed in Python
> 3.14; use ast.Constant instead
>   if isinstance(node.args[0], ast.Str):
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
> DeprecationWarning: Attribute s is deprecated and will be removed in Python
> 3.14; use value instead
>   varname = node.args[0].s
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
> DeprecationWarning: ast.Str is deprecated and will be removed in Python
> 3.14; use ast.Constant instead
>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
> DeprecationWarning: Attribute s is deprecated and will be removed in Python
> 3.14; use value instead
>   self.contains[varname].add(node.args[1].s)
> /usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed
> <StreamWriter transport=<_SelectorSocketTransport closing fd=18 read=idle
> write=<idle, bufsize=0>> reader=<StreamReader
> transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle,
> bufsize=0>>>>
>   warnings.warn(f"unclosed {self!r}", ResourceWarning)
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
> DeprecationWarning: ast.Str is deprecated and will be removed in Python
> 3.14; use ast.Constant instead
>   if isinstance(node.args[0], ast.Str):
> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
> DeprecationWarning: Attribute s is deprecated and will be removed in Python
> 3.14; use value instead
>   varname = node.args[0].s
> 4184584 16:48:37.909973 Processing Client
> 4184584 16:48:37.910125 Disconnecting Client
> 4184584 16:48:37.910729 No timeout, exiting.
> 4184584 16:48:38.011341 Exiting
>
> It is reproducible with nanbield as well and fixed in scarthgap, will
> bisect it when time allows.
>

bisect was simpler than expected as nanbield is still compatible with newer
bitbake and there weren't that many changes.

nanbield as well as kirkstone can be fixed by cherry-picking:
https://git.openembedded.org/bitbake/commit/?id=d01d684a0 asyncrpc: Add
context manager API

+ Joshua if he sees some risks in cherry-picking this to stable releases.

Cheers

[-- Attachment #2: Type: text/html, Size: 19749 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Steve Sakoman]

On Fri, Jan 5, 2024 at 6:54 AM Martin Jansa <martin.jansa@gmail.com> wrote:
>
> On Fri, Jan 5, 2024 at 5:06 PM Martin Jansa via lists.openembedded.org <martin.jansa=gmail.com@lists.openembedded.org> wrote:
>>
>> FWIW: I see the processes running after bitbake exit only in kirkstone and only with
>> PRSERV_HOST = "localhost:0"
>> if I drop PRSERV_HOST then all 3 bitbake-server processes exist together with bitbake itself.
>>
>> luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
>> luneos-kirkstone $ ps aux | grep luneos-kirkstone
>> martin   4184043  0.0  0.0   9124  2048 pts/4    S+   16:47   0:00 grep --colour=auto luneos-kirkstone
>> luneos-kirkstone $ bitbake -k zlib-native
>> NOTE: Started PRServer with DBfile: /OE/build/luneos-kirkstone/cache/prserv.sqlite3, Address: 127.0.0.1:35811, PID: 4184924
>> Loading cache: 100% |##########################################################################################################################################################################################################| Time: 0:00:04
>> Loaded 4570 entries from dependency cache.
>> ...
>> Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100% match, 100% complete)
>> NOTE: Executing Tasks
>> NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be rerun and all succeeded.
>> NOTE: Writing buildhistory
>> NOTE: Writing buildhistory took: 15 seconds
>> luneos-kirkstone $ ps aux | grep luneos-kirkstone
>> martin     51324  0.0  0.0   9124  2048 pts/4    S+   16:54   0:00 grep --colour=auto luneos-kirkstone
>> martin   4184584  8.9  0.4 668196 536892 ?       Sl   16:47   0:37 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>> martin   4184924  0.0  0.1 340280 196428 ?       S    16:47   0:00 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>> martin   4184927  0.0  0.1 340328 194440 ?       S    16:47   0:00 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>>
>> luneos-kirkstone $ cat bitbake-cookerdaemon.log
>> 4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at 2024-01-05 16:47:38.367526 ---
>> 4184584 16:47:38.370315 Started bitbake server pid 4184584
>> 4184584 16:47:38.370831 Entering server connection loop
>> 4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1, proto=0, laddr=bitbake.sock>] ([])
>> 4184584 16:47:38.376431 Processing Client
>> 4184584 16:47:38.376633 Connecting Client
>> 4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
>> 4184584 16:47:38.379355 Command Completed
>> 4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos', 'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID': '1894', 'PWD': '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME': '/home/martin-gentoo', 'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH': '/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin', 'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838', 'WINDOWID': '35651587', 'COLORTERM': 'truecolor', 'CSF_MDTVTexturesDirectory': '/usr/share/opencascade/resources/Textures', 'MACHINES': 'qemux86-64', 'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M --shift 5', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1', 'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME': '/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK': '/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf /etc/ca-certificates.conf /etc/revdep-rebuild', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES': '/usr/share/opencascade/resources/StdResource/MigrationSheet.txt', 'WINDOW': '3', 'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce', 'CSF_STEPDefaults': '/usr/share/opencascade/resources/XSTEPResource', 'EDITOR': 'vim', 'GTK_MODULES': 'canberra-gtk-module', 'XDG_SEAT': 'seat0', 'CONFIG_PROTECT': '/var/bind /usr/share/gnupg/qualified.txt /usr/share/easy-rsa /usr/share/config /usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE', 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH': '/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man', 'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults': '/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia', 'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:', 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr': '/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'QT_GRAPHICSSYSTEM': 'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory': '/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource': '/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH': '/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info', 'MOZ_GMP_PATH': '/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed', 'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC': '/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS': 'user', 'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME': '/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color', 'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s', 'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults': '/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults': '/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager', 'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0', 'CSF_XCAFDefaults': '/usr/share/opencascade/resources/StdResource', 'SHLVL': '2', 'PAGER': '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2', 'CSF_PluginDefaults': '/usr/share/opencascade/resources/StdResource', 'XDG_SESSION_ID': '2', 'CASROOT': '/usr', 'XDG_RUNTIME_DIR': '/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND': 'dconf', 'PS1': '\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w \\$\\[\\033[00m\\] ', 'CSF_XSMessage': '/usr/share/opencascade/resources/XSMessage', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop', 'CSF_UnitsDefinition': '/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY': '2067.x', 'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone', 'MAIL': '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_': '/OE/build/luneos-kirkstone/bitbake/bin/bitbake'}, ['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
>>   varname = node.args[0].s
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
>>   self.contains[varname].add(node.args[1].s)
>> /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=4184584) is multi-threaded, use of fork() may lead to deadlocks in the child.
>>   self.pid = os.fork()
>> 4184584 16:47:45.108449 Command Completed
>> 4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
>> 4184584 16:47:45.113080 Command Completed
>> 4184584 16:47:45.113399 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
>> 4184584 16:47:45.113925 Command Completed
>> 4184584 16:47:45.114218 Running command ['getSetVariable', 'BB_CONSOLELOG']
>> 4184584 16:47:45.114956 Command Completed
>> 4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
>> 4184584 16:47:45.116590 Command Completed
>> 4184584 16:47:45.122232 Running command ['getUIHandlerNum']
>> 4184584 16:47:45.122805 Command Completed
>> 4184584 16:47:45.123194 Running command ['setEventMask', 1, 20, {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19}, ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted', 'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted', 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted', 'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted', 'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider', 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted', 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed', 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded', 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted', 'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
>> 4184584 16:47:45.123558 Command Completed
>> 4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
>> 4184584 16:47:45.124228 Command Completed
>> 4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
>> 4184584 16:47:45.124858 Command Completed
>> 4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'], 'build']
>> 4184584 16:47:45.125294 Command Completed
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
>>   varname = node.args[0].s
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
>>   self.contains[varname].add(node.args[1].s)
>> /usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed <StreamWriter transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle, bufsize=0>> reader=<StreamReader transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle, bufsize=0>>>>
>>   warnings.warn(f"unclosed {self!r}", ResourceWarning)
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
>>   varname = node.args[0].s
>> 4184584 16:48:37.909973 Processing Client
>> 4184584 16:48:37.910125 Disconnecting Client
>> 4184584 16:48:37.910729 No timeout, exiting.
>> 4184584 16:48:38.011341 Exiting
>>
>> It is reproducible with nanbield as well and fixed in scarthgap, will bisect it when time allows.
>
>
> bisect was simpler than expected as nanbield is still compatible with newer bitbake and there weren't that many changes.
>
> nanbield as well as kirkstone can be fixed by cherry-picking:
> https://git.openembedded.org/bitbake/commit/?id=d01d684a0 asyncrpc: Add context manager API
>
> + Joshua if he sees some risks in cherry-picking this to stable releases.

While we wait for Joshua to comment I'll do a little autobuilder
testing to see if anything blows up with this patch.

Steve

Re: [OE-core] bitbake-server does not exit after build ends

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 17658 bytes --]

On Fri, Jan 5, 2024, 9:54 AM Martin Jansa <martin.jansa@gmail.com> wrote:

> On Fri, Jan 5, 2024 at 5:06 PM Martin Jansa via lists.openembedded.org
> <martin.jansa=gmail.com@lists.openembedded.org> wrote:
>
>> FWIW: I see the processes running after bitbake exit only in kirkstone
>> and only with
>> PRSERV_HOST = "localhost:0"
>> if I drop PRSERV_HOST then all 3 bitbake-server processes exist together
>> with bitbake itself.
>>
>> luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
>> luneos-kirkstone $ ps aux | grep luneos-kirkstone
>> martin   4184043  0.0  0.0   9124  2048 pts/4    S+   16:47   0:00 grep
>> --colour=auto luneos-kirkstone
>> luneos-kirkstone $ bitbake -k zlib-native
>> NOTE: Started PRServer with DBfile:
>> /OE/build/luneos-kirkstone/cache/prserv.sqlite3, Address: 127.0.0.1:35811,
>> PID: 4184924
>> Loading cache: 100%
>> |##########################################################################################################################################################################################################|
>> Time: 0:00:04
>> Loaded 4570 entries from dependency cache.
>> ...
>> Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100%
>> match, 100% complete)
>> NOTE: Executing Tasks
>> NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be
>> rerun and all succeeded.
>> NOTE: Writing buildhistory
>> NOTE: Writing buildhistory took: 15 seconds
>> luneos-kirkstone $ ps aux | grep luneos-kirkstone
>> martin     51324  0.0  0.0   9124  2048 pts/4    S+   16:54   0:00 grep
>> --colour=auto luneos-kirkstone
>> martin   4184584  8.9  0.4 668196 536892 ?       Sl   16:47   0:37
>> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
>> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>> /OE/build/luneos-kirkstone/bitbake.lock
>> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>> martin   4184924  0.0  0.1 340280 196428 ?       S    16:47   0:00
>> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
>> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>> /OE/build/luneos-kirkstone/bitbake.lock
>> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>> martin   4184927  0.0  0.1 340328 194440 ?       S    16:47   0:00
>> bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server
>> decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>> /OE/build/luneos-kirkstone/bitbake.lock
>> /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>>
>> luneos-kirkstone $ cat bitbake-cookerdaemon.log
>> 4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at
>> 2024-01-05 16:47:38.367526 ---
>> 4184584 16:47:38.370315 Started bitbake server pid 4184584
>> 4184584 16:47:38.370831 Entering server connection loop
>> 4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1,
>> proto=0, laddr=bitbake.sock>] ([])
>> 4184584 16:47:38.376431 Processing Client
>> 4184584 16:47:38.376633 Connecting Client
>> 4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
>> 4184584 16:47:38.379355 Command Completed
>> 4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False,
>> 'force': False, 'invalidate_stamp': None, 'dry_run': False,
>> 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False,
>> 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False,
>> 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly':
>> None, 'writeeventlog': None, 'build_verbose_shell': False,
>> 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains':
>> {}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS': 'MACHINE DISTRO
>> TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy
>> SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS
>> PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER
>> WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos',
>> 'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID':
>> '1894', 'PWD': '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME':
>> '/home/martin-gentoo', 'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH':
>> '/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin',
>> 'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838',
>> 'WINDOWID': '35651587', 'COLORTERM': 'truecolor',
>> 'CSF_MDTVTexturesDirectory': '/usr/share/opencascade/resources/Textures',
>> 'MACHINES': 'qemux86-64', 'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M
>> --shift 5', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1',
>> 'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME':
>> '/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK':
>> '/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf
>> /etc/ca-certificates.conf /etc/revdep-rebuild', 'TERMCAP':
>> 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual
>> terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:',
>> 'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES':
>> '/usr/share/opencascade/resources/StdResource/MigrationSheet.txt',
>> 'WINDOW': '3', 'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce',
>> 'CSF_STEPDefaults': '/usr/share/opencascade/resources/XSTEPResource',
>> 'EDITOR': 'vim', 'GTK_MODULES': 'canberra-gtk-module', 'XDG_SEAT': 'seat0',
>> 'CONFIG_PROTECT': '/var/bind /usr/share/gnupg/qualified.txt
>> /usr/share/easy-rsa /usr/share/config
>> /usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE',
>> 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH':
>> '/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man',
>> 'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults':
>> '/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia',
>> 'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8',
>> 'LS_COLORS':
>> 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:',
>> 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr':
>> '/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11',
>> 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0',
>> 'QT_GRAPHICSSYSTEM': 'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory':
>> '/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource':
>> '/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH':
>> '/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info',
>> 'MOZ_GMP_PATH':
>> '/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed',
>> 'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC':
>> '/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS':
>> 'user', 'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME':
>> '/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color',
>> 'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s',
>> 'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults':
>> '/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults':
>> '/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager',
>> 'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0',
>> 'CSF_XCAFDefaults': '/usr/share/opencascade/resources/StdResource',
>> 'SHLVL': '2', 'PAGER': '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2',
>> 'CSF_PluginDefaults': '/usr/share/opencascade/resources/StdResource',
>> 'XDG_SESSION_ID': '2', 'CASROOT': '/usr', 'XDG_RUNTIME_DIR':
>> '/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND': 'dconf', 'PS1':
>> '\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w
>> \\$\\[\\033[00m\\] ', 'CSF_XSMessage':
>> '/usr/share/opencascade/resources/XSMessage', 'XDG_DATA_DIRS':
>> '/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop',
>> 'CSF_UnitsDefinition':
>> '/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY': '2067.x',
>> 'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS':
>> 'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone',
>> 'MAIL': '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_':
>> '/OE/build/luneos-kirkstone/bitbake/bin/bitbake'},
>> ['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
>> DeprecationWarning: ast.Str is deprecated and will be removed in Python
>> 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
>> DeprecationWarning: Attribute s is deprecated and will be removed in Python
>> 3.14; use value instead
>>   varname = node.args[0].s
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
>> DeprecationWarning: ast.Str is deprecated and will be removed in Python
>> 3.14; use ast.Constant instead
>>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
>> DeprecationWarning: Attribute s is deprecated and will be removed in Python
>> 3.14; use value instead
>>   self.contains[varname].add(node.args[1].s)
>> /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning:
>> This process (pid=4184584) is multi-threaded, use of fork() may lead to
>> deadlocks in the child.
>>   self.pid = os.fork()
>> 4184584 16:47:45.108449 Command Completed
>> 4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
>> 4184584 16:47:45.113080 Command Completed
>> 4184584 16:47:45.113399 Running command ['getVariable',
>> 'BBINCLUDELOGS_LINES']
>> 4184584 16:47:45.113925 Command Completed
>> 4184584 16:47:45.114218 Running command ['getSetVariable',
>> 'BB_CONSOLELOG']
>> 4184584 16:47:45.114956 Command Completed
>> 4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
>> 4184584 16:47:45.116590 Command Completed
>> 4184584 16:47:45.122232 Running command ['getUIHandlerNum']
>> 4184584 16:47:45.122805 Command Completed
>> 4184584 16:47:45.123194 Running command ['setEventMask', 1, 20,
>> {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19},
>> ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY',
>> 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase',
>> 'bb.event.ParseStarted', 'bb.event.ParseProgress',
>> 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted',
>> 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted',
>> 'bb.command.CommandFailed', 'bb.command.CommandExit',
>> 'bb.command.CommandCompleted', 'bb.cooker.CookerExit',
>> 'bb.event.MultipleProviders', 'bb.event.NoProvider',
>> 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted',
>> 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed',
>> 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded',
>> 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress',
>> 'bb.event.ProcessStarted', 'bb.event.ProcessProgress',
>> 'bb.event.ProcessFinished']]
>> 4184584 16:47:45.123558 Command Completed
>> 4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
>> 4184584 16:47:45.124228 Command Completed
>> 4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
>> 4184584 16:47:45.124858 Command Completed
>> 4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'],
>> 'build']
>> 4184584 16:47:45.125294 Command Completed
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
>> DeprecationWarning: ast.Str is deprecated and will be removed in Python
>> 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
>> DeprecationWarning: Attribute s is deprecated and will be removed in Python
>> 3.14; use value instead
>>   varname = node.args[0].s
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230:
>> DeprecationWarning: ast.Str is deprecated and will be removed in Python
>> 3.14; use ast.Constant instead
>>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233:
>> DeprecationWarning: Attribute s is deprecated and will be removed in Python
>> 3.14; use value instead
>>   self.contains[varname].add(node.args[1].s)
>> /usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed
>> <StreamWriter transport=<_SelectorSocketTransport closing fd=18 read=idle
>> write=<idle, bufsize=0>> reader=<StreamReader
>> transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle,
>> bufsize=0>>>>
>>   warnings.warn(f"unclosed {self!r}", ResourceWarning)
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228:
>> DeprecationWarning: ast.Str is deprecated and will be removed in Python
>> 3.14; use ast.Constant instead
>>   if isinstance(node.args[0], ast.Str):
>> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229:
>> DeprecationWarning: Attribute s is deprecated and will be removed in Python
>> 3.14; use value instead
>>   varname = node.args[0].s
>> 4184584 16:48:37.909973 Processing Client
>> 4184584 16:48:37.910125 Disconnecting Client
>> 4184584 16:48:37.910729 No timeout, exiting.
>> 4184584 16:48:38.011341 Exiting
>>
>> It is reproducible with nanbield as well and fixed in scarthgap, will
>> bisect it when time allows.
>>
>
> bisect was simpler than expected as nanbield is still compatible with
> newer bitbake and there weren't that many changes.
>
> nanbield as well as kirkstone can be fixed by cherry-picking:
> https://git.openembedded.org/bitbake/commit/?id=d01d684a0 asyncrpc: Add
> context manager API
>
> + Joshua if he sees some risks in cherry-picking this to stable releases.
>


It looks ok. Must be the prserv client that's hanging then?


> Cheers
>
>
>

[-- Attachment #2: Type: text/html, Size: 20694 bytes --]

Re: [OE-core] bitbake-server does not exit after build ends

[Steve Sakoman]

On Fri, Jan 5, 2024 at 7:02 AM Steve Sakoman via
lists.openembedded.org <steve=sakoman.com@lists.openembedded.org>
wrote:
>
> On Fri, Jan 5, 2024 at 6:54 AM Martin Jansa <martin.jansa@gmail.com> wrote:
> >
> > On Fri, Jan 5, 2024 at 5:06 PM Martin Jansa via lists.openembedded.org <martin.jansa=gmail.com@lists.openembedded.org> wrote:
> >>
> >> FWIW: I see the processes running after bitbake exit only in kirkstone and only with
> >> PRSERV_HOST = "localhost:0"
> >> if I drop PRSERV_HOST then all 3 bitbake-server processes exist together with bitbake itself.
> >>
> >> luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
> >> luneos-kirkstone $ ps aux | grep luneos-kirkstone
> >> martin   4184043  0.0  0.0   9124  2048 pts/4    S+   16:47   0:00 grep --colour=auto luneos-kirkstone
> >> luneos-kirkstone $ bitbake -k zlib-native
> >> NOTE: Started PRServer with DBfile: /OE/build/luneos-kirkstone/cache/prserv.sqlite3, Address: 127.0.0.1:35811, PID: 4184924
> >> Loading cache: 100% |##########################################################################################################################################################################################################| Time: 0:00:04
> >> Loaded 4570 entries from dependency cache.
> >> ...
> >> Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100% match, 100% complete)
> >> NOTE: Executing Tasks
> >> NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be rerun and all succeeded.
> >> NOTE: Writing buildhistory
> >> NOTE: Writing buildhistory took: 15 seconds
> >> luneos-kirkstone $ ps aux | grep luneos-kirkstone
> >> martin     51324  0.0  0.0   9124  2048 pts/4    S+   16:54   0:00 grep --colour=auto luneos-kirkstone
> >> martin   4184584  8.9  0.4 668196 536892 ?       Sl   16:47   0:37 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
> >> martin   4184924  0.0  0.1 340280 196428 ?       S    16:47   0:00 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
> >> martin   4184927  0.0  0.1 340328 194440 ?       S    16:47   0:00 bitbake-server /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5 /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
> >>
> >> luneos-kirkstone $ cat bitbake-cookerdaemon.log
> >> 4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at 2024-01-05 16:47:38.367526 ---
> >> 4184584 16:47:38.370315 Started bitbake server pid 4184584
> >> 4184584 16:47:38.370831 Entering server connection loop
> >> 4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1, proto=0, laddr=bitbake.sock>] ([])
> >> 4184584 16:47:38.376431 Processing Client
> >> 4184584 16:47:38.376633 Connecting Client
> >> 4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
> >> 4184584 16:47:38.379355 Command Completed
> >> 4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos', 'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID': '1894', 'PWD': '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME': '/home/martin-gentoo', 'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH': '/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin', 'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER': 'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838', 'WINDOWID': '35651587', 'COLORTERM': 'truecolor', 'CSF_MDTVTexturesDirectory': '/usr/share/opencascade/resources/Textures', 'MACHINES': 'qemux86-64', 'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M --shift 5', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1', 'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME': '/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK': '/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf /etc/ca-certificates.conf /etc/revdep-rebuild', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:', 'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES': '/usr/share/opencascade/resources/StdResource/MigrationSheet.txt', 'WINDOW': '3', 'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce', 'CSF_STEPDefaults': '/usr/share/opencascade/resources/XSTEPResource', 'EDITOR': 'vim', 'GTK_MODULES': 'canberra-gtk-module', 'XDG_SEAT': 'seat0', 'CONFIG_PROTECT': '/var/bind /usr/share/gnupg/qualified.txt /usr/share/easy-rsa /usr/share/config /usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE', 'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH': '/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man', 'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults': '/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia', 'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:', 'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr': '/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'QT_GRAPHICSSYSTEM': 'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory': '/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource': '/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH': '/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info', 'MOZ_GMP_PATH': '/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed', 'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC': '/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS': 'user', 'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME': '/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color', 'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s', 'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults': '/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults': '/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager', 'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0', 'CSF_XCAFDefaults': '/usr/share/opencascade/resources/StdResource', 'SHLVL': '2', 'PAGER': '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2', 'CSF_PluginDefaults': '/usr/share/opencascade/resources/StdResource', 'XDG_SESSION_ID': '2', 'CASROOT': '/usr', 'XDG_RUNTIME_DIR': '/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND': 'dconf', 'PS1': '\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w \\$\\[\\033[00m\\] ', 'CSF_XSMessage': '/usr/share/opencascade/resources/XSMessage', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop', 'CSF_UnitsDefinition': '/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY': '2067.x', 'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone', 'MAIL': '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_': '/OE/build/luneos-kirkstone/bitbake/bin/bitbake'}, ['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
> >>   if isinstance(node.args[0], ast.Str):
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
> >>   varname = node.args[0].s
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
> >>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
> >>   self.contains[varname].add(node.args[1].s)
> >> /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=4184584) is multi-threaded, use of fork() may lead to deadlocks in the child.
> >>   self.pid = os.fork()
> >> 4184584 16:47:45.108449 Command Completed
> >> 4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
> >> 4184584 16:47:45.113080 Command Completed
> >> 4184584 16:47:45.113399 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
> >> 4184584 16:47:45.113925 Command Completed
> >> 4184584 16:47:45.114218 Running command ['getSetVariable', 'BB_CONSOLELOG']
> >> 4184584 16:47:45.114956 Command Completed
> >> 4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
> >> 4184584 16:47:45.116590 Command Completed
> >> 4184584 16:47:45.122232 Running command ['getUIHandlerNum']
> >> 4184584 16:47:45.122805 Command Completed
> >> 4184584 16:47:45.123194 Running command ['setEventMask', 1, 20, {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19}, ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord', 'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted', 'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted', 'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted', 'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted', 'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider', 'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted', 'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed', 'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded', 'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted', 'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
> >> 4184584 16:47:45.123558 Command Completed
> >> 4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
> >> 4184584 16:47:45.124228 Command Completed
> >> 4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
> >> 4184584 16:47:45.124858 Command Completed
> >> 4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'], 'build']
> >> 4184584 16:47:45.125294 Command Completed
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
> >>   if isinstance(node.args[0], ast.Str):
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
> >>   varname = node.args[0].s
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
> >>   if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
> >>   self.contains[varname].add(node.args[1].s)
> >> /usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed <StreamWriter transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle, bufsize=0>> reader=<StreamReader transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle, bufsize=0>>>>
> >>   warnings.warn(f"unclosed {self!r}", ResourceWarning)
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
> >>   if isinstance(node.args[0], ast.Str):
> >> /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
> >>   varname = node.args[0].s
> >> 4184584 16:48:37.909973 Processing Client
> >> 4184584 16:48:37.910125 Disconnecting Client
> >> 4184584 16:48:37.910729 No timeout, exiting.
> >> 4184584 16:48:38.011341 Exiting
> >>
> >> It is reproducible with nanbield as well and fixed in scarthgap, will bisect it when time allows.
> >
> >
> > bisect was simpler than expected as nanbield is still compatible with newer bitbake and there weren't that many changes.
> >
> > nanbield as well as kirkstone can be fixed by cherry-picking:
> > https://git.openembedded.org/bitbake/commit/?id=d01d684a0 asyncrpc: Add context manager API
> >
> > + Joshua if he sees some risks in cherry-picking this to stable releases.
>
> While we wait for Joshua to comment I'll do a little autobuilder
> testing to see if anything blows up with this patch.

FWIW, a-full autobuilder runs for both nanbield and kirkstone with
this patch came back all green.

Steve

> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#193373): https://lists.openembedded.org/g/openembedded-core/message/193373
> Mute This Topic: https://lists.openembedded.org/mt/102826186/3620601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bitbake-server does not exit after build ends

[Böszörményi Zoltán]

2024. 01. 05. 17:54 keltezéssel, Martin Jansa írta:
> On Fri, Jan 5, 2024 at 5:06 PM Martin Jansa via lists.openembedded.org 
> <http://lists.openembedded.org> <martin.jansa=gmail.com@lists.openembedded.org> wrote:
>
>     FWIW: I see the processes running after bitbake exit only in kirkstone and only with
>     PRSERV_HOST = "localhost:0"
>     if I drop PRSERV_HOST then all 3 bitbake-server processes exist together with
>     bitbake itself.
>
>     luneos-kirkstone $ rm -f bitbake-cookerdaemon.log
>     luneos-kirkstone $ ps aux | grep luneos-kirkstone
>     martin   4184043  0.0  0.0   9124  2048 pts/4    S+ 16:47   0:00 grep --colour=auto
>     luneos-kirkstone
>     luneos-kirkstone $ bitbake -k zlib-native
>     NOTE: Started PRServer with DBfile: /OE/build/luneos-kirkstone/cache/prserv.sqlite3,
>     Address: 127.0.0.1:35811 <http://127.0.0.1:35811>, PID: 4184924
>     Loading cache: 100%
>     |##########################################################################################################################################################################################################|
>     Time: 0:00:04
>     Loaded 4570 entries from dependency cache.
>     ...
>     Sstate summary: Wanted 10 Local 10 Mirrors 0 Missed 0 Current 11 (100% match, 100%
>     complete)
>     NOTE: Executing Tasks
>     NOTE: Tasks Summary: Attempted 92 tasks of which 92 didn't need to be rerun and all
>     succeeded.
>     NOTE: Writing buildhistory
>     NOTE: Writing buildhistory took: 15 seconds
>     luneos-kirkstone $ ps aux | grep luneos-kirkstone
>     martin     51324  0.0  0.0   9124  2048 pts/4    S+ 16:54   0:00 grep --colour=auto
>     luneos-kirkstone
>     martin   4184584  8.9  0.4 668196 536892 ?       Sl 16:47   0:37 bitbake-server
>     /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5
>     /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>     /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>     martin   4184924  0.0  0.1 340280 196428 ?       S  16:47   0:00 bitbake-server
>     /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5
>     /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>     /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0 None 0
>     martin   4184927  0.0  0.1 340328 194440 ?       S  16:47   0:00 bitbake-server
>     /OE/build/luneos-kirkstone/bitbake/bin/bitbake-server decafbad 3 5
>     /OE/build/luneos-kirkstone/bitbake-cookerdaemon.log
>     /OE/build/luneos-kirkstone/bitbake.lock /OE/build/luneos-kirkstone/bitbake.sock 0
>     None 0
>
>     luneos-kirkstone $ cat bitbake-cookerdaemon.log
>     4184584 16:47:38.367593 --- Starting bitbake server pid 4184584 at 2024-01-05
>     16:47:38.367526 ---
>     4184584 16:47:38.370315 Started bitbake server pid 4184584
>     4184584 16:47:38.370831 Entering server connection loop
>     4184584 16:47:38.372043 Accepting [<socket.socket fd=6, family=1, type=1, proto=0,
>     laddr=bitbake.sock>] ([])
>     4184584 16:47:38.376431 Processing Client
>     4184584 16:47:38.376633 Connecting Client
>     4184584 16:47:38.377632 Running command ['setFeatures', [2, 1]]
>     4184584 16:47:38.379355 Command Completed
>     4184584 16:47:38.379868 Running command ['updateConfig', {'halt': False, 'force':
>     False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [],
>     'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [],
>     'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene':
>     False, 'runall': None, 'runonly': None, 'writeeventlog': None,
>     'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20,
>     'debug_domains': {}}, {'SHELL': '/bin/bash', 'BB_ENV_PASSTHROUGH_ADDITIONS':
>     'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY
>     no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS
>     PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER
>     WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'DISTRO': 'luneos',
>     'SSH_AUTH_SOCK': '/tmp/ssh-XXXXXX2ZxYBo/agent.1893', 'SSH_AGENT_PID': '1894', 'PWD':
>     '/OE/build/luneos-kirkstone', 'LOGNAME': 'martin', 'HOME': '/home/martin-gentoo',
>     'MACHINE': 'qemux86-64', 'USER': 'martin', 'PATH':
>     '/OE/build/luneos-kirkstone/openembedded-core/scripts:/OE/build/luneos-kirkstone/bitbake/bin:/home/martin-gentoo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/etc/eselect/wine/bin:/usr/lib64/opencascade/bin:/var/lib/snapd/snap/bin',
>     'LC_ALL': 'en_US.UTF-8', 'SESSION_MANAGER':
>     'local/jama:@/tmp/.ICE-unix/1838,unix/jama:/tmp/.ICE-unix/1838', 'WINDOWID':
>     '35651587', 'COLORTERM': 'truecolor', 'CSF_MDTVTexturesDirectory':
>     '/usr/share/opencascade/resources/Textures', 'MACHINES': 'qemux86-64',
>     'XDG_CONFIG_DIRS': '/etc/xdg', 'LESS': '-R -M --shift 5', 'XDG_SESSION_PATH':
>     '/org/freedesktop/DisplayManager/Session1', 'XDG_MENU_PREFIX': 'xfce-', 'JDK_HOME':
>     '/etc/java-config-2/current-system-vm', 'CONFIG_PROTECT_MASK': '/etc/sandbox.d
>     /etc/fonts/fonts.conf /etc/gentoo-release /etc/dconf /etc/ca-certificates.conf
>     /etc/revdep-rebuild', 'TERMCAP': 'SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual
>     terminal:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:li#64:co#238:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:me=\\E[m:ms:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\\E[5i:pf=\\E[4i:Z0=\\E[?3h:Z1=\\E[?3l:Km=\\E[<:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kB=\\E[Z:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:',
>     'JAVA_HOME': '/etc/java-config-2/current-system-vm', 'CSF_MIGRATION_TYPES':
>     '/usr/share/opencascade/resources/StdResource/MigrationSheet.txt', 'WINDOW': '3',
>     'ANT_HOME': '/usr/share/ant', 'DESKTOP_SESSION': 'xfce', 'CSF_STEPDefaults':
>     '/usr/share/opencascade/resources/XSTEPResource', 'EDITOR': 'vim', 'GTK_MODULES':
>     'canberra-gtk-module', 'XDG_SEAT': 'seat0', 'CONFIG_PROTECT': '/var/bind
>     /usr/share/gnupg/qualified.txt /usr/share/easy-rsa /usr/share/config
>     /usr/lib64/libreoffice/program/sofficerc', 'XDG_SESSION_DESKTOP': 'XFCE',
>     'XDG_SESSION_TYPE': 'x11', 'PANEL_GDK_CORE_DEVICE_EVENTS': '0', 'MANPATH':
>     '/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/13/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/17/share/man:/etc/eselect/wine/share/man',
>     'XAUTHORITY': '/tmp/xauth_CEynPY', 'CSF_StandardLiteDefaults':
>     '/usr/share/opencascade/resources/StdResource', 'OPENCL_PROFILE': 'nvidia',
>     'MOTD_SHOWN': 'pam', 'GDM_LANG': 'C.UTF-8', 'LANG': 'en_US.UTF-8', 'LS_COLORS':
>     'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:',
>     'XDG_CURRENT_DESKTOP': 'XFCE', 'VTE_VERSION': '7402', 'CSF_GraphicShr':
>     '/usr/lib64/opencascade/libTKOpenGl.so', 'OPENGL_PROFILE': 'xorg-x11',
>     'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'QT_GRAPHICSSYSTEM':
>     'raster', 'XZ_OPT': '-T0', 'CSF_ShadersDirectory':
>     '/usr/share/opencascade/resources/Shaders', 'CSF_XmlOcafResource':
>     '/usr/share/opencascade/resources/XmlOcafResource', 'INFOPATH':
>     '/usr/share/gcc-data/x86_64-pc-linux-gnu/13/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.41/info:/usr/share/autoconf-2.72/info:/usr/share/autoconf-2.13/info:/usr/share/automake-1.16.5/info:/usr/share/info',
>     'MOZ_GMP_PATH': '/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed',
>     'CSF_SHMessage': '/usr/share/opencascade/resources/SHMessage', 'JAVAC':
>     '/etc/java-config-2/current-system-vm/bin/javac', 'XDG_SESSION_CLASS': 'user',
>     'VBOX_SDK_PATH': '/usr/lib64/virtualbox/sdk', 'ANDROID_HOME':
>     '/opt/android-sdk-update-manager', 'TERM': 'screen.xterm-256color',
>     'VBOX_INSTALL_PATH': '/usr/lib64/virtualbox', 'LESSOPEN': '|lesspipe %s',
>     'ANDROID_SWT': '/usr/share/swt-3.7/lib', 'CSF_StandardDefaults':
>     '/usr/share/opencascade/resources/StdResource', 'CSF_IGESDefaults':
>     '/usr/share/opencascade/resources/XSTEPResource', 'MANPAGER': 'manpager',
>     'NPM_CONFIG_GLOBALCONFIG': '/etc/npm/npmrc', 'DISPLAY': ':0.0', 'CSF_XCAFDefaults':
>     '/usr/share/opencascade/resources/StdResource', 'SHLVL': '2', 'PAGER':
>     '/usr/bin/less', 'LEX': 'flex', 'XDG_VTNR': '2', 'CSF_PluginDefaults':
>     '/usr/share/opencascade/resources/StdResource', 'XDG_SESSION_ID': '2', 'CASROOT':
>     '/usr', 'XDG_RUNTIME_DIR': '/run/user/1026', 'GCC_SPECS': '', 'GSETTINGS_BACKEND':
>     'dconf', 'PS1': '\\[\\033[01;32m\\]OE $MACHINE@$DISTRO\\[\\033[01;34m\\] \\w
>     \\$\\[\\033[00m\\] ', 'CSF_XSMessage': '/usr/share/opencascade/resources/XSMessage',
>     'XDG_DATA_DIRS':
>     '/usr/local/share:/usr/share:/etc/eselect/wine/share:/var/lib/snapd/desktop',
>     'CSF_UnitsDefinition': '/usr/share/opencascade/resources/UnitsAPI/Units.dat', 'STY':
>     '2067.x', 'VBOX_APP_HOME': '/usr/lib64/virtualbox', 'DBUS_SESSION_BUS_ADDRESS':
>     'unix:path=/run/user/1026/bus', 'INITPWD': '/OE/build/luneos-kirkstone', 'MAIL':
>     '/var/mail/martin', 'OLDPWD': '/home/martin-gentoo', '_':
>     '/OE/build/luneos-kirkstone/bitbake/bin/bitbake'},
>     ['/OE/build/luneos-kirkstone/bitbake/bin/bitbake', '-k', 'zlib-native']]
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning:
>     ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>       if isinstance(node.args[0], ast.Str):
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning:
>     Attribute s is deprecated and will be removed in Python 3.14; use value instead
>       varname = node.args[0].s
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning:
>     ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>       if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning:
>     Attribute s is deprecated and will be removed in Python 3.14; use value instead
>       self.contains[varname].add(node.args[1].s)
>     /usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This
>     process (pid=4184584) is multi-threaded, use of fork() may lead to deadlocks in the
>     child.
>       self.pid = os.fork()
>     4184584 16:47:45.108449 Command Completed
>     4184584 16:47:45.112267 Running command ['getVariable', 'BBINCLUDELOGS']
>     4184584 16:47:45.113080 Command Completed
>     4184584 16:47:45.113399 Running command ['getVariable', 'BBINCLUDELOGS_LINES']
>     4184584 16:47:45.113925 Command Completed
>     4184584 16:47:45.114218 Running command ['getSetVariable', 'BB_CONSOLELOG']
>     4184584 16:47:45.114956 Command Completed
>     4184584 16:47:45.115441 Running command ['getSetVariable', 'BB_LOGCONFIG']
>     4184584 16:47:45.116590 Command Completed
>     4184584 16:47:45.122232 Running command ['getUIHandlerNum']
>     4184584 16:47:45.122805 Command Completed
>     4184584 16:47:45.123194 Running command ['setEventMask', 1, 20,
>     {'BitBake.SigGen.HashEquiv': 19, 'BitBake.RunQueue.HashEquiv': 19},
>     ['bb.runqueue.runQueueExitWait', 'bb.event.LogExecTTY', 'logging.LogRecord',
>     'bb.build.TaskFailed', 'bb.build.TaskBase', 'bb.event.ParseStarted',
>     'bb.event.ParseProgress', 'bb.event.ParseCompleted', 'bb.event.CacheLoadStarted',
>     'bb.event.CacheLoadProgress', 'bb.event.CacheLoadCompleted',
>     'bb.command.CommandFailed', 'bb.command.CommandExit', 'bb.command.CommandCompleted',
>     'bb.cooker.CookerExit', 'bb.event.MultipleProviders', 'bb.event.NoProvider',
>     'bb.runqueue.sceneQueueTaskStarted', 'bb.runqueue.runQueueTaskStarted',
>     'bb.runqueue.runQueueTaskFailed', 'bb.runqueue.sceneQueueTaskFailed',
>     'bb.event.BuildBase', 'bb.build.TaskStarted', 'bb.build.TaskSucceeded',
>     'bb.build.TaskFailedSilent', 'bb.build.TaskProgress', 'bb.event.ProcessStarted',
>     'bb.event.ProcessProgress', 'bb.event.ProcessFinished']]
>     4184584 16:47:45.123558 Command Completed
>     4184584 16:47:45.123863 Running command ['getVariable', 'BB_DEFAULT_TASK']
>     4184584 16:47:45.124228 Command Completed
>     4184584 16:47:45.124528 Running command ['setConfig', 'cmd', 'build']
>     4184584 16:47:45.124858 Command Completed
>     4184584 16:47:45.125140 Running command ['buildTargets', ['zlib-native'], 'build']
>     4184584 16:47:45.125294 Command Completed
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning:
>     ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>       if isinstance(node.args[0], ast.Str):
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning:
>     Attribute s is deprecated and will be removed in Python 3.14; use value instead
>       varname = node.args[0].s
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:230: DeprecationWarning:
>     ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>       if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:233: DeprecationWarning:
>     Attribute s is deprecated and will be removed in Python 3.14; use value instead
>       self.contains[varname].add(node.args[1].s)
>     /usr/lib/python3.12/asyncio/streams.py:413: ResourceWarning: unclosed <StreamWriter
>     transport=<_SelectorSocketTransport closing fd=18 read=idle write=<idle, bufsize=0>>
>     reader=<StreamReader transport=<_SelectorSocketTransport closing fd=18 read=idle
>     write=<idle, bufsize=0>>>>
>       warnings.warn(f"unclosed {self!r}", ResourceWarning)
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:228: DeprecationWarning:
>     ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
>       if isinstance(node.args[0], ast.Str):
>     /OE/build/luneos-kirkstone/bitbake/lib/bb/codeparser.py:229: DeprecationWarning:
>     Attribute s is deprecated and will be removed in Python 3.14; use value instead
>       varname = node.args[0].s
>     4184584 16:48:37.909973 Processing Client
>     4184584 16:48:37.910125 Disconnecting Client
>     4184584 16:48:37.910729 No timeout, exiting.
>     4184584 16:48:38.011341 Exiting
>
>     It is reproducible with nanbield as well and fixed in scarthgap, will bisect it when
>     time allows.
>
>
> bisect was simpler than expected as nanbield is still compatible with newer bitbake and 
> there weren't that many changes.
>
> nanbield as well as kirkstone can be fixed by cherry-picking:
> https://git.openembedded.org/bitbake/commit/?id=d01d684a0 asyncrpc: Add context manager API

I have tested it by cherry-picking this commit into bitbake
in nanbield and there are no stuck bitbake-server processes
with this change under Fedora 39.

> + Joshua if he sees some risks in cherry-picking this to stable releases.
>
> Cheers
>