From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: Mikko Rapeli <mikko.rapeli@linaro.org>
Cc: bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH] command: Avoid time intensive distractions for ping
Date: Mon, 04 Sep 2023 08:07:18 +0100 [thread overview]
Message-ID: <f2989fafee47e8f5b714d976c3f7923978e1f66a.camel@linuxfoundation.org> (raw)
In-Reply-To: <ZPV9aKltBIlkR3U/@nuoska>
On Mon, 2023-09-04 at 09:47 +0300, Mikko Rapeli wrote:
> Hi,
>
> On Thu, Aug 31, 2023 at 01:01:30PM +0100, Richard Purdie wrote:
> > On Thu, 2023-08-31 at 14:30 +0300, Mikko Rapeli wrote:
> > > Hi,
> > >
> > > On Thu, Aug 31, 2023 at 09:56:33AM +0100, Richard Purdie wrote:
> > > > On Thu, 2023-08-31 at 11:51 +0300, Mikko Rapeli wrote:
> > > > > Hi,
> > > > >
> > > > > On Wed, Aug 30, 2023 at 10:33:03PM +0100, Richard Purdie wrote:
> > > > > > We noticed some users were seeing very slow ping response times which
> > > > > > caused 'server timeout' issues. There were some function calls in runCommand
> > > > > > which could be slow such as the inotify callback processing.
> > > > >
> > > > > I guess the error looks like:
> > > > >
> > > > > Timeout while waiting for a reply from the bitbake server (60s)
> > > > >
> > > > > and happens right after a succesful build to me when I try to run any bitbake
> > > > > command.
> > > > >
> > > > > Hope this fixes it. Thanks!
> > > >
> > > > That is the message however the cause may be different in your case. It
> > > > sounds like the server may be taking a long time to shut down for you
> > > > and it can't connect or start a new one whilst that is happening.
> > > >
> > > > If you have the bitbake-cookerdaemon log for your case I can see if
> > > > that sheds any light if this doesn't work for you.
> > >
> > > (Traceback (most recent call last):
> > > File "/home/builder/src/base/poky/bitbake/lib/bb/server/process.py", line 292, in main
> > > self.command_channel_reply.send(self.cooker.command.runCommand(command, self))
> > > File "/home/builder/src/base/poky/bitbake/lib/bb/server/process.py", line 868, in send
> > > self._send(obj)
> > > File "/home/builder/src/base/poky/bitbake/lib/bb/server/process.py", line 847, in _send
> > > self.writer.send_bytes(obj)
> > > File "/usr/lib/python3.10/multiprocessing/connection.py", line 200, in send_bytes
> > > self._send_bytes(m[offset:offset + size])
> > > File "/usr/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
> > > self._send(header + buf)
> > > File "/usr/lib/python3.10/multiprocessing/connection.py", line 368, in _send
> > > n = write(self._handle, buf)
> > > BrokenPipeError: [Errno 32] Broken pipe
> > > )
> > > 4131625 12:21:32.806544 Processing Client
> > > 4131625 12:21:32.806635 Disconnecting Client (socket: True)
> > > 4131625 12:21:32.807999 No timeout, exiting.
> > > 4131625 12:21:32.908254 Exiting (socket: True)
> > > 4131625 12:21:32.982669 More than one thread left?: [<_MainThread(MainThread, started 140013699211712)>, <Thread(asyncio_0, started 140012031006272)>]
> > > 4131625 12:21:34.433909 Exiting as we could obtain the lock
> > > sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/builder/src/base/build/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>
> > >
> > > I don't know what else to look for in the log.
> >
> > What happened before that?
> >
> > The pipe error was the server trying to reply to the client but the
> > client was gone. We should be able to see the command running before
> > that and if it was a ping or not and the timestamps?
>
> bitbake is quite sensitive to IO on my Ubuntu 22.04 build machine
> with zfs. This is how I trigger this reliably:
>
> $ mv tmp foo ; rm -rf foo & bitbake image
> [1] 2709733
> Timeout while waiting for a reply from the bitbake server (60s)
>
> I have rm_work enabled so tmp is not that big except for the deployed files.
That is quite a different usecase to the one being fixed by the patch.
When bitbake starts up there are a few heavy IO things it needs to do
such as loading a hash equivalence database which may be large, loading
the parse cache and iterating through all the metadata to check it's
validity.
If your system has IO challenges, doing that in parallell with a large
deletion may be challenging. If the server doesn't start quickly
enough, you'll see the timeout.
Note that it isn't a ping which is timing out but an actual command.
Later when builds are running, ping is used as a lightweight way to
check the server is still there if there has been no communication for
a while. This patch fixes the ping command to really be lightweight so
that builds don't fail on a loaded system during the build.
It would be nice to find a way to improve the startup so that the
timeouts are avoided too but I'm less sure how to do that. Even if the
ping did work, if the commands themselves are slow/stalling, that is a
different set of challenges...
Cheers,
Richard
prev parent reply other threads:[~2023-09-04 7:07 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-30 21:33 [PATCH] command: Avoid time intensive distractions for ping Richard Purdie
2023-08-31 8:51 ` [bitbake-devel] " Mikko Rapeli
2023-08-31 8:56 ` Richard Purdie
2023-08-31 11:30 ` Mikko Rapeli
2023-08-31 12:01 ` Richard Purdie
2023-09-04 6:47 ` Mikko Rapeli
2023-09-04 7:07 ` Richard Purdie [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f2989fafee47e8f5b714d976c3f7923978e1f66a.camel@linuxfoundation.org \
--to=richard.purdie@linuxfoundation.org \
--cc=bitbake-devel@lists.openembedded.org \
--cc=mikko.rapeli@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.