Re: [docs] [PATCH] bitbake-worker: add header with length of message

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2023-09-21 at 09:56 +0200, Etienne Cordonnier via
lists.yoctoproject.org wrote:
> From: Etienne Cordonnier <ecordonnier@snap.com>
> 
> The IPC mechanism between runqueue.py and bitbake-worker is currently
> not scalable:
> 
> The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker
> has no information about the size of the message. Therefore, the bitbake-worker
> is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")"
> for each chunk received.
> 
> This does not scale, because queue.find has a linear complexity relative to the size of the queue,
> and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI.
> The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each
> iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup
> 35 seconds before this fix, and 1.5 seconds after this fix).
> 
> This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be
> received, and does not need to constantly search the whole queue for </tag>.
> 
> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> ---
>  bin/bitbake-worker | 34 +++++++++++++++++++++++-----------
>  lib/bb/runqueue.py | 34 ++++++++++++++++++++++------------
>  2 files changed, 45 insertions(+), 23 deletions(-)

This does look interesting. The IPC mechanism was never intended to
carry large data and it has changed over time in ways it was never
really designed for.

I'm guessing just one of the objects is particularly large? You mention
lots of files in SRC_URI causing the issue, I'd like to better
understand that.

I'm just wondering whether instead of improving the IPC, we could avoid
some of the data in the first place?

Cheers,

Richard