Re: [bitbake-devel] hashserv performance issues

Re: [bitbake-devel] hashserv performance issues

[Richard Purdie]

Joshua reminded me of how to get the server stats (thanks).

For our public server:

$ bitbake-hashclient --address wss://hashserv.yoctoproject.org/ws stats
{
    "requests": {
        "average": 0.06250257539454776,
        "max_time": 16.712204183917493,
        "num": 381569673,
        "stdev": 0.18345958610620752,
        "total_time": 23849087.254955433
    }
}

Compared to my local system:

$ bitbake-hashclient --address ws://localhost:8686 stats
{
    "requests": {
        "average": 0.0004462178160857849,
        "max_time": 0.04537125899992134,
        "num": 3121,
        "stdev": 0.00163338261679226,
        "total_time": 1.3926458040037346
    }
}

For a world build with 38000 tasks, 38000 * 0.0625 = 2375 seconds or 39
minutes, so I suspect the hash server performance is a real issue here
:(

Michael: Is there anything we can do?

Cheers,

Richard

Re: [bitbake-devel] hashserv performance issues

[Jan-Simon Möller]

Hi !

I share your concern ... and also with an eye on PRServ then.
If we solve this here, we should apply the same lesson there. 

Can we rethink this maybe:
- both are "microservices"
- both are serving data stored in a (maybe even distributed) db
- both need the concept of an upstream server (or even servers?) to 'federate'

Essentially these bits should be a common codebase and the APIs a frontend to 
that. Otherwise we will learn lessons twice the hard way and have to write 
more code.


For the performance issue ... 
would a local hashserv + upstream hashserv connection help ?
Or do we need to query things 'always' anyway up to the upstream ... 


I'm willing to help, but would start on the deployment side as thats my 
immediate need as we disabled hashserv with scarthgap and want it back. Once 
this is up I can work my way down to code.


My 0,02€ 

Best,
Jan-Simon

Am Mittwoch, 22. Mai 2024, 15:17:02 CEST schrieb Richard Purdie via 
lists.openembedded.org:
> Joshua reminded me of how to get the server stats (thanks).
> 
> For our public server:
> 
> $ bitbake-hashclient --address wss://hashserv.yoctoproject.org/ws stats
> {
>     "requests": {
>         "average": 0.06250257539454776,
>         "max_time": 16.712204183917493,
>         "num": 381569673,
>         "stdev": 0.18345958610620752,
>         "total_time": 23849087.254955433
>     }
> }
> 
> Compared to my local system:
> 
> $ bitbake-hashclient --address ws://localhost:8686 stats
> {
>     "requests": {
>         "average": 0.0004462178160857849,
>         "max_time": 0.04537125899992134,
>         "num": 3121,
>         "stdev": 0.00163338261679226,
>         "total_time": 1.3926458040037346
>     }
> }
> 
> For a world build with 38000 tasks, 38000 * 0.0625 = 2375 seconds or 39
> minutes, so I suspect the hash server performance is a real issue here
> 
> :(
> 
> Michael: Is there anything we can do?
> 
> Cheers,
> 
> Richard

Re: [bitbake-devel] hashserv performance issues

[Richard Purdie]

On Thu, 2024-05-23 at 12:11 +0200, Jan-Simon Möller wrote:
> I share your concern ... and also with an eye on PRServ then.
> If we solve this here, we should apply the same lesson there. 
> 
> Can we rethink this maybe:
> - both are "microservices"
> - both are serving data stored in a (maybe even distributed) db
> - both need the concept of an upstream server (or even servers?) to
> 'federate'
> 
> Essentially these bits should be a common codebase and the APIs a
> frontend to  that. Otherwise we will learn lessons twice the hard way
> and have to write more code.

PRServ does not have a problem here, it isn't called in
parsing/runqueue codepaths and isn't tightly looped. 

We've talked about the above issues on calls and we still need to
convince Joshua of the need and come up with a plan, along with someone
to do the development work. I'd like to understand Joshua's concerns
and find a way to alleviate them.

> For the performance issue ... 
> would a local hashserv + upstream hashserv connection help ?
> Or do we need to query things 'always' anyway up to the upstream ... 

I doubt unfortunately since it would still need to pass queries through
to the upstream. Geo specific servers may help and was our original
plan for this but we've been struggling with time for the setup.

> I'm willing to help, but would start on the deployment side as thats
> my  immediate need as we disabled hashserv with scarthgap and want it
> back. Once this is up I can work my way down to code.

See above, I'd be happy to see work in that area but from the project
perspective, the performance issue is effectively blocking builds :(.

Cheers,

Richard