potential patchwork accuracy problems..

potential patchwork accuracy problems..

[Jakub Kicinski]

Hi!

Minor heads up that updating state in patchwork is agony, lately.
I think people are building more and more CI systems, and patchwork
can't handle the write load. So I get timeouts trying to update patch
state 2 of of 3 times. NIPA is struggling, too, but at least it has
auto-retry built in..

So long story short if something seems out of whack in patchwork,
sorry, I'm doing my best :(

Re: potential patchwork accuracy problems..

[Simon Horman]

On Thu, Aug 08, 2024 at 08:54:15AM -0700, Jakub Kicinski wrote:
> Hi!
> 
> Minor heads up that updating state in patchwork is agony, lately.
> I think people are building more and more CI systems, and patchwork
> can't handle the write load. So I get timeouts trying to update patch
> state 2 of of 3 times. NIPA is struggling, too, but at least it has
> auto-retry built in..
> 
> So long story short if something seems out of whack in patchwork,
> sorry, I'm doing my best :(

Do you know if someone who can address the problem is aware of it?
My anecdotal experiences is that access has been degraded, on and off,
for a few months.

Re: potential patchwork accuracy problems..

[Jakub Kicinski]

On Fri, 9 Aug 2024 10:17:38 +0100 Simon Horman wrote:
> On Thu, Aug 08, 2024 at 08:54:15AM -0700, Jakub Kicinski wrote:
> > Minor heads up that updating state in patchwork is agony, lately.
> > I think people are building more and more CI systems, and patchwork
> > can't handle the write load. So I get timeouts trying to update patch
> > state 2 of of 3 times. NIPA is struggling, too, but at least it has
> > auto-retry built in..
> > 
> > So long story short if something seems out of whack in patchwork,
> > sorry, I'm doing my best :(  
> 
> Do you know if someone who can address the problem is aware of it?

I have emailed, and offered help. Didn't see a reply.

> My anecdotal experiences is that access has been degraded, on and off,
> for a few months.

Yes, in the past Konstantin was able to chase down the bad actors, 
I think, as well as do some backend magic to improve the DB performance.
But it keeps happening, and patchwork has no built in protection.
(I need to find some weekend time to parse our own logs and graph 
the number of requests we send to make triple sure NIPA is behaving.)