make check bot delays

make check bot delays

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

Hi,

TL;DR: the jenkins instance running make check bot hangs daily, looking for a solution

In the past two weeks the make check bot has experienced troubles for which I've been unable to find a cause. The same jenkins instance running it for the past nine month now freezes at random times. Nothing in the logs, no error message: either the page or the slave are just doing nothing no matter what. The only reliable solution seems to be a restart of jenkins (killing the process since even restart blocks). 

Maybe it's time for the make check bot to run via teuthology instead of jenkins.

Thanks for your patience

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: make check bot delays

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

Hi,

TL;DR: the make check bot is fixed (no more delays) but only keeps the last 30 runs

It looks like the cause of the jenkins delays was the number of run kept in the archives (set to 300). Lowering it down to 30 have jenkins back to running with no delay between jobs (as opposed to using 100% of one core during 10 minutes between two jobs, apparently to do things on the archived runs).

What I can't really explain is that it also kept 300 jobs two month ago and that was no issue. What changed in between... I have no clue.

The bot is catching up.

Cheers

On 15/10/2015 00:06, Loic Dachary wrote:
> Hi,
> 
> TL;DR: the jenkins instance running make check bot hangs daily, looking for a solution
> 
> In the past two weeks the make check bot has experienced troubles for which I've been unable to find a cause. The same jenkins instance running it for the past nine month now freezes at random times. Nothing in the logs, no error message: either the page or the slave are just doing nothing no matter what. The only reliable solution seems to be a restart of jenkins (killing the process since even restart blocks). 
> 
> Maybe it's time for the make check bot to run via teuthology instead of jenkins.
> 
> Thanks for your patience
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: make check bot delays

[Dałek, Piotr]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Loic Dachary
> Sent: Thursday, October 15, 2015 11:09 AM
> 
> Hi,
> 
> TL;DR: the make check bot is fixed (no more delays) but only keeps the last
> 30 runs
>[..] 
> What I can't really explain is that it also kept 300 jobs two month ago and that
> was no issue. What changed in between... I have no clue.

Maybe there was some change in Java configuration (memory bounds in particular)?

With best regards / Pozdrawiam
Piotr Dałek

Re: make check bot delays

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]



On 15/10/2015 11:45, Dałek, Piotr wrote:
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Loic Dachary
>> Sent: Thursday, October 15, 2015 11:09 AM
>>
>> Hi,
>>
>> TL;DR: the make check bot is fixed (no more delays) but only keeps the last
>> 30 runs
>> [..] 
>> What I can't really explain is that it also kept 300 jobs two month ago and that
>> was no issue. What changed in between... I have no clue.
> 
> Maybe there was some change in Java configuration (memory bounds in particular)?

I don't think any kind of change or update or anything was done on the jenkins running the bot (and doing nothing else) in the past 9 months.

Weird.

> 
> With best regards / Pozdrawiam
> Piotr Dałek
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]