Call for jenkins slaves to improve multi operating system support

Call for jenkins slaves to improve multi operating system support

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Hi Ceph,

When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".

The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.

If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:

*  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02

  32 GB RAM
  200 GB SSD
  8 core > 2.5Ghz

*  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02

  4 GB RAM
  200 GB disk
  2 core

*  armv7, armv8 architecture for Ubuntu 14.04

  4 GB RAM
  200 GB disk
  2 core 

Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].

Cheers

[1] pull requests https://github.com/ceph/ceph/pulls
[2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
[3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
[4] gitbuilder http://ceph.com/gitbuilder.cgi
[5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: Call for jenkins slaves to improve multi operating system support

[Duan, Jiangang]

Loric,

do you mean we need give the servers to you or we just build the testing inside our own server room to do all the testing?

-jiangang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
Sent: Wednesday, April 08, 2015 6:56 PM
To: Ceph Development
Subject: Call for jenkins slaves to improve multi operating system support

Hi Ceph,

When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".

The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.

If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:

*  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02

  32 GB RAM
  200 GB SSD
  8 core > 2.5Ghz

*  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02

  4 GB RAM
  200 GB disk
  2 core

*  armv7, armv8 architecture for Ubuntu 14.04

  4 GB RAM
  200 GB disk
  2 core 

Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].

Cheers

[1] pull requests https://github.com/ceph/ceph/pulls
[2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
[3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
[4] gitbuilder http://ceph.com/gitbuilder.cgi [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91

--
Loïc Dachary, Artisan Logiciel Libre

Re: Call for jenkins slaves to improve multi operating system support

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3086 bytes --]



On 08/04/2015 15:21, Duan, Jiangang wrote:
> Loric,
> 
> do you mean we need give the servers to you or we just build the testing inside our own server room to do all the testing?

Thanks for asking, I realize that was not clear. 

The idea is not to donate hardware, because that would require manpower and extra costs to connect to the net. 

What would be useful is a machine connected to the net and dedicated to running a jenkins slave. It receives a build from the jenkins master (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh script that is found at the root of the repository and reports failure / success back to the jenkins master.

Does that make sense ?

> 
> -jiangang
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
> Sent: Wednesday, April 08, 2015 6:56 PM
> To: Ceph Development
> Subject: Call for jenkins slaves to improve multi operating system support
> 
> Hi Ceph,
> 
> When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".
> 
> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.
> 
> If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:
> 
> *  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02
> 
>   32 GB RAM
>   200 GB SSD
>   8 core > 2.5Ghz
> 
> *  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02
> 
>   4 GB RAM
>   200 GB disk
>   2 core
> 
> *  armv7, armv8 architecture for Ubuntu 14.04
> 
>   4 GB RAM
>   200 GB disk
>   2 core 
> 
> Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].
> 
> Cheers
> 
> [1] pull requests https://github.com/ceph/ceph/pulls
> [2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
> [3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
> [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: Call for jenkins slaves to improve multi operating system support

[Duan, Jiangang]

Got you.
Let me see if we can help - it may take some time - we have the servers but need find some people to set it up. 

-----Original Message-----
From: Loic Dachary [mailto:loic@dachary.org] 
Sent: Wednesday, April 08, 2015 9:33 PM
To: Duan, Jiangang; Ceph Development
Subject: Re: Call for jenkins slaves to improve multi operating system support



On 08/04/2015 15:21, Duan, Jiangang wrote:
> Loric,
> 
> do you mean we need give the servers to you or we just build the testing inside our own server room to do all the testing?

Thanks for asking, I realize that was not clear. 

The idea is not to donate hardware, because that would require manpower and extra costs to connect to the net. 

What would be useful is a machine connected to the net and dedicated to running a jenkins slave. It receives a build from the jenkins master (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh script that is found at the root of the repository and reports failure / success back to the jenkins master.

Does that make sense ?

> 
> -jiangang
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
> Sent: Wednesday, April 08, 2015 6:56 PM
> To: Ceph Development
> Subject: Call for jenkins slaves to improve multi operating system support
> 
> Hi Ceph,
> 
> When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".
> 
> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.
> 
> If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:
> 
> *  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02
> 
>   32 GB RAM
>   200 GB SSD
>   8 core > 2.5Ghz
> 
> *  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02
> 
>   4 GB RAM
>   200 GB disk
>   2 core
> 
> *  armv7, armv8 architecture for Ubuntu 14.04
> 
>   4 GB RAM
>   200 GB disk
>   2 core 
> 
> Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].
> 
> Cheers
> 
> [1] pull requests https://github.com/ceph/ceph/pulls
> [2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
> [3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
> [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: Call for jenkins slaves to improve multi operating system support

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3621 bytes --]



On 09/04/2015 12:30, Duan, Jiangang wrote:
> Got you.
> Let me see if we can help - it may take some time - we have the servers but need find some people to set it up. 

Thanks a lot, I'm looking forward to that :-)

> -----Original Message-----
> From: Loic Dachary [mailto:loic@dachary.org] 
> Sent: Wednesday, April 08, 2015 9:33 PM
> To: Duan, Jiangang; Ceph Development
> Subject: Re: Call for jenkins slaves to improve multi operating system support
> 
> 
> 
> On 08/04/2015 15:21, Duan, Jiangang wrote:
>> Loric,
>>
>> do you mean we need give the servers to you or we just build the testing inside our own server room to do all the testing?
> 
> Thanks for asking, I realize that was not clear. 
> 
> The idea is not to donate hardware, because that would require manpower and extra costs to connect to the net. 
> 
> What would be useful is a machine connected to the net and dedicated to running a jenkins slave. It receives a build from the jenkins master (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh script that is found at the root of the repository and reports failure / success back to the jenkins master.
> 
> Does that make sense ?
> 
>>
>> -jiangang
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
>> Sent: Wednesday, April 08, 2015 6:56 PM
>> To: Ceph Development
>> Subject: Call for jenkins slaves to improve multi operating system support
>>
>> Hi Ceph,
>>
>> When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".
>>
>> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.
>>
>> If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:
>>
>> *  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02
>>
>>   32 GB RAM
>>   200 GB SSD
>>   8 core > 2.5Ghz
>>
>> *  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02
>>
>>   4 GB RAM
>>   200 GB disk
>>   2 core
>>
>> *  armv7, armv8 architecture for Ubuntu 14.04
>>
>>   4 GB RAM
>>   200 GB disk
>>   2 core 
>>
>> Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].
>>
>> Cheers
>>
>> [1] pull requests https://github.com/ceph/ceph/pulls
>> [2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
>> [3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
>> [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Call for jenkins slaves to improve multi operating system support

[Sage Weil]

On Wed, 8 Apr 2015, Loic Dachary wrote:
> Hi Ceph,
> 
> When a contribution is proposed to Ceph [1], a bot compiles and run 
> tests with it to provide feedback to the developer [2]. When something 
> goes wrong the failure can be repeated on the developer machine [3] for 
> debug. This also helps the reviewer who knows the code compiles and does 
> not break anything that would be detected by "make check".
> 
> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to 
> older operating systems (headers, compiler version, etc.) may be 
> detected later, when building packages [4] and after the pull request 
> has been merged in master. This is rare but requires extra attention 
> from the reviewer and needs to be dealt with urgently when it happens.

Do additional slaves block the message from appearing on the pull 
request?  I.e., what happens if a slave is very slow (e.g., armv7) or 
broken (network issue)?

What are the connectivity requirements?  Can slaves exist on other 
(private) networks?

Thanks!
sage

Re: Call for jenkins slaves to improve multi operating system support

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

Hi Sage,

On 08/04/2015 18:59, Sage Weil wrote:
> On Wed, 8 Apr 2015, Loic Dachary wrote:
>> Hi Ceph,
>>
>> When a contribution is proposed to Ceph [1], a bot compiles and run 
>> tests with it to provide feedback to the developer [2]. When something 
>> goes wrong the failure can be repeated on the developer machine [3] for 
>> debug. This also helps the reviewer who knows the code compiles and does 
>> not break anything that would be detected by "make check".
>>
>> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to 
>> older operating systems (headers, compiler version, etc.) may be 
>> detected later, when building packages [4] and after the pull request 
>> has been merged in master. This is rare but requires extra attention 
>> from the reviewer and needs to be dealt with urgently when it happens.
> 
> Do additional slaves block the message from appearing on the pull 
> request?  I.e., what happens if a slave is very slow (e.g., armv7) or 
> broken (network issue)?

I will make it so a comment is posted as soon as the first slave succeeds / fails (it currently waits for all to finish which is inconvenient). The first slave will always be a CentOS 7 running on a fast machine so that the worst that can happen is that it's the only one to run. If slow slaves lag behind too much it would be nice to have a jenkins plugin that discards jobs randomly to prevent the queue from growing out of proportion on that specific slave.

> What are the connectivity requirements?  

Nothing more than the ability to git pull from a ceph repository. 

> Can slaves exist on other 
> (private) networks?

Yes. In that case a ssh -f -n -L tunnel to the jenkins master will be established to allow it to probe the slave when necessary.

> 
> Thanks!
> sage
> 

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Call for jenkins slaves to improve multi operating system support

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]

Hi,

Thanks to people who proposed to host jenkins slave: two new slaves should be added next week :-) The conversations we had have been summarized in a short document that will hopefully clarify how the slaves are put to work:

http://ceph-workbench.readthedocs.org/en/latest/user/jenkins.html

Cheers

On 08/04/2015 12:56, Loic Dachary wrote:
> Hi Ceph,
> 
> When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check".
> 
> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens.
> 
> If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is:
> 
> *  x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02
> 
>   32 GB RAM
>   200 GB SSD
>   8 core > 2.5Ghz
> 
> *  i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02
> 
>   4 GB RAM
>   200 GB disk
>   2 core
> 
> *  armv7, armv8 architecture for Ubuntu 14.04
> 
>   4 GB RAM
>   200 GB disk
>   2 core 
> 
> Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5].
> 
> Cheers
> 
> [1] pull requests https://github.com/ceph/ceph/pulls
> [2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064
> [3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44
> [4] gitbuilder http://ceph.com/gitbuilder.cgi
> [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]