Re: teuthology

Re: teuthology

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4184 bytes --]

Hi Kapil,

On 27/10/2014 18:01, Kapil Sharma wrote:
> Hi Loic, 
> 
> Zack from Inktank helped me a lot in setting up teuthology.
> 
> I think following areas could be worked upon to make teuthology
> easier to install and run - 
> 
> 
> 1. Currently we can by-pass the Lock server when running individual
> tests from a yaml config file. However, if we have to run
> teuthology-suite, it seems using Lock server is mandatory.I think lock
> server should be optional. A user should have an option to by-pass the
> lock server as he may not need it when running the tests outside the
> Inktank lab.

I also started to not use the lock server, a year ago. I recently experimented with paddles + teuthology-worker and it's fairly easy to get it to work. The advantage is that you can schedule jobs that will wait for others to complete. My work in progress is at

https://github.com/dachary/teuthology/compare/wip-container
https://github.com/dachary/paddles/compare/wip-container

(note that it is mixed with my attempt to make it work with docker instead of virtual machines or bare metal)
 
> 2. The word "ubuntu" is hard coded in some places in the code :-)
> 
> 3. The bootstrap script is for ubuntu machines only. Anyway, it's
> not a big task to run those commands manually or create a script of
> your own. 

Is your script available somewhere ? I plan to set it up on fedora20 and it would be most useful.

> 4. Teuthology picks up several test cases from the qa/workunits dir -
> https://github.com/ceph/ceph/tree/master/qa/workunits
> Unfortunately these tests are not documented anywhere. If one has to
> check the test coverage or document the exact test steps, it's going
> to be a humungous effort.

Yes. I'm often happy to read tests as if they were the documention of the code. But it would be useful to have documentation for the test themselves in additon ;-)

> 5. The tests executed in qa/workunits are not listed as separate test
> cases in Pulpito results. E.g qa/workunits/rbd/test_librbd_python.sh
> contains 57 tests. However, in Pupito this is listed as one job. Though
> not sure if it is possible to implement this as most of the qa/workunits
> tests are shell scripts.

Are you running pulpito ? I create a split of workunit that may be of interest to you. My motivation was not to have them show separately in pulpito. Rather to run them in // 

I first modified the script so that it could be called to group tests:

https://github.com/dachary/ceph/commit/ec0ba1a9994e4eb24955442895c6b8dd2561ba20

and then created scripts to run each group independantly

https://github.com/dachary/ceph/commit/bafd35478e310e887a808784f02c7f0041548d1e

which could also be an "exec" teuthology tasks.


> 6. I have not tried this of-late but I think teuthology can not run it's
> tests against an already deployed ceph-cluster. I need to check it again
> though.

It may be possible to convince it to do that since the install task is not mandatory.

> 7. Would be great if teuthology can cover Calamari testing too :-)

If I'm not mistaken Waren is working on making this happen. But I've not looked at it closely.

Cheers

> 
> 
> Best Regards,
> Kapil.
> 
> 
> 
> 
> 
> 
> On Mon, 2014-10-27 at 16:52 +0100, Loic Dachary wrote:
>> It was great meeting your and your family :-) Back in Paris and massively jetlagged.
>>
>> Nice to meet you Kapil ! Are there any pain points you'd like to address regarding the teuthology installation ?
>>
>> Cheers
>>
>> On 27/10/2014 11:47, Thorsten Behrens wrote:
>>> Hi Loic,
>>>
>>> was great to meet you and Patrick at the google summit!
>>>
>>> Loic, meet Kapil, who's doing lots of great stuff with QA automation
>>> and deployment on our side. And I think he and Zack were already in
>>> close contact. ;)
>>>
>>> As promised the update on what we run from teuthology:
>>>
>>> Kapil Sharma wrote:
>>>> As of now I have cnfigured, rbd, krbd and rados test suites which
>>>> are located here
>>>> [snip]
>>>>
>>>
>>> All the very best, and a safe trip home,
>>>
>>
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: teuthology

[Kapil Sharma]

Hi Loic, 

On Tue, 2014-10-28 at 16:01 +0100, Loic Dachary wrote:

> I also started to not use the lock server, a year ago. I recently experimented with paddles + teuthology-worker and it's fairly easy to get it to work. The advantage is that you can schedule jobs that will wait for others to complete. My work in progress is at
> 
> https://github.com/dachary/teuthology/compare/wip-container
> https://github.com/dachary/paddles/compare/wip-container
> 
> (note that it is mixed with my attempt to make it work with docker instead of virtual machines or bare metal)
Thanks for sharing your work. Yes, I am also using paddles +
teuthology-worker now.


>  
> > 2. The word "ubuntu" is hard coded in some places in the code :-)
> > 
> > 3. The bootstrap script is for ubuntu machines only. Anyway, it's
> > not a big task to run those commands manually or create a script of
> > your own. 
> 
> Is your script available somewhere ? I plan to set it up on fedora20 and it would be most useful.
Not yet, but I will share it with you soon. 



> 
> Yes. I'm often happy to read tests as if they were the documention of the code. But it would be useful to have documentation for the test themselves in additon ;-)
Yes, At some point I need to start this effort :-)



> Are you running pulpito ? I create a split of workunit that may be of interest to you. My motivation was not to have them show separately in pulpito. Rather to run them in // 
> 
> I first modified the script so that it could be called to group tests:
> 
> https://github.com/dachary/ceph/commit/ec0ba1a9994e4eb24955442895c6b8dd2561ba20
> 
> and then created scripts to run each group independantly
> 
> https://github.com/dachary/ceph/commit/bafd35478e310e887a808784f02c7f0041548d1e
> 
> which could also be an "exec" teuthology tasks.
Yes, I am running pulpito. Maybe a similar logic could be built to also
report the tests separately. Shall give it a try. 

> 
> 
> > 6. I have not tried this of-late but I think teuthology can not run it's
> > tests against an already deployed ceph-cluster. I need to check it again
> > though.
> 
> It may be possible to convince it to do that since the install task is not mandatory.
I found this old ticket which was closed. Need to check the latest
status - http://tracker.ceph.com/issues/6641



> > 7. Would be great if teuthology can cover Calamari testing too :-)
> If I'm not mistaken Waren is working on making this happen. But I've not looked at it closely.
Okay. Will follow up on this.



Regards,
Kapil.




> > On Mon, 2014-10-27 at 16:52 +0100, Loic Dachary wrote:
> >> It was great meeting your and your family :-) Back in Paris and massively jetlagged.
> >>
> >> Nice to meet you Kapil ! Are there any pain points you'd like to address regarding the teuthology installation ?
> >>
> >> Cheers
> >>
> >> On 27/10/2014 11:47, Thorsten Behrens wrote:
> >>> Hi Loic,
> >>>
> >>> was great to meet you and Patrick at the google summit!
> >>>
> >>> Loic, meet Kapil, who's doing lots of great stuff with QA automation
> >>> and deployment on our side. And I think he and Zack were already in
> >>> close contact. ;)
> >>>
> >>> As promised the update on what we run from teuthology:
> >>>
> >>> Kapil Sharma wrote:
> >>>> As of now I have cnfigured, rbd, krbd and rados test suites which
> >>>> are located here
> >>>> [snip]
> >>>>
> >>>
> >>> All the very best, and a safe trip home,
> >>>
> >>
> > 
> > 
> 

Re: Teuthology

[Loic Dachary]

Hi Herman,

I recently succeeded (thanks to Thierry's help ;-) to install paddles using ansible[1]. The playbook was developped recently by Zack. Even if you don't otherwise use ansible, it's a better option than more complex instructions. And easier to maintain as you can rely on Andrew, Dan and Zack (and maybe others) to maintain the playbook. One thing that I like (but is frowned upon by more seasoned ansible users) is to use ansible from the node where paddles is to be installed. For instance:

   ansible-playbook -i paddles, -c local paddles.yml

I suppose the http://docs.ceph.com/teuthology/docs/LAB_SETUP.html document should be updated with this new recommended deployment method (the main document is at https://github.com/ceph/teuthology/blob/master/docs/LAB_SETUP.rst).

Toward the end of your document, the "Submitting slave node" is where the I don't have any experience. But David Galloway and Dan Mick and maybe others do as they have deployed and configured all the hardware currently in the sepia lab. It's a significant undertaking but is fortunately well documented and all files you need are public.

I chose an easier alternative which is to rely on an OpenStack cluster to provision the resources required to run teuthology jobs[2].

The last pitfall that comes to mind, and it's a significant one, is that a number of teuthology jobs are sensitive to the infrastructure. For instance, I recently worked around a paramiko bug that only showed on OpenStack although it is by no mean specific to OpenStack[3]. Fixing ceph-qa-suite so that it does not break because the infrastructure is slower or faster, has 1G network or 10G, has fast or slow internet connections etc. is sometime non trivial. But for the most part it's easy and a little tedious: you just have to do it on a regular basis :-)

My 2cts

[1] installing paddles via ansible http://www.spinics.net/lists/ceph-devel/msg28974.html
[2] OpenStack backend https://github.com/dachary/teuthology/tree/openstack#openstack-backend
[3] openstack: SSHException: Key-exchange timed out http://tracker.ceph.com/issues/15236

On 28/03/2016 12:14, Herman Narkaytis wrote:
> We intended to publish this instruction as public document, so feedback from broader community would be valuable.
> 
> HHN.
> 
> On Mon, Mar 28, 2016 at 1:01 PM, Loic Dachary <loic@dachary.org <mailto:loic@dachary.org>> wrote:
> 
>     Hi,
> 
>     On 28/03/2016 10:55, Herman Narkaytis wrote:
>     > Loic,
>     >   Below you will find a link on Teuthology deployment instructions. I would be appreciated to get your feedback on it.
> 
>     This is a precise and detailed documentation, very useful. Do you mind if I post my remarks about it with ceph-devel in cc ? Other developers will certainly have interesting contributions to make.
> 
>     >   https://docs.google.com/document/d/18MW0NAwEZB45lGgsgjpb4fFdi8ope1YGr5OA-uvsIR8/edit
>     >
>     >   We are still figuring out how to deploy GitBuilder, but it should not be as hard as it was with Teuthology.
>     > HHN.
> 
>     :-)
> 
>     Cheers
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html