Re: vstart runner for cephfs tests

[Mark Nelson <mnelson@redhat.com>]

On 07/23/2015 07:37 AM, John Spray wrote:
>
>
> On 23/07/15 12:56, Mark Nelson wrote:
>> I had similar thoughts on the benchmarking side, which is why I
>> started writing cbt a couple years ago.  I needed the ability to
>> quickly spin up clusters and run benchmarks on arbitrary sets of
>> hardware.  The outcome isn't perfect, but it's been extremely useful
>> for running benchmarks and sort of exists as a half-way point between
>> vstart and teuthology.
>>
>> The basic idea is that you give it a yaml file that looks a little bit
>> like a teuthology yaml file and cbt will (optionally) build a cluster
>> across a number of user defined nodes with pdsh, start various
>> monitoring tools (this is ugly right now, I'm working on making it
>> modular), and then sweep through user defined benchmarks and sets of
>> parameter spaces.  I have a separate tool that will sweep through ceph
>> parameters, create ceph.conf files for each space, and run cbt with
>> each one, but the eventual goal is to integrate that into cbt itself.
>>
>> Though I never really intended it to run functional tests, I just
>> added something like looks very similar to the rados suite so I can
>> benchmark ceph_test_rados for the new community lab hardware. I
>> already had a mechanism to inject OSD down/out up/in events, so with a
>> bit of squinting it can give you a very rough approximation of a
>> workload using the osd thrasher.  If you are interested, I'd be game
>> to see if we could integrate your cephfs tests as well (I eventually
>> wanted to add cephfs benchmark capabilities anyway).
>
> Cool - my focus is very much on tightening the code-build-test loop for
> developers, but I can see us needing to extend that into a
> code-build-test-bench loop as we do performance work on cephfs in the
> future.  Does cbt rely on having ceph packages built, or does it blast
> the binaries directly from src/ onto the test nodes?

cbt doesn't handle builds/installs at all, so it's probably not 
particularly helpful in this regard.  By default it assumes binaries are 
in /usr/bin, but you can optionally override that in the yaml.  My 
workflow is usually to:

1a) build ceph from src and distribute to other nodes (manually)
1b) run a shell script that installs a given release from gitbuilder on 
all nodes
2) run a cbt yaml file that targets /usr/local, the build dir, /usr/bin, 
etc.

Definitely would be useful to have something that makes 1a) better. 
Probably not cbt's job though.

>
> John