[Lustre-devel] using LST for performance testing

[Lustre-devel] using LST for performance testing

[Nic Henke]

Hello,

	I'm hoping to get a few ideas on how we could modify LST to make doing 
performance testing easier. Right now we can use "lst stat" to get a 
rough idea of performance, but the timers are pretty rough and the data 
is a snapshot.

	Any ideas ? I've got cycles to do the coding, but not sure what would 
be the best way to fit this into the existing LST framework.


BTW - the ability to dump CSV or some other text file with per-node and 
per-group data would also be nice.

Nic

[Lustre-devel] using LST for performance testing

[Isaac Huang]

On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote:
> Hello,
> 
> 	I'm hoping to get a few ideas on how we could modify LST to make doing 
> performance testing easier. Right now we can use "lst stat" to get a 
> rough idea of performance, but the timers are pretty rough and the data 
> is a snapshot.
> 
> 	Any ideas ? I've got cycles to do the coding, but not sure what would 
> be the best way to fit this into the existing LST framework.

There's some rough edges in the stat gathering code. First, the LST
console has no idea whether the tests have stopped, and that's why the
'lst stat' command by default loops until a ^C. Test clients could
return a counter for active test batches and when it drops to 0 all
tests on the client must have completed, but servers are passive and
have no idea whether clients are done or not.

The throughput calculation also could be inaccurate. IIRC, the console
just take a snapshot of stat counters on test nodes at a fixed
interval (1 second by default), and calculate the throughput as
changes in the successive counter snapshots divided by the interval.
But, apparently the interval at which the console sends 'get_stat'
requests does not equal the interval at which snapshots are taken on
test nodes - the 'get_stat' requests could be delayed on the path when
the network is stressed (something LST was designed to do), and even
worse they could be reordered in the presence of routers. One possible
solution would be to include timestamp in the 'get_stat' replies, and
calculate the throughput as diffs in counters divided by diffs in
timestamps. Since the console only cares about the changes in
timestamps, the test nodes clocks do not need to be in sync at all
(but they do need to be monotonic and be of a same resolution).

The test servers concurrently posts one passive buffer for each
request, so for each test request there's one LNetMDAttach and one
unlink operation and both operations need to grab the one big
LNET_LOCK therefore it could be possible that the server CPU becomes a
bottleneck before the network could be saturated. The solution is to,
instead of one request per buffer, post one big buffer that could
accommodate multiple requests to amortize the per buffer processing
costs.

Refining these rough edges might likely involve protocol changes. The
LST is not a production service so strict backward compatibility is
not necessary. I think it'd suffice to do a protocol version check at
the time of 'add_node' command and simply refuse to add a test node
whose protocol version is different than that of the console.

> BTW - the ability to dump CSV or some other text file with per-node and 
> per-group data would also be nice.

That's a good idea, then users could do whatever they'd like to the data.

Thanks,
Isaac

[Lustre-devel] using LST for performance testing

[Nic Henke]

Isaac Huang wrote:
> On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote:
>   
>> Hello,
>>
>> 	I'm hoping to get a few ideas on how we could modify LST to make doing 
>> performance testing easier. Right now we can use "lst stat" to get a 
>> rough idea of performance, but the timers are pretty rough and the data 
>> is a snapshot.
>>
>> 	Any ideas ? I've got cycles to do the coding, but not sure what would 
>> be the best way to fit this into the existing LST framework.
>>     
>
> There's some rough edges in the stat gathering code. First, the LST
> console has no idea whether the tests have stopped, and that's why the
> 'lst stat' command by default loops until a ^C. Test clients could
> return a counter for active test batches and when it drops to 0 all
> tests on the client must have completed, but servers are passive and
> have no idea whether clients are done or not.
>   

I think the timing of the start/stop of each of the tests is probably 
the trickiest bit. To get really good end-to-end numbers, we'd need to 
be able to accurately time each of the tests.
> The throughput calculation also could be inaccurate. IIRC, the console
> just take a snapshot of stat counters on test nodes at a fixed
> interval (1 second by default), and calculate the throughput as
> changes in the successive counter snapshots divided by the interval.
> But, apparently the interval at which the console sends 'get_stat'
> requests does not equal the interval at which snapshots are taken on
> test nodes - the 'get_stat' requests could be delayed on the path when
> the network is stressed (something LST was designed to do), and even
> worse they could be reordered in the presence of routers. One possible
> solution would be to include timestamp in the 'get_stat' replies, and
> calculate the throughput as diffs in counters divided by diffs in
> timestamps. Since the console only cares about the changes in
> timestamps, the test nodes clocks do not need to be in sync at all
> (but they do need to be monotonic and be of a same resolution).
>   
I'm wondering if we couldn't add a new 'batch_stat' command. The idea is 
that the client code will fill in the start/stop times for each test and 
then after the test is done, 'batch_stat' would collect this data. The 
collection would still be passive and a new command should minimize the 
protocol changes. The per-test data would allow us to get accurate perf 
numbers and also provide some data into how parallel the tests were, if 
there are any unfairness issues, etc.
> The test servers concurrently posts one passive buffer for each
> request, so for each test request there's one LNetMDAttach and one
> unlink operation and both operations need to grab the one big
> LNET_LOCK therefore it could be possible that the server CPU becomes a
> bottleneck before the network could be saturated. The solution is to,
> instead of one request per buffer, post one big buffer that could
> accommodate multiple requests to amortize the per buffer processing
> costs.
>   
If we added timestamps to the data, the processing time & buffer sizing 
would be less of an issue - it wouldn't factor into the accuracy of the 
numbers are are gathering.

> Refining these rough edges might likely involve protocol changes. The
> LST is not a production service so strict backward compatibility is
> not necessary. I think it'd suffice to do a protocol version check at
> the time of 'add_node' command and simply refuse to add a test node
> whose protocol version is different than that of the console.
>
>   
OK.
>> BTW - the ability to dump CSV or some other text file with per-node and 
>> per-group data would also be nice.
>>     
>
> That's a good idea, then users could do whatever they'd like to the data.
>
>   

Nic

[Lustre-devel] using LST for performance testing

[David Dillow]

On Tue, 2009-09-29 at 11:51 -0500, Nic Henke wrote:
> I'm wondering if we couldn't add a new 'batch_stat' command. The idea is 
> that the client code will fill in the start/stop times for each test and 
> then after the test is done, 'batch_stat' would collect this data. The 
> collection would still be passive and a new command should minimize the 
> protocol changes. The per-test data would allow us to get accurate perf 
> numbers and also provide some data into how parallel the tests were, if 
> there are any unfairness issues, etc.

Along these lines, it would be nice if we could specify a run time for
each test rather than an amount of data to be transferred -- it makes it
easier to get aggregate bandwidth numbers, and often shows imbalances
nicely -- the node getting starved is the one that transfers less data.

It may also make sense to add a 'delay' parameter that causes each test
to wait a specified amount of time from the 'go' signal. This allows the
signal to propagate without running into congestion from the test,
helping to cause all of the clients to start the test closer to
simultaneously.
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

[Lustre-devel] using LST for performance testing

[Nic Henke]

David Dillow wrote:
> On Tue, 2009-09-29 at 11:51 -0500, Nic Henke wrote:
>   
>> I'm wondering if we couldn't add a new 'batch_stat' command. The idea is 
>> that the client code will fill in the start/stop times for each test and 
>> then after the test is done, 'batch_stat' would collect this data. The 
>> collection would still be passive and a new command should minimize the 
>> protocol changes. The per-test data would allow us to get accurate perf 
>> numbers and also provide some data into how parallel the tests were, if 
>> there are any unfairness issues, etc.
>>     
>
> Along these lines, it would be nice if we could specify a run time for
> each test rather than an amount of data to be transferred -- it makes it
> easier to get aggregate bandwidth numbers, and often shows imbalances
> nicely -- the node getting starved is the one that transfers less data.
>   
> It may also make sense to add a 'delay' parameter that causes each test
> to wait a specified amount of time from the 'go' signal. This allows the
> signal to propagate without running into congestion from the test,
> helping to cause all of the clients to start the test closer to
> simultaneously.
>   

Interesting - can you elaborate, perhaps in the form of a patch  ? :-) I 
like both ideas, but not signing up to code them just yet.

Nic

[Lustre-devel] using LST for performance testing

[David Dillow]

On Tue, 2009-09-29 at 14:02 -0400, Nic Henke wrote:
> David Dillow wrote:
> > Along these lines, it would be nice if we could specify a run time for
> > each test rather than an amount of data to be transferred -- it makes it
> > easier to get aggregate bandwidth numbers, and often shows imbalances
> > nicely -- the node getting starved is the one that transfers less data.
> >   
> > It may also make sense to add a 'delay' parameter that causes each test
> > to wait a specified amount of time from the 'go' signal. This allows the
> > signal to propagate without running into congestion from the test,
> > helping to cause all of the clients to start the test closer to
> > simultaneously.
> >   
> 
> Interesting - can you elaborate, perhaps in the form of a patch  ? :-) I 
> like both ideas, but not signing up to code them just yet.

Maybe, but it is going to be well down the list -- I owe Oleg a large
amount of testing before I think of doing anything else. :)
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

[Lustre-devel] using LST for performance testing

[Isaac Huang]

On Tue, Sep 29, 2009 at 01:32:48PM -0400, David Dillow wrote:
> On Tue, 2009-09-29 at 11:51 -0500, Nic Henke wrote:
> > I'm wondering if we couldn't add a new 'batch_stat' command. The idea is 
> > that the client code will fill in the start/stop times for each test and 
> > then after the test is done, 'batch_stat' would collect this data. The 
> > collection would still be passive and a new command should minimize the 
> > protocol changes. The per-test data would allow us to get accurate perf 
> > numbers and also provide some data into how parallel the tests were, if 
> > there are any unfairness issues, etc.
> 
> Along these lines, it would be nice if we could specify a run time for
> each test rather than an amount of data to be transferred -- it makes it
> easier to get aggregate bandwidth numbers, and often shows imbalances
> nicely -- the node getting starved is the one that transfers less data.

This would be a very useful feature. We're working on to add LST tests
to our automatic tests, where we met a problem that we could never
tell how long the test would run by looking at '--loop' and
'--concurrency'.

The LST already implemented a timer mechanism which is good at second
resolution, which should suffice for controlling test run time.

Isaac

[Lustre-devel] using LST for performance testing

[Nic Henke]

Nic Henke wrote:
> Isaac Huang wrote:
>   
>> On Thu, Sep 24, 2009 at 03:33:18PM -0500, Nic Henke wrote:
>>     
>
> I'm wondering if we couldn't add a new 'batch_stat' command. The idea is 
>   

So - for a current swag:

- I'm thinking of timing the 'test instance' run (sfw_test_instance_t) - 
this would get timing for each test run end-to-end but be detailed 
enough to show the different timing with --concurrency > 1 is used. 
Doing per-RPC timing seemed to heavy and I'm not sure that level of 
resolution is really needed.

- the new command 'test_stat' would collect this data ala the current 
'stat' but allow CSV dumping for more fine-grained analysis. I'd 
probably dump some node & group info as well to make easier to do 
'group' stats.


Nic

[Lustre-devel] using LST for performance testing

[Eric Barton]

What is it we really want to measure here?  Steady-state
throughput or elapsed time to run a specific test (i.e.
including ramp-up/ramp-down).

The intention behind the current stats command was to measure
steady-state throughput - i.e.once the test batch(es?) have
been started, a number of stat snapshots are taken until
throughput settles.  That also probably allows tests to be
run more quickly since they can be stopped immediately the
steady state has been observed.

Is that too hard to do with the current command set?


    Cheers,
              Eric

[Lustre-devel] using LST for performance testing

[Nic Henke]

Eric Barton wrote:
> What is it we really want to measure here?  Steady-state
> throughput or elapsed time to run a specific test (i.e.
> including ramp-up/ramp-down).
>   
Both :-)

The end-to-end performance is more interesting to me right now. The 
timing data is more accurate and we can run shorter tests that with 'lst 
stat' to get an idea of burst performance.  Having both methods is 
desirable to me.

> The intention behind the current stats command was to measure
> steady-state throughput - i.e.once the test batch(es?) have
> been started, a number of stat snapshots are taken until
> throughput settles.  That also probably allows tests to be
> run more quickly since they can be stopped immediately the
> steady state has been observed.
>
> Is that too hard to do with the current command set?
>   
It could be made cleaner and output the data to .csv, etc - but one can 
get a rough idea of steady state performance. The data will be more 
accurate once the timestamps for the data are sent over the wire instead 
of computed locally on the lst console node.

Nic

[Lustre-devel] using LST for performance testing

[Isaac Huang]

On Tue, Sep 29, 2009 at 11:51:45AM -0500, Nic Henke wrote:
>   ......
> > The test servers concurrently posts one passive buffer for each
> > request, so for each test request there's one LNetMDAttach and one
> > unlink operation and both operations need to grab the one big
> > LNET_LOCK therefore it could be possible that the server CPU becomes a
> > bottleneck before the network could be saturated. The solution is to,
> > instead of one request per buffer, post one big buffer that could
> > accommodate multiple requests to amortize the per buffer processing
> > costs.
> >   
> If we added timestamps to the data, the processing time & buffer sizing 
> would be less of an issue - it wouldn't factor into the accuracy of the 
> numbers are are gathering.

Probably not. The timestamps affect only the stat gathering RPCs which
should be far out numbered by the test RPCs (loop x concurrency x
test_client_count, for each test server).

Isaac