blktrace with fio replay for benchmarking vendor offerings

blktrace with fio replay for benchmarking vendor offerings

[Jarle Bjørgeengen]

Hello,

I'm involved in the purchasing process of block storage systems, and 
research viable benchmarking strategies for specifying and verifying 
performance requirements.

Ideally I would like to capture our traces of our current daily 
production workload  with blktrace, attach the traces to the tender and 
require the configuration to be able to run 2x that kind of workload. 
During acceptance I would like to hook enough hardware to saturate the 
system with the same workload, and measure that the requirement has been 
met.

I'm interested in comments about the practical viability of such 
approach if anyone have similar experiences.

Some open questions:

How safe is it to run blktrace on critical production environment? What 
precautions should be made?

Given that the current system consists of 3 HP EVA800 with X number of 
LUNS about 50 hosts, and the new system likely is a single system with 5 
new servers running at at full speed, how much value will the benchmark 
provide compared to "real world" ? Should I create equally many luns and 
distribute load across the 5 machines?

Is it best to scale the workload to 2X replaying all traces 
simultanously with one fio-trace-replay/lun, and then dublicate alle 
replays so that each lun serves two fio-replays rather than one, and 
then 3 and so on.

Any thoughts or comments are very much appreciated.

Best regards
Jarle Bj�rgeengen
University of Oslo / USIT

Re: blktrace with fio replay for benchmarking vendor offerings

[Jiri Horky]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Hi Jarle,

On 05/16/2012 09:21 PM, Jarle Bjørgeengen wrote:
> Hello,
>
> I'm involved in the purchasing process of block storage systems, and 
> research viable benchmarking strategies for specifying and verifying 
> performance requirements.
>
> Ideally I would like to capture our traces of our current daily 
> production workload  with blktrace, attach the traces to the tender 
> and require the configuration to be able to run 2x that kind of 
> workload. During acceptance I would like to hook enough hardware to 
> saturate the system with the same workload, and measure that the 
> requirement has been met.
>
> I'm interested in comments about the practical viability of such 
> approach if anyone have similar experiences.
>
actually, I am in the very same situation at Institute of Physics, where 
we annually buy some 1PB of raw disk space (the money equivalent). We 
ended up with very similar approach that you described. We use IOreplay 
from IOapps (http://code.google.com/p/ioapps/) application to run the 
load previously recorded by strace. So it is done on a file level 
instead of the block level. This clearly has its advantages (you may use 
different file system, benchmarks are not fatal for running systems 
etc.), but of course some drawbacks as well (strace overhead, etc., see 
webpage). If you are interested and have any more question/suggestion 
about the IOreplay, just email me, I actually wrote it as a part of my 
master thesis.
> Some open questions:
>
> How safe is it to run blktrace on critical production environment? 
> What precautions should be made?
>
> Given that the current system consists of 3 HP EVA800 with X number of 
> LUNS about 50 hosts, and the new system likely is a single system with 
> 5 new servers running at at full speed, how much value will the 
> benchmark provide compared to "real world" ? Should I create equally 
> many luns and distribute load across the 5 machines?
>
> Is it best to scale the workload to 2X replaying all traces 
> simultanously with one fio-trace-replay/lun, and then dublicate alle 
> replays so that each lun serves two fio-replays rather than one, and 
> then 3 and so on.
>
> Any thoughts or comments are very much appreciated.
I would say that all above heavily depends on your environment, whether 
the access pattern is similar to all LUNS and whether all LUNS are more 
or less expected to expand similarly in the future. Personally, I would 
go with settings that are simpler to setup (from the options you have 
proposed) unless you really know your future requirements and you are 
sure that the results obtained in both ways differ significantly.

Regards
Jiri Horky


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5655 bytes --]

Re: blktrace with fio replay for benchmarking vendor offerings

[Jarle Bjørgeengen]

On 05/16/2012 11:10 PM, Jiri Horky wrote:
> Hi Jarle,
>
> On 05/16/2012 09:21 PM, Jarle Bj�rgeengen wrote:
>> Hello,
>>
>> I'm involved in the purchasing process of block storage systems, and
>> research viable benchmarking strategies for specifying and verifying
>> performance requirements.
>>
>> Ideally I would like to capture our traces of our current daily
>> production workload with blktrace, attach the traces to the tender and
>> require the configuration to be able to run 2x that kind of workload.
>> During acceptance I would like to hook enough hardware to saturate the
>> system with the same workload, and measure that the requirement has
>> been met.
>>
>> I'm interested in comments about the practical viability of such
>> approach if anyone have similar experiences.
>>
> actually, I am in the very same situation at Institute of Physics, where
> we annually buy some 1PB of raw disk space (the money equivalent). We
> ended up with very similar approach that you described. We use IOreplay
> from IOapps (http://code.google.com/p/ioapps/) application to run the
> load previously recorded by strace. So it is done on a file level
> instead of the block level. This clearly has its advantages (you may use
> different file system, benchmarks are not fatal for running systems
> etc.), but of course some drawbacks as well (strace overhead, etc., see
> webpage). If you are interested and have any more question/suggestion
> about the IOreplay, just email me, I actually wrote it as a part of my
> master thesis.

Thanks for the tip I'll check out IOapps.

>> Some open questions:
>>
>> How safe is it to run blktrace on critical production environment?
>> What precautions should be made?
>>
>> Given that the current system consists of 3 HP EVA800 with X number of
>> LUNS about 50 hosts, and the new system likely is a single system with
>> 5 new servers running at at full speed, how much value will the
>> benchmark provide compared to "real world" ? Should I create equally
>> many luns and distribute load across the 5 machines?
>>
>> Is it best to scale the workload to 2X replaying all traces
>> simultanously with one fio-trace-replay/lun, and then dublicate alle
>> replays so that each lun serves two fio-replays rather than one, and
>> then 3 and so on.
>>
>> Any thoughts or comments are very much appreciated.
> I would say that all above heavily depends on your environment, whether
> the access pattern is similar to all LUNS and whether all LUNS are more
> or less expected to expand similarly in the future. Personally, I would
> go with settings that are simpler to setup (from the options you have
> proposed) unless you really know your future requirements and you are
> sure that the results obtained in both ways differ significantly.

Thanks.

- Jarle Bj�rgeengen