Re: blktrace with fio replay for benchmarking vendor offerings

["Jarle Bjørgeengen" <jarle.bjorgeengen@usit.uio.no>]

On 05/16/2012 11:10 PM, Jiri Horky wrote:
> Hi Jarle,
>
> On 05/16/2012 09:21 PM, Jarle Bj�rgeengen wrote:
>> Hello,
>>
>> I'm involved in the purchasing process of block storage systems, and
>> research viable benchmarking strategies for specifying and verifying
>> performance requirements.
>>
>> Ideally I would like to capture our traces of our current daily
>> production workload with blktrace, attach the traces to the tender and
>> require the configuration to be able to run 2x that kind of workload.
>> During acceptance I would like to hook enough hardware to saturate the
>> system with the same workload, and measure that the requirement has
>> been met.
>>
>> I'm interested in comments about the practical viability of such
>> approach if anyone have similar experiences.
>>
> actually, I am in the very same situation at Institute of Physics, where
> we annually buy some 1PB of raw disk space (the money equivalent). We
> ended up with very similar approach that you described. We use IOreplay
> from IOapps (http://code.google.com/p/ioapps/) application to run the
> load previously recorded by strace. So it is done on a file level
> instead of the block level. This clearly has its advantages (you may use
> different file system, benchmarks are not fatal for running systems
> etc.), but of course some drawbacks as well (strace overhead, etc., see
> webpage). If you are interested and have any more question/suggestion
> about the IOreplay, just email me, I actually wrote it as a part of my
> master thesis.

Thanks for the tip I'll check out IOapps.

>> Some open questions:
>>
>> How safe is it to run blktrace on critical production environment?
>> What precautions should be made?
>>
>> Given that the current system consists of 3 HP EVA800 with X number of
>> LUNS about 50 hosts, and the new system likely is a single system with
>> 5 new servers running at at full speed, how much value will the
>> benchmark provide compared to "real world" ? Should I create equally
>> many luns and distribute load across the 5 machines?
>>
>> Is it best to scale the workload to 2X replaying all traces
>> simultanously with one fio-trace-replay/lun, and then dublicate alle
>> replays so that each lun serves two fio-replays rather than one, and
>> then 3 and so on.
>>
>> Any thoughts or comments are very much appreciated.
> I would say that all above heavily depends on your environment, whether
> the access pattern is similar to all LUNS and whether all LUNS are more
> or less expected to expand similarly in the future. Personally, I would
> go with settings that are simpler to setup (from the options you have
> proposed) unless you really know your future requirements and you are
> sure that the results obtained in both ways differ significantly.

Thanks.

- Jarle Bj�rgeengen