achieving a workload to get specific unique duplicate blocks

achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

Firstly, thanks before in hand. :)

Using FIO, what is the way to achieve a work load with specific number
of unique blocks duplicates for specific number of times. For Example,
i want a 252k block duplicates 1000 times and i want it to be the
unique block in the complete workload.

Another query is that, for a blocksize of 256k does the io buffers are
created at 256k and written to the disk and are all those 256k blocks
unique buffer?. Can there be a change that in a 256k block, the
smaller size buffers be the same?

-- 
Srinivasa R Chamarthy

Re: achieving a workload to get specific unique duplicate blocks

[Jens Axboe]

On 04/25/2015 07:03 AM, Srinivasa Chamarthy wrote:
> Firstly, thanks before in hand. :)
>
> Using FIO, what is the way to achieve a work load with specific number
> of unique blocks duplicates for specific number of times. For Example,
> i want a 252k block duplicates 1000 times and i want it to be the
> unique block in the complete workload.

If you have very specific requirements, it might be easier to simply 
write an iolog generator that spits out what you want the pattern to 
look like. Then fio can import that, and run the workload.

> Another query is that, for a blocksize of 256k does the io buffers are
> created at 256k and written to the disk and are all those 256k blocks
> unique buffer?. Can there be a change that in a 256k block, the
> smaller size buffers be the same?

That depends on the settings for compressibility, dedupability, etc. In 
general see scramble_buffers, refill_buffers, and so on. That is all in 
the man page or HOWTO.

-- 
Jens Axboe

Re: achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

Thanks Jens. I was using refill_buffers and scramble_buffers at the
same time. If i specify them at the same time, there seems to be no
difference in the workloads. If i specify scramble_buffers then there
shouldn't be refill_buffers specified to get the buffers differently.
Thanks.
Srinivasa R Chamarthy


On Tue, May 5, 2015 at 4:39 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/25/2015 07:03 AM, Srinivasa Chamarthy wrote:
>>
>> Firstly, thanks before in hand. :)
>>
>> Using FIO, what is the way to achieve a work load with specific number
>> of unique blocks duplicates for specific number of times. For Example,
>> i want a 252k block duplicates 1000 times and i want it to be the
>> unique block in the complete workload.
>
>
> If you have very specific requirements, it might be easier to simply write
> an iolog generator that spits out what you want the pattern to look like.
> Then fio can import that, and run the workload.
>
>> Another query is that, for a blocksize of 256k does the io buffers are
>> created at 256k and written to the disk and are all those 256k blocks
>> unique buffer?. Can there be a change that in a 256k block, the
>> smaller size buffers be the same?
>
>
> That depends on the settings for compressibility, dedupability, etc. In
> general see scramble_buffers, refill_buffers, and so on. That is all in the
> man page or HOWTO.
>
> --
> Jens Axboe
>

Re: achieving a workload to get specific unique duplicate blocks

[Matthew Eaton]

It doesn't make sense to set refill_buffers and scramble_buffers at
the same time, since you won't get any repeating data with
refill_buffers set.

On Mon, May 4, 2015 at 7:23 PM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> Thanks Jens. I was using refill_buffers and scramble_buffers at the
> same time. If i specify them at the same time, there seems to be no
> difference in the workloads. If i specify scramble_buffers then there
> shouldn't be refill_buffers specified to get the buffers differently.
> Thanks.
> Srinivasa R Chamarthy
>
>
> On Tue, May 5, 2015 at 4:39 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 04/25/2015 07:03 AM, Srinivasa Chamarthy wrote:
>>>
>>> Firstly, thanks before in hand. :)
>>>
>>> Using FIO, what is the way to achieve a work load with specific number
>>> of unique blocks duplicates for specific number of times. For Example,
>>> i want a 252k block duplicates 1000 times and i want it to be the
>>> unique block in the complete workload.
>>
>>
>> If you have very specific requirements, it might be easier to simply write
>> an iolog generator that spits out what you want the pattern to look like.
>> Then fio can import that, and run the workload.
>>
>>> Another query is that, for a blocksize of 256k does the io buffers are
>>> created at 256k and written to the disk and are all those 256k blocks
>>> unique buffer?. Can there be a change that in a 256k block, the
>>> smaller size buffers be the same?
>>
>>
>> That depends on the settings for compressibility, dedupability, etc. In
>> general see scramble_buffers, refill_buffers, and so on. That is all in the
>> man page or HOWTO.
>>
>> --
>> Jens Axboe
>>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

Thanks Eaton. One question thought. From the man page for refill_buffers:

"If this option is given, fio will refill the IO buffers on every
submit. The default is to only fill it at init time and reuse that
data. Only makes sense if zero_buffers isn't specified, naturally. If
data verification is enabled, refill_buffers is also automatically
enabled."

Every IO submitted will have a new pattern for repeated workloads?
using refill_buffers, i seem to get same pattern repeatedly. Is that
how it should be? i was under the impression that refill will use new
pattern every time.

# fio --filename=test_refill --readwrite=randwrite --size=1m
--refill_buffers --bs=256k --norandommap --name=refill
# for each in {0..3}; do dd if=test_refill bs=262144 count=1
skip=$each 2>/dev/null | md5sum; done
86c3e47e173a58ad93470fd8db856f8c  -
6fe5ce86d04522889c99809d6d97cb94  -
9113ce00f3781861afc41aaf6da32428  -
f0b575c40e9da23b524f585723cbea4e  -

# fio --filename=test_refill --readwrite=randwrite --size=1m
--refill_buffers --bs=256k --norandommap --name=refill
# for each in {0..3}; do dd if=test_refill bs=262144 count=1
skip=$each 2>/dev/null | md5sum; done
86c3e47e173a58ad93470fd8db856f8c  -
6fe5ce86d04522889c99809d6d97cb94  -
9113ce00f3781861afc41aaf6da32428  -
f0b575c40e9da23b524f585723cbea4e  -
Srinivasa R Chamarthy


On Thu, May 7, 2015 at 10:19 AM, Matthew Eaton <m.eaton82@gmail.com> wrote:
> It doesn't make sense to set refill_buffers and scramble_buffers at
> the same time, since you won't get any repeating data with
> refill_buffers set.
>
> On Mon, May 4, 2015 at 7:23 PM, Srinivasa Chamarthy
> <chamarthy.raju@gmail.com> wrote:
>> Thanks Jens. I was using refill_buffers and scramble_buffers at the
>> same time. If i specify them at the same time, there seems to be no
>> difference in the workloads. If i specify scramble_buffers then there
>> shouldn't be refill_buffers specified to get the buffers differently.
>> Thanks.
>> Srinivasa R Chamarthy
>>
>>
>> On Tue, May 5, 2015 at 4:39 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>> On 04/25/2015 07:03 AM, Srinivasa Chamarthy wrote:
>>>>
>>>> Firstly, thanks before in hand. :)
>>>>
>>>> Using FIO, what is the way to achieve a work load with specific number
>>>> of unique blocks duplicates for specific number of times. For Example,
>>>> i want a 252k block duplicates 1000 times and i want it to be the
>>>> unique block in the complete workload.
>>>
>>>
>>> If you have very specific requirements, it might be easier to simply write
>>> an iolog generator that spits out what you want the pattern to look like.
>>> Then fio can import that, and run the workload.
>>>
>>>> Another query is that, for a blocksize of 256k does the io buffers are
>>>> created at 256k and written to the disk and are all those 256k blocks
>>>> unique buffer?. Can there be a change that in a 256k block, the
>>>> smaller size buffers be the same?
>>>
>>>
>>> That depends on the settings for compressibility, dedupability, etc. In
>>> general see scramble_buffers, refill_buffers, and so on. That is all in the
>>> man page or HOWTO.
>>>
>>> --
>>> Jens Axboe
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

One of the thread i went though:

http://www.spinics.net/lists/fio/msg00876.html

does state that using refill_buffers should give fully different
random data everytime.
Srinivasa R Chamarthy


On Thu, May 7, 2015 at 11:58 AM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> Thanks Eaton. One question thought. From the man page for refill_buffers:
>
> "If this option is given, fio will refill the IO buffers on every
> submit. The default is to only fill it at init time and reuse that
> data. Only makes sense if zero_buffers isn't specified, naturally. If
> data verification is enabled, refill_buffers is also automatically
> enabled."
>
> Every IO submitted will have a new pattern for repeated workloads?
> using refill_buffers, i seem to get same pattern repeatedly. Is that
> how it should be? i was under the impression that refill will use new
> pattern every time.
>
> # fio --filename=test_refill --readwrite=randwrite --size=1m
> --refill_buffers --bs=256k --norandommap --name=refill
> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
> skip=$each 2>/dev/null | md5sum; done
> 86c3e47e173a58ad93470fd8db856f8c  -
> 6fe5ce86d04522889c99809d6d97cb94  -
> 9113ce00f3781861afc41aaf6da32428  -
> f0b575c40e9da23b524f585723cbea4e  -
>
> # fio --filename=test_refill --readwrite=randwrite --size=1m
> --refill_buffers --bs=256k --norandommap --name=refill
> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
> skip=$each 2>/dev/null | md5sum; done
> 86c3e47e173a58ad93470fd8db856f8c  -
> 6fe5ce86d04522889c99809d6d97cb94  -
> 9113ce00f3781861afc41aaf6da32428  -
> f0b575c40e9da23b524f585723cbea4e  -
> Srinivasa R Chamarthy
>
>
> On Thu, May 7, 2015 at 10:19 AM, Matthew Eaton <m.eaton82@gmail.com> wrote:
>> It doesn't make sense to set refill_buffers and scramble_buffers at
>> the same time, since you won't get any repeating data with
>> refill_buffers set.
>>
>> On Mon, May 4, 2015 at 7:23 PM, Srinivasa Chamarthy
>> <chamarthy.raju@gmail.com> wrote:
>>> Thanks Jens. I was using refill_buffers and scramble_buffers at the
>>> same time. If i specify them at the same time, there seems to be no
>>> difference in the workloads. If i specify scramble_buffers then there
>>> shouldn't be refill_buffers specified to get the buffers differently.
>>> Thanks.
>>> Srinivasa R Chamarthy
>>>
>>>
>>> On Tue, May 5, 2015 at 4:39 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> On 04/25/2015 07:03 AM, Srinivasa Chamarthy wrote:
>>>>>
>>>>> Firstly, thanks before in hand. :)
>>>>>
>>>>> Using FIO, what is the way to achieve a work load with specific number
>>>>> of unique blocks duplicates for specific number of times. For Example,
>>>>> i want a 252k block duplicates 1000 times and i want it to be the
>>>>> unique block in the complete workload.
>>>>
>>>>
>>>> If you have very specific requirements, it might be easier to simply write
>>>> an iolog generator that spits out what you want the pattern to look like.
>>>> Then fio can import that, and run the workload.
>>>>
>>>>> Another query is that, for a blocksize of 256k does the io buffers are
>>>>> created at 256k and written to the disk and are all those 256k blocks
>>>>> unique buffer?. Can there be a change that in a 256k block, the
>>>>> smaller size buffers be the same?
>>>>
>>>>
>>>> That depends on the settings for compressibility, dedupability, etc. In
>>>> general see scramble_buffers, refill_buffers, and so on. That is all in the
>>>> man page or HOWTO.
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: achieving a workload to get specific unique duplicate blocks

[Jens Axboe]

On 05/06/2015 09:58 PM, Srinivasa Chamarthy wrote:
> Thanks Eaton. One question thought. From the man page for refill_buffers:
>
> "If this option is given, fio will refill the IO buffers on every
> submit. The default is to only fill it at init time and reuse that
> data. Only makes sense if zero_buffers isn't specified, naturally. If
> data verification is enabled, refill_buffers is also automatically
> enabled."
>
> Every IO submitted will have a new pattern for repeated workloads?
> using refill_buffers, i seem to get same pattern repeatedly. Is that
> how it should be? i was under the impression that refill will use new
> pattern every time.
>
> # fio --filename=test_refill --readwrite=randwrite --size=1m
> --refill_buffers --bs=256k --norandommap --name=refill
> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
> skip=$each 2>/dev/null | md5sum; done
> 86c3e47e173a58ad93470fd8db856f8c  -
> 6fe5ce86d04522889c99809d6d97cb94  -
> 9113ce00f3781861afc41aaf6da32428  -
> f0b575c40e9da23b524f585723cbea4e  -
>
> # fio --filename=test_refill --readwrite=randwrite --size=1m
> --refill_buffers --bs=256k --norandommap --name=refill
> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
> skip=$each 2>/dev/null | md5sum; done
> 86c3e47e173a58ad93470fd8db856f8c  -
> 6fe5ce86d04522889c99809d6d97cb94  -
> 9113ce00f3781861afc41aaf6da32428  -
> f0b575c40e9da23b524f585723cbea4e  -

First of all, please stop top posting. It breaks the flow of conversations.

Secondly, this is expected behavior, and it's done to get identical 
behavior between runs. If you had longer runs, each buffer would be 
different. But if you did a new run after that, the sequence of buffers 
would be the same. Same thing happens with the offsets generated, the 
buffer lengths generated, etc.

If you want a truly different run everytime, you have a few options. One 
is to simply set randrepeat=0, then OS entropy will seed the random 
generators. Or you can use randseed= to set a specific seed, which would 
allow you to reproduce a specific run.


-- 
Jens Axboe

Re: achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

[-- Attachment #1: Type: text/plain, Size: 2260 bytes --]

Sorry for that. Thanks for clarification on refill_buffers.
On 07-May-2015 10:27 pm, "Jens Axboe" <axboe@kernel.dk> wrote:

> On 05/06/2015 09:58 PM, Srinivasa Chamarthy wrote:
>
>> Thanks Eaton. One question thought. From the man page for refill_buffers:
>>
>> "If this option is given, fio will refill the IO buffers on every
>> submit. The default is to only fill it at init time and reuse that
>> data. Only makes sense if zero_buffers isn't specified, naturally. If
>> data verification is enabled, refill_buffers is also automatically
>> enabled."
>>
>> Every IO submitted will have a new pattern for repeated workloads?
>> using refill_buffers, i seem to get same pattern repeatedly. Is that
>> how it should be? i was under the impression that refill will use new
>> pattern every time.
>>
>> # fio --filename=test_refill --readwrite=randwrite --size=1m
>> --refill_buffers --bs=256k --norandommap --name=refill
>> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
>> skip=$each 2>/dev/null | md5sum; done
>> 86c3e47e173a58ad93470fd8db856f8c  -
>> 6fe5ce86d04522889c99809d6d97cb94  -
>> 9113ce00f3781861afc41aaf6da32428  -
>> f0b575c40e9da23b524f585723cbea4e  -
>>
>> # fio --filename=test_refill --readwrite=randwrite --size=1m
>> --refill_buffers --bs=256k --norandommap --name=refill
>> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
>> skip=$each 2>/dev/null | md5sum; done
>> 86c3e47e173a58ad93470fd8db856f8c  -
>> 6fe5ce86d04522889c99809d6d97cb94  -
>> 9113ce00f3781861afc41aaf6da32428  -
>> f0b575c40e9da23b524f585723cbea4e  -
>>
>
> First of all, please stop top posting. It breaks the flow of conversations.
>
> Secondly, this is expected behavior, and it's done to get identical
> behavior between runs. If you had longer runs, each buffer would be
> different. But if you did a new run after that, the sequence of buffers
> would be the same. Same thing happens with the offsets generated, the
> buffer lengths generated, etc.
>
> If you want a truly different run everytime, you have a few options. One
> is to simply set randrepeat=0, then OS entropy will seed the random
> generators. Or you can use randseed= to set a specific seed, which would
> allow you to reproduce a specific run.
>
>
> --
> Jens Axboe
>
>

[-- Attachment #2: Type: text/html, Size: 2738 bytes --]

RE: achieving a workload to get specific unique duplicate blocks

[Ruvinsky, Konstantin]

Hi,

>If you want a truly different run everytime, you have a few options. One is to simply set randrepeat=0, then OS entropy will seed the random generators. Or you can use randseed= to set a specific seed, which would allow you to reproduce a specific run.

Two points:

1. Looks like randseed is not documented in HowTo (see http://git.kernel.dk/?p=fio.git;a=blob;f=HOWTO;h=005dac251dd202917fdb446ffec22d3126757f8c;hb=HEAD)
2. Maybe make sense to add to HowTo that randrepeat=1 is the default value.

Regards
Konstantin

RE: achieving a workload to get specific unique duplicate blocks

[Ruvinsky, Konstantin]

Sorry about randseed, looked at wrong link. The right one is http://git.kernel.dk/?p=fio.git;a=blob;f=HOWTO;h=0808cc3a07e0f53a62c3476b6789f47426917829;hb=HEAD


-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Ruvinsky, Konstantin
Sent: Friday, May 08, 2015 9:04 AM
To: Jens Axboe
Cc: fio@vger.kernel.org
Subject: RE: achieving a workload to get specific unique duplicate blocks

Hi,

>If you want a truly different run everytime, you have a few options. One is to simply set randrepeat=0, then OS entropy will seed the random generators. Or you can use randseed= to set a specific seed, which would allow you to reproduce a specific run.

Two points:

1. Looks like randseed is not documented in HowTo (see http://git.kernel.dk/?p=fio.git;a=blob;f=HOWTO;h=005dac251dd202917fdb446ffec22d3126757f8c;hb=HEAD)
2. Maybe make sense to add to HowTo that randrepeat=1 is the default value.

Regards
Konstantin
N�����r��y���b�X��ǧv�^�)޺{.n�+�������\x17��ܨ}���Ơz�&j:+v���\r����zZ+��+zf���h���~����i���z�\x1e�w���?����&�)ߢ^[f

Re: achieving a workload to get specific unique duplicate blocks

[Srinivasa Chamarthy]

My reply din't seem to get through the mailing list. Blame HTML
enabled on my mobile. :)

Again. Sorry for top posting. Thought i was continuing the thread :(

Thanks for clarification on refill_buffers. I get FIO now after trials
and help from you guys.

--
Srinivasa Chamarhy

--
Srinivasa R Chamarthy


On Thu, May 7, 2015 at 11:30 PM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> Sorry for that. Thanks for clarification on refill_buffers.
>
> On 07-May-2015 10:27 pm, "Jens Axboe" <axboe@kernel.dk> wrote:
>>
>> On 05/06/2015 09:58 PM, Srinivasa Chamarthy wrote:
>>>
>>> Thanks Eaton. One question thought. From the man page for refill_buffers:
>>>
>>> "If this option is given, fio will refill the IO buffers on every
>>> submit. The default is to only fill it at init time and reuse that
>>> data. Only makes sense if zero_buffers isn't specified, naturally. If
>>> data verification is enabled, refill_buffers is also automatically
>>> enabled."
>>>
>>> Every IO submitted will have a new pattern for repeated workloads?
>>> using refill_buffers, i seem to get same pattern repeatedly. Is that
>>> how it should be? i was under the impression that refill will use new
>>> pattern every time.
>>>
>>> # fio --filename=test_refill --readwrite=randwrite --size=1m
>>> --refill_buffers --bs=256k --norandommap --name=refill
>>> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
>>> skip=$each 2>/dev/null | md5sum; done
>>> 86c3e47e173a58ad93470fd8db856f8c  -
>>> 6fe5ce86d04522889c99809d6d97cb94  -
>>> 9113ce00f3781861afc41aaf6da32428  -
>>> f0b575c40e9da23b524f585723cbea4e  -
>>>
>>> # fio --filename=test_refill --readwrite=randwrite --size=1m
>>> --refill_buffers --bs=256k --norandommap --name=refill
>>> # for each in {0..3}; do dd if=test_refill bs=262144 count=1
>>> skip=$each 2>/dev/null | md5sum; done
>>> 86c3e47e173a58ad93470fd8db856f8c  -
>>> 6fe5ce86d04522889c99809d6d97cb94  -
>>> 9113ce00f3781861afc41aaf6da32428  -
>>> f0b575c40e9da23b524f585723cbea4e  -
>>
>>
>> First of all, please stop top posting. It breaks the flow of
>> conversations.
>>
>> Secondly, this is expected behavior, and it's done to get identical
>> behavior between runs. If you had longer runs, each buffer would be
>> different. But if you did a new run after that, the sequence of buffers
>> would be the same. Same thing happens with the offsets generated, the buffer
>> lengths generated, etc.
>>
>> If you want a truly different run everytime, you have a few options. One
>> is to simply set randrepeat=0, then OS entropy will seed the random
>> generators. Or you can use randseed= to set a specific seed, which would
>> allow you to reproduce a specific run.
>>
>>
>> --
>> Jens Axboe
>>
>