Cannot verify dedupe data, is it a bug?

Cannot verify dedupe data, is it a bug?

[Changyu]

Hi,

I set "dedupe_percentage " to let FIO generate some duplicated data.  I also want read these data back and verify it.
But, there is no duplicated data after setting "verify" parameter.

checking the source code, I found that  function do_io  in file backend.c fill the buffer twice. the source code is as below.

do_io()
{
    io_u = get_io_u(td);
    .....
    if (io_u->ddir == DDIR_WRITE && td->flags & TD_F_DO_VERIFY)
        populate_verify_io_u(td, io_u);
}

In function get_io_u, the buffer is filled with data as the "dedupe_percentage " specified.
In function populate_verify_io_u, the buffer is refilled with random data. So, there would be no duplicated data.

Is this a bug?

Re: Cannot verify dedupe data, is it a bug?

[Sitsofe Wheeler]

On Wed, 17 Oct 2018 at 13:31, Changyu <feng_changyu@126.com> wrote:
>
> Hi,
>
> I set "dedupe_percentage " to let FIO generate some duplicated data.  I also want read these data back and verify it.
> But, there is no duplicated data after setting "verify" parameter.
>
> checking the source code, I found that  function do_io  in file backend.c fill the buffer twice. the source code is as below.
>
> do_io()
> {
>     io_u = get_io_u(td);
>     .....
>     if (io_u->ddir == DDIR_WRITE && td->flags & TD_F_DO_VERIFY)
>         populate_verify_io_u(td, io_u);
> }
>
> In function get_io_u, the buffer is filled with data as the "dedupe_percentage " specified.
> In function populate_verify_io_u, the buffer is refilled with random data. So, there would be no duplicated data.
>
> Is this a bug?

If it's not (and I'd say it is) then the current behaviour needs to it
needs to print a "conflicting options" error rather than silently not
doing what was asked. Someone else has reported something similar over
on https://github.com/axboe/fio/issues/449 ...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Cannot verify dedupe data, is it a bug?

[Jens Axboe]

On 10/17/18 12:32 PM, Sitsofe Wheeler wrote:
> On Wed, 17 Oct 2018 at 13:31, Changyu <feng_changyu@126.com> wrote:
>>
>> Hi,
>>
>> I set "dedupe_percentage " to let FIO generate some duplicated data.  I also want read these data back and verify it.
>> But, there is no duplicated data after setting "verify" parameter.
>>
>> checking the source code, I found that  function do_io  in file backend.c fill the buffer twice. the source code is as below.
>>
>> do_io()
>> {
>>     io_u = get_io_u(td);
>>     .....
>>     if (io_u->ddir == DDIR_WRITE && td->flags & TD_F_DO_VERIFY)
>>         populate_verify_io_u(td, io_u);
>> }
>>
>> In function get_io_u, the buffer is filled with data as the "dedupe_percentage " specified.
>> In function populate_verify_io_u, the buffer is refilled with random data. So, there would be no duplicated data.
>>
>> Is this a bug?
> 
> If it's not (and I'd say it is) then the current behaviour needs to it
> needs to print a "conflicting options" error rather than silently not
> doing what was asked. Someone else has reported something similar over
> on https://github.com/axboe/fio/issues/449 ...

They should definitely be using the same fill methods.

-- 
Jens Axboe

Re:Re: Cannot verify dedupe data, is it a bug?

[Changyu]

Hi Jens Axboe & Sitsofe Wheeler


Thank you very much for your help.

get_io_u finally call fill_io_buffer to fill the buffer, function fill_io_buffer uses td->buf_state and td->buf_state_prev to generate duplicated data if the dedupe_percentage is set.

populate_verify_io_u finally call  fill_random_buf_percentage to fill the buffer, it use td->verify_state to generate random data.

Should I create a issue on https://github.com/axboe/fio/issues?



At 2018-10-18 03:05:33, "Jens Axboe" <axboe@kernel.dk> wrote:
>On 10/17/18 12:32 PM, Sitsofe Wheeler wrote:
>> On Wed, 17 Oct 2018 at 13:31, Changyu <feng_changyu@126.com> wrote:
>>>
>>> Hi,
>>>
>>> I set "dedupe_percentage " to let FIO generate some duplicated data.  I also want read these data back and verify it.
>>> But, there is no duplicated data after setting "verify" parameter.
>>>
>>> checking the source code, I found that  function do_io  in file backend.c fill the buffer twice. the source code is as below.
>>>
>>> do_io()
>>> {
>>>     io_u = get_io_u(td);
>>>     .....
>>>     if (io_u->ddir == DDIR_WRITE && td->flags & TD_F_DO_VERIFY)
>>>         populate_verify_io_u(td, io_u);
>>> }
>>>
>>> In function get_io_u, the buffer is filled with data as the "dedupe_percentage " specified.
>>> In function populate_verify_io_u, the buffer is refilled with random data. So, there would be no duplicated data.
>>>
>>> Is this a bug?
>> 
>> If it's not (and I'd say it is) then the current behaviour needs to it
>> needs to print a "conflicting options" error rather than silently not
>> doing what was asked. Someone else has reported something similar over
>> on https://github.com/axboe/fio/issues/449 ...
>
>They should definitely be using the same fill methods.
>
>-- 
>Jens Axboe

Re: Cannot verify dedupe data, is it a bug?

[Jens Axboe]

On 10/18/18 2:16 AM, Changyu wrote:
> 
> Hi Jens Axboe & Sitsofe Wheeler
> 
> 
> Thank you very much for your help.
> 
> get_io_u finally call fill_io_buffer to fill the buffer, function
> fill_io_buffer uses td->buf_state and td->buf_state_prev to generate
> duplicated data if the dedupe_percentage is set.
> 
> populate_verify_io_u finally call  fill_random_buf_percentage to fill
> the buffer, it use td->verify_state to generate random data.
> 

As I said, I'm fine with that getting unified. The only thing to keep in
mind is that for verification data, we rely on being able to re-generate
the buffer contents, as long as we have a valid verify header. This can
be used to diagnose the nature of the corruption. As long as we retain
that property, the buffer generation for verify writes should follow the
same settings as non-verify writes.

> Should I create a issue on https://github.com/axboe/fio/issues?

Sure, probably check if it's a dupe of the issue Sitsofe referenced
first.


-- 
Jens Axboe