[fio-2.1.9] verify and bssplit do not work together

[fio-2.1.9] verify and bssplit do not work together

[Gwendal Grignou]

Jens,

I notice that when I enable bssplit and verify, verify always fails:

...
meta: verify failed at file /tmp/test offset 7143424, length 65536
open verify buf file: Read-only file system
....
In my tests, the offsets that fails are always:
7143424, 15073280, 2647654, 450397184, 88080384, 88211456

it does not happen when I set a fixed size with bs, or when I do not verify.

Here is the fio control file I use:

[stress]
filename=/tmp/test
size=107374182

readwrite=randrw
bssplit=64k/50:1M/50
;bs=64k

do_verify=1
verify=meta
verify_interval=64k
verify_dump=1
continue_on_error=verify

Also, if I set a verify_interval larger than the smallest io size in
bssplt, I get another kind of error.
Error messages with bssplt=4k/50:1M/50:

verify: bad magic header a678, wanted acca at file /tmp/test offset
9097216, length 429524186
verify: bad magic header 84b9, wanted acca at file /tmp/test offset
25268224, length 21518097
verify: bad magic header ba51, wanted acca at file /tmp/test offset
26505216, length 536429778
verify: bad magic header 1197, wanted acca at file /tmp/test offset
42139648, length 373139796
verify: bad magic header 22bf, wanted acca at file /tmp/test offset
42209280, length 493427719
meta: verify failed at file /tmp/test offset 50462720, length 65536
open verify buf file: Read-only file system
open verify buf file: Read-only file system
verify: bad magic header d1f6, wanted acca at file /tmp/test offset
52305920, length 322125536
verify: bad magic header 76ee, wanted acca at file /tmp/test offset
92106752, length 355054425
verify: bad magic header 821, wanted acca at file /tmp/test offset
94904320, length 460051914

I don't get these error when I set bs=4k.
Are you aware of such a limitation in fio?

Thanks,

Gwendal.

Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe]

On 2014-07-17 22:36, Gwendal Grignou wrote:
> Jens,
>
> I notice that when I enable bssplit and verify, verify always fails:
>
> ...
> meta: verify failed at file /tmp/test offset 7143424, length 65536
> open verify buf file: Read-only file system
> ....
> In my tests, the offsets that fails are always:
> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456
>
> it does not happen when I set a fixed size with bs, or when I do not verify.
>
> Here is the fio control file I use:
>
> [stress]
> filename=/tmp/test
> size=107374182
>
> readwrite=randrw
> bssplit=64k/50:1M/50
> ;bs=64k
>
> do_verify=1
> verify=meta
> verify_interval=64k
> verify_dump=1
> continue_on_error=verify

That does sound like a bug. Out of curiosity, does anything change if 
you set experimental_verify=1 in the job?

I'll take a look at this next week, currently away on vacation.

> Also, if I set a verify_interval larger than the smallest io size in
> bssplt, I get another kind of error.
> Error messages with bssplt=4k/50:1M/50:
>
> verify: bad magic header a678, wanted acca at file /tmp/test offset
> 9097216, length 429524186
> verify: bad magic header 84b9, wanted acca at file /tmp/test offset
> 25268224, length 21518097
> verify: bad magic header ba51, wanted acca at file /tmp/test offset
> 26505216, length 536429778
> verify: bad magic header 1197, wanted acca at file /tmp/test offset
> 42139648, length 373139796
> verify: bad magic header 22bf, wanted acca at file /tmp/test offset
> 42209280, length 493427719
> meta: verify failed at file /tmp/test offset 50462720, length 65536
> open verify buf file: Read-only file system
> open verify buf file: Read-only file system
> verify: bad magic header d1f6, wanted acca at file /tmp/test offset
> 52305920, length 322125536
> verify: bad magic header 76ee, wanted acca at file /tmp/test offset
> 92106752, length 355054425
> verify: bad magic header 821, wanted acca at file /tmp/test offset
> 94904320, length 460051914
>
> I don't get these error when I set bs=4k.
> Are you aware of such a limitation in fio?

Fio doesn't support verify intervals larger than a single written block, 
so that is something the parser or option checker should catch. 
Apparently it doesn't. I'll take a look at this too.


-- 
Jens Axboe

Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 2517 bytes --]

On 2014-07-18 09:47, Jens Axboe wrote:
> On 2014-07-17 22:36, Gwendal Grignou wrote:
>> Jens,
>>
>> I notice that when I enable bssplit and verify, verify always fails:
>>
>> ...
>> meta: verify failed at file /tmp/test offset 7143424, length 65536
>> open verify buf file: Read-only file system
>> ....
>> In my tests, the offsets that fails are always:
>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456
>>
>> it does not happen when I set a fixed size with bs, or when I do not
>> verify.
>>
>> Here is the fio control file I use:
>>
>> [stress]
>> filename=/tmp/test
>> size=107374182
>>
>> readwrite=randrw
>> bssplit=64k/50:1M/50
>> ;bs=64k
>>
>> do_verify=1
>> verify=meta
>> verify_interval=64k
>> verify_dump=1
>> continue_on_error=verify
>
> That does sound like a bug. Out of curiosity, does anything change if
> you set experimental_verify=1 in the job?
>
> I'll take a look at this next week, currently away on vacation.
>
>> Also, if I set a verify_interval larger than the smallest io size in
>> bssplt, I get another kind of error.
>> Error messages with bssplt=4k/50:1M/50:
>>
>> verify: bad magic header a678, wanted acca at file /tmp/test offset
>> 9097216, length 429524186
>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset
>> 25268224, length 21518097
>> verify: bad magic header ba51, wanted acca at file /tmp/test offset
>> 26505216, length 536429778
>> verify: bad magic header 1197, wanted acca at file /tmp/test offset
>> 42139648, length 373139796
>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset
>> 42209280, length 493427719
>> meta: verify failed at file /tmp/test offset 50462720, length 65536
>> open verify buf file: Read-only file system
>> open verify buf file: Read-only file system
>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset
>> 52305920, length 322125536
>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset
>> 92106752, length 355054425
>> verify: bad magic header 821, wanted acca at file /tmp/test offset
>> 94904320, length 460051914
>>
>> I don't get these error when I set bs=4k.
>> Are you aware of such a limitation in fio?
>
> Fio doesn't support verify intervals larger than a single written block,
> so that is something the parser or option checker should catch.
> Apparently it doesn't. I'll take a look at this too.

Looks like it's numberio biting us again... Can you try with this patch? 
For the mixed case, haven't looked into interval > bs yet.

-- 
Jens Axboe


[-- Attachment #2: verify-mixed-numberio.patch --]
[-- Type: text/x-patch, Size: 1121 bytes --]

diff --git a/verify.c b/verify.c
index e59a4b290510..7c99e15e73be 100644
--- a/verify.c
+++ b/verify.c
@@ -405,13 +405,13 @@ static int verify_io_u_meta(struct verify_header *hdr, struct vcont *vc)
 
 	/*
 	 * For read-only workloads, the program cannot be certain of the
-	 * last numberio written to a block. Checking of numberio will be done
-	 * only for workloads that write data.
-	 * For verify_only, numberio will be checked in the last iteration when
-	 * the correct state of numberio, that would have been written to each
-	 * block in a previous run of fio, has been reached.
+	 * last numberio written to a block. Checking of numberio will be
+	 * done only for workloads that write data.  For verify_only,
+	 * numberio will be checked in the last iteration when the correct
+	 * state of numberio, that would have been written to each block
+	 * in a previous run of fio, has been reached.
 	 */
-	if (td_write(td) || td_rw(td))
+	if ((td_write(td) || td_rw(td)) && (td_min_bs(td) == td_max_bs(td)))
 		if (!td->o.verify_only || td->o.loops == 0)
 			if (vh->numberio != io_u->numberio)
 				ret = EILSEQ;

Re: [fio-2.1.9] verify and bssplit do not work together

[Gwendal Grignou]

[-- Attachment #1: Type: text/plain, Size: 3195 bytes --]

Jens,

Without the patch, experimental_verify=1 produced more error, it looks like
every 64k.

With the patch above, fio behaves as expected: do_verify and
experimental_verify
works for mixed writes.

There is still a problem when bs is mixed and one argument is less than the
verify interval.
For instance:
bssplit=4k/50:1M/50
...
verify_interval=64k


But looking at the code, it won't be possible to fix, because we can not
guarantee to always have 16 4K write contiguous.

Thanks for the patch.

Gwendal.




On Mon, Jul 21, 2014 at 1:29 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 2014-07-18 09:47, Jens Axboe wrote:
>
>> On 2014-07-17 22:36, Gwendal Grignou wrote:
>>
>>> Jens,
>>>
>>> I notice that when I enable bssplit and verify, verify always fails:
>>>
>>> ...
>>> meta: verify failed at file /tmp/test offset 7143424, length 65536
>>> open verify buf file: Read-only file system
>>> ....
>>> In my tests, the offsets that fails are always:
>>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456
>>>
>>> it does not happen when I set a fixed size with bs, or when I do not
>>> verify.
>>>
>>> Here is the fio control file I use:
>>>
>>> [stress]
>>> filename=/tmp/test
>>> size=107374182
>>>
>>> readwrite=randrw
>>> bssplit=64k/50:1M/50
>>> ;bs=64k
>>>
>>> do_verify=1
>>> verify=meta
>>> verify_interval=64k
>>> verify_dump=1
>>> continue_on_error=verify
>>>
>>
>> That does sound like a bug. Out of curiosity, does anything change if
>> you set experimental_verify=1 in the job?
>>
>> I'll take a look at this next week, currently away on vacation.
>>
>>  Also, if I set a verify_interval larger than the smallest io size in
>>> bssplt, I get another kind of error.
>>> Error messages with bssplt=4k/50:1M/50:
>>>
>>> verify: bad magic header a678, wanted acca at file /tmp/test offset
>>> 9097216, length 429524186
>>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset
>>> 25268224, length 21518097
>>> verify: bad magic header ba51, wanted acca at file /tmp/test offset
>>> 26505216, length 536429778
>>> verify: bad magic header 1197, wanted acca at file /tmp/test offset
>>> 42139648, length 373139796
>>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset
>>> 42209280, length 493427719
>>> meta: verify failed at file /tmp/test offset 50462720, length 65536
>>> open verify buf file: Read-only file system
>>> open verify buf file: Read-only file system
>>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset
>>> 52305920, length 322125536
>>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset
>>> 92106752, length 355054425
>>> verify: bad magic header 821, wanted acca at file /tmp/test offset
>>> 94904320, length 460051914
>>>
>>> I don't get these error when I set bs=4k.
>>> Are you aware of such a limitation in fio?
>>>
>>
>> Fio doesn't support verify intervals larger than a single written block,
>> so that is something the parser or option checker should catch.
>> Apparently it doesn't. I'll take a look at this too.
>>
>
> Looks like it's numberio biting us again... Can you try with this patch?
> For the mixed case, haven't looked into interval > bs yet.
>
> --
> Jens Axboe
>
>

[-- Attachment #2: Type: text/html, Size: 4954 bytes --]

Re: [fio-2.1.9] verify and bssplit do not work together

[Gwendal Grignou]

Jens,

Without the patch, experimental_verify=1 produced more error, it looks
like every 64k.

With the patch above, fio behaves as expected: do_verify and
experimental_verify works for mixed writes.

There is still a problem when bs is mixed and one argument is less
than the verify interval.
For instance:
bssplit=4k/50:1M/50
...
verify_interval=64k


But looking at the code, it won't be possible to fix, because we can
not guarantee to always have 16 4K write contiguous.

Thanks for the patch.

Gwendal.

On Mon, Jul 21, 2014 at 1:29 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2014-07-18 09:47, Jens Axboe wrote:
>>
>> On 2014-07-17 22:36, Gwendal Grignou wrote:
>>>
>>> Jens,
>>>
>>> I notice that when I enable bssplit and verify, verify always fails:
>>>
>>> ...
>>> meta: verify failed at file /tmp/test offset 7143424, length 65536
>>> open verify buf file: Read-only file system
>>> ....
>>> In my tests, the offsets that fails are always:
>>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456
>>>
>>> it does not happen when I set a fixed size with bs, or when I do not
>>> verify.
>>>
>>> Here is the fio control file I use:
>>>
>>> [stress]
>>> filename=/tmp/test
>>> size=107374182
>>>
>>> readwrite=randrw
>>> bssplit=64k/50:1M/50
>>> ;bs=64k
>>>
>>> do_verify=1
>>> verify=meta
>>> verify_interval=64k
>>> verify_dump=1
>>> continue_on_error=verify
>>
>>
>> That does sound like a bug. Out of curiosity, does anything change if
>> you set experimental_verify=1 in the job?
>>
>> I'll take a look at this next week, currently away on vacation.
>>
>>> Also, if I set a verify_interval larger than the smallest io size in
>>> bssplt, I get another kind of error.
>>> Error messages with bssplt=4k/50:1M/50:
>>>
>>> verify: bad magic header a678, wanted acca at file /tmp/test offset
>>> 9097216, length 429524186
>>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset
>>> 25268224, length 21518097
>>> verify: bad magic header ba51, wanted acca at file /tmp/test offset
>>> 26505216, length 536429778
>>> verify: bad magic header 1197, wanted acca at file /tmp/test offset
>>> 42139648, length 373139796
>>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset
>>> 42209280, length 493427719
>>> meta: verify failed at file /tmp/test offset 50462720, length 65536
>>> open verify buf file: Read-only file system
>>> open verify buf file: Read-only file system
>>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset
>>> 52305920, length 322125536
>>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset
>>> 92106752, length 355054425
>>> verify: bad magic header 821, wanted acca at file /tmp/test offset
>>> 94904320, length 460051914
>>>
>>> I don't get these error when I set bs=4k.
>>> Are you aware of such a limitation in fio?
>>
>>
>> Fio doesn't support verify intervals larger than a single written block,
>> so that is something the parser or option checker should catch.
>> Apparently it doesn't. I'll take a look at this too.
>
>
> Looks like it's numberio biting us again... Can you try with this patch? For
> the mixed case, haven't looked into interval > bs yet.
>
> --
> Jens Axboe
>

Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe]

On 2014-07-21 19:49, Gwendal Grignou wrote:
> Jens,
>
> Without the patch, experimental_verify=1 produced more error, it looks
> like every 64k.
>
> With the patch above, fio behaves as expected: do_verify and
> experimental_verify works for mixed writes.
>
> There is still a problem when bs is mixed and one argument is less
> than the verify interval.
> For instance:
> bssplit=4k/50:1M/50
> ...
> verify_interval=64k
>
>
> But looking at the code, it won't be possible to fix, because we can
> not guarantee to always have 16 4K write contiguous.

As mentioned in the original email, it's a parser fix. verify_interval 
cannot be less than the minimum block size, fio will just have to catch 
that and adjust / error out.

> Thanks for the patch.

It's committed, so generally available now.

-- 
Jens Axboe

Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

On 2014-07-21 20:40, Jens Axboe wrote:
>> But looking at the code, it won't be possible to fix, because we can
>> not guarantee to always have 16 4K write contiguous.
>
> As mentioned in the original email, it's a parser fix. verify_interval
> cannot be less than the minimum block size, fio will just have to catch
> that and adjust / error out.

Attach should fix that - untested...

-- 
Jens Axboe


[-- Attachment #2: x.patch --]
[-- Type: text/x-patch, Size: 425 bytes --]

diff --git a/init.c b/init.c
index cf1d7f938fb3..cf2f82220f72 100644
--- a/init.c
+++ b/init.c
@@ -635,6 +635,9 @@ static int fixup_options(struct thread_data *td)
 		if (o->max_bs[DDIR_WRITE] != o->min_bs[DDIR_WRITE] &&
 		    !o->verify_interval)
 			o->verify_interval = o->min_bs[DDIR_WRITE];
+
+		if (o->verify_interval > o->min_bs[DDIR_WRITE])
+			o->verify_interval = o->min_bs[DDIR_WRITE];
 	}
 
 	if (o->pre_read) {

Re: [fio-2.1.9] verify and bssplit do not work together

[Gwendal Grignou]

Jens,
 I still see some verify errors when I set 'experimental_verify=1'

verify: bad magic header 977c, wanted acca at file /tmp/test offset
15478784, length 429437046
verify: bad magic header 8aac, wanted acca at file /tmp/test offset
28082176, length 302426815
verify: bad magic header c8ad, wanted acca at file /tmp/test offset
30408704, length 318473336
verify: bad magic header 3791, wanted acca at file /tmp/test offset
65024000, length 74750452

However, without this line, there are no verify errors anymore.

The fio file is:
[stress]
filename=/tmp/test
size=107374182

readwrite=randrw
bssplit=4k/50:1M/50
;bs=64k

do_verify=1
experimental_verify=1
verify=meta
verify_interval=64k
verify_dump=1
continue_on_error=verify


Thanks a lot,
Gwendal.

On Mon, Jul 21, 2014 at 11:52 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2014-07-21 20:40, Jens Axboe wrote:
>>>
>>> But looking at the code, it won't be possible to fix, because we can
>>> not guarantee to always have 16 4K write contiguous.
>>
>>
>> As mentioned in the original email, it's a parser fix. verify_interval
>> cannot be less than the minimum block size, fio will just have to catch
>> that and adjust / error out.
>
>
> Attach should fix that - untested...
>
> --
> Jens Axboe
>

Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe]

On 2014-07-21 22:24, Gwendal Grignou wrote:
> Jens,
>   I still see some verify errors when I set 'experimental_verify=1'
>
> verify: bad magic header 977c, wanted acca at file /tmp/test offset
> 15478784, length 429437046
> verify: bad magic header 8aac, wanted acca at file /tmp/test offset
> 28082176, length 302426815
> verify: bad magic header c8ad, wanted acca at file /tmp/test offset
> 30408704, length 318473336
> verify: bad magic header 3791, wanted acca at file /tmp/test offset
> 65024000, length 74750452
>
> However, without this line, there are no verify errors anymore.
>
> The fio file is:
> [stress]
> filename=/tmp/test
> size=107374182
>
> readwrite=randrw
> bssplit=4k/50:1M/50
> ;bs=64k
>
> do_verify=1
> experimental_verify=1
> verify=meta
> verify_interval=64k
> verify_dump=1
> continue_on_error=verify

I'll look into the experimental_verify bit - there's a reason it's 
experimental :-). But the hope is to make that the method for 
verification, so I'll look into what goes wrong there. It might be 
highlighting problems elsewhere (like accidental overwrites, due to the 
mixed block sizes).

-- 
Jens Axboe