Re: [fio-2.1.9] verify and bssplit do not work together

[Jens Axboe <axboe@kernel.dk>]

[-- Attachment #1: Type: text/plain, Size: 2517 bytes --]

On 2014-07-18 09:47, Jens Axboe wrote:
> On 2014-07-17 22:36, Gwendal Grignou wrote:
>> Jens,
>>
>> I notice that when I enable bssplit and verify, verify always fails:
>>
>> ...
>> meta: verify failed at file /tmp/test offset 7143424, length 65536
>> open verify buf file: Read-only file system
>> ....
>> In my tests, the offsets that fails are always:
>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456
>>
>> it does not happen when I set a fixed size with bs, or when I do not
>> verify.
>>
>> Here is the fio control file I use:
>>
>> [stress]
>> filename=/tmp/test
>> size=107374182
>>
>> readwrite=randrw
>> bssplit=64k/50:1M/50
>> ;bs=64k
>>
>> do_verify=1
>> verify=meta
>> verify_interval=64k
>> verify_dump=1
>> continue_on_error=verify
>
> That does sound like a bug. Out of curiosity, does anything change if
> you set experimental_verify=1 in the job?
>
> I'll take a look at this next week, currently away on vacation.
>
>> Also, if I set a verify_interval larger than the smallest io size in
>> bssplt, I get another kind of error.
>> Error messages with bssplt=4k/50:1M/50:
>>
>> verify: bad magic header a678, wanted acca at file /tmp/test offset
>> 9097216, length 429524186
>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset
>> 25268224, length 21518097
>> verify: bad magic header ba51, wanted acca at file /tmp/test offset
>> 26505216, length 536429778
>> verify: bad magic header 1197, wanted acca at file /tmp/test offset
>> 42139648, length 373139796
>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset
>> 42209280, length 493427719
>> meta: verify failed at file /tmp/test offset 50462720, length 65536
>> open verify buf file: Read-only file system
>> open verify buf file: Read-only file system
>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset
>> 52305920, length 322125536
>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset
>> 92106752, length 355054425
>> verify: bad magic header 821, wanted acca at file /tmp/test offset
>> 94904320, length 460051914
>>
>> I don't get these error when I set bs=4k.
>> Are you aware of such a limitation in fio?
>
> Fio doesn't support verify intervals larger than a single written block,
> so that is something the parser or option checker should catch.
> Apparently it doesn't. I'll take a look at this too.

Looks like it's numberio biting us again... Can you try with this patch? 
For the mixed case, haven't looked into interval > bs yet.

-- 
Jens Axboe


[-- Attachment #2: verify-mixed-numberio.patch --]
[-- Type: text/x-patch, Size: 1121 bytes --]

diff --git a/verify.c b/verify.c
index e59a4b290510..7c99e15e73be 100644
--- a/verify.c
+++ b/verify.c
@@ -405,13 +405,13 @@ static int verify_io_u_meta(struct verify_header *hdr, struct vcont *vc)
 
 	/*
 	 * For read-only workloads, the program cannot be certain of the
-	 * last numberio written to a block. Checking of numberio will be done
-	 * only for workloads that write data.
-	 * For verify_only, numberio will be checked in the last iteration when
-	 * the correct state of numberio, that would have been written to each
-	 * block in a previous run of fio, has been reached.
+	 * last numberio written to a block. Checking of numberio will be
+	 * done only for workloads that write data.  For verify_only,
+	 * numberio will be checked in the last iteration when the correct
+	 * state of numberio, that would have been written to each block
+	 * in a previous run of fio, has been reached.
 	 */
-	if (td_write(td) || td_rw(td))
+	if ((td_write(td) || td_rw(td)) && (td_min_bs(td) == td_max_bs(td)))
 		if (!td->o.verify_only || td->o.loops == 0)
 			if (vh->numberio != io_u->numberio)
 				ret = EILSEQ;