* [fio-2.1.9] verify and bssplit do not work together @ 2014-07-17 20:36 Gwendal Grignou 2014-07-18 7:47 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Gwendal Grignou @ 2014-07-17 20:36 UTC (permalink / raw) To: axboe, Puthikorn Voravootivat, fio Jens, I notice that when I enable bssplit and verify, verify always fails: ... meta: verify failed at file /tmp/test offset 7143424, length 65536 open verify buf file: Read-only file system .... In my tests, the offsets that fails are always: 7143424, 15073280, 2647654, 450397184, 88080384, 88211456 it does not happen when I set a fixed size with bs, or when I do not verify. Here is the fio control file I use: [stress] filename=/tmp/test size=107374182 readwrite=randrw bssplit=64k/50:1M/50 ;bs=64k do_verify=1 verify=meta verify_interval=64k verify_dump=1 continue_on_error=verify Also, if I set a verify_interval larger than the smallest io size in bssplt, I get another kind of error. Error messages with bssplt=4k/50:1M/50: verify: bad magic header a678, wanted acca at file /tmp/test offset 9097216, length 429524186 verify: bad magic header 84b9, wanted acca at file /tmp/test offset 25268224, length 21518097 verify: bad magic header ba51, wanted acca at file /tmp/test offset 26505216, length 536429778 verify: bad magic header 1197, wanted acca at file /tmp/test offset 42139648, length 373139796 verify: bad magic header 22bf, wanted acca at file /tmp/test offset 42209280, length 493427719 meta: verify failed at file /tmp/test offset 50462720, length 65536 open verify buf file: Read-only file system open verify buf file: Read-only file system verify: bad magic header d1f6, wanted acca at file /tmp/test offset 52305920, length 322125536 verify: bad magic header 76ee, wanted acca at file /tmp/test offset 92106752, length 355054425 verify: bad magic header 821, wanted acca at file /tmp/test offset 94904320, length 460051914 I don't get these error when I set bs=4k. Are you aware of such a limitation in fio? Thanks, Gwendal. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-17 20:36 [fio-2.1.9] verify and bssplit do not work together Gwendal Grignou @ 2014-07-18 7:47 ` Jens Axboe 2014-07-21 8:29 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Jens Axboe @ 2014-07-18 7:47 UTC (permalink / raw) To: Gwendal Grignou, Puthikorn Voravootivat, fio On 2014-07-17 22:36, Gwendal Grignou wrote: > Jens, > > I notice that when I enable bssplit and verify, verify always fails: > > ... > meta: verify failed at file /tmp/test offset 7143424, length 65536 > open verify buf file: Read-only file system > .... > In my tests, the offsets that fails are always: > 7143424, 15073280, 2647654, 450397184, 88080384, 88211456 > > it does not happen when I set a fixed size with bs, or when I do not verify. > > Here is the fio control file I use: > > [stress] > filename=/tmp/test > size=107374182 > > readwrite=randrw > bssplit=64k/50:1M/50 > ;bs=64k > > do_verify=1 > verify=meta > verify_interval=64k > verify_dump=1 > continue_on_error=verify That does sound like a bug. Out of curiosity, does anything change if you set experimental_verify=1 in the job? I'll take a look at this next week, currently away on vacation. > Also, if I set a verify_interval larger than the smallest io size in > bssplt, I get another kind of error. > Error messages with bssplt=4k/50:1M/50: > > verify: bad magic header a678, wanted acca at file /tmp/test offset > 9097216, length 429524186 > verify: bad magic header 84b9, wanted acca at file /tmp/test offset > 25268224, length 21518097 > verify: bad magic header ba51, wanted acca at file /tmp/test offset > 26505216, length 536429778 > verify: bad magic header 1197, wanted acca at file /tmp/test offset > 42139648, length 373139796 > verify: bad magic header 22bf, wanted acca at file /tmp/test offset > 42209280, length 493427719 > meta: verify failed at file /tmp/test offset 50462720, length 65536 > open verify buf file: Read-only file system > open verify buf file: Read-only file system > verify: bad magic header d1f6, wanted acca at file /tmp/test offset > 52305920, length 322125536 > verify: bad magic header 76ee, wanted acca at file /tmp/test offset > 92106752, length 355054425 > verify: bad magic header 821, wanted acca at file /tmp/test offset > 94904320, length 460051914 > > I don't get these error when I set bs=4k. > Are you aware of such a limitation in fio? Fio doesn't support verify intervals larger than a single written block, so that is something the parser or option checker should catch. Apparently it doesn't. I'll take a look at this too. -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-18 7:47 ` Jens Axboe @ 2014-07-21 8:29 ` Jens Axboe 2014-07-21 17:41 ` Gwendal Grignou 2014-07-21 17:49 ` Gwendal Grignou 0 siblings, 2 replies; 9+ messages in thread From: Jens Axboe @ 2014-07-21 8:29 UTC (permalink / raw) To: Gwendal Grignou, Puthikorn Voravootivat, fio [-- Attachment #1: Type: text/plain, Size: 2517 bytes --] On 2014-07-18 09:47, Jens Axboe wrote: > On 2014-07-17 22:36, Gwendal Grignou wrote: >> Jens, >> >> I notice that when I enable bssplit and verify, verify always fails: >> >> ... >> meta: verify failed at file /tmp/test offset 7143424, length 65536 >> open verify buf file: Read-only file system >> .... >> In my tests, the offsets that fails are always: >> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456 >> >> it does not happen when I set a fixed size with bs, or when I do not >> verify. >> >> Here is the fio control file I use: >> >> [stress] >> filename=/tmp/test >> size=107374182 >> >> readwrite=randrw >> bssplit=64k/50:1M/50 >> ;bs=64k >> >> do_verify=1 >> verify=meta >> verify_interval=64k >> verify_dump=1 >> continue_on_error=verify > > That does sound like a bug. Out of curiosity, does anything change if > you set experimental_verify=1 in the job? > > I'll take a look at this next week, currently away on vacation. > >> Also, if I set a verify_interval larger than the smallest io size in >> bssplt, I get another kind of error. >> Error messages with bssplt=4k/50:1M/50: >> >> verify: bad magic header a678, wanted acca at file /tmp/test offset >> 9097216, length 429524186 >> verify: bad magic header 84b9, wanted acca at file /tmp/test offset >> 25268224, length 21518097 >> verify: bad magic header ba51, wanted acca at file /tmp/test offset >> 26505216, length 536429778 >> verify: bad magic header 1197, wanted acca at file /tmp/test offset >> 42139648, length 373139796 >> verify: bad magic header 22bf, wanted acca at file /tmp/test offset >> 42209280, length 493427719 >> meta: verify failed at file /tmp/test offset 50462720, length 65536 >> open verify buf file: Read-only file system >> open verify buf file: Read-only file system >> verify: bad magic header d1f6, wanted acca at file /tmp/test offset >> 52305920, length 322125536 >> verify: bad magic header 76ee, wanted acca at file /tmp/test offset >> 92106752, length 355054425 >> verify: bad magic header 821, wanted acca at file /tmp/test offset >> 94904320, length 460051914 >> >> I don't get these error when I set bs=4k. >> Are you aware of such a limitation in fio? > > Fio doesn't support verify intervals larger than a single written block, > so that is something the parser or option checker should catch. > Apparently it doesn't. I'll take a look at this too. Looks like it's numberio biting us again... Can you try with this patch? For the mixed case, haven't looked into interval > bs yet. -- Jens Axboe [-- Attachment #2: verify-mixed-numberio.patch --] [-- Type: text/x-patch, Size: 1121 bytes --] diff --git a/verify.c b/verify.c index e59a4b290510..7c99e15e73be 100644 --- a/verify.c +++ b/verify.c @@ -405,13 +405,13 @@ static int verify_io_u_meta(struct verify_header *hdr, struct vcont *vc) /* * For read-only workloads, the program cannot be certain of the - * last numberio written to a block. Checking of numberio will be done - * only for workloads that write data. - * For verify_only, numberio will be checked in the last iteration when - * the correct state of numberio, that would have been written to each - * block in a previous run of fio, has been reached. + * last numberio written to a block. Checking of numberio will be + * done only for workloads that write data. For verify_only, + * numberio will be checked in the last iteration when the correct + * state of numberio, that would have been written to each block + * in a previous run of fio, has been reached. */ - if (td_write(td) || td_rw(td)) + if ((td_write(td) || td_rw(td)) && (td_min_bs(td) == td_max_bs(td))) if (!td->o.verify_only || td->o.loops == 0) if (vh->numberio != io_u->numberio) ret = EILSEQ; ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 8:29 ` Jens Axboe @ 2014-07-21 17:41 ` Gwendal Grignou 2014-07-21 17:49 ` Gwendal Grignou 1 sibling, 0 replies; 9+ messages in thread From: Gwendal Grignou @ 2014-07-21 17:41 UTC (permalink / raw) To: Jens Axboe; +Cc: Gwendal Grignou, Puthikorn Voravootivat, fio [-- Attachment #1: Type: text/plain, Size: 3195 bytes --] Jens, Without the patch, experimental_verify=1 produced more error, it looks like every 64k. With the patch above, fio behaves as expected: do_verify and experimental_verify works for mixed writes. There is still a problem when bs is mixed and one argument is less than the verify interval. For instance: bssplit=4k/50:1M/50 ... verify_interval=64k But looking at the code, it won't be possible to fix, because we can not guarantee to always have 16 4K write contiguous. Thanks for the patch. Gwendal. On Mon, Jul 21, 2014 at 1:29 AM, Jens Axboe <axboe@kernel.dk> wrote: > On 2014-07-18 09:47, Jens Axboe wrote: > >> On 2014-07-17 22:36, Gwendal Grignou wrote: >> >>> Jens, >>> >>> I notice that when I enable bssplit and verify, verify always fails: >>> >>> ... >>> meta: verify failed at file /tmp/test offset 7143424, length 65536 >>> open verify buf file: Read-only file system >>> .... >>> In my tests, the offsets that fails are always: >>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456 >>> >>> it does not happen when I set a fixed size with bs, or when I do not >>> verify. >>> >>> Here is the fio control file I use: >>> >>> [stress] >>> filename=/tmp/test >>> size=107374182 >>> >>> readwrite=randrw >>> bssplit=64k/50:1M/50 >>> ;bs=64k >>> >>> do_verify=1 >>> verify=meta >>> verify_interval=64k >>> verify_dump=1 >>> continue_on_error=verify >>> >> >> That does sound like a bug. Out of curiosity, does anything change if >> you set experimental_verify=1 in the job? >> >> I'll take a look at this next week, currently away on vacation. >> >> Also, if I set a verify_interval larger than the smallest io size in >>> bssplt, I get another kind of error. >>> Error messages with bssplt=4k/50:1M/50: >>> >>> verify: bad magic header a678, wanted acca at file /tmp/test offset >>> 9097216, length 429524186 >>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset >>> 25268224, length 21518097 >>> verify: bad magic header ba51, wanted acca at file /tmp/test offset >>> 26505216, length 536429778 >>> verify: bad magic header 1197, wanted acca at file /tmp/test offset >>> 42139648, length 373139796 >>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset >>> 42209280, length 493427719 >>> meta: verify failed at file /tmp/test offset 50462720, length 65536 >>> open verify buf file: Read-only file system >>> open verify buf file: Read-only file system >>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset >>> 52305920, length 322125536 >>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset >>> 92106752, length 355054425 >>> verify: bad magic header 821, wanted acca at file /tmp/test offset >>> 94904320, length 460051914 >>> >>> I don't get these error when I set bs=4k. >>> Are you aware of such a limitation in fio? >>> >> >> Fio doesn't support verify intervals larger than a single written block, >> so that is something the parser or option checker should catch. >> Apparently it doesn't. I'll take a look at this too. >> > > Looks like it's numberio biting us again... Can you try with this patch? > For the mixed case, haven't looked into interval > bs yet. > > -- > Jens Axboe > > [-- Attachment #2: Type: text/html, Size: 4954 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 8:29 ` Jens Axboe 2014-07-21 17:41 ` Gwendal Grignou @ 2014-07-21 17:49 ` Gwendal Grignou 2014-07-21 18:40 ` Jens Axboe 1 sibling, 1 reply; 9+ messages in thread From: Gwendal Grignou @ 2014-07-21 17:49 UTC (permalink / raw) To: Jens Axboe; +Cc: Gwendal Grignou, Puthikorn Voravootivat, fio Jens, Without the patch, experimental_verify=1 produced more error, it looks like every 64k. With the patch above, fio behaves as expected: do_verify and experimental_verify works for mixed writes. There is still a problem when bs is mixed and one argument is less than the verify interval. For instance: bssplit=4k/50:1M/50 ... verify_interval=64k But looking at the code, it won't be possible to fix, because we can not guarantee to always have 16 4K write contiguous. Thanks for the patch. Gwendal. On Mon, Jul 21, 2014 at 1:29 AM, Jens Axboe <axboe@kernel.dk> wrote: > On 2014-07-18 09:47, Jens Axboe wrote: >> >> On 2014-07-17 22:36, Gwendal Grignou wrote: >>> >>> Jens, >>> >>> I notice that when I enable bssplit and verify, verify always fails: >>> >>> ... >>> meta: verify failed at file /tmp/test offset 7143424, length 65536 >>> open verify buf file: Read-only file system >>> .... >>> In my tests, the offsets that fails are always: >>> 7143424, 15073280, 2647654, 450397184, 88080384, 88211456 >>> >>> it does not happen when I set a fixed size with bs, or when I do not >>> verify. >>> >>> Here is the fio control file I use: >>> >>> [stress] >>> filename=/tmp/test >>> size=107374182 >>> >>> readwrite=randrw >>> bssplit=64k/50:1M/50 >>> ;bs=64k >>> >>> do_verify=1 >>> verify=meta >>> verify_interval=64k >>> verify_dump=1 >>> continue_on_error=verify >> >> >> That does sound like a bug. Out of curiosity, does anything change if >> you set experimental_verify=1 in the job? >> >> I'll take a look at this next week, currently away on vacation. >> >>> Also, if I set a verify_interval larger than the smallest io size in >>> bssplt, I get another kind of error. >>> Error messages with bssplt=4k/50:1M/50: >>> >>> verify: bad magic header a678, wanted acca at file /tmp/test offset >>> 9097216, length 429524186 >>> verify: bad magic header 84b9, wanted acca at file /tmp/test offset >>> 25268224, length 21518097 >>> verify: bad magic header ba51, wanted acca at file /tmp/test offset >>> 26505216, length 536429778 >>> verify: bad magic header 1197, wanted acca at file /tmp/test offset >>> 42139648, length 373139796 >>> verify: bad magic header 22bf, wanted acca at file /tmp/test offset >>> 42209280, length 493427719 >>> meta: verify failed at file /tmp/test offset 50462720, length 65536 >>> open verify buf file: Read-only file system >>> open verify buf file: Read-only file system >>> verify: bad magic header d1f6, wanted acca at file /tmp/test offset >>> 52305920, length 322125536 >>> verify: bad magic header 76ee, wanted acca at file /tmp/test offset >>> 92106752, length 355054425 >>> verify: bad magic header 821, wanted acca at file /tmp/test offset >>> 94904320, length 460051914 >>> >>> I don't get these error when I set bs=4k. >>> Are you aware of such a limitation in fio? >> >> >> Fio doesn't support verify intervals larger than a single written block, >> so that is something the parser or option checker should catch. >> Apparently it doesn't. I'll take a look at this too. > > > Looks like it's numberio biting us again... Can you try with this patch? For > the mixed case, haven't looked into interval > bs yet. > > -- > Jens Axboe > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 17:49 ` Gwendal Grignou @ 2014-07-21 18:40 ` Jens Axboe 2014-07-21 18:52 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Jens Axboe @ 2014-07-21 18:40 UTC (permalink / raw) To: Gwendal Grignou; +Cc: Puthikorn Voravootivat, fio On 2014-07-21 19:49, Gwendal Grignou wrote: > Jens, > > Without the patch, experimental_verify=1 produced more error, it looks > like every 64k. > > With the patch above, fio behaves as expected: do_verify and > experimental_verify works for mixed writes. > > There is still a problem when bs is mixed and one argument is less > than the verify interval. > For instance: > bssplit=4k/50:1M/50 > ... > verify_interval=64k > > > But looking at the code, it won't be possible to fix, because we can > not guarantee to always have 16 4K write contiguous. As mentioned in the original email, it's a parser fix. verify_interval cannot be less than the minimum block size, fio will just have to catch that and adjust / error out. > Thanks for the patch. It's committed, so generally available now. -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 18:40 ` Jens Axboe @ 2014-07-21 18:52 ` Jens Axboe 2014-07-21 20:24 ` Gwendal Grignou 0 siblings, 1 reply; 9+ messages in thread From: Jens Axboe @ 2014-07-21 18:52 UTC (permalink / raw) To: Gwendal Grignou; +Cc: Puthikorn Voravootivat, fio [-- Attachment #1: Type: text/plain, Size: 402 bytes --] On 2014-07-21 20:40, Jens Axboe wrote: >> But looking at the code, it won't be possible to fix, because we can >> not guarantee to always have 16 4K write contiguous. > > As mentioned in the original email, it's a parser fix. verify_interval > cannot be less than the minimum block size, fio will just have to catch > that and adjust / error out. Attach should fix that - untested... -- Jens Axboe [-- Attachment #2: x.patch --] [-- Type: text/x-patch, Size: 425 bytes --] diff --git a/init.c b/init.c index cf1d7f938fb3..cf2f82220f72 100644 --- a/init.c +++ b/init.c @@ -635,6 +635,9 @@ static int fixup_options(struct thread_data *td) if (o->max_bs[DDIR_WRITE] != o->min_bs[DDIR_WRITE] && !o->verify_interval) o->verify_interval = o->min_bs[DDIR_WRITE]; + + if (o->verify_interval > o->min_bs[DDIR_WRITE]) + o->verify_interval = o->min_bs[DDIR_WRITE]; } if (o->pre_read) { ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 18:52 ` Jens Axboe @ 2014-07-21 20:24 ` Gwendal Grignou 2014-07-21 20:30 ` Jens Axboe 0 siblings, 1 reply; 9+ messages in thread From: Gwendal Grignou @ 2014-07-21 20:24 UTC (permalink / raw) To: Jens Axboe; +Cc: Gwendal Grignou, Puthikorn Voravootivat, fio Jens, I still see some verify errors when I set 'experimental_verify=1' verify: bad magic header 977c, wanted acca at file /tmp/test offset 15478784, length 429437046 verify: bad magic header 8aac, wanted acca at file /tmp/test offset 28082176, length 302426815 verify: bad magic header c8ad, wanted acca at file /tmp/test offset 30408704, length 318473336 verify: bad magic header 3791, wanted acca at file /tmp/test offset 65024000, length 74750452 However, without this line, there are no verify errors anymore. The fio file is: [stress] filename=/tmp/test size=107374182 readwrite=randrw bssplit=4k/50:1M/50 ;bs=64k do_verify=1 experimental_verify=1 verify=meta verify_interval=64k verify_dump=1 continue_on_error=verify Thanks a lot, Gwendal. On Mon, Jul 21, 2014 at 11:52 AM, Jens Axboe <axboe@kernel.dk> wrote: > On 2014-07-21 20:40, Jens Axboe wrote: >>> >>> But looking at the code, it won't be possible to fix, because we can >>> not guarantee to always have 16 4K write contiguous. >> >> >> As mentioned in the original email, it's a parser fix. verify_interval >> cannot be less than the minimum block size, fio will just have to catch >> that and adjust / error out. > > > Attach should fix that - untested... > > -- > Jens Axboe > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [fio-2.1.9] verify and bssplit do not work together 2014-07-21 20:24 ` Gwendal Grignou @ 2014-07-21 20:30 ` Jens Axboe 0 siblings, 0 replies; 9+ messages in thread From: Jens Axboe @ 2014-07-21 20:30 UTC (permalink / raw) To: Gwendal Grignou; +Cc: Puthikorn Voravootivat, fio On 2014-07-21 22:24, Gwendal Grignou wrote: > Jens, > I still see some verify errors when I set 'experimental_verify=1' > > verify: bad magic header 977c, wanted acca at file /tmp/test offset > 15478784, length 429437046 > verify: bad magic header 8aac, wanted acca at file /tmp/test offset > 28082176, length 302426815 > verify: bad magic header c8ad, wanted acca at file /tmp/test offset > 30408704, length 318473336 > verify: bad magic header 3791, wanted acca at file /tmp/test offset > 65024000, length 74750452 > > However, without this line, there are no verify errors anymore. > > The fio file is: > [stress] > filename=/tmp/test > size=107374182 > > readwrite=randrw > bssplit=4k/50:1M/50 > ;bs=64k > > do_verify=1 > experimental_verify=1 > verify=meta > verify_interval=64k > verify_dump=1 > continue_on_error=verify I'll look into the experimental_verify bit - there's a reason it's experimental :-). But the hope is to make that the method for verification, so I'll look into what goes wrong there. It might be highlighting problems elsewhere (like accidental overwrites, due to the mixed block sizes). -- Jens Axboe ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-07-21 20:30 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-07-17 20:36 [fio-2.1.9] verify and bssplit do not work together Gwendal Grignou 2014-07-18 7:47 ` Jens Axboe 2014-07-21 8:29 ` Jens Axboe 2014-07-21 17:41 ` Gwendal Grignou 2014-07-21 17:49 ` Gwendal Grignou 2014-07-21 18:40 ` Jens Axboe 2014-07-21 18:52 ` Jens Axboe 2014-07-21 20:24 ` Gwendal Grignou 2014-07-21 20:30 ` Jens Axboe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox