Pending fio 3.0 release

Pending fio 3.0 release

[Jens Axboe]

Hi,

CC'ing a few key people here - I want to release fio 3.0 shortly, which
is why the last released jumped to 2.99. However, before doing so, I'd
like to ensure that we are uptodate on platforms. Rebecca, is Windows
currently building fine with current -git? I suspect it is, but would be
nice to know for sure.

Any other important pending fixes that should go in before 3.0 is
released?

-- 
Jens Axboe

Re: Pending fio 3.0 release

[Sitsofe Wheeler]

Hi Jens,

On 7 August 2017 at 17:10, Jens Axboe <axboe@kernel.dk> wrote:
>
> CC'ing a few key people here - I want to release fio 3.0 shortly, which
> is why the last released jumped to 2.99. However, before doing so, I'd
> like to ensure that we are uptodate on platforms. Rebecca, is Windows
> currently building fine with current -git? I suspect it is, but would be
> nice to know for sure.
>
> Any other important pending fixes that should go in before 3.0 is
> released?

There are a few pull requests queued up over on
https://github.com/axboe/fio/pulls (json+ patches, inflight overlap
use after free fix, C++ ioengine changes, serialise overlap changes,
sheepdog ioengine, change some output to debugging only etc.).

Locally, I have an experimental libiscsi ioengine, support for
verify_state when verifying a readwrite workload, some HOWTO
reordering to improve the man page conversion and some initial
investigations into invalidation on Windows. The only one I'd have
loved to see go in is the libiscsi engine but it's kinda raw and needs
fixing up and I don't know when I'll have time to finish it...

With regard to Windows, it seems to at least be building
(https://ci.appveyor.com/project/axboe/fio/build/1.0.245 ) and running
basic workloads for me...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Pending fio 3.0 release

[Jens Axboe]

On 08/07/2017 01:05 PM, Sitsofe Wheeler wrote:
> Hi Jens,
> 
> On 7 August 2017 at 17:10, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> CC'ing a few key people here - I want to release fio 3.0 shortly, which
>> is why the last released jumped to 2.99. However, before doing so, I'd
>> like to ensure that we are uptodate on platforms. Rebecca, is Windows
>> currently building fine with current -git? I suspect it is, but would be
>> nice to know for sure.
>>
>> Any other important pending fixes that should go in before 3.0 is
>> released?
> 
> There are a few pull requests queued up over on
> https://github.com/axboe/fio/pulls (json+ patches, inflight overlap
> use after free fix, C++ ioengine changes, serialise overlap changes,
> sheepdog ioengine, change some output to debugging only etc.).

Just pulled that in, it's fine now that the more controversial bits
are out.

> Locally, I have an experimental libiscsi ioengine, support for
> verify_state when verifying a readwrite workload, some HOWTO
> reordering to improve the man page conversion and some initial
> investigations into invalidation on Windows. The only one I'd have
> loved to see go in is the libiscsi engine but it's kinda raw and needs
> fixing up and I don't know when I'll have time to finish it...

For new engines, I don't mind merging them early, as long as the people
behind them are motivated to bring them up to snuff. Up to you.

verify_state support for read/write sounds like something that would be
nice to have, but isn't a must for 3.0.

Anyway, I'll likely cut 3.0 next week, so there's still time for
whatever changes we need. But they have to be low risk at this point,
I'd hate to do a 3.0 that broke anything recent.

> With regard to Windows, it seems to at least be building
> (https://ci.appveyor.com/project/axboe/fio/build/1.0.245 ) and running
> basic workloads for me...

Ah yes, thanks!

-- 
Jens Axboe

RE: Pending fio 3.0 release

[Jeff Furlong]

Are the 3 serialize changes (pull requests  343/345/359) controversial or not complete?  I hit the same issue in April with a similar workload:

# fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1 --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress --verify=crc32c-intel --verify_fatal=1 --verify_dump=1 --verify_backlog=32768

I tried again now with fio 2.99 and reproduced after a few minutes:

verify: bad header numberio 4531, wanted 4532 at file /dev/nvme1n1 offset 2643506233344, length 2097152
       hdr_fail data dumped as nvme1n1.2643506233344.hdr_fail

So any large block random write seems to be at risk (larger the block, higher the risk).  If there are other parameters I should be setting to avoid the issue, please let me know.  Thanks.

Regards,
Jeff

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jens Axboe
Sent: Monday, August 7, 2017 12:47 PM
To: Sitsofe Wheeler <sitsofe@gmail.com>
Cc: fio@vger.kernel.org; Rebecca Cran <rebecca@bluestop.org>; Tomohiro Kusumi <tkusumi@tuxera.com>
Subject: Re: Pending fio 3.0 release

On 08/07/2017 01:05 PM, Sitsofe Wheeler wrote:
> Hi Jens,
> 
> On 7 August 2017 at 17:10, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> CC'ing a few key people here - I want to release fio 3.0 shortly, 
>> which is why the last released jumped to 2.99. However, before doing 
>> so, I'd like to ensure that we are uptodate on platforms. Rebecca, is 
>> Windows currently building fine with current -git? I suspect it is, 
>> but would be nice to know for sure.
>>
>> Any other important pending fixes that should go in before 3.0 is 
>> released?
> 
> There are a few pull requests queued up over on 
> https://github.com/axboe/fio/pulls (json+ patches, inflight overlap 
> use after free fix, C++ ioengine changes, serialise overlap changes, 
> sheepdog ioengine, change some output to debugging only etc.).

Just pulled that in, it's fine now that the more controversial bits are out.

> Locally, I have an experimental libiscsi ioengine, support for 
> verify_state when verifying a readwrite workload, some HOWTO 
> reordering to improve the man page conversion and some initial 
> investigations into invalidation on Windows. The only one I'd have 
> loved to see go in is the libiscsi engine but it's kinda raw and needs 
> fixing up and I don't know when I'll have time to finish it...

For new engines, I don't mind merging them early, as long as the people behind them are motivated to bring them up to snuff. Up to you.

verify_state support for read/write sounds like something that would be nice to have, but isn't a must for 3.0.

Anyway, I'll likely cut 3.0 next week, so there's still time for whatever changes we need. But they have to be low risk at this point, I'd hate to do a 3.0 that broke anything recent.

> With regard to Windows, it seems to at least be building
> (https://ci.appveyor.com/project/axboe/fio/build/1.0.245 ) and running 
> basic workloads for me...

Ah yes, thanks!

--
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Pending fio 3.0 release

[Sitsofe Wheeler]

On 7 August 2017 at 23:02, Jeff Furlong <jeff.furlong@wdc.com> wrote:
> Are the 3 serialize changes (pull requests  343/345/359) controversial or not complete?  I hit the same issue in April with a similar workload:

A fair question Jeff.

If memory serves https://github.com/axboe/fio/pull/343 ("Add
serialize_overlap option") is controversial because it took a heavy
handed with potentially high overhead due to checking every I/O
against every other inflight I/O to determine if it should flush to
avoid generating two in-flight I/Os that cover overlapping regions. To
use it you had to set an explicit option (which defaulted to off) and
did the job though...

https://github.com/axboe/fio/pull/345 is hopefully uncontroversial as
it just removes an optimisation that could turn off overlap checking
when it really needed for jobs that can write the same region twice
and would otherwise lead to spurious mismatches at verification time.

https://github.com/axboe/fio/pull/359 narrowed the window that was
creating a double free situation when two overlapping inflight writes
occurred. It's imperfect but it made the problem I was seeing go away
even if I can theorise it won't prevent the problem in every case.

>
> # fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1 --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress --verify=crc32c-intel --verify_fatal=1 --verify_dump=1 --verify_backlog=32768
>
> I tried again now with fio 2.99 and reproduced after a few minutes:
>
> verify: bad header numberio 4531, wanted 4532 at file /dev/nvme1n1 offset 2643506233344, length 2097152
>        hdr_fail data dumped as nvme1n1.2643506233344.hdr_fail
>
> So any large block random write seems to be at risk (larger the block, higher the risk).  If there are other parameters I should be setting to avoid the issue, please let me know.  Thanks.

If you set --overwrite=1 does the problem you're seeing go away? If so
it sounds like you're hitting https://github.com/axboe/fio/issues/335
...

-- 
Sitsofe | http://sucs.org/~sits/

RE: Pending fio 3.0 release

[Jeff Furlong]

OK, sounds like 345 and 359 are the ones desired.

I did have overwrite=1 in my original test, but tried to remove as many non-essential parameters as possible.  But I just added it back:

# fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1 --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress --verify=crc32c-intel --verify_fatal=1 --verify_dump=1 --verify_backlog=32768 --overwrite=1
crc32c: verify failed at file /dev/nvme1n1 offset 2013628727296, length 2097152]      
       Expected CRC: a6dbc3dc
       Received CRC: dfa3362b
       received data dumped as nvme1n1.2013628727296.received
       expected data dumped as nvme1n1.2013628727296.expected

Regards,
Jeff

-----Original Message-----
From: Sitsofe Wheeler [mailto:sitsofe@gmail.com] 
Sent: Monday, August 7, 2017 3:25 PM
To: Jeff Furlong <jeff.furlong@wdc.com>
Cc: Jens Axboe <axboe@kernel.dk>; fio@vger.kernel.org; Rebecca Cran <rebecca@bluestop.org>; Tomohiro Kusumi <tkusumi@tuxera.com>
Subject: Re: Pending fio 3.0 release

On 7 August 2017 at 23:02, Jeff Furlong <jeff.furlong@wdc.com> wrote:
> Are the 3 serialize changes (pull requests  343/345/359) controversial or not complete?  I hit the same issue in April with a similar workload:

A fair question Jeff.

If memory serves https://github.com/axboe/fio/pull/343 ("Add serialize_overlap option") is controversial because it took a heavy handed with potentially high overhead due to checking every I/O against every other inflight I/O to determine if it should flush to avoid generating two in-flight I/Os that cover overlapping regions. To use it you had to set an explicit option (which defaulted to off) and did the job though...

https://github.com/axboe/fio/pull/345 is hopefully uncontroversial as it just removes an optimisation that could turn off overlap checking when it really needed for jobs that can write the same region twice and would otherwise lead to spurious mismatches at verification time.

https://github.com/axboe/fio/pull/359 narrowed the window that was creating a double free situation when two overlapping inflight writes occurred. It's imperfect but it made the problem I was seeing go away even if I can theorise it won't prevent the problem in every case.

>
> # fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite 
> --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1 
> --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress 
> --verify=crc32c-intel --verify_fatal=1 --verify_dump=1 
> --verify_backlog=32768
>
> I tried again now with fio 2.99 and reproduced after a few minutes:
>
> verify: bad header numberio 4531, wanted 4532 at file /dev/nvme1n1 offset 2643506233344, length 2097152
>        hdr_fail data dumped as nvme1n1.2643506233344.hdr_fail
>
> So any large block random write seems to be at risk (larger the block, higher the risk).  If there are other parameters I should be setting to avoid the issue, please let me know.  Thanks.

If you set --overwrite=1 does the problem you're seeing go away? If so it sounds like you're hitting https://github.com/axboe/fio/issues/335
...

--
Sitsofe | http://sucs.org/~sits/

Re: Pending fio 3.0 release

[Sitsofe Wheeler]

Hmm you might be in the case where you managed to arrange for two in
flight writes to go down to the exact same region so you no longer know
what the data at that point is. If so, you would also need something
like the serialize_overlap patch (343) and you would have to set that
option to 1. Alternatively you could restrict the iodepth to 1 (no
patches required but it has an obvious drawback).

On 8 August 2017 at 00:09, Jeff Furlong <jeff.furlong@wdc.com> wrote:
> OK, sounds like 345 and 359 are the ones desired.
>
> I did have overwrite=1 in my original test, but tried to remove as many non-essential parameters as possible.  But I just added it back:
>
> # fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1 --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress --verify=crc32c-intel --verify_fatal=1 --verify_dump=1 --verify_backlog=32768 --overwrite=1
> crc32c: verify failed at file /dev/nvme1n1 offset 2013628727296, length 2097152]
>        Expected CRC: a6dbc3dc
>        Received CRC: dfa3362b
>        received data dumped as nvme1n1.2013628727296.received
>        expected data dumped as nvme1n1.2013628727296.expected
>
> Regards,
> Jeff
>
> -----Original Message-----
> From: Sitsofe Wheeler [mailto:sitsofe@gmail.com]
> Sent: Monday, August 7, 2017 3:25 PM
> To: Jeff Furlong <jeff.furlong@wdc.com>
> Cc: Jens Axboe <axboe@kernel.dk>; fio@vger.kernel.org; Rebecca Cran <rebecca@bluestop.org>; Tomohiro Kusumi <tkusumi@tuxera.com>
> Subject: Re: Pending fio 3.0 release
>
> On 7 August 2017 at 23:02, Jeff Furlong <jeff.furlong@wdc.com> wrote:
>> Are the 3 serialize changes (pull requests  343/345/359) controversial or not complete?  I hit the same issue in April with a similar workload:
>
> A fair question Jeff.
>
> If memory serves https://github.com/axboe/fio/pull/343 ("Add serialize_overlap option") is controversial because it took a heavy handed with potentially high overhead due to checking every I/O against every other inflight I/O to determine if it should flush to avoid generating two in-flight I/Os that cover overlapping regions. To use it you had to set an explicit option (which defaulted to off) and did the job though...
>
> https://github.com/axboe/fio/pull/345 is hopefully uncontroversial as it just removes an optimisation that could turn off overlap checking when it really needed for jobs that can write the same region twice and would otherwise lead to spurious mismatches at verification time.
>
> https://github.com/axboe/fio/pull/359 narrowed the window that was creating a double free situation when two overlapping inflight writes occurred. It's imperfect but it made the problem I was seeing go away even if I can theorise it won't prevent the problem in every case.
>
>>
>> # fio --name=DI_Stress --ioengine=libaio --direct=1 --rw=randwrite
>> --norandommap --randrepeat=0 --iodepth=16 --size=100% --numjobs=1
>> --bs=2048k --filename=/dev/nvme1n1 --output=DI_Stress
>> --verify=crc32c-intel --verify_fatal=1 --verify_dump=1
>> --verify_backlog=32768
>>
>> I tried again now with fio 2.99 and reproduced after a few minutes:
>>
>> verify: bad header numberio 4531, wanted 4532 at file /dev/nvme1n1 offset 2643506233344, length 2097152
>>        hdr_fail data dumped as nvme1n1.2643506233344.hdr_fail
>>
>> So any large block random write seems to be at risk (larger the block, higher the risk).  If there are other parameters I should be setting to avoid the issue, please let me know.  Thanks.
>
> If you set --overwrite=1 does the problem you're seeing go away? If so it sounds like you're hitting https://github.com/axboe/fio/issues/335
> ...
>
> --
> Sitsofe | http://sucs.org/~sits/
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.



-- 
Sitsofe | http://sucs.org/~sits/

Re: Pending fio 3.0 release

[Rebecca Cran]

On 8/7/2017 10:10 AM, Jens Axboe wrote:

> CC'ing a few key people here - I want to release fio 3.0 shortly, which
> is why the last released jumped to 2.99. However, before doing so, I'd
> like to ensure that we are uptodate on platforms. Rebecca, is Windows
> currently building fine with current -git?

Yes, it's building and running fine.

-- 
Rebecca