regression since 2.1.3 (solaris/zfs)

regression since 2.1.3 (solaris/zfs)

[Robin P. Blanchard]

My config file has direct=0, which until 2.1.4 worked as expected. Things seem to regress since.

I apologize in advance if this has already been reported. Please let me know what I can do to further help (truss/debug).

fio-2.1.4
Starting 56 threads
fio: the file system does not seem to support direct IO
fio: pid=3, err=25/file:ioengines.c:457, func=fio_set_odirect, error=Inappropriate ioctl for device


fio-2.1.5
Starting 56 threads
fio: the file system does not seem to support direct IO
fio: pid=3, err=25/file:ioengines.c:457, func=fio_set_odirect, error=Inappropriate ioctl for device



debug=io

fio-2.1.5
Starting 56 threads
io       17923 invalidate cache fio.out: 0/137438953472
fio: the file system does not seem to support direct IO
io       17923 io_u 539470, setting file failed
io       17923 get_io_u failed
io       17923 io_u_queued_completed: min=0
io       17923 getevents: 0
fio: pid=3, err=25/file:ioengines.c:457, func=fio_set_odirect, error=Inappropriate ioctl for device
io       17923 close ioengine solarisaio
io       17923 free ioengine solarisaio
io       17923 invalidate cache fio.out: 0/137438953472

debug=file drops core



# uname -a
SunOS apone 5.11 11.1 i86pc i386 i86pc Solaris

# pkg info entire |grep FMRI
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.1.16.0.5.0:20140218T165248Z

# gcc --version
gcc (GCC) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# file ~/fio/fio-2.1.5/fio
/export/home/surlypants/fio/fio-2.1.5/fio:      ELF 64-bit LSB executable AMD64 Version 1, dynamically linked, not stripped

# ldd ~/fio/fio-2.1.5/fio
        librdmacm.so.1 =>        /usr/lib/64/librdmacm.so.1
        libibverbs.so.1 =>       /usr/lib/64/libibverbs.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libz.so.1 =>     /lib/64/libz.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libgcc_s.so.1 =>         /usr/sfw/lib/64/libgcc_s.so.1
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libsoftcrypto.so.1 =>    /lib/64/libsoftcrypto.so.1
        libelf.so.1 =>   /lib/64/libelf.so.1
        libcryptoutil.so.1 =>    /lib/64/libcryptoutil.so.1

# ~/fio/fio-2.1.5/fio --enghelp
Available IO engines:
        solarisaio
        posixaio
        rdma
        net
        null
        sync
        psync
        vsync
        mmap
        cpuio



-- 
Robin P. Blanchard
Solutions Engineer
Coraid Global Field Services
www.coraid.com
+1 650.730.5140

Re: regression since 2.1.3 (solaris/zfs)

[Jens Axboe]

On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
> My config file has direct=0, which until 2.1.4 worked as expected.
 > Things seem to regress since.
>
> I apologize in advance if this has already been reported. Please
 > let me know what I can do to further help (truss/debug).

This isn't a known issue, so thanks for reporting it. The easiest way to 
debug this is to git bisect it. Looks like you are running from the tar 
balls, but I assume you have git installed? I'm assuming fio-2.1.3 
worked for you - if not, just replace fio-2.1.3 in the below with 
whatever latest version did work. If you do, the cheat sheet is 
something ala:

$ git clone git://git.kernel.dk/fio
$ cd fio; make
$ git bisect start
$ git bisect good fio-2.1.3
$ git bisect bad fio-2.1.4

This starts the bisect series, now do:

$ make clean; make

and re-run your direct=0 job file. If it worked, then you do

$ git bisect good

and if not, you do git bisect bad instead. This gets you a new point in 
the tree to test, so repeat the make clean; make and re-run the test.
Keep doing this good/bad iteration until fio tells you what commit broke 
the test for you. Then send those results here!

-- 
Jens Axboe

Re: regression since 2.1.3 (solaris/zfs)

[Robin P. Blanchard]

On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>> My config file has direct=0, which until 2.1.4 worked as expected.
> > Things seem to regress since.
>> 
>> I apologize in advance if this has already been reported. Please
> > let me know what I can do to further help (truss/debug).
> 
> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
> 
> $ git clone git://git.kernel.dk/fio
> $ cd fio; make
> $ git bisect start
> $ git bisect good fio-2.1.3
> $ git bisect bad fio-2.1.4
> 
> This starts the bisect series, now do:
> 
> $ make clean; make
> 
> and re-run your direct=0 job file. If it worked, then you do
> 
> $ git bisect good
> 
> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
> 
> -- 
> Jens Axboe
> 


Here’s where it started working again:

# git bisect good
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries

Let me know how else I can help.



Thanks,
Robin


-- 
Robin P. Blanchard
Solutions Engineer
Coraid Global Field Services
www.coraid.com
+1 650.730.5140

Re: regression since 2.1.3 (solaris/zfs)

[Jens Axboe]

On 2014-03-05 09:11, Robin P. Blanchard wrote:
>
> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>
>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>> Things seem to regress since.
>>>
>>> I apologize in advance if this has already been reported. Please
>>> let me know what I can do to further help (truss/debug).
>>
>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>
>> $ git clone git://git.kernel.dk/fio
>> $ cd fio; make
>> $ git bisect start
>> $ git bisect good fio-2.1.3
>> $ git bisect bad fio-2.1.4
>>
>> This starts the bisect series, now do:
>>
>> $ make clean; make
>>
>> and re-run your direct=0 job file. If it worked, then you do
>>
>> $ git bisect good
>>
>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>
>> --
>> Jens Axboe
>>
>
>
> Here�s where it started working again:
>
> # git bisect good
> Bisecting: 4 revisions left to test after this (roughly 2 steps)
> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>
> Let me know how else I can help.

Please keep going until it tells you what the definitively bad commit 
is. It'll end up spitting out that info, if you keep doing git bisect 
good/bad on each test point. You need just ~2 more tests after this one.

-- 
Jens Axboe

Re: regression since 2.1.3 (solaris/zfs)

[Robin P. Blanchard]

On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>> 
>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> 
>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>> Things seem to regress since.
>>>> 
>>>> I apologize in advance if this has already been reported. Please
>>>> let me know what I can do to further help (truss/debug).
>>> 
>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>> 
>>> $ git clone git://git.kernel.dk/fio
>>> $ cd fio; make
>>> $ git bisect start
>>> $ git bisect good fio-2.1.3
>>> $ git bisect bad fio-2.1.4
>>> 
>>> This starts the bisect series, now do:
>>> 
>>> $ make clean; make
>>> 
>>> and re-run your direct=0 job file. If it worked, then you do
>>> 
>>> $ git bisect good
>>> 
>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>> 
>>> --
>>> Jens Axboe
>>> 
>> 
>> 
>> Here’s where it started working again:
>> 
>> # git bisect good
>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>> 
>> Let me know how else I can help.
> 
> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
> 
> -- 
> Jens Axboe
> 

Here you go:

# git bisect good
ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
Author: Jens Axboe <axboe@kernel.dk>
Date:   Fri Oct 11 10:27:28 2013 -0600

    ppc: disable CPU clock until we can detect whether we have it or not

    The child segfault test should catch it, however it does not on
    AIX at least.

    Signed-off-by: Jens Axboe <axboe@kernel.dk>

:040000 040000 9dad9c1028514520214868c22739fcdd95413d88 572c353344ed6ff7c254fcb00b4ac9e288236109 M      arch




-- 
Robin P. Blanchard
Solutions Engineer
Coraid Global Field Services
www.coraid.com
+1 650.730.5140

Re: regression since 2.1.3 (solaris/zfs)

[Jens Axboe]

On Mar 5, 2014, at 9:27 AM, Robin P. Blanchard <robin@coraid.com> wrote:

> 
> On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>>> 
>>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>> 
>>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>>> Things seem to regress since.
>>>>> 
>>>>> I apologize in advance if this has already been reported. Please
>>>>> let me know what I can do to further help (truss/debug).
>>>> 
>>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>>> 
>>>> $ git clone git://git.kernel.dk/fio
>>>> $ cd fio; make
>>>> $ git bisect start
>>>> $ git bisect good fio-2.1.3
>>>> $ git bisect bad fio-2.1.4
>>>> 
>>>> This starts the bisect series, now do:
>>>> 
>>>> $ make clean; make
>>>> 
>>>> and re-run your direct=0 job file. If it worked, then you do
>>>> 
>>>> $ git bisect good
>>>> 
>>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>>> 
>>>> --
>>>> Jens Axboe
>>>> 
>>> 
>>> 
>>> Here’s where it started working again:
>>> 
>>> # git bisect good
>>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>>> 
>>> Let me know how else I can help.
>> 
>> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
>> 
>> -- 
>> Jens Axboe
>> 
> 
> Here you go:
> 
> # git bisect good
> ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
> commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Fri Oct 11 10:27:28 2013 -0600
> 
>    ppc: disable CPU clock until we can detect whether we have it or not
> 
>    The child segfault test should catch it, however it does not on
>    AIX at least.
> 
>    Signed-off-by: Jens Axboe <axboe@kernel.dk>

Hmm, that can’t possibly be correct. Would you mind redoing the bisection,
just to double check?

Thanks!

— 
Jens Axboe

Re: regression since 2.1.3 (solaris/zfs)

[Robin P. Blanchard]

On Mar 5, 2014, at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:

> On Mar 5, 2014, at 9:27 AM, Robin P. Blanchard <robin@coraid.com> wrote:
> 
>> 
>> On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> 
>>> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>>>> 
>>>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> 
>>>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>>>> Things seem to regress since.
>>>>>> 
>>>>>> I apologize in advance if this has already been reported. Please
>>>>>> let me know what I can do to further help (truss/debug).
>>>>> 
>>>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>>>> 
>>>>> $ git clone git://git.kernel.dk/fio
>>>>> $ cd fio; make
>>>>> $ git bisect start
>>>>> $ git bisect good fio-2.1.3
>>>>> $ git bisect bad fio-2.1.4
>>>>> 
>>>>> This starts the bisect series, now do:
>>>>> 
>>>>> $ make clean; make
>>>>> 
>>>>> and re-run your direct=0 job file. If it worked, then you do
>>>>> 
>>>>> $ git bisect good
>>>>> 
>>>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>>>> 
>>>>> --
>>>>> Jens Axboe
>>>>> 
>>>> 
>>>> 
>>>> Here’s where it started working again:
>>>> 
>>>> # git bisect good
>>>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>>>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>>>> 
>>>> Let me know how else I can help.
>>> 
>>> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
>>> 
>>> -- 
>>> Jens Axboe
>>> 
>> 
>> Here you go:
>> 
>> # git bisect good
>> ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
>> commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Fri Oct 11 10:27:28 2013 -0600
>> 
>>   ppc: disable CPU clock until we can detect whether we have it or not
>> 
>>   The child segfault test should catch it, however it does not on
>>   AIX at least.
>> 
>>   Signed-off-by: Jens Axboe <axboe@kernel.dk>
> 
> Hmm, that can’t possibly be correct. Would you mind redoing the bisection,
> just to double check?
> 
> Thanks!
> 
> — 
> Jens Axboe

Ok. Let’s try this again. Silly user, me.

# git bisect good
7cb024f89dbbc314e740885afccd9a05da056cf1 is the first bad commit
commit 7cb024f89dbbc314e740885afccd9a05da056cf1
Author: Jens Axboe <axboe@kernel.dk>
Date:   Wed Nov 6 15:37:35 2013 -0700

    solaris: ensure that -D_REENTRANT gets set

    Apparently some Solaris' require this for threadsafe
    errno.

    Signed-off-by: Jens Axboe <axboe@kernel.dk>

:100755 100755 b6bfe19aa743fc4104eb587b3ff6068fb5dc67ef ef7be0180258abecd4703ebfcf4ed63625d6392f M      configure


I have the complete git/bisect session in a screen log if you’d like it.

Thanks very much,
Robin


-- 
Robin P. Blanchard
Solutions Engineer
Coraid Global Field Services
www.coraid.com
+1 650.730.5140

Re: regression since 2.1.3 (solaris/zfs)

[Jens Axboe]

On 03/05/2014 01:39 PM, Robin P. Blanchard wrote:
>
> On Mar 5, 2014, at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:
>
>> On Mar 5, 2014, at 9:27 AM, Robin P. Blanchard <robin@coraid.com> wrote:
>>
>>>
>>> On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>>> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>>>>>
>>>>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>>>>> Things seem to regress since.
>>>>>>>
>>>>>>> I apologize in advance if this has already been reported. Please
>>>>>>> let me know what I can do to further help (truss/debug).
>>>>>>
>>>>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>>>>>
>>>>>> $ git clone git://git.kernel.dk/fio
>>>>>> $ cd fio; make
>>>>>> $ git bisect start
>>>>>> $ git bisect good fio-2.1.3
>>>>>> $ git bisect bad fio-2.1.4
>>>>>>
>>>>>> This starts the bisect series, now do:
>>>>>>
>>>>>> $ make clean; make
>>>>>>
>>>>>> and re-run your direct=0 job file. If it worked, then you do
>>>>>>
>>>>>> $ git bisect good
>>>>>>
>>>>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>>>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>>>>>
>>>>>> --
>>>>>> Jens Axboe
>>>>>>
>>>>>
>>>>>
>>>>> Here�s where it started working again:
>>>>>
>>>>> # git bisect good
>>>>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>>>>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>>>>>
>>>>> Let me know how else I can help.
>>>>
>>>> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>>
>>> Here you go:
>>>
>>> # git bisect good
>>> ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
>>> commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Fri Oct 11 10:27:28 2013 -0600
>>>
>>>    ppc: disable CPU clock until we can detect whether we have it or not
>>>
>>>    The child segfault test should catch it, however it does not on
>>>    AIX at least.
>>>
>>>    Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>
>> Hmm, that can�t possibly be correct. Would you mind redoing the bisection,
>> just to double check?
>>
>> Thanks!
>>
>> �
>> Jens Axboe
>
> Ok. Let�s try this again. Silly user, me.
>
> # git bisect good
> 7cb024f89dbbc314e740885afccd9a05da056cf1 is the first bad commit
> commit 7cb024f89dbbc314e740885afccd9a05da056cf1
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Wed Nov 6 15:37:35 2013 -0700
>
>      solaris: ensure that -D_REENTRANT gets set
>
>      Apparently some Solaris' require this for threadsafe
>      errno.
>
>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>
> :100755 100755 b6bfe19aa743fc4104eb587b3ff6068fb5dc67ef ef7be0180258abecd4703ebfcf4ed63625d6392f M      configure
>
>
> I have the complete git/bisect session in a screen log if you�d like it.

That makes a lot more sense! Can you do:

$ git checkout -f master
$ git revert 7cb024f89dbbc314e740885afccd9a05da056cf1
$ make clean; make

and retest just to be on the safe side? Also, please attach the job file 
you are using.

-- 
Jens Axboe

Re: regression since 2.1.3 (solaris/zfs)

[Robin P. Blanchard]

On Mar 5, 2014, at 16:05 PM, Jens Axboe <axboe@kernel.dk> wrote:

> On 03/05/2014 01:39 PM, Robin P. Blanchard wrote:
>> 
>> On Mar 5, 2014, at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> 
>>> On Mar 5, 2014, at 9:27 AM, Robin P. Blanchard <robin@coraid.com> wrote:
>>> 
>>>> 
>>>> On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> 
>>>>> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>>>>>> 
>>>>>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>> 
>>>>>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>>>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>>>>>> Things seem to regress since.
>>>>>>>> 
>>>>>>>> I apologize in advance if this has already been reported. Please
>>>>>>>> let me know what I can do to further help (truss/debug).
>>>>>>> 
>>>>>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>>>>>> 
>>>>>>> $ git clone git://git.kernel.dk/fio
>>>>>>> $ cd fio; make
>>>>>>> $ git bisect start
>>>>>>> $ git bisect good fio-2.1.3
>>>>>>> $ git bisect bad fio-2.1.4
>>>>>>> 
>>>>>>> This starts the bisect series, now do:
>>>>>>> 
>>>>>>> $ make clean; make
>>>>>>> 
>>>>>>> and re-run your direct=0 job file. If it worked, then you do
>>>>>>> 
>>>>>>> $ git bisect good
>>>>>>> 
>>>>>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>>>>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>>>>>> 
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Here�s where it started working again:
>>>>>> 
>>>>>> # git bisect good
>>>>>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>>>>>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>>>>>> 
>>>>>> Let me know how else I can help.
>>>>> 
>>>>> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
>>>>> 
>>>>> --
>>>>> Jens Axboe
>>>>> 
>>>> 
>>>> Here you go:
>>>> 
>>>> # git bisect good
>>>> ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
>>>> commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
>>>> Author: Jens Axboe <axboe@kernel.dk>
>>>> Date:   Fri Oct 11 10:27:28 2013 -0600
>>>> 
>>>>   ppc: disable CPU clock until we can detect whether we have it or not
>>>> 
>>>>   The child segfault test should catch it, however it does not on
>>>>   AIX at least.
>>>> 
>>>>   Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>> 
>>> Hmm, that can�t possibly be correct. Would you mind redoing the bisection,
>>> just to double check?
>>> 
>>> Thanks!
>>> 
>>> �
>>> Jens Axboe
>> 
>> Ok. Let�s try this again. Silly user, me.
>> 
>> # git bisect good
>> 7cb024f89dbbc314e740885afccd9a05da056cf1 is the first bad commit
>> commit 7cb024f89dbbc314e740885afccd9a05da056cf1
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Wed Nov 6 15:37:35 2013 -0700
>> 
>>     solaris: ensure that -D_REENTRANT gets set
>> 
>>     Apparently some Solaris' require this for threadsafe
>>     errno.
>> 
>>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>> 
>> :100755 100755 b6bfe19aa743fc4104eb587b3ff6068fb5dc67ef ef7be0180258abecd4703ebfcf4ed63625d6392f M      configure
>> 
>> 
>> I have the complete git/bisect session in a screen log if you�d like it.
> 
> That makes a lot more sense! Can you do:
> 
> $ git checkout -f master
> $ git revert 7cb024f89dbbc314e740885afccd9a05da056cf1
> $ make clean; make
> 
> and retest just to be on the safe side? Also, please attach the job file you are using.
> 
> -- 
> Jens Axboe


Actually, based on the git bisect work, I took out -D_REENTRANT from configure, and fio-2.1.6 now works as expected. Still want the job file?



# diff -u configure configure.dist
--- configure   2014-03-05 16:10:50.195429114 -0500
+++ configure.dist      2014-03-05 16:10:21.590075963 -0500
@@ -183,7 +183,7 @@
   targetos='OpenBSD'
 elif check_define __sun__ ; then
   targetos='SunOS'
-  CFLAGS="$CFLAGS"
+  CFLAGS="$CFLAGS -D_REENTRANT"
 else
   targetos=`uname -s`
 fi



-- 
Robin P. Blanchard
Solutions Engineer
Coraid Global Field Services
www.coraid.com
+1 650.730.5140

Re: regression since 2.1.3 (solaris/zfs)

[Jens Axboe]

On 03/05/2014 02:12 PM, Robin P. Blanchard wrote:
>
> On Mar 5, 2014, at 16:05 PM, Jens Axboe <axboe@kernel.dk> wrote:
>
>> On 03/05/2014 01:39 PM, Robin P. Blanchard wrote:
>>>
>>> On Mar 5, 2014, at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>>> On Mar 5, 2014, at 9:27 AM, Robin P. Blanchard <robin@coraid.com> wrote:
>>>>
>>>>>
>>>>> On Mar 5, 2014, at 11:18 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 2014-03-05 09:11, Robin P. Blanchard wrote:
>>>>>>>
>>>>>>> On Mar 3, 2014, at 10:25 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>>
>>>>>>>> On 03/02/2014 07:59 AM, Robin P. Blanchard wrote:
>>>>>>>>> My config file has direct=0, which until 2.1.4 worked as expected.
>>>>>>>>> Things seem to regress since.
>>>>>>>>>
>>>>>>>>> I apologize in advance if this has already been reported. Please
>>>>>>>>> let me know what I can do to further help (truss/debug).
>>>>>>>>
>>>>>>>> This isn't a known issue, so thanks for reporting it. The easiest way to debug this is to git bisect it. Looks like you are running from the tar balls, but I assume you have git installed? I'm assuming fio-2.1.3 worked for you - if not, just replace fio-2.1.3 in the below with whatever latest version did work. If you do, the cheat sheet is something ala:
>>>>>>>>
>>>>>>>> $ git clone git://git.kernel.dk/fio
>>>>>>>> $ cd fio; make
>>>>>>>> $ git bisect start
>>>>>>>> $ git bisect good fio-2.1.3
>>>>>>>> $ git bisect bad fio-2.1.4
>>>>>>>>
>>>>>>>> This starts the bisect series, now do:
>>>>>>>>
>>>>>>>> $ make clean; make
>>>>>>>>
>>>>>>>> and re-run your direct=0 job file. If it worked, then you do
>>>>>>>>
>>>>>>>> $ git bisect good
>>>>>>>>
>>>>>>>> and if not, you do git bisect bad instead. This gets you a new point in the tree to test, so repeat the make clean; make and re-run the test.
>>>>>>>> Keep doing this good/bad iteration until fio tells you what commit broke the test for you. Then send those results here!
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jens Axboe
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here�s where it started working again:
>>>>>>>
>>>>>>> # git bisect good
>>>>>>> Bisecting: 4 revisions left to test after this (roughly 2 steps)
>>>>>>> [3bb0a7b0fda9945973f799ab253c70d3cb0e5c8b] howto: Fix redundant entries
>>>>>>>
>>>>>>> Let me know how else I can help.
>>>>>>
>>>>>> Please keep going until it tells you what the definitively bad commit is. It'll end up spitting out that info, if you keep doing git bisect good/bad on each test point. You need just ~2 more tests after this one.
>>>>>>
>>>>>> --
>>>>>> Jens Axboe
>>>>>>
>>>>>
>>>>> Here you go:
>>>>>
>>>>> # git bisect good
>>>>> ddc0cc31a2b75b1c7dde870c8867af11fa44db92 is the first bad commit
>>>>> commit ddc0cc31a2b75b1c7dde870c8867af11fa44db92
>>>>> Author: Jens Axboe <axboe@kernel.dk>
>>>>> Date:   Fri Oct 11 10:27:28 2013 -0600
>>>>>
>>>>>    ppc: disable CPU clock until we can detect whether we have it or not
>>>>>
>>>>>    The child segfault test should catch it, however it does not on
>>>>>    AIX at least.
>>>>>
>>>>>    Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>
>>>> Hmm, that can�t possibly be correct. Would you mind redoing the bisection,
>>>> just to double check?
>>>>
>>>> Thanks!
>>>>
>>>> �
>>>> Jens Axboe
>>>
>>> Ok. Let�s try this again. Silly user, me.
>>>
>>> # git bisect good
>>> 7cb024f89dbbc314e740885afccd9a05da056cf1 is the first bad commit
>>> commit 7cb024f89dbbc314e740885afccd9a05da056cf1
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Wed Nov 6 15:37:35 2013 -0700
>>>
>>>      solaris: ensure that -D_REENTRANT gets set
>>>
>>>      Apparently some Solaris' require this for threadsafe
>>>      errno.
>>>
>>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>
>>> :100755 100755 b6bfe19aa743fc4104eb587b3ff6068fb5dc67ef ef7be0180258abecd4703ebfcf4ed63625d6392f M      configure
>>>
>>>
>>> I have the complete git/bisect session in a screen log if you�d like it.
>>
>> That makes a lot more sense! Can you do:
>>
>> $ git checkout -f master
>> $ git revert 7cb024f89dbbc314e740885afccd9a05da056cf1
>> $ make clean; make
>>
>> and retest just to be on the safe side? Also, please attach the job file you are using.
>>
>> --
>> Jens Axboe
>
>
> Actually, based on the git bisect work, I took out -D_REENTRANT from configure,
 > and fio-2.1.6 now works as expected. Still want the job file?

I do, since the direct part makes little sense to me, if you have 
direct=0 set.

-- 
Jens Axboe