fio 2.1.4 intermittently crashes on windows

fio 2.1.4 intermittently crashes on windows

[Todd Dworshak]

Hi �

I am having trouble with fio intermittently crashing on Windows 7.

Version is fio 2.1.4 64 bit, downloaded from Windows build site.

Has anyone else seen this?  If I give it fewer files in the job description, it seems to be less prone to crashing.  Since it crashes in pthreads, it looks like this may be some kind of threading bug.

Call stack of fio dump file is as follows:
                kernel32.dll!WerpReportFaultInternal()  + 0x215 bytes  
               kernel32.dll!WerpReportFault()  + 0x77 bytes     
               kernel32.dll!BasepReportFault()  + 0x1f bytes    
               kernel32.dll!UnhandledExceptionFilter()  + 0x1fc bytes  
               ntdll.dll!string "Enabling heap debug options\n"()  + 0xc9c8 bytes             
               ntdll.dll!__C_specific_handler()  + 0x9c bytes     
               ntdll.dll!RtlpExecuteHandlerForException()  + 0xd bytes                
               ntdll.dll!RtlDispatchException()  + 0x38f bytes    
               ntdll.dll!RtlRaiseStatus()  + 0x60 bytes    
               ntdll.dll!string "Enabling heap debug options\n"()  - 0x17a4e bytes           
               ntdll.dll!KiUserExceptionDispatcher()  + 0x2e bytes         
               libwinpthread-1.dll!0000000064943236()               
                [Frames below may be incorrect and/or missing, no symbols loaded for libwinpthread-1.dll]       
               000000000000000a()       
               fio.exe!0000000000421599()       
                fio.exe!0000000000437ce0()       
                fio.exe!000000000043809d()       
                fio.exe!00000000004013a5()       
                fio.exe!00000000004014d8()       
                kernel32.dll!BaseThreadInitThunk()  + 0xd bytes               
               ntdll.dll!RtlUserThreadStart()  + 0x21 bytes          

Job description follows:

[global]
ioengine=windowsaio
buffered=0
rw=read
bs=4m
size=4000m
iodepth=3
runtime=90

[file1]
filename=z\:\test1a.dat
rate=80m

[file2]
filename=z\:\test2a.dat
rate=80m

[file3]
filename=z\:\test3a.dat
rate=80m

[file4]
filename=z\:\test4a.dat
rate=80m

[file5]
filename=z\:\test5a.dat
rate=80m

[file6]
filename=z\:\test6a.dat
rate=80m

[file7]
filename=z\:\test7a.dat
rate=80m

[file8]
filename=z\:\test8a.dat
rate=80m

[file9]
filename=z\:\test9a.dat
rate=80m

[file10]
filename=z\:\test10a.dat
rate=80m

Thanks in advance for any help you can provide.

Todd Dworshak

Re: fio 2.1.4 intermittently crashes on windows

[Bruce Cran]

On 12/27/2013 6:42 PM, Todd Dworshak wrote:
> I am having trouble with fio intermittently crashing on Windows 7.
>
> Version is fio 2.1.4 64 bit, downloaded from Windows build site.
>
> Has anyone else seen this?  If I give it fewer files in the job description, it seems to be less prone to crashing.  Since it crashes in pthreads, it looks like this may be some kind of threading bug.

I'll try to replicate the problem here, but in the meantime it would be 
useful if you could try running fio under the Cygwin build of gdb to 
capture a stack trace in fio.exe.

-- 
Bruce

RE: fio 2.1.4 intermittently crashes on windows

[Todd Dworshak]

I ran fio in gdb and got the crash.  The stack trace is below.  Not sure if this is a bug in libwinpthread-1.dll or in the way fio is calling it.

Program received signal SIGSEGV, Segmentation fault.
0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
(gdb) bt
#0  0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
#1  0x0000000000000009 in ?? ()
#2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
#3  0x0000000000437ce0 in run_threads () at backend.c:1812
#4  0x000000000043809d in fio_backend () at backend.c:1925
#5  0x0000000000449411 in main (argc=<optimized out>, argv=<optimized out>,
    envp=<optimized out>) at fio.c:50
#6  0x00000000004013a5 in __tmainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
#7  0x00000000004014d8 in mainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
(gdb) f 2
#2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
151     mutex.c: No such file or directory.
(gdb) p mutex
$1 = (struct fio_mutex *) 0x79a2b0
(gdb) p *mutex
$2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0,
  magic = 280}

-----Original Message-----
From: Bruce Cran [mailto:bruce@cran.org.uk] 
Sent: Friday, December 27, 2013 9:12 PM
To: Todd Dworshak; fio@vger.kernel.org
Subject: Re: fio 2.1.4 intermittently crashes on windows

On 12/27/2013 6:42 PM, Todd Dworshak wrote:
> I am having trouble with fio intermittently crashing on Windows 7.
>
> Version is fio 2.1.4 64 bit, downloaded from Windows build site.
>
> Has anyone else seen this?  If I give it fewer files in the job description, it seems to be less prone to crashing.  Since it crashes in pthreads, it looks like this may be some kind of threading bug.

I'll try to replicate the problem here, but in the meantime it would be useful if you could try running fio under the Cygwin build of gdb to capture a stack trace in fio.exe.

--
Bruce

RE: fio 2.1.4 intermittently crashes on windows

[Elliott, Robert (Server Storage)]

I've seen something similar:
- 2.1.1 on a Windows Server 2008 R2 system: OK
- 2.1.4 on a Windows Server 2008 R2 system: crash
- 2.1.4 on a Windows Server 2012 system: OK

This is this error for 2.1.4 on Windows Server 2008 R2: 
    Faulting application name: fio.exe, version: 0.0.0.0, time stamp: 0x5294c1a1
    Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
    Exception code: 0xc0000026  (STATUS_INVALID_DISPOSITION)
    Fault offset: 0x00000000000cd7e8
    Faulting process id: 0x103c
    Faulting application start time: 0x01cf0bc5133c13bf
    Faulting application path: c:\program files\fio\fio.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Running it under cygwin gdb with the same gdb commands, I get:

Starting program: /cygdrive/c/program files/fio/fio --filename=rob.tmp --size=128m --direct=1 --rw=r
andread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iodepth=16 --numjobs=16 --runtime=60
 --group_reporting --name=4ktest --time_based
[New Thread 4760.0xe88]
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread'
 option to get rid of this warning.
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
...
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
fio-2.1.4
[New Thread 4760.0x4c4]
Starting 16 threads
[New Thread 4760.0x1050]
[New Thread 4760.0xf40]
[New Thread 4760.0x1334]
[New Thread 4760.0x948]
[New Thread 4760.0x2b8]
[New Thread 4760.0x11e8]
[New Thread 4760.0x1280]
[New Thread 4760.0x1114]
[New Thread 4760.0xea8]
[New Thread 4760.0x87c]
[New Thread 4760.0xeac]
[New Thread 4760.0x12c8]
[New Thread 4760.0x1190]
[New Thread 4760.0xebc]
[New Thread 4760.0xd48]
[New Thread 4760.0x918]
Jobs: 0 (f=0)
Program received signal SIGSEGV, Segmentation fault.
0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
(gdb) bt
#0  0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
#1  0x0000000000000001 in ?? ()
#2  0x0000000000421599 in fio_mutex_up (mutex=0x2f5b10) at mutex.c:151
#3  0x0000000000437ce0 in run_threads () at backend.c:1812
#4  0x000000000043809d in fio_backend () at backend.c:1925
#5  0x0000000000449411 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at fio.c:50
#6  0x00000000004013a5 in __tmainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
#7  0x00000000004014d8 in mainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
(gdb) f 2
#2  0x0000000000421599 in fio_mutex_up (mutex=0x2f5b10) at mutex.c:151
151     mutex.c: No such file or directory.
(gdb) p mutex
$1 = (struct fio_mutex *) 0x2f5b10
(gdb) p *mutex
$2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0, magic = 172}

---
Rob Elliott    HP Server Storage





> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Todd Dworshak
> Sent: Wednesday, 08 January, 2014 4:02 PM
> To: 'Bruce Cran'; fio@vger.kernel.org
> Subject: RE: fio 2.1.4 intermittently crashes on windows
> 
> I ran fio in gdb and got the crash.  The stack trace is below.  Not sure if this is a
> bug in libwinpthread-1.dll or in the way fio is calling it.
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000064943236 in pthread_mutex_unlock ()
>    from /cygdrive/c/program files/fio/libwinpthread-1.dll
> (gdb) bt
> #0  0x0000000064943236 in pthread_mutex_unlock ()
>    from /cygdrive/c/program files/fio/libwinpthread-1.dll
> #1  0x0000000000000009 in ?? ()
> #2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
> #3  0x0000000000437ce0 in run_threads () at backend.c:1812
> #4  0x000000000043809d in fio_backend () at backend.c:1925
> #5  0x0000000000449411 in main (argc=<optimized out>, argv=<optimized
> out>,
>     envp=<optimized out>) at fio.c:50
> #6  0x00000000004013a5 in __tmainCRTStartup ()
>     at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
> #7  0x00000000004014d8 in mainCRTStartup ()
>     at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
> (gdb) f 2
> #2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
> 151     mutex.c: No such file or directory.
> (gdb) p mutex
> $1 = (struct fio_mutex *) 0x79a2b0
> (gdb) p *mutex
> $2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0,
>   magic = 280}
> 
> -----Original Message-----
> From: Bruce Cran [mailto:bruce@cran.org.uk]
> Sent: Friday, December 27, 2013 9:12 PM
> To: Todd Dworshak; fio@vger.kernel.org
> Subject: Re: fio 2.1.4 intermittently crashes on windows
> 
> On 12/27/2013 6:42 PM, Todd Dworshak wrote:
> > I am having trouble with fio intermittently crashing on Windows 7.
> >
> > Version is fio 2.1.4 64 bit, downloaded from Windows build site.
> >
> > Has anyone else seen this?  If I give it fewer files in the job description, it
> seems to be less prone to crashing.  Since it crashes in pthreads, it looks like
> this may be some kind of threading bug.
> 
> I'll try to replicate the problem here, but in the meantime it would be useful
> if you could try running fio under the Cygwin build of gdb to capture a stack
> trace in fio.exe.
> 
> --
> Bruce
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: fio 2.1.4 intermittently crashes on windows

[Todd Dworshak]

Sounds like we are hitting the same issue.  I am encountering the crash on a Windows 7 box, which pairs up with your crash on Server 2008 R2 (both being basically the same OS).

________________________________________
From: Elliott, Robert (Server Storage) [Elliott@hp.com]
Sent: Wednesday, January 08, 2014 8:07 PM
To: Todd Dworshak; 'Bruce Cran'; fio@vger.kernel.org
Subject: RE: fio 2.1.4 intermittently crashes on windows

I've seen something similar:
- 2.1.1 on a Windows Server 2008 R2 system: OK
- 2.1.4 on a Windows Server 2008 R2 system: crash
- 2.1.4 on a Windows Server 2012 system: OK

This is this error for 2.1.4 on Windows Server 2008 R2:
    Faulting application name: fio.exe, version: 0.0.0.0, time stamp: 0x5294c1a1
    Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
    Exception code: 0xc0000026  (STATUS_INVALID_DISPOSITION)
    Fault offset: 0x00000000000cd7e8
    Faulting process id: 0x103c
    Faulting application start time: 0x01cf0bc5133c13bf
    Faulting application path: c:\program files\fio\fio.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Running it under cygwin gdb with the same gdb commands, I get:

Starting program: /cygdrive/c/program files/fio/fio --filename=rob.tmp --size=128m --direct=1 --rw=r
andread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iodepth=16 --numjobs=16 --runtime=60
 --group_reporting --name=4ktest --time_based
[New Thread 4760.0xe88]
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread'
 option to get rid of this warning.
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
...
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
fio-2.1.4
[New Thread 4760.0x4c4]
Starting 16 threads
[New Thread 4760.0x1050]
[New Thread 4760.0xf40]
[New Thread 4760.0x1334]
[New Thread 4760.0x948]
[New Thread 4760.0x2b8]
[New Thread 4760.0x11e8]
[New Thread 4760.0x1280]
[New Thread 4760.0x1114]
[New Thread 4760.0xea8]
[New Thread 4760.0x87c]
[New Thread 4760.0xeac]
[New Thread 4760.0x12c8]
[New Thread 4760.0x1190]
[New Thread 4760.0xebc]
[New Thread 4760.0xd48]
[New Thread 4760.0x918]
Jobs: 0 (f=0)
Program received signal SIGSEGV, Segmentation fault.
0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
(gdb) bt
#0  0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
#1  0x0000000000000001 in ?? ()
#2  0x0000000000421599 in fio_mutex_up (mutex=0x2f5b10) at mutex.c:151
#3  0x0000000000437ce0 in run_threads () at backend.c:1812
#4  0x000000000043809d in fio_backend () at backend.c:1925
#5  0x0000000000449411 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at fio.c:50
#6  0x00000000004013a5 in __tmainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
#7  0x00000000004014d8 in mainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
(gdb) f 2
#2  0x0000000000421599 in fio_mutex_up (mutex=0x2f5b10) at mutex.c:151
151     mutex.c: No such file or directory.
(gdb) p mutex
$1 = (struct fio_mutex *) 0x2f5b10
(gdb) p *mutex
$2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0, magic = 172}

---
Rob Elliott    HP Server Storage





> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Todd Dworshak
> Sent: Wednesday, 08 January, 2014 4:02 PM
> To: 'Bruce Cran'; fio@vger.kernel.org
> Subject: RE: fio 2.1.4 intermittently crashes on windows
>
> I ran fio in gdb and got the crash.  The stack trace is below.  Not sure if this is a
> bug in libwinpthread-1.dll or in the way fio is calling it.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000064943236 in pthread_mutex_unlock ()
>    from /cygdrive/c/program files/fio/libwinpthread-1.dll
> (gdb) bt
> #0  0x0000000064943236 in pthread_mutex_unlock ()
>    from /cygdrive/c/program files/fio/libwinpthread-1.dll
> #1  0x0000000000000009 in ?? ()
> #2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
> #3  0x0000000000437ce0 in run_threads () at backend.c:1812
> #4  0x000000000043809d in fio_backend () at backend.c:1925
> #5  0x0000000000449411 in main (argc=<optimized out>, argv=<optimized
> out>,
>     envp=<optimized out>) at fio.c:50
> #6  0x00000000004013a5 in __tmainCRTStartup ()
>     at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
> #7  0x00000000004014d8 in mainCRTStartup ()
>     at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
> (gdb) f 2
> #2  0x0000000000421599 in fio_mutex_up (mutex=0x79a2b0) at mutex.c:151
> 151     mutex.c: No such file or directory.
> (gdb) p mutex
> $1 = (struct fio_mutex *) 0x79a2b0
> (gdb) p *mutex
> $2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0,
>   magic = 280}
>
> -----Original Message-----
> From: Bruce Cran [mailto:bruce@cran.org.uk]
> Sent: Friday, December 27, 2013 9:12 PM
> To: Todd Dworshak; fio@vger.kernel.org
> Subject: Re: fio 2.1.4 intermittently crashes on windows
>
> On 12/27/2013 6:42 PM, Todd Dworshak wrote:
> > I am having trouble with fio intermittently crashing on Windows 7.
> >
> > Version is fio 2.1.4 64 bit, downloaded from Windows build site.
> >
> > Has anyone else seen this?  If I give it fewer files in the job description, it
> seems to be less prone to crashing.  Since it crashes in pthreads, it looks like
> this may be some kind of threading bug.
>
> I'll try to replicate the problem here, but in the meantime it would be useful
> if you could try running fio under the Cygwin build of gdb to capture a stack
> trace in fio.exe.
>
> --
> Bruce
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: fio 2.1.4 intermittently crashes on windows

[Sitsofe Wheeler]

On Thu, Jan 09, 2014 at 01:07:56AM +0000, Elliott, Robert (Server Storage) wrote:
> I've seen something similar:
> - 2.1.4 on a Windows Server 2008 R2 system: crash
> 
> This is this error for 2.1.4 on Windows Server 2008 R2: 

<snip>

> (gdb) f 2
> #2  0x0000000000421599 in fio_mutex_up (mutex=0x2f5b10) at mutex.c:151
> 151     mutex.c: No such file or directory.
> (gdb) p mutex
> $1 = (struct fio_mutex *) 0x2f5b10
> (gdb) p *mutex
> $2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0, magic = 172}

This is crazy - the magic value is 172 rather than 1297437765 but the
fio_mutex_up() function is protected by an assert that checks to see if
the mutex's magic number is wrong. Could you can insert something like
the following after line 149:
assert(mutex->magic == FIO_MUTEX_MAGIC);
to prove that the value has not been stomped on before
pthread_cond_signal is run?

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio 2.1.4 intermittently crashes on windows

[Sitsofe Wheeler]

On Fri, Jan 10, 2014 at 12:37:33AM +0000, Elliott, Robert (Server Storage) wrote:
> I installed Cygwin gcc, mingwg64*, make, and a few other packages...
> 
> The built fio.exe ends up complaining about a missing libpthread1-1.dll.  
> 
> Where do I find that?  README says where to get pthreadGC2.dll, but
> that's apparently not the right file any more.  Browsing up on that
> FTP site doesn't show it nearby.

You're right - the README is out of date. If you (re)run your Cygwin
setup exe then you can use its package browser to install
mingw[...]-winpthreads .

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio 2.1.4 intermittently crashes on windows

[Bruce Cran]

On 1/9/2014 10:21 PM, Sitsofe Wheeler wrote:
> On Fri, Jan 10, 2014 at 12:37:33AM +0000, Elliott, Robert (Server Storage) wrote:
>> I installed Cygwin gcc, mingwg64*, make, and a few other packages...
>>
>> The built fio.exe ends up complaining about a missing libpthread1-1.dll.
>>
>> Where do I find that?  README says where to get pthreadGC2.dll, but
>> that's apparently not the right file any more.  Browsing up on that
>> FTP site doesn't show it nearby.
> You're right - the README is out of date. If you (re)run your Cygwin
> setup exe then you can use its package browser to install
> mingw[...]-winpthreads .


Sorry - Cygwin switched from pthreads-win32 to winpthreads a couple of 
months ago and I forgot to update the README during the last set of 
changesets.

-- 
Bruce Cran

RE: fio 2.1.4 intermittently crashes on windows

[Elliott, Robert (Server Storage)]

1. In addition to updating the README, I think os/windows/install.wxs needs to reference libwinpthread-1.dll instead of pthreadGC2.dll, since the wrong .dll is being copied to c:\program files\fio alongside fio.exe.  I re-installed the original 2.1.4 binary, saved that file to a different directory, installed the newly built 2.1.4 with the added assert (whose installation process deletes libwinpthread-1.dll), and put that file back, and it runs.

os/windows/install.wxs:         <ComponentRef Id="pthreadGC2.dll"/>

2. This is where I put the assert in mutex.c:
void fio_mutex_up(struct fio_mutex *mutex)
{
	assert(mutex->magic == FIO_MUTEX_MAGIC);

	pthread_mutex_lock(&mutex->lock);
	read_barrier();
assert(mutex->magic == FIO_MUTEX_MAGIC);	// robrob this is line 150
	if (!mutex->value && mutex->waiters)
		pthread_cond_signal(&mutex->cond);
	mutex->value++;
	pthread_mutex_unlock(&mutex->lock);
}

3. The assert does not trigger, and the same crash occurs as before.

C:\Users\Administrator\fio>c:\cygwin64\bin\gdb -args "c:/program files/fio/fio.exe" --filename=rob.t
mp --size=128m --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iode
pth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --time_based
GNU gdb (GDB) 7.6.50.20130728-cvs (cygwin-special)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-cygwin".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
..
Reading symbols from /cygdrive/c/program files/fio/fio.exe...done.
 (gdb) run
Starting program: /cygdrive/c/program files/fio/fio.exe --filename=rob.tmp --size=128m --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --time_based
[New Thread 3428.0x13e0]
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread'
 option to get rid of this warning.
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
...
4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
fio-2.1.4
[New Thread 3428.0x10a4]
Starting 16 threads
[New Thread 3428.0x1104]
[New Thread 3428.0x1254]
[New Thread 3428.0xde8]
[New Thread 3428.0xeec]
[New Thread 3428.0xe58]
[New Thread 3428.0x13e8]
[New Thread 3428.0x4b8]
[New Thread 3428.0x13e4]
[New Thread 3428.0x7bc]
[New Thread 3428.0x1358]
[New Thread 3428.0x2f0]
[New Thread 3428.0x8dc]
[New Thread 3428.0x38c]
[New Thread 3428.0x133c]
[New Thread 3428.0x2c8]
[New Thread 3428.0x13c0]
Jobs: 0 (f=0)
Program received signal SIGSEGV, Segmentation fault.
0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
(gdb) bt
#0  0x0000000064943236 in pthread_mutex_unlock ()
   from /cygdrive/c/program files/fio/libwinpthread-1.dll
#1  0x000000000000000f in ?? ()
#2  0x0000000000422f79 in fio_mutex_up (mutex=0x305b70) at mutex.c:152
#3  0x0000000000438d71 in run_threads () at backend.c:1812
#4  0x000000000043918d in fio_backend () at backend.c:1925
#5  0x000000000044ba31 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at fio.c:50
#6  0x00000000004013c5 in __tmainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
#7  0x00000000004014e8 in mainCRTStartup ()
    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
(gdb) f 2
#2  0x0000000000422f79 in fio_mutex_up (mutex=0x305b70) at mutex.c:152
152                     pthread_cond_signal(&mutex->cond);
(gdb) p mutex
$1 = (struct fio_mutex *) 0x305b70
(gdb) p *mutex
$2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0, magic = 176}
(gdb)



> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Bruce Cran
> Sent: Friday, 10 January, 2014 12:11 AM
> To: Sitsofe Wheeler; Elliott, Robert (Server Storage)
> Cc: fio@vger.kernel.org
> Subject: Re: fio 2.1.4 intermittently crashes on windows
> 
> 
> On 1/9/2014 10:21 PM, Sitsofe Wheeler wrote:
> > On Fri, Jan 10, 2014 at 12:37:33AM +0000, Elliott, Robert (Server Storage)
> wrote:
> >> I installed Cygwin gcc, mingwg64*, make, and a few other packages...
> >>
> >> The built fio.exe ends up complaining about a missing libpthread1-1.dll.
> >>
> >> Where do I find that?  README says where to get pthreadGC2.dll, but
> >> that's apparently not the right file any more.  Browsing up on that
> >> FTP site doesn't show it nearby.
> > You're right - the README is out of date. If you (re)run your Cygwin
> > setup exe then you can use its package browser to install
> > mingw[...]-winpthreads .
> 
> 
> Sorry - Cygwin switched from pthreads-win32 to winpthreads a couple of
> months ago and I forgot to update the README during the last set of
> changesets.
> 
> --
> Bruce Cran
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: fio 2.1.4 intermittently crashes on windows

[Sitsofe Wheeler]

On Fri, Jan 10, 2014 at 04:46:05PM +0000, Elliott, Robert (Server Storage) wrote:
 
> 3. The assert does not trigger, and the same crash occurs as before.
> 
> C:\Users\Administrator\fio>c:\cygwin64\bin\gdb -args "c:/program files/fio/fio.exe" --filename=rob.tmp --size=128m --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --time_based
> Jobs: 0 (f=0)
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000064943236 in pthread_mutex_unlock ()
>    from /cygdrive/c/program files/fio/libwinpthread-1.dll

I've finally had time to reproduce this on a Windows 7 box. I use a
different command line:

./fio.exe --debug=all --filename=fiojob --thread --size=512 --rw=read --bs=512 --ioengine=sync --verify_pattern=0xdeadbeef --name=fiojobname

The problem appears to be that the mutex is being destroyed while it is
still being held by a different thread. Adding return; to the first line
of fio_mutex_remove in mutex.c papers over the problem...

-- 
Sitsofe | http://sucs.org/~sits/

Re: fio 2.1.4 intermittently crashes on windows

[Bruce Cran]

A fix for the pthreadGC2/libwinthread issue has just been committed - fio.exe is now statically linked.

-- 
Bruce

Sent from my iPhone

> On Jan 10, 2014, at 9:46 AM, "Elliott, Robert (Server Storage)" <Elliott@hp.com> wrote:
> 
> 1. In addition to updating the README, I think os/windows/install.wxs needs to reference libwinpthread-1.dll instead of pthreadGC2.dll, since the wrong .dll is being copied to c:\program files\fio alongside fio.exe.  I re-installed the original 2.1.4 binary, saved that file to a different directory, installed the newly built 2.1.4 with the added assert (whose installation process deletes libwinpthread-1.dll), and put that file back, and it runs.
> 
> os/windows/install.wxs:         <ComponentRef Id="pthreadGC2.dll"/>
> 
> 2. This is where I put the assert in mutex.c:
> void fio_mutex_up(struct fio_mutex *mutex)
> {
>    assert(mutex->magic == FIO_MUTEX_MAGIC);
> 
>    pthread_mutex_lock(&mutex->lock);
>    read_barrier();
> assert(mutex->magic == FIO_MUTEX_MAGIC);    // robrob this is line 150
>    if (!mutex->value && mutex->waiters)
>        pthread_cond_signal(&mutex->cond);
>    mutex->value++;
>    pthread_mutex_unlock(&mutex->lock);
> }
> 
> 3. The assert does not trigger, and the same crash occurs as before.
> 
> C:\Users\Administrator\fio>c:\cygwin64\bin\gdb -args "c:/program files/fio/fio.exe" --filename=rob.t
> mp --size=128m --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iode
> pth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --time_based
> GNU gdb (GDB) 7.6.50.20130728-cvs (cygwin-special)
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-pc-cygwin".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> ..
> Reading symbols from /cygdrive/c/program files/fio/fio.exe...done.
> (gdb) run
> Starting program: /cygdrive/c/program files/fio/fio.exe --filename=rob.tmp --size=128m --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --bs=4k --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --time_based
> [New Thread 3428.0x13e0]
> fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread'
> option to get rid of this warning.
> 4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
> ...
> 4ktest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=windowsaio, iodepth=16
> fio-2.1.4
> [New Thread 3428.0x10a4]
> Starting 16 threads
> [New Thread 3428.0x1104]
> [New Thread 3428.0x1254]
> [New Thread 3428.0xde8]
> [New Thread 3428.0xeec]
> [New Thread 3428.0xe58]
> [New Thread 3428.0x13e8]
> [New Thread 3428.0x4b8]
> [New Thread 3428.0x13e4]
> [New Thread 3428.0x7bc]
> [New Thread 3428.0x1358]
> [New Thread 3428.0x2f0]
> [New Thread 3428.0x8dc]
> [New Thread 3428.0x38c]
> [New Thread 3428.0x133c]
> [New Thread 3428.0x2c8]
> [New Thread 3428.0x13c0]
> Jobs: 0 (f=0)
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000064943236 in pthread_mutex_unlock ()
>   from /cygdrive/c/program files/fio/libwinpthread-1.dll
> (gdb) bt
> #0  0x0000000064943236 in pthread_mutex_unlock ()
>   from /cygdrive/c/program files/fio/libwinpthread-1.dll
> #1  0x000000000000000f in ?? ()
> #2  0x0000000000422f79 in fio_mutex_up (mutex=0x305b70) at mutex.c:152
> #3  0x0000000000438d71 in run_threads () at backend.c:1812
> #4  0x000000000043918d in fio_backend () at backend.c:1925
> #5  0x000000000044ba31 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>    at fio.c:50
> #6  0x00000000004013c5 in __tmainCRTStartup ()
>    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:332
> #7  0x00000000004014e8 in mainCRTStartup ()
>    at /usr/src/debug/mingw64-x86_64-runtime-3.0.0-1/crt/crtexe.c:212
> (gdb) f 2
> #2  0x0000000000422f79 in fio_mutex_up (mutex=0x305b70) at mutex.c:152
> 152                     pthread_cond_signal(&mutex->cond);
> (gdb) p mutex
> $1 = (struct fio_mutex *) 0x305b70
> (gdb) p *mutex
> $2 = {lock = 0x1bab1f00d, cond = 0x100000000, value = 0, waiters = 0, magic = 176}
> (gdb)
> 
> 
> 
>> -----Original Message-----
>> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
>> Behalf Of Bruce Cran
>> Sent: Friday, 10 January, 2014 12:11 AM
>> To: Sitsofe Wheeler; Elliott, Robert (Server Storage)
>> Cc: fio@vger.kernel.org
>> Subject: Re: fio 2.1.4 intermittently crashes on windows
>> 
>> 
>>> On 1/9/2014 10:21 PM, Sitsofe Wheeler wrote:
>>> On Fri, Jan 10, 2014 at 12:37:33AM +0000, Elliott, Robert (Server Storage)
>> wrote:
>>>> I installed Cygwin gcc, mingwg64*, make, and a few other packages...
>>>> 
>>>> The built fio.exe ends up complaining about a missing libpthread1-1.dll.
>>>> 
>>>> Where do I find that?  README says where to get pthreadGC2.dll, but
>>>> that's apparently not the right file any more.  Browsing up on that
>>>> FTP site doesn't show it nearby.
>>> You're right - the README is out of date. If you (re)run your Cygwin
>>> setup exe then you can use its package browser to install
>>> mingw[...]-winpthreads .
>> 
>> 
>> Sorry - Cygwin switched from pthreads-win32 to winpthreads a couple of
>> months ago and I forgot to update the README during the last set of
>> changesets.
>> 
>> --
>> Bruce Cran
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html