No I/O performed by windowsaio/get_io_u: zero buflen

No I/O performed by windowsaio/get_io_u: zero buflen

[smitha sunder]

Hello all,

I have a 30TB drive and I am running into an issue with random writes.
I went through this thread :
https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
already.
I see the same issue with random reads as well.
So not sure what is the issue here in my case; any help is greatly appreciated.


Read Capacity results:
   Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
   Logical block provisioning: lbpme=1, lbprz=1
   Last logical block address=58781073407 (0xdaf9fffff), Number of
logical blocks=58781073408
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block
length=4096 bytes]
   Lowest aligned logical block address=0
Hence:
   Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB


C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
--direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
--name=precond --rw=randwrite --iodepth=1 --numjobs=1
--debug=io,random
fio: set debug option io
fio: set debug option random
precond: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=windowsaio, iodepth=1
fio-3.6
Starting 1 thread
fio: file \\.\PhysicalDrive1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to
get rid of this warning.
io       3372  drop page cache \\.\PhysicalDrive1
random   3372  off rand 259043585
io       3372  fill: io_u 0A458780:
off=0x1a6a158f000,len=0x1000,ddir=1,file=\\.\PhysicalDrive1
io       3372  prep: io_u 0A458780:
off=0x1a6a158f000,len=0x1000,ddir=1,file=\\.\PhysicalDrive1
io       3372  queue: io_u 0A458780:
off=0x1a6a158f000,len=0x1000,ddir=1,file=\\.\PhysicalDrive1
io       3372  calling ->commit(), depth 1
io       3372  io_u_queued_complete: min=1
io       3372  getevents: 1
io       3372  complete: io_u 0A458780:
off=0x1a6a158f000,len=0x1000,ddir=1,file=\\.\PhysicalDrive1
random   3372  off rand 3179521932
io       3372  fill: io_u 0A458780:
off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
io       3372  get_io_u: zero buflen on 0A458780
io       3372  get_io_u failed
io       3372  drop page cache \\.\PhysicalDrive1
random   3372  off rand 17311067694306724737
io       3372  fill: io_u 0A458780:
off=0x19afd9012000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
io       3372  get_io_u: zero buflen on 0A458780
io       3372  get_io_u failed
precond: No I/O performed by windowsaio, perhaps try --debug=io option
for details?
io       3372  close ioengine windowsaio
io       3372  free ioengine windowsaio

precond: (groupid=0, jobs=1): err= 0: pid=3372: Sat Sep 15 13:58:46 2018
  write: IOPS=15, BW=60.6KiB/s (62.1kB/s)(4096B/66msec)
    slat (nsec): min=832488, max=832488, avg=832488.00, stdev= 0.00
    clat (nsec): min=790958, max=790958, avg=790958.00, stdev= 0.00
     lat (nsec): min=1623.4k, max=1623.4k, avg=1623446.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[  791],  5.00th=[  791], 10.00th=[  791], 20.00th=[  791],
     | 30.00th=[  791], 40.00th=[  791], 50.00th=[  791], 60.00th=[  791],
     | 70.00th=[  791], 80.00th=[  791], 90.00th=[  791], 95.00th=[  791],
     | 99.00th=[  791], 99.50th=[  791], 99.90th=[  791], 99.95th=[  791],
     | 99.99th=[  791]
  lat (usec)   : 1000=100.00%
  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=60.6KiB/s (62.1kB/s), 60.6KiB/s-60.6KiB/s
(62.1kB/s-62.1kB/s), io=4096B (4096B), run=66-66msec
io       3944  ioengine cpuio unregistered
io       3944  ioengine mmap unregistered
io       3944  ioengine sync unregistered
io       3944  ioengine psync unregistered
io       3944  ioengine vsync unregistered
io       3944  ioengine null unregistered
io       3944  ioengine net unregistered
io       3944  ioengine ftruncate unregistered
io       3944  ioengine filecreate unregistered
io       3944  ioengine windowsaio unregistered

C:\Program Files (x86)\fio>

I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.

Thanks
smitha

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Sitsofe Wheeler]

On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>
> Hello all,
>
> I have a 30TB drive and I am running into an issue with random writes.
> I went through this thread :
> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
> already.

I think that was something different regarding different blocksizes
per direction.

> I see the same issue with random reads as well.
> So not sure what is the issue here in my case; any help is greatly appreciated.
>
>
> Read Capacity results:
>    Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>    Logical block provisioning: lbpme=1, lbprz=1
>    Last logical block address=58781073407 (0xdaf9fffff), Number of
> logical blocks=58781073408
>    Logical block length=512 bytes
>    Logical blocks per physical block exponent=3 [so physical block
> length=4096 bytes]
>    Lowest aligned logical block address=0
> Hence:
>    Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>
>
> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
> --debug=io,random

<snip>

> io       3372  fill: io_u 0A458780:
> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
> io       3372  get_io_u: zero buflen on 0A458780
> io       3372  get_io_u failed
> io       3372  drop page cache \\.\PhysicalDrive1
> random   3372  off rand 17311067694306724737

That offset is crazy big - I'm sure it's bigger than a petabyte so my
guess is that something is overflowing. If you use --size=27g does the
job go through?

[...]

> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.

-- 
Sitsofe | http://sucs.org/~sits/

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[smitha sunder]

On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>
>> Hello all,
>>
>> I have a 30TB drive and I am running into an issue with random writes.
>> I went through this thread :
>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>> already.
>
> I think that was something different regarding different blocksizes
> per direction.
>
>> I see the same issue with random reads as well.
>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>
>>
>> Read Capacity results:
>>    Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>    Logical block provisioning: lbpme=1, lbprz=1
>>    Last logical block address=58781073407 (0xdaf9fffff), Number of
>> logical blocks=58781073408
>>    Logical block length=512 bytes
>>    Logical blocks per physical block exponent=3 [so physical block
>> length=4096 bytes]
>>    Lowest aligned logical block address=0
>> Hence:
>>    Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>
>>
>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>> --debug=io,random
>
> <snip>
>
>> io       3372  fill: io_u 0A458780:
>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>> io       3372  get_io_u: zero buflen on 0A458780
>> io       3372  get_io_u failed
>> io       3372  drop page cache \\.\PhysicalDrive1
>> random   3372  off rand 17311067694306724737
>
> That offset is crazy big - I'm sure it's bigger than a petabyte so my
> guess is that something is overflowing. If you use --size=27g does the
> job go through?
>
> [...]
>
>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>
> --
> Sitsofe | http://sucs.org/~sits/
Hi Sitsofe,

Thanks for the reply!

Yes; If I use --size=27G or if provide the exact size that the OS
displays, then the job goes through.

Thanks

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Jens Axboe]

On 9/16/18 5:24 PM, smitha sunder wrote:
> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>
>>> Hello all,
>>>
>>> I have a 30TB drive and I am running into an issue with random writes.
>>> I went through this thread :
>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>> already.
>>
>> I think that was something different regarding different blocksizes
>> per direction.
>>
>>> I see the same issue with random reads as well.
>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>
>>>
>>> Read Capacity results:
>>>    Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>    Logical block provisioning: lbpme=1, lbprz=1
>>>    Last logical block address=58781073407 (0xdaf9fffff), Number of
>>> logical blocks=58781073408
>>>    Logical block length=512 bytes
>>>    Logical blocks per physical block exponent=3 [so physical block
>>> length=4096 bytes]
>>>    Lowest aligned logical block address=0
>>> Hence:
>>>    Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>
>>>
>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>> --debug=io,random
>>
>> <snip>
>>
>>> io       3372  fill: io_u 0A458780:
>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>> io       3372  get_io_u: zero buflen on 0A458780
>>> io       3372  get_io_u failed
>>> io       3372  drop page cache \\.\PhysicalDrive1
>>> random   3372  off rand 17311067694306724737
>>
>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>> guess is that something is overflowing. If you use --size=27g does the
>> job go through?
>>
>> [...]
>>
>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>
>> --
>> Sitsofe | http://sucs.org/~sits/
> Hi Sitsofe,
> 
> Thanks for the reply!
> 
> Yes; If I use --size=27G or if provide the exact size that the OS
> displays, then the job goes through.

Are you running a 32-bit or 64-bit build of fio?

-- 
Jens Axboe

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[smitha sunder]

> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>> 
>>>> Hello all,
>>>> 
>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>> I went through this thread :
>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>> already.
>>> 
>>> I think that was something different regarding different blocksizes
>>> per direction.
>>> 
>>>> I see the same issue with random reads as well.
>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>> 
>>>> 
>>>> Read Capacity results:
>>>>   Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>   Logical block provisioning: lbpme=1, lbprz=1
>>>>   Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>> logical blocks=58781073408
>>>>   Logical block length=512 bytes
>>>>   Logical blocks per physical block exponent=3 [so physical block
>>>> length=4096 bytes]
>>>>   Lowest aligned logical block address=0
>>>> Hence:
>>>>   Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>> 
>>>> 
>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>> --debug=io,random
>>> 
>>> <snip>
>>> 
>>>> io       3372  fill: io_u 0A458780:
>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>> io       3372  get_io_u failed
>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>> random   3372  off rand 17311067694306724737
>>> 
>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>> guess is that something is overflowing. If you use --size=27g does the
>>> job go through?
>>> 
>>> [...]
>>> 
>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>> 
>>> --
>>> Sitsofe | http://sucs.org/~sits/
>> Hi Sitsofe,
>> 
>> Thanks for the reply!
>> 
>> Yes; If I use --size=27G or if provide the exact size that the OS
>> displays, then the job goes through.
> 
> Are you running a 32-bit or 64-bit build of fio?
> 
> -- 
> Jens Axboe
> 
32 bit.

Thanks
Smitha 

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Jens Axboe]

On 9/16/18 9:13 PM, smitha sunder wrote:
> 
> 
>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>
>>>>> Hello all,
>>>>>
>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>> I went through this thread :
>>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>>> already.
>>>>
>>>> I think that was something different regarding different blocksizes
>>>> per direction.
>>>>
>>>>> I see the same issue with random reads as well.
>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>
>>>>>
>>>>> Read Capacity results:
>>>>>   Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>>   Logical block provisioning: lbpme=1, lbprz=1
>>>>>   Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>>> logical blocks=58781073408
>>>>>   Logical block length=512 bytes
>>>>>   Logical blocks per physical block exponent=3 [so physical block
>>>>> length=4096 bytes]
>>>>>   Lowest aligned logical block address=0
>>>>> Hence:
>>>>>   Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>
>>>>>
>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>>> --debug=io,random
>>>>
>>>> <snip>
>>>>
>>>>> io       3372  fill: io_u 0A458780:
>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>> io       3372  get_io_u failed
>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>> random   3372  off rand 17311067694306724737
>>>>
>>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>>> guess is that something is overflowing. If you use --size=27g does the
>>>> job go through?
>>>>
>>>> [...]
>>>>
>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>
>>>> --
>>>> Sitsofe | http://sucs.org/~sits/
>>> Hi Sitsofe,
>>>
>>> Thanks for the reply!
>>>
>>> Yes; If I use --size=27G or if provide the exact size that the OS
>>> displays, then the job goes through.
>>
>> Are you running a 32-bit or 64-bit build of fio?
>>
>> -- 
>> Jens Axboe
>>
> 32 bit.

I thought so. I see a few 32-bit issues with huge devices. One of them is
this one:

http://git.kernel.dk/cgit/fio/commit/?id=604d3f5bd9f2b985568593c23f8292cbc7f4044c

but I'm sure there are others, I'll try and reproduce and get this fixed.

-- 
Jens Axboe

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[smitha sunder]

> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 9/16/18 9:13 PM, smitha sunder wrote:
>> 
>> 
>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> 
>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>> 
>>>>>> Hello all,
>>>>>> 
>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>> I went through this thread :
>>>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>>>> already.
>>>>> 
>>>>> I think that was something different regarding different blocksizes
>>>>> per direction.
>>>>> 
>>>>>> I see the same issue with random reads as well.
>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>> 
>>>>>> 
>>>>>> Read Capacity results:
>>>>>>  Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>>>  Logical block provisioning: lbpme=1, lbprz=1
>>>>>>  Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>>>> logical blocks=58781073408
>>>>>>  Logical block length=512 bytes
>>>>>>  Logical blocks per physical block exponent=3 [so physical block
>>>>>> length=4096 bytes]
>>>>>>  Lowest aligned logical block address=0
>>>>>> Hence:
>>>>>>  Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>> 
>>>>>> 
>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>>>> --debug=io,random
>>>>> 
>>>>> <snip>
>>>>> 
>>>>>> io       3372  fill: io_u 0A458780:
>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>> io       3372  get_io_u failed
>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>> random   3372  off rand 17311067694306724737
>>>>> 
>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>>>> guess is that something is overflowing. If you use --size=27g does the
>>>>> job go through?
>>>>> 
>>>>> [...]
>>>>> 
>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>> 
>>>>> --
>>>>> Sitsofe | http://sucs.org/~sits/
>>>> Hi Sitsofe,
>>>> 
>>>> Thanks for the reply!
>>>> 
>>>> Yes; If I use --size=27G or if provide the exact size that the OS
>>>> displays, then the job goes through.
>>> 
>>> Are you running a 32-bit or 64-bit build of fio?
>>> 
>>> -- 
>>> Jens Axboe
>>> 
>> 32 bit.
> 
> I thought so. I see a few 32-bit issues with huge devices. One of them is
> this one:
> 
> http://git.kernel.dk/cgit/fio/commit/?id=604d3f5bd9f2b985568593c23f8292cbc7f4044c
> 
> but I'm sure there are others, I'll try and reproduce and get this fixed.
> 
> -- 
> Jens Axboe
> 
I see the same issue even with 64 bit fio build.

Thanks
Smitha

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Jens Axboe]

On 9/16/18 9:34 PM, smitha sunder wrote:
> 
> 
>> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> On 9/16/18 9:13 PM, smitha sunder wrote:
>>>
>>>
>>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>>> I went through this thread :
>>>>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>>>>> already.
>>>>>>
>>>>>> I think that was something different regarding different blocksizes
>>>>>> per direction.
>>>>>>
>>>>>>> I see the same issue with random reads as well.
>>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>>>
>>>>>>>
>>>>>>> Read Capacity results:
>>>>>>>  Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>>>>  Logical block provisioning: lbpme=1, lbprz=1
>>>>>>>  Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>>>>> logical blocks=58781073408
>>>>>>>  Logical block length=512 bytes
>>>>>>>  Logical blocks per physical block exponent=3 [so physical block
>>>>>>> length=4096 bytes]
>>>>>>>  Lowest aligned logical block address=0
>>>>>>> Hence:
>>>>>>>  Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>>>
>>>>>>>
>>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>>>>> --debug=io,random
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> io       3372  fill: io_u 0A458780:
>>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>>> io       3372  get_io_u failed
>>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>>> random   3372  off rand 17311067694306724737
>>>>>>
>>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>>>>> guess is that something is overflowing. If you use --size=27g does the
>>>>>> job go through?
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>>>
>>>>>> --
>>>>>> Sitsofe | http://sucs.org/~sits/
>>>>> Hi Sitsofe,
>>>>>
>>>>> Thanks for the reply!
>>>>>
>>>>> Yes; If I use --size=27G or if provide the exact size that the OS
>>>>> displays, then the job goes through.
>>>>
>>>> Are you running a 32-bit or 64-bit build of fio?
>>>>
>>>> -- 
>>>> Jens Axboe
>>>>
>>> 32 bit.
>>
>> I thought so. I see a few 32-bit issues with huge devices. One of them is
>> this one:
>>
>> http://git.kernel.dk/cgit/fio/commit/?id=604d3f5bd9f2b985568593c23f8292cbc7f4044c
>>
>> but I'm sure there are others, I'll try and reproduce and get this fixed.
>>
>> -- 
>> Jens Axboe
>>
> I see the same issue even with 64 bit fio build.

The difference is that an unsigned long on 64-bit windows builds is still
32-bit, whereas it's 64-bit on linux. So the easiest for me is to test on
32-bit linux, which _probably_ then hits the same thing...

-- 
Jens Axboe

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Jens Axboe]

On 9/16/18 9:34 PM, smitha sunder wrote:
> 
> 
>> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> On 9/16/18 9:13 PM, smitha sunder wrote:
>>>
>>>
>>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>>> I went through this thread :
>>>>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>>>>> already.
>>>>>>
>>>>>> I think that was something different regarding different blocksizes
>>>>>> per direction.
>>>>>>
>>>>>>> I see the same issue with random reads as well.
>>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>>>
>>>>>>>
>>>>>>> Read Capacity results:
>>>>>>>  Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>>>>  Logical block provisioning: lbpme=1, lbprz=1
>>>>>>>  Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>>>>> logical blocks=58781073408
>>>>>>>  Logical block length=512 bytes
>>>>>>>  Logical blocks per physical block exponent=3 [so physical block
>>>>>>> length=4096 bytes]
>>>>>>>  Lowest aligned logical block address=0
>>>>>>> Hence:
>>>>>>>  Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>>>
>>>>>>>
>>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>>>>> --debug=io,random
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> io       3372  fill: io_u 0A458780:
>>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>>> io       3372  get_io_u failed
>>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>>> random   3372  off rand 17311067694306724737
>>>>>>
>>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>>>>> guess is that something is overflowing. If you use --size=27g does the
>>>>>> job go through?
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>>>
>>>>>> --
>>>>>> Sitsofe | http://sucs.org/~sits/
>>>>> Hi Sitsofe,
>>>>>
>>>>> Thanks for the reply!
>>>>>
>>>>> Yes; If I use --size=27G or if provide the exact size that the OS
>>>>> displays, then the job goes through.
>>>>
>>>> Are you running a 32-bit or 64-bit build of fio?
>>>>
>>>> -- 
>>>> Jens Axboe
>>>>
>>> 32 bit.
>>
>> I thought so. I see a few 32-bit issues with huge devices. One of them is
>> this one:
>>
>> http://git.kernel.dk/cgit/fio/commit/?id=604d3f5bd9f2b985568593c23f8292cbc7f4044c
>>
>> but I'm sure there are others, I'll try and reproduce and get this fixed.
>>
>> -- 
>> Jens Axboe
>>
> I see the same issue even with 64 bit fio build.

One more fix, and it seems to be running for me:

http://git.kernel.dk/cgit/fio/commit/?id=39c56bc010a609f6c89955cbcfa289834ffef336

Any chance you can try a new build that has those last two commits?

-- 
Jens Axboe

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[smitha sunder]

> On Sep 16, 2018, at 8:45 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 9/16/18 9:34 PM, smitha sunder wrote:
>> 
>> 
>>>> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> 
>>>> On 9/16/18 9:13 PM, smitha sunder wrote:
>>>> 
>>>> 
>>>>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>> 
>>>>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Hello all,
>>>>>>>> 
>>>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>>>> I went through this thread :
>>>>>>>> https://www.spinics.net/lists/fio/msg06294.html that seems to be fixed
>>>>>>>> already.
>>>>>>> 
>>>>>>> I think that was something different regarding different blocksizes
>>>>>>> per direction.
>>>>>>> 
>>>>>>>> I see the same issue with random reads as well.
>>>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Read Capacity results:
>>>>>>>> Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
>>>>>>>> Logical block provisioning: lbpme=1, lbprz=1
>>>>>>>> Last logical block address=58781073407 (0xdaf9fffff), Number of
>>>>>>>> logical blocks=58781073408
>>>>>>>> Logical block length=512 bytes
>>>>>>>> Logical blocks per physical block exponent=3 [so physical block
>>>>>>>> length=4096 bytes]
>>>>>>>> Lowest aligned logical block address=0
>>>>>>>> Hence:
>>>>>>>> Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>>>> 
>>>>>>>> 
>>>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio --group_reporting
>>>>>>>> --direct=1 --size=100% --bs=4K --thread --filename=\\.\PhysicalDrive1
>>>>>>>> --name=precond --rw=randwrite --iodepth=1 --numjobs=1
>>>>>>>> --debug=io,random
>>>>>>> 
>>>>>>> <snip>
>>>>>>> 
>>>>>>>> io       3372  fill: io_u 0A458780:
>>>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>>>> io       3372  get_io_u failed
>>>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>>>> random   3372  off rand 17311067694306724737
>>>>>>> 
>>>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte so my
>>>>>>> guess is that something is overflowing. If you use --size=27g does the
>>>>>>> job go through?
>>>>>>> 
>>>>>>> [...]
>>>>>>> 
>>>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>>>> 
>>>>>>> --
>>>>>>> Sitsofe | http://sucs.org/~sits/
>>>>>> Hi Sitsofe,
>>>>>> 
>>>>>> Thanks for the reply!
>>>>>> 
>>>>>> Yes; If I use --size=27G or if provide the exact size that the OS
>>>>>> displays, then the job goes through.
>>>>> 
>>>>> Are you running a 32-bit or 64-bit build of fio?
>>>>> 
>>>>> -- 
>>>>> Jens Axboe
>>>>> 
>>>> 32 bit.
>>> 
>>> I thought so. I see a few 32-bit issues with huge devices. One of them is
>>> this one:
>>> 
>>> http://git.kernel.dk/cgit/fio/commit/?id=604d3f5bd9f2b985568593c23f8292cbc7f4044c
>>> 
>>> but I'm sure there are others, I'll try and reproduce and get this fixed.
>>> 
>>> -- 
>>> Jens Axboe
>>> 
>> I see the same issue even with 64 bit fio build.
> 
> One more fix, and it seems to be running for me:
> 
> http://git.kernel.dk/cgit/fio/commit/?id=39c56bc010a609f6c89955cbcfa289834ffef336
> 
> Any chance you can try a new build that has those last two commits?
> 
> -- 
> Jens Axboe
> 

Certainly; I can give the new build a try. 

Thanks
Smitha 

RE: No I/O performed by windowsaio/get_io_u: zero buflen

[Smitha Sunder]

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jens Axboe
Sent: Sunday, September 16, 2018 8:42 PM
To: smitha sunder <sundersmitha@gmail.com>
Cc: Sitsofe Wheeler <sitsofe@gmail.com>; fio <fio@vger.kernel.org>
Subject: Re: No I/O performed by windowsaio/get_io_u: zero buflen

On 9/16/18 9:34 PM, smitha sunder wrote:
> 
> 
>> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> On 9/16/18 9:13 PM, smitha sunder wrote:
>>>
>>>
>>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>>> I went through this thread :
>>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics
>>>>>>> .net_lists_fio_msg06294.html&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=6QDNQBLcbJUsiXnq84y2HKG98aEmhvabF7uK-yKTbhk&e= that seems to be fixed already.
>>>>>>
>>>>>> I think that was something different regarding different 
>>>>>> blocksizes per direction.
>>>>>>
>>>>>>> I see the same issue with random reads as well.
>>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>>>
>>>>>>>
>>>>>>> Read Capacity results:
>>>>>>>  Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 
>>>>>>> protection]  Logical block provisioning: lbpme=1, lbprz=1  Last 
>>>>>>> logical block address=58781073407 (0xdaf9fffff), Number of 
>>>>>>> logical blocks=58781073408  Logical block length=512 bytes  
>>>>>>> Logical blocks per physical block exponent=3 [so physical block
>>>>>>> length=4096 bytes]
>>>>>>>  Lowest aligned logical block address=0
>>>>>>> Hence:
>>>>>>>  Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>>>
>>>>>>>
>>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio 
>>>>>>> --group_reporting
>>>>>>> --direct=1 --size=100% --bs=4K --thread 
>>>>>>> --filename=\\.\PhysicalDrive1 --name=precond --rw=randwrite 
>>>>>>> --iodepth=1 --numjobs=1 --debug=io,random
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> io       3372  fill: io_u 0A458780:
>>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>>> io       3372  get_io_u failed
>>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>>> random   3372  off rand 17311067694306724737
>>>>>>
>>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte 
>>>>>> so my guess is that something is overflowing. If you use 
>>>>>> --size=27g does the job go through?
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>>>
>>>>>> --
>>>>>> Sitsofe | 
>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Es
>>>>>> its_&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgR
>>>>>> BDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=N
>>>>>> aPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=ZnZsmte_bg5yC3adbwXI
>>>>>> U4EmuEn97cA9o1YpZxDU4m0&e=
>>>>> Hi Sitsofe,
>>>>>
>>>>> Thanks for the reply!
>>>>>
>>>>> Yes; If I use --size=27G or if provide the exact size that the OS 
>>>>> displays, then the job goes through.
>>>>
>>>> Are you running a 32-bit or 64-bit build of fio?
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>> 32 bit.
>>
>> I thought so. I see a few 32-bit issues with huge devices. One of 
>> them is this one:
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__git.kernel.dk_cgi
>> t_fio_commit_-3Fid-3D604d3f5bd9f2b985568593c23f8292cbc7f4044c&d=DwIDa
>> Q&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2&
>> r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4c
>> zQueVX-2LvLGD_SEwee41c&s=AJCs8E23SmJj-z7BbkuP_yxTNDg3g6BZ8FYIcV3EmAY&
>> e=
>>
>> but I'm sure there are others, I'll try and reproduce and get this fixed.
>>
>> --
>> Jens Axboe
>>
> I see the same issue even with 64 bit fio build.

The difference is that an unsigned long on 64-bit windows builds is still 32-bit, whereas it's 64-bit on linux. So the easiest for me is to test on 32-bit linux, which _probably_ then hits the same thing...

--
Jens Axboe


Jens,

Got the latest build from https://ci.appveyor.com/project/axboe/fio/build/1.0.891/job/awu51avmepn1kvgf/artifacts.

With size=100%, bs=4K , I no longer see the issue; thanks for the fix!

precond: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
fio-3.10
Starting 1 thread
fio: file \\.\PhysicalDrive1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.

precond: (groupid=0, jobs=1): err= 0: pid=2164: Mon Sep 17 09:30:52 2018
  write: IOPS=126k, BW=491MiB/s (515MB/s)(28.8GiB/60001msec)
    slat (nsec): min=1283, max=234980, avg=2264.27, stdev=1230.37
    clat (usec): min=55, max=33282, avg=240.68, stdev=46.86
     lat (usec): min=57, max=33295, avg=242.94, stdev=46.91
    clat percentiles (usec):
     |  1.00th=[  167],  5.00th=[  184], 10.00th=[  194], 20.00th=[  206],
     | 30.00th=[  217], 40.00th=[  227], 50.00th=[  235], 60.00th=[  245],
     | 70.00th=[  258], 80.00th=[  273], 90.00th=[  297], 95.00th=[  318],
     | 99.00th=[  367], 99.50th=[  388], 99.90th=[  461], 99.95th=[  611],
     | 99.99th=[ 1172]
   bw (  KiB/s): min=470928, max=608768, per=100.00%, avg=503204.03, stdev=36189.78, samples=119
   iops        : min=117732, max=152192, avg=125801.01, stdev=9047.44, samples=119
  lat (usec)   : 100=0.01%, 250=64.69%, 500=35.24%, 750=0.04%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.01%, 50=0.01%
  cpu          : usr=8.33%, sys=20.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=81.1%, 32=18.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=92.8%, 8=3.7%, 16=3.3%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7545152,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=491MiB/s (515MB/s), 491MiB/s-491MiB/s (515MB/s-515MB/s), io=28.8GiB (30.9GB), run=60001-60001msec

Thanks
smitha

Re: No I/O performed by windowsaio/get_io_u: zero buflen

[Jens Axboe]

On 9/17/18 10:33 AM, Smitha Sunder wrote:
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jens Axboe
> Sent: Sunday, September 16, 2018 8:42 PM
> To: smitha sunder <sundersmitha@gmail.com>
> Cc: Sitsofe Wheeler <sitsofe@gmail.com>; fio <fio@vger.kernel.org>
> Subject: Re: No I/O performed by windowsaio/get_io_u: zero buflen
> 
> On 9/16/18 9:34 PM, smitha sunder wrote:
>>
>>
>>> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>>> On 9/16/18 9:13 PM, smitha sunder wrote:
>>>>
>>>>
>>>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>
>>>>>>> On 9/16/18 5:24 PM, smitha sunder wrote:
>>>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I have a 30TB drive and I am running into an issue with random writes.
>>>>>>>> I went through this thread :
>>>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics
>>>>>>>> .net_lists_fio_msg06294.html&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=6QDNQBLcbJUsiXnq84y2HKG98aEmhvabF7uK-yKTbhk&e= that seems to be fixed already.
>>>>>>>
>>>>>>> I think that was something different regarding different 
>>>>>>> blocksizes per direction.
>>>>>>>
>>>>>>>> I see the same issue with random reads as well.
>>>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>> Read Capacity results:
>>>>>>>>  Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 
>>>>>>>> protection]  Logical block provisioning: lbpme=1, lbprz=1  Last 
>>>>>>>> logical block address=58781073407 (0xdaf9fffff), Number of 
>>>>>>>> logical blocks=58781073408  Logical block length=512 bytes  
>>>>>>>> Logical blocks per physical block exponent=3 [so physical block
>>>>>>>> length=4096 bytes]
>>>>>>>>  Lowest aligned logical block address=0
>>>>>>>> Hence:
>>>>>>>>  Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB
>>>>>>>>
>>>>>>>>
>>>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio 
>>>>>>>> --group_reporting
>>>>>>>> --direct=1 --size=100% --bs=4K --thread 
>>>>>>>> --filename=\\.\PhysicalDrive1 --name=precond --rw=randwrite 
>>>>>>>> --iodepth=1 --numjobs=1 --debug=io,random
>>>>>>>
>>>>>>> <snip>
>>>>>>>
>>>>>>>> io       3372  fill: io_u 0A458780:
>>>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1
>>>>>>>> io       3372  get_io_u: zero buflen on 0A458780
>>>>>>>> io       3372  get_io_u failed
>>>>>>>> io       3372  drop page cache \\.\PhysicalDrive1
>>>>>>>> random   3372  off rand 17311067694306724737
>>>>>>>
>>>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte 
>>>>>>> so my guess is that something is overflowing. If you use 
>>>>>>> --size=27g does the job go through?
>>>>>>>
>>>>>>> [...]
>>>>>>>
>>>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc.
>>>>>>>
>>>>>>> --
>>>>>>> Sitsofe | 
>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Es
>>>>>>> its_&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgR
>>>>>>> BDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=N
>>>>>>> aPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=ZnZsmte_bg5yC3adbwXI
>>>>>>> U4EmuEn97cA9o1YpZxDU4m0&e=
>>>>>> Hi Sitsofe,
>>>>>>
>>>>>> Thanks for the reply!
>>>>>>
>>>>>> Yes; If I use --size=27G or if provide the exact size that the OS 
>>>>>> displays, then the job goes through.
>>>>>
>>>>> Are you running a 32-bit or 64-bit build of fio?
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>>
>>>> 32 bit.
>>>
>>> I thought so. I see a few 32-bit issues with huge devices. One of 
>>> them is this one:
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__git.kernel.dk_cgi
>>> t_fio_commit_-3Fid-3D604d3f5bd9f2b985568593c23f8292cbc7f4044c&d=DwIDa
>>> Q&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2&
>>> r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4c
>>> zQueVX-2LvLGD_SEwee41c&s=AJCs8E23SmJj-z7BbkuP_yxTNDg3g6BZ8FYIcV3EmAY&
>>> e=
>>>
>>> but I'm sure there are others, I'll try and reproduce and get this fixed.
>>>
>>> --
>>> Jens Axboe
>>>
>> I see the same issue even with 64 bit fio build.
> 
> The difference is that an unsigned long on 64-bit windows builds is still 32-bit, whereas it's 64-bit on linux. So the easiest for me is to test on 32-bit linux, which _probably_ then hits the same thing...
> 
> --
> Jens Axboe
> 
> 
> Jens,
> 
> Got the latest build from https://ci.appveyor.com/project/axboe/fio/build/1.0.891/job/awu51avmepn1kvgf/artifacts.
> 
> With size=100%, bs=4K , I no longer see the issue; thanks for the fix!

Great, thanks a lot for reporting and testing!

For what it's worth, I think we got all of them now, puzzled why we haven't
run into this sooner. I guess not a lot of folks run 32-bit builds. I also
verified the random distribution of 32-bit, and it looks fine.

-- 
Jens Axboe