RE: fio win affinity broken

[<eliezer@integritech.solutions>]

Jeff,

I just wanted to confirm, I'm also getting the same error on a test windows 2016 server with 16 total cores and fio version 3.9 for windows:

iops-testing: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=windowsaio, iodepth=1
fio-3.9
fio_setaffinity: failed to set thread affinity (pid 17004, group 0, mask 100000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 10872, group 0, mask 400000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 17720, group 0, mask 4000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 16588, group 0, mask 40000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1336, group 0, mask 800000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 7164, group 0, mask 200000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 292, group 0, mask 1000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 9548, group 0, mask 2000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 11336, group 0, mask 10000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1676, group 0, mask 2000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 17776, group 0, mask 100000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 16912, group 0, mask 400000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 732, group 0, mask 1000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 18192, group 0, mask 2000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 8472, group 0, mask 8000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 11684, group 0, mask 20000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 10696, group 0, mask 1000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1640, group 0, mask 800000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 3100, group 0, mask 40000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 840, group 0, mask 4000000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 1012, group 0, mask 200000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 376, group 0, mask 8000000000000, GetLastError=87)
[snip]

Regards,
Eliezer





-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jeff Furlong
Sent: Tuesday, October 30, 2018 11:07 PM
To: Elliott, Robert (Persistent Memory); 'Sitsofe Wheeler'
Cc: fio
Subject: RE: fio win affinity broken

Each socket has 8 logical cores, so 16 total cores.  Agreed GetMaximumProcessorCount() returns the max possible cores, and GetActiveProcessorCount() returns the current active cores.  I can try to see if that change resolves it.  Thanks.

Regards,
Jeff

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Elliott, Robert (Persistent Memory)
Sent: Tuesday, October 30, 2018 7:21 PM
To: Jeff Furlong <jeff.furlong@wdc.com>; 'Sitsofe Wheeler' <sitsofe@gmail.com>
Cc: fio <fio@vger.kernel.org>
Subject: RE: fio win affinity broken



> -----Original Message-----
> From: Jeff Furlong [mailto:jeff.furlong@wdc.com]
> Sent: Tuesday, October 30, 2018 11:21 AM
> To: Elliott, Robert (Persistent Memory) <elliott@hpe.com>; 'Sitsofe 
> Wheeler' <sitsofe@gmail.com>
> Cc: fio <fio@vger.kernel.org>
> Subject: RE: fio win affinity broken
> 
> Robert,
> Agreed split is better.  In trying to trim down the parameter list I 
> left the policy to shared, which is default.  I haven't constrained the fio process, so not sure why the resulting clock threads fail.
> Is the mask shifted the correct number of bits?
> 
> Sitsofe,
> On a x64 version of Win2016, with dual Intel(R) Xeon(R) CPU E5-2623 v3 
> @ 3.00GHz, running
> 
> fio --name=test --ioengine=windowsaio --direct=1 --rw=write 
> --overwrite=1 --
> filename=\\.\PHYSICALDRIVE1 --runtime=1s --thread --cpus_allowed=0-1 
> --debug=process
> 

All of your threads fail starting with the 17th (mask 10000) (see below). 
Is that how many CPU cores you really have?

cpus_online() is calling GetMaximumProcessorCount() which is returning the count of 64:
    process  4788  group=0 group_start=0 group_size=64 search_cpu=0

In linux, that would return only 16 for a system with 16 online CPUs.
GetActiveProcessorCount() might be a better function to call.


fio_setaffinity: failed to set thread affinity (mask 10000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 100000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 1000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 10000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 100000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 1000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 10000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 100000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 1000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 10000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 100000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 1000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 20000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 200000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 2000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 20000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 200000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 2000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 20000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 200000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 2000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 20000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 200000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 2000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 40000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 400000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 4000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 40000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 400000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 4000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 40000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 400000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 4000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 40000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 400000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 4000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 80000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 800000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 8000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 80000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 800000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 8000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 80000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 800000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 8000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 80000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 800000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (mask 8000000000000000, GetLastError=87)


---
Robert Elliott, HPE Persistent Memory