* RE: fio win affinity broken
2018-10-29 21:22 ` Elliott, Robert (Persistent Memory)
@ 2018-10-30 16:20 ` Jeff Furlong
2018-10-31 2:20 ` Elliott, Robert (Persistent Memory)
0 siblings, 1 reply; 8+ messages in thread
From: Jeff Furlong @ 2018-10-30 16:20 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory), 'Sitsofe Wheeler'; +Cc: fio
Robert,
Agreed split is better. In trying to trim down the parameter list I left the policy to shared, which is default. I haven't constrained the fio process, so not sure why the resulting clock threads fail. Is the mask shifted the correct number of bits?
Sitsofe,
On a x64 version of Win2016, with dual Intel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz, running
fio --name=test --ioengine=windowsaio --direct=1 --rw=write --overwrite=1 --filename=\\.\PHYSICALDRIVE1 --runtime=1s --thread --cpus_allowed=0-1 --debug=process
process 4788 row_first_cpu=0 cpumask->row[0]=1
process 4788 first set cpu in mask is at index 0
process 4788 group=0 group_start=0 group_size=64 search_cpu=0
process 4788 row_last_cpu=-1 cpumask->row[7]=0
process 4788 row_last_cpu=-1 cpumask->row[6]=0
process 5868 row_first_cpu=1 cpumask->row[0]=2
process 4788 row_last_cpu=-1 cpumask->row[5]=0
process 4788 row_last_cpu=-1 cpumask->row[4]=0
process 5868 first set cpu in mask is at index 1
process 4788 row_last_cpu=-1 cpumask->row[3]=0
process 4788 row_last_cpu=-1 cpumask->row[2]=0
process 4788 row_last_cpu=-1 cpumask->row[1]=0
process 4788 row_last_cpu=0 cpumask->row[0]=1
process 5868 group=0 group_start=0 group_size=64 search_cpu=1
process 4788 last set cpu in mask is at index 0
process 5868 row_last_cpu=-1 cpumask->row[7]=0
process 4788 Returning group=0 group_mask=1
process 5868 row_last_cpu=-1 cpumask->row[6]=0
process 5868 row_last_cpu=-1 cpumask->row[5]=0
process 5868 row_last_cpu=-1 cpumask->row[4]=0
process 5868 row_last_cpu=-1 cpumask->row[3]=0
process 920 row_first_cpu=3 cpumask->row[0]=8
process 5868 row_last_cpu=-1 cpumask->row[2]=0
process 920 first set cpu in mask is at index 3
process 5868 row_last_cpu=-1 cpumask->row[1]=0
process 5868 row_last_cpu=1 cpumask->row[0]=2
process 5868 last set cpu in mask is at index 1
process 920 group=0 group_start=0 group_size=64 search_cpu=3
process 5868 Returning group=0 group_mask=2
process 920 row_last_cpu=-1 cpumask->row[7]=0
process 920 row_last_cpu=-1 cpumask->row[6]=0
process 920 row_last_cpu=-1 cpumask->row[5]=0
process 920 row_last_cpu=-1 cpumask->row[4]=0
process 920 row_last_cpu=-1 cpumask->row[3]=0
process 920 row_last_cpu=-1 cpumask->row[2]=0
process 920 row_last_cpu=-1 cpumask->row[1]=0
process 5888 row_first_cpu=2 cpumask->row[0]=4
process 920 row_last_cpu=3 cpumask->row[0]=8
process 5888 first set cpu in mask is at index 2
process 920 last set cpu in mask is at index 3
process 920 Returning group=0 group_mask=8
process 5888 group=0 group_start=0 group_size=64 search_cpu=2
process 5888 row_last_cpu=-1 cpumask->row[7]=0
process 5888 row_last_cpu=-1 cpumask->row[6]=0
process 5888 row_last_cpu=-1 cpumask->row[5]=0
process 5888 row_last_cpu=-1 cpumask->row[4]=0
process 5888 row_last_cpu=-1 cpumask->row[3]=0
process 5888 row_last_cpu=-1 cpumask->row[2]=0
process 5888 row_last_cpu=-1 cpumask->row[1]=0
process 5888 row_last_cpu=2 cpumask->row[0]=4
process 5880 row_first_cpu=4 cpumask->row[0]=16
process 5888 last set cpu in mask is at index 2
process 5888 Returning group=0 group_mask=4
process 5880 first set cpu in mask is at index 4
process 5880 group=0 group_start=0 group_size=64 search_cpu=4
process 5880 row_last_cpu=-1 cpumask->row[7]=0
process 5880 row_last_cpu=-1 cpumask->row[6]=0
process 5880 row_last_cpu=-1 cpumask->row[5]=0
process 5880 row_last_cpu=-1 cpumask->row[4]=0
process 5880 row_last_cpu=-1 cpumask->row[3]=0
process 5880 row_last_cpu=-1 cpumask->row[2]=0
process 5880 row_last_cpu=-1 cpumask->row[1]=0
process 5880 row_last_cpu=4 cpumask->row[0]=16
process 5880 last set cpu in mask is at index 4
process 5880 Returning group=0 group_mask=16
process 1064 row_first_cpu=5 cpumask->row[0]=32
process 1064 first set cpu in mask is at index 5
process 1064 group=0 group_start=0 group_size=64 search_cpu=5
process 1064 row_last_cpu=-1 cpumask->row[7]=0
process 1064 row_last_cpu=-1 cpumask->row[6]=0
process 1064 row_last_cpu=-1 cpumask->row[5]=0
process 1064 row_last_cpu=-1 cpumask->row[4]=0
process 5104 row_first_cpu=6 cpumask->row[0]=64
process 5104 first set cpu in mask is at index 6
process 1064 row_last_cpu=-1 cpumask->row[3]=0
process 5104 group=0 group_start=0 group_size=64 search_cpu=6
process 1064 row_last_cpu=-1 cpumask->row[2]=0
process 1064 row_last_cpu=-1 cpumask->row[1]=0
process 5104 row_last_cpu=-1 cpumask->row[7]=0
process 5640 row_first_cpu=14 cpumask->row[0]=16384
process 5640 first set cpu in mask is at index 14
process 1064 row_last_cpu=5 cpumask->row[0]=32
process 5104 row_last_cpu=-1 cpumask->row[6]=0
process 2108 row_first_cpu=7 cpumask->row[0]=128
process 1064 last set cpu in mask is at index 5
process 5104 row_last_cpu=-1 cpumask->row[5]=0
process 5640 group=0 group_start=0 group_size=64 search_cpu=14
process 3776 row_first_cpu=8 cpumask->row[0]=256
process 1064 Returning group=0 group_mask=32
process 5104 row_last_cpu=-1 cpumask->row[4]=0
process 5640 row_last_cpu=-1 cpumask->row[7]=0
process 2108 first set cpu in mask is at index 7
process 1176 row_first_cpu=9 cpumask->row[0]=512
process 5640 row_last_cpu=-1 cpumask->row[6]=0
process 1420 row_first_cpu=13 cpumask->row[0]=8192
process 3776 first set cpu in mask is at index 8
process 5640 row_last_cpu=-1 cpumask->row[5]=0
process 4000 row_first_cpu=10 cpumask->row[0]=1024
process 5640 row_last_cpu=-1 cpumask->row[4]=0
process 3776 group=0 group_start=0 group_size=64 search_cpu=8
process 6028 row_first_cpu=11 cpumask->row[0]=2048
process 5640 row_last_cpu=-1 cpumask->row[3]=0
process 5104 row_last_cpu=-1 cpumask->row[3]=0
process 4000 first set cpu in mask is at index 10
process 2108 group=0 group_start=0 group_size=64 search_cpu=7
process 3776 row_last_cpu=-1 cpumask->row[7]=0
process 1420 first set cpu in mask is at index 13
process 1176 first set cpu in mask is at index 9
process 5104 row_last_cpu=-1 cpumask->row[2]=0
process 5640 row_last_cpu=-1 cpumask->row[2]=0
process 1088 row_first_cpu=12 cpumask->row[0]=4096
process 3776 row_last_cpu=-1 cpumask->row[6]=0
process 2108 row_last_cpu=-1 cpumask->row[7]=0
process 5104 row_last_cpu=-1 cpumask->row[1]=0
process 6028 first set cpu in mask is at index 11
process 5640 row_last_cpu=-1 cpumask->row[1]=0
process 4000 group=0 group_start=0 group_size=64 search_cpu=10
process 3776 row_last_cpu=-1 cpumask->row[5]=0
process 2108 row_last_cpu=-1 cpumask->row[6]=0
process 1420 group=0 group_start=0 group_size=64 search_cpu=13
process 5104 row_last_cpu=6 cpumask->row[0]=64
process 1088 first set cpu in mask is at index 12
process 5640 row_last_cpu=14 cpumask->row[0]=16384
process 4000 row_last_cpu=-1 cpumask->row[7]=0
process 1176 group=0 group_start=0 group_size=64 search_cpu=9
process 1420 row_last_cpu=-1 cpumask->row[7]=0
process 5104 last set cpu in mask is at index 6
process 2108 row_last_cpu=-1 cpumask->row[5]=0
process 3776 row_last_cpu=-1 cpumask->row[4]=0
process 5640 last set cpu in mask is at index 14
process 6028 group=0 group_start=0 group_size=64 search_cpu=11
process 360 row_first_cpu=15 cpumask->row[0]=32768
process 5104 Returning group=0 group_mask=64
process 2108 row_last_cpu=-1 cpumask->row[4]=0
process 1088 group=0 group_start=0 group_size=64 search_cpu=12
process 5640 Returning group=0 group_mask=16384
process 360 first set cpu in mask is at index 15
process 2108 row_last_cpu=-1 cpumask->row[3]=0
process 6028 row_last_cpu=-1 cpumask->row[7]=0
process 3776 row_last_cpu=-1 cpumask->row[3]=0
process 1088 row_last_cpu=-1 cpumask->row[7]=0
process 2108 row_last_cpu=-1 cpumask->row[2]=0
process 6028 row_last_cpu=-1 cpumask->row[6]=0
process 3776 row_last_cpu=-1 cpumask->row[2]=0
process 2108 row_last_cpu=-1 cpumask->row[1]=0
process 4000 row_last_cpu=-1 cpumask->row[6]=0
process 360 group=0 group_start=0 group_size=64 search_cpu=15
process 6028 row_last_cpu=-1 cpumask->row[5]=0
process 1088 row_last_cpu=-1 cpumask->row[6]=0
process 3776 row_last_cpu=-1 cpumask->row[1]=0
process 1420 row_last_cpu=-1 cpumask->row[6]=0
process 2108 row_last_cpu=7 cpumask->row[0]=128
process 360 row_last_cpu=-1 cpumask->row[7]=0
process 1088 row_last_cpu=-1 cpumask->row[5]=0
process 1176 row_last_cpu=-1 cpumask->row[7]=0
process 6028 row_last_cpu=-1 cpumask->row[4]=0
process 1420 row_last_cpu=-1 cpumask->row[5]=0
process 1088 row_last_cpu=-1 cpumask->row[4]=0
process 360 row_last_cpu=-1 cpumask->row[6]=0
process 5948 row_first_cpu=17 cpumask->row[0]=131072
process 3776 row_last_cpu=8 cpumask->row[0]=256
process 5948 first set cpu in mask is at index 17
process 360 row_last_cpu=-1 cpumask->row[5]=0
process 3076 row_first_cpu=18 cpumask->row[0]=262144
process 1420 row_last_cpu=-1 cpumask->row[4]=0
process 360 row_last_cpu=-1 cpumask->row[4]=0
process 3776 last set cpu in mask is at index 8
process 360 row_last_cpu=-1 cpumask->row[3]=0
process 3076 first set cpu in mask is at index 18
process 1088 row_last_cpu=-1 cpumask->row[3]=0
process 5948 group=0 group_start=0 group_size=64 search_cpu=17
process 3776 Returning group=0 group_mask=256
process 360 row_last_cpu=-1 cpumask->row[2]=0
process 6028 row_last_cpu=-1 cpumask->row[3]=0
process 5948 row_last_cpu=-1 cpumask->row[7]=0
process 360 row_last_cpu=-1 cpumask->row[1]=0
process 3076 group=0 group_start=0 group_size=64 search_cpu=18
process 1088 row_last_cpu=-1 cpumask->row[2]=0
process 5948 row_last_cpu=-1 cpumask->row[6]=0
process 4000 row_last_cpu=-1 cpumask->row[5]=0
process 360 row_last_cpu=15 cpumask->row[0]=32768
process 3076 row_last_cpu=-1 cpumask->row[7]=0
process 1088 row_last_cpu=-1 cpumask->row[1]=0
process 6028 row_last_cpu=-1 cpumask->row[2]=0
process 5948 row_last_cpu=-1 cpumask->row[5]=0
process 1420 row_last_cpu=-1 cpumask->row[3]=0
process 360 last set cpu in mask is at index 15
process 2108 last set cpu in mask is at index 7
process 3076 row_last_cpu=-1 cpumask->row[6]=0
process 1088 row_last_cpu=12 cpumask->row[0]=4096
process 6028 row_last_cpu=-1 cpumask->row[1]=0
process 5948 row_last_cpu=-1 cpumask->row[4]=0
process 1420 row_last_cpu=-1 cpumask->row[2]=0
process 360 Returning group=0 group_mask=32768
process 4000 row_last_cpu=-1 cpumask->row[4]=0
process 3076 row_last_cpu=-1 cpumask->row[5]=0
process 1088 last set cpu in mask is at index 12
process 5948 row_last_cpu=-1 cpumask->row[3]=0
process 4000 row_last_cpu=-1 cpumask->row[3]=0
process 2108 Returning group=0 group_mask=128
process 5948 row_last_cpu=-1 cpumask->row[2]=0
process 4000 row_last_cpu=-1 cpumask->row[2]=0
process 1420 row_last_cpu=-1 cpumask->row[1]=0
process 3076 row_last_cpu=-1 cpumask->row[4]=0
process 6028 row_last_cpu=11 cpumask->row[0]=2048
process 4000 row_last_cpu=-1 cpumask->row[1]=0
process 5948 row_last_cpu=-1 cpumask->row[1]=0
process 1088 Returning group=0 group_mask=4096
process 1420 row_last_cpu=13 cpumask->row[0]=8192
process 1176 row_last_cpu=-1 cpumask->row[6]=0
process 3076 row_last_cpu=-1 cpumask->row[3]=0
process 6028 last set cpu in mask is at index 11
process 5948 row_last_cpu=17 cpumask->row[0]=131072
process 4000 row_last_cpu=10 cpumask->row[0]=1024
process 1420 last set cpu in mask is at index 13
process 4280 row_first_cpu=19 cpumask->row[0]=524288
process 3180 row_first_cpu=16 cpumask->row[0]=65536
process 4000 last set cpu in mask is at index 10
process 4280 first set cpu in mask is at index 19
process 1176 row_last_cpu=-1 cpumask->row[5]=0
process 3180 first set cpu in mask is at index 16
process 4000 Returning group=0 group_mask=1024
process 1420 Returning group=0 group_mask=8192
process 3076 row_last_cpu=-1 cpumask->row[2]=0
process 1176 row_last_cpu=-1 cpumask->row[4]=0
process 6028 Returning group=0 group_mask=2048
process 4280 group=0 group_start=0 group_size=64 search_cpu=19
process 5948 last set cpu in mask is at index 17
process 3180 group=0 group_start=0 group_size=64 search_cpu=16
process 1176 row_last_cpu=-1 cpumask->row[3]=0
process 3076 row_last_cpu=-1 cpumask->row[1]=0
process 4280 row_last_cpu=-1 cpumask->row[7]=0
process 3180 row_last_cpu=-1 cpumask->row[7]=0
process 4280 row_last_cpu=-1 cpumask->row[6]=0
process 5948 Returning group=0 group_mask=131072
process 3076 row_last_cpu=18 cpumask->row[0]=262144
process 1176 row_last_cpu=-1 cpumask->row[2]=0
process 4280 row_last_cpu=-1 cpumask->row[5]=0
process 3180 row_last_cpu=-1 cpumask->row[6]=0
process 3076 last set cpu in mask is at index 18
process 4412 row_first_cpu=20 cpumask->row[0]=1048576
process 4412 first set cpu in mask is at index 20
process 4280 row_last_cpu=-1 cpumask->row[4]=0
process 3076 Returning group=0 group_mask=262144
process 4412 group=0 group_start=0 group_size=64 search_cpu=20
process 1176 row_last_cpu=-1 cpumask->row[1]=0
process 4412 row_last_cpu=-1 cpumask->row[7]=0
process 4280 row_last_cpu=-1 cpumask->row[3]=0
process 3180 row_last_cpu=-1 cpumask->row[5]=0
process 4412 row_last_cpu=-1 cpumask->row[6]=0
process 4412 row_last_cpu=-1 cpumask->row[5]=0
process 4280 row_last_cpu=-1 cpumask->row[2]=0
process 3180 row_last_cpu=-1 cpumask->row[4]=0
process 1176 row_last_cpu=9 cpumask->row[0]=512
process 4412 row_last_cpu=-1 cpumask->row[4]=0
process 4280 row_last_cpu=-1 cpumask->row[1]=0
process 3180 row_last_cpu=-1 cpumask->row[3]=0
process 4412 row_last_cpu=-1 cpumask->row[3]=0
process 1176 last set cpu in mask is at index 9
process 4280 row_last_cpu=19 cpumask->row[0]=524288
process 4412 row_last_cpu=-1 cpumask->row[2]=0
process 3180 row_last_cpu=-1 cpumask->row[2]=0
process 4280 last set cpu in mask is at index 19
process 1176 Returning group=0 group_mask=512
process 4280 Returning group=0 group_mask=524288
process 4412 row_last_cpu=-1 cpumask->row[1]=0
process 3180 row_last_cpu=-1 cpumask->row[1]=0
process 4412 row_last_cpu=20 cpumask->row[0]=1048576
process 3180 row_last_cpu=16 cpumask->row[0]=65536
process 4412 last set cpu in mask is at index 20
process 3180 last set cpu in mask is at index 16
process 4412 Returning group=0 group_mask=1048576
process 3180 Returning group=0 group_mask=65536
fio_setaffinity: failed to set thread affinity (pid 5948, group 0, mask 20000, GetLastError=87)
process 5364 row_first_cpu=21 cpumask->row[0]=2097152
process 5364 first set cpu in mask is at index 21
process 5364 group=0 group_start=0 group_size=64 search_cpu=21
process 5364 row_last_cpu=-1 cpumask->row[7]=0
process 5364 row_last_cpu=-1 cpumask->row[6]=0
process 5364 row_last_cpu=-1 cpumask->row[5]=0
process 5364 row_last_cpu=-1 cpumask->row[4]=0
process 5364 row_last_cpu=-1 cpumask->row[3]=0
process 5364 row_last_cpu=-1 cpumask->row[2]=0
process 5364 row_last_cpu=-1 cpumask->row[1]=0
process 5364 row_last_cpu=21 cpumask->row[0]=2097152
process 5364 last set cpu in mask is at index 21
process 5364 Returning group=0 group_mask=2097152
process 3236 row_first_cpu=22 cpumask->row[0]=4194304
process 3236 first set cpu in mask is at index 22
process 3236 group=0 group_start=0 group_size=64 search_cpu=22
process 3236 row_last_cpu=-1 cpumask->row[7]=0
process 3236 row_last_cpu=-1 cpumask->row[6]=0
process 3236 row_last_cpu=-1 cpumask->row[5]=0
process 3236 row_last_cpu=-1 cpumask->row[4]=0
process 3236 row_last_cpu=-1 cpumask->row[3]=0
process 3236 row_last_cpu=-1 cpumask->row[2]=0
process 3236 row_last_cpu=-1 cpumask->row[1]=0
process 3236 row_last_cpu=22 cpumask->row[0]=4194304
process 3236 last set cpu in mask is at index 22
fio_setaffinity: failed to set thread affinity (pid 3076, group 0, mask 40000, GetLastError=87)
process 3236 Returning group=0 group_mask=4194304
process 4340 row_first_cpu=59 cpumask->row[0]=0
process 4340 first set cpu in mask is at index 59
process 4340 group=0 group_start=0 group_size=64 search_cpu=59
process 4340 row_last_cpu=-1 cpumask->row[7]=0
process 4340 row_last_cpu=-1 cpumask->row[6]=0
process 4340 row_last_cpu=-1 cpumask->row[5]=0
process 4340 row_last_cpu=-1 cpumask->row[4]=0
process 4340 row_last_cpu=-1 cpumask->row[3]=0
process 4340 row_last_cpu=-1 cpumask->row[2]=0
process 4340 row_last_cpu=-1 cpumask->row[1]=0
process 4340 row_last_cpu=59 cpumask->row[0]=0
process 4340 last set cpu in mask is at index 59
process 4340 Returning group=0 group_mask=0
process 1084 row_first_cpu=25 cpumask->row[0]=33554432
process 1084 first set cpu in mask is at index 25
process 1084 group=0 group_start=0 group_size=64 search_cpu=25
process 1084 row_last_cpu=-1 cpumask->row[7]=0
fio_setaffinity: failed to set thread affinity (pid 4280, group 0, mask 80000, GetLastError=87)
process 1084 row_last_cpu=-1 cpumask->row[6]=0
process 1128 row_first_cpu=26 cpumask->row[0]=67108864
process 1084 row_last_cpu=-1 cpumask->row[5]=0
process 1128 first set cpu in mask is at index 26
process 1084 row_last_cpu=-1 cpumask->row[4]=0
process 1084 row_last_cpu=-1 cpumask->row[3]=0
process 1084 row_last_cpu=-1 cpumask->row[2]=0
process 1128 group=0 group_start=0 group_size=64 search_cpu=26
process 1084 row_last_cpu=-1 cpumask->row[1]=0
process 1128 row_last_cpu=-1 cpumask->row[7]=0
process 1084 row_last_cpu=25 cpumask->row[0]=33554432
process 1128 row_last_cpu=-1 cpumask->row[6]=0
process 1084 last set cpu in mask is at index 25
process 1128 row_last_cpu=-1 cpumask->row[5]=0
process 1084 Returning group=0 group_mask=33554432
process 1128 row_last_cpu=-1 cpumask->row[4]=0
process 1128 row_last_cpu=-1 cpumask->row[3]=0
process 1128 row_last_cpu=-1 cpumask->row[2]=0
process 1128 row_last_cpu=-1 cpumask->row[1]=0
process 1128 row_last_cpu=26 cpumask->row[0]=67108864
process 1128 last set cpu in mask is at index 26
process 1128 Returning group=0 group_mask=67108864
process 4852 row_first_cpu=27 cpumask->row[0]=134217728
process 4852 first set cpu in mask is at index 27
process 4852 group=0 group_start=0 group_size=64 search_cpu=27
process 4852 row_last_cpu=-1 cpumask->row[7]=0
process 4852 row_last_cpu=-1 cpumask->row[6]=0
process 4852 row_last_cpu=-1 cpumask->row[5]=0
process 4852 row_last_cpu=-1 cpumask->row[4]=0
process 4852 row_last_cpu=-1 cpumask->row[3]=0
process 4852 row_last_cpu=-1 cpumask->row[2]=0
process 4852 row_last_cpu=-1 cpumask->row[1]=0
process 4852 row_last_cpu=27 cpumask->row[0]=134217728
process 4852 last set cpu in mask is at index 27
process 4852 Returning group=0 group_mask=134217728
fio_setaffinity: failed to set thread affinity (pid 4412, group 0, mask 100000, GetLastError=87)
process 2332 row_first_cpu=28 cpumask->row[0]=268435456
process 2332 first set cpu in mask is at index 28
process 2332 group=0 group_start=0 group_size=64 search_cpu=28
process 2332 row_last_cpu=-1 cpumask->row[7]=0
process 2332 row_last_cpu=-1 cpumask->row[6]=0
process 2332 row_last_cpu=-1 cpumask->row[5]=0
process 2332 row_last_cpu=-1 cpumask->row[4]=0
process 2332 row_last_cpu=-1 cpumask->row[3]=0
process 2332 row_last_cpu=-1 cpumask->row[2]=0
process 2332 row_last_cpu=-1 cpumask->row[1]=0
process 2332 row_last_cpu=28 cpumask->row[0]=268435456
process 2332 last set cpu in mask is at index 28
process 2332 Returning group=0 group_mask=268435456
process 5156 row_first_cpu=29 cpumask->row[0]=536870912
process 5156 first set cpu in mask is at index 29
process 5156 group=0 group_start=0 group_size=64 search_cpu=29
process 5156 row_last_cpu=-1 cpumask->row[7]=0
process 5156 row_last_cpu=-1 cpumask->row[6]=0
process 5156 row_last_cpu=-1 cpumask->row[5]=0
process 5156 row_last_cpu=-1 cpumask->row[4]=0
fio_setaffinity: failed to set thread affinity (pid 4852, group 0, mask 8000000, GetLastError=87)
process 5156 row_last_cpu=-1 cpumask->row[3]=0
process 5156 row_last_cpu=-1 cpumask->row[2]=0
process 5156 row_last_cpu=-1 cpumask->row[1]=0
process 5156 row_last_cpu=29 cpumask->row[0]=536870912
process 5156 last set cpu in mask is at index 29
process 5156 Returning group=0 group_mask=536870912
process 548 row_first_cpu=30 cpumask->row[0]=1073741824
process 548 first set cpu in mask is at index 30
process 548 group=0 group_start=0 group_size=64 search_cpu=30
process 548 row_last_cpu=-1 cpumask->row[7]=0
process 548 row_last_cpu=-1 cpumask->row[6]=0
process 548 row_last_cpu=-1 cpumask->row[5]=0
process 548 row_last_cpu=-1 cpumask->row[4]=0
process 548 row_last_cpu=-1 cpumask->row[3]=0
process 548 row_last_cpu=-1 cpumask->row[2]=0
process 548 row_last_cpu=-1 cpumask->row[1]=0
process 548 row_last_cpu=30 cpumask->row[0]=1073741824
process 548 last set cpu in mask is at index 30
process 548 Returning group=0 group_mask=1073741824
process 6020 row_first_cpu=31 cpumask->row[0]=2147483648
fio_setaffinity: failed to set thread affinity (pid 2332, group 0, mask 10000000, GetLastError=87)
process 6020 first set cpu in mask is at index 31
process 6020 group=0 group_start=0 group_size=64 search_cpu=31
process 6020 row_last_cpu=-1 cpumask->row[7]=0
process 6020 row_last_cpu=-1 cpumask->row[6]=0
process 6020 row_last_cpu=-1 cpumask->row[5]=0
process 6020 row_last_cpu=-1 cpumask->row[4]=0
process 6020 row_last_cpu=-1 cpumask->row[3]=0
process 1012 row_first_cpu=32 cpumask->row[0]=0
process 6020 row_last_cpu=-1 cpumask->row[2]=0
process 6020 row_last_cpu=-1 cpumask->row[1]=0
process 1012 first set cpu in mask is at index 32
process 6020 row_last_cpu=31 cpumask->row[0]=2147483648
process 6020 last set cpu in mask is at index 31
process 6020 Returning group=0 group_mask=2147483648
process 1012 group=0 group_start=0 group_size=64 search_cpu=32
process 1012 row_last_cpu=-1 cpumask->row[7]=0
process 1012 row_last_cpu=-1 cpumask->row[6]=0
process 1012 row_last_cpu=-1 cpumask->row[5]=0
process 1012 row_last_cpu=-1 cpumask->row[4]=0
process 1012 row_last_cpu=-1 cpumask->row[3]=0
process 1012 row_last_cpu=-1 cpumask->row[2]=0
process 1012 row_last_cpu=-1 cpumask->row[1]=0
process 1012 row_last_cpu=32 cpumask->row[0]=0
process 1012 last set cpu in mask is at index 32
process 1012 Returning group=0 group_mask=0
process 2272 row_first_cpu=33 cpumask->row[0]=0
process 2272 first set cpu in mask is at index 33
process 2272 group=0 group_start=0 group_size=64 search_cpu=33
process 2272 row_last_cpu=-1 cpumask->row[7]=0
process 2272 row_last_cpu=-1 cpumask->row[6]=0
process 2272 row_last_cpu=-1 cpumask->row[5]=0
process 2272 row_last_cpu=-1 cpumask->row[4]=0
process 2272 row_last_cpu=-1 cpumask->row[3]=0
process 2272 row_last_cpu=-1 cpumask->row[2]=0
process 2272 row_last_cpu=-1 cpumask->row[1]=0
process 2272 row_last_cpu=33 cpumask->row[0]=0
process 2272 last set cpu in mask is at index 33
process 2272 Returning group=0 group_mask=0
process 5508 row_first_cpu=34 cpumask->row[0]=0
process 5508 first set cpu in mask is at index 34
process 5508 group=0 group_start=0 group_size=64 search_cpu=34
process 5508 row_last_cpu=-1 cpumask->row[7]=0
process 5508 row_last_cpu=-1 cpumask->row[6]=0
process 5508 row_last_cpu=-1 cpumask->row[5]=0
process 5508 row_last_cpu=-1 cpumask->row[4]=0
process 5508 row_last_cpu=-1 cpumask->row[3]=0
process 5508 row_last_cpu=-1 cpumask->row[2]=0
fio_setaffinity: failed to set thread affinity (pid 5364, group 0, mask 200000, GetLastError=87)
process 5508 row_last_cpu=-1 cpumask->row[1]=0
process 5508 row_last_cpu=34 cpumask->row[0]=0
process 5508 last set cpu in mask is at index 34
process 5508 Returning group=0 group_mask=0
process 1904 row_first_cpu=35 cpumask->row[0]=0
process 1904 first set cpu in mask is at index 35
process 1904 group=0 group_start=0 group_size=64 search_cpu=35
process 1904 row_last_cpu=-1 cpumask->row[7]=0
process 1904 row_last_cpu=-1 cpumask->row[6]=0
process 1904 row_last_cpu=-1 cpumask->row[5]=0
process 1904 row_last_cpu=-1 cpumask->row[4]=0
process 1904 row_last_cpu=-1 cpumask->row[3]=0
process 1904 row_last_cpu=-1 cpumask->row[2]=0
process 1904 row_last_cpu=-1 cpumask->row[1]=0
process 1904 row_last_cpu=35 cpumask->row[0]=0
process 1904 last set cpu in mask is at index 35
process 1904 Returning group=0 group_mask=0
process 5356 row_first_cpu=36 cpumask->row[0]=0
process 5356 first set cpu in mask is at index 36
process 5356 group=0 group_start=0 group_size=64 search_cpu=36
process 5356 row_last_cpu=-1 cpumask->row[7]=0
process 5356 row_last_cpu=-1 cpumask->row[6]=0
process 5356 row_last_cpu=-1 cpumask->row[5]=0
process 5356 row_last_cpu=-1 cpumask->row[4]=0
process 5356 row_last_cpu=-1 cpumask->row[3]=0
process 5356 row_last_cpu=-1 cpumask->row[2]=0
process 5356 row_last_cpu=-1 cpumask->row[1]=0
fio_setaffinity: failed to set thread affinity (pid 2272, group 0, mask 200000000, GetLastError=87)
process 5356 row_last_cpu=36 cpumask->row[0]=0
process 5356 last set cpu in mask is at index 36
process 2676 row_first_cpu=37 cpumask->row[0]=0
process 5356 Returning group=0 group_mask=0
process 2676 first set cpu in mask is at index 37
process 2676 group=0 group_start=0 group_size=64 search_cpu=37
process 2676 row_last_cpu=-1 cpumask->row[7]=0
process 2676 row_last_cpu=-1 cpumask->row[6]=0
process 2676 row_last_cpu=-1 cpumask->row[5]=0
process 2676 row_last_cpu=-1 cpumask->row[4]=0
process 2676 row_last_cpu=-1 cpumask->row[3]=0
process 2676 row_last_cpu=-1 cpumask->row[2]=0
process 2676 row_last_cpu=-1 cpumask->row[1]=0
process 2676 row_last_cpu=37 cpumask->row[0]=0
process 2676 last set cpu in mask is at index 37
process 2676 Returning group=0 group_mask=0
process 4104 row_first_cpu=38 cpumask->row[0]=0
process 4104 first set cpu in mask is at index 38
process 4104 group=0 group_start=0 group_size=64 search_cpu=38
process 4104 row_last_cpu=-1 cpumask->row[7]=0
process 4104 row_last_cpu=-1 cpumask->row[6]=0
process 4104 row_last_cpu=-1 cpumask->row[5]=0
process 4104 row_last_cpu=-1 cpumask->row[4]=0
process 4104 row_last_cpu=-1 cpumask->row[3]=0
clock setaffinity failed: No error
process 4104 row_last_cpu=-1 cpumask->row[2]=0
process 4104 row_last_cpu=-1 cpumask->row[1]=0
process 4104 row_last_cpu=38 cpumask->row[0]=0
process 4104 last set cpu in mask is at index 38
process 4104 Returning group=0 group_mask=0
process 2660 row_first_cpu=39 cpumask->row[0]=0
process 2660 first set cpu in mask is at index 39
process 2660 group=0 group_start=0 group_size=64 search_cpu=39
process 2660 row_last_cpu=-1 cpumask->row[7]=0
process 2660 row_last_cpu=-1 cpumask->row[6]=0
process 2660 row_last_cpu=-1 cpumask->row[5]=0
process 2660 row_last_cpu=-1 cpumask->row[4]=0
process 2660 row_last_cpu=-1 cpumask->row[3]=0
process 2660 row_last_cpu=-1 cpumask->row[2]=0
process 2660 row_last_cpu=-1 cpumask->row[1]=0
process 2660 row_last_cpu=39 cpumask->row[0]=0
process 2660 last set cpu in mask is at index 39
process 2660 Returning group=0 group_mask=0
process 5136 row_first_cpu=40 cpumask->row[0]=0
process 5136 first set cpu in mask is at index 40
process 5136 group=0 group_start=0 group_size=64 search_cpu=40
process 5136 row_last_cpu=-1 cpumask->row[7]=0
process 5136 row_last_cpu=-1 cpumask->row[6]=0
process 5136 row_last_cpu=-1 cpumask->row[5]=0
process 5136 row_last_cpu=-1 cpumask->row[4]=0
process 5136 row_last_cpu=-1 cpumask->row[3]=0
process 5136 row_last_cpu=-1 cpumask->row[2]=0
process 5136 row_last_cpu=-1 cpumask->row[1]=0
process 5136 row_last_cpu=40 cpumask->row[0]=0
process 5136 last set cpu in mask is at index 40
process 5136 Returning group=0 group_mask=0
process 5352 row_first_cpu=41 cpumask->row[0]=0
process 5352 first set cpu in mask is at index 41
process 5352 group=0 group_start=0 group_size=64 search_cpu=41
process 5352 row_last_cpu=-1 cpumask->row[7]=0
process 5352 row_last_cpu=-1 cpumask->row[6]=0
process 5352 row_last_cpu=-1 cpumask->row[5]=0
process 5352 row_last_cpu=-1 cpumask->row[4]=0
process 5352 row_last_cpu=-1 cpumask->row[3]=0
process 5352 row_last_cpu=-1 cpumask->row[2]=0
process 5352 row_last_cpu=-1 cpumask->row[1]=0
process 5352 row_last_cpu=41 cpumask->row[0]=0
fio_setaffinity: failed to set thread affinity (pid 4340, group 0, mask 800000000000000, GetLastError=87)
process 5352 last set cpu in mask is at index 41
process 5352 Returning group=0 group_mask=0
process 5916 row_first_cpu=42 cpumask->row[0]=0
process 5916 first set cpu in mask is at index 42
process 5916 group=0 group_start=0 group_size=64 search_cpu=42
process 5916 row_last_cpu=-1 cpumask->row[7]=0
process 5916 row_last_cpu=-1 cpumask->row[6]=0
process 5916 row_last_cpu=-1 cpumask->row[5]=0
process 5916 row_last_cpu=-1 cpumask->row[4]=0
process 5916 row_last_cpu=-1 cpumask->row[3]=0
process 5916 row_last_cpu=-1 cpumask->row[2]=0
process 1492 row_first_cpu=43 cpumask->row[0]=0
fio_setaffinity: failed to set thread affinity (pid 5136, group 0, mask 10000000000, GetLastError=87)
process 1492 first set cpu in mask is at index 43
process 5916 row_last_cpu=-1 cpumask->row[1]=0
process 5916 row_last_cpu=42 cpumask->row[0]=0
process 1492 group=0 group_start=0 group_size=64 search_cpu=43
process 1492 row_last_cpu=-1 cpumask->row[7]=0
process 5916 last set cpu in mask is at index 42
process 1492 row_last_cpu=-1 cpumask->row[6]=0
process 5916 Returning group=0 group_mask=0
process 1492 row_last_cpu=-1 cpumask->row[5]=0
process 1492 row_last_cpu=-1 cpumask->row[4]=0
process 4948 row_first_cpu=44 cpumask->row[0]=0
process 1492 row_last_cpu=-1 cpumask->row[3]=0
process 1492 row_last_cpu=-1 cpumask->row[2]=0
process 4948 first set cpu in mask is at index 44
process 1492 row_last_cpu=-1 cpumask->row[1]=0
process 1492 row_last_cpu=43 cpumask->row[0]=0
process 1492 last set cpu in mask is at index 43
process 1492 Returning group=0 group_mask=0
process 4948 group=0 group_start=0 group_size=64 search_cpu=44
process 4948 row_last_cpu=-1 cpumask->row[7]=0
process 4948 row_last_cpu=-1 cpumask->row[6]=0
process 4948 row_last_cpu=-1 cpumask->row[5]=0
process 4948 row_last_cpu=-1 cpumask->row[4]=0
process 4948 row_last_cpu=-1 cpumask->row[3]=0
process 4948 row_last_cpu=-1 cpumask->row[2]=0
process 4948 row_last_cpu=-1 cpumask->row[1]=0
process 4948 row_last_cpu=44 cpumask->row[0]=0
process 4948 last set cpu in mask is at index 44
process 4948 Returning group=0 group_mask=0
process 4120 row_first_cpu=45 cpumask->row[0]=0
process 4120 first set cpu in mask is at index 45
process 4120 group=0 group_start=0 group_size=64 search_cpu=45
process 4120 row_last_cpu=-1 cpumask->row[7]=0
process 4120 row_last_cpu=-1 cpumask->row[6]=0
process 4120 row_last_cpu=-1 cpumask->row[5]=0
process 4120 row_last_cpu=-1 cpumask->row[4]=0
process 4120 row_last_cpu=-1 cpumask->row[3]=0
process 4120 row_last_cpu=-1 cpumask->row[2]=0
fio_setaffinity: failed to set thread affinity (pid 1084, group 0, mask 2000000, GetLastError=87)
process 4120 row_last_cpu=-1 cpumask->row[1]=0
process 4120 row_last_cpu=45 cpumask->row[0]=0
process 4120 last set cpu in mask is at index 45
process 4120 Returning group=0 group_mask=0
process 1924 row_first_cpu=46 cpumask->row[0]=0
process 1924 first set cpu in mask is at index 46
process 1924 group=0 group_start=0 group_size=64 search_cpu=46
process 1924 row_last_cpu=-1 cpumask->row[7]=0
process 1924 row_last_cpu=-1 cpumask->row[6]=0
process 1924 row_last_cpu=-1 cpumask->row[5]=0
process 1924 row_last_cpu=-1 cpumask->row[4]=0
process 1924 row_last_cpu=-1 cpumask->row[3]=0
process 1924 row_last_cpu=-1 cpumask->row[2]=0
process 1924 row_last_cpu=-1 cpumask->row[1]=0
process 1924 row_last_cpu=46 cpumask->row[0]=0
process 1924 last set cpu in mask is at index 46
process 1924 Returning group=0 group_mask=0
process 5848 row_first_cpu=47 cpumask->row[0]=0
process 5848 first set cpu in mask is at index 47
process 5848 group=0 group_start=0 group_size=64 search_cpu=47
process 5848 row_last_cpu=-1 cpumask->row[7]=0
process 5848 row_last_cpu=-1 cpumask->row[6]=0
process 5848 row_last_cpu=-1 cpumask->row[5]=0
process 5848 row_last_cpu=-1 cpumask->row[4]=0
process 5848 row_last_cpu=-1 cpumask->row[3]=0
process 5848 row_last_cpu=-1 cpumask->row[2]=0
process 5848 row_last_cpu=-1 cpumask->row[1]=0
process 5848 row_last_cpu=47 cpumask->row[0]=0
process 5848 last set cpu in mask is at index 47
process 5848 Returning group=0 group_mask=0
process 4156 row_first_cpu=48 cpumask->row[0]=0
process 4156 first set cpu in mask is at index 48
process 4156 group=0 group_start=0 group_size=64 search_cpu=48
process 4156 row_last_cpu=-1 cpumask->row[7]=0
process 4156 row_last_cpu=-1 cpumask->row[6]=0
process 4156 row_last_cpu=-1 cpumask->row[5]=0
process 4156 row_last_cpu=-1 cpumask->row[4]=0
fio_setaffinity: failed to set thread affinity (pid 1128, group 0, mask 4000000, GetLastError=87)
process 4156 row_last_cpu=-1 cpumask->row[3]=0
process 2096 row_first_cpu=49 cpumask->row[0]=0
process 4156 row_last_cpu=-1 cpumask->row[2]=0
process 4156 row_last_cpu=-1 cpumask->row[1]=0
process 4156 row_last_cpu=48 cpumask->row[0]=0
process 2096 first set cpu in mask is at index 49
process 4156 last set cpu in mask is at index 48
process 4156 Returning group=0 group_mask=0
process 2096 group=0 group_start=0 group_size=64 search_cpu=49
process 2096 row_last_cpu=-1 cpumask->row[7]=0
process 2096 row_last_cpu=-1 cpumask->row[6]=0
process 2096 row_last_cpu=-1 cpumask->row[5]=0
process 2096 row_last_cpu=-1 cpumask->row[4]=0
process 2096 row_last_cpu=-1 cpumask->row[3]=0
process 2096 row_last_cpu=-1 cpumask->row[2]=0
process 2096 row_last_cpu=-1 cpumask->row[1]=0
process 2096 row_last_cpu=49 cpumask->row[0]=0
process 2096 last set cpu in mask is at index 49
process 2096 Returning group=0 group_mask=0
process 6084 row_first_cpu=50 cpumask->row[0]=0
process 6084 first set cpu in mask is at index 50
process 6084 group=0 group_start=0 group_size=64 search_cpu=50
process 6084 row_last_cpu=-1 cpumask->row[7]=0
process 6084 row_last_cpu=-1 cpumask->row[6]=0
process 6084 row_last_cpu=-1 cpumask->row[5]=0
process 6084 row_last_cpu=-1 cpumask->row[4]=0
process 6084 row_last_cpu=-1 cpumask->row[3]=0
process 6084 row_last_cpu=-1 cpumask->row[2]=0
process 6084 row_last_cpu=-1 cpumask->row[1]=0
process 2772 row_first_cpu=51 cpumask->row[0]=0
process 6084 row_last_cpu=50 cpumask->row[0]=0
fio_setaffinity: failed to set thread affinity (pid 5848, group 0, mask 800000000000, GetLastError=87)
process 2772 first set cpu in mask is at index 51
process 6084 last set cpu in mask is at index 50
process 6084 Returning group=0 group_mask=0
process 2772 group=0 group_start=0 group_size=64 search_cpu=51
process 2772 row_last_cpu=-1 cpumask->row[7]=0
process 3964 row_first_cpu=52 cpumask->row[0]=0
process 2772 row_last_cpu=-1 cpumask->row[6]=0
process 3964 first set cpu in mask is at index 52
process 2772 row_last_cpu=-1 cpumask->row[5]=0
process 2772 row_last_cpu=-1 cpumask->row[4]=0
process 3964 group=0 group_start=0 group_size=64 search_cpu=52
process 3964 row_last_cpu=-1 cpumask->row[7]=0
process 2772 row_last_cpu=-1 cpumask->row[3]=0
process 3964 row_last_cpu=-1 cpumask->row[6]=0
process 3964 row_last_cpu=-1 cpumask->row[5]=0
process 2772 row_last_cpu=-1 cpumask->row[2]=0
process 3964 row_last_cpu=-1 cpumask->row[4]=0
process 3964 row_last_cpu=-1 cpumask->row[3]=0
process 2772 row_last_cpu=-1 cpumask->row[1]=0
process 3964 row_last_cpu=-1 cpumask->row[2]=0
process 3964 row_last_cpu=-1 cpumask->row[1]=0
process 2772 row_last_cpu=51 cpumask->row[0]=0
process 3964 row_last_cpu=52 cpumask->row[0]=0
process 2772 last set cpu in mask is at index 51
process 3964 last set cpu in mask is at index 52
process 2772 Returning group=0 group_mask=0
process 3964 Returning group=0 group_mask=0
process 4384 row_first_cpu=53 cpumask->row[0]=0
process 4384 first set cpu in mask is at index 53
process 4384 group=0 group_start=0 group_size=64 search_cpu=53
process 4384 row_last_cpu=-1 cpumask->row[7]=0
process 4384 row_last_cpu=-1 cpumask->row[6]=0
process 4384 row_last_cpu=-1 cpumask->row[5]=0
process 4384 row_last_cpu=-1 cpumask->row[4]=0
process 4384 row_last_cpu=-1 cpumask->row[3]=0
clock setaffinity failed: No error
process 4384 row_last_cpu=-1 cpumask->row[2]=0
process 4384 row_last_cpu=-1 cpumask->row[1]=0
process 4384 row_last_cpu=53 cpumask->row[0]=0
process 4384 last set cpu in mask is at index 53
process 4384 Returning group=0 group_mask=0
process 5244 row_first_cpu=54 cpumask->row[0]=0
process 5244 first set cpu in mask is at index 54
process 5244 group=0 group_start=0 group_size=64 search_cpu=54
process 5244 row_last_cpu=-1 cpumask->row[7]=0
process 5244 row_last_cpu=-1 cpumask->row[6]=0
process 5244 row_last_cpu=-1 cpumask->row[5]=0
process 5244 row_last_cpu=-1 cpumask->row[4]=0
process 5244 row_last_cpu=-1 cpumask->row[3]=0
process 5244 row_last_cpu=-1 cpumask->row[2]=0
process 5244 row_last_cpu=-1 cpumask->row[1]=0
process 5244 row_last_cpu=54 cpumask->row[0]=0
process 5244 last set cpu in mask is at index 54
process 5244 Returning group=0 group_mask=0
process 5076 row_first_cpu=55 cpumask->row[0]=0
process 5076 first set cpu in mask is at index 55
process 5076 group=0 group_start=0 group_size=64 search_cpu=55
process 5076 row_last_cpu=-1 cpumask->row[7]=0
process 5076 row_last_cpu=-1 cpumask->row[6]=0
process 5076 row_last_cpu=-1 cpumask->row[5]=0
process 5076 row_last_cpu=-1 cpumask->row[4]=0
process 5076 row_last_cpu=-1 cpumask->row[3]=0
process 5076 row_last_cpu=-1 cpumask->row[2]=0
process 5076 row_last_cpu=-1 cpumask->row[1]=0
clock setaffinity failed: No error
process 5076 row_last_cpu=55 cpumask->row[0]=0
process 5076 last set cpu in mask is at index 55
process 5076 Returning group=0 group_mask=0
process 5344 row_first_cpu=56 cpumask->row[0]=0
process 5344 first set cpu in mask is at index 56
process 5344 group=0 group_start=0 group_size=64 search_cpu=56
process 5344 row_last_cpu=-1 cpumask->row[7]=0
process 5344 row_last_cpu=-1 cpumask->row[6]=0
process 5344 row_last_cpu=-1 cpumask->row[5]=0
process 5344 row_last_cpu=-1 cpumask->row[4]=0
process 5344 row_last_cpu=-1 cpumask->row[3]=0
process 5344 row_last_cpu=-1 cpumask->row[2]=0
process 5344 row_last_cpu=-1 cpumask->row[1]=0
process 5344 row_last_cpu=56 cpumask->row[0]=0
process 5344 last set cpu in mask is at index 56
process 5344 Returning group=0 group_mask=0
process 3204 row_first_cpu=57 cpumask->row[0]=0
process 3204 first set cpu in mask is at index 57
process 3204 group=0 group_start=0 group_size=64 search_cpu=57
process 3204 row_last_cpu=-1 cpumask->row[7]=0
process 3204 row_last_cpu=-1 cpumask->row[6]=0
process 3204 row_last_cpu=-1 cpumask->row[5]=0
process 3204 row_last_cpu=-1 cpumask->row[4]=0
process 3204 row_last_cpu=-1 cpumask->row[3]=0
process 3204 row_last_cpu=-1 cpumask->row[2]=0
process 3204 row_last_cpu=-1 cpumask->row[1]=0
process 3204 row_last_cpu=57 cpumask->row[0]=0
process 3204 last set cpu in mask is at index 57
process 3204 Returning group=0 group_mask=0
clock setaffinity failed: No error
process 3848 row_first_cpu=58 cpumask->row[0]=0
process 3848 first set cpu in mask is at index 58
process 3848 group=0 group_start=0 group_size=64 search_cpu=58
process 3848 row_last_cpu=-1 cpumask->row[7]=0
process 3848 row_last_cpu=-1 cpumask->row[6]=0
process 3848 row_last_cpu=-1 cpumask->row[5]=0
process 3848 row_last_cpu=-1 cpumask->row[4]=0
process 3848 row_last_cpu=-1 cpumask->row[3]=0
process 3848 row_last_cpu=-1 cpumask->row[2]=0
process 3848 row_last_cpu=-1 cpumask->row[1]=0
process 3848 row_last_cpu=58 cpumask->row[0]=0
process 3848 last set cpu in mask is at index 58
process 3848 Returning group=0 group_mask=0
process 440 row_first_cpu=23 cpumask->row[0]=8388608
process 440 first set cpu in mask is at index 23
process 440 group=0 group_start=0 group_size=64 search_cpu=23
process 440 row_last_cpu=-1 cpumask->row[7]=0
process 440 row_last_cpu=-1 cpumask->row[6]=0
process 440 row_last_cpu=-1 cpumask->row[5]=0
process 440 row_last_cpu=-1 cpumask->row[4]=0
process 440 row_last_cpu=-1 cpumask->row[3]=0
process 440 row_last_cpu=-1 cpumask->row[2]=0
process 440 row_last_cpu=-1 cpumask->row[1]=0
process 440 row_last_cpu=23 cpumask->row[0]=8388608
process 440 last set cpu in mask is at index 23
fio_setaffinity: failed to set thread affinity (pid 5156, group 0, mask 20000000, GetLastError=87)
process 440 Returning group=0 group_mask=8388608
process 4420 row_first_cpu=60 cpumask->row[0]=0
process 4420 first set cpu in mask is at index 60
process 4420 group=0 group_start=0 group_size=64 search_cpu=60
process 4420 row_last_cpu=-1 cpumask->row[7]=0
process 4420 row_last_cpu=-1 cpumask->row[6]=0
process 4420 row_last_cpu=-1 cpumask->row[5]=0
process 4420 row_last_cpu=-1 cpumask->row[4]=0
process 4420 row_last_cpu=-1 cpumask->row[3]=0
process 4420 row_last_cpu=-1 cpumask->row[2]=0
process 4420 row_last_cpu=-1 cpumask->row[1]=0
process 4420 row_last_cpu=60 cpumask->row[0]=0
process 4420 last set cpu in mask is at index 60
process 4420 Returning group=0 group_mask=0
process 5164 row_first_cpu=61 cpumask->row[0]=0
process 5164 first set cpu in mask is at index 61
process 5164 group=0 group_start=0 group_size=64 search_cpu=61
process 5164 row_last_cpu=-1 cpumask->row[7]=0
process 5164 row_last_cpu=-1 cpumask->row[6]=0
process 5164 row_last_cpu=-1 cpumask->row[5]=0
process 5164 row_last_cpu=-1 cpumask->row[4]=0
process 5164 row_last_cpu=-1 cpumask->row[3]=0
process 5164 row_last_cpu=-1 cpumask->row[2]=0
process 5164 row_last_cpu=-1 cpumask->row[1]=0
fio_setaffinity: failed to set thread affinity (pid 3848, group 0, mask 400000000000000, GetLastError=87)
process 5164 row_last_cpu=61 cpumask->row[0]=0
process 5164 last set cpu in mask is at index 61
process 5164 Returning group=0 group_mask=0
process 5176 row_first_cpu=62 cpumask->row[0]=0
process 5176 first set cpu in mask is at index 62
process 5176 group=0 group_start=0 group_size=64 search_cpu=62
process 5176 row_last_cpu=-1 cpumask->row[7]=0
process 5176 row_last_cpu=-1 cpumask->row[6]=0
process 5176 row_last_cpu=-1 cpumask->row[5]=0
process 5176 row_last_cpu=-1 cpumask->row[4]=0
process 5176 row_last_cpu=-1 cpumask->row[3]=0
process 5176 row_last_cpu=-1 cpumask->row[2]=0
process 5176 row_last_cpu=-1 cpumask->row[1]=0
process 5176 row_last_cpu=62 cpumask->row[0]=0
process 5176 last set cpu in mask is at index 62
process 5176 Returning group=0 group_mask=0
process 5152 row_first_cpu=63 cpumask->row[0]=0
process 5152 first set cpu in mask is at index 63
process 5152 group=0 group_start=0 group_size=64 search_cpu=63
process 5152 row_last_cpu=-1 cpumask->row[7]=0
process 5152 row_last_cpu=-1 cpumask->row[6]=0
process 5152 row_last_cpu=-1 cpumask->row[5]=0
process 5152 row_last_cpu=-1 cpumask->row[4]=0
process 5152 row_last_cpu=-1 cpumask->row[3]=0
process 5152 row_last_cpu=-1 cpumask->row[2]=0
process 5152 row_last_cpu=-1 cpumask->row[1]=0
process 5152 row_last_cpu=63 cpumask->row[0]=0
process 5152 last set cpu in mask is at index 63
process 5152 Returning group=0 group_mask=0
clock setaffinity failed: No error
process 3852 row_first_cpu=24 cpumask->row[0]=16777216
process 3852 first set cpu in mask is at index 24
process 3852 group=0 group_start=0 group_size=64 search_cpu=24
process 3852 row_last_cpu=-1 cpumask->row[7]=0
process 3852 row_last_cpu=-1 cpumask->row[6]=0
process 3852 row_last_cpu=-1 cpumask->row[5]=0
process 3852 row_last_cpu=-1 cpumask->row[4]=0
process 3852 row_last_cpu=-1 cpumask->row[3]=0
process 3852 row_last_cpu=-1 cpumask->row[2]=0
process 3852 row_last_cpu=-1 cpumask->row[1]=0
process 3852 row_last_cpu=24 cpumask->row[0]=16777216
process 3852 last set cpu in mask is at index 24
process 3852 Returning group=0 group_mask=16777216
fio_setaffinity: failed to set thread affinity (pid 5152, group 0, mask 8000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1012, group 0, mask 100000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 3236, group 0, mask 400000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 5508, group 0, mask 400000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1904, group 0, mask 800000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 5356, group 0, mask 1000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 2676, group 0, mask 2000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 4104, group 0, mask 4000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 2660, group 0, mask 8000000000, GetLastError=87)
clock setaffinity failed: No error
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 5352, group 0, mask 20000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 5916, group 0, mask 40000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1492, group 0, mask 80000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 4948, group 0, mask 100000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 4120, group 0, mask 200000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 1924, group 0, mask 400000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 3180, group 0, mask 10000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 4156, group 0, mask 1000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 2096, group 0, mask 2000000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 6084, group 0, mask 4000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 2772, group 0, mask 8000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 3964, group 0, mask 10000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 4384, group 0, mask 20000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 5244, group 0, mask 40000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 5076, group 0, mask 80000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 5344, group 0, mask 100000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 3204, group 0, mask 200000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 548, group 0, mask 40000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 440, group 0, mask 800000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 4420, group 0, mask 1000000000000000, GetLastError=87)
clock setaffinity failed: No error
fio_setaffinity: failed to set thread affinity (pid 5164, group 0, mask 2000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 5176, group 0, mask 4000000000000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 6020, group 0, mask 80000000, GetLastError=87)
fio_setaffinity: failed to set thread affinity (pid 3852, group 0, mask 1000000, GetLastError=87)
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
clock setaffinity failed: No error
Starting 1 thread
process 3176 pid=0: runstate NOT_CREATED -> SETTING_UP
process 3176 pid=0: runstate SETTING_UP -> NOT_CREATED
process 3176 pid=0: runstate NOT_CREATED -> CREATED
process 3176 will pthread_create
process 4324 jobs pid=4324 started
process 4324 pid=4324: runstate CREATED -> INITIALIZED
process 3176 pid=4324: runstate INITIALIZED -> RUNNING
process 4324 row_first_cpu=0 cpumask->row[0]=3
process 4324 first set cpu in mask is at index 0
process 4324 group=0 group_start=0 group_size=64 search_cpu=0
process 4324 row_last_cpu=-1 cpumask->row[7]=0
process 4324 row_last_cpu=-1 cpumask->row[6]=0
process 4324 row_last_cpu=-1 cpumask->row[5]=0
process 4324 row_last_cpu=-1 cpumask->row[4]=0
process 4324 row_last_cpu=-1 cpumask->row[3]=0
process 4324 row_last_cpu=-1 cpumask->row[2]=0
process 4324 row_last_cpu=-1 cpumask->row[1]=0
process 4324 row_last_cpu=1 cpumask->row[0]=3
process 4324 last set cpu in mask is at index 1
process 4324 Returning group=0 group_mask=3
process 4324 row_first_cpu=0 cpumask->row[0]=3
process 4324 first set cpu in mask is at index 0
process 4324 group=0 group_start=0 group_size=64 search_cpu=0
process 4324 row_last_cpu=-1 cpumask->row[7]=0
process 4324 row_last_cpu=-1 cpumask->row[6]=0
process 4324 row_last_cpu=-1 cpumask->row[5]=0
process 4324 row_last_cpu=-1 cpumask->row[4]=0
process 4324 row_last_cpu=-1 cpumask->row[3]=0
process 4324 row_last_cpu=-1 cpumask->row[2]=0
process 4324 row_last_cpu=-1 cpumask->row[1]=0
process 4324 row_last_cpu=1 cpumask->row[0]=3
process 4324 last set cpu in mask is at index 1
process 4324 Returning group=0 group_mask=3
process 4324 pid=4324: runstate RUNNING -> FINISHING
process 4324 pid=4324: runstate FINISHING -> EXITED
process 3176 pid=4324: runstate EXITED -> REAPED
process 3176 terminate group_id=-1
process 3176 setting terminate on test/4324
process 2204 pid=4324: runstate REAPED -> FINISHING
process 2204 pid=4324: runstate FINISHING -> REAPED
Regards,
Jeff
-----Original Message-----
From: Elliott, Robert (Persistent Memory) [mailto:elliott@hpe.com]
Sent: Monday, October 29, 2018 2:23 PM
To: 'Sitsofe Wheeler' <sitsofe@gmail.com>; Jeff Furlong <jeff.furlong@wdc.com>
Cc: fio <fio@vger.kernel.org>
Subject: RE: fio win affinity broken
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Sitsofe Wheeler
> Sent: Monday, October 29, 2018 3:12 PM
> Subject: Re: fio win affinity broken
>
> On Mon, 29 Oct 2018 at 20:04, Jeff Furlong <jeff.furlong@wdc.com> wrote:
> >
> > Back in
> > http://git.kernel.dk/cgit/fio/commit/?id=c479640d6208236744f0562b1e7
> > 9535eec290e2b several updates were made for Windows CPU affinity. However, using the cpus_allowed since fio 3.6 and latest git appears to throw error messages. Consider a job:
> >
> > fio --name=test --ioengine=windowsaio --direct=1 --rw=write
> > --overwrite=1 --
> > filename=\\.\PHYSICALDRIVE1 --runtime=1s --thread --cpus_allowed=0-1
> > --cpus_allowed_policy=shared
Usually split is a better choice than shared - do you really want threads hopping between CPUs?
> > fio_setaffinity: failed to set thread affinity (pid 2768, group 0,
> > mask 10000, GetLastError=87)
> > fio_setaffinity: failed to set thread affinity (pid 2372, group 0,
> > mask 20000, GetLastError=87) clock setaffinity failed: No error
Those are the clock threads, not the job threads. At startup, fio spawns threads on all CPUs to measure the clocks (fio_monotonic_clocktest).
A Windows difference from linux I mentioned last year was:
If you've constrained the CPU affinity outside fio, some of those will fail. In Windows,
something like
START /AFFINITY 0x55555555 fio ...
can cause half of the clock threads to fail.
In linux, processes and threads are not restricted to their parent's
affinity mask. Although they inherit by default, they may call
sched_setaffinity with any values.
I hoped to work on a patch to limit the clock threads to the CPUs within the parent affinity mask and the logical OR of all the CPUs that are going to be used by the jobs, but then lost access to the Windows system I was using and never went back to it.
> > It seems err 87 is ERROR_INVALID_PARAMETER, but any idea why
> > "GROUP_AFFINITY struct's Reserved
> members are not initialised to 0?"
That's just one possible reason for failure; more likely causes are an invalid Mask or Group for the system under test.
---
Robert Elliott, HPE Persistent Memory
^ permalink raw reply [flat|nested] 8+ messages in thread