* fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
@ 2015-12-29 8:29 Thierry BERTAUD
2015-12-29 14:30 ` Thierry BERTAUD
2015-12-30 17:08 ` Jens Axboe
0 siblings, 2 replies; 24+ messages in thread
From: Thierry BERTAUD @ 2015-12-29 8:29 UTC (permalink / raw)
To: fio@vger.kernel.org
Fio team,
During a important workload fio is hanging.
with iodepth=128 and numjobs=128 , i have the issue but not with both set at 64.
Environment is AIX 6.1 against SAN disk (hba card 8 Gbps)
Could you take a look at this?
Error message and script file are copy-pasted below.
# cat myfio.fio
[global]
ioengine=posixaio
invalidate=1
norandommap
refill_buffers
randrepeat=0
runtime=300
time_based
direct=1
group_reporting
[test]
stonewall
filename=/dev/${HDISK}
rw=${RWTYPE}
rwmixread=${RRATIO}
bs=${BS}
iodepth=${QD}
numjobs=${NJ}
write_bw_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_bw.results
write_iops_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_iops.results
write_lat_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_lat.results
#
Command:
# HDISK=hdisk3 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-2c2c93d/fio myfio.fio
PS: Output:
# ps -ef | grep fio | wc -l
130
# ps -ef | grep fio | grep -v grep | wc -l
130
# nmon
# ps -ef | grep fio | grep -v grep | wc -l
130
# ps -ef | grep fio | grep -v grep | wc -l
130
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 02:59:07 CET 2015
17
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 02:59:12 CET 2015
17
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:01:35 CET 2015
1
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:04:12 CET 2015
10
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:10:17 CET 2015
1
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:12:18 CET 2015
18
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:13:39 CET 2015
18
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:20:52 CET 2015
34
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:25:03 CET 2015
66
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 03:26:43 CET 2015
66
#
# date;ps -ef | grep fio | grep -v grep | wc -l
Tue Dec 29 09:15:05 CET 2015
2
#
Command output:
# HDISK=hdisk3 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-2c2c93d/fio myfio.fio
test: (g=0): rw=rw, bs=64K-64K/64K-64K/64K-64K, ioengine=posixaio, iodepth=128
...
fio-2.3
Starting 128 processes
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.0 iops] [eta 17m:00s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.12.9% done] [0KB/1991KB/0KB /s] [0/31/0 iops] [eta 41m:01s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/11662KBJobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/17351KBJobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/6328KB/Jobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/7111KB/fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 113 (f=113): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [6.4%fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 111 (f=111): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(13),_(1),Wfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),WJobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),WJobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),Wfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 106 (f=106): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(5),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.:12m:03s]
Jobs: 105 (f=105): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 105 (f=105): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. 02h:18m:13s]
Jobs: 104 (f=104): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 104 (f=104): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job./0 iops] [eta 02h:24m:22s]
Jobs: 103 (f=103): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 103 (f=103): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.ops] [eta 02h:30m:34s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 100 (f=100): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.10168KB/0KB /s] [0/158/0 iops] [eta 02h:49m:19s]
Jobs: 99 (f=99): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.933KB/0KB /s] [0/233/0 iops] [eta 02h:55m:35s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 97 (f=97): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [3.3% done] [0KB/9376KB/0KB /s] [0/146/0 iops] [eta 03h:08m:09s]
Jobs: 96 (f=96): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [3.2% done] [0KB/1493KB/0KB /s] [0/23/0 iops] [eta 03h:14m:27s]
Jobs: 95 (f=95): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(1),_(1),W(3),_(1),W(3),_(1),W(5),_(2),W(1),_(1),W(2)] [3.1% done] [0KB/15573KB/0KB /s] [0/243/0 iops] [eta 03h:20m:4fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 94 (f=94): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(1),_(1),W(3),_(1),W(3),_(1),W(5),_(2),W(1),_(1),W(2)] [3.0% done] [0KB/11733KB/0KB /s] [0/183/0 iops] [eta 03h:27m:05s] fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 93 (f=93): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1)Jobs: 93 (f=93): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [2.9% done] [0KB/8675KB/0KB /s] [0/135/0 iops] [eta 03h:33m:23s]
Jobs: 92 (f=92): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.8% done] [0KB/1207KB/0KB /s] [0/18/0 iops] [eta 03h:39m:44s]
Jobs: 91 (f=91): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)Jobs: 91 (f=91): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 03h:46m:06s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 88 (f=88): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 04h:05m:21s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 86 (f=86): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [2.4% done] [0KB/19697KB/0KB /s] [0/307/0 iops] [eta 04h:18m:12s]
Jobs: 85 (f=85): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1)] [2.4% done] [0KB/18062KB/0KB /s] [0/282/0 iops] [eta 04h:24m:37s]
Jobs: 84 (f=84): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.% done] [0KB/8736KB/0KB /s] [0/136/0 iops] [eta 04h:31m:03s]
Jobs: 83 (f=83): [W(5),_(1),W(5),_(1),W(4),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.(1)] [2.3% done] [0KB/34379KB/0KB /s] [0/537/0 iops] [eta 04h:37m:31s]
Jobs: 82 (f=82): [W(5),_(1),W(5),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.1),W(1),_(1)] [2.2% done] [0KB/9447KB/0KB /s] [0/147/0 iops] [eta 04h:44m:00s]
Jobs: 81 (f=81): [W(5),_(1),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(3),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.2% done] [0KB/9173KB/0KB /s] [0/143/0 iops] [eta 04h:50mfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 78 (f=78): [W(5),_(1),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.1% done] [0KB/4830KB/0KB /s] [0/75/0fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/16763KB/0KB /s] [0/261Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/0KB/0KB /s] [0/0/0 iopJobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/11377KB/0KB /s] [0/177Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.1% done] [0KB/12017KB/0KB /s] [0/187fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 76 (f=76): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(2),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/22162KB/0KB fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 75 (f=75): [W(2),_(1),W(1),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(2),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/10fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 72 (f=72): [W(2),_(1),W(1),_(5),W(2),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.9% done] [0KB/6250KB/0KB /fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 70 (f=70): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/10fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 69 (f=69): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/78Jobs: 69 (f=69): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/12fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 68 (f=68): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/1280KB/0KB /Jobs: 68 (f=68): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(2),f(1),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/8097KB/fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 65 (f=65): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(4),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(2),W(1),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(2),W(1),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.7% done] [0KB/11649KB/0KB /s] [0/182fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 62 (f=62): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(5),_(4),W(5),_(3),W(5),_(1),W(3),_(2),W(2),_(2),W(1),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(2),W(1),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.6% done] [0KB/6257KB/0KB /s] [0/97/0fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 106 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 4 (f=4): [_(55),W(1),_(13),W(1),_(6),W(1),_(42),W(1),_(8)] [0.5% done] [0KB/105.8MB/0KB /s] [0/1681/0 iops] [eta 21h:01m:48s] fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.iops] [eta 21h:01m:02s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. 21h:08m:31s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.3s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
WHat the process do:
# truss -af -p 3015098
3015098: psargs: fio myfio.fio
3015098: = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
Pstatus: process is not stopped
#
# truss -acf -p 3015098
psargs: fio myfio.fio
Pstatus: process is not stopped
syscall seconds calls errors
_nsleep 10.44 1423
kioctl .00 56 56
statx .00 1367 1367
---- --- ---
sys totals: .00 2846 1423
usr time: .00
elapsed: .00
#
The file not exist in /tmp but the program look for it!!!
# ls -l /tmp/fio-dump-status
ls: 0653-341 The file /tmp/fio-dump-status does not exist.
#
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-29 8:29 fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out Thierry BERTAUD
@ 2015-12-29 14:30 ` Thierry BERTAUD
2015-12-30 17:08 ` Jens Axboe
1 sibling, 0 replies; 24+ messages in thread
From: Thierry BERTAUD @ 2015-12-29 14:30 UTC (permalink / raw)
To: fio@vger.kernel.org
Hello Fio team,
Just some information.
Fio is always stuck several hours later. It takes a lot of cpu (62 % of the server) but do nothing concerning IO against hdisk and hba card.
If you need more information don't hesitate to ask.
Below output from topas and gdb:
Topas:
--------------------------------------------
Topas Monitor for host: XVIOS16
Tue Dec 29 15:00:31 2015 Interval: 2
CPU User% Kern% Wait% Idle% Physc Entc
ALL 60.1 3.0 0.0 36.8 1.01 101.3
Network KBPS I-Pack O-Pack KB-In KB-Out
Total 3.6 63.0 2.5 3.3 0.2
EVENTS/QUEUES FILE/TTY
Cswitch 722 Readch 2386
Syscall 22916 Writech 386
Reads 10 Rawin 0
Writes 3 Ttyout 272
Forks 0 Igets 0
Execs 0 Namei 212
Runqueue 1.0 Dirblk 0
Waitqueue 0.0
MEMORY
PAGING Real,MB 16384
Faults 40 % Comp 18
Steals 0 % Noncomp 50
PgspIn 0 % Client 50
PgspOut 0
PageIn 0 PAGING SPACE
PageOut 0 Size,MB 512
Sios 0 % Used 4
% Free 96
Disk Busy% KBPS TPS KB-Read KB-Writ
hdisk3 0.0 0.0 0.0 0.0 0.0
hdisk2 0.0 0.0 0.0 0.0 0.0
hdisk4 0.0 0.0 0.0 0.0 0.0
hdisk1 0.0 0.0 0.0 0.0 0.0
hdisk0 0.0 0.0 0.0 0.0 0.0
hdisk6 0.0 0.0 0.0 0.0 0.0
hdisk9 0.0 0.0 0.0 0.0 0.0
hdisk10 0.0 0.0 0.0 0.0 0.0
hdisk5 0.0 0.0 0.0 0.0 0.0
hdisk7 0.0 0.0 0.0 0.0 0.0
NFS (calls/sec)
SerV2 0 WPAR Activ 0
CliV2 0 WPAR Total 0
SerV3 0 Press: "h"-help
CliV3 0 "q"-quit
SerV4 0
CliV4 0
FileSystem KBPS TPS KB-Read KB-Writ
Total 2.3 9.5 2.3 0.0
Name PID CPU% PgSp Owner
fio 11468956 62.6 4.3 root
sched 196614 0.0 0.4 root
vio_daem 10682558 0.0 46.1 root
seaproc 3997822 0.0 1.0 root
java 9764964 0.0 84.1 root
syncd 3538998 0.0 0.6 root
getty 9830486 0.0 0.7 root
sshd 9306218 0.0 1.2 padmin
solid 7995450 0.0 40.0 root
gil 2555982 0.0 0.9 root
posix_ai 53411958 0.0 0.4 root
posix_ai 19529906 0.0 0.4 root
posix_ai 58982424 0.0 0.4 root
fio 3015098 0.0 2.7 root
posix_ai 64946350 0.0 0.4 root
posix_ai 39976972 0.0 0.4 root
topas 1180092 0.0 8.7 root
posix_ai 57737462 0.0 0.4 root
posix_ai 27787362 0.0 0.4 root
posix_ai 44302422 0.0 0.4 root
gdb output:
-----------------------------
# ps -ef | grep fio
root 11468956 39518300 120 02:47:26 pts/1 468:59 /home/padmin/fio/fio-2c2c93d/fio myfio.fio
root 44105746 11206870 0 15:23:58 pts/0 0:00 grep fio
root 3015098 11141224 0 03:24:48 pts/0 0:28 fio myfio.fio
# gdb /home/padmin/fio/fio-2c2c93d/fio 3015098
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/padmin/fio/fio-2c2c93d/fio...done.
Attaching to program: /home/padmin/fio/fio-2c2c93d/fio, process 3015098
[New Thread 1]
[New Thread 258]
[Switching to Thread 258]
0xd051d470 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)
(gdb) bt
#0 0xd051d470 in _p_nsleep () from /usr/lib/libpthread.a(shr_xpg5.o)
#1 0xd011f3e8 in nsleep () from /usr/lib/libc.a(shr.o)
#2 0xd011ee38 in usleep () from /usr/lib/libc.a(shr.o)
#3 0x10001110 in fio_set_fd_nonblocking (fd=805309560, who=0x0) at libfio.c:285
#4 0xd0505c10 in _pthread_body () from /usr/lib/libpthread.a(shr_xpg5.o)
#5 0x00000000 in ?? ()
(gdb)
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-29 8:29 fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out Thierry BERTAUD
2015-12-29 14:30 ` Thierry BERTAUD
@ 2015-12-30 17:08 ` Jens Axboe
2015-12-30 18:34 ` Thierry BERTAUD
1 sibling, 1 reply; 24+ messages in thread
From: Jens Axboe @ 2015-12-30 17:08 UTC (permalink / raw)
To: Thierry BERTAUD, fio@vger.kernel.org
On 12/29/2015 01:29 AM, Thierry BERTAUD wrote:
> Fio team,
>
> During a important workload fio is hanging.
> with iodepth=128 and numjobs=128 , i have the issue but not with both set at 64.
> Environment is AIX 6.1 against SAN disk (hba card 8 Gbps)
>
> Could you take a look at this?
>
> Error message and script file are copy-pasted below.
> # cat myfio.fio
> [global]
> ioengine=posixaio
> invalidate=1
> norandommap
> refill_buffers
> randrepeat=0
> runtime=300
> time_based
> direct=1
> group_reporting
>
> [test]
> stonewall
> filename=/dev/${HDISK}
> rw=${RWTYPE}
> rwmixread=${RRATIO}
> bs=${BS}
> iodepth=${QD}
> numjobs=${NJ}
> write_bw_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_bw.results
> write_iops_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_iops.results
> write_lat_log=${BS}-${RWTYPE}-${RRATIO}R-${QD}q-${QD}-${hdisk}_lat.results
> #
>
> Command:
> # HDISK=hdisk3 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-2c2c93d/fio myfio.fio
>
> PS: Output:
> # ps -ef | grep fio | wc -l
> 130
> # ps -ef | grep fio | grep -v grep | wc -l
> 130
> # nmon
> # ps -ef | grep fio | grep -v grep | wc -l
> 130
> # ps -ef | grep fio | grep -v grep | wc -l
> 130
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 02:59:07 CET 2015
> 17
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 02:59:12 CET 2015
> 17
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:01:35 CET 2015
> 1
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:04:12 CET 2015
> 10
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:10:17 CET 2015
> 1
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:12:18 CET 2015
> 18
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:13:39 CET 2015
> 18
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:20:52 CET 2015
> 34
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:25:03 CET 2015
> 66
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 03:26:43 CET 2015
> 66
> #
> # date;ps -ef | grep fio | grep -v grep | wc -l
> Tue Dec 29 09:15:05 CET 2015
> 2
> #
>
> Command output:
> # HDISK=hdisk3 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-2c2c93d/fio myfio.fio
> test: (g=0): rw=rw, bs=64K-64K/64K-64K/64K-64K, ioengine=posixaio, iodepth=128
> ...
> fio-2.3
> Starting 128 processes
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.0 iops] [eta 17m:00s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.12.9% done] [0KB/1991KB/0KB /s] [0/31/0 iops] [eta 41m:01s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/11662KBJobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/17351KBJobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/6328KB/Jobs: 115 (f=115): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(14),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [7.3% done] [0KB/7111KB/fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 113 (f=113): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(43)] [6.4%fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 111 (f=111): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(4),_(1),W(1),_(1),W(1),_(1),W(13),_(1),Wfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),WJobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),WJobs: 108 (f=108): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(13),_(1),Wfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 106 (f=106): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(5),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.:12m:03s]
> Jobs: 105 (f=105): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 105 (f=105): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. 02h:18m:13s]
> Jobs: 104 (f=104): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 104 (f=104): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job./0 iops] [eta 02h:24m:22s]
> Jobs: 103 (f=103): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(Jobs: 103 (f=103): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.ops] [eta 02h:30m:34s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 100 (f=100): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.10168KB/0KB /s] [0/158/0 iops] [eta 02h:49m:19s]
> Jobs: 99 (f=99): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.933KB/0KB /s] [0/233/0 iops] [eta 02h:55m:35s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 97 (f=97): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [3.3% done] [0KB/9376KB/0KB /s] [0/146/0 iops] [eta 03h:08m:09s]
> Jobs: 96 (f=96): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [3.2% done] [0KB/1493KB/0KB /s] [0/23/0 iops] [eta 03h:14m:27s]
> Jobs: 95 (f=95): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(1),W(1),_(1),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(1),_(1),W(3),_(1),W(3),_(1),W(5),_(2),W(1),_(1),W(2)] [3.1% done] [0KB/15573KB/0KB /s] [0/243/0 iops] [eta 03h:20m:4fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 94 (f=94): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(1),_(1),W(3),_(1),W(3),_(1),W(5),_(2),W(1),_(1),W(2)] [3.0% done] [0KB/11733KB/0KB /s] [0/183/0 iops] [eta 03h:27m:05s] fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 93 (f=93): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1)Jobs: 93 (f=93): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(3),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [2.9% done] [0KB/8675KB/0KB /s] [0/135/0 iops] [eta 03h:33m:23s]
> Jobs: 92 (f=92): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.8% done] [0KB/1207KB/0KB /s] [0/18/0 iops] [eta 03h:39m:44s]
> Jobs: 91 (f=91): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)Jobs: 91 (f=91): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(1),W(3),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 03h:46m:06s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 88 (f=88): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1),W(2)] [2.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 04h:05m:21s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 86 (f=86): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [2.4% done] [0KB/19697KB/0KB /s] [0/307/0 iops] [eta 04h:18m:12s]
> Jobs: 85 (f=85): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job._(1)] [2.4% done] [0KB/18062KB/0KB /s] [0/282/0 iops] [eta 04h:24m:37s]
> Jobs: 84 (f=84): [W(5),_(1),W(10),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.% done] [0KB/8736KB/0KB /s] [0/136/0 iops] [eta 04h:31m:03s]
> Jobs: 83 (f=83): [W(5),_(1),W(5),_(1),W(4),_(1),W(11),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.(1)] [2.3% done] [0KB/34379KB/0KB /s] [0/537/0 iops] [eta 04h:37m:31s]
> Jobs: 82 (f=82): [W(5),_(1),W(5),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.1),W(1),_(1)] [2.2% done] [0KB/9447KB/0KB /s] [0/147/0 iops] [eta 04h:44m:00s]
> Jobs: 81 (f=81): [W(5),_(1),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(1),W(6),_(1),W(4),_(1),W(5),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(3),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.2% done] [0KB/9173KB/0KB /s] [0/143/0 iops] [eta 04h:50mfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 78 (f=78): [W(5),_(1),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.1% done] [0KB/4830KB/0KB /s] [0/75/0fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/16763KB/0KB /s] [0/261Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/0KB/0KB /s] [0/0/0 iopJobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3
),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/11377KB/0KB /s] [0/177Jobs: 77 (f=77): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(8),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.1% done] [0KB/12017KB/0KB /s] [0/187fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 76 (f=76): [W(4),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(2),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/22162KB/0KB fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 75 (f=75): [W(2),_(1),W(1),_(2),W(1),_(1),W(3),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(2),_(1),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [2.0% done] [0KB/10fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 72 (f=72): [W(2),_(1),W(1),_(5),W(2),_(1),W(4),_(1),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.9% done] [0KB/6250KB/0KB /fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 70 (f=70): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/10fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 69 (f=69): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/78Jobs: 69 (f=69): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(3),W(1),_(1),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/12fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 68 (f=68): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/1280KB/0KB /Jobs: 68 (f=68): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(1),W(1),_(2),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(1),W(2),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(2),f(1),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.8% done] [0KB/8097KB/fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 65 (f=65): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(2),W(2),_(1),W(5),_(4),W(6),_(2),W(5),_(1),W(4),_(1),W(2),_(2),W(1),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(2),W(1),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.7% done] [0KB/11649KB/0KB /s] [0/182fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 62 (f=62): [W(2),_(1),W(1),_(5),W(1),_(2),W(1),_(1),W(1),_(3),W(1),_(1),W(5),_(4),W(5),_(3),W(5),_(1),W(3),_(2),W(2),_(2),W(1),_(1),W(3),_(1),W(8),_(2),W(2),_(3),W(2),_(1),W(1),_(4),W(1),_(3),W(1),_(2),W(1),_(2),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(5),W(1),_(1),W(2),_(4),W(2),_(3),W(1),_(1),W(1),_(1)] [1.6% done] [0KB/6257KB/0KB /s] [0/97/0fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 106 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 105 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 4 (f=4): [_(55),W(1),_(13),W(1),_(6),W(1),_(42),W(1),_(8)] [0.5% done] [0KB/105.8MB/0KB /s] [0/1681/0 iops] [eta 21h:01m:48s] fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.iops] [eta 21h:01m:02s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. 21h:08m:31s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.3s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
>
> WHat the process do:
> # truss -af -p 3015098
> 3015098: psargs: fio myfio.fio
> 3015098: = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> 3015098: 117309669: statx("/tmp/fio-dump-status", 0x2FF20648, 128, 010) Err#2 ENOENT
> 3015098: 117309669: _nsleep(0x2FF20A60, 0x2FF20AD8) = 0
> Pstatus: process is not stopped
That's the main thread, what are the IO threads doing?
> # truss -acf -p 3015098
> psargs: fio myfio.fio
> Pstatus: process is not stopped
> syscall seconds calls errors
> _nsleep 10.44 1423
> kioctl .00 56 56
> statx .00 1367 1367
> ---- --- ---
> sys totals: .00 2846 1423
> usr time: .00
> elapsed: .00
> #
>
> The file not exist in /tmp but the program look for it!!!
> # ls -l /tmp/fio-dump-status
> ls: 0653-341 The file /tmp/fio-dump-status does not exist.
> #
That's expected - when the main thread goes to sleep, it checks for this
status dump file. If it exists, then it'll show the current summary of
stats, then unlink it.
--
Jens Axboe
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-30 17:08 ` Jens Axboe
@ 2015-12-30 18:34 ` Thierry BERTAUD
2015-12-31 4:02 ` Jens Axboe
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2015-12-30 18:34 UTC (permalink / raw)
To: Jens Axboe, fio@vger.kernel.org
Hello Jens,
>That's expected - when the main thread goes to sleep, it checks for this
>status dump file. If it exists, then it'll show the current summary of
>stats, then unlink it.
Thanks for explain this point.
Just to confirm my understanding the following option with engine 'posixaio'
iodepth=128 and numjobs=128 =>
I don't use 'thread' option so i think with iodpeth=128 the main fio process will fork to create 128 process.
With numjob=128 that imply, i'll clone each forked process from fio (due to iodepth) that's imply 16384 process (128 * 128).
If i use threads enabled, with iodepth=128 the main fio process will create with pthread_create 128 threads for itself.
With numjobs=128, the ain fio process will clone to have 16384 threads into itselft or it will fork to create 128 process and each process has 128 threads.
> That's the main thread, what are the IO threads doing?
If i follow disk I/O there are no activities.
This kind of information for fio process can help or not to answer the questions?
The list of thread by process this:
# ps -mo THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
root 9437428 10551472 - A 0 60 1 - 200001 pts/0 - ksh
- - - 18350143 S 0 60 1 - 10400 - - -
padmin 10551472 11534386 - A 0 60 1 - 240001 pts/0 - -rksh
- - - 49873143 S 0 60 1 - 10400 - - -
root 11206870 9437428 - A 1 60 1 - 200001 pts/0 - bash
- - - 48759011 S 1 60 1 - 410400 - - -
root 62914724 11206870 - A 6 63 1 - 200001 pts/0 - ps -mo THREAD
- - - 116850921 R 6 63 1 - 400000 - - -
#
Or thread for a given process like this:
# ps -mp 9437428 -o THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
root 9437428 10551472 - A 0 60 1 - 200001 pts/0 - ksh
- - - 18350143 S 0 60 1 - 10400 - - -
#
I tried to use gdb against the live process but i don't how to progress with it?
I can do gdb against the executable also but i don't know how to progress whit it?
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-30 18:34 ` Thierry BERTAUD
@ 2015-12-31 4:02 ` Jens Axboe
2015-12-31 16:18 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Jens Axboe @ 2015-12-31 4:02 UTC (permalink / raw)
To: Thierry BERTAUD, fio@vger.kernel.org
On 12/30/2015 11:34 AM, Thierry BERTAUD wrote:
> Hello Jens,
>
>> That's expected - when the main thread goes to sleep, it checks for this
>> status dump file. If it exists, then it'll show the current summary of
>> stats, then unlink it.
> Thanks for explain this point.
>
> Just to confirm my understanding the following option with engine 'posixaio'
> iodepth=128 and numjobs=128 =>
> I don't use 'thread' option so i think with iodpeth=128 the main fio process will fork to create 128 process.
> With numjob=128 that imply, i'll clone each forked process from fio (due to iodepth) that's imply 16384 process (128 * 128).
> If i use threads enabled, with iodepth=128 the main fio process will create with pthread_create 128 threads for itself.
> With numjobs=128, the ain fio process will clone to have 16384 threads into itselft or it will fork to create 128 process and each process has 128 threads.
>
>> That's the main thread, what are the IO threads doing?
> If i follow disk I/O there are no activities.
> This kind of information for fio process can help or not to answer the questions?
> The list of thread by process this:
> # ps -mo THREAD
> USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
> root 9437428 10551472 - A 0 60 1 - 200001 pts/0 - ksh
> - - - 18350143 S 0 60 1 - 10400 - - -
> padmin 10551472 11534386 - A 0 60 1 - 240001 pts/0 - -rksh
> - - - 49873143 S 0 60 1 - 10400 - - -
> root 11206870 9437428 - A 1 60 1 - 200001 pts/0 - bash
> - - - 48759011 S 1 60 1 - 410400 - - -
> root 62914724 11206870 - A 6 63 1 - 200001 pts/0 - ps -mo THREAD
> - - - 116850921 R 6 63 1 - 400000 - - -
> #
> Or thread for a given process like this:
> # ps -mp 9437428 -o THREAD
> USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
> root 9437428 10551472 - A 0 60 1 - 200001 pts/0 - ksh
> - - - 18350143 S 0 60 1 - 10400 - - -
> #
>
> I tried to use gdb against the live process but i don't how to progress with it?
> I can do gdb against the executable also but i don't know how to progress whit it?
Against the live process it the easiest. Just do gdb -p <pid> and
attach. If you don't use threads from fio, then they will each be a
separate process.
I'll be taking a bit of time off, but I should have access to an AIX
that I can test on sometime next week. What version are you on?
--
Jens Axboe
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-31 4:02 ` Jens Axboe
@ 2015-12-31 16:18 ` Thierry BERTAUD
2015-12-31 16:33 ` Andrey Kuzmin
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2015-12-31 16:18 UTC (permalink / raw)
To: Jens Axboe, fio@vger.kernel.org, Andrey Kuzmin
Hi Jens and Andrey,
>I'll be taking a bit of time off, but I should have access to an AIX
>that I can test on sometime next week. What version are you on?
At this time i use a VIOS 2.2.3.52 aka AIX 6100-08-01-1245. I need to build fio against AIX 6.x, 7.x.
After next week, i'll reinstall the server to go to production. So i'll need to look for a new box with a FC card so i'll use another AIX version.
For your information i don't have the issue with:
HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=4096 QD=128 NJ=128 /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
but
HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
So i think it's perhaps some hardware or OS comportement that the cause of the issue it's not fio fio (it's not the guilty but a collateral victim) but idon't find why because the statistics data from SAN seems OK.
SAN stats during test:
Latency (ms): R: 0/ W 0.17
Thoughput (M): R 0 / W 9.8
IOPS: R 0 / W 2380
1. fIo stay stuck with the last print:
fio: job 'test' hasn't exited in 97 seconds, it appears to be stuck. Doing forceful exit of this job.
1.1 The only process that's stay a live is the min fio process (all others 128 childs have been died or finished)
# ps -ef | grep fio | grep -v grep
root 41877588 48234628 120 16:18:23 pts/1 1:20 /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
#
1.3 System monitor indicate no disk activty test is over but the main process uses CPU.
Topas Monitor for host: XVIOS16
Thu Dec 31 16:28:19 2015 Interval: 2
CPU User% Kern% Wait% Idle% Physc Entc
ALL 57.4 5.8 0.0 36.8 1.01 101.5
Disk Busy% KBPS TPS KB-Read KB-Writ
hdisk3 0.0 0.0 0.0 0.0 0.0
hdisk2 0.0 0.0 0.0 0.0 0.0
hdisk9 0.0 0.0 0.0 0.0 0.0
hdisk1 0.0 0.0 0.0 0.0 0.0
hdisk0 0.0 0.0 0.0 0.0 0.0
hdisk10 0.0 0.0 0.0 0.0 0.0
hdisk7 0.0 0.0 0.0 0.0 0.0
hdisk4 0.0 0.0 0.0 0.0 0.0
hdisk5 0.0 0.0 0.0 0.0 0.0
hdisk6 0.0 0.0 0.0 0.0 0.0
Name PID CPU% PgSp Owner
fio 41877588 62.6 3.5 root
sched 196614 0.1 0.4 root
vio_daem 10682558 0.0 46.1 root
seaproc 3997822 0.0 1.0 root
syncd 3538998 0.0 0.6 root
java 9764964 0.0 84.2 root
1.5 During gdb against the process the cpu load disapear. If i don't dgb against the process the load stay alive.
2. 1: gdb trace
# gdb -p 41877588
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 41877588
0xd0119598 in ?? ()
(gdb) info threads
Id Target Id Frame
* 1 process 41877588 0xd0119598 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 41877588)]
#0 0xd0119598 in ?? ()
(gdb) bt
#0 0xd0119598 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd0119bc0 in ?? ()
#3 0xd0118f28 in ?? ()
#4 0x100625dc in ?? ()
#5 0x1006482c in ?? ()
#6 0x10064b90 in ?? ()
#7 0xd0505c10 in ?? ()
#8 0x00000000 in ?? ()
(gdb)
2.1.1 It semmes i nned to build fio with some option to have some symbole. Which option i can use for that.
At this time i just do: ./configure; gmake CC=gcc
2.2: The full log from fio:
# HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
test: (g=0): rw=rw, bs=64K-64K/64K-64K/64K-64K, ioengine=posixaio, iodepth=128
...
fio-2.3
Starting 128 processes
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.:22s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. iops] [eta 11m:34s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.] [0/138/0 iops] [eta 17m:34s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.4KB/0KB /s] [0/215/0 iops] [eta 29m:36s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.4.6% done] [0KB/7488KB/0KB /s] [0/117/0 iops] [eta 35m:36s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.1),W(21)] [12.8% done] [0KB/8617KB/0KB /s] [0/134/0 iops] [eta 41m:40s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.1),W(38)] [11.4% done] [0KB/856KB/0KB /s] [0/13/0 iops] [eta 47m:44s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 118 (f=118): [f(1),W(8),_(1),W(11),_(1),W(2),f(1),W(5),_(1),W(9),_(1),W(7),_(1),W(12),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] [9.3% done] [0KB/20057KB/0KB fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 117 (f=117): [W(9),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(4),f(1),W(2),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] [8.5% done] [0KB/7255KB/fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 116 (f=116): [W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] [7.9% done] [0KB/1130KB/Jobs: 116 (f=116): [W(2),f(1),_(1),W(5),_(1),W(2),f(1),W(8),_(1),W(8),_(1),W(4),f(1),W(4),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] Jobs: 116 (f=116): [W(3),_(1),W(5),_(1),W(1),f(1),W(9),_(1),W(8),_(1),W(4),f(1),W(4),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] [7.9%Jobs: 116 (f=116): [W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)] [7.9% done] [0KB/0KB/0KBfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 115 (f=115): [W(3),_(1),W(5),_(1),W(11),_(1),W(3),f(1),W(4),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(1),f(1),W(8),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(36),_(1),WJobs: 115 (f=115): [W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(6),f(1),W(4),_(1),W(2),_(1),W(5),_(1),W(36),_(1),W(1)] [7.4%fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 114 (f=114): [W(3),_(1),W(5),_(1),W(4),f(1),W(6),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(3),_(1),W(1),_(1),W(36),_(1),Wfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 113 (f=113): [W(3),_(1),W(5),_(1),W(11),_(1),W(6),f(1),W(1),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),WJobs: 113 (f=113): [W(3),_(1),W(5),_(1),W(11),_(1),W(6),f(1),W(1),_(1),W(2),f(2),W(5),_(1),W(7),_(1),W(1),_(1),f(1),W(9),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(Jobs: 113 (f=113): [W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(2),f(2),W(5),_(1),W(7),_(1),W(1),_(1),f(1),W(9),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 111 (f=111): [W(3),_(1),W(5),_(1),W(11),_(1),W(7),f(1),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 110 (f=110): [W(3),_(1),W(5),_(1),W(10),f(1),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(Jobs: 109 (f=109): [W(3),_(1),f(1),W(4),_(1),W(10),f(1),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.:55m:38s]
Jobs: 108 (f=108): [W(3),_(1),f(1),W(2),f(1),W(1),_(1),W(7),f(1),W(3),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(2),f(1),W(3),_(1),W(6),_(1),W(8),_(1),W(2),_(1)fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job./0KB /s] [0/142/0 iops] [eta 02h:01m:37s]
Jobs: 107 (f=107): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1Jobs: 107 (f=107): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.eta 02h:07m:35s]
Jobs: 106 (f=106): [W(3),_(1),W(5),_(1),W(6),f(1),W(1),f(1),W(2),_(1),W(5),_(1),W(2),_(1),W(7),f(1),W(1),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2)Jobs: 106 (f=106): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.42/0 iops] [eta 02h:13m:57s]
Jobs: 105 (f=105): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1Jobs: 105 (f=105): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(2),f(1),W(2),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.3/0 iops] [eta 02h:19m:56s]
Jobs: 104 (f=104): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(2),f(1),W(2),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.B/0KB /s] [0/136/0 iops] [eta 02h:26m:24s]
Jobs: 103 (f=103): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(1),f(1),_(1Jobs: 103 (f=103): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.ops] [eta 02h:32m:22s]
Jobs: 103 (f=103): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(2),f(1),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(2),_(1fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. [0/205/0 iops] [eta 02h:38m:22s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 97 (f=97): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),Jobs: 97 (f=97): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(6),f(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),Jobs: 97 (f=97): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job./0KB /s] [0/0/0 iops] [eta 03h:08m:26s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 94 (f=94): [W(3),_(1),W(4),f(1),_(1),W(11),_(1),W(5),_(1),W(2),_(1),f(1),W(8),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job. done] [0KB/14019KB/0KB /s] [0/219/0 iops] [eta 03h:26m:33s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 92 (f=92): [W(3),_(1),W(4),f(1),_(1),W(11),_(1),W(5),_(1),W(2),_(1),f(1),W(8),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.(2)] [2.9% done] [0KB/1593KB/0KB /s] [0/24/0 iops] [eta 03h:38m:38s]
Jobs: 91 (f=91): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.(2)] [2.9% done] [0KB/5694KB/0KB /s] [0/88/0 iops] [eta 03h:44m:49s]
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 89 (f=89): [W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(6),f(1),W(2),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(3),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(1),f(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(4),_(1),W(2),_(2)] [2.7% done] [0KB/15730KB/0KB /s] [0/245/0 ifio: job 'test' hasn't exited in 61 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 87 (f=87): [W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(6),f(1),W(2),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(3),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(2),f(1),_(1),f(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(4),_(1),W(2),_(2)] [2.6% done] [0KB/12825KB/0KB /s] [0/200/0 iops] fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 94 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 85 (f=85): [W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),f(1),W(2),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [2.5% done] [0KB/10149KB/0KB /s] [0/15fio: job 'test' hasn't exited in 100 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 86 (f=86): [W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(2),f(1),W(3),_(3),W(3),_(3),W(4),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),f(1),_(1),W(1),_(1),W(1),_(3),W(1),_(3),f(1),W(2),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [2.4% done] [0KB/15226KB/0KB /s] [0/23fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 83 (f=83): [W(1),_(1),W(1),_(1),W(5),_(1),W(9),f(1),W(1),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [2.2% done] [0KB/12140KB/0KB fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 81 (f=81): [W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),f(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(4),f(1),W(2),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [2.1% dofio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 81 (f=81): [W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),f(1),_(1),W(1),_(1),W(1),_(1),f(1),W(1),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [2.1% done] [fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 77 (f=77): [W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),f(1),W(1),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [1.9% done] [0KB/275KB/fio: job 'test' hasn't exited in 89 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 77 (f=77): [W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),f(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [1.9% done] [0KB/16256KB/0KBfio: job 'test' hasn't exited in 104 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 77 (f=77): [W(1),_(1),W(1),_(4),W(2),_(1),W(5),f(1),W(5),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),f(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [1.9% done] [0KB/29fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 110 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 75 (f=75): [W(1),_(1),W(1),_(4),W(2),_(1),W(5),f(1),W(5),_(1),W(5),_(1),W(2),_(1),W(1),_(1),W(7),_(2),f(1),W(3),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [1.8% done] [0fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 73 (f=73): [W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(2),f(1),W(4),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)] [1.7% done] [0KB/8633KB/0KB fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 71 (f=71): [W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(1),W(1),f(1),W(7),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)] [1.7% done] [0KB/51fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 69 (f=69): [W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),f(1),W(2),_(2),W(1),_(4),W(2),_(1),W(1),_(1),f(1),W(1),_(2)] [1.6% done] [0KB/14Jobs: 69 (f=69): [W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(3),f(1),W(3),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),f(1),W(2),_(2),W(1),_(4),W(2),_(1),W(1),_(1),f(1),W(1),_(2)] [1.6% donfio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 68 (f=68): [W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(1),f(1),W(2),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(3),f(1),W(3),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]Jobs: 67 (f=67): [W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(1),f(1),W(1),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 100 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 64 (f=64): [_(2),W(1),_(4),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),f(1),_(3),W(6),_(2),f(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(1),f(1),W(1),_(4),W(3),_(2),W(1),_(4),f(1),W(1),_(1),W(1),_(1),W(2),Jobs: 63 (f=63): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),f(1),_(3),W(6),_(2),f(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),f(1),W(1),_(1),W(1),_(1),W(2),_(2)] [1.5% done] [0fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 62 (f=62): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(6),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)] [1.5% done] [0KB/1263KB/Jobs: 60 (f=60): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(2),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(6),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(2),f(1),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]fio: job 'test' hasn't exited in 101 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 59 (f=59): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(2),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(2),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 58 (f=58): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(2),W(1),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(2),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(1),f(1),W(1),_(2),W(1),_(4),W(2),_(1),fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.
Jobs: 56 (f=56): [_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),f(1),_(2),W(1),_(2),W(1),_(1),W(1),_(2),W(1),_(1),W(3),f(1),_(2),W(3),_(2),W(1),_(3),W(1),_(1),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),f(1),W(2),_(4),W(3),_(2),W(1),_(4),W(2),fio: job 'test' hasn't exited in 92 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 96 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 100 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck. Doing forceful exit of this job.
fio: job 'test' hasn't exited in 97 seconds, it appears to be stuck. Doing forceful exit of this job.
Thanks for your feedback
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-31 16:18 ` Thierry BERTAUD
@ 2015-12-31 16:33 ` Andrey Kuzmin
2016-01-01 12:43 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Andrey Kuzmin @ 2015-12-31 16:33 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 29322 bytes --]
On Dec 31, 2015 19:19, "Thierry BERTAUD" <tbertaud@conforama.fr> wrote:
>
> Hi Jens and Andrey,
>
> >I'll be taking a bit of time off, but I should have access to an AIX
> >that I can test on sometime next week. What version are you on?
>
> At this time i use a VIOS 2.2.3.52 aka AIX 6100-08-01-1245. I need to
build fio against AIX 6.x, 7.x.
> After next week, i'll reinstall the server to go to production. So i'll
need to look for a new box with a FC card so i'll use another AIX version.
>
> For your information i don't have the issue with:
> HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=4096 QD=128 NJ=128
/home/padmin/fio/fio-5f3bd0f/fio myfio.fio
> but
> HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128
/home/padmin/fio/fio-5f3bd0f/fio myfio.fio
>
> So i think it's perhaps some hardware or OS comportement that the cause
of the issue it's not fio fio (it's not the guilty but a collateral
victim) but idon't find why because the statistics data from SAN seems OK.
> SAN stats during test:
> Latency (ms): R: 0/ W 0.17
> Thoughput (M): R 0 / W 9.8
> IOPS: R 0 / W 2380
>
>
> 1. fIo stay stuck with the last print:
> fio: job 'test' hasn't exited in 97 seconds, it appears to be stuck.
Doing forceful exit of this job.
>
> 1.1 The only process that's stay a live is the min fio process (all
others 128 childs have been died or finished)
> # ps -ef | grep fio | grep -v grep
> root 41877588 48234628 120 16:18:23 pts/1 1:20
/home/padmin/fio/fio-5f3bd0f/fio myfio.fio
> #
>
> 1.3 System monitor indicate no disk activty test is over but the main
process uses CPU.
> Topas Monitor for host: XVIOS16
> Thu Dec 31 16:28:19 2015 Interval: 2
>
> CPU User% Kern% Wait% Idle% Physc Entc
> ALL 57.4 5.8 0.0 36.8 1.01 101.5
>
> Disk Busy% KBPS TPS KB-Read KB-Writ
> hdisk3 0.0 0.0 0.0 0.0 0.0
> hdisk2 0.0 0.0 0.0 0.0 0.0
> hdisk9 0.0 0.0 0.0 0.0 0.0
> hdisk1 0.0 0.0 0.0 0.0 0.0
> hdisk0 0.0 0.0 0.0 0.0 0.0
> hdisk10 0.0 0.0 0.0 0.0 0.0
> hdisk7 0.0 0.0 0.0 0.0 0.0
> hdisk4 0.0 0.0 0.0 0.0 0.0
> hdisk5 0.0 0.0 0.0 0.0 0.0
> hdisk6 0.0 0.0 0.0 0.0 0.0
>
> Name PID CPU% PgSp Owner
> fio 41877588 62.6 3.5 root
> sched 196614 0.1 0.4 root
> vio_daem 10682558 0.0 46.1 root
> seaproc 3997822 0.0 1.0 root
> syncd 3538998 0.0 0.6 root
> java 9764964 0.0 84.2 root
>
> 1.5 During gdb against the process the cpu load disapear. If i don't dgb
against the process the load stay alive.
>
> 2. 1: gdb trace
> # gdb -p 41877588
> GNU gdb (GDB) 7.8.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "powerpc-ibm-aix5.1.0.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> Attaching to process 41877588
> 0xd0119598 in ?? ()
> (gdb) info threads
> Id Target Id Frame
> * 1 process 41877588 0xd0119598 in ?? ()
> (gdb) thread 1
> [Switching to thread 1 (process 41877588)]
> #0 0xd0119598 in ?? ()
> (gdb) bt
> #0 0xd0119598 in ?? ()
> #1 0xffffffff in ?? ()
> #2 0xd0119bc0 in ?? ()
> #3 0xd0118f28 in ?? ()
> #4 0x100625dc in ?? ()
> #5 0x1006482c in ?? ()
> #6 0x10064b90 in ?? ()
> #7 0xd0505c10 in ?? ()
> #8 0x00000000 in ?? ()
> (gdb)
>
> 2.1.1 It semmes i nned to build fio with some option to have some
symbole. Which option i can use for that.
> At this time i just do: ./configure; gmake CC=gcc
>
Try 'configure --disable-optimizations'.
Regards,
A.
> 2.2: The full log from fio:
> # HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128
/home/padmin/fio/fio-5f3bd0f/fio myfio.fio
> test: (g=0): rw=rw, bs=64K-64K/64K-64K/64K-64K, ioengine=posixaio,
iodepth=128
> ...
> fio-2.3
> Starting 128 processes
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.:22s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job. iops] [eta 11m:34s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.] [0/138/0 iops] [eta 17m:34s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.4KB/0KB /s] [0/215/0 iops] [eta 29m:36s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.4.6% done] [0KB/7488KB/0KB /s] [0/117/0
iops] [eta 35m:36s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.1),W(21)] [12.8% done] [0KB/8617KB/0KB /s]
[0/134/0 iops] [eta 41m:40s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.1),W(38)] [11.4% done] [0KB/856KB/0KB /s]
[0/13/0 iops] [eta 47m:44s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 118 (f=118):
[f(1),W(8),_(1),W(11),_(1),W(2),f(1),W(5),_(1),W(9),_(1),W(7),_(1),W(12),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
[9.3% done] [0KB/20057KB/0KB fio: job 'test' hasn't exited in 60 seconds,
it appears to be stuck. Doing forceful exit of this job.
> Jobs: 117 (f=117):
[W(9),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(4),f(1),W(2),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
[8.5% done] [0KB/7255KB/fio: job 'test' hasn't exited in 60 seconds, it
appears to be stuck. Doing forceful exit of this job.
> Jobs: 116 (f=116):
[W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
[7.9% done] [0KB/1130KB/Jobs: 116 (f=116):
[W(2),f(1),_(1),W(5),_(1),W(2),f(1),W(8),_(1),W(8),_(1),W(4),f(1),W(4),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
Jobs: 116 (f=116):
[W(3),_(1),W(5),_(1),W(1),f(1),W(9),_(1),W(8),_(1),W(4),f(1),W(4),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
[7.9%Jobs: 116 (f=116):
[W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(38)]
[7.9% done] [0KB/0KB/0KBfio: job 'test' hasn't exited in 60 seconds, it
appears to be stuck. Doing forceful exit of this job.
> Jobs: 115 (f=115):
[W(3),_(1),W(5),_(1),W(11),_(1),W(3),f(1),W(4),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(1),f(1),W(8),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(5),_(1),W(36),_(1),WJobs:
115 (f=115):
[W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(6),f(1),W(4),_(1),W(2),_(1),W(5),_(1),W(36),_(1),W(1)]
[7.4%fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 114 (f=114):
[W(3),_(1),W(5),_(1),W(4),f(1),W(6),_(1),W(8),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(3),_(1),W(1),_(1),W(36),_(1),Wfio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> Jobs: 113 (f=113):
[W(3),_(1),W(5),_(1),W(11),_(1),W(6),f(1),W(1),_(1),W(9),_(1),W(7),_(1),W(1),_(1),W(10),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),WJobs:
113 (f=113):
[W(3),_(1),W(5),_(1),W(11),_(1),W(6),f(1),W(1),_(1),W(2),f(2),W(5),_(1),W(7),_(1),W(1),_(1),f(1),W(9),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(Jobs:
113 (f=113):
[W(3),_(1),W(5),_(1),W(11),_(1),W(8),_(1),W(2),f(2),W(5),_(1),W(7),_(1),W(1),_(1),f(1),W(9),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 111 (f=111):
[W(3),_(1),W(5),_(1),W(11),_(1),W(7),f(1),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 110 (f=110):
[W(3),_(1),W(5),_(1),W(10),f(1),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(Jobs:
109 (f=109):
[W(3),_(1),f(1),W(4),_(1),W(10),f(1),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(11),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.:55m:38s]
> Jobs: 108 (f=108):
[W(3),_(1),f(1),W(2),f(1),W(1),_(1),W(7),f(1),W(3),_(1),W(8),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(2),f(1),W(3),_(1),W(6),_(1),W(8),_(1),W(2),_(1)fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job./0KB /s] [0/142/0 iops] [eta 02h:01m:37s]
> Jobs: 107 (f=107):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1Jobs:
107 (f=107):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.eta 02h:07m:35s]
> Jobs: 106 (f=106):
[W(3),_(1),W(5),_(1),W(6),f(1),W(1),f(1),W(2),_(1),W(5),_(1),W(2),_(1),W(7),f(1),W(1),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2)Jobs:
106 (f=106):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(1),W(6),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.42/0 iops] [eta 02h:13m:57s]
> Jobs: 105 (f=105):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1Jobs:
105 (f=105):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(2),f(1),W(2),_(1),W(6),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.3/0 iops] [eta 02h:19m:56s]
> Jobs: 104 (f=104):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(2),f(1),W(2),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.B/0KB /s] [0/136/0 iops] [eta 02h:26m:24s]
> Jobs: 103 (f=103):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(1),f(1),_(1Jobs:
103 (f=103):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(2),_(1),W(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.ops] [eta 02h:32m:22s]
> Jobs: 103 (f=103):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(1),W(1),_(1),W(3),_(2),W(5),_(1),W(2),f(1),_(1),W(2),_(1),W(8),_(1),W(2),_(1),W(2),_(1fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job. [0/205/0 iops] [eta 02h:38m:22s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 97 (f=97):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),Jobs:
97 (f=97):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(6),f(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),Jobs:
97 (f=97):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job./0KB /s] [0/0/0 iops] [eta 03h:08m:26s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 94 (f=94):
[W(3),_(1),W(4),f(1),_(1),W(11),_(1),W(5),_(1),W(2),_(1),f(1),W(8),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job. done] [0KB/14019KB/0KB /s] [0/219/0 iops] [eta
03h:26m:33s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 92 (f=92):
[W(3),_(1),W(4),f(1),_(1),W(11),_(1),W(5),_(1),W(2),_(1),f(1),W(8),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.(2)] [2.9% done] [0KB/1593KB/0KB /s] [0/24/0
iops] [eta 03h:38m:38s]
> Jobs: 91 (f=91):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(2),W(5),_(1),W(3),_(1),W(2),_(2),W(7),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.(2)] [2.9% done] [0KB/5694KB/0KB /s] [0/88/0
iops] [eta 03h:44m:49s]
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 89 (f=89):
[W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(6),f(1),W(2),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(3),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(1),f(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(4),_(1),W(2),_(2)]
[2.7% done] [0KB/15730KB/0KB /s] [0/245/0 ifio: job 'test' hasn't exited in
61 seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 87 (f=87):
[W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(6),f(1),W(2),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(3),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(2),f(1),_(1),f(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(4),_(1),W(2),_(2)]
[2.6% done] [0KB/12825KB/0KB /s] [0/200/0 iops] fio: job 'test' hasn't
exited in 60 seconds, it appears to be stuck. Doing forceful exit of this
job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 94 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 85 (f=85):
[W(3),_(1),W(5),_(1),W(11),_(1),W(4),f(1),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(3),W(4),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(1),f(1),W(5),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),f(1),W(2),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[2.5% done] [0KB/10149KB/0KB /s] [0/15fio: job 'test' hasn't exited in 100
seconds, it appears to be stuck. Doing forceful exit of this job.
> Jobs: 86 (f=86):
[W(3),_(1),W(5),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(2),f(1),W(3),_(3),W(3),_(3),W(4),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),f(1),_(1),W(1),_(1),W(1),_(3),W(1),_(3),f(1),W(2),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[2.4% done] [0KB/15226KB/0KB /s] [0/23fio: job 'test' hasn't exited in 60
seconds, it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 83 (f=83):
[W(1),_(1),W(1),_(1),W(5),_(1),W(9),f(1),W(1),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(6),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[2.2% done] [0KB/12140KB/0KB fio: job 'test' hasn't exited in 60 seconds,
it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 103 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 81 (f=81):
[W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),f(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(4),f(1),W(2),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[2.1% dofio: job 'test' hasn't exited in 60 seconds, it appears to be
stuck. Doing forceful exit of this job.
> Jobs: 81 (f=81):
[W(1),_(1),W(1),_(1),W(1),_(1),W(3),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),f(1),_(1),W(1),_(1),W(1),_(1),f(1),W(1),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(1),W(5),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[2.1% done] [fio: job 'test' hasn't exited in 60 seconds, it appears to be
stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 77 (f=77):
[W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),f(1),W(1),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.9% done] [0KB/275KB/fio: job 'test' hasn't exited in 89 seconds, it
appears to be stuck. Doing forceful exit of this job.
> Jobs: 77 (f=77):
[W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),f(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.9% done] [0KB/16256KB/0KBfio: job 'test' hasn't exited in 104 seconds,
it appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 77 (f=77):
[W(1),_(1),W(1),_(4),W(2),_(1),W(5),f(1),W(5),_(1),W(5),_(1),W(2),_(1),W(9),_(2),W(4),_(1),f(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(1),W(1),_(3),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.9% done] [0KB/29fio: job 'test' hasn't exited in 60 seconds, it appears
to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 110 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 75 (f=75):
[W(1),_(1),W(1),_(4),W(2),_(1),W(5),f(1),W(5),_(1),W(5),_(1),W(2),_(1),W(1),_(1),W(7),_(2),f(1),W(3),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(1),W(1),_(2),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.8% done] [0fio: job 'test' hasn't exited in 60 seconds, it appears to be
stuck. Doing forceful exit of this job.
> Jobs: 73 (f=73):
[W(1),_(1),W(1),_(4),W(2),_(1),W(11),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(2),f(1),W(4),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(2),_(3),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.7% done] [0KB/8633KB/0KB fio: job 'test' hasn't exited in 60 seconds, it
appears to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 71 (f=71):
[W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(1),W(1),f(1),W(7),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(1),W(2),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.7% done] [0KB/51fio: job 'test' hasn't exited in 60 seconds, it appears
to be stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 69 (f=69):
[W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),f(1),W(2),_(2),W(1),_(4),W(2),_(1),W(1),_(1),f(1),W(1),_(2)]
[1.6% done] [0KB/14Jobs: 69 (f=69):
[W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(4),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(3),f(1),W(3),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),f(1),W(2),_(2),W(1),_(4),W(2),_(1),W(1),_(1),f(1),W(1),_(2)]
[1.6% donfio: job 'test' hasn't exited in 60 seconds, it appears to be
stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 68 (f=68):
[W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(8),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(1),f(1),W(2),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(3),f(1),W(3),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]Jobs:
67 (f=67):
[W(1),_(1),W(1),_(4),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(4),_(1),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(2),W(7),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(1),f(1),W(1),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 100 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 64 (f=64):
[_(2),W(1),_(4),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),f(1),_(3),W(6),_(2),f(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(1),f(1),W(1),_(4),W(3),_(2),W(1),_(4),f(1),W(1),_(1),W(1),_(1),W(2),Jobs:
63 (f=63):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(1),W(2),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(1),f(1),_(3),W(6),_(2),f(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),f(1),W(1),_(1),W(1),_(1),W(2),_(2)]
[1.5% done] [0fio: job 'test' hasn't exited in 60 seconds, it appears to be
stuck. Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 62 (f=62):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(7),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(6),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(3),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]
[1.5% done] [0KB/1263KB/Jobs: 60 (f=60):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(2),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(3),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(6),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(2),f(1),_(2),W(1),_(4),W(2),_(1),W(1),_(1),W(2),_(2)]fio:
job 'test' hasn't exited in 101 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 59 (f=59):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(1),W(2),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(2),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(3),_(2),W(1),_(4),W(2),_(1),W(1),_(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 60 seconds, it appears to be stuck.
Doing forceful exit of this job.
> Jobs: 58 (f=58):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),W(1),_(1),W(2),_(2),W(1),_(1),W(1),_(2),W(1),_(1),W(4),_(2),W(3),_(2),W(1),_(3),W(2),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),W(3),_(4),W(1),f(1),W(1),_(2),W(1),_(4),W(2),_(1),fio:
job 'test' hasn't exited in 60 seconds, it appears to be stuck. Doing
forceful exit of this job.
> Jobs: 56 (f=56):
[_(7),W(2),_(1),W(1),_(2),W(1),_(1),W(6),_(2),f(1),_(2),W(1),_(2),W(1),_(1),W(1),_(2),W(1),_(1),W(3),f(1),_(2),W(3),_(2),W(1),_(3),W(1),_(1),f(1),_(3),W(2),_(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(3),W(4),_(1),W(1),_(2),W(1),_(2),W(1),_(5),W(1),_(3),W(1),_(1),W(1),_(1),W(1),_(1),W(1),_(3),f(1),W(2),_(4),W(3),_(2),W(1),_(4),W(2),fio:
job 'test' hasn't exited in 92 seconds, it appears to be stuck. Doing
forceful exit of this job.
> fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 96 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 100 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 95 seconds, it appears to be stuck.
Doing forceful exit of this job.
> fio: job 'test' hasn't exited in 97 seconds, it appears to be stuck.
Doing forceful exit of this job.
>
> Thanks for your feedback
>
> Regards,
> Thierry
>
[-- Attachment #2: Type: text/html, Size: 32553 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2015-12-31 16:33 ` Andrey Kuzmin
@ 2016-01-01 12:43 ` Thierry BERTAUD
2016-01-01 12:46 ` Thierry BERTAUD
2016-01-01 12:54 ` Andrey Kuzmin
0 siblings, 2 replies; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-01 12:43 UTC (permalink / raw)
To: Andrey Kuzmin; +Cc: fio@vger.kernel.org, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]
Hello Andrey,
I tried different options to have symbols for gdb, your suggestion and others with CFLAGS ( -g, -ggdb) but any seems the right.
> Try 'configure --disable-optimizations'.
I tried :
# ./configure --disable-optimizations
# gmake CC=gcc
I have with gdb the same output:
(gdb) info threads
Id Target Id Frame
* 1 process 54526172 0xd051d470 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 54526172)]
#0 0xd051d470 in ?? ()
(gdb) bt
#0 0xd051d470 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd011f3e8 in ?? ()
#3 0xd011ee38 in ?? ()
#4 0x1001ed50 in ?? ()
#5 0x1001fcbc in ?? ()
#6 0x10020280 in ?? ()
#7 0x10000efc in ?? ()
#8 0x100001bc in ?? ()
(gdb)
After quit gdb the fio load agaisnt CPU come back.
I changed to:
# ./configure
# CC=gcc CFLAGS='-g' gmake
# gdb -p 33095880
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 33095880
0xd0119598 in ?? ()
(gdb) info threads
Id Target Id Frame
* 1 process 33095880 0xd0119598 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 33095880)]
#0 0xd0119598 in ?? ()
(gdb) bt
#0 0xd0119598 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd0119bc0 in ?? ()
#3 0xd0118f28 in ?? ()
#4 0x100625dc in ?? ()
#5 0x1006482c in ?? ()
#6 0x10064b90 in ?? ()
#7 0xd0505c10 in ?? ()
#8 0x00000000 in ?? ()
(gdb)
I changed to:
# ./configure
# CC=gcc CFLAGS='-ggdb' gmake
# gdb -p 12124366
GNU gdb (GDB) 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 12124366
0xd051d470 in ?? ()
(gdb) info threads
Id Target Id Frame
* 1 process 12124366 0xd051d470 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 12124366)]
#0 0xd051d470 in ?? ()
(gdb) bt
#0 0xd051d470 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd011f3e8 in ?? ()
#3 0xd011ee38 in ?? ()
#4 0x10069230 in ?? ()
#5 0x10069644 in ?? ()
#6 0x100004c4 in ?? ()
#7 0x100001bc in ?? ()
(gdb)
Regards,
Thierry
Regards,
Thierry
[-- Attachment #2: Type: text/html, Size: 7738 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 12:43 ` Thierry BERTAUD
@ 2016-01-01 12:46 ` Thierry BERTAUD
2016-01-01 12:54 ` Andrey Kuzmin
1 sibling, 0 replies; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-01 12:46 UTC (permalink / raw)
To: Andrey Kuzmin; +Cc: fio@vger.kernel.org, Jens Axboe
Hello Andrey,
I tried different options to have symbols for gdb, your suggestion and others with CFLAGS ( -g, -ggdb) but any seems the right.
> Try 'configure --disable-optimizations'.
Note: After quit gdb the fio load agaisnt CPU come back.
I tried :
# ./configure --disable-optimizations
# gmake CC=gcc
I have with gdb the same output:
(gdb) info threads
Id Target Id Frame
* 1 process 54526172 0xd051d470 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 54526172)]
#0 0xd051d470 in ?? ()
(gdb) bt
#0 0xd051d470 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd011f3e8 in ?? ()
#3 0xd011ee38 in ?? ()
#4 0x1001ed50 in ?? ()
#5 0x1001fcbc in ?? ()
#6 0x10020280 in ?? ()
#7 0x10000efc in ?? ()
#8 0x100001bc in ?? ()
(gdb)
I changed to:
# ./configure
# CC=gcc CFLAGS='-g' gmake
Attaching to process 33095880
0xd0119598 in ?? ()
(gdb) info threads
Id Target Id Frame
* 1 process 33095880 0xd0119598 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 33095880)]
#0 0xd0119598 in ?? ()
(gdb) bt
#0 0xd0119598 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd0119bc0 in ?? ()
#3 0xd0118f28 in ?? ()
#4 0x100625dc in ?? ()
#5 0x1006482c in ?? ()
#6 0x10064b90 in ?? ()
#7 0xd0505c10 in ?? ()
#8 0x00000000 in ?? ()
(gdb)
I changed to:
# ./configure
# CC=gcc CFLAGS='-ggdb' gmake
0xd051d470 in ?? ()
(gdb) info threads
Id Target Id Frame
* 1 process 12124366 0xd051d470 in ?? ()
(gdb) thread 1
[Switching to thread 1 (process 12124366)]
#0 0xd051d470 in ?? ()
(gdb) bt
#0 0xd051d470 in ?? ()
#1 0xffffffff in ?? ()
#2 0xd011f3e8 in ?? ()
#3 0xd011ee38 in ?? ()
#4 0x10069230 in ?? ()
#5 0x10069644 in ?? ()
#6 0x100004c4 in ?? ()
#7 0x100001bc in ?? ()
(gdb)
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 12:43 ` Thierry BERTAUD
2016-01-01 12:46 ` Thierry BERTAUD
@ 2016-01-01 12:54 ` Andrey Kuzmin
2016-01-01 14:00 ` Thierry BERTAUD
1 sibling, 1 reply; 24+ messages in thread
From: Andrey Kuzmin @ 2016-01-01 12:54 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 4027 bytes --]
On Jan 1, 2016 15:43, "Thierry BERTAUD" <tbertaud@conforama.fr> wrote:
>
> Hello Andrey,
>
>
> I tried different options to have symbols for gdb, your suggestion and
others with CFLAGS ( -g, -ggdb) but any seems the right.
>
>
> > Try 'configure --disable-optimizations'.
>
> I tried :
>
> # ./configure --disable-optimizations
>
> # gmake CC=gcc
>
It's best to run make clean, configure --disable-optimizations, and then
make. Also, check if you have libpthreads-debug installed so that gdb
tracks libpthread symbols.
Regards,
A.
> I have with gdb the same output:
>
> (gdb) info threads
> Id Target Id Frame
> * 1 process 54526172 0xd051d470 in ?? ()
> (gdb) thread 1
> [Switching to thread 1 (process 54526172)]
> #0 0xd051d470 in ?? ()
> (gdb) bt
> #0 0xd051d470 in ?? ()
> #1 0xffffffff in ?? ()
> #2 0xd011f3e8 in ?? ()
> #3 0xd011ee38 in ?? ()
> #4 0x1001ed50 in ?? ()
> #5 0x1001fcbc in ?? ()
> #6 0x10020280 in ?? ()
> #7 0x10000efc in ?? ()
> #8 0x100001bc in ?? ()
> (gdb)
>
> After quit gdb the fio load agaisnt CPU come back.
>
> I changed to:
>
> # ./configure
> # CC=gcc CFLAGS='-g' gmake
>
> # gdb -p 33095880
> GNU gdb (GDB) 7.8.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "powerpc-ibm-aix5.1.0.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> Attaching to process 33095880
> 0xd0119598 in ?? ()
> (gdb) info threads
> Id Target Id Frame
> * 1 process 33095880 0xd0119598 in ?? ()
> (gdb) thread 1
> [Switching to thread 1 (process 33095880)]
> #0 0xd0119598 in ?? ()
> (gdb) bt
> #0 0xd0119598 in ?? ()
> #1 0xffffffff in ?? ()
> #2 0xd0119bc0 in ?? ()
> #3 0xd0118f28 in ?? ()
> #4 0x100625dc in ?? ()
> #5 0x1006482c in ?? ()
> #6 0x10064b90 in ?? ()
> #7 0xd0505c10 in ?? ()
> #8 0x00000000 in ?? ()
> (gdb)
>
> I changed to:
>
> # ./configure
> # CC=gcc CFLAGS='-ggdb' gmake
>
>
> # gdb -p 12124366
> GNU gdb (GDB) 7.8.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "powerpc-ibm-aix5.1.0.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> Attaching to process 12124366
> 0xd051d470 in ?? ()
> (gdb) info threads
> Id Target Id Frame
> * 1 process 12124366 0xd051d470 in ?? ()
> (gdb) thread 1
> [Switching to thread 1 (process 12124366)]
> #0 0xd051d470 in ?? ()
> (gdb) bt
> #0 0xd051d470 in ?? ()
> #1 0xffffffff in ?? ()
> #2 0xd011f3e8 in ?? ()
> #3 0xd011ee38 in ?? ()
> #4 0x10069230 in ?? ()
> #5 0x10069644 in ?? ()
> #6 0x100004c4 in ?? ()
> #7 0x100001bc in ?? ()
> (gdb)
>
> Regards,
>
> Thierry
>
>
>
> Regards,
>
> Thierry
[-- Attachment #2: Type: text/html, Size: 5941 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 12:54 ` Andrey Kuzmin
@ 2016-01-01 14:00 ` Thierry BERTAUD
2016-01-01 14:04 ` Andrey Kuzmin
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-01 14:00 UTC (permalink / raw)
To: Andrey Kuzmin; +Cc: fio@vger.kernel.org, Jens Axboe
Andrey,
> It's best to run make clean, configure --disable-optimizations, and then make. Also, check if you have libpthreads-debug installed so that gdb tracks libpthread symbols.
I have tyhe pthread debug library:
# find / -name "libpth*"
/usr/ccs/lib/inst_updt/libpthdebug.a
/usr/ccs/lib/libp/libpthread.a
/usr/ccs/lib/libp/libpthreads.a
/usr/ccs/lib/libp/libpthreads_compat.a
/usr/ccs/lib/libpthdebug.a <==== here
/usr/ccs/lib/libpthread.a
/usr/ccs/lib/libpthreads.a
/usr/ccs/lib/libpthreads_compat.a
/usr/ccs/lib/libt/libpthread.a
/usr/ccs/lib/libt/libpthreads.a
/usr/ccs/lib/libt/libpthreads_compat.a
/usr/ccs/lib/perf/libpthread.a
/usr/ccs/lib/perf/libpthreads.a
/usr/ccs/lib/perf/libpthreads_compat.a
/usr/lib/libpthdebug.a <==== here
/usr/lib/libpthread.a
/usr/lib/libpthreads.a
/usr/lib/libpthreads_compat.a
#
The tools procstack (from man: Prints the hexadecimal addresses and symbolic names for all the threads in the process) from ibmprocstack can read the thread:
# ldd /usr/bin/procstack
/usr/bin/procstack needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libptools.a(shr64.o)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libpthreads.a(shr_xpg5_64.o)
/usr/lib/libC.a(shr_64.o)
/usr/lib/libodm.a(shr_64.o)
/usr/lib/libtrace.a(shr_64.o)
/usr/lib/libC.a(shrcore_64.o)
/usr/lib/libC.a(ansicore_64.o)
#
Below output from procstak:
# procstack 8913040
8913040: /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
---------- tid# 84803767 (pthread ID: 1) ----------
0xd051d470 _p_nsleep(??, ??) + 0x10
0xd011f3e4 nsleep(??, ??) + 0xe4
0xd011ee34 usleep(??) + 0xd4
0x1006922c run_threads(??) + 0xaf0
0x10069640 fio_backend(??) + 0x174
0x100004c0 main(??, ??, ??) + 0xd8
0x100001b8 __start() + 0x68
---------- tid# 110231691 (pthread ID: 258) ----------
0xd0119598 __ioctl(??, ??, ??, ??, ??) + 0xb8
0xd0119bbc ioctl(0x1, 0x5800, 0x0, 0x0, 0x0, 0x4, 0x0, 0x3009f6a8) + 0x3c
0xd0118f24 isatty(??) + 0x24
0x100625d8 calc_thread_status(??, ??) + 0xb8
0x10064828 print_thread_status() + 0x58
0x10064b8c helper_thread_main(??) + 0x120
0xd0505c0c _pthread_body(??) + 0xec
#
Does the output from procstak is useful?
How to configure 'configure' or 'make' to use the /usr/lib/libpthdebug.a ?
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 14:00 ` Thierry BERTAUD
@ 2016-01-01 14:04 ` Andrey Kuzmin
2016-01-01 14:49 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Andrey Kuzmin @ 2016-01-01 14:04 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 2887 bytes --]
On Jan 1, 2016 17:00, "Thierry BERTAUD" <tbertaud@conforama.fr> wrote:
>
> Andrey,
>
> > It's best to run make clean, configure --disable-optimizations, and
then make. Also, check if you have libpthreads-debug installed so that gdb
tracks libpthread symbols.
>
> I have tyhe pthread debug library:
> # find / -name "libpth*"
> /usr/ccs/lib/inst_updt/libpthdebug.a
> /usr/ccs/lib/libp/libpthread.a
> /usr/ccs/lib/libp/libpthreads.a
> /usr/ccs/lib/libp/libpthreads_compat.a
> /usr/ccs/lib/libpthdebug.a <==== here
> /usr/ccs/lib/libpthread.a
> /usr/ccs/lib/libpthreads.a
> /usr/ccs/lib/libpthreads_compat.a
> /usr/ccs/lib/libt/libpthread.a
> /usr/ccs/lib/libt/libpthreads.a
> /usr/ccs/lib/libt/libpthreads_compat.a
> /usr/ccs/lib/perf/libpthread.a
> /usr/ccs/lib/perf/libpthreads.a
> /usr/ccs/lib/perf/libpthreads_compat.a
> /usr/lib/libpthdebug.a <==== here
> /usr/lib/libpthread.a
> /usr/lib/libpthreads.a
> /usr/lib/libpthreads_compat.a
> #
>
> The tools procstack (from man: Prints the hexadecimal addresses and
symbolic names for all the threads in the process) from ibmprocstack can
read the thread:
> # ldd /usr/bin/procstack
> /usr/bin/procstack needs:
> /usr/lib/libc.a(shr_64.o)
> /usr/lib/libptools.a(shr64.o)
> /unix
> /usr/lib/libcrypt.a(shr_64.o)
> /usr/lib/libpthreads.a(shr_xpg5_64.o)
> /usr/lib/libC.a(shr_64.o)
> /usr/lib/libodm.a(shr_64.o)
> /usr/lib/libtrace.a(shr_64.o)
> /usr/lib/libC.a(shrcore_64.o)
> /usr/lib/libC.a(ansicore_64.o)
> #
>
> Below output from procstak:
> # procstack 8913040
> 8913040: /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
> ---------- tid# 84803767 (pthread ID: 1) ----------
> 0xd051d470 _p_nsleep(??, ??) + 0x10
> 0xd011f3e4 nsleep(??, ??) + 0xe4
> 0xd011ee34 usleep(??) + 0xd4
> 0x1006922c run_threads(??) + 0xaf0
> 0x10069640 fio_backend(??) + 0x174
> 0x100004c0 main(??, ??, ??) + 0xd8
> 0x100001b8 __start() + 0x68
> ---------- tid# 110231691 (pthread ID: 258) ----------
> 0xd0119598 __ioctl(??, ??, ??, ??, ??) + 0xb8
> 0xd0119bbc ioctl(0x1, 0x5800, 0x0, 0x0, 0x0, 0x4, 0x0, 0x3009f6a8) +
0x3c
> 0xd0118f24 isatty(??) + 0x24
> 0x100625d8 calc_thread_status(??, ??) + 0xb8
> 0x10064828 print_thread_status() + 0x58
> 0x10064b8c helper_thread_main(??) + 0x120
> 0xd0505c0c _pthread_body(??) + 0xec
> #
>
> Does the output from procstak is useful?
It's the default thread that sleeps in the run_threads loop, check one of
the i/o threads.
> How to configure 'configure' or 'make' to use the /usr/lib/libpthdebug.a ?
>
GDB should pick up the right one itself when you build fio with
optimization disabled.
Regards,
A.
> Regards,
> Thierry
>
>
[-- Attachment #2: Type: text/html, Size: 3813 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 14:04 ` Andrey Kuzmin
@ 2016-01-01 14:49 ` Thierry BERTAUD
2016-01-01 16:19 ` Andrey Kuzmin
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-01 14:49 UTC (permalink / raw)
To: Andrey Kuzmin; +Cc: fio@vger.kernel.org, Jens Axboe
Andrey,
>> How to configure 'configure' or 'make' to use the /usr/lib/libpthdebug.a ?
> GDB should pick up the right one itself when you build fio with optimization disabled.
How to have sure gcc take the option via configure : configure --disable-optimizations because it's not work for me with gdb?
>> Does the output from procstak is useful?
> It's the default thread that sleeps in the run_threads loop, check one of the i/o threads.
This the info from AIX debugger dbx:
# dbx -a 8913040
Waiting to attach to process 8913040 ...
Successfully attached to fio.
Type 'help' for help.
reading symbolic information ...
stopped in _p_nsleep at 0xd051d470 ($t1)
0xd051d470 (_p_nsleep+0x10) 80410014 lwz r2,0x14(r1)
(dbx) thread
thread state-k wchan state-u k-tid mode held scope function
>$t1 run running 84803767 k no sys _p_nsleep
$t2 run running 110231691 u no sys __ioctl
(dbx) thread info 1
thread state-k wchan state-u k-tid mode held scope function
>$t1 run running 84803767 k no sys _p_nsleep
general:
pthread addr = 0x3009590c size = 0x290
vp addr = 0x30097950 size = 0x2e0
thread errno = 0
start pc = 0x100003e8
joinable = yes
pthread_t = 1
scheduler:
kernel =
user = 1 (other)
nice = 60
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x2df23000 size = 0x1fffbd0
limit = 0x2ff22bd0
sp = 0x2ff20970
(dbx) thread info 2
thread state-k wchan state-u k-tid mode held scope function
$t2 run running 110231691 u no sys __ioctl
general:
pthread addr = 0x31020cf0 size = 0x290
vp addr = 0x31023550 size = 0x2e0
thread errno = 0
start pc = 0x30071fa8
joinable = yes
pthread_t = 102
scheduler:
kernel =
user = 1 (other)
nice = 60
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x31008000 size = 0x18000
limit = 0x31020b90
sp = 0x31020820
(dbx) thread current 1
(dbx) where
_p_nsleep(??, ??) at 0xd051d470
raise.nsleep(??, ??) at 0xd011f3e4
usleep(??) at 0xd011ee34
unnamed block in run_threads(sk_out = (nil)), line 1957 in "backend.c"
unnamed block in run_threads(sk_out = (nil)), line 1957 in "backend.c"
run_threads(sk_out = (nil)), line 1957 in "backend.c"
fio_backend(sk_out = (nil)), line 2381 in "backend.c"
main(argc = 2, argv = @0x2ff22c50, envp = ??), line 60 in "fio.c"
(dbx)
(dbx) where
__ioctl(??, ??, ??, ??, ??) at 0xd0119598
ioctl(0x1, 0x5800, 0x0, 0x0, 0x0, 0x4, 0x0, 0x3009f6a8) at 0xd0119bbc
isatty(??) at 0xd0118f24
internal error: unexpected value 66 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '299,128,16;flags:-7,144,16;timerid:241,160,32;eventlist:108,192,32;timeout:432,224,128;func:441=*442=f-11,352,32;t_union:437,384,32;ipri:-1,416,32;tof:441,448,32;;'
internal error: 1283-228 expected char ';', found ',16;flags:-7,144,16;timerid:241,160,32;eventlist:108,192,32;timeout:432,224,128;func:441=*442=f-11,352,32;t_union:437,384,32;ipri:-1,416,32;tof:441,448,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sposix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
internal error: 1283-228 expected char ',', found 'posix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
internal error: 1283-228 expected char ';', found 'osix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'suthread:,0,32;userp:528=*529=xsuser:,32,32;;'
internal error: 1283-228 expected char ',', found 'uthread:,0,32;userp:528=*529=xsuser:,32,32;;'
internal error: 1283-228 expected char ';', found 'thread:,0,32;userp:528=*529=xsuser:,32,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;userp:528=*529=xsuser:,32,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'suser:,32,32;;'
internal error: 1283-228 expected char ',', found 'user:,32,32;;'
internal error: 1283-228 expected char ';', found 'ser:,32,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '32,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'spvthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ',', found 'pvthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ';', found 'vthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ',', found 'proc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ';', found 'roc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'ssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ',', found 'sigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ';', found 'igcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'srun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ',', found 'run_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ';', found 'un_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'spfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ',', found 'pfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: 1283-228 expected char ';', found 'fh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'spvproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: 1283-228 expected char ',', found 'pvproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: 1283-228 expected char ';', found 'vproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'svnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: 1283-228 expected char ',', found 'vnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: 1283-228 expected char ';', found 'node:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sksiginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
internal error: 1283-228 expected char ',', found 'ksiginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
internal error: 1283-228 expected char ';', found 'siginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sptipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ',', found 'ptipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ';', found 'tipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ',', found 'prochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ';', found 'rochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'svm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ',', found 'vm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: 1283-228 expected char ';', found 'm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 's__LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
internal error: 1283-228 expected char ',', found '__LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
internal error: 1283-228 expected char ';', found '_LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 's_LC_locale_objhdl:,64,32;;'
internal error: 1283-228 expected char ',', found '_LC_locale_objhdl:,64,32;;'
internal error: 1283-228 expected char ';', found 'LC_locale_objhdl:,64,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '64,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sfio_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ',', found 'fio_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ';', found 'io_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ',', found 'fio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ';', found 'io_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'saxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ',', found 'axmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: 1283-228 expected char ';', found 'xmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sdisk_util:,2496,32;;'
internal error: 1283-228 expected char ',', found 'disk_util:,2496,32;;'
internal error: 1283-228 expected char ';', found 'isk_util:,2496,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '2496,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: unexpected value 66 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '-11,192,32;aio_return:68,224,32;aio_errno:-1,256,32;aio_nbytes:63,288,32;aio_reqprio:-1,320,32;aio_sigevent:638,352,160;aio_word2:-1,512,32;aio_fp:-1,544,32;aio_handle:895,576,32;aio_reserved:901=ar111;0;2;66,608,96;aio_sigev_tid:108,704,32;aio_pad:-1,736,32;;'
internal error: 1283-228 expected char ';', found ',32;aio_return:68,224,32;aio_errno:-1,256,32;aio_nbytes:63,288,32;aio_reqprio:-1,320,32;aio_sigevent:638,352,160;aio_word2:-1,512,32;aio_fp:-1,544,32;aio_handle:895,576,32;aio_reserved:901=ar111;0;2;66,608,96;aio_sigev_tid:108,704,32;aio_pad:-1,736,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sworkqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
internal error: 1283-228 expected char ',', found 'workqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
internal error: 1283-228 expected char ';', found 'orkqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'ssk_out:,1088,32;;'
internal error: 1283-228 expected char ',', found 'sk_out:,1088,32;;'
internal error: 1283-228 expected char ';', found 'k_out:,1088,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '1088,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sthread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
internal error: 1283-228 expected char ',', found 'thread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
internal error: 1283-228 expected char ';', found 'hread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
internal error: unexpected value 66 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '-1,1696,32;;'
internal error: 1283-228 expected char ';', found ',32;;'
internal error: 1283-232 index("32;;", ':') failed
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sio_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
internal error: 1283-228 expected char ',', found 'io_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
internal error: 1283-228 expected char ';', found 'o_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sclient_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: 1283-228 expected char ',', found 'client_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: 1283-228 expected char ';', found 'lient_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: 1283-228 expected char ',', found 'client_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: 1283-228 expected char ';', found 'lient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sjson_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ',', found 'json_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ';', found 'son_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ',', found 'json_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ';', found 'son_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ',', found 'json_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: 1283-228 expected char ';', found 'son_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '0,32;parent_array:1141,0,32;;,160,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sthread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: 1283-228 expected char ',', found 'thread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: 1283-228 expected char ';', found 'hread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: unexpected value 120 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found 'sfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: 1283-228 expected char ',', found 'fio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: 1283-228 expected char ';', found 'io_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
internal error: unexpected value 44 at line 5176 in file stabstring.c
internal error: 1283-228 expected char ',', found '436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
calc_thread_status(je = 0x3009f6a8, force = ??), line 361 in "eta.c"
unnamed block in print_thread_status(), line 605 in "eta.c"
unnamed block in print_thread_status(), line 605 in "eta.c"
print_thread_status(), line 605 in "eta.c"
unnamed block in helper_thread_main(data = ??), line 2318 in "backend.c"
helper_thread_main(data = ??), line 2318 in "backend.c"
(dbx)
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 14:49 ` Thierry BERTAUD
@ 2016-01-01 16:19 ` Andrey Kuzmin
2016-01-01 17:31 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Andrey Kuzmin @ 2016-01-01 16:19 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: fio, Jens Axboe
[-- Attachment #1: Type: text/plain, Size: 91003 bytes --]
On Jan 1, 2016 17:49, "Thierry BERTAUD" <tbertaud@conforama.fr> wrote:
>
> Andrey,
>
> >> How to configure 'configure' or 'make' to use the
/usr/lib/libpthdebug.a ?
> > GDB should pick up the right one itself when you build fio with
optimization disabled.
> How to have sure gcc take the option via configure : configure
--disable-optimizations because it's not work for me with gdb?
>
No options involved, --disable-optimizations simply doesn't strip debug
info (as gcc usually does with -O3). Unless gcc on aix does something
weird, that should just happen by itself.
> >> Does the output from procstak is useful?
> > It's the default thread that sleeps in the run_threads loop, check one
of the i/o threads.
> This the info from AIX debugger dbx:
>
You reported 128 threads in the original problem report. Is the below a
differnt one?
> # dbx -a 8913040
> Waiting to attach to process 8913040 ...
> Successfully attached to fio.
> Type 'help' for help.
> reading symbolic information ...
> stopped in _p_nsleep at 0xd051d470 ($t1)
> 0xd051d470 (_p_nsleep+0x10) 80410014 lwz r2,0x14(r1)
> (dbx) thread
> thread state-k wchan state-u k-tid mode held scope
function
> >$t1 run running 84803767 k no sys
_p_nsleep
> $t2 run running 110231691 u no sys
__ioctl
Thread 2 sleeps in the ioctl that the trace reports as failed. May be this
is the root cause behind your issue?
Regards,
A.
> (dbx) thread info 1
> thread state-k wchan state-u k-tid mode held scope
function
> >$t1 run running 84803767 k no sys
_p_nsleep
>
>
> general:
> pthread addr = 0x3009590c size = 0x290
> vp addr = 0x30097950 size = 0x2e0
> thread errno = 0
> start pc = 0x100003e8
> joinable = yes
> pthread_t = 1
> scheduler:
> kernel =
> user = 1 (other)
> nice = 60
> event :
> event = 0x0
> cancel = enabled, deferred, not pending
> stack storage:
> base = 0x2df23000 size = 0x1fffbd0
> limit = 0x2ff22bd0
> sp = 0x2ff20970
> (dbx) thread info 2
> thread state-k wchan state-u k-tid mode held scope
function
> $t2 run running 110231691 u no sys
__ioctl
>
>
> general:
> pthread addr = 0x31020cf0 size = 0x290
> vp addr = 0x31023550 size = 0x2e0
> thread errno = 0
> start pc = 0x30071fa8
> joinable = yes
> pthread_t = 102
> scheduler:
> kernel =
> user = 1 (other)
> nice = 60
> event :
> event = 0x0
> cancel = enabled, deferred, not pending
> stack storage:
> base = 0x31008000 size = 0x18000
> limit = 0x31020b90
> sp = 0x31020820
> (dbx) thread current 1
> (dbx) where
> _p_nsleep(??, ??) at 0xd051d470
> raise.nsleep(??, ??) at 0xd011f3e4
> usleep(??) at 0xd011ee34
> unnamed block in run_threads(sk_out = (nil)), line 1957 in "backend.c"
> unnamed block in run_threads(sk_out = (nil)), line 1957 in "backend.c"
> run_threads(sk_out = (nil)), line 1957 in "backend.c"
> fio_backend(sk_out = (nil)), line 2381 in "backend.c"
> main(argc = 2, argv = @0x2ff22c50, envp = ??), line 60 in "fio.c"
> (dbx)
> (dbx) where
> __ioctl(??, ??, ??, ??, ??) at 0xd0119598
> ioctl(0x1, 0x5800, 0x0, 0x0, 0x0, 0x4, 0x0, 0x3009f6a8) at 0xd0119bbc
> isatty(??) at 0xd0118f24
> internal error: unexpected value 66 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'299,128,16;flags:-7,144,16;timerid:241,160,32;eventlist:108,192,32;timeout:432,224,128;func:441=*442=f-11,352,32;t_union:437,384,32;ipri:-1,416,32;tof:441,448,32;;'
> internal error: 1283-228 expected char ';', found
',16;flags:-7,144,16;timerid:241,160,32;eventlist:108,192,32;timeout:432,224,128;func:441=*442=f-11,352,32;t_union:437,384,32;ipri:-1,416,32;tof:441,448,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sposix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
> internal error: 1283-228 expected char ',', found
'posix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
> internal error: 1283-228 expected char ';', found
'osix_tmr:,0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;prev:444,32,32;id:241,64,32;timerid:97,96,32;flags:-7,128,16;value:-8,160,32;interval:-8,192,32;func:441,224,32;data:73,256,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'suthread:,0,32;userp:528=*529=xsuser:,32,32;;'
> internal error: 1283-228 expected char ',', found
'uthread:,0,32;userp:528=*529=xsuser:,32,32;;'
> internal error: 1283-228 expected char ';', found
'thread:,0,32;userp:528=*529=xsuser:,32,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;userp:528=*529=xsuser:,32,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found 'suser:,32,32;;'
> internal error: 1283-228 expected char ',', found 'user:,32,32;;'
> internal error: 1283-228 expected char ';', found 'ser:,32,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found '32,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'spvthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ',', found
'pvthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ';', found
'vthread:,256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'256,32;t_procp:533=*534=xsproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sproc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ',', found
'proc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ';', found
'roc:,288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'288,32;t_uaddress:525,320,64;t_ulock:-4,384,32;t_uchan:-4,416,32;t_userdata:-4,448,32;t_cv:-4,480,32;t_stackp:74,512,32;t_scp:535=*536=xssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'ssigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ',', found
'sigcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ';', found
'igcontext:,544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'544,32;t_kscp:535,576,32;t_wchan2:74,608,32;t_wchan1:74,640,32;t_eventlist:537=*530,672,32;t_pevent:241,704,32;t_wevent:241,736,32;t_usid:175,768,32;t_swchan:74,800,32;t_slist:537,832,32;t_lockowner:537,864,32;t_wchan1sid:175,896,32;t_wchan1offset:240,928,32;t_fsflags:-1,960,32;t_result:-1,992,32;t_polevel:-1,1024,32;t_threadcontrolp:234,1088,64;t_prior:537,1152,32;t_next:537,1184,32;t_graphics:90,1216,32;t_run_queue:538=*539=xsrun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'srun_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ',', found
'run_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ';', found
'un_queue:,1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'1248,32;t_time_start:268,1280,64;t_spurr_time_start:268,1344,64;t_affinity_ts:61,1408,32;t_dispct:241,1440,32;t_near_dispct:241,1472,32;t_far_dispct:241,1504,32;t_fpuct:241,1536,32;t_allowed_cpus:521,1600,512;t_adsp_flags:239,2112,16;t_ticks:239,2128,16;t_pri_band:-2,2144,8;t_savsigno:-2,2152,8;t_prefunnel_cpu:299,2160,16;t_cpuid:299,2176,16;t_scpuid:299,2192,16;t_cpu2:239,2208,16;t_cpu:239,2224,16;t_norun_secs:-2,2240,8;t_policy:-2,2248,8;t_lockpri:-2,2256,8;t_wakepri:-2,2264,8;t_time:238,2272,8;t_sav_pri:-2,2280,8;t_ceiling:-2,2288,8;t_nice:-2,2296,8;t_sigproc:-2,2304,8;t_cursig:-2,2312,8;t_whystop:-2,2320,8;t_dispatch_hist:-5,2328,8;t_sig:129,2336,64;t_sigmask:129,2400,64;t_chkblock:-1,2464,32;t_whatstop:239,2496,16;t_chkerror:239,2512,16;t_wlm_charge:107,2528,32;t_starttime:429,2560,64;t_minflt:241,2624,32;t_majflt:241,2656,32;t_wlmevtcnt:240,2688,32;t_trcgenct:240,2720,32;t_cpu_tb:240,2752,32;t_ldispcpu:299,2784,16;t_rt_ticks:240,2816,32;t_interval_start:-32,2880,64;t_cputime:-32,2944,64;t_scputime:-32,3008,64;pth_utime:-32,3072,64;pth_stime:-32,3136,64;t_credp:540=*493,3200,32;t_home_sradid:301,3232,16;t_borrow_pri:-7,3248,16;t_homecount:-1,3264,32;t_home_rq:538,3296,32;t_rs_attinfo:90,3328,32;t_rehome_tb:-10,3360,32;t_pad0:-10,3392,32;t_ulock_listp:90,3424,32;t_ipc_data:-10,3456,32;t_evm_tlv_struct:90,3488,32;t_io_priority:519,3520,16;t_flags3:239,3536,16;t_delw:541=*542=xspfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'spfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ',', found
'pfh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: 1283-228 expected char ';', found
'fh_delwait:,3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'3552,32;t_waitTm:-32,3584,64;t_advrset_maskp:543=*521,3648,32;t_runnable_start:-32,3712,64;t_runnable_total:-32,3776,64;t_running_start:-32,3840,64;t_running_total:-32,3904,64;t_borrowed_util:-8,3968,32;t_dispatch_count:-8,4000,32;t_attachmap:-8,4032,32;t_iprb_snpshtd_ptr:90,4064,32;t_eyec:520,4096,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'spvproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: 1283-228 expected char ',', found
'pvproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: 1283-228 expected char ';', found
'vproc:,96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'96,32;tv_threadlist:546=s8prevthread:531,0,32;nextthread:531,32,32;;,128,64;tv_lock_d:318,192,32;tv_synch:108,224,32;tv_tsleep:108,256,32;tv_wchan:74,288,32;tv_flags:240,320,32;tv_flags2:240,352,32;tv_totalcputime:268,384,64;tv_totalscputime:268,448,64;__tv_boost_u:547=u4tv_boost_cnt:548=ar111;0;1;-3,0,32;tv_boosted:-1,0,32;;,512,32;tv_affinity:299,544,16;tv_state:-2,560,8;tv_wtype:-2,568,8;tv_pmcontext:90,576,32;tv_pri:-2,608,8;tv_pad1:-2,616,8;tv_intercept_count:-3,624,16;tv_pad2:-1,640,32;tv_procfsvn:549=*550=xsvnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'svnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: 1283-228 expected char ',', found
'vnode:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: 1283-228 expected char ';', found
'node:,672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'672,32;tv_chkfile:90,704,32;tv_prbase:-32,768,64;tv_prpinned:551=*-32,832,32;tv_prflags:239,864,16;tv_prbufcount:239,880,16;tv_ptid:240,896,32;tv_wlm:-1,928,32;tv_class:269,960,32;tv_rset:90,992,32;tv_ptag:74,1024,32;tv_hmcr_lock:318,1056,32;tv_itags_ptr:90,1088,32;tv_itags:172,1120,64;tv_eyec:520,1184,32;tv_pad:260,1216,256;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sksiginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
> internal error: 1283-228 expected char ',', found
'ksiginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
> internal error: 1283-228 expected char ';', found
'siginfo:,1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'1536,32;p_size:-32,1600,64;p_minflt:-32,1664,64;p_majflt:-32,1728,64;p_repage:-31,1792,64;p_pctcpu:240,1856,32;p_sched_count:-1,1888,32;p_cpticks:-3,1920,16;p_msgcnt:-3,1936,16;p_majfltsec:240,1952,32;p_nice:-2,1984,8;p_sched_pri:-2,1992,8;sysidmod:-2,2000,8;unused1:-2,2008,8;p_chkblock:-1,2016,32;p_chkfile:90,2048,32;p_prtrcset:90,2080,32;p_lgpage:90,2112,32;p_rt_timer:593=ar111;0;31;444,2144,1024;p_clock:448,3168,64;p_io_priority:239,3232,16;p_memory_lbolt:61,3264,32;p_interval_start:-32,3328,64;p_acct:585,3392,640;p_diskiostat:590,4032,384;p_wlmthrds:-1,4416,32;p_wlmocthrds:-1,4448,32;p_thrsetcnt:-1,4480,32;p_thsradcnt:-1,4512,32;p_ptbpt:90,4544,32;p_ptsynch:108,4576,32;p_entry_probes:239,4608,16;p_exit_probes:239,4624,16;p_uprobe:90,4640,32;p_etrc_procd_ptr:90,4672,32;p_forktime:-32,4736,64;p_aso:587,4800,1152;p_eyec:578,5952,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sptipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ',', found
'ptipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ';', found
'tipc:,1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'1920,32;pv_dblist:544,1952,32;pv_dbnext:544,1984,32;pv_procfsvn:549,2016,32;pv_sched_next:544,2048,32;pv_sched_back:544,2080,32;pv_procp:533,2112,32;pv_child:544,2144,32;pv_siblings:544,2176,32;pv_uidinfo:580,2208,32;pv_ganchor:544,2240,32;pv_pgrpl:544,2272,32;pv_pgrpb:544,2304,32;pv_ttyl:544,2336,32;pv_cridnext:544,2368,32;pv_session_lock:318,2400,32;pv_parent_lock:318,2432,32;pv_lock_d:318,2464,32;pv_threadlist:531,2496,32;pv_synch:108,2528,32;pv_mempools:597=ar111;0;0;11,2560,64;pv_rset:90,2624,32;pv_nframes:-32,2688,64;pv_npsblks:-32,2752,64;pv_nvpages:-32,2816,64;pv_asyncio:598=*582,2880,32;pv_chksynch:108,2912,32;pv_chktv:531,2944,32;pv_totalcputime:-32,3008,64;pv_totalscputime:-32,3072,64;pv_xcputime:-32,3136,64;pv_xscputime:-32,3200,64;pv_totaldiskio:-32,3264,64;pv_termtime:61,3328,32;pv_gcipc:90,3360,32;pv_pri:-2,3392,8;pv_policy:-2,3400,8;pv_acctflags:-3,3408,16;pv_projid:-1,3424,32;pv_subproj:-32,3456,64;pv_sprojid:-1,3520,32;pv_static_environment:-1,3552,32;pv_app_dev:81,3584,64;pv_app_ino:77,3648,64;pv_app_gen:66,3712,32;pv_ewlmproc:90,3744,32;pv_handlers:599=*600=xsprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sprochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ',', found
'prochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ';', found
'rochr:,3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'3776,32;pv_cached_credp:90,3808,32;pv_handlers_lock:318,3840,32;pv_wlm_nvpages:-32,3904,64;pv_bopflags:239,3968,16;pv_bopmcnt:239,3984,16;pv_flag3:239,4000,16;pv_flag3e:238,4016,8;pv_flag3p:238,4024,8;pv_vmm_xproc_data:601=*602=xsvm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'svm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ',', found
'vm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: 1283-228 expected char ';', found
'm_xproc_data:,4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'4032,32;pv_eyec:578,4064,32;pv_kcid:577,4096,16;pv_secflag:-3,4112,16;pv_padint:-1,4128,32;pv_mcrp:90,4160,32;pv_p11_ctx:90,4192,32;pv_nodegen:241,4224,32;pv_af1:603=u32pv_aff_stat_int:163,0,256;pv_aff_stat_long:604=ar111;0;3;-4,0,128;;,4256,256;pv_aff_stat_ext:90,4512,32;pv_ncpages:11,4544,64;pv_aso_state:66,4608,32;pv_padint2:-1,4640,32;pv_sib_back:544,4672,32;pv_xproc_synch:108,4704,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
's__LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
> internal error: 1283-228 expected char ',', found
'__LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
> internal error: 1283-228 expected char ';', found
'_LC_locale:,768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'768,32;__meth_ptr:90,800,32;__data_ptr:90,832,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
's_LC_locale_objhdl:,64,32;;'
> internal error: 1283-228 expected char ',', found
'_LC_locale_objhdl:,64,32;;'
> internal error: 1283-228 expected char ';', found
'LC_locale_objhdl:,64,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found '64,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sfio_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ',', found
'fio_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ';', found
'io_mutex:,0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;rwlock:886=*887=xsfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sfio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ',', found
'fio_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ';', found
'io_rwlock:,0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;;,1088,32;:888=u64io_axmap:889=*890=xsaxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'saxmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ',', found
'axmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ';', found
'xmap:,0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;lfsr:874,0,512;;,1152,512;:891=u96zipf:871,0,768;gauss:875,0,512;;,1664,768;references:-1,2432,32;flags:877,2464,32;du:892=*893=xsdisk_util:,2496,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sdisk_util:,2496,32;;'
> internal error: 1283-228 expected char ',', found
'disk_util:,2496,32;;'
> internal error: 1283-228 expected char ';', found 'isk_util:,2496,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found '2496,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: unexpected value 66 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'-11,192,32;aio_return:68,224,32;aio_errno:-1,256,32;aio_nbytes:63,288,32;aio_reqprio:-1,320,32;aio_sigevent:638,352,160;aio_word2:-1,512,32;aio_fp:-1,544,32;aio_handle:895,576,32;aio_reserved:901=ar111;0;2;66,608,96;aio_sigev_tid:108,704,32;aio_pad:-1,736,32;;'
> internal error: 1283-228 expected char ';', found
',32;aio_return:68,224,32;aio_errno:-1,256,32;aio_nbytes:63,288,32;aio_reqprio:-1,320,32;aio_sigevent:638,352,160;aio_word2:-1,512,32;aio_fp:-1,544,32;aio_handle:895,576,32;aio_reserved:901=ar111;0;2;66,608,96;aio_sigev_tid:108,704,32;aio_pad:-1,736,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sworkqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
> internal error: 1283-228 expected char ',', found
'workqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
> internal error: 1283-228 expected char ';', found
'orkqueue:,1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'1024,32;private:90,1056,32;sk_out:940=*941=xssk_out:,1088,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found 'ssk_out:,1088,32;;'
> internal error: 1283-228 expected char ',', found 'sk_out:,1088,32;;'
> internal error: 1283-228 expected char ';', found 'k_out:,1088,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found '1088,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sthread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
> internal error: 1283-228 expected char ',', found
'thread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
> internal error: 1283-228 expected char ';', found
'hread_data:,32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'32,32;ops:956,64,288;work_seq:11,384,64;workers:967=*937,448,32;next_free_worker:-8,480,32;flush_cond:152,512,352;flush_lock:149,864,416;stat_lock:149,1280,416;wake_idle:968=B-1,1696,32;;'
> internal error: unexpected value 66 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found '-1,1696,32;;'
> internal error: 1283-228 expected char ';', found ',32;;'
> internal error: 1283-232 index("32;;", ':') failed
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sio_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
> internal error: 1283-228 expected char ',', found
'io_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
> internal error: 1283-228 expected char ';', found
'o_piece:,608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'608,32;resid:-8,640,32;error:-8,672,32;:974=u4index:-8,0,32;seen:-8,0,32;engine_data:90,0,32;;,704,32;:975=u8verify_list:864,0,64;work:936,0,64;;,736,64;end_io:976=*977=f-1,800,32;:978=u88aiocb64:902,0,704;mmap_data:90,0,32;null:11,0,64;;,832,704;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sclient_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: 1283-228 expected char ',', found
'client_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: 1283-228 expected char ';', found
'lient_eta:,9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'9120,32;eta_timeouts:-8,9152,32;cmd_list:864,9184,64;argc:9,9248,16;argv:1119=*74,9280,32;ops:1120=*1121=xsclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sclient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: 1283-228 expected char ',', found
'client_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: 1283-228 expected char ';', found
'lient_ops:,9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'9312,32;client_data:90,9344,32;files:1122=*1114,9376,32;nr_files:-8,9408,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sjson_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ',', found
'json_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ';', found
'son_object:,0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;array:1141=*1142=xsjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sjson_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ',', found
'json_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ';', found
'son_array:,0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;;,64,64;parent_type:-1,128,32;:1143=u4parent_pair:1144=*1145=xsjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sjson_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ',', found
'json_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: 1283-228 expected char ';', found
'son_pair:,0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'0,32;parent_array:1141,0,32;;,160,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sthread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: 1283-228 expected char ',', found
'thread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: 1283-228 expected char ';', found
'hread_io_list:,425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'425856,32;shm_id:-1,425888,32;io_ops:1195=*980,425920,32;cur_depth:-8,425952,32;io_u_queued:-8,425984,32;io_u_in_flight:-8,426016,32;io_u_requeues:1182,426048,128;io_u_freelist:1180,426176,96;io_u_all:1180,426272,96;io_u_lock:149,426368,416;free_cond:152,426784,352;verify_list:864,427136,64;verify_threads:1196=*147,427200,32;nr_verify_threads:-8,427232,32;verify_cond:152,427264,352;verify_thread_exit:-1,427616,32;rate_bps:882,427648,192;rate_next_io_time:1197=ar111;0;2;-10,427840,96;rate_bytes:1197,427936,96;rate_blocks:1197,428032,96;rate_io_issue_bytes:1197,428128,96;lastrate:1198=ar111;0;2;420,428224,192;last_usec:7,428416,64;poisson_state:868,428480,384;io_wq:939,428864,1728;total_io_size:11,430592,64;fill_device_size:11,430656,64;io_issues:882,430720,192;io_issue_bytes:882,430912,192;loops:11,431104,64;io_blocks:882,431168,192;this_io_blocks:882,431360,192;io_bytes:882,431552,192;this_io_bytes:882,431744,192;io_skip_bytes:11,431936,64;zone_bytes:11,432000,64;mutex:884,432064,32;bytes_done:882,432128,192;random_state:868,432320,384;start:420,432704,64;epoch:420,432768,64;last_issue:420,432832,64;time_offset:-4,432896,32;tv_cache:420,432928,64;terminate_time:420,432992,64;tv_cache_nr:-8,433056,32;tv_cache_mask:-8,433088,32;ramp_time_over:-8,433120,32;latency_ts:420,433152,64;latency_qd:-8,433216,32;latency_qd_high:-8,433248,32;latency_qd_low:-8,433280,32;latency_failed:-8,433312,32;latency_ios:11,433344,64;latency_end_run:-1,433408,32;rwmix_state:868,433472,384;rwmix_issues:-10,433856,32;rwmix_ddir:862,433888,32;ddir_seq_nr:-8,433920,32;seq_rand_state:1199=ar111;0;2;868,433984,1152;io_hist_tree:927,435136,32;io_hist_list:864,435168,64;io_hist_len:-10,435232,32;io_log_list:864,435264,64;trim_list:864,435328,64;trim_entries:-10,435392,32;next_rand_list:864,435424,64;file_service_nr:-8,435488,32;file_service_left:-8,435520,32;file_service_file:971,435552,32;sync_file_range_nr:-8,435584,32;file_size_state:868,435648,384;total_err_count:-8,436032,32;first_error:-1,436064,32;flow:1200=*1201=xsfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: unexpected value 120 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'sfio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: 1283-228 expected char ',', found
'fio_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: 1283-228 expected char ';', found
'io_flow:,436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> internal error: unexpected value 44 at line 5176 in file stabstring.c
> internal error: 1283-228 expected char ',', found
'436096,32;prof_io_ops:1057,436128,192;prof_data:90,436320,32;pinned_mem:90,436352,32;;'
> calc_thread_status(je = 0x3009f6a8, force = ??), line 361 in "eta.c"
> unnamed block in print_thread_status(), line 605 in "eta.c"
> unnamed block in print_thread_status(), line 605 in "eta.c"
> print_thread_status(), line 605 in "eta.c"
> unnamed block in helper_thread_main(data = ??), line 2318 in
"backend.c"
> helper_thread_main(data = ??), line 2318 in "backend.c"
> (dbx)
>
>
> Regards,
> Thierry
>
[-- Attachment #2: Type: text/html, Size: 96346 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out
2016-01-01 16:19 ` Andrey Kuzmin
@ 2016-01-01 17:31 ` Thierry BERTAUD
2016-01-18 23:58 ` How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-01 17:31 UTC (permalink / raw)
To: Andrey Kuzmin; +Cc: fio@vger.kernel.org, Jens Axboe
Andrey,
>> How to have sure gcc take the option via configure : configure --disable-optimizations because it's not work for me with gdb?
>No options involved, --disable-optimizations simply doesn't strip debug info (as gcc usually does with -O3). Unless gcc on aix does something weird, that should just happen by itself.
But i have symbols in the binary.
# file fio
fio: executable (RISC System/6000) or object module not stripped
#
if i check with nm:
# nm fio
:T1000=eIO_LOG_TYPE_LAT:1,IO_LOG_TYPE_CLAT:2,IO_LOG_TYPE_SLAT:3,IO_LOG_TYPE_BW:4,IO_LOG_TYPE_IOPS:5,; - 0
:T1001=eIO_LOG_TYPE_LAT:1,IO_LOG_TYPE_CLAT:2,IO_LOG_TYPE_SLAT:3,IO_LOG_TYPE_BW:4,IO_LOG_TYPE_IOPS:5,; - 0
:T1001=eIO_LOG_TYPE_LAT:1,IO_LOG_TYPE_CLAT:2,IO_LOG_TYPE_SLAT:3,IO_LOG_TYPE_BW:4,IO_LOG_TYPE_IOPS:5,; - 0
:T1002=eIP_F_ONRB:1,IP_F_ONLIST:2,IP_F_TRIMMED:4,IP_F_IN_FLIGHT:8,; - 0
...
> You reported 128 threads in the original problem report. Is the below a differnt one?
When i launch fio with: HDISK=hdisk6 RWTYPE=rw RRATIO=0 BS=65536 QD=128 NJ=128 /home/padmin/fio/fio-5f3bd0f/fio myfio.fio
Form unix with ps, i have 129 fio process running. When fio is stucking, it staying alive only 1 the main process that fork the 128 process.
>> (dbx) thread
>> thread state-k wchan state-u k-tid mode held scope function
>> >$t1 run running 84803767 k no sys _p_nsleep
>> $t2 run running 110231691 u no sys __ioctl
> Thread 2 sleeps in the ioctl that the trace reports as failed. May be this is the root cause behind your issue?
Perhaps but i don't know how to progress againt it?
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-01 17:31 ` Thierry BERTAUD
@ 2016-01-18 23:58 ` Thierry BERTAUD
2016-01-19 0:15 ` David Nellans
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-18 23:58 UTC (permalink / raw)
To: fio@vger.kernel.org
Hello fio team,
I tryed to stress my 8 Gbps hba with fio wit a lot of case:
I know that the hba card is capable of 500 000 IOPS or get data rate neat that 8 Gbps (1600 MB/sec).
i tried to change block size, iodepth, numjobs, iodepth but idon't have good resulut. (42000 IOPS and throughput 162 MB/s).
Below fio conf:
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=4k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randwrite_64_64]
rw=randwrite
numjobs=64
ioepth=64
stonewal
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-18 23:58 ` How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s Thierry BERTAUD
@ 2016-01-19 0:15 ` David Nellans
2016-01-19 0:29 ` Thierry BERTAUD
2016-01-19 0:33 ` Kulkarni, Vasu
0 siblings, 2 replies; 24+ messages in thread
From: David Nellans @ 2016-01-19 0:15 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: fio@vger.kernel.org
you have two typos in your config that will likely result in less than sufficient iodepth to hit 500k IOPS. check if those are in your actual config or not...
> On Jan 18, 2016, at 5:58 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
>
> Hello fio team,
>
> I tryed to stress my 8 Gbps hba with fio wit a lot of case:
> I know that the hba card is capable of 500 000 IOPS or get data rate neat that 8 Gbps (1600 MB/sec).
> i tried to change block size, iodepth, numjobs, iodepth but idon't have good resulut. (42000 IOPS and throughput 162 MB/s).
>
> Below fio conf:
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=4k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randwrite_64_64]
> rw=randwrite
> numjobs=64
> ioepth=64
> stonewal
>
>
>
> Regards,
> Thierry--
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 0:15 ` David Nellans
@ 2016-01-19 0:29 ` Thierry BERTAUD
[not found] ` <CAB-428m2HJBUPdtYsCKtxfYPuscV1i=ucCvCAPh4ShqWPNmTXw@mail.gmail.com>
2016-01-19 0:33 ` Kulkarni, Vasu
1 sibling, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-19 0:29 UTC (permalink / raw)
To: David Nellans; +Cc: fio@vger.kernel.org
David,
Sorry for typo.
I paste and not copy the configuration file.
# cat randwrite.fio
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=4k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randwrite_64_64]
rw=randwrite
numjobs=64
iodepth=64
stonewall
#
Cordialement,
Thierry Bertaud
Tel: 01 60 95 51 41
PS: Merci de mettre en copie pour toute demande syst�me, le groupe dsisystunix@conforama.fr
________________________________________
De : David Nellans <david@nellans.org>
Envoy� : mardi 19 janvier 2016 01:15
� : Thierry BERTAUD
Cc : fio@vger.kernel.org
Objet : Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
you have two typos in your config that will likely result in less than sufficient iodepth to hit 500k IOPS. check if those are in your actual config or not...
> On Jan 18, 2016, at 5:58 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
>
> Hello fio team,
>
> I tryed to stress my 8 Gbps hba with fio wit a lot of case:
> I know that the hba card is capable of 500 000 IOPS or get data rate neat that 8 Gbps (1600 MB/sec).
> i tried to change block size, iodepth, numjobs, iodepth but idon't have good resulut. (42000 IOPS and throughput 162 MB/s).
>
> Below fio conf:
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=4k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randwrite_64_64]
> rw=randwrite
> numjobs=64
> ioepth=64
> stonewal
>
>
>
> Regards,
> Thierry--
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 0:15 ` David Nellans
2016-01-19 0:29 ` Thierry BERTAUD
@ 2016-01-19 0:33 ` Kulkarni, Vasu
2016-01-19 0:40 ` David Nellans
1 sibling, 1 reply; 24+ messages in thread
From: Kulkarni, Vasu @ 2016-01-19 0:33 UTC (permalink / raw)
To: David Nellans; +Cc: Thierry BERTAUD, fio@vger.kernel.org
Well IOPS/througput is also dependent on the physical drive as well,
HBA alone is not sufficient to get those rates, you will need a
SSD/NVMe device to get very high IOPS as well. your harddrive
will have IOPS and max range you should check that.
On Mon, Jan 18, 2016 at 4:15 PM, David Nellans <david@nellans.org> wrote:
> you have two typos in your config that will likely result in less than sufficient iodepth to hit 500k IOPS. check if those are in your actual config or not...
>
>> On Jan 18, 2016, at 5:58 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
>>
>> Hello fio team,
>>
>> I tryed to stress my 8 Gbps hba with fio wit a lot of case:
>> I know that the hba card is capable of 500 000 IOPS or get data rate neat that 8 Gbps (1600 MB/sec).
>> i tried to change block size, iodepth, numjobs, iodepth but idon't have good resulut. (42000 IOPS and throughput 162 MB/s).
>>
>> Below fio conf:
>> [global]
>> thread
>> numjobs=1
>> iodepth=1
>> group_reporting
>> bs=4k
>> norandommap=1
>> refill_buffers
>> direct=1
>> ioengine=posixaio
>> runtime=300
>> time_based
>> filename=/dev/hdisk6
>> log_avg_msec=1000
>> [randwrite_64_64]
>> rw=randwrite
>> numjobs=64
>> ioepth=64
>> stonewal
>>
>>
>>
>> Regards,
>> Thierry--
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
A ship is safer in a harbor. But that's not what ships are built for.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 0:33 ` Kulkarni, Vasu
@ 2016-01-19 0:40 ` David Nellans
2016-01-19 1:04 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: David Nellans @ 2016-01-19 0:40 UTC (permalink / raw)
To: Kulkarni, Vasu; +Cc: Thierry BERTAUD, fio@vger.kernel.org
agree with vasu - assuming you're trying to hit numbers a vendor spec'd however on a flash drive of some sort.
you're doing random writes which is also tough on some flash drive if they're into write amplification. try changing to random reads and you only need a couple of threads (with a depth of 128 each) to generate enough iodepth to saturate most drives. 64 threads might be more threads than you have CPUs to run them so you're switching which also hurts performance
> On Jan 18, 2016, at 6:33 PM, Kulkarni, Vasu <vasu.kulkarni@gmail.com> wrote:
>
> Well IOPS/througput is also dependent on the physical drive as well,
> HBA alone is not sufficient to get those rates, you will need a
> SSD/NVMe device to get very high IOPS as well. your harddrive
> will have IOPS and max range you should check that.
>
>> On Mon, Jan 18, 2016 at 4:15 PM, David Nellans <david@nellans.org> wrote:
>> you have two typos in your config that will likely result in less than sufficient iodepth to hit 500k IOPS. check if those are in your actual config or not...
>>
>>> On Jan 18, 2016, at 5:58 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
>>>
>>> Hello fio team,
>>>
>>> I tryed to stress my 8 Gbps hba with fio wit a lot of case:
>>> I know that the hba card is capable of 500 000 IOPS or get data rate neat that 8 Gbps (1600 MB/sec).
>>> i tried to change block size, iodepth, numjobs, iodepth but idon't have good resulut. (42000 IOPS and throughput 162 MB/s).
>>>
>>> Below fio conf:
>>> [global]
>>> thread
>>> numjobs=1
>>> iodepth=1
>>> group_reporting
>>> bs=4k
>>> norandommap=1
>>> refill_buffers
>>> direct=1
>>> ioengine=posixaio
>>> runtime=300
>>> time_based
>>> filename=/dev/hdisk6
>>> log_avg_msec=1000
>>> [randwrite_64_64]
>>> rw=randwrite
>>> numjobs=64
>>> ioepth=64
>>> stonewal
>>>
>>>
>>>
>>> Regards,
>>> Thierry--
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> A ship is safer in a harbor. But that's not what ships are built for.
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 0:40 ` David Nellans
@ 2016-01-19 1:04 ` Thierry BERTAUD
2016-01-19 1:15 ` Alireza Haghdoost
0 siblings, 1 reply; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-19 1:04 UTC (permalink / raw)
To: David Nellans, Kulkarni, Vasu; +Cc: fio@vger.kernel.org
David and Vasu,
>>agree with vasu - assuming you're trying to hit numbers a vendor spec'd however on a flash drive of >>some sort.
>>you're doing random writes which is also tough on some flash drive if they're into write >>amplification. try changing to random reads and you only need a couple of threads (with a depth of >>128 each) to generate enough iodepth to saturate most drives. 64 threads might be more threads >>than you have CPUs to run them so you're switching which also hurts performance
I test against a SAN LUN (Infinidat F2000).
This my AIX LUN:
# lsattr -El hdisk6
PCM PCM/friend/NFINIDATpcm Path Control Module False
PR_key_value Persistent Reserve Key value True
algorithm round_robin Algorithm True
hcheck_cmd test_unit_rdy Health Check Command True
hcheck_interval 60 Health Check Interval True
hcheck_mode nonactive Health Check Mode True
lun_id 0x1000000000000 Logical Unit Number ID False
lun_reset_spt yes LUN Reset Supported True
max_transfer 0x80000 Maximum TRANSFER Size True
node_name 0x5742b0f00004c700 FC Node Name False
pvid none Physical volume identifier False
q_type simple Queuing TYPE True
queue_depth 64 Queue DEPTH True
reserve_policy no_reserve Reserve Policy True
rw_timeout 30 READ/WRITE time out value True
scsi_id 0x1e0800 SCSI ID False
unique_id 3B1F742b0f0000004c700000000000001d609InfiniBox08NFINIDATfcp Unique device identifier False
ww_name 0x5742b0f00004c711 FC World Wide Name False
#
With this conf:
# cat randwrite.fio
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=4k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randread_32_128]
rw=randread
numjobs=32
iodepth=128
stonewall
#
I have now bad resulsts:
IOPS: 2 800
Bandwith:11 MB/s
latency : 6ms
before with old configuration file:
IOPS: 42 000
Bandwith:164 MB/s
latency : 0.2 ms
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 1:04 ` Thierry BERTAUD
@ 2016-01-19 1:15 ` Alireza Haghdoost
2016-01-19 1:59 ` Thierry BERTAUD
0 siblings, 1 reply; 24+ messages in thread
From: Alireza Haghdoost @ 2016-01-19 1:15 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: David Nellans, Kulkarni, Vasu, fio@vger.kernel.org
I would assume you have a loop-back HBA since you are not going to get
Max Bandwidth with "Unknown" storage device in the back-end.
Your best bet is with large transfer size like:
bs=256k
and high queue depth:
iodepth=256
Make sure your HBA queue depth is not limited.
Do the math again, you cannot get 1600 MB/s with a 8Gbps fiber in theory.
--Alireza
On Mon, Jan 18, 2016 at 7:04 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
> David and Vasu,
>
>>>agree with vasu - assuming you're trying to hit numbers a vendor spec'd however on a flash drive of >>some sort.
>
>>>you're doing random writes which is also tough on some flash drive if they're into write >>amplification. try changing to random reads and you only need a couple of threads (with a depth of >>128 each) to generate enough iodepth to saturate most drives. 64 threads might be more threads >>than you have CPUs to run them so you're switching which also hurts performance
>
> I test against a SAN LUN (Infinidat F2000).
> This my AIX LUN:
> # lsattr -El hdisk6
> PCM PCM/friend/NFINIDATpcm Path Control Module False
> PR_key_value Persistent Reserve Key value True
> algorithm round_robin Algorithm True
> hcheck_cmd test_unit_rdy Health Check Command True
> hcheck_interval 60 Health Check Interval True
> hcheck_mode nonactive Health Check Mode True
> lun_id 0x1000000000000 Logical Unit Number ID False
> lun_reset_spt yes LUN Reset Supported True
> max_transfer 0x80000 Maximum TRANSFER Size True
> node_name 0x5742b0f00004c700 FC Node Name False
> pvid none Physical volume identifier False
> q_type simple Queuing TYPE True
> queue_depth 64 Queue DEPTH True
> reserve_policy no_reserve Reserve Policy True
> rw_timeout 30 READ/WRITE time out value True
> scsi_id 0x1e0800 SCSI ID False
> unique_id 3B1F742b0f0000004c700000000000001d609InfiniBox08NFINIDATfcp Unique device identifier False
> ww_name 0x5742b0f00004c711 FC World Wide Name False
> #
>
> With this conf:
> # cat randwrite.fio
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=4k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randread_32_128]
> rw=randread
> numjobs=32
> iodepth=128
> stonewall
> #
>
> I have now bad resulsts:
> IOPS: 2 800
> Bandwith:11 MB/s
> latency : 6ms
>
> before with old configuration file:
> IOPS: 42 000
> Bandwith:164 MB/s
> latency : 0.2 ms
>
> Regards,
> Thierry--
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
2016-01-19 1:15 ` Alireza Haghdoost
@ 2016-01-19 1:59 ` Thierry BERTAUD
0 siblings, 0 replies; 24+ messages in thread
From: Thierry BERTAUD @ 2016-01-19 1:59 UTC (permalink / raw)
To: Alireza Haghdoost; +Cc: David Nellans, Kulkarni, Vasu, fio@vger.kernel.org
Alireza,
I use 2 hba (fscsi1 -> fabric1, fscsi2 -> fabric2). Each fabric is connect to 3 nodes.
# lspath | grep hdisk6
Enabled hdisk6 fscsi1
Enabled hdisk6 fscsi1
Enabled hdisk6 fscsi3
Enabled hdisk6 fscsi3
Enabled hdisk6 fscsi1
Enabled hdisk6 fscsi3
#
With : bs=256k and
# cat randwrite.fio
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=256k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randread_32_256]
rw=randread
numjobs=32
iodepth=256
stonewall
#
IOPS: 23 000
Bandwith: 85 Mo/s
Latency: 0.5 ms
but with this one:
# cat randwrite.fio
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=4k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randread_64_64]
rw=randread
numjobs=64
iodepth=64
stonewall
#
IOPS: 10 000
Bandwith: 40 MB/S
Latency: 1.5 ms
but write seems better than read:
# cat randwrite.fio
[global]
thread
numjobs=1
iodepth=1
group_reporting
bs=4k
norandommap=1
refill_buffers
direct=1
ioengine=posixaio
runtime=300
time_based
filename=/dev/hdisk6
log_avg_msec=1000
[randwrite_64_64]
rw=randwrite
numjobs=64
iodepth=64
stonewall
#
IOPS: 42 000
Bandwith: 163 MB/S
latency: 0.2 ms
I don't understand why for last test randwrite is better than randread.
I'm lost and don't undertand how to stress the LUN.
Regards,
Thierry
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
[not found] ` <VI1PR05MB1343CA9499CC6AB3B7A0D2C0B1C10@VI1PR05MB1343.eurprd05.prod.outlook.com>
@ 2016-01-19 2:45 ` Alireza Haghdoost
0 siblings, 0 replies; 24+ messages in thread
From: Alireza Haghdoost @ 2016-01-19 2:45 UTC (permalink / raw)
To: Thierry BERTAUD; +Cc: David Nellans, fio@vger.kernel.org
Increase your iodepth as I said earlier. You may ask your storage
provider why read and writes behave like this. You can also increase
numjob to push more IO with more fio process/threads.
On Mon, Jan 18, 2016 at 7:34 PM, Thierry BERTAUD <tbertaud@conforama.fr> wrote:
> Alireza,
>
>
> I use 2 hba (fscsi1 -> fabric1, fscsi2 -> fabric2). Each fabric is connect
> to 3 nodes.
>
>
> # lspath | grep hdisk6
> Enabled hdisk6 fscsi1
> Enabled hdisk6 fscsi1
> Enabled hdisk6 fscsi3
> Enabled hdisk6 fscsi3
> Enabled hdisk6 fscsi1
> Enabled hdisk6 fscsi3
> #
>
> With :
>
> # cat randwrite.fio
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=256k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randread_32_256]
> rw=randread
> numjobs=32
> iodepth=256
> stonewall
> #
> IOPS: 23 000
>
> Bandwith: 85 Mo/s
>
> Latency: 0.5 ms
>
>
> but with this one:
>
> # cat randwrite.fio
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=4k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randread_64_64]
> rw=randread
> numjobs=64
> iodepth=64
> stonewall
> #
> IOPS: 10 000
>
> Bandwith: 40 MB/S
>
> Latency: 1.5 ms
>
>
> but write seems better than read:
>
> # cat randwrite.fio
> [global]
> thread
> numjobs=1
> iodepth=1
> group_reporting
> bs=4k
> norandommap=1
> refill_buffers
> direct=1
> ioengine=posixaio
> runtime=300
> time_based
> filename=/dev/hdisk6
> log_avg_msec=1000
> [randwrite_64_64]
> rw=randwrite
> numjobs=64
> iodepth=64
> stonewall
> #
> IOPS: 42 000
>
> Bandwith: 163 MB/S
>
> latency: 0.2 ms
>
>
> I don't understand why for last test randwrite is better than randread.
>
> I'm lost and don't undertand how to stress the LUN.
>
>
> Regards,
> Thierry
>
>
> ________________________________
> De : Alireza Haghdoost <alireza@cs.umn.edu>
> Envoyé : mardi 19 janvier 2016 02:06
> À : Thierry BERTAUD
> Cc : David Nellans; fio@vger.kernel.org
>
> Objet : Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
>
> I would assume you have a loop-back HBA since you are not going to get Max
> Bandwidth with "Unknown" storage device in the back-end.
> Your best bet is with large transfer size like:
> bs=256k
>
> and high queue depth:
> iodepth=256
>
> Make sure your HBA queue depth is not limited.
>
> Do the math again, you cannot get 1600 MB/s with a 8Gbps fiber in theory.
>
> --Alireza
>
> On Mon, Jan 18, 2016 at 6:29 PM, Thierry BERTAUD <tbertaud@conforama.fr>
> wrote:
>>
>> David,
>>
>> Sorry for typo.
>> I paste and not copy the configuration file.
>>
>> # cat randwrite.fio
>> [global]
>> thread
>> numjobs=1
>> iodepth=1
>> group_reporting
>> bs=4k
>> norandommap=1
>> refill_buffers
>> direct=1
>> ioengine=posixaio
>> runtime=300
>> time_based
>> filename=/dev/hdisk6
>> log_avg_msec=1000
>> [randwrite_64_64]
>> rw=randwrite
>> numjobs=64
>> iodepth=64
>> stonewall
>> #
>>
>>
>> Cordialement,
>> Thierry Bertaud
>> Tel: 01 60 95 51 41
>>
>> PS: Merci de mettre en copie pour toute demande système, le groupe
>> dsisystunix@conforama.fr
>>
>> ________________________________________
>> De : David Nellans <david@nellans.org>
>> Envoyé : mardi 19 janvier 2016 01:15
>> À : Thierry BERTAUD
>> Cc : fio@vger.kernel.org
>> Objet : Re: How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s
>>
>> you have two typos in your config that will likely result in less than
>> sufficient iodepth to hit 500k IOPS. check if those are in your actual
>> config or not...
>>
>> > On Jan 18, 2016, at 5:58 PM, Thierry BERTAUD <tbertaud@conforama.fr>
>> > wrote:
>> >
>> > Hello fio team,
>> >
>> > I tryed to stress my 8 Gbps hba with fio wit a lot of case:
>> > I know that the hba card is capable of 500 000 IOPS or get data rate
>> > neat that 8 Gbps (1600 MB/sec).
>> > i tried to change block size, iodepth, numjobs, iodepth but idon't have
>> > good resulut. (42000 IOPS and throughput 162 MB/s).
>> >
>> > Below fio conf:
>> > [global]
>> > thread
>> > numjobs=1
>> > iodepth=1
>> > group_reporting
>> > bs=4k
>> > norandommap=1
>> > refill_buffers
>> > direct=1
>> > ioengine=posixaio
>> > runtime=300
>> > time_based
>> > filename=/dev/hdisk6
>> > log_avg_msec=1000
>> > [randwrite_64_64]
>> > rw=randwrite
>> > numjobs=64
>> > ioepth=64
>> > stonewal
>> >
>> >
>> >
>> > Regards,
>> > Thierry--
>> > To unsubscribe from this list: send the line "unsubscribe fio" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2016-01-19 2:46 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-29 8:29 fio-2c2c93d and with a lot of numjobs causing fio Jobs to Time Out Thierry BERTAUD
2015-12-29 14:30 ` Thierry BERTAUD
2015-12-30 17:08 ` Jens Axboe
2015-12-30 18:34 ` Thierry BERTAUD
2015-12-31 4:02 ` Jens Axboe
2015-12-31 16:18 ` Thierry BERTAUD
2015-12-31 16:33 ` Andrey Kuzmin
2016-01-01 12:43 ` Thierry BERTAUD
2016-01-01 12:46 ` Thierry BERTAUD
2016-01-01 12:54 ` Andrey Kuzmin
2016-01-01 14:00 ` Thierry BERTAUD
2016-01-01 14:04 ` Andrey Kuzmin
2016-01-01 14:49 ` Thierry BERTAUD
2016-01-01 16:19 ` Andrey Kuzmin
2016-01-01 17:31 ` Thierry BERTAUD
2016-01-18 23:58 ` How to stress a 8Gbps card to get 500 000 IOPS or 1600 MB/s Thierry BERTAUD
2016-01-19 0:15 ` David Nellans
2016-01-19 0:29 ` Thierry BERTAUD
[not found] ` <CAB-428m2HJBUPdtYsCKtxfYPuscV1i=ucCvCAPh4ShqWPNmTXw@mail.gmail.com>
[not found] ` <VI1PR05MB1343CA9499CC6AB3B7A0D2C0B1C10@VI1PR05MB1343.eurprd05.prod.outlook.com>
2016-01-19 2:45 ` Alireza Haghdoost
2016-01-19 0:33 ` Kulkarni, Vasu
2016-01-19 0:40 ` David Nellans
2016-01-19 1:04 ` Thierry BERTAUD
2016-01-19 1:15 ` Alireza Haghdoost
2016-01-19 1:59 ` Thierry BERTAUD
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.