From: Jens Axboe <axboe@kernel.dk>
To: "Elliott, Robert (Server Storage)" <Elliott@hp.com>,
"stephenmcameron@gmail.com" <stephenmcameron@gmail.com>
Cc: "fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: fio main thread got stuck over the weekend
Date: Mon, 15 Dec 2014 10:37:41 -0700 [thread overview]
Message-ID: <548F1C65.2070501@kernel.dk> (raw)
In-Reply-To: <94D0CD8314A33A4D9D801C0FE68B40295940EEC5@G4W3202.americas.hpqcorp.net>
On 12/15/2014 10:33 AM, Elliott, Robert (Server Storage) wrote:
>>
>> If you have it running, it would be great to do a gdb attach and see
>> what the frontend is up to (or where it might be stuck)...
>>
>
> The last eta printed over 3 days ago was
> [eta 01d:17h:05m:24s]
> but it is still running.
>
> One device did report several errors during the test before the eta
> hang, and there are no IOs going to that device.
>
> There are 212 threads running out of 224 started.
>
> (gdb) info threads
> 212 Thread 0x7fa9a086e700 (LWP 6509) 0x0000003974c0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> 211 Thread 0x7fa9a006b700 (LWP 6510) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 210 Thread 0x7fa99f869700 (LWP 6511) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 209 Thread 0x7fa99f067700 (LWP 6512) last_block (td=0x7fa9a094e740, f=0x7fa9a91a8390, ddir=<value optimized out>, b=0x7fa99f066c50) at io_u.c:65
> 208 Thread 0x7fa99e865700 (LWP 6513) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 207 Thread 0x7fa99e063700 (LWP 6514) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 206 Thread 0x7fa99d861700 (LWP 6515) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 205 Thread 0x7fa99d05f700 (LWP 6516) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 204 Thread 0x7fa99c85d700 (LWP 6517) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 203 Thread 0x7fa99c05b700 (LWP 6518) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 202 Thread 0x7fa99b859700 (LWP 6519) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 201 Thread 0x7fa99b057700 (LWP 6520) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 200 Thread 0x7fa99a855700 (LWP 6521) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 199 Thread 0x7fa99a053700 (LWP 6522) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 198 Thread 0x7fa999851700 (LWP 6523) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 197 Thread 0x7fa99904e700 (LWP 6524) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 196 Thread 0x7fa99884c700 (LWP 6525) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 195 Thread 0x7fa99804a700 (LWP 6526) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 194 Thread 0x7fa997848700 (LWP 6527) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 193 Thread 0x7fa997046700 (LWP 6528) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 192 Thread 0x7fa996844700 (LWP 6529) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 191 Thread 0x7fa996042700 (LWP 6530) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 190 Thread 0x7fa995840700 (LWP 6531) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 189 Thread 0x7fa99503e700 (LWP 6532) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 188 Thread 0x7fa99483c700 (LWP 6533) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 187 Thread 0x7fa99403a700 (LWP 6534) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 186 Thread 0x7fa993838700 (LWP 6535) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 185 Thread 0x7fa993036700 (LWP 6536) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 184 Thread 0x7fa992834700 (LWP 6537) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 183 Thread 0x7fa992031700 (LWP 6538) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 182 Thread 0x7fa99182f700 (LWP 6539) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 181 Thread 0x7fa99102d700 (LWP 6540) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 180 Thread 0x7fa99082b700 (LWP 6541) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 179 Thread 0x7fa990029700 (LWP 6542) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 178 Thread 0x7fa98f827700 (LWP 6543) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 177 Thread 0x7fa98f025700 (LWP 6544) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 176 Thread 0x7fa98e823700 (LWP 6545) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 175 Thread 0x7fa98e021700 (LWP 6546) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 174 Thread 0x7fa98d81f700 (LWP 6547) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 173 Thread 0x7fa98d01d700 (LWP 6548) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 172 Thread 0x7fa98c81b700 (LWP 6549) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 171 Thread 0x7fa98c019700 (LWP 6550) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 170 Thread 0x7fa98b817700 (LWP 6551) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 169 Thread 0x7fa98b014700 (LWP 6552) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 168 Thread 0x7fa98a812700 (LWP 6553) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 167 Thread 0x7fa98a010700 (LWP 6554) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 166 Thread 0x7fa98980e700 (LWP 6555) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 165 Thread 0x7fa98900c700 (LWP 6556) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 164 Thread 0x7fa98880a700 (LWP 6557) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 163 Thread 0x7fa988008700 (LWP 6558) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 162 Thread 0x7fa987806700 (LWP 6559) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 161 Thread 0x7fa987004700 (LWP 6560) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 160 Thread 0x7fa986802700 (LWP 6561) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 159 Thread 0x7fa986000700 (LWP 6562) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 158 Thread 0x7fa9857fe700 (LWP 6563) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 157 Thread 0x7fa984ffc700 (LWP 6564) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 156 Thread 0x7fa9847fa700 (LWP 6565) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 155 Thread 0x7fa983ff7700 (LWP 6566) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 154 Thread 0x7fa9837f5700 (LWP 6567) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 153 Thread 0x7fa982ff3700 (LWP 6568) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 152 Thread 0x7fa9827f1700 (LWP 6569) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 151 Thread 0x7fa981fef700 (LWP 6570) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 150 Thread 0x7fa9817ed700 (LWP 6571) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 149 Thread 0x7fa980feb700 (LWP 6572) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 148 Thread 0x7fa9807e9700 (LWP 6573) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 147 Thread 0x7fa97ffe7700 (LWP 6574) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 146 Thread 0x7fa97f7e5700 (LWP 6575) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 145 Thread 0x7fa97efe3700 (LWP 6576) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 144 Thread 0x7fa97e7e1700 (LWP 6577) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 143 Thread 0x7fa97dfdf700 (LWP 6578) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 142 Thread 0x7fa97d7dd700 (LWP 6579) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 141 Thread 0x7fa97cfda700 (LWP 6580) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 140 Thread 0x7fa97c7d8700 (LWP 6581) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 139 Thread 0x7fa97bfd6700 (LWP 6582) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 138 Thread 0x7fa97b7d4700 (LWP 6583) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 137 Thread 0x7fa97afd2700 (LWP 6584) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 136 Thread 0x7fa97a7d0700 (LWP 6585) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 135 Thread 0x7fa979fce700 (LWP 6586) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 134 Thread 0x7fa9797cc700 (LWP 6587) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 133 Thread 0x7fa978fca700 (LWP 6588) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 132 Thread 0x7fa9787c8700 (LWP 6589) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 131 Thread 0x7fa977fc6700 (LWP 6590) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 130 Thread 0x7fa9777c4700 (LWP 6591) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 129 Thread 0x7fa976fc2700 (LWP 6592) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 128 Thread 0x7fa9767c0700 (LWP 6593) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 127 Thread 0x7fa975fbd700 (LWP 6594) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 126 Thread 0x7fa9757bb700 (LWP 6595) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 125 Thread 0x7fa974fb9700 (LWP 6596) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 124 Thread 0x7fa9747b7700 (LWP 6597) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 123 Thread 0x7fa973fb5700 (LWP 6598) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 122 Thread 0x7fa9737b3700 (LWP 6599) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 121 Thread 0x7fa972fb1700 (LWP 6600) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 120 Thread 0x7fa9727af700 (LWP 6601) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 119 Thread 0x7fa971fad700 (LWP 6602) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 118 Thread 0x7fa9717ab700 (LWP 6603) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 117 Thread 0x7fa970fa9700 (LWP 6604) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 116 Thread 0x7fa9707a7700 (LWP 6605) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 115 Thread 0x7fa96ffa5700 (LWP 6606) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 114 Thread 0x7fa96f7a3700 (LWP 6607) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 113 Thread 0x7fa9637f6700 (LWP 6622) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 112 Thread 0x7fa962ff5700 (LWP 6623) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 111 Thread 0x7fa9627f4700 (LWP 6624) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 110 Thread 0x7fa961ff3700 (LWP 6625) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 109 Thread 0x7fa9617f2700 (LWP 6626) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 108 Thread 0x7fa960ff1700 (LWP 6627) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 107 Thread 0x7fa9607f0700 (LWP 6628) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 106 Thread 0x7fa95ffef700 (LWP 6629) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 105 Thread 0x7fa95f7ee700 (LWP 6630) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 104 Thread 0x7fa95efed700 (LWP 6631) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 103 Thread 0x7fa95e7ec700 (LWP 6632) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 102 Thread 0x7fa95dfeb700 (LWP 6633) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 101 Thread 0x7fa95d7ea700 (LWP 6634) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 100 Thread 0x7fa95cfe9700 (LWP 6635) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 99 Thread 0x7fa95c7e8700 (LWP 6636) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 98 Thread 0x7fa95bfe7700 (LWP 6637) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 97 Thread 0x7fa95b7e6700 (LWP 6638) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 96 Thread 0x7fa95afe5700 (LWP 6639) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 95 Thread 0x7fa95a7e4700 (LWP 6640) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 94 Thread 0x7fa959fe3700 (LWP 6641) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 93 Thread 0x7fa9597e2700 (LWP 6642) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 92 Thread 0x7fa958fe1700 (LWP 6643) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 91 Thread 0x7fa9587e0700 (LWP 6644) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 90 Thread 0x7fa957fdf700 (LWP 6645) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 89 Thread 0x7fa9577de700 (LWP 6646) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 88 Thread 0x7fa956fdd700 (LWP 6647) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 87 Thread 0x7fa9567dc700 (LWP 6648) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 86 Thread 0x7fa955fdb700 (LWP 6649) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 85 Thread 0x7fa9557da700 (LWP 6650) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 84 Thread 0x7fa954fd9700 (LWP 6651) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 83 Thread 0x7fa9547d8700 (LWP 6652) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 82 Thread 0x7fa953fd7700 (LWP 6653) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 81 Thread 0x7fa9537d6700 (LWP 6654) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 80 Thread 0x7fa952fd5700 (LWP 6655) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 79 Thread 0x7fa9527d4700 (LWP 6656) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 78 Thread 0x7fa951fd3700 (LWP 6657) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 77 Thread 0x7fa9517d2700 (LWP 6658) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 76 Thread 0x7fa950fd1700 (LWP 6659) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 75 Thread 0x7fa9507d0700 (LWP 6660) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 74 Thread 0x7fa94ffcf700 (LWP 6661) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 73 Thread 0x7fa94f7ce700 (LWP 6662) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 72 Thread 0x7fa94efcd700 (LWP 6663) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 71 Thread 0x7fa94e7cc700 (LWP 6664) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 70 Thread 0x7fa94dfcb700 (LWP 6665) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 69 Thread 0x7fa94d7ca700 (LWP 6666) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 68 Thread 0x7fa94cfc9700 (LWP 6667) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 67 Thread 0x7fa94c7c8700 (LWP 6668) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 66 Thread 0x7fa94bfc7700 (LWP 6669) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 65 Thread 0x7fa94b7c6700 (LWP 6670) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 64 Thread 0x7fa94afc5700 (LWP 6671) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 63 Thread 0x7fa94a7c4700 (LWP 6672) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 62 Thread 0x7fa949fc3700 (LWP 6673) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 61 Thread 0x7fa9497c2700 (LWP 6674) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 60 Thread 0x7fa948fc1700 (LWP 6675) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 59 Thread 0x7fa9487c0700 (LWP 6676) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 58 Thread 0x7fa947fbf700 (LWP 6677) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 57 Thread 0x7fa9477be700 (LWP 6678) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 56 Thread 0x7fa946fbd700 (LWP 6679) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 55 Thread 0x7fa9467bc700 (LWP 6680) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 54 Thread 0x7fa945fbb700 (LWP 6681) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 53 Thread 0x7fa9457ba700 (LWP 6682) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 52 Thread 0x7fa944fb9700 (LWP 6683) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 51 Thread 0x7fa9447b8700 (LWP 6684) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 50 Thread 0x7fa943fb7700 (LWP 6685) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 49 Thread 0x7fa9437b6700 (LWP 6686) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 48 Thread 0x7fa942fb5700 (LWP 6687) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 47 Thread 0x7fa9427b4700 (LWP 6688) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 46 Thread 0x7fa941fb3700 (LWP 6689) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 45 Thread 0x7fa9417b2700 (LWP 6690) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 44 Thread 0x7fa940fb1700 (LWP 6691) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 43 Thread 0x7fa9407b0700 (LWP 6692) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 42 Thread 0x7fa93ffaf700 (LWP 6693) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 41 Thread 0x7fa93f7ae700 (LWP 6694) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 40 Thread 0x7fa93efad700 (LWP 6695) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 39 Thread 0x7fa93e7ac700 (LWP 6696) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 38 Thread 0x7fa93dfab700 (LWP 6697) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 37 Thread 0x7fa93d7aa700 (LWP 6698) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 36 Thread 0x7fa93cfa9700 (LWP 6699) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 35 Thread 0x7fa93c7a8700 (LWP 6700) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 34 Thread 0x7fa93bfa7700 (LWP 6701) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 33 Thread 0x7fa93b7a6700 (LWP 6702) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 32 Thread 0x7fa93afa5700 (LWP 6703) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 31 Thread 0x7fa93a7a4700 (LWP 6704) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 30 Thread 0x7fa939fa3700 (LWP 6705) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 29 Thread 0x7fa9397a2700 (LWP 6706) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 28 Thread 0x7fa938fa1700 (LWP 6707) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 27 Thread 0x7fa9387a0700 (LWP 6708) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 26 Thread 0x7fa937f9f700 (LWP 6709) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 25 Thread 0x7fa93779e700 (LWP 6710) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 24 Thread 0x7fa936f9d700 (LWP 6711) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 23 Thread 0x7fa93679c700 (LWP 6712) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 22 Thread 0x7fa935f9b700 (LWP 6713) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 21 Thread 0x7fa93579a700 (LWP 6714) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 20 Thread 0x7fa934f99700 (LWP 6715) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 19 Thread 0x7fa934798700 (LWP 6716) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 18 Thread 0x7fa933f97700 (LWP 6717) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 17 Thread 0x7fa933796700 (LWP 6718) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 16 Thread 0x7fa932f95700 (LWP 6719) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 15 Thread 0x7fa932794700 (LWP 6720) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 14 Thread 0x7fa931f93700 (LWP 6721) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 13 Thread 0x7fa931792700 (LWP 6722) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 12 Thread 0x7fa930f91700 (LWP 6723) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 11 Thread 0x7fa930790700 (LWP 6724) 0x000000000040b466 in td_io_getevents (td=0x7fa9a0da27c0, min=14, max=14, t=0x0) at ioengines.c:237
> 10 Thread 0x7fa92ff8f700 (LWP 6725) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 9 Thread 0x7fa92f78e700 (LWP 6726) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 8 Thread 0x7fa92ef8d700 (LWP 6727) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 7 Thread 0x7fa92e78c700 (LWP 6728) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 6 Thread 0x7fa92df8b700 (LWP 6729) 0x0000003657600614 in ?? () from /lib64/libaio.so.1
> 5 Thread 0x7fa92d78a700 (LWP 6730) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 4 Thread 0x7fa92cf89700 (LWP 6731) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 3 Thread 0x7fa92c788700 (LWP 6732) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> 2 Thread 0x7fa92bf87700 (LWP 6733) 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> * 1 Thread 0x7fa9a99b1720 (LWP 6508) 0x00000039748acced in nanosleep () from /lib64/libc.so.6
>
> I wonder if one of the time values is wrong or overflowing. There was a problem in
> handling ms vs. s units a while back...
>
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7fa92bf87700 (LWP 6733))]#0 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> (gdb) bt
> #0 0x0000003657600667 in io_submit () from /lib64/libaio.so.1
> #1 0x0000000000457058 in fio_libaio_commit (td=0x7fa9a0dd1860) at engines/libaio.c:255
> #2 0x000000000040b395 in td_io_commit (td=0x7fa9a0dd1860) at ioengines.c:396
> #3 0x000000000040bea1 in td_io_queue (td=0x7fa9a0dd1860, io_u=0x7fa8e4015400) at ioengines.c:343
> #4 0x000000000044a75d in do_io (td=0x7fa9a0dd1860) at backend.c:792
> #5 0x000000000044c209 in thread_main (data=0x7fa9a0dd1860) at backend.c:1504
> #6 0x0000003974c079d1 in start_thread () from /lib64/libpthread.so.0
> #7 0x00000039748e8b7d in clone () from /lib64/libc.so.6
> (gdb) select-frame 5
> (gdb) print elapsed
> $29 = <value optimized out>
> (gdb) print td->ts.runtime
> $30 = {68075502, 0, 0}
> (gdb) print td->start
> $31 = {tv_sec = 1099511, tv_usec = 861468}
> (gdb) print td->epoch
> $32 = {tv_sec = 1031436, tv_usec = 430008}
> (gdb) print td->ts.total_run_time
> $33 = 0
> (gdb) print td->ts.runtime
> $35 = {68075502, 0, 0}
>
> 68075502 ms is only 18.9 minutes.
->start is set on entering a new loop, so it's not that unlikely that
it's just 18.9 minutes. Total runtime is summed after the fact. The one
I think is interesting is the first thread, that is the one that should
be printing eta stats. And it seems to be stuck:
> 212 Thread 0x7fa9a086e700 (LWP 6509) 0x0000003974c0b98e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
can you do a bt on that? This looks like it _might_ be an issue that was
fixed recently with stats. Are you running fio with any output or status
options?
--
Jens Axboe
next prev parent reply other threads:[~2014-12-15 17:37 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-08-11 15:44 fio main thread got stuck over the weekend scameron
2014-08-11 16:04 ` scameron
2014-08-22 19:04 ` Jens Axboe
2014-08-22 19:09 ` scameron
2014-08-22 19:11 ` Jens Axboe
2014-12-12 20:32 ` Elliott, Robert (Server Storage)
2014-12-13 4:49 ` Jens Axboe
2014-12-15 17:33 ` Elliott, Robert (Server Storage)
2014-12-15 17:37 ` Jens Axboe [this message]
2014-12-15 19:39 ` Elliott, Robert (Server Storage)
2014-12-15 20:12 ` Jens Axboe
2014-12-15 20:31 ` Jens Axboe
2014-12-15 20:49 ` Jens Axboe
2014-12-16 0:52 ` Elliott, Robert (Server Storage)
2014-12-16 2:51 ` Jens Axboe
2014-12-16 22:43 ` Jens Axboe
2014-12-17 3:52 ` Elliott, Robert (Server Storage)
2014-12-17 5:43 ` Jens Axboe
2014-12-17 16:48 ` Elliott, Robert (Server Storage)
2014-12-17 17:27 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=548F1C65.2070501@kernel.dk \
--to=axboe@kernel.dk \
--cc=Elliott@hp.com \
--cc=fio@vger.kernel.org \
--cc=stephenmcameron@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox