* for-next branch and blktests/srp
@ 2018-12-06 16:47 Bart Van Assche
2018-12-06 17:00 ` Jens Axboe
2018-12-06 20:56 ` Bart Van Assche
0 siblings, 2 replies; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 16:47 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
Hello,
If I merge Jens' for-next branch with Linus' master branch, boot the
resulting kernel in a VM and run blktests/tests/srp/002 then that test
never finishes. The same test passes against Linus' master branch. I
think this is a regression. The following appears in the system log if
I run that test:
Call Trace:
INFO: task kworker/0:1:12 blocked for more than 120 seconds.
Call Trace:
INFO: task ext4lazyinit:2079 blocked for more than 120 seconds.
Call Trace:
INFO: task fio:2151 blocked for more than 120 seconds.
Call Trace:
INFO: task fio:2154 blocked for more than 120 seconds.
My list-pending-block-requests script produces the following output:
dm-0
dm-0/hctx0/active:0
dm-0/hctx0/dispatch:00000000b856a515 {.op=WRITE_ZEROES, .cmd_flags=SYNC, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1064, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006a2dca0a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1065, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001baa5734 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1085, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002d618841 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1025, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c8e6e436 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1250, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007af12ca3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1057, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002ad59d1e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1060, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000051c35166 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1188, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000005da9610 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1253, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a25483c5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1254, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007ef87035 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1255, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008a8f15c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1261, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d65ae959 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1263, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000088aac981 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1264, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000089489e70 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1265, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007c69e2d2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1154, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e32aa9b5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1155, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005b88a332 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1157, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001de14ef0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1159, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a380d0f3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1161, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c48cfa16 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1162, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000011d96382 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1163, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b5efd5d7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1167, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000080f5c476 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1153, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c73c244d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1266, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000069c43800 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1268, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d2ec3c96 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1269, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009b2099db {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1270, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007d433090 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1272, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000afa7d2ea {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1274, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c0f9d890 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1276, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000277c57ba {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1278, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e299166f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1279, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007d12f7a1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1169, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000041100ee6 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1170, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cbef182e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1173, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000056a9c7a8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1175, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006fa157cf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1176, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000016881f2d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1178, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000799cbaec {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1181, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006d78d0a3 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1183, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000890a4fe {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1184, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ac5fab20 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1186, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001ace99e2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1187, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a6813ac5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1189, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bce48fde {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1192, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000956251f0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1193, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000003a297eb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1237, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003b729294 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1283, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000086c5f07b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1284, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f26776c4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1285, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001091b0fa {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1286, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ddcecb7b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1287, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000075882d71 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1288, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b64a2b08 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1289, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000078d26448 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1290, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000517185f4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1291, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001d9e027f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1292, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d2f35e6b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1293, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f80f3073 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1294, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000012c0cd56 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1295, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cf6dd611 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1296, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fa1be708 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1297, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b4111b4a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1298, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008ff264cd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1299, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c225ab84 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1300, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003d257a81 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1301, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ca0c332d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1236, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000017323233 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1235, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e5bb1e25 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1260, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000033204cfa {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1302, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c11297da {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1303, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ce0e7bcc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1304, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000edc98d1b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1305, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006186ae5a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1238, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000567ee3e1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1257, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cd87086b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1259, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000097590304 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1256, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000013d23d5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1239, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005bb4d812 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1240, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000e72b8f8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1241, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009fb87abd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1242, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bd74a4f0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1243, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a2220d54 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1245, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000346bf931 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1251, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002164c010 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1252, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cf8afb6e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1258, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000085b395c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1216, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000078854d06 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1217, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000af9d1cd8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1218, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000011b7e98b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1007, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003b39baf0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=966, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000830f86c8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=977, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c0ac2d99 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=987, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001e9f1f88 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1944, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000adbffd9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=170, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001bdd41d7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=171, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000031a4eb43 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=172, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000afbc97c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=173, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002afba87e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=174, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b39122c8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=175, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000014114226 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1920, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e3462be2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2018, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000747cff6b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=20, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b5343170 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=32, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008d896c16 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1978, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e8e14028 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2006, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000002e38a52 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1983, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f33f8b2e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2011, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000068182611 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=5, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d8e84f8a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=62, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000060801d7f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=178, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000090254428 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=180, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008ae77270 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=179, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000039ab6bb2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=182, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fa6f97a6 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=185, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000062a8b22a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=184, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000137e9326 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=187, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001ffdb707 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2014, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000253a5e0e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=188, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000069869182 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=189, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bded495b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1669, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c58aa320 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=190, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000030c2307b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1670, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a3d5f5e3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1675, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ee2345f9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1671, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ef87a801 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1676, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004fdd1d0c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1678, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007242e657 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1677, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006ec28463 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1679, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005cac402f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1681, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004a3a05f6 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1680, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002c7f2dcc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1682, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fbb27d9f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1684, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fe21362d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1683, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b850bbdc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1685, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e2aad0bc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=63, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000033349182 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1686, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e50e0cd2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=9, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a471591a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=10, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000087198d52 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=13, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d6fbf80c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=22, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004b68644d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=23, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d41fd075 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=25, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000226621e3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=26, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006bb14b73 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=29, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002b36e4b7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=30, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000dd9c949a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=34, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c1926c67 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=35, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000763ef6c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=37, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b076f65c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=38, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000079cc7ceb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=40, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000def04e4c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=46, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f6d29752 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=48, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000324843f2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=50, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000071bdde3b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=53, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004910b5a9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=58, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000070571d65 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=60, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000061bcc04 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=74, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c7f39293 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=76, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000db1f04be {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=83, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ac2c8c88 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=77, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000238abcec {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=78, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000016e70cbc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=79, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000022d17790 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=80, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f0b08a8f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=81, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008b28cfbf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=82, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000074c36990 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=84, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000013c8809e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=85, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004b83cfb0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=163, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b87cbecc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=167, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000093e7f938 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=169, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001f5af434 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=176, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000601493bd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=177, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008673f1a5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=181, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d40033d4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=183, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007656d561 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=186, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fe74aa73 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=248, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002a903c93 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1687, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000768f847c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1688, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d4443f6f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1689, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000c8a9752 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1690, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000035ce87f4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1691, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008cbdbdcc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1692, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000044816f59 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1693, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003b5a927c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1694, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bc22f89d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1695, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000097df167f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=6, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000171d8839 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=253, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c1c2662e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=249, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000999d2818 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=254, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000060fe207d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=8, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d9f62a12 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=3, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000493d7341 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1749, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000075850d9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=39, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000056544de4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1750, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000067eeec15 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=27, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000039c351cd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=41, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ee44200f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=33, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d13e5409 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=42, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005cbd2919 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000caf11e82 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=44, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000057e751c7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=49, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e0065791 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=57, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c366b8d2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=56, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cb40d1bc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=103, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e4c45a67 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=115, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c621106e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=104, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b6b4a4a8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=116, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000853035e8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=126, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003fca33c7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=125, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007f225358 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=255, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000005868aaf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=192, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009a8fe9df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=75, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e192e9cd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=193, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002b115cfb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=194, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000972823d2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=196, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000020b4a46e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=203, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009bd03ccf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=205, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a529ee6f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=207, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004af732f6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=112, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007b9dc168 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=211, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006865c3a0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=130, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002d25c7c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=214, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004cdf0164 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=215, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004a08a974 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=216, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f70fe708 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=217, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cef6fa04 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=218, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009047bfe1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=219, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000abcb147 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=220, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000026a3354e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=221, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000874cb36 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=222, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009c79518c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=223, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d2049175 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=224, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008aed7492 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=225, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006452e6dd {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=226, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000024cbae99 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=227, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000081b94d50 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=228, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c480306c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=230, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006a3f7cc5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=229, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a4677973 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=231, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000049b11918 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=156, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005fd88e58 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=119, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009299bdc5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=19, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001cf2cef5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=31, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b888851f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=237, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000060fec44b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=238, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005ed27908 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=240, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000008b71cbc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=242, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000de24b44b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=244, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008d737ff9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=245, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001e14a7b9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=246, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bed1b068 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=247, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000ab3074c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=250, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a1166dac {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=251, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000416dceb3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=252, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002f9b59ca {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=256, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004708baa8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=258, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000069ca55cf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=257, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000043da84d6 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=259, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008bada904 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=261, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a5bf93df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=260, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f65b08b3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=262, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e43c68a8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=264, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002a99dd02 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=263, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000090f57825 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=265, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fae029c6 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=267, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000027c0bd25 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=266, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000f1ad49d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=124, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000014a5c29d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=273, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000042aaa54d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=272, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000369b0c0c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=274, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003289eedc {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=12, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002052017a {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1752, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000080b0bdc5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=93, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000082fad4a9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=92, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000062f02f90 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=96, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b74d9bd9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=120, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008885ac81 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=102, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001d217609 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=127, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a5ac2e97 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=87, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000081516c88 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=86, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000049069f56 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=89, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fc88c147 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=91, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c62dd0c5 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=90, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fe6525ca {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=94, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000025184095 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=123, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009d91012a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=139, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b3546627 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=138, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000045456fba {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=140, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a31c731c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=142, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000084b4efc6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=141, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000027cfcf8c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=143, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000013ee8a39 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=144, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000007a0033e {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=145, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000057393989 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=146, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e8c26d34 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=148, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000887aed1b {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=147, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000055b33b38 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=276, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000edf505dd {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=278, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001f2bbda1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=277, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005a848fd0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=279, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a24645a0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=95, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000041315939 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=155, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000deda3601 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=99, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000798240ed {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=70, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000189ff80b {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=101, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a6002fe1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=152, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009432f816 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=107, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000779a20f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=154, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005f65337a {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=282, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006e3523c0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=284, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009ed69db4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=283, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000f552b84 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=285, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005eb1f8b7 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1729, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000292c2cf8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1728, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001c163d5a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1696, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b48755e4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1666, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c1d7b392 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1665, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000765c5206 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1667, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000092972577 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1672, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000086ac6f2f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1668, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f2a0b4e7 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1673, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ca378fb7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1674, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f43011bc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1697, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009093c524 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1699, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000068f42a06 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1698, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004bfe8c64 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1700, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f7d16da2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1664, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007d614b24 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1701, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000022968545 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1702, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001f52e1d8 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1704, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ad04d966 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1703, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e1e5612e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1705, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000064704adb {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1793, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c1e42adc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1792, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000260f07dc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1794, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000ef82e8b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1796, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004011d877 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1795, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003610c09b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1797, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008194930f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1800, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000de719a93 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1799, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008bce1df0 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1801, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000002437233 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1984, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002db0ae68 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1976, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cd3a33da {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=2009, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006e89471e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=59, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000045a51949 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=117, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000681f0df2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=118, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a43a322b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=122, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c6d1a03e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1980, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a64bbed1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1981, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000068e3de60 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1982, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000f5d867c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=158, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b5407556 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=159, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000dc25cbf4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=160, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000039d35f39 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=162, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004d1b35d2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=164, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000065156d36 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=165, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000555543dc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=166, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002e9a5247 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=168, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009b0061e4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1986, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000075010611 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1987, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d860dcfb {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1988, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001ee3a599 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=204, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000028725c68 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=206, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c23d3eab {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=208, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004e6b3c80 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=209, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b167024b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=210, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005b1b8a1d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=212, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009a1e8933 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=232, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006096785d {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=213, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e4d589ee {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=326, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000072238f75 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=327, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008388b94d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=328, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a61e50f6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=329, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000488bb4b1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=330, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a989f582 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=331, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d3c23cca {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=332, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b76ed29c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=333, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000012b36220 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=334, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fcd73ae0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=128, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e7f8e284 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=233, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008ab1c9dd {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=234, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000042c74c8a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=235, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002ec2c820 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=11, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000051dabf7c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=15, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009bafbc40 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=236, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000405c282c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=239, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000047109c85 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=241, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000043a531e1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=129, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000005d369bd {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=131, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000987c5c0b {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=132, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e36e8c44 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=133, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000786f5341 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=135, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000f76da15 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=136, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f7b9a47a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=137, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000755bd27a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=157, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000672cd01 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=54, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000086404855 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=191, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000478e8f75 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=149, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000ae94107 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=150, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002c38310a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=151, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000010f8ddd6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=275, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f9f293b6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=153, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008d6fa751 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=161, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002f6bd1bc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=286, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ea53fdc2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=287, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000649da63b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=288, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d741c594 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=289, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f72656a2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=290, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000d99f513 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=291, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000059efeab9 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=292, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000043ad1792 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=293, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000168c5d8b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=294, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000048adad25 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=295, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000222f6f7c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=64, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002856dcca {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=65, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a402c93b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=66, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004860ba73 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=67, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d43538f4 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=88, .internal_tag=-1}
dm-0/hctx0/dispatch:000000008f7b7cce {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=97, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000036ec0a5c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=100, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bc97ae14 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=105, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000eec8ddd3 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=108, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002d37e5dd {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=109, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fae591f2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=111, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000551202df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=106, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d38666df {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=110, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c9d2ceaf {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1408, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ac462f42 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1409, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000e892f02 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1410, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006edb6303 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1411, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000595e154d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1413, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b14cecb4 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1412, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d6bc060d {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1414, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000434804d5 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1415, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000048103371 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1416, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000743c721e {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1417, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000afe9b0e1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1418, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003c3b18c8 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1419, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000060830e27 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1420, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000cf74b54a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1421, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002b76cad0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1422, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006f71192e {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=195, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a138d8dc {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=296, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005b141e7f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=297, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001920e100 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=298, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a79d1f5f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=299, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001b4b1ea3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=300, .internal_tag=-1}
dm-0/hctx0/dispatch:000000004b71f9ee {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=301, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000c7e9dfc3 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=302, .internal_tag=-1}
dm-0/hctx0/dispatch:000000002a17910b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=303, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ca6d3e4e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=304, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000089d492cc {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=305, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000074b9fe4f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=306, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000574720a1 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=307, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000bdf55c3a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=308, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005a6e4d81 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=309, .internal_tag=-1}
dm-0/hctx0/dispatch:000000000f3242e3 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=310, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a6fc56b0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=311, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000ad6e2271 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=312, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000fd9e0abb {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=313, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e5f3be65 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=314, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e72f30f2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=315, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000a339299f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=316, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001384c375 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=317, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003de11460 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=318, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000086424f33 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=319, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000535a93ac {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=280, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000566f43a7 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=281, .internal_tag=-1}
dm-0/hctx0/dispatch:000000003188c91b {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=320, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000923f17e6 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=321, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006a50aea4 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=322, .internal_tag=-1}
dm-0/hctx0/dispatch:000000005a2f7adf {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=323, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b907742f {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=200, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000be6ac7a0 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=197, .internal_tag=-1}
dm-0/hctx0/dispatch:000000006b762571 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=201, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000d91e05aa {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=325, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000dd2dc48b {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=324, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000080963049 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=243, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000f9a743af {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=134, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000eb1218cc {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=268, .internal_tag=-1}
dm-0/hctx0/dispatch:000000009d58e642 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=269, .internal_tag=-1}
dm-0/hctx0/dispatch:000000007f09fdef {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=270, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000010cc1a51 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT|STATS, .state=idle, .tag=271, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e2859f1c {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=1, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000b69f4bee {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=14, .internal_tag=-1}
dm-0/hctx0/dispatch:0000000057c0e072 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=121, .internal_tag=-1}
dm-0/hctx0/dispatch:000000001158ea2f {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=198, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000931cc554 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=199, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000be67eff2 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=202, .internal_tag=-1}
dm-0/hctx0/dispatch:00000000e27e528e {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=335, .internal_tag=-1}
dm-0/hctx0/flags:alloc_policy=FIFO SHOULD_MERGE|SG_MERGE
dm-0/hctx0/state:SCHED_RESTART
dm-0/hctx0/tags:nr_tags=2048
dm-0/hctx0/tags:nr_reserved_tags=0
dm-0/hctx0/tags:active_queues=0
dm-0/hctx0/tags:bitmap_tags:
dm-0/hctx0/tags:depth=2048
dm-0/hctx0/tags:busy=508
dm-0/hctx0/tags:cleared=896
dm-0/hctx0/tags:bits_per_word=64
dm-0/hctx0/tags:map_nr=32
dm-0/hctx0/tags:alloc_hint={942, 739, 1233, 1000, 2017, 361}
dm-0/hctx0/tags:wake_batch=8
dm-0/hctx0/tags:wake_index=0
dm-0/hctx0/tags:ws_active=0
dm-0/hctx0/tags:ws={
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags: {.wait_cnt=8, .wait=inactive},
dm-0/hctx0/tags:}
dm-0/hctx0/tags:round_robin=0
dm-0/hctx0/tags:min_shallow_depth=4294967295
dm-0/hctx0/tags_bitmap:00000000: ffff ffff ff57 67df ffff ffff ffff f9ff
dm-0/hctx0/tags_bitmap:00000010: ffff ffff ffff ffff ffff ffff ffff ffff
dm-0/hctx0/tags_bitmap:00000020: ffff ffff ffff ffff ffff ffff ff07 00c0
dm-0/hctx0/tags_bitmap:00000030: 0100 0000 0000 0000 ffff ffff ffff ffff
dm-0/hctx0/tags_bitmap:00000040: 0300 0000 0000 8abf ffff ffff ffff 8348
dm-0/hctx0/tags_bitmap:00000050: ff03 0000 0000 0ec6 ffff ffff ff0f 0080
dm-0/hctx0/tags_bitmap:00000060: 0100 0000 a0ef feff fbff ffff ffdf ffff
dm-0/hctx0/tags_bitmap:00000070: ffff ff07 0040 0000 ffff ffff fff9 ff3f
dm-0/hctx0/tags_bitmap:00000080: ffbf 22a9 f383 ff30 ffff ffaf 4b7f 55fe
dm-0/hctx0/tags_bitmap:00000090: ffff ffff fffb feff ffff ff3f ffff ffff
dm-0/hctx0/tags_bitmap:000000a0: ffff ffff ffff ff37 ffff ffff 8710 e3e4
dm-0/hctx0/tags_bitmap:000000b0: ff7f 4000 0d00 0000 ffff ffbf ffef fef7
dm-0/hctx0/tags_bitmap:000000c0: ffe1 ff7f ffff fff7 ffff ffff ffdf 9aff
dm-0/hctx0/tags_bitmap:000000d0: ffff ffff ff03 807f 038e 6a03 0000 0000
dm-0/hctx0/tags_bitmap:000000e0: ff0b 0d00 0000 0000 0f00 0000 8003 c020
dm-0/hctx0/tags_bitmap:000000f0: ffff 0101 0000 00f5 ffff ffff ff3f 00f6
dm-0/hctx0/cpu0/rq_list:00000000765c5f21 {.op=WRITE, .cmd_flags=SYNC|IDLE, .rq_flags=IO_STAT|STATS, .state=idle, .tag=909, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:00000000cbc163b2 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=740, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:00000000ff5bbeca {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=741, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:0000000099036a7c {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=742, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:0000000069feed1b {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=743, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:0000000014043f13 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=744, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:00000000c470b9f5 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=745, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:00000000408adf7a {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=746, .internal_tag=-1}
dm-0/hctx0/cpu1/rq_list:000000005c3065f1 {.op=READ, .cmd_flags=, .rq_flags=IO_STAT, .state=idle, .tag=747, .internal_tag=-1}
dm-0/hctx0/cpu4/rq_list:00000000c16d629a {.op=WRITE, .cmd_flags=META|PRIO|BACKGROUND, .rq_flags=IO_STAT, .state=idle, .tag=2022, .internal_tag=-1}
dm-0/hctx0/cpu4/rq_list:0000000089f6f68c {.op=WRITE, .cmd_flags=META|PRIO|BACKGROUND, .rq_flags=IO_STAT, .state=idle, .tag=2026, .internal_tag=-1}
dm-0/hctx0/cpu4/rq_list:00000000fa997151 {.op=WRITE, .cmd_flags=META|PRIO|BACKGROUND, .rq_flags=IO_STAT, .state=idle, .tag=2027, .internal_tag=-1}
dm-0/hctx0/cpu4/rq_list:00000000125bb076 {.op=WRITE, .cmd_flags=META|PRIO|BACKGROUND, .rq_flags=IO_STAT, .state=idle, .tag=2028, .internal_tag=-1}
dm-0/hctx0/cpu4/rq_list:00000000552de129 {.op=WRITE, .cmd_flags=META|PRIO|BACKGROUND, .rq_flags=IO_STAT, .state=idle, .tag=2029, .internal_tag=-1}
dm-0/hctx0/cpu5/rq_list:0000000062b531aa {.op=WRITE, .cmd_flags=SYNC|META|PRIO, .rq_flags=IO_STAT, .state=idle, .tag=362, .internal_tag=-1}
dm-0/state:SAME_COMP|NONROT|IO_STAT|INIT_DONE|STATS|REGISTERED
dm-0/queue/scheduler:[none] bfq kyber mq-deadline
dm-1
dm-2
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
nullb0
nullb1
sda
sdb
sdc
sdd
sde
sr0
vda
vdb
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 16:47 for-next branch and blktests/srp Bart Van Assche
@ 2018-12-06 17:00 ` Jens Axboe
2018-12-06 17:19 ` Bart Van Assche
2018-12-06 20:56 ` Bart Van Assche
1 sibling, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 17:00 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 9:47 AM, Bart Van Assche wrote:
> Hello,
>
> If I merge Jens' for-next branch with Linus' master branch, boot the
> resulting kernel in a VM and run blktests/tests/srp/002 then that test
> never finishes. The same test passes against Linus' master branch. I
> think this is a regression. The following appears in the system log if
> I run that test:
You are running that test on a dm device? Can you shed some light on
how that dm device is setup?
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 17:00 ` Jens Axboe
@ 2018-12-06 17:19 ` Bart Van Assche
2018-12-06 17:28 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 17:19 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On Thu, 2018-12-06 at 10:00 -0700, Jens Axboe wrote:
> On 12/6/18 9:47 AM, Bart Van Assche wrote:
> > If I merge Jens' for-next branch with Linus' master branch, boot the
> > resulting kernel in a VM and run blktests/tests/srp/002 then that test
> > never finishes. The same test passes against Linus' master branch. I
> > think this is a regression. The following appears in the system log if
> > I run that test:
>
> You are running that test on a dm device? Can you shed some light on
> how that dm device is setup?
Hi Jens,
The dm device referred to in my e-mail is a dm-mpath device created by test
srp/002. All parameters of that device are under control of that test script.
From the srp/002 test script:
DESCRIPTION="File I/O on top of multipath concurrently with logout and login (mq)"
From srp/multipath.conf:
defaults {
find_multipaths no
user_friendly_names yes
queue_without_daemon no
}
devices {
device {
vendor "LIO-ORG|SCST_BIO|FUSIONIO"
product ".*"
features "1 queue_if_no_path"
path_checker tur
}
}
Bart.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 17:19 ` Bart Van Assche
@ 2018-12-06 17:28 ` Jens Axboe
2018-12-06 17:48 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 17:28 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 10:19 AM, Bart Van Assche wrote:
> On Thu, 2018-12-06 at 10:00 -0700, Jens Axboe wrote:
>> On 12/6/18 9:47 AM, Bart Van Assche wrote:
>>> If I merge Jens' for-next branch with Linus' master branch, boot the
>>> resulting kernel in a VM and run blktests/tests/srp/002 then that test
>>> never finishes. The same test passes against Linus' master branch. I
>>> think this is a regression. The following appears in the system log if
>>> I run that test:
>>
>> You are running that test on a dm device? Can you shed some light on
>> how that dm device is setup?
>
> Hi Jens,
>
> The dm device referred to in my e-mail is a dm-mpath device created by test
> srp/002. All parameters of that device are under control of that test script.
> From the srp/002 test script:
>
> DESCRIPTION="File I/O on top of multipath concurrently with logout and login (mq)"
>
> From srp/multipath.conf:
>
> defaults {
> find_multipaths no
> user_friendly_names yes
> queue_without_daemon no
> }
>
> devices {
> device {
> vendor "LIO-ORG|SCST_BIO|FUSIONIO"
> product ".*"
> features "1 queue_if_no_path"
> path_checker tur
> }
> }
5 kernel compiles later, and I think I'm almost ready to run it...
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 17:28 ` Jens Axboe
@ 2018-12-06 17:48 ` Jens Axboe
2018-12-06 18:02 ` Bart Van Assche
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 17:48 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 10:28 AM, Jens Axboe wrote:
> On 12/6/18 10:19 AM, Bart Van Assche wrote:
>> On Thu, 2018-12-06 at 10:00 -0700, Jens Axboe wrote:
>>> On 12/6/18 9:47 AM, Bart Van Assche wrote:
>>>> If I merge Jens' for-next branch with Linus' master branch, boot the
>>>> resulting kernel in a VM and run blktests/tests/srp/002 then that test
>>>> never finishes. The same test passes against Linus' master branch. I
>>>> think this is a regression. The following appears in the system log if
>>>> I run that test:
>>>
>>> You are running that test on a dm device? Can you shed some light on
>>> how that dm device is setup?
>>
>> Hi Jens,
>>
>> The dm device referred to in my e-mail is a dm-mpath device created by test
>> srp/002. All parameters of that device are under control of that test script.
>> From the srp/002 test script:
>>
>> DESCRIPTION="File I/O on top of multipath concurrently with logout and login (mq)"
>>
>> From srp/multipath.conf:
>>
>> defaults {
>> find_multipaths no
>> user_friendly_names yes
>> queue_without_daemon no
>> }
>>
>> devices {
>> device {
>> vendor "LIO-ORG|SCST_BIO|FUSIONIO"
>> product ".*"
>> features "1 queue_if_no_path"
>> path_checker tur
>> }
>> }
>
> 5 kernel compiles later, and I think I'm almost ready to run it...
axboe@dell:/home/axboe/git/blktests $ sudo ./check srp/002
srp/002 (File I/O on top of multipath concurrently with logout and login (mq)) [failed]
runtime 33.689s ... 32.212s
--- tests/srp/002.out 2018-11-09 08:42:22.842029804 -0700
+++ /home/axboe/git/blktests/results/nodev/srp/002.out.bad 2018-12-06 10:45:35.995183521 -0700
@@ -1,2 +1 @@
Configured SRP target driver
-Passed
It runs for about 32-33s every time and then I get this. The fio output shows this:
data-integrity-test-mq: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.12-27-g68af
Starting 16 threads
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
data-integrity-test-mq: Laying out IO file (1 file / 1MiB)
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.14.0: Input/output error: read offset=724992, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.5.0: Input/output error: read offset=983040, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.7.0: Input/output error: write offset=700416, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.8.0: Input/output error: write offset=237568, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.9.0: Input/output error: write offset=1277952, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.9.0: Input/output error: write offset=970752, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.9.0: Input/output error: write offset=974848, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.9.0: Input/output error: write offset=978944, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.9.0: Input/output error: write offset=983040, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.15.0: Input/output error: write offset=172032, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.13.0: Input/output error: write offset=237568, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.3.0: Input/output error: write offset=294912, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.11.0: Input/output error: write offset=1245184, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.12.0: Input/output error: write offset=884736, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=389120, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=393216, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=397312, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=364544, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=368640, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=663552, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.1.0: Input/output error: write offset=1126400, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=102400, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=778240, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=221184, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.10.0: Input/output error: write offset=696320, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=286720, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=290816, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=647168, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=61440, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.0.0: Input/output error: write offset=73728, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=303104, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=311296, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=245760, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=65536, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=57344, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=290816, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=618496, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=503808, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=339968, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=905216, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=675840, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=311296, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=253952, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=937984, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=425984, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=114688, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=258048, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=507904, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.4.0: Input/output error: write offset=212992, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=262144, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=524288, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=266240, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=552960, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=270336, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=557056, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=245760, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.6.0: Input/output error: write offset=638976, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=249856, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1249280, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1003520, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1007616, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1150976, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=806912, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1011712, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=540672, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=544768, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=507904, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=811008, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=892928, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=585728, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1040384, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=831488, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=737280, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=638976, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1142784, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=335872, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1146880, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1085440, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1138688, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=217088, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=237568, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=241664, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1220608, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=999424, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=860160, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=864256, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=827392, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=598016, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1024000, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=663552, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=675840, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=688128, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=692224, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1064960, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=704512, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=716800, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=741376, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=778240, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=929792, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=933888, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1159168, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=303104, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1163264, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=393216, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=397312, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=1228800, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=659456, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=950272, buflen=4096
fio: io_u error on file /home/axboe/git/blktests/results/tmpdir.srp.002.7WR/mnt0/data-integrity-test-mq.2.0: Input/output error: write offset=110592, buflen=4096
fio: pid=7922, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7923, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7931, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7932, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7934, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7925, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7926, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7930, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7935, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7929, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7924, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7921, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7928, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7920, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7933, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
fio: pid=7927, err=5/file:io_u.c:1789, func=io_u error, error=Input/output error
data-integrity-test-mq: (groupid=0, jobs=16): err= 5 (file:io_u.c:1789, func=io_u error, error=Input/output error): pid=7920: Thu Dec 6 10:45:47 2018
read: IOPS=12.2k, BW=47.5MiB/s (49.8MB/s)(369MiB/7769msec)
write: IOPS=12.2k, BW=47.1MiB/s (49.4MB/s)(378MiB/8024msec); 0 zone resets
bw ( KiB/s): min=48208, max=112000, per=100.00%, avg=85859.56, stdev=1118.52, samples=144
iops : min=12050, max=28000, avg=21464.33, stdev=279.65, samples=144
cpu : usr=2.78%, sys=2.21%, ctx=93124, majf=0, minf=132
IO depths : 1=0.3%, 2=0.6%, 4=1.3%, 8=2.6%, 16=5.2%, 32=10.3%, >=64=79.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.6%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.4%, >=64=0.0%
issued rwts: total=94632,97502,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MB/s-49.8MB/s), io=369MiB (387MB), run=7769-7769msec
WRITE: bw=47.1MiB/s (49.4MB/s), 47.1MiB/s-47.1MiB/s (49.4MB/s-49.4MB/s), io=378MiB (396MB), run=8024-8024msec
Disk stats (read/write):
dm-2: ios=85051/90524, merge=9581/9463, ticks=2200798/5420861, in_queue=7709829, util=97.61%, aggrios=82306/87731, aggrmerge=0/0, aggrticks=107774/142105, aggrin_queue=264726, aggrutil=98.23%
sdh: ios=82306/87731, merge=0/0, ticks=107774/142105, in_queue=264726, util=98.23%
which would result in a non-zero exit, which should be expected for this
test?
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 17:48 ` Jens Axboe
@ 2018-12-06 18:02 ` Bart Van Assche
2018-12-06 18:08 ` Jens Axboe
2018-12-06 18:10 ` Bart Van Assche
0 siblings, 2 replies; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 18:02 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote:
> which would result in a non-zero exit, which should be expected for this
> test?
Hi Jens,
Test srp/002 simulates network failures while running fio on top of dm-mpath.
Since queue_if_no_path is enabled in multipath.conf dm-mpath will keep retrying
the block layer requests it receives if these requests fail due to path removal.
If fio reports "io_u error" for this test then that means that the test failed.
I haven't seen fio reporting any I/O errors for this test with any upstream
kernel that I tested in the past year or so.
Bart.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 18:02 ` Bart Van Assche
@ 2018-12-06 18:08 ` Jens Axboe
2018-12-06 18:10 ` Bart Van Assche
1 sibling, 0 replies; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 18:08 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 11:02 AM, Bart Van Assche wrote:
> On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote:
>> which would result in a non-zero exit, which should be expected for this
>> test?
>
> Hi Jens,
>
> Test srp/002 simulates network failures while running fio on top of dm-mpath.
> Since queue_if_no_path is enabled in multipath.conf dm-mpath will keep retrying
> the block layer requests it receives if these requests fail due to path removal.
> If fio reports "io_u error" for this test then that means that the test failed.
> I haven't seen fio reporting any I/O errors for this test with any upstream
> kernel that I tested in the past year or so.
Just ran it on current -git, and get the exact same results. Maybe there are
some clues in the log files I sent in the previous email?
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 18:02 ` Bart Van Assche
2018-12-06 18:08 ` Jens Axboe
@ 2018-12-06 18:10 ` Bart Van Assche
2018-12-06 18:12 ` Jens Axboe
1 sibling, 1 reply; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 18:10 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On Thu, 2018-12-06 at 10:02 -0800, Bart Van Assche wrote:
> On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote:
> > which would result in a non-zero exit, which should be expected for this
> > test?
>
> Test srp/002 simulates network failures while running fio on top of dm-mpath.
> Since queue_if_no_path is enabled in multipath.conf dm-mpath will keep retrying
> the block layer requests it receives if these requests fail due to path removal.
> If fio reports "io_u error" for this test then that means that the test failed.
> I haven't seen fio reporting any I/O errors for this test with any upstream
> kernel that I tested in the past year or so.
Hi Jens,
Please also verify that the version of multipath-tools that you are running includes
the following patch:
From a2675025ae9f652b005345b9082f5042b32c992c Mon Sep 17 00:00:00 2001
From: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Fri, 18 Nov 2016 13:33:44 -0800
Subject: [PATCH] Avoid that reloading a map sporadically triggers I/O errors
Avoid that reloading a map while there are no paths triggers a flush
and hence unwanted I/O errors if 'queue_if_no_path' is enabled.
Fixes: commit d569988e7528 ("libmultipath: Fixup 'DM_DEVICE_RELOAD' handling")
Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
---
libmultipath/devmapper.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libmultipath/devmapper.c b/libmultipath/devmapper.c
index 9b6b0537ba6d..1576dd017d6a 100644
--- a/libmultipath/devmapper.c
+++ b/libmultipath/devmapper.c
@@ -392,7 +392,7 @@ int dm_addmap_reload(struct multipath *mpp, char *params, int flush)
params, ADDMAP_RO, SKIP_KPARTX_OFF);
}
if (r)
- r = dm_simplecmd(DM_DEVICE_RESUME, mpp->alias, flush,
+ r = dm_simplecmd(DM_DEVICE_RESUME, mpp->alias, !flush,
1, udev_flags, 0);
return r;
}
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 18:10 ` Bart Van Assche
@ 2018-12-06 18:12 ` Jens Axboe
2018-12-06 18:16 ` Bart Van Assche
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 18:12 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 11:10 AM, Bart Van Assche wrote:
> On Thu, 2018-12-06 at 10:02 -0800, Bart Van Assche wrote:
>> On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote:
>>> which would result in a non-zero exit, which should be expected for this
>>> test?
>>
>> Test srp/002 simulates network failures while running fio on top of dm-mpath.
>> Since queue_if_no_path is enabled in multipath.conf dm-mpath will keep retrying
>> the block layer requests it receives if these requests fail due to path removal.
>> If fio reports "io_u error" for this test then that means that the test failed.
>> I haven't seen fio reporting any I/O errors for this test with any upstream
>> kernel that I tested in the past year or so.
>
> Hi Jens,
>
> Please also verify that the version of multipath-tools that you are running includes
> the following patch:
I installed from apt... Says this:
# apt show multipath-tools
Package: multipath-tools
Version: 0.6.4-5+deb9u1
I can run a self-compiled one, if this one doesn't work.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 18:12 ` Jens Axboe
@ 2018-12-06 18:16 ` Bart Van Assche
0 siblings, 0 replies; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 18:16 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On Thu, 2018-12-06 at 11:12 -0700, Jens Axboe wrote:
> On 12/6/18 11:10 AM, Bart Van Assche wrote:
> > On Thu, 2018-12-06 at 10:02 -0800, Bart Van Assche wrote:
> > > On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote:
> > > > which would result in a non-zero exit, which should be expected for this
> > > > test?
> > >
> > > Test srp/002 simulates network failures while running fio on top of dm-mpath.
> > > Since queue_if_no_path is enabled in multipath.conf dm-mpath will keep retrying
> > > the block layer requests it receives if these requests fail due to path removal.
> > > If fio reports "io_u error" for this test then that means that the test failed.
> > > I haven't seen fio reporting any I/O errors for this test with any upstream
> > > kernel that I tested in the past year or so.
> >
> > Hi Jens,
> >
> > Please also verify that the version of multipath-tools that you are running includes
> > the following patch:
>
> I installed from apt... Says this:
>
> # apt show multipath-tools
> Package: multipath-tools
> Version: 0.6.4-5+deb9u1
>
> I can run a self-compiled one, if this one doesn't work.
Please do. I use the following script to build and install multipath-tools
over the Debian binaries:
export LIB=/lib
apt-get install -y libaio-dev libdevmapper-dev libjson-c-dev librados-dev libreadline-dev libsystemd-dev liburcu-dev
make -s
make -s install
Bart.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 16:47 for-next branch and blktests/srp Bart Van Assche
2018-12-06 17:00 ` Jens Axboe
@ 2018-12-06 20:56 ` Bart Van Assche
2018-12-06 21:04 ` Jens Axboe
1 sibling, 1 reply; 13+ messages in thread
From: Bart Van Assche @ 2018-12-06 20:56 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On Thu, 2018-12-06 at 08:47 -0800, Bart Van Assche wrote:
> If I merge Jens' for-next branch with Linus' master branch, boot the
> resulting kernel in a VM and run blktests/tests/srp/002 then that test
> never finishes. The same test passes against Linus' master branch. I
> think this is a regression. The following appears in the system log if
> I run that test:
>
> Call Trace:
> INFO: task kworker/0:1:12 blocked for more than 120 seconds.
> Call Trace:
> INFO: task ext4lazyinit:2079 blocked for more than 120 seconds.
> Call Trace:
> INFO: task fio:2151 blocked for more than 120 seconds.
> Call Trace:
> INFO: task fio:2154 blocked for more than 120 seconds.
Hi Jens,
My test results so far are as follows:
* With kernel v4.20-rc5 test srp/002 passes.
* With your for-next branch test srp/002 reports the symptoms reported in my e-mail.
* With Linus' master branch from this morning test srp/002 fails in the same way as
your for-next branch.
* Also with Linus' master branch, test srp/002 passes if I revert the following commit:
ffe81d45322c ("blk-mq: fix corruption with direct issue"). So it seems like that
commit fixed one regression but introduced another regression.
Bart.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 20:56 ` Bart Van Assche
@ 2018-12-06 21:04 ` Jens Axboe
2018-12-06 22:15 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 21:04 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 1:56 PM, Bart Van Assche wrote:
> On Thu, 2018-12-06 at 08:47 -0800, Bart Van Assche wrote:
>> If I merge Jens' for-next branch with Linus' master branch, boot the
>> resulting kernel in a VM and run blktests/tests/srp/002 then that test
>> never finishes. The same test passes against Linus' master branch. I
>> think this is a regression. The following appears in the system log if
>> I run that test:
>>
>> Call Trace:
>> INFO: task kworker/0:1:12 blocked for more than 120 seconds.
>> Call Trace:
>> INFO: task ext4lazyinit:2079 blocked for more than 120 seconds.
>> Call Trace:
>> INFO: task fio:2151 blocked for more than 120 seconds.
>> Call Trace:
>> INFO: task fio:2154 blocked for more than 120 seconds.
>
> Hi Jens,
>
> My test results so far are as follows:
> * With kernel v4.20-rc5 test srp/002 passes.
> * With your for-next branch test srp/002 reports the symptoms reported in my e-mail.
> * With Linus' master branch from this morning test srp/002 fails in the same way as
> your for-next branch.
> * Also with Linus' master branch, test srp/002 passes if I revert the following commit:
> ffe81d45322c ("blk-mq: fix corruption with direct issue"). So it seems like that
> commit fixed one regression but introduced another regression.
Yes, I'm on the same page, I've been able to reproduce. It seems to be related
to dm and bypass insert, which is somewhat odd. If I just do:
diff --git a/block/blk-core.c b/block/blk-core.c
index deb56932f8c4..4c44e6fa0d08 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2637,7 +2637,8 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *
* bypass a potential scheduler on the bottom device for
* insert.
*/
- return blk_mq_request_issue_directly(rq);
+ blk_mq_request_bypass_insert(rq, true);
+ return BLK_STS_OK;
}
spin_lock_irqsave(q->queue_lock, flags);
it works fine. Well, at least this regression is less serious, I'll bang
out a fix for it and ensure we make -rc6. I'm guessing it's the bypassin
of non-read/write, which your top of dispatch also shows to be a
non-read/write. But there should be no new failure case here that wasn't
possible before, only it's easier to hit now.
--
Jens Axboe
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: for-next branch and blktests/srp
2018-12-06 21:04 ` Jens Axboe
@ 2018-12-06 22:15 ` Jens Axboe
0 siblings, 0 replies; 13+ messages in thread
From: Jens Axboe @ 2018-12-06 22:15 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-block@vger.kernel.org, Mike Snitzer
On 12/6/18 2:04 PM, Jens Axboe wrote:
> On 12/6/18 1:56 PM, Bart Van Assche wrote:
>> On Thu, 2018-12-06 at 08:47 -0800, Bart Van Assche wrote:
>>> If I merge Jens' for-next branch with Linus' master branch, boot the
>>> resulting kernel in a VM and run blktests/tests/srp/002 then that test
>>> never finishes. The same test passes against Linus' master branch. I
>>> think this is a regression. The following appears in the system log if
>>> I run that test:
>>>
>>> Call Trace:
>>> INFO: task kworker/0:1:12 blocked for more than 120 seconds.
>>> Call Trace:
>>> INFO: task ext4lazyinit:2079 blocked for more than 120 seconds.
>>> Call Trace:
>>> INFO: task fio:2151 blocked for more than 120 seconds.
>>> Call Trace:
>>> INFO: task fio:2154 blocked for more than 120 seconds.
>>
>> Hi Jens,
>>
>> My test results so far are as follows:
>> * With kernel v4.20-rc5 test srp/002 passes.
>> * With your for-next branch test srp/002 reports the symptoms reported in my e-mail.
>> * With Linus' master branch from this morning test srp/002 fails in the same way as
>> your for-next branch.
>> * Also with Linus' master branch, test srp/002 passes if I revert the following commit:
>> ffe81d45322c ("blk-mq: fix corruption with direct issue"). So it seems like that
>> commit fixed one regression but introduced another regression.
>
> Yes, I'm on the same page, I've been able to reproduce. It seems to be related
> to dm and bypass insert, which is somewhat odd. If I just do:
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index deb56932f8c4..4c44e6fa0d08 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -2637,7 +2637,8 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *
> * bypass a potential scheduler on the bottom device for
> * insert.
> */
> - return blk_mq_request_issue_directly(rq);
> + blk_mq_request_bypass_insert(rq, true);
> + return BLK_STS_OK;
> }
>
> spin_lock_irqsave(q->queue_lock, flags);
>
> it works fine. Well, at least this regression is less serious, I'll bang
> out a fix for it and ensure we make -rc6. I'm guessing it's the bypassin
> of non-read/write, which your top of dispatch also shows to be a
> non-read/write. But there should be no new failure case here that wasn't
> possible before, only it's easier to hit now.
OK, so here's the thing. As part of the corruption fix, we disallowed
direct dispatch for anything that wasn't a read or write. This means
that your WRITE_ZEROES will always fail direct dispatch. When it does,
we return busy. But next time dm will try the exact same thing again,
blk_insert_cloned_request() -> direct dispatch -> fail. Before we'd succeed
eventually, now we will always fail for that type.
The insert clone path is unique in that regard.
So we have two options - the patch I did above which always just does
bypass insert for DM, or we need to mark the request as having failed
and just not retry direct dispatch for it. I'm still not crazy about
exploring the dispatch insert off this path.
And we'd need to do this on the original request in dm, not the clone
we are passed or it won't be persistent. Hence I lean towards the
already posted patch.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2018-12-06 22:15 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-12-06 16:47 for-next branch and blktests/srp Bart Van Assche
2018-12-06 17:00 ` Jens Axboe
2018-12-06 17:19 ` Bart Van Assche
2018-12-06 17:28 ` Jens Axboe
2018-12-06 17:48 ` Jens Axboe
2018-12-06 18:02 ` Bart Van Assche
2018-12-06 18:08 ` Jens Axboe
2018-12-06 18:10 ` Bart Van Assche
2018-12-06 18:12 ` Jens Axboe
2018-12-06 18:16 ` Bart Van Assche
2018-12-06 20:56 ` Bart Van Assche
2018-12-06 21:04 ` Jens Axboe
2018-12-06 22:15 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).