From: Tomasz Grabiec <tgrabiec@scylladb.com>
To: linux-xfs@vger.kernel.org
Subject: io_submit() blocks for writes for substantial amount of time
Date: Tue, 19 Sep 2017 10:50:51 +0200 [thread overview]
Message-ID: <CAO2XSW6cctUGx18AdQQ29zHpjPiJ2vVJ-MHxrGcmoeMxzMvbbA@mail.gmail.com> (raw)
Hi,
On some systems we are seeing one of our tests to trigger io_submit()
calls to block when submitting writes for an order of 100ms [1]. This
is problematic, because we heavily rely on io_submit() being async.
Workload: open, (ftruncate, append*)*, close.
Kernel version: 4.12.9-300.fc26.x86_64
mount: /dev/nvme0n1p3 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
The blocking happens in the following places:
(1)
7fff9287472f __schedule ([kernel.kallsyms])
7fff92874d16 schedule ([kernel.kallsyms])
7fff92878d42 schedule_timeout ([kernel.kallsyms])
7fff92876478 wait_for_completion ([kernel.kallsyms])
7fffc05bf231 xfs_buf_submit_wait ([kernel.kallsyms])
7fffc05bf3d3 _xfs_buf_read ([kernel.kallsyms])
7fffc05bf4e4 xfs_buf_read_map ([kernel.kallsyms])
7fffc05f53ca xfs_trans_read_buf_map ([kernel.kallsyms])
7fffc058b432 xfs_btree_read_buf_block.constprop.34
([kernel.kallsyms])
7fffc058b504 xfs_btree_lookup_get_block ([kernel.kallsyms])
7fffc058f6ad xfs_btree_lookup ([kernel.kallsyms])
7fffc0570919 xfs_alloc_lookup_eq ([kernel.kallsyms])
7fffc0570c59 xfs_alloc_fixup_trees ([kernel.kallsyms])
7fffc0573a2d xfs_alloc_ag_vextent_near ([kernel.kallsyms])
7fffc0573db1 xfs_alloc_ag_vextent ([kernel.kallsyms])
7fffc05749cb xfs_alloc_vextent ([kernel.kallsyms])
7fffc0585ba8 xfs_bmap_btalloc ([kernel.kallsyms])
7fffc058605e xfs_bmap_alloc ([kernel.kallsyms])
7fffc0586d6d xfs_bmapi_write ([kernel.kallsyms])
7fffc05cedd1 xfs_iomap_write_direct ([kernel.kallsyms])
7fffc05cf0ec xfs_file_iomap_begin ([kernel.kallsyms])
7fff922d46ca iomap_apply ([kernel.kallsyms])
7fff922d4dfb iomap_dio_rw ([kernel.kallsyms])
7fffc05c4091 xfs_file_dio_aio_write ([kernel.kallsyms])
7fffc05c456d xfs_file_write_iter ([kernel.kallsyms])
7fff922bc5d3 aio_write ([kernel.kallsyms])
7fff922bcec1 do_io_submit ([kernel.kallsyms])
7fff922bdd40 sys_io_submit ([kernel.kallsyms])
7fff9287a6b7 entry_SYSCALL_64_fastpath ([kernel.kallsyms])
687 io_submit (/usr/lib64/libaio.so.1.0.1)
112373 seastar::reactor::flush_pending_aio
(/home/tgrabiec/src/scylla/build/release/tests/perf/perf_fast_forward_g)
(2)
7fff9287472f __schedule ([kernel.kallsyms])
7fff92874d16 schedule ([kernel.kallsyms])
7fffc05e6265 _xfs_log_force ([kernel.kallsyms])
7fffc05c2518 xfs_extent_busy_flush ([kernel.kallsyms])
7fffc0572ccd xfs_alloc_ag_vextent_size ([kernel.kallsyms])
7fffc0573d91 xfs_alloc_ag_vextent ([kernel.kallsyms])
7fffc05749cb xfs_alloc_vextent ([kernel.kallsyms])
7fffc0585ba8 xfs_bmap_btalloc ([kernel.kallsyms])
7fffc058605e xfs_bmap_alloc ([kernel.kallsyms])
7fffc0586d6d xfs_bmapi_write ([kernel.kallsyms])
7fffc05cedd1 xfs_iomap_write_direct ([kernel.kallsyms])
7fffc05cf0ec xfs_file_iomap_begin ([kernel.kallsyms])
7fff922d46ca iomap_apply ([kernel.kallsyms])
7fff922d4dfb iomap_dio_rw ([kernel.kallsyms])
7fffc05c4091 xfs_file_dio_aio_write ([kernel.kallsyms])
7fffc05c456d xfs_file_write_iter ([kernel.kallsyms])
7fff922bc5d3 aio_write ([kernel.kallsyms])
7fff922bcec1 do_io_submit ([kernel.kallsyms])
7fff922bdd40 sys_io_submit ([kernel.kallsyms])
7fff9287a6b7 entry_SYSCALL_64_fastpath ([kernel.kallsyms])
687 io_submit (/usr/lib64/libaio.so.1.0.1)
112373 seastar::reactor::flush_pending_aio
(/home/tgrabiec/src/scylla/build/release/tests/perf/perf_fast_forward_g)
Please advise, is this a known bug? When can it happen? Is there a way
to work it around to avoid blocking?
[1] https://github.com/scylladb/seastar/issues/340
Regards,
Tomasz Grabiec
next reply other threads:[~2017-09-19 8:50 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-19 8:50 Tomasz Grabiec [this message]
2017-09-19 12:27 ` io_submit() blocks for writes for substantial amount of time Brian Foster
2017-09-19 14:58 ` Christoph Hellwig
2017-09-19 16:31 ` Avi Kivity
2017-09-19 17:39 ` Brian Foster
2017-09-19 20:34 ` Christoph Hellwig
2017-09-20 6:17 ` Avi Kivity
2017-09-20 10:50 ` Brian Foster
2017-09-20 11:11 ` Avi Kivity
2017-09-20 14:49 ` Christoph Hellwig
2017-09-23 18:23 ` Avi Kivity
2017-09-19 20:34 ` Christoph Hellwig
2017-09-20 6:14 ` Avi Kivity
2017-09-19 16:29 ` Avi Kivity
2017-09-19 17:38 ` Brian Foster
2017-09-19 17:53 ` Tomasz Grabiec
2017-09-19 23:38 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAO2XSW6cctUGx18AdQQ29zHpjPiJ2vVJ-MHxrGcmoeMxzMvbbA@mail.gmail.com \
--to=tgrabiec@scylladb.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).