From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36F45C433DF for ; Thu, 18 Jun 2020 14:44:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B9B6220888 for ; Thu, 18 Jun 2020 14:44:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="LaeOybvg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9B6220888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 125918D001A; Thu, 18 Jun 2020 10:44:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0822A8D0012; Thu, 18 Jun 2020 10:44:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2E88D0019; Thu, 18 Jun 2020 10:44:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id D62168D0012 for ; Thu, 18 Jun 2020 10:44:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0C162181AC9BF for ; Thu, 18 Jun 2020 14:44:02 +0000 (UTC) X-FDA: 76942602324.02.story18_4100bef26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id D8F73343F8 for ; Thu, 18 Jun 2020 14:44:01 +0000 (UTC) X-HE-Tag: story18_4100bef26e11 X-Filterd-Recvd-Size: 7627 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:01 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id v11so2999225pgb.6 for ; Thu, 18 Jun 2020 07:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:reply-to:mime-version :content-transfer-encoding; bh=QtQZj6qHx4NyNp4faebxEBGjQhFGouVk8vIR/YM0OZ0=; b=LaeOybvgVFUedJ618PM1GTEUXkMXPY3qF3zemK0oPI0t/5c3IyDHmKKjFSXMlllNpg u4+VdHvZcEwrYY6QLxMvN9QNbcQkL40jfHhhwKnfjkRjaCBH/fdF2CXYNuGlJe/1qagv GOuftdx/mhTGZ0CDkYYM/BzpHCtO3WgFPHV5MzKmJusZp3RHe6UsIOj2jqBaJqopy9w1 8dyb1/ZNKWuCGje3eaRAvZHyHC8hhvG7uaNyzR+WZLwsMfnWHwLZir4H4MXYeJ5kkTgv S21Bzwg3kx7/cgMp23f6Z1a2ySJ2w9JTPWvpySOoRC3V8WNU7ZClsM3eXev1LkALavYt hZcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :mime-version:content-transfer-encoding; bh=QtQZj6qHx4NyNp4faebxEBGjQhFGouVk8vIR/YM0OZ0=; b=itLZItRwhYHsh1KQU/ERy8/ijLp+ZaN4l2FQ20HBvihTkH0iwbvqzoq/sOwHAkZfc0 hmI8hIKUuAP0YjmPUir8vFTwSc1R/aa0UTlOnF87zehwyMJiYY/gnUphaLtKQs8drVt8 mZbkHGDx/yuyMRYpMTjrFjiLN3jUJ69sa7uHTuit3L7nzq8vpAj7wNPHLL7EGEByBJn9 sQRxNuxJqzYuyxekTiHvGoqb8XpFuwyApl+/NKqoq4IqCqy/c6AmtbwhqINf3zBGDXSv yl1Uo22OrvlPfPuPCd5bvx1n2XWENIvs7B15vmswU0SGa1SpD2Wa2hYj5v2G3a4KYcAj ooUw== X-Gm-Message-State: AOAM5313WwExxiqlA7c8W2SwxWF/4W0jhoMkjjuuSRcdKx02SOCGgQaG 0gAJeuRJisrRtuFSvntm+nsfEw== X-Google-Smtp-Source: ABdhPJwVlCUbORgsthBtt/igrlr5i7l6tNK0VhZZRDtHj5c1nyTVCxDwaDg3pXHR9uxp5F9dJc2YrQ== X-Received: by 2002:a62:191:: with SMTP id 139mr3946645pfb.94.1592491439786; Thu, 18 Jun 2020 07:43:59 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:43:59 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org Subject: Date: Thu, 18 Jun 2020 08:43:40 -0600 Message-Id: <20200618144355.17324-1-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 Reply-To: "[PATCHSET v7 0/15]"@smtpin02.hostedemail.com, Add@smtpin02.hostedemail.com, support@smtpin02.hostedemail.com, for@smtpin02.hostedemail.com, async@smtpin02.hostedemail.com, buffered@smtpin02.hostedemail.com, reads@smtpin02.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: D8F73343F8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We technically support this already through io_uring, but it's implemented with a thread backend to support cases where we would block. This isn't ideal. After a few prep patches, the core of this patchset is adding support for async callbacks on page unlock. With this primitive, we can simply retry the IO operation. With io_uring, this works a lot like poll based retry for files that support it. If a page is currently locked and needed, -EIOCBQUEUED is returned with a callback armed. The callers callback is responsible for restarting the operation. With this callback primitive, we can add support for generic_file_buffered_read(), which is what most file systems end up using for buffered reads. XFS/ext4/btrfs/bdev is wired up, but probably trivial to add more. The file flags support for this by setting FMODE_BUF_RASYNC, similar to what we do for FMODE_NOWAIT. Open to suggestions here if this is the preferred method or not. In terms of results, I wrote a small test app that randomly reads 4G of data in 4K chunks from a file hosted by ext4. The app uses a queue depth of 32. If you want to test yourself, you can just use buffered=3D1 with ioengine=3Dio_uring with fio. No application changes are needed to use the more optimized buffered async read. preadv for comparison: real 1m13.821s user 0m0.558s sys 0m11.125s CPU ~13% Mainline: real 0m12.054s user 0m0.111s sys 0m5.659s CPU ~32% + ~50% =3D=3D ~82% This patchset: real 0m9.283s user 0m0.147s sys 0m4.619s CPU ~52% The CPU numbers are just a rough estimate. For the mainline io_uring run, this includes the app itself and all the threads doing IO on its behalf (32% for the app, ~1.6% per worker and 32 of them). Context switch rate is much smaller with the patchset, since we only have the one task performing IO. Also ran a simple fio based test case, varying the queue depth from 1 to 16, doubling every time: [buf-test] filename=3D/data/file direct=3D0 ioengine=3Dio_uring norandommap rw=3Drandread bs=3D4k iodepth=3D${QD} randseed=3D89 runtime=3D10s QD/Test Patchset IOPS Mainline IOPS 1 9046 8294 2 19.8k 18.9k 4 39.2k 28.5k 8 64.4k 31.4k 16 65.7k 37.8k Outside of my usual environment, so this is just running on a virtualized NVMe device in qemu, using ext4 as the file system. NVMe isn't very efficient virtualized, so we run out of steam at ~65K which is why we flatline on the patched side (nvme_submit_cmd() eats ~75% of the test app CPU). Before that happens, it's a linear increase. Not shown is context switch rate, which is massively lower with the new code. The old thread offload adds a blocking thread per pending IO, so context rate quickly goes through the roof. The goal here is efficiency. Async thread offload adds latency, and it also adds noticable overhead on items such as adding pages to the page cache. By allowing proper async buffered read support, we don't have X threads hammering on the same inode page cache, we have just the single app actually doing IO. Been beating on this and it's solid for me, and I'm now pretty happy with how it all turned out. Not aware of any missing bits/pieces or code cleanups that need doing. Series can also be found here: https://git.kernel.dk/cgit/linux-block/log/?h=3Dasync-buffered.7 or pull from: git://git.kernel.dk/linux-block async-buffered.7 block/blk-core.c | 6 + fs/block_dev.c | 2 +- fs/btrfs/file.c | 2 +- fs/ext4/file.c | 2 +- fs/io_uring.c | 336 ++++++++++++++++++++++++++++++++++------ fs/xfs/xfs_file.c | 2 +- include/linux/blkdev.h | 1 + include/linux/fs.h | 10 +- include/linux/pagemap.h | 75 +++++++++ mm/filemap.c | 110 ++++++++----- 10 files changed, 454 insertions(+), 92 deletions(-) Changes since v6: - Properly catch and resubmit requests that would have blocked in submission. - Fix sqthread mm usage for async buffered retry - Fix a retry condition iter setup - Rebase on master + for-5.9/io_uring Changes since v5: - Correct commit message, iocb->private -> iocb->ki_waitq - Get rid of io_uring goto, use an iter read helper Changes since v3: - io_uring: don't retry if REQ_F_NOWAIT is set - io_uring: alloc req->io if the request type didn't already - Add iocb->ki_waitq instead of (ab)using iocb->private Changes since v2: - Get rid of unnecessary wait_page_async struct, just use wait_page_async - Add another prep handler, adding wake_page_match() - Use wake_page_match() in both callers Changes since v1: - Fix an issue with inline page locking - Fix a potential race with __wait_on_page_locked_async() - Fix a hang related to not setting page_match, thus missing a wakeup --=20 Jens Axboe