From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C65FC43603 for ; Thu, 12 Dec 2019 01:29:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E45BC21655 for ; Thu, 12 Dec 2019 01:29:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="tBcfwgxJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727453AbfLLB3w (ORCPT ); Wed, 11 Dec 2019 20:29:52 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:42979 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727351AbfLLB3v (ORCPT ); Wed, 11 Dec 2019 20:29:51 -0500 Received: by mail-pf1-f193.google.com with SMTP id 4so211201pfz.9 for ; Wed, 11 Dec 2019 17:29:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=DhyX9yKmk3Bjq16+cH6CFIH0oBue170bHYeXiMOQ8dA=; b=tBcfwgxJPBhRRcAGhCQgQbyfTfpQsyiGY9DGe/XHH2/qbpQARxKq7yWjGdvqo1IllS dEWH4g1714qg29h4GEZigEfbqrKvxMzRE9LNVpEFK2NyrdjbNtkxs6UWbtUszECnOQ0D VEoKTKtnfhk/vEZGeDpWMdwNOGh8La+bbktv7kvota82pgoXLzk3sL28vntZlDhqgkLX +EyfDp8OYP1xKcWrZTib+Ng+VSUqbKeEhhTBmOwpw+Ojk7X5nKqyrRMXwloZOmKkzPiZ +9yGgbU006bu6ndm0cPeWABAqJMpqOiOI+vxaISk+h/Wium9aaoWX5gz51q7TBIt5Pl3 er3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=DhyX9yKmk3Bjq16+cH6CFIH0oBue170bHYeXiMOQ8dA=; b=HPcc5XZUzkRSMQML5c8+4IDRApjsXJp0Q8jztVgPMEonFYColnOrayn7bUn5NAnb9J q+Gf3fR2JJtdOu7QCiaLnlhh0mMjxQNR6Ird4CfwW0LQ7FqZzBUFXDDe1DAW9ol4fkvT EMaGSc8r85hP48SBzPNmwMWETxikRdqNmQEdmUpeDt0kiOlNKjaCvHjc5xxCJOl1kf0P bPC0sypWijjLbZNSxRpxP3wEJkYfZax+xjQxo0E7aS/cdyfuUAe4l9A0qk8lHYjyAmgh 3R2y9SBHzfXfZd/9B53Z4Wuq7L/u4OLOHKUdZ65E7OhaS/tztGnplKTEl8Xy7JXvPRtQ Fy/A== X-Gm-Message-State: APjAAAUVHNU7bSeo5n6YwF46zXfvzNn6erDV49+O/EXkl8NoXWIZk8wW vt6tGA3oAH7EOf+RzyX43Au+8HVZyyXwcQ== X-Google-Smtp-Source: APXvYqwIZT80qb6KK6M7DWfi9Y0s3aK7ho5OWU9hmanRrqTeBJgLbXHTk13enGqUZ5WC+TJcO9HU0Q== X-Received: by 2002:a63:fd0a:: with SMTP id d10mr7322984pgh.197.1576114191223; Wed, 11 Dec 2019 17:29:51 -0800 (PST) Received: from [192.168.1.188] ([66.219.217.145]) by smtp.gmail.com with ESMTPSA id z14sm2580858pfg.57.2019.12.11.17.29.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Dec 2019 17:29:50 -0800 (PST) Subject: Re: [PATCHSET v3 0/5] Support for RWF_UNCACHED To: Linus Torvalds Cc: Linux-MM , linux-fsdevel , linux-block , Matthew Wilcox , Chris Mason , Dave Chinner , Johannes Weiner References: <20191211152943.2933-1-axboe@kernel.dk> <0d4e3954-c467-30a7-5a8e-7c4180275533@kernel.dk> <1c93194a-ed91-c3aa-deb5-a3394805defb@kernel.dk> From: Jens Axboe Message-ID: Date: Wed, 11 Dec 2019 18:29:46 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On 12/11/19 6:22 PM, Linus Torvalds wrote: > On Wed, Dec 11, 2019 at 5:11 PM Jens Axboe wrote: >> >> 15K is likely too slow to really show an issue, I'm afraid. The 970 >> is no slouch, but your crypt setup will likely hamper it a lot. You >> don't have a non-encrypted partition on it? > > No. I normally don't need all that much disk, so I've never upgraded > my ssd from the 512G size. > > Which means that it's actually half full or so, and I never felt like > "I should keep an unencrypted partition for IO testing", since I don't > generally _do_ any IO testing. > > I can get my load up with "numjobs=8" and get my iops up to the 100k > range, though. > > But kswapd doesn't much seem to care, the CPU percentage actually does > _down_ to 0.39% when I try that. Probably simply because now my CPU's > are busy, so they are running at 4.7Ghz instead of the 800Mhz "mostly > idle" state ... > > I guess I should be happy. It does mean that the situation you see > isn't exactly the normal case. I understand why you want to do the > non-cached case, but the case I think it the worrisome one is the > regular buffered one, so that's what I'm testing (not even trying the > noaccess patches). > > So from your report I went "uhhuh, that sounds like a bug". And it > appears that it largely isn't - you're seeing it because of pushing > the IO subsystem by another order of magnitude (and then I agree that > "under those kinds of IO loads, caching just won't help") I'd very much argue that it IS a bug, maybe just doesn't show on your system. My test box is a pretty standard 2 socket system, 24 cores / 48 threads, 2 nodes. The last numbers I sent were 100K IOPS, so nothing crazy, and granted that's only 10% kswapd cpu time, but that still seems very high for those kinds of rates. I'm surprised you see essentially no kswapd time for the same data rate. We'll keep poking here, I know Johannes is spending some time looking into the reclaim side. -- Jens Axboe