From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 203B5C10F16 for ; Tue, 30 Apr 2024 18:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BC0D6B00AA; Tue, 30 Apr 2024 14:55:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 86B096B00AE; Tue, 30 Apr 2024 14:55:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7325D6B00AF; Tue, 30 Apr 2024 14:55:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 54FFF6B00AA for ; Tue, 30 Apr 2024 14:55:13 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0480EC0934 for ; Tue, 30 Apr 2024 18:55:12 +0000 (UTC) X-FDA: 82067100906.30.4A5BB16 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) by imf12.hostedemail.com (Postfix) with ESMTP id E287F40020 for ; Tue, 30 Apr 2024 18:55:10 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=FZ9gMJn6; dmarc=none; spf=pass (imf12.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.49 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714503311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SHR3RGRuxJEH/dBiKYgawHqY9Lnlphnx76oS7XwiA+c=; b=AIdapC6wsGIzjRMoo2CueSsqsS6hLGq9XRnji905H+FmMC2Q41XKJppofI9oaQWDLduqff NR30/mG6iDCmGRQ98tnkDIr5liaTvjJLCGcCl6wdwX5nyfgmdSQllPqDSuGXOPHyjMutI7 gCarHKPNa2xaKN8Q5rY86qa8G3jUEWA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714503311; a=rsa-sha256; cv=none; b=Fy0ps9Vc2/9beobFUnFdV9xIjtatycPwqp5yT8fmC/PnfGNGp2TgF5nA4t8N+F4MmByAzz IGMrwFpKsKfYBHJbPnReMV17BjwmxXE58QFiyhKn+8RAf3UgVC7iWh8Cele8ZFnqzcEs9F dz7Jdl/cn78DCaWBFwkPPCEM5/HCEOE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=FZ9gMJn6; dmarc=none; spf=pass (imf12.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.49 as permitted sender) smtp.mailfrom=axboe@kernel.dk Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-7d9c78d7f97so50738939f.3 for ; Tue, 30 Apr 2024 11:55:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1714503310; x=1715108110; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=SHR3RGRuxJEH/dBiKYgawHqY9Lnlphnx76oS7XwiA+c=; b=FZ9gMJn6bLJdaKWRDMDXqf/ctlyNBVBOiVWMQrOLBlgrdfW2t8RBeizpEzZUF6pwzt IfM8Bc8KwT7LVAAOILOF4ivg0uLtnXV/H40YC2o6qA+XmsgmG/LnxUEJ6ofhYobu2w68 kHvRV7+UyfhjuVRjVJMjcH7WJF4xHfRbD6pESVyKvWHGwkyd+bLq9LbzmVnjXCvzreOL wNsgnt9gKBPits7PLZiq/j4NFLzj/KVn/K0c0PygusAqo9yKqDZ/CG/z7+q6qWIuPwr8 NOqAHbG8W0ISVZlu+Kv1HJflxCQIgUc8Uz8O3JxijvJoHXy4NsMpNkeZpwk1fW7bNfUb tgxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714503310; x=1715108110; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SHR3RGRuxJEH/dBiKYgawHqY9Lnlphnx76oS7XwiA+c=; b=UobvPuM6IzNgnvms+1YzPzCQdPYhv5lxuL8iw2tW83mcEX7cwiQ4mJtaOaz8M4RM8+ zeHMVfpYYDPuH0tX8mFvnwZLWUupQMfGql+sa+3pFQXwidccbbgLykkHgqM76IyZJwi8 ooDiV9GnpqxWgK33LL/NiCbhFtDMkQ9OQ1xUJEO3NWkev0rLkbydoLl7ZAuC2taXZdHB 1xJdNK3nlSw3mBk5NiE/9tXF0ktxxcZQ7iyHI53FizDN2RPjK+kQy93uAFUCmalCVU8H Cmy9nUPH5+rH9sm+E/8iyJNiEQV0XZmzoTii8viCof3YArQCkyp1LeK/Z02tUATZgYGx XNXw== X-Forwarded-Encrypted: i=1; AJvYcCVm9LFzq3JSz9nQH840+ncBbUU1T2n6hzsGerGfBymu8w0uG5JxAerz02CJzZEipOKWYEmCsCUp257gmQNGoG/R1Rc= X-Gm-Message-State: AOJu0Yzzb0U3vuUhtrZ0ITAFV1i7cKPVo5SUx1OOCwes2Wrxz8dWLRgG QpkFYCGwiOP3pdUm8rlhfQXeg7MnNVxcDz9UUZJMVrHOzahUj0zll4xnXHEw/YE= X-Google-Smtp-Source: AGHT+IFtf+wSV3MsQDY8kH1lbfF76eo7FuUDr/3x1d33gkfYgxzUOO8lZB4OHFgKKuLwCBSXBvT5jw== X-Received: by 2002:a6b:f110:0:b0:7de:e04b:42c3 with SMTP id e16-20020a6bf110000000b007dee04b42c3mr828170iog.0.1714503309571; Tue, 30 Apr 2024 11:55:09 -0700 (PDT) Received: from [192.168.1.116] ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v21-20020a05663812d500b00487f9ec16b9sm389802jas.173.2024.04.30.11.55.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 30 Apr 2024 11:55:08 -0700 (PDT) Message-ID: <11f52113-7b67-4b45-ba1d-29b070050cec@kernel.dk> Date: Tue, 30 Apr 2024 12:55:05 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH net-next v8 07/14] page_pool: devmem support To: Mina Almasry Cc: David Wei , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Amritha Nambiar , Maciej Fijalkowski , Alexander Mikhalitsyn , Kaiyuan Zhang , Christian Brauner , Simon Horman , David Howells , Florian Westphal , Yunsheng Lin , Kuniyuki Iwashima , Arseniy Krasnov , Aleksander Lobakin , Michael Lass , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Richard Gobert , Sridhar Samudrala , Xuan Zhuo , Johannes Berg , Abel Wu , Breno Leitao , Pavel Begunkov , Jason Gunthorpe , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , linux-mm@kvack.org, Matthew Wilcox References: <20240403002053.2376017-1-almasrymina@google.com> <20240403002053.2376017-8-almasrymina@google.com> <8357256a-f0e9-4640-8fec-23341fc607db@davidwei.uk> Content-Language: en-US From: Jens Axboe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: E287F40020 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: px7uo4mcpp4c1re1c5e6ngqpqxnsegob X-HE-Tag: 1714503310-997802 X-HE-Meta: U2FsdGVkX1+05aS3aJb0o0m0SYUtfystK13cITZdGazkWzGD4p/1FS4s+ZCGjiTgSCnVLjFDqzmxcM/nMMR9v9CSrKkJJ5fUXxidrrSd/63focMOnk6ZdlaHg+A0UxyhqyQP5uOKEyNQh+Nr96rZRZzF332atKmvdYSh9JLfAMaCgqWRLMq2MRMHi7SUPlST9kYOVS+E3fsia2LuL9iQb5LaPCSaF50cYKqGratUifmCQA+y6iX2dc0vd+AavMTAa8+CkkQM+gRSyaY54yK09yAk/PCHEOgOITkIP2ic7hZCBYELRpRtod2ZrdlGaedBwJMaqc13m89cEmdnAkcD34+ULdhsdkHxjaajAGAQxiljfFn6/B6IjYxH+sKJcZsZHRDOVRjTcs1nCQ6khWUZGiXbBoaaEhW2iRX1nYDoMn2secONoP6rzJnIm53dmH4+dTaRvnC6fqDtA341J2AA7dCSNkH5FOuTH8fShEM5sLRoP2G4hIpyHnvXbfXAXSP856ydi8cycTSnbtwYjTmiIK7OGOWSjYd41zyz8AVEfjZ0xXfZ1kXeZeWMYBQbXdhdiawjj+wYVIz3q1iAIjbTk0jZgvBFben1xiW4MiZQYTXv/TkHTTVslY9xQhnorOSapg7CVH7NfwA7UB3Zobl2d7C5aPha03zsQ3gUWycI1RCJJ7jQLnmlujoQt++faTEJNXfTUSYEuFWGDqUfGtOMia89gmyhs6+qPZxa1+cSGW2LIpffCl1aKm3tTDbT/81NOPkaRWYBLWri3L2ETaXjO2EA4YdNdWe+mMuWD3bmuYNhtuutKxWAdgLqCva+JGXvG8vtrEK/Z5O4JOUF92lI/YwvPg9dXHD4VOoDrdZ5SVSSeFQw9m9RnzO0SwY75YNQ7rmgdKqCKvNndWz3yVI3KO06vzEUvPnWn+qBW96Sz3Z0nPLUbOxM4xTt3aoeHER2GmHBKTxR+v+AFOf/XlJ dypkrC7G wjv5k01ZhUOkM+1ss5dj1uykdE2LWlkITZEfruslLMDnQN7Bmb54VMV73rNV+r5aW1WXGDtTX0wHzC3jk5AKJGg/yOX1k01sAtCY53aNNpPurxIj1MfrR/qJHwFqSxSKVPiUUtXi+xB1i0B2FtdomZdHkbQSNA/rkSeBqXoaa3zgAxO9g13q6aGjSo8oh1KRy7y4A+UTZ7CK6Wd3X/W0LycfivCnPnaKwISJvkRR7UZHmdG+RnJk+svoSh1+ramSTFHyAfBk+W3DFr+ITHruOOhGEdN9XTypvlxvhVuKLMHRfwOnlRV82QkrV8+1Mn+JhMUscpBeBAvQYdvaUwxw0CBB58riKu872DSwoR7w7ji35hOcGz53uuVCFLHQz17M0XXJt0Ld4KTjULzCHJTKkIWrbOGcfxOoK+nCvOzoa8+k+Odnri/zCkrdCIrWUKfTV+6mpsQSG2DwRnlgeZPbk1+z/IliA8IJsZyvOy05MJPriQkVxpX9tkQ8APekjDto/NZthLGKCCBlbnPRzjU8Arfg1vfJMCF9PnGvB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/30/24 12:29 PM, Mina Almasry wrote: > On Tue, Apr 30, 2024 at 6:46?AM Jens Axboe wrote: >> >> On 4/26/24 8:11 PM, Mina Almasry wrote: >>> On Fri, Apr 26, 2024 at 5:18?PM David Wei wrote: >>>> >>>> On 2024-04-02 5:20 pm, Mina Almasry wrote: >>>>> @@ -69,20 +106,26 @@ net_iov_binding(const struct net_iov *niov) >>>>> */ >>>>> typedef unsigned long __bitwise netmem_ref; >>>>> >>>>> +static inline bool netmem_is_net_iov(const netmem_ref netmem) >>>>> +{ >>>>> +#if defined(CONFIG_PAGE_POOL) && defined(CONFIG_DMA_SHARED_BUFFER) >>>> >>>> I am guessing you added this to try and speed up the fast path? It's >>>> overly restrictive for us since we do not need dmabuf necessarily. I >>>> spent a bit too much time wondering why things aren't working only to >>>> find this :( >>> >>> My apologies, I'll try to put the changelog somewhere prominent, or >>> notify you when I do something that I think breaks you. >>> >>> Yes, this is a by-product of a discussion with regards to the >>> page_pool benchmark regressions due to adding devmem. There is some >>> background on why this was added and the impact on the >>> bench_page_pool_simple tests in the cover letter. >>> >>> For you, I imagine you want to change this to something like: >>> >>> #if defined(CONFIG_PAGE_POOL) >>> #if defined(CONFIG_DMA_SHARED_BUFFER) || defined(CONFIG_IOURING) >>> >>> or something like that, right? Not sure if this is something I should >>> do here or if something more appropriate to be in the patches you >>> apply on top. >> >> In general, attempting to hide overhead behind config options is always >> a losing proposition. It merely serves to say "look, if these things >> aren't enabled, the overhead isn't there", while distros blindly enable >> pretty much everything and then you're back where you started. >> > > The history there is that this check adds 1 cycle regression to the > page_pool fast path benchmark. The regression last I measured is 8->9 > cycles, so in % wise it's a quite significant 12.5% (more details in > the cover letter[1]). I doubt I can do much better than that to be > honest. I'm all for cycle counting, and do it myself too, but is that even measurable in anything that isn't a super targeted microbenchmark? Or even in that? > There was a desire not to pay this overhead in setups that will likely > not care about devmem, like embedded devices maybe, or setups without > GPUs. Adding a CONFIG check here seemed like very low hanging fruit, > but yes it just hides the overhead in some configs, not really removes > it. > > There was a discussion about adding this entire netmem/devmem work > under a new CONFIG. There was pushback particularly from Willem that > at the end of the day what is enabled on most distros is what matters > and we added code churn and CONFIG churn for little value. > > If there is significant pushback to the CONFIG check I can remove it. > I don't feel like it's critical, it just mirco-optimizes some setups > that doesn't really care about this work area. That is true, but in practice it'll be enabled anyway. Seems like it's not really worth it in this scenario. -- Jens Axboe