From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52D0F1B373E for ; Fri, 16 Aug 2024 15:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723821936; cv=none; b=oENECmgEaf5IHqwtJ3HgqjeN4GpY6LV2M/gD23aGRFbTUoOvE6pc7uFq10FqFyQiu/Chm6tYZlRCTU5v9wAeKyJ3bAuEmD9VraOez1zQ76I0HehiiBcp+sMBXV7TORJkjs+nbpyAD73TsCdl1qXBkc4yez36+S6YMQMfnVKnn9o= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723821936; c=relaxed/simple; bh=aBS448puOCLfZfdq/o3Me/oBWgsTW6EOTQ++9jhQTyI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jpNW4a1/V+xLOQ2JqHX3Upe1tcmJ9ijv4Hp/UxwMJlzYj3hLmwX5tTUww0bKhMUykTLllTk8ef4wZDuWdyOoL97BvBV+jiiNydZTsPMfACV/MWMxrmxkAfNydYxQXjyyRVM5n0LLCJt33dvCqMvPfxfsQZrnxvgQ/pMn3IAsOfw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=KipCaYZq; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="KipCaYZq" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-429c4a4c6a8so15669705e9.0 for ; Fri, 16 Aug 2024 08:25:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1723821932; x=1724426732; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=cTUJbqayHrtVOvzwKnfAPewIumbIhGdXnQQAZNKKC+s=; b=KipCaYZqK1SLPdNuB4qm3cu6GvOFIvJNNTWxln8kWwg4u7/SoPhiZlUStlpOUATdpv nYf9sOAVAlblEkqJPbYjjS7nxx5ZKVLhYygAT4xnXb/JYXN1C49ttmhVMMi6JjulgREQ Y2RBL9GOd3Y2saVZguY/O4K1593wYfJSuM5pQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723821932; x=1724426732; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cTUJbqayHrtVOvzwKnfAPewIumbIhGdXnQQAZNKKC+s=; b=h+BHsLsy8SBadxSnK26DJVWNmwshN8YqW3IZ8xrXnpR8SOkU9MB/AqpwcSNG8HEDMP pP+yBlXM9d4VdRmeN+/ZyUL86msHvTcna/VzmkgKrmOofCq8yGXQ45SWnLk8RoZu4aP8 w63sQzZqp+sw2s5pu6lhN1mMDC9jfwL5PfoqjP6KIxuEKmNZccTizup/rEuWywcD/iqB G3BZjnaFijGonVyAMiXmVSpdUX4sGDe+kt4dJmrsSWSurKKjcAgMIq9WvNpXwqUtWvVm SqVoqtyY0OH+34nh9QRnM54pJnzYhfbnmMQLHVg+Ww9RaGe9FZwU+pBfvlgSUpKTd0rw DqVw== X-Forwarded-Encrypted: i=1; AJvYcCXFK+yE5iZIk+NPbwc/9MqMFJfHUosLgxc5ajXR6ONxS7if1owK2PLB+KGznHfXobPyvBAnTP4gW8m0haSMNlhTuYmIF6YMwE+a X-Gm-Message-State: AOJu0Yx8ibfUjq7AqgPCeLUtUSoSVHpQAG3bhHbKE2cx+ldSs7Qw0tDk fKKNLadKWQ9N/gvADC9LFE1mN46JXwaOKqgVr208hb44iND9rFrtAFozefeBA9I= X-Google-Smtp-Source: AGHT+IHq47iCJYMwKCbcJghfwhSS/voGTaFRKxK9fCt6bzpworCDB98kJXJSD/II9rrHQu0kxbDLYA== X-Received: by 2002:a05:600c:4ecb:b0:427:ffa4:32d0 with SMTP id 5b1f17b1804b1-429ed7e4309mr20380495e9.28.1723821932332; Fri, 16 Aug 2024 08:25:32 -0700 (PDT) Received: from LQ3V64L9R2 (default-46-102-197-122.interdsl.co.uk. [46.102.197.122]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed65072asm25072915e9.13.2024.08.16.08.25.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Aug 2024 08:25:32 -0700 (PDT) Date: Fri, 16 Aug 2024 16:25:30 +0100 From: Joe Damato To: Willem de Bruijn Cc: Martin Karsten , Samiullah Khawaja , Stanislav Fomichev , netdev@vger.kernel.org, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, Alexander Lobakin , Alexander Viro , Breno Leitao , Christian Brauner , Daniel Borkmann , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jan Kara , Jiri Pirko , Johannes Berg , Jonathan Corbet , "open list:DOCUMENTATION" , "open list:FILESYSTEMS (VFS and infrastructure)" , open list , Lorenzo Bianconi , Paolo Abeni , Sebastian Andrzej Siewior Subject: Re: [RFC net-next 0/5] Suspend IRQs during preferred busy poll Message-ID: Mail-Followup-To: Joe Damato , Willem de Bruijn , Martin Karsten , Samiullah Khawaja , Stanislav Fomichev , netdev@vger.kernel.org, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, Alexander Lobakin , Alexander Viro , Breno Leitao , Christian Brauner , Daniel Borkmann , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Jan Kara , Jiri Pirko , Johannes Berg , Jonathan Corbet , "open list:DOCUMENTATION" , "open list:FILESYSTEMS (VFS and infrastructure)" , open list , Lorenzo Bianconi , Paolo Abeni , Sebastian Andrzej Siewior References: <5e52b556-fe49-4fe0-8bd3-543b3afd89fa@uwaterloo.ca> <6f40b6df-4452-48f6-b552-0eceaa1f0bbc@uwaterloo.ca> <66bf61d4ed578_17ec4b294ba@willemb.c.googlers.com.notmuch> <66bf696788234_180e2829481@willemb.c.googlers.com.notmuch> Precedence: bulk X-Mailing-List: linux-doc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <66bf696788234_180e2829481@willemb.c.googlers.com.notmuch> On Fri, Aug 16, 2024 at 10:59:51AM -0400, Willem de Bruijn wrote: > Willem de Bruijn wrote: > > Martin Karsten wrote: > > > On 2024-08-14 15:53, Samiullah Khawaja wrote: > > > > On Tue, Aug 13, 2024 at 6:19 AM Martin Karsten wrote: > > > >> > > > >> On 2024-08-13 00:07, Stanislav Fomichev wrote: > > > >>> On 08/12, Martin Karsten wrote: > > > >>>> On 2024-08-12 21:54, Stanislav Fomichev wrote: > > > >>>>> On 08/12, Martin Karsten wrote: > > > >>>>>> On 2024-08-12 19:03, Stanislav Fomichev wrote: > > > >>>>>>> On 08/12, Martin Karsten wrote: > > > >>>>>>>> On 2024-08-12 16:19, Stanislav Fomichev wrote: > > > >>>>>>>>> On 08/12, Joe Damato wrote: > > > >>>>>>>>>> Greetings: > > > > > > [snip] > > > > > > >>>>>> Note that napi_suspend_irqs/napi_resume_irqs is needed even for the sake of > > > >>>>>> an individual queue or application to make sure that IRQ suspension is > > > >>>>>> enabled/disabled right away when the state of the system changes from busy > > > >>>>>> to idle and back. > > > >>>>> > > > >>>>> Can we not handle everything in napi_busy_loop? If we can mark some napi > > > >>>>> contexts as "explicitly polled by userspace with a larger defer timeout", > > > >>>>> we should be able to do better compared to current NAPI_F_PREFER_BUSY_POLL > > > >>>>> which is more like "this particular napi_poll call is user busy polling". > > > >>>> > > > >>>> Then either the application needs to be polling all the time (wasting cpu > > > >>>> cycles) or latencies will be determined by the timeout. > > > > But if I understand correctly, this means that if the application > > > > thread that is supposed > > > > to do napi busy polling gets busy doing work on the new data/events in > > > > userspace, napi polling > > > > will not be done until the suspend_timeout triggers? Do you dispatch > > > > work to a separate worker > > > > threads, in userspace, from the thread that is doing epoll_wait? > > > > > > Yes, napi polling is suspended while the application is busy between > > > epoll_wait calls. That's where the benefits are coming from. > > > > > > The consequences depend on the nature of the application and overall > > > preferences for the system. If there's a "dominant" application for a > > > number of queues and cores, the resulting latency for other background > > > applications using the same queues might not be a problem at all. > > > > > > One other simple mitigation is limiting the number of events that each > > > epoll_wait call accepts. Note that this batch size also determines the > > > worst-case latency for the application in question, so there is a > > > natural incentive to keep it limited. > > > > > > A more complex application design, like you suggest, might also be an > > > option. > > > > > > >>>> Only when switching back and forth between polling and interrupts is it > > > >>>> possible to get low latencies across a large spectrum of offered loads > > > >>>> without burning cpu cycles at 100%. > > > >>> > > > >>> Ah, I see what you're saying, yes, you're right. In this case ignore my comment > > > >>> about ep_suspend_napi_irqs/napi_resume_irqs. > > > >> > > > >> Thanks for probing and double-checking everything! Feedback is important > > > >> for us to properly document our proposal. > > > >> > > > >>> Let's see how other people feel about per-dev irq_suspend_timeout. Properly > > > >>> disabling napi during busy polling is super useful, but it would still > > > >>> be nice to plumb irq_suspend_timeout via epoll context or have it set on > > > >>> a per-napi basis imho. > > > > I agree, this would allow each napi queue to tune itself based on > > > > heuristics. But I think > > > > doing it through epoll independent interface makes more sense as Stan > > > > suggested earlier. > > > > > > The question is whether to add a useful mechanism (one sysfs parameter > > > and a few lines of code) that is optional, but with demonstrable and > > > significant performance/efficiency improvements for an important class > > > of applications - or wait for an uncertain future? > > > > The issue is that this one little change can never be removed, as it > > becomes ABI. > > > > Let's get the right API from the start. > > > > Not sure that a global variable, or sysfs as API, is the right one. > > Sorry per-device, not global. > > My main concern is that it adds yet another user tunable integer, for > which the right value is not obvious. This is a feature for advanced users just like SO_INCOMING_NAPI_ID and countless other features. The value may not be obvious, but guidance (in the form of documentation) can be provided. > If the only goal is to safely reenable interrupts when the application > stops calling epoll_wait, does this have to be user tunable? > > Can it be either a single good enough constant, or derived from > another tunable, like busypoll_read. I believe you meant busy_read here, is that right? At any rate: - I don't think a single constant is appropriate, just as it wasn't appropriate for the existing mechanism (napi_defer_hard_irqs/gro_flush_timeout), and - Deriving the value from a pre-existing parameter to preserve the ABI, like busy_read, makes using this more confusing for users and complicates the API significantly. I agree we should get the API right from the start; that's why we've submit this as an RFC ;) We are happy to take suggestions from the community, but, IMHO, re-using an existing parameter for a different purpose only in certain circumstances (if I understand your suggestions) is a much worse choice than adding a new tunable that clearly states its intended singular purpose. - Joe