From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Gleixner Subject: Re: [RFD] Managed interrupt affinities [ Was: mlx5 broken affinity ] Date: Mon, 13 Nov 2017 21:51:05 +0100 (CET) Message-ID: References: <2187e555-2c4e-ef55-1c3a-17f5af54d762@fb.com> <573060a9-19a7-9133-ef52-fa947088dabb@fb.com> <3af0c164-8dde-b6f0-45e1-edbbb28e7f73@mellanox.com> <83d3944f-8a31-eb31-93db-294906630b0e@grimberg.me> <556f3ff5-c1d4-25c6-7bfc-9866c0d9b323@fb.com> <9384acdc-a5d8-872c-0034-9a3869f4fc8b@grimberg.me> <1d2e9304-089a-a769-9f38-a742dc066baf@grimberg.me> <67a1157e-06e7-479c-993e-bdf42fd613c6@fb.com> <69a46009-184f-d925-289c-6036f0bf2554@grimberg.me> <7f7aab77-b0b6-f8fa-0b57-3e3c1755eeaa@fb.com> <4888ac89-22d5-0700-daa1-d604e4c54970@fb.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Cc: Jens Axboe , Jes Sorensen , Tariq Toukan , Saeed Mahameed , Networking , Leon Romanovsky , Saeed Mahameed , Kernel Team , Christoph Hellwig To: Sagi Grimberg Return-path: Received: from Galois.linutronix.de ([146.0.238.70]:35066 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755221AbdKMUvN (ORCPT ); Mon, 13 Nov 2017 15:51:13 -0500 In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: On Mon, 13 Nov 2017, Sagi Grimberg wrote: > > 3) Affinity override in managed mode > > > > Doable, but there are a couple of things to think about: > > I think that it will be good to shoot for (3). Given that there are > driver requirements I'd say that driver will expose up front if it can > handle it, and if not we fallback to (1). > > > * How is this enabled? > > > > - Opt-in by driver > > > > - Extra sysfs/procfs knob > > > > We definitely should not enable it per default because that would > > surprise users/drivers which work with the current managed devices and > > rely on the affinity files to be non writeable in managed mode. > > Do you know if any exist? Would it make sense to have a survey to > understand if anyone relies on it? > > From what I've seen so far, drivers that were converted simply worked > with the non-managed facility and didn't have any special code for it. > Perhaps Christoph can comment as he convert most of them. > > But if there aren't any drivers that absolutely rely on it, maybe its > not a bad idea to allow it by default? Sure, I was just cautious and I have to admit that I have no insight into the driver side details. > > * When and how is the driver informed about the change? > > > > When: > > > > #1 Before the core tries to move the interrupt so it can veto the > > move if it cannot allocate new resources or whatever is required > > to operate after the move. > > What would the core do if a driver veto a move? Return the error code from write_affinity() as it does with any other error which fails to set the affinity. > I'm wandering in what conditions a driver will be unable to allocate > resources for move to cpu X but able to allocate for move to cpu Y. Node affine memory allocation is the only thing which comes to my mind, or some decision not to have a gazillion of queues on a single CPU. > This looks like it can work to me, but I'm probably not familiar enough > to see the full picture here. On the interrupt core side this is workable, I just need the input from the driver^Wsubsystem side if this can be implemented sanely. Thanks, tglx