From mboxrd@z Thu Jan  1 00:00:00 1970
From: Jens Axboe <axboe@kernel.dk>
Subject: Re: mlx5 broken affinity
Date: Wed, 8 Nov 2017 09:13:59 -0700
Message-ID: <98ba68ee-a21c-e509-d4c3-e21c2ec138f8@kernel.dk>
References: <2187e555-2c4e-ef55-1c3a-17f5af54d762@fb.com>
 <d65d6156-07da-14dc-88ff-fcb5c0ef5f83@grimberg.me>
 <f8624a01-f77a-ac0b-ba19-19c7150ca133@fb.com>
 <CALzJLG-Z2fJibyZ74XU7XCSx-g3=YapV9VyekAYZkS5JwvkwWw@mail.gmail.com>
 <573060a9-19a7-9133-ef52-fa947088dabb@fb.com>
 <3af0c164-8dde-b6f0-45e1-edbbb28e7f73@mellanox.com>
 <83d3944f-8a31-eb31-93db-294906630b0e@grimberg.me>
 <556f3ff5-c1d4-25c6-7bfc-9866c0d9b323@fb.com>
 <9384acdc-a5d8-872c-0034-9a3869f4fc8b@grimberg.me>
 <alpine.DEB.2.20.1711021907420.2824@nanos>
 <1d2e9304-089a-a769-9f38-a742dc066baf@grimberg.me>
 <alpine.DEB.2.20.1711071551330.1716@nanos>
 <ce53b7df-7072-0ff2-5922-8f3cb340c86a@grimberg.me>
 <063D6719AE5E284EB5DD2968C1650D6DD00B6353@AcuExch.aculab.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Cc: Jes Sorensen <jsorensen@fb.com>,
        Tariq Toukan <tariqt@mellanox.com>,
        Saeed Mahameed <saeedm@dev.mellanox.co.il>,
        Networking <netdev@vger.kernel.org>,
        Leon Romanovsky <leonro@mellanox.com>,
        Saeed Mahameed <saeedm@mellanox.com>,
        Kernel Team <kernel-team@fb.com>,
        Christoph Hellwig <hch@lst.de>
To: David Laight <David.Laight@ACULAB.COM>,
        'Sagi Grimberg' <sagi@grimberg.me>,
        Thomas Gleixner <tglx@linutronix.de>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mail-it0-f54.google.com ([209.85.214.54]:54883 "EHLO
        mail-it0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1752603AbdKHQOB (ORCPT
        <rfc822;netdev@vger.kernel.org>); Wed, 8 Nov 2017 11:14:01 -0500
Received: by mail-it0-f54.google.com with SMTP id 72so7096019itk.3
        for <netdev@vger.kernel.org>; Wed, 08 Nov 2017 08:14:01 -0800 (PST)
In-Reply-To: <063D6719AE5E284EB5DD2968C1650D6DD00B6353@AcuExch.aculab.com>
Content-Language: en-US
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On 11/08/2017 05:21 AM, David Laight wrote:
> From: Sagi Grimberg
>> Sent: 08 November 2017 07:28
> ...
>>> Why would you give the user a knob to destroy what you carefully optimized?
>>
>> Well, looks like someone relies on this knob, the question is if he is
>> doing something better for his workload. I don't know, its really up to
>> the user to say.
> 
> Maybe the user wants to ensure that nothing except some very specific
> processing happens on some (or most) of the cpu cores.
> 
> If the expected overall ethernet data rate isn't exceptionally large
> is there any reason to allocate a queue (etc) for every cpu.

There are numerous valid reasons to be able to set the affinity, for
both nics and block drivers. It's great that the kernel has a predefined
layout that works well, but users do need the flexibility to be able to
reconfigure affinities, to suit their needs.

For the particular mlx5 case, I'm actually not sure how the FB
configuration differs from the in-kernel stuff. I'll take a look at
that. It may just be that the configuration exists because the code used
to be driver private and frequently changed, setting it at bootup to a
known good configuration helped eliminate problems when upgrading
kernels. I also remember some cases of removing CPU0 from the mask.

But that particular case is completely orthogonal to whether or not we
should allow the user to reconfigure. The answer to that is clearly YES,
and we should ensure that this is possible.

-- 
Jens Axboe