From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 998BEC43381 for ; Tue, 5 Mar 2019 01:31:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 505A2206DD for ; Tue, 5 Mar 2019 01:31:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="NS7CheR/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726250AbfCEBbE (ORCPT ); Mon, 4 Mar 2019 20:31:04 -0500 Received: from mail-eopbgr40043.outbound.protection.outlook.com ([40.107.4.43]:37122 "EHLO EUR03-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726095AbfCEBbD (ORCPT ); Mon, 4 Mar 2019 20:31:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=civpnI7B8Owlh2dVtYc+uYSxhoib2r57cHsKl9ORUQQ=; b=NS7CheR/35tupX05FC0QzER5B4/pIc/tc2mmr9BEvXkU6Oin0K6an1PmQHsRw57JLrvz0Kw10HcOiB5Xoim63bO8bUDyeXjqr8N7Xx9+bKw8KWQ/UDWyXh7MZRSgyQdjMuWCbb0iUe/SejK2S+q7DEANdPYeq4AZsPCGcMg4Yds= Received: from DBBPR05MB6570.eurprd05.prod.outlook.com (20.179.44.81) by DBBPR05MB6283.eurprd05.prod.outlook.com (20.179.40.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.18; Tue, 5 Mar 2019 01:30:20 +0000 Received: from DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f]) by DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f%2]) with mapi id 15.20.1665.020; Tue, 5 Mar 2019 01:30:20 +0000 From: Jason Gunthorpe To: Jakub Kicinski CC: Jiri Pirko , "davem@davemloft.net" , "oss-drivers@netronome.com" , "netdev@vger.kernel.org" , Parav Pandit Subject: Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Index: AQHUzplETPTCP2diS0WZgKtanQm+oKXz97IAgAe1/ACAAJORAIAAB4OA Date: Tue, 5 Mar 2019 01:30:19 +0000 Message-ID: <20190305013013.GK8627@mellanox.com> References: <20190226182436.23811-1-jakub.kicinski@netronome.com> <20190226182436.23811-5-jakub.kicinski@netronome.com> <20190227123753.GB2240@nanopsycho> <20190227103000.6ea6f7c0@cakuba.netronome.com> <20190304161510.GO8627@mellanox.com> <20190304170320.10e40255@cakuba.netronome.com> In-Reply-To: <20190304170320.10e40255@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: QB1PR01CA0025.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:2d::38) To DBBPR05MB6570.eurprd05.prod.outlook.com (2603:10a6:10:d1::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [24.137.65.181] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d05cd5ca-18db-42a3-41fb-08d6a10a209a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:DBBPR05MB6283; x-ms-traffictypediagnostic: DBBPR05MB6283: x-microsoft-exchange-diagnostics: =?us-ascii?Q?1;DBBPR05MB6283;23:esi5G40sM5K+rohGdI5gamabI/4fSX5da25XPyYs2?= =?us-ascii?Q?U6H8v7E570Q37Rm4BiHrxIPcU2UZYMcGf0rcG3xtIhyZ4SZ5gIEy7rPGQ8w4?= =?us-ascii?Q?GWAoJXceYlt7I1JQ/2quijGcdpqqMWsWCJKHKO9eNnyZAOV2ppY1b41ebYPA?= =?us-ascii?Q?OlMehChI2tV6fEkFvx/qRQxoDHetyott0sePiiivPPZ0iQ1LDoryHpzMKunp?= =?us-ascii?Q?dtBazmMvlWKRM0Q9qGj47pgmafTBECikCgTWzAUdeSsKgRunPHgpimuUOGVz?= =?us-ascii?Q?oKLeBUiGUokGEZ8VU6+QT7w7pgQtaBsBMJiw4GrmKczIv+YzWHe2BH1WoeO4?= =?us-ascii?Q?T1dC4FHdzuRdizOjQHApKR9KeY8SPvqJmYoxi10RzlydvpXzqxK6afkti0rD?= =?us-ascii?Q?/uALVPvhAS9xy/or+O2p+XuQ3kQ3qNMSJa2haL8SP8eMu7yDUh/cOuJXMMXd?= =?us-ascii?Q?g74JBK2o8nz9qWj0kvujrdp9zg7WSeBXxOhkAoi/7qZF1cfr8E3yhqoXuEmG?= =?us-ascii?Q?P8WlP+Lzfz/5lfMdI/JCF596IZGTnXBselpSngKhYpBA/ja3pwcrhaSXc50m?= =?us-ascii?Q?h9x/Dq+YSANxtOkyP38f77u6WDuxF2uMtDTd9ISMria/cXywIwkNlUoB6fJ+?= =?us-ascii?Q?E0DpuXKjopIvpoWtG8/9g2DlgXbJEunfZk2Ex26/25U6Wx18tiAQMXbdp/eM?= =?us-ascii?Q?/qApGQNf3RgLWqZ1wmJ9cqo6AF2QWeX3nLZYdLz5IC+RfYZkIDAiduGInInd?= =?us-ascii?Q?QZhQpX5pMGC1z+/QjZjULHD5/iQtjCBYg6seVyaV4ihMg7GQ1MOF1AiRKJ7H?= =?us-ascii?Q?O/4JLS6ZgKATQnlnf4qLEXbLu2CgLVImQQtawctzHW1IT6C0BCAw2bptPQAy?= =?us-ascii?Q?RHVGPPz2853xqtRrwN8vQLkGCCin1T6w/TIAaKsKo6hFxZLkwJdcZFvgrKao?= =?us-ascii?Q?57q/n4csVsBxUjG5MH1FMasRAXxt/b5IkRLyXeeBiWn6hEOSP01WhIE0vHm+?= =?us-ascii?Q?Brq624NIPKDgxRGSY+tA3SkA9GxGxYYsuIcFn9Z9Ar7TB5RpdXWfkKLKnkUl?= =?us-ascii?Q?e0oN+/4n8OEkeYOGk1ohYAxnxHFdyrLJtCzrVNyMNi/9x0gftGs/ZfRgZEof?= =?us-ascii?Q?frd3YS9G1B5C48fGnlfG3NQ4oVc9VDpwjuGY6z1ybB54KaUGHIEDpUlGgiWQ?= =?us-ascii?Q?MSxGipS72HDKaXy7rHlXuA9jEVOfcH46W5VuNXdVmPFwbygEPpc3eqcjl9jD?= =?us-ascii?Q?w10pV/btZu5yiC+O603WGpJbfFUl0UKpLtOXucwlt3vV/pjjd/J4HFbao6fP?= =?us-ascii?B?QT09?= x-microsoft-antispam-prvs: x-forefront-prvs: 0967749BC1 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(346002)(136003)(376002)(396003)(366004)(39860400002)(189003)(199004)(1076003)(486006)(71190400001)(5660300002)(36756003)(105586002)(106356001)(4326008)(2616005)(11346002)(446003)(52116002)(3846002)(68736007)(478600001)(97736004)(6116002)(71200400001)(14454004)(229853002)(33656002)(76176011)(86362001)(26005)(93886005)(8676002)(54906003)(99286004)(6512007)(25786009)(8936002)(81166006)(81156014)(186003)(386003)(6506007)(6246003)(66066001)(6486002)(256004)(316002)(6916009)(6436002)(107886003)(305945005)(102836004)(7736002)(476003)(2906002)(53936002);DIR:OUT;SFP:1101;SCL:1;SRVR:DBBPR05MB6283;H:DBBPR05MB6570.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: +z7v1EyByZnoAJoHpiy80JU10GA8RwgzlHQAUk3WJRFyfPC6ymJoF+/TZbZn7ba6Hrkdg4zIDi1MIkPdIUDD8Ttw0Kx7pfqXAhfvZ4uIvIUILYWzgR80R+gsLx2Lg3zSGoP7BvKS+GxgoqNzBIhYiKH47KC7JCaf0ZlOr1rto9gd8MdqMkbtwtNqElg41MdsJe/aaNYAEZz8751gmsHXANd4JTfrXZ39XmyDGTO2rsFl8NPgPhYfj2VMMy2Rh4wBYHWWyUR5KRZl9lukC7U+vySzZLNX/j+VEsbm48kwvkdvgLp2BzbubHrvjBR3R8AjkPp7bsGaDjIIAKpTF7RQSt9/+iI7qhRBBva52RxWpuPtm9gXmBWBObTbErxYw/U0Tt3KCn83qs8+Dbyp9ZHMyuNVgj1fKkGYWDq78hEH/v8= Content-Type: text/plain; charset="us-ascii" Content-ID: <3DA3B4BBE04D1F4697C3515168AB47D3@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d05cd5ca-18db-42a3-41fb-08d6a10a209a X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Mar 2019 01:30:19.7145 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR05MB6283 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Mon, Mar 04, 2019 at 05:03:20PM -0800, Jakub Kicinski wrote: > > Don't we already have devlink instances for every mlx5 physical port > > and VF as they are unique PCI functions? >=20 > That's a very NIC-centric view of the world, though. Equating devlink > instances to ports, and further to PCI devices. Its fundamentally > different from what switches and some NICs do, where all ports are under > single devlink instance. I think, as a practical matter, it is a bit hard to recombine an asic that presents multiple PCI BDFs into a single SW object. It is tricky to give stable labels to things, to leave gaps to allow for uncertain discovey, to co-ordinate between multiple struct pci_device drivers probe functions, etc. And at least with devlink, if you have a object layer that is broader then PCI BDF, how do the devlink commands work? Are all BDFs just an alias for this hidden super object? Do any drivers attempt to provide single instant made up of merged BDFs? In other words, is a PCI BDF really the largest granularity that devlink can address today? At least in RDMA we have drivers doing all combinations of this: multiple ports per BDF, one port per BDF, and one composite RDMA device formed by combining multiple BDFs worth of ports together. > > > You guys come from the RDMA side of the world, with which I'm less > > > familiar, and the soft bus + spawning devices seems to be a popular > > > design there. Could you describe the advantages of that model for=20 > > > the sake of the netdev-only folks? :) =20 > >=20 > > I don't think we do this in RDMA at all yet, or maybe I'm not sure > > what you are thinking of? >=20 > Mm.. I caught an Intel patch set recently which was talking about buses > and spawning devices. It must have been a different kettle of fish. That sounds like scalable iov.. Jason