From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03244C43381 for ; Tue, 5 Mar 2019 22:11:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8B65520643 for ; Tue, 5 Mar 2019 22:11:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="v8C/H6VA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726886AbfCEWLy (ORCPT ); Tue, 5 Mar 2019 17:11:54 -0500 Received: from mail-eopbgr150070.outbound.protection.outlook.com ([40.107.15.70]:48910 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726367AbfCEWLx (ORCPT ); Tue, 5 Mar 2019 17:11:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gczcRB5NJrrNB97Z+7hX0/XT5FZtYo4yVjrtBCKaYq8=; b=v8C/H6VAixd9IfCP/HmZgIMgl3WX0ncAmoUAkmK6m/rKN2q061CunTzCKJuyvleHyLhDA3cbfvwGDRVqSWUpT+1ie6BRyZvoNKD1/XtQLXFbYVc/lbdadngLfuAMLB9QBuBnAy2IY9gkA0ywsAiFzrX612EuTD8QELfyszDlfVU= Received: from DBBPR05MB6570.eurprd05.prod.outlook.com (20.179.44.81) by DBBPR05MB6473.eurprd05.prod.outlook.com (20.179.42.215) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.19; Tue, 5 Mar 2019 22:11:48 +0000 Received: from DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f]) by DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f%2]) with mapi id 15.20.1665.020; Tue, 5 Mar 2019 22:11:48 +0000 From: Jason Gunthorpe To: Jakub Kicinski CC: Jiri Pirko , "davem@davemloft.net" , "oss-drivers@netronome.com" , "netdev@vger.kernel.org" , Parav Pandit Subject: Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Index: AQHUzplETPTCP2diS0WZgKtanQm+oKXz97IAgAe1/ACAAJORAIAAB4OAgAALboCAAU9yAA== Date: Tue, 5 Mar 2019 22:11:48 +0000 Message-ID: <20190305221144.GA1758@mellanox.com> References: <20190226182436.23811-1-jakub.kicinski@netronome.com> <20190226182436.23811-5-jakub.kicinski@netronome.com> <20190227123753.GB2240@nanopsycho> <20190227103000.6ea6f7c0@cakuba.netronome.com> <20190304161510.GO8627@mellanox.com> <20190304170320.10e40255@cakuba.netronome.com> <20190305013013.GK8627@mellanox.com> <20190304181107.1379e358@cakuba.netronome.com> In-Reply-To: <20190304181107.1379e358@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: QB1PR01CA0030.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:2d::43) To DBBPR05MB6570.eurprd05.prod.outlook.com (2603:10a6:10:d1::17) x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [24.137.65.181] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 94e09d02-7554-48cf-3038-08d6a1b78feb x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:DBBPR05MB6473; x-ms-traffictypediagnostic: DBBPR05MB6473: x-microsoft-exchange-diagnostics: =?us-ascii?Q?1;DBBPR05MB6473;23:xb38mcYzy+u4MxZw3UBGtm8IxHW47oAybBe1c5q3z?= =?us-ascii?Q?lRcJ890q3B/hNDFEO8YaHV3CLJyIF9PC1p5n2fkbxTJRlL/HMTUZxkNf83YQ?= =?us-ascii?Q?lYE1ZiUMxUHvXknFW8ysmJRv84DYVgor41Ll7/zu39XyPUCQdz9ECCfKtBFV?= =?us-ascii?Q?E71Wvxwm4Bg7XGKBWXnHghLZZWTET4mjo6UaBvdJncnwXZqeZaOdv1m+QDn9?= =?us-ascii?Q?t2heB/N485QdtndfrFfIkQVerSJmrejUTW1bR8BLCcUiu4FIuK4rPE1Mvzfl?= =?us-ascii?Q?kTGu7vnPZFmRk6dvlgv9j7r/r40zrEW8b094KeTSDxrWvcS9DXnUCYF6oboq?= =?us-ascii?Q?hdwGH3HGvfOSH9po90gd+xaTTcgWKzvn81JayfmcEvnqozUUj+LEtTJPCstB?= =?us-ascii?Q?9XP6tVQmT7/TXnd2d1CMurfOcq9M7Cr32GZG8H1RJyYvWVFSKIKkB7F6fpyI?= =?us-ascii?Q?reGKMrn7qXLe7rhpbS6Tm/q0wbqAoGew9YZKIxuPSTVakZQcs/L8Pyy2sqRm?= =?us-ascii?Q?wlt3xBkvLswd2M//qBI1KF2xva8IVfjFpxMMhDHB/46l6z+XcSan30zuVMUK?= =?us-ascii?Q?BW/Uqa/hYrmray814XPytDKs5k8G/+hcsQBqKQ7NRsfYuLW3NU9UZN9AXJWC?= =?us-ascii?Q?sxOXVb/dFqIRClkATfHgNcuTlPvyMvR16ff5TJ7Oaz5aFNAph4seiEtqrDub?= =?us-ascii?Q?po2wUpZ+l2lW5KLHuvXg8jnMGStkSpjUw0okPY319WH5bJlmBnihjpGRB/dg?= =?us-ascii?Q?71dYgvRNUj1lxBpdnm7Pe3BAbtgMYv0RlAaOfdxJht7g2ih5EWbPixznyMXG?= =?us-ascii?Q?fIc/OBaJQh/x81oKe7wN3qrXCCcIWaC1jnEz/g+YqWWJFQG2c8fwTxE4BZzH?= =?us-ascii?Q?7KucpchLecspXSDnw1RFa0a1D1P2AFJVoJ09jor+rrUxUSLtA1U/q13eDsw+?= =?us-ascii?Q?6ORT+GxAaCUGayQwViz42MWfj6pxYx1qW3ICDBtP5XpvBkMn1haGxloAZ7qR?= =?us-ascii?Q?zKoGfV14V7MoJQjajLIrgrcmSKNM61YcecS/b4jpErWWDjZ94/Xr2TIBoMmo?= =?us-ascii?Q?rDGAR7WTrL1lkj+a4OpzreywbB6+oD7OzUbnXJhsgO3v29bZrHlHl9WMsCf5?= =?us-ascii?Q?Hiyrvj7TnNDWMrlpwi/P9Pijhu8qqHMYYtroKQXyDwgZairShceCR6oB2JWB?= =?us-ascii?Q?uB8TVFYSNUdFfXZcbqi26oRdmDx4Wkr/lmmbEXrQ9qa1QH5fmEjsQR4lIAji?= =?us-ascii?Q?uINPjLwuRFIm6f/vqoT3rkh2rOn7tEiewf1w+v8VCl++tSWyxPXDF9AybNH1?= =?us-ascii?Q?KOiTn3+GM0RfltM+4sy1u9Qic4GpuBXK9VS/dAwIGpi?= x-microsoft-antispam-prvs: x-forefront-prvs: 0967749BC1 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(396003)(136003)(39860400002)(366004)(376002)(346002)(199004)(189003)(486006)(6916009)(256004)(229853002)(6436002)(4744005)(3846002)(2906002)(6116002)(107886003)(81166006)(81156014)(4326008)(316002)(8676002)(25786009)(386003)(6506007)(76176011)(52116002)(102836004)(6486002)(99286004)(14454004)(26005)(54906003)(6246003)(6512007)(53936002)(71190400001)(71200400001)(86362001)(6346003)(478600001)(33656002)(7736002)(305945005)(8936002)(97736004)(66066001)(36756003)(93886005)(446003)(1076003)(106356001)(105586002)(186003)(5660300002)(11346002)(2616005)(476003)(68736007);DIR:OUT;SFP:1101;SCL:1;SRVR:DBBPR05MB6473;H:DBBPR05MB6570.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: D8o/XJLk72h00/sWNNYR4Ui0G2KjyOY2r64YdDVkDKk1z5U8v6HjAdzFAgy8drZfReZslsb9AM6MAli/OQsurHhazdFjTnEFNc6OlvqItONVOzrqh8Q5tQAGvL8bjxcQYp9TH7JIbsPl1HgcW+IyhsMqgNOAeWeHaRqXNlHdXcAh3YDHavvblmsZs/eNQiBYngwBMBLoeWkuTNiT+rLZQ/YyQDYvCLDvrsPq3uB8Uf/MfvqkQkmKYvUD7SYGMrqmXW4XJsSHp1IJsOhubVPEzCugaJKx/UfmR+nDrdmi0EA1roqKh9bGaCSirVI7h/DJRgrDy1yftfP/PZeZZwkh0iM8tg+D4gnoo2DWywZZHfmo7kLFi1bp4j51eLGWrRxhNOHbovVu+6ArjZrm1jcPLy2ZNjruIZ/Ou33Y5STdB8E= Content-Type: text/plain; charset="us-ascii" Content-ID: <80455DBDA44AF54ABE01F2839ABEB77C@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 94e09d02-7554-48cf-3038-08d6a1b78feb X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Mar 2019 22:11:48.7858 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR05MB6473 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Mon, Mar 04, 2019 at 06:11:07PM -0800, Jakub Kicinski wrote: > > At least in RDMA we have drivers doing all combinations of this: > > multiple ports per BDF, one port per BDF, and one composite RDMA > > device formed by combining multiple BDFs worth of ports together. >=20 > Right, last but not least we have the case where there is one port but > multiple links (for NUMA, or just because 1 PCIe link can't really cope > with 200Gbps). In that case which DBDF would the port go to? :( > Do all internal info of the ASIC (health, regions, sbs) get registered > twice? This I don't know, at least for RDMA this configuration gets confusing very fast and devlink is the least of the worries.. Personally I would advocate for a master/slave kind of arrangement where the master BDF has a different PCI DID from the slaves. devlink and other kernel objects hang off the master. The slave port is then only used to carry selected NUMA aware data path traffic and doesn't show in devlink. Jason