From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751620AbdB1Sy4 (ORCPT ); Tue, 28 Feb 2017 13:54:56 -0500 Received: from esa2.hgst.iphmx.com ([68.232.143.124]:8991 "EHLO esa2.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751531AbdB1Sy2 (ORCPT ); Tue, 28 Feb 2017 13:54:28 -0500 X-IronPort-AV: E=Sophos;i="5.35,220,1483977600"; d="scan'208";a="93946551" Authentication-Results: spf=pass (sender IP is 74.221.232.55) smtp.mailfrom=sandisk.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac1c2134-817ff70000005688-db-58b5aa47ab39 From: Bart Van Assche To: "sebott@linux.vnet.ibm.com" CC: "gerald.schaefer@de.ibm.com" , "linux-kernel@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "dledford@redhat.com" Subject: Re: IB on s390 broken with commit 99db94940 "IB/core: Remove ib_device.dma_device" Thread-Topic: IB on s390 broken with commit 99db94940 "IB/core: Remove ib_device.dma_device" Thread-Index: AQHSkeK7VrCHEozeekuDa17C7qd8lA== Date: Tue, 28 Feb 2017 16:50:13 +0000 Message-ID: <1488300599.2370.5.camel@sandisk.com> References: <1488233058.2597.1.camel@sandisk.com> In-Reply-To: Accept-Language: nl-NL, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.28.1.254] Content-Type: text/plain; charset="iso-8859-1" Content-ID: <2C8ED9D82ADF5644BB1BDD313A99A1EA@sandisk.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrPIsWRmVeSWpSXmKPExsWyRobxn67nqq0RBn07eC1env/AarHzzQs2 i8u75rBZPDvUy2JxquswowOrx8FHh9g8HhzazOLxft9VNo/Pm+QCWKK4bFJSczLLUov07RK4 MlZs2c9UMF2s4tT8K0wNjI8Euxg5OSQETCS+nLrE3MXIxSEksIRJYs6nPWwQziVGiY1b/rOD VLEJGEnMnrCHBcQWEbCWePHqDztIEbPAK0aJO33bWUESwgKxEvsbNrJCFMVJrNo6H6pBT+JJ 63GwOIuAqsS3iauYQGxeAUOJi++/Qq1ezihx7e4ysG2cAi4Sh2ZMADqDg4NRQFai5TU3SJhZ QFzi1pP5TBBnC0gs2XOeGcIWlXj5+B8rhK0g8XnFPzaIej2JG1OnQNlWErPafzJB2NoSyxa+ Zoa4QVDi5MwnLBMYxWYhWTELSfssJO2zkLTPQtK+gJF1FaNYcWJycW56aoGhqV5xYl5KZnG2 XnJ+7iZGcDQqmuxg7NjofohRgINRiYf3QPfWCCHWxLLiytxDjBIczEoivDuKgUK8KYmVValF +fFFpTmpxYcYpTlYlMR5l2pPjRASSE8sSc1OTS1ILYLJMnFwSjUwVqm/z5nL2MYnt2ttyM94 6Y7L33t3LEw7kDTX88nlHssnvTf63px4Whj7wsczo0t/C+PlGwlzZXmqyvdsWueo9u7NzO8v 37Womn5m4rlWcPIa42S70MgFPaGuDy6otCpfav1Q+aelxkPw0rkT+ksUYs9P/nCiSijgwM9T ey1z3G9yWKX3Naa3KLEUZyQaajEXFScCAHPVaSnCAgAA X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:74.221.232.55;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10019020)(6009001)(7916002)(39410400002)(39850400002)(39840400002)(39450400003)(39860400002)(2980300002)(438002)(24454002)(377424004)(199003)(189002)(2270400002)(6916009)(97736004)(2351001)(38730400002)(54356999)(50986999)(7736002)(305945005)(229853002)(8936002)(110136004)(103116003)(356003)(54906002)(6246003)(4326008)(76176999)(5640700003)(2950100002)(106466001)(50466002)(2900100001)(106116001)(189998001)(3846002)(102836003)(6116002)(33646002)(92566002)(86362001)(8746002)(81166006)(1730700003)(47776003)(53936002)(81156014)(8676002)(2906002)(2501003)(69596002)(5660300001)(36756003)(626004)(68736007)(23756003)(7099028);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR04MB0747;H:sacsmgep15.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BN3NAM04FT033;1:nb9mIbrDwLYDeJVOeLjidwAMN5xc60PlZtkcLG2YSp+tc2EdsYuZqHufuZn5Cs1DGUAJmoP6EWuYkMItspPce1qMUvZ/Hrz81UqIZ8viJtRFU4x8yePkuabhoTTgVCqyYGNaDpGMQbvJz57xwFue1EG1pRK6Ob5RwNy4qRSMBvm6TrsU8ZC9pWXvNZ6ZyewzoU/bZ2YOQz8F+x7Nh1QrxE0lSQsjZrRnClfZhH2QIkO5tGc0MgmyxnGUZBv+5E2XmdiOA2PM+2aSUv54SL0OJOR7/6mXM7ABgmkyxbDE9eMxLNHPnudgjbcNhx7NNaq1a6/pL93mZLdbfzf4A8ffMCThgdrossLJgmvB6oVvmWJ2+Lh4R3xg310jqeOhXoZ12Ru5lW/Fba7MTWBThOh6tPdnZc/HyKHShigPxdj9cKGl08ZwFMerm4BDmzhZBmP3+Fz+0JN2NqjSpJw+bRCdUh/+Zr5Xwk8wqzM9tQs85j++91/SIvTtZ0jz3t4KHuk4Y6ymhHFKdAyZRqWROG3zzjhhG0dtcybdWhhX8IrqCT5XuTSUUiCDAlm0YOFkx6dL X-MS-Office365-Filtering-Correlation-Id: 77efa464-8368-4bf4-7f9c-08d45ff9e02b X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(8251501002);SRVR:DM5PR04MB0747; X-Microsoft-Exchange-Diagnostics: 1;DM5PR04MB0747;3:bapcEYQoULRGBSKF8vkZqYniNLKGdaryFqOF0V0LFPFOtqUxaPZKoXRxiY+o1u2KVCk3JN/qCLMeVj0uYakpuKDgnCvrAv6dZElzxcbMPLhjE72Ht1iSXzelyYyr4x9nmX9mxssYmXB1e2kU++Q2oUIv4YZMNMpZgTBbKRDWsVDySpdHb9ueD5TQNNy/kttJQqJMSbapK3EbyMuAWl+OGTa2GyGi2hJh+F5jqLlNJ1M8h0b8lBYENaZU8aFuDHqnyq2RyRZkPklweJFulAnw3Ck5w267vqt65YH0pnEuxXGLuuUoEZHNzsYTtzfripMOihlQnJPWoy5mmG0uNorI3wl8u2EQwnstiLhWL6/z6QH3v38w+06yqGXa1y2Az7WxAet/h40TTN1mmNd3imtXSQ==;25:I2myFBB7M0EbLY/TPmu1mz9UaJVw3EHES40Wh/W84YN90n6qOs1bkTUdK60dTPgNuLAp0/sP1fv3ZOFxi+/6Or5w2CRHhz5H/rrdQ2EaBNlFOKHPPBd01Gv0KKRW7QzxiJUTu0FL4HEgSxUUJoWUFhqm5xkRomJr8clrbmderJZzaebbhYiQZKAEJqFzcbONOlh/NiKz0KUIk2xJBbvLw0QMlQd0H2s0N3jRa3yqfDm74QbzxLjp2lgaEv4JsryQbPaHyirJ3iAX4h+zit81ftVhno5AR/W+ePjCtvJrfgOe8kreR6eT5K1oEY5nHxlnRy+1CzvDyL2Z5CYRVIHmGN08w/XG1vEpBOkfeWV4e7nQ01EsM0P2BKf48SzYYdI+PxjSBxa6fCOFN5AlTL2O/TQfPGfkg4krx7RSMMzznWk7tCUrdL8jHujzlt+QZ6rAXaXjq0ea3Cl8qPtrq4KFgg== X-Microsoft-Exchange-Diagnostics: 1;DM5PR04MB0747;31:lHu4Vv97CCf6nLvln6J/DFoA+Scnk5TpZRpmS9DRihLYdk3oTykfedx++zdGCErZtLG775n+v8E0CcY5aPELNEwpQJcZBdJMArIV0mYwGVdGXcdetrMziJhysXZuB7TqcNUnegaxSIT2twH1ofNlJCFQA6B/WBfSjSmLV4cteMJi8TcEKDvzv6GQCfVjVjHfXJC+wwco9vZGqOpbk2zLkVvxTXAfpgW8X4qCUMH3l20SW4iKivbxL546zrkzO/X3zsnUO0mbdneqpB0rTBCPug==;20:GhOTjwzJtdTZwrwzcOSyfzI+dg1pHDGKmEkIBIbIevEaajQ+saRHrz14VpQvrI/RZ097b3BJNp6IgiPFhRgKSJoSEsHA24f5viaq1IWE+s4P4kH3wzP9eyNpruraeQDFmw1jJ+OmIMhCT3cvXml6pOnB/Jr6c7cu7oAGy7LXxNQgGqBlT6cIAbaiMgtYBktLz+iu8pZhidM0ncKR+HLYUR646S3LmG+depy//0+e8mPj9lLagDBrXbzvp7dqoZ42/yJpC/qW2evQTJprKkKJApX/TbO+5JjwCXCsYV+so7CrbLsfrETfbjDXtltNjOYb+sujKKncrrSka7SYAn1RemrUcoFcSjTS9zQ+EHSTFVEu8ycxW5EzV9+85mcTV61BM0ZTD/8q/U+JSPeIrzFHSj3lNl9nCNTKyBpK3ELdwZWGN0lu5V0puwmKIWnoJ0zIhohLCWrzwl4Vn/0P+fgA33erHRLhy0xYs7bvAR6S7t2LHFw9AahCvRaw5HWoEori X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(13017025)(13015025)(13018025)(13023025)(13024025)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123558025)(20161123555025)(20161123560025)(20161123562025)(20161123564025)(6072148);SRVR:DM5PR04MB0747;BCL:0;PCL:0;RULEID:;SRVR:DM5PR04MB0747; X-Microsoft-Exchange-Diagnostics: 1;DM5PR04MB0747;4:JN8dbpVP3vukfma0hm8JZjac5Lvi/ywJkcrApNoH7Cr41Eu4NJf8WxrUn1K4A+4o3cPuBW83gtu5Okk+eoc2fvq3TtyiQqzAIygRMAZGKVInxX7IKmN1vVkMlAR07y6iGTJMe3x9V8vELVefH0Rf/5IDz/GEtV3SqfWWShdRzeaZYjAErA4GQso/gULilTR/A3FqBeWGAFZPX8jGbUhPHsVOYCio8sS/V8FTzTs9mw5h0WESI9iUtm9NRSyqKjO+IinbSsW8/w8afIYvdvCJq/w+whEVjNwKZ4q/wA4zddA68Z8FpBLhndoIa7LJNsp+IeZOJJJOs7rm8mHv3eEEsY1ELIssbkLLZh8WJi3cZaWSP9zsObCXlntIbHq32U9CiLU0KksEfG6GhKAqRUm5PGp8cmb2U1xp1GoNwq3Sy6Cvpflo8Pr4WMXCs5KzFV6uhiVwLYf/CEJ4jBHhnb3R/C7aN62UlzOt/YSXoQYVWMoUOGJ+c7OzZGtQkQFkyAFZYEDXpGRhrYwbzSHYK6DUVhNyy5DfnHc893hFoHWknhe3z4i9KeR+olME17q1r4ZTEmnRka9LImtfdNJE0uy697Ggan70BUeTe3Qx/jDg5Uhc8IBcqdXdsixraiiR25+L5cFfT4pHK6Q0PeIEE01nZTKNSvftNeyu0tDEV4sTIrZ5LXGiOjd5I4myYHU7kwbTNe9nrXt1crsn3T2QgLFpUTKeTUzDkH3qfjhP4UP1P2M= X-Forefront-PRVS: 0232B30BBC X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;DM5PR04MB0747;23:ERYY3ZRgN/xBXr2TRnfIA3pZg34MB42A98ppxSm?= =?iso-8859-1?Q?N5Brm5+8/4Pg3AbfHMXWHnUCL8wcLVpRTSCbCWmJkYF2yD+LNfG43zuFr1?= =?iso-8859-1?Q?0F3MRrCH/nsh/Mf+czEROJiRMf05gPDeWLv0x/qZ6PsAbCdFZB+K2qNQau?= =?iso-8859-1?Q?XSUJpH8CmgmmWo1tvrA6PvvsO5BjwqDSex2bvPySR0E0zHhpVL3tfLsFjJ?= =?iso-8859-1?Q?UxFZl3D8XrpDo1thP1TMeQthSMhLIMr+thHIsG0pHpBEVS9BNay3LI/DXs?= =?iso-8859-1?Q?ihTFhjVLS8NhHlgru2fOmDfsRM7HrGAirV5Y4XT/HBu3+Ia2uG0gVbf5IT?= =?iso-8859-1?Q?x6QKgj9821oW3kg1CFclNECPnzF+wMECLe29JY1C4xmq9XqaKLjfzgPhgB?= =?iso-8859-1?Q?wxJzvPrlqn7ufELr71wF9lFUh5rOsEnNwZFMnxoMUcbzSLW8ns86vHANcV?= =?iso-8859-1?Q?BLIiflrweL1KaiGG5BS0ZMAxIAcxPlV1Sg+1PN5ODp5DaYhXNX2dOIJqWO?= =?iso-8859-1?Q?Sr9mHfgnofFt8Ni2jct+rHHoqGgJ5v5UnHc6MSBuYqb7TDfdwBG+PkxaFc?= =?iso-8859-1?Q?4zCjY81WB91bEO8VQofnt/vcJSFBCAALLbAE3D6Tzms4K0jkgxL6Fe6+xE?= =?iso-8859-1?Q?kigkJXFkdLfnVU5ivAa2jJeiWUxPYk07lKYxf67Bmos/tCnEpKDR8xXGMU?= =?iso-8859-1?Q?qHBqB0MwkjqMAITEym6Md5CzN0u8FtrQmbEmEHFkcB68DT6O49cHHM3+K0?= =?iso-8859-1?Q?3M1V9anGzmLon8v4R0FIxCmyNQdDK2YPgpW9Kh58a4dW3PEqF7Ts5AzWEr?= =?iso-8859-1?Q?9PuMAZgHMYnRbqsRdS9i8CILs3IbLZrV3Z53a4wljoqT5I838qs00XVrBh?= =?iso-8859-1?Q?7t9J90szv0EmUamxwLM5uuLO91TB+wNYm5n7Vd4xfoq8//9AbJTKhVwcro?= =?iso-8859-1?Q?scY5rZeZ7a+3cCiQ1Xj035TD+WoNbvq7w6x20rkJdDveqHLBUQTJRRxj+I?= =?iso-8859-1?Q?CScWY+GURi7gbwc+QrQLD8oMiF/ZyYafV95e8pAlgIPdWhZH+IxNqs/7LA?= =?iso-8859-1?Q?LNfOQhyJE0ng6oTevR2sGuDNwy8jPtZRQolqnZ1KMVcCSdVbATYHdNxCyi?= =?iso-8859-1?Q?rUJAvjKBkx0j/oTTcIES60afhM/fjAqjHZXj0pj9j8NKpRBcA0FuKAvTwG?= =?iso-8859-1?Q?TvPkxHKc6qcpigh5i7VFkrfcjK9nl6AWXkCwxwvzVxhdAzpKb+jIDEHLBY?= =?iso-8859-1?Q?ko1xOQl4TYpk1ebzci9OBbqbdIVuubPJjm/GafNXk6lGu8CwQrPd9ss3Qi?= =?iso-8859-1?Q?TLYyc9oBnvlpqzIhcshcFl8bM0ciShtoMpU89lzC1Xp4eLWPYAUOhT5vdF?= =?iso-8859-1?Q?D1fcr+v12zck+HB9VfngDLtR8UPAJdZGfhhf8tw7Bkh7aO1gBlT52J4FCR?= =?iso-8859-1?Q?J3qOXzUOdOqtRTveIGSPnxSfs5pE/xsamw/5kigwd/MCTMSkHhtRgp4mA?= =?iso-8859-1?Q?=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR04MB0747;6:aBdZLgMEGyPAbw8MfEfgvKzvpA1SgTue/Zzf7TTF6UVcbns6+KPzft/QA7K0eshcAIoKWfN/6fiTYy5BQVhqsDcQlmEPU9u+LHCNDWQfQu36VCw0SEkQhbn1UapyPXmoDsAQIa+S8ZEPRtGvOhiq/C3v6307tbUXascj2AElK9GyxzTiogQlBk500eK1++1vYEo4W7lJOnfwntzen2klnGxDxKO7KzQ3AbzxlL6aK8QoDbzvAjLu48ks8vx35V35Pe2GB6z8DeZD790lytYFP9wTB1V0wbXLA0Iyz92S4Z5cPzd8xJQSEFmgOta7OZb5H9DKm5wJ8ky52inPDpwHarPa7dMhDOqxardkVWnercO01smMWVa13LyLEW7T3o7qphELskojvpUUpwREod6Xf3iD2HeWXysN23JvzfjIanY=;5:z8BeDX7HVrz+NglnQkXEJRk6ftpMjOsOIjuQnbQgwjXPciseInm72nOmqZUwoDb2Wdf3aUkTs7ZDkVAKqjiLa3cglQO0FzSsMfnrYMzzHkbutwDRV1D6O2S7r2bsA6ruuW/rUIONhHDO32MHbvpRwW8PdgqIpx+1ehhaTC4YxeM=;24:GyZVbtqV5o9UbecwTpRlK/PM3F2Ks1TpOYrAEi2Rj1N4CySfuysbihe9U64l/3aIipt8aUxzKe3R31IzaUOp1RaasKOwriVZX2GVpdwm+1E= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR04MB0747;7:b5+9/JRWfSHfP1ROCJrG8boulOEq6srSj1u7WiNUvFsM2De7LT6D3oBFPctC5RYMqLaOSih8diLgXNRuLnaTSsdjorcdUhWRBEO6d6nBSGZXInZ8yCISUF0gUGXMd63aFfvs+V03K2LmjbL9kGyoMEzdgrqDbk2L822nw5KMl8D8hP+MZJLO7kwDxeYjrl1iG6VD6MpaJFqHeSQh6eKocc/7YCLSaco3iRyE9+nZXM1aa7/C4+V5Vv9AdHPRdrv5yrwcO8NrDNt/PAnb8E3xiyIRl1LnUT3aiDvM6ZKSx/GLmq7qZ0WKDUdN2QjOdG6Zam3TVr1haoKwl+C6p1AZrg==;20:B+4YfnlvMZP5UsXJI9+k65hWVW0MXzQXxd/GRdMHPQ+w+Fo5E5cFYv/KU7aHyClUto+iBbhXmchKH7nXaPNuZVjll3n76zHwMSuK3kEr5J73q/oTGbyiSqW7Wl1Tt1t94vPwtr8MIaaF/DkfihsQJCvzG3FY6f2G7oYchlE1YbRKgFvKRXV8BKp0dhK7nIBu7fT6IYDlUrwIowe7h2Zeh1weGqX3eZ5Ao1+RYSKUaZq6l494OTPBzsjXvIpvuEg2 X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2017 16:50:17.3212 (UTC) X-MS-Exchange-CrossTenant-Id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=b61c8803-16f3-4c35-9b17-6f65f441df86;Ip=[74.221.232.55];Helo=[sacsmgep15.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR04MB0747 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id v1SIthsf006010 On Tue, 2017-02-28 at 09:53 +0100, Sebastian Ott wrote: > On Mon, 27 Feb 2017, Bart Van Assche wrote: > > > On Mon, 2017-02-27 at 21:17 +0100, Sebastian Ott wrote: > > > commit 99db94940 "IB/core: Remove ib_device.dma_device" > > > breaks infiniband on s390 (and I think also other archs that do something > > > like to_pci_dev(dev) in one of their dma_ops callbacks). > > > > > > With this commit you use the dma_ops of the device that called > > > ib_register_device but you call e.g. dma_map with ib_device->dev > > > as an argument. > > > > > > S390's (pci specific) dma_map uses to_pci_dev(dev) to look into the > > > pci device (and its arch specific data) and oopses. > > > > > > Calling dma_map with ib_device->dev.parent would work but then it > > > wouldn't make sense to copy dma_ops and mask from ib_device->dev.parent > > > to ib_device->dev.. > > > > How about something like the untested patch below? > > It works but it doesn't feel right (why should all pci devices have this > duplicated data). > > Frankly I don't get the usecase of infiniband (sometimes) using > device->dev.dma_ops instead of parent->dma_ops. Also that these values are > selectively copied from the parent looks weird (opposed to all or nothing). > > What about reintroducing dma_device (as an infiniband internal) and set it > to &ib_device->dev if you have to and to parent in all other cases? Hello Sebastian, There are three kinds of RDMA drivers: - RDMA drivers that always use DMA for transferring data between memory and HCA (e.g. mlx4, mlx5, ...). These drivers make the ULP call the PCI DMA mapping functions directly. - RDMA drivers that never use DMA directly but use another driver for transferring data (e.g. rdma_rxe). This driver makes the ULP store virtual addresses in .dma_address. - RDMA drivers that decide whether to use PIO or DMA depending on e.g. the QP type and the amount of data to be transferred (qib, hfi1). These drivers also make the ULP store virtual addresses in .dma_address and decide internally whether or not to invoke the PCI DMA mapping functions. This is why a custom DMA mapping API was introduced in the RDMA subsystem. Until recently the Linux RDMA subsystem not only had its own DMA mapping operations but also its own template for DMA mapping operations (struct ib_dma_mapping_ops). This is not only confusing but also led to a multitude of incomplete and RDMA driver DMA mapping operations of which additionally the behavior is slightly different of other DMA mapping operations. That's why we want to evolve towards a single DMA mapping API. Reintroducing the dma_device pointer in struct ib_device would make it impossible to use the standard DMA mapping API for RDMA devices. Bart.