From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A63AC433DB for ; Mon, 18 Jan 2021 12:39:49 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 81808222BB for ; Mon, 18 Jan 2021 12:39:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81808222BB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 52DBF2010E; Mon, 18 Jan 2021 12:39:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sv7zWZ6WNfQw; Mon, 18 Jan 2021 12:39:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 354282010B; Mon, 18 Jan 2021 12:39:45 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19478C088B; Mon, 18 Jan 2021 12:39:45 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0F3B5C013A for ; Mon, 18 Jan 2021 12:39:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C980185868 for ; Mon, 18 Jan 2021 12:39:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IlVwdPNwf-Gu for ; Mon, 18 Jan 2021 12:39:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by fraxinus.osuosl.org (Postfix) with ESMTPS id BEB4085816 for ; Mon, 18 Jan 2021 12:39:41 +0000 (UTC) Received: from fraeml703-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4DKB9X0kwJz67dKJ; Mon, 18 Jan 2021 20:36:32 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml703-chm.china.huawei.com (10.206.15.52) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2106.2; Mon, 18 Jan 2021 13:39:38 +0100 Received: from [10.47.8.82] (10.47.8.82) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Mon, 18 Jan 2021 12:39:37 +0000 Subject: Re: [PATCH v4 3/3] iommu/iova: Flush CPU rcache for when a depot fills To: Robin Murphy , Jean-Philippe Brucker References: <1607538189-237944-1-git-send-email-john.garry@huawei.com> <1607538189-237944-4-git-send-email-john.garry@huawei.com> <7a4f3d74-2f0d-1ffa-95cf-cfeaa81d8c7e@arm.com> From: John Garry Message-ID: <7b2b4f7c-d240-4922-4204-60cee61c8a60@huawei.com> Date: Mon, 18 Jan 2021 12:38:23 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <7a4f3d74-2f0d-1ffa-95cf-cfeaa81d8c7e@arm.com> Content-Language: en-US X-Originating-IP: [10.47.8.82] X-ClientProxiedBy: lhreml744-chm.china.huawei.com (10.201.108.194) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Cc: will@kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linuxarm@huawei.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" T24gMTUvMDEvMjAyMSAxOToyMSwgUm9iaW4gTXVycGh5IHdyb3RlOgo+Pgo+PiBJdCB3b3VsZCBi ZSBnb29kIHRvIHVuZGVyc3RhbmQgd2h5IHRoZSByY2FjaGUgZG9lc24ndCBzdGFiaWxpemUuIENv dWxkIGJlCj4+IGEgYnVnLCBvciBqdXN0IG5lZWQgc29tZSB0dW5pbmcKPj4KPj4gSW4gc3RyaWN0 IG1vZGUsIGlmIGEgZHJpdmVyIGRvZXMgQWxsb2MtRnJlZS1BbGxvYyBhbmQgdGhlIGZpcnN0IGFs bG9jCj4+IG1pc3NlcyB0aGUgcmNhY2hlLCB0aGUgc2Vjb25kIGFsbG9jYXRpb24gaGl0cyBpdC4g VGhlIHNhbWUgc2VxdWVuY2UgaW4KPj4gbm9uLXN0cmljdCBtb2RlIG1pc3NlcyB0aGUgY2FjaGUg dHdpY2UsIGJlY2F1c2UgdGhlIElPVkEgaXMgYWRkZWQgdG8gdGhlCj4+IGZsdXNoIHF1ZXVlIG9u IEZyZWUuCj4+Cj4+IFNvIHJhdGhlciB0aGFuIEFGQUZBRi4uIHdlIGdldCBBQUEuLkZGRi4uLCBv bmx5IG9uY2UgdGhlIGZxX3RpbWVyIAo+PiB0cmlnZ2Vycwo+PiBvciB0aGUgRlEgaXMgZnVsbC4K ClNvdW5kcyByaWdodAoKPj4gSW50ZXJlc3RpbmdseSB0aGUgRlEgc2l6ZSBpcyAyeCBJT1ZBX01B R19TSVpFLCBzbyB3ZQo+PiBjb3VsZCBhbGxvY2F0ZSAyIG1hZ2F6aW5lcyB3b3J0aCBvZiBmcmVz aCBJT1ZBcyBiZWZvcmUgYWxsb2Mgc3RhcnRzCj4+IGhpdHRpbmcgdGhlIGNhY2hlLiBJZiBhIGpv YiBhbGxvY2F0ZXMgbW9yZSB0aGFuIHRoYXQsIHNvbWUgbWFnYXppbmVzIGFyZQo+PiBnb2luZyB0 byB0aGUgZGVwb3QsIGFuZCB3aXRoIG11bHRpLUNQVSBqb2JzIHRob3NlIHdpbGwgZ2V0IHVzZWQg b24gb3RoZXIKPj4gQ1BVcyBkdXJpbmcgdGhlIG5leHQgYWxsb2MgYnVyc3RzLCBjYXVzaW5nIHRo ZSBwcm9ncmVzc2l2ZSBpbmNyZWFzZSBpbgo+PiByY2FjaGUgY29uc3VtcHRpb24uIEkgd29uZGVy IGlmIHNldHRpbmcgSU9WQV9NQUdfU0laRSA+IElPVkFfRlFfU0laRSAKPj4gaGVscHMKPj4gcmV1 c2Ugb2YgSU9WQXM/CgpMb29raW5nIGJhY2sgdGhyb3VnaCB0aGUgbG9yZSBoaXN0b3J5LCBJIGRv bid0IGtub3cgd2hlcmUgdGhlIApJT1ZBX0ZRX1NJWkUgPSAyNTYgY2FtZSBmcm9tLiBJIGd1ZXNz IGl0J3Mgc2l6ZSBvZiAyeCBJT1ZBX01BR19TSVpFICgxeCAKZm9yIGxvYWRlZCBhbmQgMXggZm9y IHByZXYpIGZvciB0aGUgcmVhc29uIHlvdSBtZW50aW9uLgoKPj4KPj4gVGhlbiBhZ2FpbiBJIGhh dmVuJ3Qgd29ya2VkIG91dCB0aGUgZGV0YWlscywgbWlnaHQgYmUgZW50aXJlbHkgd3JvbmcuIAo+ PiBJJ2xsCj4+IGhhdmUgYW5vdGhlciBsb29rIG5leHQgd2Vlay4KPiAKCmNoZWVycwoKPiBJIGRp ZCBzdGFydCBkaWdnaW5nIGludG8gdGhlIGRhdGEgKHRoYW5rcyBmb3IgdGhhdCEpIGJlZm9yZSBD aHJpc3RtYXMsIAo+IGJ1dCBiZXR3ZWVuIGJlaW5nIGdlbmVyYWxseSBmcmF6emxlZCBhbmQgdHJ5 aW5nIHRvIHJlbWVtYmVyIGhvdyB0byB3cml0ZSAKPiBQZXJsIHRvIG1hc3NhZ2UgdGhlIG51bWJl cnMgb3V0IG9mIHRoZSBsb2cgZHVtcCBJIG5ldmVyIGdvdCByb3VuZCB0byAKPiByZXNwb25kaW5n LCBzb3JyeS4KCkFzIHlvdSBtYXkgaGF2ZSBzZWVuOgpodHRwczovL3Jhdy5naXRodWJ1c2VyY29u dGVudC5jb20vaGlzaWxpY29uL2tlcm5lbC1kZXYvMDY0YzRkYzg4NjliM2YyYWQwN2VkZmZjZWFm ZGUwYjEyOWYyNzZiMC9sc2kzMDA4X2RtZXNnCgpJIGhhZCB0byBjaGFuZ2Ugc29tZSBibG9jayBj b25maWdzIHZpYSBzeXNmcyB0byBldmVyIGdldCBJT1ZBIGxvY2F0aW9ucyAKZm9yIHNpemUgPiAw LiBBbmQgZXZlbiB0aGVuLCBJIHN0aWxsIGdvdCBub25lIGJpZ2dlciB0aGFuIApJT1ZBX1JBTkdF X0NBQ0hFX01BWF9TSVpFLgoKTm90ZTogRm9yIGEgbG9nIGxpa2U6ClsxMzE3NS4zNjE5MTVdIHBy aW50X2lvdmEyIGlvdmFfYWxsb2NzKD01MDAwMDAwIC4uLiB0b29fYmlnPTQ3MDM2Cgo0NzAzNiBp cyBudW1iZXIgb2YgSU9WQSBzaXplID4gSU9WQV9SQU5HRV9DQUNIRV9NQVhfU0laRSwgaW4gY2Fz ZSBpdCB3YXMgCm5vdCBjbGVhci4KCkFuZCBJIG5ldmVyIGhpdCB0aGUgY3JpdGljYWwgcG9pbnQg b2YgYSBkZXBvdCBiaW4gZmlsbGluZywgYnV0IGl0IG1heSAKanVzdCB0YWtlIGV2ZW4gbG9uZ2Vy LgoKSG93ZXZlciB3aXRoIElPVkEgc2l6ZSA9IDAgYWx3YXlzIG9jY3VycmluZywgdGhlbiBJIG5v dGljZWQgdGhhdCB0aGUgCmRlcG90IHNpemUgPSAwIGJpbiBmaWxscyB1cCByZWxhdGl2ZWx5IHF1 aWNrbHkuIEFzIHN1Y2gsIEkgYW0gbm93IApzbGlnaHRseSBza2VwdGljYWwgb2YgdGhlIGFwcHJv YWNoIEkgaGF2ZSB0YWtlbiBoZXJlLCBpLmUgcHVyZ2UgdGhlIAp3aG9sZSByY2FjaGUuCgo+IAo+ IFRoZSBwYXJ0aWFsIHRob3VnaHRzIHRoYXQgSSBjYW4gcmVjYWxsIHJpZ2h0IG5vdyBhcmUgZmly c3RseSB0aGF0IHRoZSAKPiB0b3RhbCBudW1iZXJzIG9mIElPVkFzIGFyZSBhY3R1YWxseSBwcmV0 dHkgbWVhbmluZ2xlc3MsIGl0IHJlYWxseSBuZWVkcyAKPiB0byBiZSBicm9rZW4gZG93biBieSBz aXplICh0aGF0J3Mgd2hlcmUgbXkgUGVybC1oYWNraW5nIHN0YWxsZWQuLi4pOyAKPiBzZWNvbmRs eSB0aGF0IHRoZSBwYXR0ZXJuIGlzIGZhciBtb3JlIHRoYW4ganVzdCBhIHN0ZWFkeSBpbmNyZWFz ZSAtIHRoZSAKPiBDUFUgcmNhY2hlIGNvdW50IGxvb2tzIHRvIGJlIGhlYWRpbmcgYXN5bXB0b3Rp Y2FsbHkgdG93YXJkcyB+NjVLIElPVkFzIAo+IGFsbCB0aGUgdGltZSwgcmVwcmVzZW50aW5nIChJ SVJDKSB0d28gc2l6ZXMgYmVpbmcgaW4gaGVhdnkgcm90YXRpb24sIAo+IHdoaWxlIHRoZSBkZXBv dCBpcyBoYXBwaWx5IHRpY2tpbmcgYWxvbmcgaW4gYSBzdGVhZHkgc3RhdGUgYXMgZXhwZWN0ZWQs IAo+IHVudGlsIGl0IHN1ZGRlbmx5IGV4cGxvZGVzIG91dCBvZiBub3doZXJlOyB0aGlyZGx5LCBJ J2QgcmVhbGx5IGxpa2UgdG8gCj4gc2VlIGluc3RydW1lbnRhdGlvbiBvZiB0aGUgZmx1c2ggcXVl dWVzIGF0IHRoZSBzYW1lIHRpbWUsIHNpbmNlIEkgdGhpbmsgCj4gdGhleSdyZSB0aGUgcmVhbCBj dWxwcml0Lgo+IAo+IE15IHRoZW9yeSBzbyBmYXIgaXMgdGhhdCBldmVyeW9uZSBpcyBjYWxsaW5n IHF1ZXVlX2lvdmEoKSBmcmVxdWVudGx5IAo+IGVub3VnaCB0byBrZWVwIHRoZSB0aW1lciBhdCBi YXkgYW5kIHRoZWlyIG93biBxdWV1ZXMgZHJhaW5lZC4gVGhlbiBhdCAKPiB0aGUgfjE2SCBtYXJr LCAqc29tZXRoaW5nKiBoYXBwZW5zIHRoYXQgcGF1c2VzIHVubWFwcyBsb25nIGVub3VnaCBmb3Ig Cj4gdGhlIHRpbWVyIHRvIGZpcmUsIGFuZCBhdCB0aGF0IHBvaW50IGFsbCBoZWxsIGJyZWFrcyBs b29zZS4KClNvIGRvIHlvdSB0aGluayB0aGF0IHRoZSBmcmVlaW5nIHRoZSBJT1ZBIG1hZ2F6aW5l cyB3aGVuIHRoZSBkZXBvdCBmaWxscyAKaXMgdGhlIGNhdXNlIG9mIHRoaXM/IFRoYXQgd2FzIG91 ciBhbmFseXNpcy4KCj4gVGhlIGRlcG90IGlzIAo+IHN1ZGRlbmx5IGZsb29kZWQgd2l0aCBJT1ZB cyBvZiAqYWxsKiBzaXplcywgaW5kaWNhdGl2ZSBvZiBhbGwgdGhlIHF1ZXVlcyAKPiBiZWluZyBm bHVzaGVkIGF0IG9uY2UgKG5vdGUgdGhhdCB0aGUgdHdvIG1vc3QgY29tbW9uIHNpemVzIGhhdmUg YmVlbiAKPiBob3ZlcmluZyBwZXJpbG91c2x5IGNsb3NlIHRvICJmdWxsIiB0aGUgd2hvbGUgdGlt ZSksIGJ1dCB0aGVuLCAKPiBjcnVjaWFsbHksICp0aGF0IGtlZXBzIGhhcHBlbmluZyouIE15IGd1 ZXNzIGlzIHRoYXQgdGhlIGxvYWQgb2YgCj4gZnFfZmx1c2hfdGltZW91dCgpIHNsb3dzIHRoaW5n cyBkb3duIGVub3VnaCB0aGF0IHRoZSB0aGUgdGltZXIgdGhlbiAKPiBrZWVwcyBnZXR0aW5nIHRo ZSBjaGFuY2UgdG8gZXhwaXJlIGFuZCByZXBlYXQgdGhlIHNpdHVhdGlvbi4KCk5vdCBzdXJlIG9u IHRoYXQgb25lLgoKPiAKPiBUaGUgbWFpbiBjb25jbHVzaW9uIEkgZHJhdyBmcm9tIHRoaXMgaXMg dGhlIHNhbWUgb25lIHRoYXQgd2FzIG15IGluaXRpYWwgCj4gZ3V0IGZlZWxpbmc7IHRoYXQgTUFY X0dMT0JBTF9NQUdTID0gMzIgaXMgdXR0ZXIgYm9sbG9ja3MuCgpZZWFoLCBJIHRlbmQgdG8gYWdy ZWUgd2l0aCB0aGF0LiBPciwgbW9yZSBzcGVjaWZpY2FsbHksIGhvdyB0aGluZ3Mgd29yayAKdG9k YXkgaXMgYnJva2VuLCBhbmQgTUFYX0dMT0JBTF9NQUdTID0gMzIgaXMgdmVyeSBtdWNoIGludm9s dmVkIHdpdGggdGhhdC4KCj4gVGhlIENQVSByY2FjaGUgCj4gY2FwYWNpdHkgc2NhbGVzIHdpdGgg dGhlIG51bWJlciBvZiBDUFVzOyB0aGUgZmx1c2ggcXVldWUgY2FwYWNpdHkgc2NhbGVzIAo+IHdp dGggdGhlIG51bWJlciBvZiBDUFVzOyBpdCBpcyBub25zZW5zaWNhbCB0aGF0IHRoZSBkZXBvdCBz aXplIGRvZXMgbm90IAo+IGNvcnJlc3BvbmRpbmdseSBzY2FsZSB3aXRoIHRoZSBudW1iZXIgb2Yg Q1BVcyAoSSBub3RlIHRoYXQgdGhlIHRlc3RpbmcgCj4gb24gdGhlIG9yaWdpbmFsIHBhdGNoc2V0 IGNpdGVzIGEgMTYtQ1BVIHN5c3RlbSwgd2hlcmUgdGhhdCBkZXBvdCAKPiBjYXBhY2l0eSBpcyBj b252ZW5pZW50bHkgZXF1YWwgdG8gdGhlIHRvdGFsIHJjYWNoZSBjYXBhY2l0eSkuCj4gCj4gTm93 IHllcywgcHVyZ2luZyB0aGUgcmNhY2hlcyB3aGVuIHRoZSBkZXBvdCBpcyBmdWxsIGRvZXMgaW5k ZWVkIGhlbHAgCj4gbWl0aWdhdGUgdGhpcyBzY2VuYXJpbyAtIEkgYXNzdW1lIGl0IHByb3ZpZGVz IGVub3VnaCBvZiBhIGJ1ZmZlciB3aGVyZSAKPiB0aGUgcmVndWxhciBmcmVlX2lvdmFfZmFzdCgp IGNhbGxzIGRvbid0IGhpdCBxdWV1ZV9pb3ZhKCkgZm9yIGEgd2hpbGUgCj4gKGFuZCBnaXZlcyBm cV9yaW5nX2ZyZWUoKSBzb21lIHJlcHJpZXZlIG9uIHRoZSBDUFUgaGFuZGxpbmcgdGhlIAo+IHRp bWVvdXQpLCBnaXZpbmcgZW5vdWdoIGxlZXdheSBmb3IgdGhlIGZsb29kIHRvIGZpbmlzaCBiZWZv cmUgYW55b25lIAo+IHN0YXJ0cyBoaXR0aW5nIHF1ZXVlcy9sb2Nrcy9ldGMuIGFuZCBzdGFsbGlu ZyBhZ2FpbiwgYW5kIHRodXMgYnJlYWsgdGhlIAo+IHNlbGYtcGVycGV0dWF0aW5nIHRpbWVvdXQg Y3ljbGUuIEJ1dCB0aGF0J3Mgc3RpbGwgb25seSBhIGRhbWFnZSAKPiBsaW1pdGF0aW9uIGV4ZXJj aXNlISBJdCdzIHBsYW5uaW5nIGZvciBmYWlsdXJlIHRvIGp1c3QgbGllIGRvd24gYW5kIAo+IGFz c3VtZSB0aGF0IHRoZSBkZXBvdCBpcyBnb2luZyB0byBiZSBmdWxsIGlmIGZxX2ZsdXNoX3RpbWVv dXQoKSBldmVyIAo+IGZpcmVzIGJlY2F1c2UgaXQncyBzb21ldGhpbmcgbGlrZSBhbiBvcmRlciBv ZiBtYWduaXR1ZGUgc21hbGxlciB0aGFuIHRoZSAKPiBmbHVzaCBxdWV1ZSBjYXBhY2l0eSAoZXZl biBmb3IgYSB1bmlmb3JtIGRpc3RyaWJ1dGlvbiBvZiBJT1ZBIHNpemVzKSBvbiAKPiBzdXBlci1s YXJnZSBzeXN0ZW1zLgo+IAo+IEknbSBob25lc3RseSB0ZW1wdGVkIHRvIG1vdmUgbXkgcG9zaXRp b24gZnVydGhlciB0b3dhcmRzIGEgaGFyZCBOQUsgb24gCj4gdGhpcyBhcHByb2FjaCwgYmVjYXVz ZSBhbGwgdGhlIGV2aWRlbmNlIHNvIGZhciBwb2ludHMgdG8gaXQgYmVpbmcgYSAKPiBib2RnZSBh cm91bmQgYSBjbGVhciBhbmQgZWFzaWx5LWZpeGVkIHNjYWxhYmlsaXR5IG92ZXJzaWdodC4gQXQg dGhlIHZlcnkgCj4gbGVhc3QgSSdkIG5vdyB3YW50IHRvIGhlYXIgYSByZWFzb25lZCBqdXN0aWZp Y2F0aW9uIGZvciB3aHkgeW91IHdhbnQgdG8gCj4ga2VlcCB0aGUgZGVwb3QgYXQgYW4gYXJiaXRy YXJ5IGZpeGVkIHNpemUgd2hpbGUgdGhlIHdob2xlIHJlc3Qgb2YgdGhlIAo+IHN5c3RlbSBzY2Fs ZXMgCgoKPihJJ20gYXNzdW1pbmcgdGhhdCBzaW5jZSBteSBwcmV2aW91cyBzdWdnZXN0aW9uIHRv IHRyeSAKPiBjaGFuZ2VzIGluIHRoYXQgYXJlYSBzZWVtcyB0byBoYXZlIGJlZW4gaWdub3JlZCku Cj4gCgpTbyBJIHNhaWQgdGhhdCBpdCBzaG91bGQgZml4IHRoZSBwcm9ibGVtIG9mIHRoZSB0aHJv dWdocHV0IGdvaW5nIHRocm91Z2ggCnRoZSBmbG9vciBhdCB0aGlzIDE2aCBtYXJrLgoKQnV0IHdl IHNlZSAyeCB0aWdodGx5IGNvdXBsZWQgcHJvYmxlbXM6CmEuIGxlYWRpbmcgdXAgdG8gdGhlIH4x NkggY3JpdGljYWwgcG9pbnQsIHRocm91Z2hwdXQgaXMgc2xvd2x5IGRlZ3JhZGluZyAKYW5kIGJl Y29tZXMgcXVpdGUgdW5zdGFibGUgKG5vdCBzaG93biBpbiB0aGUgbG9nKQpGb3IgdGhlIExTSTMw MDggY2FyZCwgd2UgZG9uJ3Qgc2VlIHRoYXQuIEJ1dCB0aGVuIG5vIElPVkEgc2l6ZSA+IApJT1ZB X1JBTkdFX0NBQ0hFX01BWF9TSVpFIG9jY3VyIHRoZXJlLgoKYi4gYXQgdGhlIGNyaXRpY2FsIHBv aW50LCB0aHJvdWdocHV0IGdvZXMgdGhyb3VnaCB0aGUgZmxvb3IKClNvIGIuIHNob3VsZCBiZSBm aXhlZCB3aXRoIHRoZSBzdWdnZXN0aW9uIHRvIGhhdmUgdW5saW1pdGVkL2hpZ2hlciBkZXBvdCAK bWF4IGJpbiBzaXplLCBidXQgSSByZWNrb24gdGhhdCB3ZSB3b3VsZCBzdGlsbCBzZWUgYS4gQW5k IEkgcHV0IHRoYXQgCmRvd24gdG8gdGhlIGZhY3QgdGhhdCB3ZSBoYXZlIElPVkEgc2l6ZXMgPiBJ T1ZBX1JBTkdFX0NBQ0hFX01BWF9TSVpFIGF0IAphIGNlcnRhaW4gcmF0ZSBhbHdheXMuIEFzIHRo ZSByYiB0cmVlIGdyb3dzIG92ZXIgdGltZSwgdGhleSBiZWNvbWUgCnNsb3dlciBhbmQgc2xvd2Vy IHRvIGFsbG9jK2ZyZWUgLSB0aGF0J3Mgb3VyIHRoZW9yeS4gQWxsb3dpbmcgdGhlIGRlcG90IAp0 byBncm93IGZ1cnRoZXIgaXNu4oCZdCBnb2luZyB0byBoZWxwIHRoYXQuCgpNYXliZSBMZWl6aGVu J3MgaWRlYSB0byB0cmltIHRoZSByY2FjaGUgcGVyaW9kaWNhbGx5IGlzIG92ZXJhbGwgYmV0dGVy LCAKYnV0IEkgYW0gY29uY2VybmVkIG9uIGltcGxlbWVudGF0aW9uLgoKSWYgbm90LCB0aGVuIGlm IHdlIGFsbG93IGRlcG90IGJpbiBzaXplIHRvIHNjYWxlL2dyb3csIEkgd291bGQgbGlrZSB0byAK c2VlIG1vcmUgZWZmaWNpZW50IGhhbmRsaW5nIGZvciBJT1ZBIHNpemUgPiBJT1ZBX1JBTkdFX0NB Q0hFX01BWF9TSVpFLgoKVGhhbmtzLApKb2huCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCmlvbW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51eC1m b3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9s aXN0aW5mby9pb21tdQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DC99C433E0 for ; Mon, 18 Jan 2021 12:46:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DBDEC2231F for ; Mon, 18 Jan 2021 12:46:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391825AbhARMpk (ORCPT ); Mon, 18 Jan 2021 07:45:40 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]:2364 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391747AbhARMkX (ORCPT ); Mon, 18 Jan 2021 07:40:23 -0500 Received: from fraeml703-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4DKB9X0kwJz67dKJ; Mon, 18 Jan 2021 20:36:32 +0800 (CST) Received: from lhreml724-chm.china.huawei.com (10.201.108.75) by fraeml703-chm.china.huawei.com (10.206.15.52) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2106.2; Mon, 18 Jan 2021 13:39:38 +0100 Received: from [10.47.8.82] (10.47.8.82) by lhreml724-chm.china.huawei.com (10.201.108.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Mon, 18 Jan 2021 12:39:37 +0000 Subject: Re: [PATCH v4 3/3] iommu/iova: Flush CPU rcache for when a depot fills To: Robin Murphy , Jean-Philippe Brucker CC: , , , , References: <1607538189-237944-1-git-send-email-john.garry@huawei.com> <1607538189-237944-4-git-send-email-john.garry@huawei.com> <7a4f3d74-2f0d-1ffa-95cf-cfeaa81d8c7e@arm.com> From: John Garry Message-ID: <7b2b4f7c-d240-4922-4204-60cee61c8a60@huawei.com> Date: Mon, 18 Jan 2021 12:38:23 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <7a4f3d74-2f0d-1ffa-95cf-cfeaa81d8c7e@arm.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [10.47.8.82] X-ClientProxiedBy: lhreml744-chm.china.huawei.com (10.201.108.194) To lhreml724-chm.china.huawei.com (10.201.108.75) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 15/01/2021 19:21, Robin Murphy wrote: >> >> It would be good to understand why the rcache doesn't stabilize. Could be >> a bug, or just need some tuning >> >> In strict mode, if a driver does Alloc-Free-Alloc and the first alloc >> misses the rcache, the second allocation hits it. The same sequence in >> non-strict mode misses the cache twice, because the IOVA is added to the >> flush queue on Free. >> >> So rather than AFAFAF.. we get AAA..FFF.., only once the fq_timer >> triggers >> or the FQ is full. Sounds right >> Interestingly the FQ size is 2x IOVA_MAG_SIZE, so we >> could allocate 2 magazines worth of fresh IOVAs before alloc starts >> hitting the cache. If a job allocates more than that, some magazines are >> going to the depot, and with multi-CPU jobs those will get used on other >> CPUs during the next alloc bursts, causing the progressive increase in >> rcache consumption. I wonder if setting IOVA_MAG_SIZE > IOVA_FQ_SIZE >> helps >> reuse of IOVAs? Looking back through the lore history, I don't know where the IOVA_FQ_SIZE = 256 came from. I guess it's size of 2x IOVA_MAG_SIZE (1x for loaded and 1x for prev) for the reason you mention. >> >> Then again I haven't worked out the details, might be entirely wrong. >> I'll >> have another look next week. > cheers > I did start digging into the data (thanks for that!) before Christmas, > but between being generally frazzled and trying to remember how to write > Perl to massage the numbers out of the log dump I never got round to > responding, sorry. As you may have seen: https://raw.githubusercontent.com/hisilicon/kernel-dev/064c4dc8869b3f2ad07edffceafde0b129f276b0/lsi3008_dmesg I had to change some block configs via sysfs to ever get IOVA locations for size > 0. And even then, I still got none bigger than IOVA_RANGE_CACHE_MAX_SIZE. Note: For a log like: [13175.361915] print_iova2 iova_allocs(=5000000 ... too_big=47036 47036 is number of IOVA size > IOVA_RANGE_CACHE_MAX_SIZE, in case it was not clear. And I never hit the critical point of a depot bin filling, but it may just take even longer. However with IOVA size = 0 always occurring, then I noticed that the depot size = 0 bin fills up relatively quickly. As such, I am now slightly skeptical of the approach I have taken here, i.e purge the whole rcache. > > The partial thoughts that I can recall right now are firstly that the > total numbers of IOVAs are actually pretty meaningless, it really needs > to be broken down by size (that's where my Perl-hacking stalled...); > secondly that the pattern is far more than just a steady increase - the > CPU rcache count looks to be heading asymptotically towards ~65K IOVAs > all the time, representing (IIRC) two sizes being in heavy rotation, > while the depot is happily ticking along in a steady state as expected, > until it suddenly explodes out of nowhere; thirdly, I'd really like to > see instrumentation of the flush queues at the same time, since I think > they're the real culprit. > > My theory so far is that everyone is calling queue_iova() frequently > enough to keep the timer at bay and their own queues drained. Then at > the ~16H mark, *something* happens that pauses unmaps long enough for > the timer to fire, and at that point all hell breaks loose. So do you think that the freeing the IOVA magazines when the depot fills is the cause of this? That was our analysis. > The depot is > suddenly flooded with IOVAs of *all* sizes, indicative of all the queues > being flushed at once (note that the two most common sizes have been > hovering perilously close to "full" the whole time), but then, > crucially, *that keeps happening*. My guess is that the load of > fq_flush_timeout() slows things down enough that the the timer then > keeps getting the chance to expire and repeat the situation. Not sure on that one. > > The main conclusion I draw from this is the same one that was my initial > gut feeling; that MAX_GLOBAL_MAGS = 32 is utter bollocks. Yeah, I tend to agree with that. Or, more specifically, how things work today is broken, and MAX_GLOBAL_MAGS = 32 is very much involved with that. > The CPU rcache > capacity scales with the number of CPUs; the flush queue capacity scales > with the number of CPUs; it is nonsensical that the depot size does not > correspondingly scale with the number of CPUs (I note that the testing > on the original patchset cites a 16-CPU system, where that depot > capacity is conveniently equal to the total rcache capacity). > > Now yes, purging the rcaches when the depot is full does indeed help > mitigate this scenario - I assume it provides enough of a buffer where > the regular free_iova_fast() calls don't hit queue_iova() for a while > (and gives fq_ring_free() some reprieve on the CPU handling the > timeout), giving enough leeway for the flood to finish before anyone > starts hitting queues/locks/etc. and stalling again, and thus break the > self-perpetuating timeout cycle. But that's still only a damage > limitation exercise! It's planning for failure to just lie down and > assume that the depot is going to be full if fq_flush_timeout() ever > fires because it's something like an order of magnitude smaller than the > flush queue capacity (even for a uniform distribution of IOVA sizes) on > super-large systems. > > I'm honestly tempted to move my position further towards a hard NAK on > this approach, because all the evidence so far points to it being a > bodge around a clear and easily-fixed scalability oversight. At the very > least I'd now want to hear a reasoned justification for why you want to > keep the depot at an arbitrary fixed size while the whole rest of the > system scales >(I'm assuming that since my previous suggestion to try > changes in that area seems to have been ignored). > So I said that it should fix the problem of the throughput going through the floor at this 16h mark. But we see 2x tightly coupled problems: a. leading up to the ~16H critical point, throughput is slowly degrading and becomes quite unstable (not shown in the log) For the LSI3008 card, we don't see that. But then no IOVA size > IOVA_RANGE_CACHE_MAX_SIZE occur there. b. at the critical point, throughput goes through the floor So b. should be fixed with the suggestion to have unlimited/higher depot max bin size, but I reckon that we would still see a. And I put that down to the fact that we have IOVA sizes > IOVA_RANGE_CACHE_MAX_SIZE at a certain rate always. As the rb tree grows over time, they become slower and slower to alloc+free - that's our theory. Allowing the depot to grow further isn’t going to help that. Maybe Leizhen's idea to trim the rcache periodically is overall better, but I am concerned on implementation. If not, then if we allow depot bin size to scale/grow, I would like to see more efficient handling for IOVA size > IOVA_RANGE_CACHE_MAX_SIZE. Thanks, John