From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57907C7EE24 for ; Sat, 3 Jun 2023 15:58:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685807880; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=0d0LMkN2FB+K4jto4c6ztgPtbQHLiMa2QoTEGsPyags=; b=buMsWAkCXPOHAi54RW3E2opFp9KktTUiFoQhNlQRTH4wmHtHcD/bmXB7usOG0mOxDrz70Y esAHE89KOWy2MkqLDmokGz+0JaaZIyF5dXngm6IFzgZs2asPWRP/ou+9sydD8Johl4wEVf TjWLqy6Vu+pTybFF79DZDe6ki7WThmo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-314-bJxKsTOKOGKNKzqF31DAHQ-1; Sat, 03 Jun 2023 11:57:57 -0400 X-MC-Unique: bJxKsTOKOGKNKzqF31DAHQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2722129A9D28; Sat, 3 Jun 2023 15:57:55 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id C31732166B25; Sat, 3 Jun 2023 15:57:54 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 792EC19465A0; Sat, 3 Jun 2023 15:57:54 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 303EF1946595 for ; Sat, 3 Jun 2023 15:57:54 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 093D4C0297C; Sat, 3 Jun 2023 15:57:54 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 01616C0448E for ; Sat, 3 Jun 2023 15:57:53 +0000 (UTC) Received: from us-smtp-inbound-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CBD9D101A531 for ; Sat, 3 Jun 2023 15:57:53 +0000 (UTC) Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-18-3-7pb-MlMhicQbU68Z3Xlw-1; Sat, 03 Jun 2023 11:57:50 -0400 X-MC-Unique: 3-7pb-MlMhicQbU68Z3Xlw-1 Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-6237faa8677so24812506d6.1 for ; Sat, 03 Jun 2023 08:57:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685807870; x=1688399870; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h33Y8ZBTshpCakU2rwk1PXhPYpv5BZ6wPeCedcd+LbM=; b=bmpTKAEnQVaE8vY6JkQ1nxOUKVW75IcAbu6CPA9Ie/Xb0vAXKv8v6H6gUSWGo6ABE7 SEikEZs9gpvs+dQlqhFF82KICTpLREr/sKPchwNQnF9vGhmYgZzL97Mn3bfIqu5+rwBg FUamDAoOynPTpFJJT1rTuIcVi5JRbtcVD0cAXkPjb8P4+5O2S+yRrpMTt85p3QRpRSrl 0PRGbrRc+nbEeQfvbAtYQAWkeG4lSxqJ3P3gFbmHcjuKFsrUTQJdc74pUKvPkEpw8rTO oUMnHE5M1+P5/svx41X+c6dszS8PfcQutlbmD57Y2i6oGvldqtA3MCaq8egPRH1MHslp /6SA== X-Gm-Message-State: AC+VfDwDzorreOVnkBJxIZus+W+Ef3CEDtOmFucGeYs113gJY3hpx0RH QCoMUfiAHU6f2uvkR9dG8JKeYMg= X-Google-Smtp-Source: ACHHUZ4wWtOFT5sPRaDxYAshqrzDTegg4zHApyL4QgQ9q20uQleDbcB0ClYa4RtM1+bM6Hq0UDX2uA== X-Received: by 2002:ad4:5d6c:0:b0:5e9:2bad:c8fa with SMTP id fn12-20020ad45d6c000000b005e92badc8famr2204971qvb.33.1685807869654; Sat, 03 Jun 2023 08:57:49 -0700 (PDT) Received: from localhost (pool-68-160-166-30.bstnma.fios.verizon.net. [68.160.166.30]) by smtp.gmail.com with ESMTPSA id m13-20020a05621402ad00b00623839cba8csm2292876qvv.44.2023.06.03.08.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 08:57:49 -0700 (PDT) Date: Sat, 3 Jun 2023 11:57:48 -0400 From: Mike Snitzer To: Dave Chinner Message-ID: References: MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Subject: Re: [dm-devel] [PATCH v7 0/5] Introduce provisioning primitives X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jens Axboe , Christoph Hellwig , Joe Thornber , Stefan Hajnoczi , "Michael S. Tsirkin" , "Darrick J. Wong" , Jason Wang , Bart Van Assche , linux-kernel@vger.kernel.org, Joe Thornber , linux-block@vger.kernel.org, dm-devel@redhat.com, Andreas Dilger , Sarthak Kukreti , linux-fsdevel@vger.kernel.org, Theodore Ts'o , linux-ext4@vger.kernel.org, Brian Foster , Alasdair Kergon Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Content-Disposition: inline Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 T24gRnJpLCBKdW4gMDIgMjAyMyBhdCAgODo1MlAgLTA0MDAsCkRhdmUgQ2hpbm5lciA8ZGF2aWRA ZnJvbW9yYml0LmNvbT4gd3JvdGU6Cgo+IE9uIEZyaSwgSnVuIDAyLCAyMDIzIGF0IDExOjQ0OjI3 QU0gLTA3MDAsIFNhcnRoYWsgS3VrcmV0aSB3cm90ZToKPiA+IE9uIFR1ZSwgTWF5IDMwLCAyMDIz IGF0IDg6MjjigK9BTSBNaWtlIFNuaXR6ZXIgPHNuaXR6ZXJAa2VybmVsLm9yZz4gd3JvdGU6Cj4g PiA+Cj4gPiA+IE9uIFR1ZSwgTWF5IDMwIDIwMjMgYXQgMTA6NTVQIC0wNDAwLAo+ID4gPiBKb2Ug VGhvcm5iZXIgPHRob3JuYmVyQHJlZGhhdC5jb20+IHdyb3RlOgo+ID4gPgo+ID4gPiA+IE9uIFR1 ZSwgTWF5IDMwLCAyMDIzIGF0IDM6MDLigK9QTSBNaWtlIFNuaXR6ZXIgPHNuaXR6ZXJAa2VybmVs Lm9yZz4gd3JvdGU6Cj4gPiA+ID4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBBbHNvIEpvZSwgZm9yIHlv dSBwcm9wb3NlZCBkbS10aGlucCBkZXNpZ24gd2hlcmUgeW91IGRpc3RpbnF1aXNoCj4gPiA+ID4g PiBiZXR3ZWVuICJwcm92aXNpb24iIGFuZCAicmVzZXJ2ZSI6IFdvdWxkIGl0IG1ha2Ugc2Vuc2Ug Zm9yIFJFUV9NRVRBCj4gPiA+ID4gPiAoZS5nLiBhbGwgWEZTIG1ldGFkYXRhKSB3aXRoIFJFUV9Q Uk9WSVNJT04gdG8gYmUgdHJlYXRlZCBhcyBhbgo+ID4gPiA+ID4gTEJBLXNwZWNpZmljIGhhcmQg cmVxdWVzdD8gIFdoZXJlYXMgUkVRX1BST1ZJU0lPTiBvbiBpdHMgb3duIHByb3ZpZGVzCj4gPiA+ ID4gPiBtb3JlIGZyZWVkb20gdG8ganVzdCByZXNlcnZlIHRoZSBsZW5ndGggb2YgYmxvY2tzPyAo ZS5nLiBmb3IgWEZTCj4gPiA+ID4gPiBkZWxhbGxvYyB3aGVyZSBMQkEgcmFuZ2UgaXMgdW5rbm93 biwgYnV0IGRtLXRoaW5wIGNhbiBiZSBhc2tlZCB0bwo+ID4gPiA+ID4gcmVzZXJ2ZSBzcGFjZSB0 byBhY2NvbW9kYXRlIGl0KS4KPiA+ID4gPiA+Cj4gPiA+ID4KPiA+ID4gPiBNeSBwcm9wb3NhbCBv bmx5IGludm9sdmVzICdyZXNlcnZlJy4gIFByb3Zpc2lvbmluZyB3aWxsIGJlIGRvbmUgYXMgcGFy dCBvZgo+ID4gPiA+IHRoZSB1c3VhbCBpbyBwYXRoLgo+ID4gPgo+ID4gPiBPSywgSSB0aGluayB3 ZSdkIGRvIHdlbGwgdG8gcGluIGRvd24gdGhlIHRvcC1sZXZlbCBibG9jayBpbnRlcmZhY2VzIGlu Cj4gPiA+IHF1ZXN0aW9uLiBCZWNhdXNlIHRoaXMgcGF0Y2hzZXQncyBibG9jayBpbnRlcmZhY2Ug cGF0Y2ggKDIvNSkgaGVhZGVyCj4gPiA+IHNheXM6Cj4gPiA+Cj4gPiA+ICJUaGlzIHBhdGNoIGFs c28gYWRkcyB0aGUgY2FwYWJpbGl0eSB0byBjYWxsIGZhbGxvY2F0ZSgpIGluIG1vZGUgMAo+ID4g PiBvbiBibG9jayBkZXZpY2VzLCB3aGljaCB3aWxsIHNlbmQgUkVRX09QX1BST1ZJU0lPTiB0byB0 aGUgYmxvY2sKPiA+ID4gZGV2aWNlIGZvciB0aGUgc3BlY2lmaWVkIHJhbmdlLCIKPiA+ID4KPiA+ ID4gU28gaXQgd2lyZXMgdXAgYmxrZGV2X2ZhbGxvY2F0ZSgpIHRvIGNhbGwgYmxrZGV2X2lzc3Vl X3Byb3Zpc2lvbigpLiBBCj4gPiA+IHVzZXIgb2YgWEZTIGNvdWxkIHRoZW4gdXNlIGZhbGxvY2F0 ZSgpIGZvciB1c2VyIGRhdGEgLS0gd2hpY2ggd291bGQKPiA+ID4gY2F1c2UgdGhpbnAncyByZXNl cnZlIHRvIF9ub3RfIGJlIHVzZWQgZm9yIGNyaXRpY2FsIG1ldGFkYXRhLgo+IAo+IE1pa2UsIEkg dGhpbmsgeW91IG1pZ2h0IGhhdmUgbWlzdW5kZXJzdG9vZCB3aGF0IEkgaGF2ZSBiZWVuIHByb3Bv c2luZy4KPiBQb3NzaWJseSB1bmludGVudGlvbmFsbHksIEkgZGlkbid0IGNhbGwgaXQgUkVRX09Q X1BST1ZJU0lPTiBidXQKPiB0aGF0J3Mgd2hhdCBJIGludGVuZGVkIC0gdGhlIG9wZXJhdGlvbiBk b2VzIG5vdCBjb250YWluIGRhdGEgYXQgYWxsLgo+IEl0J3MgYW4gb3BlcmF0aW9uIGxpa2UgUkVR X09QX0RJU0NBUkQgb3IgUkVRX09QX1dSSVRFX1pFUk9TIC0gaXQKPiBjb250YWlucyBhIHJhbmdl IG9mIHNlY3RvcnMgdGhhdCBuZWVkIHRvIGJlIHByb3Zpc2lvbmVkIChvcgo+IGRpc2NhcmRlZCks IGFuZCBub3RoaW5nIGVsc2UuCgpObywgSSB1bmRlcnN0b29kIHRoYXQuCgo+IFRoZSB3cml0ZSBJ T3MgdGhlbXNlbHZlcyBhcmUgbm90IHRhZ2dlZCB3aXRoIGFueXRoaW5nIHNwZWNpYWwgYXQgYWxs LgoKSSBrbm93LCBidXQgSSd2ZSBiZWVuIGxvb2tpbmcgYXQgaG93IHRvIGFsc28gaGFuZGxlIHRo ZSBkZWxhbGxvYwp1c2VjYXNlIChhbmQgeWVzIEkga25vdyB5b3UgZmVlbCBpdCBkb2Vzbid0IG5l ZWQgaGFuZGxpbmcsIHRoZSBpc3N1ZQppcyBYRlMgZG9lcyBkZWFsIG5pY2VseSB3aXRoIGVuc3Vy aW5nIGl0IGhhcyBzcGFjZSB3aGVuIGl0IHRyYWNrcyBpdHMKYWxsb2NhdGlvbnMgb24gInRoaWNr IiBzdG9yYWdlIC0tIHNvIGFkZGluZyBjb29yZGluYXRpb24gYmV0d2VlbiBYRlMKYW5kIGRtLXRo aW4gbGF5ZXJzIHByb3ZpZGVzIGNvbXBhcmFibGUgc2FmZXR5Li4gdGhhdCBzYWZldHkgaXMgYW4K ZXhwZWN0ZWQgbm9ybSkuCgpCdXQgcmF0aGVyIHRoYW4gZGlzY3VzcyBpbiB0ZXJtcyBvZiBkYXRh IHZzIG1ldGFkYXRhLCB0aGUgZGlzdGluY3Rpb24KaXM6CjEpIExCQSByYW5nZSByZXNlcnZhdGlv biAobm9ybWFsIGNhc2UsIHlvdXIgcHJvcG9zYWwpCjIpIG5vbi1MQkEgcmVzZXJ2YXRpb24gKGFi c29sdXRlIHZhbHVlLCBMQkEgcmFuZ2UgaXMga25vd24gYXQgbGF0ZXIgc3RhZ2UpCgpCdXQgSSdt IGNsZWFybHkgZ29pbmcgb2ZmIHNjcmlwdCBmb3IgZHdlbGxpbmcgb24gd2FudGluZyB0byBoYW5k bGUKYm90aC4KCk15IGxvb2tpbmcgYXQgKGFiKXVzaW5nIFJFUV9NRVRBIGJlaW5nIHNldCAodXNl IDEpIHZzIG5vdCAodXNlIDIpIHdhcwphIGNydWRlIHNpbXBsaWZpY2F0aW9uIGZvciBicmFuY2hp bmcgYmV0d2VlbiB0aGUgMiBhcHByb2FjaGVzLgoKQW5kIEkgdW5kZXJzdGFuZCBJIG1hZGUgeW91 IG5lcnZvdXMgYnkgZXhwYW5kaW5nIHRoZSBzY29wZSB0byBhIG11Y2gKbW9yZSBtdWRkbGVkL3No aXR0eSBpbnRlcmZhY2UuIDspCgpXZSBhbGwganVzdCBuZWVkIHRvIGZvY3VzIG9uIHlvdXIgcHJv cG9zYWwgYW5kIEpvZSdzIGRtLXRoaW4KcmVzZXJ2YXRpb24gZGVzaWduLi4uCgpbU2FydGhhazog RllJLCB0aGlzIGltcGxpZXMgdGhhdCBpdCBkb2Vzbid0IHJlYWxseSBtYWtlIHNlbnNlIHRvIGFk ZApkbS10aGlucCBzdXBwb3J0IGJlZm9yZSBKb2UncyBkZXNpZ24gaXMgaW1wbGVtZW50ZWQuICBP dGhlcndpc2Ugd2UnbGwKaGF2ZSAyIGRpZmZlcmVudCByZXNwb25zZXMgdG8gUkVRX09QX1BST1ZJ U0lPTi4gIFRoZSBvbmUgdGhhdCBpcwpjYXB0dXJlZCBpbiB5b3VyIHBhdGNoc2V0IGlzbid0IGFk ZXF1YXRlIHRvIHByb3Blcmx5IGhhbmRsZSBlbnN1cmluZwp1cHBlciBsYXllciAobGlrZSBYRlMp IGNhbiBkZXBlbmQgb24gdGhlIHNwYWNlIGJlaW5nIGF2YWlsYWJsZSBhY3Jvc3MKc25hcHNob3Qg Ym91bmRhcmllcy5dCgo+IGkuZS4gVGhlIHByb3Bvc2FsIEkgbWFkZSBkb2VzIG5vdCB1c2UgUkVR X1BST1ZJU0lPTiBhbnl3aGVyZSBpbiB0aGUKPiBtZXRhZGF0YS9kYXRhIElPIHBhdGg7IHByb3Zp c2lvbmVkIHJlZ2lvbnMgYXJlIGNyZWF0ZWQgYnkgc2VwYXJhdGUKPiBvcGVyYXRpb25zIGFuZCBt dXN0IGJlIHRyYWNrZWQgYnkgdGhlIHVuZGVybHlpbmcgYmxvY2sgZGV2aWNlLCB0aGVuCj4gdHJl YXQgYW55IHdyaXRlIElPIHRvIHRob3NlIHJlZ2lvbnMgYXMgIm11c3Qgbm90IGZhaWwgdy8gRU5P U1BDIgo+IElPcy4KPiAKPiBUaGVyZSBzZWVtcyB0byBiZSBhIGxvdCBvZiBmZWFyIGFib3V0IHVz ZXIgZGF0YSByZXF1aXJpbmcKPiBwcm92aXNpb25pbmcuIFRoaXMgaXMgdW5mb3VuZGVkIC0gcHJv dmlzaW9uaW5nIGlzIG9ubHkgbmVlZGVkIGZvcgo+IGV4cGxpY2l0bHkgcHJvdmlzaW9uZWQgc3Bh Y2UgdmlhIGZhbGxvY2F0ZSgpLCBub3QgZXZlcnkgYnl0ZSBvZgo+IHVzZXIgZGF0YSB3cml0dGVu IHRvIHRoZSBmaWxlc3lzdGVtICh0aGUgbW9kZWwgQnJpYW4gaXMgcHJvcG9zaW5nKS4KCkFzIEkg bWVudGlvbmVkIGFib3ZlLCBJIHdhcyBqdXN0IHRyeWluZyB0byBnZXQgWEZTLW9uLXRoaW5wIHRv Cm1haW50YWluIHBhcml0eSB3aXRoIGhvdyBYRlMncyBkZWxhbGxvYyBhY2NvdW50aW5nIHdvcmtz IG9uICJ0aGljayIKc3RvcmFnZS4KCkJ1dCBoYXBweSB0byBwdXQgdGhhdCB0byBvbmUgc2lkZS4g IE1haW50YWluIGZvY3VzIGxpa2UgSSBtZW50aW9uZWQKYWJvdmUuICBJJ20gaGFwcHkgd2UgaGF2 ZSBtb21lbnR1bSBhbmQgYWdyZWVtZW50IG9uIHRoaXMgZGVzaWduIG5vdy4KUmF0aGVyIHRoYW4g YmUgY29udGVudCB3aXRoIHRoYXQsIEkgd2FzIG1pc3Rha2VubHkgbG9va2luZyBhdCBvdGhlcgph c3BlY3RzIGFuZCBpbiBkb2luZyBzbyBpbnRyb2R1Y2VkICJub2lzZSIgYmVmb3JlIHdlJ3ZlIGlt cGxlbWVudGVkCndoYXQgd2UgYWxsIGNvbXBsZXRlbHkgYWdyZWUgb246IHlvdXIgYW5kIGpvZSdz IGRlc2lnbnMuCgo+IEV4Y2Vzc2l2ZSB1c2Ugb2YgZmFsbG9jYXRlKCkgaXMgc2VsZiBjb3JyZWN0 aW5nIC0gaWYgdXNlcnMgYW5kL29yCj4gdGhlaXIgYXBwbGljYXRpb25zIHByb3Zpc2lvbiB0b28g bXVjaCwgdGhleSBhcmUgZ29pbmcgdG8gZ2V0IEVOT1NQQwo+IG9yIGhhdmUgdG8gcGF5IG1vcmUg dG8gZXhwYW5kIHRoZSBiYWNraW5nIHBvb2wgcmVzZXJ2ZXMgdGhleSBuZWVkLgo+IEJ1dCB0aGF0 J3Mgbm90IGEgcHJvYmxlbSB0aGUgYmxvY2sgZGV2aWNlIHNob3VsZCBiZSB0cnlpbmcgdG8gc29s dmU7Cj4gdGhhdCdzIGEgcHJvYmxlbSBmb3IgdGhlIHN5c2FkbWluIGFuZC9vciBiZWFuIGNvdW50 ZXJzIHRvIGFkZHJlc3MuCj4gCj4gPiA+Cj4gPiA+IFRoZSBvbmx5IHdheSB0byBkaXN0aW5xdWlz aCB0aGUgY2FsbGVyIChiZXR3ZWVuIG9uLWJlaGFsZiBvZiB1c2VyIGRhdGEKPiA+ID4gdnMgWEZT IG1ldGFkYXRhKSB3b3VsZCBiZSBSRVFfTUVUQT8KPiA+ID4KPiA+ID4gU28gc2hvdWxkIGRtLXRo aW5wIGhhdmUgYSBSRVFfTUVUQS1iYXNlZCBkaXN0aW5jdGlvbj8gT3IganVzdCB0cmVhdAo+ID4g PiBhbGwgUkVRX09QX1BST1ZJU0lPTiB0aGUgc2FtZT8KPiA+ID4KPiA+IEknbSBpbiBmYXZvciBv ZiBhIFJFUV9NRVRBLWJhc2VkIGRpc3RpbmN0aW9uLgo+IAo+IFdoeT8gV2hhdCAqcmVxdWlyZW1l bnQqIGlzIGRyaXZpbmcgdGhlIG5lZWQgZm9yIHRoaXMgZGlzdGluY3Rpb24/CgpUaGluayBJIGFu c3dlcmVkIHRoYXQgYWJvdmUsIFhGUyBkZWxhbGxvYyBhY2NvdW50aW5nIHBhcml0eSBvbiB0aGlu cC4KCj4gQXMgdGhlIHBlcnNvbiB3aG8gcHJvcG9zZWQgdGhpcyBuZXcgUkVRX09QX1BST1ZJU0lP TiBhcmNoaXRlY3R1cmUsCj4gSSdtIGRlYWQgc2V0IGFnYWluc3QgaXQuICBBbGxvd2luZyB0aGUg YmxvY2sgZGV2aWNlIHByb3ZpZGUgYSBzZXQgb2YKPiBwb29ybHkgZGVmaW5lZCAiY29uZGl0aW9u YWwgZ3VhcmFudGVlcyIgcG9saWNpZXMgaW5zdGVhZCBvZiBhCj4gbWVjaGFuaXNtIHdpdGggYSBz aW5nbGUgaXJvbmNsYWQgZ3VhcmFudGVlIGRlZmVhdHMgdGhlIGVudGlyZQo+IHB1cnBvc2Ugb2Yg dGhlIHByb3Bvc2FsLiAKPiAKPiBXZSBoYXZlIGEgcmVxdWlyZW1lbnQgZnJvbSB0aGUgKmtlcm5l bCBBQkkqIHRoYXQgKnVzZXIgZGF0YSB3cml0ZXMqCj4gbXVzdCBub3QgZmFpbCB3aXRoIEVOT1NQ QyBhZnRlciBhbiBmYWxsb2NhdGUoKSBvcGVyYXRpb24uICBUaGF0J3MKPiBvbmUgb2YgdGhlIGhp Z2ggbGV2ZWwgcG9saWNpZXMgd2UgbmVlZCB0byBpbXBsZW1lbnQuIFRoZSBmaWxlc3lzdGVtCj4g aXMgYWxyZWFkeSBjYXBhYmxlIG9mIGd1YXJhbnRlZWluZyBpdCB3b24ndCBnaXZlIHRoZSB1c2Vy IEVOT1NQQwo+IGFmdGVyIGZhbGxvY2F0ZSwgd2Ugbm93IG5lZWQgYSBndWFyYW50ZWUgZnJvbSB0 aGUgZmlsZXN5c3RlbSdzCj4gYmFja2luZyBzdG9yZSB0aGF0IGl0IHdvbid0IGdpdmUgRU5PU1BD LCB0b28uCgpZZXMsIEkgd2FzIHRyeWluZyB0byBuYXZpZ2F0ZSBKb2UncyByZWx1Y3RhbmNlIHRv IGV2ZW4gc3VwcG9ydApmYWxsb2NhdGUoKSBmb3IgYXJiaXRyYXJ5IHVzZXIgZGF0YS4gIFRoYXQn cyB3aGVyZSB0aGUgUkVRX01FVEEgdnMKZGF0YSBkaXN0aW5jdGlvbiBjcmVwdCBpbiBmb3IgbWUu ICBCdXQgYXMgeW91IHNheTogdXNpbmcgZmFsbG9jYXRlKCkKZXhjZXNzaXZlbHkgaXMgc2VsZi1j b3JyZWN0aW5nLgoKPiBUaGUgX290aGVyIHRoaW5nXyB3ZSBuZWVkIHRvIGltcGxlbWVudCBpcyBh IG1ldGhvZCBvZiBndWFyYW50ZWVpbmcKPiB0aGUgZmlsZXN5c3RlbSB3b24ndCBzaHV0IGRvd24g d2hlbiB0aGUgYmFja2luZyBkZXZpY2UgZ29lcyBFTk9TUEMKPiB1bmV4cGVjdGVkIGR1cmluZyBt ZXRhZGF0YSB3cml0ZWJhY2suICBTbyB3ZSBhbHNvIG5lZWQgdGhlIGJhY2tpbmcKPiBkZXZpY2Ug dG8gZ3VhcmFudGVlIHRoZSByZWdpb25zIHdlIHdyaXRlIG1ldGFkYXRhIHRvIHdvbid0IGdpdmUK PiBFTk9TUEMuCgpZZWFwLgoKPiBUaGF0J3MgdGhlIHdob2xlIHBvaW50IG9mIFJFUV9PUF9QUk9W SVNJT046IGZyb20gdGhlIGxheWVycyBhYm92ZQo+IHRoZSBibG9jayBkZXZpY2UsIHRoZXJlIGlz IC16ZXJvLSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIGd1YXJhbnRlZQo+IHdlIG5lZWQgZm9yIHVz ZXIgZGF0YSB3cml0ZXMgdG8gYXZvaWQgRU5PU1BDIGFuZCBmb3IgbWV0YWRhdGEgd3JpdGVzCj4g dG8gYXZvaWQgRU5PU1BDLiBUaGV5IGFyZSBvbmUgYW5kIHRoZSBzYW1lLgoKSSBrbm93LiAgVGhl IGRpZmZlcmVuY2UgY29tZXMgZnJvbSBkZWxhbGxvYyBpbml0aWFsbHkgbmVlZGluZyBhbgphYnNv bHV0ZSB2YWx1ZSBvZiByZXNlcnZlIHJhdGhlciB0aGFuIGEgc3BlY2lmaWMgTEJBIHJhbmdlLgoK PiBIZW5jZSBpZiB0aGUgYmxvY2sgZGV2aWNlIGlzIGdvaW5nIHRvIHNheSAiSSBzdXBwb3J0IHBy b3Zpc2lvbmluZyIKPiBidXQgdGhlbiBnaXZlIGRpZmZlcmVudCBjb25kaXRpb25hbCBndWFyYW50 ZWVzIGFjY29yZGluZyB0byB0aGUKPiAqdHlwZSBvZiBkYXRhKiBpbiB0aGUgSU8gcmVxdWVzdCwg dGhlbiBpdCBkb2VzIG5vdCBwcm92aWRlIHRoZQo+IGZ1bmN0aW9uYWxpdHkgdGhlIGhpZ2hlciBs YXllcnMgYWN0dWFsbHkgcmVxdWlyZSBmcm9tIGl0LgoKSSB3YXMgZ29pbmcgZm9yIHJlbGF4aW5n IHRoZSAiZHluYW1pYyIgYXBwcm9hY2ggKEJyaWFuJ3MpIHRvIGJlCmJlc3QtZWZmb3J0IC0tIGFu ZCByZWFsbHkgb25seSBmb3IgWEZTIGRlbGFsbG9jIHVzZWNhc2UuICBFdmVyeSBvdGhlcgp1c2Vj YXNlIHdvdWxkIHJlc3BlY3QgeW91ciBhbmQgSm9lJ3MgdmlzaW9uLgoKPiBJbmRlZWQsIHdoYXQg dHlwZSBvZiBkYXRhIHRoZSBJTyBjb250YWlucyBpcyAqY29udGV4dCBkZXBlbmRlbnQqLgo+IEZv ciBleGFtcGxlLCBzb21ldGltZXMgd2Ugd3JpdGUgbWV0YWRhdGEgd2l0aCB1c2VyIGRhdGEgSU8g YW5kIGJ1dAo+IHdlIHN0aWxsIG5lZWQgcHJvdmlzaW9uaW5nIGd1YXJhbnRlZXMgYXMgaWYgaXQg d2FzIGlzc3VlZCBhcwo+IG1ldGFkYXRhIElPLiBUaGlzIGlzIHRoZSBjYXNlIGZvciBta2ZzIGlu aXRpYWxpc2luZyB0aGUgZmlsZSBzeXN0ZW0KPiBieSB3cml0aW5nIGRpcmVjdGx5IHRvIHRoZSBi bG9jayBkZXZpY2UuCgpJJ20gYXdhcmUuCgo+IElPV3MsIGZpbGVzeXN0ZW0gbWV0YWRhdGEgSU8g aXNzdWVkIGZyb20ga2VybmVsIGNvbnRleHQgd291bGQgYmUKPiBjb25zaWRlcmVkIG1ldGFkYXRh IElPLCBidXQgZnJvbSB1c2Vyc3BhY2UgaXQgd291bGQgYmUgY29uc2lkZXJlZAo+IG5vcm1hbCB1 c2VyIGRhdGEgSU8gYW5kIGhlbmNlIHRyZWF0ZWQgZGlmZmVyZW50bHkuIEJ1dCB0aGUgcmVhbGl0 eQo+IGlzIHRoYXQgdGhleSBib3RoIG5lZWQgdGhlIHNhbWUgcHJvdmlzaW9uaW5nIGd1YXJhbnRl ZXMgdG8gYmUKPiBwcm92aWRlZCBieSB0aGUgYmxvY2sgZGV2aWNlLgoKV2hhdCBJIHdhcyBsb29r aW5nIGF0IGlzIG1ha2luZyB0aGUgZmFsbG9jYXRlIGludGVyZmFjZSBhYmxlIHRvCmV4cHJlc3M6 IEkgbmVlZCBkYXZlJ3MgcmVxdWlyZW1lbnRzIChib2ctc3RhbmRhcmQgYWN0dWFsbHkpIHZzIEkg bmVlZApub24tTEJBIGJlc3QgZWZmb3J0LgoKPiBTbyBob3cgZG8gdXNlcnNwYWNlIHRvb2xzIGRl YWwgd2l0aCB0aGlzIGlmIHRoZSBibG9jayBkZXZpY2UKPiByZXF1aXJlcyBSRVFfTUVUQSBvbiB1 c2VyIGRhdGEgSU9zIHRvIGRvIHRoZSByaWdodCB0aGluZyBoZXJlPyBBbmQKPiBpZiB3ZSBwcm92 aWRlIGEgbWVjaGFuaXNtIHRvIGFsbG93IHRoaXMsIGhvdyBkbyB3ZSBwcmV2ZW50IHVzZXJzcGFj ZQo+IGZvciBhbHdheXMgdXNpbmcgaXQgb24gd3JpdGVzIHRvIGZhbGxvY2F0ZSgpIHByb3Zpc2lv bmVkIHNwYWNlPwo+IAo+IEl0J3MganVzdCBub3QgcHJhY3RpY2FsIGZvciB0aGUgYmxvY2sgZGV2 aWNlIHRvIGFkZCBhcmJpdHJhcnkKPiBjb25zdHJhaW50cyBiYXNlZCBvbiB0aGUgdHlwZSBvZiBJ TyBiZWNhdXNlIHdlIHRoZW4gaGF2ZSB0byBhZGQKPiBtZWNoYW5pc21zIHRvIHVzZXJzcGFjZSBB UElzIHRvIGFsbG93IHRoZW0gdG8gY29udHJvbCB0aGUgSU8gY29udGV4dAo+IHNvIHRoZSBibG9j ayBkZXZpY2Ugd2lsbCBkbyB0aGUgcmlnaHQgdGhpbmcuIEVzcGVjaWFsbHkgY29uc2lkZXJpbmcK PiB3ZSByZWFsbHkgb25seSBuZWVkIG9uZSB0eXBlIG9mIGd1YXJhbnRlZSByZWdhcmRsZXNzIG9m IHdoZXJlIHRoZSBJTwo+IG9yaWdpbmF0ZXMgZnJvbSBvciB3aGF0IHR5cGUgb2YgZGF0YSB0aGUg SU8gY29udGFpbnMuLi4uCgpJZiBhbnl0aGluZyBteSBkaXNwb3NpdGlvbiBvbiB0aGUgY29uZGl0 aW9uYWwgdG8gcmVxdWlyZSBhIFJFUV9NRVRBCihvciBzb21lIGZhbGxvY2F0ZSBnZW5lcmF0ZWQg UkVRX1VOU0hBUkUgZGl0dG8gdG8gcmVmbGVjdCB0aGUgc2FtZSkgdG8KcGVyZm9ybSB5b3VyIGFw cHJvYWNoIHRvIFJFUV9PUF9QUk9WSVNJT04gYW5kIGhvbm9yIGZhbGxvY2F0ZSgpCnJlcXVpcmVt ZW50cyBpcyBhIGJpZyBwcm9ibGVtLiAgV291bGQgYmUgbXVjaCBiZXR0ZXIgdG8gaGF2ZSBhIGZs YWcgdG8KZXhwcmVzcyAidGhpcyByZXNlcnZhdGlvbiBkb2VzIG5vdCBoYXZlIGFuIExCQSByYW5n ZSBfeWV0XywKbmV2ZXJ0aGVsZXNzIHRyeSB0byBiZSBtaW5kZnVsIG9mIHRoaXMgZXhwZWN0ZWQg bmVhci10ZXJtIGJsb2NrCmFsbG9jYXRpb24iLgoKQnV0IEknbGwgc3RvcCBpbmxpbmluZyByZXBl dGl0aXZlIChzaW1pbGFyIGJ1dCBkaWZmZXJlbnQpIGFuc3dlcnMgdG8KeW91ciBjb25jZXJuIG5v dyA7KQogCj4gPiBEb2VzIHRoYXQgaW1wbHkgdGhhdAo+ID4gUkVRX01FVEEgYWxzbyBuZWVkcyB0 byBiZSBwYXNzZWQgdGhyb3VnaCB0aGUgYmxvY2svZmlsZXN5c3RlbSBzdGFjawo+ID4gKGVnLiBS RVFfT1BfUFJPVklPTiArIFJFUV9NRVRBIG9uIGEgbG9vcCBkZXZpY2UgdHJhbnNsYXRlcyB0byBh Cj4gPiBmYWxsb2NhdGUoPGluc2VydCBtZXRhIGZsYWcgbmFtZT4pIHRvIHRoZSB1bmRlcmx5aW5n IGZpbGUpPwo+IAo+IFRoaXMgaXMgZXhhY3RseSB0aGUgc2FtZSBjYXNlIGFzIGFib3ZlOiB0aGUg bG9vcGJhY2sgZGV2aWNlIGRvZXMKPiB1c2VyIGRhdGEgSU8gdG8gdGhlIGJhY2tpbmcgZmlsZS4g SGVuY2Ugd2UgaGF2ZSBhbm90aGVyIHNpdHVhdGlvbgo+IHdoZXJlIG1ldGFkYXRhIElPIGlzIGlz c3VlZCB0byBmYWxsb2NhdGUoKWQgdXNlciBkYXRhIHJhbmdlcyBhcyB1c2VyCj4gZGF0YSByYW5n ZXMgYW5kIHNvIHdvdWxkIGJlIGdpdmVuIGEgbGVzc2VyIGd1YXJhbnRlZSB0aGF0IHdvdWxkIGxl YWQKPiB0byB1cHBlciBmaWxlc3lzdGVtIGZhaWx1cmUuIEJPdGggdXBwZXIgYW5kIGxvd2VyIGZp bGVzeXN0ZW0gZGF0YQo+IGFuZCBtZXRhZGF0YSBuZWVkIHRvIGJlIHByb3ZpZGVkIHRoZSBzYW1l IEVOT1NQQyBndWFyYW50ZWVzIGJ5IHRoZWlyCj4gYmFja2luZyBzdG9yZXMuLi4uCj4gCj4gVGhl IHdob2xlIHBvaW50IG9mIHRoZSBSRVFfT1BfUFJPVklTSU9OIHByb3Bvc2FsIEkgbWFkZSBpcyB0 aGF0IGl0Cj4gZG9lc24ndCByZXF1aXJlIGFueSBzcGVjaWFsIGhhbmRsaW5nIGluIGNvcm5lciBj YXNlcyBsaWtlIHRoaXMuCj4gVGhlcmUgYXJlIG5vIGNyb3NzLWxheWVyIGludGVyYWN0aW9ucyBu ZWVkZWQgdG8gbWFrZSBldmVyeXRoaW5nIHdvcmsKPiBjb3JyZWN0bHkgYmVjYXVzZSB0aGUgcHJv dmlzaW9uaW5nIGd1YXJhbnRlZSBpcyBub3QgLWRhdGEgdHlwZQo+IGRlcGVuZGVudCouIFRoZSBl bnRpcmUgdXNlciBJTyBwYXRoIGNvZGUgcmVtYWlucyB1bnRvdWNoZWQgYW5kCj4gYmxpc3NmdWxs eSB1bmF3YXJlIG9mIHByb3Zpc2lvbmVkIHJlZ2lvbnMuCj4gCj4gQW5kLCByZWFsaXN0aWNhbGx5 LCBpZiB3ZSBoYXZlIHRvIHN0YXJ0IGhhbmRsaW5nIGNvbXBsZXggY29ybmVyCj4gY2FzZXMgaW4g dGhlIGZpbGVzeXN0ZW0gYW5kIElPIHBhdGggbGF5ZXJzIHRvIG1ha2UgUkVRX09QX1BST1ZJU0lP Tgo+IHdvcmsgY29ycmVjdGx5IGJlY2F1c2Ugb2YgYXJiaXRhcnkgY29uc3RyYWludHMgaW1wb3Nl ZCBieSB0aGUgYmxvY2sKPiBsYXllciBpbXBsZW1lbnRhdGlvbnMsIHRoZW4gd2UndmUgZmFpbGVk IG1pc2VyYWJseSBhdCB0aGUgZGVzaWduIGFuZAo+IGFyY2hpdGVjdHVyZSBzdGFnZS4KPiAKPiBL ZWVwIGluIG1pbmQgdGhhdCBldmVyeSBhdHRlbXB0IG1hZGUgc28gZmFyIHRvIGFkZHJlc3MgdGhl IHByb2JsZW1zCj4gd2l0aCBibG9jayBkZXZpY2UgRU5PU1BDIGVycm9ycyBoYXMgZmFpbGVkIGJl Y2F1c2Ugb2YgdGhlIGNvbXBsZXhpdHkKPiBvZiB0aGUgY29ybmVyIGNhc2VzIHRoYXQgaGF2ZSBh cmlzZW4gZHVyaW5nIGRlc2lnbiBhbmQvb3IKPiBpbXBsZW1lbnRhdGlvbi4gSXQncyBwcmV0dHkg aXJvbmljIHRoYXQgbm93IHdlIGhhdmUgYSBwcm9wb3NhbCB0aGF0Cj4gaXMgcmVtYXJrYWJseSBz aW1wbGUsIGZyZWUgb2YgY29ybmVyIGNhc2VzIGFuZCBoYXMgdmlydHVhbGx5IG5vCj4gY3Jvc3Mt bGF5ZXIgY291cGxpbmcgYXQgYWxsLCB0aGUgZmlyc3QgdGhpbmcgdGhhdCBwZW9wbGUgd2FudCB0 byBkbwo+IGlzIGFkZCBhcmJpdHJhcnkgaW1wbGVtZW50YXRpb24gY29uc3RyYWludHMgdGhhdCBy ZXN1bHQgaW4gY29tcGxleAo+IGNyb3NzLWxheWVyIGNvcm5lciBjYXNlcyB0aGF0IG5vdyBuZWVk IHRvIGJlIGhhbmRsZWQuLi4uCj4gCj4gUHV0IHNpbXBseTogaWYgd2UgcmVzdHJpY3QgUkVRX09Q X1BST1ZJU0lPTiBndWFyYW50ZWVzIHRvIGp1c3QKPiBSRVFfTUVUQSB3cml0ZXMgKG9yIGFueSBv dGhlciBzcGVjaWZpYyB0eXBlIG9mIHdyaXRlIG9wZXJhdGlvbikgdGhlbgo+IGl0J3Mgc2ltcGx5 IG5vdCB3b3J0aCBwZXJzdWluZyBhdCB0aGUgZmlsZXN5c3RlbSBsZXZlbCBiZWNhdXNlIHRoZQo+ IGd1YXJhbnRlZXMgd2UgYWN0dWFsbHkgbmVlZCBqdXN0IGFyZW4ndCB0aGVyZSBhbmQgdGhlIGNv bXBsZXhpdHkgb2YKPiBkaXNjb3ZlcmluZyBhbmQgaGFuZGxpbmcgdGhvc2UgY29ybmVyIGNhc2Vz IGp1c3QgaXNuJ3Qgd29ydGggdGhlCj4gZWZmb3J0LgoKSGVyZSBpcyB3aGVyZSBJIGdldCB0byBz YXk6IEkgdGhpbmsgeW91IG1pc3VuZGVyc3Rvb2QgbWUgKGJ1dCBpdCB3YXMKbXkgZmF1bHQgZm9y IG5vdCBiZWluZyBhYnNvbHV0ZWx5IGNsZWFyOiBJJ20gdmVyeSBtdWNoIG9uIHRoZSBzYW1lCnBh Z2UgYXMgeW91IGFuZCBKb2U7IGFuZCB5b3VyIHZpc2lvbnMgbmVlZCB0byBqdXN0IGJlIGltcGxl bWVudGVkCkFTQVApLgoKSSB3YXMgdGFraW5nIHlvdXIgZGVzaWducyBhcyBhIGdpdmVuLCBidXQg bG9va2luZyBmdXJ0aGVyIGF0OiBob3cgZG8Kd2UgYWxzbyBoYW5kbGUgdGhlIG5vbi1MQkEgKGRl bGFsbG9jKSB1c2VjYXNlIF9iZWZvcmVfIHdlIGluY2x1ZGUKUkVRX09QX1BST1ZJU0lPTiBpbiBr ZXJuZWwuCgpCdXQgSSdtIGhhcHB5IHRvIGxldCB0aGUgZGVsYWxsb2MgY2FzZSBnbyAod2UgY2Fu IHJldmlzaXQgYWRkcmVzc2luZwppdCBpZi93aGVuIG5lZWRlZCkuCgpNaWtlCgotLQpkbS1kZXZl bCBtYWlsaW5nIGxpc3QKZG0tZGV2ZWxAcmVkaGF0LmNvbQpodHRwczovL2xpc3RtYW4ucmVkaGF0 LmNvbS9tYWlsbWFuL2xpc3RpbmZvL2RtLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC276C77B73 for ; Sat, 3 Jun 2023 15:58:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230138AbjFCP6k (ORCPT ); Sat, 3 Jun 2023 11:58:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229819AbjFCP6i (ORCPT ); Sat, 3 Jun 2023 11:58:38 -0400 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFE34CA for ; Sat, 3 Jun 2023 08:57:50 -0700 (PDT) Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6261d4ea5f0so24900506d6.0 for ; Sat, 03 Jun 2023 08:57:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685807870; x=1688399870; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h33Y8ZBTshpCakU2rwk1PXhPYpv5BZ6wPeCedcd+LbM=; b=HqJxv7MTfx2vxJuoictOp9oAhBWqaKHNffHvNahAyXIlbwZAxmJoFAzECyYUrdC4kF TOlmv5iQ530r0PLuSbi2cz62oty+ZmGT/JvuCxptkSOdOohZe2whFsQcrKEzaoucrAjd Yqf9tTYv8cuQmCDHeYe6Jj7slwkZ3qcHDtusAzg8wlmFSuCxDRr2tZeONu6gu3RdM39/ qJcFnrEUGS8xdojOiVIWd9sAQPQv4tcATJzuW2DCGNtSK568uO8wZ6uYWFh0JF0ntiVe /L9p4Qyf4g8PZgvlHmds27t8ULQJkeC+K/QX2s/1g5KQ2ZUMkCE7WdPxruNlv4MN4sV+ tDEw== X-Gm-Message-State: AC+VfDwcJljItG1xw5VKbe17rhbZISnFHHr+zmM8R0xCNyB9eII1oJKS bT1Kz+qo6HLnz2W2d7O9bNiP X-Google-Smtp-Source: ACHHUZ4wWtOFT5sPRaDxYAshqrzDTegg4zHApyL4QgQ9q20uQleDbcB0ClYa4RtM1+bM6Hq0UDX2uA== X-Received: by 2002:ad4:5d6c:0:b0:5e9:2bad:c8fa with SMTP id fn12-20020ad45d6c000000b005e92badc8famr2204971qvb.33.1685807869654; Sat, 03 Jun 2023 08:57:49 -0700 (PDT) Received: from localhost (pool-68-160-166-30.bstnma.fios.verizon.net. [68.160.166.30]) by smtp.gmail.com with ESMTPSA id m13-20020a05621402ad00b00623839cba8csm2292876qvv.44.2023.06.03.08.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 08:57:49 -0700 (PDT) Date: Sat, 3 Jun 2023 11:57:48 -0400 From: Mike Snitzer To: Dave Chinner Cc: Sarthak Kukreti , Jens Axboe , linux-block@vger.kernel.org, Joe Thornber , "Michael S. Tsirkin" , Jason Wang , "Darrick J. Wong" , Brian Foster , Bart Van Assche , linux-kernel@vger.kernel.org, Christoph Hellwig , dm-devel@redhat.com, Andreas Dilger , Stefan Hajnoczi , linux-fsdevel@vger.kernel.org, Theodore Ts'o , linux-ext4@vger.kernel.org, Joe Thornber , Alasdair Kergon Subject: Re: [PATCH v7 0/5] Introduce provisioning primitives Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Fri, Jun 02 2023 at 8:52P -0400, Dave Chinner wrote: > On Fri, Jun 02, 2023 at 11:44:27AM -0700, Sarthak Kukreti wrote: > > On Tue, May 30, 2023 at 8:28 AM Mike Snitzer wrote: > > > > > > On Tue, May 30 2023 at 10:55P -0400, > > > Joe Thornber wrote: > > > > > > > On Tue, May 30, 2023 at 3:02 PM Mike Snitzer wrote: > > > > > > > > > > > > > > Also Joe, for you proposed dm-thinp design where you distinquish > > > > > between "provision" and "reserve": Would it make sense for REQ_META > > > > > (e.g. all XFS metadata) with REQ_PROVISION to be treated as an > > > > > LBA-specific hard request? Whereas REQ_PROVISION on its own provides > > > > > more freedom to just reserve the length of blocks? (e.g. for XFS > > > > > delalloc where LBA range is unknown, but dm-thinp can be asked to > > > > > reserve space to accomodate it). > > > > > > > > > > > > > My proposal only involves 'reserve'. Provisioning will be done as part of > > > > the usual io path. > > > > > > OK, I think we'd do well to pin down the top-level block interfaces in > > > question. Because this patchset's block interface patch (2/5) header > > > says: > > > > > > "This patch also adds the capability to call fallocate() in mode 0 > > > on block devices, which will send REQ_OP_PROVISION to the block > > > device for the specified range," > > > > > > So it wires up blkdev_fallocate() to call blkdev_issue_provision(). A > > > user of XFS could then use fallocate() for user data -- which would > > > cause thinp's reserve to _not_ be used for critical metadata. > > Mike, I think you might have misunderstood what I have been proposing. > Possibly unintentionally, I didn't call it REQ_OP_PROVISION but > that's what I intended - the operation does not contain data at all. > It's an operation like REQ_OP_DISCARD or REQ_OP_WRITE_ZEROS - it > contains a range of sectors that need to be provisioned (or > discarded), and nothing else. No, I understood that. > The write IOs themselves are not tagged with anything special at all. I know, but I've been looking at how to also handle the delalloc usecase (and yes I know you feel it doesn't need handling, the issue is XFS does deal nicely with ensuring it has space when it tracks its allocations on "thick" storage -- so adding coordination between XFS and dm-thin layers provides comparable safety.. that safety is an expected norm). But rather than discuss in terms of data vs metadata, the distinction is: 1) LBA range reservation (normal case, your proposal) 2) non-LBA reservation (absolute value, LBA range is known at later stage) But I'm clearly going off script for dwelling on wanting to handle both. My looking at (ab)using REQ_META being set (use 1) vs not (use 2) was a crude simplification for branching between the 2 approaches. And I understand I made you nervous by expanding the scope to a much more muddled/shitty interface. ;) We all just need to focus on your proposal and Joe's dm-thin reservation design... [Sarthak: FYI, this implies that it doesn't really make sense to add dm-thinp support before Joe's design is implemented. Otherwise we'll have 2 different responses to REQ_OP_PROVISION. The one that is captured in your patchset isn't adequate to properly handle ensuring upper layer (like XFS) can depend on the space being available across snapshot boundaries.] > i.e. The proposal I made does not use REQ_PROVISION anywhere in the > metadata/data IO path; provisioned regions are created by separate > operations and must be tracked by the underlying block device, then > treat any write IO to those regions as "must not fail w/ ENOSPC" > IOs. > > There seems to be a lot of fear about user data requiring > provisioning. This is unfounded - provisioning is only needed for > explicitly provisioned space via fallocate(), not every byte of > user data written to the filesystem (the model Brian is proposing). As I mentioned above, I was just trying to get XFS-on-thinp to maintain parity with how XFS's delalloc accounting works on "thick" storage. But happy to put that to one side. Maintain focus like I mentioned above. I'm happy we have momentum and agreement on this design now. Rather than be content with that, I was mistakenly looking at other aspects and in doing so introduced "noise" before we've implemented what we all completely agree on: your and joe's designs. > Excessive use of fallocate() is self correcting - if users and/or > their applications provision too much, they are going to get ENOSPC > or have to pay more to expand the backing pool reserves they need. > But that's not a problem the block device should be trying to solve; > that's a problem for the sysadmin and/or bean counters to address. > > > > > > > The only way to distinquish the caller (between on-behalf of user data > > > vs XFS metadata) would be REQ_META? > > > > > > So should dm-thinp have a REQ_META-based distinction? Or just treat > > > all REQ_OP_PROVISION the same? > > > > > I'm in favor of a REQ_META-based distinction. > > Why? What *requirement* is driving the need for this distinction? Think I answered that above, XFS delalloc accounting parity on thinp. > As the person who proposed this new REQ_OP_PROVISION architecture, > I'm dead set against it. Allowing the block device provide a set of > poorly defined "conditional guarantees" policies instead of a > mechanism with a single ironclad guarantee defeats the entire > purpose of the proposal. > > We have a requirement from the *kernel ABI* that *user data writes* > must not fail with ENOSPC after an fallocate() operation. That's > one of the high level policies we need to implement. The filesystem > is already capable of guaranteeing it won't give the user ENOSPC > after fallocate, we now need a guarantee from the filesystem's > backing store that it won't give ENOSPC, too. Yes, I was trying to navigate Joe's reluctance to even support fallocate() for arbitrary user data. That's where the REQ_META vs data distinction crept in for me. But as you say: using fallocate() excessively is self-correcting. > The _other thing_ we need to implement is a method of guaranteeing > the filesystem won't shut down when the backing device goes ENOSPC > unexpected during metadata writeback. So we also need the backing > device to guarantee the regions we write metadata to won't give > ENOSPC. Yeap. > That's the whole point of REQ_OP_PROVISION: from the layers above > the block device, there is -zero- difference between the guarantee > we need for user data writes to avoid ENOSPC and for metadata writes > to avoid ENOSPC. They are one and the same. I know. The difference comes from delalloc initially needing an absolute value of reserve rather than a specific LBA range. > Hence if the block device is going to say "I support provisioning" > but then give different conditional guarantees according to the > *type of data* in the IO request, then it does not provide the > functionality the higher layers actually require from it. I was going for relaxing the "dynamic" approach (Brian's) to be best-effort -- and really only for XFS delalloc usecase. Every other usecase would respect your and Joe's vision. > Indeed, what type of data the IO contains is *context dependent*. > For example, sometimes we write metadata with user data IO and but > we still need provisioning guarantees as if it was issued as > metadata IO. This is the case for mkfs initialising the file system > by writing directly to the block device. I'm aware. > IOWs, filesystem metadata IO issued from kernel context would be > considered metadata IO, but from userspace it would be considered > normal user data IO and hence treated differently. But the reality > is that they both need the same provisioning guarantees to be > provided by the block device. What I was looking at is making the fallocate interface able to express: I need dave's requirements (bog-standard actually) vs I need non-LBA best effort. > So how do userspace tools deal with this if the block device > requires REQ_META on user data IOs to do the right thing here? And > if we provide a mechanism to allow this, how do we prevent userspace > for always using it on writes to fallocate() provisioned space? > > It's just not practical for the block device to add arbitrary > constraints based on the type of IO because we then have to add > mechanisms to userspace APIs to allow them to control the IO context > so the block device will do the right thing. Especially considering > we really only need one type of guarantee regardless of where the IO > originates from or what type of data the IO contains.... If anything my disposition on the conditional to require a REQ_META (or some fallocate generated REQ_UNSHARE ditto to reflect the same) to perform your approach to REQ_OP_PROVISION and honor fallocate() requirements is a big problem. Would be much better to have a flag to express "this reservation does not have an LBA range _yet_, nevertheless try to be mindful of this expected near-term block allocation". But I'll stop inlining repetitive (similar but different) answers to your concern now ;) > > Does that imply that > > REQ_META also needs to be passed through the block/filesystem stack > > (eg. REQ_OP_PROVION + REQ_META on a loop device translates to a > > fallocate() to the underlying file)? > > This is exactly the same case as above: the loopback device does > user data IO to the backing file. Hence we have another situation > where metadata IO is issued to fallocate()d user data ranges as user > data ranges and so would be given a lesser guarantee that would lead > to upper filesystem failure. BOth upper and lower filesystem data > and metadata need to be provided the same ENOSPC guarantees by their > backing stores.... > > The whole point of the REQ_OP_PROVISION proposal I made is that it > doesn't require any special handling in corner cases like this. > There are no cross-layer interactions needed to make everything work > correctly because the provisioning guarantee is not -data type > dependent*. The entire user IO path code remains untouched and > blissfully unaware of provisioned regions. > > And, realistically, if we have to start handling complex corner > cases in the filesystem and IO path layers to make REQ_OP_PROVISION > work correctly because of arbitary constraints imposed by the block > layer implementations, then we've failed miserably at the design and > architecture stage. > > Keep in mind that every attempt made so far to address the problems > with block device ENOSPC errors has failed because of the complexity > of the corner cases that have arisen during design and/or > implementation. It's pretty ironic that now we have a proposal that > is remarkably simple, free of corner cases and has virtually no > cross-layer coupling at all, the first thing that people want to do > is add arbitrary implementation constraints that result in complex > cross-layer corner cases that now need to be handled.... > > Put simply: if we restrict REQ_OP_PROVISION guarantees to just > REQ_META writes (or any other specific type of write operation) then > it's simply not worth persuing at the filesystem level because the > guarantees we actually need just aren't there and the complexity of > discovering and handling those corner cases just isn't worth the > effort. Here is where I get to say: I think you misunderstood me (but it was my fault for not being absolutely clear: I'm very much on the same page as you and Joe; and your visions need to just be implemented ASAP). I was taking your designs as a given, but looking further at: how do we also handle the non-LBA (delalloc) usecase _before_ we include REQ_OP_PROVISION in kernel. But I'm happy to let the delalloc case go (we can revisit addressing it if/when needed). Mike