From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 923A469D16 for ; Wed, 31 Jan 2024 14:34:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706711671; cv=none; b=AxhlSUOfms5lpMUp9f27/gcbRYuDi9plUXw1hCNQ08MHJs5gCVMVnhG4l6vXKjX89B6T6V4bDKlpdhWHpqeh1u8+UM/6BesbdqPIthVyF5Qdxmmy8PgjvbCR3N17tU9O6mLbz2lzOgoJJdgj32bbgGocKwsDVaPx2R/Z2JZba+8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706711671; c=relaxed/simple; bh=xjCEbBRhazJUQAffaemogB93qHXvZg8m+6w7ABq49A0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=i6FDDeXCFLi31s50cghHKfNU+7sQF8I/uVVuaEu9GvfDpNa0lJ+7ypXzT0eDDg1IAInBjCVphL7rYPZbdJQOgUxK27gI1y/FG3PWkUBXsXknjg2Wq0LeYIOAj/tJZUayx2HRl+kyNOiLz75hCHwHBUcdQUzosMn8cnsp/5IDtLc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4I0gElCc; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4I0gElCc" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-40ef9382752so56615e9.0 for ; Wed, 31 Jan 2024 06:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706711668; x=1707316468; darn=lists.linux.dev; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=JpjaYX2Bm7BNypekCmj2qKyZLKhzYmWDn0owCWw8KnY=; b=4I0gElCc6yf34+lbec6q+apCiIVv1nsRN9yIYUX+DtO9OgXkthaj64JZ1NDpPlQkfQ n9T1K07lwAfgSnW3zlrzHjktIVRd+gIE4LjozzrptExcpIE7HjhUFgxj46mXEuKJF8rg YxKrBKp2vch67tagKOaEeujVzrt2ZjyErKbhg1s4CtsKhaz6uL9IKEurJpB5aH4geBuY +C+YAdKBQpjQpYu434fSxGafRjH/VZ5rOIze+2RCAKVj9iBlFXvvm0DLH+m5F9wzzzjo DGMQSxeXfyrM11QjzJ8yqArH8ep9obzUYTL878qJmGz4IiYDw/s8Mog8syxq7qaxvtOi IIvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706711668; x=1707316468; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JpjaYX2Bm7BNypekCmj2qKyZLKhzYmWDn0owCWw8KnY=; b=BdiAUGC8VSKK9VWavtvhIYY3H+Up1wkpviPhffZ+DoQFssON+h7iTHEVfqOozOeR7C 9yuVJdPf+ybOaDk/hVD6pLSPB83gGk1y3mBtUZVNvngks1YxJ6A4xaVM9B5rsaN4c1av /QcK6FpScWCbmwrata+pneZIm6E3TDy707YPgLWxZfWUupwzG0GSAAa4QWrxFIawxfis vAzTKaqeB5EupfjePHnR9/PDJN9psGKFBfCh2g4DOSU3GW2bZ7DtIMpToWcNloVABnLi fUA6Buuz+3/1zE1pfu1eQBjhA6Mgh6iGFvLue01WFTSln8zSdegDJNRMNQ4DG52syuMR scvg== X-Gm-Message-State: AOJu0YyR7fLxLRq6dmtGSBgWfgqeOQF0deE3eqyCZy7FQSm9W3ieplw/ go7Fb7NZm569noOP4OQZfKkG2kjBgq8IUXLg7dzU0MsCEG5EQr4rmcXhW5T1hw== X-Google-Smtp-Source: AGHT+IETjNsaUhQTAF5CrEhZRptSbgf41YUIA1YCZ/JVABMZSk9VjGL6JUdJ/nYWV9+NkjbN/0k0gw== X-Received: by 2002:a05:600c:1e13:b0:40e:fa92:e52d with SMTP id ay19-20020a05600c1e1300b0040efa92e52dmr364828wmb.2.1706711667643; Wed, 31 Jan 2024 06:34:27 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWtjp2le7tqdnQyn1m9w13bXZBXt0N1qZjohCANaKQVKdhp/0Qr+3dpXDfscX6MIVA/sU7CtDePEANPYuVz1Xqldh0OVkKtotYbMRVRXGnSYxLelJETB8Q0fbE4PPKy9j8gccBh1QhRFU4Kc0JN1Ccme5WNHnoQTc2/G1w4Xs7seR0pWPvHzP+ZuyLh+lLVSSG7BLzwGfXlhEmLraOavME+b1k0+1EFXJbh+kokxwYgVkxBO/nBpsf7TzPkTDk7lukdio56ch42mb6jbDDs229rEmSZSfCPcHY6VF+hjKTwg5yBs6Jt5TEceFLPqxJidRokMCRGN126g7xEIJGDQ3NBzX6SpsyFXVMk8OFgA28D7Y+dWHIcZMBU Received: from google.com (185.83.140.34.bc.googleusercontent.com. [34.140.83.185]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c00cd00b0040f02114906sm1753035wmm.16.2024.01.31.06.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 06:34:27 -0800 (PST) Date: Wed, 31 Jan 2024 14:34:23 +0000 From: Mostafa Saleh To: Jason Gunthorpe Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum Subject: Re: [PATCH v4 01/16] iommu/arm-smmu-v3: Make STE programming independent of the callers Message-ID: References: <0-v4-c93b774edcc4+42d2b-smmuv3_newapi_p1_jgg@nvidia.com> <1-v4-c93b774edcc4+42d2b-smmuv3_newapi_p1_jgg@nvidia.com> <20240130235611.GF1455070@nvidia.com> Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240130235611.GF1455070@nvidia.com> On Tue, Jan 30, 2024 at 07:56:11PM -0400, Jason Gunthorpe wrote: > On Tue, Jan 30, 2024 at 10:42:13PM +0000, Mostafa Saleh wrote: > > > On Thu, Jan 25, 2024 at 07:57:11PM -0400, Jason Gunthorpe wrote: > > > As the comment in arm_smmu_write_strtab_ent() explains, this routine has > > > been limited to only work correctly in certain scenarios that the caller > > > must ensure. Generally the caller must put the STE into ABORT or BYPASS > > > before attempting to program it to something else. > > > > > > The iommu core APIs would ideally expect the driver to do a hitless change > > > of iommu_domain in a number of cases: > > > > > > - RESV_DIRECT support wants IDENTITY -> DMA -> IDENTITY to be hitless > > > for the RESV ranges > > > > > > - PASID upgrade has IDENTIY on the RID with no PASID then a PASID paging > > > domain installed. The RID should not be impacted > > > > > > - PASID downgrade has IDENTIY on the RID and all PASID's removed. > > > The RID should not be impacted > > > > > > - RID does PAGING -> BLOCKING with active PASID, PASID's should not be > > > impacted > > > > > > - NESTING -> NESTING for carrying all the above hitless cases in a VM > > > into the hypervisor. To comprehensively emulate the HW in a VM we should > > > assume the VM OS is running logic like this and expecting hitless updates > > > to be relayed to real HW. > > > > From my understanding, some of these cases are not implemented (at this point). > > However, from what I see, most of these cases are related to switching from/to > > identity, which the current driver would have to block in between, is my > > understanding correct? > > Basically > > > As for NESTING -> NESTING,  how is that achieved? (and why?) > > Through iommufd and it is necessary to reflect hitless transition from > the VM to the real HW. See VFIO_DEVICE_ATTACH_IOMMUFD_PT > > > AFAICT, VFIO will do BLOCKING in between any transition, and that domain > > should never change while the a device is assigned to a VM. > > It ultimately calls iommufd_device_replace() which avoids that. Old > vfio type1 users will force a blocking, but type1 will never support > nesting so it isn't relevant. > Thanks, I will check those. > > > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > > > index 0ffb1cf17e0b2e..690742e8f173eb 100644 > > > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > > > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c > > > @@ -48,6 +48,22 @@ enum arm_smmu_msi_index { > > > ARM_SMMU_MAX_MSIS, > > > }; > > > > > > +struct arm_smmu_entry_writer_ops; > > > +struct arm_smmu_entry_writer { > > > + const struct arm_smmu_entry_writer_ops *ops; > > > + struct arm_smmu_master *master; > > > > I see only master->smmu is used, is there a reason why we have this > > struct instead? > > The CD patches in part 2 requires the master because the CD entry > memory is shared across multiple CDs so we iterate the SID list inside > the update. The STE is the opposite, each STE has its own memory so we > iterate the SID list outside the update. > > > > +struct arm_smmu_entry_writer_ops { > > > + unsigned int num_entry_qwords; > > > + __le64 v_bit; > > > + void (*get_used)(struct arm_smmu_entry_writer *writer, const __le64 *entry, > > > + __le64 *used); > > > > *writer is not used in this series, I think it would make more sense if > > it's added in the patch that introduce using it. > > Ah, I guess, I think it is used in the test bench. > > > > + void (*sync)(struct arm_smmu_entry_writer *writer); > > > +}; > > > + > > > +#define NUM_ENTRY_QWORDS (sizeof(struct arm_smmu_ste) / sizeof(u64)) > > > + > > > > Isn't that just STRTAB_STE_DWORDS, also it makes more sense to not tie > > this to the struct but with the actual hardware description that would > > never change (but the struct can change) > > The struct and the HW description are the same. The struct size cannot > change. Broadly in the series STRTAB_STE_DWORDS is being dis-favoured > for sizeof(struct arm_smmu_ste) now that we have the struct. > > After part 3 there are only two references left to that constant, so I > will likely change part 3 to remove it. But arm_smmu_ste is defined based on STRTAB_STE_DWORDS. And this macro would never change as it is tied to the HW. However, in the future we can update “struct arm_smmu_ste” to hold a refcount for some reason, then sizeof(struct arm_smmu_ste) is not the size of the STE in the hardware. IMHO, any reference to the HW STE should be done using the macro. > > > +/* > > > + * Figure out if we can do a hitless update of entry to become target. Returns a > > > + * bit mask where 1 indicates that qword needs to be set disruptively. > > > + * unused_update is an intermediate value of entry that has unused bits set to > > > + * their new values. > > > + */ > > > +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, > > > + const __le64 *entry, const __le64 *target, > > > + __le64 *unused_update) > > > +{ > > > + __le64 target_used[NUM_ENTRY_QWORDS] = {}; > > > + __le64 cur_used[NUM_ENTRY_QWORDS] = {}; > > > + u8 used_qword_diff = 0; > > > + unsigned int i; > > > + > > > + writer->ops->get_used(writer, entry, cur_used); > > > + writer->ops->get_used(writer, target, target_used); > > > + > > > + for (i = 0; i != writer->ops->num_entry_qwords; i++) { > > > + /* > > > + * Check that masks are up to date, the make functions are not > > > + * allowed to set a bit to 1 if the used function doesn't say it > > > + * is used. > > > + */ > > > + WARN_ON_ONCE(target[i] & ~target_used[i]); > > > + > > > > I think this should be a BUG. As we don't know the consequence for such change, > > and this should never happen in a non-development kernel. > > Guidance from Linus is to never use BUG, always use WARN_ON and try to > recover. If people are running in a high-sensitivity production > environment they should set the warn on panic feature to ensure any > kernel self-detection of corruption triggers a halt. > > > > +/* > > > + * Update the STE/CD to the target configuration. The transition from the > > > + * current entry to the target entry takes place over multiple steps that > > > + * attempts to make the transition hitless if possible. This function takes care > > > + * not to create a situation where the HW can perceive a corrupted entry. HW is > > > + * only required to have a 64 bit atomicity with stores from the CPU, while > > > + * entries are many 64 bit values big. > > > + * > > > + * The difference between the current value and the target value is analyzed to > > > + * determine which of three updates are required - disruptive, hitless or no > > > + * change. > > > + * > > > + * In the most general disruptive case we can make any update in three steps: > > > + * - Disrupting the entry (V=0) > > > + * - Fill now unused qwords, execpt qword 0 which contains V > > > + * - Make qword 0 have the final value and valid (V=1) with a single 64 > > > + * bit store > > > + * > > > + * However this disrupts the HW while it is happening. There are several > > > + * interesting cases where a STE/CD can be updated without disturbing the HW > > > + * because only a small number of bits are changing (S1DSS, CONFIG, etc) or > > > + * because the used bits don't intersect. We can detect this by calculating how > > > + * many 64 bit values need update after adjusting the unused bits and skip the > > > + * V=0 process. This relies on the IGNORED behavior described in the > > > + * specification. > > > + */ > > > +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, > > > + __le64 *entry, const __le64 *target) > > > +{ > > > + unsigned int num_entry_qwords = writer->ops->num_entry_qwords; > > > + __le64 unused_update[NUM_ENTRY_QWORDS]; > > > + u8 used_qword_diff; > > > + > > > + used_qword_diff = > > > + arm_smmu_entry_qword_diff(writer, entry, target, unused_update); > > > + if (hweight8(used_qword_diff) > 1) { > > > + /* > > > + * At least two qwords need their inuse bits to be changed. This > > > + * requires a breaking update, zero the V bit, write all qwords > > > + * but 0, then set qword 0 > > > + */ > > > + unused_update[0] = entry[0] & (~writer->ops->v_bit); > > > + entry_set(writer, entry, unused_update, 0, 1); > > > + entry_set(writer, entry, target, 1, num_entry_qwords - 1); > > > + entry_set(writer, entry, target, 0, 1); > > > + } else if (hweight8(used_qword_diff) == 1) { > > > + /* > > > + * Only one qword needs its used bits to be changed. This is a > > > + * hitless update, update all bits the current STE is ignoring > > > + * to their new values, then update a single "critical qword" to > > > + * change the STE and finally 0 out any bits that are now unused > > > + * in the target configuration. > > > + */ > > > + unsigned int critical_qword_index = ffs(used_qword_diff) - 1; > > > + > > > + /* > > > + * Skip writing unused bits in the critical qword since we'll be > > > + * writing it in the next step anyways. This can save a sync > > > + * when the only change is in that qword. > > > + */ > > > + unused_update[critical_qword_index] = > > > + entry[critical_qword_index]; > > > + entry_set(writer, entry, unused_update, 0, num_entry_qwords); > > > + entry_set(writer, entry, target, critical_qword_index, 1); > > > + entry_set(writer, entry, target, 0, num_entry_qwords); > > > > The STE is updated in 3 steps. > > 1) Update all bits from target (except the changed qword) > > 2) Update the changed qword > > 3) Remove the bits that are not used by the target STE. > > > > In most cases we would issue a sync for 1) and 3) although the hardware ignores > > the updates, that seems necessary, am I missing something? > > "seems [un]necessary", right? Yes, that's a typo. > All syncs are necessary because the way the SMMU HW is permitted to > cache on a qword by qword basis. > > Eg with no sync after step 1 the HW cache could have: > > QW0 Not present > QW1 Step 0 (Current) > > And then instantly after step 2 updates DW0, but before it does the > sync, the HW is permited to read. Then it would have: > > QW0 Step 2 > QW1 Step 0 (Current) > > Which is illegal. The HW is allowed to observe a mix of Step[n] and > Step[n+1] only. Never a mix of Step[n-1] and Step[n+1]. > > The sync provides a barrier that prevents this. HW can never observe > the critical qword of step 2 without also observing only new values of > step 1. > > The same argument is for step 3 -> next step 1 on a future update. I see, thanks for the explanation. Thanks, Mostafa From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42F03C47258 for ; Wed, 31 Jan 2024 14:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cDTPKYspOeQ7RhZnBJSOIlnvUP36CAnlVDLo4aSILk4=; b=4wThteEO1kP4TP 7/+ubw9ZajdCuTl9jSbkUXr+BZVY8z2eub+B2Wa6dS/jQb9gxyow+b0ieY297yBXjb3DrX+Qu4NYG gFfsBPCKw+EdlOtJc7GHVWrdLXh+wqsCMuednLTF5aTc4hjEoyjb3jxQT96T9UHI2ktsuHfREfSRk /5Blxs5gfO2/ZUjaMf02RhFrOxXHxgf1DNkzk/+ym+DEoZV1LdN25lAfkBXM5os339DinhhNIenil hBm/3UckZryKdrUyoiQ5Y38YPt5ZrBJks2gbdWbGeNXVYk4nRfyUgMh5o0cxoG1wlk/tgWzi2XiYs bHIZZ+PtAAYgEdpaUvKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVBff-00000003v5c-2wjF; Wed, 31 Jan 2024 14:34:35 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVBfa-00000003v3m-2Su8 for linux-arm-kernel@lists.infradead.org; Wed, 31 Jan 2024 14:34:34 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-40f0218476aso40835e9.1 for ; Wed, 31 Jan 2024 06:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706711668; x=1707316468; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=JpjaYX2Bm7BNypekCmj2qKyZLKhzYmWDn0owCWw8KnY=; b=sTW9IpByzZiso/o4ZsreYFbjcXB1GL2IDKUZqhoxUFI0HsJ/Cfm/MKJ/AZ1ENwLPoZ 193wh3D0SYd+1jW3hEthx/PAr56t9uEBkO1WKAfOKA10JGqk5tLS1sa0MmVN4l/kFqsn x/UrAR6XV6EHFz4f7x1JjRzCiHgQ2R4+ftkoSom1VRj+cJoNENzesY4c0yLGBkoNnHUP 2JpEz0DDmOxOOFCaO806yhw+rWewcLgpvtWAfnlyO0fs7MAnPXvExnezLwsdUx3GyRm0 06Z5/SbLxE+xvtCCfXiQDI+5Sfe5KTbYbIHhWgiwh5+FC03QItYwiLqDvEclF1jZASjw zyzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706711668; x=1707316468; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JpjaYX2Bm7BNypekCmj2qKyZLKhzYmWDn0owCWw8KnY=; b=F+iSjtlhkG9Jcm8IlCDOoJxRWiOVkmv6ON20VWnuQ/PajSKyZO4AmOhMznCuA+eguA aZYU3s7yoeeE80M40GAr6eDjbmV+KnCZL7puppC9VTz+kcS09q/q5MSUNy/Q7Ky9rwOp X6pGZxejaf5YLf1yb9PmiGVEoxt4YqMCqkTIo3jkWXpz/4XWqNxJA2v60p9E9Cn3igST SDhNin/vdk3sQGUxuT2tGT/fM5XZmwYn/4FLo6wqPZsHxb2Zcs+4XrLt5tRuyCzwZX6n 11A9bui1Fl+rHyduYXm5YwAzJ/z3MPk9y+m5XNfoFAf52IbQb8/ScndMqzC+zxSX4XRY y45w== X-Gm-Message-State: AOJu0YyvmIrPfm3AgJL4EgoY3xAvDLaT5JT/se3RhF7CPv2RKYTG1MzE Cffg3WqHcX0JPdG4HnsENQbcToCyE3uh++oUN0fKd0H6Pw1VszdPvLwPOuuPOQ== X-Google-Smtp-Source: AGHT+IETjNsaUhQTAF5CrEhZRptSbgf41YUIA1YCZ/JVABMZSk9VjGL6JUdJ/nYWV9+NkjbN/0k0gw== X-Received: by 2002:a05:600c:1e13:b0:40e:fa92:e52d with SMTP id ay19-20020a05600c1e1300b0040efa92e52dmr364828wmb.2.1706711667643; Wed, 31 Jan 2024 06:34:27 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWtjp2le7tqdnQyn1m9w13bXZBXt0N1qZjohCANaKQVKdhp/0Qr+3dpXDfscX6MIVA/sU7CtDePEANPYuVz1Xqldh0OVkKtotYbMRVRXGnSYxLelJETB8Q0fbE4PPKy9j8gccBh1QhRFU4Kc0JN1Ccme5WNHnoQTc2/G1w4Xs7seR0pWPvHzP+ZuyLh+lLVSSG7BLzwGfXlhEmLraOavME+b1k0+1EFXJbh+kokxwYgVkxBO/nBpsf7TzPkTDk7lukdio56ch42mb6jbDDs229rEmSZSfCPcHY6VF+hjKTwg5yBs6Jt5TEceFLPqxJidRokMCRGN126g7xEIJGDQ3NBzX6SpsyFXVMk8OFgA28D7Y+dWHIcZMBU Received: from google.com (185.83.140.34.bc.googleusercontent.com. [34.140.83.185]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c00cd00b0040f02114906sm1753035wmm.16.2024.01.31.06.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 06:34:27 -0800 (PST) Date: Wed, 31 Jan 2024 14:34:23 +0000 From: Mostafa Saleh To: Jason Gunthorpe Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum Subject: Re: [PATCH v4 01/16] iommu/arm-smmu-v3: Make STE programming independent of the callers Message-ID: References: <0-v4-c93b774edcc4+42d2b-smmuv3_newapi_p1_jgg@nvidia.com> <1-v4-c93b774edcc4+42d2b-smmuv3_newapi_p1_jgg@nvidia.com> <20240130235611.GF1455070@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240130235611.GF1455070@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240131_063430_695444_0F591F68 X-CRM114-Status: GOOD ( 76.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBKYW4gMzAsIDIwMjQgYXQgMDc6NTY6MTFQTSAtMDQwMCwgSmFzb24gR3VudGhvcnBl IHdyb3RlOgo+IE9uIFR1ZSwgSmFuIDMwLCAyMDI0IGF0IDEwOjQyOjEzUE0gKzAwMDAsIE1vc3Rh ZmEgU2FsZWggd3JvdGU6Cj4gCj4gPiBPbiBUaHUsIEphbiAyNSwgMjAyNCBhdCAwNzo1NzoxMVBN IC0wNDAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6Cj4gPiA+IEFzIHRoZSBjb21tZW50IGluIGFy bV9zbW11X3dyaXRlX3N0cnRhYl9lbnQoKSBleHBsYWlucywgdGhpcyByb3V0aW5lIGhhcwo+ID4g PiBiZWVuIGxpbWl0ZWQgdG8gb25seSB3b3JrIGNvcnJlY3RseSBpbiBjZXJ0YWluIHNjZW5hcmlv cyB0aGF0IHRoZSBjYWxsZXIKPiA+ID4gbXVzdCBlbnN1cmUuIEdlbmVyYWxseSB0aGUgY2FsbGVy IG11c3QgcHV0IHRoZSBTVEUgaW50byBBQk9SVCBvciBCWVBBU1MKPiA+ID4gYmVmb3JlIGF0dGVt cHRpbmcgdG8gcHJvZ3JhbSBpdCB0byBzb21ldGhpbmcgZWxzZS4KPiA+ID4gCj4gPiA+IFRoZSBp b21tdSBjb3JlIEFQSXMgd291bGQgaWRlYWxseSBleHBlY3QgdGhlIGRyaXZlciB0byBkbyBhIGhp dGxlc3MgY2hhbmdlCj4gPiA+IG9mIGlvbW11X2RvbWFpbiBpbiBhIG51bWJlciBvZiBjYXNlczoK PiA+ID4gCj4gPiA+ICAtIFJFU1ZfRElSRUNUIHN1cHBvcnQgd2FudHMgSURFTlRJVFkgLT4gRE1B IC0+IElERU5USVRZIHRvIGJlIGhpdGxlc3MKPiA+ID4gICAgZm9yIHRoZSBSRVNWIHJhbmdlcwo+ ID4gPiAKPiA+ID4gIC0gUEFTSUQgdXBncmFkZSBoYXMgSURFTlRJWSBvbiB0aGUgUklEIHdpdGgg bm8gUEFTSUQgdGhlbiBhIFBBU0lEIHBhZ2luZwo+ID4gPiAgICBkb21haW4gaW5zdGFsbGVkLiBU aGUgUklEIHNob3VsZCBub3QgYmUgaW1wYWN0ZWQKPiA+ID4gCj4gPiA+ICAtIFBBU0lEIGRvd25n cmFkZSBoYXMgSURFTlRJWSBvbiB0aGUgUklEIGFuZCBhbGwgUEFTSUQncyByZW1vdmVkLgo+ID4g PiAgICBUaGUgUklEIHNob3VsZCBub3QgYmUgaW1wYWN0ZWQKPiA+ID4gCj4gPiA+ICAtIFJJRCBk b2VzIFBBR0lORyAtPiBCTE9DS0lORyB3aXRoIGFjdGl2ZSBQQVNJRCwgUEFTSUQncyBzaG91bGQg bm90IGJlCj4gPiA+ICAgIGltcGFjdGVkCj4gPiA+IAo+ID4gPiAgLSBORVNUSU5HIC0+IE5FU1RJ TkcgZm9yIGNhcnJ5aW5nIGFsbCB0aGUgYWJvdmUgaGl0bGVzcyBjYXNlcyBpbiBhIFZNCj4gPiA+ ICAgIGludG8gdGhlIGh5cGVydmlzb3IuIFRvIGNvbXByZWhlbnNpdmVseSBlbXVsYXRlIHRoZSBI VyBpbiBhIFZNIHdlIHNob3VsZAo+ID4gPiAgICBhc3N1bWUgdGhlIFZNIE9TIGlzIHJ1bm5pbmcg bG9naWMgbGlrZSB0aGlzIGFuZCBleHBlY3RpbmcgaGl0bGVzcyB1cGRhdGVzCj4gPiA+ICAgIHRv IGJlIHJlbGF5ZWQgdG8gcmVhbCBIVy4KPiA+IAo+ID4gRnJvbSBteSB1bmRlcnN0YW5kaW5nLCBz b21lIG9mIHRoZXNlIGNhc2VzIGFyZSBub3QgaW1wbGVtZW50ZWQgKGF0IHRoaXMgcG9pbnQpLgo+ ID4gSG93ZXZlciwgZnJvbSB3aGF0IEkgc2VlLCBtb3N0IG9mIHRoZXNlIGNhc2VzIGFyZSByZWxh dGVkIHRvIHN3aXRjaGluZyBmcm9tL3RvCj4gPiBpZGVudGl0eSwgd2hpY2ggdGhlIGN1cnJlbnQg ZHJpdmVyIHdvdWxkIGhhdmUgdG8gYmxvY2sgaW4gYmV0d2VlbiwgaXMgbXkKPiA+IHVuZGVyc3Rh bmRpbmcgY29ycmVjdD8KPiAKPiBCYXNpY2FsbHkKPiAKPiA+IEFzIGZvciBORVNUSU5HIC0+IE5F U1RJTkcsIMKgaG93IGlzIHRoYXQgYWNoaWV2ZWQ/IChhbmQgd2h5PykKPiAKPiBUaHJvdWdoIGlv bW11ZmQgYW5kIGl0IGlzIG5lY2Vzc2FyeSB0byByZWZsZWN0IGhpdGxlc3MgdHJhbnNpdGlvbiBm cm9tCj4gdGhlIFZNIHRvIHRoZSByZWFsIEhXLiBTZWUgVkZJT19ERVZJQ0VfQVRUQUNIX0lPTU1V RkRfUFQKPiAKPiA+IEFGQUlDVCwgVkZJTyB3aWxsIGRvIEJMT0NLSU5HIGluIGJldHdlZW4gYW55 IHRyYW5zaXRpb24sIGFuZCB0aGF0IGRvbWFpbgo+ID4gc2hvdWxkIG5ldmVyIGNoYW5nZSB3aGls ZSB0aGUgYSBkZXZpY2UgaXMgYXNzaWduZWQgdG8gYSBWTS4KPiAKPiBJdCB1bHRpbWF0ZWx5IGNh bGxzIGlvbW11ZmRfZGV2aWNlX3JlcGxhY2UoKSB3aGljaCBhdm9pZHMgdGhhdC4gT2xkCj4gdmZp byB0eXBlMSB1c2VycyB3aWxsIGZvcmNlIGEgYmxvY2tpbmcsIGJ1dCB0eXBlMSB3aWxsIG5ldmVy IHN1cHBvcnQKPiBuZXN0aW5nIHNvIGl0IGlzbid0IHJlbGV2YW50Lgo+ClRoYW5rcywgSSB3aWxs IGNoZWNrIHRob3NlLgo+ID4gPiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9pb21tdS9hcm0vYXJtLXNt bXUtdjMvYXJtLXNtbXUtdjMuYyBiL2RyaXZlcnMvaW9tbXUvYXJtL2FybS1zbW11LXYzL2FybS1z bW11LXYzLmMKPiA+ID4gaW5kZXggMGZmYjFjZjE3ZTBiMmUuLjY5MDc0MmU4ZjE3M2ViIDEwMDY0 NAo+ID4gPiAtLS0gYS9kcml2ZXJzL2lvbW11L2FybS9hcm0tc21tdS12My9hcm0tc21tdS12My5j Cj4gPiA+ICsrKyBiL2RyaXZlcnMvaW9tbXUvYXJtL2FybS1zbW11LXYzL2FybS1zbW11LXYzLmMK PiA+ID4gQEAgLTQ4LDYgKzQ4LDIyIEBAIGVudW0gYXJtX3NtbXVfbXNpX2luZGV4IHsKPiA+ID4g IAlBUk1fU01NVV9NQVhfTVNJUywKPiA+ID4gIH07Cj4gPiA+ICAKPiA+ID4gK3N0cnVjdCBhcm1f c21tdV9lbnRyeV93cml0ZXJfb3BzOwo+ID4gPiArc3RydWN0IGFybV9zbW11X2VudHJ5X3dyaXRl ciB7Cj4gPiA+ICsJY29uc3Qgc3RydWN0IGFybV9zbW11X2VudHJ5X3dyaXRlcl9vcHMgKm9wczsK PiA+ID4gKwlzdHJ1Y3QgYXJtX3NtbXVfbWFzdGVyICptYXN0ZXI7Cj4gPiAKPiA+IEkgc2VlIG9u bHkgbWFzdGVyLT5zbW11IGlzIHVzZWQsIGlzIHRoZXJlIGEgcmVhc29uIHdoeSB3ZSBoYXZlIHRo aXMKPiA+IHN0cnVjdCBpbnN0ZWFkPwo+IAo+IFRoZSBDRCBwYXRjaGVzIGluIHBhcnQgMiByZXF1 aXJlcyB0aGUgbWFzdGVyIGJlY2F1c2UgdGhlIENEIGVudHJ5Cj4gbWVtb3J5IGlzIHNoYXJlZCBh Y3Jvc3MgbXVsdGlwbGUgQ0RzIHNvIHdlIGl0ZXJhdGUgdGhlIFNJRCBsaXN0IGluc2lkZQo+IHRo ZSB1cGRhdGUuIFRoZSBTVEUgaXMgdGhlIG9wcG9zaXRlLCBlYWNoIFNURSBoYXMgaXRzIG93biBt ZW1vcnkgc28gd2UKPiBpdGVyYXRlIHRoZSBTSUQgbGlzdCBvdXRzaWRlIHRoZSB1cGRhdGUuCj4g Cj4gPiA+ICtzdHJ1Y3QgYXJtX3NtbXVfZW50cnlfd3JpdGVyX29wcyB7Cj4gPiA+ICsJdW5zaWdu ZWQgaW50IG51bV9lbnRyeV9xd29yZHM7Cj4gPiA+ICsJX19sZTY0IHZfYml0Owo+ID4gPiArCXZv aWQgKCpnZXRfdXNlZCkoc3RydWN0IGFybV9zbW11X2VudHJ5X3dyaXRlciAqd3JpdGVyLCBjb25z dCBfX2xlNjQgKmVudHJ5LAo+ID4gPiArCQkJIF9fbGU2NCAqdXNlZCk7Cj4gPiAKPiA+ICp3cml0 ZXIgaXMgbm90IHVzZWQgaW4gdGhpcyBzZXJpZXMsIEkgdGhpbmsgaXQgd291bGQgbWFrZSBtb3Jl IHNlbnNlIGlmCj4gPiBpdCdzIGFkZGVkIGluIHRoZSBwYXRjaCB0aGF0IGludHJvZHVjZSB1c2lu ZyBpdC4KPiAKPiBBaCwgSSBndWVzcywgSSB0aGluayBpdCBpcyB1c2VkIGluIHRoZSB0ZXN0IGJl bmNoLgo+ICAKPiA+ID4gKwl2b2lkICgqc3luYykoc3RydWN0IGFybV9zbW11X2VudHJ5X3dyaXRl ciAqd3JpdGVyKTsKPiA+ID4gK307Cj4gPiA+ICsKPiA+ID4gKyNkZWZpbmUgTlVNX0VOVFJZX1FX T1JEUyAoc2l6ZW9mKHN0cnVjdCBhcm1fc21tdV9zdGUpIC8gc2l6ZW9mKHU2NCkpCj4gPiA+ICsK PiA+IAo+ID4gSXNuJ3QgdGhhdCBqdXN0IFNUUlRBQl9TVEVfRFdPUkRTLCBhbHNvIGl0IG1ha2Vz IG1vcmUgc2Vuc2UgdG8gbm90IHRpZQo+ID4gdGhpcyB0byB0aGUgc3RydWN0IGJ1dCB3aXRoIHRo ZSBhY3R1YWwgaGFyZHdhcmUgZGVzY3JpcHRpb24gdGhhdCB3b3VsZAo+ID4gbmV2ZXIgY2hhbmdl IChidXQgdGhlIHN0cnVjdCBjYW4gY2hhbmdlKQo+IAo+IFRoZSBzdHJ1Y3QgYW5kIHRoZSBIVyBk ZXNjcmlwdGlvbiBhcmUgdGhlIHNhbWUuIFRoZSBzdHJ1Y3Qgc2l6ZSBjYW5ub3QKPiBjaGFuZ2Uu IEJyb2FkbHkgaW4gdGhlIHNlcmllcyBTVFJUQUJfU1RFX0RXT1JEUyBpcyBiZWluZyBkaXMtZmF2 b3VyZWQKPiBmb3Igc2l6ZW9mKHN0cnVjdCBhcm1fc21tdV9zdGUpIG5vdyB0aGF0IHdlIGhhdmUg dGhlIHN0cnVjdC4KPiAKPiBBZnRlciBwYXJ0IDMgdGhlcmUgYXJlIG9ubHkgdHdvIHJlZmVyZW5j ZXMgbGVmdCB0byB0aGF0IGNvbnN0YW50LCBzbyBJCj4gd2lsbCBsaWtlbHkgY2hhbmdlIHBhcnQg MyB0byByZW1vdmUgaXQuCgpCdXQgYXJtX3NtbXVfc3RlIGlzIGRlZmluZWQgYmFzZWQgb24gU1RS VEFCX1NURV9EV09SRFMuIEFuZCB0aGlzIG1hY3JvIHdvdWxkCm5ldmVyIGNoYW5nZSBhcyBpdCBp cyB0aWVkIHRvIHRoZSBIVy4gSG93ZXZlciwgaW4gdGhlIGZ1dHVyZSB3ZSBjYW4gdXBkYXRlCuKA nHN0cnVjdCBhcm1fc21tdV9zdGXigJ0gdG8gaG9sZCBhIHJlZmNvdW50IGZvciBzb21lIHJlYXNv biwKdGhlbiBzaXplb2Yoc3RydWN0IGFybV9zbW11X3N0ZSkgaXMgbm90IHRoZSBzaXplIG9mIHRo ZSBTVEUgaW4gdGhlIGhhcmR3YXJlLgpJTUhPLCBhbnkgcmVmZXJlbmNlIHRvIHRoZSBIVyBTVEUg c2hvdWxkIGJlIGRvbmUgdXNpbmcgdGhlIG1hY3JvLgoKPiA+ID4gKy8qCj4gPiA+ICsgKiBGaWd1 cmUgb3V0IGlmIHdlIGNhbiBkbyBhIGhpdGxlc3MgdXBkYXRlIG9mIGVudHJ5IHRvIGJlY29tZSB0 YXJnZXQuIFJldHVybnMgYQo+ID4gPiArICogYml0IG1hc2sgd2hlcmUgMSBpbmRpY2F0ZXMgdGhh dCBxd29yZCBuZWVkcyB0byBiZSBzZXQgZGlzcnVwdGl2ZWx5Lgo+ID4gPiArICogdW51c2VkX3Vw ZGF0ZSBpcyBhbiBpbnRlcm1lZGlhdGUgdmFsdWUgb2YgZW50cnkgdGhhdCBoYXMgdW51c2VkIGJp dHMgc2V0IHRvCj4gPiA+ICsgKiB0aGVpciBuZXcgdmFsdWVzLgo+ID4gPiArICovCj4gPiA+ICtz dGF0aWMgdTggYXJtX3NtbXVfZW50cnlfcXdvcmRfZGlmZihzdHJ1Y3QgYXJtX3NtbXVfZW50cnlf d3JpdGVyICp3cml0ZXIsCj4gPiA+ICsJCQkJICAgIGNvbnN0IF9fbGU2NCAqZW50cnksIGNvbnN0 IF9fbGU2NCAqdGFyZ2V0LAo+ID4gPiArCQkJCSAgICBfX2xlNjQgKnVudXNlZF91cGRhdGUpCj4g PiA+ICt7Cj4gPiA+ICsJX19sZTY0IHRhcmdldF91c2VkW05VTV9FTlRSWV9RV09SRFNdID0ge307 Cj4gPiA+ICsJX19sZTY0IGN1cl91c2VkW05VTV9FTlRSWV9RV09SRFNdID0ge307Cj4gPiA+ICsJ dTggdXNlZF9xd29yZF9kaWZmID0gMDsKPiA+ID4gKwl1bnNpZ25lZCBpbnQgaTsKPiA+ID4gKwo+ ID4gPiArCXdyaXRlci0+b3BzLT5nZXRfdXNlZCh3cml0ZXIsIGVudHJ5LCBjdXJfdXNlZCk7Cj4g PiA+ICsJd3JpdGVyLT5vcHMtPmdldF91c2VkKHdyaXRlciwgdGFyZ2V0LCB0YXJnZXRfdXNlZCk7 Cj4gPiA+ICsKPiA+ID4gKwlmb3IgKGkgPSAwOyBpICE9IHdyaXRlci0+b3BzLT5udW1fZW50cnlf cXdvcmRzOyBpKyspIHsKPiA+ID4gKwkJLyoKPiA+ID4gKwkJICogQ2hlY2sgdGhhdCBtYXNrcyBh cmUgdXAgdG8gZGF0ZSwgdGhlIG1ha2UgZnVuY3Rpb25zIGFyZSBub3QKPiA+ID4gKwkJICogYWxs b3dlZCB0byBzZXQgYSBiaXQgdG8gMSBpZiB0aGUgdXNlZCBmdW5jdGlvbiBkb2Vzbid0IHNheSBp dAo+ID4gPiArCQkgKiBpcyB1c2VkLgo+ID4gPiArCQkgKi8KPiA+ID4gKwkJV0FSTl9PTl9PTkNF KHRhcmdldFtpXSAmIH50YXJnZXRfdXNlZFtpXSk7Cj4gPiA+ICsKPiA+IAo+ID4gSSB0aGluayB0 aGlzIHNob3VsZCBiZSBhIEJVRy4gQXMgd2UgZG9uJ3Qga25vdyB0aGUgY29uc2VxdWVuY2UgZm9y IHN1Y2ggY2hhbmdlLAo+ID4gYW5kIHRoaXMgc2hvdWxkIG5ldmVyIGhhcHBlbiBpbiBhIG5vbi1k ZXZlbG9wbWVudCBrZXJuZWwuCj4gCj4gR3VpZGFuY2UgZnJvbSBMaW51cyBpcyB0byBuZXZlciB1 c2UgQlVHLCBhbHdheXMgdXNlIFdBUk5fT04gYW5kIHRyeSB0bwo+IHJlY292ZXIuIElmIHBlb3Bs ZSBhcmUgcnVubmluZyBpbiBhIGhpZ2gtc2Vuc2l0aXZpdHkgcHJvZHVjdGlvbgo+IGVudmlyb25t ZW50IHRoZXkgc2hvdWxkIHNldCB0aGUgd2FybiBvbiBwYW5pYyBmZWF0dXJlIHRvIGVuc3VyZSBh bnkKPiBrZXJuZWwgc2VsZi1kZXRlY3Rpb24gb2YgY29ycnVwdGlvbiB0cmlnZ2VycyBhIGhhbHQu Cj4gCj4gPiA+ICsvKgo+ID4gPiArICogVXBkYXRlIHRoZSBTVEUvQ0QgdG8gdGhlIHRhcmdldCBj b25maWd1cmF0aW9uLiBUaGUgdHJhbnNpdGlvbiBmcm9tIHRoZQo+ID4gPiArICogY3VycmVudCBl bnRyeSB0byB0aGUgdGFyZ2V0IGVudHJ5IHRha2VzIHBsYWNlIG92ZXIgbXVsdGlwbGUgc3RlcHMg dGhhdAo+ID4gPiArICogYXR0ZW1wdHMgdG8gbWFrZSB0aGUgdHJhbnNpdGlvbiBoaXRsZXNzIGlm IHBvc3NpYmxlLiBUaGlzIGZ1bmN0aW9uIHRha2VzIGNhcmUKPiA+ID4gKyAqIG5vdCB0byBjcmVh dGUgYSBzaXR1YXRpb24gd2hlcmUgdGhlIEhXIGNhbiBwZXJjZWl2ZSBhIGNvcnJ1cHRlZCBlbnRy eS4gSFcgaXMKPiA+ID4gKyAqIG9ubHkgcmVxdWlyZWQgdG8gaGF2ZSBhIDY0IGJpdCBhdG9taWNp dHkgd2l0aCBzdG9yZXMgZnJvbSB0aGUgQ1BVLCB3aGlsZQo+ID4gPiArICogZW50cmllcyBhcmUg bWFueSA2NCBiaXQgdmFsdWVzIGJpZy4KPiA+ID4gKyAqCj4gPiA+ICsgKiBUaGUgZGlmZmVyZW5j ZSBiZXR3ZWVuIHRoZSBjdXJyZW50IHZhbHVlIGFuZCB0aGUgdGFyZ2V0IHZhbHVlIGlzIGFuYWx5 emVkIHRvCj4gPiA+ICsgKiBkZXRlcm1pbmUgd2hpY2ggb2YgdGhyZWUgdXBkYXRlcyBhcmUgcmVx dWlyZWQgLSBkaXNydXB0aXZlLCBoaXRsZXNzIG9yIG5vCj4gPiA+ICsgKiBjaGFuZ2UuCj4gPiA+ ICsgKgo+ID4gPiArICogSW4gdGhlIG1vc3QgZ2VuZXJhbCBkaXNydXB0aXZlIGNhc2Ugd2UgY2Fu IG1ha2UgYW55IHVwZGF0ZSBpbiB0aHJlZSBzdGVwczoKPiA+ID4gKyAqICAtIERpc3J1cHRpbmcg dGhlIGVudHJ5IChWPTApCj4gPiA+ICsgKiAgLSBGaWxsIG5vdyB1bnVzZWQgcXdvcmRzLCBleGVj cHQgcXdvcmQgMCB3aGljaCBjb250YWlucyBWCj4gPiA+ICsgKiAgLSBNYWtlIHF3b3JkIDAgaGF2 ZSB0aGUgZmluYWwgdmFsdWUgYW5kIHZhbGlkIChWPTEpIHdpdGggYSBzaW5nbGUgNjQKPiA+ID4g KyAqICAgIGJpdCBzdG9yZQo+ID4gPiArICoKPiA+ID4gKyAqIEhvd2V2ZXIgdGhpcyBkaXNydXB0 cyB0aGUgSFcgd2hpbGUgaXQgaXMgaGFwcGVuaW5nLiBUaGVyZSBhcmUgc2V2ZXJhbAo+ID4gPiAr ICogaW50ZXJlc3RpbmcgY2FzZXMgd2hlcmUgYSBTVEUvQ0QgY2FuIGJlIHVwZGF0ZWQgd2l0aG91 dCBkaXN0dXJiaW5nIHRoZSBIVwo+ID4gPiArICogYmVjYXVzZSBvbmx5IGEgc21hbGwgbnVtYmVy IG9mIGJpdHMgYXJlIGNoYW5naW5nIChTMURTUywgQ09ORklHLCBldGMpIG9yCj4gPiA+ICsgKiBi ZWNhdXNlIHRoZSB1c2VkIGJpdHMgZG9uJ3QgaW50ZXJzZWN0LiBXZSBjYW4gZGV0ZWN0IHRoaXMg YnkgY2FsY3VsYXRpbmcgaG93Cj4gPiA+ICsgKiBtYW55IDY0IGJpdCB2YWx1ZXMgbmVlZCB1cGRh dGUgYWZ0ZXIgYWRqdXN0aW5nIHRoZSB1bnVzZWQgYml0cyBhbmQgc2tpcCB0aGUKPiA+ID4gKyAq IFY9MCBwcm9jZXNzLiBUaGlzIHJlbGllcyBvbiB0aGUgSUdOT1JFRCBiZWhhdmlvciBkZXNjcmli ZWQgaW4gdGhlCj4gPiA+ICsgKiBzcGVjaWZpY2F0aW9uLgo+ID4gPiArICovCj4gPiA+ICtzdGF0 aWMgdm9pZCBhcm1fc21tdV93cml0ZV9lbnRyeShzdHJ1Y3QgYXJtX3NtbXVfZW50cnlfd3JpdGVy ICp3cml0ZXIsCj4gPiA+ICsJCQkJIF9fbGU2NCAqZW50cnksIGNvbnN0IF9fbGU2NCAqdGFyZ2V0 KQo+ID4gPiArewo+ID4gPiArCXVuc2lnbmVkIGludCBudW1fZW50cnlfcXdvcmRzID0gd3JpdGVy LT5vcHMtPm51bV9lbnRyeV9xd29yZHM7Cj4gPiA+ICsJX19sZTY0IHVudXNlZF91cGRhdGVbTlVN X0VOVFJZX1FXT1JEU107Cj4gPiA+ICsJdTggdXNlZF9xd29yZF9kaWZmOwo+ID4gPiArCj4gPiA+ ICsJdXNlZF9xd29yZF9kaWZmID0KPiA+ID4gKwkJYXJtX3NtbXVfZW50cnlfcXdvcmRfZGlmZih3 cml0ZXIsIGVudHJ5LCB0YXJnZXQsIHVudXNlZF91cGRhdGUpOwo+ID4gPiArCWlmIChod2VpZ2h0 OCh1c2VkX3F3b3JkX2RpZmYpID4gMSkgewo+ID4gPiArCQkvKgo+ID4gPiArCQkgKiBBdCBsZWFz dCB0d28gcXdvcmRzIG5lZWQgdGhlaXIgaW51c2UgYml0cyB0byBiZSBjaGFuZ2VkLiBUaGlzCj4g PiA+ICsJCSAqIHJlcXVpcmVzIGEgYnJlYWtpbmcgdXBkYXRlLCB6ZXJvIHRoZSBWIGJpdCwgd3Jp dGUgYWxsIHF3b3Jkcwo+ID4gPiArCQkgKiBidXQgMCwgdGhlbiBzZXQgcXdvcmQgMAo+ID4gPiAr CQkgKi8KPiA+ID4gKwkJdW51c2VkX3VwZGF0ZVswXSA9IGVudHJ5WzBdICYgKH53cml0ZXItPm9w cy0+dl9iaXQpOwo+ID4gPiArCQllbnRyeV9zZXQod3JpdGVyLCBlbnRyeSwgdW51c2VkX3VwZGF0 ZSwgMCwgMSk7Cj4gPiA+ICsJCWVudHJ5X3NldCh3cml0ZXIsIGVudHJ5LCB0YXJnZXQsIDEsIG51 bV9lbnRyeV9xd29yZHMgLSAxKTsKPiA+ID4gKwkJZW50cnlfc2V0KHdyaXRlciwgZW50cnksIHRh cmdldCwgMCwgMSk7Cj4gPiA+ICsJfSBlbHNlIGlmIChod2VpZ2h0OCh1c2VkX3F3b3JkX2RpZmYp ID09IDEpIHsKPiA+ID4gKwkJLyoKPiA+ID4gKwkJICogT25seSBvbmUgcXdvcmQgbmVlZHMgaXRz IHVzZWQgYml0cyB0byBiZSBjaGFuZ2VkLiBUaGlzIGlzIGEKPiA+ID4gKwkJICogaGl0bGVzcyB1 cGRhdGUsIHVwZGF0ZSBhbGwgYml0cyB0aGUgY3VycmVudCBTVEUgaXMgaWdub3JpbmcKPiA+ID4g KwkJICogdG8gdGhlaXIgbmV3IHZhbHVlcywgdGhlbiB1cGRhdGUgYSBzaW5nbGUgImNyaXRpY2Fs IHF3b3JkIiB0bwo+ID4gPiArCQkgKiBjaGFuZ2UgdGhlIFNURSBhbmQgZmluYWxseSAwIG91dCBh bnkgYml0cyB0aGF0IGFyZSBub3cgdW51c2VkCj4gPiA+ICsJCSAqIGluIHRoZSB0YXJnZXQgY29u ZmlndXJhdGlvbi4KPiA+ID4gKwkJICovCj4gPiA+ICsJCXVuc2lnbmVkIGludCBjcml0aWNhbF9x d29yZF9pbmRleCA9IGZmcyh1c2VkX3F3b3JkX2RpZmYpIC0gMTsKPiA+ID4gKwo+ID4gPiArCQkv Kgo+ID4gPiArCQkgKiBTa2lwIHdyaXRpbmcgdW51c2VkIGJpdHMgaW4gdGhlIGNyaXRpY2FsIHF3 b3JkIHNpbmNlIHdlJ2xsIGJlCj4gPiA+ICsJCSAqIHdyaXRpbmcgaXQgaW4gdGhlIG5leHQgc3Rl cCBhbnl3YXlzLiBUaGlzIGNhbiBzYXZlIGEgc3luYwo+ID4gPiArCQkgKiB3aGVuIHRoZSBvbmx5 IGNoYW5nZSBpcyBpbiB0aGF0IHF3b3JkLgo+ID4gPiArCQkgKi8KPiA+ID4gKwkJdW51c2VkX3Vw ZGF0ZVtjcml0aWNhbF9xd29yZF9pbmRleF0gPQo+ID4gPiArCQkJZW50cnlbY3JpdGljYWxfcXdv cmRfaW5kZXhdOwo+ID4gPiArCQllbnRyeV9zZXQod3JpdGVyLCBlbnRyeSwgdW51c2VkX3VwZGF0 ZSwgMCwgbnVtX2VudHJ5X3F3b3Jkcyk7Cj4gPiA+ICsJCWVudHJ5X3NldCh3cml0ZXIsIGVudHJ5 LCB0YXJnZXQsIGNyaXRpY2FsX3F3b3JkX2luZGV4LCAxKTsKPiA+ID4gKwkJZW50cnlfc2V0KHdy aXRlciwgZW50cnksIHRhcmdldCwgMCwgbnVtX2VudHJ5X3F3b3Jkcyk7Cj4gPiAKPiA+IFRoZSBT VEUgaXMgdXBkYXRlZCBpbiAzIHN0ZXBzLgo+ID4gMSkgVXBkYXRlIGFsbCBiaXRzIGZyb20gdGFy Z2V0IChleGNlcHQgdGhlIGNoYW5nZWQgcXdvcmQpCj4gPiAyKSBVcGRhdGUgdGhlIGNoYW5nZWQg cXdvcmQKPiA+IDMpIFJlbW92ZSB0aGUgYml0cyB0aGF0IGFyZSBub3QgdXNlZCBieSB0aGUgdGFy Z2V0IFNURS4KPiA+IAo+ID4gSW4gbW9zdCBjYXNlcyB3ZSB3b3VsZCBpc3N1ZSBhIHN5bmMgZm9y IDEpIGFuZCAzKSBhbHRob3VnaCB0aGUgaGFyZHdhcmUgaWdub3Jlcwo+ID4gdGhlIHVwZGF0ZXMs IHRoYXQgc2VlbXMgbmVjZXNzYXJ5LCBhbSBJIG1pc3Npbmcgc29tZXRoaW5nPwo+IAo+ICJzZWVt cyBbdW5dbmVjZXNzYXJ5IiwgcmlnaHQ/ClllcywgdGhhdCdzIGEgdHlwby4KCj4gQWxsIHN5bmNz IGFyZSBuZWNlc3NhcnkgYmVjYXVzZSB0aGUgd2F5IHRoZSBTTU1VIEhXIGlzIHBlcm1pdHRlZCB0 bwo+IGNhY2hlIG9uIGEgcXdvcmQgYnkgcXdvcmQgYmFzaXMuCj4gCj4gRWcgd2l0aCBubyBzeW5j IGFmdGVyIHN0ZXAgMSB0aGUgSFcgY2FjaGUgY291bGQgaGF2ZToKPiAKPiAgIFFXMCBOb3QgcHJl c2VudAo+ICAgUVcxIFN0ZXAgMCAoQ3VycmVudCkKPiAKPiBBbmQgdGhlbiBpbnN0YW50bHkgYWZ0 ZXIgc3RlcCAyIHVwZGF0ZXMgRFcwLCBidXQgYmVmb3JlIGl0IGRvZXMgdGhlCj4gc3luYywgdGhl IEhXIGlzIHBlcm1pdGVkIHRvIHJlYWQuIFRoZW4gaXQgd291bGQgaGF2ZToKPiAKPiAgIFFXMCBT dGVwIDIKPiAgIFFXMSBTdGVwIDAgKEN1cnJlbnQpCj4gCj4gV2hpY2ggaXMgaWxsZWdhbC4gVGhl IEhXIGlzIGFsbG93ZWQgdG8gb2JzZXJ2ZSBhIG1peCBvZiBTdGVwW25dIGFuZAo+IFN0ZXBbbisx XSBvbmx5LiBOZXZlciBhIG1peCBvZiBTdGVwW24tMV0gYW5kIFN0ZXBbbisxXS4KPiAKPiBUaGUg c3luYyBwcm92aWRlcyBhIGJhcnJpZXIgdGhhdCBwcmV2ZW50cyB0aGlzLiBIVyBjYW4gbmV2ZXIg b2JzZXJ2ZQo+IHRoZSBjcml0aWNhbCBxd29yZCBvZiBzdGVwIDIgd2l0aG91dCBhbHNvIG9ic2Vy dmluZyBvbmx5IG5ldyB2YWx1ZXMgb2YKPiBzdGVwIDEuCj4gCj4gVGhlIHNhbWUgYXJndW1lbnQg aXMgZm9yIHN0ZXAgMyAtPiBuZXh0IHN0ZXAgMSBvbiBhIGZ1dHVyZSB1cGRhdGUuCgpJIHNlZSwg dGhhbmtzIGZvciB0aGUgZXhwbGFuYXRpb24uCgpUaGFua3MsCk1vc3RhZmEKCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFp bGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlz dHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK