From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5FF93195FD for ; Fri, 21 Aug 2026 02:21:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787278863; cv=none; b=LQmqJ/sU/SfQ1c/EShXXii08mYrAPX2GLomRc1hWtj2U0kPk0lgN5lbpEyRlvOKD3lNw8fXywpa8qZgmeXZMAlk6Mra4Q07BnqpdOEU5NTRKHz6LvK9eF7sSKBvNarnNKULA1fWvCCtXEJGjGNjcpV6cx4xNSTt3BK6Rf7G4lj4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787278863; c=relaxed/simple; bh=ODEamyaNinz8ndZY25u31wy9kg0XuQk5tLuO7i1Q1AI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Sk3wc8kCuGUhu9+i4pQhYHn8dW2bEOHT3eX/5M3ebJxT8uEloLoT16TQyngGOqdM8UMlPu2CZSO12qZcsU+j0U1Xya/CsVca/rQddExnE7jcelGN7ZczvozkBklt4+68QYdi3dd1+seUC3xx4tUPAMoRpU5HzQocHiPsuk879d4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MnrbBEe2; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MnrbBEe2" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2cede6375caso48165ad.0 for ; Thu, 20 Aug 2026 19:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787278861; x=1787883661; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=vTISns22lLz3uE6OmdCyn6vQB2Si1JaLhFBQroGqygQ=; b=MnrbBEe2gHTV25ZX4gF0Ym6wkOXP1uUcryATFQ/+9qu9+v3yxiiHVhhujZS9iVhYeG h50K5uwjsmqNMP30lUy7sjnNhbPp/mwstvWO4TAsODYUbxHwqvPq/NqOFt1F5IsEFQRK 2nCj+rtKcO0rsG6U5v2kBeDqPfGDn+9fqBLCtrN6fLphhkTx+/vnWyx/qm7zwfBvzIFK Pglp60ZllhKzR8KytEEROBPnmZjqEGYTjoKyZIEN9cQoARqBwAGXmEmSQMVN7M5HAz67 EMuC2zmW+3odmR74swgDcASAeIgi2m7PKMsBHhNtqWPG5VlsZ2t69pmTjTFO1fhsQ7Fw Pbvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787278861; x=1787883661; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=vTISns22lLz3uE6OmdCyn6vQB2Si1JaLhFBQroGqygQ=; b=L3deTSBKdDJJjqf5Ygjw9Jb4LKMaRVnRQ7RzRVc1g1/RDPlNV9mFzTJ5EQUvCT8Fjf l7o2KoinLS088v/irGRkQOsy3EKSspg44lFeoSrSjI3NVi6Y6vn0z8Q8j+QStQFQ/0Bd 7HNjwKlyzCsHrkGamqzpCD9T7wcvgYEcyNCL5vnTgu0k20qFL+UcNBlQEODmaYO8+bdW O2nbYzh62xbGzSRVHA0zat9HZ+UM4DyAosuOBG/8w7MGooIrgWR4Y1/wpnW29ADin3Rj gLEgzY/GXwNcr6UB2F1dEeayvtb0iA/ctlk/EHCi8cnCzsYggZN79KLchjokt5psOUqH 1zdg== X-Forwarded-Encrypted: i=1; AHgh+Ro9AaMv2i7ApNRHCO+i4myHIWI25g7CHbEL/izm/U5LwJ9OOot5FF8fkO/Uo1qokWnXrddJNrEnBQsH8SY=@vger.kernel.org X-Gm-Message-State: AFuF++mOzl773iCV5eW9BzJakG1FLj937FXNN2MTQPLAILiFc/OQ6VBd CukX7ElAQBnlOUjDdyuckAeboYPhlWzHPlPISPds9gFKKxrPBNDiKX8EH7JYigFCkA== X-Gm-Gg: AR+sD12hGKsPlMNwMoOEfZv+Zzj7qKLm8hC85qMeHALMWS5H/GeDDYNw5ryt6KoG1Rl /kloMxSYgw1eYEn6pjIf7Xe00byXj4PUr9PnOEExKYPWKHZRcZmAcO9cDYpQXmDrIIDzDvociir BI12C+zt2VdBXAG5GObMr01L+56D52A0UHe5Ke6iUPd2ToPqOapJ5f2+mGrmvDPTym0eoB8MeY1 yeUGz2QCVDis12Q4kpegVc6GvXkK/1AOVeVvn5beUurRFH9FYBik88+zqpDjRRiEXnqB2R0h2wp Iavy0pSsR9b5aC+uD3zdGofDgq9FFwLVj2tnEEObBj2/oAKx/xZFsPRTwKLXJWFK6/M9jtEW/lT 06vH6o0p63v33q0yk3nHw6sHvixzGqYIDLCLlrTEe/tIq04mfRU+NICYaNoMm/QduFmip8bPdcF KuR1tQZdNhT1ElZJRAWkoJxOc+XoTWoLHFxt12loDjNeYzsKJbUCACBJZRP2cEhs+/MJ7tDR5f7 Gt0EWDrvoptS8eDo8bBJQIul794Z69jJeCPPode+jWW4gRuntGVAP5BSbg= X-Received: by 2002:a17:902:f552:b0:2c7:f688:f22f with SMTP id d9443c01a7336-2d650dacb5cmr1955855ad.13.1787278860193; Thu, 20 Aug 2026 19:21:00 -0700 (PDT) Received: from google.com (210.87.127.34.bc.googleusercontent.com. [34.127.87.210]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-395c4b82e5csm1054937a91.16.2026.08.20.19.20.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 19:20:59 -0700 (PDT) Date: Fri, 21 Aug 2026 02:20:55 +0000 From: Samiullah Khawaja To: Jason Gunthorpe Cc: Nicolin Chen , Alex Williamson , "Tian, Kevin" , kvm , Alex Williamson , Bjorn Helgaas , linux-kernel , linux-pci , iommu@lists.linux.dev Subject: Re: [RFC PATCH 1/5] PCI: Refuse function reset of an SR-IOV PF with enabled VFs Message-ID: References: <20260812045325.2733631-1-alex.williamson@nvidia.com> <20260812045325.2733631-2-alex.williamson@nvidia.com> <20260814083737.66bb83fb@nvidia.com> <20260817121810.GC933791@ziepe.ca> <20260818140304.GC5482@ziepe.ca> <20260818143906.1f58aecb@nvidia.com> <20260820233722.GA1110819@ziepe.ca> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20260820233722.GA1110819@ziepe.ca> On Thu, Aug 20, 2026 at 08:37:22PM -0300, Jason Gunthorpe wrote: >On Thu, Aug 20, 2026 at 10:38:07PM +0000, Samiullah Khawaja wrote: > >> > The blocking domain operation looks like it might be simplest to >> > implement in the IOMMU core. We can set a flag for a default blocking >> > domain on the IOMMU group when we take_dma_ownership of the group. Then >> > release_dma_ownership picks the blocking rather than default domain. >> > >> > This is then unwound in use_default_domain, called via dma_configure, >> > attaching the device to the default domain in probe of the next driver. >> > Therefore until probe by another driver, a device used by vfio would >> > remain in a blocking domain even while unused and unbound. >> >> The devices are expected to be attached to the default_domain even when >> these are unbound and the use_default_domain assumes that, and it only >> checks the ownership and doesn't switch the domain to default_domain. I >> guess we should add a WARN in use_default_domain() if that is not true. >> I will probably send out a patch for that separately. >> >> I think we can move the device back to default_domain after reset after >> unbind, maybe it can be done in pci_dma_cleanup() based on >> driver_managed_dma? > >This blocking domain stuff sounds very similar to what Nicolin >implemented for the per-function ATS issue? I see you are talking about this invalidation stuff: https://lore.kernel.org/all/348c50ab6e95b5ec6d48ee3fa05d529a784a34c3.1765834788.git.nicolinc@nvidia.com/ But this was the case where the device is going to be reset and to prevent ATS issues, we attach it to blocking domain before doing the reset. But of course, with the PF reset here, it induces the same kind of ATS issues on the VFs. > >Broadly we must setup a blocking domain in the iommu if ATS is >available across reset or you get these ATS related issues. > >I think at the time he looked at doing SRIOV as well but it was >tricky.. Hmm... doing that for SRIOV also, by allowing PF reset but attaching the VFs to blocking domains before the PF is reset, will only resolve the ATS issues. But the software state is still out of sync with hardware state. > >Jason Sami