From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63C071A840F for ; Fri, 17 Jan 2025 21:31:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737149491; cv=none; b=Y8zowov0VXLXTE3bAtmEfqPu3W1iRq9Vx7jeGO+AMdZySFk2+5Yica6DCKLkUkXYLpHFwpiTDTolAufvealHxDRL/m1Ogfa1ML1onksU0ld0eaYSLfCv1KgYa481htczh+TcgwWzwnT0HG0OxSjrYsP3q+kEdBvHx3e2X2Ct/mg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737149491; c=relaxed/simple; bh=URYtzwHaqHyyjSm8MWg+z/STBcL3q1UIAPmgvFW+drw=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SzVo8wtx42NpSNHNOj9znXPI2JuoBqF1NHmS2D7PbX1NnXFdzoHOUaSYYq9DMj5tTY8S4bWOogsuorIAlJ+gr6x0wnwgaCd+YXOHUtqfGre6yxeYR9b+af343OXpauUEcwKObEghT15xAyorqF6EPfFYLNAj6eNTdD6jEPeIDvA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NCsUAaem; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NCsUAaem" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737149488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CXkX6EUOkHpV3m0oGcaG/021SThKCpwYilAQGnGl0XA=; b=NCsUAaemnXFa0FquC9YNEEOWglt5nV2eXKSgRpNaXtH+4knTfUvho4CcUQCeWhVHHOTB/4 mUyiVy0dUIOPeeX3w6Kqu7EBTYHcWxviYx6FKavdsRxekSWmVYr1EKjMZIx6Qis0z/e+Ph T0Z5MKkexBtrBq9QeRSb7t3JjWjikD0= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-553-5xy7nuTeO2mdEJ-FHRu8Ig-1; Fri, 17 Jan 2025 16:31:27 -0500 X-MC-Unique: 5xy7nuTeO2mdEJ-FHRu8Ig-1 X-Mimecast-MFC-AGG-ID: 5xy7nuTeO2mdEJ-FHRu8Ig Received: by mail-io1-f70.google.com with SMTP id ca18e2360f4ac-849e7868f6eso13136139f.0 for ; Fri, 17 Jan 2025 13:31:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737149485; x=1737754285; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=CXkX6EUOkHpV3m0oGcaG/021SThKCpwYilAQGnGl0XA=; b=Yafc3ENaZWf6Fx9ysEAQ7LFEorp8zvWuZc7BwO4/2c72A9jSb5WBIDQNPPI8XCug2l OCFlC1KisqCXp7ymE4tQXGlcwizPzITdVDHYWT3hlqkVU+G8/9UkdtrDrjei7safIiJ4 YaoSFGaiEzDfCZyQUqcXYp7Af0d1VWXumngnU9trluWbbwtsuuLKyACv92LJvbLHK5Qr TU5+Q/RFhqm2Xm0k5iF97R8ClC+frb+295lbCNsoBeIrEEiIqgjlqjZdT/5Kv2vuNYxB LNHwjRlzjoUNyRlKGUDNZYoV9cpAMVZoWuM2UWowh+sPuLnCxN+kZX4InhAbCup6Qu9W d8jg== X-Forwarded-Encrypted: i=1; AJvYcCWUkhwI7ISgSkrHKMCnXpTDdrdD1MKBuQcjB/3SVT6oReghZA2usdYkLRMyNI0FYHRRLbfKpJC6zn43AEc=@vger.kernel.org X-Gm-Message-State: AOJu0YzrpZK5rUbxb8q+w83I7YWokxi/YOnPDe7+H21e3+9zICwFtZHN pAnHeHcx4o6fOdMVOEDM5U+LGmYamz8M1rYOECLsQR3pou25oGBOxaUcy+x95QWVaRxTdhF75jG pOZ+JKNTjSZP2y3Pr7NT+xLRNzWX9JtloojuvFq3wpUWArC+7IGYIzu69fmllKQ== X-Gm-Gg: ASbGncs7dWVKeG8Uc3pf589RQdSTq7DwBitjLKDPxNZwdDpuE52aDYuprTyDmYm6OY7 hhaUvXzz6k2YgQ3BmuVPQupx1o0d77bhZa581wcAG8MFECZford1T3/BQfTJzclK2j73hqIV+eb MJGngLoVG0huHOjBhNlTWAmOnrfeqeZXk1dmtbXNULl8yvS9BSBZtMFoD3V8XVXbvhuXWLdX2e7 ppIJRFRPjqdMzeB8I4d5721yNWH+qxaGYexldKjnQU9dBwDiJxtCaWaB0Mj4eoa4e+8M232Lg== X-Received: by 2002:a05:6602:8c1:b0:84a:51e2:6642 with SMTP id ca18e2360f4ac-851b618ab9emr88638339f.2.1737149485614; Fri, 17 Jan 2025 13:31:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHSd0M5RX2RVoKaQYO0/jLbCWOntW3vlF4ULNemQk85jweN5ABdI5zEy20LoW6yTsGZ24C9fA== X-Received: by 2002:a05:6602:8c1:b0:84a:51e2:6642 with SMTP id ca18e2360f4ac-851b618ab9emr88636939f.2.1737149485273; Fri, 17 Jan 2025 13:31:25 -0800 (PST) Received: from redhat.com ([38.15.36.11]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4ea753f64b4sm821316173.28.2025.01.17.13.31.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jan 2025 13:31:24 -0800 (PST) Date: Fri, 17 Jan 2025 16:31:08 -0500 From: Alex Williamson To: Ankit Agrawal Cc: Jason Gunthorpe , Yishai Hadas , "shameerali.kolothum.thodi@huawei.com" , "kevin.tian@intel.com" , Zhi Wang , Aniket Agashe , Neo Jia , Kirti Wankhede , "Tarun Gupta (SW-GPU)" , Vikram Sethi , Andy Currid , Alistair Popple , John Hubbard , Dan Williams , "Anuj Aggarwal (SW-GPU)" , Matt Ochs , "kvm@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v3 3/3] vfio/nvgrace-gpu: Check the HBM training and C2C link status Message-ID: <20250117163108.3f817d4d.alex.williamson@redhat.com> In-Reply-To: References: <20250117152334.2786-1-ankita@nvidia.com> <20250117152334.2786-4-ankita@nvidia.com> <20250117132736.408954ac.alex.williamson@redhat.com> <20250117143928.13edc014.alex.williamson@redhat.com> Organization: Red Hat Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Fri, 17 Jan 2025 21:13:52 +0000 Ankit Agrawal wrote: > >> > We're accessing device memory here but afaict the memory enable bit = of > >> > the command register is in an indeterminate state.=C2=A0 What happen= s if you > >> > use setpci to clear the memory enable bit or 'echo 0 > enable' before > >> > binding the driver?=C2=A0 Thanks, > >> > > >> > Alex =20 > >> > >> Hi Alex, sorry I didn't understand how we are accessing device memory = here if > >> the C2C_LINK_BAR0_OFFSET and HBM_TRAINING_BAR0_OFFSET are BAR0 regs. > >> But anyways, I tried 'echo 0 > /enable' before device bind= . I am not > >> observing any issue and the bind goes through. > >> > >> Or am I missing something? =20 > > > > BAR0 is what I'm referring to as device memory.=C2=A0 We cannot access > > registers in BAR0 unless the memory space enable bit of the command > > register is set.=C2=A0 The nvgrace-gpu driver makes no effort to enable= this > > and I don't think the PCI core does before probe either.=C2=A0 Disabling > > through sysfs will only disable if it was previously enabled, so > > possibly that test was invalid.=C2=A0 Please try with setpci: > > > > # Read command register > > $ setpci -s xxxx:xx:xx.x COMMAND > > # Clear memory enable > > $ setpci -s xxxx:xx:xx.x COMMAND=3D0:2 > > # Re-read command register > > $ setpci -s xxxx:xx:xx.x COMMAND > > > > Probe driver here now that the memory enable bit should re--back as > > unset.=C2=A0 Thanks, > > > > Alex =20 >=20 > Ok, yeah. I tried to disable through setpci, and the probe is failing wit= h ETIME. > Should we check if disabled and return -EIO for such situation to differe= ntiate > from timeout? No, the driver needs to enable memory on the device around the iomap rather than assuming the initial state. Thanks, Alex