From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 936AC86348 for ; Wed, 15 Jan 2025 17:05:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.176 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736960735; cv=none; b=H/Uao6fC2PQfoPz30PMZYIp9Rq6AWj1c8T9uXohCY1sV8p7o6kFmYKj9S02IrkalWY/6qP4AXgqA/wmb9c+d6oolkQadoPY2f4HKUQ+ZKw0OkJca6MheZLEkV+iwzWqILcaYzDQ1d1FKQr2evibCtdbwxFfr3RzRvpIkQrRsXWI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736960735; c=relaxed/simple; bh=74ujiHqa5JZjFipwZbxCv3A1jGWUhbO9BIXRabl1Iwg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=oo3Z+OV6iASwV67WMRB9cIHOQnpUX+34oHdFfM2jniBAoQ/8yk5XePaJmwYkwQyOTP7f7V8KcO23BLQCV/WVCTo9ECdtsi9TAJElelrFfRuxGSzDRihJhQAG6Lx71o4X5K4hlQ9FrESdXv/nbkinnXQ4Kc5xEFVpKdmJNxuhg/w= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=kcjL/6kG; arc=none smtp.client-ip=209.85.222.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="kcjL/6kG" Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7b6f0afda3fso701676885a.2 for ; Wed, 15 Jan 2025 09:05:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736960732; x=1737565532; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=+6RrQ3pBaAVcoFnAc6/GFF9heBGG2e1/nEy3oiiP8jM=; b=kcjL/6kGCiO6ZhdSZB2MItWLXadLtn9ZT94J7/X7SFc9C/Ju6SqIK+hq5AnIHww5AW STMuGsusSogox4uaofigAIlZi3kHUqc4VKMQDtFyL+EKSlvKiTXQvFx6yTJi5f7z7cWP On5xrai4fJ2atlbnJDECnTKoVw9UaCu2grqkcWRmT9iQQXkGxgh+F6ixvfTnT40st/2i +WhjVvHbyiwNahXsGNBXOdSfrqQ8f3s1oTmPi8H465NHAdejR1qCtHdzZpy0k70zGAoc 95RBIz/BIUv/ypxDfmNKLdnaAbOojxFuzOLA977EsrwbiXy5S1Bg0vZzXB9dBQ4bfwJh T2tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736960732; x=1737565532; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+6RrQ3pBaAVcoFnAc6/GFF9heBGG2e1/nEy3oiiP8jM=; b=dtNjxyjh78eDrXHZgQn5rWkDvSmzhBgEZGH51An3sZsgbNMpyHEvdardhPIi30SwIS eQDVGq1pKvw2p0/LRAH/R6CZHPlW6MxDfITsB7MWmOIqnT3WIBzDwTBPR0kwkXGTBW9v 8+50z01n/9mLN2WMUlnQsuQYsjucr+8XzJR9cjhBWtzkdvC8cRAbk9+UsGxAFDQyyDlr Opy4RGnfQ5gH0XbyrlQK1j8ssU2HvrDzTN1F3+u7CmcWdcyUN0uyAB6crwudJQbkYz9v rCVdblLI1KYqV3yih204tJmfSnH6LFADQAoxC2lQ0grAZjGP6mx6H77qoogCMB7TVsUs lQGQ== X-Forwarded-Encrypted: i=1; AJvYcCUh5zde3vbgMB2R2ILiTMzfrrQrCnK3bwqELOfUpyzv+SgM7xnNUfkI+5tD0iqSN13t/uveBYjfFUVBLqM=@vger.kernel.org X-Gm-Message-State: AOJu0Yxp1IDPT/Z4TGmLNxHaGbBSFiWSmfs66ehRZZdT+sVKmJq+qwqB o3iVC1y48Fln1yIP1zD6FgFKjgiNvZijyIthtajT7Mr/78LzK9VXTD2G7xrdYe8= X-Gm-Gg: ASbGncvwTNICVoBtBQDIYqWZm9fdQebjBnX2nPQy0FZQFxY9nnqckWXGWjVRuqlw2u7 U2bSdUVaZaR/wswezyB0Cc4XN2i048fVnMaDyQaP8dsAPlzJEq1A/J6Iy3eFLOM5AFQB8wj8jxy jSY1YvEEe2uesL1FDvNhQT5smDt3TIksVycTZkTnQGS8D5p+jSLDSwnMeAt5dx4SnipFzG2M18n XgLnhlnTkVD8f0c2EMb0OZXQF0n142pR59dDC0hSrM7jeuCs6ObkkNII8zjtrByoDYS/co1SrOG W8OgpbawEyeiMk2uStGo2I4Ue6kAFaV7HdgZOLg= X-Google-Smtp-Source: AGHT+IFEXMvJgM0dISntes7DBNI6YuCnV9MrVUYaqBMi1+QY9kA/7GzKiLqk/h7x7hOGp4rzKanxVA== X-Received: by 2002:a05:620a:4011:b0:7b7:142d:53a4 with SMTP id af79cd13be357-7bcd976d7e5mr4700725885a.51.1736960732245; Wed, 15 Jan 2025 09:05:32 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dfade73245sm66320636d6.76.2025.01.15.09.05.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jan 2025 09:05:31 -0800 (PST) Date: Wed, 15 Jan 2025 12:05:29 -0500 From: Gregory Price To: Robert Richter Cc: Alison Schofield , Vishal Verma , Ira Weiny , Dan Williams , Jonathan Cameron , Dave Jiang , Davidlohr Bueso , Terry Bowman , linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, "Fabio M. De Francesco" Subject: Re: [PATCH v1 25/29] cxl/amd: Enable Zen5 address translation using ACPI PRMT Message-ID: References: <20250107141015.3367194-1-rrichter@amd.com> <20250107141015.3367194-26-rrichter@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Wed, Jan 15, 2025 at 04:05:16PM +0100, Robert Richter wrote: > > > > Should be dpa as argument? Confusing to convert an hpa to an hpa. > > We need to handle the decoder address ranges, the argument is always > the HPA range the decoder belongs to. I see, and this is where my confusion stems from. Basically these addresses are consider "HPA" because they are programmed to decoders, and decoder addresses are "always HPA". i.e. 2 interleaved devices (endpoint decoders) with normalized addresses: dev0: base(0x0) len(0x200000000) dev1: base(0x0) len(0x200000000) These are HPAs because decoders are programmed with HPAs. It's just that in this (specific) case HPA=DPA, while root decoders and host bridge decoders will always have HPA=SPA. We're just translating up the stack from HPA range to HPA range. I've been dealing with virtualization for a long time and this has been painful for me to follow - but I think I'm getting there. > > DPA(0) > > dev0: base(0xc050000000) spa(0xc050000000) > > dev1: base(0xc050000000) spa(0xc050000000) > > > > DPA(0x1fffffffff) > > dev0: base(0xc050000000) spa(0xe04fffffff) > > dev1: base(0xc050000000) spa(0xe04fffffff) > > > > The bases seems correct, the SPAs looks suspect. > > SPA range length must be 0x4000000000 (2x 128G). That is, upper SPA > must be 0x10050000000 (0xc050000000 + 0x4000000000 - 1). This one is > too short. > > The decoder range lengths below look correct (0x2000000000), the > interleaving configuration should be checked for the decoders. > If i understand correctly, this configuration may be suspect [decoder0.0]# cat start size interleave_ways interleave_granularity 0xc050000000 0x4000000000 2 <----- root decoder reports interleave ways = 2 256 [decoder1.0]# cat start size interleave_ways interleave_granularity 0xc050000000 0x4000000000 1 <----- host bridge decoder reports interleave ways = 1 256 [decoder3.0]# cat start size interleave_ways interleave_granularity 0xc050000000 0x4000000000 1 <----- host bridge decoder reports interleave ways = 1 256 > > I do not understand this chunk here, we seem to just be chopping the HPA > > in half to acquire the DPA. But the value passed in is already a DPA. > > > > dpa = (0x1fffffffff & ~(256 * 2 - 1)) / 2 + (0x1fffffffff & (256 - 1)) > > = 0xfffffffff > > HPA is: > > HPA = 2 * 0x2000000000 - 1 = 0x3fffffffff > ... snip ... > There is probably a broken interleaving config causing half the size > of total device mem. > In my case, I never see 0x3fffffffff passed in. The value 0x1fffffffff from the endpoint decoders is always passed in. This suggests the host bridge interleave ways should be 2. I can force this and figure out why its reporting 1 and get back to you. > > dev0 (dpa -> hpa -> spa): 0x0 -> 0x0 -> 0xc050000000 > > dev1 (dpa -> hpa -> spa): 0x0 -> 0x100 -> 0xc050000100 > > dev0 (dpa -> hpa -> spa): 0x1fffffffff -> 0x3ffffffeff -> 0x1004ffffeff > > dev1 (dpa -> hpa -> spa): 0x1fffffffff -> 0x3fffffffff -> 0x1004fffffff > > Yes, would be the result without the offset applied for spa2 above. > The check above calculates the *total* length of hpa and spa with out > considering the interleaving position. This is corrected using the > offset. There is no call prm_cxl_dpa_spa(dev0, 0x1fffffffff) that > returns 0x1004fffffff, but we want to check the upper boundery of the > SPA range. > This makes sense now, there's no dpa->spa direct translation because you may have to go through multiple layers of translation to get there - so the best you can do is calculate the highest possible endpoint and say "Yeah this range is in there somewhere". Thank you for taking the time to walk me through this, I'm sorry I've been confused on DPA/HPA/SPA for so long - it's been a bit of a struggle. ~Gregory