From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4F791FBEB9 for ; Mon, 1 Sep 2025 15:50:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756741835; cv=none; b=piVJtq9Erh4K6nlUF7qx2bEm7SXw5pfAn9a2fMKECIsL1kyNiVLQThr0Qdo0jEUuk3H7u7+AcOy+BiXV9OXwtdfaTv2P0MgQlFQMTzBIlhdoWnrKW3ez+rFN+1pF/q5ieX3RVuPeMxybnY7B7yB/PqSnoXIFq0l2HDj3bYaa+tQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756741835; c=relaxed/simple; bh=DMnX/N7trPPR7m7TLJzFjgQ3Owtcg4Cjc0SYqK2Elis=; h=From:Date:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=t3DBuKlGLsHljfP6/TgmPsixDjIK1It9FGYCwEpjGEjuliWLtniSINd6QeiN4YOTZiaplcTguy5QFOgOtle7epiBSKWzbL2Gtgo9JnaPoXvDlhhmAmAuOa6wcdTpz0xhB3f18tuWFlekKdRgnLgL4ODrxIMPXzPIO8MQVfl3ONw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=RzLqP9gE; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="RzLqP9gE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756741834; x=1788277834; h=from:date:to:cc:subject:in-reply-to:message-id: references:mime-version; bh=DMnX/N7trPPR7m7TLJzFjgQ3Owtcg4Cjc0SYqK2Elis=; b=RzLqP9gEeCEo9Gk+ErsVePMbTnwkt/9p29rRvvBrkrgcIppY5lemvhhI /D/hsaDH7xCC6BE4Hn8wD+IazaT/bZ2jxASZAyfQrbWpBkEQm5mvxFHNG ZDdxzlhPlMiY/q+ckxutHoTpGBVk0DFp0g+iTfvNsNWrW2R0e949se651 dkkhCPHXt8CLxf6HtlUSsvHgwJvLkjahkijHqhkdzfEJRyzsBU+9Z/eu5 z2WVXu0mW+Mu+UzaAn8T1HISN+j8PhUcd89oVTKVN+sYm7952ybNmS5qo PXofc7fbZIUuKPO4Hmtvj0vAx9bQ30q4EU/LaNwk71sN4LI7VOjeTGaOv Q==; X-CSE-ConnectionGUID: RBfG5bSfQUemeCyIV/7uYw== X-CSE-MsgGUID: SQ/NYmU+R/2cT9M0XNF+6g== X-IronPort-AV: E=McAfee;i="6800,10657,11540"; a="84417510" X-IronPort-AV: E=Sophos;i="6.18,225,1751266800"; d="scan'208";a="84417510" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Sep 2025 08:50:33 -0700 X-CSE-ConnectionGUID: QVf3PfRTSPmkaV0Mk+9dYA== X-CSE-MsgGUID: jGs9bkVCQKuQP/0PYQDuEA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,225,1751266800"; d="scan'208";a="170330812" Received: from ijarvine-mobl1.ger.corp.intel.com (HELO localhost) ([10.245.244.193]) by orviesa006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Sep 2025 08:50:32 -0700 From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= Date: Mon, 1 Sep 2025 18:50:27 +0300 (EEST) To: Steve Oswald cc: linux-pci@vger.kernel.org Subject: Re: [BUG] Thunderbolt eGPU PCI BARs incorrectly assigned, fails to assign memory In-Reply-To: <9254be77-46ea-992f-a1bd-98bea3943520@linux.intel.com> Message-ID: References: <9254be77-46ea-992f-a1bd-98bea3943520@linux.intel.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8323328-625503797-1756741827=:947" This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323328-625503797-1756741827=:947 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE On Mon, 1 Sep 2025, Ilpo J=C3=A4rvinen wrote: > On Sun, 31 Aug 2025, Steve Oswald wrote: >=20 > > I=E2=80=99ve encountered an issue with Thunderbolt eGPU (externally con= nected > > gpu via thunderbolt 4). The change from kernel 6.10.14 to 6.11.0 broke > > the pci memory assignment of the external pcie device. I figured out > > which version broke it by using ubuntu 25.04 and downgrading the > > kernel (https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel= =2Esh/master/ubuntu-mainline-kernel.sh). > >=20 > > >From the dmesg output, on the broken 6.11.0 I see 'failed to assign'. > > The issue occurs (almost never) on previous kernel version 6.10.14. > > Using pci=3Drealloc did not change the behavior (I can produce the dmes= g > > output if necessary). > >=20 > > The issue was tested with 2 egpus (Radeon Instinct MI50 32GB, NVIDIA > > 3080 10GB). Both the amd and the nvidia driver fail to initialize the > > device because they cannot write the pcie messages. > >=20 > > System details: > > - Kernel: Linux 6.10.14-061014-generic (Ubuntu build) > 6.11.0-061100 > > - Laptop: TUXEDO InfinityBook Pro 16 - Gen8 with Thunderbolt 4 > > - eGPU: Radeon Instinct MI50 32GB, NVIDIA 3080 10GB > >=20 > > Steps to reproduce: > > 1. Boot the system with the eGPU. > > 2. Observe PCI BAR message in `dmesg`. > >=20 > > Logs: > > both kernel messages, lspci can be found here: > > https://gist.github.com/stepeos/cd060c7d66ab195f51ab4d5675b4e4af > > raw files: > > - dmesg_linux_6.11.0.log > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4= e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.11.0.log > > - dmesg_linux_6.10.14.log > > https://gist.githubusercontent.com/stepeos/cd060c7d66ab195f51ab4d5675b4= e4af/raw/f9470a06ff929d386c50ec6b5d07e0ff3f053dcf/dmesg_linux_6.10.14.log > >=20 > > If additional info is needed, I'm happy to help. >=20 > Hi Steve, >=20 > Thanks for the report. >=20 > My analysis is that the problem boils down to lack of this line with 6.11= : >=20 > pcieport 0000:00:07.0: resource 15 [mem 0x6000000000-0x601bffffff 64bit p= ref] released >=20 > It means one of the upstream bridge windows could not be released for=20 > resize as it is printed from pci_reassign_bridge_resources() which likely= =20 > occurs inside pci_resize_resource() call from amdgpu(?). >=20 > The very likely cause is this check: >=20 > /* Ignore BARs which are still in use */ > if (res->child) > continue; >=20 > ...which (until very recently) is entirely silent so there's no warning= =20 > whatsover what is the root cause. Hi again, Actually, scratch most of that. It's not during resize as the log should=20 say "releasing" (I don't know how I got this confused). "released" is from= =20 pci_bridge_release_resources() which is called from=20 pci_bus_release_bridge_resources() doesn't even try to walk upwards. But that begs question, why didn't also the bridge windows fail their=20 assignments. Resource fitting calculates size for the bridge window: pci 0000:03:00.0: bridge window [mem 0x800000000-0x10003fffff 64bit pref] t= o [bus 04-2c] add_size 100000 add_align 100000 =2E..but I cannot see assignment for that even being attempted as almost=20 immediately, this occurs: pci 0000:03:00.0: bridge window [mem 0x6000000000-0x601bffffff 64bit pref]:= assigned =2E..which is much less than 0x10003fffff-0x800000000. I cannot think of=20 anything what could make it shrink like that. I'll have to think this more, it might require a debug patch but I'll=20 think until tomorrow to see if I can understand it from the code alone. > What this means, is that there's some assigned resource underneath=20 > 0000:00:07.0 with 6.11 that wasn't there with 6.10. And it is because 6.1= 1=20 > tried harder to get your resources assigned and was successful here and= =20 > there resulting in pinning the bridge window in its place, whereas 6.10= =20 > failed to assign the same resource. >=20 > Could you provide /proc/iomem (it's enough to do that for 6.11 for now)? >=20 >=20 > You could try to use hpmmioprefsize=3D on kernel's command line to reserv= e=20 > more space for the bridge windows, the default is only 2M and these GPUs= =20 > need a magnitude more (gigabytes), you can check from 6.10 what the sizes= =20 > of the BARs on the GPU are, and round the sum upwards to the next power o= f=20 > two multiple. > > I'd also be interested to see why pci=3Drealloc failed to solve this prob= lem=20 > as it should reconfigure the entire resource tree so if you could provide= =20 > the logs with that. Please take lspci with -vvv. >=20 >=20 >=20 --=20 i. --8323328-625503797-1756741827=:947--