From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 777CE405F7 for ; Fri, 2 Jan 2026 14:24:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767363864; cv=none; b=IfNGAaX2IahH1Mfx4HQRagGfDYbyRRgLlvuvT6S4hM86IvEZDPjmr1z+8O3dQll5uXZTNxQAqKhel8A3TqTlcSeNsjh7w97Ro9lpGZVp8NNRYWaz27aFrwDjjr6Pg29mncLZ4IlhM3Eezxx8sNwdH9h418wv2w381yPsv2lpbfw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767363864; c=relaxed/simple; bh=fAtafWg1sFBzBjGB7d+SGUoSzDz5D9BUwo+weGX2YLg=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=ksx2zgqLWb8iFxahWXj3Co9m2nxJjs3IuN4uiRA8hoG0VNzimA5KcQOYO3DEWQRG2qUf9fGdz2MjAqHuYwMpalgp4YR58wJqrTiXmpCtAe6XQu7LxSHZ5kJsXNUHxs0rmN/BPXwCsgWB/pSa6Y2txb6VWF7VyaMVy7TP6yNrwQY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=L0we59nD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="L0we59nD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 42226C116B1; Fri, 2 Jan 2026 14:24:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767363864; bh=fAtafWg1sFBzBjGB7d+SGUoSzDz5D9BUwo+weGX2YLg=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=L0we59nDG+Q3GO+jsNtxGY9fMPViX52grgVcLoYYogLiRu0tpcWBslFf5vCdovms0 jAZGfkNETBQOZJyw03RwzQyCxjmF8CGIzfhFKihnFxTSondniZaSRsC6aNtytG2pJ9 laaVkcVXzn9NczWdabf8Rk3cUuuKSPVQmn8DVlvBbHXT3ee944ykDzxaOkHpbMPzFg 24mwMcVKhvB5rCloyErmB8LjDtfa8jvARawIIlSwcy1AHh5JXL+NiGEDrHi1PdVf1k 19DmldnbzoIsY/IYkz5W4aGv8oRimtMl33ZnaW4baB3T96NBSCJz69fNwuQSfj4TTW BN/2JbRLlBlFw== From: Pratyush Yadav To: Mike Rapoport Cc: Pasha Tatashin , Pratyush Yadav , Evangelos Petrongonas , Alexander Graf , Andrew Morton , Jason Miu , linux-kernel@vger.kernel.org, kexec@lists.infradead.org, linux-mm@kvack.org, nh-open-source@amazon.com Subject: Re: [PATCH] kho: add support for deferred struct page init In-Reply-To: (Mike Rapoport's message of "Wed, 31 Dec 2025 11:46:39 +0200") References: <86jyyecyzh.fsf@kernel.org> <863452cwns.fsf@kernel.org> <864ip99f1a.fsf@kernel.org> Date: Fri, 02 Jan 2026 15:24:18 +0100 Message-ID: <86wm206qjx.fsf@kernel.org> User-Agent: Gnus/5.13 (Gnus v5.13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Wed, Dec 31 2025, Mike Rapoport wrote: > On Tue, Dec 30, 2025 at 01:21:31PM -0500, Pasha Tatashin wrote: >> On Tue, Dec 30, 2025 at 12:18=E2=80=AFPM Mike Rapoport = wrote: >> > >> > On Tue, Dec 30, 2025 at 11:18:12AM -0500, Pasha Tatashin wrote: >> > > On Tue, Dec 30, 2025 at 11:16=E2=80=AFAM Mike Rapoport wrote: >> > > > >> > > > On Tue, Dec 30, 2025 at 11:05:05AM -0500, Pasha Tatashin wrote: >> > > > > On Mon, Dec 29, 2025 at 4:03=E2=80=AFPM Pratyush Yadav wrote: >> > > > > > >> > > > > > The magic is purely sanity checking. It is not used to decide = anything >> > > > > > other than to make sure this is actually a KHO page. I don't i= ntend to >> > > > > > change that. My point is, if we make sure the KHO pages are pr= operly >> > > > > > initialized during MM init, then restoring can actually be a v= ery cheap >> > > > > > operation, where you only do the sanity checking. You can even= put the >> > > > > > magic check behind CONFIG_KEXEC_HANDOVER_DEBUG if you want, bu= t I think >> > > > > > it is useful enough to keep in production systems too. >> > > > > >> > > > > It is part of a critical hotpath during blackout, should really = be >> > > > > behind CONFIG_KEXEC_HANDOVER_DEBUG >> > > > >> > > > Do you have the numbers? ;-) >> > > >> > > The fastest reboot we can achieve is ~0.4s on ARM >> > >> > I meant the difference between assigning info.magic and skipping it. >>=20 >> It is proportional to the amount of preserved memory. Extra assignment >> for each page. In our fleet we have observed IOMMU page tables to be >> 20G in size. So, let's just assume it is 20G. That is: 20 * 1024^3 / The magic check is done for each preservation, not for each page. So if the 20G of preserved memory is 1G huge pages, then you only need 20 to check the magic 20 times. > > Do you see 400ms reboot times on machines that have 20G of IOMMU page > tables? That's impressive presuming the overall size of those machines.=20 > >> 4096 =3D 5.24 million pages. If we access "struct page" only for the >> magic purpose, we fetch full 64-byte cacheline, which is 5.24 million >> * 64 bytes =3D 335 M, that is ~13ms with ~25G/s DRAM; and also each TLB >> miss will add some latency, 5.2M * 10ns =3D ~50ms. In total we can get >> 15ms ~ 50ms regression compared to 400ms, that is 4-12%. It will be >> less if we also access "struct page" for another reason at the same >> time, but still it adds up. > > Your overhead calculations are based on the assumption that we don't > access struct page, but we do. We assign page->private during > deserialization and then initialize struct page during restore. > We get the hit of cache fetches and TLB misses anyway. Exactly. The cache line will be fetched anyway. So I think the real overhead is a fetch and compare. > > It would be interesting to see the difference *measured* on those large > systems. > [...] --=20 Regards, Pratyush Yadav