Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song <muchun.song@linux.dev>]

> On Jul 10, 2026, at 03:04, James Houghton <jthoughton@google.com> wrote:
> 
> On Thu, Jul 9, 2026 at 2:55 AM Muchun Song <muchun.song@linux.dev> wrote:
>>> On Jul 9, 2026, at 00:49, James Houghton <jthoughton@google.com> wrote:
>>> 
>>> On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
>>>> Do you mean that the support for AF might vary across different CPUs?
>>>> I'm not that familiar with arm64, so it seems a bit strange to me that
>>>> such basic hardware features can differ so much from one CPU to another.
>>> 
>>> Yes, hardware updates of the Access Flag is a per-CPU feature. It is
>>> available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
>>> register; each CPU has its own. (Linux will always enable HW AF for a
>>> CPU when it is onlined[1] if support is advertised, so we simply need
>>> to check if support is advertised to know that it is in fact enabled.)
>>> 
>>> These days it is not uncommon for a system to have two (or more?)
>>> different core implementations, like with "fast" cores and "efficient"
>>> cores.
>>> 
>>> [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S
>> 
>> Thanks for your detailed explanation. When enabling HVO via the cmdline,
>> can we simply prevent CPUs that do not support AF from coming online?
>> Would implementing it this way be much simpler? In practice, developers
>> definitely know whether the current system is suitable for enabling HVO.
>> If some CPUs do not support AF, they would just need to evaluate the
>> trade-off between memory savings and having fewer online CPUs than expected.
>> 
>> For scenarios where HVO is enabled via sysctl, we simply need to check
>> if all CPUs support AF. If any do not, the system should return an error.
>> 
>> Then, we can proceed with the Pre-HVO.
> 
> I don't think it makes sense to try to implement pre-HVO.
> 
> We cannot do HVO if any boot CPUs do not support HW AF, as HW AF will
> be required to free the HugeTLB pages later, which we should continue
> to support. Pre-HVO (today anyway) happens before all boot CPUs are
> onlined. IMO it is not okay to prevent boot CPUs from onlining.

When the boot CPU starts, it first checks if the hardware (the current
boot CPU) supports AF (The selection of the Boot CPU is critical here
If the user really wants to enable HVO). If AF is supported and HVO is
enabled via cmdline, we perform a Pre-HVO. Subsequently, any CPUs that
do not support AF will be prevented from coming online.

If the boot CPU does not support AF, enabling HVO is not permitted—neither
via cmdline nor sysctl. All CPUs are allowed to be online.

Once the system has started, there is only one way to enable HVO, and
that is through sysctl. 1) If the current system includes any CPUs that do
not support AF, enabling HVO is not permitted (unless the user chooses to
take these CPUs that don't support AF offline). 2) If all online CPUs in the
current system support AF, then HVO can be enabled. In this case, the
system must also block any CPUs that do not support AF from coming online
in the future. Of course, if a user specifically wants those CPUs to be
allowed to come online, they can choose to disable HVO first (At the same time,
this means that the HVO-optimized HugeTLB must be freed first as well).

My proposal assumes that even though the system supports different types of
CPUs, their support for AF (I suppose this is a very basic feature nowadays)
is generally the same. I believe that for servers (Only servers stand to
benefit more from HVO), there shouldn't be a mix of those that support AF
and those that don't. At the very least, such a situation would be extremely
uncommon. I believe this may be acceptable. That said, if we go this route,
is there a way to simplify the code implementation even further?. Of course,
this is just my personal speculation. Please let me know if I've missed
anything.

> 
> Let's say for a moment that HVO cannot be toggled at run-time, then
> the best we can do is:
> - If a user does not specify hugetlb_free_vmemmap=1, we can always
> allow onlining of late CPUs
> - If a user specifies hugetlb_free_vmemmap=1 but not all boot CPUs
> support HW AF, we can always allow onlining of late CPUs.
> - If a user specifies hugetlb_free_vmemmap=1 and all boot CPUs support
> HW AF, we must not allow onlining incompatible late CPUs.
> 
> HVO compatibility has to be modeled as an Arm system feature. To
> support conditional onlining of late CPUs based on whether or not "HVO
> is being used", we still need the extra cpufeature.c logic.
> 
> So if HVO cannot be toggled at run-time, we can simplify the
> definition of "is HVO being used?" (for the purposes of determining if
> late CPUs can be onlined). We can simplify it from what it is now
> ("are there any optimized pages?") to
> "vmemmap_optimize_enabled==true?". That allows us to drop patch 15
> (which then needs a slight change to patch 17), but that's about it.
> So this is a slight simplification, which is nice.
> 
> If you (or the Arm folks) feel strongly, I'm happy to write this simplification.
> 
>>> I'm hoping you can look at patches 1-3 anyway. I think they're the
>>> right thing to do, even without the arm64 changes.
>> 
>> Yes, I've looked it over. I think that making modifications across
>> several areas will be much simpler on top of my refactored codebase.
>> For example:
>> 
>> - In patch 2: There is no longer a need to move vmemmap_get_tail().
>> - In patch 3: Handling partially-HVOed pages becomes straightforward
>>               because vmemmap_restore_pte() no longer relies on
>>               ->vmemmap_tail to detect whether a restoration is needed.
>> - In patch 4: I have updated the ->remap_pte callback to return an int
>>               as well, making it easy to adapt your changes.
>> 
>> Overall, the implementation is significantly cleaner, so I won't list
>> every detail here.
> 
> Thanks! Good to know I can basically drop those patches. :)
> 
> Please CC me on these HVO simplifications when you send them out, thanks!

No problem. Will do.

Thanks.