Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song <muchun.song@linux.dev>]

> On Jul 13, 2026, at 12:09, James Houghton <jthoughton@google.com> wrote:
> 
> On Thu, Jul 9, 2026 at 8:41 PM Muchun Song <muchun.song@linux.dev> wrote:
>> 
>>> On Jul 10, 2026, at 03:04, James Houghton <jthoughton@google.com> wrote:
>>> 
>>> On Thu, Jul 9, 2026 at 2:55 AM Muchun Song <muchun.song@linux.dev> wrote:
>>>>> On Jul 9, 2026, at 00:49, James Houghton <jthoughton@google.com> wrote:
>>>>> 
>>>>> On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
>>>>>> Do you mean that the support for AF might vary across different CPUs?
>>>>>> I'm not that familiar with arm64, so it seems a bit strange to me that
>>>>>> such basic hardware features can differ so much from one CPU to another.
>>>>> 
>>>>> Yes, hardware updates of the Access Flag is a per-CPU feature. It is
>>>>> available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
>>>>> register; each CPU has its own. (Linux will always enable HW AF for a
>>>>> CPU when it is onlined[1] if support is advertised, so we simply need
>>>>> to check if support is advertised to know that it is in fact enabled.)
>>>>> 
>>>>> These days it is not uncommon for a system to have two (or more?)
>>>>> different core implementations, like with "fast" cores and "efficient"
>>>>> cores.
>>>>> 
>>>>> [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S
>>>> 
>>>> Thanks for your detailed explanation. When enabling HVO via the cmdline,
>>>> can we simply prevent CPUs that do not support AF from coming online?
>>>> Would implementing it this way be much simpler? In practice, developers
>>>> definitely know whether the current system is suitable for enabling HVO.
>>>> If some CPUs do not support AF, they would just need to evaluate the
>>>> trade-off between memory savings and having fewer online CPUs than expected.
>>>> 
>>>> For scenarios where HVO is enabled via sysctl, we simply need to check
>>>> if all CPUs support AF. If any do not, the system should return an error.
>>>> 
>>>> Then, we can proceed with the Pre-HVO.
>>> 
>>> I don't think it makes sense to try to implement pre-HVO.
>>> 
>>> We cannot do HVO if any boot CPUs do not support HW AF, as HW AF will
>>> be required to free the HugeTLB pages later, which we should continue
>>> to support. Pre-HVO (today anyway) happens before all boot CPUs are
>>> onlined. IMO it is not okay to prevent boot CPUs from onlining.
>> 
>> When the boot CPU starts, it first checks if the hardware (the current
>> boot CPU) supports AF (The selection of the Boot CPU is critical here
>> If the user really wants to enable HVO). If AF is supported and HVO is
>> enabled via cmdline, we perform a Pre-HVO. Subsequently, any CPUs that
>> do not support AF will be prevented from coming online.
> 
> I understand. I don't think we should do this. I think it's much
> better to always be able to start all boot CPUs[1] and simply rely on
> "normal" HVO.
> 
> [1] Just to be clear on terminology, I'm using "boot CPUs" to refer to
> all CPUs that are onlined at boot-time, not just to *the* boot CPU
> that starts the kernel.

Thanks for the clarification. We previously had different understandings
of the boot CPU, but that should be resolved now.

> 
>> 
>> If the boot CPU does not support AF, enabling HVO is not permitted—neither
>> via cmdline nor sysctl. All CPUs are allowed to be online.
>> 
>> Once the system has started, there is only one way to enable HVO, and
>> that is through sysctl. 1) If the current system includes any CPUs that do
>> not support AF, enabling HVO is not permitted (unless the user chooses to
>> take these CPUs that don't support AF offline). 2) If all online CPUs in the
>> current system support AF, then HVO can be enabled. In this case, the
>> system must also block any CPUs that do not support AF from coming online
>> in the future.
> 
> An approach similar to this is implemented in this series: if there
> are (or were) any HVOed folios, onlining a CPU without HW AF is not
> allowed. If HW AF is not supported by any onlined CPU, HVO cannot
> proceed (though it will appear to be enabled).
> 
> I probably should have integrated
> `arch_hugetlb_vmemmap_optimization_supported()` and a `hvo_status`
> check into the sysctl handler for HVO. I'll do this if it is still
> relevant in v2.
> 
>> Of course, if a user specifically wants those CPUs to be
>> allowed to come online, they can choose to disable HVO first (At the same time,
>> this means that the HVO-optimized HugeTLB must be freed first as well).
> 
> I could make it so that, if there are no HVOed pages anymore, onlining
> incompatible CPUs becomes permitted again. I don't really see the need
> to do this.
> 
>> My proposal assumes that even though the system supports different types of
>> CPUs, their support for AF (I suppose this is a very basic feature nowadays)
>> is generally the same. I believe that for servers (Only servers stand to
>> benefit more from HVO), there shouldn't be a mix of those that support AF
>> and those that don't. At the very least, such a situation would be extremely
>> uncommon. I believe this may be acceptable. That said, if we go this route,
>> is there a way to simplify the code implementation even further?. Of course,
>> this is just my personal speculation. Please let me know if I've missed
>> anything.
> 
> I agree that systems where HVO is relevant are very unlikely to have
> mixed support for HW AF.
> 
> I don't think the approach you're suggesting in this email (as I've
> understood it) simplifies this series.
> 
> I don't want to mess with the arm64 cpufeature bits more than I've
> done here; preventing boot CPUs from onlining will certainly be more
> work. :) And then your other suggestions seem to be quite similar to
> what is already in these patches.
> 
> I am happy to get rid of the "disable HVO if it is unused and a CPU
> without support is onlined" part though and replace it with something
> similar (the simplification from the other thread[2]).

Thanks for the explanation! My main goal right now is to keep the initial
implementation as simple as possible. If any of my suggestions aren't quite
on point, feel free to adjust them as you see fit.

Thanks.

> 
> [2] https://lore.kernel.org/linux-mm/674EC6D6-B1F7-447D-B462-861ECA5A99B7@linux.dev/