Re: [PATCH 0/8] *** RFC: ARM KVM dirty tracking device ***

Re: [PATCH 0/8] *** RFC: ARM KVM dirty tracking device ***

[Oliver Upton]

Hi Lilit,

+cc kvmarm mailing list, get_maintainer is your friend :)

On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote:
> An example of a device that tracks accesses to stage-2 translations and will
> implement page_tracking_device interface is AWS Graviton Page Tracking Agent
> (PTA). We'll be posting code for the Graviton PTA device driver in a separate
> series of patches.

In order to actually review these patches, we need to see an
implementation of such a page tracking device. Otherwise it's hard to
tell that the interface accomplishes the right abstractions.

Beyond that, I have some reservations about maintaining support for
features that cannot actually be tested outside of your own environment.

> When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to
> use it via the same interface most likely with adaptations.

Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS
in hardware?

I think the best way forward here is to implement the architecture, and
hopefully after that your legacy driver can be made to fit the
interface. The FVP implements FEAT_HDBSS, so there's some (slow)
reference hardware to test against.

This is a very interesting feature, so hopefully we can move towards
something workable.

-- 
Thanks,
Oliver

Re: [PATCH 0/8] *** RFC: ARM KVM dirty tracking device ***

[Janpoladyan, Lilit]

Hi Oliver,

On 19.09.24, 11:12, "Oliver Upton" <oliver.upton@linux.dev <mailto:oliver.upton@linux.dev>> wrote:

> Hi Lilit,


> +cc kvmarm mailing list, get_maintainer is your friend :)


> On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote:
> > An example of a device that tracks accesses to stage-2 translations and will
> > implement page_tracking_device interface is AWS Graviton Page Tracking Agent
> > (PTA). We'll be posting code for the Graviton PTA device driver in a separate
> > series of patches.


> In order to actually review these patches, we need to see an
> implementation of such a page tracking device. Otherwise it's hard to
> tell that the interface accomplishes the right abstractions.

We'll be posting driver patches in the coming weeks, they should explain device
functionality.


> Beyond that, I have some reservations about maintaining support for
> features that cannot actually be tested outside of your own environment.

I understand, we'll see how we can emulate the functionality and make interface
testable.

> > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to
> > use it via the same interface most likely with adaptations.


> Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS
> in hardware?

We'd need to keep the interface for as long as hardware without FEAT_HDBSS
but with PTA is in use, hence the attempt of generalisation.

> I think the best way forward here is to implement the architecture, and
> hopefully after that your legacy driver can be made to fit the
> interface. The FVP implements FEAT_HDBSS, so there's some (slow)
> reference hardware to test against.

Thanks for the idea, we'll test with FVP, but we'd need FEAT_HDBSS
documentation for that. I don't think it's available yet, is it?

> This is a very interesting feature, so hopefully we can move towards
> something workable.


> -- 
> Thanks,
> Oliver

Thanks for the feedback, we'll be working on the discussed points,
Lilit




Amazon Web Services Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH 0/8] *** RFC: ARM KVM dirty tracking device ***

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

On Thu, 2024-09-19 at 02:11 -0700, Oliver Upton wrote:
> Hi Lilit,
> 
> +cc kvmarm mailing list, get_maintainer is your friend :)
> 
> On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote:
> > An example of a device that tracks accesses to stage-2 translations and will
> > implement page_tracking_device interface is AWS Graviton Page Tracking Agent
> > (PTA). We'll be posting code for the Graviton PTA device driver in a separate
> > series of patches.
> 
> In order to actually review these patches, we need to see an
> implementation of such a page tracking device. Otherwise it's hard to
> tell that the interface accomplishes the right abstractions.

Absolutely. That one is coming soon, but I was chasing the team to post
the API and KVM glue parts as early as possible to kick-start the
discussion, especially about the upcoming architectural solution.

> Beyond that, I have some reservations about maintaining support for
> features that cannot actually be tested outside of your own environment.

That's more about the hardware driver itself which will follow, than
the core API posted here.

I understand the reservation, but I think it's fine. In general, Linux
does support esoteric hardware that not everyone can test every
time. We do sweeping changes across all Ethernet drivers, for example,
and some of those barely even exist any more.

This particular device should be available on bare metal EC2 instances,
of course, but perhaps we should also implement it in QEMU. That would
actually be beneficial for our internal testing anyway, as it would
allow us to catch regressions much earlier in our own development
process.

> > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to
> > use it via the same interface most likely with adaptations.
> 
> Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS
> in hardware?

I don't think there is a definitive answer to that which is ready to
tape out, but it certainly seems possible that future generations will
eventually move to FEAT_HDBSS, maybe even reaching production by the
end of the decade, at the earliest? And then a decade or two later, the
existing hardware generations might even get retired, yes¹.

¹ #include <forward-looking statement.disclaimer>

> I think the best way forward here is to implement the architecture, and
> hopefully after that your legacy driver can be made to fit the
> interface. The FVP implements FEAT_HDBSS, so there's some (slow)
> reference hardware to test against.

Is there actually any documentation available about FEAT_HDBSS? We've
been asking, but haven't received it. I can find one or two mentions
e.g. https://arm.jonpalmisc.com/2023_09_sysreg/AArch64-hdbssbr_el2 but
nothing particularly useful.

The main reason for posting this series early is to make sure we do all
we can to accommodate FEAT_HDBSS. It's not the *end* of the world if
the kernel-internal API has to be tweaked slightly when FEAT_HDBSS
actually becomes reality in future, but obviously we'd prefer to
support it right from the start.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]

Re: [PATCH 0/8] *** RFC: ARM KVM dirty tracking device ***

[Oliver Upton]

On Thu, Sep 26, 2024 at 11:00:39AM +0100, David Woodhouse wrote:
> > Beyond that, I have some reservations about maintaining support for
> > features that cannot actually be tested outside of your own environment.
> 
> That's more about the hardware driver itself which will follow, than
> the core API posted here.
> 
> I understand the reservation, but I think it's fine. In general, Linux
> does support esoteric hardware that not everyone can test every
> time. We do sweeping changes across all Ethernet drivers, for example,
> and some of those barely even exist any more.

Of course, but I think it is also reasonable to say that ethernet
support in the kernel is rather mature with a good variety of hardware.
By comparison, what we have here is a brand new driver interface with
architecture / KVM code, which is pretty rare, with a single
implementation.

I'm perfectly happy to tinker on page tracking interface(s) in the
future w/o testing everything, but I must insist that we have *some*
way of testing the initial infrastructure before even considering taking
it.

> This particular device should be available on bare metal EC2 instances,
> of course, but perhaps we should also implement it in QEMU. That would
> actually be beneficial for our internal testing anyway, as it would
> allow us to catch regressions much earlier in our own development
> process.

QEMU would be interesting, but hardware is always welcome too ;-)

> > > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to
> > > use it via the same interface most likely with adaptations.
> > 
> > Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS
> > in hardware?
> 
> I don't think there is a definitive answer to that which is ready to
> tape out, but it certainly seems possible that future generations will
> eventually move to FEAT_HDBSS, maybe even reaching production by the
> end of the decade, at the earliest? And then a decade or two later, the
> existing hardware generations might even get retired, yes¹.
> 
> ¹ #include <forward-looking statement.disclaimer>

Well, hopefully that means you folks will look after it then :)

> > I think the best way forward here is to implement the architecture, and
> > hopefully after that your legacy driver can be made to fit the
> > interface. The FVP implements FEAT_HDBSS, so there's some (slow)
> > reference hardware to test against.
> 
> Is there actually any documentation available about FEAT_HDBSS? We've
> been asking, but haven't received it. I can find one or two mentions
> e.g. https://arm.jonpalmisc.com/2023_09_sysreg/AArch64-hdbssbr_el2 but
> nothing particularly useful.

Annoyingly no, the Arm ARM tends to lag the architecture by quite a bit.
The sysreg XML (from which I think this website is derived) gets updated
much more frequently.

> The main reason for posting this series early is to make sure we do all
> we can to accommodate FEAT_HDBSS. It's not the *end* of the world if
> the kernel-internal API has to be tweaked slightly when FEAT_HDBSS
> actually becomes reality in future, but obviously we'd prefer to
> support it right from the start.

Jury is still out on how FEAT_HDBSS is gonna fit with this PTA stuff.
I'm guessing your hardware has some way of disambiguating dirtied
addresses by VMID.

The architected solution, OTOH, is tied to a particular stage-2 MMU
configuration. KVM proper might need to manage the dirty tracking
hardware in that case as it'll need to be context switched on the
vcpu_load() / vcpu_put() boundary.

-- 
Thanks,
Oliver