Re: [LSF/MM/BPF TOPIC] Virtual Machine Memory Passthrough

[Pasha Tatashin <pasha.tatashin@soleen.com>]

On Wed, Feb 22, 2023 at 10:31 AM Zi Yan <ziy@nvidia.com> wrote:
>
> On 22 Feb 2023, at 8:43, Pasha Tatashin wrote:
>
> > On Mon, Feb 20, 2023 at 6:51 PM Gavin Shan <gshan@redhat.com> wrote:
> >>
> >> Hi Pasha,
> >>
> >> On 2/21/23 3:31 AM, Pasha Tatashin wrote:
> >>>
> >>> As a part of an ongoing work of replacing some containerized work load
> >>> with virtual machines within Google, I have worked on making the
> >>> memory translations faster.
> >>>
> >>> I would like to propose the following topic for this year's LSF/MM/BPF:
> >>>
> >>> Discuss  a set of techniques that can improve the guest performance,
> >>> memory footprint overhead, observability, and manageability of virtual
> >>> machines by hypervirtualizing the guest memory to the extreme. The end
> >>> goal is to allow very lightweight virtual machines to be closer in
> >>> performance to the containers.
> >>>
> >>> The following items are going to be discussed in this topic:
> >>> - Reducing the cost of SLAT page table translations.
> >>> - Reducing the memory footprint overhead.
> >>> - Reducing the memory management overhead.
> >>> - Increasing the observability of guest memory.
> >>>
> >>
> >> It's all about to understand the problem and possible solution or directions.
> >>
> >> I googled for 'SLAT' and direct me to x86's EPT. ARM64 has similar thing called
> >> stage-2 page table. The usual way to reduce page table translation cost is to map
> >> the contiguous memory through PUD/PMD. I'm not sure if there are other solutions
> >> we're heading for?
> >>
> >> Guest's memory is usually backed up by virtual memory area (VMA), which is either
> >> a anonymous or hugetlb region. As I understand, the page fault handling is excessive
> >> to populate the requested memory. I'm not sure if reducing the memory management
> >> overhead is to get it faster, or something else? :)
> >
> > Hi Gavin,
> >
> > In a non-virtualized environment, when converting VA to PA, we load
> > each level of page table, so converting to a 4K page takes 4 or 5
> > loads, depending on the page table type used. However, in a
> > virtualized environment, the number of loads to convert guest VA to
> > host PA is not a summation of SLAT page table levels and Guest page
> > table levels; rather, it is equal to: n*m + n + m. This is because
> > each guest's page table level must also be converted from guest PA to
> > host PA.
> >
> > One way to minimize the number of loads is for the guest to use huge
> > pages, for example, 1-Gbyte pages. However, this normally wastes a lot
> > of memory. The idea is that we can use guest physical memory in a
> > virtual way: create 1-Gbyte pages that are only partially backed by
> > host memory, yet improve the access performance due to fewer TLB
> > misses and faster translations through guest + SLAT page tables. I
> > would like to discuss how this can be achieved.
>
> Do you mean allocating 1GB pages in the guest and backing them using
> 2MB and/or 4KB pages in the host? From my understanding, for virtual

Yes, that is exactly right. However, backing only a subset of the 1G
page, and not zero the whole page on allocation or first fault in the
guest, fault on demand on host, as new parts of 1G page are touched.

> machines, TLB caches guestVA to hostPA, so the number of TLB entries
> would be the same as using 2MB or 4KB pages in the guest (as long as
> the guest page and the host page backing it have the same size).
> What am I missing here?

Yes, the way TLB works is the smallest page of host and guest is the
size of TLB entry. So, 1G guest pages and 2M host pages yield to 2M
TLB entries, and  1G guest pages and 4K host pages yield to 4K TLB
entries. The sabing is coming from always having 1G pages in the
guest, and if the host backs with 2M pages, the 2M TLB entries are
used.

>
> For a TLB miss, it will be faster since fewer page table walks are
> needed for 1GB pages in the guest.

That is exactly right, the faster page table walk or SLAT translation
is achieved with this approach.

Thanks,
Pasha