Re: [LSF/MM/BPF TOPIC] Virtual Machine Memory Passthrough

[Pasha Tatashin <pasha.tatashin@soleen.com>]

On Mon, Feb 20, 2023 at 6:51 PM Gavin Shan <gshan@redhat.com> wrote:
>
> Hi Pasha,
>
> On 2/21/23 3:31 AM, Pasha Tatashin wrote:
> >
> > As a part of an ongoing work of replacing some containerized work load
> > with virtual machines within Google, I have worked on making the
> > memory translations faster.
> >
> > I would like to propose the following topic for this year's LSF/MM/BPF:
> >
> > Discuss  a set of techniques that can improve the guest performance,
> > memory footprint overhead, observability, and manageability of virtual
> > machines by hypervirtualizing the guest memory to the extreme. The end
> > goal is to allow very lightweight virtual machines to be closer in
> > performance to the containers.
> >
> > The following items are going to be discussed in this topic:
> > - Reducing the cost of SLAT page table translations.
> > - Reducing the memory footprint overhead.
> > - Reducing the memory management overhead.
> > - Increasing the observability of guest memory.
> >
>
> It's all about to understand the problem and possible solution or directions.
>
> I googled for 'SLAT' and direct me to x86's EPT. ARM64 has similar thing called
> stage-2 page table. The usual way to reduce page table translation cost is to map
> the contiguous memory through PUD/PMD. I'm not sure if there are other solutions
> we're heading for?
>
> Guest's memory is usually backed up by virtual memory area (VMA), which is either
> a anonymous or hugetlb region. As I understand, the page fault handling is excessive
> to populate the requested memory. I'm not sure if reducing the memory management
> overhead is to get it faster, or something else? :)

Hi Gavin,

In a non-virtualized environment, when converting VA to PA, we load
each level of page table, so converting to a 4K page takes 4 or 5
loads, depending on the page table type used. However, in a
virtualized environment, the number of loads to convert guest VA to
host PA is not a summation of SLAT page table levels and Guest page
table levels; rather, it is equal to: n*m + n + m. This is because
each guest's page table level must also be converted from guest PA to
host PA.

One way to minimize the number of loads is for the guest to use huge
pages, for example, 1-Gbyte pages. However, this normally wastes a lot
of memory. The idea is that we can use guest physical memory in a
virtual way: create 1-Gbyte pages that are only partially backed by
host memory, yet improve the access performance due to fewer TLB
misses and faster translations through guest + SLAT page tables. I
would like to discuss how this can be achieved.

Thanks,
Pasha

>
> Thanks,
> Gavin
>