potential tss trampling, assumptions about physical memory layout

potential tss trampling, assumptions about physical memory layout

[Gabe Black]

    While continuing to try to get KVM and our M5 simulator to work
together, I ran into another issue.

    During VCPU bring up in x86 under VMX, the function init_rmode_tss
is called which seems to be writing an initial version of a TSS into
guest memory. It's not immediately clear to me why that would be a
necessary part of CPU initialization since, according to my
understanding of the ISA, the TSS should only be written by CPU itself
(as apposed to a regular load or store) when doing a task switch. This
doesn't present an immediate problem for us, but if CPUs are not all
started at the same time, it would be possible for them to trample a
shared TSS or whatever would end up beneath it.

    There has also been a strange problem that crops up when the tss is
put at an unusual address (more on that next) where copy_to_user within
kvm_clear_guest_page will work or not work depending on seemingly
arbitrary circumstances. Are there any restrictions on how to mmap the
backing memory or on how it's used at that point in CPU initialization?
I'd need to look into this more to get a good idea of what's going on,
but it seems to defy logic and any guidance would be useful.

    init_rmode_tss uses a function called rmode_tss_base to figure out
where to put this initial tss, and there seem to be a few assumptions
about what the physical memory layout looks like that are not
necessarily true. The version of the code in the kernel I'm working with
(2.6.28.9) looks like the following:

static gva_t rmode_tss_base(struct kvm *kvm)
{
        if (!kvm->arch.tss_addr) {
                gfn_t base_gfn = kvm->memslots[0].base_gfn +
                                 kvm->memslots[0].npages - 3;
                return base_gfn << PAGE_SHIFT;
        }
        return kvm->arch.tss_addr;
}

    The first assumption is that there is a valid element in the
memslots array. If not the guest may be in trouble anyway, but the
kernel may behave unpredictably in that case. Second, this code seems to
assume what I believe is the standard QEMU style memory layout with a
chunk of physical memory from 0x0 - 0xA0000, the VGA framebuffer, and
then 0xC0000 and upwards. In that case, memslots[0] would end at
0xA0000, and going three pages back from the end would still be a valid
realmode address. In our case, I was not leaving a gap for VGA and
instead allocating all ~500 MB of memory in one chunk, so three pages
from the end of memslot[0] was WAY outside the bounds of real mode
addressing. If this TSS is indeed for use by the guest in real mode,
that doesn't seem like it would work. Finally, it assumes that
memslots[0].npages is greater than 3 in the first place. If it isn't,
you'll end up with wrapping and try to install a real mode TSS at the
end of the address space, even farther from what real mode can handle
and potentially entirely invalid.

Gabe

Re: potential tss trampling, assumptions about physical memory layout

[Avi Kivity]

Gabe Black wrote:
>     While continuing to try to get KVM and our M5 simulator to work
> together, I ran into another issue.
>
>     During VCPU bring up in x86 under VMX, the function init_rmode_tss
> is called which seems to be writing an initial version of a TSS into
> guest memory. It's not immediately clear to me why that would be a
> necessary part of CPU initialization since, according to my
> understanding of the ISA, the TSS should only be written by CPU itself
> (as apposed to a regular load or store) when doing a task switch. This
> doesn't present an immediate problem for us, but if CPUs are not all
> started at the same time, it would be possible for them to trample a
> shared TSS or whatever would end up beneath it.
>
>     There has also been a strange problem that crops up when the tss is
> put at an unusual address (more on that next) where copy_to_user within
> kvm_clear_guest_page will work or not work depending on seemingly
> arbitrary circumstances. Are there any restrictions on how to mmap the
> backing memory or on how it's used at that point in CPU initialization?
> I'd need to look into this more to get a good idea of what's going on,
> but it seems to defy logic and any guidance would be useful.
>
>     init_rmode_tss uses a function called rmode_tss_base to figure out
> where to put this initial tss, and there seem to be a few assumptions
> about what the physical memory layout looks like that are not
> necessarily true. The version of the code in the kernel I'm working with
> (2.6.28.9) looks like the following:
>
> static gva_t rmode_tss_base(struct kvm *kvm)
> {
>         if (!kvm->arch.tss_addr) {
>                 gfn_t base_gfn = kvm->memslots[0].base_gfn +
>                                  kvm->memslots[0].npages - 3;
>                 return base_gfn << PAGE_SHIFT;
>         }
>         return kvm->arch.tss_addr;
> }
>
>     The first assumption is that there is a valid element in the
> memslots array. If not the guest may be in trouble anyway, but the
> kernel may behave unpredictably in that case. Second, this code seems to
> assume what I believe is the standard QEMU style memory layout with a
> chunk of physical memory from 0x0 - 0xA0000, the VGA framebuffer, and
> then 0xC0000 and upwards. In that case, memslots[0] would end at
> 0xA0000, and going three pages back from the end would still be a valid
> realmode address. In our case, I was not leaving a gap for VGA and
> instead allocating all ~500 MB of memory in one chunk, so three pages
> from the end of memslot[0] was WAY outside the bounds of real mode
> addressing. If this TSS is indeed for use by the guest in real mode,
> that doesn't seem like it would work. Finally, it assumes that
> memslots[0].npages is greater than 3 in the first place. If it isn't,
> you'll end up with wrapping and try to install a real mode TSS at the
> end of the address space, even farther from what real mode can handle
> and potentially entirely invalid.
>   

That's an early kvm design error, it was corrected with KVM_SET_TSS_ADDR.


-- 
error compiling committee.c: too many arguments to function