Scheduling of VCPUs and allocation of Guest Physical Memory

Scheduling of VCPUs and allocation of Guest Physical Memory

[Sukanto Ghosh]

Hi all,

I have two questions regarding KVM:

i) Is a VCPU scheduled like a normal Linux process or there is some 
mechanism by which performance guarantees can be provided to the VCPUs ?

ii) Who does the allocation of host physical memory to the guest, is it 
KVM module or the associated QEmu process ?

Also, I would be glad if you can provide me pointers to some 
documentation/paper/literature which discusses KVM in greater detail 
than what the paper "kvm: The Linux Virtual Machine Monitor" presents.

Thanks,
Sukanto

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Anthony Liguori]

Sukanto Ghosh wrote:
> Hi all,
>
> I have two questions regarding KVM:
>
> i) Is a VCPU scheduled like a normal Linux process

A normal process.

> or there is some mechanism by which performance guarantees can be 
> provided to the VCPUs ?

FWIW, there are various mechanisms to do scheduler tuning in Linux.  
cpusets and cgroups both provide mechanisms to ensure "performance 
guarantees".

> ii) Who does the allocation of host physical memory to the guest, is 
> it KVM module or the associated QEmu process ?

The QEMU process allocates the memory via malloc().

> Also, I would be glad if you can provide me pointers to some 
> documentation/paper/literature which discusses KVM in greater detail 
> than what the paper "kvm: The Linux Virtual Machine Monitor" presents.

The only other reference is the code itself.

Regards,

Anthony Liguori

> Thanks,
> Sukanto
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Sukanto Ghosh]

Anthony Liguori wrote:
> Sukanto Ghosh wrote:
>> Hi all,
>>
>> I have two questions regarding KVM:
>>
>> i) Is a VCPU scheduled like a normal Linux process
>
> A normal process.
>
>> or there is some mechanism by which performance guarantees can be 
>> provided to the VCPUs ?
>
> FWIW, there are various mechanisms to do scheduler tuning in Linux.  
> cpusets and cgroups both provide mechanisms to ensure "performance 
> guarantees".
are any of these mechanisms currently being used in kvm ?

>> ii) Who does the allocation of host physical memory to the guest, is 
>> it KVM module or the associated QEmu process ?
>
> The QEMU process allocates the memory via malloc().
Doesn't the QEMU process and the guest have different address-spaces ? 
So, how can it malloc for the guest ? I thought the QEMU process 
requests for the memory allocation to the kvm module via the /dev/kvm 
device node.


Thanks and regards
Sukanto

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Anthony Liguori]

Sukanto Ghosh wrote:
>>> ii) Who does the allocation of host physical memory to the guest, is 
>>> it KVM module or the associated QEmu process ?
>>
>> The QEMU process allocates the memory via malloc().
> Doesn't the QEMU process and the guest have different address-spaces ? 
> So, how can it malloc for the guest ? I thought the QEMU process 
> requests for the memory allocation to the kvm module via the /dev/kvm 
> device node.

Yes, the guest has a different address space from the host.  The QEMU 
process malloc()'s the physical memory for the guest, and tells KVM what 
the region is (via an ioctl to /dev/kvm).  When the guest tries to 
create an address space mapping a guest VA to a guest PA, KVM will build 
that mapping by using virtual memory allocated by QEMU.  To do this, it 
uses the mapping to translate guest PA to host PA.  Then it will build 
an address space for the guest mapping guest VA to host PA.

Regards,

Anthony Liguori

>
> Thanks and regards
> Sukanto
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Sukanto Ghosh]

Anthony Liguori wrote:
> Sukanto Ghosh wrote:
>>>> ii) Who does the allocation of host physical memory to the guest, 
>>>> is it KVM module or the associated QEmu process ?
>>>
>>> The QEMU process allocates the memory via malloc().
>> Doesn't the QEMU process and the guest have different address-spaces 
>> ? So, how can it malloc for the guest ? I thought the QEMU process 
>> requests for the memory allocation to the kvm module via the /dev/kvm 
>> device node.
>
> Yes, the guest has a different address space from the host.  The QEMU 
> process malloc()'s the physical memory for the guest, and tells KVM 
> what the region is (via an ioctl to /dev/kvm).


Why does the QEMU process needs to malloc() the physical memory for the 
guest ? Why can't it be done by the kernel itself ? Is it because the 
said pages will be sharable between the QEMU process and the guest, 
which will aid the QEMU process while performing DMA.



Also, when and how are host-initiated virtual interrupts delivered?  I 
guess, that it is done at the time of VM-entry. But what about the 
interrupts that need to be delivered immediately and while the VM is 
executing (assume interrupts are enabled in the guest). Is any kind of 
signal-based mechanism employed to force a VM-exit ?



Thanks and Regards,

Sukanto Ghosh

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Avi Kivity]

Sukanto Ghosh wrote:
>>
>> Yes, the guest has a different address space from the host.  The QEMU 
>> process malloc()'s the physical memory for the guest, and tells KVM 
>> what the region is (via an ioctl to /dev/kvm).
>
>
> Why does the QEMU process needs to malloc() the physical memory for 
> the guest ? Why can't it be done by the kernel itself ? Is it because 
> the said pages will be sharable between the QEMU process and the 
> guest, which will aid the QEMU process while performing DMA.
>

Userspace allocation is done in order to allow flexibility in how memory 
is allocated.  Userspace can choose to allocate large pages, apply a 
numa policy to the memory, mlock() it, etc.

It also makes swapping simple, as Linux already knows how to swap 
userspace memory.

Also, s390 has to do it this way.

>
>
> Also, when and how are host-initiated virtual interrupts delivered?  I 
> guess, that it is done at the time of VM-entry. 

Yes.

> But what about the interrupts that need to be delivered immediately 
> and while the VM is executing (assume interrupts are enabled in the 
> guest). Is any kind of signal-based mechanism employed to force a 
> VM-exit ?
>

Sending a signal to a task that is executing guest code will force it to 
exit to userspace immediately.  The in-kernel interrupt controller 
emulation also forces guest exits by sending inter-processor interrupts.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Sukanto Ghosh]

>> But what about the interrupts that need to be delivered immediately 
>> and while the VM is executing (assume interrupts are enabled in the 
>> guest). Is any kind of signal-based mechanism employed to force a 
>> VM-exit ?
>>
>
> Sending a signal to a task that is executing guest code will force it 
> to exit to userspace immediately.  The in-kernel interrupt controller 
> emulation also forces guest exits by sending inter-processor interrupts.
>
>
I saw that KVM has guest SMP support upto 4 VCPUs. Are these VCPUs 
modelled as threads of 'the process representing the VM' ?

If it is, then, when a signal is sent to the that process by the QEMU 
process, shouldn't all of the VCPUs (for that VM) exit to the userspace ?

Who does this management of delivering the interrupt to the VCPU for 
which it is intended for ? And, how ?


Also, while going through the KVM source, I found mention of opaque at 
many places. What does 'opaque' refer to ? What is an opaque data ?



Thanks and Regards,
Sukanto

Re: Scheduling of VCPUs and allocation of Guest Physical Memory

[Avi Kivity]

Sukanto Ghosh wrote:
> I saw that KVM has guest SMP support upto 4 VCPUs. Are these VCPUs 
> modelled as threads of 'the process representing the VM' ?
>

Yes.

> If it is, then, when a signal is sent to the that process by the QEMU 
> process, shouldn't all of the VCPUs (for that VM) exit to the userspace ?
>

You can send signals to a specific thread.

> Who does this management of delivering the interrupt to the VCPU for 
> which it is intended for ?

See qemu-kvm.c, look for pthread_kill().

>
>
> Also, while going through the KVM source, I found mention of opaque at 
> many places. What does 'opaque' refer to ? What is an opaque data ?

It's a common C idiom letting a lower layer call back into the upper 
layer without knowing the upper layer's details.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.