Status of SMP?

Status of SMP?

[Gregory Haskins]

Hi all,
  So what is the status of SMP support and is someone working on this?  I have some ideas about how to implement this and can spend some time with it if it would be helpful.  I just dont want to duplicate efforts.

-Greg

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Avi Kivity]

Gregory Haskins wrote:
> Hi all,
>   So what is the status of SMP support and is someone working on this?  I have some ideas about how to implement this and can spend some time with it if it would be helpful.  I just dont want to duplicate efforts.
>   

SMP is planned, though of course help is appreciated.  I'd like to start 
on an NPT capable processor so that there's no need to worry about the 
mmu and bootstrap/ipi simultaneously.

If you have ideas, please share them!


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Fwd: Re: Status of SMP?

[Gregory Haskins]

BTW: Please excuse my stupid confusion of uP (microprocessor) with UP (uniprocessor).  Duh!

>>> On Tue, Mar 13, 2007 at 11:43 AM, in message <45F68E68.BA47.005A.0-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org>,
Gregory Haskins wrote: 
> >>> On Tue, Mar 13, 2007 at 11:09 AM, in message <45F6BEC0.1010906-atKUWr5tajBWk0Htik3J/w@public.gmane.org>,
> Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote: 
>> Gregory Haskins wrote:
>>
>>>
>>> Does this sound reasonable?
>>>   
>> 
>> Much simpler (at least to start with) is to send a signal.  The signal 
>> will interrupt the guest if it is in guest mode, or if it sleeping, it 
>> will wake up the thread.  Upon receiving the signal, the thread can 
>> inject an interrupt.
>> 
>> Later on, we may wish to optimize this to avoid the signal if we're 
>> certain to be in guest mode, and only send it if we're asleep.
> 
> I will investigate it from this perspective then...but I currently dont 
> understand how a signal delivery would cause the guest to be interrupted 
> (unless you guys have already put some trickery in the kvm-kmod ;).
> 
> I can follow the logic for a uP system (I think, therefore I am.....e.g. you 
> cannot be both host and guest mode at the same time).  However, in an SMP 
> system it would be conceivable for one task to be in host context, and 
> another to be in guest context simultaneously.  If a guest was already in VM 
> context, wouldnt the signal being delivered simply be posted to the 
> pending-signals for the task and get evaluated at the next VMEXIT?  The 
> method I was describing would actually invoke the VMEXIT to begin with.  
> Perhaps the best solution is a combination of both (post a signal and an IPI 
> which evaluates to no-op in host mode).  Forgive me if I am ignorant and/or 
> being obtuse. :)
> 
> Is there something I am missing?  Or is there an issue with SMP with this 
> approach?
> 
>> 
>> 
>>> The thing I cant quite wrap my head around is how this would work in the PV 
>> driver sense.  For SMP, its clear...one VCPU thread might be in QEMU context 
>> in the LAPIC model and send the FORCE_EXIT ioctl against another VCPU to 
>> deliever the IPI.  How does this work in a uP setup for PV?  Is there some 
>> other thread in QEMU that might be waiting for AIO completions (etc) that 
>> would then be able to invoke the ioctl?  Otherwise, our one and only thread 
>> is tied up in VM context so I dont get how it would work.
>>>   
>> 
>> The scenario is a host-  side kernel-  mode device receiving a network 
>> packet and wishing to interrupt a guest.  Here, the device receive 
>> handler is a separate thread of execution.
> 
> I see.  That makes sense.
> 
> -Greg
> 
> 
> 



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Nakajima, Jun]

Avi Kivity wrote:
> Gregory Haskins wrote:
>> Hi all,
>>   So what is the status of SMP support and is someone working on
>> this?  I have some ideas about how to implement this and can spend
>> some time with it if it would be helpful.  I just dont want to
>> duplicate efforts.   
>> 
> 
> SMP is planned, though of course help is appreciated.  I'd like to
> start on an NPT capable processor so that there's no need to worry
> about the mmu and bootstrap/ipi simultaneously.
> 
> If you have ideas, please share them!

The key is the virtual MMU; getting bootstrap/IPI working is
straightforward. Without NPT or EPT, we need to have an SMP-capable
shadow pagetable code. I think one of the most efficient ways is to
re-use the code from Xen. The current one is the third generation of the
shadow page table, and it works well. 

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Avi Kivity]

Nakajima, Jun wrote:
> The key is the virtual MMU; getting bootstrap/IPI working is
> straightforward. 

I'm glad to hear you say that.


> Without NPT or EPT, we need to have an SMP-capable
> shadow pagetable code. I think one of the most efficient ways is to
> re-use the code from Xen. The current one is the third generation of the
> shadow page table, and it works well. 
>   

I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would 
be rejected.  Linux development requires incremental patches.

I'm not familiar with the current Xen shadow code, but a cursory look 
gave the following impressions:

- it is single threaded, so there's no reason to expect more scalability 
out of it
- it is lighter weight, by not maintaining a full reverse mapping.  on 
the other hand, it is less generic (requires special handling for 
linearized page tables)
- does order 2 allocations (which work in Linux, but get less reliable 
as uptime increases)

I don't see a compelling reason to change here, but of course I may have 
missed something and I'm also biased.

Getting the kvm mmu to do smp involves the following, as far as I can tell:

- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have 
touched the domain, and later, for cpus that have touched the pagetable)

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Nakajima, Jun]

Avi Kivity wrote:
> Nakajima, Jun wrote:
>> The key is the virtual MMU; getting bootstrap/IPI working is
>> straightforward.
> 
> I'm glad to hear you say that.
> 
> 
>> Without NPT or EPT, we need to have an SMP-capable
>> shadow pagetable code. I think one of the most efficient ways is to
>> re-use the code from Xen. The current one is the third generation of
>> the shadow page table, and it works well.
>> 
> 
> I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would
> be rejected.  Linux development requires incremental patches.

I assumed that the current virutal MMU code was not designed to support
SMP (when you said "there's no need to worry about the mmu"). After
reviewing the code, I think the current code is good enough, especially
in the sense that it emulates writes to guest page tables. Actually the
code at least the emulator is from Xen ;-)

This particular code keeps the shadow page table in sync with the guest
right away, avoiding major incosistencies in the SMP environment.

		spin_lock(&vcpu->kvm->lock);
		r = kvm_mmu_page_fault(vcpu, cr2, error_code);
		if (r < 0) {
			spin_unlock(&vcpu->kvm->lock);
			return r;
		}
		if (!r) {
			spin_unlock(&vcpu->kvm->lock);
			return 1;
		}

		er = emulate_instruction(vcpu, kvm_run, cr2,
error_code);
		spin_unlock(&vcpu->kvm->lock);

> 
> Getting the kvm mmu to do smp involves the following, as far as I can
> tell: 
> 
> - verifying that all the locking in place is correct
> - make sure that spurious faults (due to races) are handled well
> - install shadow ptes atomically instead of building them
> incrementally 
> - replace local tlb flushes by remote tlb flushes (for cpus that have
> touched the domain, and later, for cpus that have touched the
> pagetable) 

The other thing is how do you detect when the guest page tables become
just data (this is not an SMP issue, though)? If you don't unshadow such
pages, we'll tend to get (frequent) unnecessary #PF. 

These are mostly performance optimization issues, but SMP guests will
use local and IO APICs (and other misc timers), and we'll see more
frequent VM exits in such a guest. We may need to move them to the KVM
side if we see performance issues. 

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Avi Kivity]

Nakajima, Jun wrote:
> I assumed that the current virutal MMU code was not designed to support
> SMP (when you said "there's no need to worry about the mmu"). 

What I meant was that there is no need debug both the core smp support 
and the mmu smp support at the same time.  Certainly the mmu was written 
with smp support in mind.

> The other thing is how do you detect when the guest page tables become
> just data (this is not an SMP issue, though)? If you don't unshadow such
> pages, we'll tend to get (frequent) unnecessary #PF. 
>   

There are a few heuristics in place for zapping shadow page tables:

- user mode writes (fix_write_pf)
- unaligned writes (kvm_mmu_pre_write)
- lots of writes to the same page (kvm_mmu_pre_write)

We could probably also add a heuristic that checks that the contents 
look like a pte if PT_PRESENT is set.  This is useful mostly on pae 
where there are many reserved bits.

> These are mostly performance optimization issues, but SMP guests will
> use local and IO APICs (and other misc timers), and we'll see more
> frequent VM exits in such a guest. We may need to move them to the KVM
> side if we see performance issues. 
>   

We've actually done this, and found that performance was not affected.  
The reason is that the kernel->user transition is very fast compared to 
the guest->kernel transition.  It is still worthwhile to do this if we 
have to use the apic to inject interrupts from in-kernel paravirtualized 
devices, where it makes little sense to go back to userspace in order to 
inject the interrupt.

I'd also like to add a read-only memory type so that apic reads can 
proceed natively rather than be emulated (the TODO has this now).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Nakajima, Jun]

Avi Kivity wrote:
> 
> I'd also like to add a read-only memory type so that apic reads can
> proceed natively rather than be emulated (the TODO has this now).

I don't understand how this can work. Instead, we should use the
hardware-based virtualization for the local APIC.

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Avi Kivity]

Nakajima, Jun wrote:
> Avi Kivity wrote:
>   
>> I'd also like to add a read-only memory type so that apic reads can
>> proceed natively rather than be emulated (the TODO has this now).
>>     
>
> I don't understand how this can work. 

You're right - it can't.  I thought all read accesses are static and 
side-effect-free, but reading the timer is not static.

> Instead, we should use the
> hardware-based virtualization for the local APIC.
>   

Can you point me at documentation for that?  What I have only mentions 
tpr/cr8 shadow, but not much else.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Huang2, Wei]

I am currently working on supporting nested paging for KVM. As of now, I
am able to boot 32-bit Linux guest (ttylinux) under AMD-V nested paging.
My plan is to support a full-fledged of Linux guests and Windows guests
in the next few weeks.

The current KVM MMU code was not designed with hardware assisted paging
(i.e. nested paging or extended paging) in mind. For instance, some
functions in kvm_main.c (i.e., write_cr functions) need to become become
function pointers. Guest page table walking also needs some re-work. It
will be nice to have both SMP and hardware assisted paging (NPT and EPT)
in place for the next phase of KVM MMU code. I will be happy to provide
help and work on these these areas. :P

Best,

-Wei

Operating System Research Center
Advanced Micro Devices (AMD)

-----Original Message-----
From: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
[mailto:kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of Avi Kivity
Sent: Thursday, March 15, 2007 3:43 AM
To: Nakajima, Jun
Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
Subject: Re: [kvm-devel] Status of SMP?

Nakajima, Jun wrote:
> The key is the virtual MMU; getting bootstrap/IPI working is 
> straightforward.

I'm glad to hear you say that.


> Without NPT or EPT, we need to have an SMP-capable shadow pagetable 
> code. I think one of the most efficient ways is to re-use the code 
> from Xen. The current one is the third generation of the shadow page 
> table, and it works well.
>   

I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would
be rejected.  Linux development requires incremental patches.

I'm not familiar with the current Xen shadow code, but a cursory look
gave the following impressions:

- it is single threaded, so there's no reason to expect more scalability
out of it
- it is lighter weight, by not maintaining a full reverse mapping.  on
the other hand, it is less generic (requires special handling for
linearized page tables)
- does order 2 allocations (which work in Linux, but get less reliable
as uptime increases)

I don't see a compelling reason to change here, but of course I may have
missed something and I'm also biased.

Getting the kvm mmu to do smp involves the following, as far as I can
tell:

- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have
touched the domain, and later, for cpus that have touched the pagetable)

--
Do not meddle in the internals of kernels, for they are subtle and quick
to panic.


------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel





-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Huang2, Wei]

I am currently working on supporting nested paging for KVM. As of now, I
am able to boot 32-bit Linux guest (ttylinux) under AMD-V nested paging.
My plan is to support a full-fledged of Linux guests and Windows guests
in the next few weeks.

The current KVM MMU code was not designed with hardware assisted paging
(i.e. nested paging or extended paging) in mind. For instance, some
functions in kvm_main.c (i.e., write_cr functions) need to become become
function pointers. Guest page table walking also needs some re-work. It
will be nice to have both SMP and hardware assisted paging (NPT and EPT)
in place for the next phase of KVM MMU code. I will be happy to provide
help and work on these these areas. :P

Best,

-Wei
-----Original Message-----
From: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
[mailto:kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of Avi Kivity
Sent: Thursday, March 15, 2007 3:43 AM
To: Nakajima, Jun
Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
Subject: Re: [kvm-devel] Status of SMP?

Nakajima, Jun wrote:
> The key is the virtual MMU; getting bootstrap/IPI working is 
> straightforward.

I'm glad to hear you say that.


> Without NPT or EPT, we need to have an SMP-capable shadow pagetable 
> code. I think one of the most efficient ways is to re-use the code 
> from Xen. The current one is the third generation of the shadow page 
> table, and it works well.
>   

I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would
be rejected.  Linux development requires incremental patches.

I'm not familiar with the current Xen shadow code, but a cursory look
gave the following impressions:

- it is single threaded, so there's no reason to expect more scalability
out of it
- it is lighter weight, by not maintaining a full reverse mapping.  on
the other hand, it is less generic (requires special handling for
linearized page tables)
- does order 2 allocations (which work in Linux, but get less reliable
as uptime increases)

I don't see a compelling reason to change here, but of course I may have
missed something and I'm also biased.

Getting the kvm mmu to do smp involves the following, as far as I can
tell:

- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have
touched the domain, and later, for cpus that have touched the pagetable)

--
Do not meddle in the internals of kernels, for they are subtle and quick
to panic.


------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel





-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Status of SMP?

[Avi Kivity]

Huang2, Wei wrote:
> I am currently working on supporting nested paging for KVM. As of now, I
> am able to boot 32-bit Linux guest (ttylinux) under AMD-V nested paging.
> My plan is to support a full-fledged of Linux guests and Windows guests
> in the next few weeks.
>   

That's good to hear.  Can you post the patch you have now to the list? 
I'm quite interested in it, and I'm sure others are too.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV