Re: KVM on IBM PowerEN processor

Re: KVM on IBM PowerEN processor

[Alexander Graf]

Hi Kun,

On 08/29/2011 11:31 AM, Kun Wang wrote:
> Hi, everyone,
>
> This is Kun Wang from IBM Research China. I and my team have been working
> on IBM PowerEN processor in recent years, including its simulation,
> lib/runtime optimization and etc. Now we start the work to enable KVM on
> PowerEN processor. Since the A2 core of PowerEN follows Power ISA v2.06
> (more specifically, book3e and 64-bit), I believe 99% of our work will
> stick to the ISA, and hence can be leveraged by others.
>
> As the new one to this KVM world, I and my team definitely need your help.
> Looking forward to talking and working with you guys in the future.

Welcome to the PowerPC KVM world! I'm looking very much forward to 
working with you there. Please always CC kvm@vger.kernel.org for emails 
you send to kvm-ppc@vger, unless you think of it as completely 
off-generic-kvm discussions. I like the idea of having more clever 
people involved in the PPC KVM process and by CC'ing kvm@vger, we get 
more exposure.

The Freescale e500 cores are basically BookE 2.06 compliant with a few 
specialties here and there. Have you looked at their code? Sure, it's 
32-bit as is now but there is definitely work going on at Freescale to 
get it 64-bit too.

Maybe it makes sense to take the current code as a starting point and 
slowly move towards an architecture that is more generic. Maybe it makes 
sense to start from scratch and actually design something flexible that 
splits guest and host TLB code, so we could potentially run cross-book 
KVM in the future (which I would love to see happen! Just imagine 
running a PowerEN guest on a POWER7 system).

Either way, please make sure to coordinate any efforts with me and Scott 
so we don't walk off in different directions and only get to realize we 
went down the wrong path when there's already a 100 patches patch set on 
the mailing list :)


Again, welcome on board!

Alex

Re: KVM on IBM PowerEN processor

[Stuart Yoder]

On Tue, Aug 30, 2011 at 4:45 AM, Alexander Graf <agraf@suse.de> wrote:
> Hi Kun,
>
> On 08/29/2011 11:31 AM, Kun Wang wrote:
>>
>> Hi, everyone,
>>
>> This is Kun Wang from IBM Research China. I and my team have been working
>> on IBM PowerEN processor in recent years, including its simulation,
>> lib/runtime optimization and etc. Now we start the work to enable KVM on
>> PowerEN processor. Since the A2 core of PowerEN follows Power ISA v2.06
>> (more specifically, book3e and 64-bit), I believe 99% of our work will
>> stick to the ISA, and hence can be leveraged by others.
>>
>> As the new one to this KVM world, I and my team definitely need your help.
>> Looking forward to talking and working with you guys in the future.
>
> Welcome to the PowerPC KVM world! I'm looking very much forward to working
> with you there. Please always CC kvm@vger.kernel.org for emails you send to
> kvm-ppc@vger, unless you think of it as completely off-generic-kvm
> discussions. I like the idea of having more clever people involved in the
> PPC KVM process and by CC'ing kvm@vger, we get more exposure.
>
> The Freescale e500 cores are basically BookE 2.06 compliant with a few
> specialties here and there. Have you looked at their code? Sure, it's 32-bit
> as is now but there is definitely work going on at Freescale to get it
> 64-bit too.
>
> Maybe it makes sense to take the current code as a starting point and slowly
> move towards an architecture that is more generic. Maybe it makes sense to
> start from scratch and actually design something flexible that splits guest
> and host TLB code, so we could potentially run cross-book KVM in the future
> (which I would love to see happen! Just imagine running a PowerEN guest on a
> POWER7 system).
>
> Either way, please make sure to coordinate any efforts with me and Scott so
> we don't walk off in different directions and only get to realize we went
> down the wrong path when there's already a 100 patches patch set on the
> mailing list :)

Also, note--  we (Freescale) do have a patchset coming soon for the e500mc
which is a 2.06 embedded 32-bit CPU with category E.HV.    We've had this
running for a while and are close to finishing some cleanup before posting
upstream.  We are are also starting to look at 64-bit support for our e5500
CPU we should be a delta on top of the e500mc support.

Just want to make you aware of some current work that may overlap your
A2 KVM support...

Stuart

Re: KVM on IBM PowerEN processor

[Kun Wang]

于 2011-8-30 22:15, Stuart Yoder 写道:
> On Tue, Aug 30, 2011 at 4:45 AM, Alexander Graf<agraf@suse.de>  wrote:
>> Hi Kun,
>>
>> On 08/29/2011 11:31 AM, Kun Wang wrote:
>>> Hi, everyone,
>>>
>>> This is Kun Wang from IBM Research China. I and my team have been working
>>> on IBM PowerEN processor in recent years, including its simulation,
>>> lib/runtime optimization and etc. Now we start the work to enable KVM on
>>> PowerEN processor. Since the A2 core of PowerEN follows Power ISA v2.06
>>> (more specifically, book3e and 64-bit), I believe 99% of our work will
>>> stick to the ISA, and hence can be leveraged by others.
>>>
>>> As the new one to this KVM world, I and my team definitely need your help.
>>> Looking forward to talking and working with you guys in the future.
>> Welcome to the PowerPC KVM world! I'm looking very much forward to working
>> with you there. Please always CC kvm@vger.kernel.org for emails you send to
>> kvm-ppc@vger, unless you think of it as completely off-generic-kvm
>> discussions. I like the idea of having more clever people involved in the
>> PPC KVM process and by CC'ing kvm@vger, we get more exposure.
>>
>> The Freescale e500 cores are basically BookE 2.06 compliant with a few
>> specialties here and there. Have you looked at their code? Sure, it's 32-bit
>> as is now but there is definitely work going on at Freescale to get it
>> 64-bit too.
>>
>> Maybe it makes sense to take the current code as a starting point and slowly
>> move towards an architecture that is more generic. Maybe it makes sense to
>> start from scratch and actually design something flexible that splits guest
>> and host TLB code, so we could potentially run cross-book KVM in the future
>> (which I would love to see happen! Just imagine running a PowerEN guest on a
>> POWER7 system).
>>
>> Either way, please make sure to coordinate any efforts with me and Scott so
>> we don't walk off in different directions and only get to realize we went
>> down the wrong path when there's already a 100 patches patch set on the
>> mailing list :)
> Also, note--  we (Freescale) do have a patchset coming soon for the e500mc
> which is a 2.06 embedded 32-bit CPU with category E.HV.    We've had this
> running for a while and are close to finishing some cleanup before posting
> upstream.  We are are also starting to look at 64-bit support for our e5500
> CPU we should be a delta on top of the e500mc support.
>
> Just want to make you aware of some current work that may overlap your
> A2 KVM support...
>
> Stuart
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Stuart,

Thank you very much for your information. That's really helpful to us. I 
do think there is some overlap between your work and ours. Do you have 
any suggestions for us to avoid "re-inventing the wheel"?

Best regards,
Kun Wang

RE: KVM on IBM PowerEN processor

[Caraman Mihai Claudiu-B02008]

> -----Original Message-----
> From: kvm-ppc-owner@vger.kernel.org [mailto:kvm-ppc-
> owner@vger.kernel.org] On Behalf Of Kun Wang
> Sent: Tuesday, August 30, 2011 7:32 PM
> To: Stuart Yoder
> Cc: Kun Wang; kvm-ppc@vger.kernel.org; kvm@vger.kernel.org; Wood Scott-
> B07421; Alexander Graf
> Subject: Re: KVM on IBM PowerEN processor
> 
> ? 2011-8-30 22:15, Stuart Yoder ??:
> > On Tue, Aug 30, 2011 at 4:45 AM, Alexander Graf<agraf@suse.de>  wrote:
> >> Hi Kun,
> >>
> >> On 08/29/2011 11:31 AM, Kun Wang wrote:
> >>> Hi, everyone,
> >>>
> >>> This is Kun Wang from IBM Research China. I and my team have been
> >>> working on IBM PowerEN processor in recent years, including its
> >>> simulation, lib/runtime optimization and etc. Now we start the work
> >>> to enable KVM on PowerEN processor. Since the A2 core of PowerEN
> >>> follows Power ISA v2.06 (more specifically, book3e and 64-bit), I
> >>> believe 99% of our work will stick to the ISA, and hence can be
> leveraged by others.
> >>>
> >>> As the new one to this KVM world, I and my team definitely need your
> help.
> >>> Looking forward to talking and working with you guys in the future.
> >> Welcome to the PowerPC KVM world! I'm looking very much forward to
> >> working with you there. Please always CC kvm@vger.kernel.org for
> >> emails you send to kvm-ppc@vger, unless you think of it as completely
> >> off-generic-kvm discussions. I like the idea of having more clever
> >> people involved in the PPC KVM process and by CC'ing kvm@vger, we get
> more exposure.
> >>
> >> The Freescale e500 cores are basically BookE 2.06 compliant with a
> >> few specialties here and there. Have you looked at their code? Sure,
> >> it's 32-bit as is now but there is definitely work going on at
> >> Freescale to get it 64-bit too.
> >>
> >> Maybe it makes sense to take the current code as a starting point and
> >> slowly move towards an architecture that is more generic. Maybe it
> >> makes sense to start from scratch and actually design something
> >> flexible that splits guest and host TLB code, so we could potentially
> >> run cross-book KVM in the future (which I would love to see happen!
> >> Just imagine running a PowerEN guest on a
> >> POWER7 system).
> >>
> >> Either way, please make sure to coordinate any efforts with me and
> >> Scott so we don't walk off in different directions and only get to
> >> realize we went down the wrong path when there's already a 100
> >> patches patch set on the mailing list :)
> > Also, note--  we (Freescale) do have a patchset coming soon for the
> e500mc
> > which is a 2.06 embedded 32-bit CPU with category E.HV.    We've had
> this
> > running for a while and are close to finishing some cleanup before
> > posting upstream.  We are are also starting to look at 64-bit support
> > for our e5500 CPU we should be a delta on top of the e500mc support.
> >
> > Just want to make you aware of some current work that may overlap your
> > A2 KVM support...
> >
> > Stuart
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> > the body of a message to majordomo@vger.kernel.org More majordomo info
> > at  http://vger.kernel.org/majordomo-info.html
> Stuart,
> 
> Thank you very much for your information. That's really helpful to us. I
> do think there is some overlap between your work and ours. Do you have
> any suggestions for us to avoid "re-inventing the wheel"?
> 
> Best regards,
> Kun Wang
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

I am working on 64-bit support for Freescale e5500 core. I intend to submit the 64-bit patches as soon as e500mc patchset is applied. The changes are book3e compliant.

Regards,
Mike

Re: KVM on IBM PowerEN processor

[Benjamin Herrenschmidt]

On Tue, 2011-08-30 at 09:15 -0500, Stuart Yoder wrote:
> Also, note--  we (Freescale) do have a patchset coming soon for the e500mc
> which is a 2.06 embedded 32-bit CPU with category E.HV.    We've had this
> running for a while and are close to finishing some cleanup before posting
> upstream.  We are are also starting to look at 64-bit support for our e5500
> CPU we should be a delta on top of the e500mc support.
> 
> Just want to make you aware of some current work that may overlap your
> A2 KVM support... 

The main difference will be the MMU version (FSL is MAV1, A2 is MAV2),
and overall TLB structure.

The rest should be mostly identical :-)

I suspect the TLB virtualization code will be quite different, but it's
really going to depend what approach Kun decides to use.

You can probably start with doing simple SW load, but performance will
suck.

Other options that I've been thinking about to try to exploit indirect
TLB entries (HW load) in no specific order:

 - Use the LRAT. (How ? It's small, meant for huge translations ...
maybe on top of some large part support & demand fault it using a
really short asm code path ? Some kind of linear array of 16M
translations like we do on power7 KVM ?). Using the LRAT means that you
enable the guest direct use of indirect TLB.

 - Shadow page tables. Paravirtualize (or emulate) insertion of indirect
entries and have the HV maintain pre-translated shadows. That means all
page faults must go to KVM though, in order to handle protection
changes, and you need some reverse mapping to be able to deal with
invalidations, unless you "trick" with paravirt rather than full virt,
such as notifying KVM when you populate a PTE.

 - Like the above but additionally have KVM walk the guest page tables
directly and only forward to the guest actual PTE faults. Might require
some kind of hcall to tell kvm about the root of the tree (roots
actually) on context switch etc... might not be worth it but may also
save you overhead.

Cheers,
Ben.

Re: KVM on IBM PowerEN processor

[Kun Wang]

于 2011-8-30 17:45, Alexander Graf 写道:
> Hi Kun,
>
> On 08/29/2011 11:31 AM, Kun Wang wrote:
>> Hi, everyone,
>>
>> This is Kun Wang from IBM Research China. I and my team have been 
>> working
>> on IBM PowerEN processor in recent years, including its simulation,
>> lib/runtime optimization and etc. Now we start the work to enable KVM on
>> PowerEN processor. Since the A2 core of PowerEN follows Power ISA v2.06
>> (more specifically, book3e and 64-bit), I believe 99% of our work will
>> stick to the ISA, and hence can be leveraged by others.
>>
>> As the new one to this KVM world, I and my team definitely need your 
>> help.
>> Looking forward to talking and working with you guys in the future.
>
> Welcome to the PowerPC KVM world! I'm looking very much forward to 
> working with you there. Please always CC kvm@vger.kernel.org for 
> emails you send to kvm-ppc@vger, unless you think of it as completely 
> off-generic-kvm discussions. I like the idea of having more clever 
> people involved in the PPC KVM process and by CC'ing kvm@vger, we get 
> more exposure.
>
> The Freescale e500 cores are basically BookE 2.06 compliant with a few 
> specialties here and there. Have you looked at their code? Sure, it's 
> 32-bit as is now but there is definitely work going on at Freescale to 
> get it 64-bit too.
>
> Maybe it makes sense to take the current code as a starting point and 
> slowly move towards an architecture that is more generic. Maybe it 
> makes sense to start from scratch and actually design something 
> flexible that splits guest and host TLB code, so we could potentially 
> run cross-book KVM in the future (which I would love to see happen! 
> Just imagine running a PowerEN guest on a POWER7 system).
>
> Either way, please make sure to coordinate any efforts with me and 
> Scott so we don't walk off in different directions and only get to 
> realize we went down the wrong path when there's already a 100 patches 
> patch set on the mailing list :)
>
>
> Again, welcome on board!
>
> Alex
>
>
Thanks, Alex.

Yes, we are aware of the e500 work done by Freescale folks. I agree with 
you that we should coordinate together to achieve as more synergy.

Best regards,
Kun Wang