KVM for Sparc?

KVM for Sparc?

[Blue Swirl]

Hi,

 Sorry for cross-posting (and because I used the wrong address in the
first time for KVM, sorry for the duplicate on Sparclinux).

 Sparc host support for Qemu is getting close
 to ready, I can already run a Sparc32 system emulator on
 OpenBSD/Sparc64 and there is some limited success with recent glibc on
 Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable.
 Sparc64 target emulator can boot from several CD images, but crashes
 pretty soon.

 But I think we could already start early drafting of what KVM support
 for Sparc32 and Sparc64 would mean. Because of certain problems in the
 V9 instruction set design (V8 rett reuse for example), it may be
 difficult or even impossible to use an accelerator if the host and
 target instruction sets do not match.

 Other possibilities include porting kqemu or Xen, but I think KVM has
 the brightest future.

 I'm interested in pushing the Qemu side forward, but obviously
 something needs to be done by the kernel/KVM people too. What is the
 feeling on KVM and Sparclinux side?

Re: KVM for Sparc?

[Anthony Liguori]

Blue Swirl wrote:
> Hi,
>
>  Sorry for cross-posting (and because I used the wrong address in the
> first time for KVM, sorry for the duplicate on Sparclinux).
>
>  Sparc host support for Qemu is getting close
>  to ready, I can already run a Sparc32 system emulator on
>  OpenBSD/Sparc64 and there is some limited success with recent glibc on
>  Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable.
>  Sparc64 target emulator can boot from several CD images, but crashes
>  pretty soon.
>
>  But I think we could already start early drafting of what KVM support
>  for Sparc32 and Sparc64 would mean. Because of certain problems in the
>  V9 instruction set design (V8 rett reuse for example), it may be
>  difficult or even impossible to use an accelerator if the host and
>  target instruction sets do not match.
>   

I don't know much about the Sparc architecture, but the embedded PowerPC 
port that Hollis has spear-headed is for an architecture that does not 
natively support hardware virtualization.  As long as Sparc meets all of 
the requirements to do this sort of virtualization (all privileged 
instructions are trappable when run in non-privileged mode), it should 
be rather straight forward.

KVM has code to do shadow paging and also software TLB virtualization, 
so depending on how Sparc manages the TLB, it should be pretty straight 
forward to base the Sparc code from the appropriate code that already 
exists.

>  Other possibilities include porting kqemu or Xen, but I think KVM has
>  the brightest future.
>
>  I'm interested in pushing the Qemu side forward, but obviously
>  something needs to be done by the kernel/KVM people too.

KVM supports x86, ia64, s390, and PPC today.  I don't think there would 
be any problems adding another architecture support.  Almost all of the 
abstractions should have been flushed out already by the previous 
architecture ports.  Moreover, since there is already good support for 
Sparc in QEMU, that should simplify things significantly.

I'm looking forward to seeing the progress you make!

Regards,

Anthony Liguori

>  What is the
>  feeling on KVM and Sparclinux side?
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: KVM for Sparc?

[Hollis Blanchard]

On Mon, 2008-09-22 at 14:34 -0500, Anthony Liguori wrote:
> Blue Swirl wrote:
> > Hi,
> >
> >  Sorry for cross-posting (and because I used the wrong address in the
> > first time for KVM, sorry for the duplicate on Sparclinux).
> >
> >  Sparc host support for Qemu is getting close
> >  to ready, I can already run a Sparc32 system emulator on
> >  OpenBSD/Sparc64 and there is some limited success with recent glibc on
> >  Linux/Sparc64. Otherwise Sparc32 target emulator is pretty stable.
> >  Sparc64 target emulator can boot from several CD images, but crashes
> >  pretty soon.
> >
> >  But I think we could already start early drafting of what KVM support
> >  for Sparc32 and Sparc64 would mean. Because of certain problems in the
> >  V9 instruction set design (V8 rett reuse for example), it may be
> >  difficult or even impossible to use an accelerator if the host and
> >  target instruction sets do not match.
> >   
> 
> I don't know much about the Sparc architecture, but the embedded PowerPC 
> port that Hollis has spear-headed is for an architecture that does not 
> natively support hardware virtualization.  As long as Sparc meets all of 
> the requirements to do this sort of virtualization (all privileged 
> instructions are trappable when run in non-privileged mode), it should 
> be rather straight forward.

It would be even more interesting to implement host support on the Sparc
processors with hardware virtualization support.

Without that, functionality is possible (as we've demonstrated with KVM
on PPC440)... the trick is performance.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: KVM for Sparc?

[David Miller]

From: Hollis Blanchard <hollisb@us.ibm.com>
Date: Mon, 22 Sep 2008 15:18:39 -0500

> It would be even more interesting to implement host support on the Sparc
> processors with hardware virtualization support.

That's just a lot of protocol and userland work, rather than much
kernel stuff.

To me it's less interesting, to be honest.

Re: KVM for Sparc?

[Hollis Blanchard]

On Mon, 2008-09-22 at 13:26 -0700, David Miller wrote:
> From: Hollis Blanchard <hollisb@us.ibm.com>
> Date: Mon, 22 Sep 2008 15:18:39 -0500
> 
> > It would be even more interesting to implement host support on the Sparc
> > processors with hardware virtualization support.
> 
> That's just a lot of protocol and userland work, rather than much
> kernel stuff.
> 
> To me it's less interesting, to be honest.

Not sure what you mean. On IBM POWER servers, the hardware's
virtualization support is only accessible to an IBM-developed hypervisor
that is packaged with the system. If you're saying Sparc is in the same
position, I agree that's not very interesting.

On the other hand if those features are available to us, there would be
some pretty interesting kernel work to port KVM to use that
functionality.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: KVM for Sparc?

[David Miller]

From: Hollis Blanchard <hollisb@us.ibm.com>
Date: Mon, 22 Sep 2008 16:25:51 -0500

> On Mon, 2008-09-22 at 13:26 -0700, David Miller wrote:
> > From: Hollis Blanchard <hollisb@us.ibm.com>
> > Date: Mon, 22 Sep 2008 15:18:39 -0500
> > 
> > > It would be even more interesting to implement host support on the Sparc
> > > processors with hardware virtualization support.
> > 
> > That's just a lot of protocol and userland work, rather than much
> > kernel stuff.
> > 
> > To me it's less interesting, to be honest.
> 
> Not sure what you mean. On IBM POWER servers, the hardware's
> virtualization support is only accessible to an IBM-developed hypervisor
> that is packaged with the system. If you're saying Sparc is in the same
> position, I agree that's not very interesting.

Sun4v systems come with Sun's hypervisor.  Linux simply runs on top
of that, whether as a host or a guest.

The hypervisor source is opensource and we could technically make
changes to it, but it isn't very practical.

Re: KVM for Sparc?

[Blue Swirl]

On 9/23/08, David Miller <davem@davemloft.net> wrote:
> From: Hollis Blanchard <hollisb@us.ibm.com>
>
> Date: Mon, 22 Sep 2008 16:25:51 -0500
>
>
>  > On Mon, 2008-09-22 at 13:26 -0700, David Miller wrote:
>  > > From: Hollis Blanchard <hollisb@us.ibm.com>
>  > > Date: Mon, 22 Sep 2008 15:18:39 -0500
>  > >
>  > > > It would be even more interesting to implement host support on the Sparc
>  > > > processors with hardware virtualization support.
>  > >
>  > > That's just a lot of protocol and userland work, rather than much
>  > > kernel stuff.
>  > >
>  > > To me it's less interesting, to be honest.
>  >
>  > Not sure what you mean. On IBM POWER servers, the hardware's
>  > virtualization support is only accessible to an IBM-developed hypervisor
>  > that is packaged with the system. If you're saying Sparc is in the same
>  > position, I agree that's not very interesting.
>
>
> Sun4v systems come with Sun's hypervisor.  Linux simply runs on top
>  of that, whether as a host or a guest.
>
>  The hypervisor source is opensource and we could technically make
>  changes to it, but it isn't very practical.

Do you mean OpenxVM? I think Sun has not published the hypervisor part yet.

Re: KVM for Sparc?

[David Miller]

From: "Blue Swirl" <blauwirbel@gmail.com>
Date: Tue, 23 Sep 2008 18:34:12 +0300

> On 9/23/08, David Miller <davem@davemloft.net> wrote:
> > Sun4v systems come with Sun's hypervisor.  Linux simply runs on top
> >  of that, whether as a host or a guest.
> >
> >  The hypervisor source is opensource and we could technically make
> >  changes to it, but it isn't very practical.
> 
> Do you mean OpenxVM? I think Sun has not published the hypervisor part yet.

No, I mean OpenSPARC.  The full hypervisor source code for sun4v is in
the tarball.

	http://www.opensparc.net/

The code is there for both Niagara-T1 and Niagara-T2, for example for
Niagara-T1 click on "Get The Source" --> "OpenSPARC T1" --> "Downloads"

Then on that page you want "OpenSPARC T1 Download for Architecture and
Performance Modeling Tools".  It includes the full hypervisor and even
the openboot PROM source code.

Re: KVM for Sparc?

[Blue Swirl]

On 9/23/08, David Miller <davem@davemloft.net> wrote:
> From: "Blue Swirl" <blauwirbel@gmail.com>
>
> Date: Tue, 23 Sep 2008 18:34:12 +0300
>
>
>  > On 9/23/08, David Miller <davem@davemloft.net> wrote:
>
> > > Sun4v systems come with Sun's hypervisor.  Linux simply runs on top
>  > >  of that, whether as a host or a guest.
>  > >
>  > >  The hypervisor source is opensource and we could technically make
>  > >  changes to it, but it isn't very practical.
>  >
>  > Do you mean OpenxVM? I think Sun has not published the hypervisor part yet.
>
>
> No, I mean OpenSPARC.  The full hypervisor source code for sun4v is in
>  the tarball.
>
>         http://www.opensparc.net/
>
>  The code is there for both Niagara-T1 and Niagara-T2, for example for
>  Niagara-T1 click on "Get The Source" --> "OpenSPARC T1" --> "Downloads"
>
>  Then on that page you want "OpenSPARC T1 Download for Architecture and
>  Performance Modeling Tools".  It includes the full hypervisor and even
>  the openboot PROM source code.

I'm not sure, but I think that hypervisor is not a real hypervisor
like the PROM version but one specially designed for Legion emulator.

Re: KVM for Sparc?

[David Miller]

From: "Blue Swirl" <blauwirbel@gmail.com>
Date: Wed, 24 Sep 2008 20:22:45 +0300

> I'm not sure, but I think that hypervisor is not a real hypervisor
> like the PROM version but one specially designed for Legion emulator.

No, it's the real deal.

All the real hardware device support is there.

And Legion simulates a real, full Niagara system.

Re: KVM for Sparc?

[Javier Guerra]

On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
> It would be even more interesting to implement host support on the Sparc
> processors with hardware virtualization support.

Does Sparc processors also have 'virtualization support' as an extra
feature? i thought that 'non virtualizationability' was an
intel-specific limitation.  (come on... creating a privileged mode and
not trapping violations? who else would design like that?)

-- 
Javier

Re: KVM for Sparc?

[David Miller]

From: "Javier Guerra" <javier@guerrag.com>
Date: Mon, 22 Sep 2008 15:31:07 -0500

> On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
> > It would be even more interesting to implement host support on the Sparc
> > processors with hardware virtualization support.
> 
> Does Sparc processors also have 'virtualization support' as an extra
> feature?

Only sun4v processors (Niagara, etc.) have hypervisor support.

Re: KVM for Sparc?

[Hollis Blanchard]

On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote:
> On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
> > It would be even more interesting to implement host support on the Sparc
> > processors with hardware virtualization support.
> 
> Does Sparc processors also have 'virtualization support' as an extra
> feature? i thought that 'non virtualizationability' was an
> intel-specific limitation.  (come on... creating a privileged mode and
> not trapping violations? who else would design like that?)

Trapping on privileged instructions is enough to get you virtualization
functionality, but for good performance you really want additional
hardware support to avoid those traps in the first place.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: KVM for Sparc?

[David Miller]

From: Hollis Blanchard <hollisb@us.ibm.com>
Date: Mon, 22 Sep 2008 16:27:40 -0500

> On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote:
> > On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
> > > It would be even more interesting to implement host support on the Sparc
> > > processors with hardware virtualization support.
> > 
> > Does Sparc processors also have 'virtualization support' as an extra
> > feature? i thought that 'non virtualizationability' was an
> > intel-specific limitation.  (come on... creating a privileged mode and
> > not trapping violations? who else would design like that?)
> 
> Trapping on privileged instructions is enough to get you virtualization
> functionality, but for good performance you really want additional
> hardware support to avoid those traps in the first place.

That's not really available on sparc.

Re: KVM for Sparc?

[Blue Swirl]

On 9/23/08, David Miller <davem@davemloft.net> wrote:
> From: Hollis Blanchard <hollisb@us.ibm.com>
>
> Date: Mon, 22 Sep 2008 16:27:40 -0500
>
>
>  > On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote:
>  > > On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
>  > > > It would be even more interesting to implement host support on the Sparc
>  > > > processors with hardware virtualization support.
>  > >
>  > > Does Sparc processors also have 'virtualization support' as an extra
>  > > feature? i thought that 'non virtualizationability' was an
>  > > intel-specific limitation.  (come on... creating a privileged mode and
>  > > not trapping violations? who else would design like that?)
>  >
>  > Trapping on privileged instructions is enough to get you virtualization
>  > functionality, but for good performance you really want additional
>  > hardware support to avoid those traps in the first place.
>
>
> That's not really available on sparc.

Looking at other intercept possibilities in AMD SVM, I'm not sure what
would increase performance anyway except IO support. Because there are
no dedicated IO instructions (except maybe some ASI accesses), the
guest can be given direct HW access just by mapping the pages to guest
address space (and adding IOMMU TLBs).

Re: KVM for Sparc?

[Hollis Blanchard]

On Mon, 2008-09-22 at 15:15 -0700, David Miller wrote:
> From: Hollis Blanchard <hollisb@us.ibm.com>
> Date: Mon, 22 Sep 2008 16:27:40 -0500
> 
> > On Mon, 2008-09-22 at 15:31 -0500, Javier Guerra wrote:
> > > On Mon, Sep 22, 2008 at 3:18 PM, Hollis Blanchard <hollisb@us.ibm.com> wrote:
> > > > It would be even more interesting to implement host support on the Sparc
> > > > processors with hardware virtualization support.
> > > 
> > > Does Sparc processors also have 'virtualization support' as an extra
> > > feature? i thought that 'non virtualizationability' was an
> > > intel-specific limitation.  (come on... creating a privileged mode and
> > > not trapping violations? who else would design like that?)
> > 
> > Trapping on privileged instructions is enough to get you virtualization
> > functionality, but for good performance you really want additional
> > hardware support to avoid those traps in the first place.
> 
> That's not really available on sparc.

As far as I can see, you have a hypervisor privilege mode which allows
your guests to execute privileged instructions like WRPR without
trapping.

Without that hardware support (with only 2 privilege levels), you'd be
forced to run the guest kernel in user mode, trapping on every WRPR (for
example). That's what I mean by hardware helping you to avoid traps.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: KVM for Sparc?

[David Miller]

From: Hollis Blanchard <hollisb@us.ibm.com>
Date: Tue, 23 Sep 2008 11:07:31 -0500

> As far as I can see, you have a hypervisor privilege mode which allows
> your guests to execute privileged instructions like WRPR without
> trapping.

Sure.

> Without that hardware support (with only 2 privilege levels), you'd be
> forced to run the guest kernel in user mode, trapping on every WRPR (for
> example). That's what I mean by hardware helping you to avoid traps.

Understood.  Actually, we'll likely need to interpret instructions or
rewrite some of them when executing privileged mode code, because, as
stated, several 32-bit V8 Sparc instructions which were privileged
became non-privileged instructions for 64-bit V9.

Re: KVM for Sparc?

[David Miller]

From: Anthony Liguori <anthony@codemonkey.ws>
Date: Mon, 22 Sep 2008 14:34:21 -0500

> Blue Swirl wrote:
> >  But I think we could already start early drafting of what KVM support
> >  for Sparc32 and Sparc64 would mean. Because of certain problems in the
> >  V9 instruction set design (V8 rett reuse for example), it may be
> >  difficult or even impossible to use an accelerator if the host and
> >  target instruction sets do not match.
>
> I don't know much about the Sparc architecture, but the embedded
> PowerPC port that Hollis has spear-headed is for an architecture
> that does not natively support hardware virtualization.  As long as
> Sparc meets all of the requirements to do this sort of
> virtualization (all privileged instructions are trappable when run
> in non-privileged mode), it should be rather straight forward.

As he mentioned, the V8 rett instruction causes problems on V9 chips.

An opcode which was a V8 privileged instruction, "rett", got reused as
a non-privileged instruction in V9, for "return".

So booting a 32-bit kernel on a 64-bit cpu is going to be challenging,
at best.

> KVM supports x86, ia64, s390, and PPC today.  I don't think there
> would be any problems adding another architecture support.  Almost
> all of the abstractions should have been flushed out already by the
> previous architecture ports.  Moreover, since there is already good
> support for Sparc in QEMU, that should simplify things
> significantly.

Of course.

Re: KVM for Sparc?

[Blue Swirl]

On 9/22/08, David Miller <davem@davemloft.net> wrote:
> From: Anthony Liguori <anthony@codemonkey.ws>
>  Date: Mon, 22 Sep 2008 14:34:21 -0500
>
>
>  > Blue Swirl wrote:
>  > >  But I think we could already start early drafting of what KVM support
>  > >  for Sparc32 and Sparc64 would mean. Because of certain problems in the
>  > >  V9 instruction set design (V8 rett reuse for example), it may be
>  > >  difficult or even impossible to use an accelerator if the host and
>  > >  target instruction sets do not match.
>  >
>  > I don't know much about the Sparc architecture, but the embedded
>  > PowerPC port that Hollis has spear-headed is for an architecture
>  > that does not natively support hardware virtualization.  As long as
>  > Sparc meets all of the requirements to do this sort of
>  > virtualization (all privileged instructions are trappable when run
>  > in non-privileged mode), it should be rather straight forward.
>
>
> As he mentioned, the V8 rett instruction causes problems on V9 chips.
>
>  An opcode which was a V8 privileged instruction, "rett", got reused as
>  a non-privileged instruction in V9, for "return".

There are others: rdtbr/flushw and stdfq/stqf. Also any ASI >0x80
accesses are unprivileged on V9, though that shouldn't be a problem
since all ASIs used on V8 were <0x80. And of course MMUs are
incompatible.

>  So booting a 32-bit kernel on a 64-bit cpu is going to be challenging,
>  at best.

Maybe it would be possible to run V8 userspace with full speed
acceleration on V9 and use translation only for kernel code?

Re: KVM for Sparc?

[David Miller]

From: "Blue Swirl" <blauwirbel@gmail.com>
Date: Tue, 23 Sep 2008 18:28:06 +0300

> On 9/22/08, David Miller <davem@davemloft.net> wrote:
> > As he mentioned, the V8 rett instruction causes problems on V9 chips.
> >
> >  An opcode which was a V8 privileged instruction, "rett", got reused as
> >  a non-privileged instruction in V9, for "return".
> 
> There are others: rdtbr/flushw and stdfq/stqf. Also any ASI >0x80
> accesses are unprivileged on V9, though that shouldn't be a problem
> since all ASIs used on V8 were <0x80. And of course MMUs are
> incompatible.

Thanks for the list.  I sent a message to someone who I think might
have been responsible for these architectual design decisions, letting
them know what problems it is causing :-)

> >  So booting a 32-bit kernel on a 64-bit cpu is going to be challenging,
> >  at best.
> 
> Maybe it would be possible to run V8 userspace with full speed
> acceleration on V9 and use translation only for kernel code?

Yes, that should work.

BTW, there is another area related the ASIs.  Trap numbers.

Even through V9, traps only up to 0x7f are valid.  But sun4v extended
V9 to allow trap numbers >= 0x80, mostly these are used for hypervisor
calls.

The trap number field of the instruction is just extended one more
bit higher to accomodate this.

It turns out that binutils's assembler didn't range check the trap
number field value, so binutils needed no changes to support this :-)

Re: KVM for Sparc?

[Blue Swirl]

On 9/23/08, David Miller <davem@davemloft.net> wrote:
> From: "Blue Swirl" <blauwirbel@gmail.com>
>
> Date: Tue, 23 Sep 2008 18:28:06 +0300
>
>
>  > On 9/22/08, David Miller <davem@davemloft.net> wrote:
>
> > > As he mentioned, the V8 rett instruction causes problems on V9 chips.
>  > >
>  > >  An opcode which was a V8 privileged instruction, "rett", got reused as
>  > >  a non-privileged instruction in V9, for "return".
>  >
>  > There are others: rdtbr/flushw and stdfq/stqf. Also any ASI >0x80
>  > accesses are unprivileged on V9, though that shouldn't be a problem
>  > since all ASIs used on V8 were <0x80. And of course MMUs are
>  > incompatible.
>
>
> Thanks for the list.  I sent a message to someone who I think might
>  have been responsible for these architectual design decisions, letting
>  them know what problems it is causing :-)
>
>
>  > >  So booting a 32-bit kernel on a 64-bit cpu is going to be challenging,
>  > >  at best.
>  >
>  > Maybe it would be possible to run V8 userspace with full speed
>  > acceleration on V9 and use translation only for kernel code?
>
>
> Yes, that should work.
>
>  BTW, there is another area related the ASIs.  Trap numbers.
>
>  Even through V9, traps only up to 0x7f are valid.  But sun4v extended
>  V9 to allow trap numbers >= 0x80, mostly these are used for hypervisor
>  calls.
>
>  The trap number field of the instruction is just extended one more
>  bit higher to accomodate this.

I see, also Qemu needs to use one more bit then. Does this mean that
even V8 code written specially may use these traps to call hypervisor?
Then we would need to catch these, maybe with the some assistance from
the hypervisor.

Re: KVM for Sparc?

[Blue Swirl]

On 9/24/08, Blue Swirl <blauwirbel@gmail.com> wrote:
> On 9/23/08, David Miller <davem@davemloft.net> wrote:
>  > From: "Blue Swirl" <blauwirbel@gmail.com>
>  >
>  > Date: Tue, 23 Sep 2008 18:28:06 +0300
>  >
>  >
>  >  > On 9/22/08, David Miller <davem@davemloft.net> wrote:
>  >
>  > > > As he mentioned, the V8 rett instruction causes problems on V9 chips.
>  >  > >
>  >  > >  An opcode which was a V8 privileged instruction, "rett", got reused as
>  >  > >  a non-privileged instruction in V9, for "return".
>  >  >
>  >  > There are others: rdtbr/flushw and stdfq/stqf. Also any ASI >0x80
>  >  > accesses are unprivileged on V9, though that shouldn't be a problem
>  >  > since all ASIs used on V8 were <0x80. And of course MMUs are
>  >  > incompatible.
>  >
>  >
>  > Thanks for the list.  I sent a message to someone who I think might
>  >  have been responsible for these architectual design decisions, letting
>  >  them know what problems it is causing :-)
>  >
>  >
>  >  > >  So booting a 32-bit kernel on a 64-bit cpu is going to be challenging,
>  >  > >  at best.
>  >  >
>  >  > Maybe it would be possible to run V8 userspace with full speed
>  >  > acceleration on V9 and use translation only for kernel code?
>  >
>  >
>  > Yes, that should work.
>  >
>  >  BTW, there is another area related the ASIs.  Trap numbers.
>  >
>  >  Even through V9, traps only up to 0x7f are valid.  But sun4v extended
>  >  V9 to allow trap numbers >= 0x80, mostly these are used for hypervisor
>  >  calls.
>  >
>  >  The trap number field of the instruction is just extended one more
>  >  bit higher to accomodate this.
>
>
> I see, also Qemu needs to use one more bit then. Does this mean that
>  even V8 code written specially may use these traps to call hypervisor?
>  Then we would need to catch these, maybe with the some assistance from
>  the hypervisor.

Now I found the relevant part in the manuals. The extra sun4v bit is
not taken into account from user mode, so we can't catch privileged to
hyperprivileged mode traps easily.

Re: KVM for Sparc?

[David Miller]

From: "Blue Swirl" <blauwirbel@gmail.com>
Date: Wed, 24 Sep 2008 21:06:21 +0300

> Now I found the relevant part in the manuals. The extra sun4v bit is
> not taken into account from user mode, so we can't catch privileged to
> hyperprivileged mode traps easily.

That's right, the top bit is ignored in user mode.

Re: KVM for Sparc?

[Blue Swirl]

On 9/24/08, David Miller <davem@davemloft.net> wrote:
> From: "Blue Swirl" <blauwirbel@gmail.com>
>
> Date: Wed, 24 Sep 2008 21:06:21 +0300
>
>
>  > Now I found the relevant part in the manuals. The extra sun4v bit is
>  > not taken into account from user mode, so we can't catch privileged to
>  > hyperprivileged mode traps easily.
>
>
> That's right, the top bit is ignored in user mode.

The hypervisor uses traps 0x80, 0x83, 0x84, 0x85, and 0xff. Looking at
how these alias to low number traps: first four are unused or used for
resets (SIR, RED, XIR), so they are not in the fast path. 0xff aliases
to 0x7f, which is part of Fill 7 otherwin trap. Maybe that is not
performance critical? The Fill 7 trap entry could be shortened with
off-table jumps.

I'm thinking that we could disassemble the calling instruction on
entry to the lower traps and detect what was the true cause of the
trap.

Re: KVM for Sparc?

[David Miller]

From: "Blue Swirl" <blauwirbel@gmail.com>
Date: Wed, 24 Sep 2008 20:20:33 +0300

> I see, also Qemu needs to use one more bit then. Does this mean that
> even V8 code written specially may use these traps to call hypervisor?

No, V8 code should not set the extra bit.  Only V9 code on a processor
which is hypervisor capable should do this.

And the hypervisor calls can only be invoked from privileged mode.

This is all described in the UltraSPARC Architecture 2005 Specification
(Hyperprivileged Edition) at:

	http://www.opensparc.net/opensparc-t1/index.html