[Qemu-devel] ARM QEMU/KVM and TrustZone

[Qemu-devel] ARM QEMU/KVM and TrustZone

[Peter Maydell]

Historically for QEMU we haven't implemented TrustZone support even
though we claim to emulate CPUs that provide it. Instead we provide a
CPU which mostly looks like a variant of the real thing without the
TrustZone feature. We then bolt on a few extra cp15 registers (eg the
SCR) as a pragmatic move to get Linux guests to run. Now we're also
dealing with KVM on ARM I'd like to define things a bit more solidly
so KVM and TCG agree on what the CPU model they present is.

There are several possible environments we could provide
to a guest:
 (1) a CPU with full TrustZone support
 (2) a CPU without TrustZone at all
 (3) a TZ CPU running in NonSecure PL0/PL1
 (4) a TZ CPU running in Secure PL0/PL1

In some ways (1) is the "purist" solution -- emulate exactly what the
hardware does. However:

 * on TCG it would require a lot of work, including new functionality
   in core QEMU (to support having different CPU cores being able to
   see different views of memory, and having the S/NS attribute
   attached to memory transactions)

 * it isn't possible in KVM, because the ARM Virtualization Extensions
   don't allow you to fake the CPSR a guest sees, and so you can't
   make the guest believe it is in Monitor mode

Option (2) is architecturally sanctioned (ie TrustZone is an optional
feature, not mandatory), but it doesn't correspond to real CPUs, in
that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
modelling something that doesn't really exist.

Options (3) and (4) correspond to the environment an OS guest
typically actually uses on hardware. For ARM's devboards (versatile
express etc) Linux runs in the Secure world but it doesn't actually
use any of the TrustZone functionality, it's just a "give me full
access to everything" setup. For just about every other ARM system,
the boot rom or equivalent keeps Secure world to itself, and the OS
kernel runs in the NonSecure world. (This typically means that the
boot rom provides a set of board-specific entry points via the Secure
Monitor Call (SMC) instruction for doing operations like "invalidate
whole L2 cache" which require secure privileges.)

Proposal:

My suggestion is that we present the guest with a view that looks like
a sort of superset of (2) (3) and (4), ie sufficient that a guest
expecting any of those environments can run. In particular:

 * no cp15 registers have secure/nonsecure banking
 * there is only one memory space visible
 * secure-access-only permissions are not enforced
 * the handful of only-in-trustzone registers are implemented
   (eg VBAR, MVBAR)
 * we implement a "fake monitor mode"

The aim of the "fake monitor mode" is to allow us to provide fake
qemu-specific bootroms which implement whatever the board's SMC
interface is, without having to write specific KVM kernel code for
each board. So we don't have to run arbitrary secure-world guest code.
The rules are:
 * on an SMC instruction we enter the guest at the SMC vector
   as defined by the MVBAR (monitor vector base address register)
 * we actually run with the same access permissions as above
   (and under KVM if you look at CPSR.M it will tell you you're
   in Supervisor mode)
 * return from the SMC is via a standard exception return insn
 * we don't implement the separate memory space for the secure
   world. (This implies that you need to find space in the
   non-secure world's physical memory map for the bootrom shim;
   not a big deal I think since we already have a requirement
   for some space to put QEMU's arm_boot trivial bootloader.)

The code written for this fake monitor mode environment is likely to
be able to work OK if we ever implement full TrustZone support in TCG
QEMU.

Work required:

* Documentation: the general principles as listed above
* TCG: make sure we have implementations of all the TZ registers
* TCG: implement the SMC and fake-monitor-mode
  (I already have patches from Nokia in the qemu-linaro
  stack which can be cleaned up and used here)
* KVM: implement emulation of MVBAR
* KVM: set the config bit so SMC is trapped to the hypervisor
  and causes guest restart at the right entrypoint
* KVM: if there turns out to be anything that fake-monitor-mode
  needs to do that requires Hyp privilege we'd need a hypercall
  ABI, but I can't currently think of anything

I think that's basically a fairly small set of work to formalise
the approach we're already taking in practice, and make it a little
more flexible.

Opinions?

-- PMM

Re: [Qemu-devel] ARM QEMU/KVM and TrustZone

[Peter Maydell]

Andreas pointed out that I should have cc'd Johannes here. Sorry
for forgetting that.

-- PMM

On 22 May 2012 13:22, Peter Maydell <peter.maydell@linaro.org> wrote:
> Historically for QEMU we haven't implemented TrustZone support even
> though we claim to emulate CPUs that provide it. Instead we provide a
> CPU which mostly looks like a variant of the real thing without the
> TrustZone feature. We then bolt on a few extra cp15 registers (eg the
> SCR) as a pragmatic move to get Linux guests to run. Now we're also
> dealing with KVM on ARM I'd like to define things a bit more solidly
> so KVM and TCG agree on what the CPU model they present is.
>
> There are several possible environments we could provide
> to a guest:
>  (1) a CPU with full TrustZone support
>  (2) a CPU without TrustZone at all
>  (3) a TZ CPU running in NonSecure PL0/PL1
>  (4) a TZ CPU running in Secure PL0/PL1
>
> In some ways (1) is the "purist" solution -- emulate exactly what the
> hardware does. However:
>
>  * on TCG it would require a lot of work, including new functionality
>   in core QEMU (to support having different CPU cores being able to
>   see different views of memory, and having the S/NS attribute
>   attached to memory transactions)
>
>  * it isn't possible in KVM, because the ARM Virtualization Extensions
>   don't allow you to fake the CPSR a guest sees, and so you can't
>   make the guest believe it is in Monitor mode
>
> Option (2) is architecturally sanctioned (ie TrustZone is an optional
> feature, not mandatory), but it doesn't correspond to real CPUs, in
> that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
> modelling something that doesn't really exist.
>
> Options (3) and (4) correspond to the environment an OS guest
> typically actually uses on hardware. For ARM's devboards (versatile
> express etc) Linux runs in the Secure world but it doesn't actually
> use any of the TrustZone functionality, it's just a "give me full
> access to everything" setup. For just about every other ARM system,
> the boot rom or equivalent keeps Secure world to itself, and the OS
> kernel runs in the NonSecure world. (This typically means that the
> boot rom provides a set of board-specific entry points via the Secure
> Monitor Call (SMC) instruction for doing operations like "invalidate
> whole L2 cache" which require secure privileges.)
>
> Proposal:
>
> My suggestion is that we present the guest with a view that looks like
> a sort of superset of (2) (3) and (4), ie sufficient that a guest
> expecting any of those environments can run. In particular:
>
>  * no cp15 registers have secure/nonsecure banking
>  * there is only one memory space visible
>  * secure-access-only permissions are not enforced
>  * the handful of only-in-trustzone registers are implemented
>   (eg VBAR, MVBAR)
>  * we implement a "fake monitor mode"
>
> The aim of the "fake monitor mode" is to allow us to provide fake
> qemu-specific bootroms which implement whatever the board's SMC
> interface is, without having to write specific KVM kernel code for
> each board. So we don't have to run arbitrary secure-world guest code.
> The rules are:
>  * on an SMC instruction we enter the guest at the SMC vector
>   as defined by the MVBAR (monitor vector base address register)
>  * we actually run with the same access permissions as above
>   (and under KVM if you look at CPSR.M it will tell you you're
>   in Supervisor mode)
>  * return from the SMC is via a standard exception return insn
>  * we don't implement the separate memory space for the secure
>   world. (This implies that you need to find space in the
>   non-secure world's physical memory map for the bootrom shim;
>   not a big deal I think since we already have a requirement
>   for some space to put QEMU's arm_boot trivial bootloader.)
>
> The code written for this fake monitor mode environment is likely to
> be able to work OK if we ever implement full TrustZone support in TCG
> QEMU.
>
> Work required:
>
> * Documentation: the general principles as listed above
> * TCG: make sure we have implementations of all the TZ registers
> * TCG: implement the SMC and fake-monitor-mode
>  (I already have patches from Nokia in the qemu-linaro
>  stack which can be cleaned up and used here)
> * KVM: implement emulation of MVBAR
> * KVM: set the config bit so SMC is trapped to the hypervisor
>  and causes guest restart at the right entrypoint
> * KVM: if there turns out to be anything that fake-monitor-mode
>  needs to do that requires Hyp privilege we'd need a hypercall
>  ABI, but I can't currently think of anything
>
> I think that's basically a fairly small set of work to formalise
> the approach we're already taking in practice, and make it a little
> more flexible.
>
> Opinions?
>
> -- PMM



-- 
12345678901234567890123456789012345678901234567890123456789012345678901234567890
         1         2         3         4         5         6         7         8

Re: [Qemu-devel] ARM QEMU/KVM and TrustZone

[Christoffer Dall]

> On 22 May 2012 13:22, Peter Maydell <peter.maydell@linaro.org> wrote:
>> Historically for QEMU we haven't implemented TrustZone support even
>> though we claim to emulate CPUs that provide it. Instead we provide a
>> CPU which mostly looks like a variant of the real thing without the
>> TrustZone feature. We then bolt on a few extra cp15 registers (eg the
>> SCR) as a pragmatic move to get Linux guests to run. Now we're also
>> dealing with KVM on ARM I'd like to define things a bit more solidly
>> so KVM and TCG agree on what the CPU model they present is.
>>
>> There are several possible environments we could provide
>> to a guest:
>>  (1) a CPU with full TrustZone support
>>  (2) a CPU without TrustZone at all
>>  (3) a TZ CPU running in NonSecure PL0/PL1
>>  (4) a TZ CPU running in Secure PL0/PL1
>>
>> In some ways (1) is the "purist" solution -- emulate exactly what the
>> hardware does. However:
>>
>>  * on TCG it would require a lot of work, including new functionality
>>   in core QEMU (to support having different CPU cores being able to
>>   see different views of memory, and having the S/NS attribute
>>   attached to memory transactions)
>>
>>  * it isn't possible in KVM, because the ARM Virtualization Extensions
>>   don't allow you to fake the CPSR a guest sees, and so you can't
>>   make the guest believe it is in Monitor mode
>>
>> Option (2) is architecturally sanctioned (ie TrustZone is an optional
>> feature, not mandatory), but it doesn't correspond to real CPUs, in
>> that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
>> modelling something that doesn't really exist.
>>
>> Options (3) and (4) correspond to the environment an OS guest
>> typically actually uses on hardware. For ARM's devboards (versatile
>> express etc) Linux runs in the Secure world but it doesn't actually
>> use any of the TrustZone functionality, it's just a "give me full
>> access to everything" setup. For just about every other ARM system,
>> the boot rom or equivalent keeps Secure world to itself, and the OS
>> kernel runs in the NonSecure world. (This typically means that the
>> boot rom provides a set of board-specific entry points via the Secure
>> Monitor Call (SMC) instruction for doing operations like "invalidate
>> whole L2 cache" which require secure privileges.)
>>

Is there anything preventing people from writing a small bootloader
that switches into non-secure mode and runs kernels there as a general
approach (apart from laziness)?

>> Proposal:
>>
>> My suggestion is that we present the guest with a view that looks like
>> a sort of superset of (2) (3) and (4), ie sufficient that a guest
>> expecting any of those environments can run. In particular:
>>
>>  * no cp15 registers have secure/nonsecure banking
>>  * there is only one memory space visible
>>  * secure-access-only permissions are not enforced
>>  * the handful of only-in-trustzone registers are implemented
>>   (eg VBAR, MVBAR)
>>  * we implement a "fake monitor mode"
>>
>> The aim of the "fake monitor mode" is to allow us to provide fake
>> qemu-specific bootroms which implement whatever the board's SMC
>> interface is, without having to write specific KVM kernel code for
>> each board. So we don't have to run arbitrary secure-world guest code.
>> The rules are:
>>  * on an SMC instruction we enter the guest at the SMC vector
>>   as defined by the MVBAR (monitor vector base address register)
>>  * we actually run with the same access permissions as above
>>   (and under KVM if you look at CPSR.M it will tell you you're
>>   in Supervisor mode)
>>  * return from the SMC is via a standard exception return insn
>>  * we don't implement the separate memory space for the secure
>>   world. (This implies that you need to find space in the
>>   non-secure world's physical memory map for the bootrom shim;
>>   not a big deal I think since we already have a requirement
>>   for some space to put QEMU's arm_boot trivial bootloader.)
>>

you could have a separate set of stage-2 translation tables for this
and keep things separate for real (or would we rely on fake-SMC code
to directly be able to read fake-non-secure data, which is still
possible through a different memory map I guess).

>> The code written for this fake monitor mode environment is likely to
>> be able to work OK if we ever implement full TrustZone support in TCG
>> QEMU.
>>

what kind of operations would be required from SMC calls in a KVM
guest setting? I can see this in an embedded market, but are they not
likely to even capture Hyp mode already and set things up as required?
What I mean is, if KVM is currently targeting Calxeda-type setups will
we ever run kernels that require SMC operations as guests?

It feels a bit premature to implement all this.

>> Work required:
>>
>> * Documentation: the general principles as listed above
>> * TCG: make sure we have implementations of all the TZ registers
>> * TCG: implement the SMC and fake-monitor-mode
>>  (I already have patches from Nokia in the qemu-linaro
>>  stack which can be cleaned up and used here)
>> * KVM: implement emulation of MVBAR
>> * KVM: set the config bit so SMC is trapped to the hypervisor
>>  and causes guest restart at the right entrypoint
>> * KVM: if there turns out to be anything that fake-monitor-mode
>>  needs to do that requires Hyp privilege we'd need a hypercall
>>  ABI, but I can't currently think of anything
>>
>> I think that's basically a fairly small set of work to formalise
>> the approach we're already taking in practice, and make it a little
>> more flexible.
>>
>> Opinions?
>>

It feels like quite a bit of complexity at this point. But if we need
it, we need it. I'm just not convinced of that yet.

-Christoffer

Re: [Qemu-devel] ARM QEMU/KVM and TrustZone

[Антон Кочков]

Hm, one memory space, but what about write access restrictions, e.g.
for Non-Secure or Secure worlds for some memory addresses/blocks?

Best regards,
Anton Kochkov.


On Sat, Jun 16, 2012 at 9:37 PM, Christoffer Dall
<c.dall@virtualopensystems.com> wrote:
>> On 22 May 2012 13:22, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> Historically for QEMU we haven't implemented TrustZone support even
>>> though we claim to emulate CPUs that provide it. Instead we provide a
>>> CPU which mostly looks like a variant of the real thing without the
>>> TrustZone feature. We then bolt on a few extra cp15 registers (eg the
>>> SCR) as a pragmatic move to get Linux guests to run. Now we're also
>>> dealing with KVM on ARM I'd like to define things a bit more solidly
>>> so KVM and TCG agree on what the CPU model they present is.
>>>
>>> There are several possible environments we could provide
>>> to a guest:
>>>  (1) a CPU with full TrustZone support
>>>  (2) a CPU without TrustZone at all
>>>  (3) a TZ CPU running in NonSecure PL0/PL1
>>>  (4) a TZ CPU running in Secure PL0/PL1
>>>
>>> In some ways (1) is the "purist" solution -- emulate exactly what the
>>> hardware does. However:
>>>
>>>  * on TCG it would require a lot of work, including new functionality
>>>   in core QEMU (to support having different CPU cores being able to
>>>   see different views of memory, and having the S/NS attribute
>>>   attached to memory transactions)
>>>
>>>  * it isn't possible in KVM, because the ARM Virtualization Extensions
>>>   don't allow you to fake the CPSR a guest sees, and so you can't
>>>   make the guest believe it is in Monitor mode
>>>
>>> Option (2) is architecturally sanctioned (ie TrustZone is an optional
>>> feature, not mandatory), but it doesn't correspond to real CPUs, in
>>> that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
>>> modelling something that doesn't really exist.
>>>
>>> Options (3) and (4) correspond to the environment an OS guest
>>> typically actually uses on hardware. For ARM's devboards (versatile
>>> express etc) Linux runs in the Secure world but it doesn't actually
>>> use any of the TrustZone functionality, it's just a "give me full
>>> access to everything" setup. For just about every other ARM system,
>>> the boot rom or equivalent keeps Secure world to itself, and the OS
>>> kernel runs in the NonSecure world. (This typically means that the
>>> boot rom provides a set of board-specific entry points via the Secure
>>> Monitor Call (SMC) instruction for doing operations like "invalidate
>>> whole L2 cache" which require secure privileges.)
>>>
>
> Is there anything preventing people from writing a small bootloader
> that switches into non-secure mode and runs kernels there as a general
> approach (apart from laziness)?
>
>>> Proposal:
>>>
>>> My suggestion is that we present the guest with a view that looks like
>>> a sort of superset of (2) (3) and (4), ie sufficient that a guest
>>> expecting any of those environments can run. In particular:
>>>
>>>  * no cp15 registers have secure/nonsecure banking
>>>  * there is only one memory space visible
>>>  * secure-access-only permissions are not enforced
>>>  * the handful of only-in-trustzone registers are implemented
>>>   (eg VBAR, MVBAR)
>>>  * we implement a "fake monitor mode"
>>>
>>> The aim of the "fake monitor mode" is to allow us to provide fake
>>> qemu-specific bootroms which implement whatever the board's SMC
>>> interface is, without having to write specific KVM kernel code for
>>> each board. So we don't have to run arbitrary secure-world guest code.
>>> The rules are:
>>>  * on an SMC instruction we enter the guest at the SMC vector
>>>   as defined by the MVBAR (monitor vector base address register)
>>>  * we actually run with the same access permissions as above
>>>   (and under KVM if you look at CPSR.M it will tell you you're
>>>   in Supervisor mode)
>>>  * return from the SMC is via a standard exception return insn
>>>  * we don't implement the separate memory space for the secure
>>>   world. (This implies that you need to find space in the
>>>   non-secure world's physical memory map for the bootrom shim;
>>>   not a big deal I think since we already have a requirement
>>>   for some space to put QEMU's arm_boot trivial bootloader.)
>>>
>
> you could have a separate set of stage-2 translation tables for this
> and keep things separate for real (or would we rely on fake-SMC code
> to directly be able to read fake-non-secure data, which is still
> possible through a different memory map I guess).
>
>>> The code written for this fake monitor mode environment is likely to
>>> be able to work OK if we ever implement full TrustZone support in TCG
>>> QEMU.
>>>
>
> what kind of operations would be required from SMC calls in a KVM
> guest setting? I can see this in an embedded market, but are they not
> likely to even capture Hyp mode already and set things up as required?
> What I mean is, if KVM is currently targeting Calxeda-type setups will
> we ever run kernels that require SMC operations as guests?
>
> It feels a bit premature to implement all this.
>
>>> Work required:
>>>
>>> * Documentation: the general principles as listed above
>>> * TCG: make sure we have implementations of all the TZ registers
>>> * TCG: implement the SMC and fake-monitor-mode
>>>  (I already have patches from Nokia in the qemu-linaro
>>>  stack which can be cleaned up and used here)
>>> * KVM: implement emulation of MVBAR
>>> * KVM: set the config bit so SMC is trapped to the hypervisor
>>>  and causes guest restart at the right entrypoint
>>> * KVM: if there turns out to be anything that fake-monitor-mode
>>>  needs to do that requires Hyp privilege we'd need a hypercall
>>>  ABI, but I can't currently think of anything
>>>
>>> I think that's basically a fairly small set of work to formalise
>>> the approach we're already taking in practice, and make it a little
>>> more flexible.
>>>
>>> Opinions?
>>>
>
> It feels like quite a bit of complexity at this point. But if we need
> it, we need it. I'm just not convinced of that yet.
>
> -Christoffer
>

Re: [Qemu-devel] ARM QEMU/KVM and TrustZone

[Peter Maydell]

On 16 June 2012 18:37, Christoffer Dall <c.dall@virtualopensystems.com> wrote:
>> On 22 May 2012 13:22, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> Historically for QEMU we haven't implemented TrustZone support even
>>> though we claim to emulate CPUs that provide it. Instead we provide a
>>> CPU which mostly looks like a variant of the real thing without the
>>> TrustZone feature. We then bolt on a few extra cp15 registers (eg the
>>> SCR) as a pragmatic move to get Linux guests to run. Now we're also
>>> dealing with KVM on ARM I'd like to define things a bit more solidly
>>> so KVM and TCG agree on what the CPU model they present is.
>>>
>>> There are several possible environments we could provide
>>> to a guest:
>>>  (1) a CPU with full TrustZone support
>>>  (2) a CPU without TrustZone at all
>>>  (3) a TZ CPU running in NonSecure PL0/PL1
>>>  (4) a TZ CPU running in Secure PL0/PL1
>>>
>>> In some ways (1) is the "purist" solution -- emulate exactly what the
>>> hardware does. However:
>>>
>>>  * on TCG it would require a lot of work, including new functionality
>>>   in core QEMU (to support having different CPU cores being able to
>>>   see different views of memory, and having the S/NS attribute
>>>   attached to memory transactions)
>>>
>>>  * it isn't possible in KVM, because the ARM Virtualization Extensions
>>>   don't allow you to fake the CPSR a guest sees, and so you can't
>>>   make the guest believe it is in Monitor mode
>>>
>>> Option (2) is architecturally sanctioned (ie TrustZone is an optional
>>> feature, not mandatory), but it doesn't correspond to real CPUs, in
>>> that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
>>> modelling something that doesn't really exist.
>>>
>>> Options (3) and (4) correspond to the environment an OS guest
>>> typically actually uses on hardware. For ARM's devboards (versatile
>>> express etc) Linux runs in the Secure world but it doesn't actually
>>> use any of the TrustZone functionality, it's just a "give me full
>>> access to everything" setup. For just about every other ARM system,
>>> the boot rom or equivalent keeps Secure world to itself, and the OS
>>> kernel runs in the NonSecure world. (This typically means that the
>>> boot rom provides a set of board-specific entry points via the Secure
>>> Monitor Call (SMC) instruction for doing operations like "invalidate
>>> whole L2 cache" which require secure privileges.)
>>>
>
> Is there anything preventing people from writing a small bootloader
> that switches into non-secure mode and runs kernels there as a general
> approach (apart from laziness)?

I'm not sure what you're suggesting here. Mostly kernels do run
in NS mode on hardware, except for on ARM devboards. On ARM
devboards if you try to run the kernel in NS mode it will fall
over the first time it tries something that needs secure world
privileges. For KVM my rule of thumb is that it needs to run
the same kernel the hw runs.

To some extent the small boot loader would be the thing I describe
as a 'fake bootrom' below.

>>> Proposal:
>>>
>>> My suggestion is that we present the guest with a view that looks like
>>> a sort of superset of (2) (3) and (4), ie sufficient that a guest
>>> expecting any of those environments can run. In particular:
>>>
>>>  * no cp15 registers have secure/nonsecure banking
>>>  * there is only one memory space visible
>>>  * secure-access-only permissions are not enforced
>>>  * the handful of only-in-trustzone registers are implemented
>>>   (eg VBAR, MVBAR)
>>>  * we implement a "fake monitor mode"
>>>
>>> The aim of the "fake monitor mode" is to allow us to provide fake
>>> qemu-specific bootroms which implement whatever the board's SMC
>>> interface is, without having to write specific KVM kernel code for
>>> each board. So we don't have to run arbitrary secure-world guest code.
>>> The rules are:
>>>  * on an SMC instruction we enter the guest at the SMC vector
>>>   as defined by the MVBAR (monitor vector base address register)
>>>  * we actually run with the same access permissions as above
>>>   (and under KVM if you look at CPSR.M it will tell you you're
>>>   in Supervisor mode)
>>>  * return from the SMC is via a standard exception return insn
>>>  * we don't implement the separate memory space for the secure
>>>   world. (This implies that you need to find space in the
>>>   non-secure world's physical memory map for the bootrom shim;
>>>   not a big deal I think since we already have a requirement
>>>   for some space to put QEMU's arm_boot trivial bootloader.)
>>>
>
> you could have a separate set of stage-2 translation tables for this
> and keep things separate for real (or would we rely on fake-SMC code
> to directly be able to read fake-non-secure data, which is still
> possible through a different memory map I guess).

I'm trying to keep things simple (and separate memory maps for TCG
is a bit tricky). We can't make the monitor mode really look like
monitor mode, so there's not much point trying to implement all
the difficult complicated bits when we control the code running in
this mode anyway.

> what kind of operations would be required from SMC calls in a KVM
> guest setting? I can see this in an embedded market, but are they not
> likely to even capture Hyp mode already and set things up as required?
> What I mean is, if KVM is currently targeting Calxeda-type setups will
> we ever run kernels that require SMC operations as guests?

Yes, I think we must, because the CPU hardware doesn't let kernels
do everything in NS mode. There are usually only a handful of
operations needed. I've given some A8 examples above; I'm afraid
I don't have time to check for the A15 equivalents (I have a plane
to catch later :-)).

> It feels a bit premature to implement all this.

Basically it works at the moment for the vexpress A15 guest
because it happens to be one of the special cases which runs
in secure mode, and our cp15 emulation can just include
enough rope to let it all work. However that is likely to
drift into the ill-defined area where we are running the
guest in something that's actually NS but the cp15 emulation
lets it appear to access some S mode only registers.

In any case, I already have this problem in TCG mode for the
OMAP3 emulation, and so I want to define how it should all
work for that. We don't need to implement the KVM side just
yet as long as we're confident that we could...

-- PMM

Re: [Qemu-devel] ARM QEMU/KVM and TrustZone

[Christoffer Dall]

On Sat, Jun 16, 2012 at 9:09 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 16 June 2012 18:37, Christoffer Dall <c.dall@virtualopensystems.com> wrote:
>>> On 22 May 2012 13:22, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>> Historically for QEMU we haven't implemented TrustZone support even
>>>> though we claim to emulate CPUs that provide it. Instead we provide a
>>>> CPU which mostly looks like a variant of the real thing without the
>>>> TrustZone feature. We then bolt on a few extra cp15 registers (eg the
>>>> SCR) as a pragmatic move to get Linux guests to run. Now we're also
>>>> dealing with KVM on ARM I'd like to define things a bit more solidly
>>>> so KVM and TCG agree on what the CPU model they present is.
>>>>
>>>> There are several possible environments we could provide
>>>> to a guest:
>>>>  (1) a CPU with full TrustZone support
>>>>  (2) a CPU without TrustZone at all
>>>>  (3) a TZ CPU running in NonSecure PL0/PL1
>>>>  (4) a TZ CPU running in Secure PL0/PL1
>>>>
>>>> In some ways (1) is the "purist" solution -- emulate exactly what the
>>>> hardware does. However:
>>>>
>>>>  * on TCG it would require a lot of work, including new functionality
>>>>   in core QEMU (to support having different CPU cores being able to
>>>>   see different views of memory, and having the S/NS attribute
>>>>   attached to memory transactions)
>>>>
>>>>  * it isn't possible in KVM, because the ARM Virtualization Extensions
>>>>   don't allow you to fake the CPSR a guest sees, and so you can't
>>>>   make the guest believe it is in Monitor mode
>>>>
>>>> Option (2) is architecturally sanctioned (ie TrustZone is an optional
>>>> feature, not mandatory), but it doesn't correspond to real CPUs, in
>>>> that the hardware Cortex-A8/A9/A15 always have TrustZone. So we're
>>>> modelling something that doesn't really exist.
>>>>
>>>> Options (3) and (4) correspond to the environment an OS guest
>>>> typically actually uses on hardware. For ARM's devboards (versatile
>>>> express etc) Linux runs in the Secure world but it doesn't actually
>>>> use any of the TrustZone functionality, it's just a "give me full
>>>> access to everything" setup. For just about every other ARM system,
>>>> the boot rom or equivalent keeps Secure world to itself, and the OS
>>>> kernel runs in the NonSecure world. (This typically means that the
>>>> boot rom provides a set of board-specific entry points via the Secure
>>>> Monitor Call (SMC) instruction for doing operations like "invalidate
>>>> whole L2 cache" which require secure privileges.)
>>>>
>>
>> Is there anything preventing people from writing a small bootloader
>> that switches into non-secure mode and runs kernels there as a general
>> approach (apart from laziness)?
>
> I'm not sure what you're suggesting here. Mostly kernels do run
> in NS mode on hardware, except for on ARM devboards. On ARM
> devboards if you try to run the kernel in NS mode it will fall
> over the first time it tries something that needs secure world
> privileges. For KVM my rule of thumb is that it needs to run
> the same kernel the hw runs.

I was simply under the assumption that all operations required for a
kernel to run was allowed in NS-mode, but if that's not the case, then
never mind.

>
> To some extent the small boot loader would be the thing I describe
> as a 'fake bootrom' below.
>
>>>> Proposal:
>>>>
>>>> My suggestion is that we present the guest with a view that looks like
>>>> a sort of superset of (2) (3) and (4), ie sufficient that a guest
>>>> expecting any of those environments can run. In particular:
>>>>
>>>>  * no cp15 registers have secure/nonsecure banking
>>>>  * there is only one memory space visible
>>>>  * secure-access-only permissions are not enforced
>>>>  * the handful of only-in-trustzone registers are implemented
>>>>   (eg VBAR, MVBAR)
>>>>  * we implement a "fake monitor mode"
>>>>
>>>> The aim of the "fake monitor mode" is to allow us to provide fake
>>>> qemu-specific bootroms which implement whatever the board's SMC
>>>> interface is, without having to write specific KVM kernel code for
>>>> each board. So we don't have to run arbitrary secure-world guest code.
>>>> The rules are:
>>>>  * on an SMC instruction we enter the guest at the SMC vector
>>>>   as defined by the MVBAR (monitor vector base address register)
>>>>  * we actually run with the same access permissions as above
>>>>   (and under KVM if you look at CPSR.M it will tell you you're
>>>>   in Supervisor mode)
>>>>  * return from the SMC is via a standard exception return insn
>>>>  * we don't implement the separate memory space for the secure
>>>>   world. (This implies that you need to find space in the
>>>>   non-secure world's physical memory map for the bootrom shim;
>>>>   not a big deal I think since we already have a requirement
>>>>   for some space to put QEMU's arm_boot trivial bootloader.)
>>>>
>>
>> you could have a separate set of stage-2 translation tables for this
>> and keep things separate for real (or would we rely on fake-SMC code
>> to directly be able to read fake-non-secure data, which is still
>> possible through a different memory map I guess).
>
> I'm trying to keep things simple (and separate memory maps for TCG
> is a bit tricky). We can't make the monitor mode really look like
> monitor mode, so there's not much point trying to implement all
> the difficult complicated bits when we control the code running in
> this mode anyway.
>

ok, fair enough.

>> what kind of operations would be required from SMC calls in a KVM
>> guest setting? I can see this in an embedded market, but are they not
>> likely to even capture Hyp mode already and set things up as required?
>> What I mean is, if KVM is currently targeting Calxeda-type setups will
>> we ever run kernels that require SMC operations as guests?
>
> Yes, I think we must, because the CPU hardware doesn't let kernels
> do everything in NS mode. There are usually only a handful of
> operations needed. I've given some A8 examples above; I'm afraid
> I don't have time to check for the A15 equivalents (I have a plane
> to catch later :-)).
>
>> It feels a bit premature to implement all this.
>
> Basically it works at the moment for the vexpress A15 guest
> because it happens to be one of the special cases which runs
> in secure mode, and our cp15 emulation can just include
> enough rope to let it all work. However that is likely to
> drift into the ill-defined area where we are running the
> guest in something that's actually NS but the cp15 emulation
> lets it appear to access some S mode only registers.
>
> In any case, I already have this problem in TCG mode for the
> OMAP3 emulation, and so I want to define how it should all
> work for that. We don't need to implement the KVM side just
> yet as long as we're confident that we could...
>
If we do it in one memory space it doesn't sound very intrusive for
KVM, so in that regards I'm fine with the approach if it cannot be
avoided all together.

-Christoffer