[virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[jeshwank]

In a virtual environment, an application running in guest VM may want
to delegate security sensitive tasks to a Trusted Application (TA)
running within a Trusted Execution Environment (TEE). A TEE is a trusted
OS running in some secure environment, for example, TrustZone on ARM
CPUs, or a separate secure co-processor etc.

A virtual TEE device emulates a TEE within a guest VM. Such a virtual
TEE device supports multiple operations such as:

VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
                             TEE device.
VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
                              TEE device.
VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
                              trusted application running in TEE.
VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
                               with trusted application running in TEE.
VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
                             application running in TEE.
VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.

We would like to reserve device ID 46 for Virtio-TEE device.

Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
---
 content.tex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content.tex b/content.tex
index 0a62dce..644aa4a 100644
--- a/content.tex
+++ b/content.tex
@@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 45         &   SPI master \\
 \hline
+46         &   TEE device \\
+\hline
 \end{tabular}
 
 Some of the devices above are unspecified by this document,
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

RE: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Parav Pandit]

> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
> Behalf Of jeshwank
> Sent: Monday, September 25, 2023 4:47 PM
> To: virtio-dev@lists.oasis-open.org

I notice this repeated occurs error of choosing the mailing list as virtio-dev for spec work.
I will fix the OASIS document references.

In the meant time, the mailing list should be virtio-comment@lists... For this patch.


> 
> In a virtual environment, an application running in guest VM may want to
> delegate security sensitive tasks to a Trusted Application (TA) running within a
> Trusted Execution Environment (TEE). A TEE is a trusted OS running in some
> secure environment, for example, TrustZone on ARM CPUs, or a separate secure
> co-processor etc.
> 
> A virtual TEE device emulates a TEE within a guest VM. Such a virtual TEE device
> supports multiple operations such as:
> 
> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>                              TEE device.
> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>                               TEE device.
> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>                               trusted application running in TEE.
> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>                                with trusted application running in TEE.
> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>                              application running in TEE.
> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
> 
> We would like to reserve device ID 46 for Virtio-TEE device.
> 
> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
Reviewed-by: Parav Pandit <parav@nvidia.com>

> ---
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 0a62dce..644aa4a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}  \hline
>  45         &   SPI master \\
>  \hline
> +46         &   TEE device \\
> +\hline
>  \end{tabular}
> 
>  Some of the devices above are unspecified by this document,
> --
> 2.25.1
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[NK, JESHWANTHKUMAR]

++virtio-comment

Thanks Parav for the review..

On 25-Sep-23 9:08 PM, Parav Pandit wrote:
>
>> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
>> Behalf Of jeshwank
>> Sent: Monday, September 25, 2023 4:47 PM
>> To: virtio-dev@lists.oasis-open.org
> I notice this repeated occurs error of choosing the mailing list as virtio-dev for spec work.
> I will fix the OASIS document references.
>
> In the meant time, the mailing list should be virtio-comment@lists... For this patch.
>
>
>> In a virtual environment, an application running in guest VM may want to
>> delegate security sensitive tasks to a Trusted Application (TA) running within a
>> Trusted Execution Environment (TEE). A TEE is a trusted OS running in some
>> secure environment, for example, TrustZone on ARM CPUs, or a separate secure
>> co-processor etc.
>>
>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual TEE device
>> supports multiple operations such as:
>>
>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>                               TEE device.
>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>                                TEE device.
>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>                                trusted application running in TEE.
>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>                                 with trusted application running in TEE.
>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>                               application running in TEE.
>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>
>> We would like to reserve device ID 46 for Virtio-TEE device.
>>
>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
>
>> ---
>>   content.tex | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 0a62dce..644aa4a 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}  \hline
>>   45         &   SPI master \\
>>   \hline
>> +46         &   TEE device \\
>> +\hline
>>   \end{tabular}
>>
>>   Some of the devices above are unspecified by this document,
>> --
>> 2.25.1
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[NK, JESHWANTHKUMAR]

++virtio-comment@lists.oasis-open.org


On 25-Sep-23 9:08 PM, Parav Pandit wrote:
>
>> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
>> Behalf Of jeshwank
>> Sent: Monday, September 25, 2023 4:47 PM
>> To: virtio-dev@lists.oasis-open.org
> I notice this repeated occurs error of choosing the mailing list as virtio-dev for spec work.
> I will fix the OASIS document references.
>
> In the meant time, the mailing list should be virtio-comment@lists... For this patch.
>
>
>> In a virtual environment, an application running in guest VM may want to
>> delegate security sensitive tasks to a Trusted Application (TA) running within a
>> Trusted Execution Environment (TEE). A TEE is a trusted OS running in some
>> secure environment, for example, TrustZone on ARM CPUs, or a separate secure
>> co-processor etc.
>>
>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual TEE device
>> supports multiple operations such as:
>>
>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>                               TEE device.
>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>                                TEE device.
>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>                                trusted application running in TEE.
>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>                                 with trusted application running in TEE.
>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>                               application running in TEE.
>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>
>> We would like to reserve device ID 46 for Virtio-TEE device.
>>
>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
>
>> ---
>>   content.tex | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 0a62dce..644aa4a 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}  \hline
>>   45         &   SPI master \\
>>   \hline
>> +46         &   TEE device \\
>> +\hline
>>   \end{tabular}
>>
>>   Some of the devices above are unspecified by this document,
>> --
>> 2.25.1
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[NK, JESHWANTHKUMAR]

++virtio-comment@lists.oasis-open.org

On 25-Sep-23 9:08 PM, Parav Pandit wrote:
>
>> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
>> Behalf Of jeshwank
>> Sent: Monday, September 25, 2023 4:47 PM
>> To: virtio-dev@lists.oasis-open.org
> I notice this repeated occurs error of choosing the mailing list as virtio-dev for spec work.
> I will fix the OASIS document references.
>
> In the meant time, the mailing list should be virtio-comment@lists... For this patch.
>
>
>> In a virtual environment, an application running in guest VM may want to
>> delegate security sensitive tasks to a Trusted Application (TA) running within a
>> Trusted Execution Environment (TEE). A TEE is a trusted OS running in some
>> secure environment, for example, TrustZone on ARM CPUs, or a separate secure
>> co-processor etc.
>>
>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual TEE device
>> supports multiple operations such as:
>>
>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>                               TEE device.
>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>                                TEE device.
>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>                                trusted application running in TEE.
>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>                                 with trusted application running in TEE.
>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>                               application running in TEE.
>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>
>> We would like to reserve device ID 46 for Virtio-TEE device.
>>
>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
>
>> ---
>>   content.tex | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index 0a62dce..644aa4a 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}  \hline
>>   45         &   SPI master \\
>>   \hline
>> +46         &   TEE device \\
>> +\hline
>>   \end{tabular}
>>
>>   Some of the devices above are unspecified by this document,
>> --
>> 2.25.1
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

[-- Attachment #1: Type: text/plain, Size: 2661 bytes --]

+cc Jens

> In a virtual environment, an application running in guest VM may want
> to delegate security sensitive tasks to a Trusted Application (TA)
> running within a Trusted Execution Environment (TEE). A TEE is a trusted
> OS running in some secure environment, for example, TrustZone on ARM
> CPUs, or a separate secure co-processor etc.

I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.

Do you currently have any virtio frontend/backend implementations for this?

> 
> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
> TEE device supports multiple operations such as:
> 
> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>                              TEE device.
> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>                               TEE device.
> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>                               trusted application running in TEE.
> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>                                with trusted application running in TEE.
> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>                              application running in TEE.
> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
> 

How about shared memory support? We would like to register guest pages with the trusted OS.

-Sumit

> We would like to reserve device ID 46 for Virtio-TEE device.
> 
> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> ---
>  content.tex  <https://lore.kernel.org/all/80a2e4337affb043909c348395fb45aeeb693dc7.1695640593.git.JESHWANTHKUMAR.NK@amd.com/#Z31content.tex>  | 2 ++
>  1 file changed  <https://lore.kernel.org/all/80a2e4337affb043909c348395fb45aeeb693dc7.1695640593.git.JESHWANTHKUMAR.NK@amd.com/#related>, 2 insertions(+)
> 
> diff 
<https://lore.kernel.org/all/80a2e4337affb043909c348395fb45aeeb693dc7.1695640593.git.JESHWANTHKUMAR.NK@amd.com/#iZ31content.tex> 
--git a/content.tex b/content.tex > index 0a62dce..644aa4a 100644 > --- 
a/content.tex > +++ b/content.tex > @@ -739,6 +739,8 @@ \chapter{Device 
Types}\label{sec:Device Types} >  \hline
>  45         &   SPI master \\
>  \hline
> +46         &   TEE device \\  > +\hline > \end{tabular}
> 
>  Some of the devices above are unspecified by this document,

[-- Attachment #2: Type: text/html, Size: 4870 bytes --]

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

+cc Alex

On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>
> Hi,
>
> [+cc Arnd]
>
> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >
> > +cc Jens
> >
> > > In a virtual environment, an application running in guest VM may want
> > > to delegate security sensitive tasks to a Trusted Application (TA)
> > > running within a Trusted Execution Environment (TEE). A TEE is a trusted
> > > OS running in some secure environment, for example, TrustZone on ARM
> > > CPUs, or a separate secure co-processor etc.
> >
> > I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
> >
> > Do you currently have any virtio frontend/backend implementations for this?
> >
> > >
> > > A virtual TEE device emulates a TEE within a guest VM. Such a virtual
> > > TEE device supports multiple operations such as:
> > >
> > > VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
> > >                              TEE device.
> > > VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
> > >                               TEE device.
> > > VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
> > > VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
> > >                               trusted application running in TEE.
> > > VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
> > >                                with trusted application running in TEE.
> > > VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
> > >                              application running in TEE.
> > > VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
> > >
> >
> > How about shared memory support? We would like to register guest pages with the trusted OS.
>
> Coincidently Arnd and I (among others) discussed this in person last
> week and the conclusion was that only temporary shared memory is
> possible with virtio. So the shared memory has to be set up and torn
> down by the host during each operation, typically open-session or
> invoke-func.

Agree as I was part of those discussions. But I would like to
understand the reasoning behind it. Is there any restriction by VIRTIO
specification that we can't register guest page PAs to a device (TEE
in our case) to allow for zero copy transfers?

Alex mentioned some references to virtio GPU device. I suppose I need
to dive into its implementation to see if there are any similarities
to our use-case.

> That might not be optimal if trying to maximize
> performance, but it is portable.

IMO, the ABI should be flexible enough to support a TEE with optimum
performance.

-Sumit

>
> Cheers,
> Jens
>
> >
> > -Sumit
> >
> > > We would like to reserve device ID 46 for Virtio-TEE device.
> > >
> > > Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> > > ---
> > >  content.tex | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/content.tex b/content.tex
> > > index 0a62dce..644aa4a 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
> > >  \hline
> > >  45         &   SPI master \\
> > >  \hline
> > > +46         &   TEE device \\
> > > +\hline
> > > \end{tabular}
> > >
> > >  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Rijo Thomas]

On 9/26/2023 12:14 PM, Sumit Garg wrote:
> +cc Alex
> 
> On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>
>> Hi,
>>
>> [+cc Arnd]
>>
>> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>
>>> +cc Jens
>>>
>>>> In a virtual environment, an application running in guest VM may want
>>>> to delegate security sensitive tasks to a Trusted Application (TA)
>>>> running within a Trusted Execution Environment (TEE). A TEE is a trusted
>>>> OS running in some secure environment, for example, TrustZone on ARM
>>>> CPUs, or a separate secure co-processor etc.
>>>
>>> I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
>>>
Yes, we have tested with AMD-TEE. We have not yet tested with OP-TEE. Sure, we will try it out.

>>> Do you currently have any virtio frontend/backend implementations for this?
Yes, we have. Frontend is a Linux virtio-TEE driver, and backend is virtio-TEE device emulated in QEMU.
We used Xen hypervisor.

>>>
>>>>
>>>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
>>>> TEE device supports multiple operations such as:
>>>>
>>>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>>>                              TEE device.
>>>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>>>                               TEE device.
>>>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>>>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>>>                               trusted application running in TEE.
>>>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>>>                                with trusted application running in TEE.
>>>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>>>                              application running in TEE.
>>>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>>>
>>>
>>> How about shared memory support? We would like to register guest pages with the trusted OS.
>>
We have a command VIRTIO_TEE_CMD_REGISTER_MEM for registering shared memory buffer with Trusted OS.

In this command, the guest pages are copied into a shadow buffer in the host OS. And this shadow
buffer is mapped with Trusted OS. So, buffer-copy is involved.

One limitation, that we had was that the guest pages were non-contiguous. So, the number of physical
pages that had to be mapped with Trusted OS was exceeding 64 entries when we were testing out the
registering of guest pages. AMD-TEE Trusted OS can map a physically non-contiguous buffer, but the
number of sg entries for such a buffer must be less than 64. So, we resorted to using a shadow buffer
that is allocated within host, and gets mapped with Trusted OS.

Thanks,
Rijo

>> Coincidently Arnd and I (among others) discussed this in person last
>> week and the conclusion was that only temporary shared memory is
>> possible with virtio. So the shared memory has to be set up and torn
>> down by the host during each operation, typically open-session or
>> invoke-func.
> 
> Agree as I was part of those discussions. But I would like to
> understand the reasoning behind it. Is there any restriction by VIRTIO
> specification that we can't register guest page PAs to a device (TEE
> in our case) to allow for zero copy transfers?
> 
> Alex mentioned some references to virtio GPU device. I suppose I need
> to dive into its implementation to see if there are any similarities
> to our use-case.
> 
>> That might not be optimal if trying to maximize
>> performance, but it is portable.
> 
> IMO, the ABI should be flexible enough to support a TEE with optimum
> performance.
> 
> -Sumit
> 
>>
>> Cheers,
>> Jens
>>
>>>
>>> -Sumit
>>>
>>>> We would like to reserve device ID 46 for Virtio-TEE device.
>>>>
>>>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
>>>> ---
>>>>  content.tex | 2 ++
>>>>  1 file changed, 2 insertions(+)
>>>>
>>>> diff --git a/content.tex b/content.tex
>>>> index 0a62dce..644aa4a 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
>>>>  \hline
>>>>  45         &   SPI master \\
>>>>  \hline
>>>> +46         &   TEE device \\
>>>> +\hline
>>>> \end{tabular}
>>>>
>>>>  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

On Tue, 26 Sept 2023 at 12:53, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
>
> On 9/26/2023 12:14 PM, Sumit Garg wrote:
> > +cc Alex
> >
> > On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>
> >> Hi,
> >>
> >> [+cc Arnd]
> >>
> >> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >>>
> >>> +cc Jens
> >>>
> >>>> In a virtual environment, an application running in guest VM may want
> >>>> to delegate security sensitive tasks to a Trusted Application (TA)
> >>>> running within a Trusted Execution Environment (TEE). A TEE is a trusted
> >>>> OS running in some secure environment, for example, TrustZone on ARM
> >>>> CPUs, or a separate secure co-processor etc.
> >>>
> >>> I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
> >>>
> Yes, we have tested with AMD-TEE. We have not yet tested with OP-TEE. Sure, we will try it out.

Glad to hear that. I can help get it tested with OP-TEE as well.

>
> >>> Do you currently have any virtio frontend/backend implementations for this?
> >>>
>
> Yes, we have. Frontend is a Linux virtio-TEE driver, and backend is virtio-TEE device emulated in QEMU.
> We used the Xen hypervisor.

Can you share corresponding references? I can give it a try using Qemu with KVM.

>
> >>>>
> >>>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
> >>>> TEE device supports multiple operations such as:
> >>>>
> >>>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
> >>>>                              TEE device.
> >>>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
> >>>>                               TEE device.
> >>>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
> >>>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
> >>>>                               trusted application running in TEE.
> >>>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
> >>>>                                with trusted application running in TEE.
> >>>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
> >>>>                              application running in TEE.
> >>>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
> >>>>
> >>>
> >>> How about shared memory support? We would like to register guest pages with the trusted OS.
> We have a command VIRTIO_TEE_CMD_REGISTER_MEM for registering shared memory buffer with Trusted OS.

I suppose the commit message has to be appended then. Do you have the
draft virtio-tee device specification ready for review? I would be
interested to review that.

>
> In this command, the guest pages are copied into a shadow buffer in the host OS. And this shadow
> buffer is mapped with Trusted OS. So, buffer-copy is involved.
>
> One limitation, that we had was that the guest pages were non-contiguous. So, the number of physical
> pages that had to be mapped with Trusted OS was exceeding 64 entries when we were testing out the
> registering of guest pages. AMD-TEE Trusted OS can map a physically non-contiguous buffer, but the
> number of sg entries for such a buffer must be less than 64. So, we resorted to using a shadow buffer
> that is allocated within host, and gets mapped with Trusted OS.

I don't think OP-TEE OS has such a limitation on non-contiguous pages.
So I would suggest you to keep VIRTIO_TEE_CMD_REGISTER_MEM as part of
the ABI. It can be an optional feature for a particular trusted OS
implementation to support.

-Sumit

>
> Thanks,
> Rijo
>
> >>
> >> Coincidently Arnd and I (among others) discussed this in person last
> >> week and the conclusion was that only temporary shared memory is
> >> possible with virtio. So the shared memory has to be set up and torn
> >> down by the host during each operation, typically open-session or
> >> invoke-func.
> >
> > Agree as I was part of those discussions. But I would like to
> > understand the reasoning behind it. Is there any restriction by VIRTIO
> > specification that we can't register guest page PAs to a device (TEE
> > in our case) to allow for zero copy transfers?
> >
> > Alex mentioned some references to virtio GPU device. I suppose I need
> > to dive into its implementation to see if there are any similarities
> > to our use-case.
> >
> >> That might not be optimal if trying to maximize
> >> performance, but it is portable.
> >
> > IMO, the ABI should be flexible enough to support a TEE with optimum
> > performance.
> >
> > -Sumit
> >
> >>
> >> Cheers,
> >> Jens
> >>
> >>>
> >>> -Sumit
> >>>
> >>>> We would like to reserve device ID 46 for Virtio-TEE device.
> >>>>
> >>>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> >>>> ---
> >>>>  content.tex | 2 ++
> >>>>  1 file changed, 2 insertions(+)
> >>>>
> >>>> diff --git a/content.tex b/content.tex
> >>>> index 0a62dce..644aa4a 100644
> >>>> --- a/content.tex
> >>>> +++ b/content.tex
> >>>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
> >>>>  \hline
> >>>>  45         &   SPI master \\
> >>>>  \hline
> >>>> +46         &   TEE device \\
> >>>> +\hline
> >>>> \end{tabular}
> >>>>
> >>>>  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Rijo Thomas]

On 9/26/2023 1:19 PM, Sumit Garg wrote:
> On Tue, 26 Sept 2023 at 12:53, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
>>
>> On 9/26/2023 12:14 PM, Sumit Garg wrote:
>>> +cc Alex
>>>
>>> On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> [+cc Arnd]
>>>>
>>>> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>
>>>>> +cc Jens
>>>>>
>>>>>> In a virtual environment, an application running in guest VM may want
>>>>>> to delegate security sensitive tasks to a Trusted Application (TA)
>>>>>> running within a Trusted Execution Environment (TEE). A TEE is a trusted
>>>>>> OS running in some secure environment, for example, TrustZone on ARM
>>>>>> CPUs, or a separate secure co-processor etc.
>>>>>
>>>>> I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
>>>>>
>> Yes, we have tested with AMD-TEE. We have not yet tested with OP-TEE. Sure, we will try it out.
> 
> Glad to hear that. I can help get it tested with OP-TEE as well.
> 

We will test it out internally. Shall let you know in case we need help.

>>
>>>>> Do you currently have any virtio frontend/backend implementations for this?
>>>>>
>>
>> Yes, we have. Frontend is a Linux virtio-TEE driver, and backend is virtio-TEE device emulated in QEMU.
>> We used the Xen hypervisor.
> 
> Can you share corresponding references? I can give it a try using Qemu with KVM.
> 

We will share it in next couple of weeks. We have not yet hosted the code for external consumption.

>>
>>>>>>
>>>>>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
>>>>>> TEE device supports multiple operations such as:
>>>>>>
>>>>>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>>>>>                              TEE device.
>>>>>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>>>>>                               TEE device.
>>>>>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>>>>>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>>>>>                               trusted application running in TEE.
>>>>>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>>>>>                                with trusted application running in TEE.
>>>>>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>>>>>                              application running in TEE.
>>>>>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>>>>>
>>>>>
>>>>> How about shared memory support? We would like to register guest pages with the trusted OS.
>> We have a command VIRTIO_TEE_CMD_REGISTER_MEM for registering shared memory buffer with Trusted OS.
> 
> I suppose the commit message has to be appended then. Do you have the
> draft virtio-tee device specification ready for review? I would be
> interested to review that.
> 

Yes, the command is missed out in the commit message.

We are in the process of preparing virtio-tee device specification. We will be sending it out to this
list.

>>
>> In this command, the guest pages are copied into a shadow buffer in the host OS. And this shadow
>> buffer is mapped with Trusted OS. So, buffer-copy is involved.
>>
>> One limitation, that we had was that the guest pages were non-contiguous. So, the number of physical
>> pages that had to be mapped with Trusted OS was exceeding 64 entries when we were testing out the
>> registering of guest pages. AMD-TEE Trusted OS can map a physically non-contiguous buffer, but the
>> number of sg entries for such a buffer must be less than 64. So, we resorted to using a shadow buffer
>> that is allocated within host, and gets mapped with Trusted OS.
> 
> I don't think OP-TEE OS has such a limitation on non-contiguous pages.
> So I would suggest you to keep VIRTIO_TEE_CMD_REGISTER_MEM as part of
> the ABI. It can be an optional feature for a particular trusted OS
> implementation to support.
> 

Currently, the reg_mem (register memory) control is dictated by a flag in virtio-tee qemu code. This flag
for our testing was hard-coded as false. We will enhance our code, so that it is configurable. The value
of reg_mem shall be set to true/false depending upon whether the underlying TEE driver reports TEE_GEN_CAP_REG_MEM.

Thanks,
Rijo

> -Sumit
> 
>>
>> Thanks,
>> Rijo
>>
>>>>
>>>> Coincidently Arnd and I (among others) discussed this in person last
>>>> week and the conclusion was that only temporary shared memory is
>>>> possible with virtio. So the shared memory has to be set up and torn
>>>> down by the host during each operation, typically open-session or
>>>> invoke-func.
>>>
>>> Agree as I was part of those discussions. But I would like to
>>> understand the reasoning behind it. Is there any restriction by VIRTIO
>>> specification that we can't register guest page PAs to a device (TEE
>>> in our case) to allow for zero copy transfers?
>>>
>>> Alex mentioned some references to virtio GPU device. I suppose I need
>>> to dive into its implementation to see if there are any similarities
>>> to our use-case.
>>>
>>>> That might not be optimal if trying to maximize
>>>> performance, but it is portable.
>>>
>>> IMO, the ABI should be flexible enough to support a TEE with optimum
>>> performance.
>>>
>>> -Sumit
>>>
>>>>
>>>> Cheers,
>>>> Jens
>>>>
>>>>>
>>>>> -Sumit
>>>>>
>>>>>> We would like to reserve device ID 46 for Virtio-TEE device.
>>>>>>
>>>>>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
>>>>>> ---
>>>>>>  content.tex | 2 ++
>>>>>>  1 file changed, 2 insertions(+)
>>>>>>
>>>>>> diff --git a/content.tex b/content.tex
>>>>>> index 0a62dce..644aa4a 100644
>>>>>> --- a/content.tex
>>>>>> +++ b/content.tex
>>>>>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
>>>>>>  \hline
>>>>>>  45         &   SPI master \\
>>>>>>  \hline
>>>>>> +46         &   TEE device \\
>>>>>> +\hline
>>>>>> \end{tabular}
>>>>>>
>>>>>>  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

+cc OP-TEE ML

On Tue, 26 Sept 2023 at 13:47, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
>
> On 9/26/2023 1:19 PM, Sumit Garg wrote:
> > On Tue, 26 Sept 2023 at 12:53, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
> >>
> >> On 9/26/2023 12:14 PM, Sumit Garg wrote:
> >>> +cc Alex
> >>>
> >>> On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> [+cc Arnd]
> >>>>
> >>>> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >>>>>
> >>>>> +cc Jens
> >>>>>
> >>>>>> In a virtual environment, an application running in guest VM may want
> >>>>>> to delegate security sensitive tasks to a Trusted Application (TA)
> >>>>>> running within a Trusted Execution Environment (TEE). A TEE is a trusted
> >>>>>> OS running in some secure environment, for example, TrustZone on ARM
> >>>>>> CPUs, or a separate secure co-processor etc.
> >>>>>
> >>>>> I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
> >>>>>
> >> Yes, we have tested with AMD-TEE. We have not yet tested with OP-TEE. Sure, we will try it out.
> >
> > Glad to hear that. I can help get it tested with OP-TEE as well.
> >
>
> We will test it out internally. Shall let you know in case we need help.
>
> >>
> >>>>> Do you currently have any virtio frontend/backend implementations for this?
> >>>>>
> >>
> >> Yes, we have. Frontend is a Linux virtio-TEE driver, and backend is virtio-TEE device emulated in QEMU.
> >> We used the Xen hypervisor.
> >
> > Can you share corresponding references? I can give it a try using Qemu with KVM.
> >
>
> We will share it in next couple of weeks. We have not yet hosted the code for external consumption.
>
> >>
> >>>>>>
> >>>>>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
> >>>>>> TEE device supports multiple operations such as:
> >>>>>>
> >>>>>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
> >>>>>>                              TEE device.
> >>>>>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
> >>>>>>                               TEE device.
> >>>>>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
> >>>>>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
> >>>>>>                               trusted application running in TEE.
> >>>>>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
> >>>>>>                                with trusted application running in TEE.
> >>>>>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
> >>>>>>                              application running in TEE.
> >>>>>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
> >>>>>>
> >>>>>
> >>>>> How about shared memory support? We would like to register guest pages with the trusted OS.
> >> We have a command VIRTIO_TEE_CMD_REGISTER_MEM for registering shared memory buffer with Trusted OS.
> >
> > I suppose the commit message has to be appended then. Do you have the
> > draft virtio-tee device specification ready for review? I would be
> > interested to review that.
> >
>
> Yes, the command is missed out in the commit message.

With the commit message updated to include support for shared memory,
feel free to add:

Acked-by: Sumit Garg <sumit.garg@linaro.org>

>
> We are in the process of preparing virtio-tee device specification. We will be sending it out to this
> list.

I would also suggest you to CC: op-tee@lists.trustedfirmware.org for review.

>
> >>
> >> In this command, the guest pages are copied into a shadow buffer in the host OS. And this shadow
> >> buffer is mapped with Trusted OS. So, buffer-copy is involved.
> >>
> >> One limitation, that we had was that the guest pages were non-contiguous. So, the number of physical
> >> pages that had to be mapped with Trusted OS was exceeding 64 entries when we were testing out the
> >> registering of guest pages. AMD-TEE Trusted OS can map a physically non-contiguous buffer, but the
> >> number of sg entries for such a buffer must be less than 64. So, we resorted to using a shadow buffer
> >> that is allocated within host, and gets mapped with Trusted OS.
> >
> > I don't think OP-TEE OS has such a limitation on non-contiguous pages.
> > So I would suggest you to keep VIRTIO_TEE_CMD_REGISTER_MEM as part of
> > the ABI. It can be an optional feature for a particular trusted OS
> > implementation to support.
> >
>
> Currently, the reg_mem (register memory) control is dictated by a flag in virtio-tee qemu code. This flag
> for our testing was hard-coded as false. We will enhance our code, so that it is configurable. The value
> of reg_mem shall be set to true/false depending upon whether the underlying TEE driver reports TEE_GEN_CAP_REG_MEM.

Sounds good to me.

-Sumit

>
> Thanks,
> Rijo
>
> > -Sumit
> >
> >>
> >> Thanks,
> >> Rijo
> >>
> >>>>
> >>>> Coincidently Arnd and I (among others) discussed this in person last
> >>>> week and the conclusion was that only temporary shared memory is
> >>>> possible with virtio. So the shared memory has to be set up and torn
> >>>> down by the host during each operation, typically open-session or
> >>>> invoke-func.
> >>>
> >>> Agree as I was part of those discussions. But I would like to
> >>> understand the reasoning behind it. Is there any restriction by VIRTIO
> >>> specification that we can't register guest page PAs to a device (TEE
> >>> in our case) to allow for zero copy transfers?
> >>>
> >>> Alex mentioned some references to virtio GPU device. I suppose I need
> >>> to dive into its implementation to see if there are any similarities
> >>> to our use-case.
> >>>
> >>>> That might not be optimal if trying to maximize
> >>>> performance, but it is portable.
> >>>
> >>> IMO, the ABI should be flexible enough to support a TEE with optimum
> >>> performance.
> >>>
> >>> -Sumit
> >>>
> >>>>
> >>>> Cheers,
> >>>> Jens
> >>>>
> >>>>>
> >>>>> -Sumit
> >>>>>
> >>>>>> We would like to reserve device ID 46 for Virtio-TEE device.
> >>>>>>
> >>>>>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
> >>>>>> ---
> >>>>>>  content.tex | 2 ++
> >>>>>>  1 file changed, 2 insertions(+)
> >>>>>>
> >>>>>> diff --git a/content.tex b/content.tex
> >>>>>> index 0a62dce..644aa4a 100644
> >>>>>> --- a/content.tex
> >>>>>> +++ b/content.tex
> >>>>>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
> >>>>>>  \hline
> >>>>>>  45         &   SPI master \\
> >>>>>>  \hline
> >>>>>> +46         &   TEE device \\
> >>>>>> +\hline
> >>>>>> \end{tabular}
> >>>>>>
> >>>>>>  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Rijo Thomas]

On 9/26/2023 6:02 PM, Sumit Garg wrote:
> +cc OP-TEE ML
> 
> On Tue, 26 Sept 2023 at 13:47, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
>>
>> On 9/26/2023 1:19 PM, Sumit Garg wrote:
>>> On Tue, 26 Sept 2023 at 12:53, Rijo Thomas <Rijo-john.Thomas@amd.com> wrote:
>>>>
>>>> On 9/26/2023 12:14 PM, Sumit Garg wrote:
>>>>> +cc Alex
>>>>>
>>>>> On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> [+cc Arnd]
>>>>>>
>>>>>> On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>>>
>>>>>>> +cc Jens
>>>>>>>
>>>>>>>> In a virtual environment, an application running in guest VM may want
>>>>>>>> to delegate security sensitive tasks to a Trusted Application (TA)
>>>>>>>> running within a Trusted Execution Environment (TEE). A TEE is a trusted
>>>>>>>> OS running in some secure environment, for example, TrustZone on ARM
>>>>>>>> CPUs, or a separate secure co-processor etc.
>>>>>>>
>>>>>>> I have been exploring this area quite recently with an effort to have a common VIRIO interface which can support different trusted OS implementations. I guess you intend to test it with AMD-TEE, right? Any plans to test it with OP-TEE? As currently we have these two supported upstream.
>>>>>>>
>>>> Yes, we have tested with AMD-TEE. We have not yet tested with OP-TEE. Sure, we will try it out.
>>>
>>> Glad to hear that. I can help get it tested with OP-TEE as well.
>>>
>>
>> We will test it out internally. Shall let you know in case we need help.
>>
>>>>
>>>>>>> Do you currently have any virtio frontend/backend implementations for this?
>>>>>>>
>>>>
>>>> Yes, we have. Frontend is a Linux virtio-TEE driver, and backend is virtio-TEE device emulated in QEMU.
>>>> We used the Xen hypervisor.
>>>
>>> Can you share corresponding references? I can give it a try using Qemu with KVM.
>>>
>>
>> We will share it in next couple of weeks. We have not yet hosted the code for external consumption.
>>
>>>>
>>>>>>>>
>>>>>>>> A virtual TEE device emulates a TEE within a guest VM. Such a virtual
>>>>>>>> TEE device supports multiple operations such as:
>>>>>>>>
>>>>>>>> VIRTIO_TEE_CMD_OPEN_DEVICE – Open a communication channel with virtio
>>>>>>>>                              TEE device.
>>>>>>>> VIRTIO_TEE_CMD_CLOSE_DEVICE – Close communication channel with virtio
>>>>>>>>                               TEE device.
>>>>>>>> VIRTIO_TEE_CMD_GET_VERSION – Get version of virtio TEE.
>>>>>>>> VIRTIO_TEE_CMD_OPEN_SESSION – Open a session to communicate with
>>>>>>>>                               trusted application running in TEE.
>>>>>>>> VIRTIO_TEE_CMD_CLOSE_SESSION – Close a session to end communication
>>>>>>>>                                with trusted application running in TEE.
>>>>>>>> VIRTIO_TEE_CMD_INVOKE_FUNC – Invoke a command or function in trusted
>>>>>>>>                              application running in TEE.
>>>>>>>> VIRTIO_TEE_CMD_CANCEL_REQ – Cancel an ongoing command within TEE.
>>>>>>>>
>>>>>>>
>>>>>>> How about shared memory support? We would like to register guest pages with the trusted OS.
>>>> We have a command VIRTIO_TEE_CMD_REGISTER_MEM for registering shared memory buffer with Trusted OS.
>>>
>>> I suppose the commit message has to be appended then. Do you have the
>>> draft virtio-tee device specification ready for review? I would be
>>> interested to review that.
>>>
>>
>> Yes, the command is missed out in the commit message.
> 
> With the commit message updated to include support for shared memory,
> feel free to add:
> 
> Acked-by: Sumit Garg <sumit.garg@linaro.org>
> 

Okay. I have asked Jeshwanth to post a revised patch with updated commit message.

>>
>> We are in the process of preparing virtio-tee device specification. We will be sending it out to this
>> list.
> 
> I would also suggest you to CC: op-tee@lists.trustedfirmware.org for review.
> 

Okay.

Thanks,
Rijo

>>
>>>>
>>>> In this command, the guest pages are copied into a shadow buffer in the host OS. And this shadow
>>>> buffer is mapped with Trusted OS. So, buffer-copy is involved.
>>>>
>>>> One limitation, that we had was that the guest pages were non-contiguous. So, the number of physical
>>>> pages that had to be mapped with Trusted OS was exceeding 64 entries when we were testing out the
>>>> registering of guest pages. AMD-TEE Trusted OS can map a physically non-contiguous buffer, but the
>>>> number of sg entries for such a buffer must be less than 64. So, we resorted to using a shadow buffer
>>>> that is allocated within host, and gets mapped with Trusted OS.
>>>
>>> I don't think OP-TEE OS has such a limitation on non-contiguous pages.
>>> So I would suggest you to keep VIRTIO_TEE_CMD_REGISTER_MEM as part of
>>> the ABI. It can be an optional feature for a particular trusted OS
>>> implementation to support.
>>>
>>
>> Currently, the reg_mem (register memory) control is dictated by a flag in virtio-tee qemu code. This flag
>> for our testing was hard-coded as false. We will enhance our code, so that it is configurable. The value
>> of reg_mem shall be set to true/false depending upon whether the underlying TEE driver reports TEE_GEN_CAP_REG_MEM.
> 
> Sounds good to me.
> 
> -Sumit
> 
>>
>> Thanks,
>> Rijo
>>
>>> -Sumit
>>>
>>>>
>>>> Thanks,
>>>> Rijo
>>>>
>>>>>>
>>>>>> Coincidently Arnd and I (among others) discussed this in person last
>>>>>> week and the conclusion was that only temporary shared memory is
>>>>>> possible with virtio. So the shared memory has to be set up and torn
>>>>>> down by the host during each operation, typically open-session or
>>>>>> invoke-func.
>>>>>
>>>>> Agree as I was part of those discussions. But I would like to
>>>>> understand the reasoning behind it. Is there any restriction by VIRTIO
>>>>> specification that we can't register guest page PAs to a device (TEE
>>>>> in our case) to allow for zero copy transfers?
>>>>>
>>>>> Alex mentioned some references to virtio GPU device. I suppose I need
>>>>> to dive into its implementation to see if there are any similarities
>>>>> to our use-case.
>>>>>
>>>>>> That might not be optimal if trying to maximize
>>>>>> performance, but it is portable.
>>>>>
>>>>> IMO, the ABI should be flexible enough to support a TEE with optimum
>>>>> performance.
>>>>>
>>>>> -Sumit
>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Jens
>>>>>>
>>>>>>>
>>>>>>> -Sumit
>>>>>>>
>>>>>>>> We would like to reserve device ID 46 for Virtio-TEE device.
>>>>>>>>
>>>>>>>> Signed-off-by: Jeshwanth Kumar <jeshwanthkumar.nk@amd.com>
>>>>>>>> ---
>>>>>>>>  content.tex | 2 ++
>>>>>>>>  1 file changed, 2 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/content.tex b/content.tex
>>>>>>>> index 0a62dce..644aa4a 100644
>>>>>>>> --- a/content.tex
>>>>>>>> +++ b/content.tex
>>>>>>>> @@ -739,6 +739,8 @@ \chapter{Device Types}\label{sec:Device Types}
>>>>>>>>  \hline
>>>>>>>>  45         &   SPI master \\
>>>>>>>>  \hline
>>>>>>>> +46         &   TEE device \\
>>>>>>>> +\hline
>>>>>>>> \end{tabular}
>>>>>>>>
>>>>>>>>  Some of the devices above are unspecified by this document,

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

On Wed, 27 Sept 2023 at 21:39, Arnd Bergmann <arnd@linaro.org> wrote:
>
> On Wed, 27 Sept 2023 at 16:09, Sumit Garg <sumit.garg@linaro.org> wrote:
> >
>
> It looks like I accidentally dropped the entire Cc list in my earlier reply,
> adding them all back.

No worries.

>
> > On Wed, 27 Sept 2023 at 01:44, Arnd Bergmann <arnd@linaro.org> wrote:
> > >
> > > On Tue, 26 Sept 2023 at 08:44, Sumit Garg <sumit.garg@linaro.org> wrote:
> > > > On Tue, 26 Sept 2023 at 08:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > > > On Tue, Sep 26, 2023 at 8:00 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > > > >
> > > > > > How about shared memory support? We would like to register guest pages with the trusted OS.
> > > > >
> > > > > Coincidently Arnd and I (among others) discussed this in person last
> > > > > week and the conclusion was that only temporary shared memory is
> > > > > possible with virtio. So the shared memory has to be set up and torn
> > > > > down by the host during each operation, typically open-session or
> > > > > invoke-func.
> > > >
> > > > Agree as I was part of those discussions. But I would like to
> > > > understand the reasoning behind it. Is there any restriction by VIRTIO
> > > > specification that we can't register guest page PAs to a device (TEE
> > > > in our case) to allow for zero copy transfers?
> > > >
> > > > Alex mentioned some references to virtio GPU device. I suppose I need
> > > > to dive into its implementation to see if there are any similarities
> > > > to our use-case.
> > > >
> > > > > That might not be optimal if trying to maximize
> > > > > performance, but it is portable.
> > > >
> > > > IMO, the ABI should be flexible enough to support a TEE with optimum
> > > > performance.
> > >
> > > As we discussed last week, I can see two possible ways to implement
> > > a TEE device within the constraints of the virtio specification:
> > >
> > > a) Allocate a shared memory area in the device (host) and export it
> > >    to the driver (guest) via a virtio shared memory area. This shared
> > >    memory can be shared with userspace using mmap() if necessary.
> > >    A tee command in this case would be sent using a normal virtqueue
> > >    to the device with a pair of one transmit and one receive buffer.
> > >    Any arguments that refer to memory blocks in this case are
> > >    offsets into the shared memory area. Using this preallocated buffer
> > >    is similar to earlier TEE implementations but has some restrictions.
> > >    The command in this case has to be copied in and out by the
> > >    hypervisor implementation.
> >
> > Yeah that was the initial approach that we used for OP-TEE but it was
> > limited by the fixed size of the shared memory area. A guest client
> > application may want to share a larger data buffer with TA which can't
> > be supported via this approach.
>
> I don't know if there is a limit on the size of the virtio shared
> memory area other than the PCI MMIO space size, could
> this just be made (much) larger?

Actually it's a double edged sword. You wouldn't like to block too
much host memory which remains unused and on the other hand not too
less that can't serve guest application needs. That's the reason we
should provide an option to dynamically share guest memory with a TEE.

>
> > > b) Send all data through the virtqueue itself, pointing into normal
> > >    guest memory. The first buffer sent to the device is the request,
> > >    while the receive buffer is the result. Instead of pointers to
> > >    shared memory, this means that all data transfers would be
> > >    done in additional buffers on the same virtio transaction, and
> > >    the host would have to register the guest memory dynamically
> > >    as part of the command before forwarding them to a TEE that
> > >    relies on registering shared memory, and unmap it afterwards
> > >    since the guest might reuse the buffers for other data later
> > >    that it does not want to share with the TEE
> > >
> >
> > The TEE communication protocol has to support INOUT shared memory
> > buffers. So it will be quite tricky to support it via TX only and RX
> > only buffers (many more buffer copies).
>
> I don't remember if you can just list the same address in
> virtio for both directions, but that could solve this problem.
>

Okay in that case I think following approach can work:

- Issue VIRTIO_TEE_CMD_REGISTER_MEM to register an additional buffer
passed through virtqueue.
- Instruct TEE to perform operations on it via VIRTIO_TEE_CMD_INVOKE_FUNC.
- Issue VIRTIO_TEE_CMD_UNREGISTER_MEM to unregister that additional
buffer passed through virtqueue.

I suppose we can pass either guest user-space or kernel reference in
that virtqueue. Does this approach make sense to you?

-Sumit

> > > Registering guest memory to the TEE permanently would be
> > > a layering violation since that makes invalid assumptions about
> > > the type of virtio transport that do not make sense to a virtio
> > > driver.
> >
> > AFAICS, the VIRTIO is just a transport to relay information among
> > guest kernel drivers and host emulated devices. The registration of
> > guest memory to TEE won't be permanent but rather has a limited
> > lifetime which is alive until the guest client application closes the
> > context with TEE. So once the TEE context is closed by the guest
> > client application then all the corresponding registered memory will
> > be freed.
>
> Right, so while in theory you can implement random non-virtio
> semantics by passing other commands through a virtqueue, I would
> no longer consider it a virtio driver at that point, since it makes
> assumptions about the host system implementation beyond what
> is abstracted in virtio.
>
> > > As far as the driver is concerned, the virtqueue is a
> > > socket type interface that does transactions on input and
> > > output data in place but has no concept of guest memory.
> >
> > That's true. As part of virtio-tee, we will pass page pointers in that
> > virtio input/output data in place. I suppose it's better to discuss
> > the implementation details once AMD folks put the virtio-tee
> > specification out for review. Also, they have implementations for
> > virtio-tee frontend and backend too which they will make up for public
> > review.
> >
> > There certainly can be some essential details of virtio spec that I am
> > missing here since I only started exploring it a month back. If you
> > have some certain pointers to the spec then I will be happy to read
> > them carefully.
>
> I have not found an explicit wording that forbids you from
> referencing physical memory addresses indirectly in buffers
> that are passed through virtqueues, but the basic definition of
> a virtqueue in section 2.6 [1] describes the way that buffers
> are passed, and if a TEE driver wants to pass commands
> and their arguments in something that is not a virtqueue or
> a virtio-shmem area, I think you are clearly outside of the intended
> model.
>
>      Arnd
>
> https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html#x1-270006

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v2] virtio-tee: Reserve device ID 46 for TEE device

[Sumit Garg]

On Tue, 3 Oct 2023 at 19:06, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Thu, Sep 28, 2023, at 16:48, Sumit Garg wrote:
> > On Wed, 27 Sept 2023 at 21:39, Arnd Bergmann <arnd@linaro.org> wrote:
> >> On Wed, 27 Sept 2023 at 16:09, Sumit Garg <sumit.garg@linaro.org> wrote:
> >>
> >> I don't know if there is a limit on the size of the virtio shared
> >> memory area other than the PCI MMIO space size, could
> >> this just be made (much) larger?
> >
> > Actually it's a double edged sword. You wouldn't like to block too
> > much host memory which remains unused and on the other hand not too
> > less that can't serve guest application needs. That's the reason we
> > should provide an option to dynamically share guest memory with a TEE.
>
> To clarify: the address space in the virtio-shmem segment
> does not have to be permanently backed by host memory, it
> just provides a part of the guest-physical mmio space and
> could be populated as needed. So you might have a 1TB
> shmem region in the device itself but only use a single 4KB
> page in it.

Thanks for the clarification. It makes sense to use this approach for
TEE shared memory allocation purposes. The guest client application
should be able to mmap() the buffer allocated from virtio-shmem space.

>
> The restriction here is that the address of the shmem segment
> itself is fixed on the PCI bus (or whichever virtio transport
> is used), rather than decided by the guest from its memory.
>

I can understand this restriction.

> >> > The TEE communication protocol has to support INOUT shared memory
> >> > buffers. So it will be quite tricky to support it via TX only and RX
> >> > only buffers (many more buffer copies).
> >>
> >> I don't remember if you can just list the same address in
> >> virtio for both directions, but that could solve this problem.
> >
> > Okay in that case I think following approach can work:
> >
> > - Issue VIRTIO_TEE_CMD_REGISTER_MEM to register an additional buffer
> > passed through virtqueue.
> > - Instruct TEE to perform operations on it via VIRTIO_TEE_CMD_INVOKE_FUNC.
> > - Issue VIRTIO_TEE_CMD_UNREGISTER_MEM to unregister that additional
> > buffer passed through virtqueue.
> >
> > I suppose we can pass either guest user-space or kernel reference in
> > that virtqueue. Does this approach make sense to you?
>
> No, I'm not sure what you are suggesting here, i.e. which address
> space VIRTIO_TEE_CMD_REGISTER_MEM would operate on. This would
> normally be guest physical memory, which might correspond to pinned
> userspace pages, but I don't think that makes sense in the context
> of virtio, as we discussed before.

I suppose this is a similar situation with DMA buffers too, correct?
Is it not allowed to share DMA buffers backed by guest physical memory
with virtio devices?

As otherwise, in case of VIRTIO-TEE we will be reluctant to maintain
shadow/bounce buffers in virtio-shmem space which will be inefficient.

> If you would use the virtio
> shmem backing, VIRTIO_TEE_CMD_REGISTER_MEM could refer to an offset
> within the device shmem address space, which would then become
> backed by host memory.

Yeah that sounds like a sensible approach for buffers allocated from
virtio-shmem space.

>
> IIRC the way we had discussed this before was that the virtio-tee
> driver would not register memory at all, but instead pass all
> requests through virtqueues. When the device (host) passes the
> request up to the actual TEE, it could transparently register
> the buffer before the transaction and unregister it afterwards,
> or copy the data to a pre-registered area.

This refers to temporary shared memory in TEE terms. It is the least
efficient approach. It may turn a single VIRTIO_TEE_CMD_INVOKE_FUNC
into at max 9 transactions underneath among host and TEE: 8
register/unregister shared memory invocations and 1 actual invoke
command. This can be turned into a single transaction among host and
TEE if we can make the above discussed approaches to work.

-Sumit

>
>     Arnd

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org