[Qemu-devel] [virtio-dev][RFC v2 0/2] Signal Distribution Module virtio device specification

[Qemu-devel] [virtio-dev][RFC v2 0/2] Signal Distribution Module virtio device specification

[Christian Pinto]

Hi all,

This patch series proposes the specification of a new virtio device on which we
are working on, namely the Signal Distribution Module (SDM). 
The SDM routes inter-processor signals using a user-defined communication 
channel. A possible use-case of the SDM is an AMP system where the vairous processors
need to exhange signals (e.g., interrupts) to implement inter-processor
communication patterns. An example are RPMSG and remoteproc, both using
interrupts to kick the virtio queues used for communication (RPMSG), and a boot
signal to start the execution of a slave processor (remoteproc).

However the SDM is not limited to those use-cases and can be easily extended to
support user-defined signals and communication channels.

In addition to the virtio version, a platform device version is available as
well to be used for cases where a processor is not running Linux but another
OS/firmware that does not support virtio.

This patch series is related to :

[Qemu-devel][RFC v3 0/6] SDM Interface
https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg05121.html

where you can find the latest RFC patch set for the QEMU code of the virtio SDM 
device.

Kernel code is publicly accessible from:

https://git.virtualopensystems.com/dev/qemu-het-tools

branch sdm_test_virtio_mod_v2.

QEMU code is accessible from:

https://git.virtualopensystems.com/dev/qemu-het-tools

branch sdm-dev-v3

At the current state the SDM is implemented as a QEMU device and 
provides a local channel, for intra-QEMU signals, and a channel based on  
sockets (UNIX or TCP) to exchange signals between processors in different
instances of QEMU. 
Each communication channel exports a common interface for the sake of ease of
extension and integration of new channels.


Thanks,

Christian

---
Changes since v1:
- Fixed some typos
- Removed dependencies from QEMU
- Added explanation on how SDM can be used in AMP systems
- Explained semantics of payload field in SDMSignalData struct
---

Christian Pinto (2):
  content: reserve virtio device ID
  virtio-sdm: new device specification

 content.tex    |   4 ++
 virtio-sdm.tex | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+)
 create mode 100644 virtio-sdm.tex

-- 
1.9.1

[Qemu-devel] [virtio-dev][RFC v2 1/2] content: reserve virtio device ID

[Christian Pinto]

Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>

---
v1 -> v2:
- removed edit including virtio-sdm specs text
---
 content.tex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content.tex b/content.tex
index d989d98..3317916 100644
--- a/content.tex
+++ b/content.tex
@@ -2990,6 +2990,8 @@ Device ID  &  Virtio Device    \\
 \hline
 18         &   Input device \\
 \hline
+21         &   Signal Distribution Module \\
+\hline
 \end{tabular}
 
 Some of the devices above are unspecified by this document,
-- 
1.9.1

Re: [Qemu-devel] [virtio-dev][RFC v2 1/2] content: reserve virtio device ID

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

On Tue, Jun 28, 2016 at 01:03:43PM +0200, Christian Pinto wrote:
> Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
> Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
> 
> ---
> v1 -> v2:
> - removed edit including virtio-sdm specs text
> ---
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [virtio-dev][RFC v2 1/2] content: reserve virtio device ID

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Tue, Jun 28, 2016 at 01:03:43PM +0200, Christian Pinto wrote:
> Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
> Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
> 
> ---
> v1 -> v2:
> - removed edit including virtio-sdm specs text
> ---
>  content.tex | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index d989d98..3317916 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2990,6 +2990,8 @@ Device ID  &  Virtio Device    \\
>  \hline
>  18         &   Input device \\
>  \hline
> +21         &   Signal Distribution Module \\
> +\hline
>  \end{tabular}
>  
>  Some of the devices above are unspecified by this document,
> -- 
> 1.9.1
> 

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Christian Pinto]

This patch adds the specification of the Signal Dristribution Module virtio
device to the current virtio specification document.

Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>

---
v1 -> v2:
- Fixed some typos
- Removed dependencies from QEMU
- Added explanation on how SDM can be used in AMP systems
- Explained semantics of payload field in SDMSignalData struct
---
 content.tex    |   2 +
 virtio-sdm.tex | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 130 insertions(+)
 create mode 100644 virtio-sdm.tex

diff --git a/content.tex b/content.tex
index 3317916..7fcf779 100644
--- a/content.tex
+++ b/content.tex
@@ -5643,6 +5643,8 @@ descriptor for the \field{sense_len}, \field{residual},
 \field{status_qualifier}, \field{status}, \field{response} and
 \field{sense} fields.
 
+\input{virtio-sdm.tex}
+
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
 Currently there are three device-independent feature bits defined:
diff --git a/virtio-sdm.tex b/virtio-sdm.tex
new file mode 100644
index 0000000..76b0a7b
--- /dev/null
+++ b/virtio-sdm.tex
@@ -0,0 +1,128 @@
+\section{Signal Distribution Module}\label{sec:Device Types / SDM Device}
+
+The virtio Signal Distribution Module is meant to enable Inter Processor signal
+exchange.
+An example are Inter Processor Interrupts used in AMP systems, for the
+processors involved to notify the presence of new data in the communication
+queues.
+In AMP systems signals are used for various purposes, an example are remoteproc
+or RPMSG. In the former signals are used by a master processor to trigger
+the boot of a slave processor. While the latter, RPMSG, uses virtio queues as a
+message exchange medium between processors. In this case the SDM can be used to
+generate the interrupt associated to the kick of a virtio queue.
+
+\subsection{Device ID}\label{sec:Device Types / SDM Device / Device ID}
+
+21
+
+\subsection{Virtqueues}\label{sec:Device Types / SDM Device / Virtqueues}
+
+\begin{description}
+\item[0] hg_vq
+\item[1] gh_vq
+\end{description}
+
+Queue 0 is used for device-to-driver communication (i.e., notification of a
+signal), while queue 1 for driver-to-device communication.
+
+\subsection{Feature bits}\label{sec:Device Types / SDM Device / Feature bits}
+
+None.
+
+\subsection{Device configuration layout}\label{sec:Device Types / SDM Device /
+Device configuration layout}
+
+The device configuration is composed by three fields: \texttt{master},
+\texttt{max_slaves} and \texttt{current_slaves}.
+
+\begin{lstlisting}
+struct virtio_sdm_config {
+	u8    master;
+	u16   max_slaves;
+	u16   current_slaves;
+};
+\end{lstlisting}
+
+The SDM device can be instantiated either as a master or as a slave. The master
+slave behavior is meant on purpose to reflect the AMP like type of communication
+where a master processor controls one or more slave processors.
+A master SDM instance can send a signal to any of the slaves instances,
+while slaves can only signal the master.
+
+The \texttt{master} field of the configuration space is meant to be read by the
+driver to figure out whether a specific instance is a master or a slave.
+The \texttt{max_slaves} field contains the maximum number of slaves supported by
+the SDM device. A notification change is sent to the driver each time the value
+of \texttt{max_slaves} is changed from the device side.
+Finally, the \texttt{current_slaves} field contains the actual number of slaves
+registered to the master SDM. This field is a read only field.
+
+\subsection{Device Initialization}\label{sec:Device Types / SDM Device /
+evice Initialization}
+
+During initialization the \texttt{hg_vq} and \texttt{gh_vq} are identified and
+the device is immediately operational. A master driver instance can access the
+number of slaves registered at any time by reading the configuration space of
+the device.
+
+During the initialization phase the device connects also to the communication
+channel. It has to be noted that the behavior of the device is
+independent from the communication channel used, that is a detail of each
+specific implementation of the SDM device.
+
+\subsection{Device Operation}\label{sec:Device Types / SDM Device / Device
+peration}
+
+The SDM device handles signals coming from the two following sources:
+
+\begin{enumerate}
+\item The local processor to which the device is attached to.
+\item The communication channel connecting to other slaves/master.
+\end{enumerate}
+
+The first case is verified when the processor attached to the SDM device wants
+to send a signal to a second SDM device instance.
+The second case is instead when an SDM device instance receives a signal from
+another SDM device, to be forwarded to the local processor.
+It is important to note that due to the master/slave behavior, slaves cannot
+signal among themselves but only with the master SDM instance.
+
+In both cases, before sending over the communication channel, the signal is
+packed in the \texttt{SDMSignalData} data structure.
+
+\begin{lstlisting}
+enum sdm_signal_type {
+    SDM_IRQ,
+    SDM_BOOT,
+};
+
+struct SDMSignalData {
+    uint32_t type;
+    uint32_t slave;
+    uint32_t payload[2];
+};
+\end{lstlisting}
+
+The \texttt{type} field indicates the type of signal to be sent to the
+destination SDM. The current implementation supports two signal types.
+The \texttt{SDM_IRQ} signal is used to send an inter processor interrupt, while
+the \texttt{SDM_BOOT} signal is sent to trigger the boot of the destination
+processor. The boot signal is meant to be used in an AMP like scenario where a
+master processor boots one or more slave processors (e.g., via remoteproc).
+The \texttt{slave} field contains the id of the SDM instance destination of the
+signal. Id 0 is reserved for the master, from 1 onwards for the slaves.
+This means that the \texttt{slave} field will always contain 0 when the source
+of the signal is a slave SDM instance, while the actual id of the slave in case
+of a master.
+The \texttt{payload} is used to pass extra accompanying information with the
+signal.
+The semantics of the payload field depends on the signal itself. In case of
+\texttt{SDM_IRQ} signal, the payload field is ignored since interrupts do not
+need any extra information to be handled. In the case of \texttt{SDM_BOOT}
+signal the payload contains the boot address of the slave processor, to be used
+at the destination to initialize the program counter register before the actual
+boot process is started.
+
+
+The \texttt{SDMSignalData} structure is first filled by the source SDM kernel
+virtio driver and sent over the gh_vq.
-- 
1.9.1

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7672 bytes --]

On Tue, Jun 28, 2016 at 01:03:44PM +0200, Christian Pinto wrote:
> This patch adds the specification of the Signal Dristribution Module virtio
> device to the current virtio specification document.
> 
> Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
> Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
> 
> ---
> v1 -> v2:
> - Fixed some typos
> - Removed dependencies from QEMU
> - Added explanation on how SDM can be used in AMP systems
> - Explained semantics of payload field in SDMSignalData struct
> ---
>  content.tex    |   2 +
>  virtio-sdm.tex | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 130 insertions(+)
>  create mode 100644 virtio-sdm.tex
> 
> diff --git a/content.tex b/content.tex
> index 3317916..7fcf779 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5643,6 +5643,8 @@ descriptor for the \field{sense_len}, \field{residual},
>  \field{status_qualifier}, \field{status}, \field{response} and
>  \field{sense} fields.
>  
> +\input{virtio-sdm.tex}
> +
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
>  Currently there are three device-independent feature bits defined:
> diff --git a/virtio-sdm.tex b/virtio-sdm.tex
> new file mode 100644
> index 0000000..76b0a7b
> --- /dev/null
> +++ b/virtio-sdm.tex
> @@ -0,0 +1,128 @@
> +\section{Signal Distribution Module}\label{sec:Device Types / SDM Device}
> +
> +The virtio Signal Distribution Module is meant to enable Inter Processor signal
> +exchange.
> +An example are Inter Processor Interrupts used in AMP systems, for the
> +processors involved to notify the presence of new data in the communication
> +queues.
> +In AMP systems signals are used for various purposes, an example are remoteproc
> +or RPMSG. In the former signals are used by a master processor to trigger
> +the boot of a slave processor. While the latter, RPMSG, uses virtio queues as a
> +message exchange medium between processors. In this case the SDM can be used to
> +generate the interrupt associated to the kick of a virtio queue.
> +
> +\subsection{Device ID}\label{sec:Device Types / SDM Device / Device ID}
> +
> +21
> +
> +\subsection{Virtqueues}\label{sec:Device Types / SDM Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] hg_vq
> +\item[1] gh_vq
> +\end{description}
> +
> +Queue 0 is used for device-to-driver communication (i.e., notification of a
> +signal), while queue 1 for driver-to-device communication.
> +
> +\subsection{Feature bits}\label{sec:Device Types / SDM Device / Feature bits}
> +
> +None.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / SDM Device /
> +Device configuration layout}
> +
> +The device configuration is composed by three fields: \texttt{master},
> +\texttt{max_slaves} and \texttt{current_slaves}.
> +
> +\begin{lstlisting}
> +struct virtio_sdm_config {
> +	u8    master;

The next field has 16-bit alignment so there is a hole in this struct.
Three options:

1. Add an explicit u8 padding field.
2. Change master's type from u8 to u16.
3. Move master to the end of the struct.

> +	u16   max_slaves;
> +	u16   current_slaves;
> +};
> +\end{lstlisting}
> +
> +The SDM device can be instantiated either as a master or as a slave. The master
> +slave behavior is meant on purpose to reflect the AMP like type of communication
> +where a master processor controls one or more slave processors.
> +A master SDM instance can send a signal to any of the slaves instances,
> +while slaves can only signal the master.
> +
> +The \texttt{master} field of the configuration space is meant to be read by the
> +driver to figure out whether a specific instance is a master or a slave.
> +The \texttt{max_slaves} field contains the maximum number of slaves supported by
> +the SDM device. A notification change is sent to the driver each time the value
> +of \texttt{max_slaves} is changed from the device side.
> +Finally, the \texttt{current_slaves} field contains the actual number of slaves
> +registered to the master SDM. This field is a read only field.
> +
> +\subsection{Device Initialization}\label{sec:Device Types / SDM Device /
> +evice Initialization}
> +
> +During initialization the \texttt{hg_vq} and \texttt{gh_vq} are identified and
> +the device is immediately operational. A master driver instance can access the
> +number of slaves registered at any time by reading the configuration space of
> +the device.
> +
> +During the initialization phase the device connects also to the communication
> +channel. It has to be noted that the behavior of the device is
> +independent from the communication channel used, that is a detail of each
> +specific implementation of the SDM device.

How are SDM devices identified?  For example, if two SDM devices are
available, how does the driver know which one serves a particular
function?

> +
> +\subsection{Device Operation}\label{sec:Device Types / SDM Device / Device
> +peration}
> +
> +The SDM device handles signals coming from the two following sources:
> +
> +\begin{enumerate}
> +\item The local processor to which the device is attached to.
> +\item The communication channel connecting to other slaves/master.
> +\end{enumerate}
> +
> +The first case is verified when the processor attached to the SDM device wants
> +to send a signal to a second SDM device instance.
> +The second case is instead when an SDM device instance receives a signal from
> +another SDM device, to be forwarded to the local processor.
> +It is important to note that due to the master/slave behavior, slaves cannot
> +signal among themselves but only with the master SDM instance.
> +
> +In both cases, before sending over the communication channel, the signal is
> +packed in the \texttt{SDMSignalData} data structure.
> +
> +\begin{lstlisting}
> +enum sdm_signal_type {
> +    SDM_IRQ,
> +    SDM_BOOT,
> +};
> +
> +struct SDMSignalData {
> +    uint32_t type;
> +    uint32_t slave;
> +    uint32_t payload[2];
> +};
> +\end{lstlisting}
> +
> +The \texttt{type} field indicates the type of signal to be sent to the
> +destination SDM. The current implementation supports two signal types.
> +The \texttt{SDM_IRQ} signal is used to send an inter processor interrupt, while
> +the \texttt{SDM_BOOT} signal is sent to trigger the boot of the destination
> +processor. The boot signal is meant to be used in an AMP like scenario where a
> +master processor boots one or more slave processors (e.g., via remoteproc).
> +The \texttt{slave} field contains the id of the SDM instance destination of the
> +signal. Id 0 is reserved for the master, from 1 onwards for the slaves.
> +This means that the \texttt{slave} field will always contain 0 when the source
> +of the signal is a slave SDM instance, while the actual id of the slave in case
> +of a master.
> +The \texttt{payload} is used to pass extra accompanying information with the
> +signal.
> +The semantics of the payload field depends on the signal itself. In case of
> +\texttt{SDM_IRQ} signal, the payload field is ignored since interrupts do not
> +need any extra information to be handled. In the case of \texttt{SDM_BOOT}
> +signal the payload contains the boot address of the slave processor, to be used
> +at the destination to initialize the program counter register before the actual
> +boot process is started.
> +
> +
> +The \texttt{SDMSignalData} structure is first filled by the source SDM kernel
> +virtio driver and sent over the gh_vq.
> -- 
> 1.9.1
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Christian Pinto]

Hello Stefan,

thanks for your comments.

On Thu, Jul 14, 2016 at 2:17 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Tue, Jun 28, 2016 at 01:03:44PM +0200, Christian Pinto wrote:
> > This patch adds the specification of the Signal Dristribution Module
> virtio
> > device to the current virtio specification document.
> >
> > Signed-off-by: Christian Pinto <c.pinto@virtualopensystems.com>
> > Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
> >
> > ---
> > v1 -> v2:
> > - Fixed some typos
> > - Removed dependencies from QEMU
> > - Added explanation on how SDM can be used in AMP systems
> > - Explained semantics of payload field in SDMSignalData struct
> > ---
> >  content.tex    |   2 +
> >  virtio-sdm.tex | 128
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 130 insertions(+)
> >  create mode 100644 virtio-sdm.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 3317916..7fcf779 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5643,6 +5643,8 @@ descriptor for the \field{sense_len},
> \field{residual},
> >  \field{status_qualifier}, \field{status}, \field{response} and
> >  \field{sense} fields.
> >
> > +\input{virtio-sdm.tex}
> > +
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> >  Currently there are three device-independent feature bits defined:
> > diff --git a/virtio-sdm.tex b/virtio-sdm.tex
> > new file mode 100644
> > index 0000000..76b0a7b
> > --- /dev/null
> > +++ b/virtio-sdm.tex
> > @@ -0,0 +1,128 @@
> > +\section{Signal Distribution Module}\label{sec:Device Types / SDM
> Device}
> > +
> > +The virtio Signal Distribution Module is meant to enable Inter
> Processor signal
> > +exchange.
> > +An example are Inter Processor Interrupts used in AMP systems, for the
> > +processors involved to notify the presence of new data in the
> communication
> > +queues.
> > +In AMP systems signals are used for various purposes, an example are
> remoteproc
> > +or RPMSG. In the former signals are used by a master processor to
> trigger
> > +the boot of a slave processor. While the latter, RPMSG, uses virtio
> queues as a
> > +message exchange medium between processors. In this case the SDM can be
> used to
> > +generate the interrupt associated to the kick of a virtio queue.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / SDM Device / Device ID}
> > +
> > +21
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / SDM Device /
> Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] hg_vq
> > +\item[1] gh_vq
> > +\end{description}
> > +
> > +Queue 0 is used for device-to-driver communication (i.e., notification
> of a
> > +signal), while queue 1 for driver-to-device communication.
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / SDM Device / Feature
> bits}
> > +
> > +None.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / SDM
> Device /
> > +Device configuration layout}
> > +
> > +The device configuration is composed by three fields: \texttt{master},
> > +\texttt{max_slaves} and \texttt{current_slaves}.
> > +
> > +\begin{lstlisting}
> > +struct virtio_sdm_config {
> > +     u8    master;
>
> The next field has 16-bit alignment so there is a hole in this struct.
> Three options:
>
> 1. Add an explicit u8 padding field.
> 2. Change master's type from u8 to u16.
> 3. Move master to the end of the struct.
>

This is true, the master field would create a hole for u16 alignment. I
think that the third of the proposed options is more than fine, I will move
the master field at the end of the structure in the next spin.


>
> > +     u16   max_slaves;
> > +     u16   current_slaves;
> > +};
> > +\end{lstlisting}
> > +
> > +The SDM device can be instantiated either as a master or as a slave.
> The master
> > +slave behavior is meant on purpose to reflect the AMP like type of
> communication
> > +where a master processor controls one or more slave processors.
> > +A master SDM instance can send a signal to any of the slaves instances,
> > +while slaves can only signal the master.
> > +
> > +The \texttt{master} field of the configuration space is meant to be
> read by the
> > +driver to figure out whether a specific instance is a master or a slave.
> > +The \texttt{max_slaves} field contains the maximum number of slaves
> supported by
> > +the SDM device. A notification change is sent to the driver each time
> the value
> > +of \texttt{max_slaves} is changed from the device side.
> > +Finally, the \texttt{current_slaves} field contains the actual number
> of slaves
> > +registered to the master SDM. This field is a read only field.
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / SDM Device /
> > +evice Initialization}
> > +
> > +During initialization the \texttt{hg_vq} and \texttt{gh_vq} are
> identified and
> > +the device is immediately operational. A master driver instance can
> access the
> > +number of slaves registered at any time by reading the configuration
> space of
> > +the device.
> > +
> > +During the initialization phase the device connects also to the
> communication
> > +channel. It has to be noted that the behavior of the device is
> > +independent from the communication channel used, that is a detail of
> each
> > +specific implementation of the SDM device.
>
> How are SDM devices identified?  For example, if two SDM devices are
> available, how does the driver know which one serves a particular
> function?
>

The master-slave role is supposed to be enough to identify the device. If
as an example we consider an AMP system, the master core will only see one
SDM master device, while the slave processor will only see the slave SDM
instance. Then it is up to the implementation of the drivers to define the
signals served, while the SDM hardware is only in charge of forwarding such
signals. We do not foresee the need to have one SDM instance for each
signal type.


Thanks,

Christian


>
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / SDM Device /
> Device
> > +peration}
> > +
> > +The SDM device handles signals coming from the two following sources:
> > +
> > +\begin{enumerate}
> > +\item The local processor to which the device is attached to.
> > +\item The communication channel connecting to other slaves/master.
> > +\end{enumerate}
> > +
> > +The first case is verified when the processor attached to the SDM
> device wants
> > +to send a signal to a second SDM device instance.
> > +The second case is instead when an SDM device instance receives a
> signal from
> > +another SDM device, to be forwarded to the local processor.
> > +It is important to note that due to the master/slave behavior, slaves
> cannot
> > +signal among themselves but only with the master SDM instance.
> > +
> > +In both cases, before sending over the communication channel, the
> signal is
> > +packed in the \texttt{SDMSignalData} data structure.
> > +
> > +\begin{lstlisting}
> > +enum sdm_signal_type {
> > +    SDM_IRQ,
> > +    SDM_BOOT,
> > +};
> > +
> > +struct SDMSignalData {
> > +    uint32_t type;
> > +    uint32_t slave;
> > +    uint32_t payload[2];
> > +};
> > +\end{lstlisting}
> > +
> > +The \texttt{type} field indicates the type of signal to be sent to the
> > +destination SDM. The current implementation supports two signal types.
> > +The \texttt{SDM_IRQ} signal is used to send an inter processor
> interrupt, while
> > +the \texttt{SDM_BOOT} signal is sent to trigger the boot of the
> destination
> > +processor. The boot signal is meant to be used in an AMP like scenario
> where a
> > +master processor boots one or more slave processors (e.g., via
> remoteproc).
> > +The \texttt{slave} field contains the id of the SDM instance
> destination of the
> > +signal. Id 0 is reserved for the master, from 1 onwards for the slaves.
> > +This means that the \texttt{slave} field will always contain 0 when the
> source
> > +of the signal is a slave SDM instance, while the actual id of the slave
> in case
> > +of a master.
> > +The \texttt{payload} is used to pass extra accompanying information
> with the
> > +signal.
> > +The semantics of the payload field depends on the signal itself. In
> case of
> > +\texttt{SDM_IRQ} signal, the payload field is ignored since interrupts
> do not
> > +need any extra information to be handled. In the case of
> \texttt{SDM_BOOT}
> > +signal the payload contains the boot address of the slave processor, to
> be used
> > +at the destination to initialize the program counter register before
> the actual
> > +boot process is started.
> > +
> > +
> > +The \texttt{SDMSignalData} structure is first filled by the source SDM
> kernel
> > +virtio driver and sent over the gh_vq.
> > --
> > 1.9.1
> >
> >
>

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

On Tue, Jul 19, 2016 at 09:47:13AM +0200, Christian Pinto wrote:
> > > +During the initialization phase the device connects also to the
> > communication
> > > +channel. It has to be noted that the behavior of the device is
> > > +independent from the communication channel used, that is a detail of
> > each
> > > +specific implementation of the SDM device.
> >
> > How are SDM devices identified?  For example, if two SDM devices are
> > available, how does the driver know which one serves a particular
> > function?
> >
> 
> The master-slave role is supposed to be enough to identify the device. If
> as an example we consider an AMP system, the master core will only see one
> SDM master device, while the slave processor will only see the slave SDM
> instance. Then it is up to the implementation of the drivers to define the
> signals served, while the SDM hardware is only in charge of forwarding such
> signals. We do not foresee the need to have one SDM instance for each
> signal type.

The laissez-faire approach of allowing the implementation to define
signals breaks in an environment where there can be multiple versions of
the SDM hardware.

Virtio feature bits cannot be used to define signal-related
functionality because it's implementation-defined.  For example, there
is no way to express "CUSTOM_SIGNAL_2 is supported".

In a guest OS image that can run on two different types of AMP systems,
how do you distinguish between the set of signals to use?

I guess we can say that the driver has some external knowledge (e.g. the
board/chipset type) that allows it to know the meaning of signals and
which ones are available?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Christian Pinto]

Hello Stefan,

On Tue, Jul 19, 2016 at 10:40 AM, Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> On Tue, Jul 19, 2016 at 09:47:13AM +0200, Christian Pinto wrote:
> > > > +During the initialization phase the device connects also to the
> > > communication
> > > > +channel. It has to be noted that the behavior of the device is
> > > > +independent from the communication channel used, that is a detail of
> > > each
> > > > +specific implementation of the SDM device.
> > >
> > > How are SDM devices identified?  For example, if two SDM devices are
> > > available, how does the driver know which one serves a particular
> > > function?
> > >
> >
> > The master-slave role is supposed to be enough to identify the device. If
> > as an example we consider an AMP system, the master core will only see
> one
> > SDM master device, while the slave processor will only see the slave SDM
> > instance. Then it is up to the implementation of the drivers to define
> the
> > signals served, while the SDM hardware is only in charge of forwarding
> such
> > signals. We do not foresee the need to have one SDM instance for each
> > signal type.
>
> The laissez-faire approach of allowing the implementation to define
> signals breaks in an environment where there can be multiple versions of
> the SDM hardware.
>
> Virtio feature bits cannot be used to define signal-related
> functionality because it's implementation-defined.  For example, there
> is no way to express "CUSTOM_SIGNAL_2 is supported".
>
> In a guest OS image that can run on two different types of AMP systems,
> how do you distinguish between the set of signals to use?
>
> I guess we can say that the driver has some external knowledge (e.g. the
> board/chipset type) that allows it to know the meaning of signals and
> which ones are available?
>

Here I see two possibilities either what you propose, to demand on an
external
knowledge of the driver on the implementation of the SDM device for the
specific board/chipset. Or alternatively we could think to embed the set of
signals
supported by the implementation of the device in the configuration space.
We could univocally define signals in the specification of the device,
statically assigning each signal to an ID.
At devices initialization the driver reads the configuration and retrieves
the set of
signals supported. It is then up-to the user-level software to know the
semantic of each
signal (e.g., the meaning of the payload), that makes sense in my opinion.
We could also envision that at connection time between master and slave
there is
an handshake phase where the slave notifies the master with the set of
signals
it supports, and a slave can get rejected in case of mismatch.

Does this sound in line with the virtio philosophy?

Finally, the idea of the SDM is that in each deployment there is only one
master
and multiple slaves.



Thanks,

Christian




> Stefan
>

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3296 bytes --]

On Fri, Jul 22, 2016 at 06:18:25PM +0200, Christian Pinto wrote:
> Hello Stefan,
> 
> On Tue, Jul 19, 2016 at 10:40 AM, Stefan Hajnoczi <stefanha@redhat.com>
> wrote:
> 
> > On Tue, Jul 19, 2016 at 09:47:13AM +0200, Christian Pinto wrote:
> > > > > +During the initialization phase the device connects also to the
> > > > communication
> > > > > +channel. It has to be noted that the behavior of the device is
> > > > > +independent from the communication channel used, that is a detail of
> > > > each
> > > > > +specific implementation of the SDM device.
> > > >
> > > > How are SDM devices identified?  For example, if two SDM devices are
> > > > available, how does the driver know which one serves a particular
> > > > function?
> > > >
> > >
> > > The master-slave role is supposed to be enough to identify the device. If
> > > as an example we consider an AMP system, the master core will only see
> > one
> > > SDM master device, while the slave processor will only see the slave SDM
> > > instance. Then it is up to the implementation of the drivers to define
> > the
> > > signals served, while the SDM hardware is only in charge of forwarding
> > such
> > > signals. We do not foresee the need to have one SDM instance for each
> > > signal type.
> >
> > The laissez-faire approach of allowing the implementation to define
> > signals breaks in an environment where there can be multiple versions of
> > the SDM hardware.
> >
> > Virtio feature bits cannot be used to define signal-related
> > functionality because it's implementation-defined.  For example, there
> > is no way to express "CUSTOM_SIGNAL_2 is supported".
> >
> > In a guest OS image that can run on two different types of AMP systems,
> > how do you distinguish between the set of signals to use?
> >
> > I guess we can say that the driver has some external knowledge (e.g. the
> > board/chipset type) that allows it to know the meaning of signals and
> > which ones are available?
> >
> 
> Here I see two possibilities either what you propose, to demand on an
> external
> knowledge of the driver on the implementation of the SDM device for the
> specific board/chipset. Or alternatively we could think to embed the set of
> signals
> supported by the implementation of the device in the configuration space.
> We could univocally define signals in the specification of the device,
> statically assigning each signal to an ID.
> At devices initialization the driver reads the configuration and retrieves
> the set of
> signals supported. It is then up-to the user-level software to know the
> semantic of each
> signal (e.g., the meaning of the payload), that makes sense in my opinion.
> We could also envision that at connection time between master and slave
> there is
> an handshake phase where the slave notifies the master with the set of
> signals
> it supports, and a slave can get rejected in case of mismatch.
> 
> Does this sound in line with the virtio philosophy?
> 
> Finally, the idea of the SDM is that in each deployment there is only one
> master
> and multiple slaves.

I think the most satisfying approach from the VIRTIO spec perspective is
to include the signals in the spec.  That way feature bits can be used.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Christian Pinto]

Hello Stefan,

thanks for the feedback.I will rework the specification and send a v3 patch
series.


Christian

On Thu, Aug 4, 2016 at 10:46 AM, Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> On Fri, Jul 22, 2016 at 06:18:25PM +0200, Christian Pinto wrote:
> > Hello Stefan,
> >
> > On Tue, Jul 19, 2016 at 10:40 AM, Stefan Hajnoczi <stefanha@redhat.com>
> > wrote:
> >
> > > On Tue, Jul 19, 2016 at 09:47:13AM +0200, Christian Pinto wrote:
> > > > > > +During the initialization phase the device connects also to the
> > > > > communication
> > > > > > +channel. It has to be noted that the behavior of the device is
> > > > > > +independent from the communication channel used, that is a
> detail of
> > > > > each
> > > > > > +specific implementation of the SDM device.
> > > > >
> > > > > How are SDM devices identified?  For example, if two SDM devices
> are
> > > > > available, how does the driver know which one serves a particular
> > > > > function?
> > > > >
> > > >
> > > > The master-slave role is supposed to be enough to identify the
> device. If
> > > > as an example we consider an AMP system, the master core will only
> see
> > > one
> > > > SDM master device, while the slave processor will only see the slave
> SDM
> > > > instance. Then it is up to the implementation of the drivers to
> define
> > > the
> > > > signals served, while the SDM hardware is only in charge of
> forwarding
> > > such
> > > > signals. We do not foresee the need to have one SDM instance for each
> > > > signal type.
> > >
> > > The laissez-faire approach of allowing the implementation to define
> > > signals breaks in an environment where there can be multiple versions
> of
> > > the SDM hardware.
> > >
> > > Virtio feature bits cannot be used to define signal-related
> > > functionality because it's implementation-defined.  For example, there
> > > is no way to express "CUSTOM_SIGNAL_2 is supported".
> > >
> > > In a guest OS image that can run on two different types of AMP systems,
> > > how do you distinguish between the set of signals to use?
> > >
> > > I guess we can say that the driver has some external knowledge (e.g.
> the
> > > board/chipset type) that allows it to know the meaning of signals and
> > > which ones are available?
> > >
> >
> > Here I see two possibilities either what you propose, to demand on an
> > external
> > knowledge of the driver on the implementation of the SDM device for the
> > specific board/chipset. Or alternatively we could think to embed the set
> of
> > signals
> > supported by the implementation of the device in the configuration space.
> > We could univocally define signals in the specification of the device,
> > statically assigning each signal to an ID.
> > At devices initialization the driver reads the configuration and
> retrieves
> > the set of
> > signals supported. It is then up-to the user-level software to know the
> > semantic of each
> > signal (e.g., the meaning of the payload), that makes sense in my
> opinion.
> > We could also envision that at connection time between master and slave
> > there is
> > an handshake phase where the slave notifies the master with the set of
> > signals
> > it supports, and a slave can get rejected in case of mismatch.
> >
> > Does this sound in line with the virtio philosophy?
> >
> > Finally, the idea of the SDM is that in each deployment there is only one
> > master
> > and multiple slaves.
>
> I think the most satisfying approach from the VIRTIO spec perspective is
> to include the signals in the spec.  That way feature bits can be used.
>
> Stefan
>

Re: [Qemu-devel] [virtio-dev][RFC v2 2/2] virtio-sdm: new device specification

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Tue, Jun 28, 2016 at 01:03:44PM +0200, Christian Pinto wrote:
> +\begin{lstlisting}
> +struct virtio_sdm_config {
> +     u8    master;

The next field has 16-bit alignment so there is a hole in this struct.
Three options:

1. Add an explicit u8 padding field.
2. Change master's type from u8 to u16.
3. Move master to the end of the struct.

> +\subsection{Device Initialization}\label{sec:Device Types / SDM Device /
> +evice Initialization}
> +
> +During initialization the \texttt{hg_vq} and \texttt{gh_vq} are identified and
> +the device is immediately operational. A master driver instance can access the
> +number of slaves registered at any time by reading the configuration space of
> +the device.
> +
> +During the initialization phase the device connects also to the communication
> +channel. It has to be noted that the behavior of the device is
> +independent from the communication channel used, that is a detail of each
> +specific implementation of the SDM device.

How are SDM devices identified?  For example, if two SDM devices are
available, how does the driver know which one serves a particular
function?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]