[PATCH v3 1/3] docs: add documentation for Argo as a feature

[PATCH v3 1/3] docs: add documentation for Argo as a feature

[Christopher Clark]

Adds approachable documentation describing system components and
introduces the support statement for feature status.

Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
---
 docs/features/argo-feature.pandoc | 99 +++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 docs/features/argo-feature.pandoc

diff --git a/docs/features/argo-feature.pandoc b/docs/features/argo-feature.pandoc
new file mode 100644
index 0000000000..b6a10be78a
--- /dev/null
+++ b/docs/features/argo-feature.pandoc
@@ -0,0 +1,99 @@
+% Argo interdomain communication
+% Revision 1
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+         Status: **Tech Preview**
+
+  Architectures: x86, ARM
+
+   Component(s): Hypervisor, guest
+---------------- ----------------------------------------------------
+
+# Overview
+
+Argo is a lightweight data transport for communication between virtual
+machines, with a simple hypervisor interface that is accessible for
+building embedded systems and designed to work with familiar primitives
+within guest OS kernels. It supports policy control over communication
+and isolation between VMs.
+
+# User details
+
+Argo is present in Xen, included since Xen 4.12. A Linux device driver
+and userspace library are available and Argo is regularly tested in the
+Xen Continuous Integration system.
+
+To configure a Xen system to enable Argo:
+
+* ensure that Argo is enabled in the hypervisor with KConfig option
+
+* enable Argo on the Xen hypervisor command line
+
+* load the Argo guest kernel device driver
+
+* ensure that the Argo guest libraries are installed
+
+The guest userspace libraries support software designed for Argo
+interfaces and also enables software designed for networks to
+communicate between VMs by Argo.  This allows platform software to be
+plumbed easily between virtual machines, without requiring networking
+and with system policy controls over this communication.
+
+# Technical details
+
+## Argo components
+
+* Xen: Argo in the hypervisor provides communication between virtual
+  machines.
+
+* Guest kernel: driver provides interfaces for data transport for use
+  within the kernel, and implements familiar abstractions for
+  networking software.
+
+* Guest libraries: provide application-level support for interdomain
+  communication.
+
+## Argo implementation within Xen
+
+See the public Xen headers for the primary interface documentation.
+
+# Limitations
+
+Argo has been developed and tested on x86 and ARM architectures.
+
+# Testing
+
+The Argo test developed for the Xen Test Framework provides coverage of
+the hypercall operations.
+
+The Xen Project CI provides system test coverage of an Argo-enabled Xen
+system with Linux.
+
+# Areas for improvement
+
+## Argo and VirtIO
+
+References to design documentation for the development of an Argo
+transport for VirtIO are available via:
+https://wiki.xenproject.org/wiki/Virtio_On_Xen
+
+# Known issues
+
+* For development: sysctl/domctls for toolstack to control per-VM access
+  to Argo
+
+# References
+
+See the ARGO section of the Xen MAINTAINERS document for web reference.
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version   Notes
+---------- -------- --------- ------------------------------------------
+2025-05-28 1        Xen 4.12+ Feature included in Xen 4.12.
+---------- -------- --------- ------------------------------------------
-- 
2.34.1

[PATCH v3 2/3] MAINTAINERS: add link and keyword statements for Argo section

[Christopher Clark]

Add link to Argo documentation and to ensure that Argo maintainers and
reviewers are included in developments involving Argo, add keyword
expressions for Argo.

Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
---
 MAINTAINERS | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index c11b82eca9..697f383505 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -223,9 +223,12 @@ F:	tools/libacpi/
 ARGO
 M:	Christopher Clark <christopher.w.clark@gmail.com>
 S:	Maintained
+W:	https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
 F:	xen/include/public/argo.h
 F:	xen/include/xen/argo.h
 F:	xen/common/argo.c
+K:	\b(argo|Argo|ARGO)\b
+K:	(?i)argo[_-].*
 
 ARINC653 SCHEDULER
 M:	Nathan Studer <nathan.studer@dornerworks.com>
-- 
2.34.1

Re: [PATCH v3 2/3] MAINTAINERS: add link and keyword statements for Argo section

[Roger Pau Monné]

On Wed, May 28, 2025 at 10:10:39PM +0100, Christopher Clark wrote:
> Add link to Argo documentation and to ensure that Argo maintainers and
> reviewers are included in developments involving Argo, add keyword
> expressions for Argo.
> 
> Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
> ---
>  MAINTAINERS | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c11b82eca9..697f383505 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -223,9 +223,12 @@ F:	tools/libacpi/
>  ARGO
>  M:	Christopher Clark <christopher.w.clark@gmail.com>
>  S:	Maintained
> +W:	https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen

Could this be moved somewhere else?

The wiki is in a very bad state, with an increasing amount of stale
content.  It would be better if documentation linked here is in a more
controlled (ie: reviewed) place rather than a wiki that can be
modified by an unknown number of users.

Thanks, Roger.

[PATCH v3 3/3] MAINTAINERS: add a reviewer for Argo

[Christopher Clark]

Adding Daniel P. Smith as reviewer for the Argo subsystem.

Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 697f383505..6b129704fc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -222,6 +222,7 @@ F:	tools/libacpi/
 
 ARGO
 M:	Christopher Clark <christopher.w.clark@gmail.com>
+R:	Daniel P. Smith <dpsmith@apertussolutions.com>
 S:	Maintained
 W:	https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
 F:	xen/include/public/argo.h
-- 
2.34.1

Re: [PATCH v3 3/3] MAINTAINERS: add a reviewer for Argo

[Daniel P. Smith]

On 5/28/25 17:10, Christopher Clark wrote:
> Adding Daniel P. Smith as reviewer for the Argo subsystem.
> 
> Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
>   MAINTAINERS | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 697f383505..6b129704fc 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -222,6 +222,7 @@ F:	tools/libacpi/
>   
>   ARGO
>   M:	Christopher Clark <christopher.w.clark@gmail.com>
> +R:	Daniel P. Smith <dpsmith@apertussolutions.com>
>   S:	Maintained
>   W:	https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
>   F:	xen/include/public/argo.h

Acked-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Re: [PATCH v3 1/3] docs: add documentation for Argo as a feature

[Teddy Astie]

Hello Christopher,

Le 28/05/2025 à 23:13, Christopher Clark a écrit :
> Adds approachable documentation describing system components and
> introduces the support statement for feature status.
>
> Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
> ---
>   docs/features/argo-feature.pandoc | 99 +++++++++++++++++++++++++++++++
>   1 file changed, 99 insertions(+)
>   create mode 100644 docs/features/argo-feature.pandoc
>
> diff --git a/docs/features/argo-feature.pandoc b/docs/features/argo-feature.pandoc
> new file mode 100644
> index 0000000000..b6a10be78a
> --- /dev/null
> +++ b/docs/features/argo-feature.pandoc
> @@ -0,0 +1,99 @@
> +% Argo interdomain communication
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> +         Status: **Tech Preview**
> +
> +  Architectures: x86, ARM
> +
> +   Component(s): Hypervisor, guest
> +---------------- ----------------------------------------------------
> +
> +# Overview
> +
> +Argo is a lightweight data transport for communication between virtual
> +machines, with a simple hypervisor interface that is accessible for
> +building embedded systems and designed to work with familiar primitives
> +within guest OS kernels. It supports policy control over communication
> +and isolation between VMs.
> +
> +# User details
> +
> +Argo is present in Xen, included since Xen 4.12. A Linux device driver
> +and userspace library are available and Argo is regularly tested in the
> +Xen Continuous Integration system.
> +

Not really related to the documentation itself, but it would be
preferable to have the Linux driver upstreamed such as this sentence
sounds more as "it is supported" rather than "it's possible to make it
work".

It would also make Argo easier to integrate in Xen-based projects.

> +To configure a Xen system to enable Argo:
> +
> +* ensure that Argo is enabled in the hypervisor with KConfig option
> +
> +* enable Argo on the Xen hypervisor command line
> +
> +* load the Argo guest kernel device driver
> +
> +* ensure that the Argo guest libraries are installed
> +
> +The guest userspace libraries support software designed for Argo
> +interfaces and also enables software designed for networks to
> +communicate between VMs by Argo.  This allows platform software to be
> +plumbed easily between virtual machines, without requiring networking
> +and with system policy controls over this communication.
> +
> +# Technical details
> +
> +## Argo components
> +
> +* Xen: Argo in the hypervisor provides communication between virtual
> +  machines.
> +
> +* Guest kernel: driver provides interfaces for data transport for use
> +  within the kernel, and implements familiar abstractions for
> +  networking software.
> +
> +* Guest libraries: provide application-level support for interdomain
> +  communication.
> +

That sounds a bit confusing to me, the guest kernel provides familiar
abstractions for networking software (I hear through that something like
sockets), yet we still have guest libraries for application-level support.

I think the guest libraries provide a shim for some of the argo-specific
aspects (like ioctls and sockaddr_argo related bits); while the
interface is more unix-oriented with regular sockets operations
(sendmsg, ...).

> +## Argo implementation within Xen
> +
> +See the public Xen headers for the primary interface documentation.
> +

There is the design document in docs/designs/argo.pandoc that explains
the inner design inside Xen. I think the public headers are more for
guest consumption.

As this document also targets guest developers, it would be great to
have some guidance (e.g a docs document) on how a guest should implement
argo support.

> +# Limitations
> +
> +Argo has been developed and tested on x86 and ARM architectures.
> +
> +# Testing
> +
> +The Argo test developed for the Xen Test Framework provides coverage of
> +the hypercall operations.
> +
> +The Xen Project CI provides system test coverage of an Argo-enabled Xen
> +system with Linux.
> +
> +# Areas for improvement
> +
> +## Argo and VirtIO
> +
> +References to design documentation for the development of an Argo
> +transport for VirtIO are available via:
> +https://wiki.xenproject.org/wiki/Virtio_On_Xen
> +

Are there news regarding this ?

There is work done on virtio-msg [1], which looks fairly similar to
virtio-argo (or at least, virtio-msg could work with Argo in a similar
fashion to what's described on the virtio-argo design).

[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview

> +# Known issues
> +
> +* For development: sysctl/domctls for toolstack to control per-VM access
> +  to Argo
> +

Is it regarding disabling the argo on a per-guest basis, or regarding if
a specific VM can communicate with another VM ? i.e can the toolstack
decide to prevent 2 guest from communicating ?

IIRC, in Argo, a guest on his own can decide who can communicate with
him using separate registered rings. But I am not sure if there is more
on that regard.

> +# References
> +
> +See the ARGO section of the Xen MAINTAINERS document for web reference.
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date       Revision Version   Notes
> +---------- -------- --------- ------------------------------------------
> +2025-05-28 1        Xen 4.12+ Feature included in Xen 4.12.
> +---------- -------- --------- ------------------------------------------

Teddy


Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v3 1/3] docs: add documentation for Argo as a feature

[Jason Andryuk]

On 2025-05-29 18:40, Teddy Astie wrote:
> Hello Christopher,
> 
> Le 28/05/2025 à 23:13, Christopher Clark a écrit :

>> +## Argo and VirtIO
>> +
>> +References to design documentation for the development of an Argo
>> +transport for VirtIO are available via:
>> +https://wiki.xenproject.org/wiki/Virtio_On_Xen
>> +
> 
> Are there news regarding this ?
> 
> There is work done on virtio-msg [1], which looks fairly similar to
> virtio-argo (or at least, virtio-msg could work with Argo in a similar
> fashion to what's described on the virtio-argo design).
> 
> [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview

I think this should be dropped.  We don't need a link to a design 
document without an implementation.  You can add it once you've 
upstreamed the implementation.

>> +# Known issues
>> +
>> +* For development: sysctl/domctls for toolstack to control per-VM access
>> +  to Argo
>> +
> 
> Is it regarding disabling the argo on a per-guest basis, or regarding if
> a specific VM can communicate with another VM ? i.e can the toolstack
> decide to prevent 2 guest from communicating ?
> 
> IIRC, in Argo, a guest on his own can decide who can communicate with
> him using separate registered rings. But I am not sure if there is more
> on that regard.

Yes, I think the existing text needs to be rephrased to be more explicit 
on the issue.  I can guess what it is, but I shouldn't have to.  I'd 
recommend stating the issue as it exists, and then optionally clearly 
state a proposed solution.

Regards,
Jason

Re: [PATCH v3 1/3] docs: add documentation for Argo as a feature

[Christopher Clark]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

On Wed, Jun 4, 2025 at 5:14 PM Jason Andryuk <jason.andryuk@amd.com> wrote:

> On 2025-05-29 18:40, Teddy Astie wrote:
> > Hello Christopher,
> >
> > Le 28/05/2025 à 23:13, Christopher Clark a écrit :
>
> >> +## Argo and VirtIO
> >> +
> >> +References to design documentation for the development of an Argo
> >> +transport for VirtIO are available via:
> >> +https://wiki.xenproject.org/wiki/Virtio_On_Xen
> >> +
> >
> > Are there news regarding this ?
> >
> > There is work done on virtio-msg [1], which looks fairly similar to
> > virtio-argo (or at least, virtio-msg could work with Argo in a similar
> > fashion to what's described on the virtio-argo design).
> >
> > [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview


I appreciate the interest - I don't have additional material on it at the
moment.


>
> I think this should be dropped.  We don't need a link to a design
> document without an implementation.  You can add it once you've
> upstreamed the implementation.
>

OK, I'll remove this section for the next version of the series.


>
> >> +# Known issues
> >> +
> >> +* For development: sysctl/domctls for toolstack to control per-VM
> access
> >> +  to Argo
> >> +
> >
> > Is it regarding disabling the argo on a per-guest basis, or regarding if
> > a specific VM can communicate with another VM ? i.e can the toolstack
> > decide to prevent 2 guest from communicating ?
> >
> > IIRC, in Argo, a guest on his own can decide who can communicate with
> > him using separate registered rings. But I am not sure if there is more
> > on that regard.
>

It's to do with enabling administrative controls for the toolstack to be
able to govern access to Argo by individual VMs on a per-VM basis. At the
moment there is a host boot option that turns it on or off for the system
and there are XSM/Flask policy controls that govern access to Argo by
individual VMs on a per-VM basis, but that is less accessible for a system
administrator to apply changes to VM access and less dynamic than some use
cases may require.


>
> Yes, I think the existing text needs to be rephrased to be more explicit
> on the issue.  I can guess what it is, but I shouldn't have to.  I'd
> recommend stating the issue as it exists, and then optionally clearly
> state a proposed solution.
>

Fair, thanks, will revise it.

Christopher

[-- Attachment #2: Type: text/html, Size: 3649 bytes --]

Re: [PATCH v3 1/3] docs: add documentation for Argo as a feature

[Christopher Clark]

[-- Attachment #1: Type: text/plain, Size: 3941 bytes --]

On Sun, Jun 8, 2025 at 7:37 PM Rich Persaud <persaur@gmail.com> wrote:

> On Jun 8, 2025, at 5:49 AM, Christopher Clark <
> christopher.w.clark@gmail.com> wrote:
>
> 
>
> On Wed, Jun 4, 2025 at 5:14 PM Jason Andryuk <jason.andryuk@amd.com>
> wrote:
>
>>
>> I think this should be dropped.  We don't need a link to a design
>> document without an implementation.  You can add it once you've
>> upstreamed the implementation.
>>
>
> OK, I'll remove this section for the next version of the series.
>
>
> What's the recommended location of Xen design documents, requirements for
> future improvements, roadmaps or pointers to related work in adjacent
> open-source communities? The Xen wiki is being deprecated.
>

Others CCd are likely better qualified than I to answer these (reasonable)
questions, and I am interested to learn from any further answers.

My preferred option is not to deprecate the wiki, or at least not without a
replacement for it. I have found the wiki to be useful for many years for
both publishing technical content and for access to helpful material not
available anywhere else, even if the contents are rougher than the formal
documentation. Its accessibility is an essential part of enabling that.

If the wiki is deprecated, the available alternatives appear to be:

* Pursuing formal acceptance of documents into one of the Xen git
repositories;
From my experience of doing so, less documentation will be produced if this
is the standard required and unique knowledge will be lost at an ongoing
cost to both the developer community and users of Xen. Review capacity is
already scarce and this will further deplete that.

* Publishing on the platforms of adjacent communities;
Documentation will be less discoverable and at the mercy of external tools
and processes, again at a cost to collaboration within the Xen community.

* Repurposing the Xen issue tracker
This would be a horrible bodge. Please no.

* Self-hosting - similar to publishing in an adjacent community, but worse
I would prefer not to have to do this and may not have time available to
do so. Not everyone is able to.

* Replace the wiki with another collaboratively-edited document hosting
platform
I am open to learning more about options for this if it is the recommended
direction.


> What's the recommended way for the Xen community to discover the existence
> of Argo documentation that is not hosted by the Xen community? If
> necessary, we can create a new semantic label or Xen docs directory tree,
> to host technical content that might otherwise be lost.
>
> Should we add a sentence to Xen's Argo documentation, to the effect of
> "Please refer to the Xen [wiki in archive.org, website, mailing list],
> external sites [A, B, C, D], or your preferred [search engine, LLM] for
> more technical documents on Argo design and implementation"?
>

The Virtio & Argo design documents, produced within an adjacent community
so they are hosted there, are relevant to Xen and ought to be enabled to be
discoverable (and if necessary hostable) via Xen community platforms.
They are also a single instance of a general class: there will be
developments of other Xen-related features within XCP-ng, Qubes and Linaro
projects - and other adjacent communities - that also warrant description
and linking from within Xen community documentation.

To Jason's review point: I am willing to accept leaving the VirtIO section
out of this specific Argo feature document but I do need an appropriate
alternative place for it - eg. the Argo design document (already within the
Xen source tree) could be extended to introduce it instead. The externally
referenced design documents are part of how such future
features get implemented, so referencing prior to implementation is helpful
and supportive of the continued development of the technology.

Christopher

[-- Attachment #2: Type: text/html, Size: 5401 bytes --]

Re: [PATCH v3 1/3] docs: add documentation for Argo as a feature

[Christopher Clark]

[-- Attachment #1: Type: text/plain, Size: 3678 bytes --]

On Thu, May 29, 2025 at 11:40 PM Teddy Astie <teddy.astie@vates.tech> wrote:

> Hello Christopher,
>

Hi Teddy, thanks for the review.


>
> Le 28/05/2025 à 23:13, Christopher Clark a écrit :
> > Adds approachable documentation describing system components and
> > introduces the support statement for feature status.
> ...
> > +# Overview
> > +
> > +Argo is a lightweight data transport for communication between virtual
> > +machines, with a simple hypervisor interface that is accessible for
> > +building embedded systems and designed to work with familiar primitives
> > +within guest OS kernels. It supports policy control over communication
> > +and isolation between VMs.
> > +
> > +# User details
> > +
> > +Argo is present in Xen, included since Xen 4.12. A Linux device driver
> > +and userspace library are available and Argo is regularly tested in the
> > +Xen Continuous Integration system.
> > +
>
> Not really related to the documentation itself, but it would be
> preferable to have the Linux driver upstreamed such as this sentence
> sounds more as "it is supported" rather than "it's possible to make it
> work".
>

I agree and I can work on a better description for the next posting.


>
> It would also make Argo easier to integrate in Xen-based projects.
>

Yes (for systems with Linux).


>
> > +To configure a Xen system to enable Argo:
> > +
> > +* ensure that Argo is enabled in the hypervisor with KConfig option
> > +
> > +* enable Argo on the Xen hypervisor command line
> > +
> > +* load the Argo guest kernel device driver
> > +
> > +* ensure that the Argo guest libraries are installed
> > +
> > +The guest userspace libraries support software designed for Argo
> > +interfaces and also enables software designed for networks to
> > +communicate between VMs by Argo.  This allows platform software to be
> > +plumbed easily between virtual machines, without requiring networking
> > +and with system policy controls over this communication.
> > +
> > +# Technical details
> > +
> > +## Argo components
> > +
> > +* Xen: Argo in the hypervisor provides communication between virtual
> > +  machines.
> > +
> > +* Guest kernel: driver provides interfaces for data transport for use
> > +  within the kernel, and implements familiar abstractions for
> > +  networking software.
> > +
> > +* Guest libraries: provide application-level support for interdomain
> > +  communication.
> > +
>
> That sounds a bit confusing to me, the guest kernel provides familiar
> abstractions for networking software (I hear through that something like
> sockets), yet we still have guest libraries for application-level support.
>
> I think the guest libraries provide a shim for some of the argo-specific
> aspects (like ioctls and sockaddr_argo related bits); while the
> interface is more unix-oriented with regular sockets operations
> (sendmsg, ...).
>

OK, I wasn't sure what level of detail would be wanted for this so it's
useful to know, thanks.


>
> > +## Argo implementation within Xen
> > +
> > +See the public Xen headers for the primary interface documentation.
> > +
>
> There is the design document in docs/designs/argo.pandoc that explains
> the inner design inside Xen. I think the public headers are more for
> guest consumption.
>

OK, I'll add reference to the design document.


>
> As this document also targets guest developers, it would be great to
> have some guidance (e.g a docs document) on how a guest should implement
> argo support.
>

Thanks, I appreciate the enthusiasm for it to be written!

best,

Christopher

[-- Attachment #2: Type: text/html, Size: 5173 bytes --]