Rust in QEMU roadmap

Rust in QEMU roadmap

[Paolo Bonzini]

[several people are in bcc.  You can follow the thread at
 https://lore.kernel.org/r/cc40943e-dec1-4890-a1d9-579350ce296f@pbonzini.local]

Based on the content presented in the community calls, here are some
ideas for future Rust in QEMU subprojects.

This is by no means exhaustive, for example QAPI and async (block devices)
are missing because I have not looked at them.

Table of contents
'''''''''''''''''

* Status in QEMU 9.2
* Build system
* Feature parity for pl011 device
* Avoiding undefined behavior
* Safe QOM class definitions
* QOM method calls
* Callbacks
* Tracing/logging
* Data structure interoperability
* Features expected from recent rust


Status in QEMU 9.2
''''''''''''''''''

QEMU when built with --enable-rust compiles on all supported
build platforms.  It passes CI and "make check-unit" runs tests for
rust/qemu-api.  "make check-qtests" covers the Rust pl011 device model
including migration.  However, Rust is not enabled and its presence is
not checked for by default.

Lots of unsafe code is needed to define classes, define callbakcs, and
invoke any core functionality.  That said, the current pl011 code provides
a good roadmap for what functionality Rust code needs more urgently.

Zhao Liu is working on a conversion to Rust of the HPET device.  It has the
same limitations, but it covers a slightly different set of functionality
(e.g. timers) and it provides a second device with which to validate
the abstractions that are being built.  However, including any device
beyond these two should (in my opinion) wait until FFI abstractions are
ready and usage of unsafe code is limited (especially after initialization).

For now we're looking at simple devices.  These do not do any DMA or
memory allocation; there is hardly any benefit from Rewriting In Rust.
However, they are small and self-contained, and let us learn to walk
before we try to run.

For anybody who wants to work on FFI, it is highly recommended to consult
the subsystem maintainers and to post *early and often*.  Also include
links to the mailing list threads in https://wiki.qemu.org/RustInQemu.


Build system
''''''''''''

Posted: https://lore.kernel.org/qemu-devel/20241108180139.117112-1-pbonzini@redhat.com/

This provides developers with easy access to clippy, rustfmt and rustdoc;
and in general Cargo is the easiest way to integrate tools other than rustc.
This series also makes it possible to configure warnings with the familiar
syntax provided by Cargo.toml.

Next steps:

* Access to clippy and rustfmt could be done by Meson, similar to its
  existing "ninja clang-tidy" and "ninja clang-format" targets.  This
  removes code from QEMU and more importantly avoids that every mixed
  Rust/C project reinvents the wheel.

* rustdoc tests should be integrated into "make check".  Running "cargo
  test --doc" has the limitation that the tests are not able to link with
  C code.  I have prototyped this, but mostly to teach myself how rustdoc
  tests are run and implement them in Meson.

Long term a separate rust/ directory probably will not exist, and it'd
be nice to avoid having a proliferation of src/ directories since QEMU
has hundreds of devices.  It will be interesting to see whether at that
point there will still be a need for Cargo, or whether all required
functionality will be included in Meson before we get there.

Summary:
* The series is mostly Python and Meson code so I am comfortable merging
  it in 10.0

* Long term, cargo-like functionality should be provided by Meson.  In
  the meanwhile it can be implemented by hand, but we should upgrade
  to Meson 1.7.0 if it comes out with native support for clippy and
  friends.


Feature parity for pl011 device
'''''''''''''''''''''''''''''''

Some recent pl011 commits are missing in the Rust version.  Philippe
volunteered to port them to teach himself Rust.

Philippe also has a series to implement flow control and better use
of the PL011 FIFO (IIUC).  Porting this to Rust would be hard right
now, so its Rust implementation blocked on having chardev bindings.

Also missing is logging and tracing, but this part should not be a
blocker for defaulting to --enable-rust.

Summary:

* Philippe's FIFO work might impact the plan of defaulting to
  --enable-rust as soon as 10.0.


Avoiding undefined behavior
'''''''''''''''''''''''''''

Part 1: https://lore.kernel.org/qemu-devel/20241122074756.282142-1-pbonzini@redhat.com/

Currently, the Rust in QEMU code is not respecting the invariant that
mutable references must be either unique or obtained from an UnsafeCell<>.
It's also in some cases creating references (either shared or mutable)
that point to invalid data, for example NULL references.  This was
a necessary evil to be able to have *some* Rust code in tree and
develop build system integration, but it obviously has to be fixed.

The first part is solved by annotating the code to use the proper
interior mutability types.  These include homegrown variants of Cell
and RefCell that support the "big QEMU lock", and therefore are
usable in a multi-threaded context.  In particular, QEMU's variant of
RefCell will also check that bql_unlock() is never called with outstanding
borrows.

The second part is solved by using Linux's pinned-init crate, which is
available for use in userspace on crates.io.  The pinned-init crate avoids
exposing partially-initialized objects to safe Rust code.  Right now
it only supports nightly Rust, but I've sent a pull request to fix this;
supporting Debian's rustc 1.63.0 of this pull request is not an issue.

Summary:

* Introduce cell types as needed.  BqlCell was posted, BqlRefCell is
  written but does not have a use yet.

* Using pinned-init is planned but will come later.


Safe QOM class definitions
''''''''''''''''''''''''''

Part 1: https://lore.kernel.org/qemu-devel/20241125080507.115450-1-pbonzini@redhat.com/

QOM's object-oriented, inheritance heavy design is a challenge for Rust,
but fortunately we can learn from the experience of glib-rs developers.
There are two orthogonal parts to making QOM accessible from safe,
idiomatic Rust code: namely, defining QOM classes and calling QOM methods.

With respect to defining QOM classes, it is expected that Rust code
will mostly define *leaf* classes, which simplifies the task a bit.
Or at least, that any superclasses defined by Rust code will only be
subclassed in Rust as well.

A first attempt at this was posted by Manos, but in the review I
commented that his procedural macro approach was limited to subclasses
of DeviceState.  It would be much better to have something more easily
adaptable to other classes.

For now I've posted a series that removes code duplication but is still
very much focused on four elements of DeviceClass (realize, legacy reset,
properties, vmstate).  This part is simple.

A second step is to encode the QOM hierarchy in the type system.  A new
ClassInitImpl<ClassType> trait initializes the methods of a specific
class, for example DeviceClass, and then "recurses" to the superclass::

  trait ClassInitImpl<ClassType> {
    fn class_init(class: &mut ClassType);
  }

  impl ClassInitImpl<DeviceClass> for T where T: DeviceImpl {
    fn class_init(class: &mut DeviceClass) {
      dc.realize = rust_realize_fn::<T>;
      ...
      <Self as ClassInitImpl::<ObjectClass>>::class_init(class.parent_class);
    }
  }

  impl ClassInitImpl<ObjectClass> for T where T: ObjectImpl { ... }

This generalizes the class_init implementation that is in the already-posted
part.  The same pattern of recursion can be used to support QOM interfaces
as well.

Summary:

* Reviews are welcome on the first part.  Hopefully when I post the second
  (mid December?) the first will have been reviewed at least in part.

* Procedural macros still remain a valuable tool.  Using them for qdev
  properties should be easy to extract from Manos's submission.  In the
  review I also proposed building a small "toolkit" that can be reused
  by more procedural macros in the future.


QOM method calls
''''''''''''''''

This is a separate branch of QOM work that I had started last July, by looking
at glib-rs's implementation of GObject.  The idea is to be able to write code
that looks like native Rust but calls QOM methods.  For example::

   let dev = PL011State::new();
   dev.prop_set_chr("chardev", chr)
   dev.realize().unwrap_fatal();		// operate on Error**
   dev.mmio_map(0, addr);
   dev.connect_irq(0, irq);
   Object::into_raw(dev)

Note how methods can come from multiple classes (DeviceState, SysBusDevice,
even Object in the case of "new").  This makes it harder to implement devices
as something like SysBusDevice<PL011Properties, PL011Registers>; you'd still
have to surface methods in superclasses of SysBusDevice, for example
"prop_set_chr" or "realize" in the above example.

glib-rs's implementation is based on an IsA trait that has to be written
by hand (e.g. PL011State implements IsA<Object>, IsA<DeviceState>,
IsA<SysBusDevice>).  Rust's automatic dereferencing behavior can then be
extended to all superclasses; wrappers for methods are implemented in a
trait::

  trait DeviceMethods {
    fn realize(&self) -> Result<(), qemu_api::Error> {
      ...
      device_realize((self as *const _).cast_mut().cast::<DeviceState>, ...);
      ...
    }
  }

and the trait is applied to everything that dereferences to a subclass of
DeviceState::

  impl<R: Deref> DeviceMethods for R where R::Target: IsA<DeviceState> {}

Related to this, the IsA trait allows typesafe compile-time checked casts.
Unlike in C code, casting to a superclass can be written in such a way
that the compiler will complain if the destination type is *not* a
superclass, and with zero runtime cost.

Callbacks
'''''''''

This is the least mature of the "urgent" changes, and perhaps the more
important to have a good design for.  PL011 has callbacks for character
devices and memory regions, but other usecases include timers, bottom
halves and interrupt sinks (aka GPIO inputs).

So far all I have is a silly 100% Rust example that uses callbacks
as you would have find tham in C (so with function pointers and void*
opaques); it then implements "bindings" where you write a Rust trait
and the function pointers are filled in automatically.  You can find it
at https://gist.github.com/bonzini/2c63a905851f4f78573d022b1f196f4f.
The reason for doing this exercise with Rust, despite the code
looking nothing like idiomatic Rust, is to make sure that Miri (Rust's
experimental "undefined behavior detector") likes it.

HPET, which Zhao is working on, is particularly interesting because it
also has *arrays* of timers.

Summary:

* This requires some adventurous person to look at it.  It's not impossible
  but it's both harder and high priority.

* Linux has some examples but they are really hard to follow.  If anything
  easier is possible at the cost of leaving some easily-verifed "unsafe"
  blocks (and still provide safe callbacks for timers, memory regions, etc.)
  that may be a better short term plan.


Tracing/logging
'''''''''''''''

Tracepoints and logging are not supported yet, and no one has started working
on it.

For tracing, it's not clear to me how much C code can be reused and how
much Rust code can be automatically generated.  This would be a valid
Outreachy or Summer of Code project, as it has a limited need for unsafe
code and a well-defined scope.

A pure-Rust implementation would be interesting, but note that the usage
of printf-style %-based field formatting is pervasive in the tracing
subsystem.

Logging is a small component of QEMU and it's a good opportunity to define
a coding style for C code translated to Rust, e.g. how to name enums and
functions.  My only request is to allow using the same syntax as format!()
for logging code!

Summary:

* Simple, low priority.

* Anybody wants a small project to learn Rust FFI?


Data structure interoperability
'''''''''''''''''''''''''''''''

The poster child here is "Error", i.e. providing an easy way to convert
from QEMU's Error object to Rust's Error trait and back.

Both Marc-André and I have worked on this with two different approaches.
Marc-André based his on glib-rs, which focuses on the desired ownership
semantics (i.e. whether the resulting data is owned by Rust or C) because
glib-rs in turn was inspired by GIR (the GObject cross-language
introspection interfaces).  Mine instead tries to extend the ideas
behind the Rust traits From<>/Into<>/Clone to provide conversion and
cloning of Rust objects into C objects and vice versa.

The primitives are similar:

    from_glib_full          from_foreign
    from_glib_none          cloned_from_foreign
    to_glib_full            clone_to_foreign_ptr
    to_glib_none            clone_to_foreign

though the design and code are very different

Summary:

* I am obviously biased, so this needs more brains to help with the design.
  It's not urgent but it has to be figured out before we get to QAPI.  Maybe
  even asking on users.rust-lang.org could be an idea.

* I'm not planning any work on this for now, but I'd like whatever approach
  we choose to be an external crate that is developed outside QEMU and just
  used by it.


Features expected from recent rust
''''''''''''''''''''''''''''''''''

QEMU supports rustc version 1.63.0 and newer.  Notably, the following features
are missing:

* ``core::ffi`` (1.64.0).  Using ``std::os::raw`` and ``std::ffi`` instead.

* "let ... else" (1.65.0).  Using ``if let`` instead.  This is currently patched
  in QEMU's vendored copy of the bilge crate.

* ``CStr::from_bytes_with_nul()`` as a ``const`` function (1.72.0).

* MaybeUninit::zeroed() as a ``const`` function (1.75.0).  QEMU uses a
  ``Zeroable`` trait, and you can implement it to provide by hand a
  ``Zeroable::ZERO`` associated constant.

It is expected that QEMU will advance its minimum supported version of
rustc to 1.75.0 as soon as it stops supporting Debian bookworm.  However,
the following will still be missing:

* c"" literals (stable in 1.77.0).  QEMU provides a ``c_str!()`` macro
  to define ``CStr`` constants easily, so no big deal.

* ``offset_of!`` (stable in 1.77.0).  QEMU uses ``offset_of!()`` heavily and
  provides a ``const``-friendly replacement in the ``qemu_api`` crate.
  Also note that *nested* ``offset_of!`` was only stabilized in Rust
  1.82.0, which will probably be useful as soon as we start using BqlCell
  and BqlRefCell more widely.  So we'll be stuck with our replacement
  for a while.

* ``&raw`` (stable in 1.82.0).  Using ``addr_of!`` and ``addr_of_mut!``
  instead, though hopefully the need for raw pointers will go down over time.

* ``new_uninit`` (stable in 1.82.0).  This is used internally by the
  ``pinned_init`` crate, but it can be patched easily when/if QEMU vendors a
  copy of the crate.

* referencing statics in constants (stable in 1.83.0).  For now, using a
  const function is the sole alternative.


QEMU also supports version 0.60.x of bindgen, which is missing option
``--generate-cstr``.  This requires version 0.66.x and will be adopted
as soon as supporting these older versions is not necessary anymore.
This only affects a handful of places, which are easily found as they'll
not compile anymore once the option is addde back.

Re: Rust in QEMU roadmap

[Zhao Liu]

On Tue, Nov 26, 2024 at 06:46:45PM +0100, Paolo Bonzini wrote:
> Date: Tue, 26 Nov 2024 18:46:45 +0100
> From: Paolo Bonzini <pbonzini@redhat.com>
> Subject: Rust in QEMU roadmap
> X-Mailer: git-send-email 2.47.0
> 
> [several people are in bcc.  You can follow the thread at
>  https://lore.kernel.org/r/cc40943e-dec1-4890-a1d9-579350ce296f@pbonzini.local]
> 
> Based on the content presented in the community calls, here are some
> ideas for future Rust in QEMU subprojects.
> 
> This is by no means exhaustive, for example QAPI and async (block devices)
> are missing because I have not looked at them.
> 
> Table of contents
> '''''''''''''''''
> 
> * Status in QEMU 9.2
> * Build system
> * Feature parity for pl011 device
> * Avoiding undefined behavior
> * Safe QOM class definitions
> * QOM method calls
> * Callbacks
> * Tracing/logging
> * Data structure interoperability
> * Features expected from recent rust

Hi Paolo, thanks for your sharing and summary, do you have the plan
about custom allocator (QemuAllocator in rust/qemu-api/src/lib.rs)?

Regards,
Zhao

Re: Rust in QEMU roadmap

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Il mer 27 nov 2024, 07:28 Zhao Liu <zhao1.liu@intel.com> ha scritto:

> Hi Paolo, thanks for your sharing and summary, do you have the plan
> about custom allocator (QemuAllocator in rust/qemu-api/src/lib.rs)?
>

No plan yet, but we'll see once we introduce C data structures allocated
from Rust. Adding the libc crate to the build is not hard and it may be
simpler than enabling the QemuAllocator.

Paolo


Regards,
> Zhao
>
>

[-- Attachment #2: Type: text/html, Size: 1144 bytes --]

Re: Rust in QEMU roadmap

[Stefan Hajnoczi]

On Tue, 26 Nov 2024 at 13:04, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Tracing/logging
> '''''''''''''''
>
> Tracepoints and logging are not supported yet, and no one has started working
> on it.
>
> For tracing, it's not clear to me how much C code can be reused and how
> much Rust code can be automatically generated.  This would be a valid
> Outreachy or Summer of Code project, as it has a limited need for unsafe
> code and a well-defined scope.
>
> A pure-Rust implementation would be interesting, but note that the usage
> of printf-style %-based field formatting is pervasive in the tracing
> subsystem.

The purpose of QEMU's tracing subsystem is to support cross-platform
tracing. Instead of making QEMU code depend on a specific tracer (like
DTrace or LTTng UST) that may not be available across platforms, the
tracing subsystem generates tracer-specific C code for trace events at
compile time. The most straightforward way to expose this to Rust is
to add C functions that invoke each trace event and then call those C
functions from Rust. Tracing is designed to be lightweight and this
approach results in more overhead than in native C tracing though.

A pure Rust solution would be nice, but I'm not sure how to reconcile
that with the purpose of the tracing subsystem, which is to support
the various tracers native to QEMU's host platforms. They are geared
towards C and use low-level techniques to generate low overhead
tracing code.

About format strings, QEMU tracing is not about format strings.
Otherwise logging would just be used instead. Trace event definitions
contain a format string so that tracers that offer human-readable
formatting can use that string. Actual tracing is not about string
formatting though, it's about typed calls to <trace-event>(<arg1>,
<arg2>, ...) where those typed arguments can be captured efficiently
by the tracer (e.g. stored as binary records). Depending on the
tracer, the format string might never be used, be used in a
post-processing step during offline trace file analysis, or at runtime
(like logging).

That's what QEMU's tracing subsystem is today. But it's a low activity
subsystem where things can be changed with less disruption to users
than in most other subsystems. I think it would be nice to look at
what native Rust tracing has to offer ad reconsider some of the
requirements that shaped QEMU's C tracing subsystem. Maybe we can take
a more modern approach.

>
> Logging is a small component of QEMU and it's a good opportunity to define
> a coding style for C code translated to Rust, e.g. how to name enums and
> functions.  My only request is to allow using the same syntax as format!()
> for logging code!
>
> Summary:
>
> * Simple, low priority.
>
> * Anybody wants a small project to learn Rust FFI?

Re: Rust in QEMU roadmap

[Daniel P. Berrangé]

On Wed, Nov 27, 2024 at 05:47:19AM -0500, Stefan Hajnoczi wrote:
> On Tue, 26 Nov 2024 at 13:04, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Tracing/logging
> > '''''''''''''''
> >
> > Tracepoints and logging are not supported yet, and no one has started working
> > on it.
> >
> > For tracing, it's not clear to me how much C code can be reused and how
> > much Rust code can be automatically generated.  This would be a valid
> > Outreachy or Summer of Code project, as it has a limited need for unsafe
> > code and a well-defined scope.
> >
> > A pure-Rust implementation would be interesting, but note that the usage
> > of printf-style %-based field formatting is pervasive in the tracing
> > subsystem.
> 
> The purpose of QEMU's tracing subsystem is to support cross-platform
> tracing. Instead of making QEMU code depend on a specific tracer (like
> DTrace or LTTng UST) that may not be available across platforms, the
> tracing subsystem generates tracer-specific C code for trace events at
> compile time. The most straightforward way to expose this to Rust is
> to add C functions that invoke each trace event and then call those C
> functions from Rust. Tracing is designed to be lightweight and this
> approach results in more overhead than in native C tracing though.
> 
> A pure Rust solution would be nice, but I'm not sure how to reconcile
> that with the purpose of the tracing subsystem, which is to support
> the various tracers native to QEMU's host platforms. They are geared
> towards C and use low-level techniques to generate low overhead
> tracing code.

I think a purer Rust solution looks not too difficult for many of
the backends.

 * dtrace

   Use this crate https://crates.io/crates/usdt to generate
   Rust code from the .d file we already generate

 * ftrace

   A mere write() to a pre-opened FD

 * log

   A call to the qemu_log function, which can be exposed to rust

 * simple

   A call to some funcs in trace/simple.h which can be exposed
   to rust

 * syslog

   A call to syslog(), can be done with https://crates.io/crates/syslog

 * ust

   Unclear how to do this at a quick glance, but likely
   has a solution

Most of these rely on the trace_event_get_state macro which is
where we try to take some fast paths. So the hardest bit is likely
figuring out a nice way to represent the 'state check' in Rust
without it always turning into a function call.

> That's what QEMU's tracing subsystem is today. But it's a low activity
> subsystem where things can be changed with less disruption to users
> than in most other subsystems. I think it would be nice to look at
> what native Rust tracing has to offer ad reconsider some of the
> requirements that shaped QEMU's C tracing subsystem. Maybe we can take
> a more modern approach.

IMHO a core requirement for this must be that we can seemlessly trace
across Rust and C code using the same mechanism.

With my developer hat on, 99% of the time I'm just using the 'log'
backend because it is soooooo simple, very effective for dev use
and we enable it by default.

With my distro hat on, we enable 'dtrace' because that gives us a
low-overhead probe facility for tracing production deployments,
AND crucially it gives us system-global tracing facility across
the kernel, QEMU and all other apps.

IOW, my POV is that support for the "log" and "dtrace" backends
is mandatory for the Rust layer as a minium viable baseline.


I question the usefulness of the 'syslog' trace target. I can't see
it being desirable as a option for 'production' builds, and it seems
uneccessarily indirect for developers. What's its compelling USP ?

WRT 'ftrace', IIUC, the Linux 'perf' command has built-in support
for using USDT probes now:

  https://www.brendangregg.com/perf.html#StaticUserTracing

Do we still have a compelling USP for the 'ftrace' backend, if we
can use 'perf' for this ?

WRT 'ust', in LTTng, I see reference to the ability to use USDT probes,
so I wonder if we still need a dedicated 'ust' backend around ?

Potentially we're down to three options 'log', 'simple', and 'dtrace'.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Rust in QEMU roadmap

[Paolo Bonzini]

On Wed, Nov 27, 2024 at 1:18 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> I question the usefulness of the 'syslog' trace target. I can't see
> it being desirable as a option for 'production' builds, and it seems
> uneccessarily indirect for developers. What's its compelling USP ?
>
> WRT 'ftrace', IIUC, the Linux 'perf' command has built-in support
> for using USDT probes now:
>
>   https://www.brendangregg.com/perf.html#StaticUserTracing

I also wonder if 'ftrace' is duplicating 'log' these days. Anyhow, if
we can deprecate 'syslog' and 'ftrace' that would be good to know
independent of Rust.

Paolo

> WRT 'ust', in LTTng, I see reference to the ability to use USDT probes,
> so I wonder if we still need a dedicated 'ust' backend around ?
>
> Potentially we're down to three options 'log', 'simple', and 'dtrace'.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: Rust in QEMU roadmap

[Daniel P. Berrangé]

On Tue, Nov 26, 2024 at 06:46:45PM +0100, Paolo Bonzini wrote:
> Feature parity for pl011 device
> '''''''''''''''''''''''''''''''
> 
> Some recent pl011 commits are missing in the Rust version.  Philippe
> volunteered to port them to teach himself Rust.
> 
> Philippe also has a series to implement flow control and better use
> of the PL011 FIFO (IIUC).  Porting this to Rust would be hard right
> now, so its Rust implementation blocked on having chardev bindings.
> 
> Also missing is logging and tracing, but this part should not be a
> blocker for defaulting to --enable-rust.

I think it is OK to miss tracing as an exceptional one-off.

It is highly undesirable though for us to convert more than a
couple of devices (which currently use tracing) without having
tracing working. I consider tracing a critical feature for
supportability of QEMU.

IOW, I'm OK with lack of tracing not being a block for defaulting
to --enable-rust, provided it is accepted as being a high priority
item. Possibly call it a blocker for accepting conversions of
anything beyond pl011 & hpet, that currently has trace points.


> Tracing/logging
> '''''''''''''''
> 
> Tracepoints and logging are not supported yet, and no one has started working
> on it.
> 
> For tracing, it's not clear to me how much C code can be reused and how
> much Rust code can be automatically generated.  This would be a valid
> Outreachy or Summer of Code project, as it has a limited need for unsafe
> code and a well-defined scope.

As mentioned above, IMHO this needs to be treated as a high priority
item. While it would make a good Outreachy/GSoC project, I don't think
we should take that path, as it comes with no guarantee of a successful
timely delivery.

> A pure-Rust implementation would be interesting, but note that the usage
> of printf-style %-based field formatting is pervasive in the tracing
> subsystem.

Note that it is not unbounded printf style formatting. We are actually
pretty restricted in what we can use because our format strings are
required to be compatible with systemtap's formatter which is very
limited compared to C printf().

We've currently got an adhoc check to block use of '%m', but we could
easily make this a much stricter check on '%' and thus make it practical
to auto-convert our printf() style format strings to a format that rust
can use natively.

> Summary:
> 
> * Simple, low priority.

IMHO high priority. I don't want to enable Rust in Fedora/RHEL if
it going to degrade our ability to use tracing to debug production
deployments

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Rust in QEMU roadmap

[Paolo Bonzini]

On Tue, Nov 26, 2024 at 6:48 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
> Callbacks
> '''''''''
>
> This is the least mature of the "urgent" changes, and perhaps the more
> important to have a good design for.  PL011 has callbacks for character
> devices and memory regions, but other usecases include timers, bottom
> halves and interrupt sinks (aka GPIO inputs).

Ok, I have an idea that should make it possible to write something like this:

pub struct Timer {
    t: *mut QEMUTimer,
}

impl Timer {
    // FnCall is QEMU-specific and contains all the magic.
    unsafe extern "C" fn rust_timer_cb<
        'a, T, F: FnCall<(&'a T,)>>(opaque: *mut c_void) {
        // The argument to F::call is a tuple, otherwise this is fairly expected
        F::call((unsafe { &*(opaque.cast::<T>()) },))
    }

    #[inline]
    fn new_ns<'a, T, F: FnCall<(&'a T,)>>(
        clk: QEMUClockType,
        _cb: F,
        opaque: &'a T,
    ) -> Timer {
        let cb: unsafe extern "C" fn(*mut c_void) = rust_timer_cb::<'a, T, F>;
        Timer {
            t: unsafe { bindings::timer_new_ns(clk, cb, opaque) },
        }
    }
}

and then the caller can simply write

   self.timer = Timer::new_ns(QEMU_CLOCK_VIRTUAL, Self::timer_update, self);

The idea is that the compiler will generate a different version of
rust_timer_cb for every value of the second argument (which is unused,
except inasmuch as it helps the compiler know what "F::call" means).
In the code above "unsafe" is only needed for the unavoidable casts
and FFI calls, while the device has no unsafe at all. I haven't
written the timer bindings but the basic idea is like the above. It
can be expanded once Zhao posts his HPET conversion.

It may also be possible to use macros to make things a little less
verbose, but I'm tending to err on the side of explicitness for now.

I'm also not sure about the applicability of this trick to character
devices and memory regions, since those have multiple callbacks and/or
callbacks in structs rather than function parameters. However, for
timers this is a good option I think.

Paolo

Re: Rust in QEMU roadmap

[Daniel P. Berrangé]

On Wed, Nov 27, 2024 at 01:22:00PM +0100, Paolo Bonzini wrote:
> On Wed, Nov 27, 2024 at 1:18 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > I question the usefulness of the 'syslog' trace target. I can't see
> > it being desirable as a option for 'production' builds, and it seems
> > uneccessarily indirect for developers. What's its compelling USP ?
> >
> > WRT 'ftrace', IIUC, the Linux 'perf' command has built-in support
> > for using USDT probes now:
> >
> >   https://www.brendangregg.com/perf.html#StaticUserTracing
> 
> I also wonder if 'ftrace' is duplicating 'log' these days. Anyhow, if
> we can deprecate 'syslog' and 'ftrace' that would be good to know
> independent of Rust.

I looked back at the history of syslog and re-discovered that I raised
the same question when it was first proposed. It was said that syslog
was to be used by XenServer for some kind of logging they wanted:

  https://lists.nongnu.org/archive/html/qemu-devel/2016-08/msg00336.html

I've CC'd Paul as it'd be useful to know if Xen did end up using this
feature or not, as we don't want to rip the rug from under them.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Rust in QEMU roadmap

[Paul Durrant]

On 05/12/2024 12:00, Daniel P. Berrangé wrote:
> On Wed, Nov 27, 2024 at 01:22:00PM +0100, Paolo Bonzini wrote:
>> On Wed, Nov 27, 2024 at 1:18 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> I question the usefulness of the 'syslog' trace target. I can't see
>>> it being desirable as a option for 'production' builds, and it seems
>>> uneccessarily indirect for developers. What's its compelling USP ?
>>>
>>> WRT 'ftrace', IIUC, the Linux 'perf' command has built-in support
>>> for using USDT probes now:
>>>
>>>    https://www.brendangregg.com/perf.html#StaticUserTracing
>>
>> I also wonder if 'ftrace' is duplicating 'log' these days. Anyhow, if
>> we can deprecate 'syslog' and 'ftrace' that would be good to know
>> independent of Rust.
> 
> I looked back at the history of syslog and re-discovered that I raised
> the same question when it was first proposed. It was said that syslog
> was to be used by XenServer for some kind of logging they wanted:
> 
>    https://lists.nongnu.org/archive/html/qemu-devel/2016-08/msg00336.html
> 
> I've CC'd Paul as it'd be useful to know if Xen did end up using this
> feature or not, as we don't want to rip the rug from under them.
> 

Pretty sure we did use it in XenServer. I've not been involved for more 
than 5 years now so maybe someone else who's still working on XenServer 
knows better... Not sure of the right contact though, unfortunately.

   Paul