Re: [RFC PATCH] linux-user: add syscall-filter and fork-cpu-loop in plugin interfaces

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

Hi Ziyang,

On 10/8/25 6:41 AM, Ziyang Zhang wrote:
> This commit introduces a syscall filtering mechanism for user-mode
> emulation, allowing plugins to intercept and handle system calls.
> 
> Key features:
> - New API functions: qemu_plugin_set_syscall_filter() and
>    qemu_plugin_fork_cpu_loop()
> - A new enum qemu_plugin_syscall_filter_ret to represent filter results
> - Integration into the cpu loop for i386 and x86_64
> - A new header file include/user/syscall-filter.h
> 
> The filtering mechanism works by allowing a plugin to register a
> callback that is invoked before each system call. The callback can
> decide whether to pass, skip, or exit the current cpu loop. When
> skipping, the plugin can provide a value to be returned to the guest.
> 
> Additionally, a new function qemu_plugin_fork_cpu_loop() is provided to
> allow plugins to fork a new cpu loop, which is useful for temporarily
> resuming guest emulation (e.g., for executing callbacks) during the
> first system call.
> 
> The implementation inserts the filter checks in the cpu loop for both
> the int $0x80 (i386) and syscall (x86_64) paths.
> 
> At KVM Forum 2025, we presented Lorelei, a system designed to enable
> guest programs executed by the QEMU user-mode emulator to accelerate
> performance by invoking natively compiled host libraries via syscalls.
> We have successfully supported libraries such as SDL, OpenGL, and
> Vulkan, allowing QEMU user-mode to emulate GPU-dependent games.
> 
> Following an invitation from Mr. Bouvier to upstream Lorelei to QEMU, we
> refined its architecture to load the Lorelei module as a TCG plugin.
> 
> Since host libraries may execute guest function pointers (callbacks),
> Lorelei implements a novel nested execution model:
> 
> - During syscall filtering, Lorelei triggers a temporary cpu_loop when a
>    host library begins executing a guest callback
> - This temporarily reverts to full emulation mode to execute the guest
>    callback
> - The guest code then issues a magic syscall to exit the temporary
>    cpu_loop
> - Control returns to the host library's original execution context
> 
> 
> Slides: https://gitlab.com/qemu-project/kvm-forum/-/raw/main/_attachments/2025/Slides_DQPMaZE.pdf
> Video: https://www.youtube.com/watch?v=_jioQFm7wyU&list=PLW3ep1uCIRfxwmllXTOA2txfDWN6vUOHp&index=22
> 
> Signed-off-by: Ziyang Zhang <functioner@sjtu.edu.cn>
> ---
>   include/qemu/plugin.h         | 28 +++++++++++++++++++
>   include/qemu/qemu-plugin.h    | 44 +++++++++++++++++++++++++++++
>   include/user/syscall-filter.h | 36 ++++++++++++++++++++++++
>   linux-user/i386/cpu_loop.c    | 52 +++++++++++++++++++++++++++++++++--
>   linux-user/main.c             |  1 +
>   linux-user/user-internals.h   |  1 +
>   plugins/api.c                 | 14 ++++++++++
>   plugins/core.c                | 31 +++++++++++++++++++++
>   plugins/plugin.h              |  8 ++++++
>   9 files changed, 213 insertions(+), 2 deletions(-)
>   create mode 100644 include/user/syscall-filter.h
> 

Thanks for sharing this on the mailing list.

To add more context for Alex and Richard, the goal is indeed to support 
Lorelei by enhancing plugin interface.
By having a "magic syscall", control can be sent back to plugin.
This idea could be extended later similarly in system mode, by defining 
an hypercall mechanism.

All Lorelei libraries will be out of QEMU tree, but we need a way to 
connect this to existing QEMU, and that's the goal here.
There is a great potential to accelerate QEMU user mode by leveraging 
native libraries, so it's worth taking a look.

The idea to filter syscalls has been recently brought too by Florian 
[1]. In the end, it has been preferred to provide a set_pc interface 
[2]. While it could work, I think it's not the right direction.

Offering a syscall based interface allows to have something architecture 
independent for this kind of use case, while set_pc approach implies you 
need to find next instruction, and manually read all parameters through 
registers. As well, since syscall filtering seems to fix two different 
use cases, I would really be in favor of having such an interface 
available upstream.

Finally, there are some additional needs for Lorelei, but we can start 
talking about this syscall interface first.

Alex and Richard, what are your thoughts about this?

[1] https://lists.nongnu.org/archive/html/qemu-devel/2025-08/msg00656.html
[2] 
https://lore.kernel.org/qemu-devel/e9bcd7c7-2d67-469e-b2f3-d1a68e456b2b@epfl.ch/

Regards,
Pierrick