Re: [PATCH 00/21] system: introduce a live data recorder facility

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, Oct 16, 2026 at 11:20:11AM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Oct 15, 2026 at 4:32 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Thu, Oct 15, 2026 at 04:15:52PM +0400, marcandre.lureau@redhat.com wrote:
> > > > This series introduces a new facility called 'tldr' which is a
> > > > live data recorder. This is intended to aid support of production
> > > > QEMU deployments, by providing a data collection facility that
> > > > can be left "always on", such that in the event of a QEMU crash
> > > > or guest OS crash, certain important information will be recorded.
> > > >
> > > > The data recorder has to be enabled by adding a "tldr-config"
> > > > object instance with -object / object_add.
> > > >
> > > > With this series, the data recorder collects
> > > >
> > > >  * error_report/warn_report/error_printf messages
> > > >  * QMP events
> > >
> > > It should probably be recorded at emission, not per monitor
> >
> > Opps.
> >
> > > >  * QMP errors
> > > >  * QMP commands, which are not query-XXX/qom-list,
> > > >    or any command associated with a QMP error.
> > > >  * qemu_log output
> > > >
> > > > The last item is intended primarily to be used in combination
> > > > with tracing and its 'log' backend, to have continuous
> > > > recording of certain interesting probes.
> > > >
> > > > Each type of data is collected in a distinct circular
> > > > buffer, such that high volumes of one type of data, does
> > > > not result in overwriting other types of low volume data.
> > > >
> > > > The data recorder is set to trigger on SIGBUS, SIGSEGV,
> > > > SIGILL, SIGFPE and SIGABRT, or manually via QMP 'tldr-dump'.
> > >
> > > Instead of allocating and dumping by a signal handler, why not use a file-backed
> > > mmap or a memfd? We would need to define a small fixed header for the file
> > > layout/buckets.  This would avoid all the issues with signal handlers.
> >
> > That would impose a continuous I/O workload on data recording, which
> > is something I wanted to explicitly avoid, until the time the data is
> 
> Yes, a file-backed mapping shouldn't be the default, but memfd/tmpfs
> should have the same performance.
> 
> > needed to be preserved. This only persists the final data, not any
> > intermediate data that is overwritten as the circular buffer wraps.
> 
> Same design, the reader would handle the ring data
> 
> > Coupling the in-memory layout to the on-disk layout also means that
> > dumps would always be the max size, since we would not have grow-on
> > demand buffers.
> 
> The dump output would not be the raw data, a reader would emit the
> same text format, skipping the unused parts.

IMHO that reader has to be QEMU owned code, as otherwise we're
exposing a private data structure to a 3rd party and that makes
it public ABI/API which is highly undesirable for this.

> mapping reserves address space only, pages are allocated on first write.
> 
> The benefits:
> - no signal handler, no conflict with MCE, ASAN etc

A signal handler is integral to some extent, as captured info about
the signal is itself is an important part of the data record. IOW,
even if moving writing of the dump out of the signal handler in some
manner, I still want to be able to capture several siginfo_t fields.

You can get that info from a core dump, but the working premise
is that a core dump is not guaranteed available, whereas the data
record dumps are something we can more easily acquire.

> - data survives SIGKILL, OOM, or a crash in the dump path itself

Simplifying the signal handler to just stash the siginfo_t would
help.

> - data can be read from a hung QEMU

Data can always be read from a hung QEMU by sending it a signal.
We could wire up a non-fatal signal for this purpose if we want
to read hung QEMUs without killing them.

> Someone needs to hold the memfd to outlive QEMU, otherwise QEMU could
> create a file under a runtime dir on tmpfs

I wonder if QEMU could simply fork a child process itself that just
hangs around waiting for death, and has a shared memory region with
the main QEMU.

We already have a '-run-with async-teardown' comamnd line, which
forks a child process and uses prctl(PR_SET_PDEATHSIG) to get
notified with the parent goes away.  We could expand that child
process to also handle crash dump writing.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|