All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 00/21] system: introduce a live data recorder facility
@ 2026-10-15 11:20 Daniel P. Berrangé
  2026-10-15 11:20 ` [PATCH 01/21] osdep: add 'ignore_value(x)' macro Daniel P. Berrangé
                   ` (22 more replies)
  0 siblings, 23 replies; 29+ messages in thread
From: Daniel P. Berrangé @ 2026-10-15 11:20 UTC (permalink / raw)
  To: qemu-devel
  Cc: Marc-André Lureau, Michael Roth, Philippe Mathieu-Daudé,
	Fabiano Rosas, Richard Henderson, Dr. David Alan Gilbert,
	Paolo Bonzini, Markus Armbruster, Pierrick Bouvier, Eric Blake,
	Daniel P. Berrangé, Laurent Vivier

This series introduces a new facility called 'tldr' which is a
live data recorder. This is intended to aid support of production
QEMU deployments, by providing a data collection facility that
can be left "always on", such that in the event of a QEMU crash
or guest OS crash, certain important information will be recorded.

The data recorder has to be enabled by adding a "tldr-config"
object instance with -object / object_add.

With this series, the data recorder collects

 * error_report/warn_report/error_printf messages
 * QMP events
 * QMP errors
 * QMP commands, which are not query-XXX/qom-list,
   or any command associated with a QMP error.
 * qemu_log output

The last item is intended primarily to be used in combination
with tracing and its 'log' backend, to have continuous
recording of certain interesting probes.

Each type of data is collected in a distinct circular
buffer, such that high volumes of one type of data, does
not result in overwriting other types of low volume data.

The data recorder is set to trigger on SIGBUS, SIGSEGV,
SIGILL, SIGFPE and SIGABRT, or manually via QMP 'tldr-dump'.

There is a open question in one of the patches about SIGBUS
handling, since that is already used for MCE capture and I'm
unclear if I'm breaking something about KVM MCE reporting.

This series proposes using sigaltstack() to register an
alternative signal handling stack. This would mainly be
useful if we have a crash from a stack overflow, but the
cost is that we have 8k extra per thread, and I'm not
entirely convinced that's a beneficial tradeoff. Might
be better to just accept stack overflows as a risk.

This series only captures text based info, but as a followup
I would like to extend this to capture statistics data from
relevant subsystems. For example, the BlockBackend has its
own "always on" stats accounting code, which can be queried
with QMP. Ideally I would like to figure out a way to dump
all the BlockBackend stats in the data recorder output.
Likewise perhaps capture NIC stats and/or KVM stats in an
"always on" manner for some interesting baseline dataset.
The challenge with that is making it safe to access the
stats data without taking any mutex locks, since this must
run from a signal handler.

A typical recommended usage would be:

 $QEMU  -object tldr-config,id=tldr
        -d tldr
        -trace events=/var/lib/libvirt/qemu/trace.txt

This leaves dumps to go to QEMU's stderr on crash, but
to send them to a file

   -object tldr-config,id=tldr,crash-target=/some/path

The '-d tldr' flag redirects qemu_log() to the TLDR buffer
instead of stdout, and the '-trace events' arg defines
the "always on" tracepoints we want to capture.

An example crash dump seen if I send SIGBUS to a libvirt
managed QEMU is, with 'object*' tracepoints collected:

🚨🚨🚨 QEMU Data Recorder Start 🚨🚨🚨

🔍 Trigger:
  - Signal: SIGBUS (7)
  - Code: User process kill (0)
  - Address: 0x1e5a9e

🎥  Message recording 'command':
2026-10-15T10:59:11.318427Z {"execute": "qmp_capabilities", "id": "libvirt-1"}
2026-10-15T10:59:11.322047Z {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "events"}]}, "id": "libvirt-3"}
2026-10-15T10:59:11.347566Z {"execute": "balloon", "arguments": {"value": 4294967296}, "id": "libvirt-9"}
2026-10-15T10:59:11.354570Z {"execute": "set-action", "arguments": {"panic": "pause"}, "id": "libvirt-12"}
2026-10-15T10:59:11.358046Z {"execute": "cont", "id": "libvirt-14"}

🎥  Message recording 'event':
2026-10-15T10:59:11.358108Z {"timestamp": {"seconds": 1792061951, "microseconds": 358094}, "event": "RESUME"}
2026-10-15T10:59:20.757392Z {"timestamp": {"seconds": 1792061960, "microseconds": 757371}, "event": "VSERPORT_CHANGE", "data": {"open": true, "id": "console1"}}
2026-10-15T10:59:24.431398Z {"timestamp": {"seconds": 1792061964, "microseconds": 431379}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "net0", "path": "/machine/peripheral/net0/virtio-backend"}}
2026-10-15T10:59:32.586658Z {"timestamp": {"seconds": 1792061972, "microseconds": 586603}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[12]/rtc"}}

🎥  Message recording 'log':
28:QIO_CHANNEL_FILE)
2026-10-15T10:59:31.753125Z object_dynamic_cast_assert obj=0x557d5c828780 type=chardev-pty->chardev-pty (../chardev/char-pty.c:50:PTY_CHARDEV)
2026-10-15T10:59:31.753128Z object_dynamic_cast_assert obj=0x557d5c828780 type=chardev-pty->chardev-pty (../chardev/char-pty.c:50:PTY_CHARDEV)
2026-10-15T10:59:31.796114Z object_dynamic_cast_assert obj=0x557d5ca7d3a0 type=host-x86_64-cpu->x86_64-cpu (../target/i386/cpu-qom.h:31:X86_CPU)
2026-10-15T10:59:31.796148Z object_class_dynamic_cast_assert type=kvm-apic->apic-common (/home/berrange/src/virt/qemu/include/hw/i386/apic_internal.h:131:APIC_COMMON_GET_CLASS)

...snip...

2026-10-15T10:59:33.713641Z object_dynamic_cast_assert obj=0x557d5cc20b10 type=PCIE->PCI (/home/berrange/src/virt/qemu/include/hw/pci/pci.h:289:PCI_BUS)
2026-10-15T10:59:33.713643Z object_class_dynamic_cast_assert type=PCIE->PCI (/home/berrange/src/virt/qemu/include/hw/pci/pci.h:289:PCI_BUS_GET_CLASS)
2026-10-15T10:59:33.713648Z object_dynamic_cast_assert obj=0x557d5ca03700 type=host-x86_64-cpu->x86_64-cpu (../target/i386/cpu-qom.h:31:X86_CPU)
2026-10-15T10:59:33.713664Z object_dynamic_cast_assert obj=0x557d5eadd320 type=virtio-gpu-device->virtio-device (/home/berrange/src/virt/qemu/include/hw/virtio/virtio.h:91:VIRTIO_DEVICE)

🚨🚨🚨 QEMU Data Recorder End 🚨🚨🚨

"object_*" is not a particularly good choice for 'always on'
data collection as it is high volume and mostly noise. The
job of identifying the "interesting" low volume tracepoints
is outstanding. We can provide guidance upstream but downstream
should control what they're interested in collecting. We might
want to add further tracepoints in the code at interesting
places that we useful for post-crash debugging.

Daniel P. Berrangé (21):
  osdep: add 'ignore_value(x)' macro
  system: move SIGBUS signal handling to signal.c
  system: rename qemu_init_sigbus to qemu_init_crash_signals
  system: run SIGBUS handler on alternate signal stack
  util: expand alternate signal stack to all threads
  osdep: provide dummy siginfo_t definition
  util: introduce 'tldr', a system data recorder for crash handling
  qapi: add 'tldr-dump' command for on-demand data recorder output
  system: trigger TLDR crash dump on unhandled SIGBUS
  system: simplify signal handler logic
  main: allow SIGBUS in the main loop
  system: enable SIGBUS handling on non-Linux platforms
  system: enable crash dumps on SIGSEGV, SIGILL, SIGFPE and SIGABRT
  util: introduce capture of textual messages in data recorder
  util: capture warnings and errors in the data recorder
  monitor: remove obsolete/redundant exports
  monitor: capture monitor commands and events in data recorder
  log: enable directing log messages to data recorder
  tests/qtest: add API to declare an expected crash signal
  qtest: add 'crash' command to generate 'authentic' process crashes
  tests/qtest: add test to validate data recorder on crashes

 MAINTAINERS                 |  12 +
 include/qemu/log.h          |   1 +
 include/qemu/osdep.h        |  16 +
 include/qemu/tldr.h         |  76 ++++
 include/system/signal.h     |  25 ++
 meson.build                 |   4 +
 monitor/monitor-internal.h  |   2 -
 monitor/qmp.c               |  66 +++-
 qapi/meson.build            |   1 +
 qapi/qapi-schema.json       |   1 +
 qapi/qom.json               |  18 +
 qapi/tldr.json              |  62 +++
 qemu-options.hx             |  24 ++
 stubs/meson.build           |   1 +
 stubs/monitor-puts.c        |   9 +
 system/cpus.c               |  78 +---
 system/meson.build          |   1 +
 system/qtest.c              |  50 +++
 system/signal.c             | 149 +++++++
 tests/qtest/libqtest.c      |  62 ++-
 tests/qtest/libqtest.h      |  27 ++
 tests/qtest/meson.build     |   3 +
 tests/qtest/tldr-test.c     | 124 ++++++
 tests/unit/meson.build      |   1 +
 tests/unit/test-util-tldr.c | 377 ++++++++++++++++++
 util/error-report.c         |  43 ++-
 util/log.c                  |  42 +-
 util/main-loop.c            |   1 -
 util/meson.build            |   1 +
 util/qemu-thread-posix.c    |  20 +
 util/tldr-internal.h        |  52 +++
 util/tldr.c                 | 750 ++++++++++++++++++++++++++++++++++++
 32 files changed, 1976 insertions(+), 123 deletions(-)
 create mode 100644 include/qemu/tldr.h
 create mode 100644 include/system/signal.h
 create mode 100644 qapi/tldr.json
 create mode 100644 stubs/monitor-puts.c
 create mode 100644 system/signal.c
 create mode 100644 tests/qtest/tldr-test.c
 create mode 100644 tests/unit/test-util-tldr.c
 create mode 100644 util/tldr-internal.h
 create mode 100644 util/tldr.c

-- 
2.56.0



^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2026-10-15 17:34 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-15 11:20 [PATCH 00/21] system: introduce a live data recorder facility Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 01/21] osdep: add 'ignore_value(x)' macro Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 02/21] system: move SIGBUS signal handling to signal.c Daniel P. Berrangé
2026-10-15 12:15   ` marcandre.lureau
2026-10-15 11:20 ` [PATCH 03/21] system: rename qemu_init_sigbus to qemu_init_crash_signals Daniel P. Berrangé
2026-10-15 12:15   ` marcandre.lureau
2026-10-15 11:20 ` [PATCH 04/21] system: run SIGBUS handler on alternate signal stack Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 05/21] util: expand alternate signal stack to all threads Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 06/21] osdep: provide dummy siginfo_t definition Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 07/21] util: introduce 'tldr', a system data recorder for crash handling Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 08/21] qapi: add 'tldr-dump' command for on-demand data recorder output Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 09/21] system: trigger TLDR crash dump on unhandled SIGBUS Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 10/21] system: simplify signal handler logic Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 11/21] main: allow SIGBUS in the main loop Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 12/21] system: enable SIGBUS handling on non-Linux platforms Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 13/21] system: enable crash dumps on SIGSEGV, SIGILL, SIGFPE and SIGABRT Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 14/21] util: introduce capture of textual messages in data recorder Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 15/21] util: capture warnings and errors in the " Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 16/21] monitor: remove obsolete/redundant exports Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 17/21] monitor: capture monitor commands and events in data recorder Daniel P. Berrangé
2026-10-15 15:57   ` Dr. David Alan Gilbert
2026-10-15 16:11     ` Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 18/21] log: enable directing log messages to " Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 19/21] tests/qtest: add API to declare an expected crash signal Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 20/21] qtest: add 'crash' command to generate 'authentic' process crashes Daniel P. Berrangé
2026-10-15 11:20 ` [PATCH 21/21] tests/qtest: add test to validate data recorder on crashes Daniel P. Berrangé
2026-10-15 12:15 ` [PATCH 00/21] system: introduce a live data recorder facility marcandre.lureau
2026-10-15 12:32   ` Daniel P. Berrangé
2026-10-15 17:34 ` Pierrick Bouvier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.