[Qemu-devel] [PATCH v5 00/18] migration: add static analysis tool to check vmstate compat

[Amit Shah <amit.shah@redhat.com>]

Hello,

v5:
 - Add section / description name checking for matching names for
   changed fields.  This has the benefit of also getting rid of the
   for loop in the lookup.  However, for each section, like piix4_pm,
   all the changes have to be in one group, as against them being in
   different ones.  That's still better than earlier where sections
   were not checked.  If my Python gets better, I might come up with a
   way to improve on this as well. (Juan Quintela)
 - Add more comments to the Python script
 - Update help text to be more verbose - for qemu and for the script.
 - Simplify the check_fields() function - lesser indentations

v4:
 - Return value capped to 255 to prevent wrap-around (Eric Blake)

v3:
 - Python script returns an error code: 0 if no errors, positive for
   the number of errors identified.

v2:
 - Tabs->spaces (Dave Gilbert)
 - Several changes to the python script to make it more python-like
   (Vitaly Kuznetsov)
 - Don't store the empty fields created by VMSTATE_VALIDATE in the
   json output

This series adds a static vmstate checker to check for breakage of
live migration by analyzing the vmstate information between different
QEMU versions.

In patch 1, QEMU is modified to add a -dump-vmstate commandline
option, which takes a filename as the argument.  When invoked, QEMU
dumps the vmstate info in JSON format for the current machine type to
the file.  This patch is loosely based on a version from Andreas
Färber.

This JSON file is then fed into the Python script introduced in patch
2.  The script takes 'src' and 'dest' arguments, indicating the
direction of migration.  The script then performs a series of checks
and spews out information on inconsistencies it finds in the data.

Two stripped-down versions of JSON dumps are included in this
patchset (patch 3).

Patches 4 - 18 contain modifications to those dumps, to show the
checks that are performed by the script.  The result of running the
script against the final version of those files is appended below.

The checks are to be performed for a particular machine type, and
comparing different machine types is bound to turn up false-positives:
e.g.

(in a qemu 2.0 tree):
./x86_64-softmmu/qemu-system-x86_64 -dump-vmstate qemu-2.0.json

(in a qemu 2.2 tree:)
./x86_64-softmmu/qemu-system-x86_64 -dump-vmstate -M pc-i440fx-2.0 \
   qemu-2.2-m2.0.json

./scripts/vmstate-static-checker.py -s qemu-2.0.json -d qemu-2.2-m2.0.json

should not show any output.

The idea is to include this script in 'make check', ensuring new
commits don't break migration.

Later, this script can also be baked into the release process: vmstate
dumps for released versions of qemu for various machine types can be
stored in a directory in the tree.  At the time of freezing the tree
for releases, (like -rc2), the dump for the current release can be
updated.  A policy that says "No more live migration-breaking changes
can be accepted post -rc2" can be put in place.  Also, for checks for
older machine types on the current tree with the saved dumps should
not indicate any regressions.

I expect there to be a few false positives at the start (though there
aren't any right now).  The script will keep evolving, though, to
include new scenarios, to make it more helpful.

A later project would be to also store other guest-visible information
and use that output for more checks (including lspci output from
inside guests).

As mentioned earlier, this is the result of running the script against
the sample json data included in this patchset:

$ ./scripts/vmstate-static-checker.py --src ./tests/vmstate-static-checker-data/dump1.json  --dest ./tests/vmstate-static-checker-data/dump2.json 
Section "usb-kbd" Description "usb-kbd" Field "kbd.keycodes" size mismatch: 4 , 2
Section "PIIX3-xen" Description "PIIX3": minimum version error: 1 < 2
Section "PIIX3-xen" Description "PIIX3": Entry "Subsections" missing
Section "tpci200": Description "tpci200" missing, got "tpci2002" instead; skipping
Section "megasas", Description "PCIDevice": expected field "irq_state", while dest has no further fields
Section "SUNW,fdtwo" Description "fdc": version error: 2 > 1
Section "SUNW,fdtwo", Description "fdrive": Subsection "fdrive/media_rate" not found
Section "fusbh200-ehci-usb" version error: 2 > 1
Section "fusbh200-ehci-usb", Description "ehci-core": expected field "usbsts", got "usbsts_pending"; skipping rest
Section "intel-hda-generic", Description "intel-hda", Field "pci": missing description
Section "cfi.pflash01": Entry "Description" missing
Section "pci-serial-4x" Description "pci-serial-multi": Entry "Fields" missing
Warning: checking incompatible machine types: "pc-i440fx-2.1", "pc-i440fx-2.2"
Section "fw_cfg" does not exist in dest

$ echo $?
13

Amit Shah (18):
  migration: dump vmstate info as a json file for static analysis
  vmstate-static-checker: script to validate vmstate changes
  tests: vmstate static checker: add dump1 and dump2 files
  tests: vmstate static checker: incompat machine types
  tests: vmstate static checker: add version error in main section
  tests: vmstate static checker: version mismatch inside a Description
  tests: vmstate static checker: minimum_version_id check
  tests: vmstate static checker: remove a section
  tests: vmstate static checker: remove a field
  tests: vmstate static checker: remove last field in a struct
  tests: vmstate static checker: change description name
  tests: vmstate static checker: remove Fields
  tests: vmstate static checker: remove Description
  tests: vmstate static checker: remove Description inside Fields
  tests: vmstate static checker: remove a subsection
  tests: vmstate static checker: remove Subsections
  tests: vmstate static checker: add substructure for usb-kbd for hid
    section
  tests: vmstate static checker: add size mismatch inside substructure

 include/migration/vmstate.h                  |    2 +
 qemu-options.hx                              |   14 +
 savevm.c                                     |  139 +++
 scripts/vmstate-static-checker.py            |  345 ++++++++
 tests/vmstate-static-checker-data/dump1.json | 1163 ++++++++++++++++++++++++++
 tests/vmstate-static-checker-data/dump2.json |  968 +++++++++++++++++++++
 vl.c                                         |   13 +
 7 files changed, 2644 insertions(+)
 create mode 100755 scripts/vmstate-static-checker.py
 create mode 100644 tests/vmstate-static-checker-data/dump1.json
 create mode 100644 tests/vmstate-static-checker-data/dump2.json

-- 
1.9.3