[Qemu-devel] [RFC v3 PATCH 00/45] Initial support of multi-process qemu

[Jagannathan Raman <jag.raman@oracle.com>]

Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
the multi-process project is now a prototype and presented in this patchset.

This first series enables the emulation of lsi53c895a in a separate process.

We posted the Proof Of Concept patches [2] before the BoF session in 2018.
Subsequently, we posted RFC v1 [3] & RFC v2 [4] of this series. 

Thanks to the v1 & v2 review, we were able to incorporate the feedback into
our goals.

In the summer of 2019, we participated in a conference with folks at RedHat,
who provided us with feedback to improve the design of this project. 

We want to present version 3 of this series which incorporates the feedback
we received for v2 & the enhancements suggested in the conference. Following
are the improvements made in this series:
  - The command-line arguments are Orchestrator friendly. A new option
    named "-remote" carries the command-line of the emulation process.
    Added "rid" suboption to the "-device" option and moved "-drive"
    option to the emulation process's command-line.
  - Redesigned Monitor interface allows direct access to the remote process,
    thereby making it more scalable, maintainable & Orchestrator friendly.
    We also enabled more monitor commands for the remote process.
  - Enabled the remote process to run multiple LSI controllers
  - Redesigned the communication link to support multiple channels. MMIO
    transactions are carried out in a separate channel.
  - Fixed the bugs discovered during testing. Individual patches provide
    the details of these fixes.

Following people contributed to this patchset:

John G Johnson <john.g.johnson@oracle.com>
Jagannathan Raman <jag.raman@oracle.com>
Elena Ufimtseva <elena.ufimtseva@oracle.com>
Kanth Ghatraju <kanth.ghatraju@oracle.com>

For full concept writeup about QEMU disaggregation refer to
docs/devel/qemu-multiprocess.txt. Please refer to 
docs/qemu-multiprocess.txt for usage information.

We are planning on making the following improvements in the future:
  - Live migration
  - Performance improvements
  - Libvirt support
  - Enforcement of security policies
  - blockdev support

We welcome all your ideas, concerns, and questions for this patchset.

Thank you!

[1]: http://events17.linuxfoundation.org/sites/events/files/slides/KVM%20FORUM%20multi-process.pdf
[1]: https://www.youtube.com/watch?v=Kq1-coHh7lg
[2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
[3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
[4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html

Elena Ufimtseva (20):
  multi-process: add a command line option for debug file
  multi-process: introduce proxy object
  mutli-process: build remote command line args
  multi-process: add support of device id to communication channel
  multi-process: modify BARs read/write to support dev_id
  multi-process: support dev id in config read/write
  multi-process: configure remote side devices
  multi-process: add qdev_proxy_add to create proxy devices
  multi-process: remote: add setup_devices and setup_drive msg
    processing
  multi-process: remote: use fd for socket from parent process
  multi-process: remote: add create_done condition
  multi-process: add processing of remote drive and device command line
  multi-process: refractor vl.c code to re-use in remote
  multi-process: add remote option
  multi-process: add remote options parser
  multi-process: add parse_cmdline in remote process
  multi-process: add support for multiple devices
  multi-process: add heartbeat timer and signal handler
  multi-process: handle heartbeat messages in remote process
  multi-process: add configure and usage information

Jagannathan Raman (24):
  multi-process: memory: alloc RAM from file at offset
  multi-process: util: Add qemu_thread_cancel() to cancel running thread
  multi-process: Add stub functions to facilate build of multi-process
  multi-process: Add config option for multi-process QEMU
  multi-process: build system for remote device process
  multi-process: define proxy-link object
  multi-process: add functions to synchronize proxy and remote endpoints
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: setup memory manager for remote device
  multi-process: remote process initialization
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Add LSI device proxy object
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Introduce build flags to separate remote process code
  multi-process: Use separate MMIO communication channel
  multi-process: perform device reset in the remote process
  multi-process/mon: stub functions to enable QMP module for remote
    process
  multi-process/mon: build system for QMP module in remote process
  multi-process/mon: Refactor monitor/chardev functions out of vl.c
  multi-process/mon: trim HMP command set for remote storage processes
  multi-process/mon: Initialize QMP module for remote processes
  multi-process: prevent duplicate memory initialization in remote

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 Makefile                            |   2 +
 Makefile.objs                       |  33 ++
 Makefile.target                     |  91 +++++-
 accel/stubs/kvm-stub.c              |   5 +
 accel/stubs/tcg-stub.c              |  95 ++++++
 backends/Makefile.objs              |   2 +
 block/Makefile.objs                 |   2 +
 configure                           |  15 +
 docs/devel/qemu-multiprocess.txt    | 627 ++++++++++++++++++++++++++++++++++++
 docs/qemu-multiprocess.txt          |  86 +++++
 exec.c                              |  14 +-
 hmp-scsi-commands-info.hx           | 167 ++++++++++
 hmp-scsi-commands.hx                | 384 ++++++++++++++++++++++
 hw/Makefile.objs                    |   9 +
 hw/block/Makefile.objs              |   2 +
 hw/core/Makefile.objs               |  16 +
 hw/nvram/Makefile.objs              |   2 +
 hw/pci/Makefile.objs                |   4 +
 hw/proxy/Makefile.objs              |   1 +
 hw/proxy/memory-sync.c              | 226 +++++++++++++
 hw/proxy/proxy-lsi53c895a.c         | 168 ++++++++++
 hw/proxy/qemu-proxy.c               | 615 +++++++++++++++++++++++++++++++++++
 hw/scsi/Makefile.objs               |   2 +
 include/exec/address-spaces.h       |   2 +
 include/exec/ram_addr.h             |   2 +-
 include/glib-compat.h               |   4 +
 include/hw/pci/pci_ids.h            |   3 +
 include/hw/proxy/memory-sync.h      |  51 +++
 include/hw/proxy/proxy-lsi53c895a.h |  42 +++
 include/hw/proxy/qemu-proxy.h       | 106 ++++++
 include/hw/qdev-core.h              |   2 +
 include/io/proxy-link.h             | 202 ++++++++++++
 include/monitor/qdev.h              |  24 ++
 include/qemu-common.h               |   8 +
 include/qemu/log.h                  |   1 +
 include/qemu/mmap-alloc.h           |   3 +-
 include/qemu/thread.h               |   1 +
 include/remote/iohub.h              |  63 ++++
 include/remote/machine.h            |  48 +++
 include/remote/memory.h             |  34 ++
 include/remote/pcihost.h            |  59 ++++
 io/Makefile.objs                    |   2 +
 io/proxy-link.c                     | 363 +++++++++++++++++++++
 memory.c                            |   2 +-
 migration/Makefile.objs             |   2 +
 monitor/Makefile.objs               |   3 +
 monitor/misc.c                      |  84 ++---
 monitor/monitor-internal.h          |  38 +++
 qapi/Makefile.objs                  |   2 +
 qdev-monitor.c                      | 257 ++++++++++++++-
 qemu-options.hx                     |  21 ++
 qom/Makefile.objs                   |   5 +
 remote/Makefile.objs                |   6 +
 remote/iohub.c                      | 159 +++++++++
 remote/machine.c                    | 133 ++++++++
 remote/memory.c                     |  99 ++++++
 remote/pcihost.c                    |  84 +++++
 remote/remote-main.c                | 626 +++++++++++++++++++++++++++++++++++
 remote/remote-opts.c                | 123 +++++++
 remote/remote-opts.h                |  31 ++
 rules.mak                           |   2 +-
 stubs/gdbstub.c                     |  21 ++
 stubs/machine-init-done.c           |   4 +
 stubs/migration.c                   | 162 ++++++++++
 stubs/monitor.c                     |  65 ++++
 stubs/net-stub.c                    | 100 ++++++
 stubs/qapi-misc.c                   |  41 +++
 stubs/qapi-target.c                 |  49 +++
 stubs/replay.c                      |  18 ++
 stubs/ui-stub.c                     | 130 ++++++++
 stubs/vl-stub.c                     | 169 ++++++++++
 stubs/vmstate.c                     |  20 ++
 stubs/xen-mapcache.c                |  22 ++
 ui/Makefile.objs                    |   2 +
 util/log.c                          |   2 +
 util/mmap-alloc.c                   |   7 +-
 util/oslib-posix.c                  |   2 +-
 util/qemu-thread-posix.c            |  10 +
 vl-parse.c                          | 249 ++++++++++++++
 vl.c                                | 298 ++++++++---------
 vl.h                                |  58 ++++
 81 files changed, 6466 insertions(+), 228 deletions(-)
 create mode 100644 docs/devel/qemu-multiprocess.txt
 create mode 100644 docs/qemu-multiprocess.txt
 create mode 100644 hmp-scsi-commands-info.hx
 create mode 100644 hmp-scsi-commands.hx
 create mode 100644 hw/proxy/Makefile.objs
 create mode 100644 hw/proxy/memory-sync.c
 create mode 100644 hw/proxy/proxy-lsi53c895a.c
 create mode 100644 hw/proxy/qemu-proxy.c
 create mode 100644 include/hw/proxy/memory-sync.h
 create mode 100644 include/hw/proxy/proxy-lsi53c895a.h
 create mode 100644 include/hw/proxy/qemu-proxy.h
 create mode 100644 include/io/proxy-link.h
 create mode 100644 include/remote/iohub.h
 create mode 100644 include/remote/machine.h
 create mode 100644 include/remote/memory.h
 create mode 100644 include/remote/pcihost.h
 create mode 100644 io/proxy-link.c
 create mode 100644 remote/Makefile.objs
 create mode 100644 remote/iohub.c
 create mode 100644 remote/machine.c
 create mode 100644 remote/memory.c
 create mode 100644 remote/pcihost.c
 create mode 100644 remote/remote-main.c
 create mode 100644 remote/remote-opts.c
 create mode 100644 remote/remote-opts.h
 create mode 100644 stubs/migration.c
 create mode 100644 stubs/net-stub.c
 create mode 100644 stubs/qapi-misc.c
 create mode 100644 stubs/qapi-target.c
 create mode 100644 stubs/ui-stub.c
 create mode 100644 stubs/vl-stub.c
 create mode 100644 stubs/xen-mapcache.c
 create mode 100644 vl-parse.c
 create mode 100644 vl.h

-- 
1.8.3.1