[Qemu-devel] [PATCH FYI 00/46] Generic TLS support across VNC/chardev/migration

["Daniel P. Berrange" <berrange@redhat.com>]

Earlier in the year I posted an RFC series providing impl of
generic TLS support for VNC and chardevs.

  https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg02038.html

Since that time the initial QOM enhancements and crypto API
consolidation patches from that RFC have been merged. I
currently have a set of patches extracting the VNC server
TLS support into a generic crypto API for TLS that is under
active review, hopefully getting close to merge (v6):

  https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg00437.html

What I'm posting here is the 46 remaining patches I have that
introduce TLS support across chardevs and migration code, and in
the process standardize I/O channel handling across VNC, chardevs
and migration. I'm doing this so people can see the use context
of the TLS patches I have on review, as well as show the kind of
changes I'm making to the migration, chardev & sockets code to
facilitate conversations with other devs working on the same bits
of code. As compared to the original RFC patches, this series is
a more complete job - there is much more unit testing added and
win32 compatibility verified, and many todos/hacks cleaned up.

I still need to figure out NBD integration, which is harder than
I first thought, due to the need for me to understand the special
needs of the block AIO framework, which doesn't seem to use the
normal APIs for registering I/O callbacks with the main event loop.

I also realize that the ACL code needs to be QOM'ified to make
it possible to create/delete ACLs more generically and share then
across services.

I consider pretty much all of these patches as finished and ready
for review, but this series is a bit large to be reviewed in one
go, so I'm going to continue drip-feed it in reasonable sized chunks.

Some of these 46 patches are fairly generic and can be easily posted
in isolation for quicker merge, so I will be focusing on identifying
such patches and posting them individually to appropriate maintainers.

This series needs to be applied on top of the patch series here

  https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg00437.html

For convience I have pushed this entire branch to github

  https://github.com/berrange/qemu/commits/qemu-io-channel-18

Daniel P. Berrange (46):
  sockets: add helpers for creating SocketAddress from a socket
  sockets: move qapi_copy_SocketAddress into qemu-sockets.c
  sockets: allow port to be NULL when listening on IP address
  osdep: add qemu_fork() wrapper for safely handling signals
  coroutine: move into libqemuutil.a library
  io: add abstract QIOChannel classes
  io: add helper module for creating watches on FDs
  io: pull Buffer code out of VNC module
  io: add QIOTask class for async operations
  io: add QIOChannelSocket class
  io: add QIOChannelFile class
  io: add QIOChannelTLS class
  io: add QIOChannelWebsock class
  io: add QIOChannelCommand class
  ui: convert VNC startup code to use SocketAddress
  ui: convert VNC server to use QIOChannelSocket
  ui: convert VNC server to use QIOChannelTLS
  ui: convert VNC server to use QIOChannelWebsock
  char: remove fixed length filename allocation
  char: convert from GIOChannel to QIOChannel
  char: don't assume telnet initialization will not block
  char: introduce support for TLS encrypted TCP chardev backend
  nbd: convert to use the QAPI SocketAddress object
  qemu-nbd: convert to use the QAPI SocketAddress object
  sockets: remove use of QemuOpts from header file
  sockets: remove use of QemuOpts from socket_listen
  sockets: remove use of QemuOpts from socket_connect
  sockets: remove use of QemuOpts from socket_dgram
  migration: remove use of qemu_bufopen from vmstate tests
  migration: remove memory buffer based QEMUFile backend
  migration: move definition of struct QEMUFile back into qemu-file.c
  migration: split migration hooks out of QEMUFileOps
  migration: ensure qemu_fflush() always writes full data amount
  migration: introduce qemu_fset_blocking function on QEMUFile
  migration: force QEMUFile to blocking mode for outgoing migration
  migration: introduce a new QEMUFile impl based on QIOChannel
  migration: convert unix socket protocol to use QIOChannel
  migration: convert tcp socket protocol to use QIOChannel
  migration: convert fd socket protocol to use QIOChannel
  migration: convert exec socket protocol to use QIOChannel
  migration: convert RDMA to use QIOChannel interface
  migration: convert savevm to use QIOChannel for writing to files
  migration: delete QEMUFile sockets implementation
  migration: delete QEMUFile stdio implementation
  migration: support TLS encryption with TCP migration backend
  migration: remove support for non-iovec based write handlers

 MAINTAINERS                             |   7 +
 Makefile                                |   2 +
 Makefile.objs                           |  12 +-
 Makefile.target                         |   2 +
 block.c                                 |   2 +-
 block/nbd.c                             |  66 +--
 block/qcow2.h                           |   2 +-
 block/vdi.c                             |   2 +-
 block/write-threshold.c                 |   2 +-
 blockjob.c                              |   2 +-
 configure                               |  11 +
 coroutine-gthread.c                     |   2 +-
 coroutine-sigaltstack.c                 |   2 +-
 coroutine-ucontext.c                    |   2 +-
 coroutine-win32.c                       |   2 +-
 hw/9pfs/codir.c                         |   2 +-
 hw/9pfs/cofile.c                        |   2 +-
 hw/9pfs/cofs.c                          |   2 +-
 hw/9pfs/coxattr.c                       |   2 +-
 hw/9pfs/virtio-9p-coth.c                |   2 +-
 hw/9pfs/virtio-9p-coth.h                |   2 +-
 hw/9pfs/virtio-9p.h                     |   2 +-
 include/block/block.h                   |   2 +-
 include/block/block_int.h               |   2 +-
 include/io/buffer.h                     | 118 ++++
 include/io/channel-command.h            |  91 +++
 include/io/channel-file.h               |  93 +++
 include/io/channel-socket.h             | 297 ++++++++++
 include/io/channel-tls.h                | 142 +++++
 include/io/channel-watch.h              |  72 +++
 include/io/channel-websock.h            | 108 ++++
 include/io/channel.h                    | 374 +++++++++++++
 include/io/task.h                       | 256 +++++++++
 include/migration/qemu-file.h           |  48 +-
 include/{block => qemu}/coroutine.h     |   0
 include/{block => qemu}/coroutine_int.h |   2 +-
 include/qemu/osdep.h                    |  16 +
 include/qemu/sockets.h                  |  44 +-
 io/Makefile.objs                        |   9 +
 io/buffer.c                             |  65 +++
 io/channel-command.c                    | 370 ++++++++++++
 io/channel-file.c                       | 209 +++++++
 io/channel-socket.c                     | 718 ++++++++++++++++++++++++
 io/channel-tls.c                        | 381 +++++++++++++
 io/channel-watch.c                      | 200 +++++++
 io/channel-websock.c                    | 965 ++++++++++++++++++++++++++++++++
 io/channel.c                            | 229 ++++++++
 io/task.c                               | 150 +++++
 migration/Makefile.objs                 |   6 +-
 migration/exec.c                        |  48 +-
 migration/fd.c                          |  57 +-
 migration/migration.c                   |   9 +-
 migration/qemu-file-buf.c               | 462 ---------------
 migration/qemu-file-channel.c           | 200 +++++++
 migration/qemu-file-internal.h          |  53 --
 migration/qemu-file-stdio.c             | 194 -------
 migration/qemu-file-unix.c              | 238 --------
 migration/qemu-file.c                   | 113 ++--
 migration/rdma.c                        | 252 ++++++---
 migration/savevm.c                      |  16 +-
 migration/tcp.c                         | 372 ++++++++++--
 migration/unix.c                        | 103 ++--
 nbd.c                                   |   2 +-
 qapi-schema.json                        |   2 +
 qemu-char.c                             | 938 +++++++++++++++----------------
 qemu-coroutine-io.c                     |   2 +-
 qemu-coroutine-lock.c                   |   4 +-
 qemu-coroutine-sleep.c                  |   2 +-
 qemu-coroutine.c                        |   4 +-
 qemu-nbd.c                              | 102 ++--
 qemu-options.hx                         |  16 +-
 scripts/create_config                   |   9 +
 tests/.gitignore                        |   7 +
 tests/Makefile                          |  22 +-
 tests/io-channel-helpers.c              | 247 ++++++++
 tests/io-channel-helpers.h              |  33 ++
 tests/test-coroutine.c                  |   4 +-
 tests/test-io-channel-command.c         | 121 ++++
 tests/test-io-channel-file.c            |  89 +++
 tests/test-io-channel-socket.c          | 349 ++++++++++++
 tests/test-io-channel-tls.c             | 335 +++++++++++
 tests/test-io-task.c                    | 276 +++++++++
 tests/test-vmstate.c                    |  57 +-
 thread-pool.c                           |   2 +-
 ui/vnc-auth-sasl.c                      |  87 ++-
 ui/vnc-auth-vencrypt.c                  |  93 ++-
 ui/vnc-enc-tight.c                      |  38 +-
 ui/vnc-enc-zlib.c                       |   6 +-
 ui/vnc-enc-zrle.c                       |  18 +-
 ui/vnc-jobs.c                           |  25 +-
 ui/vnc-ws.c                             | 400 +++----------
 ui/vnc-ws.h                             |  71 +--
 ui/vnc.c                                | 799 ++++++++++++--------------
 ui/vnc.h                                |  77 +--
 util/oslib-posix.c                      |  71 +++
 util/oslib-win32.c                      |   9 +
 util/qemu-sockets.c                     | 427 ++++++++------
 97 files changed, 8875 insertions(+), 3085 deletions(-)
 create mode 100644 include/io/buffer.h
 create mode 100644 include/io/channel-command.h
 create mode 100644 include/io/channel-file.h
 create mode 100644 include/io/channel-socket.h
 create mode 100644 include/io/channel-tls.h
 create mode 100644 include/io/channel-watch.h
 create mode 100644 include/io/channel-websock.h
 create mode 100644 include/io/channel.h
 create mode 100644 include/io/task.h
 rename include/{block => qemu}/coroutine.h (100%)
 rename include/{block => qemu}/coroutine_int.h (98%)
 create mode 100644 io/Makefile.objs
 create mode 100644 io/buffer.c
 create mode 100644 io/channel-command.c
 create mode 100644 io/channel-file.c
 create mode 100644 io/channel-socket.c
 create mode 100644 io/channel-tls.c
 create mode 100644 io/channel-watch.c
 create mode 100644 io/channel-websock.c
 create mode 100644 io/channel.c
 create mode 100644 io/task.c
 delete mode 100644 migration/qemu-file-buf.c
 create mode 100644 migration/qemu-file-channel.c
 delete mode 100644 migration/qemu-file-internal.h
 delete mode 100644 migration/qemu-file-stdio.c
 delete mode 100644 migration/qemu-file-unix.c
 create mode 100644 tests/io-channel-helpers.c
 create mode 100644 tests/io-channel-helpers.h
 create mode 100644 tests/test-io-channel-command.c
 create mode 100644 tests/test-io-channel-file.c
 create mode 100644 tests/test-io-channel-socket.c
 create mode 100644 tests/test-io-channel-tls.c
 create mode 100644 tests/test-io-task.c

-- 
2.4.3