Re: [Qemu-devel] [RFC 1/5] s390x/ap-matrix: Adjunct Processor (AP) matrix object model

Re: [Qemu-devel] [RFC 1/5] s390x/ap-matrix: Adjunct Processor (AP) matrix object model

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:50 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> This patch introduces the base object model for an AP matrix device. An AP
> matrix is comprised of the AP adapters, usage domains and control domains
> assigned to a KVM guest. The matrix is represented in three bit masks:
> 
> * The AP Mask (APM) specifies the AP adapters assigned to the
>   KVM guest. The APM controls which adapters are valid for the KVM guest.
>   The bits in the mask, from left to right, correspond to APIDs
>   0 up to the number of adapters that can be assigned to the LPAR. If a bit
>   is set, the corresponding adapter is valid for use by the KVM guest.
> 
> * The AP Queue Mask (AQM) specifies the AP usage domains assigned
>   to the KVM guest. The bits in the mask, from left to right, correspond
>   to the usage domains, from 0 up to the number of domains that can be
>   assigned to the LPAR. If a bit is set, the corresponding usage domain is
>   valid for use by the KVM guest.
> 
> * The AP Domain Mask specifies the AP control domains assigned to the
>   KVM guest. The ADM bitmask controls which domains can be changed by an AP
>   command-request message sent to a usage domain from the guest. The bits in
>   the mask, from left to right, correspond to domain 0 up to the number of
>   domains that can be assigned to the LPAR. If a bit is set, the
>   corresponding domain can be modified by an AP command-request message
>   sent to a usage domain configured for the KVM guest.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  hw/s390x/Makefile.objs            |    2 +
>  hw/s390x/ap-matrix-device.c       |   33 +++++++++++++++++++++++
>  hw/s390x/ap-matrix-device.h       |   53 +++++++++++++++++++++++++++++++++++++
>  hw/s390x/s390-ap-matrix.c         |   52 ++++++++++++++++++++++++++++++++++++
>  include/hw/s390x/s390-ap-matrix.h |   39 +++++++++++++++++++++++++++
>  5 files changed, 179 insertions(+), 0 deletions(-)
>  create mode 100644 hw/s390x/ap-matrix-device.c
>  create mode 100644 hw/s390x/ap-matrix-device.h
>  create mode 100644 hw/s390x/s390-ap-matrix.c
>  create mode 100644 include/hw/s390x/s390-ap-matrix.h
> 

> diff --git a/hw/s390x/ap-matrix-device.c b/hw/s390x/ap-matrix-device.c
> new file mode 100644
> index 0000000..d036ce2
> --- /dev/null
> +++ b/hw/s390x/ap-matrix-device.c
> @@ -0,0 +1,33 @@
> +/*
> + * Common device infrastructure for AP matrix devices
> + *
> + * Copyright 2016 IBM Corp.

Hm?             ^^^^

> + * Author(s): Jing Liu <liujbjl@linux.vnet.ibm.com>

If Jing is the author, shouldn't the patch carry her s-o-b as well?

(I'm a bit confused about the history of this.)

> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +#include <ctype.h>
> +
> +#include "qemu/osdep.h"
> +#include "qemu/bitops.h"
> +#include "qemu/module.h"
> +#include "qapi/error.h"
> +#include "qapi/visitor.h"
> +#include "ap-matrix-device.h"
> +
> +static const TypeInfo ap_matrix_device_info = {
> +    .name = TYPE_AP_MATRIX_DEVICE,
> +    .parent = TYPE_DEVICE,
> +    .instance_size = sizeof(APMatrixDevice),
> +    .class_size = sizeof(APMatrixDeviceClass),
> +    .abstract = true,
> +};
> +
> +static void ap_matrix_device_register(void)
> +{
> +    type_register_static(&ap_matrix_device_info);
> +}
> +
> +type_init(ap_matrix_device_register)
> diff --git a/hw/s390x/ap-matrix-device.h b/hw/s390x/ap-matrix-device.h
> new file mode 100644
> index 0000000..af7ae2c
> --- /dev/null
> +++ b/hw/s390x/ap-matrix-device.h
> @@ -0,0 +1,53 @@
> +/*
> + * Adjunct Processor (AP) matrix device structures
> + *
> + * Copyright 2016 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#ifndef HW_S390X_AP_MATRIX_DEVICE_H
> +#define HW_S390X_AP_MATRIX_DEVICE_H
> +#include "qom/object.h"
> +#include "hw/qdev.h"
> +
> +#define AP_MATRIX_MASK_INDICES 4
> +#define AP_MATRIX_MAX_MASK_BITS 256
> +
> +typedef struct APMatrixMasks {
> +    uint64_t apm[AP_MATRIX_MASK_INDICES];
> +    uint64_t aqm[AP_MATRIX_MASK_INDICES];
> +    uint64_t adm[AP_MATRIX_MASK_INDICES];
> +} APMatrixMasks;
> +
> +typedef struct APMatrixDevice {
> +    DeviceState parent_obj;
> +    APMatrixMasks masks;
> +} APMatrixDevice;
> +
> +extern const VMStateDescription vmstate_ap_matrix_dev;
> +#define VMSTATE_AP_MATRIX_DEVICE(_field, _state)                     \
> +    VMSTATE_STRUCT(_field, _state, 1, vmstate_ap_matrix_dev, APMatrixDevice)

I'm wondering about migration: can the other side easily reconstruct
the state on the target?

> +
> +typedef struct APMatrixDeviceClass {
> +    DeviceClass parent_class;
> +} APMatrixDeviceClass;
> +
> +static inline APMatrixDevice *to_ap_matrix_dev(DeviceState *dev)
> +{
> +    return container_of(dev, APMatrixDevice, parent_obj);
> +}
> +
> +#define TYPE_AP_MATRIX_DEVICE "ap-matrix-device"
> +
> +#define AP_MATRIX_DEVICE(obj) \
> +    OBJECT_CHECK(APMatrixDevice, (obj), TYPE_AP_MATRIX_DEVICE)
> +#define AP_MATRIX_DEVICE_GET_CLASS(obj) \
> +    OBJECT_GET_CLASS(AP_MATRIXDeviceClass, (obj), TYPE_AP_MATRIX_DEVICE)
> +#define AP_MATRIX_DEVICE_CLASS(klass) \
> +    OBJECT_CLASS_CHECK(AP_MATRIXDeviceClass, (klass), TYPE_AP_MATRIX_DEVICE)
> +
> +#endif
> diff --git a/hw/s390x/s390-ap-matrix.c b/hw/s390x/s390-ap-matrix.c
> new file mode 100644
> index 0000000..6d2e234
> --- /dev/null
> +++ b/hw/s390x/s390-ap-matrix.c
> @@ -0,0 +1,52 @@
> +/*
> + * s390 AP Matrix Assignment Support
> + *
> + * Copyright 2017 IBM Corp
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2
> + * or (at your option) any later version. See the COPYING file in the
> + * top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "hw/sysbus.h"
> +#include "libgen.h"
> +#include "hw/s390x/s390-ap-matrix.h"
> +
> +static void s390_ap_matrix_device_realize(S390APMatrixDevice *sapmdev,
> +                                          APMatrixMasks *masks,
> +                                          Error **errp)
> +{
> +    size_t i;
> +    APMatrixDevice *apmdev = AP_MATRIX_DEVICE(sapmdev);
> +
> +    for (i = 0; i < AP_MATRIX_MASK_INDICES; i++) {
> +        apmdev->masks.apm[i] = masks->apm[i];
> +        apmdev->masks.aqm[i] = masks->aqm[i];
> +        apmdev->masks.adm[i] = masks->adm[i];

These masks are presumably provided in a later patch?

> +    }
> +}
> +
> +static void s390_ap_matrix_class_init(ObjectClass *klass, void *data)
> +{
> +    S390APMatrixDeviceClass *apmc = S390_AP_MATRIX_DEVICE_CLASS(klass);
> +
> +    apmc->realize = s390_ap_matrix_device_realize;
> +}
> +
> +static const TypeInfo s390_ap_matrix_info = {
> +    .name          = TYPE_S390_AP_MATRIX_DEVICE,
> +    .parent        = TYPE_AP_MATRIX_DEVICE,
> +    .instance_size = sizeof(S390APMatrixDevice),
> +    .class_size    = sizeof(S390APMatrixDeviceClass),
> +    .class_init    = s390_ap_matrix_class_init,
> +    .abstract      = true,
> +};
> +
> +static void register_s390_ap_matrix_type(void)
> +{
> +    type_register_static(&s390_ap_matrix_info);
> +}
> +
> +type_init(register_s390_ap_matrix_type)
> diff --git a/include/hw/s390x/s390-ap-matrix.h b/include/hw/s390x/s390-ap-matrix.h
> new file mode 100644
> index 0000000..b088841
> --- /dev/null
> +++ b/include/hw/s390x/s390-ap-matrix.h
> @@ -0,0 +1,39 @@
> +/*
> + * s390 AP Matrix Assignment Support
> + *
> + * Copyright 2017 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or (at
> + * your option) any later version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#ifndef HW_S390_AP_MATRIX_H
> +#define HW_S390_AP_MATRIX_H
> +
> +#include "hw/s390x/ap-matrix-device.h"
> +
> +#define AP_MATRIX_ADAPTERS_PROP_NAME    "adapters"
> +#define AP_MATRIX_DOMAINS_PROP_NAME     "domains"
> +#define AP_MATRIX_CONTROLS_PROP_NAME    "controls"

Also used in a later patch?

> +#define TYPE_S390_AP_MATRIX_DEVICE "s390-ap-matrix"
> +#define S390_AP_MATRIX_DEVICE(obj) \
> +    OBJECT_CHECK(S390APMatrixDevice, (obj), TYPE_S390_AP_MATRIX_DEVICE)
> +#define S390_AP_MATRIX_DEVICE_CLASS(klass) \
> +    OBJECT_CLASS_CHECK(S390APMatrixDeviceClass, (klass), \
> +                       TYPE_S390_AP_MATRIX_DEVICE)
> +#define S390_AP_MATRIX_DEVICE_GET_CLASS(obj) \
> +    OBJECT_GET_CLASS(S390APMatrixDeviceClass, (obj), TYPE_S390_AP_MATRIX_DEVICE)
> +
> +typedef struct S390APMatrixDevice {
> +    APMatrixDevice parent_obj;
> +} S390APMatrixDevice;
> +
> +typedef struct S390APMatrixDeviceClass {
> +    APMatrixDeviceClass parent_class;
> +    void (*realize)(S390APMatrixDevice *sapmdev, APMatrixMasks *masks,
> +                    Error **errp);
> +} S390APMatrixDeviceClass;
> +
> +#endif

Re: [Qemu-devel] [RFC 2/5] s390x/vfio: ap-matrix: Introduce VFIO AP Matrix device

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:51 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Introduces a VFIO based AP matrix device. This device will establish
> a communication pathway to the VFIO AP Matrix kernel device driver
> via a mediated AP matrix device file descriptor. This communication pathway
> will be used to:
> 
> 1. Signal the VFIO AP matrix device driver via the VFIO_AP_MATRIX_CONFIGURE
>    ioctl to configure the AP matrix for the KVM guest. The device driver
>    will set the bits in the APM, AQM and ADM fields of the CRYCB
>    referenced by the KVM guest's SIE state description according to
>    the AP matrix configuration specified for the mediated AP matrix
>    device's sysfs attribute files. The AP matrix configuration will be
>    returned to the guest from the ioctl call to notify the KVM guest
>    about the AP resources to which the guest has access.
> 
> 2. Pass intercepted AP instructions to the VFIO AP Matrix driver
>    for execution on an AP adapter assigned to the LPAR in which the
>    linux host is running.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  default-configs/s390x-softmmu.mak |    1 +
>  hw/vfio/Makefile.objs             |    1 +
>  hw/vfio/ap-matrix.c               |  179 +++++++++++++++++++++++++++++++++++++
>  include/hw/vfio/vfio-common.h     |    1 +
>  linux-headers/linux/vfio.h        |   28 +++++-
>  5 files changed, 207 insertions(+), 3 deletions(-)
>  create mode 100644 hw/vfio/ap-matrix.c
> 
> diff --git a/default-configs/s390x-softmmu.mak b/default-configs/s390x-softmmu.mak
> index 444bf16..1fe53ea 100644
> --- a/default-configs/s390x-softmmu.mak
> +++ b/default-configs/s390x-softmmu.mak
> @@ -8,3 +8,4 @@ CONFIG_S390_FLIC=y
>  CONFIG_S390_FLIC_KVM=$(CONFIG_KVM)
>  CONFIG_VFIO_CCW=$(CONFIG_LINUX)
>  CONFIG_WDT_DIAG288=y
> +CONFIG_VFIO_AP_MATRIX=$(CONFIG_LINUX)
> diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs
> index c3ab909..47e482f 100644
> --- a/hw/vfio/Makefile.objs
> +++ b/hw/vfio/Makefile.objs
> @@ -6,4 +6,5 @@ obj-$(CONFIG_SOFTMMU) += platform.o
>  obj-$(CONFIG_VFIO_XGMAC) += calxeda-xgmac.o
>  obj-$(CONFIG_VFIO_AMD_XGBE) += amd-xgbe.o
>  obj-$(CONFIG_SOFTMMU) += spapr.o
> +obj-$(CONFIG_VFIO_AP_MATRIX) += ap-matrix.o
>  endif
> diff --git a/hw/vfio/ap-matrix.c b/hw/vfio/ap-matrix.c
> new file mode 100644
> index 0000000..eeaa439
> --- /dev/null
> +++ b/hw/vfio/ap-matrix.c
> @@ -0,0 +1,179 @@
> +/*
> + * VFIO based AP matrix device assignment
> + *
> + * Copyright 2017 IBM Corp.
> + * Author(s): Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or(at
> + * your option) any version. See the COPYING file in the top-level
> + * directory.
> + */
> +
> +#include <linux/vfio.h>
> +#include <sys/ioctl.h>
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "hw/sysbus.h"
> +#include "hw/vfio/vfio.h"
> +#include "hw/vfio/vfio-common.h"
> +#include "hw/s390x/s390-ap-matrix.h"
> +#include "hw/s390x/ap-matrix-device.h"
> +#include "qemu/error-report.h"
> +
> +#define TYPE_VFIO_AP_MATRIX_DEVICE      "vfio-ap-matrix"
> +#define AP_MATRIX_SYSFSDEV_PROP_NAME    "sysfsdev"
> +
> +typedef struct VFIOAPMatrixDevice {
> +    S390APMatrixDevice apmdev;
> +    VFIODevice vdev;
> +} VFIOAPMatrixDevice;
> +
> +static void vfio_ap_matrix_compute_needs_reset(VFIODevice *vdev)
> +{
> +    vdev->needs_reset = false;

This does not look like very much computing :)

> +}
> +
> +/*
> + * We don't need vfio_hot_reset_multi and vfio_eoi operations for
> + * vfio-ap-matrix device now.
> + */
> +struct VFIODeviceOps vfio_ap_matrix_ops = {
> +    .vfio_compute_needs_reset = vfio_ap_matrix_compute_needs_reset,
> +};
> +
> +static void vfio_ap_matrix_reset(DeviceState *dev)
> +{
> +    APMatrixDevice *apmdev = DO_UPCAST(APMatrixDevice, parent_obj, dev);
> +    S390APMatrixDevice *sapmdev = DO_UPCAST(S390APMatrixDevice, parent_obj,
> +                                            apmdev);
> +    VFIOAPMatrixDevice *vapmdev = DO_UPCAST(VFIOAPMatrixDevice, apmdev,
> +                                            sapmdev);
> +
> +    ioctl(vapmdev->vdev.fd, VFIO_DEVICE_RESET);
> +}
> +
> +static void vfio_put_device(VFIOAPMatrixDevice *apmdev)
> +{
> +    g_free(apmdev->vdev.name);
> +    vfio_put_base_device(&apmdev->vdev);
> +}
> +
> +static VFIOGroup *vfio_ap_matrix_get_group(VFIOAPMatrixDevice *vapmdev,
> +                                           Error **errp)
> +{
> +    char *tmp, group_path[PATH_MAX];
> +    ssize_t len;
> +    int groupid;
> +
> +    tmp = g_strdup_printf("%s/iommu_group", vapmdev->vdev.sysfsdev);
> +    len = readlink(tmp, group_path, sizeof(group_path));
> +    g_free(tmp);
> +
> +    if (len <= 0 || len >= sizeof(group_path)) {
> +        error_setg(errp, "%s: no iommu_group found for %s",
> +                   TYPE_VFIO_AP_MATRIX_DEVICE, vapmdev->vdev.sysfsdev);
> +        return NULL;
> +    }
> +
> +    group_path[len] = 0;
> +
> +    if (sscanf(basename(group_path), "%d", &groupid) != 1) {
> +        error_setg(errp, "vfio: failed to read %s", group_path);
> +        return NULL;
> +    }
> +
> +    return vfio_get_group(groupid, &address_space_memory, errp);
> +}
> +
> +static void vfio_ap_matrix_realize(DeviceState *dev, Error **errp)
> +{
> +    VFIODevice *vbasedev;
> +    VFIOGroup *vfio_group;
> +    APMatrixDevice *apmdev = DO_UPCAST(APMatrixDevice, parent_obj, dev);
> +    S390APMatrixDevice *sapmdev = DO_UPCAST(S390APMatrixDevice, parent_obj,
> +                                            apmdev);
> +    VFIOAPMatrixDevice *vapmdev = DO_UPCAST(VFIOAPMatrixDevice, apmdev,
> +                                            sapmdev);
> +    char *mdevid;
> +    Error *local_err = NULL;
> +
> +    vfio_group = vfio_ap_matrix_get_group(vapmdev, &local_err);
> +    if (!vfio_group) {
> +        goto out_err;
> +    }
> +
> +    vapmdev->vdev.ops = &vfio_ap_matrix_ops;
> +    vapmdev->vdev.type = VFIO_DEVICE_TYPE_AP_MATRIX;
> +    mdevid = basename(vapmdev->vdev.sysfsdev);
> +    vapmdev->vdev.name = g_strdup_printf("%s-%s", TYPE_VFIO_AP_MATRIX_DEVICE,
> +                                         mdevid);
> +    vapmdev->vdev.dev = dev;
> +    QLIST_FOREACH(vbasedev, &vfio_group->device_list, next) {
> +        if (strcmp(vbasedev->name, vapmdev->vdev.name) == 0) {
> +            error_setg(&local_err,
> +                       "%s: AP matrix device %s has already been realized",
> +                       TYPE_VFIO_AP_MATRIX_DEVICE, vapmdev->vdev.name);
> +            goto out_device_err;
> +        }
> +    }
> +
> +    if (vfio_get_device(vfio_group, mdevid, &vapmdev->vdev, &local_err)) {
> +        goto out_device_err;
> +    }
> +
> +    return;
> +
> +out_device_err:
> +    vfio_put_group(vfio_group);
> +out_err:
> +    error_propagate(errp, local_err);
> +}
> +
> +static void vfio_ap_matrix_unrealize(DeviceState *dev, Error **errp)
> +{
> +    APMatrixDevice *apm_dev = DO_UPCAST(APMatrixDevice, parent_obj, dev);
> +    S390APMatrixDevice *apmdev = DO_UPCAST(S390APMatrixDevice, parent_obj,
> +                                           apm_dev);
> +    VFIOAPMatrixDevice *vapmdev = DO_UPCAST(VFIOAPMatrixDevice, apmdev, apmdev);
> +    VFIOGroup *group = vapmdev->vdev.group;
> +
> +    vfio_put_device(vapmdev);
> +    vfio_put_group(group);
> +}
> +
> +static Property vfio_ap_matrix_properties[] = {
> +    DEFINE_PROP_STRING(AP_MATRIX_SYSFSDEV_PROP_NAME, VFIOAPMatrixDevice,
> +                       vdev.sysfsdev),
> +    DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static const VMStateDescription vfio_ap_matrix_vmstate = {
> +    .name = TYPE_VFIO_AP_MATRIX_DEVICE,
> +    .unmigratable = 1,
> +};
> +
> +static void vfio_ap_matrix_class_init(ObjectClass *klass, void *data)
> +{
> +    DeviceClass *dc = DEVICE_CLASS(klass);
> +
> +    dc->props = vfio_ap_matrix_properties;
> +    dc->vmsd = &vfio_ap_matrix_vmstate;
> +    dc->desc = "VFIO-based AP matrix assignment";
> +    dc->realize = vfio_ap_matrix_realize;
> +    dc->unrealize = vfio_ap_matrix_unrealize;
> +    dc->reset = vfio_ap_matrix_reset;
> +}
> +
> +static const TypeInfo vfio_ap_matrix_info = {
> +    .name = TYPE_VFIO_AP_MATRIX_DEVICE,
> +    .parent = TYPE_S390_AP_MATRIX_DEVICE,
> +    .instance_size = sizeof(VFIOAPMatrixDevice),
> +    .class_init = vfio_ap_matrix_class_init,
> +};
> +
> +static void register_vfio_ap_matrix_type(void)
> +{
> +    type_register_static(&vfio_ap_matrix_info);
> +}
> +
> +type_init(register_vfio_ap_matrix_type)
> diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h
> index f3a2ac9..7cdc1f3 100644
> --- a/include/hw/vfio/vfio-common.h
> +++ b/include/hw/vfio/vfio-common.h
> @@ -46,6 +46,7 @@ enum {
>      VFIO_DEVICE_TYPE_PCI = 0,
>      VFIO_DEVICE_TYPE_PLATFORM = 1,
>      VFIO_DEVICE_TYPE_CCW = 2,
> +    VFIO_DEVICE_TYPE_AP_MATRIX = 3,
>  };
>  
>  typedef struct VFIOMmap {
> diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
> index 4e7ab4c..2d96c57 100644
> --- a/linux-headers/linux/vfio.h
> +++ b/linux-headers/linux/vfio.h

It would make sense to split this out as a dummy headers update.

> @@ -8,8 +8,8 @@
>   * it under the terms of the GNU General Public License version 2 as
>   * published by the Free Software Foundation.
>   */
> -#ifndef VFIO_H
> -#define VFIO_H
> +#ifndef _UAPIVFIO_H
> +#define _UAPIVFIO_H
>  
>  #include <linux/types.h>
>  #include <linux/ioctl.h>
> @@ -199,6 +199,7 @@ struct vfio_device_info {
>  #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2)	/* vfio-platform device */
>  #define VFIO_DEVICE_FLAGS_AMBA  (1 << 3)	/* vfio-amba device */
>  #define VFIO_DEVICE_FLAGS_CCW	(1 << 4)	/* vfio-ccw device */
> +#define VFIO_DEVICE_FLAGS_AP_MATRIX (1 << 5)	/* vfio-ap-matrix device */
>  	__u32	num_regions;	/* Max region index + 1 */
>  	__u32	num_irqs;	/* Max IRQ index + 1 */
>  };
> @@ -214,6 +215,7 @@ struct vfio_device_info {
>  #define VFIO_DEVICE_API_PLATFORM_STRING		"vfio-platform"
>  #define VFIO_DEVICE_API_AMBA_STRING		"vfio-amba"
>  #define VFIO_DEVICE_API_CCW_STRING		"vfio-ccw"
> +#define VFIO_DEVICE_API_AP_MATRIX_STRING	"vfio-ap-matrix"
>  
>  /**
>   * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
> @@ -714,6 +716,26 @@ struct vfio_iommu_spapr_tce_remove {
>  };
>  #define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 20)
>  
> +/**
> + * VFIO_AP_MATRIX_CONFIGURE		_IO(VFIO_TYPE, VFIO_BASE + 21
> + *
> + * Configure the AP matrix for a KVM guest.
> + */
> +#define VFIO_AP_MATRIX_CONFIGURE	_IO(VFIO_TYPE, VFIO_BASE + 21)
> +
> +#define VFIO_AP_MATRIX_MASK_INDICES	4
> +#define VFIO_AP_MATTRIX_MASK_BYTES	(VFIO_AP_MATRIX_MASK_INDICES * \
> +					 sizeof(__u64))
> +#define VFIO_AP_MATRIX_MASK_BITS	(VFIO_AP_MATTRIX_MASK_BYTES * 8)
> +
> +struct vfio_ap_matrix_config {
> +	__u32 argsz;
> +	__u32 flags;
> +	/* out */
> +	__u64 apm[VFIO_AP_MATRIX_MASK_INDICES];
> +	__u64 aqm[VFIO_AP_MATRIX_MASK_INDICES];
> +	__u64 adm[VFIO_AP_MATRIX_MASK_INDICES];
> +};
>  /* ***************************************************************** */
>  
> -#endif /* VFIO_H */
> +#endif /* _UAPIVFIO_H */

Re: [Qemu-devel] [RFC 3/5] s390x/ap-matrix: Configure AP matrix for KVM guest

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:52 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> The VFIO AP matrix mediated device driver provides an ioctl interface
> to configure the APM, ADM and APM fields contained in the
> CRYCB referenced by the guest's SIE state description. The mask
> values are specified in the mediated AP matrix device's sysfs
> attribute files. Configuring a KVM guest's AP matrix grants direct
> access to the AP adapters, domains and control domains specified in
> the matrix configuration.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  hw/vfio/ap-matrix.c        |   45 ++++++++++++++++++++++++++++++++++++++++++++
>  linux-headers/linux/vfio.h |    1 +
>  2 files changed, 46 insertions(+), 0 deletions(-)
> 
> diff --git a/hw/vfio/ap-matrix.c b/hw/vfio/ap-matrix.c
> index eeaa439..ce87c05 100644
> --- a/hw/vfio/ap-matrix.c
> +++ b/hw/vfio/ap-matrix.c
> @@ -85,6 +85,35 @@ static VFIOGroup *vfio_ap_matrix_get_group(VFIOAPMatrixDevice *vapmdev,
>      return vfio_get_group(groupid, &address_space_memory, errp);
>  }
>  
> +static int vfio_ap_matrix_configure(VFIOAPMatrixDevice *vapmdev,
> +                                    APMatrixMasks *masks, Error **errp)
> +{
> +    size_t i;
> +    struct vfio_ap_matrix_config config;
> +    int ret;
> +
> +    config.argsz = sizeof(config);
> +    config.flags = 0;
> +
> +    ret = ioctl(vapmdev->vdev.fd, VFIO_AP_MATRIX_CONFIGURE, &config);
> +    if (ret) {
> +        error_setg_errno(errp, ret,
> +                         "%s: Error configuring matrix for device %s",
> +                         TYPE_VFIO_AP_MATRIX_DEVICE, vapmdev->vdev.name);
> +        return -EFAULT;
> +    }
> +
> +    for (i = 0; i < VFIO_AP_MATRIX_MASK_INDICES; i++) {
> +        if (i < AP_MATRIX_MASK_INDICES) {
> +            masks->apm[i] = config.apm[i];
> +            masks->aqm[i] = config.aqm[i];
> +            masks->adm[i] = config.adm[i];
> +        }
> +    }
> +
> +    return 0;
> +}
> +
>  static void vfio_ap_matrix_realize(DeviceState *dev, Error **errp)
>  {
>      VFIODevice *vbasedev;
> @@ -92,8 +121,11 @@ static void vfio_ap_matrix_realize(DeviceState *dev, Error **errp)
>      APMatrixDevice *apmdev = DO_UPCAST(APMatrixDevice, parent_obj, dev);
>      S390APMatrixDevice *sapmdev = DO_UPCAST(S390APMatrixDevice, parent_obj,
>                                              apmdev);
> +    S390APMatrixDeviceClass *csapmdev =
> +            S390_AP_MATRIX_DEVICE_GET_CLASS(sapmdev);
>      VFIOAPMatrixDevice *vapmdev = DO_UPCAST(VFIOAPMatrixDevice, apmdev,
>                                              sapmdev);
> +    APMatrixMasks masks;
>      char *mdevid;
>      Error *local_err = NULL;
>  
> @@ -121,8 +153,21 @@ static void vfio_ap_matrix_realize(DeviceState *dev, Error **errp)
>          goto out_device_err;
>      }
>  
> +    if (csapmdev->realize) {
> +        if (vfio_ap_matrix_configure(vapmdev, &masks, &local_err)) {
> +            goto out_realize_err;
> +        }
> +
> +        csapmdev->realize(sapmdev, &masks, &local_err);

Ah, here's the place where you pass in the masks.

> +        if (local_err) {
> +            goto out_realize_err;
> +        }
> +    }
> +
>      return;
>  
> +out_realize_err:
> +    vfio_put_device(vapmdev);
>  out_device_err:
>      vfio_put_group(vfio_group);
>  out_err:
> diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
> index 2d96c57..bf1e25d 100644
> --- a/linux-headers/linux/vfio.h
> +++ b/linux-headers/linux/vfio.h
> @@ -736,6 +736,7 @@ struct vfio_ap_matrix_config {
>  	__u64 aqm[VFIO_AP_MATRIX_MASK_INDICES];
>  	__u64 adm[VFIO_AP_MATRIX_MASK_INDICES];
>  };
> +

Unrelated change.

>  /* ***************************************************************** */
>  
>  #endif /* _UAPIVFIO_H */

Re: [Qemu-devel] [RFC 4/5] s390x/cpumodel: enable AP facilities for guest

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:53 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> Sets up the following STFLE bits to enable the specified AP
> facilities for the guest VM:
> 	* STFLE.12: Enables the AP Query Configuration Information
>                     facility. The AP bus running in the guest uses
>                     the information returned from this instruction
>                     to configure AP adapters and domains for the
>                     guest machine.
> 	* STFLE.15: Enables the AP Special Command facility. The AP
>                     bus running in the guest sets the T bit in
>                     register 0 for the PQAP(TAPQ) instruction when
>                     scanning for AP devices if this facility is
>                     installed.
> 
> These facilities are required in order for the AP bus running on
> the KVM guest to function properly.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  target/s390x/cpu_features.c     |    2 ++
>  target/s390x/cpu_features_def.h |    2 ++
>  target/s390x/gen-features.c     |    4 ++++
>  3 files changed, 8 insertions(+), 0 deletions(-)
> 
> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
> index 31a4676..63f002c 100644
> --- a/target/s390x/cpu_features.c
> +++ b/target/s390x/cpu_features.c
> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>      FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status facility"),
>      FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE facility"),
>      FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, "Configuration-topology facility"),
> +    FEAT_INIT("apqci", S390_FEAT_TYPE_STFL, 12, "Query Adjunct Processor Configuration facility"),
>      FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range facility"),
>      FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing key-setting facility"),
> +    FEAT_INIT("apsc", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor Special Command facility"),

Are there any interdependencies for those feature bits?

>      FEAT_INIT("etf2", S390_FEAT_TYPE_STFL, 16, "Extended-translation facility 2"),
>      FEAT_INIT("msa-base", S390_FEAT_TYPE_STFL, 17, "Message-security-assist facility (excluding subfunctions)"),
>      FEAT_INIT("ldisp", S390_FEAT_TYPE_STFL, 18, "Long-displacement facility"),
> diff --git a/target/s390x/cpu_features_def.h b/target/s390x/cpu_features_def.h
> index 4b6d4e9..4e2589f 100644
> --- a/target/s390x/cpu_features_def.h
> +++ b/target/s390x/cpu_features_def.h
> @@ -27,8 +27,10 @@ typedef enum {
>      S390_FEAT_SENSE_RUNNING_STATUS,
>      S390_FEAT_CONDITIONAL_SSKE,
>      S390_FEAT_CONFIGURATION_TOPOLOGY,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
>      S390_FEAT_IPTE_RANGE,
>      S390_FEAT_NONQ_KEY_SETTING,
> +    S390_FEAT_AP_SPECIAL_CMD,
>      S390_FEAT_EXTENDED_TRANSLATION_2,
>      S390_FEAT_MSA,
>      S390_FEAT_LONG_DISPLACEMENT,
> diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
> index 68e6c31..42914cf 100644
> --- a/target/s390x/gen-features.c
> +++ b/target/s390x/gen-features.c
> @@ -425,6 +425,10 @@ static uint16_t full_GEN11_GA1[] = {
>      S390_FEAT_IPTE_RANGE,
>      S390_FEAT_ACCESS_EXCEPTION_FS_INDICATION,
>      S390_FEAT_GROUP_MSA_EXT_4,
> +    S390_FEAT_AP_QUERY_CONFIG_INFO,
> +    S390_FEAT_AP_SPECIAL_CMD,

As discussed for the kernel patch, this needs some thought. Do these
features need to be enabled conditionally?

> +    /* FIXME when GISA support is available
> +    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,*/

Looking forward to seeing some gisa patches :)

>  };
>  
>  #define full_GEN11_GA2 EmptyFeat

Re: [Qemu-devel] [RFC 4/5] s390x/cpumodel: enable AP facilities for guest

[David Hildenbrand]

On 14.11.2017 16:11, Cornelia Huck wrote:
> On Thu, 26 Oct 2017 11:54:53 -0400
> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:
> 
>> Sets up the following STFLE bits to enable the specified AP
>> facilities for the guest VM:
>> 	* STFLE.12: Enables the AP Query Configuration Information
>>                     facility. The AP bus running in the guest uses
>>                     the information returned from this instruction
>>                     to configure AP adapters and domains for the
>>                     guest machine.
>> 	* STFLE.15: Enables the AP Special Command facility. The AP
>>                     bus running in the guest sets the T bit in
>>                     register 0 for the PQAP(TAPQ) instruction when
>>                     scanning for AP devices if this facility is
>>                     installed.
>>
>> These facilities are required in order for the AP bus running on
>> the KVM guest to function properly.
>>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
>> ---
>>  target/s390x/cpu_features.c     |    2 ++
>>  target/s390x/cpu_features_def.h |    2 ++
>>  target/s390x/gen-features.c     |    4 ++++
>>  3 files changed, 8 insertions(+), 0 deletions(-)
>>
>> diff --git a/target/s390x/cpu_features.c b/target/s390x/cpu_features.c
>> index 31a4676..63f002c 100644
>> --- a/target/s390x/cpu_features.c
>> +++ b/target/s390x/cpu_features.c
>> @@ -36,8 +36,10 @@ static const S390FeatDef s390_features[] = {
>>      FEAT_INIT("srs", S390_FEAT_TYPE_STFL, 9, "Sense-running-status facility"),
>>      FEAT_INIT("csske", S390_FEAT_TYPE_STFL, 10, "Conditional-SSKE facility"),
>>      FEAT_INIT("ctop", S390_FEAT_TYPE_STFL, 11, "Configuration-topology facility"),
>> +    FEAT_INIT("apqci", S390_FEAT_TYPE_STFL, 12, "Query Adjunct Processor Configuration facility"),
>>      FEAT_INIT("ipter", S390_FEAT_TYPE_STFL, 13, "IPTE-range facility"),
>>      FEAT_INIT("nonqks", S390_FEAT_TYPE_STFL, 14, "Nonquiescing key-setting facility"),
>> +    FEAT_INIT("apsc", S390_FEAT_TYPE_STFL, 15, "Adjunct Processor Special Command facility"),
> 
> Are there any interdependencies for those feature bits?

And just as a side node, CPU features should only be exposed once
migration support is fully in place (and usually bound to the flag for
compat handling).

(haven't had time yet to have a closer look, so just as a general comment)


-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [RFC 5/5] s390x/docs: documentation for ap-matrix

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:54 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

Cool, documentation!

> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>
> ---
>  docs/ap_matrix.txt |  529 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 529 insertions(+), 0 deletions(-)
>  create mode 100644 docs/ap_matrix.txt
> 
> diff --git a/docs/ap_matrix.txt b/docs/ap_matrix.txt
> new file mode 100644
> index 0000000..ec7bd44
> --- /dev/null
> +++ b/docs/ap_matrix.txt
> @@ -0,0 +1,529 @@
> +Adjunct Processor (AP) Matrix Devices
> +=====================================
> +
> +Contents:
> +=========
> +* Introduction
> +* AP Architectural Overview
> +* Start Interpretive Execution (SIE) Instruction
> +* AP Matrix Configuration on Linux Host
> +* AP Matrix Configuration for a Linux Guest
> +* Starting a Linux Guest Configured with an AP Matrix
> +* Example: Configure AP Matrices for Two Linux Guests
> +
> +Introduction:
> +============
> +The IBM Adjunct Processor (AP) Cryptographic Facility is comprised 
> +of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
> +These AP devices provide cryptographic functions to all CPUs assigned to a
> +linux system running in an IBM Z system LPAR.

Before you start with the details: Give a very, very high level
overview? Like:

On s390x, crypto cards are exposed via the AP bus. This document
describes how those cards can be made available to KVM guests via vfio.

> + 
> +The intent of this document is to provide administrators with the basic
> +knowledge needed to provide a linux guest with direct access to one or more AP
> +adapters available to the host linux system using an AP matrix device 
> +
> +AP Architectural Overview:
> +=========================
> +In order understand the terminology used in the rest of this document, let's 
> +start with some definitions:
> +
> +* AP adapter
> +
> +  An AP adapter is a PCIe cryptographic adapter that can perform cryptographic
> +  functions. There can be from 0 to 256 AP adapters assigned to an LPAR.
> +  Each adapter is identified by a number from 0 to 255. When 
> +  installed, an AP is accessed by AP instructions executed by any CPU. 
> +
> +* AP domain
> +
> +  An adapter is partitioned into domains. Each domain can be thought of as 
> +  a set of hardware registers dedicated to an active LPAR. An adapter can hold 
> +  up to 256 domains. Each domain is identified by a number from 0 to 255. 
> +  Domains can be further classified into two types: 
> +  
> +    * Usage domains are domains that can be accessed directly to process AP 
> +      commands
> +  
> +    * Control domains are domains that are accessed indirectly by AP 
> +      commands sent to a usage domain to control or change the domain, for 
> +      example; to specify a private key that can be used by the domain to 
> +      perform cryptographic functions.
> +
> +* AP Queue
> +
> +  An AP queue is the means by which an AP command is sent to an 
> +  AP usage domain inside a specific AP. An AP queue is identified by a tuple 
> +  comprised of an AP adapter ID and a usage domain index. The index corresponds
> +  to a given usage domain within the adapter. This tuple forms an AP Queue 
> +  Number (APQN). AP instructions specify an APQN to identify the AP Queue 
> +  to which an AP command-request message is to be sent, or from which a 
> +  command-reply message is to be received. An APQN is specified in this 
> +  document with one of two formats: APQN (xx,yyyy) or simply xx.yyyy, where 
> +  xx is an adapter number and yyyy is a domain number. Both numbers will be 
> +  specified in hexidecimal format.
> +
> +* AP Instructions:
> +
> +  There are three AP instructions:
> +
> +  * NQAP: to enqueue an AP command-request message to an AP queue
> +  * DQAP: to dequeue an AP command-reply message from an AP queue
> +  * PQAP: to administer an AP queue
> +
> +Start Interpretive Execution (SIE) Instruction
> +==============================================
> +A linux guest running on an IBM Z system is started under KVM by executing the 
> +Start Interpretive Execution (SIE) instruction. The SIE state description is a 
> +control block that contains the state information for a KVM guest and is 
> +supplied as input to the SIE instruction. The SIE state description contains a 
> +field that references a Crypto Control Block (CRYCB) containing three
> +fields to identify the AP adapters, usage domains and control domains assigned 
> +to the KVM guest: 
> +
> +* The AP Mask (APM) field specifies the AP adapter numbers assigned to the 
> +  KVM guest. The APM controls which adapters are valid for the KVM guest.
> +
> +* The AP Queue Mask (AQM) field specifies the AP usage domain numbers assigned 
> +  to the KVM guest. The AQM controls which usage domains are valid for the 
> +  KVM guest.
> +
> +* The AP Domain Mask field specifies the AP control domains assigned to the 
> +  KVM guest. The ADM controls which control domains are valid for the 
> +  KVM guest.
> +
> +These three fields comprise the AP matrix for the guest. The APQNs accessible
> +to the guest is the intersection of all assigned adapter numbers (APM) and 
> +all assigned usage domain numbers (AQM). For example, if adapters 1 and 2 and 
> +usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6), (2,5) and
> +(2,6) will be valid for AP instructions executed on the guest.
> +
> +The SIE instruction is run in interpretive execution mode which means the 
> +AP instructions executed on the guest are interpreted by the hardware. This 
> +allows a guest direct access to the AP adapter cards. Since each domain within
> +a given adapter holds the master key used in the cryptographic functions it 
> +supports, each APQN must be assigned to at most one guest.
> +
> +   Example 1: Valid configuration for two guests:
> +   ---------------------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1,2  domain 7
> +
> +   This is valid because both guests have a unique set of APQNs: Guest1 has
> +   APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQN (1,7) and (2,7). There
> +   is not overlap.
> +
> +   Example 2: Invalid configuration for two guests:
> +   -----------------------------------------------
> +   Guest1: adapters 1,2  domains 5,6
> +   Guest2: adapter  1    domains 6,7
> +
> +   This is an invalid configuration because both guests have access to 
> +   APQNs (1,6).
> +
> +AP Matrix Configuration on Linux Host:
> +=====================================
> +A linux system is a guest of the LPAR in which it is running and has access to
> +the AP resources configured for the LPAR. The LPAR's AP matrix is 
> +configured using the 'Customize/Delete Activation Profiles' dialog from the HMC. 
> +This dialog displays the activation profiles configured for the linux system. 
> +Selecting the specific activation profile to be edited and clicking the 
> +'Customize Profile' button will open the 'Customize Image Profiles' dialog. 
> +Selecting the 'Crypto' link in the tree view on the left hand side of the dialog 
> +will display the AP matrix configuration in the right hand panel. There, one can 
> +assign AP adapters - called Cryptos - and domains to the LPAR. When the linux 
> +system is started using this activation profile, it will have access to the
> +AP matrix configured via the activation profile.
> +
> +When the linux system is started, the AP adapter devices will be connected to 
> +the AP bus and the following AP matrix interfaces will be created in sysfs:
> +
> +/sys/bus/ap
> +... [devices]
> +...... xx.yyyy
> +...... ...
> +...... cardxx
> +...... ...
> +
> +Where:
> +    cardxx     is adapter number xx (in hex)
> +    yyyy       is a usage domain number yyyy (in hex)
> +....xx.yyyy    is APQN (xx,yyyy)
> +
> +For example, if AP adapters 5 and 6 and domains 4 and 71 are configured for the
> +LPAR, the sysfs representation on the linux system would look like this:
> +
> +/sys/bus/ap
> +... [devices]
> +...... 05.0004
> +...... 05.0047
> +...... 06.0004
> +...... 06.0047
> +...... card05
> +...... card06
> +
> +There will also be AP device drivers created to control each type of AP matrix 
> +interface available to the IBM Z system:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex2acard]        for Crypto Express 2/3 accelerator cards
> +...... [cex2aqueue]       for AP queues served by Crypto Express 2/3 
> +                          accelerator cards
> +...... [cex4card]         for Crypto Express 4/5/6 accelerator and coprocessor 
> +                          cards
> +...... [cex4queue]        for AP queues served by Crypto Express 4/5/6 
> +                          accelerator and coprocessor cards
> +...... [pcixcccard]       for Crypto Express 2/3 coprocessor cards
> +...... [pcixccqueue]      for AP queues served by Crypto Express 2/3 
> +                          coprocessor cards
> +
> +Links to the AP interfaces controlled by each AP device driver will be created 
> +in the device driver's sysfs directory. For example, if AP adapter 5 and domains
> +4 and 71 (0x47) are assigned to the LPAR and adapter 5 is a CEX5 card, the 
> +following links will be created in the CEX5 drivers' sysfs directories:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [cex4card]
> +......... [card05]
> +...... [cex4queue]
> +......... [05.0004]
> +......... [05.0047]
> +
> +AP Matrix Configuration for a Linux Guest:
> +=========================================
> +In order to configure the AP matrix for a guest, the adapters, usage domains 
> +and control domains to be used by the guest must be identified. This section
> +describes how to configure a guest's AP matrix.
> +
> +When the linux host is booted, an AP matrix bus will be initialized. When 
> +initialized, the AP matrix bus creates a single AP matrix device to 
> +hold the APQNs to be made available to guests:
> +
> +/sys/bus/ap_matrix
> +... [devices]
> +......[matrix] symlink to the AP matrix device directory
> +
> +/sys/devices
> +... [ap_matrix]
> +......[matrix] the AP matrix device directory
> +
> +The kernel interfaces for configuring an AP matrix for a linux guest are built 
> +on the VFIO mediated device framework and are provided by the vfio_ap_matrix 
> +kernel module. The dependency chain for the vfio_ap_matrix module is:
> +
> +* vfio
> +* mdev
> +* vfio_mdev
> +* vfio_ap_matrix
> +
> +When the vfio_ap_matrix module is loaded, it will create the following sysfs 
> +interfaces:
> +
> +/sys/bus/ap
> +... [drivers]
> +...... [vfio_ap_matrix]
> +......... bind
> +
> +The vfio_ap_matrix device driver is created to provide an interface for securing
> +APQNs from use by the host linux system. This is accomplished by unbinding the
> +APQNs from the host device driver and binding them to the vfio_ap_matrix 
> +device driver. For example, suppose we want to secure APQN (05,0004). Assuming
> +for this example that AP adapter card 5 is a CEX5 coprocessor card:
> +
> +    echo 05.0004 > /sys/bus/ap/drivers/cex4queue/unbind
> +    echo 05.0004 > /sys/bus/ap/drivers/vfio_ap_matrix/bind
> +
> +This action will store the APQN in the /sys/devices/ap_matrix/matrix device 
> +which makes it available for use by a linux guest.
> +
> +Another side effect of loading the vfio_ap_matrix module is the creation of the
> +sysfs interfaces for configuring an AP matrix for a linux guest. These sysfs 
> +interfaces are built on the VFIO mediated device framework. To configure an AP 
> +matrix for a guest, a mediated matrix device must be created for the 
> +/sys/devices/ap_matrix/matrix device. A mediated matrix device must be created
> +for each guest that needs access to one or more AP queues. The sysfs interface 
> +for creating a mediated matrix device is in:
> +
> +/sys/devices
> +... [ap_matrix]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [ap_matrix-passthrough]
> +............... create
> +............... [devices]
> +
> +A mediated AP matrix device is created by writing a UUID to the attribute
> +file named 'create', for example:
> +
> +    uuidgen > create
> +
> +When a mediated AP matrix device is created, a sysfs directory named after 
> +the UUID will be created in the devices subdirectory:
> +
> +/sys/devices
> +... [ap_matrix]
> +......[matrix]
> +......... [mdev_supported_types]
> +............ [ap_matrix-passthrough]
> +............... create
> +............... [devices]
> +.................. [$uuid]
> +..................... adapters
> +..................... assign_adapter
> +..................... assign_control_domain
> +..................... assign_domain
> +..................... control_domains
> +..................... domains
> +..................... remove
> +..................... unassign_adapter
> +..................... unassign_control_domain
> +..................... unassign_domain
> +
> +There will also be three sets of attribute files created in the mediated matrix 
> +device's sysfs directory:
> +
> +1 Adapter assignment
> +    * An adapter is assigned by writing the adapter's number into the 
> +      'assign_adapter' file. This may be repeated multiple times to assign
> +      multiple adapters. For example, to assign adapters 5 and 6 to mediated 
> +      matrix device $uuid:
> +      
> +          echo 5 > assign_adapter
> +          echo 6 > assign_adapter
> +
> +    * An adapter may be unassigned by writing the adapter's number into the 
> +      'unassign_adapter' file. This may also be done multiple times to
> +      unassign multiple adapters.
> +
> +    * To view the adapter numbers assigned to the AP matrix mediated device, 
> +      print the 'adapters' file:
> +
> +          cat adapters
> +
> +1 Usage Domain assignment
> +    * A usage domain is assigned by writing the usage domain's number into the 
> +      'assign_domain' file. This may be repeated multiple times to assign
> +      multiple usage domains. For example, to assign usage domains 4 and 
> +      71 (0x47) to mediated matrix device $uuid:
> +
> +          echo 4 > assign_domain
> +          echo 47 > assign_domain
> +
> +    * A domain may be unassigned by writing the usage domain's number into the 
> +      'unassign_domain' file. This may be repeated multiple times to unassign
> +      multiple usage domains.
> +
> +    * To view the usage domain numbers assigned to the AP matrix mediated 
> +    device, print the 'domains' file:
> +
> +          cat domains
> +
> +1 Control domain assignment
> +    * A control domain is assigned by writing the control domain's number into 
> +      the 'assign_control_domain' file. This may be repeated multiple times to 
> +      assign multiple control domains. It is not necessary to assign 
> +      usage domain numbers as control domains, that is done automatically by 
> +      default. To assign control domains 4 and 37 (0x35) to mediated matrix 
> +      device $uuid:
> +      
> +          echo 4 > assign_control_domain
> +          echo 25 > assign_control_domain
> +
> +    * A control domain may be unassigned by writing the control domain's number 
> +      into the 'unassign_control_domain' file. This may be repeated multiple
> +      times to unassign multiple control domains.
> +
> +    * To view the control domain numbers assigned to the AP matrix mediated 
> +      device, print the 'control_domains' file:
> +
> +          cat control_domains
> +
> +Note: Hot plug/unplug is not currently supported for mediated AP matrix devices,
> +      so the AP matrix resulting from assignment and/or unassignment of AP 
> +      adapters, usage domains and control domains to a mediated AP matrix device 
> +      will not take affect until the linux guest is rebooted.
> +
> +Starting a Linux Guest Configured with an AP Matrix:
> +===================================================
> +In addition to providing the sysfs interfaces for configuring the AP matrix for 
> +a linux guest, a mediated AP matrix device also acts as a communication pathway 
> +between QEMU and the vfio_ap_matrix device driver. To gain access to the 
> +device driver, the following option must be specified on the QEMU command line:
> +
> +-device vfio_ap_matrix,sysfsdev=$path-to-mdev
> +
> +The sysfsdev parameter specifies the path to the mediated matrix device.
> +There are a number of ways to specify this path:
> +
> +/sys/devices/ap_matrix/matrix/$uuid
> +/sys/bus/mdev/devices/$uuid
> +/sys/bus/mdev/drivers/vfio_mdev/$uuid
> +/sys/devices/ap_matrix/matrix/mdev_supported_types/ap_matrix-passthrough/devices/$uuid
> +
> +When the linux guest is subsequently started, the guest will open the mediated 
> +matrix device's file descriptor to issue the command instructing the device 
> +driver to configure the AP matrix for the linux guest. In response, the 
> +vfio_ap_matrix device driver will update the APM, AQM, and ADM fields in the 
> +guest's CRYCB with the adapter, usage domain and control domain numbers 
> +specified via the mediated matrix device's sysfs attribute files. Programs 
> +running on the linux guest will then:
> +
> +1. Have access to the APQNs derived from the intersection of the AP adapter and
> +   usage domain numbers specified in the APM and AQM respectively
> +
> +2. Have authorization to process AP commands to change a control domains
> +   identified in an AP instruction sent to a valid APQN.
> +
> +Example: Configure AP Matrices for Two Linux Guests:
> +===================================================
> +Let's now provide an example to illustrate how KVM guests may be given
> +direct access to APQNs. For this example, we will illustrate how to configure 
> +two guests such that executing the lszcrypt command on the guests would 
> +look like this:
> +
> +Guest1
> +------
> +CARD.DOMAIN TYPE  MODE        
> +------------------------------
> +05          CEX5C CCA-Coproc  
> +05.0004     CEX5C CCA-Coproc
> +05.00ab     CEX5C CCA-Coproc  
> +06          CEX5A Accelerator 
> +06.0004     CEX5A Accelerator 
> +06.00ab     CEX5C CCA-Coproc  
> +
> +Guest2
> +------
> +CARD.DOMAIN TYPE  MODE        
> +------------------------------
> +05          CEX5A Accelerator 
> +05.0047     CEX5A Accelerator 
> +05.00ff     CEX5A Accelerator 
> +
> +These are the steps for configuring Guest1 and Guest2:
> +   
> +1. The first thing that needs to be done is to unbind each AP Queue device from
> +   its respective AP device driver to prevent access from the host linux system
> +   and to reserve it for use by a linux guest. For our example, let's assume
> +   the AP queues are bound to the cex4queue driver. 
> +
> +   /sys/bus/ap
> +   --- [drivers]
> +   ------ [cex4queue]
> +   --------- [05.0004]
> +   --------- [05.0047]
> +   --------- [05.00ab]
> +   --------- [05.00ff]
> +   --------- [06.0004]
> +   --------- [06.00ab]
> +   --------- unbind
> +
> +   To unbind AP queue 05.0004 from the cex4queue device driver:
> +
> +    echo 05.0004 > unbind
> +
> +   This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
> +   and 06.00ab.
> +
> +2. The next step is to reserve the queues for use by the two KVM guests. 
> +   This is accomplished by binding them to the VFIO AP matrix device driver:
> +
> +   /sys/bus/ap
> +   ---[drivers]
> +   ------ [vfio_ap_matrix]
> +   ---------- bind
> +
> +   For Guest1:
> +
> +    echo 05.0004 > bind
> +    echo 05.00ab > bind
> +    echo 06.0004 > bind
> +    echo 06.00ab > bind
> +
> +   For Guest2:
> +
> +   echo 05.0047 > bind
> +   echo 05.00ff > bind
> +
> +3. Create the mediated matrix devices needed to configure the AP matrices for 
> +   and to provide an interface to the vfio_ap_matrix driver for use by the 
> +   two guests:
> +
> +   /sys/devices/
> +   --- [ap_matrix]
> +   ------ [matrix] (this is the AP matrix device)
> +   --------- [mdev_supported_types]
> +   ------------ [ap_matrix-passthrough] (the mediated device type)
> +   --------------- create
> +   --------------- [devices]
> +
> +   To create the mediated devices for the two guests:
> +
> +    uuidgen > create
> +    uuidgen > create
> +
> +   This will create two mediated devices in the [devices] subdirectory named 
> +   with the UUID written to the create attribute file. We call them $uuid1
> +   and $uuid2:
> +
> +   /sys/devices/
> +   --- [ap_matrix]
> +   ------ [matrix]
> +   --------- [mdev_supported_types]
> +   ------------ [ap_matrix-passthrough]
> +   --------------- [devices]
> +   ------------------ [$uuid1]
> +   --------------------- adapters
> +   --------------------- assign_adapter
> +   --------------------- assign_control_domain
> +   --------------------- assign_domain
> +   --------------------- control_domains
> +   --------------------- domains
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +   ------------------ [$uuid2]
> +   --------------------- adapters
> +   --------------------- assign_adapter
> +   --------------------- assign_control_domain
> +   --------------------- assign_domain
> +   --------------------- control_domains
> +   --------------------- domains
> +   --------------------- unassign_adapter
> +   --------------------- unassign_control_domain
> +   --------------------- unassign_domain
> +
> +4. The administrator now needs to configure the matrices for mediated 
> +   devices $uuid1 (for Guest1) and $uuid2 (for Guest2). 
> +
> +   For Guest1:
> +   cd /sys/devices/ap_matrix/matrix/mdev_supported_types/ap_matrix_passthrough
> +   cd ./devices/$uuid1:
> +
> +   echo 5 > assign_adapter
> +   echo 6 > assign_adapter 
> +   echo 4 > assign_domain
> +   echo ab > assign_domain
> +
> +   For Guest2:
> +   cd /sys/devices/ap_matrix/matrix/mdev_supported_types/ap_matrix_passthrough
> +   cd ./devices/$uuid2:
> +
> +   echo 5 > assign_adapter 
> +   echo 47 > assign_domain
> +   echo ff > assign_domain
> +
> +   By architectural convention, all usage domains - i.e., domains assigned 
> +   via the assign_domain attribute file - will also be configured in the ADM 
> +   field of the KVM guest's CRYCB, so there is no need to assign control 
> +   domains here unless you want to assign control domains that are not 
> +   assigned as usage domains.
> +
> +5. Start Guest1
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio_ap_matrix,sysfsdev=/sys/devices/ap_matrix/matrix/$uuid1 ...
> +
> +6. Start Guest2
> +
> +   /usr/bin/qemu-system-s390x ... -device vfio_ap_matrix,sysfsdev=/sys/devices/ap_matrix/matrix/$uuid2 ...
> \ No newline at end of file

Please add a newline :)

I think this document can be improved by some ascii art for the
matrices. Especially if you put in a matrix for the host view, two
matrices for two well-configured guests and two matrices for two guests
with a bad (conflicting) configuration. That makes it more clear why we
need this interface.

Re: [Qemu-devel] [RFC 0/5] guest dedicated crypto adapters: QEMU usage

[Cornelia Huck]

On Thu, 26 Oct 2017 11:54:49 -0400
Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote:

> I was asked to post this QEMU patch set to the mailing list to illustrate
> how the KVM/kernel counterpart will be used. The KVM/kernel patches can be
> viewed at:
> 
> https://lkml.org/lkml/2017/10/13/706
> 
> The IBM Adjunct Processor (AP) Cryptographic Facility is comprised 
> of three AP instructions and from 1 to 256 PCIe cryptographic adapter cards.
> These AP adapters provide cryptographic functions to all CPUs assigned to a
> linux system running in an IBM Z system LPAR.
> 
> This patch series introduces a new QEMU command line option and QEMU object
> model to obtain direct guest access to one or more AP adapter cards assigned
> to the LPAR in which the host linux system is running. It is predicated 
> on the KVM/kernel model for providing direct guest access to AP adapters. A 
> complete description for dedicating crypto adapters to a linux guest can be 
> found in the docs/ap-matrix.txt file provided with this patch set.
> 
> Signed-off-by: Tony Krowiak <akrowiak@linux.vnet.ibm.com>

+cc qemu-devel@nongnu.org

The interface exposed by the kernel looks fine to me, and I did not
spot really broken things on a quick readthrough. I'd like more
feedback, though.

> 
> Tony Krowiak (5):
>   s390x/ap-matrix: Adjunct Processor (AP) matrix object model
>   s390x/vfio: ap-matrix: Introduce VFIO AP Matrix device
>   s390x/ap-matrix: Configure AP matrix for KVM guest
>   s390x/cpumodel: enable AP facilities for guest
>   s390x/docs: documentation for ap-matrix
> 
>  default-configs/s390x-softmmu.mak |    1 +
>  docs/ap_matrix.txt                |  529 +++++++++++++++++++++++++++++++++++++
>  hw/s390x/Makefile.objs            |    2 +
>  hw/s390x/ap-matrix-device.c       |   33 +++
>  hw/s390x/ap-matrix-device.h       |   53 ++++
>  hw/s390x/s390-ap-matrix.c         |   52 ++++
>  hw/vfio/Makefile.objs             |    1 +
>  hw/vfio/ap-matrix.c               |  224 ++++++++++++++++
>  include/hw/s390x/s390-ap-matrix.h |   39 +++
>  include/hw/vfio/vfio-common.h     |    1 +
>  linux-headers/linux/vfio.h        |   29 ++-
>  target/s390x/cpu_features.c       |    2 +
>  target/s390x/cpu_features_def.h   |    2 +
>  target/s390x/gen-features.c       |    4 +
>  14 files changed, 969 insertions(+), 3 deletions(-)
>  create mode 100644 docs/ap_matrix.txt
>  create mode 100644 hw/s390x/ap-matrix-device.c
>  create mode 100644 hw/s390x/ap-matrix-device.h
>  create mode 100644 hw/s390x/s390-ap-matrix.c
>  create mode 100644 hw/vfio/ap-matrix.c
>  create mode 100644 include/hw/s390x/s390-ap-matrix.h
>