[PATCH RESEND v2 0/8] Support VGIC save/restore using device control API

[PATCH RESEND v2 0/8] Support VGIC save/restore using device control API

[Christoffer Dall]

Implement save/restore of the VGIC state using the newer KVM Device
Control API.  This requries some number of changes to existing code in
addition to actually supporting save/restore of the necessary state.

The first patches (01-03) support creating the VGIC using the Device
Control API.  This change is necessary because there are no other
suitable KVM APIs that we can leverage to access the VGIC state from
user space and the device control API was crafted exactly for this
purpose.

Subsequent patches add the missing infrastructure and user space API
pieces necessary to actually save and restore the VGIC state.  The GIC
v2.0 architecture specification already specifies registers that can be
used to save and restore the complete VGIC state for suspend/resume
purposes on real hardware, and we can resuse this interface for the
VGIC.  The API is therefore based on the memory-mapped register accesses
defined in the specs.  See the individual patches for details.

The patches are based on kvm-arm-next:
git://git.linaro.org/people/cdall/linux-kvm-arm.git kvm-arm-next

This patch series based on the above can be cloned from:
git://git.linaro.org/people/cdall/linux-kvm-arm.git vgic-migrate-v2

User space patches for QEMU also posted on the list.  Tested on Versatile
Express TC2.

Changelogs in the individual patches.

Christoffer Dall (8):
  ARM: KVM: Allow creating the VGIC after VCPUs
  KVM: arm-vgic: Support KVM_CREATE_DEVICE for VGIC
  KVM: arm-vgic: Set base addr through device API
  irqchip: arm-gic: Define additional MMIO offsets and masks
  KVM: arm-vgic: Make vgic mmio functions more generic
  KVM: arm-vgic: Add vgic reg access from dev attr
  KVM: arm-vgic: Add GICD_SPENDSGIR and GICD_CPENDSGIR handlers
  KVM: arm-vgic: Support CPU interface reg access

 Documentation/virtual/kvm/api.txt              |    6 +-
 Documentation/virtual/kvm/devices/arm-vgic.txt |   71 ++++
 arch/arm/include/uapi/asm/kvm.h                |    8 +
 arch/arm/kvm/arm.c                             |   10 +-
 include/kvm/arm_vgic.h                         |    2 +-
 include/linux/irqchip/arm-gic.h                |   14 +
 include/linux/kvm_host.h                       |    1 +
 include/uapi/linux/kvm.h                       |    1 +
 virt/kvm/arm/vgic.c                            |  494 ++++++++++++++++++++++--
 virt/kvm/kvm_main.c                            |    5 +
 10 files changed, 580 insertions(+), 32 deletions(-)
 create mode 100644 Documentation/virtual/kvm/devices/arm-vgic.txt

-- 
1.7.10.4

[PATCH RESEND v2 1/8] ARM: KVM: Allow creating the VGIC after VCPUs

[Christoffer Dall]

Rework the VGIC initialization slightly to allow initialization of the
vgic cpu-specific state even if the irqchip (the VGIC) hasn't been
created by user space yet.  This is safe, because the vgic data
structures are already allocated when the CPU is allocated if VGIC
support is compiled into the kernel.  Further, the init process does not
depend on any other information and the sacrifice is a slight
performance degradation for creating VMs in the no-VGIC case.

The reason is that the new device control API doesn't mandate creating
the VGIC before creating the VCPU and it is unreasonable to require user
space to create the VGIC before creating the VCPUs.

At the same time move the irqchip_in_kernel check out of
kvm_vcpu_first_run_init and into the init function to make the per-vcpu
and global init functions symmetric and add comments on the exported
functions making it a bit easier to understand the init flow by only
looking at vgic.c.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 arch/arm/kvm/arm.c  |    7 ++++---
 virt/kvm/arm/vgic.c |   22 +++++++++++++++++++---
 2 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 9c697db..2b1091a 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -461,6 +461,8 @@ static void update_vttbr(struct kvm *kvm)
 
 static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 {
+	int ret;
+
 	if (likely(vcpu->arch.has_run_once))
 		return 0;
 
@@ -470,9 +472,8 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
 	 * Initialize the VGIC before running a vcpu the first time on
 	 * this VM.
 	 */
-	if (irqchip_in_kernel(vcpu->kvm) &&
-	    unlikely(!vgic_initialized(vcpu->kvm))) {
-		int ret = kvm_vgic_init(vcpu->kvm);
+	if (unlikely(!vgic_initialized(vcpu->kvm))) {
+		ret = kvm_vgic_init(vcpu->kvm);
 		if (ret)
 			return ret;
 	}
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 685fc72..5ce100f 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1243,15 +1243,19 @@ static irqreturn_t vgic_maintenance_handler(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+/**
+ * kvm_vgic_vcpu_init - Initialize per-vcpu VGIC state
+ * @vcpu: pointer to the vcpu struct
+ *
+ * Initialize the vgic_cpu struct and vgic_dist struct fields pertaining to
+ * this vcpu and enable the VGIC for this VCPU
+ */
 int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 {
 	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
 	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
 	int i;
 
-	if (!irqchip_in_kernel(vcpu->kvm))
-		return 0;
-
 	if (vcpu->vcpu_id >= VGIC_MAX_CPUS)
 		return -EBUSY;
 
@@ -1383,10 +1387,22 @@ out:
 	return ret;
 }
 
+/**
+ * kvm_vgic_init - Initialize global VGIC state before running any VCPUs
+ * @kvm: pointer to the kvm struct
+ *
+ * Map the virtual CPU interface into the VM before running any VCPUs.  We
+ * can't do this at creation time, because user space must first set the
+ * virtual CPU interface address in the guest physical address space.  Also
+ * initialize the ITARGETSRn regs to 0 on the emulated distributor.
+ */
 int kvm_vgic_init(struct kvm *kvm)
 {
 	int ret = 0, i;
 
+	if (!irqchip_in_kernel(kvm))
+		return 0;
+
 	mutex_lock(&kvm->lock);
 
 	if (vgic_initialized(kvm))
-- 
1.7.10.4

[PATCH RESEND v2 2/8] KVM: arm-vgic: Support KVM_CREATE_DEVICE for VGIC

[Christoffer Dall]

Support creating the ARM VGIC device through the KVM_CREATE_DEVICE
ioctl, which can then later be leveraged to use the
KVM_{GET/SET}_DEVICE_ATTR, which is useful both for setting addresses in
a more generic API than the ARM-specific one and is useful for
save/restore of VGIC state.

Adds KVM_CAP_DEVICE_CTRL to ARM capabilities.

Note that we change the check for creating a VGIC from bailing out if
any VCPUs were created to bailing if any VCPUs were ever run.  This is
an important distinction that doesn't break anything, but allows
creating the VGIC after the VCPUs have been created.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 Documentation/virtual/kvm/devices/arm-vgic.txt |   10 ++++++
 arch/arm/include/uapi/asm/kvm.h                |    1 -
 arch/arm/kvm/arm.c                             |    1 +
 include/linux/kvm_host.h                       |    1 +
 include/uapi/linux/kvm.h                       |    1 +
 virt/kvm/arm/vgic.c                            |   46 ++++++++++++++++++++++--
 virt/kvm/kvm_main.c                            |    5 +++
 7 files changed, 62 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/virtual/kvm/devices/arm-vgic.txt

diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
new file mode 100644
index 0000000..38f27f7
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
@@ -0,0 +1,10 @@
+ARM Virtual Generic Interrupt Controller (VGIC)
+===============================================
+
+Device types supported:
+  KVM_DEV_TYPE_ARM_VGIC_V2     ARM Generic Interrupt Controller v2.0
+
+Only one VGIC instance may be instantiated through either this API or the
+legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM interrupt
+controller, requiring emulated user-space devices to inject interrupts to the
+VGIC instead of directly to CPUs.
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index c1ee007..1c85102 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -142,7 +142,6 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM_VFP_FPINST		0x1009
 #define KVM_REG_ARM_VFP_FPINST2		0x100A
 
-
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
 #define KVM_ARM_IRQ_TYPE_MASK		0xff
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 2b1091a..ab96af2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -187,6 +187,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_IRQCHIP:
 		r = vgic_present;
 		break;
+	case KVM_CAP_DEVICE_CTRL:
 	case KVM_CAP_USER_MEMORY:
 	case KVM_CAP_SYNC_MMU:
 	case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ca645a0..2906b79 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1065,6 +1065,7 @@ struct kvm_device *kvm_device_from_filp(struct file *filp);
 
 extern struct kvm_device_ops kvm_mpic_ops;
 extern struct kvm_device_ops kvm_xics_ops;
+extern struct kvm_device_ops kvm_arm_vgic_ops;
 
 #ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
 
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 99c2533..2d50233 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -843,6 +843,7 @@ struct kvm_device_attr {
 #define KVM_DEV_TYPE_FSL_MPIC_20	1
 #define KVM_DEV_TYPE_FSL_MPIC_42	2
 #define KVM_DEV_TYPE_XICS		3
+#define KVM_DEV_TYPE_ARM_VGIC_V2	4
 
 /*
  * ioctls for VM fds
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 5ce100f..79a8bae 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1434,15 +1434,23 @@ out:
 
 int kvm_vgic_create(struct kvm *kvm)
 {
-	int ret = 0;
+	int i, ret = 0;
+	struct kvm_vcpu *vcpu;
 
 	mutex_lock(&kvm->lock);
 
-	if (atomic_read(&kvm->online_vcpus) || kvm->arch.vgic.vctrl_base) {
+	if (kvm->arch.vgic.vctrl_base) {
 		ret = -EEXIST;
 		goto out;
 	}
 
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		if (vcpu->arch.has_run_once) {
+			ret = -EBUSY;
+			goto out;
+		}
+	}
+
 	spin_lock_init(&kvm->arch.vgic.lock);
 	kvm->arch.vgic.vctrl_base = vgic_vctrl_base;
 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
@@ -1511,3 +1519,37 @@ int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
 	mutex_unlock(&kvm->lock);
 	return r;
 }
+
+static int vgic_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+static int vgic_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+static int vgic_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+static void vgic_destroy(struct kvm_device *dev)
+{
+	kfree(dev);
+}
+
+static int vgic_create(struct kvm_device *dev, u32 type)
+{
+	return kvm_vgic_create(dev->kvm);
+}
+
+struct kvm_device_ops kvm_arm_vgic_ops = {
+	.name = "kvm-arm-vgic",
+	.create = vgic_create,
+	.destroy = vgic_destroy,
+	.set_attr = vgic_set_attr,
+	.get_attr = vgic_get_attr,
+	.has_attr = vgic_has_attr,
+};
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index bf040c4..534fd3a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2265,6 +2265,11 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
 		ops = &kvm_xics_ops;
 		break;
 #endif
+#ifdef CONFIG_KVM_ARM_VGIC
+	case KVM_DEV_TYPE_ARM_VGIC_V2:
+		ops = &kvm_arm_vgic_ops;
+		break;
+#endif
 	default:
 		return -ENODEV;
 	}
-- 
1.7.10.4

[PATCH RESEND v2 3/8] KVM: arm-vgic: Set base addr through device API

[Christoffer Dall]

Support setting the distributor and cpu interface base addresses in the
VM physical address space through the KVM_{SET,GET}_DEVICE_ATTR API
in addition to the ARM specific API.

This has the added benefit of being able to share more code in user
space and do things in a uniform maner.

Also deprecate the older API at the same time, but backwards
compatibility will be maintained.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 Documentation/virtual/kvm/api.txt              |    6 +-
 Documentation/virtual/kvm/devices/arm-vgic.txt |   11 +++
 arch/arm/include/uapi/asm/kvm.h                |    9 +++
 arch/arm/kvm/arm.c                             |    2 +-
 include/kvm/arm_vgic.h                         |    2 +-
 virt/kvm/arm/vgic.c                            |   90 ++++++++++++++++++++----
 6 files changed, 105 insertions(+), 15 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 858aecf..d68b6c2 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2324,7 +2324,7 @@ This ioctl returns the guest registers that are supported for the
 KVM_GET_ONE_REG/KVM_SET_ONE_REG calls.
 
 
-4.84 KVM_ARM_SET_DEVICE_ADDR
+4.84 KVM_ARM_SET_DEVICE_ADDR (deprecated)
 
 Capability: KVM_CAP_ARM_SET_DEVICE_ADDR
 Architectures: arm, arm64
@@ -2362,6 +2362,10 @@ must be called after calling KVM_CREATE_IRQCHIP, but before calling
 KVM_RUN on any of the VCPUs.  Calling this ioctl twice for any of the
 base addresses will return -EEXIST.
 
+Note, this IOCTL is deprecated and the more flexible SET/GET_DEVICE_ATTR API
+should be used instead.
+
+
 4.85 KVM_PPC_RTAS_DEFINE_TOKEN
 
 Capability: KVM_CAP_PPC_RTAS
diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
index 38f27f7..c9febb2 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
@@ -8,3 +8,14 @@ Only one VGIC instance may be instantiated through either this API or the
 legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM interrupt
 controller, requiring emulated user-space devices to inject interrupts to the
 VGIC instead of directly to CPUs.
+
+Groups:
+  KVM_DEV_ARM_VGIC_GRP_ADDR
+  Attributes:
+    KVM_VGIC_V2_ADDR_TYPE_DIST (rw, 64-bit)
+      Base address in the guest physical address space of the GIC distributor
+      register mappings.
+
+    KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
+      Base address in the guest physical address space of the GIC virtual cpu
+      interface register mappings.
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 1c85102..587f1ae 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -142,6 +142,15 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM_VFP_FPINST		0x1009
 #define KVM_REG_ARM_VFP_FPINST2		0x100A
 
+/* Device Control API: ARM VGIC */
+#define KVM_DEV_ARM_VGIC_GRP_ADDR	0
+#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS	1
+#define KVM_DEV_ARM_VGIC_GRP_CPU_REGS	2
+#define   KVM_DEV_ARM_VGIC_CPUID_SHIFT	32
+#define   KVM_DEV_ARM_VGIC_CPUID_MASK	(0xffULL << KVM_DEV_ARM_VGIC_CPUID_SHIFT)
+#define   KVM_DEV_ARM_VGIC_OFFSET_SHIFT	0
+#define   KVM_DEV_ARM_VGIC_OFFSET_MASK	(0xffffffffULL << KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
+
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
 #define KVM_ARM_IRQ_TYPE_MASK		0xff
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index ab96af2..3ecee45 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -773,7 +773,7 @@ static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm,
 	case KVM_ARM_DEVICE_VGIC_V2:
 		if (!vgic_present)
 			return -ENXIO;
-		return kvm_vgic_set_addr(kvm, type, dev_addr->addr);
+		return kvm_vgic_addr(kvm, type, &dev_addr->addr, true);
 	default:
 		return -ENODEV;
 	}
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 7e2d158..be85127 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -144,7 +144,7 @@ struct kvm_run;
 struct kvm_exit_mmio;
 
 #ifdef CONFIG_KVM_ARM_VGIC
-int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr);
+int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write);
 int kvm_vgic_hyp_init(void);
 int kvm_vgic_init(struct kvm *kvm);
 int kvm_vgic_create(struct kvm *kvm);
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 79a8bae..d9c0fc5 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1479,6 +1479,12 @@ static int vgic_ioaddr_assign(struct kvm *kvm, phys_addr_t *ioaddr,
 {
 	int ret;
 
+	if (addr & ~KVM_PHYS_MASK)
+		return -E2BIG;
+
+	if (addr & (SZ_4K - 1))
+		return -EINVAL;
+
 	if (!IS_VGIC_ADDR_UNDEF(*ioaddr))
 		return -EEXIST;
 	if (addr + size < addr)
@@ -1491,26 +1497,41 @@ static int vgic_ioaddr_assign(struct kvm *kvm, phys_addr_t *ioaddr,
 	return ret;
 }
 
-int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
+/**
+ * kvm_vgic_addr - set or get vgic VM base addresses
+ * @kvm:   pointer to the vm struct
+ * @type:  the VGIC addr type, one of KVM_VGIC_V2_ADDR_TYPE_XXX
+ * @addr:  pointer to address value
+ * @write: if true set the address in the VM address space, if false read the
+ *          address
+ *
+ * Set or get the vgic base addresses for the distributor and the virtual CPU
+ * interface in the VM physical address space.  These addresses are properties
+ * of the emulated core/SoC and therefore user space initially knows this
+ * information.
+ */
+int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
 {
 	int r = 0;
 	struct vgic_dist *vgic = &kvm->arch.vgic;
 
-	if (addr & ~KVM_PHYS_MASK)
-		return -E2BIG;
-
-	if (addr & (SZ_4K - 1))
-		return -EINVAL;
-
 	mutex_lock(&kvm->lock);
 	switch (type) {
 	case KVM_VGIC_V2_ADDR_TYPE_DIST:
-		r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
-				       addr, KVM_VGIC_V2_DIST_SIZE);
+		if (write) {
+			r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
+					       *addr, KVM_VGIC_V2_DIST_SIZE);
+		} else {
+			*addr = vgic->vgic_dist_base;
+		}
 		break;
 	case KVM_VGIC_V2_ADDR_TYPE_CPU:
-		r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
-				       addr, KVM_VGIC_V2_CPU_SIZE);
+		if (write) {
+			r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
+					       *addr, KVM_VGIC_V2_CPU_SIZE);
+		} else {
+			*addr = vgic->vgic_cpu_base;
+		}
 		break;
 	default:
 		r = -ENODEV;
@@ -1522,16 +1543,61 @@ int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
 
 static int vgic_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 {
+	int r;
+
+	switch (attr->group) {
+	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 addr;
+		unsigned long type = (unsigned long)attr->attr;
+
+		if (copy_from_user(&addr, uaddr, sizeof(addr)))
+			return -EFAULT;
+
+		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
+		return (r == -ENODEV) ? -ENXIO : r;
+	}
+	}
+
 	return -ENXIO;
 }
 
 static int vgic_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 {
-	return -ENXIO;
+	int r = ENXIO;
+
+	switch (attr->group) {
+	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 addr;
+		unsigned long type = (unsigned long)attr->attr;
+
+		r = kvm_vgic_addr(dev->kvm, type, &addr, false);
+		if (r)
+			return (r == -ENODEV) ? -ENXIO : r;
+
+		r = 0;
+		if (copy_to_user(uaddr, &addr, sizeof(addr)))
+			return -EFAULT;
+	}
+	}
+
+	return r;
 }
 
 static int vgic_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 {
+	phys_addr_t offset;
+
+	switch (attr->group) {
+	case KVM_DEV_ARM_VGIC_GRP_ADDR:
+		switch (attr->attr) {
+		case KVM_VGIC_V2_ADDR_TYPE_DIST:
+		case KVM_VGIC_V2_ADDR_TYPE_CPU:
+			return 0;
+		}
+		break;
+	}
 	return -ENXIO;
 }
 
-- 
1.7.10.4

[PATCH RESEND v2 4/8] irqchip: arm-gic: Define additional MMIO offsets and masks

[Christoffer Dall]

Define CPU interface offsets for the GICC_ABPR, GICC_APR, and GICC_IIDR
registers.  Define distributor registers for the GICD_SPENDSGIR and the
GICD_CPENDSGIR.  KVM/ARM needs to know about these definitions to fully
support save/restore of the VGIC.

Also define some masks and shifts for the various GICH_VMCR fields.

Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 include/linux/irqchip/arm-gic.h |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index 0e5d9ec..28b28fc 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -17,6 +17,9 @@
 #define GIC_CPU_EOI			0x10
 #define GIC_CPU_RUNNINGPRI		0x14
 #define GIC_CPU_HIGHPRI			0x18
+#define GIC_CPU_ALIAS_BINPOINT		0x1c
+#define GIC_CPU_ACTIVEPRIO		0xd0
+#define GIC_CPU_IDENT			0xfc
 
 #define GIC_DIST_CTRL			0x000
 #define GIC_DIST_CTR			0x004
@@ -31,6 +34,8 @@
 #define GIC_DIST_TARGET			0x800
 #define GIC_DIST_CONFIG			0xc00
 #define GIC_DIST_SOFTINT		0xf00
+#define GIC_DIST_SGI_CLEAR		0xf10
+#define GIC_DIST_SGI_SET		0xf20
 
 #define GICH_HCR			0x0
 #define GICH_VTR			0x4
@@ -54,6 +59,15 @@
 #define GICH_LR_ACTIVE_BIT		(1 << 29)
 #define GICH_LR_EOI			(1 << 19)
 
+#define GICH_VMCR_CTRL_SHIFT		0
+#define GICH_VMCR_CTRL_MASK		(0x21f << GICH_VMCR_CTRL_SHIFT)
+#define GICH_VMCR_PRIMASK_SHIFT		27
+#define GICH_VMCR_PRIMASK_MASK		(0x1f << GICH_VMCR_PRIMASK_SHIFT)
+#define GICH_VMCR_BINPOINT_SHIFT	21
+#define GICH_VMCR_BINPOINT_MASK		(0x7 << GICH_VMCR_BINPOINT_SHIFT)
+#define GICH_VMCR_ALIAS_BINPOINT_SHIFT	18
+#define GICH_VMCR_ALIAS_BINPOINT_MASK	(0x7 << GICH_VMCR_ALIAS_BINPOINT_SHIFT)
+
 #define GICH_MISR_EOI			(1 << 0)
 #define GICH_MISR_U			(1 << 1)
 
-- 
1.7.10.4

[PATCH RESEND v2 5/8] KVM: arm-vgic: Make vgic mmio functions more generic

[Christoffer Dall]

Rename the vgic_ranges array to vgic_dist_ranges to be more specific and
to prepare for handling CPU interface register access as well (for
save/restore of VGIC state).

Pass offset from distributor or interface MMIO base to
find_matching_range function instead of the physical address of the
access in the VM memory map.  This allows other callers unaware of the
VM specifics, but with generic VGIC knowledge to reuse the function.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 virt/kvm/arm/vgic.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index d9c0fc5..1148a2e 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -602,7 +602,7 @@ struct mmio_range {
 			    phys_addr_t offset);
 };
 
-static const struct mmio_range vgic_ranges[] = {
+static const struct mmio_range vgic_dist_ranges[] = {
 	{
 		.base		= GIC_DIST_CTRL,
 		.len		= 12,
@@ -669,14 +669,13 @@ static const struct mmio_range vgic_ranges[] = {
 static const
 struct mmio_range *find_matching_range(const struct mmio_range *ranges,
 				       struct kvm_exit_mmio *mmio,
-				       phys_addr_t base)
+				       phys_addr_t offset)
 {
 	const struct mmio_range *r = ranges;
-	phys_addr_t addr = mmio->phys_addr - base;
 
 	while (r->len) {
-		if (addr >= r->base &&
-		    (addr + mmio->len) <= (r->base + r->len))
+		if (offset >= r->base &&
+		    (offset + mmio->len) <= (r->base + r->len))
 			return r;
 		r++;
 	}
@@ -713,7 +712,8 @@ bool vgic_handle_mmio(struct kvm_vcpu *vcpu, struct kvm_run *run,
 		return true;
 	}
 
-	range = find_matching_range(vgic_ranges, mmio, base);
+	offset = mmio->phys_addr - base;
+	range = find_matching_range(vgic_dist_ranges, mmio, offset);
 	if (unlikely(!range || !range->handle_mmio)) {
 		pr_warn("Unhandled access %d %08llx %d\n",
 			mmio->is_write, mmio->phys_addr, mmio->len);
-- 
1.7.10.4

[PATCH RESEND v2 6/8] KVM: arm-vgic: Add vgic reg access from dev attr

[Christoffer Dall]

Add infrastructure to handle distributor and cpu interface register
accesses through the KVM_{GET/SET}_DEVICE_ATTR interface by adding the
KVM_DEV_ARM_VGIC_GRP_DIST_REGS and KVM_DEV_ARM_VGIC_GRP_CPU_REGS groups
and defining the semantics of the attr field to be the MMIO offset as
specified in the GICv2 specs.

Missing register accesses or other changes in individual register access
functions to support save/restore of the VGIC state is added in
subsequent patches.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>

---
Changelog[v2]:
 - Added implementation specific format for the GICC_APRn registers.
---
 Documentation/virtual/kvm/devices/arm-vgic.txt |   50 +++++++++
 virt/kvm/arm/vgic.c                            |  143 ++++++++++++++++++++++++
 2 files changed, 193 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
index c9febb2..e6416f8e 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
@@ -19,3 +19,53 @@ Groups:
     KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
       Base address in the guest physical address space of the GIC virtual cpu
       interface register mappings.
+
+  KVM_DEV_ARM_VGIC_GRP_DIST_REGS
+  Attributes:
+    The attr field of kvm_device_attr encodes two values:
+    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
+    values:   |    reserved   |   cpu id   |      offset     |
+
+    All distributor regs are (rw, 32-bit)
+
+    The offset is relative to the "Distributor base address" as defined in the
+    GICv2 specs.  Getting or setting such a register has the same effect as
+    reading or writing the register on the actual hardware from the cpu
+    specified with cpu id field.  Note that most distributor fields are not
+    banked, but return the same value regardless of the cpu id used to access
+    the register.
+  Limitations:
+    - Priorities are not implemented, and registers are RAZ/WI
+  Errors:
+    - ENODEV: Getting or setting this register is not yet supported
+
+  KVM_DEV_ARM_VGIC_GRP_CPU_REGS
+  Attributes:
+    The attr field of kvm_device_attr encodes two values:
+    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
+    values:   |    reserved   |   cpu id   |      offset     |
+
+    All CPU regs are (rw, 32-bit)
+
+    The offset specifies the offset from the "CPU interface base address" as
+    defined in the GICv2 specs.  Getting or setting such a register has the
+    same effect as reading or writing the register on the actual hardware.
+
+    The Active Priorities Registers APRn are implementation defined, so we set a
+    fixed format for our implementation that fits with the model of a "GICv2
+    impementation without the security extensions" which we present to the
+    guest.  This interface always exposes four register APR[0-3] describing the
+    maximum possible 128 preemption levels.  The semantics of the register
+    indicate if any interrupts in a given preemption level are in the active
+    state by setting the corresponding bit.
+
+    Thus, preemption level X has one or more active interrupts if and only if:
+
+      APRn[X mod 32] == 0b1,  where n = X / 32
+
+    Bits for undefined preemption levels are RAZ/WI.
+
+  Limitations:
+    - Priorities are not implemented, and registers are RAZ/WI
+  Errors:
+    - ENODEV: Getting or setting this register is not yet supported
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 1148a2e..f2dc72a 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -589,11 +589,29 @@ static bool handle_mmio_sgi_reg(struct kvm_vcpu *vcpu,
 	return false;
 }
 
+static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
+				  struct kvm_exit_mmio *mmio,
+				  phys_addr_t offset)
+{
+	return false;
+}
+
+static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
+				struct kvm_exit_mmio *mmio,
+				phys_addr_t offset)
+{
+	return false;
+}
+
 /*
  * I would have liked to use the kvm_bus_io_*() API instead, but it
  * cannot cope with banked registers (only the VM pointer is passed
  * around, and we need the vcpu). One of these days, someone please
  * fix it!
+ *
+ * Note that the handle_mmio implementations should not use the phys_addr
+ * field from the kvm_exit_mmio struct as this will not have any sane values
+ * when used to save/restore state from user space.
  */
 struct mmio_range {
 	phys_addr_t base;
@@ -663,6 +681,16 @@ static const struct mmio_range vgic_dist_ranges[] = {
 		.len		= 4,
 		.handle_mmio	= handle_mmio_sgi_reg,
 	},
+	{
+		.base		= GIC_DIST_SGI_CLEAR,
+		.len		= VGIC_NR_SGIS,
+		.handle_mmio	= handle_mmio_sgi_clear,
+	},
+	{
+		.base		= GIC_DIST_SGI_SET,
+		.len		= VGIC_NR_SGIS,
+		.handle_mmio	= handle_mmio_sgi_set,
+	},
 	{}
 };
 
@@ -1541,6 +1569,80 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
 	return r;
 }
 
+static bool handle_cpu_mmio_misc(struct kvm_vcpu *vcpu,
+				 struct kvm_exit_mmio *mmio, phys_addr_t offset)
+{
+	return true;
+}
+
+static const struct mmio_range vgic_cpu_ranges[] = {
+	{
+		.base		= GIC_CPU_CTRL,
+		.len		= 12,
+		.handle_mmio	= handle_cpu_mmio_misc,
+	},
+	{
+		.base		= GIC_CPU_ALIAS_BINPOINT,
+		.len		= 4,
+		.handle_mmio	= handle_cpu_mmio_misc,
+	},
+	{
+		.base		= GIC_CPU_ACTIVEPRIO,
+		.len		= 16,
+		.handle_mmio	= handle_cpu_mmio_misc,
+	},
+	{
+		.base		= GIC_CPU_IDENT,
+		.len		= 4,
+		.handle_mmio	= handle_cpu_mmio_misc,
+	},
+};
+
+static struct kvm_exit_mmio dev_attr_mmio = { .len = 4 };
+
+static int vgic_attr_regs_access(struct kvm_device *dev,
+				 struct kvm_device_attr *attr,
+				 u32 *reg, bool is_write)
+{
+	const struct mmio_range *r = NULL;
+	phys_addr_t offset;
+	int cpuid;
+	struct kvm_vcpu *vcpu;
+	struct kvm_exit_mmio mmio;
+
+	offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
+	cpuid = (attr->attr & KVM_DEV_ARM_VGIC_CPUID_MASK) >>
+		KVM_DEV_ARM_VGIC_CPUID_SHIFT;
+
+	if (cpuid >= atomic_read(&dev->kvm->online_vcpus))
+		return -EINVAL;
+
+	vcpu = kvm_get_vcpu(dev->kvm, cpuid);
+
+	mmio.len = 4;
+	mmio.is_write = is_write;
+	if (is_write)
+		mmio_data_write(&mmio, ~0, *reg);
+
+	if (attr->group == KVM_DEV_ARM_VGIC_GRP_DIST_REGS)
+		r = find_matching_range(vgic_dist_ranges, &mmio, offset);
+	else if (attr->group == KVM_DEV_ARM_VGIC_GRP_CPU_REGS)
+		r = find_matching_range(vgic_cpu_ranges, &mmio, offset);
+
+	if (unlikely(!r || !r->handle_mmio))
+		return -ENXIO;
+
+	spin_lock(&vcpu->kvm->arch.vgic.lock);
+	offset -= r->base;
+	r->handle_mmio(vcpu, &mmio, offset);
+	spin_unlock(&vcpu->kvm->arch.vgic.lock);
+
+	if (!is_write)
+		*reg = mmio_data_read(&mmio, ~0);
+
+	return 0;
+}
+
 static int vgic_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 {
 	int r;
@@ -1557,6 +1659,18 @@ static int vgic_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
 		return (r == -ENODEV) ? -ENXIO : r;
 	}
+
+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
+		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
+		u32 reg;
+
+		if (get_user(reg, uaddr))
+			return -EFAULT;
+
+		return vgic_attr_regs_access(dev, attr, &reg, true);
+	}
+
 	}
 
 	return -ENXIO;
@@ -1579,12 +1693,35 @@ static int vgic_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 		r = 0;
 		if (copy_to_user(uaddr, &addr, sizeof(addr)))
 			return -EFAULT;
+		break;
 	}
+
+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
+		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
+		u32 reg = 0;
+
+		r = vgic_attr_regs_access(dev, attr, &reg, false);
+		if (r)
+			return r;
+		r = put_user(reg, uaddr);
+		break;
+	}
+
 	}
 
 	return r;
 }
 
+static int vgic_has_attr_regs(const struct mmio_range *ranges,
+			      phys_addr_t offset)
+{
+	if (find_matching_range(ranges, &dev_attr_mmio, offset))
+		return 0;
+	else
+		return -ENXIO;
+}
+
 static int vgic_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 {
 	phys_addr_t offset;
@@ -1597,6 +1734,12 @@ static int vgic_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
 			return 0;
 		}
 		break;
+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
+		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
+		return vgic_has_attr_regs(vgic_dist_ranges, offset);
+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS:
+		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
+		return vgic_has_attr_regs(vgic_cpu_ranges, offset);
 	}
 	return -ENXIO;
 }
-- 
1.7.10.4

[PATCH RESEND v2 7/8] KVM: arm-vgic: Add GICD_SPENDSGIR and GICD_CPENDSGIR handlers

[Christoffer Dall]

Handle MMIO accesses to the two registers which should support both the
case where the VMs want to read/write either of these registers and the
case where user space reads/writes these registers to do save/restore of
the VGIC state.

Note that the added complexity compared to simple set/clear enable
registers stems from the bookkeping of source cpu ids.  It may be
possible to change the underlying data structure to simplify the
complexity, but since this is not in the critical path, at all, this is
left as an interesting excercise to the reader.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>

---
Changelog[v2]:
 - Use struct kvm_exit_mmio accessors for ->data field.
---
 virt/kvm/arm/vgic.c |  114 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 112 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index f2dc72a..4e8c3ab 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -589,18 +589,128 @@ static bool handle_mmio_sgi_reg(struct kvm_vcpu *vcpu,
 	return false;
 }
 
+static void read_sgi_set_clear(struct kvm_vcpu *vcpu,
+			       struct kvm_exit_mmio *mmio,
+			       phys_addr_t offset)
+{
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	int i, sgi, cpu;
+	int min_sgi = (offset & ~0x3) * 4;
+	int max_sgi = min_sgi + 3;
+	int vcpu_id = vcpu->vcpu_id;
+	u32 lr, reg = 0;
+
+	/* Copy source SGIs from distributor side */
+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
+		int shift = 8 * (sgi - min_sgi);
+		reg |= (u32)dist->irq_sgi_sources[vcpu_id][sgi] << shift;
+	}
+
+	/* Copy source SGIs already on LRs */
+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
+		lr = vgic_cpu->vgic_lr[i];
+		sgi = lr & GICH_LR_VIRTUALID;
+		cpu = (lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
+		if (sgi >= min_sgi && sgi <= max_sgi) {
+			if (lr & GICH_LR_STATE)
+				reg |= (1 << cpu) << (8 * (sgi - min_sgi));
+		}
+	}
+
+	mmio_data_write(mmio, ~0, reg);
+}
+
 static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
 				  struct kvm_exit_mmio *mmio,
 				  phys_addr_t offset)
 {
-	return false;
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	int i, sgi, cpu;
+	int min_sgi = (offset & ~0x3) * 4;
+	int max_sgi = min_sgi + 3;
+	int vcpu_id = vcpu->vcpu_id;
+	u32 *lr, reg;
+	bool updated = false;
+
+	if (!mmio->is_write) {
+		read_sgi_set_clear(vcpu, mmio, offset);
+		return false;
+	}
+
+	reg = mmio_data_read(mmio, ~0);
+
+	/* Clear pending SGIs on distributor side */
+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
+		u8 mask = reg >> (8 * (sgi - min_sgi));
+		if (dist->irq_sgi_sources[vcpu_id][sgi] & mask)
+			updated = true;
+		dist->irq_sgi_sources[vcpu_id][sgi] &= ~mask;
+	}
+
+	/* Clear SGIs already on LRs */
+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
+		lr = &vgic_cpu->vgic_lr[i];
+		sgi = *lr & GICH_LR_VIRTUALID;
+		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
+
+		if (sgi >= min_sgi && sgi <= max_sgi) {
+			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
+				if (*lr & GICH_LR_PENDING_BIT)
+					updated = true;
+				*lr &= GICH_LR_PENDING_BIT;
+			}
+		}
+	}
+
+	return updated;
 }
 
 static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
 				struct kvm_exit_mmio *mmio,
 				phys_addr_t offset)
 {
-	return false;
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	int i, sgi, cpu;
+	int min_sgi = (offset & ~0x3) * 4;
+	int max_sgi = min_sgi + 3;
+	int vcpu_id = vcpu->vcpu_id;
+	u32 *lr, reg;
+	bool updated = false;
+
+	if (!mmio->is_write) {
+		read_sgi_set_clear(vcpu, mmio, offset);
+		return false;
+	}
+
+	reg = mmio_data_read(mmio, ~0);
+
+	/* Set pending SGIs on distributor side */
+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
+		u8 mask = reg >> (8 * (sgi - min_sgi));
+		if ((dist->irq_sgi_sources[vcpu_id][sgi] & mask) != mask)
+			updated = true;
+		dist->irq_sgi_sources[vcpu_id][sgi] |= mask;
+	}
+
+	/* Set active SGIs already on LRs to pending and active */
+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
+		lr = &vgic_cpu->vgic_lr[i];
+		sgi = *lr & GICH_LR_VIRTUALID;
+		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
+
+		if (sgi >= min_sgi && sgi <= max_sgi) {
+			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
+				if (!(*lr & GICH_LR_PENDING_BIT))
+					updated = true;
+				*lr |= GICH_LR_PENDING_BIT;
+			}
+		}
+	}
+
+	return updated;
 }
 
 /*
-- 
1.7.10.4

[PATCH RESEND v2 8/8] KVM: arm-vgic: Support CPU interface reg access

[Christoffer Dall]

Implement support for the CPU interface register access driven by MMIO
address offsets from the CPU interface base address.  Useful for user
space to support save/restore of the VGIC state.

This commit adds support only for the same logic as the current VGIC
support, and no more.  For example, the active priority registers are
handled as RAZ/WI, just like setting priorities on the emulated
distributor.

Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Alexander Graf <agraf@suse.de>
---
 virt/kvm/arm/vgic.c |   81 ++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 73 insertions(+), 8 deletions(-)

diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 4e8c3ab..3cfdd4d 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -71,6 +71,10 @@
 #define VGIC_ADDR_UNDEF		(-1)
 #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
 
+#define PRODUCT_ID_KVM		0x4b	/* ASCII code K */
+#define IMPLEMENTER_ARM		0x43b
+#define GICC_ARCH_VERSION_V2	0x2
+
 /* Physical address of vgic virtual cpu interface */
 static phys_addr_t vgic_vcpu_base;
 
@@ -312,7 +316,7 @@ static bool handle_mmio_misc(struct kvm_vcpu *vcpu,
 	u32 word_offset = offset & 3;
 
 	switch (offset & ~3) {
-	case 0:			/* CTLR */
+	case 0:			/* GICD_CTLR */
 		reg = vcpu->kvm->arch.vgic.enabled;
 		vgic_reg_access(mmio, &reg, word_offset,
 				ACCESS_READ_VALUE | ACCESS_WRITE_VALUE);
@@ -323,15 +327,15 @@ static bool handle_mmio_misc(struct kvm_vcpu *vcpu,
 		}
 		break;
 
-	case 4:			/* TYPER */
+	case 4:			/* GICD_TYPER */
 		reg  = (atomic_read(&vcpu->kvm->online_vcpus) - 1) << 5;
 		reg |= (VGIC_NR_IRQS >> 5) - 1;
 		vgic_reg_access(mmio, &reg, word_offset,
 				ACCESS_READ_VALUE | ACCESS_WRITE_IGNORED);
 		break;
 
-	case 8:			/* IIDR */
-		reg = 0x4B00043B;
+	case 8:			/* GICD_IIDR */
+		reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
 		vgic_reg_access(mmio, &reg, word_offset,
 				ACCESS_READ_VALUE | ACCESS_WRITE_IGNORED);
 		break;
@@ -1682,9 +1686,70 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
 static bool handle_cpu_mmio_misc(struct kvm_vcpu *vcpu,
 				 struct kvm_exit_mmio *mmio, phys_addr_t offset)
 {
-	return true;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	u32 reg, mask = 0, shift = 0;
+	bool updated = false;
+
+	switch (offset & ~0x3) {
+	case GIC_CPU_CTRL:
+		mask = GICH_VMCR_CTRL_MASK;
+		shift = GICH_VMCR_CTRL_SHIFT;
+		break;
+	case GIC_CPU_PRIMASK:
+		mask = GICH_VMCR_PRIMASK_MASK;
+		shift = GICH_VMCR_PRIMASK_SHIFT;
+		break;
+	case GIC_CPU_BINPOINT:
+		mask = GICH_VMCR_BINPOINT_MASK;
+		shift = GICH_VMCR_BINPOINT_SHIFT;
+		break;
+	case GIC_CPU_ALIAS_BINPOINT:
+		mask = GICH_VMCR_ALIAS_BINPOINT_MASK;
+		shift = GICH_VMCR_ALIAS_BINPOINT_SHIFT;
+		break;
+	}
+
+	if (!mmio->is_write) {
+		reg = (vgic_cpu->vgic_vmcr & mask) >> shift;
+		mmio_data_write(mmio, ~0, reg);
+	} else {
+		reg = mmio_data_read(mmio, ~0);
+		reg = (reg << shift) & mask;
+		if (reg != (vgic_cpu->vgic_vmcr & mask))
+			updated = true;
+		vgic_cpu->vgic_vmcr &= ~mask;
+		vgic_cpu->vgic_vmcr |= reg;
+	}
+	return updated;
 }
 
+static bool handle_mmio_abpr(struct kvm_vcpu *vcpu,
+			     struct kvm_exit_mmio *mmio, phys_addr_t offset)
+{
+	return handle_cpu_mmio_misc(vcpu, mmio, GIC_CPU_ALIAS_BINPOINT);
+}
+
+static bool handle_cpu_mmio_ident(struct kvm_vcpu *vcpu,
+				  struct kvm_exit_mmio *mmio,
+				  phys_addr_t offset)
+{
+	u32 reg;
+
+	if (mmio->is_write)
+		return false;
+
+	/* GICC_IIDR */
+	reg = (PRODUCT_ID_KVM << 20) |
+	      (GICC_ARCH_VERSION_V2 << 16) |
+	      (IMPLEMENTER_ARM << 0);
+	mmio_data_write(mmio, ~0, reg);
+	return false;
+}
+
+/*
+ * CPU Interface Register accesses - these are not accessed by the VM, but by
+ * user space for saving and restoring VGIC state.
+ */
 static const struct mmio_range vgic_cpu_ranges[] = {
 	{
 		.base		= GIC_CPU_CTRL,
@@ -1694,17 +1759,17 @@ static const struct mmio_range vgic_cpu_ranges[] = {
 	{
 		.base		= GIC_CPU_ALIAS_BINPOINT,
 		.len		= 4,
-		.handle_mmio	= handle_cpu_mmio_misc,
+		.handle_mmio	= handle_mmio_abpr,
 	},
 	{
 		.base		= GIC_CPU_ACTIVEPRIO,
 		.len		= 16,
-		.handle_mmio	= handle_cpu_mmio_misc,
+		.handle_mmio	= handle_mmio_raz_wi,
 	},
 	{
 		.base		= GIC_CPU_IDENT,
 		.len		= 4,
-		.handle_mmio	= handle_cpu_mmio_misc,
+		.handle_mmio	= handle_cpu_mmio_ident,
 	},
 };
 
-- 
1.7.10.4

[PATCH RESEND v2 2/8] KVM: arm-vgic: Support KVM_CREATE_DEVICE for VGIC

[Marc Zyngier]

Hi Christoffer,

On 2013-10-22 10:08, Christoffer Dall wrote:
> Support creating the ARM VGIC device through the KVM_CREATE_DEVICE
> ioctl, which can then later be leveraged to use the
> KVM_{GET/SET}_DEVICE_ATTR, which is useful both for setting addresses 
> in
> a more generic API than the ARM-specific one and is useful for
> save/restore of VGIC state.
>
> Adds KVM_CAP_DEVICE_CTRL to ARM capabilities.
>
> Note that we change the check for creating a VGIC from bailing out if
> any VCPUs were created to bailing if any VCPUs were ever run.  This 
> is
> an important distinction that doesn't break anything, but allows
> creating the VGIC after the VCPUs have been created.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> Reviewed-by: Alexander Graf <agraf@suse.de>
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |   10 ++++++
>  arch/arm/include/uapi/asm/kvm.h                |    1 -
>  arch/arm/kvm/arm.c                             |    1 +
>  include/linux/kvm_host.h                       |    1 +
>  include/uapi/linux/kvm.h                       |    1 +
>  virt/kvm/arm/vgic.c                            |   46
> ++++++++++++++++++++++--
>  virt/kvm/kvm_main.c                            |    5 +++
>  7 files changed, 62 insertions(+), 3 deletions(-)
>  create mode 100644 Documentation/virtual/kvm/devices/arm-vgic.txt
>
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> b/Documentation/virtual/kvm/devices/arm-vgic.txt
> new file mode 100644
> index 0000000..38f27f7
> --- /dev/null
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -0,0 +1,10 @@
> +ARM Virtual Generic Interrupt Controller (VGIC)
> +===============================================
> +
> +Device types supported:
> +  KVM_DEV_TYPE_ARM_VGIC_V2     ARM Generic Interrupt Controller v2.0
> +
> +Only one VGIC instance may be instantiated through either this API 
> or the
> +legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM
> interrupt
> +controller, requiring emulated user-space devices to inject
> interrupts to the
> +VGIC instead of directly to CPUs.
> diff --git a/arch/arm/include/uapi/asm/kvm.h
> b/arch/arm/include/uapi/asm/kvm.h
> index c1ee007..1c85102 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -142,7 +142,6 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM_VFP_FPINST		0x1009
>  #define KVM_REG_ARM_VFP_FPINST2		0x100A
>
> -

Nit: pointless change?

>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
>  #define KVM_ARM_IRQ_TYPE_MASK		0xff
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 2b1091a..ab96af2 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -187,6 +187,7 @@ int kvm_dev_ioctl_check_extension(long ext)
>  	case KVM_CAP_IRQCHIP:
>  		r = vgic_present;
>  		break;
> +	case KVM_CAP_DEVICE_CTRL:
>  	case KVM_CAP_USER_MEMORY:
>  	case KVM_CAP_SYNC_MMU:
>  	case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index ca645a0..2906b79 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1065,6 +1065,7 @@ struct kvm_device *kvm_device_from_filp(struct
> file *filp);
>
>  extern struct kvm_device_ops kvm_mpic_ops;
>  extern struct kvm_device_ops kvm_xics_ops;
> +extern struct kvm_device_ops kvm_arm_vgic_ops;
>
>  #ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
>
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 99c2533..2d50233 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -843,6 +843,7 @@ struct kvm_device_attr {
>  #define KVM_DEV_TYPE_FSL_MPIC_20	1
>  #define KVM_DEV_TYPE_FSL_MPIC_42	2
>  #define KVM_DEV_TYPE_XICS		3
> +#define KVM_DEV_TYPE_ARM_VGIC_V2	4

How about calling it GIC_V2 instead of VGIC_V2? As far as the guest is 
concerned, this is a "true" GIC, and the other names don't imply any 
distinction either...

>  /*
>   * ioctls for VM fds
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 5ce100f..79a8bae 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -1434,15 +1434,23 @@ out:
>
>  int kvm_vgic_create(struct kvm *kvm)
>  {
> -	int ret = 0;
> +	int i, ret = 0;
> +	struct kvm_vcpu *vcpu;
>
>  	mutex_lock(&kvm->lock);
>
> -	if (atomic_read(&kvm->online_vcpus) || kvm->arch.vgic.vctrl_base) {
> +	if (kvm->arch.vgic.vctrl_base) {
>  		ret = -EEXIST;
>  		goto out;
>  	}
>
> +	kvm_for_each_vcpu(i, vcpu, kvm) {
> +		if (vcpu->arch.has_run_once) {
> +			ret = -EBUSY;
> +			goto out;
> +		}
> +	}

Isn't this racy? What prevents anyone from starting a CPU while you're 
in this loop?

>  	spin_lock_init(&kvm->arch.vgic.lock);
>  	kvm->arch.vgic.vctrl_base = vgic_vctrl_base;
>  	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
> @@ -1511,3 +1519,37 @@ int kvm_vgic_set_addr(struct kvm *kvm,
> unsigned long type, u64 addr)
>  	mutex_unlock(&kvm->lock);
>  	return r;
>  }
> +
> +static int vgic_set_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
> +{
> +	return -ENXIO;
> +}
> +
> +static int vgic_get_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
> +{
> +	return -ENXIO;
> +}
> +
> +static int vgic_has_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
> +{
> +	return -ENXIO;
> +}
> +
> +static void vgic_destroy(struct kvm_device *dev)
> +{
> +	kfree(dev);
> +}
> +
> +static int vgic_create(struct kvm_device *dev, u32 type)
> +{
> +	return kvm_vgic_create(dev->kvm);
> +}
> +
> +struct kvm_device_ops kvm_arm_vgic_ops = {
> +	.name = "kvm-arm-vgic",
> +	.create = vgic_create,
> +	.destroy = vgic_destroy,
> +	.set_attr = vgic_set_attr,
> +	.get_attr = vgic_get_attr,
> +	.has_attr = vgic_has_attr,
> +};
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index bf040c4..534fd3a 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2265,6 +2265,11 @@ static int kvm_ioctl_create_device(struct kvm 
> *kvm,
>  		ops = &kvm_xics_ops;
>  		break;
>  #endif
> +#ifdef CONFIG_KVM_ARM_VGIC
> +	case KVM_DEV_TYPE_ARM_VGIC_V2:
> +		ops = &kvm_arm_vgic_ops;
> +		break;
> +#endif
>  	default:
>  		return -ENODEV;
>  	}

Cheers,

         M.
-- 
Who you jivin' with that Cosmik Debris?

[PATCH RESEND v2 3/8] KVM: arm-vgic: Set base addr through device API

[Marc Zyngier]

On 2013-10-22 10:08, Christoffer Dall wrote:
> Support setting the distributor and cpu interface base addresses in 
> the
> VM physical address space through the KVM_{SET,GET}_DEVICE_ATTR API
> in addition to the ARM specific API.
>
> This has the added benefit of being able to share more code in user
> space and do things in a uniform maner.

                                    manner?

> Also deprecate the older API at the same time, but backwards
> compatibility will be maintained.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> Reviewed-by: Alexander Graf <agraf@suse.de>
> ---
>  Documentation/virtual/kvm/api.txt              |    6 +-
>  Documentation/virtual/kvm/devices/arm-vgic.txt |   11 +++
>  arch/arm/include/uapi/asm/kvm.h                |    9 +++
>  arch/arm/kvm/arm.c                             |    2 +-
>  include/kvm/arm_vgic.h                         |    2 +-
>  virt/kvm/arm/vgic.c                            |   90
> ++++++++++++++++++++----
>  6 files changed, 105 insertions(+), 15 deletions(-)
>
> diff --git a/Documentation/virtual/kvm/api.txt
> b/Documentation/virtual/kvm/api.txt
> index 858aecf..d68b6c2 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -2324,7 +2324,7 @@ This ioctl returns the guest registers that are
> supported for the
>  KVM_GET_ONE_REG/KVM_SET_ONE_REG calls.
>
>
> -4.84 KVM_ARM_SET_DEVICE_ADDR
> +4.84 KVM_ARM_SET_DEVICE_ADDR (deprecated)
>
>  Capability: KVM_CAP_ARM_SET_DEVICE_ADDR
>  Architectures: arm, arm64
> @@ -2362,6 +2362,10 @@ must be called after calling
> KVM_CREATE_IRQCHIP, but before calling
>  KVM_RUN on any of the VCPUs.  Calling this ioctl twice for any of 
> the
>  base addresses will return -EEXIST.
>
> +Note, this IOCTL is deprecated and the more flexible 
> SET/GET_DEVICE_ATTR API
> +should be used instead.
> +
> +
>  4.85 KVM_PPC_RTAS_DEFINE_TOKEN
>
>  Capability: KVM_CAP_PPC_RTAS
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index 38f27f7..c9febb2 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -8,3 +8,14 @@ Only one VGIC instance may be instantiated through
> either this API or the
>  legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM
> interrupt
>  controller, requiring emulated user-space devices to inject
> interrupts to the
>  VGIC instead of directly to CPUs.
> +
> +Groups:
> +  KVM_DEV_ARM_VGIC_GRP_ADDR
> +  Attributes:
> +    KVM_VGIC_V2_ADDR_TYPE_DIST (rw, 64-bit)
> +      Base address in the guest physical address space of the GIC
> distributor
> +      register mappings.
> +
> +    KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
> +      Base address in the guest physical address space of the GIC
> virtual cpu
> +      interface register mappings.
> diff --git a/arch/arm/include/uapi/asm/kvm.h
> b/arch/arm/include/uapi/asm/kvm.h
> index 1c85102..587f1ae 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -142,6 +142,15 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM_VFP_FPINST		0x1009
>  #define KVM_REG_ARM_VFP_FPINST2		0x100A
>
> +/* Device Control API: ARM VGIC */
> +#define KVM_DEV_ARM_VGIC_GRP_ADDR	0
> +#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS	1
> +#define KVM_DEV_ARM_VGIC_GRP_CPU_REGS	2
> +#define   KVM_DEV_ARM_VGIC_CPUID_SHIFT	32
> +#define   KVM_DEV_ARM_VGIC_CPUID_MASK	(0xffULL <<
> KVM_DEV_ARM_VGIC_CPUID_SHIFT)
> +#define   KVM_DEV_ARM_VGIC_OFFSET_SHIFT	0
> +#define   KVM_DEV_ARM_VGIC_OFFSET_MASK	(0xffffffffULL <<
> KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
> +
>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
>  #define KVM_ARM_IRQ_TYPE_MASK		0xff
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index ab96af2..3ecee45 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -773,7 +773,7 @@ static int kvm_vm_ioctl_set_device_addr(struct 
> kvm *kvm,
>  	case KVM_ARM_DEVICE_VGIC_V2:
>  		if (!vgic_present)
>  			return -ENXIO;
> -		return kvm_vgic_set_addr(kvm, type, dev_addr->addr);
> +		return kvm_vgic_addr(kvm, type, &dev_addr->addr, true);
>  	default:
>  		return -ENODEV;
>  	}
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index 7e2d158..be85127 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -144,7 +144,7 @@ struct kvm_run;
>  struct kvm_exit_mmio;
>
>  #ifdef CONFIG_KVM_ARM_VGIC
> -int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 
> addr);
> +int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr,
> bool write);
>  int kvm_vgic_hyp_init(void);
>  int kvm_vgic_init(struct kvm *kvm);
>  int kvm_vgic_create(struct kvm *kvm);
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 79a8bae..d9c0fc5 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -1479,6 +1479,12 @@ static int vgic_ioaddr_assign(struct kvm *kvm,
> phys_addr_t *ioaddr,
>  {
>  	int ret;
>
> +	if (addr & ~KVM_PHYS_MASK)
> +		return -E2BIG;
> +
> +	if (addr & (SZ_4K - 1))
> +		return -EINVAL;
> +
>  	if (!IS_VGIC_ADDR_UNDEF(*ioaddr))
>  		return -EEXIST;
>  	if (addr + size < addr)
> @@ -1491,26 +1497,41 @@ static int vgic_ioaddr_assign(struct kvm
> *kvm, phys_addr_t *ioaddr,
>  	return ret;
>  }
>
> -int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
> +/**
> + * kvm_vgic_addr - set or get vgic VM base addresses
> + * @kvm:   pointer to the vm struct
> + * @type:  the VGIC addr type, one of KVM_VGIC_V2_ADDR_TYPE_XXX
> + * @addr:  pointer to address value
> + * @write: if true set the address in the VM address space, if false
> read the
> + *          address
> + *
> + * Set or get the vgic base addresses for the distributor and the
> virtual CPU
> + * interface in the VM physical address space.  These addresses are
> properties
> + * of the emulated core/SoC and therefore user space initially knows 
> this
> + * information.
> + */
> +int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr,
> bool write)
>  {
>  	int r = 0;
>  	struct vgic_dist *vgic = &kvm->arch.vgic;
>
> -	if (addr & ~KVM_PHYS_MASK)
> -		return -E2BIG;
> -
> -	if (addr & (SZ_4K - 1))
> -		return -EINVAL;
> -
>  	mutex_lock(&kvm->lock);
>  	switch (type) {
>  	case KVM_VGIC_V2_ADDR_TYPE_DIST:
> -		r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
> -				       addr, KVM_VGIC_V2_DIST_SIZE);
> +		if (write) {
> +			r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
> +					       *addr, KVM_VGIC_V2_DIST_SIZE);
> +		} else {
> +			*addr = vgic->vgic_dist_base;
> +		}
>  		break;
>  	case KVM_VGIC_V2_ADDR_TYPE_CPU:
> -		r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
> -				       addr, KVM_VGIC_V2_CPU_SIZE);
> +		if (write) {
> +			r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
> +					       *addr, KVM_VGIC_V2_CPU_SIZE);
> +		} else {
> +			*addr = vgic->vgic_cpu_base;
> +		}
>  		break;
>  	default:
>  		r = -ENODEV;
> @@ -1522,16 +1543,61 @@ int kvm_vgic_set_addr(struct kvm *kvm,
> unsigned long type, u64 addr)
>
>  static int vgic_set_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
>  {
> +	int r;
> +
> +	switch (attr->group) {
> +	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 addr;
> +		unsigned long type = (unsigned long)attr->attr;
> +
> +		if (copy_from_user(&addr, uaddr, sizeof(addr)))
> +			return -EFAULT;
> +
> +		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
> +		return (r == -ENODEV) ? -ENXIO : r;
> +	}
> +	}
> +
>  	return -ENXIO;
>  }
>
>  static int vgic_get_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
>  {
> -	return -ENXIO;
> +	int r = ENXIO;

Shouldn't this be a negative number?

> +	switch (attr->group) {
> +	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 addr;
> +		unsigned long type = (unsigned long)attr->attr;
> +
> +		r = kvm_vgic_addr(dev->kvm, type, &addr, false);
> +		if (r)
> +			return (r == -ENODEV) ? -ENXIO : r;
> +
> +		r = 0;

Isn't r already zero at this point?

> +		if (copy_to_user(uaddr, &addr, sizeof(addr)))
> +			return -EFAULT;
> +	}
> +	}
> +
> +	return r;
>  }
>
>  static int vgic_has_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
>  {
> +	phys_addr_t offset;
> +
> +	switch (attr->group) {
> +	case KVM_DEV_ARM_VGIC_GRP_ADDR:
> +		switch (attr->attr) {
> +		case KVM_VGIC_V2_ADDR_TYPE_DIST:
> +		case KVM_VGIC_V2_ADDR_TYPE_CPU:
> +			return 0;
> +		}
> +		break;
> +	}
>  	return -ENXIO;
>  }

Cheers,

         M.
-- 
Who you jivin' with that Cosmik Debris?

[PATCH RESEND v2 6/8] KVM: arm-vgic: Add vgic reg access from dev attr

[Marc Zyngier]

On 2013-10-22 10:08, Christoffer Dall wrote:
> Add infrastructure to handle distributor and cpu interface register
> accesses through the KVM_{GET/SET}_DEVICE_ATTR interface by adding 
> the
> KVM_DEV_ARM_VGIC_GRP_DIST_REGS and KVM_DEV_ARM_VGIC_GRP_CPU_REGS 
> groups
> and defining the semantics of the attr field to be the MMIO offset as
> specified in the GICv2 specs.
>
> Missing register accesses or other changes in individual register 
> access
> functions to support save/restore of the VGIC state is added in
> subsequent patches.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> Reviewed-by: Alexander Graf <agraf@suse.de>
>
> ---
> Changelog[v2]:
>  - Added implementation specific format for the GICC_APRn registers.
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |   50 +++++++++
>  virt/kvm/arm/vgic.c                            |  143
> ++++++++++++++++++++++++
>  2 files changed, 193 insertions(+)
>
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index c9febb2..e6416f8e 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -19,3 +19,53 @@ Groups:
>      KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
>        Base address in the guest physical address space of the GIC
> virtual cpu
>        interface register mappings.
> +
> +  KVM_DEV_ARM_VGIC_GRP_DIST_REGS
> +  Attributes:
> +    The attr field of kvm_device_attr encodes two values:
> +    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
> +    values:   |    reserved   |   cpu id   |      offset     |
> +
> +    All distributor regs are (rw, 32-bit)
> +
> +    The offset is relative to the "Distributor base address" as
> defined in the
> +    GICv2 specs.  Getting or setting such a register has the same 
> effect as
> +    reading or writing the register on the actual hardware from the 
> cpu
> +    specified with cpu id field.  Note that most distributor fields 
> are not
> +    banked, but return the same value regardless of the cpu id used
> to access
> +    the register.
> +  Limitations:
> +    - Priorities are not implemented, and registers are RAZ/WI
> +  Errors:
> +    - ENODEV: Getting or setting this register is not yet supported

-ENODEV?

> +  KVM_DEV_ARM_VGIC_GRP_CPU_REGS
> +  Attributes:
> +    The attr field of kvm_device_attr encodes two values:
> +    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
> +    values:   |    reserved   |   cpu id   |      offset     |
> +
> +    All CPU regs are (rw, 32-bit)

Nit: CPU interface registers

> +    The offset specifies the offset from the "CPU interface base 
> address" as
> +    defined in the GICv2 specs.  Getting or setting such a register 
> has the
> +    same effect as reading or writing the register on the actual 
> hardware.
> +
> +    The Active Priorities Registers APRn are implementation defined,
> so we set a
> +    fixed format for our implementation that fits with the model of 
> a "GICv2
> +    impementation without the security extensions" which we present 
> to the

implementation

> +    guest.  This interface always exposes four register APR[0-3]
> describing the
> +    maximum possible 128 preemption levels.  The semantics of the 
> register
> +    indicate if any interrupts in a given preemption level are in 
> the active
> +    state by setting the corresponding bit.
> +
> +    Thus, preemption level X has one or more active interrupts if
> and only if:
> +
> +      APRn[X mod 32] == 0b1,  where n = X / 32
> +
> +    Bits for undefined preemption levels are RAZ/WI.
> +
> +  Limitations:
> +    - Priorities are not implemented, and registers are RAZ/WI
> +  Errors:
> +    - ENODEV: Getting or setting this register is not yet supported
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 1148a2e..f2dc72a 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -589,11 +589,29 @@ static bool handle_mmio_sgi_reg(struct kvm_vcpu 
> *vcpu,
>  	return false;
>  }
>
> +static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
> +				  struct kvm_exit_mmio *mmio,
> +				  phys_addr_t offset)
> +{
> +	return false;
> +}
> +
> +static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
> +				struct kvm_exit_mmio *mmio,
> +				phys_addr_t offset)
> +{
> +	return false;
> +}
> +
>  /*
>   * I would have liked to use the kvm_bus_io_*() API instead, but it
>   * cannot cope with banked registers (only the VM pointer is passed
>   * around, and we need the vcpu). One of these days, someone please
>   * fix it!
> + *
> + * Note that the handle_mmio implementations should not use the 
> phys_addr
> + * field from the kvm_exit_mmio struct as this will not have any 
> sane values
> + * when used to save/restore state from user space.

This is quite ugly... I don't think we'd ever use that field directly, 
but reusing a well known structure for that purpose is very messy. I 
believe we'd be better off creating our own structure instead of 
re-purposing am existing one.

The other possibility would be to properly fill-in the phys_addr field. 
How difficult would that be?

>   */
>  struct mmio_range {
>  	phys_addr_t base;
> @@ -663,6 +681,16 @@ static const struct mmio_range 
> vgic_dist_ranges[] = {
>  		.len		= 4,
>  		.handle_mmio	= handle_mmio_sgi_reg,
>  	},
> +	{
> +		.base		= GIC_DIST_SGI_CLEAR,
> +		.len		= VGIC_NR_SGIS,
> +		.handle_mmio	= handle_mmio_sgi_clear,
> +	},
> +	{
> +		.base		= GIC_DIST_SGI_SET,
> +		.len		= VGIC_NR_SGIS,
> +		.handle_mmio	= handle_mmio_sgi_set,
> +	},
>  	{}
>  };
>
> @@ -1541,6 +1569,80 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned
> long type, u64 *addr, bool write)
>  	return r;
>  }
>
> +static bool handle_cpu_mmio_misc(struct kvm_vcpu *vcpu,
> +				 struct kvm_exit_mmio *mmio, phys_addr_t offset)
> +{
> +	return true;
> +}
> +
> +static const struct mmio_range vgic_cpu_ranges[] = {
> +	{
> +		.base		= GIC_CPU_CTRL,
> +		.len		= 12,
> +		.handle_mmio	= handle_cpu_mmio_misc,
> +	},
> +	{
> +		.base		= GIC_CPU_ALIAS_BINPOINT,
> +		.len		= 4,
> +		.handle_mmio	= handle_cpu_mmio_misc,
> +	},
> +	{
> +		.base		= GIC_CPU_ACTIVEPRIO,
> +		.len		= 16,
> +		.handle_mmio	= handle_cpu_mmio_misc,
> +	},
> +	{
> +		.base		= GIC_CPU_IDENT,
> +		.len		= 4,
> +		.handle_mmio	= handle_cpu_mmio_misc,
> +	},
> +};
> +
> +static struct kvm_exit_mmio dev_attr_mmio = { .len = 4 };

I'm not very fond of a half-initialized structure here. How about 
moving this "4" to the location where it is used?
Actually, what if we have several users of this through 
vgic_has_attr_regs at the same time? It feels incredibly racy. I suggest 
you nuke it and move it to live on the stack in vgic_has_attr_regs.

> +static int vgic_attr_regs_access(struct kvm_device *dev,
> +				 struct kvm_device_attr *attr,
> +				 u32 *reg, bool is_write)
> +{
> +	const struct mmio_range *r = NULL;
> +	phys_addr_t offset;
> +	int cpuid;
> +	struct kvm_vcpu *vcpu;
> +	struct kvm_exit_mmio mmio;
> +
> +	offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> +	cpuid = (attr->attr & KVM_DEV_ARM_VGIC_CPUID_MASK) >>
> +		KVM_DEV_ARM_VGIC_CPUID_SHIFT;
> +
> +	if (cpuid >= atomic_read(&dev->kvm->online_vcpus))
> +		return -EINVAL;
> +
> +	vcpu = kvm_get_vcpu(dev->kvm, cpuid);
> +
> +	mmio.len = 4;
> +	mmio.is_write = is_write;
> +	if (is_write)
> +		mmio_data_write(&mmio, ~0, *reg);
> +
> +	if (attr->group == KVM_DEV_ARM_VGIC_GRP_DIST_REGS)
> +		r = find_matching_range(vgic_dist_ranges, &mmio, offset);
> +	else if (attr->group == KVM_DEV_ARM_VGIC_GRP_CPU_REGS)
> +		r = find_matching_range(vgic_cpu_ranges, &mmio, offset);

How about having a switch statement instead?

> +	if (unlikely(!r || !r->handle_mmio))
> +		return -ENXIO;
> +
> +	spin_lock(&vcpu->kvm->arch.vgic.lock);
> +	offset -= r->base;
> +	r->handle_mmio(vcpu, &mmio, offset);
> +	spin_unlock(&vcpu->kvm->arch.vgic.lock);
> +
> +	if (!is_write)
> +		*reg = mmio_data_read(&mmio, ~0);
> +
> +	return 0;
> +}
> +
>  static int vgic_set_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
>  {
>  	int r;
> @@ -1557,6 +1659,18 @@ static int vgic_set_attr(struct kvm_device
> *dev, struct kvm_device_attr *attr)
>  		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
>  		return (r == -ENODEV) ? -ENXIO : r;
>  	}
> +
> +	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> +	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
> +		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
> +		u32 reg;
> +
> +		if (get_user(reg, uaddr))
> +			return -EFAULT;
> +
> +		return vgic_attr_regs_access(dev, attr, &reg, true);
> +	}
> +
>  	}
>
>  	return -ENXIO;
> @@ -1579,12 +1693,35 @@ static int vgic_get_attr(struct kvm_device
> *dev, struct kvm_device_attr *attr)
>  		r = 0;
>  		if (copy_to_user(uaddr, &addr, sizeof(addr)))
>  			return -EFAULT;
> +		break;
>  	}
> +
> +	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> +	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
> +		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
> +		u32 reg = 0;
> +
> +		r = vgic_attr_regs_access(dev, attr, &reg, false);
> +		if (r)
> +			return r;
> +		r = put_user(reg, uaddr);
> +		break;
> +	}
> +
>  	}
>
>  	return r;
>  }
>
> +static int vgic_has_attr_regs(const struct mmio_range *ranges,
> +			      phys_addr_t offset)
> +{
> +	if (find_matching_range(ranges, &dev_attr_mmio, offset))
> +		return 0;
> +	else
> +		return -ENXIO;
> +}
> +
>  static int vgic_has_attr(struct kvm_device *dev, struct
> kvm_device_attr *attr)
>  {
>  	phys_addr_t offset;
> @@ -1597,6 +1734,12 @@ static int vgic_has_attr(struct kvm_device
> *dev, struct kvm_device_attr *attr)
>  			return 0;
>  		}
>  		break;
> +	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> +		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> +		return vgic_has_attr_regs(vgic_dist_ranges, offset);
> +	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS:
> +		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> +		return vgic_has_attr_regs(vgic_cpu_ranges, offset);
>  	}
>  	return -ENXIO;
>  }

-- 
Who you jivin' with that Cosmik Debris?

[PATCH RESEND v2 7/8] KVM: arm-vgic: Add GICD_SPENDSGIR and GICD_CPENDSGIR handlers

[Marc Zyngier]

On 2013-10-22 10:08, Christoffer Dall wrote:
> Handle MMIO accesses to the two registers which should support both 
> the
> case where the VMs want to read/write either of these registers and 
> the
> case where user space reads/writes these registers to do save/restore 
> of
> the VGIC state.
>
> Note that the added complexity compared to simple set/clear enable
> registers stems from the bookkeping of source cpu ids.  It may be
> possible to change the underlying data structure to simplify the
> complexity, but since this is not in the critical path, at all, this 
> is
> left as an interesting excercise to the reader.
>
> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> Reviewed-by: Alexander Graf <agraf@suse.de>
>
> ---
> Changelog[v2]:
>  - Use struct kvm_exit_mmio accessors for ->data field.
> ---
>  virt/kvm/arm/vgic.c |  114
> ++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 112 insertions(+), 2 deletions(-)
>
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index f2dc72a..4e8c3ab 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -589,18 +589,128 @@ static bool handle_mmio_sgi_reg(struct 
> kvm_vcpu *vcpu,
>  	return false;
>  }
>
> +static void read_sgi_set_clear(struct kvm_vcpu *vcpu,
> +			       struct kvm_exit_mmio *mmio,
> +			       phys_addr_t offset)

set_clear is a bit unclear. How about reset?

> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +	int i, sgi, cpu;
> +	int min_sgi = (offset & ~0x3) * 4;
> +	int max_sgi = min_sgi + 3;
> +	int vcpu_id = vcpu->vcpu_id;
> +	u32 lr, reg = 0;
> +
> +	/* Copy source SGIs from distributor side */
> +	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> +		int shift = 8 * (sgi - min_sgi);
> +		reg |= (u32)dist->irq_sgi_sources[vcpu_id][sgi] << shift;
> +	}
> +
> +	/* Copy source SGIs already on LRs */
> +	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> +		lr = vgic_cpu->vgic_lr[i];
> +		sgi = lr & GICH_LR_VIRTUALID;
> +		cpu = (lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;

Please wrap these  lr accesses into separate functions. There is quite 
a bit of duplication in this patch and I wonder if we could factor 
things a bit.

At least, please isolate what is emulation related from what is 
actually what the underlying HW provides. It will help mitigating my 
headache in the future... ;-)

> +		if (sgi >= min_sgi && sgi <= max_sgi) {
> +			if (lr & GICH_LR_STATE)
> +				reg |= (1 << cpu) << (8 * (sgi - min_sgi));
> +		}
> +	}
> +
> +	mmio_data_write(mmio, ~0, reg);
> +}
> +
>  static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
>  				  struct kvm_exit_mmio *mmio,
>  				  phys_addr_t offset)
>  {
> -	return false;
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +	int i, sgi, cpu;
> +	int min_sgi = (offset & ~0x3) * 4;
> +	int max_sgi = min_sgi + 3;
> +	int vcpu_id = vcpu->vcpu_id;
> +	u32 *lr, reg;
> +	bool updated = false;
> +
> +	if (!mmio->is_write) {
> +		read_sgi_set_clear(vcpu, mmio, offset);
> +		return false;
> +	}
> +
> +	reg = mmio_data_read(mmio, ~0);
> +
> +	/* Clear pending SGIs on distributor side */
> +	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> +		u8 mask = reg >> (8 * (sgi - min_sgi));
> +		if (dist->irq_sgi_sources[vcpu_id][sgi] & mask)
> +			updated = true;
> +		dist->irq_sgi_sources[vcpu_id][sgi] &= ~mask;
> +	}
> +
> +	/* Clear SGIs already on LRs */
> +	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> +		lr = &vgic_cpu->vgic_lr[i];
> +		sgi = *lr & GICH_LR_VIRTUALID;
> +		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
> +
> +		if (sgi >= min_sgi && sgi <= max_sgi) {
> +			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
> +				if (*lr & GICH_LR_PENDING_BIT)
> +					updated = true;
> +				*lr &= GICH_LR_PENDING_BIT;
> +			}
> +		}
> +	}
> +
> +	return updated;
>  }
>
>  static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
>  				struct kvm_exit_mmio *mmio,
>  				phys_addr_t offset)
>  {
> -	return false;
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +	int i, sgi, cpu;
> +	int min_sgi = (offset & ~0x3) * 4;
> +	int max_sgi = min_sgi + 3;
> +	int vcpu_id = vcpu->vcpu_id;
> +	u32 *lr, reg;
> +	bool updated = false;
> +
> +	if (!mmio->is_write) {
> +		read_sgi_set_clear(vcpu, mmio, offset);
> +		return false;
> +	}
> +
> +	reg = mmio_data_read(mmio, ~0);
> +
> +	/* Set pending SGIs on distributor side */
> +	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> +		u8 mask = reg >> (8 * (sgi - min_sgi));
> +		if ((dist->irq_sgi_sources[vcpu_id][sgi] & mask) != mask)
> +			updated = true;
> +		dist->irq_sgi_sources[vcpu_id][sgi] |= mask;
> +	}
> +
> +	/* Set active SGIs already on LRs to pending and active */
> +	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> +		lr = &vgic_cpu->vgic_lr[i];
> +		sgi = *lr & GICH_LR_VIRTUALID;
> +		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
> +
> +		if (sgi >= min_sgi && sgi <= max_sgi) {
> +			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
> +				if (!(*lr & GICH_LR_PENDING_BIT))
> +					updated = true;
> +				*lr |= GICH_LR_PENDING_BIT;
> +			}
> +		}
> +	}
> +
> +	return updated;
>  }
>
>  /*

Overall, I feel like I've read the same function three times. Hint, 
hint... ;-)

         M.
-- 
Who you jivin' with that Cosmik Debris?

[PATCH RESEND v2 2/8] KVM: arm-vgic: Support KVM_CREATE_DEVICE for VGIC

[Christoffer Dall]

On Wed, Oct 23, 2013 at 03:55:16PM +0100, Marc Zyngier wrote:
> Hi Christoffer,
> 
> On 2013-10-22 10:08, Christoffer Dall wrote:
> >Support creating the ARM VGIC device through the KVM_CREATE_DEVICE
> >ioctl, which can then later be leveraged to use the
> >KVM_{GET/SET}_DEVICE_ATTR, which is useful both for setting
> >addresses in
> >a more generic API than the ARM-specific one and is useful for
> >save/restore of VGIC state.
> >
> >Adds KVM_CAP_DEVICE_CTRL to ARM capabilities.
> >
> >Note that we change the check for creating a VGIC from bailing out if
> >any VCPUs were created to bailing if any VCPUs were ever run.
> >This is
> >an important distinction that doesn't break anything, but allows
> >creating the VGIC after the VCPUs have been created.
> >
> >Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >Reviewed-by: Alexander Graf <agraf@suse.de>
> >---
> > Documentation/virtual/kvm/devices/arm-vgic.txt |   10 ++++++
> > arch/arm/include/uapi/asm/kvm.h                |    1 -
> > arch/arm/kvm/arm.c                             |    1 +
> > include/linux/kvm_host.h                       |    1 +
> > include/uapi/linux/kvm.h                       |    1 +
> > virt/kvm/arm/vgic.c                            |   46
> >++++++++++++++++++++++--
> > virt/kvm/kvm_main.c                            |    5 +++
> > 7 files changed, 62 insertions(+), 3 deletions(-)
> > create mode 100644 Documentation/virtual/kvm/devices/arm-vgic.txt
> >
> >diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> >b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >new file mode 100644
> >index 0000000..38f27f7
> >--- /dev/null
> >+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >@@ -0,0 +1,10 @@
> >+ARM Virtual Generic Interrupt Controller (VGIC)
> >+===============================================
> >+
> >+Device types supported:
> >+  KVM_DEV_TYPE_ARM_VGIC_V2     ARM Generic Interrupt Controller v2.0
> >+
> >+Only one VGIC instance may be instantiated through either this
> >API or the
> >+legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM
> >interrupt
> >+controller, requiring emulated user-space devices to inject
> >interrupts to the
> >+VGIC instead of directly to CPUs.
> >diff --git a/arch/arm/include/uapi/asm/kvm.h
> >b/arch/arm/include/uapi/asm/kvm.h
> >index c1ee007..1c85102 100644
> >--- a/arch/arm/include/uapi/asm/kvm.h
> >+++ b/arch/arm/include/uapi/asm/kvm.h
> >@@ -142,7 +142,6 @@ struct kvm_arch_memory_slot {
> > #define KVM_REG_ARM_VFP_FPINST		0x1009
> > #define KVM_REG_ARM_VFP_FPINST2		0x100A
> >
> >-
> 
> Nit: pointless change?
> 
> > /* KVM_IRQ_LINE irq field index values */
> > #define KVM_ARM_IRQ_TYPE_SHIFT		24
> > #define KVM_ARM_IRQ_TYPE_MASK		0xff
> >diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> >index 2b1091a..ab96af2 100644
> >--- a/arch/arm/kvm/arm.c
> >+++ b/arch/arm/kvm/arm.c
> >@@ -187,6 +187,7 @@ int kvm_dev_ioctl_check_extension(long ext)
> > 	case KVM_CAP_IRQCHIP:
> > 		r = vgic_present;
> > 		break;
> >+	case KVM_CAP_DEVICE_CTRL:
> > 	case KVM_CAP_USER_MEMORY:
> > 	case KVM_CAP_SYNC_MMU:
> > 	case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
> >diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> >index ca645a0..2906b79 100644
> >--- a/include/linux/kvm_host.h
> >+++ b/include/linux/kvm_host.h
> >@@ -1065,6 +1065,7 @@ struct kvm_device *kvm_device_from_filp(struct
> >file *filp);
> >
> > extern struct kvm_device_ops kvm_mpic_ops;
> > extern struct kvm_device_ops kvm_xics_ops;
> >+extern struct kvm_device_ops kvm_arm_vgic_ops;
> >
> > #ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
> >
> >diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> >index 99c2533..2d50233 100644
> >--- a/include/uapi/linux/kvm.h
> >+++ b/include/uapi/linux/kvm.h
> >@@ -843,6 +843,7 @@ struct kvm_device_attr {
> > #define KVM_DEV_TYPE_FSL_MPIC_20	1
> > #define KVM_DEV_TYPE_FSL_MPIC_42	2
> > #define KVM_DEV_TYPE_XICS		3
> >+#define KVM_DEV_TYPE_ARM_VGIC_V2	4
> 
> How about calling it GIC_V2 instead of VGIC_V2? As far as the guest
> is concerned, this is a "true" GIC, and the other names don't imply
> any distinction either...
> 

I thought about this, but we already have exported defines named
VGIC_something and we make all references in the kernel to VGIC in
documentaiton and so on, so I decided against that.  If you insist, do
you also want me to rename/create new defines for all other fields (like
KVM_VGIC_V2_ADDR_TYPE_DIST)?

> > /*
> >  * ioctls for VM fds
> >diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> >index 5ce100f..79a8bae 100644
> >--- a/virt/kvm/arm/vgic.c
> >+++ b/virt/kvm/arm/vgic.c
> >@@ -1434,15 +1434,23 @@ out:
> >
> > int kvm_vgic_create(struct kvm *kvm)
> > {
> >-	int ret = 0;
> >+	int i, ret = 0;
> >+	struct kvm_vcpu *vcpu;
> >
> > 	mutex_lock(&kvm->lock);
> >
> >-	if (atomic_read(&kvm->online_vcpus) || kvm->arch.vgic.vctrl_base) {
> >+	if (kvm->arch.vgic.vctrl_base) {
> > 		ret = -EEXIST;
> > 		goto out;
> > 	}
> >
> >+	kvm_for_each_vcpu(i, vcpu, kvm) {
> >+		if (vcpu->arch.has_run_once) {
> >+			ret = -EBUSY;
> >+			goto out;
> >+		}
> >+	}
> 
> Isn't this racy? What prevents anyone from starting a CPU while
> you're in this loop?
> 

It is indeed racy, nicely spotted!

Will fix in v3.

Thanks,
-Christoffer

[PATCH RESEND v2 3/8] KVM: arm-vgic: Set base addr through device API

[Christoffer Dall]

On Wed, Oct 23, 2013 at 04:10:42PM +0100, Marc Zyngier wrote:
> On 2013-10-22 10:08, Christoffer Dall wrote:
> >Support setting the distributor and cpu interface base addresses
> >in the
> >VM physical address space through the KVM_{SET,GET}_DEVICE_ATTR API
> >in addition to the ARM specific API.
> >
> >This has the added benefit of being able to share more code in user
> >space and do things in a uniform maner.
> 
>                                    manner?
> 
> >Also deprecate the older API at the same time, but backwards
> >compatibility will be maintained.
> >
> >Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >Reviewed-by: Alexander Graf <agraf@suse.de>
> >---
> > Documentation/virtual/kvm/api.txt              |    6 +-
> > Documentation/virtual/kvm/devices/arm-vgic.txt |   11 +++
> > arch/arm/include/uapi/asm/kvm.h                |    9 +++
> > arch/arm/kvm/arm.c                             |    2 +-
> > include/kvm/arm_vgic.h                         |    2 +-
> > virt/kvm/arm/vgic.c                            |   90
> >++++++++++++++++++++----
> > 6 files changed, 105 insertions(+), 15 deletions(-)
> >
> >diff --git a/Documentation/virtual/kvm/api.txt
> >b/Documentation/virtual/kvm/api.txt
> >index 858aecf..d68b6c2 100644
> >--- a/Documentation/virtual/kvm/api.txt
> >+++ b/Documentation/virtual/kvm/api.txt
> >@@ -2324,7 +2324,7 @@ This ioctl returns the guest registers that are
> >supported for the
> > KVM_GET_ONE_REG/KVM_SET_ONE_REG calls.
> >
> >
> >-4.84 KVM_ARM_SET_DEVICE_ADDR
> >+4.84 KVM_ARM_SET_DEVICE_ADDR (deprecated)
> >
> > Capability: KVM_CAP_ARM_SET_DEVICE_ADDR
> > Architectures: arm, arm64
> >@@ -2362,6 +2362,10 @@ must be called after calling
> >KVM_CREATE_IRQCHIP, but before calling
> > KVM_RUN on any of the VCPUs.  Calling this ioctl twice for any of
> >the
> > base addresses will return -EEXIST.
> >
> >+Note, this IOCTL is deprecated and the more flexible
> >SET/GET_DEVICE_ATTR API
> >+should be used instead.
> >+
> >+
> > 4.85 KVM_PPC_RTAS_DEFINE_TOKEN
> >
> > Capability: KVM_CAP_PPC_RTAS
> >diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> >b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >index 38f27f7..c9febb2 100644
> >--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> >+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >@@ -8,3 +8,14 @@ Only one VGIC instance may be instantiated through
> >either this API or the
> > legacy KVM_CREATE_IRQCHIP api.  The created VGIC will act as the VM
> >interrupt
> > controller, requiring emulated user-space devices to inject
> >interrupts to the
> > VGIC instead of directly to CPUs.
> >+
> >+Groups:
> >+  KVM_DEV_ARM_VGIC_GRP_ADDR
> >+  Attributes:
> >+    KVM_VGIC_V2_ADDR_TYPE_DIST (rw, 64-bit)
> >+      Base address in the guest physical address space of the GIC
> >distributor
> >+      register mappings.
> >+
> >+    KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
> >+      Base address in the guest physical address space of the GIC
> >virtual cpu
> >+      interface register mappings.
> >diff --git a/arch/arm/include/uapi/asm/kvm.h
> >b/arch/arm/include/uapi/asm/kvm.h
> >index 1c85102..587f1ae 100644
> >--- a/arch/arm/include/uapi/asm/kvm.h
> >+++ b/arch/arm/include/uapi/asm/kvm.h
> >@@ -142,6 +142,15 @@ struct kvm_arch_memory_slot {
> > #define KVM_REG_ARM_VFP_FPINST		0x1009
> > #define KVM_REG_ARM_VFP_FPINST2		0x100A
> >
> >+/* Device Control API: ARM VGIC */
> >+#define KVM_DEV_ARM_VGIC_GRP_ADDR	0
> >+#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS	1
> >+#define KVM_DEV_ARM_VGIC_GRP_CPU_REGS	2
> >+#define   KVM_DEV_ARM_VGIC_CPUID_SHIFT	32
> >+#define   KVM_DEV_ARM_VGIC_CPUID_MASK	(0xffULL <<
> >KVM_DEV_ARM_VGIC_CPUID_SHIFT)
> >+#define   KVM_DEV_ARM_VGIC_OFFSET_SHIFT	0
> >+#define   KVM_DEV_ARM_VGIC_OFFSET_MASK	(0xffffffffULL <<
> >KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
> >+
> > /* KVM_IRQ_LINE irq field index values */
> > #define KVM_ARM_IRQ_TYPE_SHIFT		24
> > #define KVM_ARM_IRQ_TYPE_MASK		0xff
> >diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> >index ab96af2..3ecee45 100644
> >--- a/arch/arm/kvm/arm.c
> >+++ b/arch/arm/kvm/arm.c
> >@@ -773,7 +773,7 @@ static int kvm_vm_ioctl_set_device_addr(struct
> >kvm *kvm,
> > 	case KVM_ARM_DEVICE_VGIC_V2:
> > 		if (!vgic_present)
> > 			return -ENXIO;
> >-		return kvm_vgic_set_addr(kvm, type, dev_addr->addr);
> >+		return kvm_vgic_addr(kvm, type, &dev_addr->addr, true);
> > 	default:
> > 		return -ENODEV;
> > 	}
> >diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> >index 7e2d158..be85127 100644
> >--- a/include/kvm/arm_vgic.h
> >+++ b/include/kvm/arm_vgic.h
> >@@ -144,7 +144,7 @@ struct kvm_run;
> > struct kvm_exit_mmio;
> >
> > #ifdef CONFIG_KVM_ARM_VGIC
> >-int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64
> >addr);
> >+int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr,
> >bool write);
> > int kvm_vgic_hyp_init(void);
> > int kvm_vgic_init(struct kvm *kvm);
> > int kvm_vgic_create(struct kvm *kvm);
> >diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> >index 79a8bae..d9c0fc5 100644
> >--- a/virt/kvm/arm/vgic.c
> >+++ b/virt/kvm/arm/vgic.c
> >@@ -1479,6 +1479,12 @@ static int vgic_ioaddr_assign(struct kvm *kvm,
> >phys_addr_t *ioaddr,
> > {
> > 	int ret;
> >
> >+	if (addr & ~KVM_PHYS_MASK)
> >+		return -E2BIG;
> >+
> >+	if (addr & (SZ_4K - 1))
> >+		return -EINVAL;
> >+
> > 	if (!IS_VGIC_ADDR_UNDEF(*ioaddr))
> > 		return -EEXIST;
> > 	if (addr + size < addr)
> >@@ -1491,26 +1497,41 @@ static int vgic_ioaddr_assign(struct kvm
> >*kvm, phys_addr_t *ioaddr,
> > 	return ret;
> > }
> >
> >-int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
> >+/**
> >+ * kvm_vgic_addr - set or get vgic VM base addresses
> >+ * @kvm:   pointer to the vm struct
> >+ * @type:  the VGIC addr type, one of KVM_VGIC_V2_ADDR_TYPE_XXX
> >+ * @addr:  pointer to address value
> >+ * @write: if true set the address in the VM address space, if false
> >read the
> >+ *          address
> >+ *
> >+ * Set or get the vgic base addresses for the distributor and the
> >virtual CPU
> >+ * interface in the VM physical address space.  These addresses are
> >properties
> >+ * of the emulated core/SoC and therefore user space initially
> >knows this
> >+ * information.
> >+ */
> >+int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr,
> >bool write)
> > {
> > 	int r = 0;
> > 	struct vgic_dist *vgic = &kvm->arch.vgic;
> >
> >-	if (addr & ~KVM_PHYS_MASK)
> >-		return -E2BIG;
> >-
> >-	if (addr & (SZ_4K - 1))
> >-		return -EINVAL;
> >-
> > 	mutex_lock(&kvm->lock);
> > 	switch (type) {
> > 	case KVM_VGIC_V2_ADDR_TYPE_DIST:
> >-		r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
> >-				       addr, KVM_VGIC_V2_DIST_SIZE);
> >+		if (write) {
> >+			r = vgic_ioaddr_assign(kvm, &vgic->vgic_dist_base,
> >+					       *addr, KVM_VGIC_V2_DIST_SIZE);
> >+		} else {
> >+			*addr = vgic->vgic_dist_base;
> >+		}
> > 		break;
> > 	case KVM_VGIC_V2_ADDR_TYPE_CPU:
> >-		r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
> >-				       addr, KVM_VGIC_V2_CPU_SIZE);
> >+		if (write) {
> >+			r = vgic_ioaddr_assign(kvm, &vgic->vgic_cpu_base,
> >+					       *addr, KVM_VGIC_V2_CPU_SIZE);
> >+		} else {
> >+			*addr = vgic->vgic_cpu_base;
> >+		}
> > 		break;
> > 	default:
> > 		r = -ENODEV;
> >@@ -1522,16 +1543,61 @@ int kvm_vgic_set_addr(struct kvm *kvm,
> >unsigned long type, u64 addr)
> >
> > static int vgic_set_attr(struct kvm_device *dev, struct
> >kvm_device_attr *attr)
> > {
> >+	int r;
> >+
> >+	switch (attr->group) {
> >+	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
> >+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> >+		u64 addr;
> >+		unsigned long type = (unsigned long)attr->attr;
> >+
> >+		if (copy_from_user(&addr, uaddr, sizeof(addr)))
> >+			return -EFAULT;
> >+
> >+		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
> >+		return (r == -ENODEV) ? -ENXIO : r;
> >+	}
> >+	}
> >+
> > 	return -ENXIO;
> > }
> >
> > static int vgic_get_attr(struct kvm_device *dev, struct
> >kvm_device_attr *attr)
> > {
> >-	return -ENXIO;
> >+	int r = ENXIO;
> 
> Shouldn't this be a negative number?
> 

Yes, it should.

> >+	switch (attr->group) {
> >+	case KVM_DEV_ARM_VGIC_GRP_ADDR: {
> >+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> >+		u64 addr;
> >+		unsigned long type = (unsigned long)attr->attr;
> >+
> >+		r = kvm_vgic_addr(dev->kvm, type, &addr, false);
> >+		if (r)
> >+			return (r == -ENODEV) ? -ENXIO : r;
> >+
> >+		r = 0;
> 
> Isn't r already zero at this point?
> 
yes

> >+		if (copy_to_user(uaddr, &addr, sizeof(addr)))
> >+			return -EFAULT;
> >+	}
> >+	}
> >+
> >+	return r;
> > }
> >
> > static int vgic_has_attr(struct kvm_device *dev, struct
> >kvm_device_attr *attr)
> > {
> >+	phys_addr_t offset;
> >+
> >+	switch (attr->group) {
> >+	case KVM_DEV_ARM_VGIC_GRP_ADDR:
> >+		switch (attr->attr) {
> >+		case KVM_VGIC_V2_ADDR_TYPE_DIST:
> >+		case KVM_VGIC_V2_ADDR_TYPE_CPU:
> >+			return 0;
> >+		}
> >+		break;
> >+	}
> > 	return -ENXIO;
> > }
> 
> Cheers,
> 

Thanks,
-- 
Christoffer

[PATCH RESEND v2 6/8] KVM: arm-vgic: Add vgic reg access from dev attr

[Christoffer Dall]

On Wed, Oct 23, 2013 at 04:46:56PM +0100, Marc Zyngier wrote:
> On 2013-10-22 10:08, Christoffer Dall wrote:
> >Add infrastructure to handle distributor and cpu interface register
> >accesses through the KVM_{GET/SET}_DEVICE_ATTR interface by adding
> >the
> >KVM_DEV_ARM_VGIC_GRP_DIST_REGS and KVM_DEV_ARM_VGIC_GRP_CPU_REGS
> >groups
> >and defining the semantics of the attr field to be the MMIO offset as
> >specified in the GICv2 specs.
> >
> >Missing register accesses or other changes in individual register
> >access
> >functions to support save/restore of the VGIC state is added in
> >subsequent patches.
> >
> >Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >Reviewed-by: Alexander Graf <agraf@suse.de>
> >
> >---
> >Changelog[v2]:
> > - Added implementation specific format for the GICC_APRn registers.
> >---
> > Documentation/virtual/kvm/devices/arm-vgic.txt |   50 +++++++++
> > virt/kvm/arm/vgic.c                            |  143
> >++++++++++++++++++++++++
> > 2 files changed, 193 insertions(+)
> >
> >diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt
> >b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >index c9febb2..e6416f8e 100644
> >--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> >+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> >@@ -19,3 +19,53 @@ Groups:
> >     KVM_VGIC_V2_ADDR_TYPE_CPU (rw, 64-bit)
> >       Base address in the guest physical address space of the GIC
> >virtual cpu
> >       interface register mappings.
> >+
> >+  KVM_DEV_ARM_VGIC_GRP_DIST_REGS
> >+  Attributes:
> >+    The attr field of kvm_device_attr encodes two values:
> >+    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
> >+    values:   |    reserved   |   cpu id   |      offset     |
> >+
> >+    All distributor regs are (rw, 32-bit)
> >+
> >+    The offset is relative to the "Distributor base address" as
> >defined in the
> >+    GICv2 specs.  Getting or setting such a register has the same
> >effect as
> >+    reading or writing the register on the actual hardware from
> >the cpu
> >+    specified with cpu id field.  Note that most distributor
> >fields are not
> >+    banked, but return the same value regardless of the cpu id used
> >to access
> >+    the register.
> >+  Limitations:
> >+    - Priorities are not implemented, and registers are RAZ/WI
> >+  Errors:
> >+    - ENODEV: Getting or setting this register is not yet supported
> 
> -ENODEV?
> 

indeed

> >+  KVM_DEV_ARM_VGIC_GRP_CPU_REGS
> >+  Attributes:
> >+    The attr field of kvm_device_attr encodes two values:
> >+    bits:     | 63   ....  40 | 39 ..  32  |  31   ....    0 |
> >+    values:   |    reserved   |   cpu id   |      offset     |
> >+
> >+    All CPU regs are (rw, 32-bit)
> 
> Nit: CPU interface registers
> 
> >+    The offset specifies the offset from the "CPU interface base
> >address" as
> >+    defined in the GICv2 specs.  Getting or setting such a
> >register has the
> >+    same effect as reading or writing the register on the actual
> >hardware.
> >+
> >+    The Active Priorities Registers APRn are implementation defined,
> >so we set a
> >+    fixed format for our implementation that fits with the model
> >of a "GICv2
> >+    impementation without the security extensions" which we
> >present to the
> 
> implementation
> 
> >+    guest.  This interface always exposes four register APR[0-3]
> >describing the
> >+    maximum possible 128 preemption levels.  The semantics of the
> >register
> >+    indicate if any interrupts in a given preemption level are in
> >the active
> >+    state by setting the corresponding bit.
> >+
> >+    Thus, preemption level X has one or more active interrupts if
> >and only if:
> >+
> >+      APRn[X mod 32] == 0b1,  where n = X / 32
> >+
> >+    Bits for undefined preemption levels are RAZ/WI.
> >+
> >+  Limitations:
> >+    - Priorities are not implemented, and registers are RAZ/WI
> >+  Errors:
> >+    - ENODEV: Getting or setting this register is not yet supported
> >diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> >index 1148a2e..f2dc72a 100644
> >--- a/virt/kvm/arm/vgic.c
> >+++ b/virt/kvm/arm/vgic.c
> >@@ -589,11 +589,29 @@ static bool handle_mmio_sgi_reg(struct
> >kvm_vcpu *vcpu,
> > 	return false;
> > }
> >
> >+static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
> >+				  struct kvm_exit_mmio *mmio,
> >+				  phys_addr_t offset)
> >+{
> >+	return false;
> >+}
> >+
> >+static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
> >+				struct kvm_exit_mmio *mmio,
> >+				phys_addr_t offset)
> >+{
> >+	return false;
> >+}
> >+
> > /*
> >  * I would have liked to use the kvm_bus_io_*() API instead, but it
> >  * cannot cope with banked registers (only the VM pointer is passed
> >  * around, and we need the vcpu). One of these days, someone please
> >  * fix it!
> >+ *
> >+ * Note that the handle_mmio implementations should not use the
> >phys_addr
> >+ * field from the kvm_exit_mmio struct as this will not have any
> >sane values
> >+ * when used to save/restore state from user space.
> 
> This is quite ugly... I don't think we'd ever use that field
> directly, but reusing a well known structure for that purpose is
> very messy. I believe we'd be better off creating our own structure
> instead of re-purposing am existing one.

Hmmm, I don't think this is about re-purposing an existing structure, it
is about generally using a structure in a file which happens to contain
a superflous field, which should never have to be used in this file anyway.
Now we will actually use this structure where this unnecessary field (in
this context) does not contain a sane value and we clearly document that
in the comment.  Further, introducing another type adds another memcpy
or makes the whole io_mem_abort() - vgic_handle_mmio() messy.  I
actually had a go at it that I can pass your way if you are set on this
approach...

> 
> The other possibility would be to properly fill-in the phys_addr
> field. How difficult would that be?
> 

Not really difficult at all, let me do that for a v3.

> >  */
> > struct mmio_range {
> > 	phys_addr_t base;
> >@@ -663,6 +681,16 @@ static const struct mmio_range
> >vgic_dist_ranges[] = {
> > 		.len		= 4,
> > 		.handle_mmio	= handle_mmio_sgi_reg,
> > 	},
> >+	{
> >+		.base		= GIC_DIST_SGI_CLEAR,
> >+		.len		= VGIC_NR_SGIS,
> >+		.handle_mmio	= handle_mmio_sgi_clear,
> >+	},
> >+	{
> >+		.base		= GIC_DIST_SGI_SET,
> >+		.len		= VGIC_NR_SGIS,
> >+		.handle_mmio	= handle_mmio_sgi_set,
> >+	},
> > 	{}
> > };
> >
> >@@ -1541,6 +1569,80 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned
> >long type, u64 *addr, bool write)
> > 	return r;
> > }
> >
> >+static bool handle_cpu_mmio_misc(struct kvm_vcpu *vcpu,
> >+				 struct kvm_exit_mmio *mmio, phys_addr_t offset)
> >+{
> >+	return true;
> >+}
> >+
> >+static const struct mmio_range vgic_cpu_ranges[] = {
> >+	{
> >+		.base		= GIC_CPU_CTRL,
> >+		.len		= 12,
> >+		.handle_mmio	= handle_cpu_mmio_misc,
> >+	},
> >+	{
> >+		.base		= GIC_CPU_ALIAS_BINPOINT,
> >+		.len		= 4,
> >+		.handle_mmio	= handle_cpu_mmio_misc,
> >+	},
> >+	{
> >+		.base		= GIC_CPU_ACTIVEPRIO,
> >+		.len		= 16,
> >+		.handle_mmio	= handle_cpu_mmio_misc,
> >+	},
> >+	{
> >+		.base		= GIC_CPU_IDENT,
> >+		.len		= 4,
> >+		.handle_mmio	= handle_cpu_mmio_misc,
> >+	},
> >+};
> >+
> >+static struct kvm_exit_mmio dev_attr_mmio = { .len = 4 };
> 
> I'm not very fond of a half-initialized structure here. How about
> moving this "4" to the location where it is used?
> Actually, what if we have several users of this through
> vgic_has_attr_regs at the same time? It feels incredibly racy. I
> suggest you nuke it and move it to live on the stack in
> vgic_has_attr_regs.
> 

fair enough, since this is just another way of passing the constant four
to a match function and that constant is only ever read, I don't think
there's any race here, but ok, it's completely fine to just allocate it on
the stack.

> >+static int vgic_attr_regs_access(struct kvm_device *dev,
> >+				 struct kvm_device_attr *attr,
> >+				 u32 *reg, bool is_write)
> >+{
> >+	const struct mmio_range *r = NULL;
> >+	phys_addr_t offset;
> >+	int cpuid;
> >+	struct kvm_vcpu *vcpu;
> >+	struct kvm_exit_mmio mmio;
> >+
> >+	offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> >+	cpuid = (attr->attr & KVM_DEV_ARM_VGIC_CPUID_MASK) >>
> >+		KVM_DEV_ARM_VGIC_CPUID_SHIFT;
> >+
> >+	if (cpuid >= atomic_read(&dev->kvm->online_vcpus))
> >+		return -EINVAL;
> >+
> >+	vcpu = kvm_get_vcpu(dev->kvm, cpuid);
> >+
> >+	mmio.len = 4;
> >+	mmio.is_write = is_write;
> >+	if (is_write)
> >+		mmio_data_write(&mmio, ~0, *reg);
> >+
> >+	if (attr->group == KVM_DEV_ARM_VGIC_GRP_DIST_REGS)
> >+		r = find_matching_range(vgic_dist_ranges, &mmio, offset);
> >+	else if (attr->group == KVM_DEV_ARM_VGIC_GRP_CPU_REGS)
> >+		r = find_matching_range(vgic_cpu_ranges, &mmio, offset);
> 
> How about having a switch statement instead?
> 

sure.

> >+	if (unlikely(!r || !r->handle_mmio))
> >+		return -ENXIO;
> >+
> >+	spin_lock(&vcpu->kvm->arch.vgic.lock);
> >+	offset -= r->base;
> >+	r->handle_mmio(vcpu, &mmio, offset);
> >+	spin_unlock(&vcpu->kvm->arch.vgic.lock);
> >+
> >+	if (!is_write)
> >+		*reg = mmio_data_read(&mmio, ~0);
> >+
> >+	return 0;
> >+}
> >+
> > static int vgic_set_attr(struct kvm_device *dev, struct
> >kvm_device_attr *attr)
> > {
> > 	int r;
> >@@ -1557,6 +1659,18 @@ static int vgic_set_attr(struct kvm_device
> >*dev, struct kvm_device_attr *attr)
> > 		r = kvm_vgic_addr(dev->kvm, type, &addr, true);
> > 		return (r == -ENODEV) ? -ENXIO : r;
> > 	}
> >+
> >+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> >+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
> >+		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
> >+		u32 reg;
> >+
> >+		if (get_user(reg, uaddr))
> >+			return -EFAULT;
> >+
> >+		return vgic_attr_regs_access(dev, attr, &reg, true);
> >+	}
> >+
> > 	}
> >
> > 	return -ENXIO;
> >@@ -1579,12 +1693,35 @@ static int vgic_get_attr(struct kvm_device
> >*dev, struct kvm_device_attr *attr)
> > 		r = 0;
> > 		if (copy_to_user(uaddr, &addr, sizeof(addr)))
> > 			return -EFAULT;
> >+		break;
> > 	}
> >+
> >+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> >+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS: {
> >+		u32 __user *uaddr = (u32 __user *)(long)attr->addr;
> >+		u32 reg = 0;
> >+
> >+		r = vgic_attr_regs_access(dev, attr, &reg, false);
> >+		if (r)
> >+			return r;
> >+		r = put_user(reg, uaddr);
> >+		break;
> >+	}
> >+
> > 	}
> >
> > 	return r;
> > }
> >
> >+static int vgic_has_attr_regs(const struct mmio_range *ranges,
> >+			      phys_addr_t offset)
> >+{
> >+	if (find_matching_range(ranges, &dev_attr_mmio, offset))
> >+		return 0;
> >+	else
> >+		return -ENXIO;
> >+}
> >+
> > static int vgic_has_attr(struct kvm_device *dev, struct
> >kvm_device_attr *attr)
> > {
> > 	phys_addr_t offset;
> >@@ -1597,6 +1734,12 @@ static int vgic_has_attr(struct kvm_device
> >*dev, struct kvm_device_attr *attr)
> > 			return 0;
> > 		}
> > 		break;
> >+	case KVM_DEV_ARM_VGIC_GRP_DIST_REGS:
> >+		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> >+		return vgic_has_attr_regs(vgic_dist_ranges, offset);
> >+	case KVM_DEV_ARM_VGIC_GRP_CPU_REGS:
> >+		offset = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
> >+		return vgic_has_attr_regs(vgic_cpu_ranges, offset);
> > 	}
> > 	return -ENXIO;
> > }
> 

Thanks!
-- 
Christoffer

[PATCH RESEND v2 7/8] KVM: arm-vgic: Add GICD_SPENDSGIR and GICD_CPENDSGIR handlers

[Christoffer Dall]

On Wed, Oct 23, 2013 at 05:00:43PM +0100, Marc Zyngier wrote:
> On 2013-10-22 10:08, Christoffer Dall wrote:
> >Handle MMIO accesses to the two registers which should support
> >both the
> >case where the VMs want to read/write either of these registers
> >and the
> >case where user space reads/writes these registers to do
> >save/restore of
> >the VGIC state.
> >
> >Note that the added complexity compared to simple set/clear enable
> >registers stems from the bookkeping of source cpu ids.  It may be
> >possible to change the underlying data structure to simplify the
> >complexity, but since this is not in the critical path, at all,
> >this is
> >left as an interesting excercise to the reader.
> >
> >Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
> >Reviewed-by: Alexander Graf <agraf@suse.de>
> >
> >---
> >Changelog[v2]:
> > - Use struct kvm_exit_mmio accessors for ->data field.
> >---
> > virt/kvm/arm/vgic.c |  114
> >++++++++++++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 112 insertions(+), 2 deletions(-)
> >
> >diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> >index f2dc72a..4e8c3ab 100644
> >--- a/virt/kvm/arm/vgic.c
> >+++ b/virt/kvm/arm/vgic.c
> >@@ -589,18 +589,128 @@ static bool handle_mmio_sgi_reg(struct
> >kvm_vcpu *vcpu,
> > 	return false;
> > }
> >
> >+static void read_sgi_set_clear(struct kvm_vcpu *vcpu,
> >+			       struct kvm_exit_mmio *mmio,
> >+			       phys_addr_t offset)
> 
> set_clear is a bit unclear. How about reset?
> 

it's not a reset, it handles reads of the clear/set pending registers:

/* Handle reads of GICD_CPENDSGIRn and GICD_SPENDSGIRn */
static void read_set_clear_sgi_pend_reg(...)

Does that work?

> >+{
> >+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> >+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> >+	int i, sgi, cpu;
> >+	int min_sgi = (offset & ~0x3) * 4;
> >+	int max_sgi = min_sgi + 3;
> >+	int vcpu_id = vcpu->vcpu_id;
> >+	u32 lr, reg = 0;
> >+
> >+	/* Copy source SGIs from distributor side */
> >+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> >+		int shift = 8 * (sgi - min_sgi);
> >+		reg |= (u32)dist->irq_sgi_sources[vcpu_id][sgi] << shift;
> >+	}
> >+
> >+	/* Copy source SGIs already on LRs */
> >+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> >+		lr = vgic_cpu->vgic_lr[i];
> >+		sgi = lr & GICH_LR_VIRTUALID;
> >+		cpu = (lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
> 
> Please wrap these  lr accesses into separate functions. There is
> quite a bit of duplication in this patch and I wonder if we could
> factor things a bit.
> 
> At least, please isolate what is emulation related from what is
> actually what the underlying HW provides. It will help mitigating my
> headache in the future... ;-)
> 

hmmm, yeah, this actually quite sucks, and the problems repeats itself
for other pending registers as well.

Making defines for the loop logic will be too ugly macro-mess and having
a looping function with an ops fn pointer will also suck, so I actually
think the solution is quite a different one:

Before accessing any of the register state, make sure everything is
stopped (which is probably something we should have done anyway) and
move all state from the LRs to the distributor.

I will add a separate patch for this in v3.

> >+		if (sgi >= min_sgi && sgi <= max_sgi) {
> >+			if (lr & GICH_LR_STATE)
> >+				reg |= (1 << cpu) << (8 * (sgi - min_sgi));
> >+		}
> >+	}
> >+
> >+	mmio_data_write(mmio, ~0, reg);
> >+}
> >+
> > static bool handle_mmio_sgi_clear(struct kvm_vcpu *vcpu,
> > 				  struct kvm_exit_mmio *mmio,
> > 				  phys_addr_t offset)
> > {
> >-	return false;
> >+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> >+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> >+	int i, sgi, cpu;
> >+	int min_sgi = (offset & ~0x3) * 4;
> >+	int max_sgi = min_sgi + 3;
> >+	int vcpu_id = vcpu->vcpu_id;
> >+	u32 *lr, reg;
> >+	bool updated = false;
> >+
> >+	if (!mmio->is_write) {
> >+		read_sgi_set_clear(vcpu, mmio, offset);
> >+		return false;
> >+	}
> >+
> >+	reg = mmio_data_read(mmio, ~0);
> >+
> >+	/* Clear pending SGIs on distributor side */
> >+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> >+		u8 mask = reg >> (8 * (sgi - min_sgi));
> >+		if (dist->irq_sgi_sources[vcpu_id][sgi] & mask)
> >+			updated = true;
> >+		dist->irq_sgi_sources[vcpu_id][sgi] &= ~mask;
> >+	}
> >+
> >+	/* Clear SGIs already on LRs */
> >+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> >+		lr = &vgic_cpu->vgic_lr[i];
> >+		sgi = *lr & GICH_LR_VIRTUALID;
> >+		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
> >+
> >+		if (sgi >= min_sgi && sgi <= max_sgi) {
> >+			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
> >+				if (*lr & GICH_LR_PENDING_BIT)
> >+					updated = true;
> >+				*lr &= GICH_LR_PENDING_BIT;
> >+			}
> >+		}
> >+	}
> >+
> >+	return updated;
> > }
> >
> > static bool handle_mmio_sgi_set(struct kvm_vcpu *vcpu,
> > 				struct kvm_exit_mmio *mmio,
> > 				phys_addr_t offset)
> > {
> >-	return false;
> >+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> >+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> >+	int i, sgi, cpu;
> >+	int min_sgi = (offset & ~0x3) * 4;
> >+	int max_sgi = min_sgi + 3;
> >+	int vcpu_id = vcpu->vcpu_id;
> >+	u32 *lr, reg;
> >+	bool updated = false;
> >+
> >+	if (!mmio->is_write) {
> >+		read_sgi_set_clear(vcpu, mmio, offset);
> >+		return false;
> >+	}
> >+
> >+	reg = mmio_data_read(mmio, ~0);
> >+
> >+	/* Set pending SGIs on distributor side */
> >+	for (sgi = min_sgi; sgi <= max_sgi; sgi++) {
> >+		u8 mask = reg >> (8 * (sgi - min_sgi));
> >+		if ((dist->irq_sgi_sources[vcpu_id][sgi] & mask) != mask)
> >+			updated = true;
> >+		dist->irq_sgi_sources[vcpu_id][sgi] |= mask;
> >+	}
> >+
> >+	/* Set active SGIs already on LRs to pending and active */
> >+	for_each_set_bit(i, vgic_cpu->lr_used, vgic_cpu->nr_lr) {
> >+		lr = &vgic_cpu->vgic_lr[i];
> >+		sgi = *lr & GICH_LR_VIRTUALID;
> >+		cpu = (*lr & GICH_LR_PHYSID_CPUID) >> GICH_LR_PHYSID_CPUID_SHIFT;
> >+
> >+		if (sgi >= min_sgi && sgi <= max_sgi) {
> >+			if (reg & ((1 << cpu) << (8 * (sgi - min_sgi)))) {
> >+				if (!(*lr & GICH_LR_PENDING_BIT))
> >+					updated = true;
> >+				*lr |= GICH_LR_PENDING_BIT;
> >+			}
> >+		}
> >+	}
> >+
> >+	return updated;
> > }
> >
> > /*
> 
> Overall, I feel like I've read the same function three times. Hint,
> hint... ;-)
> 

Yeah yeah yeah, you're right, this was broken :)

-- 
Christoffer