[RFC PATCH 0/1] target/arm: kvm: Support for KVM DABT without valid ISS

[RFC PATCH 0/1] target/arm: kvm: Support for KVM DABT without valid ISS

[Beata Michalska]

Some of the ARMv7 & ARMv8 load/store instructions might trigger a data abort
exception with no valid ISS info to be decoded. The lack of decode info
makes it at least tricky to emulate the instruction which is one of the
(many) reasons why KVM will not even try to do so.

So far, if a guest made an attempt to access memory outside the memory slot,
KVM reported vague ENOSYS. As a result QEMU exited with no useful information
being provided or even a clue on what has just happened.

Recently ARM KVM introduced support for notifying guest of an attempt to
execute an instruction that resulted in dabt with no valid ISS decoding info.
This still leaves QEMU to handle the case, but at least now, it can enable
further debugging of the encountered issue by being more verbose
in a (hopefully) useful way.



Beata Michalska (1):
  target/arm: kvm: Handle DABT with no valid ISS

 accel/kvm/kvm-all.c    | 15 +++++++
 accel/stubs/kvm-stub.c |  4 ++
 include/sysemu/kvm.h   |  1 +
 target/arm/cpu.h       |  3 +-
 target/arm/kvm.c       | 95 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm32.c     |  3 ++
 target/arm/kvm64.c     |  3 ++
 target/arm/kvm_arm.h   | 19 +++++++++
 8 files changed, 142 insertions(+), 1 deletion(-)

-- 
2.17.1

[RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Beata Michalska]

On ARMv7 & ARMv8 some load/store instructions might trigger a data abort
exception with no valid ISS info to be decoded. The lack of decode info
makes it at least tricky to emulate those instruction which is one of the
(many) reasons why KVM will not even try to do so.

Add suport for handling those by requesting KVM to inject external
dabt into the quest.

Signed-off-by: Beata Michalska <beata.michalska@linaro.org>
---
 accel/kvm/kvm-all.c    | 15 +++++++
 accel/stubs/kvm-stub.c |  4 ++
 include/sysemu/kvm.h   |  1 +
 target/arm/cpu.h       |  3 +-
 target/arm/kvm.c       | 95 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm32.c     |  3 ++
 target/arm/kvm64.c     |  3 ++
 target/arm/kvm_arm.h   | 19 +++++++++
 8 files changed, 142 insertions(+), 1 deletion(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index ca00daa2f5..a3ee038142 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -2174,6 +2174,14 @@ static void do_kvm_cpu_synchronize_state(CPUState *cpu, run_on_cpu_data arg)
     }
 }
 
+static void do_kvm_cpu_synchronize_state_force(CPUState *cpu,
+                                               run_on_cpu_data arg)
+{
+    kvm_arch_get_registers(cpu);
+    cpu->vcpu_dirty = true;
+}
+
+
 void kvm_cpu_synchronize_state(CPUState *cpu)
 {
     if (!cpu->vcpu_dirty) {
@@ -2181,6 +2189,13 @@ void kvm_cpu_synchronize_state(CPUState *cpu)
     }
 }
 
+void kvm_cpu_synchronize_state_force(CPUState *cpu)
+{
+    /* Force the sync */
+    run_on_cpu(cpu, do_kvm_cpu_synchronize_state_force, RUN_ON_CPU_NULL);
+}
+
+
 static void do_kvm_cpu_synchronize_post_reset(CPUState *cpu, run_on_cpu_data arg)
 {
     kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE);
diff --git a/accel/stubs/kvm-stub.c b/accel/stubs/kvm-stub.c
index 82f118d2df..e917d1d55e 100644
--- a/accel/stubs/kvm-stub.c
+++ b/accel/stubs/kvm-stub.c
@@ -58,6 +58,10 @@ void kvm_cpu_synchronize_post_init(CPUState *cpu)
 {
 }
 
+void kvm_cpu_synchronize_state_force(CPUState *cpu)
+{
+}
+
 int kvm_cpu_exec(CPUState *cpu)
 {
     abort();
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 9fe233b9bf..0cacc61d8a 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -483,6 +483,7 @@ void kvm_cpu_synchronize_state(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
 void kvm_cpu_synchronize_pre_loadvm(CPUState *cpu);
+void kvm_cpu_synchronize_state_force(CPUState *cpu);
 
 void kvm_init_cpu_signals(CPUState *cpu);
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index 5f70e9e043..e11b5e7438 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -558,7 +558,8 @@ typedef struct CPUARMState {
         uint8_t has_esr;
         uint64_t esr;
     } serror;
-
+    /* Status field for pending extarnal dabt */
+    uint8_t ext_dabt_pending;
     /* State of our input IRQ/FIQ/VIRQ/VFIQ lines */
     uint32_t irq_line_state;
 
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 5b82cefef6..10fe739c2d 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -37,6 +37,7 @@ const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
 
 static bool cap_has_mp_state;
 static bool cap_has_inject_serror_esr;
+static bool cap_has_inject_ext_dabt; /* KVM_CAP_ARM_INJECT_EXT_DABT */
 
 static ARMHostCPUFeatures arm_host_cpu_features;
 
@@ -62,6 +63,12 @@ void kvm_arm_init_serror_injection(CPUState *cs)
                                     KVM_CAP_ARM_INJECT_SERROR_ESR);
 }
 
+void kvm_arm_init_ext_dabt_injection(CPUState *cs)
+{
+    cap_has_inject_ext_dabt = kvm_check_extension(cs->kvm_state,
+                                    KVM_CAP_ARM_INJECT_EXT_DABT);
+}
+
 bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
                                       int *fdarray,
                                       struct kvm_vcpu_init *init)
@@ -218,6 +225,11 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
         ret = -EINVAL;
     }
 
+    if (kvm_check_extension(s, KVM_CAP_ARM_NISV_TO_USER))
+        if (kvm_vm_enable_cap(s, KVM_CAP_ARM_NISV_TO_USER, 0)) {
+            warn_report("Failed to enable DABT NISV cap");
+        }
+
     return ret;
 }
 
@@ -600,6 +612,10 @@ int kvm_put_vcpu_events(ARMCPU *cpu)
         events.exception.serror_esr = env->serror.esr;
     }
 
+    if (cap_has_inject_ext_dabt) {
+        events.exception.ext_dabt_pending = env->ext_dabt_pending;
+    }
+
     ret = kvm_vcpu_ioctl(CPU(cpu), KVM_SET_VCPU_EVENTS, &events);
     if (ret) {
         error_report("failed to put vcpu events");
@@ -629,6 +645,8 @@ int kvm_get_vcpu_events(ARMCPU *cpu)
     env->serror.has_esr = events.exception.serror_has_esr;
     env->serror.esr = events.exception.serror_esr;
 
+    env->ext_dabt_pending = events.exception.ext_dabt_pending;
+
     return 0;
 }
 
@@ -701,6 +719,11 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
             ret = EXCP_DEBUG;
         } /* otherwise return to guest */
         break;
+    case KVM_EXIT_ARM_NISV:
+        /* External DAB with no valid iss to decode */
+        ret = kvm_arm_handle_dabt_nisv(cs, run->arm_nisv.esr_iss,
+                                 run->arm_nisv.fault_ipa);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n",
                       __func__, run->exit_reason);
@@ -835,3 +858,75 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
 {
     return (data - 32) & 0xffff;
 }
+
+int kvm_arm_handle_dabt_nisv(CPUState *cs, uint64_t esr_iss,
+                              uint64_t fault_ipa)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+    hwaddr xlat, len;
+    AddressSpace *as = cs->as ? cs->as : &address_space_memory;
+    int store_ins = ((esr_iss >> 6) & 1); /* WnR bit */
+    int ret;
+
+    /*
+     * Note: The ioctl for SET_EVENTS will be triggered before next
+     * KVM_RUN call though the vcpu regs need to be updated before hand
+     * so to not to overwrite changes done by KVM upon injecting the abort.
+     * This sadly requires running sync twice - shame ...
+     */
+    kvm_cpu_synchronize_state(cs);
+   /*
+    * ISS [23:14] is invalid so there is a limited info
+    * on what has just happened so the only *useful* thing that can
+    * be retrieved from ISS is WnR & DFSC (though in some cases WnR
+    * might be less of a value as well)
+    */
+    /* Verify whether the memory being accessed is even recognisable */
+    if (!address_space_translate(as, fault_ipa, &xlat, &len,
+                                 store_ins, MEMTXATTRS_UNSPECIFIED)) {
+        error_report("An attemp to access memory outside of the boundries"
+                     "at 0x" TARGET_FMT_lx, (target_ulong) fault_ipa);
+    } else {
+        uint32_t ins;
+        uint8_t ins_fetched;
+
+        /*
+         * Get current PC before it will get updated to except vector entry
+         */
+        target_ulong ins_addr = is_a64(env) ? env->pc
+                                /* AArch32 mode vs T32 aka Thumb mode */
+                                : env->regs[15] - (env->thumb ? 4 : 8);
+
+        /*
+         * Get the faulting instruction
+         * Instructions have a fixed length of 32 bits
+         * and are always little-endian
+         */
+        ins_fetched = !cpu_memory_rw_debug(cs, ins_addr, (uint8_t *) &ins,
+                                           sizeof(uint32_t), store_ins) ;
+
+        error_report("Data abort exception with no valid ISS generated by "
+                     "memory access at 0x" TARGET_FMT_lx,
+                     (target_ulong)fault_ipa);
+        error_report(ins_fetched ? "%s: 0x%" PRIx32 " (encoded)" : "%s",
+                     "Unable to emulate memory instruction",
+                     (!ins_fetched ? 0 : ldl_le_p(&ins)));
+
+        error_report("Faulting instruction at 0x" TARGET_FMT_lx, ins_addr);
+    }
+    /*
+     * Set pending ext dabt amd trigger SET_EVENTS so that
+     * KVM can inject the abort
+     */
+    env->ext_dabt_pending = 1;
+
+    ret = kvm_put_vcpu_events(cpu);
+
+    /* Get the most up-tp-date state */
+    if (!ret) {
+        /* Hacky but the sync needs to be forced */
+        kvm_cpu_synchronize_state_force(cs);
+    }
+    return ret;
+}
diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
index 32bf8d6757..5539c3a3f2 100644
--- a/target/arm/kvm32.c
+++ b/target/arm/kvm32.c
@@ -240,6 +240,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
     /* Check whether userspace can specify guest syndrome value */
     kvm_arm_init_serror_injection(cs);
 
+    /* Set status for supporting the extarnal dabt injection */
+    kvm_arm_init_ext_dabt_injection(cs);
+
     return kvm_arm_init_cpreg_list(cpu);
 }
 
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 876184b8fe..3da4e2e70e 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -792,6 +792,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
     /* Check whether user space can specify guest syndrome value */
     kvm_arm_init_serror_injection(cs);
 
+    /* Set status for supporting the extarnal dabt injection */
+    kvm_arm_init_ext_dabt_injection(cs);
+
     return kvm_arm_init_cpreg_list(cpu);
 }
 
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 8e14d400e8..fe019f2fc9 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -143,6 +143,14 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu);
  */
 void kvm_arm_init_serror_injection(CPUState *cs);
 
+/**
+ * kvm_arm_init_ext_dabt_injection
+ * @cs: CPUState
+ *
+ * Set status for KVM support for Ext DABT injection
+ */
+void kvm_arm_init_ext_dabt_injection(CPUState *cs);
+
 /**
  * kvm_get_vcpu_events:
  * @cpu: ARMCPU
@@ -371,6 +379,17 @@ static inline const char *gicv3_class_name(void)
  */
 bool kvm_arm_handle_debug(CPUState *cs, struct kvm_debug_exit_arch *debug_exit);
 
+/**
+ * kvm_arm_handle_dabt_nisv
+ * @cs: CPUState
+ * @esr_iss: ISS encoding (limited) for the exception from Data Abort
+ *           ISV bit set to '0b0' -> no valid instruction syndrome
+ * @fault_ipa: faulting address for the synch data abort
+ *
+ * Returns: 0 if the exception has been handled
+ */
+int kvm_arm_handle_dabt_nisv(CPUState *cs, uint64_t esr_iss,
+                             uint64_t fault_ipa);
 /**
  * kvm_arm_hw_debug_active:
  * @cs: CPU State
-- 
2.17.1

Re: [RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Peter Maydell]

On Fri, 20 Dec 2019 at 20:27, Beata Michalska
<beata.michalska@linaro.org> wrote:
>
> On ARMv7 & ARMv8 some load/store instructions might trigger a data abort
> exception with no valid ISS info to be decoded. The lack of decode info
> makes it at least tricky to emulate those instruction which is one of the
> (many) reasons why KVM will not even try to do so.
>
> Add suport for handling those by requesting KVM to inject external
> dabt into the quest.
>
> Signed-off-by: Beata Michalska <beata.michalska@linaro.org>
> ---
>  accel/kvm/kvm-all.c    | 15 +++++++
>  accel/stubs/kvm-stub.c |  4 ++
>  include/sysemu/kvm.h   |  1 +
>  target/arm/cpu.h       |  3 +-
>  target/arm/kvm.c       | 95 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm32.c     |  3 ++
>  target/arm/kvm64.c     |  3 ++
>  target/arm/kvm_arm.h   | 19 +++++++++
>  8 files changed, 142 insertions(+), 1 deletion(-)
>
> diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
> index ca00daa2f5..a3ee038142 100644
> --- a/accel/kvm/kvm-all.c
> +++ b/accel/kvm/kvm-all.c
> @@ -2174,6 +2174,14 @@ static void do_kvm_cpu_synchronize_state(CPUState *cpu, run_on_cpu_data arg)
>      }
>  }
>
> +static void do_kvm_cpu_synchronize_state_force(CPUState *cpu,
> +                                               run_on_cpu_data arg)
> +{
> +    kvm_arch_get_registers(cpu);
> +    cpu->vcpu_dirty = true;
> +}

Why is this functionality special such that it needs a non-standard
way of synchronizing state with KVM that nothing else does ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 9fe233b9bf..0cacc61d8a 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -483,6 +483,7 @@ void kvm_cpu_synchronize_state(CPUState *cpu);
>  void kvm_cpu_synchronize_post_reset(CPUState *cpu);
>  void kvm_cpu_synchronize_post_init(CPUState *cpu);
>  void kvm_cpu_synchronize_pre_loadvm(CPUState *cpu);
> +void kvm_cpu_synchronize_state_force(CPUState *cpu);
>
>  void kvm_init_cpu_signals(CPUState *cpu);
>
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index 5f70e9e043..e11b5e7438 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -558,7 +558,8 @@ typedef struct CPUARMState {
>          uint8_t has_esr;
>          uint64_t esr;
>      } serror;
> -
> +    /* Status field for pending extarnal dabt */

"external" (I think you have the same typo later in the patch too)

> +    uint8_t ext_dabt_pending;
>      /* State of our input IRQ/FIQ/VIRQ/VFIQ lines */
>      uint32_t irq_line_state;


> @@ -701,6 +719,11 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
>              ret = EXCP_DEBUG;
>          } /* otherwise return to guest */
>          break;
> +    case KVM_EXIT_ARM_NISV:
> +        /* External DAB with no valid iss to decode */

"DABT"

> +        ret = kvm_arm_handle_dabt_nisv(cs, run->arm_nisv.esr_iss,
> +                                 run->arm_nisv.fault_ipa);

(indentation looks odd here?)

> +        break;
>      default:
>          qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n",
>                        __func__, run->exit_reason);
> @@ -835,3 +858,75 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
>  {
>      return (data - 32) & 0xffff;
>  }
> +
> +int kvm_arm_handle_dabt_nisv(CPUState *cs, uint64_t esr_iss,
> +                              uint64_t fault_ipa)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +    hwaddr xlat, len;
> +    AddressSpace *as = cs->as ? cs->as : &address_space_memory;

I don't think you should need to test cs->as for NULL;
qemu_init_vcpu() ensures that one is always set up.

Probably the neatest way to get the AddressSpace* is to
call arm_addressspace(cs, MEMTXATTRS_UNSPECIFIED).


> +    int store_ins = ((esr_iss >> 6) & 1); /* WnR bit */

This should be a bool (matching the argument type for
address_space_translate), so you can just do
   bool is_write = esr_iss & (1 << 6);

> +    int ret;
> +
> +    /*
> +     * Note: The ioctl for SET_EVENTS will be triggered before next
> +     * KVM_RUN call though the vcpu regs need to be updated before hand
> +     * so to not to overwrite changes done by KVM upon injecting the abort.
> +     * This sadly requires running sync twice - shame ...
> +     */
> +    kvm_cpu_synchronize_state(cs);
> +   /*
> +    * ISS [23:14] is invalid so there is a limited info
> +    * on what has just happened so the only *useful* thing that can
> +    * be retrieved from ISS is WnR & DFSC (though in some cases WnR
> +    * might be less of a value as well)
> +    */
> +    /* Verify whether the memory being accessed is even recognisable */
> +    if (!address_space_translate(as, fault_ipa, &xlat, &len,
> +                                 store_ins, MEMTXATTRS_UNSPECIFIED)) {
> +        error_report("An attemp to access memory outside of the boundries"

"attempt"; "boundaries". But I think what we should actually report here
is:

"Guest attempted to access memory/device at guest physical address
0x... using an instruction that KVM could not emulate (no valid ISS).
Injecting a data abort exception into guest."

It doesn't seem to me worth distinguishing whether the guest
physical address happens to have anything mapped there or not;
we're going to inject a somewhat bogus dabt anyway.

After the initial error_report(), supplemental information like
the offending instruction should be printed with error_printf(),
because it isn't a separate new error.

> +                     "at 0x" TARGET_FMT_lx, (target_ulong) fault_ipa);
> +    } else {
> +        uint32_t ins;
> +        uint8_t ins_fetched;
> +
> +        /*
> +         * Get current PC before it will get updated to except vector entry

"exception"

> +         */
> +        target_ulong ins_addr = is_a64(env) ? env->pc
> +                                /* AArch32 mode vs T32 aka Thumb mode */
> +                                : env->regs[15] - (env->thumb ? 4 : 8);
> +
> +        /*
> +         * Get the faulting instruction
> +         * Instructions have a fixed length of 32 bits
> +         * and are always little-endian

...unless they're 16-bit Thumb. In an ideal world you should handle
the Thumb case by doing a 16-bit load, and then loading another 16-bit
value if the top 3 bits of the first part are 0b111.

At least we don't have to consider the possibility ofold-style
really-big-endian-instruction-ordering :-)

> +         */
> +        ins_fetched = !cpu_memory_rw_debug(cs, ins_addr, (uint8_t *) &ins,
> +                                           sizeof(uint32_t), store_ins) ;

stray space before semicolon

> +
> +        error_report("Data abort exception with no valid ISS generated by "
> +                     "memory access at 0x" TARGET_FMT_lx,
> +                     (target_ulong)fault_ipa);
> +        error_report(ins_fetched ? "%s: 0x%" PRIx32 " (encoded)" : "%s",
> +                     "Unable to emulate memory instruction",
> +                     (!ins_fetched ? 0 : ldl_le_p(&ins)));
> +
> +        error_report("Faulting instruction at 0x" TARGET_FMT_lx, ins_addr);
> +    }
> +    /*
> +     * Set pending ext dabt amd trigger SET_EVENTS so that
> +     * KVM can inject the abort
> +     */
> +    env->ext_dabt_pending = 1;
> +
> +    ret = kvm_put_vcpu_events(cpu);
> +
> +    /* Get the most up-tp-date state */

"to"

> +    if (!ret) {
> +        /* Hacky but the sync needs to be forced */
> +        kvm_cpu_synchronize_state_force(cs);
> +    }
> +    return ret;
> +}
> diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
> index 32bf8d6757..5539c3a3f2 100644
> --- a/target/arm/kvm32.c
> +++ b/target/arm/kvm32.c
> @@ -240,6 +240,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      /* Check whether userspace can specify guest syndrome value */
>      kvm_arm_init_serror_injection(cs);
>
> +    /* Set status for supporting the extarnal dabt injection */
> +    kvm_arm_init_ext_dabt_injection(cs);
> +
>      return kvm_arm_init_cpreg_list(cpu);
>  }
>
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 876184b8fe..3da4e2e70e 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -792,6 +792,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      /* Check whether user space can specify guest syndrome value */
>      kvm_arm_init_serror_injection(cs);
>
> +    /* Set status for supporting the extarnal dabt injection */
> +    kvm_arm_init_ext_dabt_injection(cs);
> +
>      return kvm_arm_init_cpreg_list(cpu);
>  }

thanks
-- PMM

Re: [RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Beata Michalska]

On Mon, 6 Jan 2020 at 17:15, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 20 Dec 2019 at 20:27, Beata Michalska
> <beata.michalska@linaro.org> wrote:
> >
> > On ARMv7 & ARMv8 some load/store instructions might trigger a data abort
> > exception with no valid ISS info to be decoded. The lack of decode info
> > makes it at least tricky to emulate those instruction which is one of the
> > (many) reasons why KVM will not even try to do so.
> >
> > Add suport for handling those by requesting KVM to inject external
> > dabt into the quest.
> >
> > Signed-off-by: Beata Michalska <beata.michalska@linaro.org>
> > ---
> >  accel/kvm/kvm-all.c    | 15 +++++++
> >  accel/stubs/kvm-stub.c |  4 ++
> >  include/sysemu/kvm.h   |  1 +
> >  target/arm/cpu.h       |  3 +-
> >  target/arm/kvm.c       | 95 ++++++++++++++++++++++++++++++++++++++++++
> >  target/arm/kvm32.c     |  3 ++
> >  target/arm/kvm64.c     |  3 ++
> >  target/arm/kvm_arm.h   | 19 +++++++++
> >  8 files changed, 142 insertions(+), 1 deletion(-)
> >
> > diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
> > index ca00daa2f5..a3ee038142 100644
> > --- a/accel/kvm/kvm-all.c
> > +++ b/accel/kvm/kvm-all.c
> > @@ -2174,6 +2174,14 @@ static void do_kvm_cpu_synchronize_state(CPUState *cpu, run_on_cpu_data arg)
> >      }
> >  }
> >
> > +static void do_kvm_cpu_synchronize_state_force(CPUState *cpu,
> > +                                               run_on_cpu_data arg)
> > +{
> > +    kvm_arch_get_registers(cpu);
> > +    cpu->vcpu_dirty = true;
> > +}
>
> Why is this functionality special such that it needs a non-standard
> way of synchronizing state with KVM that nothing else does ?

We need the up-to-date state when handling the NISV and this is being
achieved by calling  synchronise cpu state. This will set the 'dirty' flag,
as expected. In order to get KVM to insert the external abort we need to
set vcpu events. So far so good. Still, as the cpu state is marked as 'dirty',
before entering the KVM_RUN again, Qemu will overwrite  relevant regs
-> see kvm_arch_put_registers. This messes up with  the regs
setup done by KVM when injecting the external abort. This is why we need
a sequence:
sync vcpu -> put events -> sync vcpu again
so that when entering KVM_RUN Qemu has all the updates needed.
Now, I could just set the 'dirty' flag to false and skip the regular
kvm_arch_put_registers after setting the vcpu event,
but it seemed more  sensible to have the sync version that despite the dirty
flag performs the requested sync. This is just to avoid hacky way
round the problem.
I was tempted to re-factor slightly syncing regs and events but that seemed
too invasive for this particular change. I might try to do just that though.

>
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index 9fe233b9bf..0cacc61d8a 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -483,6 +483,7 @@ void kvm_cpu_synchronize_state(CPUState *cpu);
> >  void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> >  void kvm_cpu_synchronize_post_init(CPUState *cpu);
> >  void kvm_cpu_synchronize_pre_loadvm(CPUState *cpu);
> > +void kvm_cpu_synchronize_state_force(CPUState *cpu);
> >
> >  void kvm_init_cpu_signals(CPUState *cpu);
> >
> > diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> > index 5f70e9e043..e11b5e7438 100644
> > --- a/target/arm/cpu.h
> > +++ b/target/arm/cpu.h
> > @@ -558,7 +558,8 @@ typedef struct CPUARMState {
> >          uint8_t has_esr;
> >          uint64_t esr;
> >      } serror;
> > -
> > +    /* Status field for pending extarnal dabt */
>
> "external" (I think you have the same typo later in the patch too)

And I did run the codespell  ... Will blame it on Christmas rush ...
And will fix all the typos.

>
> > +    uint8_t ext_dabt_pending;
> >      /* State of our input IRQ/FIQ/VIRQ/VFIQ lines */
> >      uint32_t irq_line_state;
>
>
> > @@ -701,6 +719,11 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
> >              ret = EXCP_DEBUG;
> >          } /* otherwise return to guest */
> >          break;
> > +    case KVM_EXIT_ARM_NISV:
> > +        /* External DAB with no valid iss to decode */
>
> "DABT"
>
> > +        ret = kvm_arm_handle_dabt_nisv(cs, run->arm_nisv.esr_iss,
> > +                                 run->arm_nisv.fault_ipa);
>
> (indentation looks odd here?)

It does indeed ....
>
> > +        break;
> >      default:
> >          qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n",
> >                        __func__, run->exit_reason);
> > @@ -835,3 +858,75 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
> >  {
> >      return (data - 32) & 0xffff;
> >  }
> > +
> > +int kvm_arm_handle_dabt_nisv(CPUState *cs, uint64_t esr_iss,
> > +                              uint64_t fault_ipa)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(cs);
> > +    CPUARMState *env = &cpu->env;
> > +    hwaddr xlat, len;
> > +    AddressSpace *as = cs->as ? cs->as : &address_space_memory;
>
> I don't think you should need to test cs->as for NULL;
> qemu_init_vcpu() ensures that one is always set up.
>
> Probably the neatest way to get the AddressSpace* is to
> call arm_addressspace(cs, MEMTXATTRS_UNSPECIFIED).
>
All right, will update that.
>
> > +    int store_ins = ((esr_iss >> 6) & 1); /* WnR bit */
>
> This should be a bool (matching the argument type for
> address_space_translate), so you can just do
>    bool is_write = esr_iss & (1 << 6);
>
Ditto
> > +    int ret;
> > +
> > +    /*
> > +     * Note: The ioctl for SET_EVENTS will be triggered before next
> > +     * KVM_RUN call though the vcpu regs need to be updated before hand
> > +     * so to not to overwrite changes done by KVM upon injecting the abort.
> > +     * This sadly requires running sync twice - shame ...
> > +     */
> > +    kvm_cpu_synchronize_state(cs);
> > +   /*
> > +    * ISS [23:14] is invalid so there is a limited info
> > +    * on what has just happened so the only *useful* thing that can
> > +    * be retrieved from ISS is WnR & DFSC (though in some cases WnR
> > +    * might be less of a value as well)
> > +    */
> > +    /* Verify whether the memory being accessed is even recognisable */
> > +    if (!address_space_translate(as, fault_ipa, &xlat, &len,
> > +                                 store_ins, MEMTXATTRS_UNSPECIFIED)) {
> > +        error_report("An attemp to access memory outside of the boundries"
>
> "attempt"; "boundaries". But I think what we should actually report here
> is:
>
> "Guest attempted to access memory/device at guest physical address
> 0x... using an instruction that KVM could not emulate (no valid ISS).
> Injecting a data abort exception into guest."
>
> It doesn't seem to me worth distinguishing whether the guest
> physical address happens to have anything mapped there or not;
> we're going to inject a somewhat bogus dabt anyway.
>
> After the initial error_report(), supplemental information like
> the offending instruction should be printed with error_printf(),
> because it isn't a separate new error.
>
Will fix that as well.

> > +                     "at 0x" TARGET_FMT_lx, (target_ulong) fault_ipa);
> > +    } else {
> > +        uint32_t ins;
> > +        uint8_t ins_fetched;
> > +
> > +        /*
> > +         * Get current PC before it will get updated to except vector entry
>
> "exception"
>
> > +         */
> > +        target_ulong ins_addr = is_a64(env) ? env->pc
> > +                                /* AArch32 mode vs T32 aka Thumb mode */
> > +                                : env->regs[15] - (env->thumb ? 4 : 8);
> > +
> > +        /*
> > +         * Get the faulting instruction
> > +         * Instructions have a fixed length of 32 bits
> > +         * and are always little-endian
>
> ...unless they're 16-bit Thumb. In an ideal world you should handle
> the Thumb case by doing a 16-bit load, and then loading another 16-bit
> value if the top 3 bits of the first part are 0b111.
>
Sigh, I have completely missed that (ashamed)

> At least we don't have to consider the possibility ofold-style
> really-big-endian-instruction-ordering :-)
>
> > +         */
> > +        ins_fetched = !cpu_memory_rw_debug(cs, ins_addr, (uint8_t *) &ins,
> > +                                           sizeof(uint32_t), store_ins) ;
>
> stray space before semicolon
>
> > +
> > +        error_report("Data abort exception with no valid ISS generated by "
> > +                     "memory access at 0x" TARGET_FMT_lx,
> > +                     (target_ulong)fault_ipa);
> > +        error_report(ins_fetched ? "%s: 0x%" PRIx32 " (encoded)" : "%s",
> > +                     "Unable to emulate memory instruction",
> > +                     (!ins_fetched ? 0 : ldl_le_p(&ins)));
> > +
> > +        error_report("Faulting instruction at 0x" TARGET_FMT_lx, ins_addr);
> > +    }
> > +    /*
> > +     * Set pending ext dabt amd trigger SET_EVENTS so that
> > +     * KVM can inject the abort
> > +     */
> > +    env->ext_dabt_pending = 1;
> > +
> > +    ret = kvm_put_vcpu_events(cpu);
> > +
> > +    /* Get the most up-tp-date state */
>
> "to"
>
> > +    if (!ret) {
> > +        /* Hacky but the sync needs to be forced */
> > +        kvm_cpu_synchronize_state_force(cs);
> > +    }
> > +    return ret;
> > +}
> > diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
> > index 32bf8d6757..5539c3a3f2 100644
> > --- a/target/arm/kvm32.c
> > +++ b/target/arm/kvm32.c
> > @@ -240,6 +240,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >      /* Check whether userspace can specify guest syndrome value */
> >      kvm_arm_init_serror_injection(cs);
> >
> > +    /* Set status for supporting the extarnal dabt injection */
> > +    kvm_arm_init_ext_dabt_injection(cs);
> > +
> >      return kvm_arm_init_cpreg_list(cpu);
> >  }
> >
> > diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> > index 876184b8fe..3da4e2e70e 100644
> > --- a/target/arm/kvm64.c
> > +++ b/target/arm/kvm64.c
> > @@ -792,6 +792,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >      /* Check whether user space can specify guest syndrome value */
> >      kvm_arm_init_serror_injection(cs);
> >
> > +    /* Set status for supporting the extarnal dabt injection */
> > +    kvm_arm_init_ext_dabt_injection(cs);
> > +
> >      return kvm_arm_init_cpreg_list(cpu);
> >  }
>

Thanks for reviewing!

BR
Beata
> thanks
> -- PMM

Re: [RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Peter Maydell]

On Fri, 20 Dec 2019 at 20:27, Beata Michalska
<beata.michalska@linaro.org> wrote:
>
> On ARMv7 & ARMv8 some load/store instructions might trigger a data abort
> exception with no valid ISS info to be decoded. The lack of decode info
> makes it at least tricky to emulate those instruction which is one of the
> (many) reasons why KVM will not even try to do so.
>
> Add suport for handling those by requesting KVM to inject external
> dabt into the quest.
>
> Signed-off-by: Beata Michalska <beata.michalska@linaro.org>
> ---
> +        /*
> +         * Get current PC before it will get updated to except vector entry
> +         */
> +        target_ulong ins_addr = is_a64(env) ? env->pc
> +                                /* AArch32 mode vs T32 aka Thumb mode */
> +                                : env->regs[15] - (env->thumb ? 4 : 8);

Another thing that occurred to me last night -- why do we need
to do this adjustment of the PC/r15 ? If this is the kernel
handing control to userspace to say "this is not an instruction
I can handle, maybe you'd like to try" then surely it should
do so with the PC pointing at the offending instruction?
Similarly, if we ask the kernel to inject a data abort I
would expect that the kernel would do the work of adjusting
the PC forwards as the architecture requires when taking
the exception.

thanks
-- PMM

Re: [RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Beata Michalska]

On Tue, 7 Jan 2020 at 14:28, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 20 Dec 2019 at 20:27, Beata Michalska
> <beata.michalska@linaro.org> wrote:
> >
> > On ARMv7 & ARMv8 some load/store instructions might trigger a data abort
> > exception with no valid ISS info to be decoded. The lack of decode info
> > makes it at least tricky to emulate those instruction which is one of the
> > (many) reasons why KVM will not even try to do so.
> >
> > Add suport for handling those by requesting KVM to inject external
> > dabt into the quest.
> >
> > Signed-off-by: Beata Michalska <beata.michalska@linaro.org>
> > ---
> > +        /*
> > +         * Get current PC before it will get updated to except vector entry
> > +         */
> > +        target_ulong ins_addr = is_a64(env) ? env->pc
> > +                                /* AArch32 mode vs T32 aka Thumb mode */
> > +                                : env->regs[15] - (env->thumb ? 4 : 8);
>
> Another thing that occurred to me last night -- why do we need
> to do this adjustment of the PC/r15 ? If this is the kernel
> handing control to userspace to say "this is not an instruction
> I can handle, maybe you'd like to try" then surely it should
> do so with the PC pointing at the offending instruction?
> Similarly, if we ask the kernel to inject a data abort I
> would expect that the kernel would do the work of adjusting
> the PC forwards as the architecture requires when taking
> the exception.
>

The code here is just for easing debugging from Qemu perspective
and that is the only reason why we even try to read the value of PC
- it is not in any way needed by kernel to inject the abort.
One can use the monitor to decode the instruction, provided it is still
available at the memory location pointed by PC (handy monitor_disas)
- that is why logging the address with decoded instruction,
as it is the only thing that is being done here. Still the address of actually
executed instruction for ARM would be PC–8 (PC–4 for Thumb)
that's why the adjustment.

BR
Beata


> thanks
> -- PMM

Re: [RFC PATCH 1/1] target/arm: kvm: Handle DABT with no valid ISS

[Peter Maydell]

On Tue, 7 Jan 2020 at 21:37, Beata Michalska <beata.michalska@linaro.org> wrote:
>
> On Tue, 7 Jan 2020 at 14:28, Peter Maydell <peter.maydell@linaro.org> wrote:
> > Another thing that occurred to me last night -- why do we need
> > to do this adjustment of the PC/r15 ? If this is the kernel
> > handing control to userspace to say "this is not an instruction
> > I can handle, maybe you'd like to try" then surely it should
> > do so with the PC pointing at the offending instruction?
> > Similarly, if we ask the kernel to inject a data abort I
> > would expect that the kernel would do the work of adjusting
> > the PC forwards as the architecture requires when taking
> > the exception.
> >
>
> The code here is just for easing debugging from Qemu perspective
> and that is the only reason why we even try to read the value of PC
> - it is not in any way needed by kernel to inject the abort.
> One can use the monitor to decode the instruction, provided it is still
> available at the memory location pointed by PC (handy monitor_disas)
> - that is why logging the address with decoded instruction,
> as it is the only thing that is being done here. Still the address of actually
> executed instruction for ARM would be PC–8 (PC–4 for Thumb)
> that's why the adjustment.

My point is that the kernel hasn't actually executed the instruction,
though -- that's why it's given us control. It seems a bit exposing
of the underlying hw mechanism for the kernel to hand control
back to userspace with the vcpu state in a sort of half-way-through
situation, rather than cleanly rolling it back to "this insn has not
executed at all yet".

thanks
-- PMM