[PATCH v3 00/39] [PATCH v2 00/39] [RISC-V] virtual interrupt controller (vAPLIC/vIMSIC) support

[PATCH v3 00/39] [PATCH v2 00/39] [RISC-V] virtual interrupt controller (vAPLIC/vIMSIC) support

[Oleksii Kurochko]

Hi all,

This series adds the initial virtual interrupt controller (vINTC) support
for RISC-V guests in Xen, based on the Advanced Interrupt Architecture
(AIA): a virtual APLIC (vAPLIC) in MSI mode backed by a virtual IMSIC
(vIMSIC) using hardware guest interrupt files.

Rather than emulating APLIC in direct-delivery mode (which requires
trap-and-emulate for every interrupt and is costly), the series targets
IMSIC from the start. AIA lets a hart implement several "guest interrupt
files" (up to GEILEN), so external interrupts can be delivered to a vCPU
directly by hardware via the VGEIN field of hstatus, without a hypervisor
round-trip. Xen only has to emulate the APLIC MMIO programming interface
and route the guest's intent onto the physical MSI topology; interrupt
delivery itself stays in hardware.

The work breaks down into a few logical blocks:

Preparatory fixes and cleanups (patches 1-6)
  - Drop the unused pregs field of struct cpu_user_regs and bug.h's
    duplicate instruction length helpers.
  - Program hstatus.VSXL explicitly, as decoding a trapped instruction
    depends on the effective XLEN of the guest.
  - csr_read64() as the counterpart of csr_write64(), used for CSR_TIME so
    that get_cycles() no longer truncates the time counter on RV32, and
    UINT64_MAX rather than ULONG_MAX to disable the VS-timer.
  - Request a G-stage flush on vmenter where VMIDs are unavailable, so a
    domain cannot run on the translations left behind by the one which ran
    on that hart before it.

APLIC groundwork and vAPLIC MMIO emulation (patches 7-11)
  - Add the missing APLIC register offsets/masks needed by both the
    physical and virtual APLIC code, rearranging asm/aplic.h in the style
    of x86's asm/msr-index.h (no functional change).
  - A per-domain MMIO handler table modelled on Arm's framework, so
    emulated devices self-register their GPA ranges and the fault path
    stays agnostic via a single try_handle_mmio() entry point.
  - vAPLIC MMIO read/write emulation. Writes are gated by the domain's
    authorised-IRQ bitmap so a guest cannot touch interrupts it does not
    own, and TARGET writes are translated from virtual to physical
    hart/guest-file indices. Delegation (SOURCECFG.D) is not yet
    supported.
  - Build the physical APLIC's target hart index with aplic_hart_field()
    as well, dropping the last in-tree duplicate of the AIA hart index
    formula, and add a helper to test for APLIC MSI mode.

vCPU context switching (patches 12-15)
  - context_switch() and the helpers it needs: save/restore of H/VS CSRs,
    the virtual timer and P2M context, and __context_switch() in assembly.
    The VMID is claimed in p2m_ctxt_switch_to() rather than at the next
    guest entry, which leaves p2m_handle_vmenter() with nothing to do.
  - Save and restore the AIA CSRs a guest can change (vsiselect and
    hviprio{1,2}), gated by hstateen0 where Smstateen is implemented.
  - vintc_ctxt_switch_{from,to}() wrappers over new ctxt_switch_{from,to}
    hooks in struct vintc_ops, called from the context switch path, plus
    the IMSIC implementation of those hooks: it records which pCPU owns a
    vCPU's guest interrupt file, as the pCPU id is part of the MSI
    address.

Trap and instruction emulation infrastructure (patches 16-26)
  - Extend the exception-table format with type/data fields and add
    EX_TYPE_TRAP_INFO so fixups can capture sepc/scause/stval, and look
    the table up for any trap taken in Xen context rather than for illegal
    instructions only, so that the hlv/hlvx sequences reach their fixup.
  - A guest page-fault handler, and trap_redirect() to forward a
    synchronous trap back into the guest's VS-mode handler for the faults
    which can never become an emulated access.
  - Resolve the faulting guest physical address from htval and stval,
    which first needs Shtvala to be detected, and define all four
    INSN_PSEUDO_VS_* values independently of the hypervisor's XLEN, so
    that a fault taken on an implicit VS-stage access is recognized as one
    rather than mistaken for an MMIO trap.
  - riscv_read_guest() (HLV/HLVX) to read guest memory and instructions
    safely, the decoding helpers shared by both access types, and the load
    and store emulation which dispatches the access through
    try_handle_mmio().

vCPU migration between pCPUs (patches 27-34) (introduced here for better context
of VGEIN fumctions usage)
  - arch_move_irqs(), dispatching through a new move_irqs hook in
    struct vintc_ops down to imsic_migrate_vcpu(), and the case where a
    vCPU has no guest interrupt file to move yet.
  - The move of a vCPU's IMSIC guest interrupt file itself, following the
    sequence the AIA spec prescribes: quiesce and save eidelivery/
    eithreshold of the old file, zero the new one, G-stage remap it,
    retarget the domain's APLIC interrupts at it
    (aplic_reconfigure_target()) and fence off straggler MSIs with a
    genmsi barrier, dump the old file's eip/eie arrays to memory, then
    restore that state into the new file and update the vCPU's
    hstatus.VGEIN.

VGEIN allocation and vCPU bring-up (patches 35-39)
  - Per-pCPU VGEIN (guest interrupt file) allocator: a bitmap of the files
    a hart implements (up to GEILEN) with helpers to assign and release
    one, and an owners[] map so a file reported pending in HGEIP can be
    traced back to the vCPU it belongs to.
  - Watch a descheduled vCPU's guest interrupt file through HGEIE, so that
    a guest blocked on an external interrupt is woken up instead of
    waiting for an unrelated event to schedule it again.
  - Stage-2 map a vCPU's physical guest interrupt file to the fixed
    per-vCPU GPA page the guest expects at offset 0.
  - continue_new_vcpu(): switch to the idle vCPU's own stack for the idle
    vCPU, and enter the guest through the new return_to_new_vcpu() path in
    entry.S for a guest one.
  - imsic_vsfile_attach(), called once the pCPU a vCPU will run on is
    known: it assigns a VGEIN, maps the guest interrupt file and records
    the IMSIC state as a consistent unit.

CI tests: https://gitlab.com/xen-project/people/olkur/xen/-/pipelines/2897809275

---
Changes in v3:
 - Address the comments from ML.
 - New patches: "drop INSN_{16,32}BIT_MASK and report unknown instruction
   lengths", "set the domain type for 64-bit kernel images" and
   "introduce CPU_NONE" (used instead of NR_CPUS as "no CPU" marker).
 - Drop patch series dependecny as it was merged to upstream/staging.
---
Changes in v2:
 - The series has grown from 17 to 39 patches. vCPU context switching, vCPU
   migration between pCPUs and the vCPU bring-up path (continue_new_vcpu(),
   attaching an IMSIC h/w interrupt file) are now part of it to have better
   context of how things are using, together with the trap-side pieces the
   MMIO emulation depends on (faulting GPA resolution, Shtvala detection,
   instruction decoding).
 - vintc_state_{save,restore}() became vintc_ctxt_switch_{from,to}() and
   vcpu_aia_init() became imsic_vsfile_attach(); "xen/riscv: manage
   IRQ_DISABLED flag in APLIC irq enable/disable callbacks" is no longer part
   of this series. The remaining changes are described in the per-patch
   changelogs.
---

Oleksii Kurochko (39):
  xen/riscv: drop INSN_{16,32}BIT_MASK and report unknown instruction
    lengths
  xen/riscv: set the domain type for 64-bit kernel images
  xen/riscv: introduce csr_volatile_read64()
  xen/riscv: request a G-stage flush on vmenter when VMIDs are disabled
  xen/riscv: use ~0 instead of ULONG_MAX to disable the VS-timer
  xen/riscv: introduce device-agnostic MMIO emulation dispatch
  xen/riscv: introduce CPU_NONE
  xen/riscv: implement virtual APLIC MMIO emulation
  xen/riscv: build the target hart index via aplic_hart_index()
  xen/riscv: add helper to check APLIC MSI mode
  xen/riscv: implement vCPU context switching
  xen/riscv: save and restore vsiselect on vCPU context switch
  xen/riscv: introduce vintc_ctxt_switch_{from,to}()
  xen/riscv: add IMSIC vCPU context switch handlers
  xen/riscv: extend exception tables with type and data fields
  xen/riscv: decouple INSN_PSEUDO_VS_* from the hypervisor's XLEN
  xen/riscv: add guest page fault handling stub
  xen/riscv: implement trap redirection to a guest
  xen/riscv: detect Shtvala
  xen/riscv: resolve the faulting guest physical address
  xen/riscv: add guest memory read helper
  xen/riscv: look up the exception table for any trap taken in Xen
    context
  xen/riscv: add helpers for decoding a trapped load or store
  xen/riscv: add guest load emulation for trapped MMIO accesses
  xen/riscv: add guest store emulation for trapped MMIO accesses
  xen/riscv: introduce arch_move_irqs()
  xen/riscv: handle the case when no vCPU migration is needed
  xen/riscv: introduce vaplic_reconfigure_target()
  xen/riscv: prepare new IMSIC VS-file
  xen/riscv: implement APLIC-IMSIC-hart sync barrier for vCPU migration
  xen/riscv: remap interrupts to new IMSIC VS-file
  xen/riscv: save and clear the old IMSIC VS-file on vCPU migration
  xen/riscv: restore register state in the new IMSIC VS-file
  xen/riscv: add basic VGEIN management for AIA guests
  xen/riscv: wake up a descheduled vCPU on a guest external interrupt
  xen/riscv: map IMSIC interrupt file for vCPUs
  xen/riscv: implement continue_new_vcpu()
  xen/riscv: set the guest's XLEN explicitly in hstatus.VSXL
  xen/riscv: introduce IMSIC h/w interrupt file attaching to vcpu

 docs/misc/riscv/booting.txt                 |  11 +
 xen/arch/riscv/Makefile                     |   2 +
 xen/arch/riscv/aia.c                        | 185 +++++
 xen/arch/riscv/aplic-priv.h                 |   2 +
 xen/arch/riscv/aplic.c                      | 223 +++++-
 xen/arch/riscv/cpufeature.c                 |   1 +
 xen/arch/riscv/domain-build.c               |   8 +
 xen/arch/riscv/domain.c                     | 338 ++++++++-
 xen/arch/riscv/emulate.c                    | 588 ++++++++++++++++
 xen/arch/riscv/entry.S                      |  87 +++
 xen/arch/riscv/extable.c                    |  48 +-
 xen/arch/riscv/guestcopy.c                  |  88 +++
 xen/arch/riscv/imsic.c                      | 734 +++++++++++++++++++-
 xen/arch/riscv/include/asm/aia.h            |   7 +
 xen/arch/riscv/include/asm/aplic.h          |  41 +-
 xen/arch/riscv/include/asm/config.h         |   3 +
 xen/arch/riscv/include/asm/cpufeature.h     |   1 +
 xen/arch/riscv/include/asm/csr.h            |  40 ++
 xen/arch/riscv/include/asm/current.h        |   4 +
 xen/arch/riscv/include/asm/domain.h         |  23 +-
 xen/arch/riscv/include/asm/emulate.h        |  10 +
 xen/arch/riscv/include/asm/extable.h        |  61 +-
 xen/arch/riscv/include/asm/gpr-num.h        |  53 ++
 xen/arch/riscv/include/asm/guest_access.h   |   4 +
 xen/arch/riscv/include/asm/imsic.h          |  26 +-
 xen/arch/riscv/include/asm/intc.h           |  12 +
 xen/arch/riscv/include/asm/irq.h            |   5 +-
 xen/arch/riscv/include/asm/mmio.h           |  63 ++
 xen/arch/riscv/include/asm/p2m.h            |   1 -
 xen/arch/riscv/include/asm/processor.h      |  14 +-
 xen/arch/riscv/include/asm/regs.h           |  23 +-
 xen/arch/riscv/include/asm/riscv_encoding.h |  69 +-
 xen/arch/riscv/include/asm/setup.h          |   2 +
 xen/arch/riscv/include/asm/system.h         |   2 +
 xen/arch/riscv/include/asm/time.h           |   4 +-
 xen/arch/riscv/include/asm/traps.h          |   9 +
 xen/arch/riscv/include/asm/vaplic.h         |  22 +
 xen/arch/riscv/intc.c                       |  22 +
 xen/arch/riscv/kernel.c                     |   2 +
 xen/arch/riscv/mmio.c                       | 175 +++++
 xen/arch/riscv/p2m.c                        | 155 +++--
 xen/arch/riscv/riscv64/asm-offsets.c        |  19 +-
 xen/arch/riscv/setup.c                      |   2 +
 xen/arch/riscv/stubs.c                      |   5 -
 xen/arch/riscv/time.c                       |   4 +-
 xen/arch/riscv/traps.c                      | 108 ++-
 xen/arch/riscv/vaplic.c                     | 447 +++++++++++-
 xen/arch/riscv/vmid.c                       |   4 +-
 48 files changed, 3600 insertions(+), 157 deletions(-)
 create mode 100644 xen/arch/riscv/emulate.c
 create mode 100644 xen/arch/riscv/include/asm/emulate.h
 create mode 100644 xen/arch/riscv/include/asm/gpr-num.h
 create mode 100644 xen/arch/riscv/include/asm/mmio.h
 create mode 100644 xen/arch/riscv/mmio.c

-- 
2.55.0

[PATCH v3 01/39] xen/riscv: drop INSN_{16,32}BIT_MASK and report unknown instruction lengths

[Oleksii Kurochko]

Both constants were inherited from OpenSBI and their names are misleading.
The length of an instruction is encoded in its low bits: bits[1:0] != 11
means 16 bits, otherwise bits[4:2] != 111 means 32 bits, and anything else
is 48 bits or wider. 0x3 and 0x1c are the patterns compared against in
these two tests, not masks selecting a 16-bit or a 32-bit instruction. As
each constant is needed in one place only, once INSN_IS_16BIT() and
INSN_IS_32BIT() encapsulate the tests, drop them and spell the values out
in those two macros instead.

While at it, make INSN_LEN() cover the case both macros already knew about:
an encoding of 48 bits or wider now yields 0 rather than being silently
taken for a 32-bit instruction. No ratified extension defines an instruction
of such a length, so rather than open-coding a decoder for encodings which
cannot legitimately occur, leave it to the caller to treat the 0 as an
illegal instruction.

No functional change: the only present caller applies INSN_LEN() to a bug
frame, i.e. to an instruction emitted by Xen itself, which is either 16 or
32 bits wide.

Suggested-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - New patch.
---
---
 xen/arch/riscv/include/asm/riscv_encoding.h | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index 03e186bcdb8c..647268e6d0d4 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -857,16 +857,18 @@
 #error "Unexpected __riscv_xlen"
 #endif
 
-#define INSN_16BIT_MASK			0x3
-#define INSN_32BIT_MASK			0x1c
+#define INSN_IS_16BIT(insn)	(((insn) & 3) != 3)
+#define INSN_IS_32BIT(insn)	(!INSN_IS_16BIT(insn) && ((insn) & 0x1c) != 0x1c)
 
-#define INSN_IS_16BIT(insn)		\
-	(((insn) & INSN_16BIT_MASK) != INSN_16BIT_MASK)
-#define INSN_IS_32BIT(insn)		\
-	(((insn) & INSN_16BIT_MASK) == INSN_16BIT_MASK && \
-	 ((insn) & INSN_32BIT_MASK) != INSN_32BIT_MASK)
-
-#define INSN_LEN(insn)			(INSN_IS_16BIT(insn) ? 2 : 4)
+/*
+ * Length in bytes of the instruction whose first parcel is @insn: 2 or 4, or
+ * 0 when the encoding is 48 bits or wider. No ratified extension defines an
+ * instruction of such a length, so rather than open-coding a decoder for
+ * encodings which cannot legitimately occur, leave it to the caller to treat
+ * the 0 as an illegal instruction.
+ */
+#define INSN_LEN(insn) \
+	(INSN_IS_16BIT(insn) ? 2 : (INSN_IS_32BIT(insn) ? 4 : 0))
 
 #if __riscv_xlen == 64
 #define LOG_REGBYTES			3
-- 
2.55.0

[PATCH v3 02/39] xen/riscv: set the domain type for 64-bit kernel images

[Oleksii Kurochko]

kernel_image64_probe() never set info->type, so every domain kept the
DOMAIN_32BIT value a zeroed KERNEL_INFO_INIT leaves behind, even though
Xen on RISC-V is only built for RV64 and accepts nothing but a 64-bit
Image.

Nothing consumes d->type on RISC-V yet, so there is no functional change.

Fixes: f6a20eda9212 ("xen/riscv: add Linux kernel loading support")
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
d->type being left wrong is a bug on its own, whether or not anything
consumes it yet. Its future user is "xen/riscv: set the guest's XLEN
explicitly in hstatus.VSXL" in this series.

The patch is placed ahead of that one, as it fixes an issue which already
exists and doesn't depend on it. If preferred, the two can be merged.
---
Changes in v3:
 - New patch.
---
---
 xen/arch/riscv/kernel.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/riscv/kernel.c b/xen/arch/riscv/kernel.c
index 8580cdb32d7a..5601b2818b37 100644
--- a/xen/arch/riscv/kernel.c
+++ b/xen/arch/riscv/kernel.c
@@ -233,6 +233,8 @@ static int __init kernel_image64_probe(struct kernel_info *info,
     info->image.text_offset = le64_to_cpu(image.text_offset);
     info->image.start = 0;
 
+    info->type = DOMAIN_64BIT;
+
     info->load = kernel_image_load;
 
     return 0;
-- 
2.55.0

[PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Oleksii Kurochko]

On RV32 a 64-bit CSR is accessed as two halves, <csr> and <csr>H. Reading
the halves with two separate instructions isn't always safe: for a CSR
which hardware increments, a carry from the low half into the high one can
happen in between, so a plain pair of reads can produce a value the CSR
never held. Therefore the high half is re-read and the sequence retried if
it changed in the meantime.

Only CSRs which change under Xen's feet need that, hence the name. The
majority can be read with a plain pair of csr_read()s, so no helper for
those is introduced here as there would be no user for it.

Use it for CSR_TIME, which is exactly such a counter, and widen cycles_t to
uint64_t. Otherwise get_cycles() would still truncate the time counter to
32 bits on RV32.

No functional change on RV64.

Fixes: a541ddadec0a ("xen/riscv: introduce time.h")
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Rename csr_read64() to csr_volatile_read64(): only CSRs which change
   under Xen's feet need the retry, the majority can be read with a plain
   split access, which isn't introduced here as it would have no user.
 - Carry the high half read at the end of an iteration over to the next
   one, instead of reading it twice per retry.
 - Update the commit message.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/include/asm/csr.h  | 27 +++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/time.h |  4 ++--
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/include/asm/csr.h b/xen/arch/riscv/include/asm/csr.h
index 888d6a2a86d6..a5cb24c863dd 100644
--- a/xen/arch/riscv/include/asm/csr.h
+++ b/xen/arch/riscv/include/asm/csr.h
@@ -39,12 +39,39 @@
     csr_write(csr, v_);             \
     csr_write(csr ## H, v_ >> 32);  \
 })
+
+/*
+ * The two halves are read by separate instructions, so a CSR which hardware
+ * increments can carry from the low half into the high one in between,
+ * yielding a value the CSR never held. Re-read the high half and retry the
+ * sequence if it changed.
+ *
+ * Only needed for CSRs which change under Xen's feet.
+ */
+#define csr_volatile_read64(csr)                    \
+({                                                  \
+    uint32_t hi_, lo_, old_;                        \
+                                                    \
+    hi_ = csr_read(csr ## H);                       \
+    do {                                            \
+        old_ = hi_;                                 \
+        lo_ = csr_read(csr);                        \
+    } while ( (hi_ = csr_read(csr ## H)) != old_ ); \
+                                                    \
+    ((uint64_t)hi_ << 32) | lo_;                    \
+})
 #else
 #define csr_write64(csr, val)       \
 ({                                  \
     csr_write(csr, val);            \
     (void)csr ## H;                 \
 })
+
+#define csr_volatile_read64(csr)    \
+({                                  \
+    (void)csr ## H;                 \
+    csr_read(csr);                  \
+})
 #endif
 
 #define csr_swap(csr, val)                                      \
diff --git a/xen/arch/riscv/include/asm/time.h b/xen/arch/riscv/include/asm/time.h
index 4d68900151a7..a7c9736ae719 100644
--- a/xen/arch/riscv/include/asm/time.h
+++ b/xen/arch/riscv/include/asm/time.h
@@ -18,11 +18,11 @@ static inline void force_update_vcpu_system_time(struct vcpu *v)
     BUG_ON("unimplemented");
 }
 
-typedef unsigned long cycles_t;
+typedef uint64_t cycles_t;
 
 static inline cycles_t get_cycles(void)
 {
-    return csr_read(CSR_TIME);
+    return csr_volatile_read64(CSR_TIME);
 }
 
 void preinit_xen_time(void);
-- 
2.55.0

[PATCH v3 04/39] xen/riscv: request a G-stage flush on vmenter when VMIDs are disabled

[Oleksii Kurochko]

vmid_handle_vmenter() reports that no flush is needed when VMIDs are
unavailable (e.g. vmid=off, or hardware with no more than one VMID bit).
Every domain then runs under VMID 0 with nothing flushed in between, so as
soon as a hart runs more than one domain, a domain entered there can use
the G-stage translations left behind by the domain which ran before it.

Report that a flush is needed on every entry instead.

Update the comment in p2m_handle_vmenter() accordingly: without VMIDs the
VS-stage TLB doesn't need a separate flush either, as the G-stage flush now
done on every entry already covers it.

While at it, spell the other early return as a bool literal.

Fixes: bff3b9ea4696 ("xen/riscv: introduce VMID allocation and manegement")
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/p2m.c  | 4 +++-
 xen/arch/riscv/vmid.c | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/xen/arch/riscv/p2m.c b/xen/arch/riscv/p2m.c
index 1cea86512c8c..de25607247a6 100644
--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -1584,7 +1584,9 @@ void p2m_handle_vmenter(void)
     /*
      * There is also no need to flush the VS-stage TLB: even if speculation
      * occurs (VSATP + old HGATP were used), it will use the old VMID, which
-     * won't be reused until need_flush is set to true.
+     * won't be reused until need_flush is set to true. When VMIDs aren't
+     * available there is no old VMID to rely on, but then need_flush is set
+     * on every entry, so the flush above covers that case.
      */
 }
 
diff --git a/xen/arch/riscv/vmid.c b/xen/arch/riscv/vmid.c
index 11c7e9d6d6c8..93714b359534 100644
--- a/xen/arch/riscv/vmid.c
+++ b/xen/arch/riscv/vmid.c
@@ -141,7 +141,7 @@ bool vmid_handle_vmenter(struct vcpu_vmid *vmid)
 
     /* Test if VCPU has valid VMID. */
     if ( read_atomic(&vmid->generation) == data->generation )
-        return 0;
+        return false;
 
     /* If there are no free VMIDs, need to go to a new generation. */
     if ( unlikely(data->next_vmid > data->max_vmid) )
@@ -164,7 +164,7 @@ bool vmid_handle_vmenter(struct vcpu_vmid *vmid)
 
  disabled:
     vmid->vmid = 0;
-    return 0;
+    return true;
 }
 
 /*
-- 
2.55.0

[PATCH v3 05/39] xen/riscv: use ~0 instead of ULONG_MAX to disable the VS-timer

[Oleksii Kurochko]

vstimecmp is a 64-bit CSR independently of XLEN, which is why it is
written with csr_write64(). On RV32 that macro splits the value into
the vstimecmp/vstimecmph pair, so passing ULONG_MAX (0xffffffff there)
writes all ones to the low half and zero to the high half, leaving the
CSR at 0x00000000ffffffff rather than at its maximum. A VS-timer irq
would then become pending as soon as (time + htimedelta) reaches 2^32,
which is exactly what the code is trying to avoid.

Use ~0 instead: csr_write64() converts its argument to uint64_t, and the
conversion of the signed -1 sign-extends to all ones, so both halves get
0xffffffff. On RV64 the result is equal to ULONG_MAX, so no functional
change there.

Fixes: 25e032730690 ("xen/riscv: allow Xen to use SSTC while hiding it from guests")
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
---
Changes in v3:
 - Add Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>.
 - Use ~0 instead of UINT64_MAX.
 - Update the subject and the commit message accordingly.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/time.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/time.c b/xen/arch/riscv/time.c
index 602c029641b8..782d9cd450f3 100644
--- a/xen/arch/riscv/time.c
+++ b/xen/arch/riscv/time.c
@@ -101,8 +101,8 @@ void __init preinit_xen_time(void)
          * A VS-timer interrupt becomes pending whenever the value of
          * (time + htimedelta) is greater than or equal to vstimecmp CSR.
          * Thereby to avoid spurious VS-timer irqs set vstimecmp CSR to
-         * ULONG_MAX.
+         * ~0.
          */
-        csr_write64(CSR_VSTIMECMP, ULONG_MAX);
+        csr_write64(CSR_VSTIMECMP, ~0);
     }
 }
-- 
2.55.0

[PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Oleksii Kurochko]

A RISC-V guest can be given several emulated devices at distinct GPA
ranges; the virtual interrupt controllers alone account for vPLIC on
legacy machines and vAPLIC together with vIMSIC on AIA-compliant ones.
Routing MMIO faults via a per-device is_access() check in the trap
handler would couple that handler to every device it must serve,
requiring a new conditional branch in the fault path for each emulated
device added.

Introduce a per-domain MMIO handler registration table, modeled
after the equivalent ARM framework, so that virtual devices
self-register their GPA ranges and read/write callbacks at domain
creation time. The MMIO fault path delegates to a single do_mmio()
entry point and remains agnostic of which device owns a particular
address.

Nothing registers a handler and do_mmio() has no callers yet, so this
patch is a no-op; the trap handler is left untouched.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
do_mmio() gets its first caller in "xen/riscv: add guest load emulation
for trapped MMIO accesses" in this series.
---
Changes in v3:
 - Update the commit message.
 - Capitalize the comments inside mmio_info_t and drop the list of access
   widths from the one on len, as it could go stale.
 - Drop the parentheses around the typedef names.
 - Reject an access width which isn't a power of 2 in do_mmio().
 - Fold the !size check of register_mmio_handler() into end <= addr.
 - Return -EADDRINUSE instead of -EEXIST for an overlapping region.
 - Drop setting d->arch.vmmio.num_entries to 0, as struct domain starts
   out zero-filled.
---
Changes in v2:
 - Drop copyright from mmio.c as it will go stale anyway as code moves
   around.
 - Drop the max_count parameter of domain_io_init() (MAX_IO_HANDLER is a
   global boundary) and embed the handler array directly in struct vmmio
   as struct mmio_handler handlers[MAX_IO_HANDLER].  This removes the
   xvzalloc_array() allocation, the max_num_entries field and
   domain_io_free() altogether; domain_io_init() consequently cannot fail
   any longer and now returns void. Note this goes slightly beyond the
   suggested variant in that max_num_entries is dropped as well, since
   ARRAY_SIZE(vmmio->handlers) serves the same purpose.
 - Drop the separate register_t argument of mmio_read_t/mmio_write_t;
   handlers now produce and consume the value through info->data.
   handle_read()/handle_write() are gone as a result, with
   try_handle_mmio() invoking ops->read()/ops->write() directly.
 - Turn mmio_read_t/mmio_write_t into function types rather than
   pointer-to-function types, so that pointer-ness is visible at the use
   sites in struct mmio_handler_ops. Constify the mmio_info_t * of the
   write callback, which has no reason to modify it any more.
 - register_mmio_handler() returns int instead of BUG_ON()ing on a full
   table: -ENOSPC now lets the caller fail domain creation. It also
   validates its inputs, rejecting a NULL ops (or one with a missing
   read/write callback) as well as zero-sized and address-wrapping
   regions with -EINVAL.
 - Guarantee the non-overlap property that cmp_mmio_handler() relies on:
   register_mmio_handler() checks the new region against both neighbours
   of its insertion slot and returns -EEXIST on overlap.
 - Replace the sort() call per registration with an insertion into the
   already sorted array: locate the slot and memmove() the tail up by
   one. sort(), swap_mmio_handler() and <xen/sort.h> are gone.
 - Extend cmp_mmio_handler()'s comment to state that it is a bsearch()
   comparator and to explain the key/elem asymmetry; document why the
   neighbours' addr + size cannot overflow.
 - Fix over-long lines and a mis-indented label.
---
---
 xen/arch/riscv/Makefile             |   1 +
 xen/arch/riscv/domain.c             |   3 +
 xen/arch/riscv/include/asm/domain.h |   3 +
 xen/arch/riscv/include/asm/mmio.h   |  63 ++++++++++
 xen/arch/riscv/mmio.c               | 175 ++++++++++++++++++++++++++++
 5 files changed, 245 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/mmio.h
 create mode 100644 xen/arch/riscv/mmio.c

diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index 665ec2693335..5454517724bd 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -14,6 +14,7 @@ obj-y += intc.o
 obj-y += irq.o
 obj-y += kernel.init.o
 obj-y += mm.o
+obj-y += mmio.o
 obj-y += p2m.o
 obj-y += paging.o
 obj-y += pt.o
diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index d94652809e36..fd6229635627 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -12,6 +12,7 @@
 #include <asm/cpufeature.h>
 #include <asm/csr.h>
 #include <asm/intc.h>
+#include <asm/mmio.h>
 #include <asm/riscv_encoding.h>
 #include <asm/vtimer.h>
 
@@ -310,6 +311,8 @@ int arch_domain_create(struct domain *d,
     if ( (rc = p2m_init(d, config)) != 0)
         goto fail;
 
+    domain_io_init(d);
+
     if ( (rc = domain_vintc_init(d)) )
         goto fail;
 
diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
index e035b33ddfdc..15e8fa19685e 100644
--- a/xen/arch/riscv/include/asm/domain.h
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -9,6 +9,7 @@
 
 #include <asm/cpufeature.h>
 #include <asm/guest-layout.h>
+#include <asm/mmio.h>
 #include <asm/p2m.h>
 #include <asm/vtimer.h>
 
@@ -101,6 +102,8 @@ struct arch_domain {
     const unsigned long *isa;
 
     struct vintc *vintc;
+
+    struct vmmio vmmio;
 };
 
 #include <xen/sched.h>
diff --git a/xen/arch/riscv/include/asm/mmio.h b/xen/arch/riscv/include/asm/mmio.h
new file mode 100644
index 000000000000..1bf48571a9f7
--- /dev/null
+++ b/xen/arch/riscv/include/asm/mmio.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef RISCV_MMIO_H
+#define RISCV_MMIO_H
+
+#include <xen/lib.h>
+#include <xen/rwlock.h>
+
+struct domain;
+struct vcpu;
+
+#define MAX_IO_HANDLER  16
+
+typedef struct {
+    paddr_t gpa;
+    unsigned int len;  /* Access width in bytes */
+    bool is_write;
+    /* Store: value to write; load: value read (set by handler) */
+    register_t data;
+} mmio_info_t;
+
+enum io_state
+{
+    IO_ABORT,       /* The IO was handled and led to an abort. */
+    IO_HANDLED,     /* The IO was successfully handled. */
+    IO_UNHANDLED,   /* No handler found for the IO. */
+};
+
+typedef enum io_state mmio_read_t(struct vcpu *v, mmio_info_t *info);
+typedef enum io_state mmio_write_t(struct vcpu *v, const mmio_info_t *info);
+
+struct mmio_handler_ops {
+    mmio_read_t *read;
+    mmio_write_t *write;
+};
+
+struct mmio_handler {
+    paddr_t addr;
+    paddr_t size;
+    const struct mmio_handler_ops *ops;
+};
+
+struct vmmio {
+    unsigned int num_entries;
+    rwlock_t lock;
+    struct mmio_handler handlers[MAX_IO_HANDLER];
+};
+
+int do_mmio(mmio_info_t *info, paddr_t fault_addr, unsigned int len);
+int register_mmio_handler(struct domain *d,
+                          const struct mmio_handler_ops *ops,
+                          paddr_t addr, paddr_t size);
+void domain_io_init(struct domain *d);
+
+#endif /* RISCV_MMIO_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/riscv/mmio.c b/xen/arch/riscv/mmio.c
new file mode 100644
index 000000000000..cf739a790afc
--- /dev/null
+++ b/xen/arch/riscv/mmio.c
@@ -0,0 +1,175 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#include <xen/bsearch.h>
+#include <xen/lib.h>
+#include <xen/rwlock.h>
+#include <xen/sched.h>
+#include <xen/string.h>
+
+#include <asm/current.h>
+#include <asm/mmio.h>
+
+/*
+ * bsearch() comparator: @key holds the address to look up in its addr field,
+ * @elem is an entry of vmmio->handlers. Relies on the regions not
+ * overlapping, which register_mmio_handler() enforces.
+ */
+static int cmp_mmio_handler(const void *key, const void *elem)
+{
+    const struct mmio_handler *handler0 = key;
+    const struct mmio_handler *handler1 = elem;
+
+    if ( handler0->addr < handler1->addr )
+        return -1;
+
+    if ( handler0->addr >= (handler1->addr + handler1->size) )
+        return 1;
+
+    return 0;
+}
+
+/*
+ * Return a copy of the matching handler rather than a pointer into
+ * vmmio->handlers: a concurrent register_mmio_handler() shifts entries
+ * up to keep the array sorted, so an escaped pointer could refer to a
+ * different (or torn) entry once the lock is dropped. The copy stays
+ * valid as the ops structures are never freed.
+ */
+static bool find_mmio_handler(struct domain *d, paddr_t gpa,
+                              struct mmio_handler *out)
+{
+    struct vmmio *vmmio = &d->arch.vmmio;
+    struct mmio_handler key = { .addr = gpa };
+    const struct mmio_handler *handler;
+
+    read_lock(&vmmio->lock);
+    handler = bsearch(&key, vmmio->handlers, vmmio->num_entries,
+                      sizeof(*handler), cmp_mmio_handler);
+    if ( handler )
+        *out = *handler;
+    read_unlock(&vmmio->lock);
+
+    return handler != NULL;
+}
+
+static enum io_state try_handle_mmio(mmio_info_t *info)
+{
+    struct vcpu *v = current;
+    struct mmio_handler handler = {};
+
+    if ( !find_mmio_handler(v->domain, info->gpa, &handler) )
+        return IO_UNHANDLED;
+
+    if ( info->is_write )
+        return handler.ops->write(v, info);
+    else
+        return handler.ops->read(v, info);
+}
+
+/*
+ * Check alignment and dispatch a decoded MMIO access to a registered
+ * handler. On success (0), info->data holds the read value for loads.
+ *
+ * There is no "retry" outcome to handle: find_mmio_handler() returns a
+ * copy of the matching handler taken under vmmio->lock and the ops
+ * structures are never freed, so the lookup result cannot go stale
+ * between finding the handler and invoking it.
+ */
+int do_mmio(mmio_info_t *info, paddr_t fault_addr, unsigned int len)
+{
+    /* Fault address should be aligned to length of MMIO */
+    if ( (len & (len - 1)) || (fault_addr & (len - 1)) )
+        return -EIO;
+
+    info->gpa = fault_addr;
+    info->len = len;
+
+    switch ( try_handle_mmio(info) )
+    {
+    case IO_HANDLED:
+        return 0;
+
+    case IO_ABORT:
+        return -EIO;
+
+    default:
+        return -EOPNOTSUPP;
+    }
+}
+
+int register_mmio_handler(struct domain *d,
+                          const struct mmio_handler_ops *ops,
+                          paddr_t addr, paddr_t size)
+{
+    struct vmmio *vmmio = &d->arch.vmmio;
+    struct mmio_handler *handlers = vmmio->handlers;
+    paddr_t end = addr + size;
+    unsigned int i;
+    int rc = 0;
+    bool overlap;
+
+    if ( !ops || !ops->read || !ops->write || end <= addr )
+        return -EINVAL;
+
+    write_lock(&vmmio->lock);
+
+    if ( vmmio->num_entries >= ARRAY_SIZE(vmmio->handlers) )
+    {
+        rc = -ENOSPC;
+        goto out;
+    }
+
+    /*
+     * The array is kept sorted by base address, so rather than appending and
+     * re-sorting, find the slot the new region belongs to and shift the tail
+     * up by one.
+     */
+    for ( i = vmmio->num_entries;
+          i > 0 && handlers[i - 1].addr > addr;
+          i-- )
+        /* Nothing to do */;
+
+    /*
+     * Regions are required not to overlap; check both neighbours. Their
+     * addr + size cannot overflow, as such regions are rejected above when
+     * they get registered.
+     */
+    overlap = (i > 0 && handlers[i - 1].addr + handlers[i - 1].size > addr) ||
+              (i < vmmio->num_entries && end > handlers[i].addr);
+
+    if ( overlap )
+    {
+        rc = -EADDRINUSE;
+        goto out;
+    }
+
+    memmove(&handlers[i + 1], &handlers[i],
+            (vmmio->num_entries - i) * sizeof(*handlers));
+
+    handlers[i] = (struct mmio_handler){
+        .addr = addr,
+        .size = size,
+        .ops = ops,
+    };
+
+    vmmio->num_entries++;
+
+ out:
+    write_unlock(&vmmio->lock);
+
+    return rc;
+}
+
+void domain_io_init(struct domain *d)
+{
+    rwlock_init(&d->arch.vmmio.lock);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.55.0

[PATCH v3 07/39] xen/riscv: introduce CPU_NONE

[Oleksii Kurochko]

Introduce CPU_NONE, a CPU number which doesn't identify any physical
CPU, to be used wherever a pCPU has to be recorded but there may be
none.

Its first user is vsfile_cpu in struct vimsic_state, which records the
pCPU whose h/w guest interrupt file backs the vCPU's IMSIC VS-file.
NR_CPUS has been used there as the value meaning that there is no such
pCPU, i.e. that the s/w VS-file is in use. NR_CPUS is however an upper
bound rather than a "no CPU" marker, and it already carries other
meanings: hartid_to_cpuid() returns it for a hart Xen doesn't know, and
pcpu_info[] uses it for an entry whose processor_id isn't valid yet.
Use CPU_NONE for vsfile_cpu instead, so that the s/w VS-file case can't
be mistaken for any of those.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
CPU_NONE is also used later in this series:
 - "xen/riscv: implement virtual APLIC MMIO emulation" checks vsfile_cpu
   against it to tell whether the target vCPU has a h/w guest interrupt
   file.
 - "xen/riscv: implement vCPU context switching" initialises the new
   arch_vcpu.last_cpu with it, and "xen/riscv: handle the case when no
   vCPU migration is needed" checks last_cpu against it.
 - "xen/riscv: remap interrupts to new IMSIC VS-file", "xen/riscv: save
   and clear the old IMSIC VS-file on vCPU migration" and "xen/riscv:
   introduce IMSIC h/w interrupt file attaching to vcpu" use it for
   vsfile_cpu of the s/w IMSIC VS-file.
---
Changes in v3:
 - New patch.
---
---
 xen/arch/riscv/imsic.c              | 4 ++--
 xen/arch/riscv/include/asm/config.h | 3 +++
 xen/arch/riscv/include/asm/imsic.h  | 2 +-
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 44b8640ea39a..402b2f14f591 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -357,9 +357,9 @@ int cf_check vcpu_imsic_init(struct vcpu *v)
     /*
      * xvzalloc() already cleared the context, so guest_file_id == 0, i.e. the
      * always-available s/w IMSIC VS-file. Only vsfile_cpu needs an explicit
-     * initializer as its s/w VS-file value is NR_CPUS rather than 0.
+     * initializer as its s/w VS-file value is CPU_NONE rather than 0.
      */
-    imsic_state->vsfile_cpu = NR_CPUS;
+    imsic_state->vsfile_cpu = CPU_NONE;
 
     v->arch.vimsic_state = imsic_state;
 
diff --git a/xen/arch/riscv/include/asm/config.h b/xen/arch/riscv/include/asm/config.h
index fd69057826e1..fb6203a7d502 100644
--- a/xen/arch/riscv/include/asm/config.h
+++ b/xen/arch/riscv/include/asm/config.h
@@ -136,6 +136,9 @@
 #define OPT_CONSOLE_STR "dtuart"
 #define INVALID_VCPU_ID MAX_VIRT_CPUS
 
+/* A CPU number which doesn't identify any physical CPU. */
+#define CPU_NONE (~0U)
+
 /* Linkage for RISCV */
 #ifdef __ASSEMBLER__
 #define CODE_FILL /* empty */
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index 2425430ed116..6baadc008d81 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -71,7 +71,7 @@ struct vimsic_state {
      */
     unsigned int guest_file_id;
     /*
-     * s/w IMSIC VS-file -> vsfile_cpu == NR_CPUS
+     * s/w IMSIC VS-file -> vsfile_cpu == CPU_NONE
      * h/w IMSIC VS-file -> vsfile_cpu < NR_CPUS
      */
     unsigned int vsfile_cpu;
-- 
2.55.0

[PATCH v3 08/39] xen/riscv: implement virtual APLIC MMIO emulation

[Oleksii Kurochko]

Guests running under Xen program interrupt routing by writing to APLIC
MMIO registers. Xen must intercept these accesses to enforce interrupt
isolation between domains and to translate guest routing intent into the
underlying physical MSI topology.

Register the vAPLIC range with the MMIO dispatch and emulate the accesses:
 - reads and writes are masked with the domain's authorised interrupt
   bitmap, so that a guest can neither observe nor affect interrupts it
   does not own;
 - target registers are shadowed per source, so that a guest reads back
   what it wrote. The h/w register instead gets the hart index and guest
   interrupt file of the pCPU the target vCPU's interrupt file lives on,
   as the APLIC uses these fields directly to compute the MSI address. As
   long as the target vCPU has no h/w guest interrupt file, only the
   shadow copy is updated;
 - only MSI delivery mode is emulated: target writes made while the
   guest's domaincfg.DM is clear are ignored.

Delegation (APLIC_SOURCECFG_D) is not yet supported.

Co-developed-by: Romain Caritey <Romain.Caritey@microchip.com>
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
- sourcecfg store emulation: mask the guest value with the new
  APLIC_SOURCECFG_WMASK (D and SM are the only implemented fields, all the
  other bits are reserved and read as zero) instead of rejecting the write
  when the value is above APLIC_SOURCECFG_SM_LEVEL_LOW. The dropped check
  compared the whole value against 7 rather than the SM field, so a legal
  SM with anything set in the reserved bits 9:3 was rejected. SM is WARL
  and vAPLIC keeps no shadow copy of sourcecfg[], so a reserved SM value
  is left to the h/w to normalize; add a TODO covering what will have to be
  emulated here once sourcecfg[] is shadowed.
- Introduce APLIC_SOURCECFG_WMASK in asm/aplic.h.
- target store emulation: reword the comment about a non-zero guest index
  to say that such a write is illegal and is therefore ignored as a whole,
  that the shadow copy keeps the zero it was allocated with, and that
  aplic_msi_target_gen() overwrites the field with vcpu_guest_file_id()
  anyway. Drop the now-redundant "Ignore such writes ..." comment.
- s/aplic_hart_field/aplic_hart_index/ (and the local variable with it):
  the value is the APLIC "Hart Index", not a field of it.
- Clarify the MSI target address documentation above aplic_hart_index():
  name the Guest Index in the address formula, say that the APLIC computes
  the MSI target address itself in MSI delivery mode, and label the two
  parts of the 14-bit Hart Index field as g and h to match the surrounding
  text.
- Don't add the MSI target address scheme diagram to asm/imsic.h: it
  would duplicate the one documented above aplic_hart_index().
- Fix the comment on APLIC_TARGET_IPRIO: the field exists when the target
  is in direct delivery mode (domaincfg.DM = 0), not "in DM mode".
- Rearrange the case blocks of vaplic_emulate_load() and
  vaplic_emulate_store() primarily by register offset: APLIC_DOMAINCFG
  comes first and APLIC_SOURCECFG handling directly follows it; the
  set/clr-num registers are also listed in offset order. No functional
  change.
- Rename the first parameter of aplic_msi_target_gen() from target_vcpu
  to v.
- Use a const-qualified pointer in aplic_hw_read_reg()'s cast to match
  readl()'s prototype.
- Put APLIC_xMSICFGADDR_PPN_HHX_MASK(), APLIC_xMSICFGADDR_PPN_LHX_MASK()
  and APLIC_xMSICFGADDR_PPN_LHX_SHIFT() on a single line each.
- Use gdprintk() instead of dprintk() for guest-triggered messages in
  vaplic.c and drop the redundant "is passed" from the word_idx message.
- Call domain_vaplic_deinit() in both failure paths of
  domain_vaplic_init() instead of open-coding the clean-up.
- aplic_msi_target_gen() takes the hart index from the pCPU the vCPU's
  interrupt file lives on (vsfile_cpu) instead of v->processor, which may
  already name the new pCPU while the vCPU is being migrated. Callers
  have to hold vsfile_lock until the value is written to the h/w, which
  the vAPLIC target write path now does.
- A target register write for a vCPU which has no h/w guest interrupt file
  yet (its vsfile_cpu is CPU_NONE) only updates the stored copy; the h/w
  register isn't programmed, as there is neither a hart nor a guest
  interrupt file to point it at. Add a TODO about s/w guest interrupt
  files, for which the h/w register will have to be written too.
- Turn vaplic_regs.target[] into vaplic_regs.sources[], an array of
  struct vaplic_source holding the guest's view of the target register
  together with a per-source lock which keeps the stored copy and the h/w
  target register updated as a pair. Document how many elements it holds
  and why index 0 is unused.
- Drop the emulation of target register writes in direct delivery mode:
  such writes are ignored with a one-time warning. Resolve the target
  vCPU only in the MSI delivery mode path, where it is used.
---
Changes in v2:
 - Shadow the guest-written target registers in a per-domain array and
   serve reads of APLIC_TARGET_* from it: the hardware register holds the
   value produced by aplic_msi_target_gen() (physical hart field, VS-file
   id), so reading it back would expose the host layout and return
   something the guest never wrote. A non-zero guest index is dropped with
   a one-time warning instead of being echoed back.
 - aplic_hart_field(): take a CPU id instead of a hartid and derive both
   the group and the hart index from msi->base_addr + msi->offset - the
   hart index bits are a part of that offset, so the hartid can't be used
   as the hart index. Add APLIC_xMSICFGADDR_PPN_LHX_{MASK,SHIFT} for that.
 - Document the IMSIC MSI target address layout and the APLIC hart index
   packing above aplic_hart_field(), and correct the corresponding diagram
   in asm/imsic.h.
 - Drop the mask parameter of aplic_hw_read_reg() and apply the
   authorization mask in vaplic_emulate_load() instead.
 - vaplic_emulate_{load,store}(): return bool instead of int, rename v/d to
   curr/currd and add ASSERT(curr == current).
 - Use domain_vcpu() instead of open-coded d->vcpu[] indexing when
   resolving the target hart index.
 - s/APLIC_REG_OFFSET_MASK/APLIC_CTRL_REGION_OFFSET_MASK/.
 - Update store emulation handling for target register to be able to deal
   with target format in both cases (MSI and Direct).
 - Update store emulation handling of domaincfg. There is no need to force
   DM/IE mode here (it will be forced/checked on Xen irq handler side).
 - Rename APLIC_DEFAULT_PRIORITY and move to aplic.h as default prioity
   value is used in vaplic code too now.
---
---
 xen/arch/riscv/aplic-priv.h         |   2 +
 xen/arch/riscv/aplic.c              | 138 +++++++++-
 xen/arch/riscv/include/asm/aplic.h  |  37 +++
 xen/arch/riscv/include/asm/vaplic.h |  19 ++
 xen/arch/riscv/vaplic.c             | 384 +++++++++++++++++++++++++++-
 5 files changed, 575 insertions(+), 5 deletions(-)

diff --git a/xen/arch/riscv/aplic-priv.h b/xen/arch/riscv/aplic-priv.h
index 35100d3a64fe..2245029b105d 100644
--- a/xen/arch/riscv/aplic-priv.h
+++ b/xen/arch/riscv/aplic-priv.h
@@ -47,4 +47,6 @@ struct aplic_priv {
  */
 extern unsigned int guest_aplic_num_sources;
 
+uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val);
+
 #endif /* ASM_RISCV_APLIC_PRIV_H */
diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index 17c96177a3b1..febc451760bf 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -16,6 +16,7 @@
 #include <xen/irq.h>
 #include <xen/mm.h>
 #include <xen/sections.h>
+#include <xen/sched.h>
 #include <xen/spinlock.h>
 #include <xen/types.h>
 #include <xen/vmap.h>
@@ -28,8 +29,6 @@
 #include <asm/io.h>
 #include <asm/riscv_encoding.h>
 
-#define APLIC_DEFAULT_PRIORITY  1
-
 static struct aplic_priv aplic = {
     .lock = SPIN_LOCK_UNLOCKED,
 };
@@ -38,6 +37,137 @@ static struct intc_info __ro_after_init aplic_info = {
     .hw_variant = INTC_APLIC,
 };
 
+/*
+ * The arrangement of IMSIC interrupt files in MMIO space follows a topology
+ * defined by the RISC-V AIA specification. An IMSIC group is a set of
+ * interrupt files (e.g., in a cluster or socket) co-located in memory.
+ *
+ * The physical address of an outgoing MSI is calculated by bitwise ORing a
+ * Base Physical Page Number (Base PPN) with the Group Index (g), the Hart
+ * Index (h) and, for a supervisor-level interrupt domain, the Guest Index:
+ *
+ *   ( Base PPN | (g << (HHXS + 12)) | (h << LHXS) | Guest Index ) << 12
+ *
+ * where Base PPN, HHXS, LHXS, HHXW and LHXW come from the {m,s}msiaddrcfg[h]
+ * registers of the interrupt domain that sends the MSI:
+ *
+ * XLEN-1       HHXS+24          LHXS+12          12          0
+ * |            |                |                |           |
+ * ------------------------------------------------------------
+ * |xxxx|   g   |xxxxxxxx|   h   |xxxx|Guest Index|     0     |
+ * ------------------------------------------------------------
+ *
+ * - g: group number.
+ * - h: hart number relative to the group.
+ * - xxxx: remaining Base PPN bits; each gap may be zero-width.
+ * - Guest Index: selects one of the 4 KiB pages right above the hart's own
+ *   supervisor-level file, i.e. it starts at bit 12; LHXS must therefore be
+ *   at least as large as the number of guest index bits.
+ * - Bits 11:0: always zero because IMSIC files are 4 KiB page-aligned.
+ *
+ * For wired interrupts in MSI delivery mode (domaincfg.DM = 1), the APLIC
+ * computes the MSI target address itself from the "Hart Index" field
+ * (bits 31:18) of the corresponding target[i] register. This 14-bit field
+ * holds both g and h:
+ *
+ * 13          lhxw+hhxw   lhxw       0
+ * |           |           |          |
+ * ------------------------------------
+ * |     0     |     g     |    h     |
+ * ------------------------------------
+ *
+ * - lhxw (Low Hart Index Width): the number of bits used for the hart number
+ *   within a group.
+ * - hhxw (High Hart Index Width): the number of bits used for the group
+ *   number; the remaining bits of the field must be zero.
+ *
+ * The Guest Index isn't a part of it: for a supervisor-level interrupt domain
+ * it has its own field (bits 17:12) in target[i].
+ *
+ * Because there are "xxxx" gaps (Base PPN bits) between the indices in the
+ * physical address (depending on HHXS and LHXS), software must extract the
+ * group and hart components separately and pack them into the APLIC-defined
+ * Hart Index format to ensure correct MSI targeting.
+ */
+static unsigned long aplic_hart_index(unsigned int cpu)
+{
+    const struct imsic_config *imsic = imsic_get_config();
+    const struct imsic_msi *msi = &imsic->msi[cpu];
+    /* Low Hart Index Shift */
+    unsigned int lhxs = imsic->guest_index_bits;
+    /* Low Hart Index Width */
+    unsigned int lhxw = imsic->hart_index_bits;
+    /* High Hart Index Width */
+    unsigned int hhxw = imsic->group_index_bits;
+    /* High Hart Index Shift */
+    unsigned int hhxs =
+        imsic->group_index_shift - APLIC_xMSICFGADDR_PPN_SHIFT * 2;
+    /*
+     * msi->base_addr is the base of the MMIO regset this CPU's interrupt
+     * files live in, and one regset can cover several harts; msi->offset
+     * selects this CPU's block inside it. The hart index bits are part of
+     * that offset, so both indexes have to be derived from the full address.
+     */
+    paddr_t target_addr = msi->base_addr + msi->offset;
+    unsigned long tppn = target_addr >> APLIC_xMSICFGADDR_PPN_SHIFT;
+    unsigned long g =
+        (tppn >> APLIC_xMSICFGADDR_PPN_HHX_SHIFT(hhxs)) &
+        APLIC_xMSICFGADDR_PPN_HHX_MASK(hhxw);
+    unsigned long h =
+        (tppn >> APLIC_xMSICFGADDR_PPN_LHX_SHIFT(lhxs)) &
+        APLIC_xMSICFGADDR_PPN_LHX_MASK(lhxw);
+
+    return (g << lhxw) | h;
+}
+
+/*
+ * v->processor can't be used here: during a migration it names the new pCPU
+ * before the interrupt file is moved there. The caller has to hold
+ * vsfile_lock until the result is written to the h/w.
+ */
+uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val)
+{
+    const struct vimsic_state *vimsic_state = v->arch.vimsic_state;
+    unsigned int guest_id = vcpu_guest_file_id(v);
+    unsigned long hart_index;
+
+    ASSERT(rw_is_locked(&vimsic_state->vsfile_lock));
+    ASSERT(vimsic_state->vsfile_cpu < NR_CPUS);
+
+    hart_index = aplic_hart_index(vimsic_state->vsfile_cpu);
+
+    base_val &= APLIC_TARGET_EIID;
+    base_val |= MASK_INSR(guest_id, APLIC_TARGET_GUEST_IDX);
+    base_val |= MASK_INSR(hart_index, APLIC_TARGET_HART_IDX);
+
+    return base_val;
+}
+
+uint32_t aplic_hw_read_reg(unsigned int offset)
+{
+    unsigned long flags;
+    uint32_t val;
+
+    ASSERT((offset < aplic.size) && IS_ALIGNED(offset, sizeof(uint32_t)));
+
+    spin_lock_irqsave(&aplic.lock, flags);
+    val = readl((const volatile void __iomem *)aplic.regs + offset);
+    spin_unlock_irqrestore(&aplic.lock, flags);
+
+    return val;
+}
+
+void aplic_hw_write_reg(unsigned int offset, uint32_t value)
+{
+    unsigned long flags;
+
+    ASSERT((offset < aplic.size) && IS_ALIGNED(offset, sizeof(uint32_t)));
+
+    spin_lock_irqsave(&aplic.lock, flags);
+    writel(value, (volatile void __iomem *)aplic.regs + offset);
+    spin_unlock_irqrestore(&aplic.lock, flags);
+}
+
 static void __init aplic_init_hw_interrupts(void)
 {
     unsigned int i;
@@ -53,9 +183,9 @@ static void __init aplic_init_hw_interrupts(void)
         /*
          * Low bits of target register contains Interrupt Priority bits which
          * can't be zero according to AIA spec.
-         * Thereby they are initialized to APLIC_DEFAULT_PRIORITY.
+         * Thereby they are initialized to APLIC_TARGET_IPRIO_DEFAULT.
          */
-        writel(APLIC_DEFAULT_PRIORITY, &aplic.regs->target[i]);
+        writel(APLIC_TARGET_IPRIO_DEFAULT, &aplic.regs->target[i]);
     }
 
     writel(APLIC_DOMAINCFG_IE | APLIC_DOMAINCFG_DM, &aplic.regs->domaincfg);
diff --git a/xen/arch/riscv/include/asm/aplic.h b/xen/arch/riscv/include/asm/aplic.h
index a2af55d54fc0..664bda2cb7eb 100644
--- a/xen/arch/riscv/include/asm/aplic.h
+++ b/xen/arch/riscv/include/asm/aplic.h
@@ -39,6 +39,13 @@
 #define  APLIC_DOMAINCFG_IE             BIT(8, U)
 #define  APLIC_DOMAINCFG_DM             BIT(2, U)
 #define  APLIC_DOMAINCFG_BE             BIT(0, U)
+/*
+ * The bits a write may change. Everything else, including the read-only zero
+ * bit 7 and the reserved bits, has to read back as zero, and BE is WARL and
+ * hardwired to 0 as Xen is little-endian only.
+ */
+#define  APLIC_DOMAINCFG_WMASK          (APLIC_DOMAINCFG_IE | \
+                                         APLIC_DOMAINCFG_DM)
 
 #define APLIC_SOURCECFG_BASE            0x0004
 #define APLIC_SOURCECFG_LAST            0x0ffc
@@ -58,6 +65,12 @@
 #define   APLIC_SOURCECFG_SM_EDGE_FALL  0x5
 #define   APLIC_SOURCECFG_SM_LEVEL_HIGH 0x6
 #define   APLIC_SOURCECFG_SM_LEVEL_LOW  0x7
+/*
+ * All other bits of sourcecfg[] are reserved and read as zero, so drop them
+ * on a write.
+ */
+#define  APLIC_SOURCECFG_WMASK          (APLIC_SOURCECFG_D | \
+                                         APLIC_SOURCECFG_SM)
 
 #define APLIC_SMSICFGADDR               0x1bc8
 #define APLIC_SMSICFGADDRH              0x1bcc
@@ -89,6 +102,9 @@
 #define  APLIC_TARGET_GUEST_IDX         GENMASK(17, 12)
 /* Bit 11 is reserved and reads as zero */
 #define  APLIC_TARGET_EIID              GENMASK(10, 0)
+/* If target is in direct delivery mode (domaincfg.DM = 0) */
+#define  APLIC_TARGET_IPRIO             GENMASK(7, 0)
+#define   APLIC_TARGET_IPRIO_DEFAULT    1U
 
 #define APLIC_IDC_SIZE                  32
 
@@ -98,6 +114,24 @@
 #define APLIC_SIZE(nr_cpus) \
     (APLIC_MIN_SIZE + APLIC_SIZE_ALIGN(APLIC_IDC_SIZE * (nr_cpus)))
 
+/*
+ * Using setip is fine here, as all SET* and CLR* register groups consist of 32
+ * registers and therefore have identical sizes.
+ *
+ * Lowest 2 bits are always zero for SET* and CLR* registers.
+ */
+#define APLIC_SETCLR_OFFSET_MASK \
+    (sizeof_field(struct aplic_regs, setip) - sizeof(uint32_t))
+
+#define APLIC_xMSICFGADDR_PPN_SHIFT IMSIC_MMIO_PAGE_SHIFT
+
+#define APLIC_xMSICFGADDR_PPN_HHX_MASK(hhxw) (BIT(hhxw, UL) - 1)
+#define APLIC_xMSICFGADDR_PPN_HHX_SHIFT(hhxs) \
+    ((hhxs) + APLIC_xMSICFGADDR_PPN_SHIFT)
+
+#define APLIC_xMSICFGADDR_PPN_LHX_MASK(lhxw) (BIT(lhxw, UL) - 1)
+#define APLIC_xMSICFGADDR_PPN_LHX_SHIFT(lhxs) (lhxs)
+
 struct aplic_regs {
     uint32_t domaincfg;         /* 0x0000 */
     uint32_t sourcecfg[1023];   /* 0x0004 */
@@ -141,4 +175,7 @@ struct aplic_regs {
     uint32_t target[1023];      /* 0x3004 */
 };
 
+uint32_t aplic_hw_read_reg(unsigned int offset);
+void aplic_hw_write_reg(unsigned int offset, uint32_t value);
+
 #endif /* ASM_RISCV_APLIC_H */
diff --git a/xen/arch/riscv/include/asm/vaplic.h b/xen/arch/riscv/include/asm/vaplic.h
index 96080bfbc23b..3a555742bbe0 100644
--- a/xen/arch/riscv/include/asm/vaplic.h
+++ b/xen/arch/riscv/include/asm/vaplic.h
@@ -11,6 +11,7 @@
 #define ASM__RISCV__VAPLIC_H
 
 #include <xen/kernel.h>
+#include <xen/spinlock.h>
 #include <xen/types.h>
 
 #include <asm/intc.h>
@@ -19,13 +20,31 @@ struct domain;
 
 #define to_vaplic(d) container_of((d)->arch.vintc, struct vaplic, vintc)
 
+/* State of one vAPLIC interrupt source. */
+struct vaplic_source {
+    /* Protects target together with the h/w target register of the source. */
+    spinlock_t lock;
+
+    /* Guest's view of the APLIC target register. */
+    uint32_t target;
+};
+
 struct vaplic_regs {
     uint32_t domaincfg;
+
+    /*
+     * Indexed by source number. The array holds d->arch.vintc->nr_virqs
+     * elements; sources[0] is unused as APLIC interrupt sources start from 1.
+     */
+    struct vaplic_source *sources;
 };
 
 struct vaplic {
     struct vintc vintc;
     struct vaplic_regs regs;
+
+    paddr_t regs_start;
+    unsigned int regs_size;
 };
 
 int domain_vaplic_init(struct domain *d);
diff --git a/xen/arch/riscv/vaplic.c b/xen/arch/riscv/vaplic.c
index 6fbdc07805fd..ea4b0dc88951 100644
--- a/xen/arch/riscv/vaplic.c
+++ b/xen/arch/riscv/vaplic.c
@@ -17,6 +17,7 @@
 #include <asm/aia.h>
 #include <asm/imsic.h>
 #include <asm/intc.h>
+#include <asm/mmio.h>
 #include <asm/vaplic.h>
 
 #include "aplic-priv.h"
@@ -27,6 +28,313 @@ unsigned int __ro_after_init guest_aplic_num_sources;
 
 #define FDT_VAPLIC_INT_CELLS 2
 
+#define AUTH_IRQ_BIT(d, irqn) \
+    (((irqn) < (d)->arch.vintc->nr_virqs) && \
+     test_bit(irqn, (d)->arch.vintc->used_irqs))
+
+/*
+ * Convert a byte offset (within a SETIP/CLRIP/SETIE/CLRIE register group) to
+ * a 32-bit word index into the used_irqs bitmap. Each word covers 32
+ * interrupt sources. For SOURCECFG and TARGET groups the same division also
+ * yields the interrupt number directly, because those arrays store one 32-bit
+ * register per source.
+ */
+#define regoffset_to_word_idx(reg_val) ((reg_val) / sizeof(uint32_t))
+
+static uint32_t vaplic_target_read(const struct domain *d, unsigned int irqn)
+{
+    const struct vaplic *vaplic = to_vaplic(d);
+
+    /* target[0] doesn't exist so irqn == 0 should be impossible */
+    if ( !irqn || irqn >= vaplic->vintc.nr_virqs )
+        return 0;
+
+    return read_atomic(&vaplic->regs.sources[irqn].target);
+}
+
+static inline uint32_t generate_auth_mask(const struct domain *currd,
+                                          unsigned int word_idx)
+{
+    unsigned int first_bit = word_idx * sizeof(uint32_t) * BITS_PER_BYTE;
+
+    if ( word_idx >= DIV_ROUND_UP(currd->arch.vintc->nr_virqs,
+                                  sizeof(uint32_t) * BITS_PER_BYTE) )
+    {
+        gdprintk(XENLOG_DEBUG, "incorrect word_idx(%u)\n", word_idx);
+
+        return 0;
+    }
+
+    return currd->arch.vintc->used_irqs[first_bit / BITS_PER_LONG] >>
+           (first_bit % BITS_PER_LONG);
+}
+
+static bool vaplic_emulate_load(const struct vcpu *curr, paddr_t addr,
+                                uint32_t *out)
+{
+    const struct domain *currd = curr->domain;
+    const struct vaplic *vaplic = to_vaplic(currd);
+    const unsigned int offset = addr & APLIC_CTRL_REGION_OFFSET_MASK;
+    uint32_t auth_mask;
+    unsigned int i;
+
+    ASSERT(curr == current);
+
+    switch ( offset )
+    {
+    case APLIC_DOMAINCFG:
+        *out = vaplic->regs.domaincfg;
+
+        return true;
+
+    case APLIC_SETIP_BASE ... APLIC_SETIP_LAST:
+    case APLIC_CLRIP_BASE ... APLIC_CLRIP_LAST:
+    case APLIC_SETIE_BASE ... APLIC_SETIE_LAST:
+        i = regoffset_to_word_idx(offset & APLIC_SETCLR_OFFSET_MASK);
+        auth_mask = generate_auth_mask(currd, i);
+
+        break;
+
+    case APLIC_SETIPNUM:
+    case APLIC_CLRIPNUM:
+    case APLIC_SETIENUM:
+    case APLIC_CLRIE_BASE ... APLIC_CLRIE_LAST:
+    case APLIC_CLRIENUM:
+    case APLIC_SETIPNUM_LE:
+        /*
+         * Based on the RISC-V AIA spec a read of these registers
+         * always returns zero
+         */
+        *out = 0;
+
+        return true;
+
+    case APLIC_TARGET_BASE ... APLIC_TARGET_LAST:
+        /*
+         * As target registers start from 1:
+         *  0x3000 genmsi
+         *  0x3004 target[1]
+         *  0x3008 target[2]
+         *   ...
+         *  0x3FFC target[1023]
+         * It is necessary to calculate an interrupt number by subtracting
+         * APLIC_GENMSI instead of APLIC_TARGET_BASE.
+         */
+        i = regoffset_to_word_idx(offset - APLIC_GENMSI);
+
+        *out = AUTH_IRQ_BIT(currd, i) ? vaplic_target_read(currd, i) : 0;
+
+        return true;
+
+    default:
+        gdprintk(XENLOG_WARNING, "Unhandled APLIC read at offset %#x\n",
+                 offset);
+
+        return false;
+    }
+
+    *out = aplic_hw_read_reg(offset) & auth_mask;
+
+    return true;
+}
+
+static bool vaplic_emulate_store(const struct vcpu *curr, paddr_t addr,
+                                 uint32_t value)
+{
+    const struct domain *currd = curr->domain;
+    unsigned int offset = addr & APLIC_CTRL_REGION_OFFSET_MASK;
+
+    ASSERT(curr == current);
+
+    switch ( offset )
+    {
+    case APLIC_DOMAINCFG:
+    {
+        struct vaplic *vaplic = to_vaplic(currd);
+
+        vaplic->regs.domaincfg = APLIC_DOMAINCFG_RO |
+                                 (value & APLIC_DOMAINCFG_WMASK);
+
+        return true;
+    }
+
+    case APLIC_SOURCECFG_BASE ... APLIC_SOURCECFG_LAST:
+        /*
+         * Only D (bit 10) and SM (bits 2:0) are implemented, the rest of the
+         * bits are reserved and read as zero, so ignore what a guest writes
+         * to them.
+         */
+        value &= APLIC_SOURCECFG_WMASK;
+
+        if ( value & APLIC_SOURCECFG_D )
+        {
+            gdprintk(XENLOG_ERR, "APLIC_SOURCECFG_D isn't supported\n");
+
+            goto fail;
+        }
+
+        /*
+         * SM is WARL, so the reserved values 0x2 and 0x3 need no handling
+         * here: vAPLIC keeps no shadow copy of sourcecfg[], the value is
+         * written straight to the h/w register and is read back from it, so
+         * it is the h/w which substitutes a legal value for an illegal one.
+         *
+         * TODO: when vAPLIC starts to shadow sourcecfg[], the WARL behaviour
+         * will have to be emulated here instead, e.g.:
+         *   if ( value == 0x2 || value == 0x3 )
+         *       value = APLIC_SOURCECFG_SM_INACTIVE;
+         */
+
+        /*
+         * As sourcecfg register starts from 1:
+         *   0x0000 domaincfg
+         *   0x0004 sourcecfg[1]
+         *   0x0008 sourcecfg[2]
+         *    ...
+         *   0x0FFC sourcecfg[1023]
+         * It is necessary to calculate an interrupt number by subtracting
+         * APLIC_DOMAINCFG instead of APLIC_SOURCECFG_BASE.
+         */
+        if ( !AUTH_IRQ_BIT(currd,
+                           regoffset_to_word_idx(offset - APLIC_DOMAINCFG)) )
+            /* Interrupt not enabled, ignore it */
+            return true;
+
+        break;
+
+    case APLIC_SETIP_BASE ... APLIC_SETIP_LAST:
+    case APLIC_CLRIP_BASE ... APLIC_CLRIP_LAST:
+    case APLIC_SETIE_BASE ... APLIC_SETIE_LAST:
+    case APLIC_CLRIE_BASE ... APLIC_CLRIE_LAST:
+    {
+        unsigned int word_idx =
+            regoffset_to_word_idx(offset & APLIC_SETCLR_OFFSET_MASK);
+
+        value &= generate_auth_mask(currd, word_idx);
+
+        break;
+    }
+
+    case APLIC_SETIPNUM:
+    case APLIC_CLRIPNUM:
+    case APLIC_SETIENUM:
+    case APLIC_CLRIENUM:
+    case APLIC_SETIPNUM_LE:
+        if ( !value || !AUTH_IRQ_BIT(currd, value) )
+            return true;
+
+        break;
+
+    case APLIC_TARGET_BASE ... APLIC_TARGET_LAST:
+    {
+        struct vaplic *vaplic = to_vaplic(currd);
+        /*
+         * Look at vaplic_emulate_load() for explanation why APLIC_GENMSI is
+         * subtracted.
+         */
+        unsigned int srcn = regoffset_to_word_idx(offset - APLIC_GENMSI);
+
+        if ( !AUTH_IRQ_BIT(currd, srcn) )
+            /* Interrupt not enabled, ignore it */
+            return true;
+
+        if ( vaplic->regs.domaincfg & APLIC_DOMAINCFG_DM )
+        {
+            struct vaplic_source *src = &vaplic->regs.sources[srcn];
+            struct vcpu *target_vcpu;
+            unsigned int guest_hart_idx =
+                MASK_EXTR(value, APLIC_TARGET_HART_IDX);
+            unsigned long flags;
+            struct vimsic_state *vimsic_state;
+
+            target_vcpu = domain_vcpu(currd, guest_hart_idx);
+
+            if ( !target_vcpu )
+            {
+                gdprintk(XENLOG_ERR, "Invalid vCPU id in target register\n");
+
+                /* Ignore such writings */
+                return true;
+            }
+
+            vimsic_state = target_vcpu->arch.vimsic_state;
+
+            /*
+             * A non-zero guest index asks for delivery to an interrupt file
+             * of a nested guest. The vIMSIC node has no riscv,guest-index-bits
+             * property, so a guest is told its harts have no guest interrupt
+             * files and the field reads as zero for them. Such a write is
+             * illegal and is therefore ignored as a whole: the stored copy
+             * keeps the zero it was allocated with, so the field never needs
+             * to be masked out here.
+             * What ends up in the h/w register is Xen's own value anyway:
+             * aplic_msi_target_gen() overwrites the field with
+             * vcpu_guest_file_id() of the target vCPU.
+             */
+            if ( MASK_EXTR(value, APLIC_TARGET_GUEST_IDX) )
+            {
+                printk_once(XENLOG_WARNING
+                            "%pd: vAPLIC target guest index != 0 is unsupported\n",
+                            currd);
+
+                return true;
+            }
+
+            /*
+             * src->lock keeps the stored copy and the h/w register updated
+             * as a pair.
+             */
+            spin_lock_irqsave(&src->lock, flags);
+
+            write_atomic(&src->target, value);
+
+            /* IRQs are already off under src->lock. */
+            read_lock(&vimsic_state->vsfile_lock);
+
+            /*
+             * The h/w register names the hart and the guest interrupt file
+             * MSIs are sent to, and there is none of either until the target
+             * vCPU is given a h/w guest interrupt file. The stored copy
+             * reaches the h/w at that point.
+             *
+             * TODO: once s/w guest interrupt files are supported, a vCPU using
+             * one still needs its MSIs, so the h/w register will have to be
+             * written for it too, pointing at an interrupt file Xen receives
+             * them in on the vCPU's behalf.
+             */
+            if ( vimsic_state->vsfile_cpu != CPU_NONE )
+                aplic_hw_write_reg(offset,
+                                   aplic_msi_target_gen(target_vcpu, value));
+
+            read_unlock(&vimsic_state->vsfile_lock);
+
+            spin_unlock_irqrestore(&src->lock, flags);
+
+            return true;
+        }
+
+        /* printk_once(), as a guest could otherwise flood the log. */
+        printk_once(XENLOG_WARNING
+                    "%pd: vAPLIC direct delivery mode isn't supported\n",
+                    currd);
+
+        return true;
+    }
+
+    default:
+    fail:
+        gdprintk(XENLOG_WARNING,
+                 "Unhandled APLIC write at offset %#x (value %#x)\n", offset,
+                 value);
+
+        return false;
+    }
+
+    aplic_hw_write_reg(offset, value);
+
+    return true;
+}
+
 static int __init cf_check vaplic_make_domu_dt_node(struct kernel_info *kinfo)
 {
     struct domain *d = kinfo->bd.d;
@@ -95,6 +403,56 @@ static const struct vintc_init_ops __initconstrel init_ops = {
     .make_domu_dt_node = vaplic_make_domu_dt_node,
 };
 
+static enum io_state cf_check vaplic_mmio_read(struct vcpu *v,
+                                               mmio_info_t *info)
+{
+    uint32_t val;
+
+    ASSERT(v == current);
+
+    if ( info->len != sizeof(uint32_t) ||
+         !IS_ALIGNED(info->gpa, sizeof(uint32_t)) )
+    {
+        gdprintk(XENLOG_DEBUG,
+                 "VAPLIC: unaligned/wrong-width read gpa=%"PRIpaddr" len=%u\n",
+                 info->gpa, info->len);
+        return IO_ABORT;
+    }
+
+    if ( !vaplic_emulate_load(v, info->gpa, &val) )
+        return IO_ABORT;
+
+    /* APLIC registers are 32-bit; zero-extend to the guest register width. */
+    info->data = val;
+
+    return IO_HANDLED;
+}
+
+static enum io_state cf_check vaplic_mmio_write(struct vcpu *v,
+                                                const mmio_info_t *info)
+{
+    ASSERT(v == current);
+
+    if ( info->len != sizeof(uint32_t) ||
+         !IS_ALIGNED(info->gpa, sizeof(uint32_t)) )
+    {
+        gdprintk(XENLOG_DEBUG,
+                 "VAPLIC: unaligned/wrong-width write gpa=%"PRIpaddr" len=%u\n",
+                 info->gpa, info->len);
+        return IO_ABORT;
+    }
+
+    if ( !vaplic_emulate_store(v, info->gpa, info->data) )
+        return IO_ABORT;
+
+    return IO_HANDLED;
+}
+
+static const struct mmio_handler_ops vaplic_mmio_ops = {
+    .read  = vaplic_mmio_read,
+    .write = vaplic_mmio_write,
+};
+
 static const struct vintc_ops vintc_ops = {
     .vcpu_init = vcpu_imsic_init,
     .vcpu_deinit = vcpu_imsic_deinit,
@@ -103,6 +461,8 @@ static const struct vintc_ops vintc_ops = {
 int domain_vaplic_init(struct domain *d)
 {
     struct vaplic *vaplic = xvzalloc(struct vaplic);
+    unsigned int i;
+    int rc;
 
     if ( !vaplic )
         return -ENOMEM;
@@ -123,7 +483,28 @@ int domain_vaplic_init(struct domain *d)
      */
     d->arch.vintc->nr_virqs = guest_aplic_num_sources + 1;
 
-    return 0;
+    /* Slot 0 is unused: APLIC source numbering starts at 1 (see used_irqs). */
+    vaplic->regs.sources = xvzalloc_array(struct vaplic_source,
+                                          d->arch.vintc->nr_virqs);
+    if ( !vaplic->regs.sources )
+    {
+        domain_vaplic_deinit(d);
+
+        return -ENOMEM;
+    }
+
+    for ( i = 0; i < d->arch.vintc->nr_virqs; i++ )
+        spin_lock_init(&vaplic->regs.sources[i].lock);
+
+    vaplic->regs_start = GUEST_APLIC_S_BASE;
+    vaplic->regs_size = APLIC_SIZE(d->max_vcpus);
+
+    rc = register_mmio_handler(d, &vaplic_mmio_ops,
+                               vaplic->regs_start, vaplic->regs_size);
+    if ( rc )
+        domain_vaplic_deinit(d);
+
+    return rc;
 }
 
 void domain_vaplic_deinit(struct domain *d)
@@ -135,5 +516,6 @@ void domain_vaplic_deinit(struct domain *d)
 
     vaplic = to_vaplic(d);
     d->arch.vintc = NULL;
+    xvfree(vaplic->regs.sources);
     xvfree(vaplic);
 }
-- 
2.55.0

[PATCH v3 09/39] xen/riscv: build the target hart index via aplic_hart_index()

[Oleksii Kurochko]

aplic_set_irq_affinity() open-coded the packing of the group and hart
indices into the target register, and got two things wrong along the
way:

 - imsic_config.msi[] is indexed by logical CPU id, but the index was
   run through cpuid_to_hartid() first. On any platform where the two
   spaces differ this picks another CPU's interrupt file, or reads past
   the array;

 - the same hart id was then used verbatim as the low hart index, and
   the group index was derived from msi[].base_addr alone. The hart
   index bits live in msi[].offset, the base address only covers the
   MMIO regset, which may hold the files of several harts. Both indices
   have to come out of base_addr + offset.

aplic_hart_index() already extracts them that way, and is what the vAPLIC
target path uses, so call it here as well and insert the result with
MASK_INSR(APLIC_TARGET_HART_IDX) instead of a bare shift, which keeps the
value from spilling out of the 14-bit field. This drops the last in-tree
duplicate of the AIA hart index formula, and with it the last user of
APLIC_TARGET_HART_IDX_SHIFT, which is removed.

desc->irq is used as the EIID as is. Assert that it is a valid APLIC
source, which also guarantees that it fits the EIID field.

No functional change on a single-group platform whose hart ids match
their CPU ids and whose IMSIC regset holds one file per hart.

Fixes: d4676a1398bc ("xen/riscv: implementation of aplic and imsic operations")
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - s/aplic_hart_field/aplic_hart_index/ (in the subject too) following the
   rename of the helper in "xen/riscv: implement virtual APLIC MMIO
   emulation".
 - Don't mask desc->irq with APLIC_TARGET_EIID. Instead, bound it by
   the APLIC source range rather than NR_IRQS, which will also have to
   cover device MSIs: add a BUILD_BUG_ON() that ARRAY_SIZE(target) fits
   the EIID field and an ASSERT() that desc->irq is a valid APLIC source
   (1..num_irqs), which also covers the target[desc->irq - 1] index.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aplic.c             | 33 ++++++++++--------------------
 xen/arch/riscv/include/asm/aplic.h |  1 -
 2 files changed, 11 insertions(+), 23 deletions(-)

diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index febc451760bf..62bbcc9afacf 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -335,9 +335,7 @@ static unsigned int aplic_get_cpu_from_mask(const cpumask_t *cpumask)
 static void cf_check aplic_set_irq_affinity(struct irq_desc *desc, const cpumask_t *mask)
 {
     unsigned int cpu;
-    uint64_t group_index, base_ppn;
-    uint32_t hhxw, lhxw, hhxs, value;
-    const struct imsic_config *imsic = aplic.imsic_cfg;
+    uint32_t value;
 
     /*
      * TODO: Currently, APLIC is supported only with MSI interrupts.
@@ -350,27 +348,18 @@ static void cf_check aplic_set_irq_affinity(struct irq_desc *desc, const cpumask
 
     ASSERT(spin_is_locked(&desc->lock));
 
-    cpu = cpuid_to_hartid(aplic_get_cpu_from_mask(mask));
-    hhxw = imsic->group_index_bits;
-    lhxw = imsic->hart_index_bits;
+    cpu = aplic_get_cpu_from_mask(mask);
+
     /*
-     * Although this variable is used only once in the calculation of
-     * group_index, and it might seem that hhxs could be defined as:
-     *   hhxs = imsic->group_index_shift - IMSIC_MMIO_PAGE_SHIFT;
-     * and then the addition of IMSIC_MMIO_PAGE_SHIFT could be omitted
-     * when calculating the group index.
-     * It was done intentionally this way to follow the formula from
-     * the AIA specification for calculating the MSI address.
+     * desc->irq is used as the EIID as-is, which is only valid for APLIC
+     * sources: they are numbered 1..num_irqs, with at most ARRAY_SIZE(target).
      */
-    hhxs = imsic->group_index_shift - IMSIC_MMIO_PAGE_SHIFT * 2;
-    base_ppn = imsic->msi[cpu].base_addr >> IMSIC_MMIO_PAGE_SHIFT;
-
-    /* Update hart and EEID in the target register */
-    group_index = (base_ppn >> (hhxs + IMSIC_MMIO_PAGE_SHIFT)) &
-                  (BIT(hhxw, UL) - 1);
-    value = desc->irq;
-    value |= cpu << APLIC_TARGET_HART_IDX_SHIFT;
-    value |= group_index << (lhxw + APLIC_TARGET_HART_IDX_SHIFT);
+    BUILD_BUG_ON(ARRAY_SIZE(aplic.regs->target) >
+                 MASK_EXTR(~0U, APLIC_TARGET_EIID));
+    ASSERT(desc->irq && desc->irq <= aplic_info.num_irqs);
+
+    value = MASK_INSR(aplic_hart_index(cpu), APLIC_TARGET_HART_IDX) |
+            desc->irq;
 
     spin_lock(&aplic.lock);
 
diff --git a/xen/arch/riscv/include/asm/aplic.h b/xen/arch/riscv/include/asm/aplic.h
index 664bda2cb7eb..9018936ca95a 100644
--- a/xen/arch/riscv/include/asm/aplic.h
+++ b/xen/arch/riscv/include/asm/aplic.h
@@ -98,7 +98,6 @@
 #define APLIC_TARGET_BASE               0x3004
 #define APLIC_TARGET_LAST               0x3ffc
 #define  APLIC_TARGET_HART_IDX          GENMASK(31, 18)
-#define  APLIC_TARGET_HART_IDX_SHIFT    18
 #define  APLIC_TARGET_GUEST_IDX         GENMASK(17, 12)
 /* Bit 11 is reserved and reads as zero */
 #define  APLIC_TARGET_EIID              GENMASK(10, 0)
-- 
2.55.0

[PATCH v3 10/39] xen/riscv: add helper to check APLIC MSI mode

[Oleksii Kurochko]

Use convient helper instead of open-coding the things.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes in v3:
 - Add Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>.
 - Add Acked-by: Jan Beulich <jbeulich@suse.com>.
---
Changes in v2:
 - Rename helper.
 - Update the commit message.
---
---
 xen/arch/riscv/aplic.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index 62bbcc9afacf..60f5f239c394 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -37,6 +37,11 @@ static struct intc_info __ro_after_init aplic_info = {
     .hw_variant = INTC_APLIC,
 };
 
+static bool aplic_msi_mode(void)
+{
+    return readl(&aplic.regs->domaincfg) & APLIC_DOMAINCFG_DM;
+}
+
 /*
  * The arrangement of IMSIC interrupt files in MMIO space follows a topology
  * defined by the RISC-V AIA specification. An IMSIC group is a set of
@@ -260,7 +265,7 @@ static void cf_check aplic_irq_enable(struct irq_desc *desc)
      *       If APLIC without MSI interrupts is required in the future,
      *       this function will need to be updated accordingly.
      */
-    ASSERT(readl(&aplic.regs->domaincfg) & APLIC_DOMAINCFG_DM);
+    ASSERT(aplic_msi_mode());
 
     ASSERT(spin_is_locked(&desc->lock));
 
@@ -291,7 +296,7 @@ static void cf_check aplic_irq_disable(struct irq_desc *desc)
      *       If APLIC without MSI interrupts is required in the future,
      *       this function will need to be updated accordingly.
      */
-    ASSERT(readl(&aplic.regs->domaincfg) & APLIC_DOMAINCFG_DM);
+    ASSERT(aplic_msi_mode());
 
     ASSERT(spin_is_locked(&desc->lock));
 
-- 
2.55.0

[PATCH v3 11/39] xen/riscv: implement vCPU context switching

[Oleksii Kurochko]

Implement context_switch() and the helpers it needs: save/restore of
H/VS CSRs, virtual timer and P2M context, and __context_switch() in
assembly, which switches Xen's own callee-saved state (and thereby the
stack) from prev to next. The virtual interrupt controller state isn't
switched here.

Add offsets of struct arch_vcpu's xen_saved_context to asm-offsets.c for
use by __context_switch().

henvcfg, htimedelta and vsie are 64-bit on both RV32 and RV64, so store
them as uint64_t and access them with csr_{read,write}64(). For that
purpose introduce csr_read64().

The VMID has to be claimed on the context switch rather than on guest
entry: once p2m_ctxt_switch_to() writes HGATP, speculation can populate
G-stage entries under whatever VMID is written there. This is done by
p2m_vmid_switch_to(), called right before the HGATP write:

 - VMIDs are a per-hart resource, so the (generation, vmid) pair of a
   vCPU which last ran on another hart is invalidated before a VMID is
   claimed for it. The local flush for a wrapped generation moves along
   with the claim.
 - Nothing is switched in for the idle vCPU, so HGATP keeps pointing at
   the p2m of the domain which ran on the hart last. Track that domain
   per hart (hgatp_owner) and keep the hart in its dirty_cpumask until a
   vCPU of another domain is switched in: p2m_tlb_flush() then goes on
   reaching the hart, and a domain which idles between two runs on the
   same hart keeps its VMIDs.
 - When the owner changes, the hart leaves the old owner's dirty_cpumask
   while its TLB may still hold that domain's G-stage translations, so it
   is moved to a new VMID generation, which makes none of them reachable
   again. It joins the new owner's dirty_cpumask before HGATP is written;
   a full barrier there, paired with one in p2m_tlb_flush(), guarantees
   that a concurrent P2M change either reaches the hart with an
   HFENCE.GVMA or is visible to it before it walks the p2m.

That leaves p2m_handle_vmenter() with nothing to do, so drop it together
with its call from check_for_pcpu_work(): a VMID is only ever invalidated
while its vCPU isn't running, and p2m_tlb_flush() drops stale entries
with a remote HFENCE.GVMA rather than by retiring VMIDs. Unlike
p2m_handle_vmenter(), HGATP is written unconditionally, as it holds the
G-stage root too, which on a context switch belongs to another domain.

While at it, fix the inclusion order of headers in asm-offsets.c: Xen's
headers go first, then arch specific ones.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Introduce csr_read64(), as vsie is 64-bit on RV32 too, i.e. has to be
   accessed as the vsie/vsieh pair there.
 - Make vsie declared as uint64_t instead of register_t.
 - Rename the parameter of save_csr_regs() to p and of restore_csr_regs()
   to n, to match their callers ctxt_switch_from()/ctxt_switch_to() and the
   rest of the context switch helpers (p2m_ctxt_switch_{from,to}(),
   vtimer_ctxt_switch_{from,to}()).
 - s/save_csr_regs/csr_regs_ctxt_switch_from/
 - s/restore_csr_regs/csr_regs_ctxt_switch_to/
 - Drop the forward declaration of struct vcpu in <asm/system.h>: the
   return type of __context_switch() already declares the tag at file
   scope.
 - Make the next parameter of __context_switch() pointer-to-const, as
   next's saved context is only read.
 - Extend the comment above __context_switch(): as ra is switched too, it
   doesn't return to its caller, but to where next last called it from
   (next's own context_switch(), or continue_new_vcpu() for a vCPU which
   has never run).
 - Call vtimer_ctxt_switch_to() after csr_regs_ctxt_switch_to(), so that
   ctxt_switch_to() restores state in the reverse order of
   ctxt_switch_from() saving it.
 - Rework p2m's VMID handling in the context switch: track the domain
   whose p2m HGATP points at in a per-CPU hgatp_owner, so a pass through
   idle no longer drops the hart from that domain's dirty_cpumask nor
   burns a VMID generation; move the dirty_cpumask update and the
   invalidation of a VMID brought from another hart from schedule_tail()
   and ctxt_switch_to() to p2m_ctxt_switch_to(), next to the VMID claim;
   add the barriers ordering the dirty_cpumask update against
   p2m_tlb_flush().
 - Initialise arch_vcpu.last_cpu to CPU_NONE instead of NR_CPUS.
 - Mark prev's dirty_cpu clean before ctxt_switch_to() and set current's
   after it.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/domain.c              | 127 ++++++++++++++++++++++
 xen/arch/riscv/entry.S               |  48 ++++++++
 xen/arch/riscv/include/asm/csr.h     |  13 +++
 xen/arch/riscv/include/asm/domain.h  |  15 ++-
 xen/arch/riscv/include/asm/p2m.h     |   1 -
 xen/arch/riscv/include/asm/system.h  |   2 +
 xen/arch/riscv/p2m.c                 | 157 ++++++++++++++++++---------
 xen/arch/riscv/riscv64/asm-offsets.c |  19 +++-
 xen/arch/riscv/stubs.c               |   5 -
 xen/arch/riscv/traps.c               |   2 -
 10 files changed, 330 insertions(+), 59 deletions(-)

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index fd6229635627..f80643e2d82b 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -11,6 +11,7 @@
 #include <asm/bitops.h>
 #include <asm/cpufeature.h>
 #include <asm/csr.h>
+#include <asm/current.h>
 #include <asm/intc.h>
 #include <asm/mmio.h>
 #include <asm/riscv_encoding.h>
@@ -152,6 +153,8 @@ int arch_vcpu_create(struct vcpu *v)
     if ( is_idle_vcpu(v) )
         return 0;
 
+    v->arch.last_cpu = CPU_NONE;
+
     vcpu_csr_init(v);
 
     if ( (rc = vcpu_vtimer_init(v)) )
@@ -323,6 +326,130 @@ int arch_domain_create(struct domain *d,
     return rc;
 }
 
+static void csr_regs_ctxt_switch_from(struct vcpu *p)
+{
+    /*
+     * There is no need to save these CSRs as only hypervisor writes them in
+     * csr_regs_ctxt_switch_to() and guest can't access them so they shouldn't
+     * be stored here. Keep them commented here just for symmetry with the
+     * restore CSRs register part.
+     *
+     * p->arch.hedeleg = csr_read(CSR_HEDELEG);
+     * p->arch.hideleg = csr_read(CSR_HIDELEG);
+     * p->arch.henvcfg = csr_read64(CSR_HENVCFG);
+     * p->arch.hcounteren = csr_read(CSR_HCOUNTEREN);
+     * p->arch.htimedelta = csr_read64(CSR_HTIMEDELTA);
+     *
+     * if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
+     *     p->arch.hstateen0 = csr_read(CSR_HSTATEEN0);
+     */
+
+    p->arch.hvip = csr_read(CSR_HVIP);
+
+    p->arch.vsstatus = csr_read(CSR_VSSTATUS);
+    p->arch.vsie = csr_read64(CSR_VSIE);
+    p->arch.vstvec = csr_read(CSR_VSTVEC);
+    p->arch.vsscratch = csr_read(CSR_VSSCRATCH);
+    p->arch.vscause = csr_read(CSR_VSCAUSE);
+    p->arch.vstval = csr_read(CSR_VSTVAL);
+    p->arch.vsepc = csr_read(CSR_VSEPC);
+}
+
+static void csr_regs_ctxt_switch_to(struct vcpu *n)
+{
+    csr_write(CSR_HEDELEG, n->arch.hedeleg);
+    csr_write(CSR_HIDELEG, n->arch.hideleg);
+    csr_write(CSR_HVIP, n->arch.hvip);
+    csr_write64(CSR_HENVCFG, n->arch.henvcfg);
+    csr_write(CSR_HCOUNTEREN, n->arch.hcounteren);
+    csr_write64(CSR_HTIMEDELTA, n->arch.htimedelta);
+
+    if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
+        csr_write(CSR_HSTATEEN0, n->arch.hstateen0);
+
+    csr_write(CSR_VSSTATUS, n->arch.vsstatus);
+    csr_write64(CSR_VSIE, n->arch.vsie);
+    csr_write(CSR_VSTVEC, n->arch.vstvec);
+    csr_write(CSR_VSSCRATCH, n->arch.vsscratch);
+    csr_write(CSR_VSCAUSE, n->arch.vscause);
+    csr_write(CSR_VSTVAL, n->arch.vstval);
+    csr_write(CSR_VSEPC, n->arch.vsepc);
+}
+
+static void ctxt_switch_from(struct vcpu *p)
+{
+    /*
+     * When the idle VCPU is running, Xen will always stay in hypervisor
+     * mode.
+     * Therefore we don't need to save the context of an idle VCPU.
+     */
+    if ( is_idle_vcpu(p) )
+        return;
+
+    p2m_ctxt_switch_from(p);
+
+    vtimer_ctxt_switch_from(p);
+
+    csr_regs_ctxt_switch_from(p);
+}
+
+static void ctxt_switch_to(struct vcpu *n)
+{
+    /*
+     * When the idle VCPU is running, Xen will always stay in hypervisor
+     * mode.
+     * Therefore we don't need to restore the context of an idle VCPU.
+     */
+    if ( is_idle_vcpu(n) )
+        return;
+
+    csr_regs_ctxt_switch_to(n);
+
+    vtimer_ctxt_switch_to(n);
+
+    p2m_ctxt_switch_to(n);
+}
+
+static void schedule_tail(struct vcpu *prev)
+{
+    unsigned int cpu = smp_processor_id();
+
+    ASSERT(prev != current);
+
+    ctxt_switch_from(prev);
+
+    write_atomic(&prev->dirty_cpu, VCPU_CPU_CLEAN);
+
+    ctxt_switch_to(current);
+
+    write_atomic(&current->dirty_cpu, cpu);
+
+    current->arch.last_cpu = cpu;
+
+    /*
+     * sched_context_switched() internally uses a spinlock,
+     * which requires interrupts to be enabled.
+     */
+    local_irq_enable();
+
+    sched_context_switched(prev, current);
+}
+
+void context_switch(struct vcpu *prev, struct vcpu *next)
+{
+    ASSERT(local_irq_is_enabled());
+    ASSERT(prev != next);
+    ASSERT(!vcpu_cpu_dirty(next));
+
+    local_irq_disable();
+
+    set_current(next);
+
+    prev = __context_switch(prev, next);
+
+    schedule_tail(prev);
+}
+
 static void __init __maybe_unused build_assertions(void)
 {
     /*
diff --git a/xen/arch/riscv/entry.S b/xen/arch/riscv/entry.S
index 202a35fb03a8..017fbb06262c 100644
--- a/xen/arch/riscv/entry.S
+++ b/xen/arch/riscv/entry.S
@@ -99,3 +99,51 @@ restore_registers:
 
         sret
 END(handle_trap)
+
+/*
+ * struct vcpu *__context_switch(struct vcpu *prev, const struct vcpu *next)
+ *
+ * This is called on prev's stack, and returns on next's. As ra is
+ * switched too, it doesn't return to its caller: it returns to where
+ * next last called it from, i.e. into next's own context_switch(), or,
+ * for a vCPU which has never run, to continue_new_vcpu() with an empty
+ * stack.
+ *
+ * a0 - prev
+ * a1 - next
+ *
+ * Returns prev in a0
+ */
+FUNC(__context_switch)
+        REG_S   s0, VCPU_XEN_SAVED_CONTEXT_S0(a0)
+        REG_S   s1, VCPU_XEN_SAVED_CONTEXT_S1(a0)
+        REG_S   s2, VCPU_XEN_SAVED_CONTEXT_S2(a0)
+        REG_S   s3, VCPU_XEN_SAVED_CONTEXT_S3(a0)
+        REG_S   s4, VCPU_XEN_SAVED_CONTEXT_S4(a0)
+        REG_S   s5, VCPU_XEN_SAVED_CONTEXT_S5(a0)
+        REG_S   s6, VCPU_XEN_SAVED_CONTEXT_S6(a0)
+        REG_S   s7, VCPU_XEN_SAVED_CONTEXT_S7(a0)
+        REG_S   s8, VCPU_XEN_SAVED_CONTEXT_S8(a0)
+        REG_S   s9, VCPU_XEN_SAVED_CONTEXT_S9(a0)
+        REG_S   s10, VCPU_XEN_SAVED_CONTEXT_S10(a0)
+        REG_S   s11, VCPU_XEN_SAVED_CONTEXT_S11(a0)
+        REG_S   sp, VCPU_XEN_SAVED_CONTEXT_SP(a0)
+        REG_S   ra, VCPU_XEN_SAVED_CONTEXT_RA(a0)
+
+        REG_L   s0, VCPU_XEN_SAVED_CONTEXT_S0(a1)
+        REG_L   s1, VCPU_XEN_SAVED_CONTEXT_S1(a1)
+        REG_L   s2, VCPU_XEN_SAVED_CONTEXT_S2(a1)
+        REG_L   s3, VCPU_XEN_SAVED_CONTEXT_S3(a1)
+        REG_L   s4, VCPU_XEN_SAVED_CONTEXT_S4(a1)
+        REG_L   s5, VCPU_XEN_SAVED_CONTEXT_S5(a1)
+        REG_L   s6, VCPU_XEN_SAVED_CONTEXT_S6(a1)
+        REG_L   s7, VCPU_XEN_SAVED_CONTEXT_S7(a1)
+        REG_L   s8, VCPU_XEN_SAVED_CONTEXT_S8(a1)
+        REG_L   s9, VCPU_XEN_SAVED_CONTEXT_S9(a1)
+        REG_L   s10, VCPU_XEN_SAVED_CONTEXT_S10(a1)
+        REG_L   s11, VCPU_XEN_SAVED_CONTEXT_S11(a1)
+        REG_L   sp, VCPU_XEN_SAVED_CONTEXT_SP(a1)
+        REG_L   ra, VCPU_XEN_SAVED_CONTEXT_RA(a1)
+
+        ret
+END(__context_switch)
diff --git a/xen/arch/riscv/include/asm/csr.h b/xen/arch/riscv/include/asm/csr.h
index a5cb24c863dd..aad82c6a6d8a 100644
--- a/xen/arch/riscv/include/asm/csr.h
+++ b/xen/arch/riscv/include/asm/csr.h
@@ -40,6 +40,13 @@
     csr_write(csr ## H, v_ >> 32);  \
 })
 
+#define csr_read64(csr)                 \
+({                                      \
+    uint64_t v_ = csr_read(csr ## H);   \
+                                        \
+    (v_ << 32) | csr_read(csr);         \
+})
+
 /*
  * The two halves are read by separate instructions, so a CSR which hardware
  * increments can carry from the low half into the high one in between,
@@ -72,6 +79,12 @@
     (void)csr ## H;                 \
     csr_read(csr);                  \
 })
+
+#define csr_read64(csr)             \
+({                                  \
+    (void)csr ## H;                 \
+    csr_read(csr);                  \
+})
 #endif
 
 #define csr_swap(csr, val)                                      \
diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
index 15e8fa19685e..77ad888a2d6c 100644
--- a/xen/arch/riscv/include/asm/domain.h
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -29,6 +29,11 @@ struct arch_vcpu_io {
 struct arch_vcpu {
     struct vcpu_vmid vmid;
 
+    /*
+     * The last CPU this vCPU ran on. Initialised to CPU_NONE.
+     */
+    unsigned int last_cpu;
+
     /*
      * Callee saved registers for Xen's state used to switch from
      * prev's stack to the next's stack during context switch.
@@ -60,11 +65,19 @@ struct arch_vcpu {
     register_t hcounteren;
     register_t hedeleg;
     register_t hideleg;
-    register_t henvcfg;
+    uint64_t   henvcfg;
     register_t hstateen0;
+    uint64_t   htimedelta;
     register_t hvip;
 
     register_t vsatp;
+    register_t vscause;
+    register_t vsepc;
+    uint64_t   vsie;
+    register_t vsscratch;
+    register_t vsstatus;
+    register_t vstval;
+    register_t vstvec;
 
     /*
      * VCPU interrupts
diff --git a/xen/arch/riscv/include/asm/p2m.h b/xen/arch/riscv/include/asm/p2m.h
index 0d1dace1a0d8..9edf78377ee5 100644
--- a/xen/arch/riscv/include/asm/p2m.h
+++ b/xen/arch/riscv/include/asm/p2m.h
@@ -262,7 +262,6 @@ struct page_info *p2m_get_page_from_gfn(struct p2m_domain *p2m, gfn_t gfn,
 
 void p2m_ctxt_switch_from(struct vcpu *p);
 void p2m_ctxt_switch_to(struct vcpu *n);
-void p2m_handle_vmenter(void);
 
 #endif /* ASM__RISCV__P2M_H */
 
diff --git a/xen/arch/riscv/include/asm/system.h b/xen/arch/riscv/include/asm/system.h
index f33af64fd2ec..d350b9c959c2 100644
--- a/xen/arch/riscv/include/asm/system.h
+++ b/xen/arch/riscv/include/asm/system.h
@@ -76,6 +76,8 @@ static inline bool local_irq_is_enabled(void)
 
 #define arch_fetch_and_add(x, v) __sync_fetch_and_add(x, v)
 
+struct vcpu *__context_switch(struct vcpu *prev, const struct vcpu *next);
+
 #endif /* __ASSEMBLER__ */
 
 #endif /* ASM__RISCV__SYSTEM_H */
diff --git a/xen/arch/riscv/p2m.c b/xen/arch/riscv/p2m.c
index de25607247a6..02e7f6364fad 100644
--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -243,6 +243,15 @@ static void p2m_tlb_flush(struct p2m_domain *p2m)
 
     p2m->need_flush = false;
 
+    /*
+     * Order the p2m updates above against the read of dirty_cpumask below,
+     * pairing with the barrier in p2m_vmid_switch_to(). Either that hart is
+     * seen here and gets an HFENCE.GVMA, or it adds itself to the mask
+     * afterwards, in which case it starts walking this p2m only once the
+     * updates are visible to it.
+     */
+    smp_mb();
+
     sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0);
 }
 
@@ -1504,16 +1513,110 @@ void p2m_ctxt_switch_from(struct vcpu *p)
      * VMID, world-switch code should zero vsatp, then swap hgatp, then
      * finally write the new vsatp value what will be done in
      * p2m_ctxt_switch_to().
-     * Note, that also HGATP update could happen in p2m_handle_vmenter().
      */
     p->arch.vsatp = csr_swap(CSR_VSATP, 0);
 
     /*
-     * Nothing to do with HGATP as it will be update in p2m_ctxt_switch_to()
-     * or/and in p2m_handle_vmenter().
+     * Nothing to do with HGATP as it will be updated in
+     * p2m_ctxt_switch_to().
      */
 }
 
+/*
+ * Domain whose p2m this hart's HGATP points at. ctxt_switch_to() bails out
+ * early for the idle vCPU, so HGATP survives a pass through idle and keeps
+ * pointing at the domain which ran here last. That domain, rather than the
+ * one the scheduler switched away from, is what owns this hart's G-stage
+ * translations. The hart also keeps its place in the owner's dirty_cpumask:
+ * p2m_tlb_flush() goes on reaching it, and a domain which idles between two
+ * runs on the same hart keeps its VMIDs.
+ *
+ * TODO: nothing resets hgatp_owner when its domain is destroyed, so a hart
+ *       which stays idle from the domain's last run until it is freed is
+ *       left with a dangling pointer. Once domain_relinquish_resources()
+ *       tears down the p2m, it has to reset hgatp_owner of every hart still
+ *       pointing at the domain (cmpxchg() against d), i.e. before the RCU
+ *       grace period of domain_destroy() and so before the domain is freed.
+ */
+static DEFINE_PER_CPU(struct domain *, hgatp_owner);
+
+/*
+ * Update this hart's place in the dirty_cpumask of the domains involved and
+ * claim a VMID for n. Must be called before HGATP is pointed at n's p2m, as
+ * it is written with the VMID claimed here.
+ */
+static void p2m_vmid_switch_to(struct vcpu *n)
+{
+    unsigned int cpu = smp_processor_id();
+    struct domain *owner = this_cpu(hgatp_owner);
+    bool need_flush;
+
+    ASSERT(!is_idle_vcpu(n));
+
+    if ( owner != n->domain )
+    {
+        /*
+         * Once this hart drops out of the owner's dirty_cpumask it stops
+         * being a target of p2m_tlb_flush(), while its TLB may still hold
+         * G-stage translations of that domain: none of the vCPUs of that
+         * domain which ran here has had its VMID invalidated. Move the hart
+         * to a new VMID generation so that none of them can be reached
+         * again.
+         *
+         * This has to precede the VMID claim below, so that n gets a VMID of
+         * the new generation.
+         */
+        if ( owner )
+        {
+            vmid_flush_hart();
+
+            cpumask_clear_cpu(cpu, owner->dirty_cpumask);
+        }
+
+        /*
+         * Mark this hart in the incoming domain's dirty_cpumask before HGATP
+         * is pointed at its p2m: from that write on the hart may cache the
+         * p2m's translations, so p2m_tlb_flush() must already reach it.
+         */
+        cpumask_set_cpu(cpu, n->domain->dirty_cpumask);
+
+        /*
+         * Pairs with the barrier in p2m_tlb_flush(). cpumask_set_cpu() is an
+         * unordered AMO on RISC-V, so without this a concurrent flusher could
+         * read the mask without this hart in it while this hart is already
+         * walking the p2m it is about to be pointed at.
+         */
+        smp_mb();
+
+        this_cpu(hgatp_owner) = n->domain;
+    }
+
+    /*
+     * A VMID is meaningful only on the hart whose pool issued it: generations
+     * are per-hart counters which all start at 1 and advance independently,
+     * so the pair a vCPU brings from another hart may match this hart's
+     * generation by coincidence, leaving the vCPU under a VMID which is live
+     * here for someone else.
+     */
+    if ( n->arch.last_cpu != cpu )
+        vmid_flush_vcpu(n);
+
+    /*
+     * Claim the VMID here rather than leaving it to the next guest entry:
+     * the caller makes HGATP live right after, and a stale VMID there pairs
+     * this domain's G-stage root with a tag which may already have been
+     * re-issued to a vCPU of another domain.
+     */
+    need_flush = vmid_handle_vmenter(&n->arch.vmid);
+
+    /*
+     * A VMID isn't re-used until the generation it was issued in wraps, so a
+     * G-stage flush is needed only when vmid_handle_vmenter() says so.
+     */
+    if ( unlikely(need_flush) )
+        local_hfence_gvma_all();
+}
+
 /*
  * As speculation may occur at any time, an incorrect set of page tables could
  * be used. Therefore, this function must be called only after all other guest
@@ -1528,11 +1631,9 @@ void p2m_ctxt_switch_to(struct vcpu *n)
     if ( is_idle_vcpu(n) )
         return;
 
+    p2m_vmid_switch_to(n);
+
     csr_write(CSR_HGATP, construct_hgatp(p2m, n->arch.vmid.vmid));
-    /*
-     * As VMID is unique per vCPU and just re-used here thereby there is no
-     * need for G-stage TLB flush here.
-     */
 
     csr_write(CSR_VSATP, n->arch.vsatp);
 
@@ -1548,48 +1649,6 @@ void p2m_ctxt_switch_to(struct vcpu *n)
     flush_tlb_guest_local();
 }
 
-void p2m_handle_vmenter(void)
-{
-    struct vcpu *curr = current;
-    struct p2m_domain *p2m = p2m_get_hostp2m(curr->domain);
-    struct vcpu_vmid *p_vmid = &curr->arch.vmid;
-    unsigned short old_vmid, new_vmid;
-    bool need_flush;
-
-    BUG_ON(is_idle_vcpu(curr));
-
-    old_vmid = p_vmid->vmid;
-    need_flush = vmid_handle_vmenter(p_vmid);
-    new_vmid = p_vmid->vmid;
-
-#ifdef P2M_DEBUG
-    printk("%pv: oldvmid(%d) new_vmid(%d), need_flush(%d)\n",
-           curr, old_vmid, new_vmid, need_flush);
-#endif
-
-    /*
-     * There is no need to set VSATP to 0 to stop speculation before updating
-     * HGATP, as VSATP is not modified here.
-     */
-    if ( old_vmid != new_vmid )
-        csr_write(CSR_HGATP, construct_hgatp(p2m, p_vmid->vmid));
-
-    /*
-     * There is also no need to flush G-stage TLB unconditionally as old VMID
-     * won't be reused until need_flush is set to true.
-     */
-    if ( unlikely(need_flush) )
-        local_hfence_gvma_all();
-
-    /*
-     * There is also no need to flush the VS-stage TLB: even if speculation
-     * occurs (VSATP + old HGATP were used), it will use the old VMID, which
-     * won't be reused until need_flush is set to true. When VMIDs aren't
-     * available there is no old VMID to rely on, but then need_flush is set
-     * on every entry, so the flush above covers that case.
-     */
-}
-
 struct page_info *get_page_from_gfn(struct domain *d, unsigned long gfn,
                                     p2m_type_t *t, p2m_query_t q)
 {
diff --git a/xen/arch/riscv/riscv64/asm-offsets.c b/xen/arch/riscv/riscv64/asm-offsets.c
index 1290b9dbbe82..c1be1614ce94 100644
--- a/xen/arch/riscv/riscv64/asm-offsets.c
+++ b/xen/arch/riscv/riscv64/asm-offsets.c
@@ -1,8 +1,10 @@
 #define COMPILE_OFFSETS
 
+#include <xen/sched.h>
+#include <xen/types.h>
+
 #include <asm/current.h>
 #include <asm/processor.h>
-#include <xen/types.h>
 
 #define DEFINE(_sym, _val)                                                 \
     asm volatile ( "\n.ascii\"==>#define " #_sym " %0 /* " #_val " */<==\""\
@@ -53,4 +55,19 @@ void asm_offsets(void)
     BLANK();
     DEFINE(PCPU_INFO_SIZE, sizeof(struct pcpu_info));
     BLANK();
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S0, struct vcpu, arch.xen_saved_context.s0);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S1, struct vcpu, arch.xen_saved_context.s1);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S2, struct vcpu, arch.xen_saved_context.s2);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S3, struct vcpu, arch.xen_saved_context.s3);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S4, struct vcpu, arch.xen_saved_context.s4);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S5, struct vcpu, arch.xen_saved_context.s5);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S6, struct vcpu, arch.xen_saved_context.s6);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S7, struct vcpu, arch.xen_saved_context.s7);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S8, struct vcpu, arch.xen_saved_context.s8);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S9, struct vcpu, arch.xen_saved_context.s9);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S10, struct vcpu, arch.xen_saved_context.s10);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_S11, struct vcpu, arch.xen_saved_context.s11);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_SP, struct vcpu, arch.xen_saved_context.sp);
+    OFFSET(VCPU_XEN_SAVED_CONTEXT_RA, struct vcpu, arch.xen_saved_context.ra);
+    BLANK();
 }
diff --git a/xen/arch/riscv/stubs.c b/xen/arch/riscv/stubs.c
index 3a7953593d93..e0febae432b2 100644
--- a/xen/arch/riscv/stubs.c
+++ b/xen/arch/riscv/stubs.c
@@ -81,11 +81,6 @@ void smp_send_state_dump(unsigned int cpu)
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
-void context_switch(struct vcpu *prev, struct vcpu *next)
-{
-    BUG_ON("unimplemented");
-}
-
 void continue_running(struct vcpu *same)
 {
     BUG_ON("unimplemented");
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index 10d6855d913b..9bb54d030be7 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -178,8 +178,6 @@ static void check_for_pcpu_work(void)
     vcpu_sync_interrupts(curr);
 
     vcpu_flush_interrupts(curr);
-
-    p2m_handle_vmenter();
 }
 
 static void timer_interrupt(void)
-- 
2.55.0

[PATCH v3 12/39] xen/riscv: save and restore vsiselect on vCPU context switch

[Oleksii Kurochko]

vsiselect is a per-hart CSR which a guest changes on its own: when V=1,
VS-mode accesses to siselect are really accesses to vsiselect.
Architecturally a vCPU has to find there the value it last wrote, but as
long as the CSR isn't part of the vCPU context it finds whatever selector
the vCPU which ran on the hart before it left behind. A guest which writes
siselect, is descheduled and then reads sireg without rewriting siselect
therefore reaches a register it never selected, and it can also observe
another guest's selector value.

When Smstateen is implemented, access to vsiselect and vsireg is gated by
hstateen0.CSRIND (bit 60, SMSTATEEN0_SVSLCT in Xen's headers), and
v->arch.hstateen0 holds the bits vcpu_csr_init() ended up with. A clear
bit there covers the two cases in which the CSR has to be skipped:

 - Xen didn't hand the guest access to it, so the guest can't have changed
   the CSR and there is no state to preserve;
 - M-mode denied the state altogether. Smstateen makes a bit which is zero
   in mstateen0 read-only zero in hstateen0, and a zero bit in mstateen0
   traps accesses from every privilege mode less privileged than M-mode,
   HS-mode included, so Xen couldn't even read the CSR to save it.

Without Smstateen no bit controls access to the CSR, so it is saved and
restored whenever Ssaia is available.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Rename from "xen/riscv: save and restore AIA state on vCPU context
   switch", as only vsiselect is switched now.
 - Save and restore only vsiselect: drop hviprio1 and hviprio2.
 - Rename vcpu_has_aia_state() to vcpu_can_access_vsiselect() and drop
   its hstateen0 bit argument.
 - Move switching vsie to uint64_t and csr_{read,write}64() to "xen/riscv:
   implement vCPU context switching".
 - Update the commit message.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/domain.c             | 28 ++++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/domain.h |  1 +
 2 files changed, 29 insertions(+)

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index f80643e2d82b..610026fd81c0 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -326,6 +326,28 @@ int arch_domain_create(struct domain *d,
     return rc;
 }
 
+/*
+ * vsiselect is a per-hart CSR, but a guest changes it on its own: when V=1,
+ * VS-mode accesses to siselect are really accesses to vsiselect. Hence it is
+ * part of the vCPU context.
+ *
+ * When Smstateen is implemented, hstateen0.CSRIND (SMSTATEEN0_SVSLCT) gates
+ * that access, and a bit staying clear in v->arch.hstateen0 (see
+ * vcpu_csr_init()) means either that the guest was never given access to the
+ * CSR, and so can't have changed it, or that M-mode denied the state
+ * altogether, in which case the CSR can't be accessed from HS-mode either.
+ */
+static bool vcpu_can_access_vsiselect(const struct vcpu *v)
+{
+    if ( !riscv_isa_extension_available(NULL, RISCV_ISA_EXT_ssaia) )
+        return false;
+
+    if ( !riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
+        return true;
+
+    return v->arch.hstateen0 & SMSTATEEN0_SVSLCT;
+}
+
 static void csr_regs_ctxt_switch_from(struct vcpu *p)
 {
     /*
@@ -353,6 +375,9 @@ static void csr_regs_ctxt_switch_from(struct vcpu *p)
     p->arch.vscause = csr_read(CSR_VSCAUSE);
     p->arch.vstval = csr_read(CSR_VSTVAL);
     p->arch.vsepc = csr_read(CSR_VSEPC);
+
+    if ( vcpu_can_access_vsiselect(p) )
+        p->arch.vsiselect = csr_read(CSR_VSISELECT);
 }
 
 static void csr_regs_ctxt_switch_to(struct vcpu *n)
@@ -374,6 +399,9 @@ static void csr_regs_ctxt_switch_to(struct vcpu *n)
     csr_write(CSR_VSCAUSE, n->arch.vscause);
     csr_write(CSR_VSTVAL, n->arch.vstval);
     csr_write(CSR_VSEPC, n->arch.vsepc);
+
+    if ( vcpu_can_access_vsiselect(n) )
+        csr_write(CSR_VSISELECT, n->arch.vsiselect);
 }
 
 static void ctxt_switch_from(struct vcpu *p)
diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
index 77ad888a2d6c..c33d83836f2e 100644
--- a/xen/arch/riscv/include/asm/domain.h
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -74,6 +74,7 @@ struct arch_vcpu {
     register_t vscause;
     register_t vsepc;
     uint64_t   vsie;
+    register_t vsiselect;
     register_t vsscratch;
     register_t vsstatus;
     register_t vstval;
-- 
2.55.0

[PATCH v3 13/39] xen/riscv: introduce vintc_ctxt_switch_{from,to}()

[Oleksii Kurochko]

Virtual interrupt controller state must be preserved across vCPU context
switches.

Introduce vintc_ctxt_switch_{from,to}() wrappers around new
ctxt_switch_{from,to}() hooks in struct vintc_ops, and call them from the
context switch path, so that this state can be saved/restored without
knowing which vINTC variant a domain uses.

No vINTC variant implements the hooks yet: the vAPLIC implementation is
added separately.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
Changes in v3:
 - s/v/{p,n} for vintc_ctxt_switch_{from,to}() arguments and also for
   hook prototypes.
 - Add Acked-by: Jan Beulich <jbeulich@suse.com>.
---
Changes in v2:
 - Update the commit message.
 - s/vintc_state_{save,restore}/vintc_ctxt_switch_{to,from}.
 - s/{re}store_state/ctxt_switch_{to,from} for vintc_ops.
 - s/vcpu/v for vintc_ctxt_switch_{to,from}() arguments.
---
---
 xen/arch/riscv/domain.c           |  4 ++++
 xen/arch/riscv/include/asm/intc.h |  9 +++++++++
 xen/arch/riscv/intc.c             | 14 ++++++++++++++
 3 files changed, 27 insertions(+)

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index 610026fd81c0..fd21cb1789e6 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -418,6 +418,8 @@ static void ctxt_switch_from(struct vcpu *p)
 
     vtimer_ctxt_switch_from(p);
 
+    vintc_ctxt_switch_from(p);
+
     csr_regs_ctxt_switch_from(p);
 }
 
@@ -433,6 +435,8 @@ static void ctxt_switch_to(struct vcpu *n)
 
     csr_regs_ctxt_switch_to(n);
 
+    vintc_ctxt_switch_to(n);
+
     vtimer_ctxt_switch_to(n);
 
     p2m_ctxt_switch_to(n);
diff --git a/xen/arch/riscv/include/asm/intc.h b/xen/arch/riscv/include/asm/intc.h
index 1bfba7c6155b..59e12f3d3c6d 100644
--- a/xen/arch/riscv/include/asm/intc.h
+++ b/xen/arch/riscv/include/asm/intc.h
@@ -64,6 +64,12 @@ struct vintc_ops {
 
     /* Deinitialize some vINTC-related stuff for a vCPU */
     void (*vcpu_deinit)(struct vcpu *v);
+
+    /* Save vINTC state of the vCPU being switched out */
+    void (*ctxt_switch_from)(struct vcpu *p);
+
+    /* Restore vINTC state of the vCPU being switched in */
+    void (*ctxt_switch_to)(struct vcpu *n);
 };
 
 struct vintc {
@@ -91,4 +97,7 @@ void domain_vintc_deinit(struct domain *d);
 
 int vintc_reserve_virq(const struct domain *d, unsigned int virq);
 
+void vintc_ctxt_switch_from(struct vcpu *p);
+void vintc_ctxt_switch_to(struct vcpu *n);
+
 #endif /* ASM__RISCV__INTERRUPT_CONTOLLER_H */
diff --git a/xen/arch/riscv/intc.c b/xen/arch/riscv/intc.c
index 810d126e263d..53ba2b04ab53 100644
--- a/xen/arch/riscv/intc.c
+++ b/xen/arch/riscv/intc.c
@@ -178,3 +178,17 @@ int __overlay_init vintc_reserve_virq(const struct domain *d,
 
     return test_and_set_bit(virq, d->arch.vintc->used_irqs) ? -EEXIST : 0;
 }
+
+void vintc_ctxt_switch_from(struct vcpu *p)
+{
+    const struct vintc_ops *ops = p->domain->arch.vintc->ops;
+
+    ops->ctxt_switch_from(p);
+}
+
+void vintc_ctxt_switch_to(struct vcpu *n)
+{
+    const struct vintc_ops *ops = n->domain->arch.vintc->ops;
+
+    ops->ctxt_switch_to(n);
+}
-- 
2.55.0

[PATCH v3 14/39] xen/riscv: add IMSIC vCPU context switch handlers

[Oleksii Kurochko]

IMSIC state currently needs to track only which physical CPU owns a vCPU's
IMSIC guest interrupt file, as the CPU id is part of the physical address
the file is mapped at.

Add imsic_ctxt_switch_from() to record that CPU when a vCPU is switched
out. A vCPU running on the s/w VS-file has no h/w file bound to a CPU, so
there is nothing to record for it. Nothing reads the recorded value yet.

imsic_ctxt_switch_to() has nothing to do: switching a vCPU in on the CPU
which owns its guest interrupt file needs no IMSIC update. Only a move to
another CPU does (a new VGEIN and a remapped file), and that isn't part of
the context switch.

Install both as the ctxt_switch_{from,to} hooks of struct vintc_ops. MSI
delivery is the only mode Xen supports (aplic_init() panics on an APLIC
without an "msi-parent" property) and the only one vAPLIC emulates, so the
vAPLIC state to save and restore is always the IMSIC one and no
vAPLIC-level forwarder is needed. Being indirect call targets, both
handlers get cf_check.

Co-developed-by: Romain Caritey <Romain.Caritey@microchip.com>
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Drop the reference to aplic_init() from the comment on the
   .ctxt_switch_{from,to} initialization in vaplic.c.
 - Use p and n instead of v as the arguments of
   imsic_ctxt_switch_{from,to}(), as the other context switch helpers do.
 - Update the commit message.
---
Changes in v2:
 - s/imsic_state_{save,restore}/imsic_ctxt_switch_{from,to}: the old names
   suggested saving and restoring register state, which isn't what these
   functions do.
 - Fix the comment in imsic_ctxt_switch_from(): it explained the
   ->vsfile_cpu sentinel while the code checks ->guest_file_id.
 - Adapt to ->vsfile_cpu holding v->processor instead of a hartid.
 - Add cf_check as both are indirect call targets now.
 - Fold in the vintc_ops hook-up, which was a separate patch in v1. It no
   longer adds vaplic_state_{save,restore}() forwarders: the
   BUG_ON("unimplemented") path in them was unreachable and
   has_msi_support() was an MMIO read done on every context switch.
 - Drop the claim that the not-yet-supported case is guarded by a BUG_ON();
   there is no such BUG_ON().
 - Update the subject accordingly.
---
---
 xen/arch/riscv/imsic.c             | 23 +++++++++++++++++++++++
 xen/arch/riscv/include/asm/imsic.h |  3 +++
 xen/arch/riscv/vaplic.c            |  6 ++++++
 3 files changed, 32 insertions(+)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 402b2f14f591..d633f59db20d 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -20,6 +20,7 @@
 #include <xen/init.h>
 #include <xen/libfdt/libfdt.h>
 #include <xen/macros.h>
+#include <xen/rwlock.h>
 #include <xen/sched.h>
 #include <xen/smp.h>
 #include <xen/spinlock.h>
@@ -342,6 +343,28 @@ static int __init imsic_parse_node(const struct dt_device_node *node,
     return 0;
 }
 
+void cf_check imsic_ctxt_switch_from(struct vcpu *p)
+{
+    struct vimsic_state *imsic_state = p->arch.vimsic_state;
+    unsigned long flags;
+
+    /*
+     * A vCPU using the s/w IMSIC VS-file (guest_file_id == 0) has no h/w
+     * VS-file bound to a physical CPU, so there is no location to record.
+     */
+    if ( !vcpu_guest_file_id(p) )
+        return;
+
+    write_lock_irqsave(&imsic_state->vsfile_lock, flags);
+    imsic_state->vsfile_cpu = p->processor;
+    write_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
+}
+
+void cf_check imsic_ctxt_switch_to(struct vcpu *n)
+{
+    /* Nothing to do */
+}
+
 int cf_check vcpu_imsic_init(struct vcpu *v)
 {
     struct vimsic_state *imsic_state;
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index 6baadc008d81..5463fb4c31b2 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -96,4 +96,7 @@ unsigned int vcpu_guest_file_id(const struct vcpu *v);
 
 int vimsic_make_domu_dt_node(struct kernel_info *kinfo, unsigned int *phandle);
 
+void imsic_ctxt_switch_from(struct vcpu *p);
+void imsic_ctxt_switch_to(struct vcpu *n);
+
 #endif /* ASM_RISCV_IMSIC_H */
diff --git a/xen/arch/riscv/vaplic.c b/xen/arch/riscv/vaplic.c
index ea4b0dc88951..4d87fdb5be42 100644
--- a/xen/arch/riscv/vaplic.c
+++ b/xen/arch/riscv/vaplic.c
@@ -456,6 +456,12 @@ static const struct mmio_handler_ops vaplic_mmio_ops = {
 static const struct vintc_ops vintc_ops = {
     .vcpu_init = vcpu_imsic_init,
     .vcpu_deinit = vcpu_imsic_deinit,
+    /*
+     * MSI delivery is the only supported mode, so the vAPLIC state to save
+     * and restore is always the IMSIC one.
+     */
+    .ctxt_switch_from = imsic_ctxt_switch_from,
+    .ctxt_switch_to = imsic_ctxt_switch_to,
 };
 
 int domain_vaplic_init(struct domain *d)
-- 
2.55.0

[PATCH v3 15/39] xen/riscv: extend exception tables with type and data fields

[Oleksii Kurochko]

Extend the RISC-V exception table format to include a type and
auxiliary data field.

The existing format only supports simple fixups. Some use cases require
additional context from the fault (e.g. capturing trap information),
which cannot be expressed with the current EX_TYPE_FIXUP entries.

Introduce a generic ASM_EXTABLE_RAW() helper to describe entries with a
handler type and associated data. Reimplement ASM_EXTABLE() in terms of
it using EX_TYPE_FIXUP for compatibility.

Add EX_TYPE_TRAP_INFO to allow handlers to retrieve trap state
(sepc/scause/stval) and pass it to the fixup path. The data field is
used to encode which GPR contains a pointer to a struct trap_info.
Provide ASM_EXTABLE_TRAP_INFO() as a convenience wrapper for this case,
and pass the trap cause down to fixup_exception() for it.

Also add gpr-num.h, providing symbolic GPR numbers for use in assembly
and inline asm. It is derived from Linux 6.16, with the register list
turned into a GPR_LIST() macro instead of being open-coded with .irp, so
that it can be used from C as well.

Resolving a GPR number to the saved value of the register relies on
struct cpu_user_regs holding x0..x31 contiguously and in register-number
order. Add regs_gpr_ptr() for that, and check the layout against
GPR_LIST() at build time. The block doesn't have to be at the start of
the structure, so anchor REG_PTR() at ->zero too rather than at the
structure itself.

Update the exception handling code to dispatch based on the entry type.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Keep the name of the asm_extable assembler macro: renaming it to
   _asm_extable was unrelated to this patch.
 - Replace regs_get_gpr() in extable.c with regs_gpr_ptr() in asm/regs.h,
   returning where the register's value is saved rather than the value
   itself, so that a single helper serves code which reads a guest register
   as well as code which writes one. Anchor the array access at
   &regs->zero rather than casting regs itself, matching the relaxed
   layout requirement below, and bound the index with array_index_nospec()
   so that release builds, where the ASSERT() is compiled out, cannot read
   past the register frame. Add NR_GPRS instead of open-coding 32.
 - Move the layout BUILD_BUG_ON()s out of regs_get_gpr() into
   build_assertions() in domain.c, so that they no longer sit in one
   particular consumer of the mapping.
 - Anchor those checks at offsetof(struct cpu_user_regs, zero) rather than
   at offset 0: the x0..x31 block only has to be contiguous and in
   register-number order, it no longer has to sit at the start of the
   structure. Update the comment above struct cpu_user_regs accordingly,
   keeping the "->zero must read as 0" requirement as a separate point.
 - REG_PTR(): anchor at &regs->zero for the same reason.
 - gpr-num.h: move the part of the comment describing the generation of the
   .L_gpr_num_<name> symbols down to the construct which actually generates
   them; GPR_LIST() itself has other users.
 - gpr-num.h: emit the separator ahead of each list entry instead of after
   it, so that the expansion doesn't end with one and whatever follows a use
   of the list has to supply its own; ASM_EXTABLE_TRAP_INFO() supplies the
   "\n" it needs.
 - gpr-num.h: state in the file header what was changed compared to the
   Linux original.
 - asm/extable.h: no functional change, put EX_TRAP_INFO_REG() on a single
   line and reflow the comment describing the exception table entry format.
 - Drop BUG_ON() in ex_handler_trap_info() as trap_info pointer should be
   guaranteed properly filled by compilation process.
---
Changes in v2:
 - regs_get_gpr(): take a GPR number instead of a byte offset, dropping the
   multiplication at the call site. A non-multiple-of-8 offset is now
   unrepresentable.
 - regs_get_gpr(): ASSERT(num < 32) instead of a range check returning 0,
   which isn't usable as an error indicator. No num == 0 check: reading x0
   is harmless out of context, and a NULL trap_info is caught by the
   existing BUG_ON() where the dereference happens.
 - regs_get_gpr(): index the register frame as an array, constify regs and
   drop the pointless inline.
 - Anchor the first BUILD_BUG_ON() at offsetof(..., zero) == 0 rather than
   at ra, pinning both ends of the x0..x31 range.
 - Drop MAX_REG_OFFSET; asm/processor.h is no longer touched by this patch.
 - ex_handler_trap_info(): use regs->sepc and the cause passed down from
   do_trap() instead of re-reading CSR_SEPC/CSR_SCAUSE; fixup_exception()
   gains a cause parameter. stval stays a CSR read as do_trap() doesn't
   read it.
 - asm/extable.h: revert the .word -> .long change, the extra parens and the
   stray semicolon after .popsection; use .half for the new type and data
   fields to match .word.
 - Make GPR_LIST() the single source of the ABI-name -> register-number
   mapping, and check struct cpu_user_regs against it at build time, so the
   two can no longer diverge.
 - Add the explanatory comment above sturc cpu_user_regs to explain an
   ordering of x0-x31 registers.
---
---
 xen/arch/riscv/domain.c                     | 22 ++++++++
 xen/arch/riscv/extable.c                    | 48 +++++++++++++++-
 xen/arch/riscv/include/asm/extable.h        | 61 ++++++++++++++-------
 xen/arch/riscv/include/asm/gpr-num.h        | 53 ++++++++++++++++++
 xen/arch/riscv/include/asm/processor.h      | 14 ++++-
 xen/arch/riscv/include/asm/regs.h           | 23 +++++++-
 xen/arch/riscv/include/asm/riscv_encoding.h |  7 ++-
 xen/arch/riscv/include/asm/traps.h          |  6 ++
 xen/arch/riscv/traps.c                      |  2 +-
 9 files changed, 210 insertions(+), 26 deletions(-)
 create mode 100644 xen/arch/riscv/include/asm/gpr-num.h

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index fd21cb1789e6..f60f871aba98 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -12,6 +12,7 @@
 #include <asm/cpufeature.h>
 #include <asm/csr.h>
 #include <asm/current.h>
+#include <asm/gpr-num.h>
 #include <asm/intc.h>
 #include <asm/mmio.h>
 #include <asm/riscv_encoding.h>
@@ -484,9 +485,30 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
 static void __init __maybe_unused build_assertions(void)
 {
+/*
+ * GPR_LIST() puts nothing between its entries, so the separator has to come
+ * from here. It is emitted ahead of each entry rather than after it, so that
+ * the expansion doesn't end with one and its use below can be terminated like
+ * an ordinary statement.
+ */
+#define CHECK_GPR_INDEX(num, name)                          \
+    ; BUILD_BUG_ON(offsetof(struct cpu_user_regs, name) !=  \
+                   offsetof(struct cpu_user_regs, zero) +   \
+                   (num) * sizeof(unsigned long))
+
     /*
      * Enforce the requirement documented in struct cpu_info that
      * guest_cpu_user_regs must be the first field.
      */
     BUILD_BUG_ON(offsetof(struct cpu_info, guest_cpu_user_regs));
+
+    /*
+     * Enforce the requirement documented in struct cpu_user_regs that the
+     * x0..x31 fields are contiguous and in architectural register-number
+     * order, which is what lets a register number be resolved to its saved
+     * value by indexing them as an array anchored at ->zero.
+     */
+    GPR_LIST(CHECK_GPR_INDEX);
+
+#undef CHECK_GPR_INDEX
 }
diff --git a/xen/arch/riscv/extable.c b/xen/arch/riscv/extable.c
index 5b89c4278c65..19d9e3617c81 100644
--- a/xen/arch/riscv/extable.c
+++ b/xen/arch/riscv/extable.c
@@ -6,8 +6,11 @@
 #include <xen/sort.h>
 #include <xen/virtual_region.h>
 
+#include <asm/csr.h>
 #include <asm/extable.h>
 #include <asm/processor.h>
+#include <asm/regs.h>
+#include <asm/traps.h>
 
 #define EX_FIELD(ptr, field) ((unsigned long)&(ptr)->field + (ptr)->field)
 
@@ -32,6 +35,12 @@ static void __init cf_check swap_ex(void *a, void *b)
 
     x->fixup = y->fixup + delta;
     y->fixup = tmp.fixup - delta;
+
+    x->type = y->type;
+    y->type = tmp.type;
+
+    x->data = y->data;
+    y->data = tmp.data;
 }
 
 static int cf_check cmp_ex(const void *a, const void *b)
@@ -59,7 +68,26 @@ static void ex_handler_fixup(const struct exception_table_entry *ex,
     regs->sepc = ex_fixup(ex);
 }
 
-bool fixup_exception(struct cpu_user_regs *regs)
+static void ex_handler_trap_info(const struct exception_table_entry *ex,
+                                 struct cpu_user_regs *regs,
+                                 unsigned long cause)
+{
+    struct trap_info *trap_info =
+        (struct trap_info *)*regs_gpr_ptr(regs, ex->data);
+
+    /*
+     * Only stval still needs a CSR read: sepc and scause were already
+     * captured by the trap entry path and do_trap() respectively. Latch
+     * trap_info->sepc before regs->sepc is pointed at the fixup code.
+     */
+    trap_info->sepc = regs->sepc;
+    trap_info->scause = cause;
+    trap_info->stval = csr_read(CSR_STVAL);
+
+    regs->sepc = ex_fixup(ex);
+}
+
+bool fixup_exception(struct cpu_user_regs *regs, unsigned long cause)
 {
     unsigned long pc = regs->sepc;
     const struct virtual_region *region = find_text_region(pc);
@@ -77,7 +105,23 @@ bool fixup_exception(struct cpu_user_regs *regs)
     if ( !ex )
         return false;
 
-    ex_handler_fixup(ex, regs);
+    switch ( ex->type )
+    {
+    case EX_TYPE_FIXUP:
+        ex_handler_fixup(ex, regs);
+        break;
+
+    case EX_TYPE_TRAP_INFO:
+        ex_handler_trap_info(ex, regs, cause);
+        break;
+
+    default:
+        printk(XENLOG_ERR
+               "Unsupported exception table entry type %u for pc %#lx\n",
+               ex->type, pc);
+
+        return false;
+    }
 
     return true;
 }
diff --git a/xen/arch/riscv/include/asm/extable.h b/xen/arch/riscv/include/asm/extable.h
index c0128a91818f..1ff557b70dd6 100644
--- a/xen/arch/riscv/include/asm/extable.h
+++ b/xen/arch/riscv/include/asm/extable.h
@@ -3,17 +3,24 @@
 #ifndef ASM__RISCV__ASM_EXTABLE_H
 #define ASM__RISCV__ASM_EXTABLE_H
 
+#include <asm/gpr-num.h>
+
+#define EX_TYPE_FIXUP       0
+#define EX_TYPE_TRAP_INFO   1
+
 #ifdef __ASSEMBLER__
 
-#define ASM_EXTABLE(insn, fixup) \
-    .pushsection .ex_table, "a"; \
-    .balign     4;               \
-    .word       (insn) - .;      \
-    .word       (fixup) - .;     \
+#define ASM_EXTABLE_RAW(insn, fixup, type, data)    \
+    .pushsection .ex_table, "a";                    \
+    .balign     4;                                  \
+    .word       (insn) - .;                         \
+    .word       (fixup) - .;                        \
+    .half       (type);                             \
+    .half       (data);                             \
     .popsection
 
 .macro asm_extable, insn, fixup
-    ASM_EXTABLE(\insn, \fixup)
+    ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_FIXUP, 0)
 .endm
 
 #else /* __ASSEMBLER__ */
@@ -23,20 +30,35 @@
 
 struct cpu_user_regs;
 
-#define ASM_EXTABLE(insn, fixup)      \
-    ".pushsection .ex_table, \"a\"\n" \
-    ".balign    4\n"                  \
-    ".word      (" #insn " - .)\n"    \
-    ".word      (" #fixup " - .)\n"   \
+#define ASM_EXTABLE_RAW(insn, fixup, type, data)    \
+    ".pushsection .ex_table, \"a\"\n"               \
+    ".balign    4\n"                                \
+    ".word      (" insn ") - .\n"                   \
+    ".word      (" fixup ") - .\n"                  \
+    ".half      (" type ")\n"                       \
+    ".half      (" data ")\n"                       \
     ".popsection\n"
 
+#define ASM_EXTABLE(insn, fixup)    \
+    ASM_EXTABLE_RAW(#insn, #fixup, __stringify(EX_TYPE_FIXUP), "0")
+
+#define EX_TRAP_INFO_REG(gpr) "(.L_gpr_num_" #gpr ")"
+
+#define ASM_EXTABLE_TRAP_INFO(insn, fixup, data)                    \
+    DEFINE_ASM_GPR_NUMS "\n"                                        \
+    ASM_EXTABLE_RAW(#insn, #fixup, __stringify(EX_TYPE_TRAP_INFO),  \
+                    EX_TRAP_INFO_REG(data))
+
 /*
- * The exception table consists of pairs of relative offsets: the first
- * is the relative offset to an instruction that is allowed to fault,
- * and the second is the relative offset at which the program should
- * continue. No general-purpose registers are modified by the exception
- * handling mechanism itself, so it is up to the fixup code to handle
- * any necessary state cleanup.
+ * Each exception table entry consists of two relative offsets and a
+ * handler description: `insn` is the relative offset to an instruction
+ * that is allowed to fault, `fixup` is the relative offset at which the
+ * program should continue in case of a fault, `type` selects how the exception
+ * is handled (EX_TYPE_*), and `data` holds auxiliary information for the
+ * handler (e.g. for EX_TYPE_TRAP_INFO, the number of the GPR that contains a
+ * pointer to a struct trap_info). No general-purpose registers are
+ * modified by the exception handling mechanism itself, so it is up to
+ * the fixup code to handle any necessary state cleanup.
  *
  * The exception table and fixup code live out of line with the main
  * instruction path. This means when everything is well, we don't even
@@ -45,14 +67,15 @@ struct cpu_user_regs;
  */
 struct exception_table_entry {
     int32_t insn, fixup;
+    uint16_t type, data;
 };
 
 extern struct exception_table_entry __start___ex_table[];
 extern struct exception_table_entry __stop___ex_table[];
 
 void sort_exception_tables(void);
-bool fixup_exception(struct cpu_user_regs *regs);
+bool fixup_exception(struct cpu_user_regs *regs, unsigned long cause);
 
-#endif /* __ASSEMBLY__ */
+#endif /* __ASSEMBLER__ */
 
 #endif /* ASM__RISCV__ASM_EXTABLE_H */
diff --git a/xen/arch/riscv/include/asm/gpr-num.h b/xen/arch/riscv/include/asm/gpr-num.h
new file mode 100644
index 000000000000..867ee172c068
--- /dev/null
+++ b/xen/arch/riscv/include/asm/gpr-num.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Based on Linux 6.16 arch/riscv/include/asm/gpr-num.h, with the
+ * register list turned into GPR_LIST() instead of being open-coded
+ * with .irp, so that it can also be used outside of assembly.
+ */
+#ifndef RISCV_GPR_NUM_H
+#define RISCV_GPR_NUM_H
+
+/*
+ * GPRs by ABI name, together with their register number (x0 .. x31).
+ *
+ * This is the single source of truth for the mapping; users expand it with
+ * their own per-register macro. Among them, the layout of struct cpu_user_regs
+ * is checked against this list at build time (see build_assertions() in
+ * domain.c), so neither list can be changed without the other.
+ */
+#define GPR_LIST(x)                                 \
+    x(0,  zero) x(1,  ra)  x(2,  sp)  x(3,  gp)     \
+    x(4,  tp)   x(5,  t0)  x(6,  t1)  x(7,  t2)     \
+    x(8,  s0)   x(9,  s1)  x(10, a0)  x(11, a1)     \
+    x(12, a2)   x(13, a3)  x(14, a4)  x(15, a5)     \
+    x(16, a6)   x(17, a7)  x(18, s2)  x(19, s3)     \
+    x(20, s4)   x(21, s5)  x(22, s6)  x(23, s7)     \
+    x(24, s8)   x(25, s9)  x(26, s10) x(27, s11)    \
+    x(28, t3)   x(29, t4)  x(30, t5)  x(31, t6)
+
+/* Number of GPRs described by GPR_LIST(). */
+#define NR_GPRS 32
+
+#ifdef __ASSEMBLER__
+
+/*
+ * Generate the .L_gpr_num_<name> assembler symbols, used to turn a register
+ * name emitted by the compiler into a register number.
+ *
+ * The separator is emitted ahead of each entry rather than after it, so that
+ * the expansion doesn't end with one: whatever follows a use of the list has
+ * to supply its own separator, instead of silently relying on a trailing one.
+ */
+#define GPR_NUM_EQU(num, name)  ; .equ .L_gpr_num_##name, num
+GPR_LIST(GPR_NUM_EQU)
+#undef GPR_NUM_EQU
+
+#else /* __ASSEMBLER__ */
+
+/* See the comment ahead of the __ASSEMBLER__ flavour above. */
+#define GPR_NUM_EQU(num, name)  "\n.equ .L_gpr_num_" #name ", " #num
+#define DEFINE_ASM_GPR_NUMS     GPR_LIST(GPR_NUM_EQU)
+
+#endif /* __ASSEMBLER__ */
+
+#endif /* RISCV_GPR_NUM_H */
diff --git a/xen/arch/riscv/include/asm/processor.h b/xen/arch/riscv/include/asm/processor.h
index b1745c107100..00c7654c3278 100644
--- a/xen/arch/riscv/include/asm/processor.h
+++ b/xen/arch/riscv/include/asm/processor.h
@@ -12,7 +12,19 @@
 
 #ifndef __ASSEMBLER__
 
-/* On stack VCPU state */
+/*
+ * On stack VCPU state.
+ *
+ * The x0..x31 fields must remain contiguous and in architectural
+ * register-number order: code which resolves a register number to its saved
+ * value indexes them as an array anchored at ->zero. Where that block sits
+ * within the structure is of no interest, but nothing may be inserted in
+ * between its fields. The layout is checked against GPR_LIST() at build
+ * time; see build_assertions() in domain.c.
+ *
+ * ->zero additionally has to always read as 0, since it supplies the value
+ * of x0 when x0 is used as a source operand.
+ */
 struct cpu_user_regs
 {
     unsigned long zero;
diff --git a/xen/arch/riscv/include/asm/regs.h b/xen/arch/riscv/include/asm/regs.h
index 531958f3d748..59573bb6013c 100644
--- a/xen/arch/riscv/include/asm/regs.h
+++ b/xen/arch/riscv/include/asm/regs.h
@@ -5,16 +5,35 @@
 #ifndef __ASSEMBLER__
 
 #include <xen/bug.h>
+#include <xen/nospec.h>
 
-#define hyp_mode(r)     (0)
+#include <asm/gpr-num.h>
+#include <asm/processor.h>
 
-struct cpu_user_regs;
+#define hyp_mode(r)     (0)
 
 static inline bool guest_mode(const struct cpu_user_regs *r)
 {
     BUG_ON("unimplemented");
 }
 
+/*
+ * Resolve an architectural register number to where its value is saved in
+ * @regs. Relies on x0..x31 being laid out as an array anchored at ->zero; see
+ * the comment above struct cpu_user_regs.
+ */
+static inline unsigned long *regs_gpr_ptr(struct cpu_user_regs *regs,
+                                          unsigned int num)
+{
+    ASSERT(num < NR_GPRS);
+
+    /*
+     * Not a speculation concern, but a convenient way to keep the access in
+     * bounds in release builds, where the assertion above is compiled out.
+     */
+    return &(&regs->zero)[array_index_nospec(num, NR_GPRS)];
+}
+
 #endif /* __ASSEMBLER__ */
 
 #endif /* ASM__RISCV__REGS_H */
diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index 647268e6d0d4..599ae3cd601a 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -911,8 +911,13 @@
 #define REG_OFFSET(insn, pos)		\
 	(SHIFT_RIGHT((insn), (pos) - LOG_REGBYTES) & REG_MASK)
 
+/*
+ * The x0..x31 block of struct cpu_user_regs is not required to sit at the
+ * start of the structure, so anchor the offset at ->zero (x0) rather than
+ * at the structure itself.
+ */
 #define REG_PTR(insn, pos, regs)	\
-	(unsigned long *)((unsigned long)(regs) + REG_OFFSET(insn, pos))
+	(unsigned long *)((unsigned long)&(regs)->zero + REG_OFFSET(insn, pos))
 
 #define GET_RM(insn)			(((insn) >> 12) & 7)
 
diff --git a/xen/arch/riscv/include/asm/traps.h b/xen/arch/riscv/include/asm/traps.h
index 21fa3c3259b3..8d4ab664bca9 100644
--- a/xen/arch/riscv/include/asm/traps.h
+++ b/xen/arch/riscv/include/asm/traps.h
@@ -7,6 +7,12 @@
 
 #ifndef __ASSEMBLER__
 
+struct trap_info {
+    register_t sepc;
+    register_t scause;
+    register_t stval;
+};
+
 void do_trap(struct cpu_user_regs *cpu_regs);
 void handle_trap(void);
 void trap_init(void);
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index 9bb54d030be7..144b5beaa2c8 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -217,7 +217,7 @@ void do_trap(struct cpu_user_regs *cpu_regs)
             break;
         }
 
-        if ( fixup_exception(cpu_regs) )
+        if ( fixup_exception(cpu_regs, cause) )
             break;
 
         fallthrough;
-- 
2.55.0

[PATCH v3 16/39] xen/riscv: decouple INSN_PSEUDO_VS_* from the hypervisor's XLEN

[Oleksii Kurochko]

htinst reports a pseudoinstruction when a guest page fault is taken on an
implicit memory access done for VS-stage address translation. Four such
values are defined, differing in the access type (read or write) and in the
access width: 4 bytes (0x2000/0x2020) or 8 bytes (0x3000/0x3020).

That width is the width of a VS-stage PTE, i.e. it follows the guest's
paging mode (4 bytes for Sv32, 8 bytes for Sv39 and wider) and has nothing
to do with the XLEN Xen itself is built for. Yet only one pair is defined,
selected by Xen's XLEN and named as if it were the only one, so the 4-byte
forms, which a guest running with VSXL=32 and Sv32 in vsatp produces, can't
be named at all on RV64.

Define all four values unconditionally instead, named after the access width
they encode rather than after the build's XLEN.

Dropping the ladder loses no build-time coverage: a build for an XLEN other
than 32 or 64 already fails on the equivalent ladders in asm/asm.h and
asm/config.h, so no replacement #error is needed here. Adding one keyed on
CONFIG_RISCV_* would in any case re-introduce exactly the conflation this
patch removes.

This diverges from the imported version of riscv_encoding.h.

No functional change: the values have no user.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message: the values no longer have a user in this
   series, as htinst_is_pseudo() recognizes a pseudoinstruction by its
   bits[1:0].
 - Use VSXLEN=32/64 instead of RV32/64 in the comments of the
   corresponding definitions.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/include/asm/riscv_encoding.h | 24 +++++++--------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index 599ae3cd601a..2dbfd4d16067 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -837,25 +837,17 @@
 #define INSN_MASK_FENCE_TSO		0xffffffff
 #define INSN_MATCH_FENCE_TSO		0x8330000f
 
-#if __riscv_xlen == 64
-
-/* 64-bit read for VS-stage address translation (RV64) */
-#define INSN_PSEUDO_VS_LOAD		0x00003000
-
-/* 64-bit write for VS-stage address translation (RV64) */
-#define INSN_PSEUDO_VS_STORE	0x00003020
+/* 64-bit read for VS-stage address translation (VSXLEN=64) */
+#define INSN_PSEUDO_VS_LOAD64		0x00003000
 
-#elif __riscv_xlen == 32
+/* 64-bit write for VS-stage address translation (VSXLEN=64) */
+#define INSN_PSEUDO_VS_STORE64		0x00003020
 
-/* 32-bit read for VS-stage address translation (RV32) */
-#define INSN_PSEUDO_VS_LOAD		0x00002000
+/* 32-bit read for VS-stage address translation (VSXLEN=32) */
+#define INSN_PSEUDO_VS_LOAD32		0x00002000
 
-/* 32-bit write for VS-stage address translation (RV32) */
-#define INSN_PSEUDO_VS_STORE	0x00002020
-
-#else
-#error "Unexpected __riscv_xlen"
-#endif
+/* 32-bit write for VS-stage address translation (VSXLEN=32) */
+#define INSN_PSEUDO_VS_STORE32		0x00002020
 
 #define INSN_IS_16BIT(insn)	(((insn) & 3) != 3)
 #define INSN_IS_32BIT(insn)	(!INSN_IS_16BIT(insn) && ((insn) & 0x1c) != 0x1c)
-- 
2.55.0

[PATCH v3 17/39] xen/riscv: add guest page fault handling stub

[Oleksii Kurochko]

Add a handler for guest page faults and hook it into the trap path,
providing the trap-side entry point which will later feed the MMIO
dispatch.

This will be used, for example, to trap accesses to APLIC registers so
that a guest can initialize and drive an emulated interrupt controller.

A G-stage (stage-2) fault has one of three causes, reported via scause:
- Fetch fault: guest tried to execute from a guest-physical address
  that is unmapped or that G-stage marks non-executable. Never
  emulatable (nothing to emulate a fetch into). On real hardware
  this raises an instruction access fault, so Xen reflects the same
  fault back to the guest. Decided here.

- Load fault / Store fault: left undecided by this patch, this is
  where MMIO emulation will hook in later.

Any of these three faults can instead be reported via a pseudoinstruction
in htinst, when both:

(a) the fault occurred on an implicit access Xen made to walk a
    VS-stage page table, and
(b) htval holds a nonzero value: the guest-physical address of that
    VS-stage PTE, not of the guest's original access.

However, none of these paths consult the p2m first, and the future MMIO path
won't either: RISC-V has no populate-on-demand, no paging, and no
mem_access, so every guest mapping is established eagerly. A G-stage
fault therefore never indicates a mapping Xen could lazily resolve to
let the access complete.

resolve_faulting_gpa() is a stub which always fails for now, so any guest
page fault other than one taken on an implicit access for VS-stage
translation currently crashes the domain. That one is reflected to the
guest through trap_redirect(), which is still a BUG_ON() placeholder
though, and so takes the host down. This is no worse than before this
patch, where these traps fell through to do_unexpected_trap() and die().

Cache the "trap came from the guest" test in a local, as the guest page
fault case needs it too.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Make resolve_faulting_gpa() return an error, which the caller checks.
 - Add a FIXME about guest page tables residing in emulated MMIO.
 - Don't refer to fixup_exception() in the comment ahead of
   BUG_ON(!from_guest): it isn't consulted for guest page faults yet.
 - Mention the from_guest local in the commit message.
 - Say in the comment above resolve_faulting_gpa() that it stores the address
   in @gf->gpa.
 - Make emulate_store() take a pointer to const, as emulate_load() does:
   neither modifies *gf.
---
Changes in v2:
 - Introduce struct guest_fault.
 - Change the prototypes of emulate_{load,store}() to take a non-const
   struct guest_fault, as emulation has to write the destination
   register and advance sepc.
 - Add handling of pseudoinstructions before the call of
   emulate_{load,store}.
 - Rename get_fault_gpa to resolve_faulting_gpa and change its prototype
   to take struct guest_fault.
 - Add handling of CAUSE_FETCH_GUEST_PAGE_FAULT now.
 - Document why the p2m is not consulted before a fault is injected, and
   add a BUILD_BUG_ON() on CONFIG_VM_EVENT to catch that assumption
   breaking.
 - Print the fault cause in the domain_crash() message rather than
   deriving an access type string which cannot cover every case.
 - Move code to introduced emulate.c instead of having it in traps.c
---
---
 xen/arch/riscv/Makefile              |   1 +
 xen/arch/riscv/emulate.c             | 183 +++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/emulate.h |  10 ++
 xen/arch/riscv/include/asm/traps.h   |   3 +
 xen/arch/riscv/traps.c               |  22 +++-
 5 files changed, 218 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/riscv/emulate.c
 create mode 100644 xen/arch/riscv/include/asm/emulate.h

diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index 5454517724bd..2b84dace6bd5 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -6,6 +6,7 @@ obj-y += domain.o
 obj-y += domain-build.init.o
 obj-$(CONFIG_DOM0LESS_BOOT) += dom0less-build.init.o
 obj-$(CONFIG_EARLY_PRINTK) += early_printk.o
+obj-y += emulate.o
 obj-y += entry.o
 obj-y += extable.o
 obj-y += guestcopy.o
diff --git a/xen/arch/riscv/emulate.c b/xen/arch/riscv/emulate.c
new file mode 100644
index 000000000000..cc4b1fc54054
--- /dev/null
+++ b/xen/arch/riscv/emulate.c
@@ -0,0 +1,183 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+/*
+ * RISC-V instruction emulation for trapped guest accesses
+ */
+
+#include <xen/bug.h>
+#include <xen/errno.h>
+#include <xen/sched.h>
+#include <xen/types.h>
+
+#include <asm/csr.h>
+#include <asm/current.h>
+#include <asm/emulate.h>
+#include <asm/riscv_encoding.h>
+#include <asm/traps.h>
+
+/*
+ * The hardware-reported details of a guest page fault, gathered once by
+ * handle_guest_page_fault() and passed down to the emulation of the faulted
+ * access.
+ */
+struct guest_fault {
+    /* The guest register state. */
+    struct cpu_user_regs *regs;
+    /* scause: a fetch, a load or a store/AMO guest page fault. */
+    unsigned long cause;
+    /*
+     * htinst: the trapped instruction in its transformed form, or one of the
+     * special values (zero, or a pseudoinstruction).
+     */
+    unsigned long htinst;
+    /* htval: as written by hardware; see resolve_faulting_gpa(). */
+    unsigned long htval;
+    /* stval: the guest virtual address of the faulting access. */
+    unsigned long stval;
+    /* The faulting guest physical address, filled by resolve_faulting_gpa(). */
+    paddr_t gpa;
+};
+
+/*
+ * Is @htinst one of the special pseudoinstruction values, reported for a guest
+ * page fault taken on an implicit memory access done for VS-stage address
+ * translation?
+ *
+ * It is enough to check only bits[1:0] as according to the spec:
+ *
+ * The value is one of the special pseudoinstructions defined later, all of
+ * which have bits 1:0 equal to 00.
+ */
+static bool htinst_is_pseudo(unsigned long htinst)
+{
+    return htinst && !(htinst & 3);
+}
+
+/*
+ * Reconstruct the guest physical address the access faulted on into @gf->gpa.
+ */
+static int resolve_faulting_gpa(struct guest_fault *gf)
+{
+    return -EOPNOTSUPP;
+}
+
+static int emulate_load(const struct guest_fault *gf)
+{
+    return -EOPNOTSUPP;
+}
+
+static int emulate_store(const struct guest_fault *gf)
+{
+    return -EOPNOTSUPP;
+}
+
+static void inject_access_fault(const struct guest_fault *gf)
+{
+    struct trap_info utrap = {
+        .sepc = gf->regs->sepc,
+        .stval = gf->stval,
+    };
+
+    switch ( gf->cause )
+    {
+    case CAUSE_FETCH_GUEST_PAGE_FAULT:
+        utrap.scause = CAUSE_FETCH_ACCESS;
+        break;
+
+    case CAUSE_LOAD_GUEST_PAGE_FAULT:
+        utrap.scause = CAUSE_LOAD_ACCESS;
+        break;
+
+    case CAUSE_STORE_GUEST_PAGE_FAULT:
+        utrap.scause = CAUSE_STORE_ACCESS;
+        break;
+
+    default:
+        domain_crash(current->domain, "Impossible cause (%#lx) in %s?\n",
+                     gf->cause, __func__);
+        return;
+    }
+
+    trap_redirect(&utrap);
+}
+
+void handle_guest_page_fault(struct cpu_user_regs *regs, unsigned long cause)
+{
+    struct guest_fault gf = {
+        .regs = regs,
+        .cause = cause,
+        .gpa = INVALID_PADDR,
+    };
+    int rc;
+
+    gf.htinst = csr_read(CSR_HTINST);
+    gf.htval = csr_read(CSR_HTVAL);
+    gf.stval = csr_read(CSR_STVAL);
+
+    /*
+     * A guest-page fault may arise due to an implicit memory access during
+     * first-stage (VS-stage) address translation, in which case a guest
+     * physical address written to htval is that of the implicit memory
+     * access that faulted - for example, the address of a VS-level page
+     * table entry that could not be read. (The guest physical address
+     * corresponding to the original virtual address is unknown when
+     * VS-stage translation fails to complete)
+     *
+     * In such cases htinst reports one of the pseudoinstructions recognized
+     * by htinst_is_pseudo(), and the fault requires separate handling (since
+     * G-stage translation failed on an unpopulated/unmapped guest physical
+     * address during a hardware page-table walk). To match bare hardware
+     * behavior, we must inject an access fault of the ORIGINAL access type
+     * (Instruction, Load, or Store/AMO) that initiated the address
+     * translation.
+     *
+     * FIXME: This assumes that guest page tables never reside in an emulated
+     * MMIO region, i.e. that an implicit access faulting at G-stage always
+     * targets an unpopulated GPA. Guests may (even transiently) place page
+     * tables in MMIO-backed memory, e.g. a video frame buffer. Supporting
+     * that would require walking the VS-stage page tables in software,
+     * accessing the PTEs (including A/D updates) through the MMIO handlers,
+     * and then emulating the original access, instead of injecting a fault.
+     */
+    if ( htinst_is_pseudo(gf.htinst) )
+    {
+        inject_access_fault(&gf);
+
+        return;
+    }
+
+    rc = resolve_faulting_gpa(&gf);
+    if ( rc )
+        goto out;
+
+    switch ( cause )
+    {
+    case CAUSE_LOAD_GUEST_PAGE_FAULT:
+        rc = emulate_load(&gf);
+        break;
+
+    case CAUSE_STORE_GUEST_PAGE_FAULT:
+        rc = emulate_store(&gf);
+        break;
+
+    case CAUSE_FETCH_GUEST_PAGE_FAULT:
+        /*
+         * Guest is trying to reach unmapped/unpopulated, or G-stage PTE doesn't
+         * allow execution (X=0). Generate fetch fault in this case.
+         */
+        inject_access_fault(&gf);
+        return;
+
+    default:
+        ASSERT_UNREACHABLE();
+        rc = -EOPNOTSUPP;
+
+        break;
+    }
+
+ out:
+    if ( rc )
+        domain_crash(current->domain,
+                     "%s: unable to handle guest page fault (cause=%#lx) at gpa %#"PRIpaddr"\n",
+                     __func__, cause, gf.gpa);
+}
diff --git a/xen/arch/riscv/include/asm/emulate.h b/xen/arch/riscv/include/asm/emulate.h
new file mode 100644
index 000000000000..59e69ca6794c
--- /dev/null
+++ b/xen/arch/riscv/include/asm/emulate.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef RISCV_EMULATE_H
+#define RISCV_EMULATE_H
+
+struct cpu_user_regs;
+
+void handle_guest_page_fault(struct cpu_user_regs *regs, unsigned long cause);
+
+#endif /* RISCV_EMULATE_H */
diff --git a/xen/arch/riscv/include/asm/traps.h b/xen/arch/riscv/include/asm/traps.h
index 8d4ab664bca9..38c6423742e0 100644
--- a/xen/arch/riscv/include/asm/traps.h
+++ b/xen/arch/riscv/include/asm/traps.h
@@ -17,6 +17,9 @@ void do_trap(struct cpu_user_regs *cpu_regs);
 void handle_trap(void);
 void trap_init(void);
 
+/* Reflect @trap back to the guest, i.e. enter its VS-mode trap handler. */
+void trap_redirect(const struct trap_info *trap);
+
 #endif /* __ASSEMBLER__ */
 
 #endif /* ASM__RISCV__TRAPS_H */
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index 144b5beaa2c8..4b9ec246e4c9 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -14,6 +14,7 @@
 
 #include <asm/extable.h>
 #include <asm/cpufeature.h>
+#include <asm/emulate.h>
 #include <asm/intc.h>
 #include <asm/processor.h>
 #include <asm/riscv_encoding.h>
@@ -193,6 +194,7 @@ void do_trap(struct cpu_user_regs *cpu_regs)
 {
     register_t pc = cpu_regs->sepc;
     unsigned long cause = csr_read(CSR_SCAUSE);
+    bool from_guest = cpu_regs->hstatus & HSTATUS_SPV;
 
     switch ( cause )
     {
@@ -203,6 +205,18 @@ void do_trap(struct cpu_user_regs *cpu_regs)
         vsbi_handle_ecall(cpu_regs);
         break;
 
+    case CAUSE_FETCH_GUEST_PAGE_FAULT:
+    case CAUSE_LOAD_GUEST_PAGE_FAULT:
+    case CAUSE_STORE_GUEST_PAGE_FAULT:
+        /*
+         * Xen doesn't access guest memory, so it can't take a guest page
+         * fault itself: only a guest can get here.
+         */
+        BUG_ON(!from_guest);
+
+        handle_guest_page_fault(cpu_regs, cause);
+        break;
+
     case CAUSE_ILLEGAL_INSTRUCTION:
         if ( do_bug_frame(cpu_regs, pc) >= 0 )
         {
@@ -251,7 +265,7 @@ void do_trap(struct cpu_user_regs *cpu_regs)
         break;
     }
 
-    if ( cpu_regs->hstatus & HSTATUS_SPV )
+    if ( from_guest )
         check_for_pcpu_work();
 }
 
@@ -275,3 +289,9 @@ enum mc_disposition arch_do_multicall_call(struct mc_state *state)
     BUG_ON("unimplemented");
     return mc_continue;
 }
+
+/* Redirect trap to Guest. */
+void trap_redirect(const struct trap_info *trap)
+{
+    BUG_ON("unimplemented");
+}
-- 
2.55.0

[PATCH v3 18/39] xen/riscv: implement trap redirection to a guest

[Oleksii Kurochko]

Two kinds of traps can't or shouldn't be handled by the hypervisor and
have to be reflected to the guest's own S-mode trap handler instead:
- Traps Xen takes on the guest's behalf, such as those taken by the
  hlv/hlvx sequences Xen uses to access guest memory.
- Access faults handle_guest_page_fault() injects for a guest-page
  fault that can never become an emulated access.

Implement trap_redirect(), until now a BUG_ON() placeholder, for that
purpose. It makes the trap appear to the guest as if it had been taken
directly in VS-mode: the trap information is transferred to the guest's
virtual supervisor CSRs and the vCPU is resumed at its exception vector in
supervisor mode, following the trap entry rules of the RISC-V privileged
specification.

The implementation is based on kvm_riscv_vcpu_trap_redirect() from Linux,
with a few deviations:
 - The function reads and writes physical VS-mode CSRs, so it is only
   meaningful for the currently running vCPU. Instead of taking a
   struct vcpu argument, it always operates on current.
 - The MODE field of vstvec is masked off explicitly when computing the
   exception target PC (exceptions always vector to BASE), rather than
   relying on the hardwired zero bit of sepc to drop it on VM entry.
 - Assertions document the preconditions: the trap must have been taken
   from virtualized mode (hstatus.SPV set), and only synchronous
   exceptions may be redirected - interrupts must be injected via hvip
   instead, so that the hardware performs VS-mode trap entry itself,
   respecting vsstatus.SIE and vectored vstvec dispatch.

STVEC_MODE_DIRECT and STVEC_MODE_VECTORED are unused for now and are
added only for completeness alongside STVEC_MODE_MASK.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Drop STVEC_BASE_MASK for now and use ~STVEC_MODE_MASK instead.
---
Changes in v2:
 - Add new defines STVEC_*. The STVEC_MODE_DIRECT/_VECTORED values are
   currently unused and were included because riscv encoding header is a
   spec mirror full of unused encodings.
 - Use STVEC_BASE_MASK instead of open-coding it.
 - Rename riscv_vcpu_trap_redirect() to trap_redirect(): unlike its KVM
   counterpart the function takes no vCPU argument, it implicitly operates
   on current, so "vcpu" in the name describes nothing.
---
 xen/arch/riscv/include/asm/riscv_encoding.h |  5 +++
 xen/arch/riscv/traps.c                      | 50 ++++++++++++++++++++-
 2 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index 2dbfd4d16067..b01da95f7ece 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -107,6 +107,11 @@
 #define SIP_SSIP			MIP_SSIP
 #define SIP_STIP			MIP_STIP
 
+/* stvec/vstvec: MODE is bits [1:0], BASE is bits [XLEN-1:2] */
+#define STVEC_MODE_MASK		_UL(0x3)
+#define STVEC_MODE_DIRECT		_UL(0x0)
+#define STVEC_MODE_VECTORED		_UL(0x1)
+
 #define PRV_U				_UL(0)
 #define PRV_S				_UL(1)
 #define PRV_M				_UL(3)
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index 4b9ec246e4c9..08aae2e5280b 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -293,5 +293,53 @@ enum mc_disposition arch_do_multicall_call(struct mc_state *state)
 /* Redirect trap to Guest. */
 void trap_redirect(const struct trap_info *trap)
 {
-    BUG_ON("unimplemented");
+    struct cpu_user_regs *regs = vcpu_guest_cpu_user_regs(current);
+    unsigned long vsstatus = csr_read(CSR_VSSTATUS);
+
+    /*
+     * Redirecting a trap makes sense only if the trap was taken from
+     * virtualized mode, i.e. sret is going to return to VS-mode.
+     */
+    ASSERT(regs->hstatus & HSTATUS_SPV);
+
+    /*
+     * Only synchronous exceptions can be redirected. Interrupts must be
+     * injected via hvip instead, so that the hardware itself performs
+     * VS-mode trap entry, respecting vsstatus.SIE and the vectored
+     * dispatch (BASE + 4 * cause) if vstvec is configured so.
+     */
+    ASSERT(!(trap->scause & CAUSE_IRQ_FLAG));
+
+    /* Change Guest SSTATUS.SPP bit */
+    vsstatus &= ~SSTATUS_SPP;
+    if ( regs->sstatus & SSTATUS_SPP )
+        vsstatus |= SSTATUS_SPP;
+
+    /* Change Guest SSTATUS.SPIE bit */
+    vsstatus &= ~SSTATUS_SPIE;
+    if ( vsstatus & SSTATUS_SIE )
+        vsstatus |= SSTATUS_SPIE;
+
+    /* Clear Guest SSTATUS.SIE bit */
+    vsstatus &= ~SSTATUS_SIE;
+
+    /* Update Guest SSTATUS */
+    csr_write(CSR_VSSTATUS, vsstatus);
+
+    /* Update Guest SCAUSE, STVAL, and SEPC */
+    csr_write(CSR_VSCAUSE, trap->scause);
+    csr_write(CSR_VSTVAL, trap->stval);
+    csr_write(CSR_VSEPC, trap->sepc);
+
+    /*
+     * Set Guest PC to Guest exception vector.
+     *
+     * vstvec's MODE field is not part of the address. Exceptions always
+     * target BASE regardless of MODE, so mask it off explicitly instead of
+     * relying on the hardwired zero bit of sepc to drop it.
+     */
+    regs->sepc = csr_read(CSR_VSTVEC) & ~STVEC_MODE_MASK;
+
+    /* Set Guest privilege mode to supervisor */
+    regs->sstatus |= SSTATUS_SPP;
 }
-- 
2.55.0

[PATCH v3 19/39] xen/riscv: detect Shtvala

[Oleksii Kurochko]

The H extension allows htval, on a guest-page fault, to be written with
either the faulting guest physical address or zero. The Shtvala extension
removes the ambiguity of this zero-write by guaranteeing that htval is
written with the faulting guest physical address in every circumstance
permitted by the ISA.

Not offered to guests: Shtvala describes htval, an HS-mode-only trap
register a VS-mode guest never touches, and the H extension it belongs to
is already hidden from guests. The guest-visible equivalent is a
separate extension, Shvstvala, covering vstval instead.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/cpufeature.c             | 1 +
 xen/arch/riscv/include/asm/cpufeature.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/xen/arch/riscv/cpufeature.c b/xen/arch/riscv/cpufeature.c
index 2d12dffae72e..c7f77d917cf5 100644
--- a/xen/arch/riscv/cpufeature.c
+++ b/xen/arch/riscv/cpufeature.c
@@ -195,6 +195,7 @@ static const struct riscv_isa_ext_entry __initconstrel riscv_isa_ext[] = {
     RISCV_ISA_EXT_ENTRY(zba,            ANY),
     RISCV_ISA_EXT_ENTRY(zbb,            ANY),
     RISCV_ISA_EXT_ENTRY(zbs,            ANY),
+    RISCV_ISA_EXT_ENTRY(shtvala,        NONE),
     RISCV_ISA_EXT_ENTRY(smaia,          ANY),
     RISCV_ISA_EXT_ENTRY(smstateen,      ANY),
     RISCV_ISA_EXT_ENTRY(ssaia,          ANY),
diff --git a/xen/arch/riscv/include/asm/cpufeature.h b/xen/arch/riscv/include/asm/cpufeature.h
index 2973eb13a513..ac8c68007072 100644
--- a/xen/arch/riscv/include/asm/cpufeature.h
+++ b/xen/arch/riscv/include/asm/cpufeature.h
@@ -36,6 +36,7 @@ enum riscv_isa_ext_id {
     RISCV_ISA_EXT_zba,
     RISCV_ISA_EXT_zbb,
     RISCV_ISA_EXT_zbs,
+    RISCV_ISA_EXT_shtvala,
     RISCV_ISA_EXT_smaia,
     RISCV_ISA_EXT_smstateen,
     RISCV_ISA_EXT_ssaia,
-- 
2.55.0

[PATCH v3 20/39] xen/riscv: resolve the faulting guest physical address

[Oleksii Kurochko]

Recover the guest physical address from htval and stval. On a guest-page
fault to hypervisor, htval holds the guest physical address shifted
right by 2, so that an address wider than HSXLEN fits, and stval holds
the faulting guest virtual address. The shift is done on paddr_t rather
than on the raw register: a guest physical address is 34 bits wide on
RV32 with Sv32x4, so shifting an XLEN-wide value would drop its top two
bits.

However, there are two cases to distinguish when recovering the
faulting GPA:
- Explicit memory access: the two least significant bits of stval are
  the same as those of the guest physical address.
- Implicit memory access for VS-stage translation: the two least
  significant bits of the guest physical address (GPA[1:0]) are zero.
These two cases can be distinguished using the value provided in
register htinst.

stval needs no check against an ISA extension: a guest-page fault writes it
with the faulting guest virtual address regardless. Sstvala would not be the
right thing to test for either (it covers stval across every trap type
which writes it, a wider guarantee than what is needed here).

htval does need one. The H extension lets an implementation write it with
either the faulting address or zero, so without Shtvala a zero htval cannot
be told apart from a genuine fault on guest physical address 0-3, and the
address has to be recovered by decoding the access and walking the VS-stage
page tables in software instead. That is left as a TODO; until it is
written, such a fault crashes the domain rather than acting on an address
which may not be the one which faulted.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Drop panic() inside resolve_faulting_gpa() and return -EOPNOTSUPP +
   printk_once().
 - Leave the comment above resolve_faulting_gpa() alone: it already says
   what the function does since "xen/riscv: add guest page fault handling
   stub".
 - Document in docs/misc/riscv/booting.txt why Shtvala is recommended.
---
Changes in v2:
 - New patch.
---
---
 docs/misc/riscv/booting.txt | 11 +++++++++++
 xen/arch/riscv/emulate.c    | 27 ++++++++++++++++++++++++++-
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/docs/misc/riscv/booting.txt b/docs/misc/riscv/booting.txt
index c99ccafa7485..f90d8603ef57 100644
--- a/docs/misc/riscv/booting.txt
+++ b/docs/misc/riscv/booting.txt
@@ -38,3 +38,14 @@ Other extensions recognised by Xen are listed in riscv_isa_ext[] in
 xen/arch/riscv/cpufeature.c. Xen uses them when present, but they are not
 needed to boot. The second column of that table says whether the extension
 is also exposed to guests.
+
+Some of them are nevertheless recommended, as functionality depends on them:
+- Shtvala:
+  Guarantees that htval is written with the faulting guest physical address
+  on a guest-page fault. Without it an implementation may write zero instead,
+  which can't be told apart from a genuine fault on guest physical addresses
+  0-3. Xen needs that address to emulate a guest's MMIO accesses; recovering
+  it in software (by walking the guest's VS-stage page tables) isn't
+  implemented, so on such hardware a guest-page fault reported with a zero
+  htval crashes the domain. See resolve_faulting_gpa() in
+  xen/arch/riscv/emulate.c.
diff --git a/xen/arch/riscv/emulate.c b/xen/arch/riscv/emulate.c
index cc4b1fc54054..a579bc4b8a15 100644
--- a/xen/arch/riscv/emulate.c
+++ b/xen/arch/riscv/emulate.c
@@ -9,6 +9,7 @@
 #include <xen/sched.h>
 #include <xen/types.h>
 
+#include <asm/cpufeature.h>
 #include <asm/csr.h>
 #include <asm/current.h>
 #include <asm/emulate.h>
@@ -58,7 +59,31 @@ static bool htinst_is_pseudo(unsigned long htinst)
  */
 static int resolve_faulting_gpa(struct guest_fault *gf)
 {
-    return -EOPNOTSUPP;
+    /*
+     * A zero htval is either a genuine fault on guest physical address 0-3, or
+     * an implementation which does not report the address at all; only Shtvala
+     * tells the two apart.
+     *
+     * TODO: where it is absent, recover the address in software rather than
+     * giving up.
+     */
+    if ( !gf->htval &&
+         !riscv_isa_extension_available(NULL, RISCV_ISA_EXT_shtvala) )
+    {
+        printk_once(XENLOG_WARNING
+                    "Shtvala isn't supported by h/w; s/w VS-stage walk required\n");
+        return -EOPNOTSUPP;
+    }
+
+    /*
+     * htval does not carry the two low bits of the address: for an explicit
+     * access they are those of the faulting guest virtual address in stval,
+     * and for an implicit access made for VS-stage translation they are zero.
+     */
+    gf->gpa = ((paddr_t)gf->htval << 2) |
+              (htinst_is_pseudo(gf->htinst) ? 0 : (gf->stval & 3));
+
+    return 0;
 }
 
 static int emulate_load(const struct guest_fault *gf)
-- 
2.55.0

[PATCH v3 21/39] xen/riscv: add guest memory read helper

[Oleksii Kurochko]

Introduce read_guest() to allow Xen to safely read guest memory using
HLV/HLVX instructions while reliably capturing trap context.

This is required for instruction fetch emulation and MMIO decoding, where
Xen must inspect guest memory that may not be directly accessible and may
fault.

The implementation is based on kvm_riscv_vcpu_unpriv_read() from Linux,
with one deviation: the hlv/hlvx instructions translate the guest address
through the live vsatp/hgatp CSRs, i.e. through the address space of the
currently running vCPU, so the function can only be called safely for
current. Instead of taking a struct vcpu argument, it always operates on
current directly.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Rename riscv_read_guest() to read_guest().
 - Document that @trap must be zero-initialised by the caller: it is
   written only when an access faults, so trap->scause == 0 on return is
   what tells the caller that the read succeeded.
 - Drop the ASSERT() on hstatus.SPV: vcpu_guest_cpu_user_regs(current)
   is the guest frame whatever context Xen is in, so SPV is always set in
   it and the check couldn't catch a call made from a nested trap. State
   in the comment above read_guest() that it must not be called from a
   nested trap taken in HS-mode instead.
 - Declare read_guest() in asm/guest_access.h.
---
Changes in v2:
 - Rename riscv_vcpu_unpriv_read() to riscv_read_guest() and make the guest
   address the first parameter. "unprivileged" described how hlv/hlvx perform
   the access rather than what the helper is for, and "unprivileged guest"
   reads as a synonym for DomU although the helper works for any domain.
 - Drop the hstatus save/restore and the local_irq_save() protecting it:
   hstatus already belongs to the vCPU which trapped, as Xen never installs
   a value of its own and does not reschedule before returning to the guest.
   ASSERT() hstatus.SPV in the saved copy instead, which also documents that
   the helper is only usable while handling a trap from a guest.
 - Poison val with ~0UL and make [val] "+&r": the fixup for the first access
   resumes past the loads without writing it, so a caller which checks
   trap->scause is no longer handed an uninitialized value.
 - Describe the trap_info write with a "+m" (*trap) operand instead of a
   "memory" clobber; the exception handler writes that structure and nothing
   else.
 - Combine the two halfwords with slli/or instead of sll/add, and use bnez
   for the instruction length check.
 - Turn the hlv.d/hlv.w selection into #if/#elif with an #error default,
   rather than silently using hlv.w for anything that is not RV64.
 - Document that at most two halfwords are fetched, i.e. that encodings
   wider than 32 bits are unsupported and cannot be completed by calling the
   helper again at guest_addr + 4, since the length check would then be
   applied to a continuation halfword.
---
---
 xen/arch/riscv/guestcopy.c                | 88 +++++++++++++++++++++++
 xen/arch/riscv/include/asm/guest_access.h |  4 ++
 2 files changed, 92 insertions(+)

diff --git a/xen/arch/riscv/guestcopy.c b/xen/arch/riscv/guestcopy.c
index 8a89212e0bea..4aa5a0d967ba 100644
--- a/xen/arch/riscv/guestcopy.c
+++ b/xen/arch/riscv/guestcopy.c
@@ -6,6 +6,7 @@
 #include <xen/string.h>
 
 #include <asm/guest_access.h>
+#include <asm/traps.h>
 
 #define COPY_from_guest     0U
 #define COPY_to_guest       BIT(0, U)
@@ -114,3 +115,90 @@ unsigned long copy_to_guest_phys(struct domain *d, paddr_t gpa, void *buf,
     return copy_guest(buf, gpa, len, GPA_INFO(d),
                       COPY_to_guest | COPY_gpa);
 }
+
+/*
+ * Read machine word from guest memory
+ *
+ * @guest_addr: Guest address to read
+ * @read_insn: Flag representing whether we are reading instruction
+ * @trap: Output pointer to trap details if something went wrong during read.
+ *        It must be zero-initialised by the caller: it is written only when
+ *        an access faults, so trap->scause == 0 on return is what tells the
+ *        caller that the read succeeded.
+ *
+ * The hlv/hlvx instructions translate guest_addr through the live
+ * vsatp/hgatp CSRs, so the read is only meaningful for the address
+ * space of the currently running vCPU. It must therefore only be called
+ * while handling a trap taken from that vCPU, never from a nested trap
+ * taken in HS-mode, which may have interrupted Xen at a point where
+ * hstatus, vsatp or hgatp don't belong to the vCPU.
+ *
+ * At most two halfwords are fetched when @read_insn is true, i.e. encodings
+ * wider than 32 bits are not supported. Such an encoding cannot be completed
+ * by calling this function again at @guest_addr + 4: the length check is
+ * applied to the first halfword read, which would then be a continuation of
+ * the instruction rather than its opcode. It is up to the caller to reject
+ * anything that is neither a 16- nor a 32-bit encoding.
+ */
+unsigned long read_guest(unsigned long guest_addr, bool read_insn,
+                         struct trap_info *trap)
+{
+    /*
+     * Poison the result: if the very first access faults, the fixup skips
+     * over the loads without writing it. Callers must check trap->scause.
+     */
+    unsigned long val = ~0UL, tmp;
+
+    /*
+     * hlv/hlvx use hstatus.SPVP for the privilege of the access, and the
+     * live vsatp/hgatp for the translation. Xen never installs a value of
+     * its own in hstatus (it is only saved on trap entry and restored
+     * before sret) and it doesn't reschedule before returning to the
+     * guest, so all three still belong to the vCPU which trapped.
+     */
+    if ( read_insn )
+    {
+        asm volatile ( "\n"
+            "1: hlvx.hu %[val], (%[addr])\n"
+            ASM_EXTABLE_TRAP_INFO(1b, 3f, %[ti])
+            "   andi %[tmp], %[val], 3\n"
+            "   addi %[tmp], %[tmp], -3\n"
+            "   bnez %[tmp], 3f\n"
+            "   addi %[addr], %[addr], 2\n"
+            "\n"
+            "2: hlvx.hu %[tmp], (%[addr])\n"
+            ASM_EXTABLE_TRAP_INFO(2b, 3f, %[ti])
+            "   slli %[tmp], %[tmp], 16\n"
+            "   or %[val], %[val], %[tmp]\n"
+            "3:\n"
+        : [val] "+&r" (val), [tmp] "=&r" (tmp), [addr] "+&r" (guest_addr),
+          "+m" (*trap)
+        : [ti] "r" (trap) );
+
+        /*
+         * Although HLVX instructions' explicit memory accesses require execute
+         * permissions, they still raise the same exceptions as other load
+         * instructions, rather than raising fetch exceptions instead.
+         */
+        if ( trap->scause == CAUSE_LOAD_PAGE_FAULT )
+            trap->scause = CAUSE_FETCH_PAGE_FAULT;
+    }
+    else
+    {
+        asm volatile ( "\n"
+            "1: "
+#if defined(CONFIG_RISCV_64)
+            "hlv.d %[val], (%[addr])\n"
+#elif defined(CONFIG_RISCV_32)
+            "hlv.w %[val], (%[addr])\n"
+#else
+# error "unsupported RISC-V variant: no hlv for a machine word"
+#endif
+            "2:\n"
+            ASM_EXTABLE_TRAP_INFO(1b, 2b, %[ti])
+        : [val] "+&r" (val), "+m" (*trap)
+        : [addr] "r" (guest_addr), [ti] "r" (trap) );
+    }
+
+    return val;
+}
diff --git a/xen/arch/riscv/include/asm/guest_access.h b/xen/arch/riscv/include/asm/guest_access.h
index 8d679319ded0..41c6ede4baae 100644
--- a/xen/arch/riscv/include/asm/guest_access.h
+++ b/xen/arch/riscv/include/asm/guest_access.h
@@ -5,6 +5,7 @@
 #include <xen/types.h>
 
 struct domain;
+struct trap_info;
 
 unsigned long raw_copy_to_guest(void *to, const void *from, unsigned len);
 unsigned long raw_copy_from_guest(void *to, const void *from, unsigned len);
@@ -25,6 +26,9 @@ unsigned long raw_clear_guest(void *to, unsigned int len);
 unsigned long copy_to_guest_phys(struct domain *d, paddr_t gpa, void *buf,
                                  unsigned long len);
 
+unsigned long read_guest(unsigned long guest_addr, bool read_insn,
+                         struct trap_info *trap);
+
 #endif /* ASM__RISCV__GUEST_ACCESS_H */
 /*
  * Local variables:
-- 
2.55.0

[PATCH v3 22/39] xen/riscv: look up the exception table for any trap taken in Xen context

[Oleksii Kurochko]

do_trap() consulted the exception table only for CAUSE_ILLEGAL_INSTRUCTION,
which covers csr_read_safe() but not the hlv/hlvx sequences reading guest
memory: those fault with load/store (guest) page fault causes and would
reach do_unexpected_trap() instead of their fixup.

Move the lookup ahead of the cause switch, and gate it on the trap having
been taken in Xen context and not being an interrupt:

- sepc of a trap taken from the guest is a guest VA/PA, which the
  guest can point at an address listed in the exception table; Xen would
  then act on that entry and, for EX_TYPE_TRAP_INFO, write through a
  pointer fully under guest control. Entries are matched by exact address,
  so this needs no more than a numerical collision.

- an interrupt taken at an address listed in the table would otherwise be
  "fixed up" as if the access itself had faulted, silently skipping it and
  handing the caller the interrupt's scause as a fault cause.

Returning early skips check_for_pcpu_work(), which is correct: that only
runs for traps taken from the guest.

With that in place a G-stage fault reaching the switch can no longer have
been caused by an hlv/hlvx covered by an entry, so anything left must have
come from the guest. Say so in the comment ahead of the BUG_ON(!from_guest)
in the guest page fault case.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the comment ahead of BUG_ON(!from_guest) in the guest page fault
   case to refer to the fixup_exception() call added here.
 - Drop "assert as much" and the paragraph about caching the "trap came from
   the guest" test from the commit message: the BUG_ON() and the local are
   introduced by "xen/riscv: add guest page fault handling stub".
 - Refer to read_guest() instead of riscv_read_guest() in the comment in
   do_trap(), following the rename of the helper.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/traps.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index 08aae2e5280b..dc938bae5b00 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -196,11 +196,34 @@ void do_trap(struct cpu_user_regs *cpu_regs)
     unsigned long cause = csr_read(CSR_SCAUSE);
     bool from_guest = cpu_regs->hstatus & HSTATUS_SPV;
 
+    /*
+     * A synchronous trap taken in Xen context may come from an access done on
+     * a vCPU's behalf, e.g. the hlv/hlvx sequences in read_guest(),
+     * or from a probing access like csr_read_safe(). Both are covered by
+     * exception table entries which record the fault details for the caller
+     * and resume execution past the faulting instruction.
+     *
+     * Traps taken from the guest must never be fixed up: sepc is then a guest
+     * address, which the guest could point at an address listed in the
+     * exception table, making Xen act on an entry (and, for EX_TYPE_TRAP_INFO,
+     * write through a pointer) fully under guest control.
+     *
+     * Interrupts must be excluded too: one taken at an address which happens
+     * to be listed in the exception table would otherwise be "fixed up" as if
+     * the access itself had faulted, silently skipping it.
+     *
+     * Returning early skips check_for_pcpu_work() below, which is correct:
+     * that only runs for traps taken from the guest.
+     */
+    if ( !from_guest && !(cause & CAUSE_IRQ_FLAG) &&
+         fixup_exception(cpu_regs, cause) )
+        return;
+
     switch ( cause )
     {
     case CAUSE_VIRTUAL_SUPERVISOR_ECALL:
         /* CAUSE_VIRTUAL_SUPERVISOR_ECALL should come from VS-mode */
-        BUG_ON(!(cpu_regs->hstatus & HSTATUS_SPV));
+        BUG_ON(!from_guest);
 
         vsbi_handle_ecall(cpu_regs);
         break;
@@ -209,8 +232,9 @@ void do_trap(struct cpu_user_regs *cpu_regs)
     case CAUSE_LOAD_GUEST_PAGE_FAULT:
     case CAUSE_STORE_GUEST_PAGE_FAULT:
         /*
-         * Xen doesn't access guest memory, so it can't take a guest page
-         * fault itself: only a guest can get here.
+         * A guest page fault taken in Xen context comes from an hlv/hlvx
+         * access made on a vCPU's behalf and is dealt with by the
+         * fixup_exception() above, so only a guest can get here.
          */
         BUG_ON(!from_guest);
 
@@ -231,9 +255,6 @@ void do_trap(struct cpu_user_regs *cpu_regs)
             break;
         }
 
-        if ( fixup_exception(cpu_regs, cause) )
-            break;
-
         fallthrough;
     default:
         if ( cause & CAUSE_IRQ_FLAG )
-- 
2.55.0

[PATCH v3 23/39] xen/riscv: add helpers for decoding a trapped load or store

[Oleksii Kurochko]

emulate_load() and emulate_store() will both need to obtain the
instruction which caused a guest MMIO trap, decode it, and locate the
register operand it names. Add what the two share, ahead of either of
them being implemented: struct decoded_insn, insn_fetch_faulted(),
decode_ldst_insn(), guest_xlen(), guest_gpr() and advance_pc(), together
with the XLEN_FIELD_*, INSN_OPCODE_* and RV_RD()/RV_RS2() definitions
they need.

Nothing calls any of this yet, so reference the functions from the
emulate_load() stub to keep the build going; the references go away once
emulate_load() gains its body later.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
How these helpers are used can be seen in the patches adding emulate_load()
and emulate_store().
---
Changes in v3:
 - Update the commit message.
 - Restore the opcode bits of a transformed instruction with 3 rather
   than INSN_16BIT_MASK: the macro is dropped by "xen/riscv: drop
   INSN_{16,32}BIT_MASK and report unknown instruction lengths", and its
   name didn't fit this use anyway.
 - Use (void) references in emulate_load() instead of __maybe_unused.
 - Use read_guest(), the new name of riscv_read_guest().
 - Shrink struct decoded_insn from 24 to 8 bytes (RV64): insn becomes
   uint32_t, as no instruction is wider than 32 bits, insn_len, len and reg
   become uint8_t (rather than bit-fields, for better code generation), and
   is_write and is_unsigned become bit-fields.
 - Don't take a custom htinst value with bits above bit[31] set for a
   transformed instruction (truncating it could yield a valid load/store
   encoding); fetch the instruction from guest memory instead.
 - Drop is_load_guest_page_fault() and open-code the check at its only
   user: with no store counterpart, such a helper is fragile.
 - Make guest_xlen() RV128-compatible by checking (xl > XLEN_FIELD_32)
   instead of (xl == XLEN_FIELD_64) before consulting vsstatus.UXL.
 - Clarify in struct decoded_insn that insn may hold the 32-bit transformed
   equivalent of the trapped instruction, and that insn_len is the length of
   the trapped instruction in guest memory rather than of the encoding in
   insn.
 - Say that none of the extensions exposed to guests has instructions wider
   than 32 bits, instead of referring to what the ISA defines, which is at
   risk of going stale.
 - Document in guest_gpr() that register bits above the guest's XLEN are not
   guaranteed to be zero or a sign extension, so callers must not rely on
   them when reading and must sign-extend what they write.
 - Explain why utrap.sepc can't be set in the initializer: on a fault
   read_guest() overwrites it with the address of its own faulting access.
 - Mention that stval, being XLEN bits wide, may not be able to hold a wider
   instruction as another reason for leaving it zero.
 - Calculate insn_len ahead of the width check and use it there, instead of
   testing INSN_IS_16BIT() and INSN_IS_32BIT() separately.
 - Drop the resets of is_write and is_unsigned in decode_ldst_insn() and
   state that @di is expected to start out zeroed instead.
 - Report an illegal instruction for a compressed encoding read back from
   guest memory when C isn't exposed to the guest.
 - Have guest_gpr() take the decoded instruction rather than a bare register
   number, and assert there that the access width fits a guest register.
 - Decode the access width from funct3 (from bits[15:13] for the
   compressed forms), which the loads and stores encode uniformly, instead
   of matching the instruction against each of them in turn. Introduce
   INSN_OPCODE_{MASK,LOAD,STORE} for that. Reject by a single check what
   the guest's XLEN doesn't allow, i.e. an access wider than a guest
   register, which neither the value an emulated access moves nor the
   register operand guest_gpr() hands out could carry.
 - Say which field of the compressed forms holds the register operand:
   RVC_RS2S() gives rd' of a load as well as rs2' of a store.
 - guest_gpr(): resolve the register via regs_gpr_ptr() from asm/regs.h
   instead of REG_PTR(), sharing its bounds check.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/emulate.c                    | 336 ++++++++++++++++++++
 xen/arch/riscv/include/asm/riscv_encoding.h |  14 +
 2 files changed, 350 insertions(+)

diff --git a/xen/arch/riscv/emulate.c b/xen/arch/riscv/emulate.c
index a579bc4b8a15..2695a3277605 100644
--- a/xen/arch/riscv/emulate.c
+++ b/xen/arch/riscv/emulate.c
@@ -13,9 +13,40 @@
 #include <asm/csr.h>
 #include <asm/current.h>
 #include <asm/emulate.h>
+#include <asm/guest_access.h>
+#include <asm/processor.h>
+#include <asm/regs.h>
 #include <asm/riscv_encoding.h>
 #include <asm/traps.h>
 
+/*
+ * Determine the trapped load or store instruction which caused a guest MMIO
+ * trap.
+ */
+struct decoded_insn {
+    /*
+     * The trapped instruction: as read from guest memory, or the 32-bit
+     * equivalent the hardware transformed it into (see insn_fetch_faulted()).
+     * None of the extensions exposed to guests has instructions wider than
+     * 32 bits, and insn_fetch_faulted() rejects anything longer.
+     */
+    uint32_t insn;
+    /*
+     * Length in bytes of the trapped instruction in guest memory: 2 or 4.
+     * This need not be the length of the encoding in insn, as a compressed
+     * instruction may have been transformed into its 32-bit equivalent.
+     */
+    uint8_t insn_len;
+    /* Width of the memory access, in bytes. */
+    uint8_t len;
+    /* Number of the register operand: rd for a load, rs2 for a store. */
+    uint8_t reg;
+    /* The access is a store rather than a load. */
+    bool is_write:1;
+    /* The load zero-extends its result rather than sign-extending it. */
+    bool is_unsigned:1;
+};
+
 /*
  * The hardware-reported details of a guest page fault, gathered once by
  * handle_guest_page_fault() and passed down to the emulation of the faulted
@@ -39,6 +70,65 @@ struct guest_fault {
     paddr_t gpa;
 };
 
+static void advance_pc(struct cpu_user_regs *regs, unsigned int step)
+{
+    regs->sepc += step;
+}
+
+/*
+ * The effective XLEN of the guest at the point of the trap: hstatus.VSXL for a
+ * trap taken from VS-mode, vsstatus.UXL for one taken from VU-mode.
+ *
+ * VSXL is consulted whichever mode the trap came from, as it also gives the
+ * width of vsstatus itself: where VSXL says 32, that register has no UXL field
+ * to consult and VU-mode is 32-bit as well, there being nothing to configure.
+ *
+ * It is needed to decode a trapped instruction: the encodings which exist only
+ * for XLEN=64 must not be recognized for a 32-bit guest. Besides those simply
+ * being reserved there, the compressed ones are ambiguous: C.LD and C.FLW
+ * share the encoding 0x6000 (mask 0xe003), and likewise C.SD/C.FSW,
+ * C.LDSP/C.FLWSP and C.SDSP/C.FSWSP.
+ *
+ * IS_ENABLED() can't be used here as HSTATUS_VSXL is defined for
+ * __riscv_xlen == 64 only, the field not existing on RV32 in the first place.
+ */
+static unsigned int guest_xlen(const struct cpu_user_regs *regs)
+{
+#ifdef CONFIG_RISCV_32
+    return 32;
+#else
+    unsigned long xl = MASK_EXTR(regs->hstatus, HSTATUS_VSXL);
+
+    if ( (xl > XLEN_FIELD_32) && !(regs->sstatus & SSTATUS_SPP) )
+        xl = MASK_EXTR(csr_read(CSR_VSSTATUS), SSTATUS64_UXL);
+
+    switch ( xl )
+    {
+    case XLEN_FIELD_32:
+        return 32;
+
+    case XLEN_FIELD_64:
+        return 64;
+
+    default:
+        /*
+         * The field holds nothing else in practice: XLEN_FIELD_128 would mean
+         * RV128, which no implementation provides, and the only value left is
+         * reserved. ASSERT_UNREACHABLE() being debug-only, a width still has
+         * to be answered in release builds.
+         *
+         * Answer 32, that being the safe way to be wrong: the decoder then
+         * fails to recognize the RV64-only encodings and emulation gives up.
+         * Answering 64 for what may well be a 32-bit guest would instead have
+         * it take C.FLW for C.LD and C.FSW for C.SD (see above), i.e. quietly
+         * emulate an access of the wrong width against the wrong register.
+         */
+        ASSERT_UNREACHABLE();
+        return 32;
+    }
+#endif
+}
+
 /*
  * Is @htinst one of the special pseudoinstruction values, reported for a guest
  * page fault taken on an implicit memory access done for VS-stage address
@@ -86,8 +176,254 @@ static int resolve_faulting_gpa(struct guest_fault *gf)
     return 0;
 }
 
+/*
+ * Where the value of a decoded instruction's register operand is held.
+ *
+ * The register is held at its full width, whatever the guest's XLEN is (see
+ * guest_xlen()). Where XLEN is narrower, the bits above it are not guaranteed
+ * to be zero, nor even a sign extension: hardware only ignores them in source
+ * operands, so they may have been left there by more privileged code running
+ * at a wider XLEN. Hence callers must not rely on those bits when reading a
+ * value, and must sign-extend what they write from bit XLEN-1, as hardware
+ * does for the result of an operation.
+ */
+static unsigned long *guest_gpr(struct cpu_user_regs *regs,
+                                const struct decoded_insn *di)
+{
+    /*
+     * decode_ldst_insn() is what fills @di in, and it rejects an access
+     * wider than a guest register.
+     */
+    ASSERT(di->len <= (guest_xlen(regs) / BITS_PER_BYTE));
+
+    return regs_gpr_ptr(regs, di->reg);
+}
+
+/*
+ * Obtain the instruction which caused a guest MMIO trap, filling in
+ * @di->insn and @di->insn_len. It either comes transformed in htinst, or has
+ * to be fetched from guest memory.
+ *
+ * Returns true if the fetch faulted in turn; the resulting trap has then
+ * already been redirected to the guest and there is nothing further for the
+ * caller to do. Where it returns false, @di has been filled in and emulation
+ * is to continue.
+ */
+static bool insn_fetch_faulted(const struct guest_fault *gf,
+                               struct decoded_insn *di)
+{
+    unsigned long htinst = gf->htinst;
+
+    /*
+     * A pseudoinstruction says nothing about the instruction the guest was
+     * executing, and comes with a guest physical address which isn't the one
+     * that instruction accessed. handle_guest_page_fault() deals with such a
+     * fault on its own, so no emulation can ever start for one.
+     */
+    ASSERT(!htinst_is_pseudo(htinst));
+
+    /*
+     * Bit[0] == 1 implies trapped instruction value is transformed instruction
+     * or custom value. A transformed instruction is a 32-bit encoding, so a
+     * value with any bit above bit[31] set is a custom one, which is of no use
+     * for decoding: fetch the instruction from guest memory then, as for zero.
+     */
+    if ( (htinst & BIT(0, UL)) && (htinst == (uint32_t)htinst) )
+    {
+        /*
+         * The transformation always yields the 32-bit format, with bits[1:0]
+         * holding a marker instead of the original opcode bits: bit[0] set to
+         * flag the transformation, bit[1] clear if the trapped instruction
+         * was a compressed one. Restoring the opcode bits makes the value the
+         * valid 32-bit encoding decode_ldst_insn() decodes. Its handling of
+         * compressed encodings exists for the branch below, where a
+         * compressed instruction is read from guest memory as is: a trapped
+         * one arrives here already expanded to its 32-bit equivalent, and the
+         * opcode bits just restored have it decoded as such.
+         *
+         * The length then cannot come from the value anymore, only from
+         * bit[1]. And only a 16- or a 32-bit instruction is ever reported
+         * this way: the standard load and store instructions the hardware
+         * transforms are all of one of these two lengths, anything else comes
+         * as the zero special value handled below.
+         */
+        di->insn = htinst | 3;
+        di->insn_len = (htinst & BIT(1, UL)) ? 4 : 2;
+    }
+    else
+    {
+        const struct cpu_user_regs *regs = gf->regs;
+        struct trap_info utrap = {};
+
+        /*
+         * Bit[0] == 0 implies trapped instruction value is zero or special
+         * value. With the pseudoinstructions ruled out above, only zero (or a
+         * custom value, see above) is left: the instruction has to be read from
+         * guest memory.
+         */
+
+        di->insn = read_guest(regs->sepc, true, &utrap);
+        if ( utrap.scause )
+        {
+            /*
+             * If during getting of trapped instruction a fault happen in
+             * G-stage translation then CAUSE_LOAD_GUEST_PAGE_FAULT is
+             * generated. Such faults during this operation is considered as
+             * bus error.
+             */
+            if ( utrap.scause == CAUSE_LOAD_GUEST_PAGE_FAULT )
+                utrap.scause = CAUSE_FETCH_ACCESS;
+
+            /*
+             * Not set in the initializer: on a fault read_guest() leaves in
+             * utrap.sepc the address of its own faulting access, whereas the
+             * trap is to be reported at the guest instruction.
+             */
+            utrap.sepc = regs->sepc;
+
+            trap_redirect(&utrap);
+
+            return true;
+        }
+
+        di->insn_len = INSN_LEN(di->insn);
+
+        /*
+         * read_guest() fetches at most two halfwords, so a wider encoding has
+         * been read in part only and cannot be decoded here.
+         *
+         * Report an illegal instruction: none of the extensions exposed to
+         * guests has instructions wider than 32 bits, so such an encoding is
+         * not a valid instruction for the guest in the first place.
+         *
+         * The same goes for a compressed encoding where C isn't exposed to the
+         * guest. The instruction in guest memory may have changed since the
+         * trap, so what is read back must not be assumed to be what trapped,
+         * nor to be valid for the guest.
+         */
+        if ( !di->insn_len ||
+             (di->insn_len == 2 &&
+              !riscv_isa_extension_available(current->domain->arch.isa,
+                                             RISCV_ISA_EXT_c)) )
+        {
+            utrap.sepc = regs->sepc;
+            utrap.scause = CAUSE_ILLEGAL_INSTRUCTION;
+            /*
+             * stval is left zero, which the spec allows for an illegal
+             * instruction: only part of the instruction is in hand, and stval,
+             * being only XLEN bits wide, may not be able to hold all of it
+             * anyway.
+             */
+
+            trap_redirect(&utrap);
+
+            return true;
+        }
+    }
+
+    return false;
+}
+
+/*
+ * Decode the load or store instruction fetched into @di, filling in the
+ * remaining fields of it (@di->insn and @di->insn_len are filled by
+ * insn_fetch_faulted()). Fields which don't apply to the instruction are left
+ * alone, so @di is expected to start out zeroed.
+ *
+ * @xlen is the effective XLEN of the guest, needed as
+ * the encodings which exist for XLEN=64 only must not be recognized for a
+ * 32-bit guest.
+ *
+ * Returns false if the instruction is not a load or store which can be
+ * emulated here.
+ */
+static bool decode_ldst_insn(struct decoded_insn *di, unsigned int xlen)
+{
+    uint32_t insn = di->insn;
+    unsigned int funct3, width_log2;
+
+    if ( INSN_IS_16BIT(insn) )
+    {
+        /*
+         * C.LW, C.LD, C.SW and C.SD (bits[1:0] == 00), and their sp-relative
+         * C.*SP forms (bits[1:0] == 10), have bits[15:13] of the form x1y:
+         * x is set for a store, and y selects a width of 4 or 8 bytes.
+         */
+        funct3 = RV_X(insn, 13, 3);
+
+        if ( (insn & 1) || !(funct3 & 2) )
+            return false;
+
+        di->is_write = funct3 & 4;
+        width_log2 = 2 + (funct3 & 1);
+
+        if ( !(insn & 2) )
+            /* Quadrant 0: bits[4:2] encode rd' (load) or rs2' (store) */
+            di->reg = RVC_RS2S(insn);
+        else if ( di->is_write )
+            /* Quadrant 2 (CSS): bits[6:2] encode rs2 for C.SWSP / C.SDSP */
+            di->reg = RVC_RS2(insn);
+        else
+        {
+            /* Quadrant 2 (CI): bits[11:7] encode rd for C.LWSP / C.LDSP */
+            di->reg = RV_RD(insn);
+
+            /* C.LWSP and C.LDSP are reserved with rd being x0. */
+            if ( !di->reg )
+                return false;
+        }
+    }
+    else
+    {
+        /*
+         * funct3[1:0] is log2 of the width in bytes, and funct3[2] selects
+         * zero-extension for a load, while being reserved for a store.
+         */
+        funct3 = RV_X(insn, 12, 3);
+        width_log2 = funct3 & 3;
+
+        switch ( insn & INSN_OPCODE_MASK )
+        {
+        case INSN_OPCODE_LOAD:
+            di->is_unsigned = funct3 & 4;
+            di->reg = RV_RD(insn);
+            break;
+
+        case INSN_OPCODE_STORE:
+            if ( funct3 & 4 )
+                return false;
+            di->is_write = true;
+            di->reg = RV_RS2(insn);
+            break;
+
+        default:
+            return false;
+        }
+    }
+
+    di->len = 1U << width_log2;
+
+    /*
+     * No access is wider than XLEN, and one as wide as XLEN exists only in
+     * its sign-extending form: this rules out the encodings which exist for
+     * XLEN=64 only on a 32-bit guest, including C.FLW for C.LD (and alike).
+     */
+    if ( (di->len * BITS_PER_BYTE > xlen) ||
+         (di->is_unsigned && di->len * BITS_PER_BYTE == xlen) )
+        return false;
+
+    return true;
+}
+
 static int emulate_load(const struct guest_fault *gf)
 {
+    /* Transient, until the helpers gain their users. */
+    (void)advance_pc;
+    (void)guest_xlen;
+    (void)guest_gpr;
+    (void)insn_fetch_faulted;
+    (void)decode_ldst_insn;
+
     return -EOPNOTSUPP;
 }
 
diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index b01da95f7ece..1af309aa4ad5 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -65,6 +65,14 @@
 #define SSTATUS64_UXL			MSTATUS_UXL
 #define SSTATUS64_SD			MSTATUS64_SD
 
+/*
+ * Width encoded by the MXL, SXL, UXL and VSXL fields, all of which share one
+ * encoding. 0 is reserved.
+ */
+#define XLEN_FIELD_32			_UL(1)
+#define XLEN_FIELD_64			_UL(2)
+#define XLEN_FIELD_128			_UL(3)
+
 #if __riscv_xlen == 64
 #define HSTATUS_VSXL			_UL(0x300000000)
 #define HSTATUS_VSXL_SHIFT		32
@@ -854,6 +862,10 @@
 /* 32-bit write for VS-stage address translation (VSXLEN=32) */
 #define INSN_PSEUDO_VS_STORE32		0x00002020
 
+#define INSN_OPCODE_MASK		0x7f
+#define INSN_OPCODE_LOAD		0x03
+#define INSN_OPCODE_STORE		0x23
+
 #define INSN_IS_16BIT(insn)	(((insn) & 3) != 3)
 #define INSN_IS_32BIT(insn)	(!INSN_IS_16BIT(insn) && ((insn) & 0x1c) != 0x1c)
 
@@ -895,6 +907,8 @@
 					 (RV_X(x, 7, 2) << 6))
 #define RVC_SDSP_IMM(x)			((RV_X(x, 10, 3) << 3) | \
 					 (RV_X(x, 7, 3) << 6))
+#define RV_RD(insn)			RV_X(insn, SH_RD, 5)
+#define RV_RS2(insn)			RV_X(insn, SH_RS2, 5)
 #define RVC_RS1S(insn)			(8 + RV_X(insn, SH_RD, 3))
 #define RVC_RS2S(insn)			(8 + RV_X(insn, SH_RS2C, 3))
 #define RVC_RS2(insn)			RV_X(insn, SH_RS2C, 5)
-- 
2.55.0

[PATCH v3 24/39] xen/riscv: add guest load emulation for trapped MMIO accesses

[Oleksii Kurochko]

Implement emulate_load() on top of the decoding interface introduced by
the previous patch: fetch the trapped instruction, decode it, dispatch
the access to a registered MMIO handler via do_mmio(), write the result
back into the destination register and step over the instruction.

Xen dispatches MMIO synchronously to an in-hypervisor handler, so unlike
KVM RISC-V there is no userspace exit/return step and no equivalent of
the kvm_io_bus_read() / KVM_EXIT_MMIO / kvm_riscv_vcpu_mmio_return()
split; the result is consumed in place.

Sign extension is done here rather than in the handlers: a signed load
is normalized by a shift pair, so a handler need only report the value
it read.

At the moment vINTC is the only backend registered with the MMIO
dispatch, so in practice this only covers vINTC traps. An access which
no handler claims currently crashes the domain; injecting an access
fault into the guest instead is left for later.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Shorten the comment above "if ( di.reg )" by dropping redundant part.
 - s/sizeof(unsigned long)/sizeof(info.data) in emulate_load().
 - Zero-initialize di, as decode_ldst_insn() expects it to start out zeroed.
 - Adjust to guest_gpr() taking the decoded instruction rather than a bare
   register number.
---
Changes in v2:
 - Move the emulation code to the new arch/riscv/emulate.c, leaving traps.c
   with trap dispatch only.
 - Split the patch up: struct decoded_insn and the decoding helpers are
   introduced by "xen/riscv: introduce the interface for trapped instruction
   decoding" and filled in by "xen/riscv: implement trapped instruction
   decoding", so only emulate_load() itself is left here.
 - do_mmio() is no longer defined alongside emulate_load(); it now lives in
   mmio.c, next to the dispatch it drives.
 - Don't write the result of a load into x0. SET_RD() wrote rd
   unconditionally, so a load into x0 clobbered regs->zero and broke the
   invariant that it reads as zero when x0 is a source operand elsewhere.
 - Recognize the XLEN=64-only encodings by the guest's effective XLEN
   (guest_xlen()) rather than by Xen's own (CONFIG_RISCV_32). Besides those
   encodings simply being reserved on RV32, the compressed ones are ambiguous
   there: C.LD and C.FLW share the encoding 0x6000 (mask 0xe003), and likewise
   C.SD/C.FSW, C.LDSP/C.FLWSP and C.SDSP/C.FSWSP.
 - Take the faulting address from struct guest_fault, filled in by
   resolve_faulting_gpa(), rather than from get_faulting_gpa().
 - Drop the description of a fault taken while re-reading the trapped
   instruction: that code is now in the patch implementing
   fetch_trapped_insn(), where a G-stage fault is reported to the guest as
   CAUSE_FETCH_ACCESS instead of hitting a BUG_ON().
 - Update the commit message accordingly.
---
---
 xen/arch/riscv/emulate.c | 39 ++++++++++++++++++++++++++++++++-------
 1 file changed, 32 insertions(+), 7 deletions(-)

diff --git a/xen/arch/riscv/emulate.c b/xen/arch/riscv/emulate.c
index 2695a3277605..ff8cba7d5116 100644
--- a/xen/arch/riscv/emulate.c
+++ b/xen/arch/riscv/emulate.c
@@ -14,6 +14,7 @@
 #include <asm/current.h>
 #include <asm/emulate.h>
 #include <asm/guest_access.h>
+#include <asm/mmio.h>
 #include <asm/processor.h>
 #include <asm/regs.h>
 #include <asm/riscv_encoding.h>
@@ -417,14 +418,38 @@ static bool decode_ldst_insn(struct decoded_insn *di, unsigned int xlen)
 
 static int emulate_load(const struct guest_fault *gf)
 {
-    /* Transient, until the helpers gain their users. */
-    (void)advance_pc;
-    (void)guest_xlen;
-    (void)guest_gpr;
-    (void)insn_fetch_faulted;
-    (void)decode_ldst_insn;
+    struct cpu_user_regs *regs = gf->regs;
+    mmio_info_t info = { .is_write = false };
+    struct decoded_insn di = {};
+    unsigned int shift = 0;
+    int rc;
 
-    return -EOPNOTSUPP;
+    /* A fault taken re-reading the instruction is redirected to the guest. */
+    if ( insn_fetch_faulted(gf, &di) )
+        return 0;
+
+    if ( !decode_ldst_insn(&di, guest_xlen(regs)) || di.is_write )
+        return -EOPNOTSUPP;
+
+    if ( !di.is_unsigned )
+        shift = BITS_PER_BYTE * (sizeof(info.data) - di.len);
+
+#ifdef EMULATE_LOAD_DEBUG
+    gdprintk(XENLOG_DEBUG, "pc=%#lx, addr=%#"PRIpaddr", len=%u, shift=%u\n",
+             regs->sepc, gf->gpa, di.len, shift);
+#endif
+
+    rc = do_mmio(&info, gf->gpa, di.len);
+    if ( rc )
+        return rc;
+
+    /* A load into x0 discards its result. */
+    if ( di.reg )
+        *guest_gpr(regs, &di) = (long)(info.data << shift) >> shift;
+
+    advance_pc(regs, di.insn_len);
+
+    return 0;
 }
 
 static int emulate_store(const struct guest_fault *gf)
-- 
2.55.0

[PATCH v3 25/39] xen/riscv: add guest store emulation for trapped MMIO accesses

[Oleksii Kurochko]

Extend the guest page fault handler with store emulation to support MMIO
write accesses.

Like emulate_load(), emulate_store() obtains the trapped instruction with
insn_fetch_faulted(), which also deals with a fault taken while re-reading
it, and decodes it with decode_ldst_insn(). As with the load path, the
access is completed synchronously through do_mmio().

At the moment only virtual interrupt controller (vINTC) traps are expected
to occur, since it is currently the only backend registered with the MMIO
handler dispatch, so in practice the store is emulated via the vINTC
backend.

Together with load emulation, this completes the basic MMIO handling path
needed for virtual interrupt controller support on RISC-V.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Zero-initialize di, as decode_ldst_insn() expects it to start out zeroed.
 - Adjust to guest_gpr() taking the decoded instruction rather than a bare
   register number.
 - Don't change the prototype of emulate_store(): the stub takes a pointer
   to const already.
---
Changes in v2:
 - Move the emulation code to the new arch/riscv/emulate.c, leaving traps.c
   with trap dispatch only.
 - Split the patch up: the instruction fetch and the mask/match chain now
   live in "xen/riscv: add helpers for decoding a trapped load or store"
   (struct decoded_insn, insn_fetch_faulted(), decode_ldst_insn(),
   guest_gpr(), advance_pc()), so only emulate_store() itself is left here.
 - Since the decoder is now shared with the load path, reject an encoding
   which is not a store (!di.is_write) explicitly; in v2 the mask/match
   chain was store-only and could not match a load.
 - Recognize the XLEN=64-only encodings by the guest's effective XLEN
   (guest_xlen()) rather than by Xen's own (CONFIG_RISCV_32). Besides those
   encodings simply being reserved on RV32, the compressed ones are ambiguous
   there: C.SD and C.FSW share an encoding, and likewise C.SDSP and C.FSWSP.
 - Read the source register through guest_gpr() instead of the
   GET_RS2()/GET_RS2S()/GET_RS2C() macros; which of the three register
   fields an encoding names is now decided by decode_ldst_insn().
 - Take the faulting address from struct guest_fault, filled in by
   resolve_faulting_gpa(), rather than from the fault_addr parameter.
 - Drop the description of a fault taken while re-reading the trapped
   instruction: that code is now in the patch adding the decoding helpers,
   where a G-stage fault is reported to the guest as CAUSE_FETCH_ACCESS.
 - Update the commit message accordingly.
---
---
 xen/arch/riscv/emulate.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/xen/arch/riscv/emulate.c b/xen/arch/riscv/emulate.c
index ff8cba7d5116..7ae3f438a726 100644
--- a/xen/arch/riscv/emulate.c
+++ b/xen/arch/riscv/emulate.c
@@ -454,7 +454,26 @@ static int emulate_load(const struct guest_fault *gf)
 
 static int emulate_store(const struct guest_fault *gf)
 {
-    return -EOPNOTSUPP;
+    struct cpu_user_regs *regs = gf->regs;
+    mmio_info_t info = { .is_write = true };
+    struct decoded_insn di = {};
+    int rc;
+
+    if ( insn_fetch_faulted(gf, &di) )
+        return 0;
+
+    if ( !decode_ldst_insn(&di, guest_xlen(regs)) || !di.is_write )
+        return -EOPNOTSUPP;
+
+    info.data = *guest_gpr(regs, &di);
+
+    rc = do_mmio(&info, gf->gpa, di.len);
+    if ( rc )
+        return rc;
+
+    advance_pc(regs, di.insn_len);
+
+    return 0;
 }
 
 static void inject_access_fault(const struct guest_fault *gf)
-- 
2.55.0

[PATCH v3 26/39] xen/riscv: introduce arch_move_irqs()

[Oleksii Kurochko]

When migrating a vCPU between pCPUs the hypervisor must also migrate
the associated virtual interrupt state. arch_move_irqs() is the
per-arch hook called by generic code to trigger that.

Replace the static inline BUG_ON placeholder in asm/irq.h with a real
implementation in intc.c dispatching through a new move_irqs vintc_ops
callback. Wire it up in vAPLIC, which delegates to imsic_migrate_vcpu(),
only a stub for now.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
---
Changes in v3:
 - Update the commit message: drop the two last paragraphs as they are
   stale.
 - Add Acked-by: Jan Beulich <jbeulich@suse.com>.
 - Add Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c             | 5 +++++
 xen/arch/riscv/include/asm/imsic.h | 2 ++
 xen/arch/riscv/include/asm/intc.h  | 3 +++
 xen/arch/riscv/include/asm/irq.h   | 5 +----
 xen/arch/riscv/intc.c              | 8 ++++++++
 xen/arch/riscv/vaplic.c            | 1 +
 6 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index d633f59db20d..c5fcdcf13b00 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -690,3 +690,8 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
 
     return fdt_end_node(fdt);
 }
+
+void imsic_migrate_vcpu(struct vcpu *v)
+{
+    BUG_ON("unimplemented");
+}
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index 5463fb4c31b2..22ce90e2e9b3 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -99,4 +99,6 @@ int vimsic_make_domu_dt_node(struct kernel_info *kinfo, unsigned int *phandle);
 void imsic_ctxt_switch_from(struct vcpu *p);
 void imsic_ctxt_switch_to(struct vcpu *n);
 
+void imsic_migrate_vcpu(struct vcpu *v);
+
 #endif /* ASM_RISCV_IMSIC_H */
diff --git a/xen/arch/riscv/include/asm/intc.h b/xen/arch/riscv/include/asm/intc.h
index 59e12f3d3c6d..a22b46dae06c 100644
--- a/xen/arch/riscv/include/asm/intc.h
+++ b/xen/arch/riscv/include/asm/intc.h
@@ -70,6 +70,9 @@ struct vintc_ops {
 
     /* Restore vINTC state of the vCPU being switched in */
     void (*ctxt_switch_to)(struct vcpu *n);
+
+    /* Move interrupts of vCPU to a different pCPU */
+    void (*move_irqs)(struct vcpu *v);
 };
 
 struct vintc {
diff --git a/xen/arch/riscv/include/asm/irq.h b/xen/arch/riscv/include/asm/irq.h
index 57e814d90cf4..311af1d16fbb 100644
--- a/xen/arch/riscv/include/asm/irq.h
+++ b/xen/arch/riscv/include/asm/irq.h
@@ -41,10 +41,7 @@ struct irq_desc *irq_to_desc(unsigned int irq);
 struct cpu_user_regs;
 struct dt_device_node;
 
-static inline void arch_move_irqs(struct vcpu *v)
-{
-    BUG_ON("unimplemented");
-}
+void arch_move_irqs(struct vcpu *v);
 
 int platform_get_irq(const struct dt_device_node *device, int index);
 
diff --git a/xen/arch/riscv/intc.c b/xen/arch/riscv/intc.c
index 53ba2b04ab53..a51eec251563 100644
--- a/xen/arch/riscv/intc.c
+++ b/xen/arch/riscv/intc.c
@@ -192,3 +192,11 @@ void vintc_ctxt_switch_to(struct vcpu *n)
 
     ops->ctxt_switch_to(n);
 }
+
+/* Move vCPU's IRQs from one pCPU to another */
+void arch_move_irqs(struct vcpu *v)
+{
+    const struct vintc_ops *ops = v->domain->arch.vintc->ops;
+
+    ops->move_irqs(v);
+}
diff --git a/xen/arch/riscv/vaplic.c b/xen/arch/riscv/vaplic.c
index 4d87fdb5be42..601b79aae430 100644
--- a/xen/arch/riscv/vaplic.c
+++ b/xen/arch/riscv/vaplic.c
@@ -462,6 +462,7 @@ static const struct vintc_ops vintc_ops = {
      */
     .ctxt_switch_from = imsic_ctxt_switch_from,
     .ctxt_switch_to = imsic_ctxt_switch_to,
+    .move_irqs = imsic_migrate_vcpu,
 };
 
 int domain_vaplic_init(struct domain *d)
-- 
2.55.0

[PATCH v3 27/39] xen/riscv: handle the case when no vCPU migration is needed

[Oleksii Kurochko]

The scheduler moves a vCPU's interrupts (arch_move_irqs(), i.e.
imsic_migrate_vcpu()) ahead of context_switch() whenever the unit is
picked up by a pCPU other than the one it was assigned to, which can
already happen for a freshly created vCPU on its very first run. No IMSIC
h/w interrupt file has been mapped for such a vCPU yet, so there is
nothing to migrate: return early while its last_cpu is still CPU_NONE,
i.e. while it hasn't run.

Keep the BUG_ON() at the end of the function until imsic_migrate_vcpu() is
fully implemented, to avoid ending up with a vCPU which isn't fully migrated
to the new IMSIC interrupt file.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Reword the last paragraph of the commit message: the BUG_ON() is already
   present in imsic_migrate_vcpu() before this patch, so it is kept rather
   than added.
 - Check last_cpu against CPU_NONE instead of NR_CPUS, as that is the
   value a vCPU which never ran holds now.
 - Describe in the commit message how imsic_migrate_vcpu() can be called
   for a vCPU which never ran, rather than referring to where the IMSIC
   interrupt file gets attached.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index c5fcdcf13b00..2f0bb29ae639 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -693,5 +693,15 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
 
 void imsic_migrate_vcpu(struct vcpu *v)
 {
+    /*
+     * The scheduler can mark a freshly created vCPU's unit as migrated and
+     * invoke this before the vCPU has ever run (see the migrated branch in
+     * schedule()). No need to do migration for such vCPUs as they aren't fully
+     * initialized (for example, context_switch() will be called after
+     * imsic_migrate_vcpu()).
+     */
+    if ( v->arch.last_cpu == CPU_NONE )
+        return;
+
     BUG_ON("unimplemented");
 }
-- 
2.55.0

[PATCH v3 28/39] xen/riscv: introduce vaplic_reconfigure_target()

[Oleksii Kurochko]

When a vCPU is migrated to a different pCPU, its IMSIC guest interrupt
file changes. Any APLIC interrupt previously configured to deliver an
MSI to the old interrupt file must be retargeted to the new one.

Implement vaplic_reconfigure_target() to walk the interrupts allocated to
the domain and rewrite the APLIC TARGET register of every one whose
guest-programmed target names the vCPU, building the new value with
aplic_msi_target_gen(). Each source is rewritten under its vaplic_source
lock, so that it can't race with a guest write of the same target
register. Nothing calls it yet.

Also assert in aplic_msi_target_gen() that the physical APLIC is in MSI
delivery mode, which all of its callers rely on.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Update the commit message.
 - Replace aplic_reconfigure_target() in aplic.c with
   vaplic_reconfigure_target() in vaplic.c: the sources to retarget are
   chosen from the guest's view of the target registers, decoded by the
   new vaplic_target_decode_vcpu(), instead of by matching the h/w
   registers against the old guest file id and pCPU, which drops the
   old_guest_file_id and old_cpu parameters. Each source is rewritten
   under its vaplic_source lock, so it can't race with a guest write of the
   same target register.
 - Hold vsfile_lock for reading while retargeting, as aplic_msi_target_gen()
   requires. Lock order is the vaplic_source lock, then vsfile_lock, then
   aplic.lock (taken by aplic_hw_write_reg()).
 - Replace the BUG_ON() on MSI delivery mode with an ASSERT() in
   aplic_msi_target_gen(), so that none of its callers can miss it; direct
   delivery mode isn't supported, so there's no way to get there in
   release builds either.
 - Update the comment above aplic_msi_target_gen() about why vsfile_lock
   has to be held.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aplic.c              | 12 +++++-
 xen/arch/riscv/include/asm/vaplic.h |  3 ++
 xen/arch/riscv/vaplic.c             | 58 ++++++++++++++++++++++++++++-
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index 60f5f239c394..283e5d6046a6 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -128,7 +128,10 @@ static unsigned long aplic_hart_index(unsigned int cpu)
 /*
  * v->processor can't be used here: during a migration it names the new pCPU
  * before the interrupt file is moved there. The caller has to hold
- * vsfile_lock until the result is written to the h/w.
+ * vsfile_lock, so that the guest file id and the pCPU owning it are read as
+ * a consistent pair. A value built from the old file just before a migration
+ * publishes the new one isn't lost: vaplic_reconfigure_target() runs after
+ * the publication and rewrites every source targeting the vCPU.
  */
 uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val)
 {
@@ -139,6 +142,13 @@ uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val)
     ASSERT(rw_is_locked(&vimsic_state->vsfile_lock));
     ASSERT(vimsic_state->vsfile_cpu < NR_CPUS);
 
+    /*
+     * The callers rely on the physical APLIC being in MSI delivery mode.
+     * Checking that here rather than in each of them means none can be
+     * missed should direct delivery mode ever be supported.
+     */
+    ASSERT(aplic_msi_mode());
+
     hart_index = aplic_hart_index(vimsic_state->vsfile_cpu);
 
     base_val &= APLIC_TARGET_EIID;
diff --git a/xen/arch/riscv/include/asm/vaplic.h b/xen/arch/riscv/include/asm/vaplic.h
index 3a555742bbe0..3cc69a3c8847 100644
--- a/xen/arch/riscv/include/asm/vaplic.h
+++ b/xen/arch/riscv/include/asm/vaplic.h
@@ -17,6 +17,7 @@
 #include <asm/intc.h>
 
 struct domain;
+struct vcpu;
 
 #define to_vaplic(d) container_of((d)->arch.vintc, struct vaplic, vintc)
 
@@ -50,4 +51,6 @@ struct vaplic {
 int domain_vaplic_init(struct domain *d);
 void domain_vaplic_deinit(struct domain *d);
 
+void vaplic_reconfigure_target(const struct vcpu *v);
+
 #endif /* ASM__RISCV__VAPLIC_H */
diff --git a/xen/arch/riscv/vaplic.c b/xen/arch/riscv/vaplic.c
index 601b79aae430..74f470ef3e52 100644
--- a/xen/arch/riscv/vaplic.c
+++ b/xen/arch/riscv/vaplic.c
@@ -52,6 +52,62 @@ static uint32_t vaplic_target_read(const struct domain *d, unsigned int irqn)
     return read_atomic(&vaplic->regs.sources[irqn].target);
 }
 
+/* Decode the target vCPU from a TARGET snapshot (NULL if none). */
+static struct vcpu *vaplic_target_decode_vcpu(const struct domain *d,
+                                              uint32_t target)
+{
+    unsigned int hart_idx = MASK_EXTR(target, APLIC_TARGET_HART_IDX);
+
+    /*
+     * Skip a source whose EIID is 0, e.g. one still holding its reset
+     * value: interrupt identity 0 isn't valid at any IMSIC interrupt file,
+     * so such a source delivers nothing and needs no retargeting.
+     * The hart index range check is only a guard: the write path refuses to
+     * record values with an out-of-range hart index.
+     */
+    if ( !(target & APLIC_TARGET_EIID) || hart_idx >= d->max_vcpus )
+        return NULL;
+
+    return d->vcpu[hart_idx];
+}
+
+void vaplic_reconfigure_target(const struct vcpu *v)
+{
+    struct domain *d = v->domain;
+    struct vaplic *vaplic = to_vaplic(d);
+    struct vimsic_state *vimsic_state = v->arch.vimsic_state;
+    unsigned int irqn;
+
+    bitmap_for_each ( irqn, d->arch.vintc->used_irqs, d->arch.vintc->nr_virqs )
+    {
+        struct vaplic_source *src = &vaplic->regs.sources[irqn];
+        unsigned long flags;
+        uint32_t target;
+
+        /* APLIC interrupt sources start from 1, there is no source 0. */
+        if ( !irqn )
+            continue;
+
+        spin_lock_irqsave(&src->lock, flags);
+
+        target = read_atomic(&src->target);
+
+        if ( vaplic_target_decode_vcpu(d, target) == v )
+        {
+            /*
+             * IRQs are already off under src->lock so no need for *_irqsave
+             * here.
+             */
+            read_lock(&vimsic_state->vsfile_lock);
+            aplic_hw_write_reg(offsetof(struct aplic_regs, target[irqn - 1]),
+                               aplic_msi_target_gen(v, target));
+            read_unlock(&vimsic_state->vsfile_lock);
+        }
+
+        spin_unlock_irqrestore(&src->lock, flags);
+    }
+}
+
 static inline uint32_t generate_auth_mask(const struct domain *currd,
                                           unsigned int word_idx)
 {
@@ -282,7 +338,7 @@ static bool vaplic_emulate_store(const struct vcpu *curr, paddr_t addr,
 
             /*
              * src->lock keeps the stored copy and the h/w register updated
-             * as a pair.
+             * as a pair against vaplic_reconfigure_target().
              */
             spin_lock_irqsave(&src->lock, flags);
 
-- 
2.55.0

[PATCH v3 29/39] xen/riscv: prepare new IMSIC VS-file

[Oleksii Kurochko]

Implement the first steps of migrating a vCPU to a different guest
interrupt file: take an interrupt file on the vCPU's new pCPU with
vgein_assign() and bring it to a clean state by setting eidelivery and
eithreshold to 0 and zeroing all implemented interrupt-pending and
interrupt-enable bits (the eip and eie arrays).

An IMSIC VS-file is reachable only through hstatus.VGEIN of the hart it
belongs to, so the clearing is done by that hart, through
imsic_call_on_cpu(). The helper doing it temporarily retargets
hstatus.VGEIN and vsiselect, which belong to whatever vCPU is running
there, and restores both before returning.

vgein_assign() is only a stub for now. Software interrupt files aren't
supported, so a vCPU which can't be given a hardware one hits a BUG_ON(),
and the BUG_ON("unimplemented") at the end of imsic_migrate_vcpu() stays,
as the migration functionality isn't complete yet.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Split the switchcase comment so each part sits above the macros it
   describes.
 - Drop imsic_switchcase_ret() as it has no users in this patch.
 - Drop the unused mrif member from struct imsic_vsfile_data.
 - Leave the trailing semicolon of the switchcase macros to their users.
 - Use __VA_ARGS__ instead of the GNU ##__VA_ARGS__ extension and
   parenthesize ireg in the switchcase expansion pyramid.
 - Make imsic_call_on_cpu() take struct imsic_vsfile_data * instead of
   void *.
 - Explain in imsic_vsfile_local_clear() why hstatus and vsiselect are
   saved and restored: the file is reached by retargeting hstatus.VGEIN,
   imsic_vs_csr_write() alters vsiselect, and both belong to the vCPU
   running on the pCPU.
 - Introduce imsic_eix_write64() to write a whole 64-bit EIx group,
   documenting the EIx register layout on RV32 vs RV64 above it, which
   lets the #ifdef CONFIG_RISCV_32 in imsic_vsfile_local_clear() go.
 - Drop the unclear "Latch the pCPU" comment.
 - Drop the redundant parentheses around the arguments of
   imsic_vs_csr_write() as they are passed to csr_*() as whole expressions.
 - Add missing break after ASSERT_UNREACHABLE() in the default case of
   imsic_eix_write().
 - Compute the number of EIx groups once in imsic_parse_node() and keep it
   in imsic_cfg.nr_eix instead of recalculating it on each migration.
 - Drop the introduction of BITS_PER_TYPE() in xen/include/xen/config.h as
   plain 64 is used instead.
 - Minor style and wording fixes: "} while (0)" in imsic_vs_csr_write(),
   no "\n" in the BUG_ON() message of vgein_assign(), and the extra args of
   the switchcase expansion pyramid are no longer described as optional.
 - Rewrite the commit message: it described saving eidelivery/eithreshold
   of the old interrupt file, which isn't done here, and adding the
   BUG_ON("unimplemented") placeholder, which is already present.
 - Drop the new_vsfile_{hgei,cpu} locals: vsfile_data.hgei and v->processor
   serve the same purpose.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aia.c               |   9 ++
 xen/arch/riscv/imsic.c             | 159 +++++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/aia.h   |   4 +
 xen/arch/riscv/include/asm/imsic.h |   7 ++
 4 files changed, 179 insertions(+)

diff --git a/xen/arch/riscv/aia.c b/xen/arch/riscv/aia.c
index e31c9c2d24b6..229ab4b8678e 100644
--- a/xen/arch/riscv/aia.c
+++ b/xen/arch/riscv/aia.c
@@ -1,8 +1,10 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/bug.h>
 #include <xen/errno.h>
 #include <xen/init.h>
 #include <xen/sections.h>
+#include <xen/sched.h>
 #include <xen/types.h>
 
 #include <asm/cpufeature.h>
@@ -21,3 +23,10 @@ void __init aia_init(void)
 
     _aia_usable = true;
 }
+
+unsigned int vgein_assign(struct vcpu *v)
+{
+    BUG_ON("unimplemented");
+
+    return 0;
+}
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 2f0bb29ae639..f1e44dc56bb8 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -26,6 +26,7 @@
 #include <xen/spinlock.h>
 #include <xen/xvmalloc.h>
 
+#include <asm/aia.h>
 #include <asm/imsic.h>
 
 #define IMSIC_HART_SIZE(guest_bits) (BIT(guest_bits, U) * IMSIC_MMIO_PAGE_SZ)
@@ -77,6 +78,72 @@ do {                            \
     csr_clear(CSR_SIREG, v);    \
 } while (0)
 
+#define imsic_vs_csr_write(c, v)    \
+do {                                \
+    csr_write(CSR_VSISELECT, c);    \
+    csr_write(CSR_VSIREG, v);       \
+} while (0)
+
+/*
+ * Leaf macros for the switchcase expansion pyramid below:
+ * imsic_switchcase_break(ireg, op, v) - emit "case ireg: op(ireg, v); break"
+ */
+#define imsic_switchcase_break(ireg, op, v) \
+    case ireg:                              \
+        op(ireg, v);                        \
+        break
+
+/*
+ * Generic switchcase expansion pyramid.
+ * F is the per-operation leaf macro, ireg is the base register index.
+ * Extra args (e.g. an operation and/or a value) are forwarded to F via
+ * __VA_ARGS__.
+ */
+#define imsic_switchcase_2(F, ireg, ...)             \
+    F((ireg) + 0, __VA_ARGS__);                      \
+    F((ireg) + 1, __VA_ARGS__)
+#define imsic_switchcase_4(F, ireg, ...)             \
+    imsic_switchcase_2(F, (ireg) + 0, __VA_ARGS__);  \
+    imsic_switchcase_2(F, (ireg) + 2, __VA_ARGS__)
+#define imsic_switchcase_8(F, ireg, ...)             \
+    imsic_switchcase_4(F, (ireg) + 0, __VA_ARGS__);  \
+    imsic_switchcase_4(F, (ireg) + 4, __VA_ARGS__)
+#define imsic_switchcase_16(F, ireg, ...)            \
+    imsic_switchcase_8(F, (ireg) + 0, __VA_ARGS__);  \
+    imsic_switchcase_8(F, (ireg) + 8, __VA_ARGS__)
+#define imsic_switchcase_32(F, ireg, ...)            \
+    imsic_switchcase_16(F, (ireg) + 0, __VA_ARGS__); \
+    imsic_switchcase_16(F, (ireg) + 16, __VA_ARGS__)
+#define imsic_switchcase_64(F, ireg, ...)            \
+    imsic_switchcase_32(F, (ireg) + 0, __VA_ARGS__); \
+    imsic_switchcase_32(F, (ireg) + 32, __VA_ARGS__)
+
+static void imsic_eix_write(unsigned int ireg, unsigned long val)
+{
+    switch ( ireg )
+    {
+    imsic_switchcase_64(imsic_switchcase_break, IMSIC_EIP0,
+                        imsic_vs_csr_write, val);
+    imsic_switchcase_64(imsic_switchcase_break, IMSIC_EIE0,
+                        imsic_vs_csr_write, val);
+    default:
+        ASSERT_UNREACHABLE();
+        break;
+    }
+}
+
+/*
+ * On RV64 a 64-bit EIx group is the even-numbered register alone, whereas on
+ * RV32 it is split into the even-numbered (low half) and the following
+ * odd-numbered (high half) register.
+ */
+static void imsic_eix_write64(unsigned int ireg, uint64_t val)
+{
+    imsic_eix_write(ireg, val);
+    if ( IS_ENABLED(CONFIG_RISCV_32) )
+        imsic_eix_write(ireg + 1, val >> 32);
+}
+
 unsigned int vcpu_guest_file_id(const struct vcpu *v)
 {
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
@@ -318,6 +385,8 @@ static int __init imsic_parse_node(const struct dt_device_node *node,
         return -ENOENT;
     }
 
+    imsic_cfg.nr_eix = DIV_ROUND_UP(imsic_cfg.nr_ids + 1, 64);
+
     /* Compute base address */
     *nr_mmios = 0;
     rc = dt_device_get_address(node, *nr_mmios, &base_addr, NULL);
@@ -389,6 +458,79 @@ int cf_check vcpu_imsic_init(struct vcpu *v)
     return 0;
 }
 
+/*
+ * The imsic_vsfile_local_*() helpers have to run on the pCPU owning the
+ * interrupt file, so they are called through imsic_call_on_cpu(), which
+ * passes them a single pointer. Their arguments are bundled here.
+ */
+struct imsic_vsfile_data {
+    /* Guest interrupt file to work on, i.e. the hstatus.VGEIN selecting it. */
+    unsigned int hgei;
+    /* Number of 64-bit EIx groups to cover, see imsic_cfg.nr_eix. */
+    unsigned int nr_eix;
+};
+
+/*
+ * Execute func() on the pCPU which owns the IMSIC interrupt file func() is
+ * going to work with.
+ *
+ * An IMSIC VS-file is reachable only through hstatus.VGEIN of the hart the
+ * file belongs to, and a guest interrupt file index is meaningless on any
+ * other hart, so such work always has to be done by that very hart.
+ *
+ * The local case runs with IRQs disabled to provide func() with the same
+ * environment it is given when it is called from the function call IPI
+ * handler.
+ */
+static void imsic_call_on_cpu(unsigned int cpu, void (*func)(void *),
+                              struct imsic_vsfile_data *data)
+{
+    if ( cpu == smp_processor_id() )
+    {
+        unsigned long flags;
+
+        local_irq_save(flags);
+        func(data);
+        local_irq_restore(flags);
+    }
+    else
+        on_selected_cpus(cpumask_of(cpu), func, data, 1);
+}
+
+static void cf_check imsic_vsfile_local_clear(void *data)
+{
+    unsigned int i;
+    const struct imsic_vsfile_data *idata = data;
+    unsigned long new_hstatus, old_hstatus, old_vsiselect;
+
+    /* We can only zero-out if we have a IMSIC VS-file */
+    if ( !idata->hgei )
+        return;
+
+    /*
+     * The file is reached by retargeting hstatus.VGEIN, and
+     * imsic_vs_csr_write() alters vsiselect. Both belong to the vCPU running
+     * on this pCPU, so restore them before returning.
+     */
+    old_vsiselect = csr_read(CSR_VSISELECT);
+    old_hstatus = csr_read(CSR_HSTATUS);
+    new_hstatus = old_hstatus & ~HSTATUS_VGEIN;
+    new_hstatus |= MASK_INSR(idata->hgei, HSTATUS_VGEIN);
+    csr_write(CSR_HSTATUS, new_hstatus);
+
+    imsic_vs_csr_write(IMSIC_EIDELIVERY, 0);
+    imsic_vs_csr_write(IMSIC_EITHRESHOLD, 0);
+
+    for ( i = 0; i < idata->nr_eix; i++ )
+    {
+        imsic_eix_write64(IMSIC_EIP0 + i * 2, 0);
+        imsic_eix_write64(IMSIC_EIE0 + i * 2, 0);
+    }
+
+    csr_write(CSR_HSTATUS, old_hstatus);
+    csr_write(CSR_VSISELECT, old_vsiselect);
+}
+
 void cf_check vcpu_imsic_deinit(struct vcpu *v)
 {
     XVFREE(v->arch.vimsic_state);
@@ -693,6 +835,10 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
 
 void imsic_migrate_vcpu(struct vcpu *v)
 {
+    struct imsic_vsfile_data vsfile_data = {
+        .nr_eix = imsic_cfg.nr_eix,
+    };
+
     /*
      * The scheduler can mark a freshly created vCPU's unit as migrated and
      * invoke this before the vCPU has ever run (see the migrated branch in
@@ -703,5 +849,18 @@ void imsic_migrate_vcpu(struct vcpu *v)
     if ( v->arch.last_cpu == CPU_NONE )
         return;
 
+    /*
+     * At this point, all interrupt producers are still using the old IMSIC
+     * VS-file.
+     */
+
+    vsfile_data.hgei = vgein_assign(v);
+
+    /* We don't support SW interrupt files at the moment. */
+    BUG_ON(!vsfile_data.hgei);
+
+    /* Zero-out new IMSIC VS-file */
+    imsic_call_on_cpu(v->processor, imsic_vsfile_local_clear, &vsfile_data);
+
     BUG_ON("unimplemented");
 }
diff --git a/xen/arch/riscv/include/asm/aia.h b/xen/arch/riscv/include/asm/aia.h
index aaa4bf91fc75..53a1efb042f8 100644
--- a/xen/arch/riscv/include/asm/aia.h
+++ b/xen/arch/riscv/include/asm/aia.h
@@ -3,8 +3,12 @@
 #ifndef RISCV_AIA_H
 #define RISCV_AIA_H
 
+struct vcpu;
+
 bool aia_usable(void);
 
 void aia_init(void);
 
+unsigned int vgein_assign(struct vcpu *v);
+
 #endif /* RISCV_AIA_H */
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index 22ce90e2e9b3..8130d125f5c3 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -55,6 +55,13 @@ struct imsic_config {
     /* Number off interrupt identities */
     unsigned int nr_ids;
 
+    /*
+     * Number of 64-bit EIx groups needed to cover all the interrupt
+     * identities, which are 0 (never valid, but it still occupies a bit) up
+     * to and including nr_ids.
+     */
+    unsigned int nr_eix;
+
     /* MSI */
     const struct imsic_msi *msi;
 
-- 
2.55.0

[PATCH v3 30/39] xen/riscv: implement APLIC-IMSIC-hart sync barrier for vCPU migration

[Oleksii Kurochko]

When a vCPU is moved to a different guest interrupt file, MSIs that the
APLIC has already sent towards the old file may still be in flight. They
must land before the old file's state is saved and the switch is done,
otherwise they would be lost.

To wait for them, use the APLIC's genmsi register. Writing it makes the
APLIC itself send an MSI (an "extempore" MSI) with a given interrupt
identity to a given hart. genmsi can only target the hart's hypervisor-
level interrupt file, not a guest one, but the AIA spec guarantees that
all MSIs previously sent by the same APLIC to the same hart become
visible at the hart's IMSIC before the extempore MSI does. So once the
extempore MSI has been delivered, no older MSI from this APLIC to the
hart can still be in flight, whichever interrupt file it targets.

genmsi's Busy bit clearing only tells that the APLIC has sent the
extempore MSI, not that it has arrived, so the sequence described in the
AIA spec under "Synchronizing interactions between a hart and the APLIC"
is split in two. aplic_genmsi_barrier() does the APLIC side, while
imsic_aplic_sync() does the hart side: it clears the pending bit of the
reserved identity before the genmsi write and spins on it afterwards,
which is what tells that the extempore MSI, and hence every older MSI
from this APLIC to this hart, has really arrived. Keeping the hart side
in imsic.c leaves the APLIC code free of the IMSIC local CSR accessors.

The last interrupt identity (nr_ids) is reserved for this purpose.

Migration doesn't use the barrier at this stage, so imsic_migrate_vcpu()
only references imsic_aplic_sync() to keep the compiler from complaining
about it being unused.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Rename from "xen/riscv: implement APLIC-hart sync barrier for vCPU
   migration", as the barrier waits for the MSIs to reach the hart's IMSIC.
 - Re-word the commit message, describing in it how the sequence is split
   between aplic_genmsi_barrier() and imsic_aplic_sync().
 - Move imsic_aplic_sync(), imsic_local_is_pending() and imsic_csr_read()
   here from "xen/riscv: remap interrupts to new IMSIC VS-file", so that
   both halves of the synchronization sequence are introduced together.
 - s/aplic_hart_field/aplic_hart_index/ following the rename of the helper
   in "xen/riscv: implement virtual APLIC MMIO emulation".
 - Drop masking of sync_id with APLIC_TARGET_EIID, which could silently
   chop off bits; instead add BUILD_BUG_ON() and ASSERT() to make it
   explicit that sync_id always fits into the EIID field.
 - Drop the redundant parentheses around the macro argument in
   imsic_csr_read().
 - Use unsigned int for isel in imsic_local_is_pending() and drop the
   redundant !! as the function returns bool.
 - Rename the unused argument of imsic_aplic_sync() to unused.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aplic.c             | 33 +++++++++++++++++++
 xen/arch/riscv/imsic.c             | 52 ++++++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/aplic.h |  3 ++
 xen/arch/riscv/include/asm/imsic.h |  8 +++++
 4 files changed, 96 insertions(+)

diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index 283e5d6046a6..5f9a894b5a76 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -27,7 +27,9 @@
 #include <asm/imsic.h>
 #include <asm/intc.h>
 #include <asm/io.h>
+#include <asm/processor.h>
 #include <asm/riscv_encoding.h>
+#include <asm/smp.h>
 
 static struct aplic_priv aplic = {
     .lock = SPIN_LOCK_UNLOCKED,
@@ -183,6 +185,37 @@ void aplic_hw_write_reg(unsigned int offset, uint32_t value)
     spin_unlock_irqrestore(&aplic.lock, flags);
 }
 
+/*
+ * As needed, synchronize with all IOMMUs and APLICs to ensure that no
+ * straggler MSIs will arrive at the old interrupt file after this step.
+ */
+void aplic_genmsi_barrier(void)
+{
+    const struct imsic_config *imsic = imsic_get_config();
+    unsigned int cpu = smp_processor_id();
+    unsigned long flags;
+    uint32_t val;
+
+    /*
+     * sync_id is nr_ids, which imsic_parse_node() limits to IMSIC_MAX_ID,
+     * so it always fits into the EIID field.
+     */
+    BUILD_BUG_ON(IMSIC_MAX_ID > MASK_EXTR(~0U, APLIC_TARGET_EIID));
+    ASSERT(imsic->sync_id <= IMSIC_MAX_ID);
+
+    val = MASK_INSR(aplic_hart_index(cpu), APLIC_TARGET_HART_IDX) |
+          imsic->sync_id;
+
+    spin_lock_irqsave(&aplic.lock, flags);
+
+    writel(val, &aplic.regs->genmsi);
+
+    while ( readl(&aplic.regs->genmsi) & APLIC_GENMSI_BUSY )
+        cpu_relax();
+
+    spin_unlock_irqrestore(&aplic.lock, flags);
+}
+
 static void __init aplic_init_hw_interrupts(void)
 {
     unsigned int i;
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index f1e44dc56bb8..8e0273eab902 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -27,6 +27,7 @@
 #include <xen/xvmalloc.h>
 
 #include <asm/aia.h>
+#include <asm/aplic.h>
 #include <asm/imsic.h>
 
 #define IMSIC_HART_SIZE(guest_bits) (BIT(guest_bits, U) * IMSIC_MMIO_PAGE_SZ)
@@ -60,6 +61,12 @@ static unsigned int __ro_after_init guest_num_msis;
 #define IMSIC_DISABLE_EITHRESHOLD   1
 #define IMSIC_ENABLE_EITHRESHOLD    0
 
+#define imsic_csr_read(c)       \
+({                              \
+    csr_write(CSR_SISELECT, c); \
+    csr_read(CSR_SIREG);        \
+})
+
 #define imsic_csr_write(c, v)   \
 do {                            \
     csr_write(CSR_SISELECT, c); \
@@ -203,6 +210,15 @@ void imsic_irq_enable(unsigned int irq)
      */
     ASSERT(!local_irq_is_enabled());
 
+    if ( irq == imsic_cfg.sync_id )
+    {
+        printk(XENLOG_WARNING
+               "irq%u is reserved for APLIC sync so shouldn't be set by %s\n",
+               irq, __func__);
+
+        return;
+    }
+
     spin_lock(&imsic_cfg.lock);
     /*
      * There is no irq - 1 here (look at aplic_set_irq_type()) because:
@@ -241,6 +257,15 @@ void imsic_irq_disable(unsigned int irq)
     spin_unlock(&imsic_cfg.lock);
 }
 
+static bool imsic_local_is_pending(unsigned int id)
+{
+    unsigned int isel =
+        (id / BITS_PER_LONG) * (BITS_PER_LONG / IMSIC_EIPx_BITS) + IMSIC_EIP0;
+    unsigned long bit = BIT(id % BITS_PER_LONG, UL);
+
+    return imsic_csr_read(isel) & bit;
+}
+
 /* Callers aren't intended to changed imsic_cfg so return const. */
 const struct imsic_config *imsic_get_config(void)
 {
@@ -387,6 +412,9 @@ static int __init imsic_parse_node(const struct dt_device_node *node,
 
     imsic_cfg.nr_eix = DIV_ROUND_UP(imsic_cfg.nr_ids + 1, 64);
 
+    /* Reserve last identity for APLIC-to-hart synchronization */
+    imsic_cfg.sync_id = imsic_cfg.nr_ids;
+
     /* Compute base address */
     *nr_mmios = 0;
     rc = dt_device_get_address(node, *nr_mmios, &base_addr, NULL);
@@ -497,6 +525,27 @@ static void imsic_call_on_cpu(unsigned int cpu, void (*func)(void *),
         on_selected_cpus(cpumask_of(cpu), func, data, 1);
 }
 
+/*
+ * Ensure that all the MSIs the APLIC has already generated for the hart this
+ * runs on have really reached the hart's IMSIC.
+ *
+ * The barrier is the one described by the AIA specification in
+ * "Synchronizing interactions between a hart and the APLIC": ask the APLIC to
+ * send an MSI to the hart itself and wait until it shows up as pending in the
+ * hart's own interrupt file. As it says nothing about MSIs on their way to
+ * any other hart, it has to be executed by the pCPU owning the interrupt file
+ * the MSIs were being sent to.
+ */
+static void cf_check imsic_aplic_sync(void *unused)
+{
+    imsic_local_eix_update(imsic_cfg.sync_id, 1, true, false);
+
+    aplic_genmsi_barrier();
+
+    while ( !imsic_local_is_pending(imsic_cfg.sync_id) )
+        cpu_relax();
+}
+
 static void cf_check imsic_vsfile_local_clear(void *data)
 {
     unsigned int i;
@@ -862,5 +911,8 @@ void imsic_migrate_vcpu(struct vcpu *v)
     /* Zero-out new IMSIC VS-file */
     imsic_call_on_cpu(v->processor, imsic_vsfile_local_clear, &vsfile_data);
 
+    /* Avoid an unused-function error until the barrier has a user. */
+    (void)imsic_aplic_sync;
+
     BUG_ON("unimplemented");
 }
diff --git a/xen/arch/riscv/include/asm/aplic.h b/xen/arch/riscv/include/asm/aplic.h
index 9018936ca95a..bfb510b2ae5e 100644
--- a/xen/arch/riscv/include/asm/aplic.h
+++ b/xen/arch/riscv/include/asm/aplic.h
@@ -94,6 +94,7 @@
 #define APLIC_SETIPNUM_LE               0x2000
 
 #define APLIC_GENMSI                    0x3000
+#define APLIC_GENMSI_BUSY               BIT(12, U)
 
 #define APLIC_TARGET_BASE               0x3004
 #define APLIC_TARGET_LAST               0x3ffc
@@ -177,4 +178,6 @@ struct aplic_regs {
 uint32_t aplic_hw_read_reg(unsigned int offset);
 void aplic_hw_write_reg(unsigned int offset, uint32_t value);
 
+void aplic_genmsi_barrier(void);
+
 #endif /* ASM_RISCV_APLIC_H */
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index 8130d125f5c3..f86407e21765 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -62,6 +62,14 @@ struct imsic_config {
      */
     unsigned int nr_eix;
 
+    /*
+     * Interrupt identity reserved exclusively for APLIC-to-hart
+     * synchronization.
+     *
+     * Must not be allocated to any interrupt source.
+     */
+    unsigned int sync_id;
+
     /* MSI */
     const struct imsic_msi *msi;
 
-- 
2.55.0

[PATCH v3 31/39] xen/riscv: remap interrupts to new IMSIC VS-file

[Oleksii Kurochko]

Once the new IMSIC VS-file has been zeroed out, it has to be mapped into
the guest's G-stage in place of the old one. Also, if any interrupts at an
APLIC are forwarded by MSIs to the old interrupt file, reconfigure the
APLIC to send them to the new interrupt file.

Taking a h/w guest interrupt file (assigning it, zeroing it out, mapping
it into G-stage and recording it in the vCPU's IMSIC state) is factored
out into imsic_vsfile_acquire().

The relevant translation tables at all IOMMUs would also have to be
updated so that MSIs for this virtual interrupt file are sent to the new
physical interrupt file, but that is skipped for now as there is no IOMMU
support for RISC-V.

Synchronize with the APLIC to ensure that no straggler MSIs will arrive at
the old interrupt file by running imsic_aplic_sync() on the pCPU which owns
that file.

If the new file can't be mapped into the guest, release the guest
external interrupt number imsic_vsfile_acquire() took for it instead of
leaking it. vgein_release() takes the pCPU explicitly instead of relying
on v->processor, as on migration v->processor already refers to the new
pCPU while the number to be released may belong to the old one.

Keep BUG_ON("unimplemented") placeholder in imsic_migrate_vcpu() to guard
against silent incorrect behaviour or unexpected panics in guest VMs until
the function is fully implemented.

Implement imsic_update_state(), which records the vCPU's guest
interrupt file id together with the pCPU owning the file. The two are
read as a pair, e.g. by imsic_migrate_vcpu() and the IMSIC context
switch hooks, so they are updated under vsfile_lock as a single unit.

imsic_map_guest_file() and vgein_release() are stubs for now.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
imsic_vsfile_acquire() isn't introduced already by "xen/riscv: prepare new
IMSIC VS-file": there, taking the new file is just vgein_assign() and zeroing
the file out. What makes it worth a helper of its own, mapping the file into
the guest, recording it in the vCPU's IMSIC state and releasing it again with
vgein_release() if the mapping fails, only comes with this patch. The helper
is also shared with the first attach of a h/w file in "xen/riscv: introduce
IMSIC h/w interrupt file attaching to vcpu".
---
Changes in v3:
 - Update the commit message.
 - Move imsic_aplic_sync() and imsic_local_is_pending() to "xen/riscv:
   implement APLIC-IMSIC-hart sync barrier for vCPU migration" and only
   call the helper here.
 - compare old_vsfile_cpu against CPU_NONE instead of NR_CPUS.
 - Call vaplic_reconfigure_target() instead of aplic_reconfigure_target().
 - Update the comment at the start of the migration: at that point the new
   IMSIC VS-file is only being prepared, the interrupt producers are moved
   to it later.
 - Factor taking a h/w guest interrupt file (assign, zero out, map into
   G-stage and record in the per-vCPU IMSIC state) out of
   imsic_migrate_vcpu() into imsic_vsfile_acquire(), introduced here
   instead of in "xen/riscv: introduce IMSIC h/w interrupt file attaching
   to vcpu".
 - Crash the domain instead of BUG_ON()ing when no h/w guest interrupt
   file is free, and report which file could not be mapped.
 - Introduce the vgein_release() stub here, together with its first
   user: release the h/w guest interrupt file taken by
   imsic_vsfile_acquire() if mapping it into the guest fails. Drop the
   stray "\n" from its BUG_ON().
 - Clarify the comment ahead of imsic_vsfile_acquire() on why setting
   hstatus.VGEIN is left to the caller.
 - Implement imsic_update_state() here instead of leaving it a stub until
   "xen/riscv: introduce IMSIC h/w interrupt file attaching to vcpu": all
   it needs, vsfile_lock and CPU_NONE, is available already. The s/w
   IMSIC VS-file is recorded with vsfile_cpu == CPU_NONE rather than
   NR_CPUS.
 - Make imsic_update_state() static and drop its declaration from
   asm/imsic.h: imsic_vsfile_acquire() is its only user.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aia.c               |   5 ++
 xen/arch/riscv/imsic.c             | 130 +++++++++++++++++++++++++++--
 xen/arch/riscv/include/asm/aia.h   |   1 +
 xen/arch/riscv/include/asm/imsic.h |   2 +
 4 files changed, 130 insertions(+), 8 deletions(-)

diff --git a/xen/arch/riscv/aia.c b/xen/arch/riscv/aia.c
index 229ab4b8678e..bd35078b7354 100644
--- a/xen/arch/riscv/aia.c
+++ b/xen/arch/riscv/aia.c
@@ -30,3 +30,8 @@ unsigned int vgein_assign(struct vcpu *v)
 
     return 0;
 }
+
+void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu)
+{
+    BUG_ON("unimplemented");
+}
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 8e0273eab902..47854264edf3 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -29,6 +29,7 @@
 #include <asm/aia.h>
 #include <asm/aplic.h>
 #include <asm/imsic.h>
+#include <asm/vaplic.h>
 
 #define IMSIC_HART_SIZE(guest_bits) (BIT(guest_bits, U) * IMSIC_MMIO_PAGE_SZ)
 
@@ -156,6 +157,18 @@ unsigned int vcpu_guest_file_id(const struct vcpu *v)
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
 }
 
+static void imsic_update_state(struct vcpu *v, unsigned int guest_file_id)
+{
+    unsigned long flags;
+    struct vimsic_state *vimsic_state = v->arch.vimsic_state;
+    unsigned int cpu = guest_file_id ? v->processor : CPU_NONE;
+
+    write_lock_irqsave(&vimsic_state->vsfile_lock, flags);
+    vimsic_state->guest_file_id = guest_file_id;
+    vimsic_state->vsfile_cpu = cpu;
+    write_unlock_irqrestore(&vimsic_state->vsfile_lock, flags);
+}
+
 void __init imsic_ids_local_delivery(bool enable)
 {
     if ( enable )
@@ -462,6 +475,11 @@ void cf_check imsic_ctxt_switch_to(struct vcpu *n)
     /* Nothing to do */
 }
 
+int imsic_map_guest_file(struct vcpu *v, unsigned int vsfile_id)
+{
+    return -EOPNOTSUPP;
+}
+
 int cf_check vcpu_imsic_init(struct vcpu *v)
 {
     struct vimsic_state *imsic_state;
@@ -882,11 +900,71 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
     return fdt_end_node(fdt);
 }
 
+/*
+ * Take a h/w guest interrupt file on the pCPU @v runs on and make it the
+ * vCPU's one.
+ *
+ * hstatus.VGEIN is deliberately left alone: it must only point at a file
+ * which already holds the vCPU's interrupt state. The file taken here is
+ * zeroed, so for imsic_migrate_vcpu() hstatus.VGEIN can only be updated once
+ * the state of the old file has been copied into it. Hence it is up to the
+ * caller to set hstatus.VGEIN at the right moment.
+ *
+ * Returns the id of the file, or 0, with the domain crashed, if none could be
+ * taken.
+ */
+static unsigned int imsic_vsfile_acquire(struct vcpu *v)
+{
+    unsigned int cpu = v->processor;
+    struct imsic_vsfile_data vsfile_data = { .nr_eix = imsic_cfg.nr_eix };
+    unsigned int vsfile_id;
+    int rc;
+
+    vsfile_id = vgein_assign(v);
+    if ( !vsfile_id )
+    {
+        /*
+         * vgein_assign() returns 0 when no free h/w guest interrupt file is
+         * available. s/w guest interrupt files aren't supported yet, so such
+         * a vCPU can't be run.
+         */
+        domain_crash(v->domain,
+                     "%pv: no free h/w guest interrupt file on CPU%u\n",
+                     v, cpu);
+        return 0;
+    }
+
+    vsfile_data.hgei = vsfile_id;
+
+    /* The file could still hold the state of its previous owner */
+    imsic_call_on_cpu(cpu, imsic_vsfile_local_clear, &vsfile_data);
+
+    rc = imsic_map_guest_file(v, vsfile_id);
+    if ( rc )
+    {
+        vgein_release(v, vsfile_id, cpu);
+
+        /* Can't continue w/o correctly mapped IMSIC interrupt file */
+        domain_crash(v->domain,
+                     "%pv: failed to map h/w guest interrupt file %u: %d\n",
+                     v, vsfile_id, rc);
+        return 0;
+    }
+
+    imsic_update_state(v, vsfile_id);
+
+    return vsfile_id;
+}
+
 void imsic_migrate_vcpu(struct vcpu *v)
 {
     struct imsic_vsfile_data vsfile_data = {
         .nr_eix = imsic_cfg.nr_eix,
     };
+    struct vimsic_state *imsic_state = v->arch.vimsic_state;
+    unsigned long flags;
+    unsigned int old_vsfile_id;
+    unsigned int old_vsfile_cpu;
 
     /*
      * The scheduler can mark a freshly created vCPU's unit as migrated and
@@ -898,21 +976,57 @@ void imsic_migrate_vcpu(struct vcpu *v)
     if ( v->arch.last_cpu == CPU_NONE )
         return;
 
+    read_lock_irqsave(&imsic_state->vsfile_lock, flags);
+    old_vsfile_id = imsic_state->guest_file_id;
+    old_vsfile_cpu = imsic_state->vsfile_cpu;
+    read_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
+
+    /*
+     * We don't support SW interrupt files at the moment. Bail out before
+     * anything is touched, as the old file has no owning pCPU in that case
+     * and there is nothing to retarget the producers away from.
+     */
+    if ( old_vsfile_cpu == CPU_NONE )
+        panic("IMSIC SW-file isn't supported\n");
+
     /*
      * At this point, all interrupt producers are still using the old IMSIC
-     * VS-file.
+     * VS-file. Allocate and clear the new one before redirecting anything
+     * to it.
      */
 
-    vsfile_data.hgei = vgein_assign(v);
+    /* Zero-out, map and start to use the new IMSIC VS-file */
+    vsfile_data.hgei = imsic_vsfile_acquire(v);
+    if ( !vsfile_data.hgei )
+        return;
 
-    /* We don't support SW interrupt files at the moment. */
-    BUG_ON(!vsfile_data.hgei);
+    /*
+     * TODO: Modify the relevant translation tables at all IOMMUs so that MSIs
+     *       for this virtual interrupt file are now sent to the new physical
+     *       interrupt file.
+     */
+    if ( iommu_enabled )
+        printk_once("IMSIC: IOMMU MSI retargeting is not implemented\n");
 
-    /* Zero-out new IMSIC VS-file */
-    imsic_call_on_cpu(v->processor, imsic_vsfile_local_clear, &vsfile_data);
+    /*
+     * If any interrupts at an APLIC are forwarded by MSIs to the old interrupt
+     * file, reconfigure the APLIC to send them to the new interrupt file.
+     */
+    vaplic_reconfigure_target(v);
 
-    /* Avoid an unused-function error until the barrier has a user. */
-    (void)imsic_aplic_sync;
+    /*
+     * Synchronizing interactions between a hart and the APLIC.
+     *
+     * The MSIs to be flushed are the ones still on their way to the old
+     * interrupt file, so the barrier has to be done by the pCPU which owns
+     * that file.
+     */
+    imsic_call_on_cpu(old_vsfile_cpu, imsic_aplic_sync, NULL);
+
+    /*
+     * At this point, all interrupt producers have been moved
+     * to the new IMSIC VS-file.
+     */
 
     BUG_ON("unimplemented");
 }
diff --git a/xen/arch/riscv/include/asm/aia.h b/xen/arch/riscv/include/asm/aia.h
index 53a1efb042f8..8e4eb2f6b14e 100644
--- a/xen/arch/riscv/include/asm/aia.h
+++ b/xen/arch/riscv/include/asm/aia.h
@@ -10,5 +10,6 @@ bool aia_usable(void);
 void aia_init(void);
 
 unsigned int vgein_assign(struct vcpu *v);
+void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu);
 
 #endif /* RISCV_AIA_H */
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index f86407e21765..e08b041eebbd 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -114,6 +114,8 @@ int vimsic_make_domu_dt_node(struct kernel_info *kinfo, unsigned int *phandle);
 void imsic_ctxt_switch_from(struct vcpu *p);
 void imsic_ctxt_switch_to(struct vcpu *n);
 
+int imsic_map_guest_file(struct vcpu *v, unsigned int vsfile_id);
+
 void imsic_migrate_vcpu(struct vcpu *v);
 
 #endif /* ASM_RISCV_IMSIC_H */
-- 
2.55.0

[PATCH v3 32/39] xen/riscv: save and clear the old IMSIC VS-file on vCPU migration

[Oleksii Kurochko]

When a vCPU migrates to another pCPU, its guest interrupt file has to
move as well: the eidelivery, eithreshold and eip/eie state of the old
IMSIC VS-file has to be carried over to the newly acquired one.

Once all interrupt producers have been retargeted to the new VS-file,
so that nothing can reach the old one anymore, read the state of the
old one and clear it at the same time. This has to be done on the pCPU
owning the old VS-file, as a guest interrupt file is only accessible
through vsiselect/vsireg with hstatus.VGEIN selecting it. After that
the old VS-file is no longer in use and its guest external interrupt
number can be released.

The state is saved into struct imsic_mrif. Its eix[] array follows the
layout of a memory-resident interrupt file (MRIF) as defined by the AIA
specification: pairs of 64-bit interrupt-pending and interrupt-enable
doublewords, each pair covering 64 interrupt identities. eithreshold
and eidelivery aren't part of an MRIF and are stored next to it as they
belong to the interrupt file state too. Using the MRIF layout allows
the same structure to hold the interrupt file of a vCPU which doesn't
own a guest interrupt file, which is how the AIA specification
describes MRIFs being used by a hypervisor.

Here the structure is only a temporary copy of a guest interrupt file:
it is filled and consumed by Xen alone and its address is never given
to an IOMMU, so no MSI is ever recorded into it. Hence plain
(non-atomic) accesses are sufficient and it can live on the stack.

The saved state still has to be restored into the new VS-file, so keep
the BUG_ON("unimplemented") placeholder at the end of
imsic_migrate_vcpu() to avoid silently losing the guest's interrupt
state.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Reword the commit message: explain why the MRIF layout is used and
   why the copy isn't an MSI target, drop references to later patches.
 - Compare vsfile_cpu against CPU_NONE instead of NR_CPUS.
 - Introduce imsic_switchcase_ret() here, together with its first user,
   drop its unused variadic form and leave the trailing semicolon to
   its users.
 - Introduce the mrif member of struct imsic_vsfile_data here, together
   with its first user.
 - Initialise vsfile_data.mrif in imsic_migrate_vcpu() with a compound
   literal instead of adding a separate tmrif local variable, so the MRIF
   the old interrupt file is dumped into is visibly the one later
   restored from.
 - Drop the redundant parentheses around the arguments of
   imsic_vs_csr_swap() and its unnecessary local variable.
 - Use plain 64 instead of BITS_PER_TYPE(uint64_t) for the number of
   interrupt identities covered by an EIx group, and explain in the
   comment ahead of IMSIC_MAX_EIX where it comes from (the MRIF layout
   from the AIA spec).
 - Keep each EIx group of the MRIF as a uint64_t and introduce
   imsic_eix_swap64() to swap a whole group, which lets the
   #ifdef CONFIG_RISCV_32 in imsic_vsfile_local_read_clear() go.
 - Add missing break after ASSERT_UNREACHABLE() in the default case of
   imsic_eix_swap().
 - Use MASK_INSR() to set HSTATUS.VGEIN in
   imsic_vsfile_local_read_clear().
 - Explain in imsic_vsfile_local_read_clear() why hstatus and vsiselect
   are saved and restored.
 - Reword the comment on why no atomic accessors are needed to store
   the values into the MRIF.
 - Take the number of EIx groups from vsfile_data.nr_eix, as the
   nr_hw_eix local variable is gone from imsic_migrate_vcpu().
 - Move the vgein_release() stub to "xen/riscv: remap interrupts to new
   IMSIC VS-file", where it gains its first user.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c | 124 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 47854264edf3..2c4838b6b228 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -57,6 +57,28 @@ static unsigned int __ro_after_init guest_num_msis;
  */
 #define GUEST_IMSIC_MAX_MSIS 255U
 
+/*
+ * The interrupt identities an IMSIC interrupt file provides are 0 (which is
+ * never valid, but still occupies a bit) up to IMSIC_MAX_ID inclusive, so
+ * IMSIC_MAX_ID + 1 bits have to be covered.
+ *
+ * Each EIx group covers 64 identities, matching the memory-resident interrupt
+ * file (MRIF) layout from the AIA spec, where pending and enable bits are
+ * stored as pairs of 64-bit doublewords.
+ */
+#define IMSIC_MAX_EIX DIV_ROUND_UP(IMSIC_MAX_ID + 1, 64)
+
+struct imsic_mrif_eix {
+    uint64_t eip;
+    uint64_t eie;
+};
+
+struct imsic_mrif {
+    struct imsic_mrif_eix eix[IMSIC_MAX_EIX];
+    unsigned long eithreshold;
+    unsigned long eidelivery;
+};
+
 #define IMSIC_DISABLE_EIDELIVERY    0
 #define IMSIC_ENABLE_EIDELIVERY     1
 #define IMSIC_DISABLE_EITHRESHOLD   1
@@ -86,6 +108,12 @@ do {                            \
     csr_clear(CSR_SIREG, v);    \
 } while (0)
 
+#define imsic_vs_csr_swap(c, v)     \
+({                                  \
+    csr_write(CSR_VSISELECT, c);    \
+    csr_swap(CSR_VSIREG, v);        \
+})
+
 #define imsic_vs_csr_write(c, v)    \
 do {                                \
     csr_write(CSR_VSISELECT, c);    \
@@ -95,12 +123,17 @@ do {                                \
 /*
  * Leaf macros for the switchcase expansion pyramid below:
  * imsic_switchcase_break(ireg, op, v) - emit "case ireg: op(ireg, v); break"
+ * imsic_switchcase_ret(ireg, op, v) - emit "case ireg: return op(ireg, v)"
  */
 #define imsic_switchcase_break(ireg, op, v) \
     case ireg:                              \
         op(ireg, v);                        \
         break
 
+#define imsic_switchcase_ret(ireg, op, v)   \
+    case ireg:                              \
+        return op(ireg, v)
+
 /*
  * Generic switchcase expansion pyramid.
  * F is the per-operation leaf macro, ireg is the base register index.
@@ -126,6 +159,22 @@ do {                                \
     imsic_switchcase_32(F, (ireg) + 0, __VA_ARGS__); \
     imsic_switchcase_32(F, (ireg) + 32, __VA_ARGS__)
 
+static unsigned long imsic_eix_swap(unsigned int ireg, unsigned long val)
+{
+    switch ( ireg )
+    {
+    imsic_switchcase_64(imsic_switchcase_ret, IMSIC_EIP0,
+                        imsic_vs_csr_swap, val);
+    imsic_switchcase_64(imsic_switchcase_ret, IMSIC_EIE0,
+                        imsic_vs_csr_swap, val);
+    default:
+        ASSERT_UNREACHABLE();
+        break;
+    }
+
+    return 0;
+}
+
 static void imsic_eix_write(unsigned int ireg, unsigned long val)
 {
     switch ( ireg )
@@ -152,6 +201,17 @@ static void imsic_eix_write64(unsigned int ireg, uint64_t val)
         imsic_eix_write(ireg + 1, val >> 32);
 }
 
+/* Swap a whole 64-bit EIx group, laid out as described above. */
+static uint64_t imsic_eix_swap64(unsigned int ireg, uint64_t val)
+{
+    uint64_t old = imsic_eix_swap(ireg, val);
+
+    if ( IS_ENABLED(CONFIG_RISCV_32) )
+        old |= (uint64_t)imsic_eix_swap(ireg + 1, val >> 32) << 32;
+
+    return old;
+}
+
 unsigned int vcpu_guest_file_id(const struct vcpu *v)
 {
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
@@ -514,6 +574,8 @@ struct imsic_vsfile_data {
     unsigned int hgei;
     /* Number of 64-bit EIx groups to cover, see imsic_cfg.nr_eix. */
     unsigned int nr_eix;
+    /* Where the interrupt file state is saved to or restored from. */
+    struct imsic_mrif *mrif;
 };
 
 /*
@@ -598,6 +660,60 @@ static void cf_check imsic_vsfile_local_clear(void *data)
     csr_write(CSR_VSISELECT, old_vsiselect);
 }
 
+static void cf_check imsic_vsfile_local_read_clear(void *data)
+{
+    unsigned int i;
+    const struct imsic_vsfile_data *idata = data;
+    struct imsic_mrif *mrif = idata->mrif;
+    unsigned long new_hstatus, old_hstatus, old_vsiselect;
+
+    /*
+     * The file is reached by retargeting hstatus.VGEIN, and
+     * imsic_vs_csr_swap() alters vsiselect. Both belong to the vCPU running
+     * on this pCPU, so restore them before returning.
+     */
+    old_vsiselect = csr_read(CSR_VSISELECT);
+    old_hstatus = csr_read(CSR_HSTATUS);
+    new_hstatus = old_hstatus & ~HSTATUS_VGEIN;
+    new_hstatus |= MASK_INSR(idata->hgei, HSTATUS_VGEIN);
+    csr_write(CSR_HSTATUS, new_hstatus);
+
+    /*
+     * No atomic accessors are needed to store the values into the MRIF here,
+     * as imsic_vsfile_read_clear() is always called with a pointer to a
+     * temporary MRIF on the stack.
+     */
+
+    mrif->eidelivery = imsic_vs_csr_swap(IMSIC_EIDELIVERY, 0);
+    mrif->eithreshold = imsic_vs_csr_swap(IMSIC_EITHRESHOLD, 0);
+    for ( i = 0; i < idata->nr_eix; i++ )
+    {
+        mrif->eix[i].eip = imsic_eix_swap64(IMSIC_EIP0 + i * 2, 0);
+        mrif->eix[i].eie = imsic_eix_swap64(IMSIC_EIE0 + i * 2, 0);
+    }
+
+    csr_write(CSR_HSTATUS, old_hstatus);
+    csr_write(CSR_VSISELECT, old_vsiselect);
+}
+
+static void imsic_vsfile_read_clear(unsigned int vsfile_id,
+                                    unsigned int vsfile_cpu,
+                                    unsigned int nr_eix,
+                                    struct imsic_mrif *mrif)
+{
+    struct imsic_vsfile_data idata = {
+        .hgei = vsfile_id,
+        .nr_eix = nr_eix,
+        .mrif = mrif,
+    };
+
+    /* We can only read clear if we have an IMSIC VS-file */
+    if ( vsfile_cpu == CPU_NONE || !vsfile_id )
+        return;
+
+    imsic_call_on_cpu(vsfile_cpu, imsic_vsfile_local_read_clear, &idata);
+}
+
 void cf_check vcpu_imsic_deinit(struct vcpu *v)
 {
     XVFREE(v->arch.vimsic_state);
@@ -960,6 +1076,7 @@ void imsic_migrate_vcpu(struct vcpu *v)
 {
     struct imsic_vsfile_data vsfile_data = {
         .nr_eix = imsic_cfg.nr_eix,
+        .mrif = &(struct imsic_mrif){ },
     };
     struct vimsic_state *imsic_state = v->arch.vimsic_state;
     unsigned long flags;
@@ -1028,5 +1145,12 @@ void imsic_migrate_vcpu(struct vcpu *v)
      * to the new IMSIC VS-file.
      */
 
+    /* Read and clear register state from old IMSIC VS-file */
+    imsic_vsfile_read_clear(old_vsfile_id, old_vsfile_cpu, vsfile_data.nr_eix,
+                            vsfile_data.mrif);
+
+    /* Free-up old IMSIC VS-file */
+    vgein_release(v, old_vsfile_id, old_vsfile_cpu);
+
     BUG_ON("unimplemented");
 }
-- 
2.55.0

[PATCH v3 33/39] xen/riscv: restore register state in the new IMSIC VS-file

[Oleksii Kurochko]

Complete the migration of a vCPU's guest interrupt file: move the state
saved from the old IMSIC VS-file into the new one and make the vCPU use
the new VS-file.

The interrupt producers have already been retargeted to the new
VS-file, so MSIs may land in it before the saved state is restored.
Hence the saved eip/eie bits are ORed into the new VS-file rather than
written to it, so that such MSIs aren't lost. eithreshold and
eidelivery are restored last: the new VS-file was cleared when it was
acquired, so it doesn't deliver interrupts until its pending and enable
bits are complete.

As with reading the old VS-file, this has to be done on the pCPU owning
the new VS-file, with hstatus.VGEIN temporarily selecting it. That is
v->processor, the pCPU the vCPU is migrating to.

Finally, point hstatus.VGEIN in the vCPU's saved context at the new
VS-file, so that it becomes the source of the vCPU's VS-level external
interrupts and the file the guest accesses through its siselect/sireg.
With that the migration is complete, so drop the BUG_ON("unimplemented")
placeholder from imsic_migrate_vcpu().

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Reword the commit message: explain why the saved state is ORed into
   the new VS-file and why eithreshold and eidelivery are restored last,
   drop the note about explanatory comments.
 - Leave the trailing semicolon of the switchcase macros to their users.
 - Drop the redundant parentheses around the arguments of
   imsic_vs_csr_set() as they are passed to csr_*() as whole
   expressions.
 - Add missing break after ASSERT_UNREACHABLE() in the default case of
   imsic_eix_set().
 - Restore the state on v->processor and take the id of the new IMSIC
   VS-file from vsfile_data.hgei, as imsic_migrate_vcpu() no longer has
   separate local variables for the new VS-file.
 - Explain why hstatus and vsiselect are saved and restored with a short
   comment in imsic_vsfile_local_update().
 - Introduce imsic_eix_set64() to set bits of a whole 64-bit EIx group,
   which lets the #ifdef CONFIG_RISCV_32 in imsic_vsfile_local_update()
   go.
 - Move the MASK_INSR() conversion and the comment rewording in
   imsic_vsfile_local_read_clear() to the patch introducing that
   function.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c | 80 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 78 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 2c4838b6b228..e35688f98bb1 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -114,6 +114,12 @@ do {                            \
     csr_swap(CSR_VSIREG, v);        \
 })
 
+#define imsic_vs_csr_set(c, v)      \
+do {                                \
+    csr_write(CSR_VSISELECT, c);    \
+    csr_set(CSR_VSIREG, v);         \
+} while (0)
+
 #define imsic_vs_csr_write(c, v)    \
 do {                                \
     csr_write(CSR_VSISELECT, c);    \
@@ -212,6 +218,28 @@ static uint64_t imsic_eix_swap64(unsigned int ireg, uint64_t val)
     return old;
 }
 
+static void imsic_eix_set(unsigned int ireg, unsigned long val)
+{
+    switch ( ireg )
+    {
+    imsic_switchcase_64(imsic_switchcase_break, IMSIC_EIP0,
+                        imsic_vs_csr_set, val);
+    imsic_switchcase_64(imsic_switchcase_break, IMSIC_EIE0,
+                        imsic_vs_csr_set, val);
+    default:
+        ASSERT_UNREACHABLE();
+        break;
+    }
+}
+
+/* Set bits of a whole 64-bit EIx group, laid out as described above. */
+static void imsic_eix_set64(unsigned int ireg, uint64_t val)
+{
+    imsic_eix_set(ireg, val);
+    if ( IS_ENABLED(CONFIG_RISCV_32) )
+        imsic_eix_set(ireg + 1, val >> 32);
+}
+
 unsigned int vcpu_guest_file_id(const struct vcpu *v)
 {
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
@@ -1016,6 +1044,47 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
     return fdt_end_node(fdt);
 }
 
+static void cf_check imsic_vsfile_local_update(void *data)
+{
+    unsigned int i;
+    const struct imsic_vsfile_data *idata = data;
+    struct imsic_mrif *mrif = idata->mrif;
+    unsigned long new_hstatus, old_hstatus, old_vsiselect;
+
+    /* We can only update if we have a HW IMSIC context */
+    if ( !idata->hgei )
+        return;
+
+    /*
+     * No atomic accessors are needed to read the values out of the MRIF here,
+     * as this is always called with a pointer to a temporary MRIF on the
+     * stack.
+     */
+
+    /*
+     * The file is reached by retargeting hstatus.VGEIN, and the
+     * imsic_vs_csr_*() accessors alter vsiselect. Both belong to the vCPU
+     * running on this pCPU, so restore them before returning.
+     */
+    old_vsiselect = csr_read(CSR_VSISELECT);
+    old_hstatus = csr_read(CSR_HSTATUS);
+    new_hstatus = old_hstatus & ~HSTATUS_VGEIN;
+    new_hstatus |= MASK_INSR(idata->hgei, HSTATUS_VGEIN);
+    csr_write(CSR_HSTATUS, new_hstatus);
+
+    for ( i = 0; i < idata->nr_eix; i++ )
+    {
+        imsic_eix_set64(IMSIC_EIP0 + i * 2, mrif->eix[i].eip);
+        imsic_eix_set64(IMSIC_EIE0 + i * 2, mrif->eix[i].eie);
+    }
+
+    imsic_vs_csr_write(IMSIC_EITHRESHOLD, mrif->eithreshold);
+    imsic_vs_csr_write(IMSIC_EIDELIVERY, mrif->eidelivery);
+
+    csr_write(CSR_HSTATUS, old_hstatus);
+    csr_write(CSR_VSISELECT, old_vsiselect);
+}
+
 /*
  * Take a h/w guest interrupt file on the pCPU @v runs on and make it the
  * vCPU's one.
@@ -1142,7 +1211,8 @@ void imsic_migrate_vcpu(struct vcpu *v)
 
     /*
      * At this point, all interrupt producers have been moved
-     * to the new IMSIC VS-file.
+     * to the new IMSIC VS-file so we move register state from
+     * the old IMSIC VS/SW-file to the new IMSIC VS-file.
      */
 
     /* Read and clear register state from old IMSIC VS-file */
@@ -1152,5 +1222,11 @@ void imsic_migrate_vcpu(struct vcpu *v)
     /* Free-up old IMSIC VS-file */
     vgein_release(v, old_vsfile_id, old_vsfile_cpu);
 
-    BUG_ON("unimplemented");
+    /* Restore register state in the new IMSIC VS-file */
+    imsic_call_on_cpu(v->processor, imsic_vsfile_local_update, &vsfile_data);
+
+    /* Set VCPU HSTATUS.VGEIN to new IMSIC VS-file */
+    vcpu_guest_cpu_user_regs(v)->hstatus &= ~HSTATUS_VGEIN;
+    vcpu_guest_cpu_user_regs(v)->hstatus |=
+        MASK_INSR(vsfile_data.hgei, HSTATUS_VGEIN);
 }
-- 
2.55.0

[PATCH v3 34/39] xen/riscv: add basic VGEIN management for AIA guests

[Oleksii Kurochko]

It was decided to add support for IMSIC from the start instead of having APLIC
operate in direct delivery mode, as it requires a trap-and-emulation approach,
which is not optimal from a performance standpoint.

AIA provides a hardware-accelerated mechanism for delivering external
interrupts to domains via "guest interrupt files" located in IMSIC.
A single physical hart can implement multiple such files (up to GEILEN),
allowing several virtual harts to receive interrupts directly from hardware.

Introduce per-CPU tracking of guest interrupt file identifiers (VGEIN)
for systems implementing AIA specification. Each CPU maintains
a bitmap describing which guest interrupt files are currently in use.

Implement helpers to initialize the bitmap based on the number of available
guest interrupt files (GEILEN), assign a VGEIN to a vCPU, and release it
when no longer needed.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
---
Also in the next patch there is other context to understand the usage of
spinlock introduced here.
---
Changes in v3:
 - Add Acked-by: Jan Beulich <jbeulich@suse.com>.
 - Add Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>.
---
Changes in v2:
 - make vgein_init() pCPU agnostic as it is working with CSR which could be
   read only on local pCPU itself.
 - Move introduction of vgein_ctrl->owners[] to separate patch.
 - Add ASSERT() and re-init vgein->bmp with 0.
 - Update the commit message (drop the last sentence as ->hstatus isn't
   filled anymore in in vgein_*() functions).
 - Introduce vgein_deinit().
---
---
 xen/arch/riscv/aia.c | 141 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 137 insertions(+), 4 deletions(-)

diff --git a/xen/arch/riscv/aia.c b/xen/arch/riscv/aia.c
index bd35078b7354..1aca07c2f70f 100644
--- a/xen/arch/riscv/aia.c
+++ b/xen/arch/riscv/aia.c
@@ -1,13 +1,31 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
-#include <xen/bug.h>
+#include <xen/bitops.h>
+#include <xen/cpu.h>
 #include <xen/errno.h>
 #include <xen/init.h>
 #include <xen/sections.h>
 #include <xen/sched.h>
+#include <xen/spinlock.h>
 #include <xen/types.h>
 
+#include <asm/aia.h>
 #include <asm/cpufeature.h>
+#include <asm/csr.h>
+#include <asm/current.h>
+
+struct vgein_ctrl {
+    /* The least-significant bits are implemented first, apart from bit 0 */
+    unsigned long bmp;
+    spinlock_t lock;
+    unsigned int geilen;
+};
+
+/*
+ * VGEIN control structure for each physical CPU to track which VS (guest)
+ * interrupt file IDs are in use.
+ */
+static DEFINE_PER_CPU(struct vgein_ctrl, vgein);
 
 static bool __ro_after_init _aia_usable;
 
@@ -16,22 +34,137 @@ bool aia_usable(void)
     return _aia_usable;
 }
 
+/* HGEIE is a per-hart CSR, so this has to run on the CPU being initialized. */
+static int vgein_init(void)
+{
+    struct vgein_ctrl *vgein = &this_cpu(vgein);
+
+    spin_lock_init(&vgein->lock);
+
+    csr_write(CSR_HGEIE, ~0UL);
+    vgein->geilen = flsl(csr_read(CSR_HGEIE) >> 1);
+    csr_write(CSR_HGEIE, 0);
+
+    vgein->bmp = 0;
+
+    if ( !vgein->geilen )
+        return -EOPNOTSUPP;
+
+    return 0;
+}
+
+static void vgein_deinit(void)
+{
+    csr_write(CSR_HGEIE, 0);
+}
+
+static int cf_check cpu_callback(struct notifier_block *nfb,
+                                 unsigned long action, void *hcpu)
+{
+    unsigned int cpu = (unsigned long)hcpu;
+    int rc = 0;
+
+    switch ( action )
+    {
+    case CPU_STARTING:
+        rc = vgein_init();
+        if ( rc )
+            printk(XENLOG_ERR "AIA: failed to init vgein for CPU%u: %d\n",
+                   cpu, rc);
+        break;
+
+    case CPU_DYING:
+        vgein_deinit();
+        break;
+    }
+
+    return notifier_from_errno(rc);
+}
+
+static struct notifier_block cpu_nfb = {
+    .notifier_call = cpu_callback,
+};
+
 void __init aia_init(void)
 {
+    int rc;
+
     if ( !riscv_isa_extension_available(NULL, RISCV_ISA_EXT_ssaia) )
+    {
+        dprintk(XENLOG_WARNING, "SSAIA isn't present in riscv,isa\n");
         return;
+    }
+
+    if ( (rc = vgein_init()) )
+    {
+        dprintk(XENLOG_ERR, "vgein_init() failed: %d\n", rc);
+        return;
+    }
 
     _aia_usable = true;
+
+    register_cpu_notifier(&cpu_nfb);
 }
 
 unsigned int vgein_assign(struct vcpu *v)
 {
-    BUG_ON("unimplemented");
+    unsigned int vgein_id;
+    struct vgein_ctrl *vgein = &per_cpu(vgein, v->processor);
+    unsigned long *bmp = &vgein->bmp;
+    unsigned long flags;
 
-    return 0;
+    if ( !vgein->geilen )
+        return 0;
+
+    spin_lock_irqsave(&vgein->lock, flags);
+    /*
+     * The vgein_id shouldn't be zero, as it will indicate that no guest
+     * external interrupt source is selected for VS-level external interrupts
+     * according to RISC-V privileged spec:
+     *   Hypervisor Status Register (hstatus) in RISC-V privileged spec:
+     *
+     *   The VGEIN (Virtual Guest External Interrupt Number) field selects
+     *   a guest external interrupt source for VS-level external interrupts.
+     *   VGEIN is a WLRL field that must be able to hold values between zero
+     *   and the maximum guest external interrupt number (known as GEILEN),
+     *   inclusive.
+     *   When VGEIN=0, no guest external interrupt source is selected for
+     *   VS-level external interrupts.
+     *
+     * So start to search from bit number 1.
+     */
+    vgein_id = find_next_zero_bit(bmp, vgein->geilen + 1, 1);
+
+    if ( vgein_id > vgein->geilen )
+        vgein_id = 0;
+    else
+        __set_bit(vgein_id, bmp);
+
+    spin_unlock_irqrestore(&vgein->lock, flags);
+
+#ifdef VGEIN_DEBUG
+    gprintk(XENLOG_DEBUG, "%s: %pv: vgein_id(%u), xen_cpu%u_bmp=%#lx\n",
+            __func__, v, vgein_id, v->processor, *bmp);
+#endif
+
+    return vgein_id;
 }
 
 void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu)
 {
-    BUG_ON("unimplemented");
+    unsigned long flags;
+    struct vgein_ctrl *vgein = &per_cpu(vgein, cpu);
+
+    if ( !vgein_id )
+        return;
+
+    spin_lock_irqsave(&vgein->lock, flags);
+    if ( !__test_and_clear_bit(vgein_id, &vgein->bmp) )
+        ASSERT_UNREACHABLE();
+    spin_unlock_irqrestore(&vgein->lock, flags);
+
+#ifdef VGEIN_DEBUG
+    gprintk(XENLOG_DEBUG, "%s: %pv: vgein_id(%u), xen_cpu%u_bmp=%#lx\n",
+            __func__, v, vgein_id, cpu, vgein->bmp);
+#endif
 }
-- 
2.55.0

[PATCH v3 35/39] xen/riscv: wake up a descheduled vCPU on a guest external interrupt

[Oleksii Kurochko]

While a vCPU is running, MSIs written to its h/w IMSIC guest interrupt
file are delivered straight to VS-mode. Once the vCPU is descheduled
nobody observes that file anymore, so a guest blocked on such an
interrupt would stay blocked until some unrelated event happens to
schedule it again.

Let Xen observe the file in that window: on deschedule set the vCPU's
bit in HGEIE, which turns an interrupt pending in its VS-file into an
HS-level SGEI, and clear the bit again on schedule-in. HGEIP only
reports an interrupt file number, so to find the vCPU to kick, keep a
per-pCPU owners[] array indexed by file number and updated in
vgein_assign()/vgein_release() together with the VGEIN bitmap. The SGEI
handler clears the HGEIE bits it has seen before kicking their owners,
so that the interrupt doesn't keep firing until the vCPU is switched in.

The same applies to a vCPU which is migrated to another pCPU while it
isn't running. Its old interrupt file is released during the migration,
and the vCPU isn't switched in on the new pCPU until something wakes it
up, so nothing would observe the new interrupt file. Hence arm HGEIE for
the new interrupt file on the new pCPU in imsic_migrate_vcpu(), and
clear the bit of the old one in imsic_vsfile_local_read_clear(), which
already runs on the pCPU owning it.

SGEIs are enabled per vCPU through v->arch.hie: vcpu_csr_init() sets
IRQ_S_GEXT in it and csr_regs_ctxt_switch_to() loads it into the CSR.

vgein_release() is only called on vCPU migration: a vCPU going away
also has to free its VGEIN slot and drop the owners[] entry, but there
is no vCPU teardown path to hook it into yet.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Drop the unrelated cpu_callback() change.
 - Read HGEIP and HGEIE in separate statements (MISRA C Rule 13.2).
 - Check that guest interrupt file ID 0 isn't reported once, ahead of
   the loop in hgei_interrupt(), instead of on every iteration.
 - Drop the VGEIN_DEBUG print from hgei_interrupt().
 - Explain in imsic_ctxt_switch_from() why setting the HGEIE bit last
   doesn't lose an interrupt.
 - Initialise hie with BIT(IRQ_S_GEXT, UL) and explain why SGEIs are
   enabled.
 - Arm HGEIE of the new interrupt file on v->processor, as the
   new_vsfile_cpu local variable is gone from imsic_migrate_vcpu().
 - Describe in the commit message why HGEIE is armed for a vCPU which
   is migrated while it isn't running.
 - Correct the commit message: vgein_release() is called on vCPU
   migration.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/aia.c                | 38 ++++++++++++++++++
 xen/arch/riscv/domain.c             |  7 ++++
 xen/arch/riscv/imsic.c              | 62 ++++++++++++++++++++++++++++-
 xen/arch/riscv/include/asm/aia.h    |  2 +
 xen/arch/riscv/include/asm/domain.h |  1 +
 xen/arch/riscv/traps.c              |  7 +++-
 6 files changed, 115 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/aia.c b/xen/arch/riscv/aia.c
index 1aca07c2f70f..d961c5c12239 100644
--- a/xen/arch/riscv/aia.c
+++ b/xen/arch/riscv/aia.c
@@ -18,6 +18,13 @@ struct vgein_ctrl {
     /* The least-significant bits are implemented first, apart from bit 0 */
     unsigned long bmp;
     spinlock_t lock;
+    /*
+     * Guest interrupt file IDs run from 1 to geilen inclusive (0 means that
+     * no guest external interrupt source is selected), and geilen can never
+     * exceed BITS_PER_LONG - 1, so indexing this array by the ID directly
+     * always fits.
+     */
+    struct vcpu *owners[BITS_PER_LONG];
     unsigned int geilen;
 };
 
@@ -138,7 +145,10 @@ unsigned int vgein_assign(struct vcpu *v)
     if ( vgein_id > vgein->geilen )
         vgein_id = 0;
     else
+    {
         __set_bit(vgein_id, bmp);
+        vgein->owners[vgein_id] = v;
+    }
 
     spin_unlock_irqrestore(&vgein->lock, flags);
 
@@ -161,6 +171,7 @@ void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu)
     spin_lock_irqsave(&vgein->lock, flags);
     if ( !__test_and_clear_bit(vgein_id, &vgein->bmp) )
         ASSERT_UNREACHABLE();
+    vgein->owners[vgein_id] = NULL;
     spin_unlock_irqrestore(&vgein->lock, flags);
 
 #ifdef VGEIN_DEBUG
@@ -168,3 +179,30 @@ void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu)
             __func__, v, vgein_id, cpu, vgein->bmp);
 #endif
 }
+
+void hgei_interrupt(void)
+{
+    unsigned long hgei_mask, flags;
+    struct vgein_ctrl *vgein = &this_cpu(vgein);
+
+    hgei_mask = csr_read(CSR_HGEIP);
+    hgei_mask &= csr_read(CSR_HGEIE);
+
+    csr_clear(CSR_HGEIE, hgei_mask);
+
+    spin_lock_irqsave(&vgein->lock, flags);
+
+    /*
+     * Bit 0 of HGEIP/HGEIE is read-only zero: guest interrupt file ID 0
+     * means that no guest external interrupt source is selected.
+     */
+    ASSERT(!(hgei_mask & 1));
+
+    for_each_set_bit ( guest_file_id, hgei_mask )
+    {
+        if ( vgein->owners[guest_file_id] )
+            vcpu_kick(vgein->owners[guest_file_id]);
+    }
+
+    spin_unlock_irqrestore(&vgein->lock, flags);
+}
diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index f60f871aba98..6cb762bee33c 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -130,6 +130,12 @@ static void vcpu_csr_init(struct vcpu *v)
         v->arch.hstateen0 = (hstateen0 & csr_masks.hstateen0) |
                             csr_masks.ro_one.hstateen0;
     }
+
+    /*
+     * Enable SGEIs, so that a guest interrupt file marked in HGEIE while
+     * the vCPU is descheduled can raise an interrupt to Xen.
+     */
+    v->arch.hie = BIT(IRQ_S_GEXT, UL);
 }
 
 static void continue_new_vcpu(struct vcpu *prev)
@@ -386,6 +392,7 @@ static void csr_regs_ctxt_switch_to(struct vcpu *n)
     csr_write(CSR_HEDELEG, n->arch.hedeleg);
     csr_write(CSR_HIDELEG, n->arch.hideleg);
     csr_write(CSR_HVIP, n->arch.hvip);
+    csr_write(CSR_HIE, n->arch.hie);
     csr_write64(CSR_HENVCFG, n->arch.henvcfg);
     csr_write(CSR_HCOUNTEREN, n->arch.hcounteren);
     csr_write64(CSR_HTIMEDELTA, n->arch.htimedelta);
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index e35688f98bb1..40d2eff14d07 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -555,12 +555,37 @@ void cf_check imsic_ctxt_switch_from(struct vcpu *p)
 
     write_lock_irqsave(&imsic_state->vsfile_lock, flags);
     imsic_state->vsfile_cpu = p->processor;
+    /*
+     * Start to observe the VS-file from HS-mode: while the vCPU isn't
+     * running an interrupt pending in its VS-file is reported through HGEIP
+     * instead of being delivered to VS-mode, which lets Xen wake the vCPU up.
+     *
+     * An MSI is recorded in the VS-file's eip[] array until the guest claims
+     * it, and HGEIP isn't latched but reflects whether the file currently
+     * has a pending-and-enabled interrupt. Hence an interrupt which arrived
+     * before this point raises an SGEI as soon as the bit is set in HGEIE;
+     * nothing is lost in between.
+     */
+    csr_set(CSR_HGEIE, BIT(imsic_state->guest_file_id, UL));
     write_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
 }
 
 void cf_check imsic_ctxt_switch_to(struct vcpu *n)
 {
-    /* Nothing to do */
+    struct vimsic_state *imsic_state = n->arch.vimsic_state;
+    unsigned long flags;
+
+    /* A s/w VS-file is never observed through HGEIP. */
+    if ( !vcpu_guest_file_id(n) )
+        return;
+
+    /*
+     * The vCPU is about to run, so hstatus.VGEIN delivers the VS-file's
+     * interrupts to it directly and there is nothing left for Xen to observe.
+     */
+    read_lock_irqsave(&imsic_state->vsfile_lock, flags);
+    csr_clear(CSR_HGEIE, BIT(imsic_state->guest_file_id, UL));
+    read_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
 }
 
 int imsic_map_guest_file(struct vcpu *v, unsigned int vsfile_id)
@@ -695,6 +720,12 @@ static void cf_check imsic_vsfile_local_read_clear(void *data)
     struct imsic_mrif *mrif = idata->mrif;
     unsigned long new_hstatus, old_hstatus, old_vsiselect;
 
+    /*
+     * The HGEIE bit imsic_ctxt_switch_from() armed belongs to the old owner
+     * only.
+     */
+    csr_clear(CSR_HGEIE, BIT(idata->hgei, UL));
+
     /*
      * The file is reached by retargeting hstatus.VGEIN, and
      * imsic_vs_csr_swap() alters vsiselect. Both belong to the vCPU running
@@ -1044,6 +1075,21 @@ int __init vimsic_make_domu_dt_node(struct kernel_info *kinfo,
     return fdt_end_node(fdt);
 }
 
+/*
+ * Start to observe the interrupt file from HS-mode, the same way
+ * imsic_ctxt_switch_from() does it for a vCPU which is switched out.
+ *
+ * The counterpart, clearing the bit of the interrupt file which is left
+ * behind, is done by imsic_vsfile_local_read_clear(), which already runs on
+ * the pCPU owning that file.
+ */
+static void cf_check imsic_local_hgeie_set(void *data)
+{
+    const struct imsic_vsfile_data *idata = data;
+
+    csr_set(CSR_HGEIE, BIT(idata->hgei, UL));
+}
+
 static void cf_check imsic_vsfile_local_update(void *data)
 {
     unsigned int i;
@@ -1225,6 +1271,20 @@ void imsic_migrate_vcpu(struct vcpu *v)
     /* Restore register state in the new IMSIC VS-file */
     imsic_call_on_cpu(v->processor, imsic_vsfile_local_update, &vsfile_data);
 
+    /*
+     * imsic_ctxt_switch_from() armed HGEIE for the old interrupt file only.
+     * Unless the vCPU is being switched in right now (a migration done by
+     * sched_context_switch(), where is_running is already set), arm it for
+     * the new file too: a blocked vCPU would otherwise never be woken up by
+     * an interrupt pending in the new file, be it restored from the old one
+     * or arriving later.
+     *
+     * For a vCPU which is being switched in, imsic_ctxt_switch_to() clears the
+     * bit anyway, as interrupts are then delivered to the vCPU directly.
+     */
+    if ( !v->is_running )
+        imsic_call_on_cpu(v->processor, imsic_local_hgeie_set, &vsfile_data);
+
     /* Set VCPU HSTATUS.VGEIN to new IMSIC VS-file */
     vcpu_guest_cpu_user_regs(v)->hstatus &= ~HSTATUS_VGEIN;
     vcpu_guest_cpu_user_regs(v)->hstatus |=
diff --git a/xen/arch/riscv/include/asm/aia.h b/xen/arch/riscv/include/asm/aia.h
index 8e4eb2f6b14e..6a05bdd8c236 100644
--- a/xen/arch/riscv/include/asm/aia.h
+++ b/xen/arch/riscv/include/asm/aia.h
@@ -12,4 +12,6 @@ void aia_init(void);
 unsigned int vgein_assign(struct vcpu *v);
 void vgein_release(struct vcpu *v, unsigned int vgein_id, unsigned int cpu);
 
+void hgei_interrupt(void);
+
 #endif /* RISCV_AIA_H */
diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
index c33d83836f2e..a3fa637b40da 100644
--- a/xen/arch/riscv/include/asm/domain.h
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -69,6 +69,7 @@ struct arch_vcpu {
     register_t hstateen0;
     uint64_t   htimedelta;
     register_t hvip;
+    register_t hie;
 
     register_t vsatp;
     register_t vscause;
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index dc938bae5b00..cf18f5185ff4 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -12,9 +12,10 @@
 #include <xen/sched.h>
 #include <xen/softirq.h>
 
-#include <asm/extable.h>
+#include <asm/aia.h>
 #include <asm/cpufeature.h>
 #include <asm/emulate.h>
+#include <asm/extable.h>
 #include <asm/intc.h>
 #include <asm/processor.h>
 #include <asm/riscv_encoding.h>
@@ -273,6 +274,10 @@ void do_trap(struct cpu_user_regs *cpu_regs)
                 timer_interrupt();
                 break;
 
+            case IRQ_S_GEXT:
+                hgei_interrupt();
+                break;
+
             default:
                 intr_handled = false;
                 break;
-- 
2.55.0

[PATCH v3 36/39] xen/riscv: map IMSIC interrupt file for vCPUs

[Oleksii Kurochko]

A guest running in VS-mode expects its own IMSIC S-file at offset 0 of its
guest-physical IMSIC block. Physically, the guest-file (G-file) assigned to
this vCPU lives at a hart-relative offset given by guest_file_id (assigned
via the vGEIN allocator). Therefore, imsic_map_guest_file() uses stage-2
translation to redirect the guest's fixed per-vCPU GPA page (offset 0) to
the specific physical guest-file page.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
---
Changes in v3:
 - Add Acked-by: Jan Beulich <jbeulich@suse.com>
 - Use dprintk() instead of printk() for the IMSIC_DEBUG message in
   imsic_map_guest_file().
 - Add Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
---
Changes in v2:
 - Use GUEST_IMSIC_S_BASE instead of imsic_cfg.base_addr as the base of the
   guest address to map to, and change the type of gaddr to paddr_t as it
   holds a guest physical address.
 - Rename guest_stride to guest_offset: it is an offset of the VS-file inside
   the pCPU's IMSIC block, not a stride.
 - Use PRIpaddr for physical addresses and %u for unsigned values in the
   debug/error messages.
 - Switch the mapping failure message from printk() to dprintk(XENLOG_ERR, ...).
 - Update the comment above imsic_map_guest_file(): vCPUs aren't pinned, they
   run on the pCPU chosen by the scheduler, and mention that on migration a
   VS-file is acquired on the new pCPU and mapped at the same GFN, so the
   stale mapping is replaced rather than explicitly torn down.
---
---
 xen/arch/riscv/imsic.c | 66 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 65 insertions(+), 1 deletion(-)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 40d2eff14d07..a8e99997c087 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -29,6 +29,7 @@
 #include <asm/aia.h>
 #include <asm/aplic.h>
 #include <asm/imsic.h>
+#include <asm/p2m.h>
 #include <asm/vaplic.h>
 
 #define IMSIC_HART_SIZE(guest_bits) (BIT(guest_bits, U) * IMSIC_MMIO_PAGE_SZ)
@@ -588,9 +589,72 @@ void cf_check imsic_ctxt_switch_to(struct vcpu *n)
     read_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
 }
 
+/*
+ * Map the physical IMSIC guest interrupt file (G-file) assigned to vCPU
+ * into the domain's stage-2 guest-physical address space.
+ *
+ * In the machine's physical address space (SPA), each hart's IMSIC
+ * supervisor-level file (S-file) is located at offset 0 of its address block,
+ * followed contiguously by GEILEN guest files at offsets of 1, 2, ..., N pages.
+ *
+ * Because a guest OS running in VS-mode expects its own supervisor-level
+ * interrupt file to be at offset 0 of its guest-physical IMSIC block, the
+ * hypervisor must use stage-2 address translation to map the vCPU's
+ * guest-physical "supervisor" page (GPA offset 0) to the specific
+ * physical guest file page (SPA offset guest_file_id) on the physical hart.
+ *
+ * A vCPU runs on the pCPU the scheduler picked for it (v->processor), and
+ * the guest file it is given (guest_file_id, from the vGEIN allocator)
+ * belongs to that very pCPU's IMSIC. A guest_file_id of 0 indicates that no
+ * hardware guest file is selected (matching the architectural behavior where
+ * vGEIN = 0 in the hstatus CSR selects no guest external interrupt source),
+ * requiring the VS-file to be emulated in software.
+ *
+ * Consequently the mapping installed here is only valid as long as the vCPU
+ * stays on that pCPU. When it migrates, a VS-file is acquired on the new
+ * pCPU and mapped at the very same GFN, so the stale mapping needs no
+ * explicit tear-down: it is simply replaced.
+ *
+ * The base guest-physical address advertised to the guest in the device
+ * tree matches offset 0 of the vCPU's virtual IMSIC block. Stage-2
+ * translation ensures that guest supervisor accesses to this page are
+ * transparently routed to the real hardware VS-file granted to it on
+ * the pCPU it currently runs on.
+ */
 int imsic_map_guest_file(struct vcpu *v, unsigned int vsfile_id)
 {
-    return -EOPNOTSUPP;
+    struct domain *d = v->domain;
+    unsigned int cpu = v->processor;
+    paddr_t gaddr = GUEST_IMSIC_S_BASE + (IMSIC_MMIO_PAGE_SZ * v->vcpu_id);
+    paddr_t paddr, guest_offset;
+    int res;
+
+    /* Nothing to map in the case of sw interrupt file. */
+    if ( !vsfile_id )
+        return 0;
+
+    guest_offset = vsfile_id * IMSIC_MMIO_PAGE_SZ;
+
+    paddr = imsic_cfg.msi[cpu].base_addr + imsic_cfg.msi[cpu].offset +
+            guest_offset;
+
+#ifdef IMSIC_DEBUG
+    dprintk(XENLOG_DEBUG,
+            "%s: %pv: ga(%#"PRIpaddr") -> pa(%#"PRIpaddr"), cpu(%u), "
+            "guest_file_id(%u) base_addr(%#"PRIpaddr") offset(%#lx)\n",
+            __func__, v, gaddr, paddr, cpu, vsfile_id,
+            imsic_cfg.msi[cpu].base_addr, imsic_cfg.msi[cpu].offset);
+#endif
+
+    res = map_regions_p2mt(d, gaddr_to_gfn(gaddr),
+                           PFN_DOWN(IMSIC_MMIO_PAGE_SZ), maddr_to_mfn(paddr),
+                           arch_dt_passthrough_p2m_type());
+    if ( res )
+        dprintk(XENLOG_ERR,
+                "%s: Failed to map %#"PRIpaddr" to the guest at %#"PRIpaddr"\n",
+                __func__, paddr, gaddr);
+
+    return res;
 }
 
 int cf_check vcpu_imsic_init(struct vcpu *v)
-- 
2.55.0

[PATCH v3 37/39] xen/riscv: implement continue_new_vcpu()

[Oleksii Kurochko]

continue_new_vcpu() is the arch hook invoked the first time a freshly
created vCPU is scheduled. Implement both cases it has to cover:
 - for the idle vCPU, switch to its own stack and jump to idle_loop();
 - for a guest vCPU, enter the guest through the new
   return_to_new_vcpu() path in entry.S, which restores hstatus and
   sepc, passes the hart id in a0 and, in a1, either the DTB address
   (boot vCPU, as expected by the RISC-V boot protocol) or the opaque
   value passed to SBI HSM hart_start() (secondary vCPUs), clears the
   remaining GPRs, sets sstatus.SPP and sstatus.SPIE and executes sret.

The remaining GPRs have to be cleared as sret neither switches stacks
nor touches the GPRs: the guest would otherwise start with Xen's values
in them, sp pointing to this vCPU's Xen stack among them.

Interrupts have to stay disabled across the restore. A trap taken in
HS-mode overwrites hstatus.SPV, sstatus.SPP and sepc (trap entry clears
hstatus.SPV in particular), so an interrupt taken between the write of
hstatus and sret would make sret return to HS-mode instead of VS-mode,
and restoring them afterwards is non-trivial. Instead interrupts are
simply kept off and sstatus.SPIE is set, so that sret re-enables them
by restoring sstatus.SIE from SPIE.

Introduce get_cpu_info() and reset_stack_and_jump() in asm/current.h,
needed by the above. get_cpu_info() is a macro rather than a static
inline because asm/current.h is pulled in by <xen/percpu.h> before
this_cpu() is defined and before <xen/sched.h> completes struct vcpu.

idle_loop() is only introduced as a stub.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Drop unneeded inclusion of asm/aia.h, asm/aplic.h and asm/imsic.h.
 - Mark return_to_new_vcpu() declaration with asmlinkage.
 - Drop the "else" after reset_stack_and_jump(idle_loop), which cannot
   return.
 - Move the restore of hstatus and the setting of sstatus.SPIE from
   continue_new_vcpu() to return_to_new_vcpu(), so that everything sret
   depends on is set up in one place, and shorten the comment in
   continue_new_vcpu() accordingly.
 - Move the tp/SSCRATCH swap out to a separate patch, as nothing reads
   SSCRATCH yet.
 - Reword the comments in return_to_new_vcpu(), which suggested that
   the values of .a0 and .a1 are established there.
 - Clear the GPRs which are not meaningful to the vCPU being started
   before sret, so that Xen register values are not exposed to a guest.
 - Update the commit message: a1 carries the opaque value passed to
   hart_start() for secondary vCPUs, describe the clearing of the GPRs,
   drop the comparison with a real CPU and the reference to a later
   implementation of idle_loop().
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/domain.c              | 25 +++++++++++++++++-
 xen/arch/riscv/entry.S               | 39 ++++++++++++++++++++++++++++
 xen/arch/riscv/include/asm/current.h |  4 +++
 3 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index 6cb762bee33c..04be628dcdb4 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -138,9 +138,27 @@ static void vcpu_csr_init(struct vcpu *v)
     v->arch.hie = BIT(IRQ_S_GEXT, UL);
 }
 
+static void schedule_tail(struct vcpu *prev);
+static void noreturn idle_loop(void);
+void asmlinkage noreturn return_to_new_vcpu(void);
+
 static void continue_new_vcpu(struct vcpu *prev)
 {
-    BUG_ON("unimplemented\n");
+    schedule_tail(prev);
+
+    if ( is_idle_vcpu(current) )
+        reset_stack_and_jump(idle_loop);
+
+    /*
+     * return_to_new_vcpu() sets up hstatus.SPV, sstatus.SPP and sepc so
+     * that sret enters the guest in VS-mode. A trap taken in HS-mode
+     * overwrites all of them (trap entry clears hstatus.SPV in particular),
+     * so interrupts have to stay disabled until sret. They are re-enabled by
+     * sret itself, as return_to_new_vcpu() also sets sstatus.SPIE.
+     */
+    local_irq_disable();
+
+    reset_stack_and_jump(return_to_new_vcpu);
 }
 
 int arch_vcpu_create(struct vcpu *v)
@@ -519,3 +537,8 @@ static void __init __maybe_unused build_assertions(void)
 
 #undef CHECK_GPR_INDEX
 }
+
+static void noreturn idle_loop(void)
+{
+    BUG_ON("unimplemented");
+}
diff --git a/xen/arch/riscv/entry.S b/xen/arch/riscv/entry.S
index 017fbb06262c..99bc0f4974c2 100644
--- a/xen/arch/riscv/entry.S
+++ b/xen/arch/riscv/entry.S
@@ -147,3 +147,42 @@ FUNC(__context_switch)
 
         ret
 END(__context_switch)
+
+/*
+ * Enter a vCPU for the first time. Must be called with interrupts disabled,
+ * see continue_new_vcpu().
+ */
+FUNC(return_to_new_vcpu)
+        REG_L   t0, CPU_USER_REGS_HSTATUS(sp)
+        csrw    CSR_HSTATUS, t0
+
+        REG_L   t0, CPU_USER_REGS_SEPC(sp)
+        csrw    CSR_SEPC, t0
+
+        /* .a0 holds the hart id */
+        REG_L   a0, CPU_USER_REGS_A0(sp)
+
+        /*
+         * .a1 holds the DTB address for the boot vCPU, or the opaque value
+         * passed to SBI HSM hart_start() for secondary vCPUs
+         */
+        REG_L   a1, CPU_USER_REGS_A1(sp)
+
+        /* Return to (V)S-mode, with interrupts re-enabled by sret */
+        li      t0, SSTATUS_SPP | SSTATUS_SPIE
+        csrs    CSR_SSTATUS, t0
+
+        /*
+         * sret doesn't switch stacks and leaves the GPRs alone, so every
+         * register which isn't meaningful to the vCPU being started has to be
+         * cleared here: otherwise the guest would see Xen's values, sp (this
+         * vCPU's Xen stack) and ra among them.
+         */
+        .irp reg, ra, sp, gp, tp, t0, t1, t2, s0, s1, a2, a3, a4, a5, a6, a7, \
+                  s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, t3, t4, t5, t6
+        mv      \reg, zero
+        .endr
+
+        /* Enter guest */
+        sret
+END(return_to_new_vcpu)
diff --git a/xen/arch/riscv/include/asm/current.h b/xen/arch/riscv/include/asm/current.h
index 78ec52fd8a35..f8babcc3d926 100644
--- a/xen/arch/riscv/include/asm/current.h
+++ b/xen/arch/riscv/include/asm/current.h
@@ -47,6 +47,8 @@ DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
 #define set_current(vcpu)  do { current = (vcpu); } while (0)
 #define get_cpu_current(cpu)  per_cpu(curr_vcpu, cpu)
 
+#define get_cpu_info() (current->arch.cpu_info)
+
 #define guest_cpu_user_regs() ({ BUG_ON("unimplemented"); NULL; })
 #define vcpu_guest_cpu_user_regs(vcpu) \
     (&(vcpu)->arch.cpu_info->guest_cpu_user_regs)
@@ -58,6 +60,8 @@ DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
     unreachable();                                          \
 } while ( false )
 
+#define reset_stack_and_jump(fn) switch_stack_and_jump(get_cpu_info(), fn)
+
 #define get_per_cpu_offset() __per_cpu_offset[smp_processor_id()]
 
 #endif /* __ASSEMBLER__ */
-- 
2.55.0

[PATCH v3 38/39] xen/riscv: set the guest's XLEN explicitly in hstatus.VSXL

[Oleksii Kurochko]

The reset value of hstatus.VSXL isn't architecturally defined, so it
has to be programmed explicitly from the domain's type instead of
relying on whatever the hardware happens to leave there.

This matters beyond the guest's own view of itself: decoding of a
trapped instruction depends on the effective XLEN of the guest, as the
encodings which exist for XLEN=64 only must not be recognized for a
32-bit one.

The field is WARL: an implementation is permitted to make it read-only
with a value that always keeps VSXLEN == HSXLEN. Probe at boot which
encodings can really be set, and refuse to construct a domain whose
width the hardware cannot provide, rather than silently running it at
the wrong XLEN.

The field occupies bits 33:32 and hence doesn't exist when HSXLEN=32,
where VSXLEN is always 32 and HSTATUS_VSXL isn't even defined. Hence
the split on CONFIG_RISCV_32, even though RV32 isn't buildable yet.

The domain's type is only established by set_domain_type(), that is
once its kernel image has been probed, whereas vCPU0 is created before
that. vcpu_csr_init() therefore leaves the field zero, and
vcpu_set_vsxl() fills it in from continue_new_vcpu() instead: that runs
once per vCPU with the type already settled, right before
return_to_new_vcpu() loads the saved hstatus into the CSR.

By then the domain is built and about to run, so it is too late to
reject a width the hardware cannot provide. construct_domain()
therefore checks it up front, before any of the domain's images are
loaded.

domain_vsxl() only maps a domain type onto the encoding, and
hstatus_vsxl_settable() tells whether the hardware accepts it. The
switch() in domain_vsxl() deliberately has no default case, so that
adding a 128-bit domain type fails to build until the mapping is
updated.

While at it drop HSTATUS_VSXL_SHIFT, which has no users and isn't
expected to gain any.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in v3:
 - Derive the width from the domain's type instead of hardcoding
   HSTATUS_VSXL_64, now that the preceding patches make d->type correct.
 - Probe at boot which VSXL encodings the hardware really accepts, and
   refuse in construct_domain() a domain whose width it cannot provide
   instead of silently running the guest at the wrong XLEN.
 - Move the write out of vcpu_csr_init(), which runs before the domain's
   type is known, into vcpu_set_vsxl() called from continue_new_vcpu().
   As a consequence the patch is moved after the one implementing
   continue_new_vcpu().
 - Handle RV32, where the field doesn't exist at all and VSXLEN is
   always 32.
 - Use XLEN_FIELD_{32,64} introduced earlier in the series instead of
   adding HSTATUS_VSXL_{32,64}: it is the same encoding, and unlike the
   latter it isn't hidden behind #if __riscv_xlen == 64, which the RV32
   paths need.
 - Drop HSTATUS_VSXL_SHIFT, which has no users.
 - Rework the commit message accordingly.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/domain-build.c               |   8 ++
 xen/arch/riscv/domain.c                     | 119 ++++++++++++++++++++
 xen/arch/riscv/include/asm/domain.h         |   3 +
 xen/arch/riscv/include/asm/riscv_encoding.h |   1 -
 xen/arch/riscv/include/asm/setup.h          |   2 +
 xen/arch/riscv/setup.c                      |   2 +
 6 files changed, 134 insertions(+), 1 deletion(-)

diff --git a/xen/arch/riscv/domain-build.c b/xen/arch/riscv/domain-build.c
index 1e3abe259ccd..1acaf0946130 100644
--- a/xen/arch/riscv/domain-build.c
+++ b/xen/arch/riscv/domain-build.c
@@ -15,9 +15,17 @@ int __init construct_domain(struct domain *d, struct kernel_info *kinfo)
 {
     struct vcpu *v = d->vcpu[0];
     struct cpu_user_regs *regs = vcpu_guest_cpu_user_regs(v);
+    unsigned int vsxl = domain_vsxl(d);
 
     BUG_ON(v->is_initialised);
 
+    /* Refuse the domain before any of its images are loaded. */
+    if ( !hstatus_vsxl_settable(vsxl) )
+    {
+        printk(XENLOG_ERR "%pd: hstatus.VSXL cannot be set to %u\n", d, vsxl);
+        return -EOPNOTSUPP;
+    }
+
     /*
      * At the moment *_load() don't return value and will just panic()
      * inside.
diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index 04be628dcdb4..9155ff4234e8 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -51,6 +51,119 @@ static struct csr_masks __ro_after_init csr_masks;
 #define HENVCFG_VALID_MASK 0xe0000003000000ffUL
 #define HSTATEEN0_VALID_MASK 0xde00000000000007UL
 
+/*
+ * Return the hstatus.VSXL value encoding the guest's XLEN. Whether the
+ * hardware can actually provide that width is a separate question, answered
+ * by hstatus_vsxl_settable().
+ *
+ * The switch() deliberately has no default case, so that adding a new
+ * domain_type (a 128-bit one, in particular) fails to build until this
+ * mapping is updated.
+ */
+unsigned int domain_vsxl(const struct domain *d)
+{
+    switch ( d->type )
+    {
+    case DOMAIN_32BIT:
+        return XLEN_FIELD_32;
+
+    case DOMAIN_64BIT:
+        return XLEN_FIELD_64;
+    }
+
+    ASSERT_UNREACHABLE();
+
+    /* A reserved encoding, which hstatus_vsxl_settable() never accepts. */
+    return 0;
+}
+
+#ifdef CONFIG_RISCV_32
+
+/*
+ * hstatus.VSXL occupies bits 33:32 and hence does not exist when HSXLEN=32.
+ * VSXLEN is then always 32, so there is nothing to probe and no width other
+ * than 32 to accept.
+ */
+void __init init_hstatus_vsxl_settable_mask(void)
+{
+}
+
+bool hstatus_vsxl_settable(unsigned int vsxl)
+{
+    return vsxl == XLEN_FIELD_32;
+}
+
+static void vcpu_set_vsxl(struct vcpu *v)
+{
+    ASSERT(is_32bit_domain(v->domain));
+}
+
+#else /* CONFIG_RISCV_64 */
+
+/*
+ * Bitmap of the hstatus.VSXL values this hardware accepts, indexed by field
+ * value. Not one of csr_masks: those are bit masks ANDed into a CSR write,
+ * this is a set of legal values of a single WARL field.
+ */
+static unsigned int __ro_after_init hstatus_vsxl_settable_mask;
+
+/*
+ * hstatus.VSXL is WARL and, in particular, an implementation is permitted to
+ * make it read-only with a value that always keeps VSXLEN == HSXLEN. Probe
+ * once which values can really be set, so that a domain whose width the
+ * hardware cannot provide is refused rather than silently run at the wrong
+ * XLEN.
+ *
+ * Every encoding the field can hold is probed, XLEN_FIELD_128 included, so
+ * that a guest width added later needs no change here.
+ *
+ * Like the CSR masks below this assumes all harts behave the same way.
+ */
+void __init init_hstatus_vsxl_settable_mask(void)
+{
+    register_t hstatus = csr_read(CSR_HSTATUS);
+
+    /* Encoding 0 is reserved, hence starting at the narrowest width. */
+    for ( unsigned int vsxl = XLEN_FIELD_32; vsxl <= XLEN_FIELD_128; vsxl++ )
+    {
+        register_t test = (hstatus & ~HSTATUS_VSXL) |
+                          MASK_INSR(vsxl, HSTATUS_VSXL);
+
+        csr_write(CSR_HSTATUS, test);
+
+        if ( MASK_EXTR(csr_read(CSR_HSTATUS), HSTATUS_VSXL) == vsxl )
+            hstatus_vsxl_settable_mask |= BIT(vsxl, U);
+    }
+
+    csr_write(CSR_HSTATUS, hstatus);
+}
+
+/*
+ * Whether the hardware really accepts the given hstatus.VSXL value, i.e.
+ * whether a guest of that XLEN can be run at all.
+ */
+bool hstatus_vsxl_settable(unsigned int vsxl)
+{
+    return hstatus_vsxl_settable_mask & BIT(vsxl, U);
+}
+
+/*
+ * Set the guest's XLEN explicitly rather than leaving it to the WARL
+ * behaviour of hstatus.VSXL.
+ */
+static void vcpu_set_vsxl(struct vcpu *v)
+{
+    struct cpu_user_regs *regs = vcpu_guest_cpu_user_regs(v);
+    unsigned int vsxl = domain_vsxl(v->domain);
+
+    ASSERT(hstatus_vsxl_settable(vsxl));
+
+    regs->hstatus = (regs->hstatus & ~HSTATUS_VSXL) |
+                    MASK_INSR(vsxl, HSTATUS_VSXL);
+}
+
+#endif /* CONFIG_RISCV_32 */
+
 void __init init_csr_masks(void)
 {
     /*
@@ -91,6 +204,10 @@ static void vcpu_csr_init(struct vcpu *v)
 {
     v->arch.hedeleg = HEDELEG_DEFAULT & csr_masks.hedeleg;
 
+    /*
+     * hstatus.VSXL is left zero here: the domain's type, and hence the
+     * guest's XLEN, is only known once its kernel image has been probed.
+     */
     vcpu_guest_cpu_user_regs(v)->hstatus = HSTATUS_SPV | HSTATUS_SPVP;
 
     v->arch.hideleg = HIDELEG_DEFAULT & csr_masks.hideleg;
@@ -149,6 +266,8 @@ static void continue_new_vcpu(struct vcpu *prev)
     if ( is_idle_vcpu(current) )
         reset_stack_and_jump(idle_loop);
 
+    vcpu_set_vsxl(current);
+
     /*
      * return_to_new_vcpu() sets up hstatus.SPV, sstatus.SPP and sepc so
      * that sret enters the guest in VS-mode. A trap taken in HS-mode
diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
index a3fa637b40da..f8fd5964fd74 100644
--- a/xen/arch/riscv/include/asm/domain.h
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -146,6 +146,9 @@ int vcpu_unset_interrupt(struct vcpu *v, unsigned int irq);
 void vcpu_sync_interrupts(struct vcpu *curr);
 void vcpu_flush_interrupts(struct vcpu *curr);
 
+bool hstatus_vsxl_settable(unsigned int vsxl);
+unsigned int domain_vsxl(const struct domain *d);
+
 #endif /* ASM__RISCV__DOMAIN_H */
 
 /*
diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h
index 1af309aa4ad5..578d772d23df 100644
--- a/xen/arch/riscv/include/asm/riscv_encoding.h
+++ b/xen/arch/riscv/include/asm/riscv_encoding.h
@@ -75,7 +75,6 @@
 
 #if __riscv_xlen == 64
 #define HSTATUS_VSXL			_UL(0x300000000)
-#define HSTATUS_VSXL_SHIFT		32
 #endif
 #define HSTATUS_VTSR			_UL(0x00400000)
 #define HSTATUS_VTW			_UL(0x00200000)
diff --git a/xen/arch/riscv/include/asm/setup.h b/xen/arch/riscv/include/asm/setup.h
index 73ce2f293348..ec36cecad50f 100644
--- a/xen/arch/riscv/include/asm/setup.h
+++ b/xen/arch/riscv/include/asm/setup.h
@@ -11,6 +11,8 @@ void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len);
 
 void init_csr_masks(void);
 
+void init_hstatus_vsxl_settable_mask(void);
+
 #endif /* ASM__RISCV__SETUP_H */
 
 /*
diff --git a/xen/arch/riscv/setup.c b/xen/arch/riscv/setup.c
index 05f93a74c9d3..fd699defcfc3 100644
--- a/xen/arch/riscv/setup.c
+++ b/xen/arch/riscv/setup.c
@@ -149,6 +149,8 @@ void __init noreturn start_xen(unsigned long bootcpu_id,
 
     init_csr_masks();
 
+    init_hstatus_vsxl_settable_mask();
+
     preinit_xen_time();
 
     intc_preinit();
-- 
2.55.0

[PATCH v3 39/39] xen/riscv: introduce IMSIC h/w interrupt file attaching to vcpu

[Oleksii Kurochko]

Introduce imsic_vsfile_attach() to give a vCPU a working guest
interrupt file.

A guest (VS) interrupt file has to be backed by one of the hardware
guest interrupt files of the pCPU the vCPU runs on, so that pCPU has to
be known first. arch_vcpu_create() is therefore not a suitable place:
the pCPU assigned to a vCPU can still change before the vCPU is first
scheduled, which would mean taking another file and remapping the
guest's IMSIC page to it. Call imsic_vsfile_attach() from
continue_new_vcpu() instead, i.e. when the vCPU is scheduled for the
first time; as that isn't an __init context, neither is
imsic_vsfile_attach().

Giving a vCPU its first file shares most of the steps with moving it to
a new one on migration, so move the body of imsic_migrate_vcpu() to
imsic_vsfile_move() and use it for both. Its first_attach argument skips
the part which moves the interrupt state out of the old file: the vCPU
has never run at that point, so the freshly acquired and zeroed file is
all the interrupt state it has, and hstatus.VGEIN can be pointed at it
right away.

vAPLIC target registers are reconfigured on the first attach as well: a
target the guest writes for a vCPU which has no h/w guest interrupt file
yet only updates the stored copy, which reaches the h/w at this point.

A vCPU which has already run is left without a h/w file only when its
first attach failed, which crashed its domain. Such a vCPU can still be
migrated, e.g. by an affinity change or a pCPU going offline, until the
domain is destroyed, so don't panic() for it, which would take the whole
host down: return early. Should a vCPU of a domain which isn't shutting
down ever get there, crash that domain rather than the host.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
imsic_vsfile_move() isn't split out of imsic_migrate_vcpu() earlier in the
series: until this patch imsic_migrate_vcpu() would be its only user, so the
first_attach path would have no caller. The first attach itself can't come
earlier either, as it is done from continue_new_vcpu(), introduced by
"xen/riscv: implement continue_new_vcpu()".
---
Changes in v3:
 - imsic_update_state() is now implemented by "xen/riscv: remap
   interrupts to new IMSIC VS-file".
 - Share the code of imsic_vsfile_attach() and imsic_migrate_vcpu() in
   imsic_vsfile_move(), whose first_attach argument skips the part moving
   state out of the old file.
 - Call vaplic_reconfigure_target() on the first attach as well, so that
   vAPLIC targets the guest wrote for the vCPU before it got its file reach
   the h/w.
 - Replace panic() for a vCPU left without a h/w file by domain_crash():
   it can only happen to a vCPU whose first imsic_vsfile_acquire() failed
   and crashed its domain, and such a vCPU can still be migrated.
 - imsic_vsfile_acquire(), together with the vgein_release() on a
   mapping failure, is now introduced by "xen/riscv: remap interrupts to
   new IMSIC VS-file", without the cpu argument (the file is always
   taken on v->processor).
 - Include asm/imsic.h for imsic_vsfile_attach().
 - Drop imsic_nr_eix() in favour of imsic_cfg.nr_eix.
 - Drop vcpu_set_vgein(): with the code shared in imsic_vsfile_move(),
   hstatus.VGEIN is updated in one place only.
 - Rework the commit message: drop the stale vcpu_aia_init() name and
   the claim that the IMSIC state has no concurrent readers yet.
---
Changes in v2:
 - Update vcpu_aia_init() to catch sw interrupt file and update some
   debug messages in it.
 - Add vgein_release() if IMSIC h/w mapping failed.
 - imsic_update_state(): store v->processor rather than
   cpuid_to_hartid(), as ->vsfile_cpu is consumed as a Xen CPU id
   (aplic_hart_field(), cpumask_of()) and its NR_CPUS sentinel lives in
   that numbering space.
 - Drop parentheses around guest_file_id ? ... in imsic_update_state().
 - Rename vcpu_aia_init to imsic_vsfile_attach() and move the code to
   imsic.c.
---
---
 xen/arch/riscv/domain.c            |  3 ++
 xen/arch/riscv/imsic.c             | 79 +++++++++++++++++++++++-------
 xen/arch/riscv/include/asm/imsic.h |  2 +
 3 files changed, 66 insertions(+), 18 deletions(-)

diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
index 9155ff4234e8..e8ae62626b76 100644
--- a/xen/arch/riscv/domain.c
+++ b/xen/arch/riscv/domain.c
@@ -13,6 +13,7 @@
 #include <asm/csr.h>
 #include <asm/current.h>
 #include <asm/gpr-num.h>
+#include <asm/imsic.h>
 #include <asm/intc.h>
 #include <asm/mmio.h>
 #include <asm/riscv_encoding.h>
@@ -268,6 +269,8 @@ static void continue_new_vcpu(struct vcpu *prev)
 
     vcpu_set_vsxl(current);
 
+    imsic_vsfile_attach(current);
+
     /*
      * return_to_new_vcpu() sets up hstatus.SPV, sstatus.SPP and sepc so
      * that sret enters the guest in VS-mode. A trap taken in HS-mode
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index a8e99997c087..bb8cdf437a6a 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -1251,7 +1251,14 @@ static unsigned int imsic_vsfile_acquire(struct vcpu *v)
     return vsfile_id;
 }
 
-void imsic_migrate_vcpu(struct vcpu *v)
+/*
+ * Give the vCPU a h/w guest interrupt file on the pCPU it runs on and move
+ * all its interrupt producers and state there.
+ *
+ * @first_attach tells that the vCPU is only now given its very first file,
+ * so there is no old file to move anything out of.
+ */
+static void imsic_vsfile_move(struct vcpu *v, bool first_attach)
 {
     struct imsic_vsfile_data vsfile_data = {
         .nr_eix = imsic_cfg.nr_eix,
@@ -1262,33 +1269,38 @@ void imsic_migrate_vcpu(struct vcpu *v)
     unsigned int old_vsfile_id;
     unsigned int old_vsfile_cpu;
 
-    /*
-     * The scheduler can mark a freshly created vCPU's unit as migrated and
-     * invoke this before the vCPU has ever run (see the migrated branch in
-     * schedule()). No need to do migration for such vCPUs as they aren't fully
-     * initialized (for example, context_switch() will be called after
-     * imsic_migrate_vcpu()).
-     */
-    if ( v->arch.last_cpu == CPU_NONE )
-        return;
-
     read_lock_irqsave(&imsic_state->vsfile_lock, flags);
     old_vsfile_id = imsic_state->guest_file_id;
     old_vsfile_cpu = imsic_state->vsfile_cpu;
     read_unlock_irqrestore(&imsic_state->vsfile_lock, flags);
 
+    ASSERT(!first_attach || old_vsfile_cpu == CPU_NONE);
+
     /*
-     * We don't support SW interrupt files at the moment. Bail out before
-     * anything is touched, as the old file has no owning pCPU in that case
-     * and there is nothing to retarget the producers away from.
+     * With no h/w file owned, the vCPU is either given its first file or
+     * uses a s/w file, and s/w files aren't supported at the moment.
+     *
+     * A vCPU which has already run has no h/w file only if its first
+     * imsic_vsfile_acquire() failed and crashed its domain. The vCPU can
+     * still be migrated until the domain is destroyed, so return early, as
+     * there is no old file to move anything out of. Should that ever happen
+     * for a domain which isn't shutting down, crash it rather than the host;
+     * for one which already is, domain_crash() does nothing but log.
+     *
+     * TODO: once s/w files are supported, move the state out of the s/w file
+     * here instead.
      */
-    if ( old_vsfile_cpu == CPU_NONE )
-        panic("IMSIC SW-file isn't supported\n");
+    if ( old_vsfile_cpu == CPU_NONE && !first_attach )
+    {
+        ASSERT(domain_shutting_down(v->domain));
+        domain_crash(v->domain, "%pv: IMSIC s/w VS-file isn't supported\n", v);
+        return;
+    }
 
     /*
      * At this point, all interrupt producers are still using the old IMSIC
-     * VS-file. Allocate and clear the new one before redirecting anything
-     * to it.
+     * VS-file, if any. Allocate and clear the new one before redirecting
+     * anything to it.
      */
 
     /* Zero-out, map and start to use the new IMSIC VS-file */
@@ -1310,6 +1322,13 @@ void imsic_migrate_vcpu(struct vcpu *v)
      */
     vaplic_reconfigure_target(v);
 
+    /*
+     * The vCPU has never run yet, so the just zeroed out file is all the
+     * interrupt state it has and HSTATUS.VGEIN can be pointed at it at once.
+     */
+    if ( first_attach )
+        goto set_vgein;
+
     /*
      * Synchronizing interactions between a hart and the APLIC.
      *
@@ -1349,8 +1368,32 @@ void imsic_migrate_vcpu(struct vcpu *v)
     if ( !v->is_running )
         imsic_call_on_cpu(v->processor, imsic_local_hgeie_set, &vsfile_data);
 
+ set_vgein:
     /* Set VCPU HSTATUS.VGEIN to new IMSIC VS-file */
     vcpu_guest_cpu_user_regs(v)->hstatus &= ~HSTATUS_VGEIN;
     vcpu_guest_cpu_user_regs(v)->hstatus |=
         MASK_INSR(vsfile_data.hgei, HSTATUS_VGEIN);
 }
+
+void imsic_migrate_vcpu(struct vcpu *v)
+{
+    /*
+     * The scheduler can mark a freshly created vCPU's unit as migrated and
+     * invoke this before the vCPU has ever run (see the migrated branch in
+     * schedule()). No need to do migration for such vCPUs as they aren't fully
+     * initialized (for example, context_switch() will be called after
+     * imsic_migrate_vcpu()).
+     */
+    if ( v->arch.last_cpu == CPU_NONE )
+        return;
+
+    imsic_vsfile_move(v, false);
+}
+
+void imsic_vsfile_attach(struct vcpu *v)
+{
+    if ( !aia_usable() )
+        return;
+
+    imsic_vsfile_move(v, true);
+}
diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
index e08b041eebbd..118c3dc94a17 100644
--- a/xen/arch/riscv/include/asm/imsic.h
+++ b/xen/arch/riscv/include/asm/imsic.h
@@ -118,4 +118,6 @@ int imsic_map_guest_file(struct vcpu *v, unsigned int vsfile_id);
 
 void imsic_migrate_vcpu(struct vcpu *v);
 
+void imsic_vsfile_attach(struct vcpu *v);
+
 #endif /* ASM_RISCV_IMSIC_H */
-- 
2.55.0

Re: [PATCH v3 02/39] xen/riscv: set the domain type for 64-bit kernel images

[Baptiste Le Duc]

On Wed, 30 Sep 2026 18:18:44 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
> kernel_image64_probe() never set info->type, so every domain kept the
> DOMAIN_32BIT value a zeroed KERNEL_INFO_INIT leaves behind, even though
> Xen on RISC-V is only built for RV64 and accepts nothing but a 64-bit
> Image.
> 
> Nothing consumes d->type on RISC-V yet, so there is no functional change.
> 
> [...]

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 01/39] xen/riscv: drop INSN_{16,32}BIT_MASK and report unknown instruction lengths

[Baptiste Le Duc]

On Wed, 30 Sep 2026 18:18:43 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
> Both constants were inherited from OpenSBI and their names are misleading.
> The length of an instruction is encoded in its low bits: bits[1:0] != 11
> means 16 bits, otherwise bits[4:2] != 111 means 32 bits, and anything else
> is 48 bits or wider. 0x3 and 0x1c are the patterns compared against in
> these two tests, not masks selecting a 16-bit or a 32-bit instruction. As
> each constant is needed in one place only, once INSN_IS_16BIT() and
> INSN_IS_32BIT() encapsulate the tests, drop them and spell the values out
> in those two macros instead.
> 
> [...]

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Baptiste Le Duc]

> On RV32 a 64-bit CSR is accessed as two halves, <csr> and <csr>H. Reading
As far as I can tell RV32 can't currently be selected (RISCV_64 is def_bool y,
no RISCV_32 in Kconfig), so this path can't even be build-tested. Is the retry
loop really needed now, or a least in this patch series?

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 04/39] xen/riscv: request a G-stage flush on vmenter when VMIDs are disabled

[Baptiste Le Duc]

On Wed, 30 Sep 2026 18:18:46 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
> vmid_handle_vmenter() reports that no flush is needed when VMIDs are
> unavailable (e.g. vmid=off, or hardware with no more than one VMID bit).
> Every domain then runs under VMID 0 with nothing flushed in between, so as
> soon as a hart runs more than one domain, a domain entered there can use
> the G-stage translations left behind by the domain which ran before it.
> 
> Report that a flush is needed on every entry instead.
> 
> [...]

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Baptiste Le Duc]

> A RISC-V guest can be given several emulated devices at distinct GPA
> ranges; the virtual interrupt controllers alone account for vPLIC on
> legacy machines and vAPLIC together with vIMSIC on AIA-compliant ones.
> Routing MMIO faults via a per-device is_access() check in the trap
> handler would couple that handler to every device it must serve,
> requiring a new conditional branch in the fault path for each emulated
> device added.
> 
> Introduce a per-domain MMIO handler registration table, modeled
> after the equivalent ARM framework, so that virtual devices
> self-register their GPA ranges and read/write callbacks at domain
> creation time. The MMIO fault path delegates to a single do_mmio()
> entry point and remains agnostic of which device owns a particular
> address.
> 
> Nothing registers a handler and do_mmio() has no callers yet, so this
> patch is a no-op; the trap handler is left untouched.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> do_mmio() gets its first caller in "xen/riscv: add guest load emulation
> for trapped MMIO accesses" in this series.
> ---
> Changes in v3:
>  - Update the commit message.
>  - Capitalize the comments inside mmio_info_t and drop the list of access
>    widths from the one on len, as it could go stale.
>  - Drop the parentheses around the typedef names.
>  - Reject an access width which isn't a power of 2 in do_mmio().
>  - Fold the !size check of register_mmio_handler() into end <= addr.
>  - Return -EADDRINUSE instead of -EEXIST for an overlapping region.
>  - Drop setting d->arch.vmmio.num_entries to 0, as struct domain starts
>    out zero-filled.
> ---
> Changes in v2:
>  - Drop copyright from mmio.c as it will go stale anyway as code moves
>    around.
>  - Drop the max_count parameter of domain_io_init() (MAX_IO_HANDLER is a
>    global boundary) and embed the handler array directly in struct vmmio
>    as struct mmio_handler handlers[MAX_IO_HANDLER].  This removes the
>    xvzalloc_array() allocation, the max_num_entries field and
>    domain_io_free() altogether; domain_io_init() consequently cannot fail
>    any longer and now returns void. Note this goes slightly beyond the
>    suggested variant in that max_num_entries is dropped as well, since
>    ARRAY_SIZE(vmmio->handlers) serves the same purpose.
>  - Drop the separate register_t argument of mmio_read_t/mmio_write_t;
>    handlers now produce and consume the value through info->data.
>    handle_read()/handle_write() are gone as a result, with
>    try_handle_mmio() invoking ops->read()/ops->write() directly.
>  - Turn mmio_read_t/mmio_write_t into function types rather than
>    pointer-to-function types, so that pointer-ness is visible at the use
>    sites in struct mmio_handler_ops. Constify the mmio_info_t * of the
>    write callback, which has no reason to modify it any more.
>  - register_mmio_handler() returns int instead of BUG_ON()ing on a full
>    table: -ENOSPC now lets the caller fail domain creation. It also
>    validates its inputs, rejecting a NULL ops (or one with a missing
>    read/write callback) as well as zero-sized and address-wrapping
>    regions with -EINVAL.
>  - Guarantee the non-overlap property that cmp_mmio_handler() relies on:
>    register_mmio_handler() checks the new region against both neighbours
>    of its insertion slot and returns -EEXIST on overlap.
>  - Replace the sort() call per registration with an insertion into the
>    already sorted array: locate the slot and memmove() the tail up by
>    one. sort(), swap_mmio_handler() and <xen/sort.h> are gone.
>  - Extend cmp_mmio_handler()'s comment to state that it is a bsearch()
>    comparator and to explain the key/elem asymmetry; document why the
>    neighbours' addr + size cannot overflow.
>  - Fix over-long lines and a mis-indented label.
> ---
> ---
>  xen/arch/riscv/Makefile             |   1 +
>  xen/arch/riscv/domain.c             |   3 +
>  xen/arch/riscv/include/asm/domain.h |   3 +
>  xen/arch/riscv/include/asm/mmio.h   |  63 ++++++++++
>  xen/arch/riscv/mmio.c               | 175 ++++++++++++++++++++++++++++
>  5 files changed, 245 insertions(+)
>  create mode 100644 xen/arch/riscv/include/asm/mmio.h
>  create mode 100644 xen/arch/riscv/mmio.c
>
> diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
> index 665ec26933..5454517724 100644
> --- a/xen/arch/riscv/Makefile
> +++ b/xen/arch/riscv/Makefile
> @@ -14,6 +14,7 @@ obj-y += intc.o
>  obj-y += irq.o
>  obj-y += kernel.init.o
>  obj-y += mm.o
> +obj-y += mmio.o
>  obj-y += p2m.o
>  obj-y += paging.o
>  obj-y += pt.o
> diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
> index d94652809e..fd62296356 100644
> --- a/xen/arch/riscv/domain.c
> +++ b/xen/arch/riscv/domain.c
> @@ -12,6 +12,7 @@
>  #include <asm/cpufeature.h>
>  #include <asm/csr.h>
>  #include <asm/intc.h>
> +#include <asm/mmio.h>
>  #include <asm/riscv_encoding.h>
>  #include <asm/vtimer.h>
>  
> @@ -310,6 +311,8 @@ int arch_domain_create(struct domain *d,
>      if ( (rc = p2m_init(d, config)) != 0)
>          goto fail;
>  
> +    domain_io_init(d);
> +
>      if ( (rc = domain_vintc_init(d)) )
>          goto fail;
>  
> diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
> index e035b33ddf..15e8fa1968 100644
> --- a/xen/arch/riscv/include/asm/domain.h
> +++ b/xen/arch/riscv/include/asm/domain.h
> @@ -9,6 +9,7 @@
>  
>  #include <asm/cpufeature.h>
>  #include <asm/guest-layout.h>
> +#include <asm/mmio.h>
>  #include <asm/p2m.h>
>  #include <asm/vtimer.h>
>  
> @@ -101,6 +102,8 @@ struct arch_domain {
>      const unsigned long *isa;
>  
>      struct vintc *vintc;
> +
> +    struct vmmio vmmio;
>  };
>  
>  #include <xen/sched.h>
> diff --git a/xen/arch/riscv/include/asm/mmio.h b/xen/arch/riscv/include/asm/mmio.h
> new file mode 100644
> index 0000000000..1bf48571a9
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/mmio.h
> @@ -0,0 +1,63 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +#ifndef RISCV_MMIO_H
Should be ASM__RISCV_MMIO_H instead to be consistant with other header
files in riscv/include/asm.
> +#define RISCV_MMIO_H
> +
> +#include <xen/lib.h>
> +#include <xen/rwlock.h>
> +
> +struct domain;
> +struct vcpu;
> +
> +#define MAX_IO_HANDLER  16
> +
> +typedef struct {
> +    paddr_t gpa;
> +    unsigned int len;  /* Access width in bytes */
> +    bool is_write;
> +    /* Store: value to write; load: value read (set by handler) */
> +    register_t data;
> +} mmio_info_t;
> +
> +enum io_state
> +{
Style: should be directly after enum io_state.
> +    IO_ABORT,       /* The IO was handled and led to an abort. */
> +    IO_HANDLED,     /* The IO was successfully handled. */
> +    IO_UNHANDLED,   /* No handler found for the IO. */
> +};
> +
> +typedef enum io_state mmio_read_t(struct vcpu *v, mmio_info_t *info);
> +typedef enum io_state mmio_write_t(struct vcpu *v, const mmio_info_t *info);
> +
> +struct mmio_handler_ops {
> +    mmio_read_t *read;
> +    mmio_write_t *write;
> +};
> +
> +struct mmio_handler {
> +    paddr_t addr;
> +    paddr_t size;
> +    const struct mmio_handler_ops *ops;
> +};
> +
> +struct vmmio {
> +    unsigned int num_entries;
> +    rwlock_t lock;
> +    struct mmio_handler handlers[MAX_IO_HANDLER];
> +};
> +
> +int do_mmio(mmio_info_t *info, paddr_t fault_addr, unsigned int len);
> +int register_mmio_handler(struct domain *d,
> +                          const struct mmio_handler_ops *ops,
> +                          paddr_t addr, paddr_t size);
> +void domain_io_init(struct domain *d);
> +
> +#endif /* RISCV_MMIO_H */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/riscv/mmio.c b/xen/arch/riscv/mmio.c
> new file mode 100644
> index 0000000000..cf739a790a
> --- /dev/null
> +++ b/xen/arch/riscv/mmio.c
> @@ -0,0 +1,175 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +
> +#include <xen/bsearch.h>
> +#include <xen/lib.h>
> +#include <xen/rwlock.h>
> +#include <xen/sched.h>
> +#include <xen/string.h>
> +
> +#include <asm/current.h>
> +#include <asm/mmio.h>
> +
> +/*
> + * bsearch() comparator: @key holds the address to look up in its addr field,
> + * @elem is an entry of vmmio->handlers. Relies on the regions not
> + * overlapping, which register_mmio_handler() enforces.
> + */
> +static int cmp_mmio_handler(const void *key, const void *elem)
> +{
> +    const struct mmio_handler *handler0 = key;
> +    const struct mmio_handler *handler1 = elem;
> +
> +    if ( handler0->addr < handler1->addr )
> +        return -1;
> +
> +    if ( handler0->addr >= (handler1->addr + handler1->size) )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +/*
> + * Return a copy of the matching handler rather than a pointer into
> + * vmmio->handlers: a concurrent register_mmio_handler() shifts entries
> + * up to keep the array sorted, so an escaped pointer could refer to a
> + * different (or torn) entry once the lock is dropped. The copy stays
> + * valid as the ops structures are never freed.
> + */
> +static bool find_mmio_handler(struct domain *d, paddr_t gpa,
> +                              struct mmio_handler *out)
> +{
> +    struct vmmio *vmmio = &d->arch.vmmio;
> +    struct mmio_handler key = { .addr = gpa };
> +    const struct mmio_handler *handler;
> +
> +    read_lock(&vmmio->lock);
> +    handler = bsearch(&key, vmmio->handlers, vmmio->num_entries,
> +                      sizeof(*handler), cmp_mmio_handler);
> +    if ( handler )
> +        *out = *handler;
> +    read_unlock(&vmmio->lock);
> +
> +    return handler != NULL;
> +}
> +
> +static enum io_state try_handle_mmio(mmio_info_t *info)
> +{
> +    struct vcpu *v = current;
> +    struct mmio_handler handler = {};
> +
> +    if ( !find_mmio_handler(v->domain, info->gpa, &handler) )
> +        return IO_UNHANDLED;
> +
> +    if ( info->is_write )
> +        return handler.ops->write(v, info);
> +    else
The else is not necessary here, misra would call that "dead code".
Here are two suggestions:

      if ( info->is_write )
          return handler.ops->write(v, info);

      return handler.ops->read(v, info);
or:

      return info->is_write ? handler.ops->write(v, info)
                            : handler.ops->read(v, info);

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 07/39] xen/riscv: introduce CPU_NONE

[Baptiste Le Duc]

> Introduce CPU_NONE, a CPU number which doesn't identify any physical
> CPU, to be used wherever a pCPU has to be recorded but there may be
> none.
> 
> Its first user is vsfile_cpu in struct vimsic_state, which records the
> pCPU whose h/w guest interrupt file backs the vCPU's IMSIC VS-file.
> NR_CPUS has been used there as the value meaning that there is no such
> pCPU, i.e. that the s/w VS-file is in use. NR_CPUS is however an upper
Nit: It's hard to understand when reading the message why s/w is mentioning.
I'd reword it to:
    ```
    NR_CPUS has been used there to mean that no h/w guest interrupt file is
    attached (i.e. the vCPU uses the s/w VS-file), so there is no pCPU to
    record.
    ```
> bound rather than a "no CPU" marker, and it already carries other
> meanings: hartid_to_cpuid() returns it for a hart Xen doesn't know, and
> pcpu_info[] uses it for an entry whose processor_id isn't valid yet.
> Use CPU_NONE for vsfile_cpu instead, so that the s/w VS-file case can't
> be mistaken for any of those.
Nit: Same as above, I'd prefer something like:
    ```
    Use CPU_NONE for vsfile_cpu instead, so that "no h/w guest interrupt
    file attached" can't be mistaken for an unknown hart or a not-yet-valid
    pcpu_info[] entry.
    ```

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 08/39] xen/riscv: implement virtual APLIC MMIO emulation

[Baptiste Le Duc]

> Guests running under Xen program interrupt routing by writing to APLIC
> MMIO registers. Xen must intercept these accesses to enforce interrupt
> isolation between domains and to translate guest routing intent into the
> underlying physical MSI topology.
> 
> Register the vAPLIC range with the MMIO dispatch and emulate the accesses:
>  - reads and writes are masked with the domain's authorised interrupt
>    bitmap, so that a guest can neither observe nor affect interrupts it
>    does not own;
>  - target registers are shadowed per source, so that a guest reads back
>    what it wrote. The h/w register instead gets the hart index and guest
>    interrupt file of the pCPU the target vCPU's interrupt file lives on,
>    as the APLIC uses these fields directly to compute the MSI address. As
>    long as the target vCPU has no h/w guest interrupt file, only the
>    shadow copy is updated;
>  - only MSI delivery mode is emulated: target writes made while the
>    guest's domaincfg.DM is clear are ignored.
> 
> Delegation (APLIC_SOURCECFG_D) is not yet supported.
> 
> Co-developed-by: Romain Caritey <Romain.Caritey@microchip.com>
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Co-developed-by: needs a matching Signed-off-by: from Romain right after
it.

> ---
> Changes in v3:
> - sourcecfg store emulation: mask the guest value with the new
>   APLIC_SOURCECFG_WMASK (D and SM are the only implemented fields, all the
>   other bits are reserved and read as zero) instead of rejecting the write
>   when the value is above APLIC_SOURCECFG_SM_LEVEL_LOW. The dropped check
>   compared the whole value against 7 rather than the SM field, so a legal
>   SM with anything set in the reserved bits 9:3 was rejected. SM is WARL
>   and vAPLIC keeps no shadow copy of sourcecfg[], so a reserved SM value
>   is left to the h/w to normalize; add a TODO covering what will have to be
>   emulated here once sourcecfg[] is shadowed.
> - Introduce APLIC_SOURCECFG_WMASK in asm/aplic.h.
> - target store emulation: reword the comment about a non-zero guest index
>   to say that such a write is illegal and is therefore ignored as a whole,
>   that the shadow copy keeps the zero it was allocated with, and that
>   aplic_msi_target_gen() overwrites the field with vcpu_guest_file_id()
>   anyway. Drop the now-redundant "Ignore such writes ..." comment.
> - s/aplic_hart_field/aplic_hart_index/ (and the local variable with it):
>   the value is the APLIC "Hart Index", not a field of it.
> - Clarify the MSI target address documentation above aplic_hart_index():
>   name the Guest Index in the address formula, say that the APLIC computes
>   the MSI target address itself in MSI delivery mode, and label the two
>   parts of the 14-bit Hart Index field as g and h to match the surrounding
>   text.
> - Don't add the MSI target address scheme diagram to asm/imsic.h: it
>   would duplicate the one documented above aplic_hart_index().
> - Fix the comment on APLIC_TARGET_IPRIO: the field exists when the target
>   is in direct delivery mode (domaincfg.DM = 0), not "in DM mode".
> - Rearrange the case blocks of vaplic_emulate_load() and
>   vaplic_emulate_store() primarily by register offset: APLIC_DOMAINCFG
>   comes first and APLIC_SOURCECFG handling directly follows it; the
>   set/clr-num registers are also listed in offset order. No functional
>   change.
> - Rename the first parameter of aplic_msi_target_gen() from target_vcpu
>   to v.
> - Use a const-qualified pointer in aplic_hw_read_reg()'s cast to match
>   readl()'s prototype.
> - Put APLIC_xMSICFGADDR_PPN_HHX_MASK(), APLIC_xMSICFGADDR_PPN_LHX_MASK()
>   and APLIC_xMSICFGADDR_PPN_LHX_SHIFT() on a single line each.
> - Use gdprintk() instead of dprintk() for guest-triggered messages in
>   vaplic.c and drop the redundant "is passed" from the word_idx message.
> - Call domain_vaplic_deinit() in both failure paths of
>   domain_vaplic_init() instead of open-coding the clean-up.
> - aplic_msi_target_gen() takes the hart index from the pCPU the vCPU's
>   interrupt file lives on (vsfile_cpu) instead of v->processor, which may
>   already name the new pCPU while the vCPU is being migrated. Callers
>   have to hold vsfile_lock until the value is written to the h/w, which
>   the vAPLIC target write path now does.
> - A target register write for a vCPU which has no h/w guest interrupt file
>   yet (its vsfile_cpu is CPU_NONE) only updates the stored copy; the h/w
>   register isn't programmed, as there is neither a hart nor a guest
>   interrupt file to point it at. Add a TODO about s/w guest interrupt
>   files, for which the h/w register will have to be written too.
> - Turn vaplic_regs.target[] into vaplic_regs.sources[], an array of
>   struct vaplic_source holding the guest's view of the target register
>   together with a per-source lock which keeps the stored copy and the h/w
>   target register updated as a pair. Document how many elements it holds
>   and why index 0 is unused.
> - Drop the emulation of target register writes in direct delivery mode:
>   such writes are ignored with a one-time warning. Resolve the target
>   vCPU only in the MSI delivery mode path, where it is used.
> ---
> Changes in v2:
>  - Shadow the guest-written target registers in a per-domain array and
>    serve reads of APLIC_TARGET_* from it: the hardware register holds the
>    value produced by aplic_msi_target_gen() (physical hart field, VS-file
>    id), so reading it back would expose the host layout and return
>    something the guest never wrote. A non-zero guest index is dropped with
>    a one-time warning instead of being echoed back.
>  - aplic_hart_field(): take a CPU id instead of a hartid and derive both
>    the group and the hart index from msi->base_addr + msi->offset - the
>    hart index bits are a part of that offset, so the hartid can't be used
>    as the hart index. Add APLIC_xMSICFGADDR_PPN_LHX_{MASK,SHIFT} for that.
>  - Document the IMSIC MSI target address layout and the APLIC hart index
>    packing above aplic_hart_field(), and correct the corresponding diagram
>    in asm/imsic.h.
>  - Drop the mask parameter of aplic_hw_read_reg() and apply the
>    authorization mask in vaplic_emulate_load() instead.
>  - vaplic_emulate_{load,store}(): return bool instead of int, rename v/d to
>    curr/currd and add ASSERT(curr == current).
>  - Use domain_vcpu() instead of open-coded d->vcpu[] indexing when
>    resolving the target hart index.
>  - s/APLIC_REG_OFFSET_MASK/APLIC_CTRL_REGION_OFFSET_MASK/.
>  - Update store emulation handling for target register to be able to deal
>    with target format in both cases (MSI and Direct).
>  - Update store emulation handling of domaincfg. There is no need to force
>    DM/IE mode here (it will be forced/checked on Xen irq handler side).
>  - Rename APLIC_DEFAULT_PRIORITY and move to aplic.h as default prioity
>    value is used in vaplic code too now.
> ---
> ---
>  xen/arch/riscv/aplic-priv.h         |   2 +
>  xen/arch/riscv/aplic.c              | 138 +++++++++-
>  xen/arch/riscv/include/asm/aplic.h  |  37 +++
>  xen/arch/riscv/include/asm/vaplic.h |  19 ++
>  xen/arch/riscv/vaplic.c             | 384 +++++++++++++++++++++++++++-
>  5 files changed, 575 insertions(+), 5 deletions(-)
>
> diff --git a/xen/arch/riscv/aplic-priv.h b/xen/arch/riscv/aplic-priv.h
> index 35100d3a64..2245029b10 100644
> --- a/xen/arch/riscv/aplic-priv.h
> +++ b/xen/arch/riscv/aplic-priv.h
> @@ -47,4 +47,6 @@ struct aplic_priv {
>   */
>  extern unsigned int guest_aplic_num_sources;
>  
> +uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val);
> +
>  #endif /* ASM_RISCV_APLIC_PRIV_H */
> diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
> index 17c96177a3..febc451760 100644
> --- a/xen/arch/riscv/aplic.c
> +++ b/xen/arch/riscv/aplic.c
> @@ -16,6 +16,7 @@
>  #include <xen/irq.h>
>  #include <xen/mm.h>
>  #include <xen/sections.h>
> +#include <xen/sched.h>
Nit: <xen/sched.h> sorts before <xen/sections.h>.

>  #include <xen/spinlock.h>
>  #include <xen/types.h>
>  #include <xen/vmap.h>
> @@ -28,8 +29,6 @@
>  #include <asm/io.h>
>  #include <asm/riscv_encoding.h>
>  
> -#define APLIC_DEFAULT_PRIORITY  1
> -
>  static struct aplic_priv aplic = {
>      .lock = SPIN_LOCK_UNLOCKED,
>  };
> @@ -38,6 +37,137 @@ static struct intc_info __ro_after_init aplic_info = {
>      .hw_variant = INTC_APLIC,
>  };
>  
> +/*
> + * The arrangement of IMSIC interrupt files in MMIO space follows a topology
> + * defined by the RISC-V AIA specification. An IMSIC group is a set of
> + * interrupt files (e.g., in a cluster or socket) co-located in memory.
> + *
> + * The physical address of an outgoing MSI is calculated by bitwise ORing a
> + * Base Physical Page Number (Base PPN) with the Group Index (g), the Hart
> + * Index (h) and, for a supervisor-level interrupt domain, the Guest Index:
Sorry but as I already mentioned before, Hart Index is equal to (g << lhxw) | h
I think it is a miss as you explained this later correctly ....

I would propose this:
    ```
    * The physical address of an outgoing MSI is calculated by bitwise ORing a
    * Base Physical Page Number (Base PPN) with the group number (g), the
    * hart number relative to the group (h) and, for a supervisor-level
    * interrupt domain, the Guest Index:
    ```

> + *
> + *   ( Base PPN | (g << (HHXS + 12)) | (h << LHXS) | Guest Index ) << 12
> + *
> + * where Base PPN, HHXS, LHXS, HHXW and LHXW come from the {m,s}msiaddrcfg[h]
> + * registers of the interrupt domain that sends the MSI:
> + *
> + * XLEN-1       HHXS+24          LHXS+12          12          0
> + * |            |                |                |           |
> + * ------------------------------------------------------------
> + * |xxxx|   g   |xxxxxxxx|   h   |xxxx|Guest Index|     0     |
> + * ------------------------------------------------------------
> + *
> + * - g: group number.
> + * - h: hart number relative to the group.
... here
> + * - xxxx: remaining Base PPN bits; each gap may be zero-width.
> + * - Guest Index: selects one of the 4 KiB pages right above the hart's own
> + *   supervisor-level file, i.e. it starts at bit 12; LHXS must therefore be
> + *   at least as large as the number of guest index bits.
> + * - Bits 11:0: always zero because IMSIC files are 4 KiB page-aligned.
> + *
> + * For wired interrupts in MSI delivery mode (domaincfg.DM = 1), the APLIC
> + * computes the MSI target address itself from the "Hart Index" field
> + * (bits 31:18) of the corresponding target[i] register. This 14-bit field
> + * holds both g and h:
> + *
> + * 13          lhxw+hhxw   lhxw       0
> + * |           |           |          |
> + * ------------------------------------
> + * |     0     |     g     |    h     |
> + * ------------------------------------
> + *
> + * - lhxw (Low Hart Index Width): the number of bits used for the hart number
> + *   within a group.
> + * - hhxw (High Hart Index Width): the number of bits used for the group
> + *   number; the remaining bits of the field must be zero.
> + *
> + * The Guest Index isn't a part of it: for a supervisor-level interrupt domain
> + * it has its own field (bits 17:12) in target[i].
> + *
> + * Because there are "xxxx" gaps (Base PPN bits) between the indices in the
> + * physical address (depending on HHXS and LHXS), software must extract the
> + * group and hart components separately and pack them into the APLIC-defined
> + * Hart Index format to ensure correct MSI targeting.
> + */
> +static unsigned long aplic_hart_index(unsigned int cpu)
> +{
> +    const struct imsic_config *imsic = imsic_get_config();
> +    const struct imsic_msi *msi = &imsic->msi[cpu];
> +    /* Low Hart Index Shift */
> +    unsigned int lhxs = imsic->guest_index_bits;
> +    /* Low Hart Index Width */
> +    unsigned int lhxw = imsic->hart_index_bits;
> +    /* High Hart Index Width */
> +    unsigned int hhxw = imsic->group_index_bits;
> +    /* High Hart Index Shift */
> +    unsigned int hhxs =
> +        imsic->group_index_shift - APLIC_xMSICFGADDR_PPN_SHIFT * 2;
> +    /*
> +     * msi->base_addr is the base of the MMIO regset this CPU's interrupt
> +     * files live in, and one regset can cover several harts; msi->offset
> +     * selects this CPU's block inside it. The hart index bits are part of
> +     * that offset, so both indexes have to be derived from the full address.
> +     */
> +    paddr_t target_addr = msi->base_addr + msi->offset;
> +    unsigned long tppn = target_addr >> APLIC_xMSICFGADDR_PPN_SHIFT;
> +    unsigned long g =
> +        (tppn >> APLIC_xMSICFGADDR_PPN_HHX_SHIFT(hhxs)) &
> +        APLIC_xMSICFGADDR_PPN_HHX_MASK(hhxw);
> +    unsigned long h =
> +        (tppn >> APLIC_xMSICFGADDR_PPN_LHX_SHIFT(lhxs)) &
> +        APLIC_xMSICFGADDR_PPN_LHX_MASK(lhxw);
> +
> +    return (g << lhxw) | h;
> +}
> +
> +/*
> + * v->processor can't be used here: during a migration it names the new pCPU
> + * before the interrupt file is moved there. The caller has to hold
> + * vsfile_lock until the result is written to the h/w.
> + */
> +uint32_t aplic_msi_target_gen(const struct vcpu *v, uint32_t base_val)
> +{
> +    const struct vimsic_state *vimsic_state = v->arch.vimsic_state;
> +    unsigned int guest_id = vcpu_guest_file_id(v);
> +    unsigned long hart_index;

My v2 remark about making hart_index const can be dropped: as Jan said
(<02cb695f-c827-4623-8f65-530bd7b740d6@suse.com>), const is wanted on
pointer targets, not on plain variables.

> +
> +    ASSERT(rw_is_locked(&vimsic_state->vsfile_lock));
> +    ASSERT(vimsic_state->vsfile_cpu < NR_CPUS);
> +
> +    hart_index = aplic_hart_index(vimsic_state->vsfile_cpu);
> +
> +    base_val &= APLIC_TARGET_EIID;
> +    base_val |= MASK_INSR(guest_id, APLIC_TARGET_GUEST_IDX);
> +    base_val |= MASK_INSR(hart_index, APLIC_TARGET_HART_IDX);
> +
> +    return base_val;
> +}
> +
> +uint32_t aplic_hw_read_reg(unsigned int offset)
> +{
> +    unsigned long flags;
> +    uint32_t val;
> +
> +    ASSERT((offset < aplic.size) && IS_ALIGNED(offset, sizeof(uint32_t)));
Nit: the parentheses around offset < aplic.size aren't needed:
        ASSERT(offset < aplic.size && IS_ALIGNED(offset, sizeof(uint32_t)));

> +
> +    spin_lock_irqsave(&aplic.lock, flags);
> +    val = readl((const volatile void __iomem *)aplic.regs + offset);
> +    spin_unlock_irqrestore(&aplic.lock, flags);
> +
> +    return val;
> +}
> +
> +void aplic_hw_write_reg(unsigned int offset, uint32_t value)
> +{
> +    unsigned long flags;
> +
> +    ASSERT((offset < aplic.size) && IS_ALIGNED(offset, sizeof(uint32_t)));
Same here.

> +
> +    spin_lock_irqsave(&aplic.lock, flags);
> +    writel(value, (volatile void __iomem *)aplic.regs + offset);
> +    spin_unlock_irqrestore(&aplic.lock, flags);
> +}
> +
>  static void __init aplic_init_hw_interrupts(void)
>  {
>      unsigned int i;
> @@ -53,9 +183,9 @@ static void __init aplic_init_hw_interrupts(void)
>          /*
>           * Low bits of target register contains Interrupt Priority bits which
>           * can't be zero according to AIA spec.
> -         * Thereby they are initialized to APLIC_DEFAULT_PRIORITY.
> +         * Thereby they are initialized to APLIC_TARGET_IPRIO_DEFAULT.
>           */
> -        writel(APLIC_DEFAULT_PRIORITY, &aplic.regs->target[i]);
> +        writel(APLIC_TARGET_IPRIO_DEFAULT, &aplic.regs->target[i]);
>      }
>  
>      writel(APLIC_DOMAINCFG_IE | APLIC_DOMAINCFG_DM, &aplic.regs->domaincfg);
> diff --git a/xen/arch/riscv/include/asm/aplic.h b/xen/arch/riscv/include/asm/aplic.h
> index a2af55d54f..664bda2cb7 100644
> --- a/xen/arch/riscv/include/asm/aplic.h
> +++ b/xen/arch/riscv/include/asm/aplic.h
> @@ -39,6 +39,13 @@
>  #define  APLIC_DOMAINCFG_IE             BIT(8, U)
>  #define  APLIC_DOMAINCFG_DM             BIT(2, U)
>  #define  APLIC_DOMAINCFG_BE             BIT(0, U)
> +/*
> + * The bits a write may change. Everything else, including the read-only zero
> + * bit 7 and the reserved bits, has to read back as zero, and BE is WARL and
> + * hardwired to 0 as Xen is little-endian only.
> + */
> +#define  APLIC_DOMAINCFG_WMASK          (APLIC_DOMAINCFG_IE | \
> +                                         APLIC_DOMAINCFG_DM)
>  
>  #define APLIC_SOURCECFG_BASE            0x0004
>  #define APLIC_SOURCECFG_LAST            0x0ffc
> @@ -58,6 +65,12 @@
>  #define   APLIC_SOURCECFG_SM_EDGE_FALL  0x5
>  #define   APLIC_SOURCECFG_SM_LEVEL_HIGH 0x6
>  #define   APLIC_SOURCECFG_SM_LEVEL_LOW  0x7
> +/*
> + * All other bits of sourcecfg[] are reserved and read as zero, so drop them
> + * on a write.
> + */
> +#define  APLIC_SOURCECFG_WMASK          (APLIC_SOURCECFG_D | \
> +                                         APLIC_SOURCECFG_SM)
>  
>  #define APLIC_SMSICFGADDR               0x1bc8
>  #define APLIC_SMSICFGADDRH              0x1bcc
> @@ -89,6 +102,9 @@
>  #define  APLIC_TARGET_GUEST_IDX         GENMASK(17, 12)
>  /* Bit 11 is reserved and reads as zero */
>  #define  APLIC_TARGET_EIID              GENMASK(10, 0)
> +/* If target is in direct delivery mode (domaincfg.DM = 0) */
> +#define  APLIC_TARGET_IPRIO             GENMASK(7, 0)
> +#define   APLIC_TARGET_IPRIO_DEFAULT    1U
>  
>  #define APLIC_IDC_SIZE                  32
>  
> @@ -98,6 +114,24 @@
>  #define APLIC_SIZE(nr_cpus) \
>      (APLIC_MIN_SIZE + APLIC_SIZE_ALIGN(APLIC_IDC_SIZE * (nr_cpus)))
>  
> +/*
> + * Using setip is fine here, as all SET* and CLR* register groups consist of 32
> + * registers and therefore have identical sizes.
> + *
> + * Lowest 2 bits are always zero for SET* and CLR* registers.
> + */
> +#define APLIC_SETCLR_OFFSET_MASK \
> +    (sizeof_field(struct aplic_regs, setip) - sizeof(uint32_t))
> +
> +#define APLIC_xMSICFGADDR_PPN_SHIFT IMSIC_MMIO_PAGE_SHIFT
> +
> +#define APLIC_xMSICFGADDR_PPN_HHX_MASK(hhxw) (BIT(hhxw, UL) - 1)
> +#define APLIC_xMSICFGADDR_PPN_HHX_SHIFT(hhxs) \
> +    ((hhxs) + APLIC_xMSICFGADDR_PPN_SHIFT)
> +
> +#define APLIC_xMSICFGADDR_PPN_LHX_MASK(lhxw) (BIT(lhxw, UL) - 1)
> +#define APLIC_xMSICFGADDR_PPN_LHX_SHIFT(lhxs) (lhxs)
> +
>  struct aplic_regs {
>      uint32_t domaincfg;         /* 0x0000 */
>      uint32_t sourcecfg[1023];   /* 0x0004 */
> @@ -141,4 +175,7 @@ struct aplic_regs {
>      uint32_t target[1023];      /* 0x3004 */
>  };
>  
> +uint32_t aplic_hw_read_reg(unsigned int offset);
> +void aplic_hw_write_reg(unsigned int offset, uint32_t value);
> +
>  #endif /* ASM_RISCV_APLIC_H */
> diff --git a/xen/arch/riscv/include/asm/vaplic.h b/xen/arch/riscv/include/asm/vaplic.h
> index 96080bfbc2..3a555742bb 100644
> --- a/xen/arch/riscv/include/asm/vaplic.h
> +++ b/xen/arch/riscv/include/asm/vaplic.h
> @@ -11,6 +11,7 @@
>  #define ASM__RISCV__VAPLIC_H
>  
>  #include <xen/kernel.h>
> +#include <xen/spinlock.h>
>  #include <xen/types.h>
>  
>  #include <asm/intc.h>
> @@ -19,13 +20,31 @@ struct domain;
>  
>  #define to_vaplic(d) container_of((d)->arch.vintc, struct vaplic, vintc)
>  
> +/* State of one vAPLIC interrupt source. */
> +struct vaplic_source {
> +    /* Protects target together with the h/w target register of the source. */
> +    spinlock_t lock;
> +
> +    /* Guest's view of the APLIC target register. */
> +    uint32_t target;
> +};
> +
>  struct vaplic_regs {
>      uint32_t domaincfg;
> +
> +    /*
> +     * Indexed by source number. The array holds d->arch.vintc->nr_virqs
> +     * elements; sources[0] is unused as APLIC interrupt sources start from 1.
> +     */
> +    struct vaplic_source *sources;
>  };
>  
>  struct vaplic {
>      struct vintc vintc;
>      struct vaplic_regs regs;
> +
> +    paddr_t regs_start;
> +    unsigned int regs_size;
>  };
>  
>  int domain_vaplic_init(struct domain *d);
> diff --git a/xen/arch/riscv/vaplic.c b/xen/arch/riscv/vaplic.c
> index 6fbdc07805..ea4b0dc889 100644
> --- a/xen/arch/riscv/vaplic.c
> +++ b/xen/arch/riscv/vaplic.c
> @@ -17,6 +17,7 @@
>  #include <asm/aia.h>
>  #include <asm/imsic.h>
>  #include <asm/intc.h>
> +#include <asm/mmio.h>
>  #include <asm/vaplic.h>
>  
>  #include "aplic-priv.h"
> @@ -27,6 +28,313 @@ unsigned int __ro_after_init guest_aplic_num_sources;
>  
>  #define FDT_VAPLIC_INT_CELLS 2
>  
> +#define AUTH_IRQ_BIT(d, irqn) \
> +    (((irqn) < (d)->arch.vintc->nr_virqs) && \
> +     test_bit(irqn, (d)->arch.vintc->used_irqs))
> +
> +/*
> + * Convert a byte offset (within a SETIP/CLRIP/SETIE/CLRIE register group) to
> + * a 32-bit word index into the used_irqs bitmap. Each word covers 32
> + * interrupt sources. For SOURCECFG and TARGET groups the same division also
> + * yields the interrupt number directly, because those arrays store one 32-bit
> + * register per source.
> + */
> +#define regoffset_to_word_idx(reg_val) ((reg_val) / sizeof(uint32_t))
> +
> +static uint32_t vaplic_target_read(const struct domain *d, unsigned int irqn)
> +{
> +    const struct vaplic *vaplic = to_vaplic(d);
> +
> +    /* target[0] doesn't exist so irqn == 0 should be impossible */
> +    if ( !irqn || irqn >= vaplic->vintc.nr_virqs )
> +        return 0;
> +
> +    return read_atomic(&vaplic->regs.sources[irqn].target);
> +}
> +
> +static inline uint32_t generate_auth_mask(const struct domain *currd,
Nit: inline in a .c file isn't needed, the compiler decides anyway.
> +                                          unsigned int word_idx)
> +{
> +    unsigned int first_bit = word_idx * sizeof(uint32_t) * BITS_PER_BYTE;
> +
> +    if ( word_idx >= DIV_ROUND_UP(currd->arch.vintc->nr_virqs,
> +                                  sizeof(uint32_t) * BITS_PER_BYTE) )
> +    {
> +        gdprintk(XENLOG_DEBUG, "incorrect word_idx(%u)\n", word_idx);
> +
> +        return 0;
> +    }
> +
> +    return currd->arch.vintc->used_irqs[first_bit / BITS_PER_LONG] >>
> +           (first_bit % BITS_PER_LONG);
> +}
> +
> +static bool vaplic_emulate_load(const struct vcpu *curr, paddr_t addr,
> +                                uint32_t *out)
> +{
> +    const struct domain *currd = curr->domain;
> +    const struct vaplic *vaplic = to_vaplic(currd);
> +    const unsigned int offset = addr & APLIC_CTRL_REGION_OFFSET_MASK;

With that, every access with addr >= APLIC_CTRL_REGION_OFFSET_MASK
aliases onto the same control region: e.g. an access to
GUEST_APLIC_S_BASE + 0x4000 (idelivery of IDC 0) ends up in the
APLIC_DOMAINCFG case, and a store there rewrites domaincfg.

It also silently relies on GUEST_APLIC_S_BASE being 16 KiB aligned.

The AIA spec ("Memory-mapped control region for an interrupt domain")
says IDC structures start at 0x4000 and are only needed for direct
delivery mode, and that all other bytes of the control region are
reserved and read-only zero, writes to them being ignored. So I'd expect
something like:

    const paddr_t offset = addr - vaplic->regs_start;

Then read/write out of control region would go to "default:" case.
and APLIC_CTRL_REGION_OFFSET_MASK could go away.

> +    uint32_t auth_mask;
> +    unsigned int i;
> +
> +    ASSERT(curr == current);
> +
> +    switch ( offset )
> +    {
> +    case APLIC_DOMAINCFG:
> +        *out = vaplic->regs.domaincfg;
> +
> +        return true;
> +
> +    case APLIC_SETIP_BASE ... APLIC_SETIP_LAST:
> +    case APLIC_CLRIP_BASE ... APLIC_CLRIP_LAST:
> +    case APLIC_SETIE_BASE ... APLIC_SETIE_LAST:
> +        i = regoffset_to_word_idx(offset & APLIC_SETCLR_OFFSET_MASK);
> +        auth_mask = generate_auth_mask(currd, i);
> +
> +        break;
> +
> +    case APLIC_SETIPNUM:
> +    case APLIC_CLRIPNUM:
> +    case APLIC_SETIENUM:
> +    case APLIC_CLRIE_BASE ... APLIC_CLRIE_LAST:
> +    case APLIC_CLRIENUM:
> +    case APLIC_SETIPNUM_LE:
> +        /*
> +         * Based on the RISC-V AIA spec a read of these registers
> +         * always returns zero
> +         */
> +        *out = 0;
> +
> +        return true;
> +
> +    case APLIC_TARGET_BASE ... APLIC_TARGET_LAST:
> +        /*
> +         * As target registers start from 1:
> +         *  0x3000 genmsi
> +         *  0x3004 target[1]
> +         *  0x3008 target[2]
> +         *   ...
> +         *  0x3FFC target[1023]
> +         * It is necessary to calculate an interrupt number by subtracting
> +         * APLIC_GENMSI instead of APLIC_TARGET_BASE.
> +         */
> +        i = regoffset_to_word_idx(offset - APLIC_GENMSI);
> +
> +        *out = AUTH_IRQ_BIT(currd, i) ? vaplic_target_read(currd, i) : 0;
> +
> +        return true;
  genmsi (0x3000) isn't covered by any case, so an access to it ends up
  in default: and faults while in MSI delivery mode it isn't read-only zero:
  Hart Index and EIID behave as in a target register, and Busy is
  read-only. I would suggest to add a dedicated switch case:
  
    ```
    case APLIC_GENMSI:
        *out = vaplic_is_msi_mode(currd) ? vaplic->regs.genmsi : 0;

        return true;
    ```
With a new helper vaplic_is_msi_mode -> bool that could also be used in
write counterpart and a new genmsi field in vaplic->regs.

If we assume the guest will never try to read genmsi then it is okay but
it might worth to at least mention it somewhere.

> +    default:
> +        gdprintk(XENLOG_WARNING, "Unhandled APLIC read at offset %#x\n",
> +                 offset);
> +
> +        return false;
> +    }
> +
> +    *out = aplic_hw_read_reg(offset) & auth_mask;
> +
> +    return true;
> +}
> +
> +static bool vaplic_emulate_store(const struct vcpu *curr, paddr_t addr,
> +                                 uint32_t value)
> +{
> +    const struct domain *currd = curr->domain;
> +    unsigned int offset = addr & APLIC_CTRL_REGION_OFFSET_MASK;

Same aliasing issue as in vaplic_emulate_load(), here with the write
going to the aliased register.

> +
> +    ASSERT(curr == current);
> +
> +    switch ( offset )
> +    {
> +    case APLIC_DOMAINCFG:
> +    {
> +        struct vaplic *vaplic = to_vaplic(currd);
> +
> +        vaplic->regs.domaincfg = APLIC_DOMAINCFG_RO |
> +                                 (value & APLIC_DOMAINCFG_WMASK);
APLIC_DOMAINCFG_WMASK includes DM, so a guest writing DM=0 reads 0 back
and assumes direct delivery mode is in effect. However, only MSI
delivery is emulated: there are no IDC structures, and with DM=0 the
target handling below drops the write, so the guest just stops getting
interrupts.

DM is WARL, so an MSI-only APLIC is expected to keep it at 1 whatever
is written, which is how software figures out that direct mode isn't
available. Can't we do the same here, as is already done for BE?
I.e. drop DM from APLIC_DOMAINCFG_WMASK, and OR APLIC_DOMAINCFG_DM in
here and in domain_vaplic_init(). That would also justify sizing the
region at APLIC_MIN_SIZE (see my comment on regs_size at the end).
> +
> +        return true;
> +    }
> +
> +    case APLIC_SOURCECFG_BASE ... APLIC_SOURCECFG_LAST:
> +        /*
> +         * Only D (bit 10) and SM (bits 2:0) are implemented, the rest of the
> +         * bits are reserved and read as zero, so ignore what a guest writes
> +         * to them.
> +         */
> +        value &= APLIC_SOURCECFG_WMASK;
> +
> +        if ( value & APLIC_SOURCECFG_D )
> +        {
> +            gdprintk(XENLOG_ERR, "APLIC_SOURCECFG_D isn't supported\n");
> +
> +            goto fail;
> +        }
> +
> +        /*
> +         * SM is WARL, so the reserved values 0x2 and 0x3 need no handling
> +         * here: vAPLIC keeps no shadow copy of sourcecfg[], the value is
> +         * written straight to the h/w register and is read back from it, so
> +         * it is the h/w which substitutes a legal value for an illegal one.
> +         *
> +         * TODO: when vAPLIC starts to shadow sourcecfg[], the WARL behaviour
> +         * will have to be emulated here instead, e.g.:
> +         *   if ( value == 0x2 || value == 0x3 )
> +         *       value = APLIC_SOURCECFG_SM_INACTIVE;
> +         */
> +
> +        /*
> +         * As sourcecfg register starts from 1:
> +         *   0x0000 domaincfg
> +         *   0x0004 sourcecfg[1]
> +         *   0x0008 sourcecfg[2]
> +         *    ...
> +         *   0x0FFC sourcecfg[1023]
> +         * It is necessary to calculate an interrupt number by subtracting
> +         * APLIC_DOMAINCFG instead of APLIC_SOURCECFG_BASE.
> +         */
> +        if ( !AUTH_IRQ_BIT(currd,
> +                           regoffset_to_word_idx(offset - APLIC_DOMAINCFG)) )
> +            /* Interrupt not enabled, ignore it */
> +            return true;
> +
> +        break;
> +
> +    case APLIC_SETIP_BASE ... APLIC_SETIP_LAST:
> +    case APLIC_CLRIP_BASE ... APLIC_CLRIP_LAST:
> +    case APLIC_SETIE_BASE ... APLIC_SETIE_LAST:
> +    case APLIC_CLRIE_BASE ... APLIC_CLRIE_LAST:
> +    {
> +        unsigned int word_idx =
> +            regoffset_to_word_idx(offset & APLIC_SETCLR_OFFSET_MASK);
> +
> +        value &= generate_auth_mask(currd, word_idx);
> +
> +        break;
> +    }
> +
> +    case APLIC_SETIPNUM:
> +    case APLIC_CLRIPNUM:
> +    case APLIC_SETIENUM:
> +    case APLIC_CLRIENUM:
> +    case APLIC_SETIPNUM_LE:
> +        if ( !value || !AUTH_IRQ_BIT(currd, value) )
> +            return true;
> +
> +        break;
> +
> +    case APLIC_TARGET_BASE ... APLIC_TARGET_LAST:
> +    {
> +        struct vaplic *vaplic = to_vaplic(currd);
> +        /*
> +         * Look at vaplic_emulate_load() for explanation why APLIC_GENMSI is
> +         * subtracted.
> +         */
> +        unsigned int srcn = regoffset_to_word_idx(offset - APLIC_GENMSI);
> +
> +        if ( !AUTH_IRQ_BIT(currd, srcn) )
> +            /* Interrupt not enabled, ignore it */
> +            return true;
> +
> +        if ( vaplic->regs.domaincfg & APLIC_DOMAINCFG_DM )
> +        {
> +            struct vaplic_source *src = &vaplic->regs.sources[srcn];
> +            struct vcpu *target_vcpu;
target_vcpu is only read, so this can be const struct vcpu *.

> +            unsigned int guest_hart_idx =
> +                MASK_EXTR(value, APLIC_TARGET_HART_IDX);
> +            unsigned long flags;
> +            struct vimsic_state *vimsic_state;
> +
> +            target_vcpu = domain_vcpu(currd, guest_hart_idx);
> +
> +            if ( !target_vcpu )
> +            {
> +                gdprintk(XENLOG_ERR, "Invalid vCPU id in target register\n");
> +
> +                /* Ignore such writings */
> +                return true;
> +            }
> +
> +            vimsic_state = target_vcpu->arch.vimsic_state;
> +
> +            /*
> +             * A non-zero guest index asks for delivery to an interrupt file
> +             * of a nested guest. The vIMSIC node has no riscv,guest-index-bits
> +             * property, so a guest is told its harts have no guest interrupt
> +             * files and the field reads as zero for them. Such a write is
> +             * illegal and is therefore ignored as a whole: the stored copy
> +             * keeps the zero it was allocated with, so the field never needs
> +             * to be masked out here.
> +             * What ends up in the h/w register is Xen's own value anyway:
> +             * aplic_msi_target_gen() overwrites the field with
> +             * vcpu_guest_file_id() of the target vCPU.
> +             */
> +            if ( MASK_EXTR(value, APLIC_TARGET_GUEST_IDX) )
> +            {
> +                printk_once(XENLOG_WARNING
> +                            "%pd: vAPLIC target guest index != 0 is unsupported\n",
> +                            currd);
> +
> +                return true;
> +            }
> +
> +            /*
> +             * src->lock keeps the stored copy and the h/w register updated
> +             * as a pair.
> +             */
> +            spin_lock_irqsave(&src->lock, flags);
> +
> +            write_atomic(&src->target, value);
> +
> +            /* IRQs are already off under src->lock. */
> +            read_lock(&vimsic_state->vsfile_lock);
> +
> +            /*
> +             * The h/w register names the hart and the guest interrupt file
> +             * MSIs are sent to, and there is none of either until the target
> +             * vCPU is given a h/w guest interrupt file. The stored copy
> +             * reaches the h/w at that point.
> +             *
> +             * TODO: once s/w guest interrupt files are supported, a vCPU using
> +             * one still needs its MSIs, so the h/w register will have to be
> +             * written for it too, pointing at an interrupt file Xen receives
> +             * them in on the vCPU's behalf.
> +             */
> +            if ( vimsic_state->vsfile_cpu != CPU_NONE )
> +                aplic_hw_write_reg(offset,
> +                                   aplic_msi_target_gen(target_vcpu, value));

The comment above says the stored copy reaches the h/w once the vCPU gets
a h/w guest interrupt file, but nothing in this patch does that: it only
happens with vaplic_reconfigure_target(), added by "xen/riscv: introduce
vaplic_reconfigure_target()" and called from imsic_vsfile_move() by
"xen/riscv: remap interrupts to new IMSIC VS-file". At this commit a
target written before the first attach never reaches the h/w. 

> +
> +            read_unlock(&vimsic_state->vsfile_lock);
> +
> +            spin_unlock_irqrestore(&src->lock, flags);
> +
> +            return true;
> +        }
> +
> +        /* printk_once(), as a guest could otherwise flood the log. */
> +        printk_once(XENLOG_WARNING
> +                    "%pd: vAPLIC direct delivery mode isn't supported\n",
> +                    currd);
> +
> +        return true;
> +    }
> +
> +    default:
> +    fail:
> +        gdprintk(XENLOG_WARNING,
> +                 "Unhandled APLIC write at offset %#x (value %#x)\n", offset,
> +                 value);
Nit: now that the arguments wrap, put offset on its own line as well.

> +
> +        return false;
So in this function you are assuming the guest will never try to write
genmsi. Is it expected? so a guest accessing it faults. Linux doesn't
use it today, but the spec defines it in MSI delivery mode. Could it at
least get an explicit case (reads as zero, writes ignored, with a TODO
about extempore MSI injection) rather than relying on guests never
touching it?

> +    }
> +
> +    aplic_hw_write_reg(offset, value);
> +
> +    return true;
> +}
> +
>  static int __init cf_check vaplic_make_domu_dt_node(struct kernel_info *kinfo)
>  {
>      struct domain *d = kinfo->bd.d;
> @@ -95,6 +403,56 @@ static const struct vintc_init_ops __initconstrel init_ops = {
>      .make_domu_dt_node = vaplic_make_domu_dt_node,
>  };
>  
> +static enum io_state cf_check vaplic_mmio_read(struct vcpu *v,
> +                                               mmio_info_t *info)

Jan asked in v1 to rename the parameter to curr once ASSERT(v ==
current) is added. Same for vaplic_mmio_write().

> +{
> +    uint32_t val;
> +
> +    ASSERT(v == current);
> +
> +    if ( info->len != sizeof(uint32_t) ||
> +         !IS_ALIGNED(info->gpa, sizeof(uint32_t)) )
> +    {
> +        gdprintk(XENLOG_DEBUG,
> +                 "VAPLIC: unaligned/wrong-width read gpa=%"PRIpaddr" len=%u\n",
> +                 info->gpa, info->len);
> +        return IO_ABORT;
Please drop the "VAPLIC:" prefix, gdprintk() already says where the
message comes from. Also a blank line before the return, as you do
elsewhere in the file.

> +    }
> +
> +    if ( !vaplic_emulate_load(v, info->gpa, &val) )
> +        return IO_ABORT;
> +
> +    /* APLIC registers are 32-bit; zero-extend to the guest register width. */
> +    info->data = val;
> +
> +    return IO_HANDLED;
> +}
> +
> +static enum io_state cf_check vaplic_mmio_write(struct vcpu *v,
> +                                                const mmio_info_t *info)
> +{
> +    ASSERT(v == current);
> +
> +    if ( info->len != sizeof(uint32_t) ||
> +         !IS_ALIGNED(info->gpa, sizeof(uint32_t)) )
> +    {
> +        gdprintk(XENLOG_DEBUG,
> +                 "VAPLIC: unaligned/wrong-width write gpa=%"PRIpaddr" len=%u\n",
> +                 info->gpa, info->len);
> +        return IO_ABORT;
Same here.

> +    }
> +
> +    if ( !vaplic_emulate_store(v, info->gpa, info->data) )
> +        return IO_ABORT;
> +
> +    return IO_HANDLED;
> +}
> +
> +static const struct mmio_handler_ops vaplic_mmio_ops = {
> +    .read  = vaplic_mmio_read,
> +    .write = vaplic_mmio_write,
> +};
> +
>  static const struct vintc_ops vintc_ops = {
>      .vcpu_init = vcpu_imsic_init,
>      .vcpu_deinit = vcpu_imsic_deinit,
> @@ -103,6 +461,8 @@ static const struct vintc_ops vintc_ops = {
>  int domain_vaplic_init(struct domain *d)
>  {
>      struct vaplic *vaplic = xvzalloc(struct vaplic);
> +    unsigned int i;
> +    int rc;
>  
>      if ( !vaplic )
>          return -ENOMEM;
> @@ -123,7 +483,28 @@ int domain_vaplic_init(struct domain *d)
>       */
>      d->arch.vintc->nr_virqs = guest_aplic_num_sources + 1;
>  
> -    return 0;
> +    /* Slot 0 is unused: APLIC source numbering starts at 1 (see used_irqs). */
> +    vaplic->regs.sources = xvzalloc_array(struct vaplic_source,
> +                                          d->arch.vintc->nr_virqs);
> +    if ( !vaplic->regs.sources )
> +    {
> +        domain_vaplic_deinit(d);
> +
> +        return -ENOMEM;
> +    }
> +
> +    for ( i = 0; i < d->arch.vintc->nr_virqs; i++ )
> +        spin_lock_init(&vaplic->regs.sources[i].lock);
> +
> +    vaplic->regs_start = GUEST_APLIC_S_BASE;
> +    vaplic->regs_size = APLIC_SIZE(d->max_vcpus);

The vAPLIC only supports MSI delivery (msi-parent → vIMSIC, no direct
mode emulated), so no IDC structures are needed and APLIC_MIN_SIZE is
enough. To make that explicit I'll also hardwire domaincfg.DM to 1
(WARL), so the guest can't select a mode the region doesn't back.

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 09/39] xen/riscv: build the target hart index via aplic_hart_index()

[Baptiste Le Duc]

> aplic_set_irq_affinity() open-coded the packing of the group and hart
> indices into the target register, and got two things wrong along the
> way:
> 
>  - imsic_config.msi[] is indexed by logical CPU id, but the index was
>    run through cpuid_to_hartid() first. On any platform where the two
>    spaces differ this picks another CPU's interrupt file, or reads past
>    the array;
> 
>  - the same hart id was then used verbatim as the low hart index, and
>    the group index was derived from msi[].base_addr alone. The hart
>    index bits live in msi[].offset, the base address only covers the
>    MMIO regset, which may hold the files of several harts. Both indices
>    have to come out of base_addr + offset.
> 
> aplic_hart_index() already extracts them that way, and is what the vAPLIC
> target path uses, so call it here as well and insert the result with
> MASK_INSR(APLIC_TARGET_HART_IDX) instead of a bare shift, which keeps the
> value from spilling out of the 14-bit field. This drops the last in-tree
> duplicate of the AIA hart index formula, and with it the last user of
> APLIC_TARGET_HART_IDX_SHIFT, which is removed.
> 
> desc->irq is used as the EIID as is. Assert that it is a valid APLIC
> source, which also guarantees that it fits the EIID field.
> 
> No functional change on a single-group platform whose hart ids match
> their CPU ids and whose IMSIC regset holds one file per hart.
> 
Nit: The message is hard to follow because it describes the old code and
its bugs in the same sentences, so you never see a plain picture of what
the code did before. I would propose this:
    ```
    aplic_set_irq_affinity() computes the target register value by hand:

        cpu         = cpuid_to_hartid(<cpu picked from mask>);
        group_index = <bits taken from msi[cpu].base_addr>;
        hart_index  = cpu;

    This is wrong in three ways:

    - msi[] is indexed by logical CPU id, not by hart id. When the two
      differ, the wrong CPU's interrupt file is used, or msi[] is read
      out of bounds.

    - The hart id is used as the low hart index. The AIA spec defines
      the low hart index as bits of the IMSIC interrupt file address,
      and the two are not guaranteed to be equal.

    - The group index is taken from msi[].base_addr only. base_addr is
      the start of the IMSIC MMIO region, which may hold the interrupt
      files of several harts; the file of this CPU is at base_addr +
      msi[].offset, and both indexes must be extracted from that
      address.

    aplic_hart_index() already does this correctly and is used by the
    vAPLIC target path. Use it here as well, and insert the result with
    MASK_INSR(APLIC_TARGET_HART_IDX) rather than a plain shift, so it
    cannot overflow the 14-bit field. APLIC_TARGET_HART_IDX_SHIFT has no
    users left and is removed.

    desc->irq is written as the EIID unchanged. Assert that it is a
    valid APLIC source number, which also ensures it fits in the EIID
    field.

    No functional change on platforms with a single group, hart ids
    equal to CPU ids, and one interrupt file per IMSIC MMIO region.
    ```
> Fixes: d4676a1398bc ("xen/riscv: implementation of aplic and imsic operations")
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in v3:
>  - s/aplic_hart_field/aplic_hart_index/ (in the subject too) following the
>    rename of the helper in "xen/riscv: implement virtual APLIC MMIO
>    emulation".
>  - Don't mask desc->irq with APLIC_TARGET_EIID. Instead, bound it by
>    the APLIC source range rather than NR_IRQS, which will also have to
>    cover device MSIs: add a BUILD_BUG_ON() that ARRAY_SIZE(target) fits
>    the EIID field and an ASSERT() that desc->irq is a valid APLIC source
>    (1..num_irqs), which also covers the target[desc->irq - 1] index.
> ---
> Changes in v2:
>  - New patch.
> ---
> ---
>  xen/arch/riscv/aplic.c             | 33 ++++++++++--------------------
>  xen/arch/riscv/include/asm/aplic.h |  1 -
>  2 files changed, 11 insertions(+), 23 deletions(-)
>
> diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
> index febc451760..62bbcc9afa 100644
> --- a/xen/arch/riscv/aplic.c
> +++ b/xen/arch/riscv/aplic.c
> @@ -335,9 +335,7 @@ static unsigned int aplic_get_cpu_from_mask(const cpumask_t *cpumask)
>  static void cf_check aplic_set_irq_affinity(struct irq_desc *desc, const cpumask_t *mask)
>  {
>      unsigned int cpu;
> -    uint64_t group_index, base_ppn;
> -    uint32_t hhxw, lhxw, hhxs, value;
> -    const struct imsic_config *imsic = aplic.imsic_cfg;
> +    uint32_t value;
>  
>      /*
>       * TODO: Currently, APLIC is supported only with MSI interrupts.
> @@ -350,27 +348,18 @@ static void cf_check aplic_set_irq_affinity(struct irq_desc *desc, const cpumask
>  
>      ASSERT(spin_is_locked(&desc->lock));
>  
> -    cpu = cpuid_to_hartid(aplic_get_cpu_from_mask(mask));
> -    hhxw = imsic->group_index_bits;
> -    lhxw = imsic->hart_index_bits;
> +    cpu = aplic_get_cpu_from_mask(mask);
> +
>      /*
> -     * Although this variable is used only once in the calculation of
> -     * group_index, and it might seem that hhxs could be defined as:
> -     *   hhxs = imsic->group_index_shift - IMSIC_MMIO_PAGE_SHIFT;
> -     * and then the addition of IMSIC_MMIO_PAGE_SHIFT could be omitted
> -     * when calculating the group index.
> -     * It was done intentionally this way to follow the formula from
> -     * the AIA specification for calculating the MSI address.
> +     * desc->irq is used as the EIID as-is, which is only valid for APLIC
> +     * sources: they are numbered 1..num_irqs, with at most ARRAY_SIZE(target).
>       */
> -    hhxs = imsic->group_index_shift - IMSIC_MMIO_PAGE_SHIFT * 2;
> -    base_ppn = imsic->msi[cpu].base_addr >> IMSIC_MMIO_PAGE_SHIFT;
> -
> -    /* Update hart and EEID in the target register */
> -    group_index = (base_ppn >> (hhxs + IMSIC_MMIO_PAGE_SHIFT)) &
> -                  (BIT(hhxw, UL) - 1);
> -    value = desc->irq;
> -    value |= cpu << APLIC_TARGET_HART_IDX_SHIFT;
> -    value |= group_index << (lhxw + APLIC_TARGET_HART_IDX_SHIFT);
> +    BUILD_BUG_ON(ARRAY_SIZE(aplic.regs->target) >
> +                 MASK_EXTR(~0U, APLIC_TARGET_EIID));
Shouldn't this function belong to aplic_init() or at least in a __init
function? Moreover, this check is only relevant in case of MSI delivery
mode but I agree it worth checking as we couldn't know at build time in
which delivery we would be.

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 30/39] xen/riscv: implement APLIC-IMSIC-hart sync barrier for vCPU migration

[Oleksii Kurochko]

On 9/30/26 6:23 PM, Oleksii Kurochko wrote:
> When a vCPU is moved to a different guest interrupt file, MSIs that the
> APLIC has already sent towards the old file may still be in flight. They
> must land before the old file's state is saved and the switch is done,
> otherwise they would be lost.
> 
> To wait for them, use the APLIC's genmsi register. Writing it makes the
> APLIC itself send an MSI (an "extempore" MSI) with a given interrupt
> identity to a given hart. genmsi can only target the hart's hypervisor-
> level interrupt file, not a guest one, but the AIA spec guarantees that
> all MSIs previously sent by the same APLIC to the same hart become
> visible at the hart's IMSIC before the extempore MSI does. So once the
> extempore MSI has been delivered, no older MSI from this APLIC to the
> hart can still be in flight, whichever interrupt file it targets.
> 
> genmsi's Busy bit clearing only tells that the APLIC has sent the
> extempore MSI, not that it has arrived, so the sequence described in the
> AIA spec under "Synchronizing interactions between a hart and the APLIC"
> is split in two. aplic_genmsi_barrier() does the APLIC side, while
> imsic_aplic_sync() does the hart side: it clears the pending bit of the
> reserved identity before the genmsi write and spins on it afterwards,
> which is what tells that the extempore MSI, and hence every older MSI
> from this APLIC to this hart, has really arrived. Keeping the hart side
> in imsic.c leaves the APLIC code free of the IMSIC local CSR accessors.
> 
> The last interrupt identity (nr_ids) is reserved for this purpose.
> 
> Migration doesn't use the barrier at this stage, so imsic_migrate_vcpu()
> only references imsic_aplic_sync() to keep the compiler from complaining
> about it being unused.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in v3:
>   - Rename from "xen/riscv: implement APLIC-hart sync barrier for vCPU
>     migration", as the barrier waits for the MSIs to reach the hart's IMSIC.
>   - Re-word the commit message, describing in it how the sequence is split
>     between aplic_genmsi_barrier() and imsic_aplic_sync().
>   - Move imsic_aplic_sync(), imsic_local_is_pending() and imsic_csr_read()
>     here from "xen/riscv: remap interrupts to new IMSIC VS-file", so that
>     both halves of the synchronization sequence are introduced together.
>   - s/aplic_hart_field/aplic_hart_index/ following the rename of the helper
>     in "xen/riscv: implement virtual APLIC MMIO emulation".
>   - Drop masking of sync_id with APLIC_TARGET_EIID, which could silently
>     chop off bits; instead add BUILD_BUG_ON() and ASSERT() to make it
>     explicit that sync_id always fits into the EIID field.
>   - Drop the redundant parentheses around the macro argument in
>     imsic_csr_read().
>   - Use unsigned int for isel in imsic_local_is_pending() and drop the
>     redundant !! as the function returns bool.
>   - Rename the unused argument of imsic_aplic_sync() to unused.
> ---
> Changes in v2:
>   - New patch.
> ---
> ---
>   xen/arch/riscv/aplic.c             | 33 +++++++++++++++++++
>   xen/arch/riscv/imsic.c             | 52 ++++++++++++++++++++++++++++++
>   xen/arch/riscv/include/asm/aplic.h |  3 ++
>   xen/arch/riscv/include/asm/imsic.h |  8 +++++
>   4 files changed, 96 insertions(+)
> 
> diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
> index 283e5d6046a6..5f9a894b5a76 100644
> --- a/xen/arch/riscv/aplic.c
> +++ b/xen/arch/riscv/aplic.c
> @@ -27,7 +27,9 @@
>   #include <asm/imsic.h>
>   #include <asm/intc.h>
>   #include <asm/io.h>
> +#include <asm/processor.h>
>   #include <asm/riscv_encoding.h>
> +#include <asm/smp.h>
>   
>   static struct aplic_priv aplic = {
>       .lock = SPIN_LOCK_UNLOCKED,
> @@ -183,6 +185,37 @@ void aplic_hw_write_reg(unsigned int offset, uint32_t value)
>       spin_unlock_irqrestore(&aplic.lock, flags);
>   }
>   
> +/*
> + * As needed, synchronize with all IOMMUs and APLICs to ensure that no
> + * straggler MSIs will arrive at the old interrupt file after this step.
> + */
> +void aplic_genmsi_barrier(void)
> +{
> +    const struct imsic_config *imsic = imsic_get_config();
> +    unsigned int cpu = smp_processor_id();
> +    unsigned long flags;
> +    uint32_t val;
> +
> +    /*
> +     * sync_id is nr_ids, which imsic_parse_node() limits to IMSIC_MAX_ID,
> +     * so it always fits into the EIID field.
> +     */
> +    BUILD_BUG_ON(IMSIC_MAX_ID > MASK_EXTR(~0U, APLIC_TARGET_EIID));
> +    ASSERT(imsic->sync_id <= IMSIC_MAX_ID);
> +
> +    val = MASK_INSR(aplic_hart_index(cpu), APLIC_TARGET_HART_IDX) |
> +          imsic->sync_id;
> +
> +    spin_lock_irqsave(&aplic.lock, flags);
> +
> +    writel(val, &aplic.regs->genmsi);
> +
> +    while ( readl(&aplic.regs->genmsi) & APLIC_GENMSI_BUSY )
> +        cpu_relax();
> +
> +    spin_unlock_irqrestore(&aplic.lock, flags);
> +}
> +
>   static void __init aplic_init_hw_interrupts(void)
>   {
>       unsigned int i;
> diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
> index f1e44dc56bb8..8e0273eab902 100644
> --- a/xen/arch/riscv/imsic.c
> +++ b/xen/arch/riscv/imsic.c
> @@ -27,6 +27,7 @@
>   #include <xen/xvmalloc.h>
>   
>   #include <asm/aia.h>
> +#include <asm/aplic.h>
>   #include <asm/imsic.h>
>   
>   #define IMSIC_HART_SIZE(guest_bits) (BIT(guest_bits, U) * IMSIC_MMIO_PAGE_SZ)
> @@ -60,6 +61,12 @@ static unsigned int __ro_after_init guest_num_msis;
>   #define IMSIC_DISABLE_EITHRESHOLD   1
>   #define IMSIC_ENABLE_EITHRESHOLD    0
>   
> +#define imsic_csr_read(c)       \
> +({                              \
> +    csr_write(CSR_SISELECT, c); \
> +    csr_read(CSR_SIREG);        \
> +})
> +
>   #define imsic_csr_write(c, v)   \
>   do {                            \
>       csr_write(CSR_SISELECT, c); \
> @@ -203,6 +210,15 @@ void imsic_irq_enable(unsigned int irq)
>        */
>       ASSERT(!local_irq_is_enabled());
>   
> +    if ( irq == imsic_cfg.sync_id )
> +    {
> +        printk(XENLOG_WARNING
> +               "irq%u is reserved for APLIC sync so shouldn't be set by %s\n",
> +               irq, __func__);
> +
> +        return;
> +    }
> +
>       spin_lock(&imsic_cfg.lock);
>       /*
>        * There is no irq - 1 here (look at aplic_set_irq_type()) because:
> @@ -241,6 +257,15 @@ void imsic_irq_disable(unsigned int irq)
>       spin_unlock(&imsic_cfg.lock);
>   }
>   
> +static bool imsic_local_is_pending(unsigned int id)
> +{
> +    unsigned int isel =
> +        (id / BITS_PER_LONG) * (BITS_PER_LONG / IMSIC_EIPx_BITS) + IMSIC_EIP0;
> +    unsigned long bit = BIT(id % BITS_PER_LONG, UL);
> +
> +    return imsic_csr_read(isel) & bit;
> +}
> +
>   /* Callers aren't intended to changed imsic_cfg so return const. */
>   const struct imsic_config *imsic_get_config(void)
>   {
> @@ -387,6 +412,9 @@ static int __init imsic_parse_node(const struct dt_device_node *node,
>   
>       imsic_cfg.nr_eix = DIV_ROUND_UP(imsic_cfg.nr_ids + 1, 64);
>   
> +    /* Reserve last identity for APLIC-to-hart synchronization */
> +    imsic_cfg.sync_id = imsic_cfg.nr_ids;
> +
>       /* Compute base address */
>       *nr_mmios = 0;
>       rc = dt_device_get_address(node, *nr_mmios, &base_addr, NULL);
> @@ -497,6 +525,27 @@ static void imsic_call_on_cpu(unsigned int cpu, void (*func)(void *),
>           on_selected_cpus(cpumask_of(cpu), func, data, 1);
>   }
>   
> +/*
> + * Ensure that all the MSIs the APLIC has already generated for the hart this
> + * runs on have really reached the hart's IMSIC.
> + *
> + * The barrier is the one described by the AIA specification in
> + * "Synchronizing interactions between a hart and the APLIC": ask the APLIC to
> + * send an MSI to the hart itself and wait until it shows up as pending in the
> + * hart's own interrupt file. As it says nothing about MSIs on their way to
> + * any other hart, it has to be executed by the pCPU owning the interrupt file
> + * the MSIs were being sent to.
> + */
> +static void cf_check imsic_aplic_sync(void *unused)
> +{
> +    imsic_local_eix_update(imsic_cfg.sync_id, 1, true, false);
> +
> +    aplic_genmsi_barrier();
> +
> +    while ( !imsic_local_is_pending(imsic_cfg.sync_id) )
> +        cpu_relax();
> +}
> +
>   static void cf_check imsic_vsfile_local_clear(void *data)
>   {
>       unsigned int i;
> @@ -862,5 +911,8 @@ void imsic_migrate_vcpu(struct vcpu *v)
>       /* Zero-out new IMSIC VS-file */
>       imsic_call_on_cpu(v->processor, imsic_vsfile_local_clear, &vsfile_data);
>   
> +    /* Avoid an unused-function error until the barrier has a user. */
> +    (void)imsic_aplic_sync;
> +
>       BUG_ON("unimplemented");
>   }
> diff --git a/xen/arch/riscv/include/asm/aplic.h b/xen/arch/riscv/include/asm/aplic.h
> index 9018936ca95a..bfb510b2ae5e 100644
> --- a/xen/arch/riscv/include/asm/aplic.h
> +++ b/xen/arch/riscv/include/asm/aplic.h
> @@ -94,6 +94,7 @@
>   #define APLIC_SETIPNUM_LE               0x2000
>   
>   #define APLIC_GENMSI                    0x3000
> +#define APLIC_GENMSI_BUSY               BIT(12, U)
>   
>   #define APLIC_TARGET_BASE               0x3004
>   #define APLIC_TARGET_LAST               0x3ffc
> @@ -177,4 +178,6 @@ struct aplic_regs {
>   uint32_t aplic_hw_read_reg(unsigned int offset);
>   void aplic_hw_write_reg(unsigned int offset, uint32_t value);
>   
> +void aplic_genmsi_barrier(void);
> +
>   #endif /* ASM_RISCV_APLIC_H */
> diff --git a/xen/arch/riscv/include/asm/imsic.h b/xen/arch/riscv/include/asm/imsic.h
> index 8130d125f5c3..f86407e21765 100644
> --- a/xen/arch/riscv/include/asm/imsic.h
> +++ b/xen/arch/riscv/include/asm/imsic.h
> @@ -62,6 +62,14 @@ struct imsic_config {
>        */
>       unsigned int nr_eix;
>   
> +    /*
> +     * Interrupt identity reserved exclusively for APLIC-to-hart
> +     * synchronization.
> +     *
> +     * Must not be allocated to any interrupt source.
> +     */
> +    unsigned int sync_id;
> +
>       /* MSI */
>       const struct imsic_msi *msi;
>   


I think it also makes sense here to apply the following fixup patch:

commit db57e94ed8405c98d7a21f4f88764fa5e0ea2d04 
(riscv-next-upstreaming-fixups)
Author: Oleksii Kurochko <oleksii.kurochko@gmail.com>
Date:   Tue Oct 6 15:59:52 2026 +0200

     fixup! xen/riscv: implement APLIC-IMSIC-hart sync barrier for vCPU 
migration

     In MSI delivery mode the MSI of APLIC source N carries identity N, by
     which Xen recognises it in its own IMSIC interrupt file. A source
     numbered sync_id or higher can't be told apart: its identity either
     isn't implemented, so its MSIs are dropped, or is the one reserved for
     the sync barrier, which it would satisfy early and whose pending bit
     would swallow its interrupt. Limit the usable sources to those below
     sync_id, and document how the identities of Xen's interrupt file are
     allocated.

diff --git a/xen/arch/riscv/aplic.c b/xen/arch/riscv/aplic.c
index 96aa4edb3958..86ff3cae984d 100644
--- a/xen/arch/riscv/aplic.c
+++ b/xen/arch/riscv/aplic.c
@@ -246,6 +246,7 @@ static int __init cf_check aplic_init(void)
      uint64_t size, paddr;
      const struct dt_device_node *imsic_node;
      const struct dt_device_node *node = aplic_info.node;
+    const struct imsic_config *imsic;
      int rc;

      /* Check for associated imsic node */
@@ -270,6 +271,22 @@ static int __init cf_check aplic_init(void)
          panic("%s: failed to get number of interrupt sources\n",
                node->full_name);

+    /*
+     * In MSI delivery mode the MSI of source N carries identity N, by 
which
+     * Xen recognises it in its own IMSIC interrupt file. Hence the sources
+     * numbered from sync_id on can't be used: their identities either 
aren't
+     * implemented or are the one reserved for the APLIC-IMSIC sync 
barrier.
+     */
+    imsic = imsic_get_config();
+    if ( aplic_info.num_irqs >= imsic->sync_id )
+    {
+        printk(XENLOG_WARNING
+               "%s: only %u of %u interrupt sources usable with %u 
IMSIC identities\n",
+               node->full_name, imsic->sync_id - 1, aplic_info.num_irqs,
+               imsic->nr_ids);
+        aplic_info.num_irqs = imsic->sync_id - 1;
+    }
+
      guest_aplic_num_sources = min(GUEST_APLIC_MAX_SOURCES, 
aplic_info.num_irqs);

      if ( aplic_info.num_irqs > ARRAY_SIZE(aplic.regs->sourcecfg) )
diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index ca5243826726..408b94b22ee9 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -533,7 +533,15 @@ static int __init imsic_parse_node(const struct 
dt_device_node *node,

      imsic_cfg.nr_eix = DIV_ROUND_UP(imsic_cfg.nr_ids + 1, 64);

-    /* Reserve last identity for APLIC-to-hart synchronization */
+    /*
+     * The identities of Xen's own interrupt file are allocated statically:
+     *  - the APLIC sources take 1 .. riscv,num-sources, as the MSI of 
source
+     *    N carries identity N (aplic_init() drops the sources which 
don't fit
+     *    below sync_id);
+     *  - the last one, nr_ids, is reserved for APLIC-to-hart 
synchronization.
+     * Anything else taking identities from this file has to stay clear of
+     * both.
+     */
      imsic_cfg.sync_id = imsic_cfg.nr_ids;

      /* Compute base address */

Any thoughts about it?

Thanks in advance.

~ Oleksii

Re: [PATCH v3 11/39] xen/riscv: implement vCPU context switching

[Baptiste Le Duc]

> Implement context_switch() and the helpers it needs: save/restore of
> H/VS CSRs, virtual timer and P2M context, and __context_switch() in
> assembly, which switches Xen's own callee-saved state (and thereby the
> stack) from prev to next. The virtual interrupt controller state isn't
> switched here.
> 
> Add offsets of struct arch_vcpu's xen_saved_context to asm-offsets.c for
> use by __context_switch().
> 
> henvcfg, htimedelta and vsie are 64-bit on both RV32 and RV64, so store
> them as uint64_t and access them with csr_{read,write}64(). For that
> purpose introduce csr_read64().
> 
> The VMID has to be claimed on the context switch rather than on guest
> entry: once p2m_ctxt_switch_to() writes HGATP, speculation can populate
> G-stage entries under whatever VMID is written there. This is done by
> p2m_vmid_switch_to(), called right before the HGATP write:
> 
>  - VMIDs are a per-hart resource, so the (generation, vmid) pair of a
>    vCPU which last ran on another hart is invalidated before a VMID is
>    claimed for it. The local flush for a wrapped generation moves along
>    with the claim.
>  - Nothing is switched in for the idle vCPU, so HGATP keeps pointing at
>    the p2m of the domain which ran on the hart last. Track that domain
>    per hart (hgatp_owner) and keep the hart in its dirty_cpumask until a
>    vCPU of another domain is switched in: p2m_tlb_flush() then goes on
>    reaching the hart, and a domain which idles between two runs on the
>    same hart keeps its VMIDs.
>  - When the owner changes, the hart leaves the old owner's dirty_cpumask
>    while its TLB may still hold that domain's G-stage translations, so it
>    is moved to a new VMID generation, which makes none of them reachable
>    again. It joins the new owner's dirty_cpumask before HGATP is written;
>    a full barrier there, paired with one in p2m_tlb_flush(), guarantees
>    that a concurrent P2M change either reaches the hart with an
>    HFENCE.GVMA or is visible to it before it walks the p2m.
> 
> That leaves p2m_handle_vmenter() with nothing to do, so drop it together
> with its call from check_for_pcpu_work(): a VMID is only ever invalidated
> while its vCPU isn't running, and p2m_tlb_flush() drops stale entries
> with a remote HFENCE.GVMA rather than by retiring VMIDs. Unlike
> p2m_handle_vmenter(), HGATP is written unconditionally, as it holds the
> G-stage root too, which on a context switch belongs to another domain.
> 
> While at it, fix the inclusion order of headers in asm-offsets.c: Xen's
> headers go first, then arch specific ones.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in v3:
>  - Update the commit message.
>  - Introduce csr_read64(), as vsie is 64-bit on RV32 too, i.e. has to be
>    accessed as the vsie/vsieh pair there.
>  - Make vsie declared as uint64_t instead of register_t.
>  - Rename the parameter of save_csr_regs() to p and of restore_csr_regs()
>    to n, to match their callers ctxt_switch_from()/ctxt_switch_to() and the
>    rest of the context switch helpers (p2m_ctxt_switch_{from,to}(),
>    vtimer_ctxt_switch_{from,to}()).
>  - s/save_csr_regs/csr_regs_ctxt_switch_from/
>  - s/restore_csr_regs/csr_regs_ctxt_switch_to/
>  - Drop the forward declaration of struct vcpu in <asm/system.h>: the
>    return type of __context_switch() already declares the tag at file
>    scope.
>  - Make the next parameter of __context_switch() pointer-to-const, as
>    next's saved context is only read.
>  - Extend the comment above __context_switch(): as ra is switched too, it
>    doesn't return to its caller, but to where next last called it from
>    (next's own context_switch(), or continue_new_vcpu() for a vCPU which
>    has never run).
>  - Call vtimer_ctxt_switch_to() after csr_regs_ctxt_switch_to(), so that
>    ctxt_switch_to() restores state in the reverse order of
>    ctxt_switch_from() saving it.
>  - Rework p2m's VMID handling in the context switch: track the domain
>    whose p2m HGATP points at in a per-CPU hgatp_owner, so a pass through
>    idle no longer drops the hart from that domain's dirty_cpumask nor
>    burns a VMID generation; move the dirty_cpumask update and the
>    invalidation of a VMID brought from another hart from schedule_tail()
>    and ctxt_switch_to() to p2m_ctxt_switch_to(), next to the VMID claim;
>    add the barriers ordering the dirty_cpumask update against
>    p2m_tlb_flush().
>  - Initialise arch_vcpu.last_cpu to CPU_NONE instead of NR_CPUS.
>  - Mark prev's dirty_cpu clean before ctxt_switch_to() and set current's
>    after it.
> ---
> Changes in v2:
>  - New patch.
> ---
> ---
>  xen/arch/riscv/domain.c              | 127 ++++++++++++++++++++++
>  xen/arch/riscv/entry.S               |  48 ++++++++
>  xen/arch/riscv/include/asm/csr.h     |  13 +++
>  xen/arch/riscv/include/asm/domain.h  |  15 ++-
>  xen/arch/riscv/include/asm/p2m.h     |   1 -
>  xen/arch/riscv/include/asm/system.h  |   2 +
>  xen/arch/riscv/p2m.c                 | 157 ++++++++++++++++++---------
>  xen/arch/riscv/riscv64/asm-offsets.c |  19 +++-
>  xen/arch/riscv/stubs.c               |   5 -
>  xen/arch/riscv/traps.c               |   2 -
>  10 files changed, 330 insertions(+), 59 deletions(-)
>
> diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
> index fd62296356..f80643e2d8 100644
> --- a/xen/arch/riscv/domain.c
> +++ b/xen/arch/riscv/domain.c
> @@ -11,6 +11,7 @@
>  #include <asm/bitops.h>
>  #include <asm/cpufeature.h>
>  #include <asm/csr.h>
> +#include <asm/current.h>
>  #include <asm/intc.h>
>  #include <asm/mmio.h>
>  #include <asm/riscv_encoding.h>
> @@ -152,6 +153,8 @@ int arch_vcpu_create(struct vcpu *v)
>      if ( is_idle_vcpu(v) )
>          return 0;
>  
> +    v->arch.last_cpu = CPU_NONE;
> +
>      vcpu_csr_init(v);
>  
>      if ( (rc = vcpu_vtimer_init(v)) )
> @@ -323,6 +326,130 @@ int arch_domain_create(struct domain *d,
>      return rc;
>  }
>  
> +static void csr_regs_ctxt_switch_from(struct vcpu *p)
> +{
> +    /*
> +     * There is no need to save these CSRs as only hypervisor writes them in
> +     * csr_regs_ctxt_switch_to() and guest can't access them so they shouldn't
> +     * be stored here. Keep them commented here just for symmetry with the
> +     * restore CSRs register part.
> +     *
> +     * p->arch.hedeleg = csr_read(CSR_HEDELEG);
> +     * p->arch.hideleg = csr_read(CSR_HIDELEG);
> +     * p->arch.henvcfg = csr_read64(CSR_HENVCFG);
> +     * p->arch.hcounteren = csr_read(CSR_HCOUNTEREN);
> +     * p->arch.htimedelta = csr_read64(CSR_HTIMEDELTA);
> +     *
> +     * if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
> +     *     p->arch.hstateen0 = csr_read(CSR_HSTATEEN0);
> +     */
> +
> +    p->arch.hvip = csr_read(CSR_HVIP);
> +
> +    p->arch.vsstatus = csr_read(CSR_VSSTATUS);
> +    p->arch.vsie = csr_read64(CSR_VSIE);
> +    p->arch.vstvec = csr_read(CSR_VSTVEC);
> +    p->arch.vsscratch = csr_read(CSR_VSSCRATCH);
> +    p->arch.vscause = csr_read(CSR_VSCAUSE);
> +    p->arch.vstval = csr_read(CSR_VSTVAL);
> +    p->arch.vsepc = csr_read(CSR_VSEPC);
> +}
> +
> +static void csr_regs_ctxt_switch_to(struct vcpu *n)
> +{
> +    csr_write(CSR_HEDELEG, n->arch.hedeleg);
> +    csr_write(CSR_HIDELEG, n->arch.hideleg);
> +    csr_write(CSR_HVIP, n->arch.hvip);
> +    csr_write64(CSR_HENVCFG, n->arch.henvcfg);
> +    csr_write(CSR_HCOUNTEREN, n->arch.hcounteren);
> +    csr_write64(CSR_HTIMEDELTA, n->arch.htimedelta);
> +
> +    if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
> +        csr_write(CSR_HSTATEEN0, n->arch.hstateen0);
> +
> +    csr_write(CSR_VSSTATUS, n->arch.vsstatus);
> +    csr_write64(CSR_VSIE, n->arch.vsie);
> +    csr_write(CSR_VSTVEC, n->arch.vstvec);
> +    csr_write(CSR_VSSCRATCH, n->arch.vsscratch);
> +    csr_write(CSR_VSCAUSE, n->arch.vscause);
> +    csr_write(CSR_VSTVAL, n->arch.vstval);
> +    csr_write(CSR_VSEPC, n->arch.vsepc);
> +}
> +
> +static void ctxt_switch_from(struct vcpu *p)
> +{
> +    /*
> +     * When the idle VCPU is running, Xen will always stay in hypervisor
> +     * mode.
> +     * Therefore we don't need to save the context of an idle VCPU.
> +     */
> +    if ( is_idle_vcpu(p) )
> +        return;
> +
> +    p2m_ctxt_switch_from(p);
> +
> +    vtimer_ctxt_switch_from(p);
> +
> +    csr_regs_ctxt_switch_from(p);
> +}
> +
> +static void ctxt_switch_to(struct vcpu *n)
> +{
> +    /*
> +     * When the idle VCPU is running, Xen will always stay in hypervisor
> +     * mode.
> +     * Therefore we don't need to restore the context of an idle VCPU.
> +     */
> +    if ( is_idle_vcpu(n) )
> +        return;
> +
> +    csr_regs_ctxt_switch_to(n);
> +
> +    vtimer_ctxt_switch_to(n);
> +
> +    p2m_ctxt_switch_to(n);
> +}
> +
> +static void schedule_tail(struct vcpu *prev)
> +{
> +    unsigned int cpu = smp_processor_id();
> +
> +    ASSERT(prev != current);
> +
> +    ctxt_switch_from(prev);
> +
> +    write_atomic(&prev->dirty_cpu, VCPU_CPU_CLEAN);
> +
> +    ctxt_switch_to(current);
> +
> +    write_atomic(&current->dirty_cpu, cpu);
> +
> +    current->arch.last_cpu = cpu;
> +
> +    /*
> +     * sched_context_switched() internally uses a spinlock,
> +     * which requires interrupts to be enabled.
> +     */
> +    local_irq_enable();
> +
> +    sched_context_switched(prev, current);
> +}
> +
> +void context_switch(struct vcpu *prev, struct vcpu *next)
> +{
> +    ASSERT(local_irq_is_enabled());
> +    ASSERT(prev != next);
> +    ASSERT(!vcpu_cpu_dirty(next));
> +
> +    local_irq_disable();
> +
> +    set_current(next);
> +
> +    prev = __context_switch(prev, next);
> +
> +    schedule_tail(prev);
> +}
> +
>  static void __init __maybe_unused build_assertions(void)
>  {
>      /*
> diff --git a/xen/arch/riscv/entry.S b/xen/arch/riscv/entry.S
> index 202a35fb03..017fbb0626 100644
> --- a/xen/arch/riscv/entry.S
> +++ b/xen/arch/riscv/entry.S
> @@ -99,3 +99,51 @@ restore_registers:
>  
>          sret
>  END(handle_trap)
> +
> +/*
> + * struct vcpu *__context_switch(struct vcpu *prev, const struct vcpu *next)
> + *
> + * This is called on prev's stack, and returns on next's. As ra is
> + * switched too, it doesn't return to its caller: it returns to where
> + * next last called it from, i.e. into next's own context_switch(), or,
> + * for a vCPU which has never run, to continue_new_vcpu() with an empty
> + * stack.
> + *
> + * a0 - prev
> + * a1 - next
> + *
> + * Returns prev in a0
> + */
> +FUNC(__context_switch)
> +        REG_S   s0, VCPU_XEN_SAVED_CONTEXT_S0(a0)
> +        REG_S   s1, VCPU_XEN_SAVED_CONTEXT_S1(a0)
> +        REG_S   s2, VCPU_XEN_SAVED_CONTEXT_S2(a0)
> +        REG_S   s3, VCPU_XEN_SAVED_CONTEXT_S3(a0)
> +        REG_S   s4, VCPU_XEN_SAVED_CONTEXT_S4(a0)
> +        REG_S   s5, VCPU_XEN_SAVED_CONTEXT_S5(a0)
> +        REG_S   s6, VCPU_XEN_SAVED_CONTEXT_S6(a0)
> +        REG_S   s7, VCPU_XEN_SAVED_CONTEXT_S7(a0)
> +        REG_S   s8, VCPU_XEN_SAVED_CONTEXT_S8(a0)
> +        REG_S   s9, VCPU_XEN_SAVED_CONTEXT_S9(a0)
> +        REG_S   s10, VCPU_XEN_SAVED_CONTEXT_S10(a0)
> +        REG_S   s11, VCPU_XEN_SAVED_CONTEXT_S11(a0)
> +        REG_S   sp, VCPU_XEN_SAVED_CONTEXT_SP(a0)
> +        REG_S   ra, VCPU_XEN_SAVED_CONTEXT_RA(a0)
> +
> +        REG_L   s0, VCPU_XEN_SAVED_CONTEXT_S0(a1)
> +        REG_L   s1, VCPU_XEN_SAVED_CONTEXT_S1(a1)
> +        REG_L   s2, VCPU_XEN_SAVED_CONTEXT_S2(a1)
> +        REG_L   s3, VCPU_XEN_SAVED_CONTEXT_S3(a1)
> +        REG_L   s4, VCPU_XEN_SAVED_CONTEXT_S4(a1)
> +        REG_L   s5, VCPU_XEN_SAVED_CONTEXT_S5(a1)
> +        REG_L   s6, VCPU_XEN_SAVED_CONTEXT_S6(a1)
> +        REG_L   s7, VCPU_XEN_SAVED_CONTEXT_S7(a1)
> +        REG_L   s8, VCPU_XEN_SAVED_CONTEXT_S8(a1)
> +        REG_L   s9, VCPU_XEN_SAVED_CONTEXT_S9(a1)
> +        REG_L   s10, VCPU_XEN_SAVED_CONTEXT_S10(a1)
> +        REG_L   s11, VCPU_XEN_SAVED_CONTEXT_S11(a1)
> +        REG_L   sp, VCPU_XEN_SAVED_CONTEXT_SP(a1)
> +        REG_L   ra, VCPU_XEN_SAVED_CONTEXT_RA(a1)
> +
> +        ret
> +END(__context_switch)
> diff --git a/xen/arch/riscv/include/asm/csr.h b/xen/arch/riscv/include/asm/csr.h
> index a5cb24c863..aad82c6a6d 100644
> --- a/xen/arch/riscv/include/asm/csr.h
> +++ b/xen/arch/riscv/include/asm/csr.h
> @@ -40,6 +40,13 @@
>      csr_write(csr ## H, v_ >> 32);  \
>  })
>  
> +#define csr_read64(csr)                 \
> +({                                      \
> +    uint64_t v_ = csr_read(csr ## H);   \
This violates Misra 20.12 rule (docs/misra/rules.rst) as we use macro
parameter `csr` as an operand of ##, while it could be expanded as a
macro (e.g. csr_read64(CSR_HENVCFG)).
> +                                        \
> +    (v_ << 32) | csr_read(csr);         \
> +})
> +
>  /*
>   * The two halves are read by separate instructions, so a CSR which hardware
>   * increments can carry from the low half into the high one in between,
> @@ -72,6 +79,12 @@
>      (void)csr ## H;                 \
>      csr_read(csr);                  \
>  })
> +
> +#define csr_read64(csr)             \
> +({                                  \
> +    (void)csr ## H;                 \
Same as above.
> +    csr_read(csr);                  \
> +})
>  #endif
>  
>  #define csr_swap(csr, val)                                      \
> diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
> index 15e8fa1968..77ad888a2d 100644
> --- a/xen/arch/riscv/include/asm/domain.h
> +++ b/xen/arch/riscv/include/asm/domain.h
> @@ -29,6 +29,11 @@ struct arch_vcpu_io {
>  struct arch_vcpu {
>      struct vcpu_vmid vmid;
>  
> +    /*
> +     * The last CPU this vCPU ran on. Initialised to CPU_NONE.
> +     */
> +    unsigned int last_cpu;
> +
>      /*
>       * Callee saved registers for Xen's state used to switch from
>       * prev's stack to the next's stack during context switch.
> @@ -60,11 +65,19 @@ struct arch_vcpu {
>      register_t hcounteren;
>      register_t hedeleg;
>      register_t hideleg;
> -    register_t henvcfg;
> +    uint64_t   henvcfg;
>      register_t hstateen0;
> +    uint64_t   htimedelta;
>      register_t hvip;
>  
>      register_t vsatp;
> +    register_t vscause;
> +    register_t vsepc;
> +    uint64_t   vsie;
> +    register_t vsscratch;
> +    register_t vsstatus;
> +    register_t vstval;
> +    register_t vstvec;
>  
>      /*
>       * VCPU interrupts
> diff --git a/xen/arch/riscv/include/asm/p2m.h b/xen/arch/riscv/include/asm/p2m.h
> index 0d1dace1a0..9edf78377e 100644
> --- a/xen/arch/riscv/include/asm/p2m.h
> +++ b/xen/arch/riscv/include/asm/p2m.h
> @@ -262,7 +262,6 @@ struct page_info *p2m_get_page_from_gfn(struct p2m_domain *p2m, gfn_t gfn,
>  
>  void p2m_ctxt_switch_from(struct vcpu *p);
>  void p2m_ctxt_switch_to(struct vcpu *n);
> -void p2m_handle_vmenter(void);
>  
>  #endif /* ASM__RISCV__P2M_H */
>  
> diff --git a/xen/arch/riscv/include/asm/system.h b/xen/arch/riscv/include/asm/system.h
> index f33af64fd2..d350b9c959 100644
> --- a/xen/arch/riscv/include/asm/system.h
> +++ b/xen/arch/riscv/include/asm/system.h
> @@ -76,6 +76,8 @@ static inline bool local_irq_is_enabled(void)
>  
>  #define arch_fetch_and_add(x, v) __sync_fetch_and_add(x, v)
>  
> +struct vcpu *__context_switch(struct vcpu *prev, const struct vcpu *next);
> +
>  #endif /* __ASSEMBLER__ */
>  
>  #endif /* ASM__RISCV__SYSTEM_H */
> diff --git a/xen/arch/riscv/p2m.c b/xen/arch/riscv/p2m.c
> index de25607247..02e7f6364f 100644
> --- a/xen/arch/riscv/p2m.c
> +++ b/xen/arch/riscv/p2m.c
> @@ -243,6 +243,15 @@ static void p2m_tlb_flush(struct p2m_domain *p2m)
>  
>      p2m->need_flush = false;
>  
> +    /*
> +     * Order the p2m updates above against the read of dirty_cpumask below,
> +     * pairing with the barrier in p2m_vmid_switch_to(). Either that hart is
> +     * seen here and gets an HFENCE.GVMA, or it adds itself to the mask
> +     * afterwards, in which case it starts walking this p2m only once the
> +     * updates are visible to it.
> +     */
> +    smp_mb();
> +
>      sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0);
>  }
>  
> @@ -1504,16 +1513,110 @@ void p2m_ctxt_switch_from(struct vcpu *p)
>       * VMID, world-switch code should zero vsatp, then swap hgatp, then
>       * finally write the new vsatp value what will be done in
>       * p2m_ctxt_switch_to().
> -     * Note, that also HGATP update could happen in p2m_handle_vmenter().
>       */
>      p->arch.vsatp = csr_swap(CSR_VSATP, 0);
>  
>      /*
> -     * Nothing to do with HGATP as it will be update in p2m_ctxt_switch_to()
> -     * or/and in p2m_handle_vmenter().
> +     * Nothing to do with HGATP as it will be updated in
> +     * p2m_ctxt_switch_to().
>       */
>  }
>  
> +/*
> + * Domain whose p2m this hart's HGATP points at. ctxt_switch_to() bails out
> + * early for the idle vCPU, so HGATP survives a pass through idle and keeps
> + * pointing at the domain which ran here last. That domain, rather than the
> + * one the scheduler switched away from, is what owns this hart's G-stage
> + * translations. The hart also keeps its place in the owner's dirty_cpumask:
> + * p2m_tlb_flush() goes on reaching it, and a domain which idles between two
> + * runs on the same hart keeps its VMIDs.
> + *
> + * TODO: nothing resets hgatp_owner when its domain is destroyed, so a hart
> + *       which stays idle from the domain's last run until it is freed is
> + *       left with a dangling pointer. Once domain_relinquish_resources()
> + *       tears down the p2m, it has to reset hgatp_owner of every hart still
> + *       pointing at the domain (cmpxchg() against d), i.e. before the RCU
> + *       grace period of domain_destroy() and so before the domain is freed.
> + */
> +static DEFINE_PER_CPU(struct domain *, hgatp_owner);
> +
> +/*
> + * Update this hart's place in the dirty_cpumask of the domains involved and
> + * claim a VMID for n. Must be called before HGATP is pointed at n's p2m, as
> + * it is written with the VMID claimed here.
> + */
> +static void p2m_vmid_switch_to(struct vcpu *n)
> +{
> +    unsigned int cpu = smp_processor_id();
> +    struct domain *owner = this_cpu(hgatp_owner);
> +    bool need_flush;
> +
> +    ASSERT(!is_idle_vcpu(n));
> +
> +    if ( owner != n->domain )
> +    {
> +        /*
> +         * Once this hart drops out of the owner's dirty_cpumask it stops
> +         * being a target of p2m_tlb_flush(), while its TLB may still hold
> +         * G-stage translations of that domain: none of the vCPUs of that
> +         * domain which ran here has had its VMID invalidated. Move the hart
> +         * to a new VMID generation so that none of them can be reached
> +         * again.
> +         *
> +         * This has to precede the VMID claim below, so that n gets a VMID of
> +         * the new generation.
> +         */
> +        if ( owner )
> +        {
> +            vmid_flush_hart();
> +
> +            cpumask_clear_cpu(cpu, owner->dirty_cpumask);
> +        }
> +
> +        /*
> +         * Mark this hart in the incoming domain's dirty_cpumask before HGATP
> +         * is pointed at its p2m: from that write on the hart may cache the
> +         * p2m's translations, so p2m_tlb_flush() must already reach it.
> +         */
> +        cpumask_set_cpu(cpu, n->domain->dirty_cpumask);
> +
> +        /*
> +         * Pairs with the barrier in p2m_tlb_flush(). cpumask_set_cpu() is an
> +         * unordered AMO on RISC-V, so without this a concurrent flusher could
> +         * read the mask without this hart in it while this hart is already
> +         * walking the p2m it is about to be pointed at.
> +         */
> +        smp_mb();
> +
> +        this_cpu(hgatp_owner) = n->domain;
> +    }
> +
> +    /*
> +     * A VMID is meaningful only on the hart whose pool issued it: generations
> +     * are per-hart counters which all start at 1 and advance independently,
> +     * so the pair a vCPU brings from another hart may match this hart's
> +     * generation by coincidence, leaving the vCPU under a VMID which is live
> +     * here for someone else.
> +     */
So if I understand correctly, we are setting generation to 0 to mark the
vCPU that it is coming from other hart right?
Nit: If it's the case I may simplify the comment to:
      /*
       * VMIDs and generations are per hart, so a pair issued on another hart
       * may match this hart's generation by coincidence. Reset the generation
       * to 0, which no hart ever uses, so that a fresh VMID is claimed from
       * this hart's pool.
       */

> +    if ( n->arch.last_cpu != cpu )
> +        vmid_flush_vcpu(n);
>
Nit: I had to read this a couple of times to get why the guest entry is too
late. "makes HGATP live" doesn't say what happens once it is: if it's
because of speculation filling the TLB with G-stage entries tagged with
the VMID in HGATP, like you explained in the commit message then it may
worth spell out:

      /*
       * Claim the VMID before the caller writes HGATP rather than on guest
       * entry: from that write on, speculation may fill the TLB with this
       * domain's G-stage entries tagged with that VMID. A stale VMID may
       * already have been re-issued to a vCPU of another domain, which would
       * then hit these entries.
       */

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 11/39] xen/riscv: implement vCPU context switching

[Jan Beulich]

On 06.10.2026 18:50, Baptiste Le Duc wrote:
>> Implement context_switch() and the helpers it needs: save/restore of
>> H/VS CSRs, virtual timer and P2M context, and __context_switch() in
>> assembly, which switches Xen's own callee-saved state (and thereby the
>> stack) from prev to next. The virtual interrupt controller state isn't
>> switched here.
>>
>> Add offsets of struct arch_vcpu's xen_saved_context to asm-offsets.c for
>> use by __context_switch().
>>
>> henvcfg, htimedelta and vsie are 64-bit on both RV32 and RV64, so store
>> them as uint64_t and access them with csr_{read,write}64(). For that
>> purpose introduce csr_read64().
>>
>> The VMID has to be claimed on the context switch rather than on guest
>> entry: once p2m_ctxt_switch_to() writes HGATP, speculation can populate
>> G-stage entries under whatever VMID is written there. This is done by
>> p2m_vmid_switch_to(), called right before the HGATP write:
>>
>>  - VMIDs are a per-hart resource, so the (generation, vmid) pair of a
>>    vCPU which last ran on another hart is invalidated before a VMID is
>>    claimed for it. The local flush for a wrapped generation moves along
>>    with the claim.
>>  - Nothing is switched in for the idle vCPU, so HGATP keeps pointing at
>>    the p2m of the domain which ran on the hart last. Track that domain
>>    per hart (hgatp_owner) and keep the hart in its dirty_cpumask until a
>>    vCPU of another domain is switched in: p2m_tlb_flush() then goes on
>>    reaching the hart, and a domain which idles between two runs on the
>>    same hart keeps its VMIDs.
>>  - When the owner changes, the hart leaves the old owner's dirty_cpumask
>>    while its TLB may still hold that domain's G-stage translations, so it
>>    is moved to a new VMID generation, which makes none of them reachable
>>    again. It joins the new owner's dirty_cpumask before HGATP is written;
>>    a full barrier there, paired with one in p2m_tlb_flush(), guarantees
>>    that a concurrent P2M change either reaches the hart with an
>>    HFENCE.GVMA or is visible to it before it walks the p2m.
>>
>> That leaves p2m_handle_vmenter() with nothing to do, so drop it together
>> with its call from check_for_pcpu_work(): a VMID is only ever invalidated
>> while its vCPU isn't running, and p2m_tlb_flush() drops stale entries
>> with a remote HFENCE.GVMA rather than by retiring VMIDs. Unlike
>> p2m_handle_vmenter(), HGATP is written unconditionally, as it holds the
>> G-stage root too, which on a context switch belongs to another domain.
>>
>> While at it, fix the inclusion order of headers in asm-offsets.c: Xen's
>> headers go first, then arch specific ones.
>>
>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>> ---
>> Changes in v3:
>>  - Update the commit message.
>>  - Introduce csr_read64(), as vsie is 64-bit on RV32 too, i.e. has to be
>>    accessed as the vsie/vsieh pair there.
>>  - Make vsie declared as uint64_t instead of register_t.
>>  - Rename the parameter of save_csr_regs() to p and of restore_csr_regs()
>>    to n, to match their callers ctxt_switch_from()/ctxt_switch_to() and the
>>    rest of the context switch helpers (p2m_ctxt_switch_{from,to}(),
>>    vtimer_ctxt_switch_{from,to}()).
>>  - s/save_csr_regs/csr_regs_ctxt_switch_from/
>>  - s/restore_csr_regs/csr_regs_ctxt_switch_to/
>>  - Drop the forward declaration of struct vcpu in <asm/system.h>: the
>>    return type of __context_switch() already declares the tag at file
>>    scope.
>>  - Make the next parameter of __context_switch() pointer-to-const, as
>>    next's saved context is only read.
>>  - Extend the comment above __context_switch(): as ra is switched too, it
>>    doesn't return to its caller, but to where next last called it from
>>    (next's own context_switch(), or continue_new_vcpu() for a vCPU which
>>    has never run).
>>  - Call vtimer_ctxt_switch_to() after csr_regs_ctxt_switch_to(), so that
>>    ctxt_switch_to() restores state in the reverse order of
>>    ctxt_switch_from() saving it.
>>  - Rework p2m's VMID handling in the context switch: track the domain
>>    whose p2m HGATP points at in a per-CPU hgatp_owner, so a pass through
>>    idle no longer drops the hart from that domain's dirty_cpumask nor
>>    burns a VMID generation; move the dirty_cpumask update and the
>>    invalidation of a VMID brought from another hart from schedule_tail()
>>    and ctxt_switch_to() to p2m_ctxt_switch_to(), next to the VMID claim;
>>    add the barriers ordering the dirty_cpumask update against
>>    p2m_tlb_flush().
>>  - Initialise arch_vcpu.last_cpu to CPU_NONE instead of NR_CPUS.
>>  - Mark prev's dirty_cpu clean before ctxt_switch_to() and set current's
>>    after it.
>> ---
>> Changes in v2:
>>  - New patch.
>> ---
>> ---
>>  xen/arch/riscv/domain.c              | 127 ++++++++++++++++++++++
>>  xen/arch/riscv/entry.S               |  48 ++++++++
>>  xen/arch/riscv/include/asm/csr.h     |  13 +++
>>  xen/arch/riscv/include/asm/domain.h  |  15 ++-
>>  xen/arch/riscv/include/asm/p2m.h     |   1 -
>>  xen/arch/riscv/include/asm/system.h  |   2 +
>>  xen/arch/riscv/p2m.c                 | 157 ++++++++++++++++++---------
>>  xen/arch/riscv/riscv64/asm-offsets.c |  19 +++-
>>  xen/arch/riscv/stubs.c               |   5 -
>>  xen/arch/riscv/traps.c               |   2 -
>>  10 files changed, 330 insertions(+), 59 deletions(-)
>>
>> diff --git a/xen/arch/riscv/domain.c b/xen/arch/riscv/domain.c
>> index fd62296356..f80643e2d8 100644
>> --- a/xen/arch/riscv/domain.c
>> +++ b/xen/arch/riscv/domain.c
>> @@ -11,6 +11,7 @@
>>  #include <asm/bitops.h>
>>  #include <asm/cpufeature.h>
>>  #include <asm/csr.h>
>> +#include <asm/current.h>
>>  #include <asm/intc.h>
>>  #include <asm/mmio.h>
>>  #include <asm/riscv_encoding.h>
>> @@ -152,6 +153,8 @@ int arch_vcpu_create(struct vcpu *v)
>>      if ( is_idle_vcpu(v) )
>>          return 0;
>>  
>> +    v->arch.last_cpu = CPU_NONE;
>> +
>>      vcpu_csr_init(v);
>>  
>>      if ( (rc = vcpu_vtimer_init(v)) )
>> @@ -323,6 +326,130 @@ int arch_domain_create(struct domain *d,
>>      return rc;
>>  }
>>  
>> +static void csr_regs_ctxt_switch_from(struct vcpu *p)
>> +{
>> +    /*
>> +     * There is no need to save these CSRs as only hypervisor writes them in
>> +     * csr_regs_ctxt_switch_to() and guest can't access them so they shouldn't
>> +     * be stored here. Keep them commented here just for symmetry with the
>> +     * restore CSRs register part.
>> +     *
>> +     * p->arch.hedeleg = csr_read(CSR_HEDELEG);
>> +     * p->arch.hideleg = csr_read(CSR_HIDELEG);
>> +     * p->arch.henvcfg = csr_read64(CSR_HENVCFG);
>> +     * p->arch.hcounteren = csr_read(CSR_HCOUNTEREN);
>> +     * p->arch.htimedelta = csr_read64(CSR_HTIMEDELTA);
>> +     *
>> +     * if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
>> +     *     p->arch.hstateen0 = csr_read(CSR_HSTATEEN0);
>> +     */
>> +
>> +    p->arch.hvip = csr_read(CSR_HVIP);
>> +
>> +    p->arch.vsstatus = csr_read(CSR_VSSTATUS);
>> +    p->arch.vsie = csr_read64(CSR_VSIE);
>> +    p->arch.vstvec = csr_read(CSR_VSTVEC);
>> +    p->arch.vsscratch = csr_read(CSR_VSSCRATCH);
>> +    p->arch.vscause = csr_read(CSR_VSCAUSE);
>> +    p->arch.vstval = csr_read(CSR_VSTVAL);
>> +    p->arch.vsepc = csr_read(CSR_VSEPC);
>> +}
>> +
>> +static void csr_regs_ctxt_switch_to(struct vcpu *n)
>> +{
>> +    csr_write(CSR_HEDELEG, n->arch.hedeleg);
>> +    csr_write(CSR_HIDELEG, n->arch.hideleg);
>> +    csr_write(CSR_HVIP, n->arch.hvip);
>> +    csr_write64(CSR_HENVCFG, n->arch.henvcfg);
>> +    csr_write(CSR_HCOUNTEREN, n->arch.hcounteren);
>> +    csr_write64(CSR_HTIMEDELTA, n->arch.htimedelta);
>> +
>> +    if ( riscv_isa_extension_available(NULL, RISCV_ISA_EXT_smstateen) )
>> +        csr_write(CSR_HSTATEEN0, n->arch.hstateen0);
>> +
>> +    csr_write(CSR_VSSTATUS, n->arch.vsstatus);
>> +    csr_write64(CSR_VSIE, n->arch.vsie);
>> +    csr_write(CSR_VSTVEC, n->arch.vstvec);
>> +    csr_write(CSR_VSSCRATCH, n->arch.vsscratch);
>> +    csr_write(CSR_VSCAUSE, n->arch.vscause);
>> +    csr_write(CSR_VSTVAL, n->arch.vstval);
>> +    csr_write(CSR_VSEPC, n->arch.vsepc);
>> +}
>> +
>> +static void ctxt_switch_from(struct vcpu *p)
>> +{
>> +    /*
>> +     * When the idle VCPU is running, Xen will always stay in hypervisor
>> +     * mode.
>> +     * Therefore we don't need to save the context of an idle VCPU.
>> +     */
>> +    if ( is_idle_vcpu(p) )
>> +        return;
>> +
>> +    p2m_ctxt_switch_from(p);
>> +
>> +    vtimer_ctxt_switch_from(p);
>> +
>> +    csr_regs_ctxt_switch_from(p);
>> +}
>> +
>> +static void ctxt_switch_to(struct vcpu *n)
>> +{
>> +    /*
>> +     * When the idle VCPU is running, Xen will always stay in hypervisor
>> +     * mode.
>> +     * Therefore we don't need to restore the context of an idle VCPU.
>> +     */
>> +    if ( is_idle_vcpu(n) )
>> +        return;
>> +
>> +    csr_regs_ctxt_switch_to(n);
>> +
>> +    vtimer_ctxt_switch_to(n);
>> +
>> +    p2m_ctxt_switch_to(n);
>> +}
>> +
>> +static void schedule_tail(struct vcpu *prev)
>> +{
>> +    unsigned int cpu = smp_processor_id();
>> +
>> +    ASSERT(prev != current);
>> +
>> +    ctxt_switch_from(prev);
>> +
>> +    write_atomic(&prev->dirty_cpu, VCPU_CPU_CLEAN);
>> +
>> +    ctxt_switch_to(current);
>> +
>> +    write_atomic(&current->dirty_cpu, cpu);
>> +
>> +    current->arch.last_cpu = cpu;
>> +
>> +    /*
>> +     * sched_context_switched() internally uses a spinlock,
>> +     * which requires interrupts to be enabled.
>> +     */
>> +    local_irq_enable();
>> +
>> +    sched_context_switched(prev, current);
>> +}
>> +
>> +void context_switch(struct vcpu *prev, struct vcpu *next)
>> +{
>> +    ASSERT(local_irq_is_enabled());
>> +    ASSERT(prev != next);
>> +    ASSERT(!vcpu_cpu_dirty(next));
>> +
>> +    local_irq_disable();
>> +
>> +    set_current(next);
>> +
>> +    prev = __context_switch(prev, next);
>> +
>> +    schedule_tail(prev);
>> +}
>> +
>>  static void __init __maybe_unused build_assertions(void)
>>  {
>>      /*
>> diff --git a/xen/arch/riscv/entry.S b/xen/arch/riscv/entry.S
>> index 202a35fb03..017fbb0626 100644
>> --- a/xen/arch/riscv/entry.S
>> +++ b/xen/arch/riscv/entry.S
>> @@ -99,3 +99,51 @@ restore_registers:
>>  
>>          sret
>>  END(handle_trap)
>> +
>> +/*
>> + * struct vcpu *__context_switch(struct vcpu *prev, const struct vcpu *next)
>> + *
>> + * This is called on prev's stack, and returns on next's. As ra is
>> + * switched too, it doesn't return to its caller: it returns to where
>> + * next last called it from, i.e. into next's own context_switch(), or,
>> + * for a vCPU which has never run, to continue_new_vcpu() with an empty
>> + * stack.
>> + *
>> + * a0 - prev
>> + * a1 - next
>> + *
>> + * Returns prev in a0
>> + */
>> +FUNC(__context_switch)
>> +        REG_S   s0, VCPU_XEN_SAVED_CONTEXT_S0(a0)
>> +        REG_S   s1, VCPU_XEN_SAVED_CONTEXT_S1(a0)
>> +        REG_S   s2, VCPU_XEN_SAVED_CONTEXT_S2(a0)
>> +        REG_S   s3, VCPU_XEN_SAVED_CONTEXT_S3(a0)
>> +        REG_S   s4, VCPU_XEN_SAVED_CONTEXT_S4(a0)
>> +        REG_S   s5, VCPU_XEN_SAVED_CONTEXT_S5(a0)
>> +        REG_S   s6, VCPU_XEN_SAVED_CONTEXT_S6(a0)
>> +        REG_S   s7, VCPU_XEN_SAVED_CONTEXT_S7(a0)
>> +        REG_S   s8, VCPU_XEN_SAVED_CONTEXT_S8(a0)
>> +        REG_S   s9, VCPU_XEN_SAVED_CONTEXT_S9(a0)
>> +        REG_S   s10, VCPU_XEN_SAVED_CONTEXT_S10(a0)
>> +        REG_S   s11, VCPU_XEN_SAVED_CONTEXT_S11(a0)
>> +        REG_S   sp, VCPU_XEN_SAVED_CONTEXT_SP(a0)
>> +        REG_S   ra, VCPU_XEN_SAVED_CONTEXT_RA(a0)
>> +
>> +        REG_L   s0, VCPU_XEN_SAVED_CONTEXT_S0(a1)
>> +        REG_L   s1, VCPU_XEN_SAVED_CONTEXT_S1(a1)
>> +        REG_L   s2, VCPU_XEN_SAVED_CONTEXT_S2(a1)
>> +        REG_L   s3, VCPU_XEN_SAVED_CONTEXT_S3(a1)
>> +        REG_L   s4, VCPU_XEN_SAVED_CONTEXT_S4(a1)
>> +        REG_L   s5, VCPU_XEN_SAVED_CONTEXT_S5(a1)
>> +        REG_L   s6, VCPU_XEN_SAVED_CONTEXT_S6(a1)
>> +        REG_L   s7, VCPU_XEN_SAVED_CONTEXT_S7(a1)
>> +        REG_L   s8, VCPU_XEN_SAVED_CONTEXT_S8(a1)
>> +        REG_L   s9, VCPU_XEN_SAVED_CONTEXT_S9(a1)
>> +        REG_L   s10, VCPU_XEN_SAVED_CONTEXT_S10(a1)
>> +        REG_L   s11, VCPU_XEN_SAVED_CONTEXT_S11(a1)
>> +        REG_L   sp, VCPU_XEN_SAVED_CONTEXT_SP(a1)
>> +        REG_L   ra, VCPU_XEN_SAVED_CONTEXT_RA(a1)
>> +
>> +        ret
>> +END(__context_switch)

First: None of the reply context above is relevant in your reply. Why
did you keep it? As indicated before, this way you make every reader
scroll through many lines, until they would finally find the first
piece of the actual reply. (I've deliberately kept it all, to show
the percentage of the overall reply that this occupied.)

>> --- a/xen/arch/riscv/include/asm/csr.h
>> +++ b/xen/arch/riscv/include/asm/csr.h
>> @@ -40,6 +40,13 @@
>>      csr_write(csr ## H, v_ >> 32);  \
>>  })
>>  
>> +#define csr_read64(csr)                 \
>> +({                                      \
>> +    uint64_t v_ = csr_read(csr ## H);   \
> This violates Misra 20.12 rule (docs/misra/rules.rst) as we use macro
> parameter `csr` as an operand of ##, while it could be expanded as a
> macro (e.g. csr_read64(CSR_HENVCFG)).

Yes. But: You say nothing as to a possible (and plausible) different
way of coding this. Imo the best way to deal with this is by a
deviation (iirc we already have a few), yet I don't think we're going
to add deviations for RISC-V until we actually are about to routinely
scan the code.

Jan

Re: [PATCH v3 11/39] xen/riscv: implement vCPU context switching

[Baptiste Le Duc]

On 2026-10-07 10:04:12+02:00, Jan Beulich wrote:
> On 06.10.2026 18:50, Baptiste Le Duc wrote:
> 
> >> Implement context_switch() and the helpers it needs: save/restore of
> 
> First: None of the reply context above is relevant in your reply. Why
> did you keep it? As indicated before, this way you make every reader
> scroll through many lines, until they would finally find the first
> piece of the actual reply. (I've deliberately kept it all, to show
> the percentage of the overall reply that this occupied.)
> 
Yes sorry, I thought b4 would have trimmed for me. I would be careful
about that in next replies.
> >> --- a/xen/arch/riscv/include/asm/csr.h
> > This violates Misra 20.12 rule (docs/misra/rules.rst) as we use macro
> > parameter `csr` as an operand of ##, while it could be expanded as a
> > macro (e.g. csr_read64(CSR_HENVCFG)).
> 
> Yes. But: You say nothing as to a possible (and plausible) different
> way of coding this. Imo the best way to deal with this is by a
> deviation (iirc we already have a few), yet I don't think we're going
> to add deviations for RISC-V until we actually are about to routinely
> scan the code.
I didn't propose anything because I didn't have any other way to code
this. Concerning the fact to not add deviation for RISC-V I agree that
it doesn't make too much sense for the moment.
> 
> Jan

Re: [PATCH v3 12/39] xen/riscv: save and restore vsiselect on vCPU context switch

[Baptiste Le Duc]

> vsiselect is a per-hart CSR which a guest changes on its own: when V=1,
> VS-mode accesses to siselect are really accesses to vsiselect.
> Architecturally a vCPU has to find there the value it last wrote, but as
> long as the CSR isn't part of the vCPU context it finds whatever selector
> the vCPU which ran on the hart before it left behind. A guest which writes
> siselect, is descheduled and then reads sireg without rewriting siselect
> therefore reaches a register it never selected, and it can also observe
> another guest's selector value.
Nit: I'd suggest a more tighter version that might be more clear. Feel
free to use it:

    ```
    vsiselect isn't saved/restored, yet the guest writes it directly (VS-mode
    accesses to siselect go to vsiselect). After a context switch a vCPU
    therefore sees the selector left by the previous vCPU on that hart: a guest
    writing siselect, getting descheduled, then accessing sireg hits a register
    it never selected, and it can observe another guest's selector value.
    ```

> 
> When Smstateen is implemented, access to vsiselect and vsireg is gated by
> hstateen0.CSRIND (bit 60, SMSTATEEN0_SVSLCT in Xen's headers), and
> v->arch.hstateen0 holds the bits vcpu_csr_init() ended up with. A clear
> bit there covers the two cases in which the CSR has to be skipped:
> 
>  - Xen didn't hand the guest access to it, so the guest can't have changed
>    the CSR and there is no state to preserve;
>  - M-mode denied the state altogether. Smstateen makes a bit which is zero
>    in mstateen0 read-only zero in hstateen0, and a zero bit in mstateen0
>    traps accesses from every privilege mode less privileged than M-mode,
>    HS-mode included, so Xen couldn't even read the CSR to save it.
> 
> Without Smstateen no bit controls access to the CSR, so it is saved and
> restored whenever Ssaia is available.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 13/39] xen/riscv: introduce vintc_ctxt_switch_{from,to}()

[Baptiste Le Duc]

On Wed, 30 Sep 2026 18:23:32 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
> Virtual interrupt controller state must be preserved across vCPU context
> switches.
> 
> Introduce vintc_ctxt_switch_{from,to}() wrappers around new
> ctxt_switch_{from,to}() hooks in struct vintc_ops, and call them from the
> context switch path, so that this state can be saved/restored without
> knowing which vINTC variant a domain uses.
> 
> [...]

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 14/39] xen/riscv: add IMSIC vCPU context switch handlers

[Baptiste Le Duc]

> IMSIC state currently needs to track only which physical CPU owns a vCPU's
> IMSIC guest interrupt file, as the CPU id is part of the physical address
> the file is mapped at.
> 
> Add imsic_ctxt_switch_from() to record that CPU when a vCPU is switched
> out. A vCPU running on the s/w VS-file has no h/w file bound to a CPU, so
> there is nothing to record for it. Nothing reads the recorded value yet.
> 
> imsic_ctxt_switch_to() has nothing to do: switching a vCPU in on the CPU
> which owns its guest interrupt file needs no IMSIC update. Only a move to
> another CPU does (a new VGEIN and a remapped file), and that isn't part of
> the context switch.
Nit: I think it could make sense to mention that if a migration happen,
it would go anyway in imsic_migrate_vcpu() before
imsic_ctxt_switch_to().
> 
> Install both as the ctxt_switch_{from,to} hooks of struct vintc_ops. MSI
> delivery is the only mode Xen supports (aplic_init() panics on an APLIC
> without an "msi-parent" property) and the only one vAPLIC emulates, so the
> vAPLIC state to save and restore is always the IMSIC one and no
> vAPLIC-level forwarder is needed. Being indirect call targets, both
> handlers get cf_check.
> 
> Co-developed-by: Romain Caritey <Romain.Caritey@microchip.com>
Should add Signed-off-by Romain Caritey.
Without that, everything is ok from my side
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> --- a/xen/arch/riscv/vaplic.c
> +++ b/xen/arch/riscv/vaplic.c
> @@ -456,6 +456,12 @@ static const struct mmio_handler_ops vaplic_mmio_ops = {
>  static const struct vintc_ops vintc_ops = {
>      .vcpu_init = vcpu_imsic_init,
>      .vcpu_deinit = vcpu_imsic_deinit,
> +    /*
> +     * MSI delivery is the only supported mode, so the vAPLIC state to save
Nit: Do we want to always support only MSI delivery? If not, maybe adapt
the comment.

Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

-- 
Baptiste Le Duc <baptiste.le-duc@vates.tech>

Re: [PATCH v3 01/39] xen/riscv: drop INSN_{16,32}BIT_MASK and report unknown instruction lengths

[Jan Beulich]

On 05.10.2026 12:01, Baptiste Le Duc wrote:
> On Wed, 30 Sep 2026 18:18:43 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
>> Both constants were inherited from OpenSBI and their names are misleading.
>> The length of an instruction is encoded in its low bits: bits[1:0] != 11
>> means 16 bits, otherwise bits[4:2] != 111 means 32 bits, and anything else
>> is 48 bits or wider. 0x3 and 0x1c are the patterns compared against in
>> these two tests, not masks selecting a 16-bit or a 32-bit instruction. As
>> each constant is needed in one place only, once INSN_IS_16BIT() and
>> INSN_IS_32BIT() encapsulate the tests, drop them and spell the values out
>> in those two macros instead.
> 
> Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v3 02/39] xen/riscv: set the domain type for 64-bit kernel images

[Jan Beulich]

On 05.10.2026 12:01, Baptiste Le Duc wrote:
> On Wed, 30 Sep 2026 18:18:44 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
>> kernel_image64_probe() never set info->type, so every domain kept the
>> DOMAIN_32BIT value a zeroed KERNEL_INFO_INIT leaves behind, even though
>> Xen on RISC-V is only built for RV64 and accepts nothing but a 64-bit
>> Image.
>>
>> Nothing consumes d->type on RISC-V yet, so there is no functional change.
>>
>> [...]
> 
> Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Jan Beulich]

On 30.09.2026 18:18, Oleksii Kurochko wrote:
> On RV32 a 64-bit CSR is accessed as two halves, <csr> and <csr>H. Reading
> the halves with two separate instructions isn't always safe: for a CSR
> which hardware increments, a carry from the low half into the high one can
> happen in between, so a plain pair of reads can produce a value the CSR
> never held. Therefore the high half is re-read and the sequence retried if
> it changed in the meantime.
> 
> Only CSRs which change under Xen's feet need that, hence the name. The
> majority can be read with a plain pair of csr_read()s, so no helper for
> those is introduced here as there would be no user for it.
> 
> Use it for CSR_TIME, which is exactly such a counter, and widen cycles_t to
> uint64_t. Otherwise get_cycles() would still truncate the time counter to
> 32 bits on RV32.
> 
> No functional change on RV64.
> 
> Fixes: a541ddadec0a ("xen/riscv: introduce time.h")
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>

Nevertheless, a few remarks below.

> --- a/xen/arch/riscv/include/asm/csr.h
> +++ b/xen/arch/riscv/include/asm/csr.h
> @@ -39,12 +39,39 @@
>      csr_write(csr, v_);             \
>      csr_write(csr ## H, v_ >> 32);  \
>  })
> +
> +/*
> + * The two halves are read by separate instructions, so a CSR which hardware
> + * increments can carry from the low half into the high one in between,
> + * yielding a value the CSR never held. Re-read the high half and retry the
> + * sequence if it changed.
> + *
> + * Only needed for CSRs which change under Xen's feet.
> + */
> +#define csr_volatile_read64(csr)                    \
> +({                                                  \
> +    uint32_t hi_, lo_, old_;                        \
> +                                                    \
> +    hi_ = csr_read(csr ## H);                       \

This could be the initializer of the variable.

> +    do {                                            \
> +        old_ = hi_;                                 \
> +        lo_ = csr_read(csr);                        \
> +    } while ( (hi_ = csr_read(csr ## H)) != old_ ); \
> +                                                    \
> +    ((uint64_t)hi_ << 32) | lo_;                    \
> +})
>  #else
>  #define csr_write64(csr, val)       \
>  ({                                  \
>      csr_write(csr, val);            \
>      (void)csr ## H;                 \

While I notice the same exists here already, I wonder what the purpose of
this and ...

>  })
> +
> +#define csr_volatile_read64(csr)    \
> +({                                  \
> +    (void)csr ## H;                 \

... this is (and why their placements differ). Generally a pure 64-bit
environment would have no need for these ...H constants, and their
availability looks to be all that's checked for here. As Baptiste did
point out on patch 11, these uses constitute Misra violations, which we
then would need to deviate. I think we want to try to limit the number of
needed deviations.

And then, seeing how patch 11 introduces an exact copy of this as
csr_read64() - can't we simply have

#define csr_volatile_read64 csr_read64

?

Jan

Re: [PATCH v3 04/39] xen/riscv: request a G-stage flush on vmenter when VMIDs are disabled

[Jan Beulich]

On 05.10.2026 12:01, Baptiste Le Duc wrote:
> On Wed, 30 Sep 2026 18:18:46 +0200, Oleksii Kurochko <oleksii.kurochko@gmail.com> wrote:
>> vmid_handle_vmenter() reports that no flush is needed when VMIDs are
>> unavailable (e.g. vmid=off, or hardware with no more than one VMID bit).
>> Every domain then runs under VMID 0 with nothing flushed in between, so as
>> soon as a hart runs more than one domain, a domain entered there can use
>> the G-stage translations left behind by the domain which ran before it.
>>
>> Report that a flush is needed on every entry instead.
>>
>> [...]
> 
> Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v3 05/39] xen/riscv: use ~0 instead of ULONG_MAX to disable the VS-timer

[Jan Beulich]

On 30.09.2026 18:18, Oleksii Kurochko wrote:
> vstimecmp is a 64-bit CSR independently of XLEN, which is why it is
> written with csr_write64(). On RV32 that macro splits the value into
> the vstimecmp/vstimecmph pair, so passing ULONG_MAX (0xffffffff there)
> writes all ones to the low half and zero to the high half, leaving the
> CSR at 0x00000000ffffffff rather than at its maximum. A VS-timer irq
> would then become pending as soon as (time + htimedelta) reaches 2^32,
> which is exactly what the code is trying to avoid.
> 
> Use ~0 instead: csr_write64() converts its argument to uint64_t, and the
> conversion of the signed -1 sign-extends to all ones, so both halves get
> 0xffffffff. On RV64 the result is equal to ULONG_MAX, so no functional
> change there.
> 
> Fixes: 25e032730690 ("xen/riscv: allow Xen to use SSTC while hiding it from guests")
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Jan Beulich]

On 05.10.2026 12:01, Baptiste Le Duc wrote:
>> --- /dev/null
>> +++ b/xen/arch/riscv/include/asm/mmio.h
>> @@ -0,0 +1,63 @@
>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>> +#ifndef RISCV_MMIO_H
> Should be ASM__RISCV_MMIO_H instead to be consistant with other header
> files in riscv/include/asm.

No, please see the bottom of ./CODING_STYLE. It's many of the other headers
which would want updating (eventually; there's no urgency I think).

Jan

Re: [PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Oleksii Kurochko]

On 10/5/26 12:01 PM, Baptiste Le Duc wrote:
>> On RV32 a 64-bit CSR is accessed as two halves, <csr> and <csr>H. Reading
> As far as I can tell RV32 can't currently be selected (RISCV_64 is def_bool y,
> no RISCV_32 in Kconfig), so this path can't even be build-tested.

I've tested that manually.

> Is the retry
> loop really needed now, or a least in this patch series?
> 

I think it is worth keeping the retry loop now so that the code is ready 
if RV32 support needs to be added in the future, reducing the changes 
required at that point. It also helps explain why a separate 
csr_volatile_read64() function is needed in the first place.

~ Oleksii

Re: [PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Oleksii Kurochko]

On 10/8/26 4:46 PM, Jan Beulich wrote:
> On 30.09.2026 18:18, Oleksii Kurochko wrote:
>> On RV32 a 64-bit CSR is accessed as two halves, <csr> and <csr>H. Reading
>> the halves with two separate instructions isn't always safe: for a CSR
>> which hardware increments, a carry from the low half into the high one can
>> happen in between, so a plain pair of reads can produce a value the CSR
>> never held. Therefore the high half is re-read and the sequence retried if
>> it changed in the meantime.
>>
>> Only CSRs which change under Xen's feet need that, hence the name. The
>> majority can be read with a plain pair of csr_read()s, so no helper for
>> those is introduced here as there would be no user for it.
>>
>> Use it for CSR_TIME, which is exactly such a counter, and widen cycles_t to
>> uint64_t. Otherwise get_cycles() would still truncate the time counter to
>> 32 bits on RV32.
>>
>> No functional change on RV64.
>>
>> Fixes: a541ddadec0a ("xen/riscv: introduce time.h")
>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Reviewed-by: Jan Beulich <jbeulich@suse.com>

Thanks.

> 
> Nevertheless, a few remarks below.
> 
>> --- a/xen/arch/riscv/include/asm/csr.h
>> +++ b/xen/arch/riscv/include/asm/csr.h
>> @@ -39,12 +39,39 @@
>>       csr_write(csr, v_);             \
>>       csr_write(csr ## H, v_ >> 32);  \
>>   })
>> +
>> +/*
>> + * The two halves are read by separate instructions, so a CSR which hardware
>> + * increments can carry from the low half into the high one in between,
>> + * yielding a value the CSR never held. Re-read the high half and retry the
>> + * sequence if it changed.
>> + *
>> + * Only needed for CSRs which change under Xen's feet.
>> + */
>> +#define csr_volatile_read64(csr)                    \
>> +({                                                  \
>> +    uint32_t hi_, lo_, old_;                        \
>> +                                                    \
>> +    hi_ = csr_read(csr ## H);                       \
> 
> This could be the initializer of the variable.
> 
>> +    do {                                            \
>> +        old_ = hi_;                                 \
>> +        lo_ = csr_read(csr);                        \
>> +    } while ( (hi_ = csr_read(csr ## H)) != old_ ); \
>> +                                                    \
>> +    ((uint64_t)hi_ << 32) | lo_;                    \
>> +})
>>   #else
>>   #define csr_write64(csr, val)       \
>>   ({                                  \
>>       csr_write(csr, val);            \
>>       (void)csr ## H;                 \
> 
> While I notice the same exists here already, I wonder what the purpose of
> this and ...

I can drop it in separate patch. The purpose was just for the symmetry 
with RV32 case.

> 
>>   })
>> +
>> +#define csr_volatile_read64(csr)    \
>> +({                                  \
>> +    (void)csr ## H;                 \
> 
> ... this is (and why their placements differ). Generally a pure 64-bit
> environment would have no need for these ...H constants, and their
> availability looks to be all that's checked for here. As Baptiste did
> point out on patch 11, these uses constitute Misra violations, which we
> then would need to deviate. I think we want to try to limit the number of
> needed deviations.

If I understood MISRA Rule 20.12 correctly (Baptiste pointed me to it on
patch 11), the issue is that the csr parameter is used both as an operand
of ## and in ordinary, macro-expanded context.

I think it would be better to just rewrite these macros so that no
deviation is needed:

#ifdef CONFIG_RISCV_32
#define csr_write64(name, val)                  \
({                                              \
     uint64_t v_ = (val);                        \
     csr_write(CSR_ ## name, v_);                \
     csr_write(CSR_ ## name ## H, v_ >> 32);     \
})

#define csr_read64(name)                        \
({                                              \
     uint64_t v_ = csr_read(CSR_ ## name ## H);  \
                                                 \
     (v_ << 32) | csr_read(CSR_ ## name);        \
})
/* csr_volatile_read64() similarly */
#else
#define csr_write64(name, val)      \
({                                  \
     csr_write(CSR_ ## name, val);   \
     (void)CSR_ ## name ## H;        \
})
...
#endif

For RV64, '(void)CSR_ ## name ## H;' could be dropped altogether, and
csr_volatile_read64() would be reworked in the same way.

IIUC, with such definitions there is no violation of MISRA Rule 20.12.

> 
> And then, seeing how patch 11 introduces an exact copy of this as
> csr_read64() - can't we simply have
> 
> #define csr_volatile_read64 csr_read64
> 
> ?

It could be an option, then I will move csr_read64 to this patch.

Thanks.

~ Oleksii

Re: [PATCH v3 03/39] xen/riscv: introduce csr_volatile_read64()

[Jan Beulich]

On 09.10.2026 16:00, Oleksii Kurochko wrote:
> On 10/8/26 4:46 PM, Jan Beulich wrote:
>> On 30.09.2026 18:18, Oleksii Kurochko wrote:
>>> --- a/xen/arch/riscv/include/asm/csr.h
>>> +++ b/xen/arch/riscv/include/asm/csr.h
>>> @@ -39,12 +39,39 @@
>>>       csr_write(csr, v_);             \
>>>       csr_write(csr ## H, v_ >> 32);  \
>>>   })
>>> +
>>> +/*
>>> + * The two halves are read by separate instructions, so a CSR which hardware
>>> + * increments can carry from the low half into the high one in between,
>>> + * yielding a value the CSR never held. Re-read the high half and retry the
>>> + * sequence if it changed.
>>> + *
>>> + * Only needed for CSRs which change under Xen's feet.
>>> + */
>>> +#define csr_volatile_read64(csr)                    \
>>> +({                                                  \
>>> +    uint32_t hi_, lo_, old_;                        \
>>> +                                                    \
>>> +    hi_ = csr_read(csr ## H);                       \
>>
>> This could be the initializer of the variable.
>>
>>> +    do {                                            \
>>> +        old_ = hi_;                                 \
>>> +        lo_ = csr_read(csr);                        \
>>> +    } while ( (hi_ = csr_read(csr ## H)) != old_ ); \
>>> +                                                    \
>>> +    ((uint64_t)hi_ << 32) | lo_;                    \
>>> +})
>>>   #else
>>>   #define csr_write64(csr, val)       \
>>>   ({                                  \
>>>       csr_write(csr, val);            \
>>>       (void)csr ## H;                 \
>>
>> While I notice the same exists here already, I wonder what the purpose of
>> this and ...
> 
> I can drop it in separate patch. The purpose was just for the symmetry 
> with RV32 case.
> 
>>
>>>   })
>>> +
>>> +#define csr_volatile_read64(csr)    \
>>> +({                                  \
>>> +    (void)csr ## H;                 \
>>
>> ... this is (and why their placements differ). Generally a pure 64-bit
>> environment would have no need for these ...H constants, and their
>> availability looks to be all that's checked for here. As Baptiste did
>> point out on patch 11, these uses constitute Misra violations, which we
>> then would need to deviate. I think we want to try to limit the number of
>> needed deviations.
> 
> If I understood MISRA Rule 20.12 correctly (Baptiste pointed me to it on
> patch 11), the issue is that the csr parameter is used both as an operand
> of ## and in ordinary, macro-expanded context.
> 
> I think it would be better to just rewrite these macros so that no
> deviation is needed:
> 
> #ifdef CONFIG_RISCV_32
> #define csr_write64(name, val)                  \
> ({                                              \
>      uint64_t v_ = (val);                        \
>      csr_write(CSR_ ## name, v_);                \
>      csr_write(CSR_ ## name ## H, v_ >> 32);     \
> })
> 
> #define csr_read64(name)                        \
> ({                                              \
>      uint64_t v_ = csr_read(CSR_ ## name ## H);  \
>                                                  \
>      (v_ << 32) | csr_read(CSR_ ## name);        \
> })
> /* csr_volatile_read64() similarly */
> #else
> #define csr_write64(name, val)      \
> ({                                  \
>      csr_write(CSR_ ## name, val);   \
>      (void)CSR_ ## name ## H;        \
> })
> ...
> #endif
> 
> For RV64, '(void)CSR_ ## name ## H;' could be dropped altogether, and
> csr_volatile_read64() would be reworked in the same way.
> 
> IIUC, with such definitions there is no violation of MISRA Rule 20.12.

That's my understanding, too. So afaic - yes please (preferably with the
casts to void of the *H constants also dropped).

Jan

Re: [PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Oleksii Kurochko]

On 10/5/26 12:01 PM, Baptiste Le Duc wrote:

[...]

>> diff --git a/xen/arch/riscv/include/asm/mmio.h b/xen/arch/riscv/include/asm/mmio.h
>> new file mode 100644
>> index 0000000000..1bf48571a9
>> --- /dev/null
>> +++ b/xen/arch/riscv/include/asm/mmio.h
>> @@ -0,0 +1,63 @@
>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>> +#ifndef RISCV_MMIO_H
> Should be ASM__RISCV_MMIO_H instead to be consistant with other header
> files in riscv/include/asm.

According to the coding style document it should be 
<arch>_<something>_H. I understand that some headers in RISC-V looks in 
the way ASM__arch_something_H only becuase they were introduced before 
the header guard recommendations have been updated.

I think I could suggest a separate patch to rename all header guards, at 
least, for RISC-V to avoid such type of confusion in future.

>> +#define RISCV_MMIO_H
>> +
>> +#include <xen/lib.h>
>> +#include <xen/rwlock.h>
>> +
>> +struct domain;
>> +struct vcpu;
>> +
>> +#define MAX_IO_HANDLER  16
>> +
>> +typedef struct {
>> +    paddr_t gpa;
>> +    unsigned int len;  /* Access width in bytes */
>> +    bool is_write;
>> +    /* Store: value to write; load: value read (set by handler) */
>> +    register_t data;
>> +} mmio_info_t;
>> +
>> +enum io_state
>> +{
> Style: should be directly after enum io_state.

I will fix that.

>> +    IO_ABORT,       /* The IO was handled and led to an abort. */
>> +    IO_HANDLED,     /* The IO was successfully handled. */
>> +    IO_UNHANDLED,   /* No handler found for the IO. */
>> +};
>> +

[...]

>> +static enum io_state try_handle_mmio(mmio_info_t *info)
>> +{
>> +    struct vcpu *v = current;
>> +    struct mmio_handler handler = {};
>> +
>> +    if ( !find_mmio_handler(v->domain, info->gpa, &handler) )
>> +        return IO_UNHANDLED;
>> +
>> +    if ( info->is_write )
>> +        return handler.ops->write(v, info);
>> +    else
> The else is not necessary here, misra would call that "dead code".
> Here are two suggestions:
> 
>        if ( info->is_write )
>            return handler.ops->write(v, info);
> 
>        return handler.ops->read(v, info);

Sure, I'll drop the else and go with the first variant in the next
version.

I am not too much into MISRA stuff and just for my education could you 
please specify what MISRA rule has been violdated here?

I suppose it is because of Rule 2.2 (or 2.1?). I don't think Rule 2.2 
applies here. MISRA C:2012 defines dead code as "any operation that is 
executed but whose removal would not affect program behaviour" (as 
quoted in [1], Section IV). The else doesn't introduce any such 
operation: both forms have the same semantics, and the read() call under 
the else is executed whenever is_write is false, so it is neither dead 
nor unreachable (Rule 2.1).

FWIW, I also compared what gcc 16 generates for RISC-V: at -O0 and -Os
the variants with and without the else are byte-identical, and at
-O1/-O2 the only difference is which of the two calls is laid out as the
fall-through path; the instructions are otherwise the same. But probably 
this is not the best way to verify things...

Also, R2.2 is disapplied project-wide for Xen anyway (see 
docs/misra/deviations.rst).

[1] https://arxiv.org/abs/2212.13933

> or:
> 
>        return info->is_write ? handler.ops->write(v, info)
>                              : handler.ops->read(v, info);
> 

If I am not confused something above then why thos option doesn't 
violate MISRA rule abouve 'dead code' isn't ternary operation just an 
equivalent to if/else?

Thanks for review!

~ Oleksii

Re: [PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Oleksii Kurochko]

On 10/8/26 4:57 PM, Jan Beulich wrote:
> On 05.10.2026 12:01, Baptiste Le Duc wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/include/asm/mmio.h
>>> @@ -0,0 +1,63 @@
>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>>> +#ifndef RISCV_MMIO_H
>> Should be ASM__RISCV_MMIO_H instead to be consistant with other header
>> files in riscv/include/asm.
> 
> No, please see the bottom of ./CODING_STYLE. It's many of the other headers
> which would want updating (eventually; there's no urgency I think).

I will send a separate patch for that, at least, for RISC-V.

Thanks.

~ Oleksii

Re: [PATCH v3 07/39] xen/riscv: introduce CPU_NONE

[Oleksii Kurochko]

On 10/5/26 4:53 PM, Baptiste Le Duc wrote:
>> Introduce CPU_NONE, a CPU number which doesn't identify any physical
>> CPU, to be used wherever a pCPU has to be recorded but there may be
>> none.
>>
>> Its first user is vsfile_cpu in struct vimsic_state, which records the
>> pCPU whose h/w guest interrupt file backs the vCPU's IMSIC VS-file.
>> NR_CPUS has been used there as the value meaning that there is no such
>> pCPU, i.e. that the s/w VS-file is in use. NR_CPUS is however an upper
> Nit: It's hard to understand when reading the message why s/w is mentioning.
> I'd reword it to:
>      ```
>      NR_CPUS has been used there to mean that no h/w guest interrupt file is
>      attached (i.e. the vCPU uses the s/w VS-file), so there is no pCPU to
>      record.
>      ```
>> bound rather than a "no CPU" marker, and it already carries other
>> meanings: hartid_to_cpuid() returns it for a hart Xen doesn't know, and
>> pcpu_info[] uses it for an entry whose processor_id isn't valid yet.
>> Use CPU_NONE for vsfile_cpu instead, so that the s/w VS-file case can't
>> be mistaken for any of those.
> Nit: Same as above, I'd prefer something like:
>      ```
>      Use CPU_NONE for vsfile_cpu instead, so that "no h/w guest interrupt
>      file attached" can't be mistaken for an unknown hart or a not-yet-valid
>      pcpu_info[] entry.
>      ```

I will apply your suggestions.

> 
> Reviewed-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
> 

Thanks!

~ Oleksii

Re: [PATCH v3 06/39] xen/riscv: introduce device-agnostic MMIO emulation dispatch

[Jan Beulich]

On 30.09.2026 18:18, Oleksii Kurochko wrote:
> --- a/xen/arch/riscv/include/asm/domain.h
> +++ b/xen/arch/riscv/include/asm/domain.h
> @@ -9,6 +9,7 @@
>  
>  #include <asm/cpufeature.h>
>  #include <asm/guest-layout.h>
> +#include <asm/mmio.h>
>  #include <asm/p2m.h>
>  #include <asm/vtimer.h>
>  
> @@ -101,6 +102,8 @@ struct arch_domain {
>      const unsigned long *isa;
>  
>      struct vintc *vintc;
> +
> +    struct vmmio vmmio;

This is a decent size structure, so I wonder whether this really wants
embedding in struct domain. If it was a pointer here (with the struct
dynamically allocated), the struct definition itself may not even need
exposing in a header.

> --- /dev/null
> +++ b/xen/arch/riscv/mmio.c
> @@ -0,0 +1,175 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +
> +#include <xen/bsearch.h>
> +#include <xen/lib.h>
> +#include <xen/rwlock.h>
> +#include <xen/sched.h>
> +#include <xen/string.h>
> +
> +#include <asm/current.h>
> +#include <asm/mmio.h>
> +
> +/*
> + * bsearch() comparator: @key holds the address to look up in its addr field,
> + * @elem is an entry of vmmio->handlers. Relies on the regions not
> + * overlapping, which register_mmio_handler() enforces.
> + */
> +static int cmp_mmio_handler(const void *key, const void *elem)
> +{
> +    const struct mmio_handler *handler0 = key;
> +    const struct mmio_handler *handler1 = elem;
> +
> +    if ( handler0->addr < handler1->addr )
> +        return -1;
> +
> +    if ( handler0->addr >= (handler1->addr + handler1->size) )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +/*
> + * Return a copy of the matching handler rather than a pointer into
> + * vmmio->handlers: a concurrent register_mmio_handler() shifts entries
> + * up to keep the array sorted, so an escaped pointer could refer to a
> + * different (or torn) entry once the lock is dropped. The copy stays
> + * valid as the ops structures are never freed.
> + */
> +static bool find_mmio_handler(struct domain *d, paddr_t gpa,

The first parameter here apparently wants to be named currd, much like ...

> +                              struct mmio_handler *out)
> +{
> +    struct vmmio *vmmio = &d->arch.vmmio;
> +    struct mmio_handler key = { .addr = gpa };
> +    const struct mmio_handler *handler;
> +
> +    read_lock(&vmmio->lock);
> +    handler = bsearch(&key, vmmio->handlers, vmmio->num_entries,
> +                      sizeof(*handler), cmp_mmio_handler);
> +    if ( handler )
> +        *out = *handler;
> +    read_unlock(&vmmio->lock);
> +
> +    return handler != NULL;
> +}
> +
> +static enum io_state try_handle_mmio(mmio_info_t *info)
> +{
> +    struct vcpu *v = current;

... this variable wants to be named curr. Unless of course there are
plans to not unconditionally use current here.

> +int register_mmio_handler(struct domain *d,
> +                          const struct mmio_handler_ops *ops,
> +                          paddr_t addr, paddr_t size)
> +{
> +    struct vmmio *vmmio = &d->arch.vmmio;
> +    struct mmio_handler *handlers = vmmio->handlers;
> +    paddr_t end = addr + size;
> +    unsigned int i;
> +    int rc = 0;
> +    bool overlap;
> +
> +    if ( !ops || !ops->read || !ops->write || end <= addr )
> +        return -EINVAL;

I think this checking goes a little too far: It's absurd for some call site
to pass NULL into this function. The other three checks are sufficiently
plausible, albeit I wonder whether this wouldn't want to be an assertion
then.

Jan

Re: [PATCH v3 07/39] xen/riscv: introduce CPU_NONE

[Jan Beulich]

On 30.09.2026 18:18, Oleksii Kurochko wrote:
> Introduce CPU_NONE, a CPU number which doesn't identify any physical
> CPU, to be used wherever a pCPU has to be recorded but there may be
> none.
> 
> Its first user is vsfile_cpu in struct vimsic_state, which records the
> pCPU whose h/w guest interrupt file backs the vCPU's IMSIC VS-file.
> NR_CPUS has been used there as the value meaning that there is no such
> pCPU, i.e. that the s/w VS-file is in use. NR_CPUS is however an upper
> bound rather than a "no CPU" marker, and it already carries other
> meanings: hartid_to_cpuid() returns it for a hart Xen doesn't know, and
> pcpu_info[] uses it for an entry whose processor_id isn't valid yet.

Yet these could (and imo should, subsequently) entirely plausibly also be
switched to CPU_NONE. IOW I don't think ...

> Use CPU_NONE for vsfile_cpu instead, so that the s/w VS-file case can't
> be mistaken for any of those.

... this is a plausible justification (nor in its re-written form as per
Baptiste's suggestion).

If you want such a distinction, you'll need separate sentinels for every
one such field.

> --- a/xen/arch/riscv/include/asm/config.h
> +++ b/xen/arch/riscv/include/asm/config.h
> @@ -136,6 +136,9 @@
>  #define OPT_CONSOLE_STR "dtuart"
>  #define INVALID_VCPU_ID MAX_VIRT_CPUS
>  
> +/* A CPU number which doesn't identify any physical CPU. */
> +#define CPU_NONE (~0U)
> +
>  /* Linkage for RISCV */
>  #ifdef __ASSEMBLER__
>  #define CODE_FILL /* empty */
> --- a/xen/arch/riscv/include/asm/imsic.h
> +++ b/xen/arch/riscv/include/asm/imsic.h
> @@ -71,7 +71,7 @@ struct vimsic_state {
>       */
>      unsigned int guest_file_id;
>      /*
> -     * s/w IMSIC VS-file -> vsfile_cpu == NR_CPUS
> +     * s/w IMSIC VS-file -> vsfile_cpu == CPU_NONE
>       * h/w IMSIC VS-file -> vsfile_cpu < NR_CPUS
>       */
>      unsigned int vsfile_cpu;

As can be seen from the comment alone, there's now the risk of
CPU_NONE < NR_CPUS, at which point users of the field will be
screwed. I think you want

    BUILD_BUG_ON(CPU_NONE < NR_CPUS);

somewhere.

Jan