[PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Florian Bezdeka]

Hi all, 

this is hopefully the final version of the dovetail KVM guest support
enablement. Dovetail failed to boot without patch 1 and crashed without
patch 2 under high IO load scenarios.

Tested with LOCKDEP / PROVE_LOCKING enabled. Stress-Tests passed on two
different hypervisors, namely KVM and VirtualBox.

All dovetail versions (down to 5.10) are affected. Backporting would be
required on my end, I need 6.1 to be fixed.

This patches apply down to 5.15, 5.10 would need manual backporting. I
can look into that once this series is accepted for 6.9.

CC: Jan Kiszka <jan.kiszka@siemens.com>
CC: Philippe Gerum <rpm@xenomai.org>

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
Changes in v5:
- Patch 2: safe_halt() -> native_safe_halt()
- Link to v4: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v4-0-77471c9473d8@siemens.com

Changes in v4:
- Patch 2: Sligthly changed the patch description 
- Patch 2: Do not stall the inband stage in task context (handle Linux
  IRQ context and task context the same way)
- Link to v3: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v3-0-ebcf517500a3@siemens.com

Changes in v3:
- Removed RFC tag
- Took care of review comments from Jan (patch 2) 
- Link to v2: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v2-0-5534fcae4eb5@siemens.com

Changes in v2:
- Drop patch one 
- Now patch 1: Handle async page faults inband, rework patch subject
- Now patch 2: Fix kvm_wait(), PV spinlocks
- Link to v1: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v1-0-1c895c256f8d@siemens.com

---
Florian Bezdeka (2):
      KVM: x86: dovetail: Move handling async page faults to the inband stage
      KVM: x86: dovetail: Allow kvm_wait() to be called from any context

 arch/x86/include/asm/idtentry.h |  2 +-
 arch/x86/kernel/irq_pipeline.c  |  6 ++++++
 arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
 3 files changed, 29 insertions(+), 8 deletions(-)
---
base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad

Best regards,
-- 
Florian Bezdeka <florian.bezdeka@siemens.com>

[PATCH dovetail v5 1/2] KVM: x86: dovetail: Move handling async page faults to the inband stage

[Florian Bezdeka]

Dovetail was stuck in the boot phase as KVM guest on a system that
reported the asynchronous page fault handling feature.

The apic_eoi() call in sysvec_kvm_asyncpf_interrupt() has no effect in
case CONFIG_IRQPIPELINE is enabled, thus those IRQs / vectors where
never ACKed at APIC level, the CPUs stalled.

Instead of migrating to __apic_oie() the IDT entry for
HYPERVISOR_CALLBACK_VECTOR for KVM on x86 is now marked as PIPELINED.
Other architectures (and other guest extensions) already used
DECLARE_IDTENTRY_SYSVEC_PIPELINED().

The imbalance of how HYPERVISOR_CALLBACK_VECTOR is being used /
pipelined goes back to 5.10. Backporting might be required.

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 arch/x86/include/asm/idtentry.h | 2 +-
 arch/x86/kernel/irq_pipeline.c  | 6 ++++++
 arch/x86/kernel/kvm.c           | 3 ++-
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 500f4e8c6542..e544b24f61bf 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -829,7 +829,7 @@ DECLARE_IDTENTRY_SYSVEC_PIPELINED(HYPERVISOR_CALLBACK_VECTOR,	sysvec_xen_hvm_cal
 #endif
 
 #ifdef CONFIG_KVM_GUEST
-DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_CALLBACK_VECTOR,	sysvec_kvm_asyncpf_interrupt);
+DECLARE_IDTENTRY_SYSVEC_PIPELINED(HYPERVISOR_CALLBACK_VECTOR,	sysvec_kvm_asyncpf_interrupt);
 #endif
 
 #undef X86_TRAP_OTHER
diff --git a/arch/x86/kernel/irq_pipeline.c b/arch/x86/kernel/irq_pipeline.c
index 15b6cbdb8f13..73e97fba55e7 100644
--- a/arch/x86/kernel/irq_pipeline.c
+++ b/arch/x86/kernel/irq_pipeline.c
@@ -155,6 +155,12 @@ static void do_sysvec_inband(struct irq_desc *desc, struct pt_regs *regs)
 		run_sysvec_on_irqstack_cond(__sysvec_acrn_hv_callback,
 					regs);
 		break;
+#endif
+#ifdef CONFIG_KVM_GUEST
+	case HYPERVISOR_CALLBACK_VECTOR:
+		run_sysvec_on_irqstack_cond(__sysvec_kvm_asyncpf_interrupt,
+					    regs);
+		break;
 #endif
 	case LOCAL_TIMER_VECTOR:
 		run_sysvec_on_irqstack_cond(__sysvec_apic_timer_interrupt,
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index c65e88f94f32..655b1cf8c9ab 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -292,7 +292,8 @@ noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
 	return true;
 }
 
-DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_asyncpf_interrupt)
+DEFINE_IDTENTRY_SYSVEC_PIPELINED(HYPERVISOR_CALLBACK_VECTOR,
+				 sysvec_kvm_asyncpf_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 	u32 token;

-- 
2.39.2

[PATCH dovetail v5 2/2] KVM: x86: dovetail: Allow kvm_wait() to be called from any context

[Florian Bezdeka]

kvm_wait() unconditionally enabled HW interrupts when called from
pipeline entry code. That broke the PV (paravirtualized) spinlock
implementation.

The following calling contexts are supported now:
  - pipeline entry, hard IRQs off:
    Wakeup on vCPU kicks only, triggered by the current lock owner via
    hypercall. Using halt().

  - Linux IRQ context, hard IRQs on, in-band IRQs off:
    Wakeup on the next hard IRQ, or by a vCPU kick.
    Using native_safe_halt() to statisfy lockdep.

  - Task context, hard IRQs on, in-band IRQs on:
    Same as Linux IRQ context.

The !CONFIG_IRQ_PIPELINE part is now re-synced with the version of
Linux to keep trace events.

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
 arch/x86/kernel/kvm.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 655b1cf8c9ab..655056316d27 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -1059,22 +1059,36 @@ static void kvm_wait(u8 *ptr, u8 val)
 	 * for irq enabled case to avoid hang when lock info is overwritten
 	 * in irq spinlock slowpath and no spurious interrupt occur to save us.
 	 */
-	if (irqs_disabled()) {
-		hard_local_irq_disable();
 
+#ifdef CONFIG_IRQ_PIPELINE
+
+	if (hard_irqs_disabled()) {
 		if (READ_ONCE(*ptr) == val)
 			halt();
+	} else {
+		hard_local_irq_disable();
+
+		if (READ_ONCE(*ptr) == val)
+			native_safe_halt(); /* enables hard IRQs */
+		else
+			hard_local_irq_enable();
+	}
 
-		hard_local_irq_enable();
+#else /* !CONFIG_IRQ_PIPELINE */
+
+	if (irqs_disabled()) {
+		if (READ_ONCE(*ptr) == val)
+			halt();
 	} else {
-		local_irq_disable_full();
+		local_irq_disable();
 
 		/* safe_halt() will enable IRQ */
 		if (READ_ONCE(*ptr) == val)
 			safe_halt();
-
-		local_irq_enable_full();
+		else
+			local_irq_enable();
 	}
+#endif
 }
 
 /*

-- 
2.39.2

Re: [PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Philippe Gerum]

Florian Bezdeka <florian.bezdeka@siemens.com> writes:

> Hi all, 
>
> this is hopefully the final version of the dovetail KVM guest support
> enablement. Dovetail failed to boot without patch 1 and crashed without
> patch 2 under high IO load scenarios.
>
> Tested with LOCKDEP / PROVE_LOCKING enabled. Stress-Tests passed on two
> different hypervisors, namely KVM and VirtualBox.
>
> All dovetail versions (down to 5.10) are affected. Backporting would be
> required on my end, I need 6.1 to be fixed.
>
> This patches apply down to 5.15, 5.10 would need manual backporting. I
> can look into that once this series is accepted for 6.9.
>
> CC: Jan Kiszka <jan.kiszka@siemens.com>
> CC: Philippe Gerum <rpm@xenomai.org>
>
> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
> Changes in v5:
> - Patch 2: safe_halt() -> native_safe_halt()
> - Link to v4: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v4-0-77471c9473d8@siemens.com
>
> Changes in v4:
> - Patch 2: Sligthly changed the patch description 
> - Patch 2: Do not stall the inband stage in task context (handle Linux
>   IRQ context and task context the same way)
> - Link to v3: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v3-0-ebcf517500a3@siemens.com
>
> Changes in v3:
> - Removed RFC tag
> - Took care of review comments from Jan (patch 2) 
> - Link to v2: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v2-0-5534fcae4eb5@siemens.com
>
> Changes in v2:
> - Drop patch one 
> - Now patch 1: Handle async page faults inband, rework patch subject
> - Now patch 2: Fix kvm_wait(), PV spinlocks
> - Link to v1: https://lore.kernel.org/r/20240531-flo-fix-kvm-guest-for-6-9-v1-0-1c895c256f8d@siemens.com
>
> ---
> Florian Bezdeka (2):
>       KVM: x86: dovetail: Move handling async page faults to the inband stage
>       KVM: x86: dovetail: Allow kvm_wait() to be called from any context
>
>  arch/x86/include/asm/idtentry.h |  2 +-
>  arch/x86/kernel/irq_pipeline.c  |  6 ++++++
>  arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
>  3 files changed, 29 insertions(+), 8 deletions(-)
> ---
> base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
> change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad
>
> Best regards,

Merged, thanks.

-- 
Philippe.

Re: [PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Florian Bezdeka]

On Wed, 2024-06-26 at 10:40 +0200, Philippe Gerum wrote:
> > Florian Bezdeka (2):
> >       KVM: x86: dovetail: Move handling async page faults to the inband stage
> >       KVM: x86: dovetail: Allow kvm_wait() to be called from any context
> > 
> >  arch/x86/include/asm/idtentry.h |  2 +-
> >  arch/x86/kernel/irq_pipeline.c  |  6 ++++++
> >  arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
> >  3 files changed, 29 insertions(+), 8 deletions(-)
> > ---
> > base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
> > change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad
> > 
> > Best regards,
> 
> Merged, thanks.

Nice. What is the procedure now to get that into the (non-rebase) 6.1
branch?

As written in the cover letter, patches should apply down to 5.15, I
can look into 5.10 again if necessary.

Best regards,
Florian

> 
> -- 
> Philippe.

Re: [PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Jan Kiszka]

On 26.06.24 12:44, Florian Bezdeka wrote:
> On Wed, 2024-06-26 at 10:40 +0200, Philippe Gerum wrote:
>>> Florian Bezdeka (2):
>>>       KVM: x86: dovetail: Move handling async page faults to the inband stage
>>>       KVM: x86: dovetail: Allow kvm_wait() to be called from any context
>>>
>>>  arch/x86/include/asm/idtentry.h |  2 +-
>>>  arch/x86/kernel/irq_pipeline.c  |  6 ++++++
>>>  arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
>>>  3 files changed, 29 insertions(+), 8 deletions(-)
>>> ---
>>> base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
>>> change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad
>>>
>>> Best regards,
>>
>> Merged, thanks.
> 
> Nice. What is the procedure now to get that into the (non-rebase) 6.1
> branch?
> 
> As written in the cover letter, patches should apply down to 5.15, I
> can look into 5.10 again if necessary.

I will take care of the 5.x updates.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Florian Bezdeka]

Hi Philippe, Hi Jan,

On Wed, 2024-06-26 at 13:29 +0200, Jan Kiszka wrote:
> On 26.06.24 12:44, Florian Bezdeka wrote:
> > On Wed, 2024-06-26 at 10:40 +0200, Philippe Gerum wrote:
> > > > Florian Bezdeka (2):
> > > >       KVM: x86: dovetail: Move handling async page faults to the inband stage
> > > >       KVM: x86: dovetail: Allow kvm_wait() to be called from any context
> > > > 
> > > >  arch/x86/include/asm/idtentry.h |  2 +-
> > > >  arch/x86/kernel/irq_pipeline.c  |  6 ++++++
> > > >  arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
> > > >  3 files changed, 29 insertions(+), 8 deletions(-)
> > > > ---
> > > > base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
> > > > change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad
> > > > 
> > > > Best regards,
> > > 
> > > Merged, thanks.
> > 
> > Nice. What is the procedure now to get that into the (non-rebase) 6.1
> > branch?
> > 
> > As written in the cover letter, patches should apply down to 5.15, I
> > can look into 5.10 again if necessary.
> 
> I will take care of the 5.x updates.
> 
> Jan

Any news about the backporting / release process? Additional help
needed?

Best regards,
Florian

> 
> -- 
> Siemens AG, Technology
> Linux Expert Center

Re: [PATCH dovetail v5 0/2] Fix KVM guest support of dovetail

[Jan Kiszka]

On 08.07.24 22:59, Florian Bezdeka wrote:
> Hi Philippe, Hi Jan,
> 
> On Wed, 2024-06-26 at 13:29 +0200, Jan Kiszka wrote:
>> On 26.06.24 12:44, Florian Bezdeka wrote:
>>> On Wed, 2024-06-26 at 10:40 +0200, Philippe Gerum wrote:
>>>>> Florian Bezdeka (2):
>>>>>       KVM: x86: dovetail: Move handling async page faults to the inband stage
>>>>>       KVM: x86: dovetail: Allow kvm_wait() to be called from any context
>>>>>
>>>>>  arch/x86/include/asm/idtentry.h |  2 +-
>>>>>  arch/x86/kernel/irq_pipeline.c  |  6 ++++++
>>>>>  arch/x86/kernel/kvm.c           | 29 ++++++++++++++++++++++-------
>>>>>  3 files changed, 29 insertions(+), 8 deletions(-)
>>>>> ---
>>>>> base-commit: 17865a05b459df6e7e7e38018b15b29a96df49af
>>>>> change-id: 20240531-flo-fix-kvm-guest-for-6-9-5ef61aac08ad
>>>>>
>>>>> Best regards,
>>>>
>>>> Merged, thanks.
>>>
>>> Nice. What is the procedure now to get that into the (non-rebase) 6.1
>>> branch?
>>>
>>> As written in the cover letter, patches should apply down to 5.15, I
>>> can look into 5.10 again if necessary.
>>
>> I will take care of the 5.x updates.
>>
>> Jan
> 
> Any news about the backporting / release process? Additional help
> needed?
> 

I'm only waiting for a dovetail release of a kernel containing those
patches.

BTW, a rebase over 6.6.37 would also be nice because of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.6.y&id=2ab254507c87b8ce6787caa31154e089a640dc20
vs.
https://source.denx.de/Xenomai/xenomai-images/-/commit/8d2f7d5cf008797caa28d3a064c1378274b52c63

Jan

-- 
Siemens AG, Technology
Linux Expert Center