Re: [PATCH] KVM: PPC: Book3S HV nestedv2: Don't drop pending doorbell across L2 entry

[Anushree Mathur <anushree.mathur@linux.ibm.com>]

On 03/08/26 9:14 AM, Vaibhav Jain wrote:
> On nestedv2 the L1 converts a pending doorbell into guest DPDES state at
> the top of kvmhv_vcpu_entry_nestedv2() and immediately forgets about it:
>
> 	if (vcpu->arch.doorbell_request) {
> 		vcpu->arch.doorbell_request = 0;
> 		kvmppc_set_dpdes(vcpu, 1);
> 	}
>
> Clearing 'doorbell_request' at this point assumes that handing DPDES to the
> L0 is equivalent to the L2 having taken the doorbell. That is not true, and
> the doorbell can be lost in two ways:
>
>    - The block runs before the lazy_irq_pending() check, so the doorbell is
>      consumed even on the path that returns 0 without ever calling
>      H_GUEST_RUN_VCPU.
>
>    - DPDES stays pending in the L2 until it is actually delivered. The L2
>      may exit for an unrelated reason (hcall, page fault, HDEC) with the
>      doorbell still set, typically because it was running with MSR[EE]=0.
>      Nothing reloads DPDES afterwards, so the L1 never learns this.
>
> Once 'doorbell_request' has been cleared, the L1 has no record of the
> pending doorbell. kvmppc_doorbell_pending() returns false, so
> kvmppc_read_dpdes() reports the target thread as idle when a sibling vCPU
> emulates 'mfspr DPDES', and the vCPU can be treated as having no work
> pending and blocked. From the L2's point of view the doorbell is silently
> lost, which shows up as an SMT guest hanging on a doorbell-based IPI.
>
> Fix this by making 'doorbell_request' track the L2's DPDES rather than
> being consumed by entry:
>
>    - inject DPDES after the early-return paths and before
>      kvmhv_nestedv2_flush_vcpu() serializes it into the vcpu run input
>      buffer, and no longer clear 'doorbell_request' there,
>
>    - after H_GUEST_RUN_VCPU, reload DPDES from the L0. The run output only
>      carries the state the L0 chose to return and the 'valids' bitmap is
>      zeroed on exit, so an explicit kvmhv_nestedv2_cached_reload() is
>      needed to see the L2's current value,
>
>    - if DPDES is still set the doorbell was not delivered, so keep
>      'doorbell_request' pending so that it is re-injected on the next
>      entry; otherwise clear it.
>
> This keeps a pending doorbell visible to the L1 for as long as the L2 has
> not consumed it, so vCPU wakeup and DPDES emulation on sibling vCPUs stay
> consistent with the L2's actual state.
>
> Fixes: 54ec2bd9e017 ("KVM: PPC: Book3S HV nestedv2: Fix doorbell emulation")
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> Assisted-by: Claude:Opus-5
> ---
>   arch/powerpc/kvm/book3s_hv.c | 20 +++++++++++++++-----
>   1 file changed, 15 insertions(+), 5 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 61dbeea317f3..40f8717b8a7d 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -15,6 +15,7 @@
>    * by Alexander Graf <agraf@suse.de>.
>    */
>   
> +#include "asm/guest-state-buffer.h"
>   #include <linux/kvm_host.h>
>   #include <linux/kernel.h>
>   #include <linux/err.h>
> @@ -4253,11 +4254,6 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,
>   	int trap;
>   	long rc;
>   
> -	if (vcpu->arch.doorbell_request) {
> -		vcpu->arch.doorbell_request = 0;
> -		kvmppc_set_dpdes(vcpu, 1);
> -	}
> -
>   	io = &vcpu->arch.nestedv2_io;
>   
>   	msr = mfmsr();
> @@ -4265,6 +4261,9 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,
>   	if (lazy_irq_pending())
>   		return 0;
>   
> +	if (vcpu->arch.doorbell_request)
> +		kvmppc_set_dpdes(vcpu, 1);
> +
>   	rc = kvmhv_nestedv2_flush_vcpu(vcpu, time_limit);
>   	if (rc < 0)
>   		return -EINVAL;
> @@ -4296,6 +4295,17 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,
>   	if (rc < 0)
>   		return -EINVAL;
>   
> +	/* Check if privileged door bell was requested and handled */
> +	if (vcpu->arch.vcore->dpdes) {
> +		kvmhv_nestedv2_cached_reload(vcpu, KVMPPC_GSID_DPDES);
> +		if (vcpu->arch.vcore->dpdes)
> +			vcpu->arch.doorbell_request |= vcpu->arch.vcore->dpdes;
> +		else
> +			cpu->arch.doorbell_request = 0;
> +	} else {
> +		vcpu->arch.doorbell_request = 0;
> +	}
> +
>   	timer_rearm_host_dec(*tb);
>   
>   	/* Record context switch and guest_run_time data */
Hi Vaibhav,

I have tested this patch and it is still giving me the issue I reported, 
so here is my analysis:

a) Without applying the patch :

1) Start the guest and run stress-ng as below for sometime
localhost:~ # stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 
1G --sched other --timeout 3600000s
stress-ng: info:  [1464] setting to a 41 days, 16 hours, 0 secs run per 
stressor
stress-ng: info:  [1464] dispatching hogs: 4 cpu, 2 vm, 2 hdd



2) Start the migration from H1 to H2:

ltc-lp7:~ # virsh migrate --live --domain sles16_anu 
qemu+ssh://10.xx.xx.xx/system --verbose --undefinesource --persistent 
--auto-converge --postcopy
(root@10.xx.xx.xx) Password:
Migration: [100.00 %]

3) Migration got completed but guest is not getting recovered from 
continuous softlockups

[ 1336.003836][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for 
977s! [htxd_monitor:1337]
[ 1336.006834][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 
1002s! [rcu_exp_par_gp_:19]
[ 1346.015839][    C0] BUG: workqueue lockup - pool cpus=1 node=0 
flags=0x0 nice=0 stuck for 1090s!
[ 1346.016355][    C0] BUG: workqueue lockup - pool cpus=3 node=0 
flags=0x0 nice=0 stuck for 1107s!
[ 1346.016874][    C0] BUG: workqueue lockup - pool cpus=7 node=0 
flags=0x0 nice=0 stuck for 1093s!
[ 1356.007835][    C6] watchdog: BUG: soft lockup - CPU#6 stuck for 
912s! [systemd:1353]
[ 1356.008835][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 
998s! [systemd-journal:570]
[ 1360.003836][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for 
999s! [htxd_monitor:1337]
[ 1360.006834][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 
1024s! [rcu_exp_par_gp_:19]
[ 1368.933835][    C4] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 1368.933973][    C4] rcu:     4-....: (1129830 ticks this GP) 
idle=afc4/1/0x4000000000000002 softirq=3694/428556 fqs=259639
[ 1368.934106][    C4] rcu:              hardirqs   softirqs  csw/system
[ 1368.934188][    C4] rcu:      number:        1     444039       0
[ 1368.934271][    C4] rcu:     cputime:        3          8 1096165  
  ==> 1110021(ms)
[ 1368.934373][    C4] rcu:     (t=1140022 jiffies g=6177 q=1684 ncpus=8)
[ 1376.224839][    C0] BUG: workqueue lockup - pool cpus=1 node=0 
flags=0x0 nice=0 stuck for 1120s!
[ 1376.225307][    C0] BUG: workqueue lockup - pool cpus=3 node=0 
flags=0x0 nice=0 stuck for 1138s!
[ 1376.225428][    C0] BUG: workqueue lockup - pool cpus=5 node=0 
flags=0x0 nice=0 stuck for 715s!
[ 1376.225548][    C0] BUG: workqueue lockup - pool cpus=6 node=0 
flags=0x0 nice=0 stuck for 1027s!
[ 1376.225667][    C0] BUG: workqueue lockup - pool cpus=7 node=0 
flags=0x0 nice=0 stuck for 1123s!
[ 1444.006835][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 
1100s! [rcu_exp_par_gp_:19]


b) Even after applying the patch also it is giving same softlockup issue 
as mentioned above:
Though I have enough vcpus and memory on the guest (16 vcpus , 13Gi of 
memory)  and ample amount of memory and cpus
present on host still these softlockups are happening after applying the 
patch too. I tried reducing stress also on the guest
but still this issue is seen.

stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G --sched 
other --timeout 3600000s

If you are planning to send next version of this patch,
Please do add my reported-by:
Reported-by: Anushree Mathur <anushree.mathur@linux.ibm.com>