Re: [PATCH] ppc/xive: Trigger the pending interrupt for a guest after migration

[Anushree Mathur <anushree.mathur@linux.ibm.com>]

On 24/08/26 12:33 PM, Gautam Menghani wrote:
> When a KVM guest is migrated to another LPAR while a stress-ng workload
> is running, lockups are seen when the guest resumes on the destination
> LPAR. This root cause is that if there was a pending interrupt for a cpu
> on the source machine but it was not presented yet when the migration
> started, the interrupt is not retriggered on the destination LPAR. This
> is evident when running 'info pic' on the destination LPAR:
>
> $ virsh qemu-monitor-command --hmp migrate 'info pic' | grep -e OS -e MSI
> CPU[0000]:   OS    00   ff  00    ff   ff  00  ff   ff  80000400
> CPU[0001]:   OS    00   ff  00    ff   ff  00  ff   ff  80000401
> CPU[0002]:   OS    00   ff  00    ff   ff  00  ff   ff  80000402
> CPU[0003]:   OS    00   ff  00    ff   ff  00  ff   ff  80000403
> CPU[0004]:   OS    00   ff  00    ff   ff  00  ff   ff  80000404
> CPU[0005]:   OS    00   ff  00    ff   ff  00  ff   ff  80000405
> CPU[0006]:   OS    80   ff  02    ff   ff  00  ff   06  80000406
> CPU[0007]:   OS    00   ff  00    ff   ff  00  ff   ff  80000407
>    00000000 MSI --    00000010   0/6   3071/16384 @3710000 ^1 [ 80000010 80000010 80000010 80000010 80000010 ^00000000 ]
>    00000001 MSI --    00000010   1/6   1852/16384 @3980000 ^1 [ 80000010 80000033 80000033 80000033 80000033 ^00000000 ]
>    00000002 MSI --    00000010   2/6   4424/16384 @40c0000 ^1 [ 80000020 80000020 80000020 80000010 80000010 ^00000000 ]
>    00000003 MSI --    00000010   3/6   1913/16384 @4220000 ^1 [ 80000010 80000032 80000010 80000010 80000010 ^00000000 ]
>    00000004 MSI --    00000010   4/6   2946/16384 @4360000 ^1 [ 80000010 80000010 80000010 80000010 80000010 ^00000000 ]
>    00000005 MSI --    00000010   5/6   2313/16384 @44a0000 ^1 [ 80000010 80000010 80000010 80000010 80000010 ^00000000 ]
>    00000006 MSI P-    00000010   6/6   5825/16384 @45f0000 ^1 [ 80000024 80000024 80000024 80000010 80000016 ^00000000 ]
>    00000007 MSI --    00000010   7/6   2346/16384 @4760000 ^1 [ 80000032 80000032 80000010 80000032 80000032 ^00000000 ]
>
> Fix this by calling xive_tctx_pipr_set() in
> vmstate_xive_tctx_post_load(), which ends up calling qemu_irq_raise() to
> present the interrupt.
>
> Signed-off-by: Gautam Menghani <gautam@linux.ibm.com>
> ---
>   hw/intc/xive.c | 3 +++
>   1 file changed, 3 insertions(+)
>
> diff --git a/hw/intc/xive.c b/hw/intc/xive.c
> index f473e6ac77..712498d4af 100644
> --- a/hw/intc/xive.c
> +++ b/hw/intc/xive.c
> @@ -1080,6 +1080,9 @@ static int vmstate_xive_tctx_post_load(void *opaque, int version_id)
>               error_report_err(local_err);
>               return ret;
>           }
> +    } else {
> +        uint8_t pipr = tctx->regs[TM_QW1_OS + TM_PIPR];
> +        xive_tctx_pipr_set(tctx, TM_QW1_OS, pipr, 0);
>       }
>   
>       return 0;
Hi Gautam,
I have tested this patch and it has been fixing the issue which I 
reported. Here is my analysis:

a) Without applying the patch :

1) Start the guest and run stress-ng as below for sometime
localhost:~ # stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G
--sched other --timeout 3600000s
stress-ng: info:  [1464] setting to a 41 days, 16 hours, 0 secs run per
stressor
stress-ng: info:  [1464] dispatching hogs: 4 cpu, 2 vm, 2 hdd



2) Start the migration from H1 to H2:

ltc-lp7:~ # virsh migrate --live --domain sles16_anu
qemu+ssh://10.xx.xx.xx/system --verbose --undefinesource --persistent
--auto-converge --postcopy
Migration: [100.00 %]

3) Migration got completed but guest is not getting recovered from
continuous softlockups even while having ample number of vcpus and
amount of memory free on host and guest both -

[ 1336.003836][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for 977s!
[htxd_monitor:1337]
[ 1336.006834][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1002s!
[rcu_exp_par_gp_:19]
[ 1346.015839][    C0] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0
nice=0 stuck for 1090s!
[ 1346.016355][    C0] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0
nice=0 stuck for 1107s!
[ 1346.016874][    C0] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0
nice=0 stuck for 1093s!
[ 1356.007835][    C6] watchdog: BUG: soft lockup - CPU#6 stuck for 912s!
[systemd:1353]
[ 1356.008835][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 998s!
[systemd-journal:570]
[ 1360.003836][    C1] watchdog: BUG: soft lockup - CPU#1 stuck for 999s!
[htxd_monitor:1337]
[ 1360.006834][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1024s!
[rcu_exp_par_gp_:19]
[ 1368.933835][    C4] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 1368.933973][    C4] rcu:     4-....: (1129830 ticks this GP)
idle=afc4/1/0x4000000000000002 softirq=3694/428556 fqs=259639
[ 1368.934106][    C4] rcu:              hardirqs   softirqs  csw/system
[ 1368.934188][    C4] rcu:      number:        1     444039 0
[ 1368.934271][    C4] rcu:     cputime:        3          8 1096165   ==>
1110021(ms)
[ 1368.934373][    C4] rcu:     (t=1140022 jiffies g=6177 q=1684 ncpus=8)
[ 1376.224839][    C0] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0
nice=0 stuck for 1120s!
[ 1376.225307][    C0] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0
nice=0 stuck for 1138s!
[ 1376.225428][    C0] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0
nice=0 stuck for 715s!
[ 1376.225548][    C0] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x0
nice=0 stuck for 1027s!
[ 1376.225667][    C0] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0
nice=0 stuck for 1123s!
[ 1444.006835][    C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1100s!
[rcu_exp_par_gp_:19]



b) After using the patch applied qemu this issue has been fixed. I tried multiple
migration instances between both the hosts and this issue is not getting seen.
Even I did try this same scenario with threads as 1/2/4/8 and for everything it worked fine.
Thanks for working on the fix.


Please feel free to add my tested-by:

Tested-by: Anushree Mathur<anushree.mathur@linux.ibm.com>

and please add my reported-by too:

Reported-by: Anushree Mathur <anushree.mathur@linux.ibm.com>

Thank you,
Anushree Mathur