Question about migration on ARM system

Question about migration on ARM system

[Zhou Wang via]

Hi,

I am tring to heterogeneous live migration on ARM64 host. Now I just use
below kernel/qemu branch to have a try:
https://github.com/hisilicon/kernel-dev/tree/private-v6.11-rc2-hisi-migrn-wip
https://github.com/hisilicon/qemu/tree/stable-8.2-kunpeng920-initial-v2

Currently guest on GICv4.1 and GICv3 host can do migration successfully. I am
not sure that it is expected.

GICv4.1 exports GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq to guest, so
guests(GICv3) on GICv4.1 and GICv3 host will have different state here. So
basically a guest on host GICv4.1 will lost state when it move to a host with
GICv3.

It seems that current mainline qemu does not consider GICD_TYPER2, and guest
OS(linux) does not use GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq(just
check cap and set req), so current test has been passed.

In fact, there was some discussions about this problem:
https://lore.kernel.org/kvmarm/20200304203330.4967-21-maz@kernel.org/

I am not sure if we should prevent users from doing migration between host
GICv3 and GICv4.1?

Thanks,
Zhou

Re: Question about migration on ARM system

[Marc Zyngier]

On Thu, 14 Nov 2024 09:03:24 +0000,
Zhou Wang <wangzhou1@hisilicon.com> wrote:
> 
> Hi,
> 
> I am tring to heterogeneous live migration on ARM64 host. Now I just use
> below kernel/qemu branch to have a try:
> https://github.com/hisilicon/kernel-dev/tree/private-v6.11-rc2-hisi-migrn-wip
> https://github.com/hisilicon/qemu/tree/stable-8.2-kunpeng920-initial-v2
> 
> Currently guest on GICv4.1 and GICv3 host can do migration successfully. I am
> not sure that it is expected.

It isn't. Or rather, it shouldn't. Are you reporting a KVM problem or
a QEMU problem?

> 
> GICv4.1 exports GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq to guest, so
> guests(GICv3) on GICv4.1 and GICv3 host will have different state here. So
> basically a guest on host GICv4.1 will lost state when it move to a host with
> GICv3.
> 
> It seems that current mainline qemu does not consider GICD_TYPER2, and guest
> OS(linux) does not use GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq(just
> check cap and set req), so current test has been passed.

Well, the guest won't test that after migration. It doesn't even know
it is migrated. But Linux definitely uses it at probe time.

> 
> In fact, there was some discussions about this problem:
> https://lore.kernel.org/kvmarm/20200304203330.4967-21-maz@kernel.org/
> 
> I am not sure if we should prevent users from doing migration between host
> GICv3 and GICv4.1?

We should allow the v3->v4.1 path (we can hide the nAS SGIs), but we
should not it the other way around.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question about migration on ARM system

[Zhou Wang via]

On 2024/11/14 20:33, Marc Zyngier wrote:
> On Thu, 14 Nov 2024 09:03:24 +0000,
> Zhou Wang <wangzhou1@hisilicon.com> wrote:
>>
>> Hi,
>>
>> I am tring to heterogeneous live migration on ARM64 host. Now I just use
>> below kernel/qemu branch to have a try:
>> https://github.com/hisilicon/kernel-dev/tree/private-v6.11-rc2-hisi-migrn-wip
>> https://github.com/hisilicon/qemu/tree/stable-8.2-kunpeng920-initial-v2
>>
>> Currently guest on GICv4.1 and GICv3 host can do migration successfully. I am
>> not sure that it is expected.
> 
> It isn't. Or rather, it shouldn't. Are you reporting a KVM problem or
> a QEMU problem?

I think KVM checks it in vgic_mmio_uaccess_write_v3_misc, however, current qemu
does not consider GICD_TYPER2 during migration.

In fact, I am wondering the description about GICD_TYPER2 in GIC spec. It said
"This register is present only when FEAT_GICv4p1 is implemented", from the view
of guest, we have a GICv3 device, however, we can see that GICD_TYPER2.nASSGIcap
is 1.

> 
>>
>> GICv4.1 exports GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq to guest, so
>> guests(GICv3) on GICv4.1 and GICv3 host will have different state here. So
>> basically a guest on host GICv4.1 will lost state when it move to a host with
>> GICv3.
>>
>> It seems that current mainline qemu does not consider GICD_TYPER2, and guest
>> OS(linux) does not use GICD_TYPER2.nASSGIcap and GICD_CTRL.nASSGIreq(just
>> check cap and set req), so current test has been passed.
> 
> Well, the guest won't test that after migration. It doesn't even know
> it is migrated. But Linux definitely uses it at probe time.
> 
>>
>> In fact, there was some discussions about this problem:
>> https://lore.kernel.org/kvmarm/20200304203330.4967-21-maz@kernel.org/
>>
>> I am not sure if we should prevent users from doing migration between host
>> GICv3 and GICv4.1?
> 
> We should allow the v3->v4.1 path (we can hide the nAS SGIs), but we
> should not it the other way around.

Make sense, in the case of v3->v4.1, source and destination will use SGI
with an active state.

Thanks,
Zhou

> 
> Thanks,
> 
> 	M.
> 

Re: Question about migration on ARM system

[Peter Maydell]

On Thu, 14 Nov 2024 at 13:48, Zhou Wang via <qemu-devel@nongnu.org> wrote:
>
> On 2024/11/14 20:33, Marc Zyngier wrote:
> > On Thu, 14 Nov 2024 09:03:24 +0000,
> > Zhou Wang <wangzhou1@hisilicon.com> wrote:
> >>
> >> Hi,
> >>
> >> I am tring to heterogeneous live migration on ARM64 host. Now I just use
> >> below kernel/qemu branch to have a try:
> >> https://github.com/hisilicon/kernel-dev/tree/private-v6.11-rc2-hisi-migrn-wip
> >> https://github.com/hisilicon/qemu/tree/stable-8.2-kunpeng920-initial-v2
> >>
> >> Currently guest on GICv4.1 and GICv3 host can do migration successfully. I am
> >> not sure that it is expected.
> >
> > It isn't. Or rather, it shouldn't. Are you reporting a KVM problem or
> > a QEMU problem?
>
> I think KVM checks it in vgic_mmio_uaccess_write_v3_misc, however, current qemu
> does not consider GICD_TYPER2 during migration.

That's because current QEMU's idea of a GIC device assumes
v3 or v4. We don't emulate v4.1 and we never updated the
kvm-arm-gicv3 device to handle anything newer than GICv3.

-- PMM