* KVM Live migration with GICv3
@ 2015-08-11 5:25 Vijay Kilari
2015-08-11 6:41 ` Pavel Fedin
2015-08-12 16:34 ` Marc Zyngier
0 siblings, 2 replies; 9+ messages in thread
From: Vijay Kilari @ 2015-08-11 5:25 UTC (permalink / raw)
To: kvmarm, Christoffer Dall, Alex Bennée, p.fedin
Hi,
I have prototyped Live migration with GICv3.
For this I have made following changes
1) Save and Restore of GICv3 registers in QEMU.
- For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3,
we have to save/restore GICD, GICR and ICC registers.
However ICC registers are system registers which cannot be
accessed @ EL0 level (SRE=1). So these ICC registers should be
accessed as mmio registers by QEMU, for this we have to add ioctl to
access ICC @ EL1 level similar to GICC registers of GICv2.
2) KVM ioctls in kernel provides only 32-bit register access to GIC
registers, where
as some registers in GICD/GICR requires 64-bit register access. I
propose to use mmio.flag to specify 32/64 bit access.
3) KVM ioctls to access ICC registers for GICv3
Please provide your initial feedback. Let me know if some of this
issues are already fixed
I am attending KVM-forum next week @ Seattle. We can discuss there as well.
Regards
Vijay
^ permalink raw reply [flat|nested] 9+ messages in thread* RE: KVM Live migration with GICv3 2015-08-11 5:25 KVM Live migration with GICv3 Vijay Kilari @ 2015-08-11 6:41 ` Pavel Fedin 2015-08-11 15:15 ` Vijay Kilari 2015-08-12 16:34 ` Marc Zyngier 1 sibling, 1 reply; 9+ messages in thread From: Pavel Fedin @ 2015-08-11 6:41 UTC (permalink / raw) To: 'Vijay Kilari', kvmarm, 'Christoffer Dall', 'Alex Bennée' Hello! > I have prototyped Live migration with GICv3. This is interesting of course, i will need this for my project too and i was also going to take a look at this later. > For this I have made following changes > > 1) Save and Restore of GICv3 registers in QEMU. > - For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3, > we have to save/restore GICD, GICR and ICC registers. > However ICC registers are system registers which cannot be > accessed @ EL0 level (SRE=1). So these ICC registers should be > accessed as mmio registers by QEMU, for this we have to add ioctl to > access ICC @ EL1 level similar to GICC registers of GICv2. Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why? The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used to access anything inside the VM, including system registers. I believe you would need only to implement handlers for GICv3 registers. > Please provide your initial feedback. Would be nice to take a look at patches. > I am attending KVM-forum next week @ Seattle. We can discuss there as well. I would love to go there, and my company wanted to send me there. But it appears to be extremely difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is running out, so i'm unlikely to attend, unfortunately. Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: KVM Live migration with GICv3 2015-08-11 6:41 ` Pavel Fedin @ 2015-08-11 15:15 ` Vijay Kilari 2015-08-11 15:24 ` Pavel Fedin 2015-08-25 11:27 ` Pavel Fedin 0 siblings, 2 replies; 9+ messages in thread From: Vijay Kilari @ 2015-08-11 15:15 UTC (permalink / raw) To: Pavel Fedin; +Cc: kvmarm On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote: > Hello! > >> I have prototyped Live migration with GICv3. > > This is interesting of course, i will need this for my project too and i was also going to take a look at this later. > >> For this I have made following changes >> >> 1) Save and Restore of GICv3 registers in QEMU. >> - For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3, >> we have to save/restore GICD, GICR and ICC registers. >> However ICC registers are system registers which cannot be >> accessed @ EL0 level (SRE=1). So these ICC registers should be >> accessed as mmio registers by QEMU, for this we have to add ioctl to >> access ICC @ EL1 level similar to GICC registers of GICv2. > > Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why? > The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used to access anything inside the VM, including system registers. I believe you would need only to implement handlers for GICv3 registers. I mean CPU interface registers GICC* defined for vgicv2 under "vgic_cpu_ranges[]" in virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well. So to access vgic_cpu_ranges[] for vgicv3, we need to register device for CPU registers space similar to gicv2 in QEMU for gicv3. ----------------------------------------------------------------------------------------- memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s), "kvm-gic_cpu", 0x1000); sysbus_init_mmio(sbd, &s->cpuiomem[0]); kvm_arm_register_device(&s->cpuiomem[0], (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT) | KVM_VGIC_V3_ADDR_TYPE_CPU, KVM_DEV_ARM_VGIC_GRP_ADDR, KVM_VGIC_V3_ADDR_TYPE_CPU, s->dev_fd); ------------------------------------------------------------------------------------------- Anyway, I will send RFC patches. > >> Please provide your initial feedback. > > Would be nice to take a look at patches. > >> I am attending KVM-forum next week @ Seattle. We can discuss there as well. > > I would love to go there, and my company wanted to send me there. But it appears to be extremely difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is running out, so i'm unlikely to attend, unfortunately. > > Kind regards, > Pavel Fedin > Expert Engineer > Samsung Electronics Research center Russia > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: KVM Live migration with GICv3 2015-08-11 15:15 ` Vijay Kilari @ 2015-08-11 15:24 ` Pavel Fedin 2015-08-12 16:17 ` Marc Zyngier 2015-08-25 11:27 ` Pavel Fedin 1 sibling, 1 reply; 9+ messages in thread From: Pavel Fedin @ 2015-08-11 15:24 UTC (permalink / raw) To: 'Vijay Kilari'; +Cc: kvmarm Hello! > I mean CPU interface registers GICC* defined for vgicv2 under > "vgic_cpu_ranges[]" in > virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well. > So to access vgic_cpu_ranges[] for vgicv3, we need to register device > for CPU registers space > similar to gicv2 in QEMU for gicv3. This thing is optional and can be missing. It is for backwards compatibility only, it allows to run GICv2 guests on GICv3. You should not use this API for accessing system registers. Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: KVM Live migration with GICv3 2015-08-11 15:24 ` Pavel Fedin @ 2015-08-12 16:17 ` Marc Zyngier 2015-08-12 16:33 ` Peter Maydell 0 siblings, 1 reply; 9+ messages in thread From: Marc Zyngier @ 2015-08-12 16:17 UTC (permalink / raw) To: Pavel Fedin, 'Vijay Kilari'; +Cc: kvmarm On 11/08/15 16:24, Pavel Fedin wrote: > Hello! > >> I mean CPU interface registers GICC* defined for vgicv2 under >> "vgic_cpu_ranges[]" in >> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well. >> So to access vgic_cpu_ranges[] for vgicv3, we need to register device >> for CPU registers space >> similar to gicv2 in QEMU for gicv3. > > This thing is optional and can be missing. It is for backwards > compatibility only, it allows to run GICv2 guests on GICv3. You > should not use this API for accessing system registers. Even more than that. The KVM emulation of GICv3 doesn't have *any* GICv2 compatibility. And that's on purpose. So trying to save the GICC registers should give you an error, because these registers *do not exist* as far as a guest is concerned. If you plan to do anything for GICv3, you should only deal with with the system register version of the CPU interface. Thanks, M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: KVM Live migration with GICv3 2015-08-12 16:17 ` Marc Zyngier @ 2015-08-12 16:33 ` Peter Maydell 0 siblings, 0 replies; 9+ messages in thread From: Peter Maydell @ 2015-08-12 16:33 UTC (permalink / raw) To: Marc Zyngier; +Cc: kvmarm@lists.cs.columbia.edu On 12 August 2015 at 17:17, Marc Zyngier <marc.zyngier@arm.com> wrote: > If you plan to do anything for GICv3, you should only deal with with the > system register version of the CPU interface. Also, please make sure that you think about (and in terms of) the underlying state inside the GIC, which is closely related to but not necessarily identical to "contents of all the registers". Migration cares about saving and restoring state. -- PMM ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: KVM Live migration with GICv3 2015-08-11 15:15 ` Vijay Kilari 2015-08-11 15:24 ` Pavel Fedin @ 2015-08-25 11:27 ` Pavel Fedin 2015-08-26 14:24 ` Vijay Kilari 1 sibling, 1 reply; 9+ messages in thread From: Pavel Fedin @ 2015-08-25 11:27 UTC (permalink / raw) To: 'Vijay Kilari'; +Cc: kvmarm Hello! Where are you? I'd like to take a look at your code. Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia > -----Original Message----- > From: Vijay Kilari [mailto:vijay.kilari@gmail.com] > Sent: Tuesday, August 11, 2015 6:16 PM > To: Pavel Fedin > Cc: kvmarm@lists.cs.columbia.edu; Christoffer Dall; Alex Bennée > Subject: Re: KVM Live migration with GICv3 > > On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote: > > Hello! > > > >> I have prototyped Live migration with GICv3. > > > > This is interesting of course, i will need this for my project too and i was also going to take a look at > this later. > > > >> For this I have made following changes > >> > >> 1) Save and Restore of GICv3 registers in QEMU. > >> - For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3, > >> we have to save/restore GICD, GICR and ICC registers. > >> However ICC registers are system registers which cannot be > >> accessed @ EL0 level (SRE=1). So these ICC registers should be > >> accessed as mmio registers by QEMU, for this we have to add ioctl to > >> access ICC @ EL1 level similar to GICC registers of GICv2. > > > > Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why? > > The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used > to access anything inside the VM, including system registers. I believe you would need only to > implement handlers for GICv3 registers. > > I mean CPU interface registers GICC* defined for vgicv2 under > "vgic_cpu_ranges[]" in > virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well. > So to access vgic_cpu_ranges[] for vgicv3, we need to register device > for CPU registers space > similar to gicv2 in QEMU for gicv3. > > ----------------------------------------------------------------------------------------- > memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s), > "kvm-gic_cpu", 0x1000); > sysbus_init_mmio(sbd, &s->cpuiomem[0]); > kvm_arm_register_device(&s->cpuiomem[0], > (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT) > | KVM_VGIC_V3_ADDR_TYPE_CPU, > KVM_DEV_ARM_VGIC_GRP_ADDR, > KVM_VGIC_V3_ADDR_TYPE_CPU, > s->dev_fd); > ------------------------------------------------------------------------------------------- > > Anyway, I will send RFC patches. > > > > >> Please provide your initial feedback. > > > > Would be nice to take a look at patches. > > > >> I am attending KVM-forum next week @ Seattle. We can discuss there as well. > > > > I would love to go there, and my company wanted to send me there. But it appears to be extremely > difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is > running out, so i'm unlikely to attend, unfortunately. > > > > Kind regards, > > Pavel Fedin > > Expert Engineer > > Samsung Electronics Research center Russia > > > > _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: KVM Live migration with GICv3 2015-08-25 11:27 ` Pavel Fedin @ 2015-08-26 14:24 ` Vijay Kilari 0 siblings, 0 replies; 9+ messages in thread From: Vijay Kilari @ 2015-08-26 14:24 UTC (permalink / raw) To: Pavel Fedin; +Cc: kvmarm On Tue, Aug 25, 2015 at 4:57 PM, Pavel Fedin <p.fedin@samsung.com> wrote: > Hello! Where are you? I'd like to take a look at your code. Sorry, I am occupied with other tasks for next 2 weeks, After that I will look into this. > > Kind regards, > Pavel Fedin > Expert Engineer > Samsung Electronics Research center Russia > > >> -----Original Message----- >> From: Vijay Kilari [mailto:vijay.kilari@gmail.com] >> Sent: Tuesday, August 11, 2015 6:16 PM >> To: Pavel Fedin >> Cc: kvmarm@lists.cs.columbia.edu; Christoffer Dall; Alex Bennée >> Subject: Re: KVM Live migration with GICv3 >> >> On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote: >> > Hello! >> > >> >> I have prototyped Live migration with GICv3. >> > >> > This is interesting of course, i will need this for my project too and i was also going to take a look at >> this later. >> > >> >> For this I have made following changes >> >> >> >> 1) Save and Restore of GICv3 registers in QEMU. >> >> - For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3, >> >> we have to save/restore GICD, GICR and ICC registers. >> >> However ICC registers are system registers which cannot be >> >> accessed @ EL0 level (SRE=1). So these ICC registers should be >> >> accessed as mmio registers by QEMU, for this we have to add ioctl to >> >> access ICC @ EL1 level similar to GICC registers of GICv2. >> > >> > Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why? >> > The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used >> to access anything inside the VM, including system registers. I believe you would need only to >> implement handlers for GICv3 registers. >> >> I mean CPU interface registers GICC* defined for vgicv2 under >> "vgic_cpu_ranges[]" in >> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well. >> So to access vgic_cpu_ranges[] for vgicv3, we need to register device >> for CPU registers space >> similar to gicv2 in QEMU for gicv3. >> >> ----------------------------------------------------------------------------------------- >> memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s), >> "kvm-gic_cpu", 0x1000); >> sysbus_init_mmio(sbd, &s->cpuiomem[0]); >> kvm_arm_register_device(&s->cpuiomem[0], >> (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT) >> | KVM_VGIC_V3_ADDR_TYPE_CPU, >> KVM_DEV_ARM_VGIC_GRP_ADDR, >> KVM_VGIC_V3_ADDR_TYPE_CPU, >> s->dev_fd); >> ------------------------------------------------------------------------------------------- >> >> Anyway, I will send RFC patches. >> >> > >> >> Please provide your initial feedback. >> > >> > Would be nice to take a look at patches. >> > >> >> I am attending KVM-forum next week @ Seattle. We can discuss there as well. >> > >> > I would love to go there, and my company wanted to send me there. But it appears to be extremely >> difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is >> running out, so i'm unlikely to attend, unfortunately. >> > >> > Kind regards, >> > Pavel Fedin >> > Expert Engineer >> > Samsung Electronics Research center Russia >> > >> > > _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: KVM Live migration with GICv3 2015-08-11 5:25 KVM Live migration with GICv3 Vijay Kilari 2015-08-11 6:41 ` Pavel Fedin @ 2015-08-12 16:34 ` Marc Zyngier 1 sibling, 0 replies; 9+ messages in thread From: Marc Zyngier @ 2015-08-12 16:34 UTC (permalink / raw) To: Vijay Kilari, kvmarm, Christoffer Dall, Alex Bennée, p.fedin On 11/08/15 06:25, Vijay Kilari wrote: > Hi, > > I have prototyped Live migration with GICv3. > For this I have made following changes > > 1) Save and Restore of GICv3 registers in QEMU. > - For GICv2, QEMU is saving/restoring GICD, GICC registers. For GICv3, > we have to save/restore GICD, GICR and ICC registers. > However ICC registers are system registers which cannot be > accessed @ EL0 level (SRE=1). So these ICC registers should be > accessed as mmio registers by QEMU, for this we have to add ioctl to > access ICC @ EL1 level similar to GICC registers of GICv2. I've already replied to this. The ICC_* registers *must* be exposed as system registers. There is no GICC_* state to save. And I don't get your EL0 access thing. > > 2) KVM ioctls in kernel provides only 32-bit register access to GIC > registers, where > as some registers in GICD/GICR requires 64-bit register access. I > propose to use mmio.flag to specify 32/64 bit access. I don't believe the MMIO registers *require* 64bit access. The spec says: "For the GITS_*, GICD_* and GICR_* registers, the upper 32 bits and the lower 32 bits can be accessed independently, unless the register requires a 64 bit access." and as far as I can see, no register mandates a 64bit access. > > 3) KVM ioctls to access ICC registers for GICv3 > > Please provide your initial feedback. Let me know if some of this > issues are already fixed > > I am attending KVM-forum next week @ Seattle. We can discuss there as well. Feel free to come and talk to us. M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-08-26 14:24 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-08-11 5:25 KVM Live migration with GICv3 Vijay Kilari 2015-08-11 6:41 ` Pavel Fedin 2015-08-11 15:15 ` Vijay Kilari 2015-08-11 15:24 ` Pavel Fedin 2015-08-12 16:17 ` Marc Zyngier 2015-08-12 16:33 ` Peter Maydell 2015-08-25 11:27 ` Pavel Fedin 2015-08-26 14:24 ` Vijay Kilari 2015-08-12 16:34 ` Marc Zyngier
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.