Live migration regarding Intel PT

Live migration regarding Intel PT

[Xiaoyao Li]

Hi Eduardo,

I have some question regrading Intel PT live migration.

Commit "e37a5c7fa459 (i386: Add Intel Processor Trace feature support)" 
expose Intel PT with a fixed capabilities of CPUID 0x14 for live 
migration. And the fixed capabilities are the value reported on 
ICX(IceLake). However, the upcoming SPR(Sapphire Rapids) has less 
capabilities of INTEL_PT_CYCLE_BITMAP than ICX. It fails to enable PT in 
guest on SPR machine.

If change the check on INTEL_PT_CYCLE_BITMAP to allow different value to 
allow it work on SPR. I think it breaks live migration, right?

For me, not making each sub-function of PT as configurable in QEMU 
indeed makes it hard for live migration. Why not make PT as unmigratable 
in the first place when introducing the support in QEMU?

Thanks,
-Xiaoyao

Re: Live migration regarding Intel PT

[Eduardo Habkost]

On Wed, Aug 25, 2021 at 02:59:37PM +0800, Xiaoyao Li wrote:
> Hi Eduardo,
> 
> I have some question regrading Intel PT live migration.
> 
> Commit "e37a5c7fa459 (i386: Add Intel Processor Trace feature support)"
> expose Intel PT with a fixed capabilities of CPUID 0x14 for live migration.
> And the fixed capabilities are the value reported on ICX(IceLake). However,
> the upcoming SPR(Sapphire Rapids) has less capabilities of
> INTEL_PT_CYCLE_BITMAP than ICX. It fails to enable PT in guest on SPR
> machine.
> 
> If change the check on INTEL_PT_CYCLE_BITMAP to allow different value to
> allow it work on SPR. I think it breaks live migration, right?

If I understand your description correctly, I don't think that
would break live migration.

Generating different CPUID data for the same CPU model+flags
would break live migration.

However, merely allowing a wider set of configurations to work
wouldn't break live migration, as long as the same CPU
model+flags generates the same CPUID data on any host.


> 
> For me, not making each sub-function of PT as configurable in QEMU indeed
> makes it hard for live migration. [...]

Making all sub-functions of PT configurable would be welcome.
actually.  That would allow us to support a wider range of
configurations and keep live migration working at the same time.


> [...] Why not make PT as unmigratable in the
> first place when introducing the support in QEMU?

I don't know, this was not my decision.  Now we support live
migration in a few specific cases (ICX hosts), and I assume we
don't want to stop supporting it.

If you just want to support a larger set of hosts when live
migration is not needed, you can add support to that use case
too.  e.g.: "-cpu host,migratable=off" and/or
"-cpu ...,intel-pt-passthrough=on" could do host passthrough of
intel-pt sub-leaves (and block live migration).

-- 
Eduardo

Re: Live migration regarding Intel PT

[Xiaoyao Li]

On 8/26/2021 4:48 AM, Eduardo Habkost wrote:
> On Wed, Aug 25, 2021 at 02:59:37PM +0800, Xiaoyao Li wrote:
>> Hi Eduardo,
>>
>> I have some question regrading Intel PT live migration.
>>
>> Commit "e37a5c7fa459 (i386: Add Intel Processor Trace feature support)"
>> expose Intel PT with a fixed capabilities of CPUID 0x14 for live migration.
>> And the fixed capabilities are the value reported on ICX(IceLake). However,
>> the upcoming SPR(Sapphire Rapids) has less capabilities of
>> INTEL_PT_CYCLE_BITMAP than ICX. It fails to enable PT in guest on SPR
>> machine.
>>
>> If change the check on INTEL_PT_CYCLE_BITMAP to allow different value to
>> allow it work on SPR. I think it breaks live migration, right?
> 
> If I understand your description correctly, I don't think that
> would break live migration.
> 
> Generating different CPUID data for the same CPU model+flags
> would break live migration.
> 
> However, merely allowing a wider set of configurations to work
> wouldn't break live migration, as long as the same CPU
> model+flags generates the same CPUID data on any host.

The easy fix in my brain to make PT work on SPR guest surely will lead 
to different CPUID data between ICX and SPR. That's why I wrote the email.

Other questions about live migration. I think a named CPU model is the 
pre-condition for live migration, right?

Then is "same QEMU version/binary" the pre-condition for live migration 
as well?

> 
>>
>> For me, not making each sub-function of PT as configurable in QEMU indeed
>> makes it hard for live migration. [...]
> 
> Making all sub-functions of PT configurable would be welcome.
> actually.  That would allow us to support a wider range of
> configurations and keep live migration working at the same time.

I think it will break old QEMU? Is it OK?

> 
>> [...] Why not make PT as unmigratable in the
>> first place when introducing the support in QEMU?
> 
> I don't know, this was not my decision.  Now we support live
> migration in a few specific cases (ICX hosts), and I assume we
> don't want to stop supporting it.
> 
> If you just want to support a larger set of hosts when live
> migration is not needed, you can add support to that use case
> too.  e.g.: "-cpu host,migratable=off" and/or
> "-cpu ...,intel-pt-passthrough=on" could do host passthrough of
> intel-pt sub-leaves (and block live migration).
> 

That will make things complicated that old use case "-cpu ...,+intel-pt" 
still means supporting live migration. And when use "-cpu 
...,+intel-pt", QEMU needs to generate fixed PT capability as previous?

Re: Live migration regarding Intel PT

[Eduardo Habkost]

On Thu, Aug 26, 2021 at 10:16:26AM +0800, Xiaoyao Li wrote:
> On 8/26/2021 4:48 AM, Eduardo Habkost wrote:
> > On Wed, Aug 25, 2021 at 02:59:37PM +0800, Xiaoyao Li wrote:
> > > Hi Eduardo,
> > > 
> > > I have some question regrading Intel PT live migration.
> > > 
> > > Commit "e37a5c7fa459 (i386: Add Intel Processor Trace feature support)"
> > > expose Intel PT with a fixed capabilities of CPUID 0x14 for live migration.
> > > And the fixed capabilities are the value reported on ICX(IceLake). However,
> > > the upcoming SPR(Sapphire Rapids) has less capabilities of
> > > INTEL_PT_CYCLE_BITMAP than ICX. It fails to enable PT in guest on SPR
> > > machine.
> > > 
> > > If change the check on INTEL_PT_CYCLE_BITMAP to allow different value to
> > > allow it work on SPR. I think it breaks live migration, right?
> > 
> > If I understand your description correctly, I don't think that
> > would break live migration.
> > 
> > Generating different CPUID data for the same CPU model+flags
> > would break live migration.
> > 
> > However, merely allowing a wider set of configurations to work
> > wouldn't break live migration, as long as the same CPU
> > model+flags generates the same CPUID data on any host.
> 
> The easy fix in my brain to make PT work on SPR guest surely will lead to
> different CPUID data between ICX and SPR. That's why I wrote the email.
> 

Yes, but I don't see where the problem is.  We only need to
generate the same CPUID data if you are running the same CPU
model + flags.  It's OK (and expected) to have "-cpu Icelake" and
"-cpu SapphireRapids" result in different CPUID data.


> Other questions about live migration. I think a named CPU model is the
> pre-condition for live migration, right?

Yes.  More precisely, it needs a migration-safe CPU model (in x86
it means all named CPU models except "max" and "host").

> 
> Then is "same QEMU version/binary" the pre-condition for live migration as
> well?

Not at all.  We support migration to different QEMU
binaries/versions.  But "same machine type" and "same -cpu
option" are both preconditions.

> 
> > 
> > > 
> > > For me, not making each sub-function of PT as configurable in QEMU indeed
> > > makes it hard for live migration. [...]
> > 
> > Making all sub-functions of PT configurable would be welcome.
> > actually.  That would allow us to support a wider range of
> > configurations and keep live migration working at the same time.
> 
> I think it will break old QEMU? Is it OK?

If it's a new feature that requires a new command-line option, it
is completely OK.

> 
> > 
> > > [...] Why not make PT as unmigratable in the
> > > first place when introducing the support in QEMU?
> > 
> > I don't know, this was not my decision.  Now we support live
> > migration in a few specific cases (ICX hosts), and I assume we
> > don't want to stop supporting it.
> > 
> > If you just want to support a larger set of hosts when live
> > migration is not needed, you can add support to that use case
> > too.  e.g.: "-cpu host,migratable=off" and/or
> > "-cpu ...,intel-pt-passthrough=on" could do host passthrough of
> > intel-pt sub-leaves (and block live migration).
> > 
> 
> That will make things complicated that old use case "-cpu ...,+intel-pt"
> still means supporting live migration. And when use "-cpu ...,+intel-pt",
> QEMU needs to generate fixed PT capability as previous?

Yes, you need to keep the existing use cases working unless you
deprecate the existing migration-safe use case (which I don't
think you want to).  But a "if (cpu->intel_pt_passthrough)" check
in cpu_x86_cpuid() would solve that.

Anyway, you only need to do that if you want to (if you believe
that's an important and useful use case).

-- 
Eduardo