Why invtsc (CPUID_APM_INVTSC) is unmigratable?

Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Xiaoyao Li]

I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could 
anyone explain it?

When the host supports invtsc, it can be exposed to guest.
When the src VM has invtsc exposed, what will forbid it to be migrated 
to a dest that also supports VMs with invtsc exposed?

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Marcelo Tosatti]

On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
> I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
> anyone explain it?


commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
Author: Marcelo Tosatti <mtosatti@redhat.com>
Date:   Wed May 14 16:30:09 2014 -0300

    target-i386: block migration and savevm if invariant tsc is exposed
    
    Invariant TSC documentation mentions that "invariant TSC will run at a
    constant rate in all ACPI P-, C-. and T-states".
    
    This is not the case if migration to a host with different TSC frequency
    is allowed, or if savevm is performed. So block migration/savevm.

So the rationale here was that without ensuring the destination host 
has the same TSC clock frequency, we can't migrate.

However, this was later extended to allow invtsc migratioon when setting
tsc-khz explicitly:

commit d99569d9d8562c480e0befab601756b0b7b5d0e0
Author: Eduardo Habkost <ehabkost@redhat.com>
Date:   Sun Jan 8 15:32:34 2017 -0200

    kvm: Allow invtsc migration if tsc-khz is set explicitly
    
    We can safely allow a VM to be migrated with invtsc enabled if
    tsc-khz is set explicitly, because:
    * QEMU already refuses to start if it can't set the TSC frequency
      to the configured value.
    * Management software is already required to keep device
      configuration (including CPU configuration) the same on
      migration source and destination.
    
    Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
    Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
    Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

And support for libvirt was added:

https://listman.redhat.com/archives/libvir-list/2017-January/141757.html

> 
> When the host supports invtsc, it can be exposed to guest.
> When the src VM has invtsc exposed, what will forbid it to be migrated to a
> dest that also supports VMs with invtsc exposed?
> 
> 

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Xiaoyao Li]

On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
> On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
>> I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
>> anyone explain it?
> 
> 
> commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> Author: Marcelo Tosatti <mtosatti@redhat.com>
> Date:   Wed May 14 16:30:09 2014 -0300
> 
>      target-i386: block migration and savevm if invariant tsc is exposed
>      
>      Invariant TSC documentation mentions that "invariant TSC will run at a
>      constant rate in all ACPI P-, C-. and T-states".
>      
>      This is not the case if migration to a host with different TSC frequency
>      is allowed, or if savevm is performed. So block migration/savevm.
> 
> So the rationale here was that without ensuring the destination host
> has the same TSC clock frequency, we can't migrate.

It seems to me the concept of invtsc was extended to "tsc freq will not 
change even after the machine is live migrated". I'm not sure it is 
correct to extend the concept of invtsc.

The main reason of introducing invtsc is to tell the tsc hardware keeps 
counting (at the same rate) even at deep C state, so long as other states.

For example, a guest is created on machine A with X GHz tsc, and invtsc 
exposed (machine A can ensure the guest's tsc counts at X GHz at any 
state). If the guest is migrated to machine B with Y GHz tsc, and 
machine B can also ensure the invtsc of its guest, i.e., the guest's tsc 
counts at Y GHz at any state. IMHO, in this case, the invtsc is 
supported at both src and dest, which means it is a migratable feature. 
However, the migration itself fails, due to mismatched/different 
configuration of tsc freq, not due to invtsc.

> However, this was later extended to allow invtsc migratioon when setting
> tsc-khz explicitly:
> 
> commit d99569d9d8562c480e0befab601756b0b7b5d0e0
> Author: Eduardo Habkost <ehabkost@redhat.com>
> Date:   Sun Jan 8 15:32:34 2017 -0200
> 
>      kvm: Allow invtsc migration if tsc-khz is set explicitly
>      
>      We can safely allow a VM to be migrated with invtsc enabled if
>      tsc-khz is set explicitly, because:
>      * QEMU already refuses to start if it can't set the TSC frequency
>        to the configured value.
>      * Management software is already required to keep device
>        configuration (including CPU configuration) the same on
>        migration source and destination.
>      
>      Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>      Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
>      Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

But in the case that user doesn't set tsc freq explicitly, the live 
migration is likely to fail or have issues even without invtsc exposed 
to guest, if the destination host has a different tsc frequency than src 
host.

So why bother checking invtsc only?

> And support for libvirt was added:
> 
> https://listman.redhat.com/archives/libvir-list/2017-January/141757.html
> 
>>
>> When the host supports invtsc, it can be exposed to guest.
>> When the src VM has invtsc exposed, what will forbid it to be migrated to a
>> dest that also supports VMs with invtsc exposed?
>>
>>
> 

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Marcelo Tosatti]

On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
> On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
> > On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
> > > I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
> > > anyone explain it?
> > 
> > 
> > commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> > Author: Marcelo Tosatti <mtosatti@redhat.com>
> > Date:   Wed May 14 16:30:09 2014 -0300
> > 
> >      target-i386: block migration and savevm if invariant tsc is exposed
> >      Invariant TSC documentation mentions that "invariant TSC will run at a
> >      constant rate in all ACPI P-, C-. and T-states".
> >      This is not the case if migration to a host with different TSC frequency
> >      is allowed, or if savevm is performed. So block migration/savevm.
> > 
> > So the rationale here was that without ensuring the destination host
> > has the same TSC clock frequency, we can't migrate.
> 
> It seems to me the concept of invtsc was extended to "tsc freq will not
> change even after the machine is live migrated". I'm not sure it is correct
> to extend the concept of invtsc.
> 
> The main reason of introducing invtsc is to tell the tsc hardware keeps
> counting (at the same rate) even at deep C state, so long as other states.
> 
> For example, a guest is created on machine A with X GHz tsc, and invtsc
> exposed (machine A can ensure the guest's tsc counts at X GHz at any state).
> If the guest is migrated to machine B with Y GHz tsc, and machine B can also
> ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz at any
> state. IMHO, in this case, the invtsc is supported at both src and dest,
> which means it is a migratable feature. However, the migration itself fails,
> due to mismatched/different configuration of tsc freq, not due to invtsc.
> 
> > However, this was later extended to allow invtsc migratioon when setting
> > tsc-khz explicitly:
> > 
> > commit d99569d9d8562c480e0befab601756b0b7b5d0e0
> > Author: Eduardo Habkost <ehabkost@redhat.com>
> > Date:   Sun Jan 8 15:32:34 2017 -0200
> > 
> >      kvm: Allow invtsc migration if tsc-khz is set explicitly
> >      We can safely allow a VM to be migrated with invtsc enabled if
> >      tsc-khz is set explicitly, because:
> >      * QEMU already refuses to start if it can't set the TSC frequency
> >        to the configured value.
> >      * Management software is already required to keep device
> >        configuration (including CPU configuration) the same on
> >        migration source and destination.
> >      Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> >      Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
> >      Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> But in the case that user doesn't set tsc freq explicitly, the live
> migration is likely to fail or have issues even without invtsc exposed to
> guest, 

Depends on how the guest is using the TSC, but yes.

> if the destination host has a different tsc frequency than src host.
> 
> So why bother checking invtsc only?

Well, if invtsc is exposed to the guest, then it might use the TSC for
timekeeping purposes. 

Therefore you don't want to fail (on the invtsc clock characteristics)
otherwise timekeeping in the guest might be problematic.

But this are all just heuristics. 

Do you have a suggestion for different behaviour?

> 
> > And support for libvirt was added:
> > 
> > https://listman.redhat.com/archives/libvir-list/2017-January/141757.html
> > 
> > > 
> > > When the host supports invtsc, it can be exposed to guest.
> > > When the src VM has invtsc exposed, what will forbid it to be migrated to a
> > > dest that also supports VMs with invtsc exposed?
> > > 
> > > 
> > 
> 
> 

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Xiaoyao Li]

On 1/23/2024 11:39 PM, Marcelo Tosatti wrote:
> On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
>> On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
>>> On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
>>>> I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
>>>> anyone explain it?
>>>
>>>
>>> commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>>> Date:   Wed May 14 16:30:09 2014 -0300
>>>
>>>       target-i386: block migration and savevm if invariant tsc is exposed
>>>       Invariant TSC documentation mentions that "invariant TSC will run at a
>>>       constant rate in all ACPI P-, C-. and T-states".
>>>       This is not the case if migration to a host with different TSC frequency
>>>       is allowed, or if savevm is performed. So block migration/savevm.
>>>
>>> So the rationale here was that without ensuring the destination host
>>> has the same TSC clock frequency, we can't migrate.
>>
>> It seems to me the concept of invtsc was extended to "tsc freq will not
>> change even after the machine is live migrated". I'm not sure it is correct
>> to extend the concept of invtsc.
>>
>> The main reason of introducing invtsc is to tell the tsc hardware keeps
>> counting (at the same rate) even at deep C state, so long as other states.
>>
>> For example, a guest is created on machine A with X GHz tsc, and invtsc
>> exposed (machine A can ensure the guest's tsc counts at X GHz at any state).
>> If the guest is migrated to machine B with Y GHz tsc, and machine B can also
>> ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz at any
>> state. IMHO, in this case, the invtsc is supported at both src and dest,
>> which means it is a migratable feature. However, the migration itself fails,
>> due to mismatched/different configuration of tsc freq, not due to invtsc.
>>
>>> However, this was later extended to allow invtsc migratioon when setting
>>> tsc-khz explicitly:
>>>
>>> commit d99569d9d8562c480e0befab601756b0b7b5d0e0
>>> Author: Eduardo Habkost <ehabkost@redhat.com>
>>> Date:   Sun Jan 8 15:32:34 2017 -0200
>>>
>>>       kvm: Allow invtsc migration if tsc-khz is set explicitly
>>>       We can safely allow a VM to be migrated with invtsc enabled if
>>>       tsc-khz is set explicitly, because:
>>>       * QEMU already refuses to start if it can't set the TSC frequency
>>>         to the configured value.
>>>       * Management software is already required to keep device
>>>         configuration (including CPU configuration) the same on
>>>         migration source and destination.
>>>       Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>       Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
>>>       Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>
>> But in the case that user doesn't set tsc freq explicitly, the live
>> migration is likely to fail or have issues even without invtsc exposed to
>> guest,
> 
> Depends on how the guest is using the TSC, but yes.
> 
>> if the destination host has a different tsc frequency than src host.
>>
>> So why bother checking invtsc only?
> 
> Well, if invtsc is exposed to the guest, then it might use the TSC for
> timekeeping purposes.
> 
> Therefore you don't want to fail (on the invtsc clock characteristics)
> otherwise timekeeping in the guest might be problematic.
> 
> But this are all just heuristics.
> 
> Do you have a suggestion for different behaviour?

I think we need to block live migration when user doesn't specify a 
certain tsc frequency explicitly, regardless of invtsc.

>>
>>> And support for libvirt was added:
>>>
>>> https://listman.redhat.com/archives/libvir-list/2017-January/141757.html
>>>
>>>>
>>>> When the host supports invtsc, it can be exposed to guest.
>>>> When the src VM has invtsc exposed, what will forbid it to be migrated to a
>>>> dest that also supports VMs with invtsc exposed?
>>>>
>>>>
>>>
>>
>>
> 

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Marcelo Tosatti]

On Wed, Jan 24, 2024 at 10:52:46PM +0800, Xiaoyao Li wrote:
> On 1/23/2024 11:39 PM, Marcelo Tosatti wrote:
> > On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
> > > On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
> > > > On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
> > > > > I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
> > > > > anyone explain it?
> > > > 
> > > > 
> > > > commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> > > > Author: Marcelo Tosatti <mtosatti@redhat.com>
> > > > Date:   Wed May 14 16:30:09 2014 -0300
> > > > 
> > > >       target-i386: block migration and savevm if invariant tsc is exposed
> > > >       Invariant TSC documentation mentions that "invariant TSC will run at a
> > > >       constant rate in all ACPI P-, C-. and T-states".
> > > >       This is not the case if migration to a host with different TSC frequency
> > > >       is allowed, or if savevm is performed. So block migration/savevm.
> > > > 
> > > > So the rationale here was that without ensuring the destination host
> > > > has the same TSC clock frequency, we can't migrate.
> > > 
> > > It seems to me the concept of invtsc was extended to "tsc freq will not
> > > change even after the machine is live migrated". I'm not sure it is correct
> > > to extend the concept of invtsc.
> > > 
> > > The main reason of introducing invtsc is to tell the tsc hardware keeps
> > > counting (at the same rate) even at deep C state, so long as other states.
> > > 
> > > For example, a guest is created on machine A with X GHz tsc, and invtsc
> > > exposed (machine A can ensure the guest's tsc counts at X GHz at any state).
> > > If the guest is migrated to machine B with Y GHz tsc, and machine B can also
> > > ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz at any
> > > state. IMHO, in this case, the invtsc is supported at both src and dest,
> > > which means it is a migratable feature. However, the migration itself fails,
> > > due to mismatched/different configuration of tsc freq, not due to invtsc.
> > > 
> > > > However, this was later extended to allow invtsc migratioon when setting
> > > > tsc-khz explicitly:
> > > > 
> > > > commit d99569d9d8562c480e0befab601756b0b7b5d0e0
> > > > Author: Eduardo Habkost <ehabkost@redhat.com>
> > > > Date:   Sun Jan 8 15:32:34 2017 -0200
> > > > 
> > > >       kvm: Allow invtsc migration if tsc-khz is set explicitly
> > > >       We can safely allow a VM to be migrated with invtsc enabled if
> > > >       tsc-khz is set explicitly, because:
> > > >       * QEMU already refuses to start if it can't set the TSC frequency
> > > >         to the configured value.
> > > >       * Management software is already required to keep device
> > > >         configuration (including CPU configuration) the same on
> > > >         migration source and destination.
> > > >       Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > >       Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
> > > >       Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > 
> > > But in the case that user doesn't set tsc freq explicitly, the live
> > > migration is likely to fail or have issues even without invtsc exposed to
> > > guest,
> > 
> > Depends on how the guest is using the TSC, but yes.
> > 
> > > if the destination host has a different tsc frequency than src host.
> > > 
> > > So why bother checking invtsc only?
> > 
> > Well, if invtsc is exposed to the guest, then it might use the TSC for
> > timekeeping purposes.
> > 
> > Therefore you don't want to fail (on the invtsc clock characteristics)
> > otherwise timekeeping in the guest might be problematic.
> > 
> > But this are all just heuristics.
> > 
> > Do you have a suggestion for different behaviour?
> 
> I think we need to block live migration when user doesn't specify a certain
> tsc frequency explicitly, regardless of invtsc.

Problem is if that guest is using kvmclock for timekeeping, then live migration 
is safe (kvmclock logic will update the tsc frequency of the destination
host upon migration).

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Xiaoyao Li]

On 1/25/2024 6:05 AM, Marcelo Tosatti wrote:
> On Wed, Jan 24, 2024 at 10:52:46PM +0800, Xiaoyao Li wrote:
>> On 1/23/2024 11:39 PM, Marcelo Tosatti wrote:
>>> On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
>>>> On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
>>>>> On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
>>>>>> I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
>>>>>> anyone explain it?
>>>>>
>>>>>
>>>>> commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
>>>>> Author: Marcelo Tosatti <mtosatti@redhat.com>
>>>>> Date:   Wed May 14 16:30:09 2014 -0300
>>>>>
>>>>>        target-i386: block migration and savevm if invariant tsc is exposed
>>>>>        Invariant TSC documentation mentions that "invariant TSC will run at a
>>>>>        constant rate in all ACPI P-, C-. and T-states".
>>>>>        This is not the case if migration to a host with different TSC frequency
>>>>>        is allowed, or if savevm is performed. So block migration/savevm.
>>>>>
>>>>> So the rationale here was that without ensuring the destination host
>>>>> has the same TSC clock frequency, we can't migrate.
>>>>
>>>> It seems to me the concept of invtsc was extended to "tsc freq will not
>>>> change even after the machine is live migrated". I'm not sure it is correct
>>>> to extend the concept of invtsc.
>>>>
>>>> The main reason of introducing invtsc is to tell the tsc hardware keeps
>>>> counting (at the same rate) even at deep C state, so long as other states.
>>>>
>>>> For example, a guest is created on machine A with X GHz tsc, and invtsc
>>>> exposed (machine A can ensure the guest's tsc counts at X GHz at any state).
>>>> If the guest is migrated to machine B with Y GHz tsc, and machine B can also
>>>> ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz at any
>>>> state. IMHO, in this case, the invtsc is supported at both src and dest,
>>>> which means it is a migratable feature. However, the migration itself fails,
>>>> due to mismatched/different configuration of tsc freq, not due to invtsc.
>>>>
>>>>> However, this was later extended to allow invtsc migratioon when setting
>>>>> tsc-khz explicitly:
>>>>>
>>>>> commit d99569d9d8562c480e0befab601756b0b7b5d0e0
>>>>> Author: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Date:   Sun Jan 8 15:32:34 2017 -0200
>>>>>
>>>>>        kvm: Allow invtsc migration if tsc-khz is set explicitly
>>>>>        We can safely allow a VM to be migrated with invtsc enabled if
>>>>>        tsc-khz is set explicitly, because:
>>>>>        * QEMU already refuses to start if it can't set the TSC frequency
>>>>>          to the configured value.
>>>>>        * Management software is already required to keep device
>>>>>          configuration (including CPU configuration) the same on
>>>>>          migration source and destination.
>>>>>        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>>        Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
>>>>>        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>
>>>> But in the case that user doesn't set tsc freq explicitly, the live
>>>> migration is likely to fail or have issues even without invtsc exposed to
>>>> guest,
>>>
>>> Depends on how the guest is using the TSC, but yes.
>>>
>>>> if the destination host has a different tsc frequency than src host.
>>>>
>>>> So why bother checking invtsc only?
>>>
>>> Well, if invtsc is exposed to the guest, then it might use the TSC for
>>> timekeeping purposes.
>>>
>>> Therefore you don't want to fail (on the invtsc clock characteristics)
>>> otherwise timekeeping in the guest might be problematic.
>>>
>>> But this are all just heuristics.
>>>
>>> Do you have a suggestion for different behaviour?
>>
>> I think we need to block live migration when user doesn't specify a certain
>> tsc frequency explicitly, regardless of invtsc.
> 
> Problem is if that guest is using kvmclock for timekeeping, then live migration
> is safe (kvmclock logic will update the tsc frequency of the destination
> host upon migration).
> 

It surprises me kvmclock can do it. Cloud you please elaborate it a bit 
how kvmclock achieve it during migration or point me to some codes in 
Linux?

Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Marcelo Tosatti]

On Fri, Jan 26, 2024 at 04:16:57PM +0800, Xiaoyao Li wrote:
> On 1/25/2024 6:05 AM, Marcelo Tosatti wrote:
> > On Wed, Jan 24, 2024 at 10:52:46PM +0800, Xiaoyao Li wrote:
> > > On 1/23/2024 11:39 PM, Marcelo Tosatti wrote:
> > > > On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
> > > > > On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
> > > > > > On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
> > > > > > > I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. Could
> > > > > > > anyone explain it?
> > > > > > 
> > > > > > 
> > > > > > commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> > > > > > Author: Marcelo Tosatti <mtosatti@redhat.com>
> > > > > > Date:   Wed May 14 16:30:09 2014 -0300
> > > > > > 
> > > > > >        target-i386: block migration and savevm if invariant tsc is exposed
> > > > > >        Invariant TSC documentation mentions that "invariant TSC will run at a
> > > > > >        constant rate in all ACPI P-, C-. and T-states".
> > > > > >        This is not the case if migration to a host with different TSC frequency
> > > > > >        is allowed, or if savevm is performed. So block migration/savevm.
> > > > > > 
> > > > > > So the rationale here was that without ensuring the destination host
> > > > > > has the same TSC clock frequency, we can't migrate.
> > > > > 
> > > > > It seems to me the concept of invtsc was extended to "tsc freq will not
> > > > > change even after the machine is live migrated". I'm not sure it is correct
> > > > > to extend the concept of invtsc.
> > > > > 
> > > > > The main reason of introducing invtsc is to tell the tsc hardware keeps
> > > > > counting (at the same rate) even at deep C state, so long as other states.
> > > > > 
> > > > > For example, a guest is created on machine A with X GHz tsc, and invtsc
> > > > > exposed (machine A can ensure the guest's tsc counts at X GHz at any state).
> > > > > If the guest is migrated to machine B with Y GHz tsc, and machine B can also
> > > > > ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz at any
> > > > > state. IMHO, in this case, the invtsc is supported at both src and dest,
> > > > > which means it is a migratable feature. However, the migration itself fails,
> > > > > due to mismatched/different configuration of tsc freq, not due to invtsc.
> > > > > 
> > > > > > However, this was later extended to allow invtsc migratioon when setting
> > > > > > tsc-khz explicitly:
> > > > > > 
> > > > > > commit d99569d9d8562c480e0befab601756b0b7b5d0e0
> > > > > > Author: Eduardo Habkost <ehabkost@redhat.com>
> > > > > > Date:   Sun Jan 8 15:32:34 2017 -0200
> > > > > > 
> > > > > >        kvm: Allow invtsc migration if tsc-khz is set explicitly
> > > > > >        We can safely allow a VM to be migrated with invtsc enabled if
> > > > > >        tsc-khz is set explicitly, because:
> > > > > >        * QEMU already refuses to start if it can't set the TSC frequency
> > > > > >          to the configured value.
> > > > > >        * Management software is already required to keep device
> > > > > >          configuration (including CPU configuration) the same on
> > > > > >          migration source and destination.
> > > > > >        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > > > >        Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
> > > > > >        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > > > 
> > > > > But in the case that user doesn't set tsc freq explicitly, the live
> > > > > migration is likely to fail or have issues even without invtsc exposed to
> > > > > guest,
> > > > 
> > > > Depends on how the guest is using the TSC, but yes.
> > > > 
> > > > > if the destination host has a different tsc frequency than src host.
> > > > > 
> > > > > So why bother checking invtsc only?
> > > > 
> > > > Well, if invtsc is exposed to the guest, then it might use the TSC for
> > > > timekeeping purposes.
> > > > 
> > > > Therefore you don't want to fail (on the invtsc clock characteristics)
> > > > otherwise timekeeping in the guest might be problematic.
> > > > 
> > > > But this are all just heuristics.
> > > > 
> > > > Do you have a suggestion for different behaviour?
> > > 
> > > I think we need to block live migration when user doesn't specify a certain
> > > tsc frequency explicitly, regardless of invtsc.
> > 
> > Problem is if that guest is using kvmclock for timekeeping, then live migration
> > is safe (kvmclock logic will update the tsc frequency of the destination
> > host upon migration).
> > 
> 
> It surprises me kvmclock can do it. Cloud you please elaborate it a bit how
> kvmclock achieve it during migration or point me to some codes in Linux?

MSR_KVM_SYSTEM_TIME_NEW description at
Documentation/virt/kvm/x86/msr.rst and

arch/x86/kernel/pvclock.c, __pvclock_clocksource_read function.