Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Bagas Sanjaya]

Hi,

I notice a regression report on Bugzilla [1]. Quoting from it:

> Hello,
> 
> I have a virtual machine running the old Windows Server 2003. On kernels 6.1.44 and 6.1.45, the QEMU VNC window stays dark, not switching to any of the guest's video modes and the VM process uses only ~64 MB of RAM of the assigned 2 GB, indefinitely. It's like the VM is paused/halted/stuck before even starting. The process can be killed successfully and then restarted again (with the same result), so it is not deadlocked in kernel or the like.
> 
> Kernel 6.1.43 works fine.
> 
> I have also tried downgrading CPU microcode from 20230808 to 20230719, but that did not help.
> 
> The CPU is AMD Ryzen 5900. I suspect some of the newly added mitigations may be the culprit?

See Bugzilla for the full thread.

Anyway, I'm adding it to regzbot as stable-specific regression:

#regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
#regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)

Thanks.

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=217799
-- 
An old man doll... just what I always wanted! - Clara

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Vitaly Kuznetsov]

Bagas Sanjaya <bagasdotme@gmail.com> writes:

> Hi,
>
> I notice a regression report on Bugzilla [1]. Quoting from it:
>
>> Hello,
>> 
>> I have a virtual machine running the old Windows Server 2003. On kernels 6.1.44 and 6.1.45, the QEMU VNC window stays dark, not switching to any of the guest's video modes and the VM process uses only ~64 MB of RAM of the assigned 2 GB, indefinitely. It's like the VM is paused/halted/stuck before even starting. The process can be killed successfully and then restarted again (with the same result), so it is not deadlocked in kernel or the like.
>> 
>> Kernel 6.1.43 works fine.
>> 
>> I have also tried downgrading CPU microcode from 20230808 to 20230719, but that did not help.
>> 
>> The CPU is AMD Ryzen 5900. I suspect some of the newly added mitigations may be the culprit?
>
> See Bugzilla for the full thread.
>
> Anyway, I'm adding it to regzbot as stable-specific regression:
>
> #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
> #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)
>
> Thanks.
>
> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217799

From KVM's PoV, I don't see any KVM/x86 patches v6.1.44..v6.1.45 and in fact
there are only two x86 patches:

f2615bb47be4 x86/CPU/AMD: Do not leak quotient data after a division by 0
98cccbd0a19a x86/hyperv: Disable IBT when hypercall page lacks ENDBR instruction

and I'm pretty certain the later is unrelated; f2615bb47be4 looks like
it can, in theory, be related. Cc: Borislav.

-- 
Vitaly

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Vitaly Kuznetsov]

Vitaly Kuznetsov <vkuznets@redhat.com> writes:

> Bagas Sanjaya <bagasdotme@gmail.com> writes:
>
>> Hi,
>>
>> I notice a regression report on Bugzilla [1]. Quoting from it:
>>
>>> Hello,
>>> 
>>> I have a virtual machine running the old Windows Server 2003. On kernels 6.1.44 and 6.1.45, the QEMU VNC window stays dark, not switching to any of the guest's video modes and the VM process uses only ~64 MB of RAM of the assigned 2 GB, indefinitely. It's like the VM is paused/halted/stuck before even starting. The process can be killed successfully and then restarted again (with the same result), so it is not deadlocked in kernel or the like.
>>> 
>>> Kernel 6.1.43 works fine.
>>> 
>>> I have also tried downgrading CPU microcode from 20230808 to 20230719, but that did not help.
>>> 
>>> The CPU is AMD Ryzen 5900. I suspect some of the newly added mitigations may be the culprit?
>>
>> See Bugzilla for the full thread.
>>
>> Anyway, I'm adding it to regzbot as stable-specific regression:
>>
>> #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
>> #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)
>>
>> Thanks.
>>
>> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217799
>
> From KVM's PoV, I don't see any KVM/x86 patches v6.1.44..v6.1.45 

Oh, sorry, my bad, in the description of the BZ it is said that 6.1.44
is already broken, so it's most likely srso stuff then:

dd5f2ef16e3c x86: fix backwards merge of GDS/SRSO bit
4f25355540ad x86/srso: Tie SBPB bit setting to microcode patch detection
77cf32d0dbfb x86/srso: Add a forgotten NOENDBR annotation
c7f2cd045542 x86/srso: Fix return thunks in generated code
c9ae63d773ca x86/srso: Add IBPB on VMEXIT
79c8091888ef x86/srso: Add IBPB
98f62883e751 x86/srso: Add SRSO_NO support
9139f4b6dd4f x86/srso: Add IBPB_BRTYPE support
ac41e90d8daa x86/srso: Add a Speculative RAS Overflow mitigation
dec3b91f2c4b x86/cpu, kvm: Add support for CPUID_80000021_EAX

it would still be great to try to bisect to the particular patch causing
the issue.

-- 
Vitaly

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Vitaly Kuznetsov]

Vitaly Kuznetsov <vkuznets@redhat.com> writes:

> Vitaly Kuznetsov <vkuznets@redhat.com> writes:
>
>> Bagas Sanjaya <bagasdotme@gmail.com> writes:
>>
>>> Hi,
>>>
>>> I notice a regression report on Bugzilla [1]. Quoting from it:
>>>
>>>> Hello,
>>>> 
>>>> I have a virtual machine running the old Windows Server 2003. On kernels 6.1.44 and 6.1.45, the QEMU VNC window stays dark, not switching to any of the guest's video modes and the VM process uses only ~64 MB of RAM of the assigned 2 GB, indefinitely. It's like the VM is paused/halted/stuck before even starting. The process can be killed successfully and then restarted again (with the same result), so it is not deadlocked in kernel or the like.
>>>> 
>>>> Kernel 6.1.43 works fine.
>>>> 
>>>> I have also tried downgrading CPU microcode from 20230808 to 20230719, but that did not help.
>>>> 
>>>> The CPU is AMD Ryzen 5900. I suspect some of the newly added mitigations may be the culprit?
>>>
>>> See Bugzilla for the full thread.
>>>
>>> Anyway, I'm adding it to regzbot as stable-specific regression:
>>>
>>> #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
>>> #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)
>>>
>>> Thanks.
>>>
>>> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217799
>>
>> From KVM's PoV, I don't see any KVM/x86 patches v6.1.44..v6.1.45 
>
> Oh, sorry, my bad, in the description of the BZ it is said that 6.1.44
> is already broken, so it's most likely srso stuff then:
>
> dd5f2ef16e3c x86: fix backwards merge of GDS/SRSO bit
> 4f25355540ad x86/srso: Tie SBPB bit setting to microcode patch detection
> 77cf32d0dbfb x86/srso: Add a forgotten NOENDBR annotation
> c7f2cd045542 x86/srso: Fix return thunks in generated code
> c9ae63d773ca x86/srso: Add IBPB on VMEXIT
> 79c8091888ef x86/srso: Add IBPB
> 98f62883e751 x86/srso: Add SRSO_NO support
> 9139f4b6dd4f x86/srso: Add IBPB_BRTYPE support
> ac41e90d8daa x86/srso: Add a Speculative RAS Overflow mitigation

Sean's https://lore.kernel.org/all/20230811155255.250835-1-seanjc@google.com/
(alteady in 'tip') can actually be related and I see it was already
tagged for stable@. Can anyone check if it really helps?

> dec3b91f2c4b x86/cpu, kvm: Add support for CPUID_80000021_EAX
>
> it would still be great to try to bisect to the particular patch causing
> the issue.

-- 
Vitaly

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Maxim Levitsky]

У ср, 2023-08-16 у 15:41 +0200, Vitaly Kuznetsov пише:
> Vitaly Kuznetsov <vkuznets@redhat.com> writes:
> 
> > Vitaly Kuznetsov <vkuznets@redhat.com> writes:
> > 
> > > Bagas Sanjaya <bagasdotme@gmail.com> writes:
> > > 
> > > > Hi,
> > > > 
> > > > I notice a regression report on Bugzilla [1]. Quoting from it:
> > > > 
> > > > > Hello,
> > > > > 
> > > > > I have a virtual machine running the old Windows Server 2003. On kernels 6.1.44 and 6.1.45, the QEMU VNC window stays dark, not switching to any of the guest's video modes and the VM process uses only ~64 MB of RAM of the assigned 2 GB, indefinitely. It's like the VM is paused/halted/stuck before even starting. The process can be killed successfully and then restarted again (with the same result), so it is not deadlocked in kernel or the like.
> > > > > 
> > > > > Kernel 6.1.43 works fine.
> > > > > 
> > > > > I have also tried downgrading CPU microcode from 20230808 to 20230719, but that did not help.
> > > > > 
> > > > > The CPU is AMD Ryzen 5900. I suspect some of the newly added mitigations may be the culprit?
> > > > 
> > > > See Bugzilla for the full thread.
> > > > 
> > > > Anyway, I'm adding it to regzbot as stable-specific regression:
> > > > 
> > > > #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
> > > > #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)
> > > > 
> > > > Thanks.
> > > > 
> > > > [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217799
> > > 
> > > From KVM's PoV, I don't see any KVM/x86 patches v6.1.44..v6.1.45 
> > 
> > Oh, sorry, my bad, in the description of the BZ it is said that 6.1.44
> > is already broken, so it's most likely srso stuff then:
> > 
> > dd5f2ef16e3c x86: fix backwards merge of GDS/SRSO bit
> > 4f25355540ad x86/srso: Tie SBPB bit setting to microcode patch detection
> > 77cf32d0dbfb x86/srso: Add a forgotten NOENDBR annotation
> > c7f2cd045542 x86/srso: Fix return thunks in generated code
> > c9ae63d773ca x86/srso: Add IBPB on VMEXIT
> > 79c8091888ef x86/srso: Add IBPB
> > 98f62883e751 x86/srso: Add SRSO_NO support
> > 9139f4b6dd4f x86/srso: Add IBPB_BRTYPE support
> > ac41e90d8daa x86/srso: Add a Speculative RAS Overflow mitigation
> 
> Sean's https://lore.kernel.org/all/20230811155255.250835-1-seanjc@google.com/
> (alteady in 'tip') can actually be related and I see it was already
> tagged for stable@. Can anyone check if it really helps?
> 
> > dec3b91f2c4b x86/cpu, kvm: Add support for CPUID_80000021_EAX
> > 
> > it would still be great to try to bisect to the particular patch causing
> > the issue.

My 0.2 cents on something that might be related:

On my Intel laptop I can't boot a windows guest with hyperv enabled inside (either regular hyperv win10 or win11 with core isolation)
I know now that 'ibt=off' on host kernel line fixes this, but I didn't yet bisected it to see which commit started it.
(I took this from https://bugzilla.redhat.com/show_bug.cgi?id=2221531, which is unrelated but I just noticed it somehow and tried the solution)

I run upstream 6.4 kernel + kvm/queue on that laptop.

Best regards,
	Maxim Levitsky

Re: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Roman Mamedov]

On Wed, 16 Aug 2023 15:41:08 +0200
Vitaly Kuznetsov <vkuznets@redhat.com> wrote:

> Sean's https://lore.kernel.org/all/20230811155255.250835-1-seanjc@google.com/
> (alteady in 'tip') can actually be related and I see it was already
> tagged for stable@. Can anyone check if it really helps?

Indeed, this patch appears to fix it. I built 6.1.46 with it added, and the
issue is no longer present. Thanks!

-- 
With respect,
Roman

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

On Wed, Aug 16, 2023 at 04:29:32PM +0700, Bagas Sanjaya wrote:
> #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
> #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)

#regzbot fix: https://lore.kernel.org/all/20230811155255.250835-1-seanjc@google.com

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Fwd: kvm: Windows Server 2003 VM fails to work on 6.1.44 (works fine on 6.1.43)

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 17.08.23 05:05, Bagas Sanjaya wrote:
> On Wed, Aug 16, 2023 at 04:29:32PM +0700, Bagas Sanjaya wrote:
>> #regzbot introduced: v6.1.43..v6.1.44 https://bugzilla.kernel.org/show_bug.cgi?id=217799
>> #regzbot title: Windows Server 2003 VM boot hang (only 64MB RAM allocated)
> 
> #regzbot fix: https://lore.kernel.org/all/20230811155255.250835-1-seanjc@google.com

#regzbot fix: b41eb316c95c98d16dba2045c4890a3020f0a5
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.