[Bug 208091] New: vcpu1, guest rIP offset ignored wrmsr or rdmsr

[Bug 208091] New: vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

            Bug ID: 208091
           Summary: vcpu1, guest rIP offset ignored wrmsr or rdmsr
           Product: Virtualization
           Version: unspecified
    Kernel Version: Linux pvx 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15
                    May 2020 15:06:08 +0200) x86_64 GNU/Linux
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: low
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: commandline@protonmail.com
        Regression: No

On launch of VM these messages are generated. Filing this report since hinted
at a reason for improvement such as for patch 7670771. First ever report, sorry
if violating any good habits.



pvx kernel: [   77.102686] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x3a
pvx kernel: [   77.102712] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0xd90
pvx kernel: [   77.102736] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x570
pvx kernel: [   77.102755] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x571
pvx kernel: [   77.102773] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x572
pvx kernel: [   77.102792] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x560
pvx kernel: [   77.102811] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x561
pvx kernel: [   77.102829] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x580
pvx kernel: [   77.102847] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x581
pvx kernel: [   77.102866] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x582

pvx kernel: [   77.102686] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x3a
pvx kernel: [   77.102712] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0xd90
pvx kernel: [   77.102736] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x570
pvx kernel: [   77.102755] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x571
pvx kernel: [   77.102773] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x572
pvx kernel: [   77.102792] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x560
pvx kernel: [   77.102811] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x561
pvx kernel: [   77.102829] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x580
pvx kernel: [   77.102847] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x581
pvx kernel: [   77.102866] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x582

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. (commandline@protonmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Hardware|All                         |x86-64
           Severity|low                         |normal

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. (commandline@protonmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://patchwork.kernel.or
                   |                            |g/patch/7670771/

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #1 from Joris L. (commandline@protonmail.com) ---
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0
address=0x7fb99eeb0]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0
address=0x7fb99ef10]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0
address=0x7fb99ef50]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0
address=0x7fb99ef70]
vfio-pci 0000:28:00.0: timed out waiting for pending transaction; performing
function level reset anyway

device=DD < anonimised

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #2 from Joris L. (commandline@protonmail.com) ---
More of the same

[  507.188509] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[  507.188524] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[  513.661635] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.661644] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.703575] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.703588] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.704769] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.704782] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.705948] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.705961] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Sean Christopherson (sean.j.christopherson@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sean.j.christopherson@intel
                   |                            |.com

--- Comment #3 from Sean Christopherson (sean.j.christopherson@intel.com) ---
The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is
not the default.  Is that enabled for any particular reason?  Or maybe a better
question is, what is it that you're trying to do and what, if any, errors are
occurring?

E.g. in the first trace, it looks like you're advertising an Intel vCPU to the
guest on top of AMD hardware.  That can be made to work, but it's not
recommended.  Ignoring MSRs in that scenario is almost guaranteed to cause
weirdness.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #4 from Joris L. (commandline@protonmail.com) ---
These message are observed when starting a Proxmox VE 6.2 VM guest with GPU
passthrough.  Current configuration is at the point there are not other error
messages visible related to VM activity.

Available to respond to and provide for actions to improve documenting these
messages etc.

The vcpu0, guest rIP messages persist while a vm is running. Stopping and
staring a VM results in no screen. messages however now show (repeat kvm ...
msr messages, repeeat block of kvm, vcpu... rIP messages) followed by ecap
messages.

[ 2762.744734] kvm_get_msr_common: 275 callbacks suppressed
[ 2762.744738] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010064
[ 2762.744775] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010065
[ 2762.744853] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010066
[ 2762.744885] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010067
[ 2762.744917] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010068
[ 2762.744948] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010069
[ 2762.744983] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc001006a
[ 2762.745014] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc001006b
[ 2762.745044] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010293
[ 2762.750132] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xe8
[ 2844.034627] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[ 2844.034641] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[ 2845.308346] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.324311] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.566461] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.569777] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.599693] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.602566] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.625243] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.627436] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.659282] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.665352] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673673] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673924] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673948] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.674104] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.674870] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675040] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675063] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675210] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.708785] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.710674] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.727404] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.727598] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.728296] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.728484] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.764636] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.765256] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.933638] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs

now the VM no longer displays the desktop.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #5 from Joris L. (commandline@protonmail.com) ---
(In reply to Sean Christopherson from comment #3)
> The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is
> not the default.  Is that enabled for any particular reason?  Or maybe a
> better question is, what is it that you're trying to do and what, if any,
> errors are occurring?
> 
> E.g. in the first trace, it looks like you're advertising an Intel vCPU to
> the guest on top of AMD hardware.  That can be made to work, but it's not
> recommended.  Ignoring MSRs in that scenario is almost guaranteed to cause
> weirdness.

Thanks Sean. I am cleaning up this config. Will check.

This is indeed a Ryzen CPU with Qemu/Proxmox on top of it.
For GPU Passthrough to a VM the kvm.ignore_msrs=1 is often recommended.
I brief test shows the VM is not working properly without this parameter, even
when booting the VM with CPU:host.

br,

Joris

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #6 from Joris L. (commandline@protonmail.com) ---
These i get with kvm ignore.msrs=1 enabled and CPU=host

[   46.512153] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.512161] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.555401] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.555406] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.556809] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.556823] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.558065] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.558078] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #7 from Joris L. (commandline@protonmail.com) ---
With the kvm ignor.msrs parameter removed i do indeed see different results.
Now i am back at these messages. Sorry for the ignorance, learning as i read
the feedback.

[10427.220982] vfio-pci 0000:17:00.0: timed out waiting for pending
transaction; performing function level reset anyway
[10428.469006] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10429.524940] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10431.668982] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10436.020672] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting
[10444.469003] vfio-pci 0000:17:00.0: not ready 16383ms after FLR; waiting
[10462.133026] vfio-pci 0000:17:00.0: not ready 32767ms after FLR; waiting
[10496.948661] vfio-pci 0000:17:00.0: not ready 65535ms after FLR; giving up

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #8 from Joris L. (commandline@protonmail.com) ---
followed by


[10956.384880] vfio-pci 0000:17:00.0: Refused to change power state, currently
in D3
[10957.109026] vfio-pci 0000:17:00.0: timed out waiting for pending
transaction; performing function level reset anyway
[10958.356694] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10959.412727] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10961.588924] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10965.940829] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. (commandline@protonmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |DOCUMENTED

--- Comment #9 from Joris L. (commandline@protonmail.com) ---
eventually, boot time parameters resolved all of these issues, sharing in case
they matter.

iommu=pt amd_iommu=forced_isolation video=vesafb:off,efifb:off rcu_nocbs=0-15
amdgpu.runpm=0 pci=nobar,noats,big_root_window
pcie_acs_override=downstream,multifunction pcie_pme=nomsi pcie_aspm=force
pcie_ports=native

-- 
You are receiving this mail because:
You are watching the assignee of the bug.