Issues on Zen4 (hw12) runner

Issues on Zen4 (hw12) runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2970 bytes --]

Hi,

I wanted to post another revision of the series adding hw12 runner,
hoping that all known issues are fixed now, but unfortunately there is
still something broken. I've rebased my series on top of staging
(ed9488a0d) and got this pipeline:

https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
(note due to some added debugging, some tests are incorrectly marked as
success even if they failed...)

1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
There supposed to be an USB ethernet device connected to the USB
controller at c3:00.4. In the PV dom0 case it's detected as:

    [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
    [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
    [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
    [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
    [    4.005349] usb 7-1.4: Manufacturer: Realtek
    [    4.005599] usb 7-1.4: SerialNumber: 684D35

But it's not there on PVH. The USB controller itself is detected, just
not device(s) connected to it. This applies to other controllers too
(there should be about 3 or 4 other USB devices - none of them show up).

2. There is a bunch of "unhandled memory read" errors during PVH dom0
startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739

    (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
    (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
    (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
    (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
    ...

This repeats several times. Could be related to the USB issue above?

There is also, likely related:

    (XEN) [    5.002036] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
    (XEN) [    5.002365] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
    (XEN) [    5.002693] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0

3. Sometimes it fails to print anything on the console, like here: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9977761447
This is likely some boot issue before Xen starts (possibly the power button
is pressed to early). Anyway, I need to fix it before adding the runner.

4. There is a bunch of unknown MSR accesses, but that's likely to be
expected. For example:

    (XEN) [    6.010446] arch/x86/pv/emul-priv-op.c:1017:d0v11 RDMSR 0xc00102b0 unimplemented
    (XEN) [    6.010798] arch/x86/pv/emul-priv-op.c:1017:d0v0 RDMSR 0xc00102b1 unimplemented


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Issues on Zen4 (hw12) runner

[Jan Beulich]

On 08.05.2025 22:44, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> I wanted to post another revision of the series adding hw12 runner,
> hoping that all known issues are fixed now, but unfortunately there is
> still something broken. I've rebased my series on top of staging
> (ed9488a0d) and got this pipeline:
> 
> https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> (note due to some added debugging, some tests are incorrectly marked as
> success even if they failed...)
> 
> 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> There supposed to be an USB ethernet device connected to the USB
> controller at c3:00.4. In the PV dom0 case it's detected as:
> 
>     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
>     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
>     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
>     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
>     [    4.005349] usb 7-1.4: Manufacturer: Realtek
>     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> 
> But it's not there on PVH. The USB controller itself is detected, just
> not device(s) connected to it. This applies to other controllers too
> (there should be about 3 or 4 other USB devices - none of them show up).
> 
> 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> 
>     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
>     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
>     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
>     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
>     ...
> 
> This repeats several times. Could be related to the USB issue above?

Yes.

> There is also, likely related:
> 
>     (XEN) [    5.002036] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
>     (XEN) [    5.002365] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
>     (XEN) [    5.002693] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0

Not very likely - these are (sadly) normal to see when MSIs are being turned
off by the kernel.

Jan

Re: Issues on Zen4 (hw12) runner

[Roger Pau Monné]

On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> I wanted to post another revision of the series adding hw12 runner,
> hoping that all known issues are fixed now, but unfortunately there is
> still something broken. I've rebased my series on top of staging
> (ed9488a0d) and got this pipeline:
> 
> https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> (note due to some added debugging, some tests are incorrectly marked as
> success even if they failed...)
> 
> 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> There supposed to be an USB ethernet device connected to the USB
> controller at c3:00.4. In the PV dom0 case it's detected as:
> 
>     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
>     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
>     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
>     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
>     [    4.005349] usb 7-1.4: Manufacturer: Realtek
>     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> 
> But it's not there on PVH. The USB controller itself is detected, just
> not device(s) connected to it. This applies to other controllers too
> (there should be about 3 or 4 other USB devices - none of them show up).
> 
> 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> 
>     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
>     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
>     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
>     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
>     ...
> 
> This repeats several times. Could be related to the USB issue above?

Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
cause of the USB issues.

> There is also, likely related:
> 
>     (XEN) [    5.002036] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
>     (XEN) [    5.002365] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
>     (XEN) [    5.002693] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0

Is this at shutdown? (doesn't look like by the timestamps).  There are
cases where Linux zeroes the MSR entries while the capability is still
enabled, and that results in those messages.  They are usually benign.

> 
> 3. Sometimes it fails to print anything on the console, like here: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9977761447
> This is likely some boot issue before Xen starts (possibly the power button
> is pressed to early). Anyway, I need to fix it before adding the runner.

That needs further debug, I'm afraid I can't provide much suggestions.

Thanks, Roger.

Re: Issues on Zen4 (hw12) runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3286 bytes --]

On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > I wanted to post another revision of the series adding hw12 runner,
> > hoping that all known issues are fixed now, but unfortunately there is
> > still something broken. I've rebased my series on top of staging
> > (ed9488a0d) and got this pipeline:
> > 
> > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > (note due to some added debugging, some tests are incorrectly marked as
> > success even if they failed...)
> > 
> > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > There supposed to be an USB ethernet device connected to the USB
> > controller at c3:00.4. In the PV dom0 case it's detected as:
> > 
> >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > 
> > But it's not there on PVH. The USB controller itself is detected, just
> > not device(s) connected to it. This applies to other controllers too
> > (there should be about 3 or 4 other USB devices - none of them show up).
> > 
> > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > 
> >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> >     ...
> > 
> > This repeats several times. Could be related to the USB issue above?
> 
> Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> cause of the USB issues.

It did got rid of those messages, but USB still doesn't work:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289

> > There is also, likely related:
> > 
> >     (XEN) [    5.002036] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> >     (XEN) [    5.002365] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> >     (XEN) [    5.002693] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> 
> Is this at shutdown? (doesn't look like by the timestamps).  There are
> cases where Linux zeroes the MSR entries while the capability is still
> enabled, and that results in those messages.  They are usually benign.

That's not shutdown. But also it's a different device than I care the
most, so I guess I can ignore it for now.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Issues on Zen4 (hw12) runner

[Roger Pau Monné]

On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > > 
> > > I wanted to post another revision of the series adding hw12 runner,
> > > hoping that all known issues are fixed now, but unfortunately there is
> > > still something broken. I've rebased my series on top of staging
> > > (ed9488a0d) and got this pipeline:
> > > 
> > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > (note due to some added debugging, some tests are incorrectly marked as
> > > success even if they failed...)
> > > 
> > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > There supposed to be an USB ethernet device connected to the USB
> > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > 
> > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > 
> > > But it's not there on PVH. The USB controller itself is detected, just
> > > not device(s) connected to it. This applies to other controllers too
> > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > 
> > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > 
> > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > >     ...
> > > 
> > > This repeats several times. Could be related to the USB issue above?
> > 
> > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > cause of the USB issues.
> 
> It did got rid of those messages, but USB still doesn't work:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289

Hm, is it possible that the usage of console=xhci is interfering with
USB devices?  Could you try to boot without console=xhci and see if
you can still reproduce the issue?  You will need the physical device
by your side, which I'm not sure it's possible.  Don't know if you
host those remotely somewhere.

> > > There is also, likely related:
> > > 
> > >     (XEN) [    5.002036] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> > >     (XEN) [    5.002365] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> > >     (XEN) [    5.002693] arch/x86/hvm/vmsi.c:845:d0v9 0000:c1:00.1: PIRQ 2484: unsupported address 0
> > 
> > Is this at shutdown? (doesn't look like by the timestamps).  There are
> > cases where Linux zeroes the MSR entries while the capability is still
> > enabled, and that results in those messages.  They are usually benign.
> 
> That's not shutdown. But also it's a different device than I care the
> most, so I guess I can ignore it for now.

Even if you see those messages the device might work OK - it's just
that at some point Linux has set the MSI address field as 0.  Xen
won't print anything when the address is switched from 0 (invalid) to
a valid value.

Thanks, Roger.

Re: Issues on Zen4 (hw12) runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

On Mon, May 12, 2025 at 04:19:59PM +0200, Roger Pau Monné wrote:
> On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> > On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > I wanted to post another revision of the series adding hw12 runner,
> > > > hoping that all known issues are fixed now, but unfortunately there is
> > > > still something broken. I've rebased my series on top of staging
> > > > (ed9488a0d) and got this pipeline:
> > > > 
> > > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > > (note due to some added debugging, some tests are incorrectly marked as
> > > > success even if they failed...)
> > > > 
> > > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > There supposed to be an USB ethernet device connected to the USB
> > > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > > 
> > > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > > 
> > > > But it's not there on PVH. The USB controller itself is detected, just
> > > > not device(s) connected to it. This applies to other controllers too
> > > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > > 
> > > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > 
> > > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > >     ...
> > > > 
> > > > This repeats several times. Could be related to the USB issue above?
> > > 
> > > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > > cause of the USB issues.
> > 
> > It did got rid of those messages, but USB still doesn't work:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289
> 
> Hm, is it possible that the usage of console=xhci is interfering with
> USB devices?  Could you try to boot without console=xhci and see if
> you can still reproduce the issue?  You will need the physical device
> by your side, which I'm not sure it's possible.  Don't know if you
> host those remotely somewhere.

I can try, but will need a proper driver there (in dom0?) - AFAIR VGA
nor efifb doesn't output to HDMI there (and eDP is not connected).
Anyway, it's IMO unlikely, given it works just fine with PV dom0...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Issues on Zen4 (hw12) runner

[Roger Pau Monné]

On Mon, May 12, 2025 at 04:40:29PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, May 12, 2025 at 04:19:59PM +0200, Roger Pau Monné wrote:
> > On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > > > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > Hi,
> > > > > 
> > > > > I wanted to post another revision of the series adding hw12 runner,
> > > > > hoping that all known issues are fixed now, but unfortunately there is
> > > > > still something broken. I've rebased my series on top of staging
> > > > > (ed9488a0d) and got this pipeline:
> > > > > 
> > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > > > (note due to some added debugging, some tests are incorrectly marked as
> > > > > success even if they failed...)
> > > > > 
> > > > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > There supposed to be an USB ethernet device connected to the USB
> > > > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > > > 
> > > > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > > > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > > > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > > > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > > > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > > > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > > > 
> > > > > But it's not there on PVH. The USB controller itself is detected, just
> > > > > not device(s) connected to it. This applies to other controllers too
> > > > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > > > 
> > > > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > 
> > > > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > >     ...
> > > > > 
> > > > > This repeats several times. Could be related to the USB issue above?
> > > > 
> > > > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > > > cause of the USB issues.
> > > 
> > > It did got rid of those messages, but USB still doesn't work:
> > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289
> > 
> > Hm, is it possible that the usage of console=xhci is interfering with
> > USB devices?  Could you try to boot without console=xhci and see if
> > you can still reproduce the issue?  You will need the physical device
> > by your side, which I'm not sure it's possible.  Don't know if you
> > host those remotely somewhere.
> 
> I can try, but will need a proper driver there (in dom0?) - AFAIR VGA
> nor efifb doesn't output to HDMI there (and eDP is not connected).
> Anyway, it's IMO unlikely, given it works just fine with PV dom0...

Oh, I see, that's a good data point that it works with PV dom0.
Handling of r/o subpage accesses is still different between PV and PVH
which could maybe explain this, but it's less likely.

Maybe I'm not spotting it, but I don't see any specific errors (like
timeouts) from the XHCI controller on the log?  Neither there seems to
be any errors or warnings from Xen.

Thanks, Roger.

Re: Issues on Zen4 (hw12) runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4231 bytes --]

On Mon, May 12, 2025 at 04:50:22PM +0200, Roger Pau Monné wrote:
> On Mon, May 12, 2025 at 04:40:29PM +0200, Marek Marczykowski-Górecki wrote:
> > On Mon, May 12, 2025 at 04:19:59PM +0200, Roger Pau Monné wrote:
> > > On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> > > > On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > > > > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > I wanted to post another revision of the series adding hw12 runner,
> > > > > > hoping that all known issues are fixed now, but unfortunately there is
> > > > > > still something broken. I've rebased my series on top of staging
> > > > > > (ed9488a0d) and got this pipeline:
> > > > > > 
> > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > > > > (note due to some added debugging, some tests are incorrectly marked as
> > > > > > success even if they failed...)
> > > > > > 
> > > > > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > There supposed to be an USB ethernet device connected to the USB
> > > > > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > > > > 
> > > > > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > > > > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > > > > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > > > > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > > > > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > > > > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > > > > 
> > > > > > But it's not there on PVH. The USB controller itself is detected, just
> > > > > > not device(s) connected to it. This applies to other controllers too
> > > > > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > > > > 
> > > > > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > > > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > 
> > > > > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > >     ...
> > > > > > 
> > > > > > This repeats several times. Could be related to the USB issue above?
> > > > > 
> > > > > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > > > > cause of the USB issues.
> > > > 
> > > > It did got rid of those messages, but USB still doesn't work:
> > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289
> > > 
> > > Hm, is it possible that the usage of console=xhci is interfering with
> > > USB devices?  Could you try to boot without console=xhci and see if
> > > you can still reproduce the issue?  You will need the physical device
> > > by your side, which I'm not sure it's possible.  Don't know if you
> > > host those remotely somewhere.
> > 
> > I can try, but will need a proper driver there (in dom0?) - AFAIR VGA
> > nor efifb doesn't output to HDMI there (and eDP is not connected).
> > Anyway, it's IMO unlikely, given it works just fine with PV dom0...
> 
> Oh, I see, that's a good data point that it works with PV dom0.
> Handling of r/o subpage accesses is still different between PV and PVH
> which could maybe explain this, but it's less likely.
> 
> Maybe I'm not spotting it, but I don't see any specific errors (like
> timeouts) from the XHCI controller on the log?  Neither there seems to
> be any errors or warnings from Xen.

I don't see any either...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Issues on Zen4 (hw12) runner

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4514 bytes --]

On Mon, May 12, 2025 at 05:11:53PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, May 12, 2025 at 04:50:22PM +0200, Roger Pau Monné wrote:
> > On Mon, May 12, 2025 at 04:40:29PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Mon, May 12, 2025 at 04:19:59PM +0200, Roger Pau Monné wrote:
> > > > On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > > > > > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I wanted to post another revision of the series adding hw12 runner,
> > > > > > > hoping that all known issues are fixed now, but unfortunately there is
> > > > > > > still something broken. I've rebased my series on top of staging
> > > > > > > (ed9488a0d) and got this pipeline:
> > > > > > > 
> > > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > > > > > (note due to some added debugging, some tests are incorrectly marked as
> > > > > > > success even if they failed...)
> > > > > > > 
> > > > > > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > > There supposed to be an USB ethernet device connected to the USB
> > > > > > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > > > > > 
> > > > > > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > > > > > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > > > > > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > > > > > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > > > > > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > > > > > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > > > > > 
> > > > > > > But it's not there on PVH. The USB controller itself is detected, just
> > > > > > > not device(s) connected to it. This applies to other controllers too
> > > > > > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > > > > > 
> > > > > > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > > > > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > > 
> > > > > > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > > >     ...
> > > > > > > 
> > > > > > > This repeats several times. Could be related to the USB issue above?
> > > > > > 
> > > > > > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > > > > > cause of the USB issues.
> > > > > 
> > > > > It did got rid of those messages, but USB still doesn't work:
> > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289
> > > > 
> > > > Hm, is it possible that the usage of console=xhci is interfering with
> > > > USB devices?  Could you try to boot without console=xhci and see if
> > > > you can still reproduce the issue?  You will need the physical device
> > > > by your side, which I'm not sure it's possible.  Don't know if you
> > > > host those remotely somewhere.
> > > 
> > > I can try, but will need a proper driver there (in dom0?) - AFAIR VGA
> > > nor efifb doesn't output to HDMI there (and eDP is not connected).
> > > Anyway, it's IMO unlikely, given it works just fine with PV dom0...
> > 
> > Oh, I see, that's a good data point that it works with PV dom0.
> > Handling of r/o subpage accesses is still different between PV and PVH
> > which could maybe explain this, but it's less likely.
> > 
> > Maybe I'm not spotting it, but I don't see any specific errors (like
> > timeouts) from the XHCI controller on the log?  Neither there seems to
> > be any errors or warnings from Xen.
> 
> I don't see any either...

Roger, it looks like your balloon patch fixes the USB case too :)

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Issues on Zen4 (hw12) runner

[Roger Pau Monné]

On Wed, May 14, 2025 at 11:12:20PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, May 12, 2025 at 05:11:53PM +0200, Marek Marczykowski-Górecki wrote:
> > On Mon, May 12, 2025 at 04:50:22PM +0200, Roger Pau Monné wrote:
> > > On Mon, May 12, 2025 at 04:40:29PM +0200, Marek Marczykowski-Górecki wrote:
> > > > On Mon, May 12, 2025 at 04:19:59PM +0200, Roger Pau Monné wrote:
> > > > > On Mon, May 12, 2025 at 03:31:19PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > On Mon, May 12, 2025 at 12:26:09PM +0200, Roger Pau Monné wrote:
> > > > > > > On Thu, May 08, 2025 at 10:44:36PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > I wanted to post another revision of the series adding hw12 runner,
> > > > > > > > hoping that all known issues are fixed now, but unfortunately there is
> > > > > > > > still something broken. I've rebased my series on top of staging
> > > > > > > > (ed9488a0d) and got this pipeline:
> > > > > > > > 
> > > > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1807819142
> > > > > > > > (note due to some added debugging, some tests are incorrectly marked as
> > > > > > > > success even if they failed...)
> > > > > > > > 
> > > > > > > > 1. USB ethernet doesn't work on PVH dom0: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > > > There supposed to be an USB ethernet device connected to the USB
> > > > > > > > controller at c3:00.4. In the PV dom0 case it's detected as:
> > > > > > > > 
> > > > > > > >     [    3.911555] usb 7-1.4: new high-speed USB device number 3 using xhci_hcd
> > > > > > > >     [    4.004201] usb 7-1.4: New USB device found, idVendor=0bda, idProduct=8153, bcdDevice=30.00
> > > > > > > >     [    4.004675] usb 7-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
> > > > > > > >     [    4.005079] usb 7-1.4: Product: USB 10/100/1000 LAN
> > > > > > > >     [    4.005349] usb 7-1.4: Manufacturer: Realtek
> > > > > > > >     [    4.005599] usb 7-1.4: SerialNumber: 684D35
> > > > > > > > 
> > > > > > > > But it's not there on PVH. The USB controller itself is detected, just
> > > > > > > > not device(s) connected to it. This applies to other controllers too
> > > > > > > > (there should be about 3 or 4 other USB devices - none of them show up).
> > > > > > > > 
> > > > > > > > 2. There is a bunch of "unhandled memory read" errors during PVH dom0
> > > > > > > > startup: https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/9978694739
> > > > > > > > 
> > > > > > > >     (XEN) [    4.026323] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > > > >     (XEN) [    4.026789] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > > > >     (XEN) [    4.027247] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0020 size 1
> > > > > > > >     (XEN) [    4.027671] arch/x86/hvm/emulate.c:417:d0v0 unhandled memory read from 0xfedc0021 size 1
> > > > > > > >     ...
> > > > > > > > 
> > > > > > > > This repeats several times. Could be related to the USB issue above?
> > > > > > > 
> > > > > > > Can you try with dom0=pf-fixup?  Those unhandled accesses might be the
> > > > > > > cause of the USB issues.
> > > > > > 
> > > > > > It did got rid of those messages, but USB still doesn't work:
> > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10006580289
> > > > > 
> > > > > Hm, is it possible that the usage of console=xhci is interfering with
> > > > > USB devices?  Could you try to boot without console=xhci and see if
> > > > > you can still reproduce the issue?  You will need the physical device
> > > > > by your side, which I'm not sure it's possible.  Don't know if you
> > > > > host those remotely somewhere.
> > > > 
> > > > I can try, but will need a proper driver there (in dom0?) - AFAIR VGA
> > > > nor efifb doesn't output to HDMI there (and eDP is not connected).
> > > > Anyway, it's IMO unlikely, given it works just fine with PV dom0...
> > > 
> > > Oh, I see, that's a good data point that it works with PV dom0.
> > > Handling of r/o subpage accesses is still different between PV and PVH
> > > which could maybe explain this, but it's less likely.
> > > 
> > > Maybe I'm not spotting it, but I don't see any specific errors (like
> > > timeouts) from the XHCI controller on the log?  Neither there seems to
> > > be any errors or warnings from Xen.
> > 
> > I don't see any either...
> 
> Roger, it looks like your balloon patch fixes the USB case too :)

Oh, that's great to hear.  I hope I can merge that one together with
the xen.config change soon.  Would you mind giving a Tested-by to the
balloon patch?

https://lore.kernel.org/xen-devel/20250514080427.28129-1-roger.pau@citrix.com/

Thanks, Roger.