ACPI timeouts when enabling KASAN

ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

Hi

I am using virtme to do some CI around linux-media.

Everything works as expected, but when I enable KASAN, I am starting
to get a lot of timeouts when the Method _PRT is executed. Eg:

[   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
[   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
[   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
[   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
[   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
[   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
[   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
[   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI

0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)

If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
always executed, but it is very annoying that I have to wait more than
5 minutes to start the vm.

Despite not having kvm enabled, the machine is quite decent, so I
would expect that it could run that method relatively fast.

Do you have any hint of what I should be looking at?

Thanks!

```
# virtme-ng --version
virtme-ng 1.23
# qemu-system-amd64 --version
QEMU emulator version 8.2.1 (Debian 1:8.2.1+ds-2)
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
# git describe
v6.9-rc3-208-g586b5dfb51b96

# virtme-configkernel --defconfig --arch x86_64
# scripts/config -e KASAN
# make olddefconfig
# make all -j 256
# virtme-run --kdir . --mods=auto --show-command --show-boot-console
--verbose  --cpus 2 --memory 4G --script-sh "echo HelloWorld"
```



-- 
Ricardo Ribalda
Software Engineer
ribalda@google.com

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> Hi

Hi Ricardo,

> 
> I am using virtme to do some CI around linux-media.
> 
> Everything works as expected, but when I enable KASAN, I am starting
> to get a lot of timeouts when the Method _PRT is executed. Eg:
> 
> [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> 
> 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> 
> If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> always executed, but it is very annoying that I have to wait more than
> 5 minutes to start the vm.
> 
> Despite not having kvm enabled, the machine is quite decent, so I
> would expect that it could run that method relatively fast.
> 
> Do you have any hint of what I should be looking at?

I'm wondering if it's a microvm-related issue...

Can you try to add --disable-microvm to your virtme command line and see
if it makes any difference?

Thanks,
-Andrea

> 
> Thanks!
> 
> ```
> # virtme-ng --version
> virtme-ng 1.23
> # qemu-system-amd64 --version
> QEMU emulator version 8.2.1 (Debian 1:8.2.1+ds-2)
> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> # git describe
> v6.9-rc3-208-g586b5dfb51b96
> 
> # virtme-configkernel --defconfig --arch x86_64
> # scripts/config -e KASAN
> # make olddefconfig
> # make all -j 256
> # virtme-run --kdir . --mods=auto --show-command --show-boot-console
> --verbose  --cpus 2 --memory 4G --script-sh "echo HelloWorld"
> ```
> 
> 
> 
> -- 
> Ricardo Ribalda
> Software Engineer
> ribalda@google.com

Re: ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

On Sat, 13 Apr 2024 at 07:56, Andrea Righi <andrea.righi@canonical.com> wrote:
>
> On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> > Hi
>
> Hi Ricardo,

Hi Andrea!

>
> >
> > I am using virtme to do some CI around linux-media.
> >
> > Everything works as expected, but when I enable KASAN, I am starting
> > to get a lot of timeouts when the Method _PRT is executed. Eg:
> >
> > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> >
> > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> >
> > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > always executed, but it is very annoying that I have to wait more than
> > 5 minutes to start the vm.
> >
> > Despite not having kvm enabled, the machine is quite decent, so I
> > would expect that it could run that method relatively fast.
> >
> > Do you have any hint of what I should be looking at?
>
> I'm wondering if it's a microvm-related issue...
>
> Can you try to add --disable-microvm to your virtme command line and see
> if it makes any difference?

I get the same results :(. BTW, this is the service that I think it is
taking lots of time:


            Method (_PRT, 0, NotSerialized)  // _PRT: PCI Routing Table
            {
                Local0 = Package (0x80){}
                Local1 = Zero
                While ((Local1 < 0x80))
                {
                    Local2 = (Local1 >> 0x02)
                    Local3 = ((Local1 + Local2) & 0x03)
                    If ((Local3 == Zero))
                    {
                        Local4 = Package (0x04)
                            {
                                Zero,
                                Zero,
                                LNKD,
                                Zero
                            }
                    }

                    If ((Local3 == One))
                    {
                        If ((Local1 == 0x04))
                        {
                            Local4 = Package (0x04)
                                {
                                    Zero,
                                    Zero,
                                    LNKS,
                                    Zero
                                }
                        }
                        Else
                        {
                            Local4 = Package (0x04)
                                {
                                    Zero,
                                    Zero,
                                    LNKA,
                                    Zero
                                }
                        }
                    }

                    If ((Local3 == 0x02))
                    {
                        Local4 = Package (0x04)
                            {
                                Zero,
                                Zero,
                                LNKB,
                                Zero
                            }
                    }

                    If ((Local3 == 0x03))
                    {
                        Local4 = Package (0x04)
                            {
                                Zero,
                                Zero,
                                LNKC,
                                Zero
                            }
                    }

                    Local4 [Zero] = ((Local2 << 0x10) | 0xFFFF)
                    Local4 [One] = (Local1 & 0x03)
                    Local0 [Local1] = Local4
                    Local1++
                }

                Return (Local0)
            }
        }
>
> Thanks,
> -Andrea
>
> >
> > Thanks!
> >
> > ```
> > # virtme-ng --version
> > virtme-ng 1.23
> > # qemu-system-amd64 --version
> > QEMU emulator version 8.2.1 (Debian 1:8.2.1+ds-2)
> > Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> > # git describe
> > v6.9-rc3-208-g586b5dfb51b96
> >
> > # virtme-configkernel --defconfig --arch x86_64
> > # scripts/config -e KASAN
> > # make olddefconfig
> > # make all -j 256
> > # virtme-run --kdir . --mods=auto --show-command --show-boot-console
> > --verbose  --cpus 2 --memory 4G --script-sh "echo HelloWorld"
> > ```
> >
> >
> >
> > --
> > Ricardo Ribalda
> > Software Engineer
> > ribalda@google.com



-- 
Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Michael S. Tsirkin]

On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> Hi
> 
> I am using virtme to do some CI around linux-media.
> 
> Everything works as expected, but when I enable KASAN, I am starting
> to get a lot of timeouts when the Method _PRT is executed. Eg:
> 
> [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> 
> 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> 
> If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> always executed, but it is very annoying that I have to wait more than
> 5 minutes to start the vm.
> 
> Despite not having kvm enabled, the machine is quite decent, so I
> would expect that it could run that method relatively fast.
> 
> Do you have any hint of what I should be looking at?
> 
> Thanks!


CC Igor.


> ```
> # virtme-ng --version
> virtme-ng 1.23
> # qemu-system-amd64 --version
> QEMU emulator version 8.2.1 (Debian 1:8.2.1+ds-2)
> Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> # git describe
> v6.9-rc3-208-g586b5dfb51b96
> 
> # virtme-configkernel --defconfig --arch x86_64
> # scripts/config -e KASAN
> # make olddefconfig
> # make all -j 256
> # virtme-run --kdir . --mods=auto --show-command --show-boot-console
> --verbose  --cpus 2 --memory 4G --script-sh "echo HelloWorld"
> ```
> 
> 
> 
> -- 
> Ricardo Ribalda
> Software Engineer
> ribalda@google.com

Re: ACPI timeouts when enabling KASAN

[Igor Mammedov]

On Sun, 14 Apr 2024 04:37:24 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> > Hi
> > 
> > I am using virtme to do some CI around linux-media.
> > 
> > Everything works as expected, but when I enable KASAN, I am starting
> > to get a lot of timeouts when the Method _PRT is executed. Eg:
> > 
> > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> > 
> > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> > 
> > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > always executed, but it is very annoying that I have to wait more than
> > 5 minutes to start the vm.
> > 
> > Despite not having kvm enabled, the machine is quite decent, so I
> > would expect that it could run that method relatively fast.
> > 
> > Do you have any hint of what I should be looking at?

The way QEMU generates _PRT haven't been changed for ages,
it's not likely to be a culprit.

CCing Rafael who might have an idea why ACPI misbehaves.

Can you share kernel config and full QEMU CLI used,
for me to try to reproduce the issue locally?



> > Thanks!  
> 
> 
> CC Igor.
> 
> 
> > ```
> > # virtme-ng --version
> > virtme-ng 1.23
> > # qemu-system-amd64 --version
> > QEMU emulator version 8.2.1 (Debian 1:8.2.1+ds-2)
> > Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
> > # git describe
> > v6.9-rc3-208-g586b5dfb51b96
> > 
> > # virtme-configkernel --defconfig --arch x86_64
> > # scripts/config -e KASAN
> > # make olddefconfig
> > # make all -j 256
> > # virtme-run --kdir . --mods=auto --show-command --show-boot-console
> > --verbose  --cpus 2 --memory 4G --script-sh "echo HelloWorld"
> > ```
> > 
> > 
> > 
> > -- 
> > Ricardo Ribalda
> > Software Engineer
> > ribalda@google.com  
> 

Re: ACPI timeouts when enabling KASAN

[Rafael J. Wysocki]

On Mon, Apr 15, 2024 at 2:52 PM Igor Mammedov <imammedo@redhat.com> wrote:
>
> On Sun, 14 Apr 2024 04:37:24 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> > > Hi
> > >
> > > I am using virtme to do some CI around linux-media.
> > >
> > > Everything works as expected, but when I enable KASAN, I am starting
> > > to get a lot of timeouts when the Method _PRT is executed. Eg:
> > >
> > > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> > >
> > > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> > >
> > > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > > always executed, but it is very annoying that I have to wait more than
> > > 5 minutes to start the vm.
> > >
> > > Despite not having kvm enabled, the machine is quite decent, so I
> > > would expect that it could run that method relatively fast.
> > >
> > > Do you have any hint of what I should be looking at?
>
> The way QEMU generates _PRT haven't been changed for ages,
> it's not likely to be a culprit.
>
> CCing Rafael who might have an idea why ACPI misbehaves.

It looks like the method evaluation time increases with KASAN enabled
and it gets aborted due to exceeding the evaluation time limit.

Thanks!

Re: ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

Hi Igor, Hi Rafael

Yes, it seems that it is just KASAN being extremely slow.
From a completely newbie here... Is there a reason why qemu generates
the table vs returning a precomputed one?

This is the config file:
https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads

And this is the qemu cli:

/usr/bin/qemu-system-x86_64 -m 4G -fsdev
local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
-device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
-display none -serial chardev:console -chardev
file,id=console,path=/proc/self/fd/2 -chardev
stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
virtserialport,name=virtme.stdin,chardev=stdin -chardev
file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
virtserialport,name=virtme.stdout,chardev=stdout -chardev
file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
virtserialport,name=virtme.stderr,chardev=stderr -chardev
file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
-device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
-chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
virtio-serial-pci -device
virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
-device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
./arch/x86/boot/bzImage -append 'nr_open=1048576
virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
virtme_root_user=1 rootfstype=9p
rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'

Regards!

On Mon, 15 Apr 2024 at 14:55, Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Mon, Apr 15, 2024 at 2:52 PM Igor Mammedov <imammedo@redhat.com> wrote:
> >
> > On Sun, 14 Apr 2024 04:37:24 -0400
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> > > > Hi
> > > >
> > > > I am using virtme to do some CI around linux-media.
> > > >
> > > > Everything works as expected, but when I enable KASAN, I am starting
> > > > to get a lot of timeouts when the Method _PRT is executed. Eg:
> > > >
> > > > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > > > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > > > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > > > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> > > >
> > > > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> > > >
> > > > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > > > always executed, but it is very annoying that I have to wait more than
> > > > 5 minutes to start the vm.
> > > >
> > > > Despite not having kvm enabled, the machine is quite decent, so I
> > > > would expect that it could run that method relatively fast.
> > > >
> > > > Do you have any hint of what I should be looking at?
> >
> > The way QEMU generates _PRT haven't been changed for ages,
> > it's not likely to be a culprit.
> >
> > CCing Rafael who might have an idea why ACPI misbehaves.
>
> It looks like the method evaluation time increases with KASAN enabled
> and it gets aborted due to exceeding the evaluation time limit.
>
> Thanks!



-- 
Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Rafael J. Wysocki]

On Mon, Apr 15, 2024 at 4:18 PM Ricardo Ribalda <ribalda@google.com> wrote:
>
> Hi Igor, Hi Rafael
>
> Yes, it seems that it is just KASAN being extremely slow.
> From a completely newbie here... Is there a reason why qemu generates
> the table vs returning a precomputed one?

I have no idea, I'm not familiar with qemu internals, sorry.

Re: ACPI timeouts when enabling KASAN

[Michael S. Tsirkin]

On Mon, Apr 15, 2024 at 04:18:22PM +0200, Ricardo Ribalda wrote:
> Hi Igor, Hi Rafael
> 
> Yes, it seems that it is just KASAN being extremely slow.
> >From a completely newbie here... Is there a reason why qemu generates
> the table vs returning a precomputed one?

We never saw this being a performance bottleneck?

-- 
MST

Re: ACPI timeouts when enabling KASAN

[Igor Mammedov]

On Mon, 15 Apr 2024 16:18:22 +0200
Ricardo Ribalda <ribalda@google.com> wrote:

> Hi Igor, Hi Rafael
> 
> Yes, it seems that it is just KASAN being extremely slow.
> From a completely newbie here... Is there a reason why qemu generates
> the table vs returning a precomputed one?

it can be a pre-generated Package
like we do with ARM (example: acpi_dsdt_add_pci_route_table)

> This is the config file:
> https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> 
> And this is the qemu cli:
> 
> /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> -display none -serial chardev:console -chardev
> file,id=console,path=/proc/self/fd/2 -chardev
> stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> virtserialport,name=virtme.stdin,chardev=stdin -chardev
> file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> virtserialport,name=virtme.stdout,chardev=stdout -chardev
> file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> virtserialport,name=virtme.stderr,chardev=stderr -chardev
> file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> virtio-serial-pci -device
> virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> ./arch/x86/boot/bzImage -append 'nr_open=1048576
> virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> virtme_root_user=1 rootfstype=9p
> rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'

boots fine for me on old Xeon E5-2630v3.
Perhaps issue is that your host is too slow,
is there reason not to use KVM instead of TCG?

Alternatively you can try using q35 machine type
instead of default 'pc', it doesn't have _PRT in
simple configuration like yours.
But then running things that depend on time is not
reliable under TCG, so you might hit timeout elsewhere.

> 
> Regards!
> 
> On Mon, 15 Apr 2024 at 14:55, Rafael J. Wysocki <rafael@kernel.org> wrote:
> >
> > On Mon, Apr 15, 2024 at 2:52 PM Igor Mammedov <imammedo@redhat.com> wrote:  
> > >
> > > On Sun, 14 Apr 2024 04:37:24 -0400
> > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >  
> > > > On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:  
> > > > > Hi
> > > > >
> > > > > I am using virtme to do some CI around linux-media.
> > > > >
> > > > > Everything works as expected, but when I enable KASAN, I am starting
> > > > > to get a lot of timeouts when the Method _PRT is executed. Eg:
> > > > >
> > > > > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > > > > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > > > > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > > > > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> > > > >
> > > > > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> > > > >
> > > > > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > > > > always executed, but it is very annoying that I have to wait more than
> > > > > 5 minutes to start the vm.
> > > > >
> > > > > Despite not having kvm enabled, the machine is quite decent, so I
> > > > > would expect that it could run that method relatively fast.
> > > > >
> > > > > Do you have any hint of what I should be looking at?  
> > >
> > > The way QEMU generates _PRT haven't been changed for ages,
> > > it's not likely to be a culprit.
> > >
> > > CCing Rafael who might have an idea why ACPI misbehaves.  
> >
> > It looks like the method evaluation time increases with KASAN enabled
> > and it gets aborted due to exceeding the evaluation time limit.
> >
> > Thanks!  
> 
> 
> 

Re: ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

Hi Igor

On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:
>
> On Mon, 15 Apr 2024 16:18:22 +0200
> Ricardo Ribalda <ribalda@google.com> wrote:
>
> > Hi Igor, Hi Rafael
> >
> > Yes, it seems that it is just KASAN being extremely slow.
> > From a completely newbie here... Is there a reason why qemu generates
> > the table vs returning a precomputed one?
>
> it can be a pre-generated Package
> like we do with ARM (example: acpi_dsdt_add_pci_route_table)
>
> > This is the config file:
> > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> >
> > And this is the qemu cli:
> >
> > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > -display none -serial chardev:console -chardev
> > file,id=console,path=/proc/self/fd/2 -chardev
> > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > virtio-serial-pci -device
> > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > virtme_root_user=1 rootfstype=9p
> > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'
>
> boots fine for me on old Xeon E5-2630v3.
> Perhaps issue is that your host is too slow,
> is there reason not to use KVM instead of TCG?

I am using a e2 instance that does not support nested virtualization :(

>
> Alternatively you can try using q35 machine type
> instead of default 'pc', it doesn't have _PRT in
> simple configuration like yours.
> But then running things that depend on time is not
> reliable under TCG, so you might hit timeout elsewhere.

I will give it a try... but you are correct, if this is running this
slow I expect that nothing from my CI will work reliably.

Thanks!


>
> >
> > Regards!
> >
> > On Mon, 15 Apr 2024 at 14:55, Rafael J. Wysocki <rafael@kernel.org> wrote:
> > >
> > > On Mon, Apr 15, 2024 at 2:52 PM Igor Mammedov <imammedo@redhat.com> wrote:
> > > >
> > > > On Sun, 14 Apr 2024 04:37:24 -0400
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >
> > > > > On Fri, Apr 12, 2024 at 11:43:22PM +0200, Ricardo Ribalda wrote:
> > > > > > Hi
> > > > > >
> > > > > > I am using virtme to do some CI around linux-media.
> > > > > >
> > > > > > Everything works as expected, but when I enable KASAN, I am starting
> > > > > > to get a lot of timeouts when the Method _PRT is executed. Eg:
> > > > > >
> > > > > > [   56.335875] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > > > [   56.529826] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > > > [   56.532391] virtio-pci 0000:00:02.0: can't derive routing for PCI INT A
> > > > > > [   56.532823] virtio-pci 0000:00:02.0: PCI INT A: no GSI
> > > > > > [   86.877471] ACPI Error: Aborting method \_SB.PCI0._PRT due to
> > > > > > previous error (AE_AML_LOOP_TIMEOUT) (20230628/psparse-529)
> > > > > > [   87.073854] ACPI Error: Method execution failed \_SB.PCI0._PRT due
> > > > > > to previous error (AE_AML_LOOP_TIMEOUT) (20230628/uteval-68)
> > > > > > [   87.075550] virtio-pci 0000:00:04.0: can't derive routing for PCI INT A
> > > > > > [   87.075810] virtio-pci 0000:00:04.0: PCI INT A: no GSI
> > > > > >
> > > > > > 0000:00:04.0 and 0000:00:02.0 are virtio devices (console and 9p-filesystem)
> > > > > >
> > > > > > If I increase the timeout (ACPI_MAX_LOOP_TIMEOUT), then the Method is
> > > > > > always executed, but it is very annoying that I have to wait more than
> > > > > > 5 minutes to start the vm.
> > > > > >
> > > > > > Despite not having kvm enabled, the machine is quite decent, so I
> > > > > > would expect that it could run that method relatively fast.
> > > > > >
> > > > > > Do you have any hint of what I should be looking at?
> > > >
> > > > The way QEMU generates _PRT haven't been changed for ages,
> > > > it's not likely to be a culprit.
> > > >
> > > > CCing Rafael who might have an idea why ACPI misbehaves.
> > >
> > > It looks like the method evaluation time increases with KASAN enabled
> > > and it gets aborted due to exceeding the evaluation time limit.
> > >
> > > Thanks!
> >
> >
> >
>


-- 
Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> Hi Igor
> 
> On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:
> >
> > On Mon, 15 Apr 2024 16:18:22 +0200
> > Ricardo Ribalda <ribalda@google.com> wrote:
> >
> > > Hi Igor, Hi Rafael
> > >
> > > Yes, it seems that it is just KASAN being extremely slow.
> > > From a completely newbie here... Is there a reason why qemu generates
> > > the table vs returning a precomputed one?
> >
> > it can be a pre-generated Package
> > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> >
> > > This is the config file:
> > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > >
> > > And this is the qemu cli:
> > >
> > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > -display none -serial chardev:console -chardev
> > > file,id=console,path=/proc/self/fd/2 -chardev
> > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > virtio-serial-pci -device
> > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > virtme_root_user=1 rootfstype=9p
> > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'
> >
> > boots fine for me on old Xeon E5-2630v3.
> > Perhaps issue is that your host is too slow,
> > is there reason not to use KVM instead of TCG?
> 
> I am using a e2 instance that does not support nested virtualization :(
> 
> >
> > Alternatively you can try using q35 machine type
> > instead of default 'pc', it doesn't have _PRT in
> > simple configuration like yours.
> > But then running things that depend on time is not
> > reliable under TCG, so you might hit timeout elsewhere.
> 
> I will give it a try... but you are correct, if this is running this
> slow I expect that nothing from my CI will work reliably.

I'm really interested to see if q35 helps here. If that's the case maybe
we should default to q35 in virtme-ng when KVM isn't available (even if
on my box q35 is actually slower than the default pc, so in that case we
may need to come up with some logic to pick the right machine type).

Thanks,
-Andrea

Re: ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

Hi Igor, Hi Andrea


On Tue, 16 Apr 2024 at 14:07, Andrea Righi <andrea.righi@canonical.com> wrote:
>
> On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> > Hi Igor
> >
> > On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:
> > >
> > > On Mon, 15 Apr 2024 16:18:22 +0200
> > > Ricardo Ribalda <ribalda@google.com> wrote:
> > >
> > > > Hi Igor, Hi Rafael
> > > >
> > > > Yes, it seems that it is just KASAN being extremely slow.
> > > > From a completely newbie here... Is there a reason why qemu generates
> > > > the table vs returning a precomputed one?
> > >
> > > it can be a pre-generated Package
> > > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> > >
> > > > This is the config file:
> > > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > > >
> > > > And this is the qemu cli:
> > > >
> > > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > > -display none -serial chardev:console -chardev
> > > > file,id=console,path=/proc/self/fd/2 -chardev
> > > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > > virtio-serial-pci -device
> > > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > > virtme_root_user=1 rootfstype=9p
> > > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'
> > >
> > > boots fine for me on old Xeon E5-2630v3.
> > > Perhaps issue is that your host is too slow,
> > > is there reason not to use KVM instead of TCG?
> >
> > I am using a e2 instance that does not support nested virtualization :(
> >
> > >
> > > Alternatively you can try using q35 machine type
> > > instead of default 'pc', it doesn't have _PRT in
> > > simple configuration like yours.
> > > But then running things that depend on time is not
> > > reliable under TCG, so you might hit timeout elsewhere.
> >
> > I will give it a try... but you are correct, if this is running this
> > slow I expect that nothing from my CI will work reliably.
>
> I'm really interested to see if q35 helps here. If that's the case maybe
> we should default to q35 in virtme-ng when KVM isn't available (even if
> on my box q35 is actually slower than the default pc, so in that case we
> may need to come up with some logic to pick the right machine type).
>

Adding
--qemu-opts -machine q35
to virtme-run did the trick for me. It boots much faster

Not sure if we should blindly default to that machine type or not. My
usecase might not be very common after all


Thanks!

> Thanks,
> -Andrea



-- 
Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Wed, Apr 17, 2024 at 02:13:57PM +0200, Ricardo Ribalda wrote:
> Hi Igor, Hi Andrea
> 
> 
> On Tue, 16 Apr 2024 at 14:07, Andrea Righi <andrea.righi@canonical.com> wrote:
> >
> > On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> > > Hi Igor
> > >
> > > On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:
> > > >
> > > > On Mon, 15 Apr 2024 16:18:22 +0200
> > > > Ricardo Ribalda <ribalda@google.com> wrote:
> > > >
> > > > > Hi Igor, Hi Rafael
> > > > >
> > > > > Yes, it seems that it is just KASAN being extremely slow.
> > > > > From a completely newbie here... Is there a reason why qemu generates
> > > > > the table vs returning a precomputed one?
> > > >
> > > > it can be a pre-generated Package
> > > > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> > > >
> > > > > This is the config file:
> > > > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > > > >
> > > > > And this is the qemu cli:
> > > > >
> > > > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > > > -display none -serial chardev:console -chardev
> > > > > file,id=console,path=/proc/self/fd/2 -chardev
> > > > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > > > virtio-serial-pci -device
> > > > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > > > virtme_root_user=1 rootfstype=9p
> > > > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'
> > > >
> > > > boots fine for me on old Xeon E5-2630v3.
> > > > Perhaps issue is that your host is too slow,
> > > > is there reason not to use KVM instead of TCG?
> > >
> > > I am using a e2 instance that does not support nested virtualization :(
> > >
> > > >
> > > > Alternatively you can try using q35 machine type
> > > > instead of default 'pc', it doesn't have _PRT in
> > > > simple configuration like yours.
> > > > But then running things that depend on time is not
> > > > reliable under TCG, so you might hit timeout elsewhere.
> > >
> > > I will give it a try... but you are correct, if this is running this
> > > slow I expect that nothing from my CI will work reliably.
> >
> > I'm really interested to see if q35 helps here. If that's the case maybe
> > we should default to q35 in virtme-ng when KVM isn't available (even if
> > on my box q35 is actually slower than the default pc, so in that case we
> > may need to come up with some logic to pick the right machine type).
> >
> 
> Adding
> --qemu-opts -machine q35
> to virtme-run did the trick for me. It boots much faster
> 
> Not sure if we should blindly default to that machine type or not. My
> usecase might not be very common after all

Hm... I'll just add a note to the troubleshooting section for now. I'm
not sure if there's a reliable way to automatically detect when it's
better to use q35 or default pc.

Thanks,
-Andrea

> 
> 
> Thanks!
> 
> > Thanks,
> > -Andrea
> 
> 
> 
> -- 
> Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Igor Mammedov]

On Tue, 16 Apr 2024 14:07:08 +0200
Andrea Righi <andrea.righi@canonical.com> wrote:

> On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> > Hi Igor
> > 
> > On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:  
> > >
> > > On Mon, 15 Apr 2024 16:18:22 +0200
> > > Ricardo Ribalda <ribalda@google.com> wrote:
> > >  
> > > > Hi Igor, Hi Rafael
> > > >
> > > > Yes, it seems that it is just KASAN being extremely slow.
> > > > From a completely newbie here... Is there a reason why qemu generates
> > > > the table vs returning a precomputed one?  
> > >
> > > it can be a pre-generated Package
> > > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> > >  
> > > > This is the config file:
> > > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > > >
> > > > And this is the qemu cli:
> > > >
> > > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > > -display none -serial chardev:console -chardev
> > > > file,id=console,path=/proc/self/fd/2 -chardev
> > > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > > virtio-serial-pci -device
> > > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > > virtme_root_user=1 rootfstype=9p
> > > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'  
> > >
> > > boots fine for me on old Xeon E5-2630v3.
> > > Perhaps issue is that your host is too slow,
> > > is there reason not to use KVM instead of TCG?  
> > 
> > I am using a e2 instance that does not support nested virtualization :(
> >   
> > >
> > > Alternatively you can try using q35 machine type
> > > instead of default 'pc', it doesn't have _PRT in
> > > simple configuration like yours.
> > > But then running things that depend on time is not
> > > reliable under TCG, so you might hit timeout elsewhere.  
> > 
> > I will give it a try... but you are correct, if this is running this
> > slow I expect that nothing from my CI will work reliably.  
> 
> I'm really interested to see if q35 helps here. If that's the case maybe
> we should default to q35 in virtme-ng when KVM isn't available (even if
> on my box q35 is actually slower than the default pc, so in that case we
> may need to come up with some logic to pick the right machine type).

it might be interesting to find out why q35 is slower (it shouldn't be)
With above config one can put all devices on hostbridge as integrated endpoints
which roughly will be the same as PCI topo in 'pc' machine)

another thing that might help is adding '-cpu max' instead of default
qemu64 cpu model.

> Thanks,
> -Andrea
> 

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Wed, Apr 17, 2024 at 02:55:44PM +0200, Igor Mammedov wrote:
> On Tue, 16 Apr 2024 14:07:08 +0200
> Andrea Righi <andrea.righi@canonical.com> wrote:
> 
> > On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> > > Hi Igor
> > > 
> > > On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:  
> > > >
> > > > On Mon, 15 Apr 2024 16:18:22 +0200
> > > > Ricardo Ribalda <ribalda@google.com> wrote:
> > > >  
> > > > > Hi Igor, Hi Rafael
> > > > >
> > > > > Yes, it seems that it is just KASAN being extremely slow.
> > > > > From a completely newbie here... Is there a reason why qemu generates
> > > > > the table vs returning a precomputed one?  
> > > >
> > > > it can be a pre-generated Package
> > > > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> > > >  
> > > > > This is the config file:
> > > > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > > > >
> > > > > And this is the qemu cli:
> > > > >
> > > > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > > > -display none -serial chardev:console -chardev
> > > > > file,id=console,path=/proc/self/fd/2 -chardev
> > > > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > > > virtio-serial-pci -device
> > > > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > > > virtme_root_user=1 rootfstype=9p
> > > > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'  
> > > >
> > > > boots fine for me on old Xeon E5-2630v3.
> > > > Perhaps issue is that your host is too slow,
> > > > is there reason not to use KVM instead of TCG?  
> > > 
> > > I am using a e2 instance that does not support nested virtualization :(
> > >   
> > > >
> > > > Alternatively you can try using q35 machine type
> > > > instead of default 'pc', it doesn't have _PRT in
> > > > simple configuration like yours.
> > > > But then running things that depend on time is not
> > > > reliable under TCG, so you might hit timeout elsewhere.  
> > > 
> > > I will give it a try... but you are correct, if this is running this
> > > slow I expect that nothing from my CI will work reliably.  
> > 
> > I'm really interested to see if q35 helps here. If that's the case maybe
> > we should default to q35 in virtme-ng when KVM isn't available (even if
> > on my box q35 is actually slower than the default pc, so in that case we
> > may need to come up with some logic to pick the right machine type).
> 
> it might be interesting to find out why q35 is slower (it shouldn't be)

Nevermind, I was comparing native kvm vs q35, of course it was slower...

> With above config one can put all devices on hostbridge as integrated endpoints
> which roughly will be the same as PCI topo in 'pc' machine)
> 
> another thing that might help is adding '-cpu max' instead of default
> qemu64 cpu model.

Doing the proper comparison (disabling kvm), adding '-cpu max' to the
equation and measuring the boot time of multiple virtme-ng runs, gives
me the following result (average of 10 runs):

                     machine
              +----------------
              | default     q35
     ---------+----------------
cpu  |default |     13s     11s
     |max     |     15s     14s

I've tried a couple of kernel configs and I get similar results.

In the scope of virtme-ng (optimize boot time) I'd say that it'd makes
sense to use '-machine q35' and default cpu settings when kvm is
unavailable.

Ricardo, do you see similar results?

Thanks,
-Andrea

Re: ACPI timeouts when enabling KASAN

[Ricardo Ribalda]

Hi Andrea

On Wed, 17 Apr 2024 at 16:38, Andrea Righi <andrea.righi@canonical.com> wrote:
>
> On Wed, Apr 17, 2024 at 02:55:44PM +0200, Igor Mammedov wrote:
> > On Tue, 16 Apr 2024 14:07:08 +0200
> > Andrea Righi <andrea.righi@canonical.com> wrote:
> >
> > > On Tue, Apr 16, 2024 at 01:36:40PM +0200, Ricardo Ribalda wrote:
> > > > Hi Igor
> > > >
> > > > On Tue, 16 Apr 2024 at 13:33, Igor Mammedov <imammedo@redhat.com> wrote:
> > > > >
> > > > > On Mon, 15 Apr 2024 16:18:22 +0200
> > > > > Ricardo Ribalda <ribalda@google.com> wrote:
> > > > >
> > > > > > Hi Igor, Hi Rafael
> > > > > >
> > > > > > Yes, it seems that it is just KASAN being extremely slow.
> > > > > > From a completely newbie here... Is there a reason why qemu generates
> > > > > > the table vs returning a precomputed one?
> > > > >
> > > > > it can be a pre-generated Package
> > > > > like we do with ARM (example: acpi_dsdt_add_pci_route_table)
> > > > >
> > > > > > This is the config file:
> > > > > > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/testdata/virtme/virtme.config?ref_type=heads
> > > > > >
> > > > > > And this is the qemu cli:
> > > > > >
> > > > > > /usr/bin/qemu-system-x86_64 -m 4G -fsdev
> > > > > > local,id=virtfs3,path=/,security_model=none,readonly=on,multidevs=remap
> > > > > > -device virtio-9p-pci,fsdev=virtfs3,mount_tag=/dev/root -device
> > > > > > i6300esb,id=watchdog0 -parallel none -net none -smp 2 -vga none
> > > > > > -display none -serial chardev:console -chardev
> > > > > > file,id=console,path=/proc/self/fd/2 -chardev
> > > > > > stdio,id=stdin,signal=on,mux=off -device virtio-serial-pci -device
> > > > > > virtserialport,name=virtme.stdin,chardev=stdin -chardev
> > > > > > file,id=stdout,path=/proc/self/fd/1 -device virtio-serial-pci -device
> > > > > > virtserialport,name=virtme.stdout,chardev=stdout -chardev
> > > > > > file,id=stderr,path=/proc/self/fd/2 -device virtio-serial-pci -device
> > > > > > virtserialport,name=virtme.stderr,chardev=stderr -chardev
> > > > > > file,id=dev_stdout,path=/proc/self/fd/1 -device virtio-serial-pci
> > > > > > -device virtserialport,name=virtme.dev_stdout,chardev=dev_stdout
> > > > > > -chardev file,id=dev_stderr,path=/proc/self/fd/2 -device
> > > > > > virtio-serial-pci -device
> > > > > > virtserialport,name=virtme.dev_stderr,chardev=dev_stderr -chardev
> > > > > > file,id=ret,path=/tmp/virtme_retefeobj4f -device virtio-serial-pci
> > > > > > -device virtserialport,name=virtme.ret,chardev=ret -no-reboot -kernel
> > > > > > ./arch/x86/boot/bzImage -append 'nr_open=1048576
> > > > > > virtme_link_mods=/builds/linux-media/media-staging/.virtme_mods/lib/modules/0.0.0
> > > > > > console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1
> > > > > > virtme.exec=`c2ggL21lZGlhLWNpL3Rlc3RkYXRhL3ZpcnRtZS90ZXN0LnNoIC9tZWRpYS1jaS90aGlyZF9wYXJ0eS92NGwtdXRpbHMgLTMy`
> > > > > > virtme_root_user=1 rootfstype=9p
> > > > > > rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> > > > > > ro init=/usr/lib/python3/dist-packages/virtme/guest/virtme-init'
> > > > >
> > > > > boots fine for me on old Xeon E5-2630v3.
> > > > > Perhaps issue is that your host is too slow,
> > > > > is there reason not to use KVM instead of TCG?
> > > >
> > > > I am using a e2 instance that does not support nested virtualization :(
> > > >
> > > > >
> > > > > Alternatively you can try using q35 machine type
> > > > > instead of default 'pc', it doesn't have _PRT in
> > > > > simple configuration like yours.
> > > > > But then running things that depend on time is not
> > > > > reliable under TCG, so you might hit timeout elsewhere.
> > > >
> > > > I will give it a try... but you are correct, if this is running this
> > > > slow I expect that nothing from my CI will work reliably.
> > >
> > > I'm really interested to see if q35 helps here. If that's the case maybe
> > > we should default to q35 in virtme-ng when KVM isn't available (even if
> > > on my box q35 is actually slower than the default pc, so in that case we
> > > may need to come up with some logic to pick the right machine type).
> >
> > it might be interesting to find out why q35 is slower (it shouldn't be)
>
> Nevermind, I was comparing native kvm vs q35, of course it was slower...
>
> > With above config one can put all devices on hostbridge as integrated endpoints
> > which roughly will be the same as PCI topo in 'pc' machine)
> >
> > another thing that might help is adding '-cpu max' instead of default
> > qemu64 cpu model.
>
> Doing the proper comparison (disabling kvm), adding '-cpu max' to the
> equation and measuring the boot time of multiple virtme-ng runs, gives
> me the following result (average of 10 runs):
>
>                      machine
>               +----------------
>               | default     q35
>      ---------+----------------
> cpu  |default |     13s     11s
>      |max     |     15s     14s
>
> I've tried a couple of kernel configs and I get similar results.
>
> In the scope of virtme-ng (optimize boot time) I'd say that it'd makes
> sense to use '-machine q35' and default cpu settings when kvm is
> unavailable.
>
> Ricardo, do you see similar results?

I see even more difference between q35 and default.
These are my kernel options:
https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/test-virtme.sh?ref_type=heads#L27



I see an issue to automatically set the machine and it is that  we
would not be able to override it with something like:

virtme-run .......... --qemu-opts -machine q35



If this patch lands in qemu we might be able to ignore all these:
https://lore.kernel.org/qemu-devel/20240417135608.2613586-1-ribalda@chromium.org/T/#u

Regards!

>
> Thanks,
> -Andrea



-- 
Ricardo Ribalda

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Wed, Apr 17, 2024 at 06:12:10PM +0200, Ricardo Ribalda wrote:
...
> > Doing the proper comparison (disabling kvm), adding '-cpu max' to the
> > equation and measuring the boot time of multiple virtme-ng runs, gives
> > me the following result (average of 10 runs):
> >
> >                      machine
> >               +----------------
> >               | default     q35
> >      ---------+----------------
> > cpu  |default |     13s     11s
> >      |max     |     15s     14s
> >
> > I've tried a couple of kernel configs and I get similar results.
> >
> > In the scope of virtme-ng (optimize boot time) I'd say that it'd makes
> > sense to use '-machine q35' and default cpu settings when kvm is
> > unavailable.
> >
> > Ricardo, do you see similar results?
> 
> I see even more difference between q35 and default.
> These are my kernel options:
> https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/test-virtme.sh?ref_type=heads#L27

Ok, I'll run some tests with these options as well.

> I see an issue to automatically set the machine and it is that  we
> would not be able to override it with something like:
> 
> virtme-run .......... --qemu-opts -machine q35

If I'm not wrong the last one should override the previous ones,
so --qemu-opts should still win over the default.

> 
> If this patch lands in qemu we might be able to ignore all these:
> https://lore.kernel.org/qemu-devel/20240417135608.2613586-1-ribalda@chromium.org/T/#u

Yep, this is much better, thanks for this fix. Let's keep the defaults
for now in virtme-ng, I'll just add a note to the troubleshooting
section.

Thanks!
-Andrea

Re: ACPI timeouts when enabling KASAN

[Andrea Righi]

On Wed, Apr 17, 2024 at 09:38:26PM +0200, Andrea Righi wrote:
> On Wed, Apr 17, 2024 at 06:12:10PM +0200, Ricardo Ribalda wrote:
> ...
> > > Doing the proper comparison (disabling kvm), adding '-cpu max' to the
> > > equation and measuring the boot time of multiple virtme-ng runs, gives
> > > me the following result (average of 10 runs):
> > >
> > >                      machine
> > >               +----------------
> > >               | default     q35
> > >      ---------+----------------
> > > cpu  |default |     13s     11s
> > >      |max     |     15s     14s
> > >
> > > I've tried a couple of kernel configs and I get similar results.
> > >
> > > In the scope of virtme-ng (optimize boot time) I'd say that it'd makes
> > > sense to use '-machine q35' and default cpu settings when kvm is
> > > unavailable.
> > >
> > > Ricardo, do you see similar results?
> > 
> > I see even more difference between q35 and default.
> > These are my kernel options:
> > https://gitlab.freedesktop.org/linux-media/media-ci/-/blob/main/test-virtme.sh?ref_type=heads#L27
> 
> Ok, I'll run some tests with these options as well.
> 
> > I see an issue to automatically set the machine and it is that  we
> > would not be able to override it with something like:
> > 
> > virtme-run .......... --qemu-opts -machine q35
> 
> If I'm not wrong the last one should override the previous ones,
> so --qemu-opts should still win over the default.
> 
> > 
> > If this patch lands in qemu we might be able to ignore all these:
> > https://lore.kernel.org/qemu-devel/20240417135608.2613586-1-ribalda@chromium.org/T/#u
> 
> Yep, this is much better, thanks for this fix. Let's keep the defaults
> for now in virtme-ng, I'll just add a note to the troubleshooting
> section.

I actually changed my mind and merged this in virtme-ng:
https://github.com/arighi/virtme-ng/pull/110

I did more tests and in terms of boot time using the q35 arch really
seems to systematically improve performance (I tested this across
multiple hardware and kernel configs).

It also seems to improve the CI run test (that is executed as a github
workflow, where kvm isn't available). As we can see here:

 - default arch, run test 34s:
   https://github.com/arighi/virtme-ng/actions/runs/8935862788/job/24545125258

 - q35 arch, run test 28s:
   https://github.com/arighi/virtme-ng/actions/runs/8936234891/job/24546173523

Ricardo's fix can help to mitigate the issue with ACPI, but for now I
think using q35 when kvm isn't available can help to speed up tests in
similar CI environments.

Thanks,
-Andrea