GPU Pass-through need help.

GPU Pass-through need help.

[Prasad Joshi]

Hello All,

I am trying to pass-through a GPU PCIe ATI Radeon to VM. Here is a log on VM.

prasad@prasad-virtual-machine:~$ dmesg | grep -i -e drm -e radeon
[    2.162294] [drm] Initialized drm 1.1.0 20060810
[    2.459594] [drm] radeon defaulting to kernel modesetting.
[    2.459596] [drm] radeon kernel modesetting enabled.
[    2.766698] radeon 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10
(level, high) -> IRQ 10
[    2.766734] radeon 0000:00:04.0: setting latency timer to 64
[    2.783512] [drm] initializing kernel modesetting (RV380 0x1002:0x5B64).
[    2.792407] [drm] register mmio base: 0x40000000
[    2.792408] [drm] register mmio size: 65536
[    2.797177] [drm] Generation 2 PCI interface, using max accessible memory
[    2.797275] radeon 0000:00:04.0: VRAM: 128M 0x00000000F8000000 -
0x00000000FFFFFFFF (128M used)
[    2.797284] radeon 0000:00:04.0: GTT: 512M 0x00000000D8000000 -
0x00000000F7FFFFFF
[    2.798370] radeon 0000:00:04.0: irq 40 for MSI/MSI-X
[    2.870703] radeon 0000:00:04.0: radeon: using MSI.
[    3.151162] [drm] radeon: irq initialized.
[    3.151539] [drm] Detected VRAM RAM=128M, BAR=128M
[    3.151541] [drm] RAM width 128bits DDR
[    3.151610] [drm] radeon: 128M of VRAM memory ready
[    3.151611] [drm] radeon: 512M of GTT memory ready.
[    3.151627] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    3.152701] [drm] radeon: 1 quad pipes, 1 Z pipes initialized.
[    3.544943] [drm] PCIE GART of 512M enabled (table at 0xF8040000).
[    3.548479] radeon 0000:00:04.0: WB enabled
[    3.549020] [drm] Loading R300 Microcode
[    3.554278] [drm] radeon: ring at 0x00000000D8001000
[    3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
[    3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22).
[    3.782542] radeon 0000:00:04.0: failled initializing CP (-22).
[    3.784006] radeon 0000:00:04.0: Disabling GPU acceleration
[    3.793709] [drm] radeon: cp finalized
[    3.834875] radeon 0000:00:04.0: ffff8800320b4c00 unpin not necessary
[    3.841759] [drm] Radeon Display Connectors
[    3.841766] [drm] Connector 0:
[    3.841771] [drm]   VGA
[    3.841777] [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[    3.841781] [drm]   Encoders:
[    3.841785] [drm]     CRT1: INTERNAL_DAC1
[    3.841788] [drm] Connector 1:
[    3.841791] [drm]   DVI-I
[    3.841794] [drm]   HPD1
[    3.841799] [drm]   DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[    3.841803] [drm]   Encoders:
[    3.841806] [drm]     CRT2: INTERNAL_DAC2
[    3.841809] [drm]     DFP1: INTERNAL_TMDS1
[    6.158753] [drm] fb mappable at 0xF8040000
[    6.158756] [drm] vram apper at 0xF8000000
[    6.158757] [drm] size 5242880
[    6.158758] [drm] fb depth is 24
[    6.158759] [drm]    pitch is 5120
[   13.914196] fb0: radeondrmfb frame buffer device
[   13.914198] drm: registered panic notifier
[   13.914683] [drm] Initialized radeon 2.7.0 20080528 for
0000:00:04.0 on minor 0

I was previously seeing a problem with the ROM BIOS, but I could solve
it by passing a correct BIOS file in function assigned_initfn. For now
I am using hardcoded file name in the code.

Can anyone help me with debugging of following errors?
[    3.778476] [drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
[    3.781010] [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22).

3444     WREG32(scratch, 0xCAFEDEAD);
3445     r = radeon_ring_lock(rdev, 2);
3446     if (r) {
3447         DRM_ERROR("radeon: cp failed to lock ring (%d).\n", r);
3448         radeon_scratch_free(rdev, scratch);
3449         return r;
3450     }
3451     radeon_ring_write(rdev, PACKET0(scratch, 0));
3452     radeon_ring_write(rdev, 0xDEADBEEF);
3453     radeon_ring_unlock_commit(rdev);
3454     for (i = 0; i < rdev->usec_timeout; i++) {
3455         tmp = RREG32(scratch);
3456         if (tmp == 0xDEADBEEF) {
3457             break;
3458         }
3459         DRM_UDELAY(1);
3460     }
3461     if (i < rdev->usec_timeout) {
3462         DRM_INFO("ring test succeeded in %d usecs\n", i);
3463     } else {
3464         DRM_ERROR("radeon: ring test failed (sracth(0x%04X)=0x%08X)\n",
3465               scratch, tmp);
3466         r = -EINVAL;
3467     }

It seems like, on line 3452 code writes DEADBEEF ins some ring and on
line 3455-3456 it reads whatever has been written on ring and it fails
in this case as it reads back 0xCAFEDEAD

How can I debug this VM problem?

BTW, the assigned device is listed in lspci output

prasad@prasad-virtual-machine:~$ lspci  | grep ATI
00:04.0 Display controller: ATI Technologies Inc RV370 5B64 [FireGL
V3100 (PCIE)] (rev 80)

I also noticed following log on the host machine.
[ 6042.798880] kvm: 1566: cpu0 unimplemented perfctr wrmsr:
0xc0010004data 0xabcd
[ 6104.970165] pci-stub 0000:02:00.0: restoring config space at
offset0x1 (was 0x100400, writing 0x100007)
[ 6105.530290] assign device 0:2:0.0
[ 6105.530351] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6105.740225] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6105.870203] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6110.575658] kvm: 2264: cpu0 unimplemented perfctr wrmsr:
0xc0010004data 0xabcd
[ 6110.874223] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6113.230216] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6113.550222] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X
[ 6113.750205] pci-stub 0000:02:00.0: irq 88 for MSI/MSI-X

Re: GPU Pass-through need help.

[Konstantin Khlebnikov]

Prasad Joshi wrote:

> I was previously seeing a problem with the ROM BIOS, but I could solve
> it by passing a correct BIOS file in function assigned_initfn. For now
> I am using hardcoded file name in the code.

Did you recheck video bios inside guest? (for example in sysfs)


If this is bootstrap vga, guest linux kernel will use bios copy placed at 0xc0000.
But in my setup qemu can only pass first 32k of option-roms,
due to curruption at seabios shadow-copying stage.
(seems like because improper i440fx PAM registers emulation)

Re: GPU Pass-through need help.

[Prasad Joshi]

On Wed, Feb 9, 2011 at 4:33 PM, Konstantin Khlebnikov
<khlebnikov@openvz.org> wrote:
>
> Prasad Joshi wrote:
>
>> I was previously seeing a problem with the ROM BIOS, but I could solve
>> it by passing a correct BIOS file in function assigned_initfn. For now
>> I am using hardcoded file name in the code.
>
> Did you recheck video bios inside guest? (for example in sysfs)
>

For GPU cards on my machine, reading from the /sys/bus/pci/devices/<device>/
rom always give error 'Invalid Argument'. I had to extract the BIOS
using some utilities. The BIOS seems to be correct as I have been able
to pass-through a old Nvidia (NVS 290) and old ATI (RV370 FireGL
V3100) card to Ubuntu Guest OS.

>
> If this is bootstrap vga, guest linux kernel will use bios copy placed at 0xc0000.
> But in my setup qemu can only pass first 32k of option-roms,
> due to curruption at seabios shadow-copying stage.
> (seems like because improper i440fx PAM registers emulation)

I am not sure if it is related to the problem you are facing. The
current SeaBIOS has very limited pci memory space from 0xf0000000  to
0xfec00000 (236MB). This includes everything, prefectable and
non-prefecthable memory. The range is very small for VGA cards to
work. I have modified the memory range on my machine for prefectable
memory.

There is one bug in qemu-kvm code, which acidently detects the
prefectable memory as non-prefectable. I had sent a patch to correct
this behavior yesterday. Hope it gets accepted.

Thanks and Regards,
Prasad

Re: GPU Pass-through need help.

[Konstantin Khlebnikov]

Prasad Joshi wrote:
> On Wed, Feb 9, 2011 at 4:33 PM, Konstantin Khlebnikov
> <khlebnikov@openvz.org <mailto:khlebnikov@openvz.org>> wrote:
>
>     Prasad Joshi wrote:
>
>         I was previously seeing a problem with the ROM BIOS, but I could
>         solve
>         it by passing a correct BIOS file in function assigned_initfn.
>         For now
>         I am using hardcoded file name in the code.
>
>
>     Did you recheck video bios inside guest? (for example in sysfs)
>
>
> For GPU cards on my machine, reading from the
> /sys/bus/pci/devices/<device>/rom always give error 'Invalid Argument'.

Yep, rom in sysfs must be enabled before via 'echo 1 > rom'
If rom was corrupted you will see warning in dmesg.

> I had to extract the BIOS using some utilities. The BIOS seems to be
> correct as I have been able to pass-through a old Nvidia (NVS 290) and
> old ATI (RV370 FireGL V3100) card to Ubuntu Guest OS.
>
>
>     If this is bootstrap vga, guest linux kernel will use bios copy
>     placed at 0xc0000.
>     But in my setup qemu can only pass first 32k of option-roms,
>     due to curruption at seabios shadow-copying stage.
>     (seems like because improper i440fx PAM registers emulation)
>
>
> I am not sure if it is related to the problem you are facing. The
> current SeaBIOS has very limited pci memory space from 0xf0000000  to
> 0xfec00000 (236MB). This includes everything, prefectable and
> non-prefecthable memory. The range is very small for VGA cards to work.
> I have modified the memory range on my machine for prefectable memory.

But if in your guest /sys/bus/pci/devices/<device>/boot_vga == 1
linux will use shadow copy from 0xc0000 instead of pci rom bar

>
> There is one bug in qemu-kvm code, which acidently detects the
> prefectable memory as non-prefectable. I had sent a patch to correct
> this behavior yesterday. Hope it gets accepted.

Yes, I saw him, IORESOURCE_PREFETCH was accidentally changed in linux kernel.
Maybe better to detect it via opening and mapping /resorce<id>_wc instad of /resource<id>,
it exist only for prefechable resources. Plus qemu will get write-combined mapping.

>
> Thanks and Regards,
> Prasad