* PCI assignement with nvidia K1: RmInitAdapter failed
@ 2013-07-09 9:18 Guillaume Thouvenin
2013-07-09 10:15 ` Guillaume Thouvenin
0 siblings, 1 reply; 7+ messages in thread
From: Guillaume Thouvenin @ 2013-07-09 9:18 UTC (permalink / raw)
To: kvm
Hello,
I'd like to test a nvidia K1 with pci assignement by using the pci-stub
module or the vfio module. I'm running an ubuntu raring as the host and
a version 1.4.0 of qemu-system-x86 1.4.0. The nvidia K1 has a grid
architecture and I see four K1 on one card.
I have unbind the device from the old driver and bind it to the
vfio-pci. Then started the VM with the following command:
$ sudo sh -c "qemu-system-x86_64 -M q35 -m 4096 --enable-kvm \
-net nic,model=virtio,macaddr=52:54:00:82:69:75 -net tap,ifname=tap0 \
-drive file=/home/thouveng/ubu13.04_amd64_base.qcow2,if=virtio \
-device vfio-pci,host=87:00.0"
Then I logged into the VM and I can see the nvidia device:
$ lspci | grep -i nvi
00:03.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID
K1] (rev a1)
I installed the nvidia driver that I downloaded from their website.
Every things seems ok. Now to validate that everyhting is working I
tried to run the command "nvidia-smi -q" and I got the following error:
$ nvidia-smi -q
NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
Unable to determine the device handle for GPU 0000:00:03.0: Unknown Error
and in the syslog of the guest I can see:
[ 1502.459123] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[ 1502.459150] NVRM: rm_init_adapter(0) failed
I checked the BAR registers in the guest but as far as I understand
them, memory regions seem correct:
$ lspci -v
00:03.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID
K1] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 099d
Flags: bus master, fast devsel, latency 0, IRQ 23
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Memory at f0000000 (64-bit, prefetchable) [size=128M]
Memory at fa000000 (64-bit, prefetchable) [size=32M]
I/O ports at c000 [size=128]
Expansion ROM at fe000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
I also tried to do the assignement with pci-stub but I have the same
problem.
The onlu clue that I could find on the web was on the page
http://us.download.nvidia.com/XFree86/Linux-x86/304.60/README/commonproblems.html where they said that the same problem occured when the "VBIOS fail to load on my Optimus system". They said about this error "Such problems are typically beyond the control of the NVIDIA driver, which relies on proper cooperation of ACPI and the System BIOS to retrieve important information about the GPU, including the Video
BIOS."
I don't really understand the problem. I will really appreciate any help to
troubleshoot the problem and any links that can help are more than welcome :).
I will also appreciate any comment about a succesfull story with PCI
assignement with a K1. I don't have all of them but I maybe will have
the opportunity to test them so any comments are welcome :).
Best regards,
Guillaume
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-09 9:18 PCI assignement with nvidia K1: RmInitAdapter failed Guillaume Thouvenin
@ 2013-07-09 10:15 ` Guillaume Thouvenin
2013-07-09 17:52 ` Alex Williamson
2013-07-10 15:30 ` Guillaume Thouvenin
0 siblings, 2 replies; 7+ messages in thread
From: Guillaume Thouvenin @ 2013-07-09 10:15 UTC (permalink / raw)
To: Guillaume Thouvenin; +Cc: kvm
Guillaume Thouvenin <guillaume.thouvenin@polymtl.ca> a écrit :
> I'd like to test a nvidia K1 with pci assignement by using the
> pci-stub module or the vfio module. I'm running an ubuntu raring as
> the host and a version 1.4.0 of qemu-system-x86 1.4.0.
I just tested with QEMU emulator version 1.5.50 that I compiled from
git but I have the same issue. The kernel running on the host is a
3.8.0-23-generic that comes with ubuntu.
I also tried to reset the card before binding it but I got the same issue:
NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
Unable to determine the device handle for GPU 0000:00:03.0: Unknown Error
even if in /dev I have:
$ ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 juil. 9 12:09 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 juil. 9 12:09 /dev/nvidiactl
Regards,
Guillaume
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-09 10:15 ` Guillaume Thouvenin
@ 2013-07-09 17:52 ` Alex Williamson
2013-07-10 12:54 ` Guillaume Thouvenin
2013-07-10 15:30 ` Guillaume Thouvenin
1 sibling, 1 reply; 7+ messages in thread
From: Alex Williamson @ 2013-07-09 17:52 UTC (permalink / raw)
To: Guillaume Thouvenin; +Cc: kvm
On Tue, 2013-07-09 at 12:15 +0200, Guillaume Thouvenin wrote:
> Guillaume Thouvenin <guillaume.thouvenin@polymtl.ca> a écrit :
>
> > I'd like to test a nvidia K1 with pci assignement by using the
> > pci-stub module or the vfio module. I'm running an ubuntu raring as
> > the host and a version 1.4.0 of qemu-system-x86 1.4.0.
>
> I just tested with QEMU emulator version 1.5.50 that I compiled from
> git but I have the same issue. The kernel running on the host is a
> 3.8.0-23-generic that comes with ubuntu.
>
> I also tried to reset the card before binding it but I got the same issue:
>
> NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
> Unable to determine the device handle for GPU 0000:00:03.0: Unknown Error
>
> even if in /dev I have:
>
> $ ls -l /dev/nvidia*
> crw-rw-rw- 1 root root 195, 0 juil. 9 12:09 /dev/nvidia0
> crw-rw-rw- 1 root root 195, 255 juil. 9 12:09 /dev/nvidiactl
Are you sure that nvidia-smi is relevant to the K1/K2 devices? Does it
work on the host? I wouldn't be surprised if a system management
interface tries to use backdoors that are not available in a VM. Are
there other tests you can do to check whether the device is otherwise
available? Also, I'm curious to see what these cards look like, could
you provide an 'sudo lspci -vvv' of the host system? Thanks,
Alex
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-09 17:52 ` Alex Williamson
@ 2013-07-10 12:54 ` Guillaume Thouvenin
2013-07-10 15:25 ` Alex Williamson
0 siblings, 1 reply; 7+ messages in thread
From: Guillaume Thouvenin @ 2013-07-10 12:54 UTC (permalink / raw)
To: Alex Williamson; +Cc: kvm
Alex Williamson <alex.williamson@redhat.com> a écrit :
> Are you sure that nvidia-smi is relevant to the K1/K2 devices? Does it
> work on the host?
Yes it works on the host and I have some information like power
consumption, temperature of the GPU, etc...
> Are
> there other tests you can do to check whether the device is otherwise
> available?
I also tried to start an X server but I got the same error reported in
the syslog: [ 435.745673] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[ 435.745695] NVRM: rm_init_adapter(0) failed
And in the xorg.log I have:
[ 423.624] (==) NVIDIA(0): Depth 24, (==) framebuffer bpp 32
[ 423.624] (==) NVIDIA(0): RGB weight 888
[ 423.624] (==) NVIDIA(0): Default visual is TrueColor
[ 423.624] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
[ 423.624] (**) NVIDIA(0): Option "NoLogo" "true"
[ 423.624] (**) NVIDIA(0): Option "UseDisplayDevice" "none"
[ 423.624] (**) NVIDIA(0): Enabling 2D acceleration
[ 423.624] (**) NVIDIA(0): Option "UseDisplayDevice" set to "none";
enabling NoScanout
[ 423.624] (**) NVIDIA(0): mode
[ 435.746] (EE) NVIDIA(0): Failed to initialize the NVIDIA GPU at
PCI:0:3:0. Please
[ 435.746] (EE) NVIDIA(0): check your system's kernel log for
additional error
[ 435.746] (EE) NVIDIA(0): messages and refer to Chapter 8:
Common Problems in the
[ 435.746] (EE) NVIDIA(0): README for additional information.
[ 435.746] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device!
[ 435.746] (EE) NVIDIA(0): Failing initialization of X screen 0
Regards,
Guillaume
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-10 12:54 ` Guillaume Thouvenin
@ 2013-07-10 15:25 ` Alex Williamson
2013-07-11 10:05 ` Guillaume Thouvenin
0 siblings, 1 reply; 7+ messages in thread
From: Alex Williamson @ 2013-07-10 15:25 UTC (permalink / raw)
To: Guillaume Thouvenin; +Cc: kvm
On Wed, 2013-07-10 at 14:54 +0200, Guillaume Thouvenin wrote:
> Alex Williamson <alex.williamson@redhat.com> a écrit :
>
> > Are you sure that nvidia-smi is relevant to the K1/K2 devices? Does it
> > work on the host?
>
> Yes it works on the host and I have some information like power
> consumption, temperature of the GPU, etc...
>
> > Are
> > there other tests you can do to check whether the device is otherwise
> > available?
>
> I also tried to start an X server but I got the same error reported in
> the syslog: [ 435.745673] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
> [ 435.745695] NVRM: rm_init_adapter(0) failed
>
> And in the xorg.log I have:
>
>
> [ 423.624] (==) NVIDIA(0): Depth 24, (==) framebuffer bpp 32
> [ 423.624] (==) NVIDIA(0): RGB weight 888
> [ 423.624] (==) NVIDIA(0): Default visual is TrueColor
> [ 423.624] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
> [ 423.624] (**) NVIDIA(0): Option "NoLogo" "true"
> [ 423.624] (**) NVIDIA(0): Option "UseDisplayDevice" "none"
> [ 423.624] (**) NVIDIA(0): Enabling 2D acceleration
> [ 423.624] (**) NVIDIA(0): Option "UseDisplayDevice" set to "none";
> enabling NoScanout
> [ 423.624] (**) NVIDIA(0): mode
> [ 435.746] (EE) NVIDIA(0): Failed to initialize the NVIDIA GPU at
> PCI:0:3:0. Please
> [ 435.746] (EE) NVIDIA(0): check your system's kernel log for
> additional error
> [ 435.746] (EE) NVIDIA(0): messages and refer to Chapter 8:
> Common Problems in the
> [ 435.746] (EE) NVIDIA(0): README for additional information.
> [ 435.746] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device!
> [ 435.746] (EE) NVIDIA(0): Failing initialization of X screen 0
AFAICT, rm_init_adapter is in the binary blob part of the nvidia driver,
so we can't simply check the code to see what's wrong. Google finds
this:
http://forums.gentoo.org/viewtopic-t-961110.html?sid=0a60d40ba001f7bbe799bb16b2921cac
That was of course on bare metal, but note that the IOMMU prevented some
accesses and the solution was to disable the IOMMU. Of course we can't
disable the IOMMU in this case. Do you see any IOMMU faults in dmesg on
the host around this error? Does nouveau work? Thanks,
Alex
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-09 10:15 ` Guillaume Thouvenin
2013-07-09 17:52 ` Alex Williamson
@ 2013-07-10 15:30 ` Guillaume Thouvenin
1 sibling, 0 replies; 7+ messages in thread
From: Guillaume Thouvenin @ 2013-07-10 15:30 UTC (permalink / raw)
To: Guillaume Thouvenin; +Cc: kvm
Guillaume Thouvenin <guillaume.thouvenin@polymtl.ca> a écrit :
> NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
> Unable to determine the device handle for GPU 0000:00:03.0: Unknown Error
And I look with strace and I can see that:
stat("/dev/nvidiactl", {st_mode=S_IFCHR|0666, st_rdev=makedev(195,
255), ...}) = 0
open("/dev/nvidiactl", O_RDWR) = 3
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
ioctl(3, 0xc04846d2, 0x7fff95a7f380) = 0
ioctl(3, 0xc00446ca, 0x7f7692482000) = 0
ioctl(3, 0xc70046c8, 0x7f7692482060) = 0
ioctl(3, 0xc020462b, 0x7fff95a7f3d0) = 0
ioctl(3, 0xc020462a, 0x7fff95a7f3b0) = 0
ioctl(3, 0xc020462a, 0x7fff95a7f3b0) = 0
ioctl(3, 0xc020462a, 0x7fff95a7f410) = 0
open("/proc/driver/nvidia/params", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f7692e8b000
read(4, "Mobile: 4294967295\nResmanDebugLe"..., 1024) = 417
close(4) = 0
munmap(0x7f7692e8b000, 4096) = 0
stat("/dev/nvidia0", {st_mode=S_IFCHR|0666, st_rdev=makedev(195, 0), ...}) = 0
open("/dev/nvidia0", O_RDWR) = -1 EIO (Input/output error)
I dig... I dig...
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: PCI assignement with nvidia K1: RmInitAdapter failed
2013-07-10 15:25 ` Alex Williamson
@ 2013-07-11 10:05 ` Guillaume Thouvenin
0 siblings, 0 replies; 7+ messages in thread
From: Guillaume Thouvenin @ 2013-07-11 10:05 UTC (permalink / raw)
To: Alex Williamson; +Cc: kvm
Alex Williamson <alex.williamson@redhat.com> a écrit :
> Do you see any IOMMU faults in dmesg on
> the host around this error?
No, the only messages produces by IOMMU occurred at boot time.
> Does nouveau work?
Not really. When I load the nouveau module in the guest I have the
following messages:
Jul 11 11:56:29 raringvm1 kernel: [ 555.659921] [drm] Initialized drm
1.1.0 20060810
Jul 11 11:56:29 raringvm1 kernel: [ 555.668919] wmi: Mapper loaded
Jul 11 11:56:29 raringvm1 kernel: [ 555.685793] checking generic
(fa000000 160000) vs hw (f0000000 8000000)
Jul 11 11:56:29 raringvm1 kernel: [ 555.685795] checking generic
(fa000000 160000) vs hw (f8000000 2000000)
Jul 11 11:56:29 raringvm1 kernel: [ 555.689434] nouveau [
DEVICE][0000:00:03.0] BOOT0 : 0x0e7320a2
Jul 11 11:56:29 raringvm1 kernel: [ 555.689438] nouveau [
DEVICE][0000:00:03.0] Chipset: GK107 (NVE7)
Jul 11 11:56:29 raringvm1 kernel: [ 555.689439] nouveau [
DEVICE][0000:00:03.0] Family : NVE0
Jul 11 11:56:29 raringvm1 kernel: [ 555.690528] nouveau [
VBIOS][0000:00:03.0] checking PRAMIN for image...
Jul 11 11:56:29 raringvm1 kernel: [ 555.690554] nouveau [
VBIOS][0000:00:03.0] ... signature not found
Jul 11 11:56:29 raringvm1 kernel: [ 555.690556] nouveau [
VBIOS][0000:00:03.0] checking PROM for image...
Jul 11 11:56:29 raringvm1 kernel: [ 555.899715] nouveau [
VBIOS][0000:00:03.0] ... appears to be valid
Jul 11 11:56:29 raringvm1 kernel: [ 555.899717] nouveau [
VBIOS][0000:00:03.0] using image from PROM
Jul 11 11:56:29 raringvm1 kernel: [ 555.899797] nouveau [
VBIOS][0000:00:03.0] BIT signature found
Jul 11 11:56:29 raringvm1 kernel: [ 555.899798] nouveau [
VBIOS][0000:00:03.0] version 80.07.4e.00.06
Jul 11 11:56:29 raringvm1 kernel: [ 555.906258] nouveau [
PFB][0000:00:03.0] RAM type: DDR3
Jul 11 11:56:29 raringvm1 kernel: [ 555.906261] nouveau [
PFB][0000:00:03.0] RAM size: 4096 MiB
Jul 11 11:56:29 raringvm1 kernel: [ 555.906262] nouveau [
PFB][0000:00:03.0] ZCOMP: 0 tags
Jul 11 11:56:29 raringvm1 kernel: [ 555.955664] nouveau [
THERM][0000:00:03.0] Found an max1617 at address 0x4c (controlled by
lm_sensors)
Jul 11 11:56:31 raringvm1 kernel: [ 555.955666] nouveau [
I2C][0000:00:03.0] detected monitoring device: max1617
Jul 11 11:56:31 raringvm1 kernel: [ 555.999204] [TTM] Zone kernel:
Available graphics memory: 2024782 kiB
Jul 11 11:56:31 raringvm1 kernel: [ 555.999205] [TTM] Initializing
pool allocator
Jul 11 11:56:31 raringvm1 kernel: [ 555.999208] [TTM] Initializing DMA
pool allocator
Jul 11 11:56:31 raringvm1 kernel: [ 555.999267] nouveau [ DRM]
VRAM: 4096 MiB
Jul 11 11:56:31 raringvm1 kernel: [ 555.999268] nouveau [ DRM]
GART: 512 MiB
Jul 11 11:56:31 raringvm1 kernel: [ 555.999270] nouveau [ DRM]
BIT BIOS found
Jul 11 11:56:31 raringvm1 kernel: [ 555.999271] nouveau [ DRM]
Bios version 80.07.4e.00
Jul 11 11:56:31 raringvm1 kernel: [ 555.999273] nouveau [ DRM]
TMDS table version 2.0
Jul 11 11:56:31 raringvm1 kernel: [ 555.999274] nouveau [ DRM]
DCB version 4.0
Jul 11 11:56:31 raringvm1 kernel: [ 555.999275] nouveau [ DRM]
DCB outp 00: 02000f00 00020030
Jul 11 11:56:31 raringvm1 kernel: [ 555.999276] nouveau [ DRM]
DCB conn 00: 00000000
Jul 11 11:56:31 raringvm1 kernel: [ 556.002406] [drm] Supports vblank
timestamp caching Rev 1 (10.10.2010).
Jul 11 11:56:31 raringvm1 kernel: [ 556.002408] [drm] No driver
support for vblank timestamp query.
Jul 11 11:56:31 raringvm1 kernel: [ 556.012527] nouveau E[
PDISP][0000:00:03.0] chid 0 mthd 0x0000 data 0x00000000 0x10001000
0x00000001
Jul 11 11:56:31 raringvm1 kernel: [ 558.005281] [TTM] Finalizing pool
allocator
Jul 11 11:56:31 raringvm1 kernel: [ 558.005285] [TTM] Finalizing DMA
pool allocator
Jul 11 11:56:31 raringvm1 kernel: [ 558.005300] [TTM] Zone kernel:
Used memory at exit: 0 kiB
Jul 11 11:56:31 raringvm1 kernel: [ 558.013388] nouveau: probe of
0000:00:03.0 failed with error -16
So I don't know what it the error -16 but when I try to start an X
server it starts well and finally complains about KMS that is not
enabled. I checked in the config file and CONFIG_FRAMEBUFFER_CONSOLE is
set to y.
As KMS seems required by the nouveau driver, X failed to use nouveau.
Here is the logs in Xorg.log:
[ 22.234] (II) LoadModule: "nouveau"
[ 22.235] (II) Loading /usr/lib/xorg/modules/drivers/nouveau_drv.so
[ 22.236] (II) Module nouveau: vendor="X.Org Foundation"
[ 22.236] compiled for 1.13.3, module version = 1.0.7
[ 22.236] Module class: X.Org Video Driver
[ 22.236] ABI class: X.Org Video Driver, version 13.1
[ 22.236] (II) NOUVEAU driver Date: Wed Mar 27 09:50:03 2013 +0100
[ 22.236] (II) NOUVEAU driver for NVIDIA chipset families :
[ 22.236] RIVA TNT (NV04)
[ 22.236] RIVA TNT2 (NV05)
[ 22.236] GeForce 256 (NV10)
[ 22.236] GeForce 2 (NV11, NV15)
[ 22.236] GeForce 4MX (NV17, NV18)
[ 22.236] GeForce 3 (NV20)
[ 22.236] GeForce 4Ti (NV25, NV28)
[ 22.236] GeForce FX (NV3x)
[ 22.236] GeForce 6 (NV4x)
[ 22.236] GeForce 7 (G7x)
[ 22.236] GeForce 8 (G8x)
[ 22.236] GeForce GTX 200 (NVA0)
[ 22.236] GeForce GTX 400 (NVC0)
[ 22.236] (--) using VT number 7
[ 22.239] (EE) [drm] KMS not enabled
[ 22.239] (EE) No devices detected.
Regards,
Guillaume
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-07-11 10:05 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-07-09 9:18 PCI assignement with nvidia K1: RmInitAdapter failed Guillaume Thouvenin
2013-07-09 10:15 ` Guillaume Thouvenin
2013-07-09 17:52 ` Alex Williamson
2013-07-10 12:54 ` Guillaume Thouvenin
2013-07-10 15:25 ` Alex Williamson
2013-07-11 10:05 ` Guillaume Thouvenin
2013-07-10 15:30 ` Guillaume Thouvenin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox