Graphics card pass-through working with two pass pci-initialization

Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

Hi Gerd,
I managed to pass through a graphics card to a Windows7 VM using your 
kraxel.q35 seabios branch 
(http://www.kraxel.org/cgit/seabios/log/?h=kraxel.q35).

Here is my setup:
Intel DX58SO
Core i7 920
Radeon HD 6950
Kernel 2.6.35.7
qemu-kvm git pull from May 26th

The gfx card is working well on the desktop even with Aero turned on.
I tried various 3D Games which worked good (some were a bit choppy ;-) 
though).

There were 2 games that made Windows crash.
What is the way to debug what went wrong here?

One thing that is not working is the pass-through of a second device, a 
sound card in my case. As soon as I pass 2 devices to the VM, Windows 
does not boot anymore.
If I continue and remove the second device from the command line the VM 
does not boot anymore with the passed through gfx card. The graphics 
card drivers crashes Windows, with the error message of not being able 
to reset the graphics card. A reboot of the host cures this problem.

These are the command lines I used to start the VM:

echo "1002 6719" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
echo 0000:03:00.0 > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 6719" > /sys/bus/pci/drivers/pci-stub/remove_id

qemu-system-x86_64 \
-boot order=cd -cpu host -vnc 192.168.3.1:0 -k de \
-smp 2,cores=2,sockets=1 \
-drive file=${IMAGE},if=ide,index=0,cache=writeback -m 4096 \
-drive file=/dev/sda3,if=ide,index=1,cache=none \ 
-drive file=/dev/sdb3,if=ide,index=2,cache=none \
-drive file=/dev/cdrom,if=ide,index=3,media=cdrom,cache=none \
-device pci-assign,host=03:00.0 \
-net nic,model=e1000,macaddr=DE:AD:BE:EF:42:42 \
-net tap,script=/usr/local/bin/qemu-ifup \
-usb -usbdevice host:045e:00f9 \
-monitor telnet:192.168.3.1:11111,server,nowait,nodelay

Thank you very much for your work.

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

On 2011-05-27 07:32, André Weidemann wrote:
> Hi Gerd,
> I managed to pass through a graphics card to a Windows7 VM using your
> kraxel.q35 seabios branch
> (http://www.kraxel.org/cgit/seabios/log/?h=kraxel.q35).
> 
> Here is my setup:
> Intel DX58SO
> Core i7 920
> Radeon HD 6950
> Kernel 2.6.35.7
> qemu-kvm git pull from May 26th
> 
> The gfx card is working well on the desktop even with Aero turned on.
> I tried various 3D Games which worked good (some were a bit choppy ;-)
> though).

Cool!

> 
> There were 2 games that made Windows crash.
> What is the way to debug what went wrong here?

Likely that's very hard to analyze, specifically as pass-through may
also widen pre-existing race windows that may reveal real bugs in the
drivers. However, you could start with tracing the host kvm events
around the crash using ftrace / trace-cmd. With a lot of luck, the
traces may contain some suspicious access patterns.

> 
> One thing that is not working is the pass-through of a second device, a
> sound card in my case. As soon as I pass 2 devices to the VM, Windows
> does not boot anymore.
> If I continue and remove the second device from the command line the VM
> does not boot anymore with the passed through gfx card. The graphics
> card drivers crashes Windows, with the error message of not being able
> to reset the graphics card. A reboot of the host cures this problem.

Does it make a difference when you preserve the function address by
specifying -device pci-assign,host=03:00.0,addr=XX.0 and -device
pci-assign,host=03:00.1,addr=XX.1 (where XX is some free slot number in
the guest)?

> 
> These are the command lines I used to start the VM:
> 
> echo "1002 6719" > /sys/bus/pci/drivers/pci-stub/new_id
> echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
> echo 0000:03:00.0 > /sys/bus/pci/drivers/pci-stub/bind
> echo "1002 6719" > /sys/bus/pci/drivers/pci-stub/remove_id
> 
> qemu-system-x86_64 \
> -boot order=cd -cpu host -vnc 192.168.3.1:0 -k de \
> -smp 2,cores=2,sockets=1 \
> -drive file=${IMAGE},if=ide,index=0,cache=writeback -m 4096 \
> -drive file=/dev/sda3,if=ide,index=1,cache=none \ -drive
> file=/dev/sdb3,if=ide,index=2,cache=none \
> -drive file=/dev/cdrom,if=ide,index=3,media=cdrom,cache=none \
> -device pci-assign,host=03:00.0 \
> -net nic,model=e1000,macaddr=DE:AD:BE:EF:42:42 \
> -net tap,script=/usr/local/bin/qemu-ifup \
> -usb -usbdevice host:045e:00f9 \
> -monitor telnet:192.168.3.1:11111,server,nowait,nodelay
> 
> Thank you very much for your work.
> 

I've started to create a wiki page to save success stories like this one
and collect the open issues, see
http://www.linux-kvm.org/wiki/index.php?title=VGA_device_assignment.
Everyone is welcome to fix, update, or otherwise enhance it.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

Hi,

On 27.05.2011 13:09, Jan Kiszka wrote:
> On 2011-05-27 07:32, André Weidemann wrote:

>> Here is my setup:
>> Intel DX58SO
>> Core i7 920
>> Radeon HD 6950
>> Kernel 2.6.35.7
>> qemu-kvm git pull from May 26th

>> One thing that is not working is the pass-through of a second device, a
>> sound card in my case. As soon as I pass 2 devices to the VM, Windows
>> does not boot anymore.
>> If I continue and remove the second device from the command line the VM
>> does not boot anymore with the passed through gfx card. The graphics
>> card drivers crashes Windows, with the error message of not being able
>> to reset the graphics card. A reboot of the host cures this problem.
>
> Does it make a difference when you preserve the function address by
> specifying -device pci-assign,host=03:00.0,addr=XX.0 and -device
> pci-assign,host=03:00.1,addr=XX.1 (where XX is some free slot number in
> the guest)?

I did some further testing in assigning both the VGA and the sound card.
If I pass the devices to the VM like this:

-device pci-assign,host=03:00.0,addr=0x4 \ #VGA
-device pci-assign,host=00:1b.0,addr=0x5 \ #Sound

Windows boots up, but the sound card is not visible as a device inside 
the VM.

Running "info pci" in the monitor shows this:
   Bus  0, device   4, function 0:
     VGA controller: PCI device 1002:6719
       IRQ 11.
       BAR0: 32 bit prefetchable memory at 0xe0000000 [0xefffffff].
       BAR2: 32 bit memory at 0xfeba0000 [0xfebbffff].
       BAR4: I/O at 0xc000 [0xc0ff].
       BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
       id ""
   Bus  0, device   5, function 0:
     Audio controller: PCI device 8086:3a3e
       IRQ 10.
       BAR0: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
       id ""


If I switch the order of the two devices to this:

-device pci-assign,host=00:1b.0,addr=0x4 \ #Sound
-device pci-assign,host=03:00.0,addr=0x5 \ #VGA

I get the following error and kvm crashes:

http://pastebin.com/KmKEqhWK

It looks like the free PCI memory was not enough to assign the graphics 
card after assigning the sound card.


If I only assign the sound card, then "info pci" in the monitor looks 
like this:
   Bus  0, device   4, function 0:
     Audio controller: PCI device 8086:3a3e
       IRQ 10.
       BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
       id ""

The device is then visible inside the VM and working.


If I am not mistaken then the graphics card needs 2 bars, one with 256MB 
and one with 128K. The sound card then needs 1 bar with 16K of PCI memory.
How big is the PCI memory with seabios?
Is there really not enough space to "squeeze" in those extra 16K?

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

On 27.05.2011 21:40, André Weidemann wrote:

> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
> and one with 128K. The sound card then needs 1 bar with 16K of PCI memory.
> How big is the PCI memory with seabios?
> Is there really not enough space to "squeeze" in those extra 16K?

I obviously forgot to add up the other memory that is used...
32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did 
not work, so I left it in. And the e1000 NIC needs another 128K.

I'll see if I can get rid of the standard VGA card. I guess that should 
free enough memory for the sound card.

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

On 27.05.2011 21:50, André Weidemann wrote:
> On 27.05.2011 21:40, André Weidemann wrote:
>
>> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>> memory.
>> How big is the PCI memory with seabios?
>> Is there really not enough space to "squeeze" in those extra 16K?
>
> I obviously forgot to add up the other memory that is used...
> 32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did
> not work, so I left it in. And the e1000 NIC needs another 128K.
>
> I'll see if I can get rid of the standard VGA card. I guess that should
> free enough memory for the sound card.

I did some more testing by starting the VM with the paramter "-vga none" 
and passed both the VGA card and the sound card to it. With this option 
the VM did not boot, but I could use the monitor to take a look at the 
PCI bar assignment. Even though the memory for the standard VGA card is 
freed, the soundcard does not seem to get the 16K bar it needs. "info 
pci" for the sound card still looks like this:
  Bus  0, device   5, function 0:
     Audio controller: PCI device 8086:3a3e
       IRQ 10.
       BAR0: 32 bit memory at 0xffffffffffffffff [0x00003ffe].

Does anyone have an idea why there was no bar assigned?
Can the kernel be too old? (2.6.35.7.)

Just to test whether or not two devices can be assigned, I passed 
through 2 sound cards. (There is an onbard sound card and the Radeon has 
one too).

Each sound card gets its bar assigned as you can see:
   Bus  0, device   4, function 0:
     Audio controller: PCI device 1002:aa80
       IRQ 10.
       BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
       id ""
   Bus  0, device   5, function 0:
     Audio controller: PCI device 8086:3a3e
       IRQ 10.
       BAR0: 32 bit memory at 0xfebf4000 [0xfebf7fff].

but the sound cards do not show inside the Windows VM.

With both sound cards still passed to the VM I then booted an Ubuntu 
10.10 image instead of Windows7. It got as far as starting gdm, but then 
the entire host and VM became very slow.
The last message I saw on the terminal before gdm started was this:
[   23.030016 ] hda_intel: azx_get_response timeout, switching to 
single_cmd mode: last cmd=0x000f0000
[   29.290017 ] hda_intel: azx_get_response timeout, switching to 
single_cmd mode: last cmd=0x200f0000

So it seems there is a problem of some kind when 2 devices are passed to 
a VM.
Did anyone else ever notice anything similar?

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3300 bytes --]

On 2011-05-26 23:19, André Weidemann wrote:
> On 27.05.2011 21:50, André Weidemann wrote:
>> On 27.05.2011 21:40, André Weidemann wrote:
>>
>>> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
>>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>>> memory.
>>> How big is the PCI memory with seabios?
>>> Is there really not enough space to "squeeze" in those extra 16K?
>>
>> I obviously forgot to add up the other memory that is used...
>> 32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did
>> not work, so I left it in. And the e1000 NIC needs another 128K.
>>
>> I'll see if I can get rid of the standard VGA card. I guess that should
>> free enough memory for the sound card.
> 
> I did some more testing by starting the VM with the paramter "-vga none"
> and passed both the VGA card and the sound card to it. With this option
> the VM did not boot,

Where did it hang, ie. what IP was reported by info cpus?

> but I could use the monitor to take a look at the
> PCI bar assignment. Even though the memory for the standard VGA card is
> freed, the soundcard does not seem to get the 16K bar it needs. "info
> pci" for the sound card still looks like this:
>  Bus  0, device   5, function 0:
>     Audio controller: PCI device 8086:3a3e
>       IRQ 10.
>       BAR0: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
> 
> Does anyone have an idea why there was no bar assigned?

Maybe Gerd's patches aren't sufficient and you still need to change
BUILD_MAX_HIGHMEM. See the hacks in
http://git.kiszka.org/?p=seabios.git;a=shortlog;h=refs/heads/vga-assign,
either replacing Gerd's patches or combined with them (I haven't checked
if the latter makes sense).

> Can the kernel be too old? (2.6.35.7.)

It would be good to check the latest kvm kernel to see if that oops is
still present. In that case, please try to collect the backtrace via
serial console, hopefully complete then. We may have an resource cleanup
issue there.

> 
> Just to test whether or not two devices can be assigned, I passed
> through 2 sound cards. (There is an onbard sound card and the Radeon has
> one too).
> 
> Each sound card gets its bar assigned as you can see:
>   Bus  0, device   4, function 0:
>     Audio controller: PCI device 1002:aa80
>       IRQ 10.
>       BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
>       id ""
>   Bus  0, device   5, function 0:
>     Audio controller: PCI device 8086:3a3e
>       IRQ 10.
>       BAR0: 32 bit memory at 0xfebf4000 [0xfebf7fff].
> 
> but the sound cards do not show inside the Windows VM.
> 
> With both sound cards still passed to the VM I then booted an Ubuntu
> 10.10 image instead of Windows7. It got as far as starting gdm, but then
> the entire host and VM became very slow.
> The last message I saw on the terminal before gdm started was this:
> [   23.030016 ] hda_intel: azx_get_response timeout, switching to
> single_cmd mode: last cmd=0x000f0000
> [   29.290017 ] hda_intel: azx_get_response timeout, switching to
> single_cmd mode: last cmd=0x200f0000

Likely some IRQ issue. Please check if latest qemu-kvm.git +
http://thread.gmane.org/gmane.comp.emulators.qemu/102540 makes any
difference.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

Hi,

On 28.05.2011 10:18, Jan Kiszka wrote:
> On 2011-05-26 23:19, André Weidemann wrote:
>> On 27.05.2011 21:50, André Weidemann wrote:
>>> On 27.05.2011 21:40, André Weidemann wrote:
>>>
>>>> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
>>>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>>>> memory.
>>>> How big is the PCI memory with seabios?
>>>> Is there really not enough space to "squeeze" in those extra 16K?
>>>
>>> I obviously forgot to add up the other memory that is used...
>>> 32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did
>>> not work, so I left it in. And the e1000 NIC needs another 128K.
>>>
>>> I'll see if I can get rid of the standard VGA card. I guess that should
>>> free enough memory for the sound card.
>>
>> I did some more testing by starting the VM with the paramter "-vga none"
>> and passed both the VGA card and the sound card to it. With this option
>> the VM did not boot,
>
> Where did it hang, ie. what IP was reported by info cpus?

I added some debug options and found out, that the VM hangs when trying 
to initialize the graphics card ROM.
See here:
http://pastebin.com/S9a8uQfU

And some additional info here:

http://pastebin.com/AC4rw8Ek (info cpus/registers)
http://pastebin.com/yYkn8jL2 (info pci)

>> but I could use the monitor to take a look at the
>> PCI bar assignment. Even though the memory for the standard VGA card is
>> freed, the soundcard does not seem to get the 16K bar it needs. "info
>> pci" for the sound card still looks like this:
>>   Bus  0, device   5, function 0:
>>      Audio controller: PCI device 8086:3a3e
>>        IRQ 10.
>>        BAR0: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
>>
>> Does anyone have an idea why there was no bar assigned?
>
> Maybe Gerd's patches aren't sufficient and you still need to change
> BUILD_MAX_HIGHMEM. See the hacks in
> http://git.kiszka.org/?p=seabios.git;a=shortlog;h=refs/heads/vga-assign,
> either replacing Gerd's patches or combined with them (I haven't checked
> if the latter makes sense).

I do not have access to the machine until tomorrow. I'm curious to see 
if extending the PCI memory window will cure the problem.

>> Can the kernel be too old? (2.6.35.7.)
>
> It would be good to check the latest kvm kernel to see if that oops is
> still present. In that case, please try to collect the backtrace via
> serial console, hopefully complete then. We may have an resource cleanup
> issue there.

I will see if I can upgrade to the latest kernel tomorrow.

>>
>> Just to test whether or not two devices can be assigned, I passed
>> through 2 sound cards. (There is an onbard sound card and the Radeon has
>> one too).
>>
>> Each sound card gets its bar assigned as you can see:
>>    Bus  0, device   4, function 0:
>>      Audio controller: PCI device 1002:aa80
>>        IRQ 10.
>>        BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
>>        id ""
>>    Bus  0, device   5, function 0:
>>      Audio controller: PCI device 8086:3a3e
>>        IRQ 10.
>>        BAR0: 32 bit memory at 0xfebf4000 [0xfebf7fff].
>>
>> but the sound cards do not show inside the Windows VM.
>>
>> With both sound cards still passed to the VM I then booted an Ubuntu
>> 10.10 image instead of Windows7. It got as far as starting gdm, but then
>> the entire host and VM became very slow.
>> The last message I saw on the terminal before gdm started was this:
>> [   23.030016 ] hda_intel: azx_get_response timeout, switching to
>> single_cmd mode: last cmd=0x000f0000
>> [   29.290017 ] hda_intel: azx_get_response timeout, switching to
>> single_cmd mode: last cmd=0x200f0000
>
> Likely some IRQ issue. Please check if latest qemu-kvm.git +
> http://thread.gmane.org/gmane.comp.emulators.qemu/102540 makes any
> difference.

See above comments. I will try this tomorrow.

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On 2010-05-28 20:49, André Weidemann wrote:
> Hi,
> 
> On 28.05.2011 10:18, Jan Kiszka wrote:
>> On 2011-05-26 23:19, André Weidemann wrote:
>>> On 27.05.2011 21:50, André Weidemann wrote:
>>>> On 27.05.2011 21:40, André Weidemann wrote:
>>>>
>>>>> If I am not mistaken then the graphics card needs 2 bars, one with
>>>>> 256MB
>>>>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>>>>> memory.
>>>>> How big is the PCI memory with seabios?
>>>>> Is there really not enough space to "squeeze" in those extra 16K?
>>>>
>>>> I obviously forgot to add up the other memory that is used...
>>>> 32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did
>>>> not work, so I left it in. And the e1000 NIC needs another 128K.
>>>>
>>>> I'll see if I can get rid of the standard VGA card. I guess that should
>>>> free enough memory for the sound card.
>>>
>>> I did some more testing by starting the VM with the paramter "-vga none"
>>> and passed both the VGA card and the sound card to it. With this option
>>> the VM did not boot,
>>
>> Where did it hang, ie. what IP was reported by info cpus?
> 
> I added some debug options and found out, that the VM hangs when trying
> to initialize the graphics card ROM.
> See here:
> http://pastebin.com/S9a8uQfU
> 
> And some additional info here:
> 
> http://pastebin.com/AC4rw8Ek (info cpus/registers)
> http://pastebin.com/yYkn8jL2 (info pci)

Yeah, you definitely run out of PCI memory.

Plus you may suffer from the PAM/SMRAM bug I wrote about in the wiki.
Try if this hack improves the situation:

http://git.kiszka.org/?p=qemu-kvm.git;a=commitdiff;h=96e600f43275310364c0310519d6ab6540bb7d25

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Graphics card pass-through working with two pass pci-initialization

[Avi Kivity]

On 05/29/2011 06:22 PM, Jan Kiszka wrote:
> Yeah, you definitely run out of PCI memory.
>
> Plus you may suffer from the PAM/SMRAM bug I wrote about in the wiki.
> Try if this hack improves the situation:
>
> http://git.kiszka.org/?p=qemu-kvm.git;a=commitdiff;h=96e600f43275310364c0310519d6ab6540bb7d25
>

Hopefully the memory API will fix this.  I'll try to push it forwards 
this week while Marcelo's minding kvm.git.

-- 
error compiling committee.c: too many arguments to function

Re: Graphics card pass-through working with two pass pci-initialization

[André Weidemann]

Hi,

On 29.05.2011 17:22, Jan Kiszka wrote:
> On 2010-05-28 20:49, André Weidemann wrote:
>> Hi,
>>
>> On 28.05.2011 10:18, Jan Kiszka wrote:
>>> On 2011-05-26 23:19, André Weidemann wrote:
>>>> On 27.05.2011 21:50, André Weidemann wrote:
>>>>> On 27.05.2011 21:40, André Weidemann wrote:
>>>>>
>>>>>> If I am not mistaken then the graphics card needs 2 bars, one with
>>>>>> 256MB
>>>>>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>>>>>> memory.
>>>>>> How big is the PCI memory with seabios?
>>>>>> Is there really not enough space to "squeeze" in those extra 16K?
>>>>>
>>>>> I obviously forgot to add up the other memory that is used...
>>>>> 32MB go to the standard VGA card. Running qemu-kvm with "-vga none" did
>>>>> not work, so I left it in. And the e1000 NIC needs another 128K.
>>>>>
>>>>> I'll see if I can get rid of the standard VGA card. I guess that should
>>>>> free enough memory for the sound card.
>>>>
>>>> I did some more testing by starting the VM with the paramter "-vga none"
>>>> and passed both the VGA card and the sound card to it. With this option
>>>> the VM did not boot,
>>>
>>> Where did it hang, ie. what IP was reported by info cpus?
>>
>> I added some debug options and found out, that the VM hangs when trying
>> to initialize the graphics card ROM.
>> See here:
>> http://pastebin.com/S9a8uQfU
>>
>> And some additional info here:
>>
>> http://pastebin.com/AC4rw8Ek (info cpus/registers)
>> http://pastebin.com/yYkn8jL2 (info pci)
>
> Yeah, you definitely run out of PCI memory.
>
> Plus you may suffer from the PAM/SMRAM bug I wrote about in the wiki.
> Try if this hack improves the situation:
>
> http://git.kiszka.org/?p=qemu-kvm.git;a=commitdiff;h=96e600f43275310364c0310519d6ab6540bb7d25

I applied above patch, but it did not make a difference. The VM did not 
boot with the parameter "-vga none".


I found the time to apply your two patches here:
http://git.kiszka.org/?p=seabios.git;a=shortlog;h=refs/heads/vga-assign
After doing so, Gerds q35 bios branch prints an error message during 
compile.
The error comes from line 33 here:
http://www.kraxel.org/cgit/seabios/tree/src/dev-q35.c?h=kraxel.q35
How do I adjust the memory area here?

Regards
  André

Re: Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2928 bytes --]

On 2011-05-29 21:30, André Weidemann wrote:
> Hi,
> 
> On 29.05.2011 17:22, Jan Kiszka wrote:
>> On 2010-05-28 20:49, André Weidemann wrote:
>>> Hi,
>>>
>>> On 28.05.2011 10:18, Jan Kiszka wrote:
>>>> On 2011-05-26 23:19, André Weidemann wrote:
>>>>> On 27.05.2011 21:50, André Weidemann wrote:
>>>>>> On 27.05.2011 21:40, André Weidemann wrote:
>>>>>>
>>>>>>> If I am not mistaken then the graphics card needs 2 bars, one with
>>>>>>> 256MB
>>>>>>> and one with 128K. The sound card then needs 1 bar with 16K of PCI
>>>>>>> memory.
>>>>>>> How big is the PCI memory with seabios?
>>>>>>> Is there really not enough space to "squeeze" in those extra 16K?
>>>>>>
>>>>>> I obviously forgot to add up the other memory that is used...
>>>>>> 32MB go to the standard VGA card. Running qemu-kvm with "-vga
>>>>>> none" did
>>>>>> not work, so I left it in. And the e1000 NIC needs another 128K.
>>>>>>
>>>>>> I'll see if I can get rid of the standard VGA card. I guess that
>>>>>> should
>>>>>> free enough memory for the sound card.
>>>>>
>>>>> I did some more testing by starting the VM with the paramter "-vga
>>>>> none"
>>>>> and passed both the VGA card and the sound card to it. With this
>>>>> option
>>>>> the VM did not boot,
>>>>
>>>> Where did it hang, ie. what IP was reported by info cpus?
>>>
>>> I added some debug options and found out, that the VM hangs when trying
>>> to initialize the graphics card ROM.
>>> See here:
>>> http://pastebin.com/S9a8uQfU
>>>
>>> And some additional info here:
>>>
>>> http://pastebin.com/AC4rw8Ek (info cpus/registers)
>>> http://pastebin.com/yYkn8jL2 (info pci)
>>
>> Yeah, you definitely run out of PCI memory.
>>
>> Plus you may suffer from the PAM/SMRAM bug I wrote about in the wiki.
>> Try if this hack improves the situation:
>>
>> http://git.kiszka.org/?p=qemu-kvm.git;a=commitdiff;h=96e600f43275310364c0310519d6ab6540bb7d25
>>
> 
> I applied above patch, but it did not make a difference. The VM did not
> boot with the parameter "-vga none".

Err, yes, of course. You also need at least legacy VGA pass-through like
in this commit:

http://git.kiszka.org/?p=qemu-kvm.git;a=commitdiff;h=5c14a1b3510d977f194b75672196caca0b488760

But there is no guarantee that the BIOS you try to run through POST
inside the guest is actually prepared for this. Maybe specifying an
empty ROM helps to get the guest a bit further:

-device pci-assign,romfile=,host=...

> 
> 
> I found the time to apply your two patches here:
> http://git.kiszka.org/?p=seabios.git;a=shortlog;h=refs/heads/vga-assign
> After doing so, Gerds q35 bios branch prints an error message during
> compile.
> The error comes from line 33 here:
> http://www.kraxel.org/cgit/seabios/tree/src/dev-q35.c?h=kraxel.q35
> How do I adjust the memory area here?

Just use my branch without any of Gerd's patches for the first step.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Graphics card pass-through working with two pass pci-initialization

[Gerd Hoffmann]

   Hi,

> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
> and one with 128K. The sound card then needs 1 bar with 16K of PCI memory.
> How big is the PCI memory with seabios?

Some comments on that (apply to the kraxel.q35 branch):

You can add this to the qemu command line to get the seabios debug 
output printed:

qemu $usualoptions -chardev stdio,id=seabios \
      -device isa-debugcon,iobase=0x402,chardev=seabios

Then you can watch seabios assigning memory.

The ordering of the cards should not matter at all, the hacked bios does 
a two-pass scan to order the memory bars by size and pack them nicely.

Finding holes in address space for pci bars is not perfect yet.  Right 
now it has sort-of hardcoded the 0xf0000000 -> 0xfec00000 range and 
tries to fit in stuff there.  If it fails to fit in everything it tries 
again but leaves out devices which are not essential for boot.  If it 
still fails the error isn't handled yet.  For q35 this is a non-starter 
as it uses the e0000000 -> efffffff range for pci mmconfig.  For the 
piix it doesn't hurt for now as the area above 0xe0000000 is unused. 
And the address space up to 0xe0000000 is used for guest ram only, thus 
it depends on the amount of memory you assign to the guest whenever this 
area is free or not.  If you assign -- say -- 2G to your guest 
everything above 0x80000000 is free.

With your setup (256m + 128m + small stuff) I'd expect seabios place the 
256m bar @ 0xe0000000, the 128m bar @ 0xf0000000 and all the small ones 
above 0xf8000000.  It will maps only the devices which it thinks are 
essential for boot (i.e. skip the sound card).

At the end of the day I want the pci address space allocation code have 
no hard-coded addresses in there but use the e820 table instead to 
figure how big the address space hole is.  Maybe even use multiple holes 
(i.e. also use the memory below mmconfig @ 0xe0000000 with q35).

cheers,
   Gerd

Re: Graphics card pass-through working with two pass pci-initialization

[Gerd Hoffmann]

   Hi,

> At the end of the day I want the pci address space allocation code have
> no hard-coded addresses in there but use the e820 table instead to
> figure how big the address space hole is. Maybe even use multiple holes
> (i.e. also use the memory below mmconfig @ 0xe0000000 with q35).

First cut hacked up & pushed to kraxel.q35

cheers,
   Gerd

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Isaku Yamahata]

On Mon, May 30, 2011 at 09:34:07AM +0200, Gerd Hoffmann wrote:
>   Hi,
>
>> If I am not mistaken then the graphics card needs 2 bars, one with 256MB
>> and one with 128K. The sound card then needs 1 bar with 16K of PCI memory.
>> How big is the PCI memory with seabios?
>
> Some comments on that (apply to the kraxel.q35 branch):
>
> You can add this to the qemu command line to get the seabios debug  
> output printed:
>
> qemu $usualoptions -chardev stdio,id=seabios \
>      -device isa-debugcon,iobase=0x402,chardev=seabios
>
> Then you can watch seabios assigning memory.
>
> The ordering of the cards should not matter at all, the hacked bios does  
> a two-pass scan to order the memory bars by size and pack them nicely.
>
> Finding holes in address space for pci bars is not perfect yet.  Right  
> now it has sort-of hardcoded the 0xf0000000 -> 0xfec00000 range and  
> tries to fit in stuff there.  If it fails to fit in everything it tries  
> again but leaves out devices which are not essential for boot.  If it  
> still fails the error isn't handled yet.  For q35 this is a non-starter  
> as it uses the e0000000 -> efffffff range for pci mmconfig.  For the  
> piix it doesn't hurt for now as the area above 0xe0000000 is unused. And 
> the address space up to 0xe0000000 is used for guest ram only, thus it 
> depends on the amount of memory you assign to the guest whenever this  
> area is free or not.  If you assign -- say -- 2G to your guest  
> everything above 0x80000000 is free.
>
> With your setup (256m + 128m + small stuff) I'd expect seabios place the  
> 256m bar @ 0xe0000000, the 128m bar @ 0xf0000000 and all the small ones  
> above 0xf8000000.  It will maps only the devices which it thinks are  
> essential for boot (i.e. skip the sound card).
>
> At the end of the day I want the pci address space allocation code have  
> no hard-coded addresses in there but use the e820 table instead to  
> figure how big the address space hole is.  Maybe even use multiple holes  
> (i.e. also use the memory below mmconfig @ 0xe0000000 with q35).

0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
If the range is determined dynamically, the area also needs to be
updated somehow dynamically.

...
            Name (_CRS, ResourceTemplate ()
...
                DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
                    0x00000000,         // Address Space Granularity
                    0xE0000000,         // Address Range Minimum
                    0xFEBFFFFF,         // Address Range Maximum 
                    0x00000000,         // Address Translation Offset
                    0x1EC00000,         // Address Length
                    ,, , AddressRangeMemory, TypeStatic)


thanks,
-- 
yamahata

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Gerd Hoffmann]

   Hi,

> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
> If the range is determined dynamically, the area also needs to be
> updated somehow dynamically.
>
> ...
>              Name (_CRS, ResourceTemplate ()
> ...
>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>                      0x00000000,         // Address Space Granularity
>                      0xE0000000,         // Address Range Minimum
>                      0xFEBFFFFF,         // Address Range Maximum
>                      0x00000000,         // Address Translation Offset
>                      0x1EC00000,         // Address Length
>                      ,, , AddressRangeMemory, TypeStatic)

Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone 
how this could be done?

cheers,
   Gerd

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Alexander Graf]

On 01.06.2011, at 09:30, Gerd Hoffmann wrote:

>  Hi,
> 
>> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
>> If the range is determined dynamically, the area also needs to be
>> updated somehow dynamically.
>> 
>> ...
>>             Name (_CRS, ResourceTemplate ()
>> ...
>>                 DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>>                     0x00000000,         // Address Space Granularity
>>                     0xE0000000,         // Address Range Minimum
>>                     0xFEBFFFFF,         // Address Range Maximum
>>                     0x00000000,         // Address Translation Offset
>>                     0x1EC00000,         // Address Length
>>                     ,, , AddressRangeMemory, TypeStatic)
> 
> Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone how this could be done?

We're facing similar issues on PPC. The equivalent of the DSDT there is the device tree, which is currently passed in as binary blob and slightly appended for dynamic configuration. I'd much rather like to see it fully generated inside of Qemu from all the information we have available there, so we don't run into consistency issues.

This will be even more required when we pass through SoC devices to the guest, which are not on a PCI bus. Without specifying them in the DT, the guest doesn't know about them. X86 has a similar issue. Take a look at the HPET for example. If you don't want an HPET inside the guest, the DSDT needs to be modified. So you need to change things at 2 places - the DSDT and Qemu.

I don't know how much work it would be to generate the DSDT dynamically from Qemu, but IMHO that's the sanest way to make things flexible. We could probably even extract most information from the Qdev tree.


Alex

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Rudolf Marek]

Sorry I forgot to include all addresses.

> I don't know how much work it would be to generate the DSDT dynamically from
> Qemu, but IMHO that's the sanest way to make things flexible. We could
> probably even extract most information from the Qdev tree.

Well I have written for coreboot a ACPI bytecode generator (check acpigen.c). 
Maybe moving to SeaBIOS+Coreboot would make sense? It has the PCI resource 
allocator and all already done.

Thanks,
Rudolf

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Gerd Hoffmann]

On 06/01/11 12:20, Rudolf Marek wrote:
> Sorry I forgot to include all addresses.
>
>> I don't know how much work it would be to generate the DSDT
>> dynamically from
>> Qemu, but IMHO that's the sanest way to make things flexible. We could
>> probably even extract most information from the Qdev tree.
>
> Well I have written for coreboot a ACPI bytecode generator (check
> acpigen.c). Maybe moving to SeaBIOS+Coreboot would make sense? It has
> the PCI resource allocator and all already done.

Probably makes more sense to move to qemu, then pass the tables to 
seabios as qemu knows which virtual hardware is requested by the user 
and thus is present and which isn't.

Having a brief look at the coreboot code it seems static stuff (compiled 
by iasl) and dynamic bits are combined into the final dsdt table, is 
that correct?

thanks,
   Gerd

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Rudolf Marek]

> Having a brief look at the coreboot code it seems static stuff (compiled by
> iasl) and dynamic bits are combined into the final dsdt table, is that correct?

Yes the dsdt is static, it has just external references to ssdt which is 
dynamically generated using the acpigen.

Acpigen can generate the packages, names and sometimes even bits of methods.

Thanks
Rudolf

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Kevin O'Connor]

On Wed, Jun 01, 2011 at 04:40:15PM +0200, Rudolf Marek wrote:
> >Having a brief look at the coreboot code it seems static stuff (compiled by
> >iasl) and dynamic bits are combined into the final dsdt table, is that correct?
> 
> Yes the dsdt is static, it has just external references to ssdt
> which is dynamically generated using the acpigen.
> 
> Acpigen can generate the packages, names and sometimes even bits of methods.

That's interesting.  SeaBIOS also has similar code - see
acpi.c:build_ssdt().

-Kevin

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Yushu Yao]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

Hi Folks,

Could anyone give me a 101 lesson on what is "two-pass PCI initialization"?

I tried some time ago to pass a LSI megaraid to a rhel5 guest, but the
initialization of the pci device inside the VM fails (because, I think, it
was already initialized in the host).

Just wondering could that be related? Have anyone tried passing a raid card
to the guest?

Thanks

_Yushu


+-------------------------------------------------+
| Yushu Yao
| Ph:1-510-486-4690
|
| Lawrence Berkeley National Lab
| Mailstop 50B-6222
| 1 Cyclotron Road
| Berkeley CA 94720-8147 - USA
+-------------------------------------------------+




On Wed, Jun 1, 2011 at 3:15 PM, Kevin O'Connor <kevin@koconnor.net> wrote:

> On Wed, Jun 01, 2011 at 04:40:15PM +0200, Rudolf Marek wrote:
> > >Having a brief look at the coreboot code it seems static stuff (compiled
> by
> > >iasl) and dynamic bits are combined into the final dsdt table, is that
> correct?
> >
> > Yes the dsdt is static, it has just external references to ssdt
> > which is dynamically generated using the acpigen.
> >
> > Acpigen can generate the packages, names and sometimes even bits of
> methods.
>
> That's interesting.  SeaBIOS also has similar code - see
> acpi.c:build_ssdt().
>
> -Kevin
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

[-- Attachment #2: Type: text/html, Size: 2115 bytes --]

Re: Graphics card pass-through working with two pass pci-initialization

[Avi Kivity]

On 06/01/2011 12:56 PM, Alexander Graf wrote:
> On 01.06.2011, at 09:30, Gerd Hoffmann wrote:
>
> >   Hi,
> >
> >>  0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
> >>  If the range is determined dynamically, the area also needs to be
> >>  updated somehow dynamically.
> >>
> >>  ...
> >>              Name (_CRS, ResourceTemplate ()
> >>  ...
> >>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
> >>                      0x00000000,         // Address Space Granularity
> >>                      0xE0000000,         // Address Range Minimum
> >>                      0xFEBFFFFF,         // Address Range Maximum
> >>                      0x00000000,         // Address Translation Offset
> >>                      0x1EC00000,         // Address Length
> >>                      ,, , AddressRangeMemory, TypeStatic)
> >
> >  Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone how this could be done?
>
> We're facing similar issues on PPC. The equivalent of the DSDT there is the device tree, which is currently passed in as binary blob and slightly appended for dynamic configuration. I'd much rather like to see it fully generated inside of Qemu from all the information we have available there, so we don't run into consistency issues.
>
> This will be even more required when we pass through SoC devices to the guest, which are not on a PCI bus. Without specifying them in the DT, the guest doesn't know about them. X86 has a similar issue. Take a look at the HPET for example. If you don't want an HPET inside the guest, the DSDT needs to be modified. So you need to change things at 2 places - the DSDT and Qemu.
>
> I don't know how much work it would be to generate the DSDT dynamically from Qemu, but IMHO that's the sanest way to make things flexible. We could probably even extract most information from the Qdev tree.
>

Generating the DSDT dynamically is hard, but the DSDT itself is 
dynamic.  You can make any function talk to the firmware configuration 
interface and return results that depend on the information there.

-- 
error compiling committee.c: too many arguments to function

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Alexander Graf]

On 01.06.2011, at 13:13, Avi Kivity wrote:

> On 06/01/2011 12:56 PM, Alexander Graf wrote:
>> On 01.06.2011, at 09:30, Gerd Hoffmann wrote:
>> 
>> >   Hi,
>> >
>> >>  0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
>> >>  If the range is determined dynamically, the area also needs to be
>> >>  updated somehow dynamically.
>> >>
>> >>  ...
>> >>              Name (_CRS, ResourceTemplate ()
>> >>  ...
>> >>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>> >>                      0x00000000,         // Address Space Granularity
>> >>                      0xE0000000,         // Address Range Minimum
>> >>                      0xFEBFFFFF,         // Address Range Maximum
>> >>                      0x00000000,         // Address Translation Offset
>> >>                      0x1EC00000,         // Address Length
>> >>                      ,, , AddressRangeMemory, TypeStatic)
>> >
>> >  Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone how this could be done?
>> 
>> We're facing similar issues on PPC. The equivalent of the DSDT there is the device tree, which is currently passed in as binary blob and slightly appended for dynamic configuration. I'd much rather like to see it fully generated inside of Qemu from all the information we have available there, so we don't run into consistency issues.
>> 
>> This will be even more required when we pass through SoC devices to the guest, which are not on a PCI bus. Without specifying them in the DT, the guest doesn't know about them. X86 has a similar issue. Take a look at the HPET for example. If you don't want an HPET inside the guest, the DSDT needs to be modified. So you need to change things at 2 places - the DSDT and Qemu.
>> 
>> I don't know how much work it would be to generate the DSDT dynamically from Qemu, but IMHO that's the sanest way to make things flexible. We could probably even extract most information from the Qdev tree.
>> 
> 
> Generating the DSDT dynamically is hard, but the DSDT itself is dynamic.  You can make any function talk to the firmware configuration interface and return results that depend on the information there.

Does that hold true for nodes as well? I thought you can only use 'functions' for specific elements?


Alex

Re: Graphics card pass-through working with two pass pci-initialization

[Avi Kivity]

On 06/01/2011 02:16 PM, Alexander Graf wrote:
> >>
> >
> >  Generating the DSDT dynamically is hard, but the DSDT itself is dynamic.  You can make any function talk to the firmware configuration interface and return results that depend on the information there.
>
> Does that hold true for nodes as well? I thought you can only use 'functions' for specific elements?

I believe that if you declare _CRS as a method, it should work (there 
are both "Name" and "Method" _CRSs in the DSDT currently).

-- 
error compiling committee.c: too many arguments to function

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Isaku Yamahata]

On Wed, Jun 01, 2011 at 09:30:12AM +0200, Gerd Hoffmann wrote:
>   Hi,
>
>> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
>> If the range is determined dynamically, the area also needs to be
>> updated somehow dynamically.
>>
>> ...
>>              Name (_CRS, ResourceTemplate ()
>> ...
>>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>>                      0x00000000,         // Address Space Granularity
>>                      0xE0000000,         // Address Range Minimum
>>                      0xFEBFFFFF,         // Address Range Maximum
>>                      0x00000000,         // Address Translation Offset
>>                      0x1EC00000,         // Address Length
>>                      ,, , AddressRangeMemory, TypeStatic)
>
> Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone  
> how this could be done?

Right now what I can think of is.
It would be possible to know the offset in AmlCode[] by
compiling dsl with -l option. The we can get the mix of source and
resulted hex with offset like

iasl -l q35-acpi-dsdt.dsl.i
=> q35-acpi-dsdt.dsl.lst

     192....                DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
     193....                    0x00000000,
     194....                    0xE0000000,
     195....                    0xFEBFFFFF,
     196....                    0x00000000,
     197....                    0x1EC00000,
     198....                    ,, , AddressRangeMemory, TypeStatic)

00000F74....87 17 00 00 0C 01 00 00     "........"
00000F7C....00 00 00 00 00 E0 FF FF     "........"
00000F84....BF FE 00 00 00 00 00 00     "........"
00000F8C....C0 1E ..................    ".."
00000F8E....79 00 ..................    "y."

But this is very iasl specific and fragile...
-- 
yamahata

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

On 2011-06-01 16:20, Isaku Yamahata wrote:
> On Wed, Jun 01, 2011 at 09:30:12AM +0200, Gerd Hoffmann wrote:
>>   Hi,
>>
>>> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
>>> If the range is determined dynamically, the area also needs to be
>>> updated somehow dynamically.
>>>
>>> ...
>>>              Name (_CRS, ResourceTemplate ()
>>> ...
>>>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>>>                      0x00000000,         // Address Space Granularity
>>>                      0xE0000000,         // Address Range Minimum
>>>                      0xFEBFFFFF,         // Address Range Maximum
>>>                      0x00000000,         // Address Translation Offset
>>>                      0x1EC00000,         // Address Length
>>>                      ,, , AddressRangeMemory, TypeStatic)
>>
>> Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone  
>> how this could be done?
> 
> Right now what I can think of is.
> It would be possible to know the offset in AmlCode[] by
> compiling dsl with -l option. The we can get the mix of source and
> resulted hex with offset like
> 
> iasl -l q35-acpi-dsdt.dsl.i
> => q35-acpi-dsdt.dsl.lst
> 
>      192....                DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
>      193....                    0x00000000,
>      194....                    0xE0000000,
>      195....                    0xFEBFFFFF,
>      196....                    0x00000000,
>      197....                    0x1EC00000,
>      198....                    ,, , AddressRangeMemory, TypeStatic)
> 
> 00000F74....87 17 00 00 0C 01 00 00     "........"
> 00000F7C....00 00 00 00 00 E0 FF FF     "........"
> 00000F84....BF FE 00 00 00 00 00 00     "........"
> 00000F8C....C0 1E ..................    ".."
> 00000F8E....79 00 ..................    "y."
> 
> But this is very iasl specific and fragile...

Why make this particular thing dynamic? Physical PCs need to define a
certain fixed PCI memory region as well and then live with it. IMHO, the
current setting is just too small for normal use.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Alex Williamson]

On Wed, 2011-06-01 at 16:31 +0200, Jan Kiszka wrote:
> On 2011-06-01 16:20, Isaku Yamahata wrote:
> > On Wed, Jun 01, 2011 at 09:30:12AM +0200, Gerd Hoffmann wrote:
> >>   Hi,
> >>
> >>> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
> >>> If the range is determined dynamically, the area also needs to be
> >>> updated somehow dynamically.
> >>>
> >>> ...
> >>>              Name (_CRS, ResourceTemplate ()
> >>> ...
> >>>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
> >>>                      0x00000000,         // Address Space Granularity
> >>>                      0xE0000000,         // Address Range Minimum
> >>>                      0xFEBFFFFF,         // Address Range Maximum
> >>>                      0x00000000,         // Address Translation Offset
> >>>                      0x1EC00000,         // Address Length
> >>>                      ,, , AddressRangeMemory, TypeStatic)
> >>
> >> Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone  
> >> how this could be done?
> > 
> > Right now what I can think of is.
> > It would be possible to know the offset in AmlCode[] by
> > compiling dsl with -l option. The we can get the mix of source and
> > resulted hex with offset like
> > 
> > iasl -l q35-acpi-dsdt.dsl.i
> > => q35-acpi-dsdt.dsl.lst
> > 
> >      192....                DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
> >      193....                    0x00000000,
> >      194....                    0xE0000000,
> >      195....                    0xFEBFFFFF,
> >      196....                    0x00000000,
> >      197....                    0x1EC00000,
> >      198....                    ,, , AddressRangeMemory, TypeStatic)
> > 
> > 00000F74....87 17 00 00 0C 01 00 00     "........"
> > 00000F7C....00 00 00 00 00 E0 FF FF     "........"
> > 00000F84....BF FE 00 00 00 00 00 00     "........"
> > 00000F8C....C0 1E ..................    ".."
> > 00000F8E....79 00 ..................    "y."
> > 
> > But this is very iasl specific and fragile...
> 
> Why make this particular thing dynamic? Physical PCs need to define a
> certain fixed PCI memory region as well and then live with it. IMHO, the
> current setting is just too small for normal use.

I don't think that's true.  It's possible a BIOS could walk the PCI bus,
figure out how much MMIO space it needs to reserve, then configure the
chipset to re-route the necessary memory and build the DSDT.  Hardly
anything is actually fixed within the chipset AIUI.  Thanks,

Alex

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Kevin O'Connor]

On Wed, Jun 01, 2011 at 11:20:29PM +0900, Isaku Yamahata wrote:
> On Wed, Jun 01, 2011 at 09:30:12AM +0200, Gerd Hoffmann wrote:
> >   Hi,
> >
> >> 0xE0000000 is hard-coded in the DSDT for both piix and q35 as below.
> >> If the range is determined dynamically, the area also needs to be
> >> updated somehow dynamically.
> >>
> >> ...
> >>              Name (_CRS, ResourceTemplate ()
> >> ...
> >>                  DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite,
> >>                      0x00000000,         // Address Space Granularity
> >>                      0xE0000000,         // Address Range Minimum
> >>                      0xFEBFFFFF,         // Address Range Maximum
> >>                      0x00000000,         // Address Translation Offset
> >>                      0x1EC00000,         // Address Length
> >>                      ,, , AddressRangeMemory, TypeStatic)
> >
> > Uhm, indeed.  I know next to nothing about ACPI though.  Ideas anyone  
> > how this could be done?
> 
> Right now what I can think of is.
> It would be possible to know the offset in AmlCode[] by
> compiling dsl with -l option. The we can get the mix of source and
> resulted hex with offset like

It's easier then this - as Avi indicated, one can turn _CRS into a
method which returns the current info with the PCI size filled in at
runtime.  Something like:

Method (_CRS, 0, NotSerialized) {
    Name (TMP, ResourceTemplate ()
    {
    ...
    })
    CreateDWordField (TMPM, 0x123, TMP)
    Store (TMPM, PCIM)
    Return (TMP)
}

This is already done for other devices - see \SB.LNKS._CRS.  For this
to work, the new variable PCIM needs to be set to the size of the PCI
region, which can be populated in the SSDT when it is built by
SeaBIOS.

As Jan points out though, is a dynamic PCI region really needed?
Those that need a large PCI region are also likely to need a large
amount of memory.  Maybe the space for PCI should just be increased.

-Kevin

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Gerd Hoffmann]

   Hi,

> As Jan points out though, is a dynamic PCI region really needed?
> Those that need a large PCI region are also likely to need a large
> amount of memory.  Maybe the space for PCI should just be increased.

Just changing it will not work as it will break live migration.

I think one option is to keep the piix-based pc as-is, with the address 
space above 0xe0000000 being available for PCI, and make the room larger 
for the upcoming q35-based pc, say starting at 0xc0000000.  That will 
give almost 1G address space.  256 MB are used by pci mmconfig though.

The other option is to make it configurable, which in turn requires a 
dynamic PCI region.

cheers,
   Gerd

Re: [SeaBIOS] Graphics card pass-through working with two pass pci-initialization

[Jan Kiszka]

On 2011-06-06 08:30, Gerd Hoffmann wrote:
>    Hi,
> 
>> As Jan points out though, is a dynamic PCI region really needed?
>> Those that need a large PCI region are also likely to need a large
>> amount of memory.  Maybe the space for PCI should just be increased.
> 
> Just changing it will not work as it will break live migration.

Changing logic in the BIOS won't break migration (the active BIOS is
included in the migration of RAM, current mappings are part of the
device states). Changing the 4G mapping in qemu's hw/pc_piix.c would
break it and needs to be coupled to the machine version.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux