PCI device passthrough / memory mapping issue

PCI device passthrough / memory mapping issue

[Fede]

I'm currently working to enable vga passthrough in kvm.

This is a current git qemu-kvm run with some debug printings:


slashack@fidel ~/kvm/vms $ qemu-system-x86_64 -hda i386ubuntu904.img
-boot c -m 1024 -net nic -net user,hostfwd=tcp::5555-:22 -pcidevice
host=01:00.0
vm_register_phys_mem:580 memory: gpa: 0, size: a0000, uaddr:
7fa65f077000, slot: 0, flags: 0
kvm_register_phys_mem:580 memory: gpa: 100000, size: 3ff00000, uaddr:
7fa65f177000, slot: 1, flags: 0
kvm_register_phys_mem:580 memory: gpa: e0000, size: 20000, uaddr:
7fa6a2fc0000, slot: 2, flags: 0
kvm_register_phys_mem:580 memory: gpa: c0000, size: 20000, uaddr:
7fa6a2f7d000, slot: 3, flags: 0
kvm_register_phys_mem:580 memory: gpa: fffe0000, size: 20000, uaddr:
7fa6a2fc0000, slot: 4, flags: 0
device: 01:00.0: driver="pci-assign" host="01:00.0"
get_real_device: region 0 size 16777216 start 0xe2000000 type 512 resource_fd 13
get_real_device: region 1 size 268435456 start 0xd0000000 type 4608
resource_fd 14
get_real_device: region 3 size 33554432 start 0xe0000000 type 512 resource_fd 15
get_real_device: region 5 size 128 start 0xe000 type 256 resource_fd 0
get_real_device: region 6 size 524288 start 0xe3000000 type 4608 resource_fd 0
assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe2000000
assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa65cf0e000
assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xd0000000
assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa64cf0e000
assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe0000000
assigned_dev_register_regions: NON_PCI_ROM_SLOT virt_base=0x7fa64af0e000
assigned_dev_register_regions: MAP PHYSICAL MEMORY e_physbase=0xe3000000
assigned_dev_register_regions: PCI_ROM_SLOT virt_base=0x7fa64ae8e000
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000010de len=2
assigned_dev_pci_read_config: (4.0): address=000e val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000010de len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00000622 len=2
assigned_dev_pci_read_config: (4.0): address=000e val=0x00000000 len=1
kvm_register_phys_mem:580 memory: gpa: f0000000, size: 1000000, uaddr:
7fa65e075000, slot: 5, flags: 0
kvm_dirty_pages_log_enable_slot:293 start f0000000 len 1000000
kvm_dirty_pages_log_change:266 slot 5 start f0000000 len 1000000 flags 1
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000010de len=2
assigned_dev_pci_read_config: (4.0): address=000e val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=000a val=0x00000300 len=2
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000010de len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00000622 len=2
assigned_dev_pci_read_config: (4.0): address=0010 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0xff000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0xf3000000 len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000008 len=4
assigned_dev_pci_write_config: (4.0): address=0014 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0xf0000008 len=4
assigned_dev_pci_write_config: (4.0): address=0014 val=0x00000008 len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000008 len=4
assigned_dev_pci_write_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0018 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0xfe000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0x10000000 len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0020 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000001 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0xffffff81 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0x00000001 len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000001 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0x0000c200 len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0x00000008 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0xfffff800 len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0xfff80008 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0x00000008 len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0x00000008 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0x12000000 len=4
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000407 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000407 len=2
assigned_dev_iomem_map: e_phys=f3000000 r_virt=0x7fa65cf0e000 type=0
len=01000000 region_num=0
kvm_register_phys_mem:580 memory: gpa: f3000000, size: 1000000, uaddr:
7fa65cf0e000, slot: 6, flags: 0
assigned_dev_iomem_map: e_phys=10000000 r_virt=0x7fa64af0e000 type=0
len=02000000 region_num=3
kvm_register_phys_mem:580 memory: gpa: 10000000, size: 2000000, uaddr:
7fa64af0e000, slot: 7,flags: 0
create_userspace_phys_mem: File exists
assigned_dev_iomem_map: Error: create new mapping failed


This worked a month ago. But after some git updates there's a problem.

When the real device regions are mapped from real virtual memory to
guest physical addresses in kvm, it overlaps region 3 with the guest
physical memory assigned to kernel space (0x10000000 to 0x10200000)

I've been trying to look for the answer to this question: Why is gpa
0x10000000 chosen and not any other free memory space?

It seems that this addresses are being chosen here (for example, for region 3):
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0xfe000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0x10000000 len=4

What does this mean? Why PCI BARs are being written and read? Why the
values that are written differs from the ones that are read after?

The last write is the gpa that is used.

Is this a bug? I can't find the source of this gpa addresses. I need
to change them.

Any help?

Thanks,
Federico

Re: PCI device passthrough / memory mapping issue

[Avi Kivity]

On 03/19/2010 12:46 AM, Fede wrote:
> I'm currently working to enable vga passthrough in kvm.
>    

> assigned_dev_iomem_map: e_phys=10000000 r_virt=0x7fa64af0e000 type=0
> len=02000000 region_num=3
> kvm_register_phys_mem:580 memory: gpa: 10000000, size: 2000000, uaddr:
> 7fa64af0e000, slot: 7,flags: 0
> create_userspace_phys_mem: File exists
> assigned_dev_iomem_map: Error: create new mapping failed
>
>
> This worked a month ago. But after some git updates there's a problem.
>
> When the real device regions are mapped from real virtual memory to
> guest physical addresses in kvm, it overlaps region 3 with the guest
> physical memory assigned to kernel space (0x10000000 to 0x10200000)
>
> I've been trying to look for the answer to this question: Why is gpa
> 0x10000000 chosen and not any other free memory space?
>    

The BIOS generally assigns addresses, then the OS updates them if it 
wants to.  Is the crash before or after the OS has started loading?

If before, suggest you add some printfs to seabios to explain its decisions.

> It seems that this addresses are being chosen here (for example, for region 3):
> assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
> assigned_dev_pci_write_config: (4.0): address=001c val=0xffffffff len=4
> assigned_dev_pci_read_config: (4.0): address=001c val=0xfe000000 len=4
>    

The above sequence is how the bios determines the BAR size. (0x2000000 
in this case).

> assigned_dev_pci_write_config: (4.0): address=001c val=0x00000000 len=4
>    

Now it clears the mess from the previous step.

> assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
> assigned_dev_pci_write_config: (4.0): address=001c val=0x10000000 len=4
>    

Here it assigns a new address.  It's clearly wrong.  A log in seabios 
will explain this.

> What does this mean? Why PCI BARs are being written and read?

> Why the
> values that are written differs from the ones that are read after?
>    

BARs are not RAM cells.  the least significant address bits are 
hardwired to zero, that's how the BIOS detects the BAR size (and 
required alignment).

> The last write is the gpa that is used.
>
> Is this a bug? I can't find the source of this gpa addresses. I need
> to change them.
>    

Looks like a bug in seabios.

-- 
error compiling committee.c: too many arguments to function