Status of pci passthrough work?

Status of pci passthrough work?

[Thomas Fjellstrom]

I'm very interested in being able to pass a few devices through to kvm guests. 
I'm wondering what exactly is working now, and how I can start testing it?

the latest kvm release doesn't seem to include any support for it in 
userspace, so I can't test it with that...

Basically what I want to do is assign a two or three physical nics (100mb and 
GiB) to one vm, some tv tuner cards to another.

Also, I'm wondering if AMD's iommu in the SB750 southbridge is supported yet? 
Or if anyone is working on it?

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

RE: Status of pci passthrough work?

[Han, Weidong]

Hi Thomas,

the patches of passthrough/VT-d on kvm.git are already checked in. With
Amit's userspace patches, you can assign device to guest. You can have a
try.

Randy (Weidong)

Thomas Fjellstrom wrote:
> I'm very interested in being able to pass a few devices through to
> kvm guests. I'm wondering what exactly is working now, and how I can
> start testing it? 
> 
> the latest kvm release doesn't seem to include any support for it in
> userspace, so I can't test it with that...
> 
> Basically what I want to do is assign a two or three physical nics
> (100mb and GiB) to one vm, some tv tuner cards to another.
> 
> Also, I'm wondering if AMD's iommu in the SB750 southbridge is
> supported yet? Or if anyone is working on it?
> 
> --
> Thomas Fjellstrom
> tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Han, Weidong wrote:
> Hi Thomas,
>
> the patches of passthrough/VT-d on kvm.git are already checked in. With
> Amit's userspace patches, you can assign device to guest. You can have a
> try.

Does that mean I need VT-d support in hardware? All I have to test with right 
now is an AMD Phenom X4  with a 780g+sb700 system. Don't think it has an 
iommu, and I'd find it odd if the intel VT-d code "just worked" on amd's 
hardware.

> Randy (Weidong)
>
> Thomas Fjellstrom wrote:
> > I'm very interested in being able to pass a few devices through to
> > kvm guests. I'm wondering what exactly is working now, and how I can
> > start testing it?
> >
> > the latest kvm release doesn't seem to include any support for it in
> > userspace, so I can't test it with that...
> >
> > Basically what I want to do is assign a two or three physical nics
> > (100mb and GiB) to one vm, some tv tuner cards to another.
> >
> > Also, I'm wondering if AMD's iommu in the SB750 southbridge is
> > supported yet? Or if anyone is working on it?
> >
> > --
> > Thomas Fjellstrom
> > tfjellstrom@shaw.ca
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

RE: Status of pci passthrough work?

[Han, Weidong]

Thomas Fjellstrom wrote:
> On Saturday 27 September 2008, Han, Weidong wrote:
>> Hi Thomas,
>> 
>> the patches of passthrough/VT-d on kvm.git are already checked in.
>> With Amit's userspace patches, you can assign device to guest. You
>> can have a try.
> 
> Does that mean I need VT-d support in hardware? All I have to test
> with right now is an AMD Phenom X4  with a 780g+sb700 system. Don't
> think it has an iommu, and I'd find it odd if the intel VT-d code
> "just worked" on amd's hardware.
> 

Yes, currently you need VT-d support in hardware to assign device. 

Randy (Weidong)

>> Randy (Weidong)
>> 
>> Thomas Fjellstrom wrote:
>>> I'm very interested in being able to pass a few devices through to
>>> kvm guests. I'm wondering what exactly is working now, and how I
>>> can start testing it? 
>>> 
>>> the latest kvm release doesn't seem to include any support for it in
>>> userspace, so I can't test it with that...
>>> 
>>> Basically what I want to do is assign a two or three physical nics
>>> (100mb and GiB) to one vm, some tv tuner cards to another.
>>> 
>>> Also, I'm wondering if AMD's iommu in the SB750 southbridge is
>>> supported yet? Or if anyone is working on it?
>>> 
>>> --
>>> Thomas Fjellstrom
>>> tfjellstrom@shaw.ca
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> --
> Thomas Fjellstrom
> tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Han, Weidong wrote:
> Thomas Fjellstrom wrote:
> > On Saturday 27 September 2008, Han, Weidong wrote:
> >> Hi Thomas,
> >>
> >> the patches of passthrough/VT-d on kvm.git are already checked in.
> >> With Amit's userspace patches, you can assign device to guest. You
> >> can have a try.
> >
> > Does that mean I need VT-d support in hardware? All I have to test
> > with right now is an AMD Phenom X4  with a 780g+sb700 system. Don't
> > think it has an iommu, and I'd find it odd if the intel VT-d code
> > "just worked" on amd's hardware.
>
> Yes, currently you need VT-d support in hardware to assign device.

So I take it the PV-DMA (or pv-dma doesn't do what I think it does...) or the 
other 1:1 device pass through work isn't working right now?

It's something I'd really like to use, but I don't have access to a platform 
with a hardware iommu. Though I might be able to pick up a replacement board 
for my new server with the SB750 southbridge which supposedly has AMD's new 
iommu hardware in it, but I haven't seen any evidence that kvm or linux 
supports it.

> Randy (Weidong)
>
> >> Randy (Weidong)
> >>
> >> Thomas Fjellstrom wrote:
> >>> I'm very interested in being able to pass a few devices through to
> >>> kvm guests. I'm wondering what exactly is working now, and how I
> >>> can start testing it?
> >>>
> >>> the latest kvm release doesn't seem to include any support for it in
> >>> userspace, so I can't test it with that...
> >>>
> >>> Basically what I want to do is assign a two or three physical nics
> >>> (100mb and GiB) to one vm, some tv tuner cards to another.
> >>>
> >>> Also, I'm wondering if AMD's iommu in the SB750 southbridge is
> >>> supported yet? Or if anyone is working on it?
> >>>
> >>> --
> >>> Thomas Fjellstrom
> >>> tfjellstrom@shaw.ca
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Thomas Fjellstrom
> > tfjellstrom@shaw.ca


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Amit Shah]

* On Saturday 27 Sep 2008 13:27:46 Thomas Fjellstrom wrote:
> On Saturday 27 September 2008, Han, Weidong wrote:
> > Thomas Fjellstrom wrote:
> > > On Saturday 27 September 2008, Han, Weidong wrote:
> > >> Hi Thomas,
> > >>
> > >> the patches of passthrough/VT-d on kvm.git are already checked in.
> > >> With Amit's userspace patches, you can assign device to guest. You
> > >> can have a try.
> > >
> > > Does that mean I need VT-d support in hardware? All I have to test
> > > with right now is an AMD Phenom X4  with a 780g+sb700 system. Don't
> > > think it has an iommu, and I'd find it odd if the intel VT-d code
> > > "just worked" on amd's hardware.
> >
> > Yes, currently you need VT-d support in hardware to assign device.
>
> So I take it the PV-DMA (or pv-dma doesn't do what I think it does...) or
> the other 1:1 device pass through work isn't working right now?

pvdma does work, but the most recent patches aren't yet published (I should do 
that). It will work for simple devices.

1:1 will also work.

> It's something I'd really like to use, but I don't have access to a
> platform with a hardware iommu. Though I might be able to pick up a
> replacement board for my new server with the SB750 southbridge which
> supposedly has AMD's new iommu hardware in it, but I haven't seen any
> evidence that kvm or linux supports it.

Linux 2.6.27 onwards supports AMD IOMMU. kvm (and device assignment) support 
for AMD IOMMU doesn't exist yet, but work is planned to start soon.

> > >> Thomas Fjellstrom wrote:
> > >>> I'm very interested in being able to pass a few devices through to
> > >>> kvm guests. I'm wondering what exactly is working now, and how I
> > >>> can start testing it?
> > >>>
> > >>> the latest kvm release doesn't seem to include any support for it in
> > >>> userspace, so I can't test it with that...

The userspace patch is undergoing pre-merge revisions.

I'll send out an email once I get my git trees synced up to my working 
revisions.

In the meantime, you can use the patches from the list.

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Sunday 28 September 2008, Amit Shah wrote:
> * On Saturday 27 Sep 2008 13:27:46 Thomas Fjellstrom wrote:
> > On Saturday 27 September 2008, Han, Weidong wrote:
> > > Thomas Fjellstrom wrote:
> > > > On Saturday 27 September 2008, Han, Weidong wrote:
> > > >> Hi Thomas,
> > > >>
> > > >> the patches of passthrough/VT-d on kvm.git are already checked in.
> > > >> With Amit's userspace patches, you can assign device to guest. You
> > > >> can have a try.
> > > >
> > > > Does that mean I need VT-d support in hardware? All I have to test
> > > > with right now is an AMD Phenom X4  with a 780g+sb700 system. Don't
> > > > think it has an iommu, and I'd find it odd if the intel VT-d code
> > > > "just worked" on amd's hardware.
> > >
> > > Yes, currently you need VT-d support in hardware to assign device.
> >
> > So I take it the PV-DMA (or pv-dma doesn't do what I think it does...) or
> > the other 1:1 device pass through work isn't working right now?
>
> pvdma does work, but the most recent patches aren't yet published (I should
> do that). It will work for simple devices.
>
> 1:1 will also work.

Once the trees are updated?

> > It's something I'd really like to use, but I don't have access to a
> > platform with a hardware iommu. Though I might be able to pick up a
> > replacement board for my new server with the SB750 southbridge which
> > supposedly has AMD's new iommu hardware in it, but I haven't seen any
> > evidence that kvm or linux supports it.
>
> Linux 2.6.27 onwards supports AMD IOMMU. kvm (and device assignment)
> support for AMD IOMMU doesn't exist yet, but work is planned to start soon.

What does the kernel supporting it help a person wanting to use it to pass 
through devices to guests if kvm doesn't support it? Also, what would the 
kernel use it for in that case?

> > > >> Thomas Fjellstrom wrote:
> > > >>> I'm very interested in being able to pass a few devices through to
> > > >>> kvm guests. I'm wondering what exactly is working now, and how I
> > > >>> can start testing it?
> > > >>>
> > > >>> the latest kvm release doesn't seem to include any support for it
> > > >>> in userspace, so I can't test it with that...
>
> The userspace patch is undergoing pre-merge revisions.
>
> I'll send out an email once I get my git trees synced up to my working
> revisions.
>
> In the meantime, you can use the patches from the list.

Its rather hard to find the patches from before I joined the list, most 
archives don't keep attachments.

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks for the heads up, and all the work on this stuff :)

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Amit Shah]

Please keep me in the CC to get faster replies.

* On Monday 29 Sep 2008 06:39:26 Thomas Fjellstrom wrote:

> > > > Yes, currently you need VT-d support in hardware to assign device.
> > >
> > > So I take it the PV-DMA (or pv-dma doesn't do what I think it does...)
> > > or the other 1:1 device pass through work isn't working right now?
> >
> > pvdma does work, but the most recent patches aren't yet published (I
> > should do that). It will work for simple devices.
> >
> > 1:1 will also work.
>
> Once the trees are updated?

pvdma, yes.

I don't have the 1:1 patches in my tree as of now. Andrea maintains those 
patches.

> > > It's something I'd really like to use, but I don't have access to a
> > > platform with a hardware iommu. Though I might be able to pick up a
> > > replacement board for my new server with the SB750 southbridge which
> > > supposedly has AMD's new iommu hardware in it, but I haven't seen any
> > > evidence that kvm or linux supports it.
> >
> > Linux 2.6.27 onwards supports AMD IOMMU. kvm (and device assignment)
> > support for AMD IOMMU doesn't exist yet, but work is planned to start
> > soon.
>
> What does the kernel supporting it help a person wanting to use it to pass
> through devices to guests if kvm doesn't support it? Also, what would the
> kernel use it for in that case?

That's right; that was just to give you an idea of how things are progressing.

> Its rather hard to find the patches from before I joined the list, most
> archives don't keep attachments.

Sure; look for an update next week.

> Thanks for the heads up, and all the work on this stuff :)

You're welcome and it's always good to hear from users.

Re: Status of pci passthrough work?

[xming]

When can we expect pvdma updates? Is it ever going to be merged into
mainline kvm?

Re: Status of pci passthrough work?

[Jan C. Bernauer]

Hi,

 I have about the same problem, so excuse me for hijacking this thread.

My hardware consists of a 780g/SB700 Mainboard and a 4850e AMD CPU, and
I'm interested in forwarding a DVB-C tuner card to the guest. Maybe
some NICs later.

I tried and 'sort of' got it working with Amit's kernel and userspace
tools.
First thing:
The dvb-c card has an interesting memory mapping, as reported by
lspci -v:
        Memory at cfdff000 (32-bit, non-prefetchable) [size=512]

Size 512 doesn't fly with a check in kvm_main.c:
if (mem->memory_size & (PAGE_SIZE - 1))
                goto out;

So I patched the userspace utilities to use 4096 instead.

With that patch, the guest saw the card, the driver got loaded,
and channel tuning works, but I get some i2c timeouts on the
guest side, and the host side has errors like:

------------[ cut here ]------------
Sep 22 02:28:54 [kernel] WARNING: at kernel/irq/manage.c:180
enable_irq+0x3a/0x55()
Sep 22 02:28:54 [kernel] Unbalanced enable for IRQ 20
Sep 22 02:28:54 [kernel] Modules linked in: sha256_generic cbc dm_crypt
crypto_blkcipher kvm_amd kvm bridge stp llc stv0297 budget_core dvb_core
saa7146 ttpci_eepr\
om ir_common k8temp i2c_core dm_snapshot dm_mirror dm_log scsi_wait_scan
[last unloaded: budget_ci]
Sep 22 02:28:54 [kernel] Pid: 5283, comm: qemu-system-x86 Tainted: G
    W 2.6.27-rc5-11874-g19561b6 #11
Sep 22 02:28:54 [kernel] Call Trace:
Sep 22 02:28:54 [kernel]  [<ffffffff80238b04>] warn_slowpath+0xb4/0xdc
Sep 22 02:28:54 [kernel]  [<ffffffff8026b099>]
__alloc_pages_internal+0xde/0x419
Sep 22 02:28:54 [kernel]  [<ffffffff802758d0>] get_user_pages+0x401/0x4ae
Sep 22 02:28:54 [kernel]  [<ffffffff80349269>] __next_cpu+0x19/0x26
Sep 22 02:28:54 [kernel]  [<ffffffff80230ce2>]
find_busiest_group+0x315/0x7c3
Sep 22 02:28:54 [kernel]  [<ffffffffa005de31>] gfn_to_hva+0x9/0x5d [kvm]
                - Last output repeated twice -
Sep 22 02:28:54 [kernel]  [<ffffffffa005e01b>]
kvm_read_guest_page+0x34/0x46 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa005e06c>] kvm_read_guest+0x3f/0x7c
[kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa0068bfe>]
paging64_walk_addr+0xe0/0x2c1 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffff80260d59>] enable_irq+0x3a/0x55
Sep 22 02:28:54 [kernel]  [<ffffffffa006df50>]
kvm_notify_acked_irq+0x17/0x30 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa00701c5>]
kvm_ioapic_update_eoi+0x2f/0x6e [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa006f6da>]
apic_mmio_write+0x24a/0x546 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa006498d>]
emulator_write_emulated_onepage+0xa1/0xf3 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffff802206f8>] paravirt_patch_call+0x13/0x2b
Sep 22 02:28:54 [kernel]  [<ffffffffa006c93e>]
x86_emulate_insn+0x366a/0x41de [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffff802206fa>] paravirt_patch_call+0x15/0x2b
Sep 22 02:28:54 [kernel]  [<ffffffffa005f90b>]
kvm_get_cs_db_l_bits+0x22/0x3a [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa006167d>]
emulate_instruction+0x198/0x25c [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa0067dfe>]
kvm_mmu_page_fault+0x46/0x83 [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa00636a9>]
kvm_arch_vcpu_ioctl_run+0x456/0x65c [kvm]
Sep 22 02:28:54 [kernel]  [<ffffffff8024d605>] hrtimer_start+0x111/0x133
Sep 22 02:28:54 [kernel]  [<ffffffffa005d451>] kvm_vcpu_ioctl+0xe0/0x459
[kvm]
Sep 22 02:28:54 [kernel]  [<ffffffffa005ee43>] kvm_vm_ioctl+0x203/0x21b
[kvm]
Sep 22 02:28:54 [kernel]  [<ffffffff802353b6>] finish_task_switch+0x2b/0xc4
Sep 22 02:28:54 [kernel]  [<ffffffff8029e3b5>] vfs_ioctl+0x21/0x6c
Sep 22 02:28:54 [kernel]  [<ffffffff8029e627>] do_vfs_ioctl+0x227/0x23d
Sep 22 02:28:54 [kernel]  [<ffffffff8029e67a>] sys_ioctl+0x3d/0x5f
Sep 22 02:28:54 [kernel]  [<ffffffff8020b45a>]
system_call_fastpath+0x16/0x1b
Sep 22 02:28:54 [kernel] ---[ end trace 7b8b990423985ddf ]---
Sep 22 02:28:54 [kernel] ------------[ cut here ]------------


Xen works with that card, but Xen has other problems, and kvm is much
nicer :) So if you need a guinea pig with basic debugging knowledge, I'm
your man.

Best regards,
Jan C. Bernauer

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Jan C. Bernauer wrote:
> Hi,
>
>  I have about the same problem, so excuse me for hijacking this thread.
>
> My hardware consists of a 780g/SB700 Mainboard and a 4850e AMD CPU, and
> I'm interested in forwarding a DVB-C tuner card to the guest. Maybe
> some NICs later.
>
> I tried and 'sort of' got it working with Amit's kernel and userspace
> tools.
> First thing:
> The dvb-c card has an interesting memory mapping, as reported by
> lspci -v:
>         Memory at cfdff000 (32-bit, non-prefetchable) [size=512]
>
> Size 512 doesn't fly with a check in kvm_main.c:
> if (mem->memory_size & (PAGE_SIZE - 1))
>                 goto out;
>
> So I patched the userspace utilities to use 4096 instead.
>
> With that patch, the guest saw the card, the driver got loaded,
> and channel tuning works, but I get some i2c timeouts on the
> guest side, and the host side has errors like:
>
> ------------[ cut here ]------------
> Sep 22 02:28:54 [kernel] WARNING: at kernel/irq/manage.c:180
> enable_irq+0x3a/0x55()
> Sep 22 02:28:54 [kernel] Unbalanced enable for IRQ 20
> Sep 22 02:28:54 [kernel] Modules linked in: sha256_generic cbc dm_crypt
> crypto_blkcipher kvm_amd kvm bridge stp llc stv0297 budget_core dvb_core
> saa7146 ttpci_eepr\
> om ir_common k8temp i2c_core dm_snapshot dm_mirror dm_log scsi_wait_scan
> [last unloaded: budget_ci]
> Sep 22 02:28:54 [kernel] Pid: 5283, comm: qemu-system-x86 Tainted: G
>     W 2.6.27-rc5-11874-g19561b6 #11
> Sep 22 02:28:54 [kernel] Call Trace:
> Sep 22 02:28:54 [kernel]  [<ffffffff80238b04>] warn_slowpath+0xb4/0xdc
> Sep 22 02:28:54 [kernel]  [<ffffffff8026b099>]
> __alloc_pages_internal+0xde/0x419
> Sep 22 02:28:54 [kernel]  [<ffffffff802758d0>] get_user_pages+0x401/0x4ae
> Sep 22 02:28:54 [kernel]  [<ffffffff80349269>] __next_cpu+0x19/0x26
> Sep 22 02:28:54 [kernel]  [<ffffffff80230ce2>]
> find_busiest_group+0x315/0x7c3
> Sep 22 02:28:54 [kernel]  [<ffffffffa005de31>] gfn_to_hva+0x9/0x5d [kvm]
>                 - Last output repeated twice -
> Sep 22 02:28:54 [kernel]  [<ffffffffa005e01b>]
> kvm_read_guest_page+0x34/0x46 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa005e06c>] kvm_read_guest+0x3f/0x7c
> [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa0068bfe>]
> paging64_walk_addr+0xe0/0x2c1 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffff80260d59>] enable_irq+0x3a/0x55
> Sep 22 02:28:54 [kernel]  [<ffffffffa006df50>]
> kvm_notify_acked_irq+0x17/0x30 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa00701c5>]
> kvm_ioapic_update_eoi+0x2f/0x6e [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa006f6da>]
> apic_mmio_write+0x24a/0x546 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa006498d>]
> emulator_write_emulated_onepage+0xa1/0xf3 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffff802206f8>]
> paravirt_patch_call+0x13/0x2b Sep 22 02:28:54 [kernel] 
> [<ffffffffa006c93e>]
> x86_emulate_insn+0x366a/0x41de [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffff802206fa>]
> paravirt_patch_call+0x15/0x2b Sep 22 02:28:54 [kernel] 
> [<ffffffffa005f90b>]
> kvm_get_cs_db_l_bits+0x22/0x3a [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa006167d>]
> emulate_instruction+0x198/0x25c [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa0067dfe>]
> kvm_mmu_page_fault+0x46/0x83 [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa00636a9>]
> kvm_arch_vcpu_ioctl_run+0x456/0x65c [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffff8024d605>] hrtimer_start+0x111/0x133
> Sep 22 02:28:54 [kernel]  [<ffffffffa005d451>] kvm_vcpu_ioctl+0xe0/0x459
> [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffffa005ee43>] kvm_vm_ioctl+0x203/0x21b
> [kvm]
> Sep 22 02:28:54 [kernel]  [<ffffffff802353b6>] finish_task_switch+0x2b/0xc4
> Sep 22 02:28:54 [kernel]  [<ffffffff8029e3b5>] vfs_ioctl+0x21/0x6c
> Sep 22 02:28:54 [kernel]  [<ffffffff8029e627>] do_vfs_ioctl+0x227/0x23d
> Sep 22 02:28:54 [kernel]  [<ffffffff8029e67a>] sys_ioctl+0x3d/0x5f
> Sep 22 02:28:54 [kernel]  [<ffffffff8020b45a>]
> system_call_fastpath+0x16/0x1b
> Sep 22 02:28:54 [kernel] ---[ end trace 7b8b990423985ddf ]---
> Sep 22 02:28:54 [kernel] ------------[ cut here ]------------
>
>
> Xen works with that card, but Xen has other problems, and kvm is much
> nicer :) So if you need a guinea pig with basic debugging knowledge, I'm
> your man.

How did you manage to pull together those patches? They all seem so old, and 
won't likely apply cleanly to git head :(

> Best regards,
> Jan C. Bernauer
>

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Jan C. Bernauer]

Thomas Fjellstrom wrote:
>
> How did you manage to pull together those patches? They all seem so
old, and
> won't likely apply cleanly to git head :(
>

Which patches do you mean? The patches for kvm?
There is a nice repository managed by Amit Shah:
Linux source:
http://git.kernel.org/?p=linux/kernel/git/amit/kvm.git;a=summary

kvm-userspace:
http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=summary

I think this compiled without too much problems.

Best regards,
Jan

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Jan C. Bernauer wrote:
> Thomas Fjellstrom wrote:
> > How did you manage to pull together those patches? They all seem so
>
> old, and
>
> > won't likely apply cleanly to git head :(
>
> Which patches do you mean? The patches for kvm?
> There is a nice repository managed by Amit Shah:
> Linux source:
> http://git.kernel.org/?p=linux/kernel/git/amit/kvm.git;a=summary
>
> kvm-userspace:
> http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=summary
>
> I think this compiled without too much problems.

Ah, the pci branches in the main kvm.git tree seemed to be months old, and I 
assumed thats where the code was kept.

> Best regards,
> Jan
>


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Thomas Fjellstrom wrote:
> On Saturday 27 September 2008, Jan C. Bernauer wrote:
> > Thomas Fjellstrom wrote:
> > > How did you manage to pull together those patches? They all seem so
> >
> > old, and
> >
> > > won't likely apply cleanly to git head :(
> >
> > Which patches do you mean? The patches for kvm?
> > There is a nice repository managed by Amit Shah:
> > Linux source:
> > http://git.kernel.org/?p=linux/kernel/git/amit/kvm.git;a=summary
> >
> > kvm-userspace:
> > http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=summar
> >y
> >
> > I think this compiled without too much problems.
>
> Ah, the pci branches in the main kvm.git tree seemed to be months old, and
> I assumed thats where the code was kept.
>
> > Best regards,
> > Jan

So I've checked out both of those trees and used head, and kvm-userspace is 
erroring out:

gcc -I. -I.. -I/root/kvm-amit-userspace/qemu/target-i386 -I/root/kvm-amit-
userspace/qemu -MMD -MT qemu-kvm-x86.o -MP -DNEED_CPU_H -D_GNU_SOURCE -
D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__user= -I/root/kvm-amit-
userspace/qemu/tcg -I/root/kvm-amit-userspace/qemu/tcg/x86_64 -I/root/kvm-
amit-userspace/qemu/fpu -DHAS_AUDIO -DHAS_AUDIO_CHOICE -I/root/kvm-amit-
userspace/qemu/slirp -I /root/kvm-amit-userspace/qemu/../libkvm  -DCONFIG_X86 
-Wall -O2 -g -fno-strict-aliasing  -m64 -I /root/kvm-amit-
userspace/kernel/include -c -o qemu-kvm-x86.o /root/kvm-amit-
userspace/qemu/qemu-kvm-x86.c
/root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:522: error: 
âKVM_FEATURE_CLOCKSOURCEâ undeclared here (not in a function)
/root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:525: error: 
âKVM_FEATURE_NOP_IO_DELAYâ undeclared here (not in a function)
/root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:528: error: âKVM_FEATURE_MMU_OPâ 
undeclared here (not in a function)

So I'm a little stuck now.

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Jan C. Bernauer]

Thomas Fjellstrom wrote:
> 
> So I've checked out both of those trees and used head, and kvm-userspace is 
> erroring out:
> 
> gcc -I. -I.. -I/root/kvm-amit-userspace/qemu/target-i386 -I/root/kvm-amit-
> userspace/qemu -MMD -MT qemu-kvm-x86.o -MP -DNEED_CPU_H -D_GNU_SOURCE -
> D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__user= -I/root/kvm-amit-
> userspace/qemu/tcg -I/root/kvm-amit-userspace/qemu/tcg/x86_64 -I/root/kvm-
> amit-userspace/qemu/fpu -DHAS_AUDIO -DHAS_AUDIO_CHOICE -I/root/kvm-amit-
> userspace/qemu/slirp -I /root/kvm-amit-userspace/qemu/../libkvm  -DCONFIG_X86 
> -Wall -O2 -g -fno-strict-aliasing  -m64 -I /root/kvm-amit-
> userspace/kernel/include -c -o qemu-kvm-x86.o /root/kvm-amit-
> userspace/qemu/qemu-kvm-x86.c
> /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:522: error: 
> âKVM_FEATURE_CLOCKSOURCEâ undeclared here (not in a function)
> /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:525: error: 
> âKVM_FEATURE_NOP_IO_DELAYâ undeclared here (not in a function)
> /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:528: error: âKVM_FEATURE_MMU_OPâ 
> undeclared here (not in a function)
> 
> So I'm a little stuck now.
> 


Try running
	make sync LINUX=/root/kvm-amit
(or whatever your kernel source dir is)
in the kernel sub directory of your kvm-amit-userspace dir.
Those KVM_FEATURE_* should be defined somewhere in kvm_para.h,
which is in there.

Best regards,
Jan

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Jan C. Bernauer wrote:
> Thomas Fjellstrom wrote:
> > So I've checked out both of those trees and used head, and kvm-userspace
> > is erroring out:
> >
> > gcc -I. -I.. -I/root/kvm-amit-userspace/qemu/target-i386
> > -I/root/kvm-amit- userspace/qemu -MMD -MT qemu-kvm-x86.o -MP -DNEED_CPU_H
> > -D_GNU_SOURCE - D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__user=
> > -I/root/kvm-amit- userspace/qemu/tcg
> > -I/root/kvm-amit-userspace/qemu/tcg/x86_64 -I/root/kvm-
> > amit-userspace/qemu/fpu -DHAS_AUDIO -DHAS_AUDIO_CHOICE -I/root/kvm-amit-
> > userspace/qemu/slirp -I /root/kvm-amit-userspace/qemu/../libkvm 
> > -DCONFIG_X86 -Wall -O2 -g -fno-strict-aliasing  -m64 -I /root/kvm-amit-
> > userspace/kernel/include -c -o qemu-kvm-x86.o /root/kvm-amit-
> > userspace/qemu/qemu-kvm-x86.c
> > /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:522: error:
> > âKVM_FEATURE_CLOCKSOURCEâ undeclared here (not in a function)
> > /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:525: error:
> > âKVM_FEATURE_NOP_IO_DELAYâ undeclared here (not in a function)
> > /root/kvm-amit-userspace/qemu/qemu-kvm-x86.c:528: error:
> > âKVM_FEATURE_MMU_OPâ undeclared here (not in a function)
> >
> > So I'm a little stuck now.
>
> Try running
> 	make sync LINUX=/root/kvm-amit
> (or whatever your kernel source dir is)
> in the kernel sub directory of your kvm-amit-userspace dir.
> Those KVM_FEATURE_* should be defined somewhere in kvm_para.h,
> which is in there.

that leaves me with:

/root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: âstruct 
kvm_msr_entryâ declared inside parameter list                                                                                 
/root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: its scope is 
only this definition or declaration, which is probably not what you want

and a bunch more errors.

> Best regards,
> Jan
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Jan C. Bernauer]

Thomas Fjellstrom wrote:
> 
> that leaves me with:
> 
> /root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: âstruct 
> kvm_msr_entryâ declared inside parameter list                                                                                 
> /root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: its scope is 
> only this definition or declaration, which is probably not what you want
> 
> and a bunch more errors.
> 
> 

Well, these are warning, and I might have ignored them :)
What are the errors?

Anyway, I'll be off now, so I won't respond till tomorrow.

Best regards,
Jan

Re: Status of pci passthrough work?

[Thomas Fjellstrom]

On Saturday 27 September 2008, Jan C. Bernauer wrote:
> Thomas Fjellstrom wrote:
> > that leaves me with:
> >
> > /root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: âstruct
> > kvm_msr_entryâ declared inside parameter list
> > /root/kvm-amit-userspace/qemu/../libkvm/libkvm.h:28: warning: its scope
> > is only this definition or declaration, which is probably not what you
> > want
> >
> > and a bunch more errors.
>
> Well, these are warning, and I might have ignored them :)
> What are the errors?
>
> Anyway, I'll be off now, so I won't respond till tomorrow.

libkvm.h:404: error: expected '=', ',' ';', 'asm' or '__attribute__' before 
'kvm_get_cr8'                                                                           
libkvm.c:145: error: expected declaration specifiers or '...' before '__u32'  


and quite a few more after that.
> Best regards,
> Jan

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Status of pci passthrough work?

[Avi Kivity]

Jan C. Bernauer wrote:
> Hi,
>
>  I have about the same problem, so excuse me for hijacking this thread.
>
> My hardware consists of a 780g/SB700 Mainboard and a 4850e AMD CPU, and
> I'm interested in forwarding a DVB-C tuner card to the guest. Maybe
> some NICs later.
>
> I tried and 'sort of' got it working with Amit's kernel and userspace
> tools.
> First thing:
> The dvb-c card has an interesting memory mapping, as reported by
> lspci -v:
>         Memory at cfdff000 (32-bit, non-prefetchable) [size=512]
>
> Size 512 doesn't fly with a check in kvm_main.c:
> if (mem->memory_size & (PAGE_SIZE - 1))
>                 goto out;
>
> So I patched the userspace utilities to use 4096 instead.
>
> With that patch, the guest saw the card, the driver got loaded,
> and channel tuning works, but I get some i2c timeouts on the
> guest side, and the host side has errors like:
>
> ------------[ cut here ]------------
> Sep 22 02:28:54 [kernel] WARNING: at kernel/irq/manage.c:180
> enable_irq+0x3a/0x55()
> Sep 22 02:28:54 [kernel] Unbalanced enable for IRQ 20
>   

That looks due to bad error handling, due to the failures you had before.

Try rebooting the host and starting again with the patched userspace.

Amit, can you take a look at the error handling paths?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Status of pci passthrough work?

[Amit Shah]

* On Sunday 28 Sep 2008 10:09:13 Avi Kivity wrote:
> Jan C. Bernauer wrote:
> > Hi,
> >
> >  I have about the same problem, so excuse me for hijacking this thread.
> >
> > My hardware consists of a 780g/SB700 Mainboard and a 4850e AMD CPU, and
> > I'm interested in forwarding a DVB-C tuner card to the guest. Maybe
> > some NICs later.
> >
> > I tried and 'sort of' got it working with Amit's kernel and userspace
> > tools.
> > First thing:
> > The dvb-c card has an interesting memory mapping, as reported by
> > lspci -v:
> >         Memory at cfdff000 (32-bit, non-prefetchable) [size=512]
> >
> > Size 512 doesn't fly with a check in kvm_main.c:
> > if (mem->memory_size & (PAGE_SIZE - 1))
> >                 goto out;
> >
> > So I patched the userspace utilities to use 4096 instead.
> >
> > With that patch, the guest saw the card, the driver got loaded,
> > and channel tuning works, but I get some i2c timeouts on the
> > guest side, and the host side has errors like:
> >
> > ------------[ cut here ]------------
> > Sep 22 02:28:54 [kernel] WARNING: at kernel/irq/manage.c:180
> > enable_irq+0x3a/0x55()
> > Sep 22 02:28:54 [kernel] Unbalanced enable for IRQ 20
>
> That looks due to bad error handling, due to the failures you had before.
>
> Try rebooting the host and starting again with the patched userspace.
>
> Amit, can you take a look at the error handling paths?

I suspect this is so because Jan's using very old trees. The 'master' branch 
used to hold the device assignment patches, They're now upstream.

The patches in the 'vtd' branch are also upstream.

The 'pvdma' branch is the one I should be updating. Currently, it's based on 
an older upstream kvm.git. I'll refresh it and drop a note.

Amit

Re: Status of pci passthrough work?

[Amit Shah]

Hello,

----- "xming" <xmingske@gmail.com> wrote:

> When can we expect pvdma updates? Is it ever going to be merged into
> mainline kvm?

The pvdma tree at

http://git.kernel.org/?p=linux/kernel/git/amit/kvm.git;a=shortlog;h=pvdma

is based of an older Linux version.

It's usable; but not ported to newer kernel versions. I can't say when I'll get around doing it. In the meanwhile if someone else is interested, drop me a line.

Amit.

Re: Status of pci passthrough work?

[xming]

> It's usable; but not ported to newer kernel versions. I can't say when I'll get around doing it. In the meanwhile if someone else is interested, drop me a line.

Do you mean interested in porting to newer versions (kvm + kernel) or
interested
in to use pvdma?

If it's the latter, I am.

Re: Status of pci passthrough work?

[Amit Shah]

----- "xming" <xmingske@gmail.com> wrote:

> > It's usable; but not ported to newer kernel versions. I can't say
> when I'll get around doing it. In the meanwhile if someone else is
> interested, drop me a line.
> 
> Do you mean interested in porting to newer versions (kvm + kernel) or
> interested
> in to use pvdma?
> 
> If it's the latter, I am.

I mean interested in porting it.

RE: Status of pci passthrough work?

[Passera, Pablo R]

Amit,
        I trying to use PVDMA. I've downloaded a kernel snapshot from the your kvm git, but I couldn't download a snapshot or the repo from your kvm-userspace tree. I tried to launch the VM using kvm-85 user space but it hangs before loading it. Should it work with kvm-85 user space? Do you have the userspace patches for PVDMA?

Thanks,
Pablo

>-----Original Message-----
>From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
>Behalf Of Amit Shah
>Sent: Tuesday, December 16, 2008 12:07 PM
>To: xming
>Cc: Thomas Fjellstrom; kvm@vger.kernel.org
>Subject: Re: Status of pci passthrough work?
>
>Hello,
>
>----- "xming" <xmingske@gmail.com> wrote:
>
>> When can we expect pvdma updates? Is it ever going to be merged into
>> mainline kvm?
>
>The pvdma tree at
>
>http://git.kernel.org/?p=linux/kernel/git/amit/kvm.git;a=shortlog;h=pvdm
>a
>
>is based of an older Linux version.
>
>It's usable; but not ported to newer kernel versions. I can't say when
>I'll get around doing it. In the meanwhile if someone else is
>interested, drop me a line.
>
>Amit.
>--
>To unsubscribe from this list: send the line "unsubscribe kvm" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of pci passthrough work?

[Amit Shah]

Hello,

On (Thu) May 14 2009 [11:08:29], Passera, Pablo R wrote:
> Amit,
>         I trying to use PVDMA. I've downloaded a kernel snapshot from the your kvm git, but I couldn't download a snapshot or the repo from your kvm-userspace tree. I tried to launch the VM using kvm-85 user space but it hangs before loading it. Should it work with kvm-85 user space? Do you have the userspace patches for PVDMA?

The pvdma userspace patches are at

http://git.kernel.org/?p=linux/kernel/git/amit/kvm-userspace.git;a=shortlog;h=pvdma

(look for the branch 'pvdma' in the tree).

		Amit

RE: Status of pci passthrough work?

[Passera, Pablo R]

Hi Amit,
        Thanks for your answer. I was able to get your userspace pvdma version. So now, I am using the PVDMA patched kernel and the PVDMA patches userspace. However, I am not able to start the VM. I am running qemu with the following options (I am trying without any pci passthrough first)

./qemu/x86_64-softmmu/qemu-system-x86_64 -hda /root/kvm/dm2.img -m 256 -net none

The SDL windows appear but it hangs after showing the message "Press F12 for boot menu.". I am not getting any message neither in qemu nor in dmesg. Do you know what could be happening? May be a kernel compile option? It would be great if you can send me the .config file that you used to compile it, just to check the options.

Thanks,
Pablo

>-----Original Message-----
>From: Amit Shah [mailto:amit.shah@redhat.com]
>Sent: Friday, May 15, 2009 8:00 AM
>To: Passera, Pablo R
>Cc: kvm@vger.kernel.org
>Subject: Re: Status of pci passthrough work?
>
>Hello,
>
>On (Thu) May 14 2009 [11:08:29], Passera, Pablo R wrote:
>> Amit,
>>         I trying to use PVDMA. I've downloaded a kernel snapshot from
>the your kvm git, but I couldn't download a snapshot or the repo from
>your kvm-userspace tree. I tried to launch the VM using kvm-85 user
>space but it hangs before loading it. Should it work with kvm-85 user
>space? Do you have the userspace patches for PVDMA?
>
>The pvdma userspace patches are at
>
>http://git.kernel.org/?p=linux/kernel/git/amit/kvm-
>userspace.git;a=shortlog;h=pvdma
>
>(look for the branch 'pvdma' in the tree).
>
>               Amit

Re: Status of pci passthrough work?

[Amit Shah]

On (Fri) May 15 2009 [07:14:05], Passera, Pablo R wrote:
> Hi Amit,
>         Thanks for your answer. I was able to get your userspace pvdma version. So now, I am using the PVDMA patched kernel and the PVDMA patches userspace. However, I am not able to start the VM. I am running qemu with the following options (I am trying without any pci passthrough first)
> 
> ./qemu/x86_64-softmmu/qemu-system-x86_64 -hda /root/kvm/dm2.img -m 256 -net none
> 
> The SDL windows appear but it hangs after showing the message "Press F12 for boot menu.". I am not getting any message neither in qemu nor in dmesg. Do you know what could be happening? May be a kernel compile option? It would be great if you can send me the .config file that you used to compile it, just to check the options.

Can you try out a few things, like booting off the 'avi' branch in
userspace and / or the 'kvm' branch of the kernel tree? Just to rule out
the bugs in the device assignment code.

		Amit

RE: Status of pci passthrough work?

[Passera, Pablo R]

Hi Amit,

        Please correct me if I am wrong, but the fact that the PVDMA module is located in the guest imposes a security problem. So, if someone in the guest has root access, he could modify the PVDMA module and then corrupt the host memory. Because of this the guest has to be trusted. I had an idea that I would like to check with you to see if this is feasible.

        What if I need pci passthrough in a known hardware platform where I know all the pci hardware devices, could I intercept the DMA commands to these PCI devices at qemu level and then do the translation from pseudo physical to physical memory address at that level? Since I know which are the devices present in the platform, I also know which are the PCI commands to configure the DMA transfers in the PCI device. Before forwarding this to the real PCI I could change the memory address to a physical one.

Do you think this is possible?

Thanks,
Pablo


>-----Original Message-----
>From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
>Behalf Of Amit Shah
>Sent: Friday, May 15, 2009 10:38 AM
>To: Passera, Pablo R
>Cc: kvm@vger.kernel.org
>Subject: Re: Status of pci passthrough work?
>
>On (Fri) May 15 2009 [07:14:05], Passera, Pablo R wrote:
>> Hi Amit,
>>         Thanks for your answer. I was able to get your userspace pvdma
>version. So now, I am using the PVDMA patched kernel and the PVDMA
>patches userspace. However, I am not able to start the VM. I am running
>qemu with the following options (I am trying without any pci passthrough
>first)
>>
>> ./qemu/x86_64-softmmu/qemu-system-x86_64 -hda /root/kvm/dm2.img -m 256
>-net none
>>
>> The SDL windows appear but it hangs after showing the message "Press
>F12 for boot menu.". I am not getting any message neither in qemu nor in
>dmesg. Do you know what could be happening? May be a kernel compile
>option? It would be great if you can send me the .config file that you
>used to compile it, just to check the options.
>
>Can you try out a few things, like booting off the 'avi' branch in
>userspace and / or the 'kvm' branch of the kernel tree? Just to rule out
>the bugs in the device assignment code.
>
>               Amit
>--
>To unsubscribe from this list: send the line "unsubscribe kvm" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of pci passthrough work?

[Amit Shah]

Hi Pablo,

On (Fri) May 29 2009 [11:41:58], Passera, Pablo R wrote:
> Hi Amit,
> 
>         Please correct me if I am wrong, but the fact that the PVDMA module is located in the guest imposes a security problem. So, if someone in the guest has root access, he could modify the PVDMA module and then corrupt the host memory. Because of this the guest has to be trusted. I had an idea that I would like to check with you to see if this is feasible.

This indeed is possible. A combination of an IOMMU along with PVDMA will
prevent this. The host can program the IOMMU to only allow guests to
access specific memory areas.

>         What if I need pci passthrough in a known hardware platform where I know all the pci hardware devices, could I intercept the DMA commands to these PCI devices at qemu level and then do the translation from pseudo physical to physical memory address at that level? Since I know which are the devices present in the platform, I also know which are the PCI commands to configure the DMA transfers in the PCI device. Before forwarding this to the real PCI I could change the memory address to a physical one.

The way device assignment works is you hand over the device to the guest
completely. The guest then programs the device registers for the DMA
addresses. If you modify qemu to intercept the MMIO for assigned devices
(this was what was done in very initial device assignment code, even
before it got merged) and verify that the addresses that are being
written to the device dma registers are from the accepted range, you can
provide the necessary security (for this case at least).

		Amit