Intel S3420GPLX VT-d VF question

Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 I have successfully booted xen on Fedora 14 x86_64 with iommu=force on 
 the kernel line, I can unbind a pci device and rebind to pci-stub and 
 see it with xm pci-list-assignable-devices.  That's cool.  Now I have an 
 Intel 82576 dual-port ET Adapter that supports SR-IOV and started 
 configuring everything to enable it's virtual functions:
 added pci_pt_e820_access=on to the module line that contains the 
 vmlinuz
 created igb.conf under /etc/modprobe.d with options igb max_vfs=7
 added blacklist igbvf to /etc/modprobe.d/blacklist.conf

 I was under the impression that I should see some kind of message about 
 MMCONFIG in xm dmesg, but there isn't anything.  I do a xm dmesg | grep 
 -i pci and it finds nothing with MMCONFIG.  I guess my next question is 
 what do I do next?  How do I troubleshoot further?  Below is the VT-d 
 messages from xm dmesg:
 (XEN) Intel VT-d Snoop Control supported.
 (XEN) Intel VT-d DMA Passthrough not supported.
 (XEN) Intel VT-d Queued Invalidation supported.
 (XEN) Intel VT-d Interrupt Remapping not supported.

 Does MMCONFIG have something to do with the fact the Interrupt 
 Remapping not supported message is displayed?
 Mike

Re: Intel S3420GPLX VT-d VF question

[Konrad Rzeszutek Wilk]

On Tue, Dec 07, 2010 at 11:52:24PM -0500, Michael A. Collins wrote:
> I have successfully booted xen on Fedora 14 x86_64 with iommu=force

Ok.. Um, what Fedora kernel are we taking about here? Do you have
the full serial log?
> on the kernel line, I can unbind a pci device and rebind to pci-stub
> and see it with xm pci-list-assignable-devices.  That's cool.  Now I
> have an Intel 82576 dual-port ET Adapter that supports SR-IOV and
> started configuring everything to enable it's virtual functions:
> added pci_pt_e820_access=on to the module line that contains the
> vmlinuz

Why? It doesn't automatically load without that parameter?
> created igb.conf under /etc/modprobe.d with options igb max_vfs=7
> added blacklist igbvf to /etc/modprobe.d/blacklist.conf

Why blacklist it?
> 
> I was under the impression that I should see some kind of message
> about MMCONFIG in xm dmesg, but there isn't anything.  I do a xm
> dmesg | grep -i pci and it finds nothing with MMCONFIG.  I guess my

Huh? Why? Do you see the 7 ethernet PCI device being created?

> next question is what do I do next?  How do I troubleshoot further?

What are you trying to troubleshoot? Did you lspci output change
after you loaded the igbvf?

> Below is the VT-d messages from xm dmesg:
> (XEN) Intel VT-d Snoop Control supported.
> (XEN) Intel VT-d DMA Passthrough not supported.
> (XEN) Intel VT-d Queued Invalidation supported.
> (XEN) Intel VT-d Interrupt Remapping not supported.
> 
> Does MMCONFIG have something to do with the fact the Interrupt
> Remapping not supported message is displayed?

No. It just means your VT-D chipset does not support interrupt
remapping.

> Mike
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Wed, 8 Dec 2010 11:32:18 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
> On Tue, Dec 07, 2010 at 11:52:24PM -0500, Michael A. Collins wrote:
>> I have successfully booted xen on Fedora 14 x86_64 with iommu=force
>
> Ok.. Um, what Fedora kernel are we taking about here? Do you have
> the full serial log?
>> on the kernel line, I can unbind a pci device and rebind to pci-stub
>> and see it with xm pci-list-assignable-devices.  That's cool.  Now I
>> have an Intel 82576 dual-port ET Adapter that supports SR-IOV and
>> started configuring everything to enable it's virtual functions:
>> added pci_pt_e820_access=on to the module line that contains the
>> vmlinuz
>
> Why? It doesn't automatically load without that parameter?
>> created igb.conf under /etc/modprobe.d with options igb max_vfs=7
>> added blacklist igbvf to /etc/modprobe.d/blacklist.conf
>
> Why blacklist it?
>>
>> I was under the impression that I should see some kind of message
>> about MMCONFIG in xm dmesg, but there isn't anything.  I do a xm
>> dmesg | grep -i pci and it finds nothing with MMCONFIG.  I guess my
>
> Huh? Why? Do you see the 7 ethernet PCI device being created?
>
>> next question is what do I do next?  How do I troubleshoot further?
>
> What are you trying to troubleshoot? Did you lspci output change
> after you loaded the igbvf?
>
>> Below is the VT-d messages from xm dmesg:
>> (XEN) Intel VT-d Snoop Control supported.
>> (XEN) Intel VT-d DMA Passthrough not supported.
>> (XEN) Intel VT-d Queued Invalidation supported.
>> (XEN) Intel VT-d Interrupt Remapping not supported.
>>
>> Does MMCONFIG have something to do with the fact the Interrupt
>> Remapping not supported message is displayed?
>
> No. It just means your VT-D chipset does not support interrupt
> remapping.
>
>> Mike
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


 I didn't know if this actually made it on the list, since it didn't 
 show up in my inbox when I sent it, so the second email from me is a 
 duplicate.  Sorry.

 The kernel I am using is myoung's pre-built rpm from 
 http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm. 
  I am also using the 4.0.1 rpms from 
 http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.
 I am following the instructions from 
 http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial.

 The pci_pt_e820_access=on setting was taken from the following blog: 
 http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.  
 I was under the impression that that option had to be set for the VFs to 
 be seen by the kernel.
 I have not tried it without, but will do that tonight.  The blacklist 
 settings was referenced in the above article, so I stuck it in.  I will 
 try to load the igbvf tonight as well.

 I do not see the 7 Virtual Functions in lspci.

 Thanks for the reply, I will try without the e820 option and loading 
 the igbvf module tonight if that works, I'll let you know.  If not, I'll 
 respond with the full xen dmesg output.
 Mike

Re: Intel S3420GPLX VT-d VF question

[Konrad Rzeszutek Wilk]

> The kernel I am using is myoung's pre-built rpm from http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm.

Sweet. That is quite up-to-date.

> I am also using the 4.0.1 rpms from
> http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.

Uhh, try a more modern version. The xen-unstable.hg - you will have
to compile it yourself. Pasi's awesome PVOPS Wiki mentions the appropiate
steps and I think Boris's blog has details as well.

> I am following the instructions from
> http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial.
> 
> The pci_pt_e820_access=on setting was taken from the following blog: http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.

Oh gosh. That is with 2.6.18 and Xen 3.4.

> I was under the impression that that option had to be set for the
> VFs to be seen by the kernel.

Well, they were workarounds for that verison of kernel. It might be
that they are required in 2.6.32.. but lets assume that those
are actually fixed.

> I have not tried it without, but will do that tonight.  The
> blacklist settings was referenced in the above article, so I stuck
> it in.  I will try to load the igbvf tonight as well.

Yeah, try it. If it works you should see suddenly seven extra NICs
(ifconfig -a) in your box. If that works then we know the kernel
driver is working. 
> 
> I do not see the 7 Virtual Functions in lspci.

Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)"

must show up. That is the proof that your max_ig.. parameter worked and the card
is now able to do SR-IOV. You might have need to upgrade your lspci - I think I had
it as "Unknown device" at some point.

Anyhow, if you see those extra devices, _then_ you can blacklist the igvf module.

> 
> Thanks for the reply, I will try without the e820 option and loading
> the igbvf module tonight if that works, I'll let you know.  If not,
> I'll respond with the full xen dmesg output.

Take your time.

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
>> The kernel I am using is myoung's pre-built rpm from 
>> http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm.
>
> Sweet. That is quite up-to-date.
>
>> I am also using the 4.0.1 rpms from
>> http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.
>
> Uhh, try a more modern version. The xen-unstable.hg - you will have
> to compile it yourself. Pasi's awesome PVOPS Wiki mentions the 
> appropiate
> steps and I think Boris's blog has details as well.
>
>> I am following the instructions from
>> http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial.
>>
>> The pci_pt_e820_access=on setting was taken from the following blog: 
>> http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.
>
> Oh gosh. That is with 2.6.18 and Xen 3.4.
>
>> I was under the impression that that option had to be set for the
>> VFs to be seen by the kernel.
>
> Well, they were workarounds for that verison of kernel. It might be
> that they are required in 2.6.32.. but lets assume that those
> are actually fixed.
>
>> I have not tried it without, but will do that tonight.  The
>> blacklist settings was referenced in the above article, so I stuck
>> it in.  I will try to load the igbvf tonight as well.
>
> Yeah, try it. If it works you should see suddenly seven extra NICs
> (ifconfig -a) in your box. If that works then we know the kernel
> driver is working.
>>
>> I do not see the 7 Virtual Functions in lspci.
>
> Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576
> Gigabit Network Connection (rev 01)
> 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
> 01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)"
>
> must show up. That is the proof that your max_ig.. parameter worked
> and the card
> is now able to do SR-IOV. You might have need to upgrade your lspci -
> I think I had
> it as "Unknown device" at some point.
>
> Anyhow, if you see those extra devices, _then_ you can blacklist the
> igvf module.
>
>>
>> Thanks for the reply, I will try without the e820 option and loading
>> the igbvf module tonight if that works, I'll let you know.  If not,
>> I'll respond with the full xen dmesg output.
>
> Take your time.


 Ok, I did try to remove the e820 option and got rid of the igbvf 
 blacklisting.  No change, the Virtual Function entries are not showing 
 up.  I beleieve that I may be jumping ahead of myself.  I am going to 
 blow away the box and reload with Fedora 14 again, but not move forward 
 with the Xen install until after I verify that SR-IOV works as it 
 should.  I will keep you posted, but I probably won't start till this 
 weekend.
 Mike

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
>> The kernel I am using is myoung's pre-built rpm from 
>> http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm.
>
> Sweet. That is quite up-to-date.
>
>> I am also using the 4.0.1 rpms from
>> http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.
>
> Uhh, try a more modern version. The xen-unstable.hg - you will have
> to compile it yourself. Pasi's awesome PVOPS Wiki mentions the 
> appropiate
> steps and I think Boris's blog has details as well.
>
>> I am following the instructions from
>> http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial.
>>
>> The pci_pt_e820_access=on setting was taken from the following blog: 
>> http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.
>
> Oh gosh. That is with 2.6.18 and Xen 3.4.
>
>> I was under the impression that that option had to be set for the
>> VFs to be seen by the kernel.
>
> Well, they were workarounds for that verison of kernel. It might be
> that they are required in 2.6.32.. but lets assume that those
> are actually fixed.
>
>> I have not tried it without, but will do that tonight.  The
>> blacklist settings was referenced in the above article, so I stuck
>> it in.  I will try to load the igbvf tonight as well.
>
> Yeah, try it. If it works you should see suddenly seven extra NICs
> (ifconfig -a) in your box. If that works then we know the kernel
> driver is working.
>>
>> I do not see the 7 Virtual Functions in lspci.
>
> Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576
> Gigabit Network Connection (rev 01)
> 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
> 01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)
> 01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function
> (rev 01)"
>
> must show up. That is the proof that your max_ig.. parameter worked
> and the card
> is now able to do SR-IOV. You might have need to upgrade your lspci -
> I think I had
> it as "Unknown device" at some point.
>
> Anyhow, if you see those extra devices, _then_ you can blacklist the
> igvf module.
>
>>
>> Thanks for the reply, I will try without the e820 option and loading
>> the igbvf module tonight if that works, I'll let you know.  If not,
>> I'll respond with the full xen dmesg output.
>
> Take your time.

 Seems as the igb module was loaded in the initramfs and was not 
 reloaded using the options specified in my igb.conf file under 
 /etc/modprobe.d.  I rmmod igb and did a modprobe igb max_vfs=7 and 
 suddenly all the Virtual Functions showed up.  So that was the problem 
 all along.  Until I figure out how to edit the initramfs to load igb 
 with max_vfs option, I'm putting the rmmod and modprobe in the rc.local 
 file, that seems to work well and it gets executed before xend.  I am 
 build xen-4.0-testing to replace the rpms I used.  Once all that is done 
 and I test passing through one of the Virtual Functions to a vm, I'll 
 report back.
 Mike

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" 
 <mike@ark-net.org> wrote:
> On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> Take your time.
>
> Seems as the igb module was loaded in the initramfs and was not
> reloaded using the options specified in my igb.conf file under
> /etc/modprobe.d.  I rmmod igb and did a modprobe igb max_vfs=7 and
> suddenly all the Virtual Functions showed up.  So that was the 
> problem
> all along.  Until I figure out how to edit the initramfs to load igb
> with max_vfs option, I'm putting the rmmod and modprobe in the
> rc.local file, that seems to work well and it gets executed before
> xend.  I am build xen-4.0-testing to replace the rpms I used.  Once
> all that is done and I test passing through one of the Virtual
> Functions to a vm, I'll report back.
> Mike


 Ok, so I can pass one of the virtual functions through to a HVM.  I 
 think I'm still not out of the water though.  It's acting very strangely 
 and I have not successfully been able to very layer-2 connectivity with 
 it.  I am seeing the following in my HVM's qemu log:
 register_real_device: Assigning real physical device 04:10.4 ...
 pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function
 pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: 
 No such file or directory: 0x4:0x10.0x4
 pt_register_regions: IO region registered (size=0x00004000 
 base_addr=0xb2448004)
 pt_register_regions: IO region registered (size=0x00004000 
 base_addr=0xb2468004)
 pt_msix_init: get MSI-X table bar base b2468000
 pt_msix_init: table_off = 0, total_entries = 3
 pt_msix_init: errno = 2
 pt_msix_init: mapping physical MSI-X table to 7fe3c4359000
 register_real_device: Real physical device 04:10.4 registered 
 successfuly!
 IRQ type = INTx
 generate a sci for PHP.
 deassert due to disable GPE bit.
 pt_pci_write_config: Warning: Guest attempt to set address to unused 
 Base Address Register. [00:06.0][Offset:30h][Length:4]
 pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 index=0 
 first_map=1
 pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 index=3 
 first_map=1


 Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which 
 leads me to believe that the pass through isn't actually working.  I am 
 still using the mayoung built xen-4.0.1-6 rpms from koji, since I have 
 not been able to successfully compile xen stubdom.  I guess gcc 4.5 is 
 just too new and untested for me to troubleshoot and get through.  
 Hopefully, someone on here can point me in the right direction.
 Mike

Re: Intel S3420GPLX VT-d VF question

[Konrad Rzeszutek Wilk]

On Mon, Dec 20, 2010 at 09:24:17PM -0500, Michael A. Collins wrote:
> On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins"
> <mike@ark-net.org> wrote:
> >On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk
> ><konrad.wilk@oracle.com> wrote:
> >>Take your time.
> >
> >Seems as the igb module was loaded in the initramfs and was not
> >reloaded using the options specified in my igb.conf file under
> >/etc/modprobe.d.  I rmmod igb and did a modprobe igb max_vfs=7 and
> >suddenly all the Virtual Functions showed up.  So that was the
> >problem
> >all along.  Until I figure out how to edit the initramfs to load igb
> >with max_vfs option, I'm putting the rmmod and modprobe in the
> >rc.local file, that seems to work well and it gets executed before
> >xend.  I am build xen-4.0-testing to replace the rpms I used.  Once
> >all that is done and I test passing through one of the Virtual
> >Functions to a vm, I'll report back.
> >Mike
> 
> 
> Ok, so I can pass one of the virtual functions through to a HVM.  I
> think I'm still not out of the water though.  It's acting very
> strangely and I have not successfully been able to very layer-2

And if you tried to use the device from dom0, or even baremetal, you
had not trouble with the VIFs, right?

> connectivity with it.  I am seeing the following in my HVM's qemu
> log:
> register_real_device: Assigning real physical device 04:10.4 ...
> pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function
> pt_iomul_init: Error: pt_iomul_init can't open file
> /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4
> pt_register_regions: IO region registered (size=0x00004000
> base_addr=0xb2448004)
> pt_register_regions: IO region registered (size=0x00004000
> base_addr=0xb2468004)
> pt_msix_init: get MSI-X table bar base b2468000
> pt_msix_init: table_off = 0, total_entries = 3
> pt_msix_init: errno = 2
                ^^^^^^^
That does not sound good, but it might be just b/c it could
not find /dev/xen/pci_iomul.

> pt_msix_init: mapping physical MSI-X table to 7fe3c4359000
> register_real_device: Real physical device 04:10.4 registered
> successfuly!
> IRQ type = INTx
> generate a sci for PHP.
> deassert due to disable GPE bit.
> pt_pci_write_config: Warning: Guest attempt to set address to unused
> Base Address Register. [00:06.0][Offset:30h][Length:4]
> pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384
> index=0 first_map=1
> pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384
> index=3 first_map=1
> 
> 
> Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which
> leads me to believe that the pass through isn't actually working.  I
> am still using the mayoung built xen-4.0.1-6 rpms from koji, since I
> have not been able to successfully compile xen stubdom.  I guess gcc
> 4.5 is just too new and untested for me to troubleshoot and get
> through.  Hopefully, someone on here can point me in the right
> direction.

Hmm. I am bit out of ideas.. Do you know which line in the QEMU code
would trigger that? You could instrument QEMU a bit?

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Wed, 22 Dec 2010 15:10:12 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
> On Mon, Dec 20, 2010 at 09:24:17PM -0500, Michael A. Collins wrote:
>> On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins"
>> <mike@ark-net.org> wrote:
>> >On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk
>> ><konrad.wilk@oracle.com> wrote:
>> >>Take your time.
>> >
>> >Seems as the igb module was loaded in the initramfs and was not
>> >reloaded using the options specified in my igb.conf file under
>> >/etc/modprobe.d.  I rmmod igb and did a modprobe igb max_vfs=7 and
>> >suddenly all the Virtual Functions showed up.  So that was the
>> >problem
>> >all along.  Until I figure out how to edit the initramfs to load 
>> igb
>> >with max_vfs option, I'm putting the rmmod and modprobe in the
>> >rc.local file, that seems to work well and it gets executed before
>> >xend.  I am build xen-4.0-testing to replace the rpms I used.  Once
>> >all that is done and I test passing through one of the Virtual
>> >Functions to a vm, I'll report back.
>> >Mike
>>
>>
>> Ok, so I can pass one of the virtual functions through to a HVM.  I
>> think I'm still not out of the water though.  It's acting very
>> strangely and I have not successfully been able to very layer-2
>
> And if you tried to use the device from dom0, or even baremetal, you
> had not trouble with the VIFs, right?
>
>> connectivity with it.  I am seeing the following in my HVM's qemu
>> log:
>> register_real_device: Assigning real physical device 04:10.4 ...
>> pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function
>> pt_iomul_init: Error: pt_iomul_init can't open file
>> /dev/xen/pci_iomul: No such file or directory: 0x4:0x10.0x4
>> pt_register_regions: IO region registered (size=0x00004000
>> base_addr=0xb2448004)
>> pt_register_regions: IO region registered (size=0x00004000
>> base_addr=0xb2468004)
>> pt_msix_init: get MSI-X table bar base b2468000
>> pt_msix_init: table_off = 0, total_entries = 3
>> pt_msix_init: errno = 2
>                 ^^^^^^^
> That does not sound good, but it might be just b/c it could
> not find /dev/xen/pci_iomul.
>
>> pt_msix_init: mapping physical MSI-X table to 7fe3c4359000
>> register_real_device: Real physical device 04:10.4 registered
>> successfuly!
>> IRQ type = INTx
>> generate a sci for PHP.
>> deassert due to disable GPE bit.
>> pt_pci_write_config: Warning: Guest attempt to set address to unused
>> Base Address Register. [00:06.0][Offset:30h][Length:4]
>> pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384
>> index=0 first_map=1
>> pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384
>> index=3 first_map=1
>>
>>
>> Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which
>> leads me to believe that the pass through isn't actually working.  I
>> am still using the mayoung built xen-4.0.1-6 rpms from koji, since I
>> have not been able to successfully compile xen stubdom.  I guess gcc
>> 4.5 is just too new and untested for me to troubleshoot and get
>> through.  Hopefully, someone on here can point me in the right
>> direction.
>
> Hmm. I am bit out of ideas.. Do you know which line in the QEMU code
> would trigger that? You could instrument QEMU a bit?


 Yeah, the VIFs are fine in dom0.  Also, I have seen reports of other 
 people receiving the /dev/xen/pci_iomul and still getting past it.  I 
 will dig around in the QEMU code for awhile and see what I can come up 
 with.  Thanks.
 Mike

RE: Intel S3420GPLX VT-d VF question

[Zhuo Song]

Can you attach some logs by dmesg in your guest just after *igbvf* module inserted?
Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you ever checked the physical ports in your card? 
If you the card takes two ports, generally, one is for odd and the other is for even function of the BDF.

Best Regards

Zhuo Song
Gif me a visky, ginger ale on the side.

---------
 On Thu, 09 Dec 2010 22:18:45 -0500, "Michael A. Collins" 
 <mike@ark-net.org> wrote:
> On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> Take your time.
>
> Seems as the igb module was loaded in the initramfs and was not
> reloaded using the options specified in my igb.conf file under
> /etc/modprobe.d.  I rmmod igb and did a modprobe igb max_vfs=7 and
> suddenly all the Virtual Functions showed up.  So that was the 
> problem
> all along.  Until I figure out how to edit the initramfs to load igb
> with max_vfs option, I'm putting the rmmod and modprobe in the
> rc.local file, that seems to work well and it gets executed before
> xend.  I am build xen-4.0-testing to replace the rpms I used.  Once
> all that is done and I test passing through one of the Virtual
> Functions to a vm, I'll report back.
> Mike


 Ok, so I can pass one of the virtual functions through to a HVM.  I 
 think I'm still not out of the water though.  It's acting very strangely 
 and I have not successfully been able to very layer-2 connectivity with 
 it.  I am seeing the following in my HVM's qemu log:
 register_real_device: Assigning real physical device 04:10.4 ...
 pt_dev_is_virtfn: 0000:04:10.4 is a SR-IOV Virtual Function
 pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: 
 No such file or directory: 0x4:0x10.0x4
 pt_register_regions: IO region registered (size=0x00004000 
 base_addr=0xb2448004)
 pt_register_regions: IO region registered (size=0x00004000 
 base_addr=0xb2468004)
 pt_msix_init: get MSI-X table bar base b2468000
 pt_msix_init: table_off = 0, total_entries = 3
 pt_msix_init: errno = 2
 pt_msix_init: mapping physical MSI-X table to 7fe3c4359000
 register_real_device: Real physical device 04:10.4 registered 
 successfuly!
 IRQ type = INTx
 generate a sci for PHP.
 deassert due to disable GPE bit.
 pt_pci_write_config: Warning: Guest attempt to set address to unused 
 Base Address Register. [00:06.0][Offset:30h][Length:4]
 pt_iomem_map: e_phys=20000000 maddr=b2448000 type=0 len=16384 index=0 
 first_map=1
 pt_iomem_map: e_phys=20004000 maddr=b2468000 type=0 len=16384 index=3 
 first_map=1


 Now, I never see a pt_msi_setup: msi mapped with pirq XX line, which 
 leads me to believe that the pass through isn't actually working.  I am 
 still using the mayoung built xen-4.0.1-6 rpms from koji, since I have 
 not been able to successfully compile xen stubdom.  I guess gcc 4.5 is 
 just too new and untested for me to troubleshoot and get through.  
 Hopefully, someone on here can point me in the right direction.
 Mike


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Fri, 24 Dec 2010 13:38:36 +0800, "Zhuo Song" 
 <alfred.z.song@gmail.com> wrote:
> Can you attach some logs by dmesg in your guest just after *igbvf*
> module inserted?
> Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you
> ever checked the physical ports in your card?
> If you the card takes two ports, generally, one is for odd and the
> other is for even function of the BDF.
>
> Best Regards
>
> Zhuo Song
> Gif me a visky, ginger ale on the side.
>

 Dec 26 19:42:35 localhost kernel: [  450.895517] pci 0000:00:02.0: BAR 
 6: [??? 0x00000000 flags 0x2] has bogus alignment
 Dec 26 19:42:35 localhost kernel: [  450.895528] pci 0000:00:07.0: BAR 
 0: assigned [mem 0x40000000-0x40003fff 64bit]
 Dec 26 19:42:35 localhost kernel: [  450.895986] pci 0000:00:07.0: BAR 
 0: set to [mem 0x40000000-0x40003fff 64bit] (PCI address 
 [0x40000000-0x40003fff]
 Dec 26 19:42:35 localhost kernel: [  450.895999] pci 0000:00:07.0: BAR 
 3: assigned [mem 0x40004000-0x40007fff 64bit]
 Dec 26 19:42:35 localhost kernel: [  450.896513] pci 0000:00:07.0: BAR 
 3: set to [mem 0x40004000-0x40007fff 64bit] (PCI address 
 [0x40004000-0x40007fff]
 Dec 26 19:42:35 localhost kernel: [  450.902103] pci 0000:00:00.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902108] pci 0000:00:00.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.902446] pci 0000:00:01.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902452] pci 0000:00:01.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.902869] ata_piix 0000:00:01.1: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902875] ata_piix 0000:00:01.1: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.903527] uhci_hcd 0000:00:01.2: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.903532] uhci_hcd 0000:00:01.2: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.903942] pci 0000:00:01.3: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.903948] pci 0000:00:01.3: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.904549] pci 0000:00:02.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.904554] pci 0000:00:02.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.904930] pci 0000:00:03.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.904933] pci 0000:00:03.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.905213] 8139cp 0000:00:04.0: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.905215] 8139cp 0000:00:04.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.905366] pci 0000:00:07.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  451.038332] Intel(R) Virtual 
 Function Network Driver - version 1.0.0-k0
 Dec 26 19:42:35 localhost kernel: [  451.038334] Copyright (c) 2009 
 Intel Corporation.
 Dec 26 19:42:35 localhost kernel: [  451.038413] igbvf 0000:00:07.0: 
 enabling device (0000 -> 0002)
 Dec 26 19:42:35 localhost kernel: [  451.070856] igbvf 0000:00:07.0: 
 Intel(R) 82576 Virtual Function
 Dec 26 19:42:35 localhost kernel: [  451.070865] igbvf 0000:00:07.0: 
 Address: 82:bb:48:c2:5f:ca
 Dec 26 19:42:35 localhost kernel: [  451.070871] igbvf 0000:00:07.0: 
 MAC: 1
 Dec 26 19:42:42 localhost kernel: [  457.800481] pci 0000:00:02.0: BAR 
 6: [??? 0x00000000 flags 0x2] has bogus alignment
 Dec 26 19:42:42 localhost kernel: [  457.800492] pci 0000:00:06.0: BAR 
 0: assigned [mem 0x40008000-0x4000bfff 64bit]
 Dec 26 19:42:42 localhost kernel: [  457.800922] pci 0000:00:06.0: BAR 
 0: set to [mem 0x40008000-0x4000bfff 64bit] (PCI address 
 [0x40008000-0x4000bfff]
 Dec 26 19:42:42 localhost kernel: [  457.800929] pci 0000:00:06.0: BAR 
 3: assigned [mem 0x4000c000-0x4000ffff 64bit]
 Dec 26 19:42:42 localhost kernel: [  457.801416] pci 0000:00:06.0: BAR 
 3: set to [mem 0x4000c000-0x4000ffff 64bit] (PCI address 
 [0x4000c000-0x4000ffff]
 Dec 26 19:42:42 localhost kernel: [  457.801744] pci 0000:00:00.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.801748] pci 0000:00:00.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804131] pci 0000:00:01.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804136] pci 0000:00:01.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804405] ata_piix 0000:00:01.1: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804409] ata_piix 0000:00:01.1: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804662] uhci_hcd 0000:00:01.2: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804665] uhci_hcd 0000:00:01.2: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804917] pci 0000:00:01.3: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804920] pci 0000:00:01.3: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809453] pci 0000:00:02.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809455] pci 0000:00:02.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809646] pci 0000:00:03.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809648] pci 0000:00:03.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809758] 8139cp 0000:00:04.0: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809759] 8139cp 0000:00:04.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809888] igbvf 0000:00:07.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809892] pci 0000:00:06.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.864627] igbvf 0000:00:06.0: 
 enabling device (0000 -> 0002)
 Dec 26 19:42:42 localhost kernel: [  457.898412] igbvf 0000:00:06.0: 
 Intel(R) 82576 Virtual Function
 Dec 26 19:42:42 localhost kernel: [  457.898416] igbvf 0000:00:06.0: 
 Address: ca:c0:62:de:59:c7
 Dec 26 19:42:42 localhost kernel: [  457.898419] igbvf 0000:00:06.0: 
 MAC: 1

 Mike

RE: Intel S3420GPLX VT-d VF question

[Zhuo Song]

Not so sure. But I think you first need to check into your guest OS for that the ethernet port which bound to your VF is UP.( after it's been UP, you can see dmesg for some info, like some words "eth0/1 is ready")
The related PF in the dom0 is UP too. It's mentioned that, like the previous talk, If the card takes two ports, generally, one is for odd and the other is for even function of the VF's BDF(lspci in dom0) which you passed thru. So one way is to try both ports.

Best Regards

Alfred Song
Gif me a visky, ginger ale on the side.

 On Fri, 24 Dec 2010 13:38:36 +0800, "Zhuo Song" 
 <alfred.z.song@gmail.com> wrote:
> Can you attach some logs by dmesg in your guest just after *igbvf*
> module inserted?
> Which vf(s) or BDF did you passed to guest? 0000:04:10.4? So have you
> ever checked the physical ports in your card?
> If you the card takes two ports, generally, one is for odd and the
> other is for even function of the BDF.
>
> Best Regards
>
> Zhuo Song
> Gif me a visky, ginger ale on the side.
>

 Dec 26 19:42:35 localhost kernel: [  450.895517] pci 0000:00:02.0: BAR 
 6: [??? 0x00000000 flags 0x2] has bogus alignment
 Dec 26 19:42:35 localhost kernel: [  450.895528] pci 0000:00:07.0: BAR 
 0: assigned [mem 0x40000000-0x40003fff 64bit]
 Dec 26 19:42:35 localhost kernel: [  450.895986] pci 0000:00:07.0: BAR 
 0: set to [mem 0x40000000-0x40003fff 64bit] (PCI address 
 [0x40000000-0x40003fff]
 Dec 26 19:42:35 localhost kernel: [  450.895999] pci 0000:00:07.0: BAR 
 3: assigned [mem 0x40004000-0x40007fff 64bit]
 Dec 26 19:42:35 localhost kernel: [  450.896513] pci 0000:00:07.0: BAR 
 3: set to [mem 0x40004000-0x40007fff 64bit] (PCI address 
 [0x40004000-0x40007fff]
 Dec 26 19:42:35 localhost kernel: [  450.902103] pci 0000:00:00.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902108] pci 0000:00:00.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.902446] pci 0000:00:01.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902452] pci 0000:00:01.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.902869] ata_piix 0000:00:01.1: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.902875] ata_piix 0000:00:01.1: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.903527] uhci_hcd 0000:00:01.2: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.903532] uhci_hcd 0000:00:01.2: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.903942] pci 0000:00:01.3: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.903948] pci 0000:00:01.3: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.904549] pci 0000:00:02.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.904554] pci 0000:00:02.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.904930] pci 0000:00:03.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.904933] pci 0000:00:03.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.905213] 8139cp 0000:00:04.0: 
 no hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  450.905215] 8139cp 0000:00:04.0: 
 using default PCI settings
 Dec 26 19:42:35 localhost kernel: [  450.905366] pci 0000:00:07.0: no 
 hotplug settings from platform
 Dec 26 19:42:35 localhost kernel: [  451.038332] Intel(R) Virtual 
 Function Network Driver - version 1.0.0-k0
 Dec 26 19:42:35 localhost kernel: [  451.038334] Copyright (c) 2009 
 Intel Corporation.
 Dec 26 19:42:35 localhost kernel: [  451.038413] igbvf 0000:00:07.0: 
 enabling device (0000 -> 0002)
 Dec 26 19:42:35 localhost kernel: [  451.070856] igbvf 0000:00:07.0: 
 Intel(R) 82576 Virtual Function
 Dec 26 19:42:35 localhost kernel: [  451.070865] igbvf 0000:00:07.0: 
 Address: 82:bb:48:c2:5f:ca
 Dec 26 19:42:35 localhost kernel: [  451.070871] igbvf 0000:00:07.0: 
 MAC: 1
 Dec 26 19:42:42 localhost kernel: [  457.800481] pci 0000:00:02.0: BAR 
 6: [??? 0x00000000 flags 0x2] has bogus alignment
 Dec 26 19:42:42 localhost kernel: [  457.800492] pci 0000:00:06.0: BAR 
 0: assigned [mem 0x40008000-0x4000bfff 64bit]
 Dec 26 19:42:42 localhost kernel: [  457.800922] pci 0000:00:06.0: BAR 
 0: set to [mem 0x40008000-0x4000bfff 64bit] (PCI address 
 [0x40008000-0x4000bfff]
 Dec 26 19:42:42 localhost kernel: [  457.800929] pci 0000:00:06.0: BAR 
 3: assigned [mem 0x4000c000-0x4000ffff 64bit]
 Dec 26 19:42:42 localhost kernel: [  457.801416] pci 0000:00:06.0: BAR 
 3: set to [mem 0x4000c000-0x4000ffff 64bit] (PCI address 
 [0x4000c000-0x4000ffff]
 Dec 26 19:42:42 localhost kernel: [  457.801744] pci 0000:00:00.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.801748] pci 0000:00:00.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804131] pci 0000:00:01.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804136] pci 0000:00:01.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804405] ata_piix 0000:00:01.1: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804409] ata_piix 0000:00:01.1: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804662] uhci_hcd 0000:00:01.2: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804665] uhci_hcd 0000:00:01.2: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.804917] pci 0000:00:01.3: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.804920] pci 0000:00:01.3: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809453] pci 0000:00:02.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809455] pci 0000:00:02.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809646] pci 0000:00:03.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809648] pci 0000:00:03.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809758] 8139cp 0000:00:04.0: 
 no hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809759] 8139cp 0000:00:04.0: 
 using default PCI settings
 Dec 26 19:42:42 localhost kernel: [  457.809888] igbvf 0000:00:07.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.809892] pci 0000:00:06.0: no 
 hotplug settings from platform
 Dec 26 19:42:42 localhost kernel: [  457.864627] igbvf 0000:00:06.0: 
 enabling device (0000 -> 0002)
 Dec 26 19:42:42 localhost kernel: [  457.898412] igbvf 0000:00:06.0: 
 Intel(R) 82576 Virtual Function
 Dec 26 19:42:42 localhost kernel: [  457.898416] igbvf 0000:00:06.0: 
 Address: ca:c0:62:de:59:c7
 Dec 26 19:42:42 localhost kernel: [  457.898419] igbvf 0000:00:06.0: 
 MAC: 1

 Mike

Re: Intel S3420GPLX VT-d VF question

[Michael A. Collins]

 On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk 
 <konrad.wilk@oracle.com> wrote:
>>
>> Thanks for the reply, I will try without the e820 option and loading
>> the igbvf module tonight if that works, I'll let you know.  If not,
>> I'll respond with the full xen dmesg output.
>
> Take your time.

 Ok, got a test vm running and the igbvf module loaded, so it looks like 
 everything is working.  I can't seem to get any traffic sent or received 
 from it, but I'll work through that eventually.  Thanks for all your 
 help!
 Mike