Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Bjorn Helgaas]

[+ linux-pci, Yinghai]

On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao <pkill.2012@gmail.com> wrote:
>> The BIOS in your machine doesn't support SR-IOV.  You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available.  Sometimes they're not.
>
> very thanks Greg,my server Dell R710 with latest BIOS version and
> option for SR-IOV(SR-IOV Global Enable->Enabled)  opened,I'm confused
> that Does R710 provide full support for SR-IOV, kernel or  ixgbe
> driver's bug? but I'm not sure where the problem lies,anyone has any
> experience about this?   .

Linux normally uses the resource assignments done by the BIOS, but it
is possible for the kernel to reassign those.  We don't have good
automatic support for that yet, but on a recent upstream kernel, you
can try "pci=realloc".  I doubt this option is in CentOS 6.3, though.

If an upstream kernel with "pci=realloc" still doesn't work, please
post the entire dmesg log.

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Yinghai Lu]

On Fri, Nov 9, 2012 at 7:26 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> [+ linux-pci, Yinghai]
>
> On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao <pkill.2012@gmail.com> wrote:
>>> The BIOS in your machine doesn't support SR-IOV.  You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available.  Sometimes they're not.
>>
>> very thanks Greg,my server Dell R710 with latest BIOS version and
>> option for SR-IOV(SR-IOV Global Enable->Enabled)  opened,I'm confused
>> that Does R710 provide full support for SR-IOV, kernel or  ixgbe
>> driver's bug? but I'm not sure where the problem lies,anyone has any
>> experience about this?   .
>
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those.  We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc".  I doubt this option is in CentOS 6.3, though.
>
> If an upstream kernel with "pci=realloc" still doesn't work, please
> post the entire dmesg log.

Even better, upstream kernel will enable realloc automatically if it
finds some SRIOV BAR is not assigned.
related logic is in pci_realloc_detect().

BTW, we may need another patch about realloc.

http://git.kernel.org/?p=linux/kernel/git/yinghai/linux-yinghai.git;a=patch;h=afb9975bddd92b17c2f9859676ce521e2e014997

>From afb9975bddd92b17c2f9859676ce521e2e014997 Mon Sep 17 00:00:00 2001
From: Yinghai Lu <yinghai@kernel.org>
Date: Wed, 3 Oct 2012 19:15:57 -0700
Subject: [PATCH] PCI: Add root bus children dev's res to fail list

We can stop trying according to try number now and do not need to use
root_bus checking as stop sign anymore.
In extreme case we could need to reallocate resource for device just
under root bus.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/setup-bus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 1e808ca..9e625d84 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -283,7 +283,7 @@ static void
assign_requested_resources_sorted(struct list_head *head,
 		idx = res - &dev_res->dev->resource[0];
 		if (resource_size(res) &&
 		    pci_assign_resource(dev_res->dev, idx)) {
-			if (fail_head && !pci_is_root_bus(dev_res->dev->bus)) {
+			if (fail_head) {
 				/*
 				 * if the failed res is for ROM BAR, and it will
 				 * be enabled later, don't add it to the list
-- 
1.7.7.6

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Don Dutile]

On 11/09/2012 10:26 AM, Bjorn Helgaas wrote:
> [+ linux-pci, Yinghai]
>
> On Thu, Nov 8, 2012 at 8:59 PM, Jason Gao<pkill.2012@gmail.com>  wrote:
>>> The BIOS in your machine doesn't support SR-IOV.  You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available.  Sometimes they're not.
>>
>> very thanks Greg,my server Dell R710 with latest BIOS version and
>> option for SR-IOV(SR-IOV Global Enable->Enabled)  opened,I'm confused
>> that Does R710 provide full support for SR-IOV, kernel or  ixgbe
>> driver's bug? but I'm not sure where the problem lies,anyone has any
>> experience about this?   .
>
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those.  We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc".  I doubt this option is in CentOS 6.3, though.
>
Try moving the device into a different slot.
You may be trying it in a non-ARI slot in the 710; that is a problem
in RHEL6 (needing to realloc bus &/or mem-space).
A non-ARI slot will want to use one bus number per VF.... which will
be problematic.
I know I've seen ixgbe's (& their vfs's) working on a dell 710;
but they may also be one of those systems that has a slot off the ich10
with no ARI support.

> If an upstream kernel with "pci=realloc" still doesn't work, please
> post the entire dmesg log.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Jason Gao]

I'm very sorry for delayed reply.now SR-IOV works for me in Centos
6.3,thank all of you.


On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> Linux normally uses the resource assignments done by the BIOS, but it
> is possible for the kernel to reassign those.  We don't have good
> automatic support for that yet, but on a recent upstream kernel, you
> can try "pci=realloc".  I doubt this option is in CentOS 6.3, though

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.



On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai <sibai.li@intel.com> wrote:
> DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012
> Please check if you configured intel_iommu=on in the grub.conf file.
> If you did, check your kernel .config file under Device Drivers-> IOMMU Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.

thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured
intel_iommu=on in the grub.conf file,but I can't find these IOMMU
options in "Device Drivers" in my kernel(2.6.32-279) .config file ,
btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like
to know what's your os version ,kernel version?

RE: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Li, Sibai]

> -----Original Message-----
> From: Jason Gao [mailto:pkill.2012@gmail.com]
> Sent: Tuesday, November 13, 2012 5:38 AM
> To: bhelgaas@google.com; Rose, Gregory V; Li, Sibai
> Cc: ddutile@redhat.com; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
> devel@lists.sourceforge.net; linux-pci@vger.kernel.org; Yinghai Lu
> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
> 
> I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all
> of you.
> 
> 
> On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> > Linux normally uses the resource assignments done by the BIOS, but it
> > is possible for the kernel to reassign those.  We don't have good
> > automatic support for that yet, but on a recent upstream kernel, you
> > can try "pci=realloc".  I doubt this option is in CentOS 6.3, though
> 
> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
> 
> 
> 
> On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai <sibai.li@intel.com> wrote:
> > DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
> > v.6.3.0 and release date is 07/24/2012 Please check if you configured
> intel_iommu=on in the grub.conf file.
> > If you did, check your kernel .config file under Device Drivers-> IOMMU
> Hardware support->enable Support for Intel IOMMU using DMA remapping
> Devices, enable Intel DMA Remapping Devices by Default, enable Support for
> Interrupt Remapping.
> 
> thank you Sibai,Our server "Dell R710",its BIOS version is just
> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
> problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Don Dutile]

On 11/13/2012 11:04 AM, Li, Sibai wrote:
>
>
>> -----Original Message-----
>> From: Jason Gao [mailto:pkill.2012@gmail.com]
>> Sent: Tuesday, November 13, 2012 5:38 AM
>> To: bhelgaas@google.com; Rose, Gregory V; Li, Sibai
>> Cc: ddutile@redhat.com; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-
>> devel@lists.sourceforge.net; linux-pci@vger.kernel.org; Yinghai Lu
>> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
>> for SR-IOV)
>>
>> I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all
>> of you.
>>
>>
>> On Fri, Nov 9, 2012 at 11:26 PM, Bjorn Helgaas<bhelgaas@google.com>  wrote:
>>> Linux normally uses the resource assignments done by the BIOS, but it
>>> is possible for the kernel to reassign those.  We don't have good
>>> automatic support for that yet, but on a recent upstream kernel, you
>>> can try "pci=realloc".  I doubt this option is in CentOS 6.3, though
>>
>> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
>>
>>
>>
>> On Sat, Nov 10, 2012 at 2:08 AM, Li, Sibai<sibai.li@intel.com>  wrote:
>>> DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
>>> v.6.3.0 and release date is 07/24/2012 Please check if you configured
>> intel_iommu=on in the grub.conf file.
>>> If you did, check your kernel .config file under Device Drivers->  IOMMU
>> Hardware support->enable Support for Intel IOMMU using DMA remapping
>> Devices, enable Intel DMA Remapping Devices by Default, enable Support for
>> Interrupt Remapping.
>>
>> thank you Sibai,Our server "Dell R710",its BIOS version is just
>> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
>> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
Sibai is referring to kernel config options.  RHEL6.3 has the IOMMU options built into
the kernel, but not enabled by default -- have to add 'intel_iommu=on' to the kernel
cmdline to enable IOMMU. SRIOV support (CONFIG_IOV) is built into the RHEL6.3 kernel as well.

>> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
>> problem solved,I'd like to know what's your os version ,kernel version?
>
> I am using RHEL6.3 with unstable kernel 3.7.0-rc
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Yinghai Lu]

On Tue, Nov 13, 2012 at 8:04 AM, Li, Sibai <sibai.li@intel.com> wrote:
>
>>
>> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
>>
>> thank you Sibai,Our server "Dell R710",its BIOS version is just
>> v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
>> the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
>> kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
>> problem solved,I'd like to know what's your os version ,kernel version?
>
> I am using RHEL6.3 with unstable kernel 3.7.0-rc

that means that config has
CONFIG_PCI_REALLOC_ENABLE_AUTO=y

So you don't need to append "pci=realloc"

Yinghai

RE: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Li, Sibai]

> -----Original Message-----
> From: yhlu.kernel@gmail.com [mailto:yhlu.kernel@gmail.com] On Behalf Of
> Yinghai Lu
> Sent: Tuesday, November 13, 2012 10:17 AM
> To: Li, Sibai
> Cc: Jason Gao; bhelgaas@google.com; Rose, Gregory V; ddutile@redhat.com;
> Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000-devel@lists.sourceforge.net;
> linux-pci@vger.kernel.org
> Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
> for SR-IOV)
> 
> On Tue, Nov 13, 2012 at 8:04 AM, Li, Sibai <sibai.li@intel.com> wrote:
> >
> >>
> >> Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for
> me.
> >>
> >> thank you Sibai,Our server "Dell R710",its BIOS version is just
> >> v.6.3.0 and release date is 07/24/2012,and I also configured
> >> intel_iommu=on in the grub.conf file,but I can't find these IOMMU
> >> options in "Device Drivers" in my
> >> kernel(2.6.32-279) .config file , btw my os is Centos
> >> 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os
> version ,kernel version?
> >
> > I am using RHEL6.3 with unstable kernel 3.7.0-rc
> 
> that means that config has
> CONFIG_PCI_REALLOC_ENABLE_AUTO=y
> 
> So you don't need to append "pci=realloc"
> 
> Yinghai

Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.

Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

[Yinghai Lu]

On Tue, Nov 13, 2012 at 10:25 AM, Li, Sibai <sibai.li@intel.com> wrote:
>
> Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.

well,  can you both post boot log with "debug ignore_loglevel" ?