Error assigning BAR 0 to hotplugged device

Error assigning BAR 0 to hotplugged device

[Chuck Tuffli]

On a system with a recent Fedora kernel (3.14.9-200.fc20.x86_64), a
hotplugged PCI device is unable to get memory mapped IO:
...
[  183.247377] pcieport 0000:9a:00.0: BAR 15: can't assign mem pref
(size 0x100000)
[  183.247384] pcieport 0000:9b:15.0: BAR 14: can't assign mem (size 0x100000)
[  183.247388] pcieport 0000:9b:15.0: BAR 15: can't assign mem pref
(size 0x100000)
[  183.247394] pci 0000:a5:00.0: BAR 6: can't assign mem pref (size 0x10000)
[  183.247397] pci 0000:a5:00.0: BAR 0: can't assign mem (size 0x4000)

The device is connected to a switch supporting hotplug (i.e. SltCap
includes HotPlug+) and I have added
pci=realloc,hpmemsize=16M
to the kernel boot parameters. This appears to have increased the
prefetchable memory but not the memory mapped IO:

99:00.0 PCI bridge: PLX Technology, Inc. Device 8733 (rev ca) (prog-if
00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Bus: primary=99, secondary=9a, subordinate=a5, sec-latency=0
    I/O behind bridge: 0000d000-0000ffff
    Memory behind bridge: fb100000-fbafffff
    Prefetchable memory behind bridge: 0000381000000000-0000381000ffffff

Should this work or am I missing a step somewhere? TIA.

Full dmesg http://pastebin.com/pZ1zzHS6
Full lspci -vv http://pastebin.com/n57UheAE

--chuck

Re: Error assigning BAR 0 to hotplugged device

[Chuck Tuffli]

On Tue, Jul 8, 2014 at 12:37 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
> On a system with a recent Fedora kernel (3.14.9-200.fc20.x86_64), a
> hotplugged PCI device is unable to get memory mapped IO:
> ...
> [  183.247377] pcieport 0000:9a:00.0: BAR 15: can't assign mem pref
> (size 0x100000)
> [  183.247384] pcieport 0000:9b:15.0: BAR 14: can't assign mem (size 0x100000)
> [  183.247388] pcieport 0000:9b:15.0: BAR 15: can't assign mem pref
> (size 0x100000)
> [  183.247394] pci 0000:a5:00.0: BAR 6: can't assign mem pref (size 0x10000)
> [  183.247397] pci 0000:a5:00.0: BAR 0: can't assign mem (size 0x4000)
>
> The device is connected to a switch supporting hotplug (i.e. SltCap
> includes HotPlug+) and I have added
> pci=realloc,hpmemsize=16M
> to the kernel boot parameters. This appears to have increased the
> prefetchable memory but not the memory mapped IO:

Further investigation suggests that pbus_size_mem() never really gets
a chance to increase the bridge aperture by the hpmemsize value
because find_free_bus_resource() returns NULL. This apparently happens
because the function "skip[s] the bus resources which have already
been assigned (that is, have non-NULL parent resource)". Specifically,
it checks

    if (r && (r->flags & type_mask) == type && !r->parent)

and in my configuration, finds that r->parent != NULL for the memory
mapped IO resource. Wouldn't r->parent be non-NULL in most cases other
than for bridges in the root complex? If so, does that imply hpmemsize
would only be used for bridges advertising hot plug in the root
complex?

--chuck

Re: Error assigning BAR 0 to hotplugged device

[Bjorn Helgaas]

[+cc Yinghai]

On Wed, Jul 16, 2014 at 11:19 AM, Chuck Tuffli <ctuffli@gmail.com> wrote:
> On Tue, Jul 8, 2014 at 12:37 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>> On a system with a recent Fedora kernel (3.14.9-200.fc20.x86_64), a
>> hotplugged PCI device is unable to get memory mapped IO:
>> ...
>> [  183.247377] pcieport 0000:9a:00.0: BAR 15: can't assign mem pref
>> (size 0x100000)
>> [  183.247384] pcieport 0000:9b:15.0: BAR 14: can't assign mem (size 0x100000)
>> [  183.247388] pcieport 0000:9b:15.0: BAR 15: can't assign mem pref
>> (size 0x100000)
>> [  183.247394] pci 0000:a5:00.0: BAR 6: can't assign mem pref (size 0x10000)
>> [  183.247397] pci 0000:a5:00.0: BAR 0: can't assign mem (size 0x4000)
>>
>> The device is connected to a switch supporting hotplug (i.e. SltCap
>> includes HotPlug+) and I have added
>> pci=realloc,hpmemsize=16M
>> to the kernel boot parameters. This appears to have increased the
>> prefetchable memory but not the memory mapped IO:
>
> Further investigation suggests that pbus_size_mem() never really gets
> a chance to increase the bridge aperture by the hpmemsize value
> because find_free_bus_resource() returns NULL. This apparently happens
> because the function "skip[s] the bus resources which have already
> been assigned (that is, have non-NULL parent resource)". Specifically,
> it checks
>
>     if (r && (r->flags & type_mask) == type && !r->parent)
>
> and in my configuration, finds that r->parent != NULL for the memory
> mapped IO resource. Wouldn't r->parent be non-NULL in most cases other
> than for bridges in the root complex? If so, does that imply hpmemsize
> would only be used for bridges advertising hot plug in the root
> complex?

Thanks for chasing this down.  I'm going to be on vacation for a few
weeks and won't be able to work on this until I return.  I added
Yinghai, who will probably have a suggestion for you.

Bjorn

Re: Error assigning BAR 0 to hotplugged device

[Yinghai Lu]

On Wed, Jul 16, 2014 at 11:54 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> [+cc Yinghai]
>
> On Wed, Jul 16, 2014 at 11:19 AM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>> On Tue, Jul 8, 2014 at 12:37 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>>> On a system with a recent Fedora kernel (3.14.9-200.fc20.x86_64), a
>>> hotplugged PCI device is unable to get memory mapped IO:
>>> ...
>>> [  183.247377] pcieport 0000:9a:00.0: BAR 15: can't assign mem pref
>>> (size 0x100000)
>>> [  183.247384] pcieport 0000:9b:15.0: BAR 14: can't assign mem (size 0x100000)
>>> [  183.247388] pcieport 0000:9b:15.0: BAR 15: can't assign mem pref
>>> (size 0x100000)
>>> [  183.247394] pci 0000:a5:00.0: BAR 6: can't assign mem pref (size 0x10000)
>>> [  183.247397] pci 0000:a5:00.0: BAR 0: can't assign mem (size 0x4000)
>>>
>>> The device is connected to a switch supporting hotplug (i.e. SltCap
>>> includes HotPlug+) and I have added
>>> pci=realloc,hpmemsize=16M
>>> to the kernel boot parameters. This appears to have increased the
>>> prefetchable memory but not the memory mapped IO:
>>
>> Further investigation suggests that pbus_size_mem() never really gets
>> a chance to increase the bridge aperture by the hpmemsize value
>> because find_free_bus_resource() returns NULL. This apparently happens
>> because the function "skip[s] the bus resources which have already
>> been assigned (that is, have non-NULL parent resource)". Specifically,
>> it checks
>>
>>     if (r && (r->flags & type_mask) == type && !r->parent)
>>
>> and in my configuration, finds that r->parent != NULL for the memory
>> mapped IO resource. Wouldn't r->parent be non-NULL in most cases other
>> than for bridges in the root complex? If so, does that imply hpmemsize
>> would only be used for bridges advertising hot plug in the root
>> complex?

Please try current tree linus/master or pci/next.
also need boot with "debug ignore_loglevel pciehp.pciehp_debug=1".

post
dmesg -s 262144
lspci -tv
lspci -vvxxx

Thanks

Yinghai

Re: Error assigning BAR 0 to hotplugged device

[Yinghai Lu]

On Wed, Jul 16, 2014 at 1:55 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Wed, Jul 16, 2014 at 11:54 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>> [+cc Yinghai]
>>
>> On Wed, Jul 16, 2014 at 11:19 AM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>>> On Tue, Jul 8, 2014 at 12:37 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>>>> On a system with a recent Fedora kernel (3.14.9-200.fc20.x86_64), a
>>>> hotplugged PCI device is unable to get memory mapped IO:
>>>> ...
>>>> [  183.247377] pcieport 0000:9a:00.0: BAR 15: can't assign mem pref
>>>> (size 0x100000)
>>>> [  183.247384] pcieport 0000:9b:15.0: BAR 14: can't assign mem (size 0x100000)
>>>> [  183.247388] pcieport 0000:9b:15.0: BAR 15: can't assign mem pref
>>>> (size 0x100000)
>>>> [  183.247394] pci 0000:a5:00.0: BAR 6: can't assign mem pref (size 0x10000)
>>>> [  183.247397] pci 0000:a5:00.0: BAR 0: can't assign mem (size 0x4000)
>>>>
>>>> The device is connected to a switch supporting hotplug (i.e. SltCap
>>>> includes HotPlug+) and I have added
>>>> pci=realloc,hpmemsize=16M
>>>> to the kernel boot parameters. This appears to have increased the
>>>> prefetchable memory but not the memory mapped IO:
>>>
>>> Further investigation suggests that pbus_size_mem() never really gets
>>> a chance to increase the bridge aperture by the hpmemsize value
>>> because find_free_bus_resource() returns NULL. This apparently happens
>>> because the function "skip[s] the bus resources which have already
>>> been assigned (that is, have non-NULL parent resource)". Specifically,
>>> it checks
>>>
>>>     if (r && (r->flags & type_mask) == type && !r->parent)
>>>
>>> and in my configuration, finds that r->parent != NULL for the memory
>>> mapped IO resource. Wouldn't r->parent be non-NULL in most cases other
>>> than for bridges in the root complex? If so, does that imply hpmemsize
>>> would only be used for bridges advertising hot plug in the root
>>> complex?
>
> Please try current tree linus/master or pci/next.
> also need boot with "debug ignore_loglevel pciehp.pciehp_debug=1".
>
> post
> dmesg -s 262144
> lspci -tv
> lspci -vvxxx

in the tree:
   80:03.0 ==> 96:00.0 ==> 97:08.0 ==> 98:00.0 ==> 99:00.0 ==> 9a:00.0
==> 9b:15.0

only 97:08.0 and 99:00.0 is hotplug+.

and kernel will honor BIOS set value at first, and realloc will only work on
unassigned/invalid assigned BARs. and hpmem_size will be only treated at
optional size even on hotplug slots.

a5:00.0 is not in 9b:15.0 at first, so yo just put the card in can
rescan the card,
right?

In this case we don't to realloc, as other devices could already have
driver loaded.

You need to fix the BIOS to have correct setting for 96:1b.0.

                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
                        Slot #21, PowerLimit 25.000W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet-
CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
                        Changed: MRL- PresDet+ LinkState+

to make it really hotplug slot.

And BIOS need to make sure parent bus/bridge have enough resource ranges
for the hotplug slots.

Yinghai

Re: Error assigning BAR 0 to hotplugged device

[Yinghai Lu]

On Wed, Jul 16, 2014 at 4:47 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Wed, Jul 16, 2014 at 1:55 PM, Yinghai Lu <yinghai@kernel.org> wrote:

>
> in the tree:
>    80:03.0 ==> 96:00.0 ==> 97:08.0 ==> 98:00.0 ==> 99:00.0 ==> 9a:00.0
> ==> 9b:15.0
>
> only 97:08.0 and 99:00.0 is hotplug+.
>
> and kernel will honor BIOS set value at first, and realloc will only work on
> unassigned/invalid assigned BARs. and hpmem_size will be only treated at
> optional size even on hotplug slots.
>
> a5:00.0 is not in 9b:15.0 at first, so yo just put the card in can
> rescan the card,
> right?
>
> In this case we don't to realloc, as other devices could already have
> driver loaded.
>
> You need to fix the BIOS to have correct setting for 96:1b.0.

should 9b:15.0

>
>                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
> HotPlug- Surprise-
>                         Slot #21, PowerLimit 25.000W; Interlock- NoCompl-
>                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet-
> CmdCplt- HPIrq- LinkChg-
>                         Control: AttnInd Unknown, PwrInd Unknown,
> Power- Interlock-
>                 SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
> PresDet+ Interlock-
>                         Changed: MRL- PresDet+ LinkState+
>
> to make it really hotplug slot.
>
> And BIOS need to make sure parent bus/bridge have enough resource ranges
> for the hotplug slots.
>
> Yinghai

Re: Error assigning BAR 0 to hotplugged device

[Chuck Tuffli]

On Wed, Jul 16, 2014 at 4:47 PM, Yinghai Lu <yinghai@kernel.org> wrote:
...
> in the tree:
>    80:03.0 ==> 96:00.0 ==> 97:08.0 ==> 98:00.0 ==> 99:00.0 ==> 9a:00.0
> ==> 9b:15.0
>
> only 97:08.0 and 99:00.0 is hotplug+.
>
> and kernel will honor BIOS set value at first, and realloc will only work on
> unassigned/invalid assigned BARs. and hpmem_size will be only treated at
> optional size even on hotplug slots.

Does this mean the only time the kernel will re-allocate the BARs and
use hpmemsize is if they have the value 0x0? If so, does realloc have
any limitations with respect to where in the tree the invalid BAR
exists? In my example, could realloc fix 80:03.0? 9b:15.0?

> a5:00.0 is not in 9b:15.0 at first, so yo just put the card in can
> rescan the card,
> right?

Correct. I did an echo 1 > /sys/bus/pci/rescan

> In this case we don't to realloc, as other devices could already have
> driver loaded.

I can understand why this is true after the PCI subsystem is up and
running, but at boot time, before any other drivers have loaded,
wouldn't it be possible for the PCI driver to honor realloc and
hpmemsize and override the values written by the BIOS? This way,
devices added later would have memory available for their BARs.

> You need to fix the BIOS to have correct setting for 96:1b.0.
>
>                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
> HotPlug- Surprise-
>                         Slot #21, PowerLimit 25.000W; Interlock- NoCompl-
>                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet-
> CmdCplt- HPIrq- LinkChg-
>                         Control: AttnInd Unknown, PwrInd Unknown,
> Power- Interlock-
>                 SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
> PresDet+ Interlock-
>                         Changed: MRL- PresDet+ LinkState+
>
> to make it really hotplug slot.

Agreed. The slot actually is hot plug capable, and I was considering
adding a quirk and then using hpmemsize to provide enough resources
for devices. But given your description, it sounds like this won't
work.

> And BIOS need to make sure parent bus/bridge have enough resource ranges
> for the hotplug slots.
>
> Yinghai

If the BIOS doesn't allocate enough memory mapped IO resources for
devices, it sounds like the kernel can't really fix this problem,
right?

--chuck

Re: Error assigning BAR 0 to hotplugged device

[Yinghai Lu]

On Wed, Jul 16, 2014 at 6:00 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
> On Wed, Jul 16, 2014 at 4:47 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> ...
>> in the tree:
>>    80:03.0 ==> 96:00.0 ==> 97:08.0 ==> 98:00.0 ==> 99:00.0 ==> 9a:00.0
>> ==> 9b:15.0
>>
>> only 97:08.0 and 99:00.0 is hotplug+.
>>
>> and kernel will honor BIOS set value at first, and realloc will only work on
>> unassigned/invalid assigned BARs. and hpmem_size will be only treated at
>> optional size even on hotplug slots.
>
> Does this mean the only time the kernel will re-allocate the BARs and
> use hpmemsize is if they have the value 0x0? If so, does realloc have
> any limitations with respect to where in the tree the invalid BAR
> exists? In my example, could realloc fix 80:03.0? 9b:15.0?

the problem is 9b:15.0 is not hotplugable, when a5:00.0 is not
inserted into the slot,
realloc will not pre-reserve range for 9b:15.0.

> If the BIOS doesn't allocate enough memory mapped IO resources for
> devices, it sounds like the kernel can't really fix this problem,
> right?

Yes. You need to fix BIOS at first.

So it will be final products or just development prototype?

Yinghai

Re: Error assigning BAR 0 to hotplugged device

[Chuck Tuffli]

On Wed, Jul 16, 2014 at 7:15 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Wed, Jul 16, 2014 at 6:00 PM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>> On Wed, Jul 16, 2014 at 4:47 PM, Yinghai Lu <yinghai@kernel.org> wrote:
>> ...
>>> in the tree:
>>>    80:03.0 ==> 96:00.0 ==> 97:08.0 ==> 98:00.0 ==> 99:00.0 ==> 9a:00.0
>>> ==> 9b:15.0
>>>
>>> only 97:08.0 and 99:00.0 is hotplug+.
>>>
>>> and kernel will honor BIOS set value at first, and realloc will only work on
>>> unassigned/invalid assigned BARs. and hpmem_size will be only treated at
>>> optional size even on hotplug slots.
>>
>> Does this mean the only time the kernel will re-allocate the BARs and
>> use hpmemsize is if they have the value 0x0? If so, does realloc have
>> any limitations with respect to where in the tree the invalid BAR
>> exists? In my example, could realloc fix 80:03.0? 9b:15.0?
>
> the problem is 9b:15.0 is not hotplugable, when a5:00.0 is not
> inserted into the slot,
> realloc will not pre-reserve range for 9b:15.0.
>
>> If the BIOS doesn't allocate enough memory mapped IO resources for
>> devices, it sounds like the kernel can't really fix this problem,
>> right?
>
> Yes. You need to fix BIOS at first.
>
> So it will be final products or just development prototype?

This is a prototype development system I'm using to develop software.
Since a BIOS fix isn't likely to happen, are there existing features I
can use to work around this problem?

--chuck

Re: Error assigning BAR 0 to hotplugged device

[Yinghai Lu]

On Thu, Jul 17, 2014 at 8:08 AM, Chuck Tuffli <ctuffli@gmail.com> wrote:
> On Wed, Jul 16, 2014 at 7:15 PM, Yinghai Lu <yinghai@kernel.org> wrote:
>>> If the BIOS doesn't allocate enough memory mapped IO resources for
>>> devices, it sounds like the kernel can't really fix this problem,
>>> right?
>>
>> Yes. You need to fix BIOS at first.
>>
>> So it will be final products or just development prototype?
>
> This is a prototype development system I'm using to develop software.
> Since a BIOS fix isn't likely to happen, are there existing features I
> can use to work around this problem?

You need have a scripts with kexec to reload kernel.
1. make sure that card is the slot 9b:15.0
2. boot system with kernel.
3. unload or unbind the drivers for devices under 80:03.0
4. use setpci to clear bridge io, mmio, mmio pref bars with 80:03.0
5. kexec load kernel and start kernel again.

Re: Error assigning BAR 0 to hotplugged device

[Chuck Tuffli]

On Thu, Jul 17, 2014 at 9:33 AM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Thu, Jul 17, 2014 at 8:08 AM, Chuck Tuffli <ctuffli@gmail.com> wrote:
>> On Wed, Jul 16, 2014 at 7:15 PM, Yinghai Lu <yinghai@kernel.org> wrote:
>>>> If the BIOS doesn't allocate enough memory mapped IO resources for
>>>> devices, it sounds like the kernel can't really fix this problem,
>>>> right?
>>>
>>> Yes. You need to fix BIOS at first.
>>>
>>> So it will be final products or just development prototype?
>>
>> This is a prototype development system I'm using to develop software.
>> Since a BIOS fix isn't likely to happen, are there existing features I
>> can use to work around this problem?
>
> You need have a scripts with kexec to reload kernel.
> 1. make sure that card is the slot 9b:15.0
> 2. boot system with kernel.
> 3. unload or unbind the drivers for devices under 80:03.0
> 4. use setpci to clear bridge io, mmio, mmio pref bars with 80:03.0
> 5. kexec load kernel and start kernel again.

Perfect. Thanks so much for your explanations and advice!

--chuck