Q: Relation between lanes to devices and to BARs

Q: Relation between lanes to devices and to BARs

[Ran Shalit]

Hello,

I am new to pci express drivers, and would please like to ask a question.
We develop a linux driver for a PCIe device board with 4 lanes.
1. Is it expected that this will be recognized as a single device or
multiple device ?
2. Is it that each lane rales to a different BAR or there is no
connection between these two ?

Thank you,
Ran

Re: Q: Relation between lanes to devices and to BARs

[Alex Williamson]

On Sun, 2016-01-31 at 02:15 -0500, Ran Shalit wrote:
> Hello,
> 
> I am new to pci express drivers, and would please like to ask a question.
> We develop a linux driver for a PCIe device board with 4 lanes.
> 1. Is it expected that this will be recognized as a single device or
> multiple device ?

It depends whether your device is multifunction.

> 2. Is it that each lane rales to a different BAR or there is no
> connection between these two ?

No relation, all PCI/e devices have the same set of standard BARs
available to them.

Re: Q: Relation between lanes to devices and to BARs

[Ran Shalit]

On Mon, Feb 1, 2016 at 6:14 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Sun, 2016-01-31 at 02:15 -0500, Ran Shalit wrote:
>> Hello,
>>
>> I am new to pci express drivers, and would please like to ask a question.
>> We develop a linux driver for a PCIe device board with 4 lanes.
>> 1. Is it expected that this will be recognized as a single device or
>> multiple device ?
>
> It depends whether your device is multifunction.
>

It is a video capture & output deivce, (video4linux), so I guess it
means it has single functionality (video).' and therefore will aprear
as a single device.


>> 2. Is it that each lane rales to a different BAR or there is no
>> connection between these two ?
>
> No relation, all PCI/e devices have the same set of standard BARs
> available to them.
>

Thanks,
So it probably means that there is no added complexity in terms of
device driver with multi lanes PCIe (compared to single lane).


Thank you,
Ran

Re: Q: Relation between lanes to devices and to BARs

[Bjorn Helgaas]

On Mon, Feb 01, 2016 at 08:41:28PM +0200, Ran Shalit wrote:
> On Mon, Feb 1, 2016 at 6:14 AM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
> > On Sun, 2016-01-31 at 02:15 -0500, Ran Shalit wrote:
> >> Hello,
> >>
> >> I am new to pci express drivers, and would please like to ask a question.
> >> We develop a linux driver for a PCIe device board with 4 lanes.
> >> 1. Is it expected that this will be recognized as a single device or
> >> multiple device ?
> >
> > It depends whether your device is multifunction.
> 
> It is a video capture & output deivce, (video4linux), so I guess it
> means it has single functionality (video).' and therefore will aprear
> as a single device.

I'm not familiar with video devices, but it sounds likely that there
would be at least two functions (one for input, another for output).

It's a little confusing because the PCI/PCIe specs use "bus",
"device", and "function" to describe device addressing, but in the
Linux kernel, when we talk about a PCI device, we mean an individual
*function*.  We maintain a struct pci_dev for each function, and a
driver binds to the pci_dev for a single function.

If you run "lspci", it shows a line for each function.  Here are a few
from my system:

  00:1b.0 Audio device: Intel Corporation Lynx Point-LP HD Audio Controller
  00:1c.0 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 1
  00:1c.3 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 4
  00:1c.5 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 6

The 00 is the PCI bus number; 1b and 1c are PCI device numbers; the
.0, .3, and .5 are PCI function numbers.  In PCI spec terms, this is
one single-function device (the audio controller at 00:1b.0) and one
multi-function device with three functions (the root ports at 00:1c).

In the kernel, we manage each function individually, so there are four
pci_dev structs for the above list, and there could be four separate
drivers bound to them.

> >> 2. Is it that each lane rales to a different BAR or there is no
> >> connection between these two ?
> >
> > No relation, all PCI/e devices have the same set of standard BARs
> > available to them.
>
> So it probably means that there is no added complexity in terms of
> device driver with multi lanes PCIe (compared to single lane).

Right.  The number of PCIe lanes is invisible to the driver.  There
are ways to find out and influence how many lanes a device uses, but
in general, there's no need to do that.

Bjorn

Re: Q: Relation between lanes to devices and to BARs

[Ran Shalit]

On Mon, Feb 1, 2016 at 10:54 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Mon, Feb 01, 2016 at 08:41:28PM +0200, Ran Shalit wrote:
>> On Mon, Feb 1, 2016 at 6:14 AM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>> > On Sun, 2016-01-31 at 02:15 -0500, Ran Shalit wrote:
>> >> Hello,
>> >>
>> >> I am new to pci express drivers, and would please like to ask a question.
>> >> We develop a linux driver for a PCIe device board with 4 lanes.
>> >> 1. Is it expected that this will be recognized as a single device or
>> >> multiple device ?
>> >
>> > It depends whether your device is multifunction.
>>
>> It is a video capture & output deivce, (video4linux), so I guess it
>> means it has single functionality (video).' and therefore will aprear
>> as a single device.
>
> I'm not familiar with video devices, but it sounds likely that there
> would be at least two functions (one for input, another for output).
>
> It's a little confusing because the PCI/PCIe specs use "bus",
> "device", and "function" to describe device addressing, but in the
> Linux kernel, when we talk about a PCI device, we mean an individual
> *function*.  We maintain a struct pci_dev for each function, and a
> driver binds to the pci_dev for a single function.
>
> If you run "lspci", it shows a line for each function.  Here are a few
> from my system:
>
>   00:1b.0 Audio device: Intel Corporation Lynx Point-LP HD Audio Controller
>   00:1c.0 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 1
>   00:1c.3 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 4
>   00:1c.5 PCI bridge: Intel Corporation Lynx Point-LP PCI Express Root Port 6
>
> The 00 is the PCI bus number; 1b and 1c are PCI device numbers; the
> .0, .3, and .5 are PCI function numbers.  In PCI spec terms, this is
> one single-function device (the audio controller at 00:1b.0) and one
> multi-function device with three functions (the root ports at 00:1c).
>
> In the kernel, we manage each function individually, so there are four
> pci_dev structs for the above list, and there could be four separate
> drivers bound to them.
>
>> >> 2. Is it that each lane rales to a different BAR or there is no
>> >> connection between these two ?
>> >
>> > No relation, all PCI/e devices have the same set of standard BARs
>> > available to them.
>>
>> So it probably means that there is no added complexity in terms of
>> device driver with multi lanes PCIe (compared to single lane).
>
> Right.  The number of PCIe lanes is invisible to the driver.  There
> are ways to find out and influence how many lanes a device uses, but
> in general, there's no need to do that.
>
> Bjorn

Hello,

If I may please ask one more on this issue.
You have made it clear the terms lanes & device in relation to
PCI/PCI-e device driver.
What about port ? Is there any relation between port to lane/device or
is there no relation between these terms.
For example, can it be 1 device with 2 lanes and 8 ports for example ?

Thank you very much,
Ran

Re: Q: Relation between lanes to devices and to BARs

[Bjorn Helgaas]

On Thu, Feb 11, 2016 at 05:22:15PM +0200, Ran Shalit wrote:
> If I may please ask one more on this issue.
> You have made it clear the terms lanes & device in relation to
> PCI/PCI-e device driver.
> What about port ? Is there any relation between port to lane/device or
> is there no relation between these terms.

In Linux, PCIe Ports (Root Ports, Switch Upstream Ports, or Switch
Downstream Ports) are treated as devices.  There is a struct pci_dev
for each Port.  A Switch contains at least two Ports (an Upstream Port
and one or more Downstream Ports), and there is a pci_dev for each of
them.

There is no connection between the number of lanes and the number of
pci_dev structs.  The Linux PCI core doesn't do anything with the
number of lanes.  My understanding is that the devices on either end
of a Link normally negotiate the link width (number of lanes) between
themselves automatically, without software intervention.

> For example, can it be 1 device with 2 lanes and 8 ports for example ?

"1 device with 8 ports" is a phrase that would make sense in the PCIe
spec, where you could be talking about a Switch with one Upstream Port
and seven Downstream Ports.

But in the Linux kernel, since we treat each PCI/PCIe *function* as an
independent device, this looks like 8 devices.  There will be 8
pci_dev structs, one for each port.

The "2 lanes" part is irrelevant.  The number of lanes makes no
difference as far as how many pci_dev structs we have.

Bjorn