* MSI-X capability display of pciutils
@ 2013-05-31 22:14 David Epping
2013-05-31 22:48 ` Bjorn Helgaas
0 siblings, 1 reply; 7+ messages in thread
From: David Epping @ 2013-05-31 22:14 UTC (permalink / raw)
To: linux-pci
Hello,
I am inspecting the MSI-X setup of some PCIe devices and have questions
about the output of lspci.
It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
As far as I understand the PCI 3.0 specification MSI-X capabilities
define a 32 bit register for the upper address portion of the MSI target
addresses (at capability offset 4) and another 32 bit register pointing
to the BAR and inter-BAR offset for the table of lower address portions
of the MSI target addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and
offset? What do they mean?
Looking at the source code of pciutils (ls-caps.c) indicates that the
PBA line is the information pointing to the table of MSI-X entries,
while the vector table line is actually the upper 32bit of each MSI
target address. This should however not be split into BAR and offset in
that case, I believe.
What did I miss?
Thanks for clarifying,
David
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: MSI-X capability display of pciutils 2013-05-31 22:14 MSI-X capability display of pciutils David Epping @ 2013-05-31 22:48 ` Bjorn Helgaas 2013-05-31 23:33 ` David Epping 0 siblings, 1 reply; 7+ messages in thread From: Bjorn Helgaas @ 2013-05-31 22:48 UTC (permalink / raw) To: David Epping; +Cc: linux-pci@vger.kernel.org On Fri, May 31, 2013 at 4:14 PM, David Epping <david.epping@missinglinkelectronics.com> wrote: > Hello, > > I am inspecting the MSI-X setup of some PCIe devices and have questions > about the output of lspci. > > It looks something like this: > Capabilities: [50] MSI-X: Enable+ Count=32 Masked- > Vector table: BAR=0 offset=00002000 > PBA: BAR=0 offset=00003000 > > As far as I understand the PCI 3.0 specification MSI-X capabilities define a > 32 bit register for the upper address portion of the MSI target addresses > (at capability offset 4) and another 32 bit register pointing to the BAR and > inter-BAR offset for the table of lower address portions of the MSI target > addresses (at capability offset 8). > Why are there two lines in the lspci output indicating some BAR and offset? > What do they mean? My understanding is that there are three data structures here: 1) The MSI-X Capability (Figure 6-10) in config space 2) The "MSI-X Table" (Figure 6-11) in PCI memory space 3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory space The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table. In MSI (not MSI-X), there is only one MSI target address, and it is in the MSI capability. In MSI-X, each MSI-X vector can have its own target address, and those addresses are not in the capability itself; they are stored in the memory decoded by a BAR. In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries). lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables. Bjorn ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: MSI-X capability display of pciutils 2013-05-31 22:48 ` Bjorn Helgaas @ 2013-05-31 23:33 ` David Epping 2013-06-01 11:30 ` David Epping 0 siblings, 1 reply; 7+ messages in thread From: David Epping @ 2013-05-31 23:33 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci@vger.kernel.org On 01.06.2013 00:48, Bjorn Helgaas wrote: > On Fri, May 31, 2013 at 4:14 PM, David Epping > <david.epping@missinglinkelectronics.com> wrote: >> Hello, >> >> I am inspecting the MSI-X setup of some PCIe devices and have questions >> about the output of lspci. >> >> It looks something like this: >> Capabilities: [50] MSI-X: Enable+ Count=32 Masked- >> Vector table: BAR=0 offset=00002000 >> PBA: BAR=0 offset=00003000 >> >> As far as I understand the PCI 3.0 specification MSI-X capabilities define a >> 32 bit register for the upper address portion of the MSI target addresses >> (at capability offset 4) and another 32 bit register pointing to the BAR and >> inter-BAR offset for the table of lower address portions of the MSI target >> addresses (at capability offset 8). >> Why are there two lines in the lspci output indicating some BAR and offset? >> What do they mean? > My understanding is that there are three data structures here: > 1) The MSI-X Capability (Figure 6-10) in config space > 2) The "MSI-X Table" (Figure 6-11) in PCI memory space > 3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory space > > The "Vector table" line tells you how to find the "MSI-X Table," and > the "PBA" line tells you how to find the "MSI-X PBA" table. > > In MSI (not MSI-X), there is only one MSI target address, and it is in > the MSI capability. > > In MSI-X, each MSI-X vector can have its own target address, and those > addresses are not in the capability itself; they are stored in the > memory decoded by a BAR. > > In your case, you would read the bus address from BAR 0, and the > vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) > from that address. The PBA should be at offsets 0x3000-0x30ff (32 > 8-byte entries). > > lspci will show you the BAR values, but it doesn't dump PCI memory > space, so you'd have to use /dev/mem or a tool like > http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the > tables. > > Bjorn > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Bjorn, thank you for your quick reply. So far the concepts of MSI and MSI-X are clear. I did not have the meaning of the PBA acronym, but now I know what it stands for. I do however not see that reflected in the spec. As your figure references are off by one and I do not have the PBA figure, we should clarify the document used. I'm looking at PCI Local Bus Specification Revision 3.0 from August 2002. According to this document (and in case I'm reading it correctly) the pending bit is part of each MSI-X Vector Table entry (lowest bit in the address portion - section 6.8.2.6). The lspci source code and its output (and your explanation) however suggest that the pending bits are stored in a separate table. In addition this specification document has the common upper address portion of each MSI target address at capability offset 4 and the BAR and intra-BAR offset at capability offset 8. The pciutils source code defines the later to be at 4 and uses offset 8 to derive the PBA. I wonder where our disconnect is. What is the title and revision of the document you are referring to? Thanks again, David ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: MSI-X capability display of pciutils 2013-05-31 23:33 ` David Epping @ 2013-06-01 11:30 ` David Epping [not found] ` <COL126-W433C75A9D7FCC1A5A561B3B7930@phx.gbl> 2013-06-01 22:08 ` Bjorn Helgaas 0 siblings, 2 replies; 7+ messages in thread From: David Epping @ 2013-06-01 11:30 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci@vger.kernel.org Bjorn, I found a ECN and presentation by PCI-SIG. Both documents describe exactly the scheme you pointed out. Seems I have been using an old or draft version. I will request the current version right now. Thank you for your kind help and sorry for any confusion caused. Best regards, David On 01.06.2013 01:33, David Epping wrote: > On 01.06.2013 00:48, Bjorn Helgaas wrote: >> On Fri, May 31, 2013 at 4:14 PM, David Epping >> <david.epping@missinglinkelectronics.com> wrote: >>> Hello, >>> >>> I am inspecting the MSI-X setup of some PCIe devices and have questions >>> about the output of lspci. >>> >>> It looks something like this: >>> Capabilities: [50] MSI-X: Enable+ Count=32 Masked- >>> Vector table: BAR=0 offset=00002000 >>> PBA: BAR=0 offset=00003000 >>> >>> As far as I understand the PCI 3.0 specification MSI-X capabilities >>> define a >>> 32 bit register for the upper address portion of the MSI target >>> addresses >>> (at capability offset 4) and another 32 bit register pointing to the >>> BAR and >>> inter-BAR offset for the table of lower address portions of the MSI >>> target >>> addresses (at capability offset 8). >>> Why are there two lines in the lspci output indicating some BAR and >>> offset? >>> What do they mean? >> My understanding is that there are three data structures here: >> 1) The MSI-X Capability (Figure 6-10) in config space >> 2) The "MSI-X Table" (Figure 6-11) in PCI memory space >> 3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory >> space >> >> The "Vector table" line tells you how to find the "MSI-X Table," and >> the "PBA" line tells you how to find the "MSI-X PBA" table. >> >> In MSI (not MSI-X), there is only one MSI target address, and it is in >> the MSI capability. >> >> In MSI-X, each MSI-X vector can have its own target address, and those >> addresses are not in the capability itself; they are stored in the >> memory decoded by a BAR. >> >> In your case, you would read the bus address from BAR 0, and the >> vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) >> from that address. The PBA should be at offsets 0x3000-0x30ff (32 >> 8-byte entries). >> >> lspci will show you the BAR values, but it doesn't dump PCI memory >> space, so you'd have to use /dev/mem or a tool like >> http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the >> tables. >> >> Bjorn >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > Bjorn, > > thank you for your quick reply. > > So far the concepts of MSI and MSI-X are clear. I did not have the > meaning of the PBA acronym, but now I know what it stands for. > I do however not see that reflected in the spec. As your figure > references are off by one and I do not have the PBA figure, we should > clarify the document used. I'm looking at PCI Local Bus Specification > Revision 3.0 from August 2002. > > According to this document (and in case I'm reading it correctly) the > pending bit is part of each MSI-X Vector Table entry (lowest bit in > the address portion - section 6.8.2.6). The lspci source code and its > output (and your explanation) however suggest that the pending bits > are stored in a separate table. > In addition this specification document has the common upper address > portion of each MSI target address at capability offset 4 and the BAR > and intra-BAR offset at capability offset 8. The pciutils source code > defines the later to be at 4 and uses offset 8 to derive the PBA. > > I wonder where our disconnect is. What is the title and revision of > the document you are referring to? > Thanks again, > David > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- David Epping david.epping@missinglinkelectronics.com Skype: david.epping Mobile US: +1 (408) 4757-574 Mobile DE: +49 (179) 9656-223 Missing Link Electronics http://www.missinglinkelectronics.com Office DE: +49 (731) 802-3930 Office US: +1 (408) 457-0700 Skype: missing.link.electronics ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <COL126-W433C75A9D7FCC1A5A561B3B7930@phx.gbl>]
* RE: MSI-X capability display of pciutils [not found] ` <COL126-W433C75A9D7FCC1A5A561B3B7930@phx.gbl> @ 2013-06-01 14:40 ` David Epping 2013-06-01 22:19 ` Bjorn Helgaas 0 siblings, 1 reply; 7+ messages in thread From: David Epping @ 2013-06-01 14:40 UTC (permalink / raw) To: Pramod KS; +Cc: Bjorn Helgaas, linux-pci@vger.kernel.org > David/All, > Not sure if I can use this thread to ask but wonderingwhere do hobbyists > can get to see the PCIe spec? > >> Date: Sat, 1 Jun 2013 13:30:28 +0200 >> From: david.epping@missinglinkelectronics.com >> To: bhelgaas@google.com >> CC: linux-pci@vger.kernel.org >> Subject: Re: MSI-X capability display of pciutils >> >> Bjorn, >> >> I found a ECN and presentation by PCI-SIG. Both documents describe >> exactly the scheme you pointed out. >> Seems I have been using an old or draft version. I will request the >> current version right now. >> >> Thank you for your kind help and sorry for any confusion caused. >> Best regards, >> David >> >> On 01.06.2013 01:33, David Epping wrote: >> > On 01.06.2013 00:48, Bjorn Helgaas wrote: >> >> On Fri, May 31, 2013 at 4:14 PM, David Epping >> >> <david.epping@missinglinkelectronics.com> wrote: >> >>> Hello, >> >>> >> >>> I am inspecting the MSI-X setup of some PCIe devices and have >> questions >> >>> about the output of lspci. >> >>> >> >>> It looks something like this: >> >>> Capabilities: [50] MSI-X: Enable+ Count=32 Masked- >> >>> Vector table: BAR=0 offset=00002000 >> >>> PBA: BAR=0 offset=00003000 >> >>> >> >>> As far as I understand the PCI 3.0 specification MSI-X capabilities >> >>> define a >> >>> 32 bit register for the upper address portion of the MSI target >> >>> addresses >> >>> (at capability offset 4) and another 32 bit register pointing to the >> >>> BAR and >> >>> inter-BAR offset for the table of lower address portions of the MSI >> >>> target >> >>> addresses (at capability offset 8). >> >>> Why are there two lines in the lspci output indicating some BAR and >> >>> offset? >> >>> What do they mean? >> >> My understanding is that there are three data structures here: >> >> 1) The MSI-X Capability (Figure 6-10) in config space >> >> 2) The "MSI-X Table" (Figure 6-11) in PCI memory space >> >> 3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory >> >> space >> >> >> >> The "Vector table" line tells you how to find the "MSI-X Table," and >> >> the "PBA" line tells you how to find the "MSI-X PBA" table. >> >> >> >> In MSI (not MSI-X), there is only one MSI target address, and it is >> in >> >> the MSI capability. >> >> >> >> In MSI-X, each MSI-X vector can have its own target address, and >> those >> >> addresses are not in the capability itself; they are stored in the >> >> memory decoded by a BAR. >> >> >> >> In your case, you would read the bus address from BAR 0, and the >> >> vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) >> >> from that address. The PBA should be at offsets 0x3000-0x30ff (32 >> >> 8-byte entries). >> >> >> >> lspci will show you the BAR values, but it doesn't dump PCI memory >> >> space, so you'd have to use /dev/mem or a tool like >> >> http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the >> >> tables. >> >> >> >> Bjorn >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe linux-pci" >> in >> >> the body of a message to majordomo@vger.kernel.org >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > >> > Bjorn, >> > >> > thank you for your quick reply. >> > >> > So far the concepts of MSI and MSI-X are clear. I did not have the >> > meaning of the PBA acronym, but now I know what it stands for. >> > I do however not see that reflected in the spec. As your figure >> > references are off by one and I do not have the PBA figure, we should >> > clarify the document used. I'm looking at PCI Local Bus Specification >> > Revision 3.0 from August 2002. >> > >> > According to this document (and in case I'm reading it correctly) the >> > pending bit is part of each MSI-X Vector Table entry (lowest bit in >> > the address portion - section 6.8.2.6). The lspci source code and its >> > output (and your explanation) however suggest that the pending bits >> > are stored in a separate table. >> > In addition this specification document has the common upper address >> > portion of each MSI target address at capability offset 4 and the BAR >> > and intra-BAR offset at capability offset 8. The pciutils source code >> > defines the later to be at 4 and uses offset 8 to derive the PBA. >> > >> > I wonder where our disconnect is. What is the title and revision of >> > the document you are referring to? >> > Thanks again, >> > David >> > >> > -- >> > To unsubscribe from this list: send the line "unsubscribe linux-pci" >> in >> > the body of a message to majordomo@vger.kernel.org >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> David Epping >> david.epping@missinglinkelectronics.com >> Skype: david.epping >> Mobile US: +1 (408) 4757-574 >> Mobile DE: +49 (179) 9656-223 >> >> Missing Link Electronics >> http://www.missinglinkelectronics.com >> Office DE: +49 (731) 802-3930 >> Office US: +1 (408) 457-0700 >> Skype: missing.link.electronics >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > AFAIK there is no public (full or reduced) version of the relevant specifications. There are some documents available for download for non-members, but for the real spec your employer is probably your best chance. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: MSI-X capability display of pciutils 2013-06-01 14:40 ` David Epping @ 2013-06-01 22:19 ` Bjorn Helgaas 0 siblings, 0 replies; 7+ messages in thread From: Bjorn Helgaas @ 2013-06-01 22:19 UTC (permalink / raw) To: David Epping; +Cc: Pramod KS, linux-pci@vger.kernel.org On Sat, Jun 1, 2013 at 8:40 AM, David Epping <david.epping@missinglinkelectronics.com> wrote: >> David/All, >> Not sure if I can use this thread to ask but wonderingwhere do hobbyists >> can get to see the PCIe spec? > AFAIK there is no public (full or reduced) version of the relevant > specifications. There are some documents available for download for > non-members, but for the real spec your employer is probably your best > chance. The MindShare books are the only real public docs I know of. But in most cases, those books are pretty good. Bjorn ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: MSI-X capability display of pciutils 2013-06-01 11:30 ` David Epping [not found] ` <COL126-W433C75A9D7FCC1A5A561B3B7930@phx.gbl> @ 2013-06-01 22:08 ` Bjorn Helgaas 1 sibling, 0 replies; 7+ messages in thread From: Bjorn Helgaas @ 2013-06-01 22:08 UTC (permalink / raw) To: David Epping; +Cc: linux-pci@vger.kernel.org Interesting. I'm using the "PCI Local Bus Specification, Rev. 3.0," but dated Feb 3, 2004. Sounds like they screwed up assigning revision numbers, and released two different things as "rev 3.0." Sigh. On Sat, Jun 1, 2013 at 5:30 AM, David Epping <david.epping@missinglinkelectronics.com> wrote: > Bjorn, > > I found a ECN and presentation by PCI-SIG. Both documents describe exactly > the scheme you pointed out. > Seems I have been using an old or draft version. I will request the current > version right now. > > Thank you for your kind help and sorry for any confusion caused. > Best regards, > David > > > On 01.06.2013 01:33, David Epping wrote: >> >> On 01.06.2013 00:48, Bjorn Helgaas wrote: >>> >>> On Fri, May 31, 2013 at 4:14 PM, David Epping >>> <david.epping@missinglinkelectronics.com> wrote: >>>> >>>> Hello, >>>> >>>> I am inspecting the MSI-X setup of some PCIe devices and have questions >>>> about the output of lspci. >>>> >>>> It looks something like this: >>>> Capabilities: [50] MSI-X: Enable+ Count=32 Masked- >>>> Vector table: BAR=0 offset=00002000 >>>> PBA: BAR=0 offset=00003000 >>>> >>>> As far as I understand the PCI 3.0 specification MSI-X capabilities >>>> define a >>>> 32 bit register for the upper address portion of the MSI target >>>> addresses >>>> (at capability offset 4) and another 32 bit register pointing to the BAR >>>> and >>>> inter-BAR offset for the table of lower address portions of the MSI >>>> target >>>> addresses (at capability offset 8). >>>> Why are there two lines in the lspci output indicating some BAR and >>>> offset? >>>> What do they mean? >>> >>> My understanding is that there are three data structures here: >>> 1) The MSI-X Capability (Figure 6-10) in config space >>> 2) The "MSI-X Table" (Figure 6-11) in PCI memory space >>> 3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory >>> space >>> >>> The "Vector table" line tells you how to find the "MSI-X Table," and >>> the "PBA" line tells you how to find the "MSI-X PBA" table. >>> >>> In MSI (not MSI-X), there is only one MSI target address, and it is in >>> the MSI capability. >>> >>> In MSI-X, each MSI-X vector can have its own target address, and those >>> addresses are not in the capability itself; they are stored in the >>> memory decoded by a BAR. >>> >>> In your case, you would read the bus address from BAR 0, and the >>> vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) >>> from that address. The PBA should be at offsets 0x3000-0x30ff (32 >>> 8-byte entries). >>> >>> lspci will show you the BAR values, but it doesn't dump PCI memory >>> space, so you'd have to use /dev/mem or a tool like >>> http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the >>> tables. >>> >>> Bjorn >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> Bjorn, >> >> thank you for your quick reply. >> >> So far the concepts of MSI and MSI-X are clear. I did not have the meaning >> of the PBA acronym, but now I know what it stands for. >> I do however not see that reflected in the spec. As your figure references >> are off by one and I do not have the PBA figure, we should clarify the >> document used. I'm looking at PCI Local Bus Specification Revision 3.0 from >> August 2002. >> >> According to this document (and in case I'm reading it correctly) the >> pending bit is part of each MSI-X Vector Table entry (lowest bit in the >> address portion - section 6.8.2.6). The lspci source code and its output >> (and your explanation) however suggest that the pending bits are stored in a >> separate table. >> In addition this specification document has the common upper address >> portion of each MSI target address at capability offset 4 and the BAR and >> intra-BAR offset at capability offset 8. The pciutils source code defines >> the later to be at 4 and uses offset 8 to derive the PBA. >> >> I wonder where our disconnect is. What is the title and revision of the >> document you are referring to? >> Thanks again, >> David >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > David Epping > david.epping@missinglinkelectronics.com > Skype: david.epping > Mobile US: +1 (408) 4757-574 > Mobile DE: +49 (179) 9656-223 > > Missing Link Electronics > http://www.missinglinkelectronics.com > Office DE: +49 (731) 802-3930 > Office US: +1 (408) 457-0700 > Skype: missing.link.electronics > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-06-01 22:19 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-31 22:14 MSI-X capability display of pciutils David Epping
2013-05-31 22:48 ` Bjorn Helgaas
2013-05-31 23:33 ` David Epping
2013-06-01 11:30 ` David Epping
[not found] ` <COL126-W433C75A9D7FCC1A5A561B3B7930@phx.gbl>
2013-06-01 14:40 ` David Epping
2013-06-01 22:19 ` Bjorn Helgaas
2013-06-01 22:08 ` Bjorn Helgaas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.