[PATCH 0/3] Add PCI domain support to R-Car drivers

[PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller drivers,
one for an external PCIe slot, the other for an internal PCI bridge to USB
controllers.

However, they currently do not work at the same time as they use the same PCI
domain and use the same root bus number. We can't use different root bus numbers
due to the way root bus numbers are assigned in pcibios_init_hw() in
arch/arm/kernel/bios32.c.

Since the two PCI controllers are completely independent, I think it makes sense
to use different PCI domains for them.

I've marked the third patch as RFC as I am not sure of the impact of enabling
PCI domains for all ARM devices. In the march to 'one kernel to rule them all',
I steered clear of mach specific changes.

These patches require the following patch from Liviu Dudau:
  [PATCH v11 07/10] OF: Introduce helper function for getting PCI domain_nr
Based on comments on this patch from Jason Gunthorpe, there is still the issue
that the domain numbers may change depending on the ordering at probe time.
However, this can be fixed later on by adding the entries in the DT files.


Phil Edworthy (3):
  PCI: rcar-pcie: Add call to get domain nr
  PCI: rcar-internal-pci: Add call to get domain nr
  ARM: Enable PCI domains

 arch/arm/Kconfig                 | 2 +-
 drivers/pci/host/pci-rcar-gen2.c | 6 ++++++
 drivers/pci/host/pcie-rcar.c     | 9 ++++++---
 3 files changed, 13 insertions(+), 4 deletions(-)

-- 
2.1.0

[PATCH 1/3] PCI: rcar-pcie: Add call to get domain nr

[Phil Edworthy]

R-Car devices (r8a7790 and r8a7791) need to place the internal PCI and external
PCIe controllers on separate domains so that they can work at the same time.

Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
---
 drivers/pci/host/pcie-rcar.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/host/pcie-rcar.c b/drivers/pci/host/pcie-rcar.c
index 4884ee5..db74371 100644
--- a/drivers/pci/host/pcie-rcar.c
+++ b/drivers/pci/host/pcie-rcar.c
@@ -128,6 +128,7 @@ struct rcar_pcie {
 	struct device		*dev;
 	void __iomem		*base;
 	struct resource		res[RCAR_PCI_MAX_RESOURCES];
+	int			domain;
 	struct resource		busn;
 	int			root_bus_nr;
 	struct clk		*clk;
@@ -393,13 +394,13 @@ static void rcar_pcie_enable(struct rcar_pcie *pcie)
 {
 	struct platform_device *pdev = to_platform_device(pcie->dev);
 
+#ifdef CONFIG_PCI_DOMAINS
+	rcar_pci.domain = pcie->domain;
+#endif
 	rcar_pci.nr_controllers = 1;
 	rcar_pci.private_data = (void **)&pcie;
 
 	pci_common_init_dev(&pdev->dev, &rcar_pci);
-#ifdef CONFIG_PCI_DOMAINS
-	rcar_pci.domain++;
-#endif
 }
 
 static int phy_wait_for_ack(struct rcar_pcie *pcie)
@@ -917,6 +918,8 @@ static int rcar_pcie_probe(struct platform_device *pdev)
 	pcie->dev = &pdev->dev;
 	platform_set_drvdata(pdev, pcie);
 
+	pcie->domain = of_pci_get_domain_nr(pdev->dev.of_node, true);
+
 	/* Get the bus range */
 	if (of_pci_parse_bus_range(pdev->dev.of_node, &pcie->busn)) {
 		dev_err(&pdev->dev, "failed to parse bus-range property\n");
-- 
2.1.0

[PATCH 2/3] PCI: rcar-internal-pci: Add call to get domain nr

[Phil Edworthy]

R-Car devices (r8a7790 and r8a7791) need to place the internal PCI and external
PCIe controllers on separate domains so that they can work at the same time.

Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
---
 drivers/pci/host/pci-rcar-gen2.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/pci/host/pci-rcar-gen2.c b/drivers/pci/host/pci-rcar-gen2.c
index 3ef854f..17681ea 100644
--- a/drivers/pci/host/pci-rcar-gen2.c
+++ b/drivers/pci/host/pci-rcar-gen2.c
@@ -99,6 +99,7 @@ struct rcar_pci_priv {
 	struct resource io_res;
 	struct resource mem_res;
 	struct resource *cfg_res;
+	int domain;
 	unsigned busnr;
 	int irq;
 	unsigned long window_size;
@@ -373,6 +374,8 @@ static int rcar_pci_probe(struct platform_device *pdev)
 
 	priv->window_size = SZ_1G;
 
+	priv->domain = of_pci_get_domain_nr(pdev->dev.of_node, true);
+
 	if (pdev->dev.of_node) {
 		struct resource busnr;
 		int ret;
@@ -397,6 +400,9 @@ static int rcar_pci_probe(struct platform_device *pdev)
 	hw.map_irq = rcar_pci_map_irq;
 	hw.ops = &rcar_pci_ops;
 	hw.setup = rcar_pci_setup;
+#ifdef CONFIG_PCI_DOMAINS
+	hw.domain = priv->domain;
+#endif
 	pci_common_init_dev(&pdev->dev, &hw);
 	return 0;
 }
-- 
2.1.0

[RFC PATCH 3/3] ARM: Enable PCI domains

[Phil Edworthy]

Since there are ARM devices with multiple independant PCI controllers, enable
PCI domains for all ARM devices.

Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
---
Marked as RFC as I am not sure of the impact of enabling PCI domains for all
ARM devices. In the march to 'one kernel to rule them all', I steered clear of
mach specific changes.

 arch/arm/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 32cbbd5..8741d3d 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1274,6 +1274,7 @@ config ISA_DMA_API
 
 config PCI
 	bool "PCI support" if MIGHT_HAVE_PCI
+	select PCI_DOMAINS
 	help
 	  Find out whether you have a PCI motherboard. PCI is the name of a
 	  bus system, i.e. the way the CPU talks to the other stuff inside
@@ -1282,7 +1283,6 @@ config PCI
 
 config PCI_DOMAINS
 	bool
-	depends on PCI
 
 config PCI_NANOENGINE
 	bool "BSE nanoEngine PCI support"
-- 
2.1.0

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Liviu Dudau]

On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller drivers,
> one for an external PCIe slot, the other for an internal PCI bridge to USB
> controllers.
> 
> However, they currently do not work at the same time as they use the same PCI
> domain and use the same root bus number. We can't use different root bus numbers
> due to the way root bus numbers are assigned in pcibios_init_hw() in
> arch/arm/kernel/bios32.c.
> 
> Since the two PCI controllers are completely independent, I think it makes sense
> to use different PCI domains for them.
> 
> I've marked the third patch as RFC as I am not sure of the impact of enabling
> PCI domains for all ARM devices. In the march to 'one kernel to rule them all',
> I steered clear of mach specific changes.
> 
> These patches require the following patch from Liviu Dudau:
>   [PATCH v11 07/10] OF: Introduce helper function for getting PCI domain_nr
> Based on comments on this patch from Jason Gunthorpe, there is still the issue
> that the domain numbers may change depending on the ordering at probe time.
> However, this can be fixed later on by adding the entries in the DT files.

Hi Phil,

I'm happy that you found use for my patch, but I can't help wondering if it is
not a better idea to convert your drivers to the whole new API in my series.
Do you have any thoughts on that?

One other thing to note: Rob Herring is not very happy with the mix of OF
parsing and domain number allocation going in together, so I might remove the
OF parsing entirely for now. Would that cause you any problems (other than
the fact that if of_pci_get_domain_nr() doesn't do anything OF related I might
rename it to pci_get_domain_nr()). You don't seem to have any alias defined
for pci-domain in the DT, so I'll guess not.

Best regards,
Liviu

> 
> 
> Phil Edworthy (3):
>   PCI: rcar-pcie: Add call to get domain nr
>   PCI: rcar-internal-pci: Add call to get domain nr
>   ARM: Enable PCI domains
> 
>  arch/arm/Kconfig                 | 2 +-
>  drivers/pci/host/pci-rcar-gen2.c | 6 ++++++
>  drivers/pci/host/pcie-rcar.c     | 9 ++++++---
>  3 files changed, 13 insertions(+), 4 deletions(-)
> 
> -- 
> 2.1.0
> 
> 

-- 
==========
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

RE: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

SGkgTGl2aXUsDQoNCk9uIDIyIFNlcHRlbWJlciAyMDE0IDEyOjI5LCBMaXZpdSB3cm90ZToNCj4g
T24gTW9uLCBTZXAgMjIsIDIwMTQgYXQgMTA6NTE6MDdBTSArMDEwMCwgUGhpbCBFZHdvcnRoeSB3
cm90ZToNCj4gPiBUaGUgUmVuZXNhcyBSLUNhciBkZXZpY2VzIChyOGE3NzkwIGFuZCByOGE3Nzkx
KSB1c2UgdHdvIFBDSSBjb250cm9sbGVyDQo+IGRyaXZlcnMsDQo+ID4gb25lIGZvciBhbiBleHRl
cm5hbCBQQ0llIHNsb3QsIHRoZSBvdGhlciBmb3IgYW4gaW50ZXJuYWwgUENJIGJyaWRnZSB0byBV
U0INCj4gPiBjb250cm9sbGVycy4NCj4gPg0KPiA+IEhvd2V2ZXIsIHRoZXkgY3VycmVudGx5IGRv
IG5vdCB3b3JrIGF0IHRoZSBzYW1lIHRpbWUgYXMgdGhleSB1c2UgdGhlDQo+IHNhbWUgUENJDQo+
ID4gZG9tYWluIGFuZCB1c2UgdGhlIHNhbWUgcm9vdCBidXMgbnVtYmVyLiBXZSBjYW4ndCB1c2Ug
ZGlmZmVyZW50IHJvb3QNCj4gYnVzIG51bWJlcnMNCj4gPiBkdWUgdG8gdGhlIHdheSByb290IGJ1
cyBudW1iZXJzIGFyZSBhc3NpZ25lZCBpbiBwY2liaW9zX2luaXRfaHcoKSBpbg0KPiA+IGFyY2gv
YXJtL2tlcm5lbC9iaW9zMzIuYy4NCj4gPg0KPiA+IFNpbmNlIHRoZSB0d28gUENJIGNvbnRyb2xs
ZXJzIGFyZSBjb21wbGV0ZWx5IGluZGVwZW5kZW50LCBJIHRoaW5rIGl0IG1ha2VzDQo+IHNlbnNl
DQo+ID4gdG8gdXNlIGRpZmZlcmVudCBQQ0kgZG9tYWlucyBmb3IgdGhlbS4NCj4gPg0KPiA+IEkn
dmUgbWFya2VkIHRoZSB0aGlyZCBwYXRjaCBhcyBSRkMgYXMgSSBhbSBub3Qgc3VyZSBvZiB0aGUg
aW1wYWN0IG9mDQo+IGVuYWJsaW5nDQo+ID4gUENJIGRvbWFpbnMgZm9yIGFsbCBBUk0gZGV2aWNl
cy4gSW4gdGhlIG1hcmNoIHRvICdvbmUga2VybmVsIHRvIHJ1bGUgdGhlbQ0KPiBhbGwnLA0KPiA+
IEkgc3RlZXJlZCBjbGVhciBvZiBtYWNoIHNwZWNpZmljIGNoYW5nZXMuDQo+ID4NCj4gPiBUaGVz
ZSBwYXRjaGVzIHJlcXVpcmUgdGhlIGZvbGxvd2luZyBwYXRjaCBmcm9tIExpdml1IER1ZGF1Og0K
PiA+ICAgW1BBVENIIHYxMSAwNy8xMF0gT0Y6IEludHJvZHVjZSBoZWxwZXIgZnVuY3Rpb24gZm9y
IGdldHRpbmcgUENJDQo+IGRvbWFpbl9ucg0KPiA+IEJhc2VkIG9uIGNvbW1lbnRzIG9uIHRoaXMg
cGF0Y2ggZnJvbSBKYXNvbiBHdW50aG9ycGUsIHRoZXJlIGlzIHN0aWxsIHRoZQ0KPiBpc3N1ZQ0K
PiA+IHRoYXQgdGhlIGRvbWFpbiBudW1iZXJzIG1heSBjaGFuZ2UgZGVwZW5kaW5nIG9uIHRoZSBv
cmRlcmluZyBhdCBwcm9iZQ0KPiB0aW1lLg0KPiA+IEhvd2V2ZXIsIHRoaXMgY2FuIGJlIGZpeGVk
IGxhdGVyIG9uIGJ5IGFkZGluZyB0aGUgZW50cmllcyBpbiB0aGUgRFQgZmlsZXMuDQo+IA0KPiBI
aSBQaGlsLA0KPiANCj4gSSdtIGhhcHB5IHRoYXQgeW91IGZvdW5kIHVzZSBmb3IgbXkgcGF0Y2gs
IGJ1dCBJIGNhbid0IGhlbHAgd29uZGVyaW5nIGlmIGl0IGlzDQo+IG5vdCBhIGJldHRlciBpZGVh
IHRvIGNvbnZlcnQgeW91ciBkcml2ZXJzIHRvIHRoZSB3aG9sZSBuZXcgQVBJIGluIG15IHNlcmll
cy4NCj4gRG8geW91IGhhdmUgYW55IHRob3VnaHRzIG9uIHRoYXQ/DQpUaGUgbWFpbiByZWFzb24g
YmVpbmcgdGhhdCBSZW5lc2FzIGFsc28gc3VwcG9ydHMgdGhlIExUU0kgdHJlZXMsIHNvIHRoaXMg
Zml4DQp3aWxsIGFsc28gbmVlZCBiYWNrcG9ydGluZyB0byB2My4xMC4gDQoNCj4gT25lIG90aGVy
IHRoaW5nIHRvIG5vdGU6IFJvYiBIZXJyaW5nIGlzIG5vdCB2ZXJ5IGhhcHB5IHdpdGggdGhlIG1p
eCBvZiBPRg0KPiBwYXJzaW5nIGFuZCBkb21haW4gbnVtYmVyIGFsbG9jYXRpb24gZ29pbmcgaW4g
dG9nZXRoZXIsIHNvIEkgbWlnaHQgcmVtb3ZlDQo+IHRoZQ0KPiBPRiBwYXJzaW5nIGVudGlyZWx5
IGZvciBub3cuIFdvdWxkIHRoYXQgY2F1c2UgeW91IGFueSBwcm9ibGVtcyAob3RoZXIgdGhhbg0K
PiB0aGUgZmFjdCB0aGF0IGlmIG9mX3BjaV9nZXRfZG9tYWluX25yKCkgZG9lc24ndCBkbyBhbnl0
aGluZyBPRiByZWxhdGVkIEkNCj4gbWlnaHQNCj4gcmVuYW1lIGl0IHRvIHBjaV9nZXRfZG9tYWlu
X25yKCkpLiBZb3UgZG9uJ3Qgc2VlbSB0byBoYXZlIGFueSBhbGlhcyBkZWZpbmVkDQo+IGZvciBw
Y2ktZG9tYWluIGluIHRoZSBEVCwgc28gSSdsbCBndWVzcyBub3QuDQpJbiB0aGUgY292ZXJpbmcg
bGV0dGVyIEkgbWVudGlvbmVkIHRoYXQgY29udHJvbGxlcnMgc2hvdWxkIGFsd2F5cyB1c2UgdGhl
IHNhbWUNCmRvbWFpbiBudW1iZXIsIHNvIEkgd2FzIHBsYW5uaW5nIHRvIGFkZCBwY2ktZG9tYWlu
IGludG8gdGhlIERUcyBsYXRlciBvbi4NCg0KVGhhbmtzDQpQaGlsDQo

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Liviu Dudau]

On Mon, Sep 22, 2014 at 12:40:42PM +0100, Phil Edworthy wrote:
> Hi Liviu,
> 
> On 22 September 2014 12:29, Liviu wrote:
> > On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> > > The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller
> > drivers,
> > > one for an external PCIe slot, the other for an internal PCI bridge to USB
> > > controllers.
> > >
> > > However, they currently do not work at the same time as they use the
> > same PCI
> > > domain and use the same root bus number. We can't use different root
> > bus numbers
> > > due to the way root bus numbers are assigned in pcibios_init_hw() in
> > > arch/arm/kernel/bios32.c.
> > >
> > > Since the two PCI controllers are completely independent, I think it makes
> > sense
> > > to use different PCI domains for them.
> > >
> > > I've marked the third patch as RFC as I am not sure of the impact of
> > enabling
> > > PCI domains for all ARM devices. In the march to 'one kernel to rule them
> > all',
> > > I steered clear of mach specific changes.
> > >
> > > These patches require the following patch from Liviu Dudau:
> > >   [PATCH v11 07/10] OF: Introduce helper function for getting PCI
> > domain_nr
> > > Based on comments on this patch from Jason Gunthorpe, there is still the
> > issue
> > > that the domain numbers may change depending on the ordering at probe
> > time.
> > > However, this can be fixed later on by adding the entries in the DT files.
> > 
> > Hi Phil,
> > 
> > I'm happy that you found use for my patch, but I can't help wondering if it is
> > not a better idea to convert your drivers to the whole new API in my series.
> > Do you have any thoughts on that?
> The main reason being that Renesas also supports the LTSI trees, so this fix
> will also need backporting to v3.10. 
> 
> > One other thing to note: Rob Herring is not very happy with the mix of OF
> > parsing and domain number allocation going in together, so I might remove
> > the
> > OF parsing entirely for now. Would that cause you any problems (other than
> > the fact that if of_pci_get_domain_nr() doesn't do anything OF related I
> > might
> > rename it to pci_get_domain_nr()). You don't seem to have any alias defined
> > for pci-domain in the DT, so I'll guess not.
> In the covering letter I mentioned that controllers should always use the same
> domain number, so I was planning to add pci-domain into the DTs later on.

Rob Herring seems to dislike the pci-domain alias solution, so it might have
to be a node in the host controller parent node, with linux,pci-domain.

Best regards,
Liviu

> 
> Thanks
> Phil


-- 
==========
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Bjorn Helgaas]

On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller drivers,
> one for an external PCIe slot, the other for an internal PCI bridge to USB
> controllers.
> 
> However, they currently do not work at the same time as they use the same PCI
> domain and use the same root bus number. We can't use different root bus numbers
> due to the way root bus numbers are assigned in pcibios_init_hw() in
> arch/arm/kernel/bios32.c.
> 
> Since the two PCI controllers are completely independent, I think it makes sense
> to use different PCI domains for them.
> 
> I've marked the third patch as RFC as I am not sure of the impact of enabling
> PCI domains for all ARM devices. In the march to 'one kernel to rule them all',
> I steered clear of mach specific changes.
> 
> These patches require the following patch from Liviu Dudau:
>   [PATCH v11 07/10] OF: Introduce helper function for getting PCI domain_nr
> Based on comments on this patch from Jason Gunthorpe, there is still the issue
> that the domain numbers may change depending on the ordering at probe time.
> However, this can be fixed later on by adding the entries in the DT files.
> 
> 
> Phil Edworthy (3):
>   PCI: rcar-pcie: Add call to get domain nr
>   PCI: rcar-internal-pci: Add call to get domain nr
>   ARM: Enable PCI domains

I'm deferring these for now because they depend on Liviu's work, which I
haven't merged yet, and I suspect some minor adaptation will be required
here.

For what it's worth, I agree with Rob's hesitation about mixing lookup with
domain number allocation in of_pci_get_domain_nr().  That seems
unnecessarily complicated.

>  arch/arm/Kconfig                 | 2 +-
>  drivers/pci/host/pci-rcar-gen2.c | 6 ++++++
>  drivers/pci/host/pcie-rcar.c     | 9 ++++++---
>  3 files changed, 13 insertions(+), 4 deletions(-)
> 
> -- 
> 2.1.0
> 

RE: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

Hi Bjorn,

On 22 September 2014 22:01, Bjorn wrote:
> On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> > The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller
> drivers,
> > one for an external PCIe slot, the other for an internal PCI bridge to USB
> > controllers.
> >
> > However, they currently do not work at the same time as they use the
> same PCI
> > domain and use the same root bus number. We can't use different root
> bus numbers
> > due to the way root bus numbers are assigned in pcibios_init_hw() in
> > arch/arm/kernel/bios32.c.
> >
> > Since the two PCI controllers are completely independent, I think it makes
> sense
> > to use different PCI domains for them.
> >
> > I've marked the third patch as RFC as I am not sure of the impact of
> enabling
> > PCI domains for all ARM devices. In the march to 'one kernel to rule them
> all',
> > I steered clear of mach specific changes.
> >
> > These patches require the following patch from Liviu Dudau:
> >   [PATCH v11 07/10] OF: Introduce helper function for getting PCI
> domain_nr
> > Based on comments on this patch from Jason Gunthorpe, there is still the
> issue
> > that the domain numbers may change depending on the ordering at probe
> time.
> > However, this can be fixed later on by adding the entries in the DT files.
> >
> >
> > Phil Edworthy (3):
> >   PCI: rcar-pcie: Add call to get domain nr
> >   PCI: rcar-internal-pci: Add call to get domain nr
> >   ARM: Enable PCI domains
> 
> I'm deferring these for now because they depend on Liviu's work, which I
> haven't merged yet, and I suspect some minor adaptation will be required
> here.
> 
> For what it's worth, I agree with Rob's hesitation about mixing lookup with
> domain number allocation in of_pci_get_domain_nr().  That seems
> unnecessarily complicated.
I could create patches to add an optional "pci-domain" property for the R-Car
PCI drivers, and just attempt to get the property in the drivers. If not found,
the drivers will assume the domain is 0.

We would then have fixed PCI domain numbering and I don't have to worry about
Liviu's work.

Would that be ok?
 
Thanks
Phil

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Liviu Dudau]

On Tue, Sep 23, 2014 at 11:10:29AM +0100, Phil Edworthy wrote:
> Hi Bjorn,
> 
> On 22 September 2014 22:01, Bjorn wrote:
> > On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> > > The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller
> > drivers,
> > > one for an external PCIe slot, the other for an internal PCI bridge to USB
> > > controllers.
> > >
> > > However, they currently do not work at the same time as they use the
> > same PCI
> > > domain and use the same root bus number. We can't use different root
> > bus numbers
> > > due to the way root bus numbers are assigned in pcibios_init_hw() in
> > > arch/arm/kernel/bios32.c.
> > >
> > > Since the two PCI controllers are completely independent, I think it makes
> > sense
> > > to use different PCI domains for them.
> > >
> > > I've marked the third patch as RFC as I am not sure of the impact of
> > enabling
> > > PCI domains for all ARM devices. In the march to 'one kernel to rule them
> > all',
> > > I steered clear of mach specific changes.
> > >
> > > These patches require the following patch from Liviu Dudau:
> > >   [PATCH v11 07/10] OF: Introduce helper function for getting PCI
> > domain_nr
> > > Based on comments on this patch from Jason Gunthorpe, there is still the
> > issue
> > > that the domain numbers may change depending on the ordering at probe
> > time.
> > > However, this can be fixed later on by adding the entries in the DT files.
> > >
> > >
> > > Phil Edworthy (3):
> > >   PCI: rcar-pcie: Add call to get domain nr
> > >   PCI: rcar-internal-pci: Add call to get domain nr
> > >   ARM: Enable PCI domains
> > 
> > I'm deferring these for now because they depend on Liviu's work, which I
> > haven't merged yet, and I suspect some minor adaptation will be required
> > here.
> > 
> > For what it's worth, I agree with Rob's hesitation about mixing lookup with
> > domain number allocation in of_pci_get_domain_nr().  That seems
> > unnecessarily complicated.
> I could create patches to add an optional "pci-domain" property for the R-Car
> PCI drivers, and just attempt to get the property in the drivers. If not found,
> the drivers will assume the domain is 0.
> 
> We would then have fixed PCI domain numbering and I don't have to worry about
> Liviu's work.

I will split the current of_pci_get_domain_nr() even further and replace it with
two functions: pci_get_domain_nr() which will do just the allocation (still based
on the boolean flag passed as parameter) and of_get_pci_domain_nr() that will
retrieve a "linux,pci-domain" value from a property belonging to a given device
node.

I plan to leave for the moment the check for mandatory presence of "linux,pci-domain"
to the host bridge driver so that everyone can implement their own policies.

How does that sound?

Best regards,
Liviu

> 
> Would that be ok?
>  
> Thanks
> Phil
> 

-- 
==========
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

RE: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

SGkgTGl2aXUsDQoNCk9uIDIzIFNlcHRlbWJlciAyMDE0IDExOjMyLCBMaXZpdSB3cm90ZToNCj4g
T24gVHVlLCBTZXAgMjMsIDIwMTQgYXQgMTE6MTA6MjlBTSArMDEwMCwgUGhpbCBFZHdvcnRoeSB3
cm90ZToNCj4gPiBIaSBCam9ybiwNCj4gPg0KPiA+IE9uIDIyIFNlcHRlbWJlciAyMDE0IDIyOjAx
LCBCam9ybiB3cm90ZToNCj4gPiA+IE9uIE1vbiwgU2VwIDIyLCAyMDE0IGF0IDEwOjUxOjA3QU0g
KzAxMDAsIFBoaWwgRWR3b3J0aHkgd3JvdGU6DQo+ID4gPiA+IFRoZSBSZW5lc2FzIFItQ2FyIGRl
dmljZXMgKHI4YTc3OTAgYW5kIHI4YTc3OTEpIHVzZSB0d28gUENJIGNvbnRyb2xsZXINCj4gPiA+
IGRyaXZlcnMsDQo+ID4gPiA+IG9uZSBmb3IgYW4gZXh0ZXJuYWwgUENJZSBzbG90LCB0aGUgb3Ro
ZXIgZm9yIGFuIGludGVybmFsIFBDSSBicmlkZ2UgdG8gVVNCDQo+ID4gPiA+IGNvbnRyb2xsZXJz
Lg0KPiA+ID4gPg0KPiA+ID4gPiBIb3dldmVyLCB0aGV5IGN1cnJlbnRseSBkbyBub3Qgd29yayBh
dCB0aGUgc2FtZSB0aW1lIGFzIHRoZXkgdXNlIHRoZQ0KPiA+ID4gc2FtZSBQQ0kNCj4gPiA+ID4g
ZG9tYWluIGFuZCB1c2UgdGhlIHNhbWUgcm9vdCBidXMgbnVtYmVyLiBXZSBjYW4ndCB1c2UgZGlm
ZmVyZW50IHJvb3QNCj4gPiA+IGJ1cyBudW1iZXJzDQo+ID4gPiA+IGR1ZSB0byB0aGUgd2F5IHJv
b3QgYnVzIG51bWJlcnMgYXJlIGFzc2lnbmVkIGluIHBjaWJpb3NfaW5pdF9odygpIGluDQo+ID4g
PiA+IGFyY2gvYXJtL2tlcm5lbC9iaW9zMzIuYy4NCj4gPiA+ID4NCj4gPiA+ID4gU2luY2UgdGhl
IHR3byBQQ0kgY29udHJvbGxlcnMgYXJlIGNvbXBsZXRlbHkgaW5kZXBlbmRlbnQsIEkgdGhpbmsg
aXQgbWFrZXMNCj4gPiA+IHNlbnNlDQo+ID4gPiA+IHRvIHVzZSBkaWZmZXJlbnQgUENJIGRvbWFp
bnMgZm9yIHRoZW0uDQo+ID4gPiA+DQo+ID4gPiA+IEkndmUgbWFya2VkIHRoZSB0aGlyZCBwYXRj
aCBhcyBSRkMgYXMgSSBhbSBub3Qgc3VyZSBvZiB0aGUgaW1wYWN0IG9mDQo+ID4gPiBlbmFibGlu
Zw0KPiA+ID4gPiBQQ0kgZG9tYWlucyBmb3IgYWxsIEFSTSBkZXZpY2VzLiBJbiB0aGUgbWFyY2gg
dG8gJ29uZSBrZXJuZWwgdG8gcnVsZSB0aGVtDQo+ID4gPiBhbGwnLA0KPiA+ID4gPiBJIHN0ZWVy
ZWQgY2xlYXIgb2YgbWFjaCBzcGVjaWZpYyBjaGFuZ2VzLg0KPiA+ID4gPg0KPiA+ID4gPiBUaGVz
ZSBwYXRjaGVzIHJlcXVpcmUgdGhlIGZvbGxvd2luZyBwYXRjaCBmcm9tIExpdml1IER1ZGF1Og0K
PiA+ID4gPiAgIFtQQVRDSCB2MTEgMDcvMTBdIE9GOiBJbnRyb2R1Y2UgaGVscGVyIGZ1bmN0aW9u
IGZvciBnZXR0aW5nIFBDSQ0KPiA+ID4gZG9tYWluX25yDQo+ID4gPiA+IEJhc2VkIG9uIGNvbW1l
bnRzIG9uIHRoaXMgcGF0Y2ggZnJvbSBKYXNvbiBHdW50aG9ycGUsIHRoZXJlIGlzIHN0aWxsIHRo
ZQ0KPiA+ID4gaXNzdWUNCj4gPiA+ID4gdGhhdCB0aGUgZG9tYWluIG51bWJlcnMgbWF5IGNoYW5n
ZSBkZXBlbmRpbmcgb24gdGhlIG9yZGVyaW5nIGF0IHByb2JlDQo+ID4gPiB0aW1lLg0KPiA+ID4g
PiBIb3dldmVyLCB0aGlzIGNhbiBiZSBmaXhlZCBsYXRlciBvbiBieSBhZGRpbmcgdGhlIGVudHJp
ZXMgaW4gdGhlIERUIGZpbGVzLg0KPiA+ID4gPg0KPiA+ID4gPg0KPiA+ID4gPiBQaGlsIEVkd29y
dGh5ICgzKToNCj4gPiA+ID4gICBQQ0k6IHJjYXItcGNpZTogQWRkIGNhbGwgdG8gZ2V0IGRvbWFp
biBucg0KPiA+ID4gPiAgIFBDSTogcmNhci1pbnRlcm5hbC1wY2k6IEFkZCBjYWxsIHRvIGdldCBk
b21haW4gbnINCj4gPiA+ID4gICBBUk06IEVuYWJsZSBQQ0kgZG9tYWlucw0KPiA+ID4NCj4gPiA+
IEknbSBkZWZlcnJpbmcgdGhlc2UgZm9yIG5vdyBiZWNhdXNlIHRoZXkgZGVwZW5kIG9uIExpdml1
J3Mgd29yaywgd2hpY2ggSQ0KPiA+ID4gaGF2ZW4ndCBtZXJnZWQgeWV0LCBhbmQgSSBzdXNwZWN0
IHNvbWUgbWlub3IgYWRhcHRhdGlvbiB3aWxsIGJlIHJlcXVpcmVkDQo+ID4gPiBoZXJlLg0KPiA+
ID4NCj4gPiA+IEZvciB3aGF0IGl0J3Mgd29ydGgsIEkgYWdyZWUgd2l0aCBSb2IncyBoZXNpdGF0
aW9uIGFib3V0IG1peGluZyBsb29rdXAgd2l0aA0KPiA+ID4gZG9tYWluIG51bWJlciBhbGxvY2F0
aW9uIGluIG9mX3BjaV9nZXRfZG9tYWluX25yKCkuICBUaGF0IHNlZW1zDQo+ID4gPiB1bm5lY2Vz
c2FyaWx5IGNvbXBsaWNhdGVkLg0KPiA+IEkgY291bGQgY3JlYXRlIHBhdGNoZXMgdG8gYWRkIGFu
IG9wdGlvbmFsICJwY2ktZG9tYWluIiBwcm9wZXJ0eSBmb3IgdGhlIFItQ2FyDQo+ID4gUENJIGRy
aXZlcnMsIGFuZCBqdXN0IGF0dGVtcHQgdG8gZ2V0IHRoZSBwcm9wZXJ0eSBpbiB0aGUgZHJpdmVy
cy4gSWYgbm90IGZvdW5kLA0KPiA+IHRoZSBkcml2ZXJzIHdpbGwgYXNzdW1lIHRoZSBkb21haW4g
aXMgMC4NCj4gPg0KPiA+IFdlIHdvdWxkIHRoZW4gaGF2ZSBmaXhlZCBQQ0kgZG9tYWluIG51bWJl
cmluZyBhbmQgSSBkb24ndCBoYXZlIHRvIHdvcnJ5IGFib3V0DQo+ID4gTGl2aXUncyB3b3JrLg0K
PiANCj4gSSB3aWxsIHNwbGl0IHRoZSBjdXJyZW50IG9mX3BjaV9nZXRfZG9tYWluX25yKCkgZXZl
biBmdXJ0aGVyIGFuZCByZXBsYWNlIGl0IHdpdGgNCj4gdHdvIGZ1bmN0aW9uczogcGNpX2dldF9k
b21haW5fbnIoKSB3aGljaCB3aWxsIGRvIGp1c3QgdGhlIGFsbG9jYXRpb24gKHN0aWxsIGJhc2Vk
DQo+IG9uIHRoZSBib29sZWFuIGZsYWcgcGFzc2VkIGFzIHBhcmFtZXRlcikgYW5kIG9mX2dldF9w
Y2lfZG9tYWluX25yKCkgdGhhdCB3aWxsDQo+IHJldHJpZXZlIGEgImxpbnV4LHBjaS1kb21haW4i
IHZhbHVlIGZyb20gYSBwcm9wZXJ0eSBiZWxvbmdpbmcgdG8gYSBnaXZlbiBkZXZpY2UNCj4gbm9k
ZS4NClRoaXMgZG9lc24ndCBzb2x2ZSB0aGUgcHJvYmxlbSBvZiBkaWZmZXJlbnQgZG9tYWluIG51
bWJlcnMgYmFzZWQgb24gZGlmZmVyZW50DQpwcm9iZSBvcmRlcmluZy4gSWYgeW91IGhhdmUgbXVs
dGlwbGUgZG9tYWlucyB0aGVuIEkgdGhpbmsgeW91IG11c3QgaGF2ZSB0aGUNCiJsaW51eCxwY2kt
ZG9tYWluIiBwcm9wZXJ0eSBmb3IgZWFjaCBjb250cm9sbGVyLg0KDQpJZiB3ZSBhc3N1bWUgdGhl
IGFib3ZlLCBvZl9nZXRfcGNpX2RvbWFpbl9ucigpIGNhbiBqdXN0IHJldHVybiBhIGJ1cyBudW1i
ZXIgb2YNCjAgaWYgdGhlICJsaW51eCxwY2ktZG9tYWluIiBwcm9wZXJ0eSBkb2Vzbid0IGV4aXN0
Lg0KDQpJIHN1cHBvc2UgeW91IGNvdWxkIGhhdmUgYSB0ZW1wb3Jhcnkgc29sdXRpb24gdGhhdCBh
bGxvY2F0ZXMgdGhlIGRvbWFpbiBudW1iZXJzDQp1bnRpbCB0aGUgbmVjZXNzYXJ5IGRyaXZlcnMg
aGF2ZSBiZWVuIHVwZGF0ZWQuDQoNCj4gSSBwbGFuIHRvIGxlYXZlIGZvciB0aGUgbW9tZW50IHRo
ZSBjaGVjayBmb3IgbWFuZGF0b3J5IHByZXNlbmNlIG9mDQo+ICJsaW51eCxwY2ktZG9tYWluIg0K
PiB0byB0aGUgaG9zdCBicmlkZ2UgZHJpdmVyIHNvIHRoYXQgZXZlcnlvbmUgY2FuIGltcGxlbWVu
dCB0aGVpciBvd24gcG9saWNpZXMuDQo+IA0KPiBIb3cgZG9lcyB0aGF0IHNvdW5kPw0KDQpUaGFu
a3MNClBoaWwNCg=

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Liviu Dudau]

On Tue, Sep 23, 2014 at 12:00:41PM +0100, Phil Edworthy wrote:
> Hi Liviu,
> 
> On 23 September 2014 11:32, Liviu wrote:
> > On Tue, Sep 23, 2014 at 11:10:29AM +0100, Phil Edworthy wrote:
> > > Hi Bjorn,
> > >
> > > On 22 September 2014 22:01, Bjorn wrote:
> > > > On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> > > > > The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller
> > > > drivers,
> > > > > one for an external PCIe slot, the other for an internal PCI bridge to USB
> > > > > controllers.
> > > > >
> > > > > However, they currently do not work at the same time as they use the
> > > > same PCI
> > > > > domain and use the same root bus number. We can't use different root
> > > > bus numbers
> > > > > due to the way root bus numbers are assigned in pcibios_init_hw() in
> > > > > arch/arm/kernel/bios32.c.
> > > > >
> > > > > Since the two PCI controllers are completely independent, I think it makes
> > > > sense
> > > > > to use different PCI domains for them.
> > > > >
> > > > > I've marked the third patch as RFC as I am not sure of the impact of
> > > > enabling
> > > > > PCI domains for all ARM devices. In the march to 'one kernel to rule them
> > > > all',
> > > > > I steered clear of mach specific changes.
> > > > >
> > > > > These patches require the following patch from Liviu Dudau:
> > > > >   [PATCH v11 07/10] OF: Introduce helper function for getting PCI
> > > > domain_nr
> > > > > Based on comments on this patch from Jason Gunthorpe, there is still the
> > > > issue
> > > > > that the domain numbers may change depending on the ordering at probe
> > > > time.
> > > > > However, this can be fixed later on by adding the entries in the DT files.
> > > > >
> > > > >
> > > > > Phil Edworthy (3):
> > > > >   PCI: rcar-pcie: Add call to get domain nr
> > > > >   PCI: rcar-internal-pci: Add call to get domain nr
> > > > >   ARM: Enable PCI domains
> > > >
> > > > I'm deferring these for now because they depend on Liviu's work, which I
> > > > haven't merged yet, and I suspect some minor adaptation will be required
> > > > here.
> > > >
> > > > For what it's worth, I agree with Rob's hesitation about mixing lookup with
> > > > domain number allocation in of_pci_get_domain_nr().  That seems
> > > > unnecessarily complicated.
> > > I could create patches to add an optional "pci-domain" property for the R-Car
> > > PCI drivers, and just attempt to get the property in the drivers. If not found,
> > > the drivers will assume the domain is 0.
> > >
> > > We would then have fixed PCI domain numbering and I don't have to worry about
> > > Liviu's work.
> > 
> > I will split the current of_pci_get_domain_nr() even further and replace it with
> > two functions: pci_get_domain_nr() which will do just the allocation (still based
> > on the boolean flag passed as parameter) and of_get_pci_domain_nr() that will
> > retrieve a "linux,pci-domain" value from a property belonging to a given device
> > node.
> This doesn't solve the problem of different domain numbers based on different
> probe ordering. If you have multiple domains then I think you must have the
> "linux,pci-domain" property for each controller.

Correct, but I've said I'm going to leave for the moment the check to the host controller(s).
So if you care about ordering or mandating the presence of "linux,pci-domain" in the
DT, please add the check in your driver.

> 
> If we assume the above, of_get_pci_domain_nr() can just return a bus number of
> 0 if the "linux,pci-domain" property doesn't exist.

You mean *domain* number of 0. But that is still a valid value and you won't be able
to distinguish between DT providing a "linux,pci-domain" value of zero and the
property not being present.

> 
> I suppose you could have a temporary solution that allocates the domain numbers
> until the necessary drivers have been updated.

And that is what pci_get_domain_nr() will do. If you don't care about ordering, one
will use that one. If you care about ordering then "linux,pci-domain" is required
and one can use of_get_pci_domain_nr() to retrieve it and fail if the information
is missing.

Does that look like a workable solution?

Best regards,
Liviu

> 
> > I plan to leave for the moment the check for mandatory presence of
> > "linux,pci-domain"
> > to the host bridge driver so that everyone can implement their own policies.
> > 
> > How does that sound?
> 
> Thanks
> Phil

-- 
==========
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

RE: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

SGkgTGl2aXUsDQoNCk9uIDIzIFNlcHRlbWJlciAyMDE0IDEyOjEwLCBMaXZpdSB3cm90ZToNCj4g
T24gVHVlLCBTZXAgMjMsIDIwMTQgYXQgMTI6MDA6NDFQTSArMDEwMCwgUGhpbCBFZHdvcnRoeSB3
cm90ZToNCj4gPiBIaSBMaXZpdSwNCj4gPg0KPiA+IE9uIDIzIFNlcHRlbWJlciAyMDE0IDExOjMy
LCBMaXZpdSB3cm90ZToNCj4gPiA+IE9uIFR1ZSwgU2VwIDIzLCAyMDE0IGF0IDExOjEwOjI5QU0g
KzAxMDAsIFBoaWwgRWR3b3J0aHkgd3JvdGU6DQo+ID4gPiA+IEhpIEJqb3JuLA0KPiA+ID4gPg0K
PiA+ID4gPiBPbiAyMiBTZXB0ZW1iZXIgMjAxNCAyMjowMSwgQmpvcm4gd3JvdGU6DQo+ID4gPiA+
ID4gT24gTW9uLCBTZXAgMjIsIDIwMTQgYXQgMTA6NTE6MDdBTSArMDEwMCwgUGhpbCBFZHdvcnRo
eSB3cm90ZToNCj4gPiA+ID4gPiA+IFRoZSBSZW5lc2FzIFItQ2FyIGRldmljZXMgKHI4YTc3OTAg
YW5kIHI4YTc3OTEpIHVzZSB0d28gUENJIGNvbnRyb2xsZXINCj4gPiA+ID4gPiBkcml2ZXJzLA0K
PiA+ID4gPiA+ID4gb25lIGZvciBhbiBleHRlcm5hbCBQQ0llIHNsb3QsIHRoZSBvdGhlciBmb3Ig
YW4gaW50ZXJuYWwgUENJIGJyaWRnZSB0byBVU0INCj4gPiA+ID4gPiA+IGNvbnRyb2xsZXJzLg0K
PiA+ID4gPiA+ID4NCj4gPiA+ID4gPiA+IEhvd2V2ZXIsIHRoZXkgY3VycmVudGx5IGRvIG5vdCB3
b3JrIGF0IHRoZSBzYW1lIHRpbWUgYXMgdGhleSB1c2UgdGhlDQo+ID4gPiA+ID4gc2FtZSBQQ0kN
Cj4gPiA+ID4gPiA+IGRvbWFpbiBhbmQgdXNlIHRoZSBzYW1lIHJvb3QgYnVzIG51bWJlci4gV2Ug
Y2FuJ3QgdXNlIGRpZmZlcmVudCByb290DQo+ID4gPiA+ID4gYnVzIG51bWJlcnMNCj4gPiA+ID4g
PiA+IGR1ZSB0byB0aGUgd2F5IHJvb3QgYnVzIG51bWJlcnMgYXJlIGFzc2lnbmVkIGluIHBjaWJp
b3NfaW5pdF9odygpIGluDQo+ID4gPiA+ID4gPiBhcmNoL2FybS9rZXJuZWwvYmlvczMyLmMuDQo+
ID4gPiA+ID4gPg0KPiA+ID4gPiA+ID4gU2luY2UgdGhlIHR3byBQQ0kgY29udHJvbGxlcnMgYXJl
IGNvbXBsZXRlbHkgaW5kZXBlbmRlbnQsIEkgdGhpbmsgaXQgbWFrZXMNCj4gPiA+ID4gPiBzZW5z
ZQ0KPiA+ID4gPiA+ID4gdG8gdXNlIGRpZmZlcmVudCBQQ0kgZG9tYWlucyBmb3IgdGhlbS4NCj4g
PiA+ID4gPiA+DQo+ID4gPiA+ID4gPiBJJ3ZlIG1hcmtlZCB0aGUgdGhpcmQgcGF0Y2ggYXMgUkZD
IGFzIEkgYW0gbm90IHN1cmUgb2YgdGhlIGltcGFjdCBvZg0KPiA+ID4gPiA+IGVuYWJsaW5nDQo+
ID4gPiA+ID4gPiBQQ0kgZG9tYWlucyBmb3IgYWxsIEFSTSBkZXZpY2VzLiBJbiB0aGUgbWFyY2gg
dG8gJ29uZSBrZXJuZWwgdG8gcnVsZSB0aGVtDQo+ID4gPiA+ID4gYWxsJywNCj4gPiA+ID4gPiA+
IEkgc3RlZXJlZCBjbGVhciBvZiBtYWNoIHNwZWNpZmljIGNoYW5nZXMuDQo+ID4gPiA+ID4gPg0K
PiA+ID4gPiA+ID4gVGhlc2UgcGF0Y2hlcyByZXF1aXJlIHRoZSBmb2xsb3dpbmcgcGF0Y2ggZnJv
bSBMaXZpdSBEdWRhdToNCj4gPiA+ID4gPiA+ICAgW1BBVENIIHYxMSAwNy8xMF0gT0Y6IEludHJv
ZHVjZSBoZWxwZXIgZnVuY3Rpb24gZm9yIGdldHRpbmcgUENJDQo+ID4gPiA+ID4gZG9tYWluX25y
DQo+ID4gPiA+ID4gPiBCYXNlZCBvbiBjb21tZW50cyBvbiB0aGlzIHBhdGNoIGZyb20gSmFzb24g
R3VudGhvcnBlLCB0aGVyZSBpcyBzdGlsbCB0aGUNCj4gPiA+ID4gPiBpc3N1ZQ0KPiA+ID4gPiA+
ID4gdGhhdCB0aGUgZG9tYWluIG51bWJlcnMgbWF5IGNoYW5nZSBkZXBlbmRpbmcgb24gdGhlIG9y
ZGVyaW5nIGF0IHByb2JlDQo+ID4gPiA+ID4gdGltZS4NCj4gPiA+ID4gPiA+IEhvd2V2ZXIsIHRo
aXMgY2FuIGJlIGZpeGVkIGxhdGVyIG9uIGJ5IGFkZGluZyB0aGUgZW50cmllcyBpbiB0aGUgRFQg
ZmlsZXMuDQo+ID4gPiA+ID4gPg0KPiA+ID4gPiA+ID4NCj4gPiA+ID4gPiA+IFBoaWwgRWR3b3J0
aHkgKDMpOg0KPiA+ID4gPiA+ID4gICBQQ0k6IHJjYXItcGNpZTogQWRkIGNhbGwgdG8gZ2V0IGRv
bWFpbiBucg0KPiA+ID4gPiA+ID4gICBQQ0k6IHJjYXItaW50ZXJuYWwtcGNpOiBBZGQgY2FsbCB0
byBnZXQgZG9tYWluIG5yDQo+ID4gPiA+ID4gPiAgIEFSTTogRW5hYmxlIFBDSSBkb21haW5zDQo+
ID4gPiA+ID4NCj4gPiA+ID4gPiBJJ20gZGVmZXJyaW5nIHRoZXNlIGZvciBub3cgYmVjYXVzZSB0
aGV5IGRlcGVuZCBvbiBMaXZpdSdzIHdvcmssIHdoaWNoIEkNCj4gPiA+ID4gPiBoYXZlbid0IG1l
cmdlZCB5ZXQsIGFuZCBJIHN1c3BlY3Qgc29tZSBtaW5vciBhZGFwdGF0aW9uIHdpbGwgYmUgcmVx
dWlyZWQNCj4gPiA+ID4gPiBoZXJlLg0KPiA+ID4gPiA+DQo+ID4gPiA+ID4gRm9yIHdoYXQgaXQn
cyB3b3J0aCwgSSBhZ3JlZSB3aXRoIFJvYidzIGhlc2l0YXRpb24gYWJvdXQgbWl4aW5nIGxvb2t1
cCB3aXRoDQo+ID4gPiA+ID4gZG9tYWluIG51bWJlciBhbGxvY2F0aW9uIGluIG9mX3BjaV9nZXRf
ZG9tYWluX25yKCkuICBUaGF0IHNlZW1zDQo+ID4gPiA+ID4gdW5uZWNlc3NhcmlseSBjb21wbGlj
YXRlZC4NCj4gPiA+ID4gSSBjb3VsZCBjcmVhdGUgcGF0Y2hlcyB0byBhZGQgYW4gb3B0aW9uYWwg
InBjaS1kb21haW4iIHByb3BlcnR5IGZvciB0aGUgUi1DYXINCj4gPiA+ID4gUENJIGRyaXZlcnMs
IGFuZCBqdXN0IGF0dGVtcHQgdG8gZ2V0IHRoZSBwcm9wZXJ0eSBpbiB0aGUgZHJpdmVycy4gSWYg
bm90IGZvdW5kLA0KPiA+ID4gPiB0aGUgZHJpdmVycyB3aWxsIGFzc3VtZSB0aGUgZG9tYWluIGlz
IDAuDQo+ID4gPiA+DQo+ID4gPiA+IFdlIHdvdWxkIHRoZW4gaGF2ZSBmaXhlZCBQQ0kgZG9tYWlu
IG51bWJlcmluZyBhbmQgSSBkb24ndCBoYXZlIHRvIHdvcnJ5IGFib3V0DQo+ID4gPiA+IExpdml1
J3Mgd29yay4NCj4gPiA+DQo+ID4gPiBJIHdpbGwgc3BsaXQgdGhlIGN1cnJlbnQgb2ZfcGNpX2dl
dF9kb21haW5fbnIoKSBldmVuIGZ1cnRoZXIgYW5kIHJlcGxhY2UgaXQgd2l0aA0KPiA+ID4gdHdv
IGZ1bmN0aW9uczogcGNpX2dldF9kb21haW5fbnIoKSB3aGljaCB3aWxsIGRvIGp1c3QgdGhlIGFs
bG9jYXRpb24gKHN0aWxsIGJhc2VkDQo+ID4gPiBvbiB0aGUgYm9vbGVhbiBmbGFnIHBhc3NlZCBh
cyBwYXJhbWV0ZXIpIGFuZCBvZl9nZXRfcGNpX2RvbWFpbl9ucigpIHRoYXQgd2lsbA0KPiA+ID4g
cmV0cmlldmUgYSAibGludXgscGNpLWRvbWFpbiIgdmFsdWUgZnJvbSBhIHByb3BlcnR5IGJlbG9u
Z2luZyB0byBhIGdpdmVuIGRldmljZQ0KPiA+ID4gbm9kZS4NCj4gPiBUaGlzIGRvZXNuJ3Qgc29s
dmUgdGhlIHByb2JsZW0gb2YgZGlmZmVyZW50IGRvbWFpbiBudW1iZXJzIGJhc2VkIG9uIGRpZmZl
cmVudA0KPiA+IHByb2JlIG9yZGVyaW5nLiBJZiB5b3UgaGF2ZSBtdWx0aXBsZSBkb21haW5zIHRo
ZW4gSSB0aGluayB5b3UgbXVzdCBoYXZlIHRoZQ0KPiA+ICJsaW51eCxwY2ktZG9tYWluIiBwcm9w
ZXJ0eSBmb3IgZWFjaCBjb250cm9sbGVyLg0KPiANCj4gQ29ycmVjdCwgYnV0IEkndmUgc2FpZCBJ
J20gZ29pbmcgdG8gbGVhdmUgZm9yIHRoZSBtb21lbnQgdGhlIGNoZWNrIHRvIHRoZSBob3N0DQo+
IGNvbnRyb2xsZXIocykuDQo+IFNvIGlmIHlvdSBjYXJlIGFib3V0IG9yZGVyaW5nIG9yIG1hbmRh
dGluZyB0aGUgcHJlc2VuY2Ugb2YgImxpbnV4LHBjaS0NCj4gZG9tYWluIiBpbiB0aGUgRFQsIHBs
ZWFzZSBhZGQgdGhlIGNoZWNrIGluIHlvdXIgZHJpdmVyLg0KPg0KPiA+DQo+ID4gSWYgd2UgYXNz
dW1lIHRoZSBhYm92ZSwgb2ZfZ2V0X3BjaV9kb21haW5fbnIoKSBjYW4ganVzdCByZXR1cm4gYSBi
dXMgbnVtYmVyIG9mDQo+ID4gMCBpZiB0aGUgImxpbnV4LHBjaS1kb21haW4iIHByb3BlcnR5IGRv
ZXNuJ3QgZXhpc3QuDQo+IA0KPiBZb3UgbWVhbiAqZG9tYWluKiBudW1iZXIgb2YgMC4gQnV0IHRo
YXQgaXMgc3RpbGwgYSB2YWxpZCB2YWx1ZSBhbmQgeW91IHdvbid0DQo+IGJlIGFibGUgdG8gZGlz
dGluZ3Vpc2ggYmV0d2VlbiBEVCBwcm92aWRpbmcgYSAibGludXgscGNpLWRvbWFpbiIgdmFsdWUg
b2YgemVybyBhbmQNCj4gdGhlIHByb3BlcnR5IG5vdCBiZWluZyBwcmVzZW50Lg0KWWVzLCBJIG1l
YW50IGRvbWFpbi4NCklmIGEgZGV2aWNlIG9ubHkgaGFzIGEgc2luZ2xlIFBDSSBkb21haW4sIG5v
ICJsaW51eCxwY2ktZG9tYWluIiBwcm9wIGlzIG5lZWRlZA0KYW5kIGEgZGVmYXVsdCBkb21haW4g
bnVtYmVyIG9mIDAgaXMgb2suDQpJZiB0aGVyZSBpcyBtb3JlIHRoYW4gb25lIGRvbWFpbiwgdXNl
cnNwYWNlIG5lZWRzIHRvIGhhdmUgY29uc2lzdGVudCBkb21haW4NCm51bWJlcmluZyBvdmVyIGtl
cm5lbCB2ZXJzaW9ucywgc28gd2UgaGF2ZSB0byBzcGVjaWZ5IHRoZSBkb21haW4gaW4gdGhlIERU
Lg0KDQpJIGFtIHByb2JhYmx5IG1pc3Npbmcgc29tZXRoaW5nLCBpdCB3b3VsZG7igJl0IGJlIHRo
ZSBmaXJzdCB0aW1lLCBidXQgSSBkb27igJl0IHNlZQ0Kd2h5IG9mX2dldF9wY2lfZG9tYWluX25y
KCkgbmVlZHMgdG8gYWxsb2NhdGUgZG9tYWluIG51bWJlcnMuDQogDQo+ID4NCj4gPiBJIHN1cHBv
c2UgeW91IGNvdWxkIGhhdmUgYSB0ZW1wb3Jhcnkgc29sdXRpb24gdGhhdCBhbGxvY2F0ZXMgdGhl
IGRvbWFpbiBudW1iZXJzDQo+ID4gdW50aWwgdGhlIG5lY2Vzc2FyeSBkcml2ZXJzIGhhdmUgYmVl
biB1cGRhdGVkLg0KPiANCj4gQW5kIHRoYXQgaXMgd2hhdCBwY2lfZ2V0X2RvbWFpbl9ucigpIHdp
bGwgZG8uIElmIHlvdSBkb24ndCBjYXJlIGFib3V0IG9yZGVyaW5nLCBvbmUNCj4gd2lsbCB1c2Ug
dGhhdCBvbmUuIElmIHlvdSBjYXJlIGFib3V0IG9yZGVyaW5nIHRoZW4gImxpbnV4LHBjaS1kb21h
aW4iIGlzIHJlcXVpcmVkDQo+IGFuZCBvbmUgY2FuIHVzZSBvZl9nZXRfcGNpX2RvbWFpbl9ucigp
IHRvIHJldHJpZXZlIGl0IGFuZCBmYWlsIGlmIHRoZSBpbmZvcm1hdGlvbg0KPiBpcyBtaXNzaW5n
Lg0KPiANCj4gRG9lcyB0aGF0IGxvb2sgbGlrZSBhIHdvcmthYmxlIHNvbHV0aW9uPw0KPiANCj4g
QmVzdCByZWdhcmRzLA0KPiBMaXZpdQ0KPiANCj4gPg0KPiA+ID4gSSBwbGFuIHRvIGxlYXZlIGZv
ciB0aGUgbW9tZW50IHRoZSBjaGVjayBmb3IgbWFuZGF0b3J5IHByZXNlbmNlIG9mDQo+ID4gPiAi
bGludXgscGNpLWRvbWFpbiINCj4gPiA+IHRvIHRoZSBob3N0IGJyaWRnZSBkcml2ZXIgc28gdGhh
dCBldmVyeW9uZSBjYW4gaW1wbGVtZW50IHRoZWlyIG93bg0KPiBwb2xpY2llcy4NCj4gPiA+DQo+
ID4gPiBIb3cgZG9lcyB0aGF0IHNvdW5kPw0KPiA+DQoNClRoYW5rcw0KUGhpbA0K

Re: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Liviu Dudau]

On Tue, Sep 23, 2014 at 12:38:28PM +0100, Phil Edworthy wrote:
> Hi Liviu,
> 
> On 23 September 2014 12:10, Liviu wrote:
> > On Tue, Sep 23, 2014 at 12:00:41PM +0100, Phil Edworthy wrote:
> > > Hi Liviu,
> > >
> > > On 23 September 2014 11:32, Liviu wrote:
> > > > On Tue, Sep 23, 2014 at 11:10:29AM +0100, Phil Edworthy wrote:
> > > > > Hi Bjorn,
> > > > >
> > > > > On 22 September 2014 22:01, Bjorn wrote:
> > > > > > On Mon, Sep 22, 2014 at 10:51:07AM +0100, Phil Edworthy wrote:
> > > > > > > The Renesas R-Car devices (r8a7790 and r8a7791) use two PCI controller
> > > > > > drivers,
> > > > > > > one for an external PCIe slot, the other for an internal PCI bridge to USB
> > > > > > > controllers.
> > > > > > >
> > > > > > > However, they currently do not work at the same time as they use the
> > > > > > same PCI
> > > > > > > domain and use the same root bus number. We can't use different root
> > > > > > bus numbers
> > > > > > > due to the way root bus numbers are assigned in pcibios_init_hw() in
> > > > > > > arch/arm/kernel/bios32.c.
> > > > > > >
> > > > > > > Since the two PCI controllers are completely independent, I think it makes
> > > > > > sense
> > > > > > > to use different PCI domains for them.
> > > > > > >
> > > > > > > I've marked the third patch as RFC as I am not sure of the impact of
> > > > > > enabling
> > > > > > > PCI domains for all ARM devices. In the march to 'one kernel to rule them
> > > > > > all',
> > > > > > > I steered clear of mach specific changes.
> > > > > > >
> > > > > > > These patches require the following patch from Liviu Dudau:
> > > > > > >   [PATCH v11 07/10] OF: Introduce helper function for getting PCI
> > > > > > domain_nr
> > > > > > > Based on comments on this patch from Jason Gunthorpe, there is still the
> > > > > > issue
> > > > > > > that the domain numbers may change depending on the ordering at probe
> > > > > > time.
> > > > > > > However, this can be fixed later on by adding the entries in the DT files.
> > > > > > >
> > > > > > >
> > > > > > > Phil Edworthy (3):
> > > > > > >   PCI: rcar-pcie: Add call to get domain nr
> > > > > > >   PCI: rcar-internal-pci: Add call to get domain nr
> > > > > > >   ARM: Enable PCI domains
> > > > > >
> > > > > > I'm deferring these for now because they depend on Liviu's work, which I
> > > > > > haven't merged yet, and I suspect some minor adaptation will be required
> > > > > > here.
> > > > > >
> > > > > > For what it's worth, I agree with Rob's hesitation about mixing lookup with
> > > > > > domain number allocation in of_pci_get_domain_nr().  That seems
> > > > > > unnecessarily complicated.
> > > > > I could create patches to add an optional "pci-domain" property for the R-Car
> > > > > PCI drivers, and just attempt to get the property in the drivers. If not found,
> > > > > the drivers will assume the domain is 0.
> > > > >
> > > > > We would then have fixed PCI domain numbering and I don't have to worry about
> > > > > Liviu's work.
> > > >
> > > > I will split the current of_pci_get_domain_nr() even further and replace it with
> > > > two functions: pci_get_domain_nr() which will do just the allocation (still based
> > > > on the boolean flag passed as parameter) and of_get_pci_domain_nr() that will
> > > > retrieve a "linux,pci-domain" value from a property belonging to a given device
> > > > node.
> > > This doesn't solve the problem of different domain numbers based on different
> > > probe ordering. If you have multiple domains then I think you must have the
> > > "linux,pci-domain" property for each controller.
> > 
> > Correct, but I've said I'm going to leave for the moment the check to the host
> > controller(s).
> > So if you care about ordering or mandating the presence of "linux,pci-
> > domain" in the DT, please add the check in your driver.
> >
> > >
> > > If we assume the above, of_get_pci_domain_nr() can just return a bus number of
> > > 0 if the "linux,pci-domain" property doesn't exist.
> > 
> > You mean *domain* number of 0. But that is still a valid value and you won't
> > be able to distinguish between DT providing a "linux,pci-domain" value of zero and
> > the property not being present.
> Yes, I meant domain.
> If a device only has a single PCI domain, no "linux,pci-domain" prop is needed
> and a default domain number of 0 is ok.

Agree, except it is not a "device" choice but a platform choice. In other words,
having a host bridge capable of running on multi-host-bridge setups, then
your HB driver will have to mandate "linux,pci-domain" presence. But the decision
to have more than one HB will be present in the device tree, not in the driver.

> If there is more than one domain, userspace needs to have consistent domain
> numbering over kernel versions, so we have to specify the domain in the DT.

Agree, and the DT will pin down the domain numbers.

> 
> I am probably missing something, it wouldn't be the first time, but I don't see
> why of_get_pci_domain_nr() needs to allocate domain numbers.

Maybe because I have confused you. Let's try again:
  - of_pci_get_domain_nr(): function present in v11, parses "pci-domain" alias,
    allocates a new domain number if "pci-domain" info is missing. Will be
    removed in v12.
  - of_get_pci_domain_nr(): subtle change of name to flag the change of behaviour
    from the previous function. Only parses "linux,pci-domain" property of the
    host bridge node, returns the property value as an integer between 0 and 255,
    or a negative value if the property is missing.
  - pci_get_new_domain_nr(): this function is only present in v12 and allocates
    a new domain number each time it is called.

Hope this helps,

Liviu

>  
> > >
> > > I suppose you could have a temporary solution that allocates the domain numbers
> > > until the necessary drivers have been updated.
> > 
> > And that is what pci_get_domain_nr() will do. If you don't care about ordering, one
> > will use that one. If you care about ordering then "linux,pci-domain" is required
> > and one can use of_get_pci_domain_nr() to retrieve it and fail if the information
> > is missing.
> > 
> > Does that look like a workable solution?
> > 
> > Best regards,
> > Liviu
> > 
> > >
> > > > I plan to leave for the moment the check for mandatory presence of
> > > > "linux,pci-domain"
> > > > to the host bridge driver so that everyone can implement their own
> > policies.
> > > >
> > > > How does that sound?
> > >
> 
> Thanks
> Phil

-- 
==========
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

RE: [PATCH 0/3] Add PCI domain support to R-Car drivers

[Phil Edworthy]

SGkgTGl2aXUsDQoNCk9uIDIzIFNlcHRlbWJlciAyMDE0IDEzOjExLCBMaXZpdSB3cm90ZToNCj4g
T24gVHVlLCBTZXAgMjMsIDIwMTQgYXQgMTI6Mzg6MjhQTSArMDEwMCwgUGhpbCBFZHdvcnRoeSB3
cm90ZToNCj4gPiBPbiAyMyBTZXB0ZW1iZXIgMjAxNCAxMjoxMCwgTGl2aXUgd3JvdGU6DQo+ID4g
PiBPbiBUdWUsIFNlcCAyMywgMjAxNCBhdCAxMjowMDo0MVBNICswMTAwLCBQaGlsIEVkd29ydGh5
IHdyb3RlOg0KPiA+ID4gPiBPbiAyMyBTZXB0ZW1iZXIgMjAxNCAxMTozMiwgTGl2aXUgd3JvdGU6
DQo+ID4gPiA+ID4gT24gVHVlLCBTZXAgMjMsIDIwMTQgYXQgMTE6MTA6MjlBTSArMDEwMCwgUGhp
bCBFZHdvcnRoeSB3cm90ZToNCj4gPiA+ID4gPiA+IEhpIEJqb3JuLA0KPiA+ID4gPiA+ID4NCj4g
PiA+ID4gPiA+IE9uIDIyIFNlcHRlbWJlciAyMDE0IDIyOjAxLCBCam9ybiB3cm90ZToNCj4gPiA+
ID4gPiA+ID4gT24gTW9uLCBTZXAgMjIsIDIwMTQgYXQgMTA6NTE6MDdBTSArMDEwMCwgUGhpbCBF
ZHdvcnRoeSB3cm90ZToNCj4gPiA+ID4gPiA+ID4gPiBUaGUgUmVuZXNhcyBSLUNhciBkZXZpY2Vz
IChyOGE3NzkwIGFuZCByOGE3NzkxKSB1c2UgdHdvIFBDSSBjb250cm9sbGVyDQo+ID4gPiA+ID4g
PiA+IGRyaXZlcnMsDQo+ID4gPiA+ID4gPiA+ID4gb25lIGZvciBhbiBleHRlcm5hbCBQQ0llIHNs
b3QsIHRoZSBvdGhlciBmb3IgYW4gaW50ZXJuYWwgUENJIGJyaWRnZSB0byBVU0INCj4gPiA+ID4g
PiA+ID4gPiBjb250cm9sbGVycy4NCj4gPiA+ID4gPiA+ID4gPg0KPiA+ID4gPiA+ID4gPiA+IEhv
d2V2ZXIsIHRoZXkgY3VycmVudGx5IGRvIG5vdCB3b3JrIGF0IHRoZSBzYW1lIHRpbWUgYXMgdGhl
eSB1c2UgdGhlDQo+ID4gPiA+ID4gPiA+IHNhbWUgUENJDQo+ID4gPiA+ID4gPiA+ID4gZG9tYWlu
IGFuZCB1c2UgdGhlIHNhbWUgcm9vdCBidXMgbnVtYmVyLiBXZSBjYW4ndCB1c2UgZGlmZmVyZW50
IHJvb3QNCj4gPiA+ID4gPiA+ID4gYnVzIG51bWJlcnMNCj4gPiA+ID4gPiA+ID4gPiBkdWUgdG8g
dGhlIHdheSByb290IGJ1cyBudW1iZXJzIGFyZSBhc3NpZ25lZCBpbiBwY2liaW9zX2luaXRfaHco
KSBpbg0KPiA+ID4gPiA+ID4gPiA+IGFyY2gvYXJtL2tlcm5lbC9iaW9zMzIuYy4NCj4gPiA+ID4g
PiA+ID4gPg0KPiA+ID4gPiA+ID4gPiA+IFNpbmNlIHRoZSB0d28gUENJIGNvbnRyb2xsZXJzIGFy
ZSBjb21wbGV0ZWx5IGluZGVwZW5kZW50LCBJIHRoaW5rIGl0IG1ha2VzDQo+ID4gPiA+ID4gPiA+
IHNlbnNlDQo+ID4gPiA+ID4gPiA+ID4gdG8gdXNlIGRpZmZlcmVudCBQQ0kgZG9tYWlucyBmb3Ig
dGhlbS4NCj4gPiA+ID4gPiA+ID4gPg0KPiA+ID4gPiA+ID4gPiA+IEkndmUgbWFya2VkIHRoZSB0
aGlyZCBwYXRjaCBhcyBSRkMgYXMgSSBhbSBub3Qgc3VyZSBvZiB0aGUgaW1wYWN0IG9mDQo+ID4g
PiA+ID4gPiA+IGVuYWJsaW5nDQo+ID4gPiA+ID4gPiA+ID4gUENJIGRvbWFpbnMgZm9yIGFsbCBB
Uk0gZGV2aWNlcy4gSW4gdGhlIG1hcmNoIHRvICdvbmUga2VybmVsIHRvIHJ1bGUgdGhlbQ0KPiA+
ID4gPiA+ID4gPiBhbGwnLA0KPiA+ID4gPiA+ID4gPiA+IEkgc3RlZXJlZCBjbGVhciBvZiBtYWNo
IHNwZWNpZmljIGNoYW5nZXMuDQo+ID4gPiA+ID4gPiA+ID4NCj4gPiA+ID4gPiA+ID4gPiBUaGVz
ZSBwYXRjaGVzIHJlcXVpcmUgdGhlIGZvbGxvd2luZyBwYXRjaCBmcm9tIExpdml1IER1ZGF1Og0K
PiA+ID4gPiA+ID4gPiA+ICAgW1BBVENIIHYxMSAwNy8xMF0gT0Y6IEludHJvZHVjZSBoZWxwZXIg
ZnVuY3Rpb24gZm9yIGdldHRpbmcgUENJDQo+ID4gPiA+ID4gPiA+IGRvbWFpbl9ucg0KPiA+ID4g
PiA+ID4gPiA+IEJhc2VkIG9uIGNvbW1lbnRzIG9uIHRoaXMgcGF0Y2ggZnJvbSBKYXNvbiBHdW50
aG9ycGUsIHRoZXJlIGlzIHN0aWxsIHRoZQ0KPiA+ID4gPiA+ID4gPiBpc3N1ZQ0KPiA+ID4gPiA+
ID4gPiA+IHRoYXQgdGhlIGRvbWFpbiBudW1iZXJzIG1heSBjaGFuZ2UgZGVwZW5kaW5nIG9uIHRo
ZSBvcmRlcmluZyBhdCBwcm9iZQ0KPiA+ID4gPiA+ID4gPiB0aW1lLg0KPiA+ID4gPiA+ID4gPiA+
IEhvd2V2ZXIsIHRoaXMgY2FuIGJlIGZpeGVkIGxhdGVyIG9uIGJ5IGFkZGluZyB0aGUgZW50cmll
cyBpbiB0aGUgRFQgZmlsZXMuDQo+ID4gPiA+ID4gPiA+ID4NCj4gPiA+ID4gPiA+ID4gPg0KPiA+
ID4gPiA+ID4gPiA+IFBoaWwgRWR3b3J0aHkgKDMpOg0KPiA+ID4gPiA+ID4gPiA+ICAgUENJOiBy
Y2FyLXBjaWU6IEFkZCBjYWxsIHRvIGdldCBkb21haW4gbnINCj4gPiA+ID4gPiA+ID4gPiAgIFBD
STogcmNhci1pbnRlcm5hbC1wY2k6IEFkZCBjYWxsIHRvIGdldCBkb21haW4gbnINCj4gPiA+ID4g
PiA+ID4gPiAgIEFSTTogRW5hYmxlIFBDSSBkb21haW5zDQo+ID4gPiA+ID4gPiA+DQo+ID4gPiA+
ID4gPiA+IEknbSBkZWZlcnJpbmcgdGhlc2UgZm9yIG5vdyBiZWNhdXNlIHRoZXkgZGVwZW5kIG9u
IExpdml1J3Mgd29yaywgd2hpY2ggSQ0KPiA+ID4gPiA+ID4gPiBoYXZlbid0IG1lcmdlZCB5ZXQs
IGFuZCBJIHN1c3BlY3Qgc29tZSBtaW5vciBhZGFwdGF0aW9uIHdpbGwgYmUgcmVxdWlyZWQNCj4g
PiA+ID4gPiA+ID4gaGVyZS4NCj4gPiA+ID4gPiA+ID4NCj4gPiA+ID4gPiA+ID4gRm9yIHdoYXQg
aXQncyB3b3J0aCwgSSBhZ3JlZSB3aXRoIFJvYidzIGhlc2l0YXRpb24gYWJvdXQgbWl4aW5nIGxv
b2t1cCB3aXRoDQo+ID4gPiA+ID4gPiA+IGRvbWFpbiBudW1iZXIgYWxsb2NhdGlvbiBpbiBvZl9w
Y2lfZ2V0X2RvbWFpbl9ucigpLiAgVGhhdCBzZWVtcw0KPiA+ID4gPiA+ID4gPiB1bm5lY2Vzc2Fy
aWx5IGNvbXBsaWNhdGVkLg0KPiA+ID4gPiA+ID4gSSBjb3VsZCBjcmVhdGUgcGF0Y2hlcyB0byBh
ZGQgYW4gb3B0aW9uYWwgInBjaS1kb21haW4iIHByb3BlcnR5IGZvciB0aGUgUi1DYXINCj4gPiA+
ID4gPiA+IFBDSSBkcml2ZXJzLCBhbmQganVzdCBhdHRlbXB0IHRvIGdldCB0aGUgcHJvcGVydHkg
aW4gdGhlIGRyaXZlcnMuIElmIG5vdCBmb3VuZCwNCj4gPiA+ID4gPiA+IHRoZSBkcml2ZXJzIHdp
bGwgYXNzdW1lIHRoZSBkb21haW4gaXMgMC4NCj4gPiA+ID4gPiA+DQo+ID4gPiA+ID4gPiBXZSB3
b3VsZCB0aGVuIGhhdmUgZml4ZWQgUENJIGRvbWFpbiBudW1iZXJpbmcgYW5kIEkgZG9uJ3QgaGF2
ZSB0byB3b3JyeSBhYm91dA0KPiA+ID4gPiA+ID4gTGl2aXUncyB3b3JrLg0KPiA+ID4gPiA+DQo+
ID4gPiA+ID4gSSB3aWxsIHNwbGl0IHRoZSBjdXJyZW50IG9mX3BjaV9nZXRfZG9tYWluX25yKCkg
ZXZlbiBmdXJ0aGVyIGFuZCByZXBsYWNlIGl0IHdpdGgNCj4gPiA+ID4gPiB0d28gZnVuY3Rpb25z
OiBwY2lfZ2V0X2RvbWFpbl9ucigpIHdoaWNoIHdpbGwgZG8ganVzdCB0aGUgYWxsb2NhdGlvbiAo
c3RpbGwgYmFzZWQNCj4gPiA+ID4gPiBvbiB0aGUgYm9vbGVhbiBmbGFnIHBhc3NlZCBhcyBwYXJh
bWV0ZXIpIGFuZCBvZl9nZXRfcGNpX2RvbWFpbl9ucigpIHRoYXQgd2lsbA0KPiA+ID4gPiA+IHJl
dHJpZXZlIGEgImxpbnV4LHBjaS1kb21haW4iIHZhbHVlIGZyb20gYSBwcm9wZXJ0eSBiZWxvbmdp
bmcgdG8gYSBnaXZlbiBkZXZpY2UNCj4gPiA+ID4gPiBub2RlLg0KPiA+ID4gPiBUaGlzIGRvZXNu
J3Qgc29sdmUgdGhlIHByb2JsZW0gb2YgZGlmZmVyZW50IGRvbWFpbiBudW1iZXJzIGJhc2VkIG9u
IGRpZmZlcmVudA0KPiA+ID4gPiBwcm9iZSBvcmRlcmluZy4gSWYgeW91IGhhdmUgbXVsdGlwbGUg
ZG9tYWlucyB0aGVuIEkgdGhpbmsgeW91IG11c3QgaGF2ZSB0aGUNCj4gPiA+ID4gImxpbnV4LHBj
aS1kb21haW4iIHByb3BlcnR5IGZvciBlYWNoIGNvbnRyb2xsZXIuDQo+ID4gPg0KPiA+ID4gQ29y
cmVjdCwgYnV0IEkndmUgc2FpZCBJJ20gZ29pbmcgdG8gbGVhdmUgZm9yIHRoZSBtb21lbnQgdGhl
IGNoZWNrIHRvIHRoZSBob3N0DQo+ID4gPiBjb250cm9sbGVyKHMpLg0KPiA+ID4gU28gaWYgeW91
IGNhcmUgYWJvdXQgb3JkZXJpbmcgb3IgbWFuZGF0aW5nIHRoZSBwcmVzZW5jZSBvZiAibGludXgs
cGNpLQ0KPiA+ID4gZG9tYWluIiBpbiB0aGUgRFQsIHBsZWFzZSBhZGQgdGhlIGNoZWNrIGluIHlv
dXIgZHJpdmVyLg0KPiA+ID4NCj4gPiA+ID4NCj4gPiA+ID4gSWYgd2UgYXNzdW1lIHRoZSBhYm92
ZSwgb2ZfZ2V0X3BjaV9kb21haW5fbnIoKSBjYW4ganVzdCByZXR1cm4gYSBidXMgbnVtYmVyIG9m
DQo+ID4gPiA+IDAgaWYgdGhlICJsaW51eCxwY2ktZG9tYWluIiBwcm9wZXJ0eSBkb2Vzbid0IGV4
aXN0Lg0KPiA+ID4NCj4gPiA+IFlvdSBtZWFuICpkb21haW4qIG51bWJlciBvZiAwLiBCdXQgdGhh
dCBpcyBzdGlsbCBhIHZhbGlkIHZhbHVlIGFuZCB5b3Ugd29uJ3QNCj4gPiA+IGJlIGFibGUgdG8g
ZGlzdGluZ3Vpc2ggYmV0d2VlbiBEVCBwcm92aWRpbmcgYSAibGludXgscGNpLWRvbWFpbiIgdmFs
dWUgb2YgemVybyBhbmQNCj4gPiA+IHRoZSBwcm9wZXJ0eSBub3QgYmVpbmcgcHJlc2VudC4NCj4g
PiBZZXMsIEkgbWVhbnQgZG9tYWluLg0KPiA+IElmIGEgZGV2aWNlIG9ubHkgaGFzIGEgc2luZ2xl
IFBDSSBkb21haW4sIG5vICJsaW51eCxwY2ktZG9tYWluIiBwcm9wIGlzIG5lZWRlZA0KPiA+IGFu
ZCBhIGRlZmF1bHQgZG9tYWluIG51bWJlciBvZiAwIGlzIG9rLg0KPiANCj4gQWdyZWUsIGV4Y2Vw
dCBpdCBpcyBub3QgYSAiZGV2aWNlIiBjaG9pY2UgYnV0IGEgcGxhdGZvcm0gY2hvaWNlLiBJbiBv
dGhlciB3b3JkcywNCj4gaGF2aW5nIGEgaG9zdCBicmlkZ2UgY2FwYWJsZSBvZiBydW5uaW5nIG9u
IG11bHRpLWhvc3QtYnJpZGdlIHNldHVwcywgdGhlbg0KPiB5b3VyIEhCIGRyaXZlciB3aWxsIGhh
dmUgdG8gbWFuZGF0ZSAibGludXgscGNpLWRvbWFpbiIgcHJlc2VuY2UuIEJ1dCB0aGUNCj4gZGVj
aXNpb24NCj4gdG8gaGF2ZSBtb3JlIHRoYW4gb25lIEhCIHdpbGwgYmUgcHJlc2VudCBpbiB0aGUg
ZGV2aWNlIHRyZWUsIG5vdCBpbiB0aGUNCj4gZHJpdmVyLg0KT2ssIEkgYWN0dWFsbHkgbWVhbnQg
dGhlIGRldmljZSBkdHNpLCBidXQgeWVzIHRoZSBkZWNpc2lvbiBsaWVzIHdpdGggdGhlIERULg0K
DQoNCj4gPiBJZiB0aGVyZSBpcyBtb3JlIHRoYW4gb25lIGRvbWFpbiwgdXNlcnNwYWNlIG5lZWRz
IHRvIGhhdmUgY29uc2lzdGVudCBkb21haW4NCj4gPiBudW1iZXJpbmcgb3ZlciBrZXJuZWwgdmVy
c2lvbnMsIHNvIHdlIGhhdmUgdG8gc3BlY2lmeSB0aGUgZG9tYWluIGluIHRoZSBEVC4NCj4gDQo+
IEFncmVlLCBhbmQgdGhlIERUIHdpbGwgcGluIGRvd24gdGhlIGRvbWFpbiBudW1iZXJzLg0KPiAN
Cj4gPg0KPiA+IEkgYW0gcHJvYmFibHkgbWlzc2luZyBzb21ldGhpbmcsIGl0IHdvdWxkbid0IGJl
IHRoZSBmaXJzdCB0aW1lLCBidXQgSSBkb24ndCBzZWUNCj4gPiB3aHkgb2ZfZ2V0X3BjaV9kb21h
aW5fbnIoKSBuZWVkcyB0byBhbGxvY2F0ZSBkb21haW4gbnVtYmVycy4NCj4gDQo+IE1heWJlIGJl
Y2F1c2UgSSBoYXZlIGNvbmZ1c2VkIHlvdS4gTGV0J3MgdHJ5IGFnYWluOg0KPiAgIC0gb2ZfcGNp
X2dldF9kb21haW5fbnIoKTogZnVuY3Rpb24gcHJlc2VudCBpbiB2MTEsIHBhcnNlcyAicGNpLWRv
bWFpbiIgYWxpYXMsDQo+ICAgICBhbGxvY2F0ZXMgYSBuZXcgZG9tYWluIG51bWJlciBpZiAicGNp
LWRvbWFpbiIgaW5mbyBpcyBtaXNzaW5nLiBXaWxsIGJlDQo+ICAgICByZW1vdmVkIGluIHYxMi4N
Cj4gICAtIG9mX2dldF9wY2lfZG9tYWluX25yKCk6IHN1YnRsZSBjaGFuZ2Ugb2YgbmFtZSB0byBm
bGFnIHRoZSBjaGFuZ2Ugb2YgYmVoYXZpb3VyDQo+ICAgICBmcm9tIHRoZSBwcmV2aW91cyBmdW5j
dGlvbi4gT25seSBwYXJzZXMgImxpbnV4LHBjaS1kb21haW4iIHByb3BlcnR5IG9mIHRoZQ0KPiAg
ICAgaG9zdCBicmlkZ2Ugbm9kZSwgcmV0dXJucyB0aGUgcHJvcGVydHkgdmFsdWUgYXMgYW4gaW50
ZWdlciBiZXR3ZWVuIDAgYW5kIDI1NSwNCj4gICAgIG9yIGEgbmVnYXRpdmUgdmFsdWUgaWYgdGhl
IHByb3BlcnR5IGlzIG1pc3NpbmcuDQo+ICAgLSBwY2lfZ2V0X25ld19kb21haW5fbnIoKTogdGhp
cyBmdW5jdGlvbiBpcyBvbmx5IHByZXNlbnQgaW4gdjEyIGFuZCBhbGxvY2F0ZXMNCj4gICAgIGEg
bmV3IGRvbWFpbiBudW1iZXIgZWFjaCB0aW1lIGl0IGlzIGNhbGxlZC4NCk9rLCBJIG1pc3NlZCB0
aGUgc3VidGxlIG5hbWUgY2hhbmdlIGFuZCBleHRyYSBmdW5jdGlvbiBmb3IgdjEyLg0KDQpTb3Vu
ZHMgZ29vZCB0byBtZSENCg0KVGhhbmtzDQpQaGlsDQo