Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

[+cc Jim, Krishna, Vidya, Shashank]

On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> PCIe controller driver.

> +++ b/drivers/pci/controller/pcie-mediatek-gen3.c

> +#define PCIE_EQ_PRESET_01_REG		0x100
> +#define PCIE_VAL_LN0_DOWNSTREAM		GENMASK(6, 0)
> +#define PCIE_VAL_LN0_UPSTREAM		GENMASK(14, 8)
> +#define PCIE_VAL_LN1_DOWNSTREAM		GENMASK(22, 16)
> +#define PCIE_VAL_LN1_UPSTREAM		GENMASK(30, 24)
> ...

> +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> +{
> ...

> +	val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> +	      FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> +	      FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> +	      FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> +	writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);

This looks like it might be for the Lane Equalization Control
registers (PCIe r6.0, sec 7.7.3.4)?

I would expect those values (0x47, 0x41) to be related to the platform
design, so maybe not completely determined by the SoC itself?  Jim and
Krishna have been working on DT schema for the equalization values,
which seems like the right place for them:

https://lore.kernel.org/linux-pci/20241018182247.41130-2-james.quinlan@broadcom.com/
https://lore.kernel.org/r/77d3a1a9-c22d-0fd3-5942-91b9a3d74a43@quicinc.com

Maybe that would be applicable here as well?  It would at least be
nice to use a common #define for the Lane Equalization Control
register offset from the capability base.

Although I see that no such #define exists in pci_regs.h, so I guess
there's nothing to do here yet.

The only users of equalization settings I could find so far are:

  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-tegra194.c?id=v6.11#n832
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-qcom-common.c?id=v6.12-rc1#n11
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-mediatek-gen3.c?id=v6.12-rc1#n909

Bjorn

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> PCIe controller driver.
> ...

> +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	int err;
> +	u32 val;
> +
> +	/*
> +	 * Wait for the time needed to complete the bulk assert in
> +	 * mtk_pcie_setup for EN7581 SoC.
> +	 */
> +	mdelay(PCIE_EN7581_RESET_TIME_MS);

It looks wrong to me to do the assert and deassert in different
places:

  mtk_pcie_setup
    reset_control_bulk_assert(pcie->phy_resets)        <--
    mtk_pcie_en7581_power_up
      mdelay(PCIE_EN7581_RESET_TIME_MS)
      reset_control_bulk_deassert(pcie->phy_resets)    <--
      mdelay(PCIE_EN7581_RESET_TIME_MS)

That makes the code hard to understand.

> +	err = phy_init(pcie->phy);
> +	if (err) {
> +		dev_err(dev, "failed to initialize PHY\n");
> +		return err;
> +	}
> +
> +	err = phy_power_on(pcie->phy);
> +	if (err) {
> +		dev_err(dev, "failed to power on PHY\n");
> +		goto err_phy_on;
> +	}
> +
> +	err = reset_control_bulk_deassert(pcie->soc->phy_resets.num_resets, pcie->phy_resets);
> +	if (err) {
> +		dev_err(dev, "failed to deassert PHYs\n");
> +		goto err_phy_deassert;
> +	}
> +
> +	/*
> +	 * Wait for the time needed to complete the bulk de-assert above.
> +	 * This time is specific for EN7581 SoC.
> +	 */
> +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> +
> +	pm_runtime_enable(dev);
> +	pm_runtime_get_sync(dev);
> +

> +	err = clk_bulk_prepare(pcie->num_clks, pcie->clks);
> +	if (err) {
> +		dev_err(dev, "failed to prepare clock\n");
> +		goto err_clk_prepare;
> +	}
> +
> +	val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> +	      FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> +	      FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> +	      FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> +	writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);
> +
> +	val = PCIE_K_PHYPARAM_QUERY | PCIE_K_QUERY_TIMEOUT |
> +	      FIELD_PREP(PCIE_K_PRESET_TO_USE_16G, 0x80) |
> +	      FIELD_PREP(PCIE_K_PRESET_TO_USE, 0x2) |
> +	      FIELD_PREP(PCIE_K_FINETUNE_MAX, 0xf);
> +	writel_relaxed(val, pcie->base + PCIE_PIPE4_PIE8_REG);

Why is this equalization stuff in the middle between
clk_bulk_prepare() and clk_bulk_enable()?  Is the split an actual
requirement, or could we use clk_bulk_prepare_enable() here, like we
do in mtk_pcie_power_up()?

If the split is required, a comment about why would be helpful.

> +	err = clk_bulk_enable(pcie->num_clks, pcie->clks);
> +	if (err) {
> +		dev_err(dev, "failed to prepare clock\n");
> +		goto err_clk_enable;
> +	}

Per https://lore.kernel.org/r/ZypgYOn7dcYIoW4i@lore-desk,
REG_PCI_CONTROL is asserted/deasserted here by en7581_pci_enable().

Is this where PERST# is asserted?  If so, a comment to that effect
would be helpful.  Where is PERST# deasserted?  Where are the required
delays before deassert done?

Bjorn

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 4118 bytes --]

> On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > PCIe controller driver.
> > ...
> 
> > +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> > +{
> > +	struct device *dev = pcie->dev;
> > +	int err;
> > +	u32 val;
> > +
> > +	/*
> > +	 * Wait for the time needed to complete the bulk assert in
> > +	 * mtk_pcie_setup for EN7581 SoC.
> > +	 */
> > +	mdelay(PCIE_EN7581_RESET_TIME_MS);

Hi Bjorn,

> 
> It looks wrong to me to do the assert and deassert in different
> places:
> 
>   mtk_pcie_setup
>     reset_control_bulk_assert(pcie->phy_resets)        <--
>     mtk_pcie_en7581_power_up
>       mdelay(PCIE_EN7581_RESET_TIME_MS)
>       reset_control_bulk_deassert(pcie->phy_resets)    <--
>       mdelay(PCIE_EN7581_RESET_TIME_MS)
> 
> That makes the code hard to understand.

The phy reset line was already asserted running reset_control_assert() in
mtk_pcie_setup() and de-asserted running reset_control_deassert() in
mtk_pcie_power_up() before adding EN7581 support. Moreover, EN7581 requires
to run phy_init()/phy_power_on() before de-asserting the phy reset lines.
I guess I can add a comment to make it more clear. Agree?

> 
> > +	err = phy_init(pcie->phy);
> > +	if (err) {
> > +		dev_err(dev, "failed to initialize PHY\n");
> > +		return err;
> > +	}
> > +
> > +	err = phy_power_on(pcie->phy);
> > +	if (err) {
> > +		dev_err(dev, "failed to power on PHY\n");
> > +		goto err_phy_on;
> > +	}
> > +
> > +	err = reset_control_bulk_deassert(pcie->soc->phy_resets.num_resets, pcie->phy_resets);
> > +	if (err) {
> > +		dev_err(dev, "failed to deassert PHYs\n");
> > +		goto err_phy_deassert;
> > +	}
> > +
> > +	/*
> > +	 * Wait for the time needed to complete the bulk de-assert above.
> > +	 * This time is specific for EN7581 SoC.
> > +	 */
> > +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> > +
> > +	pm_runtime_enable(dev);
> > +	pm_runtime_get_sync(dev);
> > +
> 
> > +	err = clk_bulk_prepare(pcie->num_clks, pcie->clks);
> > +	if (err) {
> > +		dev_err(dev, "failed to prepare clock\n");
> > +		goto err_clk_prepare;
> > +	}
> > +
> > +	val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> > +	      FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> > +	      FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> > +	      FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> > +	writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);
> > +
> > +	val = PCIE_K_PHYPARAM_QUERY | PCIE_K_QUERY_TIMEOUT |
> > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE_16G, 0x80) |
> > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE, 0x2) |
> > +	      FIELD_PREP(PCIE_K_FINETUNE_MAX, 0xf);
> > +	writel_relaxed(val, pcie->base + PCIE_PIPE4_PIE8_REG);
> 
> Why is this equalization stuff in the middle between
> clk_bulk_prepare() and clk_bulk_enable()?  Is the split an actual
> requirement, or could we use clk_bulk_prepare_enable() here, like we
> do in mtk_pcie_power_up()?

Nope, we can replace clk_bulk_enable() with clk_bulk_prepare_enable() and
remove clk_bulk_prepare() in mtk_pcie_en7581_power_up() since we actually
implements just enable callback for EN7581 in clk-en7523.c.

> 
> If the split is required, a comment about why would be helpful.
> 
> > +	err = clk_bulk_enable(pcie->num_clks, pcie->clks);
> > +	if (err) {
> > +		dev_err(dev, "failed to prepare clock\n");
> > +		goto err_clk_enable;
> > +	}
> 
> Per https://lore.kernel.org/r/ZypgYOn7dcYIoW4i@lore-desk,
> REG_PCI_CONTROL is asserted/deasserted here by en7581_pci_enable().

correct

> 
> Is this where PERST# is asserted?  If so, a comment to that effect
> would be helpful.  Where is PERST# deasserted?  Where are the required
> delays before deassert done?

I can add a comment in en7581_pci_enable() describing the PERST issue for
EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
configuring REG_PCI_CONTROL register.

https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396

Regards,
Lorenzo

> 
> Bjorn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Jim Quinlan]

On Tue, Nov 5, 2024 at 4:33 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Jim, Krishna, Vidya, Shashank]
>
> On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > PCIe controller driver.
>
> > +++ b/drivers/pci/controller/pcie-mediatek-gen3.c
>
> > +#define PCIE_EQ_PRESET_01_REG                0x100
> > +#define PCIE_VAL_LN0_DOWNSTREAM              GENMASK(6, 0)
> > +#define PCIE_VAL_LN0_UPSTREAM                GENMASK(14, 8)
> > +#define PCIE_VAL_LN1_DOWNSTREAM              GENMASK(22, 16)
> > +#define PCIE_VAL_LN1_UPSTREAM                GENMASK(30, 24)
> > ...
>
> > +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> > +{
> > ...
>
> > +     val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> > +           FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> > +           FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> > +           FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> > +     writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);

Not sure it is worth the trouble to define fields.  In fact, you are
already combining fields (rec+trans) so why not go further and just
write each lane as a u16?
>
> This looks like it might be for the Lane Equalization Control
> registers (PCIe r6.0, sec 7.7.3.4)?
>
> I would expect those values (0x47, 0x41) to be related to the platform
> design, so maybe not completely determined by the SoC itself?  Jim and
> Krishna have been working on DT schema for the equalization values,
> which seems like the right place for them:
>
> https://lore.kernel.org/linux-pci/20241018182247.41130-2-james.quinlan@broadcom.com/
> https://lore.kernel.org/r/77d3a1a9-c22d-0fd3-5942-91b9a3d74a43@quicinc.com
>
> Maybe that would be applicable here as well?  It would at least be
> nice to use a common #define for the Lane Equalization Control
> register offset from the capability base.

FWIW, these registers are HwInit/RO.  In our (Broadcom) case we have
to write them using an internal  register block that is not visible in
the config space.  In other words, we do not use the cap offset.

Regards,
Jim
Broadcom STB/CM
>
> Although I see that no such #define exists in pci_regs.h, so I guess
> there's nothing to do here yet.
>
> The only users of equalization settings I could find so far are:
>
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-tegra194.c?id=v6.11#n832
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-qcom-common.c?id=v6.12-rc1#n11
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-mediatek-gen3.c?id=v6.12-rc1#n909
>
> Bjorn
>

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > PCIe controller driver.
> > > ...
> > 
> > > +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> > > +{
> > > +	struct device *dev = pcie->dev;
> > > +	int err;
> > > +	u32 val;
> > > +
> > > +	/*
> > > +	 * Wait for the time needed to complete the bulk assert in
> > > +	 * mtk_pcie_setup for EN7581 SoC.
> > > +	 */
> > > +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> 
> > It looks wrong to me to do the assert and deassert in different
> > places:
> > 
> >   mtk_pcie_setup
> >     reset_control_bulk_assert(pcie->phy_resets)        <--
> >     mtk_pcie_en7581_power_up
> >       mdelay(PCIE_EN7581_RESET_TIME_MS)
> >       reset_control_bulk_deassert(pcie->phy_resets)    <--
> >       mdelay(PCIE_EN7581_RESET_TIME_MS)
> > 
> > That makes the code hard to understand.
> 
> The phy reset line was already asserted running reset_control_assert() in
> mtk_pcie_setup() and de-asserted running reset_control_deassert() in
> mtk_pcie_power_up() before adding EN7581 support. Moreover, EN7581 requires
> to run phy_init()/phy_power_on() before de-asserting the phy reset lines.
> I guess I can add a comment to make it more clear. Agree?

I assume the first deassert(phy_resets) in mtk_pcie_setup() is not
paired with anything in this driver.

I think it would be better to pair the other assert/deasserts in the
same functions like the below.  Then it's easy to see the matching.

While looking at this, I noticed that we assert(mac_reset) in
mtk_pcie_setup(), but it's never deasserted for EN7581.

  mtk_pcie_setup
    reset_control_bulk_deassert(phy_resets)
    mtk_pcie_en7581_power_up
      reset_control_bulk_assert(phy_resets)  # move here
      reset_control_assert(mac_reset)        # move here
      mdelay(PCIE_EN7581_RESET_TIME_MS)
      phy_init
      phy_power_on
      reset_control_deassert(mac_reset)      # add; seems missing?
      reset_control_bulk_deassert(phy_resets)
      mdelay(PCIE_EN7581_RESET_TIME_MS)

  mtk_pcie_setup
    reset_control_bulk_deassert(phy_resets)
    mtk_pcie_power_up
      reset_control_bulk_assert(phy_resets)  # move here
      reset_control_assert(mac_reset)        # move here
      reset_control_bulk_deassert(phy_resets)
      phy_init
      phy_power_on
      reset_control_deassert(mac_reset)

> > > +	err = phy_init(pcie->phy);
> > > +	if (err) {
> > > +		dev_err(dev, "failed to initialize PHY\n");
> > > +		return err;
> > > +	}
> > > +
> > > +	err = phy_power_on(pcie->phy);
> > > +	if (err) {
> > > +		dev_err(dev, "failed to power on PHY\n");
> > > +		goto err_phy_on;
> > > +	}
> > > +
> > > +	err = reset_control_bulk_deassert(pcie->soc->phy_resets.num_resets, pcie->phy_resets);
> > > +	if (err) {
> > > +		dev_err(dev, "failed to deassert PHYs\n");
> > > +		goto err_phy_deassert;
> > > +	}
> > > +
> > > +	/*
> > > +	 * Wait for the time needed to complete the bulk de-assert above.
> > > +	 * This time is specific for EN7581 SoC.
> > > +	 */
> > > +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> > > +
> > > +	pm_runtime_enable(dev);
> > > +	pm_runtime_get_sync(dev);
> > > +
> > 
> > > +	err = clk_bulk_prepare(pcie->num_clks, pcie->clks);
> > > +	if (err) {
> > > +		dev_err(dev, "failed to prepare clock\n");
> > > +		goto err_clk_prepare;
> > > +	}
> > > +
> > > +	val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> > > +	      FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> > > +	      FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> > > +	      FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> > > +	writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);
> > > +
> > > +	val = PCIE_K_PHYPARAM_QUERY | PCIE_K_QUERY_TIMEOUT |
> > > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE_16G, 0x80) |
> > > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE, 0x2) |
> > > +	      FIELD_PREP(PCIE_K_FINETUNE_MAX, 0xf);
> > > +	writel_relaxed(val, pcie->base + PCIE_PIPE4_PIE8_REG);
> > 
> > Why is this equalization stuff in the middle between
> > clk_bulk_prepare() and clk_bulk_enable()?  Is the split an actual
> > requirement, or could we use clk_bulk_prepare_enable() here, like we
> > do in mtk_pcie_power_up()?
> 
> Nope, we can replace clk_bulk_enable() with clk_bulk_prepare_enable() and
> remove clk_bulk_prepare() in mtk_pcie_en7581_power_up() since we actually
> implements just enable callback for EN7581 in clk-en7523.c.
> 
> > If the split is required, a comment about why would be helpful.
> > 
> > > +	err = clk_bulk_enable(pcie->num_clks, pcie->clks);
> > > +	if (err) {
> > > +		dev_err(dev, "failed to prepare clock\n");
> > > +		goto err_clk_enable;
> > > +	}
> > 
> > Per https://lore.kernel.org/r/ZypgYOn7dcYIoW4i@lore-desk,
> > REG_PCI_CONTROL is asserted/deasserted here by en7581_pci_enable().
> 
> correct
> 
> > Is this where PERST# is asserted?  If so, a comment to that effect
> > would be helpful.  Where is PERST# deasserted?  Where are the required
> > delays before deassert done?
> 
> I can add a comment in en7581_pci_enable() describing the PERST issue for
> EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> configuring REG_PCI_CONTROL register.
> 
> https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396

Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
required PCI delays in this driver if possible so it's easy to verify
that they are all covered.

Bjorn

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Wed, Nov 06, 2024 at 06:00:08PM -0500, Jim Quinlan wrote:
> On Tue, Nov 5, 2024 at 4:33 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > PCIe controller driver.
> >
> > > +++ b/drivers/pci/controller/pcie-mediatek-gen3.c
> >
> > > +#define PCIE_EQ_PRESET_01_REG                0x100
> > > +#define PCIE_VAL_LN0_DOWNSTREAM              GENMASK(6, 0)
> > > +#define PCIE_VAL_LN0_UPSTREAM                GENMASK(14, 8)
> > > +#define PCIE_VAL_LN1_DOWNSTREAM              GENMASK(22, 16)
> > > +#define PCIE_VAL_LN1_UPSTREAM                GENMASK(30, 24)
> > > ...
> >
> > > +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> > > +{
> > > ...
> >
> > > +     val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> > > +           FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> > > +           FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> > > +           FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> > > +     writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);
> 
> Not sure it is worth the trouble to define fields.  In fact, you are
> already combining fields (rec+trans) so why not go further and just
> write each lane as a u16?
> >
> > This looks like it might be for the Lane Equalization Control
> > registers (PCIe r6.0, sec 7.7.3.4)?
> >
> > I would expect those values (0x47, 0x41) to be related to the platform
> > design, so maybe not completely determined by the SoC itself?  Jim and
> > Krishna have been working on DT schema for the equalization values,
> > which seems like the right place for them:
> >
> > https://lore.kernel.org/linux-pci/20241018182247.41130-2-james.quinlan@broadcom.com/
> > https://lore.kernel.org/r/77d3a1a9-c22d-0fd3-5942-91b9a3d74a43@quicinc.com
> >
> > Maybe that would be applicable here as well?  It would at least be
> > nice to use a common #define for the Lane Equalization Control
> > register offset from the capability base.
> 
> FWIW, these registers are HwInit/RO.  In our (Broadcom) case we have
> to write them using an internal register block that is not visible in
> the config space.  In other words, we do not use the cap offset.

Good point.  It looks like they're a mix of HwInit/RsvdP and
Hwinit/RO.  RsvdP is for writes, so I guess the config space registers
must be write-once and subsequently read-only until reset.  In any
case, mtk is using an internal register block as well, so a cap offset
wouldn't be useful.

Maybe it would still be worthwhile to define the fields themselves in
pci_regs.h so we can someday have common code to parse the DT
properties and assemble them.  Although I suppose there's no
requirement that the registers in the internal block even be laid out
the same as the config space register.

> > Although I see that no such #define exists in pci_regs.h, so I guess
> > there's nothing to do here yet.
> >
> > The only users of equalization settings I could find so far are:
> >
> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-tegra194.c?id=v6.11#n832
> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-qcom-common.c?id=v6.12-rc1#n11
> >   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pcie-mediatek-gen3.c?id=v6.12-rc1#n909
> >
> > Bjorn
> >

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 6387 bytes --]

> On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > > PCIe controller driver.
> > > > ...
> > > 
> > > > +static int mtk_pcie_en7581_power_up(struct mtk_gen3_pcie *pcie)
> > > > +{
> > > > +	struct device *dev = pcie->dev;
> > > > +	int err;
> > > > +	u32 val;
> > > > +
> > > > +	/*
> > > > +	 * Wait for the time needed to complete the bulk assert in
> > > > +	 * mtk_pcie_setup for EN7581 SoC.
> > > > +	 */
> > > > +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> > 
> > > It looks wrong to me to do the assert and deassert in different
> > > places:
> > > 
> > >   mtk_pcie_setup
> > >     reset_control_bulk_assert(pcie->phy_resets)        <--
> > >     mtk_pcie_en7581_power_up
> > >       mdelay(PCIE_EN7581_RESET_TIME_MS)
> > >       reset_control_bulk_deassert(pcie->phy_resets)    <--
> > >       mdelay(PCIE_EN7581_RESET_TIME_MS)
> > > 
> > > That makes the code hard to understand.
> > 
> > The phy reset line was already asserted running reset_control_assert() in
> > mtk_pcie_setup() and de-asserted running reset_control_deassert() in
> > mtk_pcie_power_up() before adding EN7581 support. Moreover, EN7581 requires
> > to run phy_init()/phy_power_on() before de-asserting the phy reset lines.
> > I guess I can add a comment to make it more clear. Agree?
> 
> I assume the first deassert(phy_resets) in mtk_pcie_setup() is not
> paired with anything in this driver.

correct

> 
> I think it would be better to pair the other assert/deasserts in the
> same functions like the below.  Then it's easy to see the matching.

ack, I will post a fix for it

> 
> While looking at this, I noticed that we assert(mac_reset) in
> mtk_pcie_setup(), but it's never deasserted for EN7581.

ack, I will post a fix for it

> 
>   mtk_pcie_setup
>     reset_control_bulk_deassert(phy_resets)
>     mtk_pcie_en7581_power_up
>       reset_control_bulk_assert(phy_resets)  # move here
>       reset_control_assert(mac_reset)        # move here
>       mdelay(PCIE_EN7581_RESET_TIME_MS)
>       phy_init
>       phy_power_on
>       reset_control_deassert(mac_reset)      # add; seems missing?
>       reset_control_bulk_deassert(phy_resets)
>       mdelay(PCIE_EN7581_RESET_TIME_MS)
> 
>   mtk_pcie_setup
>     reset_control_bulk_deassert(phy_resets)
>     mtk_pcie_power_up
>       reset_control_bulk_assert(phy_resets)  # move here
>       reset_control_assert(mac_reset)        # move here
>       reset_control_bulk_deassert(phy_resets)
>       phy_init
>       phy_power_on
>       reset_control_deassert(mac_reset)
> 
> > > > +	err = phy_init(pcie->phy);
> > > > +	if (err) {
> > > > +		dev_err(dev, "failed to initialize PHY\n");
> > > > +		return err;
> > > > +	}
> > > > +
> > > > +	err = phy_power_on(pcie->phy);
> > > > +	if (err) {
> > > > +		dev_err(dev, "failed to power on PHY\n");
> > > > +		goto err_phy_on;
> > > > +	}
> > > > +
> > > > +	err = reset_control_bulk_deassert(pcie->soc->phy_resets.num_resets, pcie->phy_resets);
> > > > +	if (err) {
> > > > +		dev_err(dev, "failed to deassert PHYs\n");
> > > > +		goto err_phy_deassert;
> > > > +	}
> > > > +
> > > > +	/*
> > > > +	 * Wait for the time needed to complete the bulk de-assert above.
> > > > +	 * This time is specific for EN7581 SoC.
> > > > +	 */
> > > > +	mdelay(PCIE_EN7581_RESET_TIME_MS);
> > > > +
> > > > +	pm_runtime_enable(dev);
> > > > +	pm_runtime_get_sync(dev);
> > > > +
> > > 
> > > > +	err = clk_bulk_prepare(pcie->num_clks, pcie->clks);
> > > > +	if (err) {
> > > > +		dev_err(dev, "failed to prepare clock\n");
> > > > +		goto err_clk_prepare;
> > > > +	}
> > > > +
> > > > +	val = FIELD_PREP(PCIE_VAL_LN0_DOWNSTREAM, 0x47) |
> > > > +	      FIELD_PREP(PCIE_VAL_LN1_DOWNSTREAM, 0x47) |
> > > > +	      FIELD_PREP(PCIE_VAL_LN0_UPSTREAM, 0x41) |
> > > > +	      FIELD_PREP(PCIE_VAL_LN1_UPSTREAM, 0x41);
> > > > +	writel_relaxed(val, pcie->base + PCIE_EQ_PRESET_01_REG);
> > > > +
> > > > +	val = PCIE_K_PHYPARAM_QUERY | PCIE_K_QUERY_TIMEOUT |
> > > > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE_16G, 0x80) |
> > > > +	      FIELD_PREP(PCIE_K_PRESET_TO_USE, 0x2) |
> > > > +	      FIELD_PREP(PCIE_K_FINETUNE_MAX, 0xf);
> > > > +	writel_relaxed(val, pcie->base + PCIE_PIPE4_PIE8_REG);
> > > 
> > > Why is this equalization stuff in the middle between
> > > clk_bulk_prepare() and clk_bulk_enable()?  Is the split an actual
> > > requirement, or could we use clk_bulk_prepare_enable() here, like we
> > > do in mtk_pcie_power_up()?
> > 
> > Nope, we can replace clk_bulk_enable() with clk_bulk_prepare_enable() and
> > remove clk_bulk_prepare() in mtk_pcie_en7581_power_up() since we actually
> > implements just enable callback for EN7581 in clk-en7523.c.
> > 
> > > If the split is required, a comment about why would be helpful.
> > > 
> > > > +	err = clk_bulk_enable(pcie->num_clks, pcie->clks);
> > > > +	if (err) {
> > > > +		dev_err(dev, "failed to prepare clock\n");
> > > > +		goto err_clk_enable;
> > > > +	}
> > > 
> > > Per https://lore.kernel.org/r/ZypgYOn7dcYIoW4i@lore-desk,
> > > REG_PCI_CONTROL is asserted/deasserted here by en7581_pci_enable().
> > 
> > correct
> > 
> > > Is this where PERST# is asserted?  If so, a comment to that effect
> > > would be helpful.  Where is PERST# deasserted?  Where are the required
> > > delays before deassert done?
> > 
> > I can add a comment in en7581_pci_enable() describing the PERST issue for
> > EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> > configuring REG_PCI_CONTROL register.
> > 
> > https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396
> 
> Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
> something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
> required PCI delays in this driver if possible so it's easy to verify
> that they are all covered.

IIRC I just used the delay value used in the vendor sdk. I do not have a strong
opinion about it but I guess if we move it in the pcie-mediatek-gen3 driver, we
will need to add it in each driver where this clock is used. What do you think?

Regards,
Lorenzo

> 
> Bjorn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > > > PCIe controller driver.
> > > > > ...

> > > > Is this where PERST# is asserted?  If so, a comment to that effect
> > > > would be helpful.  Where is PERST# deasserted?  Where are the required
> > > > delays before deassert done?
> > > 
> > > I can add a comment in en7581_pci_enable() describing the PERST issue for
> > > EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> > > configuring REG_PCI_CONTROL register.
> > > 
> > > https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396
> > 
> > Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
> > something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
> > required PCI delays in this driver if possible so it's easy to verify
> > that they are all covered.
> 
> IIRC I just used the delay value used in the vendor sdk. I do not
> have a strong opinion about it but I guess if we move it in the
> pcie-mediatek-gen3 driver, we will need to add it in each driver
> where this clock is used. What do you think?

I don't know what the 250ms delay is for.  If it is for a required PCI
delay, we should use the relevant standard #define for it, and it
should be in the PCI controller driver.  Otherwise it's impossible to
verify that all the drivers are doing the correct delays.

I don't know what other drivers are using that clock.  Are you
suggesting that it may be used in non-PCI situations where the
required delay might be different?  If another user requires 250ms,
but PCI requires only 100ms, I think it would be worth having separate
delays in each user so PCI wouldn't have to pay that extra 150ms.

Bjorn

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

On Nov 07, Bjorn Helgaas wrote:
> On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > > > > PCIe controller driver.
> > > > > > ...
> 
> > > > > Is this where PERST# is asserted?  If so, a comment to that effect
> > > > > would be helpful.  Where is PERST# deasserted?  Where are the required
> > > > > delays before deassert done?
> > > > 
> > > > I can add a comment in en7581_pci_enable() describing the PERST issue for
> > > > EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> > > > configuring REG_PCI_CONTROL register.
> > > > 
> > > > https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396
> > > 
> > > Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
> > > something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
> > > required PCI delays in this driver if possible so it's easy to verify
> > > that they are all covered.
> > 
> > IIRC I just used the delay value used in the vendor sdk. I do not
> > have a strong opinion about it but I guess if we move it in the
> > pcie-mediatek-gen3 driver, we will need to add it in each driver
> > where this clock is used. What do you think?
> 
> I don't know what the 250ms delay is for.  If it is for a required PCI
> delay, we should use the relevant standard #define for it, and it
> should be in the PCI controller driver.  Otherwise it's impossible to
> verify that all the drivers are doing the correct delays.

ack, fine to me. Do you prefer to keep 250ms after clk_bulk_prepare_enable()
in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
I can check if 100ms works properly.

Regards,
Lorenzo

> 
> I don't know what other drivers are using that clock.  Are you
> suggesting that it may be used in non-PCI situations where the
> required delay might be different?  If another user requires 250ms,
> but PCI requires only 100ms, I think it would be worth having separate
> delays in each user so PCI wouldn't have to pay that extra 150ms.
> 
> Bjorn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> On Nov 07, Bjorn Helgaas wrote:
> > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > > > > > PCIe controller driver.
> > > > > > > ...
> > 
> > > > > > Is this where PERST# is asserted?  If so, a comment to that effect
> > > > > > would be helpful.  Where is PERST# deasserted?  Where are the required
> > > > > > delays before deassert done?
> > > > > 
> > > > > I can add a comment in en7581_pci_enable() describing the PERST issue for
> > > > > EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> > > > > configuring REG_PCI_CONTROL register.
> > > > > 
> > > > > https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396
> > > > 
> > > > Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
> > > > something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
> > > > required PCI delays in this driver if possible so it's easy to verify
> > > > that they are all covered.
> > > 
> > > IIRC I just used the delay value used in the vendor sdk. I do not
> > > have a strong opinion about it but I guess if we move it in the
> > > pcie-mediatek-gen3 driver, we will need to add it in each driver
> > > where this clock is used. What do you think?
> > 
> > I don't know what the 250ms delay is for.  If it is for a required PCI
> > delay, we should use the relevant standard #define for it, and it
> > should be in the PCI controller driver.  Otherwise it's impossible to
> > verify that all the drivers are doing the correct delays.
> 
> ack, fine to me. Do you prefer to keep 250ms after clk_bulk_prepare_enable()
> in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> I can check if 100ms works properly.

It's not clear to me where the relevant events are for these chips.

Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec
2.2.1, is helpful.  It shows the required timings for Power Stable,
REFCLK Stable, PERST# deassert, etc.

Per sec 2.11.2, PERST# must be asserted for at least 100us (T_PERST),
PERST# must be asserted for at least 100ms after Power Stable
(T_PVPERL), and PERST# must be asserted for at least 100us after
REFCLK Stable.

It would be helpful if we could tell by reading the source where some
of these critical events happen, and that the relevant delays are
there.  For example, if PERST# is asserted/deasserted by
"clk_enable()" or similar, it's not at all obvious from the code, so
we should have a comment to that effect.

Bjorn

Re: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 3286 bytes --]

> On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> > On Nov 07, Bjorn Helgaas wrote:
> > > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > > Introduce support for Airoha EN7581 PCIe controller to mediatek-gen3
> > > > > > > > PCIe controller driver.
> > > > > > > > ...
> > > 
> > > > > > > Is this where PERST# is asserted?  If so, a comment to that effect
> > > > > > > would be helpful.  Where is PERST# deasserted?  Where are the required
> > > > > > > delays before deassert done?
> > > > > > 
> > > > > > I can add a comment in en7581_pci_enable() describing the PERST issue for
> > > > > > EN7581. Please note we have a 250ms delay in en7581_pci_enable() after
> > > > > > configuring REG_PCI_CONTROL register.
> > > > > > 
> > > > > > https://github.com/torvalds/linux/blob/master/drivers/clk/clk-en7523.c#L396
> > > > > 
> > > > > Does that 250ms delay correspond to a PCIe mandatory delay, e.g.,
> > > > > something like PCIE_T_PVPERL_MS?  I think it would be nice to have the
> > > > > required PCI delays in this driver if possible so it's easy to verify
> > > > > that they are all covered.
> > > > 
> > > > IIRC I just used the delay value used in the vendor sdk. I do not
> > > > have a strong opinion about it but I guess if we move it in the
> > > > pcie-mediatek-gen3 driver, we will need to add it in each driver
> > > > where this clock is used. What do you think?
> > > 
> > > I don't know what the 250ms delay is for.  If it is for a required PCI
> > > delay, we should use the relevant standard #define for it, and it
> > > should be in the PCI controller driver.  Otherwise it's impossible to
> > > verify that all the drivers are doing the correct delays.
> > 
> > ack, fine to me. Do you prefer to keep 250ms after clk_bulk_prepare_enable()
> > in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> > I can check if 100ms works properly.
> 
> It's not clear to me where the relevant events are for these chips.
> 
> Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec
> 2.2.1, is helpful.  It shows the required timings for Power Stable,
> REFCLK Stable, PERST# deassert, etc.
> 
> Per sec 2.11.2, PERST# must be asserted for at least 100us (T_PERST),
> PERST# must be asserted for at least 100ms after Power Stable
> (T_PVPERL), and PERST# must be asserted for at least 100us after
> REFCLK Stable.
> 
> It would be helpful if we could tell by reading the source where some
> of these critical events happen, and that the relevant delays are
> there.  For example, if PERST# is asserted/deasserted by
> "clk_enable()" or similar, it's not at all obvious from the code, so
> we should have a comment to that effect.

I reviewed the vendor sdk and it just do something like in clk_enable():

	...
	val = readl(0x88);
	writel(val | BIT(16) | BIT(29) | BIT(26), 0x88);
	/*wait link up*/
	mdelay(1000);
	...

@Hui.Ma: is it fine use msleep(100) (so PCIE_T_PVPERL_MS) instead of msleep(1000)
(so PCIE_LINK_RETRAIN_TIMEOUT_MS)?

Regards,
Lorenzo

> 
> Bjorn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

回复: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Hui Ma (马慧)]

> On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> > On Nov 07, Bjorn Helgaas wrote:
> > > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > > Introduce support for Airoha EN7581 PCIe controller to 
> > > > > > > > mediatek-gen3 PCIe controller driver.
> > > > > > > > ...
> > > 
> > > > > > > Is this where PERST# is asserted?  If so, a comment to 
> > > > > > > that effect would be helpful.  Where is PERST# deasserted?  
> > > > > > > Where are the required delays before deassert done?
> > > > > > 
> > > > > > I can add a comment in en7581_pci_enable() describing the 
> > > > > > PERST issue for EN7581. Please note we have a 250ms delay in 
> > > > > > en7581_pci_enable() after configuring REG_PCI_CONTROL register.
> > > > > > 
> > > > > > https://github.com/torvalds/linux/blob/master/drivers/clk/cl
> > > > > > k-en7523.c#L396
> > > > > 
> > > > > Does that 250ms delay correspond to a PCIe mandatory delay, 
> > > > > e.g., something like PCIE_T_PVPERL_MS?  I think it would be 
> > > > > nice to have the required PCI delays in this driver if 
> > > > > possible so it's easy to verify that they are all covered.
> > > > 
> > > > IIRC I just used the delay value used in the vendor sdk. I do 
> > > > not have a strong opinion about it but I guess if we move it in 
> > > > the
> > > > pcie-mediatek-gen3 driver, we will need to add it in each driver 
> > > > where this clock is used. What do you think?
> > > 
> > > I don't know what the 250ms delay is for.  If it is for a required 
> > > PCI delay, we should use the relevant standard #define for it, and 
> > > it should be in the PCI controller driver.  Otherwise it's 
> > > impossible to verify that all the drivers are doing the correct delays.
> > 
> > ack, fine to me. Do you prefer to keep 250ms after 
> > clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> > I can check if 100ms works properly.
> 
> It's not clear to me where the relevant events are for these chips.
> 
> Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec 
> 2.2.1, is helpful.  It shows the required timings for Power Stable, 
> REFCLK Stable, PERST# deassert, etc.
> 
> Per sec 2.11.2, PERST# must be asserted for at least 100us (T_PERST), 
> PERST# must be asserted for at least 100ms after Power Stable 
> (T_PVPERL), and PERST# must be asserted for at least 100us after 
> REFCLK Stable.
> 
> It would be helpful if we could tell by reading the source where some 
> of these critical events happen, and that the relevant delays are 
> there.  For example, if PERST# is asserted/deasserted by 
> "clk_enable()" or similar, it's not at all obvious from the code, so 
> we should have a comment to that effect.

>I reviewed the vendor sdk and it just do something like in clk_enable():
>
>	...
>	val = readl(0x88);
>	writel(val | BIT(16) | BIT(29) | BIT(26), 0x88);
>	/*wait link up*/
>	mdelay(1000);
>	...
>
>@Hui.Ma: is it fine use msleep(100) (so PCIE_T_PVPERL_MS) instead of msleep(1000) (so PCIE_LINK_RETRAIN_TIMEOUT_MS)?
Hi Lorenzo，
	I think msleep(1000) will be safer,because some device won't link up with msleep(100).
Regards,
Hui
>
>Regards,
>Lorenzo

> 
> Bjorn

Re: 回复: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Bjorn Helgaas]

On Fri, Nov 08, 2024 at 01:23:35AM +0000, Hui Ma (马慧) wrote:
> > On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> > > On Nov 07, Bjorn Helgaas wrote:
> > > > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > > > Introduce support for Airoha EN7581 PCIe controller to 
> > > > > > > > > mediatek-gen3 PCIe controller driver.
> > > > > > > > > ...
> > > > 
> > > > > > > > Is this where PERST# is asserted?  If so, a comment to 
> > > > > > > > that effect would be helpful.  Where is PERST# deasserted?  
> > > > > > > > Where are the required delays before deassert done?
> > > > > > > 
> > > > > > > I can add a comment in en7581_pci_enable() describing the 
> > > > > > > PERST issue for EN7581. Please note we have a 250ms delay in 
> > > > > > > en7581_pci_enable() after configuring REG_PCI_CONTROL register.
> > > > > > > 
> > > > > > > https://github.com/torvalds/linux/blob/master/drivers/clk/cl
> > > > > > > k-en7523.c#L396
> > > > > > 
> > > > > > Does that 250ms delay correspond to a PCIe mandatory delay, 
> > > > > > e.g., something like PCIE_T_PVPERL_MS?  I think it would be 
> > > > > > nice to have the required PCI delays in this driver if 
> > > > > > possible so it's easy to verify that they are all covered.
> > > > > 
> > > > > IIRC I just used the delay value used in the vendor sdk. I
> > > > > do not have a strong opinion about it but I guess if we move
> > > > > it in the pcie-mediatek-gen3 driver, we will need to add it
> > > > > in each driver where this clock is used. What do you think?
> > > > 
> > > > I don't know what the 250ms delay is for.  If it is for a required 
> > > > PCI delay, we should use the relevant standard #define for it, and 
> > > > it should be in the PCI controller driver.  Otherwise it's 
> > > > impossible to verify that all the drivers are doing the correct delays.
> > > 
> > > ack, fine to me. Do you prefer to keep 250ms after 
> > > clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> > > I can check if 100ms works properly.
> > 
> > It's not clear to me where the relevant events are for these chips.
> > 
> > Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec 
> > 2.2.1, is helpful.  It shows the required timings for Power Stable, 
> > REFCLK Stable, PERST# deassert, etc.
> > 
> > Per sec 2.11.2, PERST# must be asserted for at least 100us (T_PERST), 
> > PERST# must be asserted for at least 100ms after Power Stable 
> > (T_PVPERL), and PERST# must be asserted for at least 100us after 
> > REFCLK Stable.
> > 
> > It would be helpful if we could tell by reading the source where some 
> > of these critical events happen, and that the relevant delays are 
> > there.  For example, if PERST# is asserted/deasserted by 
> > "clk_enable()" or similar, it's not at all obvious from the code, so 
> > we should have a comment to that effect.
> 
> >I reviewed the vendor sdk and it just do something like in clk_enable():
> >
> >	...
> >	val = readl(0x88);
> >	writel(val | BIT(16) | BIT(29) | BIT(26), 0x88);
> >	/*wait link up*/
> >	mdelay(1000);
> >	...
> >
> >@Hui.Ma: is it fine use msleep(100) (so PCIE_T_PVPERL_MS) instead
> >of msleep(1000) (so PCIE_LINK_RETRAIN_TIMEOUT_MS)?
>
> 	I think msleep(1000) will be safer, because some device won't
> 	link up with msleep(100).

Do you have details about this?  I guess it only hurts mediatek, but
increasing the minimum time to bring up a PCI hierarchy by almost an
entire second is a pretty big deal.

If this delay corresponds to the required T_PVPERL delay and 100ms
isn't enough for some endpoints, those endpoints should fail with many
host controllers, not just mediatek, so I would suspect the mediatek
controller or a certain platform, not the endpoint itself.

If this corresponds to T_PVPERL and mediatek needs longer, I would
document that by using "PCIE_T_PVPERL_MS * 10" and adding a comment
about why (affected platform/device, hardware erratum, etc).

Bottom line, I don't really care what the value is, but I *would* like
to be able to read pcie-mediatek-gen3.c and see the point where PCI
power is stable, a delay of at least T_PVPERL, and where PERST# is
deasserted because that's the main timing requirement on software.

Bjorn

Re: 回复: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 5144 bytes --]

> On Fri, Nov 08, 2024 at 01:23:35AM +0000, Hui Ma (马慧) wrote:
> > > On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> > > > On Nov 07, Bjorn Helgaas wrote:
> > > > > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > > > > Introduce support for Airoha EN7581 PCIe controller to 
> > > > > > > > > > mediatek-gen3 PCIe controller driver.
> > > > > > > > > > ...
> > > > > 
> > > > > > > > > Is this where PERST# is asserted?  If so, a comment to 
> > > > > > > > > that effect would be helpful.  Where is PERST# deasserted?  
> > > > > > > > > Where are the required delays before deassert done?
> > > > > > > > 
> > > > > > > > I can add a comment in en7581_pci_enable() describing the 
> > > > > > > > PERST issue for EN7581. Please note we have a 250ms delay in 
> > > > > > > > en7581_pci_enable() after configuring REG_PCI_CONTROL register.
> > > > > > > > 
> > > > > > > > https://github.com/torvalds/linux/blob/master/drivers/clk/cl
> > > > > > > > k-en7523.c#L396
> > > > > > > 
> > > > > > > Does that 250ms delay correspond to a PCIe mandatory delay, 
> > > > > > > e.g., something like PCIE_T_PVPERL_MS?  I think it would be 
> > > > > > > nice to have the required PCI delays in this driver if 
> > > > > > > possible so it's easy to verify that they are all covered.
> > > > > > 
> > > > > > IIRC I just used the delay value used in the vendor sdk. I
> > > > > > do not have a strong opinion about it but I guess if we move
> > > > > > it in the pcie-mediatek-gen3 driver, we will need to add it
> > > > > > in each driver where this clock is used. What do you think?
> > > > > 
> > > > > I don't know what the 250ms delay is for.  If it is for a required 
> > > > > PCI delay, we should use the relevant standard #define for it, and 
> > > > > it should be in the PCI controller driver.  Otherwise it's 
> > > > > impossible to verify that all the drivers are doing the correct delays.
> > > > 
> > > > ack, fine to me. Do you prefer to keep 250ms after 
> > > > clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> > > > I can check if 100ms works properly.
> > > 
> > > It's not clear to me where the relevant events are for these chips.
> > > 
> > > Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec 
> > > 2.2.1, is helpful.  It shows the required timings for Power Stable, 
> > > REFCLK Stable, PERST# deassert, etc.
> > > 
> > > Per sec 2.11.2, PERST# must be asserted for at least 100us (T_PERST), 
> > > PERST# must be asserted for at least 100ms after Power Stable 
> > > (T_PVPERL), and PERST# must be asserted for at least 100us after 
> > > REFCLK Stable.
> > > 
> > > It would be helpful if we could tell by reading the source where some 
> > > of these critical events happen, and that the relevant delays are 
> > > there.  For example, if PERST# is asserted/deasserted by 
> > > "clk_enable()" or similar, it's not at all obvious from the code, so 
> > > we should have a comment to that effect.
> > 
> > >I reviewed the vendor sdk and it just do something like in clk_enable():
> > >
> > >	...
> > >	val = readl(0x88);
> > >	writel(val | BIT(16) | BIT(29) | BIT(26), 0x88);
> > >	/*wait link up*/
> > >	mdelay(1000);
> > >	...
> > >
> > >@Hui.Ma: is it fine use msleep(100) (so PCIE_T_PVPERL_MS) instead
> > >of msleep(1000) (so PCIE_LINK_RETRAIN_TIMEOUT_MS)?
> >
> > 	I think msleep(1000) will be safer, because some device won't
> > 	link up with msleep(100).
> 
> Do you have details about this?  I guess it only hurts mediatek, but
> increasing the minimum time to bring up a PCI hierarchy by almost an
> entire second is a pretty big deal.
> 
> If this delay corresponds to the required T_PVPERL delay and 100ms
> isn't enough for some endpoints, those endpoints should fail with many
> host controllers, not just mediatek, so I would suspect the mediatek
> controller or a certain platform, not the endpoint itself.
> 
> If this corresponds to T_PVPERL and mediatek needs longer, I would
> document that by using "PCIE_T_PVPERL_MS * 10" and adding a comment
> about why (affected platform/device, hardware erratum, etc).
> 
> Bottom line, I don't really care what the value is, but I *would* like
> to be able to read pcie-mediatek-gen3.c and see the point where PCI
> power is stable, a delay of at least T_PVPERL, and where PERST# is
> deasserted because that's the main timing requirement on software.

I run some testes using 100ms delay (PCIE_T_PVPERL_MS) after
clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() and it works fine for
me (I tested with a MT7915 WiFi PCIe nic connected to the PCIe sock).
Moreover, we already poll PCIE_LINK_STATUS_REG register to check the link
status in mtk_pcie_startup_port(), right? I guess we can proceed with 100ms
delay in mtk_pcie_en7581_power_up().

Regards,
Lorenzo

> 
> Bjorn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

回复: 回复: [PATCH v4 4/4] PCI: mediatek-gen3: Add Airoha EN7581 support

[Hui Ma (马慧)]

> On Fri, Nov 08, 2024 at 01:23:35AM +0000, Hui Ma (马慧) wrote:
> > > On Thu, Nov 07, 2024 at 05:21:45PM +0100, Lorenzo Bianconi wrote:
> > > > On Nov 07, Bjorn Helgaas wrote:
> > > > > On Thu, Nov 07, 2024 at 08:39:43AM +0100, Lorenzo Bianconi wrote:
> > > > > > > On Wed, Nov 06, 2024 at 11:40:28PM +0100, Lorenzo Bianconi wrote:
> > > > > > > > > On Wed, Jul 03, 2024 at 06:12:44PM +0200, Lorenzo Bianconi wrote:
> > > > > > > > > > Introduce support for Airoha EN7581 PCIe controller 
> > > > > > > > > > to
> > > > > > > > > > mediatek-gen3 PCIe controller driver.
> > > > > > > > > > ...
> > > > > 
> > > > > > > > > Is this where PERST# is asserted?  If so, a comment to 
> > > > > > > > > that effect would be helpful.  Where is PERST# deasserted?
> > > > > > > > > Where are the required delays before deassert done?
> > > > > > > > 
> > > > > > > > I can add a comment in en7581_pci_enable() describing 
> > > > > > > > the PERST issue for EN7581. Please note we have a 250ms 
> > > > > > > > delay in
> > > > > > > > en7581_pci_enable() after configuring REG_PCI_CONTROL register.
> > > > > > > > 
> > > > > > > > https://github.com/torvalds/linux/blob/master/drivers/cl
> > > > > > > > k/cl
> > > > > > > > k-en7523.c#L396
> > > > > > > 
> > > > > > > Does that 250ms delay correspond to a PCIe mandatory 
> > > > > > > delay, e.g., something like PCIE_T_PVPERL_MS?  I think it 
> > > > > > > would be nice to have the required PCI delays in this 
> > > > > > > driver if possible so it's easy to verify that they are all covered.
> > > > > > 
> > > > > > IIRC I just used the delay value used in the vendor sdk. I 
> > > > > > do not have a strong opinion about it but I guess if we move 
> > > > > > it in the pcie-mediatek-gen3 driver, we will need to add it 
> > > > > > in each driver where this clock is used. What do you think?
> > > > > 
> > > > > I don't know what the 250ms delay is for.  If it is for a 
> > > > > required PCI delay, we should use the relevant standard 
> > > > > #define for it, and it should be in the PCI controller driver.  
> > > > > Otherwise it's impossible to verify that all the drivers are doing the correct delays.
> > > > 
> > > > ack, fine to me. Do you prefer to keep 250ms after
> > > > clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() or just use PCIE_T_PVPERL_MS (100)?
> > > > I can check if 100ms works properly.
> > > 
> > > It's not clear to me where the relevant events are for these chips.
> > > 
> > > Do you have access to the PCIe CEM spec?  The diagram in r6.0, sec 
> > > 2.2.1, is helpful.  It shows the required timings for Power 
> > > Stable, REFCLK Stable, PERST# deassert, etc.
> > > 
> > > Per sec 2.11.2, PERST# must be asserted for at least 100us 
> > > (T_PERST), PERST# must be asserted for at least 100ms after Power 
> > > Stable (T_PVPERL), and PERST# must be asserted for at least 100us 
> > > after REFCLK Stable.
> > > 
> > > It would be helpful if we could tell by reading the source where 
> > > some of these critical events happen, and that the relevant delays 
> > > are there.  For example, if PERST# is asserted/deasserted by 
> > > "clk_enable()" or similar, it's not at all obvious from the code, 
> > > so we should have a comment to that effect.
> > 
> > >I reviewed the vendor sdk and it just do something like in clk_enable():
> > >
> > >	...
> > >	val = readl(0x88);
> > >	writel(val | BIT(16) | BIT(29) | BIT(26), 0x88);
> > >	/*wait link up*/
> > >	mdelay(1000);
> > >	...
> > >
> > >@Hui.Ma: is it fine use msleep(100) (so PCIE_T_PVPERL_MS) instead 
> > >of msleep(1000) (so PCIE_LINK_RETRAIN_TIMEOUT_MS)?
> >
> > 	I think msleep(1000) will be safer, because some device won't
> > 	link up with msleep(100).
>> 
>> Do you have details about this?  I guess it only hurts mediatek, but 
>> increasing the minimum time to bring up a PCI hierarchy by almost an 
>> entire second is a pretty big deal.
>> 
>> If this delay corresponds to the required T_PVPERL delay and 100ms 
>> isn't enough for some endpoints, those endpoints should fail with many 
>> host controllers, not just mediatek, so I would suspect the mediatek 
>> controller or a certain platform, not the endpoint itself.
>> 
>> If this corresponds to T_PVPERL and mediatek needs longer, I would 
>> document that by using "PCIE_T_PVPERL_MS * 10" and adding a comment 
>> about why (affected platform/device, hardware erratum, etc).
>> 
>> Bottom line, I don't really care what the value is, but I *would* like 
>> to be able to read pcie-mediatek-gen3.c and see the point where PCI 
>> power is stable, a delay of at least T_PVPERL, and where PERST# is 
>> deasserted because that's the main timing requirement on software.

>>I run some testes using 100ms delay (PCIE_T_PVPERL_MS) after
>>clk_bulk_prepare_enable() in mtk_pcie_en7581_power_up() and it works fine for me (I tested with a MT7915 WiFi PCIe nic connected to the PCIe sock).
>>Moreover, we already poll PCIE_LINK_STATUS_REG register to check the link status in mtk_pcie_startup_port(), right? I guess we can proceed with 100ms delay in mtk_pcie_en7581_power_up().
Yes.

Hi Lorenzo/Bjorn，
	After our internal discussion and tests, we confirmed that a 100ms delay is enough.



Regards,
Hui

>Regards,
>Lorenzo

> 
> Bjorn